Python Generators and yield: Save Memory on Big Data

Imagine you have a log file with 50 million lines and you want to find the ones that mention "ERROR". If you load the whole file into a list first, you might run out of memory before you even start.

Generators solve this. They hand you one item at a time, only when you ask for it, and forget it once you've moved on. The memory they use stays small, no matter how much data flows through.

A list versus a generator

Compare these two ways to produce a million squares:

import sys

as_list = [n * n for n in range(1_000_000)]
as_generator = (n * n for n in range(1_000_000))

print("list size (bytes):", sys.getsizeof(as_list))
print("generator size (bytes):", sys.getsizeof(as_generator))

list size (bytes): 8448728
generator size (bytes): 200

Your exact numbers may differ a little depending on the Python version, but the gap will be just as dramatic.

The list holds every value at once. The generator holds almost nothing, because it only knows how to compute the next value. The only syntax difference is round brackets instead of square ones.

You can still loop over it or pass it to functions like sum:

print(sum(as_generator))

333332833333500000

Writing your own with yield

A generator expression is fine for one-liners. For anything bigger, write a generator function. It looks like a normal function, but uses yield instead of return.

def countdown(start):
    while start > 0:
        yield start
        start -= 1

for number in countdown(5):
    print(number)

5
4
3
2
1

When you call countdown(5), nothing runs yet. You get back a generator object. Each time the loop asks for the next value, Python runs the function until it hits yield, hands over that value, and pauses right there. Next time, it picks up exactly where it stopped, with all its local variables intact.

You can see this by calling next() yourself:

gen = countdown(3)
print(next(gen))
print(next(gen))
print(next(gen))

3
2
1

One more next(gen) would raise StopIteration, which is how Python signals "no more items." A for loop handles that for you quietly.

Reading a big file line by line

This is the use case that matters most in real projects. Here's the pattern, using a small in-memory stand-in for a file so you can run it anywhere:

import io

fake_file = io.StringIO(
    "INFO start\nERROR disk full\nINFO retry\nERROR timeout\nINFO done\n"
)

def error_lines(lines):
    for line in lines:
        if "ERROR" in line:
            yield line.strip()

for line in error_lines(fake_file):
    print(line)

ERROR disk full
ERROR timeout

With a real file, you'd use with open("app.log") as f: for line in error_lines(f):. Python reads one line at a time, so the file can be as large as your disk.

Chaining generators into a pipeline

Generators snap together nicely. Each stage takes an iterable and yields results, and nothing is computed until the final loop pulls values through.

def numbers():
    n = 1
    while True:        # endless stream
        yield n
        n += 1

def only_even(items):
    for item in items:
        if item % 2 == 0:
            yield item

def squared(items):
    for item in items:
        yield item * item

pipeline = squared(only_even(numbers()))

for _ in range(5):
    print(next(pipeline))

4
16
36
64
100

The numbers() generator never ends, and that's fine. Only the values we actually request ever get computed. You can't do that with a list.

To take the first few items from any iterator, use itertools.islice:

from itertools import islice

print(list(islice(squared(only_even(numbers())), 5)))

[4, 16, 36, 64, 100]

Generators are single-use

This catches everyone once. After a generator has been used up, it's empty.

gen = (n for n in range(3))
print(list(gen))
print(list(gen))

[0, 1, 2]
[]

The second call returns an empty list, with no error. If you need to loop twice, either create the generator again or store the results in a list.

Iterators in one minute

You've been using the iterator protocol this whole time. An iterable is anything you can loop over (lists, strings, files, generators). An iterator is the object doing the work, and it has a __next__ method. Generators are the easiest way to build your own iterator, with no class needed.

When a list is the better choice

Generators aren't always better. Use a list when:

• You need to loop over the data more than once.
• You need the length, indexing, or slicing (items[3]).
• The data is small anyway. A list of 200 items won't hurt anyone.

Use a generator when the data is large, comes in a stream, or when you may stop early (like "find the first match").

Key takeaways

• Generators produce values one at a time, so memory use stays flat.
• yield pauses a function and remembers where it was.
• Chain generators to build clean data pipelines.
• A generator can only be consumed once. Don't expect a second pass.