Python Fundamentals for Interviews: Data Structures, Comprehensions, and Gotchas

Generators and Iterators In Depth

def read_large_file(path):
    with open(path) as f:
        for line in f:
            yield line.strip()

A generator function computes values on demand, one at a time, instead of building the whole result in memory up front like a list comprehension would. For a huge file, [line for line in open(path)] loads everything into memory; the generator version holds only the current line. The trade-off: a generator can only be iterated once — there's no going back to the start without recreating it.

How yield actually works

Calling a generator function doesn't run any of its code — it returns a generator object immediately. The function body only executes up to the next yield each time something asks for the next value (via next() or a for loop), and the entire local state — variables, the current position in the code — is frozen in between calls:

def counter():
    print("starting")
    n = 0
    while True:
        print(f"about to yield {n}")
        yield n
        n += 1

gen = counter()      # prints nothing yet — the function body hasn't run
next(gen)             # prints "starting", "about to yield 0" — returns 0
next(gen)             # prints "about to yield 1" — returns 1, n=0 was remembered

This is fundamentally different from a normal function, which forgets all of its local state the instant it returns.

Generator expressions

The comprehension syntax with () instead of [] creates a generator instead of a list, with the same lazy, one-pass behavior:

total = sum(x * x for x in range(1_000_000))  # never builds a 1,000,000-item list

Passing a generator expression directly as a function's only argument (as above) doesn't even need extra parentheses — sum(x * x for x in range(10)) is valid; two sets of parens are only required when there are other arguments too.

The iterator protocol, briefly

for x in obj works because obj implements the iterator protocol: __iter__ returns an iterator, and the iterator's __next__ produces the next value or raises StopIteration when exhausted. Every generator, list, dict, and string satisfies this protocol — it's why they can all be used interchangeably in a for loop despite being wildly different data structures internally. A generator is simply the easiest way to write something that follows this protocol without hand-writing a class with both methods.