Module 4.3 — Iterators and generators

Scroll Core ideas → worked snippets → Practice (quiz). Attempt each question before Discuss / reveal. Use any Python REPL, script, or notebook you like.

Core ideas — iteration protocol & lazy pipelines

Anything implementing **__iter__ returning iterator with __next__ (raising StopIteration when exhausted) participates in for loops uniformly—lists, tuples, range, dictionaries (keys default), generators.

Iterator vs iterable quick distinction

• Iterable: provides __iter__ (fresh iterator starting beginning).
• Iterator: __next__ stateful object—usually exhausted after one pass (unless reset/recreated).

Calling iter(x) obtains iterator; next(it) pulls one value.

Generator functions & yield

Writing yield inside def manufactures a generator function whose call returns generator iterator object—execution suspends at each yield preserving local variables & stack position.

Why care?

• Lazy evaluation — infinite conceptual streams (counting, log tailing) without materialising huge lists.
• Pipeline composition (map, filter, itertools) can stay streaming.

Generator expressions

Syntax like list comprehensions but parentheses: (expr for x in seq if cond). Feeds consumers without building intermediate list.

Hand-simulating a generator

The fastest way to believe that a generator suspends and resumes is to step through one by hand. Consider:

def count_upto(n):
    i = 1
    while i <= n:
        yield i
        i = i + 1

Calling count_upto(3) does not run the body yet — it hands back a generator object that is paused at the very top. Each next(...) runs until the next yield, then freezes again with all locals intact:

| Call | Resumes at | Runs until | Returns | State frozen as | |------|-----------|------------|---------|-----------------| | g = count_upto(3) | — | — | (a paused generator) | before line 1 | | next(g) | top | yield i with i=1 | 1 | i = 1 | | next(g) | after yield | i=2, yield i | 2 | i = 2 | | next(g) | after yield | i=3, yield i | 3 | i = 3 | | next(g) | after yield | i=4, guard 4 <= 3 false, function ends | raises StopIteration | exhausted |

A for loop does exactly this for you: it calls next repeatedly and stops cleanly when StopIteration appears. That is why you never write the StopIteration handling yourself in ordinary loops.

Pitfalls

• Exhausted generator does not rewind—call the function again or rebuild the expression.
• len(gen) is meaningless without materialising (there is no length until the items are consumed).
• Changing a dict's set of keys while looping over it raises RuntimeError—iterate over a copied list of keys (list(d)) if you must add or remove during the loop.

Further topics (recognition only)

In larger programs, generators also support close() and throw(), which coordinate with context managers to clean up partway through a stream. You do not need to write these yet — just recognise them when you meet them in library code.

Self-study checklist

• Hand-simulate next() calls on def g(): yield 1; yield 2 until StopIteration.
• Compare memory story list comp vs genexpr for million elements (qualitative).
• Know why iterating a generator twice needs calling the generator function again (two independent iterator objects), not looping the exhausted first one again.

Worked snippets — generators & iteration

Iterator protocol manually

it = iter([10, 20])
print(next(it))
print(next(it))
# next(it)  # StopIteration

Simple generator

def count_upto(n):
    i = 1
    while i <= n:
        yield i
        i += 1


for k in count_upto(3):
    print(k)

Generator expression vs list comprehension

squares_list = [x * x for x in range(5)]
squares_lazy = (x * x for x in range(5))
print(squares_list, list(squares_lazy))

Second line doesn’t compute until iteration / list() consumes.

Stateless pipeline chunk

def positives(nums):
    for n in nums:
        if n > 0:
            yield n


def scaled(seq, factor):
    for x in seq:
        yield x * factor


stream = scaled(positives([-1, 3, -2, 7]), 10)
print(list(stream))

Each stage yields onward—no big temporary list (for long streams).

Exhaustion behaviour

def once():
    yield 1


g = once()
print(list(g), list(g))   # second list empty — same exhausted iterator