Iterators and Generators
Build lazy data flows and reason precisely about consumption, state, and resource lifetime.
By the end, you’ll be able to…
- Distinguish an iterable from its iterator.
- Compose generators without accidentally exhausting them.
- Choose eager or lazy evaluation from the consumer contract.
01A stream has a position
An iterable promises that iter(value) can produce an iterator. An iterator keeps a current position and answers next() until it raises StopIteration. A list can produce several independent iterators; a generator object normally supplies itself. This difference matters when one function counts a stream and another function expects to read the same stream afterward.
Treat iterator ownership as part of a function contract. Say whether the function consumes its input, and accept an iterable when restarting is necessary. Turning a stream into a list makes repeated access possible, but changes memory use and delays the first result. Neither choice is automatically better: choose according to whether callers need replay, indexing, or one forward pass.
02Yield suspends a function
Calling a generator function creates an object without running its body. Each next() resumes the suspended frame, preserving local variables, until the next yield or return. This makes generators useful for parsing logs and producing batches: the consumer decides how quickly work proceeds, so intermediate results need not accumulate in a large collection.
The example strips lines, skips comments, and converts accepted values as they are requested. A malformed later line would raise an exception only when iteration reaches it. Lazy validation can surprise callers who expect construction to validate everything. For an all-or-nothing import, consume and validate the complete batch before committing any external changes.
03Compose without hiding boundaries
A generator pipeline can separate normalization, filtering, and aggregation. Each stage should have one clear responsibility and preserve useful error context. yield from delegates iteration to another iterable; it is not a request to collect that iterable. itertools.islice can bound consumption, including consumption of an otherwise infinite source, without building a large temporary list.
Laziness does not guarantee constant memory. A generator may retain a giant input list, an open file, or a growing cache in its suspended frame. Keep resource ownership explicit by opening a file in the same with block that consumes its lines. If a generator itself owns a resource, arrange deterministic closure when consumers stop early rather than relying on garbage collection.
04Test consumption as behavior
Check empty inputs, partial batches, invalid sizes, and exhaustion. When a generator validates an argument inside its body, the exception occurs during iteration, not when the function is called. A test should therefore consume it inside the exception assertion. If immediate validation is required, use an outer ordinary function that validates and returns an inner generator.
Choose a custom iterator class only when named state or additional operations clarify the interface. Most finite transformations are easier to read as generator functions. Avoid manually raising StopIteration inside a generator; finish with return. A for loop handles exhaustion for you, and the next(iterator, default) form handles an ordinary end-of-stream without a broad exception handler.
The ideas worth keeping.
- An iterable can create an iterator; an iterator owns a position.
- Laziness changes when work and errors happen.
- Document consumption and close resources where they are consumed.
Make it yours.
Type and test your code right here in the interactive workspace. Run your solution, check the output, and compare with the worked guide when ready.
Running totals
Write running_totals(values), yielding the cumulative total after every input. Demonstrate that an empty iterable yields nothing.
Explore a worked solution
def running_totals(values): total = 0 for value in values: total += value yield totalassert list(running_totals([3, -1, 5])) == [3, 2, 7]assert list(running_totals([])) == []The total belongs to the generator frame. Each input is consumed exactly once, and no output list is retained inside the function.
Take until a sentinel
Yield values before the first None without consuming values after that sentinel. Prove the remaining iterator still contains the final value.
Explore a worked solution
def before_none(values): for value in values: if value is None: return yield valuesource = iter([4, 9, None, 12])assert list(before_none(source)) == [4, 9]assert next(source) == 12Returning stops the generator. The sentinel is consumed to recognize the boundary, while subsequent values remain available to the owner of source.
What stayed with you?
Check the ideas while they’re fresh. Your answers stay in this page and feedback appears as you go.
Running totals
Write running_totals(values), yielding the cumulative total after every input. Demonstrate that an empty iterable yields nothing.
[Process exited with code 0 - No output]
Keep exploring.
The official references behind this chapter, when you want a second angle or the full details.
One chapter closer.
Reading is a start. Trying it yourself makes the connection.