Iterables, Iterators, and Generators

Source: Fluent Python, 2nd Edition — Chapter 17 by Luciano Ramalho (through generator expressions)

Paul Graham: repeated shapes in code often mean you are expanding by hand a macro you should have written. Python has no Lisp macros; it added yield (2.2) so the Iterator pattern lives in the language. You almost never write a full iterator class.

range is lazy; list(range(100)) is the explicit list. for, comprehensions, unpacking, * args, list(x), and line-by-line files all sit on iterators. This half of the chapter: iter(), iterable vs iterator, the classic pattern, generator functions, laziness, generator expressions. yield from, stdlib generators, and coroutines vs generators come later.

A sequence of words

Sentence(text) iterates word by word. Take #1 is a sequence: re.findall, __getitem__, __len__. Sequences are iterable because iter(x) does this:

1. If __iter__ exists, call it (must return an iterator).
2. Else if __getitem__ exists, build a 0-based indexer (legacy; still works; new types should implement __iter__).
3. Else TypeError: C object is not iterable.

abc.Iterable only looks for __iter__ (__subclasshook__). Sentence with only __getitem__ is iterable in practice but not isinstance(..., Iterable). Most accurate check: iter(x) and catch TypeError. Skip the check if you are about to loop anyway.

Iterable vs iterator

An iterable is anything iter can hook. An iterator has __next__ (raise StopIteration when done) and __iter__ returning self. for is it = iter(s) then next until StopIteration.

abc.Iterator ⊂ Iterable. Do not call __next__ by hand; use next(it). isinstance(x, abc.Iterator) is the practical iterator test (__subclasshook__ wants both __next__ and __iter__). types.py says: iterators are a protocol, not a type.

An exhausted iterator stays empty. iter(it) does not reset it. Call iter(original_iterable) again. There is no has_next except trying next.

Anti-pattern: putting __next__ on the collection so one Sentence is its own iterator. GoF iterators exist so you can run two independent walks. iter(s) must return a new iterator each time. Collections implement __iter__, not __next__. Iterators implement both.

Quiz

iter and next

1 / 1

After it = iter(s); list(it); list(it) the second list is

Pick an answer to see if you are right.

Take #2: a classic iterator class

Sentence.__iter__ returns SentenceIterator(words) with an index. Lots of state machinery. Works; not idiomatic.

Take #3: a generator function

Any function with yield is a generator factory: calling it returns a generator object (an iterator). Sentence.__iter__ can for word in self.words: yield word. No iterator class. Falling off the end raises StopIteration; you do not raise it yourself. Say the generator yields values; the function returns a generator.

def gen_123():
    yield 1
    yield 2
    yield 3

next runs to the next yield and pauses. Prints in the body interleave with the consumer (gen_AB).

If Sentence were its own iterator, the second join would be empty.

Lazy sentences

Eager __init__ + findall builds every word even if you only need the first. re.finditer yields match objects. __init__ keeps only text; __iter__ yields match.group().

A generator expression is a lazy list comprehension: (x*3 for x in gen_AB()) does not run gen_AB until you iterate. __iter__ can return (m.group() for m in RE_WORD.finditer(self.text)) with no yield in that method — it still returns a generator.

Genexps are sugar; a generator function is always legal and often clearer when the body is more than one expression.

The starter still assigns self.words — tests pass only after you drop that cache and iterate self.text.