Generators in the Library

Source: Fluent Python, 2nd Edition — Chapter 17 by Luciano Ramalho (generator expressions through iter(func, sentinel))

Part 1 built Sentence with yield. This half: when a genexp is enough, arithmetic progressions, itertools, yield from, reducing built-ins, and iter(callable, sentinel). Generators as coroutines (.send) wait for Chapter 19 — Beazley: producers vs consumers; do not mix.

When to use a generator expression

Vector used genexps in __eq__, __hash__, __add__, and friends: a listcomp would work and waste RAM. A genexp is a one-line factory; a generator function wins when the body needs statements or will become a coroutine.

Rule of thumb: more than a couple of lines → function.

If the genexp is the only argument, drop the extra parens: Vector(n * scalar for n in self). A second argument needs (genexp).

Generators are not only for walking collections. They can invent a series.

Arithmetic progression

range is a bounded integer AP. itertools.count is unbounded. Need Fraction / Decimal / a float step and an optional end? Yield begin, then begin + step * index (index, not repeated += step, to limit float drift). Coerce with type(begin + step)(begin).

That can be a class with __iter__, or a generator function aritprog_gen. If the class exists only to wrap __iter__, drop the class.

Never list(count()). Compose: takewhile(lambda n: n < end, count(first, step)). A factory without yield that returns that iterator is still a generator factory.

itertools (and friends) by job

Know the catalog or you will rewrite it. Groups:

Filter (same items, fewer of them): filter, filterfalse, dropwhile, takewhile, compress, islice. dropwhile skips while the predicate is true, then everything after. takewhile is the opposite stop.

Map: map, starmap, enumerate, accumulate (running sum, or min / operator.mul for factorials).

Merge: chain, chain.from_iterable, zip, zip_longest, product (lazy Cartesian, repeat=).

Expand: count, cycle, repeat, combinations, combinations_with_replacement, permutations.

Rearrange: groupby (input must already be clustered/sorted by key), tee (n independent iterators), reversed (sequence or __reversed__ only).

Pipe them: they take iterators and return iterators.

yield from

yield from sub_gen() suspends the delegating generator; values (and later, .send) go straight to the client. The expression value of yield from is the subgenerator’s return. Nested for it in xs: for i in it: yield i becomes for it in xs: yield from it (chain).

Tree walk: yield cls.__name__, level then yield from tree(sub, level+1). The for over empty __subclasses__() is the base case — no extra if.

wrap in the starter uses '\\n'.join inside a JS template — that becomes \n in Python. Subclass order of A is typically B then D.

Reducing functions

all / any / max / min / sum / functools.reduce fold an iterable to one value. all([]) is True, any([]) is False. They short-circuit: any on a generator can leave items unconsumed. sorted consumes everything into a list. They need a finite iterable.

iter(func, sentinel)

Two-arg iter: call func() with no args until it returns sentinel, then stop (do not yield the sentinel). Dice until 1. File blocks: iter(partial(f.read, 64), b''). Rebuild the iterator to start over.

draw should pop from the front of box. The sentinel is not included in the output.

A conversion script that must stream ISO and .mst files into JSON keeps one record in memory by making each format a generator (iter_iso_records / iter_mst_records) and one write_json loop. New formats add a reader, not a thicker writer.

typing.Iterable[T_co] and Iterator[T_co] are the generic spellings; .send is Chapter 19.