Skip to main content

map, tqdm, and HTTP Errors

Source: Fluent Python, 2nd Edition — Chapter 21 by Luciano Ramalho (Executor.map demo through flags2 / chapter summary)

21a stopped at map preserving input order. This half: when that generator blocks, why submit + as_completed is the more general tool, then flags2 — 404 vs other HTTP, tqdm, and a Future → country dict.

No live HTTP or 4-second sleep in the sandbox. Do not flood fluentpython.com.

demo_executor_map.py​

Three workers, executor.map(loiter, range(5)). loiter(n) prints, sleep(n), returns n * 10. map returns a generator immediately — the repr is <generator object result_iterator ...>. That call does not wait for five sleeps.

Only three jobs start: loiter(0), (1), (2). loiter(0) can finish in the same second and free a worker for loiter(3) before you even iterate. loiter(4) waits for another slot (in the sample, after loiter(1) done at +1 s).

for i, result in enumerate(results): is where you block. Each next(results) is Future.result() for the next input slot, not “whoever just finished.” Slot 0 (loiter(0)) is often already done. Slot 1 waits until loiter(1)’s 1 s sleep ends. After result 2: 20 you may wait extra for loiter(4) (started late, sleeps 4 s) even if you already conceptually “finished” 3.

Play with max_workers and the delay list on your machine. max_workers=1 makes the generator’s order and the clocks line up like a for loop.

map vs submit + as_completed​

map: one callable, many arguments, results in argument order.

submit / as_completed: different callables, mix Thread and Process pool Futures in one as_completed(...), consume finish order. flags2 needs that so a progress bar can tick per response, not per original index.

Quiz

When map blocks

1 / 2

executor.map(loiter, range(5)) with three workers

Pick an answer to see if you are right.

The second test fails if you sort or if loiter is still identity.

flags2: progress, servers, CLI​

Same job as flags.py, but 676 possible codes, errors, and a tqdm bar. Sequential over that set is minutes; thread pool / asyncio (Chapter 22) can finish comparable work in seconds — only if you throttle. python -m http.server (ThreadingHTTPServer) for LOCAL files. Ramalho’s slow_server.py: DELAY (~0.5 s) and ERROR (delay + ~25% HTTP 418). Production-like chaos: Nginx + toxiproxy. REMOTE is his CDN — do not -m 1000 it.

tqdm(iterable) wraps any iterable; for ETA it wants len or total=. That is why flags2 cannot feed tqdm a silent map generator and expect a bar that moves when any download finishes.

CLI (all flags2 scripts): optional country codes / prefixes (a b c → AA–CZ), -a all, -e AA–ZZ, -l N limit, -m max concurrent (threadpool default 30, hard cap 1000), -s LOCAL|REMOTE|DELAY|ERROR, -v line-per-flag instead of the bar.

Error policy​

download_one (reused by sequential and threadpool):

  • get_flag: non-200 → raise_for_status().
  • Catch HTTPError. If 404, return Result(HTTPStatus.not_found, cc) (and optional verbose print). Any other HTTPError re-raises.
  • Success: save GIF, Result(HTTPStatus.ok, cc).
  • Connection failures and leftover HTTP errors belong to download_many (or the asyncio supervisor): tally HTTPStatus.error, keep going. Unknown exceptions can abort main.

Sequential download_many: Counter keyed by those enum values; tqdm(sorted(cc_list)) if not verbose; try around download_one.

Thread pool: to_do_map​

You iterate Futures, not codes. After as_completed you still need the cc for *** Error for {cc}. Idiom:

to_do_map = {}
for cc in sorted(cc_list):
    future = executor.submit(download_one, cc, base_url, verbose)
    to_do_map[future] = cc
done_iter = futures.as_completed(to_do_map)
if not verbose:
    done_iter = tqdm.tqdm(done_iter, total=len(cc_list))
for future in done_iter:
    try:
        res = future.result()  # does not block: already done
    except ...
        cc = to_do_map[future]

max_workers=concur_req — main already min’d with list length and MAX_CONCUR_REQ. If the pool is much smaller than the list, batches can look alphabetical even though as_completed is unordered.

future.result() re-raises whatever the worker raised — same except shape as sequential, plus the dict lookup.

Threads + concurrent.futures are the boring-good tool for I/O. ProcessPoolExecutor for embarrassingly parallel CPU. That is Chapter 21.

Quiz

404 vs the rest

1 / 2

HTTP 404 in flags2

Pick an answer to see if you are right.

Starter never catches 404: XX becomes error from wrap, so missing fails. If you swallow 500 too, server fails.