Concurrent Downloads

Source: Fluent Python, 2nd Edition — Chapter 21 by Luciano Ramalho (through Executor.map vs a homegrown process pool)

Network I/O: the CPU is idle while packets fly. A sequential client waits for each response before starting the next. Overlap the waits and twenty small files stop taking ~5× longer than they should.

Ramalho’s demo (warm CDN, 20 flag GIFs): sequential flags.py ~7.2 s; flags_threadpool.py ~1.4 s; flags_asyncio.py ~1.35 s. Concurrent runs print country codes out of list order. Hundreds of files: the gap can hit 20×. Threads vs coroutines: no meaningful HTTP-client difference if you cap how many requests you open. asyncio is Chapter 22.

Do not hammer public sites while you experiment. Twenty hardcoded URLs is one thing; a tight loop against a stranger’s server looks like a DoS. Later in the chapter he uses http.server locally.

This lesson: sequential client shape, then concurrent.futures (ThreadPoolExecutor.map, submit / as_completed, what a Future is). CPU-bound work with ProcessPoolExecutor is the same API. We will not download flags in the sandbox.

Sequential flags.py

requests (third-party, nicer than urllib.request). POP20_CC — ISO codes of the twenty most populous countries. get_flag(cc) GETs {BASE}/{cc}/{cc}.gif. save_flag writes bytes. No error handling yet — structure first.

download_many is the piece that will change: for cc in sorted(cc_list): get, save, print(cc, flush=True) so progress is not stuck in the stdout buffer until a newline. Returns len(cc_list).

main(downloader) times whatever callable you pass. Sequential script: main(download_many). Threaded and async scripts reuse get_flag / save_flag / main.

ThreadPoolExecutor.map

def download_one(cc: str):
    image = get_flag(cc)
    save_flag(image, f'{cc}.gif')
    print(cc, end=' ', flush=True)
    return cc

def download_many(cc_list: list[str]) -> int:
    with futures.ThreadPoolExecutor() as executor:
        res = executor.map(download_one, sorted(cc_list))
        return len(list(res))

The sequential loop body becomes the worker. That refactor is the usual move.

with calls shutdown(wait=True) on exit — wait for workers.

Executor.map is like map, but the callable runs in the pool. It yields a generator of return values, in argument order. list(res) drives it; if a worker raised, the exception surfaces on that next().

max_workers: required before 3.5; default None. Since 3.8:

min(32, os.cpu_count() + 4) — at least about five threads for I/O, cap 32, reuse idle workers. Override when you know your rate limit.

Quiz

I/O overlap

1 / 2

Twenty independent HTTP GETs, properly overlapped

Pick an answer to see if you are right.

If you upper in download_many and map identity, the first test can pass and upper in worker fails. If you skip map, ThreadPoolExecutor / map checks fail.

Where are the futures?

concurrent.futures.Future and asyncio.Future: a deferred result (Twisted Deferred, JS Promise). You put them in queues, ask .done(), attach .add_done_callback(fn) (fn(future) when finished).

Do not construct them. A Future means “this is scheduled.” Only the framework does that. Executor.submit(fn, *args) schedules and returns a Future. You do not flip its state.

.result(): if done, value or re-raise the worker’s exception. If not done: concurrent.futures.Future.result(timeout=...) blocks the calling thread (TimeoutError if you set a timeout). asyncio.Future.result has no timeout; you await. You cannot await a concurrent.futures.Future.

Executor.map never shows you the Future objects — __next__ already called .result().

submit and as_completed

To see them: submit in a loop, then futures.as_completed(to_do) yields each Future as it finishes (not submit order). After that, .result() does not block. Demo with max_workers=3 and five codes: first three state=running, last two pending. Worker prints can interleave with the main thread’s result: lines. max_workers=1 → sequential, submit order. max_workers=5 → more shuffle.

Starter work is identity: 3 1 2 would sort to 1 2 3, not 1 4 9.

ProcessPoolExecutor

Same Executor interface. Default max_workers is os.cpu_count(). Useless for flag downloads (start + RAM). Useful for Python CPU work (Chapter 20 primes).

proc_pool.py vs homegrown procs.py: no queues, no sentinel worker. check still returns PrimeResult(n, flag, elapsed). ProcessPoolExecutor(None) then peek _max_workers (undocumented; mypy hates it) if you want to print the pool size.

executor.map(check, numbers) returns results in numbers order, not “whoever finished.” Sort descending and the largest prime in the sample sits in slot two: the first composite with a tiny factor prints instantly; then the UI looks frozen until that prime’s Future is ready; then the rest dump at once — they already finished in other processes. Homegrown procs.py printed completion order (easy composites first). Wall time is about the same.

Quiz

map order vs finish order

1 / 2

ProcessPoolExecutor for the flag HTTP client

Pick an answer to see if you are right.

Chapter 21 continues with a demo of Executor.map timing and then error handling / http.server. Next lesson.