Skip to main content

Process Pools and Scale

Source: Fluent Python, 2nd Edition — Chapter 20 by Luciano Ramalho (process pool, GIL workarounds, “the big picture”)

20a compared threads, processes, and asyncio on a spinner. This half: queues to farm CPU work to processes, why threads lose on is_prime, and how production Python sidesteps the GIL (WSGI workers, Celery, NumPy/Dask). Chapter 21’s ProcessPoolExecutor is the same idea with less ceremony — it already uses queues inside.

await asyncio.sleep(0) to “let the spinner breathe” during CPU work is a stopgap. Real fix: loop.run_in_executor (Chapter 22) or a task queue.

Sequential baseline​

Ramalho times is_prime on 20 integers from 2 up to (10^16-1): small primes, huge primes, composites with small or large factors. sequential.py loops check(n): perf_counter, is_prime, Result(prime, elapsed). Total time ≈ sum of the rows (~40 s on his laptop) because nothing overlaps.

You will not run that sample here. The pattern is what matters: a NamedTuple result, a wall-clock around the whole loop, not just the sum of printed slices.

A homegrown process pool​

procs.py starts one worker process per core (multiprocessing.cpu_count(), overridable on the command line). Each worker is not called like a function: the library drives it, so return values go on a queue.

Queues​

FIFO by default. Threads use queue.Queue. Processes use multiprocessing queues (SimpleQueue since 3.9; older code uses Queue). asyncio has its own. There are also LIFO and priority queues.

SimpleQueue for type hints is multiprocessing.queues.SimpleQueue — the name on multiprocessing is a factory bound to a context, not the class.

Two queues:

  1. jobs — integers to test
  2. results — PrimeResult(n, prime, elapsed)

n rides with the result because completion order ≠ submit order. Fast composites finish first; a 8 s prime lands later while siblings ran in parallel. Wall time (~9.6 s vs ~40 s sequential, ~4× on a 6-core / 12-thread laptop) is not the sum of per-number times. Launch + pickle cost eat some of the speedup. Hyper-threading reports 12 CPUs; this trial-division workload behaves closer to 6 cores.

Worker and poison pill​

def worker(jobs, results):
    while n := jobs.get():
        results.put(check(n))

0 is a sentinel (“poison pill”): truthy numbers are work; 0 ends the loop. None is the other usual pill. object() is a unique sentinel in one process — after pickle, the other side has a different object, so == fails. If None is valid payload, ... (Ellipsis) pickles as itself.

main: fill jobs with NUMBERS; start N Process(target=worker, ...); then put N copies of 0. Collect with blocking results.get() until you believe you are done.

Trap: jobs.empty() after the last sentinel is usually OK here because every worker already got a 0. If the last job were a huge prime and you treated “empty jobs queue” as “all done,” a worker might still be computing. If the parent dies first, multiprocessing can dump FileNotFoundError on an internal lock. ProcessPoolExecutor (Chapter 21) is less brittle.

Experimenting with worker count: 1 process ≈ sequential; 2 ≈ ~2× if you have cores; past physical cores, contention. Ramalho’s medians dipped at 6, wiggled, then flattened — don’t overfit one laptop.

Threads are a non-solution​

threads.py is almost a find-replace of Process → Thread. For is_prime it is slower than sequential, and worse with more threads: GIL + context switches (registers, PC, stack, cache, pages).

Quiz

Pools and pills

1 / 3

Why does PrimeResult include n, not only prime and elapsed?

Pick an answer to see if you are right.

The exam uses queue.Queue in one process so the sandbox does not fork. The control flow is the procs.py worker: get until sentinel, put results.

The free lunch is over​

Herb Sutter (2005): clocks stopped being the easy speedup; cores and hyper-threads only help concurrent programs. Python began in the early 1990s, when a GIL was an obvious win: faster single-core eval, simpler C API. An extension that never talks about the GIL can still be hundreds of times faster than Python. Releasing the GIL for extra cores is optional complexity — one reason the C-extension world (and thus Python) is huge. Ruby’s MRI made a similar bet. JS has no user threads; it went async-only.

How Python still scales​

Keep web-scale envy in check (Thoughtworks): Twitter/Netflix complexity is not a starter kit.

System administration​

Fleets, SDN, cloud CLIs: scripts wait on remote commands, not on is_prime. Threads, ThreadPoolExecutor, or asyncio libraries (Facebook PE on AsyncIO, 2016) fit.

Data science​

The hot loops are C/C++/Fortran/Cython, GPUs, clusters:

  • Jupyter — notebook/lab, kernels over ZeroMQ (name: Julia / Python / R); Bokeh in the browser
  • TensorFlow / PyTorch — C++, cores/GPUs/clusters; Python is the main UI (Google / Facebook origins)
  • Dask — local processes or supercomputer jobs; NumPy/Pandas/sklearn-shaped APIs; live dashboards

Server-side​

Google, YouTube, Dropbox, Instagram, Quora, Reddit: architecture, not “one thread with a lock.” Kleppmann’s Designing Data-Intensive Applications: caches (memcached, Redis, Varnish), Postgres/MySQL, Couch/Mongo, Elasticsearch/Solr, RabbitMQ/Redis queues. Cloud vendors sell the same shapes.

Two Python-specific layers:

WSGI application servers sit under HTTP (often NGINX/Apache for static + proxy). They fork several Python processes so sequential Django/Flask/Pyramid uses every core. You can ship web apps and never import threading. Names: mod_wsgi (+ mod_wsgi-express for containers), gunicorn, uWSGI (kitchen sink, painful defaults — read production notes before copying a blog), NGINX Unit. uWSGI/Unit are polyglot; gunicorn/mod_wsgi are Python-web-centric.

ASGI is the async successor (WebSockets, long poll): aiohttp, Sanic, FastAPI; Django/Flask growing async.

Distributed task queues (Celery, RQ, or a cloud queue): the HTTP process must answer fast. Email, PDFs, retries, cron-like work go to workers, often other machines. Vocabulary: producer (e.g. a Django view) puts a message; consumers pull. Scale by adding consumers. Internally: job queue + result backend — same cartoon as jobs/results in procs.py. Redis is a common broker; Celery also RabbitMQ, SQS, …

Quiz

Scale without hugging the GIL

1 / 2

Typical WSGI deployment for CPU cores

Pick an answer to see if you are right.

Chapter 20 in one page​

Three native models (spinner). GIL: asyncio must not hog CPU; two threads can still animate; processes ignore the GIL. A process pool + queues beats sequential primes; a thread pool does not. The GIL is not the whole industry: ops is I/O; science is native/GPU; web is WSGI workers + task queues. Next: concurrent.futures (21), then asyncio in earnest (22).

Further reading if you go deeper: multiprocessing programming guidelines; Gorelick & Ozsvald High Performance Python; Hattingh Using Asyncio ch. 2 (“threads unravel”); Beazley on the GIL (empty while True is a bad microbench); PEP 554 subinterpreters; Kleppmann; Celery FAQ; Thoughtworks “web scale envy.” Actor-model libraries (Thespian, Pykka) encode no shared mutable state, one message at a time — the constraint the soapbox wants that locks do not give you.