Processes and CPU Work
Use isolated workers for substantial CPU work and account for serialization, startup, and process lifecycle.
By the end, you’ll be able to…
- Place process worker functions at importable module scope.
- Use a main guard and an explicit spawn context.
- Choose task sizes that amortize process overhead.
01Isolation can enable parallel CPU work
Separate worker processes have separate interpreters and memory. On conventional CPython, this can let CPU-heavy Python calculations execute across multiple cores despite the per-interpreter GIL. The tradeoff is startup cost, extra memory, and transferring arguments and results. A few multiplications are too small to justify that machinery; substantial independent batches are a better candidate.
ProcessPoolExecutor exposes the familiar futures interface, but the boundary is stricter than a thread boundary. Arguments and results generally must be picklable, and worker functions must be importable. A lambda or locally nested function is therefore a poor process worker. Pass compact data rather than open database connections, locks from unrelated contexts, or a huge object graph copied for every task.
02Spawn safely from a real module
With spawn, a new interpreter imports the main module before executing a worker. Creating a process pool during import can recursively create more processes. Put orchestration behind if __name__ == "__main__" and keep worker definitions at module level. Save this example as a file and run it from a terminal; interactive consoles and browser sandboxes may not provide an importable main module.
The explicit spawn context makes the lifecycle assumption visible across platforms. It also avoids depending on inherited state from fork. Inputs below are integers, outputs are integers, and the worker has no side effects. The result order is deterministic because map preserves input order, even though the actual scheduling and process identifiers vary from run to run.
from concurrent.futures import ProcessPoolExecutorfrom multiprocessing import get_contextdef sum_squares(limit): return sum(value * value for value in range(limit))def main(): with ProcessPoolExecutor(max_workers=2, mp_context=get_context('spawn')) as pool: results = list(pool.map(sum_squares, [10, 100, 1000])) print(results)if __name__ == '__main__': main()[285, 328350, 332833500]
03Transfer and batching are part of the algorithm
Serialization turns Python objects into a representation that another process reconstructs. Normal mutable objects are not automatically shared: a worker appending to its own list does not append to the parent’s list. Return an explicit result and merge in the parent, or choose a deliberate interprocess communication mechanism when the design requires collaboration.
Batch enough computation per task to amortize communication and scheduling. executor.map supports chunksize for process pools, but the best value depends on task variability: large chunks can leave workers idle behind one expensive chunk. Measure serial and process versions end to end, including startup and data preparation. A speedup in the worker body can still lose once transfers are included.
04Design failures at the process boundary
Calling future.result propagates an ordinary worker exception to the parent. Abrupt process failure can instead break the pool, so do not treat every failure as one bad input row. Keep tasks idempotent when reruns are possible and write durable outputs through a controlled parent process when feasible. Several workers writing the same file can corrupt a result even though their calculations are independent.
Avoid workers waiting on futures from the same process pool: nested dependencies can deadlock. Shutdown waits for running tasks by default, and cancelling a pending future is different from terminating active computation. For long-running jobs, make progress observable and choose a cancellation design before deployment. Process isolation supports a concurrency strategy; it does not replace resource limits or trustworthy input validation.
The ideas worth keeping.
- Use importable workers and a guarded entry point.
- Treat data transfer and startup as real costs.
- Merge explicit results and plan partial-failure behavior.
Make it yours.
Type and test your code right here in the interactive workspace. Run your solution, check the output, and compare with the worked guide when ready.
Prepare process-safe work
Write a module-level digit_sum(text) that validates decimal ASCII input and returns a number. Test it locally before using it as a process worker.
Explore a worked solution
def digit_sum(text): if not text or any(character not in '0123456789' for character in text): raise ValueError('expected nonempty ASCII digits') return sum(int(character) for character in text)assert digit_sum('1234') == 10try: digit_sum('12x')except ValueError: passelse: raise AssertionError('invalid input accepted')A small pure function is easy to validate independently and later submit by importable name. Local correctness is necessary before measuring process behavior.
Merge independent summaries
Each batch contains category names. Produce a dictionary of counts per batch, then merge those dictionaries without shared mutable worker state.
Explore a worked solution
from collections import Counterdef summarize(batch): return dict(Counter(batch))partials = [summarize(batch) for batch in [['a', 'b'], ['a', 'c', 'a']]]total = Counter()for partial in partials: total.update(partial)assert dict(total) == {'a': 3, 'b': 1, 'c': 1}The same summarize function can become a process worker. The collector merges compact dictionaries, avoiding shared counters and making a serial baseline straightforward.
What stayed with you?
Check the ideas while they’re fresh. Your answers stay in this page and feedback appears as you go.
Prepare process-safe work
Write a module-level digit_sum(text) that validates decimal ASCII input and returns a number. Test it locally before using it as a process worker.
[Process exited with code 0 - No output]
Keep exploring.
The official references behind this chapter, when you want a second angle or the full details.
One chapter closer.
Reading is a start. Trying it yourself makes the connection.