Parallelization in LLM systems: sectioning and voting

The Anthropic agents playbook names two distinct reasons to run LLM calls in parallel. They sound similar — "do multiple things at once" — but they solve completely different problems, cost differently, and fail in different ways. Conflating them produces systems that are more expensive without being meaningfully better. patterns/parallel in effective-agents-lab implements both in one module, and is also the one module in the repo where a real concurrency bug appeared during development.

The two reasons to run in parallel

Sectioning is about coverage: a task breaks into genuinely independent subtasks, so you run them concurrently and combine the results. The key word is "genuinely independent" — each specialist needs no knowledge of what the others are doing while it runs. If they do need each other's output, you want an orchestrator pattern, not parallelization.

Voting is about confidence: you run the same task multiple times with differently-angled prompts, then aggregate. It's useful when the risk you're managing is the variance of a single model call — a good prompt called twice is more reliable than a good prompt called once. What it cannot do is fix a weak prompt. Running the same flawed prompt five times gives you five flawed answers with false confidence baked in.

Sectioning in practice: the loan-risk example

The reference implementation models a loan application risk assessment. Four specialists run concurrently — market, operational, credit, and compliance — each analyzing the same application from its own angle. Here's a real trace excerpt from the module's concurrency test, four specialists writing to one shared run file from four separate threads:

{"event":"llm_call","step":"specialist_market","model":"claude-sonnet-5","cost_usd":0.00012}
{"event":"llm_call","step":"specialist_operational","model":"claude-sonnet-5","cost_usd":0.00012}
{"event":"llm_call","step":"specialist_credit","model":"claude-sonnet-5","cost_usd":0.00012}
{"event":"llm_call","step":"specialist_compliance","model":"claude-haiku-4-5-20251001","cost_usd":6e-05}
{"event":"log","step":"aggregate","decision":"approve","weighted_score":80.0,"vetoed":false,"degraded":false}

Three specialists run on claude-sonnet-5 for the heavier reasoning work; compliance runs on claude-haiku because a hard rule check doesn't need a more expensive model. The total for four specialist calls is $0.00042 in this trace. That model-tier distinction matters: sectioning lets you right-size each call independently, which you can't do when everything is one prompt.

The degraded: false in the aggregate log line is important. If any specialist times out or crashes, the others still complete and the aggregator forces a "refer" decision — it doesn't pretend it has full information. degraded: true means the result was produced with fewer inputs than expected; a downstream consumer should treat that differently from a full four-way aggregate.

Voting in practice: the guardrail layer

After sectioning produces an aggregate decision, the voting guardrail runs. Two differently-biased reviewers — a conservative risk officer and a customer advocate — both evaluate the aggregated recommendation independently:

{"event":"llm_call","step":"vote_conservative_risk_officer","model":"claude-haiku-4-5-20251001","cost_usd":6e-05}
{"event":"llm_call","step":"vote_customer_advocate","model":"claude-haiku-4-5-20251001","cost_usd":6e-05}

If fewer than two of the three judge personas agree the recommendation is justified, the result gets flagged for human review — even if the deterministic aggregator already produced a clean decision. This is the right use of voting: catching the cases where a weighted score looks fine but a differently-angled reviewer would flag it. The voter models run on Haiku because these calls are narrow judgment tasks, not complex analysis.

The full run — four specialists plus two voters — costs roughly $0.00054 in this trace. Whether that's worth it depends entirely on what a bad loan decision costs the downstream consumer.

Aggregation should be code, not another model

One of the design choices I want to call out explicitly: aggregate() is deterministic Python. Weighted scoring, explicit veto logic, threshold checks — all code you can read and test. There's no fifth model call that summarizes the four specialists' outputs and produces a decision. That's deliberate.

Compliance flagging a hard red flag (an automatic deny, regardless of every other score) is enforced by code, not by asking a model whether the compliance specialist sounded serious enough. The moment aggregation itself requires judgment, you've added complexity without reducing risk — you've just moved the judgment call one layer deeper and made it harder to audit.

If combining N parallel outputs needs as much model judgment as producing one output would have required, you haven't actually saved anything. Keep aggregation deterministic wherever you can.

The concurrency bug that actually happened

This module is the one in the repo where a real bug appeared during development, and it's worth understanding because it only shows up under actual concurrency.

The problem: sharing one contextvars.copy_context() object across multiple ThreadPoolExecutor.submit() calls raises RuntimeError: cannot enter context: <Context object> is already entered for every thread after the first, because a Context can't be entered concurrently. Everything looks fine in sequential tests, and the bug only surfaces when the parallel workers actually run simultaneously.

The fix is calling contextvars.copy_context() fresh inside each individual submit() call rather than hoisting it above the loop. One line change; the kind that only becomes obvious after you've seen the traceback. The reason this module has a dedicated concurrency test — four specialists writing to one shared run file from four separate threads — is that the next person to touch specialists.py needs a test that will catch this class of regression, not just a description of the fix.

When to use which, and when to avoid both

Sectioning earns its cost when subtasks are genuinely independent and the combination logic is simple enough to trust in deterministic code. Loan risk across market, operational, credit, and compliance is a clean fit because the underwriting domain already defines these as separate concerns. The parallel wall-clock time is strictly less than sequential (the module ships a --compare-sequential flag so you can measure this directly), and the cost is identical to sequential because you're running the same calls.

Voting earns its cost when the risk is single-call variance and you have a prompt that's already good. Don't reach for it as a fix for a prompt that produces inconsistent results — if the prompt is weak, the vote will be noisy.

Avoid both when the aggregation step becomes the hard problem. If you need a model to reconcile what four other models said, you've added latency, cost, and an extra failure surface without the benefit you were looking for. That's usually a sign the subtasks weren't independent to begin with, and the right pattern is orchestrator-workers — where one model explicitly coordinates the others — rather than parallel fan-out.