Multi-Agent Orchestration: Patterns That Survive Production

A company ran a 3-month pilot of a multi-agent research assistant. The demo was spectacular — the orchestrator decomposed a research brief, dispatched four workers to gather sources, synthesize findings, fact-check outputs, and draft the report. End-to-end in under 90 seconds. The pilot got funded.

Six months later, the system was producing wrong answers roughly a third of the time, costs had tripled their estimate, and the engineering team was spending most of their time debugging failures that couldn't be reproduced reliably. The topology was fine. The state management was not. Agents were passing their final answers to each other, not their full context — so each handoff silently dropped the reasoning that justified the answer, and subsequent agents had no way to detect when they were building on a flawed foundation.

This is the pattern: multi-agent systems fail not because the orchestration concept is wrong, but because the implementation underestimates what "passing control between agents" actually requires.

Why you might need multiple agents at all

Before getting into patterns, it is worth being explicit about when a single agent with tools is not enough — because the answer is "less often than you think."

A single agent fails you in three specific cases. First, context window exhaustion: a task that requires accumulating more information than fits in one context window (true long-document processing, multi-day research, tasks that produce large intermediate artifacts). Second, execution cost: tasks where planning requires a frontier model but the actual execution steps are mechanical enough that a cheap model handles them — and you have enough volume to care about the 10-40x cost difference. Third, quality through independent review: tasks where the same model generating and reviewing an output misses mistakes consistently, but a separate critic catches them — specifically because it evaluates with different sampling and different context.

Everything else is a single-agent problem. The agent loop article covers that foundation. The ReAct, Reflexion, and planning patterns article covers the reasoning strategies single agents use. Come back to this article once you have hit one of those three walls.

The six topologies

Orchestrator-Worker

The most common production pattern. A central orchestrator — running on a frontier model — decomposes the task, dispatches subtasks to specialized worker agents, and synthesizes their outputs. Workers are narrow: they receive a precise instruction and return a structured result. The orchestrator is the only agent that holds the full plan.

sequenceDiagram
    participant U as User
    participant O as Orchestrator (Opus/GPT-4o)
    participant W1 as Research Worker (Haiku)
    participant W2 as Code Worker (Haiku)
    participant W3 as Write Worker (Haiku)

    U->>O: "Build a market analysis + code prototype for X"
    O->>O: Decompose → 3 subtasks
    par dispatch
        O->>W1: "Research market size, competitors, trends for X"
        O->>W2: "Write Python data fetcher for X's public API"
        O->>W3: "Draft executive summary outline for market analysis"
    end
    W1-->>O: research findings + sources
    W2-->>O: working code + test results
    W3-->>O: outline + key claims
    O->>O: Synthesize, cross-check, fill gaps
    O-->>U: Final report + code

The cost math is direct. Suppose the orchestrator makes 3 planning calls at $15/M tokens (frontier pricing, as of mid-2026), consuming 2,000 tokens each: that is $0.09. Three workers each make 5 execution calls at $0.60/M tokens (cheap model), consuming 1,500 tokens each: that is $0.014 total. The overall cost is dominated by the planning passes, not the execution passes. On high-volume tasks where you would otherwise run every step on a frontier model, the savings are 40-60%.

The failure mode is context leakage at aggregation. Workers return results, but the orchestrator only sees those results — not the chain of reasoning, intermediate tool calls, or uncertainty flags the worker accumulated. If a worker found contradictory sources and silently chose one, the orchestrator has no way to know. Pass structured result envelopes, not raw answers:

# Worker returns a structured result envelope, not just the answer
from pydantic import BaseModel

class WorkerResult(BaseModel):
    answer: str
    confidence: float          # 0-1
    sources: list[str]
    caveats: list[str]         # contradictions, gaps, uncertainties
    tool_calls_made: int
    tokens_consumed: int

# Orchestrator can reason about confidence and caveats
# when synthesizing across workers

Sequential Pipeline

Each agent hands off to the next in a fixed order. Agent 1 completes, Agent 2 receives its output and continues, and so on. The analogy is a manufacturing line: each station has a specific operation, and partially-completed work moves through in sequence.

flowchart LR
    IN[Input] --> A1["Agent 1\nIngest & classify"]
    A1 --> A2["Agent 2\nEnrich & validate"]
    A2 --> A3["Agent 3\nTransform & format"]
    A3 --> A4["Agent 4\nReview & approve"]
    A4 --> OUT[Output]

    style A1 fill:#0e7490,color:#fff
    style A2 fill:#0e7490,color:#fff
    style A3 fill:#0e7490,color:#fff
    style A4 fill:#00e5ff,color:#0a0a0f

Sequential pipelines are the easiest topology to reason about and test. Each agent has a clear contract: defined inputs, defined outputs. You can evaluate each stage in isolation. Failures are localized — if Stage 3 produces bad output, you know where to look.

The limitation is latency: total time is the sum of all stage latencies. For tasks where stages could run in parallel but you forced them sequential, you pay the full latency bill. The other limitation is brittleness at fixed stage boundaries: if an input requires different processing than you anticipated, the pipeline has no way to adapt. A customer support triage that always routes through five stages struggles when a ticket requires two stages or seven.

Sequential pipelines suit tasks with stable, predictable structure — document transformation, data enrichment, content moderation at scale. For tasks with variable structure, dynamic handoff is more appropriate.

Dynamic Handoff

Instead of a fixed sequence or central orchestrator, each agent decides at runtime which other agent should receive control next. The mechanism is a set of routing tools — synthetic tool calls like transfer_to_researcher, transfer_to_coder, transfer_to_reviewer — that the current agent invokes when it determines it has done what it can.

This is the pattern behind the OpenAI Agents SDK's handoff primitive and the style of routing OpenAI describes in its agents documentation.

# Each agent has access to handoff tools for the edges it can route to
researcher_tools = [
    web_search,
    read_document,
    transfer_to_coder,   # researcher → coder
    transfer_to_writer,  # researcher → writer
]

coder_tools = [
    run_python,
    transfer_to_researcher,  # coder → researcher (to clarify requirements)
    transfer_to_reviewer,    # coder → reviewer
]

# transfer_to_coder is just a structured tool call that passes
# the full accumulated state to the receiving agent
def transfer_to_coder(task: str, context: str, findings: dict) -> None:
    """Hand off to the coding agent with current findings and task context."""
    ...

The flexibility is real: a research task that unexpectedly needs code can route itself to the coder mid-task, without a central orchestrator to update. But the debugging surface is brutal. A failure five hops into a dynamic handoff graph is nearly impossible to reproduce without full trace logging of every routing decision and the full state at each transfer point.

Routing guardrails are mandatory. Without explicit constraints, agents can create cycles (A routes to B, B routes back to A), route to agents that lack the context to continue meaningfully, or chain indefinitely. In practice, you need: a maximum hop count enforced outside the agent (not by the agent itself), explicit allowed-routing edges (not every agent can route to every other), and state validation at each receive point to confirm the incoming state is sufficient to proceed.

Evaluator-Optimizer

A generator agent produces an output; an evaluator agent scores it against explicit criteria and returns structured feedback; the generator revises. This continues until the evaluator approves or the iteration cap is reached.

flowchart LR
    IN[Task + criteria] --> GEN[Generator agent]
    GEN -->|draft| EV[Evaluator agent]
    EV -->|score + critique| GEN
    EV -->|approved| OUT[Final output]

    style GEN fill:#0e7490,color:#fff
    style EV fill:#00e5ff,color:#0a0a0f

The evaluator-optimizer loop works reliably on tasks where quality criteria can be stated precisely and independently verified. Code correctness is the canonical case: a test suite is the evaluator, and the generator rewrites code until tests pass. Translation quality measured against a reference, structured data extraction validated against a schema, report factual accuracy checked against source documents — these all work because the evaluation step is more reliable than a human doing the same check at the same speed.

The failure mode is using the same model as both generator and evaluator. The critic tends to make the same reasoning errors as the author, and can actually worsen output by reinforcing the generator's blind spots. Use a different model (or at least a different sampling configuration) for the evaluator, and keep the evaluation criteria as objective as possible. "Is this code correct per these tests?" is a better criterion than "Is this writing good?"

The other failure mode is vague criteria. An evaluator told to check for "clarity and professionalism" will produce inconsistent scores across iterations and give the generator unhelpful revision guidance. The evaluation contract should be a structured rubric with scoreable dimensions, not a prose description.

Swarm

A swarm topology has no central coordinator. Agents are peers that each hold a portion of the shared task state, communicate with each other, and self-organize toward a solution. A naming warning: OpenAI's experimental Swarm library is not this. Despite the name, Swarm-the-library runs one agent at a time and routes control via transfer_to_X tools — it is an implementation of the dynamic handoff pattern above (and the direct predecessor of the Agents SDK handoff primitive), not a decentralized topology. Genuine swarm deployments are rare enough that there is no canonical library to point at.

Swarms are appealing in theory and painful in practice. The practical use case is narrow: tasks where the work partitions naturally into independent units that agents can claim and complete without coordination — web scraping at scale, embarrassingly parallel data transformation. For tasks that require synthesis or sequential dependency, swarms degrade into expensive confusion.

The observation from production deployments: swarms are usually a workaround for not having designed a good orchestrator-worker topology. Before reaching for a swarm, ask whether the task truly cannot be decomposed upfront. Most of the time, it can.

Adaptive Planning

Adaptive planning is the evaluator-optimizer loop with the critic folded inside: a single agent checks its own progress against the plan and replans mid-task, instead of routing drafts through a separate evaluator agent. The economics favor it heavily when the check is cheap. An external evaluator pass carries a full second agent context — system prompt (~2,000 tokens), rubric, and the entire draft — on every iteration, plus a full round trip of latency. Internal replanning appends a few hundred tokens of reflection to a context you are already paying for. Over three revision cycles, that is roughly 15,000-20,000 evaluator input tokens versus ~1,000 tokens of self-critique.

The catch is the same one from the evaluator-optimizer section, made worse: the internal critic shares not just the generator's weights but its exact context and reasoning trace, so it rubber-stamps the mistakes it just made. Use adaptive planning when progress is verifiable from signals the agent can read directly — a tool call errored, tests failed, the search returned nothing. Use a separate evaluator agent when quality is a judgment call, because self-review of judgment calls converges on "looks good to me." The ReAct and planning patterns article covers the single-agent replanning mechanics in depth; this article focuses on the inter-agent case.

The state problem: what actually breaks

The six topologies above are architectural patterns. The actual engineering challenge is state management at handoff boundaries, and this is where most multi-agent systems fail.

When Agent A finishes and passes work to Agent B, what does Agent B receive? In a naive implementation, it receives the final answer that Agent A produced. But Agent A also accumulated:

• Tool call history (what it searched, what it found, what it tried that failed)
• Uncertainty and confidence about different parts of its answer
• Partial state from intermediate steps
• Caveats — contradictions in sources, edge cases flagged but not resolved

If Agent B only sees the answer, it operates on incomplete information. It cannot distinguish a confident conclusion from a best-guess. It cannot know that the caveats existed. It will build on a potentially shaky foundation without knowing to be skeptical.

The solution is structured state serialization. Every agent handoff should transmit a state envelope that includes at minimum: the task, the intermediate steps and their results, the current best answer, confidence and uncertainty metadata, and any flags that downstream agents should check.

This adds tokens — but it is better to pay for context than to debug wrong answers in production. For long pipelines, you can compress older context using a summarization pass (the context window management article covers this in detail), but never discard it entirely.

from dataclasses import dataclass

@dataclass
class AgentHandoffState:
    task_original: str
    steps_completed: list[dict]       # each step: action, result, tokens_used
    current_answer: str | None
    confidence: float                 # 0.0 - 1.0
    open_questions: list[str]         # things not resolved
    caveats: list[str]                # contradictions, edge cases
    tokens_consumed_so_far: int
    iterations_used: int
    budget_remaining: dict            # {"tokens": int, "iterations": int}

# Before handing off:
state = AgentHandoffState(
    task_original=original_task,
    steps_completed=agent_a.step_log,
    current_answer=agent_a.best_answer,
    confidence=agent_a.confidence_score,
    open_questions=agent_a.flagged_gaps,
    caveats=agent_a.caveats,
    tokens_consumed_so_far=agent_a.total_tokens,
    iterations_used=agent_a.iterations,
    budget_remaining=compute_remaining_budget(global_budget, agent_a.total_tokens),
)

What breaks

Compounding error

The math in the summary block is not academic. At 85% per-step accuracy — which is realistic for non-trivial agentic steps — a 10-step multi-agent workflow succeeds roughly 20% of the time end-to-end. Many teams build deep pipelines assuming the individual-step accuracy translates to pipeline accuracy. It does not.

The actionable response is to measure this concretely for your workflow before committing to the architecture. Run 100 test cases through your proposed pipeline and measure actual end-to-end accuracy. If it is unacceptably low, the fix is usually to reduce the number of steps — not to optimize individual step accuracy — because the exponent dominates.

{ "type": "error-compound", "accuracy": 0.85, "steps": 10, "title": "How per-step accuracy collapses across a pipeline" }

Token cost runaway

Multi-agent systems duplicate context. The system prompt appears in every agent's context. Tool schemas are repeated in every context. An orchestrator synthesizing outputs from three workers holds all three outputs simultaneously. A mid-sized orchestrator-worker run can easily cost 5-10x what you estimated from single-agent token counts.

Back-of-the-envelope for a 5-agent orchestration run:

Orchestrator (frontier): 2 planning + 2 synthesis passes,
  ~5,000 tokens each (system prompt + plan + worker outputs)
  = ~20,000 input tokens
Workers (cheap): 5 agents × 3 calls × ~3,000 tokens
  (shared system prompt + tool schemas + task + intermediate state)
  = ~45,000 input tokens
Total input tokens: ~65,000 per orchestration run

At $15/M input tokens (frontier): 20,000 tokens = $0.30
At $0.60/M (cheap):                45,000 tokens = $0.027
Total per run: ~$0.33

At 10,000 runs/day: $3,300/day = $99,000/month

Without budgeting and prompt caching on the shared prefix,
token drift from longer intermediate states makes this grow
unpredictably over time.

Prompt caching on the system prompt and tool schemas significantly changes this math — typically 50-70% reduction on the stable prefix. Enable it.

Context rot

Long-running agent loops accumulate a growing context window full of failed attempts, superseded reasoning, and stale intermediate results. After enough iterations, the model's attention is split across so much history that it starts contradicting its own earlier conclusions or repeating work it has already done. This is called context rot, and it is a documented production failure mode.

The fix is explicit context management: summarize and compress older context every N steps (using a cheap model), keep only the most recent N steps in full, and maintain a separate structured state that captures the key facts from earlier steps without the full reasoning trace. The context window management article covers the compression strategies in detail.

Handoff trust and security

Every agent handoff is a trust boundary. Agent B receiving a message from Agent A has no way to verify that the message actually came from Agent A, or that Agent A was not itself manipulated by adversarial content in the documents it processed. In a multi-agent system, a prompt injection attack against one agent can propagate across the entire pipeline if agents treat messages from other agents as unconditionally trusted.

The fix is the same as for external inputs: validate and sanitize every message at the receiving boundary, regardless of source. Treat inter-agent messages with the same skepticism as user input. The agentic AI security article covers the full attack surface.

Irreversible actions without checkpoints

A multi-agent system that sends emails, modifies databases, or makes API calls that cannot be undone needs explicit human-in-the-loop checkpoints before those actions. This is not optional. The compounding error rates above mean that a 10-step workflow has roughly 80% probability of making at least one mistake — and if step 8 sends the email, the mistake is already external.

Design the workflow to separate read and plan (fully agentic) from write and execute (human approved) wherever the action is not trivially reversible. The cost is latency on the checkpoint; the benefit is that you have not sent a wrong email to a client.

Framework options (as of mid-2026)

Framework	Model	Best for
LangGraph	Stateful graph (supports cycles)	Complex orchestration with explicit state, cycles, branching
OpenAI Agents SDK	Handoffs as first-class primitive	OpenAI-model-centric systems; handoff tracing built in
AutoGen / AG2	Conversational GroupChat	Research, multi-agent debate, flexible conversation topology
Google ADK	Hierarchical agent trees	Google Cloud / Vertex AI native deployments
Custom (bare SDK)	Full control	When framework overhead matters or topology doesn't fit a pattern

LangGraph is the dominant enterprise choice for stateful orchestration as of mid-2026, with a larger community footprint than CrewAI. Its graph model makes state explicit — every edge is a state transition, and the current state is always inspectable. This makes debugging significantly more tractable than dynamic handoff systems.

The OpenAI Agents SDK (released early 2025) introduced handoffs as a first-class concept alongside built-in tracing, which matters a lot: traces are the primary debugging tool for multi-agent failures, and any framework that does not make traces trivially available is making your life harder.

A direct implementation using the Anthropic SDK, without a framework, looks like this for a simple orchestrator-worker pattern:

import json

import anthropic
from concurrent.futures import ThreadPoolExecutor, as_completed

# timeout on the client is what actually aborts a hung API call —
# Python threads cannot be killed from outside
client = anthropic.Anthropic(timeout=30.0)

def parse_subtasks(text: str) -> list[dict]:
    """Parse the orchestrator's JSON plan. Use structured output in production."""
    return json.loads(text)

def run_worker(worker_name: str, task: str, worker_model: str = "claude-haiku-4-5") -> dict:
    """Execute a single worker agent on a narrow subtask.

    Returns the WorkerResult envelope from the orchestrator-worker
    section — answer, confidence, sources, caveats — not raw text.
    """
    response = client.messages.create(
        model=worker_model,
        max_tokens=2048,
        system=(
            f"You are a specialized {worker_name} agent. Complete the given task "
            "precisely. Return JSON with keys: answer, confidence (0-1), "
            "sources (list of strings), caveats (list of strings)."
        ),
        messages=[{"role": "user", "content": task}]
    )
    envelope = WorkerResult(
        **json.loads(response.content[0].text),  # use structured output in production
        tool_calls_made=0,
        tokens_consumed=response.usage.input_tokens + response.usage.output_tokens,
    )
    return {"worker": worker_name, "result": envelope}

def orchestrate(task: str, orchestrator_model: str = "claude-opus-4-5") -> str:
    # Step 1: Orchestrator decomposes the task
    plan_response = client.messages.create(
        model=orchestrator_model,
        max_tokens=1024,
        system="You are an orchestrator. Decompose the given task into 2-4 subtasks for specialized workers. Return a JSON array of {worker_name, task} objects.",
        messages=[{"role": "user", "content": task}]
    )
    subtasks = parse_subtasks(plan_response.content[0].text)

    # Step 2: Dispatch workers in parallel with budget guard
    results = []
    with ThreadPoolExecutor(max_workers=4) as executor:
        futures = {
            executor.submit(run_worker, st["worker_name"], st["task"]): st
            for st in subtasks
        }
        try:
            for future in as_completed(futures, timeout=60):
                results.append(future.result())
        except TimeoutError:
            # Budget expired: keep what finished, abandon the stragglers
            executor.shutdown(wait=False, cancel_futures=True)

    # Step 3: Orchestrator synthesizes from the structured envelopes
    synthesis_input = "\n\n".join(
        f"## {r['worker']} result\n{r['result'].answer}\n"
        f"Confidence: {r['result'].confidence}\n"
        f"Caveats: {'; '.join(r['result'].caveats) or 'none'}"
        for r in results
    )
    final = client.messages.create(
        model=orchestrator_model,
        max_tokens=2048,
        system="Synthesize the worker outputs into a coherent final answer. Flag any contradictions.",
        messages=[{"role": "user", "content": f"Task: {task}\n\nWorker results:\n{synthesis_input}"}]
    )
    return final.content[0].text

The timeout handling deserves a close look, because the obvious version is wrong. timeout=60 on as_completed does not kill anything — when the budget expires it raises TimeoutError, and if you let that propagate, you crash the run and throw away every result that did finish. Catch it, salvage the completed results, and call shutdown(wait=False, cancel_futures=True) so the exit path does not block waiting on stragglers. Even then, a Python thread cannot be killed from outside: the client-level timeout=30.0 is what actually aborts a hung API call and lets the thread return. You need both layers, or a stalled worker holds the orchestration indefinitely.

The decision in practice

Start with a single agent. Use well-designed tool schemas and an explicit agent loop with hard iteration and token caps. Measure its failure rate on your actual tasks, not on benchmarks.

Then apply multi-agent patterns in this order of escalating complexity:

1. Context limit hit? Add a summarization step or episodic memory (see Agent Memory Architectures) before adding agent boundaries.

2. Execution cost too high? Add orchestrator-worker split — one frontier model plans, cheap workers execute. This is the highest-ROI pattern in terms of cost reduction per added complexity.

3. Quality ceiling from single-pass generation? Add an evaluator-optimizer loop for the specific output type where quality is measurable. Do not apply it globally.

4. Task structure genuinely varies unpredictably? Consider dynamic handoff, with explicit routing constraints, trace logging on every hop, and a maximum hop count enforced outside the agents.

5. Swarm? Almost certainly not. Benchmark your actual task first.

Across these patterns, three things apply universally: serialize full state at every handoff (not just the answer), enforce token and iteration budgets at the framework level (not by trusting the agent to stop), and put a human checkpoint before every irreversible action.

The Agent Evaluation article covers how to measure all of this rigorously. Multi-agent systems that skip evaluation and go straight to production are the ones that become the cautionary stories that the next batch of engineers reads before building their own.

The gap between the pilot and production is not the architecture diagram — it is the operational discipline around state, cost, and failure modes that the demos never show. Getting that discipline right, for multi-agent systems, starts with measuring each agent's reliability in isolation before wiring them together.