Multi-Agent Debugging: Finding the Bug Across 12 Concurrent LLM Calls

What Makes Multi-Agent Bugs Different

Single-agent bugs are usually "the model got it wrong." Multi-agent bugs are usually "the system got it wrong" — the individual agents look fine in isolation, but their composition produced a wrong outcome. Two patterns dominate:

• Race conditions: two agents wrote to shared state in an order the system did not expect
• Compositional drift: each agent's output was acceptable individually, but the cumulative effect of 12 agents added up to a wrong answer

Debugging these requires tooling that single-agent debugging usually does not need.

The Trace-First Mindset

flowchart LR
    Run[Multi-agent run] --> Trace[Distributed trace<br/>with parent/child spans]
    Trace --> View[Trace viewer<br/>span explorer]
    Trace --> Replay[Replay engine]
    Trace --> Diff[Diff vs known-good run]

The single most valuable debugging investment for multi-agent systems is OpenTelemetry-shaped traces. Every LLM call, every tool call, every inter-agent message is a span with parent/child relationships and structured attributes (model, prompt hash, token counts, cost, latency).

In 2026 the open-source stack for this is: OpenTelemetry as the wire format, Phoenix or Langfuse as the viewer, and a custom or vendor (Braintrust, LangSmith, Helicone) overlay for LLM-specific attributes.

The Five Debug Patterns

1. Span Diff

Compare a failing run to a known-good run, span by span. Differences in tool inputs, prompt content, or model outputs jump out. This catches "the orchestrator slightly rephrased the task and worker C now misroutes" bugs.

2. Replay From Span

Rerun the system from a specific span using the captured inputs, with optional substitutions (different model, different prompt, different tool result). This catches "if I had used the right tool at step 7, would the rest have worked?" hypotheses.

3. Synthetic-Failure Injection

Replay a known-good run but replace one tool result with an error. Watch the agents respond. Catches "what happens if the database is slow?" failure-mode questions.

4. Token-Stream Diff

When two runs diverge, compare LLM streams token-by-token to find the exact divergence point. Catches "why did the same prompt produce different output today?"

5. Causality Tree

Build a tree of "what caused what" — every span has parents, every output has source spans. Walk backward from the bad output to the root cause. The Phoenix viewer ships this view in 2026.

A Concrete Bug Hunt

Symptom: every 50th customer-support session ends with the agent recommending the wrong product. The orchestrator and three workers all look fine in isolation.

Steps:

1. Pull all failing traces; cluster them
2. Find a common feature across failures: all involved orders shipped to a state where tax behavior differs
3. Inspect span attributes: the tax-calculator worker returns different field shapes for those states
4. The orchestrator's prompt assumes a flat shape; for the differing shape it silently picks the wrong field
5. Fix: schema-validate worker outputs at the orchestrator boundary; fail loudly on mismatch

This kind of bug is invisible without traces. With traces, it took an afternoon.

Patterns That Make Debugging Easier

• Schema-validate every inter-agent message. Pydantic on Python, Zod on TS. Strict, with errors that include the offending payload.
• Use stable IDs everywhere. Run ID, task ID, span ID. Pass them in tool calls, log them in tool results.
• Snapshot the world. Database state, queue depth, environment variables at run start. Without these, "I cannot reproduce" is your default state.
• Tag every span with the model and prompt hash. Model bumps and prompt edits are the hidden cause of drift.

A Reference Stack

flowchart LR
    Code[Agent code] -->|OTel SDK| Coll[OTel Collector]
    Coll --> Phx[Phoenix / Langfuse]
    Coll --> Met[Metrics: Grafana]
    Coll --> Logs[Logs: Loki]
    Phx --> Diff[Diff + Replay UI]
    Phx --> Repl[Replay engine]

This is the stack we run for CallSphere's multi-agent orchestration. Total instrumentation cost is a single-digit percent of agent cost; the debugging speedup is more than 10x.

Sources

• OpenTelemetry GenAI semantic conventions — https://opentelemetry.io/docs
• Phoenix tracing — https://docs.arize.com/phoenix
• Langfuse — https://langfuse.com
• "Debugging multi-agent systems" 2025 — https://arxiv.org
• Anthropic engineering on agent observability — https://www.anthropic.com/engineering

## Multi-Agent Debugging: Finding the Bug Across 12 Concurrent LLM Calls — operator perspective When teams move beyond multi-Agent Debugging, one question shows up first: where does the agent loop actually end? In practice, the boundary is rarely the model — it is the contract between the orchestrator and the tools it calls. What works in production looks unglamorous on paper — small specialized agents, explicit handoffs, deterministic retries, and dashboards that show you tool latency before they show you token spend. ## Why this matters for AI voice + chat agents Agentic AI in a real call center is a different beast than a single-LLM chatbot. Instead of one model answering one prompt, you orchestrate a small team: a router that decides intent, specialists that own a vertical (booking, intake, billing, escalation), and tools that read and write to the same Postgres your CRM trusts. Hand-offs are where most production bugs hide — when Agent A passes context to Agent B, anything that isn't explicit in the message gets lost, and the user feels it as the agent "forgetting." That's why the systems that hold up under load are the ones with typed tool schemas, deterministic state stored outside the conversation, and a hard ceiling on tool calls per session. The cost story is just as important: a multi-agent loop can quietly burn 10x the tokens of a single-LLM design if you let it think out loud at every step. The fix isn't a smarter model, it's smaller agents, shorter prompts, cached system messages, and evals that fail the build when p95 latency or per-session cost regresses. CallSphere runs this pattern across 6 verticals in production, and the rule has held every time: the agent you can debug in five minutes will out-survive the agent that's "smarter" on a benchmark. ## FAQs **Q: When does multi-Agent Debugging actually beat a single-LLM design?** A: Scaling comes from constraint, not capability. The deployments that hold up keep each agent narrow, cap tool calls per turn, cache the system prompt, and pin a smaller model for routing while reserving the larger model for synthesis. CallSphere's stack — 37 agents · 90+ tools · 115+ DB tables · 6 verticals live — is sized that way on purpose. **Q: How do you debug multi-Agent Debugging when an agent makes the wrong handoff?** A: Hard ceilings beat heuristics. A maximum step count, an idempotency key on every tool call, and a fallback to a deterministic script when confidence drops below a threshold are what keep the loop bounded. Evals that simulate noisy inputs catch the rest before they reach a real caller. **Q: What does multi-Agent Debugging look like inside a CallSphere deployment?** A: It's already in production. Today CallSphere runs this pattern in After-Hours Escalation and IT Helpdesk, alongside the other live verticals (Healthcare, Real Estate, Salon, Sales, After-Hours Escalation, IT Helpdesk). The same orchestrator code path serves voice and chat — the difference is the tool set the router exposes. ## See it live Want to see after-hours escalation agents handle real traffic? Spin up a walkthrough at https://escalation.callsphere.tech or grab 20 minutes on the calendar: https://calendly.com/sagar-callsphere/new-meeting.