Eight to ten seconds of silence: what we learned when AI agents talk to each other

If a planning agent takes 2 seconds and three workers then run one after another, the person on the phone hears 8 to 10 seconds of silence. This year we built several systems in which a question is answered by several agents with different jobs, handing work to one another. The results are mixed. For some tasks it is the only viable route; elsewhere a simple deterministic workflow is cheaper, faster and more reliable. This is what we learned.

Tool use and MCP: a shared language

Agents can only collaborate because models can call tools: the model invokes a function with structured parameters and carries on reasoning with the result. The Model Context Protocol (MCP) standardises this. Tools live on a separate server with a self-describing schema, and any agent, on any model provider, reaches them the same way.

In our systems, internal services (CRM lookup, calendar booking, call initiation, billing status) are exposed as MCP servers, and agents touch nothing except through them. That brings two benefits: tools can be tested and permissioned in one place, and the agent’s prompt carries none of the integration details.

The main lesson in tool design is that a few well-defined tools work better than many small ones. A “find the customer by email” tool is more reliable than a general-purpose SQL runner.

Planners, workers, handoffs

We use two patterns regularly.

In the planner/worker setup, one agent breaks the task into steps and hands each to a worker running on a cheaper model with a narrower context. The worker never sees the whole conversation, only its own task and the tools it needs. That cuts cost and the room for error, but the whole system is only as good as the planner.

In the handoff pattern, one agent passes the conversation to another at a given point, with the full or a filtered context. In voice-based customer support this is the standard case: a general reception agent recognises that the caller wants an appointment and passes them to the booking agent, which has narrower but more precise instructions and tools. The caller notices nothing; for the system it is a clean state change.

In both patterns communication is structured. Agents exchange schema-validated messages and do not “chat” in free text. Free-text dialogue between agents looks impressive in demos; in practice it is long, expensive and hard to debug.

Limits live in the code

More agents mean more ways to fail, and failures spread. So we put the lines of defence in code and do not leave them to the prompt:

• every tool call goes through a permission check based on the agent’s identity and the tenant;
• write operations (booking, sending email, changing status) are on an explicit allowlist, and all other tools are read-only;
• input and output are schema-validated; if the agent returns a bad format, we retry once and then report an error;
• step-count and cost limits per agent and per run, so that an agent going in circles cannot run forever.

Some decisions are never taken automatically: refunds, changes to contractual terms, closing an escalated complaint. For those the agent drafts a proposal and a human approves it, from a single line, with one click. Human-in-the-loop is a designed part of the system. One well-defined outcome of an agent’s work is “ask a human”.

One run, one trace

Debugging a multi-agent run without tracing is hopeless. Every run is a single trace, and within it each agent step, model call and tool call is a span, with input, output, token count and duration. Trace identifiers carry across handoffs, so the whole path is visible in one place.

The same data drives cost and latency analysis. With several agents, response times add up: a 2-second planner step followed by three sequential workers means an 8 to 10 second wait. In a voice system that is unacceptable, so there workers run in parallel wherever possible and the caller hears filler feedback in the meantime. Costs multiply easily too, because every agent resends its own context again and again. Prompt caching and narrow-context workers make a large difference here.

Workflow first

Multiple agents are justified when the steps of a task are not known in advance, when subtasks need different knowledge and toolsets, and when a wrong step can be reversed or tied to human approval. A typical case is a complex support ticket where identification, handling and follow-up touch different systems.

They are not justified when the order of steps is known and fixed. “Write a summary after the call, email it, update the CRM” is a plain workflow with an LLM in exactly one step, the summary. An agent-based design there only adds uncertainty, cost and latency.

Our rule is simple. First, describe the process as a deterministic workflow. Where that fails because a decision genuinely depends on context, an agent comes in. Where one agent’s context grows too large, we split it into several. Working the other way round, starting from agents, almost always produces an over-complicated system.