The hidden cost of agent retries

A customer came to us with token costs up 3x over a quarter and traffic flat. Their logs showed nothing unusual. Every agent run completed successfully.

That last part was the clue.

Successful runs can be extremely expensive

Their agent had a tool that returned a structured object. The schema had drifted, so the model's tool call failed validation. The framework's default behavior was to retry with an error message appended to the context, up to eight times.

By attempt eight, the model had usually stumbled into a shape that validated. The run completed. The logs recorded one successful agent execution.

What the logs did not record: eight model calls, each with a context window that had grown by the previous attempt's error text. The last attempt in a chain was roughly four times the token count of the first.

Why flat logging hides this

If your instrumentation logs at the agent level, a retry loop is invisible by construction. You see input, output, success. The eight intermediate calls happened inside a library you did not write.

Nested tracing makes it structural rather than something you have to think to look for:

trace: reroute-shipment            12.4s   $0.38
├─ span: plan                       1.1s   $0.02
├─ span: tool.lookup_carrier        0.3s   $0.00
├─ span: tool.check_capacity        8.9s   $0.31   ← 8 model calls
│  ├─ attempt 1  schema validation failed
│  ├─ attempt 2  schema validation failed
│  ...
│  └─ attempt 8  ok
└─ span: respond                    2.1s   $0.05

You do not need to be looking for a retry loop to see that one span cost 80% of the trace.

Three things worth doing

Cap retries and alert on the cap. Eight is a default nobody chose. Set it to two and fire an alert when you hit it, because hitting the cap means something is actually broken.

Price your spans, not just your requests. Cost attribution at the trace level tells you the total. Cost at the span level tells you where it went.

Validate tool schemas in CI. This particular bug was a schema that drifted from its consumer. A contract test would have caught it before it ever cost anything.

The fix here was one line of schema correction and a retry cap. Total spend dropped 68% below the original baseline, because the retry loop had been running at lower volume for months before anyone noticed the trend.