What observability vendors get wrong about LLMs

Every APM vendor has shipped an LLM feature in the last two years. Most of them are a trace view with token counts bolted on, and they miss the thing that makes this problem different.

Traditional monitoring assumes failures are loud

The entire model of APM is built on signals that failure announces itself: exceptions, non-2xx status codes, latency spikes, error rates. Your dashboard is green because nothing threw.

An LLM failure is a 200 OK. It arrives in well-formed JSON, in the right schema, within the p50 latency band. It is simply wrong. Every signal APM watches says the system is healthy.

This is not a gap you close by adding token count to a span.

Quality is a first-class signal or you have nothing

If wrongness is the primary failure mode, then something has to score outputs, and that scoring has to be as central as latency is in APM. Which means the platform needs:

• Somewhere to define what good means, per feature, versioned.
• A way to run that definition against historical data, not just live traffic.
• Alerting on the score, not just on errors.

Bolting a "quality" tab onto a product architected around error rates does not get you there, because the data model has no place to put ground truth.

Non-determinism breaks reproduction

The other assumption APM makes is that you can reproduce a failure by replaying the request. Same input, same output, debug from there.

Temperature above zero means the same input produces a different output every time. So debugging shifts from "reproduce it" to "find enough similar cases to see the pattern." That requires semantic search over completions, grouping by failure mode, and distribution views rather than individual traces.

Tools built on exact-match log search are the wrong shape for this.

The version dimension nobody tracks

In a normal service, code version is the thing that changed. In an LLM feature there are at least four independent version axes: your prompt, the model, the retrieval index, and the tool schemas. Any of them can move without a deploy.

A model provider silently updating a checkpoint behind the same version string has broken more features than most people realize. If your observability tool does not record which prompt revision and which model actually served each request, you cannot correlate a quality drop with what changed, and you are back to guessing.

That is the real bar. Not "do you show token counts," but "can you tell me what changed when quality dropped."