Skip to content

Token costs are not the whole story: measuring latency users feel

Dana Whitfield ·

A request that takes nine seconds but starts streaming at 400ms feels fast. A request that takes three seconds and streams nothing until the end feels broken. Total duration cannot tell those apart.

What we changed

  • Record time to first token separately from total duration.

  • Track it at p95, because the average hides exactly the requests people complain about.

  • Watch it per prompt version — a longer system prompt moves this before it moves cost.

The full walkthrough is in the documentation.

Token costs are not the whole story: measuring latency users feel | Cloudmind