How we cut our own LLM bill by 40% without changing a model

We spent four months assuming our bill was a model-choice problem. It was not. Once we could see spend broken down by feature, three quarters of it turned out to be requests nobody needed to make.

What we changed

• Cached the system prompt, which was 1,800 tokens resent on every turn.

• Stopped retrying on a validation error the model was never going to fix.

• Moved summarization to a smaller model after an eval showed no quality difference.

The full walkthrough is in the documentation.