ProductEngineering
Shipping gradual prompt rollout
How we built percentage-based traffic splitting for prompt revisions, and the design decisions that were harder than expected.
Sofia Lindqvist ·
Blog
Engineering notes, evaluation techniques and product updates from the team building Cloudmind.
How we cut our own LLM bill by 40% without changing a model
Three changes, none of them a model swap, and what each one was worth.
Dana Whitfield ·
Prompt versioning is a deploy problem, not a document problem
Why keeping prompts in a doc, or in your repo, both break at the same point.
Marcus Osei ·
- EvaluationGuides
How to build your first LLM golden dataset
A practical method for going from zero to a 50-example golden dataset that actually catches regressions, using traces you already have.
Priya Nandakumar ·
What a good eval suite looks like for a support chatbot
Twelve graders, and why the boring ones catch the most.
Priya Raman ·
Token costs are not the whole story: measuring latency users feel
Time to first token is the number your users experience. Most dashboards show the wrong one.
Dana Whitfield ·
- EvaluationOpinion
LLM-as-judge: when to trust it and when not to
LLM judges are useful and widely misused. Here is how to measure whether yours is good enough to gate a deploy on.
Marcus Hale ·
Tracing multi-agent systems without drowning in spans
What to record, what to collapse, and what to leave out entirely.
Marcus Osei ·
The three questions to ask before you fine-tune
Most teams that want to fine-tune have a prompt problem or a retrieval problem.
Priya Raman ·
- EngineeringCost
The hidden cost of agent retries
A walkthrough of how a silent retry loop tripled one team's token bill, and how nested tracing made it obvious in ten minutes.
Sofia Lindqvist ·
Why your RAG pipeline gets worse in production
The retrieval was fine in testing because you wrote the test questions.
Dana Whitfield ·
Reading a trace: a five-minute walkthrough
From a user complaint to the exact model call, with nothing skipped.
Marcus Osei ·
- EngineeringOpinion
Why prompts should not live in your codebase
The case for moving prompts out of source control and into a versioned registry, and the objections that are actually right.
Marcus Hale ·
Structured outputs, and what to do when the model ignores them
Schema enforcement moves the failure rather than removing it.
Priya Raman ·
Shipping an AI feature behind a flag
How to roll out a model change to 5% of traffic and actually learn something.
Dana Whitfield ·
- EvaluationRAG
RAG evaluation: what actually matters
Retrieval quality, not generation quality, is where most RAG systems fail. Here is how to measure the two separately.
Priya Nandakumar ·
How to run an offline eval against last week’s traffic
The eval set you need is already in your traces.
Marcus Osei ·
Hallucination is a product problem before it is a model problem
What to change in the interface when the model is confidently wrong.
Priya Raman ·
- OpinionEngineering
What observability vendors get wrong about LLMs
Traditional APM assumes failures are loud. LLM failures are quiet, well-formed and return 200 OK. That difference breaks most of the tooling.
Marcus Hale ·
- Company
Cloudmind raises $18M to make AI quality measurable
Announcing our Series A, led by Ridgeline Ventures, and what we are building next.
Cloudmind Team ·