Shipping gradual prompt rollout

Gradual rollout shipped this week. You can now send 10% of traffic to a new prompt revision, watch the eval score on each arm, and ramp or revert. Here is what was harder than we expected.

Assignment has to be sticky

The obvious implementation is a random draw per request. That is wrong for anything conversational: a user gets revision 14 on their first message and revision 13 on the follow-up, and the conversation reads as incoherent.

We hash a stable key you provide, usually a session or user ID, so a given user stays on one arm for the duration of the rollout. If you do not provide a key we fall back to per-request assignment and warn about it in the SDK, because silently doing the wrong thing is worse than being noisy.

Statistical significance is mostly a lie at this volume

Our first design showed a confidence interval on the score difference. We removed it before launch.

Most teams rolling out a prompt have hundreds of samples per arm, not tens of thousands. At that volume the interval is so wide it is uninformative, but presenting it makes people believe they are doing something rigorous. We now show the raw scores, the sample count per arm, and a plain-language note about how many samples you would need for the difference you are seeing to be meaningful.

Less impressive-looking, considerably more honest.

Failure has to default to the old revision

If the assignment service is unreachable, every request serves the current production revision. Not the new one, not an error.

This sounds obvious. It was not our first implementation, which threw on assignment failure, and that turned a degraded dependency into an outage in staging. A rollout mechanism that can take down the feature it is rolling out is not a rollout mechanism.

What we cut

Multi-armed rollouts. More than two arms at once, at typical sample volumes, means no arm gets enough data to say anything. We may add it once someone shows us a real use case with the traffic to support it.

Automatic ramping. Auto-promote when the new arm scores higher sounds great and would confidently promote noise. The human in the loop is doing real work there.

Available now on Team and above. Docs are in the prompt versioning guide.