Rollout Policies
Gradually roll out winning variations with automated safety checks.
What Rollout Policies Do
Once an experiment identifies a winning variation, you do not have to flip a switch and send 100% of traffic there immediately. Rollout policies give you controlled, gradual progression — starting with a small percentage of users and ramping up over time, with automatic safety checks at every step.
If something goes wrong during the rollout — say the winning variation causes an unexpected increase in error rates — the policy can automatically halt the rollout before it reaches all users. This gives you the confidence to ship faster, knowing that there is a safety net in place.
Four Rollout Modes
You can choose the rollout mode that best fits your risk tolerance and how much automation you want:
| Mode | How It Works | Best For |
|---|---|---|
| Manual | You control each step yourself. You decide when to increase the percentage of users seeing the new variation. The system only intervenes automatically to halt the rollout if a safety metric regresses. | High-stakes changes where you want full control over the pace, or teams new to progressive rollouts. |
| Scheduled Steps | The rollout advances automatically on a time schedule you define. For example: 5% for 24 hours, then 25% for 48 hours, then 100%. The system halts if a safety check fails. | Predictable, time-boxed rollouts where you want steady progression without manual intervention. |
| Adaptive Steps | The rollout advances only when the experiment results support it. If the data says "ship," the next step begins. If the data is inconclusive, the rollout holds at the current level. If the data indicates a problem, the rollout halts. | Data-driven rollouts where you want the pace to be dictated by statistical confidence rather than a fixed schedule. |
| Adaptive Continuous | Instead of predefined steps, the traffic percentage increases smoothly by a configured increment each time the results show confidence. This creates a gradual, continuous ramp rather than discrete jumps. | Smooth, automated rollouts where you want the finest-grained control tied directly to results. |
How Steps Work
For the Manual, Scheduled Steps, and Adaptive Steps modes, you define a series of steps that represent the progression of your rollout. Each step specifies a target traffic percentage for the winning variation.
A typical three-step rollout might look like this:
- Step 1: 5% of users see the new variation (initial canary).
- Step 2: 25% of users see the new variation (broader validation).
- Step 3: 100% of users see the new variation (full rollout).
For Scheduled Steps, each step also has a duration — how long to wait before advancing to the next step. For Adaptive Steps, advancement happens when the analysis engine confirms the results are strong enough.
Guardrails — Automatic Safety Checks
Guardrails are the most important safety feature of rollout policies. They run on every evaluation, regardless of which mode you are using. Guardrails check your guardrail goals — safety metrics that should not get worse — and halt the rollout immediately if they detect a problem.
For example, you might set up guardrails for error rates, page load times, or revenue per user. If your new feature improves signup rates (your primary goal) but causes a spike in errors (your guardrail), the rollout will halt automatically before the damage spreads to all users.
Default Safety Thresholds
The platform comes with sensible defaults that you can adjust per experiment:
| Threshold | Default Value | What It Means |
|---|---|---|
| Minimum Confidence | 95% | The analysis must be at least this confident before advancing to the next step (used by adaptive modes). |
| Guardrail Confidence | 95% | A regression in a guardrail metric must reach this confidence level before triggering a halt. |
| Minimum Regression | 0.5% | The guardrail metric must regress by at least this much to trigger a halt. This prevents halting on trivially small regressions. |
Policy Events — The Rollout Log
Every action the rollout controller takes is recorded as an event, creating a complete audit trail. You can review the history to understand exactly what happened and why at every step of the rollout:
| Event | What Happened |
|---|---|
| Advanced | Traffic was increased to the next step or incremented by the configured percentage. The event includes the old and new traffic levels. |
| Held | The controller evaluated results but chose not to advance — the data was not yet conclusive enough. The rollout stays at its current level. |
| Halted | The rollout was stopped because a guardrail detected a regression or the analysis indicated the variation is underperforming. The event includes the specific reason. |
| Completed | The rollout reached 100% and the experiment was marked as completed. The winning variation is now live for all users. |
Each event includes a human-readable explanation of why the action was taken. For example: "Advanced from 5% to 25% — primary goal confidence is 97%, which exceeds the 95% threshold."