1 Overview
Most teams do not run an A/B test and then look at it once. They open the dashboard on the first morning, again after lunch, and repeatedly over the following days. That behavior feels harmless, but it quietly changes the statistical question being answered. A conventional fixed-horizon test is calibrated for exactly one look at a pre-planned sample size. When you check repeatedly and stop when the result looks significant, you are running a different procedure with a different error rate, and the tool rarely warns you.
Sequential testing is the family of methods designed for this reality. Instead of pretending you will look once, it builds the monitoring schedule into the decision rule. Group sequential designs place boundaries at pre-specified interim looks, and always-valid confidence sequences allow continuous monitoring without inflating the false positive rate. The trade-off is real: boundaries are more conservative at each look, and you may need a somewhat larger maximum sample size to preserve power.
The core idea is that evidence accumulates, and the threshold for declaring a winner should depend on how much evidence you have collected and how many times you have looked. A tool that reports a raw p-value on a live dashboard is not necessarily wrong, but it is answering a question you probably did not intend to ask.
2 Framing the question
Before choosing a method, write down the decision the test is meant to inform. Are you deciding whether to ship a change, whether to keep iterating, or whether to abandon a direction? The answer determines what counts as a result. If the decision is binary and costly, you want a rule that controls the probability of a false positive across the whole monitoring period, not just at one arbitrary moment.
Next, separate the primary metric from the diagnostics. The primary metric is the one the decision hinges on, and it should be chosen before the test starts. Diagnostics such as guardrail metrics, segment breakdowns, and funnel steps help you interpret the primary result, but they should not become post hoc primary metrics when they happen to look good. This distinction matters because sequential methods spend their error budget on a pre-specified set of comparisons.
Finally, state the null and alternative in plain language. The null is typically that the change has no effect on the primary metric, and the alternative is that it has some effect of a size you care about. Being explicit about the smallest effect worth detecting keeps the design honest and prevents the common trap of treating any non-zero movement as a win. The question is not whether the numbers differ, but whether the difference is large enough to justify acting.
3 Designing the test
A sequential design starts with the same inputs as a fixed-horizon design: a baseline rate, a minimum detectable effect, a desired significance level, and a desired power. From these you compute a sample size, then you decide how many interim looks to allow and when they occur. Common choices include a small number of equally spaced looks or looks timed to business milestones such as a weekly review. Each look consumes some of the error budget, so more looks generally require stricter boundaries.
There are two broad families worth understanding. Group sequential designs use pre-specified analysis times and adjusted critical values, often derived from an alpha spending function that controls how much false positive risk is spent at each look. Always-valid methods, sometimes called sequential confidence sequences, produce intervals that remain valid at every sample size, which suits dashboards that update continuously. Both approaches require the stopping rule to be fixed in advance; improvising looks after seeing the data undermines the guarantee.
Randomization and assignment also belong in the design. Users or sessions should be assigned consistently, the unit of analysis should match the unit of randomization, and the ratio between variants should be checked. If the design includes multiple variants or overlapping experiments, account for those comparisons. A clean sequential design is still fragile if the underlying assignment is contaminated.
4 Sample size and validity
Sample size in a sequential test is not a single number you can copy from a fixed-horizon calculator. The maximum sample size must be inflated to account for the interim looks, and the inflation depends on the number of looks, the spending function, and the effect size you want to detect. Underpowered tests are a persistent problem: with too little data, real effects fail to cross the boundary, and the test ends inconclusive rather than negative. Inconclusive is a legitimate outcome, but it should be planned for rather than discovered.
Validity depends on discipline as much as mathematics. Peeking without an adjusted boundary, adding interim analyses after the fact, switching the primary metric, or extending the test because the result was close to significant all break the error-rate guarantee. Multiple comparisons across variants, segments, or secondary metrics also inflate false positives unless they are controlled. Sample ratio mismatch, where the observed traffic split drifts from the intended split, is a strong signal that something in the assignment or logging pipeline is wrong.
Novelty effects, seasonality, and instrumentation changes can also threaten validity. A sequential boundary does not correct for a broken experiment; it only controls error rates under the assumptions of the design. Document the planned looks, the stopping rule, and any deviations, because the credibility of the result depends on whether the analysis matched the plan.
5 Running and reading the results
When the test runs, the sequential rule should be applied mechanically. At each planned look, compare the test statistic to the boundary. If it crosses, stop and act on the pre-specified decision. If it does not, continue to the next look or to the maximum sample size. Reaching the maximum without crossing means you do not declare an effect; it does not mean the change is harmful or that the question is permanently closed. It means the data did not provide enough evidence under the rule you chose.
Report the effect estimate with an interval that respects the sequential design, and avoid presenting a raw p-value as if it came from a single fixed look. The point estimate can still be useful, but it is often biased upward when a test stops early on a positive signal, because stopping preferentially happens when random noise pushes the estimate high. Intervals from always-valid methods are wider than naive intervals, and that width is the honest price of continuous monitoring.
Read the result in the context of the decision. A boundary crossing supports shipping if the effect is meaningful and guardrails are intact. A non-crossing at maximum sample size supports stopping or redesigning, depending on the observed interval and the cost of being wrong. Either way, the interpretation should reference the pre-specified metric, the planned sample size, and the monitoring schedule.
6 What to change next
If your current tool reports live significance without adjusting for repeated looks, the first change is procedural: decide in advance how often you will look and what rule you will apply. You can adopt a group sequential plan with a small number of looks, or move to an always-valid method if continuous monitoring is part of your workflow. Either choice should be documented before the test starts and applied consistently across experiments.
Second, revisit your sample size planning. Sequential designs need a maximum sample size that preserves power after accounting for interim looks. If your tests routinely end inconclusive, the problem may be an effect size that is too small to detect with the traffic you have, not a flaw in the statistics. Consider whether a larger minimum detectable effect, a longer run time, or a different metric would make the decision clearer.
Third, build guardrails around the analysis. Log the planned looks, freeze the primary metric, check the assignment ratio, and treat unplanned analyses as exploratory rather than confirmatory. If you use a vendor tool, ask how it handles sequential monitoring and whether its reported intervals are valid under repeated looks. The goal is not to eliminate judgment but to make the error rates you are accepting explicit, so that shipping decisions rest on evidence rather than on the most flattering moment in a noisy dashboard.
7 Frequently asked questions
Is sequential testing always better than a fixed-horizon test? No. If you genuinely look once at a pre-planned sample size, a fixed-horizon test is simpler and slightly more efficient. Sequential methods earn their keep when monitoring is frequent or when early stopping has real value. The right choice depends on your workflow and the cost of continuing a test versus the cost of a wrong decision.
Does sequential testing let me stop as soon as I see significance? Only if the boundary is crossed under the pre-specified rule. Seeing a small p-value on a dashboard is not the same as crossing a sequential boundary, because the boundary accounts for how many times you have looked. Stopping on an unadjusted p-value reintroduces the false positive inflation the method was meant to prevent.
Will sequential testing make my tests longer? It can increase the maximum sample size needed to preserve power, but it may also let you stop earlier when an effect is large. The net effect depends on the effect size, the number of looks, and the spending function. Results depend on the setup and the sample size, so there is no universal answer.
Can I use sequential methods with multiple variants or metrics? Yes, but the error budget must be managed across those comparisons, and the stopping rule should be defined for the family of tests you care about. Adding variants or metrics after the test begins is a validity threat, not a feature.