1 Overview
Almost everyone who runs an A/B test has done it: you launch the experiment, then check the dashboard every few hours, and the moment the p-value dips below the conventional threshold you declare a winner. It feels decisive. It feels like good stewardship of traffic. It is also one of the most common ways to fool yourself, because the statistical guarantee attached to that threshold assumes you looked once, at a sample size you chose in advance, and then stopped. When you look repeatedly and stop on the first favorable result, you are no longer running the test you thought you were running.
The problem is not curiosity. The problem is that each peek is another opportunity for random noise to produce a flattering result. If you check often enough, a test with no real effect will eventually show a difference that crosses the threshold, and you will stop right there, having converted noise into a decision. Sequential testing exists to make repeated looks honest. Instead of pretending you will only look once, it builds the looking into the design, adjusting the evidence threshold so that the overall error rate stays under control no matter how many times you check.
This article explains why peeking breaks conventional tests, how sequential methods restore validity, and how to design, size, and read a sequential experiment. The emphasis throughout is on setup: what you decide before launch determines what your result can mean. No method rescues a poorly framed question, a contaminated assignment, or a sample too small to detect the effect you care about.
2 Framing the question
Before choosing a sequential design, write down the decision the test is meant to inform. A useful hypothesis names the change, the population, and the direction of the expected effect. For example: adding a trust badge to the checkout page will increase completed purchases among first-time visitors. That statement is specific enough to be tested and specific enough to be wrong. Vague goals such as improve the experience cannot be tested, because no outcome can confirm or disconfirm them.
Next, choose one primary metric and commit to it. The primary metric is the single number that decides the outcome. Secondary metrics can be observed, but they should not become the reason to declare victory after the fact. If you let yourself switch to whichever metric looks best, you have reintroduced the peeking problem in a different costume. Decide in advance what a meaningful change looks like, in practical terms, and what change would be too small to act on even if it were real.
Finally, specify the decision rule. What will you do if the sequential boundary is crossed in favor of the treatment? What if it is crossed against? What if the test reaches its maximum sample size without crossing? Writing these answers down before launch protects you from motivated reasoning later. It also forces you to confront whether the experiment can actually change a decision. If every possible result leads to the same action, the test is theater, not evidence.
3 Designing the test
A sequential design starts with the same foundations as any randomized experiment: random assignment, a control, a treatment, and a pre-specified primary metric. What changes is the analysis plan. Rather than fixing a single sample size and a single critical threshold, you define a sequence of analysis points, or an always-valid boundary that permits continuous monitoring, and you choose boundaries that control the overall error rate across all those looks. The boundary is stricter early, when little data has accumulated, and relaxes as evidence grows.
There are several families of sequential methods. Group sequential designs specify a small number of interim analyses, often at fractions of the planned maximum sample size, with adjusted thresholds at each look. Always-valid approaches produce a test statistic that can be evaluated at any time without inflating error, which suits dashboards that update continuously. Some designs also include futility boundaries, which allow you to stop early when the data suggest the effect is too small to matter, saving traffic for better ideas. The right choice depends on your traffic volume, how quickly you need an answer, and how costly a wrong decision would be.
Whatever method you choose, document it before launch. Record the maximum sample size, the timing or frequency of looks, the boundary values, and the stopping rules. If your tool computes sequential boundaries automatically, verify what assumptions it makes and what error rate it controls. A sequential design is only honest if the plan is fixed in advance and followed. Changing the boundary mid-test, or adding looks that were not planned, undermines the guarantee you paid for.
4 Sample size and validity
Sample size planning for sequential tests is more involved than for fixed-horizon tests, because the maximum sample size must account for the possibility of stopping early and for the adjusted boundaries. The practical inputs are the same: the baseline rate of the primary metric, the smallest effect worth detecting, the desired error rates, and the amount of traffic available. If the traffic needed exceeds what you can realistically collect, the honest conclusion is that the experiment is underpowered, not that the effect is absent.
Validity threats deserve explicit attention. Peeking without adjustment is the central one, but it is not alone. Multiple comparisons across many metrics or segments inflate false positives. Mid-test changes to the metric definition, the audience, the assignment, or the treatment experience can invalidate the comparison. Uneven exposure, bots, and duplicate accounts can contaminate the sample. Novelty and primacy effects can make early results unrepresentative of the long-run effect. Seasonal events and concurrent campaigns can confound the result if they affect arms differently.
Sequential methods address the repeated-looks problem specifically. They do not fix a biased metric, a broken randomization, or a sample too small to detect the effect you care about. Treat the sequential boundary as one safeguard among several. Pre-register the design, monitor data quality, and be willing to declare a test inconclusive when the setup cannot support a confident answer. Results always depend on the setup and the sample size, and no boundary adjustment can manufacture evidence that the data do not contain.
5 Running and reading the results
During the test, evaluate the sequential statistic only at the planned looks, or continuously if the method is always-valid. At each evaluation, compare the statistic to the pre-specified boundary. If it crosses in favor of the treatment, you may stop and treat the result as evidence of a difference. If it crosses against, you may stop and conclude the treatment is worse. If it does not cross, continue to the next planned look or to the maximum sample size. Resist the urge to interpret a near-miss as a trend; the boundary exists precisely to prevent that.
When the test reaches its maximum without a crossing, the result is not a failure of the method. It means the data did not provide enough evidence to distinguish the treatment from control at the chosen error rate. Report that honestly, along with the observed effect and its uncertainty. An inconclusive result is informative: it tells you the effect, if any, is smaller than what the test was powered to detect, or that the metric is too noisy to resolve the question at your traffic level.
If the boundary is crossed, interpret the effect size, not just the direction. A statistically detectable improvement may still be too small to justify the cost of shipping the change. Consider secondary metrics and guardrail metrics for signs of harm, while remembering that they were not the basis for stopping. Document what was decided, what evidence supported it, and what remains uncertain. The goal is a decision you can defend, not a p-value you can celebrate.
6 What to change next
After a sequential test concludes, the next step depends on what you learned. If the treatment won and the effect is practically meaningful, plan a rollout and continue monitoring the primary metric after launch, since the experimental estimate may not hold exactly in the full population. If the treatment lost or the result was inconclusive, revisit the hypothesis. Perhaps the change was too small, the audience was wrong, or the metric did not capture the intended benefit. Each of those is a different problem with a different next experiment.
Improve the experimentation system itself. If you found yourself tempted to peek, that is a signal to adopt sequential boundaries or always-valid inference so that monitoring is legitimate. If tests routinely end inconclusive, examine power: you may be asking questions that your traffic cannot answer, in which case larger effects, coarser metrics, or longer horizons are needed. If results often fail to replicate, audit assignment, logging, and metric definitions before running more tests.
Finally, build a habit of pre-registration. Write the hypothesis, primary metric, design, maximum sample size, boundary, and decision rule before launch, and store that record where the team can see it. Sequential testing is not a trick for getting significance faster; it is a discipline for looking honestly. The method matters, but the discipline of deciding in advance what would count as evidence matters more. Results depend on setup and sample size, so invest in the setup.
7 Frequently asked questions
Does sequential testing let me stop earlier than a fixed-horizon test? It can, when the effect is large, because the boundary may be crossed before the maximum sample size. But the maximum sample size is typically larger than a comparable fixed-horizon test, since the design pays a cost for the flexibility of interim looks. If the effect is small, sequential tests often take longer, not shorter. The benefit is validity under monitoring, not guaranteed speed.
Can I just use a lower p-value threshold and keep peeking? Lowering the threshold reduces the false-positive rate somewhat, but without a proper sequential method the error rate still depends on how often you look and when you stop. A fixed stricter threshold does not account for the number of looks or their timing. Use a method designed for repeated analysis rather than an arbitrary adjustment.
What if my tool reports a significant result at an unplanned look? Treat it as a signal to investigate, not as a final answer. The guarantee applies to the pre-specified plan. An unplanned crossing is exactly the situation sequential methods are meant to prevent you from overinterpreting. Continue to the planned analysis or rerun the test with a proper design.
Does sequential testing work for any metric? It works for the metric you pre-specify and analyze with the sequential method. Metrics added later, or segments chosen after seeing the data, do not inherit the validity. Keep the primary metric fixed, and treat exploratory findings as hypotheses for the next test. And remember that no method overcomes a sample too small for the effect you hope to detect.