1 Overview
Peeking at an A/B test before its planned end and stopping when the result looks significant is one of the most common ways to turn a valid experiment into a misleading one. In a fixed-horizon test, the sample size and analysis time are determined in advance, and the false-positive rate is calibrated for a single look at the data. When you check repeatedly and stop at the first significant result, you are effectively running many tests and reporting the most favorable one. This inflates the chance of a false positive, sometimes dramatically, depending on how often you peek and how early you stop.
Sequential testing offers a principled alternative. Instead of pretending you will look only once, a sequential design pre-specifies how many interim analyses you will perform and how extreme the result must be at each look to stop for success or futility. The stopping boundaries are adjusted so that the overall false-positive rate remains controlled, even though you are looking multiple times. This does not mean you can peek arbitrarily; it means you can peek legitimately if the rules are set in advance.
The trade-off is that sequential designs often require a slightly larger maximum sample size than a fixed-horizon test to achieve the same power, because the boundaries are stricter at each interim look. In exchange, you gain the ability to stop early when the effect is large, which can save time and resources. The choice between fixed-horizon and sequential testing depends on your context, but the key lesson is that unplanned peeking is not a neutral act. It changes the statistical properties of your test, and usually not in your favor.
2 Framing the question
The question a sequential test answers is not simply whether variant B differs from control A. It is whether the evidence is strong enough to stop the experiment early, given that you have looked at the data multiple times. This requires a clear hypothesis before the test begins. For example, you might hypothesize that a new checkout flow increases conversion rate compared with the current flow. The metric is the conversion rate, but the analysis is not a single comparison at the end; it is a sequence of comparisons at pre-specified information fractions, such as after 25 percent, 50 percent, 75 percent, and 100 percent of the planned sample.
Framing the question this way forces you to decide in advance what constitutes a meaningful effect and how much evidence you need to stop. It also clarifies that the goal is not to find any significant result as quickly as possible, but to make a reliable decision with controlled error rates. If you instead frame the question as "is the result significant right now?" you are inviting optional stopping, which undermines the validity of the test.
A good framing also considers the direction of the effect and whether you are testing for superiority, non-inferiority, or equivalence. Sequential boundaries can be one-sided or two-sided, and they can be symmetric or asymmetric depending on whether you want to stop early only for success, only for futility, or both. These choices should be driven by the decision you need to make, not by the convenience of the analysis software. The clearer the question, the easier it is to design a sequential test that actually answers it.
3 Designing the test
Designing a sequential test starts with the same elements as any randomized controlled experiment: a control group, one or more variants, random assignment, and a primary metric. The difference is that you must also specify the number and timing of interim analyses, the stopping boundaries, and the maximum sample size. Common approaches include group sequential designs with alpha spending functions, which allocate the total false-positive rate across the interim looks. The boundaries are wider at early looks and narrower at later looks, reflecting the fact that early evidence is noisier and more prone to false positives.
You also need to decide whether to stop only for success, only for futility, or both. Stopping for futility can be useful when the effect is small or negative, but it requires its own boundary and can affect power. The design should be pre-registered or at least documented before the test begins, including the exact test statistic, the information fractions, and the decision rules. This prevents post hoc adjustments that would invalidate the error control.
Practical constraints matter too. Sequential designs can be more complex to implement and analyze than fixed-horizon tests, and they may require specialized software or statistical expertise. If your team lacks that expertise, a simpler alternative is to commit to a fixed horizon and avoid peeking entirely, or to use a Bayesian approach with explicit priors and decision rules. The worst option is to run a fixed-horizon test while secretly peeking and stopping when it suits you. That is not a design; it is a recipe for false discoveries.
4 Sample size and validity
Sample size in a sequential test is not a single number but a maximum. You plan for a maximum sample size that gives you the desired power at the final analysis, and you allow for early stopping if a boundary is crossed. The actual sample size may be smaller, but it depends on the observed effect, the variance, the alpha level, the power, and the spending function. Because the boundaries are stricter at interim looks, the maximum sample size is often slightly larger than what a fixed-horizon test would require for the same power. This is the price of legitimate peeking.
Validity in a sequential test rests on pre-specification. If you choose the boundaries after seeing the data, or if you change the metric or population mid-test, the error control is broken. You also need to ensure that the interim analyses are based on the correct information fractions and that the test statistic is computed consistently across looks. Missing data, delayed conversions, and unequal variance can all complicate the analysis and threaten validity if not handled properly.
Another threat is multiple comparisons beyond the planned interim looks. If you also test secondary metrics or subgroups at each look without adjustment, you increase the chance of a false positive. Sequential designs control error for the primary metric and the planned looks, not for an unlimited number of exploratory analyses. Finally, the validity of any test depends on the sample size being adequate for the effect you care about. If the effect is small, even a sequential design may require a large sample to detect it reliably, and early stopping may rarely occur.
5 Running and reading the results
When you run a sequential test, you analyze the data at each pre-specified information fraction. At each look, you compute the test statistic and compare it to the stopping boundary. If the statistic crosses the boundary, you stop the test and declare a decision. If it does not, you continue to the next look or until the maximum sample size is reached. The final analysis, if you reach it, uses the adjusted boundary for the last look. The result is read as a decision, not as a p-value to be interpreted in isolation. The error rate is controlled across the entire sequence, so you can trust the decision if the design was followed.
It is important to distinguish between stopping for success and stopping for futility. Stopping for success means the evidence is strong enough to reject the null hypothesis at the adjusted boundary. Stopping for futility means the evidence suggests the effect is unlikely to reach significance even if the test continues, so you stop to save resources. Both are legitimate if pre-specified. What is not legitimate is stopping because you are impatient or because the result looks good at an unplanned moment.
Reading the results also means considering the practical significance, not just the statistical decision. A statistically significant effect at an interim look might be based on a smaller sample and could be less stable than a final result. You should report the stopping point, the boundaries, and the observed effect with a confidence interval that accounts for the sequential design. This gives stakeholders a clear picture of what was found and how much uncertainty remains, even though the error rate is controlled.
6 What to change next
If you have been peeking at fixed-horizon tests, the first change is to stop doing that. Either commit to a fixed horizon and wait until the planned sample size is reached, or adopt a sequential design with pre-specified boundaries. If you choose sequential, invest in the tools and training needed to implement it correctly. This includes software for group sequential or sequential analysis, and a clear analysis plan that specifies the information fractions, boundaries, and decision rules. Without that, you are just peeking with extra steps.
Next, review your experimentation culture. Are stakeholders demanding early results? If so, explain the trade-offs and set expectations. Sequential testing can provide early answers when effects are large, but it cannot guarantee them. For small effects, you may still need to run to the maximum sample size. Consider using a dashboard that shows the sequential boundaries and the current test statistic, so everyone can see whether the test is on track to stop early or not.
Finally, document and learn from each test. Record whether you stopped early, at which look, and what the adjusted effect estimate was. Over time, this builds intuition about how often early stopping occurs and how it affects decisions. You may find that sequential designs save time on some tests and require more planning on others. The goal is not to eliminate peeking but to make it statistically valid when it happens. That is a change worth making.
7 Frequently asked questions
Is peeking always bad? Peeking is bad when it leads to optional stopping without adjusted boundaries. If you look at the data but do not change your decision rule, and you still analyze at the planned end, the false-positive rate is not necessarily inflated. The problem is acting on what you see. Sequential testing allows you to act on interim looks because the boundaries are adjusted.
How many interim looks can I have? There is no fixed number, but more looks require stricter boundaries and a larger maximum sample size. Common designs use a handful of looks, such as three to five. The choice depends on the expected effect size, the cost of sampling, and the need for early decisions. You should pre-specify the number and timing.
Does sequential testing require a larger sample size? Often yes, for the same power, because the boundaries are stricter at interim looks. The increase depends on the number of looks and the spending function. If early stopping is likely, the expected sample size may be smaller than a fixed-horizon test, but the maximum is larger.
Can I use sequential testing for any metric? It works best for a single primary metric with a well-defined test statistic. For multiple metrics, you need to account for multiplicity, which can be complex. If you have many metrics, consider a hierarchical or composite approach, or run separate tests with appropriate adjustments.
What if I need to stop for a safety or business reason? Sequential designs can include futility boundaries, but unplanned stops for external reasons should be documented and interpreted cautiously. The statistical properties may not hold if the stop is not part of the design. In such cases, treat the result as exploratory rather than confirmatory.