1 Overview
In A/B testing, the temptation to check results early and often is nearly universal. You launch a test, and within hours you are refreshing the dashboard, hoping to see a winner. This behavior, known as peeking, is one of the most common ways to invalidate an experiment. When you analyze data repeatedly without adjusting your statistical thresholds, the chance of a false positive grows far beyond the nominal alpha level. A test that is supposed to have a 5% false positive rate can easily reach 20% or more if you peek frequently and stop when the result looks significant.
Sequential A/B tests offer a principled solution. Instead of fixing the sample size in advance and analyzing only once at the end, a sequential design allows you to monitor the data as it accumulates while controlling the overall false positive rate. This is achieved by using pre-specified stopping boundaries that become more stringent as the test progresses. The result is a test that can stop early when there is a large, clear effect, saving time and resources, while still providing valid inference. In this article, I make the case for sequential testing as a default approach for many experimentation programs, and I explain how to design, run, and interpret such tests carefully.
2 Framing the question
Before choosing a sequential design, you must frame the question your experiment is meant to answer. The hypothesis should be specific and testable: for example, 'Changing the call-to-action button color from blue to green will increase the conversion rate on the product page.' The primary metric must be defined precisely, including the time window and the population. Secondary metrics can be tracked, but the stopping rule should be based on the primary metric to avoid cherry-picking.
The decision to use a sequential design should be driven by practical considerations. If the cost of running the test longer is high, or if you expect a large effect that you want to detect quickly, sequential testing can be advantageous. However, if the effect is likely small and you need high precision, a fixed-horizon test with a large sample size may be simpler. The key is to decide in advance whether you will monitor continuously or at pre-specified interim points, and to choose a stopping rule that aligns with your tolerance for false positives and false negatives. This framing ensures that the sequential design is not an ad hoc fix but a deliberate part of the experimental plan.
3 Designing the test
Designing a sequential A/B test requires specifying the stopping boundaries and the monitoring schedule before data collection begins. A common approach is the alpha-spending function, which allocates the total Type I error rate across interim analyses. For example, the O'Brien-Fleming spending function spends very little alpha early on, making it hard to stop for efficacy at the first look, but becomes more lenient later. This preserves power while allowing early stopping for overwhelming effects. Alternatively, Pocock boundaries spend alpha more evenly, which can lead to earlier stopping but at the cost of a larger maximum sample size.
You must also decide on the number and timing of interim analyses. Some sequential designs allow continuous monitoring, where a test statistic is computed after every new observation or at regular intervals. Others use a fixed number of looks, such as after 25%, 50%, and 75% of the maximum sample size. Continuous monitoring requires more complex computations and careful control of the error rate, but it is feasible with modern software. Regardless of the schedule, the boundaries must be pre-specified and followed exactly. Any deviation, such as adding an unplanned look or changing the boundary, invalidates the error control. The design should also include a futility boundary if you want to stop early for lack of effect, but this is optional and should be justified.
4 Sample size and validity
Sample size in a sequential test is not a single number but a maximum. You determine the maximum sample size by conducting a power analysis for the smallest effect size that is practically meaningful, the desired power, and the overall alpha. The sequential design may stop before reaching this maximum if the effect is large enough to cross the efficacy boundary. However, if the true effect is small or zero, the test will typically run to the maximum sample size. This means that sequential tests are not a free lunch: they can save sample when effects are large, but they do not reduce the required sample for small effects. In fact, the maximum sample size for a sequential design is often slightly larger than for a fixed-horizon test with the same power, because of the penalty for multiple looks.
Validity in sequential testing hinges on strict adherence to the pre-specified plan. The most serious threat is peeking without adjusting boundaries, which inflates the Type I error rate. Other threats include changing the primary metric mid-experiment, unequal randomization, and external events that differentially affect the arms. Novelty effects and instrumentation changes can also bias results. To protect validity, you should document the design, use software that implements the stopping rule correctly, and avoid making decisions based on unadjusted p-values. Remember that results depend on the setup and sample size; a significant result in a sequential test is only valid if the design assumptions hold.
5 Running and reading the results
When you run a sequential A/B test, you analyze the data at each pre-specified interim look using the sequential test statistic. If the statistic crosses the efficacy boundary, you stop the test and declare a statistically significant effect. If it crosses a futility boundary, you stop for lack of effect. Otherwise, you continue to the next look or to the maximum sample size. At the end, if no boundary is crossed, you fail to reject the null hypothesis. It is crucial to use adjusted p-values and confidence intervals that account for the sequential design; naive p-values will be misleading.
Reading the results requires care. A significant result at an interim look means that, under the null hypothesis and the pre-specified stopping rule, the probability of observing such an extreme test statistic is below the adjusted threshold. However, it does not mean the effect size is large or practically important; you should always report the estimated effect and its confidence interval. If the test stops for futility, you cannot conclude that the effect is exactly zero, only that the data are consistent with a range of effects that includes zero. Finally, remember that the actual sample size at stopping is random and depends on the observed data. This randomness is accounted for in the design, but it means that the precision of your estimate varies. Always interpret results in the context of the original hypothesis and the practical significance of the effect.
6 What to change next
After completing a sequential test, use the results to inform the next iteration of your product or experiment. If the test showed a significant positive effect, consider implementing the change and monitoring its long-term impact. If the effect was not significant, you may need to rethink the hypothesis or the intervention. In either case, the sequential design itself can be refined. For example, if you found that the test frequently stopped early for large effects, you might consider using a more conservative spending function to avoid stopping too early on noisy data. Conversely, if the test rarely stopped early, you might increase the maximum sample size or use a different boundary shape.
It is also important to evaluate the operational aspects of the sequential test. Did the interim analyses cause logistical challenges? Were the stopping rules easy to implement? Did the team understand the adjusted p-values? These process questions can lead to improvements in future tests. Additionally, consider whether the primary metric was the right one; sometimes a secondary metric becomes more important as the product evolves. Finally, document what you learned about the effect size and variability, as this information can improve power analyses for future experiments. Sequential testing is not a one-size-fits-all solution, but with careful design and interpretation, it can be a powerful tool for efficient and valid experimentation.
7 Frequently asked questions
Is sequential testing the same as peeking? No. Peeking means analyzing data repeatedly without adjusting the statistical thresholds, which inflates the false positive rate. Sequential testing uses pre-specified boundaries that control the overall error rate, so it is a valid form of interim analysis. The key difference is the pre-specification and adjustment.
Can I use sequential testing for any metric? Sequential methods are most straightforward for a single primary metric. For multiple metrics, you need to account for multiplicity, which can be complex. It is best to designate one primary metric for the stopping rule and treat others as secondary or exploratory.
How do I choose between O'Brien-Fleming and Pocock boundaries? O'Brien-Fleming is conservative early and more lenient later, making it harder to stop early but preserving power for small effects. Pocock spends alpha more evenly, allowing earlier stopping but requiring a larger maximum sample size. The choice depends on your priorities: if you want to stop early for large effects, O'Brien-Fleming is often preferred; if you want more opportunities to stop, Pocock may be suitable.
Does sequential testing require special software? Yes, implementing the stopping boundaries and adjusted p-values correctly is complex. Many statistical packages and online tools support sequential designs. It is essential to use validated software and to pre-register the design.
What if I need to change the design mid-experiment? Changing the design after seeing data invalidates the error control. If you must change, you should treat the experiment as a new test with a new alpha, or use methods that account for adaptive changes, but these are advanced and should be planned in advance. Always consult a statistician if you need to deviate from the plan.