1 Overview
A/B testing is a cornerstone of data-driven optimization, but many practitioners unknowingly undermine their own experiments by repeatedly checking results before the planned sample size is reached. This behavior, known as peeking, dramatically increases the chance of false positives. When you peek, you are essentially performing multiple hypothesis tests on accumulating data, and each test carries its own error rate. Without correction, the overall false positive rate can balloon far beyond the nominal 5 percent. The result is that seemingly significant wins often vanish when the experiment is repeated, leading to poor decisions and eroded trust in experimentation.
Sequential testing offers a principled solution. Instead of fixing the sample size in advance and analyzing only at the end, sequential methods allow valid interim analyses at pre-specified points or even continuously. They adjust the significance thresholds to control the overall Type I error rate. This means you can monitor results as they come in without inflating false positives. However, sequential testing is not a free lunch: it requires careful planning, and the choice of method affects power and the interpretation of results. In this article, we will explore how to frame questions, design tests, determine sample sizes, and read results when using sequential approaches. We will also discuss common pitfalls and what to do next.
2 Framing the question
Before diving into sequential testing, you must clearly define what you are testing. A well-framed question includes a specific hypothesis, a primary metric, and a decision rule. For example, you might hypothesize that adding a testimonial section to a landing page will increase the conversion rate. The primary metric is conversion rate, defined as the proportion of visitors who complete a desired action. You should also specify the minimum detectable effect (MDE) that would be practically meaningful. This MDE, along with baseline conversion rate, will drive sample size calculations.
It is equally important to consider what threats to validity might arise. Will the test run long enough to capture weekly seasonality? Could there be novelty effects where early results are unusually positive? Are there external events that could confound the results? By addressing these questions upfront, you set the stage for a valid experiment. Moreover, you need to decide whether sequential testing is appropriate. If you expect to monitor results frequently and want the flexibility to stop early for efficacy or futility, sequential methods are valuable. But if you can commit to a fixed sample size and a single analysis, traditional methods may suffice. The key is to align your analysis plan with your monitoring behavior.
3 Designing the test
Designing an A/B test with sequential monitoring requires choosing a sequential testing framework. Common approaches include group sequential designs with alpha spending functions, and always-valid inference methods such as mixture sequential probability ratio test (mSPRT) or Bayesian sequential testing. Each has trade-offs. Group sequential designs require you to pre-specify the number and timing of interim analyses, and they adjust the significance threshold at each look using an alpha spending function. This controls the overall Type I error rate but can be less flexible if you want to peek at arbitrary times. Always-valid methods allow continuous monitoring without inflating error rates, but they may require larger sample sizes or have lower power for a given effect size.
Regardless of the method, randomization must be proper: users should be assigned to control or treatment randomly and independently. You should also ensure that the assignment is stable across sessions, often by using a consistent hash of user ID. The test duration should cover full business cycles to avoid day-of-week effects. Additionally, you must decide on the unit of analysis (e.g., user, session) and ensure that metrics are computed consistently. Finally, pre-register your analysis plan, including the sequential method, the primary metric, and the stopping rules. This prevents post-hoc rationalization and p-hacking.
4 Sample size and validity
Sample size calculation for sequential tests is more complex than for fixed-horizon tests. You need to account for the sequential design's alpha spending and the maximum number of looks. Typically, you start by calculating the sample size for a fixed-horizon test using the baseline rate, MDE, significance level (alpha), and power (1 - beta). Then, you inflate this sample size to account for the sequential monitoring. The inflation factor depends on the number and timing of interim analyses and the alpha spending function. For example, a design with many interim analyses may require a larger maximum sample size to maintain the same power. Always use software or validated formulas to compute the required sample size; do not rely on rules of thumb.
Validity in sequential testing hinges on controlling the Type I error rate. If you peek without adjustment, the false positive rate increases. Sequential methods correct for this, but they assume that the test statistic behaves as expected under the null hypothesis. Violations can occur if there is sample ratio mismatch (SRM), where the actual allocation ratio deviates from the planned ratio. SRM can indicate a bug in the randomization or data collection, and it can invalidate results. Other threats include multiple comparisons (testing many metrics without correction), novelty effects, and interference between units. To protect validity, monitor for SRM, pre-specify a single primary metric, and run the test long enough to let novelty effects dissipate. Also, consider using sequential methods that are robust to model misspecification, such as those based on always-valid confidence intervals.
5 Running and reading the results
Once the test is running, you should follow the pre-specified monitoring plan. If you are using a group sequential design, you will analyze at the planned interim points. At each look, compute the test statistic and compare it to the adjusted significance threshold. If the statistic crosses the efficacy boundary, you can stop the test early and declare a winner. If it crosses the futility boundary, you can stop for lack of effect. If neither, continue to the next look. With always-valid methods, you can monitor continuously, but you should still avoid making decisions based on random fluctuations; instead, look for consistent evidence that the effect is real.
When interpreting results, focus on the effect size and its confidence interval, not just the p-value. A statistically significant result may not be practically significant. For example, a small lift in conversion rate might not justify the implementation cost. Also, consider the direction and magnitude of the effect in the context of the business. If the test is stopped early for efficacy, the effect size estimate may be biased upward; sequential methods provide adjusted estimates, but they can still be noisy. Finally, remember that a single test is rarely definitive. Replication and robustness checks are valuable. If the result is surprising, investigate potential causes such as SRM, data pipeline issues, or external events.
6 What to change next
After an experiment concludes, the next steps depend on the outcome. If you have a clear winner, consider implementing the change and monitoring its impact in production. However, be cautious: the effect observed in the test may not fully generalize to all users or contexts. You might want to run a follow-up test to confirm the result or to test variations. If the test is inconclusive, you could increase the sample size, refine the hypothesis, or test a different intervention. If the test shows a negative effect, it is still valuable information; you can avoid rolling out a harmful change and learn about user preferences.
From a process perspective, reflect on the experiment itself. Did the sequential design work well? Were there any issues with SRM or data quality? Did you adhere to the pre-registered plan? Use these insights to improve future tests. For instance, if you found that continuous monitoring led to anxiety and premature decisions, you might switch to a group sequential design with fewer interim looks. If you struggled with sample size calculation, invest in training or tools. Remember that experimentation is a learning process. The goal is not just to find wins but to build a reliable system for making data-driven decisions. Sequential testing is a powerful tool, but it must be used correctly to avoid the very lies that peeking can create.
7 Frequently asked questions
What is peeking and why is it bad? Peeking means repeatedly checking test results before the planned sample size is reached. It is problematic because each peek is a hypothesis test, and multiple tests inflate the false positive rate. Without adjustment, you are more likely to see a false winner. Sequential testing adjusts for this by controlling the overall error rate.
Can I use sequential testing for any metric? Sequential methods can be applied to many metrics, but they work best for metrics that are approximately normally distributed or can be transformed. For proportions, you can use methods like the sequential probability ratio test or group sequential designs with appropriate adjustments. Always check assumptions.
How do I choose between group sequential and always-valid methods? Group sequential designs require pre-specified interim analysis times and are efficient if you can stick to that schedule. Always-valid methods allow continuous monitoring but may require larger sample sizes. Choose based on your monitoring needs and resources.
Does sequential testing guarantee valid results? No method can guarantee validity if there are flaws in randomization, data collection, or analysis. Sequential testing addresses the peeking problem but does not fix SRM, novelty effects, or other biases. You must still follow best practices for experimental design.
How long should I run a sequential test? The duration depends on the sample size required and the expected traffic. You should run until you reach the maximum sample size or until a stopping boundary is crossed. Do not stop early based on intuition; follow the pre-specified rules.