1 Overview
In the world of A/B testing, two broad families of methods dominate: fixed-horizon testing and sequential testing. A fixed-horizon test requires you to decide the sample size in advance, run the experiment until that sample size is reached, and then analyze the results once. A sequential test, by contrast, allows you to monitor the data as it accumulates and make a decision at any point, while still controlling the overall error rates. The question of which wins is not about a single winner in all situations; it depends on your goals, your tolerance for error, and the nature of your experiment.
This article examines the trade-offs through the lens of case studies. We will not present invented numbers or specific company examples. Instead, we will walk through the logic of how each approach behaves under different conditions. The key insight is that the choice between fixed-horizon and sequential testing is a design decision that affects the validity and efficiency of your results. Understanding these trade-offs helps you avoid common pitfalls, such as inflating false positives by peeking at fixed-horizon tests, or using sequential methods without proper boundary calibration.
Ultimately, the better choice depends on your setup. If you can reliably predict your sample size and effect size, a fixed-horizon test is simple and powerful. If you need to make decisions quickly or cannot wait for a fixed sample, a sequential test offers flexibility with valid error control. The rest of this article explores how to frame the question, design the comparison, and interpret the results.
2 Framing the question
Before comparing fixed-horizon and sequential testing, you must frame the question precisely. What decision are you trying to make? Are you testing whether a new feature increases conversion rate, reduces churn, or improves engagement? The hypothesis should be stated in terms of a metric and a direction. For example, you might hypothesize that a new checkout flow increases the completion rate compared to the current flow. The metric is the completion rate, and the test design will determine how you collect and analyze data to evaluate that hypothesis.
The framing also involves specifying the null and alternative hypotheses. In a fixed-horizon test, you typically set a significance level, such as 0.05, and a power, such as 0.80, to detect a minimum effect of interest. In a sequential test, you also set error rates, but the boundaries for stopping are adjusted to account for repeated looks at the data. The question then becomes: under what conditions does one approach yield a more reliable or efficient decision?
Another aspect of framing is the cost of errors. A false positive might lead you to adopt a harmful change, while a false negative might cause you to miss a beneficial one. The relative costs influence which error rate you prioritize and how you design the test. Additionally, the framing should consider the practical constraints: how long can you run the experiment, how much traffic do you have, and how quickly do you need an answer? These constraints often push teams toward sequential methods, but they also introduce risks if not handled carefully.
3 Designing the test
Designing a comparison between fixed-horizon and sequential testing requires careful planning. You need to define the data-generating process, the metric, and the analysis methods. In a simulation study, you might generate data from a known distribution with a specified effect size, then apply both a fixed-horizon test and a sequential test to the same data stream. The fixed-horizon test would analyze the data only at the predetermined sample size, while the sequential test would evaluate after each observation or batch, using boundaries that control the overall error rate.
For a real experiment, you cannot run both methods on the same data and get independent results, because the decision to stop affects the data collected. Instead, you can compare them conceptually or use historical data to simulate how each would have performed. The design should specify the primary metric, the randomization unit, and the conditions for stopping. In a fixed-horizon design, stopping is determined solely by reaching the target sample size. In a sequential design, stopping occurs when the test statistic crosses a boundary, which could happen early if the effect is large, or later if the effect is small.
The design also includes the choice of sequential boundaries. Common approaches include Pocock boundaries, which use a constant critical value at each look, and O'Brien-Fleming boundaries, which are conservative early and become less stringent as the sample size grows. The choice affects the expected sample size and the power. A well-designed sequential test will have a maximum sample size and an expected sample size under the alternative hypothesis. The fixed-horizon test has a fixed sample size, which is its expected sample size. The comparison should account for these differences in sample size and error rates.
4 Sample size and validity
Sample size is a critical factor in both approaches. For a fixed-horizon test, you calculate the required sample size based on the desired significance level, power, and minimum detectable effect. This calculation assumes that you will analyze the data only once, at the end. If you peek at the data before reaching the target and stop early based on the results, you inflate the false positive rate. This is a major threat to validity. The fixed-horizon test is valid only if you adhere strictly to the planned sample size and do not make decisions based on interim looks.
Sequential tests are designed to handle interim looks while maintaining the nominal error rates. However, their validity depends on correctly specifying the boundaries and the underlying assumptions. If the data are not independent and identically distributed, or if the metric is not well-behaved, the sequential boundaries may not control error rates as intended. Additionally, the expected sample size of a sequential test depends on the true effect size. If the effect is small, the sequential test may require more samples than a fixed-horizon test to reach a decision, because it needs to accumulate enough evidence to cross the boundary. If the effect is large, it may stop much earlier.
Validity is also threatened by optional stopping based on intuition rather than formal boundaries. Even with a sequential design, if you decide to stop when the results look favorable, you reintroduce bias. The boundaries must be pre-specified and followed. Another threat is unplanned changes to the metric or population during the experiment. Such changes can invalidate the assumptions and make the results unreliable. Finally, low power or small sample sizes can lead to unstable results in both approaches. With small samples, the estimates are noisy, and the chance of a false positive or false negative is higher. Therefore, sample size considerations must be grounded in the specific setup and the variability of the metric.
5 Running and reading the results
When running a fixed-horizon test, you collect data until the predetermined sample size is reached, then analyze the results. You compute a test statistic, such as a t-test or a z-test for proportions, and compare it to a critical value. If the p-value is below the significance level, you reject the null hypothesis and conclude there is a statistically significant effect. The result is read as a binary decision: significant or not. The effect size and confidence interval provide additional context. However, the validity of this reading depends on having not peeked at the data. If you did peek and stopped early, the p-value is not valid, and the false positive rate is higher than nominal.
In a sequential test, you monitor the data as it accumulates and compute the test statistic at each look. You compare it to the pre-specified boundaries. If the statistic crosses an upper boundary, you stop and declare a significant positive effect. If it crosses a lower boundary, you stop and declare no significant effect or a significant negative effect, depending on the design. If it does not cross any boundary, you continue collecting data until you reach the maximum sample size, at which point you make a final decision. The result is read with the understanding that the error rates are controlled across all looks. The sequential test may stop early, saving sample size, but the decision is still subject to the same error rates.
Reading the results also involves interpreting the practical significance. A statistically significant result may not be practically meaningful if the effect size is tiny. Both approaches should report effect sizes and confidence intervals. In a sequential test, the confidence interval after stopping may be adjusted for the sequential design. It is important to avoid overinterpreting early stopping as definitive proof; the evidence is still probabilistic. The key is to follow the pre-specified analysis plan and not to make post hoc changes.
6 What to change next
After comparing fixed-horizon and sequential testing, you may decide to change your experimentation process. If you frequently find yourself needing to peek at results, a sequential test is likely a better fit because it formally allows monitoring without inflating error rates. However, you must invest in setting up the boundaries and ensuring the team understands that stopping rules are not optional. If your experiments are short and you can reliably wait for a fixed sample size, a fixed-horizon test remains simple and powerful. The choice should be based on the trade-off between flexibility and simplicity.
Another change is to improve sample size planning. Many experiments are underpowered, leading to inconclusive results. By conducting power analyses and using sequential designs with maximum sample sizes, you can ensure that you have enough data to detect meaningful effects. Additionally, you should consider the cost of errors. If false positives are very costly, you might use more stringent significance levels or conservative sequential boundaries. If false negatives are costly, you might increase power.
Finally, you should establish clear protocols for experiment analysis. This includes pre-registering the hypothesis, metric, sample size, and analysis method. Whether you choose fixed-horizon or sequential, the validity of your results depends on following the plan. Training the team on the dangers of peeking and the correct use of sequential boundaries is essential. By making these changes, you can improve the reliability and efficiency of your A/B testing program.
7 Frequently asked questions
What is the main difference between fixed-horizon and sequential testing? Fixed-horizon testing requires a predetermined sample size and a single analysis at the end, while sequential testing allows for multiple analyses during data collection with adjusted boundaries to control error rates. The main difference is in how they handle interim looks and stopping rules.
Can I use a fixed-horizon test and just peek at the data without stopping? Peeking at a fixed-horizon test without adjusting the significance level inflates the false positive rate. If you look at the data multiple times and make decisions based on those looks, the test is no longer valid. You should either avoid peeking or use a sequential design.
Which method requires a larger sample size? It depends on the effect size and the design parameters. A fixed-horizon test has a fixed sample size. A sequential test has a maximum sample size and an expected sample size that can be smaller or larger than the fixed-horizon sample size, depending on the true effect. For large effects, sequential tests often stop earlier; for small effects, they may require more samples.
Is sequential testing always better? No. Sequential testing offers flexibility but requires more complex planning and adherence to boundaries. Fixed-horizon testing is simpler and may be more efficient when you can accurately predict the sample size and effect. The better choice depends on your specific context and constraints.
How do I choose between Pocock and O'Brien-Fleming boundaries? Pocock boundaries use a constant critical value and are better for detecting large effects early, but they require a larger maximum sample size. O'Brien-Fleming boundaries are conservative early and less stringent later, making them suitable when you want to preserve power for the final analysis. The choice depends on your priorities.
What threatens the validity of these tests? Peeking without adjustment, violating assumptions, optional stopping based on intuition, unplanned changes, and low power all threaten validity. Always follow the pre-specified plan and ensure your sample size is adequate.