1 Overview
A/B testing rests on a simple promise: if you run an experiment to a predetermined sample size and analyze it once, a statistically significant result at the five percent level means there is only a five percent chance of seeing such an extreme result if the null hypothesis were true. That promise is conditional on the analysis plan being fixed before data collection begins. In practice, many teams watch dashboards daily, refresh p-values, and stop the test the moment a metric crosses a significance threshold. This behavior is called peeking, and it quietly breaks the guarantee that makes the p-value meaningful.
The problem is not that looking at data is inherently wrong. The problem is that repeatedly testing the same accumulating data and acting on the first significant result changes the probability of a false positive. Each unadjusted look is another chance to cross the threshold by random fluctuation. The more often you look, the more likely you are to see a spurious winner even when the true effect is zero. Sequential testing methods were developed to allow monitoring while preserving error control, but they require a different design and a different interpretation than fixed-horizon tests.
This article explains why peeking inflates false positives, how sequential methods work at a conceptual level, and what practical choices keep an experiment honest. The focus is on the logic of the problem, not on specific numerical thresholds, because the exact impact depends on the number of looks, the timing, the sample size, and the underlying effect.
2 Framing the question
The core question is whether the way you monitor an experiment changes the validity of its conclusions. A fixed-horizon test assumes one analysis at a sample size chosen in advance. A peeking procedure assumes many analyses, each treated as if it were the only one. These are different statistical procedures, and they have different error rates. The hypothesis in this article is that repeated unadjusted significance testing does not preserve the nominal false-positive rate, while sequential methods can control it under continuous monitoring.
The metric of interest is the false-positive rate, also called the Type I error rate. This is the probability of declaring an effect when none exists. In a fixed-horizon test, this rate is set by the significance level, commonly five percent. Under naive peeking, the same nominal level no longer describes the actual error rate. The actual rate can be substantially higher, and it grows with the number of interim looks and with how early those looks occur. The exact inflation depends on the design, so no single number applies universally.
A related question is what happens to power and to effect estimates. Peeking does not just increase false positives; it can also bias the estimated effect size upward when a test stops early on a lucky high reading. Sequential methods address this by adjusting the decision boundaries and, in some cases, the estimates. The framing matters because the goal is not to avoid looking at data, but to look in a way that keeps the error guarantees intact.
3 Designing the test
The design of an experiment determines whether monitoring is safe. In a fixed-horizon design, you specify the sample size, the primary metric, the significance level, and the analysis plan before the test starts. You then wait until the planned sample is reached and analyze once. This is the simplest design and the one that supports the standard interpretation of a p-value. It does not forbid looking at data for operational reasons, such as checking for bugs or imbalances, but it does forbid making a stop or ship decision based on an unadjusted significance threshold before the planned end.
A sequential design explicitly allows interim analyses. There are several families of methods. Group sequential designs divide the planned sample into stages and use adjusted boundaries at each stage, with the boundaries derived from an alpha spending function. Alpha spending allocates the total Type I error across the looks, spending little at early looks and more later. Always-valid inference methods provide error control at any time, which is useful when the number of looks is not fixed in advance. Each approach has trade-offs in power, maximum sample size, and complexity.
The key design principle is prespecification. The number and timing of interim analyses, the stopping rule, and the method for adjusting boundaries must be chosen before data collection. If you decide to peek and then choose a sequential method after seeing a promising trend, the error control is no longer valid. The design also needs to account for practical constraints: how long the test can run, how much traffic is available, and whether the team can commit to the planned maximum sample size even if early results look disappointing.
4 Sample size and validity
Sample size is central to both fixed-horizon and sequential testing. In a fixed-horizon test, the required sample size depends on the baseline rate, the minimum detectable effect, the desired power, and the significance level. If the sample is too small, the test will often fail to detect a real effect. If the sample is larger than needed, the test may be unnecessarily costly. The sample size must be determined in advance because it defines the point at which the analysis will occur. Results depend heavily on this choice, and changing it after seeing data undermines validity.
In a sequential design, sample size planning is more complex. You must specify the maximum sample size and the interim analysis schedule. Early looks can allow you to stop sooner if the effect is large, but they also require more conservative boundaries, which can increase the maximum sample size needed to maintain power. The exact trade-off depends on the alpha spending function, the number of looks, and the effect size. There is no universal rule that says sequential is always better or always worse; it depends on the setup and the goals of the experiment.
Validity is threatened by anything that changes the analysis plan after data collection begins. Unplanned peeking, optional stopping, multiple comparisons across metrics or segments, and post hoc changes to the stopping rule all inflate the Type I error rate. Even a well-intentioned check can become a validity problem if the decision to stop is influenced by the observed result. The safest approach is to prespecify the design, document the stopping rule, and treat any deviation as a new analysis with its own error considerations.
5 Running and reading the results
When you run a fixed-horizon test, reading the result is straightforward if the design was followed. You compute the test statistic at the planned sample size and compare it to the critical value. A significant result means the data are unlikely under the null hypothesis, assuming the design and assumptions hold. A non-significant result means you did not detect an effect of the specified size with the available sample. In both cases, the interpretation depends on the prespecified metric, the sample size, and the assumptions of the test.
When you run a sequential test, reading the result requires attention to the stopping rule. If the test stops early for efficacy, the effect estimate may be biased upward, and confidence intervals may need adjustment. If the test continues to the maximum sample size without crossing a boundary, you conclude that the effect, if any, is smaller than the design could reliably detect. The p-value or the adjusted confidence interval must be interpreted according to the sequential method used. A naive p-value computed at an interim look is not valid for a sequential design.
The most common mistake is to run a fixed-horizon test but read it as if peeking were harmless. If you checked the dashboard daily and stopped when the p-value first dropped below the threshold, the reported significance level does not reflect the actual error rate. The result may still be real, but the evidence is weaker than it appears. The honest reading is that the test was not a fixed-horizon test; it was an unplanned sequential procedure with unknown error properties. The remedy is to either rerun the test with a proper design or to use a sequential method that accounts for the monitoring.
6 What to change next
The first change is to decide in advance how you will monitor the experiment. If you cannot commit to a single analysis at a fixed sample size, choose a sequential design. This means selecting a method, specifying the interim analysis schedule, and documenting the stopping rule before the test starts. It also means training the team to respect the rule, because the temptation to peek is strongest when results look promising or disappointing. A prespecified sequential design turns peeking from a validity threat into a planned feature.
The second change is to separate operational monitoring from decision-making. It is fine to watch for bugs, traffic imbalances, or guardrail metric failures. It is not fine to use an unadjusted significance threshold from an operational dashboard as the basis for stopping the test. If you need early signals, use a sequential method that provides valid error control, or treat the early look as exploratory and plan a confirmatory test. This distinction keeps the decision rule clean and the error guarantees intact.
The third change is to improve sample size planning and power analysis. Many peeking problems arise because the test is underpowered, so the team keeps looking for a significant result that the sample cannot reliably produce. A realistic power analysis, based on a meaningful minimum detectable effect and the available traffic, reduces the pressure to peek. It also helps set expectations: if the required sample is large, the test will take time, and that is a design fact, not a failure. Finally, document the analysis plan, the stopping rule, and any deviations. Transparency about what was actually done is the foundation for trusting the result.
7 Frequently asked questions
Is it ever okay to peek at an A/B test? It is okay to look at data for operational reasons, such as checking that the experiment is running correctly. It is not okay to make a stop or ship decision based on an unadjusted significance threshold before the planned analysis. If you want to make decisions at interim looks, use a sequential design that adjusts for multiple analyses. The distinction is between monitoring and decision-making, and it matters because the error guarantees apply only to the prespecified procedure.
Does peeking always invalidate the result? Not necessarily. A significant result after peeking may still reflect a real effect, but the reported significance level overstates the evidence. The actual false-positive rate is higher than the nominal level, and the effect estimate may be biased. The result is not automatically wrong, but it is weaker than a fixed-horizon result with the same nominal p-value. The safest interpretation is that the test was not a fixed-horizon test, so the usual guarantees do not apply.
How do sequential methods fix the problem? They adjust the decision boundaries to account for the number and timing of interim analyses. Alpha spending functions allocate the total Type I error across looks, and always-valid methods provide error control at any time. These methods preserve the false-positive rate at the desired level if the design is prespecified and followed. They may require a larger maximum sample size or more complex analysis, but they allow monitoring without sacrificing validity. The exact performance depends on the setup, the effect size, and the sample size.
What about Bayesian methods? Bayesian approaches can provide sequential inference without the same frequentist error guarantees, but they require careful prior specification and interpretation. They are not a free pass for unplanned peeking; the operating characteristics depend on the model and the prior. The choice between frequentist sequential methods and Bayesian methods should be based on the question, the team's expertise, and the decision framework. In all cases, prespecification and transparency are essential.