1 Overview
You have probably heard the warning: do not peek at your A/B test results before the test is scheduled to end. It sounds like a minor procedural quirk, but it is one of the most common ways that well-meaning experimenters fool themselves. The problem is not that early data is meaningless; it is that repeatedly checking results and deciding to stop when they look good dramatically increases the chance of declaring a winner when no real difference exists. This article explains why peeking is dangerous, how it distorts statistical inference, and what you can do instead.
The core issue is that statistical tests are designed for a fixed sample size and a single look at the data. When you peek, you give yourself multiple opportunities to see a false positive. Each peek is like buying another lottery ticket in a game where the odds of a spurious win accumulate. The more often you check, the more likely you are to cross the significance threshold by chance alone, even if the treatment has no effect. This is not a matter of discipline or intuition; it is a mathematical property of repeated hypothesis testing.
Understanding this phenomenon is essential for anyone running experiments, from product managers to data scientists. The good news is that the problem is well understood, and there are principled ways to monitor tests without inflating error rates. But first, you need to see why the naive approach fails.
2 Framing the question
Before you can interpret any test result, you must be clear about what you are testing. A well-framed experiment starts with a hypothesis that specifies the population, the intervention, and the outcome. For example, you might hypothesize that changing the color of a call-to-action button from blue to green will increase the click-through rate among new visitors. The metric is the click-through rate, and the design is a randomized controlled trial where each visitor is assigned to either the blue or green button. Without this clarity, peeking becomes even more tempting because you lack a predefined stopping rule.
The hypothesis should be falsifiable and stated in advance. It is not enough to say 'we think green is better.' You need to specify the direction of the effect and the minimum detectable effect that would be practically meaningful. This framing forces you to think about sample size and duration before you launch. It also protects you from the temptation to reinterpret results after the fact, a practice known as HARKing (hypothesizing after results are known).
A common mistake is to frame the question too broadly. If you ask 'does this change improve the user experience?' you will struggle to define a metric and a stopping point. Instead, ask a precise question: 'Does this change increase the proportion of users who complete checkout within seven days?' Precision makes it harder to fool yourself later.
3 Designing the test
A proper A/B test design includes randomization, a control group, and a pre-registered analysis plan. Randomization ensures that the groups are comparable on average, so any difference in the metric can be attributed to the treatment. The control group provides a baseline. The analysis plan specifies the primary metric, the statistical test, the significance level, and the sample size. Crucially, it also specifies when the test will stop. Without a stopping rule, you are likely to stop when the results look favorable, which is exactly the peeking problem.
There are two main approaches to designing a test that allows for monitoring without inflating false positives. The first is a fixed-horizon test: you decide in advance how many users or how many days you will run the test, and you do not look at the results until that point. This is the simplest and most robust approach. The second is a sequential test, which uses methods like alpha spending or group sequential boundaries to allow for a limited number of interim looks while controlling the overall Type I error rate. These methods require careful planning and are more complex to implement.
Regardless of the approach, you must also consider practical constraints. How long will it take to reach the required sample size? Will there be seasonality or other time-based effects? Should you run the test for full weeks to avoid day-of-week bias? These design choices affect validity and should be made before the test begins. Documenting them in advance is your best defense against the temptation to peek and stop early.
4 Sample size and validity
Sample size is the linchpin of a valid experiment. If your sample is too small, you may fail to detect a real effect (a Type II error) or you may detect a spurious effect (a Type I error). The required sample size depends on the baseline metric, the minimum detectable effect you care about, the desired significance level (often 0.05), and the desired power (often 0.80). These are not arbitrary numbers; they represent trade-offs between the risk of false positives and false negatives. You must calculate the sample size before the test, not after.
Peeking undermines validity because it changes the effective significance level. If you check results every day and stop when p < 0.05, your actual false positive rate can be much higher than 5%. For instance, with just a few peeks, the error rate can double or triple. This is not a theoretical concern; simulations and mathematical derivations show that optional stopping is a major source of false discoveries in industry. The problem is exacerbated when you also test multiple metrics or multiple variants, because each additional comparison increases the chance of a false positive.
Validity also depends on other factors: randomization must be truly random, the groups must be independent, and the metric must be measured consistently. If users can switch between variants, or if the assignment is not random, the test is confounded. Peeking is just one threat among many, but it is particularly insidious because it feels like diligence. In reality, it is a form of data dredging that invalidates the statistical guarantees you rely on.
5 Running and reading the results
Once the test is running, resist the urge to check the dashboard. If you must monitor for technical issues, look at operational metrics like sample ratio mismatch or error rates, not the primary outcome. When the pre-planned sample size is reached, analyze the results according to your analysis plan. Compute the effect size, the confidence interval, and the p-value. Remember that statistical significance does not imply practical significance. A small effect can be statistically significant with a large sample, but it may not be worth implementing.
If the result is not statistically significant, do not conclude that the treatment has no effect. The confidence interval may include both meaningful positive and negative effects. You can report the range of plausible effects and decide whether further testing is warranted. If the result is statistically significant, check for validity threats: was the sample size as planned? Were there any bugs or data quality issues? Did the test run for the intended duration? Only then can you trust the result.
It is also important to consider the possibility of a false positive. Even with a perfect design, 5% of tests will show a significant effect when there is none. If you run many tests, some will be false positives. This is why replication is valuable. A single significant result is not proof; it is evidence that should be weighed alongside other findings. The key is to avoid the peeking trap so that the evidence you gather is as reliable as possible.
6 What to change next
If you have been peeking, the first change is to adopt a pre-registered stopping rule. Decide in advance how long the test will run and how many users you need. Write it down and share it with your team. This simple act removes the discretion that leads to false positives. If you need to monitor the test, use a sequential design with proper alpha spending. Tools and libraries exist to help you implement these methods correctly, but they require statistical expertise.
The second change is to focus on effect sizes and confidence intervals, not just p-values. A p-value tells you whether an effect is likely to be nonzero, but it does not tell you how large or important the effect is. By reporting the confidence interval, you give stakeholders a sense of the uncertainty and the range of possible outcomes. This encourages more thoughtful decisions and reduces the temptation to overinterpret marginal results.
Finally, cultivate a culture that values valid experimentation over fast answers. Peeking often happens because of pressure to show results quickly. But a false positive can lead to bad product decisions and wasted resources. By educating your team about the risks and providing the tools to do it right, you can build trust in your experimentation process. Remember, the goal is not to get a significant result; it is to learn the truth about your users.
7 Frequently asked questions
Is it ever okay to peek at an A/B test? In general, no. Peeking at the primary metric and making decisions based on it inflates the false positive rate. However, you can monitor for data quality issues or sample ratio mismatch without looking at the outcome. If you must look at the outcome, use a sequential testing framework that controls the overall error rate.
What if I peek and see a significant result? A single significant result after peeking is not trustworthy. The p-value is not valid because you have given yourself multiple chances to stop. You should either continue the test to the pre-planned sample size or treat the result as exploratory and seek replication. The safest approach is to ignore any interim significant result and wait for the final analysis.
How many peeks are too many? Even one peek can inflate the error rate, especially if you would stop early based on the result. The more peeks, the worse it gets. There is no safe number of peeks without a proper sequential design. The best practice is to not peek at all until the test is complete.
Does peeking matter if the effect is huge? A huge effect is more likely to be real, but peeking still biases the estimate and the p-value. You might be seeing an exaggeration of the true effect. It is still best to wait for the planned sample size, but if the effect is enormous and consistent, it may be worth investigating further with a proper test.
Can I use Bayesian methods to avoid peeking problems? Bayesian methods can be more forgiving of optional stopping if you use a proper prior and decision rule, but they are not a free pass. You still need to pre-specify your model and decision criteria. Without that, you can fool yourself just as easily. Bayesian methods offer a different framework, but they do not eliminate the need for careful design.