1 Overview
Imagine you launch an A/B test and check the results every hour. On day one, version B is ahead by a wide margin. You feel a rush of excitement and consider calling it a win. But by day three, the difference has shrunk, and by day seven, it is gone. What happened? You fell victim to peeking, the practice of repeatedly analyzing data before the planned sample size is reached. Peeking is one of the most common and damaging mistakes in experimentation, yet it often goes unnoticed because the early results can look so convincing.
The core problem is that statistical tests are designed to control error rates only when the analysis is performed once, at a predetermined sample size. When you check results multiple times, you give random fluctuations many chances to cross the threshold for significance. This inflates the false positive rate, meaning you are more likely to declare a winner when there is no real difference. The effect is not subtle; even a few peeks can double or triple the nominal error rate. The exact inflation depends on how often you peek, how many metrics you track, and the total sample size, but the direction is always the same: more peeking, more false positives.
This article explains why peeking sabotages your results and how to avoid it. We will frame the question, design a proper test, discuss sample size and validity, show how to read results correctly, and suggest what to change next. Throughout, remember that results depend heavily on your specific setup and sample size. There is no universal fix, but understanding the mechanics will help you make better decisions.
2 Framing the question
Before running any experiment, you must frame a clear question. A good question specifies the population, the intervention, the comparison, and the outcome. For example: 'Among new visitors to our website, does a simplified checkout flow increase the proportion who complete a purchase compared to the current flow?' This question defines the metric (purchase completion rate), the groups (new visitors assigned to simplified versus current flow), and the expected direction (increase). Without such clarity, you risk chasing noise or measuring the wrong thing.
The framing also determines your hypothesis. A hypothesis is a testable statement about the relationship between the intervention and the outcome. In the checkout example, the null hypothesis is that there is no difference in purchase completion rates between the two flows. The alternative hypothesis is that there is a difference. You then choose a statistical test that will help you decide whether the observed data are consistent with the null hypothesis. Crucially, the hypothesis must be specified before you look at the data. If you change the hypothesis after seeing results, you are no longer doing confirmatory testing; you are doing exploratory analysis, which has different rules and higher risks of false discovery.
Finally, framing includes deciding what you will do with the result. Will you roll out the change if it wins? Will you iterate if it loses? Will you run a follow-up test? These decisions should be made in advance to avoid post-hoc rationalization. Peeking often arises from a desire to make decisions quickly, but without a clear framing, quick decisions can be costly. The question you ask shapes everything that follows, so invest time in getting it right.
3 Designing the test
A well-designed test has several key components: a control group, a treatment group, random assignment, a single primary metric, and a fixed sample size. Random assignment ensures that the groups are comparable on average, so any difference in the metric can be attributed to the treatment. The primary metric should be chosen before the test starts and should be directly tied to the hypothesis. Secondary metrics can be tracked, but they should not drive the decision unless you adjust for multiple comparisons.
The design also includes the analysis plan. You must specify when and how you will analyze the data. The standard approach is a fixed-horizon test: you collect data until you reach the predetermined sample size, then you analyze once. This approach controls the false positive rate at the nominal level (often 5%). If you plan to analyze multiple times, you need a sequential design that adjusts the significance threshold at each look. Such designs exist, but they require careful planning and often increase the required sample size.
Another critical design choice is the unit of randomization. It could be users, sessions, or pages, depending on the context. The unit affects the independence of observations and the appropriate statistical test. For example, if you randomize users but measure clicks, you may have multiple observations per user, which violates independence and requires methods like cluster-robust standard errors. Ignoring these details can lead to underestimated variance and inflated false positives, even without peeking. Therefore, design is not just about what you test but also about how you assign, measure, and analyze. A thoughtful design is your first line of defense against peeking and other validity threats.
4 Sample size and validity
Sample size is the backbone of a trustworthy experiment. Too small a sample and you cannot detect a meaningful effect; too large a sample and you may waste resources or detect trivial effects. More importantly, the planned sample size determines the precision of your estimate and the power of your test. Power is the probability of detecting a true effect of a given size. Conventionally, researchers aim for 80% power, but the appropriate level depends on the costs of false positives and false negatives. To calculate sample size, you need to specify the baseline rate, the minimum detectable effect, the significance level, and the desired power. These inputs should be based on prior data or domain knowledge, not guesswork.
Peeking directly threatens validity by breaking the link between the planned sample size and the error rates. When you analyze repeatedly, the effective significance level is no longer the one you set. The false positive rate climbs, sometimes dramatically. For instance, if you check a test 10 times at equally spaced intervals, the actual false positive rate can be several times higher than 5%. The exact inflation depends on the number of looks and the correlation between test statistics, but the principle holds: optional stopping invalidates the test.
Other threats to validity include multiple comparisons, unblinded analysts, and selective reporting. If you test many metrics and pick the one that shows a significant difference, you are likely to find a false positive. If the analyst knows which group is which, bias can creep in. If you report only the significant results, you distort the literature. All these threats are amplified by peeking because peeking encourages selective attention to early signals. To protect validity, pre-register your analysis plan, stick to the planned sample size, and use sequential methods if you must monitor. Remember that results depend on your setup and sample size; there is no one-size-fits-all answer.
5 Running and reading the results
Once the test is running, resist the urge to check results constantly. If you must monitor for technical issues, look at data quality metrics like sample ratio mismatch or bot traffic, not the primary outcome. For the primary outcome, wait until the planned sample size is reached. When you finally analyze, compute the test statistic and the p-value according to your pre-specified plan. A p-value is the probability of observing data as extreme as yours, assuming the null hypothesis is true. It is not the probability that the null is true, nor the probability that your result is due to chance. A common threshold is 0.05, but you should interpret it in context. A p-value just below 0.05 provides weak evidence; a very small p-value provides stronger evidence, but effect size and confidence intervals matter more.
When reading results, consider the confidence interval for the effect. It gives a range of plausible values for the true effect. If the interval includes zero, the data are consistent with no effect. If it excludes zero, you have evidence of an effect, but the width of the interval tells you about precision. A wide interval means the estimate is uncertain. Also check for practical significance: a statistically significant effect might be too small to matter for your business. Finally, beware of early stopping. If you stopped early because the results looked good, the effect size is likely inflated. The true effect is probably smaller, and the p-value is not valid. The only safe way to stop early is to use a formal sequential design that adjusts for multiple looks. Otherwise, treat any early result as a preview, not a verdict.
6 What to change next
If you have been peeking, the first change is to commit to a fixed sample size and a single analysis. This may feel uncomfortable, especially if you are used to rapid iteration. But the discipline pays off in more reliable decisions. Start by calculating the required sample size for your next test using a power analysis. Use conservative assumptions; if you are unsure about the baseline rate or the minimum detectable effect, run a pilot study or use historical data. Then, lock the analysis plan and do not deviate. If you need to monitor, use a sequential design such as group sequential or always-valid inference. These methods allow for multiple looks while controlling error rates, but they require more data and careful implementation.
Second, improve your experimentation culture. Encourage teams to pre-register hypotheses and metrics. Make it easy to wait for results by automating the analysis and reporting. Celebrate tests that are well-designed even if they produce null results. Null results are valuable; they prevent you from shipping ineffective changes. Third, invest in education. Many people peek because they do not understand the statistical consequences. A short workshop on p-values, power, and optional stopping can go a long way. Finally, consider using tools that enforce good practices, such as platforms that lock the sample size or that provide sequential boundaries. Remember that no tool can replace judgment. Results depend on your setup and sample size, so always tailor your approach to your context. By changing these habits, you will reduce false positives and make better decisions.
7 Frequently asked questions
Q: Is it ever okay to peek at an A/B test? A: It is okay to monitor data quality, but not the primary outcome. If you must look at the outcome, use a sequential design that adjusts for multiple comparisons. Otherwise, peeking inflates false positives.
Q: How often can I check without inflating error rates? A: Any unplanned look at the primary outcome inflates error rates. The more you look, the worse it gets. There is no safe number of peeks for a fixed-horizon test. If you need to look, plan it in advance with a sequential method.
Q: What if I stop early because the effect is huge? A: A huge early effect is often a random fluctuation. Stopping early based on that effect biases the estimate upward and invalidates the p-value. The true effect is likely smaller. Only stop early if your sequential design allows it.
Q: Does peeking matter with large samples? A: Yes, though the inflation is smaller. With very large samples, each look provides more information, but the multiple testing problem remains. The false positive rate still exceeds the nominal level. Always pre-specify the analysis.
Q: How do I explain peeking to stakeholders? A: Use an analogy: flipping a coin and checking after every flip for a streak of heads. Eventually you will see a streak, but it does not mean the coin is biased. Similarly, peeking finds random streaks. Explain that waiting for the planned sample size gives a fair test.
Q: What is the best way to avoid peeking? A: Pre-register your analysis plan, automate the final analysis, and use sequential designs if monitoring is required. Education and culture change also help. Remember that results depend on your setup and sample size, so consult a statistician if unsure.