1 Overview
Peeking at an A/B test is one of the most common habits in experimentation, and it is also one of the most damaging. The temptation is understandable: you launch a change, you want to know whether it works, and the dashboard updates in real time. So you check the numbers on day one, again on day two, and again after a long weekend. Each time you look, you form an impression. Each impression nudges you toward a decision. The problem is that those impressions are being formed on data that is still mostly noise, and noise moves around.
The core issue is that an experiment is a measurement device, and like any measurement device it has a settling time. Early in a test, random variation dominates. A treatment can look like a clear winner on Tuesday and a clear loser on Thursday without anything about the underlying experience having changed. If you stop when the graph looks good, you are not measuring the effect of your change; you are measuring the luckiest moment in a random walk. Letting the test run to its planned sample size is not bureaucratic caution. It is the difference between a reading and a guess.
This article explains why peeking distorts results, how to frame a question that can actually be answered, how to design and size a test, and how to read the outcome once the planned data has arrived.
2 Framing the question
A test begins long before the first visitor is assigned. It begins with a question specific enough that data can answer it. "Is the new page better?" is not a testable question, because better is undefined and the new page is a bundle of many changes. A workable question names the population, the change, and the outcome: for example, whether a simplified checkout form increases completed purchases among first-time visitors on mobile. That question implies a hypothesis, a metric, and a scope, and each of those choices shapes what the result will mean.
The hypothesis should be a statement you can falsify. It should say which direction you expect the effect to go and why, grounded in a reason such as reduced friction or clearer messaging. Writing the reason down matters because it forces you to distinguish a genuine mechanism from a vague hope. If the result later contradicts the hypothesis, the reason helps you interpret what happened instead of simply declaring the test a failure.
Choose one primary metric before launch and treat it as the decision metric. Secondary metrics are useful for context, but if you promote whichever metric moved most, you are effectively running many tests and reporting the best one. That inflates the chance of a false positive. A good framing also states the minimum effect that would justify shipping the change, because a statistically detectable effect is not automatically a meaningful one. Results depend on this setup, so the framing deserves more time than the dashboard.
3 Designing the test
The design determines what comparisons are valid. The standard approach is a randomized controlled experiment with two variants: a control that represents the current experience and a treatment that contains the change. Random assignment ensures that, on average, the two groups differ only by the change, so any difference in the metric can be attributed to it rather than to pre-existing differences between visitors. Assignment should be stable, meaning a returning visitor sees the same variant, and it should be independent of anything that could correlate with the outcome.
Decide in advance how visitors enter the experiment, how they are counted, and what counts as a conversion. Counting each visitor once avoids the distortion that comes from a small number of highly active users. Excluding internal traffic, bots, and duplicate sessions protects the comparison. If the change affects only part of the experience, restrict the test to the relevant segment rather than diluting the effect across visitors who never encounter it.
Also decide how long the test will run and what would cause you to stop. Legitimate stopping rules include reaching the planned sample size, a pre-specified safety or quality threshold, or a technical failure that invalidates the data. "The result looks significant" is not a stopping rule; it is the definition of peeking. Documenting the design before launch turns later decisions into execution rather than improvisation, and it makes the eventual result far easier to trust.
4 Sample size and validity
Sample size is where most peeking problems are born. Before launch, estimate how many visitors you need based on the baseline rate of your metric, the smallest effect you care about detecting, your chosen significance level, and your desired power. These inputs are judgments, not facts, and they should be stated openly. A test sized to detect a large effect will be small and fast but blind to subtle improvements; a test sized for a small effect will take much longer. Both can be correct depending on what decision you face.
The danger of peeking is that it changes the effective error rate of the test. If you check results repeatedly and stop when the p-value crosses a threshold, the probability of a false positive is no longer the nominal level you chose. Each look is another chance for noise to produce a flattering result. The same logic applies to comparing many metrics or many segments: the more comparisons you make, the more likely one of them looks impressive by chance. Validity also depends on practical checks such as sample ratio mismatch, which occurs when the observed split between variants deviates from the intended split and often signals a tracking or assignment bug.
None of this means results are meaningless if you look early. It means early looks should inform operations, not decisions. Check that the test is running correctly, that traffic is arriving, and that no errors are occurring. Then wait. The planned sample size is the point at which the measurement has the properties you designed it to have.
5 Running and reading the results
Once the planned sample is reached, read the result in a disciplined order. First confirm data quality: the variant split should match the design, the metric definitions should be unchanged, and the test period should not contain an obvious anomaly such as an outage or a promotion that affected only one variant. If quality checks fail, the result is not interpretable, and the right move is to fix the problem and rerun rather than to salvage the number.
Next, compare treatment and control on the primary metric. Report the estimated effect with a confidence interval rather than a single number, because the interval communicates how much uncertainty remains. A p-value summarizes how compatible the observed data is with the hypothesis of no effect, but it does not measure the size or importance of the effect. A small, precisely estimated improvement and a large, imprecisely estimated one can produce similar p-values while implying very different decisions. Always ask whether the interval includes effects too small to matter.
Finally, interpret the result in the context of the original hypothesis. If the effect is in the expected direction and large enough to justify the change, ship it. If it is inconclusive, the honest conclusion is that the test did not detect an effect at this sample size, not that no effect exists. If it is negative, that is information too, and often the most useful kind. Results depend on setup and sample size, so state the conditions under which the finding was produced.
6 What to change next
The end of a test is the beginning of the next question. If the change won, the follow-up is usually about magnitude and generalization: does the effect hold for other segments, other devices, or other traffic sources, and can the winning idea be extended further? If the change lost or was inconclusive, the follow-up is about mechanism: was the hypothesis wrong, was the implementation weak, or was the effect simply smaller than the test could detect? Each answer points to a different next experiment.
Avoid the trap of iterating on the same test by rerunning it repeatedly until it produces a favorable result. That practice converts a controlled experiment into a search for a flattering sample, and it destroys the error guarantees the design was meant to provide. If you genuinely need more data, extend the test using a pre-specified rule rather than stopping and restarting opportunistically. Sequential methods exist for legitimate continuous monitoring, but they require a design chosen in advance, not improvised after seeing the numbers.
Keep a record of what you tested, how it was sized, what you found, and what you decided. Over time that record becomes more valuable than any single result, because it shows which kinds of changes tend to work in your context and how large the effects realistically are. The discipline of letting tests run is what makes that record trustworthy.
7 Frequently asked questions
Is it ever acceptable to stop a test early? Yes, when a pre-specified rule says so. Safety issues, severe technical failures, or a formal sequential design with adjusted thresholds are legitimate reasons. The key is that the rule exists before the data arrives, so the decision is not driven by how the graph happens to look at a given moment.
Why does peeking cause false positives? Because each look is an opportunity for random noise to cross your threshold. If you keep looking and stop at the first crossing, you are selecting the most extreme moment rather than measuring the average behavior of the experiment. The result is an effect estimate that is biased upward and a false positive rate higher than the one you intended.
How long should a test run? Long enough to reach the planned sample size, and ideally long enough to cover full weekly cycles so that day-of-week patterns do not fall unevenly across variants. The exact duration depends on traffic volume and the effect size you are trying to detect, which is why the sample-size calculation comes first.
What if the result is inconclusive? Treat it as a genuine finding: at this sample size, the data cannot distinguish the treatment from the control by the amount you cared about. You can rerun with a larger sample if the question still matters, but do not declare victory or defeat from an underpowered test.