1 Overview
Peeking at an A/B test before it has reached its planned sample size is one of the most common ways teams accidentally fool themselves. When you check results repeatedly and stop as soon as a difference looks impressive, you inflate the chance of declaring a winner that is actually just noise. The fix is not to ignore the data but to design the test so that the decision rule is set before the first visitor is exposed. That means defining the hypothesis, the primary metric, the sample size, and the stopping condition in advance, then letting the test run to completion.
This article explains how to stop peeking by fixing the underlying design. It covers how to frame a testable question, how to choose a design that isolates the change, how to think about sample size and validity, and how to read results once the planned sample is reached. It also covers what to do next, whether the result is positive, negative, or inconclusive. Throughout, remember that every result depends on the specific setup, the metric, and the sample size. There is no universal answer that applies to every experiment, and no shortcut that replaces a well-planned test.
2 Framing the question
A good experiment starts with a question that can be answered by data. Instead of asking whether a new design is better, ask whether changing a specific element, such as the call-to-action text or the placement of a form, will increase a defined metric, such as the rate at which visitors complete a purchase. The hypothesis should state the expected direction of the effect and the population it applies to. For example, you might hypothesize that simplifying the checkout form will increase the completion rate for first-time visitors on mobile devices. This kind of statement is testable because it names the change, the metric, and the audience.
It is equally important to decide what you will not measure. A single primary metric keeps the decision clear and reduces the temptation to hunt for a winning segment after the fact. Secondary metrics can provide context, but they should not override the primary metric. Before launching, write down the hypothesis, the primary metric, and the guardrail metrics that must not degrade. This pre-registration is your best defense against peeking, because it removes the ambiguity about what counts as success. If you cannot state the hypothesis in one or two sentences, the test is not ready to run.
3 Designing the test
The design determines what you can conclude. A controlled experiment randomly assigns users to a control group and one or more variant groups. Randomization helps ensure that the groups are comparable on average, so that differences in the metric can be attributed to the change rather than to pre-existing differences. The control group should represent the current experience, and the variant should differ only in the element you intend to test. If you change multiple things at once, you will not know which change caused the effect. Keep the design as simple as the question allows.
Decide in advance how long the test will run and how much traffic each group will receive. The duration should cover at least one full business cycle, such as a week, to account for day-of-week patterns, but it should also be driven by the sample size needed to detect the effect you care about. Avoid stopping early because the results look promising. If you plan to analyze segments, such as new versus returning visitors, specify those segments before launch and treat them as secondary. Finally, document the design, including the randomization unit, the exposure event, and the metric definition, so that anyone reviewing the test can understand exactly what was done.
4 Sample size and validity
Sample size is not a formality; it is a core part of the design. A test that is too small may fail to detect a real effect, while a test that is too large may waste time and expose users to a worse experience. The right size depends on the baseline rate of the metric, the smallest effect that would be practically meaningful, the significance level you choose, and the power you want. These inputs are specific to your situation. There is no universal sample size that works for every experiment, and you should not borrow a number from a blog post without checking whether it fits your baseline and your minimum detectable effect.
Validity is threatened by anything that makes the groups differ for reasons other than the change. Peeking and stopping early are major threats because they change the decision rule after seeing the data. Other threats include sample ratio mismatch, where the groups are not split as intended; instrumentation errors, where the metric is not logged consistently; and external events, such as a holiday or a news cycle, that affect one group more than the other. Novelty effects can also distort results if users interact with a new design simply because it is new. A valid test is one where the design, the sample size, and the analysis plan are fixed before launch and followed without deviation.
5 Running and reading the results
Once the test is live, resist the urge to check the dashboard every hour. Interim looks are not harmless; each look is an opportunity to see a random fluctuation and mistake it for a signal. If you must monitor for technical problems, check only operational metrics such as error rates or traffic balance, not the primary success metric. When the pre-planned sample size is reached, then you can analyze the primary metric. Compare the groups using the method you specified in advance, and report the effect size along with a measure of uncertainty, such as a confidence interval. A result is not just a winner or a loser; it is an estimate with a range.
Read the result in the context of the original hypothesis. If the primary metric improved by a meaningful amount and the uncertainty is narrow enough to rule out no effect, you have evidence for the change. If the result is inconclusive, do not declare a winner based on a secondary metric or a subgroup that happened to look good. That is a form of peeking after the fact. Instead, treat the inconclusive result as information: it may mean the effect is smaller than you powered for, or that the change did not matter. In all cases, remember that the result depends on the setup and the sample size. A finding from one test does not automatically generalize to other pages, audiences, or time periods.
6 What to change next
The end of a test is the beginning of the next decision. If the result is positive and valid, the next step is to implement the change and continue to monitor the primary metric to ensure the effect holds in the long run. If the result is negative, you have learned that the change did not help in this context, which is still valuable. If the result is inconclusive, you have several options: run a larger test with the same design, test a bolder change that is more likely to produce a detectable effect, or revisit the hypothesis to see whether the metric was the right one. Each option has trade-offs, and the choice depends on the cost of running another test and the potential value of the change.
What you should not do is keep the test running indefinitely or slice the data until something looks significant. That path leads to false discoveries and eroded trust in experimentation. Instead, use the results to refine your understanding of your users and your product. Document what you learned, including the sample size, the observed effect, and any caveats. Over time, a series of well-designed tests builds a body of evidence that is more reliable than any single result. The discipline of stopping when planned and reading the result as an estimate, not a verdict, is what makes experimentation a sustainable practice.
7 Frequently asked questions
Why is peeking so harmful? Peeking is harmful because it changes the probability of a false positive. If you check results repeatedly and stop when the difference looks significant, you are effectively running many tests and reporting the most extreme one. The nominal significance level no longer applies, and you are more likely to declare a winner that is actually noise. The solution is to fix the sample size and the stopping rule in advance, and to avoid looking at the primary metric until the test is complete.
How do I know when to stop a test? Stop when you reach the pre-planned sample size or the pre-planned end date, whichever comes first. If you must stop early for operational reasons, such as a bug or a site outage, document the reason and treat the result as inconclusive rather than as a valid test. Do not stop early simply because the results look good or bad. The decision rule should be set before launch and followed consistently.
What if I need to check for errors? You can monitor operational health metrics, such as page load time, error rates, and the ratio of users assigned to each group, without looking at the primary success metric. These checks help you catch technical problems without biasing the result. If you see a sample ratio mismatch, pause the test and investigate before continuing. Once the issue is fixed, you may need to restart the test with a fresh sample. Always prioritize the integrity of the design over the convenience of a quick answer.