1 Overview
Peeking at an A/B test is one of the most common ways a well-intentioned experiment becomes misleading. The problem is not curiosity itself. The problem is making a decision, or changing the decision rule, based on data that is still accumulating. When you check results repeatedly and stop when the difference looks significant, you give random fluctuations many chances to cross a threshold. The false-positive rate is no longer the one you planned. The test may look decisive while actually being noisy.
This article explains how to run valid A/B test statistics by separating the question, the design, the sample-size plan, and the analysis. The goal is not to eliminate judgment. The goal is to make judgment happen at the right time. A valid experiment has a primary metric, a pre-specified decision rule, and a sample size that gives the test a fair chance to detect an effect if one exists. Results still depend on setup and sample size, so no single test should be treated as universal proof.
The practical alternative to peeking is not blindness. It is disciplined monitoring. You can check for instrumentation errors, sample ratio mismatch, and guardrail metrics without evaluating the primary outcome. You can also use sequential methods designed for repeated looks if you need them. The key is to decide in advance what will be monitored, what will be tested, and when the primary analysis will occur.
2 Framing the question
A valid test starts with a hypothesis that is specific enough to be wrong. Instead of saying the new page will perform better, state which change is expected to affect which behavior for which population and in which direction. The hypothesis should be written before the test begins and should not be revised after seeing the data. This protects against the common trap of turning an exploratory finding into a confirmatory claim. The hypothesis is not a prediction of the exact effect size, but it should identify the mechanism you believe is at work.
The primary metric should follow directly from the hypothesis. Choose one metric as the decision metric and treat all others as secondary or diagnostic. If you test many metrics and pick the one that looks best, the reported uncertainty is too narrow. Multiple comparisons inflate false positives, and the inflation grows with the number of independent tests. Secondary metrics can still be useful for understanding why an effect occurred, but they should not silently become the basis for shipping a change.
Define the unit of analysis as well. In many experiments the unit is a user or session, but it can also be an account, a device, or a geographic region. The unit determines what independence means and how the sample size is calculated. If the unit is mis-specified, the effective sample size may be smaller than it appears, and the uncertainty in the result will be understated. A clear question includes the population, the unit, the primary metric, and the direction of the expected effect.
3 Designing the test
A randomized controlled experiment is the standard design for causal questions. Eligible units are assigned to control or treatment using a random mechanism, and the assignment is stable for the analysis window. Randomization balances both measured and unmeasured characteristics on average, which is what allows a difference in outcomes to be attributed to the treatment rather than to pre-existing differences. The control group should represent the current experience, and the treatment group should differ only in the change being tested. Any additional difference, such as a different traffic source or a different time period, becomes a confound.
The design should also specify the analysis plan before launch. That plan includes the primary metric, the decision rule, the significance level, the direction of the test, and how missing or delayed data will be handled. It should state whether the test is one-sided or two-sided and whether any subgroups will be analyzed. Pre-specifying these choices prevents the analysis from drifting toward the most favorable result. If you plan to look at the data repeatedly, use a method designed for that, such as a sequential test or a group sequential design, rather than applying a fixed-sample rule at every glance.
Finally, consider the practical constraints. Tests can be contaminated by bots, by users who see both versions, or by changes in the product during the test. A good design includes checks for sample ratio mismatch, which occurs when the observed split between groups differs from the intended split. It also includes guardrail metrics that should not degrade, such as latency, error rates, or unsubscribe rates. These checks do not replace the primary analysis, but they help distinguish a real effect from a broken experiment.
4 Sample size and validity
Sample size is not a formality. It determines how much uncertainty remains in the estimated effect and how likely the test is to detect an effect of a given size. The required sample size depends on the baseline rate of the metric, the smallest effect you care about, the variance of the metric, and the acceptable false-positive and false-negative rates. Because these inputs are specific to the setup, there is no universal number that makes a test valid. A test with too few units may fail to detect a real effect, while a test with far more units than needed may expose users to a worse experience for longer than necessary.
Validity is threatened by several common practices. Peeking and optional stopping are the most familiar: if you check the result repeatedly and stop when it looks significant, the false-positive rate rises above the planned level. Multiple comparisons have a similar effect when many metrics or subgroups are tested without adjustment. Sample ratio mismatch suggests that assignment or instrumentation is not working as intended, and its presence can invalidate the comparison. Novelty effects, seasonality, and changes in the user population during the test can also make the observed effect unrepresentative of the long-run effect.
Sample size and validity are connected. A small sample is more sensitive to random imbalance and to outliers, and it makes early looks especially misleading. A large sample can detect tiny effects that may not be practically meaningful, so the decision rule should include a minimum effect of interest. The safest approach is to calculate the sample size in advance, freeze the analysis plan, and monitor only the health checks until the planned analysis date.
5 Running and reading the results
During the test, monitor the experiment for technical health rather than for the primary outcome. Check that the assignment ratio matches the plan, that the metric is being logged correctly, and that guardrail metrics are stable. If you find a problem, fix it and document the change. Do not evaluate the primary metric early and then decide whether to continue. If the design requires repeated looks at the primary metric, use a method that adjusts the error rate for those looks. Otherwise, wait until the pre-specified sample size is reached.
When the test ends, estimate the effect and its uncertainty. A useful result includes the difference between groups, an interval that expresses the range of plausible values, and a p-value interpreted against the pre-specified decision rule. The p-value is not the probability that the null hypothesis is true, and it is not a measure of practical importance. A small p-value indicates that the observed difference would be unlikely if the null hypothesis were true and the model assumptions held. The interval shows how precise the estimate is, and it often reveals that the data are consistent with both a meaningful improvement and a negligible one.
Read the result in the context of the design. If the sample size was too small, a non-significant result may simply mean the test lacked power. If the sample was large, a significant result may be too small to matter. Check whether the effect is consistent across pre-specified subgroups and across time. A result that appears only in one segment or only in the first days of the test deserves caution. The goal is not to declare a winner but to update your belief with the strength the data actually provide.
6 What to change next
The output of a valid test is a decision and a learning. If the treatment improved the primary metric without harming guardrails, the next step may be to ship it, but shipping should still consider the size and uncertainty of the effect. If the result is inconclusive, the next step is not to rerun the same test until it becomes significant. That practice turns the experiment into a search for a favorable random draw. Instead, revisit the hypothesis, the metric, or the sample-size plan. A larger sample or a more sensitive metric may be needed to answer the question.
If the result is negative, treat it as information about the mechanism, not as a failure. The change may not affect the intended behavior, or it may affect it in a way that is offset by another effect. Use secondary metrics and qualitative evidence to form a new hypothesis, then test that hypothesis in a fresh experiment. Avoid the temptation to slice the data until a significant subgroup appears. Subgroup findings are useful for generating hypotheses, but they usually need confirmation in a pre-specified test.
Over time, build a process rather than relying on individual tests. Document the hypothesis, the primary metric, the sample-size calculation, and the decision rule for each experiment. Keep a record of what was tested and what was learned. This makes it harder to repeat mistakes and easier to see patterns across experiments. It also reinforces the central discipline: decide how you will judge the result before you see it, and let the data speak within the limits of the design.
7 Frequently asked questions
Is it ever acceptable to peek at an A/B test? It is acceptable to monitor technical health, such as sample ratio mismatch or logging errors. It is not acceptable to evaluate the primary outcome and stop early under a fixed-sample rule. If you need to make decisions at multiple points, use a sequential or group sequential design that controls error rates across looks. The method, not willpower, should handle the repeated testing.
Why does peeking increase false positives? Each look is another chance for random noise to produce a difference that crosses the significance threshold. Under the null hypothesis, the probability of crossing at least once across many looks is higher than the probability at a single planned look. The exact inflation depends on how often you look and how you stop, which is why the analysis plan must match the monitoring plan.
Does a significant result prove the treatment works? No. It indicates that the observed difference is unlikely under the null hypothesis and the model assumptions. The estimate still has uncertainty, and the effect may not generalize to other populations or time periods. Practical significance, cost, and guardrail metrics also matter. A significant result is evidence, not proof.
How large should my sample be? Large enough to detect the smallest effect you care about with acceptable error rates, given the baseline rate and variance of your metric. Because those inputs vary by setup, the number must be calculated for each experiment. Running a test without a sample-size plan makes the result difficult to interpret and easy to overstate.