1 Overview
Peeking at an A/B test means checking whether the result is statistically significant before the experiment has reached its planned sample size, and then deciding to stop early when it looks good. It feels diligent. It feels like saving time and money. In practice it is one of the most common ways teams fool themselves, because the statistical machinery behind a significance test assumes a single look at a fixed amount of data. When you look repeatedly and stop at the first crossing of the significance threshold, you change the question being answered without changing the label on the report.
The core problem is not that early data is meaningless. Early data is noisy, and noise is exactly what a fixed-horizon test is designed to tolerate. The problem is the decision rule. If you promise to analyze once, a five percent false-positive rate means what it says. If you check after every hundred users and stop when the p-value dips below the threshold, the chance of a false positive across the whole sequence is much larger than the nominal rate. The exact inflation depends on how often you look, how long the test runs, and the underlying effect, so no single number describes every case. What is stable across setups is the direction: more looks means more chances for noise to masquerade as a result.
This article explains why peeking distorts results, how to frame a question that can survive a single honest analysis, how to design and size the test, and what to do when a stakeholder asks for a peek anyway. The goal is not to make experimentation slower for its own sake. It is to make the conclusions you act on actually mean what you think they mean.
2 Framing the question
Before any test runs, write down the decision the experiment is meant to inform. A useful framing names the change, the population it applies to, the primary metric, and the direction of improvement that would cause you to ship. If the team cannot agree on what a win looks like before launch, no amount of data will settle the argument afterward, and the temptation to peek grows precisely because the success criterion is vague.
A good question is narrow enough to be falsifiable. "Does the new checkout flow increase completed purchases per eligible session?" is testable. "Is the new checkout better?" is not, because better could mean faster, prettier, cheaper, or more profitable, and each of those implies a different metric and a different sample size. Choosing one primary metric in advance also protects you from the quieter version of peeking: scanning many metrics, finding one that moved, and telling its story. That practice multiplies comparisons and inflates false positives just as surely as repeated looks at a single metric.
Finally, decide in advance what you will do with each plausible outcome. If the treatment wins, you ship. If it loses, you keep the control and record what you learned. If the interval is too wide to distinguish a meaningful gain from a meaningful loss, you have an inconclusive result, which is information too. Committing to these branches before launch makes the eventual readout a decision procedure rather than a negotiation, and it removes much of the pressure that drives peeking in the first place.
3 Designing the test
A clean design starts with randomization. Eligible units, whether users, sessions, or accounts, are assigned to control or treatment by a random mechanism, so that on average the two groups differ only by the treatment. The unit of randomization must match the unit of analysis; if users are randomized but sessions are analyzed, dependence between sessions from the same user can distort the variance and make intervals too narrow. Decide the unit, the eligibility rules, and the exposure definition before launch, and keep them fixed.
Next, specify the analysis plan. Name the primary metric, the test or model used to compare arms, the significance level, and the decision threshold for practical significance. If the metric is a ratio or a heavily skewed quantity, say so in advance, because the appropriate method and the required sample size both change. Pre-registering the plan does not require a formal registry; a short written document that the team agrees on is enough, and it is the artifact that makes later peeking obviously off-plan.
Guard against the operational leaks that quietly invalidate tests. Check that assignment is actually random and balanced, that both arms receive comparable traffic quality, and that instrumentation fires consistently. Run the test long enough to cover full weekly cycles if behavior varies by day, and avoid overlapping experiments that could interact. None of these steps requires exotic statistics. They require deciding the rules before the data arrives, which is also what makes a single final analysis defensible.
4 Sample size and validity
Sample size is where the promise of a fixed-horizon test becomes concrete. To estimate it you need a baseline value of the metric, the smallest effect that would change your decision, the significance level, and the desired power. Plugging in those inputs yields the number of units per arm needed to detect that effect with the stated confidence. The result depends heavily on the setup: a rare conversion event needs far more traffic than a common one, and a small effect needs far more than a large one. Because these inputs are estimates, treat the calculation as a planning tool, not a guarantee.
Peeking attacks validity by changing the effective error rate. Each interim look is another opportunity for random fluctuation to cross the threshold, so optional stopping inflates false positives above the nominal level. The size of the inflation depends on the number and timing of looks and on the true effect, which is why no universal correction applies. Sequential and group-sequential methods exist precisely to allow planned interim analyses while controlling error, but they require pre-specifying the looks and the stopping boundaries. Ad hoc peeking is not a lightweight version of those methods; it is a different and uncontrolled procedure.
Other threats compound the problem. Underpowered tests produce wide intervals that invite more peeking, because nothing is ever conclusive. Changing the metric, excluding users, or extending the test after seeing results all reintroduce the same bias. Novelty effects and seasonality can make early data unrepresentative, so a result that looks strong in week one may fade. The defense is boring and effective: fix the plan, size it honestly, and let it run.
5 Running and reading the results
During the run, monitor health, not outcomes. Check that assignment is balanced, that the metric is logging correctly, and that no arm is broken by a bug. These are engineering checks, not significance checks, and they should not involve comparing conversion rates between arms. If a genuine implementation failure occurs, fix it, document it, and restart or analyze only the clean period under a revised plan. What you should not do is watch the p-value drift and decide when to stop.
At the planned analysis, read the result once. Report the estimated effect with its confidence interval, not just a yes-or-no verdict. An interval that excludes zero but spans effects from trivial to enormous is a weak basis for shipping, while an interval centered on a meaningful gain is strong evidence even if the p-value is not tiny. Compare the interval against the practical threshold you set in advance, and state the conclusion in plain language: the data are consistent with a gain of this size, or they are not precise enough to tell.
If the result is inconclusive, resist the urge to keep the test running indefinitely. Extending a test after seeing its results is a form of peeking, and it biases the eventual estimate. Instead, treat the inconclusive outcome as a design signal: the effect may be smaller than assumed, the metric noisier, or the traffic insufficient. The honest readout is the one produced by the procedure you committed to, not the one produced by the moment the numbers finally looked agreeable.
6 What to change next
The first change is procedural: require a written analysis plan before launch, including the primary metric, the sample size, the stopping rule, and the decision thresholds. Make the plan visible to stakeholders so that requests to check results early can be answered with a shared reference rather than a debate. A one-page template is usually enough, and it pays for itself the first time it prevents an early stop on noise.
Second, invest in the tooling and habits that make waiting tolerable. Dashboards should show sample accumulation and data health, not live significance. Scheduled readouts at the planned horizon give stakeholders a date to look forward to, which reduces the pressure for ad hoc peeks. If the organization genuinely needs to make decisions before the full sample arrives, adopt a pre-specified sequential design with planned interim looks and appropriate boundaries, and treat those looks as part of the plan rather than exceptions to it.
Third, build a culture that treats inconclusive results as useful. Record what was tested, what the interval showed, and what would be tested next. Over time, these records calibrate intuition about effect sizes and traffic requirements, which improves future sample-size calculations. The teams that stop peeking are not the ones with the most discipline in the moment; they are the ones whose process makes the single planned analysis the natural thing to do.
7 Frequently asked questions
Is it ever acceptable to look at results before the test ends? Looking is not the problem by itself; deciding based on what you see is. If you inspect data only to verify that logging and assignment are healthy, and you do not change the stopping rule, you have not introduced optional stopping. The danger begins when the decision to stop, extend, or ship depends on the observed effect. If you need interim decisions, plan them in advance with a sequential method that controls error.
Does peeking only matter for small effects? No. Peeking inflates false positives across effect sizes, though the practical consequences differ. With a large true effect, the test will often reach significance legitimately, and early stopping may cost little. With a small or null effect, peeking is most dangerous, because the only way the test crosses the threshold early is usually noise. Since you rarely know the true effect in advance, the safe assumption is that peeking is harmful until a planned sequential design says otherwise.
What if my boss demands a peek? Show the sample-size plan and explain that the current result is not yet interpretable. Offer a scheduled readout date and, if a decision truly cannot wait, propose a pre-specified interim analysis with proper boundaries. Frame the request as a design question rather than a refusal, and keep the conversation about what decision the data can support at its current size.
How do I know my sample size is right? You do not, exactly. It is an estimate based on assumptions about baseline rates, variance, and the effect you care about. Revisit those assumptions after the test, compare the observed variance with the planning value, and adjust future calculations. Results depend on the setup and sample size, so treat each test as a calibration opportunity rather than a final verdict on your planning process.