1 Overview
A/B testing is often taught as a simple sequence: form a hypothesis, split the traffic, wait for significance, and ship the winner. That sequence hides a large family of errors that can make a test look decisive when it is not. Peeking at results and stopping when the numbers look good is the most famous of these errors, but it is not the only one. The same experiment can be undermined by unclear questions, mismatched metrics, unequal assignment, novelty effects, and selective storytelling after the fact.
This article treats the experiment as a chain of decisions, each of which can introduce bias or noise. The goal is not to make testing sound impossibly hard, but to show where the hidden risks live and how to reduce them. A well-run test is not just a statistical calculation; it is a disciplined process that starts before the first user is exposed and continues after the last data point is collected.
Because results depend on the specific setup, the population, and the sample size, no single rule can guarantee a correct answer. Instead, the aim is to make the assumptions explicit, to pre-commit to a decision rule, and to treat every result as an estimate with uncertainty rather than a verdict. That mindset is what separates experimentation from wishful thinking.
2 Framing the question
Every credible experiment begins with a question that can be answered by data. A vague goal such as improve engagement is not a testable hypothesis. A useful hypothesis names the change, the expected direction, and the outcome it should affect. For example, adding a progress indicator to a checkout flow is expected to reduce abandonment among first-time buyers. That statement can be falsified, and it forces the team to define who is in the population and what counts as success.
The metric should follow from the hypothesis, not the other way around. Choose one primary metric that directly represents the outcome you care about, then add a small number of secondary metrics and guardrails to catch unintended consequences. If the primary metric is revenue per visitor, a guardrail might be page load time or customer support contacts. When teams collect dozens of metrics and later pick the one that moved, they are not testing a hypothesis; they are describing a coincidence.
A common hidden risk at this stage is the moving target. If the question changes after data collection begins, the original test no longer answers the new question. The safest practice is to write down the hypothesis, the primary metric, the population, and the decision rule before launch. That document does not need to be elaborate, but it should be specific enough that a different analyst could read it and agree on what the result means.
3 Designing the test
The design determines what comparisons are valid. Random assignment is the core safeguard: each unit should have a known, non-zero chance of receiving each variant, and the assignment should be independent of the outcome. When assignment is broken, for example by routing users based on their device or location, the groups may differ in ways that have nothing to do with the treatment. That is why sample ratio mismatch checks matter; if the observed split deviates from the intended split by more than chance would explain, the experiment is suspect.
Beyond randomization, the design must define the unit of analysis. Users, sessions, and page views behave differently, and analyzing at a finer level than the assignment can inflate significance. If users are assigned to variants, the analysis should usually aggregate to the user level. Interference is another design risk: in social or marketplace products, treating one user can affect another, which violates the independence assumption. In such cases, cluster randomization or switchback designs may be more appropriate.
Finally, the design should specify the analysis plan in advance. That includes the statistical test, the handling of outliers, the treatment of missing data, and the subgroups that will be examined. Pre-specification does not eliminate every problem, but it prevents the most common form of self-deception: choosing the analysis after seeing the data. A design that is clear about its assumptions is easier to audit and easier to learn from, even when the result is not the one the team hoped for.
4 Sample size and validity
Sample size is not a formality; it determines what the experiment can detect. Before launch, estimate the number of units needed based on the baseline rate of the primary metric, the smallest effect that would be practically meaningful, and the acceptable rates of false positives and false negatives. A test that is too small will often fail to detect a real effect, and a test that is too large may flag trivial differences as significant. Neither outcome is useful for decision-making.
Validity is threatened by anything that makes the observed effect differ from the true effect. Peeking and optional stopping are classic threats because they change the probability of a false positive. Multiple comparisons do the same when many metrics or subgroups are tested without correction. Novelty and primacy effects can distort early results, especially when users react to change itself rather than to the substance of the change. Instrumentation errors, bot traffic, and inconsistent logging can create effects that exist only in the data pipeline.
Because these risks interact with sample size, there is no universal threshold for when a test is trustworthy. A small experiment with a clean design and a pre-registered analysis can be more informative than a large experiment with a shifting decision rule. The practical lesson is to treat sample size and validity as a single problem: decide in advance how much data you need, how you will know the data are sound, and what you will do if the assumptions fail. Documenting those choices makes the result easier to interpret and harder to overstate.
5 Running and reading the results
Once the experiment is live, the main job is to protect the design. Monitor health checks such as assignment ratios, logging completeness, and guardrail metrics, but avoid using the primary metric to decide when to stop. If the plan allows interim looks, use a method that controls error rates, such as group sequential boundaries or always-valid inference. Otherwise, the moment you check the result becomes part of the analysis, and the reported significance no longer means what it appears to mean.
When the test ends, read the result as an estimate with uncertainty. Report the effect size, the confidence interval, and the pre-specified decision threshold. A confidence interval that includes both meaningful gains and meaningful losses is not a green light; it is a signal that the data are not precise enough to support a decision. Statistical significance is not the same as practical significance, and a non-significant result is not proof that the treatment had no effect. It may simply mean the sample was too small or the metric too noisy.
Look for explanations before celebrating or abandoning the change. Check whether the effect is consistent across pre-specified segments, whether guardrails moved in a worrying direction, and whether the result is stable over time. If the test was stopped early, or if the primary metric was chosen after the fact, treat the finding as exploratory rather than confirmatory. The most useful outcome of an experiment is often not a binary ship or no-ship decision, but a clearer understanding of how the product behaves and what to test next.
6 What to change next
The next experiment should be shaped by what the last one revealed, including its limitations. If the result was inconclusive because the sample was too small, the next step may be to increase the sample, narrow the population, or choose a metric with less variance. If the effect was significant but small, the team can decide whether the change is worth shipping for reasons beyond the primary metric, or whether a stronger intervention is needed. If guardrails moved, the next test might focus on mitigating the harm rather than amplifying the gain.
It also helps to separate the decision from the learning. A test can fail to reach significance and still teach something important about user behavior, instrumentation, or the assumptions behind the hypothesis. Capturing those lessons in a short write-up makes future experiments better. The write-up should include the original hypothesis, the design, the sample-size rationale, the result with uncertainty, and the threats to validity that were observed or suspected.
Finally, avoid the temptation to run many small tests without a coherent question. A portfolio of experiments is most valuable when each one builds on a shared model of the user and the product. That model can be updated with every result, whether positive, negative, or ambiguous. The discipline of framing, designing, sizing, and reading each test carefully is what turns a series of A/B tests into a reliable learning system rather than a collection of anecdotes.
7 Frequently asked questions
Is peeking always wrong? Peeking is not inherently wrong if the analysis plan accounts for it. The problem is unplanned peeking followed by a decision to stop or continue based on the observed result. If you want to monitor continuously, use a method designed for that, such as sequential testing or always-valid confidence sequences. Otherwise, fix the sample size and the decision rule in advance.
How many metrics can I test? You can collect many metrics, but the primary decision should rest on one pre-specified primary metric. Testing many metrics without correction increases the chance of a false positive. If you need to explore secondary metrics, label those findings as exploratory and confirm them in a follow-up experiment.
What if my sample size is smaller than planned? A smaller sample means less precision and a higher chance of missing a real effect. If you must stop early, report the result with its uncertainty and avoid claiming a definitive win or loss. The honest conclusion may be that the experiment was underpowered.
Can a non-significant result prove no effect? No. It means the data are consistent with a range of effects, including zero and possibly meaningful ones. To distinguish no effect from an undetected effect, you need a design with enough precision to rule out effects that matter. That is why sample size and validity are inseparable from interpretation.