1 Overview
A well-designed A/B test is a disciplined way to compare two versions of something and learn whether a change caused a difference in a metric. It is not a guarantee of truth. It is a method that, when followed carefully, reduces the chance that you fool yourself into seeing an effect that is not there or missing one that is. The core idea is simple: randomly assign comparable units to a control and a variant, hold everything else as constant as you can, and measure a pre-specified outcome. Randomization is what makes the groups comparable on average, including on factors you did not measure. Without it, differences between groups can reflect who was in them rather than what you changed.
Design choices determine what you can conclude. The hypothesis must be stated before the test, not after looking at the data. The metric must be defined precisely and measured the same way in both groups. The sample size must be large enough to detect an effect of practical interest, and the analysis must follow the plan. Results depend heavily on setup and sample size; a small test can be noisy, and a biased test can be confidently wrong. This article walks through the decisions that keep an experiment honest, from framing the question to interpreting the result and deciding what to do next.
2 Framing the question
Start with a specific, falsifiable hypothesis. A useful hypothesis names the change, the population, the metric, and the expected direction. For example: adding a progress indicator to a signup form will increase completion rate among new visitors. Vague goals like improve engagement cannot be tested because they do not specify what would count as support or refutation. Write the hypothesis down before you build anything. This protects you from moving the goalposts later, which is one of the most common ways experiments fool their designers.
Next, choose the primary metric. It should be closely tied to the hypothesis and sensitive to the change you are making. Define it operationally: what event counts, over what time window, for which users. Pre-specify secondary metrics that capture other important outcomes, and guardrail metrics that should not get worse, such as page load time or support contacts. Deciding these in advance prevents cherry-picking a favorable metric after the fact. Also define the unit of randomization and analysis, such as user, session, or account, and keep them consistent. If the unit of analysis differs from the unit of randomization, standard errors can be wrong and significance can be overstated. Finally, state the decision rule: what result would lead you to ship, iterate, or stop. A clear question and a clear metric make the rest of the design tractable.
3 Designing the test
The design is the structure that makes a causal claim possible. Use a randomized controlled experiment with at least one control and one variant. Randomly assign units so that, on average, the groups differ only by chance before the change is applied. If randomization is not feasible, consider whether a quasi-experimental design can still support a limited conclusion, but be explicit about its weaker guarantees. Keep assignment consistent for the duration of the test so users do not flip between experiences, which can contaminate the measurement and confuse the user.
Control everything you can. The variant should differ from the control only in the intended change; otherwise you cannot attribute an effect to that change. Ensure both groups experience the same instrumentation, the same time period, and the same external conditions as much as possible. Pre-register the analysis plan, including the primary metric, the statistical test, and how you will handle outliers or missing data. Decide in advance whether to run a one-tailed or two-tailed test and what significance level you will use. Avoid changing the design mid-flight; if you must, treat the change as a new test or use methods designed for sequential analysis. A clean design is not about complexity. It is about making the comparison fair and the conclusion defensible.
4 Sample size and validity
Sample size is a planning decision, not an afterthought. To determine it, you need a baseline rate for the metric, the smallest effect that would matter to you, the desired statistical power, and the significance level. These inputs produce an estimate of how many units you need. If the required sample is larger than you can realistically collect, you have three honest options: accept a larger minimum detectable effect, extend the test duration, or reconsider whether the experiment is worth running. Running an underpowered test and interpreting a non-significant result as evidence of no effect is a common mistake. Absence of evidence is not evidence of absence.
Validity is threatened by many forces. Selection bias occurs when groups are not comparable, often because assignment was not truly random or because of differential attrition. Confounding occurs when another change happens at the same time. Instrumentation issues, such as inconsistent logging or bot traffic, can distort the metric. Novelty and primacy effects can make short tests misleading. Peeking at results repeatedly and stopping when they look significant inflates false positives. Multiple comparisons across many metrics or variants increase the chance of a spurious finding. External events, like a holiday or outage, can affect one group differently if timing is not balanced. Document these risks and monitor for them. A test is only as trustworthy as its weakest validity safeguard.
5 Running and reading the results
Once the test is live, resist the urge to make decisions from early data. Let it run to the pre-planned sample size or duration unless a guardrail metric shows harm. Monitor data quality: check that assignment is balanced, that the metric is logged consistently, and that no obvious external event is contaminating the groups. If you must stop early for harm, say so and interpret the result cautiously. When the test completes, compute the effect estimate and its uncertainty, such as a confidence interval. The p-value tells you how often you would see an effect this large or larger if there were truly no difference; it does not tell you the probability that the null hypothesis is true. Report the effect size in the metric's natural units, because statistical significance is not the same as practical importance.
Read the result against the pre-registered decision rule. If the primary metric moved in the expected direction with acceptable uncertainty and guardrails held, you have evidence to support the change. If not, you have learned something, but be careful about claiming the change had no effect. The confidence interval may still include meaningful effects. Look at secondary metrics for context, but do not promote them to primary after the fact. Consider whether the result is likely to generalize beyond the test population and time period. A single test is one piece of evidence, not a final verdict. The goal is to update your beliefs in proportion to the strength of the evidence, not to win an argument.
6 What to change next
Every experiment should end with a decision and a next question. If the variant won, plan a careful rollout and continue to monitor guardrail metrics, because effects can fade or reverse at scale. If it lost, document what you learned and why you think it failed. If the result was inconclusive, decide whether a larger or better-designed test is warranted. Avoid the trap of running many small, underpowered tests and treating each non-significant result as proof of no effect. Instead, prioritize hypotheses with a plausible mechanism and a realistic chance of moving the primary metric.
Use the results to refine your mental model of the user and the product. A surprising result is an opportunity to generate new hypotheses, but those hypotheses need their own tests. When you plan the next experiment, revisit the framing: is the metric still the right one, is the population still relevant, and has the baseline rate shifted? Consider whether you need a different design, such as a factorial experiment to test multiple changes at once, or a switchback design when user-level randomization is impractical. Keep a written record of hypotheses, designs, and outcomes so you can learn across tests. Over time, a disciplined experimentation practice compounds into better decisions, even though no single test is definitive. The point is not to be right every time but to be less wrong in a measurable way.
7 Frequently asked questions
How long should an A/B test run? Run it long enough to reach the pre-planned sample size and to cover at least one full business cycle, such as a week, to account for day-of-week effects. Stopping early when results look good increases the chance of a false positive. If you need continuous monitoring, use sequential methods designed for that purpose rather than ad hoc peeking. The duration should be set during planning, not after you see the data.
Can I test more than one change at a time? Yes, but the design matters. A factorial experiment can test multiple changes and their interactions efficiently, but it requires more planning and a larger sample. Testing many variants against a single control increases the multiple-comparison problem; adjust your analysis or pre-specify a primary comparison. If you simply change several things at once in one variant, you will not know which change caused the effect. That is fine for a bundled treatment, but it limits what you learn.
What if my results are not statistically significant? That means the data are consistent with no effect or with an effect too small to detect given your sample size. It does not prove the change had no effect. Check the confidence interval to see the range of plausible effects. If the interval includes effects that would matter to you, the test was underpowered or too noisy. Consider whether a larger test, a more sensitive metric, or a different design would be worthwhile. Always interpret results in the context of the setup and sample size.