1 Overview
A/B tests are often treated as simple scoreboards: run two versions, wait for a winner, and ship it. In practice, the result you see at the end of a test can be misleading for several reasons that have nothing to do with the change being tested. The most common culprits are peeking at results before the planned sample size is reached, running a test with too little power to detect a realistic effect, and confusing statistical significance with practical significance. Each of these problems can make a losing change look like a winner, or hide a genuinely useful improvement.
This article explains why those problems happen and how to avoid them. It focuses on the logic of experimentation rather than on any particular tool or platform. The core message is that an A/B test is a measurement procedure with assumptions, and those assumptions determine whether the number you read at the end means what you think it means. The exact impact of any change depends on your setup, your audience, and your sample size, so no generic rule can tell you what your result will be. What you can control is the discipline of the process.
By the end, you should be able to frame a testable hypothesis, choose a primary metric, design a clean comparison, plan a sample size, and read the result without fooling yourself. You will also see why the most important question is not whether the result is significant, but whether the effect is large enough to matter for your product.
2 Framing the question
A useful A/B test starts with a hypothesis that is specific, directional, and tied to a measurable behavior. Instead of saying "the new design will be better," state what you expect to change and for whom. For example, you might hypothesize that simplifying the checkout form will increase the proportion of users who complete a purchase. The hypothesis should name the change, the expected direction of the effect, and the primary metric that will capture it. Vague hypotheses lead to vague metrics and results that can be interpreted in too many ways.
Once the hypothesis is clear, choose one primary metric. The primary metric is the single number that will decide whether the test succeeds. Secondary metrics can provide context and guard against unintended harm, but they should not be used to declare victory after the fact. For instance, if the primary metric is conversion rate, a secondary metric might be average order value or the rate of customer support contacts. Deciding these in advance prevents you from cherry-picking a favorable metric later.
Finally, define what a meaningful effect looks like. Statistical significance tells you that an observed difference is unlikely to be due to chance alone under the test's assumptions, but it does not tell you whether the difference is large enough to justify the cost of the change. Before running the test, write down the smallest improvement that would make the change worth shipping. That number, often called the minimum detectable effect or the minimum effect of interest, will guide your sample-size planning and your final decision.
3 Designing the test
The design of an A/B test determines what you can conclude from it. At minimum, you need a control group that experiences the current version and a treatment group that experiences the change. Assignment to groups should be random and independent of user characteristics, so that the groups are comparable on average. If assignment is biased, for example by giving the new version only to returning users, any difference you see may reflect the groups rather than the change. Randomization is what makes the comparison fair.
You also need to decide what unit of randomization to use. Users are the most common unit, but sessions, accounts, or geographic regions can be appropriate depending on the question. The unit matters because it affects how observations are counted and how much independence exists between them. If the same user appears in both groups, or if users influence one another, the standard statistical tests may not apply. In such cases, you may need a different design, such as a cluster-randomized test or a switchback test.
Another design choice is whether to run the test as a fixed-horizon experiment or use a sequential method. A fixed-horizon test has a planned sample size and a planned analysis time. A sequential method allows valid interim looks but requires a pre-specified stopping rule. The danger arises when you look at a fixed-horizon test repeatedly and stop when it looks good, which inflates the false-positive rate. Deciding the analysis plan before the test begins is the simplest way to protect the result's validity.
4 Sample size and validity
Sample size is not a formality; it is the main determinant of whether your test can detect the effect you care about. The required sample size depends on the baseline rate of the metric, the smallest effect you want to detect, the significance level you choose, and the power you want. Power is the probability of detecting a real effect of a given size. A low-powered test is like a telescope with a small lens: it may miss real signals and produce noisy estimates. Because these inputs vary by situation, you cannot assume a single sample size works for every test. You must calculate it for your specific metric and effect size.
Validity is threatened by several common practices. Peeking at results before the planned sample size and stopping when the result looks significant is one of the most damaging. It increases the chance of a false positive because you are effectively running many tests and reporting the most favorable one. Multiple comparisons have a similar effect: testing many metrics or many segments and highlighting the one that crosses the significance threshold will produce misleading winners. Sample ratio mismatch, where the groups end up with unequal sizes, can also indicate a problem with the randomization or tracking.
Other threats include novelty effects, where users initially react differently to a change simply because it is new, and external events that affect one group more than another. To protect validity, pre-register your hypothesis, primary metric, sample size, and analysis plan. Monitor for technical issues, but avoid making decisions based on unplanned interim looks. If you must stop early, use a method designed for sequential testing rather than a standard fixed-horizon test.
5 Running and reading the results
When the test reaches its planned sample size, analyze the primary metric first. Compute the difference between groups and a confidence interval around that difference. A confidence interval gives a range of plausible values for the true effect, which is more informative than a single significance test. If the interval includes zero, the data are consistent with no effect. If it excludes zero, the effect is statistically significant at the chosen level. But significance alone is not enough: check whether the interval excludes your minimum effect of interest. A statistically significant but tiny effect may not be worth implementing.
Also inspect the secondary metrics and guardrail metrics. A change that improves the primary metric but harms a guardrail, such as revenue or retention, may not be a net win. Look at the sample ratio to confirm the groups are close to the intended split. If the ratio is off, treat the result with caution. Finally, consider the practical significance: does the observed effect, with its uncertainty, justify the cost and risk of the change? This is a judgment call that depends on your product and strategy, not on a p-value alone.
Remember that the result is an estimate, not a certainty. Even a well-run test can produce a false positive or false negative by chance. Replicating the result in a follow-up test, or testing the change in a different context, can increase confidence. Avoid the temptation to declare a winner based on a subgroup that looks promising unless that subgroup was pre-specified. The goal is to make a decision that is robust to the noise inherent in any experiment.
6 What to change next
If the test shows a clear, practically meaningful improvement in the primary metric without harming guardrails, the next step is to ship the change and monitor it after launch. Post-launch monitoring helps catch effects that a short experiment may miss, such as long-term user adaptation or interactions with other features. If the result is inconclusive, do not simply rerun the same test unchanged. Instead, revisit your hypothesis and design. Perhaps the effect is smaller than you assumed, the metric is too noisy, or the sample size was too small. Use the confidence interval to understand what effect sizes are still plausible.
If the test shows no effect or a negative effect, that is still valuable information. It may mean the change does not work, or that it works only for a subset of users, or that the test was underpowered. Before concluding, check for validity threats such as peeking, sample ratio mismatch, or external events. If the test was valid and the effect is negligible, consider whether a different change might address the underlying user need. Experimentation is iterative; a failed test often points to a better question.
In all cases, document what you learned and update your priors. Record the hypothesis, design, sample size, result, and decision. This practice improves future tests and prevents the same mistakes from recurring. The most important change you can make is to treat each test as a learning opportunity rather than a one-off verdict. Over time, a disciplined experimentation process will produce more reliable insights than any single test.
7 Frequently asked questions
Why is peeking at A/B test results a problem? Peeking means checking results before the planned sample size and making a decision based on what you see. If you stop when the result looks significant, you are more likely to stop on a random high point, which inflates the false-positive rate. The fix is to pre-specify the sample size and analysis time, or to use a sequential testing method with a valid stopping rule. If you need to monitor, use methods designed for that purpose.
What is the difference between statistical significance and practical significance? Statistical significance means the observed difference is unlikely under the null hypothesis, given the test's assumptions. Practical significance means the difference is large enough to matter for your business or product. A result can be statistically significant but too small to justify the cost of a change. Always consider the confidence interval and your minimum effect of interest, not just the p-value.
How do I know if my test had enough power? Power depends on your baseline metric, the effect size you want to detect, your significance level, and your sample size. You can calculate the required sample size before the test using these inputs. If your test was underpowered, a non-significant result may simply mean you could not detect the effect. In that case, the confidence interval will be wide, and you should not conclude that the change had no effect. Instead, consider a larger sample or a more sensitive metric.