1 Overview
A/B testing is a powerful method for making data-driven decisions, but its results are easily distorted by common pitfalls. One of the most pervasive mistakes is peeking: repeatedly checking the results before the test is scheduled to end and making decisions based on early data. This article explains why peeking leads to false positives and how to run tests that you can trust.
The core issue is that statistical tests assume you will analyze the data only once, after reaching a predetermined sample size. When you peek, you give yourself multiple chances to see a significant result by chance alone. This inflates the false positive rate, meaning you may declare a winner when there is no real difference. The solution is not to avoid looking at data entirely, but to structure your analysis to control error rates.
We will cover the entire process: framing a clear question, designing a robust test, determining sample size, running the test without peeking, interpreting results correctly, and deciding what to do next. By following these guidelines, you can avoid fooling yourself and make more reliable decisions.
2 Framing the question
Every A/B test should start with a clear, specific hypothesis. A hypothesis is a testable statement about how a change will affect a metric. For example, 'Adding a testimonial section to the landing page will increase the conversion rate.' This hypothesis guides the entire experiment: it tells you what to change, what to measure, and what outcome you expect. Without a clear hypothesis, you risk chasing noise or measuring irrelevant metrics.
The hypothesis should be grounded in prior research or observation. It could come from user feedback, analytics data, or a theoretical framework. Importantly, it should be falsifiable: you must be able to imagine a result that would disprove it. This keeps the test focused and prevents post-hoc rationalization of ambiguous results.
Once you have a hypothesis, define the primary metric that will determine success. This metric should be directly linked to the hypothesis and sensitive to the change. For instance, if the hypothesis is about increasing sign-ups, the primary metric is the sign-up rate. Secondary metrics can provide additional context, but they should not replace the primary metric in decision-making. Clearly defining the metric upfront avoids cherry-picking later.
3 Designing the test
The design of an A/B test determines how users are assigned to groups and how the change is implemented. The gold standard is a randomized controlled experiment: users are randomly assigned to either the control group (which sees the current version) or the treatment group (which sees the new version). Randomization ensures that the groups are comparable on all factors except the change, so any difference in the metric can be attributed to the change.
It is crucial to decide on the unit of randomization. In most web experiments, the unit is the user, but it could also be a session or a page view. The choice depends on the context and the metric. For example, if the metric is user-level conversion, randomizing by user is appropriate. If the metric is click-through rate on a page, randomizing by page view might be sufficient, but it can introduce dependencies if the same user sees multiple pages.
Another key design decision is whether to run an A/B test or a more complex design like A/B/C or factorial. For simplicity, start with A/B. Also, ensure that the change is the only difference between groups. Avoid making multiple changes at once, as it becomes impossible to attribute the effect to a specific element. Finally, plan the duration of the test in advance, based on sample size calculations, and stick to it.
4 Sample size and validity
Determining the required sample size is essential for a valid test. The sample size depends on four factors: the baseline conversion rate, the minimum detectable effect (the smallest change you care about), the significance level (typically 0.05), and the power (typically 0.80). These factors determine how many users you need to reliably detect an effect if it exists. Running a test with too few users increases the risk of false negatives (missing a real effect) and false positives (due to random noise).
Validity is threatened by several factors, with peeking being a major one. Peeking means analyzing results before the planned sample size is reached. Each peek is a statistical test, and multiple tests increase the chance of a false positive. If you must monitor the test, use sequential testing methods that adjust for multiple looks, but these require careful implementation. Other threats include sample ratio mismatch (when the groups have unequal sizes due to a bug), novelty effects (users react to newness rather than the change itself), and external events (like holidays or news) that affect the metric.
To protect validity, pre-register your analysis plan: specify the primary metric, sample size, and duration. Avoid changing the metric or stopping rules mid-test. Also, ensure that your instrumentation is accurate and that you are not inadvertently filtering out certain users. By addressing these threats, you increase confidence in your results.
5 Running and reading the results
Once the test is launched, resist the urge to check results frequently. Wait until the predetermined sample size is reached. When the test concludes, analyze the data using the pre-specified statistical test. For a simple A/B test comparing conversion rates, a two-proportion z-test or a chi-square test is common. The output includes a p-value, which indicates the probability of observing a difference as extreme as the one seen if there were truly no effect. A small p-value (typically less than 0.05) suggests statistical significance.
However, statistical significance does not imply practical significance. Always consider the effect size and its confidence interval. The confidence interval provides a range of plausible values for the true effect. If the interval includes zero, the effect is not statistically significant. If it is entirely above zero, the treatment is likely better. But even if significant, ask whether the improvement is large enough to matter for the business. A tiny lift might not justify the cost of implementing the change.
Also, check for validity threats. Did the sample ratio match the intended split? Were there any anomalies during the test period? If the test was compromised, the results may be unreliable. Finally, avoid overinterpreting secondary metrics or subgroup analyses, as these are prone to false positives. Stick to the primary metric for the decision.
6 What to change next
After interpreting the results, decide on the next steps. If the treatment is a clear winner with a meaningful effect and no validity issues, consider rolling it out to all users. But before full deployment, you might run a follow-up test to confirm the effect in a different context or to optimize the implementation. If the treatment is not better, you can either abandon the change or iterate on it based on insights from the test. For example, if the change did not improve conversion, maybe it increased engagement but not enough to matter; you could refine the hypothesis and test a modified version.
If the results are inconclusive (e.g., the confidence interval includes both positive and negative effects), you may need to run a larger test or refine the metric. Inconclusive results often stem from insufficient sample size or a noisy metric. Use the learnings to design a better experiment. Also, document the test and its outcomes to build institutional knowledge and avoid repeating mistakes.
Finally, remember that A/B testing is an iterative process. Each test provides information that can inform future hypotheses. By avoiding peeking and following rigorous methods, you can build a culture of experimentation that drives continuous improvement. Always prioritize learning over winning, and be willing to accept negative results as valuable insights.
7 Frequently asked questions
Q: Can I peek at the results if I use a sequential test? A: Sequential testing methods allow for valid interim analyses by adjusting the significance threshold at each look. However, these methods require careful planning and are not a free pass to check results constantly. If you use them, pre-specify the number and timing of looks.
Q: What if my test doesn't reach the required sample size? A: If you stop early, the results are likely unreliable due to low power and inflated false positive rate. It's better to extend the test or run a new one with a larger sample. If you must stop, report the results as inconclusive.
Q: How do I handle multiple metrics? A: Designate one primary metric for the decision. Secondary metrics can be explored but should be interpreted with caution, as multiple comparisons increase the chance of false positives. Consider corrections like Bonferroni if you must test multiple metrics.
Q: What is a novelty effect and how do I avoid it? A: A novelty effect occurs when users initially engage more with a new design simply because it's new. To mitigate, run the test for at least one full business cycle (e.g., a week) and check if the effect persists over time. If the effect diminishes, it may be novelty.
Q: Can I run an A/B test with low traffic? A: Yes, but you may need to run it longer to reach sufficient sample size. Alternatively, focus on changes with larger expected effects or use more sensitive metrics. Be realistic about what you can detect.