1 Overview
A/B testing is a cornerstone of optimization, but its power is easily undermined by a common habit: peeking at results before the test concludes. Peeking means checking for statistical significance repeatedly as data accumulates, and stopping the test as soon as a significant result appears. This practice feels efficient, but it dramatically increases the chance of false positives. When you peek, you give randomness multiple opportunities to produce a misleading result. The more often you look, the higher the probability that you will eventually see a difference that is not real. This article explains why waiting for the pre-planned sample size is the best optimization tool you have. It covers the statistical reasoning, the design choices that protect validity, and how to read results responsibly. By the end, you will understand why patience is not just a virtue but a requirement for trustworthy experimentation. The goal is not to discourage monitoring but to emphasize that decisions should be based on a fixed, pre-determined analysis plan. Without that discipline, you risk shipping changes that do not actually improve your metrics, wasting resources and eroding trust in the experimentation process.
2 Framing the question
Before any test begins, you must frame a clear, falsifiable hypothesis. A good hypothesis states the expected effect of a change on a specific metric. For example, you might hypothesize that adding social proof to a landing page will increase sign-up rate. The metric is the sign-up rate, and the test design will compare the current page (control) against the modified page (variant). Framing the question also involves defining what constitutes a meaningful difference. A statistically significant result is not automatically practically significant. You need to decide in advance what minimum detectable effect (MDE) matters for your business. This MDE, along with your baseline conversion rate and desired statistical power, determines the sample size. Without a clear question, you risk chasing noise. The hypothesis should be grounded in prior research or observed user behavior, not guesswork. It should also specify the population and context. For instance, will the test run on all users or a segment? Will it run for a full business cycle to account for weekly variations? Answering these questions upfront prevents post-hoc rationalization and keeps the test focused. Remember, the goal is to learn something reliable, not just to find a winner.
3 Designing the test
Designing a test properly means committing to a fixed sample size and analysis plan before you start. The most common design is a two-arm randomized controlled trial: randomly assign users to either the control or the variant. Randomization ensures that the groups are comparable on average, so any difference in the metric can be attributed to the change. The sample size is calculated using a power analysis. Typically, you aim for 80% power to detect the MDE at a 5% significance level. This calculation requires an estimate of the baseline conversion rate and the MDE. If you do not have a reliable baseline, run a pilot or use historical data. Once the sample size is set, you must not stop the test early based on significance. Stopping early is equivalent to multiple comparisons: each peek is another chance to falsely reject the null hypothesis. To allow for interim analyses without inflating error rates, you can use sequential testing methods or alpha-spending functions. These adjust the significance threshold at each look, preserving the overall Type I error rate. However, these methods require pre-registration and careful implementation. For most teams, the simplest and most robust approach is to wait until the planned sample size is reached. This design choice is the foundation of valid inference.
4 Sample size and validity
Sample size is the linchpin of a valid A/B test. Too small a sample, and you will lack the power to detect even a real effect, leading to false negatives. Too large a sample, and you may detect trivial differences as statistically significant, wasting resources. The right sample size balances these risks based on your MDE and desired power. Validity is also threatened by peeking. When you repeatedly check for significance, the probability of at least one false positive increases. This is known as the multiple comparisons problem. For example, if you peek 10 times at a 5% significance level, the actual false positive rate can be much higher than 5%. The exact inflation depends on the number of looks and the correlation between test statistics, but the principle is clear: more looks mean more error. Other threats to validity include sample ratio mismatch (SRM), where the randomization fails and groups differ in size, and novelty effects, where user behavior changes simply because something is new. External validity is also a concern: results from a limited time frame or specific segment may not generalize. To protect validity, pre-register your analysis plan, including the sample size, primary metric, and stopping rules. Use sequential testing only if you have the expertise and tools. Otherwise, wait. The discipline of waiting is what makes the result trustworthy.
5 Running and reading the results
Once the test is running, resist the urge to check results frequently. Set a calendar reminder for the planned end date, and only then analyze the data. When you do analyze, compute the p-value for your primary metric. If the p-value is less than your pre-determined alpha (e.g., 0.05), you can reject the null hypothesis and conclude there is a statistically significant difference. If not, you fail to reject the null; this does not prove the null is true, only that you lack evidence for an effect. Always report confidence intervals to convey the uncertainty around the effect size. A significant result with a wide confidence interval may still be practically uncertain. Also, check for SRM and other data quality issues before interpreting results. If you find a significant result, consider whether it aligns with your hypothesis and whether the effect size is meaningful. If the result is not significant, do not immediately conclude the change had no effect; it may be that your test was underpowered or the effect was smaller than the MDE. In either case, avoid the temptation to run post-hoc subgroup analyses without adjustments. These can lead to false discoveries. The key is to stick to your pre-registered plan. Reading results responsibly means accepting that not every test will yield a winner, and that is okay. The goal is to learn and iterate based on reliable evidence.
6 What to change next
After a test concludes, use the results to inform your next steps. If the variant won, consider rolling it out to all users, but monitor for long-term effects. Sometimes a short-term lift fades due to novelty or seasonal factors. If the variant lost or was inconclusive, do not discard the learning. Analyze why it might have failed: was the hypothesis wrong, the implementation flawed, or the sample size insufficient? Use these insights to form new hypotheses. For example, if the change did not move the primary metric, it might have affected secondary metrics or segments. However, avoid over-interpreting noise. The best next step is often to run a follow-up test with a refined hypothesis, perhaps targeting a different segment or testing a stronger manipulation. Always calculate the sample size for the new test based on updated assumptions. If you frequently find yourself peeking and stopping early, consider adopting sequential testing methods or increasing your MDE to reduce required sample size. But remember, any method that allows early stopping must be pre-planned and correctly implemented. The overarching principle is to treat each test as a learning opportunity, not a race to significance. By waiting for the planned sample size, you build a foundation of trustworthy evidence that compounds over time. This disciplined approach is the true optimization tool.
7 Frequently asked questions
Why is peeking so bad? Peeking inflates the false positive rate because each look is another chance to see a random fluctuation that crosses the significance threshold. If you peek often enough, you are almost guaranteed to find a false positive, even if the variant has no real effect. This leads to bad decisions and wasted resources. Can I ever stop a test early? Yes, but only if you use a pre-planned sequential testing method that adjusts the significance threshold for interim looks. These methods preserve the overall Type I error rate. Without such adjustments, stopping early is not valid. How do I know my sample size is correct? Sample size is based on your baseline rate, minimum detectable effect, desired power, and significance level. If any of these inputs are wrong, your test may be underpowered or overpowered. It is wise to be conservative and round up. What if I see a significant result early? If you did not pre-plan an interim analysis, ignore it and continue until the planned sample size. The early result could be noise. Only the final analysis at the planned sample size is valid. Does waiting guarantee a correct decision? No, waiting reduces the risk of false positives but does not eliminate all error. There is still a chance of a false positive or false negative, but these risks are controlled at the levels you specified. Waiting simply ensures that your error rates are as intended. Remember, results depend on your setup and sample size; no method can guarantee truth.