1 Overview
A/B testing is a staple of modern optimization, but its power comes with a catch: the statistical guarantees only hold if you follow the rules. One of the most common mistakes is peeking at the results before the test reaches its planned sample size. Peeking means checking the data repeatedly and making decisions based on interim results. This practice inflates the false positive rate, meaning you are more likely to declare a winner when there is no real difference. Fixed sample sizes are your friend because they preserve the validity of the test. By deciding in advance how many users or sessions you need, you avoid the temptation to stop early when results look promising or to continue when they look disappointing. This article explains why peeking is problematic, how to design a test with a fixed sample size, and how to interpret results correctly. We will cover the statistical reasoning without inventing numbers, and we will emphasize that every test depends on its specific setup and sample size. The goal is to help you run experiments that yield trustworthy insights, not just quick answers.
2 Framing the question
Before you run any test, you need a clear question. A good experiment starts with a hypothesis that states what you expect to change and why. For example, you might hypothesize that simplifying a signup form will increase the conversion rate. The hypothesis should be specific and testable. Next, choose a primary metric that directly reflects the hypothesis. This metric should be sensitive to the change and aligned with business goals. Avoid using multiple primary metrics, as that increases the chance of false positives. Also consider secondary metrics for guardrails, but keep the primary metric as the sole basis for your decision. The framing stage also involves defining the population and the randomization unit. Will you randomize users, sessions, or something else? The choice affects the analysis and the sample size. Finally, specify the expected effect size. This is the minimum improvement that would be practically meaningful. Without a clear effect size, you cannot calculate a sample size. Remember, the question should be framed so that the answer can be a simple yes or no: did the change cause a difference in the metric? This clarity will guide the rest of the process.
3 Designing the test
Designing a test involves several decisions that must be made before data collection begins. First, determine the number of variants. A classic A/B test has two: control and treatment. You can include more, but each additional variant increases the sample size required and the complexity of analysis. Second, decide on the randomization method. Randomly assigning users to variants ensures that the groups are comparable on average, aside from the change being tested. Third, calculate the required sample size. This calculation uses the baseline metric, the minimum detectable effect, the desired statistical power (often 80% or 90%), and the significance level (often 5%). The sample size also depends on the variance of the metric. For rates, variance is related to the baseline rate. For continuous metrics, you need an estimate of the standard deviation. Fourth, set a fixed duration or a fixed sample size. A fixed sample size is preferable because it is tied directly to the statistical power. If you stop when you reach the sample size, you avoid peeking. Fifth, pre-register your analysis plan, including the primary metric, the statistical test, and any segmentation. This prevents post-hoc rationalization. Finally, ensure that the test runs long enough to capture weekly cycles and avoid novelty effects. All these steps require careful thought, and they depend on the specific context of your product and audience.
4 Sample size and validity
Sample size is the cornerstone of a valid experiment. If you collect too few observations, you may lack the power to detect a real effect, leading to a false negative. If you collect more than needed, you waste resources and expose more users to a potentially inferior variant. More importantly, the sample size must be fixed in advance. Peeking at the data and stopping when the result is significant is a form of optional stopping. This practice dramatically increases the Type I error rate, meaning you are more likely to see a false positive. The reason is that each peek is like a separate test; with multiple tests, the chance of at least one false positive accumulates. Statistical theory assumes that you analyze the data once, after the planned sample size is reached. If you must monitor the test, use sequential analysis methods designed for that purpose, but these require different sample size calculations. Validity is also threatened by other factors: sample ratio mismatch (when the actual allocation deviates from the planned ratio), external events that affect one variant disproportionately, and novelty effects where early behavior differs from long-term behavior. To protect validity, stick to the plan, check for data quality issues, and avoid making decisions based on partial data. Remember, results depend on the setup and sample size; a significant result in a small sample may not replicate.
5 Running and reading the results
Once the test is running, your job is to monitor for technical issues, not to check the outcome. Ensure that the randomization is working, that data is being logged correctly, and that no bugs affect one variant. You can check sample ratio mismatch, but do not look at the primary metric. When the test reaches the predetermined sample size, you can analyze the results. Start by computing the primary metric for each variant. Then perform the pre-specified statistical test, such as a t-test for means or a chi-square test for proportions. Report the effect size (the difference between variants) along with a confidence interval. The confidence interval gives a range of plausible values for the true effect. Also report the p-value, but remember that it is not the probability that the null hypothesis is true. A p-value below the significance level indicates that the observed difference is unlikely under the null hypothesis. However, statistical significance does not imply practical significance. Consider whether the effect size is large enough to matter for your business. Also check secondary metrics and guardrails to ensure no unintended harm. If the result is not significant, you cannot conclude that there is no effect; you simply lack evidence. Finally, consider the context: the result is specific to the population, time, and setup of the test. Avoid overgeneralizing.
6 What to change next
After analyzing the results, you need to decide what to do next. If the treatment is a clear winner with a practically significant effect and no guardrail issues, you might roll it out to all users. But even then, consider whether the effect will persist over time. Novelty effects can fade, so a follow-up test or a gradual rollout can help confirm. If the result is not significant, you have several options. You could accept the null and keep the control, or you could run a new test with a larger sample size if you believe the effect is real but small. However, be cautious: if you repeatedly test the same hypothesis, you increase the risk of false positives. Instead, use the results to refine your hypothesis. Maybe the change was too small, or the metric was not sensitive enough. You could also test a different variation or target a different segment. If the treatment performed worse, you learned something valuable: the change was harmful. Document the learnings and share them with your team. Every experiment, whether successful or not, contributes to institutional knowledge. Finally, update your prioritization based on what you learned. The next test should build on these insights. Remember that optimization is an iterative process, and each test is a step toward a better understanding of your users.
7 Frequently asked questions
Q: Can I peek at the results if I use a statistical correction? A: Yes, but you must use methods designed for sequential testing, such as alpha spending functions or group sequential designs. These require planning and different sample size calculations. Simply applying a Bonferroni correction to each peek is not sufficient because the peeks are not independent. Q: What if my test doesn't reach the planned sample size due to low traffic? A: You may need to extend the duration, but be aware that external factors could change. If you cannot reach the sample size, your test may be underpowered, and you should interpret results with caution. Q: How do I know if my sample size is large enough? A: It depends on the effect size you want to detect. Use a power analysis. If you want to detect a small effect, you need a larger sample. Q: Can I stop early if the results are significant? A: No, unless you planned a sequential design. Stopping early inflates false positives. Q: What is a p-value? A: It is the probability of observing data as extreme as yours, assuming the null hypothesis is true. It is not the probability that the null is true. Q: Why do results depend on setup? A: Because factors like user behavior, seasonality, and implementation details vary. A result in one context may not replicate in another. Always consider the specific conditions of your test.