1 Overview
Imagine you launch an A/B test to see if a new button color increases clicks. After a day, the results show a significant lift. You stop the test and declare victory. But what if that significant result was just a temporary fluctuation? This scenario is common, and it leads to what we call the peeking problem. Peeking refers to the practice of repeatedly checking your test results before the planned end date and making decisions based on those interim looks. While it might seem harmless or even prudent, peeking fundamentally distorts the statistical properties of your test. The p-value you see at any given moment is not a reliable indicator of the true effect. In fact, the more you peek, the more likely you are to find a false positive, even when there is no real difference between your variants. This article explains why peeking creates statistical illusions, how to design tests that avoid this trap, and what to do instead.
When you run an A/B test, you are essentially conducting a hypothesis test. You start with a null hypothesis that there is no difference between the control and treatment. You collect data and calculate a p-value, which represents the probability of observing a result as extreme as yours, assuming the null hypothesis is true. If this p-value is below a pre-defined threshold (often 0.05), you reject the null hypothesis and conclude there is a statistically significant effect. However, this logic assumes you looked at the data only once, after collecting the full sample size. If you look multiple times, the p-value loses its meaning. Each peek is like a separate test, and the chance of at least one false positive across all those tests is much higher than the nominal alpha. This is the multiple comparisons problem in disguise. The result is that your test becomes a statistical illusion: you think you have found a real effect, but you may have just found noise.
2 Framing the question
Before you even think about peeking, you need to frame a clear and specific question. A well-framed question defines the population, the intervention, the comparison, and the outcome. For example, instead of asking 'Does the new design work better?', ask 'For new visitors on the homepage, does changing the call-to-action button from blue to green increase the click-through rate within seven days?' This specificity matters because it guides your test design and prevents ambiguous interpretations later. It also helps you avoid the temptation to peek by keeping you focused on the primary metric and the planned analysis. When you have a clear question, you can pre-register your hypothesis, your primary metric, and your analysis plan. This pre-registration is a safeguard against peeking because it locks in your decisions before you see any data.
A common mistake is to start with a vague idea and then, as data comes in, refine the question to match whatever looks significant. This is a form of p-hacking known as HARKing (Hypothesizing After the Results are Known). It invalidates the test because the hypothesis was not specified in advance. To avoid this, write down your hypothesis and analysis plan before launching the test. Include details such as the expected effect size, the significance level, and the planned sample size. Also, consider what you will do if the result is not significant. Will you iterate on the design, or will you abandon the idea? Having these decisions made upfront reduces the urge to peek and stop early when results look promising. Remember, the goal of an A/B test is not to find a significant result at any cost, but to make a reliable decision based on evidence.
3 Designing the test
Designing a proper A/B test involves several key decisions: randomization, control, and the primary metric. Randomization ensures that the groups are comparable on all factors except the treatment. Without randomization, differences between groups could be due to confounding variables rather than the treatment. The control group provides a baseline against which to measure the treatment effect. The primary metric should be directly tied to your hypothesis and should be measured consistently across both groups. It is also crucial to decide on the duration of the test in advance. The duration should be long enough to capture the full effect and to account for weekly cycles, but not so long that external factors contaminate the results. A common recommendation is to run the test for at least one full business cycle, often one or two weeks, but this depends on your context and traffic.
Another critical design element is the stopping rule. You must specify when the test will end before you start. This could be based on a fixed sample size or a fixed duration. If you use a fixed sample size, you calculate the required number of users per group to detect a meaningful effect with sufficient power. If you use a fixed duration, you still need to ensure that the expected sample size is adequate. The stopping rule should not be based on interim results. If you plan to monitor the test, you can use sequential testing methods that adjust for multiple looks, but these require careful planning and are not the same as ad-hoc peeking. In the absence of such methods, the safest approach is to not look at the results until the test is complete. This might feel uncomfortable, but it is the only way to preserve the validity of your p-values and confidence intervals.
4 Sample size and validity
Sample size is the backbone of a reliable A/B test. Too small a sample, and you risk missing a real effect (low power) or getting a false positive due to random noise. Too large a sample, and you may waste resources and expose users to a suboptimal experience for longer than necessary. Calculating the required sample size involves four inputs: the baseline conversion rate, the minimum detectable effect (MDE) you care about, the significance level (alpha), and the power (1 - beta). Typically, alpha is set at 0.05 and power at 0.80, but these can vary. The MDE is the smallest relative or absolute difference in the metric that you want to be able to detect. The smaller the MDE, the larger the sample size needed. There are many online calculators and statistical formulas to compute this, but the key is to do it before the test starts and stick to it.
Peeking directly threatens validity by inflating the Type I error rate. If you check results repeatedly and stop when you see a significant p-value, the actual probability of a false positive can be much higher than the nominal alpha. For example, with just a few peeks, the false positive rate can easily double or triple. This means your test is no longer valid. Other threats to validity include sample ratio mismatch (when the groups are not equal in size due to a bug or filtering), novelty effects (where users initially react differently to a change simply because it is new), and external events that coincide with the test period. To protect validity, pre-register your analysis plan, use a fixed sample size or a proper sequential design, and monitor for data quality issues without looking at the outcome differences. If you must monitor, look only at aggregate metrics like sample size and traffic sources, not at the conversion rates by group.
5 Running and reading the results
Once the test is running, the hardest part is often to resist the urge to peek. It can be tempting to check the dashboard daily, especially if you are eager to see if your idea is working. But every peek increases the risk of a false positive. If you absolutely must monitor, set up automated alerts for data quality issues, such as a sudden drop in traffic or a sample ratio mismatch, but do not look at the primary metric until the test is scheduled to end. When the test is complete, you can analyze the results according to your pre-registered plan. Calculate the difference in the primary metric between the control and treatment groups, and compute the p-value and confidence interval. If the p-value is below your alpha, you can reject the null hypothesis and conclude there is a statistically significant effect. However, remember that statistical significance does not necessarily mean practical significance. Also consider the confidence interval: if it includes values that are not practically meaningful, the effect may be too small to matter.
If you did peek and stopped early, your results are likely biased. The observed effect size may be exaggerated, and the p-value is not valid. In such cases, the best course of action is to acknowledge the issue and re-run the test with a proper stopping rule. Alternatively, you can use statistical methods that correct for peeking, such as alpha spending functions or group sequential designs, but these must be planned in advance. If you did not plan for them, you cannot apply them post-hoc. Another common mistake is to run multiple tests simultaneously and then pick the one that shows a significant result. This is a form of multiple comparisons and also inflates the false positive rate. To avoid this, correct for multiple comparisons using methods like Bonferroni or false discovery rate, or better yet, limit the number of primary metrics and tests. Always report the results honestly, including any deviations from the plan.
6 What to change next
If your test was compromised by peeking, the first step is to recognize it and commit to better practices. Start by implementing a pre-registration process for all future tests. This can be as simple as a document that outlines the hypothesis, primary metric, sample size, and analysis plan. Share it with your team and get sign-off before launching. Next, set up your testing tool to hide the results until the test is complete. Many A/B testing platforms allow you to schedule the test and only reveal results at the end. If your tool does not have this feature, you can use a separate analytics dashboard that only updates after the test period. Also, educate your team about the dangers of peeking and the importance of statistical rigor. Create a culture where decisions are based on pre-specified analyses, not on gut feelings or early trends.
For tests that are still running, consider whether you need to stop and restart. If you have already peeked and seen a significant result, you cannot trust that result. The ethical and scientific approach is to either continue the test to the planned sample size and then analyze the full data, or to stop and re-run the test from scratch with a proper design. Continuing the test after peeking does not fix the problem because the stopping decision was influenced by the interim data. However, if you have not yet stopped and you commit to not looking again until the end, you can still salvage the test, but the p-value will be slightly conservative? Actually, no: the inflation from peeking already occurred, so the final p-value is not valid. The only clean solution is to re-run. In terms of process improvement, consider adopting sequential testing methods if you need to monitor continuously. These methods allow for valid interim analyses by adjusting the significance threshold at each look. They require planning and statistical expertise, but they can be a good compromise between peeking and waiting.
7 Frequently asked questions
Q: Can I peek just once? A: Even a single peek before the planned end can inflate the false positive rate, though the effect is smaller than with multiple peeks. The safest approach is to not peek at all. If you must look, use a proper sequential design that accounts for the interim analysis. Q: What if I see a significant result early? A: A significant result early in the test is often a false positive due to random fluctuation. The effect size may also be exaggerated. It is best to continue the test to the planned sample size and then analyze the full data. If you stop early, you cannot trust the result. Q: How can I monitor data quality without peeking? A: You can monitor aggregate metrics like sample size, traffic sources, and device breakdowns without looking at the primary metric by group. Set up alerts for anomalies such as sample ratio mismatch or sudden drops in traffic. Q: What is the difference between peeking and sequential testing? A: Peeking is ad-hoc and unplanned, and it invalidates the p-value. Sequential testing is a pre-planned method that adjusts the significance threshold at each interim look to control the overall Type I error rate. It allows for valid early stopping but requires careful design. Q: Does peeking affect confidence intervals? A: Yes, confidence intervals computed from peeking data are too narrow and do not have the nominal coverage. They overstate the precision of the estimate. Q: Can I use Bayesian methods to avoid peeking issues? A: Bayesian methods do not have the same peeking problem in terms of Type I error, but they have their own considerations. They allow for continuous monitoring, but the interpretation of results depends on the prior and the decision rule. It is not a free lunch, but it can be a valid alternative if used correctly.