1 Overview
A/B testing is often presented as a straightforward way to compare two versions of something and let the data decide. In practice, the decision rule matters as much as the data. If you check your test every day and stop as soon as the result looks significant, you are not running the test you think you are running. You are running a procedure that has a much higher chance of producing a false positive than the nominal significance level suggests. This article explains why peeking is a statistical problem, what it does to error rates, and how to design and read tests so that the result means what you think it means.
The core issue is not that looking at data is bad. The issue is that making a decision based on a threshold at a moment you choose changes the probability of crossing that threshold. A fixed-horizon test assumes one look at the data after a planned sample size. Continuous monitoring assumes many looks, and each look is another chance to cross the threshold by chance alone. The result is that the same p-value can mean very different things depending on how many times you looked and whether you stopped when it was convenient.
This article is about test design, not about any specific tool or platform. The principles apply whether you are testing a button color, an onboarding flow, or a pricing page. The goal is to help you avoid the most common way an A/B test becomes a statistical lie: by treating a peeking result as if it came from a fixed-horizon test.
2 Framing the question
Before you run a test, you need a question that can be answered with a binary decision. A good question is specific: does changing the call-to-action text from A to B increase the proportion of users who complete a desired action? The hypothesis is a statement about the difference between the two versions. The metric is the thing you measure to evaluate that statement. The design is how you assign users, how long you run, and how you decide. If any of these are vague, the test will be vague too.
A common mistake is to frame the question as 'which version is better?' without defining better. Better on what metric, for which users, over what time period? Without a pre-specified primary metric, you may be tempted to look at many metrics and pick the one that looks significant. That is a form of peeking across metrics, and it has the same effect as peeking across time: it inflates the chance of a false positive. The question should be narrow enough that you can state in advance what result would make you choose A and what result would make you choose B.
Another part of framing is deciding what you will do with the result. If the test is only for learning, you may not need a strict decision rule. But if you plan to ship the winner, you need a rule that controls error rates. That means deciding in advance whether you will use a fixed-horizon test, a sequential test, or another method that accounts for multiple looks. The framing stage is where you choose the rules of the game, and those rules determine what the final number means.
3 Designing the test
The design of an A/B test includes assignment, duration, and the decision rule. Assignment should be random and independent, so that each user has a known chance of seeing each version. Randomization helps ensure that differences between groups are due to the change, not to pre-existing differences. If assignment is biased, no amount of statistical analysis can fully repair the result. The design also includes the unit of analysis: users, sessions, or something else. The unit should match the metric and the decision you want to make.
Duration and sample size are linked. You need enough data to detect an effect of the size you care about, if that effect is real. The required sample size depends on the baseline rate, the minimum detectable effect, the desired significance level, and the desired power. These are choices you make before the test. If you plan to check the test repeatedly and stop early, you are changing the design mid-stream. That is not necessarily forbidden, but it requires a method that accounts for it, such as a sequential test or an alpha-spending approach. Simply peeking and stopping when p is below a threshold is not a valid design.
The decision rule should be written down before the test starts. It should say what metric you will use, what threshold you will apply, and when you will look at the data. If you plan to look multiple times, the rule should specify how the threshold changes at each look. Without a pre-specified rule, any result you get is hard to interpret because you cannot separate the signal from the noise introduced by your own choices. The design stage is where you earn the right to trust the final number.
4 Sample size and validity
Sample size is not just a technical detail; it is part of the meaning of the result. A small sample can produce a large difference by chance, and a large sample can detect a tiny difference that may not be practically meaningful. The right sample size depends on the effect you want to detect and the risk you are willing to take. If you run a test with too few users, you may fail to detect a real effect, or you may detect a spurious one. If you run with too many, you may spend resources to detect a difference that does not matter for your decision.
Validity is threatened by anything that makes the observed result not reflect the true effect. Optional stopping is a major threat. If you stop when the result looks good, you are more likely to stop on a random high point. This inflates the false-positive rate and exaggerates the effect size. Multiple comparisons are another threat: if you test many metrics or many segments, some will look significant by chance. Attrition and instrumentation errors can also threaten validity if they differ between groups. The common thread is that validity depends on the rules you set and follow, not just on the data you collect.
A valid test has a pre-specified primary metric, a pre-specified sample size or stopping rule, and a pre-specified analysis plan. If you deviate from these, you should report the deviation and interpret the result with caution. The result of a test is not just a p-value; it is a statement about what would happen if you repeated the procedure. If the procedure includes peeking and stopping, the statement is different from one about a fixed-horizon test. That is why sample size and validity are inseparable from the design.
5 Running and reading the results
When the test is running, the best practice is to avoid making decisions based on interim looks unless you have a pre-specified sequential design. It is fine to monitor for technical problems, such as broken tracking or uneven assignment, but you should not use interim effect estimates to decide whether to stop. If you do look, remember that the numbers will fluctuate. Early data is especially noisy, and a large early difference often shrinks as more data arrives. That pattern is not a reason to stop; it is a reason to wait for the planned sample size.
When you read the final result, start with the design. Was the sample size reached? Was the primary metric analyzed as planned? Were there any deviations? If the answer is yes to the first two and no to the third, the result can be interpreted using the pre-specified rule. If there were deviations, the result is exploratory. A p-value from an exploratory analysis does not have the same meaning as one from a confirmatory test. It may suggest a hypothesis for a future test, but it should not be used to ship a change with confidence.
Also consider the effect size and its uncertainty, not just whether the result crossed a threshold. A confidence interval gives a range of plausible values and shows how precise the estimate is. A result that is statistically significant but has a wide interval may be consistent with both a tiny effect and a large one. The decision should depend on whether the plausible effects are large enough to matter. This is why reading results is not just about significance; it is about the size and precision of the estimate in the context of your decision.
6 What to change next
If you have been peeking and stopping, the first change is to write down a decision rule before the next test. Decide on the primary metric, the minimum detectable effect, the significance level, and the sample size. If you need to look at the data before the planned end, use a method designed for sequential analysis. Many platforms offer sequential tests or always-valid inference, but you should understand what they assume and how they change the interpretation. The goal is to align your monitoring behavior with your analysis method.
The second change is to treat the test as a procedure, not a single number. Document the procedure, including any deviations, and interpret the result accordingly. If you ran a fixed-horizon test, do not stop early. If you ran a sequential test, follow its stopping rule. If you looked at many metrics, pre-specify one primary metric and treat the rest as secondary or exploratory. This discipline makes the result more trustworthy and makes it easier to learn from both successes and failures.
Finally, build a habit of learning from the test after it ends. What did you learn about the users, the product, or the metric? Even a null result can be informative if the test was well designed. The next test can build on that learning. Over time, a culture of pre-specified tests and careful reading will produce more reliable decisions than a culture of peeking and hoping. The change is not just statistical; it is about how you make decisions with data.
7 Frequently asked questions
Is it ever okay to peek at an A/B test? It is okay to monitor for technical issues, but using interim results to decide whether to stop or continue changes the statistical properties of the test. If you want to make decisions at multiple looks, use a sequential design that accounts for it. Otherwise, wait until the planned sample size.
What if I see a significant result early? A significant result early in a fixed-horizon test is not a license to stop. Early significance can be a random fluctuation. If you stop, you are more likely to be stopping on a high point, which inflates the false-positive rate and exaggerates the effect. The planned sample size exists to protect against this.
Does peeking always invalidate the test? It depends on how you peek and what you do with the information. If you look but do not change your decision rule, the impact may be limited, but the interpretation is still affected because you have seen the data. If you stop or extend based on what you see, the test is no longer a fixed-horizon test. The safest approach is to pre-specify the analysis and follow it.
How do I choose a sample size? Base it on the smallest effect you care about, the baseline rate, the desired significance level, and the desired power. These choices depend on your context and the cost of errors. There is no universal number. The key is to decide before the test and to avoid changing the plan based on interim results.