1 Overview
You have probably heard the warning: do not peek at your A/B test results before the planned end date, or you will invalidate the test. This advice is common, but it is often misunderstood. The real issue is not that looking at data is inherently wrong. The problem is that many testing tools use statistical methods that assume you will look only once, at a predetermined sample size. When you check results repeatedly with such tools, the chance of seeing a false positive increases. That is a property of the method, not a moral failing on your part.
Modern experimentation platforms offer alternatives. Sequential testing, always-valid inference, and Bayesian methods are designed to allow continuous monitoring while controlling error rates. If your tool supports these approaches, peeking is not a sin; it is a feature. The guilt you feel may be misplaced. Instead of blaming yourself for curiosity, examine whether your tool provides valid inference under repeated looks. The responsibility for valid results lies with the statistical engine, not with your self-control.
This article explains why peeking is problematic under fixed-horizon testing, how some tools solve the problem, and what you should consider when designing and monitoring experiments. The goal is to help you run tests that are both practical and statistically sound, without unnecessary guilt.
2 Framing the question
The question is not simply whether peeking is bad. The better question is: under what conditions does monitoring a test at arbitrary times lead to misleading conclusions? To answer that, we need to separate the act of looking from the statistical framework used to analyze the data. In a classic fixed-horizon test, you decide the sample size in advance, collect all the data, and then compute a single p-value. That p-value is valid only if you do not use it to decide when to stop. If you check the p-value after every new observation and stop when it crosses a threshold, you are effectively performing many tests. Each look gives the p-value another chance to fall below the threshold by random chance, even if the null hypothesis is true. This inflates the false positive rate.
However, if your tool uses a method that adjusts for multiple looks, such as group sequential boundaries or always-valid confidence sequences, then peeking does not automatically invalidate the result. The tool's error guarantees hold even if you monitor continuously. So the framing should shift from blaming the user to evaluating the tool. Does your platform provide valid inference under continuous monitoring? If not, you either need to resist peeking or switch to a tool that does. The choice depends on your goals, resources, and the cost of false positives in your context.
3 Designing the test
When designing an A/B test, you must decide upfront how you will monitor and analyze the data. This decision affects the sample size, the duration, and the statistical validity. If you plan to use a fixed-horizon test, you should commit to a sample size and avoid looking at the results until that sample is reached. That is not always practical, especially in business settings where decisions cannot wait. In that case, consider a design that accommodates peeking. Sequential designs, for example, allow for interim analyses at planned intervals while controlling the overall false positive rate. Always-valid methods go further, permitting continuous monitoring without inflating error rates.
The design also includes the hypothesis, the primary metric, and the randomization unit. You should specify these before launching. The hypothesis should be a clear statement about the expected effect of the variant compared to the control. The metric should be sensitive to the change and aligned with business goals. The randomization unit (e.g., user, session) must be consistent with the analysis. If you plan to peek, choose a tool that supports your monitoring schedule. Some tools let you set up sequential tests with alpha spending functions, which allocate the total false positive rate across looks. Others use Bayesian posterior probabilities, which can be updated continuously. The key is to match the design to the tool's capabilities, not to fight the tool's limitations.
4 Sample size and validity
Sample size is a critical factor in any experiment. It determines the power to detect an effect and the precision of your estimates. When you peek, you may be tempted to stop early if you see a significant result. But under fixed-horizon testing, stopping early based on a p-value threshold biases the effect estimate upward and increases the false positive rate. The sample size you need depends on the minimum detectable effect, the baseline conversion rate, the desired power, and the significance level. These calculations assume a single analysis at the end. If you plan to monitor continuously, you need a method that adjusts for that, which often requires a larger sample size to maintain the same power.
Validity threats in A/B testing include peeking without adjustment, multiple comparisons, and violations of the assumptions of the statistical test. For example, if you test many variants against a single control, you need to correct for multiple comparisons. If you segment the results after the fact, you increase the risk of false discoveries. The tool you use should help you manage these threats. Some tools automatically apply corrections, while others leave it to the user. Always check the documentation to understand what the tool does and does not guarantee. Remember that no statistical method can rescue a poorly designed experiment. The sample size must be adequate for the effect you hope to detect, and the validity of your conclusions depends on the integrity of the design and the appropriateness of the analysis.
5 Running and reading the results
Once the test is running, you will be tempted to check the results frequently. That is natural. The question is what you do with what you see. If your tool uses fixed-horizon p-values, you should resist the urge to stop early based on a significant result. Instead, wait until the predetermined sample size is reached. If you must monitor, use a tool that provides valid sequential inference. When reading the results, look beyond the p-value. Consider the effect size, the confidence interval, and the practical significance. A statistically significant result may not be business-relevant. Conversely, a non-significant result does not prove the null hypothesis; it may simply mean the test was underpowered.
Pay attention to the tool's error guarantees. If the tool claims to support peeking, verify that it uses a method like sequential testing or always-valid inference. Read the documentation to understand what the error rates mean under continuous monitoring. For example, an always-valid confidence interval will be wider than a fixed-horizon interval, reflecting the cost of peeking. That is not a flaw; it is the price of flexibility. When you see a result, ask yourself: does this tool's method control the false positive rate at the level I chose? If yes, you can trust the result as much as any other. If no, you should treat early looks as exploratory and wait for the final analysis. In either case, avoid making decisions based on a single metric or a short-term fluctuation.
6 What to change next
If you discover that your current tool does not support valid peeking, you have a few options. First, you can change your process: commit to a fixed sample size and avoid looking at the results until the end. This requires discipline and may not be feasible in fast-paced environments. Second, you can switch to a tool that offers sequential or always-valid methods. Many modern experimentation platforms include these features. When evaluating tools, ask specific questions: Does it adjust for multiple looks? What error guarantees does it provide? How does it handle continuous monitoring? The answers will help you choose a tool that aligns with your monitoring needs.
Third, you can adjust your design to include planned interim analyses. For example, you might specify two or three looks at predetermined sample sizes, using an alpha spending function to control the overall false positive rate. This approach is more flexible than a single final analysis but less flexible than continuous monitoring. It also requires careful planning. Finally, consider the cost of false positives in your context. If the cost is low, you might tolerate a higher false positive rate. If the cost is high, you should invest in a tool that provides strong error control. Regardless of your choice, document your monitoring plan and stick to it. The goal is not to eliminate peeking but to ensure that your statistical inferences remain valid. By aligning your behavior with the tool's assumptions, you can make better decisions and stop feeling guilty about checking your results.
7 Frequently asked questions
Is peeking always wrong? No. Peeking is problematic only when the statistical method assumes a single analysis. If the tool uses a method that adjusts for multiple looks, peeking is fine. The key is to know what your tool does.
How can I tell if my tool supports valid peeking? Check the documentation for terms like sequential testing, always-valid inference, alpha spending, or group sequential boundaries. If the tool uses fixed-horizon p-values without adjustment, it does not support valid peeking. Some tools may offer both modes; make sure you select the appropriate one.
What if I already peeked and stopped early? If you used a fixed-horizon test and stopped when the p-value crossed a threshold, your result may be biased. You can report it as exploratory, but for a confirmatory test, you should run a new experiment with a proper design. Alternatively, if the tool supports sequential methods, you can re-analyze the data using those methods, but this may not be valid if the stopping rule was not pre-specified.
Does peeking affect Bayesian methods? Bayesian methods do not have the same false positive rate concept, but they can still be affected by optional stopping if you use a decision rule based on posterior probabilities. However, always-valid Bayesian methods exist. The key is to understand the properties of the specific method.
How do I choose the right sample size if I plan to peek? You need to use a sample size calculation that accounts for the monitoring plan. For sequential designs, software or the tool itself can help. In general, peeking requires a larger sample size to maintain the same power and error control. Always consult the tool's guidance or a statistician.