1 Overview
In A/B testing, the temptation to check results early and often is strong. Marketers, product managers, and analysts frequently monitor dashboards, hoping to see a winner emerge quickly. This behavior, known as peeking, can dramatically inflate false positive rates and lead to poor decisions. This case-study guide explores a hypothetical scenario where a team tests a new call-to-action button on their website. The goal is to illustrate how peeking undermines statistical validity and how a disciplined approach yields more reliable insights.
The experiment compares the current button (control) against a redesigned version (treatment). The primary metric is click-through rate. The team plans to run the test for two weeks, but after three days, the treatment shows a significant lift. Excited, they consider stopping the test and implementing the change. However, this early peek introduces bias: the result may be due to random fluctuation rather than a true effect. By understanding the statistical principles behind sequential testing and fixed-horizon tests, the team can avoid this trap.
This article walks through the entire process, from framing the question to interpreting results, emphasizing that outcomes depend heavily on setup and sample size. It underscores that no single test guarantees truth; rather, careful design and analysis improve the odds of making correct decisions. The lessons apply broadly to any online experiment, whether in marketing, product development, or user experience research.
2 Framing the question
A well-defined question is the foundation of any experiment. In our case study, the team asks: Will changing the call-to-action button from 'Sign Up' to 'Start Your Free Trial' increase the click-through rate? This hypothesis is specific and testable. It identifies the independent variable (button text) and the dependent variable (click-through rate). The team also considers secondary metrics, such as sign-up completion rate, to ensure the change doesn't harm downstream behavior.
Framing the question also involves defining the population and the context. Are we testing on all visitors, or only new visitors? Does the change apply to mobile and desktop? These decisions affect the generalizability of the results. The team decides to include all visitors but will later analyze segments to check for consistency. They also set a clear timeframe: two weeks, to capture weekly cycles.
Finally, the team articulates the expected effect size. Based on past tests, they anticipate a relative lift of at least 10%. This expectation informs the sample size calculation. Without a clear question and expected effect, it's easy to misinterpret noise as signal. Thus, framing the question carefully sets the stage for a valid experiment.
3 Designing the test
The design of an A/B test determines how reliably it can answer the research question. In our case study, the team uses a simple two-arm randomized controlled trial. Visitors are randomly assigned to either the control (original button) or treatment (new button) group. Randomization ensures that the groups are comparable on average, aside from the manipulation. The team uses a robust random number generator and verifies that assignment is independent of user characteristics.
They also decide on the unit of randomization: the user. This prevents contamination that could occur if the same user sees both versions. They implement the test using a server-side tool that assigns users consistently across sessions. Additionally, they ensure that the test is blind: users are not aware they are in an experiment, and the analysts do not know which arm is which during data collection to avoid bias.
Other design considerations include the duration of the test and the handling of edge cases. The team chooses a fixed-horizon design: they will analyze results only after collecting the predetermined sample size. They also plan to check for sample ratio mismatch (SRM) to detect any assignment errors. By carefully designing the test, they minimize threats to validity and increase the chance that any observed difference is due to the treatment.
4 Sample size and validity
Sample size is critical for detecting a true effect and avoiding false positives. The team performs a power analysis before launching the test. They specify a baseline conversion rate (e.g., 5%), a minimum detectable effect (e.g., 10% relative lift, or 0.5 percentage points), a significance level (alpha = 0.05), and desired power (80%). Using these inputs, they calculate the required sample size per arm. This calculation ensures that if the treatment truly improves conversion by the specified amount, the test has an 80% chance of detecting it.
Validity threats are numerous. Peeking is a major one: if the team checks results daily and stops when significant, the false positive rate can exceed 30% or more, depending on how often they peek. Other threats include novelty effects (users clicking the new button out of curiosity), external validity (the result may not hold for other pages or times), and multiple comparisons (testing many metrics without correction). The team mitigates these by pre-registering the analysis plan, using a fixed sample size, and limiting primary metrics.
They also consider practical constraints: traffic volume may be lower than expected, prolonging the test. They decide to run the test until the required sample is reached, even if it takes longer than two weeks. This discipline is essential for valid inference. Without adequate sample size and control of validity threats, the experiment's conclusions are unreliable.
5 Running and reading the results
Once the test is live, the team monitors for technical issues but avoids looking at outcome data. After the predetermined sample size is reached, they analyze the results. They compute the conversion rates for each arm and the difference. Using a two-proportion z-test, they obtain a p-value and a confidence interval. Suppose the treatment conversion rate is 5.5% and control is 5.0%, a difference of 0.5 percentage points. The p-value might be 0.04, indicating statistical significance at the 5% level. The 95% confidence interval might range from 0.02 to 0.98 percentage points, suggesting the true effect is likely positive but with uncertainty.
However, statistical significance does not guarantee practical significance. The team considers the cost of implementing the change and the potential revenue impact. They also check secondary metrics: if sign-up completion rate drops, the net effect could be negative. They examine segments (mobile vs. desktop) to see if the effect is consistent. If the treatment works only for mobile, they might consider a targeted rollout.
Crucially, they avoid overinterpreting the p-value. A p-value of 0.04 means that if there were no true effect, there is a 4% chance of observing a difference this large or larger. It does not mean the treatment is 96% likely to be better. The team also considers the confidence interval, which provides a range of plausible effect sizes. By reading the results carefully, they make an informed decision.
6 What to change next
Based on the results, the team decides whether to implement the new button, keep the old one, or run further tests. If the effect is positive and consistent, they might roll out the change to all users. However, they should continue monitoring to ensure the effect persists. If the result is inconclusive or negative, they might iterate on the design, testing different variations. They could also explore why the effect occurred, using qualitative research or additional experiments.
The case study highlights the importance of learning from each test. Even a non-significant result provides information: it suggests the change didn't have a detectable effect, which can guide future hypotheses. The team documents the test details, including the hypothesis, design, sample size, and results, to build institutional knowledge. They also review the process for any deviations from the plan, such as unexpected peeking or data quality issues.
Finally, they consider the broader experimentation culture. Encouraging disciplined testing, avoiding peeking, and respecting sample size calculations are key. They might implement tools that prevent early analysis or automate sequential testing methods if continuous monitoring is necessary. By fostering a culture of statistical rigor, the organization can make better decisions over time. The next steps should always be grounded in the evidence and aligned with business goals.
7 Frequently asked questions
Q: Can I peek at my A/B test results without invalidating them? A: Peeking at fixed-horizon tests inflates false positives. If you must monitor continuously, use sequential testing methods designed for that purpose, but these require different sample size calculations and analysis.
Q: How long should I run my test? A: Run until you reach the predetermined sample size, which depends on your baseline rate, desired effect, and power. Duration should also cover full business cycles (e.g., at least one week) to account for weekly variation.
Q: What if my sample size is too small? A: Underpowered tests may miss true effects and produce unstable estimates. Consider increasing traffic, testing larger changes, or using alternative designs like switchback tests.
Q: What is sample ratio mismatch? A: SRM occurs when the observed ratio of users in each arm deviates from the intended ratio (e.g., 50/50). It indicates a bug in assignment or data collection and can invalidate results. Always check for SRM.
Q: Can I trust a significant result? A: Significance is not proof. Consider effect size, confidence intervals, practical significance, and validity threats. Replication and consistency across segments strengthen confidence.
Q: How do I handle multiple metrics? A: Predefine a primary metric and use corrections (e.g., Bonferroni) for secondary metrics, or treat them as exploratory. Avoid cherry-picking significant outcomes.