A/B Test Sample Size & Power Calculator
Calculate exact A/B test sample size, statistical power, and test duration. Features MDE reverse solving, continuous metrics, multi-armed Bonferroni, and SRM.
Regime:
Technical Reference & Architecture Guide
A/B Test Sample Size & Power Calculator Guide
Calculate exact A/B test sample size, statistical power, and test duration. Features MDE reverse solving, continuous metrics, multi-armed Bonferroni, and SRM.
Key Parameters & Controls
Baseline Conversion Rate (p₁)Float / Percentage (0.0001 - 0.9999)
Historical binomial proportion of the control cohort. Asymptotic normal approximation requires Cochran criterion n·p₁ ≥ 5 and n(1 - p₁) ≥ 5. Lower baselines inflate required observation count inversely proportional to Bernoulli variance p₁(1 - p₁).
Default: 0.03 (3.0%)
Minimum Detectable Effect (MDE / Δ)Float / Percentage (Relative or Absolute)
Target statistical sensitivity threshold. Governed by inverse-square scaling n ∝ 1/Δ²: halving the relative MDE quadruples required sample volume. Computed client-side via exact rational approximations of the inverse normal CDF (probit function) within IEEE 754 double-precision epsilon (2.22e-16).
Default: 0.05 (5.0% relative lift)
Significance Level (α / Type I Error)Float (0.001 - 0.20)
Maximum acceptable probability of false discovery (rejecting null hypothesis when true). Standard two-tailed symmetric threshold divides α across both Gaussian tails (Z_{α/2} = 1.95996 for α = 0.05). Multi-arm experiments require Bonferroni adjustment α_adj = α / (K - 1) to constrain Family-Wise Error Rate (FWER).
Default: 0.05 (95% confidence)
Statistical Power (1 - β / Sensitivity)Float (0.50 - 0.99)
Complement of Type II error rate (β). Defines the probability of correctly rejecting the null hypothesis when an authentic effect exists. Increasing power from 0.80 (Z_β = 0.8416) to 0.90 (Z_β = 1.2816) expands required visitor quotas by ~34.2% due to non-linear critical Z-score offset.
Default: 0.80 (80% power)
Number of Treatment Arms (K)Integer (2 - 20 variants)
Total experimental variants including control. In multi-arm testing (A/B/n), unadjusted comparisons suffer compounding false discovery: cumulative α_cum = 1 - (1 - α)^(K - 1). Controlled client-side via O(K) Bonferroni or Dunnett multiplicity corrections.
Default: 2 (1 Control vs 1 Treatment)
Pro Tips & Best Practices
Enforce Full 7-Day or 14-Day Calendar Cycles
Even when high-velocity platforms hit nominal significance (p < 0.05) within 48 hours, terminating early invalidates statistical inference due to temporal autocorrelation and day-of-week seasonality (weekend vs. weekday purchasing patterns). Complete at least two full 7-day cyclical periods to capture weekly variance and mitigate novelty effects.
Eliminate Continuous Peeking via Alpha Spending or mSPRT
Repeatedly evaluating dashboard p-values on accumulating data ('peeking') inflates empirical Type I error from nominal 5% to over 30% by sample maturity. If real-time early stopping is required, implement Lan-DeMets alpha spending functions with O'Brien-Fleming boundaries or Asymmetric Mixture Sequential Probability Ratio Tests (mSPRT).
Audit Sample Ratio Mismatch (SRM) Before Evaluating Metric Lift
Execute an O(K) Pearson Chi-Square Goodness-of-Fit test (χ² = Σ(O - E)² / E) on raw assignment counts prior to evaluating conversion. An SRM p-value below 0.001 signals structural tracking bias—such as asynchronous CDN worker drops, redirect cache anomalies, or bot filtering differentials—which renders subsequent hypothesis testing invalid.
Standardize on Relative MDE for Cross-Team Calibration
Always communicate Minimum Detectable Effect in relative terms (e.g., +8.0% relative lift on a 4.0% baseline equals 4.32%) rather than absolute percentage points (+0.32%). Relative metrics scale consistently across varied cohort sizes, align with financial modeling, and prevent misinterpretation of margin points versus percentage growth.
Common Gotchas & Troubleshooting
⚠️Sample size requirement is impractically large (e.g. > 500,000 visitors per variant).
Target an upstream micro-conversion proxy with higher baseline frequency (e.g., Add-to-Cart at 12% rather than Completed Order at 1.8%), relax MDE from granular targets (e.g., 2% relative) to actionable business thresholds (7%–10%), or benchmark power at 80% rather than 95%.
⚠️Sample Ratio Mismatch (SRM) alert triggers with Pearson Chi-Square p < 0.001.
Halt decision-making; do not declare a winner. Segment allocation ratios by browser engine, device class, geographic POP, and HTTP redirect latency. SRM is commonly caused by client-side script execution timeouts, aggressive WebKit Intelligent Tracking Prevention (ITP) cookie drops, or bot scrubbers disproportionately filtering variant URLs.
⚠️Continuous metric sample size (Revenue/ARPU) diverges wildly from normal projections.
Continuous e-commerce data exhibits extreme Pareto right-skewness and high variance (σ² >> μ). Mitigate heavy tails by winsorizing order values at the 99th percentile, applying a logarithmic transformation (log(1 + x)), or transitioning from Welch's t-test to non-parametric Mann-Whitney U or bootstrapped empirical distributions.
Frequently Asked Questions
How do you calculate sample size for an A/B test?▼
A/B test sample size per variant is calculated using the pooled two-proportion formula: n = [Z_{α/2}√(2p̄(1 - p̄)) + Z_β√(p₁(1 - p₁) + p₂(1 - p₂))]² / (p₂ - p₁)², where p̄ = (p₁ + p₂)/2, Z_{α/2} is the critical value for significance (1.96 for 95% confidence), and Z_β is the critical value for power (0.8416 for 80% power). This tool executes client-side rational approximations for inverse normal CDFs with O(1) time complexity and zero server data transmission.
Why is 80% statistical power the industry standard in A/B testing?▼
Statistical power of 80% (β = 0.20) represents the optimal economic trade-off between false negative risk and traffic allocation cost. Escalating power from 80% to 90% (Z_β moving from 0.84 to 1.28) demands approximately 34% more sample volume, and 95% power demands ~74% more traffic. For most digital products, prolonged test durations cause severe business opportunity costs that outweigh the residual 20% Type II error risk.
What is the difference between relative and absolute Minimum Detectable Effect (MDE)?▼
Relative MDE expresses expected lift as a percentage of baseline: for a 5.0% baseline, a 10% relative MDE targets a 5.5% conversion rate (Δ = 0.005). Absolute MDE expresses the change directly in raw percentage points: a 2.0% absolute MDE on a 5.0% baseline targets 7.0% (Δ = 0.02). Relative MDE is the standard for experimentation platforms because business growth goals and variance scaling naturally normalize around relative proportional shifts.
Should an A/B test use a one-tailed or two-tailed hypothesis test?▼
A two-tailed hypothesis test is required for production A/B testing because treatment variants can degrade user experience and depress conversion. A two-tailed test splits alpha across both distribution tails (α/2 = 0.025 on each side), ensuring mathematical protection against winning calls on negative shifts. One-tailed tests should be restricted exclusively to non-inferiority trials or cloud infrastructure migrations where performance degradation is the sole directional concern.
What is Sample Ratio Mismatch (SRM) and how do you detect it?▼
Sample Ratio Mismatch (SRM) occurs when observed variant visitor traffic significantly deviates from the assigned randomization split (e.g., 50/50 designed vs. 48.8/51.2 observed). It is detected using a Pearson Chi-Square Goodness-of-Fit test: χ² = Σ(O_i - E_i)² / E_i with K - 1 degrees of freedom. A threshold of p < 0.001 indicates severe technical distortion (e.g., edge caching anomalies, browser crashes, or asymmetric redirect latency) that invalidates all causal inference.
Why must an A/B test run for a minimum of 7 or 14 full days?▼
Experiments must span at least 7 or 14 full 24-hour cycles regardless of whether sample size quotas are met earlier, because conversion behavior exhibits strong intra-week periodicity (weekend vs. weekday purchasing patterns). Truncating early induces false discoveries via temporal autocorrelation, Simpson's paradox across day-of-week cohorts, and failure to account for multi-day conversion latency windows.
Related Tools & Utilities
All Tools →PX to REM & REM to PX ConverterDeveloper
Convert PX to REM and REM to PX instantly with custom root font sizes. Generate clean CSS clamp() fluid typography, Tailwind classes, and conversion tables.
Open tool→
HTTP Status Code Checker & Redirect TesterDeveloper
Check HTTP status codes, test redirect chains, and inspect response headers. Diagnose 1xx to 5xx error codes, RFC specifications, SEO impact, and fixes.
Open tool→
Regex Tester & Regular Expression DebuggerDeveloper
Test, debug, and optimize regular expressions in real time. Inspect regex pattern matches, capture groups, step-by-step token explanations, and replace syntax.
Open tool→
Subnet IP Calculator & CIDR RangeDeveloper
Calculate IPv4 subnet masks, CIDR network ranges, usable host addresses, and broadcast IPs. Features bitwise binary inspection, VLSM planning, and cloud VPC rules.
Open tool→
Binary Addition & Subtraction CalculatorDeveloper
Calculate binary addition and subtraction with step-by-step carry bits, two's complement signed values, bitwise logic gates, and hex conversion in real time.
Open tool→
Robots.txt Tester & Syntax ValidatorDeveloper
Test and validate robots.txt syntax against RFC 9309. Check user-agent crawl permissions, debug disallow paths, and inspect sitemaps with instant browser testing.
Open tool→