Across 200+ tests in a real CRO portfolio, only 2% reached traditional statistical significance. The other 98% shipped or reverted on something else — and most teams pretend they were stat-sig anyway.
TL;DR
- Traditional 50/50 A/B testing fails when baseline conversion is high (≥80%) or traffic is thin. The math can't detect meaningful effects in any reasonable runtime.
- Across a 200+ test portfolio, only ~2% of tests reached traditional p<0.05. The vast majority shipped or reverted on directional + secondary signals.
- Holdout-validated rollout is the legitimate methodology for these conditions — not a workaround. Ship the variant to 90-95% of traffic, hold 5-10% on control, monitor for 3 weeks.
- The decision criteria use Bayesian posterior probability + secondary metric monitoring, not frequentist p-values. Documentation must reflect the methodology distinction.
The 2% problem
A portfolio of 200+ tests run across two years of an enterprise CRO program. Stat-sig rate at traditional p<0.05: ~2%.
| Status | Count | Share | Stat sig? |
|---|---|---|---|
| Winner | ~49 | 24% | ~12% of these were non-stat-sig |
| Inconclusive | ~21 | 10% | None stat-sig by definition |
| Loser | ~10 | 5% | Some stat-sig, most directional |
| Optimization Opportunity | ~32 | 16% | Pre-launch / discovery |
| In progress / paused / disabled | ~91 | 45% | Various states |
The traditional CRO advice — "wait for statistical significance before shipping" — is workable only on tests where the math allows stat-sig in a reasonable timeframe. For most CRO tests, the math doesn't allow it. Teams ship anyway, but mislabel the methodology.
The methodology mislabel is the actual problem. It produces a test repository where "winner" means three different things — sometimes stat-sig win, sometimes directional ship, sometimes "we liked it and shipped it." Future readers can't tell which is which.
When 50/50 A/B testing fails
| Test condition | Why 50/50 A/B fails |
|---|---|
| Baseline conversion ≥80% | Hard ceiling limits detectable effect; small lifts are statistically invisible at most achievable sample sizes |
| Mobile / niche traffic <5K per arm per week | Sample size accumulates too slowly; runtimes extend beyond actionable timeframes |
| Pre-test power calculation says runtime >6 weeks for MDE >5% | Even at MDE 7-10%, runtime exceeds the team's experimentation window |
In any of these conditions, running the 50/50 anyway produces inconclusive results that programs ship on faith. That's the worst combination — the math couldn't detect the effect, so the team uses gut feel, then documents the ship as if it were a stat-sig win.
Holdout-validated rollout is the methodology that fits the constraint instead of fighting it.
The three regimes
| Regime | When to use | Output classification |
|---|---|---|
| Standard 50/50 A/B | Baseline <50%, traffic ≥5K/arm/week | stat_sig_win, stat_sig_loss, or inconclusive_park |
| Holdout-validated rollout | Baseline ≥80%, OR traffic <5K/arm/week | holdout_validated (ship) or directional_revert |
| Non-inferiority | Variant ships for non-conversion reasons (compliance, brand) | non_inferiority_pass or directional_revert |
The regime determines the methodology, the inference, and the test-repository documentation. Conflating regimes in repository documentation creates problems for whoever reads the test record next.
How holdout-validated rollout works
| Phase | Action | Duration |
|---|---|---|
| Pre-launch | Ship variant to 90% (or 95%) of traffic; hold 10% (or 5%) on control | Day 0 |
| Monitoring | Daily check on primary + downstream + guardrail metrics | Days 1-21 |
| Mid-window check | Day 14: review trends, catch early regressions | Day 14 |
| Decision | Day 21: ship to 100% if no regression; revert if regression detected | Day 21 |
The 90/10 split (rather than 50/50) maximizes the variant's exposure while preserving a control group large enough to detect a regression of the magnitude that would justify a revert. The 3-week window balances time-to-decision against the statistical noise that a 10% holdout introduces.
The decision criteria
| Signal | Pass threshold | Action |
|---|---|---|
| Bayesian P(variant > control) on primary | ≥0.80 | Continue; ship at end of window |
| Bayesian P(variant > control) | 0.40-0.80 | Continue; reassess at Day 21 |
| Bayesian P(variant > control) | <0.40 | Investigate; likely revert |
| Holdout outperforms variant by pre-defined regression threshold | Triggered | Revert immediately |
| Holdout SRM detected (chi-square p<0.01) | Triggered | Pause; verify traffic split |
| Time-on-page or engagement shows large unexplained shift | Triggered | Investigate before Day 21 |
The frequentist p-value is monitored as a sanity check but does not drive the decision. Most holdout-validated tests will not produce stat-sig results; that's why this regime exists.
Three preconditions for using this regime
| Precondition | Why it's required |
|---|---|
| Reversibility | Changes that can be undone in a single deploy (CSS, copy, conditional render). Database-schema changes don't fit. |
| Real-time observable downstream metrics | Need to detect regressions within days. Direct revenue or conversion metrics with <24-hour lag. |
| Pre-committed regression threshold | Defined before launch: "If holdout outperforms variant by >X% on metric Y, revert." Without this, the team is making decisions in real-time under pressure. |
Without all three, the regime is too risky. Stick with standard 50/50 A/B and either accept the long runtime or pick a different test.
Worked example: an 85% baseline at low traffic
A mobile verification step deep in a multi-step enrollment flow. Baseline next-step conversion ~85%; mobile traffic ~1.5K per arm per week. Pre-test power calculation: 50/50 A/B would need 6+ weeks at MDE 7%.
| Decision | Action taken | |
|---|---|---|
| Methodology | 90/10 holdout, 3-week window | |
| Pre-committed regression threshold | Holdout >2% better than variant on primary → revert | |
| Primary monitoring metric | Next-step conversion | |
| Secondary metrics | Time-on-page, scroll depth, copy interactions | |
| Day-21 result | Value | Interpretation |
| ----------------------------- | --------------------------------- | -------------------------------------- |
| Bayesian P(variant > control) | ~0.90 | Strong directional signal |
| Frequentist p-value | ~0.20 | Not stat-sig, as expected |
| Time-on-page change | -15% (~120s → ~100s) | Faster decisions, engagement preserved |
| Scroll depth + interactions | Held flat (within ±2% of control) | No skipping content |
The variant shipped under holdout_validated classification. Documentation explicitly noted: not a stat-sig A/B win; shipped on the combination of strong Bayesian posterior, time-on-page improvement, and absence of secondary-metric regression.
Documentation discipline
Every shipped result under this regime must explicitly tag the methodology:
| Field | Required content |
|---|---|
| decisionFramework | holdout_validated (not stat_sig_win) |
| outcome | winner (per team's call) |
| isSignificant | false (frequentist p ≥ 0.05) |
| Notes | "Shipped via 90/10 holdout-validated rollout. Not a stat-sig A/B win. Bayesian posterior on primary: X. Pre-committed regression threshold: Y." |
A holdout_validated result does not generalize the same way as a stat_sig_win result — the statistical claim is different and the conditions under which the result transfers are narrower. Future readers of the test repository need to know the difference.
When NOT to use holdout-validated rollout
| Situation | Why standard A/B is better |
|---|---|
| Pre-test math says 50/50 A/B will power in <6 weeks | Use the regime that produces clean inference |
| The change is high-risk or irreversible | Holdout doesn't protect against changes you can't undo |
| The team can't afford the dashboard work to monitor in real-time | Need the monitoring infrastructure for this regime to be safe |
| Stakeholders expect stat-sig win documentation | Align expectations before launch — this regime won't produce one |
Bottom line
The 2% stat-sig rate across a real CRO portfolio is the data point most CRO advice ignores. Almost every shipped CTA win in real programs is non-stat-sig at p<0.05. The professional thing to do is match the methodology to the conditions and document the methodology accurately — not run underpowered 50/50 A/B tests and pretend they produced stat-sig results.
Holdout-validated rollout is the legitimate methodology for high-baseline or thin-traffic conditions. It has its own inference framework (Bayesian posterior + regression threshold) and its own documentation requirements. Programs that adopt the regime correctly ship more defensible wins than programs that force every test into a 50/50 stat-sig frame and ship the inconclusive results on faith.