Experimentation Governance: Managing SRM, False Positives, and Bias Through a Trust Deficit Framework
TL;DR: Every statistical failure in your experimentation program — SRM, false positives, bias — costs more than the individual bad decision. It erodes organizational trust in experimentation itself, and trust deficit compounds faster than most teams realize.
Key Takeaways
- SRM, false positives, and bias are not just statistical problems — they are organizational trust problems that compound across future experiments
- The Statistical Trust Deficit framework measures how many future decisions are harmed by each bad call shipped, revealing why rigor pays back non-linearly
- Availability heuristic makes one memorable false positive (a shipped "winner" that failed) more damaging to team credibility than ten correct calls
- Sample Ratio Mismatch is the most under-detected failure mode: a 49.5%/50.5% split feels fine but often signals a tracking bug that invalidates the test
- Guardrail metrics and pre-declared success criteria are not bureaucracy — they are the only defense against survivorship bias in test interpretation
The Real Cost of a Bad Statistical Call Is Downstream
When a team ships a variant based on a false positive, the direct cost is the lost revenue from the worse variant shipping. That cost is real but bounded.
The downstream cost is larger and almost never tracked: every future experiment proposal faces more skepticism, more layers of review, and more executive second-guessing because the team's credibility took a hit. Availability heuristic — the well-documented Tversky and Kahneman finding that memorable examples dominate judgment — means the one test that shipped and failed is remembered more vividly than the twenty that shipped and worked.
This is why governance matters. Not because every decision needs a compliance process, but because the asymmetric cost of shipped false positives means statistical rigor has a compounding ROI. A team that ships three false positives in a year doesn't just lose the value of those three calls — it loses organizational willingness to trust the next dozen proposals.
"Bad testing accumulates fake lifts. Fake lifts don't convert into users — and eventually they don't convert into trust either." — Atticus Li
Sample Ratio Mismatch: The Most Under-Detected Failure
Sample Ratio Mismatch (SRM) occurs when your actual traffic split differs from what you designed — a 50/50 test that allocates 49.5/50.5, or worse, 48/52. It signals that something is wrong: a tracking bug, a bot surge, a caching layer interfering with assignment, an overlap with another test.
The reason SRM is under-detected is that small mismatches feel reasonable. A 49.5%/50.5% split looks like random variation to the eye. It isn't — at sufficient sample size, the chi-squared test on that split produces p-values that should trigger concern. Teams without automated SRM detection miss these almost every time.
Why it matters: SRM doesn't just affect your split. It signals that whatever caused the imbalance is also corrupting your treatment assignment, which corrupts every metric in the test. Shipping on SRM-affected results is shipping on garbage.
Detection thresholds: For a 50/50 test, anything with p < 0.001 on a chi-squared goodness-of-fit test is worth investigating. For larger samples, even apparently small deviations generate significant test statistics.
Common root causes: Bot traffic (one variant scraped more than the other), tracking bugs (one variant fires an event the other doesn't), user-level caching (returning users get stuck in one variant), experiment overlap (another test's assignment interferes with this one's randomization).
A platform with automated SRM alerts catches these. A spreadsheet-based system almost never does. This is one of the clearest cases where the tool actually determines whether governance is possible.
The Statistical Trust Deficit Framework
Here's the framework I use to explain why statistical rigor has non-linear payback:
Statistical Trust Deficit (STD) = (Shipped calls that failed) × (Future proposals requiring extra scrutiny as a result)
When a team ships a variant that later proves worse than control, every future proposal from that team faces additional scrutiny. Executives want more evidence, more pre-registration, more discussion. The multiplier is typically 3-10x: one bad call creates resistance against the next 3-10 proposals.
This is why shipping on an underpowered test or an SRM-affected result is so expensive. You're not just making one bad call. You're generating an STD that will slow down the next quarter's experimentation velocity by adding review overhead to every proposal.
Interpretation thresholds:
- STD below 2 — Healthy. Occasional misfires but organizational trust intact.
- STD between 2 and 5 — Warning. Team starting to feel review burden increase. Time to double down on governance.
- STD above 5 — Crisis. Experimentation program is fighting its own credibility more than it's producing learning.
Managing False Positives: Four Defenses
The standard 5% significance threshold means 5% of experiments that actually have no effect will produce a positive result by chance. When you run 50 tests a year, 2-3 of them will appear to win by random noise alone. Governance is the set of practices that catches these before they ship.
Defense 1 — Pre-registered hypotheses. Write down the specific metric, the specific directional prediction, and the specific sample size before launch. This prevents the most common failure mode: post-hoc metric hunting, where teams discover a "winning" metric they never intended to measure.
Defense 2 — Guardrail metrics with fail-stops. Every test should have at least one guardrail — a metric that, if it degrades beyond a threshold, triggers automatic rollback. Guardrails prevent the secondary failure mode: shipping a primary-metric winner that tanks retention or crashes page load.
Defense 3 — Minimum runtime discipline. Week-boundaries matter. Tests stopped on a Wednesday because "results look good" are heavily biased by day-of-week effects. The discipline of running a full week (and often multiple weeks) filters out transient patterns.
Defense 4 — Multiple comparisons corrections. When you track 10 metrics in a single test, one of them will look significant by chance at the 5% threshold even if nothing changed. Bonferroni correction, false discovery rate controls, or pre-declared primary metrics solve this.
These are not bureaucracy. Each one is a specific response to a documented failure mode, and skipping them is how false positives become shipped features.
Addressing Bias in Experiment Interpretation
Beyond SRM and false positives, bias in how results are interpreted is the third failure mode. Three patterns dominate:
Survivorship bias. Tests that completed are analyzed; tests that failed to launch or were cancelled mid-run get ignored. But the cancelled tests often contain the most important signal — they tell you where your hypotheses are systematically wrong. Abraham Wald's famous WWII analysis of bullet-hole patterns on returning planes is the classic example: you have to look at the planes that didn't come back.
Confirmation bias in metric selection. When a test shows a primary metric loss but a secondary metric win, the temptation to reframe the secondary as the real story is strong. Pre-registration defends against this.
Novelty and primacy effects. Users respond to change in ways that don't persist. A test showing a 12% lift in the first week may show 2% in week four, because the initial lift was novelty. Running tests long enough to see behavior stabilize matters — especially for features touching habitual user behavior.
The best defense against interpretation bias is writing down what you expect before you see the data, and holding yourself to analyzing the data you pre-committed to.
Common Mistakes in Governance
Treating governance as a compliance exercise. When governance feels like paperwork, teams find ways around it. When governance is framed as "this saves us from shipping losers that kill our credibility," teams adopt it.
Over-correcting after a bad call. A shipped false positive often triggers a governance crackdown that slows everything. The right response is targeted: what specific failure mode produced this call, and what specific defense would have caught it?
Relying on statistical rigor alone. Clean statistics on a bad hypothesis still produce bad decisions. Governance should include hypothesis quality review, not just statistical review.
Skipping SRM checks. This is the most common preventable failure. Every serious test should have automated SRM detection running.
Advanced: Bayesian Interpretation and Sequential Testing
For teams running high volume, frequentist significance thresholds create a velocity problem: you have to wait for a fixed sample size even when the signal is clearly strong or clearly absent.
Bayesian sequential testing methods — where you continuously update posterior probabilities and can stop when evidence is decisive — address this without the false positive inflation that frequentist peeking causes. The tradeoff is that Bayesian methods require good prior specification, which is its own discipline.
Most mid-volume teams (under 30 tests per quarter) don't need Bayesian methods. High-volume teams (50+ tests per quarter) often benefit from them significantly, but only after the basic governance foundation is in place.
Frequently Asked Questions
What's the single most important governance practice to adopt first?
Automated SRM detection. It catches the failure mode that's both most common and hardest to spot by eye. If you implement one thing from this article, implement SRM alerts.
How long should I run a test?
At minimum, one full week to capture day-of-week effects. For features with habitual user behavior (anything users do repeatedly), 2-4 weeks to see novelty effects stabilize. Never stop on a day that isn't a week boundary.
Should I use Bayesian or frequentist methods?
Start frequentist. It's simpler to reason about, has more tooling support, and the failure modes are better documented. Move to Bayesian when you're running enough volume that fixed sample sizes are genuinely slowing you down.
What's the right number of guardrail metrics?
2-4 per test. Primary metric, one or two leading indicators of the primary metric, and at least one "don't break the product" guardrail (page load time, error rate, retention).
How do I handle a test where the primary metric is inconclusive but a secondary metric moved?
Don't ship on the secondary metric. The test wasn't designed to detect movement in that metric, and treating it as a win is post-hoc interpretation. Either run a follow-up test with the secondary as the primary, or accept the inconclusive result.
Methodology note: Governance practices and trust deficit patterns reflect experience across experimentation programs in energy, SaaS, and e-commerce verticals. Specific figures are presented as ranges. Behavioral economics references draw on Kahneman and Tversky's work on availability heuristic and prospect theory.
---
Governance starts with seeing what's actually running. Browse structured experiment archives in the GrowthLayer test library — organized by funnel stage, hypothesis type, and result pattern.
Related reading: