I run experiments for a living — and I treat the practice itself as a research subject. My work sits at the intersection of experimentation methodology, causal inference, behavioral economics, and applied AI, grounded in data from 280+ production experiments rather than lab conditions.
200+ issues · 12,000+ subscribers
How organizations should measure the true value of testing programs: win rate as a vanity metric, winner’s-curse correction, sequential testing tradeoffs, and long-term holdouts as program-level audits.
Geo-incrementality, matched-market designs, holdout experiments, and attribution beyond last-click — separating demand that marketing created from demand it merely captured.
Which lab-validated effects survive contact with production traffic. Loss aversion, choice architecture, anchoring, and social proof — tested at scale rather than cited from a 1979 paper.
AI-assisted hypothesis generation, analysis acceleration, and personalization systems — and the governance layer they need: bias detection, guardrails, and transparency standards.
Practitioner research drawn from a decade of production experimentation. Papers publish first as citable write-ups on this site and Lean Experiments, with PDF versions prepared for scholarly indexing.
Experimentation programs report win rates as their headline KPI, but win rate is trivially gamed by testing safe, small changes. This paper proposes a decision-quality framework — revenue per experiment, save rate, and learning velocity — drawn from operating a 100+ test/year enterprise program.
Most organizations lose the knowledge their experiments generate within two personnel changes. This paper examines experiment repositories as institutional memory infrastructure, and the incentive structures that determine whether teams contribute to them.
Per-test impact models inflate program value through winner’s curse, novelty decay, and interaction effects. Drawing on holdout methodology popularized by Eppo and Statsig, this paper quantifies the gap between summed per-test readouts and holdout-measured cumulative lift.
Papers and technical writing I'm working through, with notes on how each one holds up against production data.
The productization of program-level measurement. The key insight: holdouts don’t just validate individual wins, they measure whether the program as a whole is worth its budget — which is the question the CFO is actually asking.
The canonical reference for experimentation at scale. The chapters on Twyman’s Law and sample ratio mismatch shaped how I QA every readout — surprising results are usually instrumentation bugs, not breakthroughs.
Still the highest-ROI paper in applied CRO. Loss framing has outperformed gain framing in my enterprise tests repeatedly — but the effect size in production is consistently smaller than the lab literature suggests, which is itself a finding worth publishing.
The experimentation operating system this research is built on — PRISM, the idea-to-investment pipeline, and honest impact measurement.
See the method → EvidenceThe production programs behind the data: $30M+ verified impact at NRG Energy, geo-incrementality at Silicon Valley Bank.
See the work → Open QuestionsProduct and research concepts I'm exploring — including AI-driven experimentation intelligence built on contributed test data.
See the ideas →Every week: one experiment, one framework, one insight to make your marketing more evidence-based and your revenue more predictable.
Free · No spam · Unsubscribe anytime
200+ issues of experiments, frameworks, and field reports from inside a Fortune 150 growth team.
Open Substack (opens in new tab)