The Framework
The Lean Experimentation System
Five steps. Infinite compounding. The framework I use to turn marketing spend into statistical certainty — built over 9 years and $30M+ in verified revenue impact.
Observe
Before a single hypothesis is written, we study the data. Heatmaps, session recordings, funnel analysis, revenue attribution. What is actually happening — not what we think is happening.
At NRG Energy, this phase revealed that 40% of our digital media spend was generating conversions that were already going to happen — spend we could reallocate to genuinely incremental acquisition.
Hypothesize
Good hypotheses connect behavioral observations to business outcomes. Not "let's try a different button color" but "reducing friction at step 3 will increase conversion by X% because users are dropping at that moment for Y reason."
The hypothesis is the contract. It defines what success looks like before you start — which means you can't move the goalposts after you see results you don't like.
Test
The test is designed to be conclusive. Proper sample size calculation, clear hold-out groups, statistical significance thresholds set before launch. We're not fishing for results — we're looking for truth.
Geo-holdout testing, A/B, multivariate — the method matches the question. For media mix questions at NRG, geo holdouts gave us the causal answer that click attribution never could.
Prove
Results must be translated into language that drives decisions. Not "the test reached 95% confidence" but "this change generates $X in additional annual revenue — here's how we know."
Executive narratives that connect experimentation to the P&L are what get programs funded, expanded, and protected when budgets get cut. The proof step is as important as the test itself.
Scale
Winners get shipped. But more importantly, winners get systematized. The learning gets documented, the pattern gets extracted, and the next experiment starts with a higher baseline.
Compounding is the point. A single good experiment is a data point. A system of experiments is a competitive advantage.
Lean Experiments Newsletter
Get the Framework in Your Inbox
Every week I break down one experiment, one framework, or one behavioral principle — with the math, the context, and the revenue implications. Free forever.
Free. No spam. Unsubscribe anytime.
Lean Experiments Newsletter
One framework. Every week.
Free forever.
Join 12,000+ growth leaders who get practical A/B testing, behavioral science, and revenue optimization frameworks — without the fluff.
Free. No spam. Unsubscribe anytime.