Know what to test, when to trust the result, and what to do next. Practical guides for analysts, growth teams, and founders.
FTC and international regulators don't just allege dark patterns anymore — their complaints now include the internal emails, A/B test data, and executive…
The lift in your test report and the lift finance sees a quarter later rarely match. Winner's curse, novelty decay, and regression all shrink it.
GA4, your server-side pipeline, and your BI tool report different numbers, and teams cite whichever fits. Why reconciliation is skipped, and the fix.
When three teams each claim the same conversion, attributed revenue exceeds reality and budget follows the best dashboard, not the best channel.
Enterprises govern reversible A/B tests like irreversible decisions, and velocity dies in the approval queue. The reframe that unlocks it.
Old experiment flags never get cleaned up, quietly contaminate new tests, and occasionally reactivate dead code. The carrying cost of zombie experiments.
An A/B test can show a clean aggregate win while the variant loses in every real segment. Simpson's paradox, why the topline lies, and the fix.
Switching A/B testing tools silently redefines your metrics and breaks historical comparability. What the sales demo never shows, and what to check first.
A winning A/B test isn't a shipped feature. The gap between the tested variant and what actually reaches production is where the value leaks away.
Some SaaS changes should raise revenue. Others should simply not break it. A billing flow rewrite, navigation cleanup, design system migration, or applied
A channel metric can rise while total revenue stays flat — the gain came from somewhere else. How to diagnose cannibalization and measure it.
Most low-traffic SaaS teams do not have a testing problem. They have a math problem. If your pricing page gets 8,000 visits a month, a small A/B testing
Token price and benchmark scores are the wrong scoreboard for choosing an AI coding model.
When you are trying to spot bot traffic in A/B tests, the numbers can be alarming. In June 2026, Cloudflare Radar reported that bots made up 57.
I have seen six-figure decisions ride on an event that never fired. The dashboard said no lift, but revenue reports told a completely different story.
A test can win on paper and still lose money. I see this all the time in B2B SaaS. A page changes, form fills rise, the dashboard looks good, then sales
A winning test can still lose you money. I see this a lot in high-stakes A/B testing. The team has a solid hypothesis, clean analytics, and good intent,
Most bad reruns do not fail in the stats tool. They fail in the story a team tells itself during A/B testing. A first test comes back weak, messy, or
If the same visitor sees variant A on Monday and variant B on Wednesday, your A/B testing efforts are not measuring behavior. They are measuring confusion.
I have seen teams ship the wrong variant because week three landed on quarter end, and buyers stopped moving. The test looked clean, but the revenue impact
Most teams pick an A/B testing method based on who can ship faster this sprint. That is how bad bets get dressed up as experimentation.
You ran the test. Signups moved. Activation moved. Revenue did not. At least not yet. This is where many SaaS teams make an expensive mistake.
Most low-traffic SaaS teams do not have a testing problem. They have a waiting problem. If you only get a few thousand meaningful users a month, a clean
Most bad product tests don't fail because the idea was weak. They fail because the test assigned treatment to the wrong unit.
Know what to test, when to trust the result, and what to do next. Practical decision guides for analysts, growth teams, and founders. Free. Weekly.
Opens Substack to confirm your subscription · Free · Unsubscribe anytime