A/B Test Sample Size & MDE Calculator Guide

Before I run any experiment, I calculate the required sample size. Not as a formality — as a genuine feasibility check. At least 30% of the test ideas I've seen would have required six months to run properly given the available traffic. Knowing that upfront saves time, money, and false confidence.

Most teams skip this step or do it wrong. Here's how to do it right — starting with the number itself, so you don't need a calculator to know whether your test is feasible.

Sample Size Per Variant: The Table Most Calculators Won't Show You

Every sample size calculator asks you for four inputs and returns one number. That is useful when you already know your baseline. It is useless when you are trying to answer the question that actually comes first: _is this test worth planning at all?_

So here are the numbers directly. Each cell is the visitors needed per variant at 95% confidence and 80% power, for a given baseline conversion rate and a given relative minimum detectable effect.

Three things to read off this table before you build anything.

Double every number for a two-arm test. These are per-variant figures. A control and one variant at a 3% baseline chasing a 10% lift needs 53,000 _each_ — 106,000 total. This single misreading is the most common planning error I see, and it is the reason so many tests get called at roughly half the power their owners believe they have.

The relationship is quadratic, not linear. Halving your MDE roughly quadruples the sample size. Look across the 3% row: 53,000 at a 10% MDE becomes 208,000 at a 5% MDE. That is the same test, same page, same traffic — asking for twice the sensitivity costs you four times the visitors.

Low baselines are brutal. A 1% baseline needs roughly eleven times the traffic of a 10% baseline for the same relative lift. If you are optimizing a low-converting flow, your MDE has to be large or your test will never finish.

If your traffic cannot support the cell you land in, the answer is not to run the test anyway. It is to change the MDE, change the page, or change the method. More on all three below.

Pro Tip: Screenshot this table and put it in your test planning doc. Most "can we test this?" conversations can be settled in thirty seconds without opening a calculator at all.

Why Sample Size Matters: The Underpowered Test Problem

An underpowered test is one that doesn't have enough data to reliably detect the effect you're looking for — even if that effect is real.

Suppose you're testing a new product page design. The new design genuinely improves conversion rate by 8% relative. But your test only reached 40% of the required sample size before someone called it. The test shows no significant result. You kill the variant. You just killed a real winner because you didn't collect enough data.

This is a Type II error — a false negative. You miss real effects. And unlike a false positive (shipping a winner that wasn't), a false negative is invisible. Nobody knows what you lost.

The rate of false negatives is controlled by statistical power. At 80% power and your required sample size, you'll correctly detect a true effect 80% of the time. At 40% of the required sample size, power falls to roughly 40-45%. You are worse than a coin flip at finding effects that are genuinely there.

Pro Tip: Run your sample size calculation before writing a single line of test code. If the test isn't feasible given your traffic, find out now — not after 6 weeks of running it.

The Four Inputs Every Sample Size Calculator Needs

Every credible sample size calculator — Optimizely's, Evan Miller's, AB Testguide — needs exactly four inputs. Mess up any one of them and your calculation is wrong.

1. Baseline conversion rate The current conversion rate of the metric you're optimizing. This must be the actual rate on the specific page being tested, not a site-wide average. If you're testing your checkout page, use checkout page conversion rate — not your overall site CVR.

2. Minimum Detectable Effect (MDE) The smallest relative lift you want the test to be able to detect. Set it at the minimum lift you'd actually ship. Check whether your calculator uses relative or absolute MDE — this guide uses relative throughout. Choosing the number itself is a business decision rather than a statistical one, and I've written a full guide to choosing the right MDE that walks through the tradeoff.

3. Statistical significance threshold Almost always 95% (alpha = 0.05, two-tailed). Some teams use 90% when running exploratory tests to reduce sample size. Use 99% when the stakes are very high (pricing tests, checkout flow changes on high-revenue pages).

4. Statistical power Almost always 80% (beta = 0.20). Use 90% for high-stakes tests where missing a real effect is costly.

Worked Example with Real Numbers

Scenario: You're testing a new checkout page layout on an e-commerce site.

• Baseline CVR: 3.0%
• MDE: 10% relative (you want to detect a lift from 3.0% to 3.3% or larger)
• Significance: 95% (two-tailed)
• Power: 80%

Running this through a standard two-proportion z-test:

Required sample size: ~53,000 visitors per variation (~106,000 total across both arms)

Now calculate test duration:

• Weekly traffic to the checkout page: 10,000 sessions
• Split: 50/50 (5,000 per variation per week)
• Weeks required: 53,000 / 5,000 = 10.6 weeks

This is where the honest answer gets uncomfortable. Ten and a half weeks is past the point where I would greenlight a test. Traffic patterns shift, campaigns launch, the site gets redesigned underneath you — a test running that long is measuring a moving target. This test needs a larger MDE, a higher-traffic page, or a different method entirely.

Run the same calculation at a 15% relative MDE and you need ~24,000 per variation, or 4.8 weeks. That is a test I would actually run. The hypothesis hasn't changed; only the honesty about what this traffic can resolve has.

Pro Tip: If a sample size calculation produces a test duration over 8 weeks, stop and revisit the MDE. Either raise the MDE to something your traffic can support, find a higher-traffic page to test the same hypothesis on, or reconsider whether the test is worth running at all.

MDE Sensitivity: The Number That Changes Everything

Understanding how sensitive sample size is to MDE changes helps you make better tradeoffs.

Using the same baseline (3% CVR, 95% significance, 80% power):

• 5% relative MDE → ~208,000 visitors per variation
• 10% relative MDE → ~53,000 visitors per variation
• 15% relative MDE → ~24,000 visitors per variation
• 20% relative MDE → ~14,000 visitors per variation

Going from a 5% to a 10% MDE cuts the required sample size by roughly 75%. Going from 10% to 20% cuts it by another 75%. The relationship is quadratic: double the MDE, cut the sample size to a quarter.

This is the most powerful lever in your test design. It is also the most abused — teams inflate the MDE to make an infeasible test look feasible, then act surprised when a "detectable" 20% lift never materializes. Raise the MDE because the business genuinely wouldn't ship a smaller lift, not because the calendar demands it.

Minimum Sample Size Per Variant: What the Number Actually Means

Here is the distinction that trips up more teams than any other: the minimum sample size per variant is not the same as the total sample size for your test. The calculator returns a per-variant number — the visitors each version needs to reach adequate statistical power. For a standard two-arm A/B test (control plus one variant), you double that number to get the total traffic required.

Say the calculator returns 53,000 per variant at 80% power, 95% confidence, and a 10% relative MDE on a 3% baseline. Your control needs 53,000 visitors and your variant needs 53,000 — 106,000 total. Miss this and you will badly underestimate how long the test runs, because you planned for 53,000 total when you actually needed twice that. A test stopped at that halfway point is running at roughly 50% power: it will miss a real effect about half the time.

The per-variant framing matters even more for multivariate and A/B/n tests. Every arm you add needs its own full allocation. A four-way test (one control, three variants) at 53,000 per variant needs 212,000 sessions — and that is before any correction for multiple comparisons, which pushes the requirement higher still. I have watched teams add a third and fourth variation "since we're already testing" without realizing they just tripled the traffic bill and the timeline.

One rule I apply before adding any variant: re-run the sample size calculation with the new arm count. If the per-variant number times the number of arms exceeds four weeks of realistic traffic, I cut variants until it fits. A clean two-arm test that concludes beats a four-arm test that gets killed at 40% power every time.

How to Read and Use the Sample Size Calculator in Optimizely

Optimizely's built-in sample size calculator (accessible in the test setup flow) uses the following defaults: 95% statistical significance, 80% power, and it accepts a relative MDE.

The key output to focus on: unique visitors per variation. This is the number each bucket needs to reach before you have adequate power. Multiply by the number of variations to get total required exposure.

Optimizely also shows a projected test duration based on historical traffic to the target URL. This estimate assumes stable traffic — if you're running a test during a campaign period or a historically volatile traffic window, apply a buffer.

One thing Optimizely's calculator doesn't do by default: adjust for multiple variations with family-wise error rate correction. If you're running 3+ variations, calculate the Bonferroni-corrected per-comparison alpha manually (alpha / number of comparisons) and use that adjusted significance level in the calculator.

Pro Tip: Cross-check Optimizely's traffic estimate against your analytics platform. Optimizely uses its own tagged session counts, which may differ from your analytics source of truth if your tag coverage isn't complete.

What to Do When Your Traffic Is Too Low

You run the calculation. The test would take 6 months. Your options:

Option 1: Increase the MDE. Accept that you'll only detect larger effects. If business context justifies this — a 15% lift would still be meaningful and shippable — this is the right call.

Option 2: Test a higher-traffic page. If you're trying to validate a hypothesis about value messaging, test it on your homepage (high traffic) before testing on your product page (lower traffic). Use a higher-traffic proxy to validate the direction, then refine on the actual target page.

Option 3: Consider qualitative research instead. If the page gets 500 sessions/week, A/B testing isn't the right tool. Run usability testing, session recordings, or user interviews. Get qualitative signal. Redesign based on that. Then monitor the before/after with analytics. That's not a test, but it's better than running an underpowered test for a year.

Option 4: Run a partial test with a directional hypothesis. If you reach roughly half the required sample size, you have about 50% power. You won't reach a definitive conclusion, but you can say "directionally positive, underpowered" and use that as input for the product roadmap without claiming a winner.

Pro Tip: Be honest with your stakeholders about what low-traffic tests can produce. "We'll run this for 2 weeks and see" on a 500-session/week page is not a test — it's guessing with extra steps. Setting that expectation correctly upfront is better than defending inconclusive results 2 weeks later.

The "We'll Run It for 2 Weeks and See" Fallacy

This is the most common mistake in experimentation practice. "Two weeks" is not a statistical input. It's an arbitrary duration driven by sprint cycles, reporting cadences, and impatience.

A 2-week test might be perfectly calibrated for your traffic and MDE. Or it might give you 30% of the required sample size. The duration doesn't tell you anything on its own. The sample size does.

The correct process: calculate required sample size → divide by available traffic per variation → that gives you required duration. Then round up to the nearest full week to capture weekly cycles. That's your test duration. Not "2 weeks" as a default.

If the calculated duration is longer than you have time for, that's a signal to revisit your MDE — not to run the test anyway and hope for the best.

Common Mistakes in Sample Size Planning

Mistake 1: Using total site traffic instead of traffic to the tested page.

This is the most expensive mistake in sample size planning. If your site gets 100,000 sessions/week but only 15% of those users visit your checkout page, the effective traffic for a checkout test is 15,000/week — not 100,000. Using the wrong denominator makes tests look feasible when they aren't.

How to fix it: Find the specific page's session count in your analytics tool, filtered to the same audience segment your test will target. That's your weekly traffic figure.

Mistake 2: Not accounting for multiple variations.

Adding variations doesn't just "split the traffic more." It requires you to adjust for multiple comparisons, or your false positive rate inflates.

With two variations (A and B), a 95% significance threshold gives you a 5% false positive rate per comparison. With three variations (A, B, and C), you have two comparisons (B vs. A, C vs. A). If you use 95% significance for each without adjustment, your family-wise false positive rate rises to approximately 9.75%.

Apply a Bonferroni correction to a 3-arm test (two comparisons, so alpha drops from 0.05 to 0.025) and the per-variant requirement rises by about 21%. Combined with the extra arm, total traffic runs roughly 1.8× a two-arm test — not the 1.5× you get from naively counting arms. A 4-arm test lands near 2.7× total. Most teams assume a third variation just splits existing traffic three ways; they don't realize the per-variation requirement climbs too.

Mistake 3: Reading the per-variant number as the total. Covered above, and worth repeating because it is the one that quietly halves your power.

Mistake 4: Not setting the sample size calculation before the test starts. Post-hoc sample size justifications are unreliable. Calculate before, commit to the number, and don't revisit it until the test is over.

Mistake 5: Treating the calculator's duration estimate as a guarantee. Traffic fluctuates. Campaigns change. A test that should take 4 weeks based on historical traffic might take 6 weeks during a slow period. Build buffer.

Mistake 6: Including sessions where the variant wasn't visible. If your test is above the fold and 30% of users don't scroll there, they shouldn't count toward your sample. Filter your analysis to users who were actually exposed to the experiment.

Mistake 7: Not checking the calculator's power assumptions. Some calculators default to 90% power instead of 80%. This isn't wrong, but it means you'll need more traffic. Know what your calculator assumes.

Pro Tip: Unless you have a specific reason to run more than two variations, stick to A/B (not A/B/C). Adding a third variation pushes total traffic to roughly 1.8× a two-arm test once you correct for multiple comparisons, and it extends duration proportionally. The marginal value of testing two variants simultaneously is rarely worth the cost.

How to Use a Free A/B Test Sample Size Calculator

The four inputs above are all any A/B test sample size calculator actually needs, so using one takes about sixty seconds once you have your numbers. Here is the exact sequence I run before every test.

Step 1 — Pull your real baseline. Open your analytics tool and find the current conversion rate for the specific page or flow you are testing, not the site-wide average. If your checkout converts at 3.0%, that 3.0% is your baseline input.

Step 2 — Set your MDE. Enter the smallest relative lift you would actually ship, say 10%. The calculator will tell you it can detect a move from 3.0% to 3.3% or larger at your chosen power and confidence.

Step 3 — Read the per-variant number, then double it. With a 3.0% baseline and a 10% relative MDE at 80% power and 95% confidence, the calculator returns roughly 53,000 visitors per variant. That is 106,000 total for a clean two-arm test — the single number most teams misread when they plan for the per-variant figure as if it were the total.

Step 4 — Convert to duration. Divide the per-variant figure by your weekly traffic per arm. At 5,000 sessions per variant per week, 53,000 visitors is about 10.6 weeks — past my eight-week ceiling, which means raising the MDE or picking a higher-traffic page. I built a free sample size calculator and a companion MDE calculator at GrowthLayer that do all four steps for you — enter your baseline and traffic, and they return visitors per variant and the detectable lift instantly.

What to Do Next

Run sample size calculations for your next 3 planned tests before building anything. If any of them come back with durations over 8 weeks, use that as the trigger to revisit MDE or page selection.

FAQ

What is the minimum sample size per variant for an A/B test?

The minimum sample size per variant is the number of visitors each version of your test needs to reach adequate statistical power, typically 80% power at 95% confidence. There is no universal number — it depends on your baseline conversion rate and your minimum detectable effect. On a 3% baseline, detecting a 10% relative lift needs roughly 53,000 per variant, while a larger 20% MDE on the same baseline needs about 14,000 per variant. Always run the calculation for your own numbers rather than reusing a figure from another test.

How many users do I need for a statistically significant A/B test?

You need enough users to reach the per-variant sample size your calculator specifies, then double it for a two-arm test. Significance is not a fixed visitor count — it is the point at which your observed difference is unlikely to be noise given your sample size and effect. Reaching the pre-calculated sample size at 80% power is what makes a significant result trustworthy; stopping early inflates false positives no matter what the p-value shows.

What sample size do I need for a 5% MDE?

A 5% relative MDE is expensive. On a 3% baseline you need roughly 208,000 visitors per variant — about 416,000 total — at 95% confidence and 80% power. On a 10% baseline it drops to about 58,000 per variant. Because sample size scales quadratically with MDE, a 5% MDE costs roughly four times what a 10% MDE costs on the same page. Most teams chasing a 5% MDE do not have the traffic to resolve it and would be better served by testing a bolder change.

Do I need to adjust sample size for more than two variants?

Yes. Every additional variant needs its own full per-variant allocation, and running multiple comparisons raises your false-positive risk unless you apply a correction such as Bonferroni or a family-wise error rate adjustment. Both effects increase the total traffic you need, so a four-arm test costs far more than double a two-arm test.

How do I calculate A/B test sample size?

Enter four inputs into a sample size calculator: your baseline conversion rate, your minimum detectable effect, your significance level (usually 95%), and your statistical power (usually 80%). The calculator returns the visitors needed per variant; double that for a two-arm test, then divide by your weekly traffic per arm to get the required duration. Always run this before you build the test, not after — a post-hoc sample size justification is unreliable.

Is a two-week A/B test long enough?

Only if two weeks happens to deliver your required sample size — duration alone tells you nothing. Two weeks on a high-traffic homepage may be well powered, while the same two weeks on a 500-session-per-week page produces a fraction of the data you need. Calculate the sample size first, convert it to weeks at your traffic level, and run for that long, rounding up to whole weeks to capture weekly cycles.

Before you build anything, size the test. I built free tools at GrowthLayer for exactly this: start with the sample size calculator to find how many visitors each variant needs, then run the MDE calculator to check whether the lift you are chasing is even detectable given your traffic. For a complete test design template — sample size planning, stopping rules, and result documentation — see my Optimizely Practitioner Toolkit.

_Correction (July 2026): an earlier version of this guide reported the worked-example sample sizes as per-variant figures when they were closer to the combined two-arm totals — understating the per-variant requirement by roughly half. It also overstated the cost of a third variation as 3× the per-variant sample size; the correct figure is about 21% more per variant, or roughly 1.8× total traffic. Every number here has been recalculated from the two-proportion z-test at 95% confidence and 80% power, and cross-checked against a power simulation._