Sample Ratio Mismatch: Causes, Checks, and Fixes

TL;DR

• Sample ratio mismatch, or SRM, means the observed allocation between experiment variants is too unlikely under the split you intended.
• SRM is not the result. It is a warning that assignment, execution, telemetry, filtering, or analysis may have broken the comparison.
• Check SRM before reading treatment effects. Microsoft’s experimentation platform uses a conservative threshold of p < 0.0005; your team should follow its platform standard or a threshold declared before results are visible.
• Diagnose the cause across the whole experiment pipeline. Start with the full assigned population, then inspect units, triggers, time windows, segments, telemetry, and data processing.
• Do not rescue a flattering result by reweighting or casually removing data. If the cause is not identified and cleanly isolated, fix the system and rerun.

The experiment result that changed sign

A Microsoft team tested whether increasing the number of cards in an MSN carousel would improve engagement. The experiment appeared to show the opposite: users shown more cards clicked less.

The experiment had enough power. Its event logging looked intact. The analysis itself ran on a mature platform. Yet a data-quality warning showed that fewer users appeared in the treatment group than the planned allocation required.

That warning was a sample ratio mismatch. After the team found and resolved its cause, the apparent negative result became positive. The feature was not the only thing separating treatment from control; the comparison itself had been distorted.

The case comes from Microsoft and Booking.com researchers’ taxonomy of sample ratio mismatch causes. It captures why SRM belongs before every lift chart and executive summary. A precise estimate from a biased comparison is still the wrong answer.

What sample ratio mismatch actually tells you

Sample ratio mismatch occurs when the observed number of units in each variant differs significantly from the allocation the experiment was configured to produce.

Suppose a test intends to assign users fifty-fifty. A small difference between the groups is expected because random assignment does not produce perfectly equal counts every time. The question is whether the difference is plausible given the total sample.

This is why eyeballing a split is unreliable. In the Microsoft research, a 50.2/49.8 split across roughly 1.64 million users was unlikely to occur by chance. The percentages looked harmless. The sample size made the discrepancy diagnostic.

SRM does not tell you which system failed. It tells you the causal comparison may no longer be trustworthy. The treatment group may contain a different population, lose telemetry at a different rate, enter the analysis through a different filter, or be processed differently downstream.

The business consequence is simple: a team can ship a losing experience because biased traffic made it look profitable, or reject a useful change because missing treatment users made it look harmful. SRM is not statistical housekeeping. It is a decision-integrity check.

That is the broader lesson from reviewing more than 200 A/B tests: the validity checks around an estimate are part of the result, not administrative detail to complete after the decision.

Detect SRM before reading the result

For a fixed allocation, the usual check is a chi-squared goodness-of-fit test comparing observed counts with expected counts.

For a two-variant fifty-fifty test:

1. Count the experiment’s randomization units in control and treatment.
2. Calculate the expected count for each variant from the planned allocation and total sample.
3. Compare observed and expected counts with the SRM test.
4. If the result crosses your predeclared threshold, hide the treatment-effect readout and investigate.

Microsoft’s Experimentation Platform documents a threshold of `p < 0.0005`. That is an example from a large-scale platform, not a universal constant. Use the standard built into your platform or select one before anyone sees which variant is winning. Changing the threshold after opening the result recreates the motivated reasoning the check is meant to prevent.

Always count the unit that was randomized. If assignment happens by user but the SRM check counts sessions, repeat visitors can create an apparent mismatch even when user assignment worked correctly. A valid check starts by aligning the assignment unit, exposure unit, and analysis unit.

Unequal allocations are also valid. A ninety-ten risk ramp should be checked against ninety-ten, not against fifty-fifty. If allocation changes during a ramp or bandit experiment, compare observed counts with the expected allocation for each time period rather than with one static ratio.

Where sample ratio mismatch enters the pipeline

The research taxonomy divides experiment delivery into four stages. This is a more useful diagnostic map than a flat list of possible bugs.

1. Assignment

The allocation system may bucket users incorrectly, use unstable identifiers, let users self-select into a variant, or expose ramp changes that were not reflected in the expected counts.

Start here when the mismatch exists in assigned-user logs before treatment is delivered.

2. Execution

The variants may not reach assigned users equally. Redirect latency, crashes, browser incompatibility, cache behavior, or variant-specific performance can make one experience disappear before exposure is recorded.

Start here when assignment counts are clean but exposure counts diverge.

3. Log processing

Telemetry can be lost, joined incorrectly, deduplicated differently, or filtered asymmetrically. A pipeline can create SRM after the product experience has already happened.

Start here when raw exposure logs look balanced but the analysis table does not.

4. Analysis

Triggered analyses and post-assignment filters can select users differently across variants. A treatment may change the behavior used to decide who enters the analysis, which makes the filter part of the treatment rather than a neutral eligibility rule.

Start here when the full assigned population is balanced but a filtered scorecard has SRM.

This four-stage model turns “randomization is broken” into a concrete question: at which handoff do balanced counts first become unbalanced?

A practical SRM diagnostic sequence

The fastest investigations narrow the search in a fixed order. Looking at every dashboard simultaneously creates noise and makes a flattering explanation too easy to choose.

Start with the full population

Compare the unfiltered assigned population with the triggered or filtered analysis. If the full population is balanced and the filtered view is not, investigate the trigger condition before the bucketing system.

A valid trigger must identify comparable control users who would have been eligible for treatment. “Users who clicked the new treatment component” cannot do that because the control group had no opportunity to click it.

Verify the counting unit

Confirm that the SRM test counts the unit used for randomization. Then inspect how identity resolution, cross-device behavior, cookie deletion, or session deduplication changes those counts.

This catches a common false diagnosis: blaming assignment when the analysis quietly changed from users to sessions.

Plot the ratio over time

Calculate observed and expected counts by hour or day. Find the first interval where the mismatch appears and compare it with launches, ramp changes, targeting edits, outages, cache changes, and telemetry releases.

The Microsoft research describes time slicing as one of the most useful ways to distinguish a start-up effect from a persistent failure. A sharp drift date often identifies the system change worth investigating.

Split by diagnostic segments

Check browser, device, geography, traffic source, app version, authentication state, and other segments that map to a plausible delivery mechanism.

The goal is not to hunt for a segment without SRM so the result can be saved. It is to locate the mechanism. A mismatch isolated to one browser version points toward execution or logging. Mismatches across unrelated experiments point toward shared infrastructure.

Follow the counts downstream

Compare counts at assignment, delivery, exposure logging, event collection, data joins, and the final scorecard. The first stage where the ratio changes is more informative than the final ratio alone.

If separate pipelines process the same exposure data, compare them. Agreement does not prove correctness, but disagreement sharply narrows the investigation.

Stop interpreting outcomes until the cause is known

An SRM check should run before treatment effects are revealed. At Microsoft, analyses must pass the SRM gate before experiment outcomes become available. That order matters because once a team sees a large lift, every exclusion rule starts to look more reasonable.

Do not ship because the primary metric is significant. Do not kill the treatment because it looks negative. Do not reweight the groups and assume the bias has disappeared. Reweighting can adjust visible counts; it cannot reconstruct missing users or prove that the remaining groups are exchangeable.

The next decision depends on the root cause, not on whether the result is attractive.

Can an experiment with SRM ever be salvaged?

Sometimes, but only after the cause is identified and its affected population can be isolated without using treatment outcomes to choose the exclusion.

The Microsoft taxonomy describes an experiment where a misconfigured search campaign forced users into one variant. Once the team established that mechanism, it could exclude those force-assigned users and analyze users who remained randomly assigned. The same paper warns that rerunning will be necessary in most cases.

That distinction is the rule:

• Consider a restricted analysis when the root cause is known, the affected records are mechanically identifiable, and removing them restores the original randomized comparison for a clearly defined population.
• Rerun when the cause is unknown, the contamination may extend beyond visible records, the treatment itself changes who is observed, or the proposed cleanup was chosen after comparing outcomes.

“The mismatch was only present during the losing week” is not a diagnosis. It is outcome-based trimming. A credible salvage case explains the system failure first and the eligible population second, without needing the lift chart to justify either choice.

Build SRM into the operating system

The 2019 cross-company study examined more than 10,000 online experiments and found SRM in approximately 6% of Microsoft experiments during the period analyzed. That is not an estimate for every company. It is evidence that the problem persisted even inside a mature experimentation organization.

The operational response should not depend on analysts remembering one more checkbox:

1. Run SRM automatically on every experiment and every planned allocation.
2. Show the check before treatment effects, not beside them.
3. Monitor assignment and exposure counts separately.
4. Preserve allocation changes, ramp history, trigger definitions, and pipeline releases.
5. Alert on variant edits or forced assignments during execution.
6. Run A/A tests to detect shared allocation or telemetry failures.
7. Record the stop, investigate, salvage, and rerun rules before launch.

This is the senior operating principle: an experiment result is not decision-ready merely because the effect estimate is precise. The chain that produced the comparison must also be intact.

SRM is one part of that chain. Why most A/B tests fail covers the surrounding program conditions that make weak hypotheses, instrumentation problems, and unreliable decisions repeatable rather than exceptional.

FAQ

Is a 53/47 split always sample ratio mismatch?

No. The answer depends on the intended allocation and total sample. With a small test, that split may be plausible by chance. With a large test, a much smaller percentage difference can be highly unlikely. Use counts and the statistical check rather than a fixed percentage rule.

Should every experiment use a fifty-fifty allocation?

No. Unequal allocations can reduce exposure to a risky treatment or support staged ramps. Test the observed counts against the allocation that was actually planned for each period.

Can I fix SRM by reweighting the variants?

Not by default. Reweighting does not identify why users are missing or why one group entered the analysis differently. Diagnose the mechanism first. If randomization cannot be recovered for a defensible population, rerun.

Should I check SRM only at the end?

No. Check during the test so delivery or telemetry failures do not waste an entire experiment cycle. Keep the treatment-effect readout hidden until the integrity check passes.

Does server-side experimentation eliminate SRM?

No. Server-side assignment avoids some browser-execution failures, but unstable identifiers, forced assignment, logging loss, filtering, joins, and analysis errors can still create a mismatch.

Bottom line

Sample ratio mismatch is a stop sign for causal interpretation, not a cosmetic warning beside the result. It tells you that treatment and control may no longer represent the comparison you planned.

Check it before reading lift. Find the first point where the ratio breaks. Diagnose assignment, execution, logging, and analysis in order. Salvage only when a known mechanism leaves a defensible randomized population; otherwise fix the system and rerun.

If your experimentation program needs an independent review of its assignment, measurement, or decision process, explore conversion rate optimization consulting.