# Sample Ratio Mismatch (SRM): How to Detect and Fix It

> A sample ratio mismatch breaks your A/B test's randomization. Run the chi-square check, then diagnose and fix the real cause before you trust the result.

- URL: https://missiongrowth.io/blog/sample-ratio-mismatch
- Published: 2026-08-13 · Updated: 2026-09-24
- Author: Ömer Furkan Aktaş, Founder, Mission Growth
- Publisher: Mission Growth. Company facts: https://missiongrowth.io/llms.txt

Your A/B test just tripped a sample ratio mismatch alert, or you're staring at a result you don't fully trust and wondering if you should.

Either way, the question is the same: can you trust the split your test actually ran on?

A sample ratio mismatch (SRM) means the traffic split your test logged doesn't match the split you configured, badly enough that random chance doesn't explain it. That's a randomization problem, not a metric one, and it comes before any p-value on the outcome you actually care about.

This guide gives you the chi-square check to run yourself, the alert threshold four platforms actually use, the five places in your pipeline an SRM typically starts, and an ordered way to find out which one you have.

In this guide:
- The chi-square test for SRM, with the formula and two worked examples
- The sample ratio mismatch threshold four vendors actually use
- The sample ratio mismatch causes, sorted into five pipeline stages
- A cost-ordered checklist for diagnosing which cause you have

## What is a sample ratio mismatch (SRM)?

A sample ratio mismatch is a statistically significant gap between the traffic split you configured for an A/B test and the split your data actually shows.

It means the comparison is untrustworthy. It does not mean any single number is "wrong."

Three words carry the whole idea, and competitor pages use "sample" three different ways, so it's worth pinning down once:

- **Sample.** The users, accounts or sessions assigned to each variant, counted by whatever unit your randomization used, never a proxy metric like page views.
- **Ratio.** The traffic split you configured for the test.
- **Mismatch.** A gap between that configured ratio and what your logged data actually shows, large enough to be a real signal rather than noise.

A fixed split is what lets an SRM check apply at all; a [multi armed bandit vs a/b testing decision](https://missiongrowth.io/blog/multi-armed-bandit) reassigns traffic continuously instead, so this guide's check doesn't apply the same way there.

An SRM is a data-integrity gate on the comparison, not a correctness check on the metric values themselves. Wikipedia defines SRM as "a statistically significant difference between the expected and actual ratios of the sizes of treatment and control groups in an experiment."

Vendors frame it as something that "breaks randomization." Both framings are true, but neither says the thing readers actually need to hear.

Your conversion rate, your revenue number, whatever the metric shows, can still be arithmetically correct while the comparison built on top of it is worthless. Statistical significance can even still appear. What you've lost is the ability to attribute the difference between groups to your treatment, because the groups are no longer the random groups the test assumed.

## How common is SRM, and what's actually at stake?

SRM shows up in roughly 6% of experiments at Microsoft, measured across five products over one year.

It also shows up in about 10% of LinkedIn's triggered analyses specifically, a narrower, historical rate from before LinkedIn built detection tooling. Those are two different, dated, named studies: Fabijan, Gupchup, Gupta, Omhover, Vermeer and Dmitriev's 2019 KDD paper for the Microsoft number, and Chen, Liu and Xu's 2018 paper for the LinkedIn one.

Kameleoon's own blog blends both into a single unsourced line:

> "research shows, about 6-10% of all tests run end up with a SRM issue"
> — Kameleoon

It names no study for either end of that range. The two real numbers aren't interchangeable: 6% covers all experiments at one company. 10% covers only a subset, triggered and zoomed-in analyses, at a different company, and reflects a period before that company had detection tooling in place.

The scale is why an SRM check should be routine. A company running ten thousand experiments a year, per the same Microsoft study, can expect to see at least one SRM a day. That's a background rate worth planning around, especially once you're tracking [how many growth experiments you run a week](https://missiongrowth.io/blog/growth-experiment-cadence).

## How to test for SRM: the chi-square check

The chi-square goodness-of-fit test detects SRM by comparing your observed group sizes to the sizes you'd expect under your configured split.

It uses the count of users or sessions assigned to each variant, never a proxy metric like page views. A returning user can inflate a visit-based count and produce a false read.

The formula, Pearson's chi-squared goodness-of-fit test:

**χ² = Σ(O − E)² / E, with df = groups − 1**

O is your observed count per group, E is the expected count under your configured split, and df is one less than the number of groups (1 for a standard two-way test).

This sample ratio mismatch example shows where the line actually sits: the same test structure, one case that clearly fails and one that clearly passes.

| Case | Total users | Expected split | Observed split | χ² | df | p-value | Verdict |
|---|---|---|---|---|---|---|---|
| Fail | 24,000 | 12,000 / 12,000 (50/50) | 11,640 / 12,360 (48.5% / 51.5%) | 21.6 | 1 | p ≈ 3.36×10⁻⁶ | SRM, reject |
| Pass | 10,000 | 5,000 / 5,000 (50/50) | 4,950 / 5,050 (1% deviation) | 1.0 | 1 | p ≈ 0.317 | No SRM |

The fail case's p-value, 3.36×10⁻⁶, is far below any reasonable threshold. The pass case's p-value, 0.317, sits comfortably inside normal sampling noise.

Same test, same formula, two different verdicts. What the test measures is the deviation's size relative to your total sample count, never the raw deviation on its own.

::figure{src="/blog/figures/sample-ratio-mismatch-1.svg" alt="A sample ratio mismatch worked example: a 24,000-user test fails at p ≈ 3.36×10⁻⁶ while a 10,000-user 1% deviation passes at p ≈ 0.317." caption="A 24,000-user test with an 11,640/12,360 split fails the SRM check at p ≈ 3.36×10⁻⁶, while a 10,000-user test with a 1% deviation passes at p ≈ 0.317." width="720" height="271"}

One measurement detail decides whether your own numbers land in the fail column or the pass column: count the users (or accounts, or sessions, whichever unit your randomization actually used), never visitors or page views. A returning user inflates a visit-based count without touching the true group size, which can manufacture an SRM that isn't there or hide one that is.

## What p-value threshold should trigger an alert?

The p-value threshold that should trigger an SRM alert ranges from 0.01 down to 0.0005 across vendors, far stricter than the conventional 0.05 significance level.

Most teams use that looser 0.05 cutoff for the test's own metric, not for an integrity check like this one.

| Vendor | Default alert threshold | Note |
|---|---|---|
| Statsig | p < 0.01 | "Exposures are Balanced" check |
| Kameleoon | p < 0.001 | Default SRM alert |
| GrowthLayer | p < 0.001 | 0.01 optional, described as more sensitive |
| Microsoft (internal ExP platform) | p < 0.0005 | Stated as conservative, to reduce false positives |

::figure{src="/blog/figures/sample-ratio-mismatch-2.svg" alt="Dot-range chart of SRM alert thresholds for Statsig, GrowthLayer, Kameleoon and Microsoft, all stricter than the 0.05 significance level." caption="Vendor default SRM alert thresholds cluster between p 0.01 and p 0.0005, all stricter than the conventional 0.05 significance level." width="720" height="314"}

Microsoft's internal threshold, p < 0.0005, is 100x stricter than the conventional significance level of 0.05 (dividing 0.05 by 0.0005 gives that ratio). That's not an arbitrary choice: run enough experiments in parallel and a loose cutoff starts flagging false alarms constantly.

Pick your own threshold by that logic. A small team can use a looser cutoff than a company running thousands of experiments a year; match your cutoff to your volume, not to whichever vendor's default ships in your tool.

## What causes SRM: the five places it starts

SRM causes sort into five stages of the experiment pipeline, a taxonomy of 25 named causes from Fabijan et al.'s 2019 KDD paper.

That paper is the only large-scale industry study of the problem. Its five stages trace the pipeline itself: a user gets assigned to a variant, the assigned variant actually executes, the exposure gets logged, the log gets pulled for analysis, and people can intervene at any of those points, which is why the taxonomy sorts by where in that flow a cause originates rather than by symptom.

| Stage | What it covers |
|---|---|
| Experiment assignment | How users get bucketed into a variant: hashing, targeting rules and the traffic-split configuration itself |
| Experiment execution | Whether an assigned user actually receives and experiences the intended variant once the code runs |
| Experiment log processing | Whether every exposure event gets logged, deduplicated and joined correctly downstream |
| Experiment analysis | The filters, joins and inclusion rules applied when the data is pulled for analysis |
| Experiment interference | Manual overrides, QA traffic, bot traffic or another team's changes touching the same users mid-test |

::figure{src="/blog/figures/sample-ratio-mismatch-3.svg" alt="Matrix table of Fabijan et al.’s five-stage SRM cause taxonomy: assignment, execution, log processing, analysis and interference." caption="Fabijan et al.’s taxonomy sorts SRM causes into five experiment-pipeline stages, from assignment through human interference." width="720" height="387"}

Knowing which stage a symptom points to narrows the search before you touch a single log. An SRM confined to one segment usually points to something specific in assignment or execution for that segment. An SRM present from day one points further upstream, toward assignment itself.

An SRM that only shows up in a triggered or zoomed-in analysis, but not in the full population, usually traces to log processing or an analysis-stage filter rather than to how users were bucketed in the first place.

## Diagnosing and fixing SRM when it fires

SRM diagnosis works cheapest check first: confirm the alert, isolate it, match it against the taxonomy, then escalate only if those simple steps fail.

In cost order:

1. **Confirm the alert.** Rerun it as an A/A test, or check it against your pre-period baseline, before you touch anything else. This is the cheapest step and the fastest signal: an SRM that shows up in an A/A test almost always points to something systemic in assignment or logging rather than to your specific experiment.
2. **Isolate the pattern.** Does it appear in one segment, one time window, or only in a triggered or zoomed-in analysis? The shape of where it shows up is itself diagnostic, per the KDD'19 rules of thumb for investigating scorecards, user segments and time segments.
3. **Match it against the five-stage taxonomy.** Use the pattern from step 2 to point at assignment, execution, log processing, analysis or interference, then check that stage first instead of guessing across all five.
4. **Escalate to a regression check only if the simple steps fail.** DoorDash's method fits a weighted linear regression of treatment assignment on pre-randomization covariates like country and platform, then reads a Wald test on each covariate to find which one predicts assignment. Reach for it only after steps one through three have failed to find the cause.

What not to do: don't filter out the affected segment and ship the rest of the results. Filtering after the fact can introduce a new selection bias instead of removing the old one, which is the exact problem DoorDash's regression method exists to solve properly.

Treat an unresolved SRM as a stop condition rather than a data point to work around. There are exactly two safe endings: cause found and fixed, so you restart the test, or cause unclear, so you don't trust the result.

Don't wait until the test ends to check, either. Optimizely's automatic SSRM detection checks traffic counts daily rather than only at the finish line, because an SRM present mid-test can get diluted and hidden by additional data before the test concludes. It's available only for Stats Engine A/B tests with a manual traffic distribution, running 45 days or less, with at least 1,000 visitors.

Checking continuously without inflating your false positive rate is its own statistics: see [sequential testing](https://missiongrowth.io/blog/sequential-testing) for running that correctly, since alpha spending and always-valid methods sit outside what an SRM check alone covers. The same discipline applies to [peeking and multiple comparisons](https://missiongrowth.io/blog/growth-experiment-cadence): an SRM check is a third, separate integrity gate alongside both.

False positives are the real cost of an unchecked SRM. DoorDash's own platform data shows experiments with an SRM produce roughly twice the rate of statistically significant metrics compared to SRM-free experiments, not a rounding error but the mechanism by which an unchecked SRM quietly doubles your odds of acting on a result that isn't real.

## FAQ

### Does a sample ratio mismatch mean my test result is wrong?
No. It means the causal comparison is untrustworthy, because the groups you're comparing aren't the random groups your test assumed. Your metric values aren't necessarily "wrong," but you can no longer credit the difference between them to your treatment alone. An SRM check gates the comparison; it does not correct or validate the numbers themselves.

### Can SRM happen in an A/A test?
Yes, and it's one of the fastest diagnostic signals you have. An SRM in an A/A test, where both groups get the identical experience, almost always points to a systemic cause in your assignment or logging pipeline rather than anything specific to the experiment you're running.

### How long should I wait before checking for SRM?
Check from day one and keep checking continuously, rather than only once at the end. A mid-test SRM can get diluted and hidden by additional data before the test concludes if you only check once at the finish line. Continuous checking has its own statistics, alpha spending and always-valid tests, covered in the sequential testing guide linked above.

### What p-value threshold should trigger an SRM alert?
Vendors range from p < 0.01 (Statsig) down to p < 0.0005 (Microsoft's internal platform), all stricter than the conventional 0.05 significance level. Pick by volume: the more experiments you run in parallel, the stricter your cutoff needs to be to avoid drowning in false alarms.

### Can SRM happen with Bayesian A/B tests?
Yes. SRM is about whether the groups you observed are the random groups you assumed, and every statistical method, frequentist or Bayesian, depends on that assumption holding. A skewed split undermines a Bayesian posterior exactly the way it undermines a p-value.

### Should I just filter out the affected segment and keep the rest of the data?
No, not without first confirming that segment is genuinely the cause. Filtering after the fact can introduce a new selection bias instead of removing the old one, which is why DoorDash built a regression method to find the actual cause instead of filtering blind.

An SRM check is a randomization-integrity gate you run before you trust any A/B test result, not a substitute for the test's own p-value. Diagnosing one is a structured search through five known pipeline stages, not a guess through an unordered list of causes. The next test that finishes, run the chi-square check against your final counts before you read the p-value on the metric you actually care about.