# Sequential Testing: A Guide to Valid Early Stopping

> Sequential testing lets you check A/B results early without inflating false positives. See how the methods work and which vendors use which one.

- URL: https://missiongrowth.io/blog/sequential-testing
- Published: 2026-08-14 · Updated: 2026-09-24
- Author: Ömer Furkan Aktaş, Founder, Mission Growth
- Publisher: Mission Growth. Company facts: https://missiongrowth.io/llms.txt

Sequential testing lets an A/B test check its own results early without breaking the 5% false-positive rate on the dashboard. Every vendor's page says some version of that. What none of them say, together, is which vendor runs which underlying method, and what "always valid" actually depends on to hold.

Spotify's 2023 engineering research is the only page that ever mapped named vendors to statistical families. It credited Optimizely, Uber, Netflix and Amplitude with mSPRT, Eppo with GAVI, and Statsig with what it called a "corrected-alpha approach." It's three years old, and nobody has reconciled it since.

Statsig's current docs describe an mSPRT-based method with no mention of "corrected-alpha," and two of today's top-ranking vendors, LaunchDarkly and Harness, aren't on Spotify's list at all.

In this guide: what sequential testing is and how it differs from a fixed-horizon test, why checking early breaks a normal p-value and the two families that fix it, a current vendor map for five tools, a four-step setup checklist mapped to an actual toggle, why an early-stopped effect size runs hot, and the two questions to ask before you trust any vendor's "always valid" claim.

## What is sequential testing?

Sequential testing analyzes an A/B test's data as it accrues, using pre-set statistical boundaries instead of one fixed sample size, so you can stop once the evidence is strong enough.

A fixed horizon test picks a sample size up front, runs until it hits that number, then looks at the result once. Sequential testing works differently:

- Sets efficacy and futility boundaries before the test starts
- Checks the data at multiple points instead of one
- Stops as soon as either boundary is crossed, or continues to the planned maximum

The term is older than online experimentation, and three fields reuse it for the same underlying idea. Statistician Abraham Wald developed the sequential probability ratio test for WWII-era industrial quality control, publishing it in a June 1945 paper in the Annals of Mathematical Statistics. Clinical research, manufacturing quality control and product experimentation all kept his method's name for the same underlying idea.

The same idea also goes by "sequential analysis," the broader statistical field the method belongs to, and "sequential sampling," its manufacturing and quality-control sense. That's the sense used in what follows: sequential A/B testing, the toggle inside an A/B testing tool, whether you're testing a checkout flow, a pricing page or an onboarding email.

Sequential testing decides when to stop a single hypothesis test. That's a different question from a [multi-armed bandit](https://missiongrowth.io/blog/multi-armed-bandit), which continuously reallocates traffic toward the better-performing variant instead of running a fixed test with a stop/go decision at all.

## Why checking early breaks a normal test, and the two ways to fix it

Checking a standard A/B test's p-value more than once inflates the real false positive rate past the number on the dashboard. Sequential testing fixes it with one of two mathematically different designs.

Here's the arithmetic behind that inflation. A z-test built for a 5% false positive rate, checked just twice during data collection, has a true false positive rate close to 10%, according to Spotify's 2023 engineering research.

Every extra look gives a random fluctuation one more chance to cross the 5% threshold, even with no real effect at all. That's the peeking problem: the 5% figure assumes you looked once, and most teams look many times.

Two families of methods fix it, and they fix it differently:

- **Group sequential design.** Set a maximum sample size in advance, then check the data at a few pre-planned points, using a spending function that allocates a shrinking slice of your error budget to each look so the total stays at your target significance level.
- **Always valid inference (mSPRT / GAVI).** Build the test around a pre-committed mixing distribution over possible effect sizes, so the resulting statistic stays valid no matter how often, or whenever, you check it, with no fixed maximum sample size to pre-declare.

::figure{src="/blog/figures/sequential-testing-3.svg" alt="One method locks in a maximum sample size before the test starts; always valid inference (mSPRT/GAVI) uses a mixing distribution instead, with no sample cap." caption="One family plans a maximum sample size upfront; the other never needs one." width="720" height="283"}

An always valid p-value doesn't stay valid by luck or by vendor promise. The mixture-based test statistic behind it is a nonnegative supermartingale under the null.

Ville's maximal inequality, the tool Howard, Ramdas, McAuliffe and Sekhon use in their 2021 confidence-sequence paper, bounds the chance that statistic ever crosses the rejection threshold, no matter when you stop looking. That's also the paper Spotify's own page points to for the mechanism, without explaining it further.

Johari, Pekelis and Walsh's 2015 paper is the one that named and popularized always valid p-values for A/B testing, and it grounds its own guarantee differently: standard martingale techniques per Siegmund 1985, plus law-of-the-iterated-logarithm bounds. Not the supermartingale argument above. It's cited here for what it actually argues, not folded into the Howard et al. mechanism.

A group sequential design's maximum planned sample size runs larger than a fixed horizon test's for the same target power. Weigl and Ponocny's 2020 worked example needed a maximum total sample size of 176 against a fixed horizon test's 170, an inflation factor of 1.034, about 3.4% more.

The sample size a sequential test actually uses is often smaller. It stops as soon as a real effect crosses the boundary; the maximum itself stays the same. Weigl and Ponocny's own real-data application used half its planned total once that happened; see [minimum detectable effect](https://missiongrowth.io/blog/minimum-detectable-effect) for how sample size and detectable effect trade off in a fixed horizon design.

Checking a number early isn't unique to A/B tests. [The same noise problem shows up in AI search visibility KPIs](https://missiongrowth.io/blog/ai-search-visibility-kpis), where day-to-day fluctuation can look like a real change if you read it too soon.

## Which vendor runs which method

Amplitude, Statsig and Eppo each disclose an always valid method today; LaunchDarkly and Harness disclose neither, and neither one appears in Spotify's older vendor list at all.

Amplitude runs mSPRT. Statsig's current docs describe the same mSPRT family, not the "corrected-alpha" approach Spotify credited it with in 2023. Eppo runs its own GAVI-tuned test. That matters because the families trade power differently.

| Vendor | Statistical family | Tuning parameter | Discloses a method name |
|---|---|---|---|
| Amplitude | mSPRT (always valid) | Not stated | Yes |
| Statsig | mSPRT (always valid), based on the approach Zhao et al. propose | Publishes two-sided and one-sided confidence-interval formulas | Yes |
| Eppo | GAVI (always valid confidence sequence) | N_tune = 10,000, set in Eppo's own docs | Yes |
| LaunchDarkly | Not named; adjustments made "behind the scenes" | Not stated | No |
| Harness | Not named; boundaries "large at first, smaller as more data comes in" | Not stated | No |

Spotify ran its own head-to-head simulation to see what these families cost in practice. At 500 users per group and 14 intermittent looks, a correctly-specified group sequential test held 0.90 to 0.93 power depending on the spending function, against 0.72 for GAVI, 0.72 for mSPRT and 0.75 for a plain Bonferroni correction.

::figure{src="/blog/figures/sequential-testing-1.svg" alt="Group sequential design holds 0.90 to 0.93 power at 14 looks, versus 0.72 for GAVI and mSPRT and 0.75 for Bonferroni, per Spotify's 2023 simulation." caption="At 14 looks, a correctly-specified group sequential test holds more power than always valid or Bonferroni alternatives." width="720" height="327"}

At that same sample size, a standard fixed horizon test runs at about 0.935 power. We got that by running a one-sided two-sample z-test at the same 500-per-group, 5% alpha and 0.2-standard-deviation effect Spotify used in its simulation.

A correctly-specified group sequential design costs almost nothing in power against the fixed horizon benchmark here; the always valid families in this simulation gave up more. This is a power comparison at one shared sample size. The maximum-versus-expected question is a separate one, covered above.

What actually costs power is misjudging the maximum sample size. Spotify's own numbers show that underestimating it by 50x costs about 15% power against a correctly-specified GAVI test and about 30% against a correctly-specified group sequential test.

Picking a vendor's sequential testing toggle is really picking a statistical family, whether or not the vendor's page says so, and the one page that used to map vendors to families is three years out of date on at least one of them.

Spotify's 2023 research named Optimizely, Uber, Netflix and Amplitude on mSPRT, Eppo on GAVI, and Statsig on what it called a "corrected-alpha approach." Statsig's own current documentation describes something else: an mSPRT-based method "based on the approach that Zhao et al. propose," with published confidence-interval formulas, no corrected-alpha approach mentioned anywhere.

That's a discrepancy between two real, dated sources, not a claim that Statsig changed its method: Spotify's 2023 page and Statsig's 2026 docs are the only two snapshots on record. Two of today's top-ranking vendors, LaunchDarkly and Harness, aren't in Spotify's list at all, and neither discloses a method name of its own.

## How to set up your first sequential test

Setting up a sequential test comes down to four decisions, made before you collect any data: name the efficacy boundary, name the futility boundary, pick a look frequency, and only then flip the vendor setting.

1. **Set your efficacy boundary (alpha spending).** This is the threshold that says "the variation is better, stop and ship it." An alpha-spending function controls it, allocating your Type I error budget across your planned looks so the total false-positive rate stays at your chosen significance level.
2. **Set your futility boundary (beta spending).** This is the threshold that says "there's no meaningful difference, stop and move on." A beta-spending function controls it the same way, on the Type II error side.
3. **Pick your look frequency.** Decide how often you'll check the data, daily is common, before the test starts, not once you're already watching the number move.
4. **Only then, enable the vendor setting.** In Statsig, that means setting Analytics Type to Frequentist, then enabling Apply Sequential Testing under Analysis Settings; Statsig lets you toggle it at any point during the experiment.

Analytics-Toolkit's glossary already pairs the efficacy and futility boundaries with their alpha-spending and beta-spending functions in one sentence, but no vendor page turns that vocabulary into an ordered sequence mapped to an actual product setting.

Amplitude's page names spending functions only generically, citing O'Brien-Fleming as an example, without the paired alpha-spending/beta-spending vocabulary or a specific vendor UI; Analytics-Toolkit names that paired vocabulary without any setup order or vendor toggle attached. The four steps above are a single ordered, vendor-mapped checklist that combines both.

Before you trust any test result, sequential or not, check for [sample ratio mismatch](https://missiongrowth.io/blog/sample-ratio-mismatch): a randomization-integrity problem that has nothing to do with peeking, but invalidates a sequential test's boundaries just as fast as a fixed-horizon test's p-value.

Variance reduction and sequential testing solve different problems, and they combine cleanly. [CUPED](https://missiongrowth.io/blog/cuped) tightens your estimate at a given sample size; this method decides only when to stop collecting more of it.

## Before you trust an early stop, correct the number

A test that stops early because it crossed its efficacy boundary reports a lift that runs hot, and that bias applies to your one test even though it disappears in the aggregate.

Schou and Marschner's 2013 study found that early stopping "is not a substantive source of bias in meta-analyses," because overestimation from trials that stopped early is balanced out, in aggregate, by underestimation from trials that ran to completion. That finding describes a pool of many trials averaged together. It says nothing about the single test in front of you.

Your one stopped-early test doesn't get the benefit of that averaging. If it crossed its efficacy boundary and stopped, its reported lift is more likely to be an overestimate than a lift measured by a test that ran to its full planned sample.

Treat an early-stopped effect size as directional, not the number you'd bake into a forecast. If the decision hinges on the lift's exact size rather than only its direction, let the test run further, or re-measure with a follow-up test, before committing to that number.

The same discipline applies to reading [SEO ROI](https://missiongrowth.io/blog/seo-roi) too early: a number measured before it's stabilized tells you less than it looks like it does.

## Ask your vendor these two questions before you flip the toggle

Before you turn on a sequential testing setting, ask your vendor two things:

1. **Which statistical family does this run: group sequential or always valid?** A group sequential design needs a maximum sample-size estimate going in; an always valid design needs a mixing distribution or tuning parameter instead. Amplitude and Statsig both disclose mSPRT. Eppo discloses GAVI, with its own N_tune=10,000 setting. LaunchDarkly and Harness disclose neither.
2. **Who chose the tuning parameter, and can you see it?** Eppo states its own N_tune value in its docs. LaunchDarkly's own advice is to leave the setting alone unless your data team specifically wants something different, without naming what actually runs underneath. Harness describes its boundaries as large at first, smaller as more data comes in, and calls results always valid without naming a method or a parameter at all.

::figure{src="/blog/figures/sequential-testing-2.svg" alt="Amplitude, Statsig and Eppo disclose their sequential testing method and tuning parameter; two of the five vendors compared here disclose neither." caption="Amplitude and Statsig name their method; two of the five vendors compared here don't." width="720" height="361"}

An unanswered question here means you're trusting an unstated design choice with your false positive rate. Our [growth experiment cadence](https://missiongrowth.io/blog/growth-experiment-cadence) guide already flags why peeking without a plan is a problem; this page is the fix, a sequential testing method built to be peeked at, once you know which family you're actually running.

Every vendor's "always valid" pitch skips the same thing: the guarantee depends on a design choice you have to make, or ask about, not something the toggle handles for you on its own. Amplitude and Statsig disclose their family; Eppo discloses its family and its tuning parameter; LaunchDarkly and Harness disclose neither.

Ask the two questions above before your next test. If your vendor doesn't have an answer, don't treat "always valid" as a fact you can skip checking.

## FAQ

### Why does a "sequential testing" search also turn up pregnancy or epidemiology results?

Google can't tell from the bare phrase which field you mean. Clinical screening, manufacturing quality control and product experimentation all kept the same statistical name for the same underlying idea.

### Can you give an example of sequential testing?

A checkout test that sets an efficacy boundary and a futility boundary before launch, checks results daily instead of waiting for a fixed date, and stops as soon as either boundary is crossed, is a sequential test. A test that's only analyzed once, at a pre-set sample size, is not.

### What is mSPRT?

mSPRT is a mixture sequential probability ratio test, the always valid method behind Amplitude's and Statsig's sequential testing settings. It stays valid at any stopping time because it's built around a pre-committed mixing distribution rather than one fixed threshold.

### Why implement sequential testing instead of a fixed-horizon test?

It lets a team act on a clear win or a clear loss before the full sample is collected, without inflating the false positive rate the way informal peeking does. The tradeoff is real: you either need a workable estimate of your maximum sample size for a group sequential design, or you accept a measurable power cost for an always valid one.

### Does sequential testing need more total users than a fixed-horizon test?

Its maximum planned sample size runs a bit larger, one published worked example puts the inflation factor at 1.034, about 3.4% more, but the number of users it actually uses is often smaller, because it stops the moment a real effect crosses the boundary. That same worked example needed only half its planned total once that happened. The extra cost only shows up if you badly misjudge the maximum going in.