Understanding mSPRT: Sequential Testing for AI Experiments

When you run an A/B test on an LLM pipeline, you want to know the answer as fast as possible. Fixed-horizon tests force you to wait for a predetermined sample size. Bayesian methods give you posterior distributions but require prior specification. mSPRT gives you something valuable: the ability to check your results at any time, stop as soon as you have enough evidence, and maintain rigorous false positive control throughout.

This post explains how mSPRT works, why it matters for AI experiments, and when you should use it versus alternatives.

The Problem with Fixed-Horizon Tests

In a classical fixed-horizon test, you calculate the required sample size before starting, run the experiment until you hit that number, then compute a p-value. The guarantee: if there is no true difference between variants, you will falsely declare a winner less than 5% of the time (assuming alpha = 0.05).

The catch: this guarantee only holds if you look at the data exactly once, at the predetermined sample size. If you peek at the results halfway through and see p = 0.04, you might be tempted to stop. But that p-value is misleading. The probability of seeing p < 0.05 at some point during the experiment, even when there is no real effect, can be 20-30% or higher depending on how often you check.

This is the peeking problem, and it is not a theoretical concern. In practice, everyone peeks. Product managers ask for updates. Engineers check dashboards. Leadership wants to know if the new model is working. Pretending nobody will look at the data is unrealistic.

For LLM experiments specifically, peeking is not just tempting -- it is necessary. If a new prompt is causing hallucinations, you need to catch it immediately, not after 10,000 observations.

How Sequential Testing Works

Sequential testing flips the script. Instead of committing to a fixed sample size and analyzing once, you analyze continuously and adjust your decision boundary to account for multiple looks.

The fundamental insight goes back to Abraham Wald's Sequential Probability Ratio Test (SPRT) from the 1940s, originally developed for quality control in manufacturing. The idea: at each observation, compute the likelihood ratio between the alternative hypothesis (there is an effect) and the null hypothesis (there is no effect). When this ratio exceeds a threshold, reject the null. When it drops below another threshold, accept the null. Otherwise, keep collecting data.

Classic SPRT has a limitation: you must specify the exact effect size under the alternative hypothesis. If you guess wrong, the test is inefficient. This is where the "mixture" in mSPRT comes in.

The mSPRT Approach

mSPRT, introduced by Johari et al. (2017) at Stanford (and later refined at Netflix and Spotify), replaces the point alternative with a mixture over possible effect sizes. Instead of testing "is the effect exactly 3%?", you test "is there some non-zero effect?" by integrating over a distribution of plausible effect sizes.

Here is the core idea, made accessible:

Step 1: Define the mixing distribution. You choose a prior distribution over effect sizes. A common choice is a normal distribution centered at zero with a variance parameter tau-squared. This tau-squared parameter controls sensitivity: larger values make the test more sensitive to large effects but slower for small effects.

Step 2: Compute the mixture likelihood ratio. At each observation, instead of computing a single likelihood ratio, you compute the expected likelihood ratio averaged over the mixing distribution. For normally distributed data, this has a clean closed-form expression:

The mixture likelihood ratio at time n with observed z-statistic Z_n is:

Lambda_n = sqrt(n * tau^2 + 1)^(-1) * exp(n * tau^2 * Z_n^2 / (2 * (n * tau^2 + 1)))

where tau^2 is the variance of the mixing distribution and Z_n is the z-statistic at time n.

Step 3: Compare to threshold. If Lambda_n exceeds 1/alpha (where alpha is your significance level, typically 0.05), reject the null hypothesis. The beauty: you can evaluate this after every observation without inflating your false positive rate.

Why it works. The mixture likelihood ratio forms a non-negative martingale under the null hypothesis. By the optional stopping theorem (with appropriate technical conditions), the probability that it ever exceeds 1/alpha is bounded by alpha. This is what gives you the "anytime valid" property: no matter when you stop, the type I error guarantee holds.

Practical Implications

Faster Decisions When Effects Are Large

If your new prompt is substantially better (or worse), mSPRT will detect it quickly. In simulations, when the true effect is a 5% lift and you are testing for it, mSPRT reaches a conclusion in roughly 40-60% of the samples a fixed-horizon test would require. For large effects (10%+ lift), the savings are even more dramatic.

This is especially valuable for AI experiments where bad variants can cause immediate harm. A prompt that hallucinates dangerous information should be caught in hours, not days.

Controlled Risk When Effects Are Small

When the true difference is tiny (say, 0.1%), mSPRT will keep running without making a premature call. It spends more time in the "keep collecting" state, which is the correct behavior: you need more evidence to detect small effects.

The trade-off: for small effects, mSPRT typically requires somewhat more samples than a properly powered fixed-horizon test (roughly 10-30% more). This is the price of continuous monitoring. In practice, this trade-off is almost always worth it.

Choosing Tau-Squared

The mixing distribution parameter tau-squared controls the test's sensitivity profile. Here are practical guidelines:

• For conversion metrics (binary outcomes like click/no-click): tau^2 between 0.001 and 0.01, depending on the baseline rate and the minimum detectable effect you care about.
• For continuous metrics (like latency, satisfaction scores): tau^2 between 0.01 and 0.1, scaled by the metric's variance.
• If unsure: start with tau^2 = 0.01 for most metrics. Winnow's default calibration handles this automatically based on historical metric variance.

mSPRT vs. Bayesian Testing

Both mSPRT and Bayesian A/B testing allow continuous monitoring. They differ in philosophy and practical details.

Bayesian testing gives you a posterior distribution over the effect size and a "probability of being best" for each variant. The advantage: you get a richer picture than a binary significant/not-significant answer. The disadvantage: results depend on your prior, and converting "probability of being best" into a decision (when do you stop?) is not straightforward without additional decision-theoretic machinery.

mSPRT gives you a frequentist guarantee: the probability of a false positive is bounded by alpha, regardless of when you stop. The advantage: clear decision rules and error control that does not depend on prior specification. The disadvantage: you get less information about the effect size distribution (though you can compute confidence intervals alongside the test).

In practice, at Winnow we use mSPRT as the primary decision engine because its error guarantees are easier to communicate and audit. We supplement it with Bayesian estimation for effect size visualization in the dashboard. You get the rigor of frequentist control with the interpretability of Bayesian posteriors.

When to Use What

Use mSPRT when:

• You need continuous monitoring (most production experiments)
• Regulatory or compliance requirements mandate frequentist error control
• You want clear, automated stopping rules
• You are running many experiments simultaneously and need consistent false positive control

Use fixed-horizon tests when:

• You have a fixed data collection window that will not change (rare in practice)
• The cost of extra samples is negligible and you do not need early stopping
• You are running a one-off study for a research paper

Use Bayesian methods when:

• You have strong prior information from previous experiments
• Decision costs are asymmetric (type I and type II errors have very different costs) and you want to encode this explicitly
• You need posterior probabilities for downstream decision models

Implementation Notes

For teams implementing mSPRT themselves (rather than using Winnow's built-in engine), a few technical notes:

Variance estimation. The z-statistic in the mSPRT formula requires an estimate of the metric variance. Use a pooled variance estimate that updates as data arrives. Be careful with small samples (n < 30): use a t-distribution correction or start computing the statistic only after a burn-in period.

Multiple metrics. If you are testing multiple metrics simultaneously, apply a correction (Bonferroni or, better, the Benjamini-Hochberg procedure) to your per-metric alpha. mSPRT controls false positives per metric, not across the family.

Non-normal data. mSPRT's theoretical guarantees assume approximately normal test statistics. For binary metrics, this is fine with moderate sample sizes (n > 100 per variant) by the central limit theorem. For highly skewed metrics like revenue, consider a variance-stabilizing transform (log or square root) before computing the z-statistic.

Wrapping Up

Sequential testing is not a luxury for AI experiments -- it is a necessity. LLM outputs are non-deterministic, failure modes can be severe, and the velocity of prompt iteration demands fast feedback loops. mSPRT gives you continuous monitoring with rigorous statistical guarantees, letting you stop early when the answer is clear and keep running when it is not.

The math is elegant, but the practical benefit is simple: you make better decisions, faster, with controlled risk.