How to A/B Test Your LLM: A Complete Guide

You ship a new prompt. Outputs look better to you. But are they actually better for your users? Without a controlled experiment, you are guessing. And guessing with LLMs is particularly dangerous because the same prompt can produce wildly different outputs across runs.

This guide walks you through setting up statistically sound A/B tests for LLM-powered features, from defining metrics to calling it with the Winnow SDK.

Why Standard A/B Testing Doesn't Work for LLMs

Traditional A/B testing was designed for deterministic systems. A button is either blue or green, and every user in variant B sees the same green button. With LLMs, you face three compounding problems:

Non-determinism. Even at temperature 0, different hardware, batching strategies, and quantization levels can produce slightly different outputs. At temperature 0.7 or higher, every call is a roll of the dice. This means variance within a single variant is much higher than in traditional A/B tests, and you need more samples to reach significance.

Multi-dimensional quality. A prompt change might improve factual accuracy but reduce conversational tone. Traditional conversion metrics (click-through rate, purchase rate) are scalar. LLM quality is a vector: helpfulness, harmlessness, honesty, relevance, formatting, latency, cost. You need to decide which dimensions matter before you start.

Delayed and subjective feedback. Users rarely click a thumbs-up button. The signal you get is often indirect -- session length, retry rate, escalation to a human. These proxy metrics carry noise.

Choosing the Right Metrics

Before writing a single line of experiment code, define your metrics. Good LLM experiment metrics have three properties:

1. Measurable automatically. If a human has to label every output, your experiment will either take forever or run on a sample too small to matter. Use LLM-as-judge evaluations, regex pattern checks, latency measurements, or downstream user actions.

2. Sensitive to the change. If you changed the system prompt to improve citation formatting, measure citation accuracy -- not overall user satisfaction, which has a hundred other drivers.

3. Directional. You need to know whether higher is better or lower is better. Define this up front so your statistical test knows what to look for.

Here is a practical set of metrics for a customer support chatbot:

• Resolution rate -- did the user's issue get resolved without escalation?
• Response latency (p50, p95) -- how long did the user wait?
• Hallucination rate -- detected via an automated fact-checking judge
• Token cost per conversation -- important for unit economics
• User satisfaction (CSAT) -- collected via post-conversation survey

Why mSPRT Over Fixed-Horizon Tests

In a classic fixed-horizon test, you decide the sample size in advance, run the experiment until you hit that number, then analyze. The problem: you cannot peek at results early. If you do, your false positive rate inflates rapidly. This is called the peeking problem.

With LLM experiments, you almost always want to peek. Model outputs can be catastrophically bad -- hallucinating medical advice, leaking PII, producing toxic content. You need the ability to monitor continuously and stop early if something goes wrong.

mSPRT (mixture Sequential Probability Ratio Test) solves this. It is designed for continuous monitoring. You can check results after every observation without inflating your false positive rate. If one variant is clearly better (or clearly worse), you stop early and save traffic. If the variants are close, you keep running until you reach the power you need.

The key insight is that mSPRT adjusts the significance threshold based on how many times you have looked at the data. Early in the experiment, the threshold is very high -- you need overwhelming evidence to call a winner. As more data accumulates, the threshold relaxes toward the traditional p < 0.05 level.

Setting Up Your First Experiment with Winnow

Here is a step-by-step walkthrough using the Winnow SDK.

Step 1: Install and Initialize

npm install @winnow/sdk

import { Winnow } from '@winnow/sdk'

const winnow = new Winnow({
  apiKey: process.env.WINNOW_API_KEY,
  environment: 'production',
})

Step 2: Define the Experiment

Create an experiment in the Winnow dashboard or via the API. Specify your variants, traffic allocation, and primary metric.

// In your application code, get the assigned variant
const variant = await winnow.getVariant('support-prompt-v2', {
  entityId: userId,  // ensures consistent assignment
})

The entityId parameter is critical. It ensures the same user always sees the same variant across sessions. Without it, a user might get variant A on Monday and variant B on Tuesday, polluting your results.

Step 3: Apply the Variant

const systemPrompt = variant === 'control'
  ? 'You are a helpful customer support agent...'
  : 'You are an expert customer support agent. Always cite the relevant help article...'

const response = await openai.chat.completions.create({
  model: 'gpt-4o',
  messages: [
    { role: 'system', content: systemPrompt },
    ...conversationHistory,
  ],
})

Step 4: Track Metrics

// Track latency
winnow.track('response_latency_ms', latencyMs, {
  experimentId: 'support-prompt-v2',
  entityId: userId,
})

// Track resolution (later, when the conversation ends)
winnow.track('resolved_without_escalation', resolved ? 1 : 0, {
  experimentId: 'support-prompt-v2',
  entityId: userId,
})

Step 5: Monitor and Decide

Open the Winnow dashboard. The mSPRT engine runs continuously. You will see:

• Current lift -- the estimated difference between variants
• Confidence interval -- how certain the estimate is
• Recommendation -- keep running, stop (winner found), or stop (no detectable difference)

When the engine reaches the significance threshold, it flags the winner. You can then promote the winning variant to 100% traffic with a single click.

Common Pitfalls

Testing too many things at once. If variant B has a different system prompt, a different model, and a different temperature, you will not know which change drove the result. Change one variable at a time.

Ignoring guardrail metrics. You might optimize for resolution rate and accidentally increase hallucination rate. Always track safety metrics alongside your primary metric, even if they are not the metric you are optimizing.

Underpowered experiments. LLM outputs have high variance. A test that would reach significance in 1,000 samples for a button color change might need 10,000 samples for a prompt change. Use a power calculator and be patient.

Not accounting for model updates. If your LLM provider ships a model update mid-experiment, your results may be confounded. Pin your model version for the duration of the experiment.

What's Next

A/B testing is the foundation, but it is just the beginning. Once you have reliable metrics and a testing framework, you can move to multi-armed bandits (which automatically shift traffic toward winners), Bayesian optimization over prompt parameters, and eventually autonomous optimization with the Winnow agent.

Start with a single experiment. Get comfortable with the workflow. Then scale.