Write your prompt, define the task, and attach the scoring rubric before you run a single token. The Author stage commits a versioned prompt + eval pair so every downstream result is traceable to the exact specification that produced it.

Prompt versioning · Eval harness · Dataset registry · LLM-as-judge + human-in-the-loop rubrics

author.pypython

from winnow import Eval

Eval("checkout-copy-v2", {
    "data": lambda: load_jsonl("checkout.jsonl"),
    "task": lambda input: llm(prompt=PROMPT_V2, model="claude-haiku-4-5"),
    "scores": [exact_match, factual_accuracy, length_penalty],
})

Evaluate

The stats layer nobody else publishes.

Offline and online share one engine. Every experiment ships with a p-value, a confidence interval, a power analysis, and an SRM check — all the things that tell you whether the number you are reading is real. mSPRT is the default because it is safe to peek. Welch, Bayesian, and CUPED are one parameter away.

mSPRT · Welch · Bayesian · CUPED · Chi-square SRM · delta method · sequential SPRT confidence sequences

evaluate.pypython

# Offline: run the eval harness
result = Eval("checkout-copy-v2").run()
print(result.p_value, result.confidence_interval)

# mSPRT: safe-to-peek sequential test (default)
# Switch to Bayesian or CUPED in one line:
result = Eval(..., test="bayesian").run()

Ship

Promoting a winner is a gate change, not a deploy.

Once an experiment closes, you roll the winner out behind a gate or a dynamic config. Gates are boolean; configs are typed JSON. Both evaluate per-user with a rule stack (user in list, email match, percent rollout). Guardrails with critical severity act as automatic kill switches when a post-launch metric regresses.

Gates · Dynamic configs · Guardrails (warn / high / critical) · Bandits · Instant rollback

ship.pypython

if winnow.check_gate("new_checkout_flow", user=u):
    render_new_flow()
else:
    render_old_flow()

# Configs carry typed params the experiment just validated
params = winnow.get_config("ranker_params", user=u)

Trace

Every call, every score, every dollar.

Instrument once and every LLM call becomes a trace: prompt, model, parameters, latency, cost, quality score. Traces carry an experiment_id and arm_id automatically, so online A/B data is a natural byproduct of doing the work — not a second pipeline to maintain.

25M+ traces/mo on Pro · 100M/mo on Team · 1B+ rows in a warehouse-grade column store for cross-experiment analytics

trace.pypython

from winnow import Winnow
winnow = Winnow()

# Auto-instrument: patches the Anthropic / OpenAI client so
# every call logs a trace with score, latency, cost.
winnow.init(auto_instrument=["anthropic"])

Score online

Real traffic, real scores, real time.

After shipping, traces flow through the online scoring pipeline. LLM judges evaluate each production call against the same rubric used offline so drift between lab and prod surfaces immediately. CUSUM detectors flag metric regressions; calibration checks flag when judges disagree.

LLM-as-judge online · CUSUM regression detection · Judge-pair calibration (Cohen's κ) · Latency + cost tracking

score-online.pypython

# Scores arrive via webhook or poll
GET /api/v1/experiments/{id}/online-scores

# {
#   "p50_quality": 0.87,
#   "cusum_status": "stable",
#   "judge_agreement": 0.92
# }

Optimize

An agent that argues with its own recommendation.

The optimization agent runs in three modes. Observer: surfaces diagnoses only. Supervised: generates candidate changes and waits for your approval before running them as experiments. Autonomous: executes approved categories of changes on its own. Every proposal includes a could_be_wrong_if clause — the exact signal that would flip the conclusion.

Observer · Supervised · Autonomous · Structured claims · Evidence pointers · Refutation conditions

optimize.pypython

POST /api/v1/agent/experiments/{id}/analyze
{
  "question": "Should we ship?",
  "depth": "deep",           // Team+
  "autonomy_mode": "supervised",  // observer | supervised | autonomous
}

// returns: claims, recommended_action,
// could_be_wrong_if, cost

Pick winner

Close the experiment, not just the ticket.

When the stats engine declares significance — or the agent proposes a winner — you review the evidence bundle: effect size, confidence interval, SRM status, and the agent's refutation conditions. One click promotes the winner arm to 100% rollout and archives the losers.

Effect size · CI · SRM · Agent rationale bundle · Audit trail · Gate auto-update

pick-winner.pypython

# Promote the winning arm
POST /api/v1/experiments/{id}/promote
{
  "arm_id": "variant-b",
  "rationale": "p=0.003, CI [+4%, +11%], no SRM"
}

# Losing arms archived; gate updated automatically

Monitor

The loop closes here — and starts again.

Post-ship monitoring watches the promoted variant for metric drift. CUSUM detectors fire when the quality or cost curve departs from the baseline established during the experiment. A firing detector creates a new signal in the Optimize stage — closing the loop and starting the next iteration.

CUSUM detectors · Guardrails (warn / high / critical) · Auto-signal creation · Instant rollback trigger

monitor.pypython

# Guardrail fires when quality drops
{
  "type": "cusum",
  "metric": "quality_score",
  "status": "regression",
  "severity": "high",
  "experiment_id": "checkout-copy-v2"
}
# → opens a new Optimize signal automatically

Close the loop

Start with a single trace. By the end of the month you will be shipping winners, not guessing.

Start Free Read the docs