Evaluation

Define and Measure AI Quality

Build structured eval suites that combine LLM judges, code assertions, and golden datasets. Capture user feedback from production. Know exactly when quality regresses -- before your users notice.

Start Free Read the Docs

eval_suite.pypython

import winnow

# Create an evaluation suite
suite = winnow.EvalSuite(
    name="summarization-quality",
    version="3.1",
    scorers=[
        winnow.LLMJudge(
            criteria="conciseness",
            model="gpt-4o",
            rubric="1-5 scale, 5 = maximally concise",
        ),
        winnow.LLMJudge(
            criteria="faithfulness",
            model="gpt-4o",
            rubric="Binary: faithful to source or not",
        ),
        winnow.CodeAssertion(
            name="length_check",
            fn=lambda output: len(output.split()) < 150,
        ),
    ],
    golden_dataset="summarization-v3",
)

# Run evaluation
results = suite.run(model="claude-sonnet", prompt="v2.1")
print(results.summary())  # mean scores, pass rates, regressions

Evaluation That Scales With You

From quick smoke tests to rigorous production monitoring -- one framework for every stage.

Eval Suite Builder

Compose suites from reusable scorers, assertions, and datasets. Version everything, diff results across runs, and set pass/fail thresholds per metric.

LLM Judge Scoring

Define rubrics in plain language and let an LLM judge score outputs on any criteria: faithfulness, conciseness, tone, safety, helpfulness, and more.

Golden Datasets (Versioned)

Maintain versioned collections of input/expected-output pairs. Track how model performance changes across dataset versions and prompt iterations.

Production Feedback Loop

Collect thumbs-up/down, corrections, and free-text feedback from end users. Automatically route low-quality outputs to your review queue.

Auto-Capture from Production

Continuously sample production inputs and outputs to build evaluation datasets. Detect distribution shift and flag novel edge cases automatically.

Research-backed

Measure pass@k, not just pass@1

A model that gets the answer right 60% of the time has a pass@1 of 0.60 and a pass@4 of about 0.97. Running a smaller model four times and keeping the best is often cheaper and better than running a larger model once. Winnow eval suites accept a samples-per-input value and report pass@k, cost-per-correct-answer, and the cost multiplier side by side, so you can compare strategies on the number that actually reflects end-to-end spend.

What you get in the product

Samples-per-input control on every eval suite
pass@k reported alongside standard pass rate
Cost-per-correct-answer column in experiment results
Sampling as a proposable axis in the optimizer

Why it matters

The Train-to-Test framework from Roberts et al. (2026) shows that once inference cost is counted, the compute-optimal move is a smaller overtrained model sampled multiple times, and the finding persists after fine-tuning. Winnow surfaces the measurement against your own traffic so the prior becomes a posterior you can deploy.

Reference: Roberts, N. et al. (2026). Test-Time Scaling Makes Overtraining Compute-Optimal. arXiv:2604.01411. University of Wisconsin-Madison and Stanford University.

Why evaluation changes everything

Without structured evaluation, every model change is a coin flip. Winnow gives you a repeatable, versioned evaluation pipeline that integrates into your CI/CD workflow and connects directly to your experiments and optimization agent.

Run evals in CI before every deploy

Compare any two model/prompt combinations side by side

Regression alerts via Slack, PagerDuty, or webhook

Human-in-the-loop review queues for ambiguous cases

Feeds directly into Experiments and Optimization Agent

Measure what matters

Create your first eval suite in minutes. Free for up to 5M agent runs per month.

Start Free