Blog

Stage 3: Adversarial Probe, Not Optional

An exploit probe tests your assertions, not the agent. Winnow's QA tab runs N deterministic strategies against every published task's assertions — if any strategy slips through, the task is held until you tighten.

Block-Mode HIL: A Queue, Not a Channel

When a customer-facing agent emits request_human_input(...) in block mode, somebody needs to answer in minutes — not in the next Slack thread. Winnow's HIL queue lives where reviewers already work, expands inline, and resumes the agent the moment the answer lands.

Calibration Center — cross-suite judge triage on Winnow

The Harness Pill: Decomposing a Scorecard Without Leaving the Page

Most regressions in agent evals don't come from the prompt — they come from the scaffold that runs the prompt. Winnow's Scorecard Overview tab now regroups itself when you flip the Arm / Model / Harness toggle. The Harness pill is the load-bearing one.

Multi-Agent Cost: Make the Dominant Agent Visible

When a manager-researcher-writer scaffold spends $24 a week instead of $17, the question isn't 'which arm' — it's 'which agent'. Winnow's B7 agent rollup turns per-span agent_id tags into one cross-trace cost table and one per-trace span tree, both colored by the same agent palette.

Run the Probe From the QA Tab

The exploit-probe QA tab now carries a Run probe now button — capture the probe spec inline, POST to the existing route, and watch the new result row land without leaving the page.

Scaffolds: Versioning the Whole Agent, Not Just the Prompt

A prompt is one ingredient. A scaffold is the whole recipe — prompts plus tools plus model config — pinned to one immutable hash that a trace can point back to. Winnow now ships a scaffold history page so you can audit which recipe shipped when.

Smart Suggestions: Three Next-Actions, Not Eleven Dashboards

Eleven eval capabilities surface eleven data points. A suite home that shows all eleven at once is a stress chart, not a workplace. Winnow's P6 smart suggestions strip is the prioritization layer that turns those eleven into up to three actionable nudges — and renders nothing when none of them fire.

Frontier vs Regression: A Two-Word Vocabulary for Release Gates

Not every failing eval should block a release. A two-bucket task classification — frontier (informational) vs regression (release-blocking) — lets your CI gate the things that matter without burying the things that are still aspirational.

When the Judge Drifts: Calibration Monitoring for LLM Evaluators

LLM judges are not stationary. Models, rubrics, and human labellers all shift. Winnow's calibration baselines catch the moment your judge stops agreeing with humans, before the dashboard lies to you.

Oracle vs Expected Output: Two Fields That Look the Same and Aren't

An oracle solution and an expected-output blurb both describe what a good answer looks like — but only one is a concrete answer the platform can auto-grade against. Why Winnow's dataset row editor now carries both, side by side.

Five Judges in One Call: Rubric-Itemized Evaluation in Winnow

Replacing N independent LLM-judge criteria with one rubric collapses N HTTP round-trips per trial into one — same per-item κ, ~Nx cost savings, no behavioral regression.

Onboarding bridges: drop Winnow into your stack without rewriting your call sites

Winnow now supports 23 sources at beta. Swap one import. Backfill from a JSONL export. Or point your OpenTelemetry exporter at us. Pick the path that costs you the least.

How many times should your model answer?

A full-lifecycle answer: instrument every sample, build the statistical prior, validate on your traffic, and jointly budget training, inference, and k under one spend ceiling.

Why does your frontier model lose on cost per correct answer?

Roberts et al. just published a scaling law that reframes which model you should run. Here is why that matters to anyone running LLM applications in production, and what Winnow ships to make it actionable.

How to A/B Test Your LLM: A Complete Guide

Standard A/B testing breaks down with non-deterministic models. Learn how to run statistically rigorous experiments on LLM outputs using mSPRT and the Winnow SDK.

Winnow vs. Manual Evaluation: Why Eyeballing LLM Outputs Doesn't Scale

The hidden costs of manual LLM evaluation and how automated, statistically rigorous testing replaces gut checks with evidence.

Understanding mSPRT: Sequential Testing for AI Experiments

A technical deep-dive into mixture Sequential Probability Ratio Tests -- the statistical engine behind continuous experiment monitoring in Winnow.

Feature Flags for AI Applications: A Practical Guide

How to use feature flags to safely manage model rollouts, prompt versioning, and gradual deployments in production AI pipelines.

Introducing the Winnow Optimization Agent

Meet the autonomous agent that observes, proposes, and executes optimizations across your AI pipeline -- with 5 levels of autonomy and built-in safety controls.