Not every failing eval should block a release. A two-bucket task classification — frontier (informational) vs regression (release-blocking) — lets your CI gate the things that matter without burying the things that are still aspirational.
Frontier vs Regression: A Two-Word Vocabulary for Release Gates
Teams that start running agent evals against every PR usually hit the same wall in week three. The suite is comprehensive, the rubric is honest, and the failure rate is high — because the suite contains both "things we ship today" and "things we aspire to ship next quarter." If the CI gate blocks on any failure, every deploy is red. If the CI gate blocks on no failure, the suite is decorative.
The fix is not better thresholds. It is a vocabulary.
Two buckets, two words
Winnow's task_set classification gives every task one of two labels:
- regression — failures block release. This is the locked-in capability set. A failing regression task means a feature the customer relies on just stopped working; the gate refuses to let the build through.
- frontier — failures are informational. This is the aspirational set. A failing frontier task means a thing you are still working toward; the gate logs it, the dashboard tracks the trend, but the build proceeds.
That is the entire vocabulary. Two words, mutually exclusive, applied per task. The release-readiness aggregation reads only the regression bucket; everything in the frontier bucket reports into a separate "Frontier signal" panel that trends over time.
Why this is not just "severity"
A severity field — critical / high / medium / low — sounds like the same thing, but it isn't. Severity asks "how bad is this failure?" task_set asks "should this failure block?" Those are different questions, and conflating them is how teams end up with a CI gate that fails on every PR because some critical-severity task is genuinely aspirational.
The trick is to keep the two axes orthogonal. A regression task can be low-severity (a deprecated edge case nobody hits much) and still block release; a frontier task can be high-severity (a serious safety gap you want to close) and still not block. The classification is about commitment, not about importance.
How it shows up in Winnow
On the Datasets list, each dataset card now carries a small task-set badge between the live-traffic chip and the row-count chip. Three states:
- Regression (orange) — the dataset's tasks are in the locked-in set.
- Frontier (purple) — the dataset's tasks are still aspirational.
- Mixed (grey) — the classification isn't set yet; the badge invites you to classify in one click.
Click the badge and a small popover offers Promote / Demote / Clear. The change persists through the existing dataset PATCH route and shows up immediately on the next load.
A HelpCircle in the popover footer routes to the wiki entry at /wiki/wolfe-task-set for the full semantics, including the auto-promotion rule (a frontier task that passes N consecutive runs auto-promotes to regression so you don't have to remember to flip it manually) and the release-readiness aggregation contract (only regression failures count toward the blocker count).
What this unlocks
Two things, both worth more than the one-line UI change suggests:
- The CI gate stops being noisy. Once the aspirational tasks are tagged
frontier, the gate's blocker count drops to the failures that genuinely matter — and the team starts trusting it again. - The aspirational set becomes legible. The "Frontier signal" panel is a trend graph of work you've committed to but haven't shipped yet. Pass rates climbing on a frontier task is the leading indicator that the capability is ready to promote.
Try it on one suite first. Tag the five tasks you're most embarrassed by as frontier, ship a PR, watch the gate turn green for the first time in weeks, and then make a real plan for promoting them.