Observability 2.0
Observability that tells the truth
Winnow is the eval and observability layer for LLM and agent systems that stays statistically honest. One wide event per call. Service-level objectives and burn-rate alerting on quality, not just infrastructure. Agent-native trajectory metrics carry confidence intervals, and the closed loop turns production failures into your next golden evals.
Drop-in OpenTelemetry and SDK ingest. Works with your existing traces. No vanity dashboards.
from opentelemetry import trace
from opentelemetry.exporter.otlp.proto.http.trace_exporter import (
OTLPSpanExporter,
)
# Point any OpenTelemetry gateway at Winnow.
# gen_ai.* span attributes map to typed columns automatically.
exporter = OTLPSpanExporter(
endpoint="https://ingest.justwinnow.com/v1/traces",
headers={"x-winnow-key": WINNOW_INGEST_KEY},
)
# Every LLM and agent call lands as one wide event:
# model, cost, ttft, cache_hit, prompt_version, build_sha,
# environment, end_user, plus prompt, response, and score.
with trace.get_tracer("checkout-agent").start_as_current_span(
"chat.completion"
) as span:
span.set_attribute("gen_ai.request.model", "claude-opus-4-8")
span.set_attribute("gen_ai.usage.cost_usd", 0.0142)
span.set_attribute("winnow.prompt_version", "v14")
span.set_attribute("winnow.build_sha", "a9d2f6b")One row per call. Every field you need to ask why.
Stop stitching logs, metrics, and traces together by hand. Every LLM and agent call lands as a single wide event, with typed columns you can slice in one query.
Typed columns on every observation
Cost, model, latency, time-to-first-token, cache hits, prompt version, build SHA, environment, and end user sit alongside the prompt, response, and score. Slice latency by prompt_version, cost by model, errors by build_sha.
OpenTelemetry-native ingest
Point any OpenTelemetry gateway at the intake endpoint. Winnow maps gen_ai.* span attributes to typed fields automatically, so you keep your existing traces and add nothing new to your emit path.
A field glossary built in
Every column carries a definition that the judges and the root-cause engine read, so a field like is_slow is never misread. High-cardinality dimension-diff runs over these columns to compute what differs about a failing cluster.
SLOs and error budgets
Service-level objectives for quality, not just uptime
Define a service-level objective directly on a judge, such as helpfulness staying above 90% over 30 days. Winnow tracks the error budget, computes a multi-window burn rate, and pages you before the budget is gone. The discipline SRE brought to latency, now applied to model quality.
Alerts fire only when both windows agree, so you page on real regressions, not a single noisy hour.
The honest Production view, offline versus online
Task success rate, trajectory efficiency, redundancy, tool-use accuracy, and step-reasoning accuracy, measured as eval criteria and then compared offline to online with a confidence interval on the delta. A number you can ship on, not a vanity dashboard.
| Metric | Offline | Production | Delta offline to online | Verdict |
|---|---|---|---|---|
| Task success rate | 0.86 (+/-.04, n=420) | 0.71 (+/-.03, n=9.1k) | -0.15 [-.20, -.11] | SIGNIFICANT GAP |
| Tool-use accuracy | 0.94 (+/-.02) | 0.92 (+/-.01) | -0.02 [-.05, +.01] | CONSISTENT |
| Trajectory efficiency | 0.78 (+/-.03) | 0.74 (+/-.02) | -0.04 [-.08, -.01] | WATCH |
| Step-reasoning accuracy | no offline data | 0.81 (+/-.02) | no comparison | PREVIEW |
Why the gap? Your offline set covers about 62% of the production intent mix, so offline overstated success. The fix is to widen the eval dataset and re-baseline, not to conclude that the model regressed.
Color is earned by a threshold, never by a point value
Every comparison carries the statistics that make it trustworthy. A metric that is not calibrated stays in preview and is never quoted as fact.
A confidence interval on the delta
Not two lonely point values. Winnow puts a confidence interval on the offline-to-online difference, with a two-proportion test and a family-wise multiple-comparison correction across the metric family. A gap turns red only when the delta interval excludes zero and online sits below the gate.
Anytime-valid and advanced detection
Peek whenever you want with anytime-valid sequential tests that hold their error rate under continuous monitoring. Bayesian estimates, CUPED variance reduction, and sample-ratio-mismatch detection are built in, so a broken split or a noisy window cannot masquerade as a result.
An explicit no-data state
When data is simply missing, Winnow shows an explicit no-data state and never fabricates a zero. When offline and online diverge, it names selection and coverage bias first, computed from your own intent clusters, before it blames the model.
Last self-test landed clean, zero dropped.
Intake health
Know your telemetry is landing, and fix it in one click
The silent killer of observability is data that never arrives. Winnow shows a live readout of what is landing, with accepted, dropped, and sampled counts and their freshness, plus a one-click self-test that round-trips a sentinel span end to end.
Production failures become your next golden evals
Most tools stop at a chart. Winnow closes the loop. When production fails, it root-causes the cluster automatically, then promotes it into your golden eval set, human-gated, with a corrected expected answer. Every incident makes the next eval run stronger.
Detect
A burn-rate alert or a production verdict flags a failing cluster of calls.
Root-cause
Dimension-diff computes what differs about the failing cluster, automatically, with no manual pivot.
Promote
Capture the failing trace as a golden case, human-reviewed, with a corrected expected answer.
Re-evaluate
The widened golden set re-runs offline. Coverage grows and selection bias shrinks.
Guard the ship
Proposed fixes clear a simulate and guardrail gate before promotion. No autonomous ships.
Typed, gated fixes
Proposed fixes are typed ActionPlans that must clear a simulate and guardrail gate before they ship. Rollback is scoped: the loop restores the last known good config on a guardrail breach, and a human or the bandit promotes the winner.
Three autonomy modes
Run the loop in Observer, Supervised, or Full-auto. Observer watches, Supervised proposes and waits for a human, and Full-auto acts within the guardrails you set. You choose how much rope the loop gets.
A trustworthy proposer
Winnow tracks a proposal-rejection rate and flags an unreliable proposer, so the loop stays honest even when an agent is writing the fix. A bad proposer gets caught, not promoted.
Generic LLM dashboards count tokens. Winnow tells the truth.
The throughline across every feature is honesty, the closed loop, and agent-native depth. Three things a token-counting dashboard structurally cannot give you.
Generic LLM dashboard
- xPoint values with no confidence, so a 2% move looks like signal.
- xFake zeros where data is simply missing.
- xToken and latency counts, with no notion of agent trajectory or tool use.
- xAlerts on raw thresholds that page you on noise.
- xCharts are the end of the road, so you copy failures by hand.
Winnow, Observability 2.0
- +Confidence intervals with anytime-valid and multiple-comparison correction on every comparison.
- +Explicit no-data states, never a fabricated zero.
- +Trajectory efficiency, redundancy, tool-use, and step-reasoning metrics.
- +Multi-window burn-rate objectives that page on real regressions.
- +The closed loop, where failures become golden evals, human-gated.
Observe your agents like you mean it
Drop in OpenTelemetry, define one service-level objective, and watch a production failure become your next eval. Statistically honest from the first event.
Start Free