Built for the full AI lifecycle
Start with setup, then follow the same Author → Evaluate → Ship → Optimize → Monitor loop you see in the product. Operate and Reference cover administration and implementation details.
Pick a stage
Each stage is one scroll. Copy-paste works end-to-end.
Install the SDK and log your first observation.
Bind the prompt, dataset, metric, and acceptance rule.
Run offline evals and governed experiments against the bound candidate.
Promote a winner behind a gate or dynamic config.
Delegate prompt and config tuning to the optimization agent.
Trace every call, catch regressions before they ship.
Org, workspace, SSO, plan, and audit log management.
REST, SDKs, MCP tools, and statistical methods.
Setup
Before the loop
Minimal integration: one install command, one client init, one observation call. Works from any language with an HTTP client — the typed SDKs are there for ergonomics, not requirements.
pip install winnow-sdk
# evsp_ is a server ingestion key; keep it in your service environment.
export WINNOW_API_KEY=evsp_...
from winnow import Winnow
winnow = Winnow()
winnow.log_observation(
experiment_id="checkout-copy-v1",
arm_id="treatment",
score=0.82,
metadata={"user_id": "u123"},
)
# Verify in Settings → SDK Setup or Traces:
# {"accepted": 1, "rejected": 0}Evaluate
Stage 2 of 5
Run an offline eval or create an experiment against the bound candidate. Winnow keeps the running mSPRT estimate Directional; a governed terminal analysis and explicit acceptance rule hand the decision to Ship.
exp = winnow.create_experiment(
experiment_id="checkout-copy-v1",
arms=[
{"arm_id": "control", "prompt_version": "v1", "is_control": True, "initial_allocation": 0.5},
{"arm_id": "treatment", "prompt_version": "v2", "initial_allocation": 0.5},
],
primary_metric="quality",
stat_method="msprt",
expected_delta=0.03,
power=0.80,
)
stats = winnow.get_experiment_stats(exp.experiment_id)Ship
Stage 3 of 5
Gates are boolean, configs are typed JSON. Both evaluate per-user with a rule stack. Shipping a winner is a gate-rollout percentage change, not a code deploy.
if winnow.check_gate("new_checkout_flow", user={"id": "u123", "country": "US"}):
render_new_flow()
else:
render_old_flow()
# Or a typed config
params = winnow.get_config("ranker_params", user={"id": "u123"})
ranker(top_k=params["top_k"], temperature=params["temperature"])Optimize
Stage 4 of 5
Three autonomy modes: Observer (suggests only), Supervised (waits for approval), Autonomous (applies bounded changes). Every proposal carries a rationale, a safety screen, and a proposer tag.
winnow.configure_agent(
experiment_id="checkout-copy-v1",
mode="supervised",
primary_metric="quality",
sensitivity="balanced",
max_experiments_per_month=20,
budget_per_cycle_usd=50,
)
proposals = winnow.list_agent_proposals("checkout-copy-v1")
for p in proposals:
print(p.id, p.proposer, p.proposal["rationale"])Monitor
Stage 5 of 5
Every observation is a trace. Guardrails are metric thresholds with severity — critical breaches block ship recommendations automatically. The reasoning agent publishes structured assessments (blockers, signals, alternatives) that the host LLM can read.
# Rules-engine recommendation, no LLM call
GET /api/v1/agent/experiments/{id}/assessment
# Cross-experiment context (similar, overlap, baselines, team history)
GET /api/v1/agent/experiments/{id}/meta
# LLM second opinion with claims + evidence + could_be_wrong_if
POST /api/v1/agent/experiments/{id}/analyze
body: { "question": "should we ship?", "depth": "standard" }Operate
Supporting guide
Single org, many workspaces (one per environment is a clean default). API and MCP calls pick the workspace from your session cookie automatically; bearer-key callers pass X-Workspace-ID (or let Winnow auto-pick if you only have one). Seats are unlimited on every tier. SSO/SAML on Team+, audit log retention scales with plan.
# Workspace auto-picks when your org has exactly one
curl https://app.justwinnow.com/api/v1/prompts -H "Authorization: Bearer evsp_your_key_here"
# Response echoes: "_workspace": { "note": "Using workspace 'production' — ..." }
# Explicit workspace via header (recommended for multi-workspace orgs)
curl https://app.justwinnow.com/api/v1/prompts \
-H "Authorization: Bearer evsp_your_key_here" \
-H "X-Workspace-ID: ws_staging_abc"
# Plan, usage, limits for the current org
GET /api/v1/billing
# Audit log (90 days on Team, unlimited on Enterprise)
GET /api/v1/audit?since=2026-01-01Reference
Supporting guide
Full OpenAPI spec at /openapi.json on every deployment. MCP setup, key boundaries, discovery, and governed owner handoffs are documented at /docs/llms.txt for agent and human readers. Core is the default when WINNOW_MCP_PROFILE is unset or blank and exposes 36 lifecycle tools. The variable accepts core, agent-evals, or legacy; Agent Evals exposes 42 tools, while explicit legacy preserves the 123-tool compatibility escape hatch. The compact profiles pass the current release payload and cold-list latency gates. Statistical method notes live alongside each method so the IDE agent picks the right one.
# Curated agent-readable index
https://justwinnow.com/docs/llms.txt
# Full corpus for ingestion
https://justwinnow.com/docs/llms-full.txt
# OpenAPI on every deployment
https://app.justwinnow.com/openapi.json
# Pricing and plan details
https://justwinnow.com/pricingReady to ship better models?
Create a free account. The Free tier covers 5 million runs and 3 agent second opinions per month.