playground / Prompt eval
Score two prompts on a synthetic eval set.
Watch refusals, judge noise, and bimodal scores trip up naive stats — and see ours hold up.
Configure the eval: how many questions, two prompts (control + treatment), the judge's score distribution (bimodal 0/1 vs continuous 0-1), refusal rate, and whether refusals are MNAR — the worst answers silently fail to log. Click "See how our statistics would conclude" to run the synthetic scores through Winnow's real statistical/ module.
Stats run server-side on the real module· No login required· Synthetic data only — never uploaded
1. Configure
live
Sample size
n per arm2,000
allocation50 / 50
Outcome
metric typebinomial
baseline pass rate0.62
treatment uplift+6.0 pp
Realism
5% of values replaced with 10× the mean — one giant order, one bot session
missing-data patternMNAR
missing NOT at random — the worst answers silently drop; bias risk
missing rate8%
2. Synthetic data
3,680 rows · MNAR 8% masked
control · n=1,840 · μ=0.590treatment · n=1,840 · μ=0.645
| stat | control | treatment | delta |
|---|---|---|---|
| questions logged | 1,840 | 1,840 | — |
| pass rate | 0.590 | 0.645 | +0.054 |
| % missing | 8.0% | 8.0% | MNAR — bias risk |
3. Test it
via /api/v1/playground/analyze
MNAR detected — caution
The treatment prompt refuses more often. Naive estimate is biased upward; the MNAR-corrected CI is what we report. No statistical method fixes MNAR — only better instrumentation does.Click the button above to pipe your synthetic data through the real statistical module. We run mSPRT with auto-cure CUPED, the SRM check, skewness-aware normality floor, and surface what we'd decide.