Roberts et al. just published a scaling law that reframes which model you should run. Here is why that matters to anyone running LLM applications in production, and what Winnow ships to make it actionable.
Why does your frontier model lose on cost per correct answer?
On April 1, 2026, a team from the University of Wisconsin-Madison and Stanford published a paper called Test-Time Scaling Makes Overtraining Compute-Optimal (arXiv:2604.01411). It is one of those papers that changes the decision you should be making when you pick a model for production.
This post explains the result in plain language, why it matters if you are running any kind of agentic workload, and what Winnow ships this week to make the insight usable rather than just interesting.
The result in one paragraph
Prior scaling work (Chinchilla, Hoffmann et al.) told you how to trade model size against training data when the only cost that matters is pretraining. Nobody spent much time formalizing the third knob: how many times you sample the model per query at inference. The Train-to-Test (T²) framework jointly optimizes all three, N (parameters), D (training tokens), and k (inference samples), under a single fixed budget C_train = 6ND plus C_inf = 2Nk. The authors present two complementary approaches, one modeling the task loss (NLL) and one modeling the pass@k accuracy directly, and both agree. The headline finding across eight real and synthetic tasks (LAMBADA, ARC-Easy, SciQ, OpenBookQA, plus four synthetic reasoning suites): once you count inference cost, the compute-optimal choice is a smaller model, trained on far more tokens than the 20-tokens-per-parameter Chinchilla rule, sampled multiple times at inference.
Put plainly: for a fixed total spend, an overtrained smaller model sampled several times can beat a substantially larger model sampled once. In the paper's own validation, overtrained 37M-parameter models outperform Chinchilla-optimal 901M-parameter models at the same inference-corrected budget on benchmarks like LAMBADA and OpenBookQA. The effect is directional, not just academic: if you are buying frontier-model calls for reasoning tasks, you are frequently on the wrong side of this tradeoff.
Why this is not just a paper
If the finding holds up, and every signal from the model providers suggests they are already building in this direction, then the model landscape is about to fragment along the (N, D, k) surface rather than collapsing to a handful of frontier options. That has two consequences for anyone running LLM applications:
- The cheapest viable configuration for your workload is almost certainly not the model you are running today. It is some smaller model, sampled more than once, with a verifier attached.
- You cannot pick that configuration from a benchmark. T² tells you what should work under clean conditions. Only your production traffic can tell you what actually works, because the cost/quality tradeoff depends on your input distribution, your verifier, and your quality floor.
This is the stack relationship we care about. T² supplies a principled starting prior. Your traffic supplies the posterior. The thing that converts the former into the latter is a statistical experimentation layer that can compare (model, prompt, k) configurations with rigor, under real cost and reward signals, and pick a winner you can deploy against.
That is exactly what Winnow does.
What changes in Winnow this week
Three product changes, each a direct application of the paper.
1. k is a first-class setting on every eval suite
Eval Builder now has a "Samples per input" control. Set it to 4 and the suite reports pass@4 alongside pass@1. pass@k is the probability that at least one of k samples is correct, which is the right quality number when your task has a verifier (tests pass, schema matches, answer checks). This makes the T² tradeoff measurable rather than theoretical.
2. Cost per correct answer, not cost per call
Cost-per-call is misleading as soon as you sample more than once. The honest comparison is:
cost_per_correct_answer = (cost_per_call * k) / pass@k
Experiment results in Winnow now show this number next to cost-per-call. Sort by it when you are comparing strategies that sample at different k. This is the quantity T² actually optimizes; every other framing leaves cost on the table.
3. Sampling is a proposable axis in the optimizer
The Optimization Agent already proposes prompt changes and model swaps. We are extending the proposer to search over k ∈ {1, 2, 4, 8, 16} when a quality gate is failing and budget has headroom. When it does, it optimizes on cost-adjusted reward (quality − λ·cost), not raw quality, because raw quality alone pushes the allocator toward configurations that are quality-optimal but cost-unviable.
What stays the same
The hardest part of shipping a T²-aware product is not the math. It is refusing to sell it as a silver bullet. Higher k only helps when:
- The task has a verifier. Open-ended generation with no verifier gets nothing from more samples, because you cannot pick the best.
- The cheaper model gets the right answer sometimes. If
pass@1is already at 0.95, extra samples are wasted spend. If it is below 0.4, you probably need a better model or a better prompt first. - You can tolerate the latency. k attempts means k× wall-clock time unless you parallelize. Live gate checks with tight SLAs may need to stay at
k=1. - Fine-tuning is in scope with caveats. The paper's RQ3 shows the overtraining advantage survives standard fine-tuning and SFT, so the finding is not limited to base models. Separate work (Springer et al., 2025) notes that overtrained checkpoints are modestly harder to fine-tune, which partially subdues the advantage but does not reverse it.
Our guide and our in-product copy are explicit about all three. We would rather say "this does not help you" than ship a feature that looks good on a landing page and wastes money in production.
Defensibility
As the model landscape fragments along (N, D, k), picking the right configuration stops being a one-time decision and becomes an ongoing optimization problem. The statistical rigor layer (mSPRT, Thompson Sampling, guardrails, drift detection) gets more valuable, not less. Winnow was built for a world in which the right answer is not obvious from a benchmark and has to be measured on your traffic.
That world just got formal in a paper.
Read the source
- Roberts, N., Cho, S., Gao, Z., Huang, T., Wu, A., Orlanski, G., Trost, A., Buchanan, K., Albarghouthi, A., & Sala, F. (2026). Test-Time Scaling Makes Overtraining Compute-Optimal. arXiv:2604.01411.
Try it
- Eval Builder with pass@k — set "Samples per input" above 1 on any new suite.
- Compute Strategy guide — worked examples for code-gen, extraction, and summarization.
- Optimization dashboard — sampling-strategy callout points at the full framing.