Blog
Notes
on
evals,
judges
and
shipping
AI
Eval methods, changelog posts and engineering notes from the Holdout team.
Changelog
3 min read
Changelog: cost diffs on every pull request
Every run now reports cost per 1k calls and p95 latency against main, right in the GitHub check.
Engineering
11 min read
Shorter system prompts break more than you think
We ran 4,800 prompt-trim experiments. Here is which instructions you can cut, and which ones hold up quality.
Guides
7 min read
Build a test set from production logs in an hour
Sample, de-duplicate and label real traffic so your evals reflect what users actually ask.
Evals
8 min read
Swapping to a cheaper model without losing quality
A step-by-step playbook for model migrations, with the eval gates that caught regressions before release.
Engineering
6 min read
Tracing agent tool calls one step at a time
Agents fail in the middle. Score each tool call as its own span and find the step where a run goes wrong.
Run your first eval in 20 minutes
Bring 240 real cases. 14 days, every feature, no credit card.