Changelog
3 min read
Changelog: cost diffs on every pull request

Jonas Weber

Every run now reports cost per 1k calls and p95 latency against main, right in the GitHub check.
What we measured
We ran every variant against the same 240 production cases, scored each with a calibrated judge, and compared pass rate, cost per 1k calls and p95 latency against main.
What we learned
Most regressions came from instructions that looked redundant. Keep the ones that constrain format, citations and refusals, and test every removal on real cases before it ships.