Guides

7 min read

Build a test set from production logs in an hour

Tom Becker

City lights seen from orbit

Sample, de-duplicate and label real traffic so your evals reflect what users actually ask.

What we measured

We ran every variant against the same 240 production cases, scored each with a calibrated judge, and compared pass rate, cost per 1k calls and p95 latency against main.

What we learned

Most regressions came from instructions that looked redundant. Keep the ones that constrain format, citations and refusals, and test every removal on real cases before it ships.

Catch

AI

regressions

before

your

users

do.

Start with 240 of your own cases. 14 days free, no credit card.

Catch

AI

regressions

before

your

users

do.

Start with 240 of your own cases. 14 days free, no credit card.

Catch

AI

regressions

before

your

users

do.

Start with 240 of your own cases. 14 days free, no credit card.

Holdout

Get the monthly changelog

1,400+ AI teams ship with Holdout.

© 2026 Holdout Labs, Inc.

All systems normal

Holdout

Get the monthly changelog

1,400+ AI teams ship with Holdout.

© 2026 Holdout Labs, Inc.

All systems normal

Create a free website with Framer, the website builder loved by startups, designers and agencies.