Catch
AI
regressions
before
your
users
do.
Holdout runs every prompt, model and tool change against your real test cases, and blocks the deploy when quality drops.
14 days free · no credit card
Gating deploys at
Northwind
Parcelwise
Finch Legal
Lumen Health
Quanta
Ostrava
Helio
Your
next
prompt
change
is
one
deploy
away
from
a
support
ticket.
Test
it
on
240
real
cases
first.
14 days free · no credit card
Product
Everything
between
a
prompt
change
and
production
Test sets, a CI gate, traces and calibrated judges in one place. Built for the engineer who owns the prompt.
Case
Tag
Last run
Refund policy, annual plan
billing
pass
Cite clause 14.2 for a non-lawyer
legal
fail
Translate error E-402 to German
i18n
pass
Which plan includes SSO?
pricing
pass
Test sets
Import real cases from production logs, tag them, and version them like code.
holdout / evals
Blocked · 11 regressions
holdout / evals
Passed · cost −41%
Deploy gate
A GitHub check that fails the PR when the pass rate drops below your bar.
Expected · 30-day notice, clause 14.2
Got · 60-day notice, clause 12.4
Judge · wrong clause cited
Trace diff
See exactly where a case failed: expected, got, and the judge’s reason.
94%
agreement with human reviewers
Calibrated judges
LLM judges checked against your reviewers, with a human queue for edge cases.
main
$0.64
this change
$0.38
Cost & latency
Every run reports cost per 1k calls and p95 latency against main.
Works with your stack
OpenAI
Anthropic
Gemini
Open-weight models
LangChain
LlamaIndex
GitHub Actions
Why Holdout
Homemade
evals
versus
a
real
deploy
gate
Most teams start with a script and a spreadsheet. It stops scaling at the first silent regression.
Homemade evals
✕
Spot-checks before a release
✕
Vibe tests in a notebook
✕
A spreadsheet nobody updates
✕
Regressions found by customers
Holdout
✓
Every change tested on real cases
✓
A deploy gate in CI
✓
A trace for each failure
✓
A cost and latency diff on every run
How it works
Your
first
gated
deploy
today
Three steps, about 20 minutes of setup, no new infrastructure.
01
Connect
Four lines of SDK around your model call. About 20 minutes.
import { holdout } from “holdout”
const run = holdout.wrap(openai)
await run.chat(messages)
holdout.gate({ minPass: 0.97 })
02
Define
Import cases from production logs, or start from our templates.
$ holdout import --from logs
Reading 30 days of traces
240 cases · 6 tags · 3 sets
Test set support-v3 ready
03
Gate
Add the GitHub check. Failing changes don’t ship.
.github/workflows/evals.yml
holdout / evals · Blocked · 11 regressions
holdout / evals · Passed · cost −41%
Merging waits for a green gate
Testimonials
What
changed
after
the
first
gated
deploy
We stopped arguing about whether the new prompt was better. The run tells us in four minutes.
Maya Ortiz
Head of AI
·
Finch Legal
71% fewer hallucinated citations
The gate caught a tool-calling regression on a Friday afternoon. Nobody had to roll back.
Daniel Okafor
Staff Engineer
·
Parcelwise
4× cheaper model, same quality
Our clinicians review 40 cases a week instead of 400. The judges agree with them 94% of the time.
Priya Raman
ML Lead
·
Lumen Health
212 regressions blocked in Q3
Setup took one standup. That same day we imported 600 cases.
Tom Becker
CTO
·
Quanta
20 min to first run
Cost and latency diffs on every PR changed how we pick models.
Lena Hoffmann
Platform Lead
·
Ostrava
−33% inference spend
It is the first eval tool our product managers actually open.
Sam Whitfield
VP Product
·
Helio
3× more prompt changes shipped
Pricing
Pricing
that
scales
with
your
eval
volume
Included eval runs on every plan, a hard spend cap, and 14 days free on all of them.
Starter
For one team’s first AI feature.
$49
/ month
✓
10,000 runs a month
✓
3 test sets
✓
GitHub check
✓
Hard spend cap
Team
Most teams
For teams shipping weekly.
$249
/ month
✓
100,000 runs a month
✓
Unlimited test sets
✓
Judges + review
✓
Cost and latency diffs
✓
Hard spend cap
Enterprise
For regulated data at scale.
$1,200
from / month
✓
Unlimited eval runs
✓
Self-host or VPC
✓
Zero retention mode
✓
SSO, SCIM, audit log
✓
Dedicated engineer
SOC 2 Type II
Audited every year.
GDPR
EU data residency available.
Self-host
Run Holdout in your VPC.
Never trains on your data
Zero retention mode on request.
FAQ