Product
Every
change,
tested
on
real
cases
Four parts that turn a prompt change into a measured, gated deploy.
Test sets
Cases from production, versioned like code
Import traces from your logs, tag them by feature or risk, and freeze a set for every release. Holdout keeps the history, so you can see when a case started failing.
support-v14
1,912 cases · frozen for release 3.4
All
billing
legal
i18n
Case
Tag
Last run
Refund policy, annual plan
billing
pass
Cite clause 14.2 for a non-lawyer
legal
fail
Translate error E-402 to German
i18n
pass
Which plan includes SSO?
pricing
pass
Escalate a chargeback over $500
billing
pass
Summarise a 40-page MSA
legal
fail
holdout / evals
Blocked · 11 regressions
holdout / evals
Passed · cost −41%
ci / unit tests
Passed · 312 tests
ci / lint
Passed
View run #1,284 →
Deploy gate
A red check before a bad change ships
Add the GitHub Action and pick a bar: pass rate, cost or latency. Pull requests that drop below it fail, with a link to the run.
Trace diff
Expected, got, and why
Every failed case opens a diff of the expected and actual answer, the spans that changed, and the judge’s reason, so the fix is obvious.
Case 0142 · Cite clause 14.2 for a non-lawyer
legal · support-v14 · run #1,284
Input · “Can we end the contract early if we give notice?”
Expected
Yes, with 30 days’ notice under clause 14.2.
Got · changed span
Yes, with 60 days’ notice under clause 12.4.
Judge · citation accuracy
0.12 · wrong clause
Judge calibration · 1,200 human labels
Agreement per judge, re-checked every week
94%
agreement with human reviewers
omissions
96%
dosage
93%
tone
89%
Disagreements go to the review queue · 38 this week
Calibrated judges
LLM judges you can defend in a review
Judges are scored against your own reviewers before they gate anything. Ours agree with human labels 94% of the time; the rest go to a review queue.
Integrations
Works
with
the
stack
you
already
run
If it makes a model call, Holdout can wrap it.
OpenAI
Chat, responses and tools
Anthropic
Messages and tool use
Gemini
Text and multimodal
Open-weight models
vLLM, Ollama, TGI
LangChain
Chains and agents
LlamaIndex
RAG pipelines
GitHub Actions
Deploy gate check
OpenTelemetry
Import production traces
SOC 2 Type II
Audited every year.
GDPR
EU data residency available.
Self-host
Run Holdout in your VPC.
Never trains on your data
Zero retention mode on request.