Customer story
Finch Legal
·
Legal AI
Hallucinated citations down 71% in 6 weeks
−38% model cost
0 incidents since
6 weeks to roll out
Hallucinated citations · 6 weeks
−71%
The challenge
Finch drafts contract summaries for in-house legal teams. Prompt changes shipped weekly, and two in a row cited the wrong clauses. Customers found them first.
The setup
The team imported 1,900 real summaries from production logs, wrote a citation judge calibrated on 300 lawyer reviews, and added the Holdout check to every prompt pull request.
The result
In six weeks hallucinated citations fell 71%, a cheaper model passed the gate at 38% lower cost, and no citation incident has reached a customer since.
We stopped arguing about whether the new prompt was better. The run tells us in four minutes.
Maya Ortiz
Head of AI