Customer story
Parcelwise
·
Logistics agents
Switched to a model 4× cheaper with no quality loss
−74% cost per 1k
98.9% pass rate held
2 days to migrate
Model
Cost / 1k
Pass rate
frontier-xl
$0.62
98.9%
compact-s
$0.16
98.9%
The challenge
Parcelwise runs agents that rebook failed deliveries. Inference cost was growing faster than revenue, but every model swap felt like a gamble.
The setup
They ran 2,400 recorded agent sessions against three cheaper models, with tool-call and outcome judges, and gated the migration branch in CI.
The result
The cheapest model held a 98.9% pass rate. Cost per 1k calls fell 74%, and one tool-calling regression was blocked before it shipped.
The gate caught a tool-calling regression on a Friday afternoon. Nobody had to roll back.
Daniel Okafor
Staff Engineer