Building Customer Support AI Agents at 100M-User Scale (Nubank)
peer-reviewed paper · source date 2026-06-01 · added 2026-08-07 15:52:43 · updated 2026-08-07 15:52:43 · Open original blog
1
Problems / challenges / motivations
- Production agent quality is gated by the quality of the evaluation loop, and most teams' LLM judges are not calibrated against anything.
- Hand-written judge prompts drift and disagree with human reviewers, which makes every downstream decision noisy.
- Optimization research is overwhelmingly benchmark-based, with little evidence tied to business outcomes.
2
Key ideas
- An evaluation-driven production framework that uses GEPA inside DSPy to optimize the LLM-as-Judge prompts themselves — optimizing the measuring instrument before the system.
- Inter-rater agreement rises from Cohen's kappa 0.00 to 0.745, alongside improved eval accuracy.
- Downstream outcomes across five production domains at 100M+ user scale: +37pp AI transactional NPS and +29pp self-service.
- Accepted at KDD 2026.
3
Why it matters for AI engineering
- This is the strongest enterprise-impact evidence in the scan and a rare peer-reviewed, large-scale production validation of DSPy/GEPA tied to A/B-tested business results.
- The ordering is the lesson: a judge at kappa 0.00 makes every optimization downstream of it meaningless, so the judge is the first thing to optimize.
- Caveat: the business metrics are first-party Nubank measurements — credible but not externally reproduced.
Comments
No comments yet.