engineering blog · source date 2026-06-03 · 0 comments ·
original1. Problems / challenges / motivations
- Self-service analytics fails in two directions: give an agent broad data access and metric definitions diverge across teams; lock access down and it cannot answer the business questions people actually have.
- Three specific failure modes drive wrong answers: concept-to-entity ambiguity (mapping a question to the...
peer-reviewed paper · source date 2026-06-01 · 0 comments ·
original1. Problems / challenges / motivations
- Production agent quality is gated by the quality of the evaluation loop, and most teams' LLM judges are not calibrated against anything.
- Hand-written judge prompts drift and disagree with human reviewers, which makes every downstream decision noisy.
- Optimization research is overwhelmingly benchmark-based, with...
engineering blog · source date 2026-05-27 · 0 comments ·
original1. Problems / challenges / motivations
- Vertical agents fail in domain-specific ways that generic benchmarks never surface.
- Practitioner corrections are the highest-value signal a deployed agent produces and usually vanish into a support queue.
- "Self-improving" typically means prompt tweaks, which does not fix root causes in the surrounding system.
2....
practitioner blog · source date 2026-03-03 · 0 comments ·
original1. Problems / challenges / motivations
- Eval methodology is well documented and badly practiced, because doing it properly is tedious.
- Coding agents are already in the loop but are not taught how to build evaluation systems.
- Error analysis — the highest-value step — is the one teams skip first.
2. Key ideas
- Publishes `evals-skills`, a plug-in set of...