arXiv paper · source date 2026-06-24 · 0 comments ·
original1. Problems / challenges / motivations
- Self-improving agents are bounded by the evaluator that scores their modifications, so a fixed evaluator caps improvement.
- In hard-to-verify domains — paper writing, proof grading — there is no benchmark to serve as the evaluator at all.
- Letting the evaluator evolve freely invites the agent and evaluator to drift...
peer-reviewed paper · source date 2026-06-01 · 0 comments ·
original1. Problems / challenges / motivations
- Production agent quality is gated by the quality of the evaluation loop, and most teams' LLM judges are not calibrated against anything.
- Hand-written judge prompts drift and disagree with human reviewers, which makes every downstream decision noisy.
- Optimization research is overwhelmingly benchmark-based, with...
practitioner blog · source date 2026-03-03 · 0 comments ·
original1. Problems / challenges / motivations
- Eval methodology is well documented and badly practiced, because doing it properly is tedious.
- Coding agents are already in the loop but are not taught how to build evaluation systems.
- Error analysis — the highest-value step — is the one teams skip first.
2. Key ideas
- Publishes `evals-skills`, a plug-in set of...