Evals Skills for Coding Agents
practitioner blog · source date 2026-03-03 · added 2026-08-07 15:52:43 · updated 2026-08-07 15:52:43 · Open original blog
1
Problems / challenges / motivations
- Eval methodology is well documented and badly practiced, because doing it properly is tedious.
- Coding agents are already in the loop but are not taught how to build evaluation systems.
- Error analysis — the highest-value step — is the one teams skip first.
2
Key ideas
- Publishes `evals-skills`, a plug-in set of agent-installable skills: eval-audit, error-analysis, write-judge-prompt, validate-evaluator, evaluate-rag, and build-review-interface.
- Operationalizes the error-analysis-first iteration loop as skills you delegate to an agent.
- Complements vendor MCP servers rather than replacing them.
3
Why it matters for AI engineering
- From one of the most-cited practitioner voices on LLM and product evals, this turns eval methodology into a reusable, agent-runnable artifact — concrete, dissectable optimization tooling rather than advice.
- `validate-evaluator` is the notable inclusion: it bakes the judge-of-judges step into the default workflow.
- Caveat: practitioner blog and tooling, not a peer-reviewed study; effectiveness claims are experiential rather than benchmarked. Value depends on the underlying error-analysis methodology being applied rigorously, not on installing the skills.
Comments
No comments yet.