Building Self-Improving Tax Agents with Codex
engineering blog · source date 2026-05-27 · added 2026-08-07 15:52:43 · updated 2026-08-07 15:52:43 · Open original blog
1
Problems / challenges / motivations
- Vertical agents fail in domain-specific ways that generic benchmarks never surface.
- Practitioner corrections are the highest-value signal a deployed agent produces and usually vanish into a support queue.
- "Self-improving" typically means prompt tweaks, which does not fix root causes in the surrounding system.
2
Key ideas
- A three-part loop: capture production traces, convert practitioner corrections into structured evals, and hand Codex scoped engineering tasks to investigate root causes and ship fixes.
- Scoped tasks come with evidence and validation gates, so the agent's fix has to pass something before it lands.
- Builds explicitly on OpenAI's harness-engineering and Symphony orchestration work.
3
Why it matters for AI engineering
- One of the clearest instantiations of the eval-driven data flywheel — traces to structured evals to scoped fixes — and a reusable playbook rather than a framework.
- The important move is that corrections become evals first and code changes second; that ordering is what keeps the loop from drifting.
- Caveat: a vendor case study; the loop and any outcome metrics are OpenAI/partner-reported and not independently reproduced. It references Symphony, whose canonical URL slug the scan flagged as unverified — cite Symphony cautiously.
Comments
No comments yet.