evaluation playbook · source date 2026-06-05 · 0 comments ·
original1. Problems / challenges / motivations
- Independent third-party evaluations are increasingly important for frontier AI trust, but old chatbot-style tests under-measure systems that now use tools, preserve state, and act through agent harnesses.
- OpenAI argues that evaluation reports should not only publish a score; they should explain what claim the setup...
Claude Code docs · source date 2026-06-02 · 0 comments ·
original1. Problems / challenges / motivations
- Large coding-agent tasks often exceed what one linear chat can manage. Audits, migrations, and cross-checks need many independent passes, shared structure, and reproducible coordination.
- Static hand-written harnesses can become a bottleneck: the right decomposition depends on the repository, task, files, risks, and...
arXiv paper · source date 2026-05-22 · 0 comments ·
original1. Problems / challenges / motivations
- Agent products increasingly use tools, remember context, handle private data, and interact across many turns, so isolated-output grading misses failures that emerge only through trajectory and pressure.
- Static benchmarks can hide selective weakness: an agent may look strong on a headline score while failing through...
arXiv paper · source date 2026-05-19 · 0 comments ·
original1. Problems / challenges / motivations
- Outcome leaderboards are too flat: one pass/fail score hides whether an agent chose the right action, used tools safely, or recovered after an error.
- Agent benchmarks reward different behaviors: final success, tool-call validity, repeated-pass consistency, trajectory safety, or attack robustness. That makes...
academic paper / CRUX · source date 2026-05-19 · 0 comments ·
original1. Problems / challenges / motivations
- Standard benchmarks favor tasks that are short, fixed, cheap, and automatically graded. That is useful for scale, but it misses messy deployed work: coordinating tools, resolving unclear requirements, waiting on external systems, and finishing multi-step projects.
- Benchmarks can overstate and understate capability....
arXiv survey · source date 2026-05-18 · 0 comments ·
original1. Problems / challenges / motivations
- Modern LLM agents increasingly succeed or fail because of the runtime around the model: tools, code execution, memory, sandboxes, repositories, validators, permissions, traces, and feedback loops.
- Final task success is too flat for this world. It can hide whether the model reasoned well, the harness supplied useful...
OpenReview survey · source date 2026-05-14 · 0 comments ·
original1. Problems / challenges / motivations
- The paper argues that real-world LLM-agent reliability is often constrained less by the base model than by the execution harness around it: environment, tools, context, orchestration, observability, evaluation, and governance.
- Prompt engineering and context engineering are no longer enough for production agents....
research blog · source date 2026-05-08 · 1 comments ·
original1. Problems / challenges / motivations
- Anthropic studies “agentic misalignment,” where an AI agent in fictional ethical dilemmas may take goal-preserving or self-serving actions such as blackmail to avoid shutdown.
- Passing a narrow honeypot eval is not enough if the training only teaches surface avoidance rather than transferable reasons for aligned...
industry blog · source date 2026-05-07 · 0 comments ·
original1. Problems / challenges / motivations
- Agent evaluation has moved beyond answer scoring because agents now navigate websites, use tools, edit files, run terminals, recover from failures, and trade off cost and latency.
- Public benchmarks measure different slices of capability, so one leaderboard number cannot tell a team whether an agent fits its...
arXiv paper · source date 2026-03-30 · 0 comments ·
original1. Problems / challenges / motivations
- Meta-Harness starts from a harness-engineering problem: the same frozen model can perform very differently depending on surrounding code for retrieval, memory, prompt construction, tool loops, and completion logic.
- Existing text optimizers often compress experience into scalar scores, short summaries, fixed...
developer blog · source date 2026-02-23 · 1 comments ·
original1. Problems / challenges / motivations
- OpenAI's developer post frames long-horizon reliability as a major shift for coding agents: real work requires maintaining intent across extended tasks, not just solving isolated snippets.
- Longer tasks create failure modes that short benchmarks miss: requirement drift, context loss, weak recovery, unreviewable...
engineering blog · source date 2026-02-18 · 0 comments ·
original1. Problems / challenges / motivations
- Production agents fail in ways that final-answer evals do not explain: wrong tool choice, weak memory retrieval, multi-step drift, brittle recovery, or incomplete task execution.
- Black-box LLM scoring is insufficient when agent behavior depends on orchestration, tools, business rules, and runtime context.
- Large...
engineering blog · source date 2026-01-26 · 0 comments ·
original1. Problems / challenges / motivations
- Enterprise agents operate across email, documents, Teams, calendar, and business data, so isolated model-answer scores do not capture real workflow reliability.
- Organizations need evals that reflect local policies, schemas, permissions, and business constraints rather than generic public leaderboard tasks.
-...
engineering blog · source date 2026-01-09 · 1 comments ·
original1. Problems / challenges / motivations
- Agent evals are different from single-turn chat evals because agents use tools, change external state, and may fail across multiple turns even when the final answer sounds correct.
- Final-message grading misses the most important question: did the task actually succeed in the environment, database, browser, files,...