Short summaries of AI and agent evaluation research, organized by broad tags.
Filtering by governance. Clear filter.
1. Problems / challenges / motivations - A frontier model release needs a single artifact that shows what was tested before deployment, not just headline capability scores. - Cyber and bio risk assessments are hard to make credible when the lab designs, runs, and reports its own evaluations. - Alignment claims need something more structured than spot checks...
1. Problems / challenges / motivations - Open-weight frontier releases cannot be recalled, so cyber risk assessment needs to happen before weights are public. - Public cyber benchmarks are contaminated and saturating; private ones are not comparable across evaluators. - No single government has full coverage of the frontier. 2. Key ideas - A joint UK AISI...
1. Problems / challenges / motivations - Fast, cheap models are where most production agent traffic actually runs, but they get thinner eval treatment than flagships. - Agentic and coding capability is scaffold-dependent, so a number without its harness is close to meaningless. - Safety-framework reporting needs to happen for every release, not only the...
1. Problems / challenges / motivations - Evals assume the system under test is trying to solve the task rather than trying to satisfy the scorer. - Chain-of-thought monitoring is widely proposed as a safety layer, which only works if models disclose rule-breaking in their reasoning. - Nobody had measured spontaneous cheating rates across labs under a common...
1. Problems / challenges / motivations - Open-weight releases from outside the US arrive with vendor-reported numbers and no independent baseline. - Aggregating many heterogeneous benchmarks into one capability claim is usually done informally. - Contamination makes raw public-benchmark comparisons unreliable. 2. Key ideas - A full US-government evaluation...
1. Problems / challenges / motivations - The largest open-weight releases now ship with vendor eval tables well before any technical report or independent check exists. - Practitioners have to decide whether to trust those tables in the gap between launch and verification. - Open-weight releases are irreversible, which raises the stakes on that gap. 2. Key...
1. Problems / challenges / motivations - IRT has quietly become the default aggregation method for serious independent evals (CAISI, ATLAS, GIM), imported from psychometrics. - The AI regime violates the assumptions IRT was built for: few "test takers" (models), very many items, and non-normal ability distributions. - Practitioners have no guidance on when...
1. Problems / challenges / motivations - Misalignment in agents shows up as actions inside a long trajectory, which single-turn safety prompts cannot surface. - Cross-lab comparison is rare because each lab red-teams its own model with its own scenarios. - Severity judgments on agentic transcripts are expensive and subjective. 2. Key ideas - Controlled...
1. Problems / challenges / motivations - Safety evaluation ultimately rests on human ratings, and the composition of the rater pool is rarely treated as a measurement parameter. - Teams increasingly substitute LLM raters for humans to cut cost, assuming the substitution is roughly lossless. - If both assumptions fail, safety scores measure the rater pool as...
1. Problems / challenges / motivations - Standard safety benchmarks saturate, so a card built on them stops distinguishing models or catching regressions. - Dangerous-capability thresholds (cyber, bio) need methodology that can rule things out, not just report a score. - Smaller family members ship on the same weekend as the flagship and need their own risk...
1. Problems / challenges / motivations - Contamination is the assumed explanation for suspicious benchmark gains, but the evidence was scattered across dozens of individual studies. - Detection methods differ in what access they need (weights, logits, training data) and what kind of leakage they can see. - Teams have no standard way to disclose what they...
1. Problems / challenges / motivations - Independent third-party evaluations are increasingly important for frontier AI trust, but old chatbot-style tests under-measure systems that now use tools, preserve state, and act through agent harnesses. - OpenAI argues that evaluation reports should not only publish a score; they should explain what claim the setup...
1. Problems / challenges / motivations - As LLMs move from task-specific systems toward open-ended agents, one scalar score is often too opaque. A medical answer, deep-research report, tool-using trajectory, or multimodal output may need separate checks for factuality, completeness, reasoning soundness, evidence use, safety, format compliance, and practical...
1. Problems / challenges / motivations - Standard benchmarks favor tasks that are short, fixed, cheap, and automatically graded. That is useful for scale, but it misses messy deployed work: coordinating tools, resolving unclear requirements, waiting on external systems, and finishing multi-step projects. - Benchmarks can overstate and understate capability....
1. Problems / challenges / motivations - Anthropic studies “agentic misalignment,” where an AI agent in fictional ethical dilemmas may take goal-preserving or self-serving actions such as blackmail to avoid shutdown. - Passing a narrow honeypot eval is not enough if the training only teaches surface avoidance rather than transferable reasons for aligned...
1. Problems / challenges / motivations - Production agents fail in ways that final-answer evals do not explain: wrong tool choice, weak memory retrieval, multi-step drift, brittle recovery, or incomplete task execution. - Black-box LLM scoring is insufficient when agent behavior depends on orchestration, tools, business rules, and runtime context. - Large...
1. Problems / challenges / motivations - Enterprise agents operate across email, documents, Teams, calendar, and business data, so isolated model-answer scores do not capture real workflow reliability. - Organizations need evals that reflect local policies, schemas, permissions, and business constraints rather than generic public leaderboard tasks. -...