AI & Agent Evaluation
2,152total visitsadmin
reading room / notes / ai engineering

Reading room

Short summaries of agent optimization and AI engineering work — harnesses, prompt and context optimization, orchestration, memory, and agentic RL.

$ evals.index --public
category: ai engineering
posts: 6
mode: short summaries
storage: postgres
status: listening
ai eval (43)ai engineering (28)
Sort by source dateLatest firstEarliest first

Filtering by benchmarks. Clear filter.

Capable Language Models Can Outgrow the Benefits of Collaboration

peer-reviewed paper · source date 2026-05-01 · 0 comments · original

1. Problems / challenges / motivations - Multi-agent debate and collaboration results were mostly established on weaker base models and assumed to hold as models improve. - Teams have no decision rule for when to add agents versus when a single strong model suffices. - Collaboration adds cost and latency that need justification. 2. Key ideas - Quantifies...

Towards a Science of Scaling Agent Systems

research blog · source date 2026-04-01 · 0 comments · original

1. Problems / challenges / motivations - Multi-agent systems are adopted on the assumption that more agents means better results, with little controlled evidence. - Comparisons across papers confound topology, model, and token budget. - Errors propagate differently depending on how agents are wired together, and that effect is rarely measured. 2. Key...

Huxley-Gödel Machine: Human-Level Coding Agent Development

peer-reviewed paper · source date 2025-10-24 · 0 comments · original

1. Problems / challenges / motivations - Self-modification search scores a variant by its own benchmark performance, but the useful property is whether it leads somewhere — the two come apart. - The paper names this the Metaproductivity–Performance Mismatch: a mediocre variant can be the ancestor of the best ones. - Expanding the wrong branches burns large...

The Landscape of Agentic Reinforcement Learning for LLMs: A Survey

peer-reviewed survey · source date 2025-09-02 · 0 comments · original

1. Problems / challenges / motivations - Agentic RL work is spread across planning, tool use, memory, reasoning, and perception with no shared taxonomy. - Standard LLM RL formulations assume a single-step MDP, which does not describe an agent acting over a long horizon with partial observability. - Environments, benchmarks, and frameworks are fragmented and...

Context Rot: How Increasing Input Tokens Impacts LLM Performance

technical report · source date 2025-07-01 · 0 comments · original

1. Problems / challenges / motivations - Long context windows are marketed as capacity, and teams treat "fits in the window" as "will be used correctly." - Needle-in-a-haystack tests use high semantic similarity between question and needle, which is the easy case. - Without evidence of degradation, there is no case for the engineering cost of compaction and...

Darwin Gödel Machine: Open-Ended Evolution of Self-Improving Agents

peer-reviewed paper · source date 2025-05-30 · 0 comments · original

1. Problems / challenges / motivations - The classical Gödel machine requires a formal proof that a self-modification is an improvement, which is infeasible for real systems. - Agent design is currently done by hand, one scaffold change at a time. - Greedy self-modification collapses into a local optimum and loses variants that would have paid off...