AI & Agent Evaluation
2,152total visitsadmin
reading room / notes / ai engineering

Reading room

Short summaries of agent optimization and AI engineering work — harnesses, prompt and context optimization, orchestration, memory, and agentic RL.

$ evals.index --public
category: ai engineering
posts: 5
mode: short summaries
storage: postgres
status: listening
ai eval (43)ai engineering (28)
Sort by source dateLatest firstEarliest first

Filtering by reliability. Clear filter.

Solve-Detect-Verify (FlexiVe): Inference-Time Scaling with a Flexible Generative Verifier

peer-reviewed paper · source date 2026-06-01 · 0 comments · original

1. Problems / challenges / motivations - Best-of-N spends the same verification compute on easy and hard candidates alike. - Generative verifiers are accurate but expensive, which limits how often they can be called. - Verifier training and verifier-guided search are usually studied separately. 2. Key ideas - FlexiVe is a generative verifier that...

MemCompiler: Compile, Don't Inject — State-Conditioned Memory for Embodied Agents

arXiv paper · source date 2026-05-01 · 0 comments · original

1. Problems / challenges / motivations - Retrieval-style memory injects a blob of past experience ahead of time, most of which is irrelevant to the current step. - Irrelevant memory costs tokens, adds latency, and actively distracts the policy. - Vector recall quality is not the same thing as memory being useful to an acting agent. 2. Key ideas - Replaces...

Capable Language Models Can Outgrow the Benefits of Collaboration

peer-reviewed paper · source date 2026-05-01 · 0 comments · original

1. Problems / challenges / motivations - Multi-agent debate and collaboration results were mostly established on weaker base models and assumed to hold as models improve. - Teams have no decision rule for when to add agents versus when a single strong model suffices. - Collaboration adds cost and latency that need justification. 2. Key ideas - Quantifies...

Towards a Science of Scaling Agent Systems

research blog · source date 2026-04-01 · 0 comments · original

1. Problems / challenges / motivations - Multi-agent systems are adopted on the assumption that more agents means better results, with little controlled evidence. - Comparisons across papers confound topology, model, and token budget. - Errors propagate differently depending on how agents are wired together, and that effect is rarely measured. 2. Key...

Context Rot: How Increasing Input Tokens Impacts LLM Performance

technical report · source date 2025-07-01 · 0 comments · original

1. Problems / challenges / motivations - Long context windows are marketed as capacity, and teams treat "fits in the window" as "will be used correctly." - Needle-in-a-haystack tests use high semantic similarity between question and needle, which is the easy case. - Without evidence of degradation, there is no case for the engineering cost of compaction and...