benchmark paper · source date 2026-07-21 · 0 comments ·
original1. Problems / challenges / motivations
- Factuality work overwhelmingly measures precision (is what was said true?) and neglects completeness (was anything important left out?).
- Rich structured rubrics are hard for an LLM judge to apply consistently.
- Rubric-based scores often move when you swap the judge model, which makes them hard to trust.
2. Key...
arXiv paper · source date 2026-07-13 · 0 comments ·
original1. Problems / challenges / motivations
- LLM-judge bias is well documented at the input-output level (position bias, verbosity bias, self-preference) but not explained mechanistically.
- Debiasing is therefore trial-and-error prompt engineering.
- There is no way to predict in advance whether a judge will fail on a new benchmark.
2. Key ideas
- Judge...
peer-reviewed paper · source date 2026-07-10 · 0 comments ·
original1. Problems / challenges / motivations
- Safety evaluation ultimately rests on human ratings, and the composition of the rater pool is rarely treated as a measurement parameter.
- Teams increasingly substitute LLM raters for humans to cut cost, assuming the substitution is roughly lossless.
- If both assumptions fail, safety scores measure the rater pool as...
engineering blog · source date 2026-07-08 · 0 comments ·
original1. Problems / challenges / motivations
- Teams adopt LLM judges to scale agent evaluation and then never check whether the judge itself is reliable.
- Observability research tends to land as separate studies rather than one usable path.
- "Ship when evals look good" needs an actual gating mechanism to mean anything.
2. Key ideas
- Rolls up five Microsoft...
preprint · source date 2026-05-31 · 0 comments ·
original1. Problems / challenges / motivations
- As LLMs move from task-specific systems toward open-ended agents, one scalar score is often too opaque. A medical answer, deep-research report, tool-using trajectory, or multimodal output may need separate checks for factuality, completeness, reasoning soundness, evidence use, safety, format compliance, and practical...