Two-Level Meta-Rubrics for Open-Ended Generation: GAMUT
benchmark paper · source date 2026-07-21 · added 2026-08-07 15:52:43 · updated 2026-08-07 15:52:43 · Open original blog
1
Problems / challenges / motivations
- Factuality work overwhelmingly measures precision (is what was said true?) and neglects completeness (was anything important left out?).
- Rich structured rubrics are hard for an LLM judge to apply consistently.
- Rubric-based scores often move when you swap the judge model, which makes them hard to trust.
2
Key ideas
- A two-level meta-rubric framework compiles a structured rubric into flat, binary, machine-gradable checklists that an LLM judge can evaluate item by item.
- Instantiated as GAMUT: 1,813 expert-verified questions grounded in real wearable imagery across 10 domains, measuring factual completeness.
- The best of 14 frontier and open models reaches only 58.7% (Gemini 3.1 Pro), so the benchmark is far from saturated.
- Scores are reported as robust to judge choice; a text-only variant is also released.
3
Why it matters for evals
- The rubric-compilation method is directly reusable in any LLM-as-judge pipeline: flattening a rubric into binary checks is what makes judge scores stable and auditable.
- Completeness is the under-measured half of factuality, and a benchmark where the frontier sits under 60% has headroom to be useful for a while.
- Caveat: an integrity flag from the scan — one scout could not confirm the arXiv abstract page for 2607.19322, and org attribution varied. Treat as a preprint and verify before citing as venue-published.
Comments
No comments yet.