Introducing GeneBench-Pro
benchmark release · source date 2026-06-30 · added 2026-08-07 15:52:43 · updated 2026-08-07 15:52:43 · Open original blog
1
Problems / challenges / motivations
- Scientific-reasoning benchmarks mostly test recall or single-step analysis, not the judgment calls that make research hard.
- Real analysis has dependent decision forks: an early wrong turn invalidates everything downstream.
- Benchmarks authored by a lab whose models top them are easy to discount.
2
Key ideas
- 129 research-level problems measuring "research taste" — multi-stage, judgment-heavy computational-biology analysis with dependent decision forks.
- Grading is deterministic, run on synthetic data with known causal structure, so the right answer is defined by construction rather than by a judge.
- GPT-5.6 Sol reaches 28.7% (31.5% Pro); the strongest non-GPT baseline is Claude Opus 4.8 at 16.0%.
- The companion paper documents estimand-based grading, external review of 82 problems, and a 50-problem holdout handed to Artificial Analysis for third-party benchmarking.
3
Why it matters for evals
- The holdout hygiene is the interesting part: giving a third party a private slice is a cheap, copyable defense against both contamination and self-scoring.
- The results highlight a "noticing versus acting" gap — models often identify the right concern but fail to act on it across a long analysis.
- Caveat: OpenAI-authored and its models top the leaderboard; the third-party holdout partly but not fully mitigates that.
Comments
No comments yet.