Long-Horizon-Terminal-Bench (LHTB)

benchmark paper · source date 2026-07-09 · added 2026-08-07 15:52:43 · updated 2026-08-07 15:52:43 · Open original blog

Terminal-agent benchmarks are saturating while real agentic work runs far longer than any of them.
Binary pass/fail on a multi-hour task throws away almost all the signal in the run.
Long tasks are expensive, so grading has to be worth the compute spent producing the trajectory.

46 long-horizon containerized tasks across 9 categories, each decomposed into fine-grained graded subtasks.
Dense intermediate rewards plus hidden verifiers give partial credit along the trajectory instead of one terminal bit.
Runs are enormous: roughly 239 episodes, ~9.9M tokens, and ~85–89 minutes per task.
Even the strongest frontier models remain far from saturation.

It pushes terminal-agent evaluation about an order of magnitude beyond Terminal-Bench 2 in horizon, and it is the clearest instance of the shift to dense, partial-credit, trajectory-aware grading.
The subtask decomposition is the reusable idea: it converts an all-or-nothing long task into a progress curve you can actually diff between models.
It was the #1 Hugging Face Paper of its day and appeared across four scan lanes.
Caveat: headline numbers and attribution are inconsistent across sources — reported bests range from ~15% pass@1 at the 0.95 reward threshold to "Grok 4.5 28.3%," and org attribution varies (Tencent HY Frontier versus Lehigh + Tencent). Reconcile against the primary source before quoting a number.

Comments

No comments yet.