Terminal-Bench 4.0
Calibrating task resources, fixing tasks, and removing saturated tasks
We've pushed a version update to the Terminal-Bench dataset and leaderboard. Terminal-Bench 4.0 calibrates task resources (time, CPU, memory), implements task fixes, and removes saturated tasks.
TB 4.0 has fewer agent timeouts and errors than 3.0, reducing measurement noise. The remaining errors in 4.0 are largely model refusals and output-token-exceeded errors. The tasks represent a wide spread of difficulty.
Calibrating resources
To calibrate resources we followed the methodology outlined in Quantifying infrastructure noise in agentic coding evals. By running the tasks with a large amount of resource headroom, we can select a reasonable setting that does not impact the final metrics.
All tasks in TB 4.0 are now set to a flat agent timeout of 8 hours. Frontier models never or rarely encounter timeouts now.
We noticed that Sonnet 5 sometimes hit timeouts and output-token-exceeded errors. The model uses large amounts of tokens (21.6B on its leaderboard run vs 6.5B for Opus 5) and struggles to stay within its max output tokens. We did not enable the 128k max-output-tokens setting, which may help with this. Regardless, we observed large variance in agent execution time for Sonnet 5.
Removing and fixing tasks
We removed 8 tasks: for saturation (2), refusals (2), public solutions (2), and unresolved quality or platform-compatibility issues (2). We considered a task "saturated" when all classes within all families of the latest generation of models solve it 5/5 times. Such a task no longer meaningfully differentiates between models. We fixed 19 tasks, updating instructions, environments, and verifiers to address flakiness or misspecification reported by users or found in our leaderboard runs.
Full details are in the GitHub release and each individual task's README changelog.
Versioning Terminal-Bench
Why is this 4.0 instead of 3.1? Terminal-Bench is now a continuous benchmark and versioning is now semantic.
This update included resource changes in the agent environment. It also changed the task set by removing saturated tasks. These represent breaking changes that require re-running trials.
In the short term, this might be confusing as the community is used to major version updates being "sequels". However, building benchmarks from scratch is enormously challenging, and benchmarks are complex simulations that contain bugs and need to be maintained. We think continuously updating, pruning, and adding tasks is ultimately a better strategy for building benchmarks.
Current results
Terminal-Bench 4.0 is built on the Harbor framework and is available on the Harbor Hub:
harbor run -d terminal-bench/terminal-bench@4.0.0There is still lots of room to improve on both cost and performance for TB 4.0, but our work does not stop here. We are looking for more difficult, diverse, and high-quality tasks.
Improving Terminal-Bench
Everything we do is in the open.
You can see how we triaged issues (validated each as a true or false positive) and implemented and reviewed fixes in the v4.0 milestone.
The milestones for upcoming releases are live:
- Terminal-Bench 4.1: verifier improvements (e.g. tamper-resistant verifiers)
- Terminal-Bench 5.0: new tasks
You are welcome to participate by reporting task issues or coming up with new tasks!
Acknowledgements
Thank you to the community, task authors, senior reviewers, and supporting organizations who opened task issues, created task fixes, and reviewed task fixes: Justin Bauer, Björn Plüster, Dariush Wahdany, Jiajun Bao, Kobe Chen, Emmy Liu, Raj Movva, Tommaso Cerruti, Stephen Benjamin, Mikhail Liamets, Ivan Bercovich, Benedikt Droste, Steven Dillmann, Ruiyang Wang, Allen Hart, Karl Krauth, Aaron Feller, Christine Sutcliffe, Jialin Ouyang, Josh Viktorov, Boxuan Li, Haowei Lin, Maksim Ivanov, Shengrui Lyu, Sherif Armanyous, Shivank Garg, Sergey Bogdanik, Walker Hughes, Bespoke Labs, Turing, ellamind AI, gNucleus AI, Scale AI, and Snorkel AI.
The leaderboard experiments are supported with grants from OpenAI, Anthropic, Z.ai, SpaceX AI, and Laude Institute.
Special thanks to Gian Segato, who devised the resource calibration methodology and ran the calibration experiments.
Written by