Terminal-Bench Science: Contribute your scientific workflows as tasks for AI Agents

Terminal-Bench Science is now open for contributions — we're looking for scientists to turn real research workflows into benchmark tasks that shape the next generation of AI agents.

ABOUT

What is Terminal-Bench Science?

Terminal-Bench Science is a benchmark for evaluating AI agents on real computational workflows from scientific research. It builds on Terminal-Bench, which has been adopted by frontier labs including Anthropic, OpenAI, and Google DeepMind and has helped drive progress in AI agents on software engineering tasks by defining what those labs measure and optimize for. Terminal-Bench Science brings the same approach to the natural sciences.

Why do we need Terminal-Bench Science?

Most existing "AI for Science" benchmarks test textbook knowledge, not real workflows. Terminal-Bench Science closes this gap with real computational workflow tasks from research labs, evaluated in containerized environments with programmatic verification. Our goal is to give scientists a direct voice in shaping AI progress: domain experts contribute scientific workflows as benchmark tasks, frontier labs evaluate and improve their AI agents against them, and the improved AI agents with stronger scientific capabilities flow back as better tools for researchers.

Domain Coverage

Terminal-Bench Science is targeting 100+ benchmark tasks across the life sciences, physical sciences, and earth sciences, but is also open to tasks from the mathematical sciences and other domains with computational workflows.

CONTRIBUTE

Why Contribute?

1. Make AI better at your science. Frontier labs optimize for what benchmarks measure. Your tasks directly incentivize them to improve their AI systems on the scientific problems in your domain.
2. Gain experience in agentic evaluation. Get hands-on with evaluating frontier AI agents — learn how to design rigorous benchmarks and see firsthand where today's best models succeed and fail on real scientific work.
3. Become a co-author. Contributors with merged tasks receive co-authorship on the Terminal-Bench Science paper.

What We're Looking For

We're looking for complex, real-world computational workflows from practicing scientists across the natural sciences that meet the following three key criteria:

1. Scientifically grounded. Tasks should reflect computational workflows from real research in the natural sciences — ideally drawn from your own work or replicating published results in your domain of expertise.

2. Objectively verifiable. Solutions must be programmatically checkable with deterministic pytest-based evaluation. We are not looking for open-ended tasks like hypothesis generation or literature review.

3. Genuinely difficult. We target tasks that today's best AI agents cannot yet reliably solve. Hard tasks expose real gaps and push capabilities forward — we're aiming for a 10–20% solve rate at release.

Tasks follow the Harbor Task Format. Check out example tasks for reference.

How to Contribute

We follow a curated contribution process to maintain quality:

1. Connect — Join our Discord, introduce yourself in #tb-science, and pitch your task idea in #tb-science-task-ideas for early feedback. Follow #tb-science-announcements for updates and weekly meetings (Mondays, 9am PT).

2. Propose — When you're ready, submit your idea via the Task Proposal Form. Proposals are posted on our Task Proposal Board and in #tb-science-task-proposals. An LLM judge evaluates it against our Task Proposal Rubric, and human reviewers use that to approve your proposal and guide you toward implementation.

3. Build — Once approved, build the task in the Harbor Task Format and submit a Pull Request following our Contributing Guide. Your implementation is evaluated against our Task Implementation Rubric, and human reviewers also assess difficulty, scientific quality, and overall fit. We work with you iteratively until it's ready to merge.

Once merged, we run frontier AI agents against all merged tasks to calibrate difficulty. Tasks that pass are included in the official Terminal-Bench Science release on the Terminal-Bench Benchmarks and Terminal-Bench Leaderboards.

Authorship & Credit

Contributors with merged tasks receive co-authorship on the Terminal-Bench Science research paper and are credited on the Terminal-Bench Contributors page. Author order is determined by the number and impact of accepted tasks. Faculty who bring in contributors or review tasks as domain experts are eligible for senior authorship.

Deadline

Tasks must be submitted and merged by August 17, 2026. Starting early is highly recommended — most tasks require a few rounds of feedback and iteration before they're ready to merge.

RESOURCES

Join our Discord and reach out to @stevendi11 on Discord or stevendi@stanford.edu to get involved. Key channels: #tb-science for general discussion, #tb-science-announcements for project updates, #tb-science-task-ideas for quick early feedback on ideas, and #tb-science-task-proposals for submitted proposals, automated reviews, and reviewer feedback. Join our weekly meeting every Monday at 9am PT.

Links

• Discord — #tb-science, #tb-science-announcements, #tb-science-task-ideas, #tb-science-task-proposals
• GitHub — source code and task submissions
• Weekly Meeting — 9am PT every Monday
• Harbor — learn the task format and run Terminal-Bench evaluations
• Example Tasks — browse existing tasks for reference
• Task Proposal Form — submit your task idea
• Task Proposal Board — browse and discuss proposals
• Contributing Guide — how to build and submit your task PR

Acknowledgements

Terminal-Bench Science is an open academic collaboration hosted by Stanford University and the Laude Institute. As part of the Terminal-Bench franchise, it is built by the Terminal-Bench & Harbor Framework team, and scientific contributors. We thank the Laude Institute and 2077AI for API credits that power benchmark evaluations.

Contact

For questions, feedback, or if you're interested in contributing, reach out to Steven Dillmann at stevendi@stanford.edu.