harbor-framework/terminal-bench
A benchmark for LLMs on complicated tasks in the terminal
Build with Backblaze B2
SDKs, agent skills, IDE extensions, and reference pipelines from Backblaze Labs. All open source.
Loading star history...
Where it stands
Rank, growth pace and release cadence relative to everything GStars tracks.
The project lists 93 contributors on GitHub.
Use Cases & Benefits
- Provides a benchmark suite to evaluate AI agents on complex terminal-based tasks autonomously.
- Enables reproducible, real-world testing of language models in sandboxed terminal environments with a comprehensive task dataset and execution harness.
- Use for benchmarking AI agents' ability to perform system-level tasks like compiling code and setting up servers.
- Use for stress-testing language models on end-to-end workflows involving training machine learning models in terminal environments.
- Use for developing and validating new AI agent frameworks that interact with text-based operating system shells.
Should you depend on it?
License terms, project hygiene and risks, from GitHub metadata and GStars' snapshots.
License
Apache 2.0
Permissive: commercial use, modification and redistribution are allowed with attribution; includes an explicit patent grant from contributors.
Community files
- README: present
Watch-outs
- No GitHub releases; pin a commit or tag instead of a release.
General guidance from repository metadata, not legal advice. How we score
From the README
Terminal-Bench is the benchmark for testing AI agents in real terminal environments. From compiling code to training models and setting up servers, Terminal-Bench evaluates how well agents can handle real-world, end-to-end tasks - autonomously.
Whether you're building LLM agents, benchmarking frameworks, or stress-testing system-level reasoning, Terminal-Bench gives you a reproducible task suite and execution harness designed for practical, real-world evaluation.
Terminal-Bench consists of two parts: a dataset of tasks, and an execution harness that connects a language model to our terminal sandbox.
Terminal-Bench is currently in beta with ~100 tasks. Over the coming months, we are going to expand Terminal-Bench into comprehensive testbed for AI agents in text-based environments. Any contributions are welcome, especially new and challenging tasks!
Our Quickstart Guide will walk you through installing the repo and contributing.
Terminal-Bench is distributed as a pip package and can be run using the Terminal-Bench CLI:
tb.
Daily snapshots
Stars gained per day, and how forks and open issues moved, from GStars' daily snapshots
Forks
564+16
Open issues
319+9
Compared with peers in CLI
Fastest-growing repositories GStars tracks in this cohort for the week of Nov 1, by 7-day star growth
| Repository | Stars | +7d | Health | Last release |
|---|---|---|---|---|
| vercel-labs/agent-desktop | 17.0K | +13.3K | 33 | today |
| mattpocock/skills | 315.4K | +8.7K | 35 | 7 days ago |
| stablyai/orca | 116.8K | +6.6K | 40 | today |
| obra/superpowers | 313.1K | +4.5K | 53 | 2 days ago |
| Panniantong/Agent-Reach | 109.3K | +3.0K | 60 | 4 days ago |
| harbor-framework/terminal-bench | 2.5K | — | 35 | — |
| Category median | 116.8K | +6.6K | 40 | — |
Also by harbor-framework
Other repositories from this maintainer that GStars tracks
Discover Repositories
Search across tracked repositories by name or description