datacurve-ai/deep-swe
Measuring frontier coding agents on original, long-horizon engineering tasks
Build with Backblaze B2
SDKs, agent skills, IDE extensions, and reference pipelines from Backblaze Labs. All open source.
Loading star history...
Where it stands
Rank, growth pace and release cadence relative to everything GStars tracks.
The project lists 2 contributors on GitHub.
Use Cases & Benefits
- Provides a benchmark to evaluate coding agents on complex, long-horizon software engineering tasks across multiple programming languages.
- Enables isolated, reproducible testing environments with program-based verifiers ensuring behavior correctness beyond code structure.
- Use for assessing AI coding agents' performance on real-world, multi-language open-source engineering challenges.
- Use for developing and tuning sandboxed coding agents with controlled network access and detailed trajectory analysis.
- Use for running targeted evaluations on subsets or individual tasks to diagnose agent capabilities or regressions.
Should you depend on it?
License terms, project hygiene and risks, from GitHub metadata and GStars' snapshots.
License
Apache 2.0
Permissive: commercial use, modification and redistribution are allowed with attribution; includes an explicit patent grant from contributors.
Community files
- README: present
Watch-outs
- No GitHub releases; pin a commit or tag instead of a release.
General guidance from repository metadata, not legal advice. How we score
From the README
DeepSWE is a benchmark for measuring frontier coding agents on original, long-horizon software engineering tasks drawn from active open-source repositories. The benchmark includes 113 tasks across TypeScript, Go, Python, JavaScript, and Rust, with isolated environments and program-based verifiers.
DeepSWE tasks use the Harbor task format:
The verifier exercises the behavior the prompt describes. It accepts any solution whose observable behavior is correct, regardless of internal symbol names or structure. The reference patch in
solution/is never used at grading time; it exists so reviewers can spot-check correctness offline.Since v1.1, grading uses Harbor's separate verifier environment, requiring Pier newer than 0.3.0. The agent works in an isolated environment and commits its work upon completion. A
[[verifier.collect]]hook in eachtask.tomlthen extracts these commits as a patch, which is applied and graded in a pristine container.
Daily snapshots
Stars gained per day, and how forks and open issues moved, from GStars' daily snapshots
+294 stars over 60 days across 6 snapshot days; best day +13 on Sep 4.
Forks
128+22
Open issues
86+14
Compared with peers in Backend
Fastest-growing repositories GStars tracks in this cohort for the week of Nov 1, by 7-day star growth
| Repository | Stars | +7d | Health | Last release |
|---|---|---|---|---|
| anthropics/skills | 187.8K | +2.9K | 15 | — |
| DuarteSantos8/openGym | 19.7K | +2.4K | 60 | 2 days ago |
| shiyu-coder/Kronos | 47.0K | +2.3K | 16 | — |
| NationalSecurityAgency/ghidra | 98.2K | +2.3K | 64 | 14 days ago |
| microsoft/VibeVoice | 56.0K | +1.1K | 29 | — |
| datacurve-ai/deep-swe | 1.9K | — | 6 | — |
| Category median | 56.0K | +2.3K | 29 | — |
Discover Repositories
Search across tracked repositories by name or description