KellerJordan/modded-nanogpt
NanoGPT (124M) in 90 seconds
Build with Backblaze B2
SDKs, agent skills, IDE extensions, and reference pipelines from Backblaze Labs. All open source.
Loading star history...
Where it stands
Rank, growth pace and release cadence relative to everything GStars tracks.
The project lists 68 contributors on GitHub.
Peaked at #58 in the weekly ranking the week of Oct 26, its best in the past year.
Use Cases & Benefits
- Implements a highly optimized training pipeline for NanoGPT to achieve rapid language model training on 8 NVIDIA H100 GPUs.
- Delivers state-of-the-art training speed by combining advanced architectural tweaks, the Muon optimizer, and system-level optimizations to drastically reduce training time and token requirements.
- Use for benchmarking and accelerating GPT-2 small model training to a target validation loss efficiently on high-end GPU clusters.
- Use for experimenting with novel optimizer designs and architectural modifications like rotary embeddings and FlexAttention in PyTorch-based language models.
- Use for research and development of competitive speedrunning techniques in large language model training to push performance boundaries.
Should you depend on it?
License terms, project hygiene and risks, from GitHub metadata and GStars' snapshots.
License
MIT
Permissive: commercial use, modification and redistribution are allowed with attribution.
Community files
- README: present
Watch-outs
- No GitHub releases; pin a commit or tag instead of a release.
General guidance from repository metadata, not legal advice. How we score
From the README
This repository hosts the NanoGPT speedrun, in which we (collaboratively|competitively) search for the fastest algorithm to use 8 NVIDIA H100 GPUs to train a language model that attains 3.28 cross-entropy loss on the FineWeb validation set.
The target (3.28 validation loss on FineWeb) follows Andrej Karpathy's GPT-2 replication in llm.c, which attains that loss after running for 45 minutes. The speedrun code also descends from llm.c's PyTorch trainer, which itself descends from NanoGPT, hence the name of the repo. Thanks to the efforts of many contributors, this repo now contains a training algorithm which attains the target performance in:
This improvement in training speed has been brought about by the following techniques:
As well as many systems optimizations.
Daily snapshots
Stars gained per day, and how forks and open issues moved, from GStars' daily snapshots
Forks
925+72
Open issues
75+28
Compared with peers in Backend
Fastest-growing repositories GStars tracks in this cohort for the week of Nov 1, by 7-day star growth
| Repository | Stars | +7d | Health | Last release |
|---|---|---|---|---|
| anthropics/skills | 187.8K | +2.9K | 15 | — |
| DuarteSantos8/openGym | 19.7K | +2.4K | 60 | 2 days ago |
| shiyu-coder/Kronos | 47.0K | +2.3K | 16 | — |
| NationalSecurityAgency/ghidra | 98.2K | +2.3K | 64 | 14 days ago |
| microsoft/VibeVoice | 56.0K | +1.1K | 29 | — |
| KellerJordan/modded-nanogpt | 6.0K | — | 35 | — |
| Category median | 56.0K | +2.3K | 29 | — |
Discover Repositories
Search across tracked repositories by name or description