Dao-AILab/flash-attention
Fast and memory-efficient exact attention
Build with Backblaze B2
SDKs, agent skills, IDE extensions, and reference pipelines from Backblaze Labs. All open source.
Loading star history...
Where it stands
Rank, growth pace and release cadence relative to everything GStars tracks.
Ranks #287 of 1,950 repositories GStars tracks this week (top 15%).
5 releases in the last 90 days, the latest today, with 212 contributors on GitHub.
Use Cases & Benefits
- Implements fast and memory-efficient exact attention mechanisms optimized for GPUs using CUDA and ROCm.
- Delivers significant speedups and memory savings over standard attention, enabling longer sequence processing with lower resource use.
- Use for accelerating training and inference of large transformer models on NVIDIA GPUs like A100 and H100.
- Use for implementing sliding window local attention and ALiBi bias in custom attention layers for efficient long-context modeling.
- Use for inference with KV cache updates and rotary embeddings to optimize autoregressive decoding in language models.
Should you depend on it?
License terms, project hygiene and risks, from GitHub metadata and GStars' snapshots.
License
BSD 3-Clause
Permissive: commercial use, modification and redistribution are allowed with attribution.
Community files
- README: present
Watch-outs
No watch-outs from GStars' checks.
General guidance from repository metadata, not legal advice. How we score
From the README
This repository provides the official implementation of FlashAttention and FlashAttention-2 from the following papers.
FlashAttention: Fast and Memory-Efficient Exact Attention with IO-Awareness Tri Dao, Daniel Y. Fu, Stefano Ermon, Atri Rudra, Christopher Ré Paper: https://arxiv.org/abs/2205.14135 IEEE Spectrum article about our submission to the MLPerf 2.0 benchmark using FlashAttention.
FlashAttention-2: Faster Attention with Better Parallelism and Work Partitioning Tri Dao
We've been very happy to see FlashAttention being widely adopted in such a short time after its release. This page contains a partial list of places where FlashAttention is being used.
FlashAttention and FlashAttention-2 are free to use and modify (see LICENSE). Please cite and credit FlashAttention if you use it.
FlashAttention-3 is optimized for Hopper GPUs (e.g. H100).
This is a beta release for testing / benchmarking before we integrate that with the rest of the repo.
Requirements: H100 / H800 GPU, CUDA >= 12.3.
We highly recommend CUDA 12.8 for best performance.
Daily snapshots
Stars gained per day, and how forks and open issues moved, from GStars' daily snapshots
Forks
3.2K+241
Open issues
1.3K+133
Compared with peers in Backend
Fastest-growing repositories GStars tracks in this cohort for the week of Oct 11, by 7-day star growth
| Repository | Stars | +7d | Health | Last release |
|---|---|---|---|---|
| DuarteSantos8/openGym | 11.1K | +5.4K | 60 | 3 days ago |
| anthropics/knowledge-work-plugins | 29.5K | +3.2K | 75 | — |
| 666ghj/MiroFish | 79.0K | +2.2K | 23 | 220 days ago |
| microsoft/mxc | 2.8K | +1.4K | 65 | 6 days ago |
| danyuchn/asd-ste100-skill | 4.7K | +1.0K | 75 | — |
| Dao-AILab/flash-attention | 25.1K | +0 | 75 | today |
| Category median | 11.1K | +2.2K | 65 | — |
Also by Dao-AILab
Other repositories from this maintainer that GStars tracks
Discover Repositories
Search across tracked repositories by name or description