Dao-AILab

    Dao-AILab/flash-attention

    Fast and memory-efficient exact attention

    backend
    Python
    BSD-3-Clause
    25.3K stars
    3.2K forks
    144 GitHub watchers
    Updated Nov 17, 2026
    View on GitHub

    Build with Backblaze B2

    SDKs, agent skills, IDE extensions, and reference pipelines from Backblaze Labs. All open source.

    Explore Backblaze Labs

    Loading star history...

    Where it stands

    Rank, growth pace and release cadence relative to everything GStars tracks.

    5 releases in the last 90 days, the latest 6 days ago, with 222 contributors on GitHub.

    Use Cases & Benefits

    • Implements fast and memory-efficient exact attention mechanisms optimized for GPUs using CUDA and ROCm.
    • Delivers significant speedups and memory savings over standard attention, enabling longer sequence processing with lower resource use.
    • Use for accelerating training and inference of large transformer models on NVIDIA GPUs like A100 and H100.
    • Use for implementing sliding window local attention and ALiBi bias in custom attention layers for efficient long-context modeling.
    • Use for inference with KV cache updates and rotary embeddings to optimize autoregressive decoding in language models.

    Should you depend on it?

    License terms, project hygiene and risks, from GitHub metadata and GStars' snapshots.

    License

    BSD 3-Clause

    Permissive: commercial use, modification and redistribution are allowed with attribution.

    Community files

    • README: present

    Watch-outs

    No watch-outs from GStars' checks.

    General guidance from repository metadata, not legal advice. How we score

    From the README

    This repository provides the official implementation of FlashAttention and FlashAttention-2 from the following papers.

    FlashAttention: Fast and Memory-Efficient Exact Attention with IO-Awareness Tri Dao, Daniel Y. Fu, Stefano Ermon, Atri Rudra, Christopher Ré Paper: https://arxiv.org/abs/2205.14135 IEEE Spectrum article about our submission to the MLPerf 2.0 benchmark using FlashAttention.

    FlashAttention-2: Faster Attention with Better Parallelism and Work Partitioning Tri Dao

    We've been very happy to see FlashAttention being widely adopted in such a short time after its release. This page contains a partial list of places where FlashAttention is being used.

    FlashAttention and FlashAttention-2 are free to use and modify (see LICENSE). Please cite and credit FlashAttention if you use it.

    FlashAttention-3 is optimized for Hopper GPUs (e.g. H100).

    This is a beta release for testing / benchmarking before we integrate that with the rest of the repo.

    Requirements: H100 / H800 GPU, CUDA >= 12.3.

    We highly recommend CUDA 12.8 for best performance.

    Excerpt from the Dao-AILab/flash-attention README on GitHub (BSD-3-Clause license). © its authors.Read the full README on GitHub

    Daily snapshots

    Stars gained per day, and how forks and open issues moved, from GStars' daily snapshots

    Forks

    3.2K+220

    Open issues

    1.4K+131

    Compared with peers in Backend

    Fastest-growing repositories GStars tracks in this cohort for the week of Nov 15, by 7-day star growth

    Dao-AILab/flash-attention compared with Backend repositories by stars, 7-day star growth, health score and last release
    RepositoryStars+7dHealthLast release
    anthropics/skills198.5K+3.9K15—
    tradermonty/claude-trading-skills7.6K+2.8K35—
    we-promise/sure17.1K+1.8K40today
    DuarteSantos8/openGym23.1K+1.8K607 days ago
    agent-substrate/substrate8.1K+1.5K651 day ago
    Dao-AILab/flash-attention25.3K—556 days ago
    Category median17.1K+1.8K40—

    Also by Dao-AILab

    Other repositories from this maintainer that GStars tracks

    Discover Repositories

    Search across tracked repositories by name or description