huggingface/tokenizers
💥 Fast State-of-the-Art Tokenizers optimized for Research and Production
Build with Backblaze B2
SDKs, agent skills, IDE extensions, and reference pipelines from Backblaze Labs. All open source.
Loading star history...
Where it stands
Rank, growth pace and release cadence relative to everything GStars tracks.
Ranks #68 among AI repositories tracked this week, and #68 over the last 30 days; #466 of 2,845 repositories GStars tracks this week (top 16%).
3 releases in the last 90 days, the latest 6 days ago, with 146 contributors on GitHub.
Peaked at #38 in the weekly ranking the week of Sep 27, its best in the past year.
Use Cases & Benefits
- Provides fast, state-of-the-art tokenizers optimized for both research and production environments.
- Delivers extremely high performance tokenization and training speeds through a Rust-based implementation.
- Use for training custom tokenizers on large text corpora with popular algorithms like BPE, WordPiece, or Unigram.
- Use for integrating efficient tokenization in NLP pipelines requiring alignment tracking and special token handling.
- Ideal for developers needing multi-language bindings to tokenize text in Python, Node.js, or Rust applications.
Should you depend on it?
License terms, project hygiene and risks, from GitHub metadata and GStars' snapshots.
License
Apache 2.0
Permissive: commercial use, modification and redistribution are allowed with attribution; includes an explicit patent grant from contributors.
Community files
- README: present
- Contributing guide: present
Watch-outs
No watch-outs from GStars' checks.
General guidance from repository metadata, not legal advice. How we score
From the README
Provides an implementation of today's most used tokenizers, with a focus on performance and versatility.
Daily snapshots
Stars gained per day, and how forks and open issues moved, from GStars' daily snapshots
+263 stars over 84 days across 12 snapshot days; best day +22 on Sep 23.
Forks
1.2K+70
Open issues
208-44
Compared with peers in AI
Fastest-growing repositories GStars tracks in this cohort for the week of Oct 11, by 7-day star growth
| Repository | Stars | +7d | Health | Last release |
|---|---|---|---|---|
| storytold/artcraft | 35.6K | +20.3K | 75 | 26 days ago |
| ifixai-ai/iFixAi | 30.7K | +6.9K | 75 | 5 days ago |
| heygen-com/hyperframes | 70.0K | +5.6K | 45 | today |
| vxcontrol/pentagi | 32.0K | +5.4K | 75 | 17 days ago |
| calesthio/OpenMontage | 70.9K | +3.2K | 75 | — |
| huggingface/tokenizers | 11.2K | +0 | 75 | 6 days ago |
| Category median | 35.6K | +5.6K | 75 | — |
Also by huggingface
Other repositories from this maintainer that GStars tracks
- Python167.6K stars
transformers
🤗 Transformers: the model-definition framework for state-of-the-art machine learning models in text, vision, audio, and multimodal models, for both inference and training.
- Python37.2K stars
pytorch-image-models
The largest collection of PyTorch image encoders / backbones. Including train, eval, inference, export scripts, and pretrained weights -- ResNet, ResNeXT, EfficientNet, NFNet, Vision Transformer (ViT), MobileNetV4, MobileNet-V3 & V2, RegNet, DPN, CSPNet, Swin Transformer, MaxViT, CoAtNet, ConvNeXt, and more
- Python34.7K stars
diffusers
🤗 Diffusers: State-of-the-art diffusion models for image, video, and audio generation in PyTorch.
- Python28.0K stars
lerobot
🤗 LeRobot: Making AI for Robotics more accessible with end-to-end learning
- Python22.0K stars
datasets
🤗 The largest hub of ready-to-use datasets for AI models with fast, easy-to-use and efficient data manipulation tools
Discover Repositories
Search across tracked repositories by name or description