pathwaycom/pathway
Python ETL framework for stream processing, real-time analytics, LLM pipelines, and RAG.
Build with Backblaze B2
SDKs, agent skills, IDE extensions, and reference pipelines from Backblaze Labs. All open source.
Loading star history...
Where it stands
Rank, growth pace and release cadence relative to everything GStars tracks.
No releases in the last 90 days; the most recent was 95 days ago, with 43 contributors on GitHub.
Peaked at #1 in the weekly ranking the week of Sep 7, its best in the past year.
Use Cases & Benefits
- Provides a Python ETL framework for stream processing, real-time analytics, LLM pipelines, and retrieval-augmented generation (RAG).
- Combines an easy-to-use Python API with a scalable Rust engine enabling multithreading, multiprocessing, and distributed computations for both batch and streaming data.
- Use for building real-time event processing and analytics pipelines integrating with Kafka and other data sources.
- Use for developing live AI pipelines that integrate large language models and RAG with real-time data streams.
- Use for deploying scalable data processing workflows in production environments using Docker and Kubernetes with persistence and consistency guarantees.
Should you depend on it?
License terms, project hygiene and risks, from GitHub metadata and GStars' snapshots.
License
Other
GitHub could not identify this license; read the LICENSE file before depending on it.
Community files
- README: present
- Contributing guide: present
- Code of conduct: present
Watch-outs
- License not identified by GitHub; read the LICENSE file before depending on it.
General guidance from repository metadata, not legal advice. How we score
From the README
Pathway is a Python ETL framework for stream processing, real-time analytics, LLM pipelines, and RAG.
Pathway comes with an easy-to-use Python API, allowing you to seamlessly integrate your favorite Python ML libraries. Pathway code is versatile and robust: you can use it in both development and production environments, handling both batch and streaming data effectively. The same code can be used for local development, CI/CD tests, running batch jobs, handling stream replays, and processing data streams.
Pathway is powered by a scalable Rust engine based on Differential Dataflow and performs incremental computation. Your Pathway code, despite being written in Python, is run by the Rust engine, enabling multithreading, multiprocessing, and distributed computations. All the pipeline is kept in memory and can be easily deployed with Docker and Kubernetes.
For any questions, you will find the community and team behind the project on Discord.
Ready to see what Pathway can do?
Available in both notebook and docker formats, these ready-to-launch examples can be launched in just a few clicks. Pick one and start your hands-on experience with Pathway today!
Daily snapshots
Stars gained per day, and how forks and open issues moved, from GStars' daily snapshots
Forks
1.7K+26
Open issues
37+2
Compared with peers in Data Engineering
Fastest-growing repositories GStars tracks in this cohort for the week of Nov 8, by 7-day star growth
| Repository | Stars | +7d | Health | Last release |
|---|---|---|---|---|
| pathwaycom/pathway | 62.0K | — | 60 | 95 days ago |
| vectordotdev/vector | 22.8K | — | 64 | 11 days ago |
| ThreeDotsLabs/watermill | 10.0K | — | 56 | 76 days ago |
| OpenDCAI/DataFlow | 8.5K | — | 54 | 10 days ago |
| Eventual-Inc/Daft | 5.8K | — | 65 | 3 days ago |
| ruc-datalab/DeepAnalyze | 4.8K | — | 15 | — |
| Category median | 8.5K | — | 56 | — |
Also by pathwaycom
Other repositories from this maintainer that GStars tracks
Discover Repositories
Search across tracked repositories by name or description