confident-ai/deepeval
The LLM Evaluation Framework
Build with Backblaze B2
SDKs, agent skills, IDE extensions, and reference pipelines from Backblaze Labs. All open source.
Loading star history...
Where it stands
Rank, growth pace and release cadence relative to everything GStars tracks.
5 releases in the last 90 days, the latest 20 days ago, with 319 contributors on GitHub.
Peaked at #39 in the weekly ranking the week of Sep 28, its best in the past year.
Use Cases & Benefits
- Provides a Python framework to evaluate large language model outputs using diverse metrics and unit testing approaches.
- Enables local, modular, and customizable LLM evaluation with integration to any CI/CD environment and support for component-level testing.
- Use for benchmarking and improving RAG pipelines, chatbots, or AI agents by testing model outputs against human-like accuracy metrics.
- Use for red-teaming LLM applications to detect safety vulnerabilities like toxicity, bias, and prompt injections with advanced attack strategies.
- Use for integrating LLM evaluation into fine-tuning workflows with real-time metrics and dataset curation on the DeepEval cloud platform.
Should you depend on it?
License terms, project hygiene and risks, from GitHub metadata and GStars' snapshots.
License
Apache 2.0
Permissive: commercial use, modification and redistribution are allowed with attribution; includes an explicit patent grant from contributors.
Community files
- README: present
- Contributing guide: present
Watch-outs
No watch-outs from GStars' checks.
General guidance from repository metadata, not legal advice. How we score
From the README
DeepEval is a simple-to-use, open-source LLM evaluation framework, for evaluating and testing large-language model systems. It is similar to Pytest but specialized for unit testing LLM outputs. DeepEval incorporates the latest research to evaluate LLM outputs based on metrics such as G-Eval, hallucination, answer relevancy, RAGAS, etc., which uses LLMs and various other NLP models that runs locally on your machine for evaluation.
Whether your LLM applications are RAG pipelines, chatbots, AI agents, implemented via LangChain or LlamaIndex, DeepEval has you covered. With it, you can easily determine the optimal models, prompts, and architecture to improve your RAG pipeline, agentic workflows, prevent prompt drifting, or even transition from OpenAI to hosting your own Deepseek R1 with confidence.
Want to talk LLM evaluation, need help picking metrics, or just to say hi? Come join our discord.
🥳 You can now share DeepEval's test results on the cloud directly on Confident AI's infrastructure
Daily snapshots
Stars gained per day, and how forks and open issues moved, from GStars' daily snapshots
Forks
2.2K+404
Open issues
807+379
Compared with peers in LLM
Fastest-growing repositories GStars tracks in this cohort for the week of Nov 1, by 7-day star growth
| Repository | Stars | +7d | Health | Last release |
|---|---|---|---|---|
| thedotmack/claude-mem | 113.6K | +8.8K | 35 | today |
| NVIDIA/auto-ontology | 7.5K | +5.6K | 37 | — |
| HKUDS/DeepTutor | 55.5K | +5.5K | 65 | 5 days ago |
| affaan-m/ECC | 297.9K | +5.2K | 63 | 35 days ago |
| DietrichGebert/ponytail | 185.3K | +5.1K | 60 | 7 days ago |
| confident-ai/deepeval | 19.3K | — | 59 | 20 days ago |
| Category median | 113.6K | +5.5K | 60 | — |
Also by confident-ai
Other repositories from this maintainer that GStars tracks
Discover Repositories
Search across tracked repositories by name or description