Awesome AI Evals

A curated collection of high-quality resources for evaluating AI systems.

Start Here

If you are new to AI evaluation, begin with Evaluation Fundamentals, then use the methodology and metrics sections to choose an approach that fits your system and evaluation goals. Each entry should explain what the resource is and why it is useful.

Table of Contents

• Evaluation Fundamentals
• Evaluation Methodology
• Metrics
• Datasets and Benchmarks
• LLM-as-a-Judge
• Human Evaluation
• Agent Evaluation
• RL Environments
• RAG Evaluation
• Safety and Red Teaming
• Evaluation Frameworks and Tools
• Papers and Surveys
• Courses, Talks, and Guides
• Contributing
• License

Evaluation Fundamentals

Resources that explain the goals, scope, and core concepts of evaluating AI systems.

Resources will be added here.

Evaluation Methodology

Resources on designing reliable evaluations, selecting test cases, comparing systems, and interpreting results.

• A Comprehensive Guide to LLM Evaluations — A vendor-authored overview of reference-based, programmatic, human, and model-based evaluation, with guidance on judge calibration and continuous regression testing. Article · Free to read

• AI Evals: Everything You Need to Know — A practical FAQ covering eval fundamentals, error analysis, evaluator design, human annotation, tooling, production use, and domain-specific workflows. FAQ · Free to read

Metrics

Definitions and practical guidance for measuring quality, reliability, efficiency, and other evaluation dimensions.

• LLM Evaluation Metrics: The Ultimate LLM Evaluation Guide — A vendor-authored guide to selecting metrics across programmatic, model-based, RAG, agent, and multi-turn evaluations, with discussions of G-Eval and metric limitations. Article · Free to read

Datasets and Benchmarks

Curated datasets and benchmark suites for testing specific capabilities or comparing model performance.

• 30 LLM Evaluation Benchmarks and How They Work — An overview of benchmark design and common LLM benchmarks, with links to datasets and papers; notes contamination, narrow coverage, and benchmark staleness as limitations. Guide · Free to read

• 250 LLM Benchmarks and Evaluation Datasets — A database of LLM benchmarks and public datasets tagged by capabilities such as reasoning, coding, safety, multimodality, and tool use; useful for discovery, but not a substitute for product-specific evaluations. Database · Free to read

• Artificial Analysis Evaluations — A public catalog of independent model evaluations and benchmark leaderboards spanning reasoning, coding, tool use, long context, multimodality, and agentic capabilities. Evaluation catalog · Free to read

• Google DeepMind Evals — A catalog of Google DeepMind benchmarks covering robotics safety, factuality, grounding, deep search, reasoning, and long-context tasks, with links to papers, datasets, and leaderboards. Benchmark catalog · Free to read

LLM-as-a-Judge

Research and tools for using language models as evaluators, including guidance on calibration, bias, and limitations.

Human Evaluation

Resources on rubric design, annotation workflows, evaluator training, quality control, and agreement.

Resources will be added here.

Agent Evaluation

Resources for evaluating systems that plan, use tools, interact with environments, or complete multi-step tasks.

• Agent Evaluation: A Detailed Guide — A long-form guide to agent evaluation covering system components, evaluation harnesses, graders, and benchmark case studies. Article · Free to read

• The Roadmap to Mastering AI Agent Evaluation — A practical roadmap covering reasoning and action layers, deterministic and model-based grading, non-determinism, and production monitoring. Article · Free to read

• Building Closed-Loop Evals for a Multimodal Agent at Scale — A talk by Uber engineers on designing evaluations for a multimodal food-enhancement agent, including practical pitfalls and lessons learned. Video · Free to watch

• Stanford CS329A: Agentic Evaluations and Long-Horizon Tasks — A Stanford lecture covering METR task horizons, GDPval, DeepScholar-Bench, and recurring long-horizon agent failure modes. Video · Free to watch

RL Environments

Resources on building, comparing, and scaling reinforcement learning environments used to train and evaluate language model agents.

• The Ultimate Guide to RL Environments: Building and Scaling Them in the LLM Era — A Hugging Face guide comparing environment frameworks, reward design, and scaling approaches for LLM training; it notes that framework APIs and ecosystem maturity are changing quickly. Guide · Free to read

RAG Evaluation

Resources for evaluating retrieval-augmented generation, including retrieval quality, grounding, attribution, and answer quality.

Resources will be added here.

Safety and Red Teaming

Resources for testing harmful behavior, misuse resistance, robustness, security, and other safety properties.

Resources will be added here.

Evaluation Frameworks and Tools

Maintained frameworks, libraries, and utilities that support repeatable AI evaluation workflows.

• DeepEval — An open-source Python framework for LLM application evaluation with unit-test-style assertions, ready-to-use metrics, component and end-to-end evaluations, and local execution. Framework · Apache-2.0

Papers and Surveys

Research papers and surveys that provide useful findings, methods, taxonomies, or evidence for AI evaluation.

• METR Research — A research index from the nonprofit METR covering autonomous capability evaluations, long-horizon tasks, AI R&D benchmarks, evaluation integrity, and frontier AI safety. Research index · Free to read

Courses, Talks, and Guides

Educational material that helps practitioners learn how to design, run, and interpret AI evaluations.

• Evals for AI Engineers — A forthcoming O'Reilly book page about systematic testing and improving AI application reliability; its table of contents is marked not yet final and full access may require a subscription. Book · Paid/Subscription

• AI Evals Advanced Masterclass in Under 57 Minutes — A masterclass with Daniel McKinnon on building agentic evals, including offline testing, task-based evaluation, domain expertise, and repeated sampling. Video · Free to watch

• Stanford CME 295: LLM Evaluation (Autumn 2025, Lecture 8) — A Stanford lecture covering human agreement, rule-based metrics, LLM judges and their biases, agent evaluation, and benchmark examples. Video · Free to watch

Contributing

Suggestions and improvements are welcome. See CONTRIBUTING.md for the submission format and review guidelines.

License

The curated list content is dedicated to the public domain under CC0 1.0.