Benchmarks

Benchmarks are the standardised tests we use to measure and compare LLMs — fixed tasks, fixed datasets, fixed scoring. They're how the field tracks progress, and how most people decide which model to use. But here's the thing nobody puts in the marketing deck: most of the famous benchmarks no longer work. MMLU, HumanEval, GSM8K — every frontier model clusters above 90%, so the scores have stopped meaning anything. Reading benchmarks well in 2026 is mostly about knowing which ones are still alive.

💡 In one line: Benchmarks are standardised tests for comparing LLMs — but most famous ones are saturated, so knowing how to read them matters more than the numbers.

What a Benchmark Is

A benchmark is a fixed dataset of tasks with a defined scoring rule, run under stated conditions. Every model takes the same test, so scores are comparable — in principle. It's the AI equivalent of a standardised exam.

Why They Matter

• Compare models on a common yardstick.
• Track progress over time across the field.
• Select a model for a job.
• Detect regressions when you change something.

The Two Diseases: Saturation & Contamination

Every static benchmark degrades over time through two mechanisms:

• Saturation — frontier models cluster near the ceiling, so the benchmark can no longer tell them apart. Once everyone scores 92–95%, the differences are noise plus prompt-format variance.
• Contamination — benchmark questions leak into training data (web crawls, GitHub mirrors, forum posts), so the model recalls the answer instead of reasoning to it. By 2026 this is the default assumption, not an edge case.

The Lifecycle (and Why Successors Keep Appearing)

MMLU launched in 2020 with frontier accuracy around 32%. By 2026, every frontier system is above 92% — and the dataset's own label errors cap the achievable ceiling near 95%. That's a benchmark that has finished its useful life.

The pattern repeats, so the community ships harder successors:

• MMLU → MMLU-Pro (10 choices, chain-of-thought pressure)
• ARC-AGI → ARC-AGI-2 (private holdout)
• HumanEval → LiveCodeBench / SWE-bench (real repositories)

Retired vs. Active (a 2026 Snapshot)

(Even GPQA Diamond is now approaching saturation at the frontier — the treadmill never stops.)

The Categories That Matter

• Knowledge — MMLU-Pro (broad factual breadth).
• Reasoning — GPQA Diamond (Google-proof PhD-level science), ARC-AGI-2 (abstract puzzles), HLE (expert questions across 100+ subjects), FrontierMath.
• Coding — SWE-bench Verified (real GitHub issues), LiveCodeBench, Aider Polyglot.
• Agentic — Terminal-Bench, GAIA, τ-bench, BFCL (tool/function calling).
• Long context — RULER.
• Multimodal — MMMU-Pro.
• Human preference — LMArena Elo (blind pairwise voting).

Reading a Benchmark Properly

A few rules that prevent expensive mistakes:

• Triangulate. Read a static academic eval and a human-preference arena and an agentic suite. Agreement across all three is the signal; a single number is close to meaningless.
• 2–3% apart is a tie. Models within that band are functionally indistinguishable on that metric.
• Check the conditions. Scaffolding, prompt format, and compute budget change scores — the same model can post different numbers under different harnesses.
• Beware selective reporting. Vendors publish the benchmarks they win.
• Match the benchmark to your job. A cholesterol test doesn't predict blood pressure; SWE-bench doesn't predict creative writing.

Whiteboard

Public Benchmarks vs. Your Evaluation

This is the part people skip. Public benchmarks measure general capability — not your task. A model that tops SWE-bench may be mediocre on your codebase; a model 3 points lower on MMLU may be better for your support bot.

So: use benchmarks to shortlist, then build your own eval set from real examples of your work, and measure what you actually care about. A hundred examples from your domain beat any public leaderboard for your decision.

(This is exactly the held-out test set from the tuning topic — the same discipline applies to model selection.)

Limitations

• Saturation and contamination (above).
• Narrow scope — one score, one narrow capability.
• Gaming — training toward the benchmark rather than the skill.
• Multiple-choice ≠ real work — recognising an answer isn't producing one.
• No cost, latency, or safety dimension in the score.

Best Practices

• Triangulate across benchmark types; treat 2–3% gaps as ties.
• Prefer active differentiators over saturated classics.
• Check the evaluation conditions before believing a number.
• Build your own eval set — it's the only one that measures your job.
• Re-check periodically: the landscape shifts every few months.

A Note on Currency

Benchmark standings change constantly as models ship, and benchmarks saturate within months now. The lists above are a mid-2026 snapshot — verify against live leaderboards (LMArena, the benchmark's own site, or vendor model cards) before making decisions.

Summary

• Benchmarks are fixed tests for comparing LLMs on a common yardstick.
• Most famous ones (MMLU, HumanEval, GSM8K) are saturated — scores above ~90% no longer differentiate.
• Contamination is now the default assumption for any public benchmark.
• Active differentiators: GPQA Diamond, SWE-bench Verified, HLE, ARC-AGI-2, BFCL, LMArena.
• Triangulate across types, treat 2–3% as a tie — and your own eval set beats any leaderboard. EOF echo created