Jagged Intelligence

Large language models can write essays, solve complex math problems, and generate code at impressive levels. Yet they frequently fail at surprisingly simple variations of the same tasks.

Melanie Mitchell describes this uneven performance as “jagged intelligence” — a landscape where models excel in certain areas while remaining brittle and unpredictable in closely related ones. Slight rewording of a problem, added irrelevant details, or minor changes in context can cause dramatic drops in performance, even on tasks where the model previously succeeded.

This jaggedness reveals a deeper issue: current LLMs are powerful pattern matchers trained on vast text, but they lack robust understanding of the world, causal reasoning, and the ability to generalize reliably. Benchmarks often overstate capabilities due to data contamination and the models’ sensitivity to superficial changes.

The article challenges both hype around near-human or superintelligent AI and overly dismissive views, arguing that we need better ways to evaluate what these systems actually know — and don’t know — before deploying them in high-stakes domains.

This Ledger Entry expands how readers think about AI capabilities by introducing the concept of "jagged intelligence" — the unpredictable and brittle nature of LLM performance — and highlighting why current benchmarks fail to capture the fundamental gaps in reasoning and world understanding that still exist in even the most advanced models.