Most LLM evaluations rely on generic benchmarks that often fail to predict performance in actual production agent systems. This experiment kept the entire agent infrastructure fixed (tools, prompts, retrieval, retry logic, and orchestration) and swapped only the underlying model across 15 LLMs.

The agent was tasked with turning natural language business questions into validated, executable queries against a large analytics store — a realistic, multi-step workflow involving planning, tool use, query generation, self-repair, and output validation.