Benchmark Results

We ran Anthropic Claude Haiku 4.5 (fast, no reasoning) on 20 questions from each dataset as a difficulty probe. The results establish a natural ordering from hardest to easiest, used throughout this report.

Full difficulty ladder

GPQA Diamond at 25% = random chance on 4-choice questions. These are graduate-level science questions designed to stump PhD students outside their field.
MMLU-Pro has 10 answer choices (random = 10%), making it much harder than standard MMLU (4-choice, random = 25%).
* MMLU time of 0.3s = cache hit from prior run.

We focus on the 3 hardest benchmarks

The blue-highlighted rows above — GPQA Diamond, MMLU-Pro, and MATH — are the only datasets that meaningfully separate frontier models. The grayed-out benchmarks below the line see most models scoring 80–95%, leaving little room to differentiate. The rest of this report concentrates on the hard trio, where reasoning capabilities, thinking budgets, and model architecture actually matter.

Our most comprehensive comparison: 100 questions, all major frontier models. MMLU-Pro uses 10 answer choices (vs MMLU's 4), making random guessing worth only 10%.

No timeouts in this run (all models responded within limit). "No Extract" = our regex couldn't find a letter answer in the model's verbose response — a parsing failure, not a model failure. Opus (12) and Grok (14) are most affected; with improved extraction most of these would be correct. See Error Breakdown for details.

Tier analysis

Flagship/Reasoning tier (OpenAI GPT-5.2 Pro, Google Gemini 3 Pro, Anthropic Claude Opus 4.6, xAI Grok 4.1 Fast Reasoning): all use thinking/reasoning tokens. Scores cluster between 78–86%. Claude Opus is 5x faster than GPT-5.2 Pro at only 6pp lower accuracy.

Fast/Cheap tier (Google Gemini 3 Flash, Anthropic Claude Haiku 4.5): much faster, 10–15pp below flagships. OpenAI GPT-5.2 non-Pro (55%) is anomalously weak — may not be activating reasoning despite effort: high.

These are not fully maxed configs. reasoning: high = ~4096 thinking tokens. Opus uses 16K thinking budget (could go 32K+). GPT-5.2 Pro at xhigh is the closest to true max. Gemini scores are depressed by answer extraction failures. See Model Configurations and Error Breakdown for details.

Simpler 4-choice benchmark. Less discriminating than MMLU-Pro but useful as a cross-check.

Random baseline at 20% confirms 4-choice expected value (25% theoretical, 20% observed at n=100). Wall times near 0 = cached responses from earlier runs.

MiniMax-M2.5 always uses internal chain-of-thought reasoning. Much slower, but significantly more accurate on hard questions — especially GPQA, where it jumps from random-chance to 93% adjusted accuracy.

Headline results on the hard trio

How timeouts are scored

Timed-out questions count as incorrect in the "Accuracy" column. "Adj. Accuracy" excludes timeouts entirely: correct / answered. For example, GPQA shows 14 correct out of 20 total (70% raw) but 5 timed out — so 14/15 answered = 93.3% adjusted. This reveals the model's capability when it actually responds in time. The 120s timeout was too aggressive for MiniMax on GPQA/MATH; a 600s timeout would likely eliminate most timeouts.

Both tested on the same 20-question samples (seed 42). Fast non-reasoning model vs always-thinking model.

Key takeaways

Thinking models earn their keep on hard problems: MiniMax gains +45pp on GPQA and +35pp on MMLU-Pro over Haiku. On easy benchmarks (Winogrande, TruthfulQA), thinking provides no measurable lift — both models saturate at 95%.

The HumanEval surprise: MiniMax (55%) is much worse than Haiku (85%) on code generation. Extended thinking doesn't help with code, and MiniMax may lack code-focused training data.

MiniMax is 10–30x slower due to internal reasoning. The 120s timeout was too low for GPQA/MATH — with 600s+ timeout, MiniMax's GPQA adjusted 93.3% would likely become raw accuracy too.

Not all errors are created equal. A model can lose points because it chose the wrong answer, because our regex couldn't extract a letter from a verbose response, or because the response was cut off.

What the error types mean

Wrong Answer

Model responded with a letter, but chose the wrong one. This is a genuine model error.

No Answer Extracted

Model gave an explanation but our regex couldn't find a letter. This is a parsing failure, not a model failure. Improved extraction recovers ~80% of these as correct.

Truncated*

Gemini models give long explanations that get cut off before stating the answer. A config/extraction issue, not a model capability issue.

Opus and Grok have 12–14 extraction failures each. With improved extraction (LLM fallback when regex fails), both would likely reach ~89–90% true accuracy. This suggests the real leaderboard is tighter than raw scores suggest: GPT-5.2 Pro 86%, Opus ~90%, Gemini Pro ~87%, Grok ~89% — all within a few points of each other.

Config caveat: These are not fully maxed configs. reasoning: high = ~4096 thinking tokens. Opus thinking: 16K is better but not max. GPT-5.2 Pro at xhigh is closest to true max. A "truly maxed" run would use thinking: 32K+ for Opus and longer timeouts for MiniMax.

Earlier ensemble/MoE experiments comparing single model vs multi-model voting and synthesis strategies.

Runner

python -m benchmarks.eval.run — custom runner using lib.llm.call_llm() via litellm. Supports all major providers (Anthropic, OpenAI, Google, xAI, MiniMax).

Sampling

Fixed seed (42), temperature 0. Smoke tests use 20 questions; full runs use 100. All models see the exact same questions for a given benchmark+sample size.

Formats

• MCQ (GPQA, MMLU-Pro, MMLU, ARC, HellaSwag, TruthfulQA, Winogrande): Cascading regex extraction for letter answers (6 patterns, ~15% failure rate on verbose models)
• MATH: \boxed{} extraction from model response
• HumanEval: Code completion + subprocess execution with test cases

Timeouts

120s per question with 1 retry. Code execution: 30s for HumanEval (prevents infinite loops). The 120s question timeout is more aggressive than standard — most eval frameworks use no timeout. MiniMax needs 600s+ for hard questions.

Answer extraction comparison

TODO: Add cheap LLM extraction fallback when regex fails (~$0.001/question using Haiku).