Local LLM Benchmarks

Models benchmarked

All MLX safetensors, run via LM Studio. Phase 1 (Qwen) verified 2026-05-19; Phase 2 (Gemma 4) verified 2026-05-22 (all 7 models, n=100 across HumanEval/MMLU/MATH/DROP/GPQA + n=50 LCB). The two right-most columns capture qualitative strengths and how each model is meant to be used in rotation. Model names link to the HuggingFace quant repo; Peak RAM is the loaded footprint from speed-probe telemetry (— where no qualifying probe — see Resource usage below); the rest is on-disk metadata.

Benchmark scoreboard

All accuracy benches (n=100), tool-calling suites, and creative-writing throughput in one matrix. Click any column header to sort ascending/descending; the best score in each numeric column is highlighted green. Hover a row to highlight it. Throughput columns are the creative-writing scenario (the most representative single number for "feels like" decode speed). The filter bar above hides models from every chart and this table at once.

Defaults: rows are in the same order as the Models table above. Click headers to re-rank. Accuracy = pass-rate %, tool-calling = score %, throughput = tok/s on the creative-writing scenario. T-Bench is on-rig Terminal-Bench 2.0 n=89 (terminus-2, --agent-timeout-multiplier 0.5, 2026-05-24 → 2026-05-29; qwen3.5-122b-a10b backfilled 2026-07-12) — all candidates measured. Phase-5 additions MiniMax-M2.5 Q3_K_S and UD-IQ2_M both appear in the charts and tie exactly on T-Bench at 25.8% (23/89).

Resource usage CPU · GPU · memory · power, from speed-probe telemetry

Per-model resource draw captured by macmon during the speed-probe runs (local models only — accuracy/throughput benchmarks and frontier models are not instrumented). Peak RAM is peak unified-memory footprint (weights + KV cache + runtime); Avg CPU/GPU are the mean per-sample utilisation; Peak power is the package total. Only runs with n ≥ 10 telemetry samples are shown — noisier runs are gated out and appear as — (qwen3-coder-next n=2, gemma-4-31b and gemma-4-e4b have no qualifying probe). The filter bar above hides rows here too.

Source: tools/local-llm-bench-m4-32gb/results/speed_probe/*_results.json (peak RAM / avg GPU / peak power / samples) + *_macmon.jsonl (per-sample cpu_usage_pct for Avg CPU). Rig: Apple M4 Max, 128 GB unified. * Gemma probe filename did not record the quant; attributed to @6bit (the primary Gemma slot).

Quality vs speed

Benchmark score against generation throughput — the trade-off the bar charts can't show. Bubble size ∝ on-disk footprint (GB). Pick the y-axis benchmark; "Composite" averages the six accuracy suites (missing suites are skipped and flagged in the tooltip). agents-a1-xl-mlx has no creative-writing throughput sample, so it has no bubble.

LiveCodeBench v6 — local (n=50)

All 7 daily-driver candidates fully characterized after Phase 1 backfill + Step B reruns (2026-05-24), regraded 2026-10-14 with the official LiveCodeBench import prelude (the earlier runner scored 6–28 pp low). Gemma 4 31B leads at 90 %; Gemma 4 26B-A4B @6bit and qwen3.6-27b tie at 88 %. Numbers are post-rerun MERGED canonical: qwen/qwen3-coder-next 84 % clean (0 trunc), qwen3.6-27b 88 % (1 trunc, Q3 abc365_c — hardest across the trio), qwen3.6-35b-a3b@6bit 82 % (6 trunc, all 65 k spirals), gemma-4-26b-a4b@4bit 76 % (8 trunc — Q2 recovered at 65 k, rest are real model limits), gemma-4-26b-a4b@6bit 88 % (1 trunc — Q19 still spirals at 65 k). Phase-5 MiniMax-M2.5 also plotted: Q3_K_S 78% @60k; UD-IQ2_M 76% @32k raw and qwen3.5-122b 62% are lower bounds (no saved answers to regrade).

LiveCodeBench v6

Coding pass@1 against the rolling contamination-resistant set. Anthropic Opus 4.x, Gemini 3 Pro, GPT-5.2, o-series and Llama 4 don't publish LCB; bars are limited to providers who do. Sonnet 4.5 is reported as LCB-Pro, not directly comparable to v6 pass@1 — kept with caveat.

Local vs frontier — published scores

Local runs (this rig, n=100, seed=42, temp=0) plotted against published first-party scores for frontier & notable open-weight models. Caveats: many newer frontier models no longer publish HumanEval/DROP; MATH often means MATH-500 or AIME 2025 (noted per-model); GPQA is Diamond throughout. Reasoning models (o3, R1, Claude 4.x with thinking, Gemini 2.5/3, GPT-5) report with test-time compute. Use for ranking, not absolute deltas — ±5pp noise at n=100. Tier pills in the filter bar toggle the frontier / open-weight reference rows.

Accuracy — 100-question benchmarks

Pass rate across reasoning/coding/knowledge suites. Phase 1 (Qwen ×3) complete; Phase 2 (Gemma 4) — 26B-A4B@4bit/@6bit complete, 31B-dense partial (MATH/DROP/GPQA still running). GPQA caveats: Qwen 27b had 15/100 and 35b-a3b had 23/100 hard-capped at max_tokens=32 768 (counted as wrong); true GPQA likely 78–85 % / 75–83 %. Gemma 4 runs used the doubled max_tokens=65 536 rule and truncated only 1–2 questions — those scores are not affected.

GPQA Diamond

Graduate-level science. The one benchmark still universally reported.

Model profile — radar

Head-to-head shape across every 0–100 benchmark axis. Pick up to 3 models. Axes are fixed 0–100 (all are percentages) so shapes stay comparable regardless of selection. Missing axes (e.g. agents-a1-xl-mlx has no MATH/DROP/GPQA/T-Bench) show as gaps.

Terminal-Bench 2.0 — local (n=89)

Multi-turn shell-agent benchmark, measured on-rig 2026-05-24 → 2026-05-29 via Harbor + LiteLLM → LM Studio (Docker linux/amd64), terminus-2 agent, concurrency=1, vanilla T-Bench 2.0 task set with --agent-timeout-multiplier 0.5 (bounds the 14 tasks declaring >60-min agent budgets — scores are a defensible floor; full-budget would land ≤5 pp higher). All 7 candidates complete: qwen/qwen3-coder-next 32.6 % (29/89, 16.8 h), qwen3.6-27b 31.5 % (28/89, 18.9 h), qwen3.6-35b-a3b@6bit 28.1 % (25/89, 15.6 h), gemma-4-31b dense 22.5 % (20/89, 17.4 h), gemma-4-26b-a4b@6bit 21.3 % (19/89, 14.4 h), @4bit 20.2 % (18/89, 16.0 h), gemma-4-e4b 4.5 % (4/89, 6.4 h). Total wall-clock ~105 h. Backfill (2026-07-12): qwen3.5-122b-a10b (GGUF Q4_K_S, thinking ON) 24.7 % (22/89, ~18.5 h) — #5, a NO-GO Planning challenger that ties the 27B one-shot but is a weaker agent (the older Qwen3.5 gen losing to the newer, smaller Qwen3.6). Headline: T-Bench cleanly inverts LCB at the top — the three Qwen3.6 models cluster on top (trained with explicit agentic-loop targets); Gemma — which tops LCB (31B 90 %, 26B @6bit 88 %) — drops to mid-pack. Canonical summaries: tools/local-llm-bench-m4-32gb/benchmarks/runs/tbench_*_summary.json; raw run dirs at bench/terminal-bench/logs/tbench-runs/<model>/result.json. Phase-5 MiniMax-M2.5 lands at 25.8% (23/89) on both quants — Q3_K_S (98.69 GB) and UD-IQ2_M (78.20 GB) tie exactly, 4th overall, above all Gemma. The 2-bit UD-IQ2_M matched the larger Q3_K_S on the agentic-shell loop measured here (remote model on the rig, MBP-side Docker, --agent-timeout-multiplier 0.5; the IQ2_M run's initial 16 Docker env-start failures were recovered on a clean-Docker resume before scoring). Budget diagnostic: IQ2_M logged 46 AgentTimeoutError at 0.5× and 46/89 = 51.7% equals MiniMax's vendor T-Bench figure — but re-running timeout tasks at 1.0× and 1.5× budget recovers only ~25–33% of them (verified-recoverable ≈ 27/89 = 30.3%, budget-boosted, not cohort-comparable). The rest of the gap to vendor is structural — throughput (2-bit @ ~40 t/s over LAN), tool-call JSON-format instability, and capability misses — not a multiplier artifact; more budget mostly lets it finish and be wrong. Full analysis: bench/minimax-m2.5/iq2m-tbench-budget-diagnostic.md.

* MiniMax-M2.5 UD-IQ2_M — with more agent-time budget: at a full/1.5× --agent-timeout-multiplier the model recovers to ~30.3% (27/89) = 23 base passes + 4 verified timeout recoveries (of the 46 AgentTimeoutError at 0.5×, ~25–33% recover). The plotted bar is the apples-to-apples 0.5× value (25.8%) — the extended-budget figure gives IQ2_M more time than every other row, so it is a diagnostic, not a cohort-comparable score, and the rest of the gap to vendor's 51.7% is structural (throughput + 2-bit tool-call format instability). Detail: bench/minimax-m2.5/iq2m-tbench-budget-diagnostic.md.

Generation throughput by scenario

Average generation tok/s — pure decode speed, excludes prefill.

Effective throughput by scenario

End-to-end tok/s including prefill — what the user actually feels.

Effective throughput vs context length (prefill-test)

How fast the model still feels at growing context. The dense 27b collapses to ~2 t/s at 8.5k.

Elapsed time per benchmark

Wall-clock minutes per suite (n=100 except LiveCodeBench n=50). 27b GPQA includes 15 max-token spirals at ~29 min each — the dominant time sink of Phase 1. Gemma 4 31B dense runs at ~13 t/s vs Gemma 26B-A4B's ~100 t/s — the dense tax shows up as ~7× longer wall-clock per bench.