Local LLM Quality vs Frontier

Local models in this comparison

On-disk identifiers & quantizations. Model names link to the HuggingFace quant repo. Peak RAM is the loaded footprint from speed-probe telemetry (— where no qualifying probe — see Resource usage below). The two right-most columns capture qualitative strengths and how each model is meant to be used in rotation — full quant deltas live in the caveats panel near the bottom.

All bars are published first-party scores from the upstream model card / announcement (or the most-cited third-party transcription, where the vendor doesn't publish). They are NOT measurements on the user's on-disk quant — see the quant-caveat panel at the bottom for expected quant-vs-base deltas. The filter bar above toggles models and tiers across every chart and the scoreboard at once.

Benchmark scoreboard

Every model on the page × every benchmark referenced, in one matrix. Click any column header to sort ascending/descending (double-click any header to reset). Best value per column is highlighted green; missing values sink to the bottom. Local rows use per-model swatch colors; frontier rows are purple-tinted, open-weight rows gray-tinted.

AIME years vary (Qwen3.6 / Gemma 4 → '26; GPT-5 / Sonnet 4.6 → '25; Kimi K2.6 → '26) — see the per-bench charts for footnotes. LCB row for gemma-4-26b-a4b is the on-rig @6bit measurement (88 %, regraded 2026-10-14); other Gemma-4-26b-a4b numbers are published BF16. All T-Bench local rows are on-rig n=89 (terminus-2, --agent-timeout-multiplier 0.5, 2026-05-24 → 2026-05-29): coder-next 32.6, qwen3.6-27b 31.5, qwen3.6-35b-a3b 28.1 (@6bit), qwen3.5-122b-a10b 24.7 (Q4_K_S, 2026-07-12 — NO-GO), gemma-4-26b-a4b 21.3 (@6bit), gemma-4-e4b 4.5. deepseek-v4-flash T-Bench value 56.9 is vendor-claimed (Phase 3 fit-test track — not measured here); other deepseek values are the base Flash model — the on-disk 2-bit-DQ quant will be lower (see caveats).

Resource usage CPU · GPU · memory · power, from speed-probe telemetry

Per-model resource draw captured by macmon during the speed-probe runs (local models only — the published quality scores above are not on-rig measurements, and frontier models are not instrumented). Peak RAM is peak unified-memory footprint (weights + KV cache + runtime); Avg CPU/GPU are mean per-sample utilisation; Peak power is the package total. Only runs with n ≥ 10 telemetry samples are shown — noisier runs are gated out and appear as — (qwen3-coder-next n=2, nemotron n=1, gemma-4-e4b has no qualifying probe). The filter bar hides rows here too.

Source: tools/local-llm-bench-m4-32gb/results/speed_probe/*_results.json (peak RAM / avg GPU / peak power / samples) + *_macmon.jsonl (per-sample cpu_usage_pct for Avg CPU). Rig: Apple M4 Max, 128 GB unified. * Gemma probe filename did not record the quant.

Local models — headline benchmarks side-by-side

MMLU-Pro / GPQA Diamond / SWE-bench Verified — the three benchmarks every relevant model reports. Bars are upstream-card scores.

MMMU / MMMU-Pro — vision & multimodal

MMMU and MMMU-Pro are different (Pro is harder). Bars are labelled (P) when Pro. Compare within the same metric only.

Terminal-Bench 2.0

Multi-turn shell agent. All 7 baseline local rows measured on-rig 2026-05-24 → 2026-05-29 via Harbor + LiteLLM → LM Studio (Docker linux/amd64, terminus-2 agent, concurrency=1, n=89 vanilla T-Bench 2.0 task set with --agent-timeout-multiplier 0.5 to bound the 14 tasks declaring >60-min agent budgets — scores are a defensible floor; full-budget would land ≤5 pp higher). Headline ranking inverts LCB: qwen3-coder-next 32.6 % (29/89, 16.8 h) leads, with qwen3.6-27b dense 31.5 % (28/89, 18.9 h) and qwen3.6-35b-a3b@6bit 28.1 % (25/89, 15.6 h) close behind — the three Qwen3.6 models cluster on top. Phase-5 MiniMax-M2.5 (remote on the rig, MBP-side Docker) lands 25.8 % (23/89) on both quants — Q3_K_S and UD-IQ2_M tie exactly, 4th overall, above every Gemma. qwen3.5-122b-a10b (GGUF Q4_K_S, thinking ON, 2026-07-12) is #5 at 24.7 % (22/89) — a NO-GO Planning-slot challenger: it ties the 27B one-shot (LCB exact tie) but is a weaker agent, the older Qwen3.5 generation losing to the smaller-but-newer Qwen3.6. Gemma slot: gemma-4-31b dense 22.5 %, gemma-4-26b-a4b@6bit 21.3 %, @4bit 20.2 % (quant-A/B within 1.1 pp — 4-bit penalty doesn't bite on agentic shell), gemma-4-e4b 4.5 % = 4B size floor. Budget diagnostic: IQ2_M logged 46 AgentTimeoutError at 0.5× (46/89 = 51.7% = MiniMax's vendor figure), but re-running timeout tasks at 1.0×/1.5× budget recovers only ~25–33% (verified-recoverable ≈ 30.3%, budget-boosted); the rest of the gap is structural — throughput (2-bit @ ~40 t/s), tool-call JSON-format instability, capability misses — not a multiplier artifact (bench/minimax-m2.5/iq2m-tbench-budget-diagnostic.md). GPT-5.5 (post-this report) reports 82.7 — original GPT-5 has no published number.

* MiniMax-M2.5 UD-IQ2_M — with more agent-time budget: at full/1.5× --agent-timeout-multiplier it recovers to ~30.3% (27/89) (23 base + 4 verified timeout recoveries; ~25–33% of the 46 timeouts recover). Chart plots the 0.5× cohort value 25.8%; the extended-budget figure gives it more time than every other row (diagnostic, not comparable), and the rest of the gap to vendor 51.7% is structural — throughput + 2-bit tool-call format instability. See bench/minimax-m2.5/iq2m-tbench-budget-diagnostic.md.

Model profile — radar

Head-to-head shape across every 0–100 benchmark axis. Pick up to 3 models (local or frontier). Axes are fixed 0–100 (all are percentages) so shapes stay comparable regardless of selection. Missing axes show as gaps.

Quantization caveats — how the on-disk quant compares to the published base

All charts above are base-weight numbers. These are the expected deltas for the user's actual on-disk quants, per community measurements.

• qwen3-coder-next (MLX 6-bit): essentially lossless. Awni Hannun's Qwen3 perplexity sweep + Unsloth's Q4_K_M analysis show ≥99% retention of BF16 SWE-bench/HumanEval at Q4-class; 6-bit lands within ≤1pt of base.
• qwen3.6-27b (MLX 6-bit): tracks card numbers within ~0.5pt. Mixed-precision oQ6 vs RTN-6bit shows ~13% lower mean KLD but both are excellent.
• qwen3.6-35b-a3b (MLX 6-bit): ≤1pt loss on static evals. But documented MoE-MLX issue: multi-turn tool-calling degrades on mlx-community 4-bit/8-bit Qwen3.x-A3B checkpoints (not yet seen on 6-bit; tool-heavy workloads may be more fragile than static evals suggest).
• ⚠ gemma-4-26b-a4b (MLX 4-bit): the weakest quant on disk. Uniform MLX 4-bit RTN runs ~4.7× worse perplexity than llama.cpp K-quants for Gemma 4. Expect 1–3pt loss on MMLU-Pro/GPQA, 3–5pt on hard math/code. Recommendation: switch to GGUF Q4_K_M or MLX-DWQ-4bit. On-rig measurement (2026-05-21, n=100): the 4-bit vs 6-bit A/B on this rig confirms the penalty — MATH 80 vs 83, GPQA 47 vs 53 (+6pt), DROP unchanged at 79, HumanEval 98 vs 97. The 6-bit lift on GPQA is the biggest single quant-driven score swing observed in Phase 2.
• gemma-4-26b-a4b (MLX 6-bit): the recommended quant for this MoE on this rig — sits ~1pt under the BF16 published numbers on most evals, beats 4-bit on every reasoning bench, and still runs at ~85 t/s generation. Tool-calling is essentially lossless vs 4-bit (jdhodges 97.5/40, veerman 83.3/12 on both quants).
• gemma-4-31b (MLX 8-bit, dense): Phase 2 complete (2026-05-22, n=100). HumanEval 95, MMLU 77, MATH 79, DROP 85, GPQA 48, LiveCodeBench 76. The 8-bit dense **ties or loses** to 26B-A4B@6bit on every bench except DROP (+6pt). Decode 13.7 t/s vs @6bit's 80.8 — 6× wall-clock for indistinguishable quality. Verdict: demote / skip in normal rotation.
• nemotron-3-nano-omni (GGUF Q4_K_M): NVIDIA publishes BF16 → FP8 → NVFP4 deltas (mean of 9 multimodal benches: 65.80 → 65.40 → 65.43). Q4_K_M typically sits between FP8 and FP4 — expect ≤1pt loss on most non-ASR multimodal tasks; ASR (WER) essentially unchanged.
• gemma-4-e4b (MLX 8-bit, 4B): Phase 2 complete (2026-05-22, n=100). HumanEval 91, MMLU 65, MATH 14, DROP 65, GPQA 34, LiveCodeBench 68, jdhodges 88, Veerman 67. MATH collapses to 14 % — 4B is too small for hard symbolic math. Useful as a FIM / quick-call slot only; not a daily-driver fallback.
• ⚠⚠ deepseek-v4-flash-dq (MLX 2-bit DQ): heaviest asterisk. V4 Flash ships natively at FP4+FP8 — dropping experts to 2-bit is destructive recompression of an already-quantized model, not a clean post-train quant. The mlx-community card explicitly warns Q4_K_M is the floor for "most of the source quality". No public benchmark deltas exist. Expect 3–8pt drop on hard reasoning, more on long-context. Treat the chart numbers as a ceiling.
• ⚠️ MiniMax-M2.5 — MLX 3-bit NO-GO, but GGUF is GO: the MLX 3-bit build reproducibly kernel-panics the Mac Studio under sustained inference (Apple GPU driver IOGPUGroupMemory / AGXG16X, macOS 25D125) — three times, independent of parallelism, context, memory pressure, and KV-cache quantization. As predicted, a different runtime revisited it: the GGUF build runs, and UD-IQ2_M (78 GB) is now the preferred quant — it cleared the 2-bit gate (no collapse), lands LCB 76% (> Q3_K_S 72%), and sits under the 80 GB co-residency line. both GGUF quants measured T-Bench 25.8% (23/89) — Q3_K_S (98.69 GB) and UD-IQ2_M (78 GB) tie exactly — and now appear on the T-Bench chart. Still excluded from the frontier-schema radar bars — no MMLU-Pro/GPQA/AIME/SWE-V/MMMU scores exist for it. Full detail in tools/local-llm-bench-m4-32gb/results/M4_MAX_128GB_NOTES.md.

Verdict — where each local model lands vs frontier

One-line takeaway per model.