Apple M4 Max · 128 GB unified memory · serving local models via mlx_lm.server + llama.cpp
Single-stream output tok/s on the M4 Max (warm), plotted against parameter count. Wall time for a 500-token task annotated.
35x spread. Qwen3-0.6B decodes at 161 tok/s; DeepSeek-R1-177B at 4.5 tok/s. The practical cliff sits at ~27B (17.5–24 tok/s) — above that, interactive work stalls behind thinking tokens.
Qwen3-Coder-30B at 128k: 75s vs 24s at 16k. DeepSeek at 1M: 510s to first token.
MMLU-Pro, GPQA Diamond, LiveCodeBench and τ²-bench across the full model ladder.
Only the 200B+ MoEs reach double-digit solve rates.
Telecom subset, 20 tasks × 3 seeds. Most local models score 0.
Same prompts with vs without thinking enabled, across 3 models × 3 runtimes × 2 temperatures (36 pairs of samples).
Thinking reduces hallucination by 53.6% (64.5% → 29.9% false claims). The gap only appears above 0.6B. On a single 4-bit quant the effect flips sign (-50.6%) — quantization quality matters as much as the thinking switch.
Accuracy by model with / without retrieval context.
Retrieval improves grounding on every model tested.
Pass rate with thinking on vs off, plus thinking-token cost.
Qwen3-30B: 71.4% → 92.9% (+21.4 pts) with thinking on.
More thinking tokens → higher accuracy, at a latency price.
Thinking is the single highest-leverage knob on this rig. The 27B at full budget beats DeepSeek R1-177B on the same multi-hop suite (85.7% vs 78.6%) while decoding 2.3x faster.
The same model driven by different coding harnesses: scores stay flat, latency does not.
HumanEval (blue) vs GSM8K (gold) — spread <10 pts despite very different harnesses.
| Comparison | Δ score | Verdict |
|---|---|---|
| Claude Code vs OpenCode | 9.4 pts | OpenCode wins (temp 0 vs retries) |
| opencode-misconfigured vs opencode | 0.0 pts | byte-identical results |
| Across runtimes (llama.cpp vs mlx) | 0.0 pts | same weights → same score |
| Latency same model, different harness | up to 17x | harness overhead dominates |
The model is the variable, the harness is not. Score differences measure the harness config, not the weights.
SVGBench questions run through coding harnesses, judged on the Chrome-rendered PNG (requirements met ÷ total). All 84 verdicts come from one judge, claude-opus-5 — which also drew 21 of the artifacts. Seven of the 105 upstream questions — not comparable to the upstream leaderboard.
Q7 — the dolphin through a hula hoop, drawn by five of the six pairs. Bars are pair means with their n; dashed line = mean over all 21 drawings.
The 43 artifacts carrying a verdict on both boards, split by who wrote the old one. Same artifacts on each pair of bars — only the judge changed.
| Pair | Drawings | Questions drawn | Mean |
|---|---|---|---|
| claude · claude-opus-5 † | 21 | Q0 · Q4 · Q5 · Q7 · Q13 | 0.693 |
| opencode · qwen3.8-flash-next | 25 | Q0 · Q4 · Q6 · Q7 · Q12 | 0.591 |
| opencode · qwen3.8-27b-8bit | 5 | Q0 · Q5 · Q7 · Q13 | 0.499 |
| opencode · gpt-5.6-terra | 28 | all seven | 0.336 |
† claude-opus-5 judged all 84 artifacts, its own 21 included. Question coverage differs per pair, so the means are not like-for-like; Q7 is the widest shared column.
| Pair | Drawings | Question drawn | Score on it |
|---|---|---|---|
| claude · fable-5-1 | 4 | Q0 — cow plowing | 0.729 |
| claude · claude-opus-4-8 | 1 | Q7 — the dolphin | 0.714 |
The re-judging changed the headline. On the old board flash-next led the one shared prompt. Under a single judge, claude-opus-5 leads Q7 0.857 to 0.786 (n=5 and n=10) and leads overall 0.693 to 0.591 (n=21 and n=25). The judge was moving the score: gpt-5.6-terra wrote all 28 verdicts on its own drawings and scored 0.421; the same 28 drawings score 0.336 under claude-opus-5, which drew none of them, and 15 of them fell. The other 12 old verdicts were written by claude-fable-5-1, which drew none of those 12; they moved +0.010. Read every figure with its n: claude-opus-5 judged its own 21 drawings, qwen3.8-27b-8bit is ranked on 5, and the two single-prompt pairs are shown per-prompt only. Detail, per requirement: the SVGBench dashboard.
Two more head-to-heads, plus a gallery that shows their artifacts next to the SVGBench drawings.
Five agent stacks build the same game from one brief — time to build, output tokens, source lines shipped, bundle size, and the defects each one caught. Open the dashboard.
One model, one brief, three design skills and a no-skill baseline — wall time and output tokens per run, five pages each, side by side. Open the dashboard.
Model output from three benchmarks on one page: 84 graded SVG drawings, the five built games, and the fifteen landing pages, each beside its numbers. Open the gallery.
Qwen3.5-27B at full thinking budget: 25 tok/s local, 85.7% multi-hop, 100% RAG grounding, hallucination down 47 pts vs no-thinking. Beats hosted DeepSeek R1-177B on accuracy and speed at zero per-token cost.