Runtimes effective-speed test
Five configurations that reached the model's native maximum context (262,144 tokens). Each panel is one metric; each line is one configuration. Hover a chart to read the per-context values.
What differs across the five configurations: runtime, checkpoint/quant,
prefix-cache mechanism and speculation engine. Sampling is identical for all (vendor):
temperature 1.0, top-p 0.95, top-k 20,
reasoning-effort xhigh. Model native context: 262,144 tokens.
All three runtimes do lossless speculative decoding and prefix caching on Apple Silicon — the target verifies every token, so output matches plain decoding. They differ in where the draft comes from and how the KV cache is tiered.
Fraction of the prefix reused per test scenario. cold = first
pass (seeding, ≈0 by design). middle_mutation = divergent prefix: MTPLX re-prefills
(~0), oMLX reuses the unchanged half (~0.49). tool_turn at 128K on MTPLX 4b/8b read
0.00 from an under-provisioned session-bank cap (48G); the confirmatory run at 100G
recovered 0.99.
* Estimated, not re-run to save time.
At 128K the sweep used a 48G session-bank cap, so MTPLX append/tool_turn
re-prefilled (measured 0.00) purely from that under-provisioning; the * values
are the reuse under an adequate 100G cap — MTPLX 4-bit tool_turn was confirmed at 0.99, the
rest inferred from it and the 256K/100G runs. At 256K, MTPLX 8-bit skipped
middle_mutation and tool_turn, following the measured MTPLX 4-bit
(0.008 / 0.996). middle_mutation stays ~0 by design (MTPLX re-prefills on divergence),
so it is not corrected.
Single source: results/cache-probe.jsonl. Scenario cold = cold
cache (full prefill); cached = the identical scenario (warm reuse, hit ≥0.97).
Effective tps = output tokens ÷ total time.