Qwen3.8 Prefix-Cache Campaign

Sustained decode

Metric

Same model per context decode tok/s · canonical

Measured arms

Mode

Runtime

Context

Class

Group

Search arm / model

canonical = vendor sampling (decision metric); greedy = diagnostic temp=0, dimmed, does not count for verdict and inflates decode. E2E does not compare across classes (code short vs audit long). Arms with cache off (W/X) have cold TTFT by design.

Runtime offerings what each one seeks, how, and the cost

Quantization offerings target of each quant and the trade-off it makes

Decode measured per runtime best decode per runtime · canonical; greedy dimmed

Decode measured per quant best decode per bits/weight · fewer bits tends to faster

What each test evaluates cache-probe (5 scenarios × 2 modes) + tool loop

Which configurations ran which tests scenario per arm · mode measured

■ canonical (temp=1, counts for verdict) · ■ greedy only (temp=0, diagnostic) · — not run. The tool loop column shows majority (passes/total). Not every arm runs all scenarios — depends on the gate.

Gate verdicts

Gate 9 — oQ8epass

T

T promoted (M4 production); U just control, no advantage >5%.

Gate 10 — MTPLXpass

V

Best decode 32K; tool loop 2/3 with corrected prompt.

Gate 11 — oQ4e+DFlashfail

X

DFlash2 +14-24% decode, but no 10% of total time (cache-off).

MTP gate — AWQ5pass

L

L beats K on total time; MTP acceptance ~0.8; tool loop 2/3.

Gate 6 — SpecPrefillfail

M/N

Cuts cold TTFT ~-55%, but zeroes the warm cache (hit 0%); warm TTFT 6-17x worse than L. None advances.

Gate 7 — ANEblocked

O

Blocked: 'Private ANE procedure-bank compiler is unavailable' in this omlx build. ANE kernels do not compile. External, not fixable in the campaign.

Gate 8 — DSpark/DFlash2pass

R/S

Speculation delivers: decode +90%/+154% and total time -57%/-70% vs Q; tool loop 2/3. But the peak (S 38 @32K) falls below MTPLX (44.5) and is 8-bit.

MTPLX Quality — Y/Z vs Speedcontrol

Y/Z

Speed ~23-31% faster in decode. Among Quality, Z(fp16) dominates Y(8-bit): same decode, tool loop 3/5 vs 1/5. Quality only for fidelity (not measured).

Ceiling 262K — decodepass

L/T/S vs V/Y

At the native maximum (262K) the MTPLX MTP collapses: decode V 6.1 / Y 8.7 tps. oMLX (L 15.4 / T 14.1) and dspark/DFlash2 (S 14.2) hold ~14-15 tps. Cause: the MTP verify re-reads the 262K KV per step; DFlash2 does not scale that way. Speed verdict at the ceiling: oMLX and dspark; MTPLX loses the advantage it had up to 128K.

Ceiling 262K — cachepass

L/T/S/V/Y

All reuse the prefix at the ceiling: identical ~1.0, append 0.95-1.0, tool_turn 0.99-1.0; middle_mutation partial (0.38-0.49). CONFIRMED by variable isolation: the MTPLX tool_turn that zeroed at 128K/cap 48G reuses 0.99 at 128K/cap 100G (same worst-case order) -> the miss was session-bank under-provisioning, not order. Rule: size the per-session cap for the context KV + tool footprint (48G is too little already at 128K; 100G suffices at 262K).

Campaign queue

8K smoke + 32K cache/MTP (greedy)done
oQ8e smoke T/U (Gate 9)done
oQ4e+DFlash W/X (Gate 11)done
MTPLX V (Gate 10)done
MTPLX Quality Y/Z vs Speed (8K + 32K)done
oMLX K + L canonical (MTP gate)done
SpecPrefill 16K (Gate 6)done
SpecPrefill 32K (Gate 6) — faileddone
ANE 16K/32K (Gate 7) — blocked (ANE compiler absent)blocked
mlx-dspark smoke + 8K/32K + tool loop (Gate 8) — passdone
cache 65Kdone
cache 128K (L/T/V/Y/S)done
native 262K (L/T/V/S full, Y partial)done
tool-loop survivorspending
confirmatory tool_turn V@128K/cap 100Gpending
re-run T/U at max_tokens=4096pending

Coverage matrix quant/model + runtime × what has been measured · cache, MTP, context

canonical = temp=1 (counts for verdict); greedy = temp=0 (diagnostic). — in context = planned arm, not yet run. MTP: ✓ native, draft via draft model, auto vendor default.

Qwen3.8-27B — prefix cache, MTP & runtime