Responsiveness test by context · October 2026 stack

Qwen3.8-Flash-Next: five runtimes from 8K to 1M

The same model family on five runtimes, with the same fixture and the vendor sampling: mlx-serve 26.10.1 and 26.10.3, oMLX 0.7.0, MTPLX 2.12.2, ds4 and Splash 1.4.0. w1 and k2 run Swift 1.5 (a fine-tune of Flash-Next), g1 and g2 the GSQ-RCO-Coder (256 of 512 experts), the others the base model. Each panel is one metric; each line is one runtime and weight file. Hover over a chart to read the values by context.

Apple M4 Max · 128 GB · measured 2026-10-07 to 2026-10-25. mlx-serve 26.10.1 and 26.10.3 · oMLX 0.7.0 · MTPLX 2.12.2 · ds4 upstream 0aaea5a · Splash 1.4.0 and a Splash build from main 3135ab4.

Splash runs other weight files. Splash loads Flash-Next from a GGUF only, so k2 (Swift 1.5 IQ4_XS GGUF) against w1 (Swift 1.5 MLX 4.7 bpw on mlx-serve 26.10.3) mixes runtime and weight format. The g2 points come from an unofficial build from main; a release can change them.
Splash re-prefills a changed history. It keeps the recurrent state of the hybrid layers only where a request ends. A change in the middle of the prompt (middle_mutation) reuses nothing on Splash in every band, against ~45% on mlx-serve; at 1M that is ~25 min of prefill.
Speed, cache and memory only. This page does not measure agent reliability. A community test on an M4 Max 128 GB found differences between these runtimes in multi-turn agent tasks (zenn.dev). Test your own agent workload before you choose.
The chip changes the result. Public numbers from an M5 Max or an M3 Ultra do not transfer to this M4 Max. The prefill ratio between oMLX and mlx-serve changes with the chip (mlx-serve issue #658).
MTPLX needs a 64-token prime. MTPLX runs a request with max_tokens ≤ 48 and no history as a background task, without a session, so it does not keep the prefix. The probe primes MTPLX with a 64-token prime. MTPLX runs from the 1-token prime are left out of this page.

Rig

Setup by candidate

What changes between runtimes: runtime, quant, prefix cache mechanism and speculation. Sampling is the same for all of them (vendor): temperature 1.0, top_p 0.95, top_k 20, reasoning xhigh, max_tokens 4096. Fixture audit_retrieval with needles at 10/50/90%. Native context: 262,144 tokens; 512K uses YaRN 2.0 on mlx-serve and ds4. Splash turns on a static YaRN by itself above 262,144 tokens and serves 1M (1,000,000 tokens) with KV int8, the KV past its resident plan on an SSD cache.

Runtimes compared

All five runtimes do speculative decoding and prefix caching on Apple Silicon: MTP on mlx-serve, oMLX, MTPLX and ds4, a DFlash2 draft on Splash. They differ in where the KV cache lives (RAM, disk or SSD) and in how they handle the memory limit: mlx-serve pins it as wired memory, oMLX spills over to SSD, MTPLX refuses the prompt, ds4 keeps the prefix cache on disk and Splash moves the KV past its resident plan to an SSD cache.

Cache hit by scenario

Fraction of the prefix reused in each scenario, for the band's selected group. cold = first pass (≈0 by design). middle_mutation = the prefix diverges in the middle: Splash reuses none of it. append and tool_turn are the gate: ≥ 0.90.

01 (full reuse)

400, 500 or 507 = scenario refused with that HTTP status (400 = context window exceeded, 507 = MTPLX memory plan). A stream error (the server cut the response) also counts as refused. — = scenario outside the band's protocol: the 256K and 512K bands of some runtimes run only cold, identical and tool_turn.

All values

Source: bench/engine-updates-2026-10/results/reports-2026-10.json, generated from the *.jsonl files of the October campaigns (engine-updates, stack-updates and splash-longctx). Per series and band, the page shows the served group with the most reps (tie: the most recent). T_turn = tool_turn TTFT + 512 / warm decode. Warm decode = median of served identical, append and tool_turn.