The best way to run Qwen3.8-Flash-Next on an M4 Max 128 GB

1 · The seven stacks

The first five rows run the base model; the last two run the Swift 1.5 fine-tune. Same test prompt, same sampling (temperature 1.0, top_p 0.95, top_k 20, reasoning effort xhigh, max 4,096 output tokens). All use speculative decoding and a prefix cache; they differ in where the cache lives and in how they handle the memory limit. Why these servers and when each ran: section 8.

2 · Agent turn time by context size

Agent turn time = the wait from sending a tool result to the end of a 512-token reply: TTFT (time to first token) + 512 ÷ decode speed. A coding agent pays this at every step. Lower is better. Pick a context size: one panel per metric, then the turn time across all sizes.

• Splash is almost flat: 6.8 s at 128K, 10.5 s at 1M with the base model; Swift 1.5 is faster again by 0.4 to 1.2 s per turn.
• mlx-serve is the fastest MLX server: 8.9 s at 128K, 14.5 s at 512K. It also has the fastest cold prompt, by up to 7%.
• oMLX is second among the MLX servers (12.1 s at 128K) and stops at 256K. MTPLX needs a flag to reach 128K and refuses 256K. ds4 serves 512K, but its turn grows to 61.9 s there.

Tokens per second is not the whole story. At 128K, MTPLX decodes almost as fast as mlx-serve (70 vs 74 tokens/s), but its turn takes 15.1 s against 8.9 s: its TTFT on the next turn is 7.8 s and its prefill runs at 274 tokens/s against 759. Compare servers on turn time and cache behavior.

3 · The cache decides the wait

In an agent session, every step sends the whole conversation again plus a small new part at the end. A server with a prefix cache keeps the already processed part and prefills only the new tokens. The wait is the tokens that miss the cache, divided by the prefill speed. No server prefilled faster than 832 tokens/s on this machine, so the scenario decides how many tokens miss. At 512K: 1K ≈ 1.5 s · 10K ≈ 15 s · 100K ≈ 2.5 min · 300K ≈ 7.5 min · 512K ≈ 13 min.

Splash is the exception. When the history changes halfway it reuses none of the prompt: 13 min at 512K with the base model, against 8 min on mlx-serve. Section 5 explains why.

A 95% cache hit is not enough at long context

All seven stacks hit the cache on the next turn. At 512K, ds4 hits 94.5% and prefills about 29K tokens again, so its next reply starts after 50 s. mlx-serve hits 99.8%, prefills about 1K tokens and starts after 4.1 s.

Cache hit by scenario

Share of the prompt the server took from the cache. First message is 0% by design. History changes halfway can reuse at most the part before the change; Splash reuses none. The two scenarios an agent repeats all day are user adds a message and next turn: above 90% is good.

4 · Context limits and memory on 128 GB

The native context is 262,144 tokens. 512K uses YaRN 2.0, a context extension that oMLX does not support. 1M ran only on Splash. A cell shows the agent turn time, or why that size did not run.

5 · Splash up to 1M, and Swift 1.5

Splash is Inco AI's inference engine on native Metal. It loads Flash-Next from a GGUF only: the base model as the unsloth UD-IQ4_XS, the Swift 1.5 fine-tune as the ukisai IQ4_XS. The tiles compare Swift 1.5 on Splash with the same fine-tune in MLX on mlx-serve 26.10.3, the stack I use every day. The gap belongs to the stack (runtime + weight format), not to the server alone.

• The gain comes from decode. Splash writes 107 tokens/s at 512K against 57 on mlx-serve. The cold prefill costs the same (731 and 732 tokens/s at 256K). At 512K, 5 of 13 replies of the base model reached the 4,096-token limit while still reasoning.
• 1M. Above 262K Splash turns YaRN on by itself and keeps the KV cache past about 860K tokens on its SSD: 9.3 s per agent turn, 2.6 s to start the next reply, 110 GB of peak memory. YaRN is static above 262K and applies to every prompt, so use the 512K or 1M setting only when the context needs it.
• A change halfway costs the whole prompt. Flash-Next is a hybrid model: some layers carry a recurrent state that cannot be rolled back, and Splash saves that state only where a request ends. An edit, a compaction, a retried turn or a fork reuses none of the prompt, against 44% on mlx-serve at 512K: 12 min at 512K and 25.5 min at 1M (splash #387, #361).

iQ-MLX and Swift 1.5 on the same mlx-serve build

Up to 128K the two quants tie on 26.10.3. Past 128K, Swift 1.5 is faster: 12.7 s against 14.5 s at 512K. The iQ pack lost decode speed from 26.10.1 to 26.10.3, while Swift 1.5 gained.

3 runs per cell, except iQ on 26.10.1 at 256K and Swift 1.5 at 256K: 1 run. 512K uses YaRN 2.0 and the 4-bit KV cache on both builds.

Swift 1.5 thinks less and scores the same

Swift 1.5 is a community fine-tune that thinks less before it answers. On mlx-serve it scored the same as the base quant within the noise of these samples and reached the final answer 2.9× faster on GPQA and 2.0× faster on LiveCodeBench. Its agent turn time is the same, because prefill and the cache do not change.

Vendor sampling, reasoning effort xhigh, one run per model. Base: ddalcu iQ-MLX 4.7 bpw. Swift 1.5: Dankpaws/Swift1.5-Qwen3.8-Flash-Next-MLX-4.7bpw. The two packs also have different n-gram tables, so the comparison is between the packs, not only the fine-tune.

6 · Run it yourself

These are the flags I measured. The data, the prompt generator and the scripts behind this page are in the repository.

Splash 1.4.0

Download unsloth/Qwen3.8-Flash-Next-GGUF (UD-IQ4_XS, 94 GB) or ukisai/Swift-1.5-Qwen3.8-Flash-Next-GGUF (IQ4_XS, 98 GB). This is the 1M setting; for 256K or 512K, set --max-context to 262144 or 524288. Splash picks the DFlash2 draft model and the 8-bit KV cache by itself.

splash serve --model unsloth/Qwen3.8-Flash-Next-GGUF:UD-IQ4_XS \
  --host 0.0.0.0 --port 8585 --max-context 1000000 \
  --revision 766911a6b7369840a91dbcd95f9f997acaab6cd6 \
  --idle-release off --language-only --no-webui --offline \
  --verify-rows auto

splash serve --model ukisai/Swift-1.5-Qwen3.8-Flash-Next-GGUF:IQ4_XS \
  --host 0.0.0.0 --port 8585 --max-context 1000000 \
  --revision 0142554c963a8653912c0c8283f05f13f63653f3 \
  --idle-release off --language-only --no-webui --offline \
  --verify-rows auto

mlx-serve 26.10.3

Download ddalcu/Qwen3.8-Flash-Next-MLX-Serve-iQ-MLX-4.7bpw (107 GB) and mlx-serve 26.10.3, which exposes OpenAI and Anthropic compatible endpoints. The prefix cache flags matter: with the default 2 GB cache, a 128K session read the whole prompt again on every turn. For 512K, use the 4-bit KV cache; with 8-bit, some long prompts did not fit in memory.

mlx-serve --model ~/models/Qwen3.8-Flash-Next-MLX-Serve-iQ-MLX-4.7bpw \
  --serve --host 127.0.0.1 --port 11234 \
  --ctx-size 262144 --mtp --ssm-checkpoint-max 16 \
  --prefix-cache-mem 16GB --prefix-cache-disk 100GB --prefix-cache-entries 64

mlx-serve --model ~/models/Qwen3.8-Flash-Next-MLX-Serve-iQ-MLX-4.7bpw \
  --serve --host 127.0.0.1 --port 11234 \
  --ctx-size 524288 --mtp --ssm-checkpoint-max 16 \
  --prefix-cache-mem 16GB --prefix-cache-disk 100GB --prefix-cache-entries 64 \
  --kv-quant 4 --kv-attn-mode fused \
  --config-overrides '{"text_config":{"rope_parameters":{"rope_type":"yarn","factor":2.0,"original_max_position_embeddings":262144},"max_position_embeddings":524288}}'

Tips for daily use

• Keep the start of the prompt stable. System prompt, tool list and project rules come first. A change there invalidates the cache for everything after it.
• Add to the end. A new message at the end costs seconds. A fork, a compaction or an edit of an earlier message costs minutes (section 3).
• Keep working in the same session. The context grows step by step and each step prefills only the new part. Only a session that starts large pays the cold prefill at once.
• Use the context you need. The turn grows with the context size, and 512K needs the 4-bit KV cache on mlx-serve. Leave memory headroom: both servers lock over 100 GB.

7 · Quality sanity check

Does a server or quant break the model? Not in these runs: the five stacks I checked are within a few cases of each other. One run per cell, on different dates, with a small sample. Read it as a sanity check, not as a ranking. ds4 and Splash with the base model were not checked.

Context sizes differ: the three base-model stacks ran at 128K, the two Swift 1.5 stacks at 262K (the 262K columns of the table below). HumanEval with the prompt's import lines added before the answer; on Splash it ran twice at 262K (158 and 159 of 164) and the bar shows the first run. LiveCodeBench v6: 50 questions (seed 42), graded with the official import lines. Instruction following: 20 prompts with format constraints, an IFEval-style probe. Tool calls A = jdhodges tool-call test; tool calls B = Veerman tool-call test. mlx-serve with the iQ pack: 2026-10-11 run on 26.10.1; oMLX and MTPLX (with --memory-limit max): 2026-10-09 runs, without instruction following. The oMLX LiveCodeBench bar comes from a 2026-10-15 run of the same pack on oMLX 0.7.1.dev1; MTPLX was not measured. Swift 1.5 stacks: 2026-10-25 runs.

Swift 1.5 on Splash, up to 1M

One run per cell, vendor sampling, reasoning effort xhigh; each column starts the server at that context size. HumanEval with the prompt's import lines, as in the chart; it ran twice at 262K and 1M on Splash, and the 1M result is within the run-to-run noise. Long-context recall: 8 questions about a document filled to the server's size.

8 · How I measured

Test prompt

A long audit document filled to the target size, with three facts hidden at 10%, 50% and 90% of the text. A reply is correct when it finds all three.

Scenarios

First message: the full prompt, no cache (cold prefill). Next turn: the same prompt plus a tool result at the end. User adds a message: the same prompt plus a new user message. History changes halfway: the prompt changes at its midpoint, so only the first half can be a cache hit.

Agent turn time

TTFT on the next turn + 512 ÷ median decode speed. It is the wait for one agent step with a 512-token reply.

Runs

One to three runs per point; the "Runs" column of the last table gives the number. A point with three runs shows the median.

Speculative decoding

The MLX servers and ds4 use the model's MTP head, a small extra head that drafts the next tokens; Splash uses a separate draft model (DFlash2). Output stays the same, decode gets faster.

YaRN

A method that stretches the model's position encoding beyond its native 262K tokens.

Harness

A script sends each scenario to the server's HTTP API and reads the timings from the stream. No coding agent sits in the loop.

Which servers, when

In October 2026 mlx-serve, oMLX, MTPLX and ds4 were the servers that ran Flash-Next on Apple Silicon with MTP and a prefix cache, each on its newest release (2026-10-07 to 2026-10-15). mlx-dspark 0.20.3 ran at 32K only and behind these four; llama.cpp, LM Studio and mlx-vlm were not measured. Splash 1.4.0 and Swift 1.5 on mlx-serve 26.10.3 ran from 2026-10-23 to 2026-10-25. On 2026-10-27 I ran the iQ pack again on 26.10.3, so both mlx-serve stacks on this page run the same build; the same-build table keeps 26.10.1 too.

Data and scripts

The raw results, the prompt generator and this page's script are in the repository (bench/engine-updates-2026-10, bench/swift15-flashnext-2026-10, bench/stack-updates-2026-10 and bench/splash-longctx-2026-10).

What this page does not measure

• Agent reliability over many turns. Speed, cache and memory only. Test your own agent workload before you choose.
• Other chips. Numbers from an M5 Max or an M3 Ultra do not transfer to this M4 Max.
• Power draw, fan noise, and several requests at once. Every run sent one request at a time.
• The quality of ds4 and of the base model on Splash (section 7).

Setup notes

• MTPLX. The default config plans only a 65,536-token window on this machine and refuses 128K; 128K needs --memory-limit max. Short requests without history skip its session cache, so the test sends a 64-token warm-up first.
• Splash. It refuses Flash-Next in MLX format, so it cannot run the same file as mlx-serve. Clients must send frequency_penalty 0, as with mlx-serve 26.10.3: with 1, the tool-call test failed on both servers.
• mlx-serve at 1M. With YaRN 3.815 and the 4-bit KV cache, mlx-serve 26.10.3 serves the first message (28.6 min), but the next requests fail with PrefillDoesNotFit: they need about 18 GB for the prefill, and 7 to 9 GB are free. 2 of 5 requests failed with Swift 1.5 and 3 of 5 with the iQ pack, so the page has no 1M turn time for mlx-serve.

All values

One row per server and context size. Times in seconds (TTFT = time to first token). Decode and prefill in tokens/s, memory in GB.