Dense Qwen3.8-27B on an M4 Max: which server runs it fastest?

Glossary

TTFT

time to first token: seconds from sending a request until the first token comes back.

Cold vs warm

cold is the first request on an empty cache, so the server reads the whole prompt; warm is a later request that reuses the cached start of the conversation.

Prefill (prompt processing)

reading the prompt before the first token; here prompt tokens ÷ cold TTFT.

Decode (token generation)

how fast the reply comes out, in tokens per second; the median over the warm requests.

Prefix cache and hit rate

the server keeps the work done on the previous prompt; the hit rate is the share of the new prompt served from that cache.

Context 32K / 128K

the test conversation fills a 32K (32,768-token) or 128K (131,072-token) context window.

Draft model (DFlash2) and speculative decoding

a small draft model proposes several tokens and the 27B checks them in one pass; accepted tokens come almost free.

MTP

multi-token prediction: extra heads inside the model propose the next tokens, with no separate draft.

Thinking effort

the reasoning_effort value of the request (xhigh or medium); more effort means more reasoning tokens and lower draft acceptance, so lower decode.

How 11 candidate setups came down to Splash

Each step keeps the setups that can still be the fastest way to run the dense 27B. 5 setups never ran: they were cut by choice to save rig time. 3 ran the 8K smoke and were not taken further after it, so their smoke decode is all we have for them. The comparison with Flash-Next is in the TL;DR and the chart below.

Agent turn time at 32K and 128K

Each bar is one agent turn: time to the first token of a turn that carries a tool result, plus 512 generated tokens, median of 3 repetitions. Thinking effort is the reasoning_effort value sent with each request. More effort means more reasoning tokens and lower draft acceptance, so a lower decode speed. Only thinking effort xhigh shows by default; tick medium to add those rows. The Flash-Next rows are the reference we compare against (thinking effort xhigh, on mlx-serve 26.10.1).

Peak memory (GB) is the highest wired memory of the whole machine sampled during the run (model weights, KV cache and anything else the system pinned), not the server process alone. Prefill is prompt tokens of the cold request ÷ cold TTFT, read from the per-request results. The harness reads an acceptance rate only from mlx-serve (draft) and the Flash-Next references (MTP); the other cells are empty.

Quality and per-task time were not measured for the 27B.

Run it yourself

The exact Splash 1.3.1 launch of the fastest setup, rebuilt from the campaign launcher. KV cache: int8, the Splash 1.3.1 default (--kv-format, not passed). --idle-release off keeps the model loaded; --language-only skips the vision tower.

• Target: mlx-community/Qwen3.8-27B-8bit @ 815b83c0 (full revision 815b83c0df8ffd1d1b5244cf75fd6ef14fca9ef9).
• Draft: incoai/Qwen3.8-27B-DFlash2 @ dedf8df6 (full revision dedf8df68adfb1afeaf7b7480c0a0243108177b4), passed as a local directory.
• Sampling (the vendor settings, sent with every request): temperature 1.0, top_p 0.95, top_k 20.
• Thinking effort: send reasoning_effort at the top level of the request body; medium gave the shorter turns here.

# once: fill the Hugging Face cache (Splash wants a Hub ID for the target and --offline reads the cache)
hf download mlx-community/Qwen3.8-27B-8bit --revision 815b83c0df8ffd1d1b5244cf75fd6ef14fca9ef9
DRAFT_DIR="$HOME/models/Qwen3.8-27B-DFlash2"   # any absolute directory you choose
hf download incoai/Qwen3.8-27B-DFlash2 --revision dedf8df68adfb1afeaf7b7480c0a0243108177b4 --local-dir "$DRAFT_DIR"

# serve (Splash 1.3.1)
splash serve --model mlx-community/Qwen3.8-27B-8bit --host 127.0.0.1 --port 8585 \
  --max-context 131072 \
  --revision 815b83c0df8ffd1d1b5244cf75fd6ef14fca9ef9 \
  --draft-model "$DRAFT_DIR" \
  --idle-release off --language-only --no-webui --offline

curl -s http://localhost:8585/v1/chat/completions \
  -H 'Content-Type: application/json' \
  -d '{"model": "mlx-community/Qwen3.8-27B-8bit", "messages": [{"role": "user", "content": "..."}], "reasoning_effort": "medium", "temperature": 1.0, "top_p": 0.95, "top_k": 20}'

Hardware and software

• Apple M4 Max, 128 GB unified memory, one machine.
• macOS 26.6.2.
• 27B servers: mlx-dspark 0.20.3, Splash 1.3.1, mlx-serve 26.10.2. Flash-Next references: mlx-serve 26.10.1.
• mlx-dspark 0.20.3 + DFlash2 (baseline): mlx-community/Qwen3.8-27B-8bit @ 815b83c0
• Splash 1.3.1 + DFlash2: mlx-community/Qwen3.8-27B-8bit @ 815b83c0
• mlx-serve 26.10.2 + DFlash2: ddalcu/Qwen3.8-27B-MLX-Serve-8bit @ 7775059e
• 3 repetitions per setup, thinking effort and context; every number is the median.

Limitations

• One machine (M4 Max, 128 GB). Other Macs will differ.
• 3 repetitions per point; medians, no confidence intervals.
• The agent turn is a synthetic probe (a long document, a tool result, a reply), not a real task.
• Quality was not measured: a fast setup is not known to answer as well as the others.
• Splash prefills faster than the baseline (+44% at 32K, +33% at 128K, same weights). The vendor credits a GPU + Neural Engine (ANE) prefill split; our harness does not record whether the ANE ran, so that cause is the vendor's claim only.
• mlx-serve 26.10.2: the tool-turn TTFT in the thinking-turn probe was 94–104 s (one request per mode, resend and drop), against 10.8 s in the cache probe at the same 128K. We have no explanation.