Qwen3.8-Flash-Next · Apple M4 Max 128 GB · mlx-serve 26.10.1 · October 2026

Where the thinking-effort line goes decides what the prefix cache can reuse

The same effort line, first or last in the prompt, on two 4.7-bit MLX packs (Swift 1.5 and iQ-MLX): effort switches at 128K, short tool-calling prompts, obedience to the level, and time, tokens and accuracy at xhigh, medium and low.

Qwen3.8-Flash-Next takes its thinking effort from one line of the prompt. Put that line first (the official template) and changing the effort in a 128K session costs 167 s, because the whole prefix cache is lost. Put it last and the switch costs 0.8–5.8 s, the model still obeys it, and accuracy holds. But the same move breaks prefix reuse on short prompts: every tool-calling turn starts cold, +0.9 to +1.4 s. On a hybrid attention/SSM model, where a variable line sits decides where the server can save its state.

Effort switch at 128K

167 → 0.8 s

time to first token, xhigh → low · cache hit 0.00 → 1.00

Short prompts, prefix reuse

53 → 0

of 55 tool-calling requests reused the system prompt · +0.9 to +1.4 s each

Obedience

×0.90–×1.05

thinking tokens, tail against official, at the same effort · rule ×0.80–×1.25

Accuracy

≤ 3

questions apart per 50 on GPQA and LiveCodeBench, every effort, both packs

What we learned

Where the effort line sits is a cache decision, not a prompt-engineering one. The model obeys it in either position. The server's prefix cache does not: with a hybrid GDN/SSM model, mlx-serve can only resume at a saved state checkpoint, and the checkpoint goes at the start of the last message.
Tail template: good for long sessions, bad for short ones. Effort switch at 128K: 167 s → 0.8–5.8 s. Same-effort control turn at 128K: 1.3 s → 3.0 s. Short tool-calling turn: +0.9 to +1.4 s, prefix reuse 53/55 → 0/55.
Both packs behave the same way. Swift 1.5 and iQ-MLX give the same cache hits at every step of the probe (0.97–1.00 on switch, 0.98 on control) and pass the same accuracy rules.
iQ-MLX cuts more thinking per level than Swift 1.5. At low, iQ-MLX thinks ×0.26 of its xhigh; Swift 1.5 ×0.45. Swift 1.5 thinks less to begin with (GPQA median 1.7K vs 5.1K tokens at xhigh on the earlier page).
At xhigh, the tail template hits the token limit a little more often (Swift 4 → 7 in 2 rounds, iQ 7 → 9 in 1), and its thinking ratio leans to ×1.04–×1.05. The intervals include 1.0, so this is not confirmed.
Decision: the official template stays the default; the harness fixes the effort per session (xhigh for code and long sessions, medium for chat and short tool calling). The tail template is kept as an optional profile for long sessions that need to change effort mid-way. Five untested ways to keep both the cheap switch and the short-prompt cache are listed below.

The two templates

Qwen3.8-Flash-Next reads its thinking effort (reasoning_effort: low, medium or xhigh) from one line in the prompt. The official chat template writes that line first. The effort-tail template moves it to a short system block after the last message. Nothing else changes: at medium the two templates render the same prompt.

officialeffort: xhighsystemtoolsuser: questionassistant

effort tailsystemtoolsuser: questioneffort: xhighassistant

Dashed blocks are the part that stays equal between two requests of one session. ▲ checkpoint marks where the server saves the state it needs to restore a cached prefix; it sits at the start of the last message before assistant. This position was inferred from the server logs, not read from the mlx-serve code.

Switching the effort in a long session

Probe 3e: a 128K-token conversation at xhigh, then one turn at medium or low, then back to xhigh. With the official template the effort line is the first token of the prompt, so changing it invalidates the whole prefix cache and the server re-reads 126K tokens: 167 s before the first token. With the effort line at the tail, the history stays cached and the switch costs 0.8–5.8 s. Median of 2 rounds per switch; the iQ-MLX official template was not re-run here, since its prompt is identical to the Swift one.

Swift 1.5 · officialSwift 1.5 · effort tailiQ-MLX · effort tail

Switching the effort at 128K

Time to the first token of the turn that changes the effort. The prompt keeps the same 126K-token history.

xhigh → medium, time to first token

official167.1 s · hit 0.00

effort tail5.7 s · hit 0.97

effort tail5.8 s · hit 0.97

xhigh → low, time to first token

official166.9 s · hit 0.00

effort tail0.8 s · hit 1.00

All turns of the probe

Median over the medium and low runs. The control turn pays more with the tail template: the server re-reads the text after its last checkpoint, which now sits before the effort block.

Turn	Pack · template	TTFT, s	cache hit
control: same effort	Swift 1.5 · official	1.34	1.00
control: same effort	Swift 1.5 · effort tail	3.04	0.98
control: same effort	iQ-MLX · effort tail	3.04	0.98
switch: new effort	Swift 1.5 · official	166.95	0.00
switch: new effort	Swift 1.5 · effort tail	3.28	0.99
switch: new effort	iQ-MLX · effort tail	3.29	0.99
back to xhigh	Swift 1.5 · official	0.88	1.00
back to xhigh	Swift 1.5 · effort tail	0.31	1.00
back to xhigh	iQ-MLX · effort tail	0.31	1.00

Short prompts lose the prefix cache

Probe 3f, Swift 1.5: 55 tool-calling cases that share the system prompt and the tool list (600–1,200 tokens each). With the official template the server reused the shared prefix in 53 of 55 requests. With the effort line at the tail it reused it in 0 of 55: every request paid the full prefill, +0.9 to +1.4 s on the first turn. Accuracy did not change (36→37 of 40, 10→10 of 12). This regression is why the tail template did not become the default.

officialeffort tail correct wrong× hit the max_tokens limit median

Tool calling · jdhodges 40 cases, xhigh, 1 round

median 2.2 s → 3.6 s

One dot per case: time of the first turn (the one that writes the tool call). Prompts are 1,218–1,287 tokens, same system prompt and tools in every case.

Tool calling · Veerman 12 cases, xhigh, 1 round

median 3.1 s → 4.0 s

One dot per case: time of the first turn (the one that writes the tool call). Prompts are 607–642 tokens, same system prompt and tools in every case.

Why. Qwen3.8-Flash-Next is a hybrid model: its attention layers keep one entry per token, but its Gated DeltaNet (SSM) layers keep one state that summarizes everything read so far. To resume from a cached prefix the server needs a saved copy of that state, a checkpoint, inside the part of the prompt that stayed equal. The logs show one checkpoint at the start of the last message. With the official template that message is the user question, right after the shared prefix, so the checkpoint is inside it. With the tail template the last message is the effort block, after the question, so when the question changes no checkpoint falls inside the equal part and the server starts cold (hybrid miss (no checkpoint ≤ 555 of 618); cold prefill). At 128K the periodic checkpoints (--ssm-checkpoint-stride) fall inside the history, so long sessions do not have this problem.

The model still obeys the effort line at the tail

Paired test on the same 100 questions (GPQA-Diamond 50 + LiveCodeBench 50): thinking tokens per question, geometric mean of the ratio, 90% bootstrap interval. Swift 1.5 has 2 rounds per arm; iQ-MLX has 1. The medium prompt is identical in both templates, so its ratio is an A/A test that shows the noise floor.

Thinking at medium and low, as a share of xhigh

Per pack and template. Lower is less thinking.

iQ-MLX reacts more to the effort line than Swift 1.5. At low, iQ-MLX thinks ×0.26 of its xhigh; Swift 1.5 thinks ×0.45. Swift 1.5 already thinks less at xhigh, so it has less to cut. The position of the line does not change these ratios.

Tail against official, same effort

Shaded band: the ×0.80–×1.25 acceptance range of the campaign rule.

Every interval sits inside the band. Swift 1.5 at low thinks 10% less with the tail template (×0.90, interval 0.86–0.94), outside the noise floor of the A/A (×0.98). At xhigh both packs lean slightly above ×1.0 (×1.04 and ×1.05), and both truncate a few more answers; the intervals include 1.0, so this is a hint, not a result.

Between the levels: what each effort buys

Official template on both packs. GPQA-Diamond and LiveCodeBench, 50 questions each (seed 42). Swift 1.5 counts are the sum of 2 rounds (out of 100); iQ-MLX counts are 1 round (out of 50).

Swift 1.5 · officialiQ-MLX · official

Median time per question · GPQA-Diamond

Official template. Swift 1.5 median over 2 rounds, iQ-MLX over 1.

xhigh

Swift 1.5 · official18.8 s

iQ-MLX · official63.9 s

medium

Swift 1.5 · official11.6 s

iQ-MLX · official16.7 s

low

Swift 1.5 · official11.1 s

iQ-MLX · official14.1 s

Median thinking tokens · GPQA-Diamond

Official template. Swift 1.5 median over 2 rounds, iQ-MLX over 1.

xhigh

Swift 1.5 · official1,398

iQ-MLX · official5,120

medium

Swift 1.5 · official976

iQ-MLX · official1,374

low

Swift 1.5 · official926

iQ-MLX · official1,180

Median time per question · LiveCodeBench v6

Official template. Swift 1.5 median over 2 rounds, iQ-MLX over 1.

xhigh

Swift 1.5 · official18.9 s

iQ-MLX · official39.1 s

medium

Swift 1.5 · official15.3 s

iQ-MLX · official16.0 s

low

Swift 1.5 · official14.0 s

iQ-MLX · official14.0 s

Median thinking tokens · LiveCodeBench v6

Official template. Swift 1.5 median over 2 rounds, iQ-MLX over 1.

xhigh

Swift 1.5 · official1,214

iQ-MLX · official2,850

medium

Swift 1.5 · official913

iQ-MLX · official948

low

Swift 1.5 · official793

iQ-MLX · official808

Suite	Effort	Swift 1.5 · official				iQ-MLX · official
Suite	Effort	correct	truncated	median time	s per correct	correct	truncated	median time	s per correct
GPQA-Diamond	`xhigh`	85/100	3	18.8 s	119 s	45/50	4	63.9 s	211 s
GPQA-Diamond	`medium`	83/100	0	11.6 s	27 s	41/50	0	16.7 s	34 s
GPQA-Diamond	`low`	75/100	0	11.1 s	18 s	39/50	0	14.1 s	19 s
LiveCodeBench v6	`xhigh`	94/100	1	18.9 s	93 s	45/50	3	39.1 s	192 s
LiveCodeBench v6	`medium`	81/100	0	15.3 s	34 s	44/50	0	16.0 s	41 s
LiveCodeBench v6	`low`	83/100	0	14.0 s	21 s	40/50	0	14.0 s	52 s

Code needs xhigh; chat does not. On LiveCodeBench, medium and low drop 5–7 problems per 50 on Swift 1.5 and 1–5 on iQ-MLX. On GPQA, medium costs 1–4 questions per 50 and low 5–6, while the time per correct answer falls 3–6×, because the long tail of truncated xhigh answers disappears. The campaign's recommendation: xhigh for code and long sessions, medium for chat and short tool calling (where it is 23–28% faster with the same score), fixed per session when the official template is in use.

Time per question, template against template

One dot per question and round: wall time to the final answer, log scale; the top row is the official template, the bottom row the effort tail. Hover a dot for the question, the time and the token count. The iQ-MLX xhigh official run comes from the Swift 1.5 vs iQ-MLX page (2026-10-10); every other run is from 2026-10-13 to 2026-10-17 on the same server build.

Swift 1.5 · officialSwift 1.5 · effort tailiQ-MLX · officialiQ-MLX · effort tail correct wrong× hit the max_tokens limit median

Swift 1.5 · GPQA-Diamond · xhigh 2 rounds

median 18.8 s → 18.3 s

Correct 85/100 → 85/100 · truncated at 65,536 tokens 3 → 5 · median tokens 1,820 → 1,745.

Swift 1.5 · GPQA-Diamond · medium 2 rounds

median 11.6 s → 11.8 s

Correct 83/100 → 83/100 · truncated at 65,536 tokens 0 → 0 · median tokens 1,338 → 1,248.

Swift 1.5 · GPQA-Diamond · low 2 rounds

median 11.1 s → 10.6 s

Correct 75/100 → 76/100 · truncated at 65,536 tokens 0 → 0 · median tokens 1,234 → 1,172.

Swift 1.5 · LiveCodeBench v6 · xhigh 2 rounds

median 18.9 s → 19.9 s

Correct 94/100 → 94/100 · truncated at 65,536 tokens 1 → 2 · median tokens 1,964 → 2,043.

Swift 1.5 · LiveCodeBench v6 · medium 2 rounds

median 15.3 s → 14.4 s

Correct 81/100 → 85/100 · truncated at 65,536 tokens 0 → 0 · median tokens 1,856 → 1,826.

Swift 1.5 · LiveCodeBench v6 · low 2 rounds

median 14.0 s → 12.7 s

Correct 83/100 → 81/100 · truncated at 65,536 tokens 0 → 0 · median tokens 1,696 → 1,606.

iQ-MLX · GPQA-Diamond · xhigh 1 round

median 63.9 s → 50.9 s

Correct 45/50 → 42/50 · truncated at 65,536 tokens 4 → 7 · median tokens 5,373 → 4,317.

iQ-MLX · GPQA-Diamond · medium 1 round

median 16.7 s → 17.5 s

Correct 41/50 → 40/50 · truncated at 65,536 tokens 0 → 0 · median tokens 1,686 → 1,746.

iQ-MLX · GPQA-Diamond · low 1 round

median 14.1 s → 12.9 s

Correct 39/50 → 39/50 · truncated at 65,536 tokens 0 → 0 · median tokens 1,468 → 1,382.

iQ-MLX · LiveCodeBench v6 · xhigh 1 round

median 39.1 s → 46.9 s

Correct 45/50 → 46/50 · truncated at 65,536 tokens 3 → 2 · median tokens 3,518 → 4,244.

iQ-MLX · LiveCodeBench v6 · medium 1 round

median 16.0 s → 17.7 s

Correct 44/50 → 43/50 · truncated at 65,536 tokens 0 → 0 · median tokens 1,846 → 2,018.

iQ-MLX · LiveCodeBench v6 · low 1 round

median 14.0 s → 13.7 s

Correct 40/50 → 41/50 · truncated at 65,536 tokens 0 → 0 · median tokens 1,810 → 1,822.

The medians move by less than the round-to-round noise. What moves is the tail: at xhigh the effort-tail template hit the token limit more often (Swift 1.5: 4 → 7 answers over 2 rounds; iQ-MLX: 7 → 9 in 1 round), and each truncated answer costs about 1,000 s. The four GPQA questions iQ-MLX lost at xhigh were all truncations.

Ways to keep both: a cheap switch and the short-prompt cache

Five options, none tested yet, written down on 2026-10-17. They all rest on the checkpoint rule inferred from the logs (one checkpoint at the start of the last message); confirming that rule in the mlx-serve source comes before options 4 and 5. Dashed blocks are the part that stays equal between the two requests of each diagram.

1. Effort block before the last user message (template)

The last message is the user question again, so the checkpoint returns to the end of the shared prefix. Short prompts reuse the system prompt as with the official template; an effort switch at 128K keeps the history and re-reads only the effort block and the last question.

Short prompts:

request Asystemtoolseffortuser: question 1assistant

request Bsystemtoolseffortuser: question 2assistant

Effort switch at 128K:

turn N, xhighsystem128K historyeffort: xhighuser Nassistant

turn N, lowsystem128K historyeffort: lowuser Nassistant

New question in a long session:

turn Nsystemhistoryeffortuser Nassistant

turn N+1systemhistoryuser Nassistant Neffortuser N+1assistant

Cost: in a long session every new question moves the effort block, so the server re-reads the whole previous turn (its question, tool results and answer). The cost is bounded by one turn, not by the session. Test: probes 3e and 3f and the obedience battery of Task 11, plus a short 3c to measure the control turn.

2. Smaller --ssm-checkpoint-stride

Keep the tail template and let the server write periodic checkpoints inside short prompts. With a stride of 512 tokens, a 600–1,200-token prompt gets one or two checkpoints inside the shared prefix.

stride todaysystemtoolsuser: questioneffortassistant

stride 512systemtoolsuser: questioneffortassistant

Cost: the server help says each checkpoint forces a chunk boundary and re-reads the expert weights during prefill, and each one stores a copy of the GDN state. Reuse is partial: it stops at the last checkpoint before the end of the shared prefix. Test: prefill time and memory at 32K, 128K and 256K, then probe 3f.

3. Hybrid template by conversation length

The Jinja template can count messages and characters. With few messages the effort line goes first (short-prompt cache intact); past a threshold it goes last (cheap switch). A session pays one full prefill when it crosses the threshold.

few messageseffortsystemtoolsuser: questionassistant

many messagessystemlong historyuser Neffortassistant

Cost: the threshold is fixed in the template and invisible to the client. No server change. A fallback if option 1 regresses on long sessions.

4. A checkpoint at every message boundary (mlx-serve)

If the server saved the SSM state at the start of every message, not only the last one, there would always be a checkpoint at the end of the shared prefix, and the position of the effort block would stop mattering.

every boundarysystemtoolsuser: questioneffortassistant

Cost: a change in mlx-serve, after the current rule is confirmed in its source. More checkpoints per prompt, each with a copy of the GDN state.

5. Take the effort out of the prompt

The client pins the effort per session (the current decision), or limits thinking with a per-request budget if the server exposes one. Without the line in the prompt, the cache does not depend on the effort.

requestsystemtoolsuser: questionassistant+ thinking budget in the API call

Cost: a per-request thinking budget is not the same control as the effort line, and it was not checked whether mlx-serve exposes one.

Suggested order: options 1 and 2 together, with the Task 11 battery, because both use the overlays and scripts that exist. Option 3 as a fallback. Options 4 and 5 depend on mlx-serve.

All numbers

Counts are sums over the rounds listed (official / tail). Tokens are thinking plus visible answer. Total time is the wall time of the whole suite.

Pack	Suite	Effort	Rounds	Correct		Truncated		Median time		Total time		Median tokens		Total tokens
Pack	Suite	Effort	Rounds	official	tail	official	tail	official	tail	official	tail	official	tail	official	tail
Swift 1.5	GPQA-Diamond	`xhigh`	2 / 2	85	85	3	5	18.8 s	18.3 s	169 min	183 min	1,820	1,745	795K	859K
Swift 1.5	GPQA-Diamond	`medium`	2 / 2	83	83	0	0	11.6 s	11.8 s	38 min	34 min	1,338	1,248	228K	209K
Swift 1.5	GPQA-Diamond	`low`	2 / 2	75	76	0	0	11.1 s	10.6 s	23 min	21 min	1,234	1,172	148K	137K
Swift 1.5	LiveCodeBench v6	`xhigh`	2 / 2	94	94	1	2	18.9 s	19.9 s	146 min	153 min	1,964	2,043	691K	731K
Swift 1.5	LiveCodeBench v6	`medium`	2 / 2	81	85	0	0	15.3 s	14.4 s	46 min	57 min	1,856	1,826	281K	315K
Swift 1.5	LiveCodeBench v6	`low`	2 / 2	83	81	0	0	14.0 s	12.7 s	29 min	29 min	1,696	1,606	193K	190K
iQ-MLX	GPQA-Diamond	`xhigh`	1 / 1	45	42	4	7	63.9 s	50.9 s	158 min	183 min	5,373	4,317	692K	795K
iQ-MLX	GPQA-Diamond	`medium`	1 / 1	41	40	0	0	16.7 s	17.5 s	23 min	23 min	1,686	1,746	137K	138K
iQ-MLX	GPQA-Diamond	`low`	1 / 1	39	39	0	0	14.1 s	12.9 s	12 min	12 min	1,468	1,382	79K	77K
iQ-MLX	LiveCodeBench v6	`xhigh`	1 / 1	45	46	3	2	39.1 s	46.9 s	144 min	135 min	3,518	4,244	615K	602K
iQ-MLX	LiveCodeBench v6	`medium`	1 / 1	44	43	0	0	16.0 s	17.7 s	30 min	26 min	1,846	2,018	164K	153K
iQ-MLX	LiveCodeBench v6	`low`	1 / 1	40	41	0	0	14.0 s	13.7 s	34 min	32 min	1,810	1,822	160K	153K

Setup and how to reproduce

Mac Studio, Apple M4 Max, 40-core GPU, 128 GB unified memory, macOS 26.6. Server: mlx-serve 26.10.1 with the same flags for every arm (--prefix-cache on, 131,072-token context for the batteries, 262,144 for the 3e probe). Sampling: temperature 1.0, top_p 0.95, top_k 20, seed 42, max_tokens 65,536 on the batteries.

Arm	Weights	Chat template
w1	Dankpaws/Swift1.5-Qwen3.8-Flash-Next-MLX-4.7bpw rev 13839b7	official (`chat_template.jinja` of the pack)
w1t	same weights, hardlink overlay	swift15-effort-tail.jinja
n2	ddalcu/Qwen3.8-Flash-Next-MLX-Serve-iQ-MLX-4.7bpw rev dafff5c	official
n2t	same weights, hardlink overlay	swift15-effort-tail.jinja (also patched into `tokenizer_config.json`)

The overlay is built by scripts/make-overlay.sh: every file of the pack hardlinked into a new directory, with only the template replaced. Probes: scripts/agent_probes.py (3e, 3f), scripts/run-battery.sh with bench2.py (GPQA, LiveCodeBench), scripts/effort_obedience.py (paired bootstrap), scripts/render_effort_tail_page.py (this page). Runbook: plan-effort-needles.md, Tasks 11 and 15; write-ups: etapa11-effort-tail.md and etapa15-iq-effort-tail.md.

Limits

GPQA-Diamond and LiveCodeBench are 50-question samples, not the full sets. Swift 1.5 has 2 rounds per effort and template, iQ-MLX has 1. A difference of 2–3 questions is within noise; the campaign rule accepts a drop of up to 3 per round.
The iQ-MLX xhigh official run is from another server session (2026-10-10). Its thinking ratio against the tail run (×1.05) is close to the same-session A/A at medium (×1.01), so the sessions look comparable, but it is not a same-day pair.
The checkpoint position (start of the last message) is inferred from the hybrid miss and inherited N checkpoints lines of the server log, not from the mlx-serve source.
The short-prompt probe (3f) ran only on Swift 1.5. The prompt is the same for iQ-MLX, so the cache behaviour should match, but it was not measured.
None of the five options in Ways to keep both was tested. They are listed as hypotheses with their expected cost.
LiveCodeBench answers graded by the old runner were regraded with the official import prelude; no passing answer turned into a failure.

Data, scripts and per-question results: github.com/snagnever/macstudio-local-llm. Generated 2026-10-17T16:31:37Z.