Qwen3.8-Flash-Next · Apple M4 Max 128 GB · mlx-serve 26.10.1 · October 2026

Swift 1.5 vs iQ-MLX: same accuracy, less waiting

Two 4.7-bit MLX packs of the same model on the same runtime: the iQ-MLX pack by ddalcu and the Swift 1.5 fine-tune by Dankpaws. Six suites, two sampling settings, one dot per question.

On an Apple M4 Max, Swift 1.5 gets to the final answer 2.9× faster on GPQA-Diamond and 2.0× faster on LiveCodeBench (median per question), with the same accuracy. Both generate at the same speed (74 vs 74 tokens/s on GPQA). The gain comes from shorter thinking, not from faster generation.

GPQA-Diamond

2.9×

faster to the final answer · median 63.9 s → 22.3 s

LiveCodeBench v6

2.0×

faster to the final answer · median 39.1 s → 19.1 s

Thinking per answer

−67%

GPQA median 5,124 → 1,689 tokens

Accuracy

≤ 3

questions apart on every suite, in both directions

Tiles use the recommended sampling (temperature 1.0). With greedy decoding the gain is smaller: 1.2× faster on GPQA-Diamond and 1.5× faster on LiveCodeBench.

Sampling for the charts below

Time per task

One dot per question: wall time from the request to the final answer, log scale. Hover a dot for the question, the time and the token count. The bar under each chart counts the questions each model finished first.

iQ-MLXSwift 1.5 correct wrong× hit the max_tokens limit median

GPQA-Diamond 50 questions, seed 42

2.9× faster (median)

Swift 1.5 faster on 38 of 50 · iQ-MLX faster or tied on 12

LiveCodeBench v6 50 problems, seed 42

2.0× faster (median)

Swift 1.5 faster on 41 of 50 · iQ-MLX faster or tied on 9

HumanEval 164 problems

1.2× faster (median)

Swift 1.5 faster on 114 of 164 · iQ-MLX faster or tied on 50

Total time to run the suite

Sum of the wall time of every question, one question at a time.

GPQA-Diamond

iQ-MLX158 min

Swift 1.595 min

LiveCodeBench v6

iQ-MLX144 min

Swift 1.581 min

HumanEval

iQ-MLX54 min

Swift 1.538 min

iQ-MLXSwift 1.5 correct wrong× hit the max_tokens limit median

GPQA-Diamond 50 questions, seed 42

1.2× faster (median)

Swift 1.5 faster on 41 of 50 · iQ-MLX faster or tied on 9

LiveCodeBench v6 50 problems, seed 42

1.5× faster (median)

Swift 1.5 faster on 40 of 50 · iQ-MLX faster or tied on 10

HumanEval 164 problems

1.2× faster (median)

Swift 1.5 faster on 126 of 164 · iQ-MLX faster or tied on 38

Total time to run the suite

Sum of the wall time of every question, one question at a time.

GPQA-Diamond

iQ-MLX140 min

Swift 1.556 min

LiveCodeBench v6

iQ-MLX100 min

Swift 1.562 min

HumanEval

iQ-MLX55 min

Swift 1.527 min

Why it is faster

Same runtime, same flags, same generation speed. Swift 1.5 writes fewer thinking tokens before it answers.

iQ-MLXSwift 1.5

Tokens per answer (median)

Thinking plus the visible answer.

GPQA-Diamond

iQ-MLX5,124 tokens

Swift 1.51,689 tokens

LiveCodeBench v6

iQ-MLX2,984 tokens

Swift 1.51,448 tokens

HumanEval

iQ-MLX468 tokens

Swift 1.5403 tokens

Generation speed (median)

Output tokens divided by the wall time of each answer.

GPQA-Diamond

iQ-MLX74.0 tok/s

Swift 1.574.3 tok/s

LiveCodeBench v6

iQ-MLX73.9 tok/s

Swift 1.573.3 tok/s

HumanEval

iQ-MLX69.1 tok/s

Swift 1.569.2 tok/s

Medians. The total token count on GPQA-Diamond drops less, from 675k to 434k tokens, because a few answers on both sides run to the max_tokens limit (4 for iQ-MLX, 2 for Swift 1.5).

iQ-MLXSwift 1.5

Tokens per answer (median)

Thinking plus the visible answer.

GPQA-Diamond

iQ-MLX1,530 tokens

Swift 1.51,148 tokens

LiveCodeBench v6

iQ-MLX1,860 tokens

Swift 1.51,285 tokens

HumanEval

iQ-MLX477 tokens

Swift 1.5368 tokens

Generation speed (median)

Output tokens divided by the wall time of each answer.

GPQA-Diamond

iQ-MLX77.4 tok/s

Swift 1.577.2 tok/s

LiveCodeBench v6

iQ-MLX75.1 tok/s

Swift 1.576.1 tok/s

HumanEval

iQ-MLX72.2 tok/s

Swift 1.571.0 tok/s

Medians. The total token count on GPQA-Diamond drops less, from 705k to 281k tokens, because a few answers on both sides run to the max_tokens limit (7 for iQ-MLX, 0 for Swift 1.5).

Accuracy

Correct answers per suite. A truncated answer counts as wrong. LiveCodeBench answers are graded with the import prelude of the official LiveCodeBench harness (regraded 2026-10-14; the first grading of this page scored both models 4–5 problems lower).

Suite	iQ-MLX	Swift 1.5	Swift − iQ
HumanEval 164 problems	158/164	158/164	0
GPQA-Diamond 50 questions, seed 42	45/50	43/50	−2
LiveCodeBench v6 50 problems, seed 42	45/50	47/50	+2
Tool calling · jdhodges 40 cases	35/40	36/40	+1
Tool calling · Veerman 12 cases	10/12	10/12	0
Instruction following 20 IFEval-style cases	20/20	19/20	−1

HumanEval is graded with the imports of the prompt added, because a correct answer that omits from typing import List fails the strict grader. Strict grade: 144 (iQ-MLX) vs 147 (Swift 1.5).

Suite	iQ-MLX	Swift 1.5	Swift − iQ
HumanEval 164 problems	155/164	157/164	+2
GPQA-Diamond 50 questions, seed 42	39/50	42/50	+3
LiveCodeBench v6 50 problems, seed 42	45/50	45/50	0
Tool calling · jdhodges 40 cases	38/40	39/40	+1
Tool calling · Veerman 12 cases	10/12	9/12	−1
Instruction following 20 IFEval-style cases	20/20	19/20	−1

HumanEval is graded with the imports of the prompt added, because a correct answer that omits from typing import List fails the strict grader. Strict grade: 138 (iQ-MLX) vs 143 (Swift 1.5).

More detail

iQ-MLXSwift 1.5

Seconds per correct answer

Total suite time divided by the number of correct answers.

GPQA-Diamond

iQ-MLX211 s

Swift 1.5132 s

LiveCodeBench v6

iQ-MLX192 s

Swift 1.5103 s

HumanEval

iQ-MLX22 s

Swift 1.516 s

LiveCodeBench by difficulty (median time)

The gain grows with the difficulty of the problem.

easy · 15 problems

iQ-MLX7.5 s 15/15 solved

Swift 1.57.7 s 15/15 solved

medium · 23 problems

iQ-MLX35.0 s 22/23 solved

Swift 1.522.4 s 22/23 solved

hard · 12 problems

iQ-MLX480.9 s 8/12 solved

Swift 1.5269.5 s 10/12 solved

Thinking + answer tokens (median of each part)

Solid: thinking. Light: the visible answer. Only the thinking shrinks.

LiveCodeBench v6

iQ-MLX2,850 + 169

Swift 1.51,214 + 158

HumanEval

iQ-MLX321 + 113

Swift 1.5249 + 130

iQ-MLXSwift 1.5

Seconds per correct answer

Total suite time divided by the number of correct answers.

GPQA-Diamond

iQ-MLX215 s

Swift 1.581 s

LiveCodeBench v6

iQ-MLX133 s

Swift 1.582 s

HumanEval

iQ-MLX24 s

Swift 1.511 s

LiveCodeBench by difficulty (median time)

The gain grows with the difficulty of the problem.

easy · 15 problems

iQ-MLX8.2 s 15/15 solved

Swift 1.56.5 s 14/15 solved

medium · 23 problems

iQ-MLX23.4 s 22/23 solved

Swift 1.515.5 s 22/23 solved

hard · 12 problems

iQ-MLX278.5 s 8/12 solved

Swift 1.5122.2 s 9/12 solved

Thinking + answer tokens (median of each part)

Solid: thinking. Light: the visible answer. Only the thinking shrinks.

LiveCodeBench v6

iQ-MLX1,656 + 150

Swift 1.5988 + 172

HumanEval

iQ-MLX339 + 114

Swift 1.5232 + 138

Short tasks: about the same

Tool calling and instruction following need little thinking, and both models answer in 2–3 seconds. For tool calling, the time adds the turn that writes the call and, for the multi-turn cases, the turn that reads the tool result.

iQ-MLXSwift 1.5 correct wrong× hit the max_tokens limit median

Tool calling · jdhodges 40 cases

about the same (median)

Tool calling · Veerman 12 cases

about the same (median)

Instruction following 20 IFEval-style cases

1.1× slower (median)

iQ-MLXSwift 1.5 correct wrong× hit the max_tokens limit median

Tool calling · jdhodges 40 cases

1.1× faster (median)

Tool calling · Veerman 12 cases

about the same (median)

Instruction following 20 IFEval-style cases

1.4× faster (median)

Agent turn at long context

One coding-agent turn on a long conversation: the time to the first token of a turn that carries a tool result, plus 512 generated tokens. Prefix cache on, median of 3 repetitions, recommended sampling. The two packs are within 4% of each other.

Context	Turn time, s		Decode, tok/s		Prefill, tok/s
Context	iQ-MLX	Swift 1.5	iQ-MLX	Swift 1.5	iQ-MLX	Swift 1.5
32K tokens	8.1	7.9	82.3	84.6	791	791
128K tokens	8.9	8.6	73.2	76.7	760	759

All numbers

Suite	Median time, s		Total time, min		Median tokens		Hit max_tokens
Suite	iQ-MLX	Swift 1.5	iQ-MLX	Swift 1.5	iQ-MLX	Swift 1.5	iQ-MLX	Swift 1.5
HumanEval	6.8	5.7	54	38	468	403	0	1
GPQA-Diamond	63.9	22.3	158	95	5,124	1,689	4	2
LiveCodeBench v6	39.1	19.1	144	81	2,984	1,448	3	1
Tool calling · jdhodges	2.4	2.5	3	2	114	109	0	0
Tool calling · Veerman	3.1	3.0	1	1	142	164	0	0
Instruction following	1.9	2.2	1	1	128	150	0	0

Suite	Median time, s		Total time, min		Median tokens		Hit max_tokens
Suite	iQ-MLX	Swift 1.5	iQ-MLX	Swift 1.5	iQ-MLX	Swift 1.5	iQ-MLX	Swift 1.5
HumanEval	6.6	5.5	55	27	477	368	3	1
GPQA-Diamond	18.3	14.9	140	56	1,530	1,148	7	0
LiveCodeBench v6	23.9	16.0	100	62	1,860	1,285	3	2
Tool calling · jdhodges	2.6	2.4	2	2	108	104	0	0
Tool calling · Veerman	3.2	3.2	1	1	170	194	0	0
Instruction following	2.5	1.8	1	1	188	129	0	0

Setup and how to reproduce

Chip

Apple M4 Max

CPU

16 cores (12P+4E)

GPU

40 cores · Metal 4

Unified memory

128 GB

macOS

26.6.2 (25G83)

Two 4.7-bit MLX packs of Qwen3.8-Flash-Next on the same mlx-serve 26.10.1 binary with the same flags. Only the weights change.

Model	Weights (Hugging Face)
iQ-MLX	ddalcu/Qwen3.8-Flash-Next-MLX-Serve-iQ-MLX-4.7bpw rev dafff5c
Swift 1.5	Dankpaws/Swift1.5-Qwen3.8-Flash-Next-MLX-4.7bpw rev 13839b7

Server flags: --ctx-size 131072 --mtp --ssm-checkpoint-max 16 --prefix-cache-mem 16GB --prefix-cache-disk 100GB --prefix-cache-entries 64. Every request sends reasoning_effort: xhigh; without it, mlx-serve answers with no thinking. Recommended sampling: temperature 1.0, top_p 0.95, top_k 20, min_p 0, presence 0, repetition 1.0; greedy decoding: temperature 0. max_tokens: HumanEval 32,768 · GPQA and LiveCodeBench 65,536 · tool calling 32,768 · instruction following 16,384. Questions run one at a time.

Scripts, per-question results and the test plan: github.com/snagnever/macstudio-local-llm.

Limits

One run per model and sampling setting. GPQA-Diamond and LiveCodeBench are 50-question samples (seed 42), not the full sets, so a difference of 2–3 questions is within run-to-run noise.
Times are medians per question. A few answers on both sides run to the max_tokens limit and dominate the totals, so the totals improve less than the medians. The page shows both.
The two packs share the same quantization layout but ship different n-gram tables, so this compares the two packs as published, not only the fine-tuned weights.
Tool calling: the runner did not record the thinking text. A median of 104–194 tokens per answer suggests little or no thinking in these suites.
Instruction following is a 20-case IFEval-style probe. It does not reproduce the IFBench score of the Swift 1.5 model card.
Agent quality (for example Terminal-Bench) is not measured.

Data, scripts and per-question results: github.com/snagnever/macstudio-local-llm. Generated 2026-10-14T21:59:27Z.