How many agent sessions fit on one local Flash-Next server?

How the test works

More sessions, slower turns

Dashed lines: at or below 1.5× passes; between 1.5× and 2.0× is a warning (passes); above 2.0× fails.

Why two sessions do not add up

The server already batches the sessions’ generation: it writes one token for each session in the same step. On this model that step gets slower as sessions are added, so the total grows little. Going from 1 to 4 sessions, the best total grows at most 1.44× the speed of one session alone at 32K and 1.26× at 128K, while each session slows down by more.

In the 2-session runs the first session started before the second and ran some turns alone, which lifts the 2-session average. An independent benchmark of the same model agrees from 128K up: about 1.4× total throughput with short prompts, but no gain over one session alone at long contexts.

Short side requests wait for long prompts

A side request takes about a second when the server is free. When it arrives while another session is reading a long prompt, it waits for that read to finish: a cold 128K prompt takes about 166 s to read and a 256K one about 351 s (one session alone, median). At 32K the slowest side requests (95th percentile) took 22.0 s–41.1 s; at 128K and 256K, 168.1 s–358.6 s.

Upstream, PR #884 lets small requests in between prefill chunks, but it excludes this model’s architecture and is not in a release yet.

Does --max-concurrent matter?

--max-concurrent sets how many requests the server runs at once. On mlx-serve 26.10.1 the default is 1, yet the boot log already says Concurrency: --max-concurrent=1, batched decode on: two sessions share generation either way. The flag is not a queue, and setting it to 2 changes nothing measurable. At 128K the side-request p95 is higher with the flag at 2 because of a single very slow request; the count of side requests over 60 s is close in both runs.

256K: two long sessions

Two 256K sessions fit only when the prompt cache kept in RAM is cut from 16 GB to 4 GB. That configuration passes with a warning: worst slowdown 1.68× against the 11.1 s reference, or 1.81× against the 10.3 s we measured for one 256K session alone over 3 runs. With the 4 GB cache, warm turns lose about a second restoring the cache from the SSD.

Recall rule at 256K: a session must get at least 11 of every 13 answered checks (85%) right, so it may miss up to 2 of 13, the same as one 256K session alone. With the 8-bit KV cache it must miss none. The table shows each session’s count.

Refused / HTTP errors (client): requests the client saw fail. The server log can count more refusals than the client sees, so the two numbers can differ.

Two real Claude Code sessions

Two real Claude Code 2.1.292 sessions (claude -p) shared the everyday server setup (128K context, default --max-concurrent 1), each on a fixed coding task with the same model as both main and fast model. Every task was done, with no errors and no refusals. Each session took 1.84×–2.36× its time alone, above the 1.5× rule.

Functional pass, time fail. The pair finished in 147.6 s on average; the two tasks one after the other take 137.6 s. Running them together finishes 7% later.

--prefill-decode-share 0.3 and 0.5

--prefill-decode-share makes generation keep a share of the time while another request reads its prompt. We tried it at 128K to clear the side-request wait. It did not: as many side requests waited over 60 s as without it, and the cold first token took about twice as long. Not adopted.

What we learned

1. mlx-serve 26.10.1 already batches generation for this model at the default --max-concurrent 1; the flag is not a queue.
2. Two sessions is the most that passes: 1.76× at 32K and 1.94× at 128K. Three fail (2.37× / 2.36×), four fail (3.37× / 3.85× with side requests, 2.84× / 3.66× without).
3. Total generation speed grows little with more sessions, so two sessions together are not faster: two real Claude Code tasks took 147.6 s together against 137.6 s one after the other.
4. A short request that arrives while another session reads a long prompt waits for it (p95 168.1 s–358.6 s at 128K and 256K); neither --max-concurrent nor --prefill-decode-share removes the wait.
5. Two 256K sessions fit only with a 4 GB prompt cache in RAM; with 16 GB or an 8-bit KV cache the server refused requests or the model missed a recall check.
6. The everyday server setup stays as it was. The launcher gained a 256k profile for two long sessions.

Related upstream work

We have not opened an issue about the admission refusals below, and their cause is not confirmed.

Not measured

• The iQ-MLX 4.7 bpw build of the same model with two sessions.
• 512K with two sessions (the 512K launcher profile forces one request at a time).
• mlx-serve 26.10.2, which changes batching at long context and the SSD cache, and PR #884 in a future release.
• More than four sessions. Three sessions ran only with side requests.
• A real queue in the client (a lock that sends one request at a time): this server version has no switch to turn batching off.
• An interactive Claude Code pair with real permission prompts and titles.

Reproduce and data

Server commands, from the repository root ($MODEL is the model directory):

tools/scripts/serve-flashnext-daily-driver.sh        # everyday setup: 128K, 16 GB prompt cache
tools/scripts/serve-flashnext-daily-driver.sh 256k   # two long sessions: 256K, 4 GB prompt cache

# the 256k profile runs:
mlx-serve --model "$MODEL" --serve --host 0.0.0.0 --port 11234 --metrics --mtp \
  --ssm-checkpoint-max 16 --prefix-cache-mem 4GB --prefix-cache-disk 100GB \
  --prefix-cache-entries 64 --ctx-size 262144 --max-concurrent 2

Runbook: plan.md. Raw results and stage reports: results/. The data behind this page is embedded in it as JSON (#data).