MTPLX 2.10.0

Measured on an M5 Max against 2.9.2, stock settings:

Every row is stock 2.9.2 against stock 2.10.0 on the same Mac, so rows that ride a raised dependency floor or a changed default show that full stock-to-stock gain; the body section for each row says exactly what was measured and how. The 48 GB row is the pinned simulated seat from the release gate.

Qwen 3.8 Flash-Next

Qwen's 125B MoE preview is a native MTPLX family. The backend is parity-exact against the reference implementation, the model's own MTP draft head rides the standard speculative lane, and the family got its own kernel campaign before release. Through the real server on an M5 Max it measures 61 tok/s plain decode and 63 to 76 tok/s with the MTP head, depending on workload.

• Two packs on Hugging Face: Bare Speed (flat 4-bit, quickest) and Optimized Speed (dynamic 4-bit with 8-bit attention, higher quality). Both appear in the app picker, onboarding, and the CLI catalog with accurate sizes, both resolve the Turbo profile by default, and mtplx pull accepts mtplx-flash-next-bare-speed, mtplx-flash-next-optimized-speed, or the HF repo names.
• The default kernel set was A/B measured lane by lane before promotion: a pipelined compiled decode step, fused GDN projections and decode step, fused conv+SiLU+norm at draft and verify widths, a fused hyper-connection read, and merged MoE gate/up projections. Capture-commit rollback is also on by default; it cut verify rounds from 51.7 to 38.8 ms and eliminated rejection re-forwards.
• The model's 32 GB n-gram table streams from SSD instead of occupying RAM, so both packs fit 96 GB Macs with headroom. A hot-row cache (MTPLX_NGRAM_HOT_MB, default 1024) recovers the speed at identical memory: 5 to 10 percent more plain decode and 7.5 to 16 percent more MTP decode, byte-identical output. Counting the table as weights had produced a false "MODEL DOES NOT FIT" and a 30 GB smaller context window on 128 GB machines.
• Reasoning effort has full 27B parity: xhigh, medium, and low in the app, --reasoning-effort on the CLI, and reasoning_effort in the API, with chat defaulting to xhigh. The family used to silently ignore the field and always think at maximum.
• Reasoning history is preserved across turns by default, matching the model card. On the old scoped fallback a mid-session turn re-prefilled 300 to 1,500 tokens and took 1.8 to 2.2 s to first token; with preserve it costs about 20 new prefill tokens and 0.11 s.
• Flash-Next verifies through the generic batched lane for now. Under greedy decoding a rare near-tie argmax can differ from single-step decode; it appeared once in 192 tokens in the release gate, and both continuations converged. The acceptance math is unchanged and exact at any temperature, and the family-native verifier is in progress.

Coding agents

The defaults are now tuned on wall-clock receipts from real multi-file tasks through OpenCode, Pi, and Hermes, and the failure modes that killed agent runs are gone.

• OpenCode and Pi default to the family's agent-lane reasoning effort, which is medium for Flash-Next. On the identical task that means 44.2 s instead of 150.2 s at the old xhigh default, with the same correct output. Chat keeps xhigh, and both effort pickers still offer it per request.
• Receipts from the release rig on Bare Speed: OpenCode built a five-file game in about 40 s per run, Pi fixed a two-bug module with passing tests in 13.2 s, and Hermes ran the same class of task in 39.8 s with its 17.7k-token system prompt answered from the session bank at 0.3 to 0.5 s first token on every turn after the first.
• The compact tool contract keeps every declared tool (#376, adapting PR #379 by @ArctifoxNL). The old byte-boundary cut deleted whole tool names from the tail, task first in Claude Code toolsets, and the model then treated the missing tools as nonexistent, which killed subagents.
• Cancellation errors name their real cause (#381), and streaming tool-call turns no longer die with a fabricated "cancelled via POST /v1/mtplx/cancel" (#332, #343). The server was misreading its own end-of-turn cancel as a foreign one, which hit hardest on slower machines and longer contexts.
• Tool calls the server cannot execute leave a truthful notice in the chat, the history, and mtplx_stats instead of silence (#349).
• Pi re-syncs update MTPLX's own compatibility keys, so a block written by an older MTPLX can no longer silently disable the effort dial after an upgrade. User-added keys survive.
• Agent requests are validated against family truth: depth respects the family ceiling (Flash-Next accepts up to 5), and requests adopt the pack's sampler instead of a generic temperature 0.6.

The engine plans memory for your Mac

MTPLX no longer assumes a 128 GB studio machine (issue #305 and friends). The serve banner prints the plan: engine budget, weights, resolved context window, and session bank. The default window is the largest one whose KV actually fits, so a 48 GB Mac serving the Speed model resolves 196,608 tokens instead of an impossible 262,144, and says so in the log instead of silently thrashing swap. On the simulated 48 GB seat at 42k context the release gate measured 33 tok/s decode and 645 tok/s prefill, with zero calls into the slow fallback path; 48 GB users on 2.9.2 had reported 3.1 to 4.6 tok/s with the app showing 129 percent memory used (#305). 128 GB machines are unchanged.

• The session bank yields idle sessions as a long request's KV grows, ahead of any swap, and the ceiling is enforced on every guard tick.
• The ceiling's reserve for generation spikes is measured, not guessed. It used to hold back a fixed 3 GiB while a deep chunked prefill really spikes up to 12 GiB, which fired the pressure banner on every deep turn. It now reserves the spike this process has actually observed, capped at half the memory left after weights, so one unusual turn cannot squeeze the bank for the rest of the session.
• The live session's own prefix chain is never evicted to make room. A 93k-token coding session had watched the ceiling walk the bank to zero mid-request and then re-prefill every following turn from scratch. The ceiling now squeezes idle sessions only, that protection holds through turns longer than its old timeout, and real system pressure still evicts whatever it needs to.
• Metal allocation failures shed caches and answer a structured HTTP 507 instead of crashing the daemon (#348 class).
• Sessions above the RAM cap spill to SSD tensor by tensor (#305, #323), so a 100k-token coding session survives a restart without a full re-prefill. Spill refusals print one line naming the entry size, the cap, and free disk.
• /health, the dashboard, and the app carry the plan, the guard events, and a pressure banner naming the exact --context-window fix. The dashboard reports the real resolved budgets instead of inventing 8G and 4G.
• MTPLX_MEMORY_BUDGET=48G reproduces any seat exactly, and the 48 GB seat is pinned by integration tests.

Decode past 128k

Past 131,072 prompt tokens the engine silently repaged dense decode into a layout that excluded the packed verify kernel, so verify paid about 30 times its normal round cost and speculative decode collapsed to plain AR speed. At 147k context on a 128 GB M5 Max, decode measured 12.0 tok/s before the fix, 16.3 after it, and 18.4 with prompt-copy speculation off.

• The ceiling is now memory-aware (MTPLX_SUSTAINED_DENSE_DECODE_MAX_CONTEXT=auto): it budgets 15 percent of device RAM against the model's KV bytes per token, clamps to the model's window, and floors at the old 131,072 so no smaller machine regresses. The resolution is announced once in the serve log, and a pinned numeric ceiling or exported env always wins.
• The packed split-KV verify kernel's q_len window extends from 4 to 8 by default, so deeper verify windows keep the fast kernel.

Long answers and rewrites hold their speed

A 34k-token generation decayed from 86 to 25 tok/s inside one request, and rewriting a file the model had just produced ran no faster than writing it fresh. Three causes were found and fixed:

• The draft head's history cache grew one row per generated token and was never trimmed during decode. It now resets and regrows every 16,384 generated tokens (MTPLX_MTP_HISTORY_LIVE_RESET_THRESHOLD, 0 disables). This affects draft conditioning only; the output stream is unchanged by construction.
• The allocator clear-cache cadence read prompt length only, so a short prompt with a long answer never cleared once and the allocator cache grew to 8.6 GB inside a single request. The cadence now arms mid-generation, and the same workload holds 0.6 GB.
• Context copy runs on the batched verify lane: when the stream's tail matches an n-gram in the prompt, up to 24 tokens are proposed as one block and verified in one forward pass, with the same probability-ratio acceptance as MTP, so sampling is identical at any temperature. On the rewrite turn this measured 87.6 tok/s against 73.8 for the fresh build, with 177 block rounds supplying 2,400 of the turn's 3,963 tokens. MTPLX_CONTEXT_COPY_BATCHED=0 disables it.
• Copy blocks start small and earn their size. A misfired block costs about four normal verify rounds, and short agent turns paid that cost fresh every turn before the lane could suspend itself, which showed up as decode dropping into the high 20s on small requests. Blocks now stay at 8 tokens until the turn's acceptance proves the content pays (MTPLX_CONTEXT_COPY_PROBATION_K), suspension arms one round earlier, and both verify lanes follow the same rule. Long rewrites still open to full blocks by their third round and keep the win above.

Receipts carry the copy-lane counters, launches with MTPLX_DROP_EVENTS=0 record per-round timings for decay attribution, and the growth-lever envs now beat the profile instead of being silently stomped.

Community lanes, adapted with credit

Four community pull requests landed this cycle, each adapted and measured before shipping.

• The warm ladder yields to live and queued requests and never stamps the traffic clock, so your first message after starting the engine is not sharing the GPU with a warm-up rung (PR #300 by @Blakeolson21).
• Margin-gate and adaptive-depth speculative lanes read the greedy draft token and its confidence in one GPU sync instead of two (PR #288 by @ArthurOstapenko). The default lanes are untouched.
• RAMP, a long-block plus fuzzy re-anchor policy for the context-copy proposer, built for temperature-0 edit-shaped agent runs where its author measured up to +54 percent decode (PR #375 by @johninthewinter). It stays off by default because our temperature-1 chat measurements favor the stock ladder, and off is byte-identical.
• A QSA rows-gather attention lane, adapted from the per-query gather in PR #380 by @maceip and on by default for Flash-Next past 16k context. Multi-row verify reads each row's selected blocks instead of the full KV, which held verify cost flat across a 16k-token answer while the dense path decayed (last-window 64.5 against 36.0 tok/s). MTPLX_QSA_GATHER=0 turns it off.

mlx floor raised to 0.32.2

A clean same-wheel A/B measured mlx 0.32.0 against 0.32.2 at 33.8 to 44.3 tok/s decode at 16k (+31%), 23.6 to 30.4 tok/s at 88k (+29%), and 379 to 535 tok/s prefill (+41%). The table's short-context row is a same-mlx engine pair on seed-identical 3,205-token answers. The mlx floor bump stacks on top of that for installs still sitting on 0.32.0; the floor pin converges everyone. The dependency floor moves so existing installs converge on the faster stack, and pin-guard tests fail loudly if an environment or a shipped profile ever drifts below it.

KV cache quantization works now

KV quantization used to crash serving at warmup, and before that bug existed it could cost half your decode speed. This release replaces the lane. Measured at 16k context on an M5 Max against quantization off, q8 costs about 4 percent decode and q4 about 19 percent, at half and a quarter the KV memory. That is still short of our zero-loss bar, so both toggles stay opt-in: q8 is a fair trade for double the context headroom, and q4 is for capacity emergencies.

• q4 routes through the exact packed-quant kernel via a persistent quantized bank instead of re-dequantizing the whole prefix every round, and the compiled verify bank promotes quantized caches instead of dropping every verify round to eager execution.
• A pre-existing q4 numerics defect at head_dim 128 fails closed, and the shipped family (head_dim 256) is exact.
• One limit remains: past the dense-decode ceiling, quantized decode still lands on a slow routing lane. The cause is located and the fix is planned for a follow-up.
• This lane covers the Qwen 3.5/3.6/3.8 attention family. Flash-Next has no toggle, and the app now explains why: its design keeps KV on only 12 of 48 layers, about 24 KB per token, and its sparse attention has no validated quantized lane yet.

Session cache and time to first token

• SSD session writes no longer starve on an idle server (#290). An idle pump persists pending sessions within seconds (MTPLX_SSD_IDLE_PUMP, default on), foreground work always wins, and SIGTERM or Ctrl-C gets a bounded flush (MTPLX_SHUTDOWN_SSD_FLUSH_S, default 10 s), so a killed daemon no longer costs a full re-prefill of sessions that were already computed.
• Session spills wait for the turn to finish. The writer already stood down while a request was running, but its pause expired after 60 seconds and a real coding turn runs one to ten minutes, so multi-gigabyte writes were landing under live decode, pushing the Mac into memory pressure and costing up to 30 percent of decode speed while they lasted. The pause now holds for 600 seconds (MTPLX_SSD_WRITER_FOREGROUND_PAUSE_MAX_S) and writes drain in the gaps between turns. An oversized session that streams straight to disk yields the same way: if a request arrives mid-write, the write aborts cleanly and retries in the next idle gap instead of making the request wait.
• A session snapshot bigger than the writer's whole backlog budget is admitted when the queue is empty instead of being refused forever (#384, reported by @sapiens77). Past roughly 50k tokens of 27B KV the old refusal had silently turned the SSD tier off. With writes landing, the reporter measured first token after a restart going from 282 s to 6.8 s.
• Preserved reasoning history keeps agent histories append-only, which is exactly what lets the session bank commit each turn without re-prefilling it.

See what the engine did

The decode cliff hid for months because fast lanes declined silently, so this release closes that class of blindness.

• Every request row records packed-route bail counters and paged-adapter engagement, so a lane that stops engaging shows up in the request log rather than in a benchmark weeks later.
• MTPLX_ROUTE_DEBUG=1 prints one line per layer naming the attention branch taken and every gate input.
• Warmup failures log their full traceback, capture-commit refusals announce their reason, the n-gram staging path and TensorOffset paged adapters carry engagement counters, and memory guard actions ride a ring buffer into the dashboard snapshot.

Server and API

• Streaming endpoints send a : keep-alive comment every 5 seconds until the first token (#358), so strict client and proxy timeouts (Claude Code, Cursor, Open WebUI, nginx, cloudflared) stop dropping healthy long prefills. MTPLX_SSE_HEARTBEAT=0 disables it and MTPLX_SSE_HEARTBEAT_INTERVAL_S tunes it.
• The Anthropic bridge (/v1/messages) forwards reasoning_effort for every family; the bridge had previously dropped the field.
• /health reports family-true sampler defaults along with the memory plan, pressure level, and guard events.
• Constructable models always run: MTP is an accelerator, not a gate. A checkpoint without an MTP head serves autoregressive with an honest "mtp_off" note instead of a false refusal.
• The app-capabilities endpoint advertises the scheduler modes the app may offer, and the experimental hyper chassis stays operator-only.

App

• Each conversation streams on its own turn stream (#324), so switching mid-generation no longer blanks or loses the reply.
• Explicit performance settings win on every serving target (#325), including "Handle multiple at once" on native Chat.
• Custom Hugging Face models the engine reports runnable are accepted (#359). Completeness is judged by the source repo's own manifest, and the failure banner names what is actually missing.
• Onboarding lists both Flash-Next packs on machines that fit them, detects existing downloads, and verifies the terminal command through your login shell, because the app's own PATH could certify a stale Homebrew install as current.
• The inference settings panel shows the family's real sampler when the engine is stopped, instead of a generic 0.6 and a "Custom model" label. With the engine running the panel always reflected the server.
• Update-dialog release notes are readable in dark mode (#367), this page included, and they render through the same template the rehearsal kit uses.
• The Memory Detail card attributes the n-gram table only when it is actually resident.
• The Live tab's acceptance panel lights within a second of a finished request even when a stream frame was missed, and the idle warm ladder no longer replaces the last real request's receipt in the dashboard.
• The memory banner says whose pressure it is. When another process pushes the Mac into memory pressure, the banner names system pressure and notes that nothing was evicted, instead of blaming the engine for it. The engine's own ceiling events keep their own wording, and the banner only claims memory was freed when the guard actually shed something. The dashboard snapshot carries the pressure source and the allocator fraction on every guard tick.

CLI

• mtplx run, mtplx chat, and mtplx ask gain --reasoning-effort with server-identical resolution, and the terminal chat gets /effort <level> to change it mid-session, the same way /reasoning works.
• The terminal chat has real line editing: arrow keys, in-line edits, and a history that persists across sessions at ~/.mtplx/history. Arrow keys used to print escape junk into the prompt.
• Prompts pipe in: echo "..." | mtplx run reads the prompt from stdin, and a piped prompt into the chat entry point answers through the same path --prompt uses. An empty pipe still refuses with the usual hint.
• Help stops hiding shipped commands: serve, run, and chat appear in the main help, and list, remove, config, env, dashboard, and integrate get their own "Server and scripting" group. A typo now gets one "Did you mean" suggestion across every registered command.
• mtplx --version prints mtplx 2.10.0 instead of mtplx 2.10.0 (2.10.0), and mtplx hardware stops telling humans "hardware acceleration confirmed: false" when the truthful reading is that this command does not measure it. The JSON documents are unchanged.
• mtplx connect opencode actually writes the config it prints; it was print-only, which surfaced as "Unexpected server error" in OpenCode.
• Daemons and tool children start Python with -P, so a source checkout or vendored mtplx/ folder in your launch directory can never shadow the installed runtime.
• The model-not-cached hint names the real command, mtplx pull <model>.
• mtplx inspect and the registry gate classify official Nemotron-H configs the same way serve routes them (#341), so inspect and serve cannot disagree.
• mtplx forge verify --stamp boots a model on the real serve path, measures it, and writes its runtime contract in place, so a family-compatible but unverified folder becomes a verified model without rebuilding or copying anything.

Fixed

• Forge routes official NVIDIA Nemotron-H configs by deriving the MTP pattern from mtp_layers_block_type (#341), and load no longer crashes.
• The dynamic-offset paged verify kernel had never compiled since its introduction and crashed at q_len above 5 once compiled. Both are fixed, and the kernel stays behind its opt-in gate pending a serve verdict.
• A model shipping without KV-geometry metadata (MTPLX_DENSE_KV_BYTES_PER_TOKEN) shows itself in the serve log instead of silently mis-budgeting dense decode.
• QSA cache state joins the cache contract, which fixes an OpenCode crash at 3.7k context from the family bring-up.
• Family-name collisions are fenced: Qwen4-preview pack names can never claim the dense-27B serving contract, and partial downloads classify as the family instead of an anonymous fallback.
• Serving profiles allowlist the family lane env keys, so a Flash-Next serve cannot boot into an unsupported-key error.
• The public Qwen 3.8 draft-depth cap stays at 3. Depths 4 through 8 were tested and lost, and an experimental raise was reverted before ship.
• A server launched with --temperature 0 couples its draft to greedy even when a client omits the temperature field. The resolver used to read the missing field instead of the effective value and leave drafts sampling at the family default, which quietly collapsed acceptance to roughly two thirds of its healthy rate. Requests that set a temperature explicitly were never affected.
• The CLI lanes that generate without the daemon (one-shot run, the terminal chat, tune) apply the same greedy coupling. Under --temperature 0 they used to keep the pack's sampled draft, which cost the same acceptance collapse on exactly the lane outside benchmarks run.
• mtplx remove refuses any ref that resolves outside the models cache and confirms before deleting, with --yes for scripts. A bare ., /, or .. used to resolve into the cache directory itself or the whole MTPLX home and delete it without a word.
• A malformed config.toml no longer bricks every command with a raw traceback. The CLI prints one line naming the file and the problem, then runs on defaults; a bad value for one key degrades that key only.
• The memory banner attributes pressure only when it can prove the source. Ties between system pressure and the engine's own allocator now name the allocator, and when the probe cannot read the allocator at all the banner stays neutral instead of assuring you the engine is steady.

Experimental, off by default

Gated lanes for operators. Everything here defaults off and changes nothing unless exported.

• --scheduler-mode hyper: the single-user manufactured-concurrency chassis (H0), trajectory-identical to serial in every gate pair.
• MTPLX_NAX_TILE_ROUTE (with MTPLX_NAX_TILE_BLOCKS): the first M5 tensor-unit attention kernel at decode shapes, +34 to +45 percent over the scalar wide-QL kernel at QL9 in gated walks.
• MTPLX_GQA_PACKED_WIDE: wide-QL packed verify routing for q_len 5 to 16.
• MTPLX_ADAPTIVE_DTEMP: acceptance-EMA adaptive draft-temperature controller.
• MTPLX_CCOPY_BANK_ROUTE (with MTPLX_CCOPY_PROF): copy-speculation dispatch through the compiled verify bank, plus its profiler.
• MTPLX_PAGED_TAILMASK_ELIDE: the resurrected paged kernel's mask-elide fast path.
• MTPLX_FORKEV_TELEMETRY: shadow pricing for speculative fork expected value.
• MTPLX_DELTA_TELEMETRY: correction-rank telemetry on every rejection.
• MTPLX_FRSPEC_DRAFT and friends: pruned draft-vocabulary prototype.
• MTPLX_QSA_FLASH: block-sparse flash-skip lane for sparse-attention decode, long-context verdict pending.
• MTPLX_DEFER_REPAIR_EVAL: lazy post-rejection repair forward.
• MTPLX_NGRAM_RESIDENT=1: wire the full n-gram table in RAM, a bench pin for 160 GB and larger machines; the streamed default meets or beats it.
• MTPLX_FUSED_MOE_DECODE, MTPLX_FUSED_GDN_OUT, MTPLX_FUSED_QSA_QKV: wired but measured slower or flat on this silicon, kept for future re-tests.

Updating

• App: Sparkle offers 2.10.0 automatically, or download the DMG at mtplx.com.
• CLI: pip install -U mtplx or brew upgrade mtplx.