MTPLX 2.1.0

This release closes out the 2.x memory reports, fixes the agent session cache end to end (including on hybrid models and across daemon restarts), fixes the app startup hang, and ships two new native backends. Most of it started as community reports and source-level diagnostics. Thank you.

Numbering note: this ships everything that was staged as 2.0.3 plus the cache completion work on top. Together it grew past a patch, so it lands as 2.1.0. If you are on 2.0.2, everything below is new.

Memory (#150)

• The MLX allocator cache is bounded by default. MLX's own cache limit tracks the memory limit (about 0.75x RAM high water), so freed transients accumulated for the process lifetime. That was the dominant bloat mechanism on 96GB and larger machines. Now RAM-tiered (2 to 8 GiB) out of the box; --mlx-cache-limit to override, off to opt out. Diagnosis by @mmmugh.
• New --memory-budget flag (env MTPLX_MEMORY_BUDGET): one knob that scales the session cache budget and the allocator bound to fit a declared RAM envelope.
• The per-session admission cap is re-clamped on machines under 96GB. The v2 auto-sizing rule silently raised the gate compared to v1.0.4's flat 8GiB, letting 64GB machines admit snapshots whose restore transients blow past physical RAM. Found by @ArthoPacini.
• The paged KV pool no longer grows past --context-window. The geometric growth step used to allocate blocks no request could address at 100k+ contexts.
• Fixed a q4 crash on the paged split-SDPA path under kv-quant (the accumulator was sized to the dequantized head dim instead of the packed storage dim). Reported by @ArthoPacini.
• The memory pressure responder is redesigned: edge-triggered with hysteresis, and it defers to an idle engine instead of tearing down the allocator mid-decode every 10 seconds.
• The SSD session cache writer has a byte-bounded backlog and an hourly write budget, which closes the disk-write half of #144.

Agent and chat sessions (#121, #159, #144 read side)

• Warm prefix reuse now survives every tool turn. Chat templates render the last assistant turn differently from earlier ones, so every tool turn used to break the token prefix and prefill skip decayed toward 24% as sessions grew. Fixed end to end: recurrent-state boundary records thread through every store, inherit across entries, and the near-prefix restore lane runs whenever it beats the exact match. In replay validation, skip stays at 83 to 87% through hostile history rewrites (previously 52% and falling). Telemetry from @mmmugh and other reporters on the thread shortened this one.
• Near-prefix restores on hybrid (recurrent attention) models no longer collapse to the oldest retained boundary. Retention was tail-biased, so after enough turns a follow-up that diverged late could restore at around token 2,048 and re-prefill 20k+ tokens, felt as 30 to 50 second waits in agent front-ends. Retention is now geometric by distance from the tail and contract-tested. Live measurement on a 21,785 token prompt: the follow-up reused 21,760 tokens and started in 0.4s, against 33.8s for the same prompt cold.
• Sessions restored after a daemon restart keep their boundary records across SSD generations, so a new session over a shared system prompt restores warm instead of silently going cold (#159). Live measurement: 99.9% of the prefix restored from SSD after a real restart, first token in 2.0s.
• Every response now reports the session cache hit in standard usage fields: usage.prompt_tokens_details.cached_tokens (OpenAI shape) and usage.cache_read_input_tokens (Anthropic shape).
• Session bank entry caps raised from 8/16 to 24/48. Tool sessions store about 3 entries per turn; bytes still bound the memory.

Sampling (#156)

• Presence and frequency penalties were silent no-ops in the batched AR lane (quickstart chat and --scheduler-mode ar_batch): the sampler never saw token history. Fixed with exact parity against the serial lane. Small models that lean on repetition penalties are the visible beneficiaries. Report, diagnosis, and PR by @mmmugh, with a convergent PR by @SuperMarioYL.

Performance

• The model-owner thread is pinned to USER_INITIATED QoS: 8 to 10% faster decode under heavy background load (Electron apps, compositing), and exactly flat when the machine is idle. MTPLX_GENERATION_QOS to override.

App

• Fixed the startup and update hang (#158). Runtime installs run off the main thread, every subprocess wait has a deadline watchdog (a wedged pip now surfaces as an error instead of freezing the app), and the python probe is bounded. The reporter's spin dump made this a fast find. A follow-up sweep watchdogged every remaining subprocess wait and closed two more main-thread beachball sites.
• The Hermes tile launches Hermes Desktop when it is installed, with the terminal flow as fallback, pinned to the MTPLX backend profile.
• Raw <tool_call> XML no longer leaks into chats that have no tools configured, including the unclosed-tag shape small models emit (#160). Reported by @FilterJoe.
• Public app builds now fail loudly if the runtime wheel or the bundled Python is missing instead of producing a hollow bundle.

CLI

• mtplx start opencode serves the same engine lane the app serves. The in-decode cache-trim cadence is aligned (256 to 1024, measured flat or better on every promoted 27B artifact) and the banner no longer prints the pre-resolution profile.

Models and backends

• New qwen3_5_mtp native backend: Qwen3.5 and 3.6 MoE MTP exports load and draft correctly (trunk shim, sidecar key remap, sanitize double-shift fix). This also fixes forge build for that model type (#147). From PR #142 by @PhilipJohnBasile.
• New hy_v3 (Tencent Hy3) native backend, also from @PhilipJohnBasile. It activates automatically once mlx-lm ships models/hy_v3 in a release.
• Hermes streams no longer show tool preambles. PR #153 by @shiftedx.
• Dependency ranges: transformers <5.14, !=5.13.0 (5.13.0 breaks mlx-lm imports) and mlx >=0.31,<0.33 (0.32 verified byte-identical on the exactness gate).

Operators

• MTPLX_COMPILED_VERIFY_MAX_CONTEXT is honored as a user env override on profiled daemons, matching the other compiled-verify knobs, so you can A/B the compiled verify window without patching site-packages.

Known issues and what is next

• Long generations at 30k+ context on 27B-class hybrids settle into a verify-cost plateau around 33 to 35 tok/s on M-series. That is kernel work and it is queued for the next performance release.
• Context-copy drafting (PR #151 by @lBroth) is planned as the headline of that release.
• Hermes Desktop opens every session with its own stock prompt of about 22k tokens (client-side, independent of the profile's system prompt, which the Hermes terminal path does honor). First turns there pay that prefill once per session. We are raising it upstream.

Credits

@mmmugh, @ArthoPacini, @gcstang, @PhilipJohnBasile, @shiftedx, @SuperMarioYL, @FilterJoe, @lBroth, and everyone who retested builds and sent diagnostics. A lot of this release is your work.