MTPLX 2.8.0

This release is about trust. Over the last week people started benchmarking MTPLX seriously and running long agentic sessions against it, and both groups found real problems: sessions that quietly stopped reusing their cache past 38k tokens, stats that could disagree with what the engine actually did, and a first-run path with sharp edges. 2.8.0 closes 45 commits of that work. The rule for the whole release was simple: every number the server reports must be the number the engine lived, and every documented path must work on the first try.

The headline: long agentic sessions stay warm

If you drive MTPLX from OpenCode, Claude Code, Pi, or any agent that resends its transcript every turn, this is the release to take.

• The session cache ceiling near 38k tokens is gone (#255, #269). The post-turn commit estimated an oversized snapshot and skipped both the write and the frontier update, so the committed frontier froze around 38k tokens and every later turn re-prefilled the whole transcript. Sessions past that point paid minutes of prefill for a turn that should have taken seconds. The oversized case now takes a zero-byte live-reference lease at the full frontier, so the next turn prefills only what is actually new. Thanks to @kmike for the receipts that pinned the plateau at exactly 38,335 tokens.
• Session commits byte-extend again (#269). The commit path built its banked prefix without the committed-reasoning substitution the next request actually sends, so commits failed every turn with retokenized_prefix_not_extending_session and cache reuse sat at 3 to 4 percent. Commit and gate now share one canonicalization choke point, proven end to end with the real Qwen 3.8 tokenizer.
• The canonicalization gate cannot inject the wrong reasoning. It refuses on tool-call changes and dropped turns instead of substituting by position, handles OpenCode's stripped preambles, and repair re-encodes preserve committed reasoning. Two system contracts moved from prefix to suffix so flipping them no longer re-prefills the whole context.
• AR mode joins the session bank (#246). --no-mtp runs now restore warm prefixes and report real cache stats, so a speculative-versus-plain control arm measures decode alone. Before this, the AR arm silently paid a full re-prefill every request and its numbers were unusable as a control.

For anyone benchmarking MTPLX

We want MTPLX measured, so 2.8.0 makes the measurement surface honest and hard to misread.

• Prompt scoring for KL quality harnesses. /v1/completions with echo: true, logprobs, and max_tokens: 0 scores a whole prompt in one call. The arrays follow OpenAI echo semantics exactly: every array has length n, token_logprobs[0] is null, the scored token always appears in its own top-K map with its true value (string collisions included), and a token_ids array gives stable identity when display strings collide. top_logprobs[0] is an empty dict rather than null because several public harness parsers iterate entries and crash on null.
• Over-context requests fail loudly. A prompt that cannot fit returns a clear 400 with context_length_exceeded instead of silently generating one token, and a fitting prompt whose max_tokens exceeds the remainder is clamped with the clamp visible in stats. No more phantom rows at the long end of a context ladder.
• finish_reason is truthful everywhere. A length cap beats tool_calls in non-streaming chat, /v1/messages maps max_tokens before stop_sequence before tool_use, the completions stream trims stop strings identically to non-stream, and a capped thinking row either recovers its content or says why it is empty (content_empty_reason: truncated_inside_reasoning).
• Streamed text equals non-streamed text. The stream leaked the blank line the model emits after its thinking block as a content delta, so diffing the two transports at temperature 0 always mismatched. Streamed content now concatenates to exactly the non-stream text.
• AR responses report honest numbers. No fabricated draft temperature on any lane, batched AR included, and the post-response bookkeeping forward pass is no longer billed into measured time on either the AR or the MTP lane.
• Repetition-guard stops stay off the wire. All streaming lanes hold back a detector-window tail while the guard is armed, so trimmed loop output never reaches a client, and a triggered stop is visible in public stats.
• /health reports degradation. A new degradation block says when compiled verify fell back to eager and why, which profile env keys an operator override beat, and the kernel bail counters. mtplx doctor prints the compiled-verify fence, including in --summary. "Looks like turbo, runs slow" is no longer invisible (#255).
• Richer per-response stats, stamped only when they apply: finish_reason, draft-sampler policy and ownership, greedy coupling, repetition-stop, content-empty reason, and clamp stats. The benchmarking guide documents response caps, thinking-off settings for capped harnesses, and the exact prompt-scoring contract.
• Bench entries stop paying hidden costs (#261). Metal memory caps and the over-context refusal apply to every bench, ladder, one-shot, and quickstart entry, rows flush as they complete, and compiled-verify prewarm happens outside measured rows and warms the exact traces real rows use. Thanks to @ArthurOstapenko for the report.

First run and every run after

• Sharing the API over your network is one line. mtplx serve --host 0.0.0.0 --api-key-file ~/.mtplx/api-key creates the key file if it is missing (0600, printed once) instead of crashing on the exact command our own error message suggests, and startup prints a Network OpenAI API Base URL with your Mac's LAN address, which is what a Parallels or VM guest should use. Keyless non-localhost binds still refuse. We are not shipping an open LLM port; we are shipping a working path to a keyed one.
• Quickstart leads with Auto. The wizard's first choice is now "Auto (recommended)", which pins nothing and lets the engine resolve the fastest verified profile per model. A previously saved wizard default of sustained migrates to Auto once; deliberate picks stay pinned. The macOS app's Auto likewise stopped emitting a profile flag, so renamed and legacy model folders no longer launch pinned to the slow profile.
• No surface claims sustained for a flagship anymore. Forge stamps, model listings, doctor, bench suites, tune, and the quickstart download branch all report what serve actually resolves.
• Branded local builds resolve by id. A forge-built pack under its bare folder name no longer makes quickstart demand a 20 GB re-download of a model it already has.
• Discover shows every MTPLX build. Case-insensitive name matching, no more slicing the top 30 by downloads before filtering, and a 100-row page, so the Qwen 3.8 family and community builds actually appear.
• Config values are real pins. A profile or sampler value in config.toml is honored as explicit in both directions and startup prints one line saying where it came from.
• Model identity comes from the artifact (#268). Family resolves from forge provenance first, and the 3.8 marker is boundary-guarded so stock Qwen/Qwen3-8B cannot be claimed by it. The macOS app now uses the same guard and the same provenance-first order. Thanks to @mmmugh for the original report and the marker analysis.
• KV quantization actually saves memory. The q8 mirror is offset-sized and released once the kernel path engages, q4 never allocates one, numerics route once per request, and the CLI text states the honest contract. Paged q8 decode runs through a dedicated kernel with counters in stats.
• The launch environment cannot steer requests. MTPLX_CLIENT is an observability label; client-specific behavior requires per-request evidence. An anonymous benchmarker's settings are honored as sent, even against an app-launched daemon. Claude Code's user agent is now recognized for observability.
• Streams end honestly. The wait between last content and the finish frame is bounded with live heartbeats and a watchdog, explicit cancels emit a terminal frame and [DONE], and client disconnects are tagged as disconnects.
• /v1/messages conformance. Parallel tool use, streamed usage accounting, and strict rejection of previously ignored fields.
• reasoning_effort: "high" maps up the engine's real ladder instead of silently using the default; unknown values return 400.
• The web chat UI names whose context cap it shows. A memory-capped launch reads "this server's 16.4k context window", not "the model's".
• Faster streaming under load. The SSE hot path uses a loop-fed queue and a constant envelope, cutting per-token server overhead at high decode speeds.

Still open

• With reasoning off, in a plain chat with no tools, Qwen 3.8 can still emit a stray tool call and end the turn early. Leave thinking on. Unchanged from 2.7.1.
• A stream that ends in a tool call can still carry a trailing blank line ahead of the tool call in some chunkings. Cosmetic; non-stream responses and plain-text streams are byte-exact.
• Dense 27B models serve concurrent requests through a serialized MTP queue by default. That is a deliberate trade: serialized MTP wins prefill-heavy agentic loads end to end, batching wins short decode-heavy loads, and --scheduler-mode ar_batch opts into the latter. A speculative batched lane for dense models is on the roadmap.

Upgrading

• CLI: pip install -U mtplx or brew upgrade mtplx.
• App: Sparkle will offer 2.8.0 (build 2008000), or grab the DMG.
• No breaking API changes. Two response-shape notes for harness authors: top_logprobs[0] on echo scoring is now {} instead of null, and streamed chat content no longer begins with the blank line that followed the model's thinking block.