Changelog

Changed

• One install, one dev extra. pip install mlx-optiq ships every command: convert, serve, eval, lora, kv-cache, latency, lab, code, game. The lab, game, cli, convert, all, vlm and audio extras are gone, kept as empty aliases so old install lines still resolve. mlx-optiq[dev] carries the test suite, browser tests, benchmark datasets and the torch/mlx-vlm oracles.
• Install footprint 985 MB -> 550 MB, shipping more by default than the old 591 MB base.
• Linux and Windows work for the API-driven path. optiq code and optiq lab against an OpenAI-compatible base_url never import mlx.core; the OS classifiers say so now. Quantizing and local inference still need Apple Silicon.

Removed

• scipy (98 MB) and data-designer (122 MB): declared core dependencies that nothing imported.
• mlx-whisper: zero usages, audio pipelines are unsupported.
• convert_vlm_to_mlx(): dead code, and the last runtime reference to mlx_vlm.

Fixed

• datasets and langdetect moved to [dev], 217 MB reached only by optiq eval and HF calibration mixes. A missing one now names the extra instead of raising a bare ModuleNotFoundError.
• Install hints in optiq lab / optiq game pointed at extras that are now empty aliases, so following them installed nothing.
• The docs advertised mlx-optiq[serve] and mlx-optiq[eval], which have never existed.
• pygame is core, so optiq game works from a plain install.

Added

• optiq game (hidden command): plays Neural Drive, a 131M diffusion world model that generates every frame of a racing game at 384x192, quantized to 86 MB with mixed 4/8-bit. Weights pull from mlx-community/neural-drive-optiq-mlx on first run; the window needs pip install 'mlx-optiq[game]'.
• optiq.core.shardwise: requantize a checkpoint shard by shard without building the model. optiq convert instantiates a model to quantize it, which a 300B checkpoint cannot survive on consumer hardware; this holds one tensor at a time, so peak memory tracks the largest tensor rather than the model (~12 GB against a 167 GB source on DeepSeek-V4). Existed only as one-off scripts in scripts/, which do not ship. requantize_shardwise(src, out, policy=moe_bit_policy(...)).
• moe_bit_policy(): the structural bit allocation for a large sparse MoE — routed experts lowest, attention and shared expert higher, router never quantized.
• optiq game: hidden command. Drives Neural Drive, a 131M latent-diffusion world model quantized to 86 MB, which generates every frame from the last 7 plus your controls. Weights download from the Hub on first run.

Fixed

• shardwise: a mixed-precision source is now read per layer. The config's global bits is only a default, and a mixed source carries per-layer overrides beside it (Hy3-Alis-MLX-Dynamic has 799). Reading only the global values fails on the first tensor that differs, which would have made every OptiQ quant unreadable by OptiQ's own requantizer.

Added

• Low-bit range-search encoder (optiq/core/lowbit.py), wired into optiq convert. Each group's quantization range is chosen by search instead of its raw min/max. Applies to affine layers at 2 and 3 bits by default; 4-bit and above are byte-identical to before. --low-bit-search/--no-low-bit-search, OPTIQ_LOWBIT_SEARCH, OPTIQ_LOWBIT_SEARCH_MAX_BITS.
• hy_v3 (tencent/Hy3) architecture. Vendored from oMLX (Apache-2.0); stock mlx-lm ships hunyuan.py and hunyuan_v1_dense.py, neither of which is hy_v3.

Fixed

• Sensitivity: MoE experts were silently excluded on the bf16-streaming path. A layer absent from the bf16 index is skipped, and LFM2.5-8B-A1B ships experts unfused as feed_forward.experts.{e}.{w1,w2,w3} against one fused switch_mlp. All 66 fused projections were skipped, never reached the allocator, and kept the default 8-bit: 8.38 bpw from a 4.5 target, twice the size of its own uniform-4-bit baseline. Now 5.15 bpw with experts at mixed precision. Affects any MoE whose source stores experts per-expert under feed_forward.
• Sensitivity: param_count measured the packed weight when the reference model is already quantized, understating quantized layers by 32/bits.

Fixed

• Sensitivity: param_count measured the packed weight when the reference model is already quantized, understating every quantized layer by 32/bits. The allocator then spent the budget on them. LFM2.5-8B-A1B came out at 8.38 bpw from a 4.5 target, twice the size of its uniform-4-bit baseline, while reporting 5.17. Affects --reference uniform_4bit on architectures that mix quantized and unquantized layers.

Added

• DeepSeek-V4 family (deepseek_v4): Flash/Pro architecture with per-layer Local / Compressed / SparseCompressed attention, hyper-connections and DeepSeekMoE. Vendored from mlx-lm PR #1192; not merged upstream.
• LFM2.5 (Liquid AI) family quants: 230M, 350M, 1.2B-Instruct, 1.2B-Thinking (Capability Score 54.14), 1.2B-JP-202606, 2.6B (Capability Score 35.19).
• deepseek_v4: chat template ported from the base repo's encoding_dsv4.py, written by the converter when the source ships none. Pinned against DeepSeek's own encoder.
• Sensitivity: the uniform-4-bit reference can stream off SSD, so it need not fit in RAM. OPTIQ_STREAM_REFERENCE=1|0 forces either path.
• IFEval reports instruction-level accuracy alongside prompt-level.
• optiq code: token usage reported, including cache hits and reasoning tokens.
• optiq convert: streaming convert for fp8/large MoE sources, allocation-aware and one block at a time.
• optiq code: replace_lines tool, addressing an edit by the line numbers read_file prints so only changed lines are sent.
• optiq code: read_file numbers its lines and its header carries a content tag ([name#a1b2: lines A-B of N]); pass the tag to replace_lines and a stale edit is rejected.
• optiq code: edit_file recovers from an anchor differing only in trailing whitespace, indentation, quote style, or copied N: prefixes. Every rung still needs exactly one matching site.
• optiq code: the stream is cut when a model starts writing its own tool result, and the fabricated text discarded.
• optiq code: nine harness thresholds are now config fields. 21 settings to 29.
• optiq.toolcalls: one tool-call parser shared by the BFCL eval and the agent loop, covering all eight family formats.

Fixed

• IFEval's loose metric was a second strict score: it cleaned the response once and scored all-or-nothing. Now builds the published eight response variants and passes an instruction if any satisfies it. Capability Score is unaffected (it reads prompt-level strict).
• Eval ran LFM2.5 without the repetition_penalty 1.1 its model card and generation_config.json both specify. Both lfm2 and lfm2_moe now carry it.
• optiq convert: the source's generation_config.json keys are preserved for every model, not just three hardcoded families. LFM2.5's do_sample was being dropped.
• optiq code: the agent loop recognised only fenced JSON tool calls, so the Pythonic, Gemma-4, Mistral and Laguna formats the eval already handled were invisible to it. It now recognises all of them, and more than one call per turn.
• optiq code: compaction could grow the context, because a dropped tool result was replaced by a placeholder longer than many results. Added a size floor and shortened the placeholder.
• optiq code: _solution_sig hashed *.py only, so every grounding mechanism silently switched off on a JS/Go/Rust repo.
• optiq code: repeated edit_file failures nudged a full-file rewrite, which fed straight into output-token truncation. The nudge now points at replace_lines.
• optiq code: empty tool results are compacted first and not protected by recency.
• optiq code: search uses ripgrep when installed, as its comment had long claimed.
• optiq code: context_window() no longer raises on a client built without a server.
• optiq convert: a VLM whose weight prefix is missing from _MM_PREFIXES no longer ships as a silently text-only quant with its vision_config stripped. Weights that are neither a known tower nor a language weight now raise and name the prefix to add.
• optiq convert: the IncompleteSnapshotError fallback added in 0.4.8 was unreachable; a local_files_only call bypassed the wrapper entirely.
• tests/model_matrix.py: mistral, laguna, nanbeige and mage-vl had docs pages but were never added, so every eval sweep silently skipped them. Both Nemotron 3 base ids were missing their -BF16 suffix and did not resolve.

Fixed

• optiq code: auto_approve was a CLI-only parameter. -y reached the TUI directly and never went through load(), so it could not be set in config.json or the environment, while every other setting obeys flag > env > repo > user > default. Unattended drivers had no way to say "don't stop and ask", so the TUI halted before every write_file/edit_file/bash waiting for a keypress that never came: the model kept talking (four turns, 22.6k tokens in the case that found it) while not one file was written, which reads exactly like a hung generation. ApprovalPolicy.gated had the same shape and is now gated_tools, so the middle ground is reachable (set it to bash to let edits through but still ask before running commands). A bare invocation passes None rather than False, so the flag cannot stamp over a config that asked for auto-approval.
• optiq code: guidance truncation was silent. fh.read(8000) cut AGENTS.md/CLAUDE.md mid-sentence and said nothing, and since guidance is prepended to every run, a long file lost its tail on every turn forever with no way for the author to notice. It now returns the overflow alongside the text and the loop logs it. 8000 chars stays the default: guidance costs ~2k tokens on every request, about 6% of a 32k local window, so the cap earns its place and the fix is telling you, not raising it blindly.
• optiq code: four values behaved like settings but could not be set (server_ready_timeout, server_host, server_port, and the guidance cap). All four now go through the same config chain, taking the surface from 16 fields to 21. server_ready_timeout matters most: a large model loading cold can exceed 180s, and that failure reads as a broken install rather than a slow one.

Fixed

• Licences for vendored third-party code were missing from the wheel. The package ships a trimmed subset of mlx-vlm (27 files, MIT, (c) 2025 Prince Canuma) and MTPLX (112 files, (c) 2026 Youssof Altoukhi), but [tool.setuptools.package-data] listed only data extensions, so setuptools silently dropped vlm/_mlxvlm/LICENSE, runtime/mtp/LICENSE.MTPLX and runtime/mtp/NOTICE.MTPLX while happily including the code they cover. Both licences require the notice travel with the source. Present from 0.4.10; absent in every release up to 0.4.9.
• OptiQ's own LICENSE file did not exist. pyproject.toml declared MIT and the classifier claimed MIT, but no licence text shipped anywhere, so the wheel asserted a licence it never included. Added at the repo root, wired in via license-files, and it now names the vendored third-party components.

Added

• OptiQ Lab chat: video attach. A clip is sampled into frames in the browser (<video> + canvas, no upload and no ffmpeg) and sent as image parts, which is how a VLM reads video. Defaults: 8 frames per clip, frames capped at 640 px on the long edge (~1.9k vision tokens for a 16:9 clip), stills downscaled to 1280 px, 16 images per turn, 512 MB per file. The composer shows the estimated vision-token cost before you send.

Fixed

• Image and video through optiq serve crashed for Mage-VL (AttributeError: 'Model' object has no attribute 'language_model'), so the documented image/video path was broken for that family in both the CLI server and the Lab. OptiqEngine._init_from_runtime assumed every VLM nests its text tower under .language_model; Mage-VL's adapter flattens it (sanitize remaps model.language_model.* -> model.*), so the loaded model is the language model. Now falls back to the model itself. Regression test in tests/test_engine_flat_vlm.py.

Changed

• optiq code: max_retries 6 -> 3 and request_timeout 900s -> 1200s. The two multiply: the SDK retries after the first attempt, so the old pair allowed 7 x 900s = 105 minutes of retrying, with nothing logged in between. A server whose generation thread has died (a Metal OOM leaves the process up and /v1/models still answering 200) was therefore indistinguishable from a hang -- observed as a 95-minute stall on a 24 GB M4. The new pair bounds the worst case at 80 min while raising the per-turn ceiling to 20 min, which a 26B on that machine needs: a single agent turn has taken ~840s.

Added

• Mage-VL VLM family (video + image): microsoft/Mage-VL, a Mage-ViT vision tower + Qwen3-4B LM. New optiq/vlm/mage_vl/ (MLX vision tower validated bit-exact vs torch, Qwen2VL-style preprocessing, frontend) and the mage_vl -> qwen3 LM adapter. First video model in the lineup; video is uniform sampled frames (the base repo's DCVC neural codec is optional and not required). Capability Score 70.27.

Fixed

• optiq code: every entry point now defaults max_tokens to 16384. run_agent, run_headless, Engine.chat and the TUI each hardcoded 8192, so a direct call got half the configured budget; all now import DEFAULT_MAX_TOKENS from config.
• optiq convert: a finished quant no longer aborts at the final save with IncompleteSnapshotError. The lean download skips non-weight files (.gitattributes, asset images, config backups) that mlx-lm's strict local_files_only re-resolve then rejected. The download wrapper completes the snapshot on demand, and convert now installs it. First hit on a Qwen3.5-VLM (Qwythos-27B).
• BFCL eval: models whose chat template silently drops the tools= argument scored 0 (no_call, the model never saw the functions). The eval now detects the missing tools in the rendered prompt and injects them into the system message with the call format. Mage-VL went 0 -> 88.5%.

Added

• OptiQ Lab Deep Research: a toggle in the Lab chat. a local (or cloud) model plans, searches the web, reads sources, and writes a cited Markdown report (draft-centric TTD-DR). A live research-trace card shows the plan, each source as it's read, the search count, and citations repaired; the report renders in the artifact pane with a short answer in chat.
• Deep Research citation grounding: verbatim-quote extraction, a verify-and-repair pass that rewrites over-reaching claims against those quotes, and a Sources list generated in code so [n] markers always resolve. Fair citation-support (union-of-cites) roughly doubles vs a plain draft-and-cite pipeline.
• OptiQ Lab artifact pane: a download button that saves a research report as .md and a model-authored HTML app as .html.
• laguna architecture (poolside Laguna-XS/S/M MoE) vendored in optiq/mlx_lm_patches/laguna.py; no upstream mlx-lm class. Sigmoid MoE with shared expert (layer-0 dense), QK-norm, GLM partial-rotary, softplus per-head attention gate, hybrid full/sliding attention with dual RoPE (YaRN for full, plain for sliding) and per-layer head counts (full 48 / sliding 64).
• optiq code TUI slash commands: /help, /init, /model, /compact, /clear, /quit (/exit, /q aliases). Input starting with / is a client command, not a goal; unknown commands hint instead of being sent to the model. Typing / opens a live autocomplete menu (filters as you type, Tab completes). /init analyzes the repo and writes an AGENTS.md; /compact drops old tool output on demand (same primitive as automatic compaction).
• optiq code reads CLAUDE.md as a project-guidance fallback (after AGENTS.md), so a repo already set up for Claude Code works with no extra file. /init reads any existing AGENTS.md/CLAUDE.md first and folds it in.
• optiq code interaction (Claude-Code parity): Esc interrupts a running turn without leaving the session; Shift-Tab cycles the approval mode (approve-each ↔ auto-approve-all, shown in the status bar); ↑/↓ walk input history; Ctrl-C copies a transcript selection if one is active, else clears the line, else (pressed twice) exits.
• optiq eval --skip-kl: skip the KL step in --task all. KL loads the candidate and a reference model at once, so a large quant on a memory-tight Mac is OOM-killed; the Capability Score is the 6 benchmarks, KL is a separate diagnostic.
• optiq code: live thinking for reasoning models — the status line shows ✻ thinking · Ns · <tail of the reasoning> as it streams, so you see progress instead of a frozen screen. It's ephemeral (status line only), so verbose reasoning never crowds the transcript.
• optiq code: the model's message text is rendered as Markdown (headings, lists, syntax-highlighted code), uncapped (was clipped to 300 plain-text chars); and the status bar shows cumulative cost on paid cloud endpoints.
• optiq code: @-file mentions — type @ to open a repo-file autocomplete (tracked files via git ls-files), Tab completes the path. Shares the autocomplete menu with slash commands.
• optiq code: web_search + web_fetch tools (Claude-Code-style WebSearch/WebFetch) — the agent can look things up (DuckDuckGo) and read a page as markdown, reusing the Lab's implementation. Read-only (no approval). ddgs + html2text moved to core deps so they work out of the box; graceful message if absent. (Running code / your app already worked via bash, like Claude Code — no separate sandbox.)
• optiq code: multiline prompt — the input is now a TextArea, so a paste keeps its newlines. Enter submits; Shift+Enter / Ctrl+J insert a newline; ↑/↓ move between lines and recall history only at the top/bottom edge; the box grows with its content.
• optiq code: /resume — opens an ↑/↓ picker of past conversations in the current repo (Enter to load, Esc to cancel), switching the active session so further turns append to it; /resume N still loads directly. In-session equivalent of -r/-c, like Claude Code's /resume.
• optiq code: double-Esc rewind — press Esc twice (idle, empty prompt) to open an ↑/↓ picker of the conversation's turns; the chosen turn and everything after it is dropped and its goal restored to the prompt to edit and resend. Jump back to any point, not just the last turn (Claude-Code style).
• optiq code: commands and rewind/resume always return focus to the prompt — with the mouse released a lost focus would strand you needing Ctrl-C to quit.
• optiq code: native text selection by default — the TUI no longer captures the mouse (mouse=false, new config knob), so click-drag select + copy works with no modifier, like Claude Code. /copy (alias /yank) puts the agent's last reply on the clipboard. Set mouse: true to restore click-to-focus and wheel-scroll (then Option-drag to select).

Changed

• optiq code: a plain text reply with no tool call ends the turn, like a chat — the agent answers a question (or finishes) and stops, no done tool required (done is now optional). Previously the loop treated "no tool call" as an error and nagged the model to keep calling tools, so a question was answered inside a hidden done summary. An empty reply with no tool call is still treated as a stall.
• optiq code: the status bar shows ↑ input tokens, a context gauge (ctx N% (used/window)) that drops when /compact or auto-compaction frees context, and cost (which never displayed before — _report_usage hardcoded 0.0; it now reads the endpoint's prompt_tokens/completion_tokens/cost).
• optiq code no longer dumps a combined diff at the end of a turn (interactive or headless), matching claude -p. Edits are shown inline as they land and left in the working tree (nothing committed); the turn ends with a one-line summary of how many files changed. Review with your own git diff.

Fixed

• optiq code: the agent's reply is now shown. When the model finishes with done(summary=…) — which is how it answers a question — the summary was hidden, so asking "what does this repo do?" printed only ⚙ done with no answer. The summary now renders as the reply (live and on resume).
• optiq code: the end-of-turn diff reflects only the files the agent changed, tracked from its write_file/edit_file calls. It diffed the whole folder against HEAD, so a project with pre-existing uncommitted work reported e.g. "40 files changed" for a turn that changed nothing (a question). An answer-only turn now reports no changes.
• optiq code: no more baseline: 0 passed / 0 failed and done — 0 passed / 0 failed noise in repos with no tests; the end-of-turn line shows tests/files only when there were any (a question shows just ■ done).
• optiq code: Shift-Tab (cycle approval) updates the status bar only — it no longer writes a line to the transcript on every press.
• optiq code: internal harness nudges (You did not call any tool…) no longer render as ▶ user turns on resume, and no longer seed ↑ history.
• optiq code: git operations are scoped to the launch folder (-- . pathspec). Run in a subdirectory of a larger repo (a project inside a monorepo), the end-of-turn diff and the status bar showed the whole repo's changes, and git add -A -N marked intent-to-add on every untracked file across the entire repo. Now confined to the folder the agent works in.
• optiq code: a cloud endpoint set only in the config file (base_url in ~/.optiq/code/config.json or <repo>/.optiq/code.json) now drives attach. It was read from $OPTIQ_BASE_URL alone, so a config-file-only setup tried to spawn a local optiq serve and exited before ready.
• optiq code: repo-tier config (<repo>/.optiq/code.json) for api_key, context_window, reasoning_effort, request_timeout, max_retries now reaches the engine — it re-loaded config without the repo, dropping that tier.
• optiq code: the interactive TUI now honors max_tokens and max_tool_output from config; only headless did, so interactive used the 8192 default and truncated large edits.
• optiq code: the [context] compaction line no longer prints above the interactive TUI (kept for headless); the same detail is available on demand via /model.
• optiq code: transcript text selection + copy works again. An earlier focus fix made the log non-focusable, which disabled Textual's native mouse selection; the prompt now keeps focus via a selection-aware click handler instead, so drag-select and Ctrl-C copy are preserved.
• optiq eval: the --task all help now lists HashHop (the suite always ran it; only the help string omitted it).
• Sensitivity checkpoint now merges with any existing file (atomic write) so it only grows — a resume/kill mid-sweep no longer truncates it and forces re-scoring of already-done layers.
• Streaming (uniform_4bit-reference) sensitivity now handles unfused per-expert MoE sources: bases that ship experts as separate mlp.experts.{e}.{gate,up,down}_proj tensors (poolside Laguna) are stacked into the running model's switch_mlp.{proj} so the experts are scored — previously they were skipped ("no bf16 source"), left the bit budget, and defaulted to the top bit-width (a Laguna-XS mixed convert came out 6.4 bpw instead of the 4.5 target).
• Converted models bake in the transformers Mistral pretokenizer-regex fix (fix_mistral_regex) when the source tokenizer triggers it, instead of copying the buggy regex through. No-op for other tokenizers.

Added

• nanbeige architecture (Nanbeige4.2 looped transformer: shared-weight layer stack run num_loops times, one KV-cache block per loop). Vendored in optiq/mlx_lm_patches/nanbeige.py and registered into mlx_lm.models on import optiq; mlx-lm has no upstream class. Reference-only features without dedicated weights (enable_double_loop_split, loop_share_kv, enable_depth_attention) raise rather than load silently.
• mlx-community/Nanbeige4.2-3B-OptiQ-4bit: mixed 4/8-bit, 5.5 bpw.

Added

• Lab chat: use any OpenAI-compatible cloud endpoint (OpenRouter and others). Pick "cloud API" as the model source, enter a base URL + your own key, and set the model to the provider's id; the Lab proxies chat there with tools. The key stays in the browser (localStorage), never persisted server-side.
• optiq code drives any OpenAI-compatible cloud endpoint (OpenRouter and others): set OPTIQ_BASE_URL + OPTIQ_API_KEY and name the model. New OPTIQ_CONTEXT_WINDOW sets the compaction budget when the endpoint has no /optiq/context. Attaches to a given --model even when /models discovery is unavailable. Docs: /docs/code/cloud.

Fixed

• Lab chat: streamed tool calls from OpenAI-compatible cloud endpoints (OpenRouter and others) now assemble correctly. The orchestrator appended whole tool-call deltas, which works for mlx-lm's buffered stream but left cloud tool calls with empty arguments (the tool then errored on missing input). Now merged by index with concatenated argument fragments.
• optiq code accepts launch flags on the bare command — optiq code -p GOAL, -c, -r ID, --model X, -y — instead of failing with "No such option" (they previously required optiq code launch …). Matches claude -p; config / export still dispatch as subcommands.
• Docs: optiq code export selects a session with --session <id> (was shown as -r).
• optiq code: the live status token counter is now cumulative for the whole turn (elapsed + total tokens), Claude-Code-style. It reset to each sub-step's count on every read/edit/test, so a multi-step turn looked like "105s · 25 tokens". The loop now banks each model call's tokens; the TUI formats large counts as 1.4k.

Added

• mistral4 decoder (optiq/models/mistral4.py): 128-routed-expert MoE with MLA attention and a softmax router. Auto-installed in the convert and streaming-load paths, so mistral3-wrapper repos with text_config.model_type == "mistral4" no longer fall back to llama, which dropped every expert.
• Mistral family: Mistral-Small-4-119B-2603-OptiQ-2bit, an extreme mixed 2/4-bit static MoE (239 GB to 43 GB). Runs on a 36 GB Mac via SSD expert streaming, ~9-12 GB resident. No Capability Score by design; ships a Flappy Bird coherence demo.
• optiq serve: per-request thinking control via a model-id suffix, always on. A request for <model>:no-think generates with enable_thinking=False, :think forces it on, no suffix is untouched. /v1/models advertises the served model with both variants so clients (and optiq code) can discover and select them. Replaces the hidden global OPTIQ_NO_THINK=1 env var for per-session use.
• optiq code: the live token counter now estimates from streamed characters, so it climbs during a tool call instead of sticking at 1-2 (mlx-lm buffers tool calls into 1-2 deltas).
• optiq code launch --yes / -y: auto-approve every tool call (no per-edit approval prompt).
• optiq code: the TUI reveals the code of a write/edit as it lands (a few lines at a time), so you see the edit take shape instead of only a "wrote N bytes" summary. The model buffers the tool call so the file arrives at once; the reveal animates it. Also makes the terminal genuinely dynamic (for screen recordings).
• optiq code: streams the model response and shows a live status line (● working · Ns · ↓N tok) while it generates, instead of a frozen screen. The engine reassembles streamed deltas into the same response shape, so the loop is unchanged; the progress callback is throttled so per-token deltas don't flood the TUI.
• Fara family: Fara1.5-4B/9B/27B-OptiQ-4bit, Qwen3.5-based computer-use / web-agent VLMs. Per-layer quant config transferred from the published Qwen3.5-*-OptiQ-4bit quants (identical architecture), so they match the Qwen3.5 family with no separate sensitivity pass. Vision tower at bf16 in optiq/.
• llada2_moe decoder (optiq/models/llada2.py): vendored MLX decoder for LLaDA2-MoE (inclusionAI LLaDA2.2-flash), a ~103B diffusion MoE — 256 routed + 1 shared expert, bidirectional attention with a fused QKV projection, per-head QK-RMSNorm, and partial RoPE. Reuses deepseek_v3's grouped-sigmoid routing + fused experts; sanitize stacks the per-expert tensors into switch_mlp; auto-installed in the convert/streaming paths. Enables a 2-bit static quant of the diffusion family (expert-streamed like the 119B).
• LLaDA2-MoE block-diffusion decode (llada2.diffusion_generate): M2T with a confidence threshold + transfer schedule and a block-causal mask. optiq serve routes llada2_moe through it (like DiffusionGemma), so the Lab and optiq code generate correctly instead of AR gibberish.
• LLaDA2.2-flash-OptiQ-2bit and KAT-Coder-V2.5-Dev-OptiQ-4bit (a qwen3_5_moe reasoning coder, mixed 4/8-bit static).

Fixed

• attach_sidecars ships text-only when a model declares vision_config but has no tower weights (e.g. a coder finetuned off a VLM base), instead of failing the convert at the sidecar step.

Docs

• Models page and Mistral guide list the 119B flagship.

Fixed

• optiq serve --kv-config / --kv-bits quantized nothing. The hook was set on mlx_lm.generate the re-exported function rather than the submodule, and the server's BatchGenerator never calls it. Startup now logs the layers actually converted.
• Quantized batch KV crashed the generation thread on the first request: an empty cache raised in nbytes/state. The server then answered nothing and reported a Metal OOM from teardown.
• Quantized batch KV failed on turn two: extract() returned a cache without merge, which _merge_caches rejects.
• Quantized batch KV corrupted offset/left_padding, producing a zero-width mask and a broadcast failure in attention.
• --prompt-cache-bytes is budgeted from free memory instead of total RAM, so a model that nearly fills the machine no longer OOMs after a few turns (24 GB Mac, gemma-4-26B: 2.40 GB → 0.50 GB).
• optiq code reported failed runs against a healthy server; the request timeout was hardcoded at 300 s, too short for a large model on a small Mac.
• BatchRotatingKVCache.merge crashed on a zero-length cache (upstream mlx-lm: [..., -l:, :] with l == 0 slices the whole buffer instead of nothing), killing the generation thread on any sliding-window model doing multi-turn with a reused prefix. Unrelated to KV quantization; patched unconditionally.
• BFCL: Mistral/Devstral [TOOL_CALLS]name[ARGS]{...} tool calls are now parsed. Devstral-Small-2-24B scored 0/200 while tool-calling correctly in production; it scores 89.5% with the fix; published Capability Score 78.60.

Added

• KV quantization on the batch path, so batching is no longer given up for it: BatchQuantizedKVCache and MergeableQuantizedKVCache. Full-attention layers only; sliding-window and recurrent layers are reported as skipped.
• optiq serve reports its effective context window at GET /v1/optiq/context.
• optiq code: context compaction. Old tool-result content is dropped past a token budget, preserving message structure so strict-alternation families stay valid. Defaults to 80% of the server's reported window.
• optiq code config: every setting, its value, and which source set it. Settings resolve in one order — flag > environment > repo config (.optiq/code.json) > user config (~/.optiq/code/config.json) > default.
• Mistral family: Devstral-Small-2-24B-Instruct-2512-OptiQ-4bit (mixed 4-bit, Capability Score 78.60, +1.95 vs uniform-4). Vision sidecar kept at bf16 for image input; loads text-only under stock mlx-lm.

Docs

• Mistral family guide at /docs/mistral; models page lists the Mistral family.

Fixed

• Mistral/Devstral tool calls are no longer dropped. They end with EOS rather than a marker, which mlx-lm's state machine reports as a stop, so the buffered call was discarded (empty message, finish_reason: "stop", non-zero completion tokens). See mlx-lm#1307.
• Anthropic endpoint: tool history is translated to structured tool_calls / role: "tool" messages instead of inlined Qwen-style <tool_call> XML, so every family renders its own markup. Affected Gemma-4, GLM, Kimi and Mistral.
• Strict-alternation templates (Mistral, Devstral) no longer 404 on two same-role turns, or on a user turn directly after tool results.
• optiq code: harness nudges ride on the preceding tool result rather than a bare user turn, so the shape is valid against any endpoint.

Added

• optiq code: identical tool calls are not re-executed when the repo has not changed; an edit clears the history.
• Tool-call healing recognizes both Mistral [TOOL_CALLS] shapes.

Docs

• Mistral Vibe integration guide.

Fixed

• Tool-call arguments are normalized to a mapping before chat templating, on both optiq serve and the cluster server. Closes the two holes in mlx-lm's conversion (a client-sent object raised TypeError; an empty string reached the template as a string).
• Cluster server: the chat-template fallback no longer raises on None or content-part messages.

Changed

• All 12 published Gemma-4 quants synced to Google's canonical chat template (2026-07-09). Fixes dropped image_url parts, null arguments rendered as None, broken turn closure on consecutive assistant messages, dropped reasoning, and an error on empty messages.

Added

• Release contract: the live audit flags a quant shipping an older chat template than its first-party base.

Added

• optiq code: a terminal coding agent for models served by optiq serve — interactive TUI, headless -p mode, and session persist / resume / export (HF Session-Traces format).
• Capability contract: a resolve-rate and turn-latency gate for optiq code on a fixed task set.
• optiq code robustness: never-empty-patch salvage, edit-apply resilience, stall force-write, and tool-call format healing.

Changed

• textual and openai are now core dependencies.

Docs

• OptiQ Code product page, docs, and launch post.

Fixed

• IFEval: a null relation kwarg crashed three verifiers and failed the whole prompt; verifiers now read the correct key and tolerate null.
• IFEval: forbidden_words matches whole words, number_paragraphs splits on the *** divider, keyword_frequency counts substrings, number_bullet_lists counts only */-, two_responses requires two distinct responses, constrained_response matches as a substring, and sentence counting tolerates abbreviations.
• IFEval: language no longer passes non-English targets unconditionally (uses langdetect, with a script fallback).
• MMLU (reasoning): the A/B/C/D extractor no longer reads a letter out of filler words, and raises if the tokenizer can't resolve four distinct letter tokens.
• GSM8K: the number regex requires a digit (no lone comma) and prefers an explicit "the answer is N".
• HumanEval: no-fence completions keep their indentation and hoist model-added imports; a program that exits before check() completes is no longer a pass; a sandbox launch/reject/cancel is a harness error, not a wrong answer.
• HashHop: prefers the terminal hash over an echoed input hash.
• BFCL: rejects arguments outside the function schema, coerces single-quoted arrays, and no longer reads "1_000" as 1000.
• Diffusion MMLU: seeds the canvas and excludes capture failures from the denominator.
• optiq eval --baseline: the comparison table no longer crashes when a benchmark fails.

Changed

• Four always-pass IFEval verifiers (number_placeholders, postscript, nth_paragraph_first_word, repeat_prompt) now check the constraint instead of returning True.
• Generation budgets standardized: 512 tokens for non-reasoning benchmarks, 4096 for --reasoning.

Added

• Regression tests for all six benchmark checkers.

Fixed

• The BFCL checker could not score a correct answer as correct. It matched a predicted argument against the ground truth by walking the list of accepted values, but for an argument whose value is itself a list (a coordinate pair, a set of ids) it compared the whole predicted list against the whole list of alternatives and always missed. Every tool-calling score was understated by roughly 16 to 18 points. Fixed, and a test now scores BFCL's own ground truth against itself and demands 100%. All published quants have been re-scored and their cards, the site, and the docs corrected.
• A MoE checkpoint with unfused experts quantized to noise. The path that handles per-expert weights only fired for the fused tensor layout; an unfused checkpoint fell through and wrote random values.
• Eval died when a checkpoint carried a weight the model does not use. One tolerant loader now covers every benchmark, so an extra tensor is dropped instead of aborting the run.
• A model with no MTP head was reported as a failure. Most models have no draft head. Its absence is now the normal case, not an error.

Changed

• --reasoning decode budget default is now 4096 (was 3072). Reasoning models were being truncated before the tool call landed.
• Vision tower is listed in model.safetensors.index.json. The sidecar stays in the optiq/ subfolder; the index just points at it, so other MLX loaders can find the tower instead of failing to load. mlx-lm and mlx-optiq are unaffected (both ignore the index for the sidecar).
• Removed the dead --task ai2d and --task wer eval tasks.

Docs

• Documented the supported loaders. OptiQ quants run fully in mlx-optiq (text, vision, MTP) and in stock mlx-lm (text). Other front-ends load through their own stack, so support there depends on that stack. See the FAQ.

Fixed

• Vision was off for every model loaded from the Hub. OptiqEngine, optiq serve and the Lab each gated the vision front-end on os.path.isdir(model_path). A Hub repo id is not a directory, and naming the model by repo id is what every model card tells you to do. So all three gates were false. Vision was disabled before the sidecar was ever looked for, and the model reported "no vision sidecar" while the sidecar sat in the snapshot, downloaded and unread. Every published VLM quant was affected. All three now resolve the path through sidecar_layout.local_model_dir. Reported on the gemma-4-26B quant.
• The "no vision sidecar" message told you the wrong thing. It claimed a missing sidecar for every cause, including the case above where the file is present. It now says which of the three it is (no local copy, no sidecar, no front-end for this arch) and where it looked.
• optiq eval grew MLX's buffer pool until the machine died. MLX parks a freed GPU buffer in a reuse pool rather than returning it, and the ceiling defaults to the whole machine. On MMLU the pool reached 30.6 GB against 0.00 GB of active memory. Metal buffers do not show in RSS, so the process looked tiny while the machine swapped and the run was killed with no traceback. The harness now drains the pool once per question, as serve and the trainer already did.
• optiq eval scored dhara-250m as a decode loop. All six benchmarks decoded with bare greedy. dhara needs a repetition penalty to stay coherent; without one it answers GSM8K with 1 + 1 + 1 + 1 … to the token cap. Every dhara build therefore scored the same, and the flat line read as "a 250M model has no benchmark headroom" when it meant "the harness breaks this model". Penalties are per-architecture in optiq/eval/decode.py; models with no entry decode exactly as before.
• One bad model output could destroy a whole benchmark run. BFCL called .strip() on a predicted tool name. dhara emitted a list, and the AttributeError killed the suite after MMLU, GSM8K and IFEval had already run. A malformed prediction now scores zero, and each benchmark fails on its own without taking the other five with it.

Added

• --repetition-penalty on optiq eval. Overrides the per-architecture default. In a --baseline run the baseline decodes the same way as the candidate.

Added

• Diffusion LLM pipeline (optiq/models/diffusion.py). optiq convert reads model_type from config.json and routes DiffusionGemma to it. Before this, an installed OptiQ could not quantize one at all.
• optiq eval on diffusion models. MMLU is scored from the canvas answer slot, since a diffusion forward predicts a canvas rather than a next token.
• --candidate-bits 8,16, a keep-bf16 tier. MLX has no 16-bit quantized format, so 16 means "leave this layer at bf16". It is for small models with no redundancy to spend. A 4-bit dhara-250m stays fluent but drifts from the reference (KL 0.0222). The 8/16 build reproduces it (KL 0.0005).
• Release contract (tests/test_release_contract.py). Audits every published quant: sidecars under optiq/, vision towers at bf16, mixed precision, metadata present, diffusion never AR-calibrated. Small quants also have to clear a KL bar against bf16, which the Capability Score cannot see at that size.
• pillow as a core dependency. The vision path ships in three published families. It was previously undeclared.

Fixed

• DiffusionGemma's vision tower did not work. It described every image as abstract noise. Three causes. The towers were quantized to the floor bit-width, because text-only calibration scores them KL = 0. They sat inline in the shards instead of the sidecar that optiq serve and the Lab gate the image path on. And save_config strips vision_config, so the tower was never rebuilt on load. The towers now ride at bf16 in optiq/optiq_vision.safetensors.
• optiq convert on a text model failed with No module named 'PIL' on a core install.
• optiq eval --task all crashed at the summary when no reference could be resolved, after all six benchmarks had already run.
• DiffusionGemma sensitivity was not reproducible. Its canvas is seeded from random tokens and the RNG was unseeded, so a resumed sweep scored the remaining layers against a different input.
• DiffusionGemma calibration sampled one point of the denoising schedule. It now samples several.
• dhara's diffusion calibration probed a single mask fraction (0.5) and re-derived the block-causal bias by hand. It now sweeps the trajectory from 1.0 down to 0.25, and uses the same function the decoder attends under.

Changed

• Republished mlx-community/dhara-250m-OptiQ-8bit (8/16 mixed, 10.25 bpw) and mlx-community/diffusiongemma-26B-A4B-it-OptiQ-4bit (vision at bf16, Capability 68.25).

Changed

• Sidecars moved under an optiq/ subfolder. OptiQ ships two weight files that are not part of a model's main weight set: the MTP head (mtp.safetensors) and a VLM's bf16 towers (optiq_vision.safetensors). They used to sit at the repo root. Every resolver checks the subfolder first and falls back to the root, so models published before this keep loading. No API changes.

Fixed

• LM Studio could not load an OptiQ quant with an MTP head. It loads through mlx-vlm. That globs every *.safetensors in the folder, picks up mtp.safetensors, and strict-loads tensors the base model has no module for. It aborts with Received 29 parameters not in model. A non-recursive glob does not descend into optiq/, so the sidecars are now invisible to it.
• --max-context auto capped hybrid-attention models at 2048 tokens. The estimator counted every layer as holding a KV cache and assumed bf16, overestimating by about 11x. It now counts only full-attention layers and honors the configured KV bit-width.

Added

• optiq cluster serve. Run a model too big for any single Mac by sharding it across several over Thunderbolt. Peers are found over Bonjour. Layers are split by each node's currently free RAM, not its installed RAM. The ring is exposed to mlx-lm as one model, so streaming, tool calls, structured output and samplers all work through the same code path a single Mac uses. A preflight refuses to load rather than swap a machine to death. Measured: Qwen3.5-122B-A10B-2bit (42.8 GiB) across a 36 GB + 24 GB pair at 20.5 tok/s, loading in 26 s.
• Tiled FlashAttention-2 backward (optiq/ops/flash_attention_tiled.py), routed by memory budget. The stock fused forward is kept. Only the backward is replaced, with recomputation over query blocks. At T=16384 the stock backward is a Metal OOM and the tiled one runs. Set OPTIQ_FLASH_ATTN=auto|always|never.
• --adapt-experts. MoE expert pools are now opt-in for LoRA. They had been silently adding about 1.2 B trainable params.
• Lab: cluster status, streaming tool calls, and runnable HTML artifacts in a side panel.

Fixed

• LoRA and DPO were broken across the whole Qwen3.5 and Qwen3.6 family. gated_delta_update dispatched to a raw Metal kernel with no vjp, so the GatedDeltaNet layers had no gradient. Both entry points are now routed.
• Flash attention made training 6-26x slower than not using it. The hand-written Metal kernel ran one thread per query row with scalar loops, and it was applied unconditionally. A full 0.8B step at 8192 went from 367.87 s to 13.23 s.
• A clean install was dead on arrival. transformers 5.13 breaks import mlx_lm. Core deps now pin transformers<5.13.

Optional LoRA-training conveniences, wired identically into optiq lora train and OptiQ Lab.

NEFTune noisy-embedding SFT (--neftune-noise-alpha)

NEFTune (Jain et al. 2023) adds uniform noise scaled by alpha / sqrt(seq_len × embed_dim) to the token embeddings during the SFT forward pass, and nothing at inference. A near-free regularizer that improves instruction-following on small datasets. Gated on the embedding's training flag, so validation loss is still measured on clean embeddings (mlx-lm flips the model to eval mode around validation). Off by default (0, matching TRL/Unsloth whose neftune_noise_alpha defaults to None); the paper suggests 5–15. SFT only, mirroring TRL, which applies NEFTune in its SFTTrainer and not the DPOTrainer.

Early stopping (--early-stopping-patience)

Halt training once the validation loss hasn't improved for N consecutive evaluations (evals fire every --steps-per-eval), so a run that has converged doesn't burn the remaining iterations. --early-stopping-min-delta sets how much improvement counts as progress. Off by default (patience 0); requires a validation set (no-op with a warning otherwise). On stop, the best-validation adapter is promoted to the returned adapter. The trailing steps that tripped the counter are worse than the best. You keep the good checkpoint, not the last one. OptiQ already snapshots the best under <adapter>/best/; this also makes it the top-level file. Works for both SFT and DPO.

Experiment logging (--report-to)

Stream train/val metrics to Weights & Biases (--report-to wandb, needs pip install wandb + WANDB_API_KEY) or swanlab, reusing mlx-lm's own reporting callbacks. Comma-separated for several; project name via --wandb-project. The tracker runs alongside the normal CLI log and the Lab live chart, not instead of them. Chained under OptiQ's best-adapter/progress hooks so nothing else changes. Off by default.

Three serving robustness features for Apple Silicon, informed by a survey of the Unsloth v0.1.48 app release (whose "safe defaults, no surprise downloads" independently matches OptiQ's 0.2.16 single-model default).

Idle auto-unload: free the model's RAM when nobody's calling (--idle-timeout)

On unified memory a served 24 GB model holds RAM the machine can't reuse even while idle. optiq serve --idle-timeout SECONDS drops the model after that many seconds with no requests and reloads it lazily on the next call (the same lazy first-request load, re-triggered). A watchdog stamps a last-access time on every request. Once idle, it detaches the provider's cached model handle. An in-flight generation keeps its own reference to the model, so a decode that overruns the timeout still finishes safely. Only the next request pays the reload. Set the timeout longer than your longest single generation. Default 0 (off, model stays resident).

Resilient downloads: Xet → HTTPS failover on stalled transfers

HuggingFace's Xet high-performance transfer is fast when it works but can stall partway through a large shard on corporate proxies, TLS-inspection middleboxes, or flaky links, hanging snapshot_download. OptiQ now retries the normal path, then, as a last resort, forces the plain HTTPS path (HF_HUB_DISABLE_XET=1) for one final attempt, resuming from cache, so completed shards are never re-fetched. Wraps mlx_lm.utils.snapshot_download, so every optiq serve / optiq convert / adapter download inherits the failover transparently, with mlx-lm's own allow_patterns intact (no extra files pre-fetched).

Memory-aware context cap: stop a long prompt from OOM-crashing the server (--max-context)

A request that runs to a big model's full native context (128k–256k) can allocate more KV than unified memory holds and kill the whole server mid-generation. --max-context auto (the default) reads the model's KV geometry and free RAM. It engages a cap only when the full native context would not fit. Machines with enough RAM see no change. Pass an integer for a hard token cap or off to disable. Uses mlx-lm's RotatingKVCache, which supports merge / trim / to_quantized, so continuous batching, prompt-cache reuse, and KV quantization all keep working. Once a prompt exceeds the cap the window rotates instead of crashing. Sliding-window models (Gemma-4, Qwen3-Next) manage their own KV and are untouched.

Website changelog

Release notes are now published at mlx-optiq.com/changelog, rendered from this file, since the source lives in a private monorepo, the site is the public record of what shipped in each PyPI release.

optiq serve is single-model by default: no more model-field 404s

optiq serve --model X hosts one model. But the OpenAI/Anthropic protocol requires a model field on every request, and stock mlx-lm reads that field as "which repo to load". So a basename, a friendly alias, or a wrong default (Claude Code's claude-… when ANTHROPIC_MODEL is unset) made it try to download that id, then 404 or hang. The server now treats the model field as a label by default: every request is served by --model X, whatever the client sends. Pass --allow-model-switch (auto-enabled by --models-dir) to opt back into stock per-request hot-swapping between cached models, where an unknown id errors instead of being pinned. This matches how the Lab already serves (one model per process, switched by restarting the server), so the Lab and Arena are unaffected. Adapter/draft fields are untouched, so per-request LoRA still works.

Mac-safe concurrency cap (--max-concurrent): stop bursts from OOM-crashing the server

Concurrent requests are already batched (mlx-lm's BatchGenerator). But each in-flight request holds its own KV cache, and mlx-lm defaults to 32 decode / 8 prefill in parallel. That is datacenter-GPU sizing, and it overruns unified memory on a Mac. A burst of requests could OOM-crash optiq serve. It now injects a Mac-safe cap: --max-concurrent (default 8) sets mlx-lm's --decode-concurrency and --prompt-concurrency (~1/4). Excess requests queue instead of crashing. Lower it for big models / long contexts, raise it if you have RAM to spare. An explicit --decode-concurrency still wins. Applied in optiq serve and the Lab API server.

Sampling-preset variants (:precise / :creative / :balanced)

The model-variant framework now carries sampling presets, not just the thinking toggle. Append :precise (temp 0, deterministic), :creative (temp 0.8 / top_p 0.95), or :balanced (temp 0.4 / top_p 0.9) to the model id. The server applies that decoding preset for the request. A client picks a decoding profile by name, without setting sampler fields. Composes with the existing :think / :no-think. All are advertised in /v1/models. Unknown suffixes pass through untouched.

Context scaling for smaller-context models behind Claude Code (--context-scale)

Agents like Claude Code auto-compact the conversation by comparing the token usage the server reports against the context window they assume the model has. That assumption is ~200k for a Claude model. Point one at a smaller-context local model and it won't compact until far past the model's real limit. The model overflows and generation fails first. --context-scale FACTOR multiplies the token counts in the reported usage, so the client's "compact at N% of the window" logic fires at the right real-token point. Set it to (window the client assumes) / (your model's context), e.g. 6.25 for a 32k model behind a 200k assumption. Only the reported usage is scaled. Generation, KV cache, and the prompt are untouched. Covers the OpenAI /v1/chat/completions path (stream + non-stream) and the Anthropic /v1/messages path (which derives its usage from the OpenAI response). In optiq serve and the Lab API server. Documented on the Claude Code integration page.

Model variants: pick thinking on/off by name (model:think / model:no-think)

optiq serve and the Lab's API server accept a variant suffix on the model id: a client sends <model>:no-think (or :think) and gets that variant. The suffix is stripped before the model is resolved and loaded. So it maps to the real model at zero extra memory, and the variant's chat_template_kwargs are applied to that request. This exposes a reasoning model's thinking toggle by name. Any OpenAI-compatible client can select it without the non-standard chat_template_kwargs body field, which many clients don't expose. The served model's variants are advertised in /v1/models.

Unknown suffixes pass through untouched, so a repo-id or local path that legitimately contains a colon is unaffected. On families whose chat template doesn't use enable_thinking (Gemma-4, VibeThinker) the variant is a harmless no-op. Wired identically into optiq serve and optiq lab's API server (the Lab chat UI already had a Reasoning toggle). Tested across Qwen3.5 / Qwen3.6 / Nemotron (toggle works) and Gemma-4 / VibeThinker (no-op).

DPO loss variants + warmup fix

• cDPO and IPO for optiq lora train --method dpo: --dpo-label-smoothing (conservative DPO for small / noisy preference data) and --dpo-loss ipo. Both exposed in the Lab finetune UI.
• DPO warmup floor for short runs: was a flat 10-step floor; now scales to ~10% of the resolved iters (capped), so tiny preference sets don't over-warm.

Long-context LoRA fine-tuning on Apple Silicon

Two memory levers extend optiq lora train to longer sequences on a Mac, where the [seq × vocab] logit tensor (vocab 248k) is the wall:

• Fused cut-cross-entropy (SFT), the LM-head loss and gradient are computed without materializing the full [seq, vocab] logits, reaching 8k context for SFT. Auto-enabled.
• --fused-dpo (DPO), DPO materializes the [B, seq, vocab] logits four times per step (policy + reference, over chosen + rejected), so the plain path OOMs around 4k on a 24 GB Mac. --fused-dpo (or OPTIQ_FUSED_DPO=1) applies the same chunked-head log-prob path, gradient-exact. Opt-in rather than auto, since the break point is VRAM-dependent. Measured on a 24 GB M4 (Qwen3.5-4B, batch 1): 2k ≈ 10 GB / ~7 min-step, 4k ≈ 11.3 GB / ~30 min-step. Past 4k it fits memory but is O(seq²)-speed-bound, so ~4k is the DPO sweet spot vs 8k for SFT.
• Exit-watchdog, resolves a final-save deadlock that could hang long (8k+) runs at checkpoint time.

Method-aware LR + epoch-based iteration defaults

OptiqLoraConfig now resolves the learning rate and iteration count by method and dataset size, so the CLI, the Lab, and direct construction all agree:

• learning_rate=None → 2e-4 (SFT) / 5e-5 (DPO) via effective_learning_rate(). This fixes a real bug. OptiqLoraConfig(method="dpo") previously fell back to the SFT 2e-4 and silently collapsed the policy. Only the CLI swapped it, so direct/Lab construction trained DPO at the wrong rate.
• iters=None → num_epochs × ceil(n_examples / batch), defaulting to 3 epochs (SFT) / 1 epoch (DPO). New --num-epochs flag and config.num_epochs field. DPO warmup keys off the resolved iters. Explicit --iters / --learning-rate / --num-epochs always win.

The Lab finetune route now routes through the same config. It had hardcoded iters=500 and never passed fused_dpo. It gains an optional epochs field and a "fused DPO logp" checkbox. Docs updated in site/docs/finetune.html.

tool_choice="required" enforcement in optiq serve

mlx-lm's server ignores tool_choice entirely. So a request that sets tool_choice="required" (or names a specific function) could be answered with plain prose, silently violating the OpenAI contract. Those values mean the API obliges the model to call a tool. Any agent that relies on required to guarantee a call got wrong behavior.

optiq serve now enforces it. When a request forces a call, OptiQ constrains generation so the model must begin with its native <tool_call> opener (a single special token for Qwen3.5/3.6). That commits it to a call, which it then completes in the normal format. Because the output is the model's native tool-call format, mlx-lm's streaming tool parser recognizes it. The enforcement holds on the streamed first turn, not just non-streaming completions. Reuses the lm-format-enforcer machinery already shipped for structured output. Falls back to a bare-JSON constraint (recovered by the tool-healer) on models without a single-token <tool_call>. tool_choice values of auto/none/absent are untouched.

Found via SeraphimSerapis/tool-eval-bench, whose tool_choice=required compliance scenario (TC-45) previously failed. The model answered "56" directly instead of calling the calculator. It now passes.

Fix: optiq serve --mtp crashed on tool-calling (and repo-id) requests

Two bugs made optiq serve --mtp unusable with a HuggingFace repo-id or with tool-calling clients. Both surfaced running the tool-eval-bench tool-calling suite against Qwen3.6-27B-OptiQ-4bit.

1. Repo-id not resolved to a local directory. --mtp with a repo-id (e.g. mlx-community/Qwen3.6-27B-OptiQ-4bit) raised FileNotFoundError: Missing config.json in mlx-community/Qwen3.6-27B-OptiQ-4bit. The MTP engine's attach() treated the repo-id string as a local path. It now resolves through a new resolve_model_dir(). A local dir with a config.json is used as-is. Otherwise the string is treated as a repo-id and mapped to the cached snapshot via snapshot_download. That fetch is a no-op when the model is already downloaded, which it always is under optiq serve, since mlx-lm loads the weights first. Serving with an explicit local snapshot path was the workaround.

2. logprobs=None crashed _serve_single. mlx-lm's server evaluates gen.logprobs[gen.token].item() on every yielded GenerationResponse. OptiQ's fast streaming loops (MTP, separate-drafter speculation, vision) don't surface a per-token logprob vector and were yielding logprobs=None. So every request through those paths died with TypeError: 'NoneType' object is not subscriptable, and tool-calling requests always hit it. A shared _NULL_LOGPROBS shim (returns 0.0 for any index) now backs all of those paths. The vision path already used it, and it's applied uniformly to the MTP and drafter paths too. The logprobs/top_logprobs response fields are placeholders on these paths (the default serving path never requests real logprobs).

With both fixed, --mtp serves tool-calling requests cleanly (verified: --mtp + repo-id + a tools= request returns proper tool_calls). MTP remains lossless speculative decoding, so this only affected the serve integration, not decoding quality.

Server-side tool-call healing in optiq serve

Quantized open-weight models often emit malformed tool calls: Hermes <tool_call> tags, fenced or bare JSON, trailing commas, fancy quotes, function-call form, the {"python": {...}} key-is-tool-name pattern. mlx-lm's server leaves those in content instead of parsing them into OpenAI tool_calls. optiq serve now runs OptiQ's heal_tool_calls on the final completion message. It recovers those into proper tool_calls and promotes finish_reason to tool_calls when a call is found. Any API client (agents, Claude Code via the OpenAI endpoint) gets clean calls. It's the same healer the Lab chat already used, now at the server layer so it applies to every client. Runs on non-streaming completions, where the full content is available. Requests without tools are untouched.

Structured / JSON output in the Lab

The constrained-decoding from 0.2.7 (response_format / guided_json, via lm-format-enforcer) was serve-only; it's now wired into the Lab. The Lab server installs the structured-output path, and the Chat UI gains a JSON-mode control (off · any valid JSON · match schema, with a schema box). response_format is threaded through both the direct path and the tool-loop route. JSON mode and the tool loop are mutually exclusive: a single constrained response vs. an agentic loop. Tools are disabled while JSON mode is on.

Fix: vision sidecar shipped audio-tower convs in PyTorch layout

build_vision_sidecar copied the vision/audio towers raw out of the bf16 base (an HF/PyTorch checkpoint) without applying mlx-vlm's conv-layout sanitize(). So the published optiq_vision.safetensors stored the audio tower convolutions channel-first (PyTorch [out, in, kH, kW]) instead of MLX channel-last [out, kH, kW, in]. That is unlike every stock mlx-community gemma-4 conversion, where mlx_vlm.convert applies that transpose before saving. OptiQ's own vision/text path never touches the audio tower, and gemma-4's SigLIP patches are Linear, no conv. So OptiQ itself was unaffected. But third-party full-VLM loaders rejected the entire multimodal graph on the shape mismatch and silently fell back to text-only LLM (reported on gemma-4-e4b-it-qat-OptiQ-4bit).

The sidecar builder now transposes subsample_conv_projection.*.conv.weight (Conv2d) and *.depthwise_conv1d.weight (Conv1d) on store, idempotently, matching mlx-vlm exactly. In practice the wrong layout shipped on the QAT SigLIP models (gemma-4-e2b-it-qat-OptiQ-4bit, gemma-4-e4b-it-qat-OptiQ-4bit). Their optiq_vision.safetensors were re-uploaded with the corrected layout, and no language-tower re-quantization was needed. The non-QAT e2b/e4b sidecars were already channel-last. The 12B (encoder-free gemma4_unified) and the 26B/31B sidecars carry no audio-tower convs, so they were unaffected.

Fair evaluation of reasoning models: optiq eval --reasoning

The Capability Score harness silently mis-scored always-on reasoning models (models that emit a <think>…</think> block before every answer and have no toggle to turn it off). Two failure modes. The generation tasks budget ~256 tokens, which truncates a multi-hundred-token think trace before the answer ever appears. And MMLU is scored by first-letter logit argmax (the answer token right after "Answer:"), which collapses to chance for a model trained to reason before answering. The result was a strong model looking near-random.

--reasoning fixes both. It lets the model think, with no suppression. It gives the generation tasks a large per-question budget: --reasoning-max-tokens, default 3072. That is a cap, not a fixed length, since greedy decoding stops at EOS so short traces cost nothing. It strips the <think> block before extracting the answer. And it scores MMLU generatively (generate with thinking, then parse the answer letter) instead of by logit argmax. Default eval behavior is unchanged. The flag is opt-in.

The 3072 default was calibrated on VibeThinker-3B (think-trace length median ~1160, p95 ~3050 tokens). On that model the difference is stark:

This makes the Capability Score usable for the whole class of thinking models going forward, not just VibeThinker.

Multi-turn agentic DPO

DPO now scores full {messages} trajectories. When chosen/rejected are message lists, the loss is taken over every assistant turn via the same multiturn.assistant_mask the multi-turn SFT path uses, not a single prompt→completion boundary. Single-string pairs keep the old behavior. The gated-delta (qwen3_next linear-attention) and full-attention Metal kernels are routed through the DPO forward/backward. Without them, DPO's four passes (policy+ref × chosen+rejected), stacked with the SFT+DPO LoRA, blow the Metal 499k buffer-count cap. Same kernels the SFT trainer uses, scoped to compatible shapes.

Structured / JSON-constrained output in optiq serve

The OpenAI-compatible server now honors response_format (json_object and json_schema) plus the vLLM-style guided_json / guided_regex / guided_choice extensions. When a request carries one, OptiQ masks the model's logits each step so it can only emit tokens that keep the output valid for the schema, regex, or choice set. This makes the served model reliable as an agent/tool-call backend: the JSON it returns parses, the enum answers stay in range, the regex always matches.

It is deliberately built on lm-format-enforcer, not xgrammar. lm-format-enforcer is pure-Python (pydantic + interegular) with no PyTorch dependency, so OptiQ stays MLX-native. The constraining logit processor adds about 1 ms/token. It is wired into mlx_lm.server's generation with a few small patches. Free-form requests (no response_format) are untouched. For reasoning models, OptiQ auto-disables thinking when a spec is present. The grammar already forbids non-JSON text, so a <think> block would only mislabel the constrained output as reasoning instead of content.

curl localhost:8080/v1/chat/completions -d '{
  "messages": [{"role":"user","content":"A person: Alice, 30."}],
  "response_format": {"type":"json_schema","json_schema":{"schema":{
    "type":"object","properties":{"name":{"type":"string"},"age":{"type":"integer"}},
    "required":["name","age"]}}}}'

Model switching + local-quant listing for coding harnesses

optiq serve already inherits mlx-lm's behavior of hot-swapping to whatever model a request asks for. It also lists every MLX model in the HuggingFace cache via /v1/models. So a coding harness (Claude Code, Codex, Cline) pointed at the server can already enumerate downloaded models and switch between them per request.

New: --models-dir DIR also advertises locally-built quants (e.g. optiq_output/<name>) that were never pushed to the hub. They show up in /v1/models and are switchable by passing their path as the request model. The served model, the local-dir quants, and the cache models are all listed together.

gpt-oss support: harmony-format evaluation

OpenAI's gpt-oss is now a supported family. Its quant evaluation needed real work: it's a reasoning model with two traits that break a naive eval harness. It emits the harmony response format, an analysis channel of chain-of-thought before a final channel with the answer. And it controls reasoning length through reasoning_effort, not the enable_thinking flag the Qwen path uses. The new optiq/eval/_harmony.py reduces channel-structured output to just the final answer before extraction (GSM8K number, MMLU letter, HumanEval code block). concise_template_kwargs probes the chat template and applies reasoning_effort='low', so the model answers within the token budget instead of truncating mid-thought. GSM8K went from 50% to 80% on a 20-question smoke once this landed. gpt-oss's tool calls arrive in a commentary channel (to=functions.NAME … <|message|>{args}), now parsed by the BFCL extractor (0% → 70.5% on the full set). HashHop's generation budget is raised for reasoning models so the 16-character hash isn't cut off mid-analysis (0% → 78%). The calibration loader also falls back to plain role-prefixed concatenation when a strict chat template rejects a sample. gpt-oss's harmony template raises on a tool turn that isn't preceded by an assistant tool_call.

The published mlx-community/gpt-oss-20b-OptiQ-4bit posts a Capability Score of 71.84 vs 53.81 for a genuine uniform-4-bit baseline (+18.03, winning all six benchmarks, HashHop +59.0, MMLU +16.7, BFCL +11.0). The margin is unusually large because gpt-oss ships natively in MXFP4: its experts are already 4-bit. Uniform-4 re-quantization compounds the loss, while OptiQ keeps the retrieval-critical attention and router at 8-bit.

Re-quantizing natively-quantized (MXFP4) experts

mlx-lm loads gpt-oss's MoE experts as QuantizedSwitchLinear (native MXFP4). nn.quantize skips them because they're already quantized, so a sub-4-bit target would pass the experts straight through unchanged. optiq.backends.mlx_backend now dequantizes pre-quantized fused-expert tensors back to bf16 before re-quantizing them to the OptiQ-assigned bits. It processes one expert slice at a time. A fused expert tensor is ~1 B elements on a 120 B model, which OOMs if dequantized whole and trips the Metal command-buffer GPU timeout if quantized in a single op. The path is a no-op for ordinary bf16 sources. (Converting a 120 B-class MXFP4 source this way still needs a Mac with more RAM than the source. ~60 GB exceeds a 36 GB machine at load time.)

SSD streaming covers SwitchGLU expert naming

is_streamable_moe and load_streaming now recognize fused routed-expert tensors stored under .mlp.experts. (Gemma / gpt-oss SwitchGLU), in addition to .switch_mlp. (Qwen3 / Nemotron SwitchMLP). The SSD expert-streaming path extends to those architectures.

Quantization methods: optiq (default) and static

optiq convert --method now selects how the per-layer bit allocation is derived. optiq (default) is the exact calibration-driven logit-KL sensitivity, the gold-standard signal. But it costs n_layers × n_bits × n_samples forward passes plus a uniform-4-bit baseline in RAM. On a large model that means hours of convert and tens of GB of memory. That is the cost that drove some users back to old 0.0.x releases. static is the fast path: it assigns bits by structural rules. Embedding and output head, the first and last block, attention and the MoE router get the high bits; the dense MLP and routed experts stay low. That feeds the same allocator at the requested candidate bits and target BPW. No calibration, no forward passes. It loads the base lazily (names and shapes only, never resident), so it allocates a 100 B-class model on a 36 GB Mac in seconds.

Measured on Qwen/Qwen3.5-0.8B (GSM8K, 200 q, target 5.0 BPW): optiq 899 s / 34.5 %, static 7.2 s / 34.5 %. Static matches optiq at 125× the convert speed and a lower BPW, because for a typical transformer, architecture alone is most of the sensitivity signal. optiq still earns its cost when you want certainty or are characterising a new family. static is the practical path for the very large bases where exact KL is impractical. That includes 2/4-bit mixed quants of 100 B-class MoEs (e.g. Qwen3.5-122B-A10B) that run on a laptop through SSD expert streaming. See the methods comparison.

SSD expert streaming: run large MoE quants that don't fit in RAM

A mixture-of-experts quant holds every expert resident even though only top-k of N fire per token. So a model like Qwen3.6-35B-A3B-OptiQ-4bit (21 GB) hard-OOMs on a 24 GB Mac. optiq.runtime.moe_stream streams the active experts' weights off SSD instead: load_streaming(...) runs that model at ~4.5 GB resident, decode ~8.8 tok/s (with concurrent reads, below), output coherent.

How it works. Lazy mmap slicing of the fused expert tensor doesn't bound memory, because MLX materializes the whole operand. So we read only the active expert rows at the byte level with safetensors.get_slice. The uint32 expert weights (the bulk) stream. The small bf16 scales/biases stay resident. A StreamingQuantizedSwitchLinear reads the unique active experts, remaps the router indices, and runs mx.gather_qmm over that compact set. That is bit-identical to the resident op. Active-expert counts are bucketed to a few fixed sizes so MLX doesn't recompile the kernel every call (prefill is ~40x faster with this). Resident params are loaded by raw byte range, not mmap. MLX mmaps each shard as one Metal buffer, and the resident scales/biases are scattered across every shard, so an mmap load would pin all ~22 GB against the Metal budget. Reading just each tensor's bytes keeps the load footprint to the resident size. That needs a manual bf16 bitcast, since safetensors can't read bf16 directly.

Wired into optiq serve (--stream-experts/--no-stream-experts, default auto, streams only a MoE too big to fit resident) and the Lab (Settings → Server → "SSD expert streaming"). The streamed model is built on the main thread at startup and handed to the server. mlx-lm's lazy model construction OOMs for a large MoE when run inside the request handler. optiq serve --stream-experts on Qwen3.6-35B-A3B answers requests at ~4.5 GB resident on a 24 GB Mac. Decode is slower than resident, since experts are read per token. Only MoE models are affected.

Decode reads the active experts with concurrent pread: parallel positional reads off a shared fd, queue depth 24. That is the big throughput lever. On Qwen3.6-35B-A3B it takes decode from 3.3 to 8.8 tok/s and prefill from 0.8 to 26.5 tok/s. A 400-token agentic generation finishes in ~49s at ~6 GB. Speculative prefetch (warm the previous token's experts during compute) is implemented but OFF by default (OPTIQ_STREAM_PREFETCH=1). When the model fits the OS page cache it adds redundant work and slightly regresses decode. It only helps when the model is far larger than RAM.

Approach validated against SharpAI's SwiftLM (MLX SSD streaming for 100B+ MoE): same residency split, same concurrent-pread design. They independently found an in-app expert cache regresses throughput vs. the OS page cache, matching our cache-off default.

Convert: reuse a pre-placed uniform-4-bit baseline

optiq convert --reference uniform_4bit now reuses an existing _uniform_4bit_baseline in the output directory instead of always rebuilding it. Drop in a published mlx-community/<model>-4bit (or a prior run's baseline) and the convert skips the bf16-quantize step. That step is the memory-heaviest part of the pipeline, and it can swap-stall on large models. Faster and more reliable for 27B+ on a 36 GB Mac.

Live tokens/sec in chat

The Lab chat view shows a decode-speed readout while the model streams. It keeps the readout on the finished message, matching the Arena. The counter starts at the first token, so it reflects steady-state generation rather than prompt-processing latency.

Lab polish

• The publish flow auto-stamps the mlx-optiq funnel banner onto quant-card READMEs at push, and the Hub surfaces each quant's Hugging Face download count.

OptiQ Lab: Arena, Hub, chat-with-files, Canvas, self-healing tools

Five additions to the local Lab UI, all reusing the existing serving plumbing.

Model Arena (/arena) loads two models and answers the same prompt side by side with tokens/sec for each. Model A runs on the main server. Model B runs in a second ApiSupervisor on port + 1, started on demand. Ideal for seeing an OptiQ mixed-precision quant against a uniform one, or a quant against its bf16 base, in real time. Both models are resident at once. So it is best used with small/fast models.

Hub (/hub) browses published OptiQ quants (auto-discovered from mlx-community), searches Hugging Face for MLX models, and lists locally-converted models. Each one gets one-click load-to-server and new-chat actions.

Chat with files (lightweight RAG with citations). Attaching a non-image file now indexes it instead of dumping the whole document into the prompt. A dependency-free BM25 retriever pulls the chunks most relevant to each question. It prepends them with [n] citation markers, and the Lab renders a sources panel under the answer. Only the retrieved chunks enter the context.

Canvas. Assistant messages that contain a fenced html block render live in a sandboxed allow-scripts iframe next to the answer. A view-source toggle sits alongside it.

Self-healing tool calling. On top of the existing malformed-call healing, a per-call retry budget stops the model from re-running the same failing tool call indefinitely. It gets 3 attempts, then it is told to change course. The tool card shows a retry limit badge when the budget is spent.

dhara-250m: the Diffusion LLM family's second model

OptiQ now ports, quantizes, and serves dhara-250m, a custom 250M tri-mode model and the second member of the Diffusion LLM family. dhara is not loadable by stock mlx-lm. It adds Canon depthwise-conv layers, QK-norm applied after RoPE, and a logit soft-cap. OptiQ ships a bit-exact mlx-native port (optiq/mlx_lm_patches/dhara_ar.py) that registers with mlx-lm. The whole pipeline, convert, LoRA, eval, serve, KV-quant, works through the autoregressive path. The Canon convs, QK-norm, and soft-cap are not Linear modules, so they stay at bf16 automatically. Only the attention and MLP projections are quantized.

It decodes three ways from one set of weights: autoregressive (the accuracy reference), self-speculation (--mtp, the recommended default), and block-diffusion. Self-speculation drafts a block in one forward and AR-verifies it (two forwards per round, prefix-cached, no commit pass). It emits output identical to autoregressive greedy decode while committing ~3–4 tokens per round. That is ~1.4× faster than token-by-token AR at the same accuracy on an M3 Max. Block-diffusion is prefix-cached too (O(block) per step, not O(sequence)).

dhara is built to be fine-tuned on a specific task (like Google's Gemma-270M). The win is size: 460 MB → 170 MB. The model is overhead-bound. 4-bit and bf16 decode at the same speed. The quant buys memory and disk, not throughput.

DiffusionGemma: OptiQ's first diffusion LLM

OptiQ now quantizes, runs, fine-tunes, and serves Google's DiffusionGemma-26B-A4B-it. It is a block/masked-diffusion image-text-to-text model that decodes by iteratively un-masking a fixed 256-token canvas instead of left-to-right. It is the founding member of a new Diffusion LLM family.

DiffusionGemma is not loadable by stock mlx-lm or mlx-vlm. So OptiQ vendors a trimmed, dependency-free decoder (a subset of mlx-vlm 0.6.3, MIT) under optiq/vlm/_mlxvlm/. Core stays mlx-lm-only, the same principle as the vision sidecars. The OptiQ pipeline measures per-layer sensitivity on the denoising-canvas logits (KL(uniform-4 ∥ layer-at-8bit)). The greedy knapsack then moves the 8-bit budget off the dense-MLP, where mlx-vlm's hand-coded recipe puts it. It goes instead onto the early-layer attention + routers the measurement flags as more sensitive. At the same ~4.66 bpw, the OptiQ quant posts a 59.90 Capability Score vs the published 4-bit's 59.84 (MMLU +2.9, HumanEval +1.2), while being 0.5 GB smaller.

What works on DiffusionGemma:

• Text + image generation, optiq.vlm.diffusion_gemma.{load, generate} (generate(..., images=[PIL.Image])). Image preprocessing reuses the Gemma-4 SigLIP path, bit-exact to mlx-vlm.
• LoRA fine-tuning, train_diffusion_lora trains with the model's native denoising objective, not autoregressive cross-entropy. The objective corrupts the target to a random noise level and predicts the clean tokens. mlx-lm's tuner can't load or correctly train a diffusion model; this can.
• optiq serve, auto-detects diffusion_gemma and routes mlx_lm.server through the vendored decode. It uses the fast confidence-threshold sampler (4.6–5× faster than the model's entropy-bound default), retrying on the occasional empty canvas.

Not applicable (diffusion is non-autoregressive): MTP / speculative drafting and KV-cache quant. The parallel canvas un-masking is the native analog of MTP, and the fixed canvas means the KV cache only holds the prompt.

Also fixes a convert bug where quantize_model dropped vision_config from the saved config. The vision weights were present but never loaded, silently text-only. Multimodal config keys are now preserved.

Published: mlx-community/diffusiongemma-26B-A4B-it-OptiQ-4bit.

Friendly error when the Lab extra is missing

optiq lab without the optional lab extra installed used to fail with a bare ModuleNotFoundError: No module named 'flask' (then jwt, then argon2 if you installed them one at a time). It was a confusing dependency-chase that looked like a broken package. lab_cmd now catches the ImportError and prints exactly what to run: pip install 'mlx-optiq[lab]'. The message notes that only the Lab web UI needs the extra, and that serve / convert / eval / lora / kv-cache all run on the core install. Pure UX. No behavior change for anyone who already has the extra.

optiq serve no longer requires the Lab extra

A core install (pip install mlx-optiq, no [lab]) followed by optiq serve raised ModuleNotFoundError: No module named 'flask'. serve runs on mlx_lm.server and does not use flask. But it reaches optiq.lab.mlx_cleanup for RAM cleanup, and the optiq.lab package __init__ eagerly imported the flask app. So loading any optiq.lab.* submodule pulled flask in at import time. The Lab app import is now lazy (optiq/lab/__init__.py via __getattr__), so serve and every other non-Lab command runs on the core dependency set. Only the Lab UI itself needs the [lab] extra. A subprocess-isolated regression test (tests/test_core_imports_without_extras.py) imports the full serve graph with flask and the other Lab-only deps blocked. A core install can't regress again. Latent since v0.1.0; never caught because dev/release machines always have the Lab extras installed.

Vision (image) support on Gemma-4 and Qwen3.5/3.6

OptiQ answers image+text prompts on the Gemma-4 and Qwen3.5 / Qwen3.6 families. The language tower is still OptiQ mixed-precision quantized and decoded by mlx-lm. The vision tower runs from a vendored encoder (no mlx-vlm runtime dependency), kept at bf16 in a sidecar (optiq_vision.safetensors). Its features are spliced into mlx-lm's decode through the input_embeddings hook. Because mlx-lm selects weights with glob("model*.safetensors"), it never sees the sidecar. The same published repo loads as a text-only model under stock mlx-lm and as a full image+text model under OptiQ. No separate vision quant; vision and audio stay bf16 (the ecosystem norm, since int4 vision hurts OCR and fine detail).

Image input now works on every Gemma-4 size and on the Qwen3.5 / Qwen3.6 family. The Qwen path vendors the Qwen3-VL vision tower (Conv3d patch embed, windowed attention with 2D rope, patch merger) and the Qwen image processor (smart-resize + merge-grouped patchify) into optiq/vlm/qwen3_5/. Both reproduce mlx-vlm's pixel values and vision features to max|Δ| = 0.0. Because Qwen's ViT carries its own 2D positional encoding, sequential backbone positions are sufficient (OCR comes out exact). No mRoPE patch to mlx-lm is needed. Validated end-to-end on Qwen3.5-0.8B at 4-bit (OCR, counting, shapes, colors, spatial, scene description), with MTP, LoRA, KV-quant, and serving all unaffected.

Gemma-4 gained two more vision arches beyond the e2b/e4b SigLIP tower. The first is the 26B/31B SigLIP variant (27 layers, no clip scalars). The second is the 12B gemma4_unified model, which is encoder-free (a light patch embedder feeding the shared backbone). The 12B needs bidirectional attention over the image-token span (use_bidirectional_attention=="vision"). A one-shot wrapper on gemma4_text._make_masks provides it (text/decode stay causal), which fixed OCR exactly. All five Gemma-4 quants and the small Qwen3.5 quants (0.8B/2B/4B/9B) now carry the optiq_vision sidecar on Hugging Face.

optiq.vlm resolves the right front-end per model_type (gemma4, gemma4_unified, qwen3_5). The sidecar build (build_vision_sidecar) downloads only the shards that hold vision/audio weights. Attaching a sidecar to a large quant doesn't pull the full bf16 base.

Gemma-4 SigLIP path

Numerically exact. Feeding mlx-vlm's own pixel values through the vendored path reproduces its vision_tower and embed_vision outputs to max|Δ| = 0.0 (266 soft tokens on gemma-4-e2b). Two correctness details the vendoring had to get right. First, mlx-lm's gemma4_text always rescales input_embeddings by embed_scale, so the vision features are pre-divided by it. Second, the per-layer inputs zero out the image-token positions before projection.

Serving and the Lab. optiq serve and the Lab's API server auto-enable image serving when the model ships an optiq_vision sidecar. Newer mlx-lm routes requests through a batch path that bypasses the old streaming hook and drops non-text content. So image requests are tagged on the per-request args at submission, forced onto the single-request path, and handed off inside the synchronous generation thread (no cross-request image leakage). The Lab Chat tab takes image uploads directly: attach an image, ask a question, get the analysis. On gemma-4-e2b at 4-bit, counting, shapes, colors, OCR, and chart reading all answer correctly.

OptiqEngine.generate / generate_stream take images= (or a full messages= list with image_url parts) for the multimodal prefill. MTP, LoRA, KV-quant, and the text path are unchanged. The vision path only runs when a request actually carries an image.

LoRA on Nemotron-H hybrid models

optiq lora train failed on NVIDIA Nemotron-3 (Mamba2 + attention hybrid) with "Could not locate transformer blocks". The trainer only looked for model.model.layers / model.language_model.model.layers. Nemotron-H registers its blocks under model.backbone.layers. The block-discovery now also handles backbone.layers and a top-level model.layers list, and the per-layer rank resolver matches backbone.layers.* metadata keys. The apply loop already skips blocks without target modules. So the pure-Mamba layers are left alone and adapters attach only to the attention blocks. Verified end-to-end on mlx-community/NVIDIA-Nemotron-3-Nano-4B-OptiQ-4bit.

New-model smoke test

Added scripts/smoke_new_model.py. It runs every basic OptiQ + Lab feature against a model and reports PASS/FAIL/SKIP: load+generate, LoRA, serve, drafter speculative decoding, quantized KV cache, eval smoketest, Lab app + dataset template. Run it before publishing any new quant so arch-specific breakage (like the Nemotron-H LoRA gap above, or the Gemma-4-unified loader) is caught up front.

Unified Gemma-4 (gemma4_unified) support

Google's unified text+vision+audio Gemma-4 checkpoints (e.g. mlx-community/gemma-4-12B-it) ship under model_type: gemma4_unified / gemma4_unified_text. The language tower is architecturally mlx-lm's gemma4_text; MoE blocks, double-wide MLP, global_head_dim, and attention_k_eq_v are all implemented there. And gemma4.py's sanitize() already drops the vision/audio towers. OptiQ now registers gemma4_unified → gemma4 and gemma4_unified_text → gemma4_text remappings in mlx_lm_patches. So optiq convert / optiq serve load, quantize, and serve the text path of these models. Vision/audio weights are dropped during conversion (text-inference artifact), consistent with how OptiQ already handles multimodal Gemma-4 bases.

mlx-lm requirement for gemma-4-12B: the unified gemma4_text path needs one mlx-lm commit (#1240, the shared-layer KV-projection sanitize fix) that postdates the 0.31.3 PyPI release. The PyPI wheel for mlx-optiq declares mlx-lm>=0.31.3, so it installs cleanly. For gemma-4-12B specifically, install mlx-lm main until upstream cuts a release with #1240: pip install "mlx-lm @ git+https://github.com/ml-explore/mlx-lm.git@df1d3f3c9a7aae402dcbb8f41d4c36bcc13a50ae". Every other model (Nemotron 4B/30B, Qwen3.5/3.6, Gemma-4 multimodal, MiniCPM5) works on stock mlx-lm 0.31.3.

mlx-lm pinned to main

The unified gemma4_text fields above postdate the 0.31.3 PyPI release, so the core mlx-lm dependency is pinned to a validated main commit. Bump the SHA in pyproject.toml when adopting newer upstream fixes.

Faster sensitivity sweep on the uniform-4-bit reference path

The --reference uniform_4bit sweep was probing every candidate bit-width per layer, including the one that equals the baseline's bit-width (4). That probe re-quantizes a layer to exactly what it already is, so KL(reference || current) is 0 by construction (observed as ~1e-18). OptiQ now records 0.0 for the baseline-equal bit directly and skips its forward passes. That roughly halves convert time for the common candidate_bits=[4, 8] case. No effect on the bit assignment. The optimizer only needs the improvement from upgrading 4 → 8, which the 8-bit probe still measures.

optiq eval KL reference no longer 404s on new bases

The KL eval's auto/uniform_4bit reference picker constructed mlx-community/<stem>-4bit and assumed it existed on the Hub. For a brand-new base (e.g. gemma-4-12B-it) that repo doesn't exist, so --task all / --task kl crashed with a 404 mid-suite. It now checks repo_exists first. When the published quant is missing, it builds a local uniform-4-bit quant of the base once and caches it under ~/.cache/optiq/kl_refs/ for reuse. Published references are still used unchanged when they exist.

Serve: explicit ready banner after mlx-lm hand-off

optiq serve previously ended at mlx-lm's "not recommended for production" UserWarning, with no follow-up "server ready" line. mlx-lm's own Starting httpd at … port … notice is logging.info. But a root logging handler already exists by then, configured at import by huggingface_hub / transformers. So mlx-lm's basicConfig(INFO) is a no-op and the line is swallowed. The server is up, but the warning reads like the process died. That bites hardest on RAM-tight machines, where the lazy model load then pauses the first request.

Now optiq serve forces the root logger to INFO so Starting httpd actually appears. It also prints its own banner with the resolved URL, plus a one-line callout that the model loads on the first request and the mlx-lm warning is harmless.

Eval: --baseline now works with --task all

optiq eval --task all --baseline X previously silently dropped the --baseline flag and ran the full 6-metric suite on a single model. The flag worked only with --task gsm8k. Anyone running the standard "OptiQ vs uniform-4-bit Capability Score comparison" before publishing to HF had to invoke optiq eval twice and stitch the table by hand. The cost of finding out is a full 8-hour re-run. That is what happened on the v0.1.5 Nemotron 4B convert.

Now both modes accept --baseline. With --task all, the candidate runs through the full suite first, then the baseline runs through the same suite. A two-row comparison table prints at the end:

Capability Score comparison  (candidate vs baseline)
──────────────────────────────────────────────────────────────────────
  metric            candidate      baseline    Δ (cand-base)
  MMLU                  64.0%         …             +… pp
  GSM8K                 81.5%         …             +… pp
  IFEval                56.2%         …             +… pp
  BFCL                  75.5%         …             +… pp
  HumanEval             77.4%         …             +… pp
  HashHop               27.0%         …             +… pp
  Capability            63.60         …             +…
  Disk (GB)              2.94         …             +…

With --output-json, the record nests both runs under candidate and baseline keys (single-model runs keep the flat schema).

Sensitivity: index already-fused switch_mlp routed experts (MoE bf16 sources)

The bf16-streaming sensitivity path (--reference uniform_4bit) recognized routed-expert tensors only in the HuggingFace raw layout: 3D *.experts.gate_up_proj / *.experts.down_proj. It splits and renames those to the mlx-lm switch_glu/switch_mlp wrapper. NVIDIA's Nemotron 3 Nano 30B-A3B (mlx-community/...-MLX-BF16) instead ships its MoE already fused in mlx-lm format: mixer.switch_mlp.fc1.weight [128, 1856, 2688] and mixer.switch_mlp.fc2.weight (a plain per-expert fc1→fc2 MLP, no gate/up split). Those names matched neither expert branch. So all 46 routed-expert tensors (23 MoE layers × 2) were skipped for "no bf16 source match," never probed, and never assigned a per-layer bit.

Consequence: mlx_lm.convert quantized the routed experts at its default precision instead of an OptiQ-optimized 4-bit. The fused experts dominate a 30B-A3B's parameter mass. So the "mixed" model ballooned to 8.34 bits-per-weight / 31.4 GB, nearly double the uniform-4-bit baseline (4.50 bpw / 17.0 GB). Meanwhile the optimizer's own achieved_bpw read a misleading 5.20, computed only over the layers it had actually seen. Dense models (including Nemotron 3 Nano 4B) were never affected. They have no switch_mlp.

The fix adds a branch to _index_bf16_layers. It indexes 3D tensors ending in .switch_mlp.fc1.weight / .switch_mlp.fc2.weight directly under their own module path (no split, no rename, the running QuantizedSwitchLinear is named identically). Routed experts now get sensitivity scores and proper mixed-precision bit assignment. The HF-raw .experts.* path is unchanged, so Qwen3.5/3.6 and Gemma-4 MoE bases convert exactly as before.

KV-cache: NemotronH hybrid model support

optiq kv-cache previously crashed with ZeroDivisionError on Mamba-attention hybrids like NVIDIA Nemotron 3 Nano (model_type: nemotron_h). Two problems compounded: (1) the code looked for layer.self_attn, but NemotronH names its attention submodule layer.mixer (same attribute holds Mamba2 mixers and MLP blocks too). (2) mlx_lm.utils.make_prompt_cache returns one cache slot per stateful layer, but NemotronH skips MLP layers in that list. On a 42-layer model the cache list has 25 entries, and caches[i] does not correspond to model.layers[i]. The result was that the four KVCache slots (one per real attention layer) got mapped to Mamba indices and rejected by a self_attn attribute check. Sensitivity analysis emitted zero results and the optimizer divided by zero.

The fix walks the layer list and classifies each one as attention / Mamba (SSM) / MLP via attribute probes (q_proj / qkv_proj / wq for attention; in_proj + conv1d/x_proj/dt_proj/A_log/D for SSM). It then builds an explicit cache_idx → layer_idx map by counting cache-bearing layers in order. The sensitivity loop indexes the prompt cache by cache_idx (correct slot). It reads attention metadata from model.layers[layer_idx] via a _attention_module lookup that tries self_attn → attention → mixer in order. The memory-estimate helper in run_kv_benchmark got the same _attention_module lookup, so the serve benchmark path stops assuming self_attn. Flat transformer architectures (Qwen3, Gemma-4, etc.) hit the identity mapping branch and behave exactly as before.

Concrete impact: on mlx-community/NVIDIA-Nemotron-3-Nano-4B-OptiQ-4bit the four attention layers (model indices 12, 17, 24, 32) now generate a valid kv_config.json. Three layers land at 4-bit and one at 8-bit, 5.0 average KV bits.

Optimizer: arch-aware per-block floor for Mamba / SSM / MLP-only blocks

_apply_block_aware_floor previously promoted n_floor_per_block=2 components on every block. That carve-out exists to keep late full-attention blocks from collapsing to all-low-bits. On Qwen3.5-4B v0.1.0, layers 23 and 31 had all four of q/k/v/o at 4-bit, and IFEval regressed -12.9 pp. Transformer blocks have ~7 linears (q/k/v/o + gate/up/down). Flooring 2 of 7 is a small tax.

Mamba / NemotronH-style mixer blocks and MLP-only blocks have just 2 linears each (in_proj/out_proj for Mamba, up_proj/down_proj for MLP). Flooring 2 of 2 promotes the entire block. Over 42 blocks, that's the whole model. Concrete case: NVIDIA-Nemotron-3-Nano-4B has 42 NemotronH blocks. With the old floor, 84 of 93 components were floored pre-knapsack. The achieved BPW landed at 7.48 against a 5.0 target, defeating mixed-precision.

The fix: blocks with <4 attention components (Mamba mixers, MLP-only blocks, linear-attention variants) cap their per-block floor at 1. Full-attention blocks (q/k/v/o all present) still honor the global n_floor of 2.

Result on Nemotron-3-Nano-4B:

Transformer-only architectures (Qwen3, Gemma-4) are unaffected. Their blocks already trip the full-attention branch. The patch lives in optiq/core/optimizer.py.

LoRA merge + model export: optiq lora merge and optiq lora export

Two new CLI commands that close the gap between "I trained adapters" and "I shipped a model."

optiq lora merge, rank-concat merges N LoRA adapters into a single composite adapter. The merge is mathematically exact, not a low-rank approximation. For sources with ranks r1, r2, … on a layer, the merged adapter on that layer has rank r1 + r2 + …. Its forward pass reproduces the sum of the original LoRA residuals exactly. Per-source scales are folded into the lora_b matrices so the merged adapter writes scale=1.0. Use case: after the standard SFT → DPO continuation (--mount-adapter), fold the two adapters into one drop-in artifact for shipping. Output is PEFT-compatible. Any LoRA-aware runtime can load it.

optiq lora merge ./adapters/sft ./adapters/dpo -o ./adapters/merged

optiq lora export, bundles a base model + one or more LoRA adapters into a single self-contained directory. The result is ready for optiq serve --model <dir> or mlx_lm.generate --model <dir>. The layout puts the merged adapter at the top level, so the directory loads as a single model. It preserves the original adapters under source_adapters/<name>/ for reference. It also writes an optiq_export.json audit trail recording the source adapters and the merge decision.

optiq lora export mlx-community/MiniCPM5-1B-OptiQ-4bit \
  --adapter ./adapters/humanizer-sft \
  --adapter ./adapters/humanizer-dpo \
  -o ./humanizer-1B-OptiQ-4bit

The exported directory is HF-Hub-ready: huggingface-cli upload <repo_id> ./humanizer-1B-OptiQ-4bit ships the model, the merged adapter, and the unmerged source adapters together.

The Lab UI's Finetune wizard Step 5 now exposes the same flow without leaving the browser:

• Combine with another adapter, checkbox + adapter picker that lists every local adapter (auto-discovered under ~/.optiq/lab/models, ./adapters/, and ~/.optiq/lab/adapters/). Calls the new POST /api/finetune/merge endpoint.
• Bundle as a self-contained model, checkbox + base-model field that runs the export. Calls POST /api/finetune/export.
• Push to Hugging Face, same button as before, now smart about which artifact to push (exported dir → merged adapter → bare trained adapter, in that preference order). New POST /api/finetune/list-adapters endpoint backs the picker.

DPO continuation from SFT: optiq lora train --mount-adapter

The standard alignment recipe is SFT first, then DPO continuing from the SFT weights, not "train DPO from a fresh zero-init LoRA on top of the base." The latter is what optiq lora train --method dpo did before this. It works in principle, but typically can't outperform SFT on small models. The preference signal alone isn't strong enough to reach the SFT distribution from scratch.

New flag --mount-adapter PATH (and corresponding OptiqLoraConfig.mount_adapter field). For every layer that the SFT adapter covers, the trainer constructs a StackedLoRALinear carrying:

• the base (quantized) Linear, frozen
• the SFT (lora_a, lora_b) loaded from the mounted adapter, frozen
• a fresh trainable DPO (lora_a, lora_b) at zero init

Forward becomes y = base(x) + sft_scale * (x @ sft_a @ sft_b) + dpo_scale * ((dropout(x) @ lora_a) @ lora_b). For the DPO reference forward, _set_lora_scale flips only the trainable self.scale to 0. The frozen SFT contribution stays active, so the KL term is anchored against base + SFT (= the SFT model). That is exactly the standard alignment-pipeline definition of "DPO continuing from SFT." Layers the SFT adapter didn't cover fall back to plain LoRALinear, so the rest of the OptiQ adapted-layer set still gets a DPO LoRA.

The trainable / frozen split is enforced via MLX's model.freeze() + model.unfreeze(keys=["lora_a", "lora_b"], recurse=True), so the optimizer and _save_adapter only see the DPO delta. The saved adapter is the clean delta-from-SFT. It composes with the SFT adapter at serving time via OptiQ's multi-LoRA registry (optiq serve --adapter ./sft --adapter ./dpo, request body "adapter": "sft+dpo").

MoE expert projections currently fall back to plain LoRA, stacked MoE is a separate refactor.

Mixed-precision KV cache now works on sliding-window models (Gemma-4, Cohere R2, OLMo 3, …)

Upstream mlx-lm ships RotatingKVCache.to_quantized() as raise NotImplementedError("RotatingKVCache Quantization NYI"). That single line blocked KV-cache quantization on 15 model families that use sliding-window attention. The list: Gemma 3 / 3n / 4 (text), Cohere Command R 2, OLMo 3, EXAONE 4 / MoE, Ministral 3, Recurrent Gemma, Baichuan M1, AFMoE, MiMo v2 Flash, Step 3.5, GPT-OSS, and the Llama variants with SWA.

optiq.runtime.kv.RotatingQuantizedKVCache is a drop-in subclass with affine-quantized storage: (packed_uint32, scales, biases) 3-tuples, same layout as mlx-lm's plain QuantizedKVCache. It adds the rotating-buffer trim / temporal-order logic preserved from the parent. update_and_fetch returns the tuple form. The standard quantized SDPA path runs unmodified, no custom Metal kernel.

patch_rotating_to_quantized() installs three patches at once. The first is the to_quantized override. The second is an SDPA dispatch fix for the Gemma-4 KV-sharing edge case, where one layer's quantized K/V tuples are passed to a downstream layer whose own cache=None. The upstream dispatcher routed that to the fp16 fast path because hasattr(None, "bits") is False. The third is a QuantizedKVCache.update_and_fetch wrapper that feeds a producer-cache registry, so the dispatch can recover bits / group_size when needed.

optiq serve installs the patch automatically when --kv-bits or --kv-config is set. The per-layer mixed-precision conversion now picks RotatingQuantizedKVCache vs QuantizedKVCache based on the source cache type. optiq kv-cache analysis also installs the patch and substitutes rotating quantized caches at probe time. Sensitivity sweeps run on Gemma-class models for the first time.

End-to-end verified on mlx-community/gemma-4-e2b-it-OptiQ-4bit. The full 4-element stack (OptiQ-4bit weights + per-layer mixed-prec KV + Gemma-4 -assistant drafter spec decoding + OptiQ-trained LoRA adapter) returns correct trained and general responses at 37.7 tok/s on M3 Max. Per-element wall-clock breakdown:

Gemma-4 kv_config.json shipped with all four pre-built quants

Mirroring the Qwen-side bundling done in v0.1.2-followup, each Gemma-4 OptiQ-4bit repo now carries the recommended kv_config.json from a real sensitivity-analysis pass. Point optiq serve --kv-config at it. Drop-in for the average user. The model still loads fine without the file.

Notes for users on earlier OptiQ versions

The new RotatingQuantizedKVCache runtime is required to use the bundled kv_config.json on any Gemma-4 quant. On optiq <= 0.1.2 the per-layer KV-quant path will still fall over with NotImplementedError: RotatingKVCache Quantization NYI. Upgrade with pip install -U mlx-optiq to land the fix.

The 13 Qwen / MiniCPM kv_configs shipped via discussion-thread followup against v0.1.2 are unaffected. Those families don't use sliding-window attention.

MiniCPM5 family quant (new)

Adds mlx-community/MiniCPM5-1B-OptiQ-4bit, a 1.08B-parameter Llama-architecture base from OpenBMB with hybrid enable_thinking reasoning. 875 MB on disk, Capability Score 30.28, +4.44 over uniform-4-bit. HumanEval jumps from 45.7% (uniform-4) to 57.9% (OptiQ mixed-precision). HashHop recovers from a 0% floor to 4.0% overall. Apache-2.0. Built with the same optiq convert --target-bpw 5.0 --candidate-bits 4,8 recipe as the rest of the family. Family guide at /docs/minicpm5.

New Labs dataset template: hf_dataset_import

Generic non-LLM template that pulls a public Hugging Face dataset by id. It optionally filters rows by a column value and slices to a row cap. Then it emits the chosen output format (text / messages_user_only / prompt_completion). Closes a gap vs Unsloth Studio's Hub picker and NeMo Data Designer's HuggingFaceSeedSource. Use it as the first stage of any pipeline that starts from a published corpus: EditLens, no_robots, dolly, or your own dataset.

Multi-LoRA serving: "base" sentinel for adapter bypass

optiq serve --adapter X auto-applies the mounted adapter to every request when only one is registered. You can now opt out of that auto-activation per request, without restarting the serve. The adapter (or adapters) request-body field accepts the literal values "base", "none", "off", or the empty string "". All of them generate from the underlying base model with no adapter active. That is useful for A/B comparing adapter vs base from the same served process. One case: constructing DPO preference data from with-adapter (chosen) vs no-adapter (rejected) samples of the same prompt. Unknown adapter names still 400 as before.

Multi-LoRA serving (CLI + Labs)

optiq serve --adapter now accepts the flag multiple times. One value keeps the classic mlx-lm --adapter-path boot. Two or more activate the OptiQ mounted-LoRA path, so all adapters stay resident on one base and switch per request via a ContextVar in the forward pass (no model reload between switches). The mounted-LoRA primitive (optiq.adapters.registry + mount.py) existed since v0.0.x, but had never been wired into either optiq serve or the Labs UI. This closes that gap.

The Labs Server settings page lists adapters auto-discovered under ~/.optiq/lab/models and lets you add arbitrary local paths. The Chat surface shows a dropdown that switches the active adapter per request once the running server has any mounted. A new GET /v1/adapters endpoint reports the registered names + per-layer mount counts for inspection.

DPO trainer memory fix

optiq lora train --method dpo used to peak at 75 GB on a 1B model at --max-seq-length 2048 on a 24 GB Mac. It spent an hour thrashing compressed swap before producing the first iteration. Two real bugs, both fixed:

1. Reordered the four DPO forward passes so reference forwards (adapter scale=0) materialize their scalar log-probs via mx.eval and detach via mx.stop_gradient before policy forwards run. Previously all four forwards were part of the same nn.value_and_grad-traced graph, and their intermediate (B, L, V) logits + per-layer activations sat in scope simultaneously. Now only the two policy forwards hold activations for backward.

2. Honored config.grad_checkpoint in the DPO path. The SFT trainer used gradient checkpointing for releases; the DPO trainer didn't reference the flag at all. Now mirrors mlx-lm's pattern by calling mlx_lm.tuner.trainer.grad_checkpoint on the first transformer block.

Verified on MiniCPM5-1B-OptiQ-4bit with the humanizer DPO data (767 train / 86 valid pairs, 24 GB M4 Mac). Peak physical memory drops from 75 GB to 18 GB. 300 iters complete in ~42 minutes.

DPO defaults (CLI + Labs)

optiq lora train --method dpo and the Labs Finetune wizard now apply DPO-aware defaults when the user does not pass a learning rate or warmup:

• Learning rate drops from the SFT default 2e-4 to 5e-5 (in line with Rafailov 2023 / TRL DPOTrainer). The CLI prints [optiq.lora] --method=dpo: using DPO learning rate 5e-05 when the default kicks in. The Labs UI auto-swaps the LR field unless you have edited it.
• Warmup defaults to 10% of --iters (floor 10) with linear ramp 0 → peak.
• Schedule decays cosine from peak to 10% of peak after warmup. Override with --dpo-lr-schedule constant or --dpo-warmup-iters N.

The trainer also adds two guardrails:

1. A loud WARNING if the resolved peak LR is > 1e-4 (the SFT default lands here, which collapses the policy within ~100 iters).

2. A one-shot collapse detector at the first val pass. It fires if loss ≈ 0 and both chosen_r / rejected_r are drifting deeply negative with a near-zero margin. It then points the user at the docstring's data-pairing requirement: chosen and rejected must both be plausible completions of the same prompt under the base distribution.

Default behavior for SFT runs is unchanged.

Fixes

• HumanEval dataset id. huggingface-hub 1.x rejects the legacy unnamespaced openai_humaneval id; eval now uses openai/openai_humaneval. Same payload.
• Family hints. optiq.lab.optiq_models._family_from_id and optiq.backends.mlx_backend._FAMILY_RECOMMENDED_SAMPLING now recognize minicpm5. The Lab picker shows the correct family label, and the model-author sampling recipe is preserved through optiq convert.

Bugfix release. The v0.1.0 wheel and sdist shipped only .py files. The bundled calibration mix (optiq/calibration/data/optiq.jsonl), the Lab UI Jinja templates, the Lab static assets (CSS, JS, fonts, favicon), and the MTP benchmark prompt fixtures were all missing from the distribution. A fresh pip install mlx-optiq then failed on optiq convert with a FileNotFoundError on the calibration mix. optiq lab failed on TemplateNotFound.

pyproject.toml now declares an explicit [tool.setuptools.package-data] block that pulls all data files into both the wheel and the sdist. Verified: wheel grows from 305 KB to 1.4 MB.

Also forwards the model-recommended sampling fix from 81da5ce. optiq serve no longer hands mlx_lm.server flags its argparse rejects (--repetition-penalty, --presence-penalty). Those flags had silently turned the server into a --help dump on Qwen3.5 / 3.6 quants whose generation_config.json carried those keys.

Major simplification + sharper LLM focus. Breaking changes, no backwards compatibility shim.

Headline: 12 / 12 OptiQ-4bit quants beat uniform 4-bit on Capability Score

Every shipped quant in this release beats stock uniform 4-bit on the six-metric Capability Score (mean of MMLU + GSM8K + IFEval + BFCL + HumanEval + HashHop). Range of Capability gains: +0.17 (Qwen3.5-27B) to +13.57 (gemma-4-e4b-it). HashHop is the new 6th benchmark. It exercises long-context multi-hop key→value retrieval, and surfaces the layers OptiQ's sensitivity-aware allocation protects. See the eval-framework blog for the methodology and models page for the full 12-model table.

Speculative decoding for all Qwen + Gemma quants

• Qwen3.5 / Qwen3.6: every OptiQ-4bit quant ships a bundled MTP head as mtp.safetensors. Enable with optiq serve --model … --mtp for ~1.4× decode (acceptance ~70 % at depth 2). MTP tensors are preserved automatically by optiq convert. The host-bit-width-matched sidecar is written next to the safetensors shards and registered in config.json.
• Gemma-4: every OptiQ-4bit quant pairs with the matching mlx-community/<size>-it-assistant-bf16 drafter via optiq serve --model … --drafter …. The drafter runs γ=1 greedy speculation through optiq.runtime.spec. Tokens are bit-identical to non-spec decode.
• Both optiq serve and optiq lab expose --mtp (Qwen) and --drafter (Gemma) flags, mutually exclusive.

DPO fine-tuning

optiq lora train --method dpo --dpo-beta 0.1 adds Direct Preference Optimization to the LoRA trainer alongside SFT. Data shape: one {"prompt", "chosen", "rejected"} per line. The reference forward pass runs through the same adapter with scale=0, so there's no second model load. Standard DPO loss (Rafailov 2023) with diagnostic metrics (margin, chosen/rejected rewards, accuracy). Surfaced in the Lab Fine-tune wizard as a training-objective dropdown with a dpo_beta input.

Model-recommended sampling published in every quant

Every Qwen3.5 / 3.6 OptiQ quant on HuggingFace now ships a generation_config.json with the model author's "instruct / non-thinking general" recipe (temperature=0.7, top_p=0.8, top_k=20, min_p=0.0, repetition_penalty=1.0, presence_penalty=1.5). Gemma-4 quants already shipped Google's "all use cases" recipe (1.0 / 0.95 / 64). optiq convert writes these for future quants automatically (existing upstream values always win the merge). The Lab Server-settings page surfaces them as pre-filled inputs the moment a model is selected. A new POST /api/server/inspect-sampling endpoint reads generation_config.json from local cache, or pulls just that one small file from HF.

OptiQ Lab UI improvements

• Drafter picker is now a free-form text input that auto-suggests the correct mlx-community/<name>-it-assistant-bf16 repo whenever a Gemma-4 target is selected. Hides entirely for non-Gemma targets (Qwen uses MTP). All four Gemma sizes mapped (was only e4b in the initial Lab build).
• Sampler inputs pre-fill with the recommended values from generation_config.json the moment a model is picked, instead of staying blank until the model loads.
• Local model picker filters drafters out. discover() and discover_hf_cache() skip any *-assistant* / *-drafter repos, so users can't accidentally try to serve a drafter as the host model.
• Dataset page header copy fixed, "Twelve templates" (was "Six").
• Fine-tune hyperparams card adds a Training-objective dropdown (SFT default, DPO selectable) with a conditional dpo_beta input.

Site refresh

• New 12-model headline table on the homepage and models page with Capability Score + Δ vs uniform-4-bit instead of single-benchmark GSM8K numbers.
• Full rewrite of the eval-framework blog presenting the six-benchmark suite as the canonical Capability Score, with a dedicated HashHop Long-Context Evaluation section.
• All --target-bpw examples now use the new 5.0 default (the prior 4.5 hint was leftover from earlier internal work).
• Single-source-of-truth sidebar across all 19 docs pages via scripts/build_docs_sidebar.py. Every page now exposes the same 24-link nav with a correctly-set is-active marker.
• Six new Lab UI screenshots under site/assets/lab-screens/ referenced from the PyPI README so users see what the app looks like before installing.

Headline numbers (refreshed for v0.1.0)

Six-benchmark Capability Score (MMLU + GSM8K + IFEval + BFCL + HumanEval + HashHop), OptiQ vs uniform 4-bit:

OptiQ Lab: local web UI

New optional component (pip install "mlx-optiq[lab]" → optiq lab). Local Flask app that opens four workflow surfaces in a browser, all using the same backend as the CLI:

• Chat, streaming playground against the served model (matches Unsloth Studio's default landing). Three local tools the model can call:
• web_search, DuckDuckGo via the ddgs library (no API key), plus a {"url": ...} mode that fetches a single page as compact markdown. Snippet replies end with a "fetch the URL for more" advisory so the model nudges itself toward full-page reads when needed.
• python, runs code in a three-tier sandbox (apple/container → macOS sandbox-exec → subprocess+rlimit) with an AST safety check (os.system, subprocess.*, signal tampering, direct network calls are blocked). Per-call wall-clock + memory caps. Matplotlib output written to the sandbox workdir is captured, base64-inlined, and rendered as an <img> in the chat UI. The model never sees the base64 payload in its context.
• terminal, runs a bash one-liner in the same sandbox. Token-aware command-position blocking, so echo "do not use sudo" works but sudo ls does not.
• Tool-call healer recovers six common malformed shapes that quantized open-weight models emit instead of structured tool_calls. It handles Hermes/Qwen <tool_call> tags, fenced JSON blocks, bare JSON, trailing commas, fancy quotes, function-call form, and the {"python": {...}} key-is-tool-name pattern. Unknown tool names are rejected so the model cannot hallucinate a tool. Healed calls are flagged in the UI with a healed chip.
• Multi-turn orchestrator runs the tool loop server-side, capped at 25 turns (matches Unsloth Studio's default). On budget exhaustion the orchestrator appends a "stop calling tools and answer now" user message. It then re-prompts the model with tools disabled for one final reply, rather than erroring out.
• Duplicate-call de-dup detects consecutive identical tool calls and substitutes a nudge ("you already called this") rather than re-executing the sandbox. Failed calls are exempt from de-dup so the model can iterate on a fix.
• Tool-error nudge appends a "try a different approach" instruction to any tool result whose body starts with a recognized error sigil (Error, Blocked:, Exit code, Search failed, etc.). Surfaced in the UI with an error chip.
• Sandbox cancellation via a Stop button. The endpoint registers a per-stream threading.Event. The Stop button sends /api/chat/cancel, the orchestrator polls the event between turns, and any running tool subprocess gets SIGKILL'd at its process group. Replaces blocking subprocess.run with a Popen + watcher inside the sandbox.
• Adjacent tool-call grouping in the UI. When the model emits N>1 tool calls in a row, they collapse into one "N tool calls" accordion with per-tool chips, so the thread doesn't get visually swamped during multi-step runs. The most-recent group expands automatically. Older groups auto-collapse when assistant text follows.
• File attach: text + code formats inline, plus PDF (pypdf) and DOCX (docx2txt). Image / audio out of scope for v0.1.0.
• Reasoning toggle: enable_thinking controllable per chat (default off for small quants).
• Quantize, 4-step wizard: paste an HF model id, slide a target-BPW dial, watch live sensitivity + knapsack progress, one-click push the result to your HF account.
• Fine-tune, 4-step wizard for sensitivity-aware LoRA training on any OptiQ quant. Live train-loss sparkline via SSE. Save + fuse + push to HF.
• Build dataset, six templates (SFT from QA pairs, DPO, style transfer, code completion, self-instruct expansion, format conversion). Outputs JSONL the fine-tune wizard reads directly. Pushes to HF as repo_type="dataset".

Plus: sidebar with live API/MTP/model status, and copy-paste configs for all five tested integrations (Claude Code, Codex, OpenCode, OpenClaw, Hermes Agent). There is also an HF settings page that stores a write-scope token encrypted at rest (Fernet key derived from the Lab password via PBKDF2-HMAC-SHA256).

Architecture: Flask + jinja2 + vendored htmx 2.0.4 + Alpine 3.14.7 (no Node build step). Background jobs run in multiprocessing.Process workers. Progress streams via SSE. SQLite at ~/.optiq/lab/lab.db. Password auth on first run (argon2id + 24h JWT cookie). Localhost-only by default. The chat tool path uses a server-side SSE orchestrator that drives the multi-turn tool loop with a 25-turn cap.

Lab depends on a small extras set: flask, argon2-cffi, pyjwt, cryptography, data-designer, plus ddgs + html2text for web search and pypdf + docx2txt for file uploads. Apple Silicon only (matches the rest of mlx-optiq).

The Lab pins mlx-lm>=0.31 (needs the qwen3_5 arch module and the --prompt-cache-bytes flag, both landed upstream in mlx-lm 0.31.0).

KV-cache quantization on tight-RAM Macs

The diagnostic that started this cycle: mlx-lm's stock 4-bit KV cache on a 24 GB Mac actually peaks higher than fp16 KV at long context. Two things cause it. (1) A conversion spike that holds both fp16 and quantized cache co-resident. (2) A prefill scores-matrix spike inside quantized_scaled_dot_product_attention. On granite-4.1-8b-4bit at 32k, stock u4 peaks at 16.35 GB vs fp16's 11.51 GB. Users enabling --kv-bits 4 on tight RAM hit OOMs instead of memory savings.

Fix is two pieces, both default-on whenever KV-quant is enabled in optiq serve:

• optiq.runtime.streaming_kv_quant, replaces mlx-lm's batched maybe_quantize_kv_cache with a per-layer streaming variant. Quantizes one layer's K, mx.eval, drops the fp16 reference, clears the buffer pool, then the same for V, repeat. Bounds the conversion transient to roughly one layer's worth (~150 MB) instead of all layers (~5 GB).
• optiq.runtime.fused_quant_sdpa, replaces mlx-lm's unfused quantized_scaled_dot_product_attention with a FlashAttention-2 N-tiled variant using mx.quantized_matmul as the inner kernel. Per-tile scores shape is bounded by n_chunk=512, reducing the prefill transient ~10x. Algorithm matches what a fused Metal kernel would do. The matmul kernel is Apple's tuned Metal, so we don't lose to a hand-rolled version.

CLI:

• optiq serve --kv-bits 4 ... and optiq serve --kv-config ... ... install both patches automatically.
• optiq serve --no-fused-kv ... opts out (for bit-exact comparison vs upstream mlx-lm).

Bench (granite-4.1-8b-4bit on M4 24 GB, NIAH retrieval at depth 0.5):

At 32k, u4 KV with our path uses 34% less peak memory than fp16 KV, and 53% less than stock mlx-lm's u4 path. The same fix unlocks 48k–96k contexts on the Qwen3.5-9B hybrid model. There, fp16 KV gets tight (11.51 GB at 64k, 18.86 GB at 96k), but our u4 stays under 10 GB peak across the whole range. Needle retrieval correct everywhere.

Speed on Qwen3.5-9B is at parity with fp16 KV (±2%). u4 = 15.16 gen_tps vs fp16 = 15.47 (−2.00%). Mixed-precision = 15.29 (−1.16%).

Accuracy (hash-hop differentiation on Qwen3.5-9B-OptiQ-4bit, 25 trials per cell, ctx=10k chars):

At hops=2 (model's comfort zone) all three KV modes are within noise. At hops=3 (reasoning + KV precision stressed) uniform 4-bit halves accuracy vs fp16, while OptiQ's mixed-precision recovers ~40% of the lost ground. Hash-hop forces exact 16-character retrieval, so any KV noise that flips one character is a total failure. That makes it sensitive to KV quality in a way NIAH-style retrieval is not.

MTP speculative decoding (new in late-cycle v0.1.0 prep)

In-checkpoint MTP heads (DeepSeek-V3 style, found in Qwen3.5 / 3.6 family) are now preserved through optiq convert and usable at serve time.

• Convert preserves MTP: optiq convert <model> automatically fetches and quantizes any mtp.* tensors from the source. Output gets a mtp.safetensors sidecar registered in config.json via the MTPLX-compatible mtp_file + mtplx_mtp_quantization keys. No-op for models without MTP. Implementation in optiq/runtime/mtp_convert.py; called from convert_llm_to_mlx and convert_llm_static_mixed post-quantize.
• Serve runs MTP: optiq serve --mtp --mtp-depth N enables in-checkpoint MTP speculative decoding. Routes generation through OptiqEngine (built on top of vendored MTPLX runtime, Apache-2.0 attribution preserved under optiq/runtime/mtp/). MTP applies to all three endpoints transparently. Default off.
• Engine streaming: OptiqEngine.generate_stream(...) yields {token, text, from_draft, done, decode_tps, ...} events per token. from_draft=True marks tokens accepted from the MTP head; False for AR steps, verified replacements, and bonus tokens.

Measured on Qwen3.5-9B-OptiQ-MTP-4bit (M4 24GB, depth=2, 3-run median): 1.36× / 1.56× / 1.78× on 2B / 4B / 9B, 1.42× floor on 27B (memory-bound). 0.8B hurts (below speculation inflection).

Three API protocols on one port

optiq serve now exposes three OpenAI-compatible endpoints from the same process:

• /v1/chat/completions (OpenAI Chat Completions; default, used by most tools)
• /v1/messages (Anthropic Messages; --anthropic, default on; used by Claude Code, OpenClaw)
• /v1/responses (OpenAI Responses; --responses, default on; required by Codex, which deprecated Chat Completions in 2026; also used by Cursor, Continue, Cline)

The Responses translator (new files optiq/responses_shim.py + optiq/responses_server.py) mirrors the existing Anthropic shim pattern. Translate the Responses body to Chat Completions, reuse mlx-lm's existing handler, then translate the response back to Responses output shape with proper response.created / response.output_text.delta / response.completed SSE events. Built-in Responses tools (web_search, file_search, mcp, computer_use) are dropped silently. Only function tools are forwarded.

Stateful previous_response_id: Responses requests that carry previous_response_id resume from the prior turn. Each completed response (stream or non-stream) is stored in a process-local TTL cache (optiq/response_store.py, 1 hr TTL, 32 MiB LRU cap). On a follow-up the stored input + prior assistant output is spliced ahead of the new input. Unknown id returns 404. Prior reasoning items are dropped from the replayed history (clients don't want chain-of-thought fed back in).

Reasoning capture (both protocols): a reasoning model (Qwen3.5, DeepSeek-R1, etc.) can emit chain-of-thought via the chat template. Both the non-streaming and the streaming paths surface it in the appropriate protocol-specific channel:

• Anthropic non-stream: {"type": "thinking", "thinking": "..."} content block (Claude 3.7 extended-thinking format) followed by {"type": "text", "text": "..."} block.
• Anthropic stream: content_block_start (type=thinking) → content_block_delta (type=thinking_delta) (many) → content_block_stop, then the text block.
• Responses non-stream: {"type": "reasoning", "summary": [{"type": "summary_text", "text": "..."}]} output item followed by {"type": "message", ...} item.
• Responses stream: response.output_item.added (reasoning) → response.reasoning_summary_text.delta (many) → response.reasoning_summary_text.done → response.output_item.done, then the message item.

Incremental SSE flush: both shims now expose AnthropicStreamTranslator / ResponsesStreamTranslator classes that the server proxy feeds chunk-by-chunk, so the client sees bytes flowing while the model is still generating. Reasoning models have first-token latency well past most SDK read timeouts. Without per-chunk flushing the connection dropped before any byte arrived.

Auth: sk-optiq-* Bearer token

New --auth / --no-auth flag on optiq serve (default on). Bearer tokens that start with sk-optiq- are accepted. Any suffix works. Missing Authorization header is allowed for local-dev curl convenience. Wrong prefix returns 401 with a clear error message. Mirrors Unsloth's sk-unsloth-* convention so integration configs feel familiar.

Integrations docs

site/docs/integrations/ adds per-tool guides with copy-pasteable configs:

• Claude Code (via Anthropic endpoint + ANTHROPIC_BASE_URL)
• Codex (via Responses endpoint + ~/.codex/config.toml)
• OpenCode (via OpenAI Chat Completions)
• OpenClaw (via Anthropic endpoint)
• Hermes Agent (via OpenAI Chat Completions)

Bonus tools that work out of the box and are listed on the integrations index: Cursor, Continue, Cline, aider, Open WebUI, LangChain / LlamaIndex / DSPy via OpenAI SDK base_url override.

Previously documented v0.1.0 changes

Sensitivity analysis: 3 methods → 1. Removed `analyze_sensitivity_fast (empirical-Fisher) and analyze_sensitivity_stream (weight-space MSE). The only remaining sensitivity entry point is analyze_sensitivity_exact`. It is calibration-driven KL divergence on logits, the gold-standard signal that actually predicts downstream quality.

The exact path now supports two execution modes (selected by the new `--reference CLI flag, or the reference= kwarg in run_llm_pipeline`):

• `--reference bf16`, load the bf16 base into RAM, swap each layer's weight in-place between bf16 and a simulate-quantized copy. Highest fidelity.
• `--reference uniform_4bit`, first build a uniform-4-bit MLX baseline from the bf16 source and load that as the running model. Then stream bf16 weights off disk one layer at a time to swap in for sensitivity probes. Lets 27 B+ models still get a calibration-driven signal on a 36 GB Mac.
• `--reference auto (default), pre-checks available RAM and routes between the two automatically. Logs a warning when falling back to uniform_4bit`.

Whichever reference is used, the FINAL OptiQ output is the same mixed-precision artifact. It is built freshly by `mlx_lm.convert` with the per-layer bit predicate (e.g. ~4.5 BPW = mix of 4-bit and 8-bit per layer).

YOLO support removed. Object detection didn't share any code with the LLM stack (separate sensitivity, separate eval, separate calibration data, separate `yolo-mlx runtime). The 5 published mlx-community/YOLO26{n,s,m,l,x}-OptiQ-6bit artifacts remain on HuggingFace as legacy from v0.0.x. They still work with yolo-mlx` directly, but the conversion path is no longer in mlx-optiq. Removed:

• `optiq/models/yolo.py`
• `optiq/eval/yolo_eval.py`
• `[yolo] extras in pyproject.toml`
• `yolo-mlx` dependency

Removed code: `analyze_sensitivity_fast, analyze_sensitivity_stream, analyze_sensitivity dispatcher, _lm_loss, _accumulate_grads, plus all the _iter_safetensors_* helpers that were stream-specific (the new uniform_4bit reference path uses different streaming logic, kept inside sensitivity.py`).

Kept: the `_chunked_quantize mlx-lm patch. It is still essential for the final mlx_lm.convert step on large MoE models (Gemma-4-26B-A4B, Qwen3.5/3.6-35B-A3B), where mlx-lm's standard nn.quantize` sweep blows the Metal GPU command-buffer timeout.

Library size: `optiq/core/sensitivity.py` shrank from 885 lines to ~520. Total deletion across all touchpoints: ~1,200 lines.

CLI changes (breaking):

• `--sensitivity {exact,fast,stream}` → removed. Only one method now.
• `--reference {auto,bf16,uniform_4bit} → new flag, default auto`.
• `--calibration-mix {optiq,/path/to/your.jsonl} → new flag, default optiq`.
• `--model-type yolo` → removed.

New: bundled six-domain calibration mix (`optiq/calibration/data/optiq.jsonl). 40 hand-curated samples across prose (5), reasoning (6), code (6), agent loops (8), function-calling (7), and constraint-bearing instructions (8, regex-filtered from HuggingFaceH4/no_robots and disjoint from google/IFEval). Chat samples are auto-rendered through the target model's tokenizer.apply_chat_template() before tokenization, so the activated subspace matches production. Replaces the prior WikiText-only calibration, which under-protected tool-call and instruction-following layers. Reproducible via python scripts/build_calibration.py. See site/blog/calibration-mix.html` for the methodology.

New: two-tier eval framework (`optiq.eval.*). optiq eval --task smoketest runs Tier-1 (KL on 64 prompts × 256 tokens + GSM8K-50, ~5 min/model). optiq eval --task all --score runs Tier-2 headline (MMLU-1k 5-shot, GSM8K-1k, IFEval full, BFCL-V3 simple-200, HumanEval-164, HashHop 25 × 4 hops). It emits a single Capability Score = mean(MMLU, GSM8K, IFEval, BFCL, HumanEval, HashHop), a simple unweighted average across the six benchmarks. disk_gb is reported separately as an honest second axis, with no hidden value judgement embedded in the formula. KL evaluator auto-resolves the reference (bf16 if it fits in RAM, else mlx-community uniform-4-bit baseline; uses HfApi.model_info for accurate per-shard sizes). HumanEval execution sandbox falls through apple/container → sandbox-exec → subprocess + rlimit. HashHop wraps the a hash-hop generator and exercises long-context multi-hop key→value retrieval at ~12 k tokens. New tasks individually addressable: --task {kl,gsm8k-50,mmlu,gsm8k,ifeval,bfcl,humaneval,hashhop}`.

Optimizer fix: KL-reduction sign-flip detection. Under `--reference uniform_4bit, the candidate's per-bit sensitivity to the uniform-4-bit baseline reverses sign relative to bf16-reference mode. KL(low) ≈ 0, because the layer is identical to baseline. KL(high) > 0, because the simulate-quantize at higher precision is *closer to bf16* than the baseline. The greedy knapsack's current.sensitivities[low] - current.sensitivities[high] term went negative, the priority heap rejected every upgrade, and 27 B+ models came out uniform-4-bit instead of mixed. Fixed in _kl_reduction()`: detects sign-flip and inverts the subtraction. Verified against a 35B-A3B checkpoint where the planner went from 80 → 400 layers at 8-bit at the same target BPW.

Memory hygiene during sensitivity probes. Calibration sequences capped at `seq_len=512 (was 1024), the memory-safe sweet spot for 27 B+ models on a 36 GB Mac during bf16-streaming probes. Each probe explicitly gc.collect + mx.clear_cache` after use. Reference logits cast to fp16 before per-probe KL math. Cuts peak resident memory by ~30 % during the layer sweep.

Description string updated to reflect LLM focus: "Mixed-precision quantization optimizer for LLMs on Apple Silicon (MLX)".

Removed: `_strip_unused_modalities (and --keep-unused-modalities flag, optiq.core.text_only, tools/strip_vision.py). The strip rewrote config.json (flipped architectures: [Qwen3_5ForConditionalGeneration] → [Qwen3_5ForCausalLM], flattened text_config, dropped vision/audio modality keys) under the assumption it was producing a cleaner text-only artifact. Empirically it provided zero disk/memory savings (mlx_lm.convert` already drops vision/audio weights from the safetensors output regardless of the strip flag). It also broke instruction-following on Qwen3.5/3.6 hybrid-attention models by 14–17 pp on IFEval (versus uniform-4 baseline at the same target BPW). Bisection: bf16 IFEval = 71.9 %, uniform-4 = 68.4 %, OptiQ all-8-bit with strip = 55.8 %, OptiQ all-8-bit without strip = 72.3 %. The arch-class flip was routing mlx-lm to a different model class with subtly different long-generation handling. The fix is to leave the config.json intact. mlx-lm loads multimodal-arch quants via the VLM-arch class, which delegates to the language path for text inference.

Sensitivity-side fix for MoE expert weights. The optimizer + planner now correctly assign per-layer bits to every expert in Gemma-4 / Qwen3.5-MoE / Qwen3.6-MoE base models. Conversion still requires more work for the largest MoE models, see "Known limit" below.

Fix: MoE expert weights were silently skipped during sensitivity analysis (optiq/core/sensitivity.py)

The Gemma-4 / Qwen3.5/3.6 MoE base models store their experts as 3-D fused tensors without a .weight suffix on disk:

• *.experts.gate_up_proj of shape (num_experts, 2 * mid, hidden), gate and up packed back-to-back along axis -2
• *.experts.down_proj of shape (num_experts, hidden, mid)

mlx-lm's sanitize() later splits these into the per-projection 3-D weights it actually loads (*.experts.switch_glu.{gate,up,down}_proj for Gemma-4, *.mlp.switch_mlp.{gate,up,down}_proj for Qwen3.5/3.6 MoE). _iter_safetensors_linears was both filtering by len(shape) != 2 and requiring key.endswith(".weight"), so every expert tensor was dropped from the sensitivity index. With no per-layer entry, make_quant_predicate fell back to default_bits (the max bit-width in the allocation, typically 8) for every expert weight. That blew up storage BPW well past the planning target.

The fix:

• _iter_safetensors_linears now also accepts ≥3-D tensors whose last dim is divisible by group_size (mx.quantize handles these natively, it operates per-row over the last axis).
• New _iter_safetensors_moe_experts(model_path, group_size) walker recognizes the on-disk MoE patterns above. It applies the same axis-(-2) split mlx-lm's sanitize() does for gate_up_proj, and emits per-leaf entries with the sanitized mlx-lm-side names. The wrapper name (switch_glu for Gemma-4, switch_mlp for Qwen3) is picked up from config.json.
• The shared loader in analyze_sensitivity_stream reads the on-disk fused tensor once per shard and applies the gate/up slice in-memory before evaluating each half independently. Param-count bookkeeping generalized from shape[0] * shape[1] to a product over all dims.

Empirical effect on google/gemma-4-26B-A4B-it (sensitivity index 400 → 490 entries; bit allocator now sees the experts):

Known limit: mlx-lm's nn.quantize graph times out on the largest MoE bases. With 30 layers × 250 M-parameter expert tensors, mlx-lm sweeps to_quantized across the full module tree. The Metal command buffer it builds exceeds the GPU's per-buffer timeout (kIOGPUCommandBufferCallbackErrorTimeout) on a 36 GB M3 Max, at any target BPW. mx.quantize itself runs in 0.01 s on a single 484 MB expert tensor, so the bottleneck is the queued pipeline, not the kernel. Until mlx-lm chunks the quantize sweep (or until OptiQ pre-quantizes the experts shard-by-shard before handing off), the largest MoE bases need a more memory-headroom-friendly machine to convert. The currently-published mlx-community/gemma-4-26B-A4B-it-OptiQ-4bit was built under v0.0.10 (24.5 GB, experts at 8-bit by fallback) and remains usable. It will be re-uploaded once the convert path is unblocked. Smaller MoE bases should benefit from the v0.0.11 sensitivity fix immediately.

Two new 27 B HF artifacts and a small set of opt-in extensions that didn't make the v0.0.9 cut. No breaking changes. All v0.0.9 APIs remain.

New HF artifacts (text-only, language-stack-only, vision/audio metadata stripped at convert time):

• mlx-community/Qwen3.5-27B-OptiQ-4bit, 343 layers @ 4-bit + 247 layers @ 8-bit, 4.50 BPW achieved, 15.7 GB on disk. Inference ~14 tok/s and ~10 GB resident on M3 Max 36 GB.
• mlx-community/Qwen3.6-27B-OptiQ-4bit, same recipe, same shape. Both produced via optiq convert ... --sensitivity stream, mmap-streamed weight-space sensitivity with peak ~9 GB RAM during conversion. That path is required for ≥ 27 B on a 36 GB Mac, since mlx_lm.load of bf16 27 B OOMs.

New: optiq/ops/, opt-in long-context training kernels

• flash_attention (tile-based forward) and flash_attention_metal + flash_attention_backward_metal (custom Metal kernels with explicit backward).
• chunked_cross_entropy and chunked_ce_training for chunked LM-head loss + grad on long sequences.
• fused_swiglu, attention_patch, sublayer_checkpoint for activation-memory experiments.
• Honest positioning: vanilla mlx-lm + grad_checkpoint=True is the right default for ≤ T = 12 000 on a 36 GB M3 Max (~82 s/step at peak T). The custom kernels here are 2–4× slower but use slightly less memory. Opt in only when the wired-limit + memory-pressure budget genuinely won't fit vanilla. See optiq.ops.attention_patch.enable_flash_attention_training for the activation switch.

New: optiq.anthropic_server, Anthropic-compatible /v1/messages endpoint

• install_anthropic_endpoint() monkey-patches mlx_lm.server.APIHandler.do_POST to recognize Anthropic's /v1/messages path while leaving every OpenAI route untouched. Lets clients written against Claude's SDK (e.g. agentic tools that hard-code the Anthropic API shape) talk to a local mlx-optiq server with no code changes.
• Idempotent installer (_INSTALLED guard), safe to call from optiq.serve startup or tests.
• Streaming + tool-call shape conversion handled in optiq.anthropic_shim (request body normalization, response shape rewrite, Anthropic event-stream protocol).

New: optiq.core.text_only.strip_multimodal_metadata(), library API

• Previously embedded in the optiq convert flow only. Now exposed as from optiq.core.text_only import strip_multimodal_metadata for users who want to clean up an existing model directory in place. It rewrites architectures, flattens text_config, drops vision/audio token IDs, and removes mrope sub-fields. Weight tensors are not touched.
• Useful when post-processing third-party VLM-text-only checkpoints that load fine in mlx-lm but still publish the wrong pipeline_tag and architecture class.

Documented: empirical training-ceiling map at system-default iogpu.wired_limit_mb=0 (M3 Max 36 GB)

Conservative optiq lora train recipes (default config: q_proj, v_proj, NL=16, rank=8, rank_scaling=by_bits) verified end-to-end against a real Hermes traces fine-tune dataset:

All five sit at peak ≤ 27.7 GB with at least 0.3 GB of headroom under the system-default cap and zero observed memory drift across iters. Going higher in T trades throughput for context coverage, and risks compressed-memory penalty. At much higher T (e.g. 2 B at T = 3200) it hits a sharp Apple-Silicon MTLResource-count cliff that fails before any byte ceiling. Don't extrapolate "more headroom in GB" → "can push T further". Each model has its own throughput knee at some seq_len.

Major: sensitivity-aware LoRA and reversible hot-swap adapters go from v0.0.8 "works programmatically" to production, with end-to-end validation. Gemma-4 variants re-uploaded with corrected weights and the gemma4_text model type. Full website refresh at mlx-optiq.pages.dev. All 6 HF model cards rewritten with the v0.0.9 positioning.

New: reversible mounted LoRA + per-request hot-swap (optiq.adapters.mount)

• MountedLoRALinear replaces the target linear modules with a wrapper that holds a dict {adapter_id: (A, B, scale)} and gates them via a ContextVar the server can flip per request.
• AdapterActivation context manager for request-scoped activation.
• prepare_model_for_mounted_lora / mount_adapter_on_model / unmount_adapter_from_model.
• AdapterRegistry wired to use the mount path, activate / deactivate / unmount are now reversible without a model reload.
• 5 rigorous tests verify: mount changes logits, activate→deactivate restores base bit-for-bit, two adapters switchable in the same process, ContextVar isolation across concurrent asyncio tasks, selective unmount preserves other adapters.

New: qwen3_5_text model type (optiq.mlx_lm_patches)

• import optiq auto-registers a qwen3_5_text module into mlx_lm.models that wraps qwen3_5.TextModel directly (no VLM .language_model prefix). Facade only, follows upstream changes automatically.
• Opt-in via tools/strip_vision.py --route-to-optiq-text for local conversions. HF-published models continue to use qwen3_5 (VLM wrapper) so they load cleanly with bare mlx-lm.

Fix: Gemma-4 weight-corruption bug in strip_vision.py

The v0.0.7 Gemma-4 strip wrote all-zero weights. mx.load returns lazy memory-mapped views into the file, and saving to the same path overwrote the file before the arrays were materialized. Fixed by forcing mx.eval(list(tensors.values())) before mx.save_safetensors to the same path.

Both Gemma-4 OptiQ variants (gemma-4-e2b-it-OptiQ-4bit, gemma-4-e4b-it-OptiQ-4bit) have been re-uploaded to HuggingFace with corrected weights and model_type: gemma4_text.

Housekeeping

• All 6 HF model cards rewritten for v0.0.9: covers bare-mlx-lm usage, mlx-optiq unlocks (KV serving, LoRA, hot-swap), and benchmarks.
• Website v0.0.6 → v0.0.9. New feature cards on the homepage (LoRA + hot-swap), new Thread 4 (sensitivity-aware LoRA) and Thread 5 (mounted hot-swap) on experiments.
• README fully rewritten with "optimized deployment for MLX" positioning, unified story across weight quant, KV quant, TurboQuant, LoRA, hot-swap, VLM-strip, YOLO26, latency prediction.
• Roadmap update: Gemma-4 KV serving blocker precisely diagnosed as gemma4_text shared-layer attention passing packed tuples to mx.fast.scaled_dot_product_attention (upstream mlx-lm fix needed).

Tests

• tests/test_release.py, 7/7 passing (config strip on all 6, load+generate all 6, qwen3_5_text routing, gemma4_text routing + weight-corruption fix verified, TurboQuant untouched, LoRA metadata reads, LoRA roundtrip).
• tests/test_hotswap.py, 5/5 passing.

Bug-fix release that makes the v0.0.7 LoRA / serve features actually work end-to-end. v0.0.7 shipped with three bugs that prevented optiq lora train and optiq serve --adapter from running to completion. None of them were caught, because the release wasn't exercised beyond --help smoke tests. Fixed:

Fixes

• optiq/lora/apply.py, removed a spurious DoRALinear import. mlx-lm 0.31.x only ships LoRALinear. DoRA is now an explicit NotImplementedError instead of ImportError.
• optiq/lora/sensitivity_rank.py, read per_layer from the top level of optiq_metadata.json (where current OptiQ pipelines write it) in addition to the legacy optimization.per_layer path. Also supports reading metadata directly from a HuggingFace repo id via hf_hub_download when the model isn't on local disk.
• optiq/lora/trainer.py, wrap mlx-lm's load_dataset output in CacheDataset before handing to the training loop. The naked TextDataset returns dicts and crashes iterate_batches.
• optiq/lora/trainer.py, _write_peft_config now emits the mlx-lm-required fine_tune_type, num_layers, and lora_parameters fields alongside the PEFT-style keys. mlx_lm.generate --adapter-path <optiq_adapter> and optiq serve --adapter <optiq_adapter> both load cleanly.

End-to-end validation (now part of the release checklist)

Qwen3.5-0.8B-OptiQ-4bit has 187 per-layer bit assignments (111 × 4-bit, 76 × 8-bit). 12 of those layers are q_proj/v_proj targets for LoRA.

Ran two 100-iter training runs on a 160/40 GSM8K subset (max_seq=512), same base, same optimizer, same data:

The sensitivity-aware configuration reaches a lower mid-training validation loss (1.045 vs 1.183 at iter 50). That is the expected pattern. Giving more capacity to layers OptiQ identified as sensitive accelerates convergence. The gap closes by iter 100 on this small dataset. Longer runs and larger datasets are needed for a definitive comparison. Full run metadata in article/assets/v007_lora/rank_scaling_ab.json.

Round-trip test

After training, the adapter saves into /tmp/optiq_lora_test/adapter/. Stock mlx_lm.generate --adapter-path loads it, and it plugs into optiq serve --adapter <path> with the OptiQ sidecar surfaced at startup. Verified with a generation of a held-out arithmetic question ("What is 9+9?" → "The answer is 18.").

Positioning shift: OptiQ is now framed as "the Apple Silicon deployment optimizer", mixed-precision quantization plus unused-component stripping plus sensitivity-aware LoRA fine-tuning, not just a quantizer.

New: sensitivity-aware LoRA fine-tuning (optiq lora train)

• Reads optiq_metadata.json for per-layer bit assignments
• Derives per-layer LoRA rank from OptiQ's KL-sensitivity measurements (--rank-scaling by_bits by default; by_kl and constant also supported)
• Adapter output is PEFT-compatible (adapter_config.json + adapters.safetensors) plus an OptiQ sidecar (optiq_lora_config.json) recording the per-layer rank distribution
• optiq lora info <adapter_dir> summarizes any adapter (OptiQ-trained or stock PEFT)
• Monkey-patches out mx.compile from mlx-lm's trainer at runtime (required to avoid Metal OOM on Qwen3.5-9B 4-bit)

New: optiq serve --adapter <id_or_path>

• Serve with a LoRA adapter applied at startup
• Accepts HuggingFace repo ids (<owner>/<adapter-repo>) and auto-downloads into ~/.cache/optiq/adapters/ (override via OPTIQ_ADAPTER_CACHE)
• Surfaces OptiQ sidecar info at startup when present (rank, per-layer rank distribution)

New: optiq convert strips unused multi-modal metadata by default

• Multi-modal base models (Qwen3.5-VLM, Gemma-4) have their vision/audio weights dropped by mlx-lm during text-only quantization. But the config.json keeps the VLM architecture class, text_config wrapper, and modality token IDs. OptiQ now cleans those up automatically, so the quantized model advertises itself accurately as text-generation
• Opt out with --keep-unused-modalities
• The same cleanup is available as a standalone reusable function: optiq.core.text_only.strip_multimodal_metadata(model_dir)
• tools/strip_vision.py script ships with the repo for retroactively cleaning previously-published OptiQ models

Housekeeping

• All six published OptiQ LLM variants (Qwen3.5-0.8B/2B/4B/9B, Gemma-4-e2b/e4b) re-uploaded to mlx-community/* with cleaned metadata. Weights are unchanged and generation output is bit-identical
• Version bump in pyproject.toml and optiq/cli.py

Docs-only release.

• README + site point at https://mlx-optiq.pages.dev/ (was wrong URL in v0.0.5).
• Added the article (Not All Layers Are Equal)

link in the site nav and footer.

New: optiq serve, an OpenAI-compatible server with mixed-precision KV cache

• optiq serve --kv-config path/to/kv_config.json wraps mlx_lm.server and injects

per-layer KV quantization at serving time. Uses mlx_lm.models.cache.QuantizedKVCache

+ mx.quantized_matmul (fused kernel: real memory savings, no fp16 materialization).

• optiq serve --kv-bits 4 for uniform quantization.
• optiq serve (no flags) forwards to fp16 mlx_lm.server unchanged.

Fix: optiq kv-cache supports hybrid-attention models

• Previously assumed every layer had self_attn. Now detects layers with a quantizable

KV cache (via hasattr(c, "to_quantized")) and skips linear-attention layers.

Required for Qwen3.5 (GatedDeltaNet + self-attention mix).

Benchmarks (Apple M3 Max, 64k context, decode tok/s vs fp16):

Known issues

• Gemma-4 (e2b, e4b): mixed-precision KV path fails on Gemma-4's shared-KV attention

layers. Upstream mlx-lm limitation, tracked for a future release. Fp16 serving

works fine.

Previous release. Weight quantization pipeline (optiq convert), KV sensitivity

analysis (optiq kv-cache), latency/eval/benchmark commands, TurboQuant library

primitives (optiq.core.turbo_kv_cache).