mlx-optiq
OptiQ CLI · optimizing compiler

Quantize, serve, and fine-tune from the command line.

The MLX-native core. Measure each layer's sensitivity, allocate mixed-precision bits, and ship a quant that keeps more quality than uniform 4-bit at the same size. Then serve it behind an OpenAI- or Anthropic-compatible API and fine-tune it with sensitivity-aware LoRA. No PyTorch, no cloud.

Per-layer bit allocation · sample LLM
Per-layer bit allocation across a 32-layer transformer: tall accent bars are 8-bit protected layers, short muted bars are 4-bit.
8-bit · sensitive layers 4-bit · robust layers
$ pip install mlx-optiq
3.2×
smaller than bf16
+15.6
best Capability gain vs U4
How it works

Measure, allocate, convert.

Which layers tolerate fewer bits is measured, not assumed.

01 · MEASURE

Per-layer KL

Simulate-quantize each layer at each bit-width, forward-pass calibration data, and measure the KL divergence from the bf16 reference.

02 · ALLOCATE

Greedy knapsack

Start every layer low, then upgrade the layer that buys the most quality per bit until the budget is met. lm_head and the first/last blocks stay high.

03 · CONVERT

Standard MLX

Hand the bit map to mlx_lm.convert. The output loads anywhere stock mlx-lm loads, with the sensitivity metadata stored alongside.

What it does

The same pass sets KV bits and adapter ranks.

Weight bits, KV-cache bits and LoRA ranks all come out of that one per-layer measurement.

i

Mixed-precision weights

Sensitive layers stay high-precision, the rest go low, at the same average size as uniform-4.

Higher accuracy at the same disk size
ii

Mixed-precision KV cache

A separate sensitivity pass on the cache. Layer 0 is often 56× more sensitive than average, so uniform 4-bit KV is catastrophic, while mixed-precision avoids it.

Faster long-context decode, kept sharp
iii

LoRA, two ways

Fine-tune with adapter rank scaled by each layer's bits, then keep N adapters mounted on one base and hot-swap per request.

Sensitivity-aware rank · hot-swap
iv

Text and images

Send pictures to vision-capable models. The vendored tower rides in a bf16 sidecar, so one repo loads text-only under mlx-lm or full image+text under OptiQ.

VLM + LLM · one artifact
v

OpenAI + Anthropic serving

optiq serve speaks both protocols from one process. Point Claude Code, Codex, or any client at a local quant.

Dual-protocol local server
vi

Expert pruning

Remove half a Mixture-of-Experts model's routed experts. Active parameters per token do not change, so it gets smaller without getting slower.

REAP, in the quantized domain
Quickstart

From a model to a serving API.

i

Convert

--reference auto picks bf16 when it fits and a uniform-4-bit baseline when it does not.

terminalbash
$ optiq convert Qwen/Qwen3.5-9B --target-bpw 5.0 --candidate-bits 4,8
ii

Serve

Dual-protocol server with mixed-precision KV and idle auto-unload.

terminalbash
$ optiq serve --model mlx-community/Qwen3.5-9B-OptiQ-4bit --idle-timeout 300
iii

Prune

Halve a sparse MoE's expert bank. It reports the divergence from the unpruned model before writing anything.

terminalbash
$ optiq prune-experts mlx-community/Qwen3.6-35B-A3B-OptiQ-4bit --target-retention 0.5
Full reference The CLI reference documents every command; How sensitivity works covers the algorithm, and Expert pruning covers what halving a MoE costs.

The compiler that builds every OptiQ quant.

Quantize, serve, and fine-tune LLMs natively on Apple Silicon.

Read the CLI docsBrowse the models