Family guide · DeepSeek-V4

DeepSeek-V4 on Apple Silicon

DeepSeek-V4-Flash is a sparse-attention mixture of experts. Across its 43 layers it runs three different attention mechanisms, chosen per layer by a compress_ratios entry in the config: five layers use sliding-window local attention, twenty compress the KV, and twenty-one compress it and then select a top-k subset to attend over. Almost all of its weight sits in the routed experts, which is what makes it quantize well.

Vendored architecture Stock mlx-lm has no deepseek_v4 class. Three separate pull requests tried to add one and none were merged, so OptiQ vendors the working implementation and registers it on import optiq. Call import optiq before mlx_lm.load.

Running it on a small machine

The 2-bit quant is 92.5 GB on disk, which is more than most Macs have in RAM. OptiQ streams the routed experts off SSD instead of holding them resident: the scales and biases stay in memory, the packed expert weights are read as a token routes to them. The model loads in a few seconds at 6.5 GB resident. Peak memory climbs with context rather than with model size, so a short prompt stays near 8.5 GB and a 900-token one reaches about 12.7 GB.

Nothing needs configuring. optiq serve and the Python API detect a MoE quant too large to sit resident and switch to streaming on their own.

The quant

Model	bf16 size	OptiQ size	Resident	Context
DeepSeek-V4-Flash-0731-OptiQ-2bit	608 GB	92.5 GB	6.5 GB	1M

Hello world

hello.pypython

import optiq  # registers the deepseek_v4 arch with mlx-lm
from optiq.runtime import moe_stream
from mlx_lm import generate

model, tok = moe_stream.load_streaming("mlx-community/DeepSeek-V4-Flash-0731-2.4bit-mixed")
print(generate(model, tok, prompt="What is 17 * 23?", max_tokens=64))

Expect it to be slow Streaming experts from SSD trades throughput for fitting at all. Decode runs at a few tokens per second on a 24 GB machine, which is fine for a batch job and painful for a chat. With enough RAM to hold the model resident it runs at normal speed.

How the bits are allocated

The routed experts hold the overwhelming majority of the parameters and tolerate 2-bit; attention, the shared experts, the embeddings and the output head do not, and stay at 4-bit or higher. Quantizing everything uniformly collapses the model into repeated fragments, which is the same result other projects report on this architecture.

This is what OptiQ's per-layer allocation does everywhere, but the effect is starker here than on a dense model, because the split between what matters and what does not lines up so cleanly with the expert boundary.

Prompt format

DeepSeek-V4 ships no chat template, so the prompt is assembled by hand:

prompttext

<|begin_of_sentence|><|User|>{your message}<|Assistant|></think>

The trailing </think> closes the reasoning block so the model answers directly. Leave it off and it thinks first.