Gemma-4 on Apple Silicon

Google's Gemma-4 instruct series spans five sizes: two compact dense (e2b, e4b), a dense 12 B (the unified text+vision model, now with image input), one sparse mixture-of-experts (26 B-A4B), and one frontier-class dense (31 B). All five ship with mlx-optiq-4-bit quants on Hugging Face, and all five take image input. They share the Gemma chat template and a distinctive shared-KV attention design.

Available quants

QAT variants

Google also ships QAT-unquantized Gemma-4 checkpoints — bf16 weights trained so that 4-bit quantization is near-lossless. OptiQ runs its sensitivity pass on those bases and still beats a uniform 4-bit quant of the same QAT base on the six-metric Capability Score. The scores below compare each OptiQ-QAT quant against a genuine uniform-4 quant of its own QAT base, holding the base fixed, so the delta is purely what the per-layer allocation adds.

The 31B flagship posts the largest absolute gain (+1.65, with a +9-point jump on long-context HashHop retrieval). The 26B-A4B sparse-MoE is the one case where 4-bit QAT already saturates the routed experts; reaching them takes a higher average bit-width, so its OptiQ quant ships larger and wins by a smaller margin. Load any of these exactly like the non-QAT quants below — same import optiq + mlx_lm.load path.

Hello world

from mlx_lm import load, generate

model, tok = load("mlx-community/gemma-4-31B-it-OptiQ-4bit")

prompt = tok.apply_chat_template(
    [{"role": "user", "content": "What's the difference between TF-IDF and BM25?"}],
    tokenize=False, add_generation_prompt=True,
)
print(generate(model, tok, prompt=prompt, max_tokens=400))

Recommended sampling

Gemma-4-it variants prefer slightly higher temperature than Qwen3.x:

from mlx_lm.sample_utils import make_sampler

# Default chat (Google's recommended)
sampler = make_sampler(temp=1.0, top_p=0.95, top_k=64)

# Reasoning / math: slightly tighter
sampler = make_sampler(temp=0.7, top_p=0.9)

Mixed-precision KV cache

Each Gemma-4 OptiQ-4bit repo on Hugging Face now bundles a recommended kv_config.json from a real sensitivity-analysis pass. Drop-in via --kv-config:

$ optiq serve \
    --model mlx-community/gemma-4-31B-it-OptiQ-4bit \
    --kv-config kv_config.json

The 26B-A4B MoE

Gemma-4-26B-A4B is a sparse mixture-of-experts: 26 B total, 4 B active per token. Different from Qwen MoE, Gemma uses switch_glu with a fused gate-and-up projection. mlx-optiq's MoE walker handles both layouts. The per-expert sensitivity rolls up into a single switch_glu tensor for allocation purposes.

Long-context

fp16 KV is still the simplest path; mixed-precision KV (above) gives faster decode at long contexts. Either way:

# Stock fp16 KV serving via mlx-optiq (no --kv-config)
$ optiq serve --model mlx-community/gemma-4-31B-it-OptiQ-4bit \
    --max-tokens 8192 --temp 1.0 --top-p 0.95

Fine-tuning recipes

Empirical training-ceiling map at iogpu.wired_limit_mb=0 on 36 GB Mac:

The 31B-dense is unusually memory-tight at long context due to its larger embedding+vocab footprint. The 26B-A4B MoE actually allows a longer sequence at the same RAM budget. Sparse activation pays off here.

$ optiq lora train mlx-community/gemma-4-26B-A4B-it-OptiQ-4bit \
    --data ./my_data \
    --max-seq-length 512 \
    --rank 8 --rank-scaling by_bits \
    --num-layers 16 --iters 2000 \
    -o ./gemma_adapter

Next: read about how sensitivity works or the LoRA fine-tuning guide.