OsaurusAI/Mistral-Medium-3.5-128B-mxfp4

Quantized mistralai/Mistral-Medium-3.5-128B for Apple Silicon (MLX) — dense 128B parameter multimodal LLM with image input + 256K context.

Source	mistralai/Mistral-Medium-3.5-128B
Architecture	mistral3 wrapper — ministral3 text decoder (88L × 12288, GQA 96/8) + pixtral vision encoder (48L × 1664)
Quant format	MXFP4 (mlx 4-bit affine, group_size=32)
Bundle size on disk	85.72 GB (78 safetensors shards)
License	Apache-2.0 (inherits from upstream)
Modalities	Text + image in / text out (no audio, no video)
Context	262 144 tokens (YaRN, factor=64 from orig=4096)

What's quantized

• Text decoder linears + embed_tokens → mlx 4-bit affine (bits=4 group_size=32)
• vision_tower.*, multi_modal_projector.*, lm_head → fp16 passthrough
• All RMSNorms → fp16 passthrough

Vision tower (model.vision_tower.*, 48 layers, 1664 hidden, patch=14, image_size=1540, spatial_merge=2) — kept bf16 → fp16 passthrough, matching upstream quantization_config.modules_to_not_convert. Images dispatch to the same pixtral encoder; embeddings are folded into the LM via a 2-layer GELU multimodal projector (also fp16 passthrough).

lm_head is fp16 passthrough (matches upstream ignored set) — no quantization noise on the final logits.

Codec round-trip validation (this bundle)

7/7 PASS at cosine ≥ 0.94 across L0 → L87 covering attn + MLP + GQA k/v projections — the expected 2-bit TurboQuant noise floor. Source FP8 e4m3 dequant uses weight_scale_inv per-tensor scale; vision/projector/lm_head pass through unchanged.

Run on Apple Silicon

pip install mlx safetensors transformers pillow
python -m jang_tools.mistral3.runtime \
    --src ~/.mlxstudio/models/OsaurusAI/Mistral-Medium-3.5-128B-mxfp4 \
    --prompt "Describe this image." \
    --image /path/to/photo.jpg \
    --max-new 64

The runtime auto-detects weight_format and dispatches; image preprocessing matches the pixtral spec (jang_tools.vl.pixtral.PixtralImageProcessor).

Build

python -m jang_tools.convert_mistral3_mxfp4 \
    ~/.mlxstudio/models/_sources/Mistral-Medium-3.5-128B \
    ~/.mlxstudio/models/OsaurusAI/Mistral-Medium-3.5-128B-mxfp4

Credits

Quantized by Jinho Jang (eric@osaurus.ai). MLX-native pipeline; 88 dense decoder layers + 48 pixtral vision layers run on M-series Macs.