Post
45
The Qwen3.5-27B performance landscape
I started gathering some numbers on the 27Bs.
You might have noticed that reported metrics differ from Thinking to Instruct models and this is expected. The mxfp8/mxfp4 are the most stable quants I could measure, and I provided Deckard(qx) quants where possible
Converting a Thinking model to Instruct
The model is thinking/instruct, and the instruct mode can be forced by setting the first line of the jinja template to
Qwen3.5-27B-Text
This is a model I tested where the vision tower was removed, its performance is the same as the VL model.
nightmedia/Qwen3.5-27B-Text-qx86-hi-mlx
DavidAU/Qwen3.5-27B-Claude-4.6-OS-INSTRUCT
On the top of the heap of the models I tested, as far as metrics go, is this model created by DavidAU. Samples of the output are provided on the model card.
nightmedia/Qwen3.5-27B-Claude-4.6-OS-INSTRUCT-mxfp8-mlx
For the Thinking version, see nightmedia/Qwen3.5-27B-Architect-Claude-qx86-hi-mlx
More metrics in comments.
-G
P.S. I will update this as soon as I have new numbers or I found a typo--whichever comes first. The models that show just the arc-check numbers are in the test queue and will be updated soon.
I started gathering some numbers on the 27Bs.
You might have noticed that reported metrics differ from Thinking to Instruct models and this is expected. The mxfp8/mxfp4 are the most stable quants I could measure, and I provided Deckard(qx) quants where possible
Converting a Thinking model to Instruct
The model is thinking/instruct, and the instruct mode can be forced by setting the first line of the jinja template to
{%- set enable_thinking = false %}Qwen3.5-27B-Text
This is a model I tested where the vision tower was removed, its performance is the same as the VL model.
nightmedia/Qwen3.5-27B-Text-qx86-hi-mlx
quant arc arc/e boolq hswag obkqa piqa wino
qx86-hi 0.443,0.498,0.857,0.701,0.372,0.770,0.752
mxfp4 0.460,0.527,0.871,0.694,0.370,0.772,0.752DavidAU/Qwen3.5-27B-Claude-4.6-OS-INSTRUCT
On the top of the heap of the models I tested, as far as metrics go, is this model created by DavidAU. Samples of the output are provided on the model card.
nightmedia/Qwen3.5-27B-Claude-4.6-OS-INSTRUCT-mxfp8-mlx
quant arc arc/e boolq hswag obkqa piqa wino
mxfp8 0.675,0.827,0.900,0.750,0.496,0.800,0.721
qx86-hi 0.667,0.824,0.902,0.752,0.502,0.791,0.725
qx64-hi 0.664,0.820,0.902
mxfp4 0.653,0.815,0.899For the Thinking version, see nightmedia/Qwen3.5-27B-Architect-Claude-qx86-hi-mlx
More metrics in comments.
-G
P.S. I will update this as soon as I have new numbers or I found a typo--whichever comes first. The models that show just the arc-check numbers are in the test queue and will be updated soon.