The Problem: Despite its new "Linear Attention" architecture (Gated Delta Networks) designed for high efficiency, the Qwen3.5-4B model is performing much slower than the older Qwen3-4B (21.0 tokens/s vs. 34.5 tokens/s).
The Cause: The current model builder forces the majority of layers (192 out of 249) into INT8 precision instead of INT4.
MatMulNBits Summary
| Precision Type |
Count |
| INT4 |
57 |
| INT8 |
192 |
| Total |
249 |
The Fix: By implementing PR #2094 (setting linear_attention_int8 to false by default), we can restore performance to a level similar to Qwen3-4B with no noticeable loss in output quality.
Performance at a Glance
| Model Version |
Decode Speed |
Status |
| Qwen3-4B (Old) |
34.5 tokens/s |
Baseline |
| Qwen3.5-4B (Current) |
21.0 tokens/s |
⚠️ Regression |
| Qwen3.5-4B (With Fix) |
28.8 tokens/s |
Optimized |
The Problem: Despite its new "Linear Attention" architecture (Gated Delta Networks) designed for high efficiency, the Qwen3.5-4B model is performing much slower than the older Qwen3-4B (21.0 tokens/s vs. 34.5 tokens/s).
The Cause: The current model builder forces the majority of layers (192 out of 249) into INT8 precision instead of INT4.
MatMulNBits Summary
The Fix: By implementing PR #2094 (setting
linear_attention_int8tofalseby default), we can restore performance to a level similar to Qwen3-4B with no noticeable loss in output quality.Performance at a Glance