Skip to content

fix: load Gemma-4 vision tower unquantized under compressed-tensors - #33988

Closed
Smallfu666 wants to merge 1 commit into
sgl-project:mainfrom
Smallfu666:fix/28018-gemma4-tower-unquantized
Closed

Smallfu666 wants to merge 1 commit into
sgl-project:mainfrom
Smallfu666:fix/28018-gemma4-tower-unquantized

Conversation

@Smallfu666

@Smallfu666 Smallfu666 commented Aug 7, 2026

Copy link
Copy Markdown
Contributor

What

Addresses the Marlin shape-selection failure reported in #28018.

google/gemma-4-31B-it-qat-w4a16-ct ignores every vision-tower linear, but its clip
wrapper makes those ignore entries end in .linear. SGLang keeps that suffix on the
unfused clippable linears; the fused qkv_proj / gate_up_proj do not, so those two per
block miss the match and get quantized. gate_up_proj is 2 * 4304 = 8608, not divisible
by Marlin's min_thread_n = 64, so load dies in gptq_marlin_repack.

So drop the compressed-tensors config for the vision tower, with a warning_once. Other
quant methods are untouched: the NVFP4 checkpoint already excludes model.vision_tower*
by glob, and I have nothing justifying de-quantizing online fp8/awq/gptq.

Second point: CompressedTensorsWNA16 and GPTQMarlinLinearKernel both dropped the
return value of check_marlin_supports_shape, which never raises, so bad shapes hit the
kernel instead of the actionable min_thread_n = 64 error. Both now use
verify_marlin_supports_shape, like awq_marlin. I left the repack kernel alone.

Tests

CPU unit tests, each confirmed to fail when its own fix alone is reverted: one builds the
vision-tower modules against a two-layer replica of the checkpoint config and asserts
UnquantizedLinearMethod; two assert n=8608 raises ValueError on each Marlin path.

End to end on one GB10, same command both sides: main dies in gptq_marlin_repack
(size_n = 8608 vs tile_n_size = 64); patched loads the checkpoint and answers a
request.

Output quality not asserted; TP>1 and image input untested.

Happy to normalize .linear generically in compressed-tensors instead, if you prefer.


CI States

Latest PR Test (Base): ❌ Run #31200478757
Latest PR Test (Extra): ❌ Run #31200478387

google/gemma-4-31B-it-qat-w4a16-ct lists every vision-tower linear in
quantization_config.ignore, but the checkpoint wraps each tower nn.Linear in a
clip module so the entries end in ".linear". SGLang keeps that suffix for the
unfused clippable linears; the fused qkv_proj and gate_up_proj do not have it,
so the ignore match fails for those two per block and they are quantized. The
fused gate_up_proj is 2 * 4304 = 8608 wide, which is not divisible by Marlin's
min_thread_n = 64, and load dies in gptq_marlin_repack.

Drop the compressed-tensors config for the vision tower, which is what the
checkpoint's ignore list asks for. Other quantization methods are left alone:
modelopt's NVFP4 Gemma-4 checkpoint already excludes model.vision_tower* by
glob, and there is no evidence to justify de-quantizing the vision tower on the
online fp8/awq/gptq paths.

Also make CompressedTensorsWNA16 and GPTQMarlinLinearKernel use
verify_marlin_supports_shape instead of check_marlin_supports_shape, whose
(ok, err_msg) return value was dropped in both, so unsupported shapes reached
the kernel instead of raising the actionable "not divisible by
min_thread_n = 64" error. awq_marlin already uses the raising variant.

Signed-off-by: Han-Yin Chang <nick20350@gmail.com>
@Smallfu666
Smallfu666 force-pushed the fix/28018-gemma4-tower-unquantized branch from 09b44da to 7e258e6 Compare August 7, 2026 17:02
@Smallfu666 Smallfu666 changed the title fix: load Gemma-4 vision/audio towers unquantized under compressed-tensors fix: load Gemma-4 vision tower unquantized under compressed-tensors Aug 8, 2026
@Smallfu666 Smallfu666 closed this Aug 21, 2026
@Smallfu666

Copy link
Copy Markdown
Contributor Author

Superseded by #35792, which narrows the fix to the fused-shard .linear ignore mismatch and adds end-to-end load validation.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

quant LLM Quantization

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant