fix: load Gemma-4 vision tower unquantized under compressed-tensors - #33988
Closed
Smallfu666 wants to merge 1 commit into
Closed
Smallfu666 wants to merge 1 commit into
Smallfu666 wants to merge 1 commit into
Conversation
Smallfu666
requested review from
Alisehen,
AniZpZ,
BBuf,
Edwardf0t1,
FlamingoPg,
HaiShaw,
OrangeRedeng,
b8zhong,
ch-wan,
kpham-sgl,
mmangkad and
pyc96
as code owners
August 7, 2026 12:19
google/gemma-4-31B-it-qat-w4a16-ct lists every vision-tower linear in quantization_config.ignore, but the checkpoint wraps each tower nn.Linear in a clip module so the entries end in ".linear". SGLang keeps that suffix for the unfused clippable linears; the fused qkv_proj and gate_up_proj do not have it, so the ignore match fails for those two per block and they are quantized. The fused gate_up_proj is 2 * 4304 = 8608 wide, which is not divisible by Marlin's min_thread_n = 64, and load dies in gptq_marlin_repack. Drop the compressed-tensors config for the vision tower, which is what the checkpoint's ignore list asks for. Other quantization methods are left alone: modelopt's NVFP4 Gemma-4 checkpoint already excludes model.vision_tower* by glob, and there is no evidence to justify de-quantizing the vision tower on the online fp8/awq/gptq paths. Also make CompressedTensorsWNA16 and GPTQMarlinLinearKernel use verify_marlin_supports_shape instead of check_marlin_supports_shape, whose (ok, err_msg) return value was dropped in both, so unsupported shapes reached the kernel instead of raising the actionable "not divisible by min_thread_n = 64" error. awq_marlin already uses the raising variant. Signed-off-by: Han-Yin Chang <nick20350@gmail.com>
Smallfu666
force-pushed
the
fix/28018-gemma4-tower-unquantized
branch
from
August 7, 2026 17:02
09b44da to
7e258e6
Compare
Contributor
Author
|
Superseded by #35792, which narrows the fix to the fused-shard .linear ignore mismatch and adds end-to-end load validation. |
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
What
Addresses the Marlin shape-selection failure reported in #28018.
google/gemma-4-31B-it-qat-w4a16-ctignores every vision-tower linear, but its clipwrapper makes those
ignoreentries end in.linear. SGLang keeps that suffix on theunfused clippable linears; the fused
qkv_proj/gate_up_projdo not, so those two perblock miss the match and get quantized.
gate_up_projis2 * 4304 = 8608, not divisibleby Marlin's
min_thread_n = 64, so load dies ingptq_marlin_repack.So drop the compressed-tensors config for the vision tower, with a
warning_once. Otherquant methods are untouched: the NVFP4 checkpoint already excludes
model.vision_tower*by glob, and I have nothing justifying de-quantizing online fp8/awq/gptq.
Second point:
CompressedTensorsWNA16andGPTQMarlinLinearKernelboth dropped thereturn value of
check_marlin_supports_shape, which never raises, so bad shapes hit thekernel instead of the actionable
min_thread_n = 64error. Both now useverify_marlin_supports_shape, likeawq_marlin. I left the repack kernel alone.Tests
CPU unit tests, each confirmed to fail when its own fix alone is reverted: one builds the
vision-tower modules against a two-layer replica of the checkpoint config and asserts
UnquantizedLinearMethod; two assertn=8608raisesValueErroron each Marlin path.End to end on one GB10, same command both sides: main dies in
gptq_marlin_repack(
size_n = 8608vstile_n_size = 64); patched loads the checkpoint and answers arequest.
Output quality not asserted; TP>1 and image input untested.
Happy to normalize
.lineargenerically in compressed-tensors instead, if you prefer.CI States
Latest PR Test (Base): ❌ Run #31200478757
Latest PR Test (Extra): ❌ Run #31200478387