Conversation
Port of upstream PR sgl-project#33988 (gemma4_mm.py hunk only; the marlin shape-check renames in that PR do not apply to v0.5.17). compressed-tensors Gemma-4 checkpoints (e.g. RedHatAI/gemma-4-26B-A4B-it-FP8-dynamic) exclude the whole vision tower via the quantization ignore list, but the entries carry the checkpoint wrapper's '.linear' suffix and unfused q/k/v names, which never match SGLang's fused modules — so the bf16 vision tower was loaded through fp8 schemes with garbage scales and produced NaN image features. Observed blast radius on v0.5.17: the default multimodal warmup request runs the broken vision path, its NaN KV is cached at the shared <bos> radix prefix, and every subsequent request greedy-decodes to <pad>. This also makes --skip-server-warmup insufficient protection for a prod deployment: any real image request re-poisons the cache. Upstream issue: sgl-project#24927 (closed, but the fix PR is still unmerged as of 2026-08-14). Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> (cherry picked from commit 29a730878ae07cd81ae5504c18a5bc2bb1efa6d0)
Port of unmerged upstream PR sgl-project#22615 (issue sgl-project#22277). Gemma-4 KV-sharing layers call attention with k=None/v=None and the triton backend reads K/V straight out of the KV cache. With --kv-cache-dtype fp8_e4m3 those tensors arrive as fp8 and the extend kernel dies compiling tl.dot(bf16_q, fp8_k) ('Unsupported rhs dtype'). Convert to the query dtype and apply k/v scales, mirroring what the non-shared path does at store time. Validated on MI325x with RedHatAI/gemma-4-26B-A4B-it-FP8-dynamic: fp8 KV doubles the pool (2.20M full + 1.76M SWA tokens vs 1.11M/891k bf16) and improves the shared-prefix bench to 31.4/33.6 req/s (NP=512/2048) vs 28.6/29.9 with bf16 KV; greedy quality unchanged. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> (cherry picked from commit 9294fcc510d41c4e484c8535fd47d8c0b9e230d4)
…project#33580) (cherry picked from commit 301a8a09c5e8ab6575a76efb0a55881888044118)
…ressure (sgl-project#33777) (cherry picked from commit f4f91ef254608f64102fb8cddfcc7a55bffc6c13)
(cherry picked from commit bacab43e9a914513a778dc058dd702443d11abe1)
(cherry picked from commit fede84057fcf579620ef6921e01ed4d8cb1eb09d)
(cherry picked from commit 7b0c65f7e5497167855824f639e20d3aea7517e3)
(cherry picked from commit b9c4c16925ca63802a3debc56cb8bc8015c5ca86)
(cherry picked from commit 41d1b33f12cf3f6dd4057d2e19e50be3485e7e1a)
…sgl-project#34644) (cherry picked from commit 26627e9) (cherry picked from commit 53668167ff1334028affe0ffe47423fccbed7b59)
…-project#34607) (cherry picked from commit 3974b00) (cherry picked from commit 2a7a56d3df71becd7a031a390a6c4502ba34342f)
…ct#35017) (cherry picked from commit c34e0793a66eda02e87c36a247389c9457ef369f)
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
7 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Stacked reorg of the CAI sglang fork, part 1 of 9 (parts 7-9 are private, in sglang-internal).
req.prefix_indiceswhen the prefix cache is disabled sgl-project/sglang#34644 Add bit-exact unified radix cache KL test for hybrid SWA + mamba sgl-project/sglang#34607) — all already merged upstream and contained in v0.5.18; this whole group dissolves at the v0.5.18 rebase.All commits carry -x provenance. Upstream status: already upstream or open upstream PRs.
🤖 Generated with Claude Code