Repository navigation
[Feat] diffusion: Native Gemma-4-Unified text encoder for LTX-2.5 - #47411
Merged
Merged
Conversation
LTX-2.5's text encoder had no SGLang implementation and loaded through
transformers.AutoModel ("doesn't have a customized version yet, using
native version"). Build its text path natively: only the language model
(the vision and audio embedders are never loaded), fused QKV and
gate/up GEMMs, k_eq_v full-attention layers, per-layer-type RoPE and
sliding-window masks, and the same op order and dtypes as the
transformers SDPA path. The transformers class-swap adapter stays for
checkpoints whose format transformers owns.
On B300 (LTX-2.5-Diffusers 4791a110) the hidden states the connectors
read are bitwise identical to transformers for six prompts including a
batch of two, encode time is unchanged (47.18 vs 47.15 ms at S=1024),
and the end-to-end MP4 has the same md5.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
![claude[bot]](https://avatars.githubusercontent.com/in/1236702?s=60&v=4){kind=small}
There was a problem hiding this comment.
Claude Code Review
This repository is configured for manual code reviews. Comment @claude review for a one-time review, or @claude review always to subscribe this PR to a review on every future push.
Tip: disable this comment in your organization's Code Review settings.
mickqian
requested review from
AgainstEntropy,
BBuf,
HaiShaw,
kevin-mii,
niehen6174 and
yichiche
as code owners
October 30, 2026 07:31
This comment has been minimized.
This comment has been minimized.
mickqian
added a commit
to mickqian/diffusion-bench-framework
that referenced
this pull request
Oct 30, 2026
LTX-2.5 joins b300x4-20261022, measured on b300final against vLLM-Omni 0da1d831 over two interleaved rounds: sglang 5.086 s, vLLM-Omni 6.032 s (1.186x). The sglang tree is main 52604e1d plus sgl-project/sglang#47411, the native Gemma-4-Unified text encoder; main still loads that encoder through transformers, which the publish gate refuses. The encoder's hidden states are bitwise equal to the transformers ones and the MP4 has the same md5. sglang now leads 17/17 cases. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
This comment has been minimized.
This comment has been minimized.
mickqian
changed the title
Collaborator
Author
|
/tag-and-rerun-ci |
Yuxingwang-intel
pushed a commit
to Yuxingwang-intel/sglang
that referenced
this pull request
Nov 6, 2026
…l-project#47411) Co-authored-by: Mick Qian <mickqian@users.noreply.github.com> Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Motivation
LTX-2.5's text encoder (
Gemma4UnifiedForConditionalGeneration, 22 GB) has no SGLang implementation. On main it loads throughtransformers.AutoModel:runtime/models/encoders/gemma_4_unified.pyonly class-swaps that transformers module so it can be layerwise-offloaded (#43683).Modifications
Gemma4UnifiedForConditionalGenerationinruntime/models/encoders/gemma_4_unified.pybuilds only the text path, because LTX-2.5 reads the embedding output plus all 48 decoder hidden states. The vision and audio embedders in the checkpoint are not loaded.QKVParallelLinear/MergedColumnParallelLinear/RowParallelLinear),k_eq_vfull-attention layers (V loads K's weights), QK/V RMSNorm, per-layer-type RoPE (default/proportional), and sliding-window plus padding masks.layer_names/host_resident_table_names, so layerwise offload and the host-resident vocab table keep working.configs/models/encoders/gemma_4_unified.py: the config now carries the repo'stext_config. The gate/up shard ids are ints.test_gemma4_unified_layerwise_offload.pycheck that a small random model matches transformers, thek_eq_v/ GQA layers, and the layerwise and host-resident declarations.Accuracy
4×B300 (sm_100),
Lightricks/LTX-2.5-Diffusersrevision 4791a110, bf16, torch 2.14.1. Native vs transformers on the same GPU:End-to-end
sglang generateLTX-2.5 (two-stage, 1920×1088×121, same seed): the MP4 md5 is identical (4abb8c3a…) with the native encoder and with the transformers one.Speed
Encode B=1 S=1024, median/min: native 47.18 / 47.15 ms, transformers 47.15 / 47.13 ms, i.e. unchanged.
Load time: native 4.4 s, transformers 4.9 s.
Serving,
ltx2.5_twostage_t2v_2gpus(2×B300, Ulysses 2 + breakable CUDA graph, client wall clock, 2 requests per fresh server): this PR 5.093 / 5.086 s, main 1ab008f with the transformers encoder 5.113 / 5.101 s. vLLM-Omni 0da1d831 on the same box: 5.962 / 6.032 s.Not tested
🤖 Generated with Claude Code
CI States
Latest PR Test (Base): ✅ Run #41020614664
Latest PR Test (Extra): ❌ Run #41020614807
Latest PR Test (AMD ROCm 10): ❌ Run #41020615198