Skip to content

Support NoPE layers in the tokenspeed_mla FP8 prefill hook - #38152

Merged
Fridge003 merged 3 commits into
mainfrom
mmangkad/fix-tokenspeed-mla-nope-prefill
Sep 5, 2026
Merged

Fridge003 merged 3 commits into
mainfrom
mmangkad/fix-tokenspeed-mla-nope-prefill

Conversation

@mmangkad

@mmangkad mmangkad commented Sep 5, 2026 •

Copy link
Copy Markdown
Contributor

Motivation

Three extra-b-test-4-gpu-b200 jobs fail on main (run 33962519343): test_kimi_linear_dcp4.py, test_kimi_linear_dcp_dspark4.py, test_unified_radix_cache_kl_dcp.py, all Kimi Linear on tokenspeed_mla + fp8_e4m3 under DCP. Every TP rank crashes during prefill and the server is SIGKILLed:

File ".../layers/attention/tokenspeed_mla_backend.py", line 265, in prepare_prefill_qkv
    cos_sin_cache=layer.rotary_emb.cos_sin_cache,
AttributeError: 'NoneType' object has no attribute 'cos_sin_cache'

Surfaced by #35770, which taught resolve_attn_backend() to unwrap HybridLinearAttnBackend.full_attn_backend. Kimi Linear is hybrid KDA/MLA, so prefill used to resolve to the wrapper, which has no prepare_prefill_qkv hook, and took the BF16 path guarded by if self.rotary_emb is not None. It now resolves to TokenspeedMLABackend, whose hook dereferences layer.rotary_emb unconditionally, and Kimi Linear builds its MLA layers with skip_rope=True, which leaves rotary_emb as None.

Modifications

_fused_rope_fp8_quantize gains a NoPE branch: when cos_sin_cache is None, keep the FP8 quantize and the packed [nope | pe] layout, drop only the rotation. It is an early return, so RoPE layers are unaffected.

The cos_sin_cache and is_neox_style reads become None checks rather than getattr defaults, and _fused_rope_fp8_quantize becomes a @staticmethod with a unit case, since nothing PR-gated covered this path.

Test Plan

Test Result
dcp/test_kimi_linear_dcp4.py OK, 3 tests
dcp/test_kimi_linear_dcp_dspark4.py OK, GSM8K 0.905 (threshold 0.88)
radix_cache/unified_radix_tree/test_unified_radix_cache_kl_dcp.py OK, avg_kl_div 0.0078 (threshold 0.01)
attention/unittests/mla/test_tokenspeed_mla.py OK

test_kimi_linear_dcp4.py was also reproduced red by stashing the patch.


CI States

Latest PR Test (Base): ❌ Run #33984976309
Latest PR Test (Extra): ❌ Run #33984976221
Latest PR Test (AMD ROCm 7.2): ❌ Run #33984976307

@mmangkad

mmangkad commented Sep 5, 2026

Copy link
Copy Markdown
Contributor Author

/rerun-test test/registered/dcp/test_kimi_linear_dcp4.py test/registered/dcp/test_kimi_linear_dcp_dspark4.py test/registered/radix_cache/unified_radix_tree/test_unified_radix_cache_kl_dcp.py test/registered/attention/unittests/mla/test_tokenspeed_mla.py

@github-actions

github-actions Bot commented Sep 5, 2026 •

Copy link
Copy Markdown
Contributor

Results for /rerun-test test/registered/dcp/test_kimi_linear_dcp4.py test/registered/dcp/test_kimi_linear_dcp_dspark4.py test/registered/radix_cache/unified_radix_tree/test_unified_radix_cache_kl_dcp.py test/registered/attention/unittests/mla/test_tokenspeed_mla.py:

🚀 4-gpu-b200 (4 tests): ✅ View workflow run

cd test/ && python3 registered/dcp/test_kimi_linear_dcp4.py
cd test/ && python3 registered/dcp/test_kimi_linear_dcp_dspark4.py
cd test/ && python3 registered/radix_cache/unified_radix_tree/test_unified_radix_cache_kl_dcp.py
cd test/ && python3 registered/attention/unittests/mla/test_tokenspeed_mla.py

🚀 1-gpu-h100 (1 test): ✅ View workflow run

cd test/ && python3 registered/attention/unittests/mla/test_tokenspeed_mla.py

@mmangkad mmangkad changed the title Support NoPE layers in the FP8 prefill hook Support NoPE layers in the tokenspeed_mla FP8 prefill hook Sep 5, 2026
@Fridge003
Fridge003 merged commit 09daea9 into main Sep 5, 2026
113 of 129 checks passed
@Fridge003
Fridge003 deleted the mmangkad/fix-tokenspeed-mla-nope-prefill branch September 5, 2026 23:51
StevenChenSE pushed a commit to StevenChenSE/sglang that referenced this pull request Sep 6, 2026
…ct#38152)

Co-authored-by: Mohammad Angkad <mohammad.angkad@radixark.ai>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants