Support Hy4-preview - #36805
Merged
Merged
Support Hy4-preview#36805
Conversation
JustinTong0323
requested review from
Alisehen,
AniZpZ,
BBuf,
DarkSharpness,
Edwardf0t1,
FlamingoPg,
Fridge003,
HaiShaw,
HydraQYH,
OrangeRedeng,
Qiaolin-Yu,
Ying1123,
b8zhong,
celve,
ch-wan,
fzyzcjy,
hebiao064,
ispobock,
merrymercy,
mmangkad and
yuan-luo
as code owners
August 28, 2026 06:24
github-actions
Bot
added
quant
JustinTong0323
force-pushed
the
xinyuan/hy4-preview-support
branch
from
August 28, 2026 09:33
55b1524 to
f4d8ba5
Compare
Model and quantization: add the hy_v4 architecture (text-only) — MLA with DSA sparse attention, iHC, gated MLA, learned attention sinks, sigmoid-gated MoE, and MTP/NextN drafting — plus MXFP8 (ModelOpt, UE8M0 group-32) weight loading with MTP-exclusion remapping and deep_gemm as the default MoE/FP8 GEMM backend. Unsupported PP/CP/DCP configurations raise at construction time instead of failing later. Attention and state: DSA backend and MLA forward changes for learned sinks, the HYV4 iHC layernorm kernel, radix-attention integration, and the DSV4-only resolve_spec_hidden_size fix. Protocol: Hunyuan tool-call/reasoning parser auto-detection (suffix structural tokens, arg_key/arg_value extraction, schema type coercion) and chat-template auto-detection. Co-authored-by: BBuf <1182563586@qq.com> Co-authored-by: alphabetc1 <2508695655@qq.com>
Hy4 dropped <tool_sep> from its vocab and emits adjacent <tool_call> blocks without separators, so the Hy3 structure_info literals never match its output. Detect the dialect from the tokenizer vocab and emit separator-free begin/end for Hy4. Required/named tool_choice fell through to a json_schema grammar that forces plain JSON the model was not trained to produce in this format; mark the detector as parsing required tool choice natively so those requests use the model's structural tool-call format instead.
… layouts Hy4 sets swiglu_limit=10.0, and the masked DeepGEMM activation path only exists as the DSV4 JIT kernel, which requires D // 8 >= num_experts and group 128. With the layout heuristic on auto, Hy4 at TP8 (D=256) or any MXFP8 group-32 config selects the masked layout and dies in the DSV4 activation assert. Force the compact layout whenever the masked branch would be entered with a shape or quant group the kernel cannot serve; DSV4's legal masked configuration (D=2048, E=256, group 128) is unaffected and pinned by a test.
Upstream ParallelContext now answers config leaves directly (get_parallel().enable_dp_lm_head, as in nemotron_h.py); the .config.<leaf> spelling from the older runtime-context API raises AttributeError at model construction.
Remove test cases that do not uniquely guard a realistic silent failure: mock self-assertions, happy-path mirrors covered by conversion/golden tests, predicate restatements, and duplicated parameter grids. Drop test_deep_gemm_masked_layout_guard.py and its CPU CI registration; the CUDA compact-vs-masked end-to-end test covers the observable path.
4 tasks
# Conflicts: # python/sglang/srt/entrypoints/openai/serving_chat.py # test/registered/unit/entrypoints/openai/test_serving_chat.py
# Conflicts: # test/registered/unit/server_args/test_server_args.py
Fridge003
approved these changes
Sep 4, 2026
handle_legacy_cp_arguments was split upstream into handle_platform_cp_compatibility and handle_legacy_cp_runtime_compatibility; the restored test still referenced the removed name and failed at collection.
# Conflicts: # python/sglang/srt/configs/model_config.py
is_sm100_supported was removed from quantization/fp8.py; gate the layout test on get_platform().is_sm100 like the production check.
Collaborator
This was referenced Sep 5, 2026
StevenChenSE
pushed a commit
to StevenChenSE/sglang
that referenced
this pull request
Sep 6, 2026
Co-authored-by: BBuf <1182563586@qq.com> Co-authored-by: alphabetc1 <2508695655@qq.com>
This was referenced Sep 7, 2026
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
3 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
Adds support for the
hy_v4architecture (Hy4-preview, text-only): MLA with DSA sparse attention, iHC, gated MLA, learned attention sinks, sigmoid-gated MoE, and MTP/NextN speculative drafting, together with MXFP8 quantization support.What is included
Validation
Limitations
CI States
Latest PR Test (Base): 🚫 Run #33934945618
Latest PR Test (Extra): 🚫 Run #33934945447
Latest PR Test (AMD ROCm 7.2): 🚫 Run #33934945566