Skip to content

[Fix] [NVIDIA] modelopt-nvfp4: Keep MoE global scales in gate/up order - #45289

Merged
Fridge003 merged 4 commits into
mainfrom
codex/nvfp4-canonical-global-scales
Oct 20, 2026
Merged

[Fix] [NVIDIA] modelopt-nvfp4: Keep MoE global scales in gate/up order#45289
Fridge003 merged 4 commits into
mainfrom
codex/nvfp4-canonical-global-scales

Conversation

@Fridge003

@Fridge003 Fridge003 commented Oct 20, 2026 •
edited by github-actions Bot
Loading

Copy link
Copy Markdown
Collaborator

Motivation

ModelOpt NVFP4 MoE global scales identify the logical gate and up projections, while some kernels require weights in physical [up, gate] order. This PR replaces the merged #44858 implementation with a loader fix that keeps global scales in canonical [gate, up] order across backends.

The contents of #44858 (78185f4b3f565cbe0fec20d71bb5deb372428e80) are reverted, including its helper extraction and added weekly test. The loader implementation retains the SwiGLU beta correction (1 / g1_alphas_up), the shared-clamp warning, and CPU tests of the actual loading and post-load paths.

Changes

  • Reorder ModelOpt NVFP4 weights and block scales in _load_w13, leaving per-tensor global scales in logical order.
  • Use use_flashinfer_trtllm_weight_layout so nominal TRT-LLM with MegaMoE retains canonical weight order.
  • Remove the backend-dependent up_first argument from GEMM1 alpha computation.
  • Cover backend layouts, TP slicing, repeated loading, non-gated experts, and SwiGLU/SiTU post-load parameters in the existing CPU suite.

Validation

CI States

Latest PR Test (Base): ✅ Run #39393353744
Latest PR Test (Extra): ❌ Run #39393353284
Latest PR Test (AMD ROCm 10): ❌ Run #39393353493

@github-actions github-actions Bot added quant LLM Quantization blackwell SM100/SM120 labels Oct 20, 2026
@Fridge003 Fridge003 added run-ci CI: run the baseline test suite on this PR run-ci-extra CI: also run the extra suite (requires run-ci) labels Oct 20, 2026
@Fridge003
Merge branch 'main' into codex/nvfp4-canonical-global-scales
f828eb9
@Fridge003
test: align MegaMoE capacity fixture with expert placement flag
d78c3b3
@Fridge003
Merge main and revert #44858 in favor of the NVFP4 loader fix
bef4f08 
Revert the contents of 78185f4 while retaining the original canonical-scale loader implementation, MegaMoE layout guard, SwiGLU beta correction, and loader/post-load regression tests. Adopt main's MegaMoE CPU fixture update.
@Jiminator Jiminator mentioned this pull request Oct 20, 2026
Merged
5 tasks done
@Fridge003

Fridge003 commented Oct 20, 2026

Copy link
Copy Markdown
Collaborator Author

base tests pass https://github.com/sgl-project/sglang/actions/runs/39393353744

@Fridge003
Fridge003 merged commit 5b1dbc2 into main Oct 20, 2026
215 of 254 checks passed
@Fridge003
Fridge003 deleted the codex/nvfp4-canonical-global-scales branch October 20, 2026 23:49
@bingxche bingxche mentioned this pull request Oct 21, 2026
Open
@FeSens FeSens mentioned this pull request Oct 28, 2026
Draft
2 of 5 tasks
detain added a commit to detain/sglang that referenced this pull request Oct 29, 2026
@detain
docs(stack): FINALIZE entries 261-274 -- header bump to qwennext-stac…
3b1b610 
…k-274, wanted_stack LANDED marks, CAVEATS docs-only ledger items

(A) Header: STACK line 260->274; three wave lines (+spec-mamba-2026-10-28
(pr45491+pr45435+pr45472+pr45670), +admission-2026-10-28(pr46272*+
pr43484*+pr46496*+pr46643*+pr45607*+pr45789*), +parsers-2026-10-28
(pr44124+pr45631+pr45855+pr46167)); tip line -> 2714457; updated
261-274 headline block (entry-119 SUPERSEDED by upstream sgl-project#45435/sgl-project#45472;
classic-pool mamba admission gate w/ upstream ImportError CAUGHT;
UNCONDITIONAL latch reset = TTFT watch; coder-parser union; logits
trilogy as CONTRACT UNIT; 14 picks, 0 integrator conflicts).

(B) wanted_stack.txt: FAM-A5 + FAM-B234 blocks deleted -> LANDED records
(chain shas, interlock audit, supersession + behavior-change pointers
into entries 261-270); SECTION-B sgl-project#44124/trilogy blocks -> LANDED;
sgl-project#37958/sgl-project#45559 block -> DONE (verdict now in CAVEATS).

(C) DOCS-ONLY LEDGER ITEMS: CAVEATS gains the sgl-project#37958/sgl-project#45559 VERIFIED-
INERT bullet (two independent hard gates, coarse-barrier fallback,
do-not-re-litigate-at-merge-sgl-project#5 line, revisit condition = dropping NEXTN
or HiCache) and the dated C0b/C0c corrections (sgl-project#44030 obsolete rationale,
sgl-project#44858 REVERSED by sgl-project#45289 [ported-pending wt-moe], sgl-project#44115 permanent N/A,
10-28 open-status re-verification list, no-pin-moves note: 0.7.2/0.4.9
floor unchanged) with the C0a tick (sgl-project#37625/sgl-project#37941/sgl-project#39727 still
outstanding; only sgl-project#43927/sgl-project#44849 partially satisfied). C0b/C0c/C8 marked
DONE in wanted_stack; C0a stays open.
detain pushed a commit to detain/sglang that referenced this pull request Oct 29, 2026
@Fridge003 @detain
[Fix] [NVIDIA] modelopt-nvfp4: Keep MoE global scales in gate/up order (
a370fb8 
sgl-project#45289)

Loader keeps canonical [gate, up] for the per-tensor global scales while
weights/block scales follow the kernel's physical row order; TRT-LLM SwiGLU
beta gains the 1/g1_alphas_up correction and the gate/up scale-mismatch
warning. Reverts sgl-project#44858's call-site workaround -- which THIS fork never had
(its skip-note in the 10-19 review record is now stale), so every sgl-project#44858 form
is absent here by construction, verified: _compute_gemm1_alphas is byte-
identical to the pre-sgl-project#44858 AND post-sgl-project#45289 state, and
_compute_trtllm_gated_act_params exists nowhere in the tree.

Picks/adaptations (cherry-pick -x 5b1dbc2):
  * fused_moe_triton/layer.py landed upstream's final form (the trtllm scale
    re-read block deleted; ModelOpt NVFP4 excluded from the reorder set).
  * modelopt_quant.py TRT-LLM params inlined with the warning + _up beta.
  * Seam carve-out: _align_fp4_moe_weights_for_flashinfer_trtllm added to the
    NVFP4 method (upstream shape from sgl-project#44666, which the oracle test mocks as
    its GPU-packing stub) wrapping the fork's identical inline call -- the
    oracle's mock.patch.object(layer.quant_method, ...) seam; behavior for a
    real boot is unchanged.
  * Fork guard: load_up_proj_weight_first reads the new
    use_flashinfer_trtllm_weight_layout term through getattr(..., False);
    entry-247's b12x CPU contract suite builds the method with __new__ and
    6 tests regressed (AttributeError) without the default. b12x itself still
    resolves False everywhere (not trtllm/cutlass/cutedsl-v2), so
    b12x_w4a16_w13_layout(up_first=False) -> "w31" (entry-247 defect-(3)
    derivation) is preserved; test_up_proj_order_is_not_flipped_for_b12x green.
  * ORACLE test_modelopt_nvfp4_moe_scales.py landed upstream's final form
    byte-identical EXCEPT the _nvfp4_loader_layer stub carries the placement
    attrs this fork's SM120 small-M gate reads at process_weights time
    (num_local_experts/num_fused_shared_experts/moe_ep_rank/moe_ep_size and
    _num_global_routed behind the property; entry-225's precedent for exactly
    this stub gap). Assertions untouched. 20 passed + 44 subtests. RED PROOF:
    reverting the beta line to the pre-sgl-project#45289 1.0/g1_alphas fails
    test_trtllm_post_load_activation_parameters[situ] here.
  * backends/test_flashinfer_trtllm_gen_moe_backend.py: sgl-project#45289's change there
    reverts sgl-project#44858's test edits -> literal no-op for this tree; file verified
    byte-identical to the upstream post-image.

Battery delta vs baseline: scales file 17P/30sub -> 20P/44sub (new oracle
tests, green); b12x 67P/46sub after the getattr guard; deferred-scales,
modelopt_nvfp4, dispatch, loader, smallm, dispatcher, kernels_namespace
(76P, inventory guard), fp8 blockwise suites unchanged.

GPU-WINDOW DEBT: end-to-end scale-ordering numerics on the real NVFP4
checkpoint (gate/up alpha mapping through a live TRT-LLM/CUTLASS forward);
this commit pins it at CPU contract level only.

(cherry picked from commit 5b1dbc2)

Co-authored-by: port-moe-contracts subagent
(cherry picked from commit 4ec7fdb603ebc300b1177364b4928554ca0a0610)
detain added a commit to detain/sglang that referenced this pull request Oct 29, 2026
@detain
MERGE: fifth merge taken from sgl-project/sglang main @ 2a6c204 (+1,1…
ae6ee52 
…20 commits since 12dc623 @ merge-sgl-project#4)

Conflicts: 167 files (149 content / 14 add/add / 4 modify/delete) at the settled
309 tip d28e7cd -- dossier predicted 146 +/-25; the +21 shift is entries
275-309 landing qsa-ring/hicache-ctrl/gdn/moe/llguard deltas into the conflict
zone. By class: A 13 (auto-takes-upstream, dedupe-verified), B 80, C 56, D 18.
102 TODO(merge5) re-plant/fold markers planted: B1 x15, B2 x29, B3 x12, B4 x23,
B5 x23. Re-plant stock: 117 files under /tmp/opencode/merge5-dossier/replant/
(76 fork-hunks patches, 19 upstream-side patches, 14 AA fork copies, 8
preremove snapshots) -- sessions 2/3 re-plant from real code, not memory.

D-list outcomes (must-not-auto-resolve, handled personally):
- sgl-project#45289-vs-sgl-project#44858 (entry 301): VERIFIED, not re-resolved -- the gate/up-order
  final form is in origin/main (5b1dbc2 in range); scales test taken as
  upstream oracle, fork 191-line version saved to replant/ for B5 diffing.
  sgl-project#44858's call-site shape CANNOT survive (our base never had it).
- entry-230 dispatcher vs sgl-project#44993/sgl-project#44994: took upstream structure in
  token_dispatcher/standard.py; published-value CONTRACT lives upstream via
  get_moe_runner_backend() in the LoRA-EP guard; fork's warn-once mismatch
  diagnostic re-plants (B5 marker). UNION GATE: standard_dispatcher 16P (was
  12P, suite grew), model_overrides 121P/1S/210sub (was 112P/1S/181sub),
  fold probe 6P/2F (upstream added a runner_backend arg to the predicate --
  B5 re-pins the probe).
- entry-219 gauge names vs sgl-project#43727 return + sgl-project#44968: PER-POOL NAMES KEPT as the
  serving schema -- hicache_host_pool_{used,total}_tokens{pool} dataclass
  fields, Gauge registrations and labels(pool=) emission re-restored over the
  #4697x structure; hicache_host_pool_evicted_tokens_total already survived
  auto-merge; producer dict restored by union in metrics_reporter. ANTI-NEEDLE
  SATISFIED: old mamba-only gauges are NOT the serving schema.
- entry-221 eager-direct add/add: DEDUPED to upstream (bodies identical except
  est_time 15->9); MEASUREMENT DEBT RECORDED: prefill t/s must be re-measured
  post-B4 before trusting the est_time.
- entry-168 D2H fence vs sgl-project#44800: the fence is ALREADY UPSTREAM (both
  load_fence_stream + write_fence_stream with the load-thread-corruption
  comment in origin/main's cache_controller.py) -- nothing re-planted; the two
  real regions resolved as import-union + comment wording (merge sgl-project#4 stale-tail
  pattern avoided). Fork hunks still saved to replant/ (belt).
- entry-209 clamp + 216/218 ordering + 288-295 hicache-ctrl-era vs #46970x KV
  rewrite: ADOPTED-UPSTREAM across unified_radix_cache/hicache_storage/
  hybrid_cache_controller/memory_pool/kv_cache_builder; EVERY fork hunk
  extracted to replant/*.fork-hunks.patch; B3 re-plant markers planted.
  sgl-project#45703's short-full-match seam arrives with upstream.
- sgl-project#45090 SGLANG_ENABLE_UNIFIED_RADIX_TREE: REMOVAL TAKEN (fork had already
  moved it to _DeprecatedEnv; now gone). grep: ZERO .py readers repo-wide;
  2 upstream-owned docs jsx snippets still export it (deepseek-v4, glm-5.2 --
  upstream's own debt). OPERATOR-OWED CHECK: confirm no /root harness exports
  it (unreadable from this seat).
- sgl-project#39727 unified L3 key: VERIFIED OPT-IN -- merged hicache_storage.py keeps
  the DEFAULT legacy suffix composition _<model>_<tp_rank>_<tp_size>
  (content-anchor: HiCacheFile._init_config_suffix, raise-guarded when
  _unified_chunk_spans is set). ANTI-NEEDLE: no silent key-scheme switch;
  warm /root/hicache_store contents stay addressable.

Modify/delete (4, all victims of upstream sgl-project#45672 test prune + URC split):
- test_unified_radix_cache_unittest.py: REMOVED (adopted split) -- its bare
  unified_tree_core_inspector imports are unreachable post-sgl-project#45236 re-home;
  fork source + hunks saved; B2 marker planted in the new core test.
- test_fused_moe_triton_config.py: REMOVED -- imports common_utils, absent
  from both sides post-sgl-project#45672; the 2 live entry-3/235 tests re-plant in B2.
- test_scheduler_batch_is_full_latch.py: REMOVED -- fork delta was 3 stub-attr
  lines documenting sgl-project#46643's absorbed arms; upstream deletion correct.
- test_qsa_kv_pool.py: KEPT -- fork's 147 added lines are entry-187/200
  sharding + 285-287 QSA-ring tests with NO upstream replacement;
  qsa_kv_pool.py sides converged byte-identical; collects clean. B2 marker.

Cascading silent-auto-merge repairs (classes not in the conflict set but
broken by auto-merge + union splices -- all found by gates, now fixed):
kernels/ops/activation/__init__.py (dead _CUDA registration dup), kernels/ops/
diffusion/__init__.py (5-tuple _SPECS entry), kernels ops embeddings/moe/
attention/mamba __init__ (KernelSpec fields removed upstream: description/
format_signature stripped, CapabilityRequirement->CUDA_ONLY), base_runner/
eager_runner cp_extend + shared_read_event imports retargeted per upstream's
"no imports in runner_utils/__init__" contract, and 4 fork helpers grafted
back onto their take-upstream homes because merged code/tests still import
them (memory_pool.spec_intermediate_bytes_per_req, kv_cache_builder.
_adjust_hicache_size_for_draft + _device_pool_size_per_token,
hybrid_cache_controller.local_restorable_mask, validation_hook.
validate_replayssm_spec_algorithm, arg_groups/moe_hook b12x validator).
flashinfer_cutlass.py flipped to FORK-OWNED (b12x route is fork-only, 760 vs
95 lines; upstream sgl-project#44993 hunks = B5 fold patch saved).

Silent-semantics notes: sgl-project#46127/sgl-project#44912 riders arrived via per-PR picks (union);
sgl-project#45520 (entry 302) is STILL OPEN upstream -- entry-230 fail-safe layer stays
as entry 303 records; if a future sync brings it, re-apply the fail-safe reads.
qwen4_exp holder-contract fused (ctor takes optional batch).

Gates at commit: import sglang OK (CVD=9, prod venv); check_registered_tests/
check_no_bare_pytest_main/check_no_registered_tests_in_package rc=0; ruff F821
+E9+format clean on the resolution set; collect-only over unit/{mem_cache,
model_executor,distributed,layers/moe} = 4337 nodes, 3 ERRORS: 2x cargo/torch-
sys libtorch probe (REPRODUCED IDENTICALLY AT THE PRE-MERGE TIP -- env issue,
not merge damage) + test_hicache_async_ready_sync.py (entry-253-era attribute
gone with the URC rewrite -- EXPECTED pre-B3).

sessions 2/3 owed: B2 test re-home, B3 mem_cache+rust re-plant (+hicache-ctrl
re-ports), B4 scheduler/admission, B5 moe/quant/dispatcher, B6 environ+C0a+
CAVEATS re-pin. PROD BOOT FORBIDDEN until B3 completes (209 clamp + 218
ordering re-plant).
detain added a commit to detain/sglang that referenced this pull request Oct 30, 2026
@detain
MERGE-5 B5-5: moe-battery red-file attribution + three test-zone re-a…
a8a6c62 
…nchors -- entries 244/247-era surfaces, 301, sgl-project#44842, sgl-project#42973

Battery damage classes re-derived per-file POST-MERGE (CVD=9, prod venv,
PYTHONPATH=python) with the B5-3/B5-4 code state:

(1) test_modelopt_nvfp4_moe_scales 2F (SUBFAILED pair): merge took
  upstream's sgl-project#45289 test file and DROPPED the fork's entry-244 stub block
  (the small-M workspace gate reads placement attrs in
  process_weights_after_loading regardless of kernel support -- kwargs
  evaluate first). Stub block re-planted VERBATIM from the 309 tip.
  20P/44sub (>= the pre-merge 20P/42sub: upstream grew two subtests).
(2) test_fp4_gemm_backend_sm120 3F: upstream sgl-project#44842 (in-range) grew the
  autotune gate with get_parallel().enable_kv_cache_sharding (sgl-project#42973) +
  the FP8->MXFP8 module-tree pre-check; the gate tests' stubs predate
  both. Test re-anchored (get_parallel stub + empty module tree); the
  three beliefs (b12x keeps the pass, cutlass autotunes, marlin doesn't)
  unchanged. 12P/0F.
(3) test_mem_pool_ep_unit 1F: the DOCUMENTED tip-era class, now attributed
  exactly: the test's id-contract predates entry-247 -- dispatcher skip
  TRUE for b12x vs the stale expected-False. Set re-pinned:
  dispatcher_global_ids += FLASHINFER_B12X_W4A16 (its expert_map contract,
  the very arm B5-3 re-planted into _keeps_global_expert_ids) and the LoRA
  subtraction gains b12x (no LoRA path). EXPERIMENTAL_SGL_TRTLLM verified
  HARMLESS without its removed arm: is_flashinfer_trtllm() covers the
  value by design (utils.py:212-217), confirming B5-3's
  upstream-churn verdict on that branch. 49P/0F/4sub -- class CLOSED, one
  fewer env-class than the pre-merge baseline.

ENV CLASSES CONFIRMED (not new, not fixable here):
  trtllm_gen 8F == documented 8F; moe_fused_gate 52F == documented ~52F
  ALL 'No CUDA GPUs are available'; triton_fused_moe 96F == documented
  ~96F; round_robin 38F (documented 37 + 1 upstream-grown CUDA case, ALL
  'No CUDA GPUs'); mega_moe_deepgemm 1F == documented 1F (test's deep_gemm
  fake vs mxfp4.py's in-range wrapper import -- NOT my redirect: fails
  identically at b3207c7 with mega_moe pre-B5-3; B2 marker :905 covers
  the test-side fix).

GATE RE-CHECK: standard_dispatcher 21P (grew again, green),
runner_backend 16P, fold probe 9P, overrides 121P/1S/210sub.
@Oseltamivir Oseltamivir mentioned this pull request Nov 5, 2026
Open
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Reviewers

@ch-wan ch-wan Awaiting requested review from ch-wan ch-wan is a code owner
@BBuf BBuf Awaiting requested review from BBuf BBuf is a code owner
@Edwardf0t1 Edwardf0t1 Awaiting requested review from Edwardf0t1 Edwardf0t1 is a code owner
@FlamingoPg FlamingoPg Awaiting requested review from FlamingoPg FlamingoPg is a code owner
@AniZpZ AniZpZ Awaiting requested review from AniZpZ AniZpZ is a code owner
@HaiShaw HaiShaw Awaiting requested review from HaiShaw HaiShaw is a code owner
@b8zhong b8zhong Awaiting requested review from b8zhong b8zhong is a code owner
@OrangeRedeng OrangeRedeng Awaiting requested review from OrangeRedeng OrangeRedeng is a code owner
@Alisehen Alisehen Awaiting requested review from Alisehen Alisehen is a code owner
@mmangkad mmangkad Awaiting requested review from mmangkad
@merrymercy merrymercy Awaiting requested review from merrymercy merrymercy is a code owner
@Ying1123 Ying1123 Awaiting requested review from Ying1123 Ying1123 is a code owner
@ispobock ispobock Awaiting requested review from ispobock ispobock is a code owner

Assignees

No one assigned

Labels

blackwell SM100/SM120 quant LLM Quantization run-ci CI: run the baseline test suite on this PR run-ci-extra CI: also run the extra suite (requires run-ci)

Projects

None yet

Milestone

No milestone

Development

Successfully merging this pull request may close these issues.

1 participant

@Fridge003