[None][fix] Enable INT8 weight-only (W8A16) MoE for non-gated activations

[Dorijan10]

Description

The INT8 weight-only per-channel (W8A16) fused-MoE path assumes gated activations (SwiGLU/GeGLU), which prevents non-gated MoE models (squared-ReLU, e.g. Nemotron) from using it; even though the underlying CUTLASS mixed-GEMM kernels already support the non-gated layout. Non-gated experts are already handled for FP8 and NVFP4 (see the Nemotron weight mapper and the FP8 fused-MoE method); this PR brings the INT8 weight-only path to parity.

• cpp/tensorrt_llm/thop/moeOp.cpp: the woq validation hardcoded fc1.inter == 2 * fc2.inter; now conditioned on isGatedActivation(), mirroring the existing non-woq branch.
• tensorrt_llm/_torch/modules/fused_moe/quantization.py (INT8WoqPerChannelFusedMoEMethod): buffer sizing, weight loading, and scale loading assumed the doubled (gate+up) layout; now handle the single up-projection when the gate weight is absent, mirroring the unquantized fused-MoE path's existing non-gated handling.

Gated models are unaffected; they retain the original code path. W8A16 checkpoints are produced the standard way (ModelOpt int8_wo, or offline per-output-channel quantization); no new runtime quantizer is introduced. On Ampere (e.g. A100), where FP8/NVFP4 are unavailable, INT8 weight-only is the natural weight-compression format for MoE inference.

Test Coverage

Validated on NVIDIA-Nemotron-3-Nano-30B-A3B (non-gated, Relu² experts) on A100-SXM4-80GB, served via trtllm-serve from a pre-quantized W8A16 checkpoint (hf_quant_config.json → quant_algo: W8A16, non-expert linears excluded). Built and validated against current main with only the two files in this PR.

Throughput: vllm bench serve (random, ISL/OSL 512/512, concurrency 32, --ignore-eos, mean of 3):

Metric	BF16	W8A16 (this PR)	Δ
Output throughput	~1203 tok/s	~1690 tok/s	+40%
Median TPOT	~23.6 ms	~16.5 ms	−30%
Weight memory	~58.9 GB	~31.5 GB	−27 GB

Lossless within eval noise on gsm8k.

Validated end-to-end on Nemotron. Includes a unit test (test_fused_moe_int8_woq_per_channel_non_gated) in tests/unittest/_torch/modules/test_fused_moe.py covering the non-gated INT8 weight-only per-channel MoE path (single-width buffer, absent gate projection), for float16 and bfloat16.

Summary by CodeRabbit

• Bug Fixes

  • Improved support for INT8 weight-only per-channel MoE so non-gated and gated activations are validated correctly.
  • Fixed handling of weight and scale shapes for MoE setups that do not use gated activations.
  • Updated loading behavior so checkpoints with only the main projection are accepted when applicable.

• Tests

  • Added coverage for a non-gated INT8 per-channel MoE path to verify correct output shape and finite results.

[coderabbitai]

📝 Walkthrough

Walkthrough

INT8 weight-only quantization per-channel MoE gains non-gated activation support. Python weight buffer and scale shapes are switched from a hardcoded ×2 intermediate size to expand_intermediate_size_per_partition, with conditional w3/w1 concatenation gated on is_gated_activation. C++ TORCH_CHECK calls in runMoe and runMoeMinLatency are updated to branch on isGatedActivation. A new unit test covers the squared-ReLU (non-gated) path end-to-end.

Changes

INT8 WOQ Non-Gated MoE Support

Layer / File(s)	Summary
Weight buffer and scale shapes for non-gated path tensorrt_llm/_torch/modules/fused_moe/quantization.py	create_weights switches w3_w1_weight and fc31_weight_scale shapes to use expand_intermediate_size_per_partition; load_expert_w3_w1_weight conditionally concatenates w3 with w1 only when w3_weight is non-empty; load_quant_scales conditionally stacks w3 and w1 scales only when is_gated_activation and w3 scales are present, otherwise writes w1 scales alone.
C++ tensor-dimension validation for gated vs non-gated cpp/tensorrt_llm/thop/moeOp.cpp	runMoe replaces an unconditional ×2 TORCH_CHECK with a branch on isGatedActivation; runMoeMinLatency removes the fixed ×2 check and adds a new post-finalization validation block that selects the correct ×2 or ×1 relationship based on mUseINT8WoqPerChannel and isGatedActivation.
Unit test for non-gated INT8 WOQ forward pass tests/unittest/_torch/modules/test_fused_moe.py	Adds ActivationType import and a new skipped test test_fused_moe_int8_woq_per_channel_non_gated that builds a CutlassFusedMoE with Relu2 (no w3 gate weights), asserts intermediate_size_expand_ratio == 1 and is_gated_activation is False, runs forward under autotune(), and checks output shape and finiteness.

Estimated code review effort

🎯 3 (Moderate) | ⏱️ ~20 minutes

🚥 Pre-merge checks | ✅ 5

✅ Passed checks (5 passed)

Check name	Status	Explanation
Description check	✅ Passed	The description covers the issue, solution, and test coverage; the PR checklist section is omitted but the core template is complete.
Docstring Coverage	✅ Passed	No functions found in the changed files to evaluate docstring coverage. Skipping docstring coverage check.
Linked Issues check	✅ Passed	Check skipped because no linked issues were found for this pull request.
Out of Scope Changes check	✅ Passed	Check skipped because no linked issues were found for this pull request.
Title check	✅ Passed	The title clearly summarizes the main change: enabling INT8 weight-only MoE support for non-gated activations.

✨ Finishing Touches

🧪 Generate unit tests (beta)

• Create PR with unit tests

---

_{Comment @coderabbitai help to get the list of available commands.}

[coderabbitai]

Actionable comments posted: 2

🧹 Nitpick comments (1)

tests/unittest/_torch/modules/test_fused_moe.py (1)

2981-3049: 📐 Maintainability & Code Quality | 🔵 Trivial | ⚡ Quick win

Coverage is still insufficient for the min-latency branch.

This test covers the regular CutlassFusedMoE.forward path well, but this PR also changes cpp/tensorrt_llm/thop/moeOp.cpp::runMoeMinLantency. Please add a companion case in tests/unittest/_torch/modules/test_fused_moe.py that enables min-latency on the same ActivationType.Relu2 + QuantAlgo.W8A16 setup, otherwise the new activation/layout validation there can regress unnoticed. As per path instructions, tests/**: “Act as a QA engineer reviewing test changes and coverage for TensorRT-LLM. Keep feedback actionable: suggest concrete list file names and whether coverage is sufficient, insufficient, or needs follow-up outside the PR.”

🤖 Prompt for AI Agents

Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.

In `@tests/unittest/_torch/modules/test_fused_moe.py` around lines 2981 - 3049,
Coverage is insufficient for the min-latency branch in runMoeMinLantency, so add
a companion test alongside test_fused_moe_int8_woq_per_channel_non_gated in
test_fused_moe.py that uses the same ActivationType.Relu2 and QuantAlgo.W8A16
setup but explicitly enables min-latency. Reuse the existing CutlassFusedMoE,
QuantConfig, and RoutingMethod setup so the new case exercises the same
non-gated INT8 weight-only path while validating the min-latency-specific
activation/layout checks. Keep the assertions on output shape and finiteness so
regressions in the new branch are caught.

Source: Path instructions

🤖 Prompt for all review comments with AI agents

Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.

Inline comments:
In `@cpp/tensorrt_llm/thop/moeOp.cpp`:
- Around line 827-858: `runMoeMinLantency()` now validates INT8-WOQ tensor
layout, but it still derives `hidden_size` and `inter_size` using the non-WOQ
index order before calling `getWorkspaceInfo(...)` and `runMoe(...)`. Update the
size selection logic in `runMoeMinLantency()` to branch on
`mUseINT8WoqPerChannel` and use the same fc1/fc2 dimension order as the new
validation block, so the downstream workspace and execution paths receive the
correct hidden/inter sizes for both WOQ and non-WOQ cases.

In `@tensorrt_llm/_torch/modules/fused_moe/quantization.py`:
- Around line 1358-1369: The gated MoE weight check in quantization.py is using
w3_weight.numel() before sharding, which breaks supported non-tensor checkpoint
entries handled by load_weight_shard(). Update the logic around the
w3_weight_shard/w1_weight_shard construction to first call load_weight_shard()
on w3_weight when present, then decide whether to concatenate based on the
sharded result’s emptiness; keep the non-gated fallback in the same fused-MoE
loading path.

---

Nitpick comments:
In `@tests/unittest/_torch/modules/test_fused_moe.py`:
- Around line 2981-3049: Coverage is insufficient for the min-latency branch in
runMoeMinLantency, so add a companion test alongside
test_fused_moe_int8_woq_per_channel_non_gated in test_fused_moe.py that uses the
same ActivationType.Relu2 and QuantAlgo.W8A16 setup but explicitly enables
min-latency. Reuse the existing CutlassFusedMoE, QuantConfig, and RoutingMethod
setup so the new case exercises the same non-gated INT8 weight-only path while
validating the min-latency-specific activation/layout checks. Keep the
assertions on output shape and finiteness so regressions in the new branch are
caught.

🪄 Autofix (Beta)

Fix all unresolved CodeRabbit comments on this PR:

• Push a commit to this branch (recommended)
• Create a new PR with the fixes

---

ℹ️ Review info

⚙️ Run configuration

Configuration used: Path: .coderabbit.yaml

Review profile: CHILL

Plan: Enterprise

Run ID: 1325b4ba-1aba-4627-83d5-d753fb661b91

📥 Commits

Reviewing files that changed from the base of the PR and between eb0cbdb and 3573e0e.

📒 Files selected for processing (3)

• cpp/tensorrt_llm/thop/moeOp.cpp
• tensorrt_llm/_torch/modules/fused_moe/quantization.py
• tests/unittest/_torch/modules/test_fused_moe.py

[Dorijan10]

One note on test placement: I added test_fused_moe_int8_woq_per_channel_non_gated to tests/unittest/_torch/modules/test_fused_moe.py, mirroring the existing test_fused_moe_int8_woq_per_channel. I see that file is now marked deprecated in favor of the unified framework in tests/unittest/_torch/modules/moe/test_moe_backend.py.

The unified framework is currently gated-only for W8A16. W8A16QuantizeUtil.create_weights always builds a gate (w3) projection and W8A16RefGatedMLPFusedMoE asserts Swiglu; so adding non-gated W8A16 coverage there needs a non-gated branch in the quantize util plus a non-gated reference module, analogous to the W4A8_MXFP4_FP8 unit-test support in #13401.

Happy to extend the unified framework that way as a follow-up if you would prefer the coverage there rather than in test_fused_moe.py.

On the min-latency test nitpick: min_latency_mode is hardcoded to False at all CutlassFusedMoE runner call sites and is not exposed through the module's forward, so a companion test reusing the existing CutlassFusedMoE setup cannot drive runMoeMinLantency, it always routes through runMoe. Exercising it would require the low-level run_moe_min_latency op with hand-constructed WOQ weights/scales/workspace, a lower layer than this test file operates at. The min-latency fix mirrors the INT8-woq size swap already present in runMoe and runGemmProfile, so the three paths are consistent by inspection; I have noted min-latency coverage as part of the same follow-up above.

[Dorijan10]

Moved the coverage into the unified framework as suggested.

test_moe_backend.py now exercises the non-gated (squared-ReLU) INT8 W8A16 path against the framework's dequantized reference, so it's a real correctness check rather than the earlier shape/finiteness smoke test. Concretely: W8A16QuantizeUtil.create_weights gained a non-gated branch and W8A16RefGatedMLPFusedMoE now forwards activation_type and loads the single up-projection when non-gated, mirroring the existing NVFP4 element-wise support; W8A16 is added to the element-wise params, CUTLASS-only. The superseded smoke test in test_fused_moe.py is removed.

Validated on A100: 20 non-gated + 20 gated W8A16 cases pass (float16 + bfloat16, CUTLASS), non-gated added, gated unchanged.

Pushed in 2d71065.

[leslie-fang25]

LGTM, thanks for the PR

[Dorijan10]

Thanks again @leslie-fang25 for the review and approval. It looks like the only thing left before merge is a maintainer approving the pending workflows. blossom-ci is showing "waiting for status to be reported." Would you be able to trigger that, or point me to the right person? The branch is slightly behind main but merges cleanly, and I would be happy to update it or address anything CI surfaces.

[Dorijan10]

While validating this at tensor-parallel sizes > 1, I found and fixed a related pre-existing bug in the same method (missing 64-alignment padding of the sharded intermediate). Made a follow-up in #16198, stacked on this branch, with a regression test.

Note that main has moved substantially in moeOp.cpp since approval, however this branch still merges cleanly today. @leslie-fang25, anything needed from my side to get blossom-ci triggered here?

[leslie-fang25]

/bot run

[leslie-fang25]

@Dorijan10 sorry for the late response, I have triggered the CI. But I am afraid I am also don't have the full access to run all the cases now. Let's see what's the result for now.

[tensorrt-cicd]

PR_Github #61457 [ run ] triggered by Bot. Commit: 2d71065 Link to invocation

[tensorrt-cicd]

PR_Github #61457 [ run ] completed with state FAILURE. Commit: 2d71065
/LLM/main/L0_MergeRequest_PR pipeline #49678 completed with status: 'FAILURE'

CI Report

⚠️ Action Required:

• Please check the failed tests and fix your PR
• If you cannot view the failures, ask the CI triggerer to share details
• Once fixed, request an NVIDIA team member to trigger CI again

CI Agent Failure Analysis

Link to invocation

[tensorrt-cicd]

PR_Github #61770 [ run ] completed with state SUCCESS. Commit: 801e981
/LLM/main/L0_MergeRequest_PR pipeline #49971 completed with status: 'FAILURE'

CI Report

⚠️ Action Required:

• Please check the failed tests and fix your PR
• If you cannot view the failures, ask the CI triggerer to share details
• Once fixed, request an NVIDIA team member to trigger CI again

CI Agent Failure Analysis

Link to invocation

[Dorijan10]

Thanks @leslie-fang25 for triggering the CI again. This time all other checks succeeded apart from the blossom-ci so I would really need a way to see which tests failed and why. I have few ideas, but seeing the CI report would make it clear for me.

@xxi-nv making me a contributor should help here I think.

[Dorijan10]

@leslie-fang25 @xxi-nv I think I automatically become a Contributor once the PR gets merged.

However, that means I still cannot get access right now to the CI report from the blossom-ci run. I think for that to happen I would need to be assigned as Collaborator. It would be very useful if I could see the CI run results and fix the issues.

[leslie-fang25]

Here is the summary of the CI failures

Root Cause(s)
RC-1: Test container/JIT cache path misconfigured (/root/.triton/cache missing)
Category: Infrastructure/CI environment
Failing job: LLM/job/main/job/L0_Test-x86_64-Single-GPU #5476
Failed test(s): Many unrelated B200/DGX_B200 tests, including:
DGX_B200-PyTorch-1/accuracy/test_llm_api_pytorch.py::TestLlama3_1_8B::test_nvfp4_streaming[stream_interval_64]
DGX_B200-PyTorch-1/kv_cache/test_kv_cache_v2_scheduler.py::TestKVCacheV2Llama::test_chunked_prefill_with_eviction
unittest/DGX_B200-PyTorch-1/unittest/_torch/modules/fused_moe/test_deepgemm_fused_expand_quant.py::*
unittest/DGX_B200-AutoDeploy-1/...
B300-PyTorch-2 MoE/executor/LoRA tests
Error:
FileNotFoundError: [Errno 2] No such file or directory: '/root/.triton/cache'

[TRT-LLM] [E] [executor] Failed to initialize executor on rank 0:
[Errno 2] No such file or directory: '/root/.triton/cache'

RuntimeError: Executor worker returned error
Additional related cache failure:

RuntimeError: Assertion error (.../deepgemm-src/.../utils/system.hpp:76):
Failed to make directory: /root/.deep_gemm/tmp, created: false, value: 17
Likely cause: The test runtime is trying to use /root for Triton/torch-inductor/DeepGEMM JIT caches, but the container/user environment cannot create or access those paths. This causes broad failures across attention, visual generation, executor, MoE, and AutoDeploy tests — not a targeted regression in the PR’s W8A16 MoE logic.
Related source files: CI/container environment or test runtime configuration, not PR source files.
Is waived?: Not applicable — this is an infrastructure/environment failure, not a new waived test assertion.
RC-2: SBSA multi-GPU Slurm job aborted/cancelled after task failure
Category: Infrastructure/Slurm
Failing job: LLM/job/main/job/L0_Test-SBSA-Multi-GPU #2664
Failed test(s): No real pytest failures identified; result collection showed empty unfinished_test.txt.
Error:
srun: error: nvl72096-T01: task 0: Exited with exit code 1
srun: Terminating StepId=5620714.0
*** STEP 5620714.0 ON nvl72096-T01 CANCELLED ... DUE TO TASK FAILURE ***

[INFRA-RETRY] GB200-8_GPUs-2_Nodes-PyTorch-1: SLURM job 5620714 terminal state=FAILED
[INFRA-RETRY] GB200-8_GPUs-2_Nodes-PyTorch-2: SLURM job 2783081 terminal state=FAILED
Slurm accounting later showed cancellations:

5620714      ... FAILED    143:0    nvl72096-T[01-02]
5620714.0    ... CANCELLED 0:15     nvl72096-T[01-02]
Likely cause: Slurm infrastructure/task failure on GB200 multi-node SBSA stages. Downstream stages were aborted/cancelled as a consequence.
Related source files: None identified.
Is waived?: Not applicable.
Recommended Actions
Re-run CI after the CI/test environment is fixed or once a clean B200/GB200 pool is available.
CI/infra owners should verify cache setup in the test containers:
Ensure HOME, TRITON_CACHE_DIR, TORCHINDUCTOR_CACHE_DIR, and DeepGEMM temp/cache paths point to writable directories.
Avoid using /root/.triton/cache and /root/.deep_gemm/tmp when tests run as a non-root or rootless/containerized user.
No PR code change is recommended from this run. PR #15550 modifies:
cpp/tensorrt_llm/thop/moeOp.cpp
tensorrt_llm/_torch/modules/fused_moe/quantization.py
MoE-related unit tests These do not explain the broad /root/.triton/cache failures across unrelated components.
After infra rerun, if only MoE/W8A16 tests remain failing, then investigate the PR-specific changes; current evidence does not support that.

Based on the first suggestions, I have rebased this PR and will trigger the CI again with the latest codebase.

[leslie-fang25]

/bot run

[tensorrt-cicd]

PR_Github #62648 [ run ] triggered by Bot. Commit: ba0c487 Link to invocation

[tensorrt-cicd]

PR_Github #62648 [ run ] completed with state FAILURE. Commit: ba0c487
/LLM/main/L0_MergeRequest_PR pipeline #50788 completed with status: 'FAILURE'

CI Report

⚠️ Action Required:

• Please check the failed tests and fix your PR
• If you cannot view the failures, ask the CI triggerer to share details
• Once fixed, request an NVIDIA team member to trigger CI again

CI Agent Failure Analysis

Link to invocation

[Dorijan10]

Hi @leslie-fang25 thanks for the triggering CI and rebase. I have checked ba0c487 locally and it preserves the changes exactly, so nothing changed here on my side. And all actions checks are green.

What I can see from the first CI report you sent is this:

RC-1: Test container/JIT cache path misconfigured (/root/.triton/cache missing)
Category: Infrastructure/CI environment

I think that when TensorRT runs during CI, it compiles some GPU kernels on the fly and caches them to disk. The test containers were pointed at /root/.triton/cache, from the looks of it a directory that did not exist or was not writable. So every test that needed to compile a kernel died with FileNotFoundError. Which would make sense as to why L0 Pipeline would fail then. I think this needs a fix on the CI environment side rather than a change in the PR.

So this was in the CI report from #49971 CI run, can you send or check if the same error appeared in the #50788? I assume it did since all that changed between the two runs is the merge and that turned out to be fine!

[leslie-fang25]

/bot run --disable-fail-fast

[leslie-fang25]

some failures looks like Irrelevant, will trigger a rerun for it.

[tensorrt-cicd]

PR_Github #62898 [ run ] triggered by Bot. Commit: ba0c487 Link to invocation

[tensorrt-cicd]

PR_Github #62898 [ run ] completed with state FAILURE. Commit: ba0c487
/LLM/main/L0_MergeRequest_PR pipeline #51021 completed with status: 'FAILURE'

CI Report

⚠️ Action Required:

• Please check the failed tests and fix your PR
• If you cannot view the failures, ask the CI triggerer to share details
• Once fixed, request an NVIDIA team member to trigger CI again

CI Agent Failure Analysis

Link to invocation

[Dorijan10]

Thanks for triggering CI again @leslie-fang25. Could you share the report for #51021? Since it had fail fast disabled it should be the complete picture, and it would be much more useful than the older report.

I think the problem might still be the container environment rather than the code, with the JIT cache paths under /root not being writeable. On my side the merge preserved the change exactly, and W8A16 is listed as SM >= 80 in CutlassFusedMoE.can_implement, which matches it running fine on SM120 in my own validation. So I do not see an arch gap on Blackwell.

[leslie-fang25]

Here is the report, seems unrelated failures with this PR, will take a re-try.

Root Cause(s)
RC-1: Unrelated KV-cache scheduler test timed out/failed on B200
Category: Test failure / timeout
Failing job: LLM/job/main/job/L0_Test-x86_64-Single-GPU #201 on Jenkins-Sub-4
Failed test(s):
DGX_B200-PyTorch-3/kv_cache/test_kv_cache_v2_scheduler.py::TestKVCacheV2Llama::test_chunked_prefill_with_eviction
Error:
Pytest failed in SLURM job 1567638 for DGX_B200-PyTorch-3

DGX_B200-PyTorch-3/kv_cache/test_kv_cache_v2_scheduler.py::TestKVCacheV2Llama::test_chunked_prefill_with_eviction
Error in file ...-slurm_run.sh on line 82:
... pytest -vv --timeout-method=thread --apply-test-list-correction --timeout=3600
--test-prefix=DGX_B200-PyTorch-3
--test-list=.../l0_b200.txt --splitting-algorithm least_duration --splits 9 --group 3 ...
(exit 1)

srun.real: error: nsc-svg-slurm-1-gpu-63: task 0: Exited with exit code 1
The post-processing also found DGX_B200-PyTorch-3/unfinished_test.txt was non-empty and generated results-timeout.xml, indicating the failed case was left unfinished.
Likely cause: The failing test is a KV-cache scheduler test that appears to have hung or exited through the pytest timeout path. It is unrelated to the PR’s MoE/INT8 weight-only changes.
Related source files:
Failing test area: tests/integration/defs/kv_cache/test_kv_cache_v2_scheduler.py
PR changed files are unrelated:
cpp/tensorrt_llm/thop/moeOp.cpp
tensorrt_llm/_torch/modules/fused_moe/quantization.py
tests/unittest/_torch/modules/moe/quantize_utils.py
tests/unittest/_torch/modules/moe/test_moe_backend.py
Is waived?: No direct waive entry found for this exact test/stage.
RC-2: SBSA multi-node GB200 MPI/UCX endpoint resolution failure
Category: Infrastructure/Transient
Failing job: LLM/job/main/job/L0_Test-SBSA-Multi-GPU #3054 on Jenkins-Sub-1
Failed test(s):
Stage was running:
GB200-8_GPUs-2_Nodes-PyTorch-1/accuracy/test_llm_api_pytorch.py::TestDeepSeekR1::test_nvfp4_multi_gpus[throughput]
But the root cause is MPI/UCX communication setup failure, not the test assertion itself.
Error:
[nvl72d048-T09:1042549] pml_ucx.c:178  Error: Failed to receive UCX worker address: Not found (-13)
[nvl72d048-T09:1042549] pml_ucx.c:482  Error: Failed to resolve UCX endpoint for rank 4
[nvl72d048-T09:1043188] pml_ucx.c:482  Error: Failed to resolve UCX endpoint for rank 5
[nvl72d048-T09:1043185] pml_ucx.c:482  Error: Failed to resolve UCX endpoint for rank 6
[nvl72d048-T09:1043217] pml_ucx.c:482  Error: Failed to resolve UCX endpoint for rank 7

mpi4py.MPI.Exception: MPI_ERR_OTHER: known error not in list
srun: error: nvl72d048-T09: tasks 1-3: Exited with exit code 1
srun: Terminating StepId=2808217.0
STEP 2808217.0 ... CANCELLED ... DUE TO TASK FAILURE
Likely cause: Multi-node SBSA/GB200 MPI transport failed to establish UCX endpoints between ranks. This is characteristic of cluster/network/MPI infrastructure instability; it is not tied to the PR’s MoE code.
Related source files: None in PR. The stage used MPI/Slurm infrastructure and failed before producing a meaningful model/test assertion.
Is waived?: Not applicable; this is infrastructure.
Recommended Actions
Re-run the CI pipeline. The SBSA multi-node failure is infrastructure, and the x86 B200 failure is in an unrelated KV-cache test.
If test_chunked_prefill_with_eviction fails again on B200, file/route to KV-cache scheduler owners with the stage/job info:
L0_Test-x86_64-Single-GPU #201
DGX_B200-PyTorch-3
Slurm job 1567638
No PR code action is indicated for #15550 based on this run. The PR modifies INT8 W8A16 non-gated MoE support and MoE unit tests; the observed failures do not exercise those changed areas.

[leslie-fang25]

/bot run --disable-fail-fast

[tensorrt-cicd]

PR_Github #63323 [ run ] triggered by Bot. Commit: ba0c487 Link to invocation

[tensorrt-cicd]

PR_Github #63323 [ run ] completed with state FAILURE. Commit: ba0c487
/LLM/main/L0_MergeRequest_PR pipeline #51318 completed with status: 'FAILURE'

CI Report

⚠️ Action Required:

• Please check the failed tests and fix your PR
• If you cannot view the failures, ask the CI triggerer to share details
• Once fixed, request an NVIDIA team member to trigger CI again

CI Agent Failure Analysis

Link to invocation

[Dorijan10]

Thanks for the report @leslie-fang25. Agreed the failures look unrelated.

On the B200 side, two of the fifteen TestKVCacheV2Llama tests scheduled in l0_b200.yml are already waived in main, test_chunked_prefill under nvbugs/6428002 and test_eviction_with_block_reuse under nvbugs/6462303. The one failing here, test_chunked_prefill_with_eviction, exercises both areas: it enables chunked prefill and uses the same _run_eviction_test helper as the second one. It also hit the 3600s timeout rather than failing an assertion, so it looks like a hang in an area with known issues rather than anything new.

The GB200 failure is a UCX endpoint resolution error on a different rack from the previous run, so that looks like general multi-node instability.

Would it make sense to maybe waive that KV-cache test alongside the others? I could also maybe merge latest main if that helps, since a fair number of waivers have landed since the last merge, according to my checks.

Please let me know!

[leslie-fang25]

/bot run --disable-fail-fast

[tensorrt-cicd]

PR_Github #63419 [ run ] triggered by Bot. Commit: ba0c487 Link to invocation

[tensorrt-cicd]

PR_Github #63419 [ run ] completed with state SUCCESS. Commit: ba0c487
/LLM/main/L0_MergeRequest_PR pipeline #51395 completed with status: 'SUCCESS'

CI Report

Link to invocation

[Dorijan10]

Excellent @leslie-fang25, all of the checks have passed!

Seems like this PR is ready for merge, let me know if you guys are ready to proceed!

[Dorijan10]

@leslie-fang25 @xxi-nv @rosong11 and other reviewers.

Thanks again for running the CI tests on this PR throughout. All checks are now green and this PR should be ready for clean merge.

It would be great if this is completed so that we can continue the work on the W8A16 path in the PR #16198 I raised as well. It is stacked on this one so it is important #15550 gets merged first. Many thanks!