[AutoDeploy][Bug]: Cutlass MOE kernel caused the accuracy drop

[nvchenghaoz]

System Info

With the H100 from cw, the cutlass moe BF16 kernel caused the accuracy drop for gsm8k.

Who can help?

@nzmora-nvidia

Information

• The official example scripts
• My own modified scripts

Tasks

• An officially supported task in the examples folder (such as GLUE/SQuAD, ...)
• My own task or dataset (give details below)

Reproduction

pytest tests/integration/defs/accuracy/test_llm_api_autodeploy.py::TestNemotronMOE -s -vv

Expected behavior

Fix the accuracy issue, or If the kernel cannot support the BF16, then let's stick with the triton..

actual behavior

N/A

additional notes

N/A

Before submitting a new issue...

• Make sure you already searched for relevant issues, and checked the documentation and examples for answers to frequently asked questions.

[nzmora-nvidia]

Hi @nvchenghaoz

I'm getting

FAILED tests/integration/defs/accuracy/test_llm_api_autodeploy.py::TestNemotronMOE::test_auto_dtype - AssertionError: Checkpoint path /home/scratch.trt_llm_data/llm-models/Nemotron-MOE/ is not a directory.

I couldn't find a similar model in /home/scratch.trt_llm_data/llm-models/ or /lustre/fs1/portfolios/coreai/projects/coreai_dlalgo_modelopt/autodeploy_data/hf_home

[nvchenghaoz]

@nzmora-nvidia you can update the model path to your local nemotron MOE BF16 model path. FYI this is the path I've been using - f"/lustre/fs1/portfolios/coreai/projects/coreai_dlalgo_modelopt/users/chengzhang/cache/huggingface/hub/models--nvidia--NVIDIA-Nemotron-Nano-31B-A3-v3/snapshots/071abcf3cfe02d4aa023a711799139e89a67094d/"

[suyoggupta]

Bumping to P0

[suyoggupta]

@nzmora-nvidia : any updates here?

[nzmora-nvidia]

As I reported in the meeting today, on main I'm getting good results.
@nvchenghaoz Let me know if you think I can proceed with this #9194 (comment)

= ACCURACY HYPOTHESIS TESTING
===========================================================
Alpha (Type I:  False Positive): 0.050
Beta  (Type II: False Negative): 0.200
Sigma (Standard deviation): 50.000
#Samples: 4096
Higher is better: True
Theta (Minimum detectable effect): 2.747
Reference accuracy: 0.000
Threshold: -1.817
===========================================================
Evaluated accuracy: 78.021
===========================================================
Generating train split: 100%|█████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████| 7473/7473 [00:00<00:00, 369801.84 examples/s]
Generating test split: 100%|██████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████| 1319/1319 [00:00<00:00, 199945.32 examples/s]
[2025-11-18 07:50:40] INFO task.py:420: Building contexts for gsm8k on rank 0...
100%|███████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████| 1319/1319 [00:03<00:00, 339.24it/s]
[2025-11-18 07:50:44] INFO evaluator.py:517: Running generate_until requests
Submitting requests: 100%|██████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████| 1319/1319 [00:02<00:00, 458.13it/s]
Fetching responses: 100%|████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████| 1319/1319 [02:15<00:00,  9.74it/s]
[11/18/2025-07:53:02] [TRT-LLM] [I] TRTLLM execution time: 138.264 seconds.
[11/18/2025-07:53:04] [TRT-LLM] [I] lm-eval gsm8k results (scores normalized to range 0~100):
|Tasks|Version|     Filter     |n-shot|  Metric   |   | Value |   |Stderr|
|-----|------:|----------------|-----:|-----------|---|------:|---|-----:|
|gsm8k|      3|flexible-extract|     5|exact_match|↑  |82.1835|±  |1.0540|
|     |       |strict-match    |     5|exact_match|↑  |81.8044|±  |1.0627|

[11/18/2025-07:53:04] [TRT-LLM] [I] lm-eval gsm8k average accuracy: 81.99
[11/18/2025-07:53:04] [TRT-LLM] [I] Hypothesis testing report:
===========================================================
= ACCURACY HYPOTHESIS TESTING
===========================================================
Alpha (Type I:  False Positive): 0.050
Beta  (Type II: False Negative): 0.200
Sigma (Standard deviation): 50.000
#Samples: 1319
Higher is better: True
Theta (Minimum detectable effect): 4.841
Reference accuracy: 0.000
Threshold: -3.203
===========================================================
Evaluated accuracy: 81.994
===========================================================