[100milliongold]

Description

cpp/tensorrt_llm/deep_ep/CMakeLists.txt classifies each entry of CMAKE_CUDA_ARCHITECTURES and re-emits it for DeepEP. The comment directly above the branch states the requirement:

# The FP4-related conversion instructions in DeepEP require SM100a, SM110a,
# or SM120a.
if(${CUDA_ARCH_MAJOR} EQUAL 10 AND ${CUDA_ARCH_MINOR} EQUAL 0)

The first branch requires CUDA_ARCH_MINOR EQUAL 0, so it only matches 100. The elseif also requires CUDA_ARCH_MINOR EQUAL 0, so it only matches 110, 120, and later x00 architectures. Every other member of the 10x family therefore falls through to the final else branch and is emitted with no suffix at all.

The regex does not capture the suffix either — [af]? sits outside the capture groups — so an incoming 103a-real or 107f-real loses its suffix on the way through.

Observed with an unmodified checkout of main (a76e0b2):

CMAKE_CUDA_ARCHITECTURES entry	emitted for DeepEP
90a-real	90-real
100f-real	100f-real	matches the comment
103a-real	103-real	suffix dropped
107f-real	107-real	suffix dropped
110f-real	110a-real	matches the comment
120f-real	120a-real	matches the comment

103 and 107 are the two 10x architectures currently reachable through cuda_configuration.cmake, and both end up with a bare target while their siblings do not.

This PR drops the CUDA_ARCH_MINOR EQUAL 0 condition from the first branch so that the whole 10x major is routed to the existing 100 handling. That matches the stated intent in #17336 that 100f kernels are meant to cover the whole 10x family:

we want 100f kernels to cover the whole 10x architectures to reduce build time and binary size

Only 103 and 107 change. 90, 100, 110 and 120 are unaffected.

Note on older CMake: when CMAKE_VERSION < 3.31 the existing fallback emits 100a and 103a because family targets cannot be expressed there. That limitation is unchanged by this PR; 107 simply follows the same path 100 already takes instead of getting a bare target.

Test Coverage

The branch logic was extracted verbatim into a standalone script and executed with cmake -P against both the current and the proposed condition, over the architecture strings cuda_configuration.cmake can produce. The table above is that output; with the change, 103a-real and 107f-real produce the same result as 100f-real, and every other input is byte-identical to current behavior.

A full build_wheel.py configure with -a "90-real;100-real;103-real;107-real" was also run on the unmodified tree to capture the current deep_ep DEEP_EP_CUDA_ARCHITECTURES: line, which is what surfaced the bare 107-real entry.

I do not have Rubin hardware, so this PR makes no claim about runtime performance on SM107. It only restores the suffix that the surrounding comment requires.

PR Checklist

• The PR title follows the required format.
• The change is limited to a single file and a single condition.
• The commit is signed off (DCO).
• Requires a maintainer to confirm the intended handling for 103, which currently also receives a bare target.

Related: #19069 (closed — that approach was wrong, and the reason is written up there), #17336 (original SM107 recognition).

Dev Engineer Review

cpp/tensorrt_llm/deep_ep/CMakeLists.txt now routes all CUDA major-version 10 architectures through SM100-family handling. This fixes suffix handling for architectures such as 103a-real and 107f-real. Architectures 90, 110, and 120 retain their existing paths. The change affects DeepEP and NVSHMEM CUDA architecture configuration only. No API, runtime, or performance changes are introduced.

QA Engineer Review

No test changes.

Per-File QA Perspective

• cpp/tensorrt_llm/deep_ep/CMakeLists.txt: Verify generated DEEP_EP_CUDA_ARCHITECTURES values for 100, 103, and 107 inputs with -real and -virtual suffixes. Confirm existing 90, 110, and 120 outputs remain unchanged.

[coderabbitai]

No actionable comments were generated in the recent review. 🎉

ℹ️ Recent review info

⚙️ Run configuration

Configuration used: Path: .coderabbit.yaml

Review profile: CHILL

Plan: Enterprise

Run ID: f08a0719-936c-42ea-a574-703a17fcb9c9

📥 Commits

Reviewing files that changed from the base of the PR and between a76e0b2 and 496e370.

📒 Files selected for processing (1)

• cpp/tensorrt_llm/deep_ep/CMakeLists.txt

Included review availability: Your plan provides up to 12 included reviews per hour; 11 remain after this review.

---

Walkthrough

The DeepEP CMake configuration now applies the special SM100/SM103 mapping to all CUDA major version 10 architectures instead of only CUDA 10.0.

Changes

CUDA architecture handling

Layer / File(s)	Summary
Update CUDA architecture matching cpp/tensorrt_llm/deep_ep/CMakeLists.txt	The special architecture branch now matches every CUDA architecture with major version 10.

Priority: ⬇️ Low

Estimated code review effort: 1 (Trivial) | ~5 minutes

Change: Bug fix

Suggested reviewers: bowenfu

Merge Risk: ⚪ Minimal · up to 496e3

This is a narrowly scoped build-configuration correction with no remaining merge-blocking risk.

🚥 Pre-merge checks | ✅ 5

✅ Passed checks (5 passed)

Check name	Status	Explanation
Title check	✅ Passed	The title follows the required format and clearly describes the main change: preserving family suffix handling for 10x CUDA architectures in DeepEP.
Description check	✅ Passed	The description explains the issue, proposed fix, affected architectures, test coverage, limitations, and checklist status. It provides sufficient context for review.
Docstring Coverage	✅ Passed	No functions found in the changed files to evaluate docstring coverage. Skipping docstring coverage check. Docstring coverage is scoped to functions touched by this diff. Analyzed 0 functions across 0…
Linked Issues check	✅ Passed	Check skipped because no linked issues were found for this pull request.
Out of Scope Changes check	✅ Passed	Check skipped because no linked issues were found for this pull request.

✨ Finishing Touches

🧪 Generate unit tests (beta)

• Create PR with unit tests

---

_{Comment @coderabbitai help to get the list of available commands.}

[100milliongold]

Follow-up: this is not only a consistency issue, it is a hard build failure. I hit it while building main with -a "90-real;100-real;103-real;107-real".

The build break

ptxas .../extension_kernels.compute_107.ptx, line 12637; error : Feature 'cvt.e2m1x2.f32' not supported on .target 'sm_107'
ptxas .../extension_kernels.compute_107.ptx, line 13592; error : Feature 'cvt.f16x2.e2m1x2' not supported on .target 'sm_107'
...
ptxas fatal : Ptx assembly aborted due to errors
gmake[3]: *** [.../deep_ep_cpp_tllm.dir/.../csrc/kernels/extension_kernels.cu.o] Error 255

cvt.e2m1x2.* are exactly the FP4-related conversion instructions the comment above the branch refers to. The generated command line confirms 107 arrives without a suffix, while its siblings get one:

--generate-code=arch=compute_90,code=[sm_90]
--generate-code=arch=compute_100a,code=[sm_100a]
--generate-code=arch=compute_103a,code=[sm_103a]
--generate-code=arch=compute_107,code=[sm_107]     <- bare

Which targets actually work

I replayed the exact failing compile (same CUDA_DEFINES / CUDA_INCLUDES / CUDA_FLAGS from flags.make, same -x cu -rdc=true -c, same working directory), varying only the generated code target:

target	result
sm_100a	compiles, 138864 B
sm_100f	compiles, 138864 B
sm_103a	compiles, 138864 B
sm_107	fails, 84 unsupported-instruction errors
sm_107a	compiles, 137296 B

The bare sm_107 target is the only one that fails, and sm_100f — what this PR routes 107 to — compiles cleanly.

Corroboration from upstream

a85c26c ("Map SM107 to the sm_100f NVRTC target in the trtllm-gen FMHA kernel"), merged today, applies the same 107 to sm_100f mapping in a different file. This PR does the equivalent thing for DeepEP's CMake-level architecture selection.

Note that on CMAKE_VERSION < 3.31 the existing fallback emits 100a/103a rather than 100f; both of those also compile, per the table above, so 107 is unblocked on old and new CMake alike.

[100milliongold]

@Barry-Delaney @leslie-fang25 Gentle ping. This has had no review since it was opened on Sep 12.

It is a build break rather than a consistency nit. Building main with -a "90-real;100-real;103-real;107-real" fails in ptxas because the family suffix is dropped for the 10x architectures in DeepEP; the error is in the comment above. One line in one file.

One thing worth flagging while you are here: the only checks that have ever run on this PR are DCO, assign-author and label-component. Check PR Title Format and Pre-commit Check have no result at all — not a pass, not a failure. On three of my other PRs the same checks were absent until the title was edited, which re-triggered them. I did not edit this one because its title already matches the required format, so I have left it alone rather than push a no-op. If that absence is what is keeping this out of a review queue, please say so and I will trigger them.

If the approach is wrong I am glad to rework it. I would mainly like to know whether the family suffix belongs here, or whether DeepEP is meant to be built for a single architecture.

[BowenFu]

Approved — Rubin release fix is correctly scoped.

The change preserves the family suffix for every CUDA major-10 target, so SM107 follows the established 100f compatibility path instead of producing invalid bare sm_107 PTX. The supplied CMake matrix and full-build compile evidence cover the affected 100/103/107 variants. I found no additional blocker.

[BowenFu]

/bot run --disable-fail-fast

[tensorrt-cicd]

PR_Github #74891 [ run ] triggered by Bot. Commit: 496e370 Link to invocation

[BowenFu]

Rubin 1.3 approval needed.

@Barry-Delaney @leslie-fang25 @NVIDIA/trt-llm-moe-devs: this is a one-line root-cause fix for the SM107 DeepEP build failure. The author supplied exact failing sm_107 evidence and successful sm_100f replay; I reviewed and approved the live head. Please provide the remaining CODEOWNER approval so full CI can start. No author change is currently requested.

[github-actions]

Automatically added "ci: full pre-merge approved" because this PR has satisfied the required GitHub review approvals. Unresolved review conversations and other required checks remain independent merge requirements.

[tensorrt-cicd]

PR_Github #74891 [ run ] completed with state SUCCESS. Commit: 496e370
/LLM/main/L0_MergeRequest_PR pipeline #61657 completed with status: 'FAILURE'

CI Report

⚠️ Action Required:

• Please check the failed tests and fix your PR
• If you cannot view the failures, ask the CI triggerer to share details
• Once fixed, request an NVIDIA team member to trigger CI again

CI Agent Failure Analysis

Link to invocation

[100milliongold]

@BowenFu Thanks for the review and for triggering the run.

The pipeline came back FAILURE — 11364 passed, 17 failed, 424 skipped on 496e370. The blossom-ci status carries only that count; the CI report, the failure analysis page and the Jenkins job are all on internal hosts I cannot reach, so I do not know which 17 tests failed. Following the bot's own instruction, I'm asking you to share them.

While waiting I checked what is visible from outside, and I'd rather show my reasoning than just assert it.

The same count appears on unrelated PRs. In the most recent ~100 PRs, 17 runs reported a numeric result. Four of them, all based on main a day after my run, report exactly 17 failures:

PR	result	what it touches
#19600	6825 passed, 17 failed, 743 skipped	jenkins/ scripts, Ray Mamba test
#19591	13798 passed, 17 failed, 748 skipped	trtllm-gen FMHA cubin artifacts
#19583	13711 passed, 17 failed, 739 skipped	tensorrt_llm/bench, executor, llmapi
#19593	12204 passed, 17 failed, 309 skipped	MoE DeepGEMM warmup fence
#19095	11364 passed, 17 failed, 424 skipped	one line of DeepEP CMake

The other counts in that sample are 2 (×4), 13 (×3), and 5/10/14/16/19/20 once each. None of those four PRs shares a file with this one.

How much of CI this change can reach. It is worth being precise, because the change is not inert in CI. Every build config in jenkins/Build.groovy passes 103-real, BUILD_DEEP_EP defaults to ON, and the CI image ships CMake 4.0.3, so the VERSION_GREATER_EQUAL 3.31 branch applies. Before this PR, 103-real fell through to the bare 103 target; after it, 103-real resolves to 100f-real. That is a real change to what CI compiles, on top of unblocking 107.

What can observe that change is the DeepEP cubin running on SM103 hardware. Across every l0_b300* / l0_gb300* list in tests/integration/test_lists/test-db/, there are 5 such tests, all in l0_dgx_b300.yml:

unittest/_torch/moe/test_moe_module.py::test_configurable_moe_multi_gpu[parallel=DEP-comm=DEEPEP-...]

Five tests cannot produce seventeen failures.

What this does not settle. Matching counts are not matching tests — I am comparing numbers, not names. Some of the 17 could still be among those five, and the count argument only bounds the blast radius; it does not empty it.

What would settle it, and it is cheap on your side: if you can paste the failing test list, or just say whether any of the five comm=DEEPEP B300 cases are in it, that answers the question outright. If none are, the failures are independent of this change and a re-run against current main should be the whole remedy.

One small thing I noticed while tracing this. With 100-real;103-real both in CMAKE_CUDA_ARCHITECTURES, both now append 100f-real, so DEEP_EP_CUDA_ARCHITECTURES ends up holding it twice (on CMake < 3.31 the older branch duplicates 100a/103a the same way). It is harmless — the build clearly succeeded — but a list(REMOVE_DUPLICATES ...) would tidy it. I have left it out deliberately: a push here would dismiss the two approvals this PR already has, and that seems a poor trade for a cosmetic fix. Happy to fold it in if you would rather have it in one go.

[100milliongold]

Short follow-up to my last comment — the situation has moved, and it makes the ask smaller rather than larger.

The 17-failure cluster is gone from main. Runs that finished today:

PR	result
#19595	Bot Pipeline Run SUCCESS, L0 Pipeline SUCCESS
#19606	5288 passed, 2 failed, 112 skipped
#19609	3840 passed, 2 failed, 112 skipped
#19601	431 passed, 2 failed, 8 skipped
#19580	12773 passed, 13 failed, 741 skipped

No run today reports 17, and #19595 went fully green. In between, #19572 and #19579 (Waive N failed cases for main in pre-merge …) merged, along with a number of fixes.

My run was pipeline #61657 on Sep 22 at 01:38 UTC, which predates all of that.

So please disregard the request for the failure list unless it is convenient — a plain /bot run is probably the whole remedy. The head is still 496e370, so this needs no push from me and the existing approvals stay intact.

If the re-run comes back with failures again, then the list would genuinely help, and the five comm=DEEPEP B300 cases from my previous comment are still the ones worth checking first.

[github-actions]

Semantic conflict review

The verdict of record is the Semantic conflict with target branch / PR #19095 commit status on the requested head commit. This summary updates on reply events and may lag between a new request and its reply.

Latest recorded state: No semantic conflict found (best effort) for head 496e370c631a60c9e19c835996df2cde2a14bdac, target ca37c9f725922f7e72017ff5f5b9794731915a34, merge base a76e0b2b4051089958e16f0dfac0ea244d84b716 (request ad54cbd1-5698-456c-96ca-087854798b20). CodeRabbit analysis.

Best-effort AI judgment for the recorded revisions. PASS, FAIL and INCONCLUSIVE may be incomplete or incorrect. PR authors and reviewers should independently verify the evidence and relevant behavior. This semantic review and its status/workflow are advisory, not required merge checks under current repository rules; other merge requirements still apply. Advisory status does not make a confirmed defect safe to ignore.

Requested (UTC)	Head	Target	Verdict	Comment
2026-10-02T16:40:31Z	496e370c631a	ca37c9f72592	PASS	reply
2026-10-02T04:42:38Z	496e370c631a	8a3c90311ea9	PASS	reply
2026-10-01T22:35:47Z	496e370c631a	269365738865	PASS	reply
2026-10-01T14:42:43Z	496e370c631a	ee510fc85d39	PASS	reply
2026-10-01T08:47:41Z	496e370c631a	d6ebc4dc07aa	PASS	reply
2026-10-01T01:24:35Z	496e370c631a	534e1f8ad9d5	PASS	reply
2026-09-30T18:46:02Z	496e370c631a	14863787ca19	PASS	reply
2026-09-30T10:40:55Z	496e370c631a	abddfc991456	PASS	reply
2026-09-30T02:50:28Z	496e370c631a	111687f92f4d	PASS	reply
2026-09-29T18:46:33Z	496e370c631a	bcb288a21105	PASS	reply
2026-09-29T12:52:46Z	496e370c631a	bcdc2d5aaf0c	PASS	reply
2026-09-29T04:43:32Z	496e370c631a	ae4aa5d6c932	PASS	reply
2026-09-28T22:35:26Z	496e370c631a	27ba69c78189	PASS	reply
2026-09-28T16:41:36Z	496e370c631a	ca937c95012b	PASS	reply
2026-09-28T08:53:24Z	496e370c631a	c2220eef33f4	PASS	reply

Processed request and reply comments are minimized to reduce timeline noise; they remain expandable for audit.

[Barry-Delaney]

Reviewed 496e370. The entire major-10 family now follows the existing 100f path, while the other architecture branches remain unchanged. I executed the actual PR normalization loop for CMake 3.30/3.31 with unsuffixed, -real, and -virtual targets (6 matrix cases), including 80/90/100/103/107/110/120 inputs; all passed. git diff --check passed. No GPU build/runtime validation was performed locally; the existing older-CMake fallback remains unchanged.