Repository navigation
[TRTLLM-14558][chore] Split cute_dsl_kernels/ into the domain kernel trees and consolidate the thop unit tests - #19275
Merged
Conversation
YihuiLu512
force-pushed
the
layout/T14T15T20T22-kernels
branch
4 times, most recently
from
September 17, 2026 05:55
ce8e7e8 to
eaeede1
Compare
YihuiLu512
marked this pull request as ready for review
September 17, 2026 05:59
YihuiLu512
requested review from
Barry-Delaney,
karljang,
kris1025,
o-stoner,
pengbowang-nv,
xxi-nv,
yiqingy0 and
yuanjingx87
September 17, 2026 05:59
Collaborator
|
PR_Github #78790 [ run ] triggered by Bot. Commit: |
yizhang-nv
approved these changes
Oct 19, 2026
Collaborator
|
PR_Github #78790 [ run ] completed with state
|
Collaborator
Author
|
/bot run --disable-fail-fast |
Collaborator
|
PR_Github #78899 [ run ] triggered by Bot. Commit: |
…o tests/microbenchmarks
Pure relocation. Path mapping (old -> new):
Benchmark drivers -- the whole subtree moves, subdirectories preserved
tests/scripts/cute_dsl_kernels/** -> tests/microbenchmarks/cute_dsl_kernels/**
(16 files: 13 run_*.py, testing.py, moe_workload_generator.py, README.md;
the moe_as_dense_gemm/, paged_mqa_logits/ and top_k/ levels are unchanged)
Ownership: .github/CODEOWNERS
- the /tests/scripts/cute_dsl_kernels rule is removed
- a /tests/microbenchmarks/cute_dsl_kernels rule is added immediately below
/tests/microbenchmarks, because that generic rule would otherwise win under
last-match-wins and put 10 of the 16 files on @NVIDIA/trt-llm-perf-devs
(the other 6 are held by the four team rules further down)
- the four team-specific rules (paged_mqa_logits, moe_as_dense_gemm,
moe_workload_generator.py, top_k) are repointed in place; they already sit
below the generic rule, so no line has to move
Net effect on CODEOWNERS is +3 lines. The effective-owner diff for this commit
is empty: all 16 files keep the team that owns them today.
Reviewer notes:
This commit is intentionally first and intentionally content-free apart from
two docstring lines. Every driver's Path(__file__).parents[N] is unchanged --
tests/scripts/... and tests/microbenchmarks/... sit at the same depth -- and
every kernel path the drivers name still exists at this commit, so nothing
has to be rewritten here. The import and sys.path repointing happens in the
two commits that actually move the kernels (C2 and C3).
The two edited lines are paged_mqa_logits/run_fp4.py:7 and run_fp8.py:7, each
a self-referential docstring naming tests/scripts/cute_dsl_kernels -- the very
directory being renamed. They are taken here rather than deferred because no
later commit has a code path that could apply them, and deferring leaves the
residue scan red at every intermediate tip.
No __init__.py and no conftest.py is added under the new tree. Adding either
would make the drivers' "try: from .testing import ..." branch start winning
over the bare-name fallback, which is a behaviour change.
This move takes these 16 files out of CI stage selection and out of the
packaged source tree. Both are declared in the PR description; see "Declared
behaviour changes" there.
Signed-off-by: Yihui Lu <269394165+YihuiLu512@users.noreply.github.com>
…ral CuteDSL and cuTile kernels
Relocation plus the reference rewrites that keep the tree importable at this
commit. Path mapping (old -> new):
Source -- new package tensorrt_llm/_torch/kernels/
_torch/cute_dsl_kernels/argmax.py -> _torch/kernels/argmax.py
_torch/cute_dsl_kernels/blackwell/*.py -> _torch/kernels/blackwell/*.py
(custom_pipeline, cute_ptx_utils, dense_blockscaled_gemm_act_fusion,
dense_blockscaled_gemm_persistent, dense_gemm_persistent, utils)
_torch/cute_dsl_kernels/blackwell/blockwise_gemm/**
-> _torch/kernels/blackwell/blockwise_gemm/**
_torch/cute_dsl_kernels/rubin/*.py -> _torch/kernels/rubin/*.py
_torch/cute_dsl_kernels/rubin/blockwise_gemm/**
-> _torch/kernels/rubin/blockwise_gemm/**
_torch/cute_dsl_kernels/utils/** -> _torch/kernels/utils/**
_torch/cuda_tile_kernels/rms_norm.py -> _torch/kernels/rms_norm.py
_torch/cuda_tile_kernels/rms_norm_fuse_residual.py
-> _torch/kernels/rms_norm_fuse_residual.py
Tests -- the in-source test leaves tensorrt_llm/
_torch/cute_dsl_kernels/test_argmax.py -> tests/unittest/_torch/kernels/parallel/test_argmax.py
Added: tensorrt_llm/_torch/kernels/{__init__.py,blackwell/__init__.py,rubin/__init__.py}
tests/microbenchmarks/cute_dsl_kernels/_offline_loader.py
Emptied and removed: tensorrt_llm/_torch/cuda_tile_kernels/
20 files move, 3 package markers are authored, 3 byte-empty markers are
filled, and 1 file is deleted.
tensorrt_llm/_torch/cuda_tile_utils.py deliberately does NOT move: the two
cuTile kernels land flat in _torch/kernels/, so their "from ..cuda_tile_utils
import IS_CUDA_TILE_AVAILABLE" stays byte-identical.
.github/CODEOWNERS drops the /tensorrt_llm/_torch/cuda_tile_kernels rule with
the directory and adds /tensorrt_llm/_torch/kernels and
/tests/unittest/_torch/kernels, all three on @NVIDIA/trt-llm-kernels-devs, so
ownership of every moved file is unchanged across the move.
Reviewer notes:
Read this commit for the rewrites, not for the renames.
Two of the authored __init__.py -- blackwell/ and rubin/ -- are the
destinations of package markers whose SOURCE is deleted only in the next
commit. This is deliberate, and a bisecting reader should not read it as a
mistake: 62 T14 files still live under _torch/cute_dsl_kernels/blackwell/ and
rubin/ at this commit. Removing those markers here would turn the directories
into PEP-420 namespace packages, which setup.py:586's bare find_packages()
does not discover -- a wheel built at this commit would silently omit them.
The blobs are byte-empty, so nothing is lost by authoring the destination and
deleting the source separately. The third, kernels/__init__.py, has no source
at all: cute_dsl_kernels/__init__.py is retired rather than relocated, and it
too is deleted in the next commit.
For the same reason, .github/CODEOWNERS keeps its
/tensorrt_llm/_torch/cute_dsl_kernels rule here and drops it only in the next
commit. Deleting it now would drop those 62 files to the
/tensorrt_llm/_torch @NVIDIA/trt-llm-runtime-devs catch-all for one commit --
a silent 62-file owner transfer that no CI check reports.
This commit rewrites every reference to a T15-owned module wherever the
referring file lives, including inside files a LATER commit will move. That is
what makes the per-task split possible. Of the 14 "from ..utils" / ".utils" /
".custom_pipeline" / ".cute_ptx_utils" lines inside T14's still-in-place
files, 10 become absolute tensorrt_llm._torch.kernels.blackwell.* and are
therefore position-independent when C3 moves those files. The remaining 4 --
all under blackwell/top_k/ -- stay relative and deepen from 2 to 4 dots
instead: they sit in the T22 offline-fallback closure, where an absolute
import would re-enter the package the fallback has just proven unimportable.
Their C3 destination has the same package depth, so the 4-dot form resolves
to tensorrt_llm._torch.kernels.blackwell.utils at both tips.
custom_ops/cuda_tile_custom_ops.py:26 is split by hand from one package-level
import into two submodule imports (..kernels.rms_norm and
..kernels.rms_norm_fuse_residual), because the aggregating package __init__.py
is retired rather than moved. Without this edit THIS COMMIT DOES NOT IMPORT.
The four run_dense_* benchmark drivers get their offline import fallback
repointed here, because this is the commit that moves the kernels they load.
The bare "sys.path.insert(<repo>/tensorrt_llm/_torch/cute_dsl_kernels)" root
they used has no successor: any directory below <repo>/tensorrt_llm raises
"attempted relative import beyond top-level package", and <repo>/tensorrt_llm
itself executes _torch/__init__.py and kills every tensorrt_llm.* import. The
four now share tests/microbenchmarks/cute_dsl_kernels/_offline_loader.py,
which seeds exactly two package objects -- tensorrt_llm and
tensorrt_llm._torch -- into sys.modules with the checkout prepended to their
__path__, after which ordinary import machinery resolves the rest. Seeding
rather than replacing keeps an install's own search locations behind the
checkout, so tensorrt_llm.bindings stays reachable. Every "parents[N]" is
unchanged: N is anchored on the repo root and set by the driver's own depth,
not by where the kernels land. The helper is deliberately a plain sibling
module -- adding __init__.py or conftest.py under this tree would flip the
drivers' "try: from .testing" arm and change behaviour.
The 3 filled byte-empty markers appear as create+delete rather than rename:
git cannot compute rename similarity against an empty blob. Git additionally
pairs the deleted cuda_tile_kernels/__init__.py with the newly authored
kernels/__init__.py at 51% (both are mostly the same Apache header). So
`git show -M --summary` prints 18 rename / 6 create / 3 delete here, not
20/3/1. Treat that tuple as a REFERENCE, never as an acceptance criterion:
51% clears git's default threshold by one point, so ~36 bytes added upstream
to cuda_tile_kernels/__init__.py flips it to 17/7/4 with nothing in this
commit having changed. What is actually asserted is per-file -- every moved
path is either a detected rename to its mapped destination, or had a
byte-empty source. There is no blame to lose on an empty file.
test_argmax.py is the one genuinely new test collection in this PR. It gains
an explicit getSMVersion() < 90 skip rather than being left to fail in the
JIT. No l0 entry is added for it here -- C4's
unittest/_torch/kernels/parallel directory entry subsumes it, and creating
both would collect it twice.
docs/source/torch/adding_custom_kernels.md tracks the move: the cuTile table
rows and worked-example links are repointed, the excerpt's import is switched
to the submodule form the real file now uses, and the sm_100+ caveat is split
into two sentences so the cute_dsl_kernels/blackwell/ half can be updated by
C3 without touching the cuTile half settled here.
The three generated lint mirrors (pyproject.toml, ruff-legacy.toml,
.pre-commit-config.yaml) are regenerated with
`python scripts/legacy_utils.py gen-configs`, never hand-edited. They are
regenerated again in C3 and C4; that is expected, they are generated files.
Signed-off-by: Yihui Lu <269394165+YihuiLu512@users.noreply.github.com>
… and VisualGen kernel trees
Relocation plus the reference rewrites that keep the tree importable at this
commit. This commit retires tensorrt_llm/_torch/cute_dsl_kernels/.
Path mapping (old -> new):
Attention -- 22 files
_torch/cute_dsl_kernels/blackwell/attention/**
-> _torch/attention/kernels/blackwell/**
(the redundant attention/ level collapses; mla/ is preserved below it)
_torch/cute_dsl_kernels/blackwell/paged_mqa_logits/
-> _torch/attention/kernels/blackwell/paged_mqa_logits/
_torch/cute_dsl_kernels/blackwell/kimi_k3_kda/
-> _torch/attention/kernels/blackwell/kimi_k3_kda/
_torch/cute_dsl_kernels/blackwell/dspark/
-> _torch/attention/kernels/blackwell/dspark/
_torch/cute_dsl_kernels/blackwell/dspark_rmsnorm_rope.py
-> _torch/attention/kernels/blackwell/dspark_rmsnorm_rope.py
_torch/cute_dsl_kernels/blackwell/minimax_m3_index_decode_score.py
-> _torch/attention/kernels/blackwell/minimax_m3_index_decode_score.py
_torch/cute_dsl_kernels/rubin/dsv4_qb_fusion/
-> _torch/attention/kernels/rubin/dsv4_qb_fusion/
MoE -- 70 files
_torch/cute_dsl_kernels/blackwell/moe_as_dense_gemm/
-> _torch/moe/kernels/blackwell/moe_as_dense_gemm/
_torch/cute_dsl_kernels/blackwell/blockscaled_contiguous_*.py (4 files)
-> _torch/moe/kernels/blackwell/
_torch/cute_dsl_kernels/cutedsl_megamoe/
-> _torch/moe/kernels/cutedsl_megamoe/
(upstream NVIDIA#17956 replaced mega_moe_nvfp4/ with this tree; the 20 files
it removed are not relocated here, and the package is self-contained,
so no import inside it changes)
_torch/cute_dsl_kernels/rubin/moe/
-> _torch/moe/kernels/rubin/
(the inner moe/ level is dropped; depth is unchanged, so
rubin_contiguous_*.py's "from ....utils" still reaches _torch/utils.py;
includes the two files NVIDIA#18357 added there, manual_mma_128dp.py and
rubin_contiguous_grouped_blockscaled_gemm_fused_fc12.py)
VisualGen -- 8 files
_torch/cute_dsl_kernels/blackwell/conv/
-> _torch/visual_gen/kernels/blackwell/conv/
_torch/cute_dsl_kernels/blackwell/pertoken_adaln/
-> _torch/visual_gen/kernels/blackwell/pertoken_adaln/
Neutral, T14-owned -- 17 files
_torch/cute_dsl_kernels/blackwell/top_k/
-> _torch/kernels/blackwell/top_k/
Fused ops -- 4 source + 3 test files
_torch/modules/fused_ops/ -> _torch/kernels/fused_ops/
tests/unittest/_torch/modules/fused_ops/
-> tests/unittest/_torch/kernels/fused_ops/
Tests that follow their kernel -- 4 files
tests/unittest/_torch/thop/parallel/test_indexer_topk.py
-> tests/unittest/_torch/kernels/parallel/test_indexer_topk.py
tests/unittest/_torch/thop/parallel/test_gvr_selfsampling_topk.py
-> tests/unittest/_torch/kernels/parallel/test_gvr_selfsampling_topk.py
tests/unittest/_torch/cute_dsl_kernels/test_nvfp4_conv3d.py
-> tests/unittest/_torch/visual_gen/kernels/test_nvfp4_conv3d.py
tests/unittest/_torch/cute_dsl_kernels/test_custom_pipeline.py
-> tests/unittest/_torch/kernels/test_custom_pipeline.py
(it loads blackwell/custom_pipeline.py by file path; that kernel is in
the neutral tree since C2, so the test follows it there rather than
into visual_gen/ with its former neighbour)
Added: tensorrt_llm/_torch/attention/kernels/rubin/__init__.py,
tensorrt_llm/_torch/moe/kernels/{__init__.py,blackwell/__init__.py},
tensorrt_llm/_torch/visual_gen/kernels/{__init__.py,blackwell/__init__.py}
Emptied and removed: tensorrt_llm/_torch/cute_dsl_kernels/,
tensorrt_llm/_torch/modules/fused_ops/,
tests/unittest/_torch/cute_dsl_kernels/,
tests/unittest/_torch/modules/fused_ops/
129 files move, 5 package markers are authored, 2 byte-empty markers at the
destinations are filled, and 3 byte-empty source markers are deleted.
Reviewer notes:
This is the largest commit and the only one with a non-empty effective-owner
diff. Read `git show -M --stat` first for the shape, then
`git show -M --diff-filter=M` for the edits, which are the part that can be
wrong. It is not splittable further without breaking the "every commit
imports" property: splitting T14 by domain strands blackwell/__init__.py and
moves top_k/ in a commit whose CODEOWNERS repoint lives elsewhere.
All 43 effective-CODEOWNERS owner changes in this PR land here, in five
classes: 19 kernels-devs -> torch-attention-devs, 12 kernels-devs -> moe-devs,
8 kernels-devs -> torch-visual-gen-devs, 3 runtime-devs -> kernels-devs, and
1 trt-llm-devs -> torch-visual-gen-devs. No file loses an owner. The full
per-file list is in the PR description; the runtime-devs -> kernels-devs class
fixes a pre-existing test/source ownership mismatch rather than creating one.
@NVIDIA/trt-llm-torch-attention-devs and @NVIDIA/trt-llm-torch-visual-gen-devs
are both blocking reviewers here; the MoE handle is @NVIDIA/trt-llm-moe-devs,
which unlike the other two carries no "torch-" segment.
In CODEOWNERS this commit removes five now-dead rules and REPOINTS exactly
two. The top-k slice of the owner diff is deliberately EMPTY: blackwell/top_k/
lands in the neutral tree and its rule is repointed, not deleted -- deleting
it would be an ownership loss for @NVIDIA/trt-llm-topk-devs. The other
repoint is test_indexer_topk.py. Both already sit below the rules they must
beat, so no CODEOWNERS line has to move. Textually the file goes 503 -> 499
lines: five rules deleted, two repointed in place, and a four-line comment
rewritten into five.
The CODEOWNERS comment above test_indexer_topk.py said the test's "final
home is settled together with the kernel's, not by this move". This is that
move: the comment is rewritten to record that the kernel landed in the
neutral tree and the test followed it out of thop/, while Attention keeps the
test as its primary consumer. Following the convention already used there, it
cites a PR rather than a tracker ID -- the literal #<PR> placeholder is
substituted for the real number before this branch is pushed.
tensorrt_llm/_torch/attention/kernels/__init__.py's docstring is widened.
Both of its previous claims -- "only production consumer is the attention
module layer" and "each submodule hosts one operator" -- became false once the
seven incoming groups (mla/, paged_mqa_logits/, kimi_k3_kda/, dspark/,
dspark_rmsnorm_rope.py, minimax_m3_index_decode_score.py and
rubin/dsv4_qb_fusion/) landed beside it: their consumers are mostly
_torch/custom_ops/ and the FMHA backend, not the module layer, and
blackwell/kimi_k3_kda/ alone hosts five modules.
@NVIDIA/trt-llm-torch-attention-devs own that file and are a blocking
reviewer.
tests/unittest/_torch/misc/test_rubin_kernel_arch.py scanned one rubin/ root.
rubin/ now splits three ways, so the constant becomes a three-root tuple and
the anti-vacuous assert moves inside the loop, per root: checking the roots in
aggregate would stay green with two of the three matching nothing. Coverage is
unchanged at 9/9 self.arch assignments, distributed 4/1/4.
The nine T22 benchmark drivers that load a T14 kernel have their offline
sys.path fallback replaced by the shared _offline_loader stub-loader
introduced in C2 (ten edits across nine files). The old fallback appended a
path below tensorrt_llm/, which cannot work once the kernels live under a
package: every parents[N] is left exactly as it was, and the installed import
path is unchanged.
The non-code surface that keys on these paths follows them: l0_b200.yml (4
entries), legacy-files.txt (22 keys repointed, 2 dropped) and its three
generated lint mirrors -- pyproject.toml, ruff-legacy.toml and
.pre-commit-config.yaml -- regenerated with
`python scripts/legacy_utils.py gen-configs`, never hand-edited, and
.test_durations. The .test_durations key for test_nvfp4_conv3d.py is rekeyed
with it; it is the one key in that file belonging to a test this commit moves.
pyproject.toml also carries two hand-maintained ruff exclude entries for the
NVIDIA#18357 Rubin files; gen-configs does not own that list, so they are
repointed by hand.
Prose that names the old tree is updated in the same commit:
docs/source/torch/adding_custom_kernels.md (the CuteDSL rows, the two
worked-example links, the code-block header and the sm_100+ half of the
compute-capability caveat that C2 split out for exactly this purpose) and
MOE_DEVELOPER_GUIDE.md. The l0 entries for the two top-k tests that leave
thop/parallel/ are re-added in C4 together with the rest of that directory;
only the PR head is built and tested, so the gap does not reach CI.
Two moved test files change their pytest ancestry in ways worth knowing:
test_nvfp4_conv3d.py lands flat under tests/unittest/_torch/visual_gen/
kernels/ (not under kernels/parallel/, which is already a directory-level l0
entry on five stages -- landing it there would change CI stage assignment),
and it therefore gains tests/unittest/_torch/visual_gen/conftest.py as an
ancestor. That conftest defines exactly one module-scoped opt-in fixture and
no hooks and no autouse, so collection and behaviour are unchanged; it is
called out in the PR description anyway.
The 2 filled byte-empty markers appear as create+delete rather than rename.
`git show -M --summary` prints 127 rename / 7 create / 5 delete here, not
129/5/3 -- a reference number, not an acceptance criterion. What is asserted
is per-file: every moved path is a detected rename to its mapped
destination, or had a byte-empty source and so carried no blame. All five
deletes measure 0 bytes at this commit's parent, so none of them could have
paired at any rename threshold.
tensorrt_llm/_torch/visual_gen/cute_dsl_kernels/ is a DIFFERENT, same-named,
out-of-scope tree of 13 files. It must stay byte-identical. Every rewrite in
this PR is anchored on a full path prefix, never on the bare token, for
exactly this reason.
Once the Attention kernels live under _torch/attention/kernels/, the custom
ops import them at module level, so `import tensorrt_llm._torch` now loads the
lazy tensorrt_llm._torch.attention package root. The package-root check in
tests/unittest/_torch/attention/test_package_exports.py therefore no longer
requires the root to be absent; it asserts that importing the root adds
nothing beyond the root itself and that nothing under the package other than
kernels/ is loaded, so the module layer and the backends stay unloaded.
The two blog links in docs/source/blogs/tech_blog/blog24_MoE_as_Dense_GEMM.md
are left pointing at /blob/main/ and registered as known-broken: published
blog paths are preserved by a separate ticket and are not silently rewritten
here.
Signed-off-by: Yihui Lu <269394165+YihuiLu512@users.noreply.github.com>
…ttest/_torch/kernels
Pure relocation plus the CI registration it invalidates. Path mapping
(old -> new):
Tests -- consolidated into tests/unittest/_torch/kernels/
tests/unittest/_torch/thop/parallel/ -> .../kernels/parallel/ (40 files)
tests/unittest/_torch/thop/parallel_hw_agnostic/ -> .../kernels/parallel_hw_agnostic/ (15 files)
tests/unittest/_torch/thop/serial/ -> .../kernels/serial/ (3 files)
58 files move, and git detects all 58 as renames (57 at 100% similarity, plus
the one test whose helper import is rewritten): this commit creates no file
and deletes none. The parallel / parallel_hw_agnostic /
serial level is preserved -- it is an xdist execution contract, not an
ownership axis.
Seven of the moved files arrived from upstream after this branch was first
cut and simply follow their directories: thop/parallel/
test_cute_dsl_bf16_bmm_fp8out.py, test_kimi_k3_fp8_weight_read_linear.py,
test_compressor_ragged.py, test_indexer_topk_ragged.py,
test_cute_dsl_bf16_dense_locality_domain_rubin.py and its helper
_cute_dsl_bf16_rubin_test_utils.py (not MoE tests, so they leave thop/ with
their neighbours), and
thop/parallel_hw_agnostic/test_minimax_m3_nvfp4_horizontal_producer.py, whose
l0_b200.yml entry and .test_durations key are repointed with the others.
Exactly one import rewrite lands in this commit:
test_cute_dsl_bf16_dense_locality_domain_rubin.py imports its helper by package
path, so "from _torch.thop.parallel._cute_dsl_bf16_rubin_test_utils import"
becomes "from _torch.kernels.parallel._cute_dsl_bf16_rubin_test_utils import"
as the two files move together. Beyond that line, the moved tests are not
reference-free either: test_fp8_block_scale_gemm.py imported
tensorrt_llm._torch.cute_dsl_kernels.rubin.dense_blockscaled_gemm_persistent,
and that line was repointed in the T15 commit, when the kernel it names moved.
The referring file moves here; its one edited line did not.
Apart from that import line, the only content edit in this commit is a C++
comment:
cpp/include/tensorrt_llm/deep_gemm/scheduler.cuh
"// Need to keep the same as the one in
tests/unittest/_torch/thop/parallel/deep_gemm_tests.py"
-> ".../kernels/parallel/deep_gemm_tests.py"
That single line is the entire cpp/ exposure of this PR; cpp/ has zero
cute_dsl_kernels and zero cuda_tile_kernels hits.
.github/CODEOWNERS is not touched by this commit (verified: zero-line diff).
The rule "/tests/unittest/_torch/kernels @NVIDIA/trt-llm-kernels-devs" already
exists -- it was added for the whole kernels/ tree earlier in this PR -- so
every moved test keeps the owner it had under "/tests/unittest/_torch/thop".
CI registration updated alongside the move:
tests/integration/test_lists/test-db/ -- 8 l0 files
tests/integration/test_lists/waives.txt -- 4 node IDs, re-sorted in place
tests/integration/defs/agg_unit_mem_df.csv -- 7 rows
tests/integration/defs/.test_durations -- 13 keys rekeyed
legacy-files.txt -- 23 keys, plus the 3 generated lint mirrors
(.pre-commit-config.yaml, pyproject.toml, ruff-legacy.toml)
Reviewer notes:
tests/unittest/_torch/thop/ does NOT disappear. T18's five MoE tests still
live there and are out of scope for this PR, so the CODEOWNERS rule
"/tests/unittest/_torch/thop @NVIDIA/trt-llm-kernels-devs" is KEPT, along
with the two moe-devs rules under it, and the thop/{parallel,serial} l0
entries are kept alongside the new kernels/ ones rather than replaced. T20's
"retire thop/" acceptance clause is therefore only partly discharged here; it
closes when T18 lands. This is stated in the PR description.
thop/parallel_hw_agnostic, by contrast, reaches zero files: all 15 of its
tracked files are in this move and the directory has no __init__.py and no
conftest.py. The l0_cpu.yml and l0_h100.yml entries for it are therefore
REPLACED, not kept -- a directory entry that collects nothing fails the stage
outright. The now-empty directory is removed.
The kernels/parallel directory entries added here are also what gives CI
coverage to the two top-k tests that the T14 commit moved into it
(test_indexer_topk.py, test_gvr_selfsampling_topk.py). Between that commit
and this one they sat in a directory no l0 list named. Nothing ships from an
intermediate commit -- CI builds and tests the PR head -- but the transitional
state is real and is recorded here rather than left to be rediscovered.
tests/integration/defs/agg_unit_mem_df.csv is the fatal-and-silent item in
this commit. Its lookup key is the byte-exact l0 entry string; a miss is a
warnings.warn plus num_workers = 1, so the tests stay green and then run out
of their TIMEOUT and get written off as flakes. Six rows are mirrored for the
new paths with their values copied unchanged, and the parallel_hw_agnostic row
is moved because its old key ceases to exist. That is 6 + 1 = 7 rows matching
unittest/_torch/kernels and 6 still matching unittest/_torch/thop. A gate that
expects 6 new rows goes red on a correct edit, and the obvious way to make it
green again is to drop the moved row -- which is the silent failure itself.
The copied values are INHERITED, NOT MEASURED; this is recorded in the ticket.
waives.txt carries the four test_fp4_linear.py::test_fp4_gemm_bias_per_backend
entries as :: node IDs, so scripts/check_test_list.py resolves them and emits
a fatal FILE NOT FOUND. This is the one place in the repo that catches a
missed path, and it blocks the commit. The four lines relocate within the file
and the file must stay sorted (the file-contents-sorter pre-commit hook
enforces it). scripts/check_test_list.py is clean at this commit.
The new kernels/parallel entry ships the same TIMEOUT (90) as thop/parallel
and thop/parallel's own value is left alone. That is strictly conservative and
deliberately unmeasured -- the tickets require the cap to be set from a real
measurement, which needs a GPU run this round cannot do. Re-tune post-merge.
jenkins/** needs zero edits: L0_MergeRequest.groovy's relatedFileList has no
thop entry, and the prefixes that do match ("tensorrt_llm/_torch/" and
"tests/unittest/_torch/") are invariant across this move. Declared here so it
does not read as an omission.
The six l0 banner comments that mention kernels stay TRUE and are not
corrections: each is scoped by the "# Covers tests/unittest/_torch/attention/"
line directly above it and describes tests/unittest/_torch/ATTENTION/kernels/,
which this PR does not touch -- not the new top-level
tests/unittest/_torch/kernels/. Clarifying them so the two cannot be confused
is optional.
Signed-off-by: Yihui Lu <269394165+YihuiLu512@users.noreply.github.com>
… casebook skill
Two skill reference documents name kernel paths that do not resolve:
.claude/skills/perf-optimization-casebook/references/kernel-and-fusion/
specialize-topk-selection-kernel.md:98
.../custom_ops/cute_dsl_kernels/blackwell/top_k/single_pass_multi_cta_radix_topk.py
-> tensorrt_llm/_torch/kernels/blackwell/top_k/single_pass_multi_cta_radix_topk.py
fuse-moe-routing-kernel.md:94
pytest tests/unittest/_torch/thop/parallel/test_noaux_tc.py
-> pytest tests/unittest/_torch/kernels/parallel_hw_agnostic/test_noaux_tc.py
Both were already wrong before this relocation started, and each for its own
reason. Measured at this branch's base, 0beff32:
* no path matching custom_ops/cute_dsl_kernels existed anywhere in the tree.
That kernel lived at tensorrt_llm/_torch/cute_dsl_kernels/blackwell/top_k/,
so the custom_ops/ segment was never right.
* tests/unittest/_torch/thop/parallel/test_noaux_tc.py did not exist either.
That test lived in thop/parallel_hw_agnostic/, so the sub-directory was
named wrongly.
They are fixed in passing while the surrounding paths are being audited, and
both new targets are tracked files at this commit. Keeping this on [None]
because neither document appears in any of the four tasks' file sets.
Reviewer notes:
AGENTS.md is deliberately NOT touched. Re-measured at 0beff32, it has zero
matches for cute_dsl, _torch/thop, fused_ops, cuda_tile and kernels/ alike --
its key-files table never named these trees. Adding a new _torch/kernels/ row
would be an addition, not a repoint, and belongs in its own change.
The bare "and test_moe.py" on the line below the second edit is left alone.
It has no directory of its own and never had one; the file it means is
tests/unittest/_torch/thop/serial/test_moe.py, which belongs to the MoE test
consolidation task and is out of scope for this PR.
This commit renames nothing and authors nothing -- it is two whole-line
substitutions in two markdown files.
Signed-off-by: Yihui Lu <269394165+YihuiLu512@users.noreply.github.com>
…WNERS comment The comment above the test_indexer_topk.py ownership rule carried a literal "#<PR>" placeholder, which could only be resolved once the pull request existed. Replace it with the real number. Comment text only; no ownership rule is changed. Signed-off-by: Yihui Lu <269394165+YihuiLu512@users.noreply.github.com>
… warmup import check - Drop the TIMEOUT (90) override on unittest/_torch/thop/parallel in l0_b200/l0_b300/l0_gb300. After the split the directory only holds the two MoE test files (~5-7 min), so the default per-entry limit is sufficient; unittest/_torch/kernels/parallel, which inherits the former load, keeps 90. - Add a GPU-free test, marked cpu_only so the CPU stages collect it, that statically resolves every try/except ImportError import in the DSA metadata warmup paths, so a stale kernel path fails CI instead of silently skipping warmup. Signed-off-by: Yihui Lu <269394165+YihuiLu512@users.noreply.github.com>
…the cute_dsl_kernels split NVIDIA#19960 added eight imports under tensorrt_llm._torch.cute_dsl_kernels, which this PR removes. The modules they name now live in the Attention and MoE kernel trees, so point the imports there: - tensorrt_llm/_torch/custom_ops/cute_dsl_custom_ops.py: mla_helpers -> attention.kernels.blackwell.mla.mla_helpers - tensorrt_llm/_torch/modules/kimi_kda/_kda_kernels.py and tests/unittest/_torch/executor/kv_cache/test_mamba_cache_manager.py: kimi_k3_kda.kda_mtp_decode -> attention.kernels.blackwell.kimi_k3_kda.kda_mtp_decode - tests/unittest/_torch/moe/test_moe_backend.py: five cutedsl_megamoe modules -> moe.kernels.cutedsl_megamoe The target modules are unchanged copies of the ones the old paths named; only the import paths change. Signed-off-by: Yihui Lu <269394165+YihuiLu512@users.noreply.github.com>
Collaborator
Author
|
/bot run --disable-fail-fast |
Collaborator
|
PR_Github #78899 [ run ] completed with state |
Collaborator
|
PR_Github #78936 [ run ] triggered by Bot. Commit: |
Collaborator
|
PR_Github #78936 [ run ] completed with state
|
Collaborator
Author
|
/bot run --disable-fail-fast |
Collaborator
|
PR_Github #79085 [ run ] triggered by Bot. Commit: |
Collaborator
|
PR_Github #79085 [ run ] completed with state |
1 task done
1 task done
1 task done
1 task done
1 task done
1 task done
1 task done
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Description
Pure relocation closing four tasks of Epic TRTLLM-14558 — T14 (TRTLLM-14844), T15 (TRTLLM-14845), T20 (TRTLLM-14850), T22 (TRTLLM-14852) — landed together because they split the same two source directories. No runtime behaviour, no numerics, no public API and no configuration change.
What moves where
tensorrt_llm/_torch/cute_dsl_kernels/_torch/attention/kernels/,_torch/moe/kernels/,_torch/visual_gen/kernels/and a new neutral_torch/kernels/tensorrt_llm/_torch/cuda_tile_kernels/_torch/kernels/tensorrt_llm/_torch/modules/fused_ops/_torch/kernels/fused_ops/tests/unittest/_torch/thop/(all but T18's MoE tests)tests/unittest/_torch/kernels/tests/unittest/_torch/modules/fused_ops/tests/unittest/_torch/kernels/fused_ops/tests/unittest/_torch/cute_dsl_kernels/test_nvfp4_conv3d.py→tests/unittest/_torch/visual_gen/kernels/,test_custom_pipeline.py→tests/unittest/_torch/kernels/tests/scripts/cute_dsl_kernels/tests/microbenchmarks/cute_dsl_kernels/Moved files are git-detected renames; the only authored files are package
__init__.pymarkers and one new module,tests/microbenchmarks/cute_dsl_kernels/_offline_loader.py. Byte-empty__init__.pyfiles are re-created at their new levels rather than renamed. Theblackwell/andrubin/architecture levels are preserved inside each destination. Files that landed upstream inside a source root while this PR was open travel with the move (see Declared).Every source root ends at zero tracked files except
tests/unittest/_torch/thop/, which keeps T18's MoE tests;thop/parallel_hw_agnosticalso ends empty.Commits
_torch/kernels/custom_opssplit,_offline_loader.pycute_dsl_kernels/by domain ownershipcute_dsl_kernels/thop/unit testsagg_unit_mem_df.csv, l0 lists,waives.txt,.test_durations[None]— repoint two stale skill-doc pathsEach commit rewrites every repo-wide reference to the modules it moves, including references inside files a later commit will move. Commits 2–4 regenerate the same three lint mirrors via
scripts/legacy_utils.py gen-configs.Ownership
All effective CODEOWNERS owner changes land in commit 3, and no file loses an owner. The classes are kernels-devs → torch-attention-devs / moe-devs / torch-visual-gen-devs, runtime-devs → kernels-devs (fixing a pre-existing test/source mismatch), moe-devs → torch-attention-devs and trt-llm-devs → torch-visual-gen-devs. The top-k slice is empty, its exact-file rules being repointed rather than deleted.
GitHub reads CODEOWNERS from the base branch, so reviewers must be requested by hand:
@NVIDIA/trt-llm-kernels-devs,@NVIDIA/trt-llm-torch-attention-devs,@NVIDIA/trt-llm-moe-devs,@NVIDIA/trt-llm-torch-visual-gen-devs,@NVIDIA/trt-llm-topk-devs,@NVIDIA/trt-llm-runtime-devs, the KV-Cache, Models and CBTS owners,@NVIDIA/trt-llm-infra-devsand@NVIDIA/trt-llm-oss-compliance.Declared
tests/microbenchmarks/is in the CBTSOUT_OF_SCOPE_PREFIXESlist and is stripped byscripts/package_trt_llm.py;tests/scripts/was on neither.cute_dsl_kernels/mega_moe_nvfp4/withcute_dsl_kernels/cutedsl_megamoe/. This PR drops the stale relocation of the removed files and relocates the new tree to_torch/moe/kernels/cutedsl_megamoe/; the package is self-contained (no relative import escapes its root), so nothing inside it is edited.cutedsl_megamoeCODEOWNERS rule is deleted rather than repointed, because/tensorrt_llm/_torch/moe @NVIDIA/trt-llm-moe-devsalready covers the destination. Effective ownership of the tree is unchanged.manual_mma_128dp.pyandrubin_contiguous_grouped_blockscaled_gemm_fused_fc12.pyundercute_dsl_kernels/rubin/moe/; they follow that directory to_torch/moe/kernels/rubin/, and theircustom_opsimport andpyproject.tomlruff-exclude entries are repointed.test_minimax_m3_*tests inthop/parallel_hw_agnostic/(their per-file l0 entries are repointed), the non-MoE tests added tothop/parallel/, andtest_custom_pipeline.py, whose file-path lookup is repointed to_torch/kernels/blackwell/custom_pipeline.py./blob/main/links in a published blog post point at files this PR moves and will 404 after merge. They are left unchanged in this PR.examples/layer_wise_benchmarks/parser_utils.pykeys on kernel module paths embedded in demangled CUDA symbols. The file is left at zero diff.tests/unittest/_torch/thop/survives holding T18's MoE tests, so its CODEOWNERS rule andthop/{parallel,serial}l0 entries are kept alongside the new ones. The newkernels/parallelentry carriesTIMEOUT (90)forward unmeasured.test_argmax.pyis a new test collection: it leavestensorrt_llm/fortests/unittest/_torch/kernels/parallel/, is picked up by a directory-level l0 entry, and gains agetSMVersion() < 90skip.test_nvfp4_conv3d.pygains one new ancestor conftest,_torch/visual_gen/conftest.py.agg_unit_mem_df.csvmirrors eachunittest/_torch/thop/{parallel,serial}row underunittest/_torch/kernels(and moves theparallel_hw_agnosticrow), and thetest_fp4_linear.py::test_fp4_gemm_bias_per_backendnode IDs inwaives.txtrelocate.sys.pathfallback is replaced by the shared_offline_loader.py, which seedstensorrt_llmandtensorrt_llm._torchintosys.modulesand prepends the checkout to their__path__.self.archassignments unchanged.jenkins/**is untouched.Verification
Re-verified at this head after the latest rebase: import and path resolution (Python AST plus path tokens in text files) compared against
mainwith no new unresolved reference, every vacated source root at zero tracked files, no CODEOWNERS pattern newly matching nothing, test-list path existence forwaives.txtand the l0 test-db, a residue scan for every old path token, andpre-commit run --from-ref main(all hooks pass, includingValidate test list entries exist in source files). Carried over from an earlier head and not re-run here: per-file blame preservation, the CBTS dry-run, and the throwaway-clone rehearsal.Collection invariance is argued, not measured: moved test files change only in the import and path references repointed for the move (plus the declared
test_argmax.pyskip), no__init__.pyappears or disappears on any ancestor test path, and conftest ancestry is unchanged except for the one declared case.The container import gate and the pytest collection gate have not been run at the current head.
Hardware preflight ran at an earlier base only, on SM100; its one failure is a pre-existing defect in
run_moe_as_dense_gemm_fc2.py. Deferred:test_argmax.py's collected-count delta and its skip firing, measured wall-clock and xdist memory fit for the newkernels/parallelentry, whethertest_fp8_block_scale_gemm.pystill findsdeep_gemm_tests.py, the post-move driver re-run, and the SM107 drivers in either direction.PR Checklist
tensorrt_llm._torch.*), every in-repo reference is rewritten in the same commit, and there are no new dependencies or new code paths.__init__.pymarkers and one small helper; copyright years are updated on every modified file.docs/source/torch/adding_custom_kernels.md,MOE_DEVELOPER_GUIDE.md, the_torch/attention/kernels/__init__.pycharter and two skill reference documents.To see a list of available CI bot commands, please comment
/bot help.Dev Engineer Review
_torch/kernelstrees._torch/thoptests moved under_torch/kernels.QA Engineer Review
test_argmax.pyadds a test collection.test_nvfp4_conv3d.pygains a VisualGen conftest ancestor.tests/integration/defs/.test_durationsandtests/integration/defs/agg_unit_mem_df.csv.6581067.Per-File QA Perspective
tensorrt_llm/_torch/**: Verify relocated imports, lazy imports, architecture-gated package boundaries, and custom-op dispatch. The intended behavior is unchanged.tests/unittest/_torch/**: Verify relocated tests, updated mocks and patches, architecture skips, Rubin root scanning, and VisualGen coverage. Corresponding CI paths were updated where applicable.tests/integration/test_lists/test-db/*.yml: Verify that each relocated suite remains assigned to the correct hardware and timeout.tests/integration/test_lists/waives.txt: Verify the four relocated FP4 linear bias waivers and bug6581067.tests/integration/defs/.test_durationsandagg_unit_mem_df.csv: Verify that duration and aggregate-metrics keys match the relocated tests.tests/microbenchmarks/cute_dsl_kernels/**: Verify new kernel imports, offline fallback behavior, repository-root detection, and compiled-artifact preservation..github/CODEOWNERS: Verify ownership for new kernel, benchmark, Attention, MoE, Top-K, and test paths..pre-commit-config.yaml,legacy-files.txt,pyproject.toml,ruff-legacy.toml, andruff-legacy-baseline.json: Verify formatting, legacy tracking, and lint-baseline coverage for new paths.cpp/include/tensorrt_llm/deep_gemm/scheduler.cuh: Verify the synchronized test-path comment.