[NV] dsr1 fp4 b200 trt agg mtp update by camiloamoreno · Pull Request #642 · SemiAnalysisAI/InferenceX

camiloamoreno · 2026-02-05T01:34:59Z

This PR contains below updates:

Update to the latest TRTLLM 1.2 release container, the recent rc6.post3
Fine-tune choice of parallelism in nvidia-master (reduce overlapping TP8/TP4)
Enable Piecewise cuda graphs optimization on specific cases

Near the top of the benchmark script (L26-35), we enable specific optimizations, mainly differentiating between cases with and without DP attention, including for the choice of MTP aggressiveness.

As in the non-mtp fp4 agg version, we use Piecewise Cuda Graphs (https://nvidia.github.io/TensorRT-LLM/features/torch_compile_and_piecewise_cuda_graph.html) which enables some components to execute thorugh cuda graphs while other components are run eagerly, to gain benefit with lower overhead. We use the formula from the documentation to generate a capture_num_tokens list depending on MAX_NUM_TOKENS.

"cuda graph max batch size" is optimized to match CONC as a natural limit, but reduced to batch_size/4 when DP_ATTENTION is enabled.

claude · 2026-02-05T23:49:41Z

Claude finished @camiloamoreno's task in 1m 52s —— View job

PR Review: [NV] dsr1 fp4 b200 trt agg mtp update

Read changed files and gather context
Validate master config (model prefixes, image accessibility)
Validate perf-changelog presence
Validate benchmark script (code style, MTP requirements)
Check for bugs and logic errors
Post review findings

LGTM — no blocking issues found.

Validation checklist:

Model prefix: dsr1 — valid
Image: nvcr.io#nvidia/tensorrt-llm/release:1.2.0rc6.post3 — public NGC image, valid
perf-changelog.yaml: Updated with matching PR link and description
--use-chat-template: Present in benchmark client (line 132)
Server launch formatting: Arguments on separate lines with \ continuations
Piecewise CUDA graphs logic: Capture token list generation correctly implements the documented formula; seq/modulo logic correctly avoids duplicates at the boundary
MAX_BATCH_SIZE: Correctly bounded to CONC/4 with DP attention, CONC otherwise; consistently used in both the YAML config and the server launch args

kedarpotdar-nv

lgtm. this should get picked up after current GPTOSS run is complete

DeepEP's low-latency combine corrupts the 256 tokens/rank rung on every Blackwell SKU we run -- B200, GB200 and GB300, EP8 and EP16, both precisions, MNNVL and RDMA alike -- while Hopper stays clean. It is stochastic at roughly 1.5-3.3% per invocation and surfaces as one wrong token row whose norm still matches, so the correctness gate catches it as a 0.07-6.6 relative error against a 0.03125 tolerance. Tracked upstream as DeepEP issue #700. Clamp the measured low-latency ladder to 128 and leave the receive sized at 256. The two are now separate constants on purpose: the receive footprint drives the transport's memory traffic and the FP8 dequant volume, so sizing it from max(ladder) -- as it was -- would have halved it the moment the ladder moved and shifted every retained rung out of comparability with the published series. Holding it at 256 also leaves the top measured rung at half occupancy, which is the ladder/capacity decoupling the original capacity probe had to hand-roll. The clamp is not silent: the harness already reports every dropped ladder point into the artifact. The likely upstream fix is DeepEP PR #642, which adds a CTA-scope fence so the combine consumer's shared-memory reads retire before the stage is recycled and the producer's next TMA load refills it -- that mechanism predicts the observed signature, and it closed #621, the same race reached from NVL72. Our pin is the head of PR #605 and was branched before #642 merged, so the fence is simply absent from our build. Raising the ladder back to 256 is therefore gated on a pin bump, deliberately not bundled here: it spans months of upstream change, re-baselines every deepep-v2 row including normal mode, and needs rewrite_deepep_v2 made tolerant first, since main already carries the 'libnccl' fix that rewrite asserts it must apply.

…lamp Fixes the defect instead of stepping around it. b368eae clamped the low-latency ladder to 128 so the corrupt 256 rung was not measured; the corruption itself was still in the kernels we publish numbers from, at roughly 1.5-3.3% per invocation on every Blackwell SKU. The fix was already upstream. DeepEP PR #642 adds a CTA-scope fence.proxy.async.shared::cta before mbarrier_arrive(empty_barriers[stage_idx]) in LOW_LATENCY_COMBINE_RECV, so the combine consumer's shared-memory reads retire before the stage is declared empty and the producer's next TMA load refills it -- signalling empty too early is what let one output row be assembled from two tiles, which matches the observed signature exactly (norm preserved to 4 s.f., 16-40% of elements wrong). It closed #621, the same race found independently on NVL72. COLLX_DEEPEP_V2_COMMIT was fa8a9b16, the head of the pre-merge PR #605 branch, cut before #642 landed. Its one unique commit was the #630 single-node V2 init fix, which main carries as 56169594e, so moving to main loses nothing and also picks up #715 (system-scope release before the GIN barrier when scale-up spans NVLink and RDMA), #688 (NCCL Device API: runtime version for ncclDevCommCreate), #178 (SM90), #641, and #640/#627 upstream's own libnccl and SO-name handling. Verified before bumping that main still exposes every API this adapter calls: legacy Buffer kwargs incl. allow_mnnvl, get_low_latency_rdma_size_hint, low_latency_dispatch/combine, ElasticBuffer's full kwarg set, topk_idx_t, and the fp8 cast helpers. rewrite_deepep_v2 now succeeds when the source already matches, because main contains #640 and the old count(old) == 1 assertion would have aborted every leg at repository-stage. Verified against all four source states: old form rewrites, fixed form is a no-op, absent and duplicated forms still fail. _LL_LADDER_CAP returns to 256. The two constants stay separate: the receive must not be sized from max(ladder), or clamping the ladder would change the footprint that drives transport traffic and fp8 dequant volume. The cap test now asserts ladder <= buffer -- strict inequality encoded the workaround, not an invariant. The backend cache key includes the pin, so this forces a rebuild rather than reusing the old build.

…pstream combine race gb300 nccl-ep low-latency BF16 failed correctness at T=256 and nowhere else: rungs T=1..128 pass with max relative error 0.0039, T=256 gave 0.4704. Both chained checks were clean there (chain_last_output_error exactly 0.0), so it is the drained oracle catching it, and the error is specifically the combine_values elementwise check -- the magnitude was computed at all, so shapes matched and it is not a payload or metadata failure. STOCHASTIC, measured: 1 failure in 5 executions of this shard (30934943352, 30969695966, 30981959724, 31200578704 pass; 31191722264 fails; an earlier container-import failure excluded as infra). The distribution is BIMODAL with nothing between the two values, which is a discrete corrupted write rather than a tolerance sitting near normal variance. MECHANISM: nccl_ep's low_latency.cu is a port of DeepEP's PRE-FIX low-latency combine. In the combine recv pipeline the reduction warps read shared memory and then mbarrier_arrive(emptyBarriers[stageIdx]) with no fence.proxy.async.shared::cta between, so the producer's next TMA load can overwrite a stage while consumer reads are in flight. DeepEP closed exactly this with a one-line fence in PR #642, and its defect signature matches ours: stochastic, Blackwell-only, top rung, one wrong row. The fence is absent at our pin AND at NVIDIA/nccl master, so this is unfixed upstream rather than a stale pin. The adapter's own known defect -- LL handles aliasing parity signal slots, NVIDIA/nccl#2303 -- is ruled out: this adapter uses one handle per group rebound per shape, and that defect wedges rather than corrupting silently. THE CLAMP IS NOT A SAFETY BOUNDARY and the code says so in those words. The fence is missing on every combine recv; T=256 is only the rung with the most pipeline iterations, and its receive plane is not even full. Lower rungs are less likely to hit the race, not immune. What the clamp buys is operational: unclamped, gb300 alone reds roughly 1 run in 5 on a defect we have already fully characterized, autopublish requires every leg green, and each red burns a full-matrix attempt plus a triage cycle -- while a stochastic red at a known site no longer teaches anyone anything. Against that: publishing a T=256 latency for a kernel we know stochastically corrupts its output at that rung, with a correctness gate that samples only 3-5 invocations per case, is itself the safety-implying act. A clamped ladder with an explicit note is the more honest artifact. Implemented as two constants mirroring ep_deepep_v2, and the split matters here more than it did there: nccl-ep sized its receive from spec.max_tokens_per_rank, which follows the CLAMPED ladder, so a naive clamp would have shrunk the transport footprint too and quietly re-based what T=128 measures. create_buffer now reads _LL_BUFFER_CAP directly. Tests pin both halves. Restore _LL_LADDER_CAP to _LL_BUFFER_CAP when a fixed wheel ships. Upstream report to NVIDIA/nccl is the primary action and is not blocked by this. 176 tests green.

… AsyncLL, DSv4-Pro workload (#2786) * CollectiveX: publish wire-basis bytes so LL bandwidth stops mixing byte definitions DeepEP/UCCL/NCCL low-latency kernels move one copy per (token, expert) assignment; normal mode and MoRI LL move rank-deduplicated (token, dest-rank) copies. byte_provenance always carried the smaller deduplicated numerator, so any bandwidth divided from it on a token-expert backend was a lower bound published as the wire rate (34.1% low on nccl-ep LL EP8 at T=128: 77.4 MB vs the 117.4 MB the kernels move) and not comparable across backends. Rows now carry wire_byte_provenance beside byte_provenance: identical for token-rank receives, per-assignment for token-expert ones, basis declared by the existing logical_copies.wire discriminator. bandwidth.py divides from the wire basis (falling back to the deduplicated one for pre-wire artifacts, which only ever understates). * CollectiveX: hold nccl-ep low-latency fleet-wide until the combine fence ships The T<=128 ladder clamp reduces exposure to the un-fenced shared-memory combine race (DeepEP #642's fence, absent at our pin and at NVIDIA master); the source has said all along it is not a safety boundary — lower rungs are less likely to hit the race, not immune. Green clamped rows are therefore not publication-valid, and describing T=256 as a capacity limit was wrong. Rows held in the registry, normal mode unaffected. * CollectiveX: charge nccl-ep's per-step routing collective to the timed HT dispatch ncclEpUpdateHandle is documented as a per-step collective and production routing changes every MoE layer, so a serving step pays it before every HT dispatch at the handle's full token capacity. Excluding it (as NVIDIA's ep_bench does) made HT dispatch the one window that omitted its routing work while deepep-v2, uccl-ep, MoRI and FlashInfer all carry theirs per call. kernel_generation bumps to nccl-ep-ht-routed — the per-row discriminator the earlier NCCL measurement changes lacked. * CollectiveX: measure MoRI low-latency on AsyncLL, the kernel production deploys SGLang's moriep dispatcher maps low-latency to AsyncLL split-phase (dispatch_send/recv + combine_send/recv, block_num 64 rdma 32 warps 8) and asserts that kernel type; IntraNodeLL is not selected by any engine, so the previous rows measured an off-production path (kernel_generation intranode-ll discriminates them). The timed windows now run AsyncLL send+recv back-to-back and the adapter fails closed on wheels whose AsyncLL lacks the recv API — single-call AsyncLL returns before any payload lands. Pending on-metal validation on mi355x. * CollectiveX: sync methodology with the wire basis, the nccl-ep LL hold and the routed HT window * CollectiveX: switch the EP workload to DeepSeek-V4-Pro routing deepseek-v4-pro: hidden 7168, top-k 6, 384 routed experts — the dsv4 shape the rest of InferenceX benchmarks (operatorx testlists, HF-config-derived). The workload name is part of every case_id, so this is a clean identity break in the durable store: no v3 row can collide with a v4-pro row. Ladders and seed unchanged. * CollectiveX: sync the README's MoRI low-latency kernel description with AsyncLL (review) * CollectiveX: rewrite the payload-bandwidth section around the wire basis (review) The Measurement section still described the rank-deduplicated count as the sole GB/s numerator and cited the retired IntraNodeLL as the dedup exemplar; bandwidth.py and the app now divide from wire_byte_provenance. The section now names both accountings, which rows differ, the 34% understatement that motivated the change, and the fallback behavior for pre-wire artifacts. * CollectiveX: exclude b300-001 and b300-016 (mlx5_4 DOWN) The full v4-pro sweep's two b300 EP16 legs each drew one of these nodes and failed closed at the network-profile gate (rdma-port-5=inactive = the 5th selector, mlx5_4). Idle-pool survey: exactly these two nodes have mlx5_4 port state 1: DOWN; the other 12 idle nodes are 4: ACTIVE — a sick node pair (likely one leaf/cable pair), not a rail-wide outage. SRE handback owed. * CollectiveX: hold gb300 low-latency until the nvshmem-over-MNNVL init regression is resolved All gb300 deepep-v2 LL shards die at the first CUDA op after Buffer init with cudaErrorUnknown (context poisoned during nvshmem symmetric-heap init over MNNVL), on every rank, both degrees and precisions. Discrimination: gb300 normal mode green at n2/n4; gb200 LL green with the same image/pin/workload; a main-branch control with the old workload fails identically — so neither this PR nor the v4-pro shape. CI-shaped 2-node hand probe (job 27430 on the pool) reproduces: bootstrap and buffer creation complete, then every rank's next CUDA call returns unknown error. Driver is 580.159.03 (installed in the KV-fix window ~2026-08-26..28, after the last green gb300 LL run). Platform escalation owed; rows held so sweeps don't burn four known-red shards each. * CollectiveX: bump the mi355x image to v0.5.18-rocm720-mi35x-20260830 (mori >= #505) AsyncLL under the DSv4-Pro workload (topk 6) dies on every shipped mi35x-mori image: ROCm/mori#505 (AsyncLL slot assignment double-allocates when top-k does not divide warpSize, fixed upstream 2026-07-31) postdates them all. On-metal probe of this date-stamped nightly (job 41563, pure mori.ops): AsyncLL topk6@384 and topk8@256 both pass, split-phase recv API present. mi355x only — the -tw SKUs keep their image until separately validated on their docker path. * CollectiveX: build DeepEP against the image-matched nvshmem (cu13) — un-breaks gb300 LL The gb300 LL cudaErrorUnknown was ours, not the platform's. 2x2 on the pool: stock in-image deep_ep (built on nvidia-nvshmem-cu13==3.4.5) runs LL clean on the exact r01 nodes and driver (580.159.03) where the CI stack dies; the CI venv's nvidia-nvshmem-cu12==3.3.9 r12 host library survives sm90/sm100 but poisons the CUDA context during symmetric-heap init over MNNVL on sm103 — buffer creation returns, the next CUDA call on every rank fails. The earlier driver-window attribution was a coincidence of timing (pin bump and driver install overlapped; hold commit e9ef099 superseded). The wheel spec now lives in common.sh (COLLX_DEEPEP_V2_NVSHMEM_SPEC), is installed by prepare_backend, and keys the venv cache so every NVIDIA deepep venv rebuilds once against cu13. gb300 ll_backends restored. * CollectiveX: torch 2.11.0 for the DeepEP venv — the actual gb300 LL fix Root cause, finally isolated by holding everything else constant: a venv identical to prepare_backend's recipe (same DeepEP pin, same nccl-cu13 2.30.4, same nvshmem-cu13 3.4.5) is green with torch==2.11.0+cu130 and red with torch==2.10.0+cu130 on the same gb300 r01 nodes (jobs 27760 vs 27430/27757). torch 2.10.0's bundled CUDA-13 userland poisons the CUDA context during nvshmem symmetric-heap init over MNNVL on sm103 + driver 580.159.03; sm90 and sm100 are unaffected. Falsified along the way, in order: the v4-pro workload (main control), the driver alone (stock image green on the failing nodes — the user's r03 canary forced that 2x2), the nvshmem cu12 wheel (cu13 rebuild still red), the DeepEP pin (old-pin venv red). 2.11.0 is what the cu130 image ships. The torch spec joins the nvshmem spec in the venv cache key, so every deepep venv rebuilds once. * CollectiveX: pin the RDC device-link arch — the real gb300 LL root cause The DeepEP build compiles every object with the right -gencode (setup.py 'Arch list: 10.3', all compile lines sm_103), but the final nvcc -dlink step receives NO gencode and falls back to nvcc's CUDA-13 default, sm_75 — the .so's single linked device image is sm_75 and its kernels can never load on the target GPU. On gb300/sm103 that surfaced as cudaErrorUnknown at the first legacy-kernel launch (LL and layout paths; ElasticBuffer JIT paths were unaffected, which is why normal mode stayed green). Proven by instrumented runs of prepare_backend's own build (job 27792: compile lines sm_103, step 9/9 -dlink bare, product sm_75) after hand builds of the identical recipe produced sm_103 and ran LL clean on the failing nodes. Fix: NVCC_PREPEND_FLAGS carries -gencode=arch=compute_<arch>,code=sm_<arch> into every nvcc invocation including the dlink. COLLX_DEEPEP_V2_BUILD_GEN (dlarch1) joins the cache key so the sm_75 venvs with valid .ready markers rebuild instead of being reused. The torch 2.10->2.11 bump (previous commit) stays: 2.11 matches the image and the 2.10-vs-2.11 A/B was run against differently-linked binaries, so its verdict is voided rather than reversed; the pin question can be revisited upstream of this fix if anyone cares to re-test.

… clamp and enable B200 EP16 The vendor shipped nccl-ep v0.2 as the nccl-extensions PyPI wheel (nccl4py stopped bundling nccl/ep at 0.4 and now provides only nccl.core). The install spec becomes two pinned wheels — nccl-extensions[cu13]==0.1.0 plus nccl4py[cu13]==0.5.0 — so a cache rebuild resolves the same tree, and the spec string change itself re-keys the shared cache. The import probe now logs get_lib_version()/get_lib_path() so a stale or shadowing libnccl_ep.so is visible in the run log. The v0.2 wheel ships the combine-recv fence v0.1's port of DeepEP lacked (the PR #642 analogue: fence_view_async_shared before mbarrier_arrive(emptyBarriers) in the shipped ll_ep.cuh) — the documented restore condition for the T<=128 ladder clamp — so the LL ladder returns to the full 256-slot receive and LL rows are enabled: EP8 on h100/h200/b300, EP8+EP16 on b200/gb200/gb300. b200 normal mode also gains EP16 (v0.2 fixes the RDMA EP16 GIN fault that pool hit under v0.1); h100/h200/b300 EP16 stays unsupported pending revalidation, since only the B200 configuration is claimed fixed. kernel_generation bumps to nccl-ep-v02-ht-routed / nccl-ep-v02-ll so pre-upgrade rows never pool with the new mover. The adapter's call surface is unchanged — every class and signature it uses is identical in v0.2. Still BF16 only: v0.2's FP8 dispatch (DS_FP8E3M4) and experimental NVFP4 combine recipes are separate bring-up. All claims beyond the fence (which is verified present in the published wheel) are validated by the correctness oracle on metal.

update nvidia master and benchmark options

444c823

camiloamoreno added the NVIDIA label Feb 5, 2026

camiloamoreno requested a review from a team as a code owner February 5, 2026 01:35

github-project-automation Bot added this to InferenceMAX Board Feb 5, 2026

camiloamoreno marked this pull request as draft February 5, 2026 04:52

camiloamoreno added 3 commits February 5, 2026 13:24

bugfix

9670ce5

perf changelog

f0fd116

Merge branch 'main' into nv/dsr1-fp4-b200-trt-agg-mtp-260204

f5f523c

camiloamoreno changed the title ~~[WIP] [NV] dsr1 fp4 b200 trt agg mtp update~~ [NV] dsr1 fp4 b200 trt agg mtp update Feb 5, 2026

camiloamoreno requested a review from kedarpotdar-nv February 5, 2026 23:46

camiloamoreno marked this pull request as ready for review February 5, 2026 23:49

camiloamoreno added the sweep-enabled label Feb 5, 2026

kedarpotdar-nv approved these changes Feb 6, 2026

View reviewed changes

cquil11 approved these changes Feb 6, 2026

View reviewed changes

Merge branch 'main' into nv/dsr1-fp4-b200-trt-agg-mtp-260204

04fd054

camiloamoreno merged commit 0aae52d into main Feb 6, 2026
5 of 33 checks passed

camiloamoreno deleted the nv/dsr1-fp4-b200-trt-agg-mtp-260204 branch February 6, 2026 17:21

github-project-automation Bot moved this to Done in InferenceMAX Board Feb 6, 2026

This was referenced Aug 30, 2026

CollectiveX: wire-basis bandwidth, nccl-ep LL hold, routed HT window, AsyncLL, DSv4-Pro workload #2786

Merged

CollectiveX: NCCL EP v0.2 via nccl-extensions — lift the LL clamp, enable LL rows and B200 EP16 #2815

Merged

Navigation Menu

Uh oh!

[NV] dsr1 fp4 b200 trt agg mtp update - #642

[NV] dsr1 fp4 b200 trt agg mtp update#642
camiloamoreno merged 5 commits into
mainfrom
nv/dsr1-fp4-b200-trt-agg-mtp-260204

camiloamoreno commented Feb 5, 2026 •

edited

Loading

Uh oh!

claude Bot commented Feb 5, 2026 •

edited

Loading

Uh oh!

kedarpotdar-nv left a comment

Uh oh!

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

3 participants

Uh oh!

Conversation

camiloamoreno commented Feb 5, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

claude Bot commented Feb 5, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

PR Review: [NV] dsr1 fp4 b200 trt agg mtp update

Uh oh!

kedarpotdar-nv left a comment

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

3 participants

camiloamoreno commented Feb 5, 2026 •

edited

Loading

claude Bot commented Feb 5, 2026 •

edited

Loading