[https://nvbugs/6342844][fix] Prevent disaggregated KV transfer stalls by chienchunhung · Pull Request #15737 · NVIDIA/TensorRT-LLM

chienchunhung · 2026-06-29T23:18:56Z

Summary

Fix the CacheSender lost-wakeup/data race that can leave generation workers stalled while a ready KV-cache response remains queued. The change also aligns synchronous transfer admission and error handling without changing the default asynchronous production path or adding a public API.

Observed qualification shows the original liveness failure is resolved for every targeted workload that reached KV transfer. Eight perf-sanity cases passed completely, two completed but failed separate performance gates, and one GB300 Kimi case has not produced a valid product result because generation ranks repeatedly crashed while loading shared model weights.

Changes

Guard the sender response queue, active request, cancellation state, and readiness predicate with mSenderMutex; remove the split mAnyReady/condition-mutex protocol that could lose a notification.
Keep NIXL/UCX, CUDA, promise, and other blocking work outside the sender critical section, and settle outstanding work during error or shutdown.
Preserve synchronous admission limits, skip asynchronous status polling only for real synchronous transfer, and bypass KV admission only for synthetic gen_only_no_context requests that perform no transfer.
Release the Python GIL around synchronous receive and propagate transfer failures into rank-aligned request handling.
Add a 64-iteration C++/NIXL synchronous-transfer stress test and focused PyExecutor unit coverage.
Raise free_gpu_memory_fraction from 0.90 to 0.95 only in the GB200 DeepSeek con3072 test configuration.

The PR is organized as three reviewable commits: production fix and unit tests, C++/NIXL stress coverage, and qualification/waiver updates.

Test scope and accounting

🟢 Passes · 🟠 Liveness fixed; separate gate fails · 🟡 No valid product result/pending · ⚪ Not addressed

Waiver status compares current upstream main with this PR after the latest rebase. Current main independently re-waived the DeepSeek perf cases under NVBug 6422339; this PR preserves those newer waivers even where an earlier PR pipeline had already demonstrated a pass.

Full test name	Stage	Result and PR scope	Waiver: main → PR	NVBug
`unittest/others/test_kv_cache_transceiver.py:: test_cpp_nixl_sync_transfer_stress`	`B200_PCIe-PackageSanityCheck- PY312-DLFW`	🟢 Direct regression: 64 sequential C++/NIXL synchronous transfers passed in #45579.	Added and unwaived	6342844
`disaggregated/test_disaggregated.py:: test_disaggregated_stress_test[ input8k-output1k-conc512-qwen3_32b_fp8_stress]` (B200)	`LLM_FUNCTION_CLUSTER_TEST`	🟡 Async sender qualification; QA execution remains pending.	Waived → unwaived	6312828, 6342844
`perf/test_perf_sanity.py::test_e2e[ disagg_upload-gen_only-gb200_deepseek-r1-fp4_ 1k1k_con1024_ctx1_dep4_gen1_dep8_eplb0_mtp0_ccb-NIXL]`	`GB200-12_GPUs-3_Nodes-PyTorch-Disagg- PerfSanity-CTX1-NODE1-GPU4-GEN1-NODE2-GPU8- Post-Merge-2`	🟢 Original liveness case; passed in #45622.	Waived (6422339) → waived	6342844
`perf/test_perf_sanity.py::test_e2e[ disagg_upload-gen_only-gb200_deepseek-r1-fp4_ 1k1k_con2048_ctx2_dep4_gen1_dep16_eplb0_mtp3_ccb-NIXL]`	`GB200-24_GPUs-6_Nodes-PyTorch-Disagg- PerfSanity-CTX2-NODE1-GPU4-GEN1-NODE4-GPU16- Post-Merge-1`	🟢 Original intermittent liveness case; passed in #45622.	Waived (6422339) → waived	6342844
`perf/test_perf_sanity.py::test_e2e[ disagg_upload-gen_only-gb200_deepseek-r1-fp4_ 1k1k_con2048_ctx2_dep4_gen1_dep16_eplb288_mtp3_ccb-NIXL]`	`GB200-24_GPUs-6_Nodes-PyTorch-Disagg- PerfSanity-CTX2-NODE1-GPU4-GEN1-NODE4-GPU16- Post-Merge-2`	🟢 Original liveness case; passed in #45622.	Waived (6422339) → waived	6342844
`perf/test_perf_sanity.py::test_e2e[ disagg_upload-gen_only-gb200_deepseek-r1-fp4_ 1k1k_con3072_ctx1_dep4_gen1_dep4_eplb0_mtp1_ccb-NIXL]`	`GB200-8_GPUs-2_Nodes-PyTorch-Disagg- PerfSanity-CTX1-NODE1-GPU4-GEN1-NODE1-GPU4- Post-Merge-3`	🟢 Passed in #45816 after the test-only KV-capacity adjustment.	Waived (6422339) → waived	6342844
`perf/test_perf_sanity.py::test_e2e[ disagg_upload-gen_only-gb300_deepseek-r1-fp4_ 1k1k_con3072_ctx1_dep4_gen1_dep4_eplb0_mtp1_ccb-NIXL]`	`GB300-8_GPUs-2_Nodes-PyTorch-Disagg- PerfSanity-CTX1-NODE1-GPU4-GEN1-NODE1-GPU4- Post-Merge-2`	🟢 Original liveness case; passed in #45622.	Waived (6422339) → waived	6342844
`perf/test_perf_sanity.py::test_e2e[ disagg_upload-gen_only-gb200_gpt-oss-120b-fp4_ 8k1k_con1024_ctx1_tp1_gen1_tp4_eplb0_mtp0_ccb-NIXL]`	`GB200-8_GPUs-2_Nodes-PyTorch-Disagg- PerfSanity-CTX1-NODE1-GPU1-GEN1-NODE1-GPU4- Post-Merge-3`	🟢 Passed in three substantive PR runs.	Unwaived → unwaived	6342844
`perf/test_perf_sanity.py::test_e2e[ disagg_upload-gen_only-gb200_kimi-k25-thinking-fp4_ 1k1k_con4096_ctx1_dep4_gen1_dep8_eplb0_mtp0_ccb-NIXL]`	`GB200-12_GPUs-3_Nodes-PyTorch-Disagg- PerfSanity-CTX1-NODE1-GPU4-GEN1-NODE2-GPU8- Post-Merge-8`	🟢 Historical liveness regression; passed in #45622.	Waived → unwaived	6323074, 6342844
`perf/test_perf_sanity.py::test_e2e[ disagg_upload-gen_only-gb300_kimi-k25-thinking-fp4_ 1k1k_con4_ctx1_dep4_gen1_tep4_eplb0_mtp0_ccb-NIXL]`	`GB300-8_GPUs-2_Nodes-PyTorch-Disagg- PerfSanity-CTX1-NODE1-GPU4-GEN1-NODE1-GPU4- Post-Merge-3`	🟢 Passed in #46022 on the final behavior.	Unwaived → unwaived	6342844
`perf/test_perf_sanity.py::test_e2e[ disagg_upload-gen_only-gb200_gpt-oss-120b-fp4_ 1k1k_con2048_ctx1_tp1_gen1_dep2_eplb0_mtp0_ccb-NIXL]`	`GB200-8_GPUs-2_Nodes-PyTorch-Disagg- PerfSanity-CTX1-NODE1-GPU1-GEN1-NODE1-GPU2- Post-Merge-2`	🟠 KV transfer finishes; #45622 failed a separate 12.39% device-step performance gate. Performance follow-up is outside this PR.	Unwaived → waived (6324123)	6324123, 6342844
`perf/test_perf_sanity.py::test_e2e[ disagg_upload-gen_only-gb300_kimi-k25-thinking-fp4_ 1k1k_con4096_ctx1_dep4_gen1_dep8_eplb0_mtp0_ccb-NIXL]`	`GB300-12_GPUs-3_Nodes-PyTorch-Disagg- PerfSanity-CTX1-NODE1-GPU4-GEN1-NODE2-GPU8- Post-Merge-4`	🟠 KV transfer finishes; #45622 failed a separate 14.98% device-step performance gate. Performance follow-up is outside this PR.	Waived → waived (6323074)	6323074, 6342844
`perf/test_perf_sanity.py::test_e2e[ disagg_upload-gen_only-gb300_kimi-k25-thinking-fp4_ 8k1k_con1024_ctx1_dep4_gen1_dep32_eplb416_mtp3_ccb-NIXL]`	`GB300-36_GPUs-9_Nodes-PyTorch-Disagg- PerfSanity-CTX1-NODE1-GPU4-GEN1-NODE8-GPU32- Post-Merge-2`	🟡 No valid product result. #46495 and #46531 show all 32 generation ranks receiving `SIGBUS` while loading shared Kimi weights, before request traffic or KV transfer.	Waived → unwaived	6323074, 6342844
`accuracy/test_disaggregated_serving.py:: TestDeepSeekV32Exp::test_auto_dtype_with_helix[ fifo-cudagraph:with_padding-pp1tp1cp4]`	`DGX_B200-8_GPUs-PyTorch-3`	⚪ Model-forward/DSA failure outside the captured `CacheSender` wait path.	Waived → waived	6396413

Explicitly not closed by this PR:

perf/test_perf_sanity.py::test_e2e[disagg_upload-gen_only-gb200_gpt-oss-120b-fp4_1k1k_con512_ctx1_tp1_gen1_dep2_eplb0_mtp0_ccb-NIXL] was not qualified.
The H100 variant of disaggregated/test_disaggregated.py::test_disaggregated_stress_test[input8k-output1k-conc512-qwen3_32b_fp8_stress] remains waived under NVBug 6312828.
GPT-OSS con2048 and GB300 Kimi con4096 performance regressions require separate follow-up; this PR establishes liveness only.
Benchmark fill-gate rank skew and unmatched-request UCX/MPI teardown are out of scope.

Validation

#45579: direct C++/NIXL synchronous-transfer stress test passed.
#45622: six perf-sanity cases passed; GPT-OSS con2048 and GB300 Kimi con4096 finished KV transfer and failed only performance gates.
#45816: GB200 DeepSeek con3072 passed after the test-only KV-capacity adjustment.
#46022: GB300 Kimi con4 passed on the final behavior.
#46495 and #46531: GB300 Kimi 8k1k con1024 did not reach the changed code because all generation ranks crashed during model-weight loading.
#46559: full L0 run reported 57,321 passed, one waived AutoDeploy failure, and one unrelated/flaky GB200 DeepSeek timeout also observed across unrelated PR and post-merge runs.
Current rebased/squashed head: all changed-file pre-commit hooks, waiver duplicate validation, AST test-list validation, DCO, git diff --check, and Python syntax compilation pass. A fresh full CI run is pending.

No valid final-behavior run has reproduced the original generation-side KV-transfer stall.

chienchunhung · 2026-06-30T00:12:02Z

/bot run --disable-fail-fast --stage-list "B200_PCIe-PackageSanityCheck-PY312-DLFW,GB200-8_GPUs-2_Nodes-PyTorch-Disagg-PerfSanity-CTX1-NODE1-GPU1-GEN1-NODE1-GPU4-Post-Merge-3"

chienchunhung · 2026-06-30T16:26:23Z

/bot run --disable-fail-fast --stage-list "GB200-12_GPUs-3_Nodes-PyTorch-Disagg-PerfSanity-CTX1-NODE1-GPU4-GEN1-NODE2-GPU8-Post-Merge-2,GB200-24_GPUs-6_Nodes-PyTorch-Disagg-PerfSanity-CTX2-NODE1-GPU4-GEN1-NODE4-GPU16-Post-Merge-1,GB200-24_GPUs-6_Nodes-PyTorch-Disagg-PerfSanity-CTX2-NODE1-GPU4-GEN1-NODE4-GPU16-Post-Merge-2,GB200-8_GPUs-2_Nodes-PyTorch-Disagg-PerfSanity-CTX1-NODE1-GPU4-GEN1-NODE1-GPU4-Post-Merge-3,GB300-8_GPUs-2_Nodes-PyTorch-Disagg-PerfSanity-CTX1-NODE1-GPU4-GEN1-NODE1-GPU4-Post-Merge-2,GB200-8_GPUs-2_Nodes-PyTorch-Disagg-PerfSanity-CTX1-NODE1-GPU1-GEN1-NODE1-GPU2-Post-Merge-2,GB200-8_GPUs-2_Nodes-PyTorch-Disagg-PerfSanity-CTX1-NODE1-GPU1-GEN1-NODE1-GPU4-Post-Merge-3,GB200-12_GPUs-3_Nodes-PyTorch-Disagg-PerfSanity-CTX1-NODE1-GPU4-GEN1-NODE2-GPU8-Post-Merge-8,GB300-12_GPUs-3_Nodes-PyTorch-Disagg-PerfSanity-CTX1-NODE1-GPU4-GEN1-NODE2-GPU8-Post-Merge-4,GB300-8_GPUs-2_Nodes-PyTorch-Disagg-PerfSanity-CTX1-NODE1-GPU4-GEN1-NODE1-GPU4-Post-Merge-3,GB300-36_GPUs-9_Nodes-PyTorch-Disagg-PerfSanity-CTX1-NODE1-GPU4-GEN1-NODE8-GPU32-Post-Merge-2"

tensorrt-cicd · 2026-06-30T16:34:16Z

PR_Github #56660 [ run ] triggered by Bot. Commit: 3bea148 Link to invocation

Combine the current NVIDIA#15238, NVIDIA#15737, NVIDIA#15794, and NVIDIA#15795 snapshots into one parent commit for NVIDIA#15738. The source PRs remain independently reviewable and mergeable. Signed-off-by: Chien-Chun Hung <2679986+chienchunhung@users.noreply.github.com>

tensorrt-cicd · 2026-06-30T19:07:57Z

PR_Github #56660 [ run ] completed with state FAILURE. Commit: 3bea148
/LLM/main/L0_MergeRequest_PR pipeline #45484 (Partly Tested) completed with status: 'FAILURE'

CI Report

⚠️ Action Required:

Please check the failed tests and fix your PR
If you cannot view the failures, ask the CI triggerer to share details
Once fixed, request an NVIDIA team member to trigger CI again

CI Agent Failure Analysis

Link to invocation

Combine the current NVIDIA#15238, NVIDIA#15737, NVIDIA#15794, and NVIDIA#15795 snapshots into one parent commit for NVIDIA#15738. The source PRs remain independently reviewable and mergeable. Signed-off-by: Chien-Chun Hung <2679986+chienchunhung@users.noreply.github.com>

chienchunhung · 2026-06-30T19:44:43Z

/bot run --disable-fail-fast --stage-list "GB200-12_GPUs-3_Nodes-PyTorch-Disagg-PerfSanity-CTX1-NODE1-GPU4-GEN1-NODE2-GPU8-Post-Merge-2,GB200-24_GPUs-6_Nodes-PyTorch-Disagg-PerfSanity-CTX2-NODE1-GPU4-GEN1-NODE4-GPU16-Post-Merge-1,GB200-24_GPUs-6_Nodes-PyTorch-Disagg-PerfSanity-CTX2-NODE1-GPU4-GEN1-NODE4-GPU16-Post-Merge-2,GB200-8_GPUs-2_Nodes-PyTorch-Disagg-PerfSanity-CTX1-NODE1-GPU4-GEN1-NODE1-GPU4-Post-Merge-3,GB300-8_GPUs-2_Nodes-PyTorch-Disagg-PerfSanity-CTX1-NODE1-GPU4-GEN1-NODE1-GPU4-Post-Merge-2,GB200-8_GPUs-2_Nodes-PyTorch-Disagg-PerfSanity-CTX1-NODE1-GPU1-GEN1-NODE1-GPU2-Post-Merge-2,GB200-8_GPUs-2_Nodes-PyTorch-Disagg-PerfSanity-CTX1-NODE1-GPU1-GEN1-NODE1-GPU4-Post-Merge-3,GB200-12_GPUs-3_Nodes-PyTorch-Disagg-PerfSanity-CTX1-NODE1-GPU4-GEN1-NODE2-GPU8-Post-Merge-8,GB300-12_GPUs-3_Nodes-PyTorch-Disagg-PerfSanity-CTX1-NODE1-GPU4-GEN1-NODE2-GPU8-Post-Merge-4,GB300-8_GPUs-2_Nodes-PyTorch-Disagg-PerfSanity-CTX1-NODE1-GPU4-GEN1-NODE1-GPU4-Post-Merge-3,GB300-36_GPUs-9_Nodes-PyTorch-Disagg-PerfSanity-CTX1-NODE1-GPU4-GEN1-NODE8-GPU32-Post-Merge-2"

tensorrt-cicd · 2026-06-30T19:50:57Z

PR_Github #56713 [ run ] triggered by Bot. Commit: a38e7d9 Link to invocation

tensorrt-cicd · 2026-06-30T19:51:08Z

PR_Github #56712 [ run ] triggered by Bot. Commit: a38e7d9 Link to invocation

tensorrt-cicd · 2026-06-30T19:51:11Z

PR_Github #56713 [ run ] completed with state ABORTED. Commit: a38e7d9

Link to invocation

chienchunhung · 2026-06-30T19:52:31Z

/bot kill

tensorrt-cicd · 2026-06-30T19:58:31Z

PR_Github #56717 [ kill ] triggered by Bot. Commit: a38e7d9 Link to invocation

tensorrt-cicd · 2026-06-30T20:01:50Z

PR_Github #56712 [ run ] completed with state ABORTED. Commit: a38e7d9

Link to invocation

tensorrt-cicd · 2026-06-30T20:02:04Z

PR_Github #56717 [ kill ] completed with state SUCCESS. Commit: a38e7d9
Successfully killed previous jobs for commit a38e7d9

Link to invocation

chienchunhung · 2026-06-30T20:02:25Z

/bot run --disable-reuse-test --disable-fail-fast --stage-list "GB200-12_GPUs-3_Nodes-PyTorch-Disagg-PerfSanity-CTX1-NODE1-GPU4-GEN1-NODE2-GPU8-Post-Merge-2,GB200-24_GPUs-6_Nodes-PyTorch-Disagg-PerfSanity-CTX2-NODE1-GPU4-GEN1-NODE4-GPU16-Post-Merge-1,GB200-24_GPUs-6_Nodes-PyTorch-Disagg-PerfSanity-CTX2-NODE1-GPU4-GEN1-NODE4-GPU16-Post-Merge-2,GB200-8_GPUs-2_Nodes-PyTorch-Disagg-PerfSanity-CTX1-NODE1-GPU4-GEN1-NODE1-GPU4-Post-Merge-3,GB300-8_GPUs-2_Nodes-PyTorch-Disagg-PerfSanity-CTX1-NODE1-GPU4-GEN1-NODE1-GPU4-Post-Merge-2,GB200-8_GPUs-2_Nodes-PyTorch-Disagg-PerfSanity-CTX1-NODE1-GPU1-GEN1-NODE1-GPU2-Post-Merge-2,GB200-8_GPUs-2_Nodes-PyTorch-Disagg-PerfSanity-CTX1-NODE1-GPU1-GEN1-NODE1-GPU4-Post-Merge-3,GB200-12_GPUs-3_Nodes-PyTorch-Disagg-PerfSanity-CTX1-NODE1-GPU4-GEN1-NODE2-GPU8-Post-Merge-8,GB300-12_GPUs-3_Nodes-PyTorch-Disagg-PerfSanity-CTX1-NODE1-GPU4-GEN1-NODE2-GPU8-Post-Merge-4,GB300-8_GPUs-2_Nodes-PyTorch-Disagg-PerfSanity-CTX1-NODE1-GPU4-GEN1-NODE1-GPU4-Post-Merge-3,GB300-36_GPUs-9_Nodes-PyTorch-Disagg-PerfSanity-CTX1-NODE1-GPU4-GEN1-NODE8-GPU32-Post-Merge-2"

tensorrt-cicd · 2026-06-30T20:09:04Z

PR_Github #56718 [ run ] triggered by Bot. Commit: a38e7d9 Link to invocation

chienchunhung · 2026-06-30T22:44:33Z

/bot run --disable-reuse-test --disable-fail-fast --stage-list "GB200-12_GPUs-3_Nodes-PyTorch-Disagg-PerfSanity-CTX1-NODE1-GPU4-GEN1-NODE2-GPU8-Post-Merge-2,GB200-24_GPUs-6_Nodes-PyTorch-Disagg-PerfSanity-CTX2-NODE1-GPU4-GEN1-NODE4-GPU16-Post-Merge-1,GB200-24_GPUs-6_Nodes-PyTorch-Disagg-PerfSanity-CTX2-NODE1-GPU4-GEN1-NODE4-GPU16-Post-Merge-2,GB200-8_GPUs-2_Nodes-PyTorch-Disagg-PerfSanity-CTX1-NODE1-GPU4-GEN1-NODE1-GPU4-Post-Merge-3,GB300-8_GPUs-2_Nodes-PyTorch-Disagg-PerfSanity-CTX1-NODE1-GPU4-GEN1-NODE1-GPU4-Post-Merge-2,GB200-8_GPUs-2_Nodes-PyTorch-Disagg-PerfSanity-CTX1-NODE1-GPU1-GEN1-NODE1-GPU2-Post-Merge-2,GB200-8_GPUs-2_Nodes-PyTorch-Disagg-PerfSanity-CTX1-NODE1-GPU1-GEN1-NODE1-GPU4-Post-Merge-3,GB200-12_GPUs-3_Nodes-PyTorch-Disagg-PerfSanity-CTX1-NODE1-GPU4-GEN1-NODE2-GPU8-Post-Merge-8,GB300-12_GPUs-3_Nodes-PyTorch-Disagg-PerfSanity-CTX1-NODE1-GPU4-GEN1-NODE2-GPU8-Post-Merge-4,GB300-8_GPUs-2_Nodes-PyTorch-Disagg-PerfSanity-CTX1-NODE1-GPU4-GEN1-NODE1-GPU4-Post-Merge-3,GB300-36_GPUs-9_Nodes-PyTorch-Disagg-PerfSanity-CTX1-NODE1-GPU4-GEN1-NODE8-GPU32-Post-Merge-2"

tensorrt-cicd · 2026-06-30T22:48:19Z

PR_Github #56718 [ run ] completed with state FAILURE. Commit: a38e7d9
/LLM/main/L0_MergeRequest_PR pipeline #45538 (Partly Tested) completed with status: 'FAILURE'

CI Report

⚠️ Action Required:

Please check the failed tests and fix your PR
If you cannot view the failures, ask the CI triggerer to share details
Once fixed, request an NVIDIA team member to trigger CI again

CI Agent Failure Analysis

Link to invocation

tensorrt-cicd · 2026-06-30T22:50:26Z

PR_Github #56750 [ run ] triggered by Bot. Commit: 0d78feb Link to invocation

Combine the current NVIDIA#15238, NVIDIA#15737, NVIDIA#15794, and NVIDIA#15795 snapshots into one parent commit for NVIDIA#15738. The source PRs remain independently reviewable and mergeable. Signed-off-by: Chien-Chun Hung <2679986+chienchunhung@users.noreply.github.com>

tensorrt-cicd · 2026-07-01T01:26:59Z

PR_Github #56750 [ run ] completed with state FAILURE. Commit: 0d78feb
/LLM/main/L0_MergeRequest_PR pipeline #45568 (Partly Tested) completed with status: 'FAILURE'

CI Report

⚠️ Action Required:

Please check the failed tests and fix your PR
If you cannot view the failures, ask the CI triggerer to share details
Once fixed, request an NVIDIA team member to trigger CI again

CI Agent Failure Analysis

Link to invocation

…tion Raise KV capacity only for the GB200 DeepSeek con3072 test configuration. Unwaive addressed Kimi and Qwen cases while retaining separate GPT-OSS and Kimi performance waivers and all newer upstream waivers. Related: https://nvbugs/6312828, https://nvbugs/6323074, https://nvbugs/6324123 Signed-off-by: Chien-Chun Hung <2679986+chienchunhung@users.noreply.github.com>

Signed-off-by: Chien-Chun Hung <2679986+chienchunhung@users.noreply.github.com>

chienchunhung · 2026-07-09T01:31:50Z

/bot run --disable-fail-fast

tensorrt-cicd · 2026-07-09T01:38:35Z

PR_Github #58338 [ run ] triggered by Bot. Commit: be8dcdc Link to invocation

tensorrt-cicd · 2026-07-09T17:05:19Z

PR_Github #58338 [ run ] completed with state SUCCESS. Commit: be8dcdc
/LLM/main/L0_MergeRequest_PR pipeline #46967 completed with status: 'FAILURE'

CI Report

⚠️ Action Required:

Please check the failed tests and fix your PR
If you cannot view the failures, ask the CI triggerer to share details
Once fixed, request an NVIDIA team member to trigger CI again

CI Agent Failure Analysis

Link to invocation

chienchunhung · 2026-07-09T17:47:12Z

/bot run --disable-fail-fast

tensorrt-cicd · 2026-07-09T17:52:41Z

PR_Github #58503 [ run ] triggered by Bot. Commit: be8dcdc Link to invocation

tensorrt-cicd · 2026-07-09T18:09:18Z

PR_Github #58503 [ run ] completed with state FAILURE. Commit: be8dcdc
/LLM/main/L0_MergeRequest_PR pipeline #47110 completed with status: 'FAILURE'

CI Report

⚠️ Action Required:

Please check the failed tests and fix your PR
If you cannot view the failures, ask the CI triggerer to share details
Once fixed, request an NVIDIA team member to trigger CI again

CI Agent Failure Analysis

Link to invocation

chienchunhung · 2026-07-09T20:00:53Z

/bot run --disable-fail-fast

tensorrt-cicd · 2026-07-09T20:07:13Z

PR_Github #58509 [ run ] triggered by Bot. Commit: be8dcdc Link to invocation

tensorrt-cicd · 2026-07-09T20:50:08Z

PR_Github #58509 [ run ] completed with state ABORTED. Commit: be8dcdc

Link to invocation

chienchunhung · 2026-07-10T00:01:05Z

/bot run --disable-fail-fast

tensorrt-cicd · 2026-07-10T00:15:40Z

PR_Github #58534 [ run ] triggered by Bot. Commit: be8dcdc Link to invocation

tensorrt-cicd · 2026-07-10T02:54:20Z

PR_Github #58534 [ run ] completed with state SUCCESS. Commit: be8dcdc
/LLM/main/L0_MergeRequest_PR pipeline #47138 completed with status: 'SUCCESS'

CI Report

Link to invocation

Tabrizian

py_executor changes LGTM.

github-actions Bot assigned chienchunhung Jun 29, 2026

chienchunhung mentioned this pull request Jun 29, 2026

[TRTLLM-12721][feat] Enable disaggregated in-flight cancellation by default #15738

Draft

chienchunhung changed the title ~~[https://nvbugs/6323889][fix] Prevent cache sender lost wakeups~~ [https://nvbugs/6342844][fix] Prevent cache sender lost wakeups Jun 29, 2026

This was referenced Jun 30, 2026

[TRTLLM-12721][fix] Harden disagg cancellation consensus and buffer ownership #15794

Draft

[TRTLLM-12721][fix] Add gated C++ NIXL in-flight cancellation and safe cleanup #15238

Merged

This was referenced Jun 30, 2026

[TRTLLM-12721][fix] Harden PyExecutor disagg cancellation lifecycle #15795

Draft

[TRTLLM-12721][fix] Negotiate disaggregated cancellation protocol and mode #15798

Draft

[TRTLLM-12721][docs] Design disaggregated peer cancellation protocol #15799

Draft

chienchunhung added 2 commits July 8, 2026 18:29

[https://nvbugs/6342844][fix] Drain synchronous KV transfer batches

be8dcdc

Signed-off-by: Chien-Chun Hung <2679986+chienchunhung@users.noreply.github.com>

chienchunhung force-pushed the codex/disagg-sender-ready-race branch from e62c970 to be8dcdc Compare July 9, 2026 01:31

chuangz0 approved these changes Jul 10, 2026

View reviewed changes

chienchunhung requested a review from Tabrizian July 10, 2026 18:00

Tabrizian approved these changes Jul 10, 2026

View reviewed changes

chienchunhung merged commit b49798f into NVIDIA:main Jul 10, 2026
7 checks passed

chienchunhung mentioned this pull request Aug 3, 2026

[None][fix] Fix gen-only async kvtransfer hang #17107

Merged

4 tasks

nv-xtf mentioned this pull request Aug 5, 2026

[https://nvbugs/6527301][fix] Do not block ranks with scheduled forward work on disagg gen KV-transfer progress #17299

Closed

1 task

Uh oh!

Conversation

chienchunhung commented Jun 29, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Summary

Changes

Test scope and accounting

Validation

Uh oh!

chienchunhung commented Jun 30, 2026

Uh oh!

chienchunhung commented Jun 30, 2026

Uh oh!

tensorrt-cicd commented Jun 30, 2026

Uh oh!

tensorrt-cicd commented Jun 30, 2026

Uh oh!

chienchunhung commented Jun 30, 2026

Uh oh!

tensorrt-cicd commented Jun 30, 2026

Uh oh!

tensorrt-cicd commented Jun 30, 2026

Uh oh!

tensorrt-cicd commented Jun 30, 2026

Uh oh!

chienchunhung commented Jun 30, 2026

Uh oh!

tensorrt-cicd commented Jun 30, 2026

Uh oh!

tensorrt-cicd commented Jun 30, 2026

Uh oh!

tensorrt-cicd commented Jun 30, 2026

Uh oh!

chienchunhung commented Jun 30, 2026

Uh oh!

tensorrt-cicd commented Jun 30, 2026

Uh oh!

chienchunhung commented Jun 30, 2026

Uh oh!

tensorrt-cicd commented Jun 30, 2026

Uh oh!

tensorrt-cicd commented Jun 30, 2026

Uh oh!

tensorrt-cicd commented Jul 1, 2026

Uh oh!

chienchunhung commented Jul 9, 2026

Uh oh!

tensorrt-cicd commented Jul 9, 2026

Uh oh!

tensorrt-cicd commented Jul 9, 2026

Uh oh!

chienchunhung commented Jul 9, 2026

Uh oh!

tensorrt-cicd commented Jul 9, 2026

Uh oh!

tensorrt-cicd commented Jul 9, 2026

Uh oh!

chienchunhung commented Jul 9, 2026

Uh oh!

tensorrt-cicd commented Jul 9, 2026

Uh oh!

tensorrt-cicd commented Jul 9, 2026

Uh oh!

chienchunhung commented Jul 10, 2026

Uh oh!

tensorrt-cicd commented Jul 10, 2026

Uh oh!

tensorrt-cicd commented Jul 10, 2026

Uh oh!

Tabrizian left a comment

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

chienchunhung commented Jun 29, 2026 •

edited

Loading