[TRTLLM-12721][fix] Harden disagg cancellation consensus and buffer ownership

[chienchunhung]

Summary

Harden the remaining local protocol and fail-closed safety guarantees for default-off disaggregated in-flight KV-transfer cancellation.

#15238 and #15737 are merged and inherited from main; this PR does not duplicate their sender-liveness, synchronous-generation, queued-cancellation, transfer-session ownership, or deferred-cleanup work. It adds the rank-consistent activation, distributed terminal-state accounting, and teardown safeguards still needed by the downstream lifecycle and peer-negotiation PRs.

Cancellation remains disabled by default. Explicit opt-in is limited to asynchronous C++ NIXL/UCX transfer with a positive timeout, overlap enabled, PP=1, CP=1, attention DP disabled, layerwise transfer disabled, and zero-copy disabled. Dynamic and preallocated transfer buffers remain supported.

Why

The merged implementation can cancel and retain transfer resources safely, but several local protocol gaps remain:

• different worker ranks can otherwise derive different effective modes or capabilities;
• a failure observed on one rank must fail the request everywhere, but must not be mistaken for proof that every rank has released local ownership;
• a poisoned transfer pool must remain sticky and topology-wide; and
• transfer-handle query/release, first-contact metadata, and plugin teardown require deterministic lifetime ordering.

Without these guarantees, an opt-in deployment can diverge across ranks, retire bookkeeping prematurely, reuse an unproven buffer, or race teardown.

Changes

Rank-consistent activation

• Resolve legacy backend selection before validating opt-in.
• Make every worker rank participate in startup preflight, including ranks where cancellation is disabled.
• Reject asymmetric requested modes or unsupported capabilities before constructing the native transceiver.
• Pass one explicit effective mode through Nanobind, CacheTransceiver, CacheTransferLayer, DataContext, formatters, and sender/receiver objects instead of rereading the process environment throughout the transfer path.
• Repeat mode and capability consensus in the native parallel group.
• Preserve existing constructor behavior by having legacy overloads select disabled mode.

Distributed transfer-state accounting

• Exchange packed per-request completion, failure, timeout, and global-poison state across ranks.
• Apply failure and timeout as a union: one rank fails the logical request everywhere.
• Retire request bookkeeping only after every rank reports a local terminal state.
• Request cancellation only after the distributed outcome is known, and retry cancellation while ownership remains active.
• Keep transfer-buffer poison sticky once any participating rank reports unsafe reuse.
• Reject unbounded wait-all polling in active mode so cancellation progress remains finite.

Lifetime and teardown hardening

• Fail closed when an active TransferSession is destroyed while it still owns reserved receive slots.
• Serialize NIXL transfer-handle status queries with explicit release only for active cancellation requests, preserving the default-off hot path.
• Retry agent metadata when a cancelled or failed first notification never reached the peer.
• Destroy sender, receiver, and connection-manager state before unloading the transport plugin, including constructor-failure paths.

Coverage

Add focused coverage for:

• failure-union versus all-ranks-terminal semantics;
• timeout winning over a late success;
• malformed state payloads, large request IDs, and topology poison;
• asymmetric rank activation and per-rank capability errors;
• dynamic-buffer qualification and explicit native-mode forwarding;
• active/default-off TransferSession cleanup;
• metadata retry and idempotent NIXL handle release; and
• active and default-off queued-context cancellation behavior.

Known limits

• This PR coordinates ranks within one executor service; it does not negotiate protocol version or effective mode between CTX and GEN. That remains [TRTLLM-12721][fix] Negotiate disaggregated cancellation protocol and mode #15798.
• Process-fatal poison propagation through PyExecutor, IPC, proxy, and worker shutdown remains [TRTLLM-12721][fix] Harden PyExecutor disagg cancellation lifecycle #15795.
• [TRTLLM-12721][docs] Design disaggregated peer cancellation protocol #15799 is a design document. Runtime peer-terminal evidence, or an explicitly approved bounded fail/restart policy with equivalent safety, is still required before default-on rollout.
• Active mode does not support synchronous generation transfer, layerwise transfer, zero-copy transfer, PP, CP, or attention DP.
• Mixed-version deployments must keep cancellation disabled until the CTX↔GEN negotiation is available.

Validation

• Reconstructed as one DCO-signed commit, e8fba959e1, directly on current main 58d8964d13, which contains merged [TRTLLM-12721][fix] Add gated C++ NIXL in-flight cancellation and safe cleanup #15238 and [https://nvbugs/6342844][fix] Prevent disaggregated KV transfer stalls #15737.
• The obsolete cumulative branch changed 43 files; the residual reconstruction changes 24 files after removing merged and downstream scope.
• Changed-file pre-commit, formatting, lint, DCO, git diff --check, and Python syntax checks pass locally.
• The two repository test-list hooks cannot start on the local Python 3.9 host because their script requires Python 3.10+; this PR does not modify test lists.
• Native C++/CUDA execution, TP=2 distributed NIXL behavior, late-deadline CTX/GEN integration, and exact-head full CI remain required before merge.

Dependency graph

Arrows point from prerequisite to dependent. #15795 and #15798 do not block this default-off PR.

flowchart LR
    MAIN["main<br/>#15238 and #15737 merged"]
    PR15794["#15794<br/>local protocol and fail-closed safety<br/>(default off)"]
    PR15795["#15795<br/>PyExecutor and IPC lifecycle"]
    PR15798["#15798<br/>CTX↔GEN negotiation"]
    PR15799["#15799<br/>peer-protocol design only"]
    SAFETY["runtime peer-terminal implementation<br/>or approved bounded fail/restart policy"]
    PR15738["#15738<br/>qualified default-on policy"]

    MAIN --> PR15794
    PR15794 --> PR15795
    PR15794 --> PR15798
    PR15798 --> PR15799
    PR15799 -.->|design input| SAFETY
    PR15795 --> PR15738
    PR15798 --> PR15738
    SAFETY --> PR15738

Loading

[chienchunhung]

/bot run --disable-fail-fast

[tensorrt-cicd]

PR_Github #56693 [ run ] triggered by Bot. Commit: 8320739 Link to invocation

[chienchunhung]

/bot run --disable-fail-fast

[tensorrt-cicd]

PR_Github #56700 [ run ] triggered by Bot. Commit: d890f64 Link to invocation

[tensorrt-cicd]

PR_Github #56693 [ run ] completed with state ABORTED. Commit: 8320739

Link to invocation

[chienchunhung]

/bot run --disable-fail-fast

[tensorrt-cicd]

PR_Github #56738 [ run ] triggered by Bot. Commit: edf9705 Link to invocation

[tensorrt-cicd]

PR_Github #56700 [ run ] completed with state ABORTED. Commit: d890f64

Link to invocation

[tensorrt-cicd]

PR_Github #56738 [ run ] completed with state FAILURE. Commit: edf9705
/LLM/main/L0_MergeRequest_PR pipeline #45557 completed with status: 'FAILURE'

CI Report

⚠️ Action Required:

• Please check the failed tests and fix your PR
• If you cannot view the failures, ask the CI triggerer to share details
• Once fixed, request an NVIDIA team member to trigger CI again

CI Agent Failure Analysis

Link to invocation

[chienchunhung]

/bot run --disable-fail-fast

[tensorrt-cicd]

PR_Github #56760 [ run ] triggered by Bot. Commit: 7db2070 Link to invocation

[tensorrt-cicd]

PR_Github #56760 [ run ] completed with state SUCCESS. Commit: 7db2070
/LLM/main/L0_MergeRequest_PR pipeline #45579 completed with status: 'FAILURE'

CI Report

⚠️ Action Required:

• Please check the failed tests and fix your PR
• If you cannot view the failures, ask the CI triggerer to share details
• Once fixed, request an NVIDIA team member to trigger CI again

CI Agent Failure Analysis

Link to invocation

[chienchunhung]

/bot run --disable-fail-fast --stage-list "A10-CPP-1,H100_PCIe-CPP-1,DGX_H100-4_GPUs-CPP-1,A100X-PyTorch-1"

[tensorrt-cicd]

PR_Github #57041 [ run ] triggered by Bot. Commit: 891251a Link to invocation

[tensorrt-cicd]

PR_Github #57041 [ run ] completed with state FAILURE. Commit: 891251a
/LLM/main/L0_MergeRequest_PR pipeline #45836 (Partly Tested) completed with status: 'FAILURE'

CI Report

⚠️ Action Required:

• Please check the failed tests and fix your PR
• If you cannot view the failures, ask the CI triggerer to share details
• Once fixed, request an NVIDIA team member to trigger CI again

CI Agent Failure Analysis

Link to invocation

[chienchunhung]

/bot run --disable-fail-fast --stage-list "A10-CPP-1,H100_PCIe-CPP-1,DGX_H100-4_GPUs-CPP-1"

[tensorrt-cicd]

PR_Github #57120 [ run ] triggered by Bot. Commit: 81dc282 Link to invocation

[tensorrt-cicd]

PR_Github #57120 [ run ] completed with state SUCCESS. Commit: 81dc282
/LLM/main/L0_MergeRequest_PR pipeline #45907 (Partly Tested) completed with status: 'FAILURE'

CI Report

⚠️ Action Required:

• Please check the failed tests and fix your PR
• If you cannot view the failures, ask the CI triggerer to share details
• Once fixed, request an NVIDIA team member to trigger CI again

CI Agent Failure Analysis

Link to invocation

[chienchunhung]

/bot run --disable-fail-fast

[tensorrt-cicd]

PR_Github #57239 [ run ] triggered by Bot. Commit: 1864ea1 Link to invocation

[tensorrt-cicd]

PR_Github #57239 [ run ] completed with state FAILURE. Commit: 1864ea1
/LLM/main/L0_MergeRequest_PR pipeline #46006 completed with status: 'FAILURE'

CI Report

⚠️ Action Required:

• Please check the failed tests and fix your PR
• If you cannot view the failures, ask the CI triggerer to share details
• Once fixed, request an NVIDIA team member to trigger CI again

CI Agent Failure Analysis

Link to invocation

[chienchunhung]

/bot run --disable-fail-fast

[tensorrt-cicd]

PR_Github #59292 [ run ] triggered by Bot. Commit: e8fba95 Link to invocation

[tensorrt-cicd]

PR_Github #59292 [ run ] completed with state SUCCESS. Commit: e8fba95
/LLM/main/L0_MergeRequest_PR pipeline #47779 completed with status: 'FAILURE'

CI Report

⚠️ Action Required:

• Please check the failed tests and fix your PR
• If you cannot view the failures, ask the CI triggerer to share details
• Once fixed, request an NVIDIA team member to trigger CI again

CI Agent Failure Analysis

Link to invocation