[https://nvbugs/6448152][perf] TEST ONLY; DO NOT REVIEW: publish PP transfer status from workers

[chienchunhung]

Summary

This PR removes the cross-rank context-transfer consensus introduced by #15139 from the scheduler hot path for the affected default-off cancellation configuration:

• The C++ cache-sender worker publishes one immutable terminal vote (completed or failed) to every context pipeline-parallel peer before making its local future ready.
• Each scheduler only drains its local MPI mailbox; it no longer needs matching scheduler participation from peer PP ranks to make progress.
• A request is committed only after every PP rank has published a terminal vote. Failure wins, and request/KV ownership is retained until that global decision.
• All configurations share one context-transfer status engine for future polling, timeout/cancellation handling, request retention, terminal-state updates, and cleanup. Only readiness reduction and consensus transport vary: the qualified path uses the worker mailbox; other configurations use the existing hierarchical collective backend; PP1 reduces locally.

Generation-transfer consensus and active in-flight cancellation behavior are unchanged.

Why

NVBUG#6448152 tracks an output-token-throughput regression in the GB300 disaggregated DeepSeek-R1 workload after #15139 added required cross-rank transfer-status consensus (1573 to 817 output tok/s in the reported run). The two synchronous hierarchical status collectives can serialize scheduler/model progress across context PP ranks.

The previous draft implementation pipelined an MPI_Iallgather, but the exact workload remained near the regressed result: a later scheduler epoch still needed matching peer participation, and a rank already inside model forward could delay the next snapshot. This reconstruction removes peer scheduler participation from terminal-vote publication while preserving the consistency requirement.

Qualified activation scope

The worker-published path is selected only when every context PP rank reports protocol v1 and all of these conditions hold:

• PyExecutor C++ cache transceiver
• MPI control plane with MPI_THREAD_MULTIPLE
• NIXL cache transport with the UCX NIXL backend
• context topology TP1, CP1, PP > 1
• attention DP disabled
• KV-transfer overlap enabled
• layer-wise transfer disabled
• in-flight cancellation disabled

Direct UCX, NIXL-libfabric, MPI/Mooncake transports, ProcessGroup/Python transceivers, TP/CP/attention-DP topologies, overlap-off/layer-wise modes, and active cancellation use the same status engine with the existing hierarchical collective consensus backend. The one-time startup allgather is same-binary configuration agreement, not rolling mixed-version negotiation; all PP workers in one deployment must use the same build.

Safety and lifecycle invariants

• The worker callback publishes exactly once and before the future becomes ready.
• Votes are immutable; all ranks must be terminal before a request is visible as complete or failed.
• A failure on any PP rank wins only after all ranks are terminal.
• No cancellation primitive is called by this optimized path.
• Ordered close markers drain earlier votes during normal teardown. Shutdown is bounded and aborts after 30 seconds on an asymmetric failure rather than freeing active MPI requests or hanging forever.

Testing

Local checks:

• Changed-file pre-commit formatting/lint checks: isort, YAPF, clang-format, cmake-format, codespell, autoflake, legacy lint, whitespace and file checks
• scripts/check_test_list.py --check-duplicate-waives and --validate with Python 3.13 (2,182 entries validated)
• Python bytecode compilation for tests/integration/defs/disaggregated/test_disaggregated.py
• git diff --check

Added coverage:

• Pure reducer, activation-gate, duplicate/conflicting vote, and version-agreement tests
• MPI async-send polling test
• Multi-rank pressure test proving a worker vote from a scheduler-stalled PP rank does not block other ranks, plus failure, independent-request, incomplete-vote, and skewed-teardown cases
• CacheSender callback ordering/exactly-once test
• Real ctxpp4_gentp4 disaggregated integration assertion that the qualified C++ NIXL/UCX path activates
• C++-transceiver PP1 lifecycle coverage for mark_complete, terminal cleanup, and no duplicate result on the next poll

A local CUDA/MPI build was not available. C++ compilation and multi-GPU execution must pass in CI.

Performance readiness gate

Functional CI is necessary but not sufficient. Before marking this PR ready, run the exact NVBUG workload as a same-container/main A/B for at least 2-3 iterations:

• Stage: GB300-12_GPUs-3_Nodes-PyTorch-Disagg-PerfSanity-CTX1-NODE1-GPU4-GEN1-NODE2-GPU8-Post-Merge-1
• Test: disagg_upload-e2e-gb300_deepseek-r1-fp4_128k8k_con256_ctx1_pp4_gen1_dep8_eplb0_mtp1_ccb-NIXL
• Require the worker-published activation log, no hangs/errors, and output-token throughput within 5% of the 1573 good baseline (at least about 1494 tok/s), with no material TTFT/tail-latency regression.

The protocol sends P * (P - 1) tiny vote messages per request. PP4 is the qualified NVBUG scope; PP8/high-request-rate stress is required before making a broader performance claim.

Related PRs

#16386 is a semantic sibling of #15794, not its parent or child. #16386 is qualified only for PP > 1 with cancellation off; #15794 owns active-cancellation safety. #15798 negotiates cancellation mode between CTX and GEN and is not part of this intra-CTX PP protocol. The changes overlap textually in cacheTransceiver.cpp, so whichever lands second must rebase carefully.

flowchart LR
    MAIN["main<br/>#15238 + #15737 merged"]
    P16386["#16386<br/>default-off PP consensus performance"]
    P15794["#15794<br/>active-cancellation protocol safety"]
    P15795["#15795<br/>PyExecutor cancellation lifecycle"]
    P15798["#15798<br/>CTX/GEN cancellation negotiation"]
    P15738["#15738<br/>qualified default-on cancellation"]

    MAIN --> P16386
    MAIN --> P15794
    P15794 --> P15795
    P15794 --> P15798
    P15795 --> P15738
    P15798 --> P15738

Loading

[chienchunhung]

/bot run --disable-fail-fast --stage-list "A30-CPP-1, DGX_H100-4_GPUs-CPP-1, DGX_H100-4_GPUs-PyTorch-Others-1, DGX_H100-4_GPUs-PyTorch-Others-2, GB300-12_GPUs-3_Nodes-PyTorch-Disagg-PerfSanity-CTX1-NODE1-GPU4-GEN1-NODE2-GPU8-Post-Merge-1"

[tensorrt-cicd]

PR_Github #59272 [ run ] triggered by Bot. Commit: 222b355 Link to invocation

[tensorrt-cicd]

PR_Github #59272 [ run ] completed with state FAILURE. Commit: 222b355
/LLM/main/L0_MergeRequest_PR pipeline #47760 (Partly Tested) completed with status: 'FAILURE'

CI Report

⚠️ Action Required:

• Please check the failed tests and fix your PR
• If you cannot view the failures, ask the CI triggerer to share details
• Once fixed, request an NVIDIA team member to trigger CI again

CI Agent Failure Analysis

Link to invocation

[chienchunhung]

/bot run --disable-fail-fast --stage-list "A30-CPP-1, DGX_H100-4_GPUs-CPP-1, DGX_H100-4_GPUs-PyTorch-Others-1, DGX_H100-4_GPUs-PyTorch-Others-2, GB300-12_GPUs-3_Nodes-PyTorch-Disagg-PerfSanity-CTX1-NODE1-GPU4-GEN1-NODE2-GPU8-Post-Merge-1"

[tensorrt-cicd]

PR_Github #59307 [ run ] triggered by Bot. Commit: 2594707 Link to invocation

[chienchunhung]

/bot run --disable-fail-fast --stage-list "A30-CPP-1, DGX_H100-4_GPUs-CPP-1, DGX_H100-4_GPUs-PyTorch-Others-1, DGX_H100-4_GPUs-PyTorch-Others-2, GB300-12_GPUs-3_Nodes-PyTorch-Disagg-PerfSanity-CTX1-NODE1-GPU4-GEN1-NODE2-GPU8-Post-Merge-1"

[tensorrt-cicd]

PR_Github #59341 [ run ] triggered by Bot. Commit: 30c27a6 Link to invocation

[tensorrt-cicd]

PR_Github #59307 [ run ] completed with state ABORTED. Commit: 2594707

Link to invocation

[tensorrt-cicd]

PR_Github #59341 [ run ] completed with state FAILURE. Commit: 30c27a6
/LLM/main/L0_MergeRequest_PR pipeline #47821 (Partly Tested) completed with status: 'FAILURE'

CI Report

⚠️ Action Required:

• Please check the failed tests and fix your PR
• If you cannot view the failures, ask the CI triggerer to share details
• Once fixed, request an NVIDIA team member to trigger CI again

CI Agent Failure Analysis

Link to invocation