[None][fix] Guard Python KV receive ownership and publication

[chienchunhung]

Design position

DBR-PR1 — receive-side ownership foundation. This PR adds the GEN-side physical-ownership and publication primitive required by deadline-bounded KV transfer retirement.

It is intentionally dormant on its own: enforce_physical_ownership defaults to False, and this PR adds no public configuration, factory wiring, source ownership, quiescence deadline, endpoint fail-close, or supported production topology.

The immediate follow-up is #18041 (DBR-PR2), which adds narrow CTX source ownership and activates both local owners for one qualified no-retry Python/NIXL FP4-MLA cell. Typed backend evidence, the non-resettable quiescence clock, and rank-aligned fail-close/restart remain DBR-PR3 through DBR-PR5.

Problem

The Python KV receive path can make a request logically terminal while transport or local CUDA work may still access its destination. That creates these deterministic hazards:

1. One failed writer can make aggregate state terminal while a sibling still accesses the same allocation.
2. Cancellation can win before receive publication, yet the task can still be created and published afterward.
3. Cancellation can overtake an authorized REQUEST_DATA publication, so the sender observes cancellation before the delayed request.
4. A repeated cancel-before-dispatch decision can lose the already-proven unpublished state and suppress release of a late idle reservation.
5. Holding the session-state lock across a blocking REQUEST_DATA send can stall terminal-result processing.
6. Cross-rank failure consensus can retire a drained rank before its siblings drain, preventing the retained ranks from reaching a later consensus.
7. Shutdown can clear a live receive owner and tear down registered transfer resources before physical access drains.

Safety boundary implemented here

For an ownership-enabled receive session, destination reuse is allowed only when:

• cancellation wins before any writer is authorized; or
• the sealed writer cohort has settled and local receive/scatter completion has finished.

Logical failure or cancellation does not erase a live writer claim. Invalid or incomplete evidence does not authorize release. This is a local receive-side invariant; it does not yet provide sender-side backend-DONE retention or deadline-bounded endpoint recovery.

What changes

• Add a receive-operation owner that tracks publication, the sealed writer cohort, per-writer terminal evidence, and local completion independently from logical task status.
• Seal publication under the RxSession state lock, but perform network sends under a separate protocol-order lock so result handling remains live and cancellation cannot overtake publication.
• Keep publication pending until the complete REQUEST_DATA fan-out returns successfully.
• Require and validate an explicit authorized writer cohort.
• Retain ownership-enabled receive sessions strongly until resources_drained() is true.
• Gate local and cross-rank failure reporting, cancellation, request removal, and session close on the shared drain predicate.
• Preserve session ownership when a defensive close refuses teardown.
• Refuse transceiver shutdown while an ownership-enabled receive session still has unproven physical access.
• Make cancel-before-dispatch cleanup idempotent so a late unpublished reservation is released exactly once.
• Preserve the legacy path when ownership enforcement is disabled.

flowchart LR
    U["Destination unpublished"] --> G{"Cancellation / publication gate"}
    G -- "cancel wins" --> S["SAFE_UNPUBLISHED<br/>no writer authorized"]
    S --> R["Release reservation"]
    G -- "publication wins" --> C["Seal writer cohort"]
    C --> P["Publish REQUEST_DATA"]
    P --> A["Destination may be accessed"]
    A --> T{"All writers settled<br/>and local work complete?"}
    T -- "no" --> H["Retain session and allocation"]
    T -- "yes" --> D["Receive owner drained<br/>reuse may proceed"]

Loading

Regression-first history

• Commit 1, c66afdab1, contains the three deterministic motivating regressions only.
• Commit 2, 12d40c2a9, adds the dormant receive-side fix and three compatibility regressions.
• Commit 3, 7e247c32f, adds and fixes the cancel-before-dispatch idempotency regression.
• Commit 4, 7a378d62f, hardens publication ordering, cohort authorization, close refusal, and cross-rank drain consensus, bringing the focused CPU suite to ten tests.
• Commit 5, c3ab2cc98, prevents shutdown from dropping an active receive owner, bringing the focused CPU suite to eleven tests.

Historical RED evidence: CPU-Generic-x86-1 PR_Github #66895 / pipeline #54452 reproduced the three original hazards on a test-only predecessor snapshot. It is motivation evidence, not patch-equivalent evidence for commit 1.

Predecessor-head GREEN evidence: CPU-Generic-x86-1 PR_Github #67564 / pipeline #55055 passed at d371644d8 with the first six regressions and compatibility cases. Current head c3ab2cc98 has eleven focused tests; full current-head CI is pending.

Change size

• Production: +498/-125 across transfer.py and transceiver.py.
• Tests: +592/-0 in one CPU regression file.
• Total: +1,090/-125 across three files.

Deferred

This PR does not add CTX source or exact backend-handle ownership, production activation, timeout/quiescence clocks, late-settlement typing, rank-aligned fail-close/restart, retry or reroute identity, broad topology/backend qualification, in-flight cancellation, or C++ parity.

#18041 is the narrow activation bridge. The remaining deadline-bounded retirement and generalization work stays in DBR-PR3 through DBR-PR8.

Dev Engineer Review

• Added dormant physical receive-ownership tracking.
• Added writer-cohort validation and terminal writer-result checks.
• Serialized publication and cancellation.
• Retained ownership-enforced sessions until resources_drained() succeeds.
• Prevented shutdown from dropping active receive ownership.
• Preserved legacy behavior because enforcement defaults to disabled.
• Updated RxSession.mark_transferring() with an optional writer_cohort argument.
• No configuration or test-list changes were identified.
• Review should confirm Python style, API consistency, error handling, and current-head CI results.

QA Engineer Review

Added these test functions:

• test_failed_writer_cannot_authorize_reuse_while_sibling_is_active()
• test_pre_cancelled_rx_session_never_publishes_destination()
• test_remote_cancel_resolves_strong_owned_session()
• test_remote_cancelled_session_is_retained_until_writers_drain()
• test_failed_receive_session_is_retained_until_writers_drain()
• test_failed_receive_consensus_waits_for_every_rank_to_drain()
• test_non_terminal_writer_result_does_not_authorize_reuse()
• test_partial_publication_failure_quarantines_destination()
• test_out_of_cohort_writer_cannot_authorize_reuse()
• test_cancel_after_publication_cannot_overtake_request_data()
• test_cancel_before_dispatch_releases_late_idle_reservation()
• test_cancel_request_retains_session_when_close_refuses()
• test_collect_done_waits_for_physical_drain()
• test_shutdown_refuses_to_drop_active_receive_owner()

No files under tests/integration/test_lists/ were modified. These tests are not covered by entries in test-db/ or qa/.

Verdict: insufficient.

[chienchunhung]

/bot run --disable-fail-fast --stage-list "A10-PyTorch-1, A10-PyTorch-2, A10-PyTorch-3"

[tensorrt-cicd]

PR_Github #66363 [ run ] triggered by Bot. Commit: 8abd3b1 Link to invocation

[chienchunhung]

/bot run --stage-list "CPU-Generic-x86-1"

[tensorrt-cicd]

PR_Github #66377 [ run ] triggered by Bot. Commit: 905fb53 Link to invocation

[tensorrt-cicd]

PR_Github #66363 [ run ] completed with state ABORTED. Commit: 8abd3b1

Link to invocation

[chienchunhung]

/bot run --stage-list "CPU-Generic-x86-1"

[tensorrt-cicd]

PR_Github #66385 [ run ] triggered by Bot. Commit: 0304618 Link to invocation

[tensorrt-cicd]

PR_Github #66377 [ run ] completed with state ABORTED. Commit: 905fb53

Link to invocation

[tensorrt-cicd]

PR_Github #66385 [ run ] completed with state SUCCESS. Commit: 0304618
/LLM/main/L0_MergeRequest_PR pipeline #54028 (Partly Tested) completed with status: 'FAILURE'

CI Report

⚠️ Action Required:

• Please check the failed tests and fix your PR
• If you cannot view the failures, ask the CI triggerer to share details
• Once fixed, request an NVIDIA team member to trigger CI again

CI Agent Failure Analysis

Link to invocation

[chienchunhung]

/bot run --stage-list "CPU-Generic-x86-1"

[chienchunhung]

/bot run --stage-list "CPU-Generic-x86-1"

[chienchunhung]

/bot run --stage-list "CPU-Generic-x86-1"

[tensorrt-cicd]

PR_Github #66436 [ run ] triggered by Bot. Commit: 09c9350 Link to invocation

[tensorrt-cicd]

PR_Github #66436 [ run ] completed with state FAILURE. Commit: 09c9350
/LLM/main/L0_MergeRequest_PR pipeline #54076 (Partly Tested) completed with status: 'FAILURE'

CI Report

⚠️ Action Required:

• Please check the failed tests and fix your PR
• If you cannot view the failures, ask the CI triggerer to share details
• Once fixed, request an NVIDIA team member to trigger CI again

CI Agent Failure Analysis

Link to invocation

[chienchunhung]

/bot run --stage-list "CPU-Generic-x86-1"

[tensorrt-cicd]

PR_Github #66556 [ run ] triggered by Bot. Commit: 0a1aaf7 Link to invocation

[tensorrt-cicd]

PR_Github #66556 [ run ] completed with state FAILURE. Commit: 0a1aaf7
/LLM/main/L0_MergeRequest_PR pipeline #54187 (Partly Tested) completed with status: 'FAILURE'

CI Report

⚠️ Action Required:

• Please check the failed tests and fix your PR
• If you cannot view the failures, ask the CI triggerer to share details
• Once fixed, request an NVIDIA team member to trigger CI again

CI Agent Failure Analysis

Link to invocation

[chienchunhung]

/bot run --disable-fail-fast

[tensorrt-cicd]

PR_Github #66849 [ run ] triggered by Bot. Commit: 4c37734 Link to invocation

[chienchunhung]

/bot run --stage-list "CPU-Generic-x86-1"

[tensorrt-cicd]

PR_Github #66895 [ run ] triggered by Bot. Commit: 3c37cb4 Link to invocation

[tensorrt-cicd]

PR_Github #66849 [ run ] completed with state ABORTED. Commit: 4c37734

Link to invocation

[tensorrt-cicd]

PR_Github #66895 [ run ] completed with state FAILURE. Commit: 3c37cb4
/LLM/main/L0_MergeRequest_PR pipeline #54452 (Partly Tested) completed with status: 'FAILURE'

CI Report

⚠️ Action Required:

• Please check the failed tests and fix your PR
• If you cannot view the failures, ask the CI triggerer to share details
• Once fixed, request an NVIDIA team member to trigger CI again

CI Agent Failure Analysis

Link to invocation

[chienchunhung]

/bot run --stage-list "CPU-Generic-x86-1"

[tensorrt-cicd]

PR_Github #70797 [ run ] completed with state SUCCESS. Commit: c79720a
/LLM/main/L0_MergeRequest_PR pipeline #57981 completed with status: 'FAILURE'

CI Report

⚠️ Action Required:

• Please check the failed tests and fix your PR
• If you cannot view the failures, ask the CI triggerer to share details
• Once fixed, request an NVIDIA team member to trigger CI again

CI Agent Failure Analysis

Link to invocation

[chienchunhung]

/bot run --disable-fail-fast

[tensorrt-cicd]

PR_Github #71025 [ run ] triggered by Bot. Commit: c79720a Link to invocation

[tensorrt-cicd]

PR_Github #71025 [ run ] completed with state SUCCESS. Commit: c79720a
/LLM/main/L0_MergeRequest_PR pipeline #58180 completed with status: 'UNSTABLE'

CI Report

⚠️ Multi-GPU Label Required:
Multi-GPU tests require the ci: full pre-merge approved label on this PR. Ask a member of NVIDIA/trt-llm-ci-approvers to add the label, then re-trigger CI with the same bot command (no rebase needed).

⚠️ Action Required:

• Please check the failed tests and fix your PR
• If you cannot view the failures, ask the CI triggerer to share details
• Once fixed, request an NVIDIA team member to trigger CI again

Link to invocation

[chienchunhung]

/bot run --disable-fail-fast

[tensorrt-cicd]

PR_Github #71051 [ run ] triggered by Bot. Commit: 6242f98 Link to invocation

[chienchunhung]

/bot run --disable-fail-fast

[tensorrt-cicd]

PR_Github #71067 [ run ] triggered by Bot. Commit: 6242f98 Link to invocation

[tensorrt-cicd]

PR_Github #71051 [ run ] completed with state ABORTED. Commit: 6242f98

Link to invocation

[chienchunhung]

/bot run --disable-fail-fast

[tensorrt-cicd]

PR_Github #71072 [ run ] triggered by Bot. Commit: bff832c Link to invocation

[tensorrt-cicd]

PR_Github #71067 [ run ] completed with state ABORTED. Commit: 6242f98

Link to invocation

[tensorrt-cicd]

PR_Github #71072 [ run ] completed with state FAILURE. Commit: bff832c
/LLM/main/L0_MergeRequest_PR pipeline #58224 completed with status: 'FAILURE'

CI Report

⚠️ Action Required:

• Please check the failed tests and fix your PR
• If you cannot view the failures, ask the CI triggerer to share details
• Once fixed, request an NVIDIA team member to trigger CI again

CI Agent Failure Analysis

Link to invocation

[chienchunhung]

/bot run --disable-fail-fast

[tensorrt-cicd]

PR_Github #71141 [ run ] triggered by Bot. Commit: bff832c Link to invocation

[tensorrt-cicd]

PR_Github #71141 [ run ] completed with state FAILURE. Commit: bff832c
/LLM/main/L0_MergeRequest_PR pipeline #58282 completed with status: 'UNSTABLE'

CI Report

⚠️ Action Required:

• Please check the failed tests and fix your PR
• If you cannot view the failures, ask the CI triggerer to share details
• Once fixed, request an NVIDIA team member to trigger CI again

Link to invocation