[https://nvbugs/6480621][fix] Preserve KV ownership in disaggregated precheck

examples/disaggregated/slurm/cache_transceiver_test/README.md

@@ -55,7 +55,9 @@ itself flows over UCX/NIXL.
 - Every rank prints `UCX_TLS` and `UCX_NET_DEVICES`; `UCX_PROTO_INFO=used` is set
   so UCX >= 1.21 captures the selected GPU↔GPU transport in the stderr logs.
 - Failing transfers are reported per cell as `TRANSFER_ERROR` / `MISMATCH` /
-  `TIMEOUT` (the run continues — a single bad case never aborts the sweep).
+  `TIMEOUT`. A verification mismatch is safe to continue; a result that does
+  not prove transfer quiescence hard-aborts only the current UCX sweep so its
+  cache manager cannot recycle or deregister pages still owned by transport.
 
 ## Outputs (under `environment.work_dir`)
 
@@ -112,10 +114,16 @@ within a `(combination)` across UCX sweeps.
   so no cross-node gather is needed.
 - A bad `UCX_NET_DEVICES`/`UCX_TLS` can hang a transfer. Hangs are handled at
   three layers so one stuck sweep never blocks the others:
-  1. **`signal.alarm`** (per cell, `timeout_per_cell_s`) recovers Python-level
-     stalls and continues within the sweep.
-  2. **Watchdog thread** (per cell) records `TIMEOUT` and `SIGKILL`s the process
-     for hangs inside a *GIL-released* native call (e.g.
+  1. **The Python sender deadline plus `signal.alarm`** bounds Python-level
+     stalls. `timeout_per_cell_s` must be greater than five seconds. The Python
+     sender deadline is five seconds lower, leaving time for CTX and GEN to
+     exchange a final ownership decision. The alarm is the per-cell bound for
+     the C++ path and for later requests after the shared cell budget is spent.
+     A timeout or any other result without explicit completion retains the
+     pages and hard-aborts the current sweep; the harness never rebuilds a
+     cache manager over possibly active transfer memory.
+  2. **Watchdog thread** (per cell) best-effort records `TIMEOUT` and `SIGKILL`s
+     the process for hangs inside a *GIL-released* native call (e.g.
      `check_*_transfer_status`); `srun --kill-on-bad-exit` then tears down the
      sweep. This is **best-effort**: it cannot fire if the hang is in a native
      call that *holds* the GIL (e.g. the UCX connection handshake inside

examples/disaggregated/slurm/cache_transceiver_test/config.yaml

@@ -98,9 +98,11 @@ ucx_env_sweep:
 
 
 run:
-  # Per (combination, request_length) cell hang threshold; a stuck transfer is reported as
-  # TIMEOUT and the harness moves on. The per-cell watchdog fires shortly after
-  # this and is always clamped below max_sweep_s.
+  # Per (combination, request_length) cell hard limit; must be >5 seconds. The
+  # Python sender deadline is set 5 seconds lower so it can return and complete
+  # the CTX/GEN ownership handshake first. The alarm remains the C++ path's
+  # per-cell bound. If quiescence is still unproven, the harness records the
+  # remaining cells and hard-aborts this sweep without reusing KV memory.
   timeout_per_cell_s: 60
   # Hard wall-clock cap for ONE full sweep (all cases x request lengths). The
   # outer `timeout` around each srun enforces this, so a hung sweep can never