diff --git a/examples/kimi_k3/disagg/README.md b/examples/kimi_k3/disagg/README.md index 8597bd8fe643..f07a6d0d870c 100644 --- a/examples/kimi_k3/disagg/README.md +++ b/examples/kimi_k3/disagg/README.md @@ -53,14 +53,12 @@ for constraints. ## Launch sequence (manual, single ctx + single gen) -Each K3 worker spans 16 GPUs (4 NVL72 nodes at 4 GPUs/node). Environment -prerequisites for every worker shell (see caveats below for why): - -```bash -export UCX_TLS=tcp,self,sm,cuda_copy,cuda_ipc # on clusters where verbs cannot - # initialize; a container-default - # UCX_TLS=tcp breaks V2 NIXL -``` +Each K3 worker spans 16 GPUs (4 NVL72 nodes at 4 GPUs/node). Leave +`UCX_TLS` unset in every worker shell so UCX selects transports itself, +RDMA/verbs included; in particular, unset any container-default +`UCX_TLS=tcp`, which breaks V2 NIXL VRAM registration. Override the +transport list only on clusters whose verbs transports cannot +initialize (see caveats 1 and 4 below for the symptom and the override). 1. Start the context server (16-rank MPI world across its 4 nodes): @@ -103,9 +101,11 @@ python3 examples/disaggregated/slurm/benchmark/submit.py \ - **Gen-only baseline**: set `benchmark.mode: gen_only_no_context` (submit.py exports `TRTLLM_DISAGG_BENCHMARK_GEN_ONLY=1` to the workers) to measure the decode-side ceiling without KV transfer. -- The harness's `start_worker.sh` clears `UCX_TLS`; the config carries - the transport pin via `TRTLLM_WORKER_UCX_TLS`, which `start_worker.sh` - re-exports as `UCX_TLS` after the clear. +- The harness's `start_worker.sh` clears any container-provided + `UCX_TLS`, so workers run with UCX's own transport selection by + default. Clusters that need a transport override (caveat 4) carry it + via `TRTLLM_WORKER_UCX_TLS` in `worker_env_var`, which + `start_worker.sh` re-exports as `UCX_TLS` after the clear. - pyxis/enroot resets image-defined variables (notably `PATH`) at container start, so the config injects the in-place TRT-LLM venv via `TRTLLM_PATH_PREPEND` / `TRTLLM_PYTHONPATH_PREPEND`, applied inside @@ -131,11 +131,14 @@ python3 examples/disaggregated/slurm/benchmark/submit.py \ 3. **Matched-DP only.** Keep ctx and gen at identical DEP16 with attention-DP on both sides; heterogeneous parallelism with attention-DP off is rejected (see constraints above). -4. **Cluster environment** (NVL72 nodes): on clusters where verbs - transports cannot initialize, pin - `UCX_TLS=tcp,self,sm,cuda_copy,cuda_ipc` (`UCX_TLS=all` hangs setup, - see caveat 1) and never run V2 NIXL with a container-default - `UCX_TLS=tcp` (breaks V2 NIXL VRAM registration) — unset/override it. +4. **Cluster environment** (NVL72 nodes): the default is to leave + `UCX_TLS` unset and let UCX pick transports (RDMA/verbs included). + Two exceptions: never run V2 NIXL with a container-default + `UCX_TLS=tcp` (breaks V2 NIXL VRAM registration; unset it), and on + clusters whose verbs transports cannot initialize, pin + `UCX_TLS=tcp,self,sm,cuda_copy,cuda_ipc` (an effective `UCX_TLS=all` + hangs setup there, see caveat 1). The pin excludes RDMA/verbs, so do + not carry it to clusters where verbs works. No bounce env override is needed: the byte gate (`TRTLLM_KV_CACHE_BOUNCE_MIN_BYTES`, default 2 MiB) is always cleared by K3 payloads (constraints section above). diff --git a/examples/kimi_k3/disagg/benchmark_kimi_k3_dep16.yaml b/examples/kimi_k3/disagg/benchmark_kimi_k3_dep16.yaml index bb0db632d279..bd5be001609c 100644 --- a/examples/kimi_k3/disagg/benchmark_kimi_k3_dep16.yaml +++ b/examples/kimi_k3/disagg/benchmark_kimi_k3_dep16.yaml @@ -57,13 +57,16 @@ environment: trtllm_wheel_path: "" work_dir: "" # Cluster notes: - # - On clusters where verbs UCX transports cannot initialize on the - # compute nodes, UCX_TLS must be pinned — UCX_TLS=all can wedge - # native NIXL init asymmetrically, leaving the surviving ranks hung - # in the V2 setup MPI collectives. start_worker.sh clears - # UCX_TLS (a container-provided UCX_TLS=tcp also breaks V2 NIXL VRAM - # registration), so the pin is carried via TRTLLM_WORKER_UCX_TLS and - # re-exported by start_worker.sh after the clear. + # - UCX transports: the default is no pin. start_worker.sh clears any + # container-provided UCX_TLS (UCX_TLS=tcp breaks V2 NIXL VRAM + # registration) and UCX then selects transports itself, RDMA/verbs + # included. Only on clusters where verbs transports cannot + # initialize on the compute nodes (symptom: UCX_TLS=all wedges + # native NIXL init asymmetrically, surviving ranks hang in the V2 + # setup MPI collectives), add + # TRTLLM_WORKER_UCX_TLS=tcp,self,sm,cuda_copy,cuda_ipc to + # worker_env_var below; start_worker.sh re-exports it as UCX_TLS + # after the clear. Note that list excludes RDMA/verbs. # - pairs with the workers' # kv_cache_bounce_size_mb (fabric-VMM bounce; measured # ~455 GB/s/GPU). The default gate of 96 blocks would silently skip @@ -73,7 +76,7 @@ environment: # in-place venv is injected via TRTLLM_PATH_PREPEND / # TRTLLM_PYTHONPATH_PREPEND, which the harness scripts apply inside # the container (setting PATH= here would be silently dropped). - worker_env_var: "TLLM_LOG_LEVEL=INFO TRTLLM_SERVER_DISABLE_GC=1 TRTLLM_WORKER_DISABLE_GC=1 ENROOT_ALLOW_DEV=yes TRTLLM_WORKER_UCX_TLS=tcp,self,sm,cuda_copy,cuda_ipc TRTLLM_PATH_PREPEND= TRTLLM_PYTHONPATH_PREPEND=" + worker_env_var: "TLLM_LOG_LEVEL=INFO TRTLLM_SERVER_DISABLE_GC=1 TRTLLM_WORKER_DISABLE_GC=1 ENROOT_ALLOW_DEV=yes TRTLLM_PATH_PREPEND= TRTLLM_PYTHONPATH_PREPEND=" server_env_var: "TRTLLM_SERVER_DISABLE_GC=1 TRTLLM_PATH_PREPEND= TRTLLM_PYTHONPATH_PREPEND=" worker_config: