[TRTLLM-15264][doc] Kimi K3 disagg: stop recommending a UCX_TLS pin by default - #17479
Conversation
…y default The K3 disagg README exported UCX_TLS=tcp,self,sm,cuda_copy,cuda_ipc as a launch prerequisite and the benchmark config hard-coded the same pin in worker_env_var. That transport list excludes RDMA/verbs, so carrying it to clusters whose inter-node traffic depends on verbs can fail initialization or silently force traffic onto TCP. Make no pin the default: leave UCX_TLS unset so UCX selects transports itself (the harness already clears any container-provided UCX_TLS=tcp, which breaks V2 NIXL VRAM registration), and move the transport override to the troubleshooting caveats, scoped to clusters where verbs transports cannot initialize. Addresses review feedback on NVIDIA#17334. Signed-off-by: Brian Nguyen <brnguyen@nvidia.com>
|
No actionable comments were generated in the recent review. 🎉 ℹ️ Recent review info⚙️ Run configurationConfiguration used: Path: .coderabbit.yaml Review profile: CHILL Plan: Enterprise Run ID: 📒 Files selected for processing (2)
WalkthroughThe Kimi K3 disaggregated deployment guidance now leaves UCX transport selection automatic by default. It documents clearing container-provided settings and using ChangesUCX transport guidance
Estimated code review effort: 2 (Simple) | ~10 minutes Suggested reviewers: 🚥 Pre-merge checks | ✅ 5✅ Passed checks (5 passed)
✨ Finishing Touches🧪 Generate unit tests (beta)
Comment |
|
/bot run |
|
PR_Github #65139 [ run ] triggered by Bot. Commit: |
|
PR_Github #65139 [ run ] completed with state
|
|
/bot run |
|
PR_Github #65190 [ run ] triggered by Bot. Commit: |
|
PR_Github #65190 [ run ] completed with state |
Description
Follow-up to review feedback on #17334 (#17334 (comment)).
The K3 disagg README exported
UCX_TLS=tcp,self,sm,cuda_copy,cuda_ipcas a launch prerequisite, and the benchmark config hard-coded the same pin inworker_env_var. That transport list excludes RDMA/verbs, so carrying it to clusters whose inter-node communication depends on verbs can fail initialization or silently force traffic onto TCP.This change makes no pin the default:
UCX_TLSunset" (UCX selects transports itself, RDMA/verbs included), with a pointer to the troubleshooting caveats.benchmark_kimi_k3_dep16.yamlno longer carriesTRTLLM_WORKER_UCX_TLSinworker_env_var; the harness'sstart_worker.shalready clears any container-providedUCX_TLS(a container-defaultUCX_TLS=tcpbreaks V2 NIXL VRAM registration), so workers default to UCX's own selection.Test Coverage
Documentation/example-config change only; no code paths affected.
PR Checklist
[TRTLLM-15264][doc]conventionDev Engineer Review
UCX_TLSunset by default.TRTLLM_WORKER_UCX_TLSfrom the example worker environment.QA Engineer Review
No test changes.