-
Notifications
You must be signed in to change notification settings - Fork 1.5k
docs(trtllm): correct the UCX env-var note and the EFA image facts on 1.4.0 #13049
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
Changes from all commits
64dc261
7d0805e
6d0ac2e
File filter
Filter by extension
Conversations
Jump to
Diff view
Diff view
There are no files selected for viewing
| Original file line number | Diff line number | Diff line change |
|---|---|---|
|
|
@@ -29,24 +29,35 @@ TensorRT-LLM supports two NIXL communication backends: UCX and LIBFABRIC. By def | |
|
|
||
| TensorRT-LLM can also leverage **UCX** (Unified Communication X) directly for KV cache transfer between prefill and decode workers. To enable UCX as the KV cache transfer backend, set `cache_transceiver_config.backend: UCX` in your engine configuration YAML file. | ||
|
|
||
| > [!NOTE] | ||
| > The environment variable `TRTLLM_USE_UCX_KVCACHE=1` with `cache_transceiver_config.backend: DEFAULT` does not enable UCX. You must explicitly set `backend: UCX` in the configuration. | ||
| `cache_transceiver_config.backend` accepts the following values: | ||
|
|
||
| | Value | Behavior | | ||
| |-------|----------| | ||
| | Not set | KV cache transfer is disabled. | | ||
| | `DEFAULT` | Uses the backend named by the first of `TRTLLM_USE_NIXL_KVCACHE`, `TRTLLM_USE_UCX_KVCACHE`, `TRTLLM_USE_MOONCAKE_KVCACHE`, or `TRTLLM_USE_MPI_KVCACHE` that is set to `1`. Uses NIXL when none of them is set. | | ||
| | `UCX`, `NIXL`, `MOONCAKE`, or `MPI` | Uses that backend and ignores the environment variables above. | | ||
|
|
||
| The precedence above matches TensorRT-LLM 1.3.0rc22; check `CacheTransceiverConfig._resolve_default_backend` in `tensorrt_llm/llmapi/llm_args.py`. | ||
|
|
||
| The two paths produce different transceivers, which is how you tell them apart at runtime: direct UCX logs `UcxConnectionManager`, while NIXL with UCX underneath logs `NixlTransferAgent ... using NIXL backend: UCX`. Read the worker's startup log for the transceiver class rather than inferring it from the configuration. | ||
|
|
||
| ## AWS EFA | ||
|
|
||
| On AWS, UCX uses the **SRD (Scalable Reliable Datagram)** transport over EFA devices. NIXL discovers EFA `rdmap*` devices automatically through UCX — no NIXL-level configuration changes are needed. | ||
|
|
||
| **Image options:** | ||
|
|
||
| - **Pre-built EFA image (AMD64 only):** A dedicated EFA image with the EFA SDK baked in is available on NGC. This is recommended for AMD64 instances (e.g. `p5.48xlarge`): | ||
| - **Pre-built EFA image:** A dedicated EFA image with the EFA SDK baked in is available on NGC, for both AMD64 and ARM64: | ||
|
|
||
| ``` | ||
| nvcr.io/nvidia/ai-dynamo/tensorrtllm-runtime:1.2.1-efa-amd64 | ||
| nvcr.io/nvidia/ai-dynamo/tensorrtllm-runtime:1.3.0-efa | ||
| ``` | ||
|
|
||
| On 1.2.1 the same image is tagged `1.2.1-efa-amd64`. Despite the suffix that tag is a multi-arch manifest covering AMD64 and ARM64; the name was corrected to `-efa` in 1.3.0. Pull the tag exactly as written, since `1.2.1-efa` was never published. | ||
|
|
||
| See [Release Artifacts](../../reference/release-artifacts.mdx) for all available EFA images. | ||
|
|
||
| - **Host-mount approach (ARM64 / GB200):** No pre-built EFA ARM64 image is published. Use the standard `tensorrtllm-runtime` image and mount the EFA SDK from the host node. This is what we tested on GB200 NVL72: | ||
| - **Host-mount approach (ARM64 / GB200):** Instead of the pre-built image, you can run the standard `tensorrtllm-runtime` image and mount the EFA SDK from the host node, which keeps the SDK in step with the host driver: | ||
|
|
||
| ```yaml | ||
| volumeMounts: | ||
|
|
@@ -58,6 +69,11 @@ volumes: | |
| path: /opt/amazon/efa | ||
| ``` | ||
|
|
||
| > [!WARNING] | ||
| > Do not use this host mount with the LIBFABRIC backend on ARM64 / GB200. With `cache_transceiver_config.backend: NIXL` and `TRTLLM_NIXL_KVCACHE_BACKEND=LIBFABRIC`, mounting the host `/opt/amazon/efa` makes NIXL fail to register CUDA VRAM with `fi_mr_reg failed: Bad address`. TensorRT-LLM asserts immediately after, and both the prefill and decode workers enter `CrashLoopBackOff`. | ||
| > | ||
| > Use the pre-built `-efa` image instead. Removing the mount and relying on the EFA SDK shipped in that image makes an otherwise identical deployment serve inference on the same pair of nodes. | ||
|
Comment on lines
+72
to
+75
Contributor
There was a problem hiding this comment. Choose a reason for hiding this commentThe reason will be displayed to describe this comment to others. Learn more. 🔍 New warning references a backend value not otherwise documented on this page The warning uses Was this helpful? React with 👍 or 👎 to provide feedback. |
||
|
|
||
| **EFA resource requests:** | ||
|
|
||
| ```yaml | ||
|
|
||
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
🔍 Recommended image tag lags the branch version and other docs still cite the AMD64-suffixed tag
The example tag is
1.3.0-efa, which matches the current stable inventory indocs/fern/reference/release-artifacts.mdx:96even though this branch is versioned 1.4.0 (Cargo.toml:38). If a 1.4.0 EFA image is published for this release branch, this snippet will be stale on day one. Also, other pages still describe1.2.1-efa-amd64as an AMD64 image (docs/fern/kubernetes/cloud-providers/eks/eks.mdx:238,docs/fern/kubernetes/disagg-communication-guide.md:305), so the multi-arch correction made here is only partially applied across the docs set.Was this helpful? React with 👍 or 👎 to provide feedback.