From 64dc261a0ac22eeeff7b18cc766b71c00cca7ce4 Mon Sep 17 00:00:00 2001 From: Dan Gil Date: Tue, 11 Aug 2026 16:15:37 -0500 Subject: [PATCH 1/2] docs(trtllm): correct the UCX env-var note and the EFA image facts on 1.4.0 Adapted port of the main-side fix. The page lives at a different path on this branch, and the release-branch copy is staler than main in a way that changes the fix, so this is not a clean pick. Three corrections: The TRTLLM_USE_UCX_KVCACHE note stated the opposite of the actual behavior. Setting it with cache_transceiver_config.backend: DEFAULT does select direct UCX and is equivalent to backend: UCX; both produce a UcxConnectionManager. The EFA image was described as AMD64 only, with the claim that no pre-built ARM64 image is published. Both are false. The -efa tags are multi-arch manifests covering amd64 and arm64, confirmed against the registry, including 1.2.1-efa-amd64 despite its suffix. This matters here because the page told ARM64 readers the host mount was their only option. The ARM64 host mount breaks NIXL LIBFABRIC CUDA VRAM registration with fi_mr_reg failed: Bad address, asserting TensorRT-LLM and putting both workers into CrashLoopBackOff. It is kept documented with a warning scoped to that backend, and the pre-built image is now a real alternative to point at rather than one the page denied existed. Signed-off-by: Dan Gil --- .../backends/trtllm/trtllm-kv-cache-transfer.md | 17 +++++++++++++---- 1 file changed, 13 insertions(+), 4 deletions(-) diff --git a/docs/fern/backends/trtllm/trtllm-kv-cache-transfer.md b/docs/fern/backends/trtllm/trtllm-kv-cache-transfer.md index c4f7359dcf5b..12dd202e3ac7 100644 --- a/docs/fern/backends/trtllm/trtllm-kv-cache-transfer.md +++ b/docs/fern/backends/trtllm/trtllm-kv-cache-transfer.md @@ -30,7 +30,9 @@ TensorRT-LLM supports two NIXL communication backends: UCX and LIBFABRIC. By def TensorRT-LLM can also leverage **UCX** (Unified Communication X) directly for KV cache transfer between prefill and decode workers. To enable UCX as the KV cache transfer backend, set `cache_transceiver_config.backend: UCX` in your engine configuration YAML file. > [!NOTE] -> The environment variable `TRTLLM_USE_UCX_KVCACHE=1` with `cache_transceiver_config.backend: DEFAULT` does not enable UCX. You must explicitly set `backend: UCX` in the configuration. +> Setting `TRTLLM_USE_UCX_KVCACHE=1` while `cache_transceiver_config.backend` is `DEFAULT` selects direct UCX, and is equivalent to setting `backend: UCX` in the configuration. Both produce a `UcxConnectionManager`. +> +> This is a different transport from the default. With `backend: DEFAULT` and no environment variable, TensorRT-LLM uses the NIXL transfer agent with UCX underneath, which logs `NixlTransferAgent ... using NIXL backend: UCX`. To confirm which one a worker chose, read its startup log for the transceiver class rather than inferring it from the configuration. ## AWS EFA @@ -38,15 +40,17 @@ On AWS, UCX uses the **SRD (Scalable Reliable Datagram)** transport over EFA dev **Image options:** -- **Pre-built EFA image (AMD64 only):** A dedicated EFA image with the EFA SDK baked in is available on NGC. This is recommended for AMD64 instances (e.g. `p5.48xlarge`): +- **Pre-built EFA image:** A dedicated EFA image with the EFA SDK baked in is available on NGC, for both AMD64 and ARM64: ``` -nvcr.io/nvidia/ai-dynamo/tensorrtllm-runtime:1.2.1-efa-amd64 +nvcr.io/nvidia/ai-dynamo/tensorrtllm-runtime:1.3.0-efa ``` +On 1.2.1 the same image is tagged `1.2.1-efa-amd64`. Despite the suffix that tag is a multi-arch manifest covering AMD64 and ARM64; the name was corrected to `-efa` in 1.3.0. Pull the tag exactly as written, since `1.2.1-efa` was never published. + See [Release Artifacts](../../reference/release-artifacts.mdx) for all available EFA images. -- **Host-mount approach (ARM64 / GB200):** No pre-built EFA ARM64 image is published. Use the standard `tensorrtllm-runtime` image and mount the EFA SDK from the host node. This is what we tested on GB200 NVL72: +- **Host-mount approach (ARM64 / GB200):** Instead of the pre-built image, you can run the standard `tensorrtllm-runtime` image and mount the EFA SDK from the host node, which keeps the SDK in step with the host driver: ```yaml volumeMounts: @@ -58,6 +62,11 @@ volumes: path: /opt/amazon/efa ``` +> [!WARNING] +> Do not use this host mount with the LIBFABRIC backend on ARM64 / GB200. With `cache_transceiver_config.backend: NIXL` and `TRTLLM_NIXL_KVCACHE_BACKEND=LIBFABRIC`, mounting the host `/opt/amazon/efa` makes NIXL fail to register CUDA VRAM with `fi_mr_reg failed: Bad address`. TensorRT-LLM asserts immediately after, and both the prefill and decode workers enter `CrashLoopBackOff`. +> +> Use the pre-built `-efa` image instead. Removing the mount and relying on the EFA SDK shipped in that image makes an otherwise identical deployment serve inference on the same pair of nodes. + **EFA resource requests:** ```yaml From 6d0ac2ea5757ff9726ccfd3e415b2a67f0a04273 Mon Sep 17 00:00:00 2001 From: Dan Gil Date: Wed, 12 Aug 2026 14:05:03 -0500 Subject: [PATCH 2/2] docs(trtllm): carry the upstream backend precedence table onto 1.4.0 Replaces the hand-written NOTE in this cherry-pick with the table Chi Xing landed on main in #13015, so the release branch matches main rather than carrying a second, differently-worded correction of the same defect. Both fix the inversion that 6590511 reports. His version is the better one: it documents all four TRTLLM_USE_*_KVCACHE variables and their precedence rather than only the UCX one, states the not-set case, and cites CacheTransceiverConfig._resolve_default_backend so the claim can be rechecked against a specific TensorRT-LLM version. Kept from the earlier revision: the sentence naming which transceiver class each path logs. That is the part QA needed to tell the two apart from a worker startup log, and #13015 does not carry it. The file lives at a different path on this branch, so this is an adapted port rather than a clean pick. Signed-off-by: Dan Gil --- .../backends/trtllm/trtllm-kv-cache-transfer.md | 15 +++++++++++---- 1 file changed, 11 insertions(+), 4 deletions(-) diff --git a/docs/fern/backends/trtllm/trtllm-kv-cache-transfer.md b/docs/fern/backends/trtllm/trtllm-kv-cache-transfer.md index 12dd202e3ac7..6f6fc542f6fc 100644 --- a/docs/fern/backends/trtllm/trtllm-kv-cache-transfer.md +++ b/docs/fern/backends/trtllm/trtllm-kv-cache-transfer.md @@ -29,10 +29,17 @@ TensorRT-LLM supports two NIXL communication backends: UCX and LIBFABRIC. By def TensorRT-LLM can also leverage **UCX** (Unified Communication X) directly for KV cache transfer between prefill and decode workers. To enable UCX as the KV cache transfer backend, set `cache_transceiver_config.backend: UCX` in your engine configuration YAML file. -> [!NOTE] -> Setting `TRTLLM_USE_UCX_KVCACHE=1` while `cache_transceiver_config.backend` is `DEFAULT` selects direct UCX, and is equivalent to setting `backend: UCX` in the configuration. Both produce a `UcxConnectionManager`. -> -> This is a different transport from the default. With `backend: DEFAULT` and no environment variable, TensorRT-LLM uses the NIXL transfer agent with UCX underneath, which logs `NixlTransferAgent ... using NIXL backend: UCX`. To confirm which one a worker chose, read its startup log for the transceiver class rather than inferring it from the configuration. +`cache_transceiver_config.backend` accepts the following values: + +| Value | Behavior | +|-------|----------| +| Not set | KV cache transfer is disabled. | +| `DEFAULT` | Uses the backend named by the first of `TRTLLM_USE_NIXL_KVCACHE`, `TRTLLM_USE_UCX_KVCACHE`, `TRTLLM_USE_MOONCAKE_KVCACHE`, or `TRTLLM_USE_MPI_KVCACHE` that is set to `1`. Uses NIXL when none of them is set. | +| `UCX`, `NIXL`, `MOONCAKE`, or `MPI` | Uses that backend and ignores the environment variables above. | + +The precedence above matches TensorRT-LLM 1.3.0rc22; check `CacheTransceiverConfig._resolve_default_backend` in `tensorrt_llm/llmapi/llm_args.py`. + +The two paths produce different transceivers, which is how you tell them apart at runtime: direct UCX logs `UcxConnectionManager`, while NIXL with UCX underneath logs `NixlTransferAgent ... using NIXL backend: UCX`. Read the worker's startup log for the transceiver class rather than inferring it from the configuration. ## AWS EFA