diff --git a/docs/fern/backends/trtllm/trtllm-kv-cache-transfer.md b/docs/fern/backends/trtllm/trtllm-kv-cache-transfer.md index c4f7359dcf5b..6f6fc542f6fc 100644 --- a/docs/fern/backends/trtllm/trtllm-kv-cache-transfer.md +++ b/docs/fern/backends/trtllm/trtllm-kv-cache-transfer.md @@ -29,8 +29,17 @@ TensorRT-LLM supports two NIXL communication backends: UCX and LIBFABRIC. By def TensorRT-LLM can also leverage **UCX** (Unified Communication X) directly for KV cache transfer between prefill and decode workers. To enable UCX as the KV cache transfer backend, set `cache_transceiver_config.backend: UCX` in your engine configuration YAML file. -> [!NOTE] -> The environment variable `TRTLLM_USE_UCX_KVCACHE=1` with `cache_transceiver_config.backend: DEFAULT` does not enable UCX. You must explicitly set `backend: UCX` in the configuration. +`cache_transceiver_config.backend` accepts the following values: + +| Value | Behavior | +|-------|----------| +| Not set | KV cache transfer is disabled. | +| `DEFAULT` | Uses the backend named by the first of `TRTLLM_USE_NIXL_KVCACHE`, `TRTLLM_USE_UCX_KVCACHE`, `TRTLLM_USE_MOONCAKE_KVCACHE`, or `TRTLLM_USE_MPI_KVCACHE` that is set to `1`. Uses NIXL when none of them is set. | +| `UCX`, `NIXL`, `MOONCAKE`, or `MPI` | Uses that backend and ignores the environment variables above. | + +The precedence above matches TensorRT-LLM 1.3.0rc22; check `CacheTransceiverConfig._resolve_default_backend` in `tensorrt_llm/llmapi/llm_args.py`. + +The two paths produce different transceivers, which is how you tell them apart at runtime: direct UCX logs `UcxConnectionManager`, while NIXL with UCX underneath logs `NixlTransferAgent ... using NIXL backend: UCX`. Read the worker's startup log for the transceiver class rather than inferring it from the configuration. ## AWS EFA @@ -38,15 +47,17 @@ On AWS, UCX uses the **SRD (Scalable Reliable Datagram)** transport over EFA dev **Image options:** -- **Pre-built EFA image (AMD64 only):** A dedicated EFA image with the EFA SDK baked in is available on NGC. This is recommended for AMD64 instances (e.g. `p5.48xlarge`): +- **Pre-built EFA image:** A dedicated EFA image with the EFA SDK baked in is available on NGC, for both AMD64 and ARM64: ``` -nvcr.io/nvidia/ai-dynamo/tensorrtllm-runtime:1.2.1-efa-amd64 +nvcr.io/nvidia/ai-dynamo/tensorrtllm-runtime:1.3.0-efa ``` +On 1.2.1 the same image is tagged `1.2.1-efa-amd64`. Despite the suffix that tag is a multi-arch manifest covering AMD64 and ARM64; the name was corrected to `-efa` in 1.3.0. Pull the tag exactly as written, since `1.2.1-efa` was never published. + See [Release Artifacts](../../reference/release-artifacts.mdx) for all available EFA images. -- **Host-mount approach (ARM64 / GB200):** No pre-built EFA ARM64 image is published. Use the standard `tensorrtllm-runtime` image and mount the EFA SDK from the host node. This is what we tested on GB200 NVL72: +- **Host-mount approach (ARM64 / GB200):** Instead of the pre-built image, you can run the standard `tensorrtllm-runtime` image and mount the EFA SDK from the host node, which keeps the SDK in step with the host driver: ```yaml volumeMounts: @@ -58,6 +69,11 @@ volumes: path: /opt/amazon/efa ``` +> [!WARNING] +> Do not use this host mount with the LIBFABRIC backend on ARM64 / GB200. With `cache_transceiver_config.backend: NIXL` and `TRTLLM_NIXL_KVCACHE_BACKEND=LIBFABRIC`, mounting the host `/opt/amazon/efa` makes NIXL fail to register CUDA VRAM with `fi_mr_reg failed: Bad address`. TensorRT-LLM asserts immediately after, and both the prefill and decode workers enter `CrashLoopBackOff`. +> +> Use the pre-built `-efa` image instead. Removing the mount and relying on the EFA SDK shipped in that image makes an otherwise identical deployment serve inference on the same pair of nodes. + **EFA resource requests:** ```yaml