Skip to content
Merged
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
26 changes: 21 additions & 5 deletions docs/fern/backends/trtllm/trtllm-kv-cache-transfer.md
Original file line number Diff line number Diff line change
Expand Up @@ -29,24 +29,35 @@ TensorRT-LLM supports two NIXL communication backends: UCX and LIBFABRIC. By def

TensorRT-LLM can also leverage **UCX** (Unified Communication X) directly for KV cache transfer between prefill and decode workers. To enable UCX as the KV cache transfer backend, set `cache_transceiver_config.backend: UCX` in your engine configuration YAML file.

> [!NOTE]
> The environment variable `TRTLLM_USE_UCX_KVCACHE=1` with `cache_transceiver_config.backend: DEFAULT` does not enable UCX. You must explicitly set `backend: UCX` in the configuration.
`cache_transceiver_config.backend` accepts the following values:

| Value | Behavior |
|-------|----------|
| Not set | KV cache transfer is disabled. |
| `DEFAULT` | Uses the backend named by the first of `TRTLLM_USE_NIXL_KVCACHE`, `TRTLLM_USE_UCX_KVCACHE`, `TRTLLM_USE_MOONCAKE_KVCACHE`, or `TRTLLM_USE_MPI_KVCACHE` that is set to `1`. Uses NIXL when none of them is set. |
| `UCX`, `NIXL`, `MOONCAKE`, or `MPI` | Uses that backend and ignores the environment variables above. |

The precedence above matches TensorRT-LLM 1.3.0rc22; check `CacheTransceiverConfig._resolve_default_backend` in `tensorrt_llm/llmapi/llm_args.py`.

The two paths produce different transceivers, which is how you tell them apart at runtime: direct UCX logs `UcxConnectionManager`, while NIXL with UCX underneath logs `NixlTransferAgent ... using NIXL backend: UCX`. Read the worker's startup log for the transceiver class rather than inferring it from the configuration.

## AWS EFA

On AWS, UCX uses the **SRD (Scalable Reliable Datagram)** transport over EFA devices. NIXL discovers EFA `rdmap*` devices automatically through UCX — no NIXL-level configuration changes are needed.

**Image options:**

- **Pre-built EFA image (AMD64 only):** A dedicated EFA image with the EFA SDK baked in is available on NGC. This is recommended for AMD64 instances (e.g. `p5.48xlarge`):
- **Pre-built EFA image:** A dedicated EFA image with the EFA SDK baked in is available on NGC, for both AMD64 and ARM64:

```
nvcr.io/nvidia/ai-dynamo/tensorrtllm-runtime:1.2.1-efa-amd64
nvcr.io/nvidia/ai-dynamo/tensorrtllm-runtime:1.3.0-efa
```

On 1.2.1 the same image is tagged `1.2.1-efa-amd64`. Despite the suffix that tag is a multi-arch manifest covering AMD64 and ARM64; the name was corrected to `-efa` in 1.3.0. Pull the tag exactly as written, since `1.2.1-efa` was never published.
Comment on lines +53 to +56

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

🔍 Recommended image tag lags the branch version and other docs still cite the AMD64-suffixed tag

The example tag is 1.3.0-efa, which matches the current stable inventory in docs/fern/reference/release-artifacts.mdx:96 even though this branch is versioned 1.4.0 (Cargo.toml:38). If a 1.4.0 EFA image is published for this release branch, this snippet will be stale on day one. Also, other pages still describe 1.2.1-efa-amd64 as an AMD64 image (docs/fern/kubernetes/cloud-providers/eks/eks.mdx:238, docs/fern/kubernetes/disagg-communication-guide.md:305), so the multi-arch correction made here is only partially applied across the docs set.

Open in Devin Review

Was this helpful? React with 👍 or 👎 to provide feedback.


See [Release Artifacts](../../reference/release-artifacts.mdx) for all available EFA images.

- **Host-mount approach (ARM64 / GB200):** No pre-built EFA ARM64 image is published. Use the standard `tensorrtllm-runtime` image and mount the EFA SDK from the host node. This is what we tested on GB200 NVL72:
- **Host-mount approach (ARM64 / GB200):** Instead of the pre-built image, you can run the standard `tensorrtllm-runtime` image and mount the EFA SDK from the host node, which keeps the SDK in step with the host driver:

```yaml
volumeMounts:
Expand All @@ -58,6 +69,11 @@ volumes:
path: /opt/amazon/efa
```

> [!WARNING]
> Do not use this host mount with the LIBFABRIC backend on ARM64 / GB200. With `cache_transceiver_config.backend: NIXL` and `TRTLLM_NIXL_KVCACHE_BACKEND=LIBFABRIC`, mounting the host `/opt/amazon/efa` makes NIXL fail to register CUDA VRAM with `fi_mr_reg failed: Bad address`. TensorRT-LLM asserts immediately after, and both the prefill and decode workers enter `CrashLoopBackOff`.
>
> Use the pre-built `-efa` image instead. Removing the mount and relying on the EFA SDK shipped in that image makes an otherwise identical deployment serve inference on the same pair of nodes.
Comment on lines +72 to +75

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

🔍 New warning references a backend value not otherwise documented on this page

The warning uses cache_transceiver_config.backend: NIXL, but the rest of the page only documents DEFAULT and UCX for that key, with the NIXL sub-backend selected via TRTLLM_NIXL_KVCACHE_BACKEND. Readers may be unsure whether NIXL is a valid literal for cache_transceiver_config.backend; a short clarification (or using DEFAULT if that is what was actually tested) would avoid ambiguity.

Open in Devin Review

Was this helpful? React with 👍 or 👎 to provide feedback.


**EFA resource requests:**

```yaml
Expand Down
Loading