fix: Update tensorrt_llm to 1.0.0rc6 (#2606)

Signed-off-by: nnshah1 <[email protected]>

Author: tanmayv25

README.md

@@ -192,7 +192,7 @@ It is recommended to use [NGC PyTorch Container](https://catalog.ngc.nvidia.com/
 
 > [!Note]
 > Ensure that you select a PyTorch container image version that matches the version of TensorRT-LLM you are using.
-> For example, if you are using `tensorrt-llm==1.0.0rc4`, use the PyTorch container image version `25.05`.
+> For example, if you are using `tensorrt-llm==1.0.0rc6`, use the PyTorch container image version `25.06`.
 > To find the correct PyTorch container version for your desired `tensorrt-llm` release, visit the [TensorRT-LLM Dockerfile.multi](https://github.com/NVIDIA/TensorRT-LLM/blob/main/docker/Dockerfile.multi) on GitHub. Switch to the branch that matches your `tensorrt-llm` version, and look for the `BASE_TAG` line to identify the recommended PyTorch container tag.
 
 > [!Important]

components/backends/trtllm/README.md

@@ -169,10 +169,6 @@ export MODEL_PATH="nvidia/DeepSeek-R1-FP4"
 ```
 
 Notes:
-- MTP is only available within the container built with the experimental TensorRT-LLM commit. Please add --use-default-experimental-tensorrtllm-commit to the arguments of the build.sh script.
-
-  Example: `./container/build.sh --framework trtllm --use-default-experimental-tensorrtllm-commit`
-
 - There is a noticeable latency for the first two inference requests. Please send warm-up requests before starting the benchmark.
 - MTP performance may vary depending on the acceptance rate of predicted tokens, which is dependent on the dataset or queries used while benchmarking. Additionally, `ignore_eos` should generally be omitted or set to `false` when using MTP to avoid speculating garbage outputs and getting unrealistic acceptance rates.
 
@@ -244,140 +240,7 @@ To benchmark your deployment with GenAI-Perf, see this utility script, configuri
 
 ## Multimodal support
 
-TRTLLM supports multimodal models with dynamo. You can provide multimodal inputs in the following ways:
-
-- By sending image URLs
-- By providing paths to pre-computed embedding files
-
-Please note that you should provide **either image URLs or embedding file paths** in a single request.
-
-### Aggregated
-
-Here are quick steps to launch Llama-4 Maverick BF16 in aggregated mode
-```bash
-cd $DYNAMO_HOME/components/backends/trtllm
-
-export AGG_ENGINE_ARGS=./engine_configs/multinode/agg.yaml
-export SERVED_MODEL_NAME="meta-llama/Llama-4-Maverick-17B-128E-Instruct"
-export MODEL_PATH="meta-llama/Llama-4-Maverick-17B-128E-Instruct"
-./launch/agg.sh
-```
-### Example Requests
-
-#### With Image URL
-
-Below is an example of an image being sent to `Llama-4-Maverick-17B-128E-Instruct` model
-
-Request :
-```bash
-curl localhost:8000/v1/chat/completions -H "Content-Type: application/json" -d '{
-    "model": "meta-llama/Llama-4-Maverick-17B-128E-Instruct",
-    "messages": [
-        {
-            "role": "user",
-            "content": [
-                {
-                    "type": "text",
-                    "text": "Describe the image"
-                },
-                {
-                    "type": "image_url",
-                    "image_url": {
-                        "url": "https://huggingface.co/datasets/huggingface/documentation-images/resolve/main/diffusers/inpaint.png"
-                    }
-                }
-            ]
-        }
-    ],
-    "stream": false,
-    "max_tokens": 160
-}'
-```
-Response :
-
-```
-{"id":"unknown-id","choices":[{"index":0,"message":{"content":"The image depicts a serene landscape featuring a large rock formation, likely El Capitan in Yosemite National Park, California. The scene is characterized by a winding road that curves from the bottom-right corner towards the center-left of the image, with a few rocks and trees lining its edge.\n\n**Key Features:**\n\n* **Rock Formation:** A prominent, tall, and flat-topped rock formation dominates the center of the image.\n* **Road:** A paved road winds its way through the landscape, curving from the bottom-right corner towards the center-left.\n* **Trees and Rocks:** Trees are visible on both sides of the road, with rocks scattered along the left side.\n* **Sky:** The sky above is blue, dotted with white clouds.\n* **Atmosphere:** The overall atmosphere of the","refusal":null,"tool_calls":null,"role":"assistant","function_call":null,"audio":null},"finish_reason":"stop","logprobs":null}],"created":1753322607,"model":"meta-llama/Llama-4-Maverick-17B-128E-Instruct","service_tier":null,"system_fingerprint":null,"object":"chat.completion","usage":null}
-```
-
-### Disaggregated
-
-Here are quick steps to launch in disaggregated mode.
-
-The following is an example of launching a model in disaggregated mode. While this example uses `Qwen/Qwen2-VL-7B-Instruct`, you can adapt it for other models by modifying the environment variables for the model path and engine configurations.
-```bash
-cd $DYNAMO_HOME/components/backends/trtllm
-
-export MODEL_PATH=${MODEL_PATH:-"Qwen/Qwen2-VL-7B-Instruct"}
-export SERVED_MODEL_NAME=${SERVED_MODEL_NAME:-"Qwen/Qwen2-VL-7B-Instruct"}
-export DISAGGREGATION_STRATEGY=${DISAGGREGATION_STRATEGY:-"decode_first"}
-export PREFILL_ENGINE_ARGS=${PREFILL_ENGINE_ARGS:-"engine_configs/multimodal/prefill.yaml"}
-export DECODE_ENGINE_ARGS=${DECODE_ENGINE_ARGS:-"engine_configs/multimodal/decode.yaml"}
-export MODALITY=${MODALITY:-"multimodal"}
-
-./launch/disagg.sh
-```
-
-For a large model like `meta-llama/Llama-4-Maverick-17B-128E-Instruct`, a multi-node setup is required for disaggregated serving, while aggregated serving can run on a single node. This is because the model with a disaggregated configuration is too large to fit on a single node's GPUs. For instance, running this model in disaggregated mode requires a setup of 2 nodes with 8xH200 GPUs or 4 nodes with 4xGB200 GPUs.
-
-In general, disaggregated serving can run on a single node, provided the model fits on the GPU. The multi-node requirement in this example is specific to the size and configuration of the `meta-llama/Llama-4-Maverick-17B-128E-Instruct` model.
-
-To deploy `Llama-4-Maverick-17B-128E-Instruct` in disaggregated mode, you will need to follow the multi-node setup instructions, which can be found [here](multinode/multinode-multimodal-example.md).
-
-### Using Pre-computed Embeddings (Experimental)
-
-Dynamo with TensorRT-LLM supports providing pre-computed embeddings directly in an inference request. This bypasses the need for the model to process an image and generate embeddings itself, which is useful for performance optimization or when working with custom, pre-generated embeddings.
-
-#### Enabling the Feature
-
-This is an experimental feature that requires using a specific TensorRT-LLM commit.
-To enable it build the dynamo container with the `--tensorrtllm-commit` flag, followed by the commit hash:
-
-```bash
-./container/build.sh --framework trtllm --tensorrtllm-commit b4065d8ca64a64eee9fdc64b39cb66d73d4be47c
-```
-
-#### How to Use
-
-Once the container is built, you can send requests with paths to local embedding files.
-
--   **Format:** Provide the embedding as part of the `messages` array, using the `image_url` content type.
--   **URL:** The `url` field should contain the absolute or relative path to your embedding file on the local filesystem.
--   **File Types:** Supported embedding file extensions are `.pt`, `.pth`, and `.bin`. Dynamo will automatically detect these extensions.
-
-When a request with a supported embedding file is received, Dynamo will load the tensor from the file and pass it directly to the model for inference, skipping the image-to-embedding pipeline.
-
-#### Example Request
-
-Here is an example of how to send a request with a pre-computed embedding file.
-
-```bash
-curl localhost:8000/v1/chat/completions -H "Content-Type: application/json" -d '{
-    "model": "meta-llama/Llama-4-Maverick-17B-128E-Instruct",
-    "messages": [
-        {
-            "role": "user",
-            "content": [
-                {
-                    "type": "text",
-                    "text": "Describe the content represented by the embeddings"
-                },
-                {
-                    "type": "image_url",
-                    "image_url": {
-                        "url": "/path/to/your/embedding.pt"
-                    }
-                }
-            ]
-        }
-    ],
-    "stream": false,
-    "max_tokens": 160
-}'
-```
-
-### Supported Multimodal Models
-
-Multimodel models listed [here](https://github.com/NVIDIA/TensorRT-LLM/blob/main/tensorrt_llm/inputs/utils.py#L221) are supported by dynamo.
+Dynamo with the TensorRT-LLM backend supports multimodal models, enabling you to process both text and images (or pre-computed embeddings) in a single request. For detailed setup instructions, example requests, and best practices, see the [Multimodal Support Guide](./multimodal_support.md).
 
 ## Performance Sweep
 

components/backends/trtllm/deploy/README.md

@@ -219,14 +219,6 @@ Send a test request to verify your deployment. See the [client section](../../..
 
 The deployment templates support various TensorRT-LLM models and configurations. You can customize model-specific arguments in the worker configuration sections of the YAML files.
 
-### Multi-Token Prediction (MTP) Support
-
-For models supporting Multi-Token Prediction (such as DeepSeek R1), special configuration is available. Note that MTP requires the experimental TensorRT-LLM commit:
-
-```bash
-./container/build.sh --framework tensorrtllm --use-default-experimental-tensorrtllm-commit
-```
-
 ## Monitoring and Health
 
 - **Frontend health endpoint**: `http://<frontend-service>:8000/health`

components/backends/trtllm/gemma3_sliding_window_attention.md

@@ -20,12 +20,9 @@ limitations under the License.
 This guide demonstrates how to deploy google/gemma-3-1b-it with Variable Sliding Window Attention (VSWA) using Dynamo. Since google/gemma-3-1b-it is a small model, each aggregated, decode, or prefill worker only requires one H100 GPU or one GB200 GPU.
 VSWA is a mechanism in which a model’s layers alternate between multiple sliding window sizes. An example of this is Gemma 3, which incorporates both global attention layers and sliding window layers.
 
-## Notes
-* To run Gemma 3 with VSWA and KV Routing with KV block reuse, ensure that the container is built using commit ID `c9eebcb4541d961ab390f0bd0a22e2c89f1bcc78` from Tensorrt-LLM.
-```bash
-./container/build.sh --framework trtllm --tensorrtllm-commit c9eebcb4541d961ab390f0bd0a22e2c89f1bcc78
-```
-* The 1.0.0rc4 release version of TensorRT-LLM can also run Gemma 3 with VSWA, but KV block reuse cannot be turned on in that version.
+> [!Note]
+> - Ensure that required services such as `nats` and `etcd` are running before starting.
+> - Request access to `google/gemma-3-1b-it` on Hugging Face and set your `HF_TOKEN` environment variable for authentication.
 
 ### Aggregated Serving
 ```bash

components/backends/trtllm/multimodal_support.md

@@ -2,10 +2,10 @@
 # SPDX-License-Identifier: Apache-2.0
 
 ARG BASE_IMAGE="nvcr.io/nvidia/pytorch"
-ARG BASE_IMAGE_TAG="25.05-py3"
+ARG BASE_IMAGE_TAG="25.06-py3"
 ARG RELEASE_BUILD
 ARG RUNTIME_IMAGE="nvcr.io/nvidia/cuda"
-ARG RUNTIME_IMAGE_TAG="12.9.0-runtime-ubuntu24.04"
+ARG RUNTIME_IMAGE_TAG="12.9.1-runtime-ubuntu24.04"
 
 # Define general architecture ARGs for supporting both x86 and aarch64 builds.
 #   ARCH: Used for package suffixes (e.g., amd64, arm64)
@@ -140,7 +140,7 @@ COPY --from=trtllm_wheel . /trtllm_wheel/
 # Note: TensorRT needs to be uninstalled before installing the TRTLLM wheel
 # because there might be mismatched versions of TensorRT between the NGC PyTorch
 # and the TRTLLM wheel.
-# Locking triton version to 3.3.1 as 3.4.0 breaks tensorrt-llm 1.0.0rc4
+# Locking triton version to 3.3.1 as 3.4.0 breaks tensorrt-llm 1.0.0rc6
 RUN [ -f /etc/pip/constraint.txt ] && : > /etc/pip/constraint.txt || true && \
     pip uninstall -y tensorrt && \
     if [ "$HAS_TRTLLM_CONTEXT" = "1" ]; then \
@@ -425,15 +425,15 @@ COPY --from=build /usr/local/cuda/lib64/libcudart.so* /usr/local/cuda/lib64/
 COPY --from=build /usr/local/cuda/nvvm /usr/local/cuda/nvvm
 
 # Copy pytorch installation from NGC PyTorch
-ARG TORCH_VER=2.8.0a0+5228986c39.nv25.5
-ARG TORCHVISION_VER=0.22.0a0
+ARG TORCH_VER=2.8.0a0+5228986c39.nv25.6
+ARG TORCHVISION_VER=0.22.0a0+95f10a4e
 ARG SETUPTOOLS_VER=78.1.1
 ARG PYTORCH_TRITON_VER=3.3.0+git96316ce52.nvinternal
 ARG JINJA2_VER=3.1.6
-ARG NETWORKX_VER=3.4.2
+ARG NETWORKX_VER=3.5
 ARG SYMPY_VER=1.14.0
 ARG PACKAGING_VER=23.2
-ARG FLASH_ATTN_VER=2.7.3
+ARG FLASH_ATTN_VER=2.7.4.post1
 ARG MPMATH_VER=1.3.0
 COPY --from=build /usr/local/lib/lib* /usr/local/lib/
 COPY --from=build /usr/local/lib/python3.12/dist-packages/torch /usr/local/lib/python3.12/dist-packages/torch
@@ -474,8 +474,8 @@ COPY --from=dev /workspace/target/release/metrics /usr/local/bin/metrics
 # NOTE: If a package (tensorrt_llm) exists on both --index-url and --extra-index-url,
 # uv will prioritize the --extra-index-url, unless --index-strategy unsafe-best-match
 # is also specified. So set the configurable index as a --extra-index-url for prioritization.
-# NOTE: locking triton version to 3.3.1 as 3.4.0 breaks tensorrt-llm 1.0.0rc4
-# NOTE: locking cuda-python version to <13 to avoid breaks with tensorrt-llm 1.0.0rc4. This
+# NOTE: locking triton version to 3.3.1 as 3.4.0 breaks tensorrt-llm 1.0.0rc6
+# NOTE: locking cuda-python version to <13 to avoid breaks with tensorrt-llm 1.0.0rc6. This
 #       can be removed after https://github.com/NVIDIA/TensorRT-LLM/pull/6703 is merged
 #       we upgrade to a published pip wheel containing this change.
 RUN uv pip install "cuda-python>=12,<13" && \

container/Dockerfile.trtllm

@@ -59,7 +59,7 @@ BUILD_CONTEXT=$(dirname "$(readlink -f "$SOURCE_DIR")")
 
 # Base Images
 TRTLLM_BASE_IMAGE=nvcr.io/nvidia/pytorch
-TRTLLM_BASE_IMAGE_TAG=25.05-py3
+TRTLLM_BASE_IMAGE_TAG=25.06-py3
 
 # Important Note: Because of ABI compatibility issues between TensorRT-LLM and NGC PyTorch,
 # we need to build the TensorRT-LLM wheel from source.
@@ -89,7 +89,7 @@ TENSORRTLLM_PIP_WHEEL_DIR="/tmp/trtllm_wheel/"
 # TensorRT-LLM commit to use for building the trtllm wheel if not provided.
 # Important Note: This commit is not used in our CI pipeline. See the CI
 # variables to learn how to run a pipeline with a specific commit.
-DEFAULT_EXPERIMENTAL_TRTLLM_COMMIT="69e9f6d48944b2ae0124ff57aa59340aa4dfae15"
+DEFAULT_EXPERIMENTAL_TRTLLM_COMMIT="a16ba6445c61ed70e7aadfe787d6f316bb422652"
 TRTLLM_COMMIT=""
 TRTLLM_USE_NIXL_KVCACHE_EXPERIMENTAL="0"
 TRTLLM_GIT_URL=""
@@ -98,7 +98,7 @@ TRTLLM_GIT_URL=""
 TENSORRTLLM_INDEX_URL="https://pypi.python.org/simple"
 # TODO: Remove the version specification from here and use the ai-dynamo[trtllm] package.
 # Need to update the Dockerfile.trtllm to use the ai-dynamo[trtllm] package.
-DEFAULT_TENSORRTLLM_PIP_WHEEL="tensorrt-llm==1.0.0rc4"
+DEFAULT_TENSORRTLLM_PIP_WHEEL="tensorrt-llm==1.0.0rc6"
 TENSORRTLLM_PIP_WHEEL=""
 
 

container/build.sh

@@ -67,7 +67,7 @@ If you are using a **GPU**, the following GPU models and architectures are suppo
 | **Build Dependency** | **Version**                                                                      |
 | :------------------- | :------------------------------------------------------------------------------- |
 | **Base Container**   | [25.03](https://catalog.ngc.nvidia.com/orgs/nvidia/containers/cuda-dl-base/tags) |
-| **TensorRT-LLM**     | 1.0.0rc4                                                                         |
+| **TensorRT-LLM**     | 1.0.0rc6                                                                         |
 | **NIXL**             | 0.4.1                                                                            |
 
 > [!Important]

docs/support_matrix.md

@@ -48,8 +48,8 @@ Repository = "https://github.com/ai-dynamo/dynamo.git"
 [project.optional-dependencies]
 trtllm =[
     "uvloop",
-    "tensorrt-llm==1.0.0rc4",
-    "triton==3.3.1",  # locking triton as version 3.4.0 breaks tensorrt-llm 1.0.0rc4
+    "tensorrt-llm==1.0.0rc6",
+    "triton==3.3.1",  # locking triton as version 3.4.0 breaks tensorrt-llm 1.0.0rc6
 ]
 
 vllm = [