diff --git a/docs/source/models/supported-models.md b/docs/source/models/supported-models.md
index 6670e8366b96..f2e450d42582 100644
--- a/docs/source/models/supported-models.md
+++ b/docs/source/models/supported-models.md
@@ -82,6 +82,7 @@ Note: Support for other models may vary. Features marked "N/A" are not applicabl
| Model Architecture/Feature | Overlap Scheduler | CUDA Graph | Chunked Prefill | Torch Sampler | TLLM C++ Sampler | KV Cache Reuse | Logits Post Processor | EPD Disaggregated Serving | Modality |
| ------------------------------------ | ----------------- | ---------- | --------------- | ------------- | ---------------- | -------------- | --------------------- | ------------------------- | --------- |
+| `Exaone4_5_ForConditionalGeneration` | Yes | Yes | Yes | Yes | Yes | Yes | Yes | No | L + I + V |
| `Gemma3ForConditionalGeneration` | Yes | Yes | N/A | Yes | Yes | N/A | Yes | No | L + I |
| `Gemma4ForConditionalGeneration` | Untested | Yes | No | Yes | Untested | No | Untested | No | L + I + A [^9] |
| `HCXVisionForCausalLM` | Yes | Yes | No | Yes | Yes | Yes | Yes | No | L + I |
diff --git a/examples/models/core/exaone/README.md b/examples/models/core/exaone/README.md
index ad5aac648027..5050a4be7822 100644
--- a/examples/models/core/exaone/README.md
+++ b/examples/models/core/exaone/README.md
@@ -14,9 +14,11 @@ This document shows how to build and run [EXAONE](https://huggingface.co/LGAI-EX
- [EXAONE-3.0](#exaone-30)
- [EXAONE-Deep](#exaone-deep)
- [EXAONE-4.0](#exaone-40)
+ - [EXAONE-4.5](#exaone-45)
- [K-EXAONE](#k-exaone)
- [PyTorch flow](#pytorch-flow)
- [Running EXAONE-4.0](#running-exaone-40)
+ - [Running EXAONE-4.5](#running-exaone-45)
- [Running K-EXAONE](#running-k-exaone)
- [MoE Backend Options](#moe-backend-options)
- [PyTorch flow Quantization](#pytorch-flow-quantization)
@@ -45,6 +47,7 @@ This document shows how to build and run [EXAONE](https://huggingface.co/LGAI-EX
* FP16
* BF16
* Tensor Parallel (TP)
+ * Multimodal (EXAONE-4.5 only)
* Expert Parallel (EP) (K-EXAONE only)
* Attention Data Parallel (ADP) (K-EXAONE only)
* Disaggregated Serving
@@ -59,7 +62,7 @@ This document shows how to build and run [EXAONE](https://huggingface.co/LGAI-EX
**Note:**
- **EXAONE-3.0** & **EXAONE-Deep** are supported using the [TRT Flow](#trt-flow).
-- **EXAONE-4.0** & **K-EXAONE** are supported using the [PyTorch flow](#pytorch-flow).
+- **EXAONE-4.0**, **EXAONE-4.5**, & **K-EXAONE** are supported using the [PyTorch flow](#pytorch-flow).
Please refer to the corresponding sections below for usage instructions and examples for each model.
@@ -90,6 +93,17 @@ export HF_MODEL_DIR=hf_models/exaone4
git clone https://huggingface.co/LGAI-EXAONE/EXAONE-4.0-32B $HF_MODEL_DIR
```
+### EXAONE-4.5
+
+EXAONE-4.5 is a multimodal model. It is supported only via the [PyTorch flow](#pytorch-flow).
+
+Download the HuggingFace checkpoint for your EXAONE-4.5 variant from the [LGAI-EXAONE](https://huggingface.co/LGAI-EXAONE) organization. The example below uses `EXAONE-4.5-33B`; replace it with the variant you want to run.
+
+```bash
+export HF_MODEL_DIR=hf_models/exaone4_5
+huggingface-cli download LGAI-EXAONE/EXAONE-4.5-33B --local-dir $HF_MODEL_DIR
+```
+
### K-EXAONE
K-EXAONE is a Mixture of Experts (MoE) model based on the EXAONE architecture. It features a hybrid architecture with both dense and MoE layers, sliding window attention, and supports FP8 and NVFP4 quantization for efficient inference.
@@ -98,7 +112,7 @@ Download the HuggingFace checkpoints of the K-EXAONE model:
```bash
export HF_MODEL_DIR=hf_models/kexaone
-git clone https://huggingface.co/LGAI-EXAONE/K-EXAONE-236B-A23B $HF_MODEL_DIR
+huggingface-cli download LGAI-EXAONE/K-EXAONE-236B-A23B --local-dir $HF_MODEL_DIR
```
## PyTorch flow
@@ -117,6 +131,21 @@ The output will be like:
[2] Prompt: 'The future of AI is', Generated text: ' not just about technology but also about how we choose to use it. We must ensure that AI is developed and deployed in a way that benefits all of humanity, not just a select few. This means prioritizing ethical considerations, transparency, and accountability in AI development. It also means involving diverse stakeholders in the conversation about AI'
```
+### Running EXAONE-4.5
+
+To quickly run EXAONE-4.5 models, you can use [examples/llm-api/quickstart_multimodal.py](../../../llm-api/quickstart_multimodal.py):
+
+```bash
+python ../../../llm-api/quickstart_multimodal.py --model_dir $HF_MODEL_DIR
+```
+
+The output will be like:
+```bash
+[0] Prompt: 'Describe the natural environment in the image.', Generated text: 'Okay, the user asked me to describe the natural environment in the image. But wait, there's no image provided here. Hmm, that's a problem. How can I describe something I can't see?\n\nFirst, I need to check if there's any image attached. The user mentioned "the image," but in the current context, there's no image data. Maybe they forgot'
+[1] Prompt: 'Describe the object and the weather condition in the image.', Generated text: 'Okay, the user asked me to describe the object and the weather condition in the image. But wait, there's no image provided here. Hmm, that's a problem. How can I describe something I can't see?\n\nFirst, I need to check if there's any image attached. The user mentioned "the image," but in the current context, there's no image data. Maybe'
+[2] Prompt: 'Describe the traffic condition on the road in the image.', Generated text: 'Okay, the user is asking me to describe the traffic condition on the road in the image. But wait, there's a problem here—I don't actually see any image. The user mentioned "the image," but in this text-based interface, there's no visual content provided. \n\nHmm, I need to handle this carefully. The user might have forgotten to attach the image or assumed I could see it'
+```
+
### Running K-EXAONE
K-EXAONE is a Mixture of Experts model that benefits from multiple parallelism strategies. You can run it with tensor parallelism (TP), expert parallelism (EP), and attention data parallelism (ADP):
diff --git a/tensorrt_llm/_torch/model_config.py b/tensorrt_llm/_torch/model_config.py
index b96e40d513e3..6b76bec4f0d0 100644
--- a/tensorrt_llm/_torch/model_config.py
+++ b/tensorrt_llm/_torch/model_config.py
@@ -608,8 +608,14 @@ def cached_file(path_or_repo_id, file_name):
return None
# Some checkpoints lack torch_dtype, populate with dtype
- pretrained_config.torch_dtype = getattr(pretrained_config, 'dtype',
- None)
+ dtype = getattr(pretrained_config, 'dtype', None)
+ # For composite VLM configs the dtype lives inside ``text_config``
+ # because the top-level config has no ``dtype`` field.
+ if dtype is None:
+ text_config = getattr(pretrained_config, 'text_config', None)
+ if text_config is not None:
+ dtype = getattr(text_config, 'dtype', None)
+ pretrained_config.torch_dtype = dtype
# Prior to transformers 5, composite configs (e.g. Qwen2_5_VLConfig) delegated attribute
# lookups to their text sub-config, so accesses like `config.vocab_size` /
diff --git a/tensorrt_llm/_torch/models/__init__.py b/tensorrt_llm/_torch/models/__init__.py
index 55177767ac5a..5b6d6a68ada5 100644
--- a/tensorrt_llm/_torch/models/__init__.py
+++ b/tensorrt_llm/_torch/models/__init__.py
@@ -11,6 +11,7 @@
from .modeling_cohere2 import Cohere2ForCausalLM
from .modeling_deepseekv3 import DeepseekV3ForCausalLM
from .modeling_exaone4 import Exaone4ForCausalLM
+from .modeling_exaone4_5 import Exaone4_5_ForConditionalGeneration
from .modeling_exaone_moe import ExaoneMoeForCausalLM
from .modeling_gemma3 import Gemma3ForCausalLM
from .modeling_gemma3vl import Gemma3VLM
@@ -54,6 +55,7 @@
"CLIPVisionModel",
"DeepseekV3ForCausalLM",
"Exaone4ForCausalLM",
+ "Exaone4_5_ForConditionalGeneration",
"ExaoneMoeForCausalLM",
"Gemma3ForCausalLM",
"Gemma3VLM",
diff --git a/tensorrt_llm/_torch/models/checkpoints/hf/exaone4_5_weight_mapper.py b/tensorrt_llm/_torch/models/checkpoints/hf/exaone4_5_weight_mapper.py
new file mode 100644
index 000000000000..7e315acced94
--- /dev/null
+++ b/tensorrt_llm/_torch/models/checkpoints/hf/exaone4_5_weight_mapper.py
@@ -0,0 +1,38 @@
+# SPDX-FileCopyrightText: Copyright (c) 2026 NVIDIA CORPORATION & AFFILIATES. All rights reserved.
+# SPDX-License-Identifier: Apache-2.0
+#
+# Licensed under the Apache License, Version 2.0 (the "License");
+# you may not use this file except in compliance with the License.
+# You may obtain a copy of the License at
+#
+# http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing, software
+# distributed under the License is distributed on an "AS IS" BASIS,
+# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+# See the License for the specific language governing permissions and
+# limitations under the License.
+
+from tensorrt_llm._torch.models.checkpoints.base_weight_loader import ConsumableWeightsDict
+from tensorrt_llm._torch.models.checkpoints.hf.weight_mapper import HfWeightMapper
+from tensorrt_llm._torch.models.modeling_utils import register_mapper
+
+
+@register_mapper("HF", "Exaone4_5_ForConditionalGeneration")
+class Exaone4_5HfWeightMapper(HfWeightMapper):
+ def preprocess_weights(self, weights: dict):
+ """Rename HF checkpoint prefixes; supports plain dict and ConsumableWeightsDict."""
+ is_consumable = isinstance(weights, ConsumableWeightsDict)
+ renamed = {}
+ for key, value in weights.items():
+ if key.startswith("model.visual."):
+ new_key = key.replace("model.visual.", "visual.")
+ renamed[new_key] = value
+ elif key.startswith("model.language_model."):
+ new_key = key.replace("model.language_model.", "model.")
+ renamed[new_key] = value
+ else:
+ renamed[key] = value
+ if is_consumable:
+ return ConsumableWeightsDict(renamed)
+ return renamed
diff --git a/tensorrt_llm/_torch/models/modeling_exaone4_5.py b/tensorrt_llm/_torch/models/modeling_exaone4_5.py
new file mode 100644
index 000000000000..1506c4fce23e
--- /dev/null
+++ b/tensorrt_llm/_torch/models/modeling_exaone4_5.py
@@ -0,0 +1,267 @@
+# SPDX-License-Identifier: Apache-2.0
+# Copyright (c) 2026, NVIDIA CORPORATION & AFFILIATES. All rights reserved.
+
+import copy
+from typing import List, Optional, Tuple, Union
+
+import torch
+from transformers import AutoConfig, AutoTokenizer, PretrainedConfig, PreTrainedModel
+from transformers.models.auto import CONFIG_MAPPING
+
+from tensorrt_llm._torch.models.checkpoints.base_weight_mapper import BaseWeightMapper
+from tensorrt_llm._torch.models.modeling_multimodal_utils import _is_disagg
+
+from ...inputs import (
+ ContentFormat,
+ ExtraProcessedInputs,
+ MultimodalPlaceholderMetadata,
+ MultimodalPlaceholderPlacement,
+ TextPrompt,
+ register_input_processor,
+)
+from ...sampling_params import SamplingParams
+from ..attention_backend import AttentionMetadata
+from .checkpoints.hf.exaone4_5_weight_mapper import Exaone4_5HfWeightMapper
+from .modeling_auto import AutoModelForCausalLM
+from .modeling_multimodal_utils import (
+ find_input_mm_embeds,
+ fuse_input_embeds,
+ get_multimodal_embeddings,
+)
+from .modeling_qwen2vl import (
+ Qwen2_5_VisionModel,
+ Qwen2VisionModelBase,
+ Qwen2VLInputProcessorBase,
+ Qwen2VLModelBase,
+)
+from .modeling_utils import ModelConfig, register_auto_model, register_vision_encoder
+
+# transformers >= 5.8 ships native Exaone4.5 configs with the same
+# sub-config-instantiation logic we'd otherwise re-implement here. Prefer
+# the HF classes when available and only register local fallbacks on older
+# releases (where the modules don't exist at all).
+try:
+ from transformers.models.exaone4_5.configuration_exaone4_5 import (
+ Exaone4_5_Config as Exaone4_5Config,
+ )
+ from transformers.models.exaone4_5.configuration_exaone4_5 import Exaone4_5_VisionConfig
+except ImportError:
+
+ class Exaone4_5_VisionConfig(PretrainedConfig):
+ model_type = "exaone4_5_vision"
+ base_config_key = "vision_config"
+
+ AutoConfig.register(Exaone4_5_VisionConfig.model_type, Exaone4_5_VisionConfig, exist_ok=True)
+
+ class Exaone4_5Config(PretrainedConfig):
+ """VLM config: nested `text_config` / `vision_config` from JSON become real sub-configs."""
+
+ model_type = "exaone4_5"
+
+ def __init__(
+ self,
+ text_config: Optional[Union[PretrainedConfig, dict]] = None,
+ vision_config: Optional[Union[PretrainedConfig, dict]] = None,
+ **kwargs,
+ ):
+ if isinstance(text_config, dict):
+ text_config = copy.deepcopy(text_config)
+ model_type = text_config.get("model_type", "exaone4")
+ # BC: EXAONE 4.5 first released with the text model type
+ # as `exaone4_5_text`, later renamed to `exaone4`.
+ if model_type == "exaone4_5_text":
+ model_type = "exaone4"
+ text_config["model_type"] = model_type
+ text_config = CONFIG_MAPPING[model_type](**text_config)
+ if isinstance(vision_config, dict):
+ vision_config = copy.deepcopy(vision_config)
+ model_type = vision_config.get("model_type", "exaone4_5_vision")
+ vision_config["model_type"] = model_type
+ vision_config = CONFIG_MAPPING[model_type](**vision_config)
+ super().__init__(text_config=text_config, vision_config=vision_config, **kwargs)
+
+ AutoConfig.register(Exaone4_5Config.model_type, Exaone4_5Config, exist_ok=True)
+
+
+class Exaone4_5InputProcessor(Qwen2VLInputProcessorBase):
+ def __init__(
+ self,
+ model_path: str,
+ config: PretrainedConfig,
+ tokenizer: AutoTokenizer,
+ trust_remote_code: bool = True,
+ **kwargs,
+ ):
+ super().__init__(
+ model_path=model_path,
+ config=config,
+ tokenizer=tokenizer,
+ trust_remote_code=trust_remote_code,
+ **kwargs,
+ )
+
+ def get_mrope_config(self, *args, **kwargs):
+ # EXAONE4.5-VL does not consume mrope metadata, so the input processor
+ # intentionally skips populating `multimodal_data['mrope_config']`.
+ # Calling the base implementation would silently produce data that the
+ # model never reads — fail loudly instead.
+ raise NotImplementedError(
+ "Exaone4_5InputProcessor does not produce mrope_config; "
+ "EXAONE4.5-VL does not use M-RoPE."
+ )
+
+ @torch.inference_mode()
+ def __call__(
+ self,
+ inputs: TextPrompt,
+ sampling_params: SamplingParams,
+ ) -> Tuple[List[int], Optional[ExtraProcessedInputs]]:
+ text_prompt, mm_data, mm_processor_kwargs = (
+ inputs.get("prompt"),
+ inputs.get("multi_modal_data", {}),
+ inputs.get("mm_processor_kwargs", {}),
+ )
+ processed_inputs = self._preprocess(text_prompt, mm_data, mm_processor_kwargs)
+
+ multimodal_data = {}
+ pixel_values = processed_inputs.get("pixel_values", None)
+ if pixel_values is not None:
+ multimodal_data["image"] = {
+ "pixel_values": pixel_values.to(self.dtype),
+ "image_grid_thw": processed_inputs.get("image_grid_thw"),
+ }
+
+ pixel_values_videos = processed_inputs.get("pixel_values_videos", None)
+ if pixel_values_videos is not None:
+ multimodal_data["video"] = {
+ "pixel_values_videos": pixel_values_videos.to(self.dtype),
+ "video_grid_thw": processed_inputs.get("video_grid_thw"),
+ }
+ fused_input_ids = processed_inputs["input_ids"][0]
+ if mm_data:
+ fused_input_ids = self._postprocess(fused_input_ids)
+
+ return fused_input_ids.to(torch.int32).tolist(), {
+ "multimodal_data": multimodal_data,
+ }
+
+
+class Exaone4_5_VisionModel(Qwen2VisionModelBase):
+ pass
+
+
+@register_vision_encoder(Exaone4_5_VisionModel, vlm_base_model=Qwen2_5_VisionModel)
+@register_auto_model("Exaone4_5_ForConditionalGeneration")
+@register_input_processor(
+ Exaone4_5InputProcessor,
+ model_type="exaone4_5",
+ placeholder_metadata=MultimodalPlaceholderMetadata(
+ placeholder_map={
+ "image": "<|image_pad|>",
+ "video": "<|video_pad|>",
+ },
+ placeholder_placement=MultimodalPlaceholderPlacement.BEFORE_TEXT,
+ content_format=ContentFormat.STRING,
+ ),
+)
+class Exaone4_5_ForConditionalGeneration(Qwen2VLModelBase):
+ def __init__(
+ self,
+ model_config: ModelConfig[PretrainedConfig],
+ *args,
+ **kwargs,
+ ) -> None:
+ self.original_arch = model_config.pretrained_config.architectures[0]
+
+ config = model_config.pretrained_config
+
+ self._supports_sdpa = True
+ # Deliberately bypass `Qwen2VLModelBase.__init__` (the natural `super()`
+ # call) because, unlike Qwen2/2.5-VL, EXAONE 4.5:
+ # 1. uses standard llama3-style RoPE — not mrope — so we must not run
+ # the mrope-specific setup (`rope_scaling['type'] = 'mrope'`,
+ # `init_mrope_embedding`, `disable_fuse_rope` plumbing);
+ # 2. ships its own causal LM (`Exaone4ForCausalLM`, read from
+ # `text_config.architectures`) and must not be forced to
+ # `["Qwen2ForCausalLM"]`;
+ # We still need the `PreTrainedModel` machinery (config wiring,
+ # `_supports_*` flags, weight-loading hooks), so we call it directly.
+ PreTrainedModel.__init__(self, config)
+
+ self.model_config = model_config
+ self.config = model_config.pretrained_config
+
+ if model_config.attn_backend != "TRTLLM":
+ raise ValueError("Exaone4.5 only supports TRTLLM backend")
+
+ llm_model_config = copy.deepcopy(model_config)
+ llm_model_config.pretrained_config = llm_model_config.pretrained_config.text_config
+ self.llm = AutoModelForCausalLM.from_config(llm_model_config)
+
+ if not _is_disagg():
+ mm_encoder_config = copy.deepcopy(model_config)
+ self.mm_encoder = Exaone4_5_VisionModel(mm_encoder_config, Qwen2_5_VisionModel)
+ else:
+ self.mm_encoder = None
+
+ def infer_max_seq_len(self) -> int:
+ return self.llm.infer_max_seq_len()
+
+ @property
+ def multimodal_data_device_paths(self) -> List[str]:
+ return [
+ "image.pixel_values",
+ "video.pixel_values_videos",
+ "multimodal_embedding",
+ ]
+
+ @torch.inference_mode()
+ def forward(
+ self,
+ attn_metadata: AttentionMetadata,
+ input_ids: Optional[torch.IntTensor] = None,
+ position_ids: Optional[torch.IntTensor] = None,
+ input_embeds: Optional[torch.Tensor] = None,
+ return_context_logits: bool = False,
+ **kwargs,
+ ) -> torch.Tensor:
+ multimodal_params = kwargs.get("multimodal_params", [])
+ mm_embeds = []
+
+ mm_multimodal_params = self._get_requests_with_mm_data(multimodal_params)
+
+ if len(mm_multimodal_params) > 0:
+ if not _is_disagg():
+ mm_embeds = get_multimodal_embeddings(
+ encoder_forward_fn=self.mm_encoder.forward,
+ multimodal_params=mm_multimodal_params,
+ )
+ else:
+ raise NotImplementedError(
+ "Exaone4.5-VL does not support disaggregated inference yet. "
+ "Unset TLLM_MULTIMODAL_DISAGGREGATED or set it to '0'."
+ )
+ mm_embeds = find_input_mm_embeds(mm_embeds, mm_multimodal_params)
+
+ input_ids, input_embeds = fuse_input_embeds(
+ self.llm.model.embed_tokens,
+ input_ids,
+ mm_embeds,
+ **kwargs,
+ )
+
+ output_prob = self.llm.forward(
+ attn_metadata=attn_metadata,
+ input_ids=input_ids,
+ position_ids=position_ids,
+ inputs_embeds=input_embeds,
+ return_context_logits=return_context_logits,
+ )
+ return output_prob
+
+ def load_weights(self, weights, weight_mapper: BaseWeightMapper):
+ assert isinstance(weight_mapper, Exaone4_5HfWeightMapper)
+ weights = weight_mapper.preprocess_weights(weights)
+ if not _is_disagg():
+ self.mm_encoder.load_weights(weights)
+ self.llm.load_weights(weights, weight_mapper)
diff --git a/tensorrt_llm/_torch/models/modeling_qwen2vl.py b/tensorrt_llm/_torch/models/modeling_qwen2vl.py
index 99af7579810f..08523ffcecc5 100644
--- a/tensorrt_llm/_torch/models/modeling_qwen2vl.py
+++ b/tensorrt_llm/_torch/models/modeling_qwen2vl.py
@@ -1,5 +1,9 @@
+# SPDX-License-Identifier: Apache-2.0
+# Copyright (c) 2026, NVIDIA CORPORATION & AFFILIATES. All rights reserved.
+
import copy
import re
+from functools import lru_cache
from typing import Any, Dict, List, Optional, Tuple, Union
import torch
@@ -8,9 +12,7 @@
from transformers import (AutoProcessor, AutoTokenizer, PretrainedConfig,
PreTrainedModel)
from transformers.models.qwen2_5_vl.modeling_qwen2_5_vl import (
- Qwen2_5_VisionPatchEmbed, Qwen2_5_VisionRotaryEmbedding,
- Qwen2_5_VisionTransformerPretrainedModel, Qwen2_5_VLVisionBlock,
- apply_rotary_pos_emb_vision)
+ Qwen2_5_VisionPatchEmbed, Qwen2_5_VisionTransformerPretrainedModel)
from transformers.models.qwen2_vl.modeling_qwen2_vl import \
Qwen2VisionTransformerPretrainedModel
@@ -39,8 +41,17 @@
from ..attention_backend import AttentionMetadata
from ..attention_backend.interface import PositionalEmbeddingParams, RopeParams
from ..attention_backend.utils import get_attention_backend
+from ..flashinfer_utils import IS_FLASHINFER_AVAILABLE
+
+# Guarded module-level import: `flashinfer_apply_rope_with_cos_sin_cache_inplace`
+# is only exported from `custom_ops` when FlashInfer is installed (see
+# `_torch/custom_ops/__init__.py`). Unconditional import would break loading
+# this module in FlashInfer-less environments; importing inside the guard mirrors
+# the pattern used in `custom_ops` itself.
+if IS_FLASHINFER_AVAILABLE:
+ from ..custom_ops import flashinfer_apply_rope_with_cos_sin_cache_inplace
from ..modules.gated_mlp import GatedMLP
-from ..modules.rotary_embedding import MRotaryEmbedding
+from ..modules.rotary_embedding import MRotaryEmbedding, RotaryEmbedding
from .modeling_auto import AutoModelForCausalLM
from .modeling_multimodal_utils import (bypass_processor_output_validation,
find_input_mm_embeds, fuse_input_embeds,
@@ -80,6 +91,10 @@ def __init__(self,
self.temporal_patch_size = getattr(self.config.vision_config,
'temporal_patch_size', 1)
+ def get_vocab_size(self) -> int:
+ """Return the vocab size of the model."""
+ return self.config.text_config.vocab_size
+
@property
def config(self) -> PretrainedConfig:
return self._config
@@ -305,10 +320,17 @@ def _preprocess(self, text: Dict[str, any], mm_data: Dict[str, any],
**mm_processor_kwargs)
def _postprocess(self, input_ids: torch.IntTensor) -> torch.IntTensor:
- masks = (input_ids == self.config.image_token_id) | (
- input_ids == self.config.vision_token_id) | (
- input_ids == self.config.video_token_id)
- input_ids[masks] = self.tllm_multimodal_token_id
+ token_ids = [
+ tid
+ for attr in ("image_token_id", "vision_token_id", "video_token_id")
+ if (tid := getattr(self.config, attr, None)) is not None
+ ]
+ if token_ids:
+ ids_tensor = torch.tensor(token_ids,
+ device=input_ids.device,
+ dtype=input_ids.dtype)
+ input_ids[torch.isin(input_ids,
+ ids_tensor)] = self.tllm_multimodal_token_id
return input_ids
def get_mrope_config(
@@ -405,6 +427,49 @@ def __init__(self, model_config: ModelConfig[PretrainedConfig],
raise NotImplementedError(
f"Model class {model_class} not implemented")
+ def _split_fused_vision_qkv_tensor(
+ self, tensor: torch.Tensor
+ ) -> Tuple[torch.Tensor, torch.Tensor, torch.Tensor]:
+ """Split HF fused `attn.qkv` along output dim (dim 0 for Linear).
+
+ Qwen2.5-VL vision is **MHA** (`num_key_value_heads == num_heads`): Q, K, and V
+ each occupy `num_heads * head_dim` — three equal blocks.
+
+ EXAONE-4.5 vision is **GQA**: Q uses `num_heads * head_dim`, K and V each use
+ `num_key_value_heads * head_dim` (asymmetric split).
+ """
+ cfg = self.config
+ num_heads = cfg.num_heads
+ num_kv_heads = getattr(cfg, "num_key_value_heads", None)
+ if num_kv_heads is None:
+ num_kv_heads = num_heads
+ head_dim, rem = divmod(cfg.hidden_size, num_heads)
+ if rem != 0:
+ raise ValueError(
+ f"vision hidden_size {cfg.hidden_size} not divisible by "
+ f"num_heads {num_heads}")
+ q_dim = num_heads * head_dim
+ kv_dim = num_kv_heads * head_dim
+ # Fused Linear out_features = Q + K + V along dim 0 of the weight (or bias).
+ fused_out_features = q_dim + 2 * kv_dim
+ leading_dim = tensor.shape[0]
+ if leading_dim == fused_out_features:
+ # GQA (e.g. EXAONE-4.5 vision) or MHA with fused length matching config.
+ return (tensor[:q_dim], tensor[q_dim:q_dim + kv_dim],
+ tensor[q_dim + kv_dim:])
+ if num_kv_heads == num_heads and leading_dim % 3 == 0:
+ # MHA (e.g. Qwen2.5-VL vision): three equal Q/K/V blocks; used if fused
+ # leading dim is a triple split but does not match `fused_out_features`.
+ dim_shape = leading_dim // 3
+ return (tensor[:dim_shape], tensor[dim_shape:2 * dim_shape],
+ tensor[2 * dim_shape:])
+ raise ValueError(
+ f"Fused vision qkv leading dim is {leading_dim}, "
+ f"want {fused_out_features} from config (q_dim={q_dim}, kv_dim={kv_dim}) "
+ f"or for MHA a length divisible by 3; "
+ f"num_heads={num_heads}, num_key_value_heads={num_kv_heads}, "
+ f"head_dim={head_dim}.")
+
def load_weights(self, weights: Dict):
visual_weights = filter_weights("visual", weights)
converted_weights = dict()
@@ -422,11 +487,11 @@ def load_weights(self, weights: Dict):
q_name = f"{prefix}attn.q_proj.{suffix}"
k_name = f"{prefix}attn.k_proj.{suffix}"
v_name = f"{prefix}attn.v_proj.{suffix}"
- dim_shape = visual_weights[name].shape[0] // 3
- converted_weights[q_name] = visual_weights[name][:dim_shape]
- converted_weights[k_name] = visual_weights[name][dim_shape:2 *
- dim_shape]
- converted_weights[v_name] = visual_weights[name][2 * dim_shape:]
+ q_part, k_part, v_part = self._split_fused_vision_qkv_tensor(
+ visual_weights[name])
+ converted_weights[q_name] = q_part
+ converted_weights[k_name] = k_part
+ converted_weights[v_name] = v_part
else:
converted_weights[name] = visual_weights[name]
pattern_mapping = {
@@ -488,6 +553,7 @@ def _parse_and_batch_multimodal_data(
return mm_content_dict, mm_extra_data
+ @nvtx_range("Qwen2VisionModelBase forward()")
@torch.inference_mode()
def forward(self, multimodal_params: List[MultimodalParams]):
@@ -518,12 +584,22 @@ def __init__(self,
reduce_output: bool = True) -> None:
config = model_config.pretrained_config.vision_config
+ # Composite VLM configs (transformers 5.x strict mode) keep
+ # `max_position_embeddings` inside `text_config` rather than at
+ # the top level; fall back to it when the parent doesn't expose it.
+ max_position_embeddings = getattr(model_config.pretrained_config,
+ "max_position_embeddings", None)
+ if max_position_embeddings is None:
+ text_config = getattr(model_config.pretrained_config, "text_config",
+ None)
+ max_position_embeddings = getattr(text_config,
+ "max_position_embeddings", None)
super().__init__(
hidden_size=config.hidden_size,
num_attention_heads=config.num_heads,
- num_key_value_heads=config.num_heads,
- max_position_embeddings=model_config.pretrained_config.text_config.
- max_position_embeddings,
+ num_key_value_heads=getattr(config, "num_key_value_heads", None)
+ or config.num_heads,
+ max_position_embeddings=max_position_embeddings,
bias=True,
pos_embd_params=None,
rope_fusion=False,
@@ -537,40 +613,79 @@ def __init__(self,
head_dim=config.hidden_size // config.num_heads,
)
+ def apply_rope(self,
+ q: torch.Tensor,
+ k: Optional[torch.Tensor],
+ v: Optional[torch.Tensor],
+ position_ids: Optional[torch.IntTensor] = None,
+ position_embeddings: Optional[Tuple[torch.Tensor,
+ torch.Tensor]] = None):
+ seq_len, _ = q.size()
+ cos, sin = position_embeddings
+
+ # FlashInfer fused RoPE assumes head_size is a multiple of 64 (see
+ # auto_deploy custom op rope docs / flashinfer tests). Qwen2.5-VL vision
+ # uses head_dim=80 (e.g. 1280 hidden / 16 heads), so use PyTorch RoPE.
+ if IS_FLASHINFER_AVAILABLE and self.head_dim % 64 == 0 and position_ids is not None:
+ try:
+ cos_sin_cache = torch.cat([cos, sin], dim=-1).contiguous()
+ flashinfer_apply_rope_with_cos_sin_cache_inplace(
+ position_ids,
+ q,
+ k,
+ self.head_dim,
+ cos_sin_cache,
+ is_neox=True,
+ )
+ return q, k, v
+ except RuntimeError as err:
+ logger.warning(
+ "Qwen2.5-VL vision RoPE: FlashInfer failed (%s); "
+ "falling back to PyTorch RotaryEmbedding.apply_rotary_pos_emb.",
+ err,
+ )
+
+ cos = cos.to(dtype=q.dtype)
+ sin = sin.to(dtype=q.dtype)
+ q = q.view(seq_len, -1, self.head_dim)
+ k = k.view(seq_len, -1, self.head_dim)
+ v = v.view(seq_len, -1, self.head_dim)
+ q = RotaryEmbedding.apply_rotary_pos_emb(q, cos, sin)
+ k = RotaryEmbedding.apply_rotary_pos_emb(k, cos, sin)
+ q, k, v = q.reshape(seq_len, -1), k.reshape(seq_len,
+ -1), v.reshape(seq_len, -1)
+ return q, k, v
+
def forward(
self,
hidden_states: torch.Tensor,
attn_metadata: AttentionMetadata,
- position_embeddings: Optional[Tuple[torch.Tensor, torch.Tensor]],
+ position_ids: Optional[torch.IntTensor] = None,
+ position_embeddings: Optional[Tuple[torch.Tensor, torch.Tensor]] = None,
**kwargs,
) -> torch.Tensor:
- # NOTE: Need separate Attention forward() for Qwen2.5-VL for multiple reasons
- # 1. We don't have the route for handing over position_embeddings to the Attention forward()
- # 2. Could not override the apply_rope() as we don't have the position_ids in the Vision Attention's rotary embedding.
- # (TODO: yechank-nvidia) Make OOTB path more modular and reusable for Attention's Rotary Embedding.
+ # NOTE: Qwen2.5-VL vision attention needs a custom forward: the generic
+ # Attention path does not accept precomputed (cos, sin) position_embeddings,
+ # and vision RoPE may use FlashInfer with explicit position_ids.
qkv = self.qkv_proj(hidden_states)
q, k, v = qkv, None, None
q, k, v = self.split_qkv(q, k, v)
- seq_length = hidden_states.shape[0]
- q, k, v = (qkv.reshape(seq_length, 3, self.num_heads,
- -1).permute(1, 0, 2, 3).unbind(0))
- cos, sin = position_embeddings
- q, k = apply_rotary_pos_emb_vision(q, k, cos, sin)
- q, k, v = q.reshape(seq_length,
- -1), k.reshape(seq_length,
- -1), v.reshape(seq_length, -1)
+ q, k, v = self.apply_rope(q, k, v, position_ids, position_embeddings)
q, k, v = self.convert_qkv(q, k, v)
- output = self.forward_impl(q=q,
- k=k,
- v=v,
- attn_metadata=attn_metadata,
- attention_mask=PredefinedAttentionMask.FULL,
- attention_window_size=None,
- attention_mask_data=None,
- mrope_config=None,
- attention_sinks=None)
+
+ output = self.forward_impl(
+ q=q,
+ k=k,
+ v=v,
+ attn_metadata=attn_metadata,
+ attention_mask=PredefinedAttentionMask.FULL,
+ attention_window_size=None,
+ attention_mask_data=None,
+ mrope_config=None,
+ attention_sinks=None,
+ )
attn_output = self.o_proj(output, layer_idx=self.layer_idx)
return attn_output
@@ -613,7 +728,7 @@ def forward(
self,
hidden_states: torch.Tensor,
attn_metadata: AttentionMetadata,
- rotary_pos_emb: Optional[torch.Tensor] = None,
+ position_ids: Optional[torch.IntTensor] = None,
position_embeddings: Optional[Tuple[torch.Tensor, torch.Tensor]] = None,
**kwargs,
) -> torch.Tensor:
@@ -623,7 +738,7 @@ def forward(
hidden_states = residual + self.attn(
hidden_states=hidden_states,
attn_metadata=attn_metadata,
- rotary_pos_emb=rotary_pos_emb,
+ position_ids=position_ids,
position_embeddings=position_embeddings,
**kwargs,
)
@@ -694,8 +809,20 @@ def __init__(self, model_config: ModelConfig[PretrainedConfig]):
embed_dim=self.config.hidden_size,
)
- head_dim = self.config.hidden_size // self.config.num_heads
- self.rotary_pos_emb = Qwen2_5_VisionRotaryEmbedding(head_dim // 2)
+ text_config = getattr(model_config.pretrained_config, "text_config",
+ model_config.pretrained_config)
+ self.config.max_position_embeddings = text_config.max_position_embeddings
+ self.config.partial_rotary_factor = 0.5
+ self.head_dim = self.config.hidden_size // self.config.num_heads
+ self.pos_embd_params = PositionalEmbeddingParams(
+ type=PositionEmbeddingType.rope_gpt_neox,
+ rope=RopeParams.from_config(self.config),
+ )
+ self.rotary_pos_emb = RotaryEmbedding(
+ self.pos_embd_params.rope,
+ head_dim=self.head_dim,
+ is_neox=self.pos_embd_params.is_neox,
+ )
self.blocks = torch.nn.ModuleList([
Qwen2_5_VLVisionBlock(model_config, layer_idx=layer_idx)
@@ -716,136 +843,187 @@ def __init__(self, model_config: ModelConfig[PretrainedConfig]):
kv_cache_manager=None,
)
- def rot_pos_emb(self, grid_thw):
- pos_ids = []
- for t, h, w in grid_thw:
- hpos_ids = torch.arange(h).unsqueeze(1).expand(-1, w)
- hpos_ids = hpos_ids.reshape(
- h // self.spatial_merge_size,
- self.spatial_merge_size,
- w // self.spatial_merge_size,
- self.spatial_merge_size,
- )
- hpos_ids = hpos_ids.permute(0, 2, 1, 3)
- hpos_ids = hpos_ids.flatten()
-
- wpos_ids = torch.arange(w).unsqueeze(0).expand(h, -1)
- wpos_ids = wpos_ids.reshape(
- h // self.spatial_merge_size,
- self.spatial_merge_size,
- w // self.spatial_merge_size,
- self.spatial_merge_size,
- )
- wpos_ids = wpos_ids.permute(0, 2, 1, 3)
- wpos_ids = wpos_ids.flatten()
- pos_ids.append(
- torch.stack([hpos_ids, wpos_ids], dim=-1).repeat(t, 1))
- pos_ids = torch.cat(pos_ids, dim=0)
- max_grid_size = grid_thw[:, 1:].max()
- rotary_pos_emb_full = self.rotary_pos_emb(max_grid_size)
- rotary_pos_emb = rotary_pos_emb_full[pos_ids].flatten(1)
- return rotary_pos_emb
-
- def get_window_index(self, grid_thw):
- window_index: List[torch.Tensor] = []
- seq_lens = []
+ def get_rotary_pos_emb_window_data(
+ self, grid_rows: List[List[int]]
+ ) -> Tuple[List[torch.Tensor], List[torch.Tensor], List[torch.Tensor],
+ List[int]]:
window_index_id = 0
- vit_merger_window_size = self.window_size // self.spatial_merge_size // self.patch_size
+ rotary_pos_emb_cos: List[torch.Tensor] = []
+ rotary_pos_emb_sin: List[torch.Tensor] = []
+ window_indices: List[torch.Tensor] = []
+ window_seq_lens: List[int] = []
+ for row in grid_rows:
+ t, h, w = int(row[0]), int(row[1]), int(row[2])
+ llm_h = h // self.spatial_merge_size
+ llm_w = w // self.spatial_merge_size
+ (cos_thw, sin_thw, window_index_thw,
+ window_seq_lens_thw) = self.get_rope_and_window_index_by_thw(
+ t, h, w)
+
+ window_indices.append(window_index_thw + window_index_id)
+ window_index_id += t * llm_h * llm_w
+
+ rotary_pos_emb_cos.append(cos_thw)
+ rotary_pos_emb_sin.append(sin_thw)
+
+ window_seq_lens.extend(window_seq_lens_thw)
+
+ return (rotary_pos_emb_cos, rotary_pos_emb_sin, window_indices,
+ window_seq_lens)
+
+ def get_window_index_by_thw(self, grid_t: int, grid_h: int,
+ grid_w: int) -> Tuple[torch.Tensor, List[int]]:
+ vit_merger_window_size = (self.window_size // self.spatial_merge_size //
+ self.patch_size)
+ llm_grid_h = grid_h // self.spatial_merge_size
+ llm_grid_w = grid_w // self.spatial_merge_size
+ index = torch.arange(grid_t * llm_grid_h * llm_grid_w,
+ dtype=torch.long).reshape(grid_t, llm_grid_h,
+ llm_grid_w)
+ pad_h = vit_merger_window_size - llm_grid_h % vit_merger_window_size
+ pad_w = vit_merger_window_size - llm_grid_w % vit_merger_window_size
+ num_windows_h = (llm_grid_h + pad_h) // vit_merger_window_size
+ num_windows_w = (llm_grid_w + pad_w) // vit_merger_window_size
+ index_padded = F.pad(index, (0, pad_w, 0, pad_h), "constant", PAD_INDEX)
+ index_padded = index_padded.reshape(
+ grid_t,
+ num_windows_h,
+ vit_merger_window_size,
+ num_windows_w,
+ vit_merger_window_size,
+ )
+ index_padded = index_padded.permute(0, 1, 3, 2, 4).reshape(
+ grid_t,
+ num_windows_h * num_windows_w,
+ vit_merger_window_size,
+ vit_merger_window_size,
+ )
+ seqlens = (index_padded != PAD_INDEX).sum([2, 3]).reshape(-1)
+ index_padded = index_padded.reshape(-1)
+ index_new = index_padded[index_padded != PAD_INDEX]
+ seqlens = seqlens * self.spatial_merge_unit
+ return index_new, seqlens.tolist()
+
+ @lru_cache(maxsize=1024) # noqa: B019
+ def get_rope_and_window_index_by_thw(
+ self, t: int, h: int, w: int
+ ) -> Tuple[torch.Tensor, torch.Tensor, torch.Tensor, Tuple[int, ...]]:
+ """CPU (cos, sin, window_idx, seqlens) in window order; cached per `(t, h, w)`."""
+ hpos_ids = torch.arange(h, dtype=torch.long).unsqueeze(1).expand(-1, w)
+ wpos_ids = torch.arange(w, dtype=torch.long).unsqueeze(0).expand(h, -1)
+ hpos_ids = (hpos_ids.reshape(h // self.spatial_merge_size,
+ self.spatial_merge_size,
+ w // self.spatial_merge_size,
+ self.spatial_merge_size).permute(
+ 0, 2, 1, 3).flatten())
+ wpos_ids = (wpos_ids.reshape(h // self.spatial_merge_size,
+ self.spatial_merge_size,
+ w // self.spatial_merge_size,
+ self.spatial_merge_size).permute(
+ 0, 2, 1, 3).flatten())
+ pos_ids = torch.stack([hpos_ids, wpos_ids], dim=-1).repeat(t, 1)
+ max_grid_size = max(h, w)
+ cos_sin = self.rotary_pos_emb.rotary_cos_sin[:max_grid_size]
+ cos, sin = cos_sin[:, 0, :], cos_sin[:, 1, :]
+ cos_flattened = cos[pos_ids].flatten(1)
+ sin_flattened = sin[pos_ids].flatten(1)
+
+ cos_thw = cos_flattened.reshape(
+ cos_flattened.shape[0] // self.spatial_merge_unit,
+ self.spatial_merge_unit,
+ -1,
+ )
+ sin_thw = sin_flattened.reshape(
+ sin_flattened.shape[0] // self.spatial_merge_unit,
+ self.spatial_merge_unit,
+ -1,
+ )
- for grid_t, grid_h, grid_w in grid_thw:
- llm_grid_h, llm_grid_w = (
- grid_h // self.spatial_merge_size,
- grid_w // self.spatial_merge_size,
- )
- index = torch.arange(grid_t * llm_grid_h * llm_grid_w).reshape(
- grid_t, llm_grid_h, llm_grid_w)
- pad_h = vit_merger_window_size - llm_grid_h % vit_merger_window_size
- pad_w = vit_merger_window_size - llm_grid_w % vit_merger_window_size
- num_windows_h = (llm_grid_h + pad_h) // vit_merger_window_size
- num_windows_w = (llm_grid_w + pad_w) // vit_merger_window_size
- index_padded = F.pad(index, (0, pad_w, 0, pad_h), "constant",
- PAD_INDEX)
- index_padded = index_padded.reshape(
- grid_t,
- num_windows_h,
- vit_merger_window_size,
- num_windows_w,
- vit_merger_window_size,
- )
- index_padded = index_padded.permute(0, 1, 3, 2, 4).reshape(
- grid_t,
- num_windows_h * num_windows_w,
- vit_merger_window_size,
- vit_merger_window_size,
- )
- seqlens = (index_padded != PAD_INDEX).sum([2, 3]).reshape(-1)
- index_padded = index_padded.reshape(-1)
- index_new = index_padded[index_padded != PAD_INDEX]
- window_index.append(index_new + window_index_id)
- seqlens = seqlens * self.spatial_merge_unit
- seq_lens.extend(seqlens.tolist())
- window_index_id += (grid_t * llm_grid_h * llm_grid_w).item()
- window_index = torch.cat(window_index, dim=0)
-
- return window_index, seq_lens
-
- def prepare_attn_metadata(self, seq_lens, attn_metadata: AttentionMetadata):
- batch_size = 1 # NOTE: Qwen2/2.5-VL concats all the pixel_values into a single tensor, so batch_size is 1
- prompt_lens = seq_lens
- seq_lens = torch.tensor(seq_lens,
- dtype=torch.int,
- pin_memory=prefer_pinned())
+ window_index_thw, seq_lens_thw = self.get_window_index_by_thw(t, h, w)
+
+ cos_thw = cos_thw[window_index_thw, :, :].reshape(-1, cos_thw.shape[-1])
+ sin_thw = sin_thw[window_index_thw, :, :].reshape(-1, sin_thw.shape[-1])
+
+ return cos_thw, sin_thw, window_index_thw, tuple(seq_lens_thw)
+
+ def prepare_attn_metadata(self, batch_size: int, seq_lens: List[int],
+ attn_metadata: AttentionMetadata):
+ batch_size = len(seq_lens)
+ seq_lens_torch = torch.tensor(seq_lens,
+ dtype=torch.int,
+ pin_memory=prefer_pinned())
request_ids = list(range(1, batch_size + 1))
attn_metadata.num_contexts = len(seq_lens)
attn_metadata.request_ids = request_ids
- attn_metadata.prompt_lens = prompt_lens
- attn_metadata.seq_lens = seq_lens
- attn_metadata.max_seq_len = seq_lens.max().item()
+ attn_metadata.prompt_lens = seq_lens
+ attn_metadata.seq_lens = seq_lens_torch
+ attn_metadata.max_seq_len = max(seq_lens)
attn_metadata.prepare()
return attn_metadata
+ @property
+ def device(self) -> torch.device:
+ return self.patch_embed.proj.weight.device
+
@torch.inference_mode()
def forward(self, pixel_values: torch.Tensor, grid_thw: torch.Tensor,
**kwargs) -> torch.Tensor:
- window_index, window_seq_lens = self.get_window_index(grid_thw)
- seq_lens = torch.repeat_interleave(grid_thw[:, 1] * grid_thw[:, 2],
- grid_thw[:, 0]).tolist()
- reverse_indices = torch.argsort(window_index)
-
- # Getting positional embedding
- rotary_pos_emb = self.rot_pos_emb(grid_thw)
- full_attn_metadata = self.prepare_attn_metadata(seq_lens,
- self.full_attn_metadata)
- window_attn_metadata = self.prepare_attn_metadata(
- window_seq_lens, self.window_attn_metadata)
-
- # From this point, pure GPU operation
hidden_states = self.patch_embed(pixel_values)
+
seq_len, _ = hidden_states.size()
+ rope_position_ids = torch.arange(seq_len,
+ dtype=torch.int32,
+ pin_memory=prefer_pinned())
+ grid_rows = grid_thw.tolist()
+
+ (rotary_pos_emb_cos, rotary_pos_emb_sin, window_indices,
+ window_seq_lens) = self.get_rotary_pos_emb_window_data(grid_rows)
+
+ window_index = torch.cat(window_indices).to(device=self.device,
+ non_blocking=True)
+
+ # Scatter sort: window_index maps original token order -> window order.
+ reverse_indices = torch.empty_like(window_index)
+ reverse_indices[window_index] = torch.arange(
+ window_index.numel(),
+ device=self.device,
+ dtype=window_index.dtype,
+ )
+
+ cos = torch.cat(rotary_pos_emb_cos).to(device=self.device,
+ non_blocking=True)
+ sin = torch.cat(rotary_pos_emb_sin).to(device=self.device,
+ non_blocking=True)
+ position_embeddings = (cos, sin)
+
+ rope_position_ids = rope_position_ids.to(device=self.device,
+ dtype=torch.int32,
+ non_blocking=True)
+ seq_lens = torch.repeat_interleave(grid_thw[:, 1] * grid_thw[:, 2],
+ grid_thw[:, 0]).tolist()
+
hidden_states = hidden_states.reshape(
seq_len // self.spatial_merge_unit, self.spatial_merge_unit, -1)
- hidden_states = hidden_states[window_index, :, :]
- hidden_states = hidden_states.reshape(seq_len, -1)
+ hidden_states = hidden_states[window_index, :, :].reshape(seq_len, -1)
- rotary_pos_emb = rotary_pos_emb.reshape(
- seq_len // self.spatial_merge_unit, self.spatial_merge_unit, -1)
- rotary_pos_emb = rotary_pos_emb[window_index, :, :]
- rotary_pos_emb = rotary_pos_emb.reshape(seq_len, -1)
- emb = torch.cat((rotary_pos_emb, rotary_pos_emb), dim=-1)
- position_embeddings = (emb.cos(), emb.sin())
+ full_attn_metadata = self.prepare_attn_metadata(len(grid_rows),
+ seq_lens,
+ self.full_attn_metadata)
+ window_attn_metadata = self.prepare_attn_metadata(
+ len(grid_rows), window_seq_lens, self.window_attn_metadata)
for layer_num, block in enumerate(self.blocks):
-
if layer_num in self.fullatt_block_indexes:
attn_metadata = full_attn_metadata
else:
attn_metadata = window_attn_metadata
hidden_states = block(
- hidden_states,
+ hidden_states=hidden_states,
attn_metadata=attn_metadata,
position_embeddings=position_embeddings,
+ position_ids=rope_position_ids,
)
hidden_states = self.merger(hidden_states)
hidden_states = hidden_states[reverse_indices, :]
diff --git a/tensorrt_llm/_torch/models/modeling_qwen3vl.py b/tensorrt_llm/_torch/models/modeling_qwen3vl.py
index 2031a4b7dc18..ecdbc5fde4b3 100644
--- a/tensorrt_llm/_torch/models/modeling_qwen3vl.py
+++ b/tensorrt_llm/_torch/models/modeling_qwen3vl.py
@@ -1,7 +1,9 @@
import copy
import re
+from functools import lru_cache
from typing import Any, Dict, List, Optional, Tuple, Union
+import numpy as np
import torch
import torch.nn as nn
from PIL import Image
@@ -10,9 +12,6 @@
from transformers.models.qwen3_vl.modeling_qwen3_vl import (
Qwen3VLVisionPatchEmbed as HFQwen3VLVisionPatchEmbed,
)
-from transformers.models.qwen3_vl.modeling_qwen3_vl import (
- Qwen3VLVisionRotaryEmbedding as HFQwen3VLVisionRotaryEmbedding,
-)
from tensorrt_llm._torch.models.modeling_multimodal_utils import _is_disagg
from tensorrt_llm.functional import PositionEmbeddingType
@@ -39,7 +38,7 @@
from ..modules.layer_norm import LayerNorm
from ..modules.linear import Linear, TensorParallelMode
from ..modules.mlp import MLP
-from ..modules.rotary_embedding import MRotaryEmbedding
+from ..modules.rotary_embedding import MRotaryEmbedding, RotaryEmbedding
from .checkpoints.base_weight_mapper import BaseWeightMapper
from .checkpoints.hf.qwen3vl_weight_mapper import Qwen3VLHfWeightMapper
from .modeling_auto import AutoModelForCausalLM
@@ -476,9 +475,9 @@ def get_prompt_token_ids(
class Qwen3VLVisionAttention(Qwen2_5_VLVisionAttention):
def __init__(self, model_config, layer_idx):
- model_config.pretrained_config.max_position_embeddings = (
- model_config.pretrained_config.text_config.max_position_embeddings
- )
+ # Qwen3-VL keeps `torch_dtype` only on `text_config` under transformers 5.x
+ # strict mode; mirror it onto `vision_config` so the parent picks it up.
+ # `max_position_embeddings` is handled by the parent's text_config fallback.
model_config.pretrained_config.vision_config.torch_dtype = (
model_config.pretrained_config.text_config.dtype
)
@@ -609,6 +608,75 @@ def forward(self, hidden_states: torch.Tensor) -> torch.Tensor:
return hidden_states
+# Referenced from https://github.com/vllm-project/vllm/blob/main/vllm/model_executor/models/qwen3_vl.py#L668
+def pos_embed_interpolate_native(
+ embed_weight: torch.Tensor,
+ t: int,
+ h: int,
+ w: int,
+ num_grid_per_side: int,
+ m_size: int,
+ dtype: torch.dtype,
+) -> torch.Tensor:
+ """Eager PyTorch bilinear position-embedding interpolation.
+
+ Returns a tensor of shape ``(t * h * w, hidden_dim)`` with the
+ bilinearly-interpolated position embeddings in spatial-merge order.
+ """
+ assert h % m_size == 0 and w % m_size == 0, f"{h=} and {w=} must be divisible by {m_size=}"
+ hidden_dim = embed_weight.shape[1]
+ device = embed_weight.device
+
+ h_idxs = torch.linspace(
+ 0,
+ num_grid_per_side - 1,
+ h,
+ dtype=torch.float32,
+ device=device,
+ )
+ w_idxs = torch.linspace(
+ 0,
+ num_grid_per_side - 1,
+ w,
+ dtype=torch.float32,
+ device=device,
+ )
+
+ h_floor = h_idxs.to(torch.long)
+ w_floor = w_idxs.to(torch.long)
+ h_ceil = torch.clamp(h_floor + 1, max=num_grid_per_side - 1)
+ w_ceil = torch.clamp(w_floor + 1, max=num_grid_per_side - 1)
+
+ dh = h_idxs - h_floor
+ dw = w_idxs - w_floor
+
+ dh_grid, dw_grid = torch.meshgrid(dh, dw, indexing="ij")
+ h_floor_grid, w_floor_grid = torch.meshgrid(h_floor, w_floor, indexing="ij")
+ h_ceil_grid, w_ceil_grid = torch.meshgrid(h_ceil, w_ceil, indexing="ij")
+
+ w11 = dh_grid * dw_grid
+ w10 = dh_grid - w11
+ w01 = dw_grid - w11
+ w00 = 1 - dh_grid - w01
+
+ h_grid = torch.stack([h_floor_grid, h_floor_grid, h_ceil_grid, h_ceil_grid])
+ w_grid = torch.stack([w_floor_grid, w_ceil_grid, w_floor_grid, w_ceil_grid])
+ h_grid_idx = h_grid * num_grid_per_side
+
+ indices = (h_grid_idx + w_grid).reshape(4, -1)
+ weights = torch.stack([w00, w01, w10, w11], dim=0).reshape(4, -1, 1)
+ weights = weights.to(dtype=dtype)
+
+ embeds = embed_weight[indices]
+ embeds *= weights
+ combined = embeds.sum(dim=0)
+
+ combined = combined.reshape(h // m_size, m_size, w // m_size, m_size, hidden_dim)
+ combined = combined.permute(0, 2, 1, 3, 4).reshape(1, -1, hidden_dim)
+ repeated = combined.expand(t, -1, -1).reshape(-1, hidden_dim)
+ return repeated.to(dtype=dtype)
+
+
class Qwen3VisionModel(torch.nn.Module):
def __init__(self, model_config: ModelConfig[PretrainedConfig]):
super().__init__()
@@ -626,8 +694,22 @@ def __init__(self, model_config: ModelConfig[PretrainedConfig]):
self.pos_embed = nn.Embedding(self.config.num_position_embeddings, self.config.hidden_size)
self.num_grid_per_side = int(self.config.num_position_embeddings**0.5)
- head_dim = self.config.hidden_size // self.config.num_heads
- self.rotary_pos_emb = HFQwen3VLVisionRotaryEmbedding(head_dim // 2)
+ text_config = getattr(
+ model_config.pretrained_config, "text_config", model_config.pretrained_config
+ )
+ self.config.max_position_embeddings = text_config.max_position_embeddings
+ self.config.partial_rotary_factor = 0.5
+ self.config.num_attention_heads = self.config.num_heads
+ self.head_dim = self.config.hidden_size // self.config.num_heads
+ self.pos_embd_params = PositionalEmbeddingParams(
+ type=PositionEmbeddingType.rope_gpt_neox,
+ rope=RopeParams.from_config(self.config),
+ )
+ self.rotary_pos_emb = RotaryEmbedding(
+ self.pos_embd_params.rope,
+ head_dim=self.head_dim,
+ is_neox=self.pos_embd_params.is_neox,
+ )
self.blocks = nn.ModuleList(
[
@@ -657,118 +739,89 @@ def __init__(self, model_config: ModelConfig[PretrainedConfig]):
kv_cache_manager=None,
)
- def rot_pos_emb(self, grid_thw: torch.Tensor) -> torch.Tensor:
- merge_size = self.spatial_merge_size
-
- max_hw = int(grid_thw[:, 1:].max().item())
- freq_table = self.rotary_pos_emb(max_hw) # (max_hw, dim // 2)
- device = freq_table.device
-
- total_tokens = int(torch.prod(grid_thw, dim=1).sum().item())
- pos_ids = torch.empty((total_tokens, 2), dtype=torch.long, device=device)
-
- offset = 0
- for num_frames, height, width in grid_thw:
- merged_h, merged_w = height // merge_size, width // merge_size
-
- block_rows = torch.arange(merged_h, device=device) # block row indices
- block_cols = torch.arange(merged_w, device=device) # block col indices
- intra_row = torch.arange(merge_size, device=device) # intra-block row offsets
- intra_col = torch.arange(merge_size, device=device) # intra-block col offsets
-
- # Compute full-resolution positions
- row_idx = block_rows[:, None, None, None] * merge_size + intra_row[None, None, :, None]
- col_idx = block_cols[None, :, None, None] * merge_size + intra_col[None, None, None, :]
-
- row_idx = row_idx.expand(merged_h, merged_w, merge_size, merge_size).reshape(-1)
- col_idx = col_idx.expand(merged_h, merged_w, merge_size, merge_size).reshape(-1)
-
- coords = torch.stack((row_idx, col_idx), dim=-1)
-
- if num_frames > 1:
- coords = coords.repeat(num_frames, 1)
-
- num_tokens = coords.shape[0]
- pos_ids[offset : offset + num_tokens] = coords
- offset += num_tokens
-
- embeddings = freq_table[pos_ids] # lookup rotary embeddings
- embeddings = embeddings.flatten(1)
- return embeddings
-
- def fast_pos_embed_interpolate(self, grid_thw):
- grid_ts, grid_hs, grid_ws = grid_thw[:, 0], grid_thw[:, 1], grid_thw[:, 2]
-
- idx_list = [[] for _ in range(4)]
- weight_list = [[] for _ in range(4)]
-
- for t, h, w in zip(grid_ts, grid_hs, grid_ws):
- h_idxs = torch.linspace(0, self.num_grid_per_side - 1, h)
- w_idxs = torch.linspace(0, self.num_grid_per_side - 1, w)
-
- h_idxs_floor = h_idxs.int()
- w_idxs_floor = w_idxs.int()
- h_idxs_ceil = (h_idxs.int() + 1).clip(max=self.num_grid_per_side - 1)
- w_idxs_ceil = (w_idxs.int() + 1).clip(max=self.num_grid_per_side - 1)
-
- dh = h_idxs - h_idxs_floor
- dw = w_idxs - w_idxs_floor
-
- base_h = h_idxs_floor * self.num_grid_per_side
- base_h_ceil = h_idxs_ceil * self.num_grid_per_side
-
- indices = [
- (base_h[None].T + w_idxs_floor[None]).flatten(),
- (base_h[None].T + w_idxs_ceil[None]).flatten(),
- (base_h_ceil[None].T + w_idxs_floor[None]).flatten(),
- (base_h_ceil[None].T + w_idxs_ceil[None]).flatten(),
- ]
-
- weights = [
- ((1 - dh)[None].T * (1 - dw)[None]).flatten(),
- ((1 - dh)[None].T * dw[None]).flatten(),
- (dh[None].T * (1 - dw)[None]).flatten(),
- (dh[None].T * dw[None]).flatten(),
- ]
-
- for i in range(4):
- idx_list[i].extend(indices[i].tolist())
- weight_list[i].extend(weights[i].tolist())
-
- idx_tensor = torch.tensor(idx_list, dtype=torch.long, device=self.pos_embed.weight.device)
- weight_tensor = torch.tensor(
- weight_list, dtype=self.pos_embed.weight.dtype, device=self.pos_embed.weight.device
+ @property
+ def device(self) -> torch.device:
+ return self.patch_embed.proj.weight.device
+
+ @staticmethod
+ @lru_cache(maxsize=1024)
+ def rot_pos_ids(h: int, w: int, spatial_merge_size: int) -> torch.Tensor:
+ hpos_ids = np.broadcast_to(np.arange(h).reshape(h, 1), (h, w))
+ h_div = h // spatial_merge_size
+ w_div = w // spatial_merge_size
+ hpos_ids = hpos_ids.reshape(
+ h_div,
+ spatial_merge_size,
+ w_div,
+ spatial_merge_size,
+ )
+ hpos_ids = hpos_ids.transpose(0, 2, 1, 3)
+ hpos_ids = hpos_ids.flatten()
+
+ wpos_ids = np.broadcast_to(np.arange(w).reshape(1, w), (h, w))
+ wpos_ids = wpos_ids.reshape(
+ h_div,
+ spatial_merge_size,
+ w_div,
+ spatial_merge_size,
)
- pos_embeds = self.pos_embed(idx_tensor) * weight_tensor[:, :, None]
- patch_pos_embeds = pos_embeds[0] + pos_embeds[1] + pos_embeds[2] + pos_embeds[3]
-
- patch_pos_embeds = patch_pos_embeds.split([h * w for h, w in zip(grid_hs, grid_ws)])
-
- patch_pos_embeds_permute = []
- merge_size = self.config.spatial_merge_size
- for pos_embed, t, h, w in zip(patch_pos_embeds, grid_ts, grid_hs, grid_ws):
- pos_embed = pos_embed.repeat(t, 1)
- pos_embed = (
- pos_embed.view(t, h // merge_size, merge_size, w // merge_size, merge_size, -1)
- .permute(0, 1, 3, 2, 4, 5)
- .flatten(0, 4)
+ wpos_ids = wpos_ids.transpose(0, 2, 1, 3)
+ wpos_ids = wpos_ids.flatten()
+
+ return torch.from_numpy(np.stack([hpos_ids, wpos_ids], axis=-1))
+
+ def rot_pos_emb(self, grid_thw: list[list[int]]):
+ max_grid_size = max(max(h, w) for _, h, w in grid_thw)
+ pos_ids = [
+ self.rot_pos_ids(h, w, self.spatial_merge_size)
+ if t == 1
+ else self.rot_pos_ids(h, w, self.spatial_merge_size).repeat(t, 1)
+ for t, h, w in grid_thw
+ ]
+ pos_ids = torch.cat(pos_ids, dim=0).to(self.device, non_blocking=True)
+
+ # Use pre-computed cos_sin_cache from RotaryEmbedding
+ cos_sin = self.rotary_pos_emb.rotary_cos_sin[:max_grid_size]
+ cos, sin = cos_sin[:, 0, :], cos_sin[:, 1, :]
+ cos_combined = cos[pos_ids].flatten(1)
+ sin_combined = sin[pos_ids].flatten(1)
+
+ return (cos_combined, sin_combined)
+
+ # Referenced from https://github.com/vllm-project/vllm/blob/main/vllm/model_executor/models/qwen3_vl.py#L668
+ def fast_pos_embed_interpolate(self, grid_thw: list[list[int]]) -> torch.Tensor:
+ interpolate_fn = pos_embed_interpolate_native
+ outputs = []
+ for t, h, w in grid_thw:
+ outputs.append(
+ interpolate_fn(
+ self.pos_embed.weight,
+ t,
+ h,
+ w,
+ self.num_grid_per_side,
+ self.spatial_merge_size,
+ self.dtype,
+ )
)
- patch_pos_embeds_permute.append(pos_embed)
- patch_pos_embeds = torch.cat(patch_pos_embeds_permute)
- return patch_pos_embeds
+ return torch.cat(outputs, dim=0)
- def prepare_attn_metadata(self, seq_lens, attn_metadata: AttentionMetadata):
- # NOTE: The single prompt is divided into multiple seq_lens, so pretending have many batch_sizes.
+ @property
+ def dtype(self) -> torch.dtype:
+ return self.patch_embed.proj.weight.dtype
+
+ def prepare_attn_metadata(
+ self, batch_size: int, seq_lens: List[int], attn_metadata: AttentionMetadata
+ ):
batch_size = len(seq_lens)
- prompt_lens = seq_lens
- seq_lens = torch.tensor(seq_lens, dtype=torch.int, pin_memory=prefer_pinned())
+ seq_lens_torch = torch.tensor(seq_lens, dtype=torch.int, pin_memory=prefer_pinned())
request_ids = list(range(1, batch_size + 1))
- attn_metadata.num_contexts = batch_size
+ attn_metadata.num_contexts = len(seq_lens)
attn_metadata.request_ids = request_ids
- attn_metadata.prompt_lens = prompt_lens
- attn_metadata.seq_lens = seq_lens
- attn_metadata.max_seq_len = seq_lens.max().item()
+ attn_metadata.prompt_lens = seq_lens
+ attn_metadata.seq_lens = seq_lens_torch
+ attn_metadata.max_seq_len = max(seq_lens)
attn_metadata.prepare()
return attn_metadata
@@ -777,28 +830,30 @@ def forward(
self, pixel_values: torch.Tensor, grid_thw: torch.Tensor, **kwargs
) -> Tuple[torch.Tensor, List[torch.Tensor]]:
seq_lens = torch.repeat_interleave(grid_thw[:, 1] * grid_thw[:, 2], grid_thw[:, 0]).tolist()
- attn_metadata = self.prepare_attn_metadata(seq_lens, self.attn_metadata)
+ grid_rows = grid_thw.detach().cpu().tolist()
+ attn_metadata = self.prepare_attn_metadata(len(grid_thw), seq_lens, self.attn_metadata)
- # Getting positional embedding
- rotary_pos_emb = self.rot_pos_emb(grid_thw)
- pos_embeds = self.fast_pos_embed_interpolate(grid_thw)
+ # Getting positional embedding (use the CPU-materialized grid to avoid
+ # converting CUDA tensors to numpy inside `rot_pos_ids`).
+ rotary_pos_emb = self.rot_pos_emb(grid_rows)
- # From this point, pure GPU operation
+ pos_embeds = self.fast_pos_embed_interpolate(grid_rows)
hidden_states = self.patch_embed(pixel_values)
hidden_states = hidden_states + pos_embeds
seq_len, _ = hidden_states.size()
+ rope_position_ids = torch.arange(seq_len, dtype=torch.int32, pin_memory=prefer_pinned())
+ rope_position_ids = rope_position_ids.to(
+ device=self.device, dtype=torch.int32, non_blocking=True
+ )
hidden_states = hidden_states.reshape(seq_len, -1)
- rotary_pos_emb = rotary_pos_emb.reshape(seq_len, -1)
- emb = torch.cat((rotary_pos_emb, rotary_pos_emb), dim=-1)
- position_embeddings = (emb.cos(), emb.sin())
-
deepstack_feature_lists = []
for layer_num, block in enumerate(self.blocks):
hidden_states = block(
- hidden_states,
+ position_ids=rope_position_ids,
+ hidden_states=hidden_states,
attn_metadata=attn_metadata,
- position_embeddings=position_embeddings,
+ position_embeddings=rotary_pos_emb,
)
if layer_num in self.deepstack_visual_indexes:
deepstack_feature = self.deepstack_merger_list[
diff --git a/tensorrt_llm/serve/chat_utils.py b/tensorrt_llm/serve/chat_utils.py
index 11e85c3ce766..f58ca8ff2399 100644
--- a/tensorrt_llm/serve/chat_utils.py
+++ b/tensorrt_llm/serve/chat_utils.py
@@ -375,7 +375,7 @@ def parse_chat_messages_coroutines(
# `content_parts` - overwriting any STRING-style placeholders inserted here.
# See also: `_resolve_content_format` (inputs/utils.py) for the full resolution used downstream.
registry_format = MULTIMODAL_PLACEHOLDER_REGISTRY.get_content_format(
- model_type)
+ type(model_config).model_type)
if registry_format is not None:
content_format = registry_format
else:
@@ -404,14 +404,16 @@ def parse_chat_messages_coroutines(
# prepend/append according to placeholder_placement.
content_parts = parsed_msg.get("content_parts")
interleave = MULTIMODAL_PLACEHOLDER_REGISTRY.get_interleave_placeholders(
- model_type)
+ type(model_config).model_type)
if content_parts and interleave:
parsed_msg["content"] = interleave_mm_placeholders(
- model_type, content_parts, msg_placeholder_counts,
+ type(model_config).model_type, content_parts,
+ msg_placeholder_counts,
mm_data_tracker.placeholder_modalities())
else:
parsed_msg["content"] = add_multimodal_placeholders(
- model_type, parsed_msg["content"], msg_placeholder_counts)
+ type(model_config).model_type, parsed_msg["content"],
+ msg_placeholder_counts)
mm_placeholder_counts.append(msg_placeholder_counts)
return conversation, mm_data_tracker.retrieve_all_async(
diff --git a/tensorrt_llm/serve/openai_server.py b/tensorrt_llm/serve/openai_server.py
index 1b0b113f0a36..ec4ad55724e9 100644
--- a/tensorrt_llm/serve/openai_server.py
+++ b/tensorrt_llm/serve/openai_server.py
@@ -389,10 +389,17 @@ def _init_llm(self, chat_template: Optional[str] = None):
if disable_harmony or self.model_config is None:
self.use_harmony = False
else:
- self.use_harmony = (self.model_config.model_type == "gpt_oss")
+ self.use_harmony = (type(self.model_config).model_type == "gpt_oss")
self.tool_call_id_type = "random" # default tool call id type is random
if self.model_config is not None:
+ # NOTE: Use the instance-level ``model_type`` (JSON-derived) here, not
+ # ``type(cfg).model_type``. ``kimi_k2`` / ``deepseek_v32`` are aliases
+ # registered in ``_CONFIG_REGISTRY`` (config_utils.py) that reuse
+ # ``DeepseekV3Config``, whose class-level ``model_type`` is ``"deepseek_v3"``.
+ # Only the JSON ``model_type`` distinguishes these variants. Other call
+ # sites that consult multimodal/chat-template registries keyed on the
+ # canonical class attribute should keep using ``type(cfg).model_type``.
if self.model_config.model_type == "kimi_k2":
self.tool_call_id_type = "kimi_k2"
elif self.model_config.model_type == "deepseek_v32":
diff --git a/tests/integration/defs/accuracy/references/mmmu.yaml b/tests/integration/defs/accuracy/references/mmmu.yaml
index 34083c6ba5d5..fc23b4cbc6eb 100644
--- a/tests/integration/defs/accuracy/references/mmmu.yaml
+++ b/tests/integration/defs/accuracy/references/mmmu.yaml
@@ -16,6 +16,8 @@ google/gemma-3-12b-it:
- quant_algo: NVFP4
kv_cache_quant_algo: FP8
accuracy: 50.11
+LGAI-EXAONE/EXAONE-4.5-33B:
+ - accuracy: 51.22
Qwen/Qwen2-VL-7B-Instruct:
- accuracy: 48.44
Qwen/Qwen2.5-VL-7B-Instruct:
diff --git a/tests/integration/defs/accuracy/test_llm_api_pytorch_multimodal.py b/tests/integration/defs/accuracy/test_llm_api_pytorch_multimodal.py
index d7623cd828ae..39ce3d05d54d 100644
--- a/tests/integration/defs/accuracy/test_llm_api_pytorch_multimodal.py
+++ b/tests/integration/defs/accuracy/test_llm_api_pytorch_multimodal.py
@@ -100,6 +100,39 @@ def test_nvfp4(self):
task.evaluate(llm, sampling_params=self.sampling_params)
+class TestExaone4_5_33B(LlmapiAccuracyTestHarness):
+ MODEL_NAME = "LGAI-EXAONE/EXAONE-4.5-33B"
+ MODEL_PATH = f"{llm_models_root()}/EXAONE-4.5-33B"
+ MAX_NUM_TOKENS = 16384
+
+ # EXAONE 4.5 ends each assistant turn with `<|endofturn|>`.
+ sampling_params = SamplingParams(
+ max_tokens=MMMU.MAX_OUTPUT_LEN,
+ truncate_prompt_tokens=MMMU.MAX_INPUT_LEN,
+ stop="<|endofturn|>",
+ )
+
+ kv_cache_config = KvCacheConfig(free_gpu_memory_fraction=0.6)
+
+ @pytest.mark.parametrize(
+ "enable_chunked_prefill,max_num_tokens",
+ [
+ (False, MAX_NUM_TOKENS),
+ (True, 1024),
+ ],
+ ids=["full_budget", "forced_chunked_prefill"],
+ )
+ def test_auto_dtype(self, enable_chunked_prefill, max_num_tokens):
+ with LLM(
+ self.MODEL_PATH,
+ enable_chunked_prefill=enable_chunked_prefill,
+ max_num_tokens=max_num_tokens,
+ kv_cache_config=self.kv_cache_config,
+ ) as llm:
+ task = MMMU(self.MODEL_NAME)
+ task.evaluate(llm, sampling_params=self.sampling_params)
+
+
class TestLlava_V1_6_Mistral_7B(LlmapiAccuracyTestHarness):
MODEL_NAME = "llava-hf/llava-v1.6-mistral-7b-hf"
MODEL_PATH = f"{llm_models_root()}/llava-v1.6-mistral-7b-hf"
diff --git a/tests/integration/test_lists/qa/llm_function_core.txt b/tests/integration/test_lists/qa/llm_function_core.txt
index 78b6dbee30bf..eda83a0aa1ab 100644
--- a/tests/integration/test_lists/qa/llm_function_core.txt
+++ b/tests/integration/test_lists/qa/llm_function_core.txt
@@ -784,6 +784,8 @@ accuracy/test_llm_api_pytorch_encode.py::TestDecoderEncode::test_encode_matches_
accuracy/test_llm_api_pytorch_encode.py::TestDecoderEncode::test_encode_matches_huggingface[qwen2-7b]
accuracy/test_llm_api_pytorch_encode.py::TestDecoderEncode::test_encode_matches_huggingface[qwen3-0.6b]
accuracy/test_llm_api_pytorch_encode.py::TestDecoderEncode::test_encode_matches_huggingface[starcoder2-3b]
+accuracy/test_llm_api_pytorch_multimodal.py::TestExaone4_5_33B::test_auto_dtype[full_budget]
+accuracy/test_llm_api_pytorch_multimodal.py::TestExaone4_5_33B::test_auto_dtype[forced_chunked_prefill]
accuracy/test_llm_api_pytorch_multimodal.py::TestGemma3_27BInstruct::test_fp8_prequantized
accuracy/test_llm_api_pytorch_multimodal.py::TestMistralLarge3_675B::test_nvfp4_4gpus[latency_moe_trtllm]
accuracy/test_llm_api_pytorch_multimodal.py::TestMistralSmall24B::test_auto_dtype[forced_chunked_prefill]
diff --git a/tests/unittest/_torch/modeling/test_modeling_exaone4_5.py b/tests/unittest/_torch/modeling/test_modeling_exaone4_5.py
new file mode 100644
index 000000000000..092db03e25a5
--- /dev/null
+++ b/tests/unittest/_torch/modeling/test_modeling_exaone4_5.py
@@ -0,0 +1,255 @@
+# SPDX-FileCopyrightText: Copyright (c) 2022-2026 NVIDIA CORPORATION & AFFILIATES. All rights reserved.
+# SPDX-License-Identifier: Apache-2.0
+
+import os
+from dataclasses import dataclass
+from typing import List
+
+import pytest
+import torch
+from test_modeling_multimodal import MultimodalScenario, TestModelingMultimodal
+from transformers import AutoProcessor
+from utils.llm_data import llm_models_root
+
+try:
+ from transformers import (
+ Exaone4_5_ForConditionalGeneration as HFExaone4_5ForConditionalGeneration,
+ )
+except ImportError:
+ # Falls back to skipping HF-vs-TRT-LLM comparison on transformers < 5.8.
+ HFExaone4_5ForConditionalGeneration = None
+
+from tensorrt_llm._torch.model_config import _mirror_text_subconfig_attrs
+from tensorrt_llm._torch.models.checkpoints.hf.exaone4_5_weight_mapper import (
+ Exaone4_5HfWeightMapper,
+)
+from tensorrt_llm._torch.models.modeling_exaone4_5 import (
+ Exaone4_5_ForConditionalGeneration,
+ Exaone4_5Config,
+)
+from tensorrt_llm._utils import get_sm_version
+
+# Reduced-size config for fast unit testing. Layer counts are shrunk so the
+# random-init HF model + TRT-LLM model fit on a single GPU while still
+# exercising the LLLG sliding/full attention pattern and at least one vision
+# full-attention block.
+EXAONE_4_5_TEST_CONFIG = {
+ "architectures": ["Exaone4_5_ForConditionalGeneration"],
+ "image_token_id": 67,
+ "model_type": "exaone4_5",
+ "text_config": {
+ "architectures": ["Exaone4ForCausalLM"],
+ "attention_dropout": 0.0,
+ "bos_token_id": 1,
+ "dtype": "bfloat16",
+ "eos_token_id": 53,
+ "hidden_act": "silu",
+ "hidden_size": 5120,
+ "initializer_range": 0.02,
+ "intermediate_size": 27392,
+ # Full LLLG cycle (4 layers) covers both sliding and full attention.
+ "layer_types": [
+ "sliding_attention",
+ "sliding_attention",
+ "sliding_attention",
+ "full_attention",
+ ],
+ "max_position_embeddings": 131072,
+ "max_window_layers": 4,
+ "model_type": "exaone4",
+ "num_attention_heads": 40,
+ "num_hidden_layers": 4,
+ "num_key_value_heads": 8,
+ "reorder_qk_norm": True,
+ "rms_norm_eps": 1e-05,
+ "rope_scaling": {
+ "factor": 16.0,
+ "high_freq_factor": 4.0,
+ "low_freq_factor": 1.0,
+ "original_max_position_embeddings": 8192,
+ "rope_type": "llama3",
+ },
+ "rope_theta": 1000000.0,
+ "sliding_window": 4096,
+ "sliding_window_pattern": "LLLG",
+ "use_cache": True,
+ "vocab_size": 153600,
+ },
+ "transformers_version": "5.8.0",
+ "video_token_id": 68,
+ "vision_config": {
+ # Reduced depth; index 0 is a global-attention block so the global
+ # path is exercised even at this size.
+ "depth": 2,
+ "dtype": "bfloat16",
+ "fullatt_block_indexes": [0],
+ "hidden_act": "silu",
+ "hidden_size": 2048,
+ "in_channels": 3,
+ "initializer_range": 0.02,
+ "intermediate_size": 5120,
+ "model_type": "exaone4_5_vision",
+ "num_heads": 32,
+ "num_key_value_heads": 8,
+ "out_hidden_size": 5120,
+ "patch_size": 14,
+ "spatial_merge_size": 2,
+ "temporal_patch_size": 2,
+ "tokens_per_second": 2,
+ "window_size": 112,
+ },
+ "vision_end_token_id": 74,
+ "vision_start_token_id": 73,
+ "vision_token_id": 67,
+ "vocab_size": 153600,
+ # Source of tokenizer / image processor / video processor files at runtime.
+ # Resolved against `LLM_MODELS_ROOT` (defaults to `/code/llm-models`).
+ "_name_or_path": str(os.path.join(llm_models_root(), "EXAONE-4.5-33B")),
+}
+
+_EXAONE_4_5_ASSET_PATH = EXAONE_4_5_TEST_CONFIG.get("_name_or_path")
+
+
+@dataclass(repr=False)
+class TestExaone4_5Scenario(MultimodalScenario):
+ """Scenario config for Exaone4.5 multimodal smoke tests."""
+
+ pass
+
+
+# Skip the whole class when local weights/processor assets are missing on
+# disk so the test doesn't fail in environments that don't have them mirrored.
+@pytest.mark.skipif(
+ not _EXAONE_4_5_ASSET_PATH or not os.path.exists(_EXAONE_4_5_ASSET_PATH),
+ reason=(
+ "Exaone4.5 multimodal test requires weights / processor assets at "
+ f"config _name_or_path (missing or not found): {_EXAONE_4_5_ASSET_PATH!r}"
+ ),
+)
+class TestExaone4_5(TestModelingMultimodal):
+ """Smoke tests for Exaone4.5 multimodal modeling.
+
+ Requires transformers >= 5.8 (where Exaone4.5 was added). On older
+ releases the HF reference model is unavailable and HF-vs-TRT-LLM
+ comparison is skipped.
+ """
+
+ @property
+ def skip_hf_inference(self) -> bool:
+ return HFExaone4_5ForConditionalGeneration is None
+
+ @property
+ def trust_remote_code(self) -> bool:
+ return True
+
+ def get_model_config(self):
+ return EXAONE_4_5_TEST_CONFIG
+
+ def create_hf_config(self):
+ # Production builds the model_config via `ModelConfig.from_pretrained`
+ # which (1) derives `torch_dtype` from the (possibly nested) `dtype`
+ # field and (2) mirrors text-side fields onto the parent VLM config.
+ # The test constructs ModelConfig directly, so replicate both steps
+ # here to keep top-level accessors (`torch_dtype`,
+ # `max_position_embeddings`, ...) working.
+ hf_config = super().create_hf_config()
+ dtype = getattr(hf_config, "dtype", None)
+ if dtype is None:
+ text_config = getattr(hf_config, "text_config", None)
+ if text_config is not None:
+ dtype = getattr(text_config, "dtype", None)
+ hf_config.torch_dtype = dtype
+ _mirror_text_subconfig_attrs(hf_config)
+ return hf_config
+
+ def get_trtllm_model_class(self):
+ return Exaone4_5_ForConditionalGeneration
+
+ def get_hf_model_class(self):
+ return HFExaone4_5ForConditionalGeneration
+
+ def get_weight_mapper_class(self):
+ return Exaone4_5HfWeightMapper
+
+ def get_model_type(self):
+ return "exaone4_5"
+
+ def get_model_config_class(self):
+ return Exaone4_5Config
+
+ def get_hf_inputs(self, modality: str, prompt: List[str], media: List[str]):
+ # On transformers < 5.8 there is no native EXAONE 4.5 processor, so
+ # `AutoProcessor.from_pretrained` returns a `TokenizersBackend`
+ # (itself a `PreTrainedTokenizerBase` subclass) without a
+ # `.tokenizer` attribute. Monkey-patch `AutoProcessor.from_pretrained`
+ # for the duration of the base implementation so it surfaces the
+ # backend as its own tokenizer. On transformers >= 5.8 the native
+ # processor already exposes `.tokenizer` and this is a no-op.
+
+ original_from_pretrained = AutoProcessor.from_pretrained
+
+ def patched_from_pretrained(*args, **kwargs):
+ processor = original_from_pretrained(*args, **kwargs)
+ if not hasattr(processor, "tokenizer"):
+ object.__setattr__(processor, "tokenizer", processor)
+ return processor
+
+ AutoProcessor.from_pretrained = patched_from_pretrained
+ try:
+ return super().get_hf_inputs(modality, prompt, media)
+ finally:
+ AutoProcessor.from_pretrained = original_from_pretrained
+
+ def get_scenarios(self) -> List[TestExaone4_5Scenario]:
+ scenarios: List[TestExaone4_5Scenario] = [
+ # ==== Modality sanity checks ====
+ TestExaone4_5Scenario(
+ modality="image",
+ use_cuda_graph=False,
+ chunked_prefill=False,
+ kv_cache_reuse=False,
+ ),
+ TestExaone4_5Scenario(
+ modality="video",
+ use_cuda_graph=False,
+ chunked_prefill=False,
+ kv_cache_reuse=False,
+ ),
+ TestExaone4_5Scenario(
+ modality="multiple_image",
+ use_cuda_graph=False,
+ chunked_prefill=False,
+ kv_cache_reuse=False,
+ ),
+ # ==== CUDA graph ====
+ TestExaone4_5Scenario(
+ modality="image",
+ use_cuda_graph=True,
+ chunked_prefill=False,
+ kv_cache_reuse=False,
+ ),
+ ]
+ # Paged context FMHA (triggered by chunked_prefill / kv_cache_reuse)
+ # is forced on for correctness on Hopper (SM90); on Blackwell (SM100)
+ # the trtllm-gen kernel set falls back to an unfused MHA path whose
+ # output diverges from the non-paged context kernel even with a single
+ # full-length chunk. Skip these scenarios outside SM90 until the
+ # Blackwell paged-context fallback matches.
+ if torch.cuda.is_available() and get_sm_version() == 90:
+ scenarios.extend(
+ [
+ TestExaone4_5Scenario(
+ modality="image",
+ use_cuda_graph=False,
+ chunked_prefill=True,
+ kv_cache_reuse=False,
+ ),
+ TestExaone4_5Scenario(
+ modality="image",
+ use_cuda_graph=False,
+ chunked_prefill=False,
+ kv_cache_reuse=True,
+ ),
+ ]
+ )
+ return scenarios
diff --git a/tests/unittest/_torch/modeling/test_modeling_qwen2_5vl.py b/tests/unittest/_torch/modeling/test_modeling_qwen2_5vl.py
index 56b631371df6..e98c4a675a37 100644
--- a/tests/unittest/_torch/modeling/test_modeling_qwen2_5vl.py
+++ b/tests/unittest/_torch/modeling/test_modeling_qwen2_5vl.py
@@ -14,6 +14,7 @@
Qwen2VLHfWeightMapper
from tensorrt_llm._torch.models.modeling_qwen2vl import (
Qwen2_5_VLModel, Qwen2VLInputProcessorBase)
+from tensorrt_llm._utils import get_sm_version
QWEN2_5_VL_7B_CONFIG = {
"architectures": ["Qwen2_5_VLForConditionalGeneration"],
@@ -235,28 +236,40 @@ def get_scenarios(self) -> List[TestQwen2_5_VLScenario]:
disable_fuse_rope=False,
chunked_prefill=False,
kv_cache_reuse=False),
-
- # ==== Chunked Prefill Scenarios ====
- TestQwen2_5_VLScenario(modality="image",
- use_cuda_graph=False,
- disable_fuse_rope=False,
- chunked_prefill=True,
- kv_cache_reuse=False),
-
- # ==== KV Cache Reuse Scenarios ====
- TestQwen2_5_VLScenario(modality="image",
- use_cuda_graph=False,
- disable_fuse_rope=False,
- chunked_prefill=False,
- kv_cache_reuse=True),
-
- # ==== Disable fuse rope scenarios ====
+ ]
+ # Paged context FMHA (triggered by chunked_prefill / kv_cache_reuse)
+ # is forced on for correctness on Hopper (SM90). On Blackwell (SM100)
+ # the trtllm-gen kernel set falls back to an unfused MHA path whose
+ # output diverges from the non-paged context kernel; gate those
+ # scenarios to SM90 until the Blackwell fallback matches.
+ if torch.cuda.is_available() and get_sm_version() == 90:
+ scenarios.extend([
+ # ==== Chunked Prefill Scenarios ====
+ TestQwen2_5_VLScenario(modality="image",
+ use_cuda_graph=False,
+ disable_fuse_rope=False,
+ chunked_prefill=True,
+ kv_cache_reuse=False),
+ # ==== KV Cache Reuse Scenarios ====
+ TestQwen2_5_VLScenario(modality="image",
+ use_cuda_graph=False,
+ disable_fuse_rope=False,
+ chunked_prefill=False,
+ kv_cache_reuse=True),
+ ])
+ # ==== Disable fuse rope scenarios ====
+ # Run last: setup_scenario rebuilds trtllm_model with
+ # disable_fuse_rope=True for this scenario, and the rebuild is not
+ # undone afterwards. Keeping it at the tail prevents the rebuilt
+ # model from leaking into chunked-prefill / kv-cache-reuse
+ # scenarios (where it surfaces as a cos/sin vs. q/k seq-len
+ # mismatch in MRotaryEmbedding.forward).
+ scenarios.append(
TestQwen2_5_VLScenario(modality="image",
use_cuda_graph=False,
disable_fuse_rope=True,
chunked_prefill=False,
- kv_cache_reuse=False),
- ]
+ kv_cache_reuse=False))
return scenarios
def get_hf_inputs(self, modality: str, prompt, media):
diff --git a/tests/unittest/_torch/modeling/test_modeling_qwen3vl.py b/tests/unittest/_torch/modeling/test_modeling_qwen3vl.py
index c6add810cc29..27691e8fa5c4 100644
--- a/tests/unittest/_torch/modeling/test_modeling_qwen3vl.py
+++ b/tests/unittest/_torch/modeling/test_modeling_qwen3vl.py
@@ -11,6 +11,7 @@
from tensorrt_llm._torch.models.checkpoints.hf.qwen3vl_weight_mapper import Qwen3VLHfWeightMapper
from tensorrt_llm._torch.models.modeling_qwen3vl import Qwen3VLInputProcessorBase, Qwen3VLModel
+from tensorrt_llm._utils import get_sm_version
QWEN3_VL_8B_CONFIG = {
"architectures": ["Qwen3VLForConditionalGeneration"],
@@ -243,31 +244,49 @@ def get_scenarios(self) -> List[TestQwen3VLScenario]:
chunked_prefill=False,
kv_cache_reuse=False,
),
- # ==== Chunked Prefill Scenarios ====
- TestQwen3VLScenario(
- modality="image",
- use_cuda_graph=False,
- disable_fuse_rope=False,
- chunked_prefill=True,
- kv_cache_reuse=False,
- ),
- # ==== KV Cache Reuse Scenarios ====
- TestQwen3VLScenario(
- modality="image",
- use_cuda_graph=False,
- disable_fuse_rope=False,
- chunked_prefill=False,
- kv_cache_reuse=True,
- ),
- # ==== Disable fuse rope scenarios ====
+ ]
+ # Paged context FMHA (triggered by chunked_prefill / kv_cache_reuse)
+ # is forced on for correctness on Hopper (SM90); the trtllm-gen
+ # kernel set on Blackwell (SM100) falls back to an unfused MHA path
+ # whose output diverges from the non-paged context kernel. Gate
+ # those scenarios to SM90 until the Blackwell fallback matches.
+ if torch.cuda.is_available() and get_sm_version() == 90:
+ scenarios.extend(
+ [
+ # ==== Chunked Prefill Scenarios ====
+ TestQwen3VLScenario(
+ modality="image",
+ use_cuda_graph=False,
+ disable_fuse_rope=False,
+ chunked_prefill=True,
+ kv_cache_reuse=False,
+ ),
+ # ==== KV Cache Reuse Scenarios ====
+ TestQwen3VLScenario(
+ modality="image",
+ use_cuda_graph=False,
+ disable_fuse_rope=False,
+ chunked_prefill=False,
+ kv_cache_reuse=True,
+ ),
+ ]
+ )
+ # ==== Disable fuse rope scenarios ====
+ # Run last: setup_scenario rebuilds trtllm_model with
+ # disable_fuse_rope=True for this scenario, and the rebuild is not
+ # undone afterwards. Keeping it at the tail prevents the rebuilt
+ # model from leaking into chunked-prefill / kv-cache-reuse
+ # scenarios (where it surfaces as a cos/sin vs. q/k seq-len
+ # mismatch in MRotaryEmbedding.forward).
+ scenarios.append(
TestQwen3VLScenario(
modality="image",
use_cuda_graph=False,
disable_fuse_rope=True,
chunked_prefill=False,
kv_cache_reuse=False,
- ),
- ]
+ )
+ )
return scenarios
def get_hf_inputs(self, modality: str, prompt, media):
diff --git a/tests/unittest/_torch/modeling/test_modeling_qwen3vl_moe.py b/tests/unittest/_torch/modeling/test_modeling_qwen3vl_moe.py
index 33b777ab9f5c..1c3dfd8bbd59 100644
--- a/tests/unittest/_torch/modeling/test_modeling_qwen3vl_moe.py
+++ b/tests/unittest/_torch/modeling/test_modeling_qwen3vl_moe.py
@@ -13,6 +13,7 @@
Qwen3VLMoeHfWeightMapper,
)
from tensorrt_llm._torch.models.modeling_qwen3vl_moe import Qwen3MoeVLModel
+from tensorrt_llm._utils import get_sm_version
QWEN3_VL_30B_A3B_CONFIG = {
"architectures": ["Qwen3VLMoeForConditionalGeneration"],
@@ -252,31 +253,49 @@ def get_scenarios(self) -> List[TestQwen3VLMoeScenario]:
chunked_prefill=False,
kv_cache_reuse=False,
),
- # ==== Chunked Prefill Scenarios ====
- TestQwen3VLMoeScenario(
- modality="image",
- use_cuda_graph=False,
- disable_fuse_rope=False,
- chunked_prefill=True,
- kv_cache_reuse=False,
- ),
- # ==== KV Cache Reuse Scenarios ====
- TestQwen3VLMoeScenario(
- modality="image",
- use_cuda_graph=False,
- disable_fuse_rope=False,
- chunked_prefill=False,
- kv_cache_reuse=True,
- ),
- # ==== Disable fuse rope scenarios ====
+ ]
+ # Paged context FMHA (triggered by chunked_prefill / kv_cache_reuse)
+ # is forced on for correctness on Hopper (SM90); the trtllm-gen
+ # kernel set on Blackwell (SM100) falls back to an unfused MHA path
+ # whose output diverges from the non-paged context kernel. Gate
+ # those scenarios to SM90 until the Blackwell fallback matches.
+ if torch.cuda.is_available() and get_sm_version() == 90:
+ scenarios.extend(
+ [
+ # ==== Chunked Prefill Scenarios ====
+ TestQwen3VLMoeScenario(
+ modality="image",
+ use_cuda_graph=False,
+ disable_fuse_rope=False,
+ chunked_prefill=True,
+ kv_cache_reuse=False,
+ ),
+ # ==== KV Cache Reuse Scenarios ====
+ TestQwen3VLMoeScenario(
+ modality="image",
+ use_cuda_graph=False,
+ disable_fuse_rope=False,
+ chunked_prefill=False,
+ kv_cache_reuse=True,
+ ),
+ ]
+ )
+ # ==== Disable fuse rope scenarios ====
+ # Run last: setup_scenario rebuilds trtllm_model with
+ # disable_fuse_rope=True for this scenario, and the rebuild is not
+ # undone afterwards. Keeping it at the tail prevents the rebuilt
+ # model from leaking into chunked-prefill / kv-cache-reuse
+ # scenarios (where it surfaces as a cos/sin vs. q/k seq-len
+ # mismatch in MRotaryEmbedding.forward).
+ scenarios.append(
TestQwen3VLMoeScenario(
modality="image",
use_cuda_graph=False,
disable_fuse_rope=True,
chunked_prefill=False,
kv_cache_reuse=False,
- ),
- ]
+ )
+ )
return scenarios
def setup_scenario(self, scenario: TestQwen3VLMoeScenario):
diff --git a/tests/unittest/llmapi/apps/test_chat_utils.py b/tests/unittest/llmapi/apps/test_chat_utils.py
index a65927ff2b11..53f282524e77 100644
--- a/tests/unittest/llmapi/apps/test_chat_utils.py
+++ b/tests/unittest/llmapi/apps/test_chat_utils.py
@@ -1,7 +1,6 @@
from unittest.mock import AsyncMock, MagicMock, patch
import pytest
-from transformers import AutoConfig
from tensorrt_llm.inputs import MultimodalDataTracker
from tensorrt_llm.inputs.media_io import AudioMediaIO
@@ -398,8 +397,12 @@ class TestMultimodalPlaceholderCounts:
],
)
def test_per_message_counts(self, messages, expected_mm_placeholder_counts):
- mock_config = MagicMock(spec=AutoConfig)
- mock_config.model_type = _MM_MODEL_TYPE
+ # Use a real class so that `type(config).model_type` (a class-attribute
+ # lookup in the production code) resolves correctly.
+ class _StubConfig:
+ model_type = _MM_MODEL_TYPE
+
+ mock_config = _StubConfig()
_, _, mm_placeholder_counts = parse_chat_messages_coroutines(messages, mock_config, None)