Skip to content
Merged
Show file tree
Hide file tree
Changes from 29 commits
Commits
Show all changes
41 commits
Select commit Hold shift + click to select a range
4c9beb0
squashed with main
rahul-steiger-nv Jun 10, 2026
a11d7c7
updated tests to run post merge
rahul-steiger-nv Jun 11, 2026
f2632d5
fix precommit
rahul-steiger-nv Jun 17, 2026
cc212bb
addressed api review
rahul-steiger-nv Jun 25, 2026
c377b20
fixed pre-commit
rahul-steiger-nv Jun 25, 2026
11ebadb
removed stale test
rahul-steiger-nv Jun 25, 2026
06ebc45
Update docs/source/models/visual-generation.md
rahul-steiger-nv Jun 26, 2026
e963008
reverted change
rahul-steiger-nv Jun 26, 2026
0eecb07
added co-resident stage example
rahul-steiger-nv Jun 26, 2026
2bf1e32
updated feature matrix
rahul-steiger-nv Jul 8, 2026
6c5aaa7
remove stale name
rahul-steiger-nv Jul 9, 2026
434b47c
Update docs/source/models/visual-generation.md
rahul-steiger-nv Jul 10, 2026
2dc9d39
Update tests/integration/test_lists/test-db/l0_b200.yml
rahul-steiger-nv Jul 10, 2026
1d76025
Update docs/source/models/visual-generation.md
rahul-steiger-nv Jul 14, 2026
864c1a3
Update docs/source/models/visual-generation.md
rahul-steiger-nv Jul 14, 2026
45b47a8
fixed real_text_lens issue
rahul-steiger-nv Jul 15, 2026
c17fc5b
Merge remote-tracking branch 'upstream/main' into visual_gen_offloadi…
rahul-steiger-nv Jul 29, 2026
8d180d0
Merge remote-tracking branch 'upstream/main' into visual_gen_offloadi…
rahul-steiger-nv Jul 30, 2026
a4c99c9
Merge branch 'main' into visual_gen_offloading
chang-l Jul 31, 2026
7617d4e
Merge latest main to resolve conflicts
chang-l Jul 31, 2026
6078df4
fix ltx test
rahul-steiger-nv Aug 2, 2026
c488467
Merge remote-tracking branch 'upstream/main' into visual_gen_offloadi…
rahul-steiger-nv Aug 2, 2026
f6961ef
Merge branch 'main' into visual_gen_offloading
rahul-steiger-nv Aug 3, 2026
6221d09
Merge branch 'main' into visual_gen_offloading
rahul-steiger-nv Aug 3, 2026
de2553f
Merge upstream/main into visual_gen_offloading
chang-l Aug 3, 2026
593c00d
Merge latest upstream/main into visual_gen_offloading
chang-l Aug 4, 2026
456aba9
Fix Cosmos3 bare pipeline test devices
chang-l Aug 4, 2026
f4a784b
Merge branch 'main' into visual_gen_offloading
rahul-steiger-nv Aug 4, 2026
9278f07
Merge branch 'main' into visual_gen_offloading
rahul-steiger-nv Aug 5, 2026
1013837
Merge upstream/main into visual_gen_offloading
rahul-steiger-nv Aug 11, 2026
562fde0
Merge remote-tracking branch 'upstream/main' into visual_gen_offloadi…
rahul-steiger-nv Aug 11, 2026
6b37bce
Merge remote-tracking branch 'upstream/main' into visual_gen_offloadi…
rahul-steiger-nv Aug 18, 2026
577de1f
Merge remote-tracking branch 'upstream/main' into visual_gen_offloadi…
rahul-steiger-nv Aug 18, 2026
2556b64
Merge remote-tracking branch 'upstream/main' into visual_gen_offloadi…
rahul-steiger-nv Aug 25, 2026
24136f2
[None][test] Set _device on remaining Cosmos3 bare pipeline test doubles
rahul-steiger-nv Aug 26, 2026
01157d6
Merge remote-tracking branch 'upstream/main' into visual_gen_offloadi…
rahul-steiger-nv Aug 26, 2026
5d0ae57
fix tests, guardrails, and transfer
rahul-steiger-nv Aug 26, 2026
9ac811b
Merge branch 'main' into visual_gen_offloading
rahul-steiger-nv Aug 27, 2026
dcbd74e
fixed pre-commit
rahul-steiger-nv Aug 27, 2026
eddccd0
Merge remote-tracking branch 'upstream/main' into visual_gen_offloadi…
rahul-steiger-nv Aug 28, 2026
d847427
fixed action tests
rahul-steiger-nv Aug 28, 2026
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
32 changes: 20 additions & 12 deletions docs/source/models/visual-generation.md
Original file line number Diff line number Diff line change
Expand Up @@ -18,6 +18,7 @@ TensorRT-LLM **VisualGen** provides a unified inference stack for diffusion mode
- Sparse attention support: see [VisualGen Sparse Attention](../visual-gen/features/sparse-attention.md).
- Multi-GPU parallelism (CFG parallel, Ulysses sequence parallel, Tensor parallelism).
- **Step caching** — two runtime caching backends (**TeaCache** and **Cache-DiT**) that skip transformer computation on steps where the step-to-step change is small.
- CPU offloading to reduce peak GPU memory usage.
- `trtllm-serve` integration with OpenAI-compatible API endpoints for image and video generation.

## Supported Models
Expand Down Expand Up @@ -48,18 +49,18 @@ Models are auto-detected from the checkpoint directory. Diffusers-format models

### Feature Matrix

| Model | FP8 blockwise | NVFP4 | TeaCache | Cache-DiT | CFG Parallelism | Ulysses Parallelism | Parallel VAE | CUDA Graph | torch.compile | trtllm-serve | Attention2D | Ring Attention | Tensor Parallelism | VSA |
|---|---|---|---|---|---|---|---|---|---|---|--|--|--|--|
| **FLUX.1** | Yes | Yes | Yes | Yes | No [^1] | Yes | No | Yes | Yes | Yes | Yes | Yes | Yes | No |
| **FLUX.2** | Yes | Yes | Yes | Yes | No [^1] | Yes | No | Yes | Yes | Yes | Yes | Yes | Yes | No |
| **Wan 2.1** | Yes | Yes | Yes | Yes | Yes | Yes | Yes | Yes | Yes | Yes | Yes | Yes | Yes | No |
| **Wan 2.1 VSA** [^2] | Yes | Yes | Yes | Yes | Yes | Yes | Yes | Yes | Yes | Yes | No | No | Yes | Yes |
| **Wan 2.2** | Yes | Yes | Yes [^3] | Yes | Yes | Yes | Yes | Yes | Yes | Yes | Yes | Yes | Yes | No |
| **LTX-2** | Yes | Yes | Yes [^4] | Yes | Yes | Yes | No | No | Yes | Yes | Yes | Yes | No | No |
| **Qwen-Image** | Yes | Yes | Yes | Yes | Yes | Yes | No | Yes | Yes | Yes | Yes | Yes | No | No |
| **Qwen-Image-Layered** [^6] | No | No | No | No | No | No | No | Yes | Yes | No | No | No | No | No |
| **Qwen-Image-Edit-2511** | Yes | Yes | No | No | Yes | No | No | Yes | Yes | No | No | No | No | No |
| **Cosmos3** | Yes | Yes | No | No | Yes | Yes | Yes | Yes | Yes | Yes | No | No | Yes | No |
| Model | FP8 blockwise | NVFP4 | TeaCache | Cache-DiT | CPU Offloading | CFG Parallelism | Ulysses Parallelism | Parallel VAE | CUDA Graph | torch.compile | trtllm-serve | Attention2D | Ring Attention | Tensor Parallelism | VSA |
|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|
| **FLUX.1** | Yes | Yes | Yes | Yes | No | No [^1] | Yes | No | Yes | Yes | Yes | Yes | Yes | Yes | No |
| **FLUX.2** | Yes | Yes | Yes | Yes | No | No [^1] | Yes | No | Yes | Yes | Yes | Yes | Yes | Yes | No |
| **Wan 2.1** | Yes | Yes | Yes | Yes | Yes [^7] | Yes | Yes | Yes | Yes | Yes | Yes | Yes | Yes | Yes | No |
| **Wan 2.1 VSA** [^2] | Yes | Yes | Yes | Yes | Yes [^7] | Yes | Yes | Yes | Yes | Yes | Yes | No | No | Yes | Yes |
| **Wan 2.2** | Yes | Yes | Yes [^3] | Yes | Yes [^7] | Yes | Yes | Yes | Yes | Yes | Yes | Yes | Yes | Yes | No |
| **LTX-2** | Yes | Yes | Yes [^4] | Yes | No | Yes | Yes | No | No | Yes | Yes | Yes | Yes | No | No |
| **Qwen-Image** | Yes | Yes | Yes | Yes | No | Yes | Yes | No | Yes | Yes | Yes | Yes | Yes | No | No |
| **Qwen-Image-Layered** [^6] | No | No | No | No | No | No | No | No | Yes | Yes | No | No | No | No | No |
| **Qwen-Image-Edit-2511** | Yes | Yes | No | No | No | Yes | No | No | Yes | Yes | No | No | No | No | No |
| **Cosmos3** | Yes | Yes | No | No | Yes [^7] | Yes | Yes | Yes | Yes | Yes | Yes | No | No | Yes | No |

[^1]: FLUX models use embedded guidance and do not have a separate negative prompt path, so CFG parallelism is not applicable.

Expand All @@ -70,6 +71,9 @@ Models are auto-detected from the checkpoint directory. Diffusers-format models
[^4]: LTX-2 has no built-in TeaCache coefficient table in TRT-LLM; set `teacache.coefficients` explicitly when enabling TeaCache.

[^6]: Qwen-Image-Layered supports baseline BF16 image-conditioned layer decomposition and returns the generated RGBA layer stack as a saveable image grid. FP8 blockwise, NVFP4, cache acceleration, attention-parallel/Sage/VSA backends, Tensor Parallelism, and `trtllm-serve` image-edit routing are not enabled for this pipeline yet.

[^7]: CPU offloading of Cosmos3 is currently validated for the text-to-video pipelines.

## Quick Start

Here is a simple example to generate a video with Wan 2.1:
Expand Down Expand Up @@ -272,6 +276,9 @@ attention_config:
vsa_sparsity: 0.90
```

### CPU Offloading

CPU offloading stages move selected Wan and Cosmos3 T2V pipeline components between CPU and GPU to reduce peak GPU memory usage; enable it with `cpu_offload_config.enable: true`.

### Multi-GPU Parallelism

Expand All @@ -285,6 +292,7 @@ Configured under `VisualGenArgs.parallel_config`. Modes can be combined:
- **Attention2D** (`attn2d_size: [N, M]`): Shards the sequence axis across an `N × M` device mesh (CP degree = `N · M`; total SP degree = `N · M · ulysses_size`).
- **Ring Attention** (`ring_size: N`): Shards the sequence axis across a 1D ring of `N` ranks, streaming K/V blocks (CP degree = `N`; total SP degree = `N · ulysses_size`; mutually exclusive with Attention2D).
- **Tensor Parallelism** (`tp_size: N`): Splits attention heads and transformer MLPs across GPUs for faster compute and reduced memory usage.

## Developer Guide

### Architecture Overview
Expand Down
15 changes: 10 additions & 5 deletions tensorrt_llm/_torch/modules/linear.py
Original file line number Diff line number Diff line change
Expand Up @@ -836,11 +836,14 @@ def rescale_fused_weights(self, module: Linear):
"""
weight_mode = module.weights_loading_config.weight_mode
shard_key_to_index = weight_mode.shard_key_to_index
compute_device = module.weight.device
if compute_device.type == "cpu" and torch.cuda.is_available():
compute_device = torch.device("cuda")

# Handle input_scale
if hasattr(module, "has_static_input_scale"):
# Compute max and replace input_scale with a new parameter
max_input_scale = module.tmp_input_scales.max()
max_input_scale = module.tmp_input_scales.to(compute_device).max()
Comment thread
NVShreyas marked this conversation as resolved.
Resolved
module.input_scale.data.copy_(max_input_scale)
# Also update inv_input_scale for static quantization
if hasattr(
Expand All @@ -854,13 +857,13 @@ def rescale_fused_weights(self, module: Linear):
module.inv_input_scale = None

# Compute max weight_scale
max_weight_scale = module.tmp_weight_scales.max()
max_weight_scale = module.tmp_weight_scales.to(compute_device).max()
module.weight_scale.data.copy_(max_weight_scale)

# Rescale each weight shard: (weight * original_scale) / max_scale
for shard_key in weight_mode.shard_keys:
idx = shard_key_to_index[shard_key]
original_scale = module.tmp_weight_scales[idx]
original_scale = module.tmp_weight_scales[idx].to(compute_device)

# Get shard position from mapping
shard_offset, shard_size = module.fused_weight_shard_indices_mapping[
Expand All @@ -869,12 +872,14 @@ def rescale_fused_weights(self, module: Linear):
# Rescale: FP8 -> BF16 -> multiply by original_scale -> divide by max_scale -> FP8
weight_shard = module.weight.data[shard_offset:shard_offset +
shard_size]
rescaled_weight = weight_shard.to(module.dtype).mul_(original_scale)
rescaled_weight = weight_shard.to(compute_device).to(
module.dtype).mul_(original_scale)
rescaled_weight = rescaled_weight.div_(
max_weight_scale.to(rescaled_weight.device)).to(
torch.float8_e4m3fn)
module.weight.data[shard_offset:shard_offset +
shard_size] = rescaled_weight
shard_size] = rescaled_weight.to(
module.weight.device)

delattr(module, "tmp_input_scales")
delattr(module, "tmp_weight_scales")
Expand Down
17 changes: 15 additions & 2 deletions tensorrt_llm/_torch/visual_gen/config.py
Original file line number Diff line number Diff line change
Expand Up @@ -35,6 +35,7 @@
CacheConfig,
CacheDiTConfig,
CompilationConfig,
CpuOffloadConfig,
CudaGraphConfig,
ParallelConfig,
TeaCacheConfig,
Expand Down Expand Up @@ -108,6 +109,7 @@ class DiffusionModelConfig(_VisualGenConfigBase):

# Unified parallelism mapping copied from the owning pipeline config.
visual_gen_mapping: Optional[Any] = None # VisualGenMapping (lazy import)
device: str = "cuda"

dynamic_weight_quant: bool = False

Expand All @@ -118,6 +120,7 @@ class DiffusionModelConfig(_VisualGenConfigBase):
compilation: CompilationConfig = PydanticField(default_factory=CompilationConfig)
torch_compile: TorchCompileConfig = PydanticField(default_factory=TorchCompileConfig)
cuda_graph: CudaGraphConfig = PydanticField(default_factory=CudaGraphConfig)
cpu_offload_config: CpuOffloadConfig = PydanticField(default_factory=CpuOffloadConfig)
attention: AttentionConfig = PydanticField(default_factory=AttentionConfig)
attention_metadata_state: Optional[Dict[str, Any]] = None
parallel: ParallelConfig = PydanticField(default_factory=ParallelConfig)
Expand Down Expand Up @@ -173,6 +176,7 @@ class DiffusionPipelineConfig(_VisualGenConfigBase):

# Unified parallelism mapping (populated by setup_visual_gen_mapping)
visual_gen_mapping: Optional[Any] = None # VisualGenMapping (lazy import)
device: str = "cuda"

dynamic_weight_quant: bool = False

Expand All @@ -183,6 +187,7 @@ class DiffusionPipelineConfig(_VisualGenConfigBase):
compilation: CompilationConfig = PydanticField(default_factory=CompilationConfig)
torch_compile: TorchCompileConfig = PydanticField(default_factory=TorchCompileConfig)
cuda_graph: CudaGraphConfig = PydanticField(default_factory=CudaGraphConfig)
cpu_offload_config: CpuOffloadConfig = PydanticField(default_factory=CpuOffloadConfig)
attention: AttentionConfig = PydanticField(default_factory=AttentionConfig)
attention_metadata_state: Optional[Dict[str, Any]] = None
parallel: ParallelConfig = PydanticField(default_factory=ParallelConfig)
Expand Down Expand Up @@ -240,12 +245,14 @@ def _make_model_config(
extra_attrs=_model_config_value(self.extra_attrs),
# Topology mappings carry distributed process-group handles.
visual_gen_mapping=_model_config_value(self.visual_gen_mapping, deep_copy=False),
device=_model_config_value(self.device),
dynamic_weight_quant=_model_config_value(self.dynamic_weight_quant),
quant_config=_model_config_value(self.quant_config),
quant_config_dict=_model_config_value(self.quant_config_dict),
compilation=_model_config_value(self.compilation),
torch_compile=_model_config_value(self.torch_compile),
cuda_graph=_model_config_value(self.cuda_graph),
cpu_offload_config=_model_config_value(self.cpu_offload_config),
attention=_model_config_value(self.attention),
attention_metadata_state=_model_config_value(self.attention_metadata_state),
parallel=_model_config_value(self.parallel),
Expand Down Expand Up @@ -460,8 +467,9 @@ def from_pretrained(
Args:
checkpoint_dir: Path to checkpoint
args: VisualGenArgs containing user config
- (compilation, torch_compile, cuda_graph, pipeline, attention, parallel, teacache,
cache_backend, cache_dit)
- (compilation_config, torch_compile_config, cuda_graph_config,
cpu_offload_config, pipeline_config, attention_config, parallel_config,
cache_config)
**kwargs: Additional config options (e.g., mapping)
"""
kwargs.pop("trust_remote_code", None)
Expand All @@ -470,6 +478,7 @@ def from_pretrained(
compilation_cfg = args.compilation_config if args else CompilationConfig()
torch_compile_cfg = args.torch_compile_config if args else TorchCompileConfig()
cuda_graph_cfg = args.cuda_graph_config if args else CudaGraphConfig()
offload_cfg = args.cpu_offload_config if args else CpuOffloadConfig()
attention_cfg = args.attention_config if args else AttentionConfig()
parallel_cfg = args.parallel_config if args else ParallelConfig()
cache_cfg = args.cache_config if args else None
Expand Down Expand Up @@ -612,6 +621,8 @@ def from_pretrained(
create_attention_metadata_state() if attention_cfg.backend == "TRTLLM" else None
)

device = kwargs.pop("device", "cuda")

Comment thread
rahul-steiger-nv marked this conversation as resolved.
Resolved
pipeline_config = cls(
quant_config=quant_config,
quant_config_dict=quant_config_dict,
Expand All @@ -621,10 +632,12 @@ def from_pretrained(
compilation=compilation_cfg,
torch_compile=torch_compile_cfg,
cuda_graph=cuda_graph_cfg,
cpu_offload_config=offload_cfg,
attention=attention_cfg,
attention_metadata_state=attention_metadata_state,
parallel=parallel_cfg,
cache=cache_cfg,
device=device,
enable_layerwise_nvtx_marker=enable_layerwise_nvtx_marker,
skip_create_weights_in_init=True,
extra_attrs=extra_attrs,
Expand Down
53 changes: 53 additions & 0 deletions tensorrt_llm/_torch/visual_gen/executor.py
Original file line number Diff line number Diff line change
Expand Up @@ -227,6 +227,21 @@ def _detect_external_launch() -> Optional[Tuple[int, int, int, str, int]]:
return None


def _cuda_memory_logging_enabled() -> bool:
"""Whether per-request CUDA peak-memory logging is enabled.

This is a development-only knob exposed as an environment variable
(rather than a public ``VisualGenArgs`` field) to keep the engine
config surface clean, mirroring the nsys trace knob
``TLLM_PROFILE_VISUAL_GEN_START_STOP``.
"""
return os.environ.get("TLLM_VISUAL_GEN_LOG_CUDA_MEMORY", "0").strip().lower() in (
"1",
"true",
"yes",
)


@dataclass
class DiffusionRequest:
"""Request for diffusion inference.
Expand Down Expand Up @@ -430,6 +445,9 @@ def _merge_defaults(self, req: DiffusionRequest):

def process_request(self, req: DiffusionRequest):
"""Process a single request."""
log_cuda_memory = _cuda_memory_logging_enabled()
if log_cuda_memory:
self._reset_cuda_peak_memory_stats()
try:
self._merge_defaults(req)
# Include request preparation in executor-side generation latency.
Expand All @@ -451,7 +469,10 @@ def process_request(self, req: DiffusionRequest):
f"Warmed-up shapes: {self.pipeline._warmed_up_shapes}"
)
output = self.pipeline.run_inference(req)
if log_cuda_memory:
self._log_cuda_peak_memory(req.request_id)
generation = time.perf_counter() - generation_start # seconds

Comment thread
rahul-steiger-nv marked this conversation as resolved.
Resolved
if self.rank == 0:
# CUDA IPC handles are invalid within the producing process, so
# a same-process client takes the media via in-process handoff.
Expand All @@ -464,6 +485,8 @@ def process_request(self, req: DiffusionRequest):
)
)
except Exception as e:
if log_cuda_memory:
self._log_cuda_peak_memory(req.request_id)
logger.error(f"Worker {self.device_id}: Error: {e}")
logger.error(traceback.format_exc())
if self.rank == 0:
Expand All @@ -475,6 +498,36 @@ def process_request(self, req: DiffusionRequest):
)
)

def _reset_cuda_peak_memory_stats(self) -> None:
"""Reset CUDA peak memory stats for this worker if CUDA is available."""
if not torch.cuda.is_available():
return

try:
torch.cuda.reset_peak_memory_stats(self.device_id)
except RuntimeError as e:
logger.warning(
f"Worker {self.device_id} rank {self.rank}: "
f"Unable to reset CUDA peak memory stats: {e}"
)
Comment thread
rahul-steiger-nv marked this conversation as resolved.
Resolved
Comment thread
rahul-steiger-nv marked this conversation as resolved.
Resolved

def _log_cuda_peak_memory(self, request_id: int) -> None:
"""Log peak CUDA memory observed for one request."""
if not torch.cuda.is_available():
return

try:
peak_allocated = torch.cuda.max_memory_allocated(self.device_id)
logger.info(
f"Worker {self.device_id} rank {self.rank}: "
f"Request {request_id} peak CUDA memory: {peak_allocated / 2**30:.2f} GiB"
)
except RuntimeError as e:
logger.warning(
f"Worker {self.device_id} rank {self.rank}: "
f"Unable to log CUDA peak memory for request {request_id}: {e}"
)


def run_diffusion_worker(
rank: int,
Expand Down
Loading
Loading