[TRTLLM-14778][perf] Add feature-mode encoder CUDA graphs for fixed-shape encoders (Whisper)

docs/source/models/encoder-decoder.md

@@ -45,7 +45,7 @@ The following table describes the supported and recommended configurations.
 | Beam search | Yes with V1 | Configure `max_beam_width` when constructing `LLM`, then set `use_beam_search=True` in `SamplingParams`. |
 | Attention backend | `TRTLLM` | Use this backend for encoder-decoder models. It is required when `tensor_parallel_size > 1`. |
 | Decoder CUDA graphs | Yes, except in FP32 | `CudaGraphConfig` captures decoder work. V1 supports greedy and beam search; V2 supports its single-beam path. FP32 encoder-decoder models decline capture at engine init and log a warning instead of failing. |
-| Encoder CUDA graphs | Yes | Set `encoder_cuda_graph_config=EncodeCudaGraphConfig(...)` and `encoder_max_batch_size`. Usually set `encoder_max_batch_size` lower than `max_batch_size`. The `TRTLLM` attention backend is required. |
+| Encoder CUDA graphs | Yes | Set `encoder_cuda_graph_config=EncodeCudaGraphConfig(...)` and `encoder_max_batch_size`. Usually set `encoder_max_batch_size` lower than `max_batch_size`. The `TRTLLM` attention backend is required. Text encoders also require `num_tokens` and `seq_lens`; a fixed-shape feature encoder such as Whisper derives both from the model and needs only `batch_sizes`. |
 | Overlap scheduler | Yes | Enabled by default. V1 supports greedy decoding and beam search; V2 remains limited to `max_beam_width=1`. |
 | Tensor parallelism | Yes | Use `tensor_parallel_size > 1` with `attn_backend="TRTLLM"`. Attention head counts must be divisible by the TP size. |
 | Pipeline parallelism | No | Keep `pipeline_parallel_size=1`. |
@@ -437,6 +437,47 @@ size, total packed tokens, and maximum sequence length. The
 limit. With beam search, decoder graph batch sizes must cover the active
 decoder sequences after beam expansion.
 
+Which encoder buckets you must supply depends on the model. A text encoder,
+such as BART or T5, packs a variable number of tokens per request, so
+`num_tokens` and `seq_lens` are part of its key space and are required; leaving
+either unset is rejected at `LLM(...)` for any architecture TensorRT-LLM
+recognizes, and at model engine initialization otherwise, since only the loaded
+model states with certainty which kind of encoder it has. An encoder whose input is a
+fixed-shape per-request feature tensor, such as Whisper's fixed 30-second
+zero-padded audio waveform (the mel transform runs inside the encoder, so the
+per-request input is the waveform itself, not a spectrogram), produces the same
+number of encoder positions for every request, so both lists follow from the
+model and are derived rather than configured. For those models
+`batch_sizes` alone enables capture, and any `num_tokens` or `seq_lens` you set
+is ignored.
+
+Batch sizes that do not fit `encoder_max_num_tokens` divided by the model's
+encoder output length are dropped, and the encoder stays eager when none fit.
+Size the encoder token budget for the largest bucket before setting the
+buckets: Whisper emits 1500 encoder positions per request, so `batch_sizes` up
+to 8 needs `encoder_max_num_tokens` of at least 12000. `encoder_max_num_tokens`
+falls back to `max_num_tokens` when unset, which is a decoder-sized number and
+usually too small.
+
+```python
+from tensorrt_llm.llmapi import EncodeCudaGraphConfig
+
+
+llm = LLM(
+    model="openai/whisper-large-v3",
+    backend="pytorch",
+    attn_backend="TRTLLM",
+    max_batch_size=8,
+    encoder_max_batch_size=8,
+    # 8 buckets * 1500 encoder positions. Leave this at the default and the
+    # 4 and 8 buckets are silently dropped.
+    encoder_max_num_tokens=12000,
+    encoder_cuda_graph_config=EncodeCudaGraphConfig(batch_sizes=[1, 2, 4, 8]),
+    # ... the remaining Whisper settings from "Transcribe audio with Whisper",
+    # whose `max_batch_size=4` this example raises to 8
+)
+```
+
 `max_batch_size` controls the total decoder concurrency, while
 `encoder_max_batch_size` controls encoder microbatch admission. For better
 performance, tune `encoder_max_batch_size`, `encoder_max_num_tokens`, and the
@@ -656,6 +697,13 @@ that the encoder graph buckets cover the request shape, and that
 `attn_backend="TRTLLM"`. Unsupported shapes and attention backends fall back to
 eager encoder execution.
 
+### `num_tokens` or `seq_lens` unset is rejected at engine construction
+
+A text encoder needs both bucket lists, so the engine raises rather than
+silently running eager. Supply them, or drop `encoder_cuda_graph_config` if you
+do not want encoder graphs. A fixed-shape feature encoder such as Whisper does
+not hit this: it derives both from the model and needs only `batch_sizes`.
+
 ### Output quality differs from the Hugging Face example
 
 Confirm that the source uses the task prefix and language settings expected by

tensorrt_llm/_torch/models/modeling_whisper.py

@@ -381,6 +381,12 @@ def __init__(self, config: WhisperConfig):
         extractor = _load_hf_feature_extractor(config)
         self.n_fft = int(extractor.n_fft)
         self.hop_length = int(extractor.hop_length)
+        # The extractor's own padded window length, kept verbatim because
+        # `WhisperInputProcessor` pads every request to exactly this many
+        # samples. `encoder_graph_spec` reports it as the fixed encoder input
+        # shape, so deriving it a second way would risk disagreeing with the
+        # tensors the processor actually produces.
+        self.n_samples = int(extractor.n_samples)
         # Pre-STFT Gaussian noise, applied where HF applies it; 0.0 (all
         # official checkpoints) disables it.
         self.dither = float(getattr(extractor, "dither", 0.0))
@@ -654,6 +660,17 @@ def _build_decoder_prompt(self) -> List[int]:
             forced = self.processor.get_decoder_prompt_ids(no_timestamps=True)
         return [int(start_id)] + [int(tok) for _, tok in sorted(forced)]
 
+    def get_decoder_prefix_len(self) -> int:
+        """Tokens every request's decoder prompt starts with.
+
+        Mixed encoder/decoder CUDA graphs capture at this query length, and a
+        mismatch makes every mixed batch miss its graph silently. This reports
+        the checkpoint default, so a request carrying a text prompt of a
+        different length (see `_resolve_decoder_prompt`) misses the mixed graph
+        and runs that batch eagerly.
+        """
+        return len(self._decoder_prompt)
+
     def _resolve_decoder_prompt(self, prompt_text: Optional[str]) -> List[int]:
         """Checkpoint-default forced prompt, or the user's decoder prompt.
 
@@ -880,6 +897,29 @@ def __pp_init__(self):
     def config(self):
         return self.model_config.pretrained_config
 
+    def encoder_graph_spec(self) -> Tuple[Tuple[int, ...], torch.dtype, int]:
+        """Fixed-shape encoder contract for enc-dec encoder CUDA graphs.
+
+        Every Whisper encoder request is an fp32 waveform zero-padded by
+        `WhisperInputProcessor` to the extractor's window, which yields exactly
+        ``max_source_positions`` encoder positions — so the encoder graph key
+        degenerates to the batch size.
+
+        The window is read from the feature extractor rather than recomputed.
+        ``max_source_positions * 2 * hop_length`` inverts the conv stem
+        correctly but only up to the truncation in the processor's own
+        ``n_samples // hop_length // 2`` check, so a checkpoint whose
+        ``n_samples`` is not an exact multiple of ``2 * hop_length`` would
+        pass that check while disagreeing with this shape — capturing graphs
+        that every runtime batch then misses. Taking the extractor's value
+        keeps both sides on one number.
+
+        Returns ``(per_request_feature_shape, dtype, fixed_seq_len)``.
+        """
+        fixed_seq_len = int(self.config.max_source_positions)
+        n_samples = int(self.model.encoder.log_mel.n_samples)
+        return ((n_samples,), torch.float32, fixed_seq_len)
+
     def forward(
         self,
         attn_metadata: AttentionMetadata,