[TRTLLM-15762][refactor] Migrate decoder execution and private resources into the runner family by lori-ren · Pull Request #19896 · NVIDIA/TensorRT-LLM

lori-ren · 2026-10-06T14:00:24Z

Dev Engineer Review

The available diff changes only import formatting in model_engine.py and test_pytorch_model_engine.py. It shows no executable behavior, API, or configuration changes. The broader runner migration described in the PR context is not present in this diff, so its implementation cannot be assessed from the available evidence.

QA Engineer Review

The only test file in the available diff is tests/unittest/_torch/executor/test_pytorch_model_engine.py. Its changes reformat imports and do not alter test cases or assertions. No test-list files changed. BART, T5, and Whisper integration tests appear in CI test lists, but the available diff does not show changes to those integration tests. Coverage verdict: needs follow-up for the runner migration described in the PR context; test execution results are unavailable.

Per-File QA Perspective

tensorrt_llm/_torch/pyexecutor/model_engine.py: Import formatting changed. No behavior change is visible in the diff.
tests/unittest/_torch/executor/test_pytorch_model_engine.py: Import formatting changed. No test behavior change is visible in the diff.

Description

This PR moves decoder execution out of PyTorchModelEngine into DecoderRunner (engine/runners/decoder/). Every model now resolves to a runner, and _fallback_to_engine is removed.

DecoderRunner owns decoder input preparation, forward, warmup and CUDA graph capture, along with the buffers and metadata memos they use. It reads its settings from the frozen DecoderRunnerConfig, including the three values previously read from llm_args.
EncoderDecoderRunner extends DecoderRunner and composes the encoder stage, renamed EncoderStage, which keeps its own worker, stream and graphs. Cross-KV projection and dummy requests, mixed CUDA graph capture and cross-attention inputs move into it. DecoderRunner no longer checks is_encoder_decoder and exposes capability flags and hooks instead. enable_encoder_decoder_mixed_cuda_graph moves to EncoderDecoderRunnerConfig.
Speculative-metadata sizing is shared with the NoKV runner through SpecDecodeRunnerConfig and helpers in engine/metadata.py. get_static_draft_len takes the spec config.
Engine members that PyExecutor and other callers still use (attn_metadata, spec_metadata, cuda_graph_runner, graph release, late-install setters, forward_encoder and encoder warmup) forward to the runner.

Two orderings change: releasing graphs frees decoder graphs before the compile backend's piecewise graphs, and warmup dummy batches free their cross-KV blocks before the other cache managers.

Test Coverage

tests/unittest/_torch/executor/engine/test_runners.py
tests/unittest/_torch/executor/engine/test_encoder_decoder.py
tests/unittest/_torch/executor/engine/test_metadata.py
tests/unittest/_torch/executor/test_pytorch_model_engine.py
tests/unittest/_torch/executor/test_pytorch_model_engine_warmup.py
tests/unittest/_torch/executor/test_eager_workspace_engine.py
tests/unittest/_torch/executor/kv_cache/test_kvcm2_integration.py
tests/unittest/_torch/speculative/ (MTP, Eagle3, PARD, draft-target, ngram with CUDA graphs)
tests/integration/defs/llmapi/test_llm_api_pytorch_{bart,t5,whisper}.py
tests/unittest/llmapi/test_llm_multi_gpu_pytorch.py::test_tinyllama_logits_processor_2gpu

PR Checklist

Please review the following before submitting your PR:

PR description clearly explains what and why. If using CodeRabbit's summary, please make sure it makes sense.
PR Follows TRT-LLM CODING GUIDELINES to the best of your knowledge.
Test cases are provided for new code paths (see test instructions)
If PR introduces API changes, an appropriate PR label is added - either api-compatible or api-breaking. For api-breaking, include BREAKING in the PR title.
Any new dependencies have been scanned for license and vulnerabilities
CODEOWNERS updated if ownership changes
Documentation updated as needed
Update tava architecture diagram if there is a significant design change in PR.
The reviewers assigned automatically/manually are appropriate for the PR.
Please check this after reviewing the above items as appropriate for this PR.

GitHub Bot Help

To see a list of available CI bot commands, please comment /bot help.

Decoder buffers, graphs, forward and warmup move from PyTorchModelEngine into DecoderRunner, and EncoderDecoderRunner now composes it with the encoder stage. The engine holds a single ModelRunner. Signed-off-by: Lori Ren <lorir@nvidia.com>

DecoderRunner reads its resolved settings through self._config and no longer takes llm_args. Signed-off-by: Lori Ren <lorir@nvidia.com>

…unner DecoderRunner and NoKVCacheRunner create and update speculative metadata through shared helpers on a common config base. Gating and memoization stay with each runner. Signed-off-by: Lori Ren <lorir@nvidia.com>

The runner's static draft length read spec_config and draft limits that moved into the config. Two test stand-ins now set spec decoding through the runner config. Signed-off-by: Lori Ren <lorir@nvidia.com>

…oderDecoderRunner DecoderRunner keeps neutral hooks and capability flags and no longer reads is_encoder_decoder. The nine enc-dec methods move unchanged. Signed-off-by: Lori Ren <lorir@nvidia.com>

…base override The mixed path gets its own method, so EncoderDecoderRunner no longer widens the base signature. ExtraInputsCollector is exported from the decoder package. Signed-off-by: Lori Ren <lorir@nvidia.com>

Replace typing.Dict, List, Optional, Union, Tuple and Type with built-in generics and X | None. Only annotations change. Signed-off-by: Lori Ren <lorir@nvidia.com>

…ation Point stale comments at DecoderRunner, tighten runner and engine annotations, and make the shared MoE A2A warmup helpers public. Add unit tests for the split encoder-decoder warmup and input paths. Signed-off-by: Lori Ren <lorir@nvidia.com>

… config get_static_draft_len takes the spec config instead of an engine or runner config, whose draft widths are derived from it. Also drop a stale comment in DecoderRunner.model_forward. Signed-off-by: Lori Ren <lorir@nvidia.com>

lori-ren · 2026-10-07T06:20:19Z

/bot run

Pooling and multimodal encoder runners never use speculative metadata, so drop the plumbing and fold SpecDecodeRunnerConfig back into DecoderRunnerConfig. Engine construction now raises when such a model sets speculative_config. Signed-off-by: Lori Ren <lorir@nvidia.com>

…package DecoderRunner is their only consumer, so they move to decoder/speculative.py. engine/metadata.py keeps only build_attention_metadata. Signed-off-by: Lori Ren <lorir@nvidia.com>

…exts DecoderRunner.forward and NoKVCacheRunner.forward enter inference mode and the model extra-attrs context themselves, and decoder warmup runs through forward. Engine.forward no longer sets them. Signed-off-by: Lori Ren <lorir@nvidia.com>

QiJune

LGTM

…nner-migration Signed-off-by: Lori Ren <lorir@nvidia.com>

lori-ren · 2026-10-08T09:31:48Z

/bot run

tensorrt-cicd · 2026-10-08T09:39:22Z

PR_Github #76717 [ run ] triggered by Bot. Commit: 6127cb2 Link to invocation

github-actions · 2026-10-08T09:40:46Z

Automatically added "ci: full pre-merge approved" because this PR has satisfied the required GitHub review approvals. Unresolved review conversations and other required checks remain independent merge requirements.

tensorrt-cicd · 2026-10-08T09:55:55Z

PR_Github #76717 [ run ] completed with state FAILURE. Commit: 6127cb2

Link to invocation

lori-ren · 2026-10-08T10:22:22Z

/bot run

tensorrt-cicd · 2026-10-08T10:30:34Z

PR_Github #76732 [ run ] triggered by Bot. Commit: 6127cb2 Link to invocation

tensorrt-cicd · 2026-10-08T16:41:44Z

PR_Github #76732 [ run ] completed with state SUCCESS. Commit: 6127cb2
/LLM/main/L0_MergeRequest_PR pipeline #63271 completed with status: 'SUCCESS'

CI Report

Link to invocation

Main's NVIDIA#19896 moved the decoder engine internals, including _should_use_full_generation_page_table, from model_engine.py into engine/runners/decoder/runner.py. Move this branch's extracted uses_full_generation_page_table helper along with it: it now lives in runner.py next to the method, and the executor creator imports it from there. Semantics unchanged on both paths. Signed-off-by: Brian Nguyen <brnguyen@nvidia.com>

Main's NVIDIA#19896 (2681ed1) moved PyTorchModelEngine's decoder execution into DecoderRunner (engine/runners/decoder/runner.py) and the encoder-decoder fast path into EncoderDecoderRunner, so this branch's model_engine.py changes conflicted. model_engine.py is main's; the branch's changes move to where that code now lives: - DecoderRunner, from NVIDIA#19816 (U1): the step-copy gather, the staged previous batch / position indices, gather_ids written by copy_to_device_if_changed, no num_accepted_draft_tokens buffer, and the CUDA graphs' static inputs; from NVIDIA#19814 (U3): the gather of vocabulary-sharded logits for post-processors. - EncoderDecoderRunner: its previous_batch_indices_cuda write clears the staged batch indices. - The two tests that called the moved members call the runner. Main's other commits since f891b3d: NVIDIA#19484, NVIDIA#19892, NVIDIA#19409, NVIDIA#19553, NVIDIA#19957, NVIDIA#19749, NVIDIA#19585, NVIDIA#19766, NVIDIA#19963, NVIDIA#19588, NVIDIA#19600, NVIDIA#19825, NVIDIA#19821, NVIDIA#19930, NVIDIA#18485. Signed-off-by: Vasanth Sabavat <vsabavat@nvidia.com>

github-actions Bot assigned lori-ren Oct 6, 2026

lori-ren added 4 commits October 6, 2026 22:48

[TRTLLM-15762][refactor] Read decoder settings from DecoderRunnerConfig

3e5b2fb

DecoderRunner reads its resolved settings through self._config and no longer takes llm_args. Signed-off-by: Lori Ren <lorir@nvidia.com>

[TRTLLM-15762][fix] Pass DecoderRunnerConfig to get_static_draft_len

514656f

The runner's static draft length read spec_config and draft limits that moved into the config. Two test stand-ins now set spec decoding through the runner config. Signed-off-by: Lori Ren <lorir@nvidia.com>

lori-ren force-pushed the refactor/decoder-runner-migration branch from 266525b to 514656f Compare October 6, 2026 22:49

lori-ren added 5 commits October 7, 2026 02:21

[TRTLLM-15762][refactor] Move encoder-decoder specialization into Enc…

b644e95

…oderDecoderRunner DecoderRunner keeps neutral hooks and capability flags and no longer reads is_encoder_decoder. The nine enc-dec methods move unchanged. Signed-off-by: Lori Ren <lorir@nvidia.com>

[TRTLLM-15762][chore] Use PEP 585/604 annotations in the engine runners

8eb97ed

Replace typing.Dict, List, Optional, Union, Tuple and Type with built-in generics and X | None. Only annotations change. Signed-off-by: Lori Ren <lorir@nvidia.com>

lori-ren marked this pull request as ready for review October 7, 2026 06:20

lori-ren requested review from a team as code owners October 7, 2026 06:20

lori-ren requested review from QiJune, Shixiaowei02, WeiHaocheng, aswinvisva, cascade812, chuangz0, lowsfer, schetlur-nv, thorjohnsen and zhaoyangwang-nvidia October 7, 2026 06:20

mzweilz approved these changes Oct 8, 2026

View reviewed changes

nvpohanh approved these changes Oct 8, 2026

View reviewed changes

YihuiLu512 approved these changes Oct 8, 2026

View reviewed changes

jieli-matrix approved these changes Oct 8, 2026

View reviewed changes

xinhe-nv approved these changes Oct 8, 2026

View reviewed changes

QiJune reviewed Oct 8, 2026

View reviewed changes

Comment thread tensorrt_llm/_torch/pyexecutor/engine/runners/no_kv_cache.py Resolved

QiJune reviewed Oct 8, 2026

View reviewed changes

Comment thread tensorrt_llm/_torch/pyexecutor/engine/runners/decoder/runner.py Resolved

lori-ren added 3 commits October 8, 2026 09:00

[TRTLLM-15762][refactor] Move spec metadata helpers into the decoder …

48850fb

…package DecoderRunner is their only consumer, so they move to decoder/speculative.py. engine/metadata.py keeps only build_attention_metadata. Signed-off-by: Lori Ren <lorir@nvidia.com>

QiJune approved these changes Oct 8, 2026

View reviewed changes

Merge remote-tracking branch 'upstream/main' into refactor/decoder-ru…

6127cb2

…nner-migration Signed-off-by: Lori Ren <lorir@nvidia.com>

lori-ren enabled auto-merge (squash) October 8, 2026 09:32

trtllm-agent added the ci: full pre-merge approved label Oct 8, 2026

lori-ren merged commit 2681ed1 into NVIDIA:main Oct 8, 2026

YihuiLu512 mentioned this pull request Oct 9, 2026

[https://nvbugs/6278337][fix] Unwaive DeepSeekV3Lite bf16 PP4 tests #20020

Merged

1 task done

vsabavat mentioned this pull request Oct 9, 2026

[None][perf] One-model speculative decoding: host work between steps, worker-produced target logits, KDA drafts' records #19989

Merged

1 task done

lori-ren mentioned this pull request Oct 12, 2026

[TRTLLM-15766][refactor] Decouple decoder tests from private methods and split DecoderRunner by phase #20134

Merged

1 task done

john-liu-dev mentioned this pull request Oct 12, 2026

[Bug]: Qwen3-VL E/PD handoff drops per-frame timestamp tokens and frame markers for video inputs; encoder-side mRoPE ids are sliced against the shorter PD prompt #20138

Open

3 of 5 tasks

xxi-nv mentioned this pull request Oct 14, 2026

[None][fix] Merge back #19593 from release/1.3: Make MoE communication timeouts follow the warmup phase #20246

Merged

1 task done

coderabbitai Bot mentioned this pull request Oct 19, 2026

[TRTLLM-15763][refactor] Narrow the runner surface, slim engine state, and make the engine's executor interface explicit #20270

Merged

1 task done

Navigation Menu

Repository navigation

[TRTLLM-15762][refactor] Migrate decoder execution and private resources into the runner family - #19896

[TRTLLM-15762][refactor] Migrate decoder execution and private resources into the runner family#19896
lori-ren merged 14 commits into
NVIDIA:mainfrom
lori-ren:refactor/decoder-runner-migration

lori-ren commented Oct 6, 2026 •

edited by coderabbitai Bot

Loading

Uh oh!

lori-ren commented Oct 7, 2026

Uh oh!

Uh oh!

Uh oh!

QiJune left a comment

Uh oh!

lori-ren commented Oct 8, 2026

Uh oh!

tensorrt-cicd commented Oct 8, 2026

Uh oh!

github-actions Bot commented Oct 8, 2026

Uh oh!

tensorrt-cicd commented Oct 8, 2026

Uh oh!

lori-ren commented Oct 8, 2026

Uh oh!

tensorrt-cicd commented Oct 8, 2026

Uh oh!

tensorrt-cicd commented Oct 8, 2026

Uh oh!

Reviewers

Uh oh!

Assignees

Labels

Projects

Milestone

Development

Uh oh!

11 participants

Conversation

lori-ren commented Oct 6, 2026 • edited by coderabbitai Bot Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Dev Engineer Review

QA Engineer Review

Per-File QA Perspective

Description

Test Coverage

PR Checklist

GitHub Bot Help

Uh oh!

lori-ren commented Oct 7, 2026

Uh oh!

Uh oh!

Uh oh!

QiJune left a comment

Choose a reason for hiding this comment

Uh oh!

lori-ren commented Oct 8, 2026

Uh oh!

tensorrt-cicd commented Oct 8, 2026

Uh oh!

github-actions Bot commented Oct 8, 2026

Uh oh!

tensorrt-cicd commented Oct 8, 2026

Uh oh!

lori-ren commented Oct 8, 2026

Uh oh!

tensorrt-cicd commented Oct 8, 2026

Uh oh!

tensorrt-cicd commented Oct 8, 2026

Uh oh!

Reviewers

Uh oh!

Assignees

Labels

Projects

Milestone

Development

Uh oh!

11 participants

lori-ren commented Oct 6, 2026 •

edited by coderabbitai Bot

Loading