Skip to content

[TRTLLM-12869][refactor] reuse encoder layouts with a compatible handoff - #20902

Draft
yechank-nvidia wants to merge 23 commits into
NVIDIA:mainfrom
yechank-nvidia:refactor/mm-handoff-compatible-20261023
Draft

yechank-nvidia wants to merge 23 commits into
NVIDIA:mainfrom
yechank-nvidia:refactor/mm-handoff-compatible-20261023

Conversation

@yechank-nvidia

@yechank-nvidia yechank-nvidia commented Oct 23, 2026 •

Copy link
Copy Markdown
Collaborator

Stacked on #20897 (bab3171542), after #20391. Review the handoff-only changes.

Description

Let prefill reuse an encoder's completed prompt/layout instead of rebuilding it. Encoded items and optional finalized layout use one versioned payload, with model-selected, P-owned state for decode.

Existing Dynamo remains compatible. Keep the existing DisaggregatedParams constructor fields, attributes, processor hook, and default result output. New consumers explicitly opt in:

legacy = encoded.disaggregated_params
versioned = encoded.export_multimodal_handoff()

The new method is a beta API addition. Both formats feed the common execution Params; postprocessing preserves the payload and each consumer's selected output. Existing APIs are not deprecated in this PR. Announce a transition period before a later removal.

Coordinated consumer: ai-dynamo/dynamo#17185, which also retains its older-backend fallback.

Test Coverage

  • Matching native build, H200 NVL: 1,085 passed, 7 skipped, 35 deselected, plus 2 subtests.
  • Existing executor/postprocessing regressions: 32 passed.
  • Compatibility matrix: unchanged Dynamo 144 passed with this backend; updated Dynamo 148 passed against both the old API and the new API.
  • Real Qwen3-VL-2B: AGG = legacy E/P = versioned E/P for one- and two-image requests, 12 generated tokens each; DeepStack width 8,192 preserved.
  • Pre-commit and production mypy roots passed. Full configured mypy 1.19.1 reports the same 12 sampler-test diagnostics on both split branches. API stability reference updated for the additive beta method; committed APIs remain unchanged.

AIPerf, peak-memory comparison, and GPU-isolated Dynamo E/P/D remain pending. Local NIXL P/D initialization did not complete on this node and is not counted as a pass. No new model/modality support or performance gain is claimed.

PR Checklist

  • Reviewed the applicable PR checklist items.

Signed-off-by: yechank <161688079+yechank-nvidia@users.noreply.github.com>
Signed-off-by: yechank <161688079+yechank-nvidia@users.noreply.github.com>
Signed-off-by: yechank <161688079+yechank-nvidia@users.noreply.github.com>
Signed-off-by: yechank <161688079+yechank-nvidia@users.noreply.github.com>
Signed-off-by: yechank <161688079+yechank-nvidia@users.noreply.github.com>
Use common item selection for partial-cache misses so prompt order, modality-local payload indices, and canonical output lengths stay aligned. Reuse validated request metadata and contiguous tensor views during input preparation.

Signed-off-by: yechank <161688079+yechank-nvidia@users.noreply.github.com>
Signed-off-by: yechank <161688079+yechank-nvidia@users.noreply.github.com>
Signed-off-by: yechank <161688079+yechank-nvidia@users.noreply.github.com>
Signed-off-by: yechank <161688079+yechank-nvidia@users.noreply.github.com>
Signed-off-by: yechank <161688079+yechank-nvidia@users.noreply.github.com>
Signed-off-by: yechank <161688079+yechank-nvidia@users.noreply.github.com>
Signed-off-by: yechank <161688079+yechank-nvidia@users.noreply.github.com>
Signed-off-by: yechank <161688079+yechank-nvidia@users.noreply.github.com>
Signed-off-by: yechank <161688079+yechank-nvidia@users.noreply.github.com>
…adata

Signed-off-by: yechank <161688079+yechank-nvidia@users.noreply.github.com>
… hashing

Signed-off-by: yechank <161688079+yechank-nvidia@users.noreply.github.com>
Signed-off-by: yechank <161688079+yechank-nvidia@users.noreply.github.com>
Signed-off-by: yechank <161688079+yechank-nvidia@users.noreply.github.com>
Signed-off-by: yechank <161688079+yechank-nvidia@users.noreply.github.com>
Signed-off-by: yechank <161688079+yechank-nvidia@users.noreply.github.com>
Signed-off-by: yechank <161688079+yechank-nvidia@users.noreply.github.com>
Signed-off-by: yechank <161688079+yechank-nvidia@users.noreply.github.com>
Signed-off-by: yechank <161688079+yechank-nvidia@users.noreply.github.com>

This branch has not been deployed

No deployments
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

api-compatible Accepted LLM API contract change that is backwards-compatible

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant