Skip to content

[None][fix] Split draft host cache during estimation - #17585

Draft
zheyuf wants to merge 1 commit into
NVIDIA:feat/m3_with_msafrom
zheyuf:agent/fix-draft-host-cache-estimation
Draft

[None][fix] Split draft host cache during estimation#17585
zheyuf wants to merge 1 commit into
NVIDIA:feat/m3_with_msafrom
zheyuf:agent/fix-draft-host-cache-estimation

Conversation

@zheyuf

@zheyuf zheyuf commented Aug 12, 2026

Copy link
Copy Markdown
Collaborator

@coderabbitai summary

Description

When one-engine speculative decoding creates separate target and draft KV cache managers, host_cache_size is a combined budget. The normal manager-construction path splits that budget by target/draft bytes per token, but the split was nested under the not estimating_kv_cache guard.

The estimation path still creates both host pools, so each manager temporarily received the full configured host budget. With a 192 GiB per-rank budget at TP8, startup could request up to 3 TiB of host memory on a 2 TiB node before reaching the later, correct split.

This change:

  • splits the host budget whenever separate draft and target managers are used, including estimation;
  • keeps GPU-budget splitting disabled during estimation, preserving the existing max-token sizing behavior; and
  • leaves the caller-owned base config unchanged.

The issue affects host offload with separate speculative draft/target KV managers; non-speculative models, shared-cache paths, and configurations without host cache are unaffected.

Test Coverage

  • Added test_estimation_splits_host_budget_before_creating_managers to cover the previously missed build_managers(..., estimating_kv_cache=True) lifecycle.
  • python -m pytest -q tests/unittest/_torch/executor/test_kv_cache_budget_split.py: 21 passed in the branch-m3-2026-08-12 nightly environment with this source file overlaid.
  • Ruff, YAPF, legacy lint baseline, and git diff --check pass for the modified files.
  • Bia TP8 Eagle3 PDL-off/on runtime smoke with the original 192 GiB per-rank setting: Slurm array 751970 (queued; results will be added to this draft).

PR Checklist

  • PR description clearly explains what and why.
  • PR follows the TRT-LLM coding guidelines to the best of my knowledge.
  • A regression test is provided for the new code path.
  • No API changes or new dependencies.
  • No CODEOWNERS, documentation, or architecture-diagram update is needed.

GitHub Bot Help

To see a list of available CI bot commands, comment /bot help.

Signed-off-by: Zheyu Fu <zheyuf@nvidia.com>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant