Skip to content

[None][fix] Allow fail-early when reuse block and legacy mamba cache #15540

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Closed
Wanli-Jiang wants to merge 1 commit into from
Closed

[None][fix] Allow fail-early when reuse block and legacy mamba cache #15540

Changes from all commits
Commits
Show all changes
1 commit
Select commit
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
26 changes: 11 additions & 15 deletions tensorrt_llm/_torch/pyexecutor/_util.py
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -316,22 +316,18 @@ def _get_model_kv_cache_manager_cls(
"KVCacheManagerV2 is not supported with %s. "
"Falling back to KVCacheManager.", incompat_str)
cls = KVCacheManager
# The V1-route hybrid mamba managers (disagg, TRTLLM_USE_CPP_MAMBA,

@Wanli-Jiang Wanli-Jiang Jul 22, 2026

Copy link
Copy Markdown
Collaborator Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

actually we can drop this part since no longer used.

# TRTLLM_USE_PY_MAMBA, or one-model speculative decoding) keep mamba
# state in a separate cache that doesn't honor block reuse. Warn at
# the routing site so users see the warning where the decision is
# actually made.
# Block reuse isn't honored by the legacy Mixed mamba manager (the unified
# CppMambaHybridCacheManager is, even under MTP). Fail fast if reuse was
# requested but routing still resolved to Mixed, rather than dropping it.
if is_hybrid_linear(model_engine.model.model_config.pretrained_config) \
and kv_cache_config.enable_block_reuse:
uses_v1_mamba_route = self._is_disagg \
or os.environ.get('TRTLLM_USE_CPP_MAMBA', '0') == '1' \
or os.environ.get('TRTLLM_USE_PY_MAMBA', '0') == '1' \
or self._speculative_config is not None
if uses_v1_mamba_route:
logger.warning(
"Block reuse does not work with MTP for hybrid linear models "
"when using the legacy MambaCacheManager (TRTLLM_USE_CPP_MAMBA=1)"
)
and kv_cache_config.enable_block_reuse \

@nv-guomingz nv-guomingz Jun 23, 2026

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Let's await this #15533 ci pipeline

and issubclass(cls, MixedMambaHybridCacheManager):
raise ValueError(
"enable_block_reuse=True is not supported by the legacy "
"MixedMambaHybridCacheManager selected for this hybrid linear "
"model. Unset TRTLLM_USE_CPP_MAMBA / TRTLLM_USE_PY_MAMBA to use "
"the unified CppMambaHybridCacheManager, which supports block "
"reuse (including under MTP).")
return cls

def _per_manager_cache_cost(self,
Expand Down
Loading