Skip to content

[fix] Release slots with spec decode + disagg #5975

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Merged
schetlur-nv merged 2 commits into from
Jul 14, 2025
Merged

[fix] Release slots with spec decode + disagg #5975

Changes from all commits
Commits
Show all changes
2 commits
Select commit Hold shift + click to select a range
3dee5fb
[fix] Release slots with spec decode + disagg
Tabrizian Jul 11, 2025
771d29d
Remove drafter
Tabrizian Jul 14, 2025
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
12 changes: 8 additions & 4 deletions tensorrt_llm/_torch/pyexecutor/py_executor.py
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -869,8 +869,7 @@ def _executor_loop(self):

self._pad_attention_dp_dummy_request()

if self.draft_model_engine is not None or is_ngram or hasattr(
self, 'drafter') and self.drafter is not None:
if self.draft_model_engine is not None or is_ngram:
self._prepare_draft_requests(self.active_requests)

scheduled_batch, fitting_disagg_gen_init_requests, num_fitting_reqs = self._schedule(
Expand Down Expand Up @@ -1595,8 +1594,13 @@ def _send_disagg_ctx_cache(self, scheduled_ctx_requests):
if req.is_context_only_request and (req.is_context_finished or
req.is_finished_due_to_length):
self.kv_cache_transceiver.respond_and_send_async(req)
self.resource_manager.resource_managers[
ResourceManagerType.SEQ_SLOT_MANAGER].free_resources(req)
for resource_mgr_type in (
ResourceManagerType.SEQ_SLOT_MANAGER,
ResourceManagerType.SPEC_RESOURCE_MANAGER):
if resource_mgr_type in self.resource_manager.resource_managers and self.resource_manager.resource_managers[
resource_mgr_type] is not None:
self.resource_manager.resource_managers[
resource_mgr_type].free_resources(req)

self.kv_cache_transceiver.check_context_transfer_status(0)

Expand Down