[P/D] NIXL Updates by robertgshaw2-redhat · Pull Request #25844 · vllm-project/vllm

robertgshaw2-redhat · 2025-09-28T21:26:49Z

Purpose

with 0.6 release, the parallel posting is now in mainline nixl. We need to opt into this in the final API. Using parallel posting improves post performance
update to ignore "double frees" in cases where requests are canceled

Test Plan

Test Result

Essential Elements of an Effective PR Description Checklist

The purpose of the PR, such as "Fix some issue (link existing issues this PR will resolve)".
The test plan, such as providing test command.
The test results, such as pasting the results comparison before and after, or e2e results
(Optional) The necessary documentation update, such as updating supported_models.md and examples for a new model.
(Optional) Release notes update. If your change is user facing, please update the release notes draft in the Google Doc.

Signed-off-by: Sage Moore <sage@neuralmagic.com> Signed-off-by: simon-mo <simon.mo@hey.com>

Signed-off-by: rentianyue-jk <rentianyue-jk@360shuke.com> Signed-off-by: Russell Bryant <rbryant@redhat.com> Co-authored-by: rentianyue-jk <rentianyue-jk@360shuke.com> Signed-off-by: simon-mo <simon.mo@hey.com>

Signed-off-by: Isotr0py <mozf@mail2.sysu.edu.cn> Co-authored-by: Isotr0py <mozf@mail2.sysu.edu.cn> Signed-off-by: simon-mo <simon.mo@hey.com>

Signed-off-by: Chenheli Hua <huachenheli@outlook.com> Signed-off-by: Russell Bryant <rbryant@redhat.com> Co-authored-by: Chenheli Hua <huachenheli@outlook.com> Signed-off-by: simon-mo <simon.mo@hey.com>

Signed-off-by: simon-mo <simon.mo@hey.com>

Signed-off-by: mgoin <mgoin64@gmail.com> Signed-off-by: simon-mo <simon.mo@hey.com>

Signed-off-by: Tyler Michael Smith <tlrmchlsmth@gmail.com> Signed-off-by: simon-mo <simon.mo@hey.com>

Signed-off-by: NickLucche <nlucches@redhat.com> Signed-off-by: simon-mo <simon.mo@hey.com>

…25810) Signed-off-by: Roger Wang <hey@rogerw.io> Signed-off-by: simon-mo <simon.mo@hey.com>

Signed-off-by: Roger Wang <hey@rogerw.io> Signed-off-by: simon-mo <simon.mo@hey.com>

…le video profiling (#25557) Signed-off-by: Isotr0py <mozf@mail2.sysu.edu.cn> Signed-off-by: Roger Wang <hey@rogerw.io> Co-authored-by: Roger Wang <hey@rogerw.io> Signed-off-by: simon-mo <simon.mo@hey.com>

Signed-off-by: Robert Shaw <robshaw@redhat.com>

tlrmchlsmth · 2025-09-29T00:11:26Z

vllm/distributed/kv_transfer/kv_connector/v1/nixl_connector.py

            remote_xfer_side_handle,
            remote_block_descs_ids,
            notif_msg=notif_id,
+            skip_desc_merge=True,


is this debug cruft?

Signed-off-by: Robert Shaw <robshaw@redhat.com>

Signed-off-by: Sage Moore <sage@neuralmagic.com> Signed-off-by: simon-mo <simon.mo@hey.com> Signed-off-by: rentianyue-jk <rentianyue-jk@360shuke.com> Signed-off-by: Russell Bryant <rbryant@redhat.com> Signed-off-by: Isotr0py <mozf@mail2.sysu.edu.cn> Signed-off-by: Chenheli Hua <huachenheli@outlook.com> Signed-off-by: mgoin <mgoin64@gmail.com> Signed-off-by: Tyler Michael Smith <tlrmchlsmth@gmail.com> Signed-off-by: NickLucche <nlucches@redhat.com> Signed-off-by: Roger Wang <hey@rogerw.io> Signed-off-by: Robert Shaw <robshaw@redhat.com> Co-authored-by: Sage Moore <sage@neuralmagic.com> Co-authored-by: Russell Bryant <rbryant@redhat.com> Co-authored-by: rentianyue-jk <rentianyue-jk@360shuke.com> Co-authored-by: Isotr0py <mozf@mail2.sysu.edu.cn> Co-authored-by: Chenheli Hua <huachenheli@outlook.com> Co-authored-by: Wentao Ye <44945378+yewentao256@users.noreply.github.com> Co-authored-by: Michael Goin <mgoin64@gmail.com> Co-authored-by: Tyler Michael Smith <tlrmchlsmth@gmail.com> Co-authored-by: Nicolò Lucchesi <nlucches@redhat.com> Co-authored-by: Roger Wang <hey@rogerw.io> Co-authored-by: Robert Shaw <robshaw@redhat.com> Signed-off-by: simon-mo <simon.mo@hey.com>

markmc · 2025-09-29T11:04:25Z

vllm/v1/core/sched/scheduler.py

+            if req_id not in self.requests:
+                logger.warning(
+                    "Got finished sending KV transfer for request %s,"
+                    "but the request is already freed.", req_id)


How would you unit test this scenario?

In #25067 the case I tested was an abort after the prefill request had finished, but @NickLucche rightly asked (AIUI):

If P is done with the request (finished, capped length), then how is the request being aborted in P ?

Whatever the scenario is ... if it is supposed to happen, we shouldn't have a warning that is not actionable by the user. But a "why this is supposed to happen" comment would be important for maintainability. Something similar to

vllm/vllm/v1/core/sched/scheduler.py

Lines 888 to 893 in 8616300

request = self.requests.get(req_id)

if request is None:

# The request is already finished. This can happen if the

# request is aborted while the model is executing it (e.g.,

# in pipeline parallelism).

continue

I agree this should be a debug log.

I will post up a diff later today that:

adds commentary about what scenarios this can be caused by

adds a unit test to the scheduler tests we have for this

I'm working on this in #25067 again

Signed-off-by: Sage Moore <sage@neuralmagic.com> Signed-off-by: simon-mo <simon.mo@hey.com> Signed-off-by: rentianyue-jk <rentianyue-jk@360shuke.com> Signed-off-by: Russell Bryant <rbryant@redhat.com> Signed-off-by: Isotr0py <mozf@mail2.sysu.edu.cn> Signed-off-by: Chenheli Hua <huachenheli@outlook.com> Signed-off-by: mgoin <mgoin64@gmail.com> Signed-off-by: Tyler Michael Smith <tlrmchlsmth@gmail.com> Signed-off-by: NickLucche <nlucches@redhat.com> Signed-off-by: Roger Wang <hey@rogerw.io> Signed-off-by: Robert Shaw <robshaw@redhat.com> Co-authored-by: Sage Moore <sage@neuralmagic.com> Co-authored-by: Russell Bryant <rbryant@redhat.com> Co-authored-by: rentianyue-jk <rentianyue-jk@360shuke.com> Co-authored-by: Isotr0py <mozf@mail2.sysu.edu.cn> Co-authored-by: Chenheli Hua <huachenheli@outlook.com> Co-authored-by: Wentao Ye <44945378+yewentao256@users.noreply.github.com> Co-authored-by: Michael Goin <mgoin64@gmail.com> Co-authored-by: Tyler Michael Smith <tlrmchlsmth@gmail.com> Co-authored-by: Nicolò Lucchesi <nlucches@redhat.com> Co-authored-by: Roger Wang <hey@rogerw.io> Co-authored-by: Robert Shaw <robshaw@redhat.com>

Signed-off-by: Sage Moore <sage@neuralmagic.com> Signed-off-by: simon-mo <simon.mo@hey.com> Signed-off-by: rentianyue-jk <rentianyue-jk@360shuke.com> Signed-off-by: Russell Bryant <rbryant@redhat.com> Signed-off-by: Isotr0py <mozf@mail2.sysu.edu.cn> Signed-off-by: Chenheli Hua <huachenheli@outlook.com> Signed-off-by: mgoin <mgoin64@gmail.com> Signed-off-by: Tyler Michael Smith <tlrmchlsmth@gmail.com> Signed-off-by: NickLucche <nlucches@redhat.com> Signed-off-by: Roger Wang <hey@rogerw.io> Signed-off-by: Robert Shaw <robshaw@redhat.com> Co-authored-by: Sage Moore <sage@neuralmagic.com> Co-authored-by: Russell Bryant <rbryant@redhat.com> Co-authored-by: rentianyue-jk <rentianyue-jk@360shuke.com> Co-authored-by: Isotr0py <mozf@mail2.sysu.edu.cn> Co-authored-by: Chenheli Hua <huachenheli@outlook.com> Co-authored-by: Wentao Ye <44945378+yewentao256@users.noreply.github.com> Co-authored-by: Michael Goin <mgoin64@gmail.com> Co-authored-by: Tyler Michael Smith <tlrmchlsmth@gmail.com> Co-authored-by: Nicolò Lucchesi <nlucches@redhat.com> Co-authored-by: Roger Wang <hey@rogerw.io> Co-authored-by: Robert Shaw <robshaw@redhat.com> Signed-off-by: yewentao256 <zhyanwentao@126.com>

We have observed a rare scenario with AsyncLLM where a client disconnect triggers an abort request after the request has finished, but before AsyncLLM has processed the request output. See vllm-project#26012, vllm-project#25067, vllm-project#25844, and llm-d/llm-d#187. Without the fix, the unit test fails with: ``` logger.warning( "Releasing expired KV blocks for request %s which were " "retrieved by %d decode worker(s) within %d seconds.", req_id, count, envs.VLLM_NIXL_ABORT_REQUEST_TIMEOUT, ) > self._reqs_to_process.remove(req_id) E KeyError: '0' vllm/distributed/kv_transfer/kv_connector/v1/nixl_connector.py:1238: KeyError ``` Signed-off-by: Mark McLoughlin <markmc@redhat.com>

This situation can occur when the API server receives a client disconnect (and thus sends an abort) around the same time a prefill completes and we keep the blocks (delay_free_blocks) around for a remote decode. We should assume the blocks may be used, and so we ignore the abort. If they are not used, they should be freed by the connector after a timeout. The original error was: ``` [scheduler.py:1183] Finished sending KV transfer for request cmpl-37c560d3-5680-4bd1-97f9-7ed31a56de60-0 File "/opt/vllm-source/vllm/v1/engine/core.py", line 292, in step engine_core_outputs = self.scheduler.update_from_output( ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/opt/vllm-source/vllm/v1/core/sched/scheduler.py", line 893, in update_from_output self._update_from_kv_xfer_finished( File "/opt/vllm-source/vllm/v1/core/sched/scheduler.py", line 1184, in _update_from_kv_xfer_finish> self._free_blocks(self.requests[req_id]) ~~~~~~~~~~~~~^^^^^^^^ KeyError: 'cmpl-37c560d3-5680-4bd1-97f9-7ed31a56de60-0' ``` But since vllm-project#25844 we would log a warning. This fix makes it so that situation in `_update_from_kv_xfer_finish()` should never occur. Signed-off-by: Mark McLoughlin <markmc@redhat.com>

Signed-off-by: Sage Moore <sage@neuralmagic.com> Signed-off-by: simon-mo <simon.mo@hey.com> Signed-off-by: rentianyue-jk <rentianyue-jk@360shuke.com> Signed-off-by: Russell Bryant <rbryant@redhat.com> Signed-off-by: Isotr0py <mozf@mail2.sysu.edu.cn> Signed-off-by: Chenheli Hua <huachenheli@outlook.com> Signed-off-by: mgoin <mgoin64@gmail.com> Signed-off-by: Tyler Michael Smith <tlrmchlsmth@gmail.com> Signed-off-by: NickLucche <nlucches@redhat.com> Signed-off-by: Roger Wang <hey@rogerw.io> Signed-off-by: Robert Shaw <robshaw@redhat.com> Co-authored-by: Sage Moore <sage@neuralmagic.com> Co-authored-by: Russell Bryant <rbryant@redhat.com> Co-authored-by: rentianyue-jk <rentianyue-jk@360shuke.com> Co-authored-by: Isotr0py <mozf@mail2.sysu.edu.cn> Co-authored-by: Chenheli Hua <huachenheli@outlook.com> Co-authored-by: Wentao Ye <44945378+yewentao256@users.noreply.github.com> Co-authored-by: Michael Goin <mgoin64@gmail.com> Co-authored-by: Tyler Michael Smith <tlrmchlsmth@gmail.com> Co-authored-by: Nicolò Lucchesi <nlucches@redhat.com> Co-authored-by: Roger Wang <hey@rogerw.io> Co-authored-by: Robert Shaw <robshaw@redhat.com> Signed-off-by: simon-mo <simon.mo@hey.com>

Signed-off-by: Sage Moore <sage@neuralmagic.com> Signed-off-by: simon-mo <simon.mo@hey.com> Signed-off-by: rentianyue-jk <rentianyue-jk@360shuke.com> Signed-off-by: Russell Bryant <rbryant@redhat.com> Signed-off-by: Isotr0py <mozf@mail2.sysu.edu.cn> Signed-off-by: Chenheli Hua <huachenheli@outlook.com> Signed-off-by: mgoin <mgoin64@gmail.com> Signed-off-by: Tyler Michael Smith <tlrmchlsmth@gmail.com> Signed-off-by: NickLucche <nlucches@redhat.com> Signed-off-by: Roger Wang <hey@rogerw.io> Signed-off-by: Robert Shaw <robshaw@redhat.com> Co-authored-by: Sage Moore <sage@neuralmagic.com> Co-authored-by: Russell Bryant <rbryant@redhat.com> Co-authored-by: rentianyue-jk <rentianyue-jk@360shuke.com> Co-authored-by: Isotr0py <mozf@mail2.sysu.edu.cn> Co-authored-by: Chenheli Hua <huachenheli@outlook.com> Co-authored-by: Wentao Ye <44945378+yewentao256@users.noreply.github.com> Co-authored-by: Michael Goin <mgoin64@gmail.com> Co-authored-by: Tyler Michael Smith <tlrmchlsmth@gmail.com> Co-authored-by: Nicolò Lucchesi <nlucches@redhat.com> Co-authored-by: Roger Wang <hey@rogerw.io> Co-authored-by: Robert Shaw <robshaw@redhat.com>

SageMoore and others added 14 commits September 27, 2025 23:32

Reduce the Cuda Graph memory footprint when running with DBO (#25779)

bb79c4d

Signed-off-by: Sage Moore <sage@neuralmagic.com> Signed-off-by: simon-mo <simon.mo@hey.com>

Validate API tokens in constant time (#25781)

ee10d7e

Signed-off-by: rentianyue-jk <rentianyue-jk@360shuke.com> Signed-off-by: Russell Bryant <rbryant@redhat.com> Co-authored-by: rentianyue-jk <rentianyue-jk@360shuke.com> Signed-off-by: simon-mo <simon.mo@hey.com>

Add filtering for chat template kwargs (#25794)

04c2b26

Signed-off-by: Isotr0py <mozf@mail2.sysu.edu.cn> Co-authored-by: Isotr0py <mozf@mail2.sysu.edu.cn> Signed-off-by: simon-mo <simon.mo@hey.com>

Add option to restrict media domains (#25783)

32335c8

Signed-off-by: Chenheli Hua <huachenheli@outlook.com> Signed-off-by: Russell Bryant <rbryant@redhat.com> Co-authored-by: Chenheli Hua <huachenheli@outlook.com> Signed-off-by: simon-mo <simon.mo@hey.com>

[Bugfix] Allow Only SDPA Backend for ViT on B200 for Qwen3-VL (#25788)

c2fa2d4

Signed-off-by: simon-mo <simon.mo@hey.com>

[CI] Fix FlashInfer AOT in release docker image (#25730)

5aa5811

Signed-off-by: mgoin <mgoin64@gmail.com> Signed-off-by: simon-mo <simon.mo@hey.com>

[Bugfix][WideEP] Apply TP Attn + EP MoE fix to other models (#24982)

26a7a33

Signed-off-by: Tyler Michael Smith <tlrmchlsmth@gmail.com> Signed-off-by: simon-mo <simon.mo@hey.com>

[Bugfix][NIXL] Fix Async Scheduler timeout issue (#25808)

b14773b

Signed-off-by: NickLucche <nlucches@redhat.com> Signed-off-by: simon-mo <simon.mo@hey.com>

[MM] Optimize memory profiling for scattered multimodal embeddings (#…

6de3d43

…25810) Signed-off-by: Roger Wang <hey@rogerw.io> Signed-off-by: simon-mo <simon.mo@hey.com>

[Bugfix] Fix Qwen3-VL regression from #24982 (#25814)

19e7ab7

Signed-off-by: Roger Wang <hey@rogerw.io> Signed-off-by: simon-mo <simon.mo@hey.com>

[VLM] Update Qwen3-VL max_num_video_tokens calculation for configurab…

4c34704

…le video profiling (#25557) Signed-off-by: Isotr0py <mozf@mail2.sysu.edu.cn> Signed-off-by: Roger Wang <hey@rogerw.io> Co-authored-by: Roger Wang <hey@rogerw.io> Signed-off-by: simon-mo <simon.mo@hey.com>

updated

6fc2f1c

Signed-off-by: Robert Shaw <robshaw@redhat.com>

updated

87d6b8b

Signed-off-by: Robert Shaw <robshaw@redhat.com>

merged

a46e572

Signed-off-by: Robert Shaw <robshaw@redhat.com>

mergify bot added v1 kv-connector labels Sep 28, 2025

Robert Shaw added 2 commits September 28, 2025 23:34

updated

6ec0c55

Signed-off-by: Robert Shaw <robshaw@redhat.com>

updated

74dc751

Signed-off-by: Robert Shaw <robshaw@redhat.com>

robertgshaw2-redhat changed the title ~~test~~ [P/D] NIXL Updates Sep 29, 2025

robertgshaw2-redhat marked this pull request as ready for review September 29, 2025 00:07

robertgshaw2-redhat requested review from ApostaC, NickLucche, WoosukKwon, alexm-redhat, comaniac, heheda12345, njhill and ywang96 as code owners September 29, 2025 00:07

robertgshaw2-redhat added this to the v0.11.0 Cherry Picks milestone Sep 29, 2025

tlrmchlsmth reviewed Sep 29, 2025

View reviewed changes

updated

4181ef7

Signed-off-by: Robert Shaw <robshaw@redhat.com>

tlrmchlsmth approved these changes Sep 29, 2025

View reviewed changes

tlrmchlsmth added the ready ONLY add when PR is ready to merge/full CI is needed label Sep 29, 2025

robertgshaw2-redhat enabled auto-merge (squash) September 29, 2025 03:35

robertgshaw2-redhat merged commit 9b44a7d into main Sep 29, 2025
50 checks passed

robertgshaw2-redhat deleted the nixl-bump branch September 29, 2025 04:46

markmc reviewed Sep 29, 2025

View reviewed changes

This was referenced Oct 7, 2025

[NIXL] Fix KeyError on abort-after-finished #26351

Closed

[NIXL] Ignore abort on already-finished request #25067

Merged

kouroshHakha mentioned this pull request Oct 7, 2025

[Bug][NIXL][PD]: Revisiting NIXL slowness on CUDA_IPC -- installation problem #26382

Closed

1 task

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Uh oh!

[P/D] NIXL Updates#25844

[P/D] NIXL Updates#25844
robertgshaw2-redhat merged 17 commits intomainfrom
nixl-bump

robertgshaw2-redhat commented Sep 28, 2025 •

edited by github-actions bot

Loading

Uh oh!

tlrmchlsmth Sep 29, 2025

Uh oh!

robertgshaw2-redhat Sep 29, 2025

Uh oh!

Uh oh!

markmc Sep 29, 2025

Uh oh!

robertgshaw2-redhat Sep 29, 2025

Uh oh!

markmc Oct 6, 2025

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

10 participants

	request = self.requests.get(req_id)
	if request is None:
	# The request is already finished. This can happen if the
	# request is aborted while the model is executing it (e.g.,
	# in pipeline parallelism).
	continue

Uh oh!

Conversation

robertgshaw2-redhat commented Sep 28, 2025 • edited by github-actions bot Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Purpose

Test Plan

Test Result

Uh oh!

tlrmchlsmth Sep 29, 2025

Choose a reason for hiding this comment

Uh oh!

robertgshaw2-redhat Sep 29, 2025

Choose a reason for hiding this comment

Uh oh!

Uh oh!

markmc Sep 29, 2025

Choose a reason for hiding this comment

Uh oh!

robertgshaw2-redhat Sep 29, 2025

Choose a reason for hiding this comment

Uh oh!

markmc Oct 6, 2025

Choose a reason for hiding this comment

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

10 participants

robertgshaw2-redhat commented Sep 28, 2025 •

edited by github-actions bot

Loading