[Bugfix][NIXL] Avoid receive reports for notification-only requests by NickLucche · Pull Request #57570 · vllm-project/vllm

NickLucche · 2026-09-18T14:05:17Z

A notif-only decode request (abort/full prefix cache hit) can still need to notify the prefiller to release KV.
If that notification's handshake fails, NIXL reports a failed receive for a request the scheduler has already removed, crashing EngineCore.
See "CI issue example" for how this relates to Buildkite 89597.

Keep notification-only entries out of receive tracking. This also covers full local prefix hits, where the KV is already local and the scheduler never waits for a receive. Producer notification is still attempted, but its failure no longer reports a receive failure or completion.

Actual receives retain both signals: completion is needed to release blocks under fail or resume scheduling under recompute.

Cancellation example

Prefiller P has computed KV for request A.
Decoder D cancels A before scheduling it. D removes A from the scheduler, but queues a notification telling P to release its KV. This has awaiting_kvs=False and no local receive blocks.
The worker attempts the handshake needed to notify P. The handshake fails—for example, because of a block-size mismatch.
Before the fix: A was registered as a receive, so the worker reports both failed_recving={A} and finished_recving={A}. The scheduler processes the completion, cannot find A, and crashes.
After the fix: A isn’t registered as a receive. The notification is still attempted and its failure logged, but no receive failure or completion reaches the scheduler.

CI issue example

In Buildkite 89597, a four-block KV receive fails because of the 1920/1024 block-size mismatch (expected for SSMs has block size is tp dependent).

While returning the generation error, D's serving wrapper submits a separate cleanup request with abort_immediately=True and the same producer KV reference.

That request never runs inference: it only tells P to release its KV. Its handshake hits the same mismatch, and NIXL reports a failed receive for a request already removed from the scheduler, crashing EngineCore.

P computes KV for request A. D's attempt to receive it fails; A finishes with an error.
D's serving error-cleanup wrapper submits a separate request B with abort_immediately=True, carrying A's producer KV reference.
The engine immediately aborts B. The connector queues an empty receive solely to notify P to release its blocks: awaiting_kvs=False, no local receive blocks. B is removed from the scheduler.
B's notification handshake hits the same block-size mismatch. Before the fix, the worker reports failed_recving={B} and finished_recving={B}; the scheduler cannot find B and crashes.
After the fix, B is not tracked as a receive. Notification is still attempted and its failure logged, but no receive failure or completion reaches the scheduler.

cc @njhill @jyizheng since you guys introduced awaiting_kvs

On Buildkite

The Buildkite adjustment is orthogonal, included in this PR to unblock the release: set both Kimi DP/EP block sizes to 2048 to avoid the 1920/1024 handshake mismatch

Co-authored-by: OpenAI Codex <codex@openai.com> Signed-off-by: NickLucche <nicolo.lucchesi@mistral.ai>

claude

Claude Code Review

This pull request is from a fork — automated review is disabled. A repository maintainer can comment @claude review to run a one-time review.

NickLucche · 2026-09-18T14:19:36Z

/ci run all

github-actions · 2026-09-18T14:19:55Z

❌ This PR is 1 commit behind upstream main. Your branch must contain every commit currently on upstream main. No new CI build was started. Merge or rebase onto the latest main, then rerun /ci run all. To test this branch at your own risk, use /ci run all --allow-stale.

NickLucche · 2026-09-18T14:32:11Z

/ci run

NickLucche · 2026-09-18T15:13:49Z

/ci run

NickLucche · 2026-09-18T15:34:40Z

/ci run

github-actions · 2026-09-18T15:34:58Z

❌ This PR is 1 commit behind upstream main. Your branch must contain every commit currently on upstream main. No new CI build was started. Merge or rebase onto the latest main, then rerun /ci run. To test this branch at your own risk, use /ci run --allow-stale.

NickLucche · 2026-09-18T15:35:10Z

/ci run --allow-stale

github-actions · 2026-09-18T15:35:31Z

✅ Triggered Buildkite CI #89872 for commit 50f9fd726ca4.

⚠️ This PR is 1 commit behind upstream main. Running CI at your own risk because --allow-stale was requested; outdated CI configuration may cause failures. Before merging, merge or rebase onto the latest main, then rerun /ci run on the latest PR commit.

njhill

Thanks @NickLucche

…57570) Signed-off-by: NickLucche <nicolo.lucchesi@mistral.ai> Co-authored-by: OpenAI Codex <codex@openai.com>

…llm-project#57570) Signed-off-by: NickLucche <nicolo.lucchesi@mistral.ai> Co-authored-by: OpenAI Codex <codex@openai.com>

Conflict resolution (30 files): - 19 files: adopted v0.32.0 (our matching cherry-picks vllm-project#57554 vllm-project#56545 vllm-project#57871 vllm-project#53187 vllm-project#57049 vllm-project#57077 vllm-project#57570 are now upstream). - Re-reverted vllm-project#53007 (attention.py / kv_cache_utils.py / test_backend_per_kind.py): page_budget=shared_page, restore _largest_kernel_block_within candidates logic. - flashinfer.py: adopted upstream SM12x NVFP4 KV (vllm-project#46963/vllm-project#60452); re-applied our D256 prefill (_nvfp4_d256_prefill + FIPrefill fields + call sites). - kernel_warmup.py: adopted upstream autotune refactor; re-applied our W4A16 CuTe-DSL autotune skip. - qwen3_dflash.py: adopted upstream quantized-draft KV fallback (dropped our NVFP4 fused dequant fast-path; DFlash2 is backup). - dflash/eagle utils.py: kept our shared-cache cache_config handling (avoid replace-copy layout loss) + upstream get_draft_load_config. - entrypoints: union imports (our per-request logging + upstream additions).

…llm-project#57570) Signed-off-by: NickLucche <nicolo.lucchesi@mistral.ai> Co-authored-by: OpenAI Codex <codex@openai.com>

NickLucche and others added 2 commits September 18, 2026 16:03

[Bugfix][NIXL] Exclude notification-only requests from receive tracking

c98f1c8

Co-authored-by: OpenAI Codex <codex@openai.com> Signed-off-by: NickLucche <nicolo.lucchesi@mistral.ai>

[CI] Match Kimi disaggregated prefill and decode block sizes

7db42e6

Co-authored-by: OpenAI Codex <codex@openai.com> Signed-off-by: NickLucche <nicolo.lucchesi@mistral.ai>

mergify Bot added ci/build bug Something isn't working kv-connector labels Sep 18, 2026

NickLucche marked this pull request as ready for review September 18, 2026 14:18

NickLucche requested review from ApostaC, Harry-Chen, ivanium, khluu, orozery and xuechendi as code owners September 18, 2026 14:18

claude Bot reviewed Sep 18, 2026

View reviewed changes

Merge branch 'main' into fix/nixl-notify-only-receive-tracking

51a0408

NickLucche added this to the v0.30.0 cherry picks milestone Sep 18, 2026

Merge branch 'main' into fix/nixl-notify-only-receive-tracking

0ac22e5

Merge branch 'main' into fix/nixl-notify-only-receive-tracking

50f9fd7

njhill approved these changes Sep 18, 2026

View reviewed changes

khluu merged commit 49e0427 into vllm-project:main Sep 18, 2026
61 checks passed

khluu pushed a commit that referenced this pull request Sep 18, 2026

[Bugfix][NIXL] Avoid receive reports for notification-only requests (#…

fa6ff06

…57570) Signed-off-by: NickLucche <nicolo.lucchesi@mistral.ai> Co-authored-by: OpenAI Codex <codex@openai.com>

njhill mentioned this pull request Sep 26, 2026

[Bugfix][NIXL] Release a dead peer's NIXL state without waiting for TTL #50047

Merged

TianDi101 mentioned this pull request Oct 13, 2026

[RFC][IO] Framework-side RAS for MoRI-IO: harden vLLM's MoRIIO KV connector ROCm/mori#759

Open

13 tasks

Navigation Menu

Repository navigation

[Bugfix][NIXL] Avoid receive reports for notification-only requests - #57570

[Bugfix][NIXL] Avoid receive reports for notification-only requests#57570
khluu merged 5 commits into
vllm-project:mainfrom
NickLucche:fix/nixl-notify-only-receive-tracking

NickLucche commented Sep 18, 2026 •

edited

Loading

Uh oh!

claude Bot left a comment

Uh oh!

NickLucche commented Sep 18, 2026

Uh oh!

github-actions Bot commented Sep 18, 2026

Uh oh!

NickLucche commented Sep 18, 2026

Uh oh!

NickLucche commented Sep 18, 2026

Uh oh!

NickLucche commented Sep 18, 2026

Uh oh!

github-actions Bot commented Sep 18, 2026

Uh oh!

NickLucche commented Sep 18, 2026

Uh oh!

github-actions Bot commented Sep 18, 2026

Uh oh!

njhill left a comment

Uh oh!

Uh oh!

Reviewers

Uh oh!

Assignees

Labels

Projects

Milestone

Development

Uh oh!

3 participants

Conversation

NickLucche commented Sep 18, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Cancellation example

CI issue example

On Buildkite

Uh oh!

claude Bot left a comment

Choose a reason for hiding this comment

Claude Code Review

Uh oh!

NickLucche commented Sep 18, 2026

Uh oh!

github-actions Bot commented Sep 18, 2026

Uh oh!

NickLucche commented Sep 18, 2026

Uh oh!

NickLucche commented Sep 18, 2026

Uh oh!

NickLucche commented Sep 18, 2026

Uh oh!

github-actions Bot commented Sep 18, 2026

Uh oh!

NickLucche commented Sep 18, 2026

Uh oh!

github-actions Bot commented Sep 18, 2026

Uh oh!

njhill left a comment

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Reviewers

Uh oh!

Assignees

Labels

Projects

Milestone

Development

Uh oh!

3 participants

NickLucche commented Sep 18, 2026 •

edited

Loading