[Bugfix][NIXL] Harden handshake listener against bad requests

[jdonkervliet]

Purpose

The prefill-side NIXL handshake listener (_nixl_handshake_listener in base_scheduler.py) is a single background thread, per engine, that lives for the whole process and answers metadata queries from every decode worker that needs to pull KV cache from it. Its only exception handling covered the idle-poll timeout; any exception raised while actually processing a received message killed the thread silently, with nothing to restart or report it.

This fails whenever anything reaches the side-channel port that isn't a well-formed request from a real REQ client: a malformed message, a request for a pp/tp rank this engine doesn't serve, or a peer that doesn't speak the REQ/ROUTER envelope (a REQ socket always auto-adds an empty delimiter frame; anything that doesn't, such as a bare DEALER socket, produces a different frame count on the ROUTER side). Once the thread dies, every subsequent handshake against that engine times out and fails for every decode worker until the process is restarted: a single bad request causing a lasting, silent, engine-wide DoS.

This isn't tied to an existing filed issue. It was found via production investigation, reproduced locally, and checked against vllm-project/vllm (issues, PRs, current main) for existing work on this listener's exception handling; nothing addresses it. The closest related work, vllm-project#48633 and vllm-project#49238 / vllm-project#50047, covers different NIXL reliability gaps (push-mode handshake latency, stale peer state after a restart), not this one.

Changes:

• base_scheduler.py: move the frame unpacking and message handling into a guarded block, narrowed to the concrete failure modes a bad request can trigger (msgspec.DecodeError, ValueError/TypeError, KeyError, zmq.ZMQError). A bad request is now logged (with the raw bytes that triggered it, safely escaped and capped) and dropped, instead of taking the whole side channel down. Anything outside that list still crashes the thread loudly, since that's a genuine bug rather than a bad peer.
• base_worker.py: wrap the decode-side handshake timeout (zmq.error.Again) with the remote address/rank and a pointer to the existing side-channel probe script (tests/v1/kv_connector/nixl_integration/nixl_side_channel_probe.py), since ZMQ hides the underlying connection state and the bare timeout gives no way to triage which of several possible causes it was.
• tests/v1/kv_connector/unit/test_nixl_connector.py: two new regression tests (test_handshake_listener_survives_bad_envelope, test_handshake_listener_survives_malformed_content) that run the real listener against a live socket and a stray non-REQ client. Verified both fail against the unpatched listener (thread dies) and pass against the fix.

AI assistance was used to investigate, reproduce, and implement this fix.

Test Plan

• ruff check / ruff format --check / pyright on changed files.
• python -m pytest tests/v1/kv_connector/unit/test_nixl_connector.py -v (full file).
• python -m pytest tests/v1/kv_connector/unit/test_bidirectional_kv_transfer.py -v (full file; calls _nixl_handshake and _nixl_handshake_listener directly).
• Live reproduction: ran the actual (unpatched) _nixl_handshake_listener against a real ROUTER socket, connected with a DEALER client (no REQ envelope) to reproduce the crash, then reran the same scenario against the patched listener to confirm it survives and keeps serving; now captured permanently as the two new regression tests above.

Test Result

• ruff/pyright: clean (0 errors).
• Full suites, run on NVIDIA GPU hardware:

  python -m pytest tests/v1/kv_connector/unit/test_nixl_connector.py tests/v1/kv_connector/unit/test_bidirectional_kv_transfer.py -v
  ================= 140 passed, 16 warnings in 207.38s (0:03:27) =================
  

• New regression tests confirmed to fail against the unpatched listener (ValueError/msgspec.DecodeError, thread dies) and pass against the fix.

---

Essential Elements of an Effective PR Description Checklist

• The purpose of the PR.
• The test plan, with test commands.
• The test results, including full-suite results on GPU hardware.
• (N/A) Documentation update.

[jdonkervliet]

Closing and reopening to force GitHub to recompute a stuck diff cache.

[jdonkervliet]

Opened upstream as vllm-project#54757.