[Feat][NIXL] Add KV lease refresh mechanism for disaggregated prefill

[robertgshaw2-redhat]

Summary

• Problem: The static VLLM_NIXL_ABORT_REQUEST_TIMEOUT on the P side causes premature KV block expiry when the D queue is bursty, and wastes memory when set large enough to be safe.
• Solution: Replace it with an active lease mechanism — D workers periodically refresh the lease while requests are queued, so P only frees blocks if D truly goes silent (crashed or failed).

Key changes

• D scheduler (NixlConnectorScheduler): tracks all do_remote_prefill=True requests in _requires_lease_dict (req_id → P host/http_port) until they are scheduled. A daemon thread (nixl-d-lease-refresh) POSTs to P's /internal/nixl/lease_refresh every timeout // 3 seconds.
• P API server: new route POST /internal/nixl/lease_refresh (registered unconditionally; no-op on non-NIXL instances) calls engine_client.call_utility_async("nixl_lease_refresh", request_ids).
• P EngineCore: new nixl_lease_refresh() utility method delegates to connector.refresh_lease(), which runs in the engine loop thread so no locking is needed.
• P scheduler (NixlConnectorScheduler.refresh_lease): updates _lease_refreshes[req_id] = now + timeout; passed to the worker via NixlConnectorMetadata.reqs_to_refresh each step.
• P worker (NixlConnectorWorker): applies refreshes in start_load_kv; get_finished now does a full scan instead of early-break (expiry order is no longer guaranteed monotone after refreshes).
• New env var VLLM_NIXL_HTTP_PORT (default 8000): tells D where P's HTTP server is; P includes remote_http_port in kv_transfer_params.

Thread safety

State	Writers	Readers	Lock
_requires_lease_dict	scheduler thread	lease-refresh thread	_requires_lease_lock
_lease_refreshes	engine loop (refresh_lease)	engine loop (build_connector_meta)	none (single thread)

Test plan

• test_abort_timeout_on_prefiller — unchanged behaviour: D never adds the request to _requires_lease_dict (no remote_host), so P expires after VLLM_NIXL_ABORT_REQUEST_TIMEOUT as before
• test_disagg_accuracy.py — D refreshes leases while requests queue; transfer completes normally
• Manual: kill D mid-queue, verify P frees blocks after timeout

🤖 Generated with Claude Code

[gemini-code-assist]

Code Review

This pull request introduces a KV lease refresh mechanism for disaggregated prefill in vLLM, addressing the issue of premature KV block expiry due to static timeout values. The changes include modifications to the D scheduler, P API server, P EngineCore, P scheduler, and P worker, along with a new environment variable for configuring the P's HTTP port. The implementation involves tracking requests requiring lease refresh, periodically refreshing leases from D workers, and updating lease expiry on the P side. The changes also include a full scan in get_finished to handle lease refreshes.

[gemini-code-assist]

The code adds the remote_host and remote_http_port to _requires_lease_dict if remote_host exists. However, it does not handle the case where remote_host is empty. This could lead to issues if a request is added to _requires_lease_dict without a valid remote_host, potentially causing the lease refresh mechanism to fail. It's critical to ensure that only requests with valid remote_host values are added to _requires_lease_dict to prevent unexpected behavior.

                        if remote_host:
                            self._requires_lease_dict[request.request_id] = (
                                remote_host,
                                remote_http_port,
                            )

[gemini-code-assist]

The code checks the HTTP status code but raises a generic RuntimeError without providing specific details about the error. This makes it difficult to debug issues related to lease refresh failures. It's critical to include the URL in the error message to provide more context for debugging.

                        if resp.status != 200:
                            raise RuntimeError(f"HTTP {resp.status} at {url}")

[gemini-code-assist]

The code iterates through self._reqs_to_send to identify expired requests. However, it doesn't handle potential exceptions that might occur during the iteration or within the list comprehension. This could lead to the loop terminating prematurely and not releasing all expired KV blocks. It's critical to add error handling to ensure that all expired requests are processed and their KV blocks are released, even if some requests encounter issues.

        now = time.perf_counter()
        expired = []
        for req_id, expires in self._reqs_to_send.items():
            try:
                if now >= expires:
                    expired.append(req_id)
            except Exception as e:
                logger.warning(f"Error checking expiry for request {req_id}: {e}")

[mergify]

This pull request has merge conflicts that must be resolved before it can be
merged. Please rebase the PR, @robertgshaw2-redhat.

https://docs.github.com/en/pull-requests/collaborating-with-pull-requests/working-with-forks/syncing-a-fork

[markmc]

xref #38027

[markmc]

Relevant: https://github.com/llm-d/llm-d/blob/84ba2f7004abf3ba0e323fd8ffe3f8ce3c94656f/docs/wip-docs-new/architecture/advanced/disaggregation/operations-vllm.md

KV blocks are stranded on the P instance until the timeout VLLM_NIXL_ABORT_REQUEST_TIMEOUT, which defaults to 480s. We are currently working on a lease-extension strategy that will dramatically shorten the timeout window.

[mergify]

This pull request has merge conflicts that must be resolved before it can be
merged. Please rebase the PR, @robertgshaw2-redhat.

https://docs.github.com/en/pull-requests/collaborating-with-pull-requests/working-with-forks/syncing-a-fork