Skip to content

[Nixl][PD] Lease renewal TTL KV blocks on P #38027

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Closed
NickLucche wants to merge 11 commits into from
+550 −26
Closed

[Nixl][PD] Lease renewal TTL KV blocks on P #38027

Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
11 commits
Select commit Hold shift + click to select a range
7b131e4
init
NickLucche Mar 20, 2026
5e76e7b
clean up and handle p tp > d tp
NickLucche Mar 24, 2026
4575240
update docs
NickLucche Mar 24, 2026
fb06246
tests update
NickLucche Mar 24, 2026
0d699ae
cruft
NickLucche Mar 24, 2026
487a4f3
update tests and default values
NickLucche Mar 24, 2026
fbb85bf
docs update
NickLucche Mar 24, 2026
c2861cf
minor
NickLucche Mar 25, 2026
3d7e34b
Merge branch 'main' into lease-ttl-kv-blocks
NickLucche Mar 30, 2026
00285f8
Merge branch 'main' into lease-ttl-kv-blocks
NickLucche Mar 31, 2026
14899ed
Merge branch 'main' into lease-ttl-kv-blocks
NickLucche Apr 1, 2026
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
15 changes: 12 additions & 3 deletions docs/features/nixl_connector_usage.md
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -126,9 +126,18 @@ python tests/v1/kv_connector/nixl_integration/toy_proxy_server.py \
- Set when prefiller and decoder are on different machines
- Connection info is passed via KVTransferParams from prefiller to decoder for handshake

- `VLLM_NIXL_ABORT_REQUEST_TIMEOUT`: Timeout (in seconds) for automatically releasing the prefiller’s KV cache for a particular request. (Optional)
- Default: 480
- If a request is aborted and the decoder has not yet read the KV-cache blocks through the nixl channel, the prefill instance will release its KV-cache blocks after this timeout to avoid holding them indefinitely.
Copy link
Copy Markdown
Member

@markmc markmc Apr 16, 2026

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Just thinking about ... "is it ok to just drop this, what about compatibility?" ... which reminds me ...

We need P and D to upgrade to this in lockstep, so that means we need to update NIXL_CONNECTOR_VERSION

- `VLLM_NIXL_KV_LEASE_DURATION`: Initial lease duration (in seconds) for KV blocks on the prefiller. (Optional)
Copy link
Copy Markdown
Member

@markmc markmc Apr 16, 2026

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

See #25700 - let's model this as config properly. Probably kv_connector_extra_config is the place

The "lease_extension" and "heartbeat_interval" configs are particularly niche - we would probably be fine with a hard-coded lease_extension = lease_duration * 2 /3 and heartbeat_interval = lease_duration / 6. And if anyone really needs to tweak these, we can add that later 👍

- Default: 30
- After a prefill completes, the prefiller holds KV blocks for this duration. The decoder sends periodic heartbeats to extend the lease.
- If no heartbeat is received within the lease duration, blocks are released.

- `VLLM_NIXL_KV_LEASE_EXTENSION`: Lease extension (in seconds) granted per heartbeat. (Optional)
- Default: 20
- Each heartbeat from the decoder extends the lease by this amount.
Comment thread
NickLucche marked this conversation as resolved.

- `VLLM_NIXL_KV_HEARTBEAT_INTERVAL`: Interval (in seconds) at which the decoder sends heartbeats. (Optional)
- Default: 5
- Should be less than `VLLM_NIXL_KV_LEASE_EXTENSION` to ensure timely renewal.

## Multi-Instance Setup

Expand Down
Loading
Loading