[albertoperdomo2]

Overview

Let an idle P2P-only offload engine report no pending work, using the tier-poll thread introduced in #58168 to service peers. Related context: #51476.

Claims

• P2P reports actual outstanding work instead of keeping the engine busy unconditionally.
• The poll thread survives sustained failures, backs off to max(tier_poll_interval_s, 1 s), and rate-limits error reports.

Validation

• .venv/bin/python -m pytest -s -q -p no:cacheprovider tests/v1/kv_offload/tiering/: 573 passed, 12 skipped.

• Instrumented engine probes verified idle queue parking and the shutdown decision; a deterministic fault probe verified bounded logging, backoff, and recovery.

• Ruff 0.14.0 check/format checks, Markdown lint, and git diff --check passed.

• Live H100 A/B validation completed: Qwen3-0.6B, two TP1/DP1 instances, P2P-only, real NIXL transfers over TCP.

  Check	Baseline	Fixed
  Pause, mode=wait&clear_cache=false	4/4 timed out at 10 s	4/4 HTTP 200 in 22.7–24.7 ms
  Idle EngineCore main-thread context switches/s	1,014–1,076	0 in both 5 s samples
  P2P transfers and matching outputs	3/3	3/3, while source acknowledged paused
  Engine shutdown	Both forced-killed after 15 s	Both normal teardown, container exit 0

  Each arm transferred 1,008 MiB / 576 chunks with zero promotion failures. Outputs also matched across images. This validates the P2P-only path; sleep, DP, multi-tier and RDMA performance were not tested.

Details

Idle polling belongs to the poll thread. Keep the engine stepping when polling is disabled or multiple secondary tiers require scheduler-driven promotions; those configurations retain their existing pause/shutdown limitation. KVCR is unchanged. Rendered documentation describes the supported behavior.

Duplicate-work searches found related #61667, #62795, and #62929, which address transfer lifetime, finalization ordering, and cache-reset fencing rather than this idle predicate. Design feedback is welcome.

AI assistance was used for implementation, tests, and review.

---

Pull Request Checklist

• I used vLLM's /pr-checklist skill. (Mandatory for agents, optional for humans).

• AI assistance was used during the creation of this PR.

• Design Fit: Minimizes impact on core components, reuses existing functionality, and justifies added complexity.

• Testing and Validation: Validates the change and ensures any added tests are meaningful and reliable, with CI coverage or documented CI resource constraints and validation performed outside CI.

• Code Quality and Style: Keeps code and comments clear and concise, and updates relevant documentation and examples.

• Pull Request Contents: Includes a brief summary and relevant links, supports claims with evidence, explains root causes and implementation trade-offs, and follows the contributing guide.

---

^{Need help on this PR? Tag @codesmith-bot with what you need.}

---

^{Need help on this PR? Tag @codesmith-bot with what you need.}

[albertoperdomo2]

I’m going to test this live soon with two P2P peers, covering cache transfers while the source is idle, pause/resume, and graceful shutdown. I’ll post the configuration and results here before marking the PR ready for review. Design feedback is welcome meanwhile.

[mergify]

Documentation preview: https://vllm--63618.org.readthedocs.build/en/63618/

[claude]

Claude Code Review

This pull request is from a fork — automated review is disabled. A repository maintainer can comment @claude review to run a one-time review.

[mergify]

This pull request has merge conflicts that must be resolved before it can be
merged. Please rebase the PR, @albertoperdomo2.

https://docs.github.com/en/pull-requests/collaborating-with-pull-requests/working-with-forks/syncing-a-fork

[liranschour]

Thanks.
Two things before this can merge:

1. Rebase onto #63779

On main, #63779 added bool(self._deferred_releases) to TieringOffloadingManager.has_pending_work(). Please keep that check when you resolve the conflict, filtered per tier the same way as _jobs in item 2. A deferred release on a polled tier completes inside poll_tiers() → _process_finished_jobs(), so it doesn't need an engine step.

2. Required: work the poll thread can finish must not block idleness

When the tier-poll thread is running, an idle lone-P2P producer handles a whole peer request without any engine step: receive the LookupMsg, parent.lookup(), pin the HITs, send LookupRespMsg, take the FetchMsg, NIXL WRITE, complete_read(). The current check still counts two parts of that as pending work:

• the HIT pin in _jobs, which stays until the fetch completes or 30 s pass;
• _server.has_inflight_transfers, through P2PSession.has_pending_work.

So a producer that peers keep fetching from can't finish pause or graceful shutdown. That's the bug this PR fixes, moved from an idle source to a busy one.

Requested change:

1. Count only work that needs an engine step. In TieringOffloadingManager.has_pending_work(), when a poll thread is alive, leave out the polled tiers: their has_pending_work(), and any _jobs / _deferred_releases entries whose tier_idx is a polled tier. Keep the rest: pending cascades, jobs on non-polled tiers, the multi-tier keep-alive, and the tier_poll_interval_s <= 0 keep-alive.
2. Fix the docs. Both has_pending_work() docstrings should say they report work that needs a step. Add a sentence to kv_offloading_usage.md that a paused engine keeps serving peers from its CPU tier: pause stops scheduling and quiesces the device, not the P2P server.

Pause with clear_cache and sleep don't depend on has_work(), because reset_cache() takes the manager lock and drains every tier. A parked lookup pin can still make that drain assert. That's tracked in #63948 and fixed by #63988, so it's out of scope here.

Tests (unit level, using the existing _attach_fake_transports / paired-manager helpers):

• Live thread, peer HIT pinned and waiting for its FetchMsg: has_pending_work() is False. The fetch then completes with no on_schedule_end(), and _jobs empties.
• Same setup with tier_poll_interval_s=0: has_pending_work() is True.

[liranschour]

Approve. Thanks

[orozery]

Can we also make comments shorter?

[orozery]

/ci run

[github-actions]

✅ @albertoperdomo2, CI is now available for this PR.

• /ci run starts upstream CI; /amd-ci run starts AMD CI only.
• Run commands test your branch with every commit currently on its upstream target branch. If it is behind, the bot merges the target branch into it first (pull before pushing again). Append --allow-stale to a run command to test an outdated branch as is, at your own risk.
• /ci retry retries failed jobs in the CI build for the current PR head. If the current head has no CI build, it starts a new CI build for the current head containing only jobs that failed in the latest earlier CI build for this PR.
• /amd-ci retry retries failed jobs in AMD CI for the current PR head. Use /amd-ci run when the current head has no AMD CI build.
• /ci cancel cancels scheduled or running CI builds for this PR branch; /amd-ci cancel does the same for AMD CI only.

[github-actions]

✅ Triggered Buildkite CI #98308 for commit 4dbd03161785.

ℹ️ This PR is 3 commits behind upstream main. Merged upstream main into this branch as 4dbd03161785 before starting CI. Pull the branch before pushing more changes.