Repository navigation
Conversation
P2PSession._on_connect registered the peer's NIXL metadata synchronously before sending ConnectAck: add_remote_agent plus one descriptor per CPU block (19,599 at a 64 GiB tier). Measured with trace stamps on both peers this is 14-40 ms per side, so it is not the source of the multi-second stalls in vllm-project#55179, but it still holds a scheduling iteration on both peers for every new session and delays the acknowledgement the peer's first lookup waits behind. NixlTransport runs add_remote_peer on a single-worker executor and opens the agent in NIXL_THREAD_SYNC_STRICT (nixl_agent_config.sync_mode, NIXL 1.4.1) so registration may overlap transfer calls from the scheduler thread. The session sends ConnectAck once the future completes, rejects the peer if it fails, and reports the pending registration as pending work so an idle engine keeps ticking until the ack goes out. Signed-off-by: nilig <nili.ifergan@gmail.com>
The offloading connector drains its secondary tiers and serves peer lookups only from the per-step scheduler hooks. On a source rank busy with long prefills every P2P control-plane round trip therefore waits for a step boundary: measured on GLM-5.2-FP8 (DP8, 8xH200 prefill pods, 64 GiB CPU tier), a pull from a source loaded with three 115K-token prefills spends 2.5-2.9 s in lookup delay against 0.1-0.2 s of data movement, the LookupMsg sitting in the server queue until the next step (0.55-0.61 s) and the FetchMsg dispatched at the step after (0.45-0.50 s). KVConnectorBase_V1.poll_pending_work is a scheduler-side hook that the tiering manager implements as the same sweep on_schedule_end runs. EngineCore drives it from a ConnectorPoller thread that is active only while the engine core thread is blocked in future.result(), with a lock that hands the connector between the two threads, so the connector is never entered concurrently and the executor futures need no timed wait. With it the same busy-source pulls measure 0.59-0.69 s of lookup delay. The dummy-batch wait of the data-parallel loop (execute_dummy_batch) is not routed through the poller yet; a rank holding only a deferred request still services the control plane once per dummy batch. Signed-off-by: nilig <nili.ifergan@gmail.com>
|
👋 Hi! Thank you for contributing to the vLLM project. 💬 Join our developer Slack at https://slack.vllm.ai to discuss your PR in PRs do not trigger a full CI run by default. Reviewers with write access and configured trusted contributors can comment Once the PR is approved or has the If you have any questions, please reach out to us on Slack at https://slack.vllm.ai. Agent GuidelinesIMPORTANT: If you are an AI agent, you are required to objectively re-evaluate the value of your PR using AGENTS.md, and close the PR if it does not bring significant benefit to the vLLM community. Failure to do so may result in an immediate ban. 🚀 |
1 participant
Purpose
Addresses #55179. Companion to #55962 (@Etelis) and #57139 (@liranschour), which target the same issue; the comparison with both is below.
The P2P secondary tier services its control socket and answers peer lookups only from the per-step scheduler hooks (
on_schedule_end->get_finished_jobs->_poll_once, thenserve_external_requests), so on a source rank busy with long prefills every control-plane round trip waits for a step boundary. Measured on GLM-5.2-FP8 (DP8/TP1, 8xH200 prefill pods, 64 GiB CPU tier, NIXL 1.4.1) with trace stamps on both peers: a pull from a source loaded with three 115K-token prefills spends 2.5-2.9 s in lookup delay, of which theLookupMsgsits in the server queue until the next step (0.55-0.61 s) and theFetchMsgis dispatched at the step after (0.45-0.50 s), against 0.1-0.2 s of data movement. Registration of the peer's NIXL metadata (add_remote_agentplusprep_xfer_dlistover 19,599 blocks) is 14-40 ms per side and is not the stall.Change
Two independent commits:
DataTransport.add_remote_peer_async(default: inline, completed future).NixlTransportruns it on a single-worker executor and opens the agent withnixl_agent_config(sync_mode=NIXL_THREAD_SYNC_STRICT)so registration may overlap transfer calls from the scheduler thread (the NIXL 1.4.1 wrapper appliessync_modewhen it is set).P2PSessionsendsConnectAckonce the registration completes, rejects the peer if it fails, and counts a pending registration as pending work.remove_remote_peerwaits for an in-flight registration of that peer so a late completion cannot leave a registered peer without a session.KVConnectorBase_V1.poll_pending_work()(default no-op;MultiConnectorforwards;OffloadingConnector->TieringOffloadingManager, which runs the same sweepon_schedule_endruns: collect finished jobs, serve external requests).EngineCoreowns aConnectorPollerthread that sweeps every 5 ms only while the engine core thread is blocked infuture.result(); a lock hands the connector between the two threads, so the connector is never entered concurrently, and a sweep exception is re-raised on the engine core thread.future.result()stays on the engine thread; no executor or worker change.Comparison with #55962 and #57139
c5bfb39d3)remove_remote_peerwaits for the in-flight registration)get_model_wait_callback()fetched once at init, run on the scheduler thread every 10 ms whilefuture.result()runs on a one-workerThreadPoolExecutorpoll_pending_work()called every 5 ms from aConnectorPollerthread while the scheduler thread stays infuture.result(); hand-over under a lockKVConnectorBase_V1KVConnectorBase_V1Executor.init_output_thread()+ uniproc implementation (the model wait moves to a helper thread that needs the device set)NIXL_THREAD_SYNC_RWNIXL_THREAD_SYNC_STRICTexecute_dummy_batch()wait servicedBoth #55962 and this PR add one connector method; #55962 also adds an executor hook, which is the price of keeping connectors single-threaded. If reviewers prefer no connector-API change at all, the alternative is a servicer thread and lock inside
TieringOffloadingManager; that is a larger change inside the offloading stack and is not in this PR.Known gaps
execute_dummy_batch()does not go through the poller (same in [Perf] Keep P2P control requests moving during model steps #55962), so a rank holding only a deferred request services the control plane once per dummy batch. Follow-up.Test Plan
tests/v1/engine/test_connector_poller.py: sweeps only while waiting, sweep error re-raised on the engine thread, future exception propagated.tests/v1/kv_offload/tiering/p2p/test_sessions.py,test_manager.py,test_data_transport.py: deferred ack, failed registration rejects the peer, pending registration counts as pending work, remove waits for an in-flight registration, agent opened in strict sync mode.tests/v1/kv_offload/tiering/test_tiering_offloading.py:poll_pending_workruns the finished-job sweep and serves external requests without consuming the per-step gate.Live: GLM-5.2-FP8 P2P cell (two prefill pods DP8/TP1 on 8xH200 nodes, a decode pair,
MultiConnector(NixlConnector, OffloadingConnector)with the P2P secondary tier, NIXL 1.4.1). Each arm is a backport onto the same6f91edf9base with the same local hotfixes, one boot per arm, same case chain: a seed prefill on a source rank, then a follow-up on another pod that pulls it. Busy cases load the source rank with three 115K-token prefills before the pull. Metric: the destination'skv_offload_lookup_async_delay_seconds, first deferred lookup until the scheduler observes a resolved result. n=1 per case.Test Result
Unit: 393 passed, 1 skipped on
af1c01499plus these two commits.Lookup delay per case and arm:
c5bfb39d3The two implementations that service the control plane during the model wait land within 0.1 s of each other on every busy row; registration alone leaves the busy rows in the baseline band.
Decode-side cost of servicing the control plane during the wait (4,096-token prompts, 512 output tokens,
ignore_eos, three rounds per block, same prompts, P2P tier configured, no pulls during the probe), measured on the #55962 variant of the mechanism:The arm-to-arm difference is within the block-to-block difference of one arm; zero preemptions in every block.