[liranschour]

Overview

Adds a fault slice (FAULTS=1) to the P2P KV-offload e2e matrix: one of two p2p sources stalls, is killed and restarted, or is removed, while a decoder keeps pulling from both. Requested in #61846 (comment) and follows the cases of the #63602 reproducer.

Stacked on #61846: only the last commit (6cfc3e21d8) is new here. Review that commit; the rest lands with #61846.

Claims

• New FAULTS=1 slice in config_sweep_accuracy_test.sh with three cases on Llama-3.2-1B, p2p mode (remote_kv_source), 3 GPUs:

  case	fault on source 0, 10 s into the run
  stall	SIGSTOP its processes for 30 s, then SIGCONT
  kill	SIGKILL, then restart on the same ports
  remove	SIGTERM, left down

• p2p_fault_check.py asserts time bounds and recovery rather than gsm8k:
  • every decoder request returns 200
  • requests naming the faulted source end within 15 s, and once the fault is detected (from 12 s on) take ≤ 2 s at the median, so the decoder prefills locally instead of waiting
  • pulls from the healthy source keep hitting (≥ 90%) within 5 s, so the fault does not freeze the decoder's p2p control plane
  • after SIGCONT or the restart, ≥ 50% of pulls from the faulted source hit again
• The stall case fails on current main and passes with [Bugfix][KV Offload][P2P] Prefill locally while a peer handshake is pending #64501.

Validation

4× H200, one host, TCP (no RDMA), branch at merge base fa3b9d1377 with a precompiled wheel. Each case run once via FAULT_CASE=<case> NUM_PREFILL_INSTANCES=2 bash tests/v1/kv_offload/tiering/p2p/run_accuracy_test.sh; the whole slice is FAULTS=1 bash tests/v1/kv_offload/tiering/p2p/config_sweep_accuracy_test.sh, about 7 minutes.

Requests naming the faulted source during the fault (n ≈ 55–60 per row), and pulls from the healthy source:

arm	case	result	faulted source p50 / max	healthy source pulled, max	faulted source pulled after recovery
main	stall	FAIL	16.85 s / 30.04 s	60/60, 0.07 s	43/51
main	kill	pass	0.01 s / 0.05 s	58/58, 0.06 s	51/51
main	remove	pass	0.01 s / 0.02 s	61/61, 0.03 s	–
+ #64501	stall	pass	0.01 s / 7.08 s	60/60, 0.06 s	41/51
+ #64501	kill	pass	0.01 s / 0.04 s	58/58, 0.06 s	51/51
+ #64501	remove	pass	0.01 s / 0.02 s	61/61, 0.04 s	–

The main stall failure:

fault faulted: n=54 pulled=0 p50=16.85s max=30.04s
E       AssertionError: a request naming the faulted source took 30.0s (bound 15.0s)

That is the gap reported in #63602 (comment): after the in-flight request ends on the 8 s lookup timeout, each new request reconnects and waits on a handshake until VLLM_P2P_HANDSHAKE_TIMEOUT_S (30 s). With #64501 the 7.08 s max is the request that was already in flight when the stop hit.

pre-commit passes on the changed files (shellcheck, ruff, test tethering).

Details

• Why p2p mode only. In pd mode the decoder waits the full 30 s load timeout while its remote_prefiller is stopped, so a time-bound check would fail until that leg has its own fix. The p2p leg is fixed by [Bugfix][KV Offload][P2P] Prefill locally while a peer handshake is pending #64501.
• No proxy. p2p_connector_proxy.py runs the prefill on the source before calling the decoder, so a stopped source would stall the proxy. The check seeds each source directly and pulls into the decoder with a remote_kv_source block, like the [Bug]: P2P KV offload tier does not fail fast on peer faults #63602 reproducer's probe. A pull counts as a hit when the decoder reports cached_tokens > 0 (the driver adds --enable-prompt-tokens-details); every prefix is unique, so the cached tokens can only have come over the p2p tier.
• Targeting one instance. Since [KVOffload][P2P][CI] Add e2e PD test matrix for the P2P KV-offload tier #61846 each instance runs in its own session, so the check signals exactly source 0 with pkill -<SIG> -s <sid>. A restarted source is started in a new session whose id is written to a file the driver's cleanup reads, so cleanup still stops only this run's processes.
• Not covered: RDMA (single-host CI runners have none, so [Bugfix][KV Offload][P2P] Exclude rc_gda lanes from the P2P NIXL agent #63498 does not apply), pd mode, and replacing a source at a new address. The kill case restarts on the same ports.
• CI. The step p2p-kv-offload-source-faults-4-gpus (L4×4, nightly, optional, 30 min) lands commented out, like every step in disaggregated.yaml since [CI] Disable jobs with unpinned runtime installs #63609. The slice needs 3 × 24 GB GPUs; L4 runtime is not measured.
• Duplicate check. No open PR adds e2e fault coverage for the P2P tier. [Bugfix][KV Offload][P2P] Fault a peer whose source WRITE stalls #64251 and [Bugfix][KV Offload][P2P] Prefill locally while a peer handshake is pending #64501 are fixes (the latter is what this slice verifies); the [Bug]: P2P KV offload tier does not fail fast on peer faults #63602 reproducer runs on Kubernetes with RDMA and is not in-tree.

AI assistance (Claude Code) was used to write this change and run the validation. I reviewed the change and the results.

---

Pull Request Checklist

• I used vLLM's /pr-checklist skill. (Mandatory for agents, optional for humans).

• AI assistance was used during the creation of this PR.

• Design Fit: Minimizes impact on core components, reuses existing functionality, and justifies added complexity.

• Testing and Validation: Validates the change and ensures any added tests are meaningful and reliable, with CI coverage or documented CI resource constraints and validation performed outside CI.

• Code Quality and Style: Keeps code and comments clear and concise, and updates relevant documentation and examples.

• Pull Request Contents: Includes a brief summary and relevant links, supports claims with evidence, explains root causes and implementation trade-offs, and follows the contributing guide.

🤖 Generated with Claude Code