Repository navigation
[Tests][KV Connector] Cover capped async-load admission and align P/D blocks - #1
Conversation
… blocks Co-authored-by: Codex <noreply@openai.com> Signed-off-by: Chenxi Ma <114001934+algebra-MCX@users.noreply.github.com>
|
Hi Robert and everyone, I've completed three matched comparisons of vllm-project#63136 on 2×H200 with real Qwen3-0.6B weights, GPU inference and NIXL P/D transfers. These GPU runs use no mocks or production monkey patches. Each comparison starts fresh prefill and decode processes. The baseline is the original PR merge-base
Both versions completed the two-request cross-block control in every run. Separately, the 11 selected CPU scheduler/allocator tests passed on the PR; base had 6 expected regression failures and 5 passing historical controls, including multiple/mixed block sizes and Mamba handling. Those CPU tests simulate the remote service and are a separate evidence category from the mock-free GPU runs. The per-run validation record is included below and in the PR description. Complete raw logs, scripts and a SHA-256 manifest are retained locally; a public archive link is pending. Two preliminary setup/harness failures are excluded from these three completed comparisons and are documented separately in the PR description. The results establish the controlled capacity-boundary behavior, not common production incidence or a throughput improvement. The full integration shell, GPU CPU-buffer mode, real hybrid/MTP models, nonzero lookahead and nonzero draft acceptance remain unvalidated here. I'd like to extend this to representative vLLM production workloads, especially to preserve the protections from vllm-project#57104/vllm-project#59880 and determine whether vllm-project#63136 addresses a problem users actually encounter. Could you share any representative, sanitized configurations and workload characteristics available, particularly:
Existing launch commands, startup cache logs and aggregate workload statistics are enough to start; private prompts or production access are not needed. I'll first screen whether the normal configuration can reach this reservation gap, then run matched GPU tests where applicable and add the configurations, results and logs here. Further synthetic repeats alone will not establish production relevance. I'm still building familiarity with this part of the scheduler and would appreciate corrections to the test design and independent maintainer validation, even if my tests pass. I'll follow up on feedback and provide further experimental evidence as needed. This follow-up remains Draft while production relevance and the remaining validation are unresolved. Validation PR: #1 AI assistance: Codex assisted with the tests, evidence collection and this write-up. Exact setup, procedure and metric observations
Each run follows the same sequence:
Across all three base126-token requests, each run retained 29 samples and every sample had: PR responses in every run reported: The NIXL logs show compatibility checks and successful transfers. Logged accepted speculative tokens were zero, so this does not establish speculative numerical correctness or effective draft acceptance. The first GPU run used Slurm 14517 (COMPLETED,4:20,exit0:0); the two additional runs used Slurm 14526 (COMPLETED,8:22,exit0:0), sequentially in the same 2-GPU allocation with fresh processes. These are three process-level comparisons, not three independent hardware samples or a performance benchmark. Executed audit submissions: sbatch PR_63136_review/tests/run-core-pd.sbatch
sbatch PR_63136_review/tests/run-core-pd-repeat.sbatchThe controlled capacity was intentionally chosen to expose the boundary. Production prevalence remains unknown. No GPU baseline timeout was repeated for the 127-token case. The full integration shell and GPU CPU-buffer mode remain pending. |
6361a43
into
krishchaudh:fix/async-load-spec-step-reserve
Overview
Supplement to vllm-project/vllm#63136, targeting its source branch. Align NIXL P/D block sizes and cover legal near-limit inputs and reservations held by another asynchronous load.
Merge recommendation: hold. This remains Draft. The core tests establish a reachable boundary liveness bug and selected safety controls; they do not establish an impact on common production workloads. That production-value requirement is currently unmet.
Claims
kv_buffer_device=cpurejects heterogeneous block sizes; CPU-only DRAM testing does not exercise that restriction.max_model_len=128truncation. Distinct prefixes prevent a local decoder cache hit from masking the second case;ignore_eosavoids early EOS affecting the completion-count assertion.max_model_lento the legalmax_model_len - 1, and add a two-request regression for capped in-flight reservations. Retain the [BugFix][KV Connector] Fix Deadlock with KVConnector + MTP under KV Pressure vllm-project/vllm#57104/[Bugfix][KV Connector] Fix speculative block reservations across KV cache groups vllm-project/vllm#59880 controls. No production scheduler or allocator code is changed by this follow-up.Validation
Three real GPU comparisons completed; production relevance remains unproven. Full integration-script and host-buffer validation also remain pending. Passing additional synthetic boundary tests would not close the production-value gap. The runtime comparison uses the exact original PR base, not today's main:
Latest results and production-configuration request: three-run validation record. Two additional mock-free H200/NIXL runs completed in Slurm job14526 (8:22, exit0:0), bringing the total to three fresh-process comparisons. In all three, base timed out on prompt126/output2, PR completed both prompt126/output2 and prompt127/output1, and both versions completed the two-request cross-block control. All29 base metric samples in each run showed waiting1/running0/KV usage0. Full raw logs are retained locally; a public archive link is pending.
fdc5fde9d786832ab638bc0c2eb2328b637ecd0a.0b767ca97f69aa1fa01e835a849ef40022ae802e.6361a43315114c7ecc19fbfc6a369e811028e183; the same unit-test file was used on both sides.The initial H200 run was Slurm job 14517, completed in 4:20 with exit
0:0; the two additional runs were job 14526, completed in 8:22 with exit0:0. Both decoders used BF16, TRITON_ATTN, eager mode, MRV2,VLLM_GPU_SYNC_CHECK=error, block size 32 and 5 total blocks including the null block. Prefill had sufficient independent capacity. Model:Qwen/Qwen3-0.6B, revisionc1899de289a04d12100db370d81485cdf75e47ca. Source trees were unchanged; both reused identical native runtime artifacts (torch2.13.0+cu129, NIXL1.5.0). The distribution version printed by that shared environment is not the tested source SHA.This establishes admission/completion, not production frequency, throughput, generated-output equivalence or nonzero draft acceptance. Logged accepted speculative tokens were zero. Real hybrid-cache models, MTP/nonzero lookahead, GPU + CPU KV buffers and all original shell phases remain untested. The independent H200 harness checks the scenarios; it does not constitute running
run_edge_case_test.shend to end.Commands and selected output
Executed in each base/PR test checkout with its environment; the supplementary unit-test file was identical:
HF_HUB_OFFLINE=1 TRANSFORMERS_OFFLINE=1 VLLM_USE_V2_MODEL_RUNNER=1 \ OMP_NUM_THREADS=4 PYTHONPATH=/tmp/pr57184-cpu-deps-20261022:. \ timeout --signal=TERM --kill-after=10s 150s .venv/bin/python -m pytest --noconftest -q \ tests/v1/kv_connector/unit/test_remote_prefill_lifecycle.py \ -k 'async_load_admits_request_whose_spec_step_fits_its_blocks or async_load_reserves_blocks_for_promotion_margin or async_load_does_not_double_reserve_mamba_spec_blocks or async_load_near_max_model_len'Base failures were the 4 new own-margin cases and 2 new in-flight-cap cases, with no collection/runtime errors in this final comparison.
GPU submission from the local validation workspace:
The audit harness starts exact-base prefill, then exact-base and original-PR decoders sequentially. It uses real
/v1/completions, returned NIXL metadata,/pause/resumefor the concurrency barrier and/metricsfor the stalled request. Selected base output:PR output:
Raw logs and complete commands are retained locally under
PR_63136_review/tests/, includinggpu-14517/results.jsonand both decode logs. Two preliminary attempts failed on missing runtime Python artifacts and a metrics barrier that cannot update while scheduling is paused; they are excluded from product results. ShutdownEngineDeadErrormessages occur after the harness intentionally sends SIGTERM, on both versions.Static validation passed: Ruff check/format, shell syntax,
git diff --check, and installed commit hooks including ShellCheck, mypy, typos and test/Buildkite tethering.Still to run in an isolated two-GPU environment:
Details
The concrete failure is a legal request for which both the prompt and the capped first reserved step fit. The old check charges an unnecessary block and never starts its asynchronous load in an otherwise idle instance. The author found it while testing the scheduler; neither that report nor this validation establishes a production incident rate.
Production evidence and a bounded validation gate
The public #63136 description explicitly identifies scheduler testing rather than a production incident. #57104 reports a GLM5.2 production deadlock, but its cause is insufficient promotion headroom; that incident cannot establish prevalence of this opposite over-reservation case. Related queue-stranding reports such as #53868, or #53130, which explicitly has no KV connector, are not confirmed incidents for this PR's cause.
Common use of P/D, speculation or high KV utilization alone does not establish common occurrence of their exact failing intersection. No claim that this is common, or that it has a measured production-performance benefit, is made here.
Further validation should proceed only through these gates:
usable_blocks > ceil(max_model_len / block_size)rules out the demonstrated idle minimum-pool stall; it does not rule out concurrent/multi-group cases.Stop condition: without an affected real deployment or representative trace, keep this as an unproven production-value boundary fix. If representative observation/replay finds no attributable admission failures or measurable user-visible loss, stop expanding synthetic/GPU coverage and do not recommend merge on a common-production-impact rationale. No further GPU validation has been run for that rationale.
Admission failure observed on H200
flowchart TD A["126-token remote hit; 4 usable blocks of 32; maxlen128"] --> B{"Admission margin"} B -->|"Old: 1 extra block"| C["Needs 4 + 1; allocation rejected"] C --> D["Idle instance: waiting1 / running0 / KV usage0"] D --> C B -->|"PR: ceil(128/32) - ceil(126/32) = 0"| E["Allocate 4 prompt blocks"] E --> F["Real NIXL transfer; request runs"] F --> G["Completes 2 output tokens"]How narrow is this boundary?
Robert's minimum-pool concern is valid under a useful restricted model: idle instance, one dense cache group, no other reservations and old margin=1. Startup requires usable blocks
U >= ceil(max_model_len / b). Old admission rejects whenceil(prompt_len / b) + 1 > U. Together these implyU = ceil(prompt_len / b) = ceil(max_model_len / b). This is equality of rounded block counts, not necessarily exact token capacity.Correction to the earlier draft:
prompt40 / maxlen2048 / 3 usable blocks of 16exercises scheduler arithmetic but fails the real startup capacity check. It is not a deployable counterexample. Realget_kv_cache_configsaccepted maxlen 127 or 128 with four usable 32-token blocks and rejected maxlen256 with the same pool.The unrestricted “IFF” is broader than the evidence supports. A real CPU planner+Scheduler probe with maxlen2048, prompt2047, block size 16 and 129 usable blocks accepted startup, yet base admission failed for spec31 (old margin=2); PR admitted and finished after simulated remote completion. For spec1 and the same 129-block pool, both versions finished. This is a scheduler result, not spec31 GPU model validation.
Minimum-pool equality need not require a deliberately tiny override. By code inspection,
--max-model-len=-1uses_auto_fit_max_model_lento maximize context within the cache budget. For a single FullAttention group limited by available memory, that can naturally makeceil(max_model_len / b)equal the usable block count. This is a static inference, not a separately executed auto-fit GPU reproducer or evidence of production frequency.Two distinct maxlen cases
flowchart LR P["Decode: 4 usable blocks x32; maxlen128"] --> A["126-token prompt"] A --> B["126 + 2 = 128; no truncation"] P --> C["127-token prompt; separate prefix"] C --> D["127 + 2 = 129; truncate to 128"] B --> E["Observed on PR: 2 outputs"] D --> F["Observed on PR: 1 output"]A prompt equal to
max_model_lenis rejected by the normal generation input processor. The original near-limit unit input is therefore changed tomax_model_len - 1.In-flight reservation is a separate path
flowchart TD A["130 total blocks: 1 null + 129 usable"] --> B["A: 2047 tokens holds 128 blocks while loading"] B --> C{"Remaining reservation for A"} C -->|"Without cap: 2049 or 2051 slots"| D["Reserve one more block; reject B"] C -->|"Cap at 2048"| E["Reserve zero more blocks"] E --> F["B: 8 tokens fits in remaining block"] F --> G["CPU: simulate transfer completion; both run, finish, free blocks"]The single-request test cannot observe
_request_remaining_blocks()affecting another load. The new two-request case covers that path for spec1/3. It does not replace live concurrent NIXL validation of that exact near-limit state.The original edge-case phase also changes from the GPU default block size 16 to32, so the full script still needs validation. This PR complements vllm-project#63136; vllm-project#63288 addresses non-async full-sequence admission and vllm-project#62979 addresses HiSparse host-page reservations. It does not open a competing upstream fix.
AI assistance: Codex assisted with inspection, code, checks, the independent runtime validation and this description. Human review remains required.
Pull Request Checklist
/pr-checklistskill. (Mandatory for agents, optional for humans).