Repository navigation
[Feature][KV Connector] Support QSA HiSparse in disaggregated serving - #3
Draft
lHrHenry233 wants to merge 2 commits into
Draft
lHrHenry233 wants to merge 2 commits into
lHrHenry233 wants to merge 2 commits into
Conversation
Track reads until the worker reports finished_recving. Client cancellation must not stop the producer heartbeat while those reads still own the source. Select only the ordinary READ heartbeat lifetime hunks from the source below. This is a source-attributed subset, not the complete prefix-sharing PR. It excludes the previously selected typed-disconnect adapter and unknown-poll guard. The original author and source DCO trailers are retained as provenance; they do not claim a new review or approval of this port by those contributors. Source-PR: vllm-project#59343 Source-Commit: e6e362a5b33244b6cfd19a1e8b38a188c61557f2 Co-authored-by: Codex <noreply@openai.com> Co-authored-by: lHrHenry233 <2381623149@qq.com> Signed-off-by: Nolen Liang <nliang@nvidia.com> Signed-off-by: bcsdhjew <fangl001@zuaa.zju.edu.cn> Signed-off-by: lHrHenry233 <2381623149@qq.com>
Map QSA main/indexer, GDN and PLE cache owners to the NIXL pull registration and payload schema. Preserve cache-group identity, rank-local allocation and active-state geometry while deduplicating physical registration. Retire split reads and notify-only imports through their aggregate completion. Build on the separately attributed single-engine/fusion and ordinary READ heartbeat changes. Keep standalone planning, model execution and shared capture fixes out of this P/D diff. Document the homogeneous producer-memory and supported topology boundaries and cover the registration/import behavior in the existing connector suites. AI assistance was used. Human line review and source-bound validation remain required before upstream submission. This change includes no experiment assets, internal model configuration or result data. Co-authored-by: OpenAI Codex <noreply@openai.com> Signed-off-by: lHrHenry233 <2381623149@qq.com>
4 of 6 tasks
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
2 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Overview
Add experimental QSA NIXL pull transfers with HiSparse enabled on decode. This draft follows #59209 and uses the explicit single-engine/fused-prepare base; its review diff includes the separate heartbeat prerequisite and QSA P/D commits.
Claims
Validation
Current-main CPU checks: 25 passed, 3 CUDA-registration skips; affected hooks and independent Standards/Spec reviews passed. CUDA selections passed 18 fused-prepare and 48 reference/cache cases, covering main/indexer BF16/FP8, strided/reclaimed/current rows, FULL replay and MTP padding. The 18-case mirror subset was rerun with default B. CPU doubles do not establish native DMA.
Current-main ordinary-Worker P/D completed a 1152-token source handoff and 16 forced output tokens. Each D rank executed target/MTP-prefill/MTP-decode FULL calls 6/6/12 with matching native replays; shared target selection and actual async scheduling were observed. Both roles drained and P accepted both rank completions. All four worker exit codes were zero, but D required the default SIGTERM fallback, so natural exit was not established. This finite real-weight functional check is not MTP accuracy or performance evidence.
GSM8K: first 200 test questions, five training shots, nonthinking chat, greedy/seed42, output limit 8192/context 16384, MTP off. Zero truncations/errors; correct off/C/A/B: 198/196/199/199. No equivalence/improvement claim.
Preliminary performance
C=unfused; A=joint prepare/separate writer; B=joint prepare/resident writes, ordinary Worker default. C4: three hosts, one paired cohort each, rotated arm order, 4 warmup+8 measured requests/arm; all 144 completed. C64: one same-host off→B pair, 64 warmup+128 measured/arm; all 384 completed. Cold55: one cold off→B pair, no warmup, all 55 P exports before D submission; both completed 55 full outputs. Request errors: zero.
C4 paired B/C throughput: −0.004%±2.167 pp sample SD; B/off: −23.061%±0.999 pp. No clear fusion benefit. C64 B/off: −8.890%, without D capacity wait/preemption. Cold55 B avoided six preemptions but had 55 deferred requests and −35.523% D-only throughput; no higher maximum concurrency or performance gain is established.
AIPerf ITL=request-average interval; TPOT independently recomputes it. Token-level tails: unmeasured. *Cold55 is finite D-only throughput; P+D rates were 1654.732→1359.090 tok/s. †Its post-first interval includes client completion/DONE, not native AIPerf ITL. Single-pair results are preliminary.
Performance configuration and paired resource/counter audit
Qwen3.8-Flash; four GPUs, CUDA capability 10.0; TP2/role. NVFP4 weights; BF16 main/indexer/conv, FP32 SSM; indexer top-k 2048/compression 4. P HiSparse off/eager/synchronous; D FULL_DECODE_ONLY/actual async=true, captures[1,4,8,16,32,64]. APC/chunked prefill/Mamba align; maxseq 64/batch 2048/maxlen 40960; default hot/mirror, logical host pool 128GiB, NIXL lease 600s. Fixed Moby-Dick slices, exactly 32000→8000 tokens, temperature0/seed42/ignore_eos.
Pairs matched requests, lengths, concurrency, parsed execution/kernel settings, frozen model/inputs and actual 34,310,750,208 B/rank GPU KV/state storage union. Evaluation source:
dde9aca41cd3ccceb1ffa902933b6678a09c83ceplus patch SHA256107e098ad6c29461fe7a67cc4a055f80ce5b2819acf4fae70eede0fee98a2b4d. Final-main compatibility checks do not remeasure quality/performance.Separate sampled peaks, approximately 1Hz; KV=P/D percent, GPU=P0/P1/D0/D1 GiB, D=running/waiting[capacity/deferred]. P running/waiting: C4=1/3, C64=1/62 (waiting=capacity); cold D-only=0/0. Gauge denominators/layouts differ, so their ratio is not HBM savings.
Counters are same-label pre/post scrape differences, engine-aggregate emitted outer windows; cold uses D-only boundary deltas. Every on point has positive misses/H2D: host→hot observed. Off/P series are missing (
null), not zero. H2D=miss×1024B software accounting, not hardware bus traffic or NIXL bytes. D2H and asynchronous emission-tail completeness are unknown. Separate NIXL receive deltas: C4 7.269GiB/point; C64 off/B 116.309/70.371GiB; cold 49.976GiB/arm. Transfer/notification/expiry error deltas were zero. Sampled peaks may miss transients; client/API cleanup does not prove natural worker exits.Details
Flow: P exports prefill metadata → D validates geometry → rank-local READs → generation/completion. Use
NixlConnectoron P andMultiConnector(NixlConnector, HiSparseConnector)on D. Require equal TP, PP/PCP/DCP=1, compatible layout/block/schema and matching draft configurations. Mixed-memory HiSparse-enabled QSA producers remain unsupported. Ordinary NIXL sleep support does not validate QSA/HiSparse sleep.Cancellation must keep the producer heartbeat alive until worker receive completion; retaining D pages alone does not protect P's source lease. Preserve original author/DCO without claiming new approval. Submission order remains single-engine support → coordinated heartbeat fix → QSA P/D; open landing/polling PRs are not added dependencies.
Low D KV usage does not imply zero refill: P/D import keeps older history host-valid and restores the writable resident tail. Diagnostic traces show 32 contexts/rank and joint/writer calls C=0/384, A=384/384, B=384/0; profiler timing is excluded. Resolver/gather remains. Refill and fusion are observed, but causal shares of transfer, mirroring, scheduling and kernels are unmeasured. Historical unfused regressions are not final B results.
This extends vllm-project#59209; #59594, vllm-project#59343 and GPU landing do not supply this QSA hybrid mapping. AI assistance: OpenAI Codex. Human line review/test confirmation remains pending before upstream submission. No remote CI/upstream publication is claimed; private assets are excluded.
Pull Request Checklist
/pr-checklistskill.