Repository navigation
Conversation
First of a series adding connector-agnostic checksums of transferred KV
for P/D disaggregation. Some transfer failures are not reported by the
transport: corrupted bytes, a read after the producer has already
reused its blocks, or blocks placed at the wrong position. With
kv_transfer_config.enable_kv_checksum on both sides, the producer
checksums a request's KV in the step that finishes its prefill, the
checksums travel with the request in kv_transfer_params, and the
consumer verifies every load against them before the request runs.
kv_checksum_fail_closed turns a failed check into a KV load failure,
handled by kv_load_failure_policy.
No connector declares a carrier in this commit, so enabling KV
checksums alone has no effect yet; the next commit makes NIXL pull the
first connector that carries them. Later in this series: KV heads
replicated across TP, different producer and consumer block sizes,
NIXL push and its connector-owned carrier, Mooncake P2P, Mamba state,
and a Triton kernel.
Design
------
- Connector-agnostic. Workers compute the checksums from the KV cache
itself, the scheduler sums and compares them, and a connector only
declares how it carries the bytes (kv_checksum_carrier). Computing
them inside each connector was rejected: every connector would
reimplement it, and the values would depend on the connector's view
of TP and cache layout.
- One value per (KV cache group, block position), independent of TP,
PP and memory layout. The scheduler, the only place that sees every
worker's record, sums the TP and PP parts on both sides, so the
consumer needs no knowledge of the producer's layout.
- A failed check is a KV load failure. It takes the path
connector-reported failures take (_handle_failed_recving), so
kv_load_failure_policy decides what happens next and no new failure
handling is added.
- No host sync on the engine step: results reach host memory on a side
copy stream and an event.
- The producer checksums the whole prompt once, in the step that
finishes it, with or without chunked prefill. Per-chunk partial sums
were rejected: they would need per-rank state in the scheduler,
cleanup on preemption, and handling of the block spanning two chunks.
The cost is that all of the prompt's checksum work lands in its last
prefill step.
What a checksum is
------------------
A checksum covers one block position of one KV cache group, over the
unsharded model: all layers of the group and all KV heads.
1. A slice is the KV of one layer and one KV head in the block: the K
and V of the block's tokens for that head. Laid out token by token
and split into little-endian 8-byte pieces, its checksum is
s = sum_j x_j * (2 * j + 1) mod 2**64
where x_j is piece p of the token at position t in the request,
j = t * P + p with P pieces per token.
2. The block's checksum adds up the slices of every layer and head,
each times its own odd factor:
c = sum_{layer, head} m * s mod 2**64
m = 2 * (layer * H + head) + 1
H = 2**16 (MAX_KV_HEADS_PER_LAYER), a fixed bound on any layer's KV
heads. Layers are numbered by the layer number in their names and
heads globally across TP, so m does not depend on the model's head
counts or its TP or PP split.
Why this definition:
- Factors and weights are odd, hence invertible mod 2**64: a change to
any single piece always changes c.
- Distinct weights catch tokens in the wrong place; distinct factors
catch swapped heads or layers. Random corruption of several pieces
goes undetected with probability about 2**-64. It is not a
cryptographic hash and does not defend against deliberate tampering.
- Numbering pieces from the request's first token makes the checksums
of consecutive blocks add up to that of the tokens they hold, so
different block sizes can later be compared without changing the
definition.
- TP splits KV heads and PP splits layers, so every slice lies whole on
one worker, and the workers' parts sum to c whatever the memory
layout (head-major, token-major, kernel-block splits).
The last block of a request only covers its first num_tokens slots,
which excludes slots that later decode or draft tokens may write.
Life of a request
-----------------
Producer:
1. In the step whose scheduled tokens cross num_prompt_tokens,
Scheduler._update_after_schedule calls KVChecksumScheduler.add_send
for requests the connector carries checksums for. build_checksum_meta
puts their blocks, per transfer group, into
SchedulerOutput.kv_checksum_scheduled.reqs_to_send.
2. KVChecksumWorker.post_forward checksums them after the forward has
written their KV. Rows for all groups are built with numpy and copied
to the device in one non-blocking H2D copy from pinned memory. The
worker's own copy stream then copies the sums to pinned host memory
and records a blocking event; KVChecksumOutput.finalize, called when
the output is pickled for the scheduler, only waits for that event.
The results ride KVConnectorOutput.kv_checksums, which
KVOutputAggregator merges across TP and PP workers.
3. Scheduler._free_request calls request_finished after the connector's
request_finished. For a request that finished normally it sums the
workers' parts and adds them to the connector's kv_transfer_params as
"kv_checksums". The proxy forwards them to the consumer with the
request.
Consumer:
1. When a request starts an async load, Scheduler.schedule calls
add_recv. It decodes the expected checksums from the request's
kv_transfer_params and puts the blocks the load writes, tokens
[num_local_computed_tokens, num_computed_tokens), into reqs_to_recv
in the same step, so workers know them before the load starts.
2. Workers checksum a load in the step whose transfer results report it
finished, before any later forward can write its blocks; a request
whose load failed or that ended is dropped. update_from_output keeps
each worker's record until the aggregated finished_recving reports
the request.
3. It then compares, after the connector's own failed_recving and
invalid_block_ids are handled (loads already known to be bad are not
compared), and before the request can be scheduled again.
On the v1 runner with a drafter that writes KV after the connector
step (defer_finalize), post_forward leaves the sends to finalize_sends,
which finalize_kv_connector calls after the drafter, on the last PP
rank only.
Which blocks
------------
- Producer: per transfer group, the blocks holding tokens
[0, num_computed_tokens). Consumer: the blocks the load writes.
- Both: leading null blocks (out of a sliding window) are left out, a
group with a null block after a non-null one is left out entirely,
and sliding-window groups keep their last cdiv(window, block_size) + 1
blocks, the clip NIXL and Mooncake apply to what they transfer. A
connector that carries checksums must load exactly these blocks.
- Only positions both sides checksummed are compared, and the last
block only if both sides hold the same number of tokens in it.
Verdicts
--------
Each group is compared on its own:
- Mismatch: some compared position differs. Logged as an error with
the groups and block positions, even if other groups are unverified.
- Unverified: the group cannot be compared: it is checksummed by one
side only, its fingerprint differs (other layers or dtype), its block
sizes differ, a worker checksummed other blocks than scheduled, or no
position is in common.
A load is also unverified as a whole when not every worker (TP * PP)
reported it, or the producer sent no (valid) checksums. Unverified
loads log a warning once per fixed reason, and the details per request
at debug level.
Each worker record carries a fingerprint per group, the sum of a hash
of each of its layers in it (name, spec type, dtype). Summed over PP
stages it identifies the group's layers whatever the PP layout; the
scheduler's own KVCacheConfig only lists the first worker's layers.
Groups a worker cannot checksum are logged at startup and reported in
the record's skipped_groups: layers without per-token KV blocks (Mamba
state), compressed caches (tokens_per_state != 1), KV heads packed into
head slots, host-resident groups, caches that are not whole 8-byte
pieces or whose kernel blocks do not tile the block size, KV heads
replicated across TP, two caches with one layer number, and layer names
without one. A group skipped on both sides is left out of the verdict,
so hybrid models verify their attention groups under fail-closed; a
group skipped on one side only is unverified.
Failure semantics
-----------------
- Fail-open (default): mismatched and unverified loads are only logged;
the loaded blocks are then used and prefix-cached as usual. This lets
checksums be rolled out, and observed, without failing requests.
- Fail-closed: mismatched and unverified loads fail through
_handle_failed_recving: kv_load_failure_policy "fail" ends the request
with an error, "recompute" sets num_computed_tokens to 0 and
recomputes the whole prompt locally on the consumer.
- The KV is never transferred again: connectors clear do_remote_prefill
on the first load. Recovery is per request, not per block.
Recomputing only from the first bad block is future work, building on
group-aware load-failure recovery (RFC vllm-project#47812, PR vllm-project#48216).
Carrier
-------
KVConnectorBase_V1.kv_checksum_carrier(request, receiving) returns how
the connector carries a request's checksums, or None (the default) if
it does not, e.g. offloading connectors. This commit defines
KVChecksumCarrier.KV_TRANSFER_PARAMS: generic code adds and reads the
"kv_checksums" key, and the connector needs no other change.
- MultiConnector: for a load, the sub-connector chosen to load it; for
a producer request, the first sub-connector with a carrier.
- A connector without a carrier can verify no load: enabling KV
checksums with it fails at startup when fail-closed and disables them
with a warning when fail-open.
Wire format: base64 of msgpack (version, tp_size, num_tokens,
[(group_id, fingerprint, block_size, start, little-endian uint64
values)]). A producer whose workers did not all report a request (or
reported other blocks) sends no checksums for it, with a warning. For a
128k-token prompt with one group and 16-token blocks this is about
87 KB of base64. Proxies must forward unknown kv_transfer_params keys.
kv_checksum_enabled disables KV checksums, with a warning, for context
parallelism (DCP/PCP > 1), and for the Ray executor with PP > 1 unless
VLLM_USE_RAY_V2_EXECUTOR_BACKEND=1: its non-last stages' connector
outputs never reach the scheduler.
Rollout and limits
------------------
- Enable on producers before consumers; otherwise consumers see loads
without checksums, unverified (failed when fail-closed).
- A producer request preempted in its final prefill step and
re-prefilled before that step's output arrives can mix the two steps'
values; the consumer then reports a mismatch or unverified, never a
false match.
- In this commit, replicated KV heads, several caches per layer and
Mamba state are skipped, different block sizes are unverified, and
the PyTorch path reads the gathered KV several times per layer; later
commits in the series address each.
Tests
-----
- test_kv_checksum.py checks every worker result against an independent
NumPy implementation of the definition: head-major and token-major
layouts and kernel-block splits; TP2 shards and PP2 stages summing to
the whole model's value; swapped tokens and layers changing it; slots
past num_tokens ignored; 16-token blocks adding up to 32-token blocks;
KV-sharing counted once; chunked gathers; every skipped group kind
next to a checksummed group; the consumer lifecycle (finish, failure,
abort); deferred sends covering KV written before finalize_sends;
fingerprints independent of the PP split; serialization and
aggregation across workers.
- test_kv_checksum_scheduler.py runs a producer's and a consumer's real
schedulers with real checksum workers, copying the producer's blocks
into the consumer's, with a minimal test connector that only declares
the kv_transfer_params carrier:
- producers of TP2, PP2 and TP2xPP2 send what a single worker sends;
requests the connector does not carry get no checksums; nothing is
sent unless every worker reported;
- correct loads verify for (TP, PP) pairs (1, 1)/(1, 1), (2, 1)/(1, 1),
(1, 1)/(2, 1), (1, 2)/(1, 1) and (1, 1)/(2, 2);
- with a local prefix hit only the loaded blocks are compared;
- a corrupted byte (logged as a mismatch), missing and malformed
checksums: only logged when fail-open, an error with policy "fail",
a local recompute with "recompute";
- different block sizes are unverified;
- a group skipped on both sides is ignored, on one side only is
unverified;
- a connector without a carrier is refused at startup when
fail-closed and disables checksums when fail-open.
Disabling the comparison,
ignoring the received checksums, summing fingerprints per worker
instead of per PP stage, numbering heads per rank, or checksumming
loads from the first block each make tests fail.
- test_scheduler.py and test_async_scheduler.py set the new scheduler
attribute on their hand-built schedulers.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Signed-off-by: David Cheung <d7cheung@gmail.com>
Make NIXL pull the first connector that carries KV checksums, in the kv_transfer_params the previous commit added them to: a producer's request_finished already returns kv_transfer_params that the proxy forwards to the consumer, so NixlPullConnector only declares the carrier. With kv_transfer_config.enable_kv_checksum on both sides, NIXL pull loads are now verified end to end. NixlPullConnector.kv_checksum_carrier returns KVChecksumCarrier.KV_TRANSFER_PARAMS for producer requests (do_remote_decode) and for regular loads. A bidirectional load (a load with do_remote_decode set, a prefiller reloading a decoder's KV from an earlier turn) reads blocks whose source sent no checksums, so it returns None: such loads are neither verified nor counted as unverified. Tests ----- test_kv_checksum_scheduler.py runs the end-to-end scheduler tests that do not depend on the connector for NIXL pull as well as for the test connector: correct loads verify for every (TP, PP) pair, and a corrupted, missing or malformed load follows kv_checksum_fail_closed and kv_load_failure_policy. Making NixlPullConnector declare no carrier makes the NIXL pull cases fail. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Signed-off-by: David Cheung <d7cheung@gmail.com>
Checksum groups whose KV heads several TP ranks hold, which the previous commit skipped: MLA-type caches, GQA with more TP ranks than KV heads, and layers with several caches, such as an attention's KV and its indexer's. Design ------ When several TP ranks hold the same KV heads, those copies must not be summed twice, and every copy should still be checked. A partial checksum is therefore keyed by the (replicas, replica_index) of its layers (ReplicaKey): - A producer only checksums replica 0: any replica describes the same KV. Its other replicas report the request with no checksums, so the scheduler still counts them. - A consumer checksums every replica. KVChecksumGroup.values gets one column per replica key, and _sum_replicas builds one check per replica: check k sums copy k % R of each layer replicated on R ranks, so every check covers each layer once and every copy is in some check. Each check must equal the producer's value, so a corrupted copy is a mismatch even if another copy is intact. - Checking every copy was preferred over checking one replica because each rank's copy is written by its own transfer. How many ranks replicate a layer (_tp_replicas): - Specs that declare max_tp_shards give it through layer_tp_replicas. - MLA-type caches (MLAAttentionSpec, SlidingWindowMLASpec) that do not declare it are held whole by every rank, as NIXL transfers them. - Other attention shards KV heads across TP and replicates one head on consecutive ranks once TP exceeds them. More than one head per rank means no replicas; one head per rank is resolved with the model's KV head count when every layer shares it (no per-layer override, no separate global head count, and a draft model with the same count). Otherwise the group is still skipped. Several caches per layer ------------------------ Distinct caches with one layer number of the same layer module (e.g. model.layers.0.self_attn.attn and model.layers.0.self_attn.indexer .k_cache) now take consecutive heads of that layer, in layer name order, so both are covered. Caches with one layer number from different modules (e.g. a main layer and an MTP layer, which PP may place on different stages) are still skipped. MiniMax-M3 ---------- MiniMaxM3IndexerCache declares max_tp_shards=1: its index_k is one head replicated to every TP rank. Its spec is an MLAAttentionSpec, so it is treated as replicated either way; the declaration states it for any other code that reads max_tp_shards. Tests ----- - test_kv_checksum.py: GQA replicas (TP4, 2 heads) and MLA replicas, declared or by spec type, are summed once by the producer and reported per replica by the consumer; an attention cache and an indexer cache of one layer take consecutive heads; the MiniMax-M3 layout (2 attention heads sharded at TP2, replicated at TP4, plus a replicated indexer) sums to the definition on the producer, and every consumer replica check equals it; layers of different modules sharing a layer number are skipped. - test_kv_checksum_scheduler.py: a TP1 producer with a TP4 consumer and a TP4 producer with a TP1 consumer (2 KV heads, so replicated) verify end to end, for NIXL pull and the kv_transfer_params test connector. Summing every replica on the producer, checking only the first replica on the consumer, not offsetting the second cache of a layer, or not treating MLA-type caches as replicated each make tests fail. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Signed-off-by: David Cheung <d7cheung@gmail.com>
NIXL transfers KV between producers and consumers with different block sizes (block_size_ratio), but so far the consumer reported such loads unverified, which fails them all under kv_checksum_fail_closed. The checksum definition already allows comparing them: pieces are numbered from the request's first token, so the checksums of consecutive blocks add up to that of the tokens they hold, whatever the block size. This commit only changes the comparison; checksums and their wire format are unchanged (the payload already carries each group's block size). Comparison ---------- The consumer's scheduler does the regrouping. With different block sizes, it sums the smaller blocks' checksums into chunks of the larger block size on each side (_chunk_sums) and compares chunks; a chunk is compared only where both sides cover it whole and hold the same number of tokens in it, blocks past the last token counting as zero. With equal block sizes a chunk is a block, as before. Mismatches are reported as the consumer's block positions, so with different block sizes a mismatch is localized to one larger block. Unverified: - block sizes that are not multiples of each other; - a consumer with smaller blocks whose first loaded block starts inside a producer block (a local prefix hit or window clip not aligned to the producer's blocks). That block's chunk also holds tokens the consumer did not load, so it cannot be compared. Skipping the chunk instead would leave the first loaded block unchecked while reporting the load verified. Adding the consumer's locally cached tokens to the chunk would report false mismatches: KV a consumer computed while decoding is in general not bit-identical to the producer's prefill of the same tokens. Known limitations ----------------- - The unaligned case above is unverified, so it fails under fail-closed and is only logged under fail-open. NIXL pull requires the consumer's block size to be a multiple of the producer's, so it arises only with larger producer blocks, as NIXL push and Mooncake P2P (later in this series) allow. The open upstream vllm-project#59747 proposes rejecting different block sizes in NIXL push at handshake. Checksumming producer blocks at a finer granularity would make the case comparable; it is not done here. Tests ----- - test_kv_checksum_scheduler.py: producer and consumer block sizes 16/32 and 32/16, for NIXL pull and the test connector: a correct load verifies and a corrupted byte fails it under fail-closed. A consumer with 16-token blocks after a 16-token local prefix hit, loading from a producer with 32-token blocks, is unverified. The previous test that different block sizes are unverified is replaced. - Comparing unsummed blocks makes the 16/32 cases fail; without the alignment check the unaligned load verifies without its first loaded block. test_kv_checksum.py (from the first commit of the series) already checks that two 16-token blocks add up to the 32-token block with the same tokens. Validation ---------- On GPU, NIXL pull with producer block size 16 and consumer block size 32, on an earlier base, reported a mismatch for 4 of 4 requests in a run without injected faults: NIXL on that base corrupted this transfer (fixed upstream by vllm-project#58860). With recompute, the consumer's outputs matched the reference. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Signed-off-by: David Cheung <d7cheung@gmail.com>
Add the second way a connector can carry KV checksums, for connectors whose consumer gets nothing back through kv_transfer_params once the producer has finished, such as NIXL push and Mooncake P2P (next commits). KVChecksumCarrier.CONNECTOR: the connector moves the checksums in its own messages. The generic code and the connector exchange only bytes; the comparison is unchanged. Carrier API ----------- - Producer: after request_finished, KVChecksumScheduler hands the summed checksums (encoded KVChecksumPayload bytes) to the new KVConnectorBase_V1.send_kv_checksums(request, checksums), instead of adding them to kv_transfer_params. - Consumer: the worker connector returns them in the new KVConnectorTransferResults.kv_checksums, keyed by request, by the time it reports the load in finished_recving. Both model runners pass them to KVChecksumWorker.post_forward, which adds them to its record (KVChecksumRecord.received), so they reach the scheduler on the existing per-worker channel. The scheduler takes the first copy for a pending load, before it compares the step's finished loads: every worker of the consumer may receive the same bytes. - MultiConnector forwards send_kv_checksums to the first sub-connector that carries the request's checksums, and merges kv_checksums of its sub-connectors' transfer results. - A load whose checksums never arrive is unverified. For both carriers, checksums are only sent for requests that finished normally (stop or length), not aborted ones. The consumer's worker connector must return the checksums by the time it reports the load finished, so the consumer compares them in the step that reports the load. Tests ----- test_kv_checksum_scheduler.py adds ConnectorCarrierConnector, a test connector that declares the connector-owned carrier and ships the checksums in its next step's metadata, and runs the end-to-end cases (layout independence, correct loads across TP/PP pairs and block sizes, and corrupted, missing or malformed checksums under each failure policy) for it as well as for the kv_transfer_params test connector and NIXL pull. Ignoring received checksums in the scheduler or the worker, or not calling send_kv_checksums, makes the connector-carrier cases fail. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Signed-off-by: David Cheung <d7cheung@gmail.com>
In NIXL push mode the producer WRITEs the KV into the consumer's blocks. The consumer's kv_transfer_params are built before the producer finishes, and nothing returns to the consumer through them afterwards, so KV checksums travel with the transfer itself: on the notif of the WRITE that delivers the KV, through the connector-owned carrier of the previous commit. Also drops an unreachable producer-side branch of the push worker's notif handling. NIXL push --------- - NixlBaseConnector.kv_checksum_carrier is now shared by pull and push; each sets its carrier (_KV_CHECKSUM_CARRIER): kv_transfer_params for pull, the connector for push. - P scheduler: send_kv_checksums keeps the checksums of a request whose finished blocks are about to be pushed, and build_connector_meta ships them with the blocks (NixlConnectorMetadata .push_finished_kv_checksums). Checksums of requests with nothing to push (e.g. aborted) are dropped, so nothing is left to clean up. - P worker: start_load_kv hands them to the writer thread with the finished blocks, in the same queue item; the writer keeps them until the request is evicted. _completion_notif builds the WRITE's notif: KV_CHECKSUM_NOTIF_PREFIX + msgpack((remote_request_id, tp_size, checksums)), instead of "remote_request_id:tp_size" when there are checksums. - D worker: _get_new_notifs parses both forms with the same completion accounting, keeps the first checksums received for a request it is loading, and get_transfer_results reports them with the request's finished_recving. - NIXL_CONNECTOR_VERSION 14 -> 15 (new notif form), so mixed versions fail the handshake instead of misreading the notif. Without checksums (disabled, or not summed on the producer) the notif keeps the plain form, so the default path is unchanged. Design choices -------------- - Why the next step's metadata: only the P worker's writer thread can send NIXL messages, and the scheduler reaches workers only through the next SchedulerOutput. The finished blocks to push already take that path (request_finished records them, the next build_connector_meta ships them), and the checksums ride along in the same metadata and queue item, so they add no latency to the WRITE. - Why the completion notif: it is already the consumer's signal that the load is done, and NIXL delivers it after the WRITE's data. The checksums therefore cannot arrive after the consumer reports the load finished, so the consumer never waits for them. A separate notif would have to be ordered against, and survive the loss of, the completion notif. - Why every writing producer rank carries them: which producer ranks write to a given consumer rank depends on the TP ratio, PP and MLA. A single designated carrier would need a proof that it always writes, otherwise the checksums would not arrive and the load would be unverified. Every rank that writes sends the same bytes (summed in the producer's scheduler), and the consumer keeps the first copy, without merging. The cost is duplicated payloads when several producer ranks write to one consumer rank. - The checksum payload inside the notif is opaque to NIXL and versioned on its own (_PAYLOAD_VERSION). Unreachable producer-side notif counting ---------------------------------------- In push mode, the only notifs D sends to P are PUSH_REG, which the writer thread consumes, and HB, which _get_new_notifs handles before the counting. Only pull_worker.py fills _pending_recv_notifs, so no code path sends P a plain "<req_id>:<n>" notif. The producer-side branch of NixlPushConnectorWorker._get_new_notifs was carried over from pull mode and never runs: P reports finished sends from its WRITE handles in get_transfer_results. This commit removes the branch, its guard, the transfer_topo assert it needed and the counter cleanup for finished pushes. _get_new_notifs now counts only consumer-side completion notifs and always returns an empty set. A completion notif for a request that is also being sent (one request both loading and saving on a kv_both instance) is now counted as a load instead of as a send. With a tripwire in the removed branch, test_nixl_push_connector, test_nixl_heartbeat and test_nixl_connector pass without reaching it, as did the GPU push runs below. Known limitations ----------------- - If a producer's lease expires while its WRITE waits for a deferred handshake, the writer drops the checksums with the request and the late WRITE (a pre-existing hazard: its blocks may have been freed) goes out without them. The consumer reports the load unverified, which fails it under fail-closed, rather than a mismatch. - A consumer load that never completes (some notifs lost) keeps its received checksums, as it already keeps its recv metadata. - The notif grows by 8 bytes per block and group (about 64 KB for a 128k-token prompt with 16-token blocks), sent once per WRITE. PUSH_REG carries block-id lists of similar size. - A checksum notif that fails to decode is logged and dropped: like a lost notif, the consumer's load then never completes and the request waits until aborted. - Push allows larger producer than consumer blocks; a consumer prefix hit that ends inside a producer block leaves the load unverified (see the block-size commit). The open upstream vllm-project#59747 proposes rejecting different block sizes in push at handshake. Tests ----- - test_kv_checksum_scheduler.py runs the end-to-end cases for NixlPushConnector as well: the producer's push metadata carries the checksums, and the consumer's workers report them with the finished load. - test_nixl_push_connector.py: the P scheduler ships checksums with the finished blocks and drops those of an aborted request; start_load_kv queues them for the writer; a completion notif built by a producer worker, with and without checksums, completes the consumer's load and reports its checksums. It no longer sets transfer_topo or checks the return value of _get_new_notifs. Not reporting the received checksums, dropping them from the notif, or not sending them makes these tests fail. Validation ---------- On GPU, end-to-end runs with a fault-injecting connector (not part of this series) met their expected results for push with TP1 -> TP1, producer TP2 -> consumer TP1 and producer TP1 -> consumer TP2: no false mismatch, every injected corruption detected, no host sync. These runs used Qwen/Qwen3-0.6B; no MLA model was run. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Signed-off-by: David Cheung <d7cheung@gmail.com>
This was referenced Oct 14, 2026
mergify
Bot
added
minimax
mrv2
sawsa307
marked this pull request as ready for review
October 14, 2026 20:16
sawsa307
requested review from
ApostaC,
NickLucche,
WoosukKwon,
ZJY0516,
alexm-redhat,
heheda12345,
ivanium,
njhill,
orozery,
robertgshaw2-redhat,
wzhao18,
yewentao256 and
ywang96
as code owners
October 14, 2026 20:16
![claude[bot]](https://avatars.githubusercontent.com/in/1236702?s=60&v=4){kind=small}
sawsa307
requested review from
ProExpertProg,
houseroad,
mgoin,
tlrmchlsmth and
xuechendi
as code owners
October 14, 2026 20:16
Contributor
|
This pull request has merge conflicts that must be resolved before it can be |
Member
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Overview
Part 3 of the KV checksum series: KV checksums for NIXL push. In push mode, the consumer's
kv_transfer_paramsare built before the producer finishes, so the checksums cannot travel in them. This PR adds a second, connector-owned carrier and uses it to send the checksums on the completion notif of the WRITE that delivers the KV.Depends on #61579 (2/N), which depends on #61578 (1/N). Only the last two commits are new here, starting at "KV checksums: connector-owned carrier". The full series and its validation are in #61488.
Not a duplicate: this extends the series; no other open PR checksums P/D transfers.
Claims
KVChecksumCarrier.CONNECTOR, lets a connector move the checksums in its own messages.KVConnectorBase_V1.send_kv_checksums.KVConnectorTransferResults.kv_checksumsby the time it reports the load finished.NIXL_CONNECTOR_VERSIONgoes from 14 to 15 for the new notif form. Without checksums, the notif keeps its old form.Validation
Unit tests (CPU):
673 passed, 1 skipped, 3 failed. Three tests fail for reasons unrelated to this change, and fail the same way on upstream main
a2fb482732:test_abort_timeout_on_prefiller[ray]and[None]need Ray and the compiled CPU ops, andtest_multi_example_connector_consistencyneeds a gated model. pre-commit (ruff, mypy) passes on every commit.test_nixl_push_connector.pybuilds a completion notif on a producer worker and parses it on a consumer worker, with and without checksums.send_kv_checksumsis never called;End to end on GPU (the full series in #61488 at
ef3d386f31):VLLM_GPU_SYNC_CHECK=error.Before marking this ready, I will add a GPU run of this PR's own tree.
Details
Two commits:
_get_new_notifsnow counts only consumer-side completion notifs. A request that both loads and saves on akv_bothinstance now has its completion notif counted as a load.Limitations:
Pull Request Checklist
I used vLLM's
/pr-checklistskill. (Mandatory for agents, optional for humans).AI assistance was used during the creation of this PR.
Design Fit: Minimizes impact on core components, reuses existing functionality, and justifies added complexity.
Testing and Validation: Validates the change and ensures any added tests are meaningful and reliable, with CI coverage or documented CI resource constraints and validation performed outside CI. (GPU run of this PR's own tree pending; see Validation.)
Code Quality and Style: Keeps code and comments clear and concise, and updates relevant documentation and examples.
Pull Request Contents: Includes a brief summary and relevant links, supports claims with evidence, explains root causes and implementation trade-offs, and follows the contributing guide.
AI assistance (Claude) was used to write and review this change. I reviewed every changed line and ran the validation above.
🤖 Generated with Claude Code