Repository navigation
Conversation
First of a series adding connector-agnostic checksums of transferred KV
for P/D disaggregation. Some transfer failures are not reported by the
transport: corrupted bytes, a read after the producer has already
reused its blocks, or blocks placed at the wrong position. With
kv_transfer_config.enable_kv_checksum on both sides, the producer
checksums a request's KV in the step that finishes its prefill, the
checksums travel with the request in kv_transfer_params, and the
consumer verifies every load against them before the request runs.
kv_checksum_fail_closed turns a failed check into a KV load failure,
handled by kv_load_failure_policy.
No connector declares a carrier in this commit, so enabling KV
checksums alone has no effect yet; the next commit makes NIXL pull the
first connector that carries them. Later in this series: KV heads
replicated across TP, different producer and consumer block sizes,
NIXL push and its connector-owned carrier, Mooncake P2P, Mamba state,
and a Triton kernel.
Design
------
- Connector-agnostic. Workers compute the checksums from the KV cache
itself, the scheduler sums and compares them, and a connector only
declares how it carries the bytes (kv_checksum_carrier). Computing
them inside each connector was rejected: every connector would
reimplement it, and the values would depend on the connector's view
of TP and cache layout.
- One value per (KV cache group, block position), independent of TP,
PP and memory layout. The scheduler, the only place that sees every
worker's record, sums the TP and PP parts on both sides, so the
consumer needs no knowledge of the producer's layout.
- A failed check is a KV load failure. It takes the path
connector-reported failures take (_handle_failed_recving), so
kv_load_failure_policy decides what happens next and no new failure
handling is added.
- No host sync on the engine step: results reach host memory on a side
copy stream and an event.
- The producer checksums the whole prompt once, in the step that
finishes it, with or without chunked prefill. Per-chunk partial sums
were rejected: they would need per-rank state in the scheduler,
cleanup on preemption, and handling of the block spanning two chunks.
The cost is that all of the prompt's checksum work lands in its last
prefill step.
What a checksum is
------------------
A checksum covers one block position of one KV cache group, over the
unsharded model: all layers of the group and all KV heads.
1. A slice is the KV of one layer and one KV head in the block: the K
and V of the block's tokens for that head. Laid out token by token
and split into little-endian 8-byte pieces, its checksum is
s = sum_j x_j * (2 * j + 1) mod 2**64
where x_j is piece p of the token at position t in the request,
j = t * P + p with P pieces per token.
2. The block's checksum adds up the slices of every layer and head,
each times its own odd factor:
c = sum_{layer, head} m * s mod 2**64
m = 2 * (layer * H + head) + 1
H = 2**16 (MAX_KV_HEADS_PER_LAYER), a fixed bound on any layer's KV
heads. Layers are numbered by the layer number in their names and
heads globally across TP, so m does not depend on the model's head
counts or its TP or PP split.
Why this definition:
- Factors and weights are odd, hence invertible mod 2**64: a change to
any single piece always changes c.
- Distinct weights catch tokens in the wrong place; distinct factors
catch swapped heads or layers. Random corruption of several pieces
goes undetected with probability about 2**-64. It is not a
cryptographic hash and does not defend against deliberate tampering.
- Numbering pieces from the request's first token makes the checksums
of consecutive blocks add up to that of the tokens they hold, so
different block sizes can later be compared without changing the
definition.
- TP splits KV heads and PP splits layers, so every slice lies whole on
one worker, and the workers' parts sum to c whatever the memory
layout (head-major, token-major, kernel-block splits).
The last block of a request only covers its first num_tokens slots,
which excludes slots that later decode or draft tokens may write.
Life of a request
-----------------
Producer:
1. In the step whose scheduled tokens cross num_prompt_tokens,
Scheduler._update_after_schedule calls KVChecksumScheduler.add_send
for requests the connector carries checksums for. build_checksum_meta
puts their blocks, per transfer group, into
SchedulerOutput.kv_checksum_scheduled.reqs_to_send.
2. KVChecksumWorker.post_forward checksums them after the forward has
written their KV. Rows for all groups are built with numpy and copied
to the device in one non-blocking H2D copy from pinned memory. The
worker's own copy stream then copies the sums to pinned host memory
and records a blocking event; KVChecksumOutput.finalize, called when
the output is pickled for the scheduler, only waits for that event.
The results ride KVConnectorOutput.kv_checksums, which
KVOutputAggregator merges across TP and PP workers.
3. Scheduler._free_request calls request_finished after the connector's
request_finished. For a request that finished normally it sums the
workers' parts and adds them to the connector's kv_transfer_params as
"kv_checksums". The proxy forwards them to the consumer with the
request.
Consumer:
1. When a request starts an async load, Scheduler.schedule calls
add_recv. It decodes the expected checksums from the request's
kv_transfer_params and puts the blocks the load writes, tokens
[num_local_computed_tokens, num_computed_tokens), into reqs_to_recv
in the same step, so workers know them before the load starts.
2. Workers checksum a load in the step whose transfer results report it
finished, before any later forward can write its blocks; a request
whose load failed or that ended is dropped. update_from_output keeps
each worker's record until the aggregated finished_recving reports
the request.
3. It then compares, after the connector's own failed_recving and
invalid_block_ids are handled (loads already known to be bad are not
compared), and before the request can be scheduled again.
On the v1 runner with a drafter that writes KV after the connector
step (defer_finalize), post_forward leaves the sends to finalize_sends,
which finalize_kv_connector calls after the drafter, on the last PP
rank only.
Which blocks
------------
- Producer: per transfer group, the blocks holding tokens
[0, num_computed_tokens). Consumer: the blocks the load writes.
- Both: leading null blocks (out of a sliding window) are left out, a
group with a null block after a non-null one is left out entirely,
and sliding-window groups keep their last cdiv(window, block_size) + 1
blocks, the clip NIXL and Mooncake apply to what they transfer. A
connector that carries checksums must load exactly these blocks.
- Only positions both sides checksummed are compared, and the last
block only if both sides hold the same number of tokens in it.
Verdicts
--------
Each group is compared on its own:
- Mismatch: some compared position differs. Logged as an error with
the groups and block positions, even if other groups are unverified.
- Unverified: the group cannot be compared: it is checksummed by one
side only, its fingerprint differs (other layers or dtype), its block
sizes differ, a worker checksummed other blocks than scheduled, or no
position is in common.
A load is also unverified as a whole when not every worker (TP * PP)
reported it, or the producer sent no (valid) checksums. Unverified
loads log a warning once per fixed reason, and the details per request
at debug level.
Each worker record carries a fingerprint per group, the sum of a hash
of each of its layers in it (name, spec type, dtype). Summed over PP
stages it identifies the group's layers whatever the PP layout; the
scheduler's own KVCacheConfig only lists the first worker's layers.
Groups a worker cannot checksum are logged at startup and reported in
the record's skipped_groups: layers without per-token KV blocks (Mamba
state), compressed caches (tokens_per_state != 1), KV heads packed into
head slots, host-resident groups, caches that are not whole 8-byte
pieces or whose kernel blocks do not tile the block size, KV heads
replicated across TP, two caches with one layer number, and layer names
without one. A group skipped on both sides is left out of the verdict,
so hybrid models verify their attention groups under fail-closed; a
group skipped on one side only is unverified.
Failure semantics
-----------------
- Fail-open (default): mismatched and unverified loads are only logged;
the loaded blocks are then used and prefix-cached as usual. This lets
checksums be rolled out, and observed, without failing requests.
- Fail-closed: mismatched and unverified loads fail through
_handle_failed_recving: kv_load_failure_policy "fail" ends the request
with an error, "recompute" sets num_computed_tokens to 0 and
recomputes the whole prompt locally on the consumer.
- The KV is never transferred again: connectors clear do_remote_prefill
on the first load. Recovery is per request, not per block.
Recomputing only from the first bad block is future work, building on
group-aware load-failure recovery (RFC vllm-project#47812, PR vllm-project#48216).
Carrier
-------
KVConnectorBase_V1.kv_checksum_carrier(request, receiving) returns how
the connector carries a request's checksums, or None (the default) if
it does not, e.g. offloading connectors. This commit defines
KVChecksumCarrier.KV_TRANSFER_PARAMS: generic code adds and reads the
"kv_checksums" key, and the connector needs no other change.
- MultiConnector: for a load, the sub-connector chosen to load it; for
a producer request, the first sub-connector with a carrier.
- A connector without a carrier can verify no load: enabling KV
checksums with it fails at startup when fail-closed and disables them
with a warning when fail-open.
Wire format: base64 of msgpack (version, tp_size, num_tokens,
[(group_id, fingerprint, block_size, start, little-endian uint64
values)]). A producer whose workers did not all report a request (or
reported other blocks) sends no checksums for it, with a warning. For a
128k-token prompt with one group and 16-token blocks this is about
87 KB of base64. Proxies must forward unknown kv_transfer_params keys.
kv_checksum_enabled disables KV checksums, with a warning, for context
parallelism (DCP/PCP > 1), and for the Ray executor with PP > 1 unless
VLLM_USE_RAY_V2_EXECUTOR_BACKEND=1: its non-last stages' connector
outputs never reach the scheduler.
Rollout and limits
------------------
- Enable on producers before consumers; otherwise consumers see loads
without checksums, unverified (failed when fail-closed).
- A producer request preempted in its final prefill step and
re-prefilled before that step's output arrives can mix the two steps'
values; the consumer then reports a mismatch or unverified, never a
false match.
- In this commit, replicated KV heads, several caches per layer and
Mamba state are skipped, different block sizes are unverified, and
the PyTorch path reads the gathered KV several times per layer; later
commits in the series address each.
Tests
-----
- test_kv_checksum.py checks every worker result against an independent
NumPy implementation of the definition: head-major and token-major
layouts and kernel-block splits; TP2 shards and PP2 stages summing to
the whole model's value; swapped tokens and layers changing it; slots
past num_tokens ignored; 16-token blocks adding up to 32-token blocks;
KV-sharing counted once; chunked gathers; every skipped group kind
next to a checksummed group; the consumer lifecycle (finish, failure,
abort); deferred sends covering KV written before finalize_sends;
fingerprints independent of the PP split; serialization and
aggregation across workers.
- test_kv_checksum_scheduler.py runs a producer's and a consumer's real
schedulers with real checksum workers, copying the producer's blocks
into the consumer's, with a minimal test connector that only declares
the kv_transfer_params carrier:
- producers of TP2, PP2 and TP2xPP2 send what a single worker sends;
requests the connector does not carry get no checksums; nothing is
sent unless every worker reported;
- correct loads verify for (TP, PP) pairs (1, 1)/(1, 1), (2, 1)/(1, 1),
(1, 1)/(2, 1), (1, 2)/(1, 1) and (1, 1)/(2, 2);
- with a local prefix hit only the loaded blocks are compared;
- a corrupted byte (logged as a mismatch), missing and malformed
checksums: only logged when fail-open, an error with policy "fail",
a local recompute with "recompute";
- different block sizes are unverified;
- a group skipped on both sides is ignored, on one side only is
unverified;
- a connector without a carrier is refused at startup when
fail-closed and disables checksums when fail-open.
Disabling the comparison,
ignoring the received checksums, summing fingerprints per worker
instead of per PP stage, numbering heads per rank, or checksumming
loads from the first block each make tests fail.
- test_scheduler.py and test_async_scheduler.py set the new scheduler
attribute on their hand-built schedulers.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Signed-off-by: David Cheung <d7cheung@gmail.com>
Make NIXL pull the first connector that carries KV checksums, in the kv_transfer_params the previous commit added them to: a producer's request_finished already returns kv_transfer_params that the proxy forwards to the consumer, so NixlPullConnector only declares the carrier. With kv_transfer_config.enable_kv_checksum on both sides, NIXL pull loads are now verified end to end. NixlPullConnector.kv_checksum_carrier returns KVChecksumCarrier.KV_TRANSFER_PARAMS for producer requests (do_remote_decode) and for regular loads. A bidirectional load (a load with do_remote_decode set, a prefiller reloading a decoder's KV from an earlier turn) reads blocks whose source sent no checksums, so it returns None: such loads are neither verified nor counted as unverified. Tests ----- test_kv_checksum_scheduler.py runs the end-to-end scheduler tests that do not depend on the connector for NIXL pull as well as for the test connector: correct loads verify for every (TP, PP) pair, and a corrupted, missing or malformed load follows kv_checksum_fail_closed and kv_load_failure_policy. Making NixlPullConnector declare no carrier makes the NIXL pull cases fail. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Signed-off-by: David Cheung <d7cheung@gmail.com>
mergify
Bot
added
mrv2
Address review on the KV checksum framework:
- Drop the Model Runner V1 hooks (gpu_model_runner.py and
kv_connector_model_runner_mixin.py). V1 now rejects
enable_kv_checksum at startup ("Model Runner V1 does not support:
KV checksums"). With them go post_forward's defer_sends and
finalize_sends: they existed for V1's drafter, which writes KV after
the connector step. V2 calls post_forward after the speculator has
proposed, so the sends already cover the drafter's KV.
- ActiveKVConnector creates the checksum worker and holds it as a
field, instead of a module-level global set by
init_kv_checksum_worker. create_kv_checksum_worker returns the
worker, or None when checksums are off. get_kv_connector takes the
KV cache config for it.
- The copy to host always runs on a torch.cuda.Stream. ROCm and XPU
provide it, and the CPU worker stubs streams and events, so the
synchronous fallback for devices without streams is gone. The
connector unit tests stub them the same way where CUDA is
unavailable (fixture cuda_stream_stubs).
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Signed-off-by: David Cheung <d7cheung@gmail.com>
vadiklyutiy
left a comment
vadiklyutiy
left a comment
There was a problem hiding this comment.
I think adding checksum is overcomplicated. I am sure we can solve the original issue without checksum
Thanks for reviewing! The checksum feature here is more of a safety net to catch the KV cache inconsistency between P and D side. The intent is to allow us to verify the KV cache from P when we initiate the transfer is indeed the same as the one received on the D side. The issue in #61229 is an example I quoted and ran into when doing local verification, but there should be a proper fix to fix it instead of relying on checksum. |
|
I think adding checksum is overcomplicated. I am sure we can solve the original issue without checksum
what another cases (except #61229) will be fixed with checksum? |
Besides #61229, checksums can catch two other NIXL transfer bugs during validation. Both corrupted KV silently: no error on either side, HTTP 200, wrong output.
I agree each of these needs its own fix, and that checksums are not a substitute for fixing them. The case for checksums is that these bugs are silent by nature. The transfer path now covers heterogeneous TP, different block sizes, MLA/replicated heads, Mamba state and push mode. |
|
@vadiklyutiy yes the motivation is multi-tenant scenarios where privacy is critical - this would be an opt-in safety net to avoid leaking one user's data to another. I think we can keep the necessary logic pretty isolated so it shouldn't be too bad complexity-wise in terms of core invasiveness. |
![claude[bot]](https://avatars.githubusercontent.com/in/1236702?s=60&v=4){kind=small}
|
Hi @sawsa307 thanks for the great work. Regarding this feature. Have you evaluated make the checksums be triggered from the model runner and the connector instead of the scheduler? Each of these hooks decides “the KV is final” from counters that were wrong this year (#34616, #58259, #58668). A checksum driven by those counters doesn’t raise false alarms in these cases, but it can’t catch them either: when the counters say rows are computed and the rows hold the wrong tokens, both sides checksum the same wrong bytes and agree. A checksum triggered by the KV writes themselves, recording which token was written at which position, does catch them. |
|
This pull request has merge conflicts that must be resolved before it can be |
Thanks @yubofredwang for the comment. The checksums only verify that the consumer received the same bytes, at the same positions, as the producer holds. If a bookkeeping bug leaves the wrong KV under a request on the producer, both sides hash the same bytes and agree. That is the intended scope: transfers that complete without error but deliver the wrong KV. Transports already checksum bytes in flight, so these come from lifetime and layout bugs, e.g. reading blocks after the producer reused them (#61229), or a P/D layout mismatch (#58860, #61526). Hashing at write time wouldn't change this: it would hash the wrong KV when it is written. I don't think that check belongs in the P/D checksums for the following reasons:
I think a better place for it is as a test oracle. Extend the randomized scheduler tests (e.g. |
Agreed. Checksums could be useful beyond connectors, but if the scope is P/D only, bugs that also hit deployments without P/D belong in the test oracle you describe, and a scheduler-driven checksum is fine for that scope. One counterexample is #34616. It was on the consumer's async-load path, so it existed only in P/D. Its shape, D computing over rows P never sent, still passes this PR under fail-closed.
That is the behavior we want. Failing or recomputing is what keeps one tenant's KV out of another tenant's request, which is @njhill's motivation. It is also what this PR does for the bugs.
For agentic multi-turn traffic it is the other way around. I benchmarked both triggers with this PR's checksum definition:
|
Yeah this is something came up when I discussed with @njhill as well. |
|
@yubofredwang fyi I have been doing some rework on top of this (actually on top of the full original PR #61488), I will push some updates soon. |
Address review on the KV checksum framework:
- Drop the Model Runner V1 hooks (gpu_model_runner.py and
kv_connector_model_runner_mixin.py). V1 now rejects
enable_kv_checksum at startup ("Model Runner V1 does not support:
KV checksums"). With them go post_forward's defer_sends and
finalize_sends: they existed for V1's drafter, which writes KV after
the connector step. V2 calls post_forward after the speculator has
proposed, so the sends already cover the drafter's KV.
- ActiveKVConnector creates the checksum worker and holds it as a
field, instead of a module-level global set by
init_kv_checksum_worker. create_kv_checksum_worker returns the
worker, or None when checksums are off. get_kv_connector takes the
KV cache config for it.
- The copy to host always runs on a torch.cuda.Stream. ROCm and XPU
provide it, and the CPU worker stubs streams and events, so the
synchronous fallback for devices without streams is gone. The
connector unit tests stub them the same way where CUDA is
unavailable (fixture cuda_stream_stubs).
Ported from vllm-project#61578 (1a2920b) onto the combined branch: kept
post_forward's received checksums and replica_zero_only, and the
layer view warm-up.
Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
Signed-off-by: David Cheung <d7cheung@gmail.com>
Signed-off-by: Nick Hill <nickhill123@gmail.com>
|
@sawsa307 OK I spent longer on this than originally intended! As mentioned I was working on the full change from #61488 along with the updates made so far in this one. Here is my branch containing a bunch of updates: main...njhill:vllm:kv-checksum-upstream, it's also rebased on newer main. Summary of main changes:
I ran some benchmarks which show no measurable overhead now of enabling checksums across different concurrencies/prompt lengths. Please take a look and let me know what you think. Have not looked at implementing the checksum caching we discussed yet, it might not be worth it now, especially if it adds nontrivial complexity. Next things to look at could be other cases that @yubofredwang was discussing, and figuring out what if anything makes sense for the offloading case too. Not sure how we want to proceed, could open my branch as a new PR (you could push a copy of a PR and open that?) And in any case let's split this change out first: #61580 (comment) |
Address review on the KV checksum framework:
- Drop the Model Runner V1 hooks (gpu_model_runner.py and
kv_connector_model_runner_mixin.py). V1 now rejects
enable_kv_checksum at startup ("Model Runner V1 does not support:
KV checksums"). With them go post_forward's defer_sends and
finalize_sends: they existed for V1's drafter, which writes KV after
the connector step. V2 calls post_forward after the speculator has
proposed, so the sends already cover the drafter's KV.
- ActiveKVConnector creates the checksum worker and holds it as a
field, instead of a module-level global set by
init_kv_checksum_worker. create_kv_checksum_worker returns the
worker, or None when checksums are off. get_kv_connector takes the
KV cache config for it.
- The copy to host always runs on a torch.cuda.Stream. ROCm and XPU
provide it, and the CPU worker stubs streams and events, so the
synchronous fallback for devices without streams is gone. The
connector unit tests stub them the same way where CUDA is
unavailable (fixture cuda_stream_stubs).
Ported from vllm-project#61578 (1a2920b) onto the combined branch: kept
post_forward's received checksums and replica_zero_only, and the
layer view warm-up.
Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
Signed-off-by: David Cheung <d7cheung@gmail.com>
Signed-off-by: Nick Hill <nickhill123@gmail.com>
Address review on the KV checksum framework:
- Drop the Model Runner V1 hooks (gpu_model_runner.py and
kv_connector_model_runner_mixin.py). V1 now rejects
enable_kv_checksum at startup ("Model Runner V1 does not support:
KV checksums"). With them go post_forward's defer_sends and
finalize_sends: they existed for V1's drafter, which writes KV after
the connector step. V2 calls post_forward after the speculator has
proposed, so the sends already cover the drafter's KV.
- ActiveKVConnector creates the checksum worker and holds it as a
field, instead of a module-level global set by
init_kv_checksum_worker. create_kv_checksum_worker returns the
worker, or None when checksums are off. get_kv_connector takes the
KV cache config for it.
- The copy to host always runs on a torch.cuda.Stream. ROCm and XPU
provide it, and the CPU worker stubs streams and events, so the
synchronous fallback for devices without streams is gone. The
connector unit tests stub them the same way where CUDA is
unavailable (fixture cuda_stream_stubs).
Ported from vllm-project#61578 (1a2920b) onto the combined branch: kept
post_forward's received checksums and replica_zero_only, and the
layer view warm-up.
Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
Signed-off-by: David Cheung <d7cheung@gmail.com>
Signed-off-by: Nick Hill <nickhill123@gmail.com>
|
General idea, I think this checksum is hard to avoid:
|
|
|
||
| enable_kv_checksum: bool = False |
There was a problem hiding this comment.
enable_kv_checksum and kv_checksum_fail_closed moved to a Literal will be more clear
kv_checksum_mode: Literal["off", "log", "enforce"]
There was a problem hiding this comment.
I have changed it to only enforce
Overview
First of a series adding connector-agnostic checksums of transferred KV for P/D disaggregation, so that transfers that complete without error but deliver the wrong KV are caught before the request runs: a read after the producer has already reused its blocks, or KV placed at the wrong position because the two sides disagree on the layout. This PR adds the framework and makes NIXL pull its first connector.
With
kv_transfer_config.enable_kv_checksumon both sides, the producer checksums a request's KV in the step that finishes its prefill. The checksums travel to the consumer with the request inkv_transfer_params, and the consumer compares every load against them. Underkv_checksum_fail_closed, a mismatched or unverifiable load is handled as a KV load failure, sokv_load_failure_policyapplies (fail, orrecomputelocally).Series (each PR stacks on the previous one; only its own commits are new):
#61488 holds the whole series in one draft as the reference implementation, along with its end-to-end validation.
Not a duplicate: #57912 and RFC #54363 add integrity checks for the filesystem KV offload tier, while this series covers producer-to-consumer transfers. No other open PR checksums P/D transfers.
Claims
Validation
Unit tests (CPU):
501 passed, 1 skipped, 3 failed. Three tests fail for reasons unrelated to this change, and fail the same way on upstream main
a2fb482732:test_abort_timeout_on_prefiller[ray]and[None]need Ray and the compiled CPU ops, andtest_multi_example_connector_consistencyneeds a gated model. pre-commit (ruff, mypy) passes on every commit.test_kv_checksum_scheduler.pyruns a real producer scheduler and a real consumer scheduler with real checksum workers, copying the producer's blocks into the consumer's. It covers a test connector that declares thekv_transfer_paramscarrier as well as NIXL pull:fail, and is recomputed withrecompute;Mutation checks: each of the following edits makes tests fail. Two are worker-side, numbering heads per rank and summing fingerprints per worker instead of per PP stage. The rest are scheduler-side: disabling the comparison, ignoring received checksums, checksumming loads from position 0, and having NIXL pull declare no carrier.
End to end on GPU. These runs used the full series in #61488 (
ef3d386f31), which contains this PR's code plus the later PRs; this PR's own tree has not been run on GPU yet. All of them are NIXL pull.VLLM_BATCH_INVARIANT=1, temperature 0, the first 200 gsm8k questions. Checksums on vs off: 200/200 outputs have identical token ids, with 0 mismatches over 200 transfers.Before marking this ready, I will add a GPU run of this PR's own tree. This PR checksums with the PyTorch path; #61488 used the Triton kernel. The run covers the determinism test and a corrupted-byte injection, with NIXL pull and
VLLM_GPU_SYNC_CHECK=error.Details
The commit messages give the design, the alternatives that were rejected, the known limitations and the tests. Two commits:
kv_transfer_params["kv_checksums"]. The consumer compares each KV cache group separately and reports it as matching, mismatched or unverified.kv_checksum_carrier, and MultiConnector delegates it to its sub-connectors.kv_transfer_paramscarrier (11 lines). Bidirectional loads, where a prefiller reloads a decoder's KV, are not checksummed.Cost: the producer does its checksum work in the step that finishes a prefill, and the consumer in the step where a load completes. This PR uses the PyTorch path; the Triton kernel comes in 6/N. With that kernel, the full series on Qwen3-8B P/D TP2 -> TP2 adds +3.5 ms TTFT at 1K prompts with concurrency 1, and −3.8% request throughput at concurrency 32. The change is under 1% for 8K prompts under saturation and for 32K prompts.
Limitations:
kv_transfer_paramskeys.Pull Request Checklist
I used vLLM's
/pr-checklistskill. (Mandatory for agents, optional for humans).AI assistance was used during the creation of this PR.
Design Fit: Minimizes impact on core components, reuses existing functionality, and justifies added complexity.
Testing and Validation: Validates the change and ensures any added tests are meaningful and reliable, with CI coverage or documented CI resource constraints and validation performed outside CI. (GPU run of this PR's own tree pending; see Validation.)
Code Quality and Style: Keeps code and comments clear and concise, and updates relevant documentation and examples.
Pull Request Contents: Includes a brief summary and relevant links, supports claims with evidence, explains root causes and implementation trade-offs, and follows the contributing guide.
AI assistance (Claude) was used to write and review this change. I reviewed every changed line and ran the validation above.
🤖 Generated with Claude Code