Conversation
…d memory Authenticate immutable model, layout and token-prefix identities in a bounded durable directory. Publish a generation only after every rank stores its complete payload. Retain SHM leases until copy completion or explicit drained cancellation; unresolved ownership stops admission. Worker-owned background transfers keep tensor payloads out of metadata RPC and require no GPU context in the sidecar. Validation: 42 index, identity and real SHM/RPC storage tests plus 81 engine-driven transfer tests passed. Coverage includes process-restart filesystem retrieval, partial rank failures, capacity rejection, read-lock eviction, late lease and completion replies, and lost-ownership admission shutdown. Pre-commit checks passed. Status: implemented; composed GPU lifetime qualification remains a release gate. Co-authored-by: Codex <noreply@openai.com> Signed-off-by: Martin Vit <martin@voipmonitor.org> (cherry picked from commit b7977f4)
Import target, recurrent, auxiliary and draft pages through the atomic vLLM checkpoint allocator. Publish only all-rank successful copies, preserve cancelled request pins until admitted transfers drain, and prevent reused public request IDs from consuming predecessor bookkeeping. CUDA work uses existing model-worker streams and pinned SHM; ordinary aligned transfers remain separate. Validation: eight real allocator/MQ ownership tests passed, including cancellation, reused request IDs, per-rank failure, LoRA namespace isolation and immutable revision requirements. Storage and transfer suites also passed. The GPU checkpoint copier retains the platform CUDA stream/event primitives. Complete-image qualification is pending; status: implemented. Co-authored-by: Codex <noreply@openai.com> Signed-off-by: Martin Vit <martin@voipmonitor.org> (cherry picked from commit 427d926)
Propagate the explicit HTTP force flag through the cache server and management module to L1 eviction. Non-forced clearing retains read/write-locked objects; an omitted flag and the argument-free CLEAR RPC retain their forced behavior. No wire identifier or payload changes are required. Validation: two real SHM regressions fail before the correction and pass afterward, covering prefetch before slot exposure, active read leases, pending write leases, unlocked eviction and release. All 94 storage and HTTP tests pass; Python pre-commit checks pass. Concurrent GPU qualification is pending; status: implemented. Co-authored-by: Codex <noreply@openai.com> Signed-off-by: Martin Vit <martin@voipmonitor.org> (cherry picked from commit 6b6074c)
…ases Reject failed begin and copy submissions with terminal ownership results, retain fatal unsafe-copy behavior, and release store admission counters even when rollback raises. Require bound worker state before emitting rank metadata. Successful copy layout, GPU kernels and durable recency semantics are unchanged. Validation: seven fault-injection cases plus 56 existing SHM, identity, index and allocator tests pass; Python pre-commit and mypy pass.
Give each checkpoint rank/storage pair a separate payload-key namespace. Checkpoint groups use different logical page positions and group-specific hashes, so they are not the aligned rank/group families required by ordinary chunk-coherent LRU eviction. The manifest and complete-payload retrieval own generation atomicity; active SHM copies retain their existing read/write leases. Retry capacity-only store reservations with the storage manager's bounded eviction backpressure, releasing all partial reservations before each retry. Conflicts and invalid generations remain immediate admission failures. Payload keys use the version-2 namespace. Version-1 filesystem objects produce safe cache misses and recomputation, never partial or mixed-generation restores. Model arithmetic, cache-page geometry, and ordinary chunk eviction are unchanged. Validation: global and isolated LRU reproduce zero eviction candidates before the fix. All 55 checkpoint storage/index and chunk-coherent eviction tests pass, including sustained stores beyond pool capacity, pinned SHM bytes, rank-failure cleanup, and filesystem restart. Python pre-commit checks pass; Rust hooks are explicitly skipped because no Rust source changes. (cherry picked from commit 6168170)
Return a negative rank completion when CUDA event creation or recording fails before copy submission. Continue draining other tasks so collective pins can be released. Unsafe in-flight copy failures remain fatal. Normal transfer dispatch is unchanged. Validation: both event fault injections fail before this change; 58 checkpoint connector, storage and index tests pass afterward. Python pre-commit hooks pass; Rust hooks are explicitly excluded for these Python-only files. Assisted-by: OpenAI Codex (cherry picked from commit 617a1b4)
Signed-off-by: derek <derek.yates@live.com>
|
Important Review skippedAuto reviews are disabled on base/target branches other than the default branch. 🗂️ Base branches to auto review (1)
Please check the settings in the CodeRabbit UI or the ⚙️ Run configurationConfiguration used: Organization UI Review profile: CHILL Plan: Advanced Run ID: You can disable this status message by setting the Use the checkbox below for a quick retry:
Thanks for using CodeRabbit! It's free for OSS, and your support helps us grow. If you like it, consider giving us a shout-out. Comment |
Signed-off-by: derek <derek.yates@live.com>
2a09b82 to
da7df92
Compare
Composed GPU qualification — passedQualified commit Correctness and ownership
Cross-replica and persistence
Write amplification
PerformanceMatched local-inference-lab sustained decode, pristine R30 → patched:
Matched cold LMCache write-through prefill: 32K Auto fairness adapted compute share across approximately 0.31–0.52 during three 65,535-token prefill/decode collisions. Decode continued without starvation and recovered after every prefill. The final provenance build and the tested build have identical root filesystem layer hashes; only the OCI revision label changed when the PR was rebased into this clean one-commit review. |
|
Public qualification image: OCI index digest: The package is public; anonymous manifest access was verified with an empty Docker credential directory. |
|
Validated and included in the two-layer R32 image, preserving Derek's original
This is one TP4 engine, not an independent reproduction of the author's Image: @coderabbitai review |
|
Tip For best results, initiate chat on the files or code changes. 🧩 Analysis chain🏁 Script executed: #!/bin/bash
set -euo pipefail
printf '%s\n' '--- Pull request metadata ---'
gh pr view 65 --repo local-inference-lab/LMCache --json number,title,baseRefName,headRefName,files,url
printf '%s\n' '--- Changed files ---'
gh pr diff 65 --repo local-inference-lab/LMCache --name-only
printf '%s\n' '--- Diff stat ---'
git diff --stat "$(git merge-base HEAD "origin/$(gh pr view 65 --repo local-inference-lab/LMCache --json baseRefName --jq .baseRefName)")" HEAD
printf '%s\n' '--- Candidate checkpoint symbols ---'
rg -n -C 3 --glob '*.py' \
'class CheckpointPayloadStore\b|def prepare_store\b|class CheckpointTransferWorker\b|CheckpointLeaseResponse\b|AdmissionFailure\b|KEY_NOT_WRITABLE\b' .Length of output: 2540 🏁 Script executed: #!/bin/bash
set -euo pipefail
printf '%s\n' '--- Pull request diff ---'
gh pr diff 65 --repo local-inference-lab/LMCache --patch
printf '%s\n' '--- Source outlines ---'
for file in \
lmcache/v1/distributed/admission.py \
lmcache/v1/multiprocess/checkpoint_storage.py \
lmcache/v1/multiprocess/checkpoint_transfer.py \
lmcache/v1/multiprocess/modules/checkpoint.py \
lmcache/v1/multiprocess/protocols/checkpoint.py \
tests/v1/multiprocess/test_checkpoint_storage.py
do
printf '\n--- %s ---\n' "$file"
ast-grep outline "$file" --items all --type class,function --view expanded || true
done
printf '%s\n' '--- Focused source contexts ---'
rg -n -C 18 --glob '*.py' \
'def prepare_store\b|class CheckpointPayloadStore\b|class CheckpointTransferWorker\b|def _.*store\b|CheckpointLeaseResponse\b|AdmissionFailure\b|KEY_NOT_WRITABLE\b|busy' \
lmcache/v1/distributed/admission.py \
lmcache/v1/multiprocess/checkpoint_storage.py \
lmcache/v1/multiprocess/checkpoint_transfer.py \
lmcache/v1/multiprocess/modules/checkpoint.py \
lmcache/v1/multiprocess/protocols/checkpoint.py \
tests/v1/multiprocess/test_checkpoint_storage.pyLength of output: 100122 🏁 Script executed: #!/bin/bash
set -euo pipefail
printf '%s\n' '--- Transfer worker implementation ---'
nl -ba lmcache/v1/multiprocess/checkpoint_transfer.py | sed -n '100,360p'
printf '%s\n' '--- Checkpoint page grouping and store admission ---'
nl -ba lmcache/v1/multiprocess/checkpoint_storage.py | sed -n '40,390p'
printf '%s\n' '--- Storage reservation API and error semantics ---'
rg -n -C 12 --glob '*.py' \
'def reserve_write_detailed\b|reserve_write_detailed\(|class L1Error\b|KEY_NOT_WRITABLE|KEY_NOT_EXIST|OUT_OF_MEMORY' \
lmcache tests
printf '%s\n' '--- Busy-status and store-timeout tests or handling ---'
rg -n -C 8 --glob '*.py' \
'status == "busy"|status in \("miss", "busy"\)|AdmissionFailure\.BUSY|rpc_timeout|_DrainedCheckpointLeaseTimeout' \
lmcache testsLength of output: 238346
Busy retry can exceed its deadline. In This does not meet the stated contract that deadline expiry returns a negative rank result. Pass the remaining busy-retry budget into the request wait, or prevent a retry request when it cannot finish within that budget. Keep the existing late-response reconciliation path for requests that already reached the server. Add a test where a busy retry reply is delayed past the retry deadline. ✏️ Learnings added
You are interacting with an AI system. |
|
Community integration receipt for #65: this exact PR head is already merged into The original contributor commits are preserved. Exact head ancestry was verified, not inferred from equivalent file contents. Closing this dev-targeted review because its community integration is complete; this is not a claim that it merged into |
2 participants
Summary
Fix concurrent request-boundary checkpoint publication when schema-2
generations share immutable content-addressed pages.
busyadmission without aborting thewaiting generation.
exponential backoff, leaving LMCache RPC handler threads free to process the
owner completion.
after the owner aborts.
cancellation ownership and filesystem deduplication.
Target branch:
dev. Merge #62 before this PR; the retained commit ancestry carries its atomic checkpoint transport. This fixes concurrent publication for content-addressed payloads represented in #62, including #64. Merging this PR must publish todev, not only to the #62 review branch.Fixes local-inference-lab/vllm#725
Root cause
The R30 transfer worker permits two checkpoint copy jobs per rank. Instruction,
prompt and response generations from one request can consequently overlap.
Their unchanged full attention pages have identical schema-2 object keys.
While the first D2H copy owns a write lease,
reserve_write_detailed(..., "new")reportsKEY_NOT_WRITABLEto a later generation. Because the page isnot readable until commit, the later store was treated as a terminal conflict.
Rank timing differed, and one negative rank acknowledgement caused the atomic
connector to abort the entire generation.
Implementation
CheckpointPayloadStore.prepare_store()now reportsAdmissionFailure.BUSYonly when every unresolved key is explicitlyKEY_NOT_WRITABLE. Missing result entries, capacity failures and invalid oraborted generations retain their prior terminal/backpressure behavior.
The checkpoint RPC represents this state as
CheckpointLeaseResponse("busy").No SHM lease or byte slot is exposed.
CheckpointTransferWorkerretries thesame immutable manifest/rank off the model and RPC threads, starting at 1 ms
and capping at 10 ms under the existing RPC deadline. Shutdown or deadline
returns a negative rank result so the scheduler performs its existing atomic
abort.
Validation
R30 and pass with this patch.
while the first copy is held, then both generations publish.
environment.
Rust hooks are explicitly skipped for this Python-only change on macOS.
Complete TP4/DCP4 GPU, cross-replica, restart, filesystem-write and performance
qualification will be added from the patched R30 image.
Signed-off-by: Derek Yates derek.yates@live.com