Conversation
Codex Review SummaryThis comment shows the latest Codex review activity on this pull request.
ℹ️ About Codex in GitHubYour team has set up Codex to review pull requests in this repo. Reviews are triggered when you
Codex reacts with 👀 while any review is running, comments if it has suggestions, and reacts with 👍 once all reviews finish with no findings. |
![chatgpt-codex-connector[bot]](https://avatars.githubusercontent.com/in/1144995?s=60&v=4){kind=small}
There was a problem hiding this comment.
💡 Codex Review
Here are some automated review suggestions for this pull request.
Reviewed commit: 453435dee8
ℹ️ About Codex in GitHub
Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you
- Open a pull request for review
- Mark a draft as ready
- Comment "@codex review".
If Codex has suggestions, it will comment; otherwise it will react with 👍.
Codex can also answer questions or update the PR. Try commenting "@codex address that feedback".
|
@codex review |
|
Codex Review: Didn't find any major issues. 🚀 Reviewed commit: ℹ️ About Codex in GitHubYour team has set up Codex to review pull requests in this repo. Reviews are triggered when you
If Codex has suggestions, it will comment; otherwise it will react with 👍. Codex can also answer questions or update the PR. Try commenting "@codex address that feedback". |
Recycle uncached scratch before cached blocks and add opt-in sparse checkpoint admission while preserving both replay boundaries. Assisted-by: OpenAI Codex Signed-off-by: Leonccaa <166551845+Leonccaa@users.noreply.github.com>
Assisted-by: OpenAI Codex Signed-off-by: Leonccaa <166551845+Leonccaa@users.noreply.github.com>
Express the opt-in interval in manager blocks so model-specific block sizes do not require token-unit conversion. Cover portable intervals and dense admission across four block sizes. Co-authored-by: OpenAI Codex <noreply@openai.com> Signed-off-by: Leonccaa <166551845+Leonccaa@users.noreply.github.com>
Adapt vLLM #45845, #47782, #53945 and #54713 to the 1Cat aligned-state layout. Replace custom interval environment variables with upstream token-based CacheConfig semantics. Co-authored-by: OpenAI Codex <noreply@openai.com> Signed-off-by: Leonccaa <166551845+Leonccaa@users.noreply.github.com>
Leonccaa
changed the title
There was a problem hiding this comment.
💡 Codex Review
Here are some automated review suggestions for this pull request.
Reviewed commit: 3ebe71d389
ℹ️ About Codex in GitHub
Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you
- Open a pull request for review
- Mark a draft as ready
- Comment "@codex review".
If Codex has suggestions, it will comment; otherwise it will react with 👍.
Codex can also answer questions or update the PR. Try commenting "@codex address that feedback".
Preserve logical event ranges and admit completed decode boundaries before the cache horizon advances. Add regression coverage and repair baseline CI typing and conflicting header checks. Co-authored-by: OpenAI Codex <noreply@openai.com> Signed-off-by: Leonccaa <166551845+Leonccaa@users.noreply.github.com>
Co-authored-by: OpenAI Codex <noreply@openai.com> Signed-off-by: Leonccaa <166551845+Leonccaa@users.noreply.github.com>
Co-authored-by: OpenAI Codex <noreply@openai.com> Signed-off-by: Leonccaa <166551845+Leonccaa@users.noreply.github.com>
Adapt the prefix-cacheable grouping invariant from vllm-project/vllm#54743 to the 1Cat offload layout. Preserve original worker group IDs and exact Mamba boundary stores. Co-authored-by: Codex <noreply@openai.com> Signed-off-by: Leonccaa <166551845+Leonccaa@users.noreply.github.com>
Keep group-local block IDs in disjoint CPU tensor pools. Derive matching scheduler and worker budgets from physical tensor sharing instead of simplified per-layer specs. This avoids evicting attention prefixes while other group slots waste the same RAM budget. Co-authored-by: Codex <noreply@openai.com> Signed-off-by: Leonccaa <166551845+Leonccaa@users.noreply.github.com>
Co-authored-by: Codex <noreply@openai.com> Signed-off-by: Leonccaa <166551845+Leonccaa@users.noreply.github.com>
Forward child lifecycle hooks and exercise fresh-process restoration, eviction, and truncated-file handling through existing tier managers. Document the remaining GPU serving and persistent namespace gates. Co-authored-by: Codex <noreply@openai.com> Signed-off-by: Leonccaa <166551845+Leonccaa@users.noreply.github.com>
Reuse existing strided mmap views and transfer handlers. Verify scheduler and worker slice visibility plus cleanup after initialization failures. Tiering spec serving integration remains gated. Co-authored-by: Codex <noreply@openai.com> Signed-off-by: Leonccaa <166551845+Leonccaa@users.noreply.github.com>
Compose existing primary and secondary tier managers per group with failure cleanup. Namespace filesystem cache by declared physical layout and validate spec routing, restart recovery and unsupported topology. GPU serving acceptance remains pending. Co-authored-by: Codex <noreply@openai.com> Signed-off-by: Leonccaa <166551845+Leonccaa@users.noreply.github.com>
Workers can create shared mappings before the scheduler and exit without unlinking. Assign scheduler final unlink responsibility and keep repeated cleanup harmless. Validated with shared-region and grouped-tiering CPU tests; real shutdown acceptance pending. Co-authored-by: Codex <noreply@openai.com> Signed-off-by: Leonccaa <166551845+Leonccaa@users.noreply.github.com>
Co-authored-by: Codex <noreply@openai.com> Signed-off-by: Leonccaa <166551845+Leonccaa@users.noreply.github.com>
Co-authored-by: Codex <noreply@openai.com> Signed-off-by: Leonccaa <166551845+Leonccaa@users.noreply.github.com>
Batch KV transfers require registered host memory. Do not continue after cudaHostRegister failure and defer the error to a later kernel or transfer. Reuse the existing construction cleanup path. Co-authored-by: Codex <noreply@openai.com> Signed-off-by: Leonccaa <166551845+Leonccaa@users.noreply.github.com>
Document RAM eviction, filesystem restart and pressure under MTP0/MTP3, ordinary single-group output comparisons, and the observed host-registration failure boundary. Co-authored-by: Codex <noreply@openai.com> Signed-off-by: Leonccaa <166551845+Leonccaa@users.noreply.github.com>
Roll back newly reserved slots through complete_store(success=False) before returning None, preserving scheduler retry cursors. Cover asymmetric pressure, reservation reuse, and existing in-flight writes. Co-authored-by: Codex <noreply@openai.com> Signed-off-by: Leonccaa <166551845+Leonccaa@users.noreply.github.com>
Exercise real LMCache conversion with fork cache specs and document the known QSA scratch exclusion gap using a strict expected failure. This does not claim serving or GPU acceptance. Co-authored-by: Codex <noreply@openai.com> Signed-off-by: Leonccaa <166551845+Leonccaa@users.noreply.github.com>
Co-authored-by: Codex <noreply@openai.com> Signed-off-by: Leonccaa <166551845+Leonccaa@users.noreply.github.com>
…al junctions Converge this PR on the offload side of sparse Mamba retention; the core policy (prefix_cache_retention_interval, replay boundaries, shared-prefix junctions, reachable_block_mask) comes from PR 1CatAI#617, on which this branch is now stacked. - Grouped CPU pools carry per-group slot counts. Mamba state slots are derived from MambaManager.reachable_block_mask for a reference request length (mamba_state_slots_reference_tokens, default max_model_len) plus one junction allowance per request, so RAM holds the states the GPU mask admits instead of one state slot per token slot. On the 16 GiB TP4 Flash-Next layout the default policy gives 390 token slots and 10 state slots per group (4.6 contexts of 64K) versus 107/107 dense. - The offloading connector records a shared-prefix junction on the request when the host tier holds a longer full-attention prefix than a sparse group can serve, so the state is materialized, kept and offloaded and the next sibling hits after a restart or GPU eviction. - Shared regions, tier managers and worker tensors honor per-group slot counts; the design doc describes the sizing and the validation history. Offload suites plus 1CatAI#617's retention tests: 300 passed, 2 skipped. Changed-file pre-commit including mypy passed. Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com> Signed-off-by: Leonccaa <166551845+Leonccaa@users.noreply.github.com>
17a3801 to
eef147c
Compare
Leonccaa
changed the title
Co-authored-by: OpenAI Codex <noreply@openai.com> Signed-off-by: Leonccaa <166551845+Leonccaa@users.noreply.github.com>
Signed-off-by: Leonccaa <166551845+Leonccaa@users.noreply.github.com>
Signed-off-by: Leonccaa <166551845+Leonccaa@users.noreply.github.com>
|
@codex review |
There was a problem hiding this comment.
💡 Codex Review
Here are some automated review suggestions for this pull request.
Reviewed commit: fb808b774f
ℹ️ About Codex in GitHub
Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you
- Open a pull request for review
- Mark a draft as ready
- Comment "@codex review".
If Codex has suggestions, it will comment; otherwise it will react with 👍.
Codex can also answer questions or update the PR. Try commenting "@codex address that feedback".
| `ModelConfig` and that value should be manually duplicated here.""" | ||
| enable_prefix_caching: bool = True | ||
| """Whether to enable prefix caching.""" | ||
| prefix_cache_retention_interval: int | None = Field(default=0, ge=0) |
There was a problem hiding this comment.
When an aligned-Mamba model uses a valid kv_connector_extra_config["block_size"] larger than its GPU block size, the default retention value keeps only the replay/junction boundary on the GPU-block grid. _build_boundary_state_store_jobs accepts that state only when its token boundary is divisible by the larger offloaded block size, so for most prompt lengths no Mamba key is stored even though the corresponding full-attention blocks are. Repeating such a prompt then gets no external prefix hit; either retain offload-aligned checkpoints or reject sparse retention with a block-size factor greater than one.
Useful? React with 👍 / 👎.
There was a problem hiding this comment.
Fixed in 0b2d937 by rejecting the combination up front: SchedulerOffloadConfig.from_spec raises when an align Mamba group is present, prefix_cache_retention_interval is not None and kv_connector_extra_config["block_size"] is larger than the GPU block size (dense retention with a factor above one keeps working). Retaining offload-aligned checkpoints would require the core mask (MambaManager.reachable_block_mask, #617) to learn the offload alignment; the connector cannot materialize a state the GPU did not keep, so that is left to the core policy. Covered by test_sparse_retention_rejects_offloaded_block_size_factor (rejects factor 2 under retention 0 and 32; accepts factor 1, dense retention, and attention-only groups). Documented in docs/design/grouped_cpu_offload.md.
| self.cpu_group_num_blocks[i] = ( | ||
| state_slots(num_blocks) if i in mamba_groups else num_blocks |
There was a problem hiding this comment.
With semantic retention this assignment can shrink each Mamba pool to only a handful of slots. If those slots are temporarily protected by GPU-to-CPU or filesystem cascade transfers, prepare_store returns None; unlike the grouped normal-store rollback, _build_boundary_state_store_jobs has already drained the one-shot handoff and only logs the failure, while the full-attention store still proceeds. The missing checkpoint is therefore never retried and the resulting attention prefix remains unusable for later restores, so boundary handoffs need a durable retry path or reserved transient capacity.
Useful? React with 👍 / 👎.
There was a problem hiding this comment.
Fixed in 0b2d937 with a durable retry: hand-offs are queued on RequestOffloadState.pending_boundary_offloads and _build_boundary_state_store_jobs retries every pending offer each step until prepare_store accepts it. The source block stays owned by the request (hashed committed blocks are never relocated by the align allocator, and the connector fences it once a job exists), so a retry a step later is safe. A re-offer of the same boundary supersedes the older entry; offers still pending when the request finishes or is preempted are dropped with a warning because the block is about to be reused. Covered by test_mamba_boundary_store_retries_while_host_tier_is_busy, test_mamba_boundary_reoffer_supersedes_pending_entry and test_pending_boundary_offloads_dropped_when_blocks_are_freed[preempted|finished]. CPU suites: 309 passed / 2 skipped; a GPU re-test of this commit is still pending.
…ck-size factor Two review findings on the offload side of sparse Mamba retention: - The core offers each Mamba align boundary-state hand-off once. With semantic retention the state pools hold only a few slots per group, so prepare_store() can fail while those slots are pinned by in-flight loads, stores or a filesystem cascade; the offer was logged and lost while the attention blocks were stored, leaving a prefix that could never be restored. Pending offers now stay on the request and are retried every step (the source block remains owned by the request); a re-offer of the same boundary supersedes the older entry, and offers still pending when the request finishes or is preempted are dropped with a warning because the block is about to be reused. - Sparse retention keeps states on the GPU block grid while boundary stores need offloaded-block alignment, so kv_connector_extra_config block_size above the GPU block size would offload attention blocks without their Mamba state for most prompt lengths. SchedulerOffloadConfig now rejects that combination at start-up (dense retention still works); retaining offload-aligned checkpoints needs the core mask to learn the offload alignment and is left to the core policy. Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com> Signed-off-by: Leonccaa <166551845+Leonccaa@users.noreply.github.com>
|
You have reached your Codex usage limits for code reviews. You can see your limits in the Codex usage dashboard. |
1 participant
Grouped RAM offload and filesystem restoration
Equal-block attention/Mamba models can lose all useful external prefix hits when native CPU offload charges every group key for the full shared tensor set. This patch preserves original KV group IDs, excludes non-prefix-cacheable scratch groups from block validation, and budgets independent group CPU pools using their actual backing tensors. Allocation, LRU/ARC eviction, reference counts and transfers reuse the existing implementations. On the offload side it consumes the sparse Mamba retention policy from #617: grouped pools are sized from the retention mask and junctions discovered in the host tier are propagated; see the section below.
The same group router now composes existing primary and secondary tier managers through TieringOffloadingSpec. Workers bind to per-group SharedOffloadRegion views, including TP slices. Partial initialization releases created tiers and mappings. Scheduler cleanup unlinks mappings even when a worker created the file first.
FileMapper accepts optional physical-layout metadata to distinguish group-row files from legacy full-row files; legacy paths remain unchanged without that metadata. Grouped tiering requires single-node TP and an explicit attention backend. Restart reuse also requires a fixed PYTHONHASHSEED and matching prefix hash algorithm. No new serializer or filesystem I/O backend is introduced. Public KV connector and LMCache adapter contracts remain unchanged; LMCache runtime compatibility is not certified; native FS files are not an LMCache interchange format.
Sparse Mamba retention on the offload side (stacked on #617)
The core retention policy is not in this PR any more: an earlier revision carried its own port of upstream's
prefix_cache_retention_interval; that was dropped in favor of #617, and this branch is now stacked on #617's head (a9ab97a), so its diff includes #617 until #617 merges. Merge order: #624, #617, then this PR. What remains here is offload-only:MambaManager.reachable_block_maskfor a reference request length (mamba_state_slots_reference_tokens, defaultmax_model_len) plus one junction allowance per request, multiplied by the requests the token pool holds. Dense retention reproduces the previous equal layout; the default policy on the 16 GiB TP4 Flash-Next layout gives 390 token slots and 10 state slots per group (4.6 contexts of 64K) instead of 107/107 (1.3 contexts).alignboundary hand-offs only carry hashed states ([Bugfix][Core] Backport upstream Mamba prefix-cache retention #617), so the host tier stores exactly what the GPU mask admits.Request.shared_prefix_boundary(as the GPU prefix cache does), so the state is materialized, kept and offloaded and the next sibling hits after a restart or GPU eviction.alignboundary state once; with only a few state slots per groupprepare_storecan fail while the slots are pinned by in-flight transfers or a filesystem cascade. Pending offers now stay on the request and are retried every step (the source block remains owned by the request); a re-offer supersedes the older entry, and offers still pending at request finish or preemption are dropped with a warning.kv_connector_extra_config["block_size"]larger than the GPU block size would offload attention blocks without their Mamba state.SchedulerOffloadConfignow rejects that combination at start-up; dense retention (None) with a factor above one keeps working. Retaining offload-aligned checkpoints would need the core mask to learn the offload alignment (core policy, [Bugfix][Core] Backport upstream Mamba prefix-cache retention #617).Offload suites plus #617's retention tests: 300 passed, 2 skipped. Changed-file pre-commit including mypy passed.
At 0b2d937: offload/worker/retention suites 237 passed, 2 skipped (test_gpu_worker needs a GPU and was deselected) plus the connector scheduler suite 72 passed; changed-file pre-commit including mypy passed. The V100 regression below predates 0b2d937; a GPU re-test of the round-3 fixes is still pending.
Validation history: the earlier revision (own core port plus an offload-only stride) passed two V100 rounds on 2026-09-13 (64K restores at the replay boundary with identical token IDs, GPU-side and externally discovered junctions served to a third request from RAM, four distinct 60K-64K contexts restored, dense retention thrashing the 107-slot pools); evidence in
/mnt/llm_hfs/builds/qsa-stride-validation-20260912and/mnt/llm_hfs/builds/qsa-retention-validation-20260913. The converged source ateef147cafd, on #617 corea9ab97a755, has now completed joint V100 RAM regression; see below.Shared mmap registration fix and 32 GiB V100 follow-up
With four workers registering the same large mmap concurrently, serving startup reproduced
cudaHostRegistererror 1. Kernel tracing identifiedpin_user_pages()returning-ENOMEMafter long-term page migration failed: 0/71 succeeded/failed pages on one worker, then 40/31 on another; later workers migrated the remaining pages and registered successfully. No new cgroup OOM kill occurred in that reproduction. This establishes a concrete migration failure mechanism for the later reproduction; it does not retrospectively identify every earlier pinning failure.pin_mmap_regionnow takes an exclusiveflockon each region's independently opened descriptor around the original CUDA registration call and releases it infinally. Other workers wait for migration/registration of that backing file to complete. Inference and transfers are not serialized. Errors remain fatal; no retry or pageable fallback is added. Tests check real descriptor contention and release after successful and exceptional registration.Updated stacked-branch CPU regression: 302 passed, 2 skipped. Re-ran the CPU/interface command below with
tests/v1/core/test_mamba_sparse_retention.pyadded, using the same documented fixtures (CPU hybrid admission; buffered I/O only for legacy unaligned FS tests, real direct I/O for grouped mmap/spawn tests). Changed-file pre-commit passed, including mypy.GPU evidence comes from the integration of upstream
7217bb5d4f, #624/#617/#598 and this fix (3fcc73b208). Its offload runtime code matches this PR update, but this is integration GPU acceptance, not a new native rebuild of the standalone stacked PR head. The native extensions were fully rebuilt before the Python-only registration fix and their hashes reverified in the final image (559d0ac9c43efa16e404fb7ece2c2990b0e1b7d5bd029c67adf4a3719fd94618). Two diagnostic starts changing only registration order and two formal-image starts passed registration and CUDA Graph capture.Restore token IDs matched their cold controls; there were no preemptions or new cgroup OOM kills. The long-request timings are primarily prefill, not decode throughput. The accepted integration is now deployed; the earlier sections below remain dated validation history. Positive retention intervals, arbitrary models/backends and broad quality/multimodal stress remain outside this follow-up.
Two operational boundaries were also verified:
OOMKilled=falsedid not exclude that worker kill. A private IPC namespace with sufficient/dev/shmcapacity reclaimed mappings across exits. This is a deployment mitigation, not a claim that the code now crash-cleans host-IPC files; no site-specific deployment script is added here.persistent_layout.config_hashincludes the vLLM version. The formal image correctly used a different FS namespace from the diagnostic version; same-version restart reuse passed. Cross-version cold misses are not a failed persistence test and compatibility hashes are unchanged.The change stays in this existing offload PR; no duplicate PR is opened. Developed and validated with OpenAI Codex assistance. See the updated host-registration and shutdown sections in
docs/design/grouped_cpu_offload.md.Joint V100 regression on #617 (2026-09-13)
Every restore clears the GPU prefix first and has zero local hits. Output token IDs match cold controls; MTP3 draft/accept counters are nonzero. Pinned cache is approximately 3.99 GiB per worker (preallocated pool, not minimum RAM per context). Independent CPU rerun: 300 passed, 2 skipped. No engine/CUDA/OOM errors in either candidate; both stopped with exit code 0. Documentation-only follow-up leaves the tested runtime source unchanged.
An old MTP3 probe assertion expected the second restore to equal the first. Actual hits grew from 62,400 to 63,200: the first restore materialized the longer Attention boundary, and all four newly stored Mamba keys were confirmed as hits on the next request. Outputs remained identical. Original failure and evidence-backed recheck are retained. MTP3 junction cases use distinct long tails to prevent the earlier replay checkpoint from bypassing missing-junction discovery; an initial cold fixture misplaced its output instruction and was corrected before the successful restore checks.
Scope: verified
b8aa829785image with 18 hash-checked Python overlays from the converged source and two diagnostic wrappers. This is joint-source regression, not complete rebuilt-image acceptance. Sparse-retention FS restart, positive-interval GPU coverage, and a complete rebuild were untested in that earlier round; the newer integration results above cover FS restart and the rebuilt image, but not positive intervals. Evidence:/mnt/llm_hfs/builds/pr598-on617-validation-20260913. No production deployment was performed in that earlier round; the later accepted integration above is deployed.CI dependency remains #624 → #617 → #598. The latest pre-validation #598 CI failures are entirely in the three baseline files fixed by #624; do not reintroduce those unrelated changes here. #624's independent CI passed.
Complete-image V100 validation
Built clean native extensions and matching Python source at
0f0139b23d, CUDA 12.8, Torch 2.10.0+cu128, SM70. All 1,954 packaged Python/native files were verified before each server start. No runtime source overlay. Image SHA256:ab3aa7184314cb59a35cbb34db8ae7f1136db66493c5256cc9ef7cb7a586dcbf.Flash-Next AWQ, TP4, FP16 KV, CUDA Graphs, 4 GiB total CPU cache: MTP0 and MTP3 each passed six RAM requests with distinct 16K contexts. After resetting GPU prefix state, repeated A restored 15,680 tokens (MTP0) / 15,200 (MTP3). After B/C exceeded capacity, recent C still hit while old A missed. Cold/restored/recomputed output token IDs matched, including across MTP0/MTP3; MTP3 draft/accept counters were nonzero.
MTP0 filesystem restart: writer exited before a new reader started with empty RAM. The reader restored 15,680 tokens, zero local hits, and the writer's exact eight output token IDs. After B/C pressure and GPU prefix reset, A restored with the same count and IDs. Writer and reader exited zero, each removing all five mmap files.
Ordinary single KV group: Qwen3-0.6B FP16 on one V100, CUDA Graphs, 256 MiB RAM cache. Disabled-offload baseline for three distinct 1,597-token prompts; enabled-offload repeats restored 1,584 tokens. Recent C survived capacity pressure, old A missed, and all 24 output token IDs matched the disabled-offload baseline.
MTP3 filesystem restart and pressure passed on the follow-up complete image at
9c47de86a6(SHA256a80544fff6f11ab5fdc54b15db0bbb2c4b4f64d34231146fb5f8ead2f05f9ca3). Native sources/artifacts were unchanged; all matching Python/native files were reverified. Both A restores had 15,200 external tokens, zero local hits, exact eight output token IDs and six drafted/six accepted tokens. Writer and reader exited zero and each removed all five mmap files.Latest CPU regression including host-registration failure handling: 167 passed, 2 skipped with the same fixture/command below. Changed-file pre-commit hooks passed, including mypy.
One initial MTP3 FS startup failed
cudaHostRegisteron two ranks. The old warning-and-continue branch left a CUDA error for the next kernel, and a separate probe showed native batch KV transfer rejects unregistered memory. The follow-up now fails immediately with rank/path/size/error and reuses construction cleanup, matching the existing simple CPU offload pinning behavior. Four-GPU isolated probes passed 100 registrations; the cause of the one serving-time registration failure is not established. Subsequent writer/reader startup and all MTP3 persistence requests passed. This does not add a pageable transfer fallback or promise that host pinning cannot fail.GPU procedure: start with fixed
PYTHONHASHSEED=0, explicitFLASH_ATTN_V100,OffloadingConnector/kv_both,cpu_bytes_to_use=4294967296; selectTieringOffloadingSpecwith the existingfssecondary tier for restart cases. Run greedy 16K requests withreturn_token_ids, reset GPU prefix state between RAM probes, exceed the RAM budget with distinct prefixes, then compare external/local hit counters and token IDs. For FS, reuse only the file directory between normally stopped writer and fresh reader. Use--shutdown-timeout 60and a 90-second container stop grace. The design document records the configuration and acceptance boundaries.CPU and interface regression
python -m pytest --noconftest -p qsa_cpu_pytest -p qsa_fs_cpu_pytest -p tests.v1.kv_connector.unit.offloading_connector.conftest -q tests/v1/kv_connector/unit/offloading_connector tests/v1/kv_offload/cpu/test_manager.py tests/v1/kv_offload/cpu/test_grouped_manager.py tests/v1/kv_offload/cpu/test_shared_offload_region.py tests/v1/kv_offload/cpu/test_grouped_tiering.py tests/v1/kv_offload/test_tiering_offloading.py tests/v1/kv_offload/test_file_mapper.py tests/v1/kv_offload/test_fs_tier.py. Local fixtures bypass GPU admission and disable O_DIRECT only for legacy FS tests with unaligned Torch buffers. New grouped mmap/spawn tests retain real O_DIRECT.python -m pytest --noconftest -p qsa_cpu_pytest -q tests/v1/kv_offload/cpu/test_shared_offload_region.py tests/v1/kv_offload/cpu/test_grouped_tiering.py. Pre-commit passed for each commit.0f0139b23d. Public connector and LMCache adapter files are unchanged. Separate unmodified LMCache SM70 build/transfer probes passed, but full Flash-Next + LMCache serving is not certified by this PR.Scope and review
This targets
1CatAI/1Cat-vLLM. Open 1Cat PR searches found no equivalent grouped native KV offload change. Official vLLM #54743 overlaps the scratch-group filtering concern; this change additionally budgets physical tensors per group and composes those pools with native filesystem tiers for 1Cat's hybrid SM70 path. It does not propose a second PR to official vLLM.Filesystem tests used NFS, so this does not claim SSD throughput, power-loss durability, disk quota/garbage collection, multi-node grouped tiering, arbitrary backend/layout compatibility, or native-FS/LMCache file interchange. RAM remains bounded by the existing LRU/ARC managers. Fixed prefix hash seed and unchanged model/layout are required for restart reuse.
Developed and tested with OpenAI Codex assistance. See
docs/design/grouped_cpu_offload.mdfor layout, lifecycle and compatibility details.Additional external LMCache metadata test
Commit 3e290f6 adds optional
test_lmcache_group_metadata.py, using the real LMCache converter/native extension and this fork's cache spec classes with small synthetic CPU tensors. On LMCache b5d109ea99a89b4d8a670ee4fc2e8cb76411ee5c: 4 passed, 1 known failure (strict xfail). Dense/hybrid engine IDs, layer mapping, DCP token spans, recurrent windows, and block-ID routing pass. QSA scratch exclusion fails: attention/scratch/Mamba returns[0, 1, 2]instead of[0, 2]; the third-party converter does not honorprefix_cacheable=False. The converter file is unchanged at LMCache dev fcb67c0ab1db2a4bad78e085b3a2df33da003b7c. This PR does not adapt the separate LMCache serving path.Reproduce with compiled LMCache installed:
.venv/bin/python -m pytest --noconftest -q -rx tests/v1/kv_connector/unit/test_lmcache_group_metadata.py. Existing local LMCache interface/event regression: 31 passed; the five new optional cases skip when LMCache is absent. Changed-file pre-commit passed. No GPU used for this metadata test, and no QSA LMCache inference/eviction/restart acceptance is claimed.Unmerged LMCache PR evaluation
Documentation commit
028e8bceaarecords independent tests of LMCache #5042 (322fecf8) and #5059 (616484d1). #5042 fixes our five metadata cases and passes 81 V100 native transfer/shape tests plus two FS tests, but real Flash-Next TP4/MTP0 startup fails during registration on its new MLA view rule for(124, 196, 1, 128). A CPU NHD/HND reproducer fails on the PR and passes with the PR-base group-edits module. Relevant CPU suite: 119 passed, 2 GLM-specific fork-API failures. Full model STORE/RETRIEVE, eviction, restart and MTP remain unaccepted. Detailed result and reproducer posted to #5042.#5059 passes 24 upstream plus 6 real-fork CPU validation tests and intentionally rejects scratch; it is an alternative rejection policy, not a compatible support patch to stack on #5042 unchanged. Result posted to #5059. Neither third-party PR was merged into this branch; production code is unchanged by this validation.