Conversation
Recycle uncached scratch before cached blocks and add opt-in sparse checkpoint admission while preserving both replay boundaries. Assisted-by: OpenAI Codex Signed-off-by: Leonccaa <166551845+Leonccaa@users.noreply.github.com>
Assisted-by: OpenAI Codex Signed-off-by: Leonccaa <166551845+Leonccaa@users.noreply.github.com>
Express the opt-in interval in manager blocks so model-specific block sizes do not require token-unit conversion. Cover portable intervals and dense admission across four block sizes. Co-authored-by: OpenAI Codex <noreply@openai.com> Signed-off-by: Leonccaa <166551845+Leonccaa@users.noreply.github.com>
Adapt vLLM #45845, #47782, #53945 and #54713 to the 1Cat aligned-state layout. Replace custom interval environment variables with upstream token-based CacheConfig semantics. Co-authored-by: OpenAI Codex <noreply@openai.com> Signed-off-by: Leonccaa <166551845+Leonccaa@users.noreply.github.com>
Leonccaa
changed the title
Leonccaa
marked this pull request as ready for review
September 13, 2026 15:44
Codex Review SummaryThis comment shows the latest Codex review activity on this pull request.
ℹ️ About Codex in GitHubYour team has set up Codex to review pull requests in this repo. Reviews are triggered when you
Codex reacts with 👀 while any review is running, comments if it has suggestions, and reacts with 👍 once all reviews finish with no findings. |
![chatgpt-codex-connector[bot]](https://avatars.githubusercontent.com/in/1144995?s=60&v=4){kind=small}
There was a problem hiding this comment.
💡 Codex Review
Here are some automated review suggestions for this pull request.
Reviewed commit: 820516bf5a
ℹ️ About Codex in GitHub
Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you
- Open a pull request for review
- Mark a draft as ready
- Comment "@codex review".
If Codex has suggestions, it will comment; otherwise it will react with 👍.
Codex can also answer questions or update the PR. Try commenting "@codex address that feedback".
Preserve logical event ranges and admit completed decode boundaries before the cache horizon advances. Add regression coverage and repair baseline CI typing and conflicting header checks. Co-authored-by: OpenAI Codex <noreply@openai.com> Signed-off-by: Leonccaa <166551845+Leonccaa@users.noreply.github.com>
Co-authored-by: OpenAI Codex <noreply@openai.com> Signed-off-by: Leonccaa <166551845+Leonccaa@users.noreply.github.com>
Co-authored-by: OpenAI Codex <noreply@openai.com> Signed-off-by: Leonccaa <166551845+Leonccaa@users.noreply.github.com>
Leonccaa
added a commit
to Leonccaa/1Cat-vLLM
that referenced
this pull request
Sep 13, 2026
…al junctions Converge this PR on the offload side of sparse Mamba retention; the core policy (prefix_cache_retention_interval, replay boundaries, shared-prefix junctions, reachable_block_mask) comes from PR 1CatAI#617, on which this branch is now stacked. - Grouped CPU pools carry per-group slot counts. Mamba state slots are derived from MambaManager.reachable_block_mask for a reference request length (mamba_state_slots_reference_tokens, default max_model_len) plus one junction allowance per request, so RAM holds the states the GPU mask admits instead of one state slot per token slot. On the 16 GiB TP4 Flash-Next layout the default policy gives 390 token slots and 10 state slots per group (4.6 contexts of 64K) versus 107/107 dense. - The offloading connector records a shared-prefix junction on the request when the host tier holds a longer full-attention prefix than a sparse group can serve, so the state is materialized, kept and offloaded and the next sibling hits after a restart or GPU eviction. - Shared regions, tier managers and worker tensors honor per-group slot counts; the design doc describes the sizing and the validation history. Offload suites plus 1CatAI#617's retention tests: 300 passed, 2 skipped. Changed-file pre-commit including mypy passed. Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com> Signed-off-by: Leonccaa <166551845+Leonccaa@users.noreply.github.com>
Contributor
Author
|
@codex review |
|
Codex Review: Didn't find any major issues. Chef's kiss. Reviewed commit: ℹ️ About Codex in GitHubYour team has set up Codex to review pull requests in this repo. Reviews are triggered when you
If Codex has suggestions, it will comment; otherwise it will react with 👍. Codex can also answer questions or update the PR. Try commenting "@codex address that feedback". |
Leonccaa
marked this pull request as draft
September 14, 2026 03:23
Leonccaa
marked this pull request as ready for review
September 14, 2026 04:37
|
You have reached your Codex usage limits for code reviews. You can see your limits in the Codex usage dashboard. |
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
CI dependency and review fixes
Baseline CI fixes have been extracted into #624. This PR's final diff contains only cache-retention code, tests and documentation. Merge #624 first, then sync the baseline; until then, the known baseline typing/header/README failures can still affect this PR's CI. The green run at
fdc0ffc896predates this split and is not the CI status of the current head.The sparse
BlockStoredevent-range fix from PR15 review and non-EAGLE completed-decode-boundary fix from PR617 review remain here unchanged. All 38 new regression cases pass; the preceding source failed 34 of them. The cache suites passed 205 tests; the prior combined run including the separate QSA scale-gate suite passed 210.Splitting the three CI files leaves all cache code and test contents byte-identical to the tested
fdc0ffc896revision. No GPU or production changes. The 37-request GPU acceptance below remains attributed to820516bf5a/5c49dbfb13; the subsequent review fixes have CPU regression coverage, not a new GPU acceptance claim.Dense Mamba state snapshots can consume the shared block pool and evict the Attention prefix of an older long conversation. A subsequent follow-up then reports zero usable prefix and repeats the full prefill. This PR adapts upstream Mamba retention to 1Cat's aligned state layout and preserves uncached blocks for reuse before evicting cached prefixes.
Upstream provenance and configuration
This is a semantic backport of established upstream work, replacing the earlier custom environment-variable design:
Configuration follows upstream main at
fa008bdccf10f2f31f84553112a227cf1b8947d7:CacheConfig.prefix_cache_retention_interval, exposed as--prefix-cache-retention-interval.None0(default)For CT252's 800-token alignment, 4000/8000/16000/32000 correspond to 5/10/20/40 blocks. The former custom environment variable is removed: its
0meant dense retention and corresponds to the newNone, not the new0.1Cat adaptation and scope
Validation
CPU metadata regressions cover dense-vs-sparse long A/B retention, identical resend and longer sibling boundaries, detected junction reuse, main/draft mixed EAGLE groups, reference counts and scratch reuse, and token intervals across 16/512/800/1024-token block sizes. The 105 retention tests pass. Existing prefix-cache tests pass after updating exact queue-order assertions for the intended uncached-first policy. Applicable pre-commit checks pass.
The broader five-file CPU run reports 237 passed / 16 failed. All 16 failing node IDs also fail on the unmodified CT252 baseline (132 passed / 16 failed), from unavailable device inference and existing simplified config fixtures; there are no new failing nodes. These CPU tests do not establish CUDA/state-tensor correctness or production acceptance.
Historical GPU evidence is for the previous custom patch, not this final upstream-aligned implementation (TP4 V100, Flash-Next AWQ + native FP8 MTP3, 800-token state alignment):
0The custom 5-block arm also exposed a first-fork miss despite a surviving Attention prefix, motivating the junction port. The subsequent 10-block arm was stopped during initialization; 20/40 were not run. No results are claimed for those arms.
The final port has now passed the bounded Flash-Next/V100/MTP3 GPU acceptance below. Production has not been patched.
Final-port GPU acceptance — 2026-09-13
PASS on exact head
820516bf5a(identical runtime source in release-port5c49dbfb13). All nine changed source files were SHA-256 verified over the original b8aa release image. No diagnostic or #616 overlay. Defaultprefix_cache_retention_interval=0; TP4 V10032GB, Flash-Next AWQ + native FP8 MTP3, FP16 KV, C2, max context262144, batch8192, GMU0.984, KV capacity493491tokens.37 requests completed; zero preemptions; no CUDA/OOM/engine errors.
Seven serial prompt/output token arrays match the historical accepted arm. Three boundary cases and the fork sibling also match independent, unique-cache-salt cold controls token-for-token. Both long follow-ups and post-idle outputs match their cold references. The first unknown fork requires replay, then the materialized junction is reusable.
The clean warmed mixed run records343200 combined A+B prefix hits and0hits for the new third request. Over a16.776s pure dual stream window, A/B deliver67.716/39.700tok/s (combined107.416); A after the third delivers77.392tok/s alone. Maximum token gaps A0.943s/B0.061s. Third TTFT21.332s includes C2 queueing and is not a pure prefill measurement. A prior mixed arm following several independent fork/control prefixes had B cold and is not used as the warmed comparison. Retention does not eliminate finite-pool capacity eviction.
Reproduction: serial A170123 repeat, B175321, A, C14017, A/B; fork at119123tokens plus instruction tail, identical and sibling retries;15999/16000/16001 boundaries with fresh-salt controls; A4096-output/B768-output concurrent streams with a third cold14023-token request under C2, followed by A/B continuations; refresh A/B, wait600seconds without inference, repeat. Sampling temperature0; long stream stress ignores EOS. All assertions pass for cached-token deltas, zero preemptions, generated-token counts and the output comparisons above. Acceptance scope is this Flash-Next/V100/MTP3 configuration, not universal model/GPU coverage.