Conversation
Recycle uncached scratch before cached blocks and add opt-in sparse checkpoint admission while preserving both replay boundaries. Assisted-by: OpenAI Codex Signed-off-by: Leonccaa <166551845+Leonccaa@users.noreply.github.com>
Express the opt-in interval in manager blocks so model-specific block sizes do not require token-unit conversion. Cover portable intervals and dense admission across four block sizes. Co-authored-by: OpenAI Codex <noreply@openai.com> Signed-off-by: Leonccaa <166551845+Leonccaa@users.noreply.github.com>
Adapt vLLM #45845, #47782, #53945 and #54713 to the 1Cat aligned-state layout. Replace custom interval environment variables with upstream token-based CacheConfig semantics. Co-authored-by: OpenAI Codex <noreply@openai.com> Signed-off-by: Leonccaa <166551845+Leonccaa@users.noreply.github.com>
Leonccaa
changed the title
Codex Review SummaryThis comment shows the latest Codex review activity on this pull request.
ℹ️ About Codex in GitHubYour team has set up Codex to review pull requests in this repo. Reviews are triggered when you
Codex reacts with 👀 while any review is running, comments if it has suggestions, and reacts with 👍 once all reviews finish with no findings. |
![chatgpt-codex-connector[bot]](https://avatars.githubusercontent.com/in/1144995?s=60&v=4){kind=small}
There was a problem hiding this comment.
💡 Codex Review
Here are some automated review suggestions for this pull request.
Reviewed commit: 5c49dbfb13
ℹ️ About Codex in GitHub
Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you
- Open a pull request for review
- Mark a draft as ready
- Comment "@codex review".
If Codex has suggestions, it will comment; otherwise it will react with 👍.
Codex can also answer questions or update the PR. Try commenting "@codex address that feedback".
| num_full_blocks=num_full_blocks, | ||
| block_size=self.block_size, | ||
| kv_cache_group_id=self.kv_cache_group_id, | ||
| block_mask=mask, |
There was a problem hiding this comment.
When KV-cache events are enabled, passing the sparse retention mask here causes cache_full_blocks to omit masked blocks from BlockStored.block_hashes while still emitting the entire newly processed token range in token_ids; for the default interval, most calls therefore emit an event with zero hashes and one block of tokens. This violates the event invariant len(token_ids) == block_size * len(block_hashes) used by KV-event consumers, so Mamba deployments using cache events can publish malformed store events or associate tokens with the wrong hashes.
Useful? React with 👍 / 👎.
Preserve logical event ranges and admit completed decode boundaries before the cache horizon advances. Add regression coverage and repair baseline CI typing and conflicting header checks. Co-authored-by: OpenAI Codex <noreply@openai.com> Signed-off-by: Leonccaa <166551845+Leonccaa@users.noreply.github.com>
1 participant
Review follow-up (2026-09-13;
9c62d99a07)BlockStoredper contiguous retained logical range, with matching tokens, hashes, extra keys and logical parent; suppress empty events. Tests cover masks, null blocks, grouped hashes, LoRA, multimodal metadata and nonzero cache horizons.GPU evidence scope: the 37-request GPU acceptance below belongs to the preceding heads
820516bf5a/5c49dbfb13. This follow-up is CPU-validated, with no new GPU acceptance claim. The tested MTP3 path used events disabled; its replay-boundary branch is unchanged. Newly fixed non-EAGLE and event-enabled paths have CPU regression coverage. No production or GPU-runtime changes were made.Dense Mamba state snapshots can consume the shared block pool and evict the Attention prefix of an older long conversation. A subsequent follow-up then reports zero usable prefix and repeats the full prefill. This PR adapts upstream Mamba retention to 1Cat's aligned state layout and preserves uncached blocks for reuse before evicting cached prefixes.
Upstream provenance and configuration
This is a semantic backport of established upstream work, replacing the earlier custom environment-variable design:
Configuration follows upstream main at
fa008bdccf10f2f31f84553112a227cf1b8947d7:CacheConfig.prefix_cache_retention_interval, exposed as--prefix-cache-retention-interval.None0(default)For CT252's 800-token alignment, 4000/8000/16000/32000 correspond to 5/10/20/40 blocks. The former custom environment variable is removed: its
0meant dense retention and corresponds to the newNone, not the new0.1Cat adaptation and scope
Validation
CPU metadata regressions cover dense-vs-sparse long A/B retention, identical resend and longer sibling boundaries, detected junction reuse, main/draft mixed EAGLE groups, reference counts and scratch reuse, and token intervals across 16/512/800/1024-token block sizes. The 105 retention tests pass. Existing prefix-cache tests pass after updating exact queue-order assertions for the intended uncached-first policy. Applicable pre-commit checks pass.
The broader five-file CPU run reports 237 passed / 16 failed. All 16 failing node IDs also fail on the unmodified CT252 baseline (132 passed / 16 failed), from unavailable device inference and existing simplified config fixtures; there are no new failing nodes. These CPU tests do not establish CUDA/state-tensor correctness or production acceptance.
Historical GPU evidence is for the previous custom patch, not this final upstream-aligned implementation (TP4 V100, Flash-Next AWQ + native FP8 MTP3, 800-token state alignment):
0The custom 5-block arm also exposed a first-fork miss despite a surviving Attention prefix, motivating the junction port. The subsequent 10-block arm was stopped during initialization; 20/40 were not run. No results are claimed for those arms.
The final port has now passed the bounded Flash-Next/V100/MTP3 GPU acceptance below. Production has not been patched.
Final-port GPU acceptance — 2026-09-13
PASS on exact head
820516bf5a(identical runtime source in release-port5c49dbfb13). All nine changed source files were SHA-256 verified over the original b8aa release image. No diagnostic or 1CatAI#616 overlay. Defaultprefix_cache_retention_interval=0; TP4 V10032GB, Flash-Next AWQ + native FP8 MTP3, FP16 KV, C2, max context262144, batch8192, GMU0.984, KV capacity493491tokens.37 requests completed; zero preemptions; no CUDA/OOM/engine errors.
Seven serial prompt/output token arrays match the historical accepted arm. Three boundary cases and the fork sibling also match independent, unique-cache-salt cold controls token-for-token. Both long follow-ups and post-idle outputs match their cold references. The first unknown fork requires replay, then the materialized junction is reusable.
The clean warmed mixed run records343200 combined A+B prefix hits and0hits for the new third request. Over a16.776s pure dual stream window, A/B deliver67.716/39.700tok/s (combined107.416); A after the third delivers77.392tok/s alone. Maximum token gaps A0.943s/B0.061s. Third TTFT21.332s includes C2 queueing and is not a pure prefill measurement. A prior mixed arm following several independent fork/control prefixes had B cold and is not used as the warmed comparison. Retention does not eliminate finite-pool capacity eviction.
Reproduction: serial A170123 repeat, B175321, A, C14017, A/B; fork at119123tokens plus instruction tail, identical and sibling retries;15999/16000/16001 boundaries with fresh-salt controls; A4096-output/B768-output concurrent streams with a third cold14023-token request under C2, followed by A/B continuations; refresh A/B, wait600seconds without inference, repeat. Sampling temperature0; long stream stress ignores EOS. All assertions pass for cached-token deltas, zero preemptions, generated-token counts and the output comparisons above. Acceptance scope is this Flash-Next/V100/MTP3 configuration, not universal model/GPU coverage.