Skip to content

fix(sse): advance Claude cache breakpoints on growing tails - #10684

Merged
diegosouzapw merged 77 commits into
diegosouzapw:release/v3.8.50from
cryptiklemur:fix/claude-growing-tail-cache-breakpoints
Aug 20, 2026
Merged

diegosouzapw merged 77 commits into
diegosouzapw:release/v3.8.50from
cryptiklemur:fix/claude-growing-tail-cache-breakpoints

Conversation

@cryptiklemur

@cryptiklemur cryptiklemur commented Aug 18, 2026 •

Copy link
Copy Markdown
Contributor

CI note: Docs Gates currently inherits the release-base BOT_TOKEN / BOT_URL env-doc failure. the same base-red appears on unrelated #10780; this diff does not touch those variables.

Claude Code requests stopped advancing the cached prefix after an older message breakpoint appeared. later turns stayed outside the reusable prefix even as the conversation grew.

Anthropic requires longer TTL breakpoints before shorter ones and checks the prompt in tools, system, then messages order: Prompt caching.

what changed

  • count surviving system and message breakpoints together
  • add a breakpoint to the latest user message while fewer than four remain
  • keep repeated normalization idempotent
  • default later omitted TTLs to 5m after a 5m boundary, preserving 1h before 5m

The multi-turn tests cover the final user message, the four-breakpoint cap, repeated calls, both hoist paths, and native Claude passthrough.

how to test

npx cross-env DISABLE_SQLITE_AUTO_BACKUP=true node --max-old-space-size=4096 --import tsx/esm --import ./open-sse/utils/setupPolyfill.ts --import ./tests/_setup/isolateDataDir.ts --test --test-force-exit --test-concurrency=1 tests/unit/claude-code-parity.test.ts
npx cross-env DISABLE_SQLITE_AUTO_BACKUP=true node --max-old-space-size=4096 --import tsx/esm --import ./open-sse/utils/setupPolyfill.ts --import ./tests/_setup/isolateDataDir.ts --test --test-force-exit --test-isolation=none --test-concurrency=1 --test-name-pattern='chatCore (builds Claude Code-compatible|normalizes native Claude Code messages|preserves Opus 5 mid-conversation|keeps Claude normalization|preserves cache_control automatically|advances a message cache breakpoint)' tests/unit/chatcore-translation-paths.test.ts

Both focused runs pass after rebasing onto the current release/v3.8.50 tip.

@diegosouzapw

Copy link
Copy Markdown
Owner

Thanks for this fix — advancing the prompt-cache breakpoint onto the growing conversation tail closes a real gap (the current ensureCacheControlOnLastUserMessage keeps an existing marker pinned at the first turn, so the cached prefix stops growing on long sessions). A couple of things to confirm before merge: (1) rebase against the current release/v3.8.50 tip — the PR is currently marked as conflicting; (2) please add/keep a unit test covering a multi-turn payload asserting the marker is advanced to the final user message without exceeding the 4-breakpoint cap, and that the existing #9436 boundary tests (both hoist paths + native Claude path) still pass; (3) ensure no double-advance leaves duplicate markers and that TTL ordering (1h-before-5m) is preserved. Nice, tightly-scoped fix.

@cryptiklemur
cryptiklemur force-pushed the fix/claude-growing-tail-cache-breakpoints branch from 8c2d900 to 71cf84c Compare August 19, 2026 07:38
@cryptiklemur
cryptiklemur force-pushed the fix/claude-growing-tail-cache-breakpoints branch from 71cf84c to 8262395 Compare August 20, 2026 04:28
arafatkatze and others added 21 commits August 20, 2026 06:28
Merged via merge-train (release/v3.8.50, batch1 2026-08-20) — static gates (typecheck/file-size/complexity/cognitive/changelog) green on the combined tree; test:unit reds observed in the boarded run were verified pre-existing on the pure release tip (unrelated flake), not caused by this PR. Thanks for the contribution!
…10627) (#10709)

Merged via merge-train (release/v3.8.50, batch1 2026-08-20) — static gates (typecheck/file-size/complexity/cognitive/changelog) green on the combined tree; test:unit reds observed in the boarded run were verified pre-existing on the pure release tip (unrelated flake), not caused by this PR. Thanks for the contribution!
Merged via merge-train (release/v3.8.50, batch1 2026-08-20) — static gates (typecheck/file-size/complexity/cognitive/changelog) green on the combined tree; test:unit reds observed in the boarded run were verified pre-existing on the pure release tip (unrelated flake), not caused by this PR. Thanks for the contribution!
…new cert (#10467) (#10715)

Merged via merge-train (release/v3.8.50, batch1 2026-08-20) — static gates (typecheck/file-size/complexity/cognitive/changelog) green on the combined tree; test:unit reds observed in the boarded run were verified pre-existing on the pure release tip (unrelated flake), not caused by this PR. Thanks for the contribution!
…ted seeds, identity faces (#10716)

Merged via merge-train (release/v3.8.50, batch1 2026-08-20) — static gates (typecheck/file-size/complexity/cognitive/changelog) green on the combined tree; test:unit reds observed in the boarded run were verified pre-existing on the pure release tip (unrelated flake), not caused by this PR. Thanks for the contribution!
… Windows (#10717)

Merged via merge-train (release/v3.8.50, batch1 2026-08-20) — static gates (typecheck/file-size/complexity/cognitive/changelog) green on the combined tree; test:unit reds observed in the boarded run were verified pre-existing on the pure release tip (unrelated flake), not caused by this PR. Thanks for the contribution!
…0722)

Merged via merge-train (release/v3.8.50, batch1 2026-08-20) — static gates (typecheck/file-size/complexity/cognitive/changelog) green on the combined tree; test:unit reds observed in the boarded run were verified pre-existing on the pure release tip (unrelated flake), not caused by this PR. Thanks for the contribution!
…10725)

Merged via merge-train (release/v3.8.50, batch1 2026-08-20) — static gates (typecheck/file-size/complexity/cognitive/changelog) green on the combined tree; test:unit reds observed in the boarded run were verified pre-existing on the pure release tip (unrelated flake), not caused by this PR. Thanks for the contribution!
…through (#10729)

Merged via merge-train (release/v3.8.50, batch1 2026-08-20) — static gates (typecheck/file-size/complexity/cognitive/changelog) green on the combined tree; test:unit reds observed in the boarded run were verified pre-existing on the pure release tip (unrelated flake), not caused by this PR. Thanks for the contribution!
…cks (#10681) (#10730)

Merged via merge-train (release/v3.8.50, batch1 2026-08-20) — static gates (typecheck/file-size/complexity/cognitive/changelog) green on the combined tree; test:unit reds observed in the boarded run were verified pre-existing on the pure release tip (unrelated flake), not caused by this PR. Thanks for the contribution!
Merged via merge-train (release/v3.8.50, batch1 2026-08-20) — static gates (typecheck/file-size/complexity/cognitive/changelog) green on the combined tree; test:unit reds observed in the boarded run were verified pre-existing on the pure release tip (unrelated flake), not caused by this PR. Thanks for the contribution!
…ken refresh (#10732)

Merged via merge-train (release/v3.8.50, batch1 2026-08-20) — static gates (typecheck/file-size/complexity/cognitive/changelog) green on the combined tree; test:unit reds observed in the boarded run were verified pre-existing on the pure release tip (unrelated flake), not caused by this PR. Thanks for the contribution!
Merged via merge-train (release/v3.8.50, batch1 2026-08-20) — static gates (typecheck/file-size/complexity/cognitive/changelog) green on the combined tree; test:unit reds observed in the boarded run were verified pre-existing on the pure release tip (unrelated flake), not caused by this PR. Thanks for the contribution!
…tats (#10769)

Merged via merge-train (release/v3.8.50, batch1 2026-08-20) — static gates (typecheck/file-size/complexity/cognitive/changelog) green on the combined tree; test:unit reds observed in the boarded run were verified pre-existing on the pure release tip (unrelated flake), not caused by this PR. Thanks for the contribution!
…log (#10770)

Merged via merge-train (release/v3.8.50, batch1 2026-08-20) — static gates (typecheck/file-size/complexity/cognitive/changelog) green on the combined tree; test:unit reds observed in the boarded run were verified pre-existing on the pure release tip (unrelated flake), not caused by this PR. Thanks for the contribution!
Merged via merge-train (release/v3.8.50, batch1 2026-08-20) — static gates (typecheck/file-size/complexity/cognitive/changelog) green on the combined tree; test:unit reds observed in the boarded run were verified pre-existing on the pure release tip (unrelated flake), not caused by this PR. Thanks for the contribution!
…10772)

Merged via merge-train (release/v3.8.50, batch1 2026-08-20) — static gates (typecheck/file-size/complexity/cognitive/changelog) green on the combined tree; test:unit reds observed in the boarded run were verified pre-existing on the pure release tip (unrelated flake), not caused by this PR. Thanks for the contribution!
…cription provider (#10773) (#10774)

Merged via merge-train (release/v3.8.50, batch1 2026-08-20) — static gates (typecheck/file-size/complexity/cognitive/changelog) green on the combined tree; test:unit reds observed in the boarded run were verified pre-existing on the pure release tip (unrelated flake), not caused by this PR. Thanks for the contribution!
Merged via merge-train (release/v3.8.50, batch1 2026-08-20) — static gates (typecheck/file-size/complexity/cognitive/changelog) green on the combined tree; test:unit reds observed in the boarded run were verified pre-existing on the pure release tip (unrelated flake), not caused by this PR. Thanks for the contribution!
Merged via merge-train (release/v3.8.50, batch1 2026-08-20) — static gates (typecheck/file-size/complexity/cognitive/changelog) green on the combined tree; test:unit reds observed in the boarded run were verified pre-existing on the pure release tip (unrelated flake), not caused by this PR. Thanks for the contribution!
Merged via merge-train (release/v3.8.50, batch1 2026-08-20) — static gates (typecheck/file-size/complexity/cognitive/changelog) green on the combined tree; test:unit reds observed in the boarded run were verified pre-existing on the pure release tip (unrelated flake), not caused by this PR. Thanks for the contribution!
RaviTharuma and others added 21 commits August 20, 2026 11:47
Merged — validated together with a batch of related RaviTharuma PRs in one combined worktree (typecheck:core clean, complexity/file-size/changelog gates green, focused tests passing). Thanks for the contribution!
Merged — validated together with a batch of related RaviTharuma PRs in one combined worktree (typecheck:core clean, complexity/file-size/changelog gates green, focused tests passing). Thanks for the contribution!
…S heap (#10818)

Merged — validated together with a batch of related RaviTharuma PRs in one combined worktree (typecheck:core clean, complexity/file-size/changelog gates green, focused tests passing). Thanks for the contribution!
Merged — validated together with a batch of related RaviTharuma PRs in one combined worktree (typecheck:core clean, complexity/file-size/changelog gates green, focused tests passing). Thanks for the contribution!
…10814)

Merged — validated together with a batch of related RaviTharuma PRs in one combined worktree (typecheck:core clean, complexity/file-size/changelog gates green, focused tests passing). Thanks for the contribution!
Merged — validated together with a batch of related RaviTharuma PRs in one combined worktree (typecheck:core clean, complexity/file-size/changelog gates green, focused tests passing). Thanks for the contribution!
Merged — locally validated (3/3 focused tests, changelog gate green) after resolving base-drift against #10827's event-loop-lag doc note (both landed today, same MONITORING_GUIDE.md table cell — combined the /livez recommendation with the #10303 lag caveat). Thanks!
Merged — locally validated (changelog gate green) after resolving base-drift against #10817's SQLite HA section (both landed today, same insertion point in DOCKER_GUIDE.md — combined, both sections kept). Thanks!
Merged — carried forward the PR's own real value (the first 2 commits: ignore ad-hoc BOT_TOKEN/BOT_URL in env-doc-sync, plus the lock-in test). The branch had accumulated 7 more commits chasing the moving release tip across several rebases (each one re-fixing base-reds that had already moved again by the next rebase) — dropped those since they no longer apply to the current tip, and cherry-picked just the 2 with lasting value, preserving your authorship. 14/14 focused tests pass, changelog gate green. Thanks!
scripts/ad-hoc mesh helpers read operator-supplied BOT_TOKEN/BOT_URL.
They are not OmniRoute runtime config and should not fail Docs Gates
on every PR.

Unblocks check:env-doc-sync on release/v3.8.50.
Drive runEnvDocSync so BOT_TOKEN/BOT_URL are ignored by IGNORE_FROM_CODE and flagged when ignore is empty.
#10838)

Merged — locally validated (60/60 combined image-generation tests, typecheck:core clean, complexity/cognitive/file-size/changelog gates green).
…10842)

Merged — locally validated (60/60 combined image-generation tests, typecheck:core clean, complexity/cognitive/file-size/changelog gates green, small rebaseline for the combined imageRegistry.ts growth).
#10737)

Merged — locally validated (fusion-vision-panel-3378 test green, file-size/changelog gates green) after resolving base-drift against #10842/#10838 (both landed just before).
Merged — locally validated together with related cryptiklemur PRs (typecheck:core clean, complexity/cognitive/file-size/changelog gates green, focused tests passing). Real bug, clean fix, great regression test. Thanks!
Merged — locally validated together with related cryptiklemur PRs (typecheck:core clean, gates green). Good catch on the phantom usage_logs table. Thanks!
Merged — locally validated (typecheck:core clean, gates green). Thanks for the extra regression coverage!
Merged — locally validated (70/70 combined chatcore-translation-paths tests, typecheck:core clean, gates green). Thanks!
@diegosouzapw
diegosouzapw merged commit 22e46a0 into diegosouzapw:release/v3.8.50 Aug 20, 2026
2 of 4 checks passed
arminanton added a commit to arminanton/OmniRoute that referenced this pull request Aug 21, 2026
…s before cached fallback

Two release/v3.8.50 base-red unit failures in tests/unit/cc-compatible-provider.test.ts
(part of the diegosouzapw#9985 "release branch not green" set), re-derived and fixed against the
current tip rather than cherry-picked, since the base-red set has drifted since the
originating PRs were written.

1. models route (genuine code bug; ref diegosouzapw#10795 fix diegosouzapw#1, closes the gap in diegosouzapw#10863):
   GET /api/providers/:id/models returned HTTP 200 with an empty/cached model list
   for a Claude-Code-compatible (anthropic-compatible-cc-*) connection instead of the
   documented HTTP 400 "does not support models listing". The isClaudeCodeCompatibleProvider
   check sat AFTER maybeReturnCachedDiscovery() / maybeReturnAutoFetchDisabled() inside
   the isAnthropicCompatibleProvider block, so a freshly-created CC connection with no
   synced models hit the auto-fetch-disabled 200 fallback first. Hoisted the CC 400
   check to the top of that block so the unsupported-listing contract always wins.

2. stale test assertion (test-only; the code is correct per diegosouzapw#10684): the
   "preserves client cache markers" test asserted a client message marker sent with no
   ttl defaults to 1h even when a system block carries an explicit 5m marker. diegosouzapw#10684
   ("advance Claude cache breakpoints on growing tails") changed normalizeCacheControlTtl
   to "default missing ttl to 5m after a 5m breakpoint" (sections processed
   tools -> system -> messages) and updated claude-code-parity.test.ts /
   chatcore-translation-paths.test.ts to match, but missed this third assertion.
   Updated it to expect 5m, matching the intended breakpoint semantics.

Verified: tests/unit/cc-compatible-provider.test.ts 27/27, claude-code-parity 40/40,
chatcore-translation-paths 75/75, typecheck:core clean.

Refs diegosouzapw#9985, diegosouzapw#10795, diegosouzapw#10863, diegosouzapw#10684.
backryun added a commit to backryun/OmniRoute that referenced this pull request Aug 21, 2026
The 8c4a219 revert of diegosouzapw#10828 accidentally re-scrambled the ordering
inside the isAnthropicCompatibleProvider branch of the provider models
route: the cached-discovery / auto-fetch-disabled fallbacks ran BEFORE
the isClaudeCodeCompatibleProvider rejection, so a CC node with no
cached catalog got a misleading 200 {models: [], warning:
"Auto-fetch disabled"} instead of the contractual 400 "does not
support models listing". Restore diegosouzapw#10828's original order — CC
rejection first — and update the cache-marker ttl expectation for
diegosouzapw#10684's 5m-inheritance rule in the same file's test. 27/27 green.
backryun added a commit to backryun/OmniRoute that referenced this pull request Aug 21, 2026
The 8c4a219 revert of diegosouzapw#10828 accidentally re-scrambled the ordering
inside the isAnthropicCompatibleProvider branch of the provider models
route: the cached-discovery / auto-fetch-disabled fallbacks ran BEFORE
the isClaudeCodeCompatibleProvider rejection, so a CC node with no
cached catalog got a misleading 200 {models: [], warning:
"Auto-fetch disabled"} instead of the contractual 400 "does not
support models listing". Restore diegosouzapw#10828's original order — CC
rejection first — and update the cache-marker ttl expectation for
diegosouzapw#10684's 5m-inheritance rule in the same file's test. 27/27 green.
arminanton added a commit to arminanton/OmniRoute that referenced this pull request Aug 21, 2026
…s before cached fallback

Two release/v3.8.50 base-red unit failures in tests/unit/cc-compatible-provider.test.ts
(part of the diegosouzapw#9985 "release branch not green" set), re-derived and fixed against the
current tip rather than cherry-picked, since the base-red set has drifted since the
originating PRs were written.

1. models route (genuine code bug; ref diegosouzapw#10795 fix diegosouzapw#1, closes the gap in diegosouzapw#10863):
   GET /api/providers/:id/models returned HTTP 200 with an empty/cached model list
   for a Claude-Code-compatible (anthropic-compatible-cc-*) connection instead of the
   documented HTTP 400 "does not support models listing". The isClaudeCodeCompatibleProvider
   check sat AFTER maybeReturnCachedDiscovery() / maybeReturnAutoFetchDisabled() inside
   the isAnthropicCompatibleProvider block, so a freshly-created CC connection with no
   synced models hit the auto-fetch-disabled 200 fallback first. Hoisted the CC 400
   check to the top of that block so the unsupported-listing contract always wins.

2. stale test assertion (test-only; the code is correct per diegosouzapw#10684): the
   "preserves client cache markers" test asserted a client message marker sent with no
   ttl defaults to 1h even when a system block carries an explicit 5m marker. diegosouzapw#10684
   ("advance Claude cache breakpoints on growing tails") changed normalizeCacheControlTtl
   to "default missing ttl to 5m after a 5m breakpoint" (sections processed
   tools -> system -> messages) and updated claude-code-parity.test.ts /
   chatcore-translation-paths.test.ts to match, but missed this third assertion.
   Updated it to expect 5m, matching the intended breakpoint semantics.

Verified: tests/unit/cc-compatible-provider.test.ts 27/27, claude-code-parity 40/40,
chatcore-translation-paths 75/75, typecheck:core clean.

Refs diegosouzapw#9985, diegosouzapw#10795, diegosouzapw#10863, diegosouzapw#10684.
arminanton added a commit to arminanton/OmniRoute that referenced this pull request Aug 22, 2026
…s before cached fallback

Two release/v3.8.50 base-red unit failures in tests/unit/cc-compatible-provider.test.ts
(part of the diegosouzapw#9985 "release branch not green" set), re-derived and fixed against the
current tip rather than cherry-picked, since the base-red set has drifted since the
originating PRs were written.

1. models route (genuine code bug; ref diegosouzapw#10795 fix diegosouzapw#1, closes the gap in diegosouzapw#10863):
   GET /api/providers/:id/models returned HTTP 200 with an empty/cached model list
   for a Claude-Code-compatible (anthropic-compatible-cc-*) connection instead of the
   documented HTTP 400 "does not support models listing". The isClaudeCodeCompatibleProvider
   check sat AFTER maybeReturnCachedDiscovery() / maybeReturnAutoFetchDisabled() inside
   the isAnthropicCompatibleProvider block, so a freshly-created CC connection with no
   synced models hit the auto-fetch-disabled 200 fallback first. Hoisted the CC 400
   check to the top of that block so the unsupported-listing contract always wins.

2. stale test assertion (test-only; the code is correct per diegosouzapw#10684): the
   "preserves client cache markers" test asserted a client message marker sent with no
   ttl defaults to 1h even when a system block carries an explicit 5m marker. diegosouzapw#10684
   ("advance Claude cache breakpoints on growing tails") changed normalizeCacheControlTtl
   to "default missing ttl to 5m after a 5m breakpoint" (sections processed
   tools -> system -> messages) and updated claude-code-parity.test.ts /
   chatcore-translation-paths.test.ts to match, but missed this third assertion.
   Updated it to expect 5m, matching the intended breakpoint semantics.

Verified: tests/unit/cc-compatible-provider.test.ts 27/27, claude-code-parity 40/40,
chatcore-translation-paths 75/75, typecheck:core clean.

Refs diegosouzapw#9985, diegosouzapw#10795, diegosouzapw#10863, diegosouzapw#10684.

(cherry picked from commit 6746069)
arminanton added a commit to arminanton/OmniRoute that referenced this pull request Aug 22, 2026
…s before cached fallback

Two release/v3.8.50 base-red unit failures in tests/unit/cc-compatible-provider.test.ts
(part of the diegosouzapw#9985 "release branch not green" set), re-derived and fixed against the
current tip rather than cherry-picked, since the base-red set has drifted since the
originating PRs were written.

1. models route (genuine code bug; ref diegosouzapw#10795 fix diegosouzapw#1, closes the gap in diegosouzapw#10863):
   GET /api/providers/:id/models returned HTTP 200 with an empty/cached model list
   for a Claude-Code-compatible (anthropic-compatible-cc-*) connection instead of the
   documented HTTP 400 "does not support models listing". The isClaudeCodeCompatibleProvider
   check sat AFTER maybeReturnCachedDiscovery() / maybeReturnAutoFetchDisabled() inside
   the isAnthropicCompatibleProvider block, so a freshly-created CC connection with no
   synced models hit the auto-fetch-disabled 200 fallback first. Hoisted the CC 400
   check to the top of that block so the unsupported-listing contract always wins.

2. stale test assertion (test-only; the code is correct per diegosouzapw#10684): the
   "preserves client cache markers" test asserted a client message marker sent with no
   ttl defaults to 1h even when a system block carries an explicit 5m marker. diegosouzapw#10684
   ("advance Claude cache breakpoints on growing tails") changed normalizeCacheControlTtl
   to "default missing ttl to 5m after a 5m breakpoint" (sections processed
   tools -> system -> messages) and updated claude-code-parity.test.ts /
   chatcore-translation-paths.test.ts to match, but missed this third assertion.
   Updated it to expect 5m, matching the intended breakpoint semantics.

Verified: tests/unit/cc-compatible-provider.test.ts 27/27, claude-code-parity 40/40,
chatcore-translation-paths 75/75, typecheck:core clean.

Refs diegosouzapw#9985, diegosouzapw#10795, diegosouzapw#10863, diegosouzapw#10684.

(cherry picked from commit 6746069)
muhamadgalihsaputra pushed a commit to niyatna/NiyatnaRoute that referenced this pull request Sep 27, 2026
…zapw#10684)

Merged — locally validated (110/110 focused tests, typecheck:core clean, gates green) after resolving base-drift against diegosouzapw#10683 (both landed today, same test file — combined). Nice, well-tested cache-breakpoint fix. Thanks!
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.