Preserve Codex cost usage window under byte pressure

[Whiteknight07]

Summary

• protect completed files that touch the requested Codex cost-usage window from both row- and byte-budget deletion
• keep the 256 MiB database cap best-effort once only requested-window or recently active data remains
• remove the fallback that stripped protected scan detail and immediately scheduled it for reconstruction
• add repeated-enforcement, report-integrity, fork-parent, out-of-window pruning, and non-Gregorian calendar coverage
• move the post-0.49.1 cost-cache CPU fix into the correct 0.49.2 changelog section

Root cause

The row-budget path protected the requested reporting window, but the byte-budget path explicitly disabled that protection. Deleting a file cascaded into file_day_aggregates, so rebuilding global aggregates permanently removed older requested days from Spend Dashboard.

The initial PR patch correctly enabled requested-window protection, but then fell back to stripOldestRebuildableDetail. That reset protected files to incomplete, deleted their token/row/accumulator state, and marked catch-up pending. The next refresh would reconstruct the same detail, exceed the same cap, strip it again, and potentially repeat indefinitely.

The maintainer revision uses a stable policy: delete only safe out-of-window completed files. Once every remaining file is protected, stop enforcement without mutating scan state. The current report and fork baselines remain authoritative, catch-up does not restart, and the database may remain above its best-effort byte cap.

Storage-policy decision

VISION.md requires explicit sign-off for data-storage behavior changes. This PR deliberately prioritizes requested-window report fidelity and stable refresh behavior over an absolute 256 MiB SQLite cap.

Recommendation: approve the fidelity-first policy. A hard cap cannot be maintained for arbitrarily large requested windows without either deleting displayed history or introducing a larger durable aggregate-only storage design.

Validation

• CODEXBAR_SUPPRESS_TEST_KEYCHAIN_ACCESS=1 swift test --filter CostUsageStoreTests — 60 tests passed
• CODEXBAR_SUPPRESS_TEST_KEYCHAIN_ACCESS=1 swift test --filter CostUsagePerformanceGateTests — 25 tests passed
• make check — passed
• CODEXBAR_SUPPRESS_TEST_KEYCHAIN_ACCESS=1 make test — 834 selections in 70 groups passed; zero failures or retries
• structured autoreview — secret scan clean; no accepted/actionable findings
• regression coverage uses the real CostUsageStore/SQLite path and proves two consecutive over-cap enforcement passes retain in-window rows and report aggregates without setting catch-up pending

Fixes #2823

[clawsweeper]

🦞👀
ClawSweeper picked this up.

Pull request received. I will update this pull request when review starts.

[clawsweeper]

Codex review: needs real behavior proof before merge. Reviewed August 13, 2026, 7:13 AM ET / 11:13 UTC.

ClawSweeper review

What this changes

The PR changes Codex cost-cache retention so byte-pressure cleanup preserves requested Spend Dashboard history and adds regression coverage for protected data.

Regression provenance

Possible regression — probable (reproduction; reviewed change). No predecessor PR is attributed.

Merge readiness

⛔ Blocked until real behavior proof from a real setup is added - 4 items remain

The current main branch still permits byte-budget deletion of requested-window cost data. This patch removes that path with focused SQLite regression coverage, but needs real-cache proof and explicit approval of the intentional best-effort storage-cap policy.

Priority: P1
Reviewed head: 8ee63cc7902328bf20c57c42cbfa2e0d6b894490
Owner decision: Required. See Decision needed.

Review scores

Measure	Result	What it means
Overall readiness	🦪 silver shellfish (2/6)	The focused patch and regression coverage are strong, but real behavior proof for the rebased head is still limited to tests.
Proof confidence	🦪 silver shellfish (2/6)	Needs real behavior proof before merge: The PR provides focused SQLite-store test claims but no after-fix run against a real oversized Codex cache; add redacted terminal output or runtime logs showing retained requested-window history and the resulting cache size. After adding proof, update the PR body; ClawSweeper should re-review automatically. If it does not, the PR author or someone with repository write access can comment @clawsweeper re-review.
Patch quality	🐚 platinum hermit (4/6)	No actionable review findings were identified.

Verification

Check	Result	Evidence
Real behavior	Needs proof	Needs real behavior proof before merge: The PR provides focused SQLite-store test claims but no after-fix run against a real oversized Codex cache; add redacted terminal output or runtime logs showing retained requested-window history and the resulting cache size. After adding proof, update the PR body; ClawSweeper should re-review automatically. If it does not, the PR author or someone with repository write access can comment @clawsweeper re-review.
Evidence reviewed	5 items	Current-main defect: The byte-budget loop explicitly disables requested-window protection and then strips rebuildable detail when no deletion candidate remains. Proposed retention policy: The PR changes the byte loop to protect the requested window and removes the detail-stripping fallback, leaving the byte cap best-effort once only protected data remains. Regression coverage: The branch exercises consecutive over-cap enforcement, in-window aggregate retention, out-of-window pruning, and fork-parent protection through the real CostUsageStore SQLite seam.
Findings	None	None.
Security	None	None.

How this fits together

CodexBar scans local Codex cache files into a SQLite usage store that feeds Spend Dashboard history. Retention applies row and byte budgets before that stored history is read by the dashboard.

flowchart LR
A[Codex cache files] --> B[Usage scanner]
B --> C[SQLite cost cache]
C --> D[Retention budgets]
D --> E[Requested report window]
E --> F[Spend Dashboard]

Loading

Decision needed

Question	Recommendation
Should CodexBar allow the local cost cache to remain above 256 MiB when enforcing that cap would remove requested Spend Dashboard history?	Approve fidelity-first retention: Keep requested-window and required fork data intact, treating the 256 MiB cap as best effort after safe out-of-window pruning.

Why: The patch intentionally changes a data-storage behavior from preserving the cap to preserving displayed history, and VISION.md requires sign-off for that class of change.

Before merge

• Add real behavior proof - Needs real behavior proof before merge: The PR provides focused SQLite-store test claims but no after-fix run against a real oversized Codex cache; add redacted terminal output or runtime logs showing retained requested-window history and the resulting cache size. After adding proof, update the PR body; ClawSweeper should re-review automatically. If it does not, the PR author or someone with repository write access can comment @clawsweeper re-review.
• Resolve merge risk (P1) - After upgrade, a large Codex cache may remain above 256 MiB when all remaining data is protected, increasing local disk use relative to the former effective hard cap.
• Complete next step (P2) - The remaining blockers are explicit storage-policy approval and contributor-supplied real-cache proof, neither of which is a safe automated repair task.

Agent review details

Security

None.

Review metrics

Metric	Value	Why it matters
Patch footprint	3 files; production +5/-50, tests +72/-59	The production change removes the destructive fallback while the larger test delta covers retention-policy boundaries.

Merge-risk options

Maintainer options:

1. Approve the fidelity-first policy (recommended)
   Accept best-effort byte enforcement when safe pruning cannot reduce the cache without removing displayed history.
2. Pause for a bounded-storage design
   Require an alternative persistence strategy if a strict local-disk limit must remain guaranteed.

Technical review

Best possible solution:

Approve the fidelity-first policy only if allowing an over-cap local cache is acceptable, then retain this narrow safeguard and document the cap as best-effort behavior.

Do we have a high-confidence way to reproduce the issue?

Yes from source: current main passes protectRequestedWindow: false in the byte-budget loop, and the branch’s focused SQLite tests exercise the over-cap retention path. This read-only review did not execute the test suite.

Is this the best way to solve the issue?

Unclear pending sign-off: changing the protection flag and removing the rebuild loop is the narrowest repair for lost displayed history, but accepting an over-cap database is a deliberate storage-policy choice.

AGENTS.md: found and applied where relevant.

Codex review notes: model internal, reasoning high; reviewed against 1f5bdb856e1d.

Labels

Label justifications:

• P1: The current released behavior can silently remove requested Spend Dashboard history for users with large Codex caches.
• merge-risk: 🚨 compatibility: Existing installations can retain a larger local cache after upgrade when all remaining data is protected.
• rating: 🦪 silver shellfish: Overall readiness is 🦪 silver shellfish; proof is 🦪 silver shellfish and patch quality is 🐚 platinum hermit.
• status: 📣 needs proof: The PR needs real behavior proof before ClawSweeper can clear the contributor ask. Needs real behavior proof before merge: The PR provides focused SQLite-store test claims but no after-fix run against a real oversized Codex cache; add redacted terminal output or runtime logs showing retained requested-window history and the resulting cache size. After adding proof, update the PR body; ClawSweeper should re-review automatically. If it does not, the PR author or someone with repository write access can comment @clawsweeper re-review.

Evidence

What I checked:

• Current-main defect: The byte-budget loop explicitly disables requested-window protection and then strips rebuildable detail when no deletion candidate remains. (Sources/CodexBarCore/Vendored/CostUsage/CostUsageStore+Retention.swift:301, 1f5bdb856e1d)
• Proposed retention policy: The PR changes the byte loop to protect the requested window and removes the detail-stripping fallback, leaving the byte cap best-effort once only protected data remains. (Sources/CodexBarCore/Vendored/CostUsage/CostUsageStore+Retention.swift:303, 8ee63cc79023)
• Regression coverage: The branch exercises consecutive over-cap enforcement, in-window aggregate retention, out-of-window pruning, and fork-parent protection through the real CostUsageStore SQLite seam. (Tests/CodexBarTests/CostUsageStoreTests.swift:1698, 8ee63cc79023)
• Storage-policy sign-off: The repository VISION.md requires sign-off for behavior changes affecting data storage, which this fidelity-over-cap policy intentionally changes. (VISION.md:19, 1f5bdb856e1d)
• Feature history: Current history shows sustained work on CostUsage retention by Peter Steinberger and a recent related retention change by Yuxin Qiao. (Sources/CodexBarCore/Vendored/CostUsage/CostUsageStore+Retention.swift:260, c41a419421a3)

Likely related people:

• steipete: Current-main history includes the SQLite retention restoration and subsequent CostUsage store work; steipete also authored the rebased PR head. (role: recent cost-store contributor; confidence: high; commits: 38ca30ab93bc, 17afef7b84eb; files: Sources/CodexBarCore/Vendored/CostUsage/CostUsageStore+Retention.swift, Sources/CodexBarCore/Vendored/CostUsage/CostUsageStore+CodexCache.swift)
• Yuxin-Qiao: Recent main history attributes the related active-window retention change to Yuxin Qiao. (role: recent retention contributor; confidence: medium; commits: c41a419421a3; files: Sources/CodexBarCore/Vendored/CostUsage/CostUsageStore+Retention.swift, Tests/CodexBarTests/CostUsageStoreTests.swift)

Rank-up moves

Optional improvements that raise the rating; they are not merge blockers.

• Add redacted output from a real oversized Codex cache showing the requested reporting window remains intact while the cache stays over the best-effort cap.
• Update the PR body with the proof; if a review does not refresh automatically, ask a maintainer to request re-review.

Rating scale

Score	Internal tier	Crab rank	Meaning
6/6	S	🦀 challenger crab	Exceptional readiness
5/6	A	🦞 diamond lobster	Very strong readiness
4/6	B	🐚 platinum hermit	Good normal PR; ordinary maintainer review
3/6	C	🦐 gold shrimp	Useful, but confidence is limited
2/6	D	🦪 silver shellfish	Proof or implementation needs work
1/6	F	🧂 unranked krab	Not merge-ready
N/A	NA	🌊 off-meta tidepool	Rating does not apply

Overall follows the weaker of proof and patch quality.
Shiny media proof means a screenshot, video, or linked artifact directly shows the changed behavior. Runtime, network, CSP, and security claims still need visible diagnostics.

Workflow

• ClawSweeper keeps one durable marker-backed review comment per issue or PR.
• Re-runs edit this comment so the latest verdict, findings, and automation markers stay together instead of adding duplicate bot comments.
• A fresh review can be triggered by eligible @clawsweeper re-review comments, exact-item GitHub events, scheduled/background review runs, or manual workflow dispatch.
• PR/issue authors and users with repository write access can comment @clawsweeper re-review or @clawsweeper re-run on an open PR or issue to request a fresh review only.
• Maintainers can also comment @clawsweeper review to request a fresh review only.
• Fresh-review commands do not start repair, autofix, rebase, CI repair, or automerge.
• Maintainer-only repair and merge flows require explicit commands such as @clawsweeper autofix, @clawsweeper automerge, @clawsweeper fix ci, or @clawsweeper address review.
• Maintainers can comment @clawsweeper explain to ask for more context, or @clawsweeper stop to stop active automation.

History

Review history (6 earlier review cycles)

• reviewed 2026-08-10T00:30:31.119Z sha 502d123 :: needs real behavior proof before merge. :: none
• reviewed 2026-08-10T00:35:18.726Z sha 502d123 :: needs real behavior proof before merge. :: none
• reviewed 2026-08-10T00:48:11.629Z sha 502d123 :: needs maintainer review before merge. :: none
• reviewed 2026-08-10T08:24:15.632Z sha 166d5e0 :: needs real behavior proof before merge. :: none
• reviewed 2026-08-11T13:12:03.872Z sha 166d5e0 :: needs real behavior proof before merge. :: none
• reviewed 2026-08-12T15:59:02.784Z sha 166d5e0 :: needs real behavior proof before merge. :: [P1] Rebase the release note onto the current unreleased section

[Whiteknight07]

@clawsweeper re-review

[clawsweeper]

🦞🧹
ClawSweeper re-review requested.

I asked ClawSweeper to review this item again.
Action: item re-review queued (workflow sweep.yml, event repository_dispatch).
Result: when the review finishes, ClawSweeper will create the durable review comment if needed or update the existing comment in place.

Re-review progress:

• State: Failed
• Detail: Durable publication exhausted its retry budget and was retained for operator dead-letter recovery.
• Run: https://github.com/openclaw/clawsweeper/actions/runs/31345955488
• Updated: 2026-08-10T01:01:08.197Z

[steipete]

Rebased onto current main (1f5bdb856e) and refreshed the implementation, head is now 8ee63cc790.

Changes vs the previous head:

• Re-applied the retention fix on top of the new identical-content save path that landed with Skip identical cost cache database rewrites #2852: the tightened-budget assertion in identical codex cache save skips content rewrites but advances the scan timestamp now expects catchUpRequired == false with the store staying over the best-effort cap and in-window rows intact, matching the protected-window policy.
• Hardened byte budget preserves fork parent detail required by a surviving child: the fork parent is now out-of-window and stale, so the test exercises the fork-dependency protection clause rather than passing via requested-window protection alone.
• Moved the changelog entry into the current 0.49.4 — Unreleased section and dropped the stray duplicate of the 0.49.2 CPU-fix bullet.

Proof: swift test --filter CostUsageStoreTests (71 tests), swift test --filter CostUsagePerformanceGateTests (28 tests), and make check all pass locally; full make test running.