Fix Codex usage history indexing and catch-up completion

[Quicksaver]

Summary

Fix Codex Usage & Spend indexing for very large local histories. The scanner now keeps bounded passes bounded across discovery, candidate selection, file parsing, and cache reconciliation, while avoiding corpus-wide cache searches for every session file.

The catch-up state is also made durable and self-healing: completed files can leave stale lookback queues, completed caches no longer restart from zero after relaunch, appended sessions resume correctly, and refreshes cannot lose the worker that should process newly discovered tail work.

What Changed

• Cap time-limited Codex catch-up candidate work and check the byte/time budget before starting additional files or discovery work.
• Page current-window, flat-root, and older-partition directory discovery through a persisted 512-entry cursor instead of listing and sorting the entire active corpus before candidate selection.
• Build one file-identity alias index per refresh instead of scanning the entire cache for every session file.
• Use normalized path sets for discovery and deduplication, including /private/var aliases.
• Allow already-complete cached files retained in a stale lookback queue to be scheduled once and acknowledged as complete.
• Reconcile at most 512 persisted stale-lookback paths per cache load/save and defer exact full-inventory proof until no bounded work remains.
• Prepare at most 512 retained lookback paths per bounded pass, so root validation and URL materialization cannot walk the full queue before candidate selection.
• Keep post-scan progress accounting incremental during catch-up; exact byte totals, metadata validation, and sorted inventory persistence now run only after bounded work is exhausted.
• Persist an explicit completion-proof sentinel so the final non-empty work slice cannot trigger full accounting or rediscover the newest sessions; the following zero-work pass performs the exact proof and publishes a distinct completion progress key.
• Persist the exact current scan inventory and reconcile stale catch-up metadata only when every file in that inventory is complete.
• Preserve completion across relaunches when APFS exposes the same inode with a different device component, while validating size, mtime, and the indexed-content anchor before reusing cached state.
• Normalize persisted file identities against the current Codex root device and invalidate completion safely when a file was appended or changed.
• Queue one same-mode catch-up restart for both the primary Codex view and Spend Dashboard so a refresh that discovers tail work cannot race with worker teardown, while ensuring both Stop paths clear queued restarts before an active pass exits.
• Add work counters and regression coverage for bounded large-corpus scans, linear alias lookup, exact inventory completion, relaunch identity drift, appends, cancellation, and catch-up restart races.

Why

A real Codex history of approximately 6.8 GB exposed several compounding failures. Initial indexing became progressively slower as more files entered the cache, eventually appearing stalled after hours. Once the scanner was made faster, it could reach 100% but still fail while finalizing, leaving charts unavailable. Relaunching CodexBar then restarted the multi-gigabyte analysis, and later attempts repeatedly advanced only a small amount before returning to the same error state.

The symptoms were not one isolated failure: they combined unbounded work outside the parser budget, quadratic cache-identity lookup, stale retained-lookback entries, inconsistent persisted completion metadata, filesystem identity drift across relaunches, and a worker lifecycle race.

Evidence

Incident observations

• The original corpus was about 6.68 GB and continued growing with normal Codex use.
• Throughput initially advanced by megabytes every few minutes, then degraded to dozens of megabytes, then only a few megabytes; after many hours only about 1.67 GB had been processed.
• After the first performance fixes, the same history rapidly reached roughly 6.83 GB / 6.83 GB, proving that raw parsing was no longer the dominant blocker, but finalization still errored and the data/charts were not published.
• Relaunching showed the analysis beginning again near 1.7 MB / 6.84 GB, demonstrating that completion was not being restored durably.
• Subsequent runs advanced only about 20 MB before reproducing the completed-analysis error, which pointed to stale pending state and worker/cache reconciliation rather than an unreadable source record.

Root-cause findings and corresponding fixes

1. Cache alias lookup grew quadratically with the corpus. Each file could search all cached files for a matching filesystem identity. The scanner now creates one identity-to-path index and visits only the matching bucket. A 1,500-file regression corpus asserts 1,500 alias candidates visited rather than approximately 2.25 million, processes no unchanged usage rows, and stays within the timing gate.
2. A bounded pass still performed unbounded candidate work. Large retained lookback queues could be fully filtered before the 512-file cap took effect. Time-limited passes now inspect at most 512 unique candidate paths before cache-eligibility checks, reuse the URL identities already established during discovery, and consult the time/byte budget before additional files and discovery work. Regression coverage verifies that two passes each inspect and scan exactly 512 files while preserving and advancing the remainder.
3. Completed files could remain permanently pending. Older cache state could retain an already-complete file in pendingFilePaths, while candidate filtering excluded complete files. That made the UI reach 100% but prevented finalization forever. Retained paths are now admitted once so completion can acknowledge and remove them.
4. Persisted aggregate progress could disagree with the current scan inventory. catchUpPending and zero/partial counters could survive after every current file was complete. Conversely, counting every file retained from a wider history window could falsely satisfy the smaller current-scan count while current files were still absent. The scanner now persists the exact current inventory paths, and cache loading verifies every one against current metadata and indexed-content anchors before clearing pending state. Wider retained files cannot satisfy that proof.
5. Filesystem identity was not stable enough across relaunches. APFS can expose the same inode with a different st_dev component. Treating the full persisted device:inode value as permanently stable caused completed files to look new after reopening the app. Restore now maps persisted inodes to the current Codex root device and validates mtime, size, and the token-index content anchor before adopting the current identity.
6. Appended files needed precise invalidation and resume. If metadata differs after identity normalization, unchanged complete files remain reusable, genuine appends retain their identity but become incomplete, and rewrites drop reusable scan state. Tests reconstruct both stale-progress and append-after-completion caches and verify a single-file resume returns them to an exact completed state.
7. Same-mode refreshes could lose their catch-up worker. A refresh could discover new tail work while an existing automatic worker was finishing; the old early return then allowed teardown to leave pending work with no owner. Primary and Spend Dashboard catch-up now queue one restart and launch it after the current task clears its token and scope.
8. Cancel could immediately restart an active catch-up pass. Both the primary and Spend Dashboard stop paths returned while a pass was running before clearing an already queued restart. Deferred teardown could then launch another worker and reset the stop flag. Both Stop paths now clear their queued restart before that return.
9. Persisted reconciliation bypassed the bounded scanner. Cache load and save synchronously validated every retained lookback path before candidate scheduling, and exact inventory proof was evaluated before cheap pending-work guards. Reconciliation now validates at most 512 pending paths per invocation without first indexing the full cache, while full current-inventory validation runs only when no active lookback, discovery, or incomplete file work remains.
10. Post-scan progress accounting bypassed the bounded pass. After scanning at most 512 candidates, the scanner still sorted every discovered path and fetched metadata for the whole corpus to rebuild byte/file progress and inventory state. Catch-up now advances file progress from the retained lookback state without a corpus traversal, marks byte totals indeterminate while that exact denominator is unavailable, and performs the exact metadata/inventory proof once no bounded work remains.
11. Retained-lookback preparation bypassed candidate limits. Before the 512-candidate selector ran, every persisted pending path was root-validated and materialized. Bounded passes now prepare at most 512 paths and feed only that slice to selection and finalization; a no-op discovery pass also avoids reconstructing and sorting the unchanged full pending queue.
12. The final work slice could never reach a distinct proof pass. Deferring exact accounting without a durable marker allowed cache reconciliation to discard the empty lookback state, after which the newest active files were rediscovered and retried forever. The scanner now persists an explicit awaitingExactInventory sentinel, preserves it across the store round trip, clears it on the following zero-work pass, and includes pending/progress totals in the catch-up status identity so charts publish the completed snapshot.
13. Discovery itself bypassed every downstream bound. The scanner listed and sorted every JSONL file in the current date window, walked the flat root, and reconstructed cached-file candidates before applying the 512-file selector. Oversized active-day directories therefore still imposed corpus-sized work on every pass. Discovery now shares the 512-entry work budget across current-window, flat-root, and older date-partition pages, persists the logical day/directory cursor, and defers recursive legacy discovery plus the exact cached inventory to the final proof pass. Relaunch cursor rehydration is also bounded and does not reparse session JSONL.
14. The parser-hash update rejected every v0.49.2 Cost Usage database. The store now recognizes the exact released v0.49.2 parser hash as a compatible scheduling-only upgrade, updates the schema/hash atomically, and retains completed file rows. A focused upgrade regression verifies no rebuild occurs and all completed state remains available.
15. Device-drift recovery could stat the entire persisted corpus during load. Identity validation now stops at 512 files, persists the remainder in the active lookback queue, preserves that queue across scan-window resets, and removes entries only after a positively validated current snapshot. Missing or changed files stay queued for scanner handling; ordinary append/migration queues retain their existing behavior.

Automated regression evidence

• An oversized active-day directory with 1,500 JSONL files advances in bounded 512-entry discovery and scan slices without first listing, sorting, or progress-accounting over the full corpus. The first pass records first=512 discovery=512 attempts=512; after resetting the live directory handle, the bounded rehydration pass records relaunched=512 discovery=512 attempts=0; the following slice records second=1024 discovery=512 visits=512 attempts=512 accounting=0.
• Simulated relaunch rehydration proves that restoring the logical cursor remains bounded, does not reparse cached JSONL, preserves the 512 indexed files, and resumes forward progress on the following slice.
• Persisted cache save and the subsequent cache load each stop reconciliation at exactly 512 visits: save=512 load=512 pending=921.
• A 600-file finalization regression proves the final 88-file work slice performs no whole-inventory accounting, while the following zero-work pass performs exactly 600 accounting visits and clears pending state: finalWorkAttempts=88 finalWorkAccounting=0 proofAttempts=0 proofAccounting=600.
• Warm 1,500-file refreshes perform linear indexed alias lookup and parse zero unchanged usage rows.
• Retained complete files clear stale lookback state and repair processed/total byte and file counters.
• Completion reconciliation refuses to use a complete file from a wider retained window as proof for a missing current-inventory file.
• Completion survives persisted device-component drift only after content validation.
• Appended session files invalidate completion and converge again without replaying the whole corpus.
• Primary and Spend Dashboard same-mode refresh races retain a worker and finish pending tail work.
• Cancel during an active primary or Spend Dashboard pass clears any queued restart.
• Existing oversized-record, bounded-byte, wall-clock deadline, fork resume, sparse checkpoint, and exact-result convergence gates remain covered.

Validation

• CostUsageCatchUpCompletionTests: 5/5 passed, including a 600-file identity-drift corpus (512 validated, 88 deferred), missing-file retention, scan-window queue preservation, and normalized-alias completion.
• CostUsageBoundedFinalizationTests: 1 test passed, proving exact inventory work is isolated to the zero-work completion pass.
• UsageStoreCodexCostCatchUpTests: 6 tests passed, including primary active-pass cancellation with a queued restart.
• UsageStoreSpendDashboardCodexCostCatchUpTests: 6 tests passed, including active-pass cancellation with a queued restart.
• CostUsagePerformanceGateTests: 27 tests passed; oversized active-day discovery, candidate selection, JSONL scanning, and persisted reconciliation each stopped at 512 visits. Simulated relaunch rehydration performed zero JSONL scan attempts and preserved the 512 already-indexed files before the next slice advanced to 1,024.
• CostUsageBoundedProgressTests: 10/10 passed after the identity-queue repair, preserving existing append, rewrite, missing-file, cancellation, completion, and migration queue behavior.
• CostUsageStoreTests: 61/61 passed, including an exact v0.49.2 database upgrade that retains completed rows, installs the current schema/hash, and records zero rebuilds.
• ProviderArchitectureGatekeeperTests: 38 tests passed after rebasing its exact source anchors onto the latest merged upstream behavior.
• The latest 1,500-file warm-cache proof completed in 1,075 ms with 1,500 indexed lookups, 1,500 candidates visited, and zero unchanged usage rows parsed.
• make check passed, including generated-source checks, SwiftFormat, and strict SwiftLint with zero violations.
• Latest upstream/main at e5528d452 merged before final validation, including the macOS 15 Cost Usage store executor-isolation fix.
• Full make test reached 45 successful groups before the same unrelated, reproducible locale-sensitive MiniMaxMenuCardBillingTests failure on group 46 and its retry: the test expects comma-grouped 1,234/5,678, while this machine's current locale renders non-breaking-space grouping. The failing suite does not touch Codex cost scanning or any file changed by this branch.
• Manual real-data validation against the same approximately 6.8 GB Codex history that exposed the failures.

Proof

After the final fixes, the same approximately 6.83–6.84 GB local Codex dataset used to discover the performance and completion failures was quickly and successfully analyzed in only a few minutes. This validates the complete path against the original incident corpus, not merely a reduced fixture.

Redacted real-history debug log and persisted-store snapshot

The excerpt below is from the real incident dataset. Provider/account details, home paths, and spend values were removed. No conversation content was read or included.

[2026-08-10T18:54:00.362Z] [INFO] CodexBar startup [provider-state details redacted; Codex enabled]
[2026-08-10T18:54:31.626Z] [INFO] Codex cost scan applied work limits bytesConsumed=790493 deferredByBudget=0 deferredByTime=1 maxBytesPerRefresh=536870912 maxFileBytes=268435456 partialFiles=0
[2026-08-10T18:54:48.139Z] [INFO] cost usage success provider=codex duration=34.31s today=[REDACTED_SPEND] historyDays=30 windowCost=[REDACTED_SPEND]

[2026-08-10T18:57:02.992Z] [INFO] CodexBar startup [provider-state details redacted; Codex enabled]
[2026-08-10T18:57:27.755Z] [INFO] Codex cost scan applied work limits bytesConsumed=404384 deferredByBudget=0 deferredByTime=1 maxBytesPerRefresh=536870912 maxFileBytes=268435456 partialFiles=0
[2026-08-10T18:57:44.201Z] [INFO] cost usage success provider=codex duration=34.81s today=[REDACTED_SPEND] historyDays=30 windowCost=[REDACTED_SPEND]

[2026-08-10T19:57:26.551Z] [INFO] Codex cost scan applied work limits bytesConsumed=4600693 deferredByBudget=0 deferredByTime=1 maxBytesPerRefresh=536870912 maxFileBytes=268435456 partialFiles=0
[2026-08-10T19:57:43.578Z] [INFO] cost usage success provider=codex duration=31.53s today=[REDACTED_SPEND] historyDays=30 windowCost=[REDACTED_SPEND]

# Read-only query of the persisted cost-usage store after the final line above:
catch_up_pending=0
processed_bytes=6887714668
total_bytes=6887714668
completed_files=10949
total_files=10949
last_scan_utc=2026-08-10 19:57:12

This shows two distinct relaunches reusing the existing multi-gigabyte cache and scanning only 790,493 bytes and 404,384 bytes before successful publication, rather than replaying the corpus. The later 4,600,693-byte delta demonstrates appended history being incorporated. The persisted state then records exact byte/file completion with no pending catch-up.

A later read-only snapshot after normal Codex use showed the same cache still complete while the corpus had grown: catch_up_pending=0, processed_bytes=6898985724, total_bytes=6898985724, completed_files=10955, total_files=10955.

Current-head bounded traversal and completion-proof output

This current-head proof uses generated multi-thousand-file history and contains no provider identity, paths, spend, or conversation content.

[discovery-proof] first=512, discovery=512, attempts=512
[discovery-proof] relaunched=512, discovery=512, attempts=0
[discovery-proof] second=1024, discovery=512, visits=512, attempts=512, accounting=0
[alias-index-proof] warm refresh 1500 files: 1075 ms, lookups=1500, candidates=1500
[finalization-proof] finalWorkAttempts=88, finalWorkAccounting=0, proofAttempts=0, proofAccounting=600

The counters demonstrate that each pending phase is independently bounded: directory discovery, candidate selection, scanning, persisted reconciliation, and progress accounting. Relaunch rehydration visits only one bounded page, does not parse JSONL, and the next slice resumes forward progress. Exact whole-inventory accounting occurs only on the separate pass with zero file-scan attempts.

The same behaviors are also captured by synthetic regressions so the proof does not depend solely on one machine's local history.

Redacted current-head v0.49.2 upgrade and bounded identity-drift proof

The final source tree was run against an isolated copy of the real large-history database and read-only session corpus. The copy was relabeled with the exact v0.49.2 parser hash/schema and its stored device components were changed only inside that copy. Paths, identifiers, spend, identity values, and conversation content were neither printed nor retained.

[current-head-live-proof] mode=upgrade-drift-load elapsed_ms=9910 retained_files=10978 load_validation=512 pending=1
[current-head-live-proof] mode=relaunch-catch-up elapsed_ms=32170 retained_files=10978 attempts=512 discovery=512 accounting=0 pending=1 processed_bytes=0 total_bytes=0
[persisted-store] parser_hash=f1d04f67770b64cf schema=current retained_files=10978 pending_paths=9954

This proves the released database upgrades without deletion, cache load stops identity validation at exactly 512, the next relaunch performs exactly one bounded 512-file discovery/scan slice, and the remaining queue survives in the final-schema store. The temporary harness and database copy were deleted after the aggregate markers were captured; the real cache and roughly 7.1 GiB/12,400-file session corpus remained read-only.

Cache-wide migration queue proof

The latest maintainer follow-up closes a migration-specific hole in the bounded scanner. Pricing/model catalog, project metadata, turn-ID, priority metadata, and priority-turn migrations now reseed the already-discovered normalized path inventory into the existing durable active-lookback queue, even when roots and the scan window are unchanged and the old queue has drained. No new persisted schema is introduced: migration metadata can advance on the first pass because the queue itself owns every remaining revisit, and later passes drain it without reseeding.

The six-owner 600-file regression matrix proves the first bounded migration pass attempts exactly 512 files, retains 88 paths, and performs zero full progress-accounting visits. The second pass attempts exactly those remaining 88 paths and proves a path beyond the first slice was reparsed. The separate exact-validation pass reaches 600/600, and the following ordinary warm refresh performs zero candidate visits, reparses, or progress-accounting visits. Additional 513-file and partially drained 600-file tests prove migration reseeding preserves newest-first order while retaining any out-of-inventory pending work.

Current validation:

• Migration suite: 3 tests passed, including all 6 migration-owner cases.
• Existing bounded-progress suite: 10/10 passed, preserving append, rewrite, missing-file, cancellation, completion, and queue behavior.
• Requested focused scanner/store/catch-up/architecture matrix: 151 selections passed with the global reconciliation-hook suite run in isolation.
• Parser hash check and make check: passed; SwiftLint reported zero violations.
• Full make test reached 45 successful groups before the same unrelated, reproducible locale-sensitive MiniMaxMenuCardBillingTests failure on group 46 and its retry; the remaining 119 tests in that exact post-merge shard passed.

[clawsweeper]

🦞👀
ClawSweeper picked this up.

Pull request received. I will update this pull request when review starts.

[clawsweeper]

Codex review: needs maintainer review before merge. Reviewed August 13, 2026, 5:17 PM ET / 21:17 UTC.

ClawSweeper review

What this changes

The PR bounds large Codex Usage & Spend history scans, preserves persisted catch-up state across relaunches and appends, and prevents catch-up task restart races.

Regression provenance

Possible regression — probable (reproduction; reviewed change). No predecessor PR is attributed.

Merge readiness

⚠️ Ready for maintainer review - 5 items remain

The PR remains worth maintaining: it has real after-fix evidence and the repository owner has explicitly left the compatibility-sensitive persisted-cache design pending sign-off. Local read-only inspection could not run because the review sandbox failed during command startup, so this verdict relies on the supplied PR discussion and diff context.

Priority: P1
Reviewed head: 01f72618cd1c223618dd0b9157c1d887c2251892
Owner decision: Required. See Decision needed.

Review scores

Measure	Result	What it means
Overall readiness	🐚 platinum hermit (4/6)	Strong real-history proof and extensive focused regression coverage support the patch, with owner approval and a clean rebase still required for its persisted-state design.
Proof confidence	🦞 diamond lobster (5/6)	Sufficient (logs): Redacted after-fix logs and persisted-store results from a real multi-gigabyte history show relaunch reuse, appended-history processing, and a completed catch-up state; private paths and values were redacted.
Patch quality	🐚 platinum hermit (4/6)	No actionable review findings were identified.

Verification

Check	Result	Evidence
Real behavior	Verified	Sufficient (logs): Redacted after-fix logs and persisted-store results from a real multi-gigabyte history show relaunch reuse, appended-history processing, and a completed catch-up state; private paths and values were redacted.
Evidence reviewed	4 items	Sandbox limitation: The attempted read-only repository inspection failed before command execution with bwrap: loopback: Failed RTM_NEWADDR: Operation not permitted; no local source, git history, or test result is asserted from this review. Maintainer-owned design decision: The repository owner states that the branch is intentionally unmerged pending owner sign-off on persisted inventory/storage ownership after adopting a queue-only migration continuation design. Persisted cache compatibility surface: The patch adds persisted inventory and lookback cursor fields and whitelists the released v0.49.2 parser hash as a compatible upgrade, so it affects existing Cost Usage stores rather than only in-memory scheduling.
Findings	None	None.
Security	None	None.

How this fits together

CodexBar scans local Codex session JSONL history into a persisted Cost Usage cache used by the primary Usage view and Spend Dashboard. Background catch-up workers advance that cache in bounded passes and publish progress or completion to those views.

flowchart LR
  A[Local Codex session files] --> B[Bounded file discovery]
  B --> C[Catch-up scan queue]
  C --> D[Persisted Cost Usage cache]
  D --> E[Background catch-up worker]
  E --> F[Usage and Spend Dashboard]
  E --> G[Completion and progress state]

Loading

Decision needed

Question	Recommendation
Should CodexBar adopt this PR’s queue-only ownership model for persisted Cost Usage inventory and cache-wide migration continuation?	Approve the queue-owned model: Accept the bounded queue as the owner of migration continuation, then rebase and verify the current merge result before landing.

Why: The repository owner explicitly states that this data-ownership design remains intentionally unmerged pending owner sign-off; automated review cannot choose the permanent persisted-state contract.

Before merge

• Resolve merge risk (P1) - Existing Cost Usage databases may retain or discard completion state incorrectly across the new inventory, cursor, and compatible-parser-hash migration paths.
• Resolve merge risk (P1) - A lifecycle error in the new queued-restart behavior could leave long catch-up work without a worker or restart it after a user stops it.
• Resolve merge risk (P1) - The current PR is conflict-marked (mergeableState: dirty), so a refreshed merge result is required before landing.
• Complete next step (P2) - Owner approval of the persisted-state ownership model and a clean rebased merge result are required before this PR can land.

Agent review details

Security

None.

Review metrics

Metric	Value	Why it matters
Patch surface	21 files, +4,058 / -183 lines	The change crosses scanner, persistence, orchestration, and substantial regression coverage.
Persisted-state additions	1 inventory field and 7 optional lookback-state fields	These fields make upgrade compatibility and state ownership central merge concerns.

Merge-risk options

Maintainer options:

1. Rebase and prove both store paths (recommended)
   Resolve the dirty merge state and run focused fresh-cache and v0.49.2-upgrade catch-up validation on the rebased head before merge.
2. Accept the migration risk
   Approve the new persisted queue and inventory semantics based on the existing real-history and regression evidence.
3. Pause for a smaller redesign
   Defer this branch if maintainers do not want the queue-only persisted-state ownership model.

Technical review

Best possible solution:

Obtain owner approval for the queue-owned persisted migration model, rebase the branch onto current main, and validate both a fresh history and a v0.49.2-to-current cache upgrade before merge.

Do we have a high-confidence way to reproduce the issue?

Yes, source-level regression paths and a redacted multi-gigabyte real-history log are supplied, but this reviewer could not execute local inspection because the sandbox command startup failed.

Is this the best way to solve the issue?

Unclear: the bounded scanning approach is supported by targeted proof, but the owner must decide whether its queue-only persisted migration ownership model is the desired permanent contract.

AGENTS.md: found and applied where relevant.

Codex review notes: model internal, reasoning high; reviewed against daf02ae7a7a4.

Labels

Label justifications:

• P1: Large existing Codex histories can otherwise stall or fail to publish Usage and Spend data.
• merge-risk: 🚨 compatibility: The PR changes persisted Cost Usage cache fields and accepts a released parser hash without rebuilding completed history.
• merge-risk: 🚨 availability: The scanner work budget and queued worker restarts determine whether long-running catch-up finishes or stops correctly.
• rating: 🐚 platinum hermit: Overall readiness is 🐚 platinum hermit; proof is 🦞 diamond lobster and patch quality is 🐚 platinum hermit.
• status: 👀 ready for maintainer look: ClawSweeper has no concrete contributor-facing blocker left for this PR. Sufficient (logs): Redacted after-fix logs and persisted-store results from a real multi-gigabyte history show relaunch reuse, appended-history processing, and a completed catch-up state; private paths and values were redacted.
• proof: sufficient: Contributor real behavior proof is sufficient. Redacted after-fix logs and persisted-store results from a real multi-gigabyte history show relaunch reuse, appended-history processing, and a completed catch-up state; private paths and values were redacted.

Evidence

What I checked:

• Sandbox limitation: The attempted read-only repository inspection failed before command execution with bwrap: loopback: Failed RTM_NEWADDR: Operation not permitted; no local source, git history, or test result is asserted from this review.
• Maintainer-owned design decision: The repository owner states that the branch is intentionally unmerged pending owner sign-off on persisted inventory/storage ownership after adopting a queue-only migration continuation design. (Sources/CodexBarCore/Vendored/CostUsage/CostUsageStore+CodexCache.swift:244, 11b9cd7d9847)
• Persisted cache compatibility surface: The patch adds persisted inventory and lookback cursor fields and whitelists the released v0.49.2 parser hash as a compatible upgrade, so it affects existing Cost Usage stores rather than only in-memory scheduling. (Sources/CodexBarCore/Vendored/CostUsage/CostUsageCacheModels.swift:18, 01f72618cd1c)
• After-fix real behavior proof: The contributor supplied redacted logs from a multi-gigabyte local history showing successful relaunch reuse, later appended-history processing, and a persisted store with zero pending catch-up and equal processed/total bytes and files. (01f72618cd1c)

Likely related people:

• steipete: Owner-authored commits on the branch cover bounded queue ownership and finalization, and the owner explicitly reserved persisted inventory/storage ownership for sign-off. (role: recent area contributor and decision owner; confidence: high; commits: 80ab22d76092, 26a07bb64369, 087ca11c2fc5; files: Sources/CodexBarCore/Vendored/CostUsage/CostUsageScanner.swift, Sources/CodexBarCore/Vendored/CostUsage/CostUsageStore+CodexCache.swift)
• Quicksaver: The branch’s core bounded-scan, cache-upgrade, and regression-test commits were authored by Quicksaver. (role: recent area contributor; confidence: high; commits: f50968f92a72, 5cdc0171ea0c, 6cf344300ed3; files: Sources/CodexBarCore/Vendored/CostUsage/CostUsageScanner.swift, Tests/CodexBarTests/CostUsageBoundedProgressTests.swift)

Rank-up moves

Optional improvements that raise the rating; they are not merge blockers.

• Resolve the dirty merge state against current main.
• Obtain owner sign-off on persisted inventory and migration ownership.
• Run fresh-cache and v0.49.2-upgrade validation on the rebased head.

Rating scale

Score	Internal tier	Crab rank	Meaning
6/6	S	🦀 challenger crab	Exceptional readiness
5/6	A	🦞 diamond lobster	Very strong readiness
4/6	B	🐚 platinum hermit	Good normal PR; ordinary maintainer review
3/6	C	🦐 gold shrimp	Useful, but confidence is limited
2/6	D	🦪 silver shellfish	Proof or implementation needs work
1/6	F	🧂 unranked krab	Not merge-ready
N/A	NA	🌊 off-meta tidepool	Rating does not apply

Overall follows the weaker of proof and patch quality.
Shiny media proof means a screenshot, video, or linked artifact directly shows the changed behavior. Runtime, network, CSP, and security claims still need visible diagnostics.

Workflow

• ClawSweeper keeps one durable marker-backed review comment per issue or PR.
• Re-runs edit this comment so the latest verdict, findings, and automation markers stay together instead of adding duplicate bot comments.
• A fresh review can be triggered by eligible @clawsweeper re-review comments, exact-item GitHub events, scheduled/background review runs, or manual workflow dispatch.
• PR/issue authors and users with repository write access can comment @clawsweeper re-review or @clawsweeper re-run on an open PR or issue to request a fresh review only.
• Maintainers can also comment @clawsweeper review to request a fresh review only.
• Fresh-review commands do not start repair, autofix, rebase, CI repair, or automerge.
• Maintainer-only repair and merge flows require explicit commands such as @clawsweeper autofix, @clawsweeper automerge, @clawsweeper fix ci, or @clawsweeper address review.
• Maintainers can comment @clawsweeper explain to ask for more context, or @clawsweeper stop to stop active automation.

History

Review history (10 earlier review cycles; latest 8 shown)

• reviewed 2026-08-10T21:12:48.649Z sha a1aab85 :: needs changes before merge. :: [P2] Bound candidate selection before applying the cap
• reviewed 2026-08-10T21:55:51.930Z sha 3f7c44d :: needs changes before merge. :: [P2] Bound persisted lookback reconciliation
• reviewed 2026-08-10T22:15:59.809Z sha 295987c :: found issues before merge. :: [P2] Keep post-scan accounting inside the bounded pass
• reviewed 2026-08-10T22:39:05.157Z sha 40ff7ee :: needs real behavior proof before merge. :: [P2] Bound retained-lookback preparation before selection
• reviewed 2026-08-11T08:16:46.797Z sha 31c1c2b :: needs changes before merge. :: [P2] Bound discovery before building the candidate corpus
• reviewed 2026-08-11T08:30:31.564Z sha 11b9cd7 :: found issues before merge. :: [P2] Bound discovery before constructing the candidate corpus
• reviewed 2026-08-11T10:40:18.510Z sha 5cdc017 :: needs real behavior proof before merge. :: [P1] Preserve existing Cost Usage stores across upgrade | [P2] Bound identity-drift validation during cache load
• reviewed 2026-08-11T13:01:34.000Z sha e50a58a :: needs maintainer review before merge. :: none

[chatgpt-codex-connector]

💡 Codex Review

Here are some automated review suggestions for this pull request.

Reviewed commit: 8a976c44e4

ℹ️ About Codex in GitHub

Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you

• Open a pull request for review
• Mark a draft as ready
• Comment "@codex review".

If Codex has suggestions, it will comment; otherwise it will react with 👍.

Codex can also answer questions or update the PR. Try commenting "@codex address that feedback".

[Quicksaver]

Responding to the request for inspectable real-behavior proof: the PR body now contains this same redacted artifact under Proof.

Redacted real-history debug log and persisted-store snapshot

Provider/account details, home paths, and spend values were removed. No conversation content was read or included.

[2026-08-10T18:54:00.362Z] [INFO] CodexBar startup [provider-state details redacted; Codex enabled]
[2026-08-10T18:54:31.626Z] [INFO] Codex cost scan applied work limits bytesConsumed=790493 deferredByBudget=0 deferredByTime=1 maxBytesPerRefresh=536870912 maxFileBytes=268435456 partialFiles=0
[2026-08-10T18:54:48.139Z] [INFO] cost usage success provider=codex duration=34.31s today=[REDACTED_SPEND] historyDays=30 windowCost=[REDACTED_SPEND]

[2026-08-10T18:57:02.992Z] [INFO] CodexBar startup [provider-state details redacted; Codex enabled]
[2026-08-10T18:57:27.755Z] [INFO] Codex cost scan applied work limits bytesConsumed=404384 deferredByBudget=0 deferredByTime=1 maxBytesPerRefresh=536870912 maxFileBytes=268435456 partialFiles=0
[2026-08-10T18:57:44.201Z] [INFO] cost usage success provider=codex duration=34.81s today=[REDACTED_SPEND] historyDays=30 windowCost=[REDACTED_SPEND]

[2026-08-10T19:57:26.551Z] [INFO] Codex cost scan applied work limits bytesConsumed=4600693 deferredByBudget=0 deferredByTime=1 maxBytesPerRefresh=536870912 maxFileBytes=268435456 partialFiles=0
[2026-08-10T19:57:43.578Z] [INFO] cost usage success provider=codex duration=31.53s today=[REDACTED_SPEND] historyDays=30 windowCost=[REDACTED_SPEND]

# Read-only query of the persisted cost-usage store after the final line above:
catch_up_pending=0
processed_bytes=6887714668
total_bytes=6887714668
completed_files=10949
total_files=10949
last_scan_utc=2026-08-10 19:57:12

The two startup markers show relaunches reusing the existing multi-gigabyte cache: only 790,493 bytes and 404,384 bytes were consumed before successful Codex publication, rather than replaying the corpus. The later 4,600,693-byte delta shows appended history being incorporated. The persisted store then records equal processed/total bytes and completed/total files with no pending catch-up.

The two line-review findings were also fixed on top of merged upstream behavior:

• Completion now persists and validates the exact current scan inventory, so wider retained files cannot falsely satisfy completion; a regression removes a current file while retaining a complete wider file and verifies catch-up remains pending.
• Spend Dashboard Stop clears a queued restart before returning from an active pass; a regression covers that cancellation race.

@clawsweeper re-review

[clawsweeper]

🦞🧹
ClawSweeper re-review requested.

I asked ClawSweeper to review this item again.
Action: item re-review queued (workflow sweep.yml, event repository_dispatch).
Result: when the review finishes, ClawSweeper will create the durable review comment if needed or update the existing comment in place.

Re-review progress:

• State: Complete
• Detail: The durable review result and its route handoff completed.
• Run: https://github.com/openclaw/clawsweeper/actions/runs/31428524762
• Updated: 2026-08-10T20:20:36.555Z

[Quicksaver]

Resolved the updated ClawSweeper finding in a1aab85fa.

The primary stopCodexCostCatchUp() path now clears codexCostCatchUpRestartRequested before returning from an active pass, so deferred teardown cannot launch a replacement worker after Stop. I added the symmetric primary stopping an active pass clears a queued restart regression.

Validation after merging latest upstream/main:

• UsageStoreCodexCostCatchUpTests: 6 passed
• UsageStoreSpendDashboardCodexCostCatchUpTests: 6 passed
• CostUsageCatchUpCompletionTests: 2 passed
• ProviderArchitectureGatekeeperTests: 38 passed
• make check: passed with zero lint violations

@clawsweeper re-review

[clawsweeper]

🦞🧹
ClawSweeper re-review requested.

I asked ClawSweeper to review this item again.
Action: item re-review queued (workflow sweep.yml, event repository_dispatch).
Result: when the review finishes, ClawSweeper will create the durable review comment if needed or update the existing comment in place.

Re-review progress:

• State: Complete
• Detail: The durable review result and its route handoff completed.
• Run: https://github.com/openclaw/clawsweeper/actions/runs/31433905955
• Updated: 2026-08-10T21:27:23.270Z

[Quicksaver]

Fixed the updated candidate-selection finding in 3f7c44d28.

The bounded path now stops after inspecting 512 unique candidate path keys, before cache-eligibility checks. Pending lookback paths are consumed directly, and the selector reuses the URL identity already established during discovery so cache accumulation remains stable across passes.

Concrete 1,500-file regression proof from CostUsagePerformanceGateTests:

[candidate-selection-proof] first=512, second=1024, overlap=512, pending=476, visits=512, attempts=512

This proves that the second pass inspected only 512 candidates (not the full retained queue), scanned 512 files, retained all 512 first-pass cache entries, and advanced the cache to 1,024 files with 476 remaining.

Validation:

• CostUsagePerformanceGateTests: 27/27 passed
• CostUsageCatchUpCompletionTests: 2/2 passed
• UsageStoreCodexCostCatchUpTests: 6/6 passed
• UsageStoreSpendDashboardCodexCostCatchUpTests: 6/6 passed
• make check: passed, zero SwiftLint violations

make test also reached 45 successful groups before the existing locale-sensitive MiniMaxMenuCardBillingTests failure in group 46 and its retry (1,234 expected versus this machine’s locale rendering 1 234); that suite does not touch these scanner changes.

@clawsweeper re-review

[clawsweeper]

🦞🧹
ClawSweeper re-review requested.

I asked ClawSweeper to review this item again.
Action: item re-review queued (workflow sweep.yml, event repository_dispatch).
Result: when the review finishes, ClawSweeper will create the durable review comment if needed or update the existing comment in place.

Re-review progress:

• State: Complete
• Detail: The durable review result and its route handoff completed.
• Run: https://github.com/openclaw/clawsweeper/actions/runs/31436203940
• Updated: 2026-08-10T21:57:50.219Z

[Quicksaver]

Fixed the persisted reconciliation finding in 295987c71.

Changes:

• Each cache load/save now validates at most 512 retained lookback paths
• The reconciliation path no longer builds a full-cache identity index first; it uses bounded direct/normalized path aliases
• Exact current-inventory validation is deferred until catch-up is pending but there is no active lookback, discovery work, or incomplete/buffered file work

Concrete 1,500-file regression output:

[reconcile-proof] save=512 load=512 pending=988
[candidate-selection-proof] first=512, second=1024, overlap=512, pending=476, visits=512, attempts=512

The first line proves that both save and the subsequent load stop after 512 metadata/completion visits even though the retained queue contains 988 paths. The second proves the scanner then remains bounded and advances another 512 files without losing the first-pass cache.

Validation:

• CostUsagePerformanceGateTests: 27/27 passed
• CostUsageCatchUpCompletionTests: 2/2 passed
• make check: passed, zero SwiftLint violations

The full local make test remains blocked only by the unrelated locale-sensitive MiniMaxMenuCardBillingTests expectation documented in the PR body; the branch CI macOS shards exercise the complete suite in their standard environment.

@clawsweeper re-review

[clawsweeper]

🦞🧹
ClawSweeper re-review requested.

I asked ClawSweeper to review this item again.
Action: item re-review queued (workflow sweep.yml, event repository_dispatch).
Result: when the review finishes, ClawSweeper will create the durable review comment if needed or update the existing comment in place.

Re-review progress:

• State: Complete
• Detail: The durable review result and its route handoff completed.
• Run: https://github.com/openclaw/clawsweeper/actions/runs/31438569828
• Updated: 2026-08-10T22:30:44.768Z

[Quicksaver]

Fixed the latest bounded-pass finding in 40ff7ee. Pending catch-up passes now maintain file progress incrementally and deliberately leave exact byte totals/inventory indeterminate, so they no longer sort or stat the full discovered corpus after the 512-file scan. Exact metadata accounting and sorted inventory persistence run only once bounded work is exhausted.

The counted 1,500-file regression proves two consecutive bounded passes advance 512 files each while performing zero whole-corpus progress-accounting visits: first=512, second=1024, overlap=512, pending=476, visits=512, attempts=512, accounting=0. CostUsagePerformanceGateTests (27), CostUsageCatchUpCompletionTests (2), both catch-up orchestration suites (6 + 6), and make check pass. The PR Evidence/Validation sections now include this proof.

@clawsweeper review

[clawsweeper]

🦞🧹
ClawSweeper re-review requested.

I asked ClawSweeper to review this item again.
Action: item re-review queued (workflow sweep.yml, event repository_dispatch).
Result: when the review finishes, ClawSweeper will create the durable review comment if needed or update the existing comment in place.

Re-review progress:

• State: Complete
• Detail: The durable review result and its route handoff completed.
• Run: https://github.com/openclaw/clawsweeper/actions/runs/31439270017
• Updated: 2026-08-10T22:40:41.625Z

[Quicksaver]

Addressed the current durable ClawSweeper requirements at merged head e50a58a25 (6cf344300, 9279695ac; latest upstream/main e5528d452 merged).

• P1 — preserve v0.49.2 stores: added an exact compatible parser-hash/schema upgrade. The focused regression opens a completed v0.49.2 store, retains its files, installs the current schema/hash atomically, and records zero rebuilds.
• P2 — bound identity drift: cache load validates at most 512 drifted identities, persists the remainder in the active lookback queue, carries that queue across scan-window resets, retains missing/changed candidates for scanner handling, and drains only positively validated identity-drift entries. The 600-file regression validates 512 and defers 88.
• Current-head live proof: an isolated copy of the real >12,400-file, ~7.1 GiB history/store was relabeled with the exact v0.49.2 schema/hash and had only the copy's identity metadata changed. Real sessions and the active cache stayed read-only. Paths, identifiers, identities, spend, and content were not printed or retained.

[current-head-live-proof] mode=upgrade-drift-load elapsed_ms=9910 retained_files=10978 load_validation=512 pending=1
[current-head-live-proof] mode=relaunch-catch-up elapsed_ms=32170 retained_files=10978 attempts=512 discovery=512 accounting=0 pending=1 processed_bytes=0 total_bytes=0
[persisted-store] parser_hash=f1d04f67770b64cf schema=current retained_files=10978 pending_paths=9954

The temporary proof harness/database were deleted afterward. The earlier redacted real-history evidence in the PR body still demonstrates normal appended history being incorporated; this new run proves the final merged head's upgrade/relaunch bound.

Validation on the merged head:

• CostUsageCatchUpCompletionTests: 5/5
• CostUsageBoundedProgressTests: 10/10
• CostUsagePerformanceGateTests: 27/27
• CostUsageStoreTests: 61/61
• CostUsageStoreExecutorIsolationTests: 2/2
• make check: pass, current parser hash, formatting clean, 0 lint violations
• Full make test: 45 groups pass; group 46 and its retry fail only the pre-existing locale-sensitive MiniMaxMenuCardBillingTests assertions (1,234/5,678 expected, Lisbon locale renders nonbreaking-space grouping). The other 119 tests in that shard pass.

The prior CI failures were checked before this push: Linux could not resolve DIR, and SwiftLint rejected an over-length test file. Both are fixed in the pushed commits; the old macOS shards were already green.

[Quicksaver]

@clawsweeper re-review

[clawsweeper]

🦞🧹
ClawSweeper re-review requested.

I asked ClawSweeper to review this item again.
Action: item re-review queued (workflow sweep.yml, event repository_dispatch).
Result: when the review finishes, ClawSweeper will create the durable review comment if needed or update the existing comment in place.

Re-review progress:

• State: Complete
• Detail: The durable review result and its route handoff completed.
• Run: https://github.com/openclaw/clawsweeper/actions/runs/31494213170
• Updated: 2026-08-11T13:03:39.747Z

[steipete]

Merged via #2918, which contains every commit of this PR verbatim (your head e50a58a252 is an ancestor of the merged branch), plus rebase corrections against the moved main (restored Equatable for the #2852 fast-save comparison, preserved skipIdenticalContent save semantics, switched to the general parser-hash predecessor set) and the companion fixes for the Finish-now window and semantic progress detection.

Closing this PR since the same commits are now on main — full authorship credit is preserved in history and the changelog. Thank you for the substantial indexing/durability work, @Quicksaver — it's the backbone of the fix.