Drain processed zero-row files from Codex cost catch-up - #3075
Conversation
|
🦞👀 Pull request received. I will update this pull request when review starts. |
|
ClawSweeper status: review started. I am starting a fresh review of this pull request: Drain processed zero-row files from Codex cost catch-up This is item 1/1 in the current shard. Shard 0/1. This placeholder means the worker is alive and reading the current context. I will edit this same comment with the actual review when the claws are done clicking. Crustacean status: shell secured, claws on keyboard, evidence pebbles being sorted. |
![chatgpt-codex-connector[bot]](https://avatars.githubusercontent.com/in/1144995?s=60&v=4){kind=small}
There was a problem hiding this comment.
💡 Codex Review
Here are some automated review suggestions for this pull request.
Reviewed commit: 0690f76c8c
ℹ️ About Codex in GitHub
Codex has been enabled to automatically review pull requests in this repo. Reviews are triggered when you
- Open a pull request for review
- Mark a draft as ready
- Comment "@codex review".
If Codex has suggestions, it will comment; otherwise it will react with 👍.
When you sign up for Codex through ChatGPT, Codex can also answer questions or update the PR, like "@codex address that feedback".
| state: &state, | ||
| maxBytesToRead: fullRescanAllowedBytes) | ||
| context.scanBudget?.consume(workBytes: fullRescanAllowedBytes) | ||
| return .processed |
There was a problem hiding this comment.
When a time- or byte-bounded rescan reads only a prefix of a file whose session already contributed through another rollout, rescanCodexFile can remove the cache entry because the prefix contains no unique rows, even though parsedBytes has not reached the file size. Returning .processed unconditionally here makes the new cacheless-path logic drain that file and exclude it from the exact inventory, so an old touched/resumed session can permanently lose valid in-window usage in its unread suffix. The outcome must remain deferred unless the rescan actually reached EOF or otherwise proved completion.
Useful? React with 👍 / 👎.
1 participant
Summary
Fixes #3071. When an old Codex session file (content days entirely outside the scan window) gained a recent mtime — rsync, backup restore,
touch— the active lookback queued it intopendingFilePathspermanently: queue completion required a cache entry, but zero-in-window-row files intentionally never get one.catchUpPendingstayed true forever, the exact-inventory proof pass never ran, and the UI showed a perpetual "catching up".The scanner now tracks processed-vs-deferred separately: a file that was fully processed drains from the queue even when it contributes zero in-window rows, and cache-less processed paths are excluded from the exact inventory. Deferred (genuinely unprocessed) files stay queued. Existing stuck queues self-heal on the next scan pass — no state reset needed.
Test plan
make checkclean