Skip to content

recurate: a recuration of a span's middle leaves both sides, and the consumers run before a plan (#192) - #208

Merged
ojungo69 merged 4 commits into
mainfrom
fix/recurate-parts
Sep 28, 2026
Merged

ojungo69 merged 4 commits into
mainfrom
fix/recurate-parts

Conversation

@ojungo69

@ojungo69 ojungo69 commented Sep 28, 2026 •

Copy link
Copy Markdown
Owner

Closes #192. From the Codex review of #189 (rounds 7 and 8). In each case nothing was lost: a range already curated again was sent again, which spent a free cap, a subscription window or a paid entry's budget once more.

What

  • A recuration of a queued span's middle leaves the parts on both sides (item 1).
    • recurate rows are now keyed by (op_device, op_seq, from_seq), so one claim op's queued window can be split.
    • The claims consumer subtracts each recuration's window and covers from every queued span it touches. It keeps the parts before and after, and keeps whole a record it curated only in part, as before.
    • queue() still queues an op once: an op already queued, whole or in the parts a recuration left, stays as it is. The old key did the same, and this way a trimmed row and a fresh whole row never overlap in a plan.
    • A knowledge.db built with the old key gets the table made again in one savepoint, with its rows kept (claims::schema).
  • A recuration of a skipped span's middle leaves both sides (item 2). skipped_spans subtracts every later recuration with Span::minus, which gives a part on each side, offsets and all.
  • The consumers run before oboete recurate plans (item 3). A recuration appended before a crash or a failed run_once is read first, so its span is not listed or sent again. This is worker::run_once, the same pass --yes already runs after sending.

Tests

  • a_recuration_of_a_spans_middle_leaves_both_sides:
    • queued 1-3 and skipped 1-3, then a recuration of 2-2: both give 1-1 and 3-3;
    • then a recuration of 3-3 leaves 1-1;
    • queuing the claim op again for a record curated since changes nothing;
    • the next oboete recurate lists one span in one window.
  • a_recuration_the_consumers_have_not_read_is_not_sent_again: a recuration appended with no consumer run afterwards. oboete recurate then lists nothing.
  • the_queue_of_an_older_knowledge_db_is_keyed_again: a table with the old key keeps its row and takes a second part of the same op.
  • Four mutations each turn one of these red:
    • queuing an op again;
    • no drain before the plan;
    • no part before the recuration;
    • no migration.

Review stop rule (declared before the first round)

A finding is fixed here only if a plain environment reaches it (the default or documented config, Linux, macOS or Windows) and it loses or duplicates a recuration. Everything else goes to an issue. After two rounds in a row whose findings all go to issues, this PR is merged with no further commit once check and CodeQL are green.

🤖 Generated with Claude Code

https://claude.ai/code/session_012wbKzQZyG5Aq6poybwciHt

Summary by Sourcery

Preserve recuration span fragments and drain pending consumers before planning to prevent lost or duplicated recuration work.

New Features:

  • Preserve both sides of queued and skipped spans when a recuration affects their middle records.
  • Drain consumers before planning recuration work so already-recorded recurations are not sent again.

Bug Fixes:

  • Prevent recuration windows from being duplicated or lost when spans are split and claim operations are queued again.
  • Migrate existing knowledge databases to the split-capable recuration queue schema without losing queued rows.

Enhancements:

  • Support subtracting arbitrary spans while retaining record boundaries and offsets.

Tests:

  • Add coverage for middle-span splitting, duplicate queueing, pre-plan consumer draining, and legacy database migration.

@chatgpt-codex-connector

Copy link
Copy Markdown

You have reached your Codex usage limits for security reviews. Please try again later.

@sourcery-ai sourcery-ai Bot left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Sorry @ojungo69, you've used your own review budget of 250,000 diff characters for the last 7 days.

You can request another review in 2 hours and 49 minutes by commenting @sourcery-ai review. Upgrade to get a review now.

@coderabbitai

coderabbitai Bot commented Sep 28, 2026 •

Copy link
Copy Markdown

Important

  • 🔍 Trigger review

This repository does not receive automatic reviews because it has fewer than 10 stars.

⚙️ Run configuration

Configuration used: Organization UI

Review profile: ASSERTIVE

Plan: Advanced

Run ID: afca4dc5-8888-40bf-a701-cde6c95f4c75


Thanks for using CodeRabbit! It's free for OSS, and your support helps us grow. If you like it, consider giving us a shout-out.

❤️ Share

Comment @coderabbitai help to get the list of available commands.

@chatgpt-codex-connector

chatgpt-codex-connector Bot commented Sep 28, 2026 •

Copy link
Copy Markdown

Codex Review Summary

This comment shows the latest Codex review activity on this pull request.

Review Status Commit Review trigger
📝 Code Review ✅ Completed 2026-09-28T09:54:05.072503Z 7751ed1 Manual request
ℹ️ About Codex in GitHub

Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you

  • Open a pull request for review
  • Mark a draft as ready
  • Comment "@codex review" or "@codex security review".

Codex reacts with 👀 while any review is running, comments if it has suggestions, and reacts with 👍 once all reviews finish with no findings.

@sourcery-ai

sourcery-ai Bot commented Sep 28, 2026

Copy link
Copy Markdown

Reviewer's Guide

Updates recuration tracking to preserve disjoint portions of queued and skipped spans, deduplicate claim operations, migrate legacy queue schemas, and run consumers before planning so pending recuration events cannot be sent twice.

Sequence diagram for recuration planning after consumer drain

sequenceDiagram
    participant User
    participant Recurate as curate::recurate
    participant Worker as worker::run_once
    participant Claims as Claims consumer
    participant Planner as Recuration planner

    User->>Recurate: recurate()
    Recurate->>Worker: run_once()
    Worker->>Claims: consume pending recuration
    Claims->>Claims: subtract curated span from queued spans
    Worker-->>Recurate: consumers drained
    Recurate->>Planner: plan remaining spans
    Planner-->>User: list or send recuration windows
Loading

Entity relationship diagram for recuration queue keys

erDiagram
    RECURATE {
        string device
        int from_seq
        int to_seq
        string op_device
        int op_seq
    }
    CLAIM_OPERATION {
        string op_device
        int op_seq
    }
    CLAIM_OPERATION ||--o{ RECURATE : queues
Loading

Flow diagram for splitting queued and skipped spans

flowchart LR
    Original[Queued or skipped span 1-3] --> Recuration[Recuration of middle 2-2]
    Recuration --> Before[Remaining part 1-1]
    Recuration --> After[Remaining part 3-3]
    Before --> Next[Plan only remaining spans]
    After --> Next
Loading

File-Level Changes

Change Details Files
Split queued recuration windows around each newly processed recuration while preserving op-level deduplication.
  • Key recuration rows by claim operation and starting sequence so one operation can occupy multiple disjoint ranges.
  • Subtract each recuration span and its covered ranges from overlapping queued spans, inserting the remaining left and right parts.
  • Prevent re-queuing an operation if any range for that operation already exists.
  • Rebuild the legacy table key inside a savepoint while preserving existing rows.
src/claims.rs
src/consumer/claims.rs
Apply middle-span subtraction consistently to skipped recuration candidates.
  • Add Span::minus to retain both non-overlapping portions, including offset-aware boundaries.
  • Fold all later recurations over skipped spans and return every surviving part.
src/curate.rs
Drain consumers before constructing a recuration plan.
  • Run worker::run_once before loading and planning queued recurations so appended-but-unconsumed recuration events are reflected.
  • Expose shared span and queue helpers needed by the consumer and planning flow.
src/curate.rs
src/consumer/claims.rs
Add regression coverage for splitting, deduplication, migration, and consumer ordering.
  • Verify queued and skipped spans retain both sides of a middle recuration and shrink correctly at an end.
  • Verify re-queuing a curated operation does not create another entry.
  • Verify unconsumed recuration events are not sent again.
  • Verify legacy databases migrate without losing rows.
src/claims.rs
src/consumer/claims.rs
src/curate.rs

Assessment against linked issues

Issue Objective Addressed Explanation
#192 When an explicit recuration covers the middle of a queued span, subtract the recurated range while preserving the untouched portions on both sides, including record-granular queue behavior. ✅
#192 When an explicit recuration covers the middle of a skipped span, subtract the recurated range while returning the remaining portions on both sides from skipped-span queries. ✅

Possibly linked issues


Tips and commands

Interacting with Sourcery

  • Trigger a new review: Comment @sourcery-ai review on the pull request.
  • Continue discussions: Reply directly to Sourcery's review comments.
  • Generate a GitHub issue from a review comment: Ask Sourcery to create an
    issue from a review comment by replying to it. You can also reply to a
    review comment with @sourcery-ai issue to create an issue from it.
  • Generate a pull request title: Write @sourcery-ai anywhere in the pull
    request title to generate a title at any time. You can also comment
    @sourcery-ai title on the pull request to (re-)generate the title at any time.
  • Generate a pull request summary: Write @sourcery-ai summary anywhere in
    the pull request body to generate a PR summary at any time exactly where you
    want it. You can also comment @sourcery-ai summary on the pull request to
    (re-)generate the summary at any time.
  • Generate reviewer's guide: Comment @sourcery-ai guide on the pull
    request to (re-)generate the reviewer's guide at any time.
  • Resolve all Sourcery comments: Comment @sourcery-ai resolve on the
    pull request to resolve all Sourcery comments. Useful if you've already
    addressed all the comments and don't want to see them anymore.
  • Dismiss all Sourcery reviews: Comment @sourcery-ai dismiss on the pull
    request to dismiss all existing Sourcery reviews. Especially useful if you
    want to start fresh with a new review - don't forget to comment
    @sourcery-ai review to trigger a new review!

Customizing Your Experience

Access your dashboard to:

  • Enable or disable review features such as the Sourcery-generated pull request
    summary, the reviewer's guide, and others.
  • Change the review language.
  • Add, remove or edit custom review instructions.
  • Adjust other review settings.

Getting Help

@codacy-production

codacy-production Bot commented Sep 28, 2026 •

Copy link
Copy Markdown

Up to standards ✅

🟢 Issues 0 issues

Results:
0 new issues

View in Codacy

NEW Get contextual insights on your PRs based on Codacy's metrics, along with PR and Jira context, without leaving GitHub. Enable AI reviewer
TIP This summary will be updated as you push new changes.

Comment thread src/curate.rs Fixed

@chatgpt-codex-connector chatgpt-codex-connector Bot left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

💡 Codex Review

Here are some automated review suggestions for this pull request.

Reviewed commit: 00f6ec0502

ℹ️ About Codex in GitHub

Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you

  • Open a pull request for review
  • Mark a draft as ready
  • Comment "@codex review".

If Codex has suggestions, it will comment; otherwise it will react with 👍.

Codex can also answer questions or update the PR. Try commenting "@codex address that feedback".

Comment thread src/curate.rs Outdated
ojungo69 added a commit that referenced this pull request Sep 28, 2026
…r it, and held through the plan and its sending (review on #208)

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_012wbKzQZyG5Aq6poybwciHt
@chatgpt-codex-connector

Copy link
Copy Markdown

You have reached your Codex usage limits for security reviews. Please try again later.

@chatgpt-codex-connector chatgpt-codex-connector Bot left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

💡 Codex Review

Here are some automated review suggestions for this pull request.

Reviewed commit: 017c94f561

ℹ️ About Codex in GitHub

Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you

  • Open a pull request for review
  • Mark a draft as ready
  • Comment "@codex review".

If Codex has suggestions, it will comment; otherwise it will react with 👍.

Codex can also answer questions or update the PR. Try commenting "@codex address that feedback".

Comment thread src/curate.rs
ojungo69 added a commit that referenced this pull request Sep 28, 2026
… run once more whether it sent or not (review on #208)

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_012wbKzQZyG5Aq6poybwciHt
@chatgpt-codex-connector

Copy link
Copy Markdown

You have reached your Codex usage limits for security reviews. Please try again later.

ojungo69 and others added 4 commits September 28, 2026 18:46
…des, and the consumers run before a plan (#192)

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_012wbKzQZyG5Aq6poybwciHt
…r it, and held through the plan and its sending (review on #208)

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_012wbKzQZyG5Aq6poybwciHt
… run once more whether it sent or not (review on #208)

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_012wbKzQZyG5Aq6poybwciHt
@ojungo69

Copy link
Copy Markdown
Owner Author

@codex review

@chatgpt-codex-connector

Copy link
Copy Markdown

You have reached your Codex usage limits for security reviews. Please try again later.

1 similar comment
@chatgpt-codex-connector

Copy link
Copy Markdown

You have reached your Codex usage limits for security reviews. Please try again later.

@sonarqubecloud

Copy link
Copy Markdown

@chatgpt-codex-connector

Copy link
Copy Markdown

Codex Review: Didn't find any major issues. What shall we delve into next?

Reviewed commit: 7751ed1275

ℹ️ About Codex in GitHub

Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you

  • Open a pull request for review
  • Mark a draft as ready
  • Comment "@codex review".

If Codex has suggestions, it will comment; otherwise it will react with 👍.

Codex can also answer questions or update the PR. Try commenting "@codex address that feedback".

@ojungo69
ojungo69 merged commit 2ca3bf2 into main Sep 28, 2026
16 checks passed
@ojungo69
ojungo69 deleted the fix/recurate-parts branch September 28, 2026 09:55
ojungo69 added a commit that referenced this pull request Sep 28, 2026
The owner asked for this (2026-09-28): the open-code-review workflow's
model effort goes from `max` to `high`.

## What

`llm_reasoning_effort: max` becomes `high` in both review steps, the
primary model and the fallback. `effort: high`, the action's own review
depth, was already `high` and is unchanged. Nothing else changes.

## Note

The review job failed on the recent PRs (#205, #206 and #208: "The
primary review failed and no fallback review succeeded"). Whether `max`
caused that has not been checked. The workflow runs from `main`
(`pull_request_target`), so this PR's own run still uses `max`. The new
value applies to runs after the merge.

## Review stop rule (declared before the first round)

A finding is fixed here only if it shows the workflow would fail, or
would run with weaker permissions or trust than today. Everything else
goes to an issue. After two rounds in a row whose findings all go to
issues, or a 👍, this PR is merged with no further commit once `check`
and CodeQL are green.

🤖 Generated with [Claude Code](https://claude.com/claude-code)

https://claude.ai/code/session_012wbKzQZyG5Aq6poybwciHt

## Summary by Sourcery

Set the open-code-review workflow to use high reasoning effort for both
configured models.

Enhancements:
- Reduce the model reasoning effort used by both the primary and
fallback open-code-review models from maximum to high while retaining
the workflow’s high review depth.

Documentation:
- Update open-code-review documentation to reflect the high model
reasoning effort and revised timeout rationale.

---------

Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
ojungo69 added a commit that referenced this pull request Sep 28, 2026
…216)

Closes #213.

Two knowledge.db migrations in `claims::schema` checked the old schema
and then changed it with no lock between the two. A worker and a command
(`oboete recurate`, `oboete search`) opening the same older knowledge.db
at once could both find the old schema; the second then failed on
"duplicate column name: claim_at", or rebuilt the queue table a second
time.

## What

- `claims::migrate(k, needed, change)`: reads `needed` first, so a
current knowledge.db takes no write lock. When a change is needed:
- outside a transaction, it takes `BEGIN IMMEDIATE`, asks `needed`
again, and changes only if it still holds;
- inside one (a consumer's step or rewind, both immediate), the lock is
already held, so it changes at once, as before.
- The `claim_at` column (#210) and the queue's key rebuild (#208) both
go through it.
- `db::ensure_column`, which the issue named, takes `&mut Connection`;
`schema` is called with the step's transaction (`&Connection`), so the
same check-again shape is written here with
`Transaction::new_unchecked`.

## Test

`two_connections_migrate_one_older_knowledge_db_at_once`: for an older
evidence table, then an older queue table, a first connection holds the
write lock (as a step does) while a second runs `schema`; the first then
migrates and commits. Both succeed, the column is there once, the queue
has its three-column key, and the rows are kept. On `main`'s `claims.rs`
it fails 3 runs in 3 with "duplicate column name: claim_at".

## Review stop rule (declared before the first round)

A finding is fixed here only if a plain environment reaches it (a
worker, a hook and the CLI commands on one home; Linux, macOS or
Windows) and it loses a row, stops a worker run, or leaves the schema
half changed. Anything else goes to an issue. After two rounds in a row
whose findings all go to issues, this PR is merged with no further
commit once `check` and CodeQL are green.

🤖 Generated with [Claude Code](https://claude.com/claude-code)

https://claude.ai/code/session_012wbKzQZyG5Aq6poybwciHt

## Summary by Sourcery

Make knowledge.db schema migrations safe when workers and commands open
an older database concurrently.

Bug Fixes:
- Prevent concurrent knowledge.db migrations from applying schema
changes more than once or failing with duplicate-column errors.

Enhancements:
- Centralize migration checks so current databases avoid write locks
while outdated databases recheck their schema under an immediate
transaction.
- Apply lock-safe migration handling to the claim_at column and recurate
queue key migrations.

Tests:
- Add coverage for concurrent migrations of older evidence and queue
schemas, verifying successful upgrades and preservation of rows.

Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

recurate: a recuration of a span's middle leaves the whole span queued and skipped (from #189)

2 participants