feat(wal,ingester,server): segment rotation + rotation-triggered snapshot cadence (RFC0008.6) by jensholdgaard · Pull Request #188 · jensholdgaard/ourios

jensholdgaard · 2026-06-12T20:42:03Z

What

Segment rotation (RFC0008.6) plus the §6.9 rotation-triggered snapshot cadence it was blocking.

ourios-wal — rotation in append, before the write:

Size cap: the check includes the frame about to land, so a frame never straddles segments (the §6.9 lower bound guarantees any legal frame fits a fresh segment).
Age cap: from the segment UUIDv7's embedded mint time — §6.5's "since its header was written", surviving reopen with no extra persisted state. An empty segment never age-rotates (no recovery window to bound, only file churn).
Sequence: fdatasync(old) → create fresh segment + 24 B header → fsync(parent dir) — the new entry is durable before any frame lands in it, and a closed segment can never carry a torn tail (recovery treats that as RFC0008.5 corruption).
Quiesce: any rotation step failing sets a quiesce flag — the triggering append surfaces the underlying Io error, every later append returns QuiescedAfterRotationFsyncFailure until an operator intervenes (a fresh open). sync stays available: the old segment is still the append target, and acking frames already written to it is safe.

Rotation cadence (ourios-ingester / ourios-server):

IngestPipeline::with_rotation_hook: fired once per detected segment change with the miner as it stands and the rotation-point high-water mark (the old segment's last durable offset), before the rotating batch's records reach the miner — so a snapshot taken by the hook reflects exactly the frames ≤ the mark.
serve() wires the hook to the per-tenant snapshot writer (best-effort, like the post-recovery and shutdown cadence points). This activates the §6.9 primary cadence and completes the deferral noted in feat(miner,ingester,server): snapshot restore v2 — per-tenant restore + startup recovery driver #187.

Tests

All three RFC0008.6 arms live: size-cap (16 MiB + 2 MiB at the 17 MiB minimum; closed segment ends exactly at the first frame's offset; replay sees both frames once, in order), time-cap (1 s minimum age via real sleep; pins the empty-segment no-rotate rule), and quiesce (#[cfg(unix)] read-only-root injection; quiesce persists after the condition clears; sync stays Ok; no half-created segment).
RFC0008.10 snapshot-cadence arm: a real WAL driven across its age cap through the live pipeline — the artefact is stamped with the old segment's offset and reflects only the pre-rotation batch; a steady-state control proves the hook never fires without a rotation.
Colocated pipeline unit test pins the hook contract (fires once, with the old segment's last durable offset, before the batch reaches the miner).

Invariants / hazards

§3.4 WAL-before-ack / H3: the ack path is unchanged — rotation adds no fsync between append and ack in the steady state (the rotation's fdatasync replaces, not supplements, the close-time obligation, and happens at most once per segment). The quiesce rule is §3.4's loud-failure side: continuing after a failed rotation could put a torn tail on a closed segment (recovery halt) or a frame in a segment without a durable directory entry (silent loss on power cut).
Hazard docs(rfc-0001): fill in drafted-bar content for the template miner #5 / §6.9 cadence: the hook ordering (before the rotating batch reaches the miner) is what makes the stamped high-water mark honest — the snapshot never claims coverage of records it hasn't folded.
wal_unflushed_bytes resets on the rotation's closing fdatasync, keeping the RFC0008.9 bound achievable.

Checks run

cargo fmt --all --check, cargo clippy --all-targets --all-features -- -D warnings, cargo test --all-features (full workspace, zero failures), cargo doc --no-deps --all-features, cargo bench --no-run — all green locally.

RFC 0008 remaining red after this: .1 (wal-before-ack), .3 (recovery O(n)), .4/.5 (torn-write/corruption arms), .8 (batched fsync), .9 (unflushed bound).

🤖 Generated with Claude Code

Summary by CodeRabbit

Release Notes

New Features
- Added automatic snapshot generation at WAL segment rotation points to improve recovery efficiency and establish durability checkpoints.
- Implemented rotation failure safeguards that prevent data corruption by refusing subsequent operations when rotation encounters critical errors.
Tests
- Added comprehensive rotation scenario tests covering size-based rotation, time-based rotation, and failure recovery behavior.

…esce on failure (RFC0008.6) Rotation happens in append, before the write, so a frame never straddles segments: the size check includes the frame about to land (the §6.9 lower bound guarantees any legal frame fits a fresh segment), and the age comes from the segment UUIDv7's embedded mint time — §6.5's 'since its header was written', surviving reopen with no extra persisted state. An empty segment never age-rotates. The rotation sequence is fdatasync(old) -> create fresh segment + header -> fsync(parent dir), so the new entry is durable before any frame lands in it and a closed segment can never carry a torn tail (which recovery treats as RFC0008.5 corruption). Any step failing quiesces the WAL: the triggering append surfaces the IO error, every later append returns QuiescedAfterRotationFsyncFailure until an operator intervenes; sync stays available — acking frames already in the old segment is safe. All three RFC0008.6 arms are live. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>

…§6.9 / RFC0008.10 arm) The pipeline detects a WAL segment change between consecutive durable offsets and fires an installable rotation hook with the rotation-point high-water mark — the old segment's last durable offset — BEFORE the rotating batch's records reach the miner, so a snapshot taken by the hook reflects exactly the frames at or below the mark. serve() wires the hook to the per-tenant snapshot writer (best-effort, like the other cadence points). This activates §6.9's primary cadence, previously blocked on rotation itself; the RFC0008.10 snapshot-cadence arm is live (real WAL driven across its age cap; artefact stamped with the old segment's offset and reflecting only the pre-rotation batch). Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>

coderabbitai · 2026-06-12T20:42:12Z

Warning

Review limit reached

@jensholdgaard, we couldn't start this review because you've reached your PR review rate limit.

More reviews will be available in 36 minutes and 54 seconds. Learn how PR review limits work.

Your organization has run out of usage credits. Purchase more credits in the billing tab to continue.

⌛ How to resolve this issue?

After more reviews become available, a review can be triggered using the @coderabbitai review command as a PR comment. Alternatively, push new commits to this PR.

We recommend that you space out your commits to avoid hitting the rate limit.

🚦 How do rate limits work?

CodeRabbit enforces hourly rate limits for each developer per organization.

Our paid plans include higher PR review limits than trial, open-source, and free plans. In all cases, reviews become available again over time. During sustained high-volume PR review activity, CodeRabbit may temporarily slow when the next review becomes available.

Please see our Fair Usage Limits Policy for further information.

ℹ️ Review info

⚙️ Run configuration

Configuration used: defaults

Review profile: CHILL

Plan: Pro Plus

Run ID: 387414b0-4abc-4e09-97e9-0b1deb3c69f5

📥 Commits

Reviewing files that changed from the base of the PR and between 1201db2 and c39d9bc.

📒 Files selected for processing (4)

crates/ourios-ingester/src/receiver/pipeline.rs
crates/ourios-ingester/tests/rfc0008_10_rotation_cadence.rs
crates/ourios-wal/src/lib.rs
crates/ourios-wal/tests/rfc0008_6_rotation.rs

📝 Walkthrough

Walkthrough

This PR implements WAL segment rotation mechanics with a rotation-cadence hook that fires when durable segments change. The WAL gains quiesced-on-failure state, pre-write rotation ordering, and age-based rotation. The ingester pipeline detects rotation via durable offset changes and fires an installed callback before handing records to the miner. The server wires the hook to persist per-tenant snapshots at rotation boundaries.

Changes

WAL Rotation and Rotation-Cadence Snapshots

Layer / File(s)	Summary
WAL rotation-failure quiescing `crates/ourios-wal/src/lib.rs`	Adds `quiesced` flag to `Wal` initialized to `false` on open; set to `true` on any rotation failure (segment close, header flush, or parent-dir fsync); early check in `append` returns `QuiescedAfterRotationFailure` instead of continuing.
WAL pre-write rotation ordering and age calculation `crates/ourios-wal/src/lib.rs`	Refactors rotation evaluation to occur before write using computed `frame_len`; adds `segment_age_secs` helper deriving age from UUIDv7 timestamps; updates rotation-due logic to consider both size and age before write.
WAL rotation execution and error handling `crates/ourios-wal/src/lib.rs`	Reworks `rotate()` to set `quiesced = true` on any failure and reset on success; resets `unflushed_bytes` after closing; replaces `AppendError::QuiescedAfterRotationFsyncFailure` with `QuiescedAfterRotationFailure` and updates display and source implementations.
WAL rotation integration tests `crates/ourios-wal/tests/rfc0008_6_rotation.rs`	Adds test helpers for WAL config, segment enumeration, and replay capture; implements size-cap rotation test verifying no frame drop/duplication, time-cap rotation test validating age-based triggers, and Unix rotation-failure quiesce test confirming `AppendError::Io` → `QuiescedAfterRotationFailure` sequence and read-only root behavior.
WAL crate documentation `crates/ourios-wal/src/lib.rs`	Updates crate-level docs to reflect implemented rotation features and refines `current_segment_path` field docs for housekeeping rename-guard behavior.
Rotation hook type and builder method `crates/ourios-ingester/src/receiver/pipeline.rs`	Defines `RotationHook` type as `FnMut(&MinerCluster, WalOffset) + Send`; adds `with_rotation_hook` builder method with documentation specifying callback fires before rotated records reach the miner using previous segment's durable offset.
Pipeline struct and ingest rotation detection `crates/ourios-ingester/src/receiver/pipeline.rs`	Adds optional `rotation_hook` field to `IngestPipeline`; updates `ingest` to capture previous durable offset, sync the batch, compute new durable offset, detect segment change (previous and new offsets exist and differ), fire hook with miner and previous offset before ingesting records into miner.
Ingester rotation-cadence tests `crates/ourios-ingester/src/receiver/pipeline.rs`, `crates/ourios-ingester/tests/rfc0008_10_rotation_cadence.rs`	Adds `SequenceJournal` for scripted offsets and unit test verifying hook fires once with prior-segment offset; includes `rotating_pipeline` helper, end-to-end test validating snapshot writes with correct `wal_high_water` at rotation point, and no-rotation test confirming zero hook firings when age cap is not crossed.
Server-side rotation hook wiring `crates/ourios-server/src/receiver.rs`	Updates `serve()` to register rotation hook that clones `snapshots_root`, writes per-tenant snapshots at rotation-point high-water mark via `recovery::write_snapshots`, and logs failures to stderr without propagating errors.

Estimated code review effort

🎯 4 (Complex) | ⏱️ ~60 minutes

Possibly related issues

jensholdgaard/ourios#80: Directly addresses the RFC0008 WAL segment rotation slice by implementing rotation mechanics, quiesced-on-failure state, rotation sequencing, and the ingest pipeline rotation-cadence hook.

Possibly related PRs

jensholdgaard/ourios#134: Both PRs modify IngestPipeline::ingest flow in crates/ourios-ingester/src/receiver/pipeline.rs; #134 introduces the WAL-before-ack append+sync-then-miner pipeline, and this PR extends it with rotation-cadence hook detection.
jensholdgaard/ourios#187: This PR's rotation-cadence hook in IngestPipeline::ingest (tracking previous/new durable offsets and firing on segment change) directly builds on the durable-offset plumbing (IngestPipeline::last_durable and Journal::sync → Option<WalOffset>) introduced in #187.

Poem

🐰 Segments rotate and journals spin,
Durable offsets track where we've been.
When WAL turns a page so bright,
The hook catches snapshots—all just right! ✨
Quiesced on failure, age-aware and keen,
Rotation cadence keeps recovery clean.

🚥 Pre-merge checks | ✅ 5

✅ Passed checks (5 passed)

Check name	Status	Explanation
Title check	✅ Passed	The title clearly and concisely summarizes the main changes: WAL segment rotation (RFC0008.6) and rotation-triggered snapshot cadence (RFC0008.10), directly matching the PR's primary objectives.
Description check	✅ Passed	The description is comprehensive and well-structured, covering the 'What' section with detailed implementation notes, a 'Tests' section documenting coverage, and a 'Checks run' section confirming verification steps. However, it is missing explicit confirmation of the Checklist items (cargo fmt, clippy, tests, docs/CHANGELOG, RFC link).
Docstring Coverage	✅ Passed	No functions found in the changed files to evaluate docstring coverage. Skipping docstring coverage check.
Linked Issues check	✅ Passed	Check skipped because no linked issues were found for this pull request.
Out of Scope Changes check	✅ Passed	Check skipped because no linked issues were found for this pull request.

_{✏️ Tip: You can configure your own custom pre-merge checks in the settings.}

✨ Finishing Touches

🧪 Generate unit tests (beta)

Create PR with unit tests
Commit unit tests in branch feat/rfc0008-6-rotation

Thanks for using CodeRabbit! It's free for OSS, and your support helps us grow. If you like it, consider giving us a shout-out.

❤️ Share

_{Comment @coderabbitai help to get the list of available commands and usage tips.}

Copilot

Pull request overview

Implements RFC0008.6 WAL segment rotation (size-cap + age-cap) and wires a rotation-triggered snapshot cadence hook through the ingester pipeline into the server, enabling RFC0008.10’s rotation cadence behavior.

Changes:

Add pre-append rotation logic to Wal::append (size/age triggers) with quiesce-on-rotation-failure behavior.
Introduce IngestPipeline::with_rotation_hook and fire it on detected segment changes before the rotating batch reaches the miner.
Add integration tests covering WAL rotation arms and rotation-triggered snapshot cadence end-to-end.

Reviewed changes

Copilot reviewed 5 out of 5 changed files in this pull request and generated 1 comment.

Show a summary per file

File	Description
crates/ourios-wal/tests/rfc0008_6_rotation.rs	Adds RFC0008.6 tests for size-cap, time-cap, and quiesce rotation behavior.
crates/ourios-wal/src/lib.rs	Implements rotation + quiesce behavior inside `append`, plus UUIDv7-based age calculation.
crates/ourios-server/src/receiver.rs	Wires the ingester rotation hook to snapshot writing as the primary cadence point.
crates/ourios-ingester/tests/rfc0008_10_rotation_cadence.rs	Adds RFC0008.10 test asserting rotation triggers correctly stamped snapshots.
crates/ourios-ingester/src/receiver/pipeline.rs	Adds rotation hook API + firing semantics before miner ingest on segment change.

💡 Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.

…rent-dir fsync Copilot caught the ordering gap: the parent fsync made the new directory entry power-cut-durable while the 24 B header bytes were not, so a crash in that window would surface the next open's header read as OpenError::Corrupt — a benign crash turned unrecoverable. The header is now fdatasync'd first; failure quiesces like every other rotation step. No new test: the effect is kernel cache ordering, unobservable from userspace without filesystem fault injection — the quiesce path the new step joins is already covered by the RFC0008.6 quiesce arm. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>

Copilot

Pull request overview

Copilot reviewed 5 out of 5 changed files in this pull request and generated 2 comments.

Copilot

Pull request overview

Copilot reviewed 5 out of 5 changed files in this pull request and generated 2 comments.

…lure The variant covers any rotation-step failure (closing fdatasync, fresh-segment creation, parent-dir fsync), so the fsync-specific name misled operators and log parsing; the Display string and the quiesce arm's test name follow. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>

coderabbitai

Actionable comments posted: 1

🤖 Prompt for all review comments with AI agents

Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.

Inline comments:
In `@crates/ourios-ingester/src/receiver/pipeline.rs`:
- Around line 176-181: The direct call to rotation_hook (the hook(&self.miner,
prev) invocation inside the if-let) can panic and unwind through the ingest
pipeline, poisoning the shared pipeline mutex; wrap the hook invocation in
std::panic::catch_unwind so panics are contained: call the hook inside a closure
passed to catch_unwind (use std::panic::AssertUnwindSafe if necessary to satisfy
bounds), log or otherwise swallow the Err returned by catch_unwind, and continue
without letting the panic propagate; keep the if-let condition
(self.rotation_hook.as_mut(), before, self.last_durable) and only replace the
direct hook(...) call with the catch_unwind-protected invocation.

🪄 Autofix (Beta)

Fix all unresolved CodeRabbit comments on this PR:

Push a commit to this branch (recommended)
Create a new PR with the fixes

ℹ️ Review info

⚙️ Run configuration

Configuration used: defaults

Review profile: CHILL

Plan: Pro Plus

Run ID: d6324e7d-6a23-4346-9548-0a37cd172039

📥 Commits

Reviewing files that changed from the base of the PR and between 6649d34 and 1201db2.

📒 Files selected for processing (5)

crates/ourios-ingester/src/receiver/pipeline.rs
crates/ourios-ingester/tests/rfc0008_10_rotation_cadence.rs
crates/ourios-server/src/receiver.rs
crates/ourios-wal/src/lib.rs
crates/ourios-wal/tests/rfc0008_6_rotation.rs

… never unwind ingest Review round 3: the age comparison now uses the full Duration — whole-second truncation delayed the cap by up to a second past the configured bound (the time-cap arm tightened accordingly: the very next append after crossing the cap rotates, and the empty-segment skip is now the load-bearing protection in the first phase). The rotation hook runs under catch_unwind: a panicking snapshot writer must not poison the shared pipeline mutex and halt ingestion over a best-effort cache write — pinned by a test that panics in the hook and asserts the batch still acks, reaches the miner, and the pipeline stays usable. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>

Copilot

Pull request overview

Copilot reviewed 5 out of 5 changed files in this pull request and generated 1 comment.

…ion age Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>

jensholdgaard and others added 2 commits June 12, 2026 22:19

jensholdgaard requested a review from Copilot June 12, 2026 20:42

Copilot started reviewing on behalf of jensholdgaard June 12, 2026 20:42 View session

Copilot AI reviewed Jun 12, 2026

View reviewed changes

Comment thread crates/ourios-wal/src/lib.rs

jensholdgaard requested a review from Copilot June 12, 2026 20:52

Copilot started reviewing on behalf of jensholdgaard June 12, 2026 20:52 View session

Copilot AI reviewed Jun 12, 2026

View reviewed changes

Comment thread crates/ourios-wal/src/lib.rs Outdated

Comment thread crates/ourios-wal/tests/rfc0008_6_rotation.rs Outdated

jensholdgaard requested a review from Copilot June 12, 2026 21:07

Copilot started reviewing on behalf of jensholdgaard June 12, 2026 21:07 View session

Copilot AI reviewed Jun 12, 2026

View reviewed changes

Comment thread crates/ourios-ingester/tests/rfc0008_10_rotation_cadence.rs

Comment thread crates/ourios-wal/src/lib.rs Outdated

coderabbitai Bot reviewed Jun 12, 2026

View reviewed changes

Comment thread crates/ourios-ingester/src/receiver/pipeline.rs

jensholdgaard requested a review from Copilot June 12, 2026 21:28

Copilot started reviewing on behalf of jensholdgaard June 12, 2026 21:28 View session

Copilot AI reviewed Jun 12, 2026

View reviewed changes

Comment thread crates/ourios-ingester/tests/rfc0008_10_rotation_cadence.rs Outdated

docs(ingester): cadence test sleep rationale reflects duration-precis…

c39d9bc

…ion age Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>

jensholdgaard merged commit 8c2c28c into main Jun 12, 2026
11 checks passed

jensholdgaard deleted the feat/rfc0008-6-rotation branch June 12, 2026 21:43

coderabbitai Bot mentioned this pull request Jun 13, 2026

feat(ingester): batched-fsync group commit (RFC0008.8) #191

Merged

jensholdgaard mentioned this pull request Jun 13, 2026

docs(rfc-0001,rfc-0008): advance both to green on the maturity ladder #192

Merged

coderabbitai Bot mentioned this pull request Jun 17, 2026

feat(server): wire the RFC 0014 data write path live (rfc0013 green .6) #245

Merged

coderabbitai Bot mentioned this pull request Jul 20, 2026

fix(ingester): rfc 0035 review fixes — capture-slot unwind safety, honest barrier docs #579

Merged

Navigation Menu

feat(wal,ingester,server): segment rotation + rotation-triggered snapshot cadence (RFC0008.6) - #188

feat(wal,ingester,server): segment rotation + rotation-triggered snapshot cadence (RFC0008.6)#188
jensholdgaard merged 6 commits into
mainfrom
feat/rfc0008-6-rotation

jensholdgaard commented Jun 12, 2026 •

edited by coderabbitai Bot

Loading

Uh oh!

coderabbitai Bot commented Jun 12, 2026 •

edited

Loading

Review limit reached

Walkthrough

Changes

Estimated code review effort

Possibly related issues

Possibly related PRs

Poem

Uh oh!

Copilot AI left a comment

Uh oh!

Uh oh!

Copilot AI left a comment

Uh oh!

Uh oh!

Uh oh!

Copilot AI left a comment

Uh oh!

Uh oh!

Uh oh!

coderabbitai Bot left a comment

Uh oh!

Uh oh!

Copilot AI left a comment

Uh oh!

Uh oh!

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

2 participants

Conversation

jensholdgaard commented Jun 12, 2026 • edited by coderabbitai Bot Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

What

Tests

Invariants / hazards

Checks run

Summary by CodeRabbit

Release Notes

Uh oh!

coderabbitai Bot commented Jun 12, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Review limit reached

Walkthrough

Changes

Estimated code review effort

Possibly related issues

Possibly related PRs

Poem

Uh oh!

Copilot AI left a comment

Choose a reason for hiding this comment

Pull request overview

Reviewed changes

Uh oh!

Uh oh!

Copilot AI left a comment

Choose a reason for hiding this comment

Pull request overview

Uh oh!

Uh oh!

Uh oh!

Copilot AI left a comment

Choose a reason for hiding this comment

Pull request overview

Uh oh!

Uh oh!

Uh oh!

coderabbitai Bot left a comment

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Copilot AI left a comment

Choose a reason for hiding this comment

Pull request overview

Uh oh!

Uh oh!

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

2 participants

jensholdgaard commented Jun 12, 2026 •

edited by coderabbitai Bot

Loading

coderabbitai Bot commented Jun 12, 2026 •

edited

Loading