feat(compaction): reclaim crash orphans + RFC0009.4 crash-recovery test - #206
Conversation
Closes the RFC0009.4 (crash safety) §5 evidence gap. The commit point is
the atomic manifest swap, so a crash always freezes a partition at a clean
generation (RFC0009.3); the missing half was that the dead files a crash
leaves — superseded inputs the post-commit GC never reached, a pre-commit
consolidated file, an interrupted `*.parquet.tmp` publish — are
*reclaimable*.
- `gc_orphans(partition_dir)` in ourios-parquet: manifest-authoritative
reclamation (a `*.parquet` not named by the manifest is provably dead;
no manifest ⇒ glob ⇒ only stray `*.parquet.tmp` are orphans). Idempotent;
never unlinks a live file. Returns `OrphanGc { reclaimed, failures }`.
- RFC0009.4 test: builds the on-disk state a SIGKILL at each commit point
would leave (pre-swap, post-swap-pre-GC, stray `.tmp` / no manifest) and
asserts orphans are reclaimable while reads stay at a clean generation
and live files are untouched. Faithful because the only commit is one
`rename`.
- Wired into the ingester sweep (`SweepReport.orphans_reclaimed`) so a
prior crash's orphans are reclaimed on the next sweep.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
|
Warning Review limit reached
More reviews will be available in 42 minutes. Learn how PR review limits work. Your organization has used up its prepaid credits, and credit purchases are no longer available. Enable the review add-on in the billing tab to keep reviews running — you're only billed for reviews past your plan's rate limits ($0.25/file). ⌛ How to resolve this issue?After more reviews become available, a review can be triggered using the We recommend that you space out your commits to avoid hitting the rate limit. 🚦 How do rate limits work?CodeRabbit enforces hourly rate limits for each developer per organization. Our paid plans include higher PR review limits than trial, open-source, and free plans. In all cases, reviews become available again over time. During sustained high-volume PR review activity, CodeRabbit may temporarily slow when the next review becomes available. Please see our Fair Usage Limits Policy for further information. ℹ️ Review info⚙️ Run configurationConfiguration used: defaults Review profile: CHILL Plan: Pro Plus Run ID: 📒 Files selected for processing (2)
📝 WalkthroughWalkthroughAdds orphan file garbage collection to the sealed-partition compaction path. ChangesOrphan GC for sealed-partition compaction
Estimated code review effort🎯 3 (Moderate) | ⏱️ ~25 minutes Possibly related issues
Possibly related PRs
Poem
🚥 Pre-merge checks | ✅ 5✅ Passed checks (5 passed)
✏️ Tip: You can configure your own custom pre-merge checks in the settings. ✨ Finishing Touches🧪 Generate unit tests (beta)
Thanks for using CodeRabbit! It's free for OSS, and your support helps us grow. If you like it, consider giving us a shout-out. Comment |
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
There was a problem hiding this comment.
Pull request overview
Adds crash-orphan reclamation to the compaction subsystem and introduces an RFC0009.4 acceptance test to close the crash-safety evidence gap around post-crash cleanup behavior.
Changes:
- Introduces
gc_orphans(partition_dir) -> OrphanGcto reclaim manifest-excluded*.parquetand stray*.parquet.tmpfiles safely. - Adds an RFC0009.4 test that constructs crash-point on-disk states (pre-swap, post-swap/pre-GC, and no-manifest stray
.tmp) and validates reclaim + read correctness. - Wires orphan GC into
ourios-ingestersweep execution and extends metrics tests for the expandedSweepReport.
Reviewed changes
Copilot reviewed 4 out of 4 changed files in this pull request and generated 3 comments.
| File | Description |
|---|---|
| crates/ourios-parquet/src/lib.rs | Re-exports gc_orphans / OrphanGc from the compaction module. |
| crates/ourios-parquet/src/compaction.rs | Implements orphan GC + adds RFC0009.4 crash-recovery test coverage. |
| crates/ourios-ingester/src/metrics.rs | Updates SweepReport test fixture to include the new field. |
| crates/ourios-ingester/src/compactor.rs | Runs gc_orphans for each candidate partition and tracks reclaimed count in SweepReport. |
💡 Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.
![coderabbitai[bot]](https://avatars.githubusercontent.com/in/347564?s=60&v=4){kind=small}
There was a problem hiding this comment.
Actionable comments posted: 3
🤖 Prompt for all review comments with AI agents
Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.
Inline comments:
In `@crates/ourios-ingester/src/compactor.rs`:
- Around line 82-86: The orphans_reclaimed field accumulated in run_sweep is not
being published to telemetry by CompactionMetrics::record_sweep, making the
crash recovery path invisible operationally. Update the record_sweep method to
emit orphan-file telemetry for the orphans_reclaimed field alongside the
existing gc_failures metric—either by folding it into an existing instrument or
creating a dedicated one. Then add an assertion in the metrics test to verify
that this metric is being recorded when orphans are reclaimed during a sweep.
- Around line 173-179: The gc_orphans function is only being called on
partitions after they pass the plan_candidates filter, which excludes partitions
with fewer than two live files. However, the crash states this code is meant to
recover from result in partitions with exactly one live file (either a
consolidated file after manifest swap, or one parquet file with a stray tmp
file), which are filtered out before reaching the gc_orphans call. Move the
gc_orphans execution to run across all sealed partitions before the
plan_candidates filtering step, or execute it via a separate sealed-partition
scan that is independent of candidate filtering. Additionally, add a sweep-level
regression test that validates these one-live-file partition cases are properly
handled, building on the existing
rfc0009_4_orphans_reclaimable_and_live_files_untouched test patterns.
In `@crates/ourios-parquet/src/compaction.rs`:
- Around line 311-316: Update the `# Errors` documentation section for the
`gc_orphans` function to include `CompactionError::Manifest` as a documented
error case. Currently, the documentation only mentions `CompactionError::Io` for
directory scan failures, but the function can also fail when reading or parsing
the manifest before the directory scan begins. Add a statement documenting that
manifest read/parse failures will result in `CompactionError::Manifest` being
returned, ensuring the documented error contract matches the actual
implementation.
🪄 Autofix (Beta)
Fix all unresolved CodeRabbit comments on this PR:
- Push a commit to this branch (recommended)
- Create a new PR with the fixes
ℹ️ Review info
⚙️ Run configuration
Configuration used: defaults
Review profile: CHILL
Plan: Pro Plus
Run ID: 5472e694-05cc-4b55-9719-1298cb1673ad
📒 Files selected for processing (4)
crates/ourios-ingester/src/compactor.rscrates/ourios-ingester/src/metrics.rscrates/ourios-parquet/src/compaction.rscrates/ourios-parquet/src/lib.rs
… too_many_lines) Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
- mark OrphanGc #[non_exhaustive] (forward-compat for the new pub struct) - gc_orphans # Errors doc: also documents CompactionError::Manifest - sweep gc-orphans error now names the partition, not just the tenant Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
What
Closes the RFC0009.4 (crash safety) §5 evidence gap for background compaction (epic #94) — the one criterion with implementation but no acceptance test.
Why
Compaction's only commit point is the atomic
manifest.jsongeneration swap, so a crash always freezes a partition at a clean generation (the no-torn-read half isatomic_publish_is_never_torn_across_the_swap, RFC0009.3). The missing half: the dead files a crash leaves — superseded inputs the post-commit GC never reached, a pre-commit consolidated file, or an interrupted*.parquet.tmppublish — must be reclaimable (theManifestdoc itself says manifest-excluded files "are orphans awaiting GC").Changes
ourios-parquetgc_orphans(partition_dir) -> OrphanGc— manifest-authoritative reclamation: a*.parquetnot named by the manifest is provably dead; with no manifest the glob is the live set so only stray*.parquet.tmpare orphans. Idempotent; never unlinks a live file.SIGKILLwould leave at each commit point (pre-swap, post-swap-pre-GC, stray.tmp/no-manifest) and asserts orphans are reclaimable, reads stay at a clean pre-/post-generation, and live files are untouched. Faithful because the commit is a singlerename.run_sweepreclaims a candidate partition's prior-crash orphans (SweepReport.orphans_reclaimed).Invariants (CLAUDE.md §3)
§3.6 (object storage source of truth) / RFC0009.2-.3 unaffected:
gc_orphansonly removes manifest-excluded files (provably dead), never a live row. Row conservation (RFC0009.2) and atomicity (RFC0009.3) tests stay green.Verification
cargo fmt --check,cargo clippy -p ourios-parquet -p ourios-ingester --all-targets --all-features -D warnings, and both crates' full test suites pass locally.Part of driving RFC 0009
specified → green; next: RFC0009.1 (post-compaction file-size histogram) + RFC0009.6 (union-schema merge).🤖 Generated with Claude Code
Summary by CodeRabbit
New Features
Improvements