feat(ingester): scaffold crate + background compaction runner (RFC 0009 §3.2) - #101
Merged
Conversation
…09 §3.2) Establishes the `ourios-ingester` crate (CLAUDE.md §7 layout; added to the workspace) as the ingester role's home, and lands its first working subsystem: the background compaction daemon. - `run_sweep(bucket_root, now_unix_nanos, policy)` — one synchronous, deterministic pass: enumerate tenants (decoding `tenant_id=<enc>` dirs), `plan_candidates` per tenant, `compact_partition` each, into a `SweepReport`. This is the unit the tests exercise. - `Compactor::run(on_sweep)` — the daemon loop: a tokio interval ticks `run_sweep` on the blocking pool (compaction is blocking I/O) and hands each result to a caller observer, so a failing sweep is observed, not fatal (telemetry wiring is a later slice). - `ourios-parquet::percent_decode_tenant` — inverse of the tenant encoder, needed to recover raw tenant ids from directory names when sweeping; round-trip + malformed-escape tests. - `receiver` module is a documented placeholder: the OTLP ingest path (RFC 0003, `drafted`) and WAL-before-ack (RFC 0008) land when those RFCs reach `red`. This scaffold implements only the RFC 0009 §3.2 compaction host; it does not front-run the receiver. Tests: sweep compacts a sealed candidate, skips unsealed, scans every tenant, empty store is zero; the daemon loop runs a sweep end-to-end under tokio until cancelled. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
|
No actionable comments were generated in the recent review. 🎉 ℹ️ Recent review info⚙️ Run configurationConfiguration used: defaults Review profile: CHILL Plan: Pro Plus Run ID: 📒 Files selected for processing (2)
📝 WalkthroughWalkthroughThis PR adds a new ChangesIngester Crate and Background Compaction
Sequence Diagram(s)sequenceDiagram
participant Compactor
participant run_sweep
participant Planner as plan_candidates
participant Merger as compact_partition
participant Storage as bucket_root
Compactor->>run_sweep: invoke with bucket_root, now_unix_nanos, policy
run_sweep->>Storage: list tenants under data/
run_sweep->>Planner: plan_candidates(tenant)
Planner-->>run_sweep: candidate list
run_sweep->>Merger: compact_partition(candidate)
Merger-->>run_sweep: CompactionOutcome
run_sweep-->>Compactor: SweepReport
Estimated code review effort🎯 4 (Complex) | ⏱️ ~45 minutes Possibly related issues
Possibly related PRs
Poem
🚥 Pre-merge checks | ✅ 5✅ Passed checks (5 passed)
✏️ Tip: You can configure your own custom pre-merge checks in the settings. ✨ Finishing Touches📝 Generate docstrings
🧪 Generate unit tests (beta)
Warning Review ran into problems🔥 ProblemsGit: Failed to clone repository. Please run the Comment |
Contributor
There was a problem hiding this comment.
Pull request overview
This PR introduces a new workspace crate, ourios-ingester, and implements the first ingester subsystem: a background compaction runner/daemon that periodically sweeps the object-store layout, plans sealed compaction candidates per tenant, and invokes Parquet compaction (per RFC 0009 §3.2). It also adds tenant percent-decoding support to ourios-parquet to enable enumerating tenants from tenant_id=<enc> directory names.
Changes:
- Add
ourios-ingestercrate scaffold with a synchronousrun_sweep(deterministic, testable) and an asyncCompactor::rundaemon loop usingspawn_blocking. - Add
percent_decode_tenanttoourios-parquetand re-export it for use by sweep enumeration logic. - Register the new crate in the workspace and lockfile.
Reviewed changes
Copilot reviewed 7 out of 8 changed files in this pull request and generated 2 comments.
Show a summary per file
| File | Description |
|---|---|
| crates/ourios-parquet/src/partition.rs | Adds percent_decode_tenant plus round-trip/malformed-escape tests. |
| crates/ourios-parquet/src/lib.rs | Re-exports percent_decode_tenant from the crate root. |
| crates/ourios-ingester/src/receiver.rs | Placeholder module documenting deferred ingest/WAL work per RFC maturity. |
| crates/ourios-ingester/src/lib.rs | New crate root: documents role/RFC scope and exports compactor API. |
| crates/ourios-ingester/src/compactor.rs | Implements tenant enumeration, run_sweep, daemon loop, and tests. |
| crates/ourios-ingester/Cargo.toml | Declares the new crate and its dependencies (ourios-core, ourios-parquet, tokio). |
| Cargo.toml | Adds crates/ourios-ingester to the workspace members list. |
| Cargo.lock | Records the new workspace package entry for ourios-ingester. |
💡 Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.
Owner
Author
|
@coderabbitai review |
✅ Action performedReview finished.
|
![coderabbitai[bot]](https://avatars.githubusercontent.com/in/347564?s=60&v=4){kind=small}
There was a problem hiding this comment.
Actionable comments posted: 3
🤖 Prompt for all review comments with AI agents
Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.
Inline comments:
In `@crates/ourios-ingester/src/compactor.rs`:
- Around line 86-95: The loop currently uses `?` on `plan_candidates` and
`compact_partition`, which aborts the entire sweep on the first error; change
both calls to handle errors locally instead of propagating: call
`plan_candidates(bucket_root, &tenant, now_unix_nanos, policy)` and
`compact_partition(bucket_root, &partition)` and match on the Result — on Err
log the error (including tenant/partition identifiers) and increment an
appropriate failure counter on `report` (add `plan_failures` /
`partition_failures` if needed), then continue to the next tenant/partition; on
Ok proceed as before and update `report.partitions_compacted`,
`report.rows_compacted` and `report.gc_failures` from `outcome`. This ensures
`Compactor::run` won’t be aborted by a single tenant/partition error.
In `@crates/ourios-ingester/src/lib.rs`:
- Around line 15-16: The doc comment contains a broken intra-doc link: the code
span "[`ourios_parquet:: plan_candidates`]" is split across lines causing
rustdoc to emit broken_intra_doc_links; edit the module-level doc comment in
lib.rs and join the link into a single uninterrupted path like
[`ourios_parquet::plan_candidates`] (remove the newline between :: and
plan_candidates) so the intra-doc link resolves correctly.
In `@crates/ourios-parquet/src/partition.rs`:
- Around line 198-213: percent_decode_tenant currently accepts any non-'%' byte
verbatim which lets non-canonical names (e.g., spaces, non-ASCII) through;
update percent_decode_tenant to reject any unescaped byte that would have been
percent-encoded by percent_encode_tenant by validating each non-'%' byte against
the canonical safe set (the same character class used in percent_encode_tenant)
and return None if a byte is not in that set or if any percent-sequence is
invalid; keep the existing hex-decoding logic for '%' sequences but add the
per-byte check for non-'%' bytes so only canonical unescaped characters are
allowed.
🪄 Autofix (Beta)
Fix all unresolved CodeRabbit comments on this PR:
- Push a commit to this branch (recommended)
- Create a new PR with the fixes
ℹ️ Review info
⚙️ Run configuration
Configuration used: defaults
Review profile: CHILL
Plan: Pro Plus
Run ID: 2e89d531-f3b4-4dbc-bc7d-f0c097176455
⛔ Files ignored due to path filters (1)
Cargo.lockis excluded by!**/*.lock
📒 Files selected for processing (7)
Cargo.tomlcrates/ourios-ingester/Cargo.tomlcrates/ourios-ingester/src/compactor.rscrates/ourios-ingester/src/lib.rscrates/ourios-ingester/src/receiver.rscrates/ourios-parquet/src/lib.rscrates/ourios-parquet/src/partition.rs
…runner (RFC 0009 §3.2)
Owner
Author
|
@coderabbitai review |
✅ Actions performedReview triggered.
|
There was a problem hiding this comment.
Actionable comments posted: 1
🤖 Prompt for all review comments with AI agents
Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.
Inline comments:
In `@crates/ourios-parquet/src/partition.rs`:
- Around line 191-197: The decoder currently accepts lowercase hex via
hex_value, letting `%2f` decode though percent_encode_tenant only emits `%2F`;
update the percent-decoding logic (the code using hex_value in the percent
decode function used by tenants()/percent_decode_tenant) to reject lowercase hex
digits: when parsing the two hex chars after `%`, verify each char matches
'0'..'9' or 'A'..'F' (uppercase) before converting (or otherwise check that the
original characters are uppercase) and return None on any lowercase a..f so the
decoder is the strict inverse of percent_encode_tenant.
🪄 Autofix (Beta)
Fix all unresolved CodeRabbit comments on this PR:
- Push a commit to this branch (recommended)
- Create a new PR with the fixes
ℹ️ Review info
⚙️ Run configuration
Configuration used: defaults
Review profile: CHILL
Plan: Pro Plus
Run ID: 052614ad-a267-4255-b39e-dada6d6da6a0
📒 Files selected for processing (3)
crates/ourios-ingester/src/compactor.rscrates/ourios-ingester/src/lib.rscrates/ourios-parquet/src/partition.rs
…action runner (RFC 0009 §3.2)
Owner
Author
|
@coderabbitai review |
✅ Actions performedReview triggered.
|
10 tasks
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
What
Scaffolds the
ourios-ingestercrate (the ingester role,CLAUDE.md§1/§7 — now a workspace member) and lands its first working subsystem: the background compaction daemon (RFC 0009 §3.2). Closes the "Background task in the ingester" slice on epic #94.New crate — scope & RFC standing
A new crate is an architectural commitment (§7).
ourios-ingesteris in the §7 target layout, and RFC 0009 §3.2 (specified) says compaction is "hosted in the ingester role" — so this scaffold is RFC-backed. It deliberately implements only the compaction host; the OTLP ingest path (RFC 0003, stilldrafted) and WAL-before-ack (RFC 0008) are a documented placeholder (receivermodule) that lands when those RFCs reachred. No ingest is front-run.Changes
run_sweep(bucket_root, now_unix_nanos, policy) -> Result<SweepReport, IngestError>— one synchronous, deterministic pass: enumerate tenants (decodetenant_id=<enc>dirs),plan_candidatesper tenant,compact_partitioneach, accumulate aSweepReport(tenants scanned, partitions compacted, rows, gc_failures). The testable unit.Compactor::run(on_sweep)— the daemon loop: atokio::time::intervalticksrun_sweepon the blocking pool (compaction is blocking FS/Parquet work) and hands eachResultto a caller-supplied observer, so a failing sweep is observed, not fatal, and the loop keeps ticking. (OTel metering is a later slice; the observer is the seam.)ourios_parquet::percent_decode_tenant— inverse of the tenant encoder, needed to recover raw tenant ids from directory names when sweeping. Round-trip + malformed-escape tests.Tests
run_sweep: compacts a sealed candidate; skips an unsealed partition; scans every tenant (compacts only the candidate one); empty store → zero.Compactor::run: spawns the loop under tokio, awaits the first sweep result via the observer, asserts the candidate was compacted, then cancels.percent_decode_tenant: round-trips the encoder across unreserved/delimiter/UTF-8 tenants; rejects malformed escapes.Invariants / hazards
compact_partition(viaReader::open_partition) validates row-vs-path, andplan_candidatesonly returns canonical partition dirs (feat(parquet): add compaction candidate planner (RFC 0009 §3.3) #100).Verification (local)
cargo fmt --all --check✅cargo clippy -p ourios-ingester -p ourios-parquet --all-targets --all-features -- -D warnings✅cargo test -p ourios-ingester -p ourios-parquet --all-features✅ (ingester 5; parquet 52 + integration)Not in scope (later epic #94 slices)
Standalone orphan-GC sweep, audit events + telemetry (incl. the H4 file-size histogram, and OTel metering of
SweepReport), and the crash-recovery/proptest + D2/D3 bench. Wiring the compactor into a running server binary awaitsourios-server.Epic: #94.
🤖 Generated with Claude Code
Summary by CodeRabbit
New Features
Chores
Tests