feat(ingester): instrument the compaction sweep with RFC 0009 §3.6 metrics by jensholdgaard · Pull Request #106 · jensholdgaard/ourios

jensholdgaard · 2026-06-03T18:57:52Z

What

Phase 3 of the telemetry code work: records the RFC 0009 §3.6 compaction metric set through OpenTelemetry, using the generated ourios-semconv name constants (PR #105) and the global meter (RFC 0001 §6.8 API/SDK split — the ingester depends only on the lightweight opentelemetry API; the SDK + OTLP exporter live in ourios-telemetry, PR #104).

Changes

CompactionMetrics (new metrics.rs): builds the ourios.compaction instruments and seeds each counter with a zero measurement at construction, so the full set is visible to the exporter before the first sweep (§6.8 collect-on-read). The duration histogram is deliberately not seeded — a synthetic record(0) would distort its distribution.
Compactor::run builds the instruments once (seeded) before the loop and records every sweep:
- ourios.compaction.sweeps (+ duration histogram) carry ourios.compaction.result = committed / noop / error;
- partitions / files / rows / orphan.files carry the volume. Sweep wall-clock is timed inside the blocking task.
SweepReport.files_compacted added (sum of each consolidated partition's files_before) for the H4 ourios.compaction.files signal.

Deferred (need data this slice lacks)

ourios.compaction.io (per-compaction byte counts aren't on CompactionOutcome), ourios.compaction.backlog (gauge semantics), and ourios.storage.parquet.file.size (an all-files H4 detector that belongs to the writer / a partition scan, not the compactor). The ourios-semconv constants for these already exist; instrumentation lands when the data is plumbed.

Hazard note (H4, RFC 0009 §3.6)

ourios.compaction.files (input files merged away) is the small-file mitigation signal; instruments are seeded so they don't silently disappear at zero traffic.

Test

Stands up an in-memory MeterProvider via ourios-telemetry::init_in_memory, records a sweep, force-flushes, and asserts the exported stream carries every recorded instrument.

Verification

cargo fmt --all --check, clippy --all-targets --all-features -D warnings, cargo test --workspace --all-features — all green.

🤖 Generated with Claude Code

Summary by CodeRabbit

New Features
- Added compaction metrics to monitor sweeps, partitions, files (including files_compacted), rows, orphan files, and sweep duration.
- Exposed a public compaction metrics API to create and record telemetry; sweep timing and a required result attribute are recorded for each sweep.
- Counters are zero-seeded so they appear before the first sweep.
Tests
- Added unit tests validating metrics collection, exported metric names, required attributes, and sweep classification.

…trics Phase 3 of the telemetry code work. Records the compaction metric set through OpenTelemetry, using the generated `ourios-semconv` name constants and the global meter (RFC 0001 §6.8 API/SDK split — the ingester depends only on the lightweight `opentelemetry` API; the SDK + OTLP exporter live in `ourios-telemetry`, configured by the binary). - New `CompactionMetrics` (crates/ourios-ingester/src/metrics.rs): builds the `ourios.compaction` instruments and seeds each counter with a zero measurement at construction so the full set is visible to the exporter before the first sweep (§6.8 collect-on-read). The duration histogram is deliberately not seeded — a synthetic `record(0)` would distort it. - `Compactor::run` builds the instruments once (seeded) before the loop and records each sweep: `sweeps` + `duration` carry the `ourios.compaction.result` attribute (committed / noop / error), and `partitions` / `files` / `rows` / `orphan.files` carry the volume. Sweep wall-clock is timed inside the blocking task. - `SweepReport` gains `files_compacted` (sum of each consolidated partition's `files_before`) for the H4 `ourios.compaction.files` signal. Deferred to a follow-up (need data this slice doesn't have): `ourios.compaction.io` (per-compaction byte counts aren't on `CompactionOutcome`), `ourios.compaction.backlog` (gauge semantics), and `ourios.storage.parquet.file.size` (an all-files H4 detector that belongs to the writer / a partition scan, not the compactor). Test: stands up an in-memory MeterProvider via `ourios-telemetry`'s `init_in_memory`, records a sweep, force-flushes, and asserts the exported stream carries every recorded instrument. Verification: cargo fmt --all --check, clippy --all-targets --all-features -D warnings, cargo test --workspace --all-features — all green. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>

jensholdgaard · 2026-06-03T18:57:55Z

@coderabbitai review

coderabbitai · 2026-06-03T18:58:00Z

✅ Action performed

Review finished.

Note: CodeRabbit is an incremental review system and does not re-review already reviewed commits. This command is applicable only when automatic reviews are paused.

coderabbitai · 2026-06-03T18:58:05Z

No actionable comments were generated in the recent review. 🎉

ℹ️ Recent review info

⚙️ Run configuration

Configuration used: defaults

Review profile: CHILL

Plan: Pro Plus

Run ID: 2b1980c9-91b1-4498-bb1a-65f7490512fa

📥 Commits

Reviewing files that changed from the base of the PR and between 900484d and 655cab1.

📒 Files selected for processing (1)

crates/ourios-ingester/src/compactor.rs

🚧 Files skipped from review as they are similar to previous changes (1)

crates/ourios-ingester/src/compactor.rs

📝 Walkthrough

Walkthrough

Adds OpenTelemetry metrics for compaction sweeps: new CompactionMetrics (counters + histogram), per-sweep timing and recording in the compactor daemon, a SweepReport.files_compacted field, and telemetry dependency/test wiring.

Changes

Compaction Metrics Instrumentation

Layer / File(s)	Summary
Telemetry dependencies `crates/ourios-ingester/Cargo.toml`	Adds `opentelemetry` (metrics-only) and `ourios-semconv` to production dependencies; adds `ourios-telemetry` (testing), `opentelemetry_sdk` (metrics + testing), and multi-thread tokio runtime to dev-dependencies.
CompactionMetrics instrumentation module `crates/ourios-ingester/src/metrics.rs`	Defines `CompactionMetrics` with counters for sweeps, partitions, files, rows, orphan files and a histogram for sweep duration. Implements `new`, `record_sweep`, `Default`, seeds attribute-free counters, and includes an in-memory telemetry unit test validating metric names and result attributes.
SweepReport data tracking `crates/ourios-ingester/src/compactor.rs`	Adds `files_compacted: u64` to `SweepReport`, a saturating `to_u64` helper, sets `files_compacted` from `outcome.files_before` on commit, and updates the related test to assert the value.
Daemon loop metrics integration `crates/ourios-ingester/src/compactor.rs`	Instantiates `CompactionMetrics` before the daemon loop, times each `run_sweep` call with `Instant`, records metrics via `record_sweep(&result, elapsed)` before forwarding the result to the caller observer, and documents seeding-before-loop behavior.
Public API exports `crates/ourios-ingester/src/lib.rs`	Exports new `metrics` module and re-exports `CompactionMetrics` from the crate root.

Sequence Diagram

sequenceDiagram
  participant Daemon as Compactor::run
  participant Metrics as CompactionMetrics
  participant Meter as ourios.compaction meter
  Daemon->>Metrics: record_sweep(result, elapsed)
  Metrics->>Meter: record sweeps & duration (+ optional counters on Ok)

Estimated code review effort

🎯 3 (Moderate) | ⏱️ ~25 minutes

Possibly related PRs

jensholdgaard/ourios#104: Provides the ourios-telemetry in-memory testing scaffold used by the new metrics unit test.
jensholdgaard/ourios#105: Adds the ourios-semconv generated constants crate used for metric and attribute names.
jensholdgaard/ourios#101: Earlier compactor changes that this PR extends (SweepReport and daemon loop behavior).

Poem

"I nibble logs and count each sweep,
tiny hops record the data deep.
Timed whiskers, counters sown so neat,
seeded zeros make the metrics meet.
A rabbit cheers — the telemetry's complete."

🚥 Pre-merge checks | ✅ 5

✅ Passed checks (5 passed)

Check name	Status	Explanation
Title check	✅ Passed	The title clearly and specifically summarizes the main change: instrumenting the compaction sweep with RFC 0009 §3.6 metrics.
Description check	✅ Passed	The description comprehensively covers all template sections with detailed context, changes, deferred work, testing approach, and verification results.
Docstring Coverage	✅ Passed	Docstring coverage is 100.00% which is sufficient. The required threshold is 80.00%.
Linked Issues check	✅ Passed	Check skipped because no linked issues were found for this pull request.
Out of Scope Changes check	✅ Passed	Check skipped because no linked issues were found for this pull request.

_{✏️ Tip: You can configure your own custom pre-merge checks in the settings.}

✨ Finishing Touches

📝 Generate docstrings

Create stacked PR
Commit on current branch

🧪 Generate unit tests (beta)

Create PR with unit tests
Commit unit tests in branch feat/compaction-metrics

_{Comment @coderabbitai help to get the list of available commands and usage tips.}

Copilot

Pull request overview

Adds OpenTelemetry instrumentation for the ingester compaction sweep per RFC 0009 §3.6, using ourios-semconv generated name constants and the global meter provider pattern from RFC 0001 §6.8.

Changes:

Introduces CompactionMetrics to build and record compaction sweep counters + duration histogram.
Wires metrics recording into Compactor::run, including wall-clock timing inside the blocking sweep task.
Extends SweepReport with files_compacted to support the ourios.compaction.files signal.

Reviewed changes

Copilot reviewed 4 out of 5 changed files in this pull request and generated 3 comments.

Show a summary per file

File	Description
crates/ourios-ingester/src/metrics.rs	New compaction metrics instruments + recording + in-memory export test.
crates/ourios-ingester/src/lib.rs	Exposes the new `metrics` module and re-exports `CompactionMetrics`.
crates/ourios-ingester/src/compactor.rs	Records metrics per sweep, times blocking work, and reports `files_compacted`.
crates/ourios-ingester/Cargo.toml	Adds OTel API + semconv constants dependency; adds telemetry/sdk test deps.
Cargo.lock	Updates lockfile for new dependencies.

💡 Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.

…§3.6 metrics

jensholdgaard · 2026-06-03T19:08:49Z

@coderabbitai review

coderabbitai · 2026-06-03T19:08:54Z

✅ Action performed

Review finished.

Note: CodeRabbit is an incremental review system and does not re-review already reviewed commits. This command is applicable only when automatic reviews are paused.

Copilot

Pull request overview

Copilot reviewed 4 out of 5 changed files in this pull request and generated 1 comment.

…C 0009 §3.6 metrics

jensholdgaard · 2026-06-03T19:17:02Z

@coderabbitai review

coderabbitai · 2026-06-03T19:17:07Z

✅ Action performed

Review finished.

Note: CodeRabbit is an incremental review system and does not re-review already reviewed commits. This command is applicable only when automatic reviews are paused.

Copilot

Pull request overview

Copilot reviewed 4 out of 5 changed files in this pull request and generated 1 comment.

…with RFC 0009 §3.6 metrics

jensholdgaard · 2026-06-03T19:30:34Z

@coderabbitai review

coderabbitai · 2026-06-03T19:30:39Z

✅ Action performed

Review finished.

Note: CodeRabbit is an incremental review system and does not re-review already reviewed commits. This command is applicable only when automatic reviews are paused.

Copilot

Pull request overview

Copilot reviewed 4 out of 5 changed files in this pull request and generated no new comments.

jensholdgaard requested a review from Copilot June 3, 2026 18:57

Copilot started reviewing on behalf of jensholdgaard June 3, 2026 18:58 View session

jensholdgaard mentioned this pull request Jun 3, 2026

Epic: Background compaction — small-file consolidation (RFC 0009) #94

Closed

10 tasks

Copilot AI reviewed Jun 3, 2026

View reviewed changes

Comment thread crates/ourios-ingester/src/metrics.rs Outdated

Comment thread crates/ourios-ingester/src/compactor.rs Outdated

Comment thread crates/ourios-ingester/src/metrics.rs

fixup! feat(ingester): instrument the compaction sweep with RFC 0009 …

5daa3fe

…§3.6 metrics

jensholdgaard requested a review from Copilot June 3, 2026 19:08

Copilot started reviewing on behalf of jensholdgaard June 3, 2026 19:08 View session

Copilot AI reviewed Jun 3, 2026

View reviewed changes

Comment thread crates/ourios-ingester/src/metrics.rs

fixup! fixup! feat(ingester): instrument the compaction sweep with RF…

900484d

…C 0009 §3.6 metrics

jensholdgaard requested a review from Copilot June 3, 2026 19:17

Copilot started reviewing on behalf of jensholdgaard June 3, 2026 19:17 View session

Copilot AI reviewed Jun 3, 2026

View reviewed changes

Comment thread crates/ourios-ingester/src/compactor.rs

fixup! fixup! fixup! feat(ingester): instrument the compaction sweep …

655cab1

…with RFC 0009 §3.6 metrics

jensholdgaard requested a review from Copilot June 3, 2026 19:30

Copilot started reviewing on behalf of jensholdgaard June 3, 2026 19:30 View session

Copilot AI reviewed Jun 3, 2026

View reviewed changes

jensholdgaard merged commit e1e7ca9 into main Jun 3, 2026
12 checks passed

Conversation

jensholdgaard commented Jun 3, 2026 • edited by coderabbitai Bot Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

What

Changes

Deferred (need data this slice lacks)

Hazard note (H4, RFC 0009 §3.6)

Test

Verification

Summary by CodeRabbit

Uh oh!

jensholdgaard commented Jun 3, 2026

Uh oh!

coderabbitai Bot commented Jun 3, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

coderabbitai Bot commented Jun 3, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Walkthrough

Changes

Sequence Diagram

Estimated code review effort

Possibly related PRs

Poem

Uh oh!

Copilot AI left a comment

Choose a reason for hiding this comment

Pull request overview

Reviewed changes

Uh oh!

Uh oh!

Uh oh!

Uh oh!

jensholdgaard commented Jun 3, 2026

Uh oh!

coderabbitai Bot commented Jun 3, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

Copilot AI left a comment

Choose a reason for hiding this comment

Pull request overview

Uh oh!

Uh oh!

jensholdgaard commented Jun 3, 2026

Uh oh!

coderabbitai Bot commented Jun 3, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

Copilot AI left a comment

Choose a reason for hiding this comment

Pull request overview

Uh oh!

Uh oh!

jensholdgaard commented Jun 3, 2026

Uh oh!

coderabbitai Bot commented Jun 3, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

Copilot AI left a comment

Choose a reason for hiding this comment

Pull request overview

Uh oh!

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

2 participants

jensholdgaard commented Jun 3, 2026 •

edited by coderabbitai Bot

Loading

coderabbitai Bot commented Jun 3, 2026 •

edited

Loading

coderabbitai Bot commented Jun 3, 2026 •

edited

Loading

coderabbitai Bot commented Jun 3, 2026 •

edited

Loading

coderabbitai Bot commented Jun 3, 2026 •

edited

Loading

coderabbitai Bot commented Jun 3, 2026 •

edited

Loading