test(bench): D2/D3 compaction throughput + small-file-collapse bench (RFC0009.7)

[jensholdgaard]

What

First step toward RFC 0009 → validated: the D2/D3 bench code, run on the ci-runner for indicative numbers (the cheap-first path; the authoritative baseline-8vcpu-32gib run is a later opt-in). RFC0009.7 maps validated to D2 (compaction throughput) + D3 (small-file count under load) + a B2 re-run; this lands D2 + D3.

• crates/ourios-bench/benches/compaction.rs (criterion, harness = false — same pattern as b1/b2/recovery):
  • d2/compaction-throughput — times compact_partition over a backlog of N small files in one partition (swept 8/32), via iter_batched (each iteration rebuilds the backlog, since compaction is destructive). Prints bytes_read/bytes_written so the criterion time → MiB/s.
  • d3/small-file-collapse — consolidates a representative 64-file backlog; asserts + prints the structural collapse (N live files → 1, every row conserved).
• .github/workflows/compaction-bench.yml — lean, corpus-free, workflow_dispatch-only criterion run on the ci-runner (deliberately not coupled to query-bench's heavy Zenodo corpus fetch). SHA-pinned actions + least-privilege permissions, matching the repo's hardened convention. Job summary surfaces the D2/D3 lines + criterion timings; uploads the log + criterion artifact.

Verified locally

cargo bench -p ourios-bench --bench compaction -- --warm-up-time 1 --measurement-time 3:

• D2: 8 files (~8.5 MB) → ~30 ms (~270 MiB/s); 32 files (~34 MB) → ~107 ms (~306 MiB/s) — throughput ~linear in backlog bytes ("keeps up").
• D3: 64 files → 1, 128 000 rows conserved.

Scope caveat (honest)

D3's absolute target (output files 256 MiB–2 GiB, < 5% under 128 MiB — benchmarks.md D3) needs real corpus volume and the §1 baseline host; a synthetic ci-runner run produces sub-MiB files. So the CI run shows the structural collapse (count + row conservation) + the throughput shape only — stated in the bench doc-comment and the workflow summary. The size-band number is the later authoritative-baseline measurement (with the green → validated flip).

Invariants

Non-gating, indicative bench (the structural side is already pinned by ourios-parquet's rfc0009_1_* / compaction_conserves_every_row tests). No §3 invariant or §4 hazard touched; no product-code change.

🤖 Generated with Claude Code

[coderabbitai]

Warning

Review limit reached

@jensholdgaard, we couldn't start this review because you've reached your PR review rate limit.

More reviews will be available in 3 minutes and 49 seconds. Learn how PR review limits work.

Your organization has used up its prepaid credits, and credit purchases are no longer available. Enable the review add-on in the billing tab to keep reviews running — you're only billed for reviews past your plan's rate limits ($0.25/file).

⌛ How to resolve this issue?

After more reviews become available, a review can be triggered using the @coderabbitai review command as a PR comment. Alternatively, push new commits to this PR.

We recommend that you space out your commits to avoid hitting the rate limit.

🚦 How do rate limits work?

CodeRabbit enforces hourly rate limits for each developer per organization.

Our paid plans include higher PR review limits than trial, open-source, and free plans. In all cases, reviews become available again over time. During sustained high-volume PR review activity, CodeRabbit may temporarily slow when the next review becomes available.

Please see our Fair Usage Limits Policy for further information.

ℹ️ Review info

⚙️ Run configuration

Configuration used: defaults

Review profile: CHILL

Plan: Pro Plus

Run ID: 3872c43d-4a5c-4c44-813b-f25f4471618e

📥 Commits

Reviewing files that changed from the base of the PR and between 9861806 and 9ac6653.

📒 Files selected for processing (3)

• .github/workflows/compaction-bench.yml
• crates/ourios-bench/Cargo.toml
• crates/ourios-bench/benches/compaction.rs

✨ Finishing Touches

🧪 Generate unit tests (beta)

• Create PR with unit tests
• Commit unit tests in branch bench-rfc0009-d2-d3

---

Thanks for using CodeRabbit! It's free for OSS, and your support helps us grow. If you like it, consider giving us a shout-out.

❤️ Share

• X
• Mastodon
• Reddit
• LinkedIn

_{Comment @coderabbitai help to get the list of available commands and usage tips.}

[Copilot]

Pull request overview

Adds a new Criterion benchmark suite and an on-demand GitHub Actions workflow to collect indicative D2/D3 compaction measurements for RFC 0009.7, without pulling any external corpora.

Changes:

• Registers a new compaction Criterion bench target in ourios-bench (non-harness).
• Adds benches/compaction.rs implementing D2 (throughput sweep) and D3 (small-file collapse + row conservation checks).
• Adds a workflow_dispatch-only compaction-bench.yml workflow to run the bench on hosted runners and publish a concise job summary + artifacts.

Reviewed changes

Copilot reviewed 3 out of 3 changed files in this pull request and generated 1 comment.

File	Description
crates/ourios-bench/Cargo.toml	Registers the new compaction benchmark target (harness = false).
crates/ourios-bench/benches/compaction.rs	Implements D2/D3 compaction benchmarks using compact_partition over synthetic backlogs.
.github/workflows/compaction-bench.yml	Adds a manual workflow to run the bench, summarize key lines, and upload logs/criterion output.

---

💡 Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.

[Copilot]

Pull request overview

Copilot reviewed 3 out of 3 changed files in this pull request and generated no new comments.