feat(bench): RFC 0009 D1/D2 sustained-ingest soak harness - #558
Merged
Conversation
Adds `ourios-bench soak`: an in-process soak that drives the real ingest path — OTLP export -> IngestPipeline -> group-commit WAL fsync (100 ms window) -> miner -> Parquet record sink on a local Store — at a paced target rate, and samples the compaction backlog while driving `run_sweep` manually. The core mechanism is a synthetic clock: compaction only acts on sealed partitions (hour end + grace), which is time-driven logic, so record timestamps advance on a compressed timeline (`time_compression` synthetic seconds per wall second; default 60 = one wall-minute of load per synthetic hour) and the same synthetic now feeds `run_sweep`/`plan_candidates`. The sealing logic runs unmodified — only the timestamps it compares are compressed — so the real seal -> sweep -> compact path is exercised without waiting real hours. D1 throughput and ack latency are measured in wall-clock time and are unaffected; ack latency is taken at the `IngestPipeline::ingest` boundary (commit wait + in-order miner hand-off), a conservative upper bound on the WAL-commit latency and exactly what an OTLP client sees. The report (JSON via --out + stdout summary) carries the achieved and per-core rates, ack-latency percentiles, the backlog timeseries, and D1/D2 verdicts with their exact bars (>= 100_000 lines/s/core with p99 <= 200 ms; backlog bounded, returning to zero and draining to a final zero). A workflow_dispatch-only soak-bench.yml runs it on the ci-runner (indicative, non-authoritative) with a verdict-table job summary and the JSON as artifact. Hazard #3 (WAL durability vs. latency): the harness changes no ingest code; it measures the existing batched-fsync path with the production 100 ms window through a shared-Wal Journal wrapper, on a multi-thread runtime so the coordinator's spawn_blocking fsync offload behaves as in the server. Hazard #4 (small files): the sink flush target is deliberately small so sustained ingest produces the multi-file partitions compaction exists to consolidate, and D2 asserts the sweep drains them. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01WQY9wfrfRggqSpMLH8Xj3Y
|
Warning Review limit reached
Next review available in: 18 minutes Enable usage-based reviews in Billing to review now. Otherwise, wait until the next included review is available. How can I continue?After more reviews become available, a review can be triggered using the To avoid repeated limits, reduce automatic review volume by pausing incremental auto-reviews earlier, using label-based review opt-in, excluding WIP or generated PR titles, or requesting reviews manually when the PR is ready. If your team needs uninterrupted high-volume reviews, an organization admin can enable usage-based reviews. How do review limits work?CodeRabbit enforces per-developer PR review limits for each organization. Most developers receive the normal plan review availability. For paid Pro and Pro+ PR reviews, CodeRabbit uses adaptive limits for sustained high-volume activity. When a developer's recent PR review activity reaches the 95th percentile or higher among CodeRabbit users, additional reviews become available more gradually as earlier reviews age out of the rolling window. Please refer docs for additional details. Review details⚙️ Run configurationConfiguration used: defaults Review profile: CHILL Plan: Pro Plus Run ID: 📒 Files selected for processing (5)
✨ Finishing Touches🧪 Generate unit tests (beta)
Thanks for using CodeRabbit! It's free for OSS, and your support helps us grow. If you like it, consider giving us a shout-out. Comment |
Contributor
There was a problem hiding this comment.
Pull request overview
Note
Copilot couldn't run its full agentic review because it didn't start before the timeout. Make sure your repository has a runner available, or add a copilot-code-review.yml file specifying one with the runs-on attribute. See the docs for more details.
Adds an RFC 0009 D1/D2 sustained-ingest “soak” harness to ourios-bench, with a new soak subcommand that drives the real in-process ingest pipeline under a synthetic clock while sampling compaction/WAL backlog, plus a workflow to run it on-demand.
Changes:
- Introduces
crates/ourios-bench/src/soak.rsimplementing the soak load generator, synthetic clock, sampler, verdict/report types, and tests. - Adds a
soaksubcommand and summary/report output plumbing tocrates/ourios-bench/src/main.rs. - Publishes the soak API from
lib.rs, updatesCargo.tomldeps, and adds aworkflow_dispatchGitHub Action to run the soak.
Reviewed changes
Copilot reviewed 5 out of 5 changed files in this pull request and generated 6 comments.
Show a summary per file
| File | Description |
|---|---|
| crates/ourios-bench/src/soak.rs | New soak harness implementation (load pacing, synthetic clock, sampler, backlog accounting, verdict/report, tests). |
| crates/ourios-bench/src/main.rs | Adds soak subcommand parsing/execution and a human-readable summary printer. |
| crates/ourios-bench/src/lib.rs | Exposes soak config/report/verdict APIs from the bench crate. |
| crates/ourios-bench/Cargo.toml | Promotes ingest/WAL deps to regular deps; expands Tokio features for soak runtime needs. |
| .github/workflows/soak-bench.yml | Adds workflow-dispatch soak runner that emits a job summary and uploads artifacts. |
💡 Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.
jensholdgaard
added a commit
that referenced
this pull request
Jul 20, 2026
… wall split, single-list backlog PR #558 review fixes, all six accepted findings: 1. Load loop checks the deadline AFTER awaiting the tick, so a tick landing past the deadline schedules no extra batch (the 5 s smoke now acks exactly duration/pace batches). Test: load_loop_schedules_no_batch_past_the_deadline. 2. Sampler runs on an aligned interval (interval_at + MissedTickBehavior::Delay) instead of sleep-then-work, so the period no longer stretches by each sample's blocking time; the cadence comment now matches the behavior. 3. Ack latencies are bounded: LatencyRecorder caps stored samples at 2^22 and decimates by two on overflow, keeping a systematic 1-in-2^k sample of the stream (percentiles stay valid at bounded memory); the report carries latency_samples_stored/_total. Test: latency_recorder_decimates_to_a_systematic_sample. 4. wall_secs split into load_wall_secs (the D1 rate denominator, measured at end of load drain) and total_wall_secs (covers the sampler join + drain sweep, so no sample timestamp exceeds it); JSON, stdout summary, and the workflow jq updated. Smoke test asserts the ordering invariants. 5. D2Verdict::returned_to_zero_after_max docstring now states plainly that the post-load drain sweep reaching zero is itself the return-to-zero evidence; logic unchanged, pinned by an extended d2_verdict test. 6. Backlog bytes come from ONE tenant-wide listing per sample matched against the candidates' partition prefixes, not one listing per candidate — bounded list ops per sample on non-local stores. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01WQY9wfrfRggqSpMLH8Xj3Y
…, single-list backlog PR #558 review fixes, all accepted findings: 1. Load loop checks the deadline AFTER awaiting the tick, so a tick landing past the deadline schedules no extra batch (the 5 s smoke now acks exactly duration/pace batches). Test: load_loop_schedules_no_batch_past_the_deadline. 2. Both tickers pace with MissedTickBehavior::Delay: the sampler runs on an aligned interval (interval_at) instead of sleep-then-work, so the period no longer stretches by each sample's blocking time, and the load ticker no longer bursts back-to-back catch-up batches after a stall (which broke the paced-load assumption and inflated in-flight depth). 3. Ack latencies are bounded: LatencyRecorder caps stored samples at 2^22 and decimates by two on overflow, keeping a systematic 1-in-2^k sample of the stream (percentiles stay valid at bounded memory); the report carries latency_samples_stored/_total. Test: latency_recorder_decimates_to_a_systematic_sample. 4. wall_secs split into load_wall_secs (the D1 rate denominator, measured at end of load drain) and total_wall_secs (covers the sampler join + drain sweep, so no sample timestamp exceeds it); JSON, stdout summary, and the workflow jq updated. Smoke test asserts the ordering invariants. 5. D2Verdict::returned_to_zero_after_max docstring now states plainly that the post-load drain sweep reaching zero is itself the return-to-zero evidence; logic unchanged, pinned by an extended d2_verdict test. 6. Backlog bytes come from ONE tenant-wide listing per sample matched against the candidates' partition prefixes, not one listing per candidate — bounded list ops per sample on non-local stores. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01WQY9wfrfRggqSpMLH8Xj3Y
jensholdgaard
force-pushed
the
rfc0009-d1-d2-soak
branch
from
July 20, 2026 04:09
6f5d6b7 to
cc02a7e
Compare
Two more accepted PR #558 review findings: 1. The batch generator writes each param into the body via `write!` (std::fmt::Write) instead of allocating an intermediate String per record slot in the hot loop. 2. Sampler shutdown is prompt: the stop flag rides a tokio watch channel and races the tick in a `select!`, so the sampler exits as soon as stop is flagged instead of idling out up to one full `sample_every_secs` tick (which inflated total_wall_secs for nothing — the drain sample already exists). The aligned-cadence / no-overlap properties are unchanged: samples still run strictly between ticks, one at a time. tokio grows the `macros` feature for the `select!`. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01WQY9wfrfRggqSpMLH8Xj3Y
This was referenced Jul 20, 2026
jensholdgaard
added a commit
that referenced
this pull request
Jul 20, 2026
The only .await is oidc-gated; the async signature is the cross-feature API. Per-package clippy without the feature tripped on it (#558 review noted it twice). Claude-Session: https://claude.ai/code/session_01WQY9wfrfRggqSpMLH8Xj3Y Co-authored-by: Claude Fable 5 <noreply@anthropic.com>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
What
The RFC 0009 D1/D2 sustained-ingest soak harness — the two unrun hot-path criteria (docs/benchmarks.md §D1/§D2) get a runnable measure: a
soaksubcommand on theourios-benchbinary that assembles the real end-to-end in-process pipeline (IngestPipeline+CommitCoordinatorgroup commit at the 100 ms WAL batch window + manual compaction sweeps) and drives paced synthetic OTLP load against it.The synthetic clock (the design's one trick, documented in the module doc): compaction only acts on sealed partitions (hour-end + grace), so a one-hour wall-clock soak would never compact. Record timestamps advance on a compressed timeline (
--time-compression) and the same synthetic "now" feedsrun_sweep/plan_candidates— the real seal→sweep→compact path runs continuously. D1's ack measurement stays on the real wall clock.IngestPipeline::ingestboundary — a conservative upper bound on the commit-only definition (a pass implies the bar) — p50/p95/p99/max, per-core rate vs the ≥100k lines/s/core + p99 ≤ 200 ms bars.list_with_sizes), then a drain sweep past hour+grace;final_backlog == 0 && returned-to-zero-after-max. A drain-sweep error reportsusize::MAX— a loud FAIL, never a silently "drained" run..github/workflows/soak-bench.yml:workflow_dispatch(duration/rate/time-compression inputs), SHA pins copied from compaction-bench.yml, verdict table in the job summary marked "ci-runner (indicative)", JSON artifact.Verification
cargo fmt --checkclean;cargo clippy --all-targets --all-features -D warningsclean workspace-wide; full workspacecargo nextest run1177/1177 including 8 new soak tests (the end-to-end smoke soak runs in ~5 s, not ignored); binary smoke run exercised the full path (its D1 verdict honestly FAILs on an E-core debug build — the harness measures, it does not flatter); workflow YAML parses and the jq summary was validated against the real smoke JSON.Pre-existing note (untouched):
-p ourios-bench --all-featuresalone surfaces a latentunused_asyncinourios-ingester/src/receiver/auth.rs:170withoidcoff; CI's workspace-wide--all-featuresmasks it.Invariants / hazards
Hazard #3 (WAL durability vs latency) and #4 (small-file problem) are what this measures — no production code changes at all;
ourios-wal/ourios-ingestermoved from dev- to regular deps of the bench crate only.🤖 Generated with Claude Code
https://claude.ai/code/session_01WQY9wfrfRggqSpMLH8Xj3Y