test(ingester): land rfc0003_2 crash-before-ack — RFC 0003 fully green by jensholdgaard · Pull Request #137 · jensholdgaard/ourios

jensholdgaard · 2026-06-06T19:00:00Z

What

The final OTLP-receiver scenario (RFC0003.2 crash-before-ack) — with this, all 15 §5 acceptance criteria are live and RFC 0003 is fully implemented.

A real-SIGKILL fixture (receiver_crash_fixture, a [[bin]] mirroring wal_crash_fixture from #126): builds an IngestPipeline over a real Wal, ingests one batch (append + fsync), prints READY, parks. The test spawns it, waits for READY, SIGKILLs it (after fsync, before any transport ack), reopens the WAL + replays, and asserts the fsync'd OtlpBatch frame survives and recovers the input record's body.

No dedup assertion — this is the no-loss half of at-least-once: a batch synced-but-not-acked survives the crash; a client that never saw the ack retries, producing a duplicate the OTLP duplicate-data section accepts. Stable across repeated SIGKILL runs.

Verification

cargo test -p ourios-ingester ✓ — 0 ignored; all 15 rfc0003_* files live. Workspace green.
cargo fmt --all --check ✓ · cargo clippy --all-targets --all-features -- -D warnings ✓

After this merges

I'll advance RFC 0003 toward green (or flag it as a maintainer process call per the maturity ladder), record the milestone in memory, and surface the §9 served-socket binary (wiring the HTTP/gRPC listeners into ourios-server with spawn_blocking ingest) as the natural next step — the listeners are currently exercised in-process only.

🤖 Generated with Claude Code

Final OTLP-receiver scenario (RFC0003.2). A real-SIGKILL fixture (receiver_crash_fixture, a [[bin]] mirroring wal_crash_fixture from #126): builds an IngestPipeline over a real Wal, ingests one batch (append + fsync), prints READY, parks. The test spawns it, waits for READY, SIGKILLs it (after fsync, before any transport ack), reopens the WAL + replays, and asserts the fsync'd OtlpBatch frame survives and recovers the input record's body. No dedup assertion — the fsync'd-but-not-acked batch survives, and a client that never saw the ack retries (a duplicate, accepted per the OTLP duplicate-data section). Stable across repeated SIGKILL runs. All 15 §5 acceptance criteria (RFC0003.1–.15) are now live; 0 ignored. The crate doc is updated to reflect the receiver is fully implemented (the remaining gap is a served-socket binary, §9 process-model). Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>

jensholdgaard · 2026-06-06T19:00:04Z

@coderabbitai review

coderabbitai · 2026-06-06T19:00:07Z

Warning

Review limit reached

@jensholdgaard, we couldn't start this review because you've reached your PR review rate limit.

More reviews will be available in 29 minutes and 52 seconds. Learn how PR review limits work.

Your organization has run out of usage credits. Purchase more in the billing tab.

⌛ How to resolve this issue?

After more reviews become available, a review can be triggered using the @coderabbitai review command as a PR comment. Alternatively, push new commits to this PR.

We recommend that you space out your commits to avoid hitting the rate limit.

🚦 How do rate limits work?

CodeRabbit enforces hourly rate limits for each developer per organization.

Our paid plans include higher PR review limits than trial, open-source, and free plans. In all cases, reviews become available again over time. During sustained high-volume PR review activity, CodeRabbit may temporarily slow when the next review becomes available.

Please see our Fair Usage Limits Policy for further information.

ℹ️ Review info

⚙️ Run configuration

Configuration used: defaults

Review profile: CHILL

Plan: Pro Plus

Run ID: 0e2e4007-2209-42f7-9ab0-032df2a88368

📥 Commits

Reviewing files that changed from the base of the PR and between 11a8eea and 65afae3.

📒 Files selected for processing (4)

crates/ourios-ingester/Cargo.toml
crates/ourios-ingester/src/lib.rs
crates/ourios-ingester/tests/fixtures/receiver_crash_fixture.rs
crates/ourios-ingester/tests/rfc0003_2_crash_before_ack.rs

✨ Finishing Touches

🧪 Generate unit tests (beta)

Create PR with unit tests
Commit unit tests in branch feat/otlp-receiver-crash

Thanks for using CodeRabbit! It's free for OSS, and your support helps us grow. If you like it, consider giving us a shout-out.

❤️ Share

_{Comment @coderabbitai help to get the list of available commands and usage tips.}

coderabbitai · 2026-06-06T19:00:09Z

✅ Action performed

Review finished.

Note: CodeRabbit is an incremental review system and does not re-review already reviewed commits. This command is applicable only when automatic reviews are paused.

Copilot

Pull request overview

This PR completes RFC 0003’s final acceptance test slice (RFC0003.2) by adding a real-process crash harness that SIGKILLs an ingesting receiver after WAL fsync but before any transport-level ack, then verifies the durable OtlpBatch frame survives WAL replay.

Changes:

Turn rfc0003_2_crash_before_ack from an ignored stub into a live SIGKILL-based crash-recovery test.
Add a dedicated receiver_crash_fixture [[bin]] to act as the killable subprocess that performs a real ingest + fsync and then parks.
Update ourios-ingester crate docs to reflect RFC 0003 acceptance criteria are now all live.

Reviewed changes

Copilot reviewed 4 out of 4 changed files in this pull request and generated 1 comment.

File	Description
crates/ourios-ingester/tests/rfc0003_2_crash_before_ack.rs	Implements the live crash-before-ack recovery assertion using a SIGKILL’d subprocess and WAL replay.
crates/ourios-ingester/tests/fixtures/receiver_crash_fixture.rs	Adds the subprocess fixture binary that ingests one batch to a real WAL, prints `READY`, then parks for SIGKILL.
crates/ourios-ingester/src/lib.rs	Updates crate-level docs to reflect RFC 0003 is fully implemented (tests live) and notes remaining §9 wiring.
crates/ourios-ingester/Cargo.toml	Registers `receiver_crash_fixture` as a non-product `[[bin]]` for test harness use.

💡 Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.

jensholdgaard · 2026-06-06T19:06:37Z

+use std::io::{BufRead, BufReader};
+use std::process::{Command, Stdio};
+
+use opentelemetry_proto::tonic::common::v1::any_value::Value;
+use ourios_ingester::receiver::decode_protobuf;
+use ourios_wal::{FrameKind, FrameSink, RecoveryError, Wal, WalConfig};
+
+fn wal_config(root: &std::path::Path) -> WalConfig {
+    WalConfig {
+        root: root.to_path_buf(),
+        batch_window_ms: 100,
+        segment_size_bytes: 128 * 1024 * 1024,
+        segment_age_secs: 600,
+        housekeeping_secs: 60,
+        macos_full_fsync: false,
+    }
+}


Fixed — rfc0003_2 now does mod ingest_support; use ingest_support::replay_frames; and drops its duplicated wal_config + CollectingSink. One source for the WAL config (the fixture is a separate [[bin]] so it keeps its own literal, but the test side no longer duplicates the helper).

rfc0003_2 duplicated wal_config + a CollectingSink that tests/ingest_support already provides via replay_frames. Use the shared helper (one source for the WAL config, no drift). Surfaced in review. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>

Copilot

Pull request overview

Copilot reviewed 4 out of 4 changed files in this pull request and generated 2 comments.

jensholdgaard · 2026-06-06T19:12:19Z

+    // Arrange: a real WAL root the fixture and this test both open (via
+    // the shared `ingest_support` helper, so the config can't drift).


Fixed in the latest commit — the Arrange comment now says only this test reads the WAL back via the shared helper, and the fixture (a separate bin) opens the same root with a matching WalConfig literal.

jensholdgaard · 2026-06-06T19:12:20Z

+    assert_eq!(
+        line.trim(),
+        "READY",
+        "fixture signalled READY (got {line:?}) — it died before fsync",
+    );


Fixed — the assert message now states what the check proves ("a different first line means it failed before reaching the post-fsync READY print") rather than implying a specific pre-fsync cause.

…Y assert) Review wording fixes: the Arrange comment no longer implies the fixture uses the shared helper (it has its own matching WalConfig literal), and the READY assertion message states what it proves (the fixture did not reach the post-fsync READY print) rather than over-specifying the cause. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>

Copilot

Pull request overview

Copilot reviewed 4 out of 4 changed files in this pull request and generated 3 comments.

+//! A real-process crash: `receiver_crash_fixture` ingests one batch
+//! (append + fsync) over a real `Wal`, prints `READY`, and parks; this
+//! test `SIGKILL`s it after `READY` — i.e. after the batch is durable but
+//! before any transport ack would be sent — then reopens the WAL and
+//! replays. The fsync'd `OtlpBatch` frame must survive and recover the


+        "fixture must signal READY (got {line:?}) — a different first line \
+         means it failed before reaching the post-fsync READY print",
+    );
+    child.kill().expect("SIGKILL fixture");


+//! Not a product binary — declared as a `[[bin]]` only so the test can
+//! spawn it as a real OS process and `SIGKILL` it. The crate is
+//! `#![deny(unsafe_code)]` (workspace lint), so a `fork()` harness is
+//! out; a child driven by `Child::kill()` is the no-`unsafe` way to
+//! exercise a genuine crash. Mirrors `ourios-wal`'s `wal_crash_fixture`.


jensholdgaard requested a review from Copilot June 6, 2026 19:00

Copilot started reviewing on behalf of jensholdgaard June 6, 2026 19:00 View session

Copilot AI reviewed Jun 6, 2026

View reviewed changes

jensholdgaard requested a review from Copilot June 6, 2026 19:06

Copilot started reviewing on behalf of jensholdgaard June 6, 2026 19:06 View session

Copilot AI reviewed Jun 6, 2026

View reviewed changes

jensholdgaard requested a review from Copilot June 6, 2026 19:12

Copilot started reviewing on behalf of jensholdgaard June 6, 2026 19:12 View session

Copilot AI reviewed Jun 6, 2026

View reviewed changes

jensholdgaard merged commit 77d809f into main Jun 6, 2026
11 checks passed

jensholdgaard deleted the feat/otlp-receiver-crash branch June 6, 2026 19:18

jensholdgaard mentioned this pull request Jun 6, 2026

docs(rfc-0003): advance status specified → green #138

Merged

Navigation Menu

test(ingester): land rfc0003_2 crash-before-ack — RFC 0003 fully green - #137

test(ingester): land rfc0003_2 crash-before-ack — RFC 0003 fully green#137
jensholdgaard merged 3 commits into
mainfrom
feat/otlp-receiver-crash

jensholdgaard commented Jun 6, 2026

Uh oh!

jensholdgaard commented Jun 6, 2026

Uh oh!

coderabbitai Bot commented Jun 6, 2026 •

edited

Loading

Review limit reached

Uh oh!

coderabbitai Bot commented Jun 6, 2026 •

edited

Loading

Uh oh!

Copilot AI left a comment

Uh oh!

jensholdgaard Jun 6, 2026

Uh oh!

Copilot AI left a comment

Uh oh!

jensholdgaard Jun 6, 2026

Uh oh!

jensholdgaard Jun 6, 2026

Uh oh!

Copilot AI left a comment

Uh oh!

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

2 participants

		// Arrange: a real WAL root the fixture and this test both open (via
		// the shared `ingest_support` helper, so the config can't drift).

Conversation

jensholdgaard commented Jun 6, 2026

What

Verification

After this merges

Uh oh!

jensholdgaard commented Jun 6, 2026

Uh oh!

coderabbitai Bot commented Jun 6, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Review limit reached

Uh oh!

coderabbitai Bot commented Jun 6, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

Copilot AI left a comment

Choose a reason for hiding this comment

Pull request overview

Reviewed changes

Uh oh!

jensholdgaard Jun 6, 2026

Choose a reason for hiding this comment

Uh oh!

Copilot AI left a comment

Choose a reason for hiding this comment

Pull request overview

Uh oh!

jensholdgaard Jun 6, 2026

Choose a reason for hiding this comment

Uh oh!

jensholdgaard Jun 6, 2026

Choose a reason for hiding this comment

Uh oh!

Copilot AI left a comment

Choose a reason for hiding this comment

Pull request overview

Uh oh!

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

2 participants

coderabbitai Bot commented Jun 6, 2026 •

edited

Loading

coderabbitai Bot commented Jun 6, 2026 •

edited

Loading