Skip to content
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
72 changes: 72 additions & 0 deletions crates/ourios-ingester/tests/rfc0014_ingest_write_path.rs
Original file line number Diff line number Diff line change
@@ -0,0 +1,72 @@
//! RFC 0014 — ingest write path (record sink + flush policy) acceptance
//! scenarios (§5).
//!
//! **Status: `red`.** These are the failing stubs that drive the `green`
//! implementation: each encodes one RFC 0014 §5 scenario and currently
//! `todo!()`s. They are `#[ignore]`d so the default `cargo test` (and CI)
//! stays green while the buffering `ParquetRecordSink` is built — `green`
//! replaces each body with a real assertion against the sink (per-partition
//! buffers; hybrid size + age + WAL-rotation flush; hard byte-ceiling with
//! blocking backpressure) and removes the `#[ignore]`.
//!
//! Placement may shift at `green`: RFC0014.5 (crash recovery) extends the RFC
//! 0008 WAL harness here in `ourios-ingester`; the buffer-trigger scenarios
//! (.1–.4, .6) may move next to the sink wherever it lands.
//!
//! See `docs/rfcs/0014-ingest-write-path.md` §5/§6.

/// Scenario RFC0014.1 — Size trigger: the emit that crosses the size target
/// flushes the partition to one right-sized Parquet object.
/// See `docs/rfcs/0014-ingest-write-path.md` §5.
#[test]
#[ignore = "RFC0014.1 — red until the ParquetRecordSink + flush policy land (green)"]
fn rfc0014_1_size_trigger() {
todo!("RFC0014.1: a partition flushes on the emit that crosses the size target")
}

/// Scenario RFC0014.2 — Age trigger: a sub-target low-volume partition flushes
/// on the next batch-window tick once its oldest record reaches `max_buffer_age`.
/// See `docs/rfcs/0014-ingest-write-path.md` §5.
#[test]
#[ignore = "RFC0014.2 — red until the ParquetRecordSink + flush policy land (green)"]
fn rfc0014_2_age_trigger() {
todo!("RFC0014.2: low-volume partition flushes on age")
}

/// Scenario RFC0014.3 — Rotation force-flush: a WAL segment rotation flushes
/// every partition (including sub-threshold ones); nothing un-flushed predates
/// the sealed segment.
/// See `docs/rfcs/0014-ingest-write-path.md` §5.
#[test]
#[ignore = "RFC0014.3 — red until the ParquetRecordSink + flush policy land (green)"]
fn rfc0014_3_rotation_force_flush() {
todo!("RFC0014.3: rotation flushes every partition")
}

/// Scenario RFC0014.4 — Bounded memory: the sink early-flushes under pressure
/// and, at the hard ceiling, `emit` blocks so buffered bytes never exceed it.
/// See `docs/rfcs/0014-ingest-write-path.md` §5.
#[test]
#[ignore = "RFC0014.4 — red until the ParquetRecordSink + flush policy land (green)"]
fn rfc0014_4_bounded_memory() {
todo!("RFC0014.4: hard ceiling, never exceeded")
}

/// Scenario RFC0014.5 — No acknowledged-data loss: a crash with a non-empty
/// buffer loses nothing — WAL replay re-mines every un-flushed acknowledged
/// record (`CLAUDE.md` §3.4).
/// See `docs/rfcs/0014-ingest-write-path.md` §5.
#[test]
#[ignore = "RFC0014.5 — red until the ParquetRecordSink + flush policy land (green)"]
fn rfc0014_5_no_acknowledged_data_loss() {
todo!("RFC0014.5: crash mid-buffer loses no acknowledged data (WAL replay)")
}

/// Scenario RFC0014.6 — Tenant isolation: a flush produces an object holding
/// only one tenant's rows; no buffer or flush crosses tenants (`CLAUDE.md` §3.7).
/// See `docs/rfcs/0014-ingest-write-path.md` §5.
#[test]
#[ignore = "RFC0014.6 — red until the ParquetRecordSink + flush policy land (green)"]
fn rfc0014_6_tenant_isolation() {
todo!("RFC0014.6: no cross-tenant buffer or flush")
}
9 changes: 7 additions & 2 deletions docs/rfcs/0014-ingest-write-path.md
Original file line number Diff line number Diff line change
@@ -1,7 +1,7 @@
---
rfc: 0014
title: Ingest write path — record sink and flush policy
status: specified
status: red
author: Jens Holdgaard Pedersen <jens@holdgaard.org>
drafting-assistance: Claude
created: 2026-06-17
Expand All @@ -11,7 +11,7 @@ superseded-by: —

# RFC 0014 — Ingest write path: record sink and flush policy

> **Status note.** **`specified`** (2026-06-17). The conspicuous gap in the
> **Status note.** **`red`** (2026-06-17). The conspicuous gap in the
> ingest stack: today the miner (RFC 0001) emits each mined `MinedRecord`
> into a `RecordSink`, and **production wires `NoOpRecordSink` — the records
> are dropped.** Every other layer is built and tested (OTLP → WAL → miner;
Expand All @@ -35,6 +35,11 @@ superseded-by: —
> than exceed it (§3.4, RFC0014.4). The remaining §7 questions (defaults,
> early-flush victim, rotation-hook surface, size estimation) are tuning /
> implementation detail, decided across the `red`/`green` PRs.
>
> **`red`** lands the six `#[ignore]`d acceptance stubs (RFC0014.1–.6) in
> `crates/ourios-ingester/tests/rfc0014_ingest_write_path.rs` (CI stays green).
> `green` builds the buffering `ParquetRecordSink` and wires it into the miner
> in place of `NoOpRecordSink`.

## 1. Summary

Expand Down
Loading