docs(rfc-0009): flip compaction to validated (D2/D3/B2-post on baseline, §9.7)

docs/benchmarks.md

@@ -813,3 +813,39 @@ component (contrast A1's writer-environment sensitivity, §9.4). The
 value of this run is the authoritative `hardware_kind` stamp on the
 two gates that, under RFC 0011, define RFC 0001's `validated`: **both
 PASS on a representative ≥ 1 M-line corpus on §1 baseline hardware.**
+
+### 9.7 Results — 2026-06-15 (authoritative, `baseline-8vcpu-32gib`) — D2 / D3 / B2-post (RFC 0009 compaction)
+
+**Hardware.** `baseline-8vcpu-32gib` — the §1 baseline (8 dedicated
+vCPU, 32 GiB RAM, local SSD), provisioned for this run and torn down
+immediately after. These are the **authoritative** D2 / D3 / B2-post
+numbers for RFC 0009's `validated` measure (RFC0009.7).
+**Run.** Dedicated baseline host (no CI run id): the `ourios-bench`
+`compaction` bench at git `4d52288`. Two invocations — the band-scale
+one-shot (`OURIOS_COMPACTION_BASELINE=1`, `FILES=32`, `ROWS=4800`,
+`BODY_BYTES=4096`) for D2/D3, then the `b2-post-compaction` criterion
+group. Synthetic (no corpus): D2/D3 drive one partition of 32 small
+files (~485 MiB) through `compact_partition`; B2-post queries
+32-files-vs-1-file with the result set held constant.
+
+| measure | result | verdict |
+|---|---|---|
+| **D2** compaction throughput | 32 files (485.2 MiB) → 1 in 2.91 s = **166.8 MiB/s**; 153,600 rows conserved | keeps up — single-partition / single-threaded, ≫ any per-partition seal rate, so the backlog drains |
+| **D3** small-file size band | output **456.7 MiB** — **IN** the 256 MiB–2 GiB band; **0%** of live files < 128 MiB (target < 5%) | PASS |
+| **B2-post** query latency | template query: uncompacted **12.78 ms** (32 row groups, 33.5 MiB read, 32 files) → compacted **2.10 ms** (1 row group, 1.05 MiB, 1 file) = **6.1×** | PASS |
+
+**Reading.** D3 is the headline: a band-scale compaction lands its
+output squarely in the H4 256 MiB–2 GiB target with zero sub-128 MiB
+files — the small-file problem, eliminated. D2 shows consolidation
+runs at ~167 MiB/s on one partition/thread, far above any plausible
+per-partition seal rate, so a backlog drains (the "keeps up"
+property). B2-post quantifies the query payoff that motivated RFC 0009
+(the PR #92 B2 finding that per-file footer/metadata reads dominate):
+collapsing 32 files → 1 cuts the footer reads ~6× on this query. The
+structural reductions (32 → 1 files / row groups, rows conserved) are
+hardware-independent and also pinned in `ourios-parquet`'s
+`rfc0009_1_*` / `compaction_conserves_every_row` tests; these
+wall-clock figures are the baseline-hardware stamp for RFC 0009's
+`validated`. The full sustained-ingest soak (D2's "backlog returns to
+zero in a one-hour window at D1's rate") and D1 itself remain unrun —
+the throughput here is the RFC0009.7 D2 measure, not that soak.

docs/rfcs/0009-compaction.md

@@ -1,7 +1,7 @@
 ---
 rfc: 0009
 title: Background compaction — small-file consolidation
-status: green
+status: validated
 author: Jens Holdgaard Pedersen <jens@holdgaard.org>
 drafting-assistance: Claude
 created: 2026-06-02
@@ -11,8 +11,18 @@ superseded-by: —
 
 # RFC 0009 — Background compaction: small-file consolidation
 
-> **Status note.** **`green`** (2026-06-15) — every RFC0009 §5 acceptance
-> criterion has a live, passing test: **.1** small-file count collapses
+> **Status note.** **`validated`** (2026-06-15) — the RFC0009.7 D2/D3/B2-post
+> benches were measured authoritatively on `baseline-8vcpu-32gib`
+> (`benchmarks.md` §9.7, git `4d52288`): **D3** PASS (a band-scale compaction
+> output lands at 456.7 MiB, IN the 256 MiB–2 GiB H4 band, 0% under 128 MiB);
+> **D2** compaction throughput 166.8 MiB/s single-partition (≫ any
+> per-partition seal rate → backlog drains); **B2-post** query latency
+> 12.78 ms → 2.10 ms (≈6.1×) as 32 files / row groups collapse to 1. The
+> sustained-ingest soak (D2's literal one-hour window at D1's rate) and D1
+> itself remain unrun — the throughput is the RFC0009.7 D2 measure, not that
+> soak. The prior `green` status (flipped earlier the same day) rested on
+> every RFC0009 §5 acceptance criterion having a live, passing test: **.1**
+> small-file count collapses
 > (`rfc0009_1_*`), **.2** row conservation (`compaction_conserves_every_row`
 > proptest), **.3** atomic publish / no torn read (`atomic_publish_…` +
 > the `ourios-querier` `rfc0009_3_*` manifest tests), **.4** crash safety
@@ -25,20 +35,22 @@ superseded-by: —
 > the runner lives in `ourios-ingester` (`run_sweep`/`Compactor`) with the
 > §3.6 OTel metrics + audit event.
 >
-> **RFC0009.7 is the `validated`-side measure, not a `green` gate.** Per
-> §6 it is the D2/D3 `criterion` benches (compaction throughput, file
-> count under load) + a post-compaction re-run of the RFC 0007 §6 B2
-> latency bench — wall-clock, deferred exactly as RFC 0007's B1/B2 were;
-> its *structural* half (file count falls under compaction) is evidenced
-> by RFC0009.1.
+> **RFC0009.7 — measured.** The D2/D3 `criterion` benches (compaction
+> throughput, small-file size band) + the post-compaction B2 re-run live in
+> `ourios-bench`'s `compaction` bench — CI-indicative via
+> `compaction-bench.yml`, authoritative on `baseline-8vcpu-32gib` in §9.7.
+> Its *structural* half (file count falls under compaction, every row
+> conserved) is also pinned deterministically by RFC0009.1 /
+> `compaction_conserves_every_row`.
 >
-> **Open for `validated` / follow-up (§7):** the D2/D3 benches; and the §7
-> implementation items — late-arriving data re-flagging an already-compacted
-> partition (the manifest is authoritative, so a new write into such a
-> partition must be picked up by the candidate scan / folded into the
-> manifest — confirm `plan_candidates` covers it), the S3 atomic-swap
-> primitive + single-writer lease for object storage (local FS uses
-> `rename`), and the RFC 0004 cadence defaults.
+> **Open follow-ups (§7, post-`validated`):** the full D2 sustained-ingest
+> soak (backlog-returns-to-zero in a one-hour window at D1's rate) + a
+> measured D1; late-arriving data re-flagging an already-compacted partition
+> (the manifest is authoritative, so a new write into such a partition must
+> be picked up by the candidate scan / folded into the manifest — confirm
+> `plan_candidates` covers it); the S3 atomic-swap primitive + single-writer
+> lease for object storage (local FS uses `rename`); and the RFC 0004 cadence
+> defaults.
 
 ## 1. Summary
 

docs/roadmap.md

@@ -2,9 +2,11 @@
 
 > Living document. Refreshed at phase boundaries (§4) and whenever
 > a merged PR materially changes the *current state* in §3.
-> Last updated: **2026-06-15** — RFC 0005 (Parquet storage) and RFC 0010
-> (audit-stream / drift queries) flipped to `green` (RFC0005.6 row-group
-> sizing landed; RFC 0010's eight §5 drift scenarios all pass). Earlier, on
+> Last updated: **2026-06-15** — RFC 0009 (background compaction) flipped to
+> `validated` (RFC0009.7 D2/D3/B2-post measured on `baseline-8vcpu-32gib`,
+> §9.7); RFC 0005 (Parquet storage) and RFC 0010 (audit-stream / drift
+> queries) flipped to `green` (RFC0005.6 row-group sizing landed; RFC 0010's
+> eight §5 drift scenarios all pass). Earlier, on
 > 2026-06-14, RFC 0001, RFC 0008, and RFC 0011 flipped
 > to `accepted` (maintainer sign-off). RFC 0001 reached `validated` first
 > (C1/C2 pass authoritatively on the `benchmarks.md` §1 baseline hardware,
@@ -108,7 +110,7 @@ captured by B1/B2 (see `benchmarks.md` §2 / §7).
 | 0006 | Bench harness | `green` |
 | 0007 | Querier (DataFusion + logs DSL) | **`validated`** |
 | 0008 | WAL | **`accepted`** |
-| 0009 | Background compaction | **`green`** — §5 RFC0009.1–.6 pass (manifest + `gc_orphans` + sweep; #206/#207/#208/#209); .7 (D2/D3 benches) deferred to `validated` |
+| 0009 | Background compaction | **`validated`** — §5 RFC0009.1–.6 pass; RFC0009.7 D2/D3/B2-post measured authoritatively on `baseline-8vcpu-32gib` (§9.7: D3 in 256 MiB–2 GiB band, D2 166.8 MiB/s, B2-post ≈6.1×) |
 | 0010 | Audit-stream / drift queries | **`green`** — all 8 §5 scenarios pass (`crates/ourios-querier/tests/drift.rs`); discharges RFC 0001 H5.3; §9 items are `accepted`-gating; general audit aggregation deferred (§3.2) |
 | 0011 | A1 re-scope | **`accepted`** |
 
@@ -147,9 +149,10 @@ test` job gates every PR on the exact head; the coverage job runs
 alongside it but is informational (`continue-on-error`), not gating.
 
 **What remains** is post-MVP shipping shape (§5 — Helm chart, the
-production deployment surface) plus the open RFCs above: RFC 0005 →
-`green`, RFC 0009 → `validated` (the D2/D3 compaction benches), and the
-items tracked in the RFCs' §7/§9 open-questions.
+production deployment surface) and the items tracked in the RFCs'
+§7/§9 open-questions (e.g. RFC 0009's full D2 sustained-ingest soak +
+a measured D1, and the S3 atomic-swap primitive). RFC 0005 (`green`)
+and RFC 0009 (`validated`) are no longer open.
 
 ---