diff --git a/docs/benchmarks.md b/docs/benchmarks.md index a98e93c8..f87443cc 100644 --- a/docs/benchmarks.md +++ b/docs/benchmarks.md @@ -2168,24 +2168,28 @@ pruning-granularity-over-bytes trade RFC 0036 §3.3 exists to make, now measured on realistic data: it is a good trade, and it gets *better* the finer the threshold, bounded only by footer/index overhead. -**Finding 3 — 32 and 64 MiB are indistinguishable because arrow's default -row-group *row* cap dominates, not the byte threshold.** The compacted -writer sets no `max_row_group_size`, so parquet-rs's default **1,048,576-row** -cap applies. On 2,160,000 rows that forces ~3 groups of ~1 M rows each -*regardless* of a byte threshold ≳ 16 MiB: at 32 and 64 MiB the row cap -trips before the byte threshold does (byte-identical 3-group files), and -only 16 MiB (~590 k rows/group) is genuinely byte-capped. So above ~16 MiB -the compacted **byte** threshold does not actually govern granularity on a -high-row corpus — the row cap does, silently. This is a real gap versus -RFC 0036 §3.3's byte-driven-rotation premise (pruning still works — the -RFC0036.2 gate passes — just at coarser granularity than the byte threshold -implies). **Actionable follow-up:** to let the byte threshold control -rotation (and realise the finer pruning at 32 MiB, not only at 16 MiB), -raise or remove the row cap (`WriterProperties::set_max_row_group_size`) on -the compacted writer. Separating 32 from 64 MiB by *bytes* alone then needs -the baseline corpus's volume too. What this run establishes regardless: -**finer effective groups win the materialization trade at a sub-1% disk -cost.** +**Finding 3 — 32 and 64 MiB are near-identical because arrow's default +row-group *row* cap sets a ~30 MiB granularity floor, finer than either +byte threshold.** The compacted writer sets no `max_row_group_size`, so +parquet-rs's default **1,048,576-row** cap applies. At this corpus's ~30 +encoded bytes/row that cap fills a group at **~30 MiB**, which trips +*before* the 32 MiB byte flush (and well before 64 MiB): so 32 and 64 MiB +are both **row-capped** at ~3 groups of ~30 MiB (byte-identical files), and +only 16 MiB (~590 k rows → ~15 MiB, under the cap) is genuinely +byte-governed → 5 finer groups. + +**Correction to an earlier framing.** A prior draft of this finding called +the row cap a "gap" and suggested raising `max_row_group_size` "so the byte +threshold bites." That is backwards: raising the cap would let the 32/64 MiB +byte flush govern and produce *fewer, coarser* groups (~2 of ~32 MiB) — the +**wrong** direction for pruning. The row cap is a granularity *floor* that +is currently *helping*. The lever that unambiguously improves window +pruning is a **smaller byte threshold** (16 MiB halves window +materialization here), the §7 authoritative-sweep question — not a larger +row cap. Making group sizing byte-*uniform* (raising the cap) is a separate +predictability choice, not a pruning win. Net: **finer effective groups win +the materialization trade at sub-1% disk cost, and the byte-threshold +*value* — not the row cap — is the lever.** **Conclusion.** 32 MiB stays a reasonable, defensible shipped default (it already delivers the pruning mechanism the RFC0036.2 gate enforces). The diff --git a/docs/rfcs/0031-comparative-evaluation-loki.md b/docs/rfcs/0031-comparative-evaluation-loki.md index 6d0e0c3b..2352fc8d 100644 --- a/docs/rfcs/0031-comparative-evaluation-loki.md +++ b/docs/rfcs/0031-comparative-evaluation-loki.md @@ -1,7 +1,7 @@ --- rfc: 0031 title: Comparative evaluation against Grafana Loki -status: validated +status: accepted author: Jens Holdgaard Pedersen drafting-assistance: Claude created: 2026-07-11 @@ -11,7 +11,8 @@ superseded-by: — # RFC 0031 — Comparative evaluation against Grafana Loki -> **Status note.** **`validated`** (2026-07-21). The §7 gate set is +> **Status note.** **`accepted`** (2026-07-22, maintainer sign-off — the +> terminal state). Reached `validated` 2026-07-21: the §7 gate set is > frozen and CI-enforcing (`green` via the asserting ci-runner series, > §9.13–§9.18), and the first fully authoritative comparative run on > `baseline-8vcpu-32gib` (§9.24, PR #583) passed all 11 frozen gate diff --git a/docs/rfcs/0034-d1-rescope.md b/docs/rfcs/0034-d1-rescope.md index 61e92cc7..8d310afc 100644 --- a/docs/rfcs/0034-d1-rescope.md +++ b/docs/rfcs/0034-d1-rescope.md @@ -1,7 +1,7 @@ --- rfc: 0034 title: D1 re-scope — ingest throughput is a per-node capacity on baseline hardware, not a per-core rate -status: specified +status: accepted author: Jens Holdgaard Pedersen drafting-assistance: Claude created: 2026-07-20 @@ -11,9 +11,10 @@ superseded-by: — # RFC 0034 — D1 re-scope -> **Status note.** **`specified`**, with all three §5 criteria now in -> force (2026-07-21); the terminal flip to `accepted` is the -> maintainer's sign-off, not this note. RFC0034.1 is **enacted** by +> **Status note.** **`accepted`** (2026-07-22, maintainer sign-off — the +> terminal state for this re-scope RFC, which has no thesis-gate of its +> own to `validate`). All three §5 criteria were in force from +> 2026-07-21. RFC0034.1 is **enacted** by > this change: the `docs/benchmarks.md` § D1 block is recast per-node > (≥ 100 000 lines/s on `baseline-8vcpu-32gib`, multi-tenant, shared > commit stream, p99 ack ≤ 200 ms at the sustained rate) with the old diff --git a/docs/rfcs/0036-write-side-layout.md b/docs/rfcs/0036-write-side-layout.md index 41cccb0c..9a615a78 100644 --- a/docs/rfcs/0036-write-side-layout.md +++ b/docs/rfcs/0036-write-side-layout.md @@ -1,7 +1,7 @@ --- rfc: 0036 title: Write-side layout — compacted-partition clustering and row-group sizing -status: validated +status: accepted author: Jens Holdgaard Pedersen drafting-assistance: Claude created: 2026-07-21 @@ -11,9 +11,11 @@ superseded-by: — # RFC 0036 — Write-side layout -> **Status note.** **`validated`** (2026-07-22) — all five §5 scenarios -> green, plus the comparative evidence below. `accepted` is a maintainer -> flip. **RFC0036.1** (footer inspection — compacted threshold, +> **Status note.** **`accepted`** (2026-07-22, maintainer sign-off — the +> terminal state). Reached `validated` the same day on all five §5 +> scenarios green plus the comparative evidence below; the §7 threshold +> sweep (§9.28) and its row-cap finding landed with it. +> **RFC0036.1** (footer inspection — compacted threshold, > `sorting_columns`, per-group service min/max — plus the §6 merge > property), **RFC0036.3** (D3 file-band + forced-spill memory bound), > **RFC0036.4** (shuffled-listing byte-identity rebuild), and @@ -538,13 +540,14 @@ Mapped to `CLAUDE.md` §6.2; techniques per §5 scenario id: one-service window materialises **half** the bytes at 16 MiB (14.66 MiB) vs 32/64 MiB (28.94 MiB), same answer, for that +0.80% disk cost — the pruning-granularity trade is *good* on compressible data and improves as - T shrinks; (3) 32 and 64 MiB are indistinguishable because arrow's default - 1,048,576-row group cap (the compacted writer sets no - `max_row_group_size`) trips before either byte threshold on the 2.16 M-row - corpus — so above ~16 MiB the **byte** threshold does not govern - granularity, the row cap does (a gap vs §3.3's byte-driven premise; - actionable follow-up: `set_max_row_group_size` on the compacted writer so - the byte threshold bites). **Disposition:** 32 MiB stands as the + T shrinks; (3) 32 and 64 MiB are near-identical because arrow's default + 1,048,576-row group cap (no `max_row_group_size` is set) fills a group at + ~30 MiB on this ~30 B/row corpus and so trips before either byte + threshold — a granularity *floor* finer than 32/64 MiB. It is **not** a + bug to "fix" by raising the cap: doing so would let 32/64 MiB coarsen (~2 + groups), the wrong way for pruning. The pruning lever is a *smaller byte + threshold* (16 MiB, byte-governed → finer groups), the §7 sweep question — + not the row cap. **Disposition:** 32 MiB stands as the shipped default (it already delivers the mechanism the RFC0036.2 gate enforces); the data leans toward *smaller* thresholds, so **16 MiB is flagged as a candidate for the authoritative sweep and a maintainer diff --git a/docs/roadmap.md b/docs/roadmap.md index 3fec1c60..13261b3a 100644 --- a/docs/roadmap.md +++ b/docs/roadmap.md @@ -168,12 +168,12 @@ captured by B1/B2 (see `benchmarks.md` §2 / §7). | 0028 | Build-feedback program (test-harness + workspace decomposition) | `green` | | 0029 | OIDC bearer layer (issuer-agnostic, Dex-validated) | `green` | | 0030 | TLS/mTLS on the data-plane listeners | `green` | -| 0031 | Comparative evaluation against Grafana Loki | **`validated`** — all four must-win classes measured, §7 gates frozen and CI-enforcing; the first fully authoritative run (`benchmarks.md` §9.24, `baseline-8vcpu-32gib`) passed all 11 frozen gate decisions; losses published per §5 (L6 *storage* is a recorded diagnostic, not a win) | +| 0031 | Comparative evaluation against Grafana Loki | **`accepted`** (2026-07-22) — all four must-win classes measured, §7 gates frozen and CI-enforcing; the first fully authoritative run (`benchmarks.md` §9.24, `baseline-8vcpu-32gib`) passed all 11 frozen gate decisions; losses published per §5 (L6 *storage* is a recorded diagnostic, not a win) | | 0032 | Query-schema and cost-model resource for the MCP surface | `green` | | 0033 | Cached template-map artifact | `green` | -| 0034 | D1 re-scope: per-node ingest-throughput bar | `specified` — enacted: RFC0034.1–.3 satisfied by the §9.20–§9.23 measurement series; `accepted` is a maintainer flip | +| 0034 | D1 re-scope: per-node ingest-throughput bar | **`accepted`** (2026-07-22) — enacted: RFC0034.1–.3 satisfied by the §9.20–§9.23 measurement series (a re-scope RFC with no thesis-gate of its own; `specified`→`accepted`) | | 0035 | Ingest concurrency (ordered mining, concurrent encode/publish) | `green` — §9.22 A/B plus the §9.23 asserting soak; the #578 sweep-publish durability window closed alongside | -| 0036 | Write-side layout (compaction-time service/time sort) | `specified` — awaiting maintainer design review before `red` | +| 0036 | Write-side layout (compaction-time service/time sort) | **`accepted`** (2026-07-22) — all five §5 green (real compaction); RFC0036.2 scanned-count gate + in-repo before/after (§9.27, 1.43×); baseline no-regression (§9.26) + §7 threshold sweep (§9.28); the comparative harness single-file limit + the row-cap interaction are documented follow-ups | **Crates — all ten product crates are implemented** (`ourios-core`, `-miner`, `-wal`, `-parquet`, `-ingester`, `-querier`, `-server`,