From 3dcf5e2550af48703493a81524a55ab1c629d38d Mon Sep 17 00:00:00 2001 From: Jens Holdgaard Pedersen Date: Sun, 14 Jun 2026 21:06:18 +0200 Subject: [PATCH 1/2] =?UTF-8?q?docs:=20enact=20RFC=200012=20=E2=80=94=20?= =?UTF-8?q?=C2=A72=20pillar-#2=20is=20a=20logical=2050=E2=80=93200=C3=97,?= =?UTF-8?q?=20not=20byte-level?= MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit Maintainer-approved enactment of RFC 0012 (the meta: RFC; CLAUDE.md's footer requires meta: RFC + majority approval — satisfied). - CLAUDE.md §2 pillar #2: the 50–200× is reworded as a **logical** reduction whose payoff is query pruning (gates B1/B2), not on-disk bytes vs a byte codec; A1 named a recorded diagnostic (RFC 0011). The "single biggest engineering risk" sentence is preserved verbatim. - CLAUDE.md footer: 2026-06-14 changelog line + Last-updated bump. - Coupled reconciliations (RFC 0012 §3.2): benchmarks.md §2 A1 "Why this bar" → "Why recorded (diagnostic)"; README.md template-miner bullet; RFC 0001 §1 summary (accepted-RFC factual reword + RFC 0011 pointer). - RFC 0012 → `accepted`. After this, a repo grep for the framing phrases ("before any byte-level codec" / "over a competent byte codec") returns only RFC 0011 and RFC 0012 (which quote the old wording to describe the change) — RFC0012.2 satisfied. Co-Authored-By: Claude Opus 4.8 (1M context) --- CLAUDE.md | 22 ++++++++++++++------ README.md | 4 ++-- docs/benchmarks.md | 9 +++++--- docs/rfcs/0001-template-miner.md | 6 ++++-- docs/rfcs/0012-claude-md-pillar-2-wording.md | 11 +++++++++- 5 files changed, 38 insertions(+), 14 deletions(-) diff --git a/CLAUDE.md b/CLAUDE.md index 952ea1c5..e9d1f9fa 100644 --- a/CLAUDE.md +++ b/CLAUDE.md @@ -51,9 +51,16 @@ decision, not a PR-level one. and page indexes. Query performance comes from skipping row groups via footer reads, not from scanning. 2. **Drain-derived online template mining.** Log lines collapse to - `(template_id, params)` at ingest time. This is where the 50–200× - compression comes from — before any byte-level codec runs. Correctness - of this layer is the single biggest engineering risk in the project. + `(template_id, params)` at ingest time — a **logical** 50–200× + reduction (many near-identical lines become rows keyed by one small, + stable `template_id`). That reduction is what lets a selective query + read a handful of row groups instead of scanning the corpus, so the + payoff is **query pruning** (pillar #1's footer-skip; benchmark gates + B1/B2), **not** fewer on-disk bytes than a byte codec — RFC 0011 + showed a whole-stream codec captures the same redundancy, so the + on-disk-compression-vs-zstd ratio (A1) is a recorded diagnostic, not a + gate. Correctness of this layer is the single biggest engineering risk + in the project. 3. **DataFusion as the query engine.** We do not write a vectorised execution engine. We hand DataFusion logical plans and let it work. @@ -394,11 +401,14 @@ the wrong thing. --- -*Last updated: 2026-05-13. Original draft 2026-04-23. 2026-04-26 revision +*Last updated: 2026-06-14. Original draft 2026-04-23. 2026-04-26 revision aligned §7 with the present-day repo and added §6.7 + an mdBook entry to §6.6 (commits 1c806f5..84eed86). 2026-05-13 revision adds the "Tests are specifications, not friction" bullet to §6.2, codifying a discipline called out by Dave Farley as a recurring AI-assisted-development failure mode (informal `meta:` RFC waiver per maintainer discussion of the same -date; precedent: b50067d). This document is load-bearing; further changes -require a `meta:` RFC and majority maintainer approval.* +date; precedent: b50067d). 2026-06-14 revision rewords §2 pillar #2 so the +50–200× reads as a **logical** reduction (query pruning, benchmark gates +B1/B2) rather than an on-disk-bytes-beat-the-codec claim, per **RFC 0012** +(maintainer-approved `meta:` RFC) propagating RFC 0011's A1 demotion. This document is load-bearing; further changes require a `meta:` +RFC and majority maintainer approval.* diff --git a/README.md b/README.md index b8021cd7..a4ac8307 100644 --- a/README.md +++ b/README.md @@ -21,8 +21,8 @@ parts: - **Apache Parquet** — columnar storage, with predicate pushdown via row-group statistics, bloom filters, and page indexes. - **A Drain-derived online template miner** — collapses log lines to - `(template_id, params)` at ingest time, before any byte-level codec - runs. + `(template_id, params)` at ingest time: a *logical* 50–200× reduction + that drives query pruning (not a bytes-vs-codec claim; see RFC 0011). - **Apache DataFusion** — a production-ready vectorised SQL execution engine. We hand it logical plans; it does the work. diff --git a/docs/benchmarks.md b/docs/benchmarks.md index 5d539437..eb011580 100644 --- a/docs/benchmarks.md +++ b/docs/benchmarks.md @@ -121,9 +121,12 @@ codecs run. - **Falsifier**: if any representative corpus yields ≤ 2× improvement over zstd-alone, the template-mining pillar is not pulling its weight on that class of logs. Open an RFC. -- **Why this bar**: the project's headline claim (§2, CLAUDE.md) is - "50–200× over raw, ≥ 5× over a competent byte codec." Below 3× the - headline is dishonest. +- **Why recorded** (diagnostic, not a bar): `CLAUDE.md` §2 pillar #2 + describes a **logical** 50–200× reduction (lines → `(template_id, + params)`) whose payoff is query pruning (B1/B2), *not* on-disk bytes vs + a byte codec. A1 tracks the on-disk ratio as the columnar queryability + premium + a codec-regression guard; RFC 0011 (`accepted`) demoted it + from a gate to this diagnostic. ### A2 — Bytes per line, amortised diff --git a/docs/rfcs/0001-template-miner.md b/docs/rfcs/0001-template-miner.md index 456b7a35..c139fab6 100644 --- a/docs/rfcs/0001-template-miner.md +++ b/docs/rfcs/0001-template-miner.md @@ -80,8 +80,10 @@ rather than a property-test exception `[§3.3]`, bounds parameter values at 256 B with overflow to a side `body` column `[§3.2]`, and tracks template structural changes via a monotonic `template_version` so that schema drift across deploys is a first-class query rather than -a silent count drop `[§3.5]`. The compression target is 50–200× over -raw bytes before any byte-level codec runs. +a silent count drop `[§3.5]`. The 50–200× figure is a **logical** +reduction (lines → `(template_id, params)`), realised as query pruning +(gates B1/B2), not on-disk bytes versus a byte codec — see RFC 0011, which +demoted the compression-vs-zstd ratio (A1) to a diagnostic. ## 2. Motivation diff --git a/docs/rfcs/0012-claude-md-pillar-2-wording.md b/docs/rfcs/0012-claude-md-pillar-2-wording.md index 761e8300..a503d851 100644 --- a/docs/rfcs/0012-claude-md-pillar-2-wording.md +++ b/docs/rfcs/0012-claude-md-pillar-2-wording.md @@ -1,7 +1,7 @@ --- rfc: 0012 title: "meta: CLAUDE.md §2 pillar-#2 wording — template mining's 50–200× is a logical reduction, not on-disk bytes" -status: drafted +status: accepted author: Jens Holdgaard Pedersen drafting-assistance: Claude created: 2026-06-14 @@ -11,6 +11,15 @@ superseded-by: — # RFC 0012 — meta: CLAUDE.md §2 pillar-#2 wording +> **Status note. `accepted` (2026-06-14, maintainer-approved + enacted.)** +> The `CLAUDE.md` §2 pillar-#2 reword (§3.1) and all three coupled +> reconciliations (§3.2 — `benchmarks.md` §2, `README.md`, RFC 0001 §1) +> were applied in the enacting PR; RFC 0001's accepted prose took the +> recommended factual reword (with the RFC 0011 pointer). The §7 footer +> changelog line + `Last updated` bump landed in the same diff. This +> meta-RFC required majority maintainer approval per `CLAUDE.md`'s footer; +> that gate is satisfied. + > **This is a `meta:` RFC.** It proposes a change to `CLAUDE.md`, which > its own footer declares load-bearing: *"This document is load-bearing; > further changes require a `meta:` RFC and majority maintainer From 05cebfc6261dd9182a0015d86c08af694267a4c9 Mon Sep 17 00:00:00 2001 From: Jens Holdgaard Pedersen Date: Sun, 14 Jun 2026 21:18:28 +0200 Subject: [PATCH 2/2] docs: link RFC 0011 in README + rewrap CLAUDE.md footer (Copilot) MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit Two format nits on the RFC 0012 enactment: link the README's RFC 0011 reference like its other RFC bullets, and rewrap the CLAUDE.md footer so "This document is load-bearing…" is its own wrapped line rather than jammed after the new changelog sentence. Co-Authored-By: Claude Opus 4.8 (1M context) --- CLAUDE.md | 5 +++-- README.md | 3 ++- 2 files changed, 5 insertions(+), 3 deletions(-) diff --git a/CLAUDE.md b/CLAUDE.md index e9d1f9fa..8f90fa35 100644 --- a/CLAUDE.md +++ b/CLAUDE.md @@ -410,5 +410,6 @@ mode (informal `meta:` RFC waiver per maintainer discussion of the same date; precedent: b50067d). 2026-06-14 revision rewords §2 pillar #2 so the 50–200× reads as a **logical** reduction (query pruning, benchmark gates B1/B2) rather than an on-disk-bytes-beat-the-codec claim, per **RFC 0012** -(maintainer-approved `meta:` RFC) propagating RFC 0011's A1 demotion. This document is load-bearing; further changes require a `meta:` -RFC and majority maintainer approval.* +(maintainer-approved `meta:` RFC) propagating RFC 0011's A1 demotion. +This document is load-bearing; further changes require a `meta:` RFC and +majority maintainer approval.* diff --git a/README.md b/README.md index a4ac8307..fa14b2cb 100644 --- a/README.md +++ b/README.md @@ -22,7 +22,8 @@ parts: row-group statistics, bloom filters, and page indexes. - **A Drain-derived online template miner** — collapses log lines to `(template_id, params)` at ingest time: a *logical* 50–200× reduction - that drives query pruning (not a bytes-vs-codec claim; see RFC 0011). + that drives query pruning (not a bytes-vs-codec claim; see + [RFC 0011](docs/rfcs/0011-a1-rescope.md)). - **Apache DataFusion** — a production-ready vectorised SQL execution engine. We hand it logical plans; it does the work.