Skip to content
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
23 changes: 17 additions & 6 deletions CLAUDE.md
Original file line number Diff line number Diff line change
Expand Up @@ -51,9 +51,16 @@ decision, not a PR-level one.
and page indexes. Query performance comes from skipping row groups via
footer reads, not from scanning.
2. **Drain-derived online template mining.** Log lines collapse to
`(template_id, params)` at ingest time. This is where the 50–200×
compression comes from — before any byte-level codec runs. Correctness
of this layer is the single biggest engineering risk in the project.
`(template_id, params)` at ingest time — a **logical** 50–200×
reduction (many near-identical lines become rows keyed by one small,
stable `template_id`). That reduction is what lets a selective query
read a handful of row groups instead of scanning the corpus, so the
payoff is **query pruning** (pillar #1's footer-skip; benchmark gates
B1/B2), **not** fewer on-disk bytes than a byte codec — RFC 0011
showed a whole-stream codec captures the same redundancy, so the
on-disk-compression-vs-zstd ratio (A1) is a recorded diagnostic, not a
gate. Correctness of this layer is the single biggest engineering risk
in the project.
3. **DataFusion as the query engine.** We do not write a vectorised
execution engine. We hand DataFusion logical plans and let it work.

Expand Down Expand Up @@ -394,11 +401,15 @@ the wrong thing.

---

*Last updated: 2026-05-13. Original draft 2026-04-23. 2026-04-26 revision
*Last updated: 2026-06-14. Original draft 2026-04-23. 2026-04-26 revision
aligned §7 with the present-day repo and added §6.7 + an mdBook entry to
§6.6 (commits 1c806f5..84eed86). 2026-05-13 revision adds the "Tests are
specifications, not friction" bullet to §6.2, codifying a discipline
called out by Dave Farley as a recurring AI-assisted-development failure
mode (informal `meta:` RFC waiver per maintainer discussion of the same
date; precedent: b50067d). This document is load-bearing; further changes
require a `meta:` RFC and majority maintainer approval.*
date; precedent: b50067d). 2026-06-14 revision rewords §2 pillar #2 so the
50–200× reads as a **logical** reduction (query pruning, benchmark gates
B1/B2) rather than an on-disk-bytes-beat-the-codec claim, per **RFC 0012**
(maintainer-approved `meta:` RFC) propagating RFC 0011's A1 demotion.
This document is load-bearing; further changes require a `meta:` RFC and
majority maintainer approval.*
5 changes: 3 additions & 2 deletions README.md
Original file line number Diff line number Diff line change
Expand Up @@ -21,8 +21,9 @@ parts:
- **Apache Parquet** — columnar storage, with predicate pushdown via
row-group statistics, bloom filters, and page indexes.
- **A Drain-derived online template miner** — collapses log lines to
`(template_id, params)` at ingest time, before any byte-level codec
runs.
`(template_id, params)` at ingest time: a *logical* 50–200× reduction
that drives query pruning (not a bytes-vs-codec claim; see
[RFC 0011](docs/rfcs/0011-a1-rescope.md)).
- **Apache DataFusion** — a production-ready vectorised SQL execution
engine. We hand it logical plans; it does the work.

Expand Down
9 changes: 6 additions & 3 deletions docs/benchmarks.md
Original file line number Diff line number Diff line change
Expand Up @@ -121,9 +121,12 @@ codecs run.
- **Falsifier**: if any representative corpus yields ≤ 2× improvement
over zstd-alone, the template-mining pillar is not pulling its
weight on that class of logs. Open an RFC.
- **Why this bar**: the project's headline claim (§2, CLAUDE.md) is
"50–200× over raw, ≥ 5× over a competent byte codec." Below 3× the
headline is dishonest.
- **Why recorded** (diagnostic, not a bar): `CLAUDE.md` §2 pillar #2
describes a **logical** 50–200× reduction (lines → `(template_id,
params)`) whose payoff is query pruning (B1/B2), *not* on-disk bytes vs
a byte codec. A1 tracks the on-disk ratio as the columnar queryability
premium + a codec-regression guard; RFC 0011 (`accepted`) demoted it
from a gate to this diagnostic.

### A2 — Bytes per line, amortised

Expand Down
6 changes: 4 additions & 2 deletions docs/rfcs/0001-template-miner.md
Original file line number Diff line number Diff line change
Expand Up @@ -80,8 +80,10 @@ rather than a property-test exception `[§3.3]`, bounds parameter
values at 256 B with overflow to a side `body` column `[§3.2]`, and
tracks template structural changes via a monotonic `template_version`
so that schema drift across deploys is a first-class query rather than
a silent count drop `[§3.5]`. The compression target is 50–200× over
raw bytes before any byte-level codec runs.
a silent count drop `[§3.5]`. The 50–200× figure is a **logical**
reduction (lines → `(template_id, params)`), realised as query pruning
(gates B1/B2), not on-disk bytes versus a byte codec — see RFC 0011, which
demoted the compression-vs-zstd ratio (A1) to a diagnostic.

## 2. Motivation

Expand Down
11 changes: 10 additions & 1 deletion docs/rfcs/0012-claude-md-pillar-2-wording.md
Original file line number Diff line number Diff line change
@@ -1,7 +1,7 @@
---
rfc: 0012
title: "meta: CLAUDE.md §2 pillar-#2 wording — template mining's 50–200× is a logical reduction, not on-disk bytes"
status: drafted
status: accepted
author: Jens Holdgaard Pedersen <jens@holdgaard.org>
drafting-assistance: Claude
created: 2026-06-14
Expand All @@ -11,6 +11,15 @@ superseded-by: —

# RFC 0012 — meta: CLAUDE.md §2 pillar-#2 wording

> **Status note. `accepted` (2026-06-14, maintainer-approved + enacted.)**
> The `CLAUDE.md` §2 pillar-#2 reword (§3.1) and all three coupled
> reconciliations (§3.2 — `benchmarks.md` §2, `README.md`, RFC 0001 §1)
> were applied in the enacting PR; RFC 0001's accepted prose took the
> recommended factual reword (with the RFC 0011 pointer). The §7 footer
> changelog line + `Last updated` bump landed in the same diff. This
> meta-RFC required majority maintainer approval per `CLAUDE.md`'s footer;
> that gate is satisfied.

> **This is a `meta:` RFC.** It proposes a change to `CLAUDE.md`, which
> its own footer declares load-bearing: *"This document is load-bearing;
> further changes require a `meta:` RFC and majority maintainer
Expand Down
Loading