From 3e1d72998eb2ddf4a913262843a601c938167453 Mon Sep 17 00:00:00 2001 From: Jens Holdgaard Pedersen Date: Tue, 9 Jun 2026 00:36:47 +0200 Subject: [PATCH 1/6] =?UTF-8?q?docs(rfc-0001):=20define=20the=20=C2=A76.6?= =?UTF-8?q?=20reader-render=20contract=20+=20lossy=20warning=20marker?= MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit H7.3 referenced "the §6.6 warning marker" that §6.6 never defined. Add a Reader render contract subsection defining the marker as a structured, out-of-band per-row Reconstruction signal (Faithful | RetainedVerbatim) — not a body-byte mutation — and pinning the lossy short-circuit (return body verbatim, do not call reconstruct). Reject a body-byte annotation as the marker (it would break the verbatim guarantee). The clean-path read-time template registry is explicitly deferred to RFC 0007. Update H7.3 and the §8 reader-behaviour test to resolve to the defined signal. RFC stays specified; no schema or algorithm change. Co-Authored-By: Claude Opus 4.8 --- docs/rfcs/0001-template-miner.md | 114 +++++++++++++++++++++++++++---- 1 file changed, 101 insertions(+), 13 deletions(-) diff --git a/docs/rfcs/0001-template-miner.md b/docs/rfcs/0001-template-miner.md index ed4e85f71..e755af4c2 100644 --- a/docs/rfcs/0001-template-miner.md +++ b/docs/rfcs/0001-template-miner.md @@ -386,8 +386,11 @@ direction and the primary obligation lives in those other RFCs. > **Scenario H7.3 — Reader emits body verbatim when lossy_flag is true** > - **Given** a record with `lossy_flag = true` > - **When** the reader renders the row -> - **Then** the rendered output is the `body` column verbatim -> with the §6.6 warning marker +> - **Then** the rendered bytes are the `body` column verbatim +> (byte-for-byte, no prefix or in-band marker) +> - **And** the rendered row carries the §6.6 reconstruction signal +> `Reconstruction::RetainedVerbatim` (the out-of-band warning +> marker the §6.6 *Reader render contract* defines) > - **And** `reconstruct()` is NOT called for that row > **Scenario H7.4 — Widened literal slot reconstructs via STR fallback** @@ -1418,6 +1421,22 @@ any param has `type_tag == OVERFLOW`. ### 6.6 Body reconstruction `[§3.3]` +> **Amendment 2026-06-08 (reader render contract).** H7.3 (§5) +> referenced "the §6.6 warning marker," but §6.6 defined no such +> marker — the prose *Reader behaviour* paragraph it replaced +> described one only informally. This amendment adds the *Reader +> render contract* subsection below, defining the marker as a +> structured, out-of-band per-row `Reconstruction` signal +> (`Faithful` | `RetainedVerbatim`) the reader attaches to the +> rendered row — never a mutation of the body bytes — and pinning +> the lossy short-circuit H7.3 requires (return `body` verbatim, +> do not call `reconstruct`). The clean-path read-time template +> lookup (a registry mapping `(template_id, template_version) → +> tokens` at read time) is explicitly **out of scope** and deferred +> to the querier's reader-materialisation story (RFC 0007). RFC +> 0001 stays `specified`; this clarifies the §6.6 contract, it does +> not change the on-disk schema or the mining algorithm. + **Capture, always.** Every successful tokenization in §6.2 step 1 populates the `separators` array with the bytes between adjacent tokens (and the leading and trailing bytes of the line). The array @@ -1468,13 +1487,76 @@ reconstruction from template + params + separators is still expected to match. The flag is reserved for the cases where the record genuinely cannot be reconstructed. -**Reader behaviour.** When rendering rows, the reader checks -`lossy_flag` first. For lossy rows it emits the `body` verbatim -with an explicit warning marker ("this row's body cannot be -reconstructed from the template; the displayed value is the -original bytes as ingested"). For non-lossy rows it calls -`reconstruct`. The reader never silently substitutes one for the -other. +#### Reader render contract + +The functions above run at *write* time: `reconstruct` is the +property-test oracle and the in-process renderer the miner +exercises while the template is in hand. The *reader* — the +read-side path that materialises a stored Parquet row back into +the effective original line for a query result — is a distinct +caller, and H7.3 pins the contract it must honour. + +**The render result is `(bytes, reconstruction)`.** Rendering a +stored row yields two things: the effective original line **bytes** +and a per-row **reconstruction signal** + +``` +enum Reconstruction { + Faithful, // bytes == ingested line, reconstructed from template + RetainedVerbatim, // bytes are the retained `body` column, not reconstructed +} +``` + +The `Reconstruction` signal **is** the "§6.6 warning marker" that +H7.3 references. It is structured, out-of-band metadata attached to +the rendered row — *not* a mutation of the body bytes. A consumer +(the DSL output layer, RFC 0007; a UI) renders `RetainedVerbatim` +as a "reconstruction not faithful — original line retained verbatim" +warning beside the row, exactly as it would render any other per-row +annotation. + +**A body-byte annotation is explicitly rejected as the marker.** +Prefixing the body with a sentinel string, wrapping it in a marker +character, or otherwise editing the bytes to carry the warning would +break the verbatim guarantee: the whole point of the lossy path is +that an operator asking "show me what was actually logged" gets the +ingested bytes back unchanged `[§3.3]`. The marker therefore lives +beside the bytes, never inside them. + +**Lossy path (H7.3).** For a row with `lossy_flag = true` — the +tokenizer-failure / explicit-rejection cases enumerated under +*`lossy_flag` semantics* above — the reader returns the `body` +column **verbatim** with `Reconstruction::RetainedVerbatim`, and +does **not** invoke `reconstruct`: no template lookup, no token +walk. The same short-circuit applies to the other rows §6.6 and +§6.5 mark as non-reconstructable when they carry a retained body — +an `OVERFLOW` param (§6.5), or a `body_kind = Structured` row whose +canonically-encoded body is itself the authoritative rendering +(§6.1). `reconstruct`'s own `lossy_flag` / `OVERFLOW` early returns +remain in place as a **defensive guard** for callers that reach it +anyway; the reader's contract is to short-circuit *before* that +call, so the guard is belt-and-braces, not the primary mechanism. + +**Clean path.** For a faithful row (`lossy_flag = false`, no +`OVERFLOW` param, template available) the reader invokes +`reconstruct(record)` and attaches `Reconstruction::Faithful`. This +path requires resolving `(template_id, template_version) → tokens` +against a template registry available at *read* time. That registry +— how the reader obtains the template a stored row was mined +against, given that the miner's in-memory tree is an ingest-side +structure — is a **separate concern and out of scope of this +amendment**. Today `reconstruct` is exercised only where the +template is already in hand (the write-side property test H7.1, and +H7.4); the read-time lookup mechanism is future work, tracked with +the querier's reader-materialisation story (RFC 0007). This +amendment pins the render contract and the lossy path; it does +**not** specify a template-registry design. + +The reader never silently substitutes one rendering for the other: +every rendered row carries its `Reconstruction` signal, and the +lossy/clean branch is selected solely by the row's `lossy_flag` +(and the §6.5 / §6.1 non-reconstructable cases), never inferred +from the bytes. **Property test.** For every row `r` in the corpus where `r.lossy_flag == false`: @@ -2138,10 +2220,16 @@ resolves bidirectionally between RFC and tests. *Covers:* RFC0001.12, RFC0001.13, RFC0001.14, RFC0001.15, RFC0001.16. -- **Reader behaviour test**: assert the §6.6 reader emits the - `body` column verbatim (with the warning marker) for - `lossy_flag = true` rows, and calls `reconstruct()` for the - rest. The reader never silently substitutes one for the other. +- **Reader behaviour test**: assert the §6.6 *Reader render + contract* — for a `lossy_flag = true` row the reader returns the + `body` bytes verbatim (no in-band prefix/marker) carrying + `Reconstruction::RetainedVerbatim`, and does not call + `reconstruct()`; for a faithful row it calls `reconstruct()` and + carries `Reconstruction::Faithful`. The reader never silently + substitutes one rendering for the other, and never mutates the + body bytes to carry the marker. (The clean-path read-time + template-registry lookup is out of scope of this scenario per + §6.6.) *Covers:* H7.3. - **Overflow-path tests**: synthesize a parameter exceeding the From b292cf720e44ad929a6aef337042ae8f7175b845 Mon Sep 17 00:00:00 2001 From: Jens Holdgaard Pedersen Date: Tue, 9 Jun 2026 00:53:32 +0200 Subject: [PATCH 2/6] docs(rfc-0001): scope the reader-render marker to String bodies; defer structured MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit A body_kind=Structured row renders its canonical body, but its Faithful-vs-not classification turns on the deferred OTLP-canonical-JSON decision (RFC0001.9). Scope the §6.6 Reconstruction marker + lossy/clean split to the implemented String path; structured-body rendering lands with RFC0001.9. Co-Authored-By: Claude Opus 4.8 --- docs/rfcs/0001-template-miner.md | 32 ++++++++++++++++++++------------ 1 file changed, 20 insertions(+), 12 deletions(-) diff --git a/docs/rfcs/0001-template-miner.md b/docs/rfcs/0001-template-miner.md index e755af4c2..a224d90f9 100644 --- a/docs/rfcs/0001-template-miner.md +++ b/docs/rfcs/0001-template-miner.md @@ -1528,14 +1528,22 @@ tokenizer-failure / explicit-rejection cases enumerated under *`lossy_flag` semantics* above — the reader returns the `body` column **verbatim** with `Reconstruction::RetainedVerbatim`, and does **not** invoke `reconstruct`: no template lookup, no token -walk. The same short-circuit applies to the other rows §6.6 and -§6.5 mark as non-reconstructable when they carry a retained body — -an `OVERFLOW` param (§6.5), or a `body_kind = Structured` row whose -canonically-encoded body is itself the authoritative rendering -(§6.1). `reconstruct`'s own `lossy_flag` / `OVERFLOW` early returns -remain in place as a **defensive guard** for callers that reach it -anyway; the reader's contract is to short-circuit *before* that -call, so the guard is belt-and-braces, not the primary mechanism. +walk. The same short-circuit applies to a `String`-body row carrying +an `OVERFLOW` param (§6.5): its retained body is returned verbatim +with `Reconstruction::RetainedVerbatim`. `reconstruct`'s own +`lossy_flag` / `OVERFLOW` early returns remain in place as a +**defensive guard** for callers that reach it anyway; the reader's +contract is to short-circuit *before* that call, so the guard is +belt-and-braces, not the primary mechanism. + +**Structured bodies are out of scope of this amendment.** A +`body_kind = Structured` row renders its canonically-encoded `body` +column, but whether that is `Reconstruction::Faithful` (the canonical +JSON round-trips the source `AnyValue`) turns on the OTLP-canonical +JSON encoding decision (RFC0001.9), which is deferred. This amendment +defines the `Reconstruction` marker for the implemented **`String`** +path; structured-body rendering and its classification land with +RFC0001.9. **Clean path.** For a faithful row (`lossy_flag = false`, no `OVERFLOW` param, template available) the reader invokes @@ -1553,10 +1561,10 @@ amendment pins the render contract and the lossy path; it does **not** specify a template-registry design. The reader never silently substitutes one rendering for the other: -every rendered row carries its `Reconstruction` signal, and the -lossy/clean branch is selected solely by the row's `lossy_flag` -(and the §6.5 / §6.1 non-reconstructable cases), never inferred -from the bytes. +every rendered `String`-body row carries its `Reconstruction` signal, +and the lossy/clean branch is selected solely by the row's +`lossy_flag` (and the §6.5 `OVERFLOW` case), never inferred from the +bytes. **Property test.** For every row `r` in the corpus where `r.lossy_flag == false`: From 65a8bf6a861c5a166757b32e1d8971209bd51abc Mon Sep 17 00:00:00 2001 From: Jens Holdgaard Pedersen Date: Tue, 9 Jun 2026 00:55:30 +0200 Subject: [PATCH 3/6] docs(rfc-0001): rust fence on the Reconstruction enum + generalize the marker wording Addresses review: language-tag the enum code fence; reword the RetainedVerbatim consumer guidance to "rendered from the retained body bytes, not reconstructed" (accurate for the lossy + overflow String paths). Co-Authored-By: Claude Opus 4.8 --- docs/rfcs/0001-template-miner.md | 4 ++-- 1 file changed, 2 insertions(+), 2 deletions(-) diff --git a/docs/rfcs/0001-template-miner.md b/docs/rfcs/0001-template-miner.md index a224d90f9..fed0df217 100644 --- a/docs/rfcs/0001-template-miner.md +++ b/docs/rfcs/0001-template-miner.md @@ -1500,7 +1500,7 @@ caller, and H7.3 pins the contract it must honour. stored row yields two things: the effective original line **bytes** and a per-row **reconstruction signal** -``` +```rust enum Reconstruction { Faithful, // bytes == ingested line, reconstructed from template RetainedVerbatim, // bytes are the retained `body` column, not reconstructed @@ -1511,7 +1511,7 @@ The `Reconstruction` signal **is** the "§6.6 warning marker" that H7.3 references. It is structured, out-of-band metadata attached to the rendered row — *not* a mutation of the body bytes. A consumer (the DSL output layer, RFC 0007; a UI) renders `RetainedVerbatim` -as a "reconstruction not faithful — original line retained verbatim" +as a "rendered from the retained `body` bytes, not reconstructed" warning beside the row, exactly as it would render any other per-row annotation. From ee6ba1c2618b0dbe79327949cf828b6a0df1ce61 Mon Sep 17 00:00:00 2001 From: Jens Holdgaard Pedersen Date: Tue, 9 Jun 2026 01:04:29 +0200 Subject: [PATCH 4/6] docs(rfc-0001): reframe structured render as out-of-scope, not deferred MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit Avoid contradicting §6.1 (the canonical-encoding rule is specified) and the RFC0001.9 scenario: the §6.6 reader-render amendment scopes the Reconstruction marker to String bodies; structured-body render classification is out of scope here, specified with RFC0001.9. Co-Authored-By: Claude Opus 4.8 --- docs/rfcs/0001-template-miner.md | 11 +++++------ 1 file changed, 5 insertions(+), 6 deletions(-) diff --git a/docs/rfcs/0001-template-miner.md b/docs/rfcs/0001-template-miner.md index fed0df217..9ae4e2deb 100644 --- a/docs/rfcs/0001-template-miner.md +++ b/docs/rfcs/0001-template-miner.md @@ -1538,12 +1538,11 @@ belt-and-braces, not the primary mechanism. **Structured bodies are out of scope of this amendment.** A `body_kind = Structured` row renders its canonically-encoded `body` -column, but whether that is `Reconstruction::Faithful` (the canonical -JSON round-trips the source `AnyValue`) turns on the OTLP-canonical -JSON encoding decision (RFC0001.9), which is deferred. This amendment -defines the `Reconstruction` marker for the implemented **`String`** -path; structured-body rendering and its classification land with -RFC0001.9. +column (the §6.1 canonical-encoding rule); how that render is +classified under the `Reconstruction` signal is out of scope here. +This amendment defines the marker for the implemented **`String`** +path; structured-body rendering and its classification are specified +with the RFC0001.9 structured short-circuit. **Clean path.** For a faithful row (`lossy_flag = false`, no `OVERFLOW` param, template available) the reader invokes From 724780dae2793f5abe92c7643b5152d247b147ac Mon Sep 17 00:00:00 2001 From: Jens Holdgaard Pedersen Date: Tue, 9 Jun 2026 01:08:05 +0200 Subject: [PATCH 5/6] docs(rfc-0001): do not claim RFC0001.9 specifies the Reconstruction signal RFC0001.9 covers structured-body rendering (the short-circuit) but does not define this amendment new Reconstruction marker. Reword: structured render is exercised by RFC0001.9; mapping a structured render to a Reconstruction signal is left open here, settled when structured render is wired. Co-Authored-By: Claude Opus 4.8 --- docs/rfcs/0001-template-miner.md | 10 +++++----- 1 file changed, 5 insertions(+), 5 deletions(-) diff --git a/docs/rfcs/0001-template-miner.md b/docs/rfcs/0001-template-miner.md index 9ae4e2deb..3fc6f6ef6 100644 --- a/docs/rfcs/0001-template-miner.md +++ b/docs/rfcs/0001-template-miner.md @@ -1538,11 +1538,11 @@ belt-and-braces, not the primary mechanism. **Structured bodies are out of scope of this amendment.** A `body_kind = Structured` row renders its canonically-encoded `body` -column (the §6.1 canonical-encoding rule); how that render is -classified under the `Reconstruction` signal is out of scope here. -This amendment defines the marker for the implemented **`String`** -path; structured-body rendering and its classification are specified -with the RFC0001.9 structured short-circuit. +column (the §6.1 canonical-encoding rule, exercised by the RFC0001.9 +structured short-circuit). This amendment defines the `Reconstruction` +marker for the implemented **`String`** path only; how a structured +render maps to a `Reconstruction` signal is left open here, to be +settled when structured-body rendering is wired. **Clean path.** For a faithful row (`lossy_flag = false`, no `OVERFLOW` param, template available) the reader invokes From c0082bd189eec8b6deba87f0abc6551820186e47 Mon Sep 17 00:00:00 2001 From: Jens Holdgaard Pedersen Date: Tue, 9 Jun 2026 01:11:31 +0200 Subject: [PATCH 6/6] docs(rfc-0001): scope the render-result intro to String-body rows The (bytes, reconstruction) intro read as a universal contract; scope it to body_kind=String up front so structured rows (out of scope here) aren't implied to carry a Reconstruction value. Co-Authored-By: Claude Opus 4.8 --- docs/rfcs/0001-template-miner.md | 10 ++++++---- 1 file changed, 6 insertions(+), 4 deletions(-) diff --git a/docs/rfcs/0001-template-miner.md b/docs/rfcs/0001-template-miner.md index 3fc6f6ef6..d3e7c360f 100644 --- a/docs/rfcs/0001-template-miner.md +++ b/docs/rfcs/0001-template-miner.md @@ -1494,11 +1494,13 @@ property-test oracle and the in-process renderer the miner exercises while the template is in hand. The *reader* — the read-side path that materialises a stored Parquet row back into the effective original line for a query result — is a distinct -caller, and H7.3 pins the contract it must honour. +caller, and H7.3 pins the contract it must honour. The contract below +covers `String`-body rows (`body_kind = String`); structured bodies +are out of scope (see the end of this subsection). -**The render result is `(bytes, reconstruction)`.** Rendering a -stored row yields two things: the effective original line **bytes** -and a per-row **reconstruction signal** +**For a `String`-body row, the render result is `(bytes, +reconstruction)`.** Rendering yields two things: the effective +original line **bytes** and a per-row **reconstruction signal** ```rust enum Reconstruction {