From 39fed6239c38e4594e55379bee1108bc1cc9b02d Mon Sep 17 00:00:00 2001 From: jp Date: Thu, 17 Sep 2026 19:41:40 -0700 Subject: [PATCH 1/2] docs(specs): design spec for a failure-shape index keyed on the action Records the class the 2g-c6 session named: an instrument returning a TRUE answer to a NARROWER question than the one asked, with a PASSING positive control. Seeds it with 25 measured instances (21 individually citable) from Oracle PART 25, this wave's mempalace cards, and 2g-c6's night. The class defeats the corpus's standing remedy. "Get a positive control before trusting a zero" assumes a control that can fail; here it passes, because it proves the reader works rather than that the reader was aimed at the question asked. Applies 2g's LAWS-INDEX arrival-phrasing model (n=11: 10 of 11 arrival phrasings return zero against cards that exist) and names #497's pre-mutation hook as the caller that already produces the query key. The minimal slice measures before it builds: a labelled corpus plus a recall harness that interleaves baseline and candidate per query, since #477 measured a card moving rank 13 to 1 on an unmodified tree within 30 minutes. Four falsifiers are written down, including the one most likely to produce a false success -- triggers authored by the lane that wrote the cards. No code. Spec only. Part of #503 Co-Authored-By: Claude Opus 5 --- FORK_CHANGELOG.md | 49 ++++ README.md | 1 + .../2026-09-17-failure-shape-index.yaml | 48 ++++ docs/specs/2026-09-17-failure-shape-index.md | 218 ++++++++++++++++++ website/public/llms-full.txt | 1 + 5 files changed, 317 insertions(+) create mode 100644 docs/fork-changes/2026-09-17-failure-shape-index.yaml create mode 100644 docs/specs/2026-09-17-failure-shape-index.md diff --git a/FORK_CHANGELOG.md b/FORK_CHANGELOG.md index aaa2201c76..70bf53ea28 100644 --- a/FORK_CHANGELOG.md +++ b/FORK_CHANGELOG.md @@ -18,6 +18,55 @@ The format is based on [Keep a Changelog](https://keepachangelog.com/en/1.1.0/). --- +## [2026-09-17] + + +### Added + + +- **design spec for indexing failure SHAPES, keyed on the action about to be taken** (`HEAD` — pending resolution) + A design spec only — `docs/specs/2026-09-17-failure-shape-index.md`, no code. + + The 2g-c6 session reported eight errors in one night with a single shape: an + instrument returning a **true** answer to a **narrower** question than the one + asked, with a **passing** positive control. The memorypalace wave of 2026-09-11 + recorded eleven more and Oracle PART 25 six, so the spec opens with a seed corpus + of 25 measured instances (21 individually citable — four of the 2g-c6 eight are + not described in #503, and the spec says so rather than rounding the number up). + + The class matters because the corpus's standing remedy cannot see it. "Get a + positive control before trusting a zero" assumes a failing control is available; + here the control passes, because it proves the reader works rather than that the + reader was aimed at the question asked. Every instance in the seed corpus was + green at the moment it was wrong. + + Two things shape the design. First, 2g's `docs/LAWS-INDEX.md` measured why a + correct corpus goes unread: n=11 arrival phrasings, 10 returning zero against + cards that exist, while the terms that do work are the ones you only know if you + already know the card. Retrieval succeeded exactly where it was not needed. + Second, #497's pre-mutation hook already builds a short **action sentence** at the + moment of a mutation and searches on it — so the query key exists and is produced + at the right time; what is missing is anything on the answering side that a verb + phrase can match. A failure-shape record would be the first record type in the + palace whose key is a trigger rather than a subject. + + The spec deliberately does not propose shipping an index first. Its minimal slice + is a labelled corpus plus a recall harness that interleaves baseline and candidate + per query (#477 measured a card moving rank 13 → 1 on an unmodified tree inside + 30 minutes, so two-halves measurement is invalid here). Four falsifiers are + written down, including the one most likely to produce a false success: triggers + authored by the same lane that wrote the cards would reproduce 2g's + "keyed-on terms work" regime and report it as a win, so the held-out phrasings + have to come from a lane that did not write the corpus. + + Builds on what already exists rather than proposing new surfaces: `source_kind` + (#452) classifies a curated shape card for free, #477's curated-above-transcript + ordering already puts it on the right side of the ranking, and `why`, `tunnels`, + `rate` and the AGE graph each have a role with no new route. + + *Files:* `docs/specs/2026-09-17-failure-shape-index.md` + + ## [2026-09-11] diff --git a/README.md b/README.md index 4249ec38bf..33b8ab10ba 100644 --- a/README.md +++ b/README.md @@ -290,6 +290,7 @@ The full enumeration of fork-ahead changes. The canonical source is [`docs/fork- | Description | Upstream PR | Fork commit | |---|---|---| +| design spec for indexing failure SHAPES, keyed on the action about to be taken | — | `HEAD` — pending resolution | | purge / prune / mined share one open-and-refuse sequence and one exit-code contract | — | [`459efab`](https://github.com/techempower-org/mempalace/commit/459efab) | | Entry shas resolve from the commit that added the entry file; strict check rejects an unresolved placeholder on main | — | [`b9d168a`](https://github.com/techempower-org/mempalace/commit/b9d168a) | | KG write-through commits once per batch instead of once per drawer | — | [`41f1251`](https://github.com/techempower-org/mempalace/commit/41f1251) | diff --git a/docs/fork-changes/2026-09-17-failure-shape-index.yaml b/docs/fork-changes/2026-09-17-failure-shape-index.yaml new file mode 100644 index 0000000000..5ce3dc5b9f --- /dev/null +++ b/docs/fork-changes/2026-09-17-failure-shape-index.yaml @@ -0,0 +1,48 @@ +seq: 149 +id: failure-shape-index +date: '2026-09-17' +bucket: Added +commit: HEAD +area: Docs +summary: design spec for indexing failure SHAPES, keyed on the action about to be taken +body: | + A design spec only — `docs/specs/2026-09-17-failure-shape-index.md`, no code. + + The 2g-c6 session reported eight errors in one night with a single shape: an + instrument returning a **true** answer to a **narrower** question than the one + asked, with a **passing** positive control. The memorypalace wave of 2026-09-11 + recorded eleven more and Oracle PART 25 six, so the spec opens with a seed corpus + of 25 measured instances (21 individually citable — four of the 2g-c6 eight are + not described in #503, and the spec says so rather than rounding the number up). + + The class matters because the corpus's standing remedy cannot see it. "Get a + positive control before trusting a zero" assumes a failing control is available; + here the control passes, because it proves the reader works rather than that the + reader was aimed at the question asked. Every instance in the seed corpus was + green at the moment it was wrong. + + Two things shape the design. First, 2g's `docs/LAWS-INDEX.md` measured why a + correct corpus goes unread: n=11 arrival phrasings, 10 returning zero against + cards that exist, while the terms that do work are the ones you only know if you + already know the card. Retrieval succeeded exactly where it was not needed. + Second, #497's pre-mutation hook already builds a short **action sentence** at the + moment of a mutation and searches on it — so the query key exists and is produced + at the right time; what is missing is anything on the answering side that a verb + phrase can match. A failure-shape record would be the first record type in the + palace whose key is a trigger rather than a subject. + + The spec deliberately does not propose shipping an index first. Its minimal slice + is a labelled corpus plus a recall harness that interleaves baseline and candidate + per query (#477 measured a card moving rank 13 → 1 on an unmodified tree inside + 30 minutes, so two-halves measurement is invalid here). Four falsifiers are + written down, including the one most likely to produce a false success: triggers + authored by the same lane that wrote the cards would reproduce 2g's + "keyed-on terms work" regime and report it as a win, so the held-out phrasings + have to come from a lane that did not write the corpus. + + Builds on what already exists rather than proposing new surfaces: `source_kind` + (#452) classifies a curated shape card for free, #477's curated-above-transcript + ordering already puts it on the right side of the ranking, and `why`, `tunnels`, + `rate` and the AGE graph each have a role with no new route. +files: +- docs/specs/2026-09-17-failure-shape-index.md diff --git a/docs/specs/2026-09-17-failure-shape-index.md b/docs/specs/2026-09-17-failure-shape-index.md new file mode 100644 index 0000000000..f9dd2624ad --- /dev/null +++ b/docs/specs/2026-09-17-failure-shape-index.md @@ -0,0 +1,218 @@ +# Failure-shape retrieval — index by the mistake, keyed on the action + +**Status:** design spec, no implementation. **Issue:** #503. **Date:** 2026-09-17. +**Related:** #497 (pre-mutation query — the caller), #451 / #452 (provenance), #477 +(curated ordering), #489 (curation gap), #428 (fleet memory umbrella). + +--- + +## 1. What a "failure shape" record is + +A **failure shape** is a recurring *mechanism* by which a correct-looking check +produces a wrong answer. The class this spec is built around, named by the 2g-c6 +session: + +> **An instrument returns a TRUE answer to a NARROWER question than the one asked, +> and its positive control passes.** + +Three properties make it worth a record type of its own: + +1. **It is invisible to the standard remedy.** The corpus's own rule is *get a + positive control before trusting a zero*. Here the control passes — it proves + the reader works, not that the reader was pointed at the question. Every + instance below had a green control. +2. **It is content-dissimilar from its own trigger.** The card that would have + prevented "deploy a cert change to a live service" is about *predicate + selection*. No content-similarity search connects those two strings. +3. **It recurs across unrelated domains.** Git plumbing, shell globbing, process + tables, RF debugging, markdown diffing. The domain is noise; the shape is signal. + +A record therefore has a different shape from a drawer. Minimum fields: + +| field | why | +|---|---| +| `shape` | one sentence, the mechanism — *not* the incident | +| `asked` / `answered` | the two questions, side by side. This pair **is** the card | +| `control_passed` | bool. A shape where the control catches it is a different class | +| `triggers[]` | arrival phrasings — what you would SAY or DO just before making it | +| `instances[]` | measured occurrences: date, lane, instrument, one-line evidence | +| `remedy` | the predicate to choose instead, stated as a check | + +`asked`/`answered` is load-bearing: it is the only field that distinguishes this +class from "a bug happened". If a proposed card cannot fill both, it is not a +failure shape. + +### Seed corpus — 25 measured instances, 21 individually citable + +**Oracle PART 25 (six, 2026-09-11):** + +| asked | instrument | actually answered | +|---|---|---| +| does the defect exist? | a read of a live uncommitted worktree | did it exist *at read time*? (fixed 2 min later) | +| what did this branch do? | `git diff main..branch` | tree-vs-tree — peer's additions render as this branch's deletions | +| is the file present after merge? | one hardcoded path | is it at *that* path? (a rename falsifies it; blob identical) | +| is it under the char limit? | a byte count | how many bytes? (23,309 B vs 22,936 chars, UTF-8 emoji) | +| did the merge take both sides? | "it merged" | did a command exit 0? (`rev-list --parents` shows the real answer) | +| are the links there? | `r'...([a-z\\-]+)\\.md'` in a quoted heredoc | is there a literal backslash? (matched nothing; reported "no links" about a landed fix) | + +**This wave, mempalace (eleven, 2026-09-11 → 09-17):** + +| asked | instrument | actually answered | +|---|---|---| +| is this sha on main? | `git cat-file -e` | does this object exist anywhere? | +| is this entry resolved? | `merge-base --is-ancestor HEAD HEAD` | is the tip its own ancestor? (trivially yes) | +| is this field set? | a field grep | does this text appear? | +| what added this file on main? | `git log --diff-filter=A` on a branch | what added it *here*? (squash orphans it) | +| did the data change? | diff of rendered output | did the output change? (loader normalises the differing field) | +| is the phrase present? | `grep -c 'CHANNEL LIST'` | is it present *on one line*? (hard-wrapped → 0 on correct text) | +| how many lines changed? | diff filter `^[+-][^+-]` | …excluding every markdown list line (all start `+- `) → "0 changed" | +| is the branch merged? | `git branch --merged` | is it an *ancestor*? (a squash merge is not) | +| is this file tracked? | `git check-ignore` | would it be ignored? (deleted 5 tracked files under an ignored dir) | +| is the command registered? | a grep for `add_parser(` | is it registered *on one line*? (multi-line call → false 0) | +| is the command wired? | `--help` smoke | does the parser build? (exits before dispatch is read) | + +**2g-c6 (eight in one night, four named in #503):** a count that measured code +formatting · a regex that measured spacing · `pgrep` matching its own wrapper · a +frequency table that cannot represent a zero. + +> ⚠️ Four of the 2g-c6 eight are not individually described in #503. The corpus is +> **25 on record, 21 citable**; any retrieval measurement must state which it used. + +--- + +## 2. Why index by arrival phrasing — 2g's measurement + +`~/Projects/2g/docs/LAWS-INDEX.md` exists because a 468 KB corpus of correct cards +was unreachable in practice. Its author measured why, n=11, phrasing each situation +the way an agent actually meets it: + +``` +ARRIVAL PHRASINGS 10 of 11 return ZERO, and every one names a card that EXISTS + "false clean" 0 · "silently skips" 0 · "option parsing" 0 · "stale issue" 0 + "already fixed" 0 · "double count" 0 · "wrong denominator" 0 · "premise moved" 0 + "parsed as an option" 0 · "publish the fix" 0 ("partial enumeration" 1 — lone hit) + +KEYED-ON TERMS work, if you already know the word + "positive control" 10 · "truncat" 17 · "denominator" 14 · "specimen" 7 + +TOO COMMON return hits, locate nothing + control 113 · instrument 107 · measured 138 · zero 95 · grep 105 +``` + +Three regimes, and the middle one is the trap: **retrieval works exactly when you +already know the card exists.** The index's response was not to rewrite the cards +but to add a lookup keyed on *"what am I about to do / what would I say"* — the +`if you would say…` column. It cites **search anchors and commit shas, never line +numbers**, after line numbers broke on the very commit that made the index findable. + +Two properties transfer directly to the palace: + +- **Key on the arrival, not the subject.** The palace indexes what a drawer *says*. + An agent arrives with what it is *about to do*. Those are different strings, and + §1's property 2 says they are not content-similar. +- **Cite something stable under motion.** A palace analogue of the anchor/sha rule: + cite a shape by a stable `shape_id`, never by drawer id or rank, both of which + move on every re-mine (measured: a card moved rank 13 → 1 on an unmodified tree + within 30 minutes, #477). + +--- + +## 3. Retrieval keys on the action sentence + +**#497 is the caller.** Its pre-mutation hook already does the hard half: it matches +mutation shapes (`systemctl restart|reload|stop`, `deploy.sh`, `scp`/`rsync` to a +host, writes under `/etc`, `*.service`, `git push --force`, `rm -rf` outside the +repo, cert/key writes), **builds a short action sentence**, and runs +`mempalace search "" --limit 3 --format compact` with a 2 s timeout, +advisory-only, exit 0 always. + +So the query key already exists and is already being produced at the right moment. +What is missing is anything on the answering side that a *verb phrase* can match. + +``` + #497 hook ──"about to deploy a cert change to a live service"──▶ search + │ + today: content similarity over drawers ──▶ transcripts that mention certs + wanted: trigger match over shape records ──▶ "a check that passed on the + narrower question" + its remedy +``` + +The auto-query hook fires on **topic changes**; #497's own observation is that the +expensive misses happen at **actions**, where no topic changed. A failure-shape +record is the first record type in the palace whose key is a trigger rather than a +subject, which is why it is the right thing to index first: it is the only class +where the caller's string and the card's string have no lexical overlap by design. + +--- + +## 4. What the palace already has, and what it lacks + +**Has — reusable today:** + +- **`source_kind` (#452)** — `transcript` · `memory` · `diary` · `file` · `unknown`, + plus `source_stale`, `source_indexed_at`, `provenance_note`. A shape card is a + curated file, so it inherits the highest-trust classification for free, and + `all_transcript()` / `no_curated_source()` already exist as result-set predicates. +- **Curated ordering (#477)** — curated hits already rank above the transcripts that + quote them. A shape corpus is curated by construction, so it lands on the right + side of an ordering rule that is already shipped and measured. +- **`why`** — explains why a drawer surfaced (location, tags, graph, tunnels). + The natural place to render *which trigger matched*, with no new surface. +- **`tunnels`** — cross-wing links. A shape recurring in `2g` and `memorypalace` is + literally a tunnel; the mechanism for "this shape is not domain-specific" exists. +- **`rate`** — records feedback that nudges ranking. The obvious signal channel for + "this shape was the right one" without building a new feedback path. +- **KG / `cypher` / `walk`** — an edge type from shape → instance → wing is + expressible in AGE today. + +**Lacks:** + +1. **No trigger field.** Nothing in a drawer is indexed as "the thing you would be + doing when you need this". Content search cannot synthesise it. +2. **No shape/instance distinction.** A drawer that *describes* a failure mode and + one that merely *mentions* one are the same object to search. #489 is the same + gap from the other side: a conclusion that exists only inside a diff chunk. +3. **No labelled retrieval test.** There is no harness that asks "for query Q, is + card C in the top N?" over a fixed corpus — so no change to ranking can currently + be shown to help or hurt. This is the real blocker, and it is cheap to remove. +4. **Ranking instability under re-mine** (#477's rank 13 → 1) means any measurement + must interleave baseline and candidate per query, never run them as two halves. + +--- + +## 5. Minimal first slice — one PR, and it measures before it builds + +**Do not ship an index first.** The honest first slice is 2g's own method applied +to the palace: *measure the retrieval gap on a labelled corpus.* + +**One PR delivers:** + +1. `docs/failure-shapes/` — the 21 citable instances as YAML, one file per shape, + fields per §1. Curated, mineable, `source_kind: file`. +2. `scripts/failure_shape_recall.py` — for each shape, issue its `triggers[]` + against the live search at `--limit 3`, record whether the shape's own drawer is + returned. Emits recall@3, recall@10, and the per-trigger table. + **Interleaves baseline and candidate per query** (#477 instability). +3. The measurement, committed as a dated result — the `n=11`-equivalent for this + corpus, at whatever n it actually reaches. + +No new CLI verb, no ranking change, no daemon route. Roughly the size of #493. + +### What would falsify the idea + +| outcome | reading | +|---|---| +| baseline recall@3 is **high** (say ≥0.6) on arrival phrasings | the index is unnecessary — plain search already connects actions to shapes; close #503 | +| baseline low, and **trigger-indexed retrieval does not beat it** | the bottleneck is the embedding, not the index; a trigger field is the wrong fix | +| baseline low, trigger retrieval **beats it but only on triggers authored with the card** | the index is overfit — it retrieves phrasings someone already thought of, which is the keyed-on-terms regime, not arrival | +| baseline low, trigger retrieval beats it on **held-out phrasings written by a different lane** | the idea holds; proceed to a retrieval surface | + +That third row is the one to guard hardest, and it is the reason the triggers for +the held-out test **must be written by a lane that did not author the cards** — +otherwise the measurement reproduces 2g's "keyed-on phrasings work" regime and +reports it as success. + +> **Pre-register the expected answer before running the harness** (Oracle PART 23): +> write down the predicted recall@3 for baseline, then diff. A harness whose result +> is read without a prediction is an instrument nobody has proven can see — which is +> the very shape this corpus exists to index. diff --git a/website/public/llms-full.txt b/website/public/llms-full.txt index eb9e1b36e8..d7dc108e18 100644 --- a/website/public/llms-full.txt +++ b/website/public/llms-full.txt @@ -309,6 +309,7 @@ The full enumeration of fork-ahead changes. The canonical source is [`docs/fork- | Description | Upstream PR | Fork commit | |---|---|---| +| design spec for indexing failure SHAPES, keyed on the action about to be taken | — | `HEAD` — pending resolution | | purge / prune / mined share one open-and-refuse sequence and one exit-code contract | — | [`459efab`](https://github.com/techempower-org/mempalace/commit/459efab) | | Entry shas resolve from the commit that added the entry file; strict check rejects an unresolved placeholder on main | — | [`b9d168a`](https://github.com/techempower-org/mempalace/commit/b9d168a) | | KG write-through commits once per batch instead of once per drawer | — | [`41f1251`](https://github.com/techempower-org/mempalace/commit/41f1251) | From 2d62de1c5153b7136208be2c82b577e95bce6d7d Mon Sep 17 00:00:00 2001 From: jp Date: Thu, 17 Sep 2026 19:44:48 -0700 Subject: [PATCH 2/2] docs(fork-changes): record fork_pr 505 on the failure-shape-index entry Part of #503 Co-Authored-By: Claude Opus 5 --- docs/fork-changes/2026-09-17-failure-shape-index.yaml | 1 + 1 file changed, 1 insertion(+) diff --git a/docs/fork-changes/2026-09-17-failure-shape-index.yaml b/docs/fork-changes/2026-09-17-failure-shape-index.yaml index 5ce3dc5b9f..79e626cd4f 100644 --- a/docs/fork-changes/2026-09-17-failure-shape-index.yaml +++ b/docs/fork-changes/2026-09-17-failure-shape-index.yaml @@ -46,3 +46,4 @@ body: | `rate` and the AGE graph each have a role with no new route. files: - docs/specs/2026-09-17-failure-shape-index.md +fork_pr: 505