ADR-011 stage 2: one staleness axis, one identity mode, and no raw file reads by paddymul · Pull Request #218 · buckaroo-data/tallyman

paddymul · 2026-09-23T02:06:24Z

Stage 2 of ADR-011 (plans/ADR-011-sources-are-aliases.md), stacked on #217. It deletes the staleness source axis (D6) and the source-identity modes (D8), rewrites every call site that authored a raw file read onto imported source aliases, and takes the stage-1 escape hatch with them.

Net −616 lines (+2341 / −2957 across 80 files). Raw-read call sites in tests/ go from 357 to 22, and the 22 that remain are the tests of the refusal itself.

What is deleted

The staleness source axis. An entry is stale when a followed alias moved, and that is the only reason. manifest.sources goes entirely rather than narrowing to a retention record, because once a source version is an entry the retention closure is the DAG. dependents.sources_of and io._note_parent_records — the function whose folding of a parent's digests into its child is the direct cause of the defect ADR-011 exists to fix — go with it.
The source-identity modes. TALLYMAN_SOURCE_IDENTITY, its off and salt modes, and salted_hash. "Identity off" and "every input is versioned" cannot both be true.
The ordered-copy store. compute_cache/ordered_sources/, copy_key, manifest.ordered_copies, and ensure_ordered_copy / existing_ordered_copy / recreate_ordered_copy. The copy is the source entry's snapshot now.
source_digests.json and digest_for's stat memo, which have no callers left: nothing digests a file outside an import.
TALLYMAN_LEGACY_FILE_READS, the stage-1 escape hatch, and the tests/conftest.py line that set it for the whole suite. There is now no way to author a raw file read.

A real bug the rewrite exposed

update_and_depend(path, alias, project=X) failed whenever X was not the active project. The generated recipe's read_project_file resolved the ambient project while the contextvar naming the entry being minted said X; the two disagreed, so the importer's own recipe hit the D2 refusal written for authored recipes and the import died complaining about a file tallyman does not own.

project= is an override, so the read now follows the entry being minted. Every call site in the repo passed before this, because the project fixture also activates its project — the cache lab, which warms xorq in a project of its own without activating it, is what found it. Red test 2f8350a, fix 502c7a5.

Two expectations changed because the behaviour is now right

Neither is a test bent to stay green:

The page-load profiler measures three entries where it measured two, because an imported source version is an entry. The corpus fixture says so.
The reset round-trip has to import its second file after the first checkpoint. Import it before, and that source entry survives the reset back, so no clone is left referenced only by a retired entry and the case ADR-007 D13 is about stops being exercised at all. Arranged correctly, the test is the proof that deleting manifest.sources did not break retention: a backward reset parks the clone in the bullpen instead of unlinking it, and a forward reset restores it. That was the failure mode that could have destroyed data silently.

ADRs amended

ADR-002 (keeps its clone store, loses the modes, the sources map and the reconstruction caveat — which is now an error, not a fallback to live bytes), ADR-005 (its reader runs at import; the schema DSL, inference ladder and suggestion contract are unchanged, and options are fixed per alias), ADR-008 (its refusal extends to read_project_file itself, and the ordered copy becomes the source entry's snapshot). ADR-011 records stage 2 and answers its first open question: keep the raw bytes, which is also what makes the snapshot cache rather than data.

TDD

Red 2d01e6f (run 35809031979, failed) and 2f8350a (run 35811124364, failed); green a803ba7 (run 35854181437) and 75381c5, both with ruff, fast and integration passing.

Four agents worked on this branch and all four died mid-flight — three on a stall watchdog, one on an API error — after I made the mistake of giving three of them one shared worktree, where they raced on a single git index and one wiped eight of another's files with a git restore. ec4f86e is their salvaged work, committed as-is to protect it; everything after it was finished by hand.

Outstanding, deliberately not here

docs/architecture.md, docs/caching.md, docs/expression-lifecycle.md and docs/system-contract.md all describe the deleted source axis and are now wrong. PR #216 is rewriting all four; a second rewrite here would conflict badly, so they are untouched.

The pinned layout costs +57% on high-cardinality numeric data (16.2 MB → 25.5 MB on a 1.5M-row fixture), because 122,880 int64s dictionary-encode to just under pyarrow's 1 MB dictionary page limit and the dictionary never spills to plain. That is a question about _PARQUET_OPTIONS for every snapshot, which is ADR-009's to answer; nothing here writes a source snapshot differently from a computed one to avoid it. Measured sizes are in ADR-011's implementation notes.

🤖 Generated with Claude Code

The assertions stage 2 has to make true, ahead of the code that makes them true. A scan of a project with three imported sources and two children takes no digest at all and reports no axis it could not evaluate; a manifest records neither sources nor ordered copies; source_identity has no mode, no salt and no stat-memoized digest, and no module names TALLYMAN_SOURCE_IDENTITY; the ordered-copy store and its copy key are gone, so compute_cache holds one kind of file; io folds no parent source records; the rebuild script replays a source entry through an import rather than through build_and_persist; and TALLYMAN_LEGACY_FILE_READS buys an authored recipe nothing. The digest counter is its own positive control: it counts the two digests each import takes (the file as given, then the clone as written) and zero across the scan, so the claim is measured rather than read off the code. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>

update_and_depend(path, alias, project=X) fails whenever X is not the active project. The recipe the importer generates calls read_project_file with no project argument (source_import.py:348), so the read resolves the ambient active project, the _SOURCE_ENTRY contextvar it is checked against names X, and the two disagree — the importer's own recipe then hits the D2 refusal written for authored recipes and the import dies with an error about a file tallyman does not own. Every call site in the repo passes today because the project fixture also makes its project active. Rewriting the suite onto imports found it: the cache lab warms xorq in a project of its own, without activating it. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>

…11 D6, D8) The source axis, manifest.sources, manifest.ordered_copies, the ordered-copy store and the source-identity modes are all deleted. read_project_file and tallyman_read_csv now always refuse an authored recipe, and the TALLYMAN_LEGACY_FILE_READS escape hatch that let the suite keep authoring raw reads is gone with them. The test rewrite onto imported source aliases is PARTIAL: about half the suite is moved over and the rest still calls the deleted helpers. Committed as-is because four agents were editing this worktree at once and one of them lost eight files to a git restore. Follow-up commits finish it. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>

update_and_depend(path, alias, project=X) failed whenever X was not the active project. The recipe the importer generates calls read_project_file with no project argument, so the read resolved the ambient active project while the contextvar naming the entry being minted said X. The two disagreed, the importer's own recipe hit the D2 refusal written for authored recipes, and the import died complaining about a file tallyman does not own. project= is an override, so the read now follows the entry being minted rather than whichever project happens to be active. source_entry_context returns the (project, hash) pair it already held instead of filtering on a project passed in, and neither entry point resolves the ambient project any more. Every call site in the repo passed before this because the project fixture also activates its project. Rewriting the suite onto imports is what exposed it: the cache lab warms xorq in a project of its own without activating it. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>

The call-site rewrite left four files still opening a file in a recipe. Each now imports it as a source alias and reads the alias. Two of the four failures were the new behaviour, not breakage: - the page-load profiler measures three entries where it measured two, because an imported source version IS an entry (D1). The corpus fixture says so rather than the assertion excusing it. - the reset scenario has to import extra.parquet AFTER s1. Importing it before keeps its source entry alive across the reset back, so no clone is left referenced only by a retired entry and the case ADR-007 D13 is about stops being exercised. With the import after s1 the test proves what it claims: a backward reset parks the clone in the bullpen instead of unlinking it, and a forward reset restores it. test_replay's storyboard imported the parquet and then authored a raw read of it anyway; the stdio round trip did the same over the real MCP transport. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>

Each of the three ADRs that described how a raw input reaches a build now says what ADR-011 changed, in its status block, so a reader who starts at any of them is not misled: - ADR-002 keeps its clone store and loses the rest: the mode switch and salted_hash, manifest.sources, digest_for and its stat memo, and the reconstruction caveat, which is now an error rather than a fallback to live bytes. - ADR-005's reader runs at import, not in a recipe. The schema DSL, the inference ladder and the suggestion contract are unchanged; they just live inside catalog_import_source now, and the options are fixed per alias. - ADR-008's refusal of a raw read extends to read_project_file itself, and the ordered copy stops being a separate kind of file: it is the source entry's snapshot, named by its content hash. ADR-011 records stage 2 in its implementation notes, marks itself implemented, and answers its first open question: keep the raw bytes, which is also what makes the snapshot cache rather than data. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>

Same bytes mint the same entry hash in both projects, but each holds its own entry, snapshot and clone: a heal, a clone sweep or a re-import in one never reaches the other, and the one-alias-per-bytes rule is per project. Also pins the way to give a source a second name: a catalog entry whose recipe reads it. Its hash is xorq's hash of the expression, not the md5 a source entry is named by, so the two never coincide and nothing has to be added to the recipe to force them apart. These pass today; they guard the duplicate-import fix that follows. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

A second import of bytes another alias already holds minted nothing new but re-ran _mint over the shared entry: it rewrote the first alias's manifest and recipe with the second alias's name, and a failure in the build step rmtree'd the entry the first alias points at. - the second import raises, naming the alias and version that hold the bytes and the catalog_create way to a second name, and leaves the first entry byte-for-byte as it was - the same holds for bytes of an older version of another alias - catalog_import_source returns it as an error and records no revision - a re-import that repairs a missing snapshot keeps the existing entry when the build step fails Replaces test_two_aliases_over_identical_bytes_share_one_entry, which pinned the behaviour being removed. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

A source entry is named by its bytes and reader options, so a second import of the same bytes under another alias landed on the first alias's entry and re-ran _mint over it: the manifest's provenance and the recipe were rewritten with the second alias's name, and a failure in the build step rmtree'd the entry the first alias points at. - update_and_depend refuses a mint whose entry is a version of another alias of the project, at any version. The error names that alias and version and the way to a second name: a catalog entry whose recipe is tracked_expr_from_alias(<that alias>). Keyed on the entry hash, so a CSV read two ways is still two imports (D12); per project. - _mint removes the entry directory on failure only when it created it, so a re-import repairing a missing snapshot keeps the existing entry. - ADR-011 D1 now says one set of bytes is one version under one alias, and records that it first said the opposite and why that changed. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

The kind lives on the alias ("catalog" or "source" in aliases.jsonl), but whether an entry is a source entry lives on the entry (its manifest carries provenance), and nothing ties the two together. catalog_alias checks only the kind of the name, so catalog_alias(<source entry hash>, "x") makes a catalog alias whose head is an imported file. catalog_revise("x", ...) is then allowed, which bypasses D1's "a source version has no recipe to revise". Three tests pin the invariant: - set_alias refuses a catalog alias onto a source entry; - set_alias refuses a source alias onto a computed entry; - MCP catalog_alias on a source entry returns an error naming the source alias-version (orders-v1) and steering to catalog_create over tracked_expr_from_alias, creates no alias, and the steer then works. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

manifest.provenance.alias and .version record the name a source version was imported under, once. A rename carries the alias's history and kind to the new name, and an unalias drops it, but three places still read the import-time name as the current one: - materialize.pinned_reason and the _heal_a_source error name the version a_src-v1 after a rename to renamed_src, and the heal error advises catalog_import_source(path, 'a_src', pinned_version=1). Following that advice mints a second source alias, a_src, over the same entry. - The generated recipe's header reads "# a_src-v1: a source version", so the Code tab of renamed_src-v1 presents the old name as the entry's. - scripts/rebuild_native_catalog.py replays a source entry under provenance["alias"] and orders its versions by that alias's history. After a rename the replay mints a_src, never makes renamed_src before the children build, and a child reading renamed_src fails; the versions lose their ordering edge and fall into hash order. These tests pin the current name in both messages, advice that repairs the renamed alias without minting a new one (through update_and_depend and through the MCP tools), wording that does not send an unaliased version back under its dead name, a recipe header that records the import name as history, and a rebuild that replays a renamed source under its current name and in version order. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

set_alias now reads the manifest of the entry it is pointing at. A manifest that records provenance is a source entry, a version of an imported file; any other manifest is a computed entry. A catalog alias onto a source entry, or a source alias onto a computed entry, raises AliasKindMismatch. The check lives in set_alias because every route that names an entry goes through it, so catalog_alias, a revise, a promoted diff, a recalc and an import all keep an alias's kind and its entries' kind the same without each tool repeating it. A hash with no readable manifest is not checked: the alias bookkeeping is used and tested with hashes that name no entry. A passing test pins that boundary. For a source entry the message names the source alias-version that holds it (via version_of_hash) and gives the ADR-011 way to a second name: catalog_create('<name>', "...expr = tracked_expr_from_alias('<source>')"), a catalog entry that reads the source and follows it when it is imported again. MCP catalog_alias returns that message as {"error": ...}. The other set_alias callers only alias a hash they just built (catalog_create, catalog_revise, promote_diff in MCP and the companion, the companion's PUT /api/code, recalc, the rebuild and perf scripts) or just imported (update_and_depend), so the new check cannot fire for them. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

provenance.alias and .version stay what they are: the record of the import. Everything that names a source version as it is now asks the alias store. - source_import.source_version_in finds the source alias whose history holds an entry, trying the name it was imported as first. It is aliases.version_of_hash restricted to source aliases, because an import advances nothing else. current_source_version applies it to a project. - materialize.pinned_reason and the _heal_a_source error name the version by that alias and add "imported as <old>-vN" when a rename changed it. The advised catalog_import_source(...) names that alias and version, so running it repairs the snapshot and mints nothing. When no source alias holds the entry, both messages say so, and the error says an import of the same bytes writes them again as a new version of whichever alias it names. - The generated recipe's first line reads "Generated by catalog_import_source when the file was imported as a_src-v1", which stays true after a rename. - scripts/rebuild_native_catalog.py orders a source's versions and replays each one under the source alias that holds it in the old catalog, pinned to its version there, so an out-of-order replay is an error. The re-point loop now skips only the alias the import pointed and sets any other holder with the kind it had; the pin needs a second source alias over the same bytes to be on that version before its next one replays. - SourceProvenance's docstring says alias and version are the import name. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

…ntry' into fix/adr-011-duplicate-import

…rename' into fix/adr-011-duplicate-import

… routes refuse source aliases (ADR-011) Found reviewing #219. - A re-import of a version whose snapshot is gone went through _mint: it rebuilt the entry, rewrote the manifest (provenance path, imported_at, result_digest, row_count) and took whatever it wrote as the new truth. A probe with a reader that drops a row showed 10 -> 9 rows under the same hash and no unfaithful_heal record, where ensure_materialized records one. The repair should rewrite nothing but the snapshot, and be verified like any heal. - An entry directory with no manifest (a crash part-way through an import) could not be repaired by re-importing: the no-op branch read the manifest and raised. - The heal error's advised catalog_import_source(...) dropped a CSV's schema and reader options, so running it as written was refused as "not orders-v1". The pinned-version refusal also blamed the file when only the reader options differed. - PUT /api/code/<source alias> and POST /api/promote_diff onto a target name that is a source alias built an entry, failed to point the source alias at it and answered 500. Both should refuse before building, 409. Replaces test_a_failed_repair_leaves_the_existing_entry_in_place, which pinned the repair going through _mint. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

D11's refusal (bytes that match an older version of the same alias) ends "or import these bytes under a different alias". Since #219 that import is refused as well: bytes one alias holds cannot be imported under another. The refusal should offer ways back that work, a reset or reading the old version with pinned_expr_from_alias('<alias>-v<N>'). Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

…es refuse source aliases (ADR-011) - update_and_depend never rewrites an entry that already exists. A no-op (or a new alias over an entry no alias holds) whose snapshot is gone restores the clone from the caller's bytes when it is gone too, then heals through ensure_materialized: from the clone, checked against the recorded result_digest, recorded as an unfaithful heal when the rows differ. It used to go through _mint, which rebuilt the entry, rewrote provenance and recorded whatever digest it wrote. An entry directory with no readable manifest is not an entry, and is written again. - source_import.import_call prints the catalog_import_source call that imports a file the way its entry records it was read, with a CSV's schema= and reader_options=. The heal error's advice uses it, so the advice runs as written. The pinned refusal says the reader options may be what differs, and D11's refusal offers pinned_expr_from_alias instead of a second alias, which D1 refuses. - The source-alias refusal text moves to aliases.source_alias_refusal. The companion's PUT /api/code and promote-diff routes refuse with it, 409, before building; they used to build, fail in set_alias and 500 with the entry left under no alias. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

- ADR-011: status names PR #219; D1 gains the kind rule and says provenance keeps the name a version was imported as; D3's table gains the one-alias error and a note on what an import of an existing version does; D11 and D12 say "the same bytes read the same way" and that advice carries the reader options; Testing drops the import_once_and_depend bullet (not shipped) and lists the new cases; a "Review fixes (PR #219)" implementation note records what changed and why. Also fixes two stale lines: the arena holds snapshots, not ordered copies, and ADR-002's sources map goes rather than narrowing. - docs/mcp-server.md: catalog_import_source's one-alias rule, the second-name recipe and the healing no-op; catalog_alias refuses a source entry. - tests/conftest.py: orders_parquet's docstring no longer suggests importing it under another alias beside orders_src. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

paddymul · 2026-09-24T11:28:44Z

Review of #217 and #218: findings and how each was handled

Fixes are on #219 (stacked on this PR), each test first.

A second alias importing the same bytes rewrote the first alias's entry, and a failure deleted it. Probe: after a simulated build_expr failure under b_src, a_src still headed the hash with no entry directory. Fixed in c7a0339: the import is refused, naming the alias and version that hold the bytes.
After a rename, the heal error and pin reason used the import-time name, and the advised re-import minted a second source alias. The rebuild script had the same fault. Fixed in 6aefd0f (from ADR-011: a source version is named by the alias that holds it now #221).
catalog_alias could put a catalog name on a source entry, opening it to catalog_revise. Fixed in da53ca7 (from ADR-011: an alias's kind matches the entries it points at #220): set_alias keeps an alias's kind equal to its entries'.
ADR-011 contradicted itself (the arena's ordered copies, ADR-002's sources map "narrowing", an import_once_and_depend test for a function not shipped). Fixed in 9f46fc8.
recon_cas_path has no callers after ADR-011 stage 2: one staleness axis, one identity mode, and no raw file reads #218. Open.
catalog_promote_diff still writes bare hashes into its generated recipe, against D5. Open; diffs are tabled, and the ADR does not yet carve it out.
This PR's description (ADR-011 stage 1: a file enters the catalog only by an explicit import #217's too) is stale in places: ADR-011 stage 1: a file enters the catalog only by an explicit import #217 still says the snapshot is data. Open.
README.md and docs/reactive-recalc.md still teach catalog_load_parquet / read_project_file, along with the four docs docs: describe the system as built with ADR-007, ADR-008 and ADR-009 #216 rewrites. Open until docs: describe the system as built with ADR-007, ADR-008 and ADR-009 #216 lands. docs/mcp-server.md was brought up to date in 9f46fc8.

ADR-011 review fixes: one alias per bytes, alias kinds, current names, verified repairs Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

ADR-011 held back docs/architecture.md, caching.md, expression-lifecycle.md and system-contract.md until this PR landed, so they would be rewritten once. Now that the PR is rebased onto the merged ADR-011 stack (#217, #218, #219), they describe what exists: a file enters only by catalog_import_source, as a source entry under a source alias; a source entry's hash is an md5 of its bytes and reader options; its snapshot is cache, healed from the clone under data/.cas, and pinned once the clone is gone; recipes name aliases, never files or bare hashes; alias kinds and the rule that an alias's kind matches its entries; one set of bytes is one version under one alias; and staleness has one axis. The ordered copy, the identity modes, manifest.sources, the source-digest memo and the source axis are gone from every doc except where one says what was removed. Known defects #197, #198, #207, #211 and #191, and the two staleness defects with no issue, move to a note that they no longer apply. reactive-recalc.md's walk-through now imports orders and advances it by a re-import. mcp-server.md, installing.md (TALLYMAN_SOURCE_IDENTITY is gone), the README and the code-derived sections of tallyman_explanation.md follow suit. The #193 to #196 passages are unchanged; those fixes are being ported onto this branch separately. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

paddymul and others added 8 commits September 22, 2026 22:06

paddymul mentioned this pull request Sep 24, 2026

ADR-011 review fixes: one alias per bytes, alias kinds, current names, verified repairs #219

Merged

paddymul and others added 2 commits September 24, 2026 05:57

paddymul mentioned this pull request Sep 24, 2026

ADR-011: an alias's kind matches the entries it points at #220

Merged

paddymul mentioned this pull request Sep 24, 2026

ADR-011: a source version is named by the alias that holds it now #221

Merged

paddymul and others added 8 commits September 24, 2026 06:45

Merge remote-tracking branch 'origin/fix/adr-011-alias-kind-matches-e…

45c177a

…ntry' into fix/adr-011-duplicate-import

Merge remote-tracking branch 'origin/fix/adr-011-source-names-follow-…

f48ccc8

…rename' into fix/adr-011-duplicate-import

paddymul mentioned this pull request Sep 24, 2026

fix(xorq): an ordered copy keeps its source's types and reader options, and its digest file is written atomically (#197, #198, #211) #215

Closed

Merge pull request #219 from buckaroo-data/fix/adr-011-duplicate-import

968d502

ADR-011 review fixes: one alias per bytes, alias kinds, current names, verified repairs Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

paddymul merged commit 10a0c72 into feat/adr-011-sources-are-aliases Sep 24, 2026
1 check passed

This was referenced Sep 24, 2026

fix(xorq): a failed build keeps the snapshot already on disk for its hash (ADR-011 port of #213) #222

Merged

fix(cache): a pin survives a reset and a dismissed error banner (#194, #195, #196), on the ADR-011 stack #223

Merged

paddymul mentioned this pull request Sep 24, 2026

docs: describe the system as built with ADR-007, ADR-008 and ADR-009 #216

Merged

Navigation Menu

Repository navigation

ADR-011 stage 2: one staleness axis, one identity mode, and no raw file reads - #218

ADR-011 stage 2: one staleness axis, one identity mode, and no raw file reads#218
paddymul merged 20 commits into
feat/adr-011-sources-are-aliasesfrom
feat/adr-011-stage-2

paddymul commented Sep 23, 2026 •

edited

Loading

Uh oh!

paddymul commented Sep 24, 2026

Uh oh!

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

1 participant

Conversation

paddymul commented Sep 23, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

What is deleted

A real bug the rewrite exposed

Two expectations changed because the behaviour is now right

ADRs amended

TDD

Outstanding, deliberately not here

Uh oh!

paddymul commented Sep 24, 2026

Review of #217 and #218: findings and how each was handled

Uh oh!

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

1 participant

paddymul commented Sep 23, 2026 •

edited

Loading