docs: describe the system as built with ADR-007, ADR-008 and ADR-009

[paddymul]

Stacked on #189: the base is feat/adr-007-009-cache-redesign, not main. Docs only; no code or test changes, and ADR-010's text is untouched.

Rebased on 2026-09-24 onto #189's branch after the ADR-011 stack (#217, #218, #219) was merged into it, and extended to cover ADR-011. ADR-011 (a raw input is a source alias whose versions are entries, a file enters only by catalog_import_source, recipes name aliases and never bare hashes, and staleness has one axis) held back docs/architecture.md, docs/caching.md, docs/expression-lifecycle.md and docs/system-contract.md until this PR landed, so they would be rewritten once. They are rewritten now, in 5b6c163, with the rest of the docs; 2650674 updates the plans' status notes. The first three commits are the original ones, rebased; the only conflicts were in the status notes of ADR-002 and ADR-005, which ADR-011 had also edited, and both keep ADR-011's notes with this PR's merged in after them.

Rebased again on 2026-09-24 onto 1f8cb02, after #222 (the #193 fix) and #223 (the #194 to #196 fixes) were merged. The rebase had no conflicts. 357f0a2 rewrites every passage about those four issues to describe the fixed behaviour: a create stages its snapshot at a temp name and publishes it after the manifest, so a failed build leaves the file already at the path; the pin is read from the manifest alone (reproducible: false, unfaithful_heal_digest, or a source entry whose clone is gone), so it moves through a reset and outlasts the error banner's dismiss; a retired entry's snapshot is judged by the manifest parked in the bullpen and listed as retired on the Cache page; a reset lets the live entry dir replace a parked one unless it has no manifest; and errors.jsonl readers skip a line that is not a JSON object. It also corrects the sentences that said the error log holds the pin, that nothing rewrites a manifest, and that nothing live reads the bullpen, and restores the docs/system-contract.md write-path step and ADR-007 implementation note that #213 had written and the port left out.

ADR headers state the design as built (f98bd07). At Paddy's request the plan headers this PR edited carry no merge or rebase history. ADR-011's Status is "Accepted (2026-09-22), implemented."; what it amends is its own bullet; the origin and Paddy's two quotes are in Context; and its body describes what the system does, with implementation notes that carry no PR numbers or order of events. ADR-002, ADR-005, ADR-007, ADR-008 and ADR-009 state their final status with amends, supersedes and amended-by bullets, and the status notes of recalc-mechanism, auto-recalc-on-revise, cache-soundness-audit and datafusion-scan-order-findings state current behaviour only.

Coherence pass (a61943b), checked against the code at 1f8cb02. It corrects claims that contradicted the code: nothing reads the recipe zip back; a promoted diff's recipe names two entries by hash and records no edge; a source entry's snapshot has its own writer, so invariant I2 names two; three kinds of snapshot are pinned, and a pin also depends on whether a source's clone is on disk; ADR-007's account of ADR-006 now matches ADR-006; auto-recalc also follows an import and a re-pointing promoted diff. It also removes leftover mentions of removed mechanisms (hash pins, ordered copies, salt mode, catalog_load_parquet), fixes the #90 and #168 citations, takes PR and commit references out of the status lines this PR added, and deletes two stray tool-call lines from the end of docs/installing.md.

• docs/architecture-new.md (71f9264, cf446c6) is a new description of the system as built after ADR-011, meant to be read in one sitting. It was written from scratch against the code at 1f8cb02, using the other docs and the ADRs only as sources, and docs/architecture.md is left as it was. Its Known defects section adds two defects with no issue, each confirmed with a probe in a scratch home: catalog_import_source lets a CSV that fails to parse escape as a plain ValueError, with nothing written to errors.jsonl, and importing a version's bytes again restores a lost clone only when the snapshot is gone too.
• plans/open-bugs-2026-09-24.md (5ce41b7) lists every open issue checked against 1f8cb02, by priority, including the ones that are fixed or obsolete but still open. importing a parquet file with a fixed_size_binary column fails with a KeyError traceback that names no column #224 to alias, notebook and config writes are read-modify-write with no lock, so two overlapping writers lose one change #240 were filed from this pass, and the known-defects lists in architecture-new.md and architecture.md cite them.
• caching: concurrent .execute() on the shared default backend raises "Already borrowed" — api_data has no execute lock (second cause of #79 two-tab 500) #118, two tallyman servers on one project is unsupported and nothing detects it — each holds in-process state the other never sees #183, cache_worthy falls back to "a snapshot exists" when the manifest is missing — a worthy entry that has lost both is served as cheap #204 and a tz-aware timestamp in an import schema reads offset-less text as UTC and converts it, contrary to ADR-005 D9(a) #231 are fixed (664972c, 9efffe3). 664972c merges feat/adr-007-009-cache-redesign at 63bcdd6, which brings in fix(xorq): executions on the shared backend run one at a time per process (#118) #242 (caching: concurrent .execute() on the shared default backend raises "Already borrowed" — api_data has no execute lock (second cause of #79 two-tab 500) #118), fix(csv): a zone on offset-less CSV text is attached to the wall-clock time (#231) #244 (a tz-aware timestamp in an import schema reads offset-less text as UTC and converts it, contrary to ADR-005 D9(a) #231), fix(cache): a read refuses a directory with no manifest instead of guessing its worthiness from the snapshot (#204) #245 (cache_worthy falls back to "a snapshot exists" when the manifest is missing — a worthy entry that has lost both is served as cheap #204) and fix: one tallyman server per data dir, and a second tallyman on its own data dir reaches its own companion (#183) #241 (two tallyman servers on one project is unsupported and nothing detects it — each holds in-process state the other never sees #183). 9efffe3 describes the fixed behaviour and takes the four out of the defect lists (architecture-new.md, architecture.md, caching.md, system-contract.md, ADR-007's Tickets and D11, and open-bugs-2026-09-24.md, which moves them to "Closed in this pass"). The docs now describe a process-wide execution lock around every execution on the shared backend, taken after the project lock; one tallyman run per data dir, held through <data dir>/server.lock, whose owner record is the only way a client finds its companion; every read refusing an entry dir with no manifest, with a BuildError and a 500 from the entry routes; and a zoned CSV timestamp parsed in its zone. It also drops TALLYMAN_COMPANION_URL from docs/installing.md, since nothing reads it.

The docs describe the system as it is on #189's branch, where ADR-007, ADR-008, ADR-009 and ADR-011 are implemented. Where an open issue says the code does not yet do what a doc would promise (#199 to #210 except #204, #185 to #188, #190, and a few older ones such as #157 and #170), the doc describes what the code does and cites the issue. #193 to #196 (#222, #223) and #118, #183, #204 and #231 (#242, #241, #245, #244) are fixed on this branch and are described as fixed.

Terms, as docs/architecture.md now defines them: an entry is one catalog computation, stored under its content hash. A worthy entry is one tallyman materializes into a snapshot, a parquet file of its rows; a cheap entry is row-preserving over one file and keeps no file of its own. An import (catalog_import_source) is the one way a file enters; it makes a source entry, one version of a source alias, whose snapshot holds the file's rows in file order and whose manifest records provenance. The clone is the imported bytes under data/.cas/. A heal re-creates a missing snapshot and checks it. The bullpen is the directory a reset moves retired files into.

What changed

File	Change
docs/architecture.md	Rewritten as the overview and map: a Terms glossary, the component map, the full on-disk layout, one section per core concept (including imports and source aliases, and one staleness axis), the request paths (create, import, revise and cascade, grid, diff, reset), a Known defects section with notes on the defects ADR-011 made moot and on #193 to #196 (fixed), and currency notes dated 2026-09-22 and 2026-09-24 that list ADR-001 to ADR-011.
docs/caching.md	The defects that touch each cache, what the project lock covers, the staged publish of a create, pins read from the manifest, the Cache page's retired and orphan rows, the bullpen's one live reader, source entries (hash, snapshot, heal from the clone, the pin when the clone is gone), the heal table, what the nondeterminism lint flags, the SPA's missing unfaithful_heal listener, and that Buckaroo 0.15.6 ignores row_order_column.
docs/expression-lifecycle.md	The build timeline (the expanded build written at create, what a failed build deletes, the alias and notebook writes), a step for the import that comes before any recipe, the catalog and notebook pages' session loads, and the Buckaroo log locations.
docs/system-contract.md	Part 2 opens with imports and source entries (the governing rule, the hash, the D3 case table, one set of bytes per alias); recipes, entries, the content hash, the manifest (provenance, unfaithful_heal_digest), alias kinds, materialization (the staged publish, pins read from the manifest) and the read path are restated for them; a new "import path" section; one staleness axis; and a "Known deviations" section mapping each open issue to the rule it breaks.
docs/reactive-recalc.md	The walk-through imports orders and advances it by a re-import; one staleness axis replaces the two; pinned_expr_from_alias takes a version reference only; the recorded graph is manifest.parents alone; the scan output (live, orphan_stale), the cone's live-head filter, the SPA's recalc handling, and imports in the trigger model.
docs/mcp-server.md	catalog_chart_errors (31 tools), the klass file paths, catalog_diff's __row_order handling (#200), the checkpoint and notify conditions, #170 and the display-klass reset gap, and ADR-011's data-sourcing rules, auto-recalc on import and the scan that reads no file.
docs/installing.md	Project listing, the active-project precedence (#39), the environment variable table (TALLYMAN_SOURCE_IDENTITY is gone), and the sharing section (tallyman pack, no grids under serve, #209).
README.md	The tool list (catalog_import_source), routes, SPA pages and SSE listeners, the recommended prompts, the recipe conventions, and the Buckaroo hand-off and serve and pack notes. In the owner's intro, only the two clauses that described raw files read by programs and changes on disk counting as updates are changed (a61943b).
tallyman_explanation.md	The feature list, the walkthrough and the CS section, including imports and one staleness axis. The owner's own sections are unchanged, among them the appendix "the original framing", which still says a changed source file counts as staleness.
plans/ADR-007…, plans/ADR-008…, plans/ADR-009…	Status "Accepted (2026-09-22). Implemented in #189", with the history lines kept and the related defects named, #193 to #196 as fixed; ADR-008 notes that #197, #198 and #211 went with the ordered copy; ADR-007 gains the implementation note for #193's staged publish.
plans/ADR-003… to plans/ADR-006…	Each status block says which later ADR decision supersedes what.
plans/ADR-011…	Header reduced to status, amends, context, tickets and code; the body describes the design as built, without PR or stage history.
plans/ADR-002… and ten other plans	A short dated status note where a plan asserts behaviour that has changed, revised for ADR-011 where it touched sources.

Where the docs and the code disagree in a way that looks like a code bug

These are described in the docs as current behaviour, or found while checking them. None has an issue yet. Two from the first version of this list no longer apply: ADR-011 deleted the source-digest memo the scan wiped, and a hash-pinned child can no longer be written.

• A recipe resolves its project from the active_project file, not the MCP session's project. tracked_expr_from_alias and pinned_expr_from_alias call resolve_project(), while the MCP tool builds into the session's sticky project. After another session switches projects the recipe looks its aliases up in the other project. Related to no way to pin an MCP process to a project: active-project file outranks TALLYMAN_PROJECT and --project #39.
• A raw read of a snapshot by path gets past the no-bare-hashes rule. _raw_parquet_read_check (build.py:305) allows xo.deferred_read_parquet of any file under the project's compute_cache/, so a recipe can read compute_cache/result_cache/<hash>.parquet directly: a bare hash in a recipe (ADR-011 D5) with no parent edge recorded and no ensure_materialized first. Found by reading the code, not reproduced.
• An unfaithful heal of a source entry is attributed to the wrong cause. When a source snapshot re-created from its clone differs from the recorded digest, _verify_self_heal falls through to "execution (caching: a recipe with deferred nondeterminism (sample/now/unordered limit) serves different bytes under one content_hash on recompute — hash-invisible, and it breaks eviction's faithful-recompute assumption #83) — a fixed graph that runs differently each execute, or source drift under off": a source entry has no graph to run, and the off mode is gone. The likely cause for a source entry is a reader change.
• catalog_list does not show an alias's kind, though ADR-011 D1 records the kind "so catalog_list can separate inputs from computations". The docs do not claim it does.
• recon_cas_path has no callers. ADR-011 D9 describes its error branch, but nothing calls it.
• The SPA ignores six event kinds the server publishes: entry_added (both promote paths), alias_changed, alias_renamed, display_changed, project_reset and unfaithful_heal. None of them refreshes an open page, so a promoted diff or a CLI reset-to shows up only after some other event.
• project_root handed to Buckaroo points at artifacts/ but klasses live under artifacts/catalog/ — project stats and grid post-processing never load #170, confirmed against the Buckaroo 0.15.6 wheel. Buckaroo scans <project_root>/stats, post_processing and display; tallyman sends artifacts/. Display klasses load; project stats and post-processing functions never do.
• Smaller. catalog_chart_errors, catalog_list_display_klasses and POST /api/chart_error each commit an empty revision per call. Display klasses live outside the catalog repo, so their tools' checkpoints record nothing and a reset does not undo them. tallyman pack --exclude-cache skips paths containing /cache/, which matches nothing in the current layout. Alias, notebook and chart writes take no lock, so an MCP edit and a browser edit of one file at the same moment can lose one. tests/test_promote_diff.py:253 still sets TALLYMAN_SOURCE_IDENTITY, which nothing reads.
• Stale comments in code (not changed here): io.py's module docstring says user code references files under data/ by relative name; tracked_expr_from_alias says to read by hash with pinned_expr_from_alias, and pinned_expr_from_alias's first line says it takes a content hash, which it refuses; tallyman_read_csv and ADR-005 showed catalog_import_source(path, alias, separator=...), but the tool takes reader_options={...} (ADR-005 is fixed here); ds_modeling_workflow says its dataset is a parquet under data/; paths.py shows ~/.tallyman/; preload_plan says it writes nothing, but loading a build can write .xorq_build_expanded/; api_reset says a timeline scrubber in the SPA calls it, and nothing does.

Checked

Every claim in the edited docs was checked against the code on this branch, and against the installed Buckaroo 0.15.6 wheel for claims about Buckaroo; for the ADR-011 pass, against source_import.py, io.py, materialize.py, staleness.py, source_identity.py, catalog_state.py, aliases.py, build.py and the MCP tools. Every relative link and #anchor in the edited docs resolves, and git diff --check is clean. CI not watched (docs only).

🤖 Generated with Claude Code