feat(cache): implement ADR-007, ADR-008 and ADR-009 (tallyman-owned materialization, row order, digest stability)

[paddymul]

Implements the cache redesign that #184 proposes: ADR-007 (tallyman owns result materialization), ADR-008 (every file carries __row_order, every page sorts by it) and ADR-009 (digest stability).

Stacked on #184. The first four commits are that PR's ADRs, so merge it first and this diff shrinks to the implementation. Draft, like #184.

Status — 2026-09-22: the fix PRs stacked on this branch

This branch is the direction I am taking, and it is not ready for main yet. Reviewing it turned up 25 follow-up issues (#183–#211). Rather than reopen this PR later, each fix lands as its own PR into this branch, TDD'd red then green on CI — 605a677 changed the workflow so pull requests into feat/** run it.

Open on top of this branch, all four green and none merged:

PR	Closes	What it changes
#213	#193	A failed build keeps the snapshot already on disk for its hash. materialize(..., publish=False) stages the file under a temp name; the build publishes it after write_manifest, so a failure between the two no longer deletes a good file.
#214	#194, #195, #196	A pin survives a reset and a dismissed error banner. A retired entry dir stays in the bullpen and a live dir wins when the same hash is re-added; an unfaithful heal is recorded in the manifest (unfaithful_heal_digest) instead of being inferred from errors.jsonl; a corrupt line in errors.jsonl is skipped instead of hiding the log. The Cache page labels a snapshot a reset retired.
#215	#197, #198, #211	An ordered copy keeps its source's column types, because pyarrow writes it instead of polars. The first CSV copy uses the reader options the caller passed, a polars panic becomes a BuildError, and the .digest sidecar is written through a temp file so a reader never sees an empty one.
#216	—	The docs describe the system as built: ADR-007, ADR-008 and ADR-009 marked implemented, ADR-010 marked rejected, and docs/architecture.md rewritten as the overview doc. Docs and plans only.

Merge order: #213, #214 and #215 first, in any order, then #216 rebased — it currently describes those bugs as open in about twenty places.

Still open against this branch

• Next tier: full_diff keeps __row_order as a data column — one inserted row shows every later row's position as changed #200 (full_diff shows __row_order as a changed column), the snapshot writer drops any column named __row_order_right, including one the author made #206 with assert_joinable refuses semi and anti join chains over entries — they add no right-side columns, so nothing collides #199 (the right side's __row_order and semi/anti chains), a build on the companion's event loop freezes the whole UI while it waits for the project lock — no way to show a busy indicator #190 with reload_project_sessions posts /reload_expr once per catalog entry, one after another, from async handlers #201 (builds and session reloads run on the companion's event loop), cache_worthy falls back to "a snapshot exists" when the manifest is missing — a worthy entry that has lost both is served as cheap #204 with _ResultPlan keeps the unused loaded build, and its backend objects, alive in a 256-entry LRU #210 (a manifest-less entry is served as cheap; the plan LRU holds unused builds alive), an unfaithful heal of a worthy parent changes its cheap children's rows under their hashes — only the parent is flagged and refreshed #208 with an unfaithful heal runs its checks and the companion's forced Buckaroo reload while holding the project lock #203 (an unfaithful heal changes cheap children under their hashes, and runs under the project lock), every grid open now posts /load_expr — concurrent opens load twice, and a promoted diff re-runs Buckaroo's stats pipeline on each open #202 (every grid open posts /load_expr).
• After that: ordered copies left behind by source edits are never listed or removed — compute_cache/ grows with no way to see it #207, ensure_expanded_build's marker does not record the project root — a copied project keeps reading the old path #209, the canonical sort leaves nested columns out of the tie-break — rows tied on every sortable column can still differ #205, an ungrouped float SUM/AVG depends on the parent file's row-group layout, even single-partition — a heal after a format change would be flagged unfaithful #187, non-pure recipes: the purity verdict is never recorded and does not propagate — an entry built on a non-pure parent is recorded as reproducible #185, one project-wide write lock turns slow work into blocked work — a page request that needs a heal waits behind any build in the other process #186, two tallyman servers on one project is unsupported and nothing detects it — each holds in-process state the other never sees #183, diffs: build every diff as an entry before it is displayed — moved out of ADR-007 (was D10) as a follow-on to the cache redesign #188.
• Found while rewriting the docs, no issue filed yet: every staleness scan overwrites source_digests.json with a single entry, so the next build re-hashes every source; a child pinned to a parent by raw hash is permanently stale on the source axis and reports as an UNEXPLAINED orphan; a recipe resolves the project from the active_project file while the MCP tool builds into the session's project; the SPA ignores six SSE events (entry_added, alias_changed, alias_renamed, display_changed, project_reset, unfaithful_heal).
• Two open decisions on fix(xorq): an ordered copy keeps its source's types and reader options, and its digest file is written atomically (#197, #198, #211) #215: whether to dictionary-encode string columns only (an ordered copy of numeric data is 47% larger than its source, because 122,880 distinct int64s fit under pyarrow's 1 MiB dictionary page limit), and whether a function-valued CSV reader option should be refused outright (its repr carries a memory address, so the entry's hash forks on every build).

What changed

Terms, once: an entry is one catalog computation. A worthy entry is one tallyman writes a parquet file for, called its snapshot, because it does expensive work (an aggregate, join, sort, window function, UDF, union). A cheap entry is a row-preserving plan over one file and keeps no file of its own. An ordered copy is the parquet copy of a source that carries __row_order. A clone is the copy of a source under data/.cas/, named by the digest of its bytes.

• No xorq cache node is in any build (ADR-007 D1). A recipe that calls .cache() is a build error. rewrite_cache_dirs, classify_build, snapshot_key, entry_graph_expr and the source-read injection are deleted, and a test keeps XORQ_CACHE_DIR empty through a build, a chained child, a view, a delete and a reopen.
• One writer, materialize. It runs the entry's frozen build on a single-partition connection, writes the snapshot in a pinned layout (zstd, 1,048,576-row groups, a parquet page index, __row_order last) to a temp name and replaces the file atomically, under the project's write lock. A create runs the query twice; a recipe that is not reproducible is recorded and its file is pinned (ADR-007 D4, ADR-009 D1, D3, D6).
• ensure_materialized makes every file an entry's plan reads exist before anything runs: a snapshot by recursing on the hash in its name, an ordered copy from its clone with the reader options in the manifest, a clone from the live source while the bytes still match. What it writes is checked against the recorded digest; a mismatch is loud and pinned (ADR-007 D5, D13).
• Chaining a worthy parent is a bare read of its snapshot, so a child's hash follows its parent's and a filter over an aggregate is cheap (ADR-007 D3).
• __row_order is an int64 0..N-1 last column on every snapshot and ordered copy. /api/data pages by ORDER BY __row_order. A cheap entry that drops the column fails to build with the fix shown; assigning to it is an error; every order_by gets the natural order as a tie-break and a non-final one is kept; three-way joins get an instruction; the diff drops the column from both sides (ADR-008).
• Cheap or worthy is one allow-list test, decided when the entry is built and recorded in the manifest. expr.yaml is no longer parsed for it (ADR-008 D4, closes Cache classifier parses xorq build YAML with regex — silent misclassification if the format drifts #12).
• Every source enters through an ordered copy keyed by the source's content digest and reader options, under compute_cache/ordered_sources/. Editing a CSV and re-running a recipe now forks the hash (closes tallyman_read_csv bypasses source identity — content-blind intermediate key, in-place re-materialization, no manifest.sources #168). A raw xo.deferred_read_parquet is a build error (ADR-008 D2, D7, D12).
• result_digest is arrow-sha256:, a digest of the file's Arrow content read back, so it does not move with the codec, row-group size or pyarrow version. An unfaithful heal names an engine change when the recorded versions differ (ADR-009 D2, D4).
• Buckaroo is handed files that exist. A worthy entry's grid gets a view build of its snapshot, a cheap entry its own build. Session ids are entry-<project>-<hash> and tallyman keeps no record of them; a klass reload posts one request per entry; an unfaithful heal forces a reload; every /load_expr names row_order_column (ADR-007 D6, ADR-008 D8).
• Reset leaves compute_cache/ alone and moves clones into the bullpen instead of deleting them, which fixes an entry that failed after a reset back and forward. The project lock is re-entrant per thread (ADR-007 D11, D14).
• Cache page: lists every snapshot file (a file whose entry is gone is an orphan row), marks a snapshot pinned, and refuses to delete a pinned one with the reason.

A snapshot's name is now the entry's content hash, which is the same on every machine, so this should also fix #77 (an empty grid on a clone of the project at another path). I did not test it on a second machine.

docs/system-contract.md and the descriptive docs are rewritten to match. Each of ADR-004 to ADR-009 has an updated status line, and ADR-007, ADR-008 and ADR-009 have an "Implementation notes" section that records where the code differs from the text. The differences a reviewer should look at: the ordered-copy key uses the source digest and not the clone's path, the reader options live in a new manifest.ordered_copies, a three-way join is refused by an explicit check because ibis no longer raises through the new canonical sort, and hoisting a non-final order_by uses only the keys the author wrote.

How it was built

1. f9bb94b: 180 failing tests for the three ADRs in one commit. CI ran it: ruff passed and the fast suite failed with 180 failed and 688 passed, which matches the local run.
2. The implementation, in the commits after it. CI on 283c483: ruff, the fast suite (861 passed), the integration suite (7) and the perf report all pass.
3. Updates to the existing tests. About 115 changed for the two deliberate build rules: a cheap recipe now has to keep __row_order, and a raw parquet read is refused. Tests of retired mechanisms (xorq cache nodes, the session map, compute-cache pruning) were deleted or rewritten against the new one; the commit message lists each deleted test and the decision that retired it.

Checked locally before pushing: the fast suite (860 passed), the integration suite against a real Buckaroo (7), the on-demand cache_lab suite (11), a scripted create, revise, cascade, live diff, promote and reset back and forward against a real Buckaroo, and the Cache page in Chromium.

Not in this PR

• The audit's independent bugs (ADR-007 D9, step 3): the chart error loop, eager notebook sessions, the staleness scan re-hashing every source, tallyman pack shipping compute_cache/ (it now ships the ordered copies too), the primary-key search, the over-broad stat-cache wipes. They change no hash.
• Buckaroo's half of the paging fix, xorq paging is not repeatable: window_to_parquet sorts on one key with no tie-breaker, and does not order at all when unsorted buckaroo#974. Tallyman sends the hint; Buckaroo 0.15.6 ignores it, so the grid's own paging is still unstable above 10 MB until that lands.
• non-pure recipes: the purity verdict is never recorded and does not propagate — an entry built on a non-pure parent is recorded as reproducible #185 (non-pure recipes), one project-wide write lock turns slow work into blocked work — a page request that needs a heal waits behind any build in the other process #186 (the wait behind the project lock), diffs: build every diff as an entry before it is displayed — moved out of ADR-007 (was D10) as a follow-on to the cache redesign #188 (diffs built as entries).
• A snapshot that is not valid parquet is not detected by ensure_materialized, which only checks that the file exists. verify_result_faithful raises on it and the verify sweep reports it under errors.

After merge

Every worthy entry's hash, snapshot path and digest change, so the corpus has to be rebuilt once (ADR-007 D9). ~/.cache/xorq/result_cache and the older ~/.cache/xorq/parquet/ can then be deleted by hand.

🤖 Generated with Claude Code

[paddymul]

Review pass (/code-review high, main...this branch)

Four findings, plus one more turned up while fixing the fourth.

1. An ordered copy couldn't be made again for an entry whose manifest doesn't record it (ordered_copy.py). recreate_ordered_copy looked for the record only in the manifest of the entry being opened. Anything that reached a cheap entry's graph without _note_parent_records failed with SourceUnavailable once compute_cache/ was gone: a promoted diff, or cached_result_expr called from a recipe. Fixed: the record now comes from the owner's manifest, else from any entry in the catalog. The build that first reads a source records its copy, and entries are never deleted. Test in d7466bb (seen failing on CI), fix in 8bbd36b.

2. Two async def handlers (api_promote_diff, put_code) build on the event loop while holding the blocking project lock, which freezes the companion while they wait. Not fixed here; a blocked build leaves the workflow stuck either way. The UI side (a busy indicator, which first needs the loop to stay free) is a build on the companion's event loop freezes the whole UI while it waits for the project lock — no way to show a busy indicator #190.

3. A CSV outside data/ is recorded by absolute path, and staleness can't resolve it (reports unknown source: on every scan, never flags an edit). Not fixed here; filed as staleness never resolves a CSV recorded outside data/ — every scan reports it unknown, and an edit is never flagged #191.

4. scripts/profile_pageload.py imported the removed result_cache.classify_build. Fixed in 8bbd36b. The profiler had also drifted from the Tier-B harness in three ways, fixed in the same commit and checked on a corpus built with this branch:

   • it found entries by the retired result.parquet and so saw none;
   • it read the old aliases.json;
   • it paged over the bare build without making its files exist.

   It now reads the worthiness verdict from the manifest, and shares entry discovery, alias lookup and a new _paged_build helper with tests/test_perf_integration.py.

The review also checked these and found them fine: the build hash doesn't depend on snapshot mtime/inode, reductions inside mutate/filter count as worthy, the content digest doesn't depend on how rows are batched, and project_lock can be re-taken within a thread.