Skip to content

docs(plans): ADR-007 tallyman owns result materialization (proposed) - #180

Closed
paddymul wants to merge 6 commits into
mainfrom
docs/adr-007-tallyman-owned-materialization
Closed

paddymul wants to merge 6 commits into
mainfrom
docs/adr-007-tallyman-owned-materialization

Conversation

@paddymul

@paddymul paddymul commented Sep 18, 2026 •

Copy link
Copy Markdown
Contributor

Draft ADR for discussion. Adds plans/ADR-007-tallyman-owned-materialization.md and the spike it cites, scripts/spike_bare_read_chaining.py. No behaviour change.

The proposal removes xorq's cache from tallyman's data path. rewrite_for_build stops calling .cache() (the only place in src/ that creates a result cache node), and the things that call gave us for free get tallyman-owned replacements:

Today (a consequence of the CachedNode) Proposed
File named by xorq's snapshot key, recorded as manifest.snapshot_key and asserted on read compute_cache/result_cache/<content_hash>.parquet, a function of the hash (D2)
Written by xorq's ParquetStorage as a side effect of count().execute() One materialize() used by the build and every heal, unique temp name, os.replace, per-hash flock (D4)
Missing ancestors regenerated by nested cache nodes ensure_materialized(), driven by the snapshot paths the build itself reads (D5)
Children inline the parent's graph and cache node Children of a worthy parent read its snapshot, so the child hash is a function of the parent hash (D3)
Buckaroo replays the entry's build Buckaroo gets a one-node view of the snapshot and never executes an expensive graph (D6)

Revised 2026-09-20 in the grilling session. Three additions:

  • A governing rule, stated by Paddy: Buckaroo is a displayer that runs queries only for summary stats, sorting and paging, and tallyman runs an entry's computation or a diff to completion before asking it to show anything.
  • Question 1 of the session is resolved under D5. The self-repairing build that ADR-006 decision D4 (chaining inlines the parent's cache node) bought had one user, Buckaroo's replay of builds, which the rule forbids, so nothing is given up.
  • D10: every diff takes the promote path and is built before it is displayed. An unnamed diff is an ephemeral entry under compute_cache/, because the checkpoint commits every directory under entries/. Promote becomes "put an alias on it".

The ADR now has a Terms section, and every other ADR's decision label is written with its ADR number and what it decides.

Supersedes ADR-006 D4 and D8 if accepted. ADR-006 rejected bare-read chaining because "the pre-heal choreography stays load-bearing forever"; D5 answers that directly and states what is given up. The existing ADRs are not edited here. Their status lines change when this one is accepted.

Spike results (uv run python scripts/spike_bare_read_chaining.py): the child's hash is unchanged when the snapshot's bytes change and changes when its path does; a child cannot be built while the parent's snapshot is absent; a filter plus a computed column over an aggregate's snapshot is classified cheap where the inlined design classifies it worthy; nothing is written under XORQ_CACHE_DIR.

One of three ADR PRs from the 2026-09-18 cache audit. ADR-008 (row order of reads, #181) and ADR-009 (digest stability, #182) are separate PRs, so the references to them here do not resolve until those merge. The documents are independent. The implementation is not: the hash-changing parts of all three share one corpus rebuild (D9).

Checked locally: ruff check and ruff format --check on the script, the script runs from a clean temp dir, and every path and line reference in the ADR resolves on a tree with all three ADRs present. Docs-only, so CI was not watched.

🤖 Generated with Claude Code

Proposes removing xorq cache nodes from builds: snapshots at
compute_cache/result_cache/<content_hash>.parquet written by one
tallyman-side writer, bare-read chaining through worthy parents,
ensure_materialized as the single entry point that makes files exist,
and a one-node view build for Buckaroo. Supersedes ADR-006 D4 and D8.

scripts/spike_bare_read_chaining.py is the pre-check the ADR cites: a
bare-read child's hash depends on the snapshot's path string only, the
child is classified cheap, and nothing is written under XORQ_CACHE_DIR.

Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
paddymul and others added 5 commits September 18, 2026 17:08
The diff compare post (app.py:1237) is a second handoff besides
load_session. Its expression is composed from cached_result_expr, so it
is covered at post time; the eviction rule now includes diff sessions.

Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
D6 claimed Buckaroo never executes a Join on tallyman's behalf. That is
true for entry grids only: _build_compare_expr posts the outer-join
compare expression itself. Scope the claim and add the gap as an open
question.

Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
…), and defined terms

From the grilling session.

- Governing rule (Paddy): Buckaroo is a displayer that runs queries only
  for summary stats, sorting and paging; tallyman runs an entry's
  computation or a diff to completion before asking it to show anything.
- Question 1 resolved under D5: the self-repairing build that ADR-006
  decision D4 bought had one user, Buckaroo's replay, which the rule
  forbids, so nothing is given up.
- D6 rewritten as the rule applied to entry grids; a cheap entry is
  handed as a view, pending confirmation (open question 4).
- D10: every diff takes the promote path and is built before it is
  displayed; an unnamed diff is an ephemeral entry under compute_cache/,
  because the checkpoint commits every directory under entries/.
- manifest.parents is shown incomplete: a promoted diff records no parent
  edges, which supports reading required snapshots from the build.
- Adds a Terms section; every other ADR's decision label now carries its
  ADR number and what it decides.

Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
Paddy: tallyman runs the query and materializes the parquet if necessary
when the entry is created, and does not call Buckaroo to display an entry
until that has finished. So a cheap entry's grid is handed the entry's
own build as a view, and nothing is written when an entry is viewed.
Closes open question 4.

Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
…efore speed

Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
@paddymul

Copy link
Copy Markdown
Contributor Author

Superseded by #184, which combined the ADR-007, ADR-008 and ADR-009 drafts; that text is in #189.

@paddymul paddymul closed this Sep 24, 2026
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant