diff --git a/README.md b/README.md index 3251b2f..4cafd4c 100644 --- a/README.md +++ b/README.md @@ -4,7 +4,7 @@ Tallyman — a data science environment designed for coding agents. Stop squinti Two windows: Claude Code in one, a browser in the other. You work by defining a set of named results that can depend on each other, then making sure those results live up to their name. You describe them; the agent writes the queries. You tell it what `likely_customers` should mean, and a moment later the answer is a table in the browser at full size, sortable and searchable, with statistics over every column. Your prompt sits above the query the agent wrote, so you read your intent, the code that came out of it, and the result together. You notice it's catching people who already churned, so you sharpen the sentence and the agent revises the query. Everything built on top of that name updates to match, fast enough that you don't lose your place. How big the data is, what has already been computed, and what needs recomputing never enter into it. -Those queries are expressions: self-contained programs that declare the raw files and other results they read. Writing the expression is the agent's entire job. An expression is declarative and can be introspected, so tallyman reads its dependencies straight off it and builds a directed graph of the project's aliased expressions. When a parent updates, its children are recomputed; a raw file changing on disk counts as an update too. Tallyman executes each expression once and writes the result and its summary statistics to disk. Reading it back is out of core: scrolling, sorting, and searching pull only the pieces of data needed to fill the screen, so nothing has to fit in memory and four million rows opens like four thousand. +Those queries are expressions: self-contained programs that name, by alias, the imported data and the other results they read. Writing the expression is the agent's entire job. An expression is declarative and can be introspected, so tallyman reads its dependencies straight off it and builds a directed graph of the project's aliased expressions. When a parent updates, its children are recomputed; importing a new version of a data file counts as an update too. Tallyman executes each expression once and writes the result and its summary statistics to disk. Reading it back is out of core: scrolling, sorting, and searching pull only the pieces of data needed to fill the screen, so nothing has to fit in memory and four million rows opens like four thousand. You aren't typing `df.sort_values('lifetime_sales')` just to see which customers are at the top, you aren't waiting for the LLM to print a 10 row table like it's coming out of a 1200 baud modem. You aren't running out of memory in the middle of a session. You aren't building the ad-hoc cache that every long notebook grows, the pickle in /tmp behind an if not exists guard that you never quite trust. You aren't nursing a kernel along for days because one cell takes five minutes to rerun, or bracing yourself before you close the window. None of that is in your head while you work. What's in your head is the data and what it means. @@ -50,61 +50,77 @@ index of all the docs — start with [docs/architecture.md](docs/architecture.md ## V0 scope -End-to-end: a Claude Code MCP tool that compiles a xorq expression, materializes -a result to a content-hashed catalog entry on disk, and pushes a live update -to a browser companion via SSE. +End-to-end: a Claude Code MCP tool that compiles a xorq expression into a +content-hashed catalog entry on disk (running it, and writing its result to a +file when the entry does expensive work), and pushes a live update to a browser +companion via SSE. What's working: -- **MCP tools** (FastMCP over stdio): - - Catalog: `catalog_run`, `catalog_load_parquet`, `catalog_create`, +- **MCP tools** (FastMCP over stdio, 31 tools): + - Catalog: `catalog_import_source`, `catalog_run`, `catalog_create`, `catalog_revise`, `catalog_alias`, `catalog_rename`, `catalog_unalias`, - `catalog_list`, `catalog_diff`, `catalog_chart`, `catalog_recalc`, plus the - summary-stat / post-processing / display-klass authoring tools. + `catalog_list`, `catalog_diff`, `catalog_promote_diff`, `catalog_chart`, + `catalog_chart_errors`, `catalog_scan_staleness`, `catalog_recalc`, + `catalog_export_marimo`, plus the summary-stat / post-processing / + display-klass authoring tools. - Notebook: `notebook_reorder`, `notebook_remove`, `notebook_edit_markdown`. - Project: `project_list`, `project_new`, `project_switch`. - See [docs/architecture.md](docs/architecture.md) for the full tool surface. + See [docs/mcp-server.md](docs/mcp-server.md) for every tool and its side + effects. - **Companion** (FastAPI on `:7860`) — serves the React SPA (`packages/app/dist`) as a catch-all and exposes a JSON API + SSE under `/{project}/api/*`: - - SPA tabs: **Catalog** (entry list + detail with V_n chips and forensic - history), **Notebook** (curated narrative anchored on aliases, drag-reorder, - inline markdown editor, × remove), **Diff** (code diff, schema diff, - per-column stats, key-joined side-by-side, head() side-by-side), **Cache** - (per-entry cache footprint), and **Log** (linear, filterable activity view). - - JSON: `/{project}/api/{entries,entry/,aliases,notebook,errors,log, - data/,diff_data/...,disk_usage,result_cache,staleness}`, plus the - mutation routes (`PATCH notebook`, `PUT code/`, + - SPA pages: **Catalog** (entry list + detail with V_n chips, forensic + history, and a metadata tab with the entry's disk footprint, sources, + parents and children), **Notebook** (curated narrative anchored on aliases, + drag-reorder, inline markdown editor, × remove), **Diff** (code diff, schema + diff, per-column stats, key-joined side-by-side, head() side-by-side, and a + promote button), **Cache** (the result snapshots on disk, with a delete + button; pinned snapshots cannot be deleted, and a snapshot whose entry a + reset retired is labelled as such), **Log** (linear, filterable + activity view), and the project list. + - JSON: `/{project}/api/{entries,entry/,entry_cache/, + session/,aliases,notebook,notebook_full,errors,error/,log, + data/,diff_data/...,disk_usage,result_cache,staleness,telemetry}`, + plus the mutation routes (`PATCH notebook`, `PUT code/`, `PUT markdown/`, `POST reset`, `POST recalc`, - `POST promote_diff/...`). - - `/{project}/api/sse` — live updates (`new_entry`, `build_failed`, - `alias_changed`, `notebook_changed`, `recalc`, `summary_stat_changed`). + `POST promote_diff/...`, `DELETE result_cache/`, `DELETE errors`). + - `/{project}/api/sse` — live updates. The SPA listens for `new_entry`, + `build_failed`, `notebook_changed`, `chart_attached`, + `post_processing_changed`, `summary_stat_changed`, `recalc` and + `project_switched`. - `/internal/notify` — the MCP server's notification hook; fans out to SSE. - **Buckaroo subprocess** — `tallyman run` spawns `python -m buckaroo.server` on `:8700` (falls back to a random port if busy), watches for the - `BUCKAROO_PORT=...` handshake, and lazily opens per-entry sessions on - first view by POSTing a build dir to Buckaroo's `/load_expr` endpoint - (PR 776), after making sure every file the entry reads exists — sort/search - push down to the xorq backend. A *worthy* entry (one whose query tallyman - materialized to a result file when the entry was created) is handed a view - build, a build that is one read of that file (`.xorq_view_build/`). A *cheap* - entry (a filter, selection or computed column over one file, which keeps no - file of its own) is handed its own build, expanded into a stable per-entry - path (`.xorq_build_expanded/`, gated by a `.complete` marker) so - `${TALLYMAN_PROJECT_ROOT}` placeholders are resolved before xorq's loader sees - them. A session's id is derived from the project and the content hash, so - tallyman keeps no session file. Tear-down rides along with the companion. - Disable with `--no-buckaroo`. + `BUCKAROO_PORT=...` handshake, and opens a session for an entry by POSTing a + build dir to Buckaroo's `/load_expr` endpoint (PR 776), after making sure + every file the entry reads exists. It does this when an entry's catalog page + opens, and for every cell each time the notebook page loads. A *worthy* + entry (one whose query tallyman materialized to a result file when the entry + was created) is handed a view build, a build that is one read of that file + (`.xorq_view_build/`). A *cheap* entry (a filter, selection or computed column + over one file, which keeps no file of its own) is handed its own build, + expanded into a stable per-entry path (`.xorq_build_expanded/`, gated by a + `.complete` marker) so `${TALLYMAN_PROJECT_ROOT}` placeholders are resolved + before xorq's loader sees them. Buckaroo's sorting, search and summary stats + run as queries over that build. A session's id is derived from the project and + the content hash, so tallyman keeps no session file. Tear-down rides along with + the companion. Disable with `--no-buckaroo`. - **Build artifacts are portable.** xorq's absolute filesystem paths are - rewritten to `${TALLYMAN_PROJECT_ROOT}` on write and expanded back on load. + rewritten to `${TALLYMAN_PROJECT_ROOT}` on write and expanded back on load + (with one known gap for copied projects, #209). - **`tallyman serve `** — read-only companion against a project - directory that may live anywhere on disk. Mutation routes return 403. + directory that may live anywhere on disk. Mutation routes return 403, and no + Buckaroo subprocess runs, so entry grids do not load. What's NOT yet implemented: 1. Column-level lineage (xorq has the data; there is no lineage view today). -2. ML training pipeline (storyboard beats 7-8). +2. A dedicated ML training tool, `catalog_train` (storyboard beats 7-8, #2). + Models can already be fitted as catalog entries with `xorq.ml`, as + `catalog_run`'s tool description shows. ## Running the spike @@ -128,7 +144,8 @@ In another terminal, launch Claude Code from this directory; it picks up Recommended prompts: -> Use catalog_load_parquet to load `orders.parquet`. +> Use catalog_import_source to import +> `~/.tallyman-notebooks/projects/spike/data/orders.parquet` as `orders`. > > Now use catalog_create to make a named entry `shoe_sales` that groups orders > by region and totals the price. @@ -150,26 +167,36 @@ tar xzf my-project.tgz -C ~/projects/ uv run tallyman serve ~/projects/spike ``` -The companion runs read-only: same catalog, same forensic history, no edit -affordances. Mutation routes return 403. +The companion runs read-only: same catalog, same forensic history, same charts. +The edit controls still show, but mutation routes return 403, and there is no +Buckaroo grid. +The archive includes `compute_cache/`, and a known defect (#209) makes a copied +project's cheap entries read from the original location; see +[docs/installing.md](docs/installing.md#sharing-a-project). ## Conventions worth knowing -- Recipes read data with `read_project_file` (parquet under `data/`), - `tallyman_read_csv` (CSV) and `tracked_expr_from_alias` / - `pinned_expr_from_alias` (catalog entries). `xo.deferred_read_parquet` on a - file outside the project's `compute_cache/` is a build error, since the file - would get no content digest and no `__row_order` column, and `xo.read_parquet` - resolves through ibis's backend loader and fails. Use - `import xorq.api as xo` and `import xorq.vendor.ibis as ibis`. Do NOT - `import ibis` directly. -- Prefer `from tallyman_xorq.io import read_project_file; t = read_project_file("name.parquet")` - over absolute paths — the catalog records project-relative intent and the - build is portable across machines/users. +- A file enters the catalog only through `catalog_import_source(path, alias)`, + which copies its bytes into the project and makes each version of it an entry + under a **source alias**; the path can be anywhere and is never read again. + Import it again to bring in new data: different bytes mint the next version, + and the entries downstream are recalculated. A CSV takes its `schema` and + reader options in the import call, and they are fixed there. +- Recipes read entries by alias, never files: + `tracked_expr_from_alias("orders")` follows an alias and + `pinned_expr_from_alias("orders-v2")` pins one version. `read_project_file`, + `tallyman_read_csv` and `xo.deferred_read_csv` are build errors that name the + import to use, and so is `xo.deferred_read_parquet` of any file outside the + project's `compute_cache/`. `xo.read_parquet` resolves through ibis's backend + loader and fails. Use `import xorq.api as xo` and `import xorq.vendor.ibis as + ibis`. Do NOT `import ibis` directly. - Content hash is xorq's build hash — same code + same inputs → same hash → same - entry dir (idempotent). -- All catalog state lives on disk. The MCP server holds no in-memory state; the - companion only holds the SSE subscriber list. + entry dir (idempotent). A source entry's hash is instead an md5 of the + imported bytes and the reader options. +- All catalog state lives on disk. The MCP server and the companion keep only + in-memory caches of things that never change (loaded builds and reads, keyed + by content hash), plus the MCP session's active project and the companion's + SSE subscribers and diff sessions. - `TALLYMAN_PROJECT_PATH` overrides project_dir() resolution for the active project. Used by `tallyman serve` to point at a project directory anywhere on disk. diff --git a/docs/architecture-new.md b/docs/architecture-new.md new file mode 100644 index 0000000..1405b41 --- /dev/null +++ b/docs/architecture-new.md @@ -0,0 +1,859 @@ +# Tallyman architecture + +This document describes tallyman as the code in `src/` builds it. Where it and +the code disagree, the code is right. The rules the system promises are stated +normatively in [system-contract.md](system-contract.md), and +[caching.md](caching.md), [expression-lifecycle.md](expression-lifecycle.md), +[reactive-recalc.md](reactive-recalc.md) and [mcp-server.md](mcp-server.md) each +go deeper on one subject. The reasons for the design are in +[ADR-007](../plans/ADR-007-tallyman-owned-materialization.md) (tallyman writes +its own result files), [ADR-008](../plans/ADR-008-row-order-of-reads.md) (every +file carries a row-order column), [ADR-009](../plans/ADR-009-digest-stability.md) +(what the result digest covers) and +[ADR-011](../plans/ADR-011-sources-are-aliases.md) (a raw input is an alias). + +## 1. What tallyman is + +Tallyman is a notebook without cells. Its unit of work is an **entry**: one +dataframe result, stored on disk under a content hash and usually given a name. +An agent, Claude Code, writes each entry as a short Python **recipe** that +builds a xorq expression (a deferred dataframe computation, which runs only when +executed) over other entries, which it reads by name. Tallyman freezes the +expression, runs it, stores the result and records which entries it read, so +that revising one entry can recompute the entries built on it. Data files come +in through an explicit import that makes each version of a file an entry too. A +React app shows the catalog in the browser, and Buckaroo, a separate data-grid +server, draws each entry's rows. + +## 2. The processes + +| Process | Started by | Owns | +|---|---|---| +| MCP server, `src/tallyman_mcp/server.py` | Claude Code runs `tallyman mcp`, one per session, over stdio | the session's active project; the builds and imports the agent asks for | +| Companion, `src/tallyman_companion/app.py` | `tallyman run`, a FastAPI app on port 7860 unless `--port` names another | the API and event stream the browser uses; the builds, recalcs and resets the browser asks for; the Cache page's delete; the Buckaroo subprocess | +| Buckaroo | the companion, through `BuckarooManager` (`buckaroo_lifecycle.py`) | grid sessions and their queries: paging, sorting, search and summary statistics | + +**The MCP server** has 31 tools and one prompt. It remembers the session's +active project, read from `~/.tallyman-notebooks/active_project` on the first +call and changed only by `project_switch` or `project_new`, which go through the +companion. `_with_checkpoint` commits a checkpoint (one git commit of the +catalog, section 10) after every tool that returns without an error, except +those in `_NO_CHECKPOINT`: most read-only tools, the project tools, and +`catalog_recalc` and `catalog_promote_diff`, which commit their own. After a +change the server posts to the companion's `/internal/notify`, best effort, and +it posts nothing when no companion serves its projects (below). + +**The companion** serves the React app, a JSON API under `/{project}/api/`, and +a stream of Server-Sent Events (SSE, named messages on an HTTP response the +browser keeps open) at `/{project}/api/sse`. It builds through +`PUT /{project}/api/code/{alias}` (the code editor) and +`POST /{project}/api/promote_diff/{alias}/{va}/{vb}`, and its middleware +`_checkpoint_after_mutation` commits a checkpoint after every successful non-GET +request, except on routes that commit their own or change no authored state. +`tallyman serve ` runs a read-only companion with no Buckaroo. + +**One server per data dir.** The **data dir** is the directory that holds every +project, `~/.tallyman-notebooks` unless `TALLYMAN_HOME` names another. +`tallyman run` claims it before it starts anything else +(`tallyman_core/server_lock.py`): it takes an exclusive `flock` on +`/server.lock` and holds it on a descriptor it keeps open while it +serves, so the kernel drops the lock however the process exits. The file also +holds the **owner record**, JSON naming the server's pid, host, port, bind +address, start time, data dir and command line. A second `tallyman run` on the +same data dir is refused with a message that names the holder and says how to +run a second tallyman on its own data dir and port. Each server holds state no +other process sees (its SSE subscribers, its Buckaroo sessions, its memos), so +two serving one project would each show a view the other's writes never reach. +The MCP server and `tallyman reset-to` find their companion on each call with +`server_lock.companion_url()`, from the owner record's port, and believe the +record only while its lock is held. Nothing else names the companion: no +default port and no environment variable. With no server on the data dir there +is no companion, so notifies are skipped, tool replies carry `url: null`, and +`project_switch` and `project_new` return an error. Each notify, project switch +and project creation carries its data dir as `home`, and a companion serving +another data dir answers 409 and changes nothing; the browser sends no `home` +and is not checked. `tallyman mcp`, one per Claude Code session, and +`tallyman serve` claim nothing. + +**Buckaroo** runs as `python -m buckaroo.server --port 8700 --no-browser +--stdio-control` and exits when its stdin closes. It displays, and queries only +for statistics, sorting, search and paging; tallyman finishes an entry's +computation before handing it over, so a computation never fails inside a grid +query. Buckaroo keeps each grid's state as a **session** in memory, and drops a +session that has had no browser attached for an hour. + +**xorq** is a library. Tallyman uses its expression API (`xorq.vendor.ibis`, an +ibis fork), `build_expr` to freeze an expression to a directory named by its +12-character hash, `load_expr` to load one back, and its embedded DataFusion +engine, the only backend. xorq's hash covers a graph's structure and the paths +of the files it reads, not their contents. Tallyman does not use xorq's cache: +no build contains a xorq cache node, a recipe that calls `.cache()` fails to +build, and tallyman writes, names, verifies and re-creates every result file +itself (`tallyman_xorq/materialize.py`). The catalog store is tallyman's own +(`tallyman_core/catalog.py`, `catalog_state.py`). + +Both the MCP server and the companion write the catalog. The **project lock** +(`catalog_state.project_lock`, a `flock` on `artifacts/catalog/.checkpoint.lock`) +makes their builds, imports, heals (re-creations of a missing result file, +section 7), checkpoints and resets happen one at a time. It is re-entrant within +a thread and blocks with no timeout. Smaller writes (an alias, a notebook cell, a +chart, a display config, `config.json`) take no lock and replace their whole +file atomically. + +**The execution lock.** Each process executes reads on one DataFusion session, +xorq's default backend (`xorq.config.default_backend()`), and two threads +executing on it at once fail with `RuntimeError: Already borrowed`. Both +processes run work on thread pools (FastAPI's for requests, FastMCP's for tool +calls), so every execution on that backend holds `execution.execution_lock`, one +re-entrant lock per process: `/api/data` pages, post-processing runs, the +primary-key probes and `full_diff`'s Buckaroo helpers. Executions in one process +therefore run one at a time, and a long one, such as a diff's summaries, makes +that process's page reads wait. Another process has its own backend and its own +lock. The project lock comes first: anything that can heal +(`cached_result_expr`, `ensure_materialized`) runs before the execution lock is +taken, and `project_lock` raises `RuntimeError` in a thread that holds the +execution lock and would take a new `flock`. Two executions run on connections +of their own and take no execution lock, so a build or heal does not hold up +page reads: a materialization's stream (`materialize._stream_to_parquet`) and a +cheap entry's row count at build (`result_cache.stream_row_count`). +`tests/test_execution_lock.py` checks the rule in the source: every execution in +`src/` sits inside `with execution_lock():` or is one of those two, and nothing +that can take a project lock is called inside that block. + +## 3. The project on disk + +A project is `~/.tallyman-notebooks/projects//`; `TALLYMAN_HOME` moves +the root. A file is **cache** when tallyman can make it again from files that +are not cache and check what it made, so anything may delete it (ADR-007's +rule). A file is **record** when nothing can re-create it. A **log** only +explains what happened. Paths are under `artifacts/catalog/`, the catalog's git +repository, unless they start with `artifacts/` or `data/`. + +| Path | Holds | Kind | +|---|---|---| +| `aliases.jsonl` | one line per alias, `{alias, latest, history, kind}` | record, tracked | +| `entries.jsonl` | the complete entry directories the last checkpoint saw | record, tracked | +| `entries/.zip` | the **recipe zip**: a deterministic archive of the entry's `expr.py`, `schema.json`, `manifest.json` and `xorq_build/`, written by the first checkpoint after the entry is made; nothing reads it back | record, tracked | +| `config.json` | project settings, one key: `auto_recalc` | record, tracked | +| `notebook.jsonl`, `chart_specs/`, `display_configs/`, `prompts/`, `post_processing/`, `stats/` | notebook cells, per-entry charts and grid settings, the prompts each entry was built from, the project's statistic and post-processing functions | record, tracked | +| `entries//` | the entry directory (section 4) | record, untracked: every read uses it, and nothing re-creates it | +| `entries//.xorq_build_expanded/`, `.xorq_view_build/`, `.buckaroo_stat_cache/`, `primary_key.json` | per-entry derived files | cache | +| `compute_cache/result_cache/.parquet` | snapshots, the result files of worthy entries (section 4) | cache, unless pinned because it cannot be made again faithfully (section 8) | +| `bullpen/entries//`, `bullpen/cas/` | the **bullpen**: entry directories and clones a reset retired, kept so a reset forward can bring them back (section 10) | record, parked | +| `diff_stat_cache/-/` | Buckaroo statistics per diffed pair | cache | +| `data/.cas/` | clones: the bytes of every imported file | record | +| `artifacts/display/` | display classes, outside the repository | record, untracked | +| `artifacts/errors.jsonl`, `events.jsonl`, `telemetry.jsonl` | failures, the activity log, grid-load timings | log | + +The catalog's `.gitignore` keeps the untracked paths out of `git add -A`, and +`catalog.assert_catalog_consistent`, run after every reset, rejects any tracked +path outside `TRACKED_SURFACE`. Since nothing reads a recipe zip, cloning the +catalog repository does not recreate entry directories. The logs sit outside the +repository, so a reset does not rewind them, and `errors.jsonl` holds no state: +dismissing the error banner deletes it. `tallyman init` writes a fixture at +`data/orders.parquet`, which no build reads until it is imported. + +Two files sit at the top of the data dir, outside every project: +`active_project`, the name of the active project, and `server.lock`, the claim +and owner record of the `tallyman run` serving the data dir (section 2). A +server that exits leaves its record in the file, and nothing believes it once +the lock is gone. + +## 4. Core objects + +### Entries + +An **entry** is one computation, built once over inputs fixed forever, and +stored in `entries//`. Its **content hash** is its name: for a +computed entry, xorq's hash of its expression after tallyman's rewrite +(section 6). Every file that expression reads is another entry's snapshot, named +by that entry's hash, so a child's hash is a function of its parents'. The chain +ends at source entries, the entries an import makes, one per version of a file, +whose hash comes from the imported bytes (section 5). The absolute project path +is part of those read paths, so the same recipe in a project at another path has +another hash. An entry directory holds: + +- `expr.py`, the **recipe**, with the project path replaced by + `${TALLYMAN_PROJECT_ROOT}`. It names inputs by alias, so running it again can + mean something else; tallyman re-runs one only to mint a new entry in a + recalc, and in a diagnostic after an **unfaithful heal** (a re-created result + whose rows differ from the ones recorded, section 7). +- `xorq_build/`, the **build**: the expression as `build_expr` froze it, with + every input fixed and paths made portable (`portable.make_portable_inplace`). + A worthy parent appears as the path of its snapshot, a cheap parent as its + graph inlined. Once an entry exists, its build is what it means and its recipe + is documentation. +- `schema.json`: columns, types and the row count. +- `manifest.json`, the **manifest**: what the build does not record. It is the + directory's last write, made atomically by `manifest.write_manifest`, and a + directory without one is not an entry. After create, only an unfaithful heal + rewrites it. + +| Manifest field | Meaning | +|---|---| +| `content_hash`, `project`, `created_at`, `prompt` | identity and authorship | +| `parents` | `[{hash, ref, follow}]`, the parent edges | +| `cache_worthy`, `cache_worthy_why` | the worthy-or-cheap verdict and its reason | +| `row_count`, `execute_seconds`, `compile_seconds`, `cache_bytes` | measurements | +| `result_digest` | the content digest of a worthy entry's snapshot | +| `reproducible`, `nonreproducible_columns` | whether two runs at create gave the same digest | +| `unfaithful_heal_digest` | the digest an unfaithful heal wrote; it pins the snapshot | +| `snapshot_format`, `engine_versions` | the snapshot format version and the xorq, xorq-datafusion and pyarrow versions | +| `provenance` | a source entry only: where its data came from, `{alias, version, path, digest, suffix, reader, imported_at}` | + +A worthy entry's **snapshot** is its result as one parquet file, +`compute_cache/result_cache/.parquet` (`materialize.snapshot_path`), +a path that depends on the content hash alone. + +### Worthy and cheap entries + +`worthiness.classify_expr` decides once, at build, on the author's expression, +whether an entry is **worthy** or **cheap**. Tallyman **materializes** a worthy +entry: it runs it to completion and writes its snapshot when the entry is +created. A cheap entry has no file; its small plan runs again on every read, +over files that exist. + +An entry is cheap only if every relation in it is a file read, filter, +projection (renames, casts and computed columns), column drop, null drop or null +fill; it reads exactly one file; and no value in it multiplies rows (`Unnest`), +depends on row arrival order (a window function) or is impure (`random()`, +`uuid()`, `now()`, `today()`, any UDF). Anything else is worthy, including every +aggregate, join, sort, limit and union. The test is an allow-list because a cheap +entry pages by the `__row_order` column of the one file it reads (every +snapshot ends in one, numbering its rows `0..N-1`), so a wrong "cheap" +gives unstable paging where a wrong "worthy" costs a copy. A source entry is +worthy by definition, and a filter over a worthy entry is cheap, since it reads +a snapshot. + +### Aliases and versions + +An **alias** is a mutable name for a line of entries, a record +`{alias, latest, history, kind}` in `aliases.jsonl` (`tallyman_core/aliases.py`). +`latest` is the head, `history` holds versions V1 to Vn, and `-v` names +version N; a name matching that pattern is refused. `set_alias` moves the head +and appends to the history. + +A **catalog alias** names computed entries and moves by create, revise, recalc +and promote. A **source alias** names the versions of an imported dataset and +moves only by an import. A name is one kind or the other, and `set_alias` keeps +an alias's kind equal to its entries' kind on every route (`AliasKindMismatch`). +A source alias can be renamed or removed; the entry keeps the name it was +imported as, so a message naming a version asks the alias store which alias +holds it (`source_import.current_source_version`). An entry no alias has held, +built by `catalog_run`, is a **scratch entry**, and must be named with +`catalog_alias` before a recipe can read it. + +### Parent edges + +While a recipe runs, the readers in `tallyman_xorq/io.py` record each entry it +reads as a **parent edge** in `manifest.parents` (through `parent_capture.py`). +`tracked_expr_from_alias("sales")` records a **followed edge**, +`{hash: , ref: "sales", follow: true}`: the child goes stale when the alias +moves. `pinned_expr_from_alias("sales-v2")` records a **pinned edge**, with +`ref: "sales-v2"` and `follow: false`: the child stays on that version. Both +return the parent's result through `cached_result_expr` (section 7). The edges +are the whole record of what an entry depends on. `follow` decides what a recalc +does and has no effect on reads, which go through the build. + +### What a recipe may read + +A recipe names aliases, and a build reads only the **arena**, the files tallyman +owns: the clones under `data/.cas/` and the snapshots under +`compute_cache/result_cache/`. Everything else is refused at build, and the +message says what to write instead. + +| A recipe that | Is refused by | +|---|---| +| calls `read_project_file` or `tallyman_read_csv` | `io._source_entry_read` | +| calls `xo.deferred_read_csv` | `build._csv_direct_read_check` | +| calls `xo.deferred_read_parquet` on a file outside `compute_cache/` | `build._raw_parquet_read_check` | +| passes a content hash to either reader | `io` (ADR-011 D5, a bare content hash is refused in a recipe) | +| passes a bare alias to `pinned_expr_from_alias` | `io`: it would pin whatever the head happened to be | +| names an alias or version that does not exist | `io` (`ProjectDataNotFound`) | +| reads in-memory data (`ibis.memtable`, `read_in_memory`) | `source_cache.rewrite_for_build` | +| calls `.cache()` | `rewrite_for_build` | +| assigns to `__row_order` | `row_order.assert_not_assigned` | +| is cheap and drops `__row_order` | `row_order.require_on_cheap`, showing the corrected select | +| joins three entries in one chain, first and two right-hand sides carrying `__row_order` | `row_order.assert_joinable`, showing the `.drop("__row_order")` fix | +| sorts, then loses a sort key in a later order-keeping step | `row_order.canonical_sorted` | +| is a revision that follows its own alias | `catalog_revise` and `PUT /code`, after the build | + +Two generated recipes break the naming rule on purpose. A source entry's recipe +calls `read_project_file`, which a context variable +(`source_import._SOURCE_ENTRY`) resolves to that entry's own snapshot; the path +in the call is provenance. A promoted diff's recipe names two content hashes +(section 11). Operations are refused too: a revise or diff promotion aimed at a +source alias is refused before anything is built, with the text of +`aliases.source_alias_refusal` (the companion answers 409), and `catalog_alias` +refuses to give a source entry a catalog name. + +## 5. Data in: importing a file + +This picture covers sections 5 to 7. Solid arrows happen when an entry is +created, dotted ones when a missing snapshot is healed. + +```mermaid +flowchart LR + F["outside file"] -->|"catalog_import_source"| CL["clone in data/.cas"] + CL --> SS["source entry snapshot"] + SS -->|"tracked_expr_from_alias"| B["build_and_persist"] + B -->|"worthy: materialize"| WS["worthy entry snapshot"] + B -->|"cheap: no file"| FB["frozen build"] + CL -.->|"heal"| SS + FB -.->|"heal"| WS + WS --> R["cached_result_expr"] + FB --> R + WS -->|"view build"| G["Buckaroo grid"] + FB -->|"expanded build"| G +``` + +A file enters the catalog only by an **import**, the MCP tool +`catalog_import_source(outside_path, alias, pinned_version=None, prompt="", +schema=None, reader_options=None)`, which calls `source_import.update_and_depend`. +It copies the bytes into the arena and points a source alias at a new **source +entry**, an ordinary entry whose rows are the file's rows. It refuses a +directory, a missing file, an alias name in version syntax, a name that is a +catalog alias, and a suffix other than `.parquet`, `.pq`, `.csv`, `.tsv` or +`.txt`. It fixes the reader options, digests the file (md5) and computes the +entry hash, then, under the project lock, picks the case (`_plan_version`), +writes the entry if it does not exist (`_mint`) and moves the alias. "Bytes" +below means the bytes read with the given reader options. + +| State | Result | +|---|---| +| the alias does not exist | mint v1 | +| no `pinned_version`, bytes differ from the head | mint the next version | +| no `pinned_version`, bytes equal the head | nothing changes; the head is returned | +| bytes equal an older version | refused, naming `reset_to`, and `pinned_expr_from_alias("-v")` to read that version | +| `pinned_version=N` exists and matches | nothing changes; vN is returned and the head stays | +| `pinned_version=N` exists and does not match | refused: the file, or its reader options, are not that version's | +| `pinned_version` is head + 1, bytes are new | mint it | +| `pinned_version` beyond head + 1, or below 1 | refused: versions cannot be skipped | +| bytes are a version of another alias | refused, naming that alias and version | + +**The entry hash.** `source_entry_hash` is `md5("source||")` cut to 12 hex characters, the shape of xorq's hash. It cannot come +from xorq, because the entry's recipe reads the snapshot the hash names. It +covers the bytes and the reader options and nothing else, so two projects that +import one file each hold their own entry, snapshot and clone under one hash. + +**Reader options.** A parquet file takes none. A CSV takes a `schema` (a dict by +column name, or ADR-005's positional list with `"&rest"`) and `polars.scan_csv` +options such as `separator`; `infer_schema_length` and `schema_overrides` are +refused. The options must survive a round trip through JSON, so a callable is +refused (`_reader_for`). They are recorded on the entry and hashed with the +bytes (`ordered_copy._reader_signature`), so a file is read one way for good, +and a CSV read two ways is two imports under two aliases. A message that advises +an import prints them (`source_import.import_call`), since a call without them +names another entry. + +**The clone.** `source_identity.ensure_cas_path` copies the file to +`data/.cas/`, copy-on-write where the filesystem offers it, +digests the copy, and refuses it (`CloneDigestMismatch`) if the file changed +during the copy. The **clone** is the bytes as imported; it re-creates the +source entry's snapshot, and nothing deletes it (section 8). + +**The snapshot: pyarrow and polars.** The import writes the file's rows in file +order, plus a last `__row_order` column numbered `0..N-1`, to the entry's +snapshot path. Both readers feed one pyarrow writer, +`materialize.write_pinned_parquet`, with every snapshot's settings (zstd level 3, +parquet format 2.6, statistics, a page index) in row groups of 122,880 rows +(`ordered_copy.ORDERED_COPY_ROW_GROUP_ROWS`). A parquet file needs no parser: +pyarrow reads it with `ParquetFile.iter_batches`, so the file's types survive +exactly. A CSV is parsed by polars (`io._materialize_ordered`), which keeps row +order and runs ADR-005's schema language and inference ladder (100 rows, then +10,000, then the whole file, unless every column is pinned), and whose batches +reach the writer through `collect_batches` without the frame being held whole. + +**Zoned timestamps in a CSV.** A column whose schema type is a timestamp with a +zone, such as `timestamp('America/New_York')`, is read as text and parsed after +the scan (`io._parse_zoned`), because polars' CSV reader would parse text with no +UTC offset as UTC and convert it. Text with an offset is an instant and is +converted into the zone. Text without one is a wall-clock time in the zone and +keeps it, so `09:30` stays `09:30` New York time. This is ADR-005's rule for a +zoned schema on offset-less text (its D9(a): attach the zone, do not convert). +The import raises, naming the column, the row and the value, for offset-less +text that names a time the zone skips or repeats at a daylight-saving change, +for a column that mixes text with and without an offset (polars infers one +format per column from its first non-null value), and for text that is not a +timestamp. The first non-null value is checked before the read, since polars +1.40.1 writes nulls instead of raising when that value matches no format. The +declared precision is kept, `ns` included. + +**The entry.** `_mint` writes a generated `expr.py` whose header records the +path, digest and reader options, a real `xorq_build/`, `schema.json`, and last +the manifest, which marks the entry worthy and records the snapshot's +`result_digest` and the version's **provenance**, where it came from. The +presence of `provenance` is what makes an entry a source entry. Its `path` is +never read again, so moving, editing or deleting the original changes no build, +and its `alias` and `version` are the name the version was imported as. + +**One set of bytes, one alias.** Bytes that are already a version of another +alias in the project are refused (`_refuse_bytes_held_elsewhere`), and the error +suggests a catalog entry whose recipe is `tracked_expr_from_alias("")` as a second name. The rule keys on the entry hash, so one CSV under two +sets of reader options can be two aliases. + +**An existing version.** When the entry exists, the import rewrites none of its +files. If its snapshot is gone, the import restores the clone from the given +file (when the clone is gone too) and heals the snapshot through +`ensure_materialized`, verified against the recorded digest, so a repair never +changes a version's rows. A directory a crash left without a manifest is not an +entry, and is written again. + +**As a catalog operation.** A minted version is recorded as an `alias_set` +event, gets a notebook cell at v1 (or the previous version's chart and display +config), is announced with `new_entry`, and runs auto-recalc for the alias's +followers (section 9), all committed by one checkpoint. An import that changes +nothing records and announces nothing, and its checkpoint is an empty commit. + +## 6. The write path + +`catalog_run` builds a scratch entry. `catalog_create(name, code)` refuses a +source-alias name, an existing alias or a version-shaped name, then builds and +sets the alias. `catalog_revise(name, code)` and `PUT /code` refuse a source or +unknown alias, build, refuse a recipe that follows its own alias, and move the +alias. All call `build.build_and_persist`, which holds the project lock for the +whole build, so a child's build waits for its parent's materialization: + +1. **Run the recipe** (`_import_script`) with the parent collector armed. This is + the one moment names resolve; refused reads fail here. +2. **Lint.** `_nondeterminism_warnings` warns about `now()`, `today()`, + `random()`, `uuid()` and an unseeded `sample()`. +3. **Refuse raw reads** (`_csv_direct_read_check`, `_raw_parquet_read_check`). +4. **Classify and rewrite.** `classify_expr` gives the verdict, and + `source_cache.rewrite_for_build` refuses what section 4 lists, gives a worthy + entry the **canonical sort** (the author's keys, then `__row_order`, then every + other sortable column, a total order), and moves a cheap entry's + `__row_order` last. +5. **Freeze.** `build_expr` builds into a temporary directory named by the + content hash. If that entry exists with a manifest, the build appends the + prompt and returns it. +6. **Lay down** the entry directory: `xorq_build/`, made portable, and `expr.py`. +7. **Execute, with the reproducibility check.** A worthy entry is materialized by + `materialize(project, hash, check_reproducible=True, publish=False)`: two runs + of the frozen build, their digests compared, the first file left complete at + a temporary name. A cheap entry is streamed once in full (`stream_row_count`) + and nothing is kept, so a failing cast surfaces here, in tallyman. +8. **Record** `schema.json`, then the manifest (`write_manifest`). +9. **Publish.** `materialize.publish_snapshot` moves the staged file over the + snapshot path with one `os.replace`, once the entry is complete. +10. **Prompt log.** `_append_prompt` appends to `prompts/.jsonl`. + +**Materialization.** `materialize` loads the frozen build +(`result_cache.load_entry_expr`) and rebinds it onto a fresh single-partition +connection (`single_partition_backend`, batches of 8,192), so a float aggregate +merges in one order on any machine. `_stream_to_parquet` drops any inherited +`__row_order` and ibis's `__row_order_right`, numbers the rows in a new last +`__row_order`, and writes row groups of 1,048,576 rows with the settings of +section 5, to a unique temporary name under the project lock. It returns the +file's digest, read back (`digest.file_digests`). When step 7's runs differ, the +entry builds anyway: the manifest records `reproducible: false` and the columns +that differed, the snapshot is pinned, and the reply warns the author. A heal +runs once and replaces the file at once. + +**After the build**, the tool sets the alias, records an event in `events.jsonl`, +appends a notebook cell for a new alias, carries chart and display config +forward on a revise (`carry_forward_entry_config`), notifies the companion with +`new_entry`, and runs auto-recalc after a revise. The checkpoint comes last, as +the tool returns, and is taken even when nothing changed. + +**What a failure leaves behind.** + +- Before step 6, nothing under `entries/`. Parent snapshots the recipe's reads + re-created stay, as correct cache. +- From step 6 on, the build removes the directory it created and its temporary + file; the file already at the snapshot path is untouched. +- A killed process can leave a directory with no manifest, which every writer + treats as absent and every read refuses (section 7), and a `...tmp` + file in `result_cache/` that nothing removes. Killed between steps 8 and 9, it + leaves a complete entry over whatever file was at the path, or none, which the + next read heals. +- The MCP tools record the failure in `errors.jsonl`, add a `build_error` event, + notify `build_failed` and skip the checkpoint; `PUT /code` answers 400. +- A revision refused for following its own alias is already built, so its entry + stays without an alias and the next checkpoint records it. +- A failed checkpoint is logged; the next one's `git add -A` takes in the change. +- A failed import removes the directory it created; a clone or snapshot it wrote + stays, named by content, for the next import of the same bytes. + +## 7. The read path + +**Loading a build.** `result_cache.load_entry_expr` fills the project path back +into a stable copy of the build, the **expanded build** in +`.xorq_build_expanded/` (`portable.ensure_expanded_build`, gated by a +`.complete` marker), and loads it with `load_expr`. A missing or unloadable build +is an error naming the entry. Nothing falls back to `expr.py`, which would read +the aliases' current heads instead of the entry's parents. + +**The canonical read.** Every consumer inside tallyman's processes (`/api/data` +pages, charts, diffs, post-processing, a recipe's readers) reads a result through +`result_cache.cached_result_expr(project, hash)`. It calls `ensure_materialized`, +then returns, for a worthy entry, one bare read of its snapshot +(`deferred_read_parquet`, memoized in `_snapshot_read`), without loading the +build; this read is also how a worthy parent enters a child's build. For a cheap +entry it returns the loaded graph rebound onto the default backend +(`_resolve_result_plan`, memoized, through `rebind_onto`). Both are +single-backend expressions, so entries compose in joins, unions and diffs. + +**Making files exist.** `materialize.ensure_materialized` puts every file an +entry reads, and its own snapshot, on disk before anything runs: + +1. A worthy entry whose snapshot exists is done, with nothing loaded. +2. A source entry whose snapshot is missing is healed from its clone + (`_heal_a_source`). +3. Otherwise the build is loaded, and each missing file it reads, always another + entry's snapshot, is made by recursing on the hash in its name (`_recreate`). +4. A worthy entry whose snapshot is missing is healed (`_heal`). + +**A directory with no manifest is refused.** The manifest holds what a read +needs: the worthy-or-cheap verdict, the digest a heal is checked against, the +pin, and a source entry's provenance. `result_cache.entry_manifest` reads it and, +when the file is missing, raises `BuildError` with the message +`entry in '' has no manifest.json: `. A hash with no +directory at all gets the same error. `cache_worthy` reads only the manifest, so +a file at the snapshot path says nothing about the verdict. +`cached_result_expr` and `ensure_materialized` raise before they load or write +anything, and a child that reads such an entry raises the parent's error as it +is. Nothing answers around the error: `/api/data`, `/api/entry`, +`/api/entry_cache` and `/api/notebook_full` answer 500, and `/api/session` +answers status `error` with the message. (`/api/data`, `/api/entry` and +`/api/entry_cache` answer 404 for a hash with no directory, which they check +first.) Running the recipe again, or importing the file again for a source +entry, writes the entry again under the same hash. + +A **heal** re-creates a missing snapshot and checks it against `result_digest`. +A computed entry heals by running `materialize` once. A source entry heals from +its clone, which `source_import.rewrite_source_snapshot` parses again with the +reader options in `provenance`. Both take the project lock and look again for the +file first. If a source entry's clone is gone too, the heal fails with an error +naming both files and the `catalog_import_source` call, reader options and +`pinned_version` included, that restores the version. + +**Verification.** `result_cache._verify_self_heal` compares a healed file's +digest with the recorded one. A mismatch is an unfaithful heal: the rows are +served, as the honest output of the frozen build, and the heal logs a warning +that blames an engine upgrade (`_engine_change`), a recipe whose graph hashes +differently when re-run (`recipe_is_structurally_nondeterministic`) or a graph +that runs differently each time; writes the new digest into +`unfaithful_heal_digest`, which pins the file; records an `unfaithful_heal` +error for the page's banner; deletes the entry's Buckaroo statistics; and calls +`UNFAITHFUL_HEAL_HOOKS`, where the companion registers a forced grid reload +(section 11). `catalog_scan_staleness(verify_results=True)` runs the same +comparison over every snapshot on disk (`staleness.verify_sweep`) and writes +nothing. + +**Row order.** Every snapshot ends in `__row_order`, an int64 column holding +`0..N-1` in physical order. `row_order.page` orders a page by any sort keys it is +given and then `__row_order`, which has no ties, so a request returns the same +rows in any process; `/api/data` gives no keys. The build rules of section 4 keep +the column meaningful, and the canonical sort decides the order a worthy entry's +rows are numbered in. ADR-008 gives the reasons. + +**Digest stability.** `result_digest` is `arrow-sha256:` +(`digest.content_digest`), a SHA-256 over the file's Arrow data read back, with +separate streams per column for validity, lengths and values. It ignores the +row-group size, codec, writer version and string encoding, and changes with any +value, null, name, type or row order. Layout matters in one place: an ungrouped +float total depends on the batch boundaries of the file it reads, so the +row-group sizes (1,048,576 computed, 122,880 source) and the batch size are fixed +by `SNAPSHOT_FORMAT_VERSION`, which the manifest records (ADR-009). + +**Memos.** Each process keeps `_resolve_result_plan` (256 loaded builds) and +`_snapshot_read` (1,024 snapshot reads), cleared by +`cached_result_expr.cache_clear()`. File existence is checked on every call, +outside the memos, so they change latency and never rows. + +## 8. Pins and deletion + +A **pinned** snapshot is one the Cache page refuses to delete because it cannot +be made again faithfully. `materialize.pinned_reason` decides from the manifest +that speaks for the file (`snapshot_manifest`: the live entry's, or the copy a +reset parked) and, for a source entry, from whether its clone exists. + +| Pin reason | Read from | +|---|---| +| a source version whose clone is gone (for a retired one, from `bullpen/cas/` too) | `provenance` and the clone's path | +| a recipe whose two runs at create gave different digests | `reproducible: false` | +| a heal that produced different rows than were built | `unfaithful_heal_digest` | + +The pin travels with the manifest through a reset, and dismissing the error +banner does not lift it. It protects the file from the Cache page and nothing +else. + +**The Cache page is the one deleter.** Nothing in tallyman deletes a snapshot on +its own, and nothing writes one speculatively: a snapshot is written when its +entry is created or when a read needs it. +`GET /{project}/api/result_cache` lists every file in `result_cache/` with two +labels. A **retired** file has no live entry, but a reset parked one in the +bullpen, and that manifest decides its pin. An **orphan** has no entry, live or +parked, and is never pinned. `DELETE /{project}/api/result_cache/{hash}` unlinks +a file, answering 400 for a malformed hash, 403 in serve mode, 404 for no +snapshot and 409 with the reason for a pinned one. The next read heals the file; +a grid already open on it fails until the entry is opened again. + +**Clones.** A clone is record, the only copy of the imported bytes, and no read, +heal or import deletes one. A source snapshot is cache while its clone exists +and pinned once it is gone. Importing the same bytes again restores the clone +only when the snapshot is gone as well. `source_identity.gc_cas` is the only code +that removes a clone from `data/.cas/`, and only `reset_to` calls it, moving each +clone that no source entry in `entries.jsonl` names +(`catalog_state._live_source_digests`) to `bullpen/cas/`. If the bullpen already +holds that file, the live copy is removed; if any surviving manifest is +unreadable, the sweep is skipped. + +## 9. Staleness and recalc + +**One axis.** An entry is stale when the alias of one of its followed edges +points at a different hash than the edge recorded, and for no other reason +(`staleness.entry_staleness`). A pinned edge never makes an entry stale, and a +changed data file matters only once it is imported, which moves a source alias. +The scan reads `aliases.jsonl` and the manifests and opens no data file. An edge +whose alias is missing is reported under `unknown_axes`. + +**The scan.** `staleness.scan` judges every entry. Only an alias head can be +**directly stale** (`live: true`); a superseded version reports `live: false`, +since rebuilding it would move no alias. A live entry that is not directly stale +but sits downstream of one that is, is **transitively stale**. +`catalog_scan_staleness` and `GET /{project}/api/staleness` expose the scan and, +with auto-recalc on, add `orphan_stale`: each directly stale entry, classified by +`recalc.classify_orphans` as self-following, explained by a recorded error, or +unexplained. + +**Recalc.** `recalc.recalc(project, roots, dry_run)` rebuilds a **cone**: the +roots and every entry reachable from them through edges, parents first +(`dependents.descendant_cone`, which reports a cycle), with superseded versions +dropped unless named as roots. A dry run, the default, reports planned actions. +A real run (`_replay_cone`) replays each member's recipe through +`build_and_persist`, and before its children replay, moves every alias whose +head was that member to the new hash, so a child's `tracked_expr_from_alias` +reads the new head. A member whose inputs did not move, such as a pinned child, +replays to the same hash and is left alone. The walk stops at the first failure, +keeps the rebuilt prefix, records the failure and the stale entries it skipped in +`errors.jsonl`, and reports `status: "failed"`. A real run that changed something +takes one checkpoint. With no roots, `catalog_recalc` and +`POST /{project}/api/recalc` use every directly stale entry. + +**Auto-recalc.** When a project enables it (`config.auto_recalc_enabled`: the +`TALLYMAN_AUTO_RECALC` variable, then `auto_recalc` in `config.json`, then on), +an operation that moves an existing alias recomputes that alias's followers +within the operation: a revise from either surface, an import that mints a +version, and a diff promotion that re-points an existing alias. +`recalc.auto_recalc` takes as roots the entries this alias made directly stale +(`followers_of`), replays their cone without a checkpoint, and logs every other +directly stale entry, reporting it in `orphan_stale`. The operation's checkpoint +commits the head move and the cascade together, so one reset undoes both, and a +`recalc` event carries the `{old hash: new hash}` remap. + +## 10. Checkpoints and reset + +A **checkpoint** (`catalog_state.checkpoint_catalog`) is one git commit of the +catalog, tagged `step-NNN`, its **step**. Under the project lock it writes +`entries.jsonl` from the entry directories that have a manifest, zips any +without a tracked zip (`catalog.zip_pending_entries`), runs `git add -A`, commits +and tags. `genesis` records `step-000` when a project is created. + +**Reset.** `catalog_state.reset_to(project, ref)` returns the catalog to a step +or label, under the project lock: + +1. `git reset --hard` restores every tracked file, so source aliases rewind with + the rest. +2. `prune_entries` moves each entry directory `entries.jsonl` does not list into + the **bullpen**, `bullpen/entries//`, where a reset parks what it + retires so a later reset forward can bring it back. +3. `restore_from_bullpen` copies back each listed directory that is missing, and + the clones restored source entries name. It copies, so the bullpen keeps its + set and a project can go back and forth repeatedly. +4. `_retire_cas_clones` parks clones no surviving source entry names (section 8). +5. `catalog.assert_catalog_consistent` checks the allow-list, that the recipe + zips match `entries.jsonl`, and that every hash an alias, chart or display + config names has a zip. + +A reset leaves `compute_cache/` alone. A snapshot is named by its entry's hash, +so one left by a retired entry can never be served for another; the Cache page +shows it as retired, and a snapshot missing after a reset is healed as usual. + +**A parked directory replaced by the live one.** When a reset retires an entry +the bullpen already holds, `_retire` replaces the parked copy with the live one. +They can differ (`created_at`, `prompt`, and the `result_digest` of a recipe that +is not reproducible), and the live one matches the snapshot on disk, since a +create always rewrites the snapshot. A live directory without a manifest is +dropped instead. **A reset forward** therefore brings back the manifest that +matches the file, and while an entry is retired the Cache page reads its parked +manifest, counting a parked clone as present. + +After a reset the process clears its memos. The companion's +`POST /{project}/api/reset` also clears `diff_stat_cache/`, reloads the grids and +publishes `project_reset`; `tallyman reset-to ` asks a running companion to +do the same through `/internal/notify`. + +## 11. The viewer + +**Opening an entry.** When an entry's data tab opens, the browser asks +`GET /{project}/api/session/{hash}`, and `BuckarooManager.load_session` +restarts a dead Buckaroo (at most every 30 seconds), calls +`ensure_materialized` so any heal is done and verified first, and posts +`/load_expr`. A worthy entry is handed a **view build**, a build whose whole +graph is one read of its snapshot, written once to `.xorq_view_build/` with a +marker recording the snapshot path (`ensure_view_build`); a cheap entry is handed +its expanded build. The post carries the session id `entry--`, +`project_root` (`artifacts/`), `cache_storage_path` (`.buckaroo_stat_cache/`), +`row_order_column` and, for an entry with a display config, its +`column_config_overrides`. A failure before the post is answered as status +`error` with the reason. + +Tallyman keeps no map of sessions: the id is a function of project and hash, and +every open posts again. +Buckaroo skips the work when it holds that session with the same build directory +and the post carries no configuration, and re-creates a session it dropped. The +notebook page's route, `GET /{project}/api/notebook_full`, opens one for every +cell. + +**Events.** The companion publishes SSE events from its own routes and +republishes what the MCP server and the command line post to +`/internal/notify`. On `recalc`, `project_reset` and every **klass** change (a +statistic, post-processing function or display class written for the project), +it reloads grids with `reload_project_sessions`, which posts +`/reload_expr/` for every entry and treats 404 or 400 as "not open". +The browser refetches on `new_entry`, `build_failed`, `notebook_changed`, +`chart_attached`, `post_processing_changed`, `summary_stat_changed`, `recalc` +and `project_switched`, and follows a `recalc` remap in a background tab. It +ignores `entry_added`, `alias_changed`, `alias_renamed`, `display_changed`, +`project_reset` and `unfaithful_heal`. + +**The forced reload after an unfaithful heal.** The companion's hook calls +`BuckarooManager.force_reload_session`, which posts `/load_expr` for the entry's +session with `force_reload: true`, so Buckaroo recomputes the statistics the heal +deleted instead of showing ones from the old rows. The hook also publishes +`unfaithful_heal`. + +**Diffs.** `GET /{project}/api/diff_data/{alias}/{va}/{vb}` reads two versions +through `cached_result_expr`, finds a join key (`primary_key.diff_keys`, cached in +`primary_key.json`; a search whose queries run longer than one second in total +answers 504, and time spent waiting for the execution lock does not count), and +computes the summaries (`tallyman_xorq.diff.full_diff`). For the grid, +`tallyman_companion.diff.build_compare_expr` builds an outer join on the key, +with `__row_order` dropped from both sides and `membership`, `_eq`, `_pct_delta` +and `_abs_delta` columns, posted as session `diff--`. Diff sessions, +unlike entry sessions, are remembered until Buckaroo restarts. The join is not +materialized, so Buckaroo runs it for every grid query. + +**Promoted diffs.** `catalog_promote_diff` and the companion's promote route +save a diff as an entry whose generated recipe is: + +```python +from tallyman_companion.diff import build_diff_expr +expr = build_diff_expr(a_hash="", b_hash="", keys=[...]) +``` + +It names two content hashes and records no parent edge, because +`build_diff_expr` reads both sides through `cached_result_expr`, not through the +readers that record edges; a promoted diff never goes stale or takes part in a +recalc as a follower. It contains a join, so it is worthy. Its alias is +`diff__v_v` unless the MCP tool is given another; re-pointing an +existing one runs auto-recalc. Both surfaces commit their own checkpoint and +announce `entry_added`. + +## 12. Invariants + +Where the code breaks one of these, section 13 names the issue. + +- **A content hash names one result**: every read, warm or cold, in either + process, returns the rows the entry's build fixed, unless the recipe is itself + nondeterministic, which the digest detects. +- **A recipe names aliases**, never a file or a bare content hash. +- **A build reads only the arena**: every file it reads is a snapshot named by + its entry's content hash. +- **Names resolve once**, when an entry is minted, and no read, heal or grid + hand-off resolves a name to decide rows. +- **The manifest completes an entry**: it is the last write, atomic, and a + directory without one is not an entry. Writers build over it, and every read + refuses it; nothing stands in for the manifest's verdict. +- **A snapshot changes only by an atomic replace of a complete file**, under the + project lock. +- **Two routines write result bytes**, `materialize` and the import's + `_write_snapshot`, with the same parquet settings. +- **Every snapshot ends in `__row_order`**, and every `/api/data` page is ordered + by it. +- **A re-created snapshot is verified before it is served**, and a mismatch pins + it. +- **Only the user deletes a snapshot**, through the Cache page, which refuses a + pinned one. +- **No clone's bytes are deleted**: a reset parks a clone no surviving entry + names in the bullpen. +- **An alias's kind matches its entries' kind**, and a source alias moves only by + an import. +- **A source alias's history only grows**, and no two of its versions hold the + same bytes read the same way. +- **An import never shifts a zoned CSV time**: offset-less text keeps its + wall-clock time in the column's zone, and a wall-clock time the zone skips or + repeats is an error. +- **Staleness has one axis**: a followed alias that points somewhere new. +- **One operation, one checkpoint**: a head move and its cascade commit together. +- **One write at a time per project**: builds, imports, heals, checkpoints and + resets hold the project lock. +- **One execution at a time per process** on its shared default backend, under + the execution lock, and a thread that holds it takes no new project lock. +- **One server per data dir**: `tallyman run` holds `server.lock` while it + serves, and a client of a data dir reaches only that data dir's companion. + +## 13. Known defects + +The open issues that touch this design, as of `63bcdd6`. The full list, including +the viewer, export and hint issues, with a priority for each, is +[`plans/open-bugs-2026-09-24.md`](../plans/open-bugs-2026-09-24.md), made at +`1f8cb02`. Since then four were fixed on this branch, and the sections above +describe the fixed behaviour: #118 by #242 (the execution lock, section 2), #183 +by #241 (one server per data dir, section 2), #204 by #245 (a directory with no +manifest is refused, section 7) and #231 by #244 (zoned CSV times, section 5). + +Wrong rows, pins or edges without an error: + +- #229: a pinned child and a tracked child with the same query compile to one content hash, so they are one entry and the first build's edge wins; a pinned alias can then move when its parent is revised. +- #228: `build._raw_parquet_read_check` allows `xo.deferred_read_parquet` of any file under `compute_cache/`, so a recipe can read a snapshot by its path: a bare content hash, with no parent edge and no `ensure_materialized` first. +- #232: an unfaithful heal of a source entry is attributed to a graph that runs differently each time ("or source drift under off"), though the likely cause is a reader change; a corrupt clone is healed from and pinned. +- #205: the canonical sort leaves nested columns out of its tie-break, so rows tied on every sortable column can come out in either order. +- #206: the snapshot writer drops any column named `__row_order_right`, including one the author made. +- #208: an unfaithful heal of a worthy parent changes its cheap children's rows under their hashes, and only the parent is flagged and reloaded. +- #209: an expanded build's marker does not record the project path, so a copied project keeps reading the old location; the marker is also trusted when the expanded folder is gone. +- #200: `full_diff` keeps `__row_order` as a data column, so one inserted row shows every later row as changed. +- #142: the CSV reader null-fills a short row instead of raising. + +Import: + +- #224: a parquet file with a `fixed_size_binary` or UUID column fails at import with a long traceback that names no column. +- #225: a failed import leaves its clone in `data/.cas/` and its snapshot in `result_cache/`, and nothing deletes the clone. +- #227: `catalog_import_source` lets a CSV parse error, or a clone that fails its digest check, escape as a plain `ValueError`; nothing reaches `errors.jsonl`, and the message names the clone and `tallyman_read_csv`. +- #234: the error for an older version's bytes names reset commands that do not exist. +- #239: importing a version's bytes again restores a lost clone only when the snapshot is gone too, so a version whose clone alone is lost stays pinned. +- #237: `catalog_list` does not report an alias's kind. + +Writes, the lock and processes: + +- #240: alias, notebook and `config.json` writes load, change and replace the file without the project lock, so two overlapping writers lose one change. +- #233: a recipe's alias reads take the project from the `active_project` file, while the MCP tool builds into its session's project and a companion route into the project in its URL. +- #186: the project lock blocks with no timeout, so a page read that needs a heal waits behind any build in the other process. +- #190: `PUT /code` and `POST /promote_diff` build on the companion's event loop, so the whole UI stops answering while they wait for the lock or build. +- #226: a writer killed mid-write leaves a `.tmp` in `result_cache/` or `data/.cas/` that nothing lists or deletes. +- #230: every failed build leaves its temporary `tallyman_expr_.py` in the OS temp directory. + +Recalc and row order: + +- #238: `catalog_recalc` with a source entry as a root fails, because the replay runs the generated `read_project_file` outside the import. +- #199: the three-way join check also refuses chains of semi and anti joins, which add no right-hand columns and cannot collide. +- #185: whether a recipe is pure is neither recorded nor passed on, so an entry built on a non-reproducible parent is recorded as reproducible, and `today()` passes the check at create. +- #187: an ungrouped float `SUM` or `AVG` depends on the row-group layout of the file it reads, which only the snapshot format version holds fixed. + +The viewer: + +- #170: Buckaroo is pointed at `artifacts/`, so it never finds the project's statistics and post-processing functions under `artifacts/catalog/`. +- #172: diff sessions are keyed by the two hashes with no project, so two projects holding the same source versions can share one. +- #188: the live diff grid hands Buckaroo an unmaterialized join, which Buckaroo runs for every query. +- #201: a klass reload posts `/reload_expr` once per catalog entry, one after another, from the event loop. +- #202: every grid open posts `/load_expr`, so two opens at once both load, and a promoted diff runs Buckaroo's statistics again on every open. +- #203: an unfaithful heal runs its checks and the forced Buckaroo reload while holding the project lock, and the reload opens a session nobody has open. +- #210: the plan memo keeps every loaded build and its backend objects alive, up to 256 of them. +- #235: the SPA has no listener for `project_reset`, `unfaithful_heal`, `entry_added` or `alias_changed`, so an open page goes stale. +- buckaroo-data/buckaroo#974: Buckaroo ignores `row_order_column`, so grid pages are not ordered by `__row_order`. + +Code and text left behind by ADR-011: #236. diff --git a/docs/architecture.md b/docs/architecture.md index 587b408..6187134 100644 --- a/docs/architecture.md +++ b/docs/architecture.md @@ -1,479 +1,1013 @@ # Tallyman architecture overview -This is the top-level map of the tallyman codebase. Read it first, then follow -the cross-references into the per-subsystem docs. For a document index with a -currency note on each file, see [Related documentation](#related-documentation) -at the end. +This is the top-level map of tallyman: how the system works from end to end, +and where each part is documented in more depth. Read it first. The +[Related documentation](#related-documentation) section at the end lists every +doc, with a note on how current each one is. ## What tallyman is -Tallyman is a deconstructed notebook platform. Instead of a notebook file with -inline cells and outputs, the unit of work is a **catalog entry**: a single -xorq expression, compiled and stored on disk under a content hash. Claude Code -is the author — it drives tallyman through an MCP server, creating and revising -entries as xorq expressions. The entries accumulate in an on-disk, -content-addressed, git-backed catalog. A FastAPI companion server plus a React -SPA visualize that catalog in a browser, and a Buckaroo subprocess provides -interactive data grids over each entry's result. +Tallyman is a notebook without cells. Its unit of work is a **catalog entry**: +one xorq expression (a deferred dataframe computation, which runs only when +asked), compiled and stored on disk under its **content hash**. Claude Code is +the author. It drives tallyman through an MCP server, sending Python that +builds an expression, and each successful call becomes an entry in a +git-backed catalog on disk. A FastAPI companion server and a React single-page +app (SPA) show the catalog in a browser, and a Buckaroo subprocess draws the +interactive data grid for each entry. -Nothing in tallyman is a long-lived application server holding state in memory. -The catalog on disk is the source of truth. The running processes (companion, -Buckaroo, MCP server) are views and editors over it; the MCP server holds no -in-memory catalog state, and the companion holds only its SSE subscriber list. +The catalog on disk is the source of truth, and the running processes are +views of it and editors of it. Each process keeps in-memory caches keyed by +content hash, which change how fast an answer comes back and never what the +answer is. Beyond those caches, the MCP server remembers which project its +session is working on, and the companion holds its open SSE streams and the +diff sessions it has opened in Buckaroo. So one server runs per **data dir**, +the directory that holds every project (`~/.tallyman-notebooks` unless +`TALLYMAN_HOME` names another): `tallyman run` holds an exclusive lock on +`/server.lock` while it serves, and a second `tallyman run` on the same +data dir is refused with a message naming the one that holds it. See +[One server per data dir](#one-server-per-data-dir). + +### Terms + +These are the project's own terms. The other docs use them with the same +meaning. + +- **Catalog:** a project's entries, aliases and notebook, kept in a git + repository at `/artifacts/catalog/`. +- **Entry:** one catalog computation, stored in `entries//` and + committed as `entries/.zip`. +- **Content hash:** an entry's identity, xorq's 12-character hash of the + entry's expression after tallyman's rewrite. Every file the expression reads + is a snapshot named by the content hash of the entry it holds, so the hash + covers the inputs as well as the structure. A source entry (below) is the + exception that makes this work: its hash is an md5 of the imported bytes and + the reader options, truncated to the same 12 characters. +- **Recipe:** the Python the author submitted, kept as the entry's `expr.py`. + It names its inputs by alias, so run again later it can mean something else. +- **Build:** the entry's `xorq_build/` directory, the expression frozen to disk + with every input fixed. A read of a cheap entry and every heal of a computed + entry load the build (a source entry heals from its clone); + a worthy entry whose snapshot exists is read from the snapshot alone. The + recipe is run again only to make a new entry (a revise or a recalc), and by + one diagnostic after a heal that went wrong. +- **Manifest:** the entry's `manifest.json`, which records what the build does + not say: parent hashes, the cheap-or-worthy verdict, the result digest, and + for a source entry where its data came from. +- **Alias:** a mutable name, such as `sales`, that points at the latest content + hash of a logical entry and keeps every hash it has pointed at, as versions + V1, V2 and so on. An alias has a kind: a **catalog alias** names + computations, and a **source alias** names versions of an imported file. +- **Import:** `catalog_import_source`, the one way a file enters the catalog. + It copies the file's bytes into the project, writes one snapshot of them and + points a source alias at the new entry. A recipe never opens a file. +- **Source entry:** one version of a source alias. It is an ordinary entry + whose manifest also records `provenance`: the path the file was imported + from, its digest, the reader options, and the name it was imported as. The + path is never read again. +- **Clone:** the imported bytes, kept under `data/.cas/`, + named by their md5 digest and checked against it when written. A source + entry's snapshot is made again from its clone. +- **`__row_order`:** an int64 column holding `0..N-1` in a file's physical row + order. It is the last column of every file tallyman writes, and pages of an + entry are sorted by it. +- **Worthy and cheap entries:** a **worthy** entry is one tallyman + materializes, because its plan does work that is expensive or that cannot + keep its input's row order (an aggregate, a join, a sort, a window function, + among others). A **cheap** entry is row-preserving over one file (filters, + column selections, computed columns) and has no file of its own; its small + plan re-runs on every read. +- **Snapshot:** the parquet file that holds a worthy entry's result, + `compute_cache/result_cache/.parquet`. A source entry is worthy, + and its snapshot holds the imported rows in the file's order. +- **Materialize:** run an entry's build to completion and write the result to + its snapshot. +- **Heal:** make a missing snapshot again (a source entry's from its clone, + any other by running the entry's build) and check it against what was + recorded. +- **Result digest:** a content digest of a snapshot, recorded in the manifest + when the snapshot is first written; every heal is checked against it. +- **Pinned snapshot:** one the Cache page refuses to delete, because it cannot + be made again faithfully. +- **Checkpoint:** one git commit of the catalog repository, tagged `step-NNN`. + Each operation that changes the catalog lands as one checkpoint. +- **Reset:** `reset_to`, which returns the catalog to an earlier checkpoint. +- **Bullpen:** the directory a reset moves retired entry directories and + clones into, so that a later reset forward can bring them back. +- **Project lock:** a file lock on the project that builds, materializations, + checkpoints and resets take, so that they happen one at a time. +- **Session:** one grid's state inside the Buckaroo process. +- **View build:** a build whose whole graph is one read of a worthy entry's + snapshot. It is what Buckaroo is handed for a worthy entry. +- **Klass:** a summary statistic, post-processing function or display class + written for the project, for Buckaroo's grids to use. ### Big-picture flow ``` Claude Code - │ (MCP tool calls over stdio) + │ MCP tool calls over stdio ▼ - tallyman MCP server ──────────────► on-disk catalog - (compiles xorq exprs, ~/.tallyman-notebooks/projects// - checkpoints to git) (content-addressed entries, aliases, + tallyman MCP server ──────────────► on-disk project + (builds entries, ~/.tallyman-notebooks/projects// + checkpoints to git) (entries, snapshots, aliases, │ notebook, git history) │ best-effort HTTP notify ▲ - ▼ │ reads/writes + ▼ │ reads and writes tallyman companion (FastAPI :7860) ────────┘ │ REST + SSE ▼ React SPA (packages/app) in the browser │ embeds grids ▼ - Buckaroo subprocess (:8700) ◄─── companion POSTs build dirs - (interactive dataframe grids, sort/search push down to xorq) + Buckaroo subprocess (:8700) ◄─── companion posts a build to display + (grids: paging, sorting, search, summary statistics) ``` -The typical loop: Claude Code calls an MCP tool to create or revise an entry, -the MCP server compiles the xorq expression and writes a content-addressed -entry plus a git checkpoint, then notifies the companion. The companion emits a -Server-Sent Event (SSE, a push message over an HTTP stream the browser holds -open; see [Live updates over SSE](#live-updates-over-sse)), the SPA refetches -the affected data, and when the user opens -an entry the companion warms a Buckaroo session so the grid loads. See -[expression-lifecycle.md](expression-lifecycle.md) for the full -create-to-view path. +The typical loop: Claude Code calls an MCP tool to create or revise an entry. +The MCP server imports the code and builds the entry. It checks the +expression, decides whether the entry is worthy, freezes it, runs its query to +completion (writing the snapshot of a worthy entry), and writes the entry +directory. It then notifies the companion, best effort, and commits a +checkpoint as the tool returns. The companion publishes a Server-Sent Event +(SSE, a message on an HTTP stream the browser keeps open; see +[Live updates over SSE](#live-updates-over-sse)), and the SPA refetches what +changed. When the user opens the entry, the companion first makes sure every +file the entry reads exists, then asks Buckaroo to open a session for it, and +the grid connects to that session over a WebSocket. +[expression-lifecycle.md](expression-lifecycle.md) follows one entry through +all of this. -Three processes run on one machine: `tallyman_mcp` (spawned by Claude Code over -stdio), `tallyman_companion` (the FastAPI app, started with `tallyman run`), and -a `buckaroo` server subprocess that the CLI/companion supervises. +Three processes run on one machine: `tallyman_mcp`, which Claude Code spawns +over stdio (one per Claude Code session); `tallyman_companion`, the FastAPI app +that `tallyman run` starts; and the Buckaroo server, which `tallyman run` +spawns and stops. The MCP server and the companion both write to the catalog. +The project lock makes their builds, materializations, checkpoints and resets +happen one at a time; smaller writes (an alias, a notebook cell, a chart) are +atomic file replacements that take no lock. ## Component map -Tallyman is six subsystems: five Python packages under `src/` and one frontend -workspace under `packages/`. The dependency direction runs core ← xorq ← {mcp, -companion} ← cli; nothing in `tallyman_core` imports the web, MCP, or compute -layers. - -**tallyman_core** (`src/tallyman_core/`) is the native catalog model. It owns -the on-disk representation: versioned entries keyed by content hash, mutable -aliases with version history, the notebook cell list, chart specs, display -configs, project-global post-processing and summary-stat functions, and the -prompt/error/event logs. It manages the git-backed checkpoint transaction -(capture pointers, zip recipes, stage, commit, tag) and the reset-to-revision -operation that rewinds the catalog and reconciles untracked entry directories -and source clones through a holding area called the bullpen. A key invariant lives here: -`assert_catalog_consistent` enforces an allow-listed tracked surface and -verifies that every hash referenced by an alias, chart, or display config has a -durable recipe zip. The call direction is one-way: `catalog_state` calls -`catalog`, never the reverse. Design: [native-catalog-store.md](../plans/native-catalog-store.md). - -**tallyman_xorq** (`src/tallyman_xorq/`) is the xorq integration layer. It -compiles a xorq expression into a content-addressed entry, computes the content -hash from the expression structure (which covers each source's content, because -a recipe reads an ordered copy named by the source's digest), decides once -whether the entry is worthy of a materialized result file, writes that file -itself (`materialize`), and writes a portable build directory whose absolute -paths are rewritten to `${TALLYMAN_PROJECT_ROOT}` placeholders. It also implements -reactive staleness detection (comparing recorded manifest fields against the -current world) and the recalc cone that recomputes dependents in dependency -order. Reconstruction of an entry's expression from its persisted `expr.py` -happens here, using context variables that pin the entry's recorded source -digests. See [caching.md](caching.md) and [reactive-recalc.md](reactive-recalc.md). - -**tallyman_companion** (`src/tallyman_companion/`) is the FastAPI web server on -port 7860. It surfaces catalog views over REST, pushes live updates over SSE, -manages the Buckaroo subprocess lifecycle, and handles browser-initiated -mutations (code revision, diff promotion, resets). There are no server-side -HTML templates — it serves the compiled React SPA (`packages/app/dist`) as a -catch-all and mounts `/assets` and `/static`. A checkpoint middleware wraps -mutating routes so each authored change lands as one git revision, with an -opt-out denylist for routes that self-checkpoint or should not checkpoint at -all. It builds diff expressions (outer join with membership and per-column -delta/equality sentinel columns) and computes the Buckaroo column-config -overrides that color them. No dedicated subsystem doc yet — the route table is -in `app.py`; the create-to-view path is in [expression-lifecycle.md](expression-lifecycle.md). - -**tallyman_mcp** (`src/tallyman_mcp/`) is the FastMCP server that Claude Code -talks to over stdio. It exposes the catalog, notebook, and project tools -(`catalog_run`, `catalog_create`, `catalog_revise`, `catalog_alias`, -`catalog_diff`, `catalog_recalc`, the chart/display/stat/post-processing tools, -`notebook_*`, and `project_*`). Checkpointing is opt-out: every tool -auto-checkpoints at the dispatch boundary unless it is on the no-checkpoint -list. The active project is session-sticky (seeded on first tool call, surviving -disk changes within the session), and project lifecycle changes are POSTed to -the companion rather than written directly so SSE stays honest. Notifications to -the companion are best-effort and never raise. Every tool, its parameters, and -its side effects (checkpoint, SSE notify, auto-recalc) are documented in -[mcp-server.md](mcp-server.md). - -**tallyman_cli** (`src/tallyman_cli/`) is the Click command-line interface -(entry point `tallyman`). It initializes projects (with synthetic fixture data), -runs the companion and the MCP service, and owns the Buckaroo subprocess via -`BuckarooManager`, which spawns `python -m buckaroo.server` on port 8700 (or a -random free port) and exits when its stdin closes. It also provides `serve` -(read-only companion against a project directory anywhere on disk), `pack` -(portable tarball of the project directory), `reset-to` / `revisions`, -and storyboard `replay` for deterministic rehearsal. See [installing.md](installing.md). - -**Frontend** (`packages/app/`) is the React 18 + Vite SPA. It builds to `dist/` -and is served by FastAPI as a catch-all; it drives refetches off an SSE version -counter rather than polling, defers grid loads until scroll via -`LazyBuckarooEmbed`, and remaps views to new hashes when a recalc event arrives. -Its `BuckarooEmbed` component mounts `BuckarooServerView` from `buckaroo-js-core` -directly and connects over WebSocket. No dedicated frontend doc yet. - -## On-disk catalog layout - -A project lives at `~/.tallyman-notebooks/projects//`. The home root is -`~/.tallyman-notebooks/` by default and is overridable with the `TALLYMAN_HOME` -environment variable (`paths.py:tallyman_home`). The single active project name -is recorded in `~/.tallyman-notebooks/active_project` (one line). The catalog -itself is a git repository at `/artifacts/catalog/`. +Tallyman is six parts: five Python packages under `src/` and one frontend +workspace under `packages/`. Imports mostly run one way, core ← xorq ← +{companion, mcp} ← cli. The exceptions: `tallyman_core` imports `tallyman_xorq` +lazily inside three functions (`reset_to` clears the read memo and retires +clones, and `run_post_processing` reads an entry's result), the MCP server +imports the companion's diff helpers for `catalog_promote_diff`, and the +companion imports the CLI's fixture writer for `/api/projects/new`. + +**tallyman_core** (`src/tallyman_core/`) is the catalog model and store. It +owns the on-disk representation: entries keyed by content hash, aliases and +their version history, the notebook's cell list, chart specs, display configs, +the project's post-processing and summary-stat functions, per-project settings +(`config.json`), and the prompt, error and event logs. It runs the checkpoint +(record the entry pointers, zip new recipes, `git add -A`, commit once, tag the +step) and `reset_to`, which rewinds the catalog to a step and reconciles the +files git does not track through the bullpen. It holds the project lock +(`catalog_state.project_lock`). `assert_catalog_consistent` enforces an +allow-listed set of tracked paths and checks that every hash an alias, chart or +display config names has a committed recipe zip. The call direction is one +way: `catalog_state` calls `catalog`, never the reverse. Design: +[native-catalog-store.md](../plans/native-catalog-store.md). + +**tallyman_xorq** (`src/tallyman_xorq/`) turns recipes into entries and serves +their results. `build.py` imports a recipe and builds the entry. `io.py` holds +what recipes read other entries with (`tracked_expr_from_alias`, +`pinned_expr_from_alias`) and the refusals of a raw file read. +`source_import.py` imports a file as a source entry: `source_identity.py` keeps +the clones, and `ordered_copy.py` holds the reader options a source entry +records and the row-group size of its snapshot. +`worthiness.py` decides cheap or worthy. `source_cache.py` and `row_order.py` +check and rewrite the expression before it is frozen. `materialize.py` writes +snapshots and makes sure the files an entry reads exist. `digest.py` computes +content digests. `result_cache.py` is the one read of an entry's result. +`portable.py` makes builds relocatable. `staleness.py`, `dependents.py` and +`recalc.py` are the reactive system, and `primary_key.py` and `diff.py` serve +version diffs. See [caching.md](caching.md) and +[reactive-recalc.md](reactive-recalc.md). + +**tallyman_companion** (`src/tallyman_companion/`) is the FastAPI web server +on port 7860. It serves the catalog over REST, pushes live updates over SSE, +handles edits made in the browser (a code revision, a diff promotion, notebook +edits, a recalc, deleting a snapshot, project switching), and talks to Buckaroo +through `BuckarooManager` (`buckaroo_lifecycle.py`), which spawns the +subprocess, opens sessions and reloads them when klasses change. It has no +server-side HTML templates: it serves the built SPA (`packages/app/dist`) for +every GET that no route matches (an unmatched `/api` path gets a JSON 404), and +mounts the SPA's `/assets`. A checkpoint middleware commits one git revision +after each successful mutating request, except on an explicit list of exempt +routes (`/internal/*` and the project routes, cache and log clears, telemetry, +reset, and the routes that checkpoint themselves). `diff.py` builds the diff +comparison, an outer join with a membership column and per-column delta and +equality columns, and the Buckaroo column settings that colour it. No +dedicated doc covers the route table yet; it is in `app.py`, and the +create-to-view path is in [expression-lifecycle.md](expression-lifecycle.md). + +**tallyman_mcp** (`src/tallyman_mcp/server.py`) is the FastMCP server that +Claude Code talks to over stdio: 31 tools and one prompt. Every tool +checkpoints after it succeeds unless it is on the `_NO_CHECKPOINT` list. The +active project is sticky for the session: seeded on the first call, then +changed only by `project_switch` or `project_new`, which go through the +companion so that its SSE stream stays honest. It finds the companion on each +call from the port recorded in the data dir's `server.lock`; with no server on +the data dir it sends no notifications, its replies carry no entry links, and +`project_switch` and `project_new` return an error. Notifications to the +companion are best effort and never raise. [mcp-server.md](mcp-server.md) +documents every tool, its parameters and its side effects. + +**tallyman_cli** (`src/tallyman_cli/main.py`) is the Click command line, +`tallyman`. `init` creates a project, with a synthetic `data/orders.parquet` +unless given `--no-fixture` (written, not imported: a recipe reads it only after +`catalog_import_source`), and records its step-000 checkpoint. `run` claims the +data dir, refusing to start when another server holds it, then starts the +companion and, unless given `--no-buckaroo`, the Buckaroo subprocess +(`python -m buckaroo.server --stdio-control`, which exits when its stdin +closes) on port 8700, or on a random port if 8700 is taken. `mcp` starts the +MCP server. `serve` runs a read-only companion, without Buckaroo, against a +project directory anywhere on disk. `pack` tars a project directory. `reset-to`, +`revisions` and `revisions label` move through and name checkpoints. `replay` +runs a storyboard of MCP tool calls. See [installing.md](installing.md). + +**Frontend** (`packages/app/`) is a React 18 and Vite SPA with pages for the +catalog, the notebook, diffs, the Cache page, the activity log and the project +list. It refetches when an SSE event arrives instead of polling. The catalog +page's data tab asks for the entry's Buckaroo session as soon as the entry +opens, and shows a spinner, then the grid, or the reason it failed with a retry +button (#133). The notebook page's data route opens a Buckaroo session for every +cell each time the page loads (#202), and each cell's grid connects only when +the cell scrolls near the viewport (`LazyBuckarooEmbed`). The grid itself is +`BuckarooServerView` from `buckaroo-js-core`, connected to the Buckaroo process +over a WebSocket. No dedicated frontend doc yet. + +## On-disk layout + +A project lives at `~/.tallyman-notebooks/projects//`. The home root +is `~/.tallyman-notebooks/` by default and can be moved with the +`TALLYMAN_HOME` environment variable (`paths.tallyman_home`). The active +project's name is the one line of `~/.tallyman-notebooks/active_project`. +`server.lock` beside it is the claim of the `tallyman run` serving the data dir, +and its owner record ([One server per data dir](#one-server-per-data-dir)). ``` ~/.tallyman-notebooks/ - active_project # one line: the active project name + active_project # one line: the active project's name + server.lock # held by the running tallyman run; its pid, port and start time projects// artifacts/ - catalog/ # git repo — the tracked catalog - entries/.zip # tracked recipe zip (expr.py, schema.json, xorq_build/) - entries// # untracked build dir (gitignored, reconciled on reset) - entries.jsonl # pointer list, one {hash} per line - aliases.jsonl # one {alias, latest, history:[V1,V2,...]} per line - notebook.jsonl # one {cell_id, alias, markdown} per cell - config.json # project settings, e.g. {auto_recalc: bool} - chart_specs/.vl.json # Vega-Lite specs, keyed by content hash - display_configs/.json # {column_config_overrides, diff_provenance} - post_processing/.py # process(expr) functions (_disabled/ = soft-deleted) - stats/.py # compute(col) functions (_disabled/ = soft-deleted) - prompts/.jsonl # append-only per-entry prompt history - bullpen/ # untracked holding area (entries/, cas/) a reset retires files to - compute_cache/ # untracked cache files tallyman writes: - result_cache/.parquet # snapshots of worthy entries - ordered_sources/.parquet # ordered copies of sources - diff_stat_cache/ # untracked Buckaroo diff-stat caches per entry pair - .gitignore # deny-by-default (entries/*/, bullpen/, caches, *.tmp) - display/.py # display klass files (ColAnalysis subclasses) - errors.jsonl # append-only error log (outside catalog repo) - events.jsonl # append-only activity log (outside catalog repo) - exports/ # marimo .py, screenshots, CSVs - data/ # user input parquets (fixtures) - data/.cas/ # content-addressed source clones (CoW), cas mode only + catalog/ # git repo: the catalog + entries/.zip # tracked recipe zip (expr.py, schema.json, manifest.json, xorq_build/) + entries// # untracked entry directory (below) + entries.jsonl # the entries a checkpoint recorded, one {hash} per line + aliases.jsonl # one {alias, latest, history, kind} per line + notebook.jsonl # one {cell_id, alias, markdown} per cell + config.json # project settings, e.g. {"auto_recalc": true} + chart_specs/.vl.json # Vega-Lite specs, by content hash + display_configs/.json # {column_config_overrides, diff_provenance} + post_processing/.py # process(expr) functions (_disabled/ holds removed ones) + stats/.py # compute(col) functions (_disabled/ holds removed ones) + prompts/.jsonl # the prompts each entry was built from + .gitignore # keeps the untracked paths below out of git add -A + .checkpoint.lock # the project lock (untracked) + compute_cache/ # untracked; files tallyman can make again + result_cache/.parquet # snapshots of worthy entries (source entries too) + bullpen/ # untracked; entries/ and cas/ that a reset retired + diff_stat_cache/-/ # untracked; Buckaroo statistics per diffed pair + display/.py # display klasses (outside the catalog repo) + errors.jsonl # error log (outside the catalog repo) + events.jsonl # activity log (outside the catalog repo) + telemetry.jsonl # Buckaroo grid-load timings (outside the catalog repo) + exports/ + data/ # tallyman init's fixture (an import takes any path) + data/.cas/ # clones: the bytes of every imported file + buckaroo.log # the Buckaroo subprocess's stderr (tallyman run) + notebook_marimo.py # written by catalog_export_marimo ``` -Key formats and what's tracked vs untracked: - -- **Recipe zip** (`entries/.zip`) is the durable, deterministic, - content-addressed entry. It contains `expr.py` (the author's literal source, - with portable placeholders), `schema.json` (field names and types), and - `xorq_build/` (the portable expression directory with `expr.yaml` plus deps). - The zip writer only runs inside a checkpoint. -- **Entry build dir** (`entries//`) is ephemeral and gitignored. It holds - `manifest.json` (the build-completeness sentinel), the load-time-expanded - `.xorq_build_expanded/`, the `.xorq_view_build/` handed to Buckaroo for a - worthy entry, and per-entry caches (`.buckaroo_stat_cache/`). A - directory without a `manifest.json` is treated as partial/crashed and is not - zipped. -- **Manifest** (`manifest.json`) carries entry metadata: `content_hash`, - `result_digest`, `sources` (`{rel_path: digest}`), `ordered_copies` (how each - ordered copy of a source was made), `parents` (DAG edges), `row_count`, - `compile_seconds`, `execute_seconds`, `cache_worthy` and `cache_worthy_why`, - `cache_bytes`, `reproducible`, `snapshot_format` and `engine_versions`. - Written atomically (temp file + `os.replace`). -- **JSONL pointer file** (`entries.jsonl`) and the structured logs - (`prompts/`, `errors.jsonl`, `events.jsonl`) are append-oriented and - line-delimited. They replaced the single `catalog.yaml` that older docs - reference. -- **Activity logs** (`events.jsonl`, `errors.jsonl`) live in `artifacts/`, - outside the catalog git repo, so they survive reset-to-revision and have no - size cap. The UI filters them on read. -- **No per-entry `result.parquet`.** That layer was removed (#104). A worthy - entry's rows live in its snapshot, `compute_cache/result_cache/.parquet`, - which tallyman writes when the entry is created; a cheap entry keeps no copy - and recomputes on read. - -All catalog writers use atomic writes, so a crash mid-write or a checkpoint -firing during a write window leaves a whole file, not a torn one. +An entry directory, `entries//`, holds the four recipe members +(`expr.py`, `schema.json`, `manifest.json`, `xorq_build/`) and the per-entry +caches, which are made again on demand: `.xorq_build_expanded/` (the build with +the project path filled back in), `.xorq_view_build/` (a worthy entry's view +build), `.buckaroo_stat_cache/` (Buckaroo's summary statistics) and +`primary_key.json` (the key a diff joins on). `.xorq_build_expanded/` and +`.xorq_view_build/` each have a sibling `.complete` marker, written last. + +Key formats, and what is tracked: + +- **Recipe zip** (`entries/.zip`) is the committed record of an entry. + Nothing reads it back, so a clone of the catalog repository does not recreate + entry directories from it. The checkpoint writes it, deterministically, once + per entry, and nothing else does, so it keeps the manifest as it was at + create. +- **Manifest** (`manifest.json`) records `content_hash`, `project`, + `created_at`, `prompt`, `row_count`, `execute_seconds`, `compile_seconds`, + `cache_worthy` and `cache_worthy_why` (the cheap-or-worthy verdict and its + reason), `cache_bytes` (the snapshot's size), `result_digest`, `reproducible` + and `nonreproducible_columns`, `snapshot_format` and `engine_versions`, + `parents` (`[{hash, ref, follow}]`), `unfaithful_heal_digest` (set by an + unfaithful heal, and a pin) and, on a source entry only, `provenance` + (`{alias, version, path, digest, suffix, reader, imported_at}`, where `alias` + and `version` are the name it was imported as). It is the entry directory's + last write, atomic, and its presence means the entry is complete: the entry + list, the checkpoint, recalc and the build skip or rebuild a directory that + has none. Every read refuses such a directory: `result_cache.entry_manifest` + raises a `BuildError` naming the missing `manifest.json` before anything is + loaded or written, the companion's entry routes answer 500, and running the + recipe again writes the entry again. After create only an unfaithful heal + rewrites the manifest, to record `unfaithful_heal_digest`, atomically, under + the project lock. +- **`compute_cache/`** holds the files tallyman writes and can make again: + snapshots, a source entry's included. It is untracked, a reset leaves it + alone, and anything may delete it: the next read makes what it needs again. + The exceptions are the pinned snapshots, which cannot be made again; one of + them is a source entry's snapshot once its clone is gone. +- **Logs** (`errors.jsonl`, `events.jsonl`, `telemetry.jsonl`) live in + `artifacts/`, outside the catalog repository, so a reset does not rewind + them and a recorded failure survives it. Dismissing the error banner deletes + `errors.jsonl`, which holds no pin (pins are decided from the manifest), and the + readers of `errors.jsonl` skip a line that is not a JSON object, so one torn + append does not break the banner or the error page. +- **Display klasses** (`artifacts/display/`) are also outside the catalog + repository, so no checkpoint commits them and no reset rewinds them. + Summary stats and post-processing functions are inside it. +- **No per-entry `result.parquet`.** That layer was removed in #104; the + companion still sweeps any old one away once per project at startup. + +Tracked catalog files are written to a temporary file and renamed into place, +so a checkpoint that fires during a write commits a whole file. +`entries.jsonl` is written only inside the checkpoint, under the project lock. ## Core domain concepts -**Content hash (identity).** Every entry is identified by a hash derived from -its expression structure (xorq's tokenization) and, depending on the -source-identity mode, its source file digests. Identity is structural: two -entries with the same expression and inputs collapse to the same hash, which is -what makes builds idempotent. Because hashes are content-addressed and globally -unique by construction, a Buckaroo session's id is derived from the project and -the content hash (`entry--`), and tallyman keeps no record of -sessions. The source-identity mode (`off` / `cas` / `salt`, default -`cas`) is decided in [ADR-002-source-identity-content-hash.md](../plans/ADR-002-source-identity-content-hash.md). - -**Result digest.** A second identity axis, recorded for *worthy* (materialized) -entries only. The content hash keys the expression graph; the `result_digest` -keys the executed *result*. It is `arrow-sha256:`, a SHA-256 over the -Arrow data of the entry's snapshot parquet read back (`snapshot_file_digest`), -so it does not depend on how the writer batched or grouped the rows, on the -codec, or on the writer's version. It does depend on every value and on the -order of the rows: the snapshot is written in a canonical total order (the -author's sort keys, then `__row_order`, then the remaining sortable columns) on -a single-partition connection, so the same build gives the same digest run to -run. Cheap, row-preserving entries record no digest: they have no snapshot to -digest and recompute live. A mismatch when a deleted snapshot is re-created -points at execution nondeterminism (sampling, `now()`, an impure UDF, source -drift) or at an engine change, which the manifest's `engine_versions` lets the -error record say. A worthy entry's query is also run twice when it is created, -so a recipe that is not reproducible is known from the start. Design: -[ADR-004-result-digest-canonical-ordering.md](../plans/ADR-004-result-digest-canonical-ordering.md) -and [ADR-009-digest-stability.md](../plans/ADR-009-digest-stability.md). - -**Alias and V_n versions.** An alias is a named, mutable pointer (for example -`sales`) to the latest content hash of a logical entry. Each alias carries an -ordered `history` list of every hash it has pointed at (V1, V2, …, oldest -first). Revising an alias mints a new content hash, advances `latest`, and -appends to `history`; the old versions remain as forensic lineage. -`catalog_diff` resolves version indices (-1 latest, -2 previous) through this -history. - -**Parent edges and following.** At build time each entry records its direct DAG -parents as `{hash, ref, follow}`. `tracked_expr_from_alias('name')` records -`follow=True`: the edge names an alias, and the child goes stale as that alias -advances. `pinned_expr_from_alias('hash')` records `follow=False`: the edge pins -an exact hash and is never disturbed by recalc. Aliases are resolved to hashes -at read time, not baked into the edge. - -**Staleness.** Read-only and side-effect-free. An entry is stale on the alias -axis when a `follow=True` parent's alias head no longer equals the recorded -hash, or on the source axis when a recorded source digest no longer matches the -file's current digest. Computing staleness never executes anything; it only -compares the manifest against the current world and returns reasons. - -**Recalc cone.** When an alias head advances, its dependents form a cone of -entries that may now be stale. The cone is recomputed in topological order -(Kahn's algorithm over intra-cone parent edges) so parents rebuild before -children. Auto-recalc only recomputes the followers of the alias that just -moved; pre-existing ("orphan") staleness is left in place, logged, and -classified against the recorded error store rather than treated as an -unexplained invariant break. See [reactive-recalc.md](reactive-recalc.md) and -[recalc-mechanism.md](../plans/recalc-mechanism.md). - -**Worthy and cheap entries (materialization).** At build, tallyman decides once -whether an entry is *worthy* (it does expensive work, or work that cannot -inherit a row order: an aggregate, join, sort, limit, window function, union, -distinct, unnest, a second file, a non-pure operation or a UDF) or *cheap* -(row-preserving over one file). A worthy entry is materialized: tallyman runs -its query when the entry is created and writes the result to a **snapshot** -parquet file, which every read after that uses. A cheap entry writes nothing -and re-runs its small plan on every read. Tallyman owns the files and xorq's own -cache is not involved. `ensure_materialized` makes every file an entry reads -exist before anything runs, re-creating a missing snapshot, ordered copy or -source clone and checking what it made, so a deleted file heals on the next -read. Sources enter through **ordered copies**: a parquet copy of each source, -in file order, with a last column `__row_order`. See [caching.md](caching.md). - -**Portability.** A build directory embeds absolute filesystem paths in -`expr.yaml`. On write these are rewritten to `${TALLYMAN_PROJECT_ROOT}` -placeholders; on load they are expanded into a stable per-entry directory marked -complete by a sentinel, so a project can be copied or packed and run from -anywhere on disk, and the expanded path stays consistent so Buckaroo's -stat-cache keys match across restarts. - -**Checkpoint and reset-to-revision.** A checkpoint is an atomic git transaction -under a per-project file lock: it captures pointers, zips pending recipes, -stages all tracked files, commits once, and tags the step. Reset-to-revision -does a hard git reset to a commit and then reconciles untracked entry directories -back to the recorded pointers, retiring to or restoring from the bullpen without -recompute (a forward reset copies back from the bullpen). It also moves the -source clones no surviving entry refers to into the bullpen instead of deleting -them. It leaves `compute_cache/` alone: those files are named by content hash and -can be made again. Live operations never read the bullpen. -Every write to a project (a build, a materialization, a checkpoint) takes the -same re-entrant project lock, so there is one writer at a time. - -**Live updates over SSE.** The companion pushes changes to the browser with -Server-Sent Events (SSE): the SPA opens one long-lived HTTP stream to -`GET /{project}/api/sse` through the browser's native `EventSource`, and the -server writes named event messages down it. This is the inverse of polling. The -browser never asks "anything new?" on a timer; it holds the stream open and the -server speaks when something changes. The SPA registers listeners for these -kinds: `new_entry`, `build_failed`, `notebook_changed`, `chart_attached`, -`post_processing_changed`, `summary_stat_changed`, `recalc`, and -`project_switched` (plus `hello` and `ping`, which open and keep the connection -alive). An event is a signal, not a data payload: every one increments a -monotonic `version` counter held in a React context (`SSEContext.tsx`), and -components key their effects on that counter, so a bump triggers exactly one -refetch of the affected REST resource. That is what "refetches off an SSE -version counter rather than polling" means. Two events also carry state the -refetch cannot derive: `recalc` ships the `{oldHash: newHash}` remap so an open -entry view can follow its entry to the new hash, and `project_switched` ships -the project name to navigate to. If the stream drops, the context flips to -`offline`. Sources: `SSEContext.tsx` on the browser side, the `/{project}/api/sse` -route and the `/internal/notify` fan-out in `app.py` on the server side. +### Identity: the content hash + +An entry's content hash is the name xorq gives the build directory of the +entry's rewritten expression. xorq hashes a file read by its path alone, so +tallyman puts the content in the path: every file a recipe's expression reads is +a worthy entry's snapshot, named by that entry's content hash, and a child's +hash is therefore a function of its parent's. The chain bottoms out at source +entries, whose hash is `md5("source||")` truncated to +12 hex characters (`source_import.source_entry_hash`): the md5 digest of the +imported bytes and the reader options they were parsed with, and nothing else. +It cannot come from xorq, because a source entry's generated recipe reads the +snapshot that the hash names. So the hash of any entry covers the bytes of every +file under it. Two entries with the same expression over the same inputs +collapse to one hash, which makes building idempotent. The absolute path of the +project is part of a computed entry's hash, so the same recipe in a project at +another path gets another hash; a source entry's is not, so two projects that +import one file each hold their own entry under the same hash. The clone store +underneath is [ADR-002](../plans/ADR-002-source-identity-content-hash.md)'s, +narrowed by [ADR-011](../plans/ADR-011-sources-are-aliases.md) to one mode: +every import digests and clones. + +### Aliases and versions + +An alias is a line `{alias, latest, history, kind}` in `aliases.jsonl`. +Revising an alias builds a new entry, moves `latest` to its +hash and appends that hash to `history`; the old versions stay as they were. +The kind is `catalog` or `source`, and a name is one or the other, never both. +`set_alias` keeps an alias's kind matching its entries: a catalog alias never +points at a source entry and a source alias never points at a computed one, +whichever route sets it. A source alias moves only by an import: revising it, +promoting a diff onto it and `catalog_alias` onto a source entry are refused, +in the MCP tools and in the companion's code-edit and promote-diff routes +alike, before anything is built. Renaming and removing a source alias work as +for a catalog alias. +`catalog_diff` and the diff page pick versions from this history (`-1` is the +latest, `-2` the one before). A Buckaroo session id is derived from the project +and the content hash, `entry--`, so tallyman keeps no record of +sessions. + +### Parent edges + +At build time each entry records the entries its recipe read, as +`{hash, ref, follow}` in `manifest.parents`. +`tracked_expr_from_alias("sales")` records `follow=True`: the child follows the +alias and goes stale when the alias moves. `pinned_expr_from_alias` takes only a +version reference such as `"sales-v2"` and records `follow=False`: the child +stays on that entry. A bare alias is refused (#166), because it would pin +whatever the head happened to be, and so is a bare content hash (ADR-011 D5, no +bare hashes in recipes), so every edge an authored recipe records names an +alias (a promoted diff's generated recipe is the exception: it names its two +entries by hash and records no edge); an entry with no alias, +one built by `catalog_run`, has to be named before anything can build on it. A +source alias is read the same way as any other. The edge stores the hash the +alias pointed at when the child was built, and the staleness scan looks the +alias up again to compare. + +### Worthy and cheap entries + +Tallyman decides once, when an entry is built, whether it is worthy, and +records the verdict in the manifest as `cache_worthy`, with a short reason in +`cache_worthy_why` +(`worthiness.classify_expr`). An entry is cheap only if every relation +operation in it is a file read, a filter, a column selection or computed +column, a column drop, a drop of null rows or a fill of nulls; it reads exactly +one file; and no value in it multiplies rows (`unnest`), depends on the order +rows arrive in (a window function) or is not pure (`random()`, `uuid()`, +`now()`, `today()`, any UDF). Everything else is worthy. The test is an +allow-list, so an operation nobody has classified costs a copy instead of +unstable paging. A cheap entry must keep `__row_order`, because it pages by the +column of the file it reads: a select that drops the column fails the build, +and the error shows the corrected select. + +### Materialization + +A worthy entry is materialized when it is created, by one routine, +`materialize`, which every heal of a computed entry also uses. (A source entry's +snapshot is written by the import, and made again from its clone, with the same +writer settings.) `materialize` runs the entry's build on a single-partition +connection (so a float total is merged in one order), streams the rows through a +writer with a pinned layout (zstd, row groups of 1,048,576 rows, a page index), +numbers them in a last `__row_order` column, writes a temporary file and renames +it over the snapshot, all under the project lock, and returns the content digest +of the file it wrote. A heal renames at once; a create leaves the file at its +temporary name and renames it after the manifest is written, so a failed build +never touches the file already at the path. At create it runs the query twice +and compares the two digests. If they differ, the entry still builds, the +manifest records `reproducible: false` with the columns that differed, and the +snapshot is pinned. A cheap entry writes nothing: its plan is streamed once in +full at create, so an error in it fails the tool call. Every file that holds a +result is tallyman's; no build contains a xorq cache node, and xorq's own cache +is not used. [caching.md](caching.md) has the details. + +### Reads: `cached_result_expr` and `ensure_materialized` + +Every consumer reads an entry's result through `cached_result_expr`: `/api/data` +pages, charts, diffs, post-processing, and a child recipe chaining off the +entry. It first calls `ensure_materialized`, which makes every file the entry's +plan reads exist before anything runs: a missing snapshot is made again by +running its entry's build, or, for a source entry, by parsing its clone again +with the reader options it recorded (`_heal_a_source`). No plan reads a clone, +and nothing in a read makes one again. With the clone gone the source entry's +snapshot is the last copy of its rows, so it is pinned; if it is gone too, the +read fails with an error naming the missing clone and the +`catalog_import_source` call, reader options included, that repairs the version. +The Buckaroo hand-off calls `ensure_materialized` too. A worthy entry then reads +as one bare read of its snapshot, without loading its build when the file +exists, and a cheap entry as its frozen plan, re-run over files that exist. A +healed snapshot is checked against the recorded `result_digest`. A mismatch is +still served, since the rows are the honest output of the frozen build, but the +heal records the digest it wrote in the manifest's `unfaithful_heal_digest`, +which pins the file, logs an `unfaithful_heal` error for the error banner, and +wipes the entry's Buckaroo statistics; when the heal runs in the companion, +Buckaroo is also told to reload the entry's grid. The pin is part of the +manifest, so it moves with the entry through a reset and survives dismissing the +banner, which deletes the error log. + +### Sources: imports and source aliases + +A recipe names aliases, and a build reads only files tallyman owns +([ADR-011](../plans/ADR-011-sources-are-aliases.md)). A file enters the catalog +by one call, `catalog_import_source(outside_path, alias, pinned_version=None, +schema=..., reader_options=...)` (`source_import.update_and_depend`). It +digests the file, clones the bytes to `data/.cas/` and checks +the clone against the digest, writes the source entry's snapshot, writes the +entry itself (a generated recipe, a frozen build, a schema and a manifest with +`provenance`), and appends the entry to the source alias's history. The path +can be anywhere, and after the import it is never read again, so editing, +moving or deleting the original changes no build. New data arrives by importing +again under the same alias: different bytes mint the next version, and every +entry that follows the alias goes stale exactly as it would after a revise. +The import emits the same events as a revise, runs auto-recalc on the same +switch, and lands with its cascade as one checkpoint. + +The call's outcome depends on the alias's history (ADR-011 D3, the case table). +Identical bytes are a no-op returning the current version. `pinned_version=N` +claims the file is version N, or the next version to mint, and is refused if it +is neither. Versions cannot be skipped, and history only grows: bytes equal to +an older version are refused, with `reset_to` named as the way back and +`pinned_expr_from_alias("-v")` as the way to read that version. Bytes +another alias of the project already holds are refused too, naming that alias: +one set of bytes, read one way, is one version under one alias, and a second +name for it is a catalog entry whose recipe is `tracked_expr_from_alias("")`. A re-import of a version whose snapshot is gone rewrites nothing of +the entry: it restores the clone from the given file if needed and heals the +snapshot the way a read would, verified against the recorded digest. + +Reader options are fixed at import (ADR-011 D12). A parquet file takes none: its +snapshot is written by pyarrow in the file's order, so it keeps the file's +types. A CSV is parsed by polars under the schema and `scan_csv` options named in +the call, through ADR-005's schema language and inference ladder, and its rows +go to the same pyarrow writer in batches. A timestamp column whose schema type +has a zone is read as text and parsed in that zone (`io._parse_zoned`): text with +a UTC offset is converted into the zone, and text without one keeps its +wall-clock time. Offset-less text that names a time the zone skips or repeats at +a daylight-saving change, and a column that mixes text with and without an +offset, are errors naming the column, row and value. The options are recorded +on the entry and are part of its hash, so a CSV read two ways is two imports +under two aliases. A source entry's snapshot is written with the same pyarrow settings as +any other snapshot, in row groups of 122,880 rows rather than 1,048,576. + +The build refuses every other way of reading a file: `read_project_file`, +`tallyman_read_csv`, `xo.deferred_read_csv`, and `xo.deferred_read_parquet` of a +file outside `compute_cache/`, each with an error naming the import to use. +`read_project_file` survives only inside the recipe the importer generates, where +a context variable resolves it to the entry's own snapshot. + +### Row order + +Every file tallyman writes ends in `__row_order`, so a page served by `/api/data` is +sorted by `__row_order`, and the same request returns the same rows in any +process. (The paging helper also takes user sort keys and puts `__row_order` +after them; no route passes any yet, and Buckaroo's grid does not use the +column yet, buckaroo-data/buckaroo#974.) Every `order_by` in a recipe also gets +`__row_order` and then the remaining columns appended as tie-breakers, and a +sort followed only by steps that keep row order (filters, selections, limits) +still decides the order that is written. +[system-contract.md](system-contract.md) states the rules. + +### Result digest + +`result_digest` is `arrow-sha256:`, a SHA-256 over the Arrow data of a +snapshot as read back (`digest.py`). It does not change with how +the rows were batched, the codec, the row-group size or the writer's version, +and it does change with any value, any null, the order of the rows, and the +column names and types. Worthy entries record it; a cheap entry has no snapshot +and records none. Its one job is to show whether a re-created snapshot +reproduced the original. A mismatch is attributed to an engine change (the +manifest records the xorq, xorq-datafusion and pyarrow versions and the +snapshot format version), to a recipe that bakes a changing value into its +graph (#88), or to a graph that runs differently each time (#83). Running each +new worthy entry twice finds most such recipes at create; what it cannot find, +such as `today()` or a non-pure parent, is #185. Design: +[ADR-004](../plans/ADR-004-result-digest-canonical-ordering.md) and +[ADR-009](../plans/ADR-009-digest-stability.md). + +### Staleness + +Staleness is a judgment that runs no query, opens no data file and changes no +catalog state; it reads `aliases.jsonl` and the manifests. An entry is stale +when a `follow=True` parent's alias now points at a different hash than the one +recorded, and for no other reason (ADR-011 D6, one staleness axis). A pinned +parent never makes its child stale. A changed input file is not a reason until +it is imported: the import moves the source alias, and the entries following +it go stale through the same rule. Only an entry that is the current head of an +alias counts as directly stale (#154); a superseded version is reported with +`live=False`. A parent alias that no longer exists is reported under +`unknown_axes`. + +### Recalc cone + +When an alias head advances, the entries that followed it become directly +stale. They and every current head built on them form the +cone. Recalc replays each member's recipe in topological order (Kahn's +algorithm over the edges inside the cone), so a parent rebuilds and its alias +moves before its children replay. A member whose inputs did not move, such as +a child that pins a version of its parent, replays to the same hash and is left +alone. Auto-recalc, on by default for each project, runs this for the followers +of the alias a revise or an import just moved; staleness from any other cause +is left in place, logged, and classified against the recorded errors. See +[reactive-recalc.md](reactive-recalc.md). + +### Portability + +A build contains absolute paths in its `expr.yaml`. The build step rewrites the +project's path to a `${TALLYMAN_PROJECT_ROOT}` placeholder, +and a read fills it back in, into the stable per-entry directory +`.xorq_build_expanded/`. A project can therefore be packed, copied or cloned to +another path, and the expanded build keeps one path across restarts. The +expanded directory's marker +does not record which project path it was filled in with, so a copy that +carries the expanded directories along keeps reading the old location (#209). + +### Checkpoint and reset + +A checkpoint takes the project lock, records the complete entry directories in +`entries.jsonl`, zips any entry that has no recipe zip yet, runs `git add -A`, +commits once and tags `step-NNN`. The MCP server checkpoints after each tool, +and the companion after each mutating request; a recalc and a diff promotion +checkpoint themselves, once each. `reset_to` takes the lock, runs +`git reset --hard` to the step, and reconciles the files git does not track. Entry +directories the step does not list move to the bullpen; listed ones that are +missing are copied back from it. Source clones that no surviving entry refers to +move to `bullpen/cas/`, never deleted, and clones a restored entry needs are +copied back. A reset leaves `compute_cache/` alone: its files are named by +content hash, a leftover cannot be served for another entry, and a file that is +missing afterwards is healed like any other. An entry directory retired a second +time replaces the copy already parked under its name, since the live one is the +one that agrees with the snapshot on disk; a live directory with no manifest (an +interrupted build's) is dropped instead. The one live reader of the bullpen is +the Cache page: it lists a retired entry's snapshot as `retired` and decides its +pin from the manifest parked in the bullpen, counting a retired source version's +clone as present when it is parked in `bullpen/cas/`, since a reset forward +brings both back. + +### The project lock + +One re-entrant file lock, `catalog_state.project_lock` (a `flock` on +`artifacts/catalog/.checkpoint.lock`), is taken by a build (for its whole +length, recipe import included), an import, a materialization or heal, a +checkpoint and a reset. It holds between the MCP server and the companion, and +it is re-entrant within a thread, since a build materializes, and an import +heals, while it holds the lock. It blocks with no timeout, so a +page request whose entry needs a heal waits behind any build in the other +process (#186), and the two companion routes that build on the event loop, +`PUT /code` and `POST /promote_diff`, freeze the whole UI while they wait or +build (#190). Smaller writes to tracked files (aliases, notebook cells, charts, +display configs, `config.json`) take no lock: each replaces its whole file +atomically, so two processes editing the same file at the same moment can lose +one of the edits. The lock covers no reads either; they have the execution lock, +below. + +### The execution lock + +Each process executes reads on one DataFusion session, xorq's default backend, +and two threads executing on it at once fail with +`RuntimeError: Already borrowed`. The companion serves requests from FastAPI's +thread pool and the MCP server runs tool calls on FastMCP's, so +`execution.execution_lock`, one re-entrant lock per process, is held around +every execution on that backend: a `/api/data` page, a post-processing run, a +primary-key probe and `full_diff`'s Buckaroo helpers. Executions in one process +run one at a time, and another +process has its own backend and lock. The project lock comes first: anything +that can heal (`cached_result_expr`, `ensure_materialized`) runs before the +execution lock is taken, and `project_lock` raises in a thread that holds the +execution lock and would take a new file lock. A materialization's stream and a +cheap entry's row count at build run on connections of their own and take no +execution lock, so a build does not hold up page reads. +`tests/test_execution_lock.py` checks in the source that every other execution +sits inside `with execution_lock():`. + +### One server per data dir + +`tallyman run` claims its data dir before it starts anything else +(`tallyman_core/server_lock.py`): an exclusive `flock` on +`/server.lock`, held on a descriptor the server keeps open, so the +kernel drops it however the process exits. The file also holds an owner record +(pid, host, port, bind address, start time, data dir, command line), which names +the holder when a second `tallyman run` on the same data dir is refused. The MCP +server and `tallyman reset-to` find their companion from that record's port +(`companion_url()`), and believe it only while the lock is held; no default port +or environment variable stands in. Each notify, project switch and project +creation names its data dir as `home`, and a companion serving another data dir +answers 409. `tallyman mcp` and `tallyman serve` claim nothing. A second tallyman +runs on its own data dir and port: +`TALLYMAN_HOME= tallyman run --port `. + +### Live updates over SSE + +The companion pushes changes to the browser with Server-Sent Events: the SPA +opens one long-lived HTTP stream to +`GET /{project}/api/sse` through the browser's `EventSource`, and the server +writes named events down it. The browser never polls. The SPA listens for +`new_entry`, `build_failed`, `notebook_changed`, `chart_attached`, +`post_processing_changed`, `summary_stat_changed`, `recalc` and +`project_switched`, plus `hello` and `ping`, which open and keep the stream +alive. Each listened event except `project_switched` increments a `version` +counter in a React context (`SSEContext.tsx`), and components refetch their +REST resource when it changes. Two events carry state a refetch cannot derive: +`recalc` carries the `{oldHash: newHash}` remap, so that an entry view open in a +background tab moves to the entry's new hash (a focused tab stays put), and +`project_switched` carries the project to navigate to. The server also +publishes `entry_added`, `alias_changed`, `alias_renamed`, `display_changed`, +`project_reset` and `unfaithful_heal`, which the SPA has no listener for, so +they cause no refetch. If the stream drops, the context reports `offline`. +Sources: `SSEContext.tsx` in the browser; the `/{project}/api/sse` route and the +`/internal/notify` fan-out in `app.py` on the server. ## Request and data-flow paths -### catalog_run / catalog_create — author a new entry - -1. Claude Code calls the MCP tool with xorq source. -2. `tallyman_xorq` compiles the expression, computes the content hash, and runs - the build, writing the entry build dir: `expr.py`, `xorq_build/`, - `schema.json`, and finally `manifest.json` (written last, atomically, as the - completeness sentinel). A `cache_worthy` entry is materialized during the - build: its query runs twice and its snapshot is written to - `compute_cache/result_cache/`. -3. The MCP dispatch boundary fires a checkpoint: `tallyman_core` zips the recipe, - stages the tracked surface, commits one git revision, and tags it. -4. The MCP server best-effort POSTs a notification to the companion. -5. The companion emits an SSE `new_entry` event; the SPA bumps its version - counter and refetches the entry list. See [expression-lifecycle.md](expression-lifecycle.md). - -### catalog_revise + auto-recalc — revise an entry and cascade - -1. A revision arrives from `catalog_revise` (MCP) or `PUT /code` (companion). It - mints a new content hash, advances the alias `latest`, and appends the old - hash to `history`. Charts and display configs carry forward from the old hash - to the new one only where the new hash does not already define them. - Self-alias references are rejected — an entry following its own alias would be - permanently stale by design. -2. Auto-recalc, if enabled for the project, walks the recalc cone of the alias's - followers in topological order and rebuilds each, re-pointing aliases before - replaying children. This walk is checkpoint-free. -3. The whole thing lands as one checkpoint, so head advance plus cascade is a - single git revision that reset-to-revision undoes atomically. -4. The companion emits an SSE `recalc` event carrying `{oldHash: newHash}` for - each remapped entry; backgrounded SPA views navigate to the new hash, focused - views stay put. See [reactive-recalc.md](reactive-recalc.md) and - [auto-recalc-on-revise.md](../plans/auto-recalc-on-revise.md). +### catalog_run / catalog_create: author a new entry + +1. Claude Code calls the tool with Python that binds `expr`. +2. `build_and_persist` takes the project lock and imports the code. While it + runs, `tracked_expr_from_alias` and `pinned_expr_from_alias` record each + parent edge and make the parent's files exist. +3. The build refuses what cannot become a sound entry: a raw file read, a bare + content hash or bare alias passed to `pinned_expr_from_alias`, an in-memory + table, a `.cache()` call, an assignment to `__row_order`, a cheap entry that + drops it, and a join chain over three entries that all carry it. + It classifies the entry, adds the canonical sort to a worthy entry, and + freezes the expression with xorq's `build_expr`, whose directory name is the + content hash. If a complete entry with that hash is already on disk, the + build stops and returns it. +4. It writes the entry directory (`xorq_build/` with portable paths, + `expr.py`) and runs the query: a worthy entry is materialized, twice, and its + snapshot written at a temporary name; a cheap entry is streamed once and + nothing is kept. +5. It writes `schema.json` (read from the snapshot for a worthy entry) and then + `manifest.json`, atomically. Only then is a worthy entry's snapshot moved + into place, as the last write to the entry's result. +6. `catalog_create` sets the alias and adds a notebook cell. The tool notifies + the companion, and as it returns the dispatch wrapper commits a checkpoint, + which zips the recipe. +7. The companion publishes `new_entry`, and the SPA refetches the entry list. + +If the build fails after it has created the entry directory, the directory is +removed, along with the snapshot's temporary file. The file at the snapshot's +path is untouched, so a snapshot that was already there, such as one a reset +left behind, survives a failed re-add. + +### catalog_import_source: bring a file in + +1. Claude Code calls the tool with a path, a source alias and, for a CSV, a + schema and reader options. +2. `update_and_depend` refuses a directory, a name that is already a catalog + alias, and a file that is neither parquet nor CSV. It decides the reader, + digests the file and computes the entry hash, then takes the project lock + and consults the alias's history (the case table under + [Sources](#sources-imports-and-source-aliases)). +3. To mint a version it clones the bytes to `data/.cas/`, writes the snapshot, + and writes the entry directory: a generated `expr.py` that records the path, + digest and reader options in its header, a frozen `xorq_build/`, + `schema.json`, and `manifest.json` with `provenance`, last. A failure removes + the entry directory it created. It then appends the entry to the alias. +4. The tool records an `alias_set` event, adds a notebook cell for a new alias + (or carries charts and display configs forward from the previous version), + notifies the companion with `new_entry`, and runs auto-recalc for the + alias's followers. The checkpoint as the tool returns takes in the import + and its cascade. A no-op import records and notifies nothing. + +### catalog_revise + auto-recalc: revise an entry and cascade + +1. A revision arrives from `catalog_revise` (MCP) or `PUT /code` (companion); + both refuse a source alias, before building anything, with an error that + names the import. It builds a new entry, moves the alias's `latest` to the + new hash and appends it to `history`. Charts and display configs carry + forward from the old hash to the new one where the new hash has none of its + own. A revision that reads its own alias by name is rejected: it would follow + its own head and be stale forever. +2. If auto-recalc is on for the project (it is by default), the followers the + revise made stale, and every current head built on them, are rebuilt in + topological order, each alias re-pointed before its children replay. The + walk takes no checkpoint of its own. +3. The revise and the cascade land as one checkpoint, so a reset to the step + before undoes both. +4. A cascade that changed anything publishes a `recalc` SSE event with the + remap, and the companion reloads the project's Buckaroo sessions. See + [reactive-recalc.md](reactive-recalc.md). ### Viewing an entry grid -1. The SPA entry-detail pane mounts `LazyBuckarooEmbed`, which waits for the grid - to scroll near the viewport (IntersectionObserver) and then polls the - companion for the entry's session (`GET /{project}/api/session/{hash}`) until - it returns a WebSocket URL. -2. The companion first runs `ensure_materialized`, so every file the entry reads - exists and anything it had to re-create is verified. Only then does it POST to - the Buckaroo subprocess's `/load_expr`, with a session id derived from the - project and the hash. A worthy entry is handed a *view build*, a build whose - whole graph is one read of its snapshot (`/.xorq_view_build/`); a cheap - entry is handed its own build, expanded into a stable per-entry path with - `${TALLYMAN_PROJECT_ROOT}` resolved. The payload names `__row_order` as the - row-order column. Buckaroo runs queries only for summary stats, sorting and - paging; it never executes an aggregate or join on tallyman's behalf. -3. Buckaroo creates the session and streams the grid over WebSocket; sort and - search push down to the xorq backend rather than paging a materialized - parquet. -4. The data tab stays mounted (hidden) across tab switches to keep the WS session - alive. Tallyman keeps no session record. Opening the entry again POSTs the same - id, which Buckaroo answers from the session it holds or rebuilds if it has - dropped it (it does after an idle hour, and after a restart). +1. The catalog page's data tab requests `GET /{project}/api/session/{hash}` as + soon as it opens. On the notebook page, the data route (`/api/notebook_full`) + opens a session for every cell each time the page loads or refetches (on any + SSE event), and each cell's grid asks `/api/session` for its WebSocket URL + once the cell scrolls near the viewport (#202). +2. The companion calls `load_session`, which runs `ensure_materialized` first, + so a missing file is made again and checked before Buckaroo is involved. A + failure there shows in the page with its reason and a retry button. +3. It posts `/load_expr` to Buckaroo with the session id + `entry--`. A worthy entry is handed its view build + (`/.xorq_view_build/`, written once); a cheap entry its own build, + expanded into `/.xorq_build_expanded/`. The body also names the entry's + statistics cache directory, `__row_order` as the `row_order_column`, and the + `project_root` Buckaroo searches for klasses. That root is `artifacts/`, so + Buckaroo finds the display klasses there but not the stats and + post-processing functions, which tallyman writes under `artifacts/catalog/` + (#170). +4. Buckaroo creates the session, or answers from the one it holds, and the grid + connects over a WebSocket. Paging, sorting, search and summary statistics are + Buckaroo's queries over the build it was handed, which for a worthy entry is a + read of the snapshot. Buckaroo 0.15.6, the pinned version, ignores + `row_order_column`, so the grid's pages are not yet ordered by `__row_order` + (buckaroo-data/buckaroo#974); `/api/data` pages are. +5. Tallyman keeps no record of sessions. Every open posts `/load_expr` again + with the same id. Buckaroo skips the work while it holds that session with + the same build directory, and creates it again after dropping it (it drops a + session idle for an hour, and loses them all on a restart). Two opens at once + both post, and a promoted diff entry, which sends its column colouring, + re-runs Buckaroo's statistics on every open (#202). ### Diffing versions -1. The SPA diff page resolves the version pair (defaulting to V_{n-1} vs V_n via - the alias history) and requests the diff from the companion. -2. The companion builds a compare expression: an outer join of the two versions - with a membership column (a-only / b-only / both) plus per-column `{col}_eq`, - `{col}_pct_delta`, and `{col}_abs_delta` sentinel columns. The compare - expression is memoized for the process lifetime because the two entry hashes - are immutable. -3. Buckaroo column-config overrides color the diff: categorical coloring for - key/equality columns, numeric coloring for the delta columns. -4. The diff grid loads through `/load_expr` with a compare build, with diff stats - cached per entry pair under `diff_stat_cache/`. Both sides are read through - `cached_result_expr` with `__row_order` dropped, so the files exist before the - join is built. The join itself is not materialized first; that is tracked in - #188. +1. The diff page resolves the version pair (by default V_{n-1} against V_n) and + requests `/{project}/api/diff_data/{alias}/{va}/{vb}`. +2. The companion reads both sides through `cached_result_expr`, so both sides' + files exist first, and computes the code, schema, statistics, head and keyed + diffs (`full_diff`). Those summaries still include `__row_order` as a data + column (#200). +3. For the grid it builds a compare expression: an outer join of the two + versions on the diff key, with `__row_order` dropped from both sides, a + membership column (a only, b only, both), and per-column `{col}_eq`, + `{col}_pct_delta` and `{col}_abs_delta` columns. The expression is memoized + per pair and key (an LRU of 128, cleared on a reset or recalc). +4. The compare build is posted to Buckaroo as session `diff--` (the first + 12 characters of each hash), with statistics cached per pair under + `diff_stat_cache/`. The join is not materialized first, so Buckaroo runs it + for every query of the diff grid (#188). + +`catalog_promote_diff` and the diff page's promote button turn a diff into an +entry of its own, whose recipe calls `build_diff_expr(a_hash, b_hash, keys)`. +It contains a join, so it is worthy and materialized like any other entry. + +### Resetting to a revision + +`tallyman reset-to ` (CLI) and `POST /{project}/api/reset` (companion) +call `reset_to`, described under [Checkpoint and reset](#checkpoint-and-reset). +The companion then clears its in-memory result and compare memos and the +`diff_stat_cache/` directory, reloads the project's Buckaroo sessions, and +publishes `project_reset`. The CLI posts `project_reset` to the companion's +`/internal/notify`, which does the same clean-up. `tallyman revisions` lists the +steps, and `tallyman revisions label ` names one. + +## Known defects + +These open issues describe places where the system does not yet do what the +rest of the docs say it should. The docs describe the current behaviour and cite +the issue where it matters. + +Writes, the lock and processes: + +- #186: the project lock is one blocking lock with no timeout, so slow work in + one process blocks page reads in the other. +- #190: `PUT /code` and `POST /promote_diff` build on the companion's event + loop, which freezes the UI while they wait for the lock or build. + +Row order and diffs: + +- #199: the three-way join check also refuses chains of semi and anti joins. +- #200: `full_diff` keeps `__row_order` as a data column. +- #205: the canonical sort leaves nested columns out of its tie-break. +- #206: the snapshot writer drops any column named `__row_order_right`, + including one the author made. +- #188: the live diff grid hands Buckaroo an unmaterialized join. + +Heals and Buckaroo: + +- #202: every grid open posts `/load_expr`; concurrent opens load twice, and a + promoted diff re-runs Buckaroo's statistics on every open. +- #201: a klass reload posts `/reload_expr` once per catalog entry, one after + another, on the event loop. +- #203: an unfaithful heal runs its checks and the forced Buckaroo reload while + holding the project lock, and the reload opens a session for an entry nobody + has open. +- #208: an unfaithful heal of a worthy parent changes its cheap children's rows + under their hashes; only the parent is flagged. +- #209: a copied project keeps reading the old path through its expanded + builds. +- #210: the plan memo keeps each loaded build and its backend objects alive. + +Design questions still open: #185 (a non-pure recipe's verdict is not recorded +or passed on to entries built on it) and #187 (an ungrouped float `SUM` depends +on the layout of the file it reads, which the snapshot format version pins). +Found while checking these docs (#233): a recipe's +`tracked_expr_from_alias` and `pinned_expr_from_alias` resolve the project from +the `active_project` file, not the MCP session's own project, so the two can +disagree after another session switches projects and a recipe then looks its +aliases up in the other project (related to #39). +Older open issues in the same areas: #170 (Buckaroo is not pointed at the +project's stats and post-processing functions) and #157 (Buckaroo's on-disk +statistics cache has not been seen to give a first-load hit). + +Filed on 2026-09-24 against the same code, and described one by one in +[architecture-new.md](architecture-new.md#13-known-defects) and +[plans/open-bugs-2026-09-24.md](../plans/open-bugs-2026-09-24.md): wrong rows or +edges without an error (#228, #229, #232), the import (#224, #225, #227, +#234, #237, #239), writes and leftovers (#226, #230, #240), recalc of a source +entry (#238), the SPA's missing SSE listeners (#235) and ADR-011 leftovers in +the code (#236). + +These no longer apply since ADR-011 (a raw input is a source alias, PRs #217 to +#219), because the mechanism each was about is gone: #197 (a parquet source's +copy changed column types; pyarrow writes a source snapshot now), #198 (a CSV's +first read got JSON-rewritten reader options; the options are fixed at import +and must be plain values), #211 and #207 (the ordered copy's `.digest` file and +the copies left behind by source edits; ordered copies no longer exist), #191 +(staleness could not resolve a CSV outside `data/`; staleness no longer reads +files), and the two staleness defects the first version of this list named (the +scan wiping the source-digest memo, and a hash-pinned child stale for good on +the source axis). + +Fixed on this branch: + +- #193 by #222: a failed build deleted the snapshot already on disk for its + hash. +- #194, #195 and #196 by #223: a reset could pair a non-reproducible entry's + older manifest with its newer snapshot, a retired entry's snapshot lost its + pin, and the pin from an unfaithful heal lived in `errors.jsonl`, so + dismissing the error banner lifted it. +- #118 by #242: concurrent executions on a process's shared backend failed with + `Already borrowed`. See [The execution lock](#the-execution-lock). +- #183 by #241: two servers on one project went undetected. See + [One server per data dir](#one-server-per-data-dir). +- #204 by #245: with its manifest gone, an entry's worthiness was guessed from + whether a snapshot existed, so a worthy entry that had lost both was served + as cheap. Every read now refuses such a directory, which reverses #95, where + `/api/data` served it with a total of 0. +- #231 by #244: a zoned timestamp in a CSV schema read offset-less text as UTC + and converted it, so `09:30` in New York was stored as `04:30-05:00`. ## Related documentation -Currency notes below reflect a docs-vs-code audit on 2026-06-25. They will drift; -when in doubt, the code wins. - -### Architecture docs (`docs/`) — describe the current system - -- [system-contract.md](system-contract.md) — **normative**: the invariants and - binding rules the system guarantees (identity, read/write/cache contracts). - Where the descriptive docs and this contract disagree, the difference is a - bug. The read path implements it as of the #163 fix (PR #167). **Current.** -- [expression-lifecycle.md](expression-lifecycle.md) — one expression from MCP - ingest to rendered rows, naming every artifact and cache write. **Current.** -- [reactive-recalc.md](reactive-recalc.md) — revise an alias, recompute its - dependents; the cone and the dependency graph. **Current.** -- [caching.md](caching.md) — the caches across the stack and their invalidation. - **Mostly current.** -- [installing.md](installing.md) — install and run the spike. **Mostly current.** -- [mcp-server.md](mcp-server.md) — every MCP tool and prompt Claude Code drives, - with parameters, return shapes, and per-tool side effects. **Current.** - -### Design records / ADRs (`plans/`) - -- [ADR-002-source-identity-content-hash.md](../plans/ADR-002-source-identity-content-hash.md) - — content-addressed source reads so `content_hash` tracks source data. - **Mostly current.** -- [ADR-001-git-subprocess-threading.md](../plans/ADR-001-git-subprocess-threading.md) — - calling git from the multithreaded server (fork-safe `posix_spawn`). - **Mostly current.** -- [ADR-003-result-cache-cost-rubric.md](../plans/ADR-003-result-cache-cost-rubric.md) — - a *proposed* cost-vs-size cache rubric. **Partially stale / not adopted:** the - structural `cache_worthy` admission test it proposes to remove is still the - live gatekeeper, and `ensure_result` it names was removed (#73). -- [ADR-004-result-digest-canonical-ordering.md](../plans/ADR-004-result-digest-canonical-ordering.md) - — a canonically-ordered snapshot and a digest of it, replacing the per-row - Python digest (#137). **Partially superseded:** the canonical sort stays; the - digest is now a content digest of the file read back (ADR-009), and the CSV - row-index column is `__row_order` (ADR-008). -- [ADR-006-read-path-loads-builds.md](../plans/ADR-006-read-path-loads-builds.md) - — reads load the frozen build (#163). **Partially superseded** by ADR-007: the - parent's cache node no longer travels in a child's build, and the - snapshot-key tripwire is gone. +Currency notes below reflect a docs-against-code check on 2026-09-22 against +the branch of #189 (ADR-007, ADR-008 and ADR-009 implemented), redone on +2026-09-24 after ADR-011 was merged into it. They will drift; when in doubt, the +code wins. + +### Architecture docs (`docs/`): the current system + +- [system-contract.md](system-contract.md): **normative**. The invariants and + rules the system guarantees (identity, reads, writes, materialization, row + order). Where the code or a descriptive doc disagrees with it, the difference + is a bug; its last section lists the known ones. **Current.** +- [expression-lifecycle.md](expression-lifecycle.md): one expression from MCP + ingest to rendered rows, naming every file written and when. **Current.** +- [caching.md](caching.md): every cache in xorq, tallyman and Buckaroo, what it + saves, what keys it, and what invalidates it. **Current.** +- [reactive-recalc.md](reactive-recalc.md): revise an alias, recompute its + dependents; the dependency graph, staleness and the cone. **Current.** +- [mcp-server.md](mcp-server.md): every MCP tool and the prompt, with + parameters, return shapes and side effects. **Current.** +- [installing.md](installing.md): install and run tallyman. **Current.** + +### Design records (`plans/ADR-*.md`) + +- [ADR-001-git-subprocess-threading.md](../plans/ADR-001-git-subprocess-threading.md): + calling git from the multithreaded server (fork-free `posix_spawn`). + **Accepted; mostly current.** +- [ADR-002-source-identity-content-hash.md](../plans/ADR-002-source-identity-content-hash.md): + content-addressed source clones, so `content_hash` tracks source data. + **Narrowed by ADR-011:** the clone store stands, and a reset moves + unreferenced clones to the bullpen instead of deleting them (ADR-007); the + identity modes, `manifest.sources` and the source-digest memo are gone, and a + clone is written only by an import. +- [ADR-003-result-cache-cost-rubric.md](../plans/ADR-003-result-cache-cost-rubric.md): + a cost-against-size cache rubric. **Proposed, not adopted.** The structural + cheap-or-worthy test it would remove still decides, now as ADR-008's + allow-list recorded in the manifest; `classify_build` and `ensure_result`, + which it names, are gone; ADR-007's bare-read chaining addressed its + motivating case; its budget and eviction half is still open. +- [ADR-004-result-digest-canonical-ordering.md](../plans/ADR-004-result-digest-canonical-ordering.md): + a canonically ordered snapshot and a digest of it, replacing the per-row + Python digest (#137). **Partly superseded:** the digest is a content digest of + the file read back (ADR-009), and the row-index column is `__row_order`, + written by every writer and appended to every sort (ADR-008). +- [ADR-005-intelligent-csv-import.md](../plans/ADR-005-intelligent-csv-import.md): + the CSV reader's schema and error contract. **Partly superseded** by ADR-008 + and ADR-011: the column is `__row_order`, the trailing `order_by` is gone, and + the reader runs inside `catalog_import_source` with its options fixed there; + the parsed rows are the source entry's snapshot, made again from the clone + when missing. The schema language, inference ladder and error contract are + unchanged. +- [ADR-006-read-path-loads-builds.md](../plans/ADR-006-read-path-loads-builds.md): + reads load the frozen build (#163). **Partly superseded** by ADR-007: ADR-006 + D4 (chaining inlines the parent's cache node) and ADR-006 D8 (the manifest + records a snapshot key that reads check) are retired, and ADR-006 D5 (the + canonical sort) is amended by ADR-008 and ADR-009. - [ADR-007-tallyman-owned-materialization.md](../plans/ADR-007-tallyman-owned-materialization.md), [ADR-008-row-order-of-reads.md](../plans/ADR-008-row-order-of-reads.md) and - [ADR-009-digest-stability.md](../plans/ADR-009-digest-stability.md) — the cache - redesign this doc and [caching.md](caching.md) describe: tallyman writes its - own result files, every file carries `__row_order`, and a rewritten file is - flagged only when the result changed. + [ADR-009-digest-stability.md](../plans/ADR-009-digest-stability.md): the cache + redesign that this doc and [caching.md](caching.md) describe. Tallyman writes + its own result files, every file carries `__row_order`, and a re-created file + is flagged only when the result changed. **Accepted (2026-09-22), implemented + in #189.** Each has an "Implementation notes" section saying where the code + differs from its text. +- [ADR-010-immutable-store-one-owner.md](../plans/ADR-010-immutable-store-one-owner.md): + an immutable result store, every entry materialized, one owning process. + **Rejected (2026-09-22)**; kept for the record. ADR-007, ADR-008 and ADR-009 + stand. +- [ADR-011-sources-are-aliases.md](../plans/ADR-011-sources-are-aliases.md): + a raw input is a source alias whose versions are entries, a file enters only + by `catalog_import_source`, recipes name aliases and never bare hashes, and + staleness has one axis. **Accepted (2026-09-22), implemented** in #217, #218 + and #219, merged into #189's branch. Its implementation notes record where the + code differs from the decisions. ### Plans (`plans/`) -- [native-catalog-store.md](../plans/native-catalog-store.md) — the native - `tallyman_core.catalog` that replaced xorq's catalog package. **Mostly current.** -- [recalc-mechanism.md](../plans/recalc-mechanism.md) — how reactive recalc - works. **Mostly current.** -- [auto-recalc-on-revise.md](../plans/auto-recalc-on-revise.md) — atomic - auto-recalc on revise. **Partially stale:** line numbers and a couple of - function names drifted (`_recalc_walk` → `_replay_cone`), and the "future - Stage C" frontend SSE listener already shipped. -- [remove-ondemand-result-parquet.md](../plans/remove-ondemand-result-parquet.md) - — removing the on-demand `result.parquet` layer (#104). **Current.** -- [project_switcher.md](../plans/project_switcher.md) — the project switcher. - **Mostly current.** -- [89-determinism-prereqs-execution.md](../plans/89-determinism-prereqs-execution.md) - — clearing #89's determinism prerequisites. **Mostly current.** -- [catalog-xorq-integration-tests.md](../plans/catalog-xorq-integration-tests.md) - — coexistence/reset integration tests. **Partially stale:** references the old - `catalog.yaml` / `aliases.json` formats since replaced by JSONL. -- [llm-summary-stats.md](../plans/llm-summary-stats.md) — LLM-authored summary - stats. **Partially stale:** the Buckaroo-side klass pattern it describes - (`_Generated_*` classes) is now a `@stat()` decorator; a few signatures and - the notify `kind` differ. - -### Research notes / experiment logs (`plans/`, `demo/`) — point-in-time records - -The digest investigation behind #137 (current): +- [native-catalog-store.md](../plans/native-catalog-store.md): the native + catalog store that replaced xorq's catalog package. **Mostly current:** + `compute_cache.jsonl` and the reset's compute-cache prune are gone (ADR-007). +- [recalc-mechanism.md](../plans/recalc-mechanism.md): how reactive recalc + works. **Partly stale:** auto-recalc on revise now exists, and new data + arrives by an import, which advances a source alias like a revise; there is + no source axis. +- [auto-recalc-on-revise.md](../plans/auto-recalc-on-revise.md): atomic + auto-recalc on revise. **Implemented; partly stale:** function names drifted + (`_recalc_walk` is `_replay_cone`), and its "future Stage C" SSE listener + shipped. +- [remove-ondemand-result-parquet.md](../plans/remove-ondemand-result-parquet.md): + removing the on-demand `result.parquet` layer (#104). **Partly stale:** the + single materialized copy is now tallyman's snapshot, not xorq's cache + (ADR-007). +- [project_switcher.md](../plans/project_switcher.md): the project switcher. + **Mostly current:** the home root is `~/.tallyman-notebooks/`, and there is + no Buckaroo session file (ADR-007). +- [89-determinism-prereqs-execution.md](../plans/89-determinism-prereqs-execution.md): + clearing #89's determinism prerequisites. **Point in time;** the digest and + heal verification it describes were replaced by ADR-007 and ADR-009. +- [cache-soundness-audit.md](../plans/cache-soundness-audit.md): an inventory + of the #163 bug class, 2026-07-30. **Point in time;** #167 and #189 changed + several findings. +- [catalog-xorq-integration-tests.md](../plans/catalog-xorq-integration-tests.md): + coexistence and reset integration tests. **Partly stale:** it refers to the + old `catalog.yaml` and `aliases.json` formats. +- [llm-summary-stats.md](../plans/llm-summary-stats.md): LLM-authored summary + stats. **Partly stale:** Buckaroo's `_Generated_*` classes are now a `@stat()` + decorator, a few signatures and the notify `kind` differ, and the stats are + not found by Buckaroo (#170). + +### Research notes and experiment logs (`plans/`, `demo/`, `docs/research/`) + +Point-in-time records. The digest investigation behind #137: [datafusion-scan-order-findings.md](../plans/datafusion-scan-order-findings.md) -— why a parallel datafusion scan emits rows in a different order each run, and -the polars-ingest decision — and +(why a parallel DataFusion scan emits rows in a different order each run, and +the decision to ingest with polars, which stands for CSV: an import parses a CSV +with polars, and pyarrow copies a parquet file in file order) and [result-digest-vs-xorq-staleness.md](../plans/result-digest-vs-xorq-staleness.md) -— why `result_digest` can't reuse xorq's content-aware cache staleness. +(why `result_digest` cannot reuse xorq's cache staleness). Both have a status +note for what #189 changed. Older, historical: [eda-prompt-research.md](../plans/eda-prompt-research.md), @@ -483,23 +1017,25 @@ Older, historical: [plotting-testcases.md](../plans/plotting-testcases.md), [xorq-sklearn-assessment.md](../plans/xorq-sklearn-assessment.md), [ds-demo-scripts.md](../plans/ds-demo-scripts.md), -[demo/datasets.md](../demo/datasets.md). These are historical; staleness mostly -doesn't apply, except where they assert current system behavior (a few reference -the removed `result.parquet` and the old `~/.tallyman/` path). +[demo/datasets.md](../demo/datasets.md), and the CSV importer research under +[docs/research/csv-importers/](research/csv-importers/). Some of them mention +the removed `result.parquet`, xorq's result cache and the old `~/.tallyman/` +path; `ds-demo-scripts.md` has a status note saying so. -### Root & meta +### Root and meta -- [README.md](../README.md) — V0 spike overview and run instructions. **Current.** -- [proposal.md](../proposal.md) — the talk pitch. **Current** (it's a pitch, not - a spec). +- [README.md](../README.md): what tallyman is, and how to run it. **Current.** +- [tallyman_explanation.md](../tallyman_explanation.md): the owner's framing of + tallyman, a feature list and a walkthrough. **Current** in the sections + written from the code; the owner's own sections are his and are not checked. +- [proposal.md](../proposal.md): the talk pitch. **Current** (a pitch, not a + spec). -The original `plan.md` (V0.6 plan) and `TICKETS.md` (V0 punchlist) were removed -as stale cruft — they predated the React SPA migration, the `result.parquet` -removal, and the JSONL catalog format. This doc supersedes them as the -architecture reference. +The original `plan.md` (V0.6 plan) and `TICKETS.md` (V0 punch list) were removed +as stale: they predated the React SPA, the `result.parquet` removal and the +JSONL catalog format. This doc replaces them as the architecture reference. > Gaps: there is no dedicated reference for the REST API, the CLI, or the -> frontend SPA architecture. (The MCP tool surface and the authoring extension -> points it exposes — display klasses, summary stats, post-processing — are now -> covered in [mcp-server.md](mcp-server.md).) See the project's gap tracking for -> the current list. +> frontend. The MCP tool surface, including the extension points it exposes +> (display klasses, summary stats, post-processing), is covered in +> [mcp-server.md](mcp-server.md). diff --git a/docs/caching.md b/docs/caching.md index 4c31c85..3bb3452 100644 --- a/docs/caching.md +++ b/docs/caching.md @@ -13,11 +13,19 @@ results are tallyman's own. The xorq section is here because Buckaroo's stat cache (the summary statistics Buckaroo computes for each column of a grid) is built on it, and because it explains what the design replaced. +Terms follow [architecture.md](architecture.md#terms). An **entry** is one +catalog computation, stored under its **content hash**, xorq's hash of the +entry's expression. Every file the expression reads is a snapshot named by the +content hash of the entry it holds, and a **source entry** (one version of an +imported file) is hashed from the md5 of its bytes and its reader options, so +the hash also covers the bytes of every input and the identity of every parent. + The dominant pattern everywhere is content-addressing: keys are derived from immutable inputs (an expression's structure, an entry's content -hash, a source file's digest), so entries never go stale and "invalidation" +hash, an imported file's digest), so entries never go stale and "invalidation" is usually a space decision, not a correctness one. The exceptions are -called out as they appear and collected at the end. +called out as they appear and collected at the end. Where an open issue says +tallyman does not yet behave as described, the paragraph says so and cites it. ## xorq: the expression cache @@ -52,9 +60,9 @@ Snapshot's blindness to data changes is deliberate (reproducible build artifacts keyed on what the expression *is*, not on what the source file happened to contain), but it is the one real footgun in the stack: `ParquetSnapshotCache`'s own docstring notes it does not re-key when -source files change. Tallyman puts a source's content in the path of the -file a recipe reads, so xorq's path-only keys are content-honest without -tallyman using either strategy; see below. +source files change. Tallyman names every file a recipe reads by a content +hash, so xorq's path-only keys are content-honest without tallyman using +either strategy; see below. ### Storage backends @@ -82,14 +90,15 @@ would mask filesystem changes within one long-lived process. ## tallyman -Tallyman writes its own result files. It decides at build time whether an -entry is worth materializing, writes the file once when the entry is created, -makes sure every file an entry reads exists before anything executes, and -reads sources through copies that carry a row-order column. xorq's own -cache is not in that path. ADR-007 (`plans/ADR-007-tallyman-owned-materialization.md`) -records the problems that led there: cache files landing under -`~/.cache/xorq` outside the project, a writer that races on a fixed temporary -file, and a build that re-ran an expensive parent whenever a child was built. +Tallyman writes its own result files. It decides at build time whether an entry +is worth materializing, writes the file once when the entry is created, makes +sure every file an entry reads exists before anything executes, and lets a data +file in only by an import, as a source entry whose snapshot carries a row-order +column. xorq's own cache is not in that path. ADR-007 +(`plans/ADR-007-tallyman-owned-materialization.md`) records the problems that +led there: cache files landing under `~/.cache/xorq` outside the project, a +writer that races on a fixed temporary file, and a build that re-ran an +expensive parent whenever a child was built. ### Worthy and cheap entries (`src/tallyman_xorq/worthiness.py`) @@ -114,17 +123,21 @@ An entry is cheap only if all of these hold: Everything else is worthy: an aggregate, join, sort, limit, union, distinct, sample, and any operation nobody has considered yet. The test is an allow-list -because a cheap entry pages by its parent's `__row_order` (see Row order -below), so a wrong "cheap" gives unstable paging and a wrong "worthy" costs a -copy. A UDF is matched by its base class, so a scalar UDF makes an entry +because a cheap entry pages by the `__row_order` of the one file it reads (see +Row order below), so a wrong "cheap" gives unstable paging and a wrong "worthy" +costs a copy. A UDF is matched by its base class, so a scalar UDF makes an entry worthy without any other expensive operation (#81). ### Snapshots (`src/tallyman_xorq/materialize.py`) -A worthy entry's **snapshot** is `/result_cache/.parquet` -(`snapshot_path`), so its name is a function of the content hash and nothing -else. `materialize` is the one writer of snapshots, used by the build and by -every heal (a **heal** re-creates a snapshot that is missing from disk): +A worthy entry's **snapshot** is +`/result_cache/.parquet` (`snapshot_path`), so its +name is a function of the content hash and nothing else. `materialize` writes +the snapshot of every computed entry, used by the build and by every heal of one +(a **heal** re-creates a snapshot that is missing from disk); a source entry's +snapshot is written by the import's writer (`source_import._write_snapshot`), at +import and when it is healed from its clone (see "Source entries" below). +`materialize`: - it loads the entry's frozen build and rebinds every backend in it onto a fresh single-partition connection (`single_partition_backend`, with @@ -139,21 +152,42 @@ every heal (a **heal** re-creates a snapshot that is missing from disk): - it writes to a unique temporary name in the destination directory and then replaces the destination with `os.replace`, under the project's write lock. -A create always runs the query and replaces whatever file is at the path. A -create of a worthy entry runs it twice through the writer and compares the -content digests of the two files (`check_reproducible`), since at create time -nothing is recorded to compare against. If they differ the build still -succeeds, and the manifest records `reproducible: false` with the columns that -differed. The file is then **pinned**: the Cache page will not delete it, since -it cannot be re-created faithfully. A heal runs the query once and checks the -result against the recorded digest. - -The row-group size and the batch size decide the batch boundaries that an -entry built on the file sees, and an ungrouped float total depends on them, so -they are part of the reproducibility contract. `SNAPSHOT_FORMAT_VERSION` -stands for both, and for the layout of ordered copies; the manifest records it -as `snapshot_format`, next to the xorq, xorq-datafusion and pyarrow versions in -`engine_versions`. Changing any of them is a corpus rebuild. +A create always runs the query, and what it writes replaces whatever file is at +the path. A create of a worthy entry runs it twice through the writer and +compares the content digests of the two files (`check_reproducible`), since at +create time nothing is recorded to compare against. If they differ the build +still succeeds, and the manifest records `reproducible: false` with the columns +that differed. The file is then **pinned**: the Cache page will not delete it, +since it cannot be re-created faithfully. A heal runs the query once, replaces +the file at once, and checks the result against the recorded digest. + +A create does not replace the file straight away. It calls +`materialize(..., publish=False)`, which leaves the finished file at its +temporary name, and the build moves it into place (`publish_snapshot`) as its +last step, after the manifest is written. A build that fails before then removes +only its temporary file, so a file already at the path is kept, such as one a +reset left on disk, which for an entry that is not reproducible is the only copy +of its rows (#193, fixed in #222). A source entry needs no such staging: the +import keeps a snapshot already at its path, which holds the same rows, since +the path is named by the bytes and the reader options. + +**Pins.** `pinned_reason` decides from the entry's manifest and, for a source +entry, whether its clone is on disk. A snapshot is pinned when the manifest says +`reproducible: false`, when it holds `unfaithful_heal_digest` (the digest an +unfaithful heal wrote, below), or when it is a source entry whose clone is gone +(for a retired one, from both `data/.cas/` and the bullpen). Because the first +two are part of the manifest, the pin moves with the entry through a reset and +survives the error banner's dismiss, which deletes `errors.jsonl` (#196, fixed +in #223). For a file whose entry a reset retired, the manifest parked in the +bullpen speaks for it (`snapshot_manifest`), so the pin holds while the entry is +retired (#195). + +The row-group size and the batch size decide the batch boundaries that an entry +built on the file sees, and an ungrouped float total depends on them (#187), so +they are part of the reproducibility contract. `SNAPSHOT_FORMAT_VERSION` stands +for both, and for the row groups of a source entry's snapshot; the manifest +records it as `snapshot_format`, next to the xorq, xorq-datafusion and pyarrow +versions in `engine_versions`. Changing any of them is a corpus rebuild. `result_digest` in the manifest is `arrow-sha256:`, a SHA-256 over the file's Arrow data read back (`src/tallyman_xorq/digest.py`). It does not depend @@ -163,45 +197,66 @@ on every value, on which slots are null, on the order of the rows, and on the column names and types. Per-column digests name the columns that differ between two runs. -### Ordered copies of sources (`src/tallyman_xorq/ordered_copy.py`) - -A recipe never reads a source file directly. `read_project_file` and -`tallyman_read_csv` take the source's content digest (md5, memoized on the -file's stat). In the default `cas` mode they also clone the source -copy-on-write to `/data/.cas/`, the **clone**. Then -polars writes an **ordered copy**: `/ordered_sources/.parquet`, -holding the source's rows in file order plus a last column `__row_order`, in -row groups of 122,880 rows. The recipe reads the copy. `` is an md5 of the -digest and the reader options (for a CSV, the schema and the `scan_csv` -options), so the copy's name, and with it the entry's content hash, follows -the source's content in every identity mode. Editing a source and running the -same recipe forks a new entry, and the old entry keeps the rows it was built -from (#168). - -The manifest records each copy under `ordered_copies`: the source path, its -digest, the reader options, and the content digest of the copy when it was -first written. That record is what lets `ensure_materialized` make a deleted -copy again. A `.digest` file next to a copy caches its content digest. - -`TALLYMAN_SOURCE_IDENTITY` (default `cas`) now decides what else is recorded. -`cas` keeps the clone and records `manifest.sources`. `salt` records -`manifest.sources`, mixes the digests into the entry hash, and reads the live -file when it writes a copy. `off` records no `sources`, so the source axis of -staleness reports `unknown`, and also reads the live file. `rewrite_for_build` -no longer branches on the mode. - -A recipe that calls `xo.deferred_read_parquet` on a file that is not under -`compute_cache/`, or `xo.deferred_read_csv`, is a build error: such a read gets -no digest, no clone and no ordered copy. +### Source entries (`src/tallyman_xorq/source_import.py`) + +A recipe never reads a data file. A file enters the catalog only through +`catalog_import_source` (`update_and_depend`), which makes it a **source +entry**, a version of a **source alias** (ADR-011, +`plans/ADR-011-sources-are-aliases.md`). The import: + +- digests the file (md5) and clones its bytes, copy-on-write where the + filesystem supports it and a plain copy elsewhere, to + `/data/.cas/`, the **clone**. The copy is digested + again after it is written and refused if it does not match its name; +- writes the entry's snapshot, `/result_cache/.parquet`, + holding the file's rows in file order plus a last column `__row_order`. A + parquet file is copied by pyarrow, so its column types survive; a CSV is parsed + by polars under the schema and `scan_csv` options named in the import call, + and its batches go to the same pyarrow writer. Either way the file has the + pinned layout of every snapshot, in row groups of 122,880 rows; +- writes the entry: a generated recipe, a frozen build, a schema, and a manifest + whose `provenance` records the outside path, the digest, the reader options and + the name it was imported as. + +The entry's content hash is `md5("source||")`, cut to +12 hex characters, so it is a function of the bytes and the reader options and +nothing else. The reader options are fixed at import and must be plain values: a +callable, whose `repr` would carry a memory address, is refused. The outside path +is provenance and is never read again, so editing or deleting the original file +changes nothing. New data arrives by importing again under the same alias, which +mints the next version under a new hash; the old versions keep their rows. + +A source entry is worthy, and its snapshot is cache in the sense of ADR-007 D13 +(a file is cache only if it can be made again): the clone holds the bytes and the +manifest holds the reader options, so a deleted snapshot is written again from +the clone (`_heal_a_source`) and checked against the recorded `result_digest` +like any other heal. Nothing in a read writes a clone. With the clone gone, the +snapshot is the last copy of those rows, so `pinned_reason` pins it; if the +snapshot is gone as well, the read fails with an error naming the missing clone +and the `catalog_import_source` call, reader options included, that restores +it. Importing the same bytes again under the alias that holds them rewrites +nothing of the entry. When the snapshot is gone, it writes the clone back from +the given file, verified against the digest, and heals the snapshot. When the +snapshot is still there, it leaves a lost clone lost, so the version stays +pinned (#239). + +Every other way of reading a file is a build error: `read_project_file`, +`tallyman_read_csv`, `xo.deferred_read_csv`, and `xo.deferred_read_parquet` of a +file outside `compute_cache/`. `read_project_file` is still called by the recipe +the importer generates, where a context variable (`_SOURCE_ENTRY`) resolves it to +the entry's own snapshot. ### Reads (`cached_result_expr` and `ensure_materialized`) -Every consumer reads an entry's result through one function, -`cached_result_expr`, whose internals are the canonical read of -`docs/system-contract.md` (#163): the entry's frozen `xorq_build/` is expanded -and loaded, and reads never re-import `expr.py`. A missing or unloadable build -is a hard error naming the entry. `cached_result_expr` calls -`ensure_materialized(project, content_hash)` and then returns: +Every in-process consumer (page reads, charts, diffs, post-processing, a child +recipe) reads an entry's result through one function, `cached_result_expr`, +whose internals are the canonical read of `docs/system-contract.md` (#163): the +entry's frozen `xorq_build/` is expanded and loaded, and a read never re-imports +`expr.py`. (Only the attribution of an unfaithful heal re-imports it, as a +diagnostic.) Buckaroo's grid is handed a build instead; see "Buckaroo +integration caches" below. A missing or unloadable build is a hard error naming +the entry. `cached_result_expr` calls `ensure_materialized(project, +content_hash)` and then returns: - for a worthy entry, one bare `deferred_read_parquet` of the snapshot on the default backend, memoized per `(project, content_hash)`. When the file exists @@ -212,31 +267,53 @@ is a hard error naming the entry. `cached_result_expr` calls its own snapshot, is on disk before anything executes: 1. A worthy entry whose snapshot exists is done, and no build is loaded. -2. Otherwise it loads the frozen build and collects the file each `Read` node - points at. -3. It re-creates each missing file by the rule for its class (table below). - When nothing can, the error names the source file. +2. A source entry whose snapshot is missing is healed from its clone, and + nothing else is needed: its own build reads the very file that is missing. +3. Otherwise it loads the frozen build and collects the file each `Read` node + points at. Every one is another entry's snapshot, and a missing one is made + again by recursing on the hash in its name. 4. For a worthy entry it then heals the entry's own snapshot, under the project lock and after re-checking that the file is still missing, and verifies it. +Whether an entry is worthy comes from its manifest (`cache_worthy`), and nothing +stands in for it: a file at the snapshot path says nothing about the verdict. An +entry directory with no manifest, which a build or import killed before its last +write leaves, is refused. `result_cache.entry_manifest` raises a `BuildError` +naming the missing `manifest.json` before anything is loaded or written, a child +whose build reads that entry's snapshot raises the same error, and running the +recipe again (or importing the file again, for a source entry) writes the entry +again under the same hash. + | File | Written by | If it is missing | |---|---|---| -| Snapshot, `compute_cache/result_cache/.parquet` | `materialize` | re-run the entry's build, and verify the digest | -| Ordered copy, `compute_cache/ordered_sources/.parquet` | polars, at ingest | re-run ingest on the clone with the recorded reader options, and check it | -| Clone, `data/.cas/` | `ensure_cas_path` | copy the live source again, but only while its bytes still hash to the digest | +| Snapshot of a computed entry, `compute_cache/result_cache/.parquet` | `materialize` | re-run the entry's build, and verify the digest | +| Snapshot of a source entry, same directory | the import | parse the clone again with the recorded reader options, and verify the digest; with the clone gone too, raise an error naming the clone and the import that repairs it | +| Clone, `data/.cas/` | the import (`ensure_cas_path`) | nothing in a read makes it again; the snapshot is pinned while it exists; a re-import of the same bytes writes the clone back only when the snapshot is gone too (#239) | -A healed snapshot is checked against the recorded `result_digest`. A mismatch -is still served, since the rows are the honest output of the frozen build, but +A healed snapshot is checked against the recorded `result_digest`. A mismatch is +still served, since the rows are the honest output of the frozen build, but never silently (`_verify_self_heal`). It logs a warning that attributes the change: an engine version that differs from the manifest's `engine_versions`, a recipe that re-derives a different graph hash (#88), or a fixed graph that runs -differently each time (#83). It records a durable `unfaithful_heal` error, -which also pins the file, wipes the entry's Buckaroo stat cache, and fires the -registered hooks. In the companion the hooks force Buckaroo to reload the open -grid and push an SSE event. A re-created ordered copy is checked the same way -against its recorded content digest, and a mismatch records an -`unfaithful_ordered_copy` error. +differently each time (#83). It records the digest it wrote in the manifest's +`unfaithful_heal_digest`, which pins the file, records a durable +`unfaithful_heal` error for the error banner, wipes the entry's Buckaroo stat +cache, and fires the registered hooks. That field is the only one written after +create: an unfaithful heal is the one thing that rewrites a manifest, +atomically, under the heal's lock. In the companion the hook posts a forced +reload of the entry's grid to Buckaroo and publishes an `unfaithful_heal` SSE +event. The SPA has no listener for that event; the error appears in the catalog +page's error banner the next time the page refetches. All of this runs while the +heal still holds the project lock, and the forced reload is posted whether or +not a grid is open, without a promoted diff's colouring (#203). Only the healed +entry is flagged: a cheap child of it reads the same snapshot, so the child's +rows change under its hash with no record and no reload (#208). The MCP server +registers no hook, so a heal that runs there records the pin and the error and +wipes the stat cache only. A source entry's snapshot made again from its clone +goes through the same check, so if a reader now parses the bytes differently, +the difference is recorded as an unfaithful heal and the manifest keeps the +digest of the rows that were imported. Both shapes are single-backend expressions, so two entries compose (`union`, `join`, a diff) without tripping xorq's "multiple backends" guard. Chaining @@ -246,10 +323,11 @@ snapshot by its literal path, which contains the parent's content hash: the child's identity is a function of the parent's, and does not change when the file's bytes do. The parent's snapshot is made to exist before the child is built, since a bare read cannot be composed over a missing file. A cheap -parent's graph is inlined instead. The child's manifest records every source -digest its parent recorded, and the parent's ordered-copy records as well when -the parent is cheap, because the child's build then reads those copies itself. -The viewer's paginated reads, diffs, and post-processing all go through +parent's graph is inlined instead. The child's manifest records only its parent +edges: a source version is an entry, so the parent edges are the whole record of +what the child depends on (ADR-011 D6 deleted `manifest.sources`, the per-file +digest map children used to inherit). The viewer's paginated reads, diffs, and +post-processing all go through `cached_result_expr`. ### Row order (`src/tallyman_xorq/row_order.py`) @@ -258,9 +336,11 @@ Every file tallyman writes ends in an int64 `__row_order` column holding `0..N-1` in the file's physical row order, and every page request sorts by it: `ORDER BY __row_order`, or the user's sort keys and then `__row_order`, so the same request returns the same rows in any process and any cache state -(`row_order.page`, which `/api/data` uses). Buckaroo pages in its own process, -and is told the column's name with `row_order_column` in the `/load_expr` -payload. The build enforces what makes that safe: +(`row_order.page`, which `/api/data` uses; that route takes no user sort yet). +Buckaroo pages the grid in its own process. Tallyman tells it the column's name +with `row_order_column` in the `/load_expr` payload, but Buckaroo 0.15.6, the +pinned version, ignores the hint, so the grid's pages are not yet ordered by it +(buckaroo-data/buckaroo#974). The build enforces what makes the rule safe: - A cheap entry has no file of its own and pages by its parent's column, so it must keep it. A select that omits it fails the build with the corrected @@ -273,54 +353,96 @@ payload. The build enforces what makes that safe: sortable columns, appended as tie-breakers, so a sort that feeds a `limit` decides the same rows on any connection. A sort that is not the last step is hoisted: the top-level sort leads with its keys. If a key was dropped or - overwritten, the build fails and names it. + overwritten, the build fails and names it. Nested columns (lists, structs, + maps) are left out of the tie-break, so two rows that tie on every other + column can still come out in either order (#205). - Every entry carries the column, and ibis names a join's right-hand copy `__row_order_right`, so joining three entries in one recipe needs - `.drop("__row_order")` on the right-hand inputs. The snapshot writer drops - `__row_order_right`, so a join entry can be joined again. A diff drops the - column from both sides, and the primary-key search skips it. + `.drop("__row_order")` on the right-hand inputs. The check counts right-hand + inputs without looking at the join kind, so a chain of semi or anti joins, + which adds no right-hand columns, is refused too (#199). The snapshot writer + drops `__row_order_right`, so a join entry can be joined again; it drops any + column of that name, including one the author made, and a join with a + non-default `rname` leaves its own collision column behind (#206). +- The diff grid's compare expression drops the column from both sides, but + `full_diff`, which computes the diff page's summaries and `catalog_diff`'s + reply, keeps it as a data column (#200). The primary-key search skips it. ### Compute cache (`compute_cache_dir` in `src/tallyman_core/paths.py`) -`/artifacts/catalog/compute_cache/` holds two directories of files -that tallyman writes: `result_cache/` (snapshots) and `ordered_sources/` -(ordered copies and their `.digest` files). By rule everything in it is cache: -a file lives here only if `ensure_materialized` can re-create it and check what -it made, so the cold state is an empty `compute_cache/`, and reading any entry -then re-creates every file it needs. +`/artifacts/catalog/compute_cache/` holds one directory of files that +tallyman writes: `result_cache/`, the snapshots of worthy entries, source +entries included. By rule everything in it is cache: a file lives here only if +`ensure_materialized` can re-create it and check what it made, so the cold state +is an empty `compute_cache/`, and reading any entry then re-creates every file +it needs. Three kinds of snapshot are exceptions, and are pinned (see "Pins" +above): the snapshot of an entry recorded as not reproducible, which cannot be +made again faithfully; one whose heal already produced different rows than were +built (`unfaithful_heal_digest`); and the snapshot of a source entry whose clone +is gone, which is the last copy of the imported rows. A pin protects the file +from the Cache page and from nothing else (#185). Files are deleted only by an explicit user action, and written only because -something is about to read them. The startup warm-up writes nothing, the verify -sweep (`catalog_scan_staleness(verify_results=True)`) reads and never writes, -and a reset leaves the directory alone. The Cache page's delete is the one -deleter. It answers 409 with the reason for a pinned snapshot, and it lists a -snapshot whose entry is no longer in the catalog as an orphan row so that it -can be deleted. - -Every write takes the project's write lock (`catalog_state.project_lock`): a -build, a materialization, a heal and a checkpoint. It is a file lock on -`.checkpoint.lock`, so it holds between the MCP server and the companion, and -it is re-entrant within a thread. It covers writes only. Concurrent reads on -the shared default backend can still raise `Already borrowed` (#118). +something is about to read them. The startup warm-up writes nothing here, the +verify sweep (`catalog_scan_staleness(verify_results=True)`) reads and never +writes, and a reset leaves the directory alone. The Cache page's delete is the +one deleter. The page answers 409 with the reason for a pinned snapshot. It +lists a snapshot whose entry a reset retired as a `retired` row, whose pin comes +from the manifest parked in the bullpen, and a snapshot that no entry names, +live or retired, as an `orphan` row, so that the user can delete either. A +source entry's snapshot is listed like any other. + +The project's write lock (`catalog_state.project_lock`) is taken by a build +(for its whole length), an import, a materialization or heal, a checkpoint and +a reset. Other writes take no lock: alias and notebook updates, +chart and display configs and `config.json` each replace a whole file +atomically, the logs append a line, and the Cache page's delete unlinks the +file. The lock is a file +lock on `.checkpoint.lock`, so it holds between the MCP server and the +companion, and it is re-entrant within a thread. It blocks with no timeout: a +page read whose entry needs a heal waits behind any build in the other process +(#186), and the companion's `PUT /code` and `POST /promote_diff` routes build on +the event loop, so the whole UI stops answering while they wait or build (#190). +It covers no reads. Executions have a lock of their own, +`execution.execution_lock`, one re-entrant lock per process, held around every +execution on the process's shared default backend, which fails with +`RuntimeError: Already borrowed` when two threads execute on it at once. A heal +runs before that lock is taken, since the project lock comes first, and +`project_lock` raises in a thread that holds the execution lock and would take a +new file lock. A materialization's stream and a cheap entry's row count at build +run on connections of their own and take no execution lock. `reset_to` (`src/tallyman_core/catalog_state.py`) returns the catalog with `git reset --hard` and reconciles entry directories through the bullpen (the directory a reset moves retired files into, so that a later reset forward can -bring them back), but it does not manage `compute_cache/` (ADR-007 D14, a reset -leaves `compute_cache/` alone). Snapshots are named by content hash, so one left behind by a retired -entry cannot be served for another entry: it is unreferenced disk until that -entry comes back or the user deletes it. A clone is data, the only frozen copy -of the bytes an entry was built from once the live file is edited, and -`/data/.cas` lives outside the catalog git repo. So `reset_to` moves -the clones no surviving entry's `manifest.sources` refers to into -`/bullpen/cas/` and never deletes one (`source_identity.gc_cas` with a -bullpen), and a reset forward copies them back. `compute_cache.jsonl` no -longer exists. +bring them back), but it does not manage `compute_cache/` (ADR-007 D14, a +reset leaves `compute_cache/` alone). Snapshots are named by content hash, so +one left behind by a retired entry cannot be served for another entry: it is +unreferenced disk until that entry comes back or the user deletes it. A clone +is data, the only copy tallyman has of bytes it imported (the outside file is +never read again), and `/data/.cas` lives outside the catalog git repo. +So `reset_to` moves the clones that no surviving source entry's +`manifest.provenance` names into `/bullpen/cas/` and never deletes one +(`source_identity.gc_cas` with a bullpen), and a reset forward copies back the +clones a restored entry names. A reset that cannot read every surviving +manifest skips the sweep. `compute_cache.jsonl` no longer exists. + +When a reset retires an entry whose directory the bullpen already holds (an +entry retired once, restored, and retired again), the live directory replaces +the parked one, since it is the one that agrees with the snapshot on disk: a +create always rewrites the snapshot, and an entry that is not reproducible +records another `result_digest` each time (#194, fixed in #223). A live +directory with no manifest, which an interrupted build leaves, never replaces a +parked one and is dropped instead. The bullpen has one live reader besides +`reset_to`: the Cache page reads a retired entry's parked manifest to decide its +snapshot's pin, and for a retired source version it counts a clone parked in +`bullpen/cas/` as present, since a reset forward brings the entry and the clone +back together (#195). ### In-memory caches in the companion -All bounded LRUs over immutable keys, so eviction means a cheap rebuild -and staleness is impossible: +The first two are bounded LRUs over immutable keys, so eviction means a cheap +rebuild and staleness is impossible; the third is a short TTL: - `cached_result_expr` — two memos. `_resolve_result_plan` is `lru_cache(256)` keyed `(project, content_hash)`: it holds a loaded build, @@ -329,7 +451,9 @@ and staleness is impossible: snapshot, so one read has one table name in the shared backend. `cached_result_expr.cache_clear()` clears both. Whether each file exists is checked on every call, inside `ensure_materialized`, since existence is the - one input that stays mutable. + one input that stays mutable. The plan memo also keeps the build as it was + loaded, which nothing reads again, so up to 256 unused DataFusion backends + stay in memory (#210). The MCP server has the same two memos. - `_build_compare_expr` — `lru_cache(128)` keyed `(project, a_hash, b_hash, keys)`; saves rebuilding diff outer-join expressions. Build dirs land under `$TMPDIR/tallyman_diff_builds/`. @@ -338,10 +462,11 @@ and staleness is impossible: time-based cache in tallyman; it coalesces filesystem walks during SSE bursts. -On companion startup a 3-second warmup budget loads the frozen builds of cheap -entries into the plan memo, so the first page request does not pay for the -load. A worthy entry is skipped, since reading it does not load its build, and -the warmup writes no file. +On companion startup a 3-second warmup budget loads the frozen builds of the +active project's cheap entries into the plan memo, so the first page request +does not pay for the load. A worthy entry is skipped, since reading it does not +load its build. The warmup writes nothing under `compute_cache/`; loading a +cheap entry's build can write its `.xorq_build_expanded/`. ### Buckaroo integration caches (`src/tallyman_companion/buckaroo_lifecycle.py`) @@ -351,62 +476,84 @@ Buckaroo process, and its id is `entry--` (`BuckarooManager.session_id_for`), a function of the two and nothing else, so it is never stale. There is no session file. -- **Opening a grid** — `load_session` calls `ensure_materialized` first, so a +- **Opening a grid**: `load_session` calls `ensure_materialized` first, so a failure of the computation surfaces in tallyman and never inside a grid query, then POSTs `/load_expr` with the derived id every time. Buckaroo skips the work when it already holds a session with that id and the same build directory and the post carries none of `component_config`, `column_config_overrides`, `extra_grid_config`, `init_sd` or `skip_stat_columns`. It creates the session again if it has dropped it, as it - does after a session has been idle for an hour. A promoted diff entry sends - `column_config_overrides`, so it reloads on every open. -- **What Buckaroo is handed** — for a worthy entry, a **view build**: a build + does after a session has been idle for an hour. Nothing coalesces two opens + of one entry, so opens that overlap both post and both run Buckaroo's + pipeline; a promoted diff entry sends `column_config_overrides`, so it + re-runs the pipeline on every open; and the notebook page's data route posts + for every cell on every page load (#202). +- **What Buckaroo is handed**: for a worthy entry, a **view build**, a build whose whole graph is one read of the entry's snapshot, written once to `/.xorq_view_build/` and rebuilt if the snapshot's path changes (a sibling `.xorq_view_build.complete` file records the path it was made for). Buckaroo never executes an aggregate, join or sort on tallyman's behalf and never writes a snapshot. For a cheap entry, the entry's own expanded build, - a small plan over files that exist. Both directories are stable, because - Buckaroo's stat-cache keys include the build directory's path. The body also - carries `row_order_column`, the name `__row_order`. -- **After an unfaithful heal** — the companion's hook wipes the entry's stat - cache and POSTs `/load_expr` for the entry's id with `force_reload: true`, so - an open grid does not keep stats computed from the old rows. -- **After a klass change** — a klass is a summary stat, post-processing or + a small plan over files that exist. Both directories are stable per-entry + paths, so Buckaroo is handed the same build after a restart, which is what its + on-disk stat cache relies on. The body also + carries `row_order_column`, the name `__row_order`, which Buckaroo 0.15.6 + ignores, and `project_root`, where Buckaroo looks for the project's `stats/`, + `post_processing/` and `display/` klasses. Tallyman sends `artifacts/`, + which holds `display/`, but it writes stats and post-processing functions + under `artifacts/catalog/`, so Buckaroo does not find them (#170). +- **After an unfaithful heal**: `_verify_self_heal` wipes the entry's stat + cache, and the companion's hook POSTs `/load_expr` for the entry's id with + `force_reload: true`, so an open grid does not keep stats computed from the + old rows. The post goes out whether or not a grid is open, which opens a + session nobody asked for, and it carries no column colouring (#203). +- **After a klass change**: a klass is a summary stat, post-processing or display class written for the project. `reload_project_sessions` POSTs - `/reload_expr/` for every entry of the project, treats the 404 that - Buckaroo answers for an unknown session as "not open", and clears the stat - cache of each grid it reloaded. That is one request per entry per change. -- **Per-entry stat cache** — `/.buckaroo_stat_cache/parquet/`, a + `/reload_expr/` for every entry of the project, treats the 404 (or 400) + that Buckaroo answers for an unknown session as "not open", and clears the + stat cache of each grid it reloaded. That is one request per entry per + change, sent one after another from the companion's event loop (#201), and + the stat-cache wipe after each reload is more than a klass change needs + (#177). A reset or a recalc that moved an alias reloads sessions the same way. +- **Per-entry stat cache**: `/.buckaroo_stat_cache/parquet/`, a `ParquetSnapshotCache` Buckaroo writes its summary stats into (the companion passes the path at `/load_expr` time). Deleted wholesale on stat reload so Buckaroo recomputes. Always on, orthogonal to whether an entry - is worthy or cheap. -- **Diff stat cache** — `/artifacts/catalog/diff_stat_cache/` + is worthy or cheap. In practice a first load after a Buckaroo restart has not + been seen to hit it (#157). +- **Diff stat cache**: `/artifacts/catalog/diff_stat_cache/` `{a_hash[:12]}-{b_hash[:12]}/`, the same idea for a comparison session, keyed by the entry pair. Re-opening the same diff reuses the per-column - stats instead of recomputing over the full join. The live diff still posts an - unmaterialized join to Buckaroo (ADR-007 D10, tracked in #188), and its - session ids are remembered in memory and cleared when `/health` reports a new + stats instead of recomputing over the full join. A reset or a recalc that + moved an alias deletes the whole directory. The live diff still posts an + unmaterialized join to Buckaroo (ADR-007 D10, which would have built every + diff as an entry first, was moved to #188). Its session ids, + `diff--`, are remembered in the companion's memory and + forgotten when the companion starts a Buckaroo whose `/health` reports a new `started` timestamp. ### Per-entry immutable records -Not caches in the eviction sense — immutable build outputs, valid forever -because the entry they describe never changes. `paths.py` draws the line -explicitly: `ENTRY_ARTIFACT_NAMES` (`xorq_build/`, `manifest.json`, -`schema.json`) are the immutable artifacts; the write-isolated perf overlay -symlinks them read-only — safe because nothing ever rewrites them — and omits +Not caches in the eviction sense — build outputs, valid forever because the +entry they describe never changes. `paths.py` draws the line explicitly: +`ENTRY_ARTIFACT_NAMES` (`xorq_build/`, `manifest.json`, `schema.json`) are the +artifacts; the write-isolated perf overlay symlinks them read-only — safe +because the only later write, an unfaithful heal recording +`unfaithful_heal_digest` in `manifest.json`, is an atomic replace that swaps +the overlay's link for a file and leaves the original alone — and omits `ENTRY_CACHE_NAMES` (`.buckaroo_stat_cache`, `.xorq_build_expanded`, `.xorq_view_build`) so a benchmark starts honestly cold. -The entry directory itself is gitignored. Its durable, git-tracked form is the -recipe zip `entries/.zip` — a deterministic archive of `expr.py`, -`xorq_build/`, `manifest.json`, and `schema.json`, written by the checkpoint -(`tallyman_core/catalog.py`). So the build dir is untracked-but-durable: the -recipe zip carries it across a clone, and `entries.jsonl` records which dirs -should exist so `reset_to` can reconcile them from the bullpen. (See the native -catalog store, `catalog.py` / `catalog_state.py`, for the full tracked surface.) +The entry directory itself is gitignored. What the catalog repository tracks for +it is the recipe zip `entries/.zip`, a deterministic archive of `expr.py`, +`xorq_build/`, `manifest.json`, and `schema.json`, written once, by the first +checkpoint after create (`tallyman_core/catalog.py`), so its manifest does not +carry an `unfaithful_heal_digest` recorded later. Nothing reads the zip back +(`catalog.py`): it records what a checkpoint committed, and a clone of the +catalog repository does not recreate entry directories from it. `entries.jsonl` +records which dirs should exist so `reset_to` can reconcile them from the +bullpen. (See the native catalog store, `catalog.py` / `catalog_state.py`, for +the full tracked surface.) - **Primary key** (`src/tallyman_xorq/primary_key.py`) — `/primary_key.json` saves a full-table cardinality scan. A cheap @@ -416,23 +563,31 @@ catalog store, `catalog.py` / `catalog_state.py`, for the full tracked surface.) - **Portable build expansion** (`src/tallyman_xorq/portable.py`) — `/.xorq_build_expanded/` plus a `.complete` marker, written last, gates reuse; a crashed expansion redoes on next access. The - expansion must live at a stable path, because Buckaroo's stat-cache keys - include the build directory's path and a random tmp dir would miss them on - every process restart. (Content-addressed but regenerable, so it is an + expansion lives at a stable path so that Buckaroo is handed the same build + directory after every process restart, which its on-disk stat cache was + meant to rely on (#157). (Content-addressed but regenerable, so it is an `ENTRY_CACHE_NAME`, not an artifact — the overlay omits it rather than - symlinking it.) + symlinking it.) The marker does not record which project path the build was + expanded with, so a project copied together with these directories keeps + reading the old location, and fails once that is gone (#209). A worthy + entry's build is first expanded at create, when `materialize` loads it; a + cheap entry's the first time something loads it (a read, the grid, a child's + build, the companion's startup warm-up). - **Manifest / schema** — `/manifest.json`, `/schema.json`: row counts, timings (including #87's cache-admission fields: `compile_seconds`, `cache_worthy`, `cache_worthy_why`, `cache_bytes`), the `result_digest` (a content digest of the snapshot, worthy entries only), `reproducible` and `nonreproducible_columns`, `snapshot_format` and - `engine_versions`, the `ordered_copies` records, `sources` and `parents`. + `engine_versions`, `parents`, `unfaithful_heal_digest` once an unfaithful + heal has pinned the file, and on a source entry `provenance` (where the file + came from, its digest, the reader options and the name it was imported as). A worthy entry's schema is read from the file `materialize` wrote, which is why a `timestamp[s]` column is recorded as `timestamp[ms]`. Every entry's schema ends in `__row_order`. All of it is fixed at build so later reads don't re-walk the expression. - **Alias history** — `/aliases.jsonl`, one line per alias holding - its current head and the append-only version log (`aliases.py`). It is a + its current head, the append-only version log and its kind, `catalog` or + `source` (`aliases.py`). It is a git-tracked file in the catalog repo, not a per-entry artifact, so it rolls back with `git reset` on a `reset_to`. - **Per-hash config** — `/chart_specs/.vl.json` and @@ -499,57 +654,59 @@ catalog store, `catalog.py` / `catalog_state.py`, for the full tracked surface.) ## Invalidation, in one view Most of the stack never invalidates because it never can be stale: tallyman -entry hashes, snapshot paths (named by content hash), ordered copies (named by -a source's digest and its reader options), primary-key files, and manifests all -rely on "same key, same rows, forever". Cleanup for those is a space concern. -Only the user deletes a file, and a deleted file is made again and verified the -next time something reads it. +entry hashes (a source entry's named by its bytes and reader options), snapshot +paths (named by content hash), primary-key files, and manifests all rely on +"same key, same rows, forever". Cleanup for those is a space concern. +Only the user deletes a file, and a deleted file is made again and verified +the next time something reads it. One documented hole breaks "never stale": execution nondeterminism. An entry whose recipe calls `now()` / `random()` / an unseeded `sample()` or an impure -UDF produces different rows each run under one content hash, so a cold -recompute can disagree with what was built (#88). The build flags these as -advisory lint warnings (`_nondeterminism_warnings`, `build.py`), and a worthy -entry is also run twice when it is created, so a recipe that is not -reproducible is known from the start: the manifest records -`reproducible: false`, the build result names the columns that differed, and the -snapshot is pinned. That check cannot see `today()`, since both runs agree (the -lint does), or what an entry inherits from a non-reproducible parent, since both -runs read the same parent file. The runtime backstop is `result_digest`: every -heal verifies the repopulated snapshot against it before serving -(`_verify_self_heal`), and an unfaithful heal wipes the entry's stat cache, -records a durable error, pins the file, and forces Buckaroo to reload the open -grid. An engine or writer upgrade that changes results is attributed to the -versions in `engine_versions`, and the remedy is a corpus rebuild. The second -hole this section used to document — cold reads re-running `expr.py` and -re-digesting live sources, serving edited bytes under the original hash -(#115/#163) — is closed: reads load the frozen build, whose leaves are ordered -copies named by the source digest, bare reads of parent snapshots, and inlined -cheap parent graphs, so a cold read cannot see a post-build edit at all. +UDF produces different rows each run under one content hash, so a cold recompute +can disagree with what was built (#88). The build flags `now()`, `today()`, +`random()`, `uuid()` and an unseeded `sample()` as advisory lint warnings +(`_nondeterminism_warnings`, `build.py`); an impure UDF is not flagged, since +purity cannot be read off the graph. A worthy entry is also run twice when it is +created, so a recipe that is not reproducible is known from the start: the +manifest records `reproducible: false`, the build result names the columns that +differed, and the snapshot is pinned. That check cannot see `today()`, since +both runs agree (the lint does), or what an entry inherits from a +non-reproducible parent, since both runs read the same parent file (#185). The +runtime backstop is `result_digest`: every heal verifies the repopulated +snapshot against it before serving (`_verify_self_heal`), and an unfaithful heal +pins the file in its manifest, wipes the entry's stat cache, records a durable +error, and, in the companion, forces Buckaroo to reload the entry's grid. It +does not reach the cheap entries that read the healed snapshot, whose rows +change with it (#208). An engine or writer upgrade that changes results is +attributed to the versions in `engine_versions`, and the remedy is a corpus +rebuild. The second hole this section used to document — cold reads re-running +`expr.py` and re-digesting live sources, serving edited bytes under the original +hash (#115/#163) — is closed: reads load the frozen build, whose leaves are bare +reads of snapshots named by content hash and inlined cheap parent graphs, and a +data file enters only by an import, so a cold read cannot see a post-build edit +at all. Where staleness is actually possible, it is handled explicitly: - **mtime tracking** — Buckaroo's file cache invalidates automatically when the - source file changes, and tallyman's source-digest memo - (`source_identity.digest_for`) skips re-hashing a file whose mtime, size and - inode are unchanged. Content stays the truth: the staleness scan forces a - re-hash. -- **TTL** — xorq's `ParquetTTLStorage` (default one day) and the - companion's 3-second disk-usage cache are the only clocks in the - system. + source file changes. Tallyman keeps no stat-keyed memo: a file is digested + once, when it is imported, and the staleness scan reads no file at all. +- **TTL** — xorq's `ParquetTTLStorage` (default one day), the + companion's 3-second disk-usage cache, and Buckaroo's eviction of a session + idle for an hour are the only clocks in the system. - **State-change purges** — Buckaroo's op-chain key change, the JS `purgeInfiniteCache` on sort/ops change, the diff-session bookkeeping - clearing on Buckaroo restart, and the stat-cache deletion on a klass reload - or an unfaithful heal. + clearing on Buckaroo restart, the stat-cache deletion on a klass reload + or an unfaithful heal, and the deletion of `diff_stat_cache/` on a reset or + a recalc. The one rule to remember: a cache keyed on a path does not notice upstream data -changes, so tallyman puts the content in the path. In every identity mode the -file a recipe reads is an ordered copy named by the source's digest and reader -options. An edited source therefore forks the entry hash at build instead of -deduping to the stale entry, and reads — warm, cold, or healing — resolve -through the frozen build to the copy the entry was built from, never the live -file. `cas` also keeps the clone, the only frozen copy of the bytes once the -live file is edited, which is what lets `ensure_materialized` make a deleted -ordered copy again. Under `off` and `salt` there is no clone, so a deleted copy -can be made again only while the live file still has the recorded bytes; -otherwise the error names the source file. +changes, so tallyman puts the content in the path. Every file a recipe reads is +a snapshot named by a content hash, and a source entry's hash is a function of +the bytes it imported and the reader options. New data arrives only as an +import, which mints a new source version under a new hash and makes the entries +following that alias stale, and reads — warm, cold, or healing — resolve through +the frozen build to the snapshot the entry was built from, never the outside +file. The clone is what lets `ensure_materialized` make a deleted source +snapshot again; with the clone gone the snapshot is pinned, and once both are +gone the error names the clone and the import that restores it. diff --git a/docs/expression-lifecycle.md b/docs/expression-lifecycle.md index 035b748..b946c8a 100644 --- a/docs/expression-lifecycle.md +++ b/docs/expression-lifecycle.md @@ -7,14 +7,19 @@ Two walkthroughs: a brand-new expression (everything cold), then the same expression on a later visit (warm). It is a companion to `caching.md`, which maps the caches statically; this doc puts them on a timeline. -The one fact that makes the timeline make sense: an entry is written to disk -at *build* time, and so is the only materialized copy of its result. A -worthy entry's **snapshot** (a parquet file of its rows, which tallyman writes -itself) is written during the build, not on first read. Only two things are -populated *lazily*, at first view: Buckaroo's stat cache on first `/load_expr`, -and the view build directory a worthy entry's grid is handed. So "build" and -"first view" each write different things, and a restart re-runs the view-time -work against caches that survived on disk. +The one fact that makes the timeline make sense: an **entry** (one catalog +computation, stored under its content hash) is written to disk at *build* +time, and so is the only materialized copy of its result. A **worthy** entry, +one whose plan does expensive work such as an aggregate or a join, has a +**snapshot** (a parquet file of its rows, which tallyman writes itself), and +the snapshot is written during the build, not on first read. A **cheap** entry, +a filter or selection over one file, keeps no copy of its rows. The rest is +written lazily, when the entry is first read or viewed: Buckaroo's stat cache +on the first `/load_expr`, the view build a worthy entry's grid is handed, a +cheap entry's expanded build, and the primary key a diff joins on. So "build" +and "first view" each write different things, and a restart re-runs the +view-time work against caches that survived on disk. Other terms follow +[architecture.md](architecture.md#terms). ## Cast @@ -23,14 +28,15 @@ work against caches that survived on disk. | MCP server | `src/tallyman_mcp/server.py` | receives code + alias, calls the builder | | Builder | `src/tallyman_xorq/build.py` | imports the recipe, materializes a worthy entry, persists the entry | | xorq compiler | `build_expr` / `load_expr` | tokenizes to a content hash, runs the graph | -| Ingest | `src/tallyman_xorq/io.py`, `ordered_copy.py` | reads a source through its clone into an ordered copy that ends in `__row_order` | +| Import | `src/tallyman_xorq/source_import.py`, `source_identity.py` | `catalog_import_source`: clones a file's bytes and writes a source entry, whose snapshot ends in `__row_order` | +| Reads in a recipe | `src/tallyman_xorq/io.py` | `tracked_expr_from_alias`, `pinned_expr_from_alias`, and the refusals of a raw file read | | Classifier and rewrite | `worthiness.py`, `source_cache.py`, `row_order.py` | worthy or cheap, the canonical sort, the `__row_order` rules | | Materialization | `src/tallyman_xorq/materialize.py` | `materialize` writes snapshots; `ensure_materialized` makes every file an entry reads exist | | Reads | `src/tallyman_xorq/result_cache.py` | `cached_result_expr`, the one canonical read | | State + git | `src/tallyman_core/catalog_state.py`, `catalog.py` | per-concern tracked files in tallyman's native catalog git repo; the checkpoint commits | | Companion | `src/tallyman_companion/app.py` | HTTP/SSE API the React SPA talks to | | Buckaroo mgr | `src/tallyman_companion/buckaroo_lifecycle.py` | owns the Buckaroo subprocess and opens its sessions | -| Paths | `src/tallyman_core/paths.py` | single source of truth for every on-disk location | +| Paths | `src/tallyman_core/paths.py` | the project's directory layout (`compute_cache/result_cache/` is named in `materialize.py`, and `data/.cas/` in `source_identity.py`) | Throughout, `` is `entry_dir(project, content_hash)` = `/artifacts/catalog/entries//`, and `` is @@ -55,78 +61,106 @@ The model calls one of three tools (`server.py`): the companion's `put_code` code-edit path. The code is a Python string that must bind a variable `expr` to an -ibis/xorq expression, typically built from `tracked_expr_from_alias("alias")` (another -entry), `read_project_file("file")` (a raw parquet source) or -`tallyman_read_csv(path, schema=...)` (a CSV). All three tools converge on -`_run_and_record` → `build_and_persist(project, code, prompt)`. +ibis/xorq expression built from other entries: `tracked_expr_from_alias("alias")` +follows an alias, and `pinned_expr_from_alias("alias-v2")` pins one version of +it. A recipe never opens a file. All three tools converge on `_run_and_record` → +`build_and_persist(project, code, prompt)`. The companion's code editor +(`PUT /{project}/api/code/{alias}`) calls `build_and_persist` directly, after +refusing a source alias. + +### Before this: the data is imported + +The data at the bottom of every chain entered the catalog earlier, through +`catalog_import_source(outside_path, alias, ...)` (`source_import.update_and_depend`, +ADR-011). That call is not a build of a recipe. It digests the file, clones its +bytes to `/data/.cas/` (checked against the digest after +the copy), writes the rows in file order with a last `__row_order` column to +`compute_cache/result_cache/.parquet`, and writes an entry +directory of its own: a generated `expr.py` whose header records the path, the +digest and the reader options, a frozen `xorq_build/`, `schema.json`, and a +`manifest.json` whose `provenance` says where the data came from. The content +hash is an md5 of the bytes' digest and the reader options. It then appends the +entry to the source alias, so the recipe above can read it with +`tracked_expr_from_alias("orders")`. The outside file is never read again. ## 2. Build + execute (`build_and_persist`) -The whole build holds the project's write lock (`catalog_state.project_lock`), so -there is one writer at a time per project. In order: +The whole build holds the project's write lock (`catalog_state.project_lock`), +so builds, materializations, checkpoints and resets in either process happen one +at a time. In order: 1. **Import the user code** in a fresh module scope (`_import_script`). During - the import, each loader announces itself. `read_project_file` and - `tallyman_read_csv` take the source's content digest, clone the source - under the `cas` identity mode (the default) to `data/.cas/`, - and write an **ordered copy** of it under `compute_cache/ordered_sources/` - (the source's rows in file order, plus a last column `__row_order`), unless - that copy is already on disk. `tracked_expr_from_alias` records each parent - edge, and makes the parent's files exist first (`ensure_materialized`), + the import, `tracked_expr_from_alias` and `pinned_expr_from_alias` resolve + each alias, record the parent edge (`parent_capture` collects them for the + manifest), and make the parent's files exist first (`ensure_materialized`), because a child cannot be built over a snapshot that is missing. - `source_identity`, `parent_capture` and `ordered_copy` each collect what was - announced, for the manifest. + `read_project_file` and `tallyman_read_csv` raise here, naming the import to + use, and so does a bare content hash or bare alias passed to + `pinned_expr_from_alias`. 2. **Check, classify, rewrite, compile.** The build refuses a raw `xo.deferred_read_csv` and a raw `xo.deferred_read_parquet` of a file that - is not under `compute_cache/`, since such a read gets no digest, clone or - ordered copy. `worthiness.classify_expr` then decides, once and on the + is not under `compute_cache/`, since such a read gets no digest and no + clone. `worthiness.classify_expr` then decides, once and on the expression the author wrote, whether the entry is **worthy** (materialized) or **cheap** (row-preserving over one file, no file of its own); the verdict goes in the manifest. `rewrite_for_build` rejects in-memory reads, a recipe that already contains a xorq cache node, an assignment to - `__row_order`, and a cheap entry that drops that column. It gives a worthy + `__row_order`, a join chain over three entries that all carry that column + (the error shows the `.drop("__row_order")` fix; chains of semi or anti + joins are refused too, #199), and a cheap entry that drops the column. It + gives a worthy entry the canonical sort (the author's sort keys, then `__row_order`, then the remaining sortable columns), and moves `__row_order` to the last position of a cheap entry. `build_expr(rewritten, builds_dir=)` then compiles into a throwaway temp dir whose name *is* the content hash: xorq's dask-style token over the rewritten expression structure (`content_hash = build_path.name`). The hash follows the content of every - source in every identity mode, because the path xorq tokenizes is the - ordered copy's, whose name is a function of the source's digest. Under - `salt` the digests are also folded in afterward (`si.salted_hash`). `expr.py` - keeps the author's literal source. -3. **Idempotency check**: if `` already exists for this hash, nothing is - rebuilt — the prompt is appended as a re-run event and the existing - `BuildResult` is returned. (This is the cheap path that Part 2 leans on.) + input, because every path xorq tokenizes is a snapshot named by a content + hash, down to the source entries, whose hash is a function of their bytes. + `expr.py` keeps the author's literal source. +3. **Idempotency check**: if `/manifest.json` already exists for this + hash, nothing is rebuilt: the prompt is appended as a re-run event and the + existing `BuildResult` is returned. (This is the cheap path that Part 2 + leans on.) A directory with no manifest, left by a build that crashed, is + built again. 4. **Lay down the entry**: create `/`, copy xorq's build output to `/xorq_build/`, rewrite project-root paths to the `${TALLYMAN_PROJECT_ROOT}` placeholder (`make_portable_inplace`), and write the author's source to `/expr.py`. 5. **Execute.** This is the one place the entry's query runs, and what it writes depends on the verdict. A **worthy** entry is materialized: `materialize` - loads the entry's frozen build, rebinds it onto a single-partition - connection, and streams the rows through the snapshot writer into - `compute_cache/result_cache/.parquet`, numbering them in a - last `__row_order` column. It runs the query twice and compares the content + loads the entry's frozen build (which writes the expanded build, + `/.xorq_build_expanded/`), rebinds it onto a single-partition + connection, and streams the rows through the snapshot writer, numbering + them in a last `__row_order` column. The file stays at a temporary name + beside `compute_cache/result_cache/.parquet` until step 6 + moves it into place. It runs the query twice and compares the content digests of the two files, so a recipe that is not reproducible is known from the start (`reproducible: false`, and the file is pinned). A **cheap** entry streams its frozen plan once, through `stream_row_count`, which forces - row-level evaluation (a bad cast or UDF fails here, in tallyman's process) - and yields the exact row count, but keeps no rows. No `result.parquet` is - written either way. + row-level evaluation (a bad cast fails here, in tallyman's process; any UDF + makes an entry worthy, so a UDF fails in `materialize`) and yields the exact + row count, but keeps no rows. No `result.parquet` is written either way. If + anything from step 4 on fails, the build removes the entry directory it + created and its own temporary file, and nothing else: a file already at the + snapshot's path, such as one a reset left behind, stays as it was (#193, + fixed in #222). 6. **Derive metadata**: the row count and the `result_digest` (an `arrow-sha256:` content digest of the snapshot, worthy entries only) come from the execute above. A worthy entry's schema is read from the file `materialize` wrote, since parquet changes some types (a `timestamp[s]` column comes back `timestamp[ms]`) and the writer adds `__row_order`; a cheap entry's schema comes from the expression, which already ends in that column. - Write `/schema.json` and `/manifest.json`, and append the - prompt to `/prompts/.jsonl`. The manifest carries the verdict + Write `/schema.json` and `/manifest.json`. The manifest carries the verdict (`cache_worthy`, `cache_worthy_why`), `compile_seconds` and `cache_bytes` (the snapshot's size, or `None` for a cheap entry), `result_digest`, - `reproducible`, `snapshot_format`, `engine_versions`, `ordered_copies`, - `sources` and `parents`, and is written last and atomically, so its - presence means the entry is complete. + `reproducible`, `snapshot_format`, `engine_versions` and `parents`, and is + the entry directory's last write, atomic, so its presence means the entry + is complete. (Only a source entry's manifest has `provenance`, written by + the import.) Then, for a worthy entry, `publish_snapshot` moves the staged + file into place with one atomic replace. That is the last write to the + entry's result, so the snapshot's path changes only once the entry is + complete. Last, the prompt is appended to `/prompts/.jsonl`. 7. **Mark persisted; the checkpoint commits.** `build_and_persist` sets `catalog_registered = True`, meaning the entry dir is fully on disk — it does not write git itself. Durability is the *checkpoint's* job: when the MCP tool @@ -143,30 +177,31 @@ there is one writer at a time per project. In order: |---|---|---| | Build recipe | `/xorq_build/` (portable) | step 4 | | User source | `/expr.py` | step 4 | -| **Snapshot** (worthy only) | `/compute_cache/result_cache/.parquet` | `materialize`, step 5 | -| **Ordered copy** of each source | `/compute_cache/ordered_sources/.parquet` (and a `.digest` file) | `read_project_file` / `tallyman_read_csv`, step 1, unless already there | +| **Snapshot** (worthy only) | `/compute_cache/result_cache/.parquet` | `materialize` at a temporary name, step 5; `publish_snapshot`, step 6 | +| Expanded build (worthy only) | `/.xorq_build_expanded/` and its `.complete` marker | `materialize` loading the build, step 5 | | Schema / manifest | `/schema.json`, `/manifest.json` | step 6 | | Prompt history | `/prompts/.jsonl` | step 6 | | Chart / display config (if attached or carried forward) | `/chart_specs/.vl.json`, `/display_configs/.json` | UI, or `carry_forward_entry_config` on revise (§3) | -| Source digests | `/artifacts/source_digests.json` (stat-keyed memo) + the manifest's `sources` map (`cas` and `salt`) | `source_identity`, step 1 | -| Content source clones (`cas` only) | `/data/.cas/` (outside the catalog repo) | `read_project_file` → `ensure_cas_path`, step 1 | | Alias | `/aliases.jsonl` | `set_alias` (§3 below) | | Recipe zip + pointers + commit | `/entries/.zip`, `entries.jsonl` + a git commit | the checkpoint, step 7 | `xorq_build/`, `manifest.json`, and `schema.json` are `ENTRY_ARTIFACT_NAMES` in -`paths.py` — immutable build outputs, partitioned from the regenerable -`ENTRY_CACHE_NAMES` (`.buckaroo_stat_cache`, `.xorq_build_expanded`, -`.xorq_view_build`). The perf overlay symlinks the artifacts read-only and omits -the caches, which is the operational proof of the split: nothing ever rewrites -an artifact. There is no `result.parquet` in this list — none is written. The -single materialized copy of a worthy entry's rows is its snapshot (§6), and a -cheap entry keeps no copy at all. - -The entry directory is gitignored; its git-tracked durable form is the recipe -zip `entries/.zip` that the checkpoint commits (step 7). So the build dir -is untracked-but-durable — the recipe zip carries it across a clone, and -`entries.jsonl` records which dirs should exist so `reset_to` reconciles them -from the bullpen, never silently rewriting them. +`paths.py` — build outputs, partitioned from the regenerable `ENTRY_CACHE_NAMES` +(`.buckaroo_stat_cache`, `.xorq_build_expanded`, `.xorq_view_build`). The perf +overlay symlinks the artifacts read-only and omits the caches, which is the +operational proof of the split. One artifact is written again after create: an +unfaithful heal records `unfaithful_heal_digest` in `manifest.json`, by an +atomic replace that swaps an overlay's link for a file and leaves the original +alone. There is no `result.parquet` in this list — none is written. The single +materialized copy of a worthy entry's rows is its snapshot (§6), and a cheap +entry keeps no copy at all. + +The entry directory is gitignored; what the catalog repository tracks for it is +the recipe zip `entries/.zip` that the checkpoint commits (step 7). +Nothing reads the zip back (`catalog.py`): it records what a checkpoint +committed, and a clone of the catalog repository does not recreate entry +directories from it. `entries.jsonl` records which dirs should exist so +`reset_to` can reconcile them from the bullpen. What is **not** written yet — these are lazy, and that is the whole point of the timeline: @@ -175,6 +210,10 @@ the timeline: Populated on the first `/load_expr`, by Buckaroo, at view time (§5). - **`/.xorq_view_build/`** — for a worthy entry, the build Buckaroo is handed (§5). Written on the first view. +- **`/.xorq_build_expanded/`** — for a cheap entry, its build with the + project's path filled back in. Written the first time something loads the + build: a page read, the grid, a child's build, or the companion's startup + warm-up. - **`/primary_key.json`** — written on the first primary-key scan. The snapshot is deliberately *not* on this list: it is written at build, in @@ -183,17 +222,20 @@ step 5, not on first read (§6). ## 3. Alias registration + notify For a named entry, `catalog_create` calls `set_alias(project, name, hash, -expect_exists=False)`, which writes a line to the tracked -`/aliases.jsonl` — `{"alias", "latest", "history": [...]}` — and -appends a notebook cell. Aliases live in their own tracked file now, not in a -`catalog.yaml`; #103 made that possible by giving tallyman its own native -catalog repo (the old xorq catalog package rejected extra tracked files, which -is why aliases used to be smuggled into `catalog.yaml`). Then +expect_exists=False)`, which rewrites the tracked `/aliases.jsonl` (one +`{"alias", "latest", "history": [...], "kind"}` line per alias) with the new +alias in it, and then appends a notebook cell. The kind is `catalog` here; +`set_alias` refuses a name that is already a source alias, and refuses to point +a catalog alias at a source entry. Aliases live in their own tracked file now, +not in a `catalog.yaml`; #103 made that possible by giving tallyman its own +native catalog repo (the old xorq catalog package rejected extra tracked files, +which is why aliases used to be smuggled into `catalog.yaml`). Then `_notify("new_entry", content_hash=…, alias=…, version=…)` POSTs the companion's -`/internal/notify`, which publishes an SSE event to any connected SPA -(`app.py`). The viewer's catalog list refreshes; nothing touches Buckaroo yet. -The git commit for all of this lands when the tool returns, at the checkpoint -(§2 step 7). +`/internal/notify`, which publishes an SSE event (a message on the HTTP stream +each open browser tab holds) to any connected SPA (`app.py`). The viewer's +catalog list refreshes; nothing touches Buckaroo yet. The git commit for all of +this lands when the tool returns, at the checkpoint (§2 step 7), so the +notification goes out a moment before the commit. ## 4. The user clicks the new expression @@ -208,9 +250,15 @@ The handler: loading a grid can mean re-creating a missing file, which must not block the page (#133). -The grid loads separately. `LazyBuckarooEmbed` waits until the grid scrolls near -the viewport, then polls `GET /{project}/api/session/{hash}` (`get_session`) -until it returns a websocket URL. +The grid loads separately. On the catalog page the data tab +(`BuckarooDataPane`) requests `GET /{project}/api/session/{hash}` +(`get_session`) as soon as the entry opens, shows a spinner with the elapsed +time while the request blocks, and then shows the grid, or the reason it failed +with a retry button (#133). The tab stays mounted, hidden, when another tab is +shown, so its websocket stays open. On the notebook page, `LazyBuckarooEmbed` +waits until a cell scrolls near the viewport, then polls the same route until it +returns a websocket URL. The notebook's data route (`/api/notebook_full`) has +also already posted `/load_expr` for every cell (#202). ## 5. `load_session` → Buckaroo `/load_expr` (`buckaroo_lifecycle.py`) @@ -234,35 +282,43 @@ precise message and a retry. never writes a snapshot. For a **cheap** entry, `ensure_expanded_build` materializes `${TALLYMAN_PROJECT_ROOT}` in the entry's `xorq_build/` into `entry_expanded_build_dir(project, hash)`, a **stable per-entry path**, never - a random tmp dir, gated by a `.complete` marker. Both paths are stable - because Buckaroo's stat-cache keys include the build directory's path; a - random path would miss the stat cache on every restart. + a random tmp dir, gated by a `.complete` marker. Both paths are stable so + that Buckaroo is handed the same build directory after a restart, which is + what its on-disk stat cache is meant to rely on. 4. **Stat-cache dir**: `stat_cache = /.buckaroo_stat_cache`, `mkdir(exist_ok=True)` — preserves an existing cache, never wipes it. 5. **POST `/load_expr`** with `session=entry--`, - `build_dir` (from step 3), `project_root=` (where Buckaroo - scans for project-authored stat/post-processing klasses), - `cache_storage_path=`, and `row_order_column="__row_order"`. A - promoted diff entry adds `column_config_overrides`, and the companion adds - `telemetry_url` when it knows its own address. Buckaroo loads the xorq - expression, creates the session, and returns its `session` id. + `build_dir` (from step 3), `project_root=`, + `cache_storage_path=`, and `row_order_column="__row_order"`. + Buckaroo looks for the project's klasses (its summary stats, + post-processing functions and display classes) in `stats/`, + `post_processing/` and `display/` under `project_root`; tallyman keeps + display classes in `artifacts/display/` but writes the other two under + `artifacts/catalog/`, so Buckaroo does not find them (#170). A promoted diff + entry adds `column_config_overrides`, and the companion adds `telemetry_url` + when it knows its own address. Buckaroo loads the xorq expression, creates + the session, and returns its `session` id. 6. Nothing is remembered. The session id is a function of the project and the hash, so tallyman keeps no session map and no session file. Because the stat cache is empty for a new entry, Buckaroo computes summary -stats from scratch and **writes `/.buckaroo_stat_cache/parquet/`** — -this is the cold population. With buckaroo 0.15.0 + `BUCKAROO_PERF=1` this -shows up in `~/.buckaroo/logs/server.log` as `misses=N`, `snapshots -written>0`, and a `firstpull.summary_stats secs=…` span. +stats from scratch and **writes `/.buckaroo_stat_cache/parquet/`**; +this is the cold population. Buckaroo logs one line per run for it +(`xorq stat cache […]: N hit(s), M miss(es), K snapshot(s) written …`) to its +own `~/.buckaroo/logs/server.log`, and posts its per-load timings to the +companion as `firstpull.*` spans (kept in `artifacts/telemetry.jsonl`, shown when +a `buckaroo` event is expanded in the Log tab). The subprocess's stderr goes to +`/buckaroo.log`, in the project `tallyman run` started with. ## 6. The websocket and first data pull The SPA opens `ws://…/ws/{session}` from the URL the session route returned. Over the socket Buckaroo streams the first row window and the summary-stats payload. The JS `SmartRowCache` holds row segments client-side from here on (see -`caching.md`). Buckaroo pages in its own process, and pages an entry by the -`row_order_column` it was told; builds of Buckaroo that predate that hint ignore -it (buckaroo-data/buckaroo#974). +`caching.md`). Buckaroo pages in its own process. Tallyman named `__row_order` +as the `row_order_column`, but Buckaroo 0.15.6, the pinned version, ignores that +hint, so the grid's pages are not yet ordered by it +(buckaroo-data/buckaroo#974). The **result** is read through one function, `cached_result_expr`, whose internals load the entry's frozen build (the canonical read of @@ -279,20 +335,26 @@ first, and then: backend, recomputed on read over files that exist. If a snapshot is missing, `ensure_materialized` re-creates it under the project -lock, after re-checking that it is still missing, by running the frozen build once -through the same writer, and verifies the result against the `result_digest` -recorded at build. Missing files it reads are made first: a parent's snapshot by -recursing on the hash in its name, an ordered copy from its clone with the reader -options in the manifest, a clone from the live source while the bytes still match. -A mismatch is still served, but never silently: `_verify_self_heal` records a -durable `unfaithful_heal` error (which also pins the file), wipes the entry's -stat cache, and fires the hooks, which in the companion force Buckaroo to reload -the open grid and push an SSE event. +lock, after re-checking that it is still missing, by running the frozen build +once through the same writer, and verifies the result against the +`result_digest` recorded at build. Missing files it reads are made first, each a +parent's snapshot, by recursing on the hash in its name. A source entry's +snapshot is made again from its clone with the reader options in its manifest, +and checked the same way; if the clone is gone too, the error names it and the +import call that repairs the version. A mismatch is still served, but never +silently: `_verify_self_heal` pins the file by recording the digest it wrote in +the manifest's `unfaithful_heal_digest`, records a durable `unfaithful_heal` +error for the error banner, wipes the entry's stat cache, and fires the hooks. +In the companion the hook posts a forced reload of the entry's grid to Buckaroo +and publishes an `unfaithful_heal` SSE event, which the SPA has no listener for; +the error shows in the catalog page's error banner. The checks and the hook run +while the heal holds the project lock (#203), and cheap entries that read the +healed snapshot are not flagged (#208). Every reader takes this path: the paginated viewer (`api_data` pages the entry with `row_order.page`, `ORDER BY __row_order` and then `LIMIT/OFFSET`, so the same -request returns the same rows), both sides of a diff, chaining a child recipe, and -post-processing. +request returns the same rows), charts (which fetch up to 100,000 rows through +`api_data`), both sides of a diff, chaining a child recipe, and post-processing. --- @@ -303,13 +365,14 @@ Three distinct "warm" scenarios, because they hit different caches. ## A. Re-submitting identical code (build idempotency) `build_and_persist` runs `build_expr` again, gets the **same content hash**, -finds `` already present, appends the prompt, and returns the existing -`BuildResult`. No execution, no materialization, and no new file: the ordered -copies of its sources are already on disk, so ingest finds them and writes -nothing. The expression is never recomputed when its structure and its sources' -content are unchanged. If a source *has* changed, its digest changes, so the -ordered copy has a new name and the same recipe forks a new entry; the old entry -keeps the rows it was built from. +finds `` and its manifest already present, appends the prompt, and +returns the existing `BuildResult`. No execution, no materialization, and no new +file. The expression is never recomputed when its structure and its parents are +unchanged. A changed data file changes nothing here until it is imported again: +the import mints the next version of its source alias, the entries that follow +the alias go stale, and recalc replays their recipes into new entries (at once, +when auto-recalc is on, as it is by default). The old entries keep the rows they +were built from. Importing unchanged bytes is itself a no-op. ## B. Revisiting an entry in the same companion process (RAM-warm) @@ -318,14 +381,18 @@ session route then runs `ensure_materialized` (a manifest read and a `stat` for worthy entry) and POSTs `/load_expr` again with the same derived id. Buckaroo still holds that session, with the same build dir, and the post carries none of the config-bearing fields, so it answers from the session it holds. No Buckaroo -recompute, no stat-cache touch. The SPA reuses the same websocket. +recompute, no stat-cache touch. The grid connects to the same session again. +Two exceptions (#202): a promoted diff entry's post carries its +`column_config_overrides`, so Buckaroo re-runs its pipeline for it on every +open, and two opens of one entry at the same moment both post and both run it. ## C. Revisiting after a restart (disk-warm, RAM-cold) -When Buckaroo restarts, the companion detects a new `started` timestamp from -`/health` and clears its in-RAM bookkeeping of diff sessions -(`_reset_session_bookkeeping_if_restarted`) — but it touches nothing on disk. -There is no entry-session record to clear. So on the next visit: +When the companion starts (or restarts) the Buckaroo subprocess, it reads the +`started` timestamp from `/health` and, if it is new, clears its in-RAM +bookkeeping of diff sessions (`_reset_session_bookkeeping_if_restarted`). It +touches nothing on disk, and there is no entry-session record to clear. So on +the next visit: 1. `load_session` POSTs `/load_expr`, and Buckaroo, which no longer holds the session, creates it. @@ -337,16 +404,18 @@ There is no entry-session record to clear. So on the next visit: `hits>0, misses=0` in `server.log`; `firstpull.summary_stats secs=Y` with `Y ≪` the cold time.) -The cache survives the restart precisely because (a) nothing in the restart -path deletes it, and (b) the build path is stable, so the stat-cache key is -identical across processes. +The cache is meant to survive the restart because (a) nothing in the restart +path deletes it, and (b) Buckaroo is handed the same build over the same files, +so it computes the same stat-cache keys. In practice a first load after a +restart has not been seen to hit it (#157). -The only things that *invalidate* the stat cache are a klass-changing event — a -summary-stat / post-processing / display-config change, a project reset, or a -recalc — and an unfaithful heal. The first group routes through +The only things that *invalidate* the stat cache are a klass-changing event (a +summary-stat, post-processing or display change), a project reset, a recalc +that moved an alias, and an unfaithful heal. All but the last route through `reload_project_sessions`, which POSTs `/reload_expr/` for every entry of the -project (a 404 means the grid is not open) and clears the stat cache of each grid -it reloaded (`_clear_stat_cache`). A plain restart is not such an event. +project, one after another (a 404 or 400 means the grid is not open; #201), and +clears the stat cache of each grid it reloaded (`_clear_stat_cache`). A plain +restart is not such an event. Similarly, the snapshot of a worthy entry survives the restart untouched — it is named by the entry's content hash under `compute_cache/` — so @@ -358,30 +427,35 @@ restart never does. # One-line summary of cache write phases -- **Build time** (`build_and_persist`): during the recipe import, the ordered - copy of each source under `compute_cache/ordered_sources/` and (`cas` only) its - `data/.cas/` clone; then the snapshot under - `compute_cache/result_cache/` (worthy entries only), `schema.json` / - `manifest.json`, source digests (`cas` and `salt`), `expr.py`, and the prompt - history. +- **Import time** (`catalog_import_source`, before any recipe reads the data): + the clone under `data/.cas/`, the source entry's snapshot under + `compute_cache/result_cache/`, and its entry directory (`expr.py`, + `xorq_build/`, `schema.json`, `manifest.json` with `provenance`); then the + source alias in `aliases.jsonl`. +- **Build time** (`build_and_persist`): `xorq_build/` and `expr.py`, the snapshot + under `compute_cache/result_cache/` and the expanded build (worthy entries + only), `schema.json` / `manifest.json`, and the prompt history. +- **Naming** (right after the build, in the tool): `aliases.jsonl` via + `set_alias`, and the notebook cell. - **Checkpoint** (when the tool returns, `checkpoint_catalog`): the recipe zip - `entries/.zip`, the `aliases.jsonl` / `entries.jsonl` tracked surface, - and one git commit + step tag. + `entries/.zip` and `entries.jsonl`, and one git commit + step tag that + also takes in `aliases.jsonl` and the rest of the tracked surface. - **First view** (`load_session` → `/load_expr`): `.buckaroo_stat_cache/`, and the - stable expanded build (cheap entry) or view build (worthy entry). No session - record is written anywhere. + stable expanded build (cheap entry, unless a read already wrote it) or view + build (worthy entry). No session record is written anywhere. - **First primary-key scan**: `/primary_key.json`. -Everything keyed on the content hash is immutable, and a deleted file is made -again and verified the next time something reads it. The only invalidations are -a klass-changing edit, a reset or recalc, or an unfaithful heal (all of them -the stat cache). Only the user -deletes a file (the Cache page; a pinned snapshot is refused). A `reset_to` -leaves `compute_cache/` alone, and moves the `data/.cas` source clones that no -surviving entry's `manifest.sources` refers to into `/bullpen/cas/` -instead of deleting them, so a reset forward brings them back. A child's -`manifest.sources` records every clone its parents' builds read, so a clone stays -referenced as long as any entry built on it survives. See `caching.md` for the -full invalidation table. (The cold-reconstruction staleness hole this section -used to reference — #74/#115 — is closed: reads load the frozen build, so a cold -read cannot see a post-build source edit.) +Everything keyed on the content hash is immutable (apart from the pin an +unfaithful heal adds to a manifest), and a deleted file is made again and +verified the next time something reads it. The only invalidations are a +klass-changing edit, a reset or recalc, or an unfaithful heal (all of them the +stat cache). Only the user deletes a file (the Cache page; a pinned snapshot is +refused), and a failed build leaves the file at its entry's path alone. A +`reset_to` leaves `compute_cache/` alone, and moves the `data/.cas` clones that +no surviving source entry's `manifest.provenance` names into +`/bullpen/cas/` instead of deleting them, so a reset forward brings +them back. A clone therefore stays as long as the source version that imported +it survives. See `caching.md` for the full invalidation table. (The +cold-reconstruction staleness hole this section used to reference — #74/#115 — +is closed: reads load the frozen build, so a cold read cannot see a post-build +edit of a data file.) diff --git a/docs/installing.md b/docs/installing.md index 9ffdcd4..f3af473 100644 --- a/docs/installing.md +++ b/docs/installing.md @@ -58,24 +58,31 @@ A "project" is a named catalog/notebook workspace on disk. uv run tallyman init spike ``` -This creates `~/.tallyman-notebooks/projects/spike/` and a starter fixture. -List or inspect projects later with `uv run tallyman project_list` (via MCP) or -just by looking under the projects root. +This creates `~/.tallyman-notebooks/projects/spike/`, writes a starter fixture +(`data/orders.parquet`; pass `--no-fixture` to skip it) and records the +project's first checkpoint, step 0. There is no CLI command that lists projects: +look under `~/.tallyman-notebooks/projects/`, use the project list in the +browser, or ask Claude to call the `project_list` MCP tool. ## 5. Start the companion app -The companion is a FastAPI app on `http://127.0.0.1:7860`. The MCP server pushes -live updates to it over SSE. +The companion is a FastAPI app on `http://127.0.0.1:7860`. The MCP server tells +it about each change with an HTTP request, and it pushes live updates on to the +browser over SSE (a stream of events the browser keeps open). ```sh uv run tallyman run --project spike ``` Leave this running in its own terminal. Open to watch -the catalog, notebook, lineage, and diffs update as you work. +the catalog, notebook and diffs update as you work. -`tallyman run` also spawns a Buckaroo subprocess on `:8700` for in-table recon. -Disable it with `--no-buckaroo` if you don't need it. +`tallyman run` also spawns a Buckaroo subprocess on `:8700` (or on a random port +if 8700 is taken), which draws each entry's data grid. Its stderr goes to +`~/.tallyman-notebooks/projects/spike/buckaroo.log`, and Buckaroo keeps its own +log in `~/.buckaroo/logs/server.log`. Disable it with `--no-buckaroo` if you +don't need the grids; an entry's page then shows its row count and a note that +Buckaroo is not available. ## 6. Configure and approve the MCP server @@ -97,7 +104,7 @@ server: } ``` -If you cloned to a different path, update `cwd` to point at your checkout. +The copy in the repo has the owner's path in `cwd`; set it to your checkout. Because `.mcp.json` is a project file, Claude Code will **not** trust it automatically. Launch Claude Code from the repo directory and approve the @@ -140,18 +147,30 @@ of the catalog with a `V_n` chip; previous versions stay in forensic history. ## Sharing a project -A project directory is portable: +A project directory is portable: paths inside each entry's build are stored +relative to the project, so it can be read from any location. ```sh -tar czf my-project.tgz -C ~/.tallyman-notebooks/projects spike +uv run tallyman pack spike -o my-project.tgz # on the other machine: tar xzf my-project.tgz -C ~/projects/ uv run tallyman serve ~/projects/spike ``` +`tallyman pack` tars the whole project directory, including `compute_cache/` +(the result files tallyman can make again), so the archive can be large. + `tallyman serve` runs the companion **read-only** against any project directory -on disk — same catalog and history, no edit affordances (mutation routes return -403). +on disk: same catalog, history and charts. The browser still shows the edit +controls, but the server answers their requests with 403. It starts no Buckaroo +subprocess, so entry grids do not load. + +One known defect affects copies (#209). An archive carries each entry's +`.xorq_build_expanded/` directory, a copy of the build with the original +project's absolute path filled in, and tallyman reuses it without checking the +path. A cheap entry (a filter or selection over one file, with no result file of +its own) then reads from the original location, and fails once that location is +gone. ## Reference @@ -166,20 +185,32 @@ on disk — same catalog and history, no edit affordances (mutation routes retur | Variable | Default | Purpose | |----------|---------|---------| -| `TALLYMAN_PROJECT` | `spike` | Active project name | | `TALLYMAN_HOME` | `~/.tallyman-notebooks` | Root for all project state | +| `TALLYMAN_PROJECT` | none | Seeds the `active_project` file when that file does not exist yet and the named project does (see below) | +| `TALLYMAN_AUTO_RECALC` | unset | `1`/`0` (or `true`/`false`) overrides the project's auto-recalc switch (on by default) | +| `TALLYMAN_LOG_LEVEL` | `INFO` | Log level of the MCP server | **State on disk** lives under `TALLYMAN_HOME`: ``` ~/.tallyman-notebooks/ ├── active_project # one-line plain text; the active project +├── server.lock # held by the `tallyman run` serving this data dir; names its pid and port └── projects/ └── / # catalog, notebook, build artifacts per project ``` -The active project is resolved from `TALLYMAN_PROJECT` first, then the -`active_project` file. +[architecture.md](architecture.md#on-disk-layout) has the full layout. + +The active project is the one named in the `active_project` file. A command's +`--project` flag (`tallyman run --project spike`) names a project explicitly, and +`tallyman run` writes it to the file. `TALLYMAN_PROJECT` is read only when the +file does not exist yet and the project it names exists, to create the file, so +once the file exists it wins over the variable (#39). (`tallyman serve` is the +exception: it sets the variable to the served directory's name and reads it +while it runs.) The MCP server takes its project from the file at its first tool +call and keeps using it for the rest of the Claude Code session, until +`project_switch` or `project_new` changes it. ## Troubleshooting @@ -200,5 +231,12 @@ The active project is resolved from `TALLYMAN_PROJECT` first, then the its replies carry no entry links, and `project_switch` / `project_new` fail. - **Port 7860 already in use** — something else is listening there; pass `--port` to `tallyman run`. - - +- **An entry's data tab says "Buckaroo not available"** — the companion was + started with `--no-buckaroo`, or Buckaroo failed to start; `tallyman run` + prints why. If the tab shows an error instead, its detail says whether + tallyman could not prepare the entry or Buckaroo could not load it, and + `buckaroo.log` in the project directory has Buckaroo's side. +- **The browser stops responding while Claude builds an entry** — a build holds + the project's write lock, and a page that needs the same lock (to re-create a + deleted result file, or an edit made in the browser) waits for it (#186, + #190). diff --git a/docs/mcp-server.md b/docs/mcp-server.md index e2e0e55..e295108 100644 --- a/docs/mcp-server.md +++ b/docs/mcp-server.md @@ -2,10 +2,12 @@ The MCP server (`src/tallyman_mcp/server.py`) is the surface Claude Code drives. It is a [FastMCP](https://github.com/jlowin/fastmcp) server that exposes the -catalog as **30 tools and one prompt** over stdio. Claude Code is the only -caller; a human never invokes these directly. Each tool compiles or mutates the -on-disk catalog through `tallyman_core` / `tallyman_xorq`, commits a git -revision, and best-effort notifies the running companion so the browser updates. +catalog as **31 tools and one prompt** over stdio. Claude Code is the only +caller; a human never invokes these directly. Most tools compile or change the +on-disk catalog through `tallyman_core` / `tallyman_xorq`, commit a git +revision (a checkpoint), and best-effort notify the running companion so the +browser updates. Terms (entry, alias, worthy and cheap entries, snapshot, klass) +follow [architecture.md](architecture.md#terms). This doc lists every tool, its parameters, its return shape, and — the part that matters most when reasoning about a session — its **side effects**: whether it @@ -45,7 +47,7 @@ fail are logged and dropped. ## Cross-cutting behavior -Four mechanisms apply to (almost) every tool, so they are documented once here +Five mechanisms apply to (almost) every tool, so they are documented once here rather than repeated per tool. **Auto-checkpoint (opt-out).** `mcp.tool` is monkeypatched to @@ -78,8 +80,14 @@ tagging. **Notifications.** `_notify(kind, hash, **extra)` is a 2-second best-effort POST to the companion's `/internal/notify`, which re-publishes it as an SSE event of that `kind`. It never raises. Pass extra data as flat kwargs (`remap=...`), not -`extra={...}`, or it nests and is silently dropped. The "Notifies" line per tool -lists the kinds it emits. +`extra={...}`, or it nests and is silently dropped. The "SSE notify" column +below lists the kinds each tool emits. The SPA listens for `new_entry`, +`build_failed`, `notebook_changed`, `chart_attached`, +`post_processing_changed`, `summary_stat_changed`, `recalc` and +`project_switched`; `alias_changed`, `alias_renamed`, `display_changed` and +`entry_added` reach the browser but trigger no refetch. For +`summary_stat_changed`, `post_processing_changed`, `display_changed` and +`recalc` the companion also reloads the project's Buckaroo sessions. **Sticky active project.** `_resolve_active_project()` returns the in-process `_mcp_active_project` (set by `project_switch` / `project_new`) ahead of the @@ -87,12 +95,13 @@ on-disk `active_project` file, so switching once is sticky for the rest of the session regardless of what another session writes to disk. The first tool call seeds it. `SESSION_ID` (an 8-hex uuid) tags this process in `events.jsonl`. -**Auto-recalc.** `catalog_revise` and `catalog_promote_diff` advance an existing -alias head, which can make followers stale. Both route through +**Auto-recalc.** `catalog_revise`, `catalog_promote_diff` and +`catalog_import_source` (when it mints a new version) advance an existing alias +head, which can make followers stale. All three route through `_auto_recalc_after_head_advance` (when the project's `auto_recalc` switch is on): a checkpoint-free cascade rebuilds the followers in dependency order and leaves them in the working tree, so the head advance and the whole cascade land as one -git revision. A non-empty cascade emits a `recalc` SSE event. +git revision. A cascade whose remap is non-empty emits a `recalc` SSE event. ## Side-effect matrix @@ -109,6 +118,7 @@ git revision. A non-empty cascade emits a `recalc` SSE event. | `notebook_remove` | yes | `notebook_changed` | no | | `notebook_edit_markdown` | yes | `notebook_changed` | no | | `catalog_chart` | yes | `chart_attached` | no | +| `catalog_chart_errors` | yes ⚠ | — | no | | `catalog_diff` | no | — | no | | `catalog_promote_diff` | self³ | `entry_added`, `recalc`⁴ | yes | | `catalog_scan_staleness` | no | — | no | @@ -129,46 +139,51 @@ git revision. A non-empty cascade emits a `recalc` SSE event. | `project_switch` | no | — ⁵ | no | | `project_new` | no | — ⁵ | no | -¹ `notebook_changed` only when `name` is supplied. ² only when a cell was -actually removed. ³ self-checkpoints exactly one revision; on the opt-out list so -the dispatch boundary does not double-commit. ⁴ only on a committing run that -produced a non-empty remap. ⁵ the *companion* broadcasts `project_switched`; the -MCP tool itself sends no `_notify`. ⚠ see [Known quirks](#known-quirks). +¹ `notebook_changed` only when the import mints v1 of a new alias, which appends +a notebook cell. ² only when a cell was actually removed. ³ self-checkpoints one +revision (for `catalog_recalc`, only a committing run whose remap is non-empty); +on the opt-out list so the dispatch boundary does not double-commit. ⁴ only on a +committing run that produced a non-empty remap. ⁵ the *companion* broadcasts +`project_switched`; the MCP tool itself sends no `_notify`. ⚠ see [Known +quirks](#known-quirks). Any tool whose result carries an `error` key skips its checkpoint and sends no -notify; failures come back inline in the response (most as `{error, error_id}` -for build failures, `{error}` for validation/precondition failures), not as -raised exceptions. +success notify (a failed build in the authoring tools still sends +`build_failed`; `catalog_promote_diff` sends nothing on failure); failures come +back inline in the response (most as `{error, error_id}` for build failures, +`{error}` for validation/precondition failures), not as raised exceptions. --- ## Authoring tools The compile-and-persist family. The `code` argument is a self-contained Python -script that binds a top-level `expr` to a xorq/ibis expression; -`catalog_run`'s docstring is the canonical cookbook for writing it (namespaces, -the datafusion-only backend, data sourcing, and the `xorq.ml` MODELING section). -Data sourcing inside `code` uses four helpers from `tallyman_xorq.io`: -`tracked_expr_from_alias` (a catalog entry, recorded as a lineage parent), -`pinned_expr_from_alias` (a catalog entry by hash or `"name-vN"` version -reference, no following — a bare alias is rejected, #166), -`read_project_file` (a raw parquet under `data/`), and `tallyman_read_csv` (CSV -ingest; use it for all CSV reads instead of `xo.deferred_read_csv`, #137). Both -read the file through an **ordered copy** (the source's rows in file order, plus a -last column `__row_order` holding `0..N-1`), so editing a source and re-running -the same recipe creates a new entry and the old one keeps its rows. Reading a -parquet file any other way (`xo.deferred_read_parquet`) is a build error. +script that binds a top-level `expr` to a xorq/ibis expression; `catalog_run`'s +docstring is the canonical cookbook for writing it (namespaces, the +datafusion-only backend, data sourcing, and the `xorq.ml` MODELING section). +Data sourcing inside `code` uses two helpers from `tallyman_xorq.io`: +`tracked_expr_from_alias` (a catalog entry by alias, recorded as a lineage +parent that the entry follows) and `pinned_expr_from_alias` (one version of an +alias, by `"name-vN"` version reference, no following; a bare alias is rejected, +#166, and so is a bare content hash, ADR-011 D5). A recipe never opens a file: +data files come in through `catalog_import_source` (below) as **source +aliases**, and a recipe reads one like any other alias. `read_project_file`, +`tallyman_read_csv` and `xo.deferred_read_csv` are build errors whose message +names the import to use, and so is `xo.deferred_read_parquet` of any file +outside the project's `compute_cache/`. Every entry's result ends in `__row_order`, and pages are ordered by it. A recipe that only filters, selects or adds columns is *cheap* and must keep the column: `t.select("region", "price")` fails the build, and the error shows the fix, `t.select("region", "price", "__row_order")`. A recipe that aggregates, joins, -sorts, uses a window function, union, distinct, unnest or UDF is *worthy*: the -server writes its result to a snapshot file when the entry is created and -renumbers the column to match. Assigning to `__row_order` is an error, an -`order_by` that is followed by more steps is kept if its key columns survive, and -joining three entries in one recipe needs `.drop("__row_order")` on the -right-hand inputs. `catalog_run`'s docstring has the details. +sorts, uses a window function, union, distinct, unnest, `random()`, `now()` or a +UDF, or reads a second file, is *worthy*: the server writes its result to a +snapshot file when the entry is created (running the query twice) and renumbers +the column to match. Assigning to `__row_order` is an error, an `order_by` that +is followed by more steps is kept if its key columns survive, and joining three +entries in one recipe needs `.drop("__row_order")` on the right-hand inputs +(chains of semi or anti joins are refused too, #199). `catalog_run`'s docstring +has the details. ### `catalog_run(code, prompt="") -> dict` Execute an expression and persist it as an **unnamed (scratch)** entry. Claude's @@ -180,17 +195,17 @@ default authoring tool for a one-off run. false` with `nonreproducible_columns` when a worthy entry's query gave different results on its two runs at create time. `{error, error_id}` on a `BuildError`. -- **Writes:** the entry build dir under `catalog/entries//`; the ordered - copy of each source and, for a worthy entry, its snapshot under - `catalog/compute_cache/`; a `build_ok` or `build_error` event to - `events.jsonl`. +- **Writes:** the entry build dir under `catalog/entries//`; for a worthy + entry, its snapshot under `catalog/compute_cache/result_cache/`; a `build_ok` + or `build_error` event to `events.jsonl`. - **Promote** a scratch entry to a name afterward with `catalog_alias`. -### `catalog_import_source(outside_path, alias, pinned_version=None, prompt="", schema=None, reader_options=None) -> dict` -Import a parquet or CSV into the catalog and point the source alias `alias` at -it (ADR-011). The only way a file enters: the bytes are copied into the arena, -one snapshot of them is written in file order with a `__row_order` column, and -a version of `alias` is minted. Recipes then read +### `catalog_import_source(outside_path, alias, pinned_version=None, prompt="", +schema=None, reader_options=None) -> dict` Import a parquet or CSV into the +catalog and point the source alias `alias` at it (ADR-011). The only way a file +enters: the bytes are copied into the project's clone store (`data/.cas/`), one +snapshot of them is written in file order with a `__row_order` column, and a +version of `alias` is minted. Recipes then read `tracked_expr_from_alias(alias)`; the original path is provenance and is never read again. - **Params:** `outside_path` (required) — any path, `data/` is not special; @@ -231,7 +246,8 @@ carry chart/display config forward, and cascade-recompute the alias's stale followers. The canonical head-advance path. - **Params:** `name` (required) — existing alias; `code` (required) — a self-contained recipe that **must not reference its own alias by name** - (rejected, #135 — inline the source or pin the previous version by hash); + (rejected, #135 — inline the source, or pin the previous version with + `pinned_expr_from_alias("-v")`; a bare hash is refused); `prompt` — optional. - **Returns:** `catalog_run` shape plus `alias`, `version`; `carried_over` (subset of `chart` / `display_config`) when non-empty; and a `recalc` @@ -303,12 +319,25 @@ hash). but-JSON spec only fails in the browser). - **Returns:** `{hash, spec_path}`. `{error}` for an unknown target or invalid JSON. -- **Writes:** `catalog/chart_specs/.vl.json`. +- **Writes:** `catalog/chart_specs/.vl.json`. The browser fetches the + chart's data from `/{project}/api/data/?limit=100000`. + +### `catalog_chart_errors(hash_or_alias) -> list` +Chart render failures the browser reported for an entry, most recent first. The +page posts one to `/{project}/api/chart_error` when vega-embed fails, and it is +stored in `errors.jsonl` with `tool="chart_render"`. Use it after attaching or +editing a chart, once someone has viewed the page. +- **Params:** `hash_or_alias` (required); resolved like `catalog_chart`. +- **Returns:** a list of error records (up to the 500 most recent errors are + searched), empty if none were reported; `[{error}]` for an unknown target. +- It is read-only but not on the opt-out list, so it checkpoints on every call + (see [Known quirks](#known-quirks)). ### `catalog_diff(name, va=-2, vb=-1) -> dict` **Read-only** diff of two versions of an alias (default previous vs latest). Each side's expression comes from `cached_result_expr`, which makes any missing -file exist first, and the diff drops `__row_order` from both sides. +file exist first. The diff (`full_diff`) keeps `__row_order` as a data column, so +after an inserted row the stats diff reports it as changed (#200). - **Params:** `name` (required); `va`, `vb` — version indices, 1-based or negative-from-end (`-1` latest, `-2` penultimate). - **Returns:** `{alias, before:{version,hash}, after:{version,hash}, schema, @@ -338,9 +367,11 @@ marimo-exportable. ## Staleness and recalc tools ### `catalog_scan_staleness(verify_results=False) -> dict` -**Read-only** scan of every live entry for staleness against its recorded inputs. -Purely diagnostic — never rebuilds, repoints, or checkpoints. Call it first to -see what is stale. +**Read-only** scan of every complete entry (current alias heads and superseded +versions alike; only heads can be directly stale) for staleness against its +recorded inputs. Purely diagnostic — never rebuilds, repoints, or checkpoints, +and opens no data file: an entry is stale only when an alias it follows has +moved (ADR-011 D6, one staleness axis). Call it first to see what is stale. - **Params:** `verify_results` — also check that each snapshot on disk still has its recorded `result_digest`. It reads the files that exist and writes nothing, so a snapshot that was deleted is reported and stays deleted. @@ -355,16 +386,19 @@ see what is stale. Recompute stale entries and their dependents in dependency order. Defaults to a non-committing **dry-run preview**; call again with `dry_run=False` to commit. - **Params:** `roots` — content hashes to recompute with their descendant cone; - `None` defaults to every directly-stale entry. Roots naming no live entry are - silently dropped. `dry_run` — preview when `True` (default). + `None` defaults to every directly-stale entry. Roots that name no complete + entry (no `manifest.json`) are silently dropped; a named root that is a + superseded version is kept and replayed. `dry_run` — preview when `True` + (default). - **Returns:** the `RecalcReport` (`project, roots, cone, entries, dry_run, status, remap, checkpoint_step, error`). Action vocabulary differs by mode (`rebuild`/`cascade`/`unchanged` on dry-run; `rebuilt`/`noop`/`failed`/ `skipped` on a real run). A nothing-stale fast path returns a short `{status:'ok', ..., note:'nothing stale to recompute'}`. - **Self-checkpoints arg-aware:** a dry-run takes no checkpoint; a committing run - takes exactly one for the whole walk, so the recompute is a single revision - `reset-to` can undo atomically. A committing run with a non-empty remap emits + that changed something (a non-empty remap) takes exactly one for the whole + walk, so the recompute is a single revision `reset-to` can undo atomically, + and one that changed nothing takes none. A committing run with a non-empty remap emits `recalc`. Build failures are encoded in `status`/`error`/per-entry `error` (the walk halts, the rebuilt prefix stays committed); a dependency cycle yields `status=='cycle'` rather than raising. @@ -375,9 +409,18 @@ Project-authored per-column statistics. Source runs in a restricted-globals sandbox; errors come back inline rather than later in the Buckaroo log. There is no separate update tool — re-adding the same `name` overwrites. +Buckaroo looks for project stats in `/stats/`, and tallyman sends +`project_root` as `/artifacts/`, while it writes stats to +`/artifacts/catalog/stats/`. So a stat added here is validated, +committed and reloaded, but Buckaroo does not find it and the grid does not show +it (#170). Post-processing functions have the same problem; display klasses, +which live in `artifacts/display/`, do not. + ### `catalog_add_summary_stat(name, source) -> dict` -Validate a `compute(col)` function against a 1-row ibis memtable, write it to -`/stats/.py`, and hot-reload it into open Buckaroo sessions. +Validate a `compute(col)` function against a 3-row, one-column ibis memtable, +write it to `/artifacts/catalog/stats/.py` (tracked in the +catalog repository), and ask the companion to reload the project's open Buckaroo +sessions. - **Params:** `name` (required) — a valid Python identifier; `source` (required) — must define `compute(col)` taking one ibis column and returning an ibis scalar. @@ -405,8 +448,10 @@ styling base classes already in scope; validated in a restricted sandbox before the file lands. ### `catalog_add_display_klass(name, source) -> dict` -Validate and persist a display klass to `display/.py`, then hot-reload it -into open sessions (`display_changed`). The primary use is extending +Validate and persist a display klass to `/artifacts/display/.py`, +then hot-reload it into open sessions (`display_changed`). That directory is +outside the catalog repository, so the checkpoint this tool takes commits +nothing for it, and a reset does not undo it. The primary use is extending `DefaultMainStyling` (`df_display_name='main'`) or `DefaultSummaryStatsStyling` (`'summary'`) with a `pinned_rows` entry so a stat shows as a frozen row. - **Params:** `name` (required) — valid identifier; `source` (required) — must @@ -433,7 +478,8 @@ sandbox as summary stats (third-party imports like numpy/sklearn raise ### `catalog_add_post_processing(name, source) -> dict` Validate a `process(expr)` function and write it to -`/post_processing/.py`. +`/artifacts/catalog/post_processing/.py` (tracked in the catalog +repository). Buckaroo does not find it there yet (#170, above). - **Params:** `name` (required) — valid identifier, becomes the dropdown label; `source` (required) — defines `process(expr)` returning an ibis expression or a pandas DataFrame. @@ -441,8 +487,9 @@ Validate a `process(expr)` function and write it to - **Validator gotcha:** the dry-run table has only columns `{a, b}`, so a `process` that references real column names is rejected here even when `catalog_run_post_processing` previewed it fine — guard with - `if 'col' not in expr.columns: return expr`. The new option appears only on the - next session load (V1 does not hot-swap loaded sessions). + `if 'col' not in expr.columns: return expr`. The tool's docstring says the new + option appears on the next session load; the companion also posts + `/reload_expr` for each of the project's entries, as for any klass change. ### `catalog_remove_post_processing(name) -> dict` Soft-delete to `post_processing/_disabled/`. Still shows in the list with @@ -490,11 +537,13 @@ it references exact column names rather than guessing. ## Project-lifecycle tools -These require the companion (`tallyman run`) to be active: the **companion** is -the sole writer of the `active_project` file and the genesis baseline, reached -via `_companion_post`. All three are on the opt-out list. On success the -companion broadcasts a `project_switched` SSE event itself (the MCP tool sends no -`_notify`). Each updates the in-process sticky `_mcp_active_project`. +`project_switch` and `project_new` require the companion (`tallyman run`) to be +active: for these tools the **companion** writes the `active_project` file and +the genesis baseline, reached via `_companion_post`, so that it can publish the +SSE event. All three are on the opt-out +list. On success the companion broadcasts a `project_switched` SSE event itself +(the MCP tool sends no `_notify`), and the tool updates the in-process sticky +`_mcp_active_project`. ### `project_list() -> dict` List projects on disk and report the active one. **Safe even when the companion @@ -517,7 +566,7 @@ Create a new project on disk (the companion runs `ensure_project` + genesis baseline) and switch to it. - **Params:** `name` (required) — `^[a-z0-9][a-z0-9_-]{0,31}$`, not reserved, not colliding; `with_fixture` — when `True`, the companion seeds the shoe-orders - demo parquet into `data/`. + demo parquet into `data/`, where `catalog_import_source` can import it. - **Returns:** `{name, active}`. On failure `{error}` (sticky unchanged): same shapes as `project_switch`, with 409 on a name collision. @@ -525,12 +574,13 @@ baseline) and switch to it. ### `ds_modeling_workflow(dataset, target="") -> str` An `@mcp.prompt()` (not a tool) that returns a static guidance string scaffolding -a modeling workflow: load → engineer features → split → fit via `xorq.ml` → +a modeling workflow: import → engineer features → split → fit via `xorq.ml` → score → chart. Pure function — no side effects, no I/O, not routed through the checkpoint or tagging wrappers. -- **Params:** `dataset` (required) — a parquet under `data/`, interpolated into - the text; `target` — column to predict (empty renders as `` for - unsupervised work). +- **Params:** `dataset` (required) — the data file, interpolated into the text; + the workflow's first step imports it with `catalog_import_source`. `target` — + column to predict (empty renders as `` for unsupervised + work). - It encodes the hard constraints the model must follow: fit the model *as a catalog entry* with `xorq.ml` (never in a post-processing sandbox, which blocks sklearn/scipy/numpy); copy exact fit/predict signatures from the MODELING @@ -541,11 +591,18 @@ checkpoint or tagging wrappers. ## Known quirks -- **`catalog_list_display_klasses` checkpoints.** It is the one read-only `*_list*` - tool missing from the `_NO_CHECKPOINT` set (`server.py`), so `_with_checkpoint` - runs `checkpoint_catalog`, which commits with `--allow-empty` and advances the - step tag. Each call therefore appends an **empty** catalog revision — harmless - to correctness but it inflates the git history and the step count. Adding it to - `_NO_CHECKPOINT` (next to `catalog_list_summary_stats` / - `catalog_list_post_processings`) would bring it in line with the other list - tools. +- **`catalog_list_display_klasses` and `catalog_chart_errors` checkpoint.** They + are read-only but missing from the `_NO_CHECKPOINT` set (`server.py`), so + `_with_checkpoint` runs `checkpoint_catalog`, which commits with + `--allow-empty` and advances the step tag. Each call therefore appends an + **empty** catalog revision — harmless to correctness but it inflates the git + history and the step count. Adding them to `_NO_CHECKPOINT` (next to + `catalog_list_summary_stats` / `catalog_list_post_processings`) would bring + them in line with the other read-only tools. +- **Display klasses are outside the catalog repository.** `catalog_add_display_klass` + and `catalog_remove_display_klass` checkpoint, but `artifacts/display/` is not + in the repository, so the revision records nothing about them and a reset does + not undo them. +- **A build holds the project lock for its whole length.** Parallel tool calls + that build queue behind each other, and behind any build or heal the companion + is running (#186). diff --git a/docs/reactive-recalc.md b/docs/reactive-recalc.md index 9b81eb6..14b30df 100644 --- a/docs/reactive-recalc.md +++ b/docs/reactive-recalc.md @@ -1,27 +1,29 @@ # The reactive system: revise an alias, recompute its dependents -A tallyman catalog is a closed system. Nothing reads a database, and no source -file changes underneath you without your knowing. The thing that moves is *you*: -you revise an alias — a source projection or an aggregation — to a new recipe, -and now everything built on top of the old version is out of date. The reactive -system is what notices that and recomputes the followers, in dependency order, on -demand. +A tallyman catalog is a closed system. Nothing reads a database, and no data +file changes underneath you: a file enters the catalog only when you import it, +and every build reads tallyman's copy. The thing that moves is *you*: you revise +an alias (a named pointer to the latest version of an entry, such as an +aggregation) to a new recipe, or import a new version of a data file under its +source alias, and now everything built on top of the old version is out of +date. The reactive system is what notices +that and recomputes the followers (the entries that read that alias by name), in +dependency order. Terms follow [architecture.md](architecture.md#terms). This doc starts with the two things you actually do — build a small catalog, then revise an alias and recompute its dependents — and then documents the surfaces -(MCP tools and the companion's HTTP API) and the edges (the staleness axes, +(MCP tools and the companion's HTTP API) and the edges (the staleness rule, checkpoints, failure handling) underneath. The short version of how it works today: - Building an alias does not recompute anything downstream; it advances that one - alias. *Revising* an alias, by default, recomputes its stale followers in the - same atomic checkpoint (auto-recalc — see *Trigger model*). The switch is - per-project and env-overridable; turn it off for the explicit path below. + alias. *Revising* an alias, or importing new data under a source alias, by + default recomputes its stale followers in the same atomic checkpoint + (auto-recalc — see *Trigger model*). The switch is per-project and + env-overridable; turn it off for the explicit path below. - A scan tells you which followers are stale. With auto-recalc off, a separate - explicit recalc recomputes them and re-points their aliases — and that explicit - recalc is still the path for source-file drift, which the revise trigger doesn't - cover. + explicit recalc recomputes them and re-points their aliases. - A recalc that rebuilds an entry advances that entry's alias to a **new revision** (the alias history is append-only), and takes **one** catalog checkpoint for the whole walk. @@ -31,29 +33,30 @@ The short version of how it works today: Every catalog entry is content-addressed: its `content_hash` is a function of its recipe and its resolved inputs. An *alias* is a mutable name that points at the latest hash for a concept and keeps the full history of hashes it has pointed at -(`aliases.jsonl`, one line per alias: `{"alias", "latest", "history": [...]}`). +(`aliases.jsonl`, one line per alias: `{"alias", "latest", "history": [...], +"kind"}`). The kind is `source` for an alias whose versions are imported files +and `catalog` for one whose versions are computations. A recipe is a self-contained Python script that binds a top-level `expr`. It -reads a raw file with `read_project_file(...)` and reads another catalog entry with -`tracked_expr_from_alias(...)`. Build a three-node chain — a source projection, an -aggregation over it, and a filter over that: +reads other catalog entries, by alias, with `tracked_expr_from_alias(...)`; it +never opens a file. Data enters through `catalog_import_source`, which makes each +version of a file an entry of its own under a source alias. Build a three-node +chain — an imported file, an aggregation over it, and a filter over that: ```python -from tallyman_mcp.server import catalog_create +from tallyman_mcp.server import catalog_create, catalog_import_source -# Node 1 — a source alias: project some columns out of a raw parquet. -catalog_create("orders", ''' -from tallyman_xorq.io import read_project_file -t = read_project_file("orders.parquet") -expr = t.select("region", "price") -''') -# -> {"hash": , "alias": "orders", "version": 1, ...} +# Node 1 — a source alias: import a parquet file. The import copies the bytes into +# the project and writes them, in file order with a last __row_order column, as +# the snapshot of a new entry. The path is recorded and never read again. +catalog_import_source("/path/to/orders.parquet", "orders") +# -> {"hash": , "alias": "orders", "version": 1, "created": True, ...} # Node 2 — an aggregation alias that FOLLOWS orders by name. catalog_create("by_region", ''' from tallyman_xorq.io import tracked_expr_from_alias t = tracked_expr_from_alias("orders") -expr = t.group_by("region").agg(total=t.price.sum()) +expr = t.group_by("region").aggregate(total=t.price.sum()) ''') # -> {"hash": , "alias": "by_region", "version": 1, ...} @@ -69,18 +72,23 @@ expr = t.filter(t.total > 1000) `catalog_create(name, code)` builds the recipe, persists the entry, and assigns the alias at **version 1**. It errors if the alias already exists (use `catalog_revise` for an existing name). There is no `project` argument on the -tool: it operates on the session's active project (set by `project_switch`); the -optional `project=` kwarg on `read_project_file`/`tracked_expr_from_alias` defaults to that same -active project, so recipes omit it. - -A recipe pulls in data through one of three functions in `tallyman_xorq.io`, and +tool: it operates on the session's active project (set by `project_switch`). The +optional `project=` kwarg on `tracked_expr_from_alias`/`pinned_expr_from_alias` +defaults to the project named in the `active_project` file, which is the +session's project unless another session has switched projects since; then the +two differ and the recipe looks its aliases up in the other project (related to +#39). Recipes omit the kwarg. + +A recipe pulls in data through one of two functions in `tallyman_xorq.io`, and which one you call is what records the dependency edge: | Function | Reads | Records | Goes stale when | |---|---|---|---| -| `read_project_file("orders.parquet")` | a raw file under `/data/` | a **source** leaf (`rel_path → digest`) | the file's bytes change on disk | -| `tracked_expr_from_alias("orders")` | a catalog entry, by **alias** | a **parent** edge, `follow=True` | the alias advances to a new head | -| `pinned_expr_from_alias(...)` | a catalog entry, by alias **or** hash | a **parent** edge, `follow=False` | never (it pinned that exact revision) | +| `tracked_expr_from_alias("orders")` | a catalog entry, by **alias** (a source alias or a catalog alias) | a **parent** edge, `follow=True` | the alias advances to a new head | +| `pinned_expr_from_alias("orders-v1")` | a catalog entry, by version reference | a **parent** edge, `follow=False` | never (it pinned that exact revision) | + +`read_project_file`, `tallyman_read_csv`, and a raw `xo.deferred_read_*` of a +file tallyman did not write are build errors that name the import to use. The follow relationship is the whole game: @@ -89,39 +97,44 @@ The follow relationship is the whole game: follow — resolves to the alias's current head at build time, and records **follow=True**. The child goes stale and recomputes when `orders` advances. This is normal chaining. -- `pinned_expr_from_alias("orders")` or `pinned_expr_from_alias("")` is the - deliberate opt-out. It accepts an alias or a hash, records **follow=False**, and - the child stays on that exact revision: it never goes stale on the alias axis, - and recalc finds it but leaves it alone. -- `read_project_file` is the root of every chain: a raw file with no catalog - identity, the leaf the graph bottoms out in. +- `pinned_expr_from_alias("orders-v1")` is the deliberate opt-out. It accepts + only a version reference (`"-v"`, 1-based into the alias's history). + It refuses a bare alias, which would pin whatever the head happened to be + (#166), and a bare content hash (ADR-011 D5), so every edge an authored recipe + records names an alias (a promoted diff's generated recipe names its two + entries by hash and records no edge). It + records **follow=False**, and the child stays on that exact revision: it never + goes stale, and recalc finds it but leaves it alone. +- Source aliases are the roots of every chain. Each version is a source entry, + an ordinary entry whose snapshot holds an imported file's rows, and the graph + bottoms out in them. So after these three calls: `orders → by_region → top_regions`, each aliased at version 1, each following its parent by name (`tracked_expr_from_alias`). ## Revising an alias and recomputing its dependents -This is the core workflow. You revise an alias — it does not matter whether it is -a source projection (`orders`) or an aggregation (`by_region`) — and then you -recompute whatever followed it. +This is the core workflow. You advance an alias — by importing new data under a +source alias (`orders`) or by revising a catalog alias (`by_region`) — and then +you recompute whatever followed it. The walk-through below is the explicit path, +which runs when auto-recalc is off (`TALLYMAN_AUTO_RECALC=0`, or +`"auto_recalc": false` in the catalog's `config.json`). With auto-recalc on, the +default, the import (or `catalog_revise`) does steps 1 to 3 itself (see *Trigger +model*). ```python -from tallyman_mcp.server import catalog_revise, catalog_scan_staleness, catalog_recalc +from tallyman_mcp.server import catalog_import_source, catalog_scan_staleness, catalog_recalc -# Revise the source alias: keep an extra column this time. -catalog_revise("orders", ''' -from tallyman_xorq.io import read_project_file -t = read_project_file("orders.parquet") -expr = t.select("region", "price", "qty") -''') -# -> {"hash": , "alias": "orders", "version": 2, ...} +# The file has new rows: import it again under the same alias. +catalog_import_source("/path/to/orders.parquet", "orders") +# -> {"hash": , "alias": "orders", "version": 2, "created": True, ...} ``` -`catalog_revise(name, code)` advances `orders` to a new hash and **version 2**. -The previous hash stays in the catalog as a forensic artifact and in -`orders`'s alias history. Nothing downstream has moved: `by_region` and -`top_regions` still point at their version-1 builds, which were computed against -`orders` v1. They are now stale. +The bytes differ from `orders` v1, so the import mints **version 2** of the +source alias under a new hash. (Importing unchanged bytes is a no-op that returns +v1.) The previous version stays in the catalog and in `orders`'s alias history. +Nothing downstream has moved: `by_region` and `top_regions` still point at their +version-1 builds, which were computed against `orders` v1. They are now stale. Step 1 — scan to see what moved: @@ -132,11 +145,11 @@ catalog_scan_staleness() ``` `by_region` followed `orders` by name, and `orders`'s head no longer matches the -hash `by_region` recorded at build, so `by_region` is **directly stale** on the -alias axis. `top_regions` followed `by_region`, whose head has *not* moved yet, so -it is not directly stale — but it is **transitively stale**: a recalc rooted at -the stale set will carry it along. `orders` itself is not stale; it is the fresh -head you just built. +hash `by_region` recorded at build, so `by_region` is **directly stale**. +`top_regions` followed `by_region`, whose head has *not* moved yet, so it is not +directly stale — but it is **transitively stale**: a recalc rooted at the stale +set will carry it along. `orders` is the fresh head you just imported, and it +reads no other entry. Step 2 — preview the recalc (always a dry run first): @@ -163,18 +176,19 @@ The walk replays `by_region` first. Because dependency order re-points `tracked_expr_from_alias("by_region")` reads the advanced parent and rebuilds against it. Both aliases now point at fresh, version-2 hashes; a re-scan is clean. -The same flow drives a revision of an *aggregation* alias. Revise `by_region` and -`top_regions` goes stale and recomputes; `orders` (its parent) is untouched. The -direction is always downstream: revising an alias makes its **followers** stale, -never its parents. +The same flow drives a revision of a catalog alias. Revise `by_region` with +`catalog_revise` and `top_regions` goes stale and recomputes; `orders` (its +parent) is untouched. The direction is always downstream: advancing an alias +makes its **followers** stale, never its parents. A source alias cannot be +revised, since there is no recipe to change: new data is a new import. -This three-step flow (revise, scan, recalc) is the explicit path — the one that -runs when auto-recalc is **off**. By default it is **on**, and `catalog_revise` -folds the recalc into itself: it advances the alias *and* recomputes its stale -followers in the same atomic checkpoint, so the scan-and-recalc dance above -collapses to a single `catalog_revise` call. See *Trigger model* for the switch -and what the auto path reports. The scan stays passive either way; it never -writes. +This three-step flow (advance, scan, recalc) is the explicit path — the one that +runs when auto-recalc is **off**. By default it is **on**, and the import or +`catalog_revise` folds the recalc into itself: it advances the alias *and* +recomputes its stale followers in the same atomic checkpoint, so the +scan-and-recalc dance above collapses to a single call. See *Trigger model* for +the switch and what the auto path reports. The scan stays passive either way: it +rebuilds nothing, opens no data file and changes no catalog state. ## When are new alias revisions created @@ -184,9 +198,12 @@ moves the head (`alias_map[name] = hash`) and appends to the history that history. So a new alias revision is created exactly when an alias head advances to a hash it was not already pointing at: -- **On create** — `catalog_create("orders", ...)` writes version 1. -- **On revise** — `catalog_revise("orders", ...)` appends version 2 for the one - alias you edited. +- **On create** — `catalog_create("by_region", ...)` writes version 1, and so + does the first `catalog_import_source(..., "orders")`. +- **On revise** — `catalog_revise("by_region", ...)` appends version 2 for the + one alias you edited. +- **On import** — `catalog_import_source(..., "orders")` with bytes that differ + from the head appends the next version of the source alias. - **On recalc** — each dependent that **rebuilds to a new hash** has its alias advanced to a new revision. In the example above, the single committed recalc appended `by_region` v2 and `top_regions` v2. This is the "new alias revisions @@ -198,13 +215,14 @@ Three edges make that precise: - A **`noop`** entry — one that replays to the *same* hash because its inputs didn't really move — gets no new revision. Its alias is left untouched (and `set_alias` dedupes a consecutive duplicate hash regardless). -- The revision is appended **per advanced alias head, not per rebuilt entry**. A - rebuilt entry that has no alias on its head — an unnamed scratch entry, or an - intermediate node referenced only by a hash pin — produces a new hash and a - `remap` entry but **zero** alias revisions. A head carrying two aliases advances - both. -- A **hash-pinned** child (`pinned_expr_from_alias("")`) re-resolves to the same - parent and is a `noop`, so it neither rebuilds nor advances its alias. +- The revision is appended **per advanced alias head, not per rebuilt entry**. + Since #154 the cone holds only current alias heads, plus any entry you named + in `roots`. A named root with no alias on it (an unnamed scratch entry, say) + produces a new hash and a `remap` entry but **zero** alias revisions. A head + carrying two aliases advances both. +- A **version-pinned** child (`pinned_expr_from_alias("orders-v1")`) re-resolves + to the same parent and is a `noop`, so it neither rebuilds nor advances its + alias. This per-alias revision is distinct from the catalog-level checkpoint. The alias head advancing is bookkeeping inside `aliases.jsonl`; the checkpoint is the single @@ -222,29 +240,29 @@ each query (cheap for a notebook-sized catalog — one small JSON read per entry ### Where the edges live -Each entry has a `manifest.json` with two fields that carry the graph: +Each entry has a `manifest.json`, and one field in it carries the graph: - **`manifest.parents`** — the resolved cross-entry edges, a list of `{hash, ref, follow}`. `hash` is the parent's build-time content hash, `ref` the - original argument (alias or hash), `follow` its read-intent. Empty for a root (an - entry that reads no other entry). -- **`manifest.sources`** — the raw-file leaves, `{rel_path: digest}`: the files the - recipe read via `read_project_file` or `tallyman_read_csv`, each with the content - digest it was built against, plus every source its parents recorded. `None` when - the entry was built under `off` identity mode (so the source axis can't be - evaluated); `{}` when the recipe read no raw files. - -`tracked_`/`pinned_expr_from_alias` write `parents`; `read_project_file` and -`tallyman_read_csv` write `sources`. Together they are the only record of the -graph. + original argument (an alias or a version reference), `follow` its read-intent. + Empty for a root (an entry that reads no other entry). + +`tracked_`/`pinned_expr_from_alias` write it, and it is the only record of the +graph. A source entry's manifest also records `provenance` (the path the file +was imported from, its digest, the reader options and the name it was imported +as), which says where its data came from and is not an edge: the file is never +read again. There used to be a second field, `manifest.sources`, a per-file digest +map that children inherited from their parents; ADR-011 D6 deleted it once every +input became an entry, since the parent edges then record everything. ### Roots, leaves, and direction This doc uses the **data-flow** convention (as in Airflow, dbt, Spark lineage): edges point downstream, the direction data moves. -- A **source / root** has no incoming edges — nothing it depends on: a - `read_project_file` raw input, or an entry with empty `manifest.parents`. +- A **source / root** has no incoming edges — nothing it depends on: a source + entry (a version of an imported file), or any entry with empty + `manifest.parents`. - A **leaf / sink** has no outgoing edges — nothing depends on it: a final aggregation nobody chains off. @@ -252,14 +270,15 @@ A recalc starts at the changed roots and flows down to the leaves. (Build-system tools — Make, Bazel — point the arrows the other way and swap these names, which is the usual source of confusion.) -### Cheap vs worthy parents — and what lands in `manifest.sources` +### Cheap vs worthy parents When a recipe reads a parent with `tracked_expr_from_alias`, what comes back depends on whether the parent was classified **worthy** or **cheap** at build. This is the materialized-vs-not distinction: - A **worthy** parent (an aggregate, join, sort, window function, union, distinct, - unnest, a second file, a non-pure operation, or a UDF) is materialized: its + unnest, a second file, a non-pure operation, or a UDF, and every source entry) + is materialized: its result was written to a snapshot file when it was built. A child reading it gets a *bare read of that snapshot* — the parent's work is computed once and shared, and the child does not re-run it. The snapshot's path contains the parent's @@ -268,15 +287,12 @@ This is the materialized-vs-not distinction: - A **cheap** parent (row-preserving over one file: select, filter, mutate) has no file of its own. A child reading it gets the parent's *frozen graph inlined* — pushdown makes re-running it ~free — so the parent's expression, down to the - ordered copy of its source, is composed into the child. + snapshot it reads, is composed into the child. -Either way, building the child folds the parent's recorded `sources` into the -child's own `manifest.sources`, so the source digests a child depends on are -recorded on the child and its clones stay referenced as long as it survives. When -the parent is cheap the child also records the parent's `ordered_copies`, since -the child's build reads those copies itself. Editing a raw file that anywhere up -the chain feeds the child therefore makes the child **directly** stale on the -source axis, not merely transitively stale. +Either way the child records one edge, to the parent it named, and nothing about +the parent's own inputs. New data reaches it through the aliases: an import +advances the source alias, the entries that follow it go directly stale, and the +entries built on those are transitively stale. Which work is materialized is otherwise an implementation detail you don't see: `tracked_expr_from_alias` returns an expression either way, and the child never @@ -288,12 +304,11 @@ it has to. ### Build-time capture, not live introspection The edges are captured the moment a recipe runs, not by walking the built -expression afterward. While `build_and_persist` imports the recipe, two collectors -are armed, and each loader announces itself as it executes: `read_project_file` -and `tallyman_read_csv` note the source digest, `tracked_`/`pinned_expr_from_alias` -notes the resolved parent hash. After the import those bags are written into the -manifest. The capture -happens once, at build; nothing re-derives it later. +expression afterward. While `build_and_persist` imports the recipe, one +collector is armed (`parent_capture`), and `tracked_`/`pinned_expr_from_alias` +note the resolved parent hash in it as they execute. After the import the +collected edges are written into the manifest. The capture happens once, at +build; nothing re-derives it later. This is a **tallyman** mechanism, not an xorq one. xorq's unit is a single expression and its content hash; it has no concept of a catalog, an alias, an @@ -320,8 +335,9 @@ pins, and those live only in the manifest too. All graph reads go through `tallyman_xorq.dependents`, which scans the manifests and assembles the views the reactive system needs: -- `parents_of(hash)` / `sources_of(hash)` — one entry's recorded inputs. -- `build_dag()` — forward edges for every live entry, `{child: [parents]}`. +- `parents_of(hash)` — one entry's recorded parent edges. +- `build_dag()` — forward edges for every complete entry (one with a + `manifest.json`), `{child: [parents]}`. - `dependents_index()` — the reverse index, `{parent: {children}}`: how a recalc goes from a changed entry to everything downstream. - `descendant_cone(roots)` — the roots and all their transitive dependents, @@ -331,65 +347,50 @@ Both `staleness` and `recalc` go through this module rather than touching manife fields directly, so it is the single seam over the recorded graph: swapping the implementation (a persistent index, say) is a change to `dependents` alone. -One consequence, and a candidate future check: because a cheap parent's -edge is gone from xorq's flattened expression, you can't fully reconcile tallyman's -recorded parents against xorq's graph. What you *can* reconcile is the source leaves — a -child's `sources` should equal its own raw-file reads plus the union of `sources` -over its parents. That invariant is checkable; the parent edges are only as -good as what the side-channel captured at build. - -## The two staleness axes - -The workflow above drives the **alias axis**. There is a second axis for source -files that change on disk, which a closed notebook catalog mostly doesn't hit, but -it exists and is worth knowing. - -An entry is judged on two independent axes, each tied to a kind of recorded input: - -- **alias** — a `follow=True` parent (recorded when the recipe referenced a parent - by alias name, `tracked_expr_from_alias("orders")`) is stale when that alias now resolves to - a different hash than the one recorded at build. This is what fires when you - revise an upstream alias. A `follow=False` parent (a `pinned_expr_from_alias` - reference, by alias or hash) is never stale on this axis: the recipe asked for - *that* revision and still gets it. -- **source** — a recorded `(rel_path, digest)` is stale when the file on disk no - longer digests to the recorded value. The check forces a faithful re-read so an - in-place content swap can't be masked by a cached digest. In a closed catalog - you don't expect this to fire; it covers the case where a raw parquet under - `data/` is replaced. - -A note on chains: a child's `manifest.sources` carries the source files its parents -recorded, whether each parent is cheap or worthy (see *Cheap vs worthy parents* -above). Editing such a source makes the child **directly** stale on the source -axis, not merely transitively stale. This is expected. +One consequence: because a cheap parent's edge is gone from xorq's flattened +expression, you can't fully reconcile tallyman's recorded parents against xorq's +graph, and the parent edges are only as good as what the side-channel captured +at build. + +## One staleness axis + +An entry is stale when a `follow=True` parent — recorded when the recipe +referenced a parent by alias name, `tracked_expr_from_alias("orders")` — now +resolves to a different hash than the one recorded at build, and for no other +reason (ADR-011 D6, `plans/ADR-011-sources-are-aliases.md`). This is what fires +when you revise an upstream alias or import new data under a source alias. A +`follow=False` parent (a `pinned_expr_from_alias("orders-v1")` reference) is +never stale: the recipe asked for *that* revision and still gets it. + +A data file that changes outside tallyman is not a reason. Until it is imported +again nothing in the catalog has changed, and the builds read tallyman's copy of +the bytes, never the file. So the scan opens no data file; it reads +`aliases.jsonl` and the manifests. There used to be a second axis, a recorded +source digest no longer matching the file on disk. It could not tell how an +entry had come to depend on a file, so a child pinned to its parent by hash read +as stale forever, and the import model removed it. + +Only an entry that is the current head of an alias can be directly stale (#154). +A superseded version still records inputs that have since moved, but it heads no +alias, so recomputing it would re-point nothing; the scan reports it with +`live: false` and `stale: false`, and keeps its `reasons` for the record. + +A note on chains: an entry records only its own parent edges, so an import makes +the direct followers of the source alias **directly** stale and everything built +on them **transitively** stale, whether the parents in between are cheap or +worthy. `result_digest` is deliberately not a staleness input. An entry that recomputes to the same `content_hash` but a different result is *nondeterministic*, not stale, and recompute can't make it fresh. That is the #83/#121 concern, separate from this system. -### Prerequisite: `cas` source identity (source axis only) - -Source-axis staleness only works under content-addressed source identity, which is -the default. `source_identity.mode()` reads `TALLYMAN_SOURCE_IDENTITY` and falls -back to `"cas"`. In `cas` mode, `read_project_file` and `tallyman_read_csv` take -the source's content digest, clone it copy-on-write to `data/.cas/`, -and read an ordered copy of the clone whose file name is a function of the digest. -The path xorq hashes is therefore the content identity: editing a source in place -yields a different digest, the manifest records that digest at build time, and a -later scan can tell the file moved. The clone also means an old entry's copy can be -made again from the bytes it was built from after you edit the source. - -Under the legacy `off` mode the manifest records no source digests, so the source -axis cannot be evaluated. A scan reports it as `unknown` rather than silently -fresh. If you have a corpus built under `off`, rebuild it — there is no migration -path and none is wanted (single-user repo). The **alias axis is unaffected** by -this mode; it works regardless. - ## Detecting staleness (the scan) -The scan classifies every live entry and tells you two things per entry: whether it -is **directly stale** (its own recorded inputs moved) and whether it is +The scan classifies every complete entry (every entry directory with a +`manifest.json`, current alias heads and superseded versions alike) and tells +you two things per entry: whether it is **directly stale** (its own recorded +inputs moved, and it is a current alias head) and whether it is **transitively stale** (a clean entry sitting downstream of a directly stale ancestor, so a recalc would carry it along). @@ -411,17 +412,22 @@ Read-only, takes no checkpoint. It returns: "stale": true, "reasons": [{"axis": "alias", "ref": "orders", "was": "", "now": ""}], "unknown_axes": [], - "transitively_stale": false + "transitively_stale": false, + "live": true // the entry is a current alias head }, ... - } + }, + "orphan_stale": [ ... ] // empty unless auto-recalc is on; see Trigger model } ``` -Each `reason` names the axis, the `ref` (alias name or source rel_path), the value -`was` at build time, and the value `now` (or `null` if the input is gone). An axis -that can't be evaluated — alias deleted, source file removed, or the entry was -built under `off` mode — lands in `unknown_axes` and never forces `stale: true`. +Each `reason` names the axis (always `"alias"`; the field is kept so a reason +has the same shape everywhere), the `ref` (the alias name), the parent hash that +`was` recorded at build, and the alias head `now`. A parent alias that no longer +exists can't be evaluated; it lands in `unknown_axes` as `alias:` and never +forces `stale: true`. `catalog_scan_staleness(verify_results=True)` +also checks every snapshot on disk against its recorded `result_digest`; see +[mcp-server.md](mcp-server.md). ### HTTP @@ -432,22 +438,24 @@ GET /{project}/api/staleness Read-only, same data shape plus the project name: ```json -{ "project": "...", "stale": [...], "transitively_stale": [...], "entries": {...} } +{ "project": "...", "stale": [...], "transitively_stale": [...], "entries": {...}, "orphan_stale": [...] } ``` -This is the scan-on-load surface. The SPA is expected to fire it on project load -and after a recalc to drive a per-entry badge (the badge UI itself is not built yet -— see Current limits). +This is meant as the scan-on-load surface for a per-entry staleness badge. The +SPA does not call it yet, and there is no badge (see Current limits). ## Recomputing (the recalc) Recalc takes a set of *roots*, computes their descendant cone (every entry -reachable downstream, topologically ordered parents-before-children), and replays -each recipe in that order through `build_and_persist` — the same primitive -`catalog_revise` uses. A replay reads the *current* alias heads, so once a parent -has been re-pointed to its fresh hash, its dependents replay against the advanced -parent. An entry whose inputs didn't actually move replays to the same -`content_hash` and short-circuits to a no-op. +reachable downstream through recorded parent edges, topologically ordered +parents-before-children), drops from it every entry that is not a current alias +head unless it was named as a root (#154: replaying a superseded version would +only make an entry nothing points at), and replays each recipe in that order +through `build_and_persist` — the same primitive `catalog_revise` uses. A replay +reads the *current* alias heads, so once a parent has been re-pointed to its +fresh hash, its dependents replay against the advanced parent. An entry whose +inputs didn't actually move replays to the same `content_hash` and +short-circuits to a no-op. ### Preview, then commit @@ -459,7 +467,8 @@ Dry-run actions: - `rebuild` — directly stale, its own inputs moved. - `cascade` — clean, but a followed parent inside the cone will advance. -- `unchanged` — pinned to an out-of-cone parent, or a root that isn't stale. +- `unchanged` — not stale itself, and no followed parent inside the cone: a child + that pins a version of its parent, or a root that isn't stale. A real run (`dry_run=False`) replays for real and reports: @@ -514,10 +523,11 @@ Content-Type: application/json Both body fields are optional; `dry_run` defaults to True and `roots` defaults to the stale set, matching the MCP tool. The response is the same report (without the -`note` field on the empty case). A committed run additionally invalidates the -companion's result/compare caches, reloads any live buckaroo sessions, and -publishes a `recalc` SSE event. The route is blocked with a 403 in read-only -(serve) mode. +`note` field on the empty case). A committed run that changed something +additionally clears the companion's result and compare memos and its +`diff_stat_cache/` directory, reloads the project's Buckaroo sessions (one +`/reload_expr` per catalog entry, #201), and publishes a `recalc` SSE event. The +route is blocked with a 403 in read-only (serve) mode. ### One revision, undoable atomically @@ -531,13 +541,14 @@ it was, rolling back the whole cascade at once. ### Failure and cycles -The cascade-failure policy is **stop-and-report**, the plan's default. If a replay -raises, the walk stops at that entry: the already-recomputed prefix stays committed -(so its checkpoint still records), the failing entry is reported with +The cascade-failure policy is **stop-and-report**. If a replay raises, the walk +stops at that entry: the already-recomputed prefix stays committed (so its +checkpoint still records), the failing entry is reported with `action: "failed"` and the error text, and everything after it is `skipped`. The call does not raise — a caller gets a structured `status: "failed"` report, not a -500. The two other policies in the ADR (skip-and-continue, all-or-nothing) are not -implemented. +500. The recalc design also considered skip-and-continue and all-or-nothing; +neither is implemented. The failure, and every skipped entry that was itself +directly stale, is recorded in `errors.jsonl` under its hash. If the dependency graph contains a cycle, recalc detects it while building the cone, before any recipe runs, and returns `status: "cycle"` with the cycle described in @@ -574,21 +585,19 @@ event's `kind`. A client must therefore register a **named** listener, What works today is the backend, end to end, through both the MCP and HTTP surfaces: scan, preview, commit, atomic revision, cross-process invalidation, and -the published SSE event. What is not yet built: - -- **Frontend rendering.** The SPA's `SSEContext` listens for nine named events - (`hello`, `ping`, `new_entry`, `build_failed`, `notebook_changed`, - `chart_attached`, `post_processing_changed`, `summary_stat_changed`, - `project_switched`) but **not** `recalc` — the backend emits it, nothing consumes - it, so open views do not auto-refresh after a recalc yet. Auto-recalc-on-revise - does not change this: it emits the same `recalc` event, which the SPA still - ignores, so a view open on a cascaded dependent needs a manual refresh. Wiring the - listener is the one piece that makes the cascade visible in the viewer. There is - also no staleness badge and no recalc affordance in the UI, and nothing calls - `/api/staleness` or `/api/recalc` from the frontend. For now, drive recalc through - MCP or a direct HTTP call and refresh the view manually. -- **Deletion-side semantics.** Archive / delete-cone behavior and multi-parent - unbind (ADR Q3/Q4) are out of scope for this system as built. +the published SSE event. The SPA listens for `recalc`: each event refetches the +catalog list, and an entry view open in a background tab whose hash was remapped +navigates to the new hash. A focused tab stays put, so that a code edit in +progress is not thrown away; its URL still names the old hash, so a reload shows +the old version too, and the new one is reached from the refreshed catalog list. +What is not yet built: + +- **Staleness and recalc in the UI.** There is no staleness badge and no recalc + button, and nothing in the SPA calls `/api/staleness` or `/api/recalc`. Drive an + explicit recalc through MCP or a direct HTTP call. +- **Deletion-side semantics.** Archive and delete-cone behaviour, and unbinding + one parent of an entry that has several, are out of scope for this system as + built. - **Alternative cascade policies.** Only stop-and-report exists. ## Trigger model @@ -600,12 +609,17 @@ scan-then-recalc path. The scan surfaces (`catalog_scan_staleness` / ### Auto-recalc on revise (default) Revising an alias recomputes its stale followers in the **same** checkpoint as the -head advance. Every surface that re-points an *existing* alias triggers it — -`catalog_revise`, the companion `PUT …/api/code/{alias}`, and `catalog_promote_diff` -when it re-points an existing target. The cascade is scoped to *this* revise's -followers: roots are the entries the advance made directly stale on the alias axis, -and the descendant cone expands from there. Pre-existing staleness from any other -cause is left untouched. +head advance. Every surface that advances an *existing* alias to a new version +triggers it — `catalog_revise`, the companion `PUT …/api/code/{alias}`, a diff +promotion (`catalog_promote_diff` or the companion's promote route) when it +re-points an existing target, and `catalog_import_source` when it mints a new +version of a source alias. `catalog_alias`, which points a name at an entry that +already exists, does not. The cascade is scoped to *this* advance's followers: +roots are the entries it made directly stale, and the descendant cone expands +from there. Pre-existing staleness from any other cause is left untouched. +The companion's two routes build the new entry on its event loop, so the whole +UI stops answering until that build (and any wait for the project lock) is over +(#190); the cascade itself runs on a worker thread. The whole thing is one git revision. The head advance, the rebuilt dependents, and their re-pointed aliases are swept into the surface's single per-operation @@ -616,8 +630,8 @@ the rest are skipped. The revise reply carries a `recalc` sub-report — `catalog_recalc`'s shape (`cone`, `entries`, `remap`, `status`) plus `orphan_stale`. `orphan_stale` lists every entry -that was directly stale but *not* a follower of this revise (a drifted source, a -different alias not yet recalced, or an invariant break): not recomputed, logged at +that was directly stale but *not* a follower of this revise (a different alias +not yet recalced, or an invariant break): not recomputed, logged at WARNING, and classified against the durable error store — "explained by error ``" when a recorded recalc/build failure carries that hash, else "UNEXPLAINED … file a bug." Cascade failures are persisted to `errors.jsonl` (keyed by the @@ -634,5 +648,6 @@ path below. With the switch off, advancing an alias marks its followers stale but recomputes nothing. `catalog_recalc` / `POST …/api/recalc` is then the deliberate action that writes — preview with a dry run, commit with `dry_run=False`. This is also the path -for staleness the auto trigger never sees: a drifted source file, or any alias -left stale because auto-recalc was off when it was revised. +for any alias left stale because auto-recalc was off when it was revised or +imported. A source entry passed as a root fails instead of being skipped +(#238). diff --git a/docs/system-contract.md b/docs/system-contract.md index 02c115f..0a2170d 100644 --- a/docs/system-contract.md +++ b/docs/system-contract.md @@ -2,12 +2,21 @@ - **Status:** Normative, implemented. Describes the system as built; where code or the descriptive docs disagree with it, the disagreement is a bug. + The known disagreements, each with its open issue, are listed under + [Known deviations](#known-deviations) at the end. Design decisions and history: `plans/ADR-006-read-path-loads-builds.md` (the read path, still in force for what the later ADRs did not change) and `plans/ADR-007-tallyman-owned-materialization.md`, `plans/ADR-008-row-order-of-reads.md` and `plans/ADR-009-digest-stability.md` (tallyman writes its own result files, - every file carries `__row_order`, and the digest is a content digest). + every file carries `__row_order`, and the digest is a content digest), + accepted on 2026-09-22 and implemented in #189. + `plans/ADR-010-immutable-store-one-owner.md`, a later proposal to replace + those three, was rejected the same day. + `plans/ADR-011-sources-are-aliases.md` (a raw input is a source alias whose + versions are entries, a file enters only by an explicit import, recipes name + aliases and never bare hashes, and staleness has one axis), accepted the same + day and implemented in #217, #218 and #219. - **Audience:** no prior xorq knowledge assumed. The xorq section below covers exactly as much of xorq as the rest of the doc needs, and no more. @@ -73,7 +82,8 @@ it: the build forbids the authoring patterns that create such nodes (in-memory reads are a build error), so every leaf is a file read and the refusal is a loud assertion that the build gate failed, never a live copy path. Beside it, rebinding fails loudly if a build ever spans more than one -distinct backend content profile (ADR D3). +distinct backend content profile (ADR-006 D3, rebind composition onto the +default backend with a one-group guard). ## 3. Builds: freezing an expression to disk @@ -104,7 +114,7 @@ Three properties matter: (build artifacts should be reproducible from what the expression *is*), and it is the single most consequential xorq design choice for tallyman: **if you want content identity, you must put the content in the path.** Tallyman - does (Part 2, CAS). + does (Part 2, "Project, imports and source entries"). `load_expr(build_dir)` is the inverse: it reads the yaml and mints **fresh backend objects** from the profiles (nothing is memoized — two loads of the @@ -151,75 +161,109 @@ adds the output axis itself (Part 2, `result_digest`). answer to new — built to serve the current file's answer fast, nothing more. Tallyman needs the opposite: entries are history, so identity must hold still while data moves (that is what makes V3-vs-V4 meaningful), and - change is handled explicitly instead — the manifest records what the - world *was* (source digests, parent hashes), the staleness scan detects - by comparing that record against the live world, recalc mints new entries - and advances aliases, and old entries keep serving their old bytes - forever. A key whose job is to move when data moves has nothing to hang - that on. Bolting it under tallyman's caches would also collide with - identity mechanically: the expression hash is hardwired to path-only - normalization (measured in the source-identity ADR), so mtime-keyed - caches would move while entry hashes stood still — fresh bytes under an - old name, the #163 failure shape. CAS serves both needs with one - mechanism: content in the path gives the hash and every cache key an - identity that moves exactly when content moves, and the recorded digests - give staleness something durable to compare. (One stat use survives, as - an accelerator: the source-digest memo skips re-hashing files whose stat - is unchanged; content stays the truth.) + change is handled explicitly instead — new data is imported as a new + version of a source alias, the manifest records which version of each + parent an entry was built on, the staleness scan compares that record with + the alias heads, recalc mints new entries and advances aliases, and old + entries keep serving their old bytes forever. A key whose job is to move + when data moves has nothing to hang that on. Bolting it under tallyman's + caches would also collide with identity mechanically: the expression hash + is hardwired to path-only normalization (measured in the source-identity + ADR), so mtime-keyed caches would move while entry hashes stood still — + fresh bytes under an old name, the #163 failure shape. Content-named files + serve both needs with one mechanism: the hash and every cache key move + exactly when content moves, and an import is the one event that moves + them. --- # Part 2 — The tallyman primitives -## Project, sources, and CAS +## Project, imports and source entries A **project** is a directory (`~/.tallyman-notebooks/projects//`) holding -user data files (`data/`) and the catalog (`artifacts/catalog/`, a git repo). - -A **source** is a user-provided data file. Sources are mutable on disk — the -user can overwrite `trips.parquet` any time — so tallyman never lets an entry -depend on a live source path. At build time each source is: - -1. digested (md5, memoized on stat so unchanged files hash once), -2. cloned copy-on-write to `data/.cas/`, -3. read through the clone. - -This is **CAS** (content-addressed sources), and it is the answer to xorq's -path-only hashing: the path *is* the digest, so every xorq-level key (the -expression hash, and the name of every file tallyman writes) becomes -content-honest for free. Edit a source and rebuild: the digest changes, the path -changes, the hash changes, a new entry forks. The clone is the entry's immutable -input forever; the live file is merely where the *next* build will look. - -A recipe never reads the source or its clone directly. Ingest writes an -**ordered copy** of the clone under the project's `compute_cache/ordered_sources/`: -polars reads the clone in file order and writes a parquet file with the same -columns and one more at the end, `__row_order`, `0..N-1`. The copy is named by -the source's digest and the reader options (the schema and `scan_csv` options -of a CSV), and the recipe reads that file. So a recipe's reads are all files -tallyman wrote, each carrying the column that pages sort by (Part 2, "Row -order"). +the catalog (`artifacts/catalog/`, a git repo) and the clone store +(`data/.cas/`). + +**A recipe names aliases. A build reads only files tallyman owns.** A data file +the user has is mutable and outside tallyman's control, so no build ever reads +one. It enters the catalog by one explicit act, an **import** +(`catalog_import_source`, `source_import.update_and_depend`), which: + +1. digests the file (md5) and clones its bytes, copy-on-write where the + filesystem offers it, to `data/.cas/` (the **clone**), + digesting the clone again and refusing it if it does not match its name; +2. writes one parquet file of its rows, in file order plus a last `__row_order` + column, `0..N-1`, to `compute_cache/result_cache/.parquet`: + pyarrow copies a parquet file, keeping its types, and polars parses a CSV + under the schema and `scan_csv` options the call names (a timestamp column + whose schema type has a zone keeps the wall-clock time of text with no UTC + offset, converts text with one into the zone, and refuses a time the zone + skips or repeats and a column that mixes the two kinds of text); +3. writes an entry, the **source entry**, with a generated recipe, a frozen + build, a schema and a manifest whose `provenance` records the outside path, + the digest, the reader options and the name it was imported as; +4. appends that entry to a **source alias**, as its next version. + +The outside path is provenance from then on and is never read again: editing, +moving or deleting the file changes no build. To bring in new data, import the +file again under the same alias. Different bytes mint the next version, and the +entries that follow the alias go stale exactly as after a revise. The same bytes +are a no-op. + +A source entry's content hash is `md5("source||")`, +truncated to 12 hex characters: the bytes and the reader options, and nothing +else. This matters because xorq hashes a file read by its path alone. Every file +a recipe's expression reads is a snapshot, named by the content hash of the +entry it holds, so every xorq-level key (the expression hash, and the name of +every file tallyman writes) is content-honest, and the chain of names ends at +hashes of bytes. The reader options are fixed at import (a CSV read two ways is +two imports under two aliases), and they must be plain values that the entry can +record: a callable option is refused. + +The import's outcomes form a table (ADR-011 D3). With no `pinned_version`: a new +alias mints v1, bytes that differ from the head mint the next version, and bytes +equal to the head are a no-op. With `pinned_version=N`: the file must be version +N (a no-op) or, as N = head + 1, new bytes; anything else is an error, and +versions cannot be skipped. History is append-only, so bytes equal to a version +older than the head are refused, naming `reset_to` as the way back. And **one set +of bytes, read one way, is one version under one alias**: bytes another alias of +the project already holds are refused, naming that alias; a second name for them +is a catalog entry whose recipe reads `tracked_expr_from_alias("")`. +The rule is per project: two projects importing one file each hold their own +entry, snapshot and clone under the same hash. ## Recipe A **recipe** (`expr.py`) is the LLM-authored Python that defines a computation. It must bind a variable `expr`, built from: -- `read_project_file("trips.parquet")` — read a source (through CAS); -- `tracked_expr_from_alias("trips")` — build on another entry, named by alias, - recording a `follow=True` parent edge: this expression depends on the parent - alias, and when that alias advances, recalc mints a new version of this - expression; -- `pinned_expr_from_alias()` — same, but `follow=False`: - recalc never touches this expression when the parent moves. Pins name an - exact version — a hash, or an explicit `"trips-v3"` version reference; a - bare alias is rejected as ambiguous (it would silently pin whatever the - head happened to be when the recipe was built — #166). - -The defining property of a recipe: **it binds by name.** "trips.parquet" -means whatever bytes sit there right now; "trips" means whatever entry that -alias points at right now. A recipe therefore has a different meaning at -different moments. That is exactly what you want when *authoring* — and +- `tracked_expr_from_alias("trips")` — build on another entry, named by alias + (a source alias or a catalog alias), recording a `follow=True` parent edge: + this expression depends on the parent alias, and when that alias advances, + recalc mints a new version of this expression; +- `pinned_expr_from_alias("trips-v3")` — same, but `follow=False`: recalc never + touches this expression when the parent moves. A pin names an exact version + with an explicit version reference. A bare alias is rejected as ambiguous (it + would silently pin whatever the head happened to be when the recipe was built, + #166), and so is a bare content hash (ADR-011 D5), so every parent edge an + authored recipe records names an alias, and no opaque hash appears in one. An + entry with no alias has to be named before anything can build on it. The one + generated exception is a promoted diff, whose recipe calls + `build_diff_expr(a_hash=..., b_hash=...)` with the two hashes and records no + parent edge (below, "Composition"). + +A recipe never opens a file. `read_project_file`, `tallyman_read_csv`, +`xo.deferred_read_csv`, and `xo.deferred_read_parquet` of a file outside +`compute_cache/` are build errors, each naming the import to use. The one recipe +that calls `read_project_file` is the one the importer generates for a source +entry, where a context variable resolves the call to that entry's own snapshot; +the path it names is provenance. + +The defining property of a recipe: **it binds by name.** "trips" means whatever +entry that alias points at right now, and for a source alias that is whichever +version of the file was imported last. A recipe therefore has a different +meaning at different moments. That is exactly what you want when *authoring* — and exactly what you must never consult again afterward. ## Entry @@ -233,7 +277,7 @@ entries// xorq_build/ # the frozen build (Part 1 §3), paths made portable manifest.json # the closure record (below) schema.json -entries/.zip # git-tracked durable form, written at checkpoint +entries/.zip # git-tracked record, written at checkpoint, never read back ``` An entry carries **two representations of its computation, with different @@ -241,7 +285,7 @@ authority**: | | `expr.py` (recipe) | `xorq_build/` (build) | |---|---|---| -| binds inputs by | name (aliases, live paths) | value (parent graphs inlined, CAS paths) | +| binds inputs by | name (aliases) | value (a worthy parent's snapshot path, a source version's included; a cheap parent's graph inlined) | | meaning over time | drifts as names move | fixed forever | | authoritative for | authoring: revise, display, the *next* build | semantics: every read, forever | @@ -254,6 +298,10 @@ resolving names to current heads is the point. (#163 is what happens when this rule is broken: reads re-ran recipes, so historical entries silently re-bound to today's parents.) +A source entry has both representations too, and the rule holds for it +trivially: its recipe is generated by the import and never re-run to make +anything, and the build the import records reads its own snapshot. + ## Content hash An entry's `content_hash` is xorq's expression hash (Part 1 §3), taken at @@ -261,24 +309,27 @@ build time. Two details determine what it covers: - the expression hashed is the one after the rewrite (below), which adds the canonical sort to a worthy entry and adds no cache node to anything; -- every file read in the graph is a file tallyman wrote under a name that - carries content: an ordered copy named by its source's digest, or a worthy - parent's snapshot named by the parent's content hash. So the hash covers the - input bytes, and a child's hash is a function of its parent's. +- every file read in the graph is a worthy parent's snapshot, named by the + parent's content hash, so a child's hash is a function of its parent's. + +A source entry's hash is the exception, and the base of every chain: it is +computed from the imported bytes and the reader options (Part 2, "Project, +imports and source entries") rather than taken from xorq, because its generated +recipe reads the snapshot that the hash names. -The hash therefore names "this computation over these exact input bytes." +So the hash names "this computation over these exact input bytes." That one property makes builds idempotent and history append-only. Rebuild the same computation over unchanged inputs and you land on the existing entry: the build recognizes the hash and stops. Change anything that alters -the computation or its inputs — the recipe's logic, a source's bytes, a -parent's identity — and a new entry forks under a new hash, while every existing -entry keeps its name and its meaning. +the computation or its inputs — the recipe's logic, a parent's identity, and +through a new source version the imported bytes — and a new entry forks under a +new hash, while every existing entry keeps its name and its meaning. Two limits are deliberate. The hash cannot see execution behavior (Part 1 §5): a recipe calling `sample()` or `now()` hashes identically run to run, which is the gap `result_digest` (below) exists to police. And it makes no -attempt at cross-machine portability: absolute path prefixes participate in the -hash. +attempt at cross-machine portability: absolute path prefixes participate in a +computed entry's hash. (A source entry's hash has no path in it.) ## Manifest: the closure record @@ -289,23 +340,43 @@ itself doesn't state, so that no later operation ever needs to resolve a name: |---|---| | `content_hash` | the entry's identity | | `parents` | `[{hash, ref, follow}]` — each alias reference, **resolved to the exact hash it meant at build time** | -| `sources` | `{rel_path: digest}` — each source, pinned to the bytes read | -| `ordered_copies` | `{key: {source, digest, reader, content_digest}}` — each ordered copy the plan reads, with the reader options and the digest that let it be made again | +| `provenance` | a source entry only: `{alias, version, path, digest, suffix, reader, imported_at}` — the name it was imported as, the outside path (never read again), the digest of the bytes, and the reader options that let its snapshot be made again from the clone; its presence is what makes an entry a source entry | | `cache_worthy`, `cache_worthy_why`, `cache_bytes` | whether the entry is materialized, decided once at build, and the evidence | | `result_digest` | `arrow-sha256:` digest of the snapshot's content (worthy entries) — the output identity | | `reproducible`, `nonreproducible_columns` | whether two runs at create gave the same digest, and the columns that differed | +| `unfaithful_heal_digest` | the digest the last unfaithful heal wrote; set, it pins the snapshot. The only field written after create | | `snapshot_format`, `engine_versions` | the format version and the xorq, xorq-datafusion and pyarrow versions at build | | `row_count`, `execute_seconds`, `compile_seconds`, timings | build measurements | -The manifest is written last, atomically: its presence is the "this entry is -complete" sentinel. An entry directory without one is treated as absent. +The manifest is the entry directory's last write, atomic: its presence is the +"this entry is complete" sentinel. After that only an unfaithful heal rewrites +it, to record `unfaithful_heal_digest`, by an atomic replace under the project +lock. The recipe zip, written by the first checkpoint, keeps the manifest as it +was at create. An entry directory without a manifest is treated as absent by the +entry list, the checkpoint, recalc and the build, which builds it again. Every +read refuses it: `result_cache.entry_manifest` raises a `BuildError` naming the +missing `manifest.json` before anything is loaded or written, a child that reads +the entry raises the same error, and no route answers around it. Nothing stands +in for the manifest's `cache_worthy`. ## Alias -An **alias** is a mutable name: `{alias, latest, history}` in a git-tracked -file. `latest` is the head; `history` is every hash it has pointed at (V1…Vn, -oldest first). Revising an alias mints a new entry, advances `latest`, appends -to `history`. Old entries remain, immutable, as the version history. +An **alias** is a mutable name: `{alias, latest, history, kind}` in a +git-tracked file. `latest` is the head; `history` is every hash it has pointed +at (V1…Vn, oldest first). Revising an alias mints a new entry, advances +`latest`, appends to `history`. Old entries remain, immutable, as the version +history. + +An alias has a **kind**. A **catalog alias** names computations and advances by +revise, promote and recalc. A **source alias** names imported data and advances +only by an import: there is no recipe to revise and nothing to promote onto it, +and every surface that offers those refuses a source alias before building +anything. A name is one kind or the other, never both. **An alias's kind +matches its entries' kind**: `set_alias` never points a catalog alias at a +source entry or a source alias at a computed one, whichever route asks. +A source version is named by the alias that holds it now: `provenance` keeps +the name it was imported as, and a rename or an unalias does not rewrite it, so +a message that names a version or advises an import asks the alias store. **Where alias resolution is legal** — names resolve in exactly three situations, all of them *about* choosing or minting, never about serving: @@ -314,9 +385,8 @@ situations, all of them *about* choosing or minting, never about serving: hashes in the new entry's manifest. 2. **Selecting** (UI, diff version arithmetic): resolve "by_hour" or "V-1" to a hash, *then* serve that hash. -3. **Judging** (staleness scan): compare recorded parent hashes and source - digests against current heads and current files — read-only, executing - nothing. +3. **Judging** (staleness scan): compare recorded parent hashes against + current heads — read-only, executing nothing and opening no data file. Once an entry is selected, serving it consults no name again. `follow` is a **recalc** policy (should a parent's advance mint a new version of this @@ -353,9 +423,10 @@ The canonical sort, described next, is added only to a worthy entry. Every file tallyman writes ends in an `int64` column named `__row_order` holding `0..N-1` in the file's physical row order. It is the last column, and it is -visible in every table. Two writers produce it: ingest (an ordered copy of a -source) and `materialize` (a snapshot), and each materialization overwrites an -inherited one with positions in its own file. It is what makes a page of an +visible in every table. Two writers produce it: the import (a source entry's +snapshot, numbered in the imported file's order) and `materialize` (every other +snapshot), and each overwrites an inherited one with positions in its own +file. It is what makes a page of an entry a function of `(content_hash, sort, offset, limit)`: with no user sort a page is `ORDER BY __row_order`, and with one the user's keys come first and `__row_order` is the last key, which breaks every tie. @@ -369,28 +440,53 @@ page is `ORDER BY __row_order`, and with one the user's keys come first and them: the writer numbers the result in that order. - Every `order_by` in a recipe gets `__row_order`, then the remaining sortable columns, as its last keys, so the sort is total wherever the recipe put it. A - sort that is not the recipe's last step is kept: the top-level sort leads with + sort followed only by steps that keep row order (filters, limits, selections, + column drops, drops or fills of nulls) is kept: the top-level sort leads with its keys, and the build fails, naming the key, if a later step dropped or - changed it. + changed it or if the sort was by an expression rather than a column. Above an + aggregate, a join or a union the order of rows is gone, and the top-level sort + is the tie-break alone. - A join of two entries leaves the right side's copy under ibis's collision name, `__row_order_right`, and the writer drops it. Joining three entries in one recipe needs `.drop("__row_order")` on the right-hand inputs, and the build - says so. -- A diff carries no row-order column from either side. + says so. (Today the writer drops any column of that name, including an + author's, #206, and the check also refuses semi and anti join chains, which + cannot collide, #199.) +- A diff carries no row-order column from either side. The compare grid and a + promoted diff drop it; `full_diff`, behind the diff page's summaries and + `catalog_diff`, does not yet (#200). ## Materialization -`materialize(project, hash)` is the one routine that writes a snapshot. The -build calls it and so does every heal, so result bytes are manufactured in one -place. It runs the entry's frozen build on a **single-partition** connection -(so a float total is merged in one order and is bit-stable on any machine), -streams the rows through a writer with a pinned layout (zstd, row groups of -1,048,576 rows, a parquet page index, `__row_order` last), writes to a unique -temp name and replaces the final file atomically, all under the project's write -lock, and returns the content digest of the file it wrote, read back. A create -runs the query twice and compares the digests; if they differ the recipe is not +`materialize(project, hash)` is the one routine that writes a computed entry's +snapshot. The build calls it and so does every **heal** of one (the re-creation +of a snapshot that is missing from disk), so result bytes are manufactured in +one place. A source entry's snapshot is not a result: it is written by the +import, and made again from the clone by the same code the import used +(`source_import.rewrite_source_snapshot`), through the same pinned writer. It +runs the entry's frozen build on a **single-partition** connection (so a float +total is merged in one order and is bit-stable on any machine), streams the rows +through a writer with a pinned layout (zstd, row groups of 1,048,576 rows, a +parquet page index, `__row_order` last), writes to a unique temp name and +replaces the final file atomically, all under the project's write lock, and +returns the content digest of the file it wrote, read back. A heal replaces the +file at once. A create leaves the finished file at its temp name +(`materialize(..., publish=False)`), and the build moves it into place +(`publish_snapshot`) after the manifest is written, so a build that fails +removes only its temp file and never the file already at the path. A create runs +the query twice and compares the digests; if they differ the recipe is not reproducible, the entry still builds, and its file is **pinned**: the Cache -page's delete leaves it alone. +page's delete leaves it alone. A snapshot changes only by an atomic replace of a +complete file. + +**Pins are read from the entry's manifest and, for a source entry, from whether +its clone is on disk** (`pinned_reason`): a snapshot is pinned when the manifest +says `reproducible: false`, when it holds `unfaithful_heal_digest`, or when it +is a source entry whose clone is gone. So a pin moves with its entry through a +reset, and nothing outside the entry, such as the error log, can lift it. A +snapshot whose entry a reset retired is judged by the manifest parked in the +bullpen (the directory a reset moves retired entries into), and a retired source +version's clone counts as present when a reset parked it there too. `ensure_materialized(project, hash)` is the one entry point that makes files exist, and every consumer that composes or executes an entry goes through it @@ -398,20 +494,26 @@ exist, and every consumer that composes or executes an entry goes through it 1. A worthy entry whose snapshot exists is done, and no build is loaded. 2. Otherwise load the build and collect every file its `Read` nodes point at. -3. Re-create each that is missing by the rule for its class. A snapshot is made - again by recursing on the hash in its file name. An ordered copy is made again - from its source's clone with the reader options in the manifest, and checked - against the digest recorded when it was first written. A clone is copied again - from the live source while the live bytes still hash to its name. +3. Re-create each that is missing. Every one is another entry's snapshot, made + again by recursing on the hash in its file name. 4. If the entry is worthy, materialize it and verify the result against - `result_digest`. + `result_digest`. A source entry skips steps 2 and 3: its snapshot is written + again from its clone with the reader options in its manifest, and verified + the same way. + +Whether the entry is worthy is read from the manifest, never derived again. +With the manifest missing the read raises; a file at the snapshot path says +nothing about the verdict. A file is cache only if this function can re-create it from files that are not -cache. Snapshots and ordered copies satisfy that and live under -`compute_cache/`, which anything may delete. Clones are data: once the live -source is edited a clone is the only copy of the bytes an entry was built from, -so nothing deletes one. When nothing can re-create a file (the clone is gone and -the live source has changed) the error names the source file. +cache. Snapshots satisfy that, a source entry's included, and live under +`compute_cache/`, which anything may delete. Clones are data: a clone is the +only copy tallyman has of bytes it imported, so nothing deletes one (a reset +moves a clone no surviving source entry names into the bullpen). A source entry +whose clone is gone is the one case where a snapshot is the last copy of its +rows, so that snapshot is pinned; if it is deleted anyway, the read fails with +an error naming the missing clone and the import call, reader options included, +that repairs the version. Files are deleted only by an explicit user action, and a file is written only because something is about to read it. The startup warm-up, the verify sweep and @@ -419,12 +521,14 @@ a reset write and delete nothing under `compute_cache/`. ## Result digest -For worthy entries, the build records `result_digest`: `arrow-sha256:`, a +For worthy entries, the build (for a source entry, the import) records +`result_digest`: `arrow-sha256:`, a SHA-256 over the snapshot's ordered Arrow data, computed from the file read back. It is independent of the row-group size, the codec, the writer's version, whether a text column is `string` or `large_string`, and what a null slot holds; it depends on every value, on which slots are null, on the order of the rows -(fixed by the canonical sort) and on the column names and types. It is the +(fixed by the canonical sort, or for a source entry by the file's order) and on +the column names and types. It is the **output** identity axis, and it has exactly one job: witnessing that a later rematerialization reproduced the original result. It is never a staleness input (an entry whose recompute differs is *nondeterministic*, not stale — recomputing @@ -442,35 +546,77 @@ the moment the entry is born. ## The write path (build) `build_and_persist(project, code)` holds the project's write lock for the whole -build (one write at a time per project, so two builds of one entry cannot end with -the failing one deleting the winner's directory): +build (one build at a time per project, so two builds of one entry cannot end +with the failing one deleting the winner's directory). The lock is a file lock, +so it holds between the MCP server and the companion, which both build. One +companion serves a data dir (the directory that holds every project, +`TALLYMAN_HOME`): `tallyman run` holds an exclusive lock on +`/server.lock` while it serves, a second `tallyman run` on the same +data dir is refused, and a client of the data dir reaches only the companion +that lock names. The build: 1. **Import the recipe** — the single moment of name resolution. During the - import, `read_project_file` and `tallyman_read_csv` digest each source, clone - it (CAS), write its ordered copy and record `{rel_path: digest}` and the copy's - reader options; `tracked_expr_from_alias` resolves each alias to its current - head, records the parent edge, and returns the parent's result (below). + import, `tracked_expr_from_alias` resolves each alias to its current head, + records the parent edge, and returns the parent's result (below), and + `pinned_expr_from_alias` does the same for a version reference. A raw file + read, a bare hash and a bare alias raise here. 2. **Check and rewrite** — reject what cannot become a sound entry (an in-memory - read, a `.cache()` call, a raw parquet read, an assignment to `__row_order`, a - cheap entry that drops it), classify the entry once (cheap or worthy), and add - the canonical sort to a worthy entry. + read, a `.cache()` call, a raw parquet or CSV read, an assignment to + `__row_order`, a cheap entry that drops it, a join chain over three entries + that all carry it), classify the entry once (cheap or worthy), add the + canonical sort to a worthy entry, and move `__row_order` to the last column + of a cheap one. 3. **Freeze** — `build_expr` serializes the rewritten expression; `content_hash` = the build's name. If an entry with this hash already exists, stop: append the prompt, return the existing entry (idempotency). 4. **Lay down the entry** — copy the build in, make paths portable (`${TALLYMAN_PROJECT_ROOT}` placeholders), write `expr.py`. -5. **Execute once** — a worthy entry is materialized (Part 2, "Materialization"), - which writes the snapshot and yields its digest, and the build records the - digest, the reproducibility verdict and the schema read from the written file. - A cheap entry is streamed once in full and keeps nothing (honest evaluation, - fails fast). -6. **Record** — schema, manifest (written last, atomic). +5. **Execute** — a worthy entry is materialized (Part 2, "Materialization"), + which runs its query twice, writes the snapshot and yields its digest, and + the build records the digest, the reproducibility verdict and the schema + read from the written file. The snapshot stays at its temp name. A cheap + entry is streamed once in full and keeps nothing (honest evaluation, fails + fast). +6. **Record** — schema, manifest (atomic, the entry directory's last write), and + then the snapshot, moved into place from its temp name. A build that fails + before that leaves any file already at the snapshot's path as it was. 7. **Checkpoint** — when the MCP tool returns: recipe zip, tracked pointers, one git commit. The build's obligation in one line: **record everything a reader will ever need, because the reader is forbidden from resolving anything.** +## The import path + +`update_and_depend(outside_path, alias, pinned_version=None, schema=None, +**reader_options)`, behind the `catalog_import_source` tool, is the only way a +source alias advances. It fixes the reader from the file's suffix and the +options, digests the file, computes the entry hash, and then, under the project +lock, decides the case (Part 2, "Project, imports and source entries"): + +- **Mint:** clone the bytes and verify the clone, write the snapshot, write the + entry (generated `expr.py`, frozen build, schema, manifest with `provenance`, + written last), and append it to the alias. A failure removes the entry + directory the import created, and nothing else. +- **The version already exists** (a no-op, or a repair): the entry's recipe, + build and manifest are the record of the import that minted it, and none of + them is rewritten. If its snapshot is gone, the clone is restored from the + given file when it is gone too (verified against the digest), and the + snapshot is healed exactly as `ensure_materialized` heals it: from the clone, + verified against `result_digest`. If a reader now parses the bytes + differently, the healed file is served but recorded as an unfaithful heal + (which pins it), and the manifest keeps the digest of the rows the version + was imported with: a repair never re-records a version's rows. A directory a + crash left without a manifest is not an entry, and is written again. +- **Refuse:** a directory, a name that is a catalog alias, a file that is not + parquet or CSV, a CSV option that does not survive JSON, and every error row + of the case table, each with a message saying what to do instead. + +A minted version is a catalog operation like a revise: the tool records the +same events, notifies the companion, runs auto-recalc for the alias's followers +when the project enables it, and lands the import and its cascade as one +checkpoint. + ## The read path One canonical read, used by every consumer: @@ -485,13 +631,18 @@ read(project, content_hash): - **Worthy entry:** a bare read of `compute_cache/result_cache/.parquet`, served without loading the entry's build when the file exists. If the file is missing, `ensure_materialized` re-runs the *frozen* build (whose reads are - ordered copies and parents' snapshots, all made to exist first), writes the - snapshot, and verifies it against `result_digest` before it is served. + parents' snapshots, all made to exist first), writes the snapshot, and + verifies it against `result_digest` before it is served. For a source entry + it parses the clone again instead. - **Cheap entry:** execution re-runs the frozen graph, reading files that exist. Same rows every time, by construction. - **The cold state is an empty `compute_cache/`:** any entry can be read after it is deleted, and must produce the same result as the warm read. That property - is the standing regression test for every read-path change. + is the standing regression test for every read-path change. The known + exceptions are an entry recorded as not reproducible, whose snapshot is pinned + because it cannot be made again faithfully, and the entries built on it + (#185, #208), and a source entry whose clone is gone, whose snapshot is the + last copy of its rows. `cached_result_expr(project, hash)` is the function every in-process consumer calls. It is `ensure_materialized` plus the read above, and it memoizes the loaded @@ -500,19 +651,31 @@ backend) per `(project, content_hash)`. Removing the memo must change latency an nothing else. The per-call existence check stays outside the memo, because file existence is the one input that remains mutable. -**Chaining** (`tracked_expr_from_alias` at build time) uses the same read. A worthy -parent is a bare read of its snapshot, so the child's build holds the literal path -of that file, which contains the parent's content hash: the child's identity is a -function of its parent's. A filter over an aggregate's snapshot is therefore a -cheap entry. A cheap parent's graph is inlined. The parent's snapshot is made to -exist before the child is composed, since a child cannot be built over a file -that is missing. +**One execution at a time per process.** A process's default backend is one +DataFusion session, which fails with `Already borrowed` when two threads execute +on it at once. So every execution on it (a page, a post-processing run, a +primary-key probe, a diff's summaries) holds `execution.execution_lock`, one +re-entrant lock per process. The read above comes first and the execution after, +because a heal takes the project lock and the order is the project lock first, +then the execution lock: `project_lock` raises in a thread that holds the +execution lock and would take a new file lock. An execution on a connection of +its own (a materialization's stream, a cheap entry's row count at build) needs +no execution lock. + +**Chaining** (`tracked_expr_from_alias` at build time) uses the same read. A +worthy parent, a source entry included, is a bare read of its snapshot, so the +child's build holds the literal path of that file, which contains the parent's +content hash: the child's identity is a function of its parent's. A filter over +an aggregate's snapshot is therefore a cheap entry. A cheap parent's graph is +inlined. The parent's snapshot is made to exist before the child is composed, +since a child cannot be built over a file that is missing. The recipe-reconstruction machinery survives only as a diagnostic. An entry whose build is missing or unloadable is a **hard error** naming the entry and the remedy (rebuild) — there is no automatic recipe fallback, because a warning on a background read is exactly how #163-class behavior stays -invisible (decided in `plans/ADR-006-read-path-loads-builds.md`, D6).[^recon] +invisible (ADR-006 D6, a missing or unloadable build is a hard error, in +`plans/ADR-006-read-path-loads-builds.md`).[^recon] ## Composition: diff and beyond @@ -521,13 +684,14 @@ expression): 1. read each entry (canonical read above) — each read makes its files exist, and each load minted fresh backend objects; -2. **rebind onto shared backends, one per distinct content profile** (profile - identity = profile minus `idx`), using `replace_sources`. For today's - catalogs every profile is the same embedded engine, so this collapses to - one shared backend; -3. compose, dropping `__row_order` from both sides first. The result is a - single-backend expression: it executes in-process, and `build_expr` serializes - it into a normal single-profile build that Buckaroo's `/load_expr` accepts. +2. **rebind onto the process's default backend**, using `replace_sources`. + Every profile in a tallyman build is the same embedded engine (profile + identity = profile minus `idx`), and a build that spans more than one + distinct content profile fails loudly instead of being rebound; +3. compose, dropping `__row_order` from both sides first (`full_diff` does not + yet, #200). The result is a single-backend expression: it executes + in-process, and `build_expr` serializes it into a normal single-profile build + that Buckaroo's `/load_expr` accepts. Composition of frozen builds was never the problem; backend object identity was (#75, rediagnosed in #163). The rebind is cheap graph surgery, no data @@ -536,9 +700,10 @@ moves. A **promoted diff** is just an entry whose recipe pins two hashes (`build_diff_expr(a_hash, b_hash)`) — name-free, deterministic, and built through the ordinary write path. It contains a join, so it is worthy and is -materialized like any other. The live diff grid, which is not an entry, still hands -Buckaroo an unmaterialized join (`plans/ADR-007-tallyman-owned-materialization.md` -D10 moved that to a follow-on). +materialized like any other. The live diff grid, which is not an entry, still +hands Buckaroo an unmaterialized join: ADR-007 D10, which would have built +every diff as an entry before showing it, was moved out of +`plans/ADR-007-tallyman-owned-materialization.md` to #188. ## Handing an entry to Buckaroo @@ -550,51 +715,73 @@ process and never inside a grid query. - A **worthy** entry's grid is handed a **view build**: a build whose whole graph is one bare read of the entry's snapshot, written once to a stable per-entry - directory (Buckaroo's stat-cache keys include the build directory's path). + directory, so Buckaroo is handed the same build after a restart. - A **cheap** entry's grid is handed its own expanded build, a stored plan over files that exist. -- Tallyman keeps no record of Buckaroo's sessions. A session id is - `entry--`, posted on every open: Buckaroo skips the work - while it holds that session with the same build directory, and creates the - session again if it dropped it (it does after an hour without a browser). A - klass reload posts `/reload_expr/` for each entry of the project and treats - the 404 for an id Buckaroo does not hold as "not open". +- Tallyman keeps no record of Buckaroo's sessions. A **session** (one grid's + state in the Buckaroo process) has the id `entry--`, + posted on every open: Buckaroo skips the work while it holds that session with + the same build directory and the post carries no configuration, and creates + the session again if it dropped it (it does after an hour without a browser). + A klass (a project-authored stat, post-processing or display class) reload + posts `/reload_expr/` for each entry of the project and treats the 404 for + an id Buckaroo does not hold as "not open". - Every `/load_expr` names `__row_order` as the row-order column, so Buckaroo can - order its pages by it (buckaroo-data/buckaroo#974 is Buckaroo's half of that). + order its pages by it. Buckaroo 0.15.6, the pinned version, ignores the hint; + buckaroo-data/buckaroo#974 is Buckaroo's half of that. - After an unfaithful heal, the entry's stat cache is wiped and Buckaroo is told to reload the grid (`force_reload`). ## Reset -`reset_to` returns the catalog to an earlier step. It restores every tracked file -with `git reset --hard`, and reconciles the untracked entry directories through the -**bullpen**, the directory a reset moves retired files into so a reset forward can -bring them back. It leaves `compute_cache/` alone: snapshots and ordered copies are -named by content, and a file that is missing afterwards is made again and verified -like any other. It moves the source clones no surviving entry refers to into the -bullpen and never deletes them. - -## Staleness and recalc (unchanged, stated for completeness) - -**Staleness** is a read-only judgment: an entry is stale on the alias axis -when a `follow=True` parent's recorded hash no longer equals that alias's -head, and on the source axis when a recorded digest no longer matches the -live file's digest. Computing staleness executes nothing and mutates nothing. +`reset_to` returns the catalog to an earlier step. It restores every tracked +file with `git reset --hard`, and reconciles the untracked entry directories +through the **bullpen**, the directory a reset moves retired files into so a +reset forward can bring them back. It leaves `compute_cache/` alone: snapshots +are named by content hash, and a file that is missing afterwards is made again +and verified like any other. It moves the clones no surviving source entry names +in its `provenance` into the bullpen and never deletes them, and a reset forward +copies back the clones a restored source entry names. Source aliases rewind with +every other alias, since `aliases.jsonl` is a tracked file. An entry directory +retired when the bullpen already holds one under its name replaces the parked +copy, because the live one agrees with the snapshot on disk; one with no +manifest, left by an interrupted build, is dropped instead. So a reset forward +brings back the manifest that matches the file. The bullpen has one live reader +besides `reset_to`, the Cache page, which reads a retired entry's parked +manifest (and a retired source version's parked clone) to keep its snapshot's +pin. + +## Staleness and recalc + +**Staleness** is a read-only judgment with one axis: an entry is stale when a +`follow=True` parent's recorded hash no longer equals that alias's head, and +for no other reason. A pinned parent never makes its child stale. A data file +that changed outside tallyman is not a reason: until it is imported again +nothing has changed in the catalog, and the import moves a source alias, which +is the one axis. Computing staleness executes nothing, opens no data file and +changes no catalog state. Only an entry that is the current head of an alias is +actionably stale; a superseded version is reported with `live=False` (#154). A +parent alias that no longer exists is reported under `unknown_axes`. **Recalc** is the one sanctioned re-execution of recipes. When an alias head -advances (a revise), the entries that may be affected form its **cone**: -every entry reachable by walking `follow=True` parent edges backwards from -that alias — its followers, their followers, and so on. Pinned -(`follow=False`) edges are not in the cone. It runs automatically after a -revise when the project enables auto-recalc, or on demand. +advances (a revise, or an import that mints a new version of a source alias), +the entries that followed it by name are directly stale; they are the roots, and +the entries that may be affected form their **cone**: the roots and every +current alias head reachable from them through recorded parent edges, followers +of followers and so on. It runs automatically, when the project enables +auto-recalc (the default), after a revise, after an import that mints a version, +and after a promoted diff that re-points an existing alias; otherwise it runs on +demand. Recalc rebuilds the cone in topological order, parents before children. For each member it re-imports the member's *recipe* — the one situation where name resolution is the point, since the goal is a new version against the new heads — builds the result as an ordinary new entry, and advances the member's alias before any of its children replay, so each child chains off -its parent's fresh head. Old entries are untouched; every member gains a -version, none loses one. +its parent's fresh head. A member whose inputs did not move, such as a child +that pins a version of its parent (`follow=False`), replays to the same hash and +is left alone. Old entries are untouched; a member that rebuilds gains a version, +and none loses one. Two disciplines keep it predictable. **Scope:** recalc touches only followers of the alias that moved; pre-existing staleness elsewhere is reported, not @@ -611,20 +798,24 @@ not only in tests: - on every heal (a snapshot `ensure_materialized` writes is checked before it is served); -- on demand, corpus-wide, via `catalog_scan_staleness(verify_results=True)`, which - reads and never writes: a snapshot that is missing is reported as `absent` and - checked at the moment it next exists. - -A failure is surfaced loudly — a durable `unfaithful_heal` record in -`errors.jsonl` (the UI badge, and the pin: the Cache page's delete leaves the file -alone), a stat cache wipe, a forced reload of the open Buckaroo grid, an SSE -event — never only a log line. Its attribution has four classes with four -different fixes: +- on demand, corpus-wide, via `catalog_scan_staleness(verify_results=True)`, whose + verify sweep reads and never writes: a snapshot that is missing is reported as + `absent` and checked at the moment it next exists. + +A failure is surfaced loudly, never only as a log line: the pin, the digest the +heal wrote recorded as `unfaithful_heal_digest` in the manifest (the Cache +page's delete leaves the file alone), a durable `unfaithful_heal` record in +`errors.jsonl` (shown in the catalog page's error banner), a stat cache wipe, +and in the companion a forced reload of the entry's Buckaroo grid and an +`unfaithful_heal` SSE event. Today the SPA has no listener for that event, the +forced reload is sent even when no grid is open (#203), and cheap entries that +read the healed snapshot are not flagged (#208). Its attribution has four +classes with four different fixes: | class | detector | meaning | response | |---|---|---|---| | engine | the xorq, xorq-datafusion or pyarrow version, or the snapshot format, differs from the one recorded at build | a library upgrade changed the result | rebuild the entry; the recipe is not implicated | -| structural (#88) | recipe re-derives a different hash, sources unchanged | author-time value baked into the graph (`pd.Timestamp.now()`) | lint; rewrite recipe | +| structural (#88) | recipe re-derives a different hash, inputs unchanged | author-time value baked into the graph (`pd.Timestamp.now()`) | lint; rewrite recipe | | execution (#83) | fixed graph, digest moves across runs | `sample()`, `now()`, impure UDF | lint; the file is pinned | | lineage (#163) | a read resolved a name post-build | machinery bug | impossible by construction under this contract | @@ -636,15 +827,16 @@ design: with reads going through the build, lineage drift has no mechanism left. [^recon]: The subsystem the fix demoted: `_recipe_expr` re-imports an entry's `expr.py` to recover its expression, patching over the recipe's name-binding with context variables for the duration of the import. - `_RECON_SOURCES` carries the entry's recorded `{path: digest}` map so - that `read_project_file`, when called inside such a re-import, resolves - each source to its frozen `.cas/` clone instead of re-digesting - the live file (#115), and `_resolve_noncyclic_hash` walks a - self-referencing alias back to its previous version to break re-import - cycles (#74). Each hook re-derives a binding the frozen build already - contains, which is why the fix retired their read-time role (and #162's - proposed `_RECON_PARENTS` sibling was never built) rather than adding a - fourth. The diagnostic use stays, and is the only reason the machinery + `_resolve_noncyclic_hash` walks a self-referencing alias back to its + previous version to break re-import cycles (#74). A second hook, + `_RECON_SOURCES`, once resolved each raw file read to its frozen + `.cas/` clone (#115); ADR-011 deleted it with `manifest.sources`, + since a recipe no longer reads files at all, and a source entry's own + generated recipe resolves its one read to its snapshot (`_SOURCE_ENTRY`). + Each hook re-derives a binding the frozen build already contains, which is + why the fix retired their read-time role (and #162's proposed + `_RECON_PARENTS` sibling was never built) rather than adding another. The + diagnostic use stays, and is the only reason the machinery still exists: `recipe_is_structurally_nondeterministic` re-derives the hash from the recipe on purpose, because re-running the recipe is exactly how you detect that a recipe fails to reproduce its own graph. @@ -666,9 +858,11 @@ wrong even if every test passes. from immutable inputs; cached values are reproducible from the frozen build alone; every read path has a cold seam (an empty `compute_cache/`); a self-heal is reproduce-and-verify, never manufacture. Result bytes are - manufactured by one routine, `materialize`, which the build and every heal - call — a read path that writes bytes some other way has become a second, - unaudited build path. + manufactured in two places: `materialize`, which the build and every heal of + a computed entry call, and the import's writer (`source_import._write_snapshot`), + which writes a source entry's snapshot at import and at a heal from its clone. + A read path that writes bytes any other way has become a second, unaudited + build path. - **I3 — One read semantics.** Every consumer that materializes an entry reads the frozen build through the one canonical read. The recipe is never re-executed on behalf of an existing entry. @@ -676,7 +870,9 @@ wrong even if every test passes. the manifest; after that the entry is closed. Aliases at read time select *which* entry to serve, never *what* an entry means. (Grep-able form: `get_alias`/`previous_version`/`version_of_hash` appear only in minting, - selecting, and judging code — never in materializing code.) + selecting, and judging code — never in materializing code. The one alias + lookup near materializing code names a source version in the text of an + error or a pin reason, `current_source_version`, and decides no rows.) - **I5 — One question, one path.** Any question answerable two ways (grid vs API bytes, manifest row count vs live count) either shares one canonical path or carries an explicit check tying the two together; disagreement is @@ -688,6 +884,48 @@ wrong even if every test passes. --- +# Known deviations + +Where the code breaks a rule above today. Each is an open issue; none is a +change of the rule. + +- **Verification reaches one entry.** An unfaithful heal of a worthy parent + changes its cheap children's rows under their hashes without a record + (#208), and the purity of an entry is not passed on to entries built on it, + so a child of a non-reproducible parent is recorded as reproducible (#185). + Both are gaps in I1 and I2. +- **Row order.** The canonical sort's tie-break leaves out nested columns + (#205); the snapshot writer drops any column named `__row_order_right` (#206); + the three-way join check refuses semi and anti joins (#199); `full_diff` keeps + `__row_order` as data (#200); and Buckaroo's grid does not yet order pages by + `__row_order` (buckaroo-data/buckaroo#974), so I6 holds for `/api/data` and + not yet for the grid. +- **Handing an entry to Buckaroo.** Concurrent opens both post `/load_expr`, and + a promoted diff re-runs Buckaroo's statistics on every open (#202); the + forced reload after an unfaithful heal runs under the project lock and opens a + session nobody asked for (#203); a klass reload posts once per entry from the + companion's event loop (#201); Buckaroo is pointed at `artifacts/` and does + not find the project's stats and post-processing functions (#170); the live + diff grid is an unmaterialized join (#188). +- **One writer at a time.** The project lock blocks with no timeout (#186), and + two companion routes build on the event loop and freeze the UI while they wait + (#190). The lock covers builds, materializations, checkpoints and resets + only: alias, notebook, chart, display-config and `config.json` writes replace + their file atomically without it, so an MCP edit and a browser edit of the + same file at the same moment can lose one of the two (#240). +- **Names resolve once, in the right project.** A recipe's alias readers + (`tracked_expr_from_alias`, `pinned_expr_from_alias`) resolve the project from + the `active_project` file, while the MCP tool builds into the session's own + project; after another session switches projects the two differ, and the + recipe looks its aliases up in the other project (#233; related to #39). +- **Portability.** An expanded build does not record the project path it was + filled in with, so a copied project reads the old location (#209). +- **Float totals.** An ungrouped float `SUM` depends on the row-group layout of + the file it reads; the snapshot format version pins that layout, and a change + of it is a corpus rebuild (#187). + +--- + # History This contract began as the proposed design for the #163 fix; the pre-fix @@ -697,8 +935,27 @@ The redesign that replaced xorq's cache nodes with tallyman's own materializatio added `__row_order` and redefined the digest is in [`plans/ADR-007-tallyman-owned-materialization.md`](../plans/ADR-007-tallyman-owned-materialization.md), [`plans/ADR-008-row-order-of-reads.md`](../plans/ADR-008-row-order-of-reads.md) and -[`plans/ADR-009-digest-stability.md`](../plans/ADR-009-digest-stability.md). -The wider audit of the same bug class is +[`plans/ADR-009-digest-stability.md`](../plans/ADR-009-digest-stability.md), +accepted on 2026-09-22 and implemented in #189. +[`plans/ADR-010-immutable-store-one-owner.md`](../plans/ADR-010-immutable-store-one-owner.md) +proposed replacing them and was rejected. +[`plans/ADR-011-sources-are-aliases.md`](../plans/ADR-011-sources-are-aliases.md) +made a raw input a source alias whose versions are entries, which removed the +source axis of staleness, the identity modes, `manifest.sources` and the ordered +copy, and with them the defects listed here before it (#191, #197, #198, #207, +#211, and the two staleness defects that had no issue). Four deviations this +section listed were fixed on the same branch: a failed build deleted the +snapshot already at its path (#193, fixed in #222), and a reset could pair a +non-reproducible entry's older manifest with its newer snapshot, a retired +entry's snapshot lost its pin, and dismissing the error banner lifted an +unfaithful heal's pin (#194, #195 and #196, fixed in #223). Three more were +fixed later on the same branch: with its manifest missing, an entry's worthiness +was guessed from whether its snapshot existed, so a worthy entry that had lost +both was read as cheap (#204, fixed in #245, which makes every read refuse such +a directory); two servers on one project went undetected (#183, fixed in #241, +one server per data dir); and concurrent executions on a process's shared +backend could fail with `Already borrowed` (#118, fixed in #242, the execution +lock). The wider audit of the same bug class is [`plans/cache-soundness-audit.md`](../plans/cache-soundness-audit.md) (#168–#172, buckaroo#955–#957) — the contract's rules apply to those axes too. diff --git a/plans/89-determinism-prereqs-execution.md b/plans/89-determinism-prereqs-execution.md index 50455cf..d92edda 100644 --- a/plans/89-determinism-prereqs-execution.md +++ b/plans/89-determinism-prereqs-execution.md @@ -3,6 +3,13 @@ - **Status:** Execution plan (2026-06-18). Drives the still-open implementable items in the reactive-recalc prerequisites epic (#89) to done, so the staleness layer's design questions are the only thing left blocking it. +- **Status note (2026-09-22):** point in time. Reads no longer re-run recipes + (`plans/ADR-006-read-path-loads-builds.md`); a reset no longer prunes + `compute_cache/`, so `prune_compute_cache` is gone + (`plans/ADR-007-tallyman-owned-materialization.md` D14); `result_digest` is a + content digest of the snapshot read back + (`plans/ADR-009-digest-stability.md`); and `_warn_if_self_heal_unfaithful` + became `_verify_self_heal`, which also names an engine change as a cause. - **Epic:** buckaroo-data/tallyman#89. Design in `.claude-worktrees/docs-reactive-catalog/plans/adr-reactive-catalog-recalc.md`. - **Branch:** `feat/89-determinism-prereqs` off `main`. diff --git a/plans/ADR-002-source-identity-content-hash.md b/plans/ADR-002-source-identity-content-hash.md index 7ff95be..eca7d55 100644 --- a/plans/ADR-002-source-identity-content-hash.md +++ b/plans/ADR-002-source-identity-content-hash.md @@ -1,32 +1,31 @@ # ADR: Content-addressed source reads (CAS) so `content_hash` tracks source data -- **Status:** Accepted (2026-06-18 — default flipped `off` → `cas` in #86, with - `cp --reflink=auto` on Linux and `.cas` GC wired into `reset_to`; supersedes - the 2026-06-10 Proposed draft below. See the reconstruction caveat under - Consequences.) **Narrowed by `plans/ADR-011-sources-are-aliases.md` - (2026-09-22, PR #218), which keeps the clone store and deletes the rest of - the machinery around it:** - - The mode switch is gone. `TALLYMAN_SOURCE_IDENTITY`, its `off` and `salt` - modes and `salted_hash` are deleted (ADR-011 D8). There is one behaviour: - an import digests the bytes and clones them. "Identity off" and "every - input is versioned" cannot both be true. - - `manifest.sources` is gone (ADR-011 D6). It was written as the retention - closure `gc_cas` walks and read by the staleness scan as a freshness - record, and that second reading is the defect ADR-011 exists to remove. - Retention is now the DAG: a source version is an entry, so its clone is - alive exactly while that entry is, and `catalog_state._live_source_digests` - collects `manifest.provenance.digest` instead. - - `digest_for` and its `source_digests.json` stat memo are gone. Nothing - digests a file outside an import any more, so the memo has no callers and - the scan touches no file at all. - - The reconstruction caveat under Consequences is resolved rather than - inherited: `recon_cas_path` no longer falls back to serving live bytes when - the clone is missing and the file has drifted. That branch is an error - (ADR-011 D9), because the live file is no longer a build input in the first - place. - What survives unchanged is the substance of this ADR: content-addressed - clones under `data/.cas/`, written copy-on-write where the - filesystem offers it, parked in the bullpen by a reset rather than unlinked. +- **Status:** Accepted (2026-06-18). It supersedes the 2026-06-10 Proposed + draft below. The clone store is the only mode (see "Narrowed by"), and + `reset_to` parks the clones no surviving entry names in the bullpen. +- **Narrowed by:** `plans/ADR-011-sources-are-aliases.md`, which keeps the clone + store and removes the machinery around it. What stands is the substance of + this ADR: content-addressed clones under `data/.cas/`, written + copy-on-write where the filesystem offers it, parked in the bullpen by a reset + rather than unlinked. What is gone: + - The mode switch. `TALLYMAN_SOURCE_IDENTITY`, its `off` and `salt` modes and + `salted_hash` do not exist (ADR-011 D8). An import digests the bytes and + clones them, always. + - `manifest.sources` (ADR-011 D6). It served as the retention closure `gc_cas` + walks and was read by the staleness scan as a freshness record, and that + second reading is the defect ADR-011 exists to remove. Retention is the + DAG: a source version is an entry, so its clone is alive exactly while that + entry is (`catalog_state._live_source_digests` collects + `manifest.provenance.digest`). + - `digest_for` and its `source_digests.json` stat memo. Nothing digests a file + outside an import, and the staleness scan touches no file at all. + - The reconstruction caveat under Consequences. Reads load the frozen build and + do not re-run recipes (`plans/ADR-006-read-path-loads-builds.md`, #163), a + recipe reads no file, and a source entry's snapshot is re-created from its + clone. When the clone is gone the snapshot is pinned as the last copy; + nothing serves live bytes in its place (ADR-011 D9). +- **Also out of date below:** there is no `result.parquet`, and xorq's snapshot + cache is not used (`plans/ADR-007-tallyman-owned-materialization.md`). - **Context ticket:** buckaroo-data/tallyman#30 (precondition for the result-cache rubric's content-stable `content_hash` key) - **Affected code:** `src/tallyman_xorq/source_identity.py` (new), diff --git a/plans/ADR-003-result-cache-cost-rubric.md b/plans/ADR-003-result-cache-cost-rubric.md index 71850ed..aaed78e 100644 --- a/plans/ADR-003-result-cache-cost-rubric.md +++ b/plans/ADR-003-result-cache-cost-rubric.md @@ -1,6 +1,15 @@ # ADR: Result-cache admission and eviction by measured cost vs size -- **Status:** Proposed (2026-06-10) +- **Status:** Proposed (2026-06-10); not adopted. Partly overtaken (noted 2026-09-22). The structural + cheap-or-worthy test this ADR would remove still decides whether an entry is materialized, now as the + allow-list of `plans/ADR-008-row-order-of-reads.md` D4 (cheap means row-preserving over one file), computed + once and recorded in the manifest. `classify_build`, the regex over `expr.yaml` that it names, is gone, and + so is `ensure_result` (#73). Its motivating case, computed-column revisions below a join that each wrote a + full copy, is answered by `plans/ADR-007-tallyman-owned-materialization.md` D3 (a child of a worthy entry + reads that entry's snapshot, so such a child is cheap). Its measurement half is implemented + (`compile_seconds` and `cache_bytes` in the manifest, beside `execute_seconds`). Its budget and eviction + half is still open: under ADR-007 D12 files are deleted only by an explicit user action, and a disk budget + waits on a rewrite of this ADR against that design. - **Context ticket:** buckaroo-data/tallyman#30 (measured cost/size cache rubric; supersedes #12, feeds #10 and #21) - **Affected code:** `src/tallyman_xorq/result_cache.py` (`classify_build`, `cache_worthy`, `cached_result_expr`, `ensure_result`), `src/tallyman_xorq/build.py` (`build_and_persist`, manifest write), `src/tallyman_core/manifest.py` - **Related ADR:** `plans/ADR-002-source-identity-content-hash.md` (why `content_hash` is a stable cache key) diff --git a/plans/ADR-004-result-digest-canonical-ordering.md b/plans/ADR-004-result-digest-canonical-ordering.md index 3fe801a..797df72 100644 --- a/plans/ADR-004-result-digest-canonical-ordering.md +++ b/plans/ADR-004-result-digest-canonical-ordering.md @@ -1,9 +1,16 @@ # ADR: Result digest as a row-multiset, and canonical snapshot ordering -- **Status:** Proposed (2026-06-25) - Amended by `plans/ADR-009-digest-stability.md` (implemented in #189): `result_digest` is a digest of the - snapshot's Arrow content read back, `arrow-sha256:`, and no longer a SHA-256 of the file's bytes, which - is what Option A below recommends. The canonical sort it describes is unchanged. +- **Status:** Implemented (Option A below). +- **Superseded in part by:** + - `plans/ADR-009-digest-stability.md` D2: `result_digest` is a digest of the + snapshot's Arrow content read back, `arrow-sha256:`, not a SHA-256 of + the file's bytes, which is what Option A recommends. + - `plans/ADR-008-row-order-of-reads.md`: the synthetic row-index column is + `__row_order` (ADR-008 D2), the last, visible column of every file tallyman + writes, so decision 4 below ("not a data column") no longer holds; it is + appended, then the remaining columns, to every sort in a recipe (ADR-008 + D10), and a sort that is not the recipe's last step is kept (ADR-008 D11). A + deterministic total order before the snapshot is written still stands. - **Affected code:** `src/tallyman_xorq/result_cache.py` (`_digest_update`, `count_and_result_digest`, `result_digest`, `verify_result_faithful`), `src/tallyman_xorq/build.py` (`build_and_persist` execute path, diff --git a/plans/ADR-005-intelligent-csv-import.md b/plans/ADR-005-intelligent-csv-import.md index 0d011c6..7507ebe 100644 --- a/plans/ADR-005-intelligent-csv-import.md +++ b/plans/ADR-005-intelligent-csv-import.md @@ -1,25 +1,30 @@ # ADR: Intelligent CSV import — a deterministic reader with a great error/suggestion contract -- **Status:** Proposed (2026-06-26). Lands on top of #137 (the polars - order-stable reader swap), which is merged to `main` (`52afca7`). Resolves the - CSV-ingest-contract cluster filed during the #137 review: **#141, #143, #144, - #145**; **#142 is deferred** to the holistic edge-case pass (see below). - Amended by `plans/ADR-008-row-order-of-reads.md` (implemented in #189): INV-1's row-index column is named - `__row_order` (not `original_row_order`) and INV-2's trailing `order_by` is gone, so `tallyman_read_csv` - returns a plain read of an ordered copy that is keyed by the CSV's content. - Amended again by `plans/ADR-011-sources-are-aliases.md` (2026-09-22, PR #218): **this reader now runs at - import time, not in a recipe.** `tallyman_read_csv` is refused in an authored recipe alongside - `read_project_file` (ADR-011 D2) — both open a file the catalog does not own, and leaving the CSV reader - open would be a hole in the rule. A CSV enters through - `catalog_import_source(path, alias, separator=..., schema=...)`, and everything this ADR designed goes - with it: the schema DSL and its normaliser, the inference ladder (100 → 10k → whole file), the - suggestion engine and the error contract all run inside the import, over - `source_import._write_csv_snapshot`. The recipe then reads the resulting source alias. - Two consequences for the text below. The reader options are **fixed at import** and recorded on the - source entry (ADR-011 D12), so one file is read one way forever; to read it a second way, import it again - under a second alias. And the retry loop this ADR is built around — read, see the suggestion, adjust the - schema, read again — is a loop over imports, which is why ADR-011 keeps the raw bytes in the clone store - rather than only the parsed snapshot (its Open question 1). +- **Status:** Implemented. It addresses #141, #143, #144 and #145, which are + still open on GitHub; #142 is deferred to the holistic edge-case pass (see + below). +- **Amended by:** + - `plans/ADR-008-row-order-of-reads.md`: INV-1's row-index column is named + `__row_order` (not `original_row_order`), and INV-2's trailing `order_by` is + gone. + - `plans/ADR-011-sources-are-aliases.md`: **this reader runs at import time, + not in a recipe.** A CSV enters through `catalog_import_source(path, alias, + schema=..., reader_options={"separator": ...})`, and the schema DSL and its + normaliser, the inference ladder (100 → 10k → whole file), the suggestion + engine and the error contract all run inside the import + (`source_import._write_csv_snapshot`); `tallyman_read_csv` is refused in a + recipe, which reads the resulting source alias. The reader options are + fixed at import and recorded on the source entry (ADR-011 D12), so one file + is read one way; to read it a second way, import it again under a second + alias. The retry loop this ADR is built around (read, see the suggestion, + adjust the schema, read again) is a loop over imports, which is why ADR-011 + keeps the raw bytes in the clone store. INV-3 holds: the parsed parquet is + the source entry's snapshot, `compute_cache/result_cache/.parquet`, + whose hash covers the CSV's digest and the reader options, and it is + re-created from the clone, never from the outside CSV. D5's drift check does + not exist, because staleness has one axis (ADR-011 D6): editing the outside + CSV changes nothing until it is imported again, which mints the next version + of its source alias. - **Affected code:** `src/tallyman_xorq/io.py` (`tallyman_read_csv`, `_polars_overrides`, `_IBIS_TO_POLARS`, a new schema-spec normaliser and a suggestion engine), `src/tallyman_mcp/server.py` (the `catalog_run` CSV diff --git a/plans/ADR-006-read-path-loads-builds.md b/plans/ADR-006-read-path-loads-builds.md index ceba6c8..28832fb 100644 --- a/plans/ADR-006-read-path-loads-builds.md +++ b/plans/ADR-006-read-path-loads-builds.md @@ -1,15 +1,15 @@ # ADR: Reads load the frozen build — the #163 read-path decisions -- **Status:** Accepted (2026-07-30) — decided in the grilling session over - `docs/system-contract.md`. Supersedes the read-path behavior introduced by - #74/#75 (recipe re-import on read). All follow-up questions resolved - (D9–D12, and the D5 amendment); implemented on PR #167. - Two of its decisions are superseded by `plans/ADR-007-tallyman-owned-materialization.md` (implemented in - #189): D4 (chaining inlines the parent's cache node) and D8 (the manifest records the snapshot key and reads - assert it), since no build holds a cache node any more. D5 (the canonical sort) is amended by - `plans/ADR-008-row-order-of-reads.md` (every sort gets the natural order, a non-final sort is kept) and by - `plans/ADR-009-digest-stability.md` (`result_digest` is a content digest, no longer a file hash). D2, D3, D6, - D7, D10 and D12 keep their intent. +- **Status:** Accepted (2026-07-30) — decided in the grilling session over `docs/system-contract.md`. + Supersedes the read-path behavior introduced by #74/#75 (recipe re-import on read). All follow-up questions + resolved (D9–D12, and the D5 amendment); implemented on PR #167. Two of its decisions are superseded by + `plans/ADR-007-tallyman-owned-materialization.md`: D4 (chaining inlines the parent's cache node) and D8 (the + manifest records the snapshot key and reads assert it), since no build holds a cache node any more. D5 (the + canonical sort) is amended by `plans/ADR-008-row-order-of-reads.md` (every sort gets the natural order, a + non-final sort is kept) and by `plans/ADR-009-digest-stability.md` (`result_digest` is a content digest, no + longer a file hash). D2 keeps its facade, but the snapshot's path is now a function of the content hash + (ADR-007 D2) and no loader takes a `cache_dir`. D10's session eviction became a forced Buckaroo reload, + since tallyman keeps no record of sessions (ADR-007 D6). D3, D6, D7, D9 and D12 keep their intent. - **Context ticket:** buckaroo-data/tallyman#163 (read path reconstructs from `expr.py`; a content hash does not name a fixed result). Normative design: `docs/system-contract.md`. Bug-class survey: diff --git a/plans/ADR-007-tallyman-owned-materialization.md b/plans/ADR-007-tallyman-owned-materialization.md index cd814b9..8ceb79e 100644 --- a/plans/ADR-007-tallyman-owned-materialization.md +++ b/plans/ADR-007-tallyman-owned-materialization.md @@ -1,9 +1,32 @@ # ADR: Tallyman owns result materialization (no xorq cache nodes in builds) -- **Status:** Implemented in buckaroo-data/tallyman#189 (2026-09-21), at Paddy's request to implement the - set. Where this text says a decision is "not yet confirmed" or "Proposed", it was implemented as - written; the differences between the text and the code are under "Implementation notes" below. - Original status: Proposed (2026-09-18, revised 2026-09-20 in the grilling session, which added the governing rule, decisions D10 to D12, and the resolution recorded under D5, and again the same day after a review of PR #184: D10 moved out to #188, the verify sweep left D5's callers, D6's session-ending clause was dropped, and D12's rule was restated, and a third time that day after a second review of PR #184: two kinds of entry were confirmed (open question 1), D5 lost its accepted gap, D6 gained the klass reload, and D13 and D14 are new). Awaiting Paddy's review; nothing here is implemented. Supersedes two decisions of `plans/ADR-006-read-path-loads-builds.md`: its D4 (chaining inlines the parent's cache node) and its D8 (the manifest records the snapshot key and reads assert it). Five other ADR-006 decisions keep their intent: D5 (the canonical sort), D6 (a missing build is a hard error), D7 (verification runs in production and is loud), D10 (an unfaithful heal wipes the entry's Buckaroo state) and D12 (unfaithful entries are pinned and badged). The last three attach to `ensure_materialized`, which is this ADR's D5. +- **Status:** Accepted (2026-09-22), implemented. Where the text below says a + decision is "not yet confirmed" or "Proposed", it was implemented as written; + the differences between the text and the code are under "Implementation + notes". Open defects: `docs/architecture.md` ("Known defects"). +- **Supersedes:** two decisions of `plans/ADR-006-read-path-loads-builds.md`: + ADR-006 D4 (chaining inlines the parent's cache node) and ADR-006 D8 (the + manifest records the snapshot key and reads assert it), since no build holds a + cache node. It changes two more. ADR-006 D2 (keep the facade, replace the + internals) keeps its facade, but the snapshot's path is a function of the + content hash (this ADR's D2) and no loader takes a `cache_dir`. ADR-006 D10 (an + unfaithful heal wipes the entry's Buckaroo state) keeps the wipe, but its + session eviction became a forced Buckaroo reload, since tallyman keeps no + record of sessions (this ADR's D6). ADR-006 D5 (the canonical sort) is amended + by ADR-008 and ADR-009, not by this ADR. These keep their intent: ADR-006 D3 + (rebind composition onto the default backend, with a one-group guard), ADR-006 + D6 (a missing or unloadable build is a hard error), ADR-006 D7 (verification + runs in production and is loud), ADR-006 D9 (no cheap-entry digests) and + ADR-006 D12 (unfaithful entries are pinned and badged). ADR-006 D7, D10 and D12 + attach to `ensure_materialized`, which is this ADR's D5. +- **Amended by:** `plans/ADR-011-sources-are-aliases.md`. A data file enters + only by an import, as a source entry whose snapshot takes the place of the + ordered copy. The implementation notes below that describe ordered copies + (`compute_cache/ordered_sources/`, the `.digest` sidecar, + `manifest.sources`, `manifest.ordered_copies`, and `ordered_copy.py` making + and re-making copies) describe code ADR-011 removed. D13's rule (a file is + cache only if it can be re-created) holds for a source entry's snapshot, + which is re-created from its clone. - **Reading decision labels:** a bare label such as "D5" in this document always means this ADR's own decision. Another ADR's decision is always written with its ADR number and a few words saying what it decides. @@ -14,12 +37,14 @@ possible for caching." - **Tickets:** #188 (diffs, moved out of this ADR), #186 (the waiting that D11's lock causes), #185 (non-pure recipes), #183 (two servers on one - project), #168 (CSV source identity, which the shared rebuild of D9 needs), - #118 (concurrent reads on the shared backend, which D11 does not cover), - #77 (an empty grid on a clone of the project at another path, which D2 - closes), #76 (closed: how bare-read chaining failed the last time it was - tried, which D5 answers), #22 (the checkpoint's cost grows with the cache, - which D14 removes). + project, now refused per data dir), #168 (CSV source identity, which the + shared rebuild of D9 needs), #118 (concurrent executions on the shared + backend, which the project lock does not cover and a per-process execution + lock does), #77 (an empty grid on a clone of the project at another path, + which D2 closes), #76 (closed: how bare-read chaining failed the last time it + was tried, which D5 answers), #22 (the checkpoint's cost grows with the + cache, which D14 removes). D11 (one write at a time per project) records how + #183 and #118 are handled. - **Affected code:** `src/tallyman_xorq/source_cache.py` (`rewrite_for_build`), `src/tallyman_xorq/result_cache.py` (`_resolve_result_plan`, `cached_result_expr`, `entry_graph_expr`, `baked_snapshot_path`, @@ -567,9 +592,12 @@ discover them: lock cannot be held across an `await`. - Whether a recalc takes the lock once per build or once for the whole walk is left to the implementation. Either is correct. -- The lock covers writes only. Concurrent reads on the shared default backend - fail with `RuntimeError: Already borrowed`. That is #118, and this ADR does - not change it. +- The lock covers writes only. Two threads executing on a process's shared + default backend at once fail with `RuntimeError: Already borrowed`, so every + execution there holds a second lock, `execution.execution_lock`: one per + process, re-entrant, and ordered after the project lock. Anything that can + heal runs before it is taken, and `project_lock` raises in a thread that + holds it and would take a new file lock (#118, implemented by #242). The lock is blocking and has no timeout, and the work it now covers is long: a materialization runs single-partition, and a create runs its query twice @@ -581,8 +609,14 @@ process. Paddy, 2026-09-20: correct first. #186 tracks the waiting. Two tallyman servers pointed at one project is unsupported. Paddy: "you have done something diabolical and deserve the results." The file lock would still serialize their writes on one machine, and nothing else about them is safe, -because each holds in-process state the other never sees. -buckaroo-data/tallyman#183 tracks detecting that case and refusing to start. +because each holds in-process state the other never sees. So `tallyman run` +claims its data dir (`TALLYMAN_HOME`, the directory that holds every project) +with an exclusive `flock` on `/server.lock`, and a second +`tallyman run` on the same data dir is refused with a message naming the one +that holds it. The MCP server finds its companion from the port recorded in that +file, and with no server on the data dir it notifies nothing +(buckaroo-data/tallyman#183, implemented by #241). `tallyman mcp` and +`tallyman serve` claim nothing. ### D12. Files are deleted only by an explicit user action @@ -854,6 +888,13 @@ What the implementation does that the text above does not say, or says different file no entry names `orphan`; the delete route answers 409 with the reason. - **The lock (D11).** `catalog_state.project_lock` is public and re-entrant per thread. `build_and_persist` holds it for the whole build, including the recipe import, so a chained build waits for its parent's materialization. +- **A create publishes its snapshot last (D4).** The build calls `materialize(..., publish=False)`, + which leaves the file complete at its temp name, and moves it into place with `publish_snapshot` after the manifest + is written. A build that fails before then removes only its temp file, so a file already at the path, such as the + one a reset left on disk (D14), is kept. A heal publishes at once. A crash between the manifest write and the + publish leaves a complete entry over the file that was there before, or over none, which a heal then makes, and + leaves the temp file behind. A source entry's import needs no staging: it keeps a snapshot already at its path, + which holds the same rows, since that path is named by the bytes and the reader options. - **Clones (D13, D14).** `ensure_cas_path` clones to a unique temp name, since two builders cloning one source shared one. `gc_cas` moves clones into `/bullpen/cas/` when given a bullpen. - **Not done here.** The independent bugs listed under D9 step 3 (the chart error loop, eager notebook sessions, the diff --git a/plans/ADR-008-row-order-of-reads.md b/plans/ADR-008-row-order-of-reads.md index 3b2e7ef..259350d 100644 --- a/plans/ADR-008-row-order-of-reads.md +++ b/plans/ADR-008-row-order-of-reads.md @@ -1,24 +1,29 @@ # ADR: Row order of reads (every file carries `__row_order`, every page sorts by it) -- **Status:** Implemented in buckaroo-data/tallyman#189 (2026-09-21), at Paddy's request to implement the - set. Where this text says a decision is "not yet confirmed" or "Proposed", it was implemented as - written; the differences between the text and the code are under "Implementation notes" below. - Original status: Proposed (2026-09-18, revised 2026-09-20 in the grilling session, and again the same day after a review of PR #184, which made the fix for #168 a precondition of D2 and D7, and a third time that day after a second review: D4 and D6 were tightened, and D10 to D12 are new). Awaiting Paddy's review; nothing here is implemented. The first draft pinned row order with an engine setting. Paddy proposed baking a row-order column into every file tallyman writes and sorting every page by it. The measurements below favour that, so it is now the decision and the engine setting is the rejected alternative under D5. Amends `plans/ADR-005-intelligent-csv-import.md` INV-1 (the name and position of the row-order column) and INV-2 (the trailing `order_by`). Corrects a threshold quoted in `plans/ADR-006-read-path-loads-builds.md` decision D5 (the canonical sort) and three other places. - **Extended by `plans/ADR-011-sources-are-aliases.md` (2026-09-22, PR #218).** D2 and D7 here make a raw - `xo.deferred_read_parquet` a build error, so that every source enters through an ordered copy. ADR-011 D2 - applies the same rule to `read_project_file` and `tallyman_read_csv` themselves: there is now no way to - author a read of a file the catalog does not own, and the sanctioned route in is an explicit import. - The ordered copy is then not a separate kind of file. An imported source version is an ordinary entry - (ADR-011 D1) and the copy **is** that entry's snapshot: one parquet in - `compute_cache/result_cache/`, named by the entry's content hash, written by pyarrow in the pinned - layout of ADR-009. `compute_cache/ordered_sources/`, the `copy_key` - (`md5(digest ǀ reader signature)`) that named files in it, `manifest.ordered_copies` and the - `ensure_ordered_copy` / `existing_ordered_copy` / `recreate_ordered_copy` trio are all deleted. - What survives from `ordered_copy.py` is what describes a read rather than performs one: - `ORDERED_COPY_ROW_GROUP_ROWS`, the reader descriptors (`parquet_reader`, `csv_reader`) and the reader - signature that the source entry's hash is built from. - Everything this ADR decided about row order is unchanged: every file tallyman writes still carries - `__row_order` as a last `int64` column, `0..N-1`, and every page still sorts by it. +- **Status:** Accepted (2026-09-22), implemented. Where the text below says a + decision is "not yet confirmed" or "Proposed", it was implemented as written; + the differences between the text and the code are under "Implementation + notes". Open defects touching it: #199, #200, #205 and #206. +- **Amends:** + - `plans/ADR-005-intelligent-csv-import.md` INV-1 (the name and position of the + row-order column) and INV-2 (the trailing `order_by`). + - `plans/ADR-006-read-path-loads-builds.md` decision D5 (the canonical sort): + corrects a threshold it quotes, and three other places. +- **Amended by:** `plans/ADR-011-sources-are-aliases.md`. The refusal of D2 and + D7 extends to `read_project_file` and `tallyman_read_csv`, so no recipe can + read a file the catalog does not own, and a file enters only by an explicit + import. The ordered copy of D2 is not a separate kind of file: an imported + source version is an ordinary entry (ADR-011 D1), and the copy is that entry's + snapshot, one parquet in `compute_cache/result_cache/` named by the entry's + content hash and written by pyarrow in the pinned layout of ADR-009. + `compute_cache/ordered_sources/`, the copy key, `manifest.ordered_copies` and + the functions that made and re-made copies do not exist. What remains of + `ordered_copy.py` describes a read rather than performing one: + `ORDERED_COPY_ROW_GROUP_ROWS`, the reader descriptors (`parquet_reader`, + `csv_reader`) and the reader signature the source entry's hash is built from. + Everything this ADR decides about row order holds: every file tallyman writes + carries `__row_order` as a last `int64` column, `0..N-1`, and every page sorts + by it. - **Context:** the 2026-09-18 cache audit (tallyman @ `a748ea6`, buckaroo 0.15.4, xorq 0.3.26, xorq-datafusion 0.2.7). - **Tickets:** #168 (CSV sources bypass source identity; D2 and D7 depend on diff --git a/plans/ADR-009-digest-stability.md b/plans/ADR-009-digest-stability.md index 181e47a..88096a4 100644 --- a/plans/ADR-009-digest-stability.md +++ b/plans/ADR-009-digest-stability.md @@ -1,9 +1,20 @@ # ADR: Digest stability (a heal is flagged only when the result changed) -- **Status:** Implemented in buckaroo-data/tallyman#189 (2026-09-21), at Paddy's request to implement the - set. Where this text says a decision is "not yet confirmed" or "Proposed", it was implemented as - written; the differences between the text and the code are under "Implementation notes" below. - Original status: Proposed (2026-09-18, revised 2026-09-20 in the grilling session: D3 gains two format requirements from `plans/ADR-008-row-order-of-reads.md`, D1 lost its speed gate, and D6 is new; and again the same day after a review of PR #184: D1 and D3 now say what single-partition execution leaves undetermined, and D6's cheap-entry half moved to #185; and a third time that day after a second review: D1 now says how a loaded build gets onto the single-partition connection, and D3's format version covers the ordered copies of sources). Awaiting Paddy's review; nothing here is implemented. Amends `plans/ADR-004-result-digest-canonical-ordering.md` (Option A's "hash the snapshot bytes") and decision D5 of `plans/ADR-006-read-path-loads-builds.md` (the canonical sort), which said "`result_digest` keeps its file-hash definition". The canonical sort itself is unchanged and is still required. +- **Status:** Accepted (2026-09-22), implemented. Where the text below says a + decision is "not yet confirmed" or "Proposed", it was implemented as written; + the differences between the text and the code are under "Implementation + notes". Open defect touching it: #208 (an unfaithful heal reaches only the + healed entry). +- **Amends:** `plans/ADR-004-result-digest-canonical-ordering.md` (Option A's + "hash the snapshot bytes") and decision D5 of + `plans/ADR-006-read-path-loads-builds.md` (the canonical sort), which said + "`result_digest` keeps its file-hash definition". The canonical sort itself is + unchanged and is still required. +- **Amended by:** `plans/ADR-011-sources-are-aliases.md`. The ordered copy of a + source that this ADR mentions (written by polars, in its own layout) does not + exist: a source entry's snapshot is written by pyarrow with the settings of + every other snapshot, in row groups of 122,880 rows, and D3's format version + covers it. - **Reading decision labels:** a bare label such as "D2" in this document always means this ADR's own decision. Another ADR's decision is always written with its ADR number and a few words saying what it decides. diff --git a/plans/ADR-011-sources-are-aliases.md b/plans/ADR-011-sources-are-aliases.md index c83c5ad..2c480e7 100644 --- a/plans/ADR-011-sources-are-aliases.md +++ b/plans/ADR-011-sources-are-aliases.md @@ -1,31 +1,14 @@ # ADR: A raw input is an alias, and files enter only by an explicit import -- **Status:** Accepted (2026-09-22), implemented. Stage 1 — the import path and - the refusals (D1, D2, D3, D5, D9, D10, D12) — is PR #217, with two later - decisions of the same day folded into D1: the snapshot is written by pyarrow - in the pinned layout, and it is cache that `ensure_materialized` re-creates - from the clone. Stage 2 — D6, D8 and the rewrite of every call site that - authored a raw file read — is PR #218, stacked on it. The fixes from reviewing - both are PR #219, stacked on #218: one set of bytes is one version under one - alias, an alias's kind matches its entries' kind, a version is named by the - alias that holds it now, and a re-import that finds a version already there - rewrites nothing but a missing snapshot, verified like any heal (D1, D3, D11, - D12 and the PR #219 implementation notes). Amends - `plans/ADR-002-source-identity-content-hash.md` (its modes, its `sources` map - and its reconstruction caveat go; the clone store stays), - `plans/ADR-005-intelligent-csv-import.md` (its reader runs at import, not in - a recipe) and `plans/ADR-008-row-order-of-reads.md` (its refusal extends to - `read_project_file`, and the ordered copy becomes the source entry's - snapshot). Still outstanding: `docs/architecture.md`, `docs/caching.md`, - `docs/expression-lifecycle.md` and `docs/system-contract.md` all describe the - deleted source axis, and are left until PR #216 lands to avoid rewriting the - same four files twice. - Written from Paddy's design session the same day, after a review of PR #189 - found that a child pinned to its parent by content hash is permanently stale - and reports itself as an UNEXPLAINED orphan. The direction is his: "treat the - orders.parquet like an alias that got updated externally", and "we want - people pointing at aliases not hashes. aliases are what gives us the dag, - hashes are brittle." +- **Status:** Accepted (2026-09-22), implemented. +- **Amends:** + - `plans/ADR-002-source-identity-content-hash.md` (its modes, its `sources` map + and its reconstruction caveat go; the clone store stays). + - `plans/ADR-005-intelligent-csv-import.md` (its reader runs at import, not in + a recipe). + - `plans/ADR-008-row-order-of-reads.md` (its refusal extends to + `read_project_file`, and the ordered copy becomes the source entry's + snapshot). - **Reading decision labels:** a bare label such as "D5" in this document always means this ADR's own decision. Another ADR's decision is always written with its ADR number and a few words saying what it decides. @@ -33,34 +16,34 @@ turned up a class of defect that no local fix addresses: staleness is computed on two axes, and the second one (a recorded source digest no longer matches the file on disk) asks a question the system cannot always answer. - This ADR removes the axis by making raw inputs first-class catalog objects. -- **Tickets:** none filed yet. Supersedes the open design question in the #189 - review notes ("a hash-pinned child stays stale on the source axis"), and - removes the cause of the staleness scan's digest-memo wipe - (`source_identity.py:87-101`). -- **Affected code:** `src/tallyman_xorq/io.py` (`read_project_file`, - `tracked_expr_from_alias`, `pinned_expr_from_alias`, - `_note_parent_records`), `src/tallyman_xorq/source_identity.py` (`mode`, - `digest_for`, `ensure_cas_path`, `recon_cas_path`, `gc_cas`), - `src/tallyman_xorq/staleness.py` (axis 2, `_force_source_rehash`), - `src/tallyman_xorq/recalc.py` (`_classify_orphan`), - `src/tallyman_xorq/ordered_copy.py` (the copy key and its reader options), - `src/tallyman_core/aliases.py` (a second kind of alias), - `src/tallyman_core/manifest.py` (`sources`), - `src/tallyman_mcp/server.py` (`catalog_load_parquet`), - `docs/system-contract.md`. -- **Related ADRs:** `plans/ADR-002-source-identity-content-hash.md` (the `.cas` - clone store this builds on; its `off` and `salt` modes go, and so does its - `sources` map, D6), - `plans/ADR-005-intelligent-csv-import.md` (reader options and the suggestion - contract, which move to import time), - `plans/ADR-008-row-order-of-reads.md` (its D2 and D7 — every source enters - through an ordered copy, a raw read is a build error — are the precedent this - extends to `read_project_file` itself), + The review found that a child pinned to its parent by content hash is + permanently stale and reports itself as an UNEXPLAINED orphan. This ADR + removes the axis by making raw inputs first-class catalog objects, the + direction Paddy set in the design session that followed: "treat the + orders.parquet like an alias that got updated externally", and "we want + people pointing at aliases not hashes. aliases are what gives us the dag, + hashes are brittle." +- **Tickets:** none of its own. It removes the mechanisms that #191, #197, + #198, #207 and #211 were about (the source axis and the ordered copy). +- **Code:** `src/tallyman_xorq/source_import.py` (the import: `update_and_depend`, + the entry hash, the refusals, the advised import call), + `src/tallyman_xorq/source_identity.py` (the clone store), + `src/tallyman_xorq/io.py` (reading aliases, and refusing raw reads), + `src/tallyman_xorq/build.py` (refusing `xo.deferred_read_*` of a file tallyman + did not write), `src/tallyman_xorq/staleness.py` (one axis), + `src/tallyman_xorq/materialize.py` (healing and pinning a source snapshot), + `src/tallyman_xorq/ordered_copy.py` (reader descriptors and the snapshot's + row-group size), `src/tallyman_core/aliases.py` (alias kinds), + `src/tallyman_core/manifest.py` (`provenance`), + `src/tallyman_core/catalog_state.py` (clone retention), + `src/tallyman_mcp/server.py` (`catalog_import_source`) and the companion's + code-edit and promote-diff routes. +- **Related ADRs:** the three it amends, and `plans/ADR-007-tallyman-owned-materialization.md` (D13, a file is cache only if it can be re-created; a source version's snapshot passes that test, because the clone of the imported bytes re-creates it). -- **Evidence:** the probes in the 2026-09-22 session, reproduced under Problem. +- **Evidence:** the probes of the 2026-09-22 design session, reproduced under + Problem. ## Terms @@ -90,7 +73,7 @@ That map is written as a retention record. `io._note_parent_records` `gc_cas` keeps the clones alive for as long as any live entry needs them. The scan then reads the same map as a freshness record. One field, two meanings. -Probed on this branch, with a child pinned to its parent by hash: +Probed on #189's code, with a child pinned to its parent by hash: ``` parent 8bb2a4661c1a worthy=True sources={orders.parquet: 2d20a090} @@ -149,10 +132,11 @@ existing `VERSION_REF_RE` and `resolve_version_ref` with no new syntax. A source alias is a distinct kind. `catalog_revise` is refused on one (there is no recipe to revise), as is promoting a diff onto it, on every surface that -offers those (the MCP tools and the companion's code-edit and promote-diff -routes, which refuse before building anything). Rename and unalias behave as -they do for catalog aliases. The kind is recorded in the alias store so -`catalog_list` can separate inputs from computations. +offers those (the MCP tools, and the companion's code-edit and promote-diff +routes, which answer 409). Every surface refuses before building anything, with +one text (`aliases.source_alias_refusal`), so no entry is left behind under no +alias. Rename and unalias behave as they do for catalog aliases. The kind is +recorded in the alias store; `catalog_list` does not show it yet. **An alias's kind matches the kind of the entries it points at.** A catalog alias never points at a source entry (one whose manifest records `provenance`), @@ -164,7 +148,7 @@ A source version's `provenance` keeps the name it was **imported as**. A rename moves the alias's history to a new name and an unalias drops it, and neither rewrites that record, so anything that names the version as it is now (the pin reason, the heal error and the import it advises, the rebuild's replay) asks -the alias store which source alias holds it. +the alias store which source alias holds it (`current_source_version`). Consequence: a child of a source records `{hash: , ref: "orders", follow: True}` — the same edge shape as any catalog parent. The DAG @@ -192,19 +176,20 @@ never looks at it. Only when the clone is gone as well are the rows unrecoverable. Then the snapshot is the last copy, so it is pinned, the delete answers 409, and a read of an entry whose snapshot is already gone fails with an error naming the -missing clone and the re-import that repairs it. A source version is therefore +missing clone and the re-import that repairs it. For a version whose entry a +reset retired, a clone the reset parked in the bullpen counts as present, since +a reset forward brings both back. A source version is therefore the one kind of entry whose "cache or data" answer depends on a second file being present, which is the price of not storing the bytes three times. **One set of bytes, read one way, is one source version under one alias.** Importing bytes that another alias of the project already holds, at any version, -is an error naming that alias and version. The likely way to get there is not -knowing the bytes are already in the project, and letting two aliases share the -entry meant one entry directory carrying one alias's name in its provenance and -recipe while another alias pointed at it. (This document first said two such -imports share one entry. PR #219 reversed that after the #217/#218 review found -the second import rewrote the first alias's manifest, and a failure part-way -deleted its entry.) +is an error naming that alias and version (`_refuse_bytes_held_elsewhere`). The +likely way to get there is not knowing the bytes are already in the project. +Letting two aliases share the entry would leave one entry directory carrying one +alias's name in its provenance and recipe while another alias pointed at it, and +a second import over it could rewrite the first alias's manifest, or delete its +entry by failing part-way. A second name for a source is a catalog entry whose recipe reads it: `catalog_create("orders_eu", "... expr = tracked_expr_from_alias('orders')")`. @@ -255,7 +240,9 @@ that minted it. If its snapshot is gone, the import heals it the way `ensure_materialized` does, from the clone (restored from the caller's bytes first when it is gone too) and checked against the recorded `result_digest`, so a reader that now parses the bytes differently is recorded as an unfaithful heal -rather than becoming the version's rows. +and the manifest keeps the digest of the rows that were imported. A directory +without a manifest, which a crash part-way through an import leaves, is not an +entry, and the import writes it again. **Considered and not shipped: `import_once_and_depend(outside_path, alias)`.** An idempotent form for a standalone script that wants the same data wherever it @@ -296,14 +283,15 @@ This alone removes the defect under Problem: the child that pinned Delete axis 2. An entry is stale when a followed alias has moved, and that is the only reason. -`manifest.sources` goes with it, rather than merely narrowing. It exists as the -retention closure `gc_cas` walks, and once every input is an entry the closure -is the DAG: a source version's file is alive exactly while its entry is alive, -which the parent edges already record. `manifest.ordered_copies` goes for the -same reason (D1 makes the copy a snapshot), and `_note_parent_records` -(`io.py:537`) — the function that folds a parent's sources and copies into its -child, and so the immediate cause of the defect under Problem — is deleted -entirely. +`manifest.sources` goes with it, rather than merely narrowing. It existed as +the retention closure `gc_cas` walks, and once every input is an entry the +closure is the DAG: a source version's file is alive exactly while its entry is +alive, which the parent edges already record, and `gc_cas` keeps the clones that +surviving source entries name in `manifest.provenance` +(`catalog_state._live_source_digests`). `manifest.ordered_copies` goes for the +same reason (D1 makes the copy a snapshot), and so do `dependents.sources_of` +and `_note_parent_records`, the function that folded a parent's sources and +copies into its child and so the immediate cause of the defect under Problem. Deleted with them: `_force_source_rehash`, `source_digests.json`, `digest_for`'s stat memo, the `unknown` verdict for a removed source file, and the source-axis @@ -337,8 +325,9 @@ disk now. The new MCP tool is `catalog_import_source(outside_path, alias, pinned_version=None, **reader_options)`, mapping to `update_and_depend`. The old -tool's semantics change too much to keep the name: today re-running it with an -existing alias is an error, and under this ADR it mints the next version. +tool's semantics changed too much to keep the name: re-running it with an +existing alias was an error, and an import under an existing alias mints the +next version. ### D11. Version history is append-only and monotonic @@ -367,7 +356,7 @@ options with it (`source_import.import_call`, in the MCP tool's `schema=` and `reader_options=` shapes), and a pinned import refused for naming another entry says the reader options may be what differs. -This removes the hash fork found in the #215 review, where a function-valued +This removes a hash fork, where a function-valued reader option's `repr` carries a memory address and re-derives a new copy key on every build. Options are evaluated once, at import, and stored. @@ -376,9 +365,7 @@ every build. Options are evaluated once, at import, and stored. - A child of a source alias is stale after an import advances it, and clear after recalc. The permanently-stale case of Problem cannot be constructed, because D5 refuses the recipe. -- Every row of D3's `update_and_depend` table, including the four errors. (The - `import_once_and_depend` test this list first named went with the function, - which D3 records as considered and not shipped.) +- Every row of D3's `update_and_depend` table, including the four errors. - The same bytes under a second alias are refused, naming the alias and version that hold them, and the first alias's entry is left byte for byte as it was. - Two projects importing one file each hold their own entry, snapshot and @@ -417,39 +404,40 @@ every build. Options are evaluated once, at import, and stored. ## Consequences -- Every entry's hash changes, because a source entry's hash now stands between - a recipe and its data. The corpus is rebuilt once, as it is for ADR-007 D9. - Nothing is migrated: per the project rule, existing catalogs are rebuilt and - existing recipes are rewritten. In this repo that is 309 `read_project_file` - references across 55 test files, plus nine `src/` modules — the largest - single piece of work in the ADR, and mechanical. -- The staleness scan stops touching the filesystem outside the arena, so its - cost becomes a read of `aliases.jsonl` and the manifests. The digest-memo - defect disappears rather than being fixed. -- A source's history becomes inspectable: which versions existed, when each was +- Every entry's hash depends on a source entry's hash, which stands between a + recipe and its data, so a corpus built before this design is rebuilt rather + than migrated, as for ADR-007 D9 (per the project rule, existing catalogs are + rebuilt and existing recipes rewritten). +- The staleness scan does not touch the filesystem outside the arena, so its + cost is a read of `aliases.jsonl` and the manifests. There is no digest memo + for it to wipe. +- A source's history is inspectable: which versions existed, when each was imported, what each was imported from, and which entries were built on each. Diffing v1 against v2 uses the existing diff, since both are entries. -- Data no longer arrives by being dropped in `data/`. That is a real loss of +- Data does not arrive by being dropped in `data/`. That is a real loss of convenience, and the import step is the price of the model. -- `docs/architecture.md`, `docs/caching.md`, `docs/expression-lifecycle.md` and - `docs/system-contract.md` all describe the source axis and need rewriting. - ADR-002 is narrowed rather than superseded: its clone store survives, its modes and its `sources`-as-freshness reading do not. ## Implementation notes -**Stage 1 (PR #217, 2026-09-22): the import path and the refusals — D1, D2, D3, -D5, D9, D10, D12.** What the code does that this document did not say: +Where the code says more than the decisions above. - **A source entry's content hash is `md5("source||")`, - truncated to xorq's 12 hex.** It cannot come from `build_expr`, because the - generated recipe reads the snapshot and the snapshot is named by the hash. The - bytes and the reader options are the whole identity (`source_import.py`). -- **The generated recipe still calls `read_project_file`**, and a contextvar - (`_SOURCE_ENTRY`) is what makes that call legal and resolves it to the entry's - own snapshot. `result_cache._recipe_expr` sets the same contextvar when it - reconstructs a source entry. The path in the recipe is provenance; nothing - opens it. + truncated to xorq's 12 hex characters** (`source_import.source_entry_hash`). + It cannot come from `build_expr`, because the generated recipe reads the + snapshot and the snapshot is named by the hash. The bytes and the reader + options are the whole identity. Reader options must be plain values the entry + can record; a callable option is refused. +- **The generated recipe still calls `read_project_file`**, and a context + variable (`_SOURCE_ENTRY`) is what makes that call legal and resolves it to + the entry's own snapshot. `result_cache._recipe_expr` sets the same variable + when it reconstructs a source entry. The path in the recipe is provenance; + nothing opens it. The read follows the entry being minted, not the active + project, so `update_and_depend(path, alias, project=X)` works whichever + project is active. +- **`tallyman_read_csv` is refused alongside `read_project_file`.** D2 names + only the latter, but both open a file the catalog does not own. - **pyarrow writes the snapshot; polars only parses a CSV.** `result_cache/` holds one shape of parquet file, so the import writes through the settings every computed snapshot uses (`materialize._PARQUET_OPTIONS`, via @@ -461,146 +449,66 @@ D5, D9, D10, D12.** What the code does that this document did not say: which is the only reader that holds the file's row order and applies the schema DSL and inference ladder of ADR-005, and its batches go to the same writer through `collect_batches`, so nothing collects the whole frame. -- **A newer polars would not have helped.** Checked 2026-09-22: 1.40.1 is - installed, 1.44.2 is the latest on PyPI, and neither exposes the parquet - format version or a page-index option — `sink_parquet` writes format 1.0 and - the request is pola-rs/polars#12752, still open. `use_pyarrow=True` is just - pyarrow doing the write. So there is no polars upgrade that reaches the pinned - layout, and writing the file twice to re-encode it was rejected. +- **No polars version writes the pinned layout.** As of 2026-09-22 (1.40.1 + installed, 1.44.2 the latest on PyPI), polars exposes neither the parquet + format version nor a page-index option: `sink_parquet` writes format 1.0, and + the request is pola-rs/polars#12752. `use_pyarrow=True` is just pyarrow doing + the write, and writing the file twice to re-encode it was rejected. - **The pinned layout costs size on high-cardinality numeric columns.** On the 1.5M-row `tests/big_parquet.py` fixture (`id` sequential int64, `g` 200 - values, `v` random float) the snapshot goes from 16,188,593 bytes (polars) to - 25,495,419 (pyarrow), +57%, with the same 13 row groups of 122,880 rows. - Measured cause: pyarrow dictionary-encodes by default, one 122,880-row chunk - of int64 is ~983 KB, just under its 1 MB `dictionary_pagesize_limit`, so the - dictionary never overflows to PLAIN and the file carries both. The same data - is 16,205,749 bytes with `use_dictionary=False`, and 17,827,724 with - 1,048,576-row groups (where the limit does kick in). That is a question about - `_PARQUET_OPTIONS` and about whether the two row-group sizes should be one, - which is ADR-009's to answer for every snapshot at once; nothing here writes a - source snapshot differently from a computed one to avoid it. Also measured: - the page index is present either way — polars writes one without being asked — - so the format version and the encoder are what actually differed. -- **`SNAPSHOT_FORMAT_VERSION` stays 1**, as it did for the same change to the - ordered copies in #215, even though the bytes of a source snapshot do change. - A source entry imported before this change records the digest of the - polars-written file, so re-creating its snapshot now writes different bytes - and the heal is recorded as unfaithful, attributed to the recipe rather than to - the format. Bumping the version would attribute that one case correctly and - would misattribute every computed entry, whose layout did not change; per the - project rule the corpus is rebuilt instead, so there is no such entry. + values, `v` random float) the snapshot is 25,495,419 bytes written by pyarrow + against 16,188,593 written by polars, +57%, with the same 13 row groups of + 122,880 rows. Measured cause: pyarrow dictionary-encodes by default, one + 122,880-row chunk of int64 is ~983 KB, just under its 1 MB + `dictionary_pagesize_limit`, so the dictionary never overflows to PLAIN and + the file carries both. The same data is 16,205,749 bytes with + `use_dictionary=False`, and 17,827,724 with 1,048,576-row groups (where the + limit does kick in). That is a question about `_PARQUET_OPTIONS` and about + whether the two row-group sizes should be one, which is ADR-009's to answer for + every snapshot at once; nothing writes a source snapshot differently from a + computed one to avoid it. The page index is present either way. +- **`SNAPSHOT_FORMAT_VERSION` is 1** for source snapshots written by pyarrow as + well as for the polars-written files they replaced. A heal of a polars-written + source snapshot would therefore be recorded as unfaithful and attributed to + the recipe rather than the format; bumping the version would have + misattributed every computed entry instead, whose layout did not change. Per + the project rule the corpus is rebuilt, so there is no such entry. - **Provenance lives in one manifest field**, `manifest.provenance` (`{alias, version, path, digest, suffix, reader, imported_at}`), and its - presence is what makes an entry a source entry. `manifest.sources` stays empty - for one, so `catalog_state._live_source_digests` had to learn to keep a clone - alive from `provenance.digest` as well — otherwise a reset retires the only - copy of the imported bytes. - Its `alias` and `version` are the name the version was imported as (D1): a - rename or unalias does not rewrite them. -- **`entry_staleness` skips axis 2 for a source entry** rather than reporting it - unknown. A small piece of D6, forced: every source entry would otherwise carry - a permanent "source axis unknown". -- **`data/` stops being special.** An import takes any path. A relative path - resolves against the working directory, and the replay CLI expands + presence is what makes an entry a source entry. Its `alias` and `version` are + the name the version was imported as (D1). +- **`StaleReason.axis` is always `"alias"`.** The field is kept so a staleness + reason has the same shape in the API and the UI; a one-value field says the + axis is no longer a choice. +- **`data/` is not special.** An import takes any path. A relative path resolves + against the working directory, and the replay CLI expands `${TALLYMAN_PROJECT_ROOT}` in storyboard arguments so `demo/storyboard.json` can name a file shipped with the project. -- **Open question 1 is answered "keep both".** The imported bytes stay in - `data/.cas/` beside the ordered snapshot, because ADR-005's - suggestion-and-retry contract has to re-read the file as imported and the - outside path may be gone. Every import therefore stores the data twice, and - the second copy now earns its keep twice over: it is also what makes the - snapshot cache rather than data (D1). Dropping the clone would not halve the - storage, it would move the same bytes into the pinned column. -- **`tallyman_read_csv` is refused alongside `read_project_file`.** D2 names only - the latter, but both open a file the catalog does not own, and leaving the CSV - reader open would be a hole in the rule. -- A source entry keeps a real `xorq_build/`, so `load_entry_expr`, the diff and - the viewer treat it as an ordinary entry. Nothing reads it on the normal path: - a worthy entry whose snapshot exists is served by a bare read of that file. - -**Deliberate debt, now paid.** `TALLYMAN_LEGACY_FILE_READS=1` disabled D2's -refusal for the whole suite so stage 1 could land before the call-site rewrite. -Stage 2 deletes the variable, the `tests/conftest.py` line that set it and the -branch in `io.py` that read it. There is no way to author a raw file read. - -**Stage 2 (PR #218, 2026-09-23): D6, D8 and the call-site rewrite.** What the -code does that this document did not say: - -- **`manifest.sources` is deleted, not narrowed.** D6 says the field reverts to - being the retention closure; in the event it has no readers left at all, since - a source version is an entry and the closure is the DAG. `dependents.sources_of` - goes with it, and `io._note_parent_records` — the function whose folding of a - parent's digests into its child is the direct cause of the defect under - Problem — is deleted outright. -- **`StaleReason.axis` survives as a field that is always `"alias"`.** Deleting - it would change the shape of every staleness reason in the API and the UI for - no gain; a one-value field reads the same and says the axis is no longer a - choice. -- **The fixture split is `orders_parquet` and `orders_src`.** The first is the - file, for a test about the bytes; the second imports it and returns the alias, - for a test that needs a recipe. The alias is `orders_src` rather than `orders` - because several tests already create a catalog alias called `orders`, and a - name is one kind or the other and never both. -- **An import must not depend on which project is active.** `update_and_depend(path, - alias, project=X)` failed whenever `X` was not the active project: the generated - recipe's `read_project_file` resolved the ambient project while `_SOURCE_ENTRY` - named `X`, so the importer's own recipe hit the refusal written for authored - recipes. The read follows the entry being minted. Every call site passed before - the rewrite because the `project` fixture also activates its project; the cache - lab, which warms xorq in a project of its own without activating it, is what - exposed it. -- **Two test expectations changed because the behaviour is now right**, not to - keep them green. The page-load profiler measures three entries where it - measured two, because an imported source version is an entry. And the reset - round-trip has to import its second file *after* the first checkpoint: import - it before, and its source entry survives the reset back, so no clone is left - referenced only by a retired entry and the case ADR-007 D13 is about stops - being exercised at all. Arranged correctly, that test is the proof that a - backward reset parks a clone in the bullpen rather than unlinking it — the - failure mode that deleting `manifest.sources` could otherwise have caused - silently. - -**Review fixes (PR #219, 2026-09-24).** Found reviewing #217 and #218; each -landed test first. What the code does that the decisions above did not say: - -- **Duplicate bytes were a shared entry.** Stage 1 let a second alias import - bytes another alias held, on the grounds that the hash made it one entry. The - second import re-ran `_mint` over that entry, rewriting the first alias's - provenance and recipe, and a failure part-way deleted the entry directory the - first alias pointed at. D1 now refuses the import (`_refuse_bytes_held_elsewhere`). -- **The kind rule lives in `set_alias`** (D1), not in each tool, after - `catalog_alias` put a catalog name on a source entry and opened it to - `catalog_revise`. The companion's `PUT /api/code` and promote-diff routes had - no refusal at all: they built the entry, failed to point the source alias at - it and answered 500 with the entry left under no alias. They now refuse with - the tools' text (`aliases.source_alias_refusal`) before building, answering 409. -- **A version is named by the alias that holds it now** - (`source_import.current_source_version`). Naming it by `provenance.alias` - made the heal advice mint a second alias under the old name after a rename, - and made `scripts/rebuild_native_catalog.py` replay a renamed source under the - old name: children reading the new name failed to rebuild, and the order of - the source's versions was lost. -- **A repair import heals** (D3). It used to go through `_mint`, which rebuilt - the entry, rewrote the manifest and recorded whatever digest it wrote: a - probe with a reader that drops a row went from 10 rows to 9 under the same - hash with nothing recorded. An entry directory without a manifest, which a - crash part-way through an import leaves, is not an entry, and a re-import - writes it again. -- **Advised imports carry the reader options** (D12). Without them the heal - error's advice for a CSV named another entry and was refused. +- **A source entry keeps a real `xorq_build/`**, so `load_entry_expr`, the diff + and the viewer treat it as an ordinary entry. Nothing reads it on the normal + path: a worthy entry whose snapshot exists is served by a bare read of that + file. +- **Test fixtures.** `orders_parquet` is the file, for a test about the bytes; + `orders_src` imports it and returns the alias, for a test that needs a recipe. + The alias is `orders_src` rather than `orders` because several tests create a + catalog alias called `orders`, and a name is one kind or the other. The + page-load profiler counts an imported source version as an entry. The reset + round-trip test imports its second file after the first checkpoint, so that a + backward reset leaves a clone referenced only by a retired entry and parks it + in the bullpen rather than unlinking it (the case of ADR-007 D13). ## Open questions -1. ~~**Does a source version keep its raw bytes as well as its snapshot?**~~ - **Answered 2026-09-22: keep both.** A CSV imported with the wrong schema - would otherwise be fixable only from the outside file, which may be gone, - and ADR-005's suggestion-and-retry contract assumes the bytes can be - re-read. Keeping the clone also decides D1's other half: the snapshot is - re-creatable from it, so it is cache in the sense of ADR-007 D13 rather than - irreplaceable data. Every import stores the data twice; the clone is - copy-on-write where the filesystem offers it, so the initial cost is near - zero, but the snapshot is a real second materialization. See the - implementation notes for the measured sizes. +1. **Does a source version keep its raw bytes as well as its snapshot?** + Answered 2026-09-22: keep both. A CSV imported with the wrong schema would + otherwise be fixable only from the outside file, which may be gone, and + ADR-005's suggestion-and-retry contract assumes the bytes can be re-read. + Keeping the clone also decides D1's other half: the snapshot is re-creatable + from it, so it is cache in the sense of ADR-007 D13 rather than irreplaceable + data. Every import stores the data twice; the clone is copy-on-write where + the filesystem offers it, so the initial cost is near zero, but the snapshot + is a real second materialization. See the implementation notes for the + measured sizes. 2. **Directories and multi-file datasets.** A dataset that arrives as `orders/part-0000.parquet`, `part-0001.parquet`, … rather than one file: does `update_and_depend("~/exports/orders/", "orders")` import the directory diff --git a/plans/auto-recalc-on-revise.md b/plans/auto-recalc-on-revise.md index b0d6e66..6e9df92 100644 --- a/plans/auto-recalc-on-revise.md +++ b/plans/auto-recalc-on-revise.md @@ -6,6 +6,14 @@ dependents in the same atomic checkpoint.** The recalc engine already exists; this is about *triggering* it from revise, folding it into *one* git transaction, and *reporting* what cascaded. No new compute. +- **Status note (2026-09-24):** implemented, on by default. Three things below + are out of date: the SPA listens for the `recalc` SSE event; the + checkpoint-free walk is `_replay_cone`, not `_recalc_walk`; and there is no + source axis (`plans/ADR-011-sources-are-aliases.md`): new data arrives by + `catalog_import_source`, which takes the place of the `catalog_load_parquet` + named below, and an import that mints a version triggers auto-recalc like a + revise. `docs/reactive-recalc.md` ("Trigger model") describes the current + behaviour. - **Builds on:** `recalc.py` (the cone walk + re-point + single checkpoint), `catalog_revise` (`tallyman_mcp/server.py`), the companion revise route (`tallyman_companion/app.py`), and the normalized cross-process notify/SSE path diff --git a/plans/cache-soundness-audit.md b/plans/cache-soundness-audit.md index d688196..3763ea9 100644 --- a/plans/cache-soundness-audit.md +++ b/plans/cache-soundness-audit.md @@ -3,6 +3,17 @@ - **Status:** Research note (2026-07-30). Companion to `docs/system-contract.md` (the proposed contract) — this is the inventory of where the codebase currently violates it *beyond* the instances #163 already lists. +- **Status note (2026-09-24):** a point-in-time inventory. Several findings no + longer hold: reads load the frozen build + (`plans/ADR-006-read-path-loads-builds.md`); tallyman writes its own result + files (ADR-007 to ADR-009); a data file enters only by an import, as a source + entry named by its bytes and reader options, so a build never reads a file on + a path the user controls (`plans/ADR-011-sources-are-aliases.md`), which removes + the mechanism of W3 (#168 is still open on GitHub); + entry session ids include the project and tallyman keeps no session map, + though diff session ids still do not include it (B1, #172); every snapshot a + heal re-creates is verified before it is served (W6); and `csv_ordered/` is + gone. Check an issue's current state before relying on a finding here. - **Template:** the #163 disease — a cache or code path whose key claims immutability while its value depends on mutable state; read-time resolution of a live name on behalf of a frozen artifact; self-heal that manufactures diff --git a/plans/datafusion-scan-order-findings.md b/plans/datafusion-scan-order-findings.md index 13eaadc..782e6ee 100644 --- a/plans/datafusion-scan-order-findings.md +++ b/plans/datafusion-scan-order-findings.md @@ -1,6 +1,12 @@ # DataFusion scan ordering — findings and decision - **Status:** Resolved (2026-06-25) — **stick with polars for the ingest reader.** +- **Status note (2026-09-24):** the polars decision stands for CSV: polars parses + a CSV once, when it is imported (`plans/ADR-011-sources-are-aliases.md`), and + a parquet file is copied in file order by pyarrow, which keeps its types. The + row-index column is `__row_order`, not `original_row_order`, and + `result_digest` is a content digest of the snapshot read back, not a hash of + its bytes (`plans/ADR-009-digest-stability.md`). - **Origin:** Investigation behind `plans/ADR-004-result-digest-canonical-ordering.md`, prompted by datafusion's nondeterministic parallel scan order producing false "drift" on the order-sensitive result digest. diff --git a/plans/ds-demo-scripts.md b/plans/ds-demo-scripts.md index ba041d7..4ba5e68 100644 --- a/plans/ds-demo-scripts.md +++ b/plans/ds-demo-scripts.md @@ -2,6 +2,8 @@ Generated 2026-05-30. Five DS-oriented demo scripts on freshly-downloaded public datasets, chosen by a fan-out design pass + critic for distinct DS techniques, plotting variety, and coverage of the recently-changed code (diff backends, result cache, primary-key inference, post-processing live-reload). +Status note (2026-09-24): the probes below describe the code of 2026-05-30. Their statements about `result.parquet`, the `ParquetSnapshotCache` result cache, `classify_build` and `ensure_result` no longer match the code, and the scripts' `catalog_load_parquet` and `read_project_file` calls no longer run: a file enters only through `catalog_import_source`, and a recipe reads it by alias (`plans/ADR-011-sources-are-aliases.md`). `docs/caching.md` describes the current design. + **Datasets** are staged at `/tmp/tallyman_demo_data/`. Before running a script, copy its parquet(s) into the target project: `cp /tmp/tallyman_demo_data/ ~/.tallyman/projects//data/`. **Verified blocker (post_processing.py:35-61,110-119):** the `process(expr)` sandbox builds `__builtins__` from a fixed allowlist with no `__import__`, so `import numpy/scipy/sklearn/pandas` fails inside any post-processing function. The pandas-DataFrame return path only works via `expr.execute()` + DataFrame methods; third-party stats/ML is unreachable. Scripts 2 and 5 are designed to hit this wall. diff --git a/plans/native-catalog-store.md b/plans/native-catalog-store.md index 51e1948..178c565 100644 --- a/plans/native-catalog-store.md +++ b/plans/native-catalog-store.md @@ -5,6 +5,14 @@ post-implementation bug-hunt + hardening pass (2026-06-17, see below). Refined through a parallel code investigation and a grilling pass; the decisions below are settled unless marked open. +Status note (2026-09-24): `plans/ADR-007-tallyman-owned-materialization.md` D14 +(a reset leaves `compute_cache/` alone) retired `compute_cache.jsonl` and the +reset's compute-cache prune and restore described below. `entries.jsonl` is the +only pointer file, and a reset moves the source clones no surviving entry refers +to into `bullpen/cas/` instead of deleting them. The source-identity `salt` mode +and its salted hash, which the zip naming below allows for, are gone +(`plans/ADR-011-sources-are-aliases.md` D8). + ## Implementation notes (deviations from the plan as written) The cut landed as: `1a` failing-first markers → native recipe store + checkpoint diff --git a/plans/open-bugs-2026-09-24.md b/plans/open-bugs-2026-09-24.md new file mode 100644 index 0000000..01be541 --- /dev/null +++ b/plans/open-bugs-2026-09-24.md @@ -0,0 +1,138 @@ +# Open bugs, 2026-09-24 + +- **As of:** `1f8cb02`, the tip of `feat/adr-007-009-cache-redesign` (PR #189), which + carries ADR-007, ADR-008, ADR-009 and ADR-011. Nothing here is on `main` yet. +- **Scope:** every open issue in buckaroo-data/tallyman, checked against that commit. + #190 to #240 were reproduced there, by a probe or by reading the code. The older + issues were re-read against the code, with a probe where one was cheap. +- **Status words:** + - **confirmed**: the bug reproduces at `1f8cb02` as filed. + - **changed**: the mechanism was rewritten and the bug survives in a different form, described in the row. + - **new**: filed on 2026-09-24. +- **Priority:** + - **P1**: rows, pins or edges come out wrong without an error. + - **P2**: a request fails, hangs, or the UI shows the wrong state. + - **P3**: leaks, misleading messages, missing hints and cleanup. + +## P1: wrong data without an error + +| # | status | what goes wrong | +|---|---|---| +| #229 | new | A tracked child and a pinned child with the same query are one entry, so the first build's edge wins. A pinned alias can then move when its parent is revised. | +| #228 | new | `_raw_parquet_read_check` exempts every file under `compute_cache/`, so a recipe can read an entry's snapshot by hash. It records no parent and never goes stale. | +| #232 | new | An unfaithful heal of a source version is blamed on the recipe. A corrupt clone is healed from, pinned, and not repaired by a re-import. | +| #208 | confirmed | An unfaithful heal of a worthy parent changes its cheap children's rows under their hashes, and only the parent is flagged. | +| #209 | confirmed | The expanded-build marker does not record the project root, so a copied project reads the old project's paths. A comment adds that the marker is trusted even when the expanded folder is gone. | +| #200 | confirmed | `full_diff` treats `__row_order` as data, so one inserted row shows every later row as changed. | +| #13 | confirmed, wider than filed | The diff's membership sentinel is the first non-key column, so a null in that column puts a row in the wrong bucket. Ordinary data triggers it, not only key-only diffs. | +| #206 | confirmed | The snapshot writer drops any column named `__row_order_right`, including one the author made. | +| #205 | confirmed (code) | The canonical sort (ADR-008's fixed row order for a worthy entry) leaves nested columns out of the tie-break, so rows tied on every sortable column can still come out in different orders. | +| #142 | confirmed | The CSV reader, which runs at import, null-fills short rows instead of raising. ADR-005 deferred it. | +| #169 | confirmed | Primary-key inheritance finds the parent by an alphabetical alias scan and checks the key by column name only, so a diff can join on a non-unique key. | +| #172 | changed | Entry sessions are fixed. Diff sessions are still keyed `diff--` with no project, and source hashes are the same across projects, so one project can be served another's diff session. | + +## P2: failures, hangs, and a UI that shows the wrong state + +| # | status | what goes wrong | +|---|---|---| +| #190 | confirmed | `put_code` and `api_promote_diff` build on the companion's event loop, so the whole UI freezes while one waits for the project lock. | +| #201 | confirmed | `reload_project_sessions` posts `/reload_expr` once per catalog entry (source versions included), one after another, on the event loop. | +| #202 | confirmed | Every grid open posts `/load_expr`. Concurrent opens load twice, and a promoted diff re-runs Buckaroo's stats on every open. | +| #203 | confirmed | An unfaithful heal runs its checks, and the companion's forced Buckaroo reload, while holding the project lock. `_heal_a_source` is a second place this happens. | +| #233 | new | A recipe's alias reads resolve against the active-project file, not the project being built. An MCP session or a companion route on another project gets a refusal that names the wrong project's file. | +| #39 | confirmed | The active-project file outranks `TALLYMAN_PROJECT` and `mcp --project`, so an MCP process cannot be pinned to a project. Pinning it would also make #233 the normal case. | +| #164 | confirmed | The post-processing validator dry-runs on a fixed `{a, b}` table, which forces a guard that hides shape errors and blocks diff-scoped filters. | +| #240 | new | Alias, notebook and `config.json` writes load, change and replace the file without the project lock. If a second writer loads the file before the first one's replace lands, the first change is lost. | +| #238 | new | `catalog_recalc` with a source entry as a root fails, because the replay runs the generated `read_project_file` outside the import, and the failure is recorded against good data. | +| #227 | new | `catalog_import_source` lets a CSV reader error, or a clone that fails its digest check, escape. Nothing is recorded, and the message names the clone and `tallyman_read_csv`. | +| #224 | new | Importing a parquet file with a `fixed_size_binary` or UUID column fails with a 3,000-character KeyError traceback that names no column. | +| #235 | new | The SPA has no listener for `project_reset`, `unfaithful_heal`, `entry_added` or `alias_changed`, so an open page goes stale. | +| #199 | confirmed | `assert_joinable` refuses semi and anti join chains over entries, though they add no right-side columns. | +| #161 | confirmed | Auto-recalc rebuilds dependents with no prompt, so they show "no prompt" from v2 on. | +| #111 | confirmed, wider than filed | The alias is moved before the entry config is carried forward. This happens in four paths (revise, `PUT /code`, recalc, import). In recalc, a carry-forward failure escapes the build's `try`. | +| #170 | confirmed | Buckaroo is given `project_root = artifacts/`, but klasses live under `artifacts/catalog/`, so project stats and grid post-processing never load. | +| #177 | confirmed | Every klass reload wipes the entry's stat cache, which forces a cold recompute. | +| #156 | changed | The global session lock is gone. A load longer than `10 s + 1 s per million rows` still reports a timeout although Buckaroo finishes it, and same-hash opens are not merged. | +| #34 | changed | A timeout now returns a typed status with a retry. Loads are still synchronous and not deduplicated (see #202). | +| #37 | changed | An alias gets a distinct `no_build` status, but `GET /api/session/{alias}` still does not resolve aliases. | +| #1 | confirmed | `extra_grid_config` from the display config never reaches `/load_expr`. Sending it would also trigger #202's reload on every open. | +| #45 | confirmed (not re-measured) | The diff page computes the full diff synchronously on first open, then waits up to 30 s on `/load_expr`. | +| #210 | confirmed | Dropping `_ResultPlan.loaded` frees nothing. xorq's unbounded `translate_from_yaml` cache keeps a backend per `load_expr`, so the fix has to be on tallyman's side. | +| #22 | changed | A checkpoint no longer lists `compute_cache/`, but it still runs under the lock before the response, and its cost grows with the entry count. The bullpen only grows. | +| #186 | confirmed | The project lock blocks with no timeout and no busy indicator, and page heals and imports take it. | +| #23 | confirmed | The marimo export omits post-processing, summary stats, charts and display config. | +| #43 | confirmed | Exported cells set `TALLYMAN_PROJECT` but not `TALLYMAN_HOME`. | +| #44 | confirmed | Exported cells re-run the full query and hand the whole result to `mo.ui.table`. | + +## P3: leaks, messages, hints, cleanup + +| # | status | what goes wrong | +|---|---|---| +| #225 | new | A failed import leaves its clone in `data/.cas/` and its snapshot in `result_cache/`. Nothing ever deletes the clone. | +| #226 | new | A writer killed mid-write leaves a `.tmp` in `result_cache/` or `data/.cas/` that the Cache page never lists and nothing deletes. | +| #230 | new | Every failed build leaves a `tallyman_expr_.py` in the OS temp dir. This machine has 4,163 of them. | +| #35 | confirmed (partly fixed) | A build that raises cleans up its entry dir. A killed one still leaves a dir without a manifest, and nothing sweeps it. | +| #239 | new | Re-importing a version whose clone is lost does not restore the clone while its snapshot exists, so the version stays pinned. | +| #234 | new | The import error for an older version's bytes names reset commands that don't exist (`tallyman reset`, `catalog_reset_to`). | +| #38 | confirmed | The customization tools (add stat, klass, post-processing) return errors without `record_error`, so they get no `error_id` and don't appear in `/api/errors`. | +| #40, #42, #107 | confirmed | Missing build-error hints: `try_cast` raises an Arrow cast error; `window(partition_by=)` raises TypeError; a window over a `lag()` column fails to plan. | +| #41 | confirmed | An aggregate over an ordered `ibis.window()` covers the whole partition (`[3,3,3]`, not the running `[1,2,3]`), and no lint catches it. | +| #236 | new | ADR-011 leftovers: dead helpers (`recon_cas_path`, `_spec_sig`, `_polars_overrides`), text describing ordered copies and identity modes, and no test for the callable reader-option refusal. | +| #110 | confirmed | `carry_forward_entry_config`'s guard branches have no tests. | +| #138 | confirmed | `packages/embed/` is still in the tree and nothing builds it. | +| #157 | changed | The stat cache now hits across a Buckaroo restart. What remains is #177's wipe and Buckaroo's cache log being thrown away. | + +## Features and design questions still open + +#237 (`catalog_list` should report an alias's kind), #185 (the purity verdict is not +recorded or passed to children), #187 (float aggregates over parent row-group +layout: only the research list remains), #188 (build every diff as an entry), #146 +(re-scoped to window `order_by` and ordered aggregates), #150 (exact decimals), #151 +(pattern-based parents), #125 (hard-fail nondeterministic recipes), #140 (remove +`api_data`), #4 (`api_data` still round-trips through pandas and JSON), #10 (cache +eviction), #60 (log levels), #78 (per-column fingerprints), #106 (per-project venvs), +#2 (`catalog_train`: fitting works in a recipe, the storyboard beats are missing), #3, +#6, #7 (scratch entries are never recalculated, and nothing in the UI says so), #8. + +## Fixed or obsolete, still open on GitHub + +These could be closed with a comment giving the evidence. None has been closed yet. + +| # | verdict | evidence at `1f8cb02` | +|---|---|---| +| #12 | fixed | `worthiness.classify_expr` walks the live expression; the regex classifier is gone. | +| #77 | fixed | Snapshots are named by content hash and read through `${TALLYMAN_PROJECT_ROOT}`. A project copied to a new home reads and heals. The rest is #209. | +| #132 | fixed | The `entry_cache` route exists, CI builds the SPA, and the bundle warns when its revision differs from the server's. | +| #133 | fixed | The detail request no longer touches Buckaroo, and the grid tab shows a spinner, then the grid or the error with a retry. | +| #134 | fixed | The metadata tab shows imported bytes, last-modified, the cheap or expensive label with its reason, and parent and child links. | +| #135 | fixed | `catalog_revise` refuses a recipe that names its own alias, and parent references render as links. | +| #141 | fixed | A named schema that misses the header raises; a tuple schema binds by position. | +| #143 | fixed | Inference escalates from 100 rows to 10k to the whole file, and a failed pinned type suggests a schema. | +| #144 | fixed | `time`, `decimal` (as float64) and non-nullable types map. | +| #145 | fixed | The zone and precision are kept. The offset-less text case was #231, since fixed (below). | +| #89 | mostly fixed | Scan, recalc, auto-recalc and their tools shipped. Only the archive overlay is unbuilt. | +| #168 | obsolete | ADR-011 removed the mechanism: `tallyman_read_csv` is refused in recipes, and an edited file is imported as a new version. | +| #147 | obsolete | A UDF makes an entry worthy, and a worthy entry is canonically sorted. | +| #117 | obsolete | Snapshots are named by content hash, so a UDF entry has one stable path. | +| #98 | obsolete | The native catalog store replaced xorq's catalog; `assert_consistency` is no longer called. | +| #30 | obsolete | "Worthy" is a row-order correctness rule (ADR-008 D4), not a cost choice. Eviction is #10. | +| #31 | obsolete | ADR-007 D6 (tallyman hands Buckaroo a view build over the snapshot) settles the cache-hint protocol. | +| #28 | obsolete | The endpoint exists only on unmerged PR #26. | + +## Closed in this pass + +Closed with a comment linking the fix, on 2026-09-24 unless noted. Each fix is on #189's +branch and reaches `main` with #189. + +- **Fixed:** + - #193 (a failed build deleted the snapshot on disk; fixed by #222). + - #194, #195, #196 (reset and banner dismissal lost the pin; fixed by #223). + - #197, #198 (the ordered copy changed types and rewrote reader options; fixed by ADR-011). + - #118 (page reads executed on the shared backend with no lock and failed with `Already borrowed`; + fixed by #242, closed 2026-09-26). + - #183 (two tallyman servers on one project went undetected; fixed by #241, closed 2026-09-26). + - #204 (`cache_worthy` fell back to "a snapshot exists" when the manifest was missing; fixed by + #245, closed 2026-09-26). + - #231 (a zoned timestamp in an import schema read offset-less text as UTC and converted it; fixed + by #244, closed 2026-09-26). +- **Obsolete** (ADR-011 removed the mechanism): #191, #207, #211. diff --git a/plans/project_switcher.md b/plans/project_switcher.md index cfe29f2..3b48f74 100644 --- a/plans/project_switcher.md +++ b/plans/project_switcher.md @@ -4,6 +4,12 @@ Status: **post-grill, ready to ticket and implement.** Fifteen decisions locked through interactive grilling; recorded below with the reasoning that landed each one. Implementation order at the bottom. +Status note (2026-09-22): implemented. Two things below have since changed. The +home root is `~/.tallyman-notebooks/` (movable with `TALLYMAN_HOME`), not +`~/.tallyman/`. And decision 11's session file is gone: a Buckaroo session id +is `entry--`, derived and never stored +(`plans/ADR-007-tallyman-owned-materialization.md` D6). + ## Goal A user-visible project switcher in the companion chrome: diff --git a/plans/recalc-mechanism.md b/plans/recalc-mechanism.md index bcbded9..c0decce 100644 --- a/plans/recalc-mechanism.md +++ b/plans/recalc-mechanism.md @@ -5,6 +5,15 @@ `plans/reactive-staleness-recalc.md` (the staged plan) and `plans/adr-reactive-catalog-recalc.md` (the design questions). Stage 1 (staleness detection) shipped in PR #126; the #115 cold-read blocker in PR #127. +- **Status note (2026-09-24):** three things below no longer hold. A cheap child's + build inlines its parent's frozen graph and does not re-run the parent's + recipe. There is no source axis (`plans/ADR-011-sources-are-aliases.md` D6): a + data file is a source alias whose versions are imports, so new data advances an + alias, its followers go directly stale and the rest of the cone transitively, + exactly as for a revise. Auto-recalc on revise and on import exists and is on + by default (`plans/auto-recalc-on-revise.md`). And a pin is written + `pinned_expr_from_alias("-v")`; a bare hash is refused (ADR-011 D5). + `docs/reactive-recalc.md` describes the current behaviour. - **Code:** `src/tallyman_xorq/recalc.py` (the engine), `tallyman_xorq/staleness.py` and `tallyman_xorq/dependents.py` (Stage 1, the inputs it reads), the MCP tools `catalog_scan_staleness` / `catalog_recalc` in `tallyman_mcp/server.py`, and the diff --git a/plans/remove-ondemand-result-parquet.md b/plans/remove-ondemand-result-parquet.md index 43d7516..6822ff6 100644 --- a/plans/remove-ondemand-result-parquet.md +++ b/plans/remove-ondemand-result-parquet.md @@ -10,6 +10,15 @@ implementation and re-introduced part of the layer, so the work grew an and were corrected during implementation. Those deltas are called out in [What the #101 plan got wrong](#what-the-101-plan-got-wrong) rather than buried. +Status note (2026-09-24): the removal of `result.parquet` still holds. Its +premise that xorq's `ParquetSnapshotCache` is the single materialized copy was +replaced by `plans/ADR-007-tallyman-owned-materialization.md`: no build holds a +xorq cache node, and the single copy is tallyman's own snapshot, +`compute_cache/result_cache/.parquet`, written by `materialize`. +The source-identity `salt` mode discussed below is gone +(`plans/ADR-011-sources-are-aliases.md` D8). `docs/caching.md` describes the +current design. + ## Goal Delete every "materialise a per-entry `result.parquet` on demand" path. xorq's diff --git a/plans/result-digest-vs-xorq-staleness.md b/plans/result-digest-vs-xorq-staleness.md index c640592..91bb01e 100644 --- a/plans/result-digest-vs-xorq-staleness.md +++ b/plans/result-digest-vs-xorq-staleness.md @@ -1,6 +1,12 @@ # result_digest vs xorq's content-aware cache staleness - **Status:** Research note (2026-06-26) +- **Status note (2026-09-22):** tallyman no longer uses xorq's cache at all + (`plans/ADR-007-tallyman-owned-materialization.md`): no build holds a cache + node, and tallyman writes its own snapshots. `result_digest` is now a content + digest of the snapshot read back (`plans/ADR-009-digest-stability.md`), not a + hash of its bytes. The note's argument for keeping an output digest separate + from input identity still holds. - **Question:** xorq already ships a content-aware cache that knows when a cached result is stale. Why does tallyman maintain a separate `result_digest` / `verify_result_faithful`? Can we reuse xorq's staleness mechanism instead — or a diff --git a/tallyman_explanation.md b/tallyman_explanation.md index 9e714ad..4bebb89 100644 --- a/tallyman_explanation.md +++ b/tallyman_explanation.md @@ -72,8 +72,8 @@ When the expression is added, it is also executed and cached, along with summary ## Authoring — the LLM writes expressions, over MCP - Every analysis step is a **xorq expression** submitted through an MCP tool - call, not code typed into a cell. Claude Code (or any MCP client) is the - editor. + call. Claude Code (or any MCP client) is the editor, and the browser's Code + tab can submit a revision too. - **Named aliases** (`orders`, `by_region`) are the unit of work. An alias is a mutable pointer to the latest content hash of a logical step. - **Expressions depend on other expressions by name.** A child either *follows* @@ -83,18 +83,26 @@ When the expression is added, it is also executed and cached, along with summary produced an expression is stored alongside it, versioned with it. No "write docs for this" round-trip, and the doc can't drift from the code because they were minted together. -- **Intelligent CSV/parquet import** with a deterministic reader and an error +- **Intelligent CSV import** with a deterministic reader and an error contract designed to hand the LLM a usable suggestion when a file doesn't - parse cleanly. + parse cleanly. Parquet and CSV files alike are read once, in file order, + when they are imported. +- **Data files are aliases too.** A file enters the catalog only by an explicit + import, `catalog_import_source`, which copies its bytes in and makes each + version of it an entry under a *source alias*. Import it again to bring in new + data: the next version is minted and everything downstream recomputes. A + recipe never opens a file, so editing or deleting the original changes + nothing. - **Projects** — multiple independent catalogs, with `project_new`, `project_list`, `project_switch`. ## Identity and caching — execute once - **Content-addressed entries.** An entry's identity is a hash of its - expression structure plus its source-file digests. Same code + same inputs → - same hash → same directory on disk. Resubmitting identical work is free. -- **Materialize what is expensive.** Each input file is read once into an - ordered copy that numbers its rows, and an entry judged expensive (aggregates, + expression structure and its inputs, down to the bytes of every imported file. + Same code + same inputs → same hash → same directory on disk. Resubmitting + identical work is free. +- **Materialize what is expensive.** Each imported file is read once into a + snapshot that numbers its rows, and an entry judged expensive (aggregates, joins, sorts, windows, UDFs) has its result written to a snapshot file when it is created. Cheap row-preserving entries recompute rather than pay for storage. @@ -102,18 +110,21 @@ When the expression is added, it is also executed and cached, along with summary the digest recorded when it was first written, before anything reads it. - **Result digest.** Materialized entries record a digest of their result, computed from the contents of the file and not its bytes, so a different - parquet writer or row-group size does not change it. A new entry's query is - also run twice when it is created. If the two runs, or a later rebuild, - disagree with the recorded digest, the expression is nondeterministic - (sampling, `now()`, an impure UDF, source drift) and tallyman says so instead - of silently serving different numbers. + parquet writer or row-group size does not change it. A new materialized + entry's query is also run twice when it is created. If the two runs disagree, + or a deleted snapshot comes back different from the recorded digest, the + expression is nondeterministic (sampling, `now()`, an impure UDF) or the + engine changed, and tallyman says so instead of silently serving different + numbers. - **Nothing lives in a kernel.** All catalog state is on disk. The MCP server - holds no in-memory state at all. + keeps nothing in memory but caches of things that never change and the + project its session is working on. ## Reactivity — revise one alias, the graph catches up -- **Staleness on two axes**: an alias axis (a followed parent advanced) and a - source axis (an input file's content digest changed). Computing staleness is - read-only; it executes nothing and reports reasons. +- **Staleness on one axis**: an entry is stale when an alias it follows has + advanced, whether by a revise or by an import of new data. Computing staleness + runs no query, opens no data file and changes no catalog state; it reports + reasons. - **Recalc cone.** When an alias head moves, its dependents are recomputed in topological order, parents before children. - **Auto-recalc on revise**, with preview-then-commit for explicit recalcs. Only @@ -127,34 +138,42 @@ When the expression is added, it is also executed and cached, along with summary stays on disk as forensic lineage, each with its own prompt and code. - **Diff two versions** across five surfaces: code diff, schema diff, per-column summary stats, key-joined row-level side-by-side, and `head()` side-by-side. -- **Promote a diff** to accept a version. +- **Promote a diff** to a catalog entry of its own, which can be charted, + post-processed and exported like any other. - **Git-backed checkpoints.** Each mutation (and each whole recalc walk) is one atomic commit under a per-project lock. - **Reset to any revision**, with build artifacts reconciled back to the - recorded pointers — restored from the bullpen instead of recomputed. + recorded pointers — restored from the bullpen instead of recomputed. Result + files are named by content, so a reset leaves them where they are. ## Viewing — a grid over data that was never in memory -- **Buckaroo grid per entry.** The viewer is handed the entry's xorq *build - directory*, so sort, search, and summary-stat computation push down into the - xorq backend. Millions of rows without materializing them in the browser or - in a kernel. +- **Buckaroo grid per entry.** The viewer is handed a xorq *build directory*: + for a materialized entry, one read of its result file; for a cheap entry, the + entry's own small plan. Sort, search, and summary-stat computation run as + queries on it in the xorq backend. Millions of rows without loading them into + the browser or a kernel. - **Automatic per-column summary stats and histograms** on every entry. - **Charts.** Vega-Lite specs attached to an entry and rendered above its grid. - **Live updates over SSE.** The browser reflects each tool call as it lands. You and the model watch the same screen. - **Log tab** — a linear, filterable activity view, including build failures. -- **Cache tab** — per-entry disk footprint. +- **Cache tab** — the result snapshots on disk, with a delete button; each + entry's own disk footprint is on its metadata tab. ## The LLM extends the analysis UI, not just the data -- **Project-authored summary stats.** The LLM writes a `compute(col)` function; - it then appears as a pinned row in *every* grid in the project. Validated by a - dry run at tool-call time, so a bad stat fails at the MCP call rather than in - a subprocess log. -- **Post-processing functions** — a `process(expr)` per file, surfaced as an - option in the grid's post-processing dropdown. +- **Project-authored summary stats.** The LLM writes a `compute(col)` function, + meant to appear in the statistics of *every* grid in the project (a display + klass can pin it as a row of the main table). Validated by a dry run at + tool-call time, so a bad stat fails at the MCP call rather than in a + subprocess log. Today the grid does not show it: tallyman points Buckaroo one + directory above where it writes the file (#170). +- **Post-processing functions** — a `process(expr)` per file, meant to appear + as an option in the grid's post-processing dropdown (same gap, #170). - **Display klasses** — project-specific styling and formatting rules for the main and summary views. -- All three are soft-deletable and versioned with the project. +- All three are soft-deletable. Summary stats and post-processing functions are + committed with the catalog; display klasses live outside the catalog + repository, so a reset does not rewind them. ## Narrative and export - **Catalog vs. notebook are different views.** The catalog is everything ever @@ -166,10 +185,12 @@ When the expression is added, it is also executed and cached, along with summary ## Sharing - **Portable builds.** Absolute paths in build artifacts are rewritten to `${TALLYMAN_PROJECT_ROOT}` on write and expanded on load, so a project runs - from anywhere on disk. + from anywhere on disk. (A copy that carries the expanded builds along keeps + reading the old location, #209.) - **`tallyman pack` → `.tgz` → `tallyman serve`.** A colleague gets the whole - catalog, history, and grids as a read-only companion with no environment to - reproduce and no kernel to start. + catalog, history, prompts and charts as a read-only companion with no + environment to reproduce and no kernel to start. `serve` does not start + Buckaroo, so there are no grids. # Introduction to Tallyman — a walkthrough @@ -182,8 +203,9 @@ You start the way you always do: > Load `violations.csv` and show me what's in it. -The agent reads the file, writes an expression, and sends it to tallyman. A -second later the browser has an entry in it called `violations`, and clicking it +The agent imports the file, so tallyman keeps its own copy of the bytes, writes +an expression over it, and sends that to tallyman. A second later the browser +has an entry in it called `violations`, and clicking it gives you a real grid: 4 million rows, every column, with a histogram and summary statistics sitting above each column. Not `df.head()`. Not the first five rows wrapped across your terminal at 80 characters. The whole result, and @@ -209,8 +231,9 @@ you shouldn't have to wait while a model writes that code for you > Group these by violation code and count them, sorted descending. A new entry appears at the top of the catalog, named `by_code`. Above the grid -is the sentence you just typed. Below it is the expression the agent wrote. You -read both, in that order, and decide whether they agree. +is the sentence you just typed, and the Code tab beside the grid holds the +expression the agent wrote. You read both, in that order, and decide whether +they agree. That pairing is the second change. In a notebook you'd have a code cell and, if you were disciplined, a markdown cell above it explaining why. Here the @@ -253,8 +276,8 @@ enough that nobody would trust it. Tallyman can only hold one kind of thing. Every entry is a xorq expression that evaluates to a table, so the system knows the shape of every object it stores. -Diff isn't a heuristic bolted on top of arbitrary Python; it's a well-defined -operation on two tables with known schemas, and it can be specific: which +Diff is a well-defined operation on two tables with known schemas, with no +guessing about arbitrary Python involved, and it can be specific: which columns appeared, which types changed, how each column's distribution moved, which rows differ when joined on a key. @@ -296,11 +319,13 @@ expression itself. > Add a summary stat for the percent of nulls in each column. -The agent writes one small function. It appears as a new row in the statistics -block on *every* grid in the project, including the ones you built an hour ago. -The same works for formatting rules and for post-processing views. The agent -isn't just producing tables; it's customizing the tool you're reviewing them -with. +The agent writes one small function. It is meant to appear as a new row in the +statistics block on *every* grid in the project, including the ones you built an +hour ago, and the same goes for formatting rules and post-processing views. So +the agent also customizes the tool you review its tables with. (Today +formatting rules reach the grid, but project statistics and +post-processing views do not, because tallyman points Buckaroo at the wrong +directory for them: #170.) ## You do a review pass @@ -325,8 +350,9 @@ presentation you're about to give. When you're ready, export that ordering to a Marimo notebook (Jupyter is coming) and use their presentation ecosystem, or pack the project into a `.tgz` and hand it to a colleague. They run one command and get the whole thing -read-only — catalog, history, grids, prompts — with no environment to reproduce -and no kernel to start. +read-only (catalog, history, prompts, charts) with no environment to reproduce +and no kernel to start. The read-only server does not start Buckaroo, so the +grids are the one thing they do not get. ## What you never did @@ -338,10 +364,10 @@ read every line the agent wrote, which is the part that actually needed you. # For the CS-minded — what is actually happening The user-facing behavior above comes out of a small number of design decisions. -None of them are novel on their own; the leverage is in constraining the -problem space enough that they can all hold at once. +None of them is novel on its own. They work together because the problem space +is constrained enough that they can all hold at once. -**The unit is an expression, not a cell.** Every step is a xorq expression — a +**The unit of work is an expression.** Every step is a xorq expression — a method-chained relational transformation that compiles to a query plan (xorq is built on Ibis and DataFusion). Constraining the language to tabular transformations is what makes everything downstream tractable. You cannot @@ -349,12 +375,13 @@ express arbitrary Python, so tallyman never has to reason about arbitrary Python. **Identity is content-addressed.** An entry's identity is a hash of its -expression structure plus the content digests of its source files. Same code +expression structure and its inputs, which bottom out in the content digests of +the files it was imported from. Same code over the same inputs produces the same hash and lands in the same directory. Resubmitting identical work is a no-op, which makes builds idempotent and makes "did this change?" a pointer comparison rather than a diff. -**The graph is recorded, not inferred.** At build time each entry writes down +**The graph is recorded at build time.** At build time each entry writes down its direct parents as edges. An edge is either *following* (it names an alias and tracks whatever that alias currently points at) or *pinned* (it names an exact hash and never moves). Marimo infers its DAG by static analysis of Python @@ -367,12 +394,13 @@ Revising mints a new hash, advances the head, and appends to the history. Old versions are never garbage; they're the lineage, and they're what makes diff possible. -**Staleness is a read-only computation on two axes.** An entry is stale on the -alias axis when a following parent's head no longer matches the recorded hash, -and on the source axis when an input file's content digest no longer matches -what was recorded. Detecting staleness executes nothing — it compares the -manifest against the world and returns reasons. This is why the scan is instant -even on a large catalog. +**Staleness is computed on one axis, without running a query.** An entry is +stale when a parent it follows by name has a head that no longer matches the +recorded hash. Data files are no exception: a file is an alias whose versions are +imports, so new data advances an alias like a revise does, and a file edited on +disk but not imported changes nothing. Detecting staleness runs no query and +opens no data file: it compares the manifests with the alias heads and returns +reasons. **Recalc walks a cone in topological order.** When an alias head advances, its dependents form a cone. Kahn's algorithm over the intra-cone edges orders the @@ -380,9 +408,11 @@ rebuild so parents finish before children, and each entry is rebuilt exactly once no matter how many paths reach it. Pre-existing staleness elsewhere in the catalog is left alone and logged rather than swept into the walk. -**Materialization follows a worthiness rubric.** Each input file is read once -into an ordered copy, which numbers the rows in file order so that every page of -every entry is the same page on every request. A result is written to a snapshot +**Materialization follows a worthiness rubric.** Each imported file is read +once into a snapshot, which numbers the rows in file order so that a page of an +entry read through tallyman's API is the same page on every request (the grid +will do the same once Buckaroo uses the column, buckaroo-data/buckaroo#974). A +result is written to a snapshot only for entries judged expensive — those whose plan contains an aggregate, join, sort, window, or UDF, or that cannot inherit a row order. A cheap row-preserving projection recomputes on read instead of paying storage, because @@ -391,27 +421,31 @@ and runs an entry's query when the entry is created, so a failure shows up in the tool call that made the entry. A deleted snapshot is rewritten, and checked, the next time something reads it. -**Determinism is audited, not assumed.** Materialized entries record a digest of +**Determinism is checked.** Materialized entries record a digest of their result, computed from the contents of the file and in a fixed row order, so scan-order nondeterminism and writer details don't produce false alarms. A -new entry's query is run twice when it is created. If the runs, or a later -rebuild, disagree with the recorded digest, something in the expression is -nondeterministic — sampling, `now()`, an impure UDF, or source drift — and -tallyman reports it rather than silently serving different numbers under the -same hash. - -**State lives on disk; there is no kernel.** The MCP server holds no in-memory -state at all, and the companion holds only its list of SSE subscribers. This is -the design decision the whole system rests on, and the reason it suits agents: +new materialized entry's query is run twice when it is created, on a +single-partition connection so that float sums add up in one order. If the runs +disagree, or a deleted snapshot comes back different from the recorded digest, +something in the expression is nondeterministic — sampling, `now()`, or an +impure UDF — or the engine version changed, and tallyman reports it +rather than silently serving different numbers under the same hash. + +**State lives on disk; there is no kernel.** The MCP server and the companion +keep only caches of things that never change, keyed by content hash, plus a few +facts about the session: which project the MCP session is on, and the +companion's open SSE streams and diff sessions. This is the design decision the +whole system rests on, and the reason it suits agents: an agent working in a live Jupyter kernel has to reason about invisible state it cannot inspect — what's in memory, what's stale, what ran in what order. Here there is nothing invisible to reason about. Every fact is a file, addressed by content. -**The grid pushes computation down.** The viewer (Buckaroo) is handed the -entry's compiled build directory rather than a materialized dataframe, so sort, -search, and summary-statistic computation execute in the query engine against -the data on disk. Nothing needs to fit in browser memory, and nothing needs to +**The grid pushes computation down.** The viewer (Buckaroo) is handed a +compiled build directory rather than a dataframe: for a materialized entry, one +read of its result file; for a cheap entry, its own small plan. Sort, search, +and summary-statistic computation execute in the query engine against the data +on disk. Nothing needs to fit in browser memory, and nothing needs to fit in a Python process either. This is why 4 million rows opens like 4 thousand. @@ -424,7 +458,9 @@ the analysis is a real history you can walk backwards. **Builds are relocatable.** Compiled artifacts embed absolute paths; on write these are rewritten to a `${TALLYMAN_PROJECT_ROOT}` placeholder and expanded on load into a stable per-entry directory. That's what makes `pack` and `serve` -work: a project is a directory you can move to another machine and open. +work: a project is a directory you can move to another machine and open. (The +expanded directories do not record which path they were expanded for, so a copy +that carries them keeps reading the old location until #209 is fixed.) **The browser is push-driven.** The companion holds one long-lived SSE stream per client. Each MCP mutation notifies the companion, which fans out a named