feat(cache): implement ADR-007, ADR-008 and ADR-009 (tallyman-owned materialization, row order, digest stability)

.github/workflows/tests.yml

@@ -4,7 +4,8 @@ on:
   push:
     branches: [main, "spike/**"]
   pull_request:
-    branches: [main]
+    # feat/** too: fix PRs stacked on a feature branch (e.g. PRs into #189's branch) need CI for the red/green cycle.
+    branches: [main, "feat/**"]
 
 # CI runs three sequential jobs: lint → fast tests → integration tests.
 #

.mcp.json

@@ -5,8 +5,7 @@
       "args": ["run", "tallyman", "mcp"],
       "cwd": "/Users/paddy/tallyman",
       "env": {
-        "TALLYMAN_PROJECT": "spike",
-        "TALLYMAN_COMPANION_URL": "http://127.0.0.1:7860"
+        "TALLYMAN_PROJECT": "spike"
       }
     }
   }

README.md

@@ -4,7 +4,7 @@ Tallyman — a data science environment designed for coding agents. Stop squinti
 
 Two windows: Claude Code in one, a browser in the other. You work by defining a set of named results that can depend on each other, then making sure those results live up to their name. You describe them; the agent writes the queries. You tell it what `likely_customers` should mean, and a moment later the answer is a table in the browser at full size, sortable and searchable, with statistics over every column. Your prompt sits above the query the agent wrote, so you read your intent, the code that came out of it, and the result together. You notice it's catching people who already churned, so you sharpen the sentence and the agent revises the query. Everything built on top of that name updates to match, fast enough that you don't lose your place. How big the data is, what has already been computed, and what needs recomputing never enter into it.
 
-Those queries are expressions: self-contained programs that declare the raw files and other results they read. Writing the expression is the agent's entire job. An expression is declarative and can be introspected, so tallyman reads its dependencies straight off it and builds a directed graph of the project's aliased expressions.  When a parent updates, its children are recomputed; a raw file changing on disk counts as an update too. Tallyman executes each expression once and writes the result and its summary statistics to disk. Reading it back is out of core: scrolling, sorting, and searching pull only the pieces of data needed to fill the screen, so nothing has to fit in memory and four million rows opens like four thousand.
+Those queries are expressions: self-contained programs that name, by alias, the imported data and the other results they read. Writing the expression is the agent's entire job. An expression is declarative and can be introspected, so tallyman reads its dependencies straight off it and builds a directed graph of the project's aliased expressions.  When a parent updates, its children are recomputed; importing a new version of a data file counts as an update too. Tallyman executes each expression once and writes the result and its summary statistics to disk. Reading it back is out of core: scrolling, sorting, and searching pull only the pieces of data needed to fill the screen, so nothing has to fit in memory and four million rows opens like four thousand.
 
 
 You aren't typing `df.sort_values('lifetime_sales')` just to see which customers are at the top, you aren't waiting for the LLM to print a 10 row table like it's coming out of a 1200 baud modem.  You aren't running out of memory in the middle of a session.  You aren't building the ad-hoc cache that every long notebook grows, the pickle in /tmp behind an if not exists guard that you never quite trust. You aren't nursing a kernel along for days because one cell takes five minutes to rerun, or bracing yourself before you close the window. None of that is in your head while you work. What's in your head is the data and what it means.
@@ -50,58 +50,77 @@ index of all the docs — start with [docs/architecture.md](docs/architecture.md
 
 ## V0 scope
 
-End-to-end: a Claude Code MCP tool that compiles a xorq expression, materializes
-a result to a content-hashed catalog entry on disk, and pushes a live update
-to a browser companion via SSE.
+End-to-end: a Claude Code MCP tool that compiles a xorq expression into a
+content-hashed catalog entry on disk (running it, and writing its result to a
+file when the entry does expensive work), and pushes a live update to a browser
+companion via SSE.
 
 What's working:
 
-- **MCP tools** (FastMCP over stdio):
-  - Catalog: `catalog_run`, `catalog_load_parquet`, `catalog_create`,
+- **MCP tools** (FastMCP over stdio, 31 tools):
+  - Catalog: `catalog_import_source`, `catalog_run`, `catalog_create`,
     `catalog_revise`, `catalog_alias`, `catalog_rename`, `catalog_unalias`,
-    `catalog_list`, `catalog_diff`, `catalog_chart`, `catalog_recalc`, plus the
-    summary-stat / post-processing / display-klass authoring tools.
+    `catalog_list`, `catalog_diff`, `catalog_promote_diff`, `catalog_chart`,
+    `catalog_chart_errors`, `catalog_scan_staleness`, `catalog_recalc`,
+    `catalog_export_marimo`, plus the summary-stat / post-processing /
+    display-klass authoring tools.
   - Notebook: `notebook_reorder`, `notebook_remove`, `notebook_edit_markdown`.
   - Project: `project_list`, `project_new`, `project_switch`.
 
-    See [docs/architecture.md](docs/architecture.md) for the full tool surface.
+    See [docs/mcp-server.md](docs/mcp-server.md) for every tool and its side
+    effects.
 - **Companion** (FastAPI on `:7860`) — serves the React SPA
   (`packages/app/dist`) as a catch-all and exposes a JSON API + SSE under
   `/{project}/api/*`:
-  - SPA tabs: **Catalog** (entry list + detail with V_n chips and forensic
-    history), **Notebook** (curated narrative anchored on aliases, drag-reorder,
-    inline markdown editor, × remove), **Diff** (code diff, schema diff,
-    per-column stats, key-joined side-by-side, head() side-by-side), **Cache**
-    (per-entry cache footprint), and **Log** (linear, filterable activity view).
-  - JSON: `/{project}/api/{entries,entry/<hash>,aliases,notebook,errors,log,
-    data/<hash>,diff_data/...,disk_usage,result_cache,staleness}`, plus the
-    mutation routes (`PATCH notebook`, `PUT code/<alias>`,
+  - SPA pages: **Catalog** (entry list + detail with V_n chips, forensic
+    history, and a metadata tab with the entry's disk footprint, sources,
+    parents and children), **Notebook** (curated narrative anchored on aliases,
+    drag-reorder, inline markdown editor, × remove), **Diff** (code diff, schema
+    diff, per-column stats, key-joined side-by-side, head() side-by-side, and a
+    promote button), **Cache** (the result snapshots on disk, with a delete
+    button; pinned snapshots cannot be deleted, and a snapshot whose entry a
+    reset retired is labelled as such), **Log** (linear, filterable
+    activity view), and the project list.
+  - JSON: `/{project}/api/{entries,entry/<hash>,entry_cache/<hash>,
+    session/<hash>,aliases,notebook,notebook_full,errors,error/<id>,log,
+    data/<hash>,diff_data/...,disk_usage,result_cache,staleness,telemetry}`,
+    plus the mutation routes (`PATCH notebook`, `PUT code/<alias>`,
     `PUT markdown/<cell_id>`, `POST reset`, `POST recalc`,
-    `POST promote_diff/...`).
-  - `/{project}/api/sse` — live updates (`new_entry`, `build_failed`,
-    `alias_changed`, `notebook_changed`, `recalc`, `summary_stat_changed`).
+    `POST promote_diff/...`, `DELETE result_cache/<hash>`, `DELETE errors`).
+  - `/{project}/api/sse` — live updates. The SPA listens for `new_entry`,
+    `build_failed`, `notebook_changed`, `chart_attached`,
+    `post_processing_changed`, `summary_stat_changed`, `recalc` and
+    `project_switched`.
   - `/internal/notify` — the MCP server's notification hook; fans out to SSE.
 - **Buckaroo subprocess** — `tallyman run` spawns `python -m buckaroo.server`
   on `:8700` (falls back to a random port if busy), watches for the
-  `BUCKAROO_PORT=...` handshake, and lazily creates per-entry sessions on
-  first view by POSTing the entry's `xorq_build/` dir to Buckaroo's
-  `/load_expr` endpoint (PR 776) — sort/search push down to the xorq
-  backend rather than paging over a materialised parquet. The build dir is
+  `BUCKAROO_PORT=...` handshake, and opens a session for an entry by POSTing a
+  build dir to Buckaroo's `/load_expr` endpoint (PR 776), after making sure
+  every file the entry reads exists. It does this when an entry's catalog page
+  opens, and for every cell each time the notebook page loads. A *worthy*
+  entry (one whose query tallyman materialized to a result file when the entry
+  was created) is handed a view build, a build that is one read of that file
+  (`.xorq_view_build/`). A *cheap* entry (a filter, selection or computed column
+  over one file, which keeps no file of its own) is handed its own build,
   expanded into a stable per-entry path (`.xorq_build_expanded/`, gated by a
   `.complete` marker) so `${TALLYMAN_PROJECT_ROOT}` placeholders are resolved
-  before xorq's loader sees them. Sessions are persisted in a global
-  `~/.tallyman-notebooks/buckaroo_sessions.json` (keyed by content hash, shared
-  across projects) and invalidated by start-time when Buckaroo restarts.
-  Tear-down rides along with the companion. Disable with `--no-buckaroo`.
+  before xorq's loader sees them. Buckaroo's sorting, search and summary stats
+  run as queries over that build. A session's id is derived from the project and
+  the content hash, so tallyman keeps no session file. Tear-down rides along with
+  the companion. Disable with `--no-buckaroo`.
 - **Build artifacts are portable.** xorq's absolute filesystem paths are
-  rewritten to `${TALLYMAN_PROJECT_ROOT}` on write and expanded back on load.
+  rewritten to `${TALLYMAN_PROJECT_ROOT}` on write and expanded back on load
+  (with one known gap for copied projects, #209).
 - **`tallyman serve <project_dir>`** — read-only companion against a project
-  directory that may live anywhere on disk. Mutation routes return 403.
+  directory that may live anywhere on disk. Mutation routes return 403, and no
+  Buckaroo subprocess runs, so entry grids do not load.
 
 What's NOT yet implemented:
 
 1. Column-level lineage (xorq has the data; there is no lineage view today).
-2. ML training pipeline (storyboard beats 7-8).
+2. A dedicated ML training tool, `catalog_train` (storyboard beats 7-8, #2).
+   Models can already be fitted as catalog entries with `xorq.ml`, as
+   `catalog_run`'s tool description shows.
 
 ## Running the spike
 
@@ -125,7 +144,8 @@ In another terminal, launch Claude Code from this directory; it picks up
 
 Recommended prompts:
 
-> Use catalog_load_parquet to load `orders.parquet`.
+> Use catalog_import_source to import
+> `~/.tallyman-notebooks/projects/spike/data/orders.parquet` as `orders`.
 >
 > Now use catalog_create to make a named entry `shoe_sales` that groups orders
 > by region and totals the price.
@@ -147,22 +167,36 @@ tar xzf my-project.tgz -C ~/projects/
 uv run tallyman serve ~/projects/spike
 ```
 
-The companion runs read-only: same catalog, same forensic history, no edit
-affordances. Mutation routes return 403.
+The companion runs read-only: same catalog, same forensic history, same charts.
+The edit controls still show, but mutation routes return 403, and there is no
+Buckaroo grid.
+The archive includes `compute_cache/`, and a known defect (#209) makes a copied
+project's cheap entries read from the original location; see
+[docs/installing.md](docs/installing.md#sharing-a-project).
 
 ## Conventions worth knowing
 
-- xorq 0.3.x reads use `xo.deferred_read_parquet` (NOT `xo.read_parquet` — that
-  resolves through ibis's backend loader and fails). Use
-  `import xorq.api as xo` and `import xorq.vendor.ibis as ibis`. Do NOT
-  `import ibis` directly.
-- Prefer `from tallyman_xorq.io import read_project_file; t = read_project_file("name.parquet")`
-  over absolute paths — the catalog records project-relative intent and the
-  build is portable across machines/users.
+- A file enters the catalog only through `catalog_import_source(path, alias)`,
+  which copies its bytes into the project and makes each version of it an entry
+  under a **source alias**; the path can be anywhere and is never read again.
+  Import it again to bring in new data: different bytes mint the next version,
+  and the entries downstream are recalculated. A CSV takes its `schema` and
+  reader options in the import call, and they are fixed there.
+- Recipes read entries by alias, never files:
+  `tracked_expr_from_alias("orders")` follows an alias and
+  `pinned_expr_from_alias("orders-v2")` pins one version. `read_project_file`,
+  `tallyman_read_csv` and `xo.deferred_read_csv` are build errors that name the
+  import to use, and so is `xo.deferred_read_parquet` of any file outside the
+  project's `compute_cache/`. `xo.read_parquet` resolves through ibis's backend
+  loader and fails. Use `import xorq.api as xo` and `import xorq.vendor.ibis as
+  ibis`. Do NOT `import ibis` directly.
 - Content hash is xorq's build hash — same code + same inputs → same hash → same
-  entry dir (idempotent).
-- All catalog state lives on disk. The MCP server holds no in-memory state; the
-  companion only holds the SSE subscriber list.
+  entry dir (idempotent). A source entry's hash is instead an md5 of the
+  imported bytes and the reader options.
+- All catalog state lives on disk. The MCP server and the companion keep only
+  in-memory caches of things that never change (loaded builds and reads, keyed
+  by content hash), plus the MCP session's active project and the companion's
+  SSE subscribers and diff sessions.
 - `TALLYMAN_PROJECT_PATH` overrides project_dir() resolution for the active project.
   Used by `tallyman serve` to point at a project directory anywhere on disk.
 
@@ -172,3 +206,7 @@ affordances. Mutation routes return 403.
 uv run pytest                       # full suite
 uv run pytest tests/test_pack.py    # the pack / portability proof
 ```
+
+## License
+
+Tallyman is licensed under the GNU Affero General Public License, version 3 only (`AGPL-3.0-only`). The full text is in [LICENSE](LICENSE).

demo/datasets.md

@@ -97,19 +97,19 @@ df['tip_pct'] = (df['tip_amount'] / df['fare_amount'].replace(0, float('nan')))
 Type these into Claude Code with the tallyman MCP server running:
 
 ```
-Use catalog_load_parquet to load /tmp/nyc_taxi/yellow_tripdata_2024-01.parquet,
-name it yellow_jan_2024. Prompt: "NYC yellow cab trips January 2024".
+Use catalog_import_source to import /tmp/nyc_taxi/yellow_tripdata_2024-01.parquet
+under the alias yellow_jan_2024. Prompt: "NYC yellow cab trips January 2024".
 ```
 
 ```
-Create a named entry trips_by_zone that joins yellow_jan_2024 with the taxi zone
-lookup at /tmp/nyc_taxi/taxi_zone_lookup.csv on PULocationID, groups by Zone,
-and counts trips with mean fare. Name it trips_by_zone.
+Import /tmp/nyc_taxi/taxi_zone_lookup.csv under the alias taxi_zones, then create
+a named entry trips_by_zone that joins yellow_jan_2024 with taxi_zones on
+PULocationID, groups by Zone, and counts trips with mean fare.
 ```
 
 ```
-Use catalog_load_parquet to load /tmp/citibike/citibike_h1_2024.parquet,
-name it citibike_h1. Prompt: "Citibike trips January–June 2024".
+Use catalog_import_source to import /tmp/citibike/citibike_h1_2024.parquet
+under the alias citibike_h1. Prompt: "Citibike trips January–June 2024".
 ```
 
 ```

demo/storyboard.json

@@ -2,9 +2,13 @@
   "project": "spike",
   "steps": [
     {
-      "tool": "catalog_load_parquet",
-      "args": {"rel_path": "orders.parquet", "name": "orders", "prompt": "raw orders dataset"},
-      "narration": "beat 1: load the orders parquet and name it"
+      "tool": "catalog_import_source",
+      "args": {
+        "outside_path": "${TALLYMAN_PROJECT_ROOT}/data/orders.parquet",
+        "alias": "orders",
+        "prompt": "raw orders dataset"
+      },
+      "narration": "beat 1: import the orders parquet under a source alias"
     },
     {
       "tool": "catalog_create",
@@ -21,7 +25,7 @@
         "code": "from tallyman_xorq.io import tracked_expr_from_alias\nt = tracked_expr_from_alias('shoe_sales')\nexpr = t.order_by('total')\n",
         "prompt": "verify shoe_sales totals are reasonable"
       },
-      "narration": "beat 4: scratch verification — sort the result"
+      "narration": "beat 4: scratch verification \u2014 sort the result"
     },
     {
       "tool": "catalog_revise",
@@ -39,8 +43,14 @@
         "vega_spec": {
           "mark": "bar",
           "encoding": {
-            "x": {"field": "region", "type": "nominal"},
-            "y": {"field": "total", "type": "quantitative"}
+            "x": {
+              "field": "region",
+              "type": "nominal"
+            },
+            "y": {
+              "field": "total",
+              "type": "quantitative"
+            }
           }
         }
       },
@@ -52,7 +62,7 @@
         "cell_id": "PLACEHOLDER",
         "markdown": "## Shoe sales by region\n\nBroader bucket: boots, sneakers, sandals."
       },
-      "narration": "beat 10: rewrite the markdown for the audience (skipped — needs cell_id resolution)",
+      "narration": "beat 10: rewrite the markdown for the audience (skipped \u2014 needs cell_id resolution)",
       "skip": true
     }
   ]