ADR-011 stage 1: a file enters the catalog only by an explicit import

demo/datasets.md

@@ -97,19 +97,19 @@ df['tip_pct'] = (df['tip_amount'] / df['fare_amount'].replace(0, float('nan')))
 Type these into Claude Code with the tallyman MCP server running:
 
 ```
-Use catalog_load_parquet to load /tmp/nyc_taxi/yellow_tripdata_2024-01.parquet,
-name it yellow_jan_2024. Prompt: "NYC yellow cab trips January 2024".
+Use catalog_import_source to import /tmp/nyc_taxi/yellow_tripdata_2024-01.parquet
+under the alias yellow_jan_2024. Prompt: "NYC yellow cab trips January 2024".
 ```
 
 ```
-Create a named entry trips_by_zone that joins yellow_jan_2024 with the taxi zone
-lookup at /tmp/nyc_taxi/taxi_zone_lookup.csv on PULocationID, groups by Zone,
-and counts trips with mean fare. Name it trips_by_zone.
+Import /tmp/nyc_taxi/taxi_zone_lookup.csv under the alias taxi_zones, then create
+a named entry trips_by_zone that joins yellow_jan_2024 with taxi_zones on
+PULocationID, groups by Zone, and counts trips with mean fare.
 ```
 
 ```
-Use catalog_load_parquet to load /tmp/citibike/citibike_h1_2024.parquet,
-name it citibike_h1. Prompt: "Citibike trips January–June 2024".
+Use catalog_import_source to import /tmp/citibike/citibike_h1_2024.parquet
+under the alias citibike_h1. Prompt: "Citibike trips January–June 2024".
 ```
 
 ```

demo/storyboard.json

@@ -2,9 +2,13 @@
   "project": "spike",
   "steps": [
     {
-      "tool": "catalog_load_parquet",
-      "args": {"rel_path": "orders.parquet", "name": "orders", "prompt": "raw orders dataset"},
-      "narration": "beat 1: load the orders parquet and name it"
+      "tool": "catalog_import_source",
+      "args": {
+        "outside_path": "${TALLYMAN_PROJECT_ROOT}/data/orders.parquet",
+        "alias": "orders",
+        "prompt": "raw orders dataset"
+      },
+      "narration": "beat 1: import the orders parquet under a source alias"
     },
     {
       "tool": "catalog_create",
@@ -21,7 +25,7 @@
         "code": "from tallyman_xorq.io import tracked_expr_from_alias\nt = tracked_expr_from_alias('shoe_sales')\nexpr = t.order_by('total')\n",
         "prompt": "verify shoe_sales totals are reasonable"
       },
-      "narration": "beat 4: scratch verification — sort the result"
+      "narration": "beat 4: scratch verification \u2014 sort the result"
     },
     {
       "tool": "catalog_revise",
@@ -39,8 +43,14 @@
         "vega_spec": {
           "mark": "bar",
           "encoding": {
-            "x": {"field": "region", "type": "nominal"},
-            "y": {"field": "total", "type": "quantitative"}
+            "x": {
+              "field": "region",
+              "type": "nominal"
+            },
+            "y": {
+              "field": "total",
+              "type": "quantitative"
+            }
           }
         }
       },
@@ -52,7 +62,7 @@
         "cell_id": "PLACEHOLDER",
         "markdown": "## Shoe sales by region\n\nBroader bucket: boots, sneakers, sandals."
       },
-      "narration": "beat 10: rewrite the markdown for the audience (skipped — needs cell_id resolution)",
+      "narration": "beat 10: rewrite the markdown for the audience (skipped \u2014 needs cell_id resolution)",
       "skip": true
     }
   ]

docs/installing.md

@@ -128,7 +128,8 @@ it. To remove it later: `claude mcp remove tallyman -s project`.
 With the companion running (step 5) and the MCP server approved (step 6), the
 `mcp__tallyman__*` tools are available in chat. Try:
 
-> Use `catalog_load_parquet` to load `orders.parquet`.
+> Use `catalog_import_source` to import `orders.parquet` from the project's
+> `data/` directory under the alias `orders`.
 >
 > Now use `catalog_create` to make a named entry `shoe_sales` that groups orders
 > by region and totals the price.

docs/mcp-server.md

@@ -94,7 +94,7 @@ git revision. A non-empty cascade emits a `recalc` SSE event.
 | Tool | Checkpoints | SSE notify | Auto-recalc |
 |---|---|---|---|
 | `catalog_run` | yes | `new_entry`, `build_failed` | no |
-| `catalog_load_parquet` | yes | `new_entry`, `build_failed`, `notebook_changed`¹ | no |
+| `catalog_import_source` | yes | `new_entry`, `build_failed`, `notebook_changed`¹, `recalc` | yes |
 | `catalog_create` | yes | `new_entry`, `build_failed`, `notebook_changed` | no |
 | `catalog_revise` | yes | `new_entry`, `build_failed`, `recalc` | yes |
 | `catalog_alias` | yes | `alias_changed`, `notebook_changed` | no |
@@ -181,16 +181,34 @@ default authoring tool for a one-off run.
   `events.jsonl`.
 - **Promote** a scratch entry to a name afterward with `catalog_alias`.
 
-### `catalog_load_parquet(rel_path, prompt="", name="") -> dict`
-Register a parquet under `<project>/data/` as an entry by synthesizing a
-`read_project_file(rel_path)` recipe — the no-code "load this file" path. With
-`name`, behaves like `catalog_create` (names the entry, appends a notebook cell).
-- **Params:** `rel_path` (required) — path under `data/`; `prompt` — optional
-  intent / cell markdown; `name` — optional alias.
-- **Returns:** same as `catalog_run`, plus `alias` and `version` when named.
-  `{error}` if the alias already exists; `{error, error_id}` on a missing file.
-- Unlike `catalog_run`, it does **not** emit a `build_ok` event; success shows
-  only via the notifies.
+### `catalog_import_source(outside_path, alias, pinned_version=None, prompt="", schema=None, reader_options=None) -> dict`
+Import a parquet or CSV into the catalog and point the source alias `alias` at
+it (ADR-011). The only way a file enters: the bytes are copied into the arena,
+one snapshot of them is written in file order with a `__row_order` column, and
+a version of `alias` is minted. Recipes then read
+`tracked_expr_from_alias(alias)`; the original path is provenance and is never
+read again.
+- **Params:** `outside_path` (required) — any path, `data/` is not special;
+  `alias` (required) — the source alias, which may not name a catalog alias;
+  `pinned_version` — the version you claim the file is, checked rather than
+  assumed; `prompt` — optional intent / cell markdown; `schema` and
+  `reader_options` — CSV column types and `polars.scan_csv` options, recorded on
+  the entry and never re-derived (ADR-011 D12).
+- **Returns:** `hash`, `alias`, `version`, `created`, `row_count`, `schema`,
+  `path`, `digest`, `url`, plus `recalc` when advancing the alias cascaded.
+  `{error, error_id}` for a missing path, an alias collision, bytes another
+  alias already holds (the error names that alias and version), or any row of
+  the ADR-011 D3 table that is an error.
+- Re-running it on **unchanged** bytes is a no-op that returns the current
+  version, so it is safe in a script. If that version's snapshot is gone, the
+  no-op heals it from the clone and checks it against the recorded digest; it
+  never rewrites the version. Re-running it on **changed** bytes mints the next
+  version and cascades like a revise.
+- **One set of bytes is one version under one alias** (ADR-011 D1). To give a
+  source a second name, `catalog_create` an entry whose recipe is
+  `tracked_expr_from_alias("<source alias>")`; it follows the source when it is
+  re-imported. A CSV read with other reader options is a different entry and
+  may be imported under its own alias (ADR-011 D12).
 
 ### `catalog_create(name, code, prompt="") -> dict`
 Execute and persist as a **named** entry (alias) and append a notebook cell.
@@ -229,7 +247,9 @@ cell. Used after a `catalog_run` when a scratch entry earns a permanent name.
 - **Params:** `hash` (required) — must name an on-disk entry; `name` (required)
   — must be free.
 - **Returns:** `{hash, alias, version, url}`. `{error}` if the hash has no entry
-  or the name is taken.
+  or the name is taken, and if the entry is a source version: a source entry
+  belongs to its source alias, and the error names the `catalog_create` over
+  `tracked_expr_from_alias` that gives the source a second name (ADR-011 D1).
 
 ### `catalog_rename(old_name, new_name) -> dict`
 Rename an alias, preserving its full version history and notebook position.

packages/app/src/pages/CatalogPage.tsx

@@ -506,8 +506,8 @@ function EntryCacheView({ project, hash }: { project: string; hash: string }) {
         </span>
         <span className="meta">
           {data.source_tracked
-            ? `${data.source_formatted} raw source${data.sources.length > 1 ? ` (${data.sources.length} files)` : ""}`
-            : "raw source size not tracked (source-identity off)"}
+            ? `${data.source_formatted} imported bytes, kept so the snapshot can be written again`
+            : "no imported bytes — this expression is a computation over other aliases"}
           {data.modified_at && <> · last modified {data.modified_at.slice(0, 19).replace("T", " ")}</>}
         </span>
         <span className="meta cache-note">

plans/ADR-002-source-identity-content-hash.md

@@ -3,7 +3,30 @@
 - **Status:** Accepted (2026-06-18 — default flipped `off` → `cas` in #86, with
   `cp --reflink=auto` on Linux and `.cas` GC wired into `reset_to`; supersedes
   the 2026-06-10 Proposed draft below. See the reconstruction caveat under
-  Consequences.)
+  Consequences.) **Narrowed by `plans/ADR-011-sources-are-aliases.md`
+  (2026-09-22, PR #218), which keeps the clone store and deletes the rest of
+  the machinery around it:**
+  - The mode switch is gone. `TALLYMAN_SOURCE_IDENTITY`, its `off` and `salt`
+    modes and `salted_hash` are deleted (ADR-011 D8). There is one behaviour:
+    an import digests the bytes and clones them. "Identity off" and "every
+    input is versioned" cannot both be true.
+  - `manifest.sources` is gone (ADR-011 D6). It was written as the retention
+    closure `gc_cas` walks and read by the staleness scan as a freshness
+    record, and that second reading is the defect ADR-011 exists to remove.
+    Retention is now the DAG: a source version is an entry, so its clone is
+    alive exactly while that entry is, and `catalog_state._live_source_digests`
+    collects `manifest.provenance.digest` instead.
+  - `digest_for` and its `source_digests.json` stat memo are gone. Nothing
+    digests a file outside an import any more, so the memo has no callers and
+    the scan touches no file at all.
+  - The reconstruction caveat under Consequences is resolved rather than
+    inherited: `recon_cas_path` no longer falls back to serving live bytes when
+    the clone is missing and the file has drifted. That branch is an error
+    (ADR-011 D9), because the live file is no longer a build input in the first
+    place.
+  What survives unchanged is the substance of this ADR: content-addressed
+  clones under `data/.cas/<digest><suffix>`, written copy-on-write where the
+  filesystem offers it, parked in the bullpen by a reset rather than unlinked.
 - **Context ticket:** buckaroo-data/tallyman#30 (precondition for the
   result-cache rubric's content-stable `content_hash` key)
 - **Affected code:** `src/tallyman_xorq/source_identity.py` (new),

plans/ADR-005-intelligent-csv-import.md

@@ -7,6 +7,19 @@
   Amended by `plans/ADR-008-row-order-of-reads.md` (implemented in #189): INV-1's row-index column is named
   `__row_order` (not `original_row_order`) and INV-2's trailing `order_by` is gone, so `tallyman_read_csv`
   returns a plain read of an ordered copy that is keyed by the CSV's content.
+  Amended again by `plans/ADR-011-sources-are-aliases.md` (2026-09-22, PR #218): **this reader now runs at
+  import time, not in a recipe.** `tallyman_read_csv` is refused in an authored recipe alongside
+  `read_project_file` (ADR-011 D2) — both open a file the catalog does not own, and leaving the CSV reader
+  open would be a hole in the rule. A CSV enters through
+  `catalog_import_source(path, alias, separator=..., schema=...)`, and everything this ADR designed goes
+  with it: the schema DSL and its normaliser, the inference ladder (100 → 10k → whole file), the
+  suggestion engine and the error contract all run inside the import, over
+  `source_import._write_csv_snapshot`. The recipe then reads the resulting source alias.
+  Two consequences for the text below. The reader options are **fixed at import** and recorded on the
+  source entry (ADR-011 D12), so one file is read one way forever; to read it a second way, import it again
+  under a second alias. And the retry loop this ADR is built around — read, see the suggestion, adjust the
+  schema, read again — is a loop over imports, which is why ADR-011 keeps the raw bytes in the clone store
+  rather than only the parsed snapshot (its Open question 1).
 - **Affected code:** `src/tallyman_xorq/io.py` (`tallyman_read_csv`,
   `_polars_overrides`, `_IBIS_TO_POLARS`, a new schema-spec normaliser and a
   suggestion engine), `src/tallyman_mcp/server.py` (the `catalog_run` CSV

plans/ADR-008-row-order-of-reads.md

@@ -4,6 +4,21 @@
   set. Where this text says a decision is "not yet confirmed" or "Proposed", it was implemented as
   written; the differences between the text and the code are under "Implementation notes" below.
   Original status: Proposed (2026-09-18, revised 2026-09-20 in the grilling session, and again the same day after a review of PR #184, which made the fix for #168 a precondition of D2 and D7, and a third time that day after a second review: D4 and D6 were tightened, and D10 to D12 are new). Awaiting Paddy's review; nothing here is implemented. The first draft pinned row order with an engine setting. Paddy proposed baking a row-order column into every file tallyman writes and sorting every page by it. The measurements below favour that, so it is now the decision and the engine setting is the rejected alternative under D5. Amends `plans/ADR-005-intelligent-csv-import.md` INV-1 (the name and position of the row-order column) and INV-2 (the trailing `order_by`). Corrects a threshold quoted in `plans/ADR-006-read-path-loads-builds.md` decision D5 (the canonical sort) and three other places.
+  **Extended by `plans/ADR-011-sources-are-aliases.md` (2026-09-22, PR #218).** D2 and D7 here make a raw
+  `xo.deferred_read_parquet` a build error, so that every source enters through an ordered copy. ADR-011 D2
+  applies the same rule to `read_project_file` and `tallyman_read_csv` themselves: there is now no way to
+  author a read of a file the catalog does not own, and the sanctioned route in is an explicit import.
+  The ordered copy is then not a separate kind of file. An imported source version is an ordinary entry
+  (ADR-011 D1) and the copy **is** that entry's snapshot: one parquet in
+  `compute_cache/result_cache/`, named by the entry's content hash, written by pyarrow in the pinned
+  layout of ADR-009. `compute_cache/ordered_sources/`, the `copy_key`
+  (`md5(digest ǀ reader signature)`) that named files in it, `manifest.ordered_copies` and the
+  `ensure_ordered_copy` / `existing_ordered_copy` / `recreate_ordered_copy` trio are all deleted.
+  What survives from `ordered_copy.py` is what describes a read rather than performs one:
+  `ORDERED_COPY_ROW_GROUP_ROWS`, the reader descriptors (`parquet_reader`, `csv_reader`) and the reader
+  signature that the source entry's hash is built from.
+  Everything this ADR decided about row order is unchanged: every file tallyman writes still carries
+  `__row_order` as a last `int64` column, `0..N-1`, and every page still sorts by it.
 - **Context:** the 2026-09-18 cache audit (tallyman @ `a748ea6`, buckaroo
   0.15.4, xorq 0.3.26, xorq-datafusion 0.2.7).
 - **Tickets:** #168 (CSV sources bypass source identity; D2 and D7 depend on