Skip to content
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
22 changes: 16 additions & 6 deletions docs/mcp-server.md
Original file line number Diff line number Diff line change
Expand Up @@ -189,18 +189,26 @@ a version of `alias` is minted. Recipes then read
`tracked_expr_from_alias(alias)`; the original path is provenance and is never
read again.
- **Params:** `outside_path` (required) — any path, `data/` is not special;
`alias` (required) — the source alias, which may not name a catalog entry;
`alias` (required) — the source alias, which may not name a catalog alias;
`pinned_version` — the version you claim the file is, checked rather than
assumed; `prompt` — optional intent / cell markdown; `schema` and
`reader_options` — CSV column types and `polars.scan_csv` options, recorded on
the entry and never re-derived (ADR-011 D12).
- **Returns:** `hash`, `alias`, `version`, `created`, `row_count`, `schema`,
`path`, `digest`, `url`, plus `recalc` when advancing the alias cascaded.
`{error, error_id}` for a missing path, an alias collision, or any row of the
ADR-011 D3 table that is an error.
`{error, error_id}` for a missing path, an alias collision, bytes another
alias already holds (the error names that alias and version), or any row of
the ADR-011 D3 table that is an error.
- Re-running it on **unchanged** bytes is a no-op that returns the current
version, so it is safe in a script. Re-running it on **changed** bytes mints
the next version and cascades like a revise.
version, so it is safe in a script. If that version's snapshot is gone, the
no-op heals it from the clone and checks it against the recorded digest; it
never rewrites the version. Re-running it on **changed** bytes mints the next
version and cascades like a revise.
- **One set of bytes is one version under one alias** (ADR-011 D1). To give a
source a second name, `catalog_create` an entry whose recipe is
`tracked_expr_from_alias("<source alias>")`; it follows the source when it is
re-imported. A CSV read with other reader options is a different entry and
may be imported under its own alias (ADR-011 D12).

### `catalog_create(name, code, prompt="") -> dict`
Execute and persist as a **named** entry (alias) and append a notebook cell.
Expand Down Expand Up @@ -239,7 +247,9 @@ cell. Used after a `catalog_run` when a scratch entry earns a permanent name.
- **Params:** `hash` (required) — must name an on-disk entry; `name` (required)
— must be free.
- **Returns:** `{hash, alias, version, url}`. `{error}` if the hash has no entry
or the name is taken.
or the name is taken, and if the entry is a source version: a source entry
belongs to its source alias, and the error names the `catalog_create` over
`tracked_expr_from_alias` that gives the source a second name (ADR-011 D1).

### `catalog_rename(old_name, new_name) -> dict`
Rename an alias, preserving its full version history and notebook position.
Expand Down
138 changes: 119 additions & 19 deletions plans/ADR-011-sources-are-aliases.md
Original file line number Diff line number Diff line change
Expand Up @@ -5,7 +5,12 @@
decisions of the same day folded into D1: the snapshot is written by pyarrow
in the pinned layout, and it is cache that `ensure_materialized` re-creates
from the clone. Stage 2 — D6, D8 and the rewrite of every call site that
authored a raw file read — is PR #218, stacked on it. Amends
authored a raw file read — is PR #218, stacked on it. The fixes from reviewing
both are PR #219, stacked on #218: one set of bytes is one version under one
alias, an alias's kind matches its entries' kind, a version is named by the
alias that holds it now, and a re-import that finds a version already there
rewrites nothing but a missing snapshot, verified like any heal (D1, D3, D11,
D12 and the PR #219 implementation notes). Amends
`plans/ADR-002-source-identity-content-hash.md` (its modes, its `sources` map
and its reconstruction caveat go; the clone store stays),
`plans/ADR-005-intelligent-csv-import.md` (its reader runs at import, not in
Expand Down Expand Up @@ -45,8 +50,8 @@
`src/tallyman_mcp/server.py` (`catalog_load_parquet`),
`docs/system-contract.md`.
- **Related ADRs:** `plans/ADR-002-source-identity-content-hash.md` (the `.cas`
clone store this builds on; its `off` and `salt` modes go, and its `sources`
map narrows to a retention record),
clone store this builds on; its `off` and `salt` modes go, and so does its
`sources` map, D6),
`plans/ADR-005-intelligent-csv-import.md` (reader options and the suggestion
contract, which move to import time),
`plans/ADR-008-row-order-of-reads.md` (its D2 and D7 — every source enters
Expand All @@ -64,8 +69,8 @@
- **Source entry:** one version of a source alias, stored as an ordinary entry
with a content hash, a recipe and a manifest.
- **The arena:** the files tallyman owns — the clone store under `data/.cas/`
and the ordered copies under `compute_cache/`. A file outside the arena is
never read by a build.
and the snapshots under `compute_cache/result_cache/`. A file outside the arena
is never read by a build.
- **Import:** the explicit act of copying a file into the arena and pointing a
source alias at the resulting entry.
- **Provenance path:** the outside path a version was imported from, recorded
Expand Down Expand Up @@ -137,16 +142,30 @@ Everything below follows from those two sentences.
### D1. A raw input is an alias whose versions are entries

Importing a file mints an entry and points a source alias at it. The entry is
ordinary: a content hash, a recipe, a manifest, an ordered copy carrying
ordinary: a content hash, a recipe, a manifest, a snapshot carrying
`__row_order`. The alias lives in the same `aliases.jsonl` as catalog aliases,
with the same head-plus-history shape, so `orders-v2` resolves through the
existing `VERSION_REF_RE` and `resolve_version_ref` with no new syntax.

A source alias is a distinct kind. `catalog_revise` is refused on one (there is
no recipe to revise), as is promoting a diff onto it. Rename and unalias behave
as they do for catalog aliases. The kind is recorded in the alias store so
no recipe to revise), as is promoting a diff onto it, on every surface that
offers those (the MCP tools and the companion's code-edit and promote-diff
routes, which refuse before building anything). Rename and unalias behave as
they do for catalog aliases. The kind is recorded in the alias store so
`catalog_list` can separate inputs from computations.

**An alias's kind matches the kind of the entries it points at.** A catalog
alias never points at a source entry (one whose manifest records `provenance`),
and a source alias never points at a computed one. `set_alias` enforces it, so
it holds on every route that names an entry: `catalog_alias`, a revise, a
promoted diff, a recalc, an import.

A source version's `provenance` keeps the name it was **imported as**. A rename
moves the alias's history to a new name and an unalias drops it, and neither
rewrites that record, so anything that names the version as it is now (the pin
reason, the heal error and the import it advises, the rebuild's replay) asks
the alias store which source alias holds it.

Consequence: a child of a source records `{hash: <version's entry hash>, ref:
"orders", follow: True}` — the same edge shape as any catalog parent. The DAG
gains its roots. Source versions become visible in the UI, diffable against
Expand Down Expand Up @@ -177,8 +196,24 @@ missing clone and the re-import that repairs it. A source version is therefore
the one kind of entry whose "cache or data" answer depends on a second file
being present, which is the price of not storing the bytes three times.

Two imports of identical bytes under different aliases produce the same entry
hash and share one file, with both aliases pointing at it.
**One set of bytes, read one way, is one source version under one alias.**
Importing bytes that another alias of the project already holds, at any version,
is an error naming that alias and version. The likely way to get there is not
knowing the bytes are already in the project, and letting two aliases share the
entry meant one entry directory carrying one alias's name in its provenance and
recipe while another alias pointed at it. (This document first said two such
imports share one entry. PR #219 reversed that after the #217/#218 review found
the second import rewrote the first alias's manifest, and a failure part-way
deleted its entry.)

A second name for a source is a catalog entry whose recipe reads it:
`catalog_create("orders_eu", "... expr = tracked_expr_from_alias('orders')")`.
Nothing has to be added to force the hashes apart, because a source entry's hash
is an md5 of its bytes and a recipe's is xorq's hash of the expression, and the
new entry follows `orders`, so a re-import advances it. The rule keys on the entry
hash, so a CSV read two ways is still two imports under two aliases (D12), and
it is per project: two projects importing one file each hold their own entry,
snapshot and clone under the same hash.

### D2. Files enter only by an explicit import

Expand Down Expand Up @@ -210,6 +245,17 @@ The official, and only, way to advance a source alias:
| `pinned_version` beyond head+1 | error: versions cannot be skipped |
| `pinned_version=N` < head, digest matches | return vN; the head does not move |
| bytes match a version older than the head | error: see D11 |
| bytes are a version of another alias | error: one set of bytes, one alias (D1) |

"Bytes" in this table means the bytes read with the given reader options, which
is what the entry hash covers (D12). A row that ends on a version that already
exists (the no-ops, and a new alias over an entry no alias holds any more) never
rewrites that entry: its recipe, build and manifest are the record of the import
that minted it. If its snapshot is gone, the import heals it the way
`ensure_materialized` does, from the clone (restored from the caller's bytes
first when it is gone too) and checked against the recorded `result_digest`, so
a reader that now parses the bytes differently is recorded as an unfaithful heal
rather than becoming the version's rows.

**Considered and not shipped: `import_once_and_depend(outside_path, alias)`.**
An idempotent form for a standalone script that wants the same data wherever it
Expand Down Expand Up @@ -297,8 +343,11 @@ existing alias is an error, and under this ADR it mints the next version.
### D11. Version history is append-only and monotonic

An import whose bytes match a version older than the head is an error, naming
`reset_to` as the way back. Two version numbers never denote the same bytes, and
a version number never moves backwards while history is intact.
`reset_to` as the way back, and `pinned_expr_from_alias("<alias>-v<N>")` as the
way to read the old version without moving the head. Importing the bytes under
a different alias is not offered, because D1 refuses it. Two version numbers
never denote the same bytes read the same way, and a version number never moves
backwards while history is intact.

The case is unlikely — it takes a source file that was edited and then restored
exactly — and the error keeps the alternative (a v3 whose content equals v1's,
Expand All @@ -308,7 +357,15 @@ or a head that jumps backwards) out of the model.

A CSV's delimiter, schema overrides and inference settings are named once, in
the import call, and recorded on the source entry. Two recipes cannot read one
file two ways; import it twice under two aliases.
file two ways; import it twice under two aliases. That is the one case in which
the same bytes are under two aliases, and it is two entries, because the entry
hash covers the reader options.

Because the hash covers them, an import that leaves them out names another
entry. Every message that advises an import therefore prints the recorded
options with it (`source_import.import_call`, in the MCP tool's `schema=` and
`reader_options=` shapes), and a pinned import refused for naming another entry
says the reader options may be what differs.

This removes the hash fork found in the #215 review, where a function-valued
reader option's `repr` carries a memory address and re-derives a new copy key on
Expand All @@ -319,14 +376,27 @@ every build. Options are evaluated once, at import, and stored.
- A child of a source alias is stale after an import advances it, and clear
after recalc. The permanently-stale case of Problem cannot be constructed,
because D5 refuses the recipe.
- `import_once_and_depend` returns v1 after the alias has advanced to v3, and
does not read the outside file.
- Every row of D3's `update_and_depend` table, including the three errors.
- Every row of D3's `update_and_depend` table, including the four errors. (The
`import_once_and_depend` test this list first named went with the function,
which D3 records as considered and not shipped.)
- The same bytes under a second alias are refused, naming the alias and version
that hold them, and the first alias's entry is left byte for byte as it was.
- Two projects importing one file each hold their own entry, snapshot and
clone; a heal, a clone sweep or a re-import in one never reaches the other.
- `catalog_alias` of a source entry is refused, as is a source alias pointed at
a computed entry.
- After a rename, the pin reason and the heal error name the alias the version
is under now, and running the advised import repairs that version.
- A re-import of a version whose snapshot is gone rebuilds nothing, rewrites
nothing of the entry, and records an unfaithful heal when the rows differ.
- The advised import for a CSV carries its schema and reader options and, run
as written, repairs the version.
- An authored recipe calling `read_project_file` fails to build, and the error
names the import call.
- An authored recipe passing a bare content hash to `pinned_expr_from_alias`
fails, and the error names `<alias>-v<N>`.
- A source alias refuses `catalog_revise` and refuses a promoted diff.
- A source alias refuses `catalog_revise` and refuses a promoted diff, through
the MCP tools and through the companion's routes alike.
- `reset_to` rewinds a source alias to v1 and a reset forward restores v2 from
the bullpen.
- Editing the outside file after import changes nothing: the same build, the
Expand Down Expand Up @@ -374,8 +444,7 @@ D5, D9, D10, D12.** What the code does that this document did not say:
- **A source entry's content hash is `md5("source|<digest>|<reader signature>")`,
truncated to xorq's 12 hex.** It cannot come from `build_expr`, because the
generated recipe reads the snapshot and the snapshot is named by the hash. The
bytes and the reader options are the whole identity, which is what makes two
imports of one file under two aliases mint one entry (`source_import.py`).
bytes and the reader options are the whole identity (`source_import.py`).
- **The generated recipe still calls `read_project_file`**, and a contextvar
(`_SOURCE_ENTRY`) is what makes that call legal and resolves it to the entry's
own snapshot. `result_cache._recipe_expr` sets the same contextvar when it
Expand Down Expand Up @@ -426,6 +495,8 @@ D5, D9, D10, D12.** What the code does that this document did not say:
for one, so `catalog_state._live_source_digests` had to learn to keep a clone
alive from `provenance.digest` as well — otherwise a reset retires the only
copy of the imported bytes.
Its `alias` and `version` are the name the version was imported as (D1): a
rename or unalias does not rewrite them.
- **`entry_staleness` skips axis 2 for a source entry** rather than reporting it
unknown. A small piece of D6, forced: every source entry would otherwise carry
a permanent "source axis unknown".
Expand Down Expand Up @@ -489,6 +560,35 @@ code does that this document did not say:
failure mode that deleting `manifest.sources` could otherwise have caused
silently.

**Review fixes (PR #219, 2026-09-24).** Found reviewing #217 and #218; each
landed test first. What the code does that the decisions above did not say:

- **Duplicate bytes were a shared entry.** Stage 1 let a second alias import
bytes another alias held, on the grounds that the hash made it one entry. The
second import re-ran `_mint` over that entry, rewriting the first alias's
provenance and recipe, and a failure part-way deleted the entry directory the
first alias pointed at. D1 now refuses the import (`_refuse_bytes_held_elsewhere`).
- **The kind rule lives in `set_alias`** (D1), not in each tool, after
`catalog_alias` put a catalog name on a source entry and opened it to
`catalog_revise`. The companion's `PUT /api/code` and promote-diff routes had
no refusal at all: they built the entry, failed to point the source alias at
it and answered 500 with the entry left under no alias. They now refuse with
the tools' text (`aliases.source_alias_refusal`) before building, answering 409.
- **A version is named by the alias that holds it now**
(`source_import.current_source_version`). Naming it by `provenance.alias`
made the heal advice mint a second alias under the old name after a rename,
and made `scripts/rebuild_native_catalog.py` replay a renamed source under the
old name: children reading the new name failed to rebuild, and the order of
the source's versions was lost.
- **A repair import heals** (D3). It used to go through `_mint`, which rebuilt
the entry, rewrote the manifest and recorded whatever digest it wrote: a
probe with a reader that drops a row went from 10 rows to 9 under the same
hash with nothing recorded. An entry directory without a manifest, which a
crash part-way through an import leaves, is not an entry, and a re-import
writes it again.
- **Advised imports carry the reader options** (D12). Without them the heal
error's advice for a CSV named another entry and was refused.

## Open questions

1. ~~**Does a source version keep its raw bytes as well as its snapshot?**~~
Expand Down
Loading
Loading