From 230110c3233129de46a66af26ce2cf483ee1a270 Mon Sep 17 00:00:00 2001 From: Paddy Mullen Date: Tue, 16 Jun 2026 18:47:47 -0400 Subject: [PATCH 01/27] docs: plan native tallyman_core.catalog (drop xorq catalog package) Plan for replacing xorq's catalog repo manager with a tallyman-native content-addressed recipe store, and decomposing catalog.yaml into git-tracked per-concern files (JSONL for list state, one JSON per nested blob), now that dropping xorq removes assert_consistency's extra-tracked-file constraint (#48). Two cuts, one PR: the WRITE cut (native store + catalog.yaml decomposition, checkpoint owns zip creation) and the RUNTIME cut (replace import xorq.api with narrow catalog-free imports + a datafusion connect shim, since import xorq.api transitively loads the whole xorq.catalog package). Captures the investigation findings, resolved design decisions (D1-D4), module boundary, risk register, and the ordered TDD implementation sequence. Co-Authored-By: Claude Opus 4.8 (1M context) --- plans/native-catalog-store.md | 302 ++++++++++++++++++++++++++++++++++ 1 file changed, 302 insertions(+) create mode 100644 plans/native-catalog-store.md diff --git a/plans/native-catalog-store.md b/plans/native-catalog-store.md new file mode 100644 index 0000000..fa841c3 --- /dev/null +++ b/plans/native-catalog-store.md @@ -0,0 +1,302 @@ +# Plan: native `tallyman_core.catalog` — drop xorq's catalog package + +Status: proposed (2026-06-16). Refined through a parallel code investigation and +a grilling pass; the decisions below are settled unless marked open. + +## Problem + +tallyman drives xorq's `catalog/` package by subprocess from +`tallyman_core/xorq_catalog.py` (`uv run xorq catalog add / add-alias / +remove-alias`, all `--no-sync`). That package is the source of the recurring +catalog pain: + +- **`assert_consistency` runs on every add, independent of `--no-sync`** (xorq + `catalog.py:335`, and unconditionally in `CatalogEntry.__attrs_post_init__` + `1212-1213`). It asserts the tracked tree equals exactly `catalog.yaml` + each + entry's zip/metadata + alias symlinks, so any *extra tracked file* makes it + raise. This is #48, and the reason every piece of tallyman state had to be + smuggled into `catalog.yaml` keys instead of living in its own files. +- **The add subprocess commits outside tallyman's flock**, racing + `checkpoint_catalog`'s `.git/index.lock` (`catalog_state.py:383-388` already + apologizes for it and silently returns `None` when it loses). +- **#48 is live on disk**: `first-project` has four build dirs but only one + tracked recipe zip — three adds silently no-opped. Reset on that corpus + already fails `_assert_catalog_consistent`. A rebuild is required, which the + single-user no-migration rule (project `CLAUDE.md`) already permits. + +xorq splits into three layers; only the innermost hurts. We cut **(1)** and keep +**(2)/(3)**: +1. **Catalog repo manager** — `xorq/catalog/` (~7.7k LOC), driven by subprocess. +2. **Serializer** — `xorq/ibis_yaml/` (`build_expr`/`load_expr`). +3. **Engine** — `xorq.vendor.ibis` + datafusion + `xorq.expr`. + +## Scope: two cuts, one PR + +### Cut A — the WRITE cut +Delete `xorq_catalog.py`; stand up `tallyman_core/catalog.py` as the native +content-addressed recipe store; and **decompose `catalog.yaml`** so every piece +of mutable catalog state becomes its own git-tracked file. The #48 unlock +(tracking arbitrary files) is what makes the decomposition possible. + +### Cut B — the RUNTIME cut +`import xorq.api` transitively loads the entire `xorq.catalog` package +(`xorq/api.py:65 import xorq.catalog.api as catalog`; verified: bare +`import xorq` loads zero catalog modules, `import xorq.api` loads 12). Replace +tallyman's ~9 `import xorq.api as xo` sites with narrow catalog-free imports plus +a datafusion `connect()` shim. + +These must land in the same PR (ordered commits): if only Cut A lands the package +stays resident; if only Cut B lands the `uv run xorq catalog` child still runs it. + +## Ownership: `tallyman_core.catalog` and the xorq boundary + +This is not a vendored copy of xorq's `catalog.py`. tallyman already owns the +catalog *model*; the only remaining xorq tendril is the `xorq_catalog.py` +subprocess. `tallyman_core/catalog.py` is the store, written natively. We **drop** +xorq's `assert_consistency`, three-way merge, and zip contract — none carry +forward. + +After this PR, tallyman code touches xorq's catalog **never**: + +| Touchpoint today | After | +|---|---| +| `xorq catalog add` writes the zip | stdlib writer in `tallyman_core/catalog.py` | +| xorq commits zip + metadata + alias symlinks | tallyman's checkpoint is the only committer | +| `assert_consistency` on every add | never invoked against our repo | +| three-way merge | already dead (no remote, `--no-sync`); stays dead | +| zip internal format / `REQUIRED_ARCHIVE_NAMES` | irrelevant — nothing reads the zip back; format is ours | +| xorq's `entries`/`aliases` `catalog.yaml` keys | not written; nothing reads them | +| old on-disk corpus | rebuilt, never read | + +The honest boundary: this severs the *catalog*, not the *engine*. Entry contents +are still xorq build artifacts (`build_expr`/`load_expr` produce the dir we zip; +`vendor.ibis` runs it), all catalog-free. The one asterisk on "never": a +`from_catalog` reconstruction re-execs a recipe whose `import xorq.api as xo` +*passively loads* (never calls) the catalog package — see Decision D4. + +## Confirmed facts the design rests on + +1. **Nothing reads the zip back.** The only consumer of `entries/.zip` is + `_tracked_recipe_hashes` (`catalog_state.py:433`), which reads *names* via + `git ls-files`. `load_expr` consumes the expanded build *dir*, never the + archive. So the zip format is entirely ours. +2. **Two load paths read different files.** `from_catalog`/chaining + (`cached_result_expr` → `_recipe_expr`, `result_cache.py:148`) re-imports + `entry_dir/expr.py`; the Buckaroo viewer (`load_entry`) reads + `entry_dir/xorq_build/`. Both `expr.py` and `xorq_build/` must therefore be in + the durable artifact — today's zip has only `xorq_build/`, so the durability + invariant currently guards the wrong file (Decision D3 fixes this). +3. **`run_git` uses `os.posix_spawn`, not `subprocess.run`** (`git_util.py`), so + the clean "no subprocess" guard test is: monkeypatch `subprocess.run` to + raise, run a full cycle, assert no raise. +4. **tallyman never merges** (no remote, `--no-sync` everywhere). No merge + surface to port. +5. **`pyproject.toml` can't be narrowed** — `xorq.catalog` ships in the single + `xorq==0.3.26` wheel. "Drop" means stop importing/calling, enforced by a test. +6. **`.xorq_build_expanded/` is derived + machine-local** (`portable.py`): the + placeholder-expanded copy of `xorq_build/`, regenerated on load, holding this + machine's absolute paths. Never archive or track it. + +## The design (end state) + +### Tracked tree +The catalog repo gets a `.gitignore` (there is none today; only an explicit +pathspec kept staging honest). Ignore the heavy/derived artifacts: + +``` +entries/*/ # untracked build dirs (recipe lives in entries/*.zip) +bullpen/ +result_cache/ +compute_cache/ +diff_stat_cache/ +.checkpoint.lock +``` + +Then `git add -A` stages exactly the meaningful state, and the tracked tree is: + +``` +entries/.zip # immutable recipe, one blob per entry +aliases.jsonl # one line per alias +notebook.jsonl # one line per cell +entries.jsonl # pointer list: untracked build dirs (bullpen reconcile) +compute_cache.jsonl # pointer list: untracked compute-cache warm set +chart_specs/.vl.json # one Vega-Lite spec per entry +display_configs/.json # one Buckaroo/display config per entry +post_processing/.py # project-global functions (moved under the repo) +stats/.py # project-global functions (moved under the repo) +``` + +No YAML anywhere. The former `catalog.yaml` is gone; its residual pointer +bookkeeping is `entries.jsonl` + `compute_cache.jsonl`. + +### The recipe store (immutable) +- `entries/.zip` archives an **allowlist** of the entry dir: + `{expr.py, xorq_build/, schema.json, manifest.json}`. Allowlist, not + "everything except," so a future derived artifact dropped in `entry_dir` can't + leak into the durable zip. Excludes `.xorq_build_expanded/` (fact 6) and + `result.parquet` (the untracked heavy result). +- Members stored under arcname `/` (single top-level prefix, so a + future restore can reuse it). **Named by the salted `content_hash`** when + source-identity salt mode is on, not `build_path.name` — otherwise + `_tracked_recipe_hashes` (zip names) diverges from `entries.jsonl` pointers. +- **Deterministic archive** (`date_time=(1980,1,1,0,0,0)`, + `external_attr=0o644<<16`, sorted iteration, `ZIP_DEFLATED`) so the blob + doesn't churn on identical rebuilds. +- **Drop the wheel + requirements.txt** xorq injected (~178KB → ~6KB per entry); + nothing reads them, and the payoff is git *history* (every checkpoint re-stages + the zip). +- **The checkpoint owns zip creation, not the build.** The build persists a + complete `entries//` on success and cleans up the dir if this call + created it and the build failed (no partial dirs, no orphan zips). The + checkpoint, under `_project_lock`, zips any entry dir lacking a tracked zip, + then `git add -A` + commit. This is the sole git transaction and sole zip + writer: no out-of-lock writes, no loose orphans, a committed pointer always has + its recipe in the same commit, and it self-heals (a skipped/failed checkpoint + leaves a complete gitignored dir the next checkpoint picks up). + +### The decomposition (mutable state → tracked files) +The recipe is immutable; charts, styling, post-processing, stats, aliases, and +the notebook are **mutable** — edited on an existing entry without rebuilding, so +they must not live inside the content-addressed recipe zip. Today they round-trip +through `catalog.yaml` (`capture_tallyman_state` reads live files into keys; +`materialize` writes them back) only because `assert_consistency` forbade +tracking extra files. With xorq gone: +- The mutators already write live files (`set_chart`, `set_display_config`, the + `.py` writers, `set_alias`, the notebook editors). Those files become **tracked + directly**; checkpoint commits them via `git add -A`; `reset_to`'s + `git reset --hard` restores them. +- **`capture`/`materialize` for these six sections is deleted.** `capture` + shrinks to writing `entries.jsonl` + `compute_cache.jsonl`. +- Each mutation is already one checkpoint commit (dispatch-boundary), so the + revision granularity is unchanged — but a chart revision becomes a legible + `git diff` to its file instead of opaque `catalog.yaml` churn. +- `aliases.py`'s `_read`/`_write` switch from `catalog_state` keys to + `aliases.jsonl` (one line per alias: `{"alias","latest","history":[...]}`); + every other alias function rides on `_read`/`_write` untouched, and the only + reader off `read_tallyman_state` is `aliases.py` itself (verified). +- `post_processing/` and `stats/` move from `artifacts/` (outside the repo) + to under `catalog/` so they can be tracked. Layout change, trivially a rebuild. + +Atomic reset is preserved: `git reset --hard ` restores all tracked files +at once regardless of how many — spreading state across files doesn't weaken it. + +### Cut B — catalog-free imports +Replace `import xorq.api as xo` with narrow imports (all verified +symbol-identical and catalog-free): `xorq.ibis_yaml.compiler` (build/load), +`xorq.expr.api` (`deferred_read_*`, `memtable`, `to_sql`), `xorq.caching` +(`ParquetSnapshotCache`). The one non-mechanical swap is **no-arg `xo.connect()`** +(used at `source_cache.py:133,151`), which is bound to `xorq.api.connect` and +absent from `xorq.expr.api`. Add a tallyman datafusion connect shim in +`tallyman_xorq` (`Backend(); do_connect(None)`; returns the identical `Backend`), +with a determinism-equivalence test (the shim-built `ParquetSnapshotCache` must +produce byte-identical content-addressed snapshots, since source-cache keys feed +content-addressing). Sites: `source_cache.py`, `io.py`, `build.py:419`, +`result_cache.py`, `post_processing.py`, `summary_stats.py`, `app.py`, `diff.py`. + +Definition of done is **not** a `sys.modules` assertion (user recipes keep +`import xorq.api`, D4). It is: (a) monkeypatch-`subprocess.run`-to-raise guard +over a full cycle, and (b) a static grep that `src/` has no +`import xorq.catalog` / `from xorq.catalog` / `xorq catalog` argv. + +## Decisions (resolved in the grilling pass) + +- **D1 — zip timing:** checkpoint owns zip creation; build owns only the complete + entry dir. (Collapses the orphan-zip / `--allow-empty`-masking / partial-failure + risks.) +- **D2 — `catalog_registered`:** means "entry dir persisted" (a local fact at + build time); durability is the next checkpoint's job and always succeeds because + the dir is complete. Rewrite the `#48` "not durable" log accordingly. +- **D3 — archive scope:** the recipe allowlist `{expr.py, xorq_build/, + schema.json, manifest.json}` as one immutable zip (fixes the wrong-artifact + durability gap from fact 2). +- **D4 — recipe imports:** keep user recipes on `import xorq.api as xo`; + catalog-free guarantee scoped to `src/`. Real recipes call `xo.connect()`/ + `xo.desc`, so switching the authoring surface needs its own codegen-validation + pass and is a separate follow-up. +- **Decomposition scope:** all six mutable sections (charts, display, + post-processing, stats, aliases, notebook) become tracked files; `catalog.yaml` + reduces to the two JSONL pointer files. One PR, ordered commits. +- **Formats:** JSONL for list-shaped state (aliases, notebook cells, pointer + lists); one pretty-printed JSON object per file for nested blobs (chart specs, + display configs); `.py` for scripts. + +## Module boundary + +- **`tallyman_core/catalog.py` (new — the store):** `write_recipe_zip` / + zip-an-entry-dir, `tracked_recipe_hashes`, `assert_catalog_consistent` (#52), + the `entries.jsonl`/`compute_cache.jsonl` pointer bookkeeping, the bullpen + reconciliation (`prune_entries`/`restore_from_bullpen`), and the `.gitignore`/ + tracked-surface definition. +- **`catalog_state.py` (stays — the versioning engine):** `_project_lock`, + `ensure_catalog_repo`, `checkpoint_catalog` (capture pointers → zip pending via + `catalog.py` → `git add -A` → commit → tag), `reset_to`, `list_revisions`, + `current_step`, genesis. Slims substantially. +- Call direction is one-way (`catalog_state` → `catalog`); `assert_catalog_consistent` + takes the pointer set as an argument to avoid a cycle. Don't rename + `catalog_state.py` (test churn) — slim it in place. +- **Keep `entry_hashes` + the consistency check for now.** Once we own the tracked + zips, the entry set is derivable from `git ls-files entries/*.zip`, making + `entry_hashes` and the two-views check redundant — but that collapse touches the + bullpen reconciliation and is a deliberate separate look. + +## Risk register + +| # | Risk | Sev | Status / mitigation | +|---|---|---|---| +| 1 | Stage build dirs / `result.parquet` into git | High | Retired by `.gitignore` (`entries/*/` etc.) + `git add -A`. Regression test asserts no build-dir file is tracked. | +| 2 | `sys.modules`-based "dropped" assertion is unsatisfiable (recipes load `xorq.api`) | High | Definition of done = subprocess guard + `src/` static grep, not `sys.modules`. | +| 3 | Naive `xorq.api`→`xorq.expr.api` swap breaks no-arg `xo.connect()` | High | Datafusion connect shim + determinism-equivalence test. | +| 4 | `test_catalog_xorq_integration.py` imports `xorq.catalog.catalog.Catalog` | High | Rewrite in the fix commits: drop `_open_xorq_catalog`/alias-symlink tests; keep the `git ls-files` durability guards. | +| 5 | Salt-mode zip must be named by the salted hash | High | Name `entries/.zip`; test under `TALLYMAN_SOURCE_IDENTITY=salt`. | +| 6 | Commit timing inverts (inline → checkpoint); tests that build with no checkpoint and assert tracked fail | Med | Canonical failing-first test; update `test_xorq_catalog_add_registers_after_genesis` to checkpoint first. | +| 7 | Orphan loose zip / `--allow-empty` masks a pointer-without-recipe | Med | Retired by D1 (checkpoint is the sole zip writer over complete dirs). | +| 8 | In-process git reintroduces the macOS fork-from-threads SIGSEGV | Med | All native catalog git via `git_util.run_git` (posix_spawn); never GitPython. `test_no_bare_subprocess_git_in_src` enforces. | +| 9 | `test_capture_seeds_xorq_keys` / decomposed-section capture tests pin dropped behavior | Low | Delete/rewrite in the fix commits alongside the decomposition. | +| 10 | Existing corpus is #48-inconsistent; two zip formats coexist | Low | Rebuild from scratch (project rule). | + +## Implementation sequence (ordered commits, TDD) + +All new failing tests bundle into one commit, seen red on CI, before the fixes. + +1. **Failing-first commit.** `test_*_uses_no_subprocess` (build/alias paths, + monkeypatch `subprocess.run`), `test_zip_staged_not_committed_until_checkpoint`, + `test_n_builds_track_n_zips` (#48 blind spot), `test_salt_zip_named_by_salted_hash`, + `test_build_dir_files_not_staged`, `test_failed_build_no_orphan_pointer`. Confirm + `run_git` is posix_spawn (it is) before relying on the monkeypatch. +2. **Native recipe store** — `tallyman_core/catalog.py` `write_recipe_zip`; + `build.py` persists the complete dir + cleans up on failure; delete + `xorq_catalog.py`. +3. **Checkpoint owns zipping** — `.gitignore`; `checkpoint_catalog` zips pending + + `git add -A`; `prune_entries`/consistency move into `catalog.py`. +4. **Decompose `catalog.yaml`** — aliases → `aliases.jsonl`; charts/display/pp/stats + tracked directly; notebook → `notebook.jsonl`; pointers → `entries.jsonl` + + `compute_cache.jsonl`; delete `capture`/`materialize` for the six sections; move + `post_processing/`+`stats/` under the repo. +5. **Cut B** — narrow catalog-free imports + connect shim across the 8 modules. +6. **Test-cleanup commit** — rewrite/delete the xorq-coupled integration tests; + re-target `test_failed_registration_is_observable` to a native failure; drop the + xorq rationale from `test_catalog_repo_branch_is_main`. +7. **Rebuild + verify** — rebuild the corpus; `git ls-files` shows only the tracked + tree above; genesis→create×2→reset roundtrip passes; full suite green locally + before push. + +## Scope boundaries / deferred + +- No `pyproject.toml` change (fact 5). No merge/sync surface (fact 4). No xorq + `assert_consistency`/`BuildZip`/`test_zip` contract in the native store (fact 1). +- No clone-restore / reconstruct-from-zip implementation; D3 keeps the zip *able* + to support it (whole recipe + `/` prefix). +- No on-disk migration; rebuild the corpus. +- Recipe authoring-surface switch (D4) is a separate, codegen-validated follow-up. +- Collapsing `entry_hashes` into `git ls-files` is a separate look. + +## Open / not yet grilled + +- **Rebuild operational path.** What you actually run to rebuild a corpus + (re-execute stored `expr.py` via `build_and_persist` in a loop vs re-drive the + MCP/codegen session) and whether a `tallyman rebuild` affordance is warranted — + `first-project` is ~119MB, `parking_ticket_analysis_1m` ~10 entries, so a + from-scratch rebuild is minutes-to-tens-of-minutes per large project. Needs a CLI + check before settling. +- Genesis baseline behavior post-decomposition; exact `.gitignore`; final + failing-first test list once the above is settled. From 1f5034b3ba4bcd4c4582a0de49e83d4b7307d942 Mon Sep 17 00:00:00 2001 From: Paddy Mullen Date: Tue, 16 Jun 2026 19:16:13 -0400 Subject: [PATCH 02/27] docs: refine native-catalog-store plan from review pass Addresses four findings from a code-investigation review of the plan: - prompts.jsonl: it is append-mutable (grows on a re-run with no rebuild) AND runtime-read (entry-detail API -> React "N prompts seen"), yet the plan omitted it entirely, so it would be lost on reset/restore (entry dirs are gitignored; manifest keeps only the first prompt). Make it a seventh tracked decomposition file prompts/.jsonl, not a zip member. Wired through tracked tree, recipe-store exclusions, the decomposition section, D3, decomposition-scope, and steps 1/4. - Cut B: state "8 runtime imports" and note the 9th grep hit (server.py:219) is the codegen-guidance docstring, retained per D4. - Staging: git add -A is allow-by-default; assert_catalog_consistent (#52) now enforces the tracked surface as a path allowlist (deny-by-default). Risk row 1, module boundary, and steps 1/3 updated. - Deterministic archive: pin ZIP_DEFLATED compresslevel. Co-Authored-By: Claude Opus 4.8 (1M context) --- plans/native-catalog-store.md | 85 ++++++++++++++++++++++++++--------- 1 file changed, 65 insertions(+), 20 deletions(-) diff --git a/plans/native-catalog-store.md b/plans/native-catalog-store.md index fa841c3..dae4668 100644 --- a/plans/native-catalog-store.md +++ b/plans/native-catalog-store.md @@ -122,6 +122,7 @@ entries.jsonl # pointer list: untracked build dirs (bullpen reco compute_cache.jsonl # pointer list: untracked compute-cache warm set chart_specs/.vl.json # one Vega-Lite spec per entry display_configs/.json # one Buckaroo/display config per entry +prompts/.jsonl # append-only authoring-prompt history per entry post_processing/.py # project-global functions (moved under the repo) stats/.py # project-global functions (moved under the repo) ``` @@ -129,19 +130,32 @@ stats/.py # project-global functions (moved under the repo) No YAML anywhere. The former `catalog.yaml` is gone; its residual pointer bookkeeping is `entries.jsonl` + `compute_cache.jsonl`. +Because `git add -A` is allow-by-default, the `.gitignore` denylist alone does +not keep the tracked tree honest — `assert_catalog_consistent`'s path allowlist +(Module boundary, #52) does. (The `metadata/` `*.zip.metadata.yaml` sidecars and +the `aliases/` symlink tree that `git add -A` would otherwise stage are xorq +`catalog add` subprocess output; they stop being written once Cut A removes the +subprocess, and a from-scratch rebuild never recreates them.) + ### The recipe store (immutable) - `entries/.zip` archives an **allowlist** of the entry dir: `{expr.py, xorq_build/, schema.json, manifest.json}`. Allowlist, not "everything except," so a future derived artifact dropped in `entry_dir` can't - leak into the durable zip. Excludes `.xorq_build_expanded/` (fact 6) and - `result.parquet` (the untracked heavy result). + leak into the durable zip. Excludes `.xorq_build_expanded/` (fact 6), + `result.parquet` (the untracked heavy result), and `prompts.jsonl` — the last + is append-mutable (it grows on a re-run of an existing hash with no rebuild, + `build.py:335`), so a content-addressed zip member would churn the blob in git + history on every re-run; it becomes a tracked decomposition file instead (see + below). - Members stored under arcname `/` (single top-level prefix, so a future restore can reuse it). **Named by the salted `content_hash`** when source-identity salt mode is on, not `build_path.name` — otherwise `_tracked_recipe_hashes` (zip names) diverges from `entries.jsonl` pointers. - **Deterministic archive** (`date_time=(1980,1,1,0,0,0)`, - `external_attr=0o644<<16`, sorted iteration, `ZIP_DEFLATED`) so the blob - doesn't churn on identical rebuilds. + `external_attr=0o644<<16`, sorted iteration, `ZIP_DEFLATED` at a pinned + `compresslevel`) so the blob doesn't churn on identical rebuilds. (Single + machine, so cross-zlib reproducibility is moot; pinning the level keeps the + blob byte-stable across same-machine rebuilds regardless of zlib defaults.) - **Drop the wheel + requirements.txt** xorq injected (~178KB → ~6KB per entry); nothing reads them, and the payoff is git *history* (every checkpoint re-stages the zip). @@ -155,9 +169,10 @@ bookkeeping is `entries.jsonl` + `compute_cache.jsonl`. leaves a complete gitignored dir the next checkpoint picks up). ### The decomposition (mutable state → tracked files) -The recipe is immutable; charts, styling, post-processing, stats, aliases, and -the notebook are **mutable** — edited on an existing entry without rebuilding, so -they must not live inside the content-addressed recipe zip. Today they round-trip +The recipe is immutable; charts, styling, post-processing, stats, aliases, the +notebook, and the per-entry authoring-prompt history are **mutable** — edited +(or, for prompts, appended to) on an existing entry without rebuilding, so they +must not live inside the content-addressed recipe zip. Today they round-trip through `catalog.yaml` (`capture_tallyman_state` reads live files into keys; `materialize` writes them back) only because `assert_consistency` forbade tracking extra files. With xorq gone: @@ -176,6 +191,18 @@ tracking extra files. With xorq gone: reader off `read_tallyman_state` is `aliases.py` itself (verified). - `post_processing/` and `stats/` move from `artifacts/` (outside the repo) to under `catalog/` so they can be tracked. Layout change, trivially a rebuild. +- **Prompt history is a seventh tracked file, with a different origin.** Unlike + the six above it never round-tripped through `catalog.yaml` — it has always + been a loose `entry_dir/prompts.jsonl` (written by `build.py:35`; read back by + the entry-detail API at `app.py:580` → the React "N prompts seen for this hash" + disclosure). Because `entries/*/` is now gitignored, that loose file would be + lost on `reset_to`/clone-restore, and `manifest.json` preserves only the + *first* build's prompt — so the re-run history the disclosure exists to show + would be gone. Relocate the `build.py` writer and the `read_prompts` reader + from `entry_dir` to a tracked root-level `prompts/.jsonl` (mirroring + `chart_specs/`). Layout change, trivially a rebuild. `capture`/`materialize` + do not touch it — it was never in `catalog.yaml`, so the six-section deletion + above is unaffected. Atomic reset is preserved: `git reset --hard ` restores all tracked files at once regardless of how many — spreading state across files doesn't weaken it. @@ -190,8 +217,12 @@ absent from `xorq.expr.api`. Add a tallyman datafusion connect shim in `tallyman_xorq` (`Backend(); do_connect(None)`; returns the identical `Backend`), with a determinism-equivalence test (the shim-built `ParquetSnapshotCache` must produce byte-identical content-addressed snapshots, since source-cache keys feed -content-addressing). Sites: `source_cache.py`, `io.py`, `build.py:419`, -`result_cache.py`, `post_processing.py`, `summary_stats.py`, `app.py`, `diff.py`. +content-addressing). Sites (8 runtime imports): `source_cache.py`, `io.py`, +`build.py:419`, `result_cache.py`, `post_processing.py`, `summary_stats.py`, +`app.py`, `diff.py`. The 9th grep hit, `tallyman_mcp/server.py:219`, is the +codegen-guidance docstring teaching recipe authors the `import xorq.api as xo` +namespace — it is *not* an executed import; left as-is per D4 (the recipe +authoring surface is a separate follow-up). Definition of done is **not** a `sys.modules` assertion (user recipes keep `import xorq.api`, D4). It is: (a) monkeypatch-`subprocess.run`-to-raise guard @@ -208,13 +239,17 @@ over a full cycle, and (b) a static grep that `src/` has no the dir is complete. Rewrite the `#48` "not durable" log accordingly. - **D3 — archive scope:** the recipe allowlist `{expr.py, xorq_build/, schema.json, manifest.json}` as one immutable zip (fixes the wrong-artifact - durability gap from fact 2). + durability gap from fact 2). `prompts.jsonl` is deliberately *not* a member: it + is append-mutable, runtime-read provenance, so it joins the decomposition as a + tracked `prompts/.jsonl` rather than churning the content-addressed blob. - **D4 — recipe imports:** keep user recipes on `import xorq.api as xo`; catalog-free guarantee scoped to `src/`. Real recipes call `xo.connect()`/ `xo.desc`, so switching the authoring surface needs its own codegen-validation pass and is a separate follow-up. -- **Decomposition scope:** all six mutable sections (charts, display, - post-processing, stats, aliases, notebook) become tracked files; `catalog.yaml` +- **Decomposition scope:** all six `catalog.yaml`-smuggled mutable sections + (charts, display, post-processing, stats, aliases, notebook) become tracked + files, plus a seventh — the per-entry prompt history, relocated from the + now-gitignored entry dir to a tracked `prompts/.jsonl`; `catalog.yaml` reduces to the two JSONL pointer files. One PR, ordered commits. - **Formats:** JSONL for list-shaped state (aliases, notebook cells, pointer lists); one pretty-printed JSON object per file for nested blobs (chart specs, @@ -223,10 +258,13 @@ over a full cycle, and (b) a static grep that `src/` has no ## Module boundary - **`tallyman_core/catalog.py` (new — the store):** `write_recipe_zip` / - zip-an-entry-dir, `tracked_recipe_hashes`, `assert_catalog_consistent` (#52), - the `entries.jsonl`/`compute_cache.jsonl` pointer bookkeeping, the bullpen - reconciliation (`prune_entries`/`restore_from_bullpen`), and the `.gitignore`/ - tracked-surface definition. + zip-an-entry-dir, `tracked_recipe_hashes`, `assert_catalog_consistent` (#52 — + enforces the tracked surface as a path **allowlist** (deny-by-default): every + `git ls-files` path must fnmatch the tracked-surface set or it raises, so a + stray dir that `git add -A` would otherwise stage is rejected instead of + committed silently), the `entries.jsonl`/`compute_cache.jsonl` pointer + bookkeeping, the bullpen reconciliation (`prune_entries`/`restore_from_bullpen`), + and the `.gitignore`/tracked-surface definition. - **`catalog_state.py` (stays — the versioning engine):** `_project_lock`, `ensure_catalog_repo`, `checkpoint_catalog` (capture pointers → zip pending via `catalog.py` → `git add -A` → commit → tag), `reset_to`, `list_revisions`, @@ -243,7 +281,7 @@ over a full cycle, and (b) a static grep that `src/` has no | # | Risk | Sev | Status / mitigation | |---|---|---|---| -| 1 | Stage build dirs / `result.parquet` into git | High | Retired by `.gitignore` (`entries/*/` etc.) + `git add -A`. Regression test asserts no build-dir file is tracked. | +| 1 | `git add -A` is allow-by-default; a `.gitignore` denylist silently tracks any unlisted artifact (build dirs, `result.parquet`, future strays) | High | `.gitignore` (`entries/*/` etc.) covers today's heavy artifacts, but the durable guard is `assert_catalog_consistent` enforcing the tracked surface as a path **allowlist** (deny-by-default). Regression test asserts no build-dir file — and no unlisted path — is tracked. | | 2 | `sys.modules`-based "dropped" assertion is unsatisfiable (recipes load `xorq.api`) | High | Definition of done = subprocess guard + `src/` static grep, not `sys.modules`. | | 3 | Naive `xorq.api`→`xorq.expr.api` swap breaks no-arg `xo.connect()` | High | Datafusion connect shim + determinism-equivalence test. | | 4 | `test_catalog_xorq_integration.py` imports `xorq.catalog.catalog.Catalog` | High | Rewrite in the fix commits: drop `_open_xorq_catalog`/alias-symlink tests; keep the `git ls-files` durability guards. | @@ -261,16 +299,23 @@ All new failing tests bundle into one commit, seen red on CI, before the fixes. 1. **Failing-first commit.** `test_*_uses_no_subprocess` (build/alias paths, monkeypatch `subprocess.run`), `test_zip_staged_not_committed_until_checkpoint`, `test_n_builds_track_n_zips` (#48 blind spot), `test_salt_zip_named_by_salted_hash`, - `test_build_dir_files_not_staged`, `test_failed_build_no_orphan_pointer`. Confirm + `test_build_dir_files_not_staged`, `test_failed_build_no_orphan_pointer`, + `test_prompt_history_survives_reset` (re-run appends a prompt, checkpoint, + `reset_to`, assert all prompts present — fails today because the loose + `entry_dir/prompts.jsonl` is gitignored), `test_unlisted_path_not_tracked` + (stage a stray file, assert `assert_catalog_consistent` raises). Confirm `run_git` is posix_spawn (it is) before relying on the monkeypatch. 2. **Native recipe store** — `tallyman_core/catalog.py` `write_recipe_zip`; `build.py` persists the complete dir + cleans up on failure; delete `xorq_catalog.py`. 3. **Checkpoint owns zipping** — `.gitignore`; `checkpoint_catalog` zips pending + - `git add -A`; `prune_entries`/consistency move into `catalog.py`. + `git add -A`; `prune_entries`/consistency move into `catalog.py`, with + `assert_catalog_consistent` extended to enforce the tracked-path allowlist. 4. **Decompose `catalog.yaml`** — aliases → `aliases.jsonl`; charts/display/pp/stats tracked directly; notebook → `notebook.jsonl`; pointers → `entries.jsonl` + - `compute_cache.jsonl`; delete `capture`/`materialize` for the six sections; move + `compute_cache.jsonl`; delete `capture`/`materialize` for the six sections; + relocate the `prompts.jsonl` writer/reader (`build.py`, `read_prompts`, + `app.py`) from `entry_dir` to tracked `prompts/.jsonl`; move `post_processing/`+`stats/` under the repo. 5. **Cut B** — narrow catalog-free imports + connect shim across the 8 modules. 6. **Test-cleanup commit** — rewrite/delete the xorq-coupled integration tests; From 08efa47cbcf04f5d9b30599f44cf2e0886814bca Mon Sep 17 00:00:00 2001 From: Paddy Mullen Date: Tue, 16 Jun 2026 19:32:56 -0400 Subject: [PATCH 03/27] docs: fix connect/notebook/aliases/line-num/zip-cost findings in native-catalog plan MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit Folds five review findings into plans/native-catalog-store.md: - F1: xorq.expr.api exports a connect, but it is ibis's connect(resource, **kwargs), not the no-arg datafusion xorq.api.connect — a naive swap fails at call time, not import. Correct Cut B prose + Risk #3. - F2: relocate the notebook (project_dir/notebooks/) under catalog/ alongside post_processing/ and stats/; note chart_specs/ and display_configs/ already live under catalog_dir (no move). - F3: strip aliases.py's module-level import xorq_catalog + the _xcat.add_alias/remove_alias calls in set_alias/remove_alias in the same commit that deletes xorq_catalog.py, else the package is un-importable. - F5: xorq api.py catalog import is at line 62 (was cited :65). ~7.7k LOC is correct (non-test). - F6: deterministic content-addressed zip is stored once by git; dropping the embedded wheel saves blob + working-tree size, not per-checkpoint churn. F4 (catalog.yaml r/w + reset_to.materialize teardown) tracked as #105. Co-Authored-By: Claude Opus 4.8 (1M context) --- plans/native-catalog-store.md | 46 ++++++++++++++++++++++++++--------- 1 file changed, 34 insertions(+), 12 deletions(-) diff --git a/plans/native-catalog-store.md b/plans/native-catalog-store.md index dae4668..a3fe964 100644 --- a/plans/native-catalog-store.md +++ b/plans/native-catalog-store.md @@ -40,7 +40,7 @@ of mutable catalog state becomes its own git-tracked file. The #48 unlock ### Cut B — the RUNTIME cut `import xorq.api` transitively loads the entire `xorq.catalog` package -(`xorq/api.py:65 import xorq.catalog.api as catalog`; verified: bare +(`xorq/api.py:62 import xorq.catalog.api as catalog`; verified: bare `import xorq` loads zero catalog modules, `import xorq.api` loads 12). Replace tallyman's ~9 `import xorq.api as xo` sites with narrow catalog-free imports plus a datafusion `connect()` shim. @@ -157,8 +157,10 @@ subprocess, and a from-scratch rebuild never recreates them.) machine, so cross-zlib reproducibility is moot; pinning the level keeps the blob byte-stable across same-machine rebuilds regardless of zlib defaults.) - **Drop the wheel + requirements.txt** xorq injected (~178KB → ~6KB per entry); - nothing reads them, and the payoff is git *history* (every checkpoint re-stages - the zip). + nothing reads them. Because the zip is deterministic and content-addressed, git + stores each blob once and a `git add -A` over an unchanged zip is a no-op that + adds no history — so the saving is a smaller durable blob and working tree per + entry, not avoided per-checkpoint churn (there is none once the zip is byte-stable). - **The checkpoint owns zip creation, not the build.** The build persists a complete `entries//` on success and cleans up the dir if this call created it and the build failed (no partial dirs, no orphan zips). The @@ -186,11 +188,22 @@ tracking extra files. With xorq gone: revision granularity is unchanged — but a chart revision becomes a legible `git diff` to its file instead of opaque `catalog.yaml` churn. - `aliases.py`'s `_read`/`_write` switch from `catalog_state` keys to - `aliases.jsonl` (one line per alias: `{"alias","latest","history":[...]}`); - every other alias function rides on `_read`/`_write` untouched, and the only - reader off `read_tallyman_state` is `aliases.py` itself (verified). -- `post_processing/` and `stats/` move from `artifacts/` (outside the repo) - to under `catalog/` so they can be tracked. Layout change, trivially a rebuild. + `aliases.jsonl` (one line per alias: `{"alias","latest","history":[...]}`); the + read-only alias functions ride on `_read`/`_write` untouched, and the only reader + off `read_tallyman_state` is `aliases.py` itself (verified). But `aliases.py` also + has a *module-level* `import xorq_catalog as _xcat` (`aliases.py:24`) and + `set_alias`/`remove_alias` still call `_xcat.add_alias`/`remove_alias` + (`:129`,`:155`) — the xorq subprocess. Both must be stripped in the *same* commit + that deletes `xorq_catalog.py`: `aliases.py` is its only remaining caller once + `build.py`'s recipe path is rewritten, and because the import is module-level, + deleting `xorq_catalog.py` without it makes `aliases.py` — and the whole package — + un-importable. +- `post_processing/`, `stats/`, and the notebook move from outside the repo + (`artifacts/post_processing`, `artifacts/stats`, and `project_dir/notebooks/` + respectively) to under `catalog/` so they can be tracked. `chart_specs/` and + `display_configs/` already live under `catalog/` (`charts.py`/`display_configs.py` + write to `catalog_dir`), so they need no move — only `post_processing/`, `stats/`, + and the notebook do. Layout change, trivially a rebuild. - **Prompt history is a seventh tracked file, with a different origin.** Unlike the six above it never round-tripped through `catalog.yaml` — it has always been a loose `entry_dir/prompts.jsonl` (written by `build.py:35`; read back by @@ -212,8 +225,13 @@ Replace `import xorq.api as xo` with narrow imports (all verified symbol-identical and catalog-free): `xorq.ibis_yaml.compiler` (build/load), `xorq.expr.api` (`deferred_read_*`, `memtable`, `to_sql`), `xorq.caching` (`ParquetSnapshotCache`). The one non-mechanical swap is **no-arg `xo.connect()`** -(used at `source_cache.py:133,151`), which is bound to `xorq.api.connect` and -absent from `xorq.expr.api`. Add a tallyman datafusion connect shim in +(used at `source_cache.py:133,151`). `xorq.expr.api` is *not* a drop-in here: it +does export a `connect`, but it is ibis's `connect(resource, **kwargs)` +(re-exported via `from xorq.vendor.ibis.expr.api import *`), a different function +requiring a positional `resource` — not the no-arg datafusion `xorq.api.connect` +(verified: `xorq.expr.api.connect is xorq.api.connect` → `False`). So a naive swap +resolves the symbol (no `ImportError`) and then fails at *call* time on the missing +arg, or silently builds the wrong backend. Add a tallyman datafusion connect shim in `tallyman_xorq` (`Backend(); do_connect(None)`; returns the identical `Backend`), with a determinism-equivalence test (the shim-built `ParquetSnapshotCache` must produce byte-identical content-addressed snapshots, since source-cache keys feed @@ -283,7 +301,7 @@ over a full cycle, and (b) a static grep that `src/` has no |---|---|---|---| | 1 | `git add -A` is allow-by-default; a `.gitignore` denylist silently tracks any unlisted artifact (build dirs, `result.parquet`, future strays) | High | `.gitignore` (`entries/*/` etc.) covers today's heavy artifacts, but the durable guard is `assert_catalog_consistent` enforcing the tracked surface as a path **allowlist** (deny-by-default). Regression test asserts no build-dir file — and no unlisted path — is tracked. | | 2 | `sys.modules`-based "dropped" assertion is unsatisfiable (recipes load `xorq.api`) | High | Definition of done = subprocess guard + `src/` static grep, not `sys.modules`. | -| 3 | Naive `xorq.api`→`xorq.expr.api` swap breaks no-arg `xo.connect()` | High | Datafusion connect shim + determinism-equivalence test. | +| 3 | `xo.connect()` survives the `xorq.api`→`xorq.expr.api` swap as ibis's `connect(resource, …)` (present, not absent), so it fails at *call* time / builds the wrong backend rather than at import — easy to miss | High | Datafusion connect shim + determinism-equivalence test. | | 4 | `test_catalog_xorq_integration.py` imports `xorq.catalog.catalog.Catalog` | High | Rewrite in the fix commits: drop `_open_xorq_catalog`/alias-symlink tests; keep the `git ls-files` durability guards. | | 5 | Salt-mode zip must be named by the salted hash | High | Name `entries/.zip`; test under `TALLYMAN_SOURCE_IDENTITY=salt`. | | 6 | Commit timing inverts (inline → checkpoint); tests that build with no checkpoint and assert tracked fail | Med | Canonical failing-first test; update `test_xorq_catalog_add_registers_after_genesis` to checkpoint first. | @@ -307,7 +325,11 @@ All new failing tests bundle into one commit, seen red on CI, before the fixes. `run_git` is posix_spawn (it is) before relying on the monkeypatch. 2. **Native recipe store** — `tallyman_core/catalog.py` `write_recipe_zip`; `build.py` persists the complete dir + cleans up on failure; delete - `xorq_catalog.py`. + `xorq_catalog.py` *and* strip its only other caller's coupling in the same + commit — `aliases.py`'s module-level `import xorq_catalog` plus the + `_xcat.add_alias`/`remove_alias` calls in `set_alias`/`remove_alias` — so the + package stays importable (the failing-first suite from commit 1 can't run red + against an un-importable tree). 3. **Checkpoint owns zipping** — `.gitignore`; `checkpoint_catalog` zips pending + `git add -A`; `prune_entries`/consistency move into `catalog.py`, with `assert_catalog_consistent` extended to enforce the tracked-path allowlist. From 83dcf72a1209ad9645f35d04c78b8cf9cc0a7650 Mon Sep 17 00:00:00 2001 From: Paddy Mullen Date: Tue, 16 Jun 2026 19:37:44 -0400 Subject: [PATCH 04/27] docs: propagate notebook relocation + fix citations (verification pass) MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit Adversarial verification of the prior fixes confirmed F1/F3/F5/F6 clean but the consistency critic found the F2 notebook relocation was not propagated from the Decomposition prose to the two places that enumerate it concretely: - Implementation-sequence commit 4 said "move post_processing/+stats/ under the repo" — now also relocates the notebook (project_dir/notebooks/). - Tracked-tree block annotated post_processing/stats as "(moved under the repo)" but not notebook.jsonl — now annotated. Also fixes two cosmetic citations the verifiers caught: the 2nd xo.connect() call is source_cache.py:152 (not :151), and the aliases.py import is spelled `from tallyman_core import xorq_catalog as _xcat` (not a bare import). Co-Authored-By: Claude Opus 4.8 (1M context) --- plans/native-catalog-store.md | 14 ++++++++------ 1 file changed, 8 insertions(+), 6 deletions(-) diff --git a/plans/native-catalog-store.md b/plans/native-catalog-store.md index a3fe964..df648a7 100644 --- a/plans/native-catalog-store.md +++ b/plans/native-catalog-store.md @@ -117,7 +117,7 @@ Then `git add -A` stages exactly the meaningful state, and the tracked tree is: ``` entries/.zip # immutable recipe, one blob per entry aliases.jsonl # one line per alias -notebook.jsonl # one line per cell +notebook.jsonl # one line per cell (moved under the repo) entries.jsonl # pointer list: untracked build dirs (bullpen reconcile) compute_cache.jsonl # pointer list: untracked compute-cache warm set chart_specs/.vl.json # one Vega-Lite spec per entry @@ -191,7 +191,8 @@ tracking extra files. With xorq gone: `aliases.jsonl` (one line per alias: `{"alias","latest","history":[...]}`); the read-only alias functions ride on `_read`/`_write` untouched, and the only reader off `read_tallyman_state` is `aliases.py` itself (verified). But `aliases.py` also - has a *module-level* `import xorq_catalog as _xcat` (`aliases.py:24`) and + has a *module-level* `from tallyman_core import xorq_catalog as _xcat` + (`aliases.py:24`) and `set_alias`/`remove_alias` still call `_xcat.add_alias`/`remove_alias` (`:129`,`:155`) — the xorq subprocess. Both must be stripped in the *same* commit that deletes `xorq_catalog.py`: `aliases.py` is its only remaining caller once @@ -225,7 +226,7 @@ Replace `import xorq.api as xo` with narrow imports (all verified symbol-identical and catalog-free): `xorq.ibis_yaml.compiler` (build/load), `xorq.expr.api` (`deferred_read_*`, `memtable`, `to_sql`), `xorq.caching` (`ParquetSnapshotCache`). The one non-mechanical swap is **no-arg `xo.connect()`** -(used at `source_cache.py:133,151`). `xorq.expr.api` is *not* a drop-in here: it +(used at `source_cache.py:133,152`). `xorq.expr.api` is *not* a drop-in here: it does export a `connect`, but it is ibis's `connect(resource, **kwargs)` (re-exported via `from xorq.vendor.ibis.expr.api import *`), a different function requiring a positional `resource` — not the no-arg datafusion `xorq.api.connect` @@ -326,7 +327,7 @@ All new failing tests bundle into one commit, seen red on CI, before the fixes. 2. **Native recipe store** — `tallyman_core/catalog.py` `write_recipe_zip`; `build.py` persists the complete dir + cleans up on failure; delete `xorq_catalog.py` *and* strip its only other caller's coupling in the same - commit — `aliases.py`'s module-level `import xorq_catalog` plus the + commit — `aliases.py`'s module-level `from tallyman_core import xorq_catalog` plus the `_xcat.add_alias`/`remove_alias` calls in `set_alias`/`remove_alias` — so the package stays importable (the failing-first suite from commit 1 can't run red against an un-importable tree). @@ -337,8 +338,9 @@ All new failing tests bundle into one commit, seen red on CI, before the fixes. tracked directly; notebook → `notebook.jsonl`; pointers → `entries.jsonl` + `compute_cache.jsonl`; delete `capture`/`materialize` for the six sections; relocate the `prompts.jsonl` writer/reader (`build.py`, `read_prompts`, - `app.py`) from `entry_dir` to tracked `prompts/.jsonl`; move - `post_processing/`+`stats/` under the repo. + `app.py`) from `entry_dir` to tracked `prompts/.jsonl`; relocate + `post_processing/`, `stats/`, and the notebook (`project_dir/notebooks/`) under + the repo. 5. **Cut B** — narrow catalog-free imports + connect shim across the 8 modules. 6. **Test-cleanup commit** — rewrite/delete the xorq-coupled integration tests; re-target `test_failed_registration_is_observable` to a native failure; drop the From 312612605e3884e4592618a7ae276a74ef387e63 Mon Sep 17 00:00:00 2001 From: Paddy Mullen Date: Tue, 16 Jun 2026 19:44:03 -0400 Subject: [PATCH 05/27] docs: be honest about what dropping the entry wheel forfeits MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit The wheel + requirements.txt are xorq's per-expression reproducibility mechanism (WheelPackager/PackagedRunner in ibis_yaml/packager.py: bundle a built wheel + pinned requirements so `xorq run` re-executes an entry in its own isolated uv env). The prior "nothing reads them" rationale undersold the tradeoff. tallyman doesn't use that path: it pins the env once (xorq==0.3.26 + uv.lock) and runs every expression in one shared .venv via in-process build_expr (build.py:317), never a per-expression uv env (the slow/expensive thing it avoids). The in-process build output never held the wheel anyway — only the catalog zip did, and nothing reads it back (fact 1). So the drop forfeits xorq's per-entry env reconstruction (unused here), not tallyman's reproducibility, which rides on the pinned project env. Co-Authored-By: Claude Opus 4.8 (1M context) --- plans/native-catalog-store.md | 23 ++++++++++++++++++----- 1 file changed, 18 insertions(+), 5 deletions(-) diff --git a/plans/native-catalog-store.md b/plans/native-catalog-store.md index df648a7..ffbcff9 100644 --- a/plans/native-catalog-store.md +++ b/plans/native-catalog-store.md @@ -156,11 +156,24 @@ subprocess, and a from-scratch rebuild never recreates them.) `compresslevel`) so the blob doesn't churn on identical rebuilds. (Single machine, so cross-zlib reproducibility is moot; pinning the level keeps the blob byte-stable across same-machine rebuilds regardless of zlib defaults.) -- **Drop the wheel + requirements.txt** xorq injected (~178KB → ~6KB per entry); - nothing reads them. Because the zip is deterministic and content-addressed, git - stores each blob once and a `git add -A` over an unchanged zip is a no-op that - adds no history — so the saving is a smaller durable blob and working tree per - entry, not avoided per-checkpoint churn (there is none once the zip is byte-stable). +- **Drop the wheel + requirements.txt** the `xorq catalog add` subprocess injects + (~178KB → ~6KB per entry). These are xorq's *per-expression* reproducibility + mechanism — `WheelPackager`/`PackagedBuilder`/`PackagedRunner` + (`xorq/ibis_yaml/packager.py`) bundle a built wheel + pinned `requirements.txt` so + `xorq run` can re-execute an entry in its *own* isolated `uv` env. tallyman does + not use that path: it pins the whole environment once (`xorq==0.3.26` + `uv.lock`) + and runs every expression in one shared `.venv` via in-process + `build_expr`/`load_expr` (`build.py:317`), never a per-expression `uv` env — that + per-entry env is the slow, expensive thing tallyman deliberately avoids. The + in-process build output (`xorq_build/`) never held the wheel anyway (the + `build_expr` file set is `expr.yaml`/SQL/`deferred_reads.yaml`/`profiles.yaml`); only + the catalog zip did, and nothing reads it back (fact 1). So the drop forfeits + xorq's per-entry env reconstruction — **unused here** — not tallyman's + reproducibility, which rides on the pinned project env. Separately: because the zip + is deterministic and content-addressed, git stores each blob once and a + `git add -A` over an unchanged zip is a no-op that adds no history — so the saving + is a smaller durable blob and working tree per entry, not avoided per-checkpoint + churn (there is none once the zip is byte-stable). - **The checkpoint owns zip creation, not the build.** The build persists a complete `entries//` on success and cleans up the dir if this call created it and the build failed (no partial dirs, no orphan zips). The From f24e761482e01353ccf6a3bc2d22767d50a9dc17 Mon Sep 17 00:00:00 2001 From: Paddy Mullen Date: Tue, 16 Jun 2026 20:00:58 -0400 Subject: [PATCH 06/27] docs: per-catalog-repo reproducibility via a tracked pyproject.toml + uv.lock MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit Decision: capture reproducibility at catalog-repo granularity, not per expression. Track a single pyproject.toml + uv.lock in the catalog repo (the reproducibility unit); rebuild = recreate one venv per project from the lock and re-run the recipes. Per-expression venvs are deferred (the slow/expensive xorq model we don't want yet). - Tracked tree: add pyproject.toml + uv.lock. - Wheel-drop bullet: reproducibility now rides on the tracked per-repo lock, not the ambient tallyman dev env. - Scope boundary: disambiguate this from "no change to tallyman's own pyproject" (fact 5) — they are different files. - Open/rebuild: env half settled (recreate venv from tracked lock); remaining open is orchestration + when the lock is snapshotted/refreshed. Also records (chat finding, not yet in doc): xorq's decompiler is ibis's generic one and is lossy for deferred reads (drops the path, emits a non-compiling hyphenated identifier), so reconstructing recipes by decompiling xorq_build/ is not viable — keep persisting expr.py. Co-Authored-By: Claude Opus 4.8 (1M context) --- plans/native-catalog-store.md | 42 ++++++++++++++++++++++------------- 1 file changed, 26 insertions(+), 16 deletions(-) diff --git a/plans/native-catalog-store.md b/plans/native-catalog-store.md index ffbcff9..60c4575 100644 --- a/plans/native-catalog-store.md +++ b/plans/native-catalog-store.md @@ -125,6 +125,8 @@ display_configs/.json # one Buckaroo/display config per entry prompts/.jsonl # append-only authoring-prompt history per entry post_processing/.py # project-global functions (moved under the repo) stats/.py # project-global functions (moved under the repo) +pyproject.toml # per-repo env spec — the reproducibility unit +uv.lock # pinned lock; recreate one venv per repo to rebuild ``` No YAML anywhere. The former `catalog.yaml` is gone; its residual pointer @@ -161,15 +163,17 @@ subprocess, and a from-scratch rebuild never recreates them.) mechanism — `WheelPackager`/`PackagedBuilder`/`PackagedRunner` (`xorq/ibis_yaml/packager.py`) bundle a built wheel + pinned `requirements.txt` so `xorq run` can re-execute an entry in its *own* isolated `uv` env. tallyman does - not use that path: it pins the whole environment once (`xorq==0.3.26` + `uv.lock`) - and runs every expression in one shared `.venv` via in-process - `build_expr`/`load_expr` (`build.py:317`), never a per-expression `uv` env — that - per-entry env is the slow, expensive thing tallyman deliberately avoids. The - in-process build output (`xorq_build/`) never held the wheel anyway (the - `build_expr` file set is `expr.yaml`/SQL/`deferred_reads.yaml`/`profiles.yaml`); only - the catalog zip did, and nothing reads it back (fact 1). So the drop forfeits - xorq's per-entry env reconstruction — **unused here** — not tallyman's - reproducibility, which rides on the pinned project env. Separately: because the zip + not use that path: it builds every expression in-process via + `build_expr`/`load_expr` (`build.py:317`), and captures reproducibility at + **catalog-repo granularity** — a single tracked `pyproject.toml` + `uv.lock` (see + Tracked tree), one venv per project, recreated to rebuild the corpus. A + per-expression `uv` env is the slow, expensive thing tallyman avoids; per-expression + venvs are a bridge to cross later if ever wanted. The in-process build output + (`xorq_build/`) never held the wheel anyway (the `build_expr` file set is + `expr.yaml`/SQL/`deferred_reads.yaml`/`profiles.yaml`); only the catalog zip did, + and nothing reads it back (fact 1). So the drop forfeits xorq's per-entry env + reconstruction — **unused here** — not tallyman's reproducibility, which is the + tracked per-repo lock. Separately: because the zip is deterministic and content-addressed, git stores each blob once and a `git add -A` over an unchanged zip is a no-op that adds no history — so the saving is a smaller durable blob and working tree per entry, not avoided per-checkpoint @@ -364,7 +368,10 @@ All new failing tests bundle into one commit, seen red on CI, before the fixes. ## Scope boundaries / deferred -- No `pyproject.toml` change (fact 5). No merge/sync surface (fact 4). No xorq +- No change to *tallyman's own* `pyproject.toml`/xorq pin (fact 5 — `xorq.catalog` + can't be narrowed out of the single wheel). The `pyproject.toml` + `uv.lock` added + to the tracked tree are a *separate*, per-catalog-repo env lock (the reproducibility + unit), not a change to tallyman's deps. No merge/sync surface (fact 4). No xorq `assert_consistency`/`BuildZip`/`test_zip` contract in the native store (fact 1). - No clone-restore / reconstruct-from-zip implementation; D3 keeps the zip *able* to support it (whole recipe + `/` prefix). @@ -374,11 +381,14 @@ All new failing tests bundle into one commit, seen red on CI, before the fixes. ## Open / not yet grilled -- **Rebuild operational path.** What you actually run to rebuild a corpus - (re-execute stored `expr.py` via `build_and_persist` in a loop vs re-drive the - MCP/codegen session) and whether a `tallyman rebuild` affordance is warranted — - `first-project` is ~119MB, `parking_ticket_analysis_1m` ~10 entries, so a - from-scratch rebuild is minutes-to-tens-of-minutes per large project. Needs a CLI - check before settling. +- **Rebuild operational path.** The *env* half is settled: recreate the project's + one venv from the tracked `uv.lock`, then rebuild every expression in it (one venv + per catalog repo, not per expression). Still open: what you actually run to + re-execute (re-run stored `expr.py` via `build_and_persist` in a loop vs re-drive + the MCP/codegen session), whether a `tallyman rebuild` affordance is warranted + (`first-project` ~119MB, `parking_ticket_analysis_1m` ~10 entries → minutes-to-tens + per large project), and **when `pyproject.toml`/`uv.lock` get snapshotted/refreshed** + (snapshot from the active env at genesis, plus a refresh when a recipe needs a new + dep). Needs a CLI check before settling. - Genesis baseline behavior post-decomposition; exact `.gitignore`; final failing-first test list once the above is settled. From 7689a053bb4607c946cbe59e16209f7a47bb03b5 Mon Sep 17 00:00:00 2001 From: Paddy Mullen Date: Tue, 16 Jun 2026 20:10:57 -0400 Subject: [PATCH 07/27] docs: cross-link #106 (runtime per-project venv isolation) as deferred scope Records the runtime per-project venv change as an explicit deferred follow-on: this plan settles the reproducibility record (tracked pyproject.toml + uv.lock) only; building each project inside its own venv recreated from that lock is the larger architecture change tracked in #106. Co-Authored-By: Claude Opus 4.8 (1M context) --- plans/native-catalog-store.md | 4 ++++ 1 file changed, 4 insertions(+) diff --git a/plans/native-catalog-store.md b/plans/native-catalog-store.md index 60c4575..2eb7511 100644 --- a/plans/native-catalog-store.md +++ b/plans/native-catalog-store.md @@ -378,6 +378,10 @@ All new failing tests bundle into one commit, seen red on CI, before the fixes. - No on-disk migration; rebuild the corpus. - Recipe authoring-surface switch (D4) is a separate, codegen-validated follow-up. - Collapsing `entry_hashes` into `git ls-files` is a separate look. +- Runtime per-project venv isolation (build each project *inside* its own venv, + recreated from the tracked lock, instead of the shared in-process env) is a larger + architecture change — tracked in #106. This plan settles the reproducibility + *record* (the tracked `pyproject.toml` + `uv.lock`) only; the in-process build stays. ## Open / not yet grilled From ba2d942a97c1ab68787c793b436037eca6d32200 Mon Sep 17 00:00:00 2001 From: Paddy Mullen Date: Tue, 16 Jun 2026 20:15:25 -0400 Subject: [PATCH 08/27] docs: record why expr.py stays durable (chaining needs a shared backend) Settles the "why not regenerate expr.py from xorq_build/" question under D3 with evidence, so it is not re-proposed: - The from_catalog chaining path re-execs expr.py so a cheap entry's recompute expression roots on the shared in-process backend; two entries union/join on one backend (#75). Reconstructing the way xorq's catalog does (load_expr per entry) gives each its own backend -> "Multiple backends found" (verified: two load_expr results won't union; two re-exec'd sources do). xorq never hits this because it runs one entry at a time and never composes. - Both source-regeneration escape routes were tested and rejected: the ibis decompiler drops the read path (non-compiling source), and expr.unbind() composes but strips the path and can't execute without re-registering each source by name. Both reimplement, more fragilely, what the already-written expr.py (build.py:367) gives for free. Conclusion unchanged: keep expr.py in the D3 allowlist. Co-Authored-By: Claude Opus 4.8 (1M context) --- plans/native-catalog-store.md | 19 +++++++++++++++++-- 1 file changed, 17 insertions(+), 2 deletions(-) diff --git a/plans/native-catalog-store.md b/plans/native-catalog-store.md index 2eb7511..8c0cd2b 100644 --- a/plans/native-catalog-store.md +++ b/plans/native-catalog-store.md @@ -275,8 +275,23 @@ over a full cycle, and (b) a static grep that `src/` has no the dir is complete. Rewrite the `#48` "not durable" log accordingly. - **D3 — archive scope:** the recipe allowlist `{expr.py, xorq_build/, schema.json, manifest.json}` as one immutable zip (fixes the wrong-artifact - durability gap from fact 2). `prompts.jsonl` is deliberately *not* a member: it - is append-mutable, runtime-read provenance, so it joins the decomposition as a + durability gap from fact 2). **`expr.py` is durable, not regenerated from + `xorq_build/`.** The `from_catalog` chaining path re-execs it so a cheap entry's + recompute expression roots on the *shared* in-process backend and two entries + `union`/`join` on one backend (#75). Reconstructing the way xorq's own catalog + does — `load_expr` per entry — gives each its *own* backend and raises + `Multiple backends found` (verified: two `load_expr` results won't `union`; two + re-exec'd sources do). xorq never hits this because its catalog runs one entry at + a time and never composes; tallyman's chaining is the feature that needs the + shared backend. Two source-regeneration routes were tested and rejected: xorq's + decompiler is ibis's generic one and drops the read path (emits non-compiling, + path-less source); `expr.unbind()` composes but likewise strips the read path and + can't execute without re-registering each source by name. Both reimplement — more + fragilely — what the one already-written `expr.py` (`build.py:367`) gives for + free, so it stays. (Expensive entries don't actually need it for data — they read + a baked snapshot and use `expr.py` only to *locate* it — but the cheap-entry + recompute case does.) `prompts.jsonl` is deliberately *not* a member: it is + append-mutable, runtime-read provenance, so it joins the decomposition as a tracked `prompts/.jsonl` rather than churning the content-addressed blob. - **D4 — recipe imports:** keep user recipes on `import xorq.api as xo`; catalog-free guarantee scoped to `src/`. Real recipes call `xo.connect()`/ From cb44fb2d8d3c0fa58588214c91fe041e53b19518 Mon Sep 17 00:00:00 2001 From: Paddy Mullen Date: Tue, 16 Jun 2026 20:24:27 -0400 Subject: [PATCH 09/27] chore: gitignore .claude/ and .claude-worktrees/ (multi-GB local tooling state) These are Claude Code's local session/tooling state (~1.6G) and agent worktrees (~3.3G), neither of which belongs in the repo. Ignore them so they never get swept into a commit. Co-Authored-By: Claude Opus 4.8 (1M context) --- .gitignore | 3 +++ 1 file changed, 3 insertions(+) diff --git a/.gitignore b/.gitignore index dae1fbd..11657ae 100644 --- a/.gitignore +++ b/.gitignore @@ -18,3 +18,6 @@ catalog/ projects/ # cache lab measurement reports (tests/test_cache_lab.py) tests/cache_lab_reports/ +# Claude Code local tooling state + agent worktrees (multi-GB; never commit) +.claude/ +.claude-worktrees/ From c3b38de87c633b0c3dbad69941d956a571413a40 Mon Sep 17 00:00:00 2001 From: Paddy Mullen Date: Tue, 16 Jun 2026 20:50:39 -0400 Subject: [PATCH 10/27] docs: pin the missing test-plan coverage in native-catalog-store Audit of the embedded TDD section found three load-bearing claims with no named test (recipe-zip byte-stability, recipe-zip member allowlist, Cut-B connect-shim determinism-equivalence) and D3's chaining-after-reset durability untested in the gating suite. The failing-first commit also mixed assertion-red tests with ones that would fail by collection error or false-green. - Add named tests: test_recipe_zip_byte_stable_across_rebuilds, test_recipe_zip_members_are_allowlist, test_recipe_zip_contains_expr_py, test_connect_shim_snapshot_is_byte_identical, test_two_entries_union_after_reset (+ contrapositive), test_src_has_no_xorq_catalog_argv, test_display_config_survives_reset. - Split commit 1 into 1a (assertion-red today) / 1b (lands catalog.py + .gitignore stubs in-commit); build-path no-subprocess guard asserts spy.call_count == 0, scoped past the cp -c clone. - Step 2 now deletes test_catalog_xorq_integration.py in the same commit as xorq_catalog.py (module-level import would collection-error the suite). - Step 6 reclassifies fix-induced breaks (delete vs re-target). - Risk register rows 11-15; design callouts bind each invariant to its test; rebuild-path open item carries xorq's execution-equivalence lesson. Co-Authored-By: Claude Opus 4.8 (1M context) --- plans/native-catalog-store.md | 174 ++++++++++++++++++++++++++++------ 1 file changed, 143 insertions(+), 31 deletions(-) diff --git a/plans/native-catalog-store.md b/plans/native-catalog-store.md index 8c0cd2b..8384cd3 100644 --- a/plans/native-catalog-store.md +++ b/plans/native-catalog-store.md @@ -148,7 +148,9 @@ subprocess, and a from-scratch rebuild never recreates them.) is append-mutable (it grows on a re-run of an existing hash with no rebuild, `build.py:335`), so a content-addressed zip member would churn the blob in git history on every re-run; it becomes a tracked decomposition file instead (see - below). + below). The allowlist-not-denylist membership (and the three exclusions) is + pinned by `test_recipe_zip_members_are_allowlist`; `test_recipe_zip_contains_expr_py` + guards that `expr.py` is present (D3). - Members stored under arcname `/` (single top-level prefix, so a future restore can reuse it). **Named by the salted `content_hash`** when source-identity salt mode is on, not `build_path.name` — otherwise @@ -158,6 +160,8 @@ subprocess, and a from-scratch rebuild never recreates them.) `compresslevel`) so the blob doesn't churn on identical rebuilds. (Single machine, so cross-zlib reproducibility is moot; pinning the level keeps the blob byte-stable across same-machine rebuilds regardless of zlib defaults.) + Pinned by `test_recipe_zip_byte_stable_across_rebuilds` — without it a defaulted + `ZipInfo` mtime embeds the current time and silently re-churns the git blob. - **Drop the wheel + requirements.txt** the `xorq catalog add` subprocess injects (~178KB → ~6KB per entry). These are xorq's *per-expression* reproducibility mechanism — `WheelPackager`/`PackagedBuilder`/`PackagedRunner` @@ -261,9 +265,16 @@ namespace — it is *not* an executed import; left as-is per D4 (the recipe authoring surface is a separate follow-up). Definition of done is **not** a `sys.modules` assertion (user recipes keep -`import xorq.api`, D4). It is: (a) monkeypatch-`subprocess.run`-to-raise guard -over a full cycle, and (b) a static grep that `src/` has no -`import xorq.catalog` / `from xorq.catalog` / `xorq catalog` argv. +`import xorq.api`, D4). It is three named tests: (a) `test_uses_no_subprocess_build` ++ `test_uses_no_subprocess_alias` — a `subprocess.run` call-count spy over a full +cycle asserting zero *xorq* invocations (the build path must assert +`spy.call_count == 0`, not "did not raise": `build.py:477-478` swallows the +exception; scope the spy past the legitimate `cp -c` clone at +`source_identity.py:111`); (b) `test_src_has_no_xorq_catalog_argv` — a static grep +that `src/` has no `import xorq.catalog` / `from xorq.catalog` and no split-token +`xorq … catalog … add` argv (skipping the `tallyman_mcp/server.py:219` codegen +docstring); and (c) `test_connect_shim_snapshot_is_byte_identical` — the datafusion +shim preserves byte-identical content-addressed snapshots. ## Decisions (resolved in the grilling pass) @@ -293,6 +304,12 @@ over a full cycle, and (b) a static grep that `src/` has no recompute case does.) `prompts.jsonl` is deliberately *not* a member: it is append-mutable, runtime-read provenance, so it joins the decomposition as a tracked `prompts/.jsonl` rather than churning the content-addressed blob. + The shared-backend half is already covered in the fast suite + (`test_result_cache.py:214,232`); the **durability** half — that `expr.py` + round-trips via the zip once `entries/*/` is gitignored — is *not*, so + `test_two_entries_union_after_reset` (commit 1b, no `integration`/`cache_lab`/`perf` + marker, since CI deselects those) is the gating test, with the contrapositive + `test_two_load_expr_results_cannot_union` pinning why `expr.py` must be durable. - **D4 — recipe imports:** keep user recipes on `import xorq.api as xo`; catalog-free guarantee scoped to `src/`. Real recipes call `xo.connect()`/ `xo.desc`, so switching the authoring surface needs its own codegen-validation @@ -334,35 +351,108 @@ over a full cycle, and (b) a static grep that `src/` has no |---|---|---|---| | 1 | `git add -A` is allow-by-default; a `.gitignore` denylist silently tracks any unlisted artifact (build dirs, `result.parquet`, future strays) | High | `.gitignore` (`entries/*/` etc.) covers today's heavy artifacts, but the durable guard is `assert_catalog_consistent` enforcing the tracked surface as a path **allowlist** (deny-by-default). Regression test asserts no build-dir file — and no unlisted path — is tracked. | | 2 | `sys.modules`-based "dropped" assertion is unsatisfiable (recipes load `xorq.api`) | High | Definition of done = subprocess guard + `src/` static grep, not `sys.modules`. | -| 3 | `xo.connect()` survives the `xorq.api`→`xorq.expr.api` swap as ibis's `connect(resource, …)` (present, not absent), so it fails at *call* time / builds the wrong backend rather than at import — easy to miss | High | Datafusion connect shim + determinism-equivalence test. | -| 4 | `test_catalog_xorq_integration.py` imports `xorq.catalog.catalog.Catalog` | High | Rewrite in the fix commits: drop `_open_xorq_catalog`/alias-symlink tests; keep the `git ls-files` durability guards. | +| 3 | `xo.connect()` survives the `xorq.api`→`xorq.expr.api` swap as ibis's `connect(resource, …)` (present, not absent), so it fails at *call* time / builds the wrong backend rather than at import — easy to miss | High | Datafusion connect shim + `test_connect_shim_snapshot_is_byte_identical` (commit 1b) — was prose-only, now a named failing-first test. | +| 4 | `test_catalog_xorq_integration.py` imports `xorq.catalog.catalog.Catalog` **at module level** (`:45` also imports the to-be-deleted `xorq_catalog`) | High | Delete/rewrite in the **step-2** commit (same one that deletes `xorq_catalog.py`), not step 6 — a module-level import of the deleted module collection-errors the whole suite from step 2 on. Drop `_open_xorq_catalog`/alias-symlink tests; keep the `git ls-files` durability guards. | | 5 | Salt-mode zip must be named by the salted hash | High | Name `entries/.zip`; test under `TALLYMAN_SOURCE_IDENTITY=salt`. | -| 6 | Commit timing inverts (inline → checkpoint); tests that build with no checkpoint and assert tracked fail | Med | Canonical failing-first test; update `test_xorq_catalog_add_registers_after_genesis` to checkpoint first. | +| 6 | Commit timing inverts (inline → checkpoint); tests that build with no checkpoint and assert tracked fail | Med | `test_xorq_catalog_add_registers_after_genesis` is a **fix-induced break**, not failing-first: edit to checkpoint-then-assert in the step-6 cleanup commit. | | 7 | Orphan loose zip / `--allow-empty` masks a pointer-without-recipe | Med | Retired by D1 (checkpoint is the sole zip writer over complete dirs). | -| 8 | In-process git reintroduces the macOS fork-from-threads SIGSEGV | Med | All native catalog git via `git_util.run_git` (posix_spawn); never GitPython. `test_no_bare_subprocess_git_in_src` enforces. | -| 9 | `test_capture_seeds_xorq_keys` / decomposed-section capture tests pin dropped behavior | Low | Delete/rewrite in the fix commits alongside the decomposition. | +| 8 | In-process git reintroduces the macOS fork-from-threads SIGSEGV | Med | All native catalog git via `git_util.run_git` (posix_spawn); never GitPython. `test_no_bare_subprocess_git_in_src` enforces (note: it matches bare `git` argv only — the `xorq catalog` argv guard is the separate `test_src_has_no_xorq_catalog_argv`). | +| 9 | `test_capture_seeds_xorq_keys` / decomposed-section capture tests pin dropped behavior | Low | **Delete** (not rewrite) in the step-6 cleanup commit alongside the decomposition. | | 10 | Existing corpus is #48-inconsistent; two zip formats coexist | Low | Rebuild from scratch (project rule). | +| 11 | Deterministic-archive claim untested — a defaulted `ZipInfo` mtime silently re-churns the content-addressed git blob on every rebuild | High | `test_recipe_zip_byte_stable_across_rebuilds` (commit 1b): build the same code twice, assert byte-identical archives. | +| 12 | Zip-member allowlist is the "only guard against leak" but lives only as D3 prose; a denylist impl that skips today's three known names passes existence checks yet leaks a future artifact | High | `test_recipe_zip_members_are_allowlist` (commit 1b): assert `namelist()` equals exactly the four members; reject an unknown `foo.bin`. (Distinct surface from `test_unlisted_path_not_tracked`, which guards the git-tracked repo root, not the zip bytes.) | +| 13 | D3 half (b) — `expr.py`-via-zip durability for chaining — untested in the gating suite; the only `reset`+`from_catalog` file is marker-deselected (`pyproject.toml:77`), so a `load_expr`-per-entry refactor would pass the fast suite and silently break chaining | High | `test_two_entries_union_after_reset` + `test_recipe_zip_contains_expr_py` (commit 1b), carrying **no** `integration`/`cache_lab`/`perf` marker. | +| 14 | `display_configs` is the one decomposed section with no survives-reset test today; its only coverage is the capture/materialize round-trip tests step 4 deletes | Med | `test_display_config_survives_reset` (commit 1b) — new reset test so net coverage doesn't regress. | +| 15 | Commit-1 failing-first is internally inconsistent with the TDD rule: some named tests fail by collection error (`test_unlisted_path_not_tracked` → missing API), false-green (`test_uses_no_subprocess` build path swallowed at `build.py:477-478`; `test_prompt_history_survives_reset` survives via bullpen until step-3 `.gitignore`) | High | Split into commit 1a (assertion-red today) and 1b (lands API/`.gitignore` stubs in-commit); build-path guard asserts `spy.call_count == 0`, scoped to the xorq argv (exempt the `cp -c` clone at `source_identity.py:111`). | ## Implementation sequence (ordered commits, TDD) -All new failing tests bundle into one commit, seen red on CI, before the fixes. - -1. **Failing-first commit.** `test_*_uses_no_subprocess` (build/alias paths, - monkeypatch `subprocess.run`), `test_zip_staged_not_committed_until_checkpoint`, - `test_n_builds_track_n_zips` (#48 blind spot), `test_salt_zip_named_by_salted_hash`, - `test_build_dir_files_not_staged`, `test_failed_build_no_orphan_pointer`, - `test_prompt_history_survives_reset` (re-run appends a prompt, checkpoint, - `reset_to`, assert all prompts present — fails today because the loose - `entry_dir/prompts.jsonl` is gitignored), `test_unlisted_path_not_tracked` - (stage a stray file, assert `assert_catalog_consistent` raises). Confirm - `run_git` is posix_spawn (it is) before relying on the monkeypatch. +Every new test must be seen red on CI **by a real assertion**, never by a +collection/import error (the repo TDD rule excludes those). That forces the +failing-first work into two commits, not one: some assertions go red against +today's inline-xorq tree, but others reference symbols +(`assert_catalog_consistent`, `write_recipe_zip`) or a `.gitignore` that only +later commits create. Commit 1a holds the first kind; commit 1b lands thin stubs +of those symbols + the `.gitignore` in the *same* commit as its tests, so the +assertion is what's red, not the import. Confirm `run_git` is posix_spawn (it is) +before relying on any `subprocess.run` monkeypatch. + +1a. **Failing-first, assertion-red against today's tree** (use existing public + surfaces only — `build_and_persist`, `checkpoint_catalog`, `git ls-files` — + never the not-yet-created `tallyman_core.catalog` API, or these collect-error): + - `test_uses_no_subprocess_alias` — monkeypatch `subprocess.run` to raise; + `set_alias` raises today (`aliases.py:129` → `_xcat.add_alias`, no try/except). + - `test_uses_no_subprocess_build` — assert a **call-count spy** records zero + *xorq* `subprocess.run` invocations. Must NOT assert "did not raise": + `build.py:477-478` swallows the exception into `catalog_registered=False`, so a + raising stub false-greens. Scope the spy to the xorq argv so it doesn't trip on + the legitimate `cp -c` CoW clone at `source_identity.py:111`. + - `test_zip_staged_not_committed_until_checkpoint` (D1 — committed inline today). + - `test_n_builds_track_n_zips` (#48 blind spot — 4 build dirs, 1 tracked zip today). + - `test_salt_zip_named_by_salted_hash` (Risk #5 — xorq names by `build_path.name`). + - `test_build_dir_files_not_staged` (Risk #1 — no `.gitignore` today). + - `test_failed_build_no_orphan_pointer` (D1/Risk #7). + +1b. **Failing-first, assertion-red once this commit's stubs land.** Land + stub `tallyman_core/catalog.py` (`assert_catalog_consistent`, `write_recipe_zip` + as no-op/identity) **and** the `.gitignore` (`entries/*/` etc.) in this commit so + the tests below fail by assertion: + - `test_unlisted_path_not_tracked` — stage a stray file, assert + `assert_catalog_consistent` raises (stub returns without raising → red). + - `test_recipe_zip_byte_stable_across_rebuilds` — build the same code twice + (`res2.content_hash == res1.content_hash`, the `test_portable.py:134-145` + idempotent pattern), zip each entry dir, assert the two archives are + byte-identical. Guards the deterministic-archive claim — a defaulted `ZipInfo` + mtime silently re-churns the git blob on every rebuild and nothing else catches it. + - `test_recipe_zip_members_are_allowlist` — drop strays + (`.xorq_build_expanded/x`, `result.parquet`, `prompts.jsonl`, an unknown + `foo.bin`) into a real entry dir, run `write_recipe_zip`, assert + `set(namelist())` under the `/` prefix equals exactly + `{expr.py, xorq_build/**, schema.json, manifest.json}`. Pins allowlist-not-denylist: + a denylist skipping only today's three known names passes existence checks but + leaks `foo.bin`. + - `test_prompt_history_survives_reset` — re-run appends a prompt + (`build.py:335`), checkpoint, `reset_to`, assert all prompts present. Red only + once `entries/*/` is gitignored (pre-staged in this commit); the loose + `entry_dir/prompts.jsonl` is then lost on reset. (Verify the bullpen copytree + does not otherwise preserve it before relying on this.) + - `test_two_entries_union_after_reset` — two `from_catalog` entries `union` after a + `reset_to` that restores `expr.py` from the zip only (D3 half (b)). **Carries no + `integration`/`cache_lab`/`perf` marker** — CI runs bare `uv run pytest` and + deselects those (`pyproject.toml:77`), so a marked test would share the existing + blind spot. Optionally add the contrapositive `test_two_load_expr_results_cannot_union` + to pin why `expr.py` is load-bearing. + - `test_recipe_zip_contains_expr_py` — assert `expr.py` is a zip member (D3). + - `test_connect_shim_snapshot_is_byte_identical` — the shim-built and + `xo.connect()`-built `ParquetSnapshotCache` must produce byte-identical + content-addressed snapshots (Cut B / Risk #3; the no-arg `xo.connect()` sites are + `src/tallyman_xorq/source_cache.py:133,152`). The central Cut-B correctness claim. + - `test_src_has_no_xorq_catalog_argv` — static grep over `src/` for + `import xorq.catalog` / `from xorq.catalog` **and** the split-token catalog argv + (`_xorq(project, "catalog", "add", …)` in `xorq_catalog.py:31`, so a literal + `"xorq catalog"` string match misses it). Model on + `test_no_bare_subprocess_git_in_src` (`test_reset_to_revision.py:81-89`); add a + sanctioned skip for the `tallyman_mcp/server.py:219` codegen docstring (D4). + - `test_display_config_survives_reset` — edit a display config on an existing hash, + checkpoint, `reset_to`, assert restored. The one decomposed section with no reset + coverage today (`test_reset_roundtrip_over_every_kind`, + `test_reset_to_revision.py:215-232`, covers pp/stats/charts/notebook, not display). 2. **Native recipe store** — `tallyman_core/catalog.py` `write_recipe_zip`; `build.py` persists the complete dir + cleans up on failure; delete - `xorq_catalog.py` *and* strip its only other caller's coupling in the same - commit — `aliases.py`'s module-level `from tallyman_core import xorq_catalog` plus the - `_xcat.add_alias`/`remove_alias` calls in `set_alias`/`remove_alias` — so the - package stays importable (the failing-first suite from commit 1 can't run red - against an un-importable tree). + `xorq_catalog.py` *and*, in the same commit, strip every module-level coupling to + it so the tree stays importable (commit 1a's suite can't run red against an + un-importable tree). Three module-level imports of the deleted module must go in + this commit, not later: + - `aliases.py:24` `from tallyman_core import xorq_catalog` + the + `_xcat.add_alias`/`remove_alias` calls in `set_alias`/`remove_alias` + (`:129`,`:155`). + - `tests/test_catalog_xorq_integration.py:45` `from tallyman_core import + xorq_catalog as xc` (and `:99` `from xorq.catalog.catalog import Catalog`). The + plan otherwise defers this file's rewrite to step 6, but its import is + module-level: the moment `xorq_catalog.py` is deleted the **whole suite** goes + red by collection error, not assertion. Delete/rewrite it here (drop the + `_open_xorq_catalog`/alias-symlink tests; keep the `git ls-files` durability + guards), so steps 2–5 stay green locally. 3. **Checkpoint owns zipping** — `.gitignore`; `checkpoint_catalog` zips pending + `git add -A`; `prune_entries`/consistency move into `catalog.py`, with `assert_catalog_consistent` extended to enforce the tracked-path allowlist. @@ -374,9 +464,27 @@ All new failing tests bundle into one commit, seen red on CI, before the fixes. `post_processing/`, `stats/`, and the notebook (`project_dir/notebooks/`) under the repo. 5. **Cut B** — narrow catalog-free imports + connect shim across the 8 modules. -6. **Test-cleanup commit** — rewrite/delete the xorq-coupled integration tests; - re-target `test_failed_registration_is_observable` to a native failure; drop the - xorq rationale from `test_catalog_repo_branch_is_main`. +6. **Test-cleanup commit** — the remaining edits are **fix-induced breaks** (an + existing test goes red *because* the fix landed), not failing-first tests; they + belong with the fixes, never bundled into commit 1: + - `test_failed_registration_is_observable` — **delete**, don't re-target. Its + premise (a swallowed best-effort `catalog_registered` bool) vanishes with the + registration path; the stray-file→raise behavior it gestures at is exactly what + `test_unlisted_path_not_tracked` now covers, so re-targeting would just duplicate. + - `test_capture_seeds_xorq_keys` — **delete** (not "rewrite"): it asserts `capture` + seeds empty `entries`/`aliases` keys in `catalog.yaml` so `xorq catalog add` + won't `KeyError`; with `catalog.yaml` and the subprocess both gone the invariant + ceases to exist. Breaks when step 4 stops writing `catalog.yaml`. + - `test_xorq_catalog_add_registers_after_genesis` — goes red on the fix branch + because the zip is now written at checkpoint, not inline at build. Edit to + checkpoint-then-assert (passes only post-fix), so it lives here, not in + failing-first. Its "best-effort xorq registration must land" docstring is dead. + - `test_catalog_repo_branch_is_main` — docstring-only edit (drop the xorq + rationale; the `main`-branch assertion stays valid for our native repo). + - The capture/materialize round-trip tests in `test_display_configs.py` break when + step 4 deletes those primitives; this commit removes them, and the new + `test_display_config_survives_reset` (commit 1b) keeps net reset coverage from + regressing. 7. **Rebuild + verify** — rebuild the corpus; `git ls-files` shows only the tracked tree above; genesis→create×2→reset roundtrip passes; full suite green locally before push. @@ -408,6 +516,10 @@ All new failing tests bundle into one commit, seen red on CI, before the fixes. (`first-project` ~119MB, `parking_ticket_analysis_1m` ~10 entries → minutes-to-tens per large project), and **when `pyproject.toml`/`uv.lock` get snapshotted/refreshed** (snapshot from the active env at genesis, plus a refresh when a recipe needs a new - dep). Needs a CLI check before settling. -- Genesis baseline behavior post-decomposition; exact `.gitignore`; final - failing-first test list once the above is settled. + dep). Needs a CLI check before settling. Whatever the mechanism, its verification + should assert **execution-equivalence** (rebuilt entries execute to identical + data), not just a structural roundtrip — xorq's `test_rebuild_produces_consistent_target` + exists because a rebuild can match on schema/recipe/hash while silently returning + wrong rows. +- Genesis baseline behavior post-decomposition and the exact `.gitignore` contents. + (The failing-first list is now settled above as commits 1a/1b.) From 411f701e1c547e77a6d25f1be4bbc55323027f07 Mon Sep 17 00:00:00 2001 From: Paddy Mullen Date: Tue, 16 Jun 2026 21:36:37 -0400 Subject: [PATCH 11/27] =?UTF-8?q?test:=20native=20catalog=20store=20?= =?UTF-8?q?=E2=80=94=20no=20xorq=20subprocess,=20checkpoint=20owns=20the?= =?UTF-8?q?=20zip=20(failing-first)?= MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit Commit 1a of the native-catalog-store cut (plans/native-catalog-store.md): the failing-first tests that are assertion-red against today's xorq-subprocess tree, using only existing public surfaces (build_and_persist, checkpoint_catalog, git ls-files). - test_uses_no_subprocess_alias: set_alias shells out to `xorq catalog add-alias` today (aliases.py:129, no try/except) — must not. - test_uses_no_subprocess_build: a full build invokes `xorq catalog add`; the spy asserts call_count==0 (not "did not raise" — build.py swallows the registration exception into catalog_registered=False). - test_zip_staged_not_committed_until_checkpoint (D1): xorq commits the recipe zip inline at build, outside the flock; the checkpoint must own it. - test_failed_build_no_orphan_pointer (D1/Risk #7): a load_expr failure after target.mkdir leaves a partial entry dir → orphan pointer; the build must clean up. Two planned 1a guards (n-builds-track-n-zips, build-dir-files-not-staged) were verified GREEN today — the merged #48 fix already registers every entry in a fresh project, and checkpoint uses an explicit `git add catalog.yaml` pathspec (not `git add -A`). Per the TDD rule they ship with their fix commit, not here. Co-Authored-By: Claude Opus 4.8 (1M context) --- tests/test_native_catalog_store.py | 131 +++++++++++++++++++++++++++++ 1 file changed, 131 insertions(+) create mode 100644 tests/test_native_catalog_store.py diff --git a/tests/test_native_catalog_store.py b/tests/test_native_catalog_store.py new file mode 100644 index 0000000..7f5ebfb --- /dev/null +++ b/tests/test_native_catalog_store.py @@ -0,0 +1,131 @@ +"""Native catalog store (drop xorq's catalog package) — the Cut-A/Cut-B suite. + +See plans/native-catalog-store.md. These pin the contract the native +``tallyman_core.catalog`` store must satisfy: the build/alias paths invoke no +xorq ``catalog`` subprocess, the checkpoint (not the build) owns the durable +recipe zip, and a failed build leaves no orphan pointer. + +Carries no ``integration``/``cache_lab``/``perf`` marker, so a bare +``uv run pytest`` (CI default, pyproject.toml:77) exercises them. +""" + +from __future__ import annotations + +import subprocess +from pathlib import Path + +from tallyman_core import catalog_state as cs +from tallyman_core import paths +from tallyman_xorq import build_and_persist + + +def _agg_code(parquet_path: Path) -> str: + return ( + "import xorq.api as xo\n" + f"t = xo.deferred_read_parquet({str(parquet_path)!r})\n" + "expr = t.group_by('region').aggregate(total=t.price.sum(), n=t.count())\n" + ) + + +def _tracked_zip_hashes(project: str) -> set[str]: + rc, out, _ = cs.run_git(["ls-files", "entries"], cwd=paths.catalog_dir(project)) # type: ignore[attr-defined] + return {ln[len("entries/") : -len(".zip")] for ln in out.split() if ln.endswith(".zip")} + + +class _XorqSpy: + """Wrap ``subprocess.run`` and count only ``uv run xorq catalog ...`` calls. + + Scoped to the xorq argv so it never trips on the ``cp -c`` CoW clone + (source_identity.py:111) or any other subprocess. Delegates everything to + the real ``subprocess.run`` so behaviour is unchanged. + """ + + def __init__(self, real): + self._real = real + self.xorq_calls = 0 + + def __call__(self, argv, *a, **k): + if isinstance(argv, (list, tuple)) and list(argv[:4]) == ["uv", "run", "xorq", "catalog"]: + self.xorq_calls += 1 + return self._real(argv, *a, **k) + + +# --------------------------------------------------------------------------- +# Cut A — no xorq subprocess on the build / alias paths +# --------------------------------------------------------------------------- + + +def test_uses_no_subprocess_alias(project, monkeypatch): + """set_alias must not shell out to ``xorq catalog add-alias``. + + Today aliases.py:129 calls the xorq subprocess with no try/except, so a + raising ``subprocess.run`` propagates out of set_alias. Native aliases write + a tracked ``aliases.jsonl`` and never spawn xorq, so this must not raise. + """ + from tallyman_core import aliases as al + + cs.genesis(project) # the catalog repo must exist or add_alias returns early + + def _boom(*a, **k): + raise AssertionError("set_alias spawned a subprocess (xorq catalog add-alias)") + + monkeypatch.setattr(subprocess, "run", _boom) + al.set_alias(project, "myalias", "deadbeefdead", expect_exists=False) + + +def test_uses_no_subprocess_build(project, orders_parquet, monkeypatch): + """A full build must invoke zero ``xorq catalog`` subprocesses. + + Asserts a call-count spy of zero, NOT "did not raise": build.py:539-541 + swallows a raising registration into ``catalog_registered=False``, so a + raising stub would false-green. + """ + cs.genesis(project) # so add_entry reaches the subprocess rather than returning early + spy = _XorqSpy(subprocess.run) + monkeypatch.setattr(subprocess, "run", spy) + build_and_persist(project, _agg_code(orders_parquet)) + assert spy.xorq_calls == 0, f"build spawned {spy.xorq_calls} xorq catalog subprocess(es)" + + +# --------------------------------------------------------------------------- +# Cut A — the checkpoint owns the durable recipe zip (D1) +# --------------------------------------------------------------------------- + + +def test_zip_staged_not_committed_until_checkpoint(project, orders_parquet): + """The durable ``entries/.zip`` is written by the checkpoint, not the + build. Today xorq's ``catalog add`` commits it inline (outside the flock), + so it is tracked before any tallyman checkpoint — D1 moves that to the + checkpoint under the project lock. + """ + cs.genesis(project) + res = build_and_persist(project, _agg_code(orders_parquet)) + assert res.content_hash not in _tracked_zip_hashes(project), ( + "recipe zip was committed at build time; D1 requires the checkpoint to own it" + ) + cs.checkpoint_catalog(project, "create") + assert res.content_hash in _tracked_zip_hashes(project), "checkpoint did not commit the recipe zip" + + +def test_failed_build_no_orphan_pointer(project, orders_parquet, monkeypatch): + """A build that fails after creating its entry dir leaves no dir behind, so + the next checkpoint records no pointer without a durable recipe (D1/Risk #7). + """ + import xorq.ibis_yaml.compiler as compiler + + cs.genesis(project) + real_load = compiler.load_expr + + def _boom_load(*a, **k): + raise RuntimeError("induced load failure after target.mkdir") + + monkeypatch.setattr(compiler, "load_expr", _boom_load) + try: + build_and_persist(project, _agg_code(orders_parquet)) + except Exception: + pass + monkeypatch.setattr(compiler, "load_expr", real_load) + + ed = paths.entries_dir(project) + leftover = [c.name for c in ed.iterdir() if c.is_dir()] if ed.exists() else [] + assert not leftover, f"failed build left orphan entry dir(s): {leftover}" From 6c828663d6ce3f4f0b7366b6b6d19383f0c8b856 Mon Sep 17 00:00:00 2001 From: Paddy Mullen Date: Tue, 16 Jun 2026 22:07:27 -0400 Subject: [PATCH 12/27] feat: native content-addressed recipe store; checkpoint owns the zip (Cut A) Stand up tallyman_core/catalog.py as the native recipe store and delete the xorq catalog subprocess. The build now persists a complete entry dir (and cleans it up on failure); the checkpoint, under the project lock, is the sole zip writer and sole git transaction. - catalog.py: write_recipe_zip (deterministic, allowlisted {expr.py, xorq_build/, schema.json, manifest.json} under one / prefix), tracked_recipe_hashes, zip_pending_entries, assert_catalog_consistent (tracked-surface path allowlist + pointer/recipe agreement, #52), and the catalog-repo .gitignore. - catalog_state.py: ensure_catalog_repo writes the .gitignore; checkpoint zips pending entries then `git add -A` (was `git add catalog.yaml`); reset_to delegates consistency to catalog.assert_catalog_consistent. Drops the inline _tracked_recipe_hashes/_assert_catalog_consistent/_TRACKED. - build.py: drop the `xorq catalog add` subprocess; catalog_registered now means "entry dir persisted" (D2). rmtree the entry dir on a load/execute failure so a partial build leaves no orphan pointer (D1/Risk #7). - aliases.py: strip the module-level xorq_catalog import and the add_alias/ remove_alias subprocess calls (alias state stays in catalog.yaml for now; decomposition to aliases.jsonl is the next commit). - Delete xorq_catalog.py. Delete the all-@integration test_catalog_xorq_integration.py (built on Catalog.from_kwargs / alias symlinks) and port its still-valid durability guards into test_native_catalog_store.py as fast in-process tests. The four commit-1a failing-first tests now pass; added allowlist / byte-stable / expr.py-member / no-xorq-argv / consistency / salt-naming / reset-durability coverage. The catalog repo's first .gitignore also ignores the #104 migration marker so `git add -A` stays honest. Full fast suite: 512 passed. Co-Authored-By: Claude Opus 4.8 (1M context) --- src/tallyman_core/aliases.py | 4 - src/tallyman_core/catalog.py | 200 +++++++++++++++ src/tallyman_core/catalog_state.py | 70 ++---- src/tallyman_core/xorq_catalog.py | 101 -------- src/tallyman_xorq/build.py | 34 +-- tests/test_catalog_xorq_integration.py | 323 ------------------------- tests/test_native_catalog_store.py | 251 ++++++++++++++++--- 7 files changed, 445 insertions(+), 538 deletions(-) create mode 100644 src/tallyman_core/catalog.py delete mode 100644 src/tallyman_core/xorq_catalog.py delete mode 100644 tests/test_catalog_xorq_integration.py diff --git a/src/tallyman_core/aliases.py b/src/tallyman_core/aliases.py index a249598..54765bf 100644 --- a/src/tallyman_core/aliases.py +++ b/src/tallyman_core/aliases.py @@ -21,7 +21,6 @@ from __future__ import annotations -from tallyman_core import xorq_catalog as _xcat from tallyman_core.paths import ensure_project @@ -126,8 +125,6 @@ def set_alias(project: str, name: str, content_hash: str, *, expect_exists: bool _write(project, aliases, history) - _xcat.add_alias(project, content_hash, alias=name) - return { "name": name, "hash": content_hash, @@ -152,4 +149,3 @@ def remove_alias(project: str, name: str) -> None: aliases.pop(name, None) history.pop(name, None) _write(project, aliases, history) - _xcat.remove_alias(project, name) diff --git a/src/tallyman_core/catalog.py b/src/tallyman_core/catalog.py new file mode 100644 index 0000000..293f796 --- /dev/null +++ b/src/tallyman_core/catalog.py @@ -0,0 +1,200 @@ +"""The native content-addressed recipe store (replaces xorq's catalog package). + +tallyman owns the catalog *model*; this module is the *store*. It writes one +immutable, deterministic ``entries/.zip`` per entry from an allowlist of +the entry dir, and enforces the tracked surface as a path allowlist +(deny-by-default) so a stray file ``git add -A`` would otherwise stage is +rejected instead of committed silently (#52). + +Call direction is one-way: ``catalog_state`` (the versioning engine) calls into +here; this module imports only ``paths``/``git_util`` and never +``catalog_state`` — ``assert_catalog_consistent`` takes the pointer set as an +argument to keep it acyclic. + +What we deliberately drop from xorq's catalog: ``assert_consistency`` (its tree +contract), the three-way merge, the per-expression wheel + requirements.txt, and +the zip internal format / ``REQUIRED_ARCHIVE_NAMES``. Nothing reads our zip back +(the only consumer is ``tracked_recipe_hashes`` over zip *names*), so the format +is entirely ours. +""" + +from __future__ import annotations + +import fnmatch +import zipfile +from pathlib import Path + +from tallyman_core.git_util import run_git +from tallyman_core.paths import catalog_dir, entries_dir + +# The immutable recipe is an ALLOWLIST of the entry dir, not "everything except": +# a future derived artifact dropped in the entry dir can't leak into the durable +# zip. Top-level files plus the xorq_build/ tree. Excludes .xorq_build_expanded/ +# (machine-local), result.parquet (untracked heavy result), and prompts.jsonl +# (append-mutable provenance — a tracked decomposition file instead). +RECIPE_TOP_LEVEL = ("expr.py", "schema.json", "manifest.json") +RECIPE_TREE_DIRS = ("xorq_build",) + +# Deterministic archive constants so an identical rebuild yields a byte-identical +# blob (no churn in git history). Single machine, so cross-zlib reproducibility +# is moot; pinning the level keeps bytes stable regardless of zlib defaults. +_ZIP_DATE_TIME = (1980, 1, 1, 0, 0, 0) +_ZIP_EXTERNAL_ATTR = 0o644 << 16 +_ZIP_COMPRESSLEVEL = 6 + +# The git-tracked surface as fnmatch patterns (deny-by-default). Every +# ``git ls-files`` path must match one of these or assert_catalog_consistent +# raises. The heavy/derived artifacts are kept out by the repo .gitignore; this +# allowlist is the durable guard, since ``git add -A`` is allow-by-default. +TRACKED_SURFACE = ( + ".gitignore", + "catalog.yaml", + "entries/*.zip", + "chart_specs/*.vl.json", + "display_configs/*.json", + "aliases.jsonl", + "notebook.jsonl", + "entries.jsonl", + "compute_cache.jsonl", + "prompts/*.jsonl", + "post_processing/*.py", + "post_processing/_disabled/*.py", + "stats/*.py", + "stats/_disabled/*.py", + "pyproject.toml", + "uv.lock", +) + +# The repo .gitignore: keep the heavy/derived artifacts out of ``git add -A``. +# Trailing-slash patterns match directories only, so ``entries/*/`` ignores the +# build dirs while ``entries/.zip`` files stay trackable, and +# ``compute_cache/`` ignores the cache dir while ``compute_cache.jsonl`` (the +# pointer file) stays trackable. +GITIGNORE_LINES = ( + "entries/*/", + "bullpen/", + "result_cache/", + "compute_cache/", + "diff_stat_cache/", + ".checkpoint.lock", + ".xorq_build_expanded/", + ".migrated_*", # machine-local upgrade-migration markers (e.g. #104), not catalog content +) + + +def write_gitignore(project: str) -> Path: + """Write the catalog repo's .gitignore (idempotent). The native store is the + first to introduce one — before, an explicit ``git add catalog.yaml`` kept + staging honest; now ``git add -A`` needs the denylist (plus the allowlist + consistency guard).""" + p = catalog_dir(project) / ".gitignore" + body = "\n".join(GITIGNORE_LINES) + "\n" + if not p.exists() or p.read_text() != body: + p.parent.mkdir(parents=True, exist_ok=True) + p.write_text(body) + return p + + +def _recipe_members(entry_dir: Path) -> list[tuple[Path, str]]: + """(abs_path, relpath) for every allowlisted recipe file, sorted by relpath.""" + members: list[tuple[Path, str]] = [] + for name in RECIPE_TOP_LEVEL: + f = entry_dir / name + if f.is_file(): + members.append((f, name)) + for d in RECIPE_TREE_DIRS: + root = entry_dir / d + if root.is_dir(): + for f in root.rglob("*"): + if f.is_file(): + members.append((f, str(f.relative_to(entry_dir)))) + return sorted(members, key=lambda t: t[1]) + + +def write_recipe_zip(entry_dir: Path, dest: Path, content_hash: str) -> Path: + """Archive the recipe allowlist of *entry_dir* into *dest*, deterministically. + + Members stored under arcname ``/`` (single top-level + prefix so a future clone-restore can reuse it). Named by the (possibly + salted) *content_hash*, not ``build_path.name`` — otherwise the zip names + diverge from the entry pointers under source-identity salt mode (Risk #5). + """ + members = _recipe_members(entry_dir) + dest.parent.mkdir(parents=True, exist_ok=True) + tmp = dest.with_name(dest.name + ".tmp") + with zipfile.ZipFile(tmp, "w", compression=zipfile.ZIP_DEFLATED, compresslevel=_ZIP_COMPRESSLEVEL) as zf: + for path, rel in members: + info = zipfile.ZipInfo(f"{content_hash}/{rel}", date_time=_ZIP_DATE_TIME) + info.compress_type = zipfile.ZIP_DEFLATED + info.external_attr = _ZIP_EXTERNAL_ATTR + zf.writestr(info, path.read_bytes()) + tmp.replace(dest) + return dest + + +def tracked_recipe_hashes(project: str) -> set[str]: + """Hashes of the git-tracked ``entries/.zip`` recipes — the durable, + content-addressed artifact set (what survives a clone or ``git reset``), as + opposed to the untracked ``entries//`` build dirs the bullpen + reconciles. Sourced from ``git ls-files`` so it reflects the committed tree. + """ + rc, out, _ = run_git(["ls-files", "entries"], cwd=catalog_dir(project)) + if rc != 0: + return set() + return {ln[len("entries/") : -len(".zip")] for ln in out.split() if ln.endswith(".zip")} + + +def zip_pending_entries(project: str) -> list[str]: + """Zip every complete entry dir that lacks a tracked recipe zip. + + The sole zip writer, called by the checkpoint under the project lock. A + "complete" dir has a manifest (the build's last write), so a partial dir + from an interrupted build is skipped, not zipped. Deterministic + content- + addressed, so re-zipping an unchanged entry is a byte-identical no-op. + """ + tracked = tracked_recipe_hashes(project) + ed = entries_dir(project) + if not ed.exists(): + return [] + written: list[str] = [] + for child in sorted(ed.iterdir()): + if not child.is_dir() or child.name in tracked: + continue + if not (child / "manifest.json").is_file(): + continue # incomplete build dir — let the next checkpoint pick it up + write_recipe_zip(child, ed / f"{child.name}.zip", child.name) + written.append(child.name) + return written + + +def assert_catalog_consistent(project: str, pointers: set[str]) -> None: + """Two guards over the committed tree (#52), raising on either violation. + + 1. **Tracked-surface allowlist** (deny-by-default): every ``git ls-files`` + path must fnmatch ``TRACKED_SURFACE`` or it raises — so a stray dir that + ``git add -A`` would otherwise stage is rejected, not silently committed. + 2. **Pointer/recipe agreement**: the git-tracked recipe set + (``entries/.zip``) must equal the tallyman *pointers* + (``entry_hashes``), so a pointer with no durable recipe — or a recipe + with no pointer — fails loudly instead of returning a masked divergence. + + Takes *pointers* as an argument (rather than reading ``catalog_state``) to + keep the call direction one-way and acyclic. + """ + cd = catalog_dir(project) + rc, out, _ = run_git(["ls-files"], cwd=cd) + if rc == 0: + unlisted = [p for p in out.split() if not any(fnmatch.fnmatch(p, pat) for pat in TRACKED_SURFACE)] + if unlisted: + raise RuntimeError( + "catalog tracks paths outside the allowed surface (a stray file `git add -A` would commit): " + f"{sorted(unlisted)}" + ) + tracked = tracked_recipe_hashes(project) + if tracked != pointers: + missing = sorted(pointers - tracked) # pointer present, no durable recipe + orphan = sorted(tracked - pointers) # durable recipe, no pointer + raise RuntimeError( + "catalog inconsistent: the git-tracked recipe set and the entry_hashes pointers disagree " + f"(pointers with no durable recipe: {missing}; tracked recipes with no pointer: {orphan})" + ) diff --git a/src/tallyman_core/catalog_state.py b/src/tallyman_core/catalog_state.py index a35dc0b..5055f5c 100644 --- a/src/tallyman_core/catalog_state.py +++ b/src/tallyman_core/catalog_state.py @@ -48,7 +48,7 @@ import yaml -from tallyman_core import charts, notebook +from tallyman_core import catalog, charts, notebook from tallyman_core import display_configs as dc from tallyman_core import post_processing as pp from tallyman_core import summary_stats as ss @@ -68,10 +68,6 @@ # git-level identity flags, so commits work without a global git config. _GIT_ID = ["-c", "user.email=tallyman@local", "-c", "user.name=tallyman"] -# Only catalog.yaml is git-tracked in the catalog repo. Alias bookkeeping is -# carried as catalog.yaml keys, not separate files — tracking those alongside -# xorq's managed set made its assert_consistency reject the repo (#48). -_TRACKED = ("catalog.yaml",) _STEP_RE = re.compile(r"^step-(\d+)$") # Refs and labels are operator input that ends up as git arguments. A plain # tag-shaped name only: no leading dash (option injection), no revision @@ -356,12 +352,15 @@ def ensure_catalog_repo(project: str) -> bool: """ cd = catalog_dir(project) cd.mkdir(parents=True, exist_ok=True) - if (cd / ".git").exists(): - return True - rc, _, err = run_git(["init", "-q", "-b", "main"], cwd=cd) - if rc != 0: - log.warning("catalog git init failed in %s: %s", cd, err) - return False + if not (cd / ".git").exists(): + rc, _, err = run_git(["init", "-q", "-b", "main"], cwd=cd) + if rc != 0: + log.warning("catalog git init failed in %s: %s", cd, err) + return False + # The native store's .gitignore keeps the heavy/derived artifacts out of the + # checkpoint's `git add -A`; the allowlist consistency guard is the durable + # backstop. Idempotent, so existing repos pick it up too. + catalog.write_gitignore(project) return True @@ -390,9 +389,12 @@ def checkpoint_catalog(project: str, message: str, *, step: int | None = None, l with _project_lock(project): ensure_catalog_repo(project) capture_tallyman_state(project) - existing = [f for f in _TRACKED if (cd / f).exists()] - if existing: - run_git(["add", *existing], cwd=cd) + # The checkpoint is the sole zip writer and sole git transaction: zip any + # complete entry dir lacking a tracked recipe, then stage the whole + # tracked surface. A committed pointer always has its recipe in the same + # commit; the .gitignore + allowlist keep `git add -A` honest. + catalog.zip_pending_entries(project) + run_git(["add", "-A"], cwd=cd) if step is None: tags = _step_tags(project) step = tags[-1] + 1 if tags else 0 @@ -422,44 +424,6 @@ def _resolve_tag(project: str, tag: str) -> str: return out -def _tracked_recipe_hashes(project: str) -> set[str]: - """Hashes of the git-tracked ``entries/.zip`` recipes. - - The durable, content-addressed xorq artifact set — what survives a clone or - a ``git reset`` — as opposed to the untracked ``entries//`` build dirs - the bullpen reconciles. Sourced from ``git ls-files`` so it reflects the - committed tree, not whatever the working dir happens to hold. - """ - rc, out, _ = run_git(["ls-files", "entries"], cwd=catalog_dir(project)) - if rc != 0: - return set() - return {line[len("entries/") : -len(".zip")] for line in out.split() if line.endswith(".zip")} - - -def _assert_catalog_consistent(project: str) -> None: - """The two views of "what entries exist" must agree after a reset. - - ``git reset --hard`` restores the durable xorq recipe set - (``entries/.zip``); the bullpen restores the untracked build dirs - against the ``entry_hashes`` pointers. They are reconciled by different - mechanisms that never see each other, so a pointer whose recipe was never - committed (any interrupted or failed best-effort ``xorq catalog add``) comes - back from the bullpen as a build-dir copy while the recipe it should recompute - from does not exist. Surface that divergence instead of returning a catalog - whose two views silently disagree and reporting success (#52). - """ - pointers = set(read_tallyman_state(project)["entry_hashes"]) - tracked = _tracked_recipe_hashes(project) - if tracked != pointers: - missing = sorted(pointers - tracked) # pointer present, no durable recipe - orphan = sorted(tracked - pointers) # durable recipe, no pointer - raise RuntimeError( - "catalog inconsistent after reset: the git-tracked xorq recipe set and the " - f"tallyman entry_hashes pointers disagree (pointers with no durable recipe: " - f"{missing}; tracked recipes with no pointer: {orphan})" - ) - - def reset_to(project: str, ref: int | str) -> None: """Restore the catalog to a step/label: git reset --hard, materialize the derived files, then reconcile the untracked artifacts to the recorded @@ -478,7 +442,7 @@ def reset_to(project: str, ref: int | str) -> None: prune_entries(project) prune_compute_cache(project) restore_from_bullpen(project) - _assert_catalog_consistent(project) + catalog.assert_catalog_consistent(project, set(read_tallyman_state(project)["entry_hashes"])) def genesis(project: str) -> int | None: diff --git a/src/tallyman_core/xorq_catalog.py b/src/tallyman_core/xorq_catalog.py deleted file mode 100644 index c683930..0000000 --- a/src/tallyman_core/xorq_catalog.py +++ /dev/null @@ -1,101 +0,0 @@ -"""Thin wrappers that keep the xorq catalog git repo in sync. - -All functions are best-effort: failures are logged but never propagate to the -caller so a missing or misconfigured catalog repo never breaks the main build. - -Each function takes an explicit ``project`` argument — the xorq catalog -repo lives at ``/artifacts/catalog/``, and the calling layer -(aliases.py, build.py) already knows which project it's mutating. Using -``resolve_project()`` here would risk writing to the wrong catalog if the -user switched projects mid-operation. -""" - -from __future__ import annotations - -import logging -import shutil -import subprocess -import tempfile -from pathlib import Path - -from tallyman_core.paths import catalog_dir - -log = logging.getLogger(__name__) - - -def _repo_path(project: str) -> Path: - return catalog_dir(project) - - -def _xorq(project: str, *args: str, **kw) -> subprocess.CompletedProcess: - return subprocess.run( - ["uv", "run", "xorq", "catalog", "--path", str(_repo_path(project)), *args], - capture_output=True, - text=True, - **kw, - ) - - -def add_entry( - project: str, - build_dir: Path, - aliases: list[str] = (), - *, - entry_name: str | None = None, -) -> bool: - """Add *build_dir* to the xorq catalog, optionally tagging with *aliases*. - - *entry_name* lets the caller control the name xorq uses for the entry - (xorq derives it from the directory name). When supplied the build dir - is copied into a temp location first. - - Returns True on success. - """ - repo = _repo_path(project) - if not repo.exists(): - log.warning("xorq catalog repo not found at %s — skipping", repo) - return False - - alias_flags: list[str] = [] - for a in aliases: - alias_flags += ["-a", a] - - def _run(d: Path) -> bool: - r = _xorq(project, "add", "--no-sync", str(d), *alias_flags) - if r.returncode != 0: - # error, not warning: a failed add means the recipe never reached git - # and is silently lost (#48). The caller surfaces this on BuildResult. - log.error("xorq catalog add failed for %s: %s", d, r.stderr.strip()) - return False - return True - - if entry_name and entry_name != build_dir.name: - with tempfile.TemporaryDirectory() as tmp: - named = Path(tmp) / entry_name - shutil.copytree(build_dir, named) - return _run(named) - return _run(build_dir) - - -def add_alias(project: str, content_hash: str, alias: str) -> bool: - """Point *alias* at *content_hash* in the project's xorq catalog.""" - repo = _repo_path(project) - if not repo.exists(): - log.warning("xorq catalog repo not found at %s — skipping", repo) - return False - r = _xorq(project, "add-alias", content_hash, alias, "--no-sync") - if r.returncode != 0: - log.warning("xorq catalog add-alias failed: %s", r.stderr.strip()) - return False - return True - - -def remove_alias(project: str, alias: str) -> bool: - repo = _repo_path(project) - if not repo.exists(): - return False - r = _xorq(project, "remove-alias", alias, "--no-sync") - if r.returncode != 0: - log.warning("xorq catalog remove-alias failed: %s", r.stderr.strip()) - return False - return True diff --git a/src/tallyman_xorq/build.py b/src/tallyman_xorq/build.py index 3a5dbb4..0585272 100644 --- a/src/tallyman_xorq/build.py +++ b/src/tallyman_xorq/build.py @@ -4,6 +4,7 @@ import json import logging import re +import shutil import sys import tempfile import time @@ -60,9 +61,10 @@ class BuildResult: row_count: int execute_seconds: float schema: dict - # Whether the entry's recipe landed in the xorq catalog git repo. None on the - # re-run path (entry already on disk, registration not re-attempted); True/False - # on a fresh build. False means the recipe is NOT durable — see #48. + # Whether the complete entry dir was persisted to disk (D2). None on the + # re-run path (entry already on disk); True on a fresh build. Durability — + # committing the recipe zip — is the next checkpoint's job, which always + # succeeds because the dir is complete, so there is no False case anymore. catalog_registered: bool | None = None # Advisory build-time lints (#88): execution-nondeterministic ops whose result # varies run-to-run under one content_hash. Empty when the recipe is clean. @@ -437,6 +439,7 @@ def build_and_persist( cache_dir.mkdir(parents=True, exist_ok=True) loaded = load_expr(build_path, cache_dir=cache_dir) except Exception as exc: + shutil.rmtree(target, ignore_errors=True) # no orphan dir → no orphan pointer (D1) hint = _ibis_import_hint(str(exc), code) raise BuildError(f"load_expr failed: {exc}{hint}\n{traceback.format_exc()}") from exc @@ -451,6 +454,7 @@ def build_and_persist( arrow_schema = loaded.schema().to_pyarrow() row_count = int(loaded.count().execute()) except Exception as exc: + shutil.rmtree(target, ignore_errors=True) # no orphan dir → no orphan pointer (D1) hint = _ibis_import_hint(str(exc), code) raise BuildError(f"build execution failed: {exc}{hint}\n{traceback.format_exc()}") from exc execute_seconds = round(time.monotonic() - t0, 3) @@ -527,24 +531,12 @@ def build_and_persist( except OSError: pass - # Best-effort by design (a missing/misconfigured catalog repo must not break - # the build), but the outcome is surfaced on the result instead of swallowed: - # a failed add means the recipe never reached git and is not durable (#48). - catalog_registered: bool | None = None - cause = "" - try: - from tallyman_core.xorq_catalog import add_entry as _xcat_add - - catalog_registered = _xcat_add(project, xorq_build_dir, entry_name=content_hash) - except Exception as exc: - catalog_registered = False - cause = f" ({exc})" - if catalog_registered is False: - logging.getLogger(__name__).error( - "xorq catalog add failed for %s%s — entry recipe is not durable in the catalog repo (#48)", - content_hash, - cause, - ) + # The complete entry dir (recipe + manifest + schema) is now persisted on + # disk. catalog_registered reports that local fact — "entry dir persisted" — + # not durability: committing the content-addressed recipe zip is the next + # checkpoint's job, which always succeeds because the dir is complete (D2). + # No subprocess, no out-of-lock writer, no silent #48 no-op. + catalog_registered = True return BuildResult( content_hash=content_hash, diff --git a/tests/test_catalog_xorq_integration.py b/tests/test_catalog_xorq_integration.py deleted file mode 100644 index 5710e9d..0000000 --- a/tests/test_catalog_xorq_integration.py +++ /dev/null @@ -1,323 +0,0 @@ -"""catalog ↔ xorq coexistence across multi-entry adds and reset round-trips. - -Companion to ``test_reset_to_revision.py``. That suite drives synthetic -``entries/aaaa`` dirs and never runs xorq; its one real-xorq integration test -(``test_xorq_catalog_add_registers_after_genesis``) adds exactly *one* entry, so -it never crosses the checkpoint that commits the bookkeeping files into the -xorq catalog repo. That blind spot is how #48 shipped through all of V0. - -These tests reproduce the real demo flow (genesis → catalog_create → -catalog_create → reset_to) through the production MCP surface, so the -dispatch-boundary checkpoint wrapper fires exactly as it does in production — -the actual #48 trigger. Two distinct bugs were surfaced here: - -1. **#48** — the checkpoint commits tallyman's ``aliases.json``/ - ``alias_history.json`` files into the xorq catalog repo, ``assert_consistency`` - then fails, and every ``xorq catalog add`` after the first silently no-ops - (Group A, D). The fix carries that state as ``catalog.yaml`` keys instead, so - no extra files are tracked. -2. **reset ↔ xorq-catalog divergence** — a reset round-trips the tallyman - ``entries//`` dir through the bullpen, but the durable xorq ``.zip`` - recipe is reconciled by ``git reset`` and never regained. The two views of - "what entries exist" drift apart and ``reset_to`` never re-validates the - xorq catalog (Group B). - -Groups A–E originally landed as the failing-test half of a TDD split (the #48 -fix was a separate PR, now merged); they pass today and stand as the regression -guard. Group F characterizes a remaining follow-up (#56): the entry/alias -registration outcome is computed but not yet surfaced to the MCP caller — -``test_set_alias_swallows_failed_xorq_registration`` pins the current swallow -and flips when that signal lands. See ``plans/catalog-xorq-integration-tests.md``. - -Like the reset suite, these shell out to git directly to assert on the catalog -repo — the sanctioned exception to the no-bare-subprocess-git guard. -""" - -from __future__ import annotations - -import subprocess - -import pytest - -from tallyman_core import aliases as al -from tallyman_core import catalog_state as cs -from tallyman_core import paths -from tallyman_core import xorq_catalog as xc -from tallyman_xorq import build_and_persist - -# Distinct aggregations → distinct exprs → distinct xorq build hashes, so each -# entry is a genuinely separate catalog add (not a content-hash re-run no-op). -_AGGS = [ - "total=t.price.sum(), n=t.count()", - "avg=t.price.mean()", - "mx=t.price.max()", - "mn=t.price.min()", -] - - -def _agg_code(parquet_path, agg: str) -> str: - return ( - "import xorq.api as xo\n" - f"t = xo.deferred_read_parquet({str(parquet_path)!r})\n" - f"expr = t.group_by('region').aggregate({agg})\n" - ) - - -def _git(project: str, *args: str) -> str: - cd = paths.catalog_dir(project) - return subprocess.run(["git", "-C", str(cd), *args], capture_output=True, text=True).stdout.strip() - - -def _tracked_entry_zips(project: str) -> set[str]: - """Bare hashes of the git-tracked ``entries/.zip`` recipes. - - The durable, content-addressed xorq artifact set — what survives a clone or - a ``git reset``, as opposed to the untracked ``entries//`` working - dirs the tallyman bullpen reconciles. - """ - out = _git(project, "ls-files", "entries") - return {line[len("entries/") : -len(".zip")] for line in out.split() if line.endswith(".zip")} - - -def _tracked_alias_links(project: str) -> set[str]: - """Git-tracked alias symlinks under ``aliases/`` in the xorq catalog repo. - - The durable record of an alias: what survives a clone or a ``git reset``. - tallyman's own bookkeeping lives in ``catalog.yaml`` (#48) and is *not* - this — these are xorq's symlinks, written only by a successful - ``xorq catalog add-alias``.""" - return set(_git(project, "ls-files", "aliases").split()) - - -def _open_xorq_catalog(project: str): - """Open the project's xorq catalog with consistency checking on. - - ``from_kwargs`` defaults ``check_consistency=True``, so this runs - ``assert_consistency`` — the exact check #48 breaks. Returns the catalog or - re-raises, letting a test assert it opens. - """ - from xorq.catalog.catalog import Catalog - - return Catalog.from_kwargs(path=paths.catalog_dir(project), init=False) - - -def _make_entries(project: str, orders_parquet, n: int) -> list[str]: - """Drive ``catalog_create`` n times through the real MCP tool (so the - checkpoint wrapper fires), returning the content hashes in order.""" - from tallyman_mcp.server import catalog_create - - cs.genesis(project) # branch-pinned, xorq-keyed catalog.yaml baseline - hashes: list[str] = [] - for i in range(n): - res = catalog_create(name=f"agg{i}", code=_agg_code(orders_parquet, _AGGS[i])) - assert "error" not in res, f"catalog_create #{i} errored: {res}" - hashes.append(res["hash"]) - return hashes - - -# --------------------------------------------------------------------------- -# Group A — multi-entry catalog add (#48 core) -# --------------------------------------------------------------------------- - - -@pytest.mark.integration -def test_every_entry_registers_in_xorq_catalog(project, orders_parquet): - """Every entry's recipe must land in the xorq catalog, not just the first. - - Pre-fix only entry 1's ``.zip`` was tracked: entry 1's checkpoint committed - the bookkeeping files, ``assert_consistency`` then failed, and entries 2+ - add silently no-op'd. Guards against a regression.""" - hashes = _make_entries(project, orders_parquet, 3) - tracked = _tracked_entry_zips(project) - missing = [h for h in hashes if h not in tracked] - assert not missing, f"entries never registered in xorq catalog: {missing} (tracked: {tracked})" - - -@pytest.mark.integration -def test_catalog_openable_after_checkpoint(project, orders_parquet): - """The xorq catalog must stay openable after a checkpoint. Pins the - ``assert_consistency`` root cause: genesis + one ``catalog_create`` (which - checkpoints) left it unopenable pre-fix.""" - _make_entries(project, orders_parquet, 1) - try: - _open_xorq_catalog(project) - except AssertionError as exc: - pytest.fail(f"xorq catalog unopenable after checkpoint (#48 assert_consistency): {exc!r}") - - -@pytest.mark.integration -def test_bookkeeping_files_not_tracked_in_xorq_repo(project, orders_parquet): - """The precise #48 invariant: tallyman's alias bookkeeping must not be - git-tracked as separate files inside the xorq catalog repo. The checkpoint's - ``_TRACKED`` set committed both pre-fix, which is what broke - ``assert_consistency``. The fix (carry alias state as ``catalog.yaml`` keys - instead of ``aliases.json``/``alias_history.json`` files) makes this pass.""" - _make_entries(project, orders_parquet, 1) - tracked = set(_git(project, "ls-files").split()) - offenders = tracked & {"aliases.json", "alias_history.json"} - assert not offenders, f"tallyman bookkeeping tracked in xorq catalog repo: {sorted(offenders)}" - - -# --------------------------------------------------------------------------- -# Group B — reset round-trips keep the xorq catalog consistent (second bug) -# --------------------------------------------------------------------------- - - -@pytest.mark.integration -def test_reset_keeps_xorq_catalog_consistent(project, orders_parquet): - """A reset must leave the xorq catalog openable. Pre-fix the reset target - (step-1) was itself checkpointed with the bookkeeping files tracked, so - ``git reset --hard`` restored an inconsistent tree and ``reset_to`` never - re-validated it.""" - _make_entries(project, orders_parquet, 2) - cs.reset_to(project, 1) - try: - _open_xorq_catalog(project) - except AssertionError as exc: - pytest.fail(f"xorq catalog unopenable after reset_to(step-1): {exc!r}") - - -@pytest.mark.integration -def test_xorq_entry_set_matches_tallyman_pointers(project, orders_parquet): - """The durable xorq recipe set must match tallyman's ``entry_hashes`` - pointers. Pre-fix they diverged — 1 tracked ``.zip`` against 2 pointers — - the visible signature of "git-backed, recompute from history" being false - for every entry after the first.""" - _make_entries(project, orders_parquet, 2) - pointers = set(cs.read_tallyman_state(project)["entry_hashes"]) - tracked = _tracked_entry_zips(project) - assert tracked == pointers, f"xorq tracked zips {tracked} diverge from tallyman pointers {pointers}" - - -@pytest.mark.integration -def test_back_then_forward_restores_xorq_recipe(project, orders_parquet): - """Reset back then forward must restore entry 2's *durable* xorq recipe, - not merely its bullpen dir copy. Pre-fix the ``.zip`` never existed (its add - no-op'd), so a forward reset could not regain it — the divergence pinned end - to end.""" - hashes = _make_entries(project, orders_parquet, 2) - cs.reset_to(project, 1) # back to entry-1-only - cs.reset_to(project, 2) # forward to both - entry2 = hashes[1] - assert entry2 in _tracked_entry_zips(project), ( - f"entry 2 recipe {entry2} not durable in xorq catalog after back→forward reset" - ) - - -# --------------------------------------------------------------------------- -# Group D — silent-swallow surfacing (#48 secondary) -# --------------------------------------------------------------------------- - - -@pytest.mark.integration -def test_failed_registration_is_observable(project, orders_parquet): - """A failed xorq catalog add must be observable, not reported as success. - - Pre-fix this failure happened on its own (entry 1's checkpoint committed the - bookkeeping files, leaving the catalog inconsistent so the next add fails) — - that was bug #48 itself, and ``add_entry`` swallowed the failure with no - signal. The fix removes that spontaneous failure, so to test the *signal* we - inject the same kind of inconsistency a stray tracked file gives, then assert - ``build_and_persist`` reports ``catalog_registered is False`` rather than - claiming success.""" - _make_entries(project, orders_parquet, 1) - - # A tracked file outside xorq's managed set is exactly the shape #48 created: - # assert_consistency then rejects the repo and the next add fails. - cd = paths.catalog_dir(project) - (cd / "stray.txt").write_text("not xorq-managed\n") - _git(project, "add", "stray.txt") - subprocess.run( - ["git", "-C", str(cd), "-c", "user.email=t@t", "-c", "user.name=t", "commit", "-q", "-m", "stray"], - capture_output=True, - text=True, - ) - - result = build_and_persist(project, _agg_code(orders_parquet, _AGGS[1])) - assert result.catalog_registered is False, ( - "failed xorq catalog add must be observable: build_and_persist should expose " - f"catalog_registered=False, got {result.catalog_registered!r}" - ) - - -# --------------------------------------------------------------------------- -# Group E — reset rolls back alias state (regression guard shipping WITH the fix) -# --------------------------------------------------------------------------- - - -@pytest.mark.integration -def test_reset_rolls_back_alias_state(project, orders_parquet): - """A reset must roll alias bookkeeping back. - - The #48 fix carries alias state as ``catalog.yaml`` keys, so ``reset_to``'s - ``git reset --hard`` rolls it back with the rest of the file and - ``load_aliases`` (which reads catalog.yaml) reflects the reset target — no - separate reconcile. This guards that round-trip.""" - _make_entries(project, orders_parquet, 2) # creates aliases agg0, agg1 - assert set(al.load_aliases(project)) == {"agg0", "agg1"} - cs.reset_to(project, 1) # back to the one-entry step - assert set(al.load_aliases(project)) == {"agg0"}, ( - f"reset_to(step-1) did not roll alias state back: {al.load_aliases(project)}" - ) - - -# --------------------------------------------------------------------------- -# Group F — alias registration durability (the #48 follow-up) -# -# build_and_persist now surfaces entry-add durability via catalog_registered. -# The *alias* add (`xorq catalog add-alias`, driven by set_alias) is a separate -# call with its own failure modes, and its outcome is still swallowed. These -# exercise that boundary: the xorq-side contract, the happy path the fix -# restores, and the unsurfaced-failure gap. -# --------------------------------------------------------------------------- - - -@pytest.mark.integration -def test_add_alias_returns_false_for_unregistered_entry(project, orders_parquet): - """xorq refuses to alias an entry it doesn't know. - - ``add-alias`` constructs ``CatalogEntry(name, require_exists=True)``, whose - ``__attrs_post_init__`` asserts the entry exists (xorq catalog.py:1214). For - an unregistered hash the CLI exits non-zero, so ``_xcat.add_alias`` returns - ``False``. This is exactly the signal ``set_alias`` discards — and the shape - a failed ``add_entry`` leaves behind (the alias of a recipe that never - reached git).""" - cs.genesis(project) - assert xc.add_alias(project, "deadbeefdeadbeef", "ghost") is False - - -@pytest.mark.integration -def test_registered_entry_gets_durable_alias_symlink(project, orders_parquet): - """The happy path the #48 fix restores end to end. - - A real ``catalog_create`` registers the entry, so the following - ``add-alias`` succeeds and xorq writes a git-tracked ``aliases/`` - symlink — the durable record, distinct from tallyman's own bookkeeping in - ``catalog.yaml``.""" - _make_entries(project, orders_parquet, 1) # catalog_create('agg0') -> set_alias - links = _tracked_alias_links(project) - assert any("agg0" in link for link in links), f"alias symlink not durable in xorq repo: {links}" - - -@pytest.mark.integration -def test_set_alias_swallows_failed_xorq_registration(project, orders_parquet): - """``set_alias`` reports success even when the alias never reaches xorq. - - Aliasing a hash xorq doesn't know makes ``_xcat.add_alias`` return ``False``, - but ``set_alias`` discards that bool: it writes its ``catalog.yaml`` - bookkeeping and returns version info regardless. Unlike - ``build_and_persist``'s ``catalog_registered``, nothing surfaces the failed - alias registration. This pins that gap (characterization — flip the final - assertion when the signal lands).""" - cs.genesis(project) - ghost = "deadbeefdeadbeef" - - info = al.set_alias(project, "ghost", ghost, expect_exists=False) - - # set_alias claims success and writes its own bookkeeping... - assert info == {"name": "ghost", "hash": ghost, "version": 1} - assert al.load_aliases(project) == {"ghost": ghost} - # ...but the alias never reached the xorq catalog repo, and no signal said so. - assert not _tracked_alias_links(project), ( - "alias reported as set but not durable in the xorq catalog — set_alias swallowed the failed registration" - ) diff --git a/tests/test_native_catalog_store.py b/tests/test_native_catalog_store.py index 7f5ebfb..1d63201 100644 --- a/tests/test_native_catalog_store.py +++ b/tests/test_native_catalog_store.py @@ -1,21 +1,28 @@ """Native catalog store (drop xorq's catalog package) — the Cut-A/Cut-B suite. See plans/native-catalog-store.md. These pin the contract the native -``tallyman_core.catalog`` store must satisfy: the build/alias paths invoke no -xorq ``catalog`` subprocess, the checkpoint (not the build) owns the durable -recipe zip, and a failed build leaves no orphan pointer. +``tallyman_core.catalog`` store satisfies: the build/alias paths invoke no xorq +``catalog`` subprocess, the checkpoint (not the build) owns the deterministic, +allowlisted recipe zip, a failed build leaves no orphan pointer, and the +tracked surface is a path allowlist (deny-by-default). Carries no ``integration``/``cache_lab``/``perf`` marker, so a bare -``uv run pytest`` (CI default, pyproject.toml:77) exercises them. +``uv run pytest`` (CI default, pyproject.toml:77) exercises them. The durability +guards here used to live in the xorq-specific ``test_catalog_xorq_integration.py`` +(deleted with the subprocess) as slow ``@integration`` tests; they are now +in-process and fast. """ from __future__ import annotations import subprocess +import zipfile from pathlib import Path +import pytest + +from tallyman_core import catalog, paths from tallyman_core import catalog_state as cs -from tallyman_core import paths from tallyman_xorq import build_and_persist @@ -27,9 +34,25 @@ def _agg_code(parquet_path: Path) -> str: ) +def _agg_code_n(parquet_path: Path, agg: str) -> str: + return ( + "import xorq.api as xo\n" + f"t = xo.deferred_read_parquet({str(parquet_path)!r})\n" + f"expr = t.group_by('region').aggregate({agg})\n" + ) + + +_AGGS = ["total=t.price.sum()", "avg=t.price.mean()", "mx=t.price.max()", "mn=t.price.min()"] + + def _tracked_zip_hashes(project: str) -> set[str]: - rc, out, _ = cs.run_git(["ls-files", "entries"], cwd=paths.catalog_dir(project)) # type: ignore[attr-defined] - return {ln[len("entries/") : -len(".zip")] for ln in out.split() if ln.endswith(".zip")} + return catalog.tracked_recipe_hashes(project) + + +def _build_checkpoint(project: str, code: str, message: str) -> str: + res = build_and_persist(project, code) + cs.checkpoint_catalog(project, message) + return res.content_hash class _XorqSpy: @@ -56,15 +79,10 @@ def __call__(self, argv, *a, **k): def test_uses_no_subprocess_alias(project, monkeypatch): - """set_alias must not shell out to ``xorq catalog add-alias``. - - Today aliases.py:129 calls the xorq subprocess with no try/except, so a - raising ``subprocess.run`` propagates out of set_alias. Native aliases write - a tracked ``aliases.jsonl`` and never spawn xorq, so this must not raise. - """ + """set_alias must not shell out to ``xorq catalog add-alias``.""" from tallyman_core import aliases as al - cs.genesis(project) # the catalog repo must exist or add_alias returns early + cs.genesis(project) def _boom(*a, **k): raise AssertionError("set_alias spawned a subprocess (xorq catalog add-alias)") @@ -74,37 +92,126 @@ def _boom(*a, **k): def test_uses_no_subprocess_build(project, orders_parquet, monkeypatch): - """A full build must invoke zero ``xorq catalog`` subprocesses. - - Asserts a call-count spy of zero, NOT "did not raise": build.py:539-541 - swallows a raising registration into ``catalog_registered=False``, so a - raising stub would false-green. - """ - cs.genesis(project) # so add_entry reaches the subprocess rather than returning early + """A full build invokes zero ``xorq catalog`` subprocesses (call-count spy, + not "did not raise" — the swallowed-registration path would false-green).""" + cs.genesis(project) spy = _XorqSpy(subprocess.run) monkeypatch.setattr(subprocess, "run", spy) build_and_persist(project, _agg_code(orders_parquet)) assert spy.xorq_calls == 0, f"build spawned {spy.xorq_calls} xorq catalog subprocess(es)" +def test_src_has_no_xorq_catalog_argv(): + """No src/ module imports ``xorq.catalog`` or builds a ``xorq catalog`` argv. + + A static guard (the macOS-only fork crash and the catalog package both evade + a runtime check). Matches ``import/from xorq.catalog`` and the split-token + ``"xorq", "catalog"`` argv (so a literal ``"xorq catalog"`` search misses + it). Modelled on ``test_no_bare_subprocess_git_in_src``. + """ + import re + + src = Path(__file__).resolve().parent.parent / "src" + imp = re.compile(r"(?:from|import)\s+xorq\.catalog\b") + argv = re.compile(r"""["']xorq["']\s*,\s*["']catalog["']""") + offenders = [] + for p in src.rglob("*.py"): + text = p.read_text() + offenders += [f"{p.relative_to(src)} (import)" for _ in imp.finditer(text)] + offenders += [f"{p.relative_to(src)} (argv)" for _ in argv.finditer(text)] + # The MCP codegen docstring (server.py) teaches `import xorq.api as xo` to + # recipe authors — not a catalog import, not the argv — so it never matches. + assert not offenders, f"xorq.catalog import/argv in src/: {offenders}" + + # --------------------------------------------------------------------------- -# Cut A — the checkpoint owns the durable recipe zip (D1) +# Cut A — the recipe zip: allowlist, determinism, expr.py durability # --------------------------------------------------------------------------- -def test_zip_staged_not_committed_until_checkpoint(project, orders_parquet): - """The durable ``entries/.zip`` is written by the checkpoint, not the - build. Today xorq's ``catalog add`` commits it inline (outside the flock), - so it is tracked before any tallyman checkpoint — D1 moves that to the - checkpoint under the project lock. +def test_recipe_zip_members_are_allowlist(project, orders_parquet, tmp_path): + """write_recipe_zip archives exactly the allowlist — never a stray. + + Allowlist-not-denylist: a denylist skipping only today's known strays would + pass existence checks yet leak a future artifact (here ``foo.bin``). """ - cs.genesis(project) + res = build_and_persist(project, _agg_code(orders_parquet), prompt="seed") + ed = paths.entry_dir(project, res.content_hash) + # strays that must NOT leak into the durable zip + (ed / "result.parquet").write_bytes(b"heavy") + (ed / "foo.bin").write_bytes(b"\x00") + (ed / ".xorq_build_expanded").mkdir(exist_ok=True) + (ed / ".xorq_build_expanded" / "x.yaml").write_text("x") + assert (ed / "prompts.jsonl").exists() # append-mutable provenance — also excluded + + dest = tmp_path / "recipe.zip" + catalog.write_recipe_zip(ed, dest, res.content_hash) + names = set(zipfile.ZipFile(dest).namelist()) + prefix = f"{res.content_hash}/" + assert names and all(n.startswith(prefix) for n in names), "every member under one / prefix" + rels = {n[len(prefix) :] for n in names} + top = {r for r in rels if "/" not in r} + tree = {r for r in rels if "/" in r} + assert top == {"expr.py", "schema.json", "manifest.json"}, top + assert all(r.startswith("xorq_build/") for r in tree), tree + assert not any(s in r for r in rels for s in ("foo.bin", "result.parquet", "prompts.jsonl", "xorq_build_expanded")) + + +def test_recipe_zip_byte_stable_across_rebuilds(project, orders_parquet, tmp_path): + """The archive is byte-identical across rebuilds — a defaulted ZipInfo mtime + would embed the wall clock and silently re-churn the content-addressed git + blob on every rebuild. Bumping file mtimes between the two zips is what a + naive (file-mtime) writer leaks and a deterministic one absorbs.""" + import os + import time + res = build_and_persist(project, _agg_code(orders_parquet)) - assert res.content_hash not in _tracked_zip_hashes(project), ( - "recipe zip was committed at build time; D1 requires the checkpoint to own it" + ed = paths.entry_dir(project, res.content_hash) + z1 = tmp_path / "a.zip" + catalog.write_recipe_zip(ed, z1, res.content_hash) + + bumped = time.time() + 10_000 + for p in ed.rglob("*"): + if p.is_file(): + os.utime(p, (bumped, bumped)) + z2 = tmp_path / "b.zip" + catalog.write_recipe_zip(ed, z2, res.content_hash) + + assert z1.read_bytes() == z2.read_bytes() + + +def test_recipe_zip_contains_expr_py(project, orders_parquet, tmp_path): + """expr.py is a durable zip member (D3): the from_catalog chaining path + re-execs it so a cheap entry's recompute roots on the shared in-process + backend, and two entries union/join on one backend (#75).""" + res = build_and_persist(project, _agg_code(orders_parquet)) + ed = paths.entry_dir(project, res.content_hash) + dest = tmp_path / "recipe.zip" + catalog.write_recipe_zip(ed, dest, res.content_hash) + assert f"{res.content_hash}/expr.py" in zipfile.ZipFile(dest).namelist() + + +def test_salt_zip_named_by_salted_hash(project, monkeypatch): + """Under source-identity salt mode the durable zip is named by the salted + content_hash (= the entry dir name), not xorq's path-identity build hash, so + the tracked zip set and the entry pointers don't diverge (Risk #5).""" + monkeypatch.setenv("TALLYMAN_SOURCE_IDENTITY", "salt") + from tallyman_cli.fixtures import write_shoe_orders + + write_shoe_orders(paths.data_dir(project) / "orders.parquet", n_rows=120, seed=1) + code = ( + "from tallyman_xorq.io import from_project\n" + "t = from_project('orders.parquet')\n" + "expr = t.group_by('region').aggregate(n=t.count())\n" ) + cs.genesis(project) + res = build_and_persist(project, code) + import json + + manifest = json.loads((paths.entry_dir(project, res.content_hash) / "manifest.json").read_text()) + assert manifest.get("sources"), "salt mode should have captured a source digest" cs.checkpoint_catalog(project, "create") - assert res.content_hash in _tracked_zip_hashes(project), "checkpoint did not commit the recipe zip" + assert res.content_hash in _tracked_zip_hashes(project) def test_failed_build_no_orphan_pointer(project, orders_parquet, monkeypatch): @@ -114,18 +221,90 @@ def test_failed_build_no_orphan_pointer(project, orders_parquet, monkeypatch): import xorq.ibis_yaml.compiler as compiler cs.genesis(project) - real_load = compiler.load_expr def _boom_load(*a, **k): raise RuntimeError("induced load failure after target.mkdir") monkeypatch.setattr(compiler, "load_expr", _boom_load) - try: + with pytest.raises(Exception): build_and_persist(project, _agg_code(orders_parquet)) - except Exception: - pass - monkeypatch.setattr(compiler, "load_expr", real_load) + monkeypatch.undo() ed = paths.entries_dir(project) leftover = [c.name for c in ed.iterdir() if c.is_dir()] if ed.exists() else [] assert not leftover, f"failed build left orphan entry dir(s): {leftover}" + + +# --------------------------------------------------------------------------- +# Cut A — the checkpoint owns zipping; the tracked surface is an allowlist +# --------------------------------------------------------------------------- + + +def test_zip_staged_not_committed_until_checkpoint(project, orders_parquet): + """The durable ``entries/.zip`` is written by the checkpoint, not the + build (D1).""" + cs.genesis(project) + res = build_and_persist(project, _agg_code(orders_parquet)) + assert res.content_hash not in _tracked_zip_hashes(project), "zip must not exist before a checkpoint" + cs.checkpoint_catalog(project, "create") + assert res.content_hash in _tracked_zip_hashes(project), "checkpoint did not commit the recipe zip" + + +def test_n_builds_track_n_zips(project, orders_parquet): + """N distinct builds, each checkpointed, leave N durable tracked zips (the + #48 failure mode was N build dirs but fewer tracked recipes).""" + cs.genesis(project) + hashes = [_build_checkpoint(project, _agg_code_n(orders_parquet, _AGGS[i]), f"c{i}") for i in range(4)] + tracked = _tracked_zip_hashes(project) + missing = [h for h in hashes if h not in tracked] + assert not missing, f"entries with no durable tracked zip: {missing} (tracked: {tracked})" + + +def test_build_dir_files_not_staged(project, orders_parquet): + """The untracked build dir (``entries//xorq_build/...``) is never + git-tracked under ``git add -A`` — the .gitignore + allowlist keep it out + (Risk #1).""" + cs.genesis(project) + build_and_persist(project, _agg_code(orders_parquet)) + cs.checkpoint_catalog(project, "create") + rc, out, _ = cs.run_git(["ls-files"], cwd=paths.catalog_dir(project)) + staged = [p for p in out.split() if "/xorq_build/" in p or p.endswith("/expr.py") or p.endswith("/manifest.json")] + assert not staged, f"build-dir files were staged: {staged}" + + +def test_unlisted_path_not_tracked(project, orders_parquet): + """A tracked path outside the allowed surface fails the consistency check — + the durable guard against ``git add -A`` silently committing a stray.""" + cs.genesis(project) + res = build_and_persist(project, _agg_code(orders_parquet)) + cs.checkpoint_catalog(project, "create") + cd = paths.catalog_dir(project) + (cd / "stray.txt").write_text("not part of the tracked surface\n") + cs.run_git(["add", "stray.txt"], cwd=cd) + with pytest.raises(RuntimeError, match="outside the allowed surface"): + catalog.assert_catalog_consistent(project, {res.content_hash}) + + +# --------------------------------------------------------------------------- +# Cut A — reset keeps the durable recipe set and the pointers in agreement +# --------------------------------------------------------------------------- + + +def test_reset_keeps_pointer_zip_agreement(project, orders_parquet): + """A reset to an earlier step leaves the tracked recipe set equal to the + entry_hashes pointers (the consistency check passes, #52).""" + cs.genesis(project) + hashes = [_build_checkpoint(project, _agg_code_n(orders_parquet, _AGGS[i]), f"c{i}") for i in range(2)] + cs.reset_to(project, 1) # back to the one-entry step + pointers = set(cs.read_tallyman_state(project)["entry_hashes"]) + assert pointers == _tracked_zip_hashes(project) == {hashes[0]} + + +def test_back_then_forward_restores_recipe(project, orders_parquet): + """Reset back then forward restores entry 2's durable recipe zip, not merely + its bullpen dir copy.""" + cs.genesis(project) + hashes = [_build_checkpoint(project, _agg_code_n(orders_parquet, _AGGS[i]), f"c{i}") for i in range(2)] + cs.reset_to(project, 1) + cs.reset_to(project, 2) + assert hashes[1] in _tracked_zip_hashes(project) From ec0320db1bcae04a954392889939e6459d4d7d02 Mon Sep 17 00:00:00 2001 From: Paddy Mullen Date: Tue, 16 Jun 2026 22:27:00 -0400 Subject: [PATCH 13/27] refactor: decompose catalog.yaml into tracked files (native store, #52) MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit Every piece of mutable catalog state is now its own git-tracked file the native store allows, written by its own mutator and rolled back by reset_to's `git reset --hard` directly — no catalog.yaml round-trip. - aliases.py: alias map/history move from catalog.yaml keys to a tracked aliases.jsonl (one line per alias: {alias, latest, history}); aliases.py no longer reaches into catalog_state. - notebook.py: from a single project_dir/notebooks/default.json object to a tracked catalog/notebook.jsonl (one cell per line). - paths.py: post_processing/ and stats/ move under the catalog repo (were artifacts/ siblings); add prompts_dir/prompts_path; drop the vestigial notebooks/ from the project layout. - build.py/app.py: per-entry prompt history moves from a loose (now gitignored) entries//prompts.jsonl to a tracked prompts/.jsonl, so the re-run history survives a reset/clone. read_prompts/_append_prompt take (project, content_hash). - catalog_state.py: catalog.yaml + capture/materialize of the six sections are gone. capture/read/write_tallyman_state shrink to the two pointer files (entries.jsonl + compute_cache.jsonl) for the untracked artifacts the bullpen reconciles; prune/restore read those; reset_to drops the materialize step. - catalog.py: drop catalog.yaml from the tracked-surface allowlist. Tests: delete the obsolete capture/materialize and catalog.yaml-coexistence tests (machinery removed); update read_prompts call sites and the path-layout assertions; add test_prompt_history_survives_reset + test_display_config_survives_reset so net reset coverage doesn't regress. Full fast suite: 506 passed. Co-Authored-By: Claude Opus 4.8 (1M context) --- src/tallyman_companion/app.py | 2 +- src/tallyman_core/aliases.py | 50 +++-- src/tallyman_core/catalog.py | 1 - src/tallyman_core/catalog_state.py | 306 ++++++++++------------------- src/tallyman_core/notebook.py | 37 ++-- src/tallyman_core/paths.py | 18 +- src/tallyman_xorq/build.py | 25 ++- tests/test_display_configs.py | 38 +--- tests/test_native_catalog_store.py | 49 ++++- tests/test_pack.py | 5 +- tests/test_paths.py | 16 +- tests/test_project_switcher.py | 1 - tests/test_prompts.py | 6 +- tests/test_reset_to_revision.py | 101 ++-------- 14 files changed, 267 insertions(+), 388 deletions(-) diff --git a/src/tallyman_companion/app.py b/src/tallyman_companion/app.py index 4501dce..33966f7 100644 --- a/src/tallyman_companion/app.py +++ b/src/tallyman_companion/app.py @@ -569,7 +569,7 @@ def api_entry_detail(project: str, content_hash: str): for i, h in enumerate(hist_hashes, start=1): forensic_history.append({"hash": h, "version": i, "is_current": h == content_hash}) - prompt_history = read_prompts(entry) + prompt_history = read_prompts(project, content_hash) chart_spec = get_chart(project, content_hash) build_artifacts: list[dict] = [] diff --git a/src/tallyman_core/aliases.py b/src/tallyman_core/aliases.py index 54765bf..c9baa94 100644 --- a/src/tallyman_core/aliases.py +++ b/src/tallyman_core/aliases.py @@ -21,7 +21,9 @@ from __future__ import annotations -from tallyman_core.paths import ensure_project +import json + +from tallyman_core.paths import catalog_dir, ensure_project class AliasExists(ValueError): @@ -32,24 +34,46 @@ class AliasNotFound(KeyError): pass +def _aliases_file(project: str): + return catalog_dir(project) / "aliases.jsonl" + + def _read(project: str) -> tuple[dict[str, str], dict[str, list[str]]]: - """Read the alias map + history out of catalog.yaml. + """Read the alias map + history out of the tracked ``aliases.jsonl``. - Lazy import: ``aliases`` is imported very early in ``tallyman_core``'s - package init (before ``catalog_state``'s transitive deps), so resolving - ``catalog_state`` at call time keeps that ordering clean. + One line per alias: ``{"alias", "latest", "history": [...]}``. The native + store tracks this file directly, so ``reset_to``'s ``git reset`` rolls alias + state back with the rest of the tree (no separate reconcile). """ - from tallyman_core import catalog_state as cs - - state = cs.read_tallyman_state(project) - return state["alias_map"], state["alias_history"] + p = _aliases_file(project) + alias_map: dict[str, str] = {} + history: dict[str, list[str]] = {} + if p.exists(): + for line in p.read_text().splitlines(): + if not line.strip(): + continue + rec = json.loads(line) + alias_map[rec["alias"]] = rec["latest"] + history[rec["alias"]] = rec.get("history", []) + return alias_map, history def _write(project: str, aliases: dict[str, str], history: dict[str, list[str]]) -> None: - """Persist the alias map + history into catalog.yaml (preserving xorq's keys).""" - from tallyman_core import catalog_state as cs - - cs.write_tallyman_state(project, alias_map=aliases, alias_history=history) + """Persist the alias map + history to ``aliases.jsonl`` (sorted for a stable + diff; an empty map removes the file).""" + ensure_project(project) + p = _aliases_file(project) + p.parent.mkdir(parents=True, exist_ok=True) + if not aliases: + p.unlink(missing_ok=True) + return + body = "".join( + json.dumps({"alias": name, "latest": aliases[name], "history": history.get(name, [])}) + "\n" + for name in sorted(aliases) + ) + tmp = p.with_name(p.name + ".tmp") + tmp.write_text(body) + tmp.replace(p) def load_aliases(project: str) -> dict[str, str]: diff --git a/src/tallyman_core/catalog.py b/src/tallyman_core/catalog.py index 293f796..3a36be1 100644 --- a/src/tallyman_core/catalog.py +++ b/src/tallyman_core/catalog.py @@ -48,7 +48,6 @@ # allowlist is the durable guard, since ``git add -A`` is allow-by-default. TRACKED_SURFACE = ( ".gitignore", - "catalog.yaml", "entries/*.zip", "chart_specs/*.vl.json", "display_configs/*.json", diff --git a/src/tallyman_core/catalog_state.py b/src/tallyman_core/catalog_state.py index 5055f5c..cf6c51b 100644 --- a/src/tallyman_core/catalog_state.py +++ b/src/tallyman_core/catalog_state.py @@ -1,38 +1,30 @@ -"""Pydata-owned state in the xorq catalog's catalog.yaml + the reset engine. - -catalog.yaml is the versioned state carrier (see plans/adr-reset-to-revision.md). -xorq owns ``entries`` and ``aliases`` in that file; tallyman owns: - - post_processing: [{name, source, disabled}] - stats: [{name, source, disabled}] - charts: [{content_hash, spec}] - display_configs: [{content_hash, config}] - notebook: {cells: [{cell_id, alias, markdown}]} - entry_hashes: [content_hash, ...] # pointer to untracked entries// - compute_cache: [relpath, ...] # pointer to untracked compute cache - alias_map: {alias: latest_hash} # tallyman's alias bookkeeping - alias_history: {alias: [hash, ...]} # tallyman's alias bookkeeping - -The catalog git repo tracks only ``catalog.yaml``. tallyman's alias bookkeeping -lives *in* it, as the ``alias_map``/``alias_history`` keys above (owned by -``aliases.py``) — committing separate ``aliases.json``/``alias_history.json`` -files inside the repo made xorq's ``assert_consistency`` reject it and every -``xorq catalog add`` after the first silently no-op (#48). Keeping the state as -catalog.yaml keys means there are no extra tracked files, and ``reset_to``'s -``git reset`` rolls alias state back with the rest of the file (no separate -reconcile). The heavy artifacts (entries/, the caches) are content-addressed, -additive, and -untracked — ``git reset`` can't roll them back, so ``reset_to`` reconciles them -to the recorded pointer lists: evictions retire to the bullpen (not deleted), -and anything the restored step records that is missing comes back from the -bullpen by copy. Live operations never read the bullpen, so a re-added -expression still computes cold. The derived files (.py scripts, chart specs, -the notebook) live outside the repo, so they are reconstructed by -``materialize``. - -A checkpoint captures the live on-disk state into catalog.yaml and commits once, -through the fork-safe git primitive (git_util) under a per-project lock. This -keeps the four invariants #33 kept re-breaking structural rather than reviewed. +"""The native catalog's versioning engine (see plans/adr-reset-to-revision.md). + +Every piece of mutable catalog state is now its own git-tracked file the +native store (#52) allows (charts under ``chart_specs/``, display configs under +``display_configs/``, post-processing/stats ``.py`` under the repo, aliases in +``aliases.jsonl``, the notebook in ``notebook.jsonl``, per-entry prompt history +in ``prompts/.jsonl``). Their own mutators write them; ``reset_to``'s +``git reset --hard`` restores them all at once. There is no longer a +``catalog.yaml`` round-trip (no ``capture``/``materialize`` of those sections). + +What remains here is the *pointer* bookkeeping for the two **untracked** +artifact sets — the entry build dirs and the compute-cache warm set — in two +tracked JSONL files: + + entries.jsonl: {"hash": content_hash} # untracked entries// dirs + compute_cache.jsonl: {"path": relpath} # untracked compute-cache warm set + +Those heavy artifacts are content-addressed, additive, and gitignored, so +``git reset`` can't roll them back; ``reset_to`` reconciles them to the recorded +pointers via the bullpen — evictions retire (not deleted), and anything a +restored step records but is missing comes back by copy. Live operations never +read the bullpen, so a re-added expression still computes cold. + +A checkpoint captures the pointer lists, zips any pending recipe (catalog.py), +``git add -A``, and commits once — through the fork-safe git primitive +(git_util) under a per-project lock — keeping the four invariants #33 kept +re-breaking structural rather than reviewed. """ from __future__ import annotations @@ -46,12 +38,7 @@ import shutil from pathlib import Path -import yaml - -from tallyman_core import catalog, charts, notebook -from tallyman_core import display_configs as dc -from tallyman_core import post_processing as pp -from tallyman_core import summary_stats as ss +from tallyman_core import catalog from tallyman_core.git_util import run_git from tallyman_core.paths import ( ENTRIES_DIRNAME, @@ -59,9 +46,6 @@ catalog_dir, compute_cache_dir, entries_dir, - notebooks_dir, - post_processing_dir, - stats_dir, ) log = logging.getLogger(__name__) @@ -76,72 +60,63 @@ # --------------------------------------------------------------------------- -# catalog.yaml read / write (read-modify-write so xorq's keys survive) +# pointer bookkeeping: the two tracked JSONL files that replace catalog.yaml # --------------------------------------------------------------------------- -def _catalog_yaml(project: str) -> Path: - return catalog_dir(project) / "catalog.yaml" +def _entries_file(project: str) -> Path: + return catalog_dir(project) / "entries.jsonl" + +def _compute_cache_file(project: str) -> Path: + return catalog_dir(project) / "compute_cache.jsonl" -def _read_raw(project: str) -> dict: - p = _catalog_yaml(project) - if not p.exists(): - return {} - return yaml.safe_load(p.read_text()) or {} + +def _read_jsonl(path: Path, field: str) -> list[str] | None: + """The *field* of each line, or None when the file is absent — so callers + can tell "never recorded" (no-op) from a recorded-empty list (reconcile to + nothing).""" + if not path.exists(): + return None + return [json.loads(line)[field] for line in path.read_text().splitlines() if line.strip()] + + +def _write_jsonl(path: Path, field: str, values: list[str]) -> None: + path.parent.mkdir(parents=True, exist_ok=True) + path.write_text("".join(json.dumps({field: v}) + "\n" for v in values)) def read_tallyman_state(project: str) -> dict: - """The tallyman-owned keys, each defaulted so callers never KeyError.""" - raw = _read_raw(project) + """The pointer lists for the *untracked* artifacts the bullpen reconciles — + entry build dirs (``entry_hashes``) and the compute-cache warm set + (``compute_cache``). Each defaults to [] so callers never KeyError. + + The decomposed mutable sections (charts, display, post-processing, stats, + aliases, notebook) are tracked files now, restored by ``git reset`` directly, + so they are no longer carried here. + """ return { - "post_processing": raw.get("post_processing", []), - "stats": raw.get("stats", []), - "charts": raw.get("charts", []), - "display_configs": raw.get("display_configs", []), - "notebook": raw.get("notebook", {"cells": []}), - "entry_hashes": raw.get("entry_hashes", []), - "compute_cache": raw.get("compute_cache", []), - "alias_map": raw.get("alias_map", {}), - "alias_history": raw.get("alias_history", {}), + "entry_hashes": _read_jsonl(_entries_file(project), "hash") or [], + "compute_cache": _read_jsonl(_compute_cache_file(project), "path") or [], } -def write_tallyman_state(project: str, **keys) -> None: - """Merge tallyman keys into catalog.yaml atomically, preserving xorq's keys. - - xorq's keys are also *seeded* when absent: its ``CatalogYaml.contents`` - indexes ``entries``/``aliases`` directly (no defaulting for dict-shaped - files), so a catalog.yaml born from genesis/capture without them makes - every best-effort ``xorq catalog add`` fail — silently. xorq round-trips - unknown keys, so the two empty lists are the whole coexistence contract. - """ - p = _catalog_yaml(project) - p.parent.mkdir(parents=True, exist_ok=True) - raw = _read_raw(project) - raw.setdefault("entries", []) - raw.setdefault("aliases", []) - for k, v in keys.items(): - if v is not None: - raw[k] = v - tmp = p.with_name(p.name + ".tmp") - # safe_dump, to mirror _read_raw's safe_load: the full Dumper would write a - # stray non-plain value (a Path in a chart spec) as a !!python/object tag, - # after which every read fails until the file is hand-repaired. Raising - # here — before the tmp replace — keeps the previous file readable. - tmp.write_text(yaml.safe_dump(raw, default_flow_style=False, sort_keys=False)) - tmp.replace(p) +def write_tallyman_state( + project: str, *, entry_hashes: list[str] | None = None, compute_cache: list[str] | None = None +) -> None: + """Persist whichever pointer list is given to its tracked JSONL (the other + is left untouched).""" + if entry_hashes is not None: + _write_jsonl(_entries_file(project), "hash", entry_hashes) + if compute_cache is not None: + _write_jsonl(_compute_cache_file(project), "path", compute_cache) # --------------------------------------------------------------------------- -# capture: live on-disk state -> catalog.yaml keys +# capture: live untracked-artifact listings -> the two tracked pointer files # --------------------------------------------------------------------------- -def _scan_scripts(listing: list[dict]) -> list[dict]: - return [{"name": e["name"], "source": e["source"], "disabled": e["disabled"]} for e in listing] - - def _list_cache_files(root: Path) -> list[str]: if not root.exists(): return [] @@ -149,92 +124,14 @@ def _list_cache_files(root: Path) -> list[str]: def capture_tallyman_state(project: str) -> dict: - """Read the live authored state + artifact listings into catalog.yaml.""" + """Snapshot the pointer lists for the untracked artifacts into their tracked + JSONL files. The decomposed sections write their own tracked files, so + capture no longer touches charts/display/pp/stats/aliases/notebook.""" ed = entries_dir(project) - state = { - "post_processing": _scan_scripts(pp.list_post_processings(project)), - "stats": _scan_scripts(ss.list_stats(project)), - "charts": [{"content_hash": h, "spec": charts.get_chart(project, h)} for h in charts.list_charts(project)], - "display_configs": [ - {"content_hash": h, "config": dc.get_display_config(project, h)} for h in dc.list_display_configs(project) - ], - "notebook": notebook.load(project), - "entry_hashes": sorted(c.name for c in ed.iterdir() if c.is_dir()) if ed.exists() else [], - "compute_cache": _list_cache_files(compute_cache_dir(project)), - } - # alias_map/alias_history are written live by aliases.py and already in - # catalog.yaml; write_tallyman_state preserves keys it isn't given, so - # capture leaves them untouched rather than re-snapshotting. - write_tallyman_state(project, **state) - return state - - -# --------------------------------------------------------------------------- -# materialize: catalog.yaml keys -> on-disk derived files (non-destructive) -# --------------------------------------------------------------------------- - - -def _materialize_scripts(target: Path, entries: list[dict]) -> None: - if target.exists(): - shutil.rmtree(target) # removes _disabled/ with it - target.mkdir(parents=True, exist_ok=True) - for e in entries or []: - dest = (target / "_disabled") if e.get("disabled") else target - dest.mkdir(parents=True, exist_ok=True) - src = e.get("source", "") - (dest / f"{e['name']}.py").write_text(src if src.endswith("\n") else src + "\n") - - -def _materialize_charts(project: str, entries: list[dict]) -> None: - recorded = {c["content_hash"]: c["spec"] for c in entries or []} - for h in list(charts.list_charts(project)): - if h not in recorded: - charts.remove_chart(project, h) - for h, spec in recorded.items(): - charts.set_chart(project, h, spec) - - -def _materialize_display_configs(project: str, entries: list[dict]) -> None: - recorded = {c["content_hash"]: c["config"] for c in entries or []} - for h in list(dc.list_display_configs(project)): - if h not in recorded: - dc.remove_display_config(project, h) - for h, config in recorded.items(): - if config is not None: - dc.set_display_config(project, h, config) - - -def _materialize_notebook(project: str, nb: dict) -> None: - nbfile = notebooks_dir(project) / "default.json" - cells = (nb or {}).get("cells") or [] - if cells: - notebooks_dir(project).mkdir(parents=True, exist_ok=True) - nbfile.write_text(json.dumps({"cells": cells}, indent=2)) - else: - nbfile.unlink(missing_ok=True) - - -def materialize(project: str) -> None: - """Reconstruct on-disk derived files from catalog.yaml. - - A section is rebuilt only when its key is *present*: an absent key means - the project never recorded that state (e.g. predates these keys), so its - on-disk files are left alone rather than wiped. A present-but-empty list is - a deliberate "none at this step" and clears the section. - """ - raw = _read_raw(project) - if "post_processing" in raw: - _materialize_scripts(post_processing_dir(project), raw["post_processing"]) - if "stats" in raw: - _materialize_scripts(stats_dir(project), raw["stats"]) - if "charts" in raw: - _materialize_charts(project, raw["charts"]) - if "display_configs" in raw: - _materialize_display_configs(project, raw["display_configs"]) - if "notebook" in raw: - _materialize_notebook(project, raw["notebook"] or {"cells": []}) - # alias_map/alias_history need no materialize step: aliases.py reads them - # straight from catalog.yaml, which git reset has already rolled back. + entry_hashes = sorted(c.name for c in ed.iterdir() if c.is_dir()) if ed.exists() else [] + compute_cache = _list_cache_files(compute_cache_dir(project)) + write_tallyman_state(project, entry_hashes=entry_hashes, compute_cache=compute_cache) + return {"entry_hashes": entry_hashes, "compute_cache": compute_cache} # --------------------------------------------------------------------------- @@ -258,11 +155,13 @@ def _retire(src: Path, dest: Path) -> None: def prune_entries(project: str) -> int: - """Retire entry dirs not named by ``entry_hashes``. No-op when key absent.""" - raw = _read_raw(project) - if "entry_hashes" not in raw: + """Retire entry dirs not named by ``entries.jsonl``. No-op when the pointer + file is absent (never captured), so a build before the first checkpoint is + not mistaken for an eviction.""" + valid = _read_jsonl(_entries_file(project), "hash") + if valid is None: return 0 - valid = set(raw["entry_hashes"] or []) + valid = set(valid) ed = entries_dir(project) if not ed.exists(): return 0 @@ -274,46 +173,41 @@ def prune_entries(project: str) -> int: return removed -def _prune_cache(project: str, root: Path, key: str) -> int: - raw = _read_raw(project) - if key not in raw: +def prune_compute_cache(project: str) -> int: + valid = _read_jsonl(_compute_cache_file(project), "path") + if valid is None: return 0 - valid = set(raw[key] or []) + valid = set(valid) + root = compute_cache_dir(project) if not root.exists(): return 0 removed = 0 for p in root.rglob("*"): if p.is_file() and (rel := str(p.relative_to(root))) not in valid: - _retire(p, bullpen_dir(project) / key / rel) + _retire(p, bullpen_dir(project) / "compute_cache" / rel) removed += 1 return removed -def prune_compute_cache(project: str) -> int: - return _prune_cache(project, compute_cache_dir(project), "compute_cache") - - def restore_from_bullpen(project: str) -> int: """Copy recorded-but-missing artifacts back from the bullpen. The inverse of the prunes, for a reset that walks forward: anything the - restored catalog.yaml's pointer lists name that is absent from the live - tree comes back by *copy*, so the bullpen keeps its set and the - back/forward rehearsal loop can repeat. Only ``reset_to`` calls this — - live operations never see the bullpen, which is what keeps a re-added - expression honest (cold). + restored pointer files name that is absent from the live tree comes back by + *copy*, so the bullpen keeps its set and the back/forward rehearsal loop can + repeat. Only ``reset_to`` calls this — live operations never see the + bullpen, which is what keeps a re-added expression honest (cold). """ - raw = _read_raw(project) bp = bullpen_dir(project) restored = 0 ed = entries_dir(project) - for h in raw.get("entry_hashes") or []: + for h in _read_jsonl(_entries_file(project), "hash") or []: live, parked = ed / h, bp / ENTRIES_DIRNAME / h if not live.exists() and parked.is_dir(): shutil.copytree(parked, live) restored += 1 root = compute_cache_dir(project) - for rel in raw.get("compute_cache") or []: + for rel in _read_jsonl(_compute_cache_file(project), "path") or []: live, parked = root / rel, bp / "compute_cache" / rel if not live.exists() and parked.is_file(): live.parent.mkdir(parents=True, exist_ok=True) @@ -375,7 +269,7 @@ def _step_tags(project: str) -> list[int]: def checkpoint_catalog(project: str, message: str, *, step: int | None = None, label: str | None = None) -> int | None: - """Capture state, commit catalog.yaml once, tag the step. + """Capture pointers, zip pending recipes, commit the tracked surface once, tag the step. One commit per operation (not per mutator), through the fork-safe primitive, under the per-project lock. Returns the step number, or None when the @@ -425,12 +319,13 @@ def _resolve_tag(project: str, tag: str) -> str: def reset_to(project: str, ref: int | str) -> None: - """Restore the catalog to a step/label: git reset --hard, materialize the - derived files, then reconcile the untracked artifacts to the recorded - pointers — evictions retire to the bullpen, recorded-but-missing files - come back from it. Finally re-validate the git-tracked xorq recipe set - against the tallyman pointers, so a step whose two views disagree fails - loudly rather than returning a masked divergence (#52).""" + """Restore the catalog to a step/label: ``git reset --hard`` (which restores + every tracked file — the recipe zips and all decomposed mutable state — at + once), then reconcile the *untracked* artifacts to the recorded pointers: + evictions retire to the bullpen, recorded-but-missing files come back from + it. Finally re-validate the tracked recipe set against the pointers, so a + step whose two views disagree fails loudly rather than returning a masked + divergence (#52).""" cd = catalog_dir(project) tag = f"step-{ref:03d}" if isinstance(ref, int) else str(ref) with _project_lock(project): @@ -438,7 +333,6 @@ def reset_to(project: str, ref: int | str) -> None: rc, _, err = run_git(["reset", "--hard", commit], cwd=cd) if rc != 0: raise RuntimeError(f"catalog reset to {tag!r} failed: {err}") - materialize(project) prune_entries(project) prune_compute_cache(project) restore_from_bullpen(project) diff --git a/src/tallyman_core/notebook.py b/src/tallyman_core/notebook.py index 6af3668..0ee3e2b 100644 --- a/src/tallyman_core/notebook.py +++ b/src/tallyman_core/notebook.py @@ -7,14 +7,11 @@ version pinning. Forensic versions stay in the catalog, not in the notebook. -V0 has one default notebook per project at `notebooks/default.json`: +V0 has one default notebook per project, tracked under the catalog repo at +`notebook.jsonl` — one cell per line: - { - "cells": [ - {"cell_id": "abc123...", "alias": "shoe_sales", "markdown": "..."}, - ... - ] - } + {"cell_id": "abc123...", "alias": "shoe_sales", "markdown": "..."} + ... """ from __future__ import annotations @@ -23,7 +20,7 @@ import uuid from pathlib import Path -from tallyman_core.paths import ensure_project, project_dir +from tallyman_core.paths import catalog_dir, ensure_project class CellNotFound(KeyError): @@ -31,26 +28,34 @@ class CellNotFound(KeyError): def _path(project: str, name: str = "default") -> Path: - return project_dir(project) / "notebooks" / f"{name}.json" + fname = "notebook.jsonl" if name == "default" else f"notebook.{name}.jsonl" + return catalog_dir(project) / fname def load(project: str, name: str = "default") -> dict: p = _path(project, name) if not p.exists(): return {"cells": []} - try: - data = json.loads(p.read_text()) - except json.JSONDecodeError: - return {"cells": []} - data.setdefault("cells", []) - return data + cells: list[dict] = [] + for line in p.read_text().splitlines(): + if not line.strip(): + continue + try: + cells.append(json.loads(line)) + except json.JSONDecodeError: + continue + return {"cells": cells} def _save(project: str, data: dict, name: str = "default") -> None: ensure_project(project) p = _path(project, name) p.parent.mkdir(parents=True, exist_ok=True) - p.write_text(json.dumps(data, indent=2)) + cells = data.get("cells", []) + if not cells: + p.unlink(missing_ok=True) # an empty notebook is no file (a clean diff) + return + p.write_text("".join(json.dumps(c) + "\n" for c in cells)) def find_cell_by_alias(project: str, alias: str, name: str = "default") -> dict | None: diff --git a/src/tallyman_core/paths.py b/src/tallyman_core/paths.py index e2b8271..62c9d2b 100644 --- a/src/tallyman_core/paths.py +++ b/src/tallyman_core/paths.py @@ -238,11 +238,24 @@ def diff_stat_cache_dir(project: str, a_hash: str, b_hash: str) -> Path: def post_processing_dir(project: str) -> Path: - return artifacts_dir(project) / "post_processing" + # Under the catalog repo (was artifacts/) so the native store tracks the + # scripts directly instead of round-tripping them through catalog.yaml. + return catalog_dir(project) / "post_processing" def stats_dir(project: str) -> Path: - return artifacts_dir(project) / "stats" + return catalog_dir(project) / "stats" + + +def prompts_dir(project: str) -> Path: + return catalog_dir(project) / "prompts" + + +def prompts_path(project: str, content_hash: str) -> Path: + """Tracked per-entry authoring-prompt history (was a loose + ``entries//prompts.jsonl``, lost on a clone now that build dirs are + gitignored).""" + return prompts_dir(project) / f"{content_hash}.jsonl" def display_dir(project: str) -> Path: @@ -369,7 +382,6 @@ def ensure_project(project: str) -> Path: display_dir(project), exports_dir(project), data_dir(project), - notebooks_dir(project), ): d.mkdir(parents=True, exist_ok=True) return p diff --git a/src/tallyman_xorq/build.py b/src/tallyman_xorq/build.py index 0585272..b8c9b2a 100644 --- a/src/tallyman_xorq/build.py +++ b/src/tallyman_xorq/build.py @@ -33,21 +33,32 @@ perf_log = logging.getLogger("tallyman.perf") -def _append_prompt(entry_path: Path, prompt: str | None) -> None: - """Append a prompt event to the entry's prompts.jsonl.""" +def _append_prompt(project: str, content_hash: str, prompt: str | None) -> None: + """Append a prompt event to the entry's tracked ``prompts/.jsonl``. + + Relocated out of the (now gitignored) entry dir so the re-run history the + UI disclosure shows survives a reset/clone — it is append-mutable provenance + that the content-addressed recipe zip deliberately excludes. + """ if not prompt: return + from tallyman_core.paths import prompts_path + record = { "prompt": prompt, "at": datetime.now(timezone.utc).isoformat(), } - with (entry_path / "prompts.jsonl").open("a") as fh: + p = prompts_path(project, content_hash) + p.parent.mkdir(parents=True, exist_ok=True) + with p.open("a") as fh: fh.write(json.dumps(record) + "\n") -def read_prompts(entry_path: Path) -> list[dict]: +def read_prompts(project: str, content_hash: str) -> list[dict]: """Return every prompt seen for this entry, oldest first.""" - p = entry_path / "prompts.jsonl" + from tallyman_core.paths import prompts_path + + p = prompts_path(project, content_hash) if not p.exists(): return [] with p.open() as fh: @@ -395,7 +406,7 @@ def build_and_persist( manifest_path = entry_manifest_path(project, content_hash) if manifest_path.exists(): meta = json.loads(manifest_path.read_text()) - _append_prompt(target, prompt) + _append_prompt(project, content_hash, prompt) return BuildResult( content_hash=content_hash, entry_path=target, @@ -523,7 +534,7 @@ def build_and_persist( parents=parents or None, ) write_manifest(target, manifest) - _append_prompt(target, prompt) + _append_prompt(project, content_hash, prompt) # Best-effort: drop the temp script. try: diff --git a/tests/test_display_configs.py b/tests/test_display_configs.py index 482a7c1..41c08a0 100644 --- a/tests/test_display_configs.py +++ b/tests/test_display_configs.py @@ -64,41 +64,9 @@ def test_remove_display_config(project: str): assert get_display_config(project, "a") is None -# --------------------------------------------------------------------------- -# catalog_state capture / materialize round-trip -# --------------------------------------------------------------------------- - - -def test_display_configs_captured_in_catalog_state(project: str): - from tallyman_core.catalog_state import capture_tallyman_state, read_tallyman_state - - set_display_config(project, "abc123", SAMPLE_CONFIG) - capture_tallyman_state(project) - state = read_tallyman_state(project) - hashes = [c["content_hash"] for c in state["display_configs"]] - assert "abc123" in hashes - - -def test_display_configs_materialize_round_trip(project: str): - from tallyman_core.catalog_state import capture_tallyman_state, materialize - - set_display_config(project, "abc123", SAMPLE_CONFIG) - capture_tallyman_state(project) - remove_display_config(project, "abc123") - assert get_display_config(project, "abc123") is None - materialize(project) - restored = get_display_config(project, "abc123") - assert restored is not None - assert restored["diff_provenance"]["source_alias"] == "sales" - - -def test_display_configs_materialize_removes_unrecorded(project: str): - from tallyman_core.catalog_state import materialize, write_tallyman_state - - set_display_config(project, "orphan", SAMPLE_CONFIG) - write_tallyman_state(project, display_configs=[]) - materialize(project) - assert get_display_config(project, "orphan") is None +# The native store tracks display_configs/.json directly, so the old +# capture/materialize catalog.yaml round-trip is gone; survives-reset coverage +# now lives in test_native_catalog_store.py::test_display_config_survives_reset. # --------------------------------------------------------------------------- diff --git a/tests/test_native_catalog_store.py b/tests/test_native_catalog_store.py index 1d63201..2d85dee 100644 --- a/tests/test_native_catalog_store.py +++ b/tests/test_native_catalog_store.py @@ -137,12 +137,14 @@ def test_recipe_zip_members_are_allowlist(project, orders_parquet, tmp_path): """ res = build_and_persist(project, _agg_code(orders_parquet), prompt="seed") ed = paths.entry_dir(project, res.content_hash) - # strays that must NOT leak into the durable zip + # strays that must NOT leak into the durable zip (prompts.jsonl is relocated + # to a tracked prompts/.jsonl now; drop one here to prove the zip + # excludes it even if a stale copy lingers in the entry dir) (ed / "result.parquet").write_bytes(b"heavy") (ed / "foo.bin").write_bytes(b"\x00") + (ed / "prompts.jsonl").write_text('{"prompt": "x"}\n') (ed / ".xorq_build_expanded").mkdir(exist_ok=True) (ed / ".xorq_build_expanded" / "x.yaml").write_text("x") - assert (ed / "prompts.jsonl").exists() # append-mutable provenance — also excluded dest = tmp_path / "recipe.zip" catalog.write_recipe_zip(ed, dest, res.content_hash) @@ -308,3 +310,46 @@ def test_back_then_forward_restores_recipe(project, orders_parquet): cs.reset_to(project, 1) cs.reset_to(project, 2) assert hashes[1] in _tracked_zip_hashes(project) + + +# --------------------------------------------------------------------------- +# Decomposition — mutable state is tracked files that roll back with git reset +# --------------------------------------------------------------------------- + + +def test_prompt_history_survives_reset(project, orders_parquet): + """The per-entry prompt history is a tracked ``prompts/.jsonl`` now, so + a re-run's appended prompt rolls back with the tree on reset (it used to be a + loose, untracked ``entries//prompts.jsonl`` git reset never touched).""" + from tallyman_xorq import read_prompts + + cs.genesis(project) + res = build_and_persist(project, _agg_code(orders_parquet), prompt="first") + h = res.content_hash + cs.checkpoint_catalog(project, "c1") + build_and_persist(project, _agg_code(orders_parquet), prompt="second") # re-run, same hash, appends + cs.checkpoint_catalog(project, "c2") + assert [p["prompt"] for p in read_prompts(project, h)] == ["first", "second"] + cs.reset_to(project, 1) # only "first" had been recorded at step 1 + assert [p["prompt"] for p in read_prompts(project, h)] == ["first"] + cs.reset_to(project, 2) + assert [p["prompt"] for p in read_prompts(project, h)] == ["first", "second"] + + +def test_display_config_survives_reset(project, orders_parquet): + """A display config (tracked ``display_configs/.json``) rolls back with + git reset — the one decomposed section with no survives-reset coverage before + (its old capture/materialize round-trip tests are deleted with this cut).""" + from tallyman_core import display_configs as dc + + cs.genesis(project) + res = build_and_persist(project, _agg_code(orders_parquet)) + h = res.content_hash + cs.checkpoint_catalog(project, "c1") # step 1: entry, no display config + dc.set_display_config(project, h, {"pinned_rows": [1, 2]}) + cs.checkpoint_catalog(project, "c2") # step 2: display config set + assert dc.get_display_config(project, h) == {"pinned_rows": [1, 2]} + cs.reset_to(project, 1) + assert dc.get_display_config(project, h) is None + cs.reset_to(project, 2) + assert dc.get_display_config(project, h) == {"pinned_rows": [1, 2]} diff --git a/tests/test_pack.py b/tests/test_pack.py index 08b3d5a..6f17719 100644 --- a/tests/test_pack.py +++ b/tests/test_pack.py @@ -36,8 +36,9 @@ def test_pack_creates_tgz(project: str, orders_parquet: Path, isolated_home: Pat with tarfile.open(out_path) as tar: names = tar.getnames() - # Alias bookkeeping ships inside catalog.yaml (tallyman keys), not as files. - assert f"{project}/artifacts/catalog/catalog.yaml" in names + # Alias bookkeeping ships as its own tracked file now (native store), not + # smuggled into catalog.yaml. + assert f"{project}/artifacts/catalog/aliases.jsonl" in names assert f"{project}/data/orders.parquet" in names assert any(f"{project}/artifacts/catalog/entries/{h}/" in n for n in names) diff --git a/tests/test_paths.py b/tests/test_paths.py index 57cbb32..d5ea391 100644 --- a/tests/test_paths.py +++ b/tests/test_paths.py @@ -164,14 +164,14 @@ def test_validate_project_name_rejects(name: str): def test_ensure_project_creates_artifacts_layout(isolated_home: Path): p = ensure_project("alpha") assert p == project_dir("alpha") - # New tree: artifacts/ holds everything the system produces. + # New tree: artifacts/ holds everything the system produces. post_processing + # and stats moved under the catalog repo so the native store tracks them. assert (p / "artifacts" / "catalog" / "entries").is_dir() - assert (p / "artifacts" / "post_processing").is_dir() - assert (p / "artifacts" / "stats").is_dir() + assert (p / "artifacts" / "catalog" / "post_processing").is_dir() + assert (p / "artifacts" / "catalog" / "stats").is_dir() assert (p / "artifacts" / "exports").is_dir() - # Inputs and document stay at project root. + # Inputs stay at project root. assert (p / "data").is_dir() - assert (p / "notebooks").is_dir() def test_ensure_project_validates_name(isolated_home: Path): @@ -192,11 +192,11 @@ def test_path_helpers_under_artifacts(isolated_home: Path): assert catalog_dir("alpha") == p / "artifacts" / "catalog" assert entries_dir("alpha") == p / "artifacts" / "catalog" / "entries" assert entry_dir("alpha", "abc123") == entries_dir("alpha") / "abc123" - assert post_processing_dir("alpha") == p / "artifacts" / "post_processing" - assert stats_dir("alpha") == p / "artifacts" / "stats" + assert post_processing_dir("alpha") == p / "artifacts" / "catalog" / "post_processing" + assert stats_dir("alpha") == p / "artifacts" / "catalog" / "stats" assert exports_dir("alpha") == p / "artifacts" / "exports" assert errors_path("alpha") == p / "artifacts" / "errors.jsonl" - # data/ and notebooks/ stay at project root. + # data/ stays at project root. assert data_dir("alpha") == p / "data" diff --git a/tests/test_project_switcher.py b/tests/test_project_switcher.py index e023dee..1fc3b8b 100644 --- a/tests/test_project_switcher.py +++ b/tests/test_project_switcher.py @@ -108,7 +108,6 @@ def test_api_projects_new_creates_and_switches(isolated_home: Path): # Layout was created. p = isolated_home / "projects" / "fresh" assert (p / "artifacts" / "catalog" / "entries").is_dir() - assert (p / "notebooks").is_dir() # Fixture default off — no orders.parquet. assert not (p / "data" / "orders.parquet").exists() diff --git a/tests/test_prompts.py b/tests/test_prompts.py index 4210655..7c4174a 100644 --- a/tests/test_prompts.py +++ b/tests/test_prompts.py @@ -17,7 +17,7 @@ def _code(parquet: Path) -> str: def test_first_build_records_one_prompt(project: str, orders_parquet: Path): res = build_and_persist(project, _code(orders_parquet), prompt="first") - history = read_prompts(res.entry_path) + history = read_prompts(project, res.content_hash) assert len(history) == 1 assert history[0]["prompt"] == "first" @@ -25,7 +25,7 @@ def test_first_build_records_one_prompt(project: str, orders_parquet: Path): def test_re_run_appends_prompt(project: str, orders_parquet: Path): a = build_and_persist(project, _code(orders_parquet), prompt="first") build_and_persist(project, _code(orders_parquet), prompt="second") - history = read_prompts(a.entry_path) + history = read_prompts(project, a.content_hash) assert [p["prompt"] for p in history] == ["first", "second"] @@ -40,7 +40,7 @@ def test_manifest_prompt_remains_first(project: str, orders_parquet: Path): def test_no_prompt_does_not_write_history(project: str, orders_parquet: Path): res = build_and_persist(project, _code(orders_parquet), prompt=None) - assert read_prompts(res.entry_path) == [] + assert read_prompts(project, res.content_hash) == [] def test_entry_detail_renders_prompt_history(fresh_companion_app, project: str, orders_parquet: Path): diff --git a/tests/test_reset_to_revision.py b/tests/test_reset_to_revision.py index 4826e3d..d690073 100644 --- a/tests/test_reset_to_revision.py +++ b/tests/test_reset_to_revision.py @@ -96,48 +96,11 @@ def test_run_git_returns_code_out_err(project): assert ".git" in out -# --------------------------------------------------------------------------- -# Invariant 3 — non-destructive materialize (G3) -# --------------------------------------------------------------------------- - - -def test_capture_materialize_roundtrip(project): - pp.write_post_processing(project, "pp1", PP_SRC) - ss.write_stat(project, "st1", ST_SRC) - charts.set_chart(project, "deadbeef", SPEC) - notebook.append(project, "alias1", markdown="hello") - - state = cs.capture_tallyman_state(project) - assert any(e["name"] == "pp1" for e in state["post_processing"]) - assert any(e["name"] == "st1" for e in state["stats"]) - assert any(c["content_hash"] == "deadbeef" for c in state["charts"]) - assert state["notebook"]["cells"][0]["alias"] == "alias1" - - # Drift the on-disk tree, then materialize back from catalog.yaml. - pp.write_post_processing(project, "pp2_added_later", PP_SRC) - charts.set_chart(project, "cafef00d", SPEC) - cs.materialize(project) - - names = {e["name"] for e in pp.list_post_processings(project)} - assert names == {"pp1"} # pp2 reconciled away - assert charts.list_charts(project) == ["deadbeef"] # cafef00d gone - assert {c["alias"] for c in notebook.load(project)["cells"]} == {"alias1"} - - -def test_materialize_absent_key_preserves_files(project): - """A key absent from catalog.yaml means 'not recorded' — never wipe it.""" - pp.write_post_processing(project, "legacy", PP_SRC) - # catalog.yaml has no post_processing key written. - cs.materialize(project) - assert any(e["name"] == "legacy" for e in pp.list_post_processings(project)) - - -def test_materialize_present_empty_clears(project): - """A present-but-empty list is a deliberate 'none at this step'.""" - pp.write_post_processing(project, "doomed", PP_SRC) - cs.write_tallyman_state(project, post_processing=[]) - cs.materialize(project) - assert pp.list_post_processings(project) == [] +# Invariant 3 (non-destructive materialize) is retired by the native store: the +# six mutable sections are tracked files now, restored by `git reset --hard` +# directly, so there is no capture/materialize catalog.yaml round-trip. The +# survives-reset behavior is covered by test_reset_roundtrip_over_every_kind +# (below) and test_native_catalog_store.py's per-section reset tests. # --------------------------------------------------------------------------- @@ -626,51 +589,9 @@ def test_reset_surfaces_pointer_without_durable_recipe(project): cs.reset_to(project, s1) # forward: bullpen restores the dir, recipe is gone -# --------------------------------------------------------------------------- -# xorq catalog coexistence — genesis must not break `xorq catalog add` -# --------------------------------------------------------------------------- - - -def test_capture_seeds_xorq_keys(project): - """xorq's CatalogYaml.contents indexes `entries`/`aliases` directly, with - no defaulting for dict-shaped files (only the legacy list form gets - defaults) — so a catalog.yaml born from genesis/capture must carry both - keys, or every `xorq catalog add` fails (Error: 'aliases') and - registration silently no-ops. xorq round-trips unknown keys, so seeding - its two keys is the whole coexistence contract.""" - import yaml as _yaml - - cs.capture_tallyman_state(project) - raw = _yaml.safe_load((paths.catalog_dir(project) / "catalog.yaml").read_text()) - assert raw.get("entries") == [] - assert raw.get("aliases") == [] - - -def test_write_tallyman_state_rejects_unrepresentable_values(project): - """write_tallyman_state must never produce a catalog.yaml that safe_load - cannot read back. The reader (_read_raw) is yaml.safe_load, while the - writer's full Dumper happily emits python-object tags for a stray Path — - one such write bricks every later read: checkpoint, materialize, and prune - all degrade to log warnings until the file is hand-repaired. The write must - fail loudly instead, before the tmp-file replace, leaving the previous - catalog.yaml intact.""" - import yaml as _yaml - - cs.write_tallyman_state(project, charts=[]) # a good file exists first - bad_spec = {"content_hash": "x", "spec": {"path": Path("/tmp/x")}} - with pytest.raises(_yaml.YAMLError): - cs.write_tallyman_state(project, charts=[bad_spec]) - assert cs.read_tallyman_state(project)["charts"] == [] # previous file untouched - - -@pytest.mark.integration -def test_xorq_catalog_add_registers_after_genesis(project, orders_parquet): - """End to end: genesis (branch-pinned, xorq-keyed catalog.yaml) then a - build — the best-effort xorq registration must actually land, leaving the - build zip tracked in the catalog repo at entries/.zip.""" - assert cs.genesis(project) == 0 - res = build_and_persist(project, _agg_code(orders_parquet)) - zip_path = paths.catalog_dir(project) / "entries" / f"{res.content_hash}.zip" - assert zip_path.exists(), "xorq catalog add did not register the build" - tracked = _git(project, "ls-files", "entries") - assert f"entries/{res.content_hash}.zip" in tracked.split() +# The xorq-catalog-coexistence section (catalog.yaml entries/aliases seeding, +# write_tallyman_state yaml-safety, and the best-effort xorq-add-after-genesis +# durability) is retired with the subprocess. The native store's +# zip-at-checkpoint durability is covered by +# test_native_catalog_store.py::test_zip_staged_not_committed_until_checkpoint +# and test_n_builds_track_n_zips. From 863a45001256da3015ed229b7a4665754d5e3a6a Mon Sep 17 00:00:00 2001 From: Paddy Mullen Date: Tue, 16 Jun 2026 22:37:57 -0400 Subject: [PATCH 14/27] =?UTF-8?q?refactor:=20Cut=20B=20=E2=80=94=20narrow?= =?UTF-8?q?=20catalog-free=20imports=20+=20datafusion=20connect=20shim?= MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit Replace tallyman's own `import xorq.api as xo` runtime sites with narrow imports that don't transitively load the xorq.catalog package: xorq.expr.api (deferred_read_parquet / memtable / to_sql), xorq.ibis_yaml.compiler (build_expr), xorq.caching (ParquetSnapshotCache). - backend.py (new): catalog-free no-arg datafusion connect shim — Backend(); do_connect(None), the exact body of xorq.api.connect. The naive narrowing xorq.expr.api.connect is ibis's connect(resource, **kwargs), a different function that would fail at call time or build the wrong backend (Risk #3). - source_cache.py: ParquetSnapshotCache(source=connect()) via the shim. - io.py / result_cache.py / build.py / app.py: narrow read/memtable/to_sql/ build_expr imports. - post_processing.py / summary_stats.py: narrow the xo.memtable validate/run sites; KEEP the full `import xorq.api` bind in _restricted_globals — that is the user-authored stat/post-processing exec surface (D4), like recipes. Scope (D4 / Risk #2): the guarantee is that src/ imports no catalog package, enforced by test_src_has_no_xorq_catalog_argv — NOT a sys.modules assertion. A recipe's expr.py still does `import xorq.api as xo`, so re-execing one (build, cached_result_expr) loads the catalog package; switching the recipe authoring surface is a separate codegen-validated follow-up. test_connect_shim_matches_xorq_api_connect pins the shim equivalence: the shim-built and xorq.api.connect-built ParquetSnapshotCache derive the same content-addressed snapshot key and bake the same data (raw parquet bytes aren't stable — datafusion group-by order isn't deterministic). Full fast suite: 507 passed. Co-Authored-By: Claude Opus 4.8 (1M context) --- src/tallyman_companion/app.py | 4 +-- src/tallyman_core/post_processing.py | 8 +++--- src/tallyman_core/summary_stats.py | 4 +-- src/tallyman_xorq/backend.py | 28 +++++++++++++++++++ src/tallyman_xorq/build.py | 4 +-- src/tallyman_xorq/io.py | 4 +-- src/tallyman_xorq/result_cache.py | 6 ++-- src/tallyman_xorq/source_cache.py | 10 +++---- tests/test_native_catalog_store.py | 41 ++++++++++++++++++++++++++++ 9 files changed, 89 insertions(+), 20 deletions(-) create mode 100644 src/tallyman_xorq/backend.py diff --git a/src/tallyman_companion/app.py b/src/tallyman_companion/app.py index 33966f7..1ee234f 100644 --- a/src/tallyman_companion/app.py +++ b/src/tallyman_companion/app.py @@ -202,7 +202,7 @@ def _build_compare_expr(project: str, a_hash: str, b_hash: str, keys: tuple[str, """ import tempfile - import xorq.api as xo + from xorq.ibis_yaml.compiler import build_expr a = cached_result_expr(project, a_hash) b = cached_result_expr(project, b_hash) @@ -211,7 +211,7 @@ def _build_compare_expr(project: str, a_hash: str, b_hash: str, keys: tuple[str, builds_dir = Path(tempfile.gettempdir()) / "tallyman_diff_builds" builds_dir.mkdir(parents=True, exist_ok=True) - build_path = Path(xo.build_expr(expr, builds_dir=str(builds_dir))) + build_path = Path(build_expr(expr, builds_dir=str(builds_dir))) return build_path, overrides diff --git a/src/tallyman_core/post_processing.py b/src/tallyman_core/post_processing.py index 6247931..17777ef 100644 --- a/src/tallyman_core/post_processing.py +++ b/src/tallyman_core/post_processing.py @@ -151,7 +151,7 @@ def validate_post_processing_source(name: str, source: str) -> None: raise PostProcessingSourceError(f"process() must take exactly one parameter, got {len(params)}") try: - import xorq.api as xo # noqa: PLC0415 + from xorq.expr.api import memtable # noqa: PLC0415 except ImportError as e: raise PostProcessingSourceError(f"xorq is not installed: {e}") from None @@ -160,7 +160,7 @@ def validate_post_processing_source(name: str, source: str) -> None: # 1-row single-column dry-run (like stats use) wouldn't exercise # the common shape. Three rows × two cols is small enough that any # ``process`` that runs at all will return quickly. - table = xo.memtable( + table = memtable( {"a": [1, 2, 3], "b": ["x", "y", "z"]}, name="_post_processing_dry_run", ) @@ -238,7 +238,7 @@ def run_post_processing(project: str, entry_name_or_hash: str, source: str) -> d ) try: - import xorq.api as xo # noqa: PLC0415 + from xorq.expr.api import memtable # noqa: PLC0415 except ImportError as exc: raise PostProcessingRunError(f"missing dependency: {exc}") from None @@ -246,7 +246,7 @@ def run_post_processing(project: str, entry_name_or_hash: str, source: str) -> d # entry recomputes, an expensive one reads its baked snapshot, and an evicted # snapshot self-heals — all inside cached_result_expr. No result.parquet. df = cached_result_expr(project, content_hash).execute() - ibis_table = xo.memtable(df, name="_post_processing_run") + ibis_table = memtable(df, name="_post_processing_run") # Exec the source and extract process(). ns: dict = {} diff --git a/src/tallyman_core/summary_stats.py b/src/tallyman_core/summary_stats.py index 1bc84b3..1a43dfc 100644 --- a/src/tallyman_core/summary_stats.py +++ b/src/tallyman_core/summary_stats.py @@ -151,11 +151,11 @@ def validate_stat_source(name: str, source: str) -> None: raise StatSourceError(f"compute() must take exactly one parameter, got {len(params)}") try: - import xorq.api as xo # noqa: PLC0415 + from xorq.expr.api import memtable # noqa: PLC0415 except ImportError as e: raise StatSourceError(f"xorq is not installed: {e}") from None - table = xo.memtable({"a": [1, 2, 3]}, name="_stat_dry_run") + table = memtable({"a": [1, 2, 3]}, name="_stat_dry_run") col = table["a"] try: result = compute(col) diff --git a/src/tallyman_xorq/backend.py b/src/tallyman_xorq/backend.py new file mode 100644 index 0000000..b995af8 --- /dev/null +++ b/src/tallyman_xorq/backend.py @@ -0,0 +1,28 @@ +"""Catalog-free datafusion connect shim (Cut B). + +``xorq.api.connect()`` is the no-arg datafusion ``Backend`` factory tallyman's +source cache needs, but ``import xorq.api`` transitively loads the whole +``xorq.catalog`` package (12 modules). This replicates ``connect`` exactly — +``Backend(); do_connect(None)`` (xorq/api.py) — from the catalog-free backend +module instead. + +The naive narrowing ``xorq.expr.api.connect`` is *not* a drop-in: that is ibis's +``connect(resource, **kwargs)`` (a different function requiring a positional +arg), not this no-arg datafusion factory — so a swap to it would resolve the +symbol and then fail at call time or build the wrong backend. The +determinism-equivalence is pinned by +``test_native_catalog_store.py::test_connect_shim_snapshot_is_byte_identical``. +""" + +from __future__ import annotations + +from xorq.backends.xorq_datafusion import Backend + + +def connect() -> Backend: + """A fresh no-arg datafusion backend — equivalent to ``xorq.api.connect()``, + without importing ``xorq.api`` (and therefore without loading the catalog + package).""" + backend = Backend() + backend.do_connect(None) + return backend diff --git a/src/tallyman_xorq/build.py b/src/tallyman_xorq/build.py index b8c9b2a..4770ce9 100644 --- a/src/tallyman_xorq/build.py +++ b/src/tallyman_xorq/build.py @@ -493,10 +493,10 @@ def build_and_persist( compile_seconds: float | None = None try: - import xorq.api as xo + from xorq.expr.api import to_sql t_compile = time.monotonic() - xo.to_sql(author_expr) + to_sql(author_expr) compile_seconds = round(time.monotonic() - t_compile, 3) except Exception: # Best-effort: a DAG xorq can't render to SQL must not break the build. diff --git a/src/tallyman_xorq/io.py b/src/tallyman_xorq/io.py index 2a2a1f9..ea70395 100644 --- a/src/tallyman_xorq/io.py +++ b/src/tallyman_xorq/io.py @@ -46,7 +46,7 @@ def from_project(rel_path: str, project: str | None = None): for build_and_persist to mix into the entry hash; `off` is the plain path read. """ - import xorq.api as xo + from xorq.expr.api import deferred_read_parquet from tallyman_xorq import source_identity as si @@ -57,7 +57,7 @@ def from_project(rel_path: str, project: str | None = None): si.note_source(rel_path, digest) if si.mode() == "cas": path = si.ensure_cas_path(proj, path, digest) - return xo.deferred_read_parquet(str(path)) + return deferred_read_parquet(str(path)) def from_catalog(alias_or_hash: str, project: str | None = None): diff --git a/src/tallyman_xorq/result_cache.py b/src/tallyman_xorq/result_cache.py index a5b338b..6a68645 100644 --- a/src/tallyman_xorq/result_cache.py +++ b/src/tallyman_xorq/result_cache.py @@ -264,7 +264,7 @@ def cached_result_expr(project: str, content_hash: str): expression uncached under salt, so there is no snapshot to dereference). Both are single-backend recompute expressions — content-honest, no parquet. """ - import xorq.api as xo + from xorq.expr.api import deferred_read_parquet # This function is LRU-memoised, so its body runs only on a miss: every line # below is a cold read. Tag each return with the #74 path it took and the @@ -300,5 +300,5 @@ def _read(expr, tag): return _read(raw, "worthy-recompute-fallback") if not path.exists(): baked.count().execute() # evicted snapshot: repopulate once, then read - return _read(xo.deferred_read_parquet(str(path)), "evicted-self-heal") - return _read(xo.deferred_read_parquet(str(path)), "baked-read") + return _read(deferred_read_parquet(str(path)), "evicted-self-heal") + return _read(deferred_read_parquet(str(path)), "baked-read") diff --git a/src/tallyman_xorq/source_cache.py b/src/tallyman_xorq/source_cache.py index f3a0b58..4812251 100644 --- a/src/tallyman_xorq/source_cache.py +++ b/src/tallyman_xorq/source_cache.py @@ -44,6 +44,7 @@ def _should_cache_read(method_name: str) -> bool: """ return method_name not in _EXEMPT_READS + _IN_MEMORY_MSG = ( "expression reads in-memory data (read_in_memory / ibis.memtable). This " "usually means the source was loaded into pandas (e.g. pd.read_csv) and " @@ -106,13 +107,14 @@ def rewrite_for_build(expr, project: str): path-only snapshot keys would collide across salted entries with identical paths but different content. In-memory rejection still applies. """ - import xorq.api as xo import xorq.vendor.ibis.expr.operations as ops + from xorq.caching import ParquetSnapshotCache from xorq.common.utils.graph_utils import replace_nodes, walk_nodes from xorq.expr.relations import Read from tallyman_core.paths import compute_cache_dir from tallyman_xorq import source_identity as si + from tallyman_xorq.backend import connect reads = walk_nodes(Read, expr) if walk_nodes(ops.InMemoryTable, expr) or any(r.method_name == "read_in_memory" for r in reads): @@ -130,7 +132,7 @@ def rewrite_for_build(expr, project: str): # same Read — without the memo it would wrap it again, forever. targets = {r for r in reads if _should_cache_read(r.method_name)} if targets: - src_cache = xo.ParquetSnapshotCache.from_kwargs(source=xo.connect(), base_path=base) + src_cache = ParquetSnapshotCache.from_kwargs(source=connect(), base_path=base) wrapped: dict = {} def replacer(node, kwargs): @@ -148,9 +150,7 @@ def replacer(node, kwargs): # relative_path keeps baked results inspectable apart from source parses # under the same compute-cache root. if _is_worthy_expr(expr): - result_cache = xo.ParquetSnapshotCache.from_kwargs( - source=xo.connect(), base_path=base, relative_path="result_cache" - ) + result_cache = ParquetSnapshotCache.from_kwargs(source=connect(), base_path=base, relative_path="result_cache") expr = expr.cache(cache=result_cache) return expr diff --git a/tests/test_native_catalog_store.py b/tests/test_native_catalog_store.py index 2d85dee..10fa48d 100644 --- a/tests/test_native_catalog_store.py +++ b/tests/test_native_catalog_store.py @@ -312,6 +312,47 @@ def test_back_then_forward_restores_recipe(project, orders_parquet): assert hashes[1] in _tracked_zip_hashes(project) +# --------------------------------------------------------------------------- +# Cut B — the catalog-free datafusion connect shim +# --------------------------------------------------------------------------- + + +def test_connect_shim_matches_xorq_api_connect(project, orders_parquet, monkeypatch): + """The catalog-free connect shim is equivalent to xorq.api.connect(): it + builds the same datafusion backend, so a ParquetSnapshotCache(source=shim) + derives the SAME content-addressed snapshot key and bakes the SAME result + data. A naive swap to xorq.expr.api.connect (ibis's connect(resource, + **kwargs)) would resolve the symbol then build the wrong backend or fail at + call time (Risk #3) — this is the central Cut-B correctness claim. + + Not raw-byte equality: datafusion's group-by output order (and parquet + metadata) isn't deterministic, so even one backend re-baking the same expr + yields different bytes. The content-addressed *key* and the *data* are what + must match. + """ + import pandas as pd + import xorq.api as xo + + from tallyman_xorq.result_cache import _cached_node_path, _recipe_expr + from tallyman_xorq.source_cache import rewrite_for_build + + res = build_and_persist(project, _agg_code(orders_parquet)) # bakes via the shim + raw = _recipe_expr(project, res.content_hash) + shim_path = _cached_node_path(rewrite_for_build(raw, project)) + assert shim_path is not None and shim_path.exists(), "the shim must bake a real snapshot" + shim_df = pd.read_parquet(shim_path).sort_values("region").reset_index(drop=True) + + # rewrite_for_build does `from tallyman_xorq.backend import connect` at call + # time, so patching the shim's source reroutes the next derive/bake. + monkeypatch.setattr("tallyman_xorq.backend.connect", xo.connect) + api_expr = rewrite_for_build(raw, project) + assert _cached_node_path(api_expr) == shim_path, "shim and xorq.api.connect must share the snapshot key" + shim_path.unlink() # evict, then re-bake via xorq.api.connect + api_expr.count().execute() + api_df = pd.read_parquet(shim_path).sort_values("region").reset_index(drop=True) + pd.testing.assert_frame_equal(shim_df, api_df) + + # --------------------------------------------------------------------------- # Decomposition — mutable state is tracked files that roll back with git reset # --------------------------------------------------------------------------- From 356ee1cf3fba5e1901014a54830cbc17de856206 Mon Sep 17 00:00:00 2001 From: Paddy Mullen Date: Tue, 16 Jun 2026 22:40:40 -0400 Subject: [PATCH 15/27] chore: sweep stale xorq-catalog docs + dead code after the native cut MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit Non-functional cleanup of references the native store made obsolete: - aliases.py / paths.py / catalog_state.py: rewrite the module docstrings and the project-layout diagram to the decomposed tree (aliases.jsonl, notebook.jsonl, prompts/, post_processing+stats under catalog/, the two pointer JSONLs) — no more catalog.yaml / xorq assert_consistency framing. - paths.py: delete catalog_repo_path(), dead since the `xorq catalog --path` subprocess was removed. - test_reset_to_revision.py: retarget test_catalog_repo_branch_is_main's docstring (branch pin is now for deterministic, host-config-independent layout, not xorq compat); the main-branch assertion stays. The fix-induced test breaks themselves (capture/materialize round-trips, catalog.yaml coexistence, read_prompts signature, path-layout assertions) were already resolved with the commits that caused them. 98 affected tests pass. Co-Authored-By: Claude Opus 4.8 (1M context) --- src/tallyman_core/aliases.py | 15 ++++++------ src/tallyman_core/catalog_state.py | 7 +++--- src/tallyman_core/paths.py | 39 +++++++++--------------------- tests/test_reset_to_revision.py | 8 +++--- 4 files changed, 24 insertions(+), 45 deletions(-) diff --git a/src/tallyman_core/aliases.py b/src/tallyman_core/aliases.py index c9baa94..7714493 100644 --- a/src/tallyman_core/aliases.py +++ b/src/tallyman_core/aliases.py @@ -5,14 +5,13 @@ alias_map: {alias: latest_hash} alias_history: {alias: [hash_v1, hash_v2, ...]} # oldest first -Both live as tallyman-owned keys in the catalog's ``catalog.yaml`` (read and -written through ``catalog_state``), not as separate files. catalog.yaml is the -versioned state carrier the catalog git repo tracks, so alias state clones and -versions with the catalog and ``reset_to``'s ``git reset`` rolls it back for -free — a sidecar store would have to be reconciled separately, and committing -one *inside* the repo broke xorq's ``assert_consistency`` (#48). xorq keeps its -own ``aliases`` key + alias symlinks in the same file; these tallyman keys sit -alongside, and xorq round-trips them untouched. +Both live in a tracked ``aliases.jsonl`` in the catalog repo — one line per +alias, ``{"alias", "latest", "history": [...]}``. The native store tracks the +file directly, so alias state clones and versions with the catalog and +``reset_to``'s ``git reset`` rolls it back with the rest of the tree (no +separate reconcile). Before the native cut this had to be smuggled into +catalog.yaml keys, because committing a separate file inside the (then xorq) +catalog repo broke its ``assert_consistency`` (#48); that constraint is gone. Identity is the content hash; the alias is a *name* for a concept that may evolve. The latest hash is always the current best answer; older hashes diff --git a/src/tallyman_core/catalog_state.py b/src/tallyman_core/catalog_state.py index cf6c51b..236ce98 100644 --- a/src/tallyman_core/catalog_state.py +++ b/src/tallyman_core/catalog_state.py @@ -239,10 +239,9 @@ def _project_lock(project: str): def ensure_catalog_repo(project: str) -> bool: """Idempotently git-init the catalog repo. Returns True if it exists after. - The branch is pinned to ``main`` (git ≥ 2.28): xorq's catalog layer - hardcodes it, and inheriting the host's ``init.defaultBranch`` (master on - stock git) makes every ``xorq catalog add`` fail — silently, because - registration is best-effort. + The branch is pinned to ``main`` (git ≥ 2.28) rather than inheriting the + host's ``init.defaultBranch`` (master on stock git), so the repo layout is + deterministic and host-config-independent. """ cd = catalog_dir(project) cd.mkdir(parents=True, exist_ok=True) diff --git a/src/tallyman_core/paths.py b/src/tallyman_core/paths.py index 62c9d2b..cd0c287 100644 --- a/src/tallyman_core/paths.py +++ b/src/tallyman_core/paths.py @@ -7,14 +7,19 @@ ├── buckaroo_sessions.json # global Buckaroo session map └── projects// ├── artifacts/ # everything the system produces - │ ├── catalog/entries// # the xorq catalog git repo - │ │ # (alias bookkeeping is catalog.yaml keys, not files) - │ ├── post_processing/.py - │ ├── stats/.py + │ ├── catalog/ # the native catalog git repo + │ │ ├── entries/.zip # immutable recipe (one blob per entry) + │ │ ├── entries// # untracked build dir (gitignored) + │ │ ├── aliases.jsonl # tracked alias bookkeeping + │ │ ├── notebook.jsonl # tracked notebook cells + │ │ ├── chart_specs/.vl.json + │ │ ├── display_configs/.json + │ │ ├── post_processing/.py , stats/.py + │ │ ├── prompts/.jsonl + │ │ └── entries.jsonl , compute_cache.jsonl # untracked-artifact pointers │ ├── exports/... # marimo .py, screenshots, CSVs │ └── errors.jsonl - ├── data/ # input parquets (fixtures or user) - └── notebooks/notebook.json + └── data/ # input parquets (fixtures or user) The active project lives in a single one-line file at ``~/.tallyman/active_project``. ``resolve_project()`` reads it on every @@ -338,28 +343,6 @@ def resolve_project(explicit: str | None = None) -> str | None: return None -# --------------------------------------------------------------------------- -# xorq catalog repo path (per-project) -# --------------------------------------------------------------------------- - - -def catalog_repo_path() -> Path: - """Path the xorq CLI uses for its ``catalog`` git repo. - - Lives at ``/artifacts/catalog/`` — same dir as the tallyman - catalog entries. ``xorq catalog init`` writes a ``.git/`` alongside - them. - - ``TALLYMAN_CATALOG_REPO`` env overrides for tests or external repos. - """ - override = os.environ.get("TALLYMAN_CATALOG_REPO") - if override: - return Path(override) - project = resolve_project() - if project is None: - raise RuntimeError("catalog_repo_path called with no active project") - return catalog_dir(project) - # --------------------------------------------------------------------------- # Project lifecycle diff --git a/tests/test_reset_to_revision.py b/tests/test_reset_to_revision.py index d690073..e54cdd8 100644 --- a/tests/test_reset_to_revision.py +++ b/tests/test_reset_to_revision.py @@ -431,11 +431,9 @@ def test_cross_process_checkpoint(project, isolated_home): def test_catalog_repo_branch_is_main(project, monkeypatch): - """xorq's catalog layer hardcodes its branch as `main`; a plain `git init` - inherits init.defaultBranch (master on a stock machine and on CI), after - which `xorq catalog add` fails forever with "refs/heads/main does not - exist" — silently, because registration is best-effort. The repo branch - must be pinned, not inherited from host config.""" + """The catalog repo pins its branch to `main` rather than inheriting the + host's init.defaultBranch (master on a stock machine and on CI), so the + repo's layout is deterministic and independent of host git config.""" monkeypatch.setenv("GIT_CONFIG_GLOBAL", "/dev/null") monkeypatch.setenv("GIT_CONFIG_SYSTEM", "/dev/null") cs.ensure_catalog_repo(project) From 62fe9d044fcbda6bbbb03652545e7bbe2f7607e9 Mon Sep 17 00:00:00 2001 From: Paddy Mullen Date: Tue, 16 Jun 2026 22:47:30 -0400 Subject: [PATCH 16/27] test: pin the decomposed tracked tree end to end; mark the plan implemented MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit - test_tracked_tree_is_the_decomposed_surface: build an entry, touch every decomposed section (alias, chart, display config, post-processing, stat, notebook, prompt), checkpoint, and assert `git ls-files` equals exactly the native decomposed surface — no catalog.yaml, no build dirs, no xorq metadata/symlinks. The whole-cut regression guard. - plans/native-catalog-store.md: status → implemented, with the deviations from the plan as written (merged commits 2+3, the diff.py phantom Cut-B site, the full xorq.api bind kept for user-exec globals, key+data equivalence instead of byte-identical snapshots, the green-today 1a tests folded into fix commits, and the deferred corpus rebuild). Verified: fast suite 508 passed; integration 6 passed; cache_lab 16 passed (off mode) with the content-identity test passing under salt. A manual genesis→build×2→checkpoint→reset roundtrip produces the expected tree and rolls every section back. Co-Authored-By: Claude Opus 4.8 (1M context) --- plans/native-catalog-store.md | 42 ++++++++++++++++++++++++++++-- tests/test_native_catalog_store.py | 39 +++++++++++++++++++++++++++ 2 files changed, 79 insertions(+), 2 deletions(-) diff --git a/plans/native-catalog-store.md b/plans/native-catalog-store.md index 8384cd3..1b38cdf 100644 --- a/plans/native-catalog-store.md +++ b/plans/native-catalog-store.md @@ -1,7 +1,45 @@ # Plan: native `tallyman_core.catalog` — drop xorq's catalog package -Status: proposed (2026-06-16). Refined through a parallel code investigation and -a grilling pass; the decisions below are settled unless marked open. +Status: **implemented** (2026-06-16) on `plan/native-catalog-store`. Refined +through a parallel code investigation and a grilling pass; the decisions below +are settled unless marked open. + +## Implementation notes (deviations from the plan as written) + +The cut landed as: `1a` failing-first markers → native recipe store + checkpoint +owns the zip → catalog.yaml decomposition → Cut B → doc/dead-code sweep → +verify. Where reality diverged from the plan text: + +- **Plan commits 2 and 3 merged.** The recipe store (`write_recipe_zip`) is inert + until the checkpoint calls it, and `reset_to`'s consistency check needs both + halves, so splitting them would leave `reset_to` transiently broken. Landed as + one coherent "Cut A" commit. +- **Line numbers were uniformly stale** (~+60 in `build.py`, e.g. the registration + swallow is :539-541 not :477-478, the re-run prompt append is :396 not :335). + Re-derived against the post-#104 tree before writing any test. +- **#104 already removed `result.parquet`.** The zip allowlist's "exclude + result.parquet" is now about a stray that no longer naturally appears; the real + heavy exclusion is `.xorq_build_expanded/`. The dead `migrate_drop_result_parquet` + marker is gitignored (`.migrated_*`). +- **Several planned "failing-first" 1a tests were green today** (the merged #48 + fix already registers every entry; the salted-zip naming already works via the + `entry_name` copytree; checkpoint used an explicit `git add catalog.yaml`). + Per the TDD rule they shipped with their fix commit as guards, not in 1a. +- **Cut B — `diff.py` is a phantom site.** Neither `diff.py` imports `xorq.api`; + the real `xo.build_expr` site is `app.py._build_compare_expr`. And + `post_processing.py`/`summary_stats.py` keep the full `import xorq.api` bind in + `_restricted_globals` — that is the user-authored exec surface (D4), like + recipes; only their `xo.memtable` uses were narrowed. +- **Connect-shim test asserts key + data equivalence, not byte-identical.** The + baked snapshot parquet is *not* byte-deterministic even for one backend + (datafusion's group-by output order varies), so the meaningful equivalence is + the content-addressed snapshot *key* + the *data*, not raw bytes. +- **Corpus rebuild deferred.** Eight on-disk projects under + `~/.tallyman-notebooks` are still old-format (`catalog.yaml` + xorq zips with + `metadata/` sidecars); they need rebuilding (the single-user rule permits it), + but the rebuild *mechanism* is the open item below, so it was not auto-run. + End-to-end verification ran on a fresh project instead + (`test_tracked_tree_is_the_decomposed_surface`). ## Problem diff --git a/tests/test_native_catalog_store.py b/tests/test_native_catalog_store.py index 10fa48d..ac9dc1c 100644 --- a/tests/test_native_catalog_store.py +++ b/tests/test_native_catalog_store.py @@ -394,3 +394,42 @@ def test_display_config_survives_reset(project, orders_parquet): assert dc.get_display_config(project, h) is None cs.reset_to(project, 2) assert dc.get_display_config(project, h) == {"pinned_rows": [1, 2]} + + +def test_tracked_tree_is_the_decomposed_surface(project, orders_parquet): + """End to end: after building an entry and touching every decomposed section, + the git-tracked tree is *exactly* the native decomposed surface — no + catalog.yaml, no build dirs, no xorq metadata/symlinks. Pins the whole cut.""" + from tallyman_core import aliases as al + from tallyman_core import charts, notebook + from tallyman_core import display_configs as dc + from tallyman_core import post_processing as pp + from tallyman_core import summary_stats as ss + + cs.genesis(project) + res = build_and_persist(project, _agg_code(orders_parquet), prompt="first") + h = res.content_hash + al.set_alias(project, "by_region", h, expect_exists=False) + charts.set_chart(project, h, {"mark": "bar"}) + dc.set_display_config(project, h, {"pinned_rows": [1]}) + pp.write_post_processing(project, "pp1", "def process(expr):\n return expr\n") + ss.write_stat(project, "st1", "def compute(col):\n return col.count()\n") + notebook.append(project, "by_region", markdown="hello") + cs.checkpoint_catalog(project, "create") + + rc, out, _ = cs.run_git(["ls-files"], cwd=paths.catalog_dir(project)) + tracked = set(out.split()) + assert tracked == { + ".gitignore", + "aliases.jsonl", + "notebook.jsonl", + "entries.jsonl", + "compute_cache.jsonl", + f"entries/{h}.zip", + f"chart_specs/{h}.vl.json", + f"display_configs/{h}.json", + "post_processing/pp1.py", + "stats/st1.py", + f"prompts/{h}.jsonl", + }, tracked + assert "catalog.yaml" not in tracked and not any("/xorq_build/" in t for t in tracked) From 778c55f9bfcece7b94f1c477544a22232e730994 Mon Sep 17 00:00:00 2001 From: Paddy Mullen Date: Wed, 17 Jun 2026 01:51:03 -0400 Subject: [PATCH 17/27] test: native catalog store integrity + warm-evict self-heal (failing-first) Four bug-hunt regressions, each seen failing before the fix: - assert_catalog_consistent's allowlist uses fnmatch, whose * spans /, so a stray nested under an allowlisted dir prefix (post_processing/sub/leak.py, chart_specs/deep/leak.vl.json, prompts/_disabled/leak.jsonl, ...) slips past the deny-by-default guard. A companion test pins that the legitimate two-level *_disabled members must still pass. - An interrupted atomic-write *.tmp (aliases._write / write_recipe_zip) is staged by the checkpoint's git add -A and durably wedges a later reset_to. - A checkpoint over a manifest-less mid-build entry dir records a pointer with no durable recipe zip (capture_tallyman_state and zip_pending_entries disagree), producing a self-inconsistent step reset_to permanently rejects. - cached_result_expr memoised the snapshot existence check, so an expensive entry's snapshot evicted after a warm read dangled on a stale deferred_read_parquet ("At least one path is required"). Co-Authored-By: Claude Opus 4.8 (1M context) --- tests/test_native_catalog_store.py | 92 ++++++++++++++++++++++++++++++ tests/test_result_cache.py | 29 ++++++++++ 2 files changed, 121 insertions(+) diff --git a/tests/test_native_catalog_store.py b/tests/test_native_catalog_store.py index ac9dc1c..3fbe55e 100644 --- a/tests/test_native_catalog_store.py +++ b/tests/test_native_catalog_store.py @@ -433,3 +433,95 @@ def test_tracked_tree_is_the_decomposed_surface(project, orders_parquet): f"prompts/{h}.jsonl", }, tracked assert "catalog.yaml" not in tracked and not any("/xorq_build/" in t for t in tracked) + + +# --------------------------------------------------------------------------- +# tracked-surface integrity (bug-hunt regressions) +# --------------------------------------------------------------------------- + + +@pytest.mark.parametrize( + "rel", + [ + "post_processing/sub/leak.py", + "chart_specs/deep/leak.vl.json", + "prompts/_disabled/leak.jsonl", + "stats/nested/leak.py", + "display_configs/nested/leak.json", + ], +) +def test_nested_stray_under_allowlisted_prefix_is_rejected(project, orders_parquet, rel): + """A stray nested *under* an allowlisted dir prefix must be rejected. + + ``fnmatch``'s ``*`` spans ``/``, so ``post_processing/sub/leak.py`` wrongly + matched the ``post_processing/*.py`` pattern and slipped past the + deny-by-default guard (none of these paths are gitignored either). The + allowlist must be path-segment-aware. + """ + cs.genesis(project) + build_and_persist(project, _agg_code(orders_parquet)) + cs.checkpoint_catalog(project, "create") + cd = paths.catalog_dir(project) + p = cd / rel + p.parent.mkdir(parents=True, exist_ok=True) + p.write_text("LEAK\n") + cs.run_git(["add", "-A"], cwd=cd) + _, out, _ = cs.run_git(["ls-files"], cwd=cd) + assert rel in out.split() # `git add -A` staged it (not gitignored) + pointers = set(cs.read_tallyman_state(project)["entry_hashes"]) + with pytest.raises(RuntimeError, match="outside the allowed surface"): + catalog.assert_catalog_consistent(project, pointers) + + +def test_legit_disabled_subdir_member_still_passes(project, orders_parquet): + """The segment-aware allowlist must not over-reject the legitimate two-level + members it already tracks (``post_processing/_disabled/*.py``, &c.).""" + cs.genesis(project) + build_and_persist(project, _agg_code(orders_parquet)) + cs.checkpoint_catalog(project, "create") + cd = paths.catalog_dir(project) + for rel in ("post_processing/_disabled/old.py", "stats/_disabled/old.py"): + p = cd / rel + p.parent.mkdir(parents=True, exist_ok=True) + p.write_text("def process(t):\n return t\n") + cs.run_git(["add", "-A"], cwd=cd) + pointers = set(cs.read_tallyman_state(project)["entry_hashes"]) + catalog.assert_catalog_consistent(project, pointers) # must NOT raise + + +def test_interrupted_atomic_write_tmp_not_committed(project, orders_parquet): + """A ``*.tmp`` left by a crash mid atomic-write (``aliases._write`` / + ``write_recipe_zip``) must never be staged by the checkpoint's ``git add -A``, + so it cannot durably wedge a later ``reset_to`` via the tracked-surface + allowlist.""" + from tallyman_core import aliases as al + + cs.genesis(project) + res = build_and_persist(project, _agg_code(orders_parquet)) + al.set_alias(project, "regions", res.content_hash) + cs.checkpoint_catalog(project, "cp1") + cd = paths.catalog_dir(project) + # Simulate a crash between tmp.write_text and tmp.replace in aliases._write. + (cd / "aliases.jsonl.tmp").write_text('{"alias":"x","latest":"d","history":["d"]}\n') + s2 = cs.checkpoint_catalog(project, "cp2") # nothing consumes the .tmp + _, out, _ = cs.run_git(["ls-files"], cwd=cd) + assert "aliases.jsonl.tmp" not in out.split(), "interrupted atomic-write .tmp committed by `git add -A`" + cs.reset_to(project, s2) # the committed step must remain reset-able + + +def test_checkpoint_skips_manifestless_entry_dir(project): + """A checkpoint firing during the mid-build window (entry dir with + ``expr.py`` + ``schema.json`` but no ``manifest.json``) must not commit a + pointer it has no durable recipe zip for; the committed step must remain + ``reset_to``-able. ``capture_tallyman_state`` must apply the same manifest + filter ``zip_pending_entries`` already uses.""" + cs.genesis(project) + ed = paths.entry_dir(project, "deadbeef0001") + ed.mkdir(parents=True) + (ed / "expr.py").write_text("expr = 1\n") + (ed / "schema.json").write_text("{}") # NO manifest.json — a mid-build dir + step = cs.checkpoint_catalog(project, "checkpoint over a manifest-less mid-build dir") + pointers = set(cs.read_tallyman_state(project)["entry_hashes"]) + assert pointers == catalog.tracked_recipe_hashes(project) # pointer set == durable recipe set + catalog.assert_catalog_consistent(project, pointers) # must not raise + cs.reset_to(project, step) # must not raise diff --git a/tests/test_result_cache.py b/tests/test_result_cache.py index 7e6b76d..a82975a 100644 --- a/tests/test_result_cache.py +++ b/tests/test_result_cache.py @@ -471,3 +471,32 @@ def test_ensure_result_is_removed(): assert not hasattr(rc, "ensure_result") assert not hasattr(rc, "ensure_result_build") assert not hasattr(rc, "ensure_viewer_expanded_build") + + +def test_cached_result_expr_self_heals_after_warm_then_evict(project, orders_parquet, monkeypatch): + """An expensive entry's baked snapshot evicted *after* a warm read must still + self-heal on the next read, not dangle on a stale ``deferred_read_parquet``. + + ``cached_result_expr`` memoised the snapshot existence check, so once an + expensive entry was warm in the LRU, evicting its snapshot left the cached + expression pointing at a now-missing path — the next ``.execute()`` raised + ``ValueError: At least one path is required``. The existence check + self-heal + must run on every call, not only on a cold LRU miss. + """ + from tallyman_xorq.result_cache import baked_snapshot_path, cached_result_expr + + monkeypatch.setenv("TALLYMAN_PROJECT", project) + catalog_create("agg", _agg_code(project)) + agg_h = _hash_of(project) + + # Warm the LRU via a real read, exactly as the public from_catalog API does. + expected = len(cached_result_expr(project, agg_h).execute()) + + # Evict the baked snapshot while the LRU stays WARM (no cache_clear). + p = baked_snapshot_path(project, agg_h) + assert p is not None and p.exists() + p.unlink() + + # Re-read the SAME entry: must self-heal, not dangle on the stale deferred read. + df = cached_result_expr(project, agg_h).execute() + assert len(df) == expected From a0d639f0e9aece8841565dcbd5f0fad3be74e4eb Mon Sep 17 00:00:00 2001 From: Paddy Mullen Date: Wed, 17 Jun 2026 01:58:27 -0400 Subject: [PATCH 18/27] fix: close three tracked-surface integrity holes in the native store MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit - assert_catalog_consistent matched the allowlist with fnmatch, whose * spans /, so a stray nested under an allowlisted dir prefix passed the deny-by-default guard (and entries/sub/x.zip slipped past entries/*.zip). Add a path-segment-aware _surface_match (equal segment count + per-segment fnmatchcase) so * never crosses /. - Add *.tmp to the repo .gitignore so an interrupted atomic-write temp (aliases._write, write_recipe_zip, the JSONL writers) is never staged by the checkpoint's git add -A and cannot wedge a later reset_to. - capture_tallyman_state recorded every entry dir as a pointer, including a manifest-less mid-build dir that zip_pending_entries skips — committing a pointer with no durable recipe zip, a self-inconsistent step reset_to permanently rejects. Apply zip_pending_entries' manifest filter to capture so the two agree on the recorded set. Updates two reset tests whose premises the manifest filter changes: the back-then-forward fixture now gives its synthetic entry a manifest (real entries always have one); the pointer-without-recipe defense-in-depth test now injects the divergence directly, since capture no longer creates it. Co-Authored-By: Claude Opus 4.8 (1M context) --- src/tallyman_core/catalog.py | 15 ++++++++++++++- src/tallyman_core/catalog_state.py | 9 ++++++++- tests/test_reset_to_revision.py | 30 ++++++++++++++++++------------ 3 files changed, 40 insertions(+), 14 deletions(-) diff --git a/src/tallyman_core/catalog.py b/src/tallyman_core/catalog.py index 3a36be1..9be97a8 100644 --- a/src/tallyman_core/catalog.py +++ b/src/tallyman_core/catalog.py @@ -78,6 +78,7 @@ ".checkpoint.lock", ".xorq_build_expanded/", ".migrated_*", # machine-local upgrade-migration markers (e.g. #104), not catalog content + "*.tmp", # an interrupted atomic-write temp (aliases._write, write_recipe_zip, ...) — never a committable artifact ) @@ -94,6 +95,18 @@ def write_gitignore(project: str) -> Path: return p +def _surface_match(path: str, pattern: str) -> bool: + """fnmatch per path segment, so a ``*`` never spans ``/``. + + ``post_processing/*.py`` matches ``post_processing/foo.py`` but NOT + ``post_processing/sub/leak.py`` — plain ``fnmatch`` lets ``*`` cross ``/`` and + would wave the nested stray through (and ``entries/*.zip`` past + ``entries/sub/x.zip``), defeating the deny-by-default allowlist. + """ + segs, pats = path.split("/"), pattern.split("/") + return len(segs) == len(pats) and all(fnmatch.fnmatchcase(s, p) for s, p in zip(segs, pats)) + + def _recipe_members(entry_dir: Path) -> list[tuple[Path, str]]: """(abs_path, relpath) for every allowlisted recipe file, sorted by relpath.""" members: list[tuple[Path, str]] = [] @@ -183,7 +196,7 @@ def assert_catalog_consistent(project: str, pointers: set[str]) -> None: cd = catalog_dir(project) rc, out, _ = run_git(["ls-files"], cwd=cd) if rc == 0: - unlisted = [p for p in out.split() if not any(fnmatch.fnmatch(p, pat) for pat in TRACKED_SURFACE)] + unlisted = [p for p in out.split() if not any(_surface_match(p, pat) for pat in TRACKED_SURFACE)] if unlisted: raise RuntimeError( "catalog tracks paths outside the allowed surface (a stray file `git add -A` would commit): " diff --git a/src/tallyman_core/catalog_state.py b/src/tallyman_core/catalog_state.py index 236ce98..ee6132c 100644 --- a/src/tallyman_core/catalog_state.py +++ b/src/tallyman_core/catalog_state.py @@ -128,7 +128,14 @@ def capture_tallyman_state(project: str) -> dict: JSONL files. The decomposed sections write their own tracked files, so capture no longer touches charts/display/pp/stats/aliases/notebook.""" ed = entries_dir(project) - entry_hashes = sorted(c.name for c in ed.iterdir() if c.is_dir()) if ed.exists() else [] + # Only COMPLETE entry dirs (a manifest is the build's last write) — the same + # filter zip_pending_entries uses, so capture and the zip writer agree on the + # set a checkpoint records. Otherwise a checkpoint racing a mid-build dir + # records a pointer with no durable recipe zip, and the step is permanently + # un-reset-able (the pointer/recipe consistency guard rejects it). + entry_hashes = ( + sorted(c.name for c in ed.iterdir() if c.is_dir() and (c / "manifest.json").is_file()) if ed.exists() else [] + ) compute_cache = _list_cache_files(compute_cache_dir(project)) write_tallyman_state(project, entry_hashes=entry_hashes, compute_cache=compute_cache) return {"entry_hashes": entry_hashes, "compute_cache": compute_cache} diff --git a/tests/test_reset_to_revision.py b/tests/test_reset_to_revision.py index e54cdd8..4459f33 100644 --- a/tests/test_reset_to_revision.py +++ b/tests/test_reset_to_revision.py @@ -540,6 +540,7 @@ def test_scrub_back_then_forward_restores_from_bullpen(project): ed = paths.entry_dir(project, "bbbb") ed.mkdir(parents=True) + (ed / "manifest.json").write_text("{}") # a complete entry — capture only records manifest-bearing dirs (ed / "result.parquet").write_bytes(b"big") _stage_recipe_zip(project, "bbbb") # durable recipe, so the step's two views agree (#52) cc = paths.compute_cache_dir(project) @@ -568,23 +569,28 @@ def test_reset_surfaces_pointer_without_durable_recipe(project): pointer names an entry whose durable xorq recipe (entries/.zip) was never committed (#52). - This is the shape any interrupted or failed best-effort ``xorq catalog add`` - leaves behind: the build dir + the ``entry_hashes`` pointer land, but the - content-addressed recipe never reaches git. On a forward reset - ``restore_from_bullpen`` copies the build dir back, so the tallyman view - claims an entry the durable xorq catalog cannot reproduce — its only copy is - the evictable bullpen dir. The two views of "what entries exist" disagree; - reset must surface that divergence, not mask it and return success.""" + Defense-in-depth: the normal capture path no longer *creates* this divergence + — a manifest-less mid-build dir is not recorded as a pointer (see + test_native_catalog_store.py::test_checkpoint_skips_manifestless_entry_dir). + But should a committed step's ``entry_hashes`` pointer name an entry whose + durable recipe zip is absent anyway (manual corruption, or a future code path + that bypasses the capture filter), reset must still surface the divergence — + the consistency guard is the backstop — not mask it and return success. So + inject the divergent step directly rather than via capture.""" cs.ensure_catalog_repo(project) s0 = cs.checkpoint_catalog(project, "baseline") # no entries yet - # A build dir + pointer whose recipe zip was never committed (failed add). - paths.entry_dir(project, "deadbeef").mkdir(parents=True) - s1 = cs.checkpoint_catalog(project, "entry whose recipe never reached git") + # Commit a step whose pointer has no durable recipe zip, bypassing the + # capture manifest-filter (which would otherwise drop the bogus pointer). + cs.write_tallyman_state(project, entry_hashes=["deadbeef"]) + cd = paths.catalog_dir(project) + cs.run_git(["add", "-A"], cwd=cd) + cs.run_git([*cs._GIT_ID, "commit", "-m", "pointer whose recipe never reached git", "--allow-empty"], cwd=cd) + cs.run_git(["tag", "-f", "step-001"], cwd=cd) - cs.reset_to(project, s0) # evict deadbeef to the bullpen + cs.reset_to(project, s0) # move away from the divergent step with pytest.raises(RuntimeError, match="deadbeef"): - cs.reset_to(project, s1) # forward: bullpen restores the dir, recipe is gone + cs.reset_to(project, 1) # restoring it must surface the pointer-without-recipe divergence # The xorq-catalog-coexistence section (catalog.yaml entries/aliases seeding, From ee0a90a31dd716834b90a93808cce9c26f2a18c4 Mon Sep 17 00:00:00 2001 From: Paddy Mullen Date: Wed, 17 Jun 2026 01:58:35 -0400 Subject: [PATCH 19/27] fix: cached_result_expr re-checks the baked snapshot on every read MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit The whole function was @lru_cache-memoised, including the snapshot existence-check + self-heal, so once an expensive entry was warm in the LRU, evicting its baked snapshot left the cached deferred_read_parquet pointing at a now-missing path — the next .execute() raised "At least one path is required". The cold self-heal (after cache_clear) worked, masking the warm gap. Split the expensive reconstruction (recipe import + rewrite_for_build + snapshot-path resolution) into a memoised _resolve_result_plan, and make cached_result_expr a thin non-memoised wrapper that re-checks path.exists() and self-heals on every call. The perf-namespace cold-read tag still fires once per genuine reconstruct; .cache_clear/.cache_info are re-exposed on the public name so existing callers (companion app, conftest, cache tests) keep working. Co-Authored-By: Claude Opus 4.8 (1M context) --- src/tallyman_xorq/result_cache.py | 95 ++++++++++++++++++++----------- 1 file changed, 61 insertions(+), 34 deletions(-) diff --git a/src/tallyman_xorq/result_cache.py b/src/tallyman_xorq/result_cache.py index 6a68645..1307445 100644 --- a/src/tallyman_xorq/result_cache.py +++ b/src/tallyman_xorq/result_cache.py @@ -230,6 +230,46 @@ def baked_snapshot_path(project: str, content_hash: str) -> Path | None: @functools.lru_cache(maxsize=256) +def _resolve_result_plan(project: str, content_hash: str): + """The expensive, memoisable half of ``cached_result_expr``: reconstruct the + recipe and decide how the entry's result is read. Returns either + ``("recompute", expr)`` for a cheap/fallback entry, or ``("baked", baked, path)`` + for an expensive entry whose snapshot ``cached_result_expr`` re-checks on every + call (so a snapshot evicted *after* a warm read still self-heals rather than + returning a dangling ``deferred_read_parquet``). + """ + t0 = time.monotonic() + + def _cold(tag): + perf_log.debug( + "cached_result_expr cold read %s: path=%s wall_ms=%.1f", + content_hash, + tag, + (time.monotonic() - t0) * 1000, + ) + + raw = _recipe_expr(project, content_hash) + if not cache_worthy(project, content_hash): + _cold("cheap-recompute") + return ("recompute", raw) + + # Expensive: locate the baked snapshot via the same rewrite the build used. + # rewrite_for_build wraps the expression in a ParquetSnapshotCache whose + # structural key matches the file baked at build time, so storage/calc_key + # give its on-disk path without re-executing. + from tallyman_xorq.source_cache import rewrite_for_build + + baked = rewrite_for_build(raw, project) + path = _cached_node_path(baked) + if path is None: + # classify_build (serialized) and _is_worthy_expr (live) disagreed — fall + # back to recompute rather than dereference a cache that wasn't baked. + _cold("worthy-recompute-fallback") + return ("recompute", raw) + _cold("baked-read") + return ("baked", baked, path) + + def cached_result_expr(project: str, content_hash: str): """The entry's result as a single-backend expression on the default backend. @@ -263,42 +303,29 @@ def cached_result_expr(project: str, content_hash: str): one via ``worthy-recompute-fallback`` (its ``rewrite_for_build`` returns the expression uncached under salt, so there is no snapshot to dereference). Both are single-backend recompute expressions — content-honest, no parquet. + + The expensive reconstruction is memoised in ``_resolve_result_plan``; the + snapshot's on-disk presence is re-checked HERE on every call, so an expensive + entry whose snapshot is evicted *after* a warm read self-heals on the next + read instead of handing back a stale ``deferred_read_parquet`` (which would + ``ValueError: At least one path is required`` at execute time). The + perf-namespace cold-read tag (#87) is emitted from the memoised half, so it + still fires once per genuine reconstruct. """ from xorq.expr.api import deferred_read_parquet - # This function is LRU-memoised, so its body runs only on a miss: every line - # below is a cold read. Tag each return with the #74 path it took and the - # reconstruct wall-clock (#82's lever) on the perf namespace (#87), so a - # Join-descendant that is structurally worthy but cost-cheap shows up on the - # read path, not just in lifecycle counts. - t0 = time.monotonic() - - def _read(expr, tag): - perf_log.debug( - "cached_result_expr cold read %s: path=%s wall_ms=%.1f", - content_hash, - tag, - (time.monotonic() - t0) * 1000, - ) - return expr - - raw = _recipe_expr(project, content_hash) - if not cache_worthy(project, content_hash): - return _read(raw, "cheap-recompute") - - # Expensive: locate the baked snapshot via the same rewrite the build used, - # then read it as a bare parquet. rewrite_for_build wraps the expression in a - # ParquetSnapshotCache whose structural key matches the file baked at build - # time, so storage/calc_key give its on-disk path without re-executing. - from tallyman_xorq.source_cache import rewrite_for_build - - baked = rewrite_for_build(raw, project) - path = _cached_node_path(baked) - if path is None: - # classify_build (serialized) and _is_worthy_expr (live) disagreed — fall - # back to recompute rather than dereference a cache that wasn't baked. - return _read(raw, "worthy-recompute-fallback") + plan = _resolve_result_plan(project, content_hash) + if plan[0] == "recompute": + return plan[1] + _, baked, path = plan if not path.exists(): baked.count().execute() # evicted snapshot: repopulate once, then read - return _read(deferred_read_parquet(str(path)), "evicted-self-heal") - return _read(deferred_read_parquet(str(path)), "baked-read") + perf_log.debug("cached_result_expr self-heal %s: evicted-self-heal", content_hash) + return deferred_read_parquet(str(path)) + + +# Existing callers clear the result-expr memo via cached_result_expr.cache_clear() +# (companion app, conftest, cache tests); the memo now lives on the plan resolver, +# so re-expose its cache controls on the public name. +cached_result_expr.cache_clear = _resolve_result_plan.cache_clear +cached_result_expr.cache_info = _resolve_result_plan.cache_info From e0296574564de0874d39557fa008111fc0297417 Mon Sep 17 00:00:00 2001 From: Paddy Mullen Date: Wed, 17 Jun 2026 02:02:21 -0400 Subject: [PATCH 20/27] feat: rebuild a project's catalog into the native store by re-execing recipes The plan's open item (#52, 'Rebuild + verify'): scripts/rebuild_native_catalog.py re-executes each entry's expr.py into the native content-addressed store, the single-user 'rebuild, don't migrate' path off the old xorq-catalog layout (catalog.yaml + metadata/ sidecars + aliases.json). Re-exec does not reproduce the old content hashes (the build pipeline + xorq serialization have moved, and the hash is sensitive to the absolute source path), so the rebuild treats hashes as not preserved: it topologically sorts the from_catalog graph, re-points alias refs at rebuilt parents before their children build, rewrites literal parent-hash refs old->new, expands the persisted placeholder back to the real path, and remaps charts + reconstructs alias history with the new hashes. Reads either aliases.jsonl (native) or aliases.json + alias_history.json (old), so it round-trips a native catalog and upgrades an old one through one code path. --dry-run prints the plan without writing. Validated by a native round-trip test (a same-path rebuild is hash-stable, so it asserts exact entry/alias-history/chart/prompt preservation) and end-to-end against a real old-format project. Co-Authored-By: Claude Opus 4.8 (1M context) --- scripts/rebuild_native_catalog.py | 283 +++++++++++++++++++++++++++ tests/test_rebuild_native_catalog.py | 119 +++++++++++ 2 files changed, 402 insertions(+) create mode 100644 scripts/rebuild_native_catalog.py create mode 100644 tests/test_rebuild_native_catalog.py diff --git a/scripts/rebuild_native_catalog.py b/scripts/rebuild_native_catalog.py new file mode 100644 index 0000000..4b7b2b1 --- /dev/null +++ b/scripts/rebuild_native_catalog.py @@ -0,0 +1,283 @@ +"""Rebuild a project's catalog into the native content-addressed store by +re-executing its recipes. + +Why a rebuild (not a migration). The catalog on-disk format changed with the +native store (#52): the old xorq-catalog layout (``catalog.yaml`` + per-entry +``metadata/.zip.metadata.yaml`` sidecars + ``aliases.json`` / +``alias_history.json`` + in-entry ``prompts.jsonl``) is replaced by a decomposed +tracked surface (``aliases.jsonl``, ``notebook.jsonl``, ``prompts/.jsonl``, +relocated ``post_processing`` / ``stats``) plus native ``entries/.zip`` +recipes. The project ``CLAUDE.md`` rule is "rebuild the corpus, don't write +migration code" — so this re-executes each entry's ``expr.py`` instead of +transcoding bytes. + +The hash-drift fact that shapes everything. A re-exec does NOT reproduce the old +content hashes — empirically the hash drifts even re-running the identical recipe +against the identical data at the same path (the build pipeline + xorq +serialization have moved since the corpus was built; the hash is also sensitive +to the absolute source path). So the rebuild treats hashes as NOT preserved and +remaps every hash-keyed artifact through an ``old_hash -> new_hash`` table built +as it goes: + +* recipes that chain by ALIAS (``from_catalog("citibike")``) survive untouched — + the alias is re-pointed at the rebuilt parent before the child is built; +* recipes that chain by literal HASH (``from_catalog("763193211746")``) are + rewritten in place (old hash -> new hash) before the child builds, which is why + the build order is a topological sort of the from_catalog dependency graph; +* charts (``chart_specs/.vl.json``) and notebook cells are remapped; +* alias history is reconstructed with the remapped hashes; +* per-entry prompt history is carried across to the new ``prompts/.jsonl``. + +Reads either the native ``aliases.jsonl`` or the old ``aliases.json`` + +``alias_history.json``, so it round-trips a native catalog and upgrades an old +one with the same code path. + +Usage:: + + TALLYMAN_HOME=~/.tallyman-notebooks uv run python scripts/rebuild_native_catalog.py first-project + # inspect the plan without writing anything: + TALLYMAN_HOME=~/.tallyman-notebooks uv run python scripts/rebuild_native_catalog.py first-project --dry-run + +The rebuild is IN-PLACE: it reads every recipe into memory, wipes the catalog +metadata (keeping ``data/``), re-execs, and checkpoints. Destructive to the +catalog bookkeeping (recipes are regenerated); the single-user no-migration rule +permits it. Use ``--dry-run`` first. +""" + +from __future__ import annotations + +import argparse +import json +import re +import shutil +import sys +from dataclasses import dataclass, field + +# 12-char lowercase-hex content hash (the entry-dir / pointer naming). +_HASH_RE = re.compile(r"\b[0-9a-f]{12}\b") +# from_catalog("ref") / from_catalog('ref', ...) — first positional arg only. +_FROM_CAT_RE = re.compile(r"from_catalog\(\s*['\"]([^'\"]+)['\"]") + + +@dataclass +class OldCatalog: + """Everything the rebuild needs, read out of the old catalog before it is wiped.""" + + recipes: dict[str, str] = field(default_factory=dict) # old_hash -> expr.py text + aliases: dict[str, str] = field(default_factory=dict) # alias -> current old_hash + history: dict[str, list[str]] = field(default_factory=dict) # alias -> [old_hash, ...] + charts: dict[str, str] = field(default_factory=dict) # old_hash -> vega-lite spec text + prompts: dict[str, list[dict]] = field(default_factory=dict) # old_hash -> [{prompt, at}, ...] + post_processing: dict[str, str] = field(default_factory=dict) # name -> source + stats: dict[str, str] = field(default_factory=dict) # name -> source + + +def read_old_catalog(project: str) -> OldCatalog: + """Read the pre-rebuild catalog. Tolerates both the native (``aliases.jsonl``) + and the old (``aliases.json`` + ``alias_history.json``) alias stores.""" + from tallyman_core.paths import catalog_dir # noqa: PLC0415 + + cat = catalog_dir(project) + oc = OldCatalog() + + entries = cat / "entries" + if entries.is_dir(): + for d in sorted(entries.iterdir()): + if d.is_dir() and (d / "expr.py").is_file() and (d / "manifest.json").is_file(): + oc.recipes[d.name] = (d / "expr.py").read_text() + pj = d / "prompts.jsonl" + if pj.is_file(): + oc.prompts[d.name] = [json.loads(x) for x in pj.read_text().splitlines() if x.strip()] + + # prompts may already be decomposed (native layout) under prompts/.jsonl + pdir = cat / "prompts" + if pdir.is_dir(): + for f in pdir.glob("*.jsonl"): + oc.prompts.setdefault(f.stem, [json.loads(x) for x in f.read_text().splitlines() if x.strip()]) + + aliases_jsonl = cat / "aliases.jsonl" + if aliases_jsonl.is_file(): + for line in aliases_jsonl.read_text().splitlines(): + if line.strip(): + rec = json.loads(line) + oc.aliases[rec["alias"]] = rec["latest"] + oc.history[rec["alias"]] = rec.get("history", []) + else: + aj = cat / "aliases.json" + if aj.is_file(): + oc.aliases = json.loads(aj.read_text()) + ah = cat / "alias_history.json" + if ah.is_file(): + oc.history = json.loads(ah.read_text()) + # an alias with no recorded history defaults to its single current hash + for name, h in oc.aliases.items(): + oc.history.setdefault(name, [h]) + + cs = cat / "chart_specs" + if cs.is_dir(): + for f in cs.glob("*.vl.json"): + oc.charts[f.name[: -len(".vl.json")]] = f.read_text() + + # post_processing / stats: old layout under artifacts/, native under the repo + for holder, dest in (("post_processing", oc.post_processing), ("stats", oc.stats)): + for base in (cat / holder, cat.parent / holder): + if base.is_dir(): + for f in base.glob("*.py"): + dest.setdefault(f.stem, f.read_text()) + return oc + + +def parse_deps(expr_text: str, aliases: dict[str, str], known: set[str]) -> set[str]: + """The set of old hashes this recipe depends on (alias refs resolved to their + current hash, literal hash refs taken as-is), restricted to entries we have.""" + out: set[str] = set() + for ref in _FROM_CAT_RE.findall(expr_text): + out.add(aliases.get(ref, ref)) + return out & known + + +def toposort(recipes: dict[str, str], aliases: dict[str, str]) -> list[str]: + """Dependency order (parents before children) over the from_catalog graph.""" + known = set(recipes) + dmap = {h: parse_deps(t, aliases, known) for h, t in recipes.items()} + order: list[str] = [] + placed: set[str] = set() + while len(placed) < len(recipes): + ready = sorted(h for h in recipes if h not in placed and dmap[h] <= placed) + if not ready: + stuck = {h: sorted(dmap[h] - placed) for h in recipes if h not in placed} + raise RuntimeError(f"unresolvable from_catalog dependencies (cycle or missing parent): {stuck}") + order.extend(ready) + placed.update(ready) + return order + + +def rewrite_hash_refs(expr_text: str, remap: dict[str, str]) -> str: + """Replace literal old-hash from_catalog refs with their rebuilt new hash. + + Only rewrites hashes already in *remap* (parents built earlier in topo order); + alias refs are left alone (the alias is re-pointed before the child builds). + """ + + def sub(m: re.Match) -> str: + return remap.get(m.group(0), m.group(0)) + + return _HASH_RE.sub(sub, expr_text) + + +def rebuild_project(project: str, *, dry_run: bool = False, log=print) -> dict[str, str]: + """Re-exec a project's recipes into the native store. Returns old->new hash map. + + On ``dry_run`` it reads the old catalog, prints the topo plan, and returns the + (empty) remap without writing anything. + """ + from tallyman_core import aliases as al # noqa: PLC0415 + from tallyman_core import ( # noqa: PLC0415 + catalog, + catalog_state, + set_active_project, # noqa: PLC0415 + ) + from tallyman_core.charts import set_chart # noqa: PLC0415 + from tallyman_core.paths import catalog_dir, ensure_project, project_dir, prompts_path # noqa: PLC0415 + from tallyman_core.post_processing import write_post_processing # noqa: PLC0415 + from tallyman_core.summary_stats import write_stat # noqa: PLC0415 + from tallyman_xorq import build_and_persist # noqa: PLC0415 + + oc = read_old_catalog(project) + if not oc.recipes: + raise RuntimeError(f"no rebuildable entries (expr.py + manifest.json) found in project {project!r}") + order = toposort(oc.recipes, oc.aliases) + log( + f"project {project!r}: {len(oc.recipes)} entries, {len(oc.aliases)} aliases, " + f"{len(oc.charts)} charts, {len(oc.post_processing)} post-processing, {len(oc.stats)} stats" + ) + log(f"build order (topological): {order}") + + if dry_run: + log("dry-run: no changes written.") + return {} + + cat = catalog_dir(project) + shutil.rmtree(cat) # wipe catalog bookkeeping; data/ is untouched + ensure_project(project) + set_active_project(project) # recipes resolve project_path/from_catalog against the active project + catalog_state.genesis(project) + + # alias whose CURRENT target is this old hash -> must be set right after it + # builds so a child's from_catalog(alias) resolves at build time. + cur_alias_for: dict[str, list[str]] = {} + for name, h in oc.aliases.items(): + cur_alias_for.setdefault(h, []).append(name) + + # The persisted recipe is portable: build.py rewrites the data path to a + # ${TALLYMAN_PROJECT_ROOT} placeholder. Expand it back so the re-exec reads + # the real source (recipes that read via project_path()/from_project() carry + # no placeholder, so this is a no-op for them). + root = str(project_dir(project)) + + remap: dict[str, str] = {} + for old_hash in order: + recipe = oc.recipes[old_hash].replace("${TALLYMAN_PROJECT_ROOT}", root) + recipe = rewrite_hash_refs(recipe, remap) # fix literal parent-hash refs + prompts = oc.prompts.get(old_hash, []) + first = prompts[0].get("prompt") if prompts else None + res = build_and_persist(project, recipe, prompt=first) + remap[old_hash] = res.content_hash + tag = "" if res.content_hash == old_hash else f" (rehashed -> {res.content_hash})" + log(f" built {old_hash}{tag}") + for name in cur_alias_for.get(old_hash, []): + al.set_alias(project, name, res.content_hash) + if len(prompts) > 1: # build wrote only the first; carry the rest + p = prompts_path(project, res.content_hash) + p.parent.mkdir(parents=True, exist_ok=True) + p.write_text("".join(json.dumps(x) + "\n" for x in prompts)) + + # full alias map + history, remapped (overwrites the incremental sets above) + al._write( + project, + {name: remap.get(h, h) for name, h in oc.aliases.items()}, + {name: [remap.get(x, x) for x in hs] for name, hs in oc.history.items()}, + ) + + for old_h, spec in oc.charts.items(): + nh = remap.get(old_h) + if nh: + set_chart(project, nh, spec) + else: + log(f" WARN chart for {old_h} dropped (no rebuilt entry)") + + for name, src in oc.post_processing.items(): + try: + write_post_processing(project, name, src) + except Exception as e: # noqa: BLE001 + log(f" WARN post-processing {name!r} failed re-validation: {e}") + for name, src in oc.stats.items(): + try: + write_stat(project, name, src) + except Exception as e: # noqa: BLE001 + log(f" WARN stat {name!r} failed re-validation: {e}") + + step = catalog_state.checkpoint_catalog(project, "rebuild into native catalog format") + log(f" checkpoint -> step {step}") + + pointers = set(catalog_state.read_tallyman_state(project)["entry_hashes"]) + catalog.assert_catalog_consistent(project, pointers) + log(f" consistency OK: {len(pointers)} entries in the native store") + return remap + + +def main(argv: list[str] | None = None) -> int: + ap = argparse.ArgumentParser(description="Rebuild a project's catalog into the native store by re-execing recipes.") + ap.add_argument("project", help="project name under TALLYMAN_HOME/projects") + ap.add_argument("--dry-run", action="store_true", help="read + print the build plan, write nothing") + args = ap.parse_args(argv) + try: + rebuild_project(args.project, dry_run=args.dry_run) + except Exception as e: # noqa: BLE001 + print(f"rebuild failed: {e}", file=sys.stderr) + return 1 + return 0 + + +if __name__ == "__main__": + raise SystemExit(main()) diff --git a/tests/test_rebuild_native_catalog.py b/tests/test_rebuild_native_catalog.py new file mode 100644 index 0000000..cb25a3b --- /dev/null +++ b/tests/test_rebuild_native_catalog.py @@ -0,0 +1,119 @@ +"""Round-trip test for ``scripts/rebuild_native_catalog.py``. + +Builds a small native catalog (an alias with a revision, a from_catalog child, +a chart, prompt history), rebuilds it through the script's importable core, and +asserts the rebuild re-execs every recipe, reconstructs the decomposed surface, +and leaves a consistent native store. A same-path rebuild is hash-stable (the +build pipeline is deterministic), so the entry set, alias history, and chart key +are preserved exactly. + +The script is loaded via importlib (scripts/ is not a package), mirroring how +test_perf_integration.py loads scripts/perf_report.py. +""" + +from __future__ import annotations + +import importlib.util +import sys +from pathlib import Path + +from tallyman_core import aliases as al +from tallyman_core import catalog +from tallyman_core import catalog_state as cs +from tallyman_core.charts import get_chart +from tallyman_xorq import build_and_persist, read_prompts +from tallyman_xorq.result_cache import cached_result_expr + +_REPO_ROOT = Path(__file__).resolve().parents[1] + + +def _load_rebuild(): + path = _REPO_ROOT / "scripts" / "rebuild_native_catalog.py" + spec = importlib.util.spec_from_file_location("rebuild_native_catalog", path) + mod = importlib.util.module_from_spec(spec) + assert spec and spec.loader + sys.modules[spec.name] = mod # so @dataclass can resolve cls.__module__ under importlib load + spec.loader.exec_module(mod) + return mod + + +def _agg(parquet: Path, *, avg: bool = False) -> str: + extra = ", avg=t.price.mean()" if avg else "" + return ( + "import xorq.api as xo\n" + f"t = xo.deferred_read_parquet({str(parquet)!r})\n" + f"expr = t.group_by('region').aggregate(n=t.count(){extra})\n" + ) + + +def _child(alias: str, project: str) -> str: + return ( + "from tallyman_xorq import from_catalog\n" + f"expr = from_catalog({alias!r}, {project!r}).filter(lambda r: r.n > 0)\n" + ) + + +def _build_corpus(project: str, parquet: Path) -> dict: + cs.genesis(project) + a = build_and_persist(project, _agg(parquet), prompt="agg by region") + al.set_alias(project, "regions", a.content_hash) + a2 = build_and_persist(project, _agg(parquet, avg=True), prompt="add avg") + al.set_alias(project, "regions", a2.content_hash) # revision: history [a, a2] + b = build_and_persist(project, _child("regions", project), prompt="filter regions") + from tallyman_core.charts import set_chart + + set_chart(project, a2.content_hash, {"mark": "bar", "encoding": {"x": {"field": "region"}}}) + cs.checkpoint_catalog(project, "corpus") + return {"a": a.content_hash, "a2": a2.content_hash, "b": b.content_hash} + + +def test_rebuild_preserves_entries_aliases_charts_prompts(project, orders_parquet): + rb = _load_rebuild() + before = _build_corpus(project, orders_parquet) + + remap = rb.rebuild_project(project, log=lambda *a: None) + + # Same-path rebuild is hash-stable: every entry rebuilt to its own hash. + assert set(remap) == set(before.values()) + assert all(remap[h] == h for h in before.values()), remap + + # Native store is internally consistent after the rebuild. + pointers = set(cs.read_tallyman_state(project)["entry_hashes"]) + assert pointers == set(before.values()) + catalog.assert_catalog_consistent(project, pointers) + + # Alias + its full revision history survived the round-trip. + assert al.get_alias(project, "regions") == before["a2"] + assert al.history_for(project, "regions") == [before["a"], before["a2"]] + + # Chart re-attached to the (preserved) hash; prompts carried across. + assert get_chart(project, before["a2"]) is not None + assert [p["prompt"] for p in read_prompts(project, before["a2"])] == ["add avg"] + + # Every rebuilt entry — including the from_catalog child — reloads. + for h in before.values(): + assert len(cached_result_expr(project, h).execute()) >= 0 + + +def test_rebuild_dry_run_writes_nothing(project, orders_parquet): + rb = _load_rebuild() + before = _build_corpus(project, orders_parquet) + head_before = cs.list_revisions(project) + + remap = rb.rebuild_project(project, dry_run=True, log=lambda *a: None) + + assert remap == {} # nothing rebuilt + # Catalog untouched: same entries, same revision timeline. + assert set(cs.read_tallyman_state(project)["entry_hashes"]) == set(before.values()) + assert cs.list_revisions(project) == head_before + + +def test_rebuild_toposort_orders_parents_before_children(project, orders_parquet): + rb = _load_rebuild() + recipes = { + "child1": "from tallyman_xorq import from_catalog\nexpr = from_catalog('base')\n", + "base000child": "import xorq.api as xo\nexpr = xo.memtable({'a': [1]})\n", + } + # "base" alias points at the root hash; the child must sort after it. + order = rb.toposort(recipes, {"base": "base000child"}) + assert order.index("base000child") < order.index("child1") From b51216fff91891666916d83b1f20171b17b955f0 Mon Sep 17 00:00:00 2001 From: Paddy Mullen Date: Wed, 17 Jun 2026 02:03:42 -0400 Subject: [PATCH 21/27] test: read aliases from the native aliases.jsonl in the Tier-B perf harness The native cut (#52) removed catalog.yaml / aliases.json / alias_history.json / the metadata sidecars, so the perf integration harness's alias discovery returned {} against a native corpus: _diff_pairs found no revision pairs and a TALLYMAN_PERF_ENTRIES= subset always skipped the run. Read the native aliases.jsonl first (latest + history per line), falling back to the legacy files so an old corpus still runs, and link the native decomposed surface in the overlay alongside the legacy names. Marker-gated (perf), local-only, never CI. Co-Authored-By: Claude Opus 4.8 (1M context) --- tests/test_perf_integration.py | 51 +++++++++++++++++++++++++++++----- 1 file changed, 44 insertions(+), 7 deletions(-) diff --git a/tests/test_perf_integration.py b/tests/test_perf_integration.py index ecca6bb..5eb67f9 100644 --- a/tests/test_perf_integration.py +++ b/tests/test_perf_integration.py @@ -99,7 +99,25 @@ assert set(_ENTRY_LINK_NAMES).isdisjoint(ENTRY_CACHE_NAMES) # no artifact is also a cache # Catalog bookkeeping the read paths need; the caches are deliberately omitted so # the overlay starts cold. -_CATALOG_LINK_NAMES = ("aliases.json", "alias_history.json", "catalog.yaml", "aliases", "chart_specs", "metadata") +_CATALOG_LINK_NAMES = ( + # native decomposed surface (#52+) + "aliases.jsonl", + "notebook.jsonl", + "entries.jsonl", + "compute_cache.jsonl", + "prompts", + "post_processing", + "stats", + "display_configs", + "chart_specs", + # legacy xorq-catalog layout — _link skips whichever are absent, so a native + # corpus links the files above and an old one links these. + "aliases.json", + "alias_history.json", + "catalog.yaml", + "aliases", + "metadata", +) # --------------------------------------------------------------------------- @@ -249,19 +267,38 @@ def _resolve_corpus_or_skip() -> tuple[str, Path, list[str]]: def _load_aliases(real_dir: Path) -> dict[str, str]: - p = real_dir / "artifacts" / "catalog" / "aliases.json" - return json.loads(p.read_text()) if p.exists() else {} + """Alias -> latest hash, from the native ``aliases.jsonl`` or legacy ``aliases.json``.""" + catalog = real_dir / "artifacts" / "catalog" + jsonl = catalog / "aliases.jsonl" + if jsonl.exists(): + out: dict[str, str] = {} + for line in jsonl.read_text().splitlines(): + if line.strip(): + rec = json.loads(line) + out[rec["alias"]] = rec["latest"] + return out + legacy = catalog / "aliases.json" + return json.loads(legacy.read_text()) if legacy.exists() else {} def _load_alias_history(real_dir: Path) -> dict[str, list[str]]: """Alias revision chains, from whichever store the corpus uses. - Pre-#57 catalogs kept these in ``alias_history.json``; a corpus rebuilt - through the fixed machinery (``scripts/rebuild_parking_catalog.py``) carries - them as the ``alias_history`` key in ``catalog.yaml``. Read the legacy file - if present, else fall back to the catalog.yaml key. + Native (#52+) keeps them as the ``history`` key per line of ``aliases.jsonl``. + Pre-#57 catalogs kept them in ``alias_history.json``; a corpus rebuilt through + the old ``scripts/rebuild_parking_catalog.py`` carries them as the + ``alias_history`` key in ``catalog.yaml``. Read native first, then the legacy + forms. """ catalog = real_dir / "artifacts" / "catalog" + jsonl = catalog / "aliases.jsonl" + if jsonl.exists(): + out: dict[str, list[str]] = {} + for line in jsonl.read_text().splitlines(): + if line.strip(): + rec = json.loads(line) + out[rec["alias"]] = rec.get("history", []) + return out legacy = catalog / "alias_history.json" if legacy.exists(): return json.loads(legacy.read_text()) From acaa03179f41bfcd705921e3107a05826169f7b5 Mon Sep 17 00:00:00 2001 From: Paddy Mullen Date: Wed, 17 Jun 2026 02:07:19 -0400 Subject: [PATCH 22/27] docs: record the bug-hunt hardening pass + close the rebuild-mechanism open item Four in-scope regressions found by the post-implementation bug-hunt and fixed test-first (allowlist nesting hole, interrupted-.tmp commit, manifest-less pointer, warm-evict self-heal), the out-of-scope pre-existing findings filed for separate decision, and the corpus-rebuild open item closed by scripts/rebuild_native_catalog.py (the eight real projects remain the user's to run). Co-Authored-By: Claude Opus 4.8 (1M context) --- plans/native-catalog-store.md | 59 ++++++++++++++++++++++++++++++----- 1 file changed, 52 insertions(+), 7 deletions(-) diff --git a/plans/native-catalog-store.md b/plans/native-catalog-store.md index 1b38cdf..40a144c 100644 --- a/plans/native-catalog-store.md +++ b/plans/native-catalog-store.md @@ -1,6 +1,7 @@ # Plan: native `tallyman_core.catalog` — drop xorq's catalog package -Status: **implemented** (2026-06-16) on `plan/native-catalog-store`. Refined +Status: **implemented** (2026-06-16) on `plan/native-catalog-store`, plus a +post-implementation bug-hunt + hardening pass (2026-06-17, see below). Refined through a parallel code investigation and a grilling pass; the decisions below are settled unless marked open. @@ -34,12 +35,56 @@ verify. Where reality diverged from the plan text: baked snapshot parquet is *not* byte-deterministic even for one backend (datafusion's group-by output order varies), so the meaningful equivalence is the content-addressed snapshot *key* + the *data*, not raw bytes. -- **Corpus rebuild deferred.** Eight on-disk projects under - `~/.tallyman-notebooks` are still old-format (`catalog.yaml` + xorq zips with - `metadata/` sidecars); they need rebuilding (the single-user rule permits it), - but the rebuild *mechanism* is the open item below, so it was not auto-run. - End-to-end verification ran on a fresh project instead - (`test_tracked_tree_is_the_decomposed_surface`). +- **Corpus rebuild — mechanism now exists, not yet run on the real corpus.** + Eight on-disk projects under `~/.tallyman-notebooks` are still old-format + (`catalog.yaml` + xorq zips with `metadata/` sidecars). `scripts/rebuild_native_catalog.py` + (added in the hardening pass) re-execs a project's recipes into the native + store; it was validated end-to-end against a disposable copy of `first-project` + and by `tests/test_rebuild_native_catalog.py`, but the eight real projects are + the user's to run (`--dry-run` first). Key fact the rebuild rests on: a re-exec + does **not** reproduce the old content hashes (the build pipeline + xorq + serialization have moved, and the hash is sensitive to the absolute source + path), so the rebuild remaps every hash-keyed artifact old→new, rewrites literal + `from_catalog()` refs, and expands the persisted `${TALLYMAN_PROJECT_ROOT}` + placeholder; alias-ref recipes survive untouched. + +## Post-implementation hardening (bug-hunt pass, 2026-06-17) + +A fan-out bug-hunt (eight surfaces, each adversarially probed and every finding +independently re-reproduced) over the native store + Cut B found the baseline +suite green on all eight and surfaced four in-scope regressions, all fixed +test-first: + +- **Allowlist let nested strays through.** `assert_catalog_consistent` matched + `TRACKED_SURFACE` with `fnmatch`, whose `*` spans `/`, so `post_processing/sub/leak.py` + matched `post_processing/*.py` (and `entries/sub/x.zip` matched `entries/*.zip`). + Now a path-segment-aware `_surface_match` (equal segment count + per-segment + `fnmatchcase`). +- **Interrupted atomic-write `.tmp` committed + wedged reset.** A crash between + `tmp.write_text` and `tmp.replace` (aliases/notebook/prompts writers, + `write_recipe_zip`) left a `*.tmp` the checkpoint's `git add -A` committed, + which then failed the allowlist on every later `reset_to`. Added `*.tmp` to the + repo `.gitignore`. +- **Manifest-less mid-build dir recorded as a pointer.** `capture_tallyman_state` + listed every entry dir, including a mid-build one without `manifest.json` that + `zip_pending_entries` skips — committing a pointer with no durable recipe, a + self-inconsistent step `reset_to` permanently rejected. Capture now applies the + same manifest filter the zip writer uses. +- **`cached_result_expr` self-heal broke once warm.** The whole function was + `@lru_cache`-memoised, so an expensive entry's snapshot evicted *after* a warm + read returned a dangling `deferred_read_parquet` (`"At least one path is + required"`). Split into a memoised plan resolver + a per-call existence check; + `.cache_clear`/`.cache_info` re-exposed on the public name. + +Out of scope (pre-existing, filed for separate decision, **not** fixed here): +reset-then-recheckpoint skips a step number and orphans the future-step tag; +`build_and_persist(project=X)` doesn't anchor X for the recipe's ambient +`project_path`/`from_catalog` resolution; `label_step` accepts `main` (the pinned +branch name); `version_of_hash`/`previous_version` use first-match `list.index`; +corrupt tracked JSON surfaces as a bare HTTP 500. + +End-to-end verification still runs on a fresh project +(`test_tracked_tree_is_the_decomposed_surface`). ## Problem From 40f09aa7d2520095c0578031ea6e6fd241016bc9 Mon Sep 17 00:00:00 2001 From: Paddy Mullen Date: Wed, 17 Jun 2026 02:20:39 -0400 Subject: [PATCH 23/27] feat: rebuild handles catalog.yaml-era corpora, self-chaining revises, in-process reuse Stress-testing the rebuild against the real corpus (a disposable copy of the 15-entry parking_ticket_analysis_1m) surfaced three gaps, all fixed: - Most real projects are catalog.yaml-era, not aliases.json-era: aliases live in alias_map/alias_history and charts/post_processing/stats/notebook are embedded in catalog.yaml with no decomposed files. read_old_catalog now reads each section from its decomposed file if present, else from catalog.yaml, and reconstructs the notebook (alias-keyed cells, no remap). - A self-chaining revise (from_catalog of one's own alias, #74) made the topo sort see a false self-cycle. parse_deps now resolves a self-alias-ref to the PRIOR revision in history (the build-time parent), and the rebuild re-points an alias at each revision as it is rebuilt, oldest-first. - An in-process rebuild left the process-global cached_result_expr memo populated with pre-wipe plans, so a from_catalog child re-execed against a stale parent. Clear the memo after wiping the catalog (a fresh-process CLI run is unaffected). Documents the remaining build-time-alias-revision limitation: a recipe chains a parent by alias with no version pin, so an alias revised after a child was built (with a schema change) re-points to its latest revision and the child can miss a column; pinning cross-alias refs to manifest.parents (#84) is the next step. Co-Authored-By: Claude Opus 4.8 (1M context) --- scripts/rebuild_native_catalog.py | 140 ++++++++++++++++++++++----- tests/test_rebuild_native_catalog.py | 71 +++++++++++++- 2 files changed, 182 insertions(+), 29 deletions(-) diff --git a/scripts/rebuild_native_catalog.py b/scripts/rebuild_native_catalog.py index 4b7b2b1..4dcf657 100644 --- a/scripts/rebuild_native_catalog.py +++ b/scripts/rebuild_native_catalog.py @@ -42,6 +42,18 @@ metadata (keeping ``data/``), re-execs, and checkpoints. Destructive to the catalog bookkeeping (recipes are regenerated); the single-user no-migration rule permits it. Use ``--dry-run`` first. + +Known limitation — build-time alias revision. A recipe chains a parent by ALIAS +with no version pin (``from_catalog("habitual_speeder_stats")``), so when that +alias was revised *after* a child was built and the revision changed the parent's +schema, the rebuild re-points the alias at its FINAL latest revision and the +child can fail to find a column the older revision had. The exact build-time +parent hash is recorded in ``manifest.parents`` (#84); pinning cross-alias refs +to it (rewriting the ref to the build-time parent hash, like the literal-hash +path already does) would close this — the next step for deeply-revised corpora. +Validated end-to-end on `first-project` (4 entries, alias chain) and the native +round-trip test; a 15-entry corpus with schema-evolving alias revisions +(`parking_ticket_analysis_1m`) reaches this limit. """ from __future__ import annotations @@ -70,16 +82,35 @@ class OldCatalog: prompts: dict[str, list[dict]] = field(default_factory=dict) # old_hash -> [{prompt, at}, ...] post_processing: dict[str, str] = field(default_factory=dict) # name -> source stats: dict[str, str] = field(default_factory=dict) # name -> source + notebook_cells: list[dict] = field(default_factory=list) # [{cell_id, alias, markdown}, ...] def read_old_catalog(project: str) -> OldCatalog: - """Read the pre-rebuild catalog. Tolerates both the native (``aliases.jsonl``) - and the old (``aliases.json`` + ``alias_history.json``) alias stores.""" + """Read the pre-rebuild catalog across all three layouts this project has seen: + + * native (#52+): ``aliases.jsonl`` + ``chart_specs/`` files + ``prompts/`` + + relocated ``post_processing`` / ``stats`` .py; + * aliases.json-era: ``aliases.json`` + ``alias_history.json`` + ``chart_specs/`` + + ``artifacts/{post_processing,stats}``; + * catalog.yaml-era (most of the real corpus): everything embedded in + ``catalog.yaml`` — ``alias_map`` / ``alias_history`` / ``charts`` / + ``post_processing`` / ``stats`` / ``notebook`` — with no decomposed files. + + Each section reads from its decomposed file if present, else falls back to + ``catalog.yaml``. + """ from tallyman_core.paths import catalog_dir # noqa: PLC0415 cat = catalog_dir(project) oc = OldCatalog() + cy = cat / "catalog.yaml" + ydata: dict = {} + if cy.is_file(): + import yaml # noqa: PLC0415 + + ydata = yaml.safe_load(cy.read_text()) or {} + entries = cat / "entries" if entries.is_dir(): for d in sorted(entries.iterdir()): @@ -102,14 +133,15 @@ def read_old_catalog(project: str) -> OldCatalog: rec = json.loads(line) oc.aliases[rec["alias"]] = rec["latest"] oc.history[rec["alias"]] = rec.get("history", []) - else: - aj = cat / "aliases.json" - if aj.is_file(): - oc.aliases = json.loads(aj.read_text()) + elif (cat / "aliases.json").is_file(): + oc.aliases = json.loads((cat / "aliases.json").read_text()) ah = cat / "alias_history.json" - if ah.is_file(): - oc.history = json.loads(ah.read_text()) - # an alias with no recorded history defaults to its single current hash + oc.history = json.loads(ah.read_text()) if ah.is_file() else {} + for name, h in oc.aliases.items(): # default a historyless alias to its current hash + oc.history.setdefault(name, [h]) + else: # catalog.yaml-era: aliases embedded as alias_map / alias_history + oc.aliases = dict(ydata.get("alias_map", {}) or {}) + oc.history = {k: list(v) for k, v in (ydata.get("alias_history", {}) or {}).items()} for name, h in oc.aliases.items(): oc.history.setdefault(name, [h]) @@ -117,29 +149,69 @@ def read_old_catalog(project: str) -> OldCatalog: if cs.is_dir(): for f in cs.glob("*.vl.json"): oc.charts[f.name[: -len(".vl.json")]] = f.read_text() + if not oc.charts: # catalog.yaml-era: charts as [{content_hash, spec}] + for rec in ydata.get("charts", []) or []: + oc.charts[rec["content_hash"]] = json.dumps(rec["spec"]) - # post_processing / stats: old layout under artifacts/, native under the repo + # post_processing / stats: decomposed .py (old artifacts/ layout or native under + # the repo), else the catalog.yaml-embedded [{name, source}] lists. for holder, dest in (("post_processing", oc.post_processing), ("stats", oc.stats)): for base in (cat / holder, cat.parent / holder): if base.is_dir(): for f in base.glob("*.py"): dest.setdefault(f.stem, f.read_text()) + if not dest: + for rec in ydata.get(holder, []) or []: + dest[rec["name"]] = rec["source"] + + # notebook cells (alias-keyed, so no hash remap): native notebook.jsonl else + # the catalog.yaml-embedded {cells: [...]}. + nb_jsonl = cat / "notebook.jsonl" + if nb_jsonl.is_file(): + oc.notebook_cells = [json.loads(x) for x in nb_jsonl.read_text().splitlines() if x.strip()] + else: + oc.notebook_cells = list((ydata.get("notebook", {}) or {}).get("cells", []) or []) return oc -def parse_deps(expr_text: str, aliases: dict[str, str], known: set[str]) -> set[str]: - """The set of old hashes this recipe depends on (alias refs resolved to their - current hash, literal hash refs taken as-is), restricted to entries we have.""" +def parse_deps( + expr_text: str, + self_hash: str, + aliases: dict[str, str], + history: dict[str, list[str]], + known: set[str], +) -> set[str]: + """The old hashes this recipe depends on, as the build resolved them. + + A literal hash ref is itself. An alias ref resolves to the alias's *build-time* + target: if this entry is a revision of that alias (the documented self-chaining + revise — ``from_catalog`` of one's own alias, #74), the dependency is the + PREVIOUS revision in the alias history, not the current latest (which would be + this very entry, a false self-cycle); otherwise it is the alias's current hash. + """ out: set[str] = set() for ref in _FROM_CAT_RE.findall(expr_text): - out.add(aliases.get(ref, ref)) + if ref in aliases: + hist = history.get(ref, []) + if self_hash in hist: + i = hist.index(self_hash) + dep = hist[i - 1] if i > 0 else None # build-time parent = the prior revision + else: + dep = aliases[ref] # a cross-alias chain resolves to the current latest + else: + dep = ref # literal hash pin + if dep: + out.add(dep) return out & known -def toposort(recipes: dict[str, str], aliases: dict[str, str]) -> list[str]: +def toposort( + recipes: dict[str, str], aliases: dict[str, str], history: dict[str, list[str]] | None = None +) -> list[str]: """Dependency order (parents before children) over the from_catalog graph.""" + history = history or {} known = set(recipes) - dmap = {h: parse_deps(t, aliases, known) for h, t in recipes.items()} + dmap = {h: parse_deps(t, h, aliases, history, known) for h, t in recipes.items()} order: list[str] = [] placed: set[str] = set() while len(placed) < len(recipes): @@ -186,7 +258,7 @@ def rebuild_project(project: str, *, dry_run: bool = False, log=print) -> dict[s oc = read_old_catalog(project) if not oc.recipes: raise RuntimeError(f"no rebuildable entries (expr.py + manifest.json) found in project {project!r}") - order = toposort(oc.recipes, oc.aliases) + order = toposort(oc.recipes, oc.aliases, oc.history) log( f"project {project!r}: {len(oc.recipes)} entries, {len(oc.aliases)} aliases, " f"{len(oc.charts)} charts, {len(oc.post_processing)} post-processing, {len(oc.stats)} stats" @@ -202,12 +274,24 @@ def rebuild_project(project: str, *, dry_run: bool = False, log=print) -> dict[s ensure_project(project) set_active_project(project) # recipes resolve project_path/from_catalog against the active project catalog_state.genesis(project) - - # alias whose CURRENT target is this old hash -> must be set right after it - # builds so a child's from_catalog(alias) resolves at build time. - cur_alias_for: dict[str, list[str]] = {} - for name, h in oc.aliases.items(): - cur_alias_for.setdefault(h, []).append(name) + # The catalog just changed under the process-global result-expr memo; a stale + # plan would make a from_catalog child re-exec against the pre-wipe parent. + # (A fresh-process CLI run has an empty memo; an in-process rebuild does not.) + from tallyman_xorq.result_cache import cached_result_expr # noqa: PLC0415 + + cached_result_expr.cache_clear() + + # Re-point an alias at each of its revisions AS that revision is rebuilt + # (topo order replays history oldest-first), so a later self-chaining revise + # resolves from_catalog(alias) to its build-time parent, and a cross-alias + # child resolves to the latest-built revision. Keyed on history membership, + # not just the final target. + revises_at: dict[str, list[str]] = {} + for name, hist in oc.history.items(): + for h in hist: + revises_at.setdefault(h, []).append(name) + if oc.aliases.get(name) not in hist and name in oc.aliases: + revises_at.setdefault(oc.aliases[name], []).append(name) # latest absent from history # The persisted recipe is portable: build.py rewrites the data path to a # ${TALLYMAN_PROJECT_ROOT} placeholder. Expand it back so the re-exec reads @@ -225,7 +309,7 @@ def rebuild_project(project: str, *, dry_run: bool = False, log=print) -> dict[s remap[old_hash] = res.content_hash tag = "" if res.content_hash == old_hash else f" (rehashed -> {res.content_hash})" log(f" built {old_hash}{tag}") - for name in cur_alias_for.get(old_hash, []): + for name in revises_at.get(old_hash, []): al.set_alias(project, name, res.content_hash) if len(prompts) > 1: # build wrote only the first; carry the rest p = prompts_path(project, res.content_hash) @@ -257,6 +341,14 @@ def rebuild_project(project: str, *, dry_run: bool = False, log=print) -> dict[s except Exception as e: # noqa: BLE001 log(f" WARN stat {name!r} failed re-validation: {e}") + # notebook cells are alias-keyed (no hash remap); re-append in order. + from tallyman_core import notebook # noqa: PLC0415 + + for cell in oc.notebook_cells: + alias = cell.get("alias") + if alias: + notebook.append(project, alias, markdown=cell.get("markdown")) + step = catalog_state.checkpoint_catalog(project, "rebuild into native catalog format") log(f" checkpoint -> step {step}") diff --git a/tests/test_rebuild_native_catalog.py b/tests/test_rebuild_native_catalog.py index cb25a3b..20401bf 100644 --- a/tests/test_rebuild_native_catalog.py +++ b/tests/test_rebuild_native_catalog.py @@ -73,16 +73,22 @@ def test_rebuild_preserves_entries_aliases_charts_prompts(project, orders_parque remap = rb.rebuild_project(project, log=lambda *a: None) - # Same-path rebuild is hash-stable: every entry rebuilt to its own hash. + # Every original entry was rebuilt. assert set(remap) == set(before.values()) - assert all(remap[h] == h for h in before.values()), remap + # A same-path rebuild is hash-stable for the recipe-deterministic roots. The + # from_catalog child bakes its aggregate parent's snapshot, whose bytes are + # not datafusion-order-deterministic, so its hash may drift — the rebuild + # remaps it (and re-points aliases) rather than relying on exact preservation. + assert remap[before["a"]] == before["a"] + assert remap[before["a2"]] == before["a2"] + new_hashes = set(remap.values()) # Native store is internally consistent after the rebuild. pointers = set(cs.read_tallyman_state(project)["entry_hashes"]) - assert pointers == set(before.values()) + assert pointers == new_hashes catalog.assert_catalog_consistent(project, pointers) - # Alias + its full revision history survived the round-trip. + # Alias + its full revision history survived the round-trip (stable roots). assert al.get_alias(project, "regions") == before["a2"] assert al.history_for(project, "regions") == [before["a"], before["a2"]] @@ -91,7 +97,7 @@ def test_rebuild_preserves_entries_aliases_charts_prompts(project, orders_parque assert [p["prompt"] for p in read_prompts(project, before["a2"])] == ["add avg"] # Every rebuilt entry — including the from_catalog child — reloads. - for h in before.values(): + for h in new_hashes: assert len(cached_result_expr(project, h).execute()) >= 0 @@ -117,3 +123,58 @@ def test_rebuild_toposort_orders_parents_before_children(project, orders_parquet # "base" alias points at the root hash; the child must sort after it. order = rb.toposort(recipes, {"base": "base000child"}) assert order.index("base000child") < order.index("child1") + + +def test_toposort_self_chaining_revise_resolves_to_prior_revision(project): + """A revision that chains off its own alias (the documented self-chaining + revise, #74) depends on the PRIOR revision, not itself — otherwise the + dependency graph has a false self-cycle and the topo sort cannot proceed.""" + rb = _load_rebuild() + recipes = { + "v1aaaaaaaaaa": "import xorq.api as xo\nexpr = xo.memtable({'x': [1]})\n", + "v2bbbbbbbbbb": "from tallyman_xorq import from_catalog\nexpr = from_catalog('a')\n", + } + aliases = {"a": "v2bbbbbbbbbb"} # latest is v2 + history = {"a": ["v1aaaaaaaaaa", "v2bbbbbbbbbb"]} + # v2's self-alias-ref resolves to v1 (its build-time parent), not itself. + assert rb.parse_deps(recipes["v2bbbbbbbbbb"], "v2bbbbbbbbbb", aliases, history, set(recipes)) == {"v1aaaaaaaaaa"} + assert rb.toposort(recipes, aliases, history) == ["v1aaaaaaaaaa", "v2bbbbbbbbbb"] + + +def test_read_old_catalog_reads_catalog_yaml_era(project): + """Most of the real corpus is catalog.yaml-era: aliases live in + alias_map/alias_history and charts/pp/stats/notebook are embedded in + catalog.yaml (no aliases.json, no chart_specs files). read_old_catalog must + read all of it, not just the aliases.json layout.""" + import yaml + + from tallyman_core.paths import catalog_dir, ensure_project, entry_dir + + rb = _load_rebuild() + ensure_project(project) + cd = catalog_dir(project) + cd.mkdir(parents=True, exist_ok=True) + for h in ("aaaa11112222", "bbbb33334444"): + ed = entry_dir(project, h) + ed.mkdir(parents=True) + (ed / "expr.py").write_text("expr = 1\n") + (ed / "manifest.json").write_text("{}") + (cd / "catalog.yaml").write_text( + yaml.safe_dump( + { + "alias_map": {"thing": "bbbb33334444"}, + "alias_history": {"thing": ["aaaa11112222", "bbbb33334444"]}, + "charts": [{"content_hash": "bbbb33334444", "spec": {"mark": "bar"}}], + "post_processing": [{"name": "pp1", "source": "def process(t):\n return t\n"}], + "stats": [{"name": "st1", "source": "def compute(c):\n return c.count()\n"}], + "notebook": {"cells": [{"cell_id": "x", "alias": "thing", "markdown": "note"}]}, + } + ) + ) + oc = rb.read_old_catalog(project) + assert oc.aliases == {"thing": "bbbb33334444"} + assert oc.history == {"thing": ["aaaa11112222", "bbbb33334444"]} + assert "bbbb33334444" in oc.charts + assert oc.post_processing.get("pp1") + assert oc.stats.get("st1") + assert [c["alias"] for c in oc.notebook_cells] == ["thing"] From b7acc1048f40327ec2beae64a864081cc48809c1 Mon Sep 17 00:00:00 2001 From: Paddy Mullen Date: Wed, 17 Jun 2026 11:33:22 -0400 Subject: [PATCH 24/27] test: failing tests for PR #103 review findings (M1/L1, L4, L6) MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit Bundles the failing-first tests for the verifiable behavioral findings from the PR #103 review, ahead of their fixes: - M1/L1 — assert_catalog_consistent must reject decomposed sidecars whose content hash names no durable recipe: a dangling alias latest hash, a dangling alias *history* hash, an orphan chart hash, and an orphan display-config hash. The guard currently validates only entries.jsonl ↔ entries/*.zip, so all four DID NOT RAISE. - L4 — a build that fails after the entry dir is populated but past the load_expr/execute paths (here make_portable_inplace) must leave no orphan entry dir. Only the two ad-hoc rmtree paths clean up today, so the dir lingers. (Drops the misleading monkeypatch.undo() that the existing no-orphan test also carries — undo() reverts isolated_home's TALLYMAN_HOME setenv on the shared monkeypatch, pointing entries_dir at the real home and making the check vacuous.) - L6 — notebook.load must fail loudly on a malformed JSONL line (matching aliases._read / read_prompts) rather than silently dropping it. Co-Authored-By: Claude Opus 4.8 (1M context) --- tests/test_native_catalog_store.py | 93 ++++++++++++++++++++++++++++++ tests/test_notebook.py | 13 +++++ 2 files changed, 106 insertions(+) diff --git a/tests/test_native_catalog_store.py b/tests/test_native_catalog_store.py index 3fbe55e..5bf725b 100644 --- a/tests/test_native_catalog_store.py +++ b/tests/test_native_catalog_store.py @@ -525,3 +525,96 @@ def test_checkpoint_skips_manifestless_entry_dir(project): assert pointers == catalog.tracked_recipe_hashes(project) # pointer set == durable recipe set catalog.assert_catalog_consistent(project, pointers) # must not raise cs.reset_to(project, step) # must not raise + + +# --------------------------------------------------------------------------- +# Cut A — the guard validates every decomposed hash-reference, not just +# entries.jsonl ↔ entries/*.zip (M1/L1) +# --------------------------------------------------------------------------- + + +def test_consistency_guard_surfaces_dangling_alias_hash(project, orders_parquet): + """A committed ``aliases.jsonl`` whose latest/history hash names no durable + recipe must fail the consistency guard, not reset clean (M1). A dangling + alias head later blows up the self-chaining ``from_catalog`` build, so + ``reset_to`` must refuse to restore the step rather than mask the divergence.""" + from tallyman_core import aliases as al + + cs.genesis(project) + res = build_and_persist(project, _agg_code(orders_parquet)) + al.set_alias(project, "ghost", "ffffffffffff", expect_exists=False) # no such entry + cs.checkpoint_catalog(project, "dangling alias") + pointers = set(cs.read_tallyman_state(project)["entry_hashes"]) + assert pointers == {res.content_hash} # the real entry's pointer/recipe agree (guard 2 passes) + with pytest.raises(RuntimeError, match="ffffffffffff"): + catalog.assert_catalog_consistent(project, pointers) + + +def test_consistency_guard_surfaces_dangling_alias_history_hash(project, orders_parquet): + """The alias check covers history, not only the latest head — a revised alias + whose *older* version's recipe is missing is just as dangling (M1: 'at minimum + cover aliases.jsonl history').""" + from tallyman_core import aliases as al + + cs.genesis(project) + res = build_and_persist(project, _agg_code(orders_parquet)) + al.set_alias(project, "regions", "ffffffffffff", expect_exists=False) # v1: no recipe + al.set_alias(project, "regions", res.content_hash) # v2: real head; history = [ffff…, res] + cs.checkpoint_catalog(project, "alias with a dangling history entry") + pointers = set(cs.read_tallyman_state(project)["entry_hashes"]) + with pytest.raises(RuntimeError, match="ffffffffffff"): + catalog.assert_catalog_consistent(project, pointers) + + +def test_consistency_guard_surfaces_orphan_chart_hash(project, orders_parquet): + """A committed ``chart_specs/.vl.json`` whose hash names no durable + recipe must fail the guard (L1) — the guard validates every content-addressed + sidecar against the live recipe set, not just the recipe↔pointer pair.""" + from tallyman_core import charts + + cs.genesis(project) + build_and_persist(project, _agg_code(orders_parquet)) + charts.set_chart(project, "ffffffffffff", {"mark": "bar"}) # no such entry + cs.checkpoint_catalog(project, "orphan chart") + pointers = set(cs.read_tallyman_state(project)["entry_hashes"]) + with pytest.raises(RuntimeError, match="ffffffffffff"): + catalog.assert_catalog_consistent(project, pointers) + + +def test_consistency_guard_surfaces_orphan_display_config_hash(project, orders_parquet): + """A committed ``display_configs/.json`` whose hash names no durable + recipe must fail the guard (L1), the display-config twin of the orphan-chart + case.""" + from tallyman_core import display_configs as dc + + cs.genesis(project) + build_and_persist(project, _agg_code(orders_parquet)) + dc.set_display_config(project, "ffffffffffff", {"pinned_rows": [1]}) # no such entry + cs.checkpoint_catalog(project, "orphan display config") + pointers = set(cs.read_tallyman_state(project)["entry_hashes"]) + with pytest.raises(RuntimeError, match="ffffffffffff"): + catalog.assert_catalog_consistent(project, pointers) + + +def test_failed_build_after_copy_leaves_no_orphan_dir(project, orders_parquet, monkeypatch): + """A build that fails AFTER the entry dir is populated but past the + load_expr/execute paths (here: ``make_portable_inplace``) must also leave no + orphan entry dir (L4). The D1 'no partial dirs' invariant covers every + population step, not just the two that had ad-hoc ``rmtree`` cleanup.""" + import tallyman_xorq.build as build_mod + + cs.genesis(project) + + def _boom(*a, **k): + raise RuntimeError("induced failure after the entry dir was populated") + + monkeypatch.setattr(build_mod, "make_portable_inplace", _boom) + with pytest.raises(Exception): + build_and_persist(project, _agg_code(orders_parquet)) + # No monkeypatch.undo(): it would also revert isolated_home's TALLYMAN_HOME + # setenv (same monkeypatch instance), pointing entries_dir at the real home + # and making this check vacuous. monkeypatch auto-reverts at teardown. + + ed = paths.entries_dir(project) + leftover = [c.name for c in ed.iterdir() if c.is_dir()] if ed.exists() else [] + assert not leftover, f"failed build left orphan entry dir(s): {leftover}" diff --git a/tests/test_notebook.py b/tests/test_notebook.py index 3433ff6..b964625 100644 --- a/tests/test_notebook.py +++ b/tests/test_notebook.py @@ -1,5 +1,6 @@ from __future__ import annotations +import json from pathlib import Path import pytest @@ -112,6 +113,18 @@ def test_remove_alias_from_cells(project: str): assert "keep_me" in aliases +def test_load_raises_on_malformed_line(project: str): + """A malformed JSONL line fails loudly (matching ``aliases._read`` / + ``read_prompts``) rather than being silently dropped (L6). A silent skip plus + the next ``_save`` would make the loss permanent; with atomic ``_save`` the + only source of a torn line is genuine corruption, which must surface.""" + notebook.append(project, "a", markdown="hi") + p = notebook._path(project) + p.write_text(p.read_text() + "{not json}\n") + with pytest.raises(json.JSONDecodeError): + notebook.load(project) + + # --------------------------------------------------------------------------- # MCP tools: catalog_create/revise/alias/rename interact with notebook # --------------------------------------------------------------------------- From e4e55277765f9dc2288b59efe253233f46f479a3 Mon Sep 17 00:00:00 2001 From: Paddy Mullen Date: Wed, 17 Jun 2026 11:47:37 -0400 Subject: [PATCH 25/27] =?UTF-8?q?fix:=20address=20PR=20#103=20review=20?= =?UTF-8?q?=E2=80=94=20consistency=20guard,=20atomic=20writes,=20cleanup?= MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit Resolves the confirmed findings from the PR #103 review. Integrity guard (M1/L1): assert_catalog_consistent gains a third guard — every decomposed sidecar keyed by content hash (aliases.jsonl latest+history, chart_specs/, display_configs/) must name a hash in the durable recipe set, so a committed step with a dangling alias/chart/display ref fails loudly instead of resetting clean. Charts/display are read from `git ls-files` (the committed tree) so a transient untracked sidecar isn't flagged. Notebook cells key on alias name, not hash, so they stay out of scope. Atomic writes (L2/L3/L5/C1/C2): new fsutil.atomic_write_text (tmp + os.replace) routes every tracked-surface writer — charts, display configs, notebook._save, manifest, post_processing/stats sources, the build's schema.json, the prompt append, and aliases._write (consolidated onto the shared helper). A crash or a cross-process checkpoint `git add -A` can no longer stage a torn file. Build cleanup (L4): the entry-dir population is wrapped so any failure past target.mkdir removes the partial dir, not just the load_expr/execute paths that had ad-hoc rmtrees (now dropped). The target.exists() early-return guarantees we created the dir, so cleanup can't clobber a prior entry. notebook.load (L6): malformed JSONL now raises (matching aliases._read / read_prompts) rather than silently dropping the line; with atomic _save a torn line is genuine corruption worth surfacing. Stale comments (L9): the result-cache delete endpoint and its test no longer claim cache_clear() averts the dangling-deref 500 — since ee0a90a cached_result_expr self-heals on every read; cache_clear is defense-in-depth. Coverage (M2/L8): test_alias_state_survives_reset (alias map+history roll back with git reset) and test_consistency_guard_surfaces_orphan_recipe (guard 2's orphan branch). Dropped the misleading monkeypatch.undo() from the no-orphan test — it reverted isolated_home's TALLYMAN_HOME and made the check vacuous. Plan doc (L11/L12): drop the settled .gitignore open item; fix the connect-shim test name/wording (test_connect_shim_matches_xorq_api_connect, key+data equivalence, not the never-shipped *_byte_identical). Co-Authored-By: Claude Opus 4.8 (1M context) --- plans/native-catalog-store.md | 25 ++- src/tallyman_companion/app.py | 13 +- src/tallyman_core/__init__.py | 2 + src/tallyman_core/aliases.py | 5 +- src/tallyman_core/catalog.py | 51 ++++- src/tallyman_core/charts.py | 4 +- src/tallyman_core/display_configs.py | 4 +- src/tallyman_core/fsutil.py | 32 +++ src/tallyman_core/manifest.py | 9 +- src/tallyman_core/notebook.py | 12 +- src/tallyman_core/post_processing.py | 4 +- src/tallyman_core/summary_stats.py | 4 +- src/tallyman_xorq/build.py | 310 ++++++++++++++------------- tests/test_cache_inspector.py | 10 +- tests/test_native_catalog_store.py | 61 +++++- 15 files changed, 358 insertions(+), 188 deletions(-) create mode 100644 src/tallyman_core/fsutil.py diff --git a/plans/native-catalog-store.md b/plans/native-catalog-store.md index 40a144c..5ed37dc 100644 --- a/plans/native-catalog-store.md +++ b/plans/native-catalog-store.md @@ -356,8 +356,10 @@ exception; scope the spy past the legitimate `cp -c` clone at `source_identity.py:111`); (b) `test_src_has_no_xorq_catalog_argv` — a static grep that `src/` has no `import xorq.catalog` / `from xorq.catalog` and no split-token `xorq … catalog … add` argv (skipping the `tallyman_mcp/server.py:219` codegen -docstring); and (c) `test_connect_shim_snapshot_is_byte_identical` — the datafusion -shim preserves byte-identical content-addressed snapshots. +docstring); and (c) `test_connect_shim_matches_xorq_api_connect` — the datafusion +shim derives the same content-addressed snapshot key and bakes the same data as +`xo.connect()` (key + data equivalence, not byte-identical — datafusion's group-by +output order isn't deterministic). ## Decisions (resolved in the grilling pass) @@ -434,7 +436,7 @@ shim preserves byte-identical content-addressed snapshots. |---|---|---|---| | 1 | `git add -A` is allow-by-default; a `.gitignore` denylist silently tracks any unlisted artifact (build dirs, `result.parquet`, future strays) | High | `.gitignore` (`entries/*/` etc.) covers today's heavy artifacts, but the durable guard is `assert_catalog_consistent` enforcing the tracked surface as a path **allowlist** (deny-by-default). Regression test asserts no build-dir file — and no unlisted path — is tracked. | | 2 | `sys.modules`-based "dropped" assertion is unsatisfiable (recipes load `xorq.api`) | High | Definition of done = subprocess guard + `src/` static grep, not `sys.modules`. | -| 3 | `xo.connect()` survives the `xorq.api`→`xorq.expr.api` swap as ibis's `connect(resource, …)` (present, not absent), so it fails at *call* time / builds the wrong backend rather than at import — easy to miss | High | Datafusion connect shim + `test_connect_shim_snapshot_is_byte_identical` (commit 1b) — was prose-only, now a named failing-first test. | +| 3 | `xo.connect()` survives the `xorq.api`→`xorq.expr.api` swap as ibis's `connect(resource, …)` (present, not absent), so it fails at *call* time / builds the wrong backend rather than at import — easy to miss | High | Datafusion connect shim + `test_connect_shim_matches_xorq_api_connect` (commit 1b) — was prose-only, now a named test asserting key + data equivalence (not byte-identical; see Implementation notes). | | 4 | `test_catalog_xorq_integration.py` imports `xorq.catalog.catalog.Catalog` **at module level** (`:45` also imports the to-be-deleted `xorq_catalog`) | High | Delete/rewrite in the **step-2** commit (same one that deletes `xorq_catalog.py`), not step 6 — a module-level import of the deleted module collection-errors the whole suite from step 2 on. Drop `_open_xorq_catalog`/alias-symlink tests; keep the `git ls-files` durability guards. | | 5 | Salt-mode zip must be named by the salted hash | High | Name `entries/.zip`; test under `TALLYMAN_SOURCE_IDENTITY=salt`. | | 6 | Commit timing inverts (inline → checkpoint); tests that build with no checkpoint and assert tracked fail | Med | `test_xorq_catalog_add_registers_after_genesis` is a **fix-induced break**, not failing-first: edit to checkpoint-then-assert in the step-6 cleanup commit. | @@ -506,10 +508,12 @@ before relying on any `subprocess.run` monkeypatch. blind spot. Optionally add the contrapositive `test_two_load_expr_results_cannot_union` to pin why `expr.py` is load-bearing. - `test_recipe_zip_contains_expr_py` — assert `expr.py` is a zip member (D3). - - `test_connect_shim_snapshot_is_byte_identical` — the shim-built and - `xo.connect()`-built `ParquetSnapshotCache` must produce byte-identical - content-addressed snapshots (Cut B / Risk #3; the no-arg `xo.connect()` sites are - `src/tallyman_xorq/source_cache.py:133,152`). The central Cut-B correctness claim. + - `test_connect_shim_matches_xorq_api_connect` — the shim-built and + `xo.connect()`-built `ParquetSnapshotCache` must derive the same + content-addressed snapshot key and bake the same data (Cut B / Risk #3; the + no-arg `xo.connect()` sites are `src/tallyman_xorq/source_cache.py:133,152`). + Key + data equivalence, not byte-identical — datafusion's group-by order isn't + deterministic. The central Cut-B correctness claim. - `test_src_has_no_xorq_catalog_argv` — static grep over `src/` for `import xorq.catalog` / `from xorq.catalog` **and** the split-token catalog argv (`_xorq(project, "catalog", "add", …)` in `xorq_catalog.py:31`, so a literal @@ -604,5 +608,8 @@ before relying on any `subprocess.run` monkeypatch. data), not just a structural roundtrip — xorq's `test_rebuild_produces_consistent_target` exists because a rebuild can match on schema/recipe/hash while silently returning wrong rows. -- Genesis baseline behavior post-decomposition and the exact `.gitignore` contents. - (The failing-first list is now settled above as commits 1a/1b.) +- Genesis baseline behavior post-decomposition. (The failing-first list is now + settled above as commits 1a/1b.) The `.gitignore` contents are no longer open: + they are concretely defined by `GITIGNORE_LINES` (`src/tallyman_core/catalog.py`) + and written by `write_gitignore`. The genuinely-open follow-up is the build-time + alias-revision limitation (#84). diff --git a/src/tallyman_companion/app.py b/src/tallyman_companion/app.py index 1ee234f..017d92f 100644 --- a/src/tallyman_companion/app.py +++ b/src/tallyman_companion/app.py @@ -1099,12 +1099,13 @@ async def api_delete_result_cache(project: str, content_hash: str): p.unlink() except OSError as exc: raise HTTPException(500, str(exc)) - # Drop the read-path memo so the snapshot actually self-heals on next read. - # A warm cached_result_expr holds a deferred_read of the file just unlinked; - # left in place, the next viewer read (api_data) — and ensure_session's - # pre-/load_expr heal — would dereference a missing file instead of - # re-baking. lru_cache has no per-key eviction, and a full clear is cheap - # for an admin delete: entries are content-addressed, so warm exprs simply + # Defense-in-depth, not load-bearing for correctness: since ee0a90a, + # cached_result_expr is a thin non-memoised wrapper that re-checks + # path.exists() and self-heals on every read, so the next viewer read + # (api_data) / ensure_session heal re-bakes the evicted snapshot even with a + # warm memo left in place (proven by + # test_cached_result_expr_self_heals_after_warm_then_evict). We still clear + # it because it's cheap for an admin delete: content-addressed entries just # re-reconstruct (and the evicted one re-bakes) on the next read. cached_result_expr.cache_clear() return {"ok": True, "hash": content_hash} diff --git a/src/tallyman_core/__init__.py b/src/tallyman_core/__init__.py index 7b4c6fe..a227b1f 100644 --- a/src/tallyman_core/__init__.py +++ b/src/tallyman_core/__init__.py @@ -20,6 +20,7 @@ write_display_klass, ) from tallyman_core.errors import clear_errors, get_error, list_errors, record_error +from tallyman_core.fsutil import atomic_write_text from tallyman_core.manifest import Manifest, read_manifest, write_manifest from tallyman_core.marimo_export import export_notebook_path, notebook_to_marimo from tallyman_core.notebook import CellNotFound @@ -93,6 +94,7 @@ "write_post_processing", "write_stat", "alias_for_hash", + "atomic_write_text", "artifacts_dir", "catalog_dir", "clear_errors", diff --git a/src/tallyman_core/aliases.py b/src/tallyman_core/aliases.py index 7714493..78b4a28 100644 --- a/src/tallyman_core/aliases.py +++ b/src/tallyman_core/aliases.py @@ -22,6 +22,7 @@ import json +from tallyman_core.fsutil import atomic_write_text from tallyman_core.paths import catalog_dir, ensure_project @@ -70,9 +71,7 @@ def _write(project: str, aliases: dict[str, str], history: dict[str, list[str]]) json.dumps({"alias": name, "latest": aliases[name], "history": history.get(name, [])}) + "\n" for name in sorted(aliases) ) - tmp = p.with_name(p.name + ".tmp") - tmp.write_text(body) - tmp.replace(p) + atomic_write_text(p, body) def load_aliases(project: str) -> dict[str, str]: diff --git a/src/tallyman_core/catalog.py b/src/tallyman_core/catalog.py index 9be97a8..565e2dd 100644 --- a/src/tallyman_core/catalog.py +++ b/src/tallyman_core/catalog.py @@ -78,7 +78,7 @@ ".checkpoint.lock", ".xorq_build_expanded/", ".migrated_*", # machine-local upgrade-migration markers (e.g. #104), not catalog content - "*.tmp", # an interrupted atomic-write temp (aliases._write, write_recipe_zip, ...) — never a committable artifact + "*.tmp", # an interrupted atomic write's temp (atomic_write_text / write_recipe_zip), never committable ) @@ -179,8 +179,23 @@ def zip_pending_entries(project: str) -> list[str]: return written +def _committed_sidecar_hashes(cd: Path, prefix: str, suffix: str) -> set[str]: + """Content hashes of the git-tracked ``/`` sidecars. + + Reads ``git ls-files`` (the committed tree, like ``tracked_recipe_hashes``), + so a transient *untracked* sidecar on disk — set but not yet checkpointed — + is never mistaken for a committed orphan. + """ + rc, out, _ = run_git(["ls-files", prefix], cwd=cd) + if rc != 0: + return set() + return { + ln[len(prefix) + 1 : -len(suffix)] for ln in out.split() if ln.startswith(f"{prefix}/") and ln.endswith(suffix) + } + + def assert_catalog_consistent(project: str, pointers: set[str]) -> None: - """Two guards over the committed tree (#52), raising on either violation. + """Three guards over the committed tree (#52), raising on any violation. 1. **Tracked-surface allowlist** (deny-by-default): every ``git ls-files`` path must fnmatch ``TRACKED_SURFACE`` or it raises — so a stray dir that @@ -189,6 +204,14 @@ def assert_catalog_consistent(project: str, pointers: set[str]) -> None: (``entries/.zip``) must equal the tallyman *pointers* (``entry_hashes``), so a pointer with no durable recipe — or a recipe with no pointer — fails loudly instead of returning a masked divergence. + 3. **Sidecar hash-reference integrity**: every decomposed file keyed by + content hash (``aliases.jsonl`` latest+history, ``chart_specs/``, + ``display_configs/``) must name a hash in the durable recipe set. + Guard 2 only pairs recipes with pointers; without this a committed step + whose alias/chart/display points at a hash with no recipe would reset + clean and report OK — and a dangling alias head later blows up the + self-chaining ``from_catalog`` build. (Notebook cells key on alias *name*, + not hash, so they are deliberately out of scope here.) Takes *pointers* as an argument (rather than reading ``catalog_state``) to keep the call direction one-way and acyclic. @@ -210,3 +233,27 @@ def assert_catalog_consistent(project: str, pointers: set[str]) -> None: "catalog inconsistent: the git-tracked recipe set and the entry_hashes pointers disagree " f"(pointers with no durable recipe: {missing}; tracked recipes with no pointer: {orphan})" ) + + # Guard 3 — sidecar hash-reference integrity. Function-level import keeps the + # call direction one-way: aliases imports only paths/fsutil, never catalog or + # catalog_state, so this stays acyclic. Aliases are read on-disk (== the + # committed tree after reset_to's git reset --hard); charts/display from git. + from tallyman_core import aliases as _aliases + + alias_refs: set[str] = set(_aliases.load_aliases(project).values()) + for history in _aliases.load_history(project).values(): + alias_refs.update(history) + + dangling: dict[str, list[str]] = {} + bad_aliases = sorted(h for h in alias_refs if h not in tracked) + if bad_aliases: + dangling["aliases.jsonl"] = bad_aliases + for prefix, suffix in (("chart_specs", ".vl.json"), ("display_configs", ".json")): + bad = sorted(h for h in _committed_sidecar_hashes(cd, prefix, suffix) if h not in tracked) + if bad: + dangling[prefix] = bad + if dangling: + raise RuntimeError( + "catalog inconsistent: decomposed sidecars name content hashes with no durable recipe " + f"(orphans by surface: {dangling})" + ) diff --git a/src/tallyman_core/charts.py b/src/tallyman_core/charts.py index b2e90c2..7779d10 100644 --- a/src/tallyman_core/charts.py +++ b/src/tallyman_core/charts.py @@ -14,6 +14,7 @@ import json from pathlib import Path +from tallyman_core.fsutil import atomic_write_text from tallyman_core.paths import catalog_dir @@ -44,8 +45,7 @@ def set_chart(project: str, content_hash: str, spec) -> Path: out = chart_path(project, content_hash) out.parent.mkdir(parents=True, exist_ok=True) - out.write_text(json.dumps(spec, indent=2)) - return out + return atomic_write_text(out, json.dumps(spec, indent=2)) def get_chart(project: str, content_hash: str) -> dict | None: diff --git a/src/tallyman_core/display_configs.py b/src/tallyman_core/display_configs.py index f207826..0e39c67 100644 --- a/src/tallyman_core/display_configs.py +++ b/src/tallyman_core/display_configs.py @@ -11,6 +11,7 @@ import json from pathlib import Path +from tallyman_core.fsutil import atomic_write_text from tallyman_core.paths import catalog_dir @@ -26,8 +27,7 @@ def set_display_config(project: str, content_hash: str, config: dict) -> Path: """Persist a display configuration for content_hash.""" out = config_path(project, content_hash) out.parent.mkdir(parents=True, exist_ok=True) - out.write_text(json.dumps(config, indent=2)) - return out + return atomic_write_text(out, json.dumps(config, indent=2)) def get_display_config(project: str, content_hash: str) -> dict | None: diff --git a/src/tallyman_core/fsutil.py b/src/tallyman_core/fsutil.py new file mode 100644 index 0000000..1854754 --- /dev/null +++ b/src/tallyman_core/fsutil.py @@ -0,0 +1,32 @@ +"""Filesystem helpers shared across the tracked catalog surface. + +The decomposed catalog files (aliases, notebook, charts, display configs, +post-processing/stats sources, the entry manifest + schema, prompt history) are +rewritten in place by their own mutators. A bare ``write_text`` is +``.is_file()``-true the instant it is truncated, so a crash mid-write — or the +checkpoint's ``git add -A`` firing in the write window from a separate process — +can stage a torn file a later reader chokes on. Routing every such writer +through ``atomic_write_text`` makes the rewrite a single rename: the destination +is either the old bytes or the new bytes, never a prefix. +""" + +from __future__ import annotations + +from pathlib import Path + + +def atomic_write_text(path: Path, text: str) -> Path: + """Write *text* to *path* atomically (write a sibling ``*.tmp``, then + ``os.replace`` it into place). + + The tmp sibling shares *path*'s directory so the replace is a + same-filesystem rename (atomic on POSIX). ``*.tmp`` is gitignored + (``catalog.GITIGNORE_LINES``), so an interrupted write leaves an + uncommittable temp rather than a truncated *path*. The caller ensures the + parent dir exists (every catalog writer already ``mkdir(parents=True)``s). + """ + path = Path(path) + tmp = path.with_name(path.name + ".tmp") + tmp.write_text(text) + tmp.replace(path) + return path diff --git a/src/tallyman_core/manifest.py b/src/tallyman_core/manifest.py index 6d2bf79..0c5016c 100644 --- a/src/tallyman_core/manifest.py +++ b/src/tallyman_core/manifest.py @@ -6,6 +6,7 @@ from pydantic import BaseModel, Field +from tallyman_core.fsutil import atomic_write_text from tallyman_core.paths import ( ENTRY_MANIFEST_FILENAME, ENTRY_SCHEMA_FILENAME, @@ -56,9 +57,13 @@ class Manifest(BaseModel): def write_manifest(entry_path: Path, manifest: Manifest) -> Path: + # Atomic: manifest.json is the build's completeness sentinel — zip_pending_entries + # gates on (child / "manifest.json").is_file(), and the checkpoint runs from a + # separate process. A plain write_text is .is_file()-true the instant it is + # truncated, so a checkpoint firing in the write window could zip a partial + # manifest member; tmp + replace makes the member appear whole or not at all. out = entry_path / ENTRY_MANIFEST_FILENAME - out.write_text(json.dumps(manifest.model_dump(), indent=2)) - return out + return atomic_write_text(out, json.dumps(manifest.model_dump(), indent=2)) def read_manifest(entry_path: Path) -> Manifest: diff --git a/src/tallyman_core/notebook.py b/src/tallyman_core/notebook.py index 0ee3e2b..9fdba1f 100644 --- a/src/tallyman_core/notebook.py +++ b/src/tallyman_core/notebook.py @@ -20,6 +20,7 @@ import uuid from pathlib import Path +from tallyman_core.fsutil import atomic_write_text from tallyman_core.paths import catalog_dir, ensure_project @@ -40,10 +41,11 @@ def load(project: str, name: str = "default") -> dict: for line in p.read_text().splitlines(): if not line.strip(): continue - try: - cells.append(json.loads(line)) - except json.JSONDecodeError: - continue + # Fail loudly on a malformed line, matching aliases._read / read_prompts. + # _save is atomic (no torn writes), so a bad line is genuine corruption; + # silently dropping it here would let the next _save make the loss + # permanent. notebook.jsonl is git-tracked, so the fix is recoverable. + cells.append(json.loads(line)) return {"cells": cells} @@ -55,7 +57,7 @@ def _save(project: str, data: dict, name: str = "default") -> None: if not cells: p.unlink(missing_ok=True) # an empty notebook is no file (a clean diff) return - p.write_text("".join(json.dumps(c) + "\n" for c in cells)) + atomic_write_text(p, "".join(json.dumps(c) + "\n" for c in cells)) def find_cell_by_alias(project: str, alias: str, name: str = "default") -> dict | None: diff --git a/src/tallyman_core/post_processing.py b/src/tallyman_core/post_processing.py index 17777ef..f97e7f1 100644 --- a/src/tallyman_core/post_processing.py +++ b/src/tallyman_core/post_processing.py @@ -27,6 +27,7 @@ import re from pathlib import Path +from tallyman_core.fsutil import atomic_write_text from tallyman_core.paths import post_processing_dir as _post_processing_dir # Mirrors the allowlist in ``buckaroo/server/xorq_loading.py``. Keeping @@ -197,8 +198,7 @@ def write_post_processing(project: str, name: str, source: str) -> Path: d = post_processing_dir(project) d.mkdir(parents=True, exist_ok=True) path = d / f"{name}.py" - path.write_text(source if source.endswith("\n") else source + "\n") - return path + return atomic_write_text(path, source if source.endswith("\n") else source + "\n") class PostProcessingRunError(ValueError): diff --git a/src/tallyman_core/summary_stats.py b/src/tallyman_core/summary_stats.py index 1a43dfc..7ddc7cc 100644 --- a/src/tallyman_core/summary_stats.py +++ b/src/tallyman_core/summary_stats.py @@ -26,6 +26,7 @@ import re from pathlib import Path +from tallyman_core.fsutil import atomic_write_text from tallyman_core.paths import stats_dir as _stats_dir # Mirrors the allowlist in ``buckaroo/server/xorq_loading.py``. Keeping @@ -180,8 +181,7 @@ def write_stat(project: str, name: str, source: str) -> Path: d = stats_dir(project) d.mkdir(parents=True, exist_ok=True) path = d / f"{name}.py" - path.write_text(source if source.endswith("\n") else source + "\n") - return path + return atomic_write_text(path, source if source.endswith("\n") else source + "\n") def remove_stat(project: str, name: str) -> Path | None: diff --git a/src/tallyman_xorq/build.py b/src/tallyman_xorq/build.py index 4770ce9..ea87c2f 100644 --- a/src/tallyman_xorq/build.py +++ b/src/tallyman_xorq/build.py @@ -17,6 +17,7 @@ from tallyman_core import ( ENTRY_MANIFEST_FILENAME, Manifest, + atomic_write_text, ensure_project, entries_dir, entry_build_dir, @@ -50,8 +51,12 @@ def _append_prompt(project: str, content_hash: str, prompt: str | None) -> None: } p = prompts_path(project, content_hash) p.parent.mkdir(parents=True, exist_ok=True) - with p.open("a") as fh: - fh.write(json.dumps(record) + "\n") + # Atomic read-modify-write rather than an O_APPEND write: _append_prompt runs + # outside the project lock (only the checkpoint holds it), so the checkpoint's + # git add -A can fire mid-append from a separate process. tmp + replace makes + # prompts/.jsonl always whole — never a torn trailing line (C2). + existing = p.read_text() if p.exists() else "" + atomic_write_text(p, existing + json.dumps(record) + "\n") def read_prompts(project: str, content_hash: str) -> list[dict]: @@ -383,157 +388,166 @@ def build_and_persist( except InMemoryReadError as exc: raise BuildError(str(exc)) from exc - # Use a temp builds_dir so xorq's hash naming doesn't collide; we move - # things into our catalog layout afterwards. - with tempfile.TemporaryDirectory(prefix="tallyman_build_") as builds_str: - builds_dir = Path(builds_str) - try: - build_path = Path(build_expr(expr_obj, builds_dir=builds_dir)) - except Exception as exc: - hint = _ibis_import_hint(str(exc), code) - raise BuildError(f"build_expr failed: {exc}{hint}\n{traceback.format_exc()}") from exc - - content_hash = build_path.name - if si.mode() == "salt" and sources: - # xorq's hash is path-identity only; mixing the source digests in - # makes the entry hash content-sensitive (same shape, 12 hex). - content_hash = si.salted_hash(content_hash, sources) - target = entry_dir(project, content_hash) - - if target.exists(): - # Same content hash already on disk — nothing to do beyond return. - # Record this invocation's prompt as a re-run event. - manifest_path = entry_manifest_path(project, content_hash) - if manifest_path.exists(): - meta = json.loads(manifest_path.read_text()) - _append_prompt(project, content_hash, prompt) - return BuildResult( - content_hash=content_hash, - entry_path=target, - row_count=meta.get("row_count", 0), - execute_seconds=meta.get("execute_seconds", 0.0), - schema=json.loads(entry_schema_path(project, content_hash).read_text()), - lint_warnings=lint_warnings, - compile_seconds=meta.get("compile_seconds"), - cache_worthy=meta.get("cache_worthy"), - cache_worthy_why=meta.get("cache_worthy_why"), - cache_bytes=meta.get("cache_bytes"), - ) - - target.mkdir(parents=True, exist_ok=True) - - # Move xorq's build directory contents (expr.yaml + deps) under the entry. - xorq_build_dir = entry_build_dir(project, content_hash) - xorq_build_dir.mkdir(exist_ok=True) - for item in build_path.rglob("*"): - dest = xorq_build_dir / item.relative_to(build_path) - if item.is_dir(): - dest.mkdir(parents=True, exist_ok=True) - else: - dest.parent.mkdir(parents=True, exist_ok=True) - dest.write_bytes(item.read_bytes()) - - # Rewrite project-root substrings to a portable placeholder so the - # build is loadable on any machine with the same project laid out. - make_portable_inplace(xorq_build_dir, project_dir(project)) - - # Persist the user's source, also rewriting any project-root paths. - code_persisted = code.replace(str(project_dir(project)), "${TALLYMAN_PROJECT_ROOT}") - (target / "expr.py").write_text(code_persisted) - - # Load + execute. - try: - # Per-project compute cache (not the global ~/.cache/xorq), so a - # reset can prune it to the revision's warm-set and a freshly added - # expression computes cold. See plans/adr-reset-to-revision.md. - cache_dir = compute_cache_dir(project) - cache_dir.mkdir(parents=True, exist_ok=True) - loaded = load_expr(build_path, cache_dir=cache_dir) - except Exception as exc: - shutil.rmtree(target, ignore_errors=True) # no orphan dir → no orphan pointer (D1) - hint = _ibis_import_hint(str(exc), code) - raise BuildError(f"load_expr failed: {exc}{hint}\n{traceback.format_exc()}") from exc - - # Execute (#73). A worthy entry's build carries a baked result-cache - # node (rewrite_for_build), so executing materialises it once into the - # compute cache — and every later loader (the Buckaroo viewer, diffs, - # from_catalog) reads that cached result instead of re-running the DAG. - # A cheap entry materialises nothing: this is a pushdown count() over a - # columnar source plus a schema read, no parquet written. - t0 = time.monotonic() - try: - arrow_schema = loaded.schema().to_pyarrow() - row_count = int(loaded.count().execute()) - except Exception as exc: - shutil.rmtree(target, ignore_errors=True) # no orphan dir → no orphan pointer (D1) - hint = _ibis_import_hint(str(exc), code) - raise BuildError(f"build execution failed: {exc}{hint}\n{traceback.format_exc()}") from exc - execute_seconds = round(time.monotonic() - t0, 3) - - # Schema + row count come from the expression directly (no result.parquet to - # read back); a worthy entry's rows are already materialised in its cache. - schema_doc = { - "fields": [{"name": f.name, "type": str(f.type)} for f in arrow_schema], - "row_count": row_count, - } - entry_schema_path(project, content_hash).write_text(json.dumps(schema_doc, indent=2)) - - # Cache-admission instrumentation (#87): record the structural verdict and - # the measured value-per-byte inputs side by side, so #30's structural→ - # measured flip is decidable from data. classify_build's verdict is what - # cache_worthy already computes (and throws away); compile_seconds times the - # author DAG's expr->backend-plan step (the dominant per-view cost per #30), - # separate from execute; cache_bytes is the baked snapshot size (None for a - # cheap entry that bakes nothing). loaded is the build_path expression whose - # execute just baked the snapshot, so its CachedNode names that exact file. - from tallyman_xorq.result_cache import _cached_node_path, classify_build - - verdict = classify_build(xorq_build_dir) - cache_worthy_v, cache_worthy_why = verdict["worthy"], verdict["why"] - - compile_seconds: float | None = None + created_target = False try: - from xorq.expr.api import to_sql - - t_compile = time.monotonic() - to_sql(author_expr) - compile_seconds = round(time.monotonic() - t_compile, 3) - except Exception: - # Best-effort: a DAG xorq can't render to SQL must not break the build. - pass - - cache_bytes: int | None = None - snap = _cached_node_path(loaded) - if snap is not None and snap.exists(): + # Use a temp builds_dir so xorq's hash naming doesn't collide; we move + # things into our catalog layout afterwards. + with tempfile.TemporaryDirectory(prefix="tallyman_build_") as builds_str: + builds_dir = Path(builds_str) + try: + build_path = Path(build_expr(expr_obj, builds_dir=builds_dir)) + except Exception as exc: + hint = _ibis_import_hint(str(exc), code) + raise BuildError(f"build_expr failed: {exc}{hint}\n{traceback.format_exc()}") from exc + + content_hash = build_path.name + if si.mode() == "salt" and sources: + # xorq's hash is path-identity only; mixing the source digests in + # makes the entry hash content-sensitive (same shape, 12 hex). + content_hash = si.salted_hash(content_hash, sources) + target = entry_dir(project, content_hash) + + if target.exists(): + # Same content hash already on disk — nothing to do beyond return. + # Record this invocation's prompt as a re-run event. + manifest_path = entry_manifest_path(project, content_hash) + if manifest_path.exists(): + meta = json.loads(manifest_path.read_text()) + _append_prompt(project, content_hash, prompt) + return BuildResult( + content_hash=content_hash, + entry_path=target, + row_count=meta.get("row_count", 0), + execute_seconds=meta.get("execute_seconds", 0.0), + schema=json.loads(entry_schema_path(project, content_hash).read_text()), + lint_warnings=lint_warnings, + compile_seconds=meta.get("compile_seconds"), + cache_worthy=meta.get("cache_worthy"), + cache_worthy_why=meta.get("cache_worthy_why"), + cache_bytes=meta.get("cache_bytes"), + ) + + target.mkdir(parents=True, exist_ok=True) + created_target = True # reached only past the target.exists() early-return + + # Move xorq's build directory contents (expr.yaml + deps) under the entry. + xorq_build_dir = entry_build_dir(project, content_hash) + xorq_build_dir.mkdir(exist_ok=True) + for item in build_path.rglob("*"): + dest = xorq_build_dir / item.relative_to(build_path) + if item.is_dir(): + dest.mkdir(parents=True, exist_ok=True) + else: + dest.parent.mkdir(parents=True, exist_ok=True) + dest.write_bytes(item.read_bytes()) + + # Rewrite project-root substrings to a portable placeholder so the + # build is loadable on any machine with the same project laid out. + make_portable_inplace(xorq_build_dir, project_dir(project)) + + # Persist the user's source, also rewriting any project-root paths. + code_persisted = code.replace(str(project_dir(project)), "${TALLYMAN_PROJECT_ROOT}") + (target / "expr.py").write_text(code_persisted) + + # Load + execute. + try: + # Per-project compute cache (not the global ~/.cache/xorq), so a + # reset can prune it to the revision's warm-set and a freshly added + # expression computes cold. See plans/adr-reset-to-revision.md. + cache_dir = compute_cache_dir(project) + cache_dir.mkdir(parents=True, exist_ok=True) + loaded = load_expr(build_path, cache_dir=cache_dir) + except Exception as exc: + hint = _ibis_import_hint(str(exc), code) + raise BuildError(f"load_expr failed: {exc}{hint}\n{traceback.format_exc()}") from exc + + # Execute (#73). A worthy entry's build carries a baked result-cache + # node (rewrite_for_build), so executing materialises it once into the + # compute cache — and every later loader (the Buckaroo viewer, diffs, + # from_catalog) reads that cached result instead of re-running the DAG. + # A cheap entry materialises nothing: this is a pushdown count() over a + # columnar source plus a schema read, no parquet written. + t0 = time.monotonic() + try: + arrow_schema = loaded.schema().to_pyarrow() + row_count = int(loaded.count().execute()) + except Exception as exc: + hint = _ibis_import_hint(str(exc), code) + raise BuildError(f"build execution failed: {exc}{hint}\n{traceback.format_exc()}") from exc + execute_seconds = round(time.monotonic() - t0, 3) + + # Schema + row count come from the expression directly (no result.parquet to + # read back); a worthy entry's rows are already materialised in its cache. + schema_doc = { + "fields": [{"name": f.name, "type": str(f.type)} for f in arrow_schema], + "row_count": row_count, + } + atomic_write_text(entry_schema_path(project, content_hash), json.dumps(schema_doc, indent=2)) + + # Cache-admission instrumentation (#87): record the structural verdict and + # the measured value-per-byte inputs side by side, so #30's structural→ + # measured flip is decidable from data. classify_build's verdict is what + # cache_worthy already computes (and throws away); compile_seconds times the + # author DAG's expr->backend-plan step (the dominant per-view cost per #30), + # separate from execute; cache_bytes is the baked snapshot size (None for a + # cheap entry that bakes nothing). loaded is the build_path expression whose + # execute just baked the snapshot, so its CachedNode names that exact file. + from tallyman_xorq.result_cache import _cached_node_path, classify_build + + verdict = classify_build(xorq_build_dir) + cache_worthy_v, cache_worthy_why = verdict["worthy"], verdict["why"] + + compile_seconds: float | None = None try: - cache_bytes = snap.stat().st_size - except OSError: + from xorq.expr.api import to_sql + + t_compile = time.monotonic() + to_sql(author_expr) + compile_seconds = round(time.monotonic() - t_compile, 3) + except Exception: + # Best-effort: a DAG xorq can't render to SQL must not break the build. pass - perf_log.info( - "cache admission %s: structural worthy=%s (%s) | compile=%ss execute=%ss bytes=%s", - content_hash, - cache_worthy_v, - cache_worthy_why, - compile_seconds, - execute_seconds, - cache_bytes, - ) + cache_bytes: int | None = None + snap = _cached_node_path(loaded) + if snap is not None and snap.exists(): + try: + cache_bytes = snap.stat().st_size + except OSError: + pass + + perf_log.info( + "cache admission %s: structural worthy=%s (%s) | compile=%ss execute=%ss bytes=%s", + content_hash, + cache_worthy_v, + cache_worthy_why, + compile_seconds, + execute_seconds, + cache_bytes, + ) - manifest = Manifest( - content_hash=content_hash, - project=project, - prompt=prompt, - row_count=row_count, - execute_seconds=execute_seconds, - compile_seconds=compile_seconds, - cache_worthy=cache_worthy_v, - cache_worthy_why=cache_worthy_why, - cache_bytes=cache_bytes, - sources=sources or None, - parents=parents or None, - ) - write_manifest(target, manifest) + manifest = Manifest( + content_hash=content_hash, + project=project, + prompt=prompt, + row_count=row_count, + execute_seconds=execute_seconds, + compile_seconds=compile_seconds, + cache_worthy=cache_worthy_v, + cache_worthy_why=cache_worthy_why, + cache_bytes=cache_bytes, + sources=sources or None, + parents=parents or None, + ) + write_manifest(target, manifest) + except Exception: + # No partial entry dir survives a failed build: every population step + # is covered, not just the load_expr/execute paths that had ad-hoc + # cleanup before (D1/L4). The target.exists() early-return guarantees + # we created this dir, so removing it can't clobber a prior entry. + if created_target: + shutil.rmtree(target, ignore_errors=True) + raise _append_prompt(project, content_hash, prompt) # Best-effort: drop the temp script. diff --git a/tests/test_cache_inspector.py b/tests/test_cache_inspector.py index 143fd39..4d33bb0 100644 --- a/tests/test_cache_inspector.py +++ b/tests/test_cache_inspector.py @@ -131,10 +131,12 @@ def test_cache_delete_removes_only_snapshot(fresh_companion_app, project, orders def test_cache_delete_then_read_self_heals_warm_cache(fresh_companion_app, project, orders_parquet, monkeypatch): # Deleting a snapshot must self-heal on the next read even when the read path's - # cached_result_expr was already warm for that entry — the companion memoises a - # `deferred_read_parquet` of the snapshot, so without clearing that lru the next - # read dereferences the just-deleted file and 500s ("At least one path is - # required") instead of re-baking. The delete endpoint promises a self-heal. + # cached_result_expr was already warm for that entry. Since ee0a90a the read + # wrapper re-checks path.exists() on every call and self-heals regardless of the + # warm memo (see test_cached_result_expr_self_heals_after_warm_then_evict), so + # the delete endpoint's cache_clear() is now defense-in-depth, not what averts + # the "At least one path is required" 500. This pins the end-to-end promise + # through the endpoint. monkeypatch.setenv("TALLYMAN_PROJECT", project) from tallyman_xorq.result_cache import cached_result_expr diff --git a/tests/test_native_catalog_store.py b/tests/test_native_catalog_store.py index 5bf725b..8b86408 100644 --- a/tests/test_native_catalog_store.py +++ b/tests/test_native_catalog_store.py @@ -230,7 +230,9 @@ def _boom_load(*a, **k): monkeypatch.setattr(compiler, "load_expr", _boom_load) with pytest.raises(Exception): build_and_persist(project, _agg_code(orders_parquet)) - monkeypatch.undo() + # No monkeypatch.undo(): it shares the instance that isolated_home used to set + # TALLYMAN_HOME, so undo() would point entries_dir at the real home and make + # this check vacuous. monkeypatch auto-reverts at teardown. ed = paths.entries_dir(project) leftover = [c.name for c in ed.iterdir() if c.is_dir()] if ed.exists() else [] @@ -618,3 +620,60 @@ def _boom(*a, **k): ed = paths.entries_dir(project) leftover = [c.name for c in ed.iterdir() if c.is_dir()] if ed.exists() else [] assert not leftover, f"failed build left orphan entry dir(s): {leftover}" + + +def test_consistency_guard_surfaces_orphan_recipe(project, orders_parquet): + """The orphan branch of guard 2 (a tracked ``entries/.zip`` with no + pointer) must raise — the untested half of the pointer/recipe agreement check + (L8). The missing-pointer half is covered by + ``test_reset_to_revision.py::test_reset_surfaces_pointer_without_durable_recipe``.""" + cs.genesis(project) + build_and_persist(project, _agg_code(orders_parquet)) + cs.checkpoint_catalog(project, "create") # tracks entries/.zip + with pytest.raises(RuntimeError, match="tracked recipes with no pointer"): + catalog.assert_catalog_consistent(project, set()) # pointers empty, recipe tracked + + +# --------------------------------------------------------------------------- +# Decomposition — alias state rolls back with git reset (M2) +# --------------------------------------------------------------------------- + + +def test_alias_state_survives_reset(project, orders_parquet): + """The alias map + history (the tracked ``aliases.jsonl`` this cut introduced) + rolls back with ``git reset`` — the carrier whose only coverage was the deleted + ``test_reset_rolls_back_alias_state`` (M2). Mirrors the display-config / + prompt-history survives-reset tests.""" + from tallyman_core import aliases as al + + cs.genesis(project) + h0 = _build_checkpoint(project, _agg_code_n(orders_parquet, _AGGS[0]), "c0") # step 1 + al.set_alias(project, "regions", h0, expect_exists=False) + cs.checkpoint_catalog(project, "c1") # step 2: regions -> h0 (history [h0]) + h1 = build_and_persist(project, _agg_code_n(orders_parquet, _AGGS[1])).content_hash + al.set_alias(project, "regions", h1) # revise: history [h0, h1] + al.set_alias(project, "avg", h1, expect_exists=False) + cs.checkpoint_catalog(project, "c2") # step 3: regions -> h1 (history [h0, h1]) + avg -> h1 + + assert al.load_aliases(project) == {"regions": h1, "avg": h1} + cs.reset_to(project, 2) + assert al.load_aliases(project) == {"regions": h0} + assert al.history_for(project, "regions") == [h0] + cs.reset_to(project, 3) + assert al.load_aliases(project) == {"regions": h1, "avg": h1} + assert al.history_for(project, "regions") == [h0, h1] + + +def test_append_prompt_is_atomic_and_complete(project, orders_parquet): + """``_append_prompt`` rewrites ``prompts/.jsonl`` via tmp + replace (C2): + it runs outside the project lock (only the checkpoint holds it), so the file + must stay whole JSONL across appends with no leftover ``*.tmp`` the + checkpoint's ``git add -A`` could stage.""" + from tallyman_xorq import read_prompts + + cs.genesis(project) + res = build_and_persist(project, _agg_code(orders_parquet), prompt="first") + h = res.content_hash + build_and_persist(project, _agg_code(orders_parquet), prompt="second") # re-run, same hash, appends + assert [r["prompt"] for r in read_prompts(project, h)] == ["first", "second"] + assert not list(paths.prompts_path(project, h).parent.glob("*.tmp")) # no interrupted temp left behind From 4f7f78b52429e86d5a131734cd0d3cc9677707f8 Mon Sep 17 00:00:00 2001 From: Paddy Mullen Date: Wed, 17 Jun 2026 12:37:18 -0400 Subject: [PATCH 26/27] test: cheap build must validate the row-level projection (failing-first) MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit A cheap entry (pure projection — no Aggregate/Join/Sort/window/UDF) only ran schema() + count() at build. DataFusion satisfies count() from source metadata and prunes the row projection, so a failing row-level op (cast / arithmetic / UDF) never evaluated at build: the broken entry committed + aliased as a successful build and threw only on the first materialising read — silent broken state at the head of an alias chain (pack01 finding, PR #103 comment). test_cheap_build_validates_row_projection builds a cheap mutate with an out-of-range date cast and asserts the build raises. Red today: the build does not raise (DID NOT RAISE BuildError). Also pins the valid-cast positive case so the streaming row-count path is exercised, not just the error path. Co-Authored-By: Claude Opus 4.8 (1M context) --- tests/test_build.py | 39 +++++++++++++++++++++++++++++++++++++++ 1 file changed, 39 insertions(+) diff --git a/tests/test_build.py b/tests/test_build.py index e90891f..2c99034 100644 --- a/tests/test_build.py +++ b/tests/test_build.py @@ -110,6 +110,45 @@ def test_build_cheap_entry_records_no_snapshot_bytes(project: str, orders_parque assert res.cache_bytes is None +def test_cheap_build_validates_row_projection(project: str): + # A cheap entry (pure projection — no Aggregate/Join/Sort/window/UDF) must + # still evaluate its row-level projection at build time. count() is satisfied + # from source metadata and prunes the projection, so a failing row-level op + # (here a cast of an out-of-range date string) never runs unless the build + # forces a full evaluation. Without it the broken entry commits + aliases as a + # successful build and throws only on the first materialising read — silent + # broken state at the head of an alias chain (pack01 finding). A worthy entry + # escapes this because its result-cache node materialises on execute. + import pyarrow as pa + import pyarrow.parquet as pq + + from tallyman_core.paths import data_dir + + p = data_dir(project) / "bad_dates.parquet" + p.parent.mkdir(parents=True, exist_ok=True) + # month 20 is out of range — datafusion's date32 cast rejects it at row-eval + # time, but count() prunes the cast so it never runs at build unless forced. + pq.write_table(pa.table({"d": ["2020-01-01", "2020-20-07", "2021-12-31"]}), p) + + def _cast_code(parquet: Path) -> str: + return f""" +import xorq.api as xo +t = xo.deferred_read_parquet({str(parquet)!r}) +expr = t.mutate(dd=t.d.cast("date")) +""" + + with pytest.raises(BuildError): + build_and_persist(project, _cast_code(p)) + + # The streaming validation is also the cheap path's row-count source, so a + # valid cheap projection must still build and report the right count. + good = data_dir(project) / "good_dates.parquet" + pq.write_table(pa.table({"d": ["2020-01-01", "2021-12-31"]}), good) + res = build_and_persist(project, _cast_code(good)) + assert res.cache_worthy is False # pure projection — the cheap path under test + assert res.row_count == 2 + + def test_build_idempotent_same_code(project: str, orders_parquet: Path): code = _agg_code(orders_parquet) a = build_and_persist(project, code, prompt="first") From 8e686f8ab171c52e2576aeb539923bc444eedf19 Mon Sep 17 00:00:00 2001 From: Paddy Mullen Date: Wed, 17 Jun 2026 12:40:50 -0400 Subject: [PATCH 27/27] fix: cheap build validates row-level ops; doc-nit cleanup MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit pack01 finding (PR #103 comment): a cheap entry (no Aggregate/Join/Sort/window/ UDF) only ran schema() + count() at build, and DataFusion satisfies count() from source metadata — pruning the row projection — so a failing cast/arithmetic/UDF never evaluated at author time. The broken entry committed + aliased as a successful build and threw only on the first materialising read. build.py now classifies before executing and, for the cheap path, streams the full result via to_pyarrow_batches() and discards it — forcing row-level evaluation at build (constant memory; the one pass also yields the exact row count). A worthy entry is unchanged: baking its result-cache snapshot already evaluates every row, so count() there both validates and counts. Pinned by test_cheap_build_validates_row_projection (4f7f78b), with a valid-cast positive case covering the streaming row-count path. Also fixes three review doc-nits: - backend.py cited a nonexistent test (test_connect_shim_snapshot_is_byte_ identical) and the wrong "byte-identical" claim; now names the shipped test_connect_shim_matches_xorq_api_connect (key + data equivalence). - build.py's failed-build cleanup comment overclaimed "we created this dir"; reworded — created_target is also set for a manifest-less mid-build leftover, which is still safe to rmtree (never a complete, durable entry). - assert_catalog_consistent guard-3 docstring omitted the hash-keyed prompts/.jsonl; now states why it (append-only, inert when orphaned) and the notebook are out of scope. Co-Authored-By: Claude Opus 4.8 (1M context) --- src/tallyman_core/catalog.py | 11 +++++--- src/tallyman_xorq/backend.py | 7 +++--- src/tallyman_xorq/build.py | 49 +++++++++++++++++++++++------------- 3 files changed, 42 insertions(+), 25 deletions(-) diff --git a/src/tallyman_core/catalog.py b/src/tallyman_core/catalog.py index 565e2dd..2c197e7 100644 --- a/src/tallyman_core/catalog.py +++ b/src/tallyman_core/catalog.py @@ -204,14 +204,17 @@ def assert_catalog_consistent(project: str, pointers: set[str]) -> None: (``entries/.zip``) must equal the tallyman *pointers* (``entry_hashes``), so a pointer with no durable recipe — or a recipe with no pointer — fails loudly instead of returning a masked divergence. - 3. **Sidecar hash-reference integrity**: every decomposed file keyed by - content hash (``aliases.jsonl`` latest+history, ``chart_specs/``, + 3. **Sidecar hash-reference integrity**: every durable, content-addressed + sidecar that can dangle into a wrong-data or build-failure path + (``aliases.jsonl`` latest+history, ``chart_specs/``, ``display_configs/``) must name a hash in the durable recipe set. Guard 2 only pairs recipes with pointers; without this a committed step whose alias/chart/display points at a hash with no recipe would reset clean and report OK — and a dangling alias head later blows up the - self-chaining ``from_catalog`` build. (Notebook cells key on alias *name*, - not hash, so they are deliberately out of scope here.) + self-chaining ``from_catalog`` build. Two other hash/name-keyed surfaces + are deliberately out of scope: notebook cells key on alias *name* not hash, + and ``prompts/.jsonl`` is append-only provenance that is inert when + orphaned (read only for an entry the UI is already showing). Takes *pointers* as an argument (rather than reading ``catalog_state``) to keep the call direction one-way and acyclic. diff --git a/src/tallyman_xorq/backend.py b/src/tallyman_xorq/backend.py index b995af8..79706f9 100644 --- a/src/tallyman_xorq/backend.py +++ b/src/tallyman_xorq/backend.py @@ -9,9 +9,10 @@ The naive narrowing ``xorq.expr.api.connect`` is *not* a drop-in: that is ibis's ``connect(resource, **kwargs)`` (a different function requiring a positional arg), not this no-arg datafusion factory — so a swap to it would resolve the -symbol and then fail at call time or build the wrong backend. The -determinism-equivalence is pinned by -``test_native_catalog_store.py::test_connect_shim_snapshot_is_byte_identical``. +symbol and then fail at call time or build the wrong backend. The equivalence is +pinned by ``test_native_catalog_store.py::test_connect_shim_matches_xorq_api_connect`` +(key + data equivalence, not raw bytes — datafusion's group-by output order isn't +deterministic). """ from __future__ import annotations diff --git a/src/tallyman_xorq/build.py b/src/tallyman_xorq/build.py index ea87c2f..a9a086c 100644 --- a/src/tallyman_xorq/build.py +++ b/src/tallyman_xorq/build.py @@ -428,7 +428,7 @@ def build_and_persist( ) target.mkdir(parents=True, exist_ok=True) - created_target = True # reached only past the target.exists() early-return + created_target = True # past the complete-entry early-return; safe to rmtree on failure # Move xorq's build directory contents (expr.yaml + deps) under the entry. xorq_build_dir = entry_build_dir(project, content_hash) @@ -461,16 +461,32 @@ def build_and_persist( hint = _ibis_import_hint(str(exc), code) raise BuildError(f"load_expr failed: {exc}{hint}\n{traceback.format_exc()}") from exc + # Classify before executing (reads the serialized build, no eval): the + # verdict selects the execution strategy below and is reused for the #87 + # admission instrumentation after the temp dir is gone. + from tallyman_xorq.result_cache import _cached_node_path, classify_build # noqa: PLC0415 + + verdict = classify_build(xorq_build_dir) + cache_worthy_v, cache_worthy_why = verdict["worthy"], verdict["why"] + # Execute (#73). A worthy entry's build carries a baked result-cache # node (rewrite_for_build), so executing materialises it once into the # compute cache — and every later loader (the Buckaroo viewer, diffs, - # from_catalog) reads that cached result instead of re-running the DAG. - # A cheap entry materialises nothing: this is a pushdown count() over a - # columnar source plus a schema read, no parquet written. + # from_catalog) reads that cached result instead of re-running the DAG; + # baking evaluates every row, so count() there both validates and counts. + # A cheap entry materialises nothing, so count() alone is satisfied from + # source metadata and PRUNES the row projection — a failing cast / + # arithmetic / UDF would never evaluate at build, committing + aliasing a + # broken entry that throws only on the first materialising read. Stream + # the full result and discard it to force row-level evaluation at author + # time (constant memory; the one pass also yields the exact row count). t0 = time.monotonic() try: arrow_schema = loaded.schema().to_pyarrow() - row_count = int(loaded.count().execute()) + if cache_worthy_v: + row_count = int(loaded.count().execute()) + else: + row_count = sum(batch.num_rows for batch in loaded.to_pyarrow_batches()) except Exception as exc: hint = _ibis_import_hint(str(exc), code) raise BuildError(f"build execution failed: {exc}{hint}\n{traceback.format_exc()}") from exc @@ -484,19 +500,14 @@ def build_and_persist( } atomic_write_text(entry_schema_path(project, content_hash), json.dumps(schema_doc, indent=2)) - # Cache-admission instrumentation (#87): record the structural verdict and - # the measured value-per-byte inputs side by side, so #30's structural→ - # measured flip is decidable from data. classify_build's verdict is what - # cache_worthy already computes (and throws away); compile_seconds times the + # Cache-admission instrumentation (#87): record the structural verdict + # (computed above, before execute, since it now also selects the execution + # strategy) alongside the measured value-per-byte inputs, so #30's + # structural→measured flip is decidable from data. compile_seconds times the # author DAG's expr->backend-plan step (the dominant per-view cost per #30), # separate from execute; cache_bytes is the baked snapshot size (None for a # cheap entry that bakes nothing). loaded is the build_path expression whose # execute just baked the snapshot, so its CachedNode names that exact file. - from tallyman_xorq.result_cache import _cached_node_path, classify_build - - verdict = classify_build(xorq_build_dir) - cache_worthy_v, cache_worthy_why = verdict["worthy"], verdict["why"] - compile_seconds: float | None = None try: from xorq.expr.api import to_sql @@ -541,10 +552,12 @@ def build_and_persist( ) write_manifest(target, manifest) except Exception: - # No partial entry dir survives a failed build: every population step - # is covered, not just the load_expr/execute paths that had ad-hoc - # cleanup before (D1/L4). The target.exists() early-return guarantees - # we created this dir, so removing it can't clobber a prior entry. + # No partial entry dir survives a failed build: every population step is + # covered, not just the load_expr/execute paths that had ad-hoc cleanup + # before (D1/L4). created_target is set once we pass the complete-entry + # (manifest-bearing) early-return, so the dir we remove is either one this + # call made or a manifest-less mid-build leftover — never a complete, + # durable entry, which always early-returns before created_target is set. if created_target: shutil.rmtree(target, ignore_errors=True) raise