feat: native tallyman_core.catalog store (drop xorq catalog package)

[paddymul]

What

Replaces xorq's catalog repo manager with a tallyman-native, content-addressed recipe store, and decomposes the monolithic catalog.yaml into per-concern git-tracked files. Design + investigation record: plans/native-catalog-store.md.

Why

xorq's catalog/ package is the source of the recurring catalog pain: assert_consistency runs on every add (even --no-sync) and rejects any extra tracked file (#48), and the add subprocess commits outside tallyman's flock, racing index.lock. Dropping the package removes the constraint by construction — tallyman already owns the catalog model; the subprocess was the last tendril.

What shipped (two cuts)

• Cut A — native store + decomposition. Deletes xorq_catalog.py; adds tallyman_core/catalog.py (a TRACKED_SURFACE allowlist, .gitignore, deterministic write_recipe_zip, assert_catalog_consistent). The checkpoint owns zip creation under the project lock (build owns only the complete entry dir), so there are no out-of-lock git writes or orphan zips — the root cause of tallyman commits aliases.json/alias_history.json into the xorq catalog repo, so xorq's assert_consistency fails and catalog add silently no-ops for every entry after the first — entry recipes never reach git #48. With assert_consistency gone, catalog.yaml decomposes: charts, display configs, post-processing, stats, aliases, and the notebook each become their own git-tracked file (aliases.jsonl, notebook.jsonl, chart_specs/*.vl.json, …); catalog.yaml reduces to pointer lists.
• Cut B — catalog-free runtime. Replaces import xorq.api as xo (which transitively loads all 12 xorq.catalog modules) with narrow catalog-free imports + a datafusion connect() shim.

Key decisions

• Recipe zip archives the allowlist {expr.py, xorq_build/, schema.json, manifest.json}, deterministic; the per-expression wheel is dropped (~178KB → ~6KB).
• Checkpoint owns zip creation under the project lock.
• All mutable state → tracked files that roll back with git reset.
• User recipes stay on import xorq.api as xo (catalog-free guarantee scoped to src/); the authoring-surface switch is a separate codegen-validated follow-up.

Review hardening (PR #103)

A multi-agent review pass landed on top of the implementation: a third assert_catalog_consistent guard (decomposed sidecars must name a hash in the durable recipe set), a shared atomic_write_text across every tracked-surface writer, entry-dir cleanup on every failed-build path, a loud notebook.load on malformed JSONL, and added reset/guard coverage. See the review thread for the per-finding trail.

plans/native-catalog-store.md carries the full investigation findings, risk register, module boundary, and the ordered TDD implementation sequence.

🤖 Generated with Claude Code

[paddymul]

Code review (Claude)

Docs-only PR: a design plan (plans/native-catalog-store.md) plus a 3-line .gitignore addition. I verified the load-bearing technical claims against the actual tree before reviewing the prose.

Verification — the claims hold exactly:

• import xorq loads 0 catalog modules; import xorq.api loads 12 ✓
• xorq.expr.api.connect is xorq.api.connect → False, with incompatible signatures (connect(session_config=None) vs. connect(resource, **kwargs)), so Risk T-33: Memory-scaling integration test for React embed #3's "resolves the symbol, fails at call time" is real ✓
• pyproject.toml:77 deselects integration/cache_lab/perf (and perf_tier_a) ✓
• Every cited file:line (aliases.py:24/129/155, build.py:335/367/477-478, catalog_state.py:383-388/433, source_cache.py:133/152, server.py:219) is accurate ✓
• All four zip-allowlist members (expr.py, xorq_build/, schema.json, manifest.json) are co-located at the entry-dir top level, and schema.json persists across the same-hash early-return ✓
• All five existing tests slated for delete/rewrite exist at the cited locations ✓

Unusually rigorous plan. Findings are about implementation sequencing and a few concrete gaps, not the direction.

1. The 1a/1b split sits in tension with the TDD rule; two "failing-first" tests are really fix-induced breaks (High)

The plan splits failing-first into two commits. Stub symbols bundled with their tests (1b) are legitimate scaffolding. But test_prompt_history_survives_reset and test_two_entries_union_after_reset go red only because 1b pre-stages the .gitignore that ignores entries/*/ — on today's tree both pass (bullpen / tracked entry dir preserve the loose prompts.jsonl and expr.py across reset). Their lifecycle is green → red (when .gitignore lands) → green (when durable restore lands): not "test the existing bug then fix it," but "introduce the new storage model, which creates the regression, then add durability." The .gitignore is a behavior change, not an inert stub, and landing it in a failing-first commit risks turning existing green reset/bullpen tests red there too. Recommend deciding: (a) collapse 1a+1b into one failing-first commit but keep .gitignore out; or (b) treat the two .gitignore-dependent durability tests as fix-induced breaks and move them (with .gitignore) into commit 3.

2. The plan's own "real assertion, never collection/import error" bar isn't met by some 1b reds (Med)

test_two_entries_union_after_reset with a no-op write_recipe_zip + gitignored entries/*/ goes red because expr.py is gone (missing-file/import error mid-act), not the "Multiple backends found" RuntimeError that motivates it, and not a clean assert. Cheap fix: assert presence first (assert (entry_dir/"expr.py").exists() post-reset) so the red is an assertion.

3. assert_catalog_consistent's allowlist must include .gitignore and the full tracked set (Med)

The check is deny-by-default over git ls-files. The .gitignore introduced in commit 3 is tracked at the repo root but isn't in the plan's tracked-pattern enumeration — a strict allowlist would reject it and fail every checkpoint. Cover .gitignore, pyproject.toml, uv.lock, and each glob. Add a positive assertion (a normal checkpoint of every tracked kind passes), since test_unlisted_path_not_tracked only checks the negative.

4. Zip byte-stability depends on xorq_build/ determinism, not just pinned ZipInfo (Med)

test_recipe_zip_byte_stable_across_rebuilds leans on content_hash equality, which guarantees the hash is stable, not that every byte in xorq_build/ is identical run-to-run. An embedded non-project-root absolute path or unordered YAML dump churns the zip while the hash matches. Have the test diff the expanded xorq_build/ tree across two rebuilds so a non-deterministic build file is attributed to the builder, not the zip writer.

5. Minor

• test_recipe_zip_members_are_allowlist must prefix-match xorq_build/** while exact-matching the three top-level files — not a literal 4-element set compare.
• The plan says CI deselects "integration/cache_lab/perf"; addopts also deselects perf_tier_a (immaterial; enumeration is one short).
• The .gitignore addition (.claude/, .claude-worktrees/) is unrelated to the plan doc — tiny and already committed, just noting the scope mix.

Bottom line

Technically sound, internally consistent, verified-accurate — approvable as a plan. The one thing worth settling before writing code is finding #1 (1a/1b structure + where .gitignore lives), since it collides with the project's TDD rule. #3 and #4 are concrete traps to bake into the commit-1 tests now rather than discover at step 7.

🤖 Generated with Claude Code

[paddymul]

Tier-B perf integration run + catalog integrity audit

Ran the perf harness against the PR and audited the catalog's referential integrity (prompted by "what if a chart references an expression/alias that isn't in the catalog").

Perf: pytest -m perf tests/test_perf_integration.py

1 passed in 8.59s. Run against parking_ticket_analysis_1m (the post-#73 truncated corpus — the default parking_ticket_analysis still embeds per-entry result.parquet, so it's pre-#73 and won't self-heal). The harness's catalog.yaml-era alias discovery worked — it resolved all 9 aliases and built the diff pairs. The measurements split into two groups.

6 source-rooted entries — clean numbers:

entry	rows	exec cold	exec warm	cold stats	warm stats	1st page
09d21553	1M	0.74s	0.60s	0.94s	0.10s	0.01s
4e90d0b9	1M	0.60s	0.59s	0.48s	0.09s	0.01s
7a3f2dac	1M	0.16s	0.16s	0.25s	0.04s	0.01s
a89f5e91	1M	0.60s	0.60s	0.49s	0.09s	0.01s
bd654ec5	1M	0.16s	0.15s	0.39s	0.05s	0.01s
cf52068f	1M	0.14s	0.13s	0.22s	0.04s	0.01s

Signal: warm stats are 5–10× faster than cold (e.g. 0.94→0.10s), with heal_s≈0.0 and load_s≈0.02s. The on-disk stat cache survives a fresh dataflow — the #34 empty-cache failure (warm≈cold) is not occurring on this branch.

9 from_catalog entries + all 4 diff pairs — errored, single root cause:

from_catalog('all_violations' | 'tickets_per_day' | 'habitual_speeder_stats' | 'camera_select_columns')
→ ProjectDataNotFound: catalog entry not found in project 'parking_ticket_analysis_1m'

Not a regression. from_catalog resolves an alias through get_alias → aliases.jsonl only (io.py:99, aliases.py:37); the 1m corpus is catalog.yaml-era with no aliases.jsonl (aliases live in catalog.yaml's alias_map). The harness discovers those aliases via the catalog.yaml fallback, but the native runtime can't resolve them at execute time. No native-format corpus exists on disk yet, so a fully-green from_catalog/diff run needs scripts/rebuild_native_catalog.py first (the "8 real projects still need rebuilding" item). The 6 source-rooted rows above are the real measurement this run produced.

Integrity audit: referential checks are partial and asymmetric

Recipe→parent refs fail loudly; chart/alias/display refs to a missing entry are unchecked and dangle silently. The perf errors above are a live demo of the loud half — from_catalog to a missing alias raises rather than corrupting state.

• assert_catalog_consistent (catalog.py:182) is not referential integrity. It's a deny-by-default tracked-surface allowlist (rejects stray files git add -A would commit) plus pointer↔recipe agreement. It says nothing about whether a chart/alias target hash exists.
• from_catalog(<hash|alias>) is checked at build time and every reconstruction (io.py:97-99) — a missing target raises ProjectDataNotFound. Loud, every time.

Chart → missing entry/alias (the original question):

• At creation via MCP it's guarded — catalog_chart checks entry_dir(...).exists() then get_alias(...), returning {"error": "no entry for ..."} otherwise (server.py:689-692).
• But core set_chart (charts.py:32) has no check — writes a .vl.json for any hash.
• And no cleanup: remove_chart exists but has zero call sites in any entry-removal/prune path. Attach a chart to entry A, evict/prune A, and A's chart spec dangles silently in chart_specs/. get_chart still returns it (keyed on the chart file, not the entry); the browser falls through to "no chart."

Same silent-dangle gap on the other hash-keyed surfaces, none validating the target: aliases (latest/history, set_alias), display_configs (display_configs/<hash>.json), notebook cells (cell→alias). Post-processing and summary stats carry no entry refs (N/A). No test covers any dangling case.

Gap: there is no proactive integrity check — "scan the catalog, flag any chart/alias/display/notebook ref pointing at a missing entry." A catalog doctor pass (or extending assert_catalog_consistent with a referential check) would close it.

[paddymul]

Code review — native tallyman_core.catalog (drop xorq catalog package)

Reviewed ea06845..40f09aa (28 files, +2383/−1002) via a multi-agent pass: cluster finders across the changed surface → adversarial verification of every finding (each reproduced or refuted against the code at 40f09aa) → completeness critic. Severities below are the verifier-adjusted values (several downgraded after reproduction).

Scope: findings for the manual rebuild CLI (scripts/rebuild_native_catalog.py) are tracked separately and excluded here. This covers the native store, checkpoint/reset, the build.py rework, the Cut-B import change, the catalog.yaml decomposition, companion/cache touch points, tests, and PR hygiene. No back-compat/migration concerns are raised (single-user repo).

Result: 14 confirmed (2 medium, 12 low) + 2 gaps; 1 dismissed false positive. No high-severity findings in scope.

---

TL;DR — worth addressing before merge

1. [MEDIUM] The consistency guard is asymmetric (M1). assert_catalog_consistent validates only entries.jsonl ↔ entries/*.zip. Aliases (latest+history), charts, display-configs, and notebook hashes are never checked against the live recipe set, so a step whose alias/chart/display/notebook points at a non-existent hash passes silently and reset_to reports clean.
2. [MEDIUM] Test-coverage regression on a carrier this PR changed (M2). Alias-state rollback-on-reset lost its only test in the deleted test_catalog_xorq_integration.py with no native replacement, even though this PR moves aliases into the standalone tracked aliases.jsonl.

Everything else is low: a consistent non-atomic-write pattern across several tracked-surface writers, a couple of stale comments/doc text, and the docs:-titled-but-actually-a-feat PR metadata.

---

MEDIUM

M1 — Consistency guard ignores aliases/charts/display/notebook · src/tallyman_core/catalog.py:182-212 · integrity · high → medium
assert_catalog_consistent enforces only (1) the TRACKED_SURFACE path allowlist and (2) entries/<hash>.zip set == entries.jsonl pointers. No cross-check that aliases.jsonl, chart_specs/<hash>.vl.json, display_configs/<hash>.json, or notebook hashes resolve to a durable recipe. reset_to (catalog_state.py:345) calls only this guard. Reproduced: committed set_alias("ghost","ffff…") + an orphan chart/display for that hash; guard passed, reset_to succeeded. Realistic triggers: a roll-back/prune leaving a sidecar, or a setter called with a no-entry hash (set_alias/set_chart/set_display_config do no existence check). Impact bounded — orphan chart/display are inert; a dangling alias-history hash surfaces as a loud build error on the self-chaining from_catalog path, not silent wrong rows.
Fix: extend the guard (or a sibling reset_to calls) to verify every aliases.jsonl latest+history hash, chart/display hash, and notebook hash resolves to a tracked recipe; raise listing orphans.

M2 — Lost coverage: alias-state rollback on reset has no native replacement · tests/test_reset_to_revision.py:178-195 (gap) · test-coverage · medium
The deleted test_catalog_xorq_integration.py had test_reset_rolls_back_alias_state (builds 2 entries, reset_to(1), asserts load_aliases == {"agg0"}). This PR moves aliases into the tracked aliases.jsonl (ec0320d) — exactly the carrier such a test guards — but no native test resets and asserts alias state. test_reset_roundtrip_over_every_kind covers pp/stats/charts/notebook and omits aliases; test_aliases.py has zero reset/checkpoint coverage. The PR's own test_display_config_survives_reset establishes the standard that each decomposed section gets a survives-reset test; aliases now occupies that gap.
Fix: add test_alias_state_survives_reset (two aliases across two steps, reset to step 1, assert only step-1; reset forward, assert both).

---

LOW

L1 — assert_catalog_consistent never detects orphaned chart_specs/display_configs · catalog.py:182-212 · integrity · medium → low
Guard 2 checks only the recipe-zip set; chart_specs/display_configs/prompts are validated for path shape only, never live-entry membership. remove_chart is only re-exported (never called by prune/reset); remove_display_config is never called anywhere — nothing GCs these. Narrower surface of M1. MCP catalog_chart does gate writes to live entries, so the realistic vector is roll-back/prune. Fix: the M1 guard, or document + test the tolerated-orphan stance.

L2 — Chart / display-config writers are non-atomic · catalog.py:81 (comment) + charts.py:47, display_configs.py:29 · integrity · low
The GITIGNORE_LINES comment claims *.tmp covers atomic writers, but set_chart/set_display_config (and notebook._save) do bare write_text. A crash mid-write leaves a truncated file the next git add -A commits; get_chart/get_display_config do unguarded json.loads. Fix: tmp + Path.replace, matching aliases._write.

L3 — Build's completeness sentinel manifest.json is written non-atomically · manifest.py:58-61 · concurrency · medium → low
The "manifest = complete entry dir" design gates on .is_file(), but write_manifest is a plain write_text — .is_file()-true the instant it's truncated. build_and_persist holds no lock; the companion (separate process) checkpoints on a threadpool while the MCP server is mid-build, so a checkpoint can zip a truncated manifest member. Currently latent — nothing reads the zip's manifest member back, and the capture path never json.loads it; violates the byte-identical-rebuild promise but doesn't break reads today. Fix: atomic write_manifest (tmp + replace); consider the same for schema.json at build.py:479.

L4 — Orphan entry-dir cleanup covers only the load_expr/execute exception paths · build.py:423-536 · correctness · low
shutil.rmtree(target) runs on exactly two failure paths (453, 468). The xorq_build copy loop, make_portable_inplace, and the expr.py/schema.json/manifest writes have no cleanup — an exception there leaves a partial entries/<hash>/ dir. Durability is preserved (the manifest filter skips incomplete dirs), so it's gitignored junk, but the D1 "no partial dirs" invariant is only partially realized and the no-orphan test exercises only the path that already cleans up. Fix: wrap the whole entry-dir population in a try/except that rmtrees when this call created the dir (or drop the ad-hoc rmtrees and document the manifest filter as the sole guard); add a test for a non-load_expr failure site.

L5 — notebook._save rewrites notebook.jsonl non-atomically · notebook.py:50-58 · integrity · medium → low
The one JSONL list-rewriter in the decomposition that isn't atomic; a torn write can be committed by git add -A. Bounded — cells are one-per-line and load() skips bad lines, so worst case is dropping the trailing cell(s), and it's git-recoverable. Fix: mirror aliases._write (tmp + replace), keeping the empty-notebook unlink branch.

L6 — notebook.load silently drops malformed JSONL lines · notebook.py:40-47 · integrity · medium → low
On JSONDecodeError, load() does continue; the next _save() makes the loss permanent. Inconsistent with aliases._read/read_prompts, which raise loudly. (The PR actually reduced blast radius vs the old loader, which dropped the whole notebook on one bad line.) Fix: let json.loads raise, or log + preserve the raw line so _save round-trips it.

L7 — Orphan chart_specs/display_configs/prompts files caught by neither guard, no test · catalog.py:182-212 · test-coverage · low
Test-coverage twin of L1/M1. test_nested_stray_under_allowlisted_prefix_is_rejected covers strays nested under the prefixes, not top-level orphan-hash files. By-design lenience (sidecars roll back via git reset, read by hash so inert) — the gap is that it's incidental, not pinned. Fix: decide intent — reject (guard + failing test) or allow (test documenting an orphan-hash sidecar passes).

L8 — assert_catalog_consistent's orphan-recipe branch (tracked - pointers) is untested · catalog.py:205-212 · test-coverage · medium → low
Guard 2 raises in both directions, but only missing = pointers - tracked is tested. No test stages a tracked zip absent from pointers, so the orphan branch never executes — a refactor dropping that half passes green. Defense-in-depth backstop (manifest filter prevents the divergence arising). Fix: commit an orphan zip and assert the guard raises matching "tracked recipes with no pointer".

L9 — api_delete_result_cache comment + cache_clear() left stale by the ee0a90a self-heal fix · app.py:1102-1109 · convention · low
ee0a90a made cached_result_expr re-check path.exists() and self-heal on every read. The endpoint's comment still claims cache_clear() is what prevents a dangling deref — now false (proven by test_cached_result_expr_self_heals_after_warm_then_evict, which self-heals with no cache_clear). Same stale rationale in tests/test_cache_inspector.py:133-137. Fix: update both comments.

L10 — PR title (docs:) and body ("design plan (no code)") contradict the contents · PR metadata · pr-hygiene · medium → low
The PR ships a multi-thousand-line implementation (deletes xorq_catalog.py, adds the native store, Cut B, the decomposition, new tests) and the plan doc says Status: implemented, yet the title is docs: and the body opens "A design plan (no code)…". Fix: retitle to feat:/refactor: and drop the "design plan (no code)" framing (the detailed sections can stay).

L11 — Plan doc "Open / not yet grilled" section is stale · plans/native-catalog-store.md:592-608 · convention · low
Still lists "the exact .gitignore contents" as open, but the catalog .gitignore is now concrete (GITIGNORE_LINES, catalog.py:72-82, written by write_gitignore). Fix: drop the .gitignore item; narrow what's genuinely open to the build-time-alias-revision limit (#84).

L12 — Risk register / Cut B cite a connect-shim test name that doesn't match · plans/native-catalog-store.md:437, 509 · convention · low
The doc names test_connect_shim_snapshot_is_byte_identical ("byte-identical"), but the shipped test is test_connect_shim_matches_xorq_api_connect and asserts key+data equivalence, not byte equality (datafusion group-by order isn't deterministic). The deviation is disclosed in "Implementation notes" but the risk-register rows weren't updated. Fix: update the name + wording.

---

Additional gaps (low)

C1 — post_processing/stats writers are non-atomic too · post_processing.py:200, summary_stats.py:183 · integrity. Same plain-write_text hazard as L2 on the tracked *.py surfaces. Fix: one shared atomic-write helper for all of these.

C2 — No test for a checkpoint racing the build's out-of-lock prompt append · build.py:409, 537 · concurrency. _append_prompt writes tracked prompts/<hash>.jsonl outside the project lock (only the checkpoint holds it); the re-run path appends while a dispatch-boundary checkpoint runs git add -A. Same class as L3, untested. Fix: atomic append, or a concurrency test; at minimum document the ordering requirement.

---

Dismissed (false positive)

test_api_projects_new_creates_and_switches asserts only dirs, not git/genesis init — genesis is covered by test_genesis_on_companion_project_new (tests/test_reset_to_revision.py:400-406), which POSTs the same route and asserts step-000 is tagged. The new behavior is tested.

---

Cross-cutting themes

1. The integrity guard is the recurring weak point. M1/L1/L7/L8 all trace to assert_catalog_consistent checking only recipe↔pointer. One third guard (every aliases/chart/display/notebook hash resolves to a tracked recipe) closes M1 and L1 at once; L7/L8 become the tests that pin it.
2. Non-atomic writes are a consistent pattern. L2/L3/L5 + C1 + C2 are the same gap on several writers (manifest, notebook, charts, display, pp, stats, prompts) — only aliases._write and write_recipe_zip got tmp+replace in the hardening pass. A shared atomic_write(path, body) helper across the tracked-surface writers resolves all of them and matches the convention this PR already established.

_{🤖 Generated with Claude Code — multi-agent review; every confirmed finding independently reproduced or refuted against the code at 40f09aa.}

[paddymul]

Addendum from the pack01 sonnet run on this branch — one in-scope build-path finding the review missed

Reran pack01 verbatim against this PR (40f09aa, sonnet driver, isolated home; notes in tallyman-prompt-packs/runs/2026-06-17-pack01-sonnet/). One build.py correctness issue surfaced that the review above doesn't cover — its build.py findings are L3 (manifest atomicity) and L4 (orphan-dir cleanup on exception paths); this is neither.

[MEDIUM — arguably high] A cheap build never exercises the row-level projection — a recipe that throws on every read is committed and aliased as a successful build · src/tallyman_xorq/build.py:463-471 · correctness/integrity

For a cheap entry (no Aggregate/Join/Sort/window/UDF) the build's only execution is:

arrow_schema = loaded.schema().to_pyarrow()   # logical schema, no data eval
row_count = int(loaded.count().execute())     # pushdown count, prunes the row projection

DataFusion satisfies count() from source/parquet metadata without evaluating the projection, and schema() is logical, so a row-level evaluation error in the recipe (a failing cast, arithmetic, a UDF) is never exercised at build. The surrounding try/except (build.py:467) only catches what count/schema themselves raise. A worthy entry bakes a full result and would catch it; a cheap mutate escapes.

Result: the entry is persisted, registered in entries.jsonl, and aliased, with a manifest row_count and a successful tool response (is_error:false). Every later read throws. The author gets no signal at author time.

Reproduced (pack01 T1 — a mixed-format date column parsed with a plain .cast("date")):

camera v2 (e69c6232a7f9): build OK, row_count 23,925,000, execute_seconds 0.013, cache_worthy=false ("cheap")
full scan: ValueError: Arrow error: Cast error: Cannot cast string '2020-20-07' to value of Date32 type   (0.8s)

The driver's T1 transcript shows 10 tallyman calls and 0 in-band errors — it reported "camera is now date32" and moved on. The error only surfaced at T2, when the first worthy build reading camera forced materialization (registry error_id 67ab20a581ef).

Impact — bounded but worse than a build failure: the broken entry is committed + aliased and invisible at author time, and it sits at the head of an alias that downstream from_catalog recipes chain on. It surfaces loudly on the first materializing read (viewer, a worthy downstream build, diff, marimo export), so it is silent-broken-state rather than silent-wrong-rows. The pre-native build did a full to_parquet and rejected the same cast at author time with nothing persisted — this PR's cheap path trades that validation for the metadata-only count.

Fix options (the tension: the cheap path exists precisely to avoid writing a parquet):

• Exercise the projection at build without persisting — a streaming execute() that discards rows, or a temp to_parquet that's dropped — restoring author-time validation at the cost of the cheap-path scan. Cleanest for correctness.
• Or classify a projection containing fallible row ops (cast / arithmetic / UDF) as worthy so it materializes; leave pure rename/select mutates on the cheap path.
• A bounded .limit(N).execute() probe is not sufficient — the offending row is rare and won't be in the first N.

---

Same run also produced F2 (stacked window functions fail to plan; the WindowShift error misleads authors into thinking lag is unsupported) → tallyman-side hint filed as #107, and F3 (the resulting row_number()+self-join over-correction silently mis-orders tied dates, 0.92% of sprees) — F3 is a model-authored-recipe wrong-data consequence prevented by the #107 hint, not a separate defect in this PR, so it's not filed standalone. Details for both in the run notes.

[paddymul]

Review findings addressed

Implemented following the repo's TDD rule: failing tests first in b7acc10 (CI red), fixes in e4e5527 (CI green). Per-finding trail below.

Medium

• M1 — asymmetric consistency guard → e4e5527. assert_catalog_consistent gains a third guard: every decomposed sidecar keyed by content hash (aliases.jsonl latest+history, chart_specs/<hash>, display_configs/<hash>) must resolve to a tracked recipe; otherwise it raises listing the orphans by surface. Pinned by test_consistency_guard_surfaces_dangling_alias_hash / ..._history_hash (b7acc10). One deliberate scope cut: notebook cells key on alias name, not hash, so they're out of scope — enforcing notebook→alias existence would break the legitimate "cell for an alias not yet created" pattern (test_reset_roundtrip_over_every_kind).
• M2 — lost alias-rollback coverage → e4e5527. Added test_alias_state_survives_reset (two aliases across two steps, reset back asserts only step-1 state + history, reset forward asserts both), mirroring the display-config/prompt survives-reset tests.

Low

• L1 — orphan chart/display not detected → e4e5527. Same third guard covers chart + display hashes. I took the reject option (over document-and-allow). Charts/display are read from git ls-files (the committed tree) so a transient untracked sidecar isn't false-flagged. Pinned by test_consistency_guard_surfaces_orphan_chart_hash / ..._display_config_hash (b7acc10).
• L2 / L3 / L5 / C1 — non-atomic writers → e4e5527. New tallyman_core/fsutil.atomic_write_text (tmp + os.replace); routed charts, display configs, notebook._save, write_manifest, the build's schema.json, post-processing/stats sources, and consolidated aliases._write onto it. The GITIGNORE_LINES comment now references the shared helper.
• L4 — partial entry-dir cleanup → e4e5527. The entry-dir population is wrapped in one try/except that rmtrees on any failure past target.mkdir; the two ad-hoc rmtrees are dropped. The target.exists() early-return guarantees we created the dir, so cleanup can't clobber a prior entry. New test_failed_build_after_copy_leaves_no_orphan_dir (b7acc10) induces a failure at make_portable_inplace. While here I also fixed a latent bug in the existing no-orphan test: its monkeypatch.undo() reverted isolated_home's TALLYMAN_HOME setenv (shared instance), pointing entries_dir at the real home and making the check vacuous — removed from both.
• L6 — silent malformed-line drop → e4e5527. notebook.load now lets json.loads raise (matching aliases._read / read_prompts); with the now-atomic _save, a torn line is genuine corruption worth surfacing. Pinned by test_load_raises_on_malformed_line (b7acc10).
• L7 — orphan sidecar intent unpinned → e4e5527. Resolved as part of L1 (reject + tests pin it).
• L8 — untested orphan-recipe branch → e4e5527. Added test_consistency_guard_surfaces_orphan_recipe.
• L9 — stale cache_clear rationale → e4e5527. Updated the delete-endpoint comment and test_cache_inspector.py comment: since ee0a90a, cached_result_expr self-heals on every read, so cache_clear() is defense-in-depth, not what averts the 500. Kept the call (cheap).
• L10 — PR title/body → retitled to feat: and the body now describes the shipped change (dropped the "design plan (no code)" framing).
• L11 — stale .gitignore open item → e4e5527. Dropped from the plan's open list (it's now concrete in GITIGNORE_LINES); narrowed to the genuinely-open catalog: restore the cross-entry DAG by recording resolved from_catalog parents in the manifest ({hash, ref, follow}) — catalog_parents went dark under #74 #84 limitation.
• L12 — wrong connect-shim test name → e4e5527. Risk register / Cut-B / sequence rows now name test_connect_shim_matches_xorq_api_connect and say "key + data equivalence," not the never-shipped *_byte_identical.

Additional gaps

• C1 — folded into the atomic-write pass above.
• C2 — out-of-lock prompt append race → e4e5527. _append_prompt is now atomic read-modify-write (tmp + replace) so the file is always whole JSONL. Pinned by test_append_prompt_is_atomic_and_complete.

Dismissed

• The test_api_projects_new_creates_and_switches genesis-coverage false positive — agreed, no action.

Full suite green locally (531 fast + 6 integration) and on CI.

[paddymul]

pack01 cheap-build finding + three doc-nits addressed

Followed the repo TDD cadence: failing test in 4f7f78b (CI red on test_cheap_build_validates_row_projection — DID NOT RAISE BuildError, 1 failed / 531 passed), fix in 8e686f8 (CI green — 532 fast + 6 integration).

[MEDIUM] Cheap build never exercised the row-level projection → 8e686f8

Confirmed the reproduction with a minimal in-process case: a cheap t.mutate(dd=t.d.cast("date")) over a parquet with one out-of-range date (2020-20-07) built successfully — count() is satisfied from source metadata and prunes the projection, so the cast never evaluated at author time.

The fix splits the build's execution by the structural verdict, which is now computed before execute (classify_build reads serialized YAML, no eval):

• worthy entry (Aggregate/Join/Sort/window/UDF): unchanged — baking its result-cache snapshot already evaluates every row, so count().execute() both validates and counts.
• cheap entry: streams the full result via loaded.to_pyarrow_batches() and discards it, forcing row-level evaluation at build. Constant memory, and the one pass also yields the exact row_count (no separate count()).

This restores the author-time validation the pre-native full-to_parquet build had, at the cost of a scan on cheap builds — the tradeoff the finding favored (option 1). The .limit(N) probe was rejected for the reason given (rare offending row). test_cheap_build_validates_row_projection pins both the error path and a valid-cast positive case (so the streaming row-count is covered, not just the raise).

Note on the same-run items: F2/F3 are tracked separately (#107 + run notes) and aren't changes to this PR, so they're out of scope here.

Doc-nits from the review pass → 8e686f8

• backend.py cited a test that doesn't exist (test_connect_shim_snapshot_is_byte_identical) and the wrong "byte-identical" claim. Now names the shipped test_connect_shim_matches_xorq_api_connect and says key + data equivalence (datafusion group-by order isn't deterministic). This was the source-code survivor of the earlier L12 (the plan doc was already corrected).
• build.py failed-build cleanup comment overclaimed "the target.exists() early-return guarantees we created this dir." Reworded: created_target is also set for a manifest-less mid-build leftover dir, which is still safe to rmtree (never a complete, durable entry — those early-return before the flag is set).
• catalog.py guard-3 docstring said "every decomposed file keyed by content hash" but silently omitted the hash-keyed prompts/<hash>.jsonl. Now states why it (append-only provenance, inert when orphaned) and the notebook (keyed by alias name) are out of scope.

🤖 Generated with Claude Code