fix(xorq): an ordered copy keeps its source's types and reader options, and its digest file is written atomically (#197, #198, #211)

[paddymul]

Stacked on #189: the base is feat/adr-007-009-cache-redesign, not main. Fixes #197, Fixes #198, Fixes #211.

An ordered copy is the parquet copy of a source that a recipe reads instead of the source: the source's rows in file order plus a last column, __row_order, 0..N-1 (ADR-008 D2, every file tallyman reads carries __row_order). It lives at compute_cache/ordered_sources/<key>.parquet, and a <key>.digest file beside it caches its content digest. The manifest records the reader options it was made with, so ensure_materialized can make a deleted copy again (ADR-007 D13, which files are cache).

What changed

• ordered copies of parquet sources change column types — date64 becomes a timestamp, a map becomes array<struct>, decimal256 panics polars #197, the parquet writer (1ac8fc7). pyarrow writes the copy of a parquet source (ordered_copy._write_parquet_copy). It streams ParquetFile.iter_batches() in file order, regroups the rows into 122,880-row groups whatever the source's row groups are, appends __row_order from np.arange, and writes with the snapshot's parquet settings (materialize._PARQUET_OPTIONS: zstd level 3, format 2.6, statistics, a page index). The copy's Arrow schema is the source's plus __row_order, so date64, map, time32, time64 and decimal256 columns keep their types, and a recipe sees the ibis types a direct read of the source gives. A worthy aggregate over a decimal256(40, 2) column builds and sums it exactly.
• tallyman_read_csv hands polars JSON-rewritten reader options on the first ingest — a callable option arrives as a string and panics #198, and ordered copies of parquet sources change column types — date64 becomes a timestamp, a map becomes array<struct>, decimal256 panics polars #197's panic on the CSV path (07eb3d0). tallyman_read_csv passes the caller's own schema and scan_csv options to the first write. Their JSON form in the manifest still names the copy and is what recreate_ordered_copy replays. That function already refuses a record marked lossless: False, so a deleted copy of a read with a function option says it cannot be made again. polars still parses a CSV, and a PanicException from it now becomes a BuildError that names the source. The panic is a BaseException, so before this it got past both the build's except Exception and the MCP tool's except BuildError. A with_column_names function that raises is one way to cause one.
• an ordered copy's .digest sidecar is read without the lock and written in place — a build can record an empty content digest #211 (c267f7d). The sidecar is written to a unique temp name and moved into place with os.replace, before the copy itself is, so a reader that finds the copy finds its whole digest beside it. An empty sidecar is treated as missing: the digest is read back from the copy and the sidecar written again. The race as the issue describes it needs an unlocked writer, because a build and both copy writers hold the project lock. Two things could still leave the sidecar empty: an unlocked read_project_file (a notebook process) that rewrites a missing sidecar, or an in-place write cut off after its truncate. The fix covers both.

Docs: ADR-008's and ADR-009's "Implementation notes", and the lines in docs/caching.md and docs/system-contract.md that said polars writes every copy.

The snapshot format version stays at 1

SNAPSHOT_FORMAT_VERSION (ADR-009 D3, the snapshot format) stands for the settings that decide the batch boundaries an entry sees, and the ordered copy's 122,880-row groups are one of them. The pyarrow copy's bytes differ from the polars copy's, but its row groups are where they were, and polars also wrote a page index. I measured the 1.5M-row test source (tests/big_parquet.py) on the single-partition materialization connection. The two copies have the same content digest. The SUM and AVG of the float column come out bit for bit the same ungrouped, filtered on the float, and over four ranges of the sorted id column, which the page index prunes. The two copies of orders.parquet also have matching digests. A version bump would make any later unfaithful heal of an existing entry blame a format change that did not change what the entry read.

What does change is the columns polars did not keep. An entry built before this over a date64, map, time32 or time64 column read a copy with other types. If its copy is made again, recreate_ordered_copy finds that the content digest differs from the recorded one and records unfaithful_ordered_copy. Rebuilding such an entry is the fix.

How it was built

1. 2665f80: six failing tests. CI run: ruff passed and the fast suite failed on exactly those six (6 failed, 864 passed), each for the reason its issue gives.
2. 1ac8fc7, 07eb3d0, c267f7d: one fix per issue, each making its own tests pass. CI run: ruff, the fast suite (870 passed) and the integration suite (7 passed) all pass.

Locally, on c267f7d: the fast suite gives 863 passed, 6 skipped and 1 failed. The failure is test_fouc.py::test_unknown_api_path_404s, which gets a 503 because this worktree has no packages/app/dist, and it fails the same way without these commits. I also checked the new writer on an empty source, a source whose only column is __row_order, a pandas categorical spanning row groups with different dictionaries, and list and struct columns.

Not in this PR

• fixed_size_binary: xorq 0.3.26 cannot read it (KeyError: FixedSizeBinaryType while it builds the table's schema), from the source or from the copy. polars turned it into binary, so a recipe over one used to build. Now it fails the way a direct read of the source does.
• The copy of a source with unique numeric columns is larger: 25.5 MB against 16.2 MB from polars for the 1.5M-row test source. The snapshot settings dictionary-encode every column, and an int64 or float64 dictionary never overflows in a 122,880-row group. Snapshots pay the same cost. Changing use_dictionary would change both files, so I left it.
• The copy's key includes the JSON form of the reader options, and for a function that is its repr, which holds the function's memory address. So two builds of the same recipe with a function option share an entry only if the function lands at the same address. Four builds, two in each of two processes, gave four entries. Within one process the address is never reused, because each build's recipe module stays in sys.modules.
• _row_groups repeats the regrouping in materialize._stream_to_parquet. I left materialize.py alone because fix(xorq): a failed build keeps the snapshot already on disk for its hash #213 edits it.

🤖 Generated with Claude Code

[paddymul]

Superseded by the ADR-011 stack (#217, #218, #219). ADR-011 deleted the ordered copy; the #197 fix (pyarrow writes a parquet source's snapshot) and the #198 fix (reader options fixed at import) are in the stack, and the .digest sidecar #211 was about no longer exists.