docs(plans): ADR-009 digest stability (proposed)

[paddymul]

Draft ADR for discussion. Adds plans/ADR-009-digest-stability.md and the two spikes it cites, scripts/spike_float_aggregate_digest.py and scripts/spike_logical_digest.py. No behaviour change.

A heal whose digest does not match is loud and sticky: a durable unfaithful_heal record, an SSE event, a stat-cache wipe, session eviction, and a pin and badge on the entry (ADR-006 D7, D10, D12). Two things trigger that today without any change in the result.

1. A float SUM/AVG is not bit-reproducible under DataFusion's parallel partial aggregation. The same canonically ordered group-by over 3,000,000 rows gives 6 distinct digests in 6 runs at the default partition count and 1 at target_partitions = 1; an integer-only aggregate is stable either way. Aggregate is the usual reason an entry is worthy, so most expensive entries with a float measure are flagged after any eviction.
2. File bytes depend on things that are not the result: how the stream was batched, and the writer's version string in the parquet footer, so every heal after a pyarrow upgrade would mismatch.

Decisions proposed:

• D1: materialization runs on a connection with target_partitions = 1, for the build and every heal. About 3x on aggregation at spike scale, paid per bake and never per read. Gated on measuring a parking-corpus rebuild, with a recorded float-only fallback.
• D2: result_digest becomes a SHA-256 over the snapshot's ordered Arrow content, computed from the file as read back by one function shared by build and verify. It is invariant to batch size, codec and row-group size, survives a trip through DataFusion, and still detects a changed value, a null replaced by 0.0, and a row swap. It costs 0.12 s to read back and hash a 57 MB file against 0.02 s for a file-bytes hash. This reverses part of ADR-006 D5's reasoning, and the ADR says so.
• D3: the snapshot format for tallyman's writer: zstd, 1,048,576-row groups, each combined before writing. 57.1 MB and a 2.7 KB footer against 100.3 MB and a 228.5 KB footer in the shape xorq writes.
• D3 (revised 2026-09-20): the format also takes two requirements from ADR-008 (docs(plans): ADR-008 row order of reads via a baked __row_order column (proposed) #181): the writer numbers rows in a last column named __row_order, and it writes a parquet page index, which takes a range request from 79-90 ms to 19-24 ms.
• D4: an unfaithful_heal record carries engine and writer versions at build and at heal, so an upgrade is not blamed on the recipe.

One of three ADR PRs from the 2026-09-18 cache audit, with ADR-007 (#180) and ADR-008 (#181). References to the other two do not resolve until those merge. D2 and D3 assume ADR-007's tallyman-side writer; D1 stands without it.

Checked locally: ruff check and ruff format --check on both scripts, both run from a clean temp dir and their output matches the tables in the ADR, and every path and line reference in the ADR resolves on a tree with all three ADRs present. Docs-only, so CI was not watched.

🤖 Generated with Claude Code

[paddymul]

Superseded by #184, which combined the ADR-007, ADR-008 and ADR-009 drafts; that text is in #189.