perf(api): paginate /api/data off cached_result_expr, not a full result.parquet (#90)

[paddymul]

Addresses #90.

What

/api/data (app.py:518) paginated a windowed read by first materialising the entry's entire result to disk: it called ensure_result unconditionally, which cached_result_expr(...).to_parquet(result_path)s the full result, then read offset+limit rows back and sliced. An O(limit) request did O(rows) work and wrote a file — for a cheap entry that reverses #74's "materialise nothing"; for an expensive entry it duplicates the baked snapshot under the entry.

This serves the page straight off the expression cached_result_expr already hands back:

• total comes from the manifest's row_count (recorded at build; api_entry_detail reads it the same way), so no file is needed to populate it.
• The page is cached_result_expr(project, content_hash).limit(limit, offset=offset).execute(). The window pushes down — for an expensive entry it lowers to a windowed read of the baked snapshot, for a cheap entry the limit pushes through to the source read. Neither writes result.parquet.

ensure_result stays for the callers that need a parquet path (downloads, promote-diff, post-processing) — this is the one read path that needs rows, not a path.

The now-unused _read_head_rows helper (and its pyarrow import) are removed.

Tests

test_api_data_cheap_entry_serves_page_without_materialising and test_api_data_expensive_entry_paginates_without_per_entry_parquet assert no per-entry result.parquet is written when a page is served (cheap path returns the right page+total; expensive path paginates in order across a deep offset). Both are committed first so the regression is visible failing on CI, then the fix lands.

Context

Surfaced while reconciling the caching docs with #74. Same ensure_result-on-read mechanism as #79 (concurrent materialisation race); adjacent to #71 (viewer re-runs the recipe on load).

🤖 Generated with Claude Code

[paddymul]

Post-merge review

Reviewed #92 after merge. The change is correct and a real win — it drops an O(rows) full-result write per first page load and restores #74's "materialise nothing" on the read path. Verified the property it now leans on: order-sensitive ops (Aggregate/Join/Sort) are classified worthy → baked to a stable snapshot, so only order-preserving ops take the per-page re-execute path; limit(0)/offset semantics and cross-execution scan-order stability check out empirically.

Findings and dispositions:

#	Finding	Disposition
1	api_data guards on entry_build_dir but reads manifest.json unguarded for total; the build dir is created before the manifest, so a half-built/pruned entry → FileNotFoundError → 500.	Fixed in #95 — fix(#90) 6c5b99c (guarded read, best-effort total; repro abd5e66 seen failing on CI first).
2	total is build-time row_count, not actual rows.	Accepted — consistent with api_entry_detail; only drifts for a nondeterministic source, which #88 lints against.
3	cached_result_expr's @lru_cache outlives a compute-cache prune (reset_to), so a post-reset read executes an expression pointing at a deleted snapshot; the path.exists() self-heal only runs on a cache miss. #92 widened the blast radius to the per-page endpoint.	Filed as #96.
4	Pagination ordering across pages for a cheap, unsorted entry (independent per-page re-execution) was untested — the cheap test fetched only offset=0; the deep-offset test used order_by.	Fixed in #95 — test(#90) 3515bf0 (two pages == matching slices of one full read).
5	Nits: inlines entry_dir(...) / ENTRY_MANIFEST_FILENAME over the entry_manifest_path() helper; tests hardcode "result.parquet".	Accepted — matches the neighbouring api_entry_detail / existing test style.
6	PR body lists "downloads" as a remaining ensure_result caller; actual callers are companion/xorq diff + post_processing + salt-mode result_cache.	Doc-only; no code change.

Fixes for #1 and #4 are in #95 (green on CI: ruff ✓, fast ✓, integration ✓). #3 is tracked in #96.