Repository navigation
perf(viewer): serve cache-worthy entries from result.parquet + share the path with Tier-B (#71) - #102
Merged
Merged
Conversation
…the recipe A cache-worthy entry (Aggregate/Join/Sort) is posted to Buckaroo's /load_expr as its raw recipe, so count() and every LIMIT 300 page re-run the full DAG instead of reading the materialised result.parquet. The new test asserts the posted build_dir is a cheap parquet read (classify_build worthy=False); it fails today because ensure_session hands over the expensive recipe. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
…recipe (#71) Opening a cache-worthy entry (Aggregate/Join/Sort) in the viewer re-ran the whole expression DAG on every count() and every LIMIT 300 page, because ensure_session posted the raw recipe build dir to /load_expr. The materialised result.parquet the build already wrote went unused. ensure_session now, for a cache-worthy entry, hands Buckaroo a build whose expression is a deferred_read_parquet of the materialised result instead of the recipe — count()/paging become a parquet scan, and push-down still holds on a parquet scan. Cheap entries keep loading the recipe (their recompute ≈ a parquet read). On any failure in result-build prep it falls back to the recipe so a cache hiccup can't blank the detail page. - ensure_result_build (result_cache.py): builds a portable deferred_read_parquet of the result (ensure_result guarantees the parquet first), marker-gated and lock-serialized like ensure_expanded_build. - new entry_result_build_dir / entry_result_expanded_build_dir path helpers, both registered as per-entry caches (regenerated on demand). - the recipe-expansion unit tests now use a cheap entry so they keep exercising the recipe path; cache-worthy entries take the new result-read path. Closes #71. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
…er-B proxy (#71) Pull the recipe-vs-result-read selection out of ensure_session into a single ensure_viewer_expanded_build(project, content_hash) in result_cache.py, and point both the companion's ensure_session and the Tier-B page-load proxy at it, so the harness pages over exactly what Buckaroo is served and the two paths can't drift. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Resolve conflicts against the #73 result-cache rework on main: - result_cache.py: keep main's reworked classify_build / cached_result_expr / ensure_result base; layer #71's additive ensure_result_build + ensure_viewer_expanded_build on top (they only need ensure_result to return a parquet path, which main still provides). Union the import/logger block. - paths.py: keep main's ENTRY_RESULT_FILENAME / entry_result_path and add #71's entry_result_build_dir / entry_result_expanded_build_dir; drop the now-unused result_cache_dir (main removed it). - buckaroo_lifecycle.ensure_session auto-merged: main's pre-lock snapshot self-heal (cached_result_expr) and #71's ensure_viewer_expanded_build coexist. - tests: keep both new ensure_session tests (cache-worthy-serves-result-read + self-heals-evicted-snapshot); Tier-B proxy uses ensure_viewer_expanded_build plus main's cached_result_expr heal. test_buckaroo / test_result_cache / test_paths / test_source_cache pass; ruff clean. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
paddymul
added a commit
that referenced
this pull request
Jun 16, 2026
…cache() Delete every "materialise a per-entry result.parquet on demand" path. The xorq .cache() snapshot (baked at build for expensive entries) is the single materialised copy; cheap entries recompute on read. Finishes the #73/#71/#90 arc and unwinds #102's viewer-from-result.parquet layer, which merged after the plan and was built on the same on-demand parquet. Source: - result_cache: delete ensure_result and the salt read branch (salt entries fall through to recompute); delete #102's ensure_result_build / ensure_viewer_expanded_build. - post_processing / companion.diff.build_diff_expr: read cached_result_expr. - xorq.diff.full_diff: xorq-only; require a_expr/b_expr (ValueError if missing); drop the pandas/polars file-reading backends, _ensure_entry_parquet and the backend kwarg. Call sites drop backend="xorq". - paths: delete ENTRY_RESULT_FILENAME / entry_result_path and #102's .xorq_result_build* names + helpers; trim ENTRY_CACHE_NAMES. manifest: drop result_path. core.__init__: drop the exports. - companion.app: drop the disk-usage "results" bucket; drive the cache inspector off the manifest (cache_bytes, #87), filtered to entries that bake a snapshot; delete unlinks the baked snapshot. - buckaroo_lifecycle: ensure_session serves the recipe build again (_entry_parquet_exists -> _entry_exists, keyed on the build dir). - build: bump the migration marker; sweep result.parquet + dead #102 dirs. - Frontend: reword CachePage/CatalogPage/NotebookPage off "result.parquet". Tests: retarget the ensure_result/result.parquet fixtures onto cached_result_expr / baked snapshots (result_cache, diff, cache_inspector, disk_usage, buckaroo, buckaroo_multi_project, paths, manifest, perf_integration); migrate the cache lab onto cached_result_expr (snapshot cold-vs-warm). Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
This was referenced Jun 17, 2026
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Problem
Opening a cache-worthy catalog entry (Aggregate/Join/Sort) in the viewer recomputes the entire expression DAG on every page interaction — once for the row count and again for each row window — instead of reading the materialised
result.parquetthe build already wrote.BuckarooManager.ensure_sessionposts the entry's raw recipe build dir to/load_expr, so push-down lands on the full recipe rather than a parquet scan.Change
ensure_sessionnow serves adeferred_read_parquetof the materialised result for cache-worthy entries (cheap entries keep loading the recipe; any cache hiccup falls back to the recipe so a hiccup can't blank the detail page).ensure_viewer_expanded_build(project, content_hash)inresult_cache.py, and bothensure_session(production) and the Tier-B page-load proxy call it — so the harness pages over exactly what Buckaroo is served and the two paths can't drift.Status — superseded, preserved for reference
This is the original #71 line, built on the on-demand
result.parquetlayer.mainhas since moved tocached_result_expr/ xorq.cache(), PR #100 already uses that route, and PR #101 plans to remove the on-demandresult.parquetlayer entirely. Opening this so the work (and theensure_viewer_expanded_buildconsolidation) is reviewable in one place — not as a merge candidate over the newer stack. The original PR #72 was closed in favour of that reorg.Tests:
test_result_cache.py+test_buckaroo.py(22) pass locally; ruff clean.🤖 Generated with Claude Code