Repository navigation
Conversation
…the recipe A cache-worthy entry (Aggregate/Join/Sort) is posted to Buckaroo's /load_expr as its raw recipe, so count() and every LIMIT 300 page re-run the full DAG instead of reading the materialised result.parquet. The new test asserts the posted build_dir is a cheap parquet read (classify_build worthy=False); it fails today because ensure_session hands over the expensive recipe. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
…recipe (#71) Opening a cache-worthy entry (Aggregate/Join/Sort) in the viewer re-ran the whole expression DAG on every count() and every LIMIT 300 page, because ensure_session posted the raw recipe build dir to /load_expr. The materialised result.parquet the build already wrote went unused. ensure_session now, for a cache-worthy entry, hands Buckaroo a build whose expression is a deferred_read_parquet of the materialised result instead of the recipe — count()/paging become a parquet scan, and push-down still holds on a parquet scan. Cheap entries keep loading the recipe (their recompute ≈ a parquet read). On any failure in result-build prep it falls back to the recipe so a cache hiccup can't blank the detail page. - ensure_result_build (result_cache.py): builds a portable deferred_read_parquet of the result (ensure_result guarantees the parquet first), marker-gated and lock-serialized like ensure_expanded_build. - new entry_result_build_dir / entry_result_expanded_build_dir path helpers, both registered as per-entry caches (regenerated on demand). - the recipe-expansion unit tests now use a cheap entry so they keep exercising the recipe path; cache-worthy entries take the new result-read path. Closes #71. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Contributor
Author
|
no longer relevant after 74 |
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Problem
Opening a cache-worthy catalog entry (Aggregate/Join/Sort) in the viewer recomputes the entire expression DAG on every page interaction — once for the row count and again for each row window — instead of reading the materialised
result.parquetthe build already wrote.BuckarooManager.ensure_sessionposts the entry's raw recipe build dir to/load_expr. That dir is the full recipe (from_catalog(...).group_by('plate').agg(...)over the multi-million-row source) with no read ofresult.parquet. So everyexpr.count().execute()and everyLIMIT 300window re-runs the whole DAG against datafusion. For an Aggregate,LIMIT 300can't push down past the group-by, so even the first 300-row page pays the full aggregation. Per the profiling in #71, Aggregate/Join recipes cost 0.9–1.3s on each of count and window, tracking input size, not output.Fix
In
ensure_session, for a cache-worthy entry (cache_worthy(project, content_hash)), hand Buckaroo a build whose expression is adeferred_read_parquetof the materialised result rather than the raw recipe:ensure_result_build(inresult_cache.py) callsensure_result(...)to guarantee the parquet is present, then builds a tiny portabledeferred_read_parquet(result)build, marker-gated and lock-serialized likeensure_expanded_build.ensure_sessionexpands that build into a dedicated.xorq_result_build_expandeddir and posts it. Cheap entries keep loading the recipe (their recompute ≈ a parquet read, so indirection buys nothing).Push-down still holds: a parquet scan pushes filters/limit/sort down too, so reading
result.parquetkeeps the push-down benefit while dropping the re-aggregation.Tests
test_unit_ensure_session_cache_worthy_serves_result_read— the posted build classifies as a cheap parquet read (classify_buildworthy=False) while the recipe stays worthy=True. Seen failing on CI before the fix.Closes #71.
🤖 Generated with Claude Code