Skip to content

Push filters that contain references to outer/lateral scopes.#3386

Merged
nicktobey merged 4 commits intomainfrom
nicktobey/push-filters-from-outer-scope
Jan 23, 2026
Merged

Push filters that contain references to outer/lateral scopes.#3386
nicktobey merged 4 commits intomainfrom
nicktobey/push-filters-from-outer-scope

Conversation

@nicktobey
Copy link
Copy Markdown
Contributor

We attempt to push filter expressions deeper into the tree so that they can reduce the size of intermediate iterators. Ideally, we want to push filters to directly above the data source that they reference.

Previously, we only pushed filters if they only referenced a single table, since pushing a filter that referenced multiple tables could potentially move the filter to a location where one of the referenced tables is no longer in scope. However, if the extra table references refer to a table in an outer scope or lateral scope, pushing the filter is completely safe. GetFields that reference an outer or lateral scope can be effectively treated as literals for the purpose of this optimization.

This PR changes getFiltersByTable, a function that maps tables onto the filters that reference those tables. Previously it would ignore filters that reference multiple tables. Now, it allows those filters provided that the extra references are to outer/lateral scopes.

This improves many of the plan tests:

  • The changed test in tpch_plans.go pushes a filter into the leftmost table lookup
  • The second changed test in query_plans.go replaces a naive InnerJoin with a CrossHashjoin
  • integration_plans.go shows many queries that now have an IndexedTableAccess instead of a table scan, or where we push a filter deeper into a join.

A small number of neutral / slightly negative changes:

  • One of the changes in integration_plans.go introduces a redundant filter that was previously being removed. In practice this is pretty benign because filters rarely impact the runtime unless they require type conversions.
  • The first changed test in query_plans.go replaces a LookupJoin with a LateralCrossJoin on an IndexedTableAccess. These two plans are effectively equivalent, but the LateralCrossJoin is harder to analyze, has a larger estimated cost and larger row estimate, and could in theory inhibit subsequent optimizations. I imagine we could create a new analysis pass that converts this kind of LateralCrossJoin into a LookupJoin.

Copy link
Copy Markdown
Contributor

@fulghum fulghum left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Nice work! LGTM

@nicktobey nicktobey merged commit 53aea15 into main Jan 23, 2026
8 checks passed
@nicktobey nicktobey deleted the nicktobey/push-filters-from-outer-scope branch January 23, 2026 04:40
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants