Skip to content

fix: Prevent predicate pushdown across Sort with baked-in slice - #26804

Merged
ritchie46 merged 2 commits into
pola-rs:mainfrom
gautamvarmadatla:fix/predicate-pushdown-sort-slice
Mar 5, 2026
Merged

ritchie46 merged 2 commits into
pola-rs:mainfrom
gautamvarmadatla:fix/predicate-pushdown-sort-slice

Conversation

@gautamvarmadatla

Copy link
Copy Markdown
Contributor

Fixes : #26803

So, TLDR is #26495 swapped the optimizer pass order so that slice pushdown now runs before predicate pushdown, which caused the issue. Previously, the standalone Slice IR node acted as a hard barrier in predicate pushdown, calling no_pushdown_restart_opt to stop all predicates from passing through. With slice pushdown running first, that Slice node gets baked into Sort { slice: Some(...) } before predicate pushdown runs. Predicate pushdown treats Sort as push-through (it calls pushdown_and_continue), and there was no special-case barrier behavior for Sort { slice: Some(...) }. So user predicates now slip past the row limit.

Also, the DynamicPred change in the same commit compounds the problem for single-column sorts by injecting an additional predicate into the same pushdown flow, but it is not the fundamental reason user predicates cross the row-limit boundary.

To fix this i basically drained accumulated predicates to local when Sort has a baked-in slice and reapplied them above the node via optional_apply_predicate.

@github-actions github-actions Bot added fix Bug fix python Related to Python Polars rust Related to Rust Polars title needs formatting labels Mar 4, 2026
@gautamvarmadatla gautamvarmadatla changed the title fix: prevent predicate pushdown across Sort with baked-in slice fix: Prevent predicate pushdown across Sort with baked-in slice Mar 4, 2026
@codecov

codecov Bot commented Mar 4, 2026 •

Copy link
Copy Markdown

Codecov Report

✅ All modified and coverable lines are covered by tests.
✅ Project coverage is 81.58%. Comparing base (9f1a742) to head (ab37e5c).
⚠️ Report is 2 commits behind head on main.

Additional details and impacted files
@@            Coverage Diff             @@
##             main   #26804      +/-   ##
==========================================
+ Coverage   81.30%   81.58%   +0.27%     
==========================================
  Files        1802     1802              
  Lines      246972   246991      +19     
  Branches     3086     3086              
==========================================
+ Hits       200810   201498     +688     
+ Misses      45371    44703     -668     
+ Partials      791      790       -1     

☔ View full report in Codecov by Sentry.
📢 Have feedback on the report? Share it here.

🚀 New features to boost your workflow:
  • ❄️ Test Analytics: Detect flaky tests, report on failures, and find test suite problems.

@ritchie46

Copy link
Copy Markdown
Member

Thanks @gautamvarmadatla

@ritchie46
ritchie46 merged commit 451f123 into pola-rs:main Mar 5, 2026
29 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

fix Bug fix python Related to Python Polars rust Related to Rust Polars

Projects

None yet

Development

Successfully merging this pull request may close these issues.

predicate pushdown optimizer incorrectly pushes .filter() past .sort().head() (and related row-limiting operations) in lazy queries

2 participants