Skip to content

patch: Avoid pyspark no partition for window ops warning#2429

Merged
FBruzzesi merged 1 commit intomainfrom
patch/pyspark-no-partition-for-window-ops
Apr 24, 2025
Merged

patch: Avoid pyspark no partition for window ops warning#2429
FBruzzesi merged 1 commit intomainfrom
patch/pyspark-no-partition-for-window-ops

Conversation

@FBruzzesi
Copy link
Member

What type of PR is this? (check all applicable)

  • 💾 Refactor
  • ✨ Feature
  • 🐛 Bug Fix
  • 🔧 Optimization
  • 📝 Documentation
  • ✅ Test
  • 🐳 Other

Related issues

First noticed in #2417 comment

Checklist

  • Code follows style guide (ruff)
  • Tests added
  • Documented the changes

If you have comments or can explain your changes, please do so below

Depending on pyspark and python version I started to get the following warning:

No Partition Defined for Window operation! Moving all data to a single partition, this can cause serious performance degradation.

@FBruzzesi FBruzzesi added fix pyspark Issue is related to pyspark backend labels Apr 24, 2025
Copy link
Member

@MarcoGorelli MarcoGorelli left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

thanks @FBruzzesi

we should probably expressify partition_by / order_by at some point too

@FBruzzesi FBruzzesi mentioned this pull request Apr 24, 2025
10 tasks
@FBruzzesi FBruzzesi changed the title (RFC) patch: Avoid pyspark no partition for window ops warning patch: Avoid pyspark no partition for window ops warning Apr 24, 2025
@FBruzzesi FBruzzesi merged commit 57c9442 into main Apr 24, 2025
34 checks passed
@FBruzzesi FBruzzesi deleted the patch/pyspark-no-partition-for-window-ops branch April 24, 2025 14:19
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

fix pyspark Issue is related to pyspark backend

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants