Skip to content

Conversation

colin-ho
Copy link
Contributor

@colin-ho colin-ho commented Sep 4, 2025

Changes Made

Adds logic to elide shuffles for either side of a hash join when input num partitions != num partitions for join or input is not hash partitioned on the join key. Same logic as https://github.com/Eventual-Inc/Daft/blob/main/src/daft-physical-plan/src/physical_planner/translate.rs#L1441

Related Issues

Closes #5127

Checklist

  • Documented in API Docs (if applicable)
  • Documented in User Guide (if applicable)
  • If adding a new documentation page, doc is added to docs/mkdocs.yml navigation
  • Documentation builds and is formatted properly (tag @/ccmao1130 for docs review)

@github-actions github-actions bot added the feat label Sep 4, 2025
@srilman
Copy link
Contributor

srilman commented Sep 4, 2025

Dude you have so many actively queued PRs, I'm not doing this on purpose

@colin-ho colin-ho marked this pull request as ready for review September 4, 2025 16:14
@colin-ho colin-ho requested a review from srilman September 4, 2025 16:14
Copy link

codecov bot commented Sep 9, 2025

Codecov Report

❌ Patch coverage is 0% with 50 lines in your changes missing coverage. Please review.
✅ Project coverage is 74.25%. Comparing base (33884be) to head (c4f02e9).
⚠️ Report is 3 commits behind head on main.

Files with missing lines Patch % Lines
...stributed/src/pipeline_node/join/translate_join.rs 0.00% 50 Missing ⚠️
Additional details and impacted files

Impacted file tree graph

@@            Coverage Diff             @@
##             main    #5135      +/-   ##
==========================================
+ Coverage   73.81%   74.25%   +0.43%     
==========================================
  Files         957      957              
  Lines      124278   123286     -992     
==========================================
- Hits        91740    91547     -193     
+ Misses      32538    31739     -799     
Files with missing lines Coverage Δ
...ft-distributed/src/pipeline_node/join/hash_join.rs 0.00% <ø> (ø)
...stributed/src/pipeline_node/join/translate_join.rs 0.00% <0.00%> (ø)

... and 26 files with indirect coverage changes

🚀 New features to boost your workflow:
  • ❄️ Test Analytics: Detect flaky tests, report on failures, and find test suite problems.
  • 📦 JS Bundle Analysis: Save yourself from yourself by tracking and limiting bundle sizes in JS merges.

@colin-ho colin-ho merged commit 10ccc4a into main Sep 9, 2025
68 of 71 checks passed
@colin-ho colin-ho deleted the colin/elide-shuffle-for-hash-join branch September 9, 2025 19:08
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
Projects
None yet
Development

Successfully merging this pull request may close these issues.

Improve the performance of hash joins on Flotilla when inputs only have 1 partition
2 participants