Add hook for sharing join state in distributed execution by thinkharderdev · Pull Request #12523 · apache/datafusion

thinkharderdev · 2024-09-18T14:00:22Z

Which issue does this PR close?

Closes #12454

Rationale for this change

What changes are included in this PR?

Are these changes tested?

Are there any user-facing changes?

thinkharderdev · 2024-09-18T14:01:22Z

Set as draft for now as I still need to integrate into NestedLoopJoinStream but the basic approach is all here

thinkharderdev · 2024-09-18T14:02:14Z

                let reservation =
                    MemoryConsumer::new("HashJoinInput").register(context.memory_pool());
+
+                let probe_threads = shared_state


If shared state is available we use the num_task_partitions instead of output partitioning to determine the number of local probe threads

thinkharderdev · 2024-09-18T14:02:54Z

+            return Poll::Ready(Ok(StatefulStreamResult::Continue));
+        }
+
+        if let Some(shared_state) = build_side.left_data.shared_state.as_ref() {


When all local probe threads are complete and there is shared state we need to probe that before competing

thinkharderdev · 2024-09-18T14:03:43Z

        assert_batches_sorted_eq!(expected, &batches);
    }

+    struct Coordinator {


Usage example here. In a real-world scenario the Coordinator would likely be an external service which we communicate with through an rpc call

comphead · 2024-09-20T19:06:20Z

 use parking_lot::Mutex;

+/// `SharedJoinState` provides an extension point allowing
+/// `HashJoinStream` to share the `visited_indices_bitmap` of the build side of a join


is it left side or right side indices?

comphead · 2024-09-20T19:06:34Z

@korowa FYI

comphead · 2024-09-20T19:07:56Z

+/// across probe tasks without shared memory.
+///
+/// This can be used to, for example, implement a left outer join efficiently as a broadcast join
+/// if the left side is small


is it left side small or right? My feeling was the left(driving) table is huge and right is small

No, it's the opposite. The left (build) side is small and can be efficiently broadcast. Then the right (probe) side can be partitioned across multiple nodes with the build side broadcast to all of them.

comphead · 2024-09-20T19:10:24Z

+    }
+}
+
+fn merge_bitmap(m1: &mut BooleanBufferBuilder, m2: BooleanBuffer) -> Result<()> {


bitmap here is boolean bitmask of what was visited or matched by join/filter?

Correct. What will be shared and merged is the JoinLeftData::visited_indices_bitmap

comphead · 2024-09-20T19:16:31Z

Maybe its easier to build some diagram in draw.io or something?
I got the point about shared state but I'm not sure how it will be travelling from caller side and to caller side

thinkharderdev · 2024-09-20T20:33:41Z

Maybe its easier to build some diagram in draw.io or something? I got the point about shared state but I'm not sure how it will be travelling from caller side and to caller side

Yeah, good idea I'll work something up

github-actions · 2024-11-20T02:02:21Z

Thank you for your contribution. Unfortunately, this pull request is stale because it has been open 60 days with no activity. Please remove the stale label or comment or this will be closed in 7 days.

github-actions · 2025-01-20T01:58:42Z

Thank you for your contribution. Unfortunately, this pull request is stale because it has been open 60 days with no activity. Please remove the stale label or comment or this will be closed in 7 days.

github-actions · 2025-03-31T02:08:29Z

Thank you for your contribution. Unfortunately, this pull request is stale because it has been open 60 days with no activity. Please remove the stale label or comment or this will be closed in 7 days.

github-actions · 2025-06-14T02:09:37Z

Thank you for your contribution. Unfortunately, this pull request is stale because it has been open 60 days with no activity. Please remove the stale label or comment or this will be closed in 7 days.

--- [Cherry-pick summary: v46→v47] Source commit: 76905a1 (Hook for doing distributed `CollectLeft` joins (#269/apache#12523)) Strategy: cherry-picked, minor adaptions (3 conflicts in hash_join.rs; v47 refactored execute() to use try_once instead of once, requiring Ok() wrapper; partition check preserved; stale None first-arg dropped; mod.rs re-export conflict resolved by keeping both PhysicalExprRef import and new pub use exports) Upstream PR: fork-only (apache#12523 ref) Test coverage: adequate Tests: cargo nextest run -p datafusion-physical-plan passed (899 tests) Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>

Combines three fork-only commits from v49: - Hook for doing distributed CollectLeft joins (#269/apache#12523) - Add JoinContext with JoinLeftData to TaskContext in HashJoinExec (#300) - Make HASH_JOIN_SEED public (fork-only) Adds SharedJoinState/SharedJoinStateImpl trait for distributed probe coordination, JoinContext for sharing build-side state via TaskContext, contains_hash on JoinHashMapType, and converts process_unmatched_build_batch to async for shared state polling. Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>

Add hook for sharing join state in distributed execution

31faf2f

thinkharderdev requested review from alamb and comphead September 18, 2024 14:00

github-actions Bot added the physical-expr Changes to the physical-expr crates label Sep 18, 2024

thinkharderdev commented Sep 18, 2024

View reviewed changes

thinkharderdev mentioned this pull request Sep 18, 2024

Proposal: Hook to better support CollectLeft joins in distributed execution #12454

Open

comphead reviewed Sep 20, 2024

View reviewed changes

github-actions Bot added the Stale PR has not had any activity for some time label Nov 20, 2024

Dandandan removed the Stale PR has not had any activity for some time label Nov 20, 2024

Dandandan mentioned this pull request Nov 20, 2024

Left/full outer join incorrect for CollectLeft / broadcast apache/datafusion-ballista#1055

Open

github-actions Bot added the Stale PR has not had any activity for some time label Jan 20, 2025

github-actions Bot closed this Jan 27, 2025

Dandandan reopened this Jan 29, 2025

github-actions Bot removed the Stale PR has not had any activity for some time label Jan 30, 2025

github-actions Bot added the Stale PR has not had any activity for some time label Mar 31, 2025

github-actions Bot closed this Apr 10, 2025

Dandandan reopened this Apr 14, 2025

github-actions Bot removed the Stale PR has not had any activity for some time label Apr 15, 2025

github-actions Bot added the Stale PR has not had any activity for some time label Jun 14, 2025

github-actions Bot closed this Jun 21, 2025

avantgardnerio pushed a commit to coralogix/arrow-datafusion that referenced this pull request Dec 15, 2025

Hook for doing distributed CollectLeft joins (#269/apache#12523)

925734a

avantgardnerio pushed a commit to coralogix/arrow-datafusion that referenced this pull request Dec 15, 2025

Hook for doing distributed CollectLeft joins (#269/apache#12523)

1f1e211

dispanser pushed a commit to coralogix/arrow-datafusion that referenced this pull request Jan 7, 2026

Hook for doing distributed CollectLeft joins (#269/apache#12523)

76905a1

dispanser mentioned this pull request Feb 26, 2026

Datafusion upgrade to DF 47 coralogix/arrow-datafusion#397

Closed

Conversation

thinkharderdev commented Sep 18, 2024

Which issue does this PR close?

Rationale for this change

What changes are included in this PR?

Are these changes tested?

Are there any user-facing changes?

Uh oh!

thinkharderdev commented Sep 18, 2024

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

comphead commented Sep 20, 2024

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

comphead commented Sep 20, 2024

Uh oh!

thinkharderdev commented Sep 20, 2024

Uh oh!

github-actions Bot commented Nov 20, 2024

Uh oh!

github-actions Bot commented Jan 20, 2025

Uh oh!

github-actions Bot commented Mar 31, 2025

Uh oh!

github-actions Bot commented Jun 14, 2025

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

3 participants