Skip to content

chore: improve RepartitionExec display tree #15606

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Merged
merged 2 commits into from
Apr 8, 2025
Merged

chore: improve RepartitionExec display tree #15606

Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
2 commits
Select commit Hold shift + click to select a range
f3b75b3
fix RepartitionExec tree displayformat
getChan Apr 6, 2025
c044bc3
pass sqllogictests
getChan Apr 6, 2025
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
2 changes: 1 addition & 1 deletion datafusion/physical-plan/src/repartition/mod.rs
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -510,7 +510,7 @@ impl DisplayAs for RepartitionExec {
writeln!(f, "partitioning_scheme={}", self.partitioning(),)?;
writeln!(
f,
"output_partition_count={}",
"input_partition_count={}",
self.input.output_partitioning().partition_count()
Copy link
Contributor

@2010YOUY01 2010YOUY01 Apr 7, 2025

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I think the output partition number is required to display in this node, but it's not implemented correctly. (or maybe now the planner always set the output partition the same as the input partition count)
I suggest to change the implementation instead, maybe we can display both the input and output partition count.

Copy link
Contributor Author

@getChan getChan Apr 7, 2025

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

thanks for review.
The output partition number is already included in the partitioning_scheme, so it is displayed.
ex)

13)┌─────────────┴─────────────┐
14)│      RepartitionExec      │
15)│    --------------------   │
16)│   input_partition_count:  │
17)│             1             │
18)│                           │
19)│    partitioning_scheme:   │
20)│     RoundRobinBatch(4)    │
21)└─────────────┬─────────────┘

Copy link
Contributor

@2010YOUY01 2010YOUY01 Apr 7, 2025 •
edited
Loading

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Maybe as a follow-up task we can update it to be more clear like

input_partition_count: 1
output_partition_count: 4

partition_scheme: RoundRobin

)?;
if self.preserve_order {
Expand Down
76 changes: 38 additions & 38 deletions datafusion/sqllogictest/test_files/explain_tree.slt
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -180,7 +180,7 @@ physical_plan
13)┌─────────────┴─────────────┐
14)│ RepartitionExec │
15)│ -------------------- │
16)│ output_partition_count: │
16)│ input_partition_count: │
17)│ 1 │
18)│ │
19)│ partitioning_scheme: │
Expand Down Expand Up @@ -218,7 +218,7 @@ physical_plan
18)┌─────────────┴─────────────┐
19)│ RepartitionExec │
20)│ -------------------- │
21)│ output_partition_count: │
21)│ input_partition_count: │
22)│ 4 │
23)│ │
24)│ partitioning_scheme: │
Expand All @@ -236,7 +236,7 @@ physical_plan
36)┌─────────────┴─────────────┐
37)│ RepartitionExec │
38)│ -------------------- │
39)│ output_partition_count: │
39)│ input_partition_count: │
40)│ 1 │
41)│ │
42)│ partitioning_scheme: │
Expand Down Expand Up @@ -311,7 +311,7 @@ physical_plan
19)┌─────────────┴─────────────┐┌─────────────┴─────────────┐
20)│ RepartitionExec ││ RepartitionExec │
21)│ -------------------- ││ -------------------- │
22)│ output_partition_count: ││ output_partition_count: │
22)│ input_partition_count: ││ input_partition_count: │
23)│ 4 ││ 4 │
24)│ ││ │
25)│ partitioning_scheme: ││ partitioning_scheme: │
Expand All @@ -320,7 +320,7 @@ physical_plan
28)┌─────────────┴─────────────┐┌─────────────┴─────────────┐
29)│ RepartitionExec ││ RepartitionExec │
30)│ -------------------- ││ -------------------- │
31)│ output_partition_count: ││ output_partition_count: │
31)│ input_partition_count: ││ input_partition_count: │
32)│ 1 ││ 1 │
33)│ ││ │
34)│ partitioning_scheme: ││ partitioning_scheme: │
Expand Down Expand Up @@ -386,7 +386,7 @@ physical_plan
40)-----------------------------┌─────────────┴─────────────┐┌─────────────┴─────────────┐
41)-----------------------------│ RepartitionExec ││ RepartitionExec │
42)-----------------------------│ -------------------- ││ -------------------- │
43)-----------------------------│ output_partition_count: ││ output_partition_count: │
43)-----------------------------│ input_partition_count: ││ input_partition_count: │
44)-----------------------------│ 4 ││ 4 │
45)-----------------------------│ ││ │
46)-----------------------------│ partitioning_scheme: ││ partitioning_scheme: │
Expand All @@ -395,7 +395,7 @@ physical_plan
49)-----------------------------┌─────────────┴─────────────┐┌─────────────┴─────────────┐
50)-----------------------------│ RepartitionExec ││ RepartitionExec │
51)-----------------------------│ -------------------- ││ -------------------- │
52)-----------------------------│ output_partition_count: ││ output_partition_count: │
52)-----------------------------│ input_partition_count: ││ input_partition_count: │
53)-----------------------------│ 1 ││ 1 │
54)-----------------------------│ ││ │
55)-----------------------------│ partitioning_scheme: ││ partitioning_scheme: │
Expand Down Expand Up @@ -434,7 +434,7 @@ physical_plan
17)┌─────────────┴─────────────┐
18)│ RepartitionExec │
19)│ -------------------- │
20)│ output_partition_count: │
20)│ input_partition_count: │
21)│ 1 │
22)│ │
23)│ partitioning_scheme: │
Expand Down Expand Up @@ -496,7 +496,7 @@ physical_plan
41)┌─────────────┴─────────────┐
42)│ RepartitionExec │
43)│ -------------------- │
44)│ output_partition_count: │
44)│ input_partition_count: │
45)│ 1 │
46)│ │
47)│ partitioning_scheme: │
Expand Down Expand Up @@ -530,7 +530,7 @@ physical_plan
13)┌─────────────┴─────────────┐
14)│ RepartitionExec │
15)│ -------------------- │
16)│ output_partition_count: │
16)│ input_partition_count: │
17)│ 1 │
18)│ │
19)│ partitioning_scheme: │
Expand Down Expand Up @@ -566,7 +566,7 @@ physical_plan
15)┌─────────────┴─────────────┐
16)│ RepartitionExec │
17)│ -------------------- │
18)│ output_partition_count: │
18)│ input_partition_count: │
19)│ 1 │
20)│ │
21)│ partitioning_scheme: │
Expand Down Expand Up @@ -599,7 +599,7 @@ physical_plan
13)┌─────────────┴─────────────┐
14)│ RepartitionExec │
15)│ -------------------- │
16)│ output_partition_count: │
16)│ input_partition_count: │
17)│ 1 │
18)│ │
19)│ partitioning_scheme: │
Expand Down Expand Up @@ -633,7 +633,7 @@ physical_plan
13)┌─────────────┴─────────────┐
14)│ RepartitionExec │
15)│ -------------------- │
16)│ output_partition_count: │
16)│ input_partition_count: │
17)│ 1 │
18)│ │
19)│ partitioning_scheme: │
Expand Down Expand Up @@ -694,7 +694,7 @@ physical_plan
13)┌─────────────┴─────────────┐
14)│ RepartitionExec │
15)│ -------------------- │
16)│ output_partition_count: │
16)│ input_partition_count: │
17)│ 1 │
18)│ │
19)│ partitioning_scheme: │
Expand Down Expand Up @@ -727,7 +727,7 @@ physical_plan
13)┌─────────────┴─────────────┐
14)│ RepartitionExec │
15)│ -------------------- │
16)│ output_partition_count: │
16)│ input_partition_count: │
17)│ 1 │
18)│ │
19)│ partitioning_scheme: │
Expand Down Expand Up @@ -922,7 +922,7 @@ physical_plan
13)┌─────────────┴─────────────┐
14)│ RepartitionExec │
15)│ -------------------- │
16)│ output_partition_count: │
16)│ input_partition_count: │
17)│ 1 │
18)│ │
19)│ partitioning_scheme: │
Expand Down Expand Up @@ -1031,7 +1031,7 @@ physical_plan
13)┌─────────────┴─────────────┐
14)│ RepartitionExec │
15)│ -------------------- │
16)│ output_partition_count: │
16)│ input_partition_count: │
17)│ 1 │
18)│ │
19)│ partitioning_scheme: │
Expand Down Expand Up @@ -1089,7 +1089,7 @@ physical_plan
12)┌─────────────┴─────────────┐
13)│ RepartitionExec │
14)│ -------------------- │
15)│ output_partition_count: │
15)│ input_partition_count: │
16)│ 1 │
17)│ │
18)│ partitioning_scheme: │
Expand Down Expand Up @@ -1123,7 +1123,7 @@ physical_plan
13)┌─────────────┴─────────────┐
14)│ RepartitionExec │
15)│ -------------------- │
16)│ output_partition_count: │
16)│ input_partition_count: │
17)│ 1 │
18)│ │
19)│ partitioning_scheme: │
Expand Down Expand Up @@ -1209,7 +1209,7 @@ physical_plan
22)┌─────────────┴─────────────┐┌─────────────┴─────────────┐
23)│ RepartitionExec ││ RepartitionExec │
24)│ -------------------- ││ -------------------- │
25)│ output_partition_count: ││ output_partition_count: │
25)│ input_partition_count: ││ input_partition_count: │
26)│ 4 ││ 4 │
27)│ ││ │
28)│ partitioning_scheme: ││ partitioning_scheme: │
Expand All @@ -1220,7 +1220,7 @@ physical_plan
33)┌─────────────┴─────────────┐┌─────────────┴─────────────┐
34)│ ProjectionExec ││ RepartitionExec │
35)│ -------------------- ││ -------------------- │
36)│ CAST(table1.string_col AS ││ output_partition_count: │
36)│ CAST(table1.string_col AS ││ input_partition_count: │
37)│ Utf8View): ││ 1 │
38)│ CAST(string_col AS ││ │
39)│ Utf8View) ││ partitioning_scheme: │
Expand All @@ -1237,7 +1237,7 @@ physical_plan
50)┌─────────────┴─────────────┐┌─────────────┴─────────────┐
51)│ RepartitionExec ││ DataSourceExec │
52)│ -------------------- ││ -------------------- │
53)│ output_partition_count: ││ files: 1 │
53)│ input_partition_count: ││ files: 1 │
54)│ 1 ││ format: parquet │
55)│ ││ │
56)│ partitioning_scheme: ││ │
Expand Down Expand Up @@ -1281,7 +1281,7 @@ physical_plan
24)┌─────────────┴─────────────┐┌─────────────┴─────────────┐
25)│ RepartitionExec ││ RepartitionExec │
26)│ -------------------- ││ -------------------- │
27)│ output_partition_count: ││ output_partition_count: │
27)│ input_partition_count: ││ input_partition_count: │
28)│ 4 ││ 4 │
29)│ ││ │
30)│ partitioning_scheme: ││ partitioning_scheme: │
Expand All @@ -1292,7 +1292,7 @@ physical_plan
35)┌─────────────┴─────────────┐┌─────────────┴─────────────┐
36)│ ProjectionExec ││ RepartitionExec │
37)│ -------------------- ││ -------------------- │
38)│ CAST(table1.string_col AS ││ output_partition_count: │
38)│ CAST(table1.string_col AS ││ input_partition_count: │
39)│ Utf8View): ││ 1 │
40)│ CAST(string_col AS ││ │
41)│ Utf8View) ││ partitioning_scheme: │
Expand All @@ -1309,7 +1309,7 @@ physical_plan
52)┌─────────────┴─────────────┐┌─────────────┴─────────────┐
53)│ RepartitionExec ││ DataSourceExec │
54)│ -------------------- ││ -------------------- │
55)│ output_partition_count: ││ files: 1 │
55)│ input_partition_count: ││ files: 1 │
56)│ 1 ││ format: parquet │
57)│ ││ │
58)│ partitioning_scheme: ││ │
Expand Down Expand Up @@ -1356,7 +1356,7 @@ physical_plan
27)-----------------------------┌─────────────┴─────────────┐
28)-----------------------------│ RepartitionExec │
29)-----------------------------│ -------------------- │
30)-----------------------------│ output_partition_count: │
30)-----------------------------│ input_partition_count: │
31)-----------------------------│ 1 │
32)-----------------------------│ │
33)-----------------------------│ partitioning_scheme: │
Expand All @@ -1380,7 +1380,7 @@ physical_plan
04)┌─────────────┴─────────────┐┌─────────────┴─────────────┐
05)│ DataSourceExec ││ RepartitionExec │
06)│ -------------------- ││ -------------------- │
07)│ files: 1 ││ output_partition_count: │
07)│ files: 1 ││ input_partition_count: │
08)│ format: csv ││ 1 │
09)│ ││ │
10)│ ││ partitioning_scheme: │
Expand Down Expand Up @@ -1505,7 +1505,7 @@ physical_plan
33)┌─────────────┴─────────────┐┌─────────────┴─────────────┐
34)│ RepartitionExec ││ RepartitionExec │
35)│ -------------------- ││ -------------------- │
36)│ output_partition_count: ││ output_partition_count: │
36)│ input_partition_count: ││ input_partition_count: │
37)│ 4 ││ 4 │
38)│ ││ │
39)│ partitioning_scheme: ││ partitioning_scheme: │
Expand All @@ -1514,7 +1514,7 @@ physical_plan
42)┌─────────────┴─────────────┐┌─────────────┴─────────────┐
43)│ RepartitionExec ││ RepartitionExec │
44)│ -------------------- ││ -------------------- │
45)│ output_partition_count: ││ output_partition_count: │
45)│ input_partition_count: ││ input_partition_count: │
46)│ 1 ││ 1 │
47)│ ││ │
48)│ partitioning_scheme: ││ partitioning_scheme: │
Expand Down Expand Up @@ -1606,7 +1606,7 @@ physical_plan
18)┌─────────────┴─────────────┐
19)│ RepartitionExec │
20)│ -------------------- │
21)│ output_partition_count: │
21)│ input_partition_count: │
22)│ 1 │
23)│ │
24)│ partitioning_scheme: │
Expand Down Expand Up @@ -1648,7 +1648,7 @@ physical_plan
19)┌─────────────┴─────────────┐
20)│ RepartitionExec │
21)│ -------------------- │
22)│ output_partition_count: │
22)│ input_partition_count: │
23)│ 1 │
24)│ │
25)│ partitioning_scheme: │
Expand Down Expand Up @@ -1689,7 +1689,7 @@ physical_plan
19)┌─────────────┴─────────────┐
20)│ RepartitionExec │
21)│ -------------------- │
22)│ output_partition_count: │
22)│ input_partition_count: │
23)│ 1 │
24)│ │
25)│ partitioning_scheme: │
Expand Down Expand Up @@ -1728,7 +1728,7 @@ physical_plan
17)┌─────────────┴─────────────┐
18)│ RepartitionExec │
19)│ -------------------- │
20)│ output_partition_count: │
20)│ input_partition_count: │
21)│ 1 │
22)│ │
23)│ partitioning_scheme: │
Expand Down Expand Up @@ -1771,7 +1771,7 @@ physical_plan
20)┌─────────────┴─────────────┐
21)│ RepartitionExec │
22)│ -------------------- │
23)│ output_partition_count: │
23)│ input_partition_count: │
24)│ 1 │
25)│ │
26)│ partitioning_scheme: │
Expand Down Expand Up @@ -1815,7 +1815,7 @@ physical_plan
19)┌─────────────┴─────────────┐
20)│ RepartitionExec │
21)│ -------------------- │
22)│ output_partition_count: │
22)│ input_partition_count: │
23)│ 1 │
24)│ │
25)│ partitioning_scheme: │
Expand Down Expand Up @@ -1869,7 +1869,7 @@ physical_plan
25)-----------------------------┌─────────────┴─────────────┐
26)-----------------------------│ RepartitionExec │
27)-----------------------------│ -------------------- │
28)-----------------------------│ output_partition_count: │
28)-----------------------------│ input_partition_count: │
29)-----------------------------│ 1 │
30)-----------------------------│ │
31)-----------------------------│ partitioning_scheme: │
Expand Down Expand Up @@ -1983,7 +1983,7 @@ physical_plan
22)┌─────────────┴─────────────┐
23)│ RepartitionExec │
24)│ -------------------- │
25)│ output_partition_count: │
25)│ input_partition_count: │
26)│ 1 │
27)│ │
28)│ partitioning_scheme: │
Expand Down Expand Up @@ -2062,7 +2062,7 @@ physical_plan
19)┌─────────────┴─────────────┐
20)│ RepartitionExec │
21)│ -------------------- │
22)│ output_partition_count: │
22)│ input_partition_count: │
23)│ 1 │
24)│ │
25)│ partitioning_scheme: │
Expand Down