Commit 9f603fc
[SPARK-1955][GraphX]: VertexRDD can incorrectly assume index sharing
Fixes the issue whereby when VertexRDD's are `diff`ed, `innerJoin`ed, or `leftJoin`ed and have different partition sizes they fail under the `zipPartitions` method. This fix tests whether the partitions are equal or not and, if not, will repartition the other to match the partition size of the calling VertexRDD.
Author: Brennon York <[email protected]>
Closes #4705 from brennonyork/SPARK-1955 and squashes the following commits:
0882590 [Brennon York] updated to properly handle differently-partitioned vertexRDDs1 parent a777c65 commit 9f603fc
File tree
1 file changed
+9
-3
lines changed- graphx/src/main/scala/org/apache/spark/graphx/impl
1 file changed
+9
-3
lines changedLines changed: 9 additions & 3 deletions
| Original file line number | Diff line number | Diff line change | |
|---|---|---|---|
| |||
104 | 104 | | |
105 | 105 | | |
106 | 106 | | |
| 107 | + | |
| 108 | + | |
| 109 | + | |
| 110 | + | |
| 111 | + | |
| 112 | + | |
107 | 113 | | |
108 | | - | |
| 114 | + | |
109 | 115 | | |
110 | 116 | | |
111 | 117 | | |
| |||
133 | 139 | | |
134 | 140 | | |
135 | 141 | | |
136 | | - | |
| 142 | + | |
137 | 143 | | |
138 | 144 | | |
139 | 145 | | |
| |||
162 | 168 | | |
163 | 169 | | |
164 | 170 | | |
165 | | - | |
| 171 | + | |
166 | 172 | | |
167 | 173 | | |
168 | 174 | | |
| |||
0 commit comments