[HUDI-9410] Moved the code from hudi-spark3-common to hudi-spark-common module#13301
Conversation
|
@yihua Ethan, review please and merge if it's ok. and i'll continue fixing spark4 support PR. |
| @@ -15,7 +15,7 @@ | |||
| * limitations under the License. | |||
| */ | |||
|
|
|||
| package org.apache.spark.sql.catalyst.plans.logcal | |||
There was a problem hiding this comment.
Finally, someone submitted a fix for this. Thank you!
There was a problem hiding this comment.
nit: should these classes be renamed to *Spark* from *Spark3*?
There was a problem hiding this comment.
I thought about it too, and even tried to do so, but found that some classes already have such "brothers" (classes with the same names but without Spark version). So i decided to not rename anything. But i'll revisit this one more time in Spark 4 support PR.
yihua
left a comment
yihua
left a comment
There was a problem hiding this comment.
LGTM. Thanks for the refactoring!
| */ | ||
|
|
||
| package org.apache.hudi.spark3.internal; | ||
| package org.apache.hudi.spark.internal; |
There was a problem hiding this comment.
We should revisit in a separate JIRA and PR to see if we can move these bulk insert tests to hudi-spark to avoid test code duplication across version-specific modules.
There was a problem hiding this comment.
…on module (apache#13301) Co-authored-by: Vova Kolmakov <kolmakov.vladimir@huawei.com> (cherry picked from commit b5a25f7)
…on module (apache#13301) Co-authored-by: Vova Kolmakov <kolmakov.vladimir@huawei.com>
4 participants
Change Logs
Right now there are many duplicated classes which will make the maintenance Hudi Spark integration much harder.
As Spark 2 is not supported anymore, it's reasonable to merge the spark3-common code into spark-common.
So i:
hudi-spark3-commontohudi-spark-commonmodulehudi-spark3-commonmoduleIt's a prerequisite PR to make Spark 4 support easier (#12772) and avoid code duplication in the future.
Addressing these comments:
#12772 (comment)
#12772 (comment)
Impact
reduced code duplication in spark-x modules
Risk level (write none, low medium or high below)
none
Documentation Update
none
ticket number here and follow the instruction to make
changes to the website.
Contributor's checklist