HBASE-25326 Allow running and building hbase-connectors with Apache Spark 3.0 #75
Conversation
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
|
💔 -1 overall
This message was automatically generated. |
|
💔 -1 overall
This message was automatically generated. |
saintstack
reviewed
Nov 24, 2020
Contributor
saintstack
left a comment
saintstack
left a comment
There was a problem hiding this comment.
LGTM
Spark2 still works?
Thanks for working on this.
Contributor
|
The failing seems to be because of this... |
Contributor
Author
|
Thanks @saintstack for looking at this. |
LucaCanali
force-pushed
the
compileWithSpark3
branch
from
76d7c48 to
31e8f9c
Compare
|
💔 -1 overall
This message was automatically generated. |
LucaCanali
force-pushed
the
compileWithSpark3
branch
from
31e8f9c to
b1d2458
Compare
|
💔 -1 overall
This message was automatically generated. |
|
🎊 +1 overall
This message was automatically generated. |
Contributor
Author
Yes, Spark 2 still works. BTW, it is worth mentioning that there will have to be separate releases of the spark connector for each supported Scala version (notably for Scala 2.11 and 2.12) and that Spark 3.0 only supports Scala 2.12. |
symat
approved these changes
Jan 28, 2021
symat
left a comment
symat
left a comment
There was a problem hiding this comment.
LGTM, thanks for the patch @LucaCanali !
I just found this patch after I got compilation errors when trying to build hbase-connectors with Spark 3.2 and Scala 2.12. Applying this patch, now the following command works for me:
mvn clean install -Dhadoop.profile=3.0 -Dhadoop-three.version=3.2.0 -Dspark.version=3.1.0 -Dscala.version=2.12.10 -Dscala.binary.version=2.12
also I verified the patch with spark 2, using:
mvn clean install -Dhadoop.profile=3.0 -Dhadoop-three.version=3.2.0 -Dspark.version=2.4.5 -Dscala.version=2.11.12 -Dscala.binary.version=2.11
Compilation and unit tests passed for both commands above. CI also seems to be green now, as far as I understood.
@saintstack what do you think?
@meszibalu can you take a look?
(I haven't actually tested the hbase connector with spark 3.1 just yet, but that will be my next step. Still, this patch seems to me a valuable contribution in its current form.)
|
I just tested the hbase-connectors built with this patch on a cluster. I was able to use both Spark2 and Spark3 with the following setup:
|
Member
|
Seems fine to me, admittedly I'm not a spark/scala expert in any sense :) Glancing at the Yetus personality, it doesn't look like we have anything to test both Spark2 and Spark3 during precommit. Should we also add that? |
saintstack
approved these changes
Feb 1, 2021
Contributor
|
LGTM. Leaving open while Josh's question is outstanding. |
Member
I'm OK to track this as a follow-on. I trust that Maté and the author have done adequate testing already. |
saintstack
changed the title
Contributor
|
Merged it then @joshelser ... We can do build infra for s2 vs s3 in follow-on as you suggest. |
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
6 participants
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
Currently hbase-spark connector only works with Spark 2.x. Apache Spark 3.0 has been relesead in June 2020.
This addresses the changes needed to run the connector with Spark 3.0 and also to be able to compile the connector using Spark 3.0 as a dependency.
This has been manually tested with Apache Spark 3.0.1 and HBase 2.2.4.