Skip to content

Conversation

@attilapiros
Copy link
Contributor

In https://issues.apache.org/jira/browse/HBASE-28137 to support Spark 3.4 the scala-parser-combinators was added as direct dependency to HBase Spark Connector. This was needed as Spark 3.4 is not using scala-parser-combinators and it is not inherited as transitive dependency.

But this solution has a disadvantage. As the HBase Spark Connector assembly jar does not include any 3rd party libraries the scala-parser-combinators must be added to the spark classpath for HBase Spark Connector to work. A much better solution is to replace scala.util.parsing.json with org.json4s.jackson which is used by Spark core, see https://github.com/apache/spark/blob/branch-3.4/core/pom.xml#L279-L280.

In https://issues.apache.org/jira/browse/HBASE-28137 to support Spark 3.4 the scala-parser-combinators was added as direct dependency to HBase Spark Connector.
This was needed as Spark 3.4 is not using scala-parser-combinators and it is not inherited as transitive dependency.

But this solution has a disadvantage. As the HBase Spark Connector assembly jar does not include any 3rd party libraries the scala-parser-combinators must be added to the spark classpath for HBase Spark Connector to work.
A much better solution is to replace scala.util.parsing.json with org.json4s.jackson which is used by Spark core, see https://github.com/apache/spark/blob/branch-3.4/core/pom.xml#L279-L280.
@Apache-HBase
Copy link

💔 -1 overall

Vote Subsystem Runtime Comment
+0 🆗 reexec 1m 2s Docker mode activated.
_ Prechecks _
+1 💚 dupname 0m 0s No case conflicting files found.
+1 💚 @author 0m 0s The patch does not contain any @author tags.
-0 ⚠️ test4tests 0m 0s The patch doesn't appear to include any new or modified tests. Please justify why no new tests are needed for this patch. Also please list what manual steps were performed to verify this patch.
_ master Compile Tests _
+0 🆗 mvndep 0m 21s Maven dependency ordering for branch
+1 💚 mvninstall 1m 25s master passed
+1 💚 compile 1m 19s master passed
+1 💚 spotless 0m 14s branch has no errors when running spotless:check.
+1 💚 javadoc 1m 17s master passed
+1 💚 scaladoc 0m 46s master passed
_ Patch Compile Tests _
+0 🆗 mvndep 0m 3s Maven dependency ordering for patch
+1 💚 mvninstall 0m 45s the patch passed
+1 💚 compile 1m 18s the patch passed
+1 💚 javac 1m 18s the patch passed
+1 💚 scalac 1m 18s the patch passed
+1 💚 whitespace 0m 0s The patch has no whitespace issues.
+1 💚 xml 0m 2s The patch has no ill-formed XML file.
-1 ❌ spotless 0m 10s patch has 22 errors when running spotless:check, run spotless:apply to fix.
+1 💚 javadoc 1m 17s the patch passed
+1 💚 scaladoc 0m 45s the patch passed
_ Other Tests _
+1 💚 unit 7m 52s spark in the patch passed.
+1 💚 unit 7m 57s hbase-spark in the patch passed.
27m 54s
Subsystem Report/Notes
Docker ClientAPI=1.43 ServerAPI=1.43 base: https://ci-hbase.apache.org/job/HBase-Connectors-PreCommit/job/PR-126/1/artifact/yetus-precommit-check/output/Dockerfile
GITHUB PR #126
Optional Tests dupname javac javadoc unit spotless xml compile scalac scaladoc
uname Linux b203a8db83b2 5.4.0-1103-aws #111~18.04.1-Ubuntu SMP Tue May 23 20:04:10 UTC 2023 x86_64 GNU/Linux
Build tool hb_maven
Personality dev-support/jenkins/hbase-personality.sh
git revision master / 1fb67c9
Default Java Oracle Corporation-1.8.0_282-b08
spotless https://ci-hbase.apache.org/job/HBase-Connectors-PreCommit/job/PR-126/1/artifact/yetus-precommit-check/output/patch-spotless.txt
Test Results https://ci-hbase.apache.org/job/HBase-Connectors-PreCommit/job/PR-126/1/testReport/
Max. process+thread count 944 (vs. ulimit of 12500)
modules C: spark spark/hbase-spark U: spark
Console output https://ci-hbase.apache.org/job/HBase-Connectors-PreCommit/job/PR-126/1/console
versions git=2.20.1
Powered by Apache Yetus 0.12.0 https://yetus.apache.org

This message was automatically generated.

@Apache-HBase
Copy link

🎊 +1 overall

Vote Subsystem Runtime Comment
+0 🆗 reexec 0m 29s Docker mode activated.
_ Prechecks _
+1 💚 dupname 0m 0s No case conflicting files found.
+1 💚 @author 0m 0s The patch does not contain any @author tags.
-0 ⚠️ test4tests 0m 0s The patch doesn't appear to include any new or modified tests. Please justify why no new tests are needed for this patch. Also please list what manual steps were performed to verify this patch.
_ master Compile Tests _
+0 🆗 mvndep 0m 20s Maven dependency ordering for branch
+1 💚 mvninstall 1m 25s master passed
+1 💚 compile 1m 19s master passed
+1 💚 spotless 0m 13s branch has no errors when running spotless:check.
+1 💚 javadoc 1m 18s master passed
+1 💚 scaladoc 0m 47s master passed
_ Patch Compile Tests _
+0 🆗 mvndep 0m 4s Maven dependency ordering for patch
+1 💚 mvninstall 0m 44s the patch passed
+1 💚 compile 1m 19s the patch passed
+1 💚 javac 1m 19s the patch passed
+1 💚 scalac 1m 19s the patch passed
+1 💚 whitespace 0m 0s The patch has no whitespace issues.
+1 💚 xml 0m 1s The patch has no ill-formed XML file.
+1 💚 spotless 0m 10s patch has no errors when running spotless:check.
+1 💚 javadoc 1m 18s the patch passed
+1 💚 scaladoc 0m 46s the patch passed
_ Other Tests _
+1 💚 unit 8m 0s spark in the patch passed.
+1 💚 unit 7m 48s hbase-spark in the patch passed.
27m 24s
Subsystem Report/Notes
Docker ClientAPI=1.43 ServerAPI=1.43 base: https://ci-hbase.apache.org/job/HBase-Connectors-PreCommit/job/PR-126/2/artifact/yetus-precommit-check/output/Dockerfile
GITHUB PR #126
Optional Tests dupname javac javadoc unit spotless xml compile scalac scaladoc
uname Linux b92f5b6f81e0 5.4.0-1103-aws #111~18.04.1-Ubuntu SMP Tue May 23 20:04:10 UTC 2023 x86_64 GNU/Linux
Build tool hb_maven
Personality dev-support/jenkins/hbase-personality.sh
git revision master / 1fb67c9
Default Java Oracle Corporation-1.8.0_282-b08
Test Results https://ci-hbase.apache.org/job/HBase-Connectors-PreCommit/job/PR-126/2/testReport/
Max. process+thread count 943 (vs. ulimit of 12500)
modules C: spark spark/hbase-spark U: spark
Console output https://ci-hbase.apache.org/job/HBase-Connectors-PreCommit/job/PR-126/2/console
versions git=2.20.1
Powered by Apache Yetus 0.12.0 https://yetus.apache.org

This message was automatically generated.

@meszibalu meszibalu merged commit e628718 into apache:master Oct 16, 2023
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

3 participants