Skip to content

[SPARK-39948][BUILD] Exclude hive-vector-code-gen dependency #38978

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Closed
zhouyifan279 wants to merge 5 commits into from
Closed

[SPARK-39948][BUILD] Exclude hive-vector-code-gen dependency #38978

Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
5 commits
Select commit Hold shift + click to select a range
3b34e74
[SPARK-39948] Exclude hive-vector-code-gen dependency to solve CVE-20…
zhouyifan279 Dec 8, 2022
f609898
trigger-test
zhouyifan279 Dec 8, 2022
54fd846
[SPARK-39948] Exclude hive-vector-code-gen dependency to solve CVE-20…
zhouyifan279 Dec 8, 2022
d862020
trigger-test
zhouyifan279 Dec 8, 2022
359607e
trigger-test
zhouyifan279 Dec 8, 2022
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
2 changes: 0 additions & 2 deletions dev/deps/spark-deps-hadoop-2-hive-2.3
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -101,7 +101,6 @@ hive-shims-common/2.3.9//hive-shims-common-2.3.9.jar
hive-shims-scheduler/2.3.9//hive-shims-scheduler-2.3.9.jar
hive-shims/2.3.9//hive-shims-2.3.9.jar
hive-storage-api/2.7.3//hive-storage-api-2.7.3.jar
hive-vector-code-gen/2.3.9//hive-vector-code-gen-2.3.9.jar
hk2-api/2.6.1//hk2-api-2.6.1.jar
hk2-locator/2.6.1//hk2-locator-2.6.1.jar
hk2-utils/2.6.1//hk2-utils-2.6.1.jar
Expand Down Expand Up @@ -261,7 +260,6 @@ threeten-extra/1.7.1//threeten-extra-1.7.1.jar
tink/1.7.0//tink-1.7.0.jar
transaction-api/1.1//transaction-api-1.1.jar
univocity-parsers/2.9.1//univocity-parsers-2.9.1.jar
velocity/1.5//velocity-1.5.jar
xbean-asm9-shaded/4.22//xbean-asm9-shaded-4.22.jar
xercesImpl/2.12.2//xercesImpl-2.12.2.jar
xml-apis/1.4.01//xml-apis-1.4.01.jar
Expand Down
2 changes: 0 additions & 2 deletions dev/deps/spark-deps-hadoop-3-hive-2.3
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -89,7 +89,6 @@ hive-shims-common/2.3.9//hive-shims-common-2.3.9.jar
hive-shims-scheduler/2.3.9//hive-shims-scheduler-2.3.9.jar
hive-shims/2.3.9//hive-shims-2.3.9.jar
hive-storage-api/2.7.3//hive-storage-api-2.7.3.jar
hive-vector-code-gen/2.3.9//hive-vector-code-gen-2.3.9.jar
hk2-api/2.6.1//hk2-api-2.6.1.jar
hk2-locator/2.6.1//hk2-locator-2.6.1.jar
hk2-utils/2.6.1//hk2-utils-2.6.1.jar
Expand Down Expand Up @@ -248,7 +247,6 @@ threeten-extra/1.7.1//threeten-extra-1.7.1.jar
tink/1.7.0//tink-1.7.0.jar
transaction-api/1.1//transaction-api-1.1.jar
univocity-parsers/2.9.1//univocity-parsers-2.9.1.jar
velocity/1.5//velocity-1.5.jar
wildfly-openssl/1.0.7.Final//wildfly-openssl-1.0.7.Final.jar
xbean-asm9-shaded/4.22//xbean-asm9-shaded-4.22.jar
xz/1.9//xz-1.9.jar
Expand Down
4 changes: 4 additions & 0 deletions pom.xml
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -2042,6 +2042,10 @@
<groupId>${hive.group}</groupId>
<artifactId>hive-ant</artifactId>
</exclusion>
<exclusion>
<groupId>${hive.group}</groupId>
<artifactId>hive-vector-code-gen</artifactId>
</exclusion>
<!-- break the loop -->
<exclusion>
<groupId>${hive.group}</groupId>
Expand Down
9 changes: 6 additions & 3 deletions sql/hive/src/main/scala/org/apache/spark/sql/hive/client/package.scala
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -108,7 +108,8 @@ package object client {
"org.apache.calcite.avatica:avatica",
"com.fasterxml.jackson.core:*",
"org.apache.curator:*",
"org.pentaho:pentaho-aggdesigner-algorithm"))
"org.pentaho:pentaho-aggdesigner-algorithm",
"org.apache.hive:hive-vector-code-gen"))

// Since Hive 3.0, HookUtils uses org.apache.logging.log4j.util.Strings
// Since HIVE-14496, Hive.java uses calcite-core
Expand All @@ -117,7 +118,8 @@ package object client {
"org.apache.derby:derby:10.14.1.0"),
exclusions = Seq("org.apache.calcite:calcite-druid",
"org.apache.curator:*",
"org.pentaho:pentaho-aggdesigner-algorithm"))
"org.pentaho:pentaho-aggdesigner-algorithm",
"org.apache.hive:hive-vector-code-gen"))
Copy link
Member

@sunchao sunchao Dec 8, 2022

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

do other Hive versions like v2_2 have the same issue?

Copy link
Member

@dongjoon-hyun dongjoon-hyun Dec 8, 2022

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

@zhouyifan279 Could you answer @sunchao 's question?

Copy link
Contributor Author

@zhouyifan279 zhouyifan279 Dec 9, 2022

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

@sunchao According to this commit in Hive Project, hive-vector-code-gen is added as hive-exec dependency since hive-2.3.0.

Copy link
Member

@dongjoon-hyun dongjoon-hyun Dec 9, 2022

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Thank you for confirming, @zhouyifan279 .


// Since Hive 3.0, HookUtils uses org.apache.logging.log4j.util.Strings
// Since HIVE-14496, Hive.java uses calcite-core
Expand All @@ -126,7 +128,8 @@ package object client {
"org.apache.derby:derby:10.14.1.0"),
exclusions = Seq("org.apache.calcite:calcite-druid",
"org.apache.curator:*",
"org.pentaho:pentaho-aggdesigner-algorithm"))
"org.pentaho:pentaho-aggdesigner-algorithm",
"org.apache.hive:hive-vector-code-gen"))

val allSupportedHiveVersions =
Set(v12, v13, v14, v1_0, v1_1, v1_2, v2_0, v2_1, v2_2, v2_3, v3_0, v3_1)
Expand Down