Skip to content

Commit a842622

Browse files
dongjoon-hyunsweisdb
authored andcommitted
[SPARK-44115][BUILD] Upgrade Apache ORC to 2.0.0
### What changes were proposed in this pull request? This PR aims to Upgrade Apache ORC to 2.0.0 for Apache Spark 4.0.0. Apache ORC community has 3-year support policy which is longer than Apache Spark. It's aligned like the following. - Apache ORC 2.0.x <-> Apache Spark 4.0.x - Apache ORC 1.9.x <-> Apache Spark 3.5.x - Apache ORC 1.8.x <-> Apache Spark 3.4.x - Apache ORC 1.7.x (Supported) <-> Apache Spark 3.3.x (End-Of-Support) ### Why are the changes needed? **Release Note** - https://github.com/apache/orc/releases/tag/v2.0.0 **Milestone** - https://github.com/apache/orc/milestone/20?closed=1 - apache/orc#1728 - apache/orc#1801 - apache/orc#1498 - apache/orc#1627 - apache/orc#1497 - apache/orc#1509 - apache/orc#1554 - apache/orc#1708 - apache/orc#1733 - apache/orc#1760 - apache/orc#1743 ### Does this PR introduce _any_ user-facing change? No. ### How was this patch tested? Pass the CIs. ### Was this patch authored or co-authored using generative AI tooling? No. Closes apache#45443 from dongjoon-hyun/SPARK-44115. Authored-by: Dongjoon Hyun <[email protected]> Signed-off-by: Dongjoon Hyun <[email protected]>
1 parent c5133f5 commit a842622

File tree

3 files changed

+25
-4
lines changed

3 files changed

+25
-4
lines changed

dev/deps/spark-deps-hadoop-3-hive-2.3

+4-3
Original file line numberDiff line numberDiff line change
@@ -227,9 +227,10 @@ opencsv/2.3//opencsv-2.3.jar
227227
opentracing-api/0.33.0//opentracing-api-0.33.0.jar
228228
opentracing-noop/0.33.0//opentracing-noop-0.33.0.jar
229229
opentracing-util/0.33.0//opentracing-util-0.33.0.jar
230-
orc-core/1.9.2/shaded-protobuf/orc-core-1.9.2-shaded-protobuf.jar
231-
orc-mapreduce/1.9.2/shaded-protobuf/orc-mapreduce-1.9.2-shaded-protobuf.jar
232-
orc-shims/1.9.2//orc-shims-1.9.2.jar
230+
orc-core/2.0.0/shaded-protobuf/orc-core-2.0.0-shaded-protobuf.jar
231+
orc-format/1.0.0/shaded-protobuf/orc-format-1.0.0-shaded-protobuf.jar
232+
orc-mapreduce/2.0.0/shaded-protobuf/orc-mapreduce-2.0.0-shaded-protobuf.jar
233+
orc-shims/2.0.0//orc-shims-2.0.0.jar
233234
oro/2.0.8//oro-2.0.8.jar
234235
osgi-resource-locator/1.0.3//osgi-resource-locator-1.0.3.jar
235236
paranamer/2.8//paranamer-2.8.jar

pom.xml

+16-1
Original file line numberDiff line numberDiff line change
@@ -141,7 +141,7 @@
141141
<!-- After 10.17.1.0, the minimum required version is JDK19 -->
142142
<derby.version>10.16.1.1</derby.version>
143143
<parquet.version>1.13.1</parquet.version>
144-
<orc.version>1.9.2</orc.version>
144+
<orc.version>2.0.0</orc.version>
145145
<orc.classifier>shaded-protobuf</orc.classifier>
146146
<jetty.version>11.0.20</jetty.version>
147147
<jakartaservlet.version>5.0.0</jakartaservlet.version>
@@ -2593,13 +2593,28 @@
25932593
</exclusions>
25942594
</dependency>
25952595

2596+
<dependency>
2597+
<groupId>org.apache.orc</groupId>
2598+
<artifactId>orc-format</artifactId>
2599+
<version>1.0.0</version>
2600+
<classifier>${orc.classifier}</classifier>
2601+
<scope>${orc.deps.scope}</scope>
2602+
</dependency>
25962603
<dependency>
25972604
<groupId>org.apache.orc</groupId>
25982605
<artifactId>orc-core</artifactId>
25992606
<version>${orc.version}</version>
26002607
<classifier>${orc.classifier}</classifier>
26012608
<scope>${orc.deps.scope}</scope>
26022609
<exclusions>
2610+
<exclusion>
2611+
<groupId>org.apache.orc</groupId>
2612+
<artifactId>orc-format</artifactId>
2613+
</exclusion>
2614+
<exclusion>
2615+
<groupId>com.aayushatharva.brotli4j</groupId>
2616+
<artifactId>brotli4j</artifactId>
2617+
</exclusion>
26032618
<exclusion>
26042619
<groupId>org.apache.hadoop</groupId>
26052620
<artifactId>hadoop-common</artifactId>

sql/core/pom.xml

+5
Original file line numberDiff line numberDiff line change
@@ -93,6 +93,11 @@
9393
<groupId>org.scala-lang.modules</groupId>
9494
<artifactId>scala-parallel-collections_${scala.binary.version}</artifactId>
9595
</dependency>
96+
<dependency>
97+
<groupId>org.apache.orc</groupId>
98+
<artifactId>orc-format</artifactId>
99+
<classifier>${orc.classifier}</classifier>
100+
</dependency>
96101
<dependency>
97102
<groupId>org.apache.orc</groupId>
98103
<artifactId>orc-core</artifactId>

0 commit comments

Comments
 (0)