Skip to content

[SPARK-20202][BUILD][SQL] Remove references to org.spark-project.hive (Hive 1.2.1) #29936

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Closed
dongjoon-hyun wants to merge 2 commits into from
Closed

[SPARK-20202][BUILD][SQL] Remove references to org.spark-project.hive (Hive 1.2.1) #29936

Show file tree
Hide file tree
Changes from 1 commit
Commits
Show all changes
2 commits
Select commit Hold shift + click to select a range
16b3452
Remove Hive 1.2
dongjoon-hyun Oct 2, 2020
c855260
Remove hive-1.2 profile
dongjoon-hyun Oct 5, 2020
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
The table of contents is too big for display.
Diff view
Diff view
  •  
  •  
  •  
1 change: 0 additions & 1 deletion dev/run-tests.py
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -325,7 +325,6 @@ def get_hive_profiles(hive_version):
"""

sbt_maven_hive_profiles = {
"hive1.2": ["-Phive-1.2"],
"hive2.3": ["-Phive-2.3"],
}

Expand Down
6 changes: 1 addition & 5 deletions dev/test-dependencies.sh
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -32,7 +32,6 @@ export LC_ALL=C
HADOOP_MODULE_PROFILES="-Phive-thriftserver -Pmesos -Pkubernetes -Pyarn -Phive"
MVN="build/mvn"
HADOOP_HIVE_PROFILES=(
hadoop-2.7-hive-1.2
hadoop-2.7-hive-2.3
hadoop-3.2-hive-2.3
)
Expand Down Expand Up @@ -71,12 +70,9 @@ for HADOOP_HIVE_PROFILE in "${HADOOP_HIVE_PROFILES[@]}"; do
if [[ $HADOOP_HIVE_PROFILE == **hadoop-3.2-hive-2.3** ]]; then
HADOOP_PROFILE=hadoop-3.2
HIVE_PROFILE=hive-2.3
elif [[ $HADOOP_HIVE_PROFILE == **hadoop-2.7-hive-2.3** ]]; then
HADOOP_PROFILE=hadoop-2.7
HIVE_PROFILE=hive-2.3
else
HADOOP_PROFILE=hadoop-2.7
HIVE_PROFILE=hive-1.2
HIVE_PROFILE=hive-2.3
fi
echo "Performing Maven install for $HADOOP_HIVE_PROFILE"
$MVN $HADOOP_MODULE_PROFILES -P$HADOOP_PROFILE -P$HIVE_PROFILE jar:jar jar:test-jar install:install clean -q
Expand Down
2 changes: 2 additions & 0 deletions docs/sql-migration-guide.md
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -42,6 +42,8 @@ license: |

- In Spark 3.1, incomplete interval literals, e.g. `INTERVAL '1'`, `INTERVAL '1 DAY 2'` will fail with IllegalArgumentException. In Spark 3.0, they result `NULL`s.

- In Spark 3.1, we remove the built-in Hive 1.2. You need to migrate your custom SerDes to Hive 2.3. See [HIVE-15167](https://issues.apache.org/jira/browse/HIVE-15167) for more details.

## Upgrading from Spark SQL 3.0 to 3.0.1

- In Spark 3.0, JSON datasource and JSON function `schema_of_json` infer TimestampType from string values if they match to the pattern defined by the JSON option `timestampFormat`. Since version 3.0.1, the timestamp type inference is disabled by default. Set the JSON option `inferTimestamp` to `true` to enable such type inference.
Expand Down
22 changes: 1 addition & 21 deletions pom.xml
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -2970,13 +2970,9 @@
<sourceDirectories>
<directory>${basedir}/src/main/java</directory>
<directory>${basedir}/src/main/scala</directory>
<directory>${basedir}/v${hive.version.short}/src/main/java</directory>
<directory>${basedir}/v${hive.version.short}/src/main/scala</directory>
</sourceDirectories>
<testSourceDirectories>
<directory>${basedir}/src/test/java</directory>
<directory>${basedir}/v${hive.version.short}/src/test/java</directory>
<directory>${basedir}/v${hive.version.short}/src/test/scala</directory>
</testSourceDirectories>
<configLocation>dev/checkstyle.xml</configLocation>
<outputFile>${basedir}/target/checkstyle-output.xml</outputFile>
Expand Down Expand Up @@ -3150,23 +3146,7 @@

<profile>
<id>hive-1.2</id>
<properties>
<hive.group>org.spark-project.hive</hive.group>
<hive.classifier></hive.classifier>
<!-- Version used in Maven Hive dependency -->
<hive.version>1.2.1.spark2</hive.version>
<!-- Version used for internal directory structure -->
<hive.version.short>1.2</hive.version.short>
<hive.parquet.scope>${hive.deps.scope}</hive.parquet.scope>
<hive.storage.version>2.6.0</hive.storage.version>
<hive.storage.scope>provided</hive.storage.scope>
<hive.common.scope>provided</hive.common.scope>
<hive.llap.scope>provided</hive.llap.scope>
<hive.serde.scope>provided</hive.serde.scope>
<hive.shims.scope>provided</hive.shims.scope>
<orc.classifier>nohive</orc.classifier>
<datanucleus-core.version>3.2.10</datanucleus-core.version>
</properties>
<!-- Exists only for backward compatibility. No-op. -->

@viirya viirya Oct 5, 2020 •
edited
Loading

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

So this profile is still useful after we remove all the hive-1.2 stuffs?

@srowen srowen Oct 5, 2020

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

No, it's just that you get an error if you specify -Phive-1.2 if it's removed entirely. Now, maybe that's a good thing? previously for Scala profiles I had left them in because, for example -Pscala-2.12 was not the default before. When it became the default, that would have caused an error even though it was already set up for 2.12. But this case is probably different: someone selecting Hive 1.x support should see an error, probably? in which case this should be removed.

@viirya viirya Oct 5, 2020

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Yeah, this sounds more correct.

@dongjoon-hyun dongjoon-hyun Oct 5, 2020

Copy link
Copy Markdown
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Got it. I'll remove this and raise error explicitly, @viirya and @srowen . :)

</profile>

<profile>
Expand Down
3 changes: 0 additions & 3 deletions sql/core/pom.xml
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -221,8 +221,6 @@
</goals>
<configuration>
<sources>
<source>v${hive.version.short}/src/main/scala</source>
<source>v${hive.version.short}/src/main/java</source>
<source>src/main/scala-${scala.binary.version}</source>
</sources>
</configuration>
Expand All @@ -235,7 +233,6 @@
</goals>
<configuration>
<sources>
<source>v${hive.version.short}/src/test/scala</source>
<source>src/test/gen-java</source>
</sources>
</configuration>
Expand Down
0 ...tion/datasources/orc/OrcColumnVector.java → ...tion/datasources/orc/OrcColumnVector.java
View file
Open in desktop
File renamed without changes.
0 ...xecution/datasources/orc/OrcFilters.scala → ...xecution/datasources/orc/OrcFilters.scala
View file
Open in desktop
File renamed without changes.
0 ...cution/datasources/orc/OrcShimUtils.scala → ...cution/datasources/orc/OrcShimUtils.scala
View file
Open in desktop
File renamed without changes.
0 ...tion/datasources/orc/OrcFilterSuite.scala → ...tion/datasources/orc/OrcFilterSuite.scala
View file
Open in desktop
File renamed without changes.
208 changes: 0 additions & 208 deletions ...re/v1.2/src/main/java/org/apache/spark/sql/execution/datasources/orc/OrcColumnVector.java
View file
Open in desktop

This file was deleted.

79 changes: 0 additions & 79 deletions ...ore/v1.2/src/main/scala/org/apache/spark/sql/execution/datasources/orc/DaysWritable.scala
View file
Open in desktop

This file was deleted.

Loading