Skip to content

[HUDI-2057][HUDI-5499] CTAS Generate An External Table When Create Managed Table #6419

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Open
dongkelun wants to merge 5 commits into
base: master
Choose a base branch
from
Open

[HUDI-2057][HUDI-5499] CTAS Generate An External Table When Create Managed Table #6419

Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
5 commits
Select commit Hold shift + click to select a range
9eed30f
[HUDI-2057] CTAS Generate An External Table When Create Managed Table
dongkelun Nov 4, 2022
f1abf8a
Merge branch 'master' of https://github.com/apache/hudi into HUDI-2057
dongkelun Nov 6, 2022
5c1cb21
Add validate CTAS table type
dongkelun Nov 7, 2022
c945db2
Add validate CTAS table type
dongkelun Nov 8, 2022
cfd2056
Restore code to verify the failure reason of check
dongkelun Nov 8, 2022
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
2 changes: 1 addition & 1 deletion ...rce/hudi-spark-common/src/main/scala/org/apache/spark/sql/hudi/ProvidesHoodieConfig.scala
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -171,7 +171,7 @@ trait ProvidesHoodieConfig extends Logging {

logInfo(s"Insert statement use write operation type: $operation, payloadClass: $payloadClassName")

withSparkConf(sparkSession, catalogProperties) {
withSparkConf(sparkSession, catalogProperties ++ extraOptions) {
Copy link
Contributor

@yihua yihua Sep 17, 2022

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Could you add a test that should fail before this fix, to guard any breaking changes in the future?

Copy link
Contributor Author

@dongkelun dongkelun Sep 19, 2022

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I can't figure out how to implement the test case.Do you have any ideas?

Map(
"path" -> path,
TABLE_TYPE.key -> tableType,
Expand Down
60 changes: 60 additions & 0 deletions packaging/bundle-validation/spark_hadoop_mr/validateCtasTableType.scala
View file
Open in desktop
Original file line number Diff line number Diff line change
@@ -0,0 +1,60 @@
/*
* Licensed to the Apache Software Foundation (ASF) under one
* or more contributor license agreements. See the NOTICE file
* distributed with this work for additional information
* regarding copyright ownership. The ASF licenses this file
* to you under the Apache License, Version 2.0 (the
* "License"); you may not use this file except in compliance
* with the License. You may obtain a copy of the License at
*
* http://www.apache.org/licenses/LICENSE-2.0
*
* Unless required by applicable law or agreed to in writing, software
* distributed under the License is distributed on an "AS IS" BASIS,
* WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
* See the License for the specific language governing permissions and
* limitations under the License.
*/
import org.apache.spark.sql.hudi.HoodieSqlCommonUtils

val database = "default"

if (HoodieSqlCommonUtils.isUsingHiveCatalog(spark)) {
val tableName1 = "test_ctas_1"
spark.sql(
s"""
| create table $tableName1 using hudi
| tblproperties(
| primaryKey = 'id'
| )
| AS
| select 1 as id, 'a1' as name, 10 as price, 1000 as ts
""".stripMargin)
if (spark.catalog.getTable(tableName1).tableType.equals("MANAGED")) {
val tableName2 = "test_ctas_2"
spark.sql(
s"""
| create table $tableName2 using hudi
| tblproperties(
| primaryKey = 'id'
| )
| location '/tmp/$tableName2'
| AS
| select 1 as id, 'a1' as name, 10 as price, 1000 as ts
Copy link
Member

@xushiyan xushiyan Nov 9, 2022

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

please move this to a proper unit test. Bundle validation is not the right place to execute such tests.

Copy link
Contributor Author

@dongkelun dongkelun Nov 9, 2022

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

@xushiyan Hello, the purpose here is to test the type of the sync hive table, but sync hive cannot be realized in unit test

""".stripMargin)
if (spark.catalog.getTable(tableName2).tableType.equals("EXTERNAL")) {
System.out.println("CTAS hive table type validation passed.")
System.exit(0)
} else {
System.err.println(s"CTAS hive table type validation failed:\n\tThe table type of $database.$tableName2 should be EXTERNAL")
System.exit(1)
}
} else {
System.err.println(s"CTAS hive table type validation failed:\n\tThe table type of $database.$tableName1 should be MANAGED")
System.exit(1)
}
} else {
System.err.println(s"CTAS hive table type validation failed:\n\tSpark should us Hive as Session's Catalog")
System.exit(1)
}

26 changes: 26 additions & 0 deletions packaging/bundle-validation/validate.sh
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -75,6 +75,27 @@ test_spark_hadoop_mr_bundles () {
echo "::warning::validate.sh spark & hadoop-mr bundles validation was successful."
}

##
# Function to test the hive table type with hive sync by CTAS in spark bundle.
#
# env vars (defined in container):
# HIVE_HOME: path to the hive directory
# DERBY_HOME: path to the derby directory
# SPARK_HOME: path to the spark directory
##
test_CTAS_hiveTableType () {
echo "::warning::validate.sh setting up hive metastore for CTAS hive table type validation"

$DERBY_HOME/bin/startNetworkServer -h 0.0.0.0 &
$HIVE_HOME/bin/hiveserver2 &
echo "::warning::validate.sh hive metastore setup complete. Testing"
$SPARK_HOME/bin/spark-shell --jars $JARS_DIR/spark.jar < $WORKDIR/spark_hadoop_mr/validateCtasTableType.scala
if [ "$?" -ne 0 ]; then
echo "::error::validate.sh failed validate CTAS hive table type"
exit 1
fi
echo "::warning::validate.sh CTAS hive table type validation successful"
}

##
# Function to test the utilities bundle and utilities slim bundle + spark bundle.
Expand Down Expand Up @@ -168,6 +189,11 @@ if [ "$?" -ne 0 ]; then
fi
echo "::warning::validate.sh done validating spark & hadoop-mr bundle"

test_CTAS_hiveTableType
if [ "$?" -ne 0 ]; then
exit 1
fi

if [[ $SPARK_HOME == *"spark-2.4"* ]] || [[ $SPARK_HOME == *"spark-3.1"* ]]
then
echo "::warning::validate.sh validating utilities bundle"
Expand Down