Skip to content

[SPARK-31828][SQL] Retain table properties at CreateTableLikeCommand #28647

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Closed
ulysses-you wants to merge 34 commits into from
Closed

[SPARK-31828][SQL] Retain table properties at CreateTableLikeCommand #28647

Show file tree
Hide file tree
Changes from 2 commits
Commits
Show all changes
34 commits
Select commit Hold shift + click to select a range
61dfe05
init
ulysses-you May 27, 2020
37ba3b3
fix
ulysses-you May 27, 2020
594a1bb
update doc
ulysses-you May 28, 2020
b30d62c
remove metastore proerties
ulysses-you May 29, 2020
29aba43
update doc
ulysses-you May 29, 2020
162627a
add comment
ulysses-you May 29, 2020
9eacf1e
add meta test
ulysses-you May 29, 2020
175d0e2
fix view
ulysses-you May 29, 2020
ed0877e
fix
ulysses-you May 29, 2020
dd04dcc
fix
ulysses-you May 30, 2020
4507e7f
fix
ulysses-you May 30, 2020
bc52948
fix
ulysses-you May 30, 2020
6b523e1
fix
ulysses-you May 31, 2020
0e79ed3
update doc
ulysses-you May 31, 2020
46d7a7b
fix comment
ulysses-you May 31, 2020
1edc619
Merge branch 'master' of https://github.com/apache/spark into SPARK-3…
ulysses-you Jun 24, 2020
78ff34f
update reserved properties
ulysses-you Jun 24, 2020
2251181
revert last commit
ulysses-you Jun 24, 2020
5c63477
check format
ulysses-you Jun 28, 2020
183c209
fix
ulysses-you Jun 28, 2020
fcc8b3b
update doc
ulysses-you Jun 29, 2020
006ec47
update doc
ulysses-you Jul 1, 2020
319fbfb
update doc and comment
ulysses-you Jul 1, 2020
7f9f685
remove useless properties in HiveClientImpl
ulysses-you Jul 1, 2020
93a0c75
add view test
ulysses-you Jul 1, 2020
3e8af07
add test
ulysses-you Jul 1, 2020
0a3b1cf
fix
ulysses-you Jul 1, 2020
1e1349f
fix
ulysses-you Jul 2, 2020
19f398b
update constants
ulysses-you Jul 8, 2020
03966ae
add using hive
ulysses-you Jul 10, 2020
dc00260
Merge branch 'master' of https://github.com/apache/spark into SPARK-3…
ulysses-you Nov 2, 2020
2d470fc
minor
ulysses-you Nov 9, 2020
4b55575
Merge branch 'master' of https://github.com/apache/spark into SPARK-3…
ulysses-you Nov 17, 2020
c45489a
Merge branch 'master' of https://github.com/apache/spark into SPARK-3…
ulysses-you Nov 24, 2020
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
3 changes: 2 additions & 1 deletion sql/core/src/main/scala/org/apache/spark/sql/execution/command/tables.scala
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -115,6 +115,7 @@ case class CreateTableLikeCommand(
CatalogTableType.EXTERNAL
}

val newProperties = sourceTableDesc.properties ++ properties
val newTableDesc =
CatalogTable(
identifier = targetTable,
Expand All @@ -124,7 +125,7 @@ case class CreateTableLikeCommand(
provider = newProvider,
partitionColumnNames = sourceTableDesc.partitionColumnNames,
bucketSpec = sourceTableDesc.bucketSpec,
properties = properties,
properties = newProperties,
tracksPartitionsInCatalog = sourceTableDesc.tracksPartitionsInCatalog)

catalog.createTable(newTableDesc, ifNotExists)
Expand Down
20 changes: 19 additions & 1 deletion sql/hive/src/test/scala/org/apache/spark/sql/hive/execution/HiveDDLSuite.scala
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -2632,7 +2632,6 @@ class HiveDDLSuite
assert(source.properties("a") == "apple")
sql("CREATE TABLE t LIKE s STORED AS parquet TBLPROPERTIES('f'='foo', 'b'='bar')")
val table = catalog.getTableMetadata(TableIdentifier("t"))
assert(table.properties.get("a") === None)
Copy link
Contributor

@cloud-fan cloud-fan May 27, 2020

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Seems like we intentionally don't keep the table properties from the original table. @maropu @dongjoon-hyun @viirya are you OK with the behavior change proposed by this PR?

Copy link
Member

@viirya viirya May 27, 2020

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Based on the description of CreateTableLikeCommand:

The CatalogTable attributes copied from the source table are storage(inputFormat, outputFormat, serde, compressed, properties), schema, provider, partitionColumnNames, bucketSpec by default.

So we don't say table properties are copied from source table too. Not sure about why we didn't copy table properties.

I feel it is OK as seems copying original table properties should not be harmful. And we already copy storage properties.

But we should update the doc together.

Copy link
Member

@maropu maropu May 28, 2020

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

hm, I have the same opinion with @viirya; I couldn't find any strong reason not to copy the properites. Rather, is this a kind of bugs? Anyway, yea, we should clearly descibe the behaivour in our doc...

assert(table.properties("f") == "foo")
assert(table.properties("b") == "bar")
}
Expand Down Expand Up @@ -2720,4 +2719,23 @@ class HiveDDLSuite
checkAnswer(sql("SHOW PARTITIONS ta_part"), Row("ts=10") :: Nil)
}
}

test("SPARK-31828: Retain table properties at CreateTableLikeCommand") {
val catalog = spark.sessionState.catalog
withTable("t1", "t2", "t3") {
sql("CREATE TABLE t1(c1 int) TBLPROPERTIES('k1'='v1', 'k2'='v2')")
Copy link
Contributor

@cloud-fan cloud-fan Jul 9, 2020

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

can we add USING hive? otherwise if we change the default table format, this test will be broken.

Copy link
Contributor Author

@ulysses-you ulysses-you Jul 10, 2020

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

added.

val t1 = catalog.getTableMetadata(TableIdentifier("t1"))
assert(t1.properties("k1") == "v1")
assert(t1.properties("k2") == "v2")
sql("CREATE TABLE t2 LIKE t1 TBLPROPERTIES('k2'='v3', 'k4'='v4')")
val t2 = catalog.getTableMetadata(TableIdentifier("t2"))
assert(t2.properties("k1") == "v1")
assert(t2.properties("k2") == "v3")
assert(t2.properties("k4") == "v4")
sql("CREATE TABLE t3 LIKE t1")
val t3 = catalog.getTableMetadata(TableIdentifier("t3"))
assert(t3.properties("k1") == "v1")
assert(t3.properties("k2") == "v2")
}
}
}