[SPARK-19723][SQL]create datasource table with an non-existent location should work #17055
Conversation
|
Test build #73408 has started for PR 17055 at commit |
|
retest this please |
|
Test build #73415 has finished for PR 17055 at commit
|
| withTable("t", "t1") { | ||
| withTempDir { | ||
| dir => | ||
| dir.delete() |
There was a problem hiding this comment.
there are lots of dir existed test case in DDLSuit or HiveDDLSuit, so we just add non-existent test cases
|
retest this please |
|
Test build #73427 has finished for PR 17055 at commit
|
|
cc @gatorsmile @cloud-fan I will appreciate that you could help to review this pr~ |
|
ping @gatorsmile |
|
I will review this PR in the next few days. Thanks! |
|
ok, thanks very much~ |
|
Test build #73762 has finished for PR 17055 at commit
|
| withTable("t", "t1") { | ||
| withTempDir { | ||
| dir => | ||
| dir.delete() |
| test("create datasource table with a non-existing location") { | ||
| withTable("t", "t1") { | ||
| withTempDir { | ||
| dir => |
There was a problem hiding this comment.
Nit: style issue. the above two lines can be combined together.
| |CREATE TABLE t(a int, b int) | ||
| |USING parquet | ||
| |LOCATION '$dir' | ||
| """.stripMargin) |
There was a problem hiding this comment.
The test case is already pretty long. Please reduce the sql statement to a single line.
| } | ||
| } | ||
|
|
||
| test("create datasource table with a non-existing location") { |
There was a problem hiding this comment.
Let me try to combine the HiveDDLSuite.scala and DDLSuite.scala. Otherwise, the test cases need to be duplicated in every PR.
|
Test build #74003 has finished for PR 17055 at commit
|
|
Test build #74010 has finished for PR 17055 at commit
|
|
Test build #74015 has finished for PR 17055 at commit
|
|
@gatorsmile this pr could be merged? |
|
Test build #74261 has finished for PR 17055 at commit
|
|
@gatorsmile I have merged with master，if it is ok, could you help to merge it? |
|
Test build #74263 has finished for PR 17055 at commit
|
|
Test build #74264 has finished for PR 17055 at commit
|
|
I will review it today. Thanks! |
|
OK thanks a lot～ |
| Seq(true, false).foreach { shouldDelete => | ||
| val tcName = if (shouldDelete) "non-existent" else "existed" | ||
| val tcName = if (shouldDelete) "non-existing" else "existed" | ||
| test(s"CTAS for external data source table with a $tcName location") { |
There was a problem hiding this comment.
There was a problem hiding this comment.
oh, it is. after we also qualified the location path for datasource table , the code for equal two location are the same with HiveExternalCatalog. thanks~
| withTempDir { dir => | ||
| if (shouldDelete) { | ||
| dir.delete() | ||
| } |
There was a problem hiding this comment.
if (shouldDelete) dir.delete()
| spark.sql( | ||
| s""" | ||
| |CREATE TABLE t1(a int, b int) USING parquet PARTITIONED BY(a) LOCATION '$dir' | ||
| """.stripMargin) |
|
Test build #74360 has finished for PR 17055 at commit
|
|
Test build #74361 has finished for PR 17055 at commit
|
|
thanks, merging to master! |
4 participants
What changes were proposed in this pull request?
This JIRA is a follow up work after SPARK-19583
As we discussed in that PR
The following DDL for datasource table with an non-existent location should work:
Currently it will throw exception that path not exists for datasource table for datasource table
How was this patch tested?
unit test added