[SPARK-19724][SQL]create a managed table with an existed default table should throw an exception #17272

windpiger · 2017-03-13T07:26:34Z

What changes were proposed in this pull request?

This JIRA is a follow up work after SPARK-19583

As we discussed in that PR

The following DDL for a managed table with an existed default location should throw an exception:

CREATE TABLE ... (PARTITIONED BY ...) AS SELECT ...
CREATE TABLE ... (PARTITIONED BY ...)

Currently there are some situations which are not consist with above logic:

CREATE TABLE ... (PARTITIONED BY ...) succeed with an existed default location
situation: for both hive/datasource(with HiveExternalCatalog/InMemoryCatalog)
CREATE TABLE ... (PARTITIONED BY ...) AS SELECT ...
situation: hive table succeed with an existed default location

This PR is going to make above two situations consist with the logic that it should throw an exception
with an existed default location.

How was this patch tested?

unit test added

…e path should throw an exception

windpiger · 2017-03-13T07:27:22Z

sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/catalog/InMemoryCatalog.scala

    val db = tableDefinition.identifier.database.get
    requireDbExists(db)
    val table = tableDefinition.identifier.table
-    if (tableExists(db, table)) {


we have check if the table exists in SessionCatalog.createtable

SparkQA · 2017-03-13T07:28:59Z

Test build #74436 has finished for PR 17272 at commit a0ba419.

This patch fails Scala style tests.
This patch merges cleanly.
This patch adds no public classes.

windpiger · 2017-03-13T07:29:00Z

sql/catalyst/src/test/scala/org/apache/spark/sql/catalyst/catalog/ExternalCatalogSuite.scala

    assert(actual.tableType === CatalogTableType.EXTERNAL)
  }

-  test("create table when the table already exists") {


since in InMemoryCatalog and HiveExternalCatalog does not check if the table exists, here move the test case to SessionCatalogSuite(using InMemoryCatalog) and HiveExternalCatalogSuite

windpiger · 2017-03-13T07:30:28Z

sql/core/src/main/scala/org/apache/spark/sql/execution/command/createDataSourceTables.scala

        table.storage.locationUri
      }
+
+      sparkSession.sessionState.catalog.checkTableOrPathExists(table, ignoreIfExists = false)


because in following saveDataIntoTable will create the table path and store data in it, we should not check the path in following createTable, we check it here before saveDataIntoTable

SparkQA · 2017-03-13T08:48:40Z

Test build #74437 has finished for PR 17272 at commit 65d7ea9.

This patch fails Spark unit tests.
This patch merges cleanly.
This patch adds no public classes.

SparkQA · 2017-03-13T09:21:08Z

Test build #74441 has finished for PR 17272 at commit 0e753cc.

This patch fails Spark unit tests.
This patch merges cleanly.
This patch adds no public classes.

windpiger · 2017-03-13T11:31:10Z

retest this please

SparkQA · 2017-03-13T19:03:53Z

Test build #74445 has finished for PR 17272 at commit c8d9b77.

This patch fails PySpark unit tests.
This patch merges cleanly.
This patch adds no public classes.

windpiger · 2017-03-13T23:27:39Z

retest this please

SparkQA · 2017-03-14T01:16:56Z

Test build #74468 has finished for PR 17272 at commit c8d9b77.

This patch fails PySpark unit tests.
This patch merges cleanly.
This patch adds no public classes.

SparkQA · 2017-03-14T02:19:07Z

Test build #74472 has finished for PR 17272 at commit cd4a091.

This patch fails PySpark unit tests.
This patch merges cleanly.
This patch adds no public classes.

SparkQA · 2017-03-14T06:17:34Z

Test build #74493 has started for PR 17272 at commit 2ac70b4.

windpiger · 2017-03-14T08:10:02Z

retest this please

SparkQA · 2017-03-14T09:59:52Z

Test build #74499 has finished for PR 17272 at commit 2ac70b4.

This patch fails PySpark unit tests.
This patch merges cleanly.
This patch adds no public classes.

windpiger · 2017-03-14T12:10:12Z

python/pyspark/sql/tests.py

python test failed

the location('file:/home/jenkins/workspace/SparkPullRequestBuilder/spark-warehouse/tab1') of table('`default`.`tab1`') already exists.;

the location of tab1 does not deleted , I have tried to found out in which test case it forget to delete it(search all test cases containing tab1 and run it), but they are all ok to delete the location of tab1, so here we change the table name to work around it.

SparkQA · 2017-03-14T13:46:22Z

Test build #74516 has finished for PR 17272 at commit 516c4e4.

This patch fails Spark unit tests.
This patch merges cleanly.
This patch adds no public classes.

windpiger · 2017-03-14T13:52:52Z

retest this please

SparkQA · 2017-03-14T15:41:36Z

Test build #74524 has finished for PR 17272 at commit 516c4e4.

This patch fails PySpark unit tests.
This patch merges cleanly.
This patch adds no public classes.

SparkQA · 2017-03-14T17:48:02Z

Test build #74535 has finished for PR 17272 at commit 739f207.

This patch fails PySpark unit tests.
This patch merges cleanly.
This patch adds no public classes.

SparkQA · 2017-03-15T14:34:56Z

Test build #74604 has finished for PR 17272 at commit 4351cd7.

This patch fails Python style tests.
This patch merges cleanly.
This patch adds no public classes.

HyukjinKwon · 2017-05-11T14:46:35Z

@windpiger is this still active?

gatorsmile · 2017-06-19T22:18:30Z

cc @ltnwgl Could you please take this over? Thanks!

gatorsmile · 2018-03-19T19:04:20Z

cc @dongjoon-hyun Are you interested in this PR? Just take it over?

gengliangwang · 2018-03-20T04:01:14Z

@gatorsmile I will take it over :)

…le should throw an exception ## What changes were proposed in this pull request? This PR is to finish #17272 This JIRA is a follow up work after SPARK-19583 As we discussed in that PR The following DDL for a managed table with an existed default location should throw an exception: CREATE TABLE ... (PARTITIONED BY ...) AS SELECT ... CREATE TABLE ... (PARTITIONED BY ...) Currently there are some situations which are not consist with above logic: CREATE TABLE ... (PARTITIONED BY ...) succeed with an existed default location situation: for both hive/datasource(with HiveExternalCatalog/InMemoryCatalog) CREATE TABLE ... (PARTITIONED BY ...) AS SELECT ... situation: hive table succeed with an existed default location This PR is going to make above two situations consist with the logic that it should throw an exception with an existed default location. ## How was this patch tested? unit test added Author: Gengliang Wang <[email protected]> Closes #20886 from gengliangwang/pr-17272.

…le should throw an exception ## What changes were proposed in this pull request? This PR is to finish apache#17272 This JIRA is a follow up work after SPARK-19583 As we discussed in that PR The following DDL for a managed table with an existed default location should throw an exception: CREATE TABLE ... (PARTITIONED BY ...) AS SELECT ... CREATE TABLE ... (PARTITIONED BY ...) Currently there are some situations which are not consist with above logic: CREATE TABLE ... (PARTITIONED BY ...) succeed with an existed default location situation: for both hive/datasource(with HiveExternalCatalog/InMemoryCatalog) CREATE TABLE ... (PARTITIONED BY ...) AS SELECT ... situation: hive table succeed with an existed default location This PR is going to make above two situations consist with the logic that it should throw an exception with an existed default location. ## How was this patch tested? unit test added Author: Gengliang Wang <[email protected]> Closes apache#20886 from gengliangwang/pr-17272.

## What changes were proposed in this pull request? This PR proposes to close PRs ... - inactive to the review comments more than a month - WIP and inactive more than a month - with Jenkins build failure but inactive more than a month - suggested to be closed and no comment against that - obviously looking inappropriate (e.g., Branch 0.5) To make sure, I left a comment for each PR about a week ago and I could not have a response back from the author in these PRs below: Closes apache#11129 Closes apache#12085 Closes apache#12162 Closes apache#12419 Closes apache#12420 Closes apache#12491 Closes apache#13762 Closes apache#13837 Closes apache#13851 Closes apache#13881 Closes apache#13891 Closes apache#13959 Closes apache#14091 Closes apache#14481 Closes apache#14547 Closes apache#14557 Closes apache#14686 Closes apache#15594 Closes apache#15652 Closes apache#15850 Closes apache#15914 Closes apache#15918 Closes apache#16285 Closes apache#16389 Closes apache#16652 Closes apache#16743 Closes apache#16893 Closes apache#16975 Closes apache#17001 Closes apache#17088 Closes apache#17119 Closes apache#17272 Closes apache#17971 Added: Closes apache#17778 Closes apache#17303 Closes apache#17872 ## How was this patch tested? N/A Author: hyukjinkwon <[email protected]> Closes apache#18017 from HyukjinKwon/close-inactive-prs.

[SPARK-19724][SQL]create a managed table with an existed default tabl…

a0ba419

…e path should throw an exception

windpiger commented Mar 13, 2017

View reviewed changes

windpiger added 2 commits March 13, 2017 15:31

remove empty lines

65d7ea9

fix comment

0e753cc

fix test failed

c8d9b77

fix test failed

cd4a091

windpiger force-pushed the managedTableLocationExists branch from 2ac70b4 to cd4a091 Compare March 14, 2017 12:08

windpiger commented Mar 14, 2017

View reviewed changes

reset catalog in pyspark tests

739f207

windpiger force-pushed the managedTableLocationExists branch from 516c4e4 to 739f207 Compare March 14, 2017 16:00

to find why jenkins failed

4351cd7

HyukjinKwon mentioned this pull request May 17, 2017

[INFRA] Close stale PRs #18017

Closed

asfgit closed this in 5d2750a May 18, 2017

gengliangwang mentioned this pull request Mar 22, 2018

[SPARK-19724][SQL]create a managed table with an existed default table should throw an exception #20886

Closed

[SPARK-19724][SQL]create a managed table with an existed default table should throw an exception #17272

[SPARK-19724][SQL]create a managed table with an existed default table should throw an exception #17272

Uh oh!

Conversation

windpiger commented Mar 13, 2017

What changes were proposed in this pull request?

How was this patch tested?

Uh oh!

windpiger Mar 13, 2017

Choose a reason for hiding this comment

Uh oh!

SparkQA commented Mar 13, 2017

Uh oh!

windpiger Mar 13, 2017 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

windpiger Mar 13, 2017

Choose a reason for hiding this comment

Uh oh!

SparkQA commented Mar 13, 2017

Uh oh!

SparkQA commented Mar 13, 2017

Uh oh!

windpiger commented Mar 13, 2017

Uh oh!

SparkQA commented Mar 13, 2017

Uh oh!

windpiger commented Mar 13, 2017

Uh oh!

SparkQA commented Mar 14, 2017

Uh oh!

SparkQA commented Mar 14, 2017

Uh oh!

SparkQA commented Mar 14, 2017

Uh oh!

windpiger commented Mar 14, 2017

Uh oh!

SparkQA commented Mar 14, 2017

Uh oh!

windpiger Mar 14, 2017

Choose a reason for hiding this comment

Uh oh!

SparkQA commented Mar 14, 2017

Uh oh!

windpiger commented Mar 14, 2017

Uh oh!

SparkQA commented Mar 14, 2017

Uh oh!

SparkQA commented Mar 14, 2017

Uh oh!

SparkQA commented Mar 15, 2017

Uh oh!

HyukjinKwon commented May 11, 2017

Uh oh!

gatorsmile commented Jun 19, 2017

Uh oh!

gatorsmile commented Mar 19, 2018

Uh oh!

gengliangwang commented Mar 20, 2018

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

5 participants

windpiger Mar 13, 2017 •

edited

Loading