[SPARK-4943][SPARK-5251][SQL] Allow table name having dot for db/catalog #4062

scwf · 2015-01-15T14:17:14Z

Follow up for #3941 and #3960, in this PR
1 enable ddl table name having dot
2 enable hivecontext related api support dot table name with tableIdentifier
3 remove QualifiedTableName which do not match the other API, using tableIdentifier instead.

SparkQA · 2015-01-15T14:22:36Z

Test build #25602 has started for PR 4062 at commit e0d1960.

This patch merges cleanly.

SparkQA · 2015-01-15T15:29:43Z

Test build #25602 has finished for PR 4062 at commit e0d1960.

This patch passes all tests.
This patch merges cleanly.
This patch adds no public classes.

AmplabJenkins · 2015-01-15T15:29:46Z

Test PASSed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/25602/
Test PASSed.

marmbrus · 2015-01-16T21:47:27Z

sql/hive/src/main/scala/org/apache/spark/sql/hive/HiveContext.scala

Why hardcode default here? When a database isn't specified its up to the catalog to use the current one, where as this will ignore any USE statements that have been run.

@marmbrus yes, this just inherited from the old version. Now invalidateTable do not support database as you suggest, so here is a todo in line 102, how about make another PR to fix this? or do it in this PR?

@marmbrus I will make a new PR to fix the database issue after this in.

scwf · 2015-01-20T14:59:30Z

@marmbrus, any other comments here?

scwf · 2015-01-22T15:44:33Z

ping @marmbrus

SparkQA · 2015-01-30T01:22:51Z

Test build #26359 has started for PR 4062 at commit 8857756.

This patch merges cleanly.

SparkQA · 2015-01-30T02:28:33Z

Test build #26359 has finished for PR 4062 at commit 8857756.

This patch passes all tests.
This patch merges cleanly.
This patch adds no public classes.

AmplabJenkins · 2015-01-30T02:28:36Z

Test PASSed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/26359/
Test PASSed.

scwf · 2015-01-30T03:23:06Z

ping @marmbrus, is this ok to go?

scwf · 2015-02-02T03:49:57Z

ping

SparkQA · 2015-02-04T03:12:49Z

Test build #26708 has started for PR 4062 at commit fcfa126.

This patch merges cleanly.

SparkQA · 2015-02-04T03:13:44Z

Test build #26708 has finished for PR 4062 at commit fcfa126.

This patch fails Scala style tests.
This patch merges cleanly.
This patch adds no public classes.

AmplabJenkins · 2015-02-04T03:13:45Z

Test FAILed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/26708/
Test FAILed.

SparkQA · 2015-02-04T03:52:37Z

Test build #26715 has started for PR 4062 at commit 057d23e.

This patch merges cleanly.

SparkQA · 2015-02-04T05:03:46Z

Test build #26715 has finished for PR 4062 at commit 057d23e.

This patch passes all tests.
This patch merges cleanly.
This patch adds no public classes.

AmplabJenkins · 2015-02-04T05:03:49Z

Test PASSed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/26715/
Test PASSed.

marmbrus · 2015-03-18T02:24:06Z

Okay, sorry this didn't make it into 1.3. Mind bringing it up to date?

scwf · 2015-03-19T09:29:23Z

yeah later back to update this

liancheng · 2015-04-15T17:09:06Z

Sorry... Would you mind to rebase this again? Thanks!

SparkQA · 2015-04-22T11:43:41Z

Test build #30753 has started for PR 4062 at commit e63c4ed.

SparkQA · 2015-04-22T11:47:43Z

Test build #30753 has finished for PR 4062 at commit e63c4ed.

This patch fails to build.
This patch merges cleanly.
This patch adds no public classes.
This patch does not change any dependencies.

AmplabJenkins · 2015-04-22T11:47:44Z

Test FAILed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/30753/
Test FAILed.

SparkQA · 2015-04-22T12:28:40Z

Test build #30755 has started for PR 4062 at commit 4a318f3.

SparkQA · 2015-04-22T14:06:35Z

Test build #30755 has finished for PR 4062 at commit 4a318f3.

This patch passes all tests.
This patch merges cleanly.
This patch adds no public classes.
This patch does not change any dependencies.

AmplabJenkins · 2015-04-22T14:06:40Z

Test PASSed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/30755/
Test PASSed.

SparkQA · 2015-04-22T16:28:42Z

Test build #30766 has started for PR 4062 at commit e437e89.

scwf · 2015-04-22T16:34:21Z

Updated and added test cases.

SparkQA · 2015-04-22T18:07:20Z

Test build #30766 has finished for PR 4062 at commit e437e89.

This patch passes all tests.
This patch merges cleanly.
This patch adds no public classes.
This patch does not change any dependencies.

AmplabJenkins · 2015-04-22T18:07:25Z

Test PASSed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/30766/
Test PASSed.

scwf · 2015-04-22T23:43:09Z

To summarize this:
1 enable ddl table name having dot
2 enable HiveContext/DataFrame/SQLContext related api support dot table name with tableIdentifier
3 remove QualifiedTableName which do not match the other API from HiveMetastoreCatalog, using tableIdentifier instead.
4 tablename x.y.z.....m.n indicate that m is the db name and n is the table name

scwf · 2015-04-22T23:44:05Z

/cc @marmbrus @liancheng

scwf · 2015-04-22T23:47:37Z

sql/hive/src/main/scala/org/apache/spark/sql/hive/HiveMetastoreCatalog.scala

should not rename this method

scwf · 2015-04-25T01:45:43Z

Jenkins, retest this please

scwf · 2015-04-27T06:26:37Z

@liancheng , rebased and updated, any comments here?

scwf · 2015-05-03T01:27:00Z

Jenkins, retest this please

AmplabJenkins · 2015-05-03T01:27:11Z

Merged build triggered.

AmplabJenkins · 2015-05-03T01:27:20Z

Merged build started.

SparkQA · 2015-05-03T01:27:42Z

Test build #31680 has started for PR 4062 at commit 7c65386.

SparkQA · 2015-05-03T03:14:42Z

Test build #31680 has finished for PR 4062 at commit 7c65386.

This patch passes all tests.
This patch merges cleanly.
This patch adds no public classes.

AmplabJenkins · 2015-05-03T03:14:47Z

Merged build finished. Test PASSed.

AmplabJenkins · 2015-05-03T03:14:47Z

Test PASSed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/31680/
Test PASSed.

scwf · 2015-05-06T21:53:22Z

ping

scwf added 5 commits January 14, 2015 22:40

use tableIdentifier in metastore

fcab9b0

using tableidentfier when create table in hive

111331c

using tableidentifier in ddl also

481ba95

Merge branch 'tableIdentifier' into createtable

3c1fcbb

fix conflicts

e0d1960

marmbrus reviewed Jan 16, 2015
View reviewed changes

This was referenced Jan 16, 2015

[SPARK-5240][SQL] Adding createDataSourceTable interface to Catalog #4036

Closed

[SPARK-5251][SQL] Using tableIdentifier in hive metastore #4045

Closed

scwf changed the title ~~[SPARK-4943][SQL] Allow table name having dot for db/catalog~~ [SPARK-4943][SPARK-5251][SQL] Allow table name having dot for db/catalog Jan 17, 2015

resolve conflicts

8857756

conflicts fix

fcfa126

style fix

057d23e

fix conflicts

4a318f3

scwf force-pushed the createtable branch from e63c4ed to 4a318f3 Compare April 22, 2015 12:23

scwf added 2 commits April 22, 2015 22:27

improvement

a7139fa

added test cases

e437e89

scwf reviewed Apr 22, 2015
View reviewed changes

sql/hive/src/main/scala/org/apache/spark/sql/hive/HiveMetastoreCatalog.scala Outdated

Copy link

Contributor Author

scwf Apr 22, 2015

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

should not rename this method

hiveTableFilePath -> hiveDefaultTableFilePath

7c65386

scwf closed this Jul 1, 2015

[SPARK-4943][SPARK-5251][SQL] Allow table name having dot for db/catalog #4062

[SPARK-4943][SPARK-5251][SQL] Allow table name having dot for db/catalog #4062

Uh oh!

Conversation

scwf commented Jan 15, 2015

Uh oh!

SparkQA commented Jan 15, 2015

Uh oh!

SparkQA commented Jan 15, 2015

Uh oh!

AmplabJenkins commented Jan 15, 2015

Uh oh!

marmbrus Jan 16, 2015

Choose a reason for hiding this comment

Uh oh!

scwf Jan 16, 2015

Choose a reason for hiding this comment

Uh oh!

scwf Jan 17, 2015

Choose a reason for hiding this comment

Uh oh!

scwf commented Jan 20, 2015

Uh oh!

scwf commented Jan 22, 2015

Uh oh!

SparkQA commented Jan 30, 2015

Uh oh!

SparkQA commented Jan 30, 2015

Uh oh!

AmplabJenkins commented Jan 30, 2015

Uh oh!

scwf commented Jan 30, 2015

Uh oh!

scwf commented Feb 2, 2015

Uh oh!

SparkQA commented Feb 4, 2015

Uh oh!

SparkQA commented Feb 4, 2015

Uh oh!

AmplabJenkins commented Feb 4, 2015

Uh oh!

SparkQA commented Feb 4, 2015

Uh oh!

SparkQA commented Feb 4, 2015

Uh oh!

AmplabJenkins commented Feb 4, 2015

Uh oh!

marmbrus commented Mar 18, 2015

Uh oh!

scwf commented Mar 19, 2015

Uh oh!

liancheng commented Apr 15, 2015

Uh oh!

SparkQA commented Apr 22, 2015

Uh oh!

SparkQA commented Apr 22, 2015

Uh oh!

AmplabJenkins commented Apr 22, 2015

Uh oh!

SparkQA commented Apr 22, 2015

Uh oh!

SparkQA commented Apr 22, 2015

Uh oh!

AmplabJenkins commented Apr 22, 2015

Uh oh!

SparkQA commented Apr 22, 2015

Uh oh!

scwf commented Apr 22, 2015

Uh oh!

SparkQA commented Apr 22, 2015

Uh oh!

AmplabJenkins commented Apr 22, 2015

Uh oh!

scwf commented Apr 22, 2015

Uh oh!

scwf commented Apr 22, 2015

Uh oh!

scwf Apr 22, 2015

Choose a reason for hiding this comment

Uh oh!