Skip to content

[SPARK-2817] [SQL] add "show create table" support #1760

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Closed
tianyi wants to merge 13 commits into from
Closed

[SPARK-2817] [SQL] add "show create table" support #1760

Show file tree
Hide file tree
Changes from 9 commits
Commits
Show all changes
13 commits
Select commit Hold shift + click to select a range
a03db77
[SPARK-2817] add "show create table" support
tianyi Aug 4, 2014
a337bd6
[SPARK-2817] add "show create table" support
tianyi Aug 4, 2014
bbf6b42
[SPARK-2817] add a systemProperties named "test.tmp.dir" to pass the …
tianyi Aug 5, 2014
bb82726
[SPARK-2817] remove test which requires a system from the whitelist.
tianyi Aug 5, 2014
bde800a
[SPARK-2817]
tianyi Aug 7, 2014
bfc2999
[SPARK-2817] add "File.separator" support, create a "testTmpDir" outs…
tianyi Aug 8, 2014
9f97586
[SPARK-2817] remove test.tmp.dir from pom.xml
Aug 8, 2014
9b2ba11
[SPARK-2817] fix the line length problem
Aug 8, 2014
dbe1031
[SPARK-2817] move some code out of function rewritePaths, as it may b…
Aug 8, 2014
60d48a9
[SPARK-2817] use system temporary folder instead of temporary files i…
tianyi Aug 12, 2014
565ec14
[SPARK-2817] fix the case problem
tianyi Aug 12, 2014
cbffe8b
[SPARK-2817] fix the case problem
tianyi Aug 12, 2014
7d28b15
[SPARK-2817] fix too short prefix problem
tianyi Aug 12, 2014
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
8 changes: 8 additions & 0 deletions ...atibility/src/test/scala/org/apache/spark/sql/hive/execution/HiveCompatibilitySuite.scala
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -635,6 +635,14 @@ class HiveCompatibilitySuite extends HiveQueryFileTest with BeforeAndAfter {
"serde_regex",
"serde_reported_schema",
"set_variable_sub",
"show_create_table_partitioned",
"show_create_table_delimited",
"show_create_table_alter",
"show_create_table_view",
"show_create_table_serde",
"show_create_table_db_table",
"show_create_table_does_not_exist",
"show_create_table_index",
"show_describe_func_quotes",
"show_functions",
"show_partitions",
Expand Down
1 change: 1 addition & 0 deletions sql/hive/src/main/scala/org/apache/spark/sql/hive/HiveQl.scala
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -51,6 +51,7 @@ private[hive] object HiveQl {
protected val nativeCommands = Seq(
"TOK_DESCFUNCTION",
"TOK_DESCDATABASE",
"TOK_SHOW_CREATETABLE",
"TOK_SHOW_TABLESTATUS",
"TOK_SHOWDATABASES",
"TOK_SHOWFUNCTIONS",
Expand Down
14 changes: 14 additions & 0 deletions sql/hive/src/main/scala/org/apache/spark/sql/hive/TestHive.scala
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -60,6 +60,8 @@ class TestHiveContext(sc: SparkContext) extends HiveContext(sc) {
// without restarting the JVM.
System.clearProperty("spark.hostPort")



Copy link
Contributor

@marmbrus marmbrus Aug 11, 2014

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Can you remove these spurious new line additions? Below as well.

lazy val warehousePath = getTempFilePath("sparkHiveWarehouse").getCanonicalPath
lazy val metastorePath = getTempFilePath("sparkHiveMetastore").getCanonicalPath

Expand All @@ -70,6 +72,17 @@ class TestHiveContext(sc: SparkContext) extends HiveContext(sc) {
set("hive.metastore.warehouse.dir", warehousePath)
}

val testTmpDir = if (System.getProperty("user.dir").endsWith("sql" +
Copy link
Contributor

@marmbrus marmbrus Aug 11, 2014

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Could this just be:

val testTempDir = File.createtempfile("", "spark.hive.tmp")
testTempDir.delete()
testTempDir.mkdir()

I'm not sure why we would want to keep temporary files in the source tree instead of the system temporary folder.

File.separator + "hive")) {
new File(System.getProperty("user.dir") + File.separator + "tmp")
Copy link
Contributor

@chenghao-intel chenghao-intel Aug 8, 2014

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

All we need is the temporal folder, not sure why we have to check the user working directory if end with sql/hive,
Can we just use val testTmpDir = new File(System.getProperty("user.dir") + File.separator + "tmp") instead?

Copy link
Contributor Author

@tianyi tianyi Aug 8, 2014

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

@chenghao-intel I did not understand why we have to check that too. I just follow the same way of handling "inRepoTests"

Copy link
Contributor

@marmbrus marmbrus Aug 8, 2014

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

in repo tests is checking to see if the tests are there or if we should look for them in the hive distribution (they weren't always bundled with the source tree). If we need a temporary directory, we should use the mechanisms provided by the JVM for that (File.createTempFile)

} else {
new File(System.getProperty("user.dir") + File.separator + "sql" +
File.separator + "hive" + File.separator + "tmp")
}

// For some hive test case which contain ${system:test.tmp.dir}
System.setProperty("test.tmp.dir", testTmpDir.getCanonicalPath)

configure() // Must be called before initializing the catalog below.

/** The location of the compiled hive distribution */
Expand Down Expand Up @@ -109,6 +122,7 @@ class TestHiveContext(sc: SparkContext) extends HiveContext(sc) {
hiveFilesTemp.mkdir()
hiveFilesTemp.deleteOnExit()


val inRepoTests = if (System.getProperty("user.dir").endsWith("sql" + File.separator + "hive")) {
new File("src" + File.separator + "test" + File.separator + "resources" + File.separator)
} else {
Expand Down
Empty file added ...hive/src/test/resources/golden/show_create_table_alter-0-813886d6cf0875c62e89cd1d06b8b0b4
View file
Open in desktop
Empty file.
18 changes: 18 additions & 0 deletions ...hive/src/test/resources/golden/show_create_table_alter-1-2a91d52719cf4552ebeb867204552a26
View file
Open in desktop
Original file line number Diff line number Diff line change
@@ -0,0 +1,18 @@
CREATE EXTERNAL TABLE tmp_showcrt1(
key smallint,
value float)
CLUSTERED BY (
key)
SORTED BY (
value DESC)
INTO 5 BUCKETS
ROW FORMAT SERDE
'org.apache.hadoop.hive.serde2.lazy.LazySimpleSerDe'
STORED AS INPUTFORMAT
'org.apache.hadoop.mapred.TextInputFormat'
OUTPUTFORMAT
'org.apache.hadoop.hive.ql.io.HiveIgnoreKeyTextOutputFormat'
LOCATION
'file:/tmp/sparkHiveWarehouse1280221975983654134/tmp_showcrt1'
TBLPROPERTIES (
'transient_lastDdlTime'='1407132100')
Empty file added ...ive/src/test/resources/golden/show_create_table_alter-10-259d978ed9543204c8b9c25b6e25b0de
View file
Open in desktop
Empty file.
Empty file added ...hive/src/test/resources/golden/show_create_table_alter-2-928cc85c025440b731e5ee33e437e404
View file
Open in desktop
Empty file.
22 changes: 22 additions & 0 deletions ...hive/src/test/resources/golden/show_create_table_alter-3-2a91d52719cf4552ebeb867204552a26
View file
Open in desktop
Original file line number Diff line number Diff line change
@@ -0,0 +1,22 @@
CREATE TABLE tmp_showcrt1(
key smallint,
value float)
COMMENT 'temporary table'
CLUSTERED BY (
key)
SORTED BY (
value DESC)
INTO 5 BUCKETS
ROW FORMAT SERDE
'org.apache.hadoop.hive.serde2.lazy.LazySimpleSerDe'
STORED AS INPUTFORMAT
'org.apache.hadoop.mapred.TextInputFormat'
OUTPUTFORMAT
'org.apache.hadoop.hive.ql.io.HiveIgnoreKeyTextOutputFormat'
LOCATION
'file:/tmp/sparkHiveWarehouse1280221975983654134/tmp_showcrt1'
TBLPROPERTIES (
'EXTERNAL'='FALSE',
'last_modified_by'='tianyi',
'last_modified_time'='1407132100',
'transient_lastDdlTime'='1407132100')
Empty file added ...hive/src/test/resources/golden/show_create_table_alter-4-c2cb6a7d942d4dddd1aababccb1239f9
View file
Open in desktop
Empty file.
21 changes: 21 additions & 0 deletions ...hive/src/test/resources/golden/show_create_table_alter-5-2a91d52719cf4552ebeb867204552a26
View file
Open in desktop
Original file line number Diff line number Diff line change
@@ -0,0 +1,21 @@
CREATE EXTERNAL TABLE tmp_showcrt1(
key smallint,
value float)
COMMENT 'changed comment'
CLUSTERED BY (
key)
SORTED BY (
value DESC)
INTO 5 BUCKETS
ROW FORMAT SERDE
'org.apache.hadoop.hive.serde2.lazy.LazySimpleSerDe'
STORED AS INPUTFORMAT
'org.apache.hadoop.mapred.TextInputFormat'
OUTPUTFORMAT
'org.apache.hadoop.hive.ql.io.HiveIgnoreKeyTextOutputFormat'
LOCATION
'file:/tmp/sparkHiveWarehouse1280221975983654134/tmp_showcrt1'
TBLPROPERTIES (
'last_modified_by'='tianyi',
'last_modified_time'='1407132100',
'transient_lastDdlTime'='1407132100')
Empty file added sql/hive/src/test/resources/golden/show_create_table_alter-6-fdd1bd7f9acf0b2c8c9b7503d4046cb
View file
Open in desktop
Empty file.
21 changes: 21 additions & 0 deletions ...hive/src/test/resources/golden/show_create_table_alter-7-2a91d52719cf4552ebeb867204552a26
View file
Open in desktop
Original file line number Diff line number Diff line change
@@ -0,0 +1,21 @@
CREATE EXTERNAL TABLE tmp_showcrt1(
key smallint,
value float)
COMMENT 'changed comment'
CLUSTERED BY (
key)
SORTED BY (
value DESC)
INTO 5 BUCKETS
ROW FORMAT SERDE
'org.apache.hadoop.hive.serde2.lazy.LazySimpleSerDe'
STORED AS INPUTFORMAT
'org.apache.hadoop.mapred.TextInputFormat'
OUTPUTFORMAT
'org.apache.hadoop.hive.ql.io.HiveIgnoreKeyTextOutputFormat'
LOCATION
'file:/tmp/sparkHiveWarehouse1280221975983654134/tmp_showcrt1'
TBLPROPERTIES (
'last_modified_by'='tianyi',
'last_modified_time'='1407132101',
'transient_lastDdlTime'='1407132101')
Empty file added ...hive/src/test/resources/golden/show_create_table_alter-8-22ab6ed5b15a018756f454dd2294847e
View file
Open in desktop
Empty file.
21 changes: 21 additions & 0 deletions ...hive/src/test/resources/golden/show_create_table_alter-9-2a91d52719cf4552ebeb867204552a26
View file
Open in desktop
Original file line number Diff line number Diff line change
@@ -0,0 +1,21 @@
CREATE EXTERNAL TABLE tmp_showcrt1(
key smallint,
value float)
COMMENT 'changed comment'
CLUSTERED BY (
key)
SORTED BY (
value DESC)
INTO 5 BUCKETS
ROW FORMAT SERDE
'org.apache.hadoop.hive.serde2.lazy.LazySimpleSerDe'
STORED BY
'org.apache.hadoop.hive.ql.metadata.DefaultStorageHandler'
WITH SERDEPROPERTIES (
'serialization.format'='1')
LOCATION
'file:/tmp/sparkHiveWarehouse1280221975983654134/tmp_showcrt1'
TBLPROPERTIES (
'last_modified_by'='tianyi',
'last_modified_time'='1407132101',
'transient_lastDdlTime'='1407132101')
Empty file added ...e/src/test/resources/golden/show_create_table_db_table-0-67509558a4b2d39b25787cca33f52635
View file
Open in desktop
Empty file.
2 changes: 2 additions & 0 deletions ...e/src/test/resources/golden/show_create_table_db_table-1-549981e00a3d95f03dd5a9ef6044aa20
View file
Open in desktop
Original file line number Diff line number Diff line change
@@ -0,0 +1,2 @@
default
tmp_feng
Empty file added ...ve/src/test/resources/golden/show_create_table_db_table-2-34ae7e611d0aedbc62b6e420347abee
View file
Open in desktop
Empty file.
Empty file added ...e/src/test/resources/golden/show_create_table_db_table-3-7a9e67189d3d4151f23b12c22bde06b5
View file
Open in desktop
Empty file.
13 changes: 13 additions & 0 deletions ...e/src/test/resources/golden/show_create_table_db_table-4-b585371b624cbab2616a49f553a870a0
View file
Open in desktop
Original file line number Diff line number Diff line change
@@ -0,0 +1,13 @@
CREATE TABLE tmp_feng.tmp_showcrt(
key string,
value int)
ROW FORMAT SERDE
'org.apache.hadoop.hive.serde2.lazy.LazySimpleSerDe'
STORED AS INPUTFORMAT
'org.apache.hadoop.mapred.TextInputFormat'
OUTPUTFORMAT
'org.apache.hadoop.hive.ql.io.HiveIgnoreKeyTextOutputFormat'
LOCATION
'file:/tmp/sparkHiveWarehouse1280221975983654134/tmp_feng.db/tmp_showcrt'
TBLPROPERTIES (
'transient_lastDdlTime'='1407132107')
Empty file added ...e/src/test/resources/golden/show_create_table_db_table-5-964757b7e7f2a69fe36132c1a5712199
View file
Open in desktop
Empty file.
Empty file added ...e/src/test/resources/golden/show_create_table_db_table-6-ac09cf81e7e734cf10406f30b9fa566e
View file
Open in desktop
Empty file.
Empty file added .../src/test/resources/golden/show_create_table_delimited-0-97228478b9925f06726ceebb6571bf34
View file
Open in desktop
Empty file.
17 changes: 17 additions & 0 deletions .../src/test/resources/golden/show_create_table_delimited-1-2a91d52719cf4552ebeb867204552a26
View file
Open in desktop
Original file line number Diff line number Diff line change
@@ -0,0 +1,17 @@
CREATE TABLE tmp_showcrt1(
key int,
value string,
newvalue bigint)
ROW FORMAT DELIMITED
FIELDS TERMINATED BY ','
COLLECTION ITEMS TERMINATED BY '|'
MAP KEYS TERMINATED BY '%'
LINES TERMINATED BY '\n'
STORED AS INPUTFORMAT
'org.apache.hadoop.mapred.TextInputFormat'
OUTPUTFORMAT
'org.apache.hadoop.hive.ql.io.HiveIgnoreKeyTextOutputFormat'
LOCATION
'file:/tmp/tmp_showcrt1'
TBLPROPERTIES (
'transient_lastDdlTime'='1407132730')
Empty file added .../src/test/resources/golden/show_create_table_delimited-2-259d978ed9543204c8b9c25b6e25b0de
View file
Open in desktop
Empty file.
Empty file added ...rc/test/resources/golden/show_create_table_partitioned-0-4be9a3b1ff0840786a1f001cba170a0c
View file
Open in desktop
Empty file.
16 changes: 16 additions & 0 deletions ...rc/test/resources/golden/show_create_table_partitioned-1-2a91d52719cf4552ebeb867204552a26
View file
Open in desktop
Original file line number Diff line number Diff line change
@@ -0,0 +1,16 @@
CREATE EXTERNAL TABLE tmp_showcrt1(
key string,
newvalue boolean COMMENT 'a new value')
COMMENT 'temporary table'
PARTITIONED BY (
value bigint COMMENT 'some value')
ROW FORMAT SERDE
'org.apache.hadoop.hive.serde2.lazy.LazySimpleSerDe'
STORED AS INPUTFORMAT
'org.apache.hadoop.mapred.TextInputFormat'
OUTPUTFORMAT
'org.apache.hadoop.hive.ql.io.HiveIgnoreKeyTextOutputFormat'
LOCATION
'file:/tmp/sparkHiveWarehouse1280221975983654134/tmp_showcrt1'
TBLPROPERTIES (
'transient_lastDdlTime'='1407132112')
Empty file added ...rc/test/resources/golden/show_create_table_partitioned-2-259d978ed9543204c8b9c25b6e25b0de
View file
Open in desktop
Empty file.
Empty file added ...hive/src/test/resources/golden/show_create_table_serde-0-33f15d91810b75ee05c7b9dea0abb01c
View file
Open in desktop
Empty file.
15 changes: 15 additions & 0 deletions ...hive/src/test/resources/golden/show_create_table_serde-1-2a91d52719cf4552ebeb867204552a26
View file
Open in desktop
Original file line number Diff line number Diff line change
@@ -0,0 +1,15 @@
CREATE TABLE tmp_showcrt1(
key int,
value string,
newvalue bigint)
COMMENT 'temporary table'
ROW FORMAT SERDE
'org.apache.hadoop.hive.serde2.columnar.ColumnarSerDe'
STORED AS INPUTFORMAT
'org.apache.hadoop.hive.ql.io.RCFileInputFormat'
OUTPUTFORMAT
'org.apache.hadoop.hive.ql.io.RCFileOutputFormat'
LOCATION
'file:/tmp/sparkHiveWarehouse1280221975983654134/tmp_showcrt1'
TBLPROPERTIES (
'transient_lastDdlTime'='1407132115')
Empty file added ...hive/src/test/resources/golden/show_create_table_serde-2-259d978ed9543204c8b9c25b6e25b0de
View file
Open in desktop
Empty file.
Empty file added ...hive/src/test/resources/golden/show_create_table_serde-3-fd12b3e0fe30f5d71c67676791b4a33b
View file
Open in desktop
Empty file.
14 changes: 14 additions & 0 deletions ...hive/src/test/resources/golden/show_create_table_serde-4-2a91d52719cf4552ebeb867204552a26
View file
Open in desktop
Original file line number Diff line number Diff line change
@@ -0,0 +1,14 @@
CREATE EXTERNAL TABLE tmp_showcrt1(
key string,
value boolean)
ROW FORMAT SERDE
'org.apache.hadoop.hive.serde2.columnar.ColumnarSerDe'
STORED BY
'org.apache.hadoop.hive.ql.metadata.DefaultStorageHandler'
WITH SERDEPROPERTIES (
'serialization.format'='$',
'field.delim'=',')
LOCATION
'file:/tmp/sparkHiveWarehouse1280221975983654134/tmp_showcrt1'
TBLPROPERTIES (
'transient_lastDdlTime'='1407132115')
Empty file added ...hive/src/test/resources/golden/show_create_table_serde-5-259d978ed9543204c8b9c25b6e25b0de
View file
Open in desktop
Empty file.
Empty file added sql/hive/src/test/resources/golden/show_create_table_view-0-ecef6821e4e9212e553ca38142fd0250
View file
Open in desktop
Empty file.
1 change: 1 addition & 0 deletions sql/hive/src/test/resources/golden/show_create_table_view-1-1e931ea3fa6065107859ffbb29bb0ed7
View file
Open in desktop
Original file line number Diff line number Diff line change
@@ -0,0 +1 @@
CREATE VIEW tmp_copy_src AS SELECT `src`.`key`, `src`.`value` FROM `default`.`src`
Empty file added sql/hive/src/test/resources/golden/show_create_table_view-2-ed97e9e56d95c5b3db57485cba5ad17f
View file
Open in desktop
Empty file.
1 change: 1 addition & 0 deletions sql/hive/src/test/scala/org/apache/spark/sql/hive/execution/HiveComparisonTest.scala
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -161,6 +161,7 @@ abstract class HiveComparisonTest
"transient_lastDdlTime",
"grantTime",
"lastUpdateTime",
"last_modified_by",
"last_modified_time",
"Owner:",
// The following are hive specific schema parameters which we do not need to match exactly.
Expand Down