[SPARK-33382][SQL][TESTS] Unify datasource v1 and v2 SHOW TABLES tests #30287

MaxGekk · 2020-11-07T18:40:45Z

What changes were proposed in this pull request?

In the PR, I propose to gather common SHOW TABLES tests into one trait org.apache.spark.sql.execution.command.ShowTablesSuite, and put datasource specific tests to the v1.ShowTablesSuite and v2.ShowTablesSuite. Also tests for parsing SHOW TABLES are extracted to ShowTablesParserSuite.

Why are the changes needed?

The unification will allow to run common SHOW TABLES tests for both DSv1 and DSv2
We can detect missing features and differences between DSv1 and DSv2 implementations.

Does this PR introduce any user-facing change?

No

How was this patch tested?

By running new test suites:

org.apache.spark.sql.execution.command.v1.ShowTablesSuite
org.apache.spark.sql.execution.command.v2.ShowTablesSuite
ShowTablesParserSuite

…ck to v1"

…atalog is set"

cloud-fan · 2020-11-11T05:26:35Z

thanks, merging to master!

### What changes were proposed in this pull request? 1. Create the separate test suite `org.apache.spark.sql.hive.execution.command.ShowTablesSuite`. 2. Re-use V1 SHOW TABLES tests added by #30287 in the Hive test suites. 3. Add new test case for the pattern `'table_name_1*|table_name_2*'` in the common test suite. ### Why are the changes needed? To test V1 + common SHOW TABLES tests in Hive. ### Does this PR introduce _any_ user-facing change? No ### How was this patch tested? By running v1/v2 and Hive v1 `ShowTablesSuite`: ``` $ build/sbt -Phive-2.3 -Phive-thriftserver "test:testOnly *ShowTablesSuite" ``` Closes #30340 from MaxGekk/show-tables-hive-tests. Authored-by: Max Gekk <[email protected]> Signed-off-by: Wenchen Fan <[email protected]>

### What changes were proposed in this pull request? 1. Move `SHOW PARTITIONS` parsing tests to `ShowPartitionsParserSuite` 2. Place Hive tests for `SHOW PARTITIONS` from `HiveCommandSuite` to the base test suite `v1.ShowPartitionsSuiteBase`. This will allow to run the tests w/ and w/o Hive. The changes follow the approach of #30287. ### Why are the changes needed? - The unification will allow to run common `SHOW PARTITIONS` tests for both DSv1 and Hive DSv1, DSv2 - We can detect missing features and differences between DSv1 and DSv2 implementations. ### Does this PR introduce _any_ user-facing change? No ### How was this patch tested? By running: - new test suites `build/sbt -Phive-2.3 -Phive-thriftserver "test:testOnly *ShowPartitionsSuite"` - and old one `build/sbt -Phive-2.3 -Phive-thriftserver "test:testOnly org.apache.spark.sql.hive.execution.HiveCommandSuite"` Closes #30377 from MaxGekk/unify-dsv1_v2-show-partitions-tests. Authored-by: Max Gekk <[email protected]> Signed-off-by: Wenchen Fan <[email protected]>

…rceV2SQLSuite.scala ### What changes were proposed in this pull request? After #30287, `runShowTablesSql()` in `DataSourceV2SQLSuite.scala` is no longer used. This PR removes the unused method. ### Why are the changes needed? To remove unused method. ### Does this PR introduce _any_ user-facing change? No ### How was this patch tested? Existing test. Closes #31022 from imback82/33382-followup. Authored-by: Terry Kim <[email protected]> Signed-off-by: Dongjoon Hyun <[email protected]>

### What changes were proposed in this pull request? 1. Move `DESCRIBE DATABASE/NAMESPACE` parsing tests to `DescribeNamespaceParserSuite`. 2. Put common `DESCRIBE NAMESPACE` tests into one trait `org.apache.spark.sql.execution.command. DescribeNamespaceSuiteBase`, and put datasource specific tests to the `v1.DescribeNamespaceSuite` and `v2.DescribeNamespaceSuite`. The changes follow the approach of #30287. ### Why are the changes needed? 1. The unification will allow to run common `DESCRIBE NAMESPACE` tests for both DSv1/Hive DSv1 and DSv2 2. We can detect missing features and differences between DSv1 and DSv2 implementations. ### Does this PR introduce _any_ user-facing change? No ### How was this patch tested? Existing unit tests and new tests. Closes #34305 from imback82/migrate_desc_namespace. Lead-authored-by: Terry Kim <[email protected]> Co-authored-by: Wenchen Fan <[email protected]> Signed-off-by: Wenchen Fan <[email protected]>

### What changes were proposed in this pull request? 1. Move SHOW TBLPROPERTIES parsing tests to `ShowTblPropertiesParserSuite`. 2. Define the class `command.ShowTblPropertiesSuiteBase` that is parent of `v1.ShowTblPropertiesSuiteBase` and `v2.ShowTblPropertiesSuite`. 3. Define the class `v1.ShowTblPropertiesSuiteBase` that is parent of `v1.ShowTblPropertiesSuite` and `hive.execution.command.ShowTblPropertiesSuite`. 4. move test case from `DataSourceV2SQLSessionCatalogSuite` to `command.ShowTblPropertiesSuiteBase` to run for V2/V1/HIVE V1 5. move some test case from `HiveCommandSuite` to `command.ShowTblPropertiesSuiteBase` 6. move some test case from `HiveCommandSuite` to `v1.ShowTblPropertiesSuiteBase ` to run for V1/HIVE V1 7. Add two test case for `v1.ShowTblPropertiesSuite ` for view. The changes follow the approach of [#30287](#30287) [#34305](#34305) ### Why are the changes needed? 1. The unification will allow to run common `SHOW TBLPROPERTIES` tests for both DSv1/Hive DSv1 and DSv2 2. We can detect missing features and differences between DSv1 and DSv2 implementations. ### Does this PR introduce _any_ user-facing change? No ### How was this patch tested? By running v1/v2 and Hive v1 ShowTblPropertiesSuite: $ build/sbt -Phive-2.3 -Phive-thriftserver "test:testOnly *ShowTblPropertiesSuite" Closes #34476 from Peng-Lei/SPARK-37195. Authored-by: PengLei <[email protected]> Signed-off-by: Wenchen Fan <[email protected]>

…tests ### What changes were proposed in this pull request? 1. Move `ALTER NAMESPACE ... SET LOCATION` parsing tests to `AlterNamespaceSetLocationParserSuite`. 2. Put common `ALTER NAMESPACE ... SET LOCATION` tests into one trait `org.apache.spark.sql.execution.command.AlterNamespaceSetLocationSuiteBase`, and put datasource specific tests to the `v1.AlterNamespaceSetLocationSuite` and `v2.AlterNamespaceSetLocationSuite`. The changes follow the approach of #30287. ### Why are the changes needed? 1. The unification will allow to run common `ALTER NAMESPACE ... SET LOCATION` tests for both DSv1/Hive DSv1 and DSv2 2. We can detect missing features and differences between DSv1 and DSv2 implementations. ### Does this PR introduce _any_ user-facing change? No ### How was this patch tested? Existing unit tests and new tests. Closes #34610 from imback82/alter_namespace_set_loation_tests2. Authored-by: Terry Kim <[email protected]> Signed-off-by: Wenchen Fan <[email protected]>

### What changes were proposed in this pull request? 1. Move the `DROP NAMESPACE` parsing tests from `DDLParserSuite` to `DropNamespaceParserSuite` 2. Define the class `command.DropNamespaceSuiteBase` that is parent of `v1.DropNamespaceSuiteBase` and `v2.DropNamespaceSuite`. 3. Define the class `v1.DropNamespaceSuiteBase` that is parent of `v1.DropNamespaceSuite` and `hive.execution.command.DropNamespaceSuite`. 4. Place the v1 tests for `DROP NAMESPACE` from `DDLSuite` and v2 tests from `DataSourceV2SQLSuite` to the common trait `DropNamespaceSuiteBase`. 5. Move TODO test from `DescribeNamespaceSuiteBase` to `v1.DropNamespaceSuiteBase` and `v2.DropNamespaceSuite` The changes follow the approach of [#30287](#30287) ### Why are the changes needed? 1. The unification will allow to run common `DROP NAMESPACE` tests for both DSv1/Hive DSv1 and DSv2 2. We can detect missing features and differences between DSv1 and DSv2 implementations. ### Does this PR introduce _any_ user-facing change? No ### How was this patch tested? v1/v2 and Hive v1 DropNamespaceSuite: ```$ build/sbt -Phive-2.3 -Phive-thriftserver "test:testOnly *DropNamespaceSuite"``` Closes #34819 from dchvn/unify-drop-namespace-tests. Authored-by: dch nguyen <[email protected]> Signed-off-by: Wenchen Fan <[email protected]>

…S tests ### What changes were proposed in this pull request? 1. Move `ALTER NAMESPACE ... SET PROPERTIES` parsing tests to `AlterNamespaceSetPropertiesParserSuite`. 2. Put common `ALTER NAMESPACE ... SET PROPERTIES` tests into one trait `org.apache.spark.sql.execution.command.AlterNamespaceSetPropertiesSuiteBase`, and put datasource specific tests to the `v1.AlterNamespaceSetPropertiesSuite` and `v2.AlterNamespaceSetPropertiesSuite`. The changes follow the approach of #30287. ### Why are the changes needed? 1. The unification will allow to run common `ALTER NAMESPACE ... SET PROPERTIES` tests for both DSv1/Hive DSv1 and DSv2 2. We can detect missing features and differences between DSv1 and DSv2 implementations. ### Does this PR introduce _any_ user-facing change? No ### How was this patch tested? Existing unit tests and new tests. Closes #34842 from imback82/alter_namespace_set_properties_tests. Authored-by: Terry Kim <[email protected]> Signed-off-by: Wenchen Fan <[email protected]>

### What changes were proposed in this pull request? 1. Move `CREATE NAMESPACE` parsing tests to `CreateNamespaceParserSuite`. 2. Put common `CREATE NAMESPACE` tests into one trait `org.apache.spark.sql.execution.command.CreateNamespaceSuiteBase`, and put datasource specific tests to the `v1.CreateNamespaceSuite` and `v2.CreateNamespaceSuite`. The changes follow the approach of #30287. ### Why are the changes needed? 1. The unification will allow to run common `CREATE NAMESPACE` tests for both DSv1/Hive DSv1 and DSv2 2. We can detect missing features and differences between DSv1 and DSv2 implementations. ### Does this PR introduce _any_ user-facing change? No ### How was this patch tested? Existing unit tests and new tests. Closes #35093 from imback82/v2_create_namespace. Authored-by: Terry Kim <[email protected]> Signed-off-by: Wenchen Fan <[email protected]>

### What changes were proposed in this pull request? unify test case: 1. Move the testcase from `DDLParserSuite.scala` to `ShowCreateTableParserSuite.scala`. 2. Move the testcase from `DataSourceV2SQLSuite` to `v2.ShowCreateTableSuite`. If the test case also work with V1/Hive, then move to ShowCreateTableSuiteBase 3. Move the testcase from `sql/ShowCreateTableSuite.scala` to `v1.ShowCreateTableSuite` If the test case also work with Hive, then move to `v1.ShowCreateTableSuiteBase` If the test case also work with V2/Hive, then move to `ShowCreateTableSuiteBase` 4. Move the testcase from `HiveShowCreateTableSuite` to `hive.ShowCreateTableSuite` If the test case also work with V1/V2, then move to `ShowCreateTableSuiteBase` If the test case also work with V1, then move to `v1.ShowCreateTableSuiteBase` 5. Use `getShowCreateDDL` instead of `checkCreateTable` to check the result. fix diff behavior: 1. Add one space after `create table xxx` for the command `SHOW CREATE TABLE AS SERDE` to unify the output. 2. Add one space after `OPTIONS` for v2 command `SHOW CREATE TABLE` to unify the output. The changes follow the approach of [#30287](#30287) [#34305](#34305) ### Why are the changes needed? 1. The unification will allow to run common `SHOW CREATE TABLE` tests for both DSv1/Hive DSv1 and DSv2 2. We can detect missing features and differences between DSv1 and DSv2 implementations. ### Does this PR introduce _any_ user-facing change? No ### How was this patch tested? $ build/sbt -Phive-2.3 -Phive-thriftserver "test:testOnly *ShowCreateTableSuite" Closes #34719 from Peng-Lei/SPARK-37381. Authored-by: PengLei <[email protected]> Signed-off-by: Wenchen Fan <[email protected]>

### What changes were proposed in this pull request? unify test case: 1. Move the testcase from `DDLParserSuite.scala` to `ShowCreateTableParserSuite.scala`. 2. Move the testcase from `DataSourceV2SQLSuite` to `v2.ShowCreateTableSuite`. If the test case also work with V1/Hive, then move to ShowCreateTableSuiteBase 3. Move the testcase from `sql/ShowCreateTableSuite.scala` to `v1.ShowCreateTableSuite` If the test case also work with Hive, then move to `v1.ShowCreateTableSuiteBase` If the test case also work with V2/Hive, then move to `ShowCreateTableSuiteBase` 4. Move the testcase from `HiveShowCreateTableSuite` to `hive.ShowCreateTableSuite` If the test case also work with V1/V2, then move to `ShowCreateTableSuiteBase` If the test case also work with V1, then move to `v1.ShowCreateTableSuiteBase` 5. Use `getShowCreateDDL` instead of `checkCreateTable` to check the result. fix diff behavior: 1. Add one space after `create table xxx` for the command `SHOW CREATE TABLE AS SERDE` to unify the output. 2. Add one space after `OPTIONS` for v2 command `SHOW CREATE TABLE` to unify the output. The changes follow the approach of [apache#30287](apache#30287) [apache#34305](apache#34305) ### Why are the changes needed? 1. The unification will allow to run common `SHOW CREATE TABLE` tests for both DSv1/Hive DSv1 and DSv2 2. We can detect missing features and differences between DSv1 and DSv2 implementations. ### Does this PR introduce _any_ user-facing change? No ### How was this patch tested? $ build/sbt -Phive-2.3 -Phive-thriftserver "test:testOnly *ShowCreateTableSuite" Closes apache#34719 from Peng-Lei/SPARK-37381. Authored-by: PengLei <[email protected]> Signed-off-by: Wenchen Fan <[email protected]>

### What changes were proposed in this pull request? 1. Move `DESCRIBE TABLE` parsing tests to `DescribeRelationParserSuite`. 2. Put common `DESCRIBE TABLE` tests into one trait `org.apache.spark.sql.execution.command.DescribeTableSuiteBase`, and put datasource specific tests to the `v1.DescribeTableSuite` and `v2.DescribeTableSuite`. The changes follow the approach of #30287. Closes #36671. ### Why are the changes needed? 1. The unification will allow to run common `DESCRIBE TABLE` tests for both DSv1/Hive DSv1 and DSv2 2. We can detect missing features and differences between DSv1 and DSv2 implementations. ### Does this PR introduce _any_ user-facing change? No. ### How was this patch tested? By running the modified test suites: ``` $ build/sbt "testOnly *DDLParserSuite" $ build/sbt -Phive-2.3 -Phive-thriftserver "test:testOnly *HiveDDLSuite" ``` and new test suites: ``` $ build/sbt "sql/test:testOnly *DescribeTableParserSuite" $ build/sbt -Phive-2.3 -Phive-thriftserver "test:testOnly *DescribeTableSuite" ``` Closes #36912 from MaxGekk/unify-describe-table-tests-4. Authored-by: Max Gekk <[email protected]> Signed-off-by: Max Gekk <[email protected]>

MaxGekk added 30 commits November 5, 2020 22:27

Draft of ShowTablesSuite

58f4134

ShowTablesSuite for v2

7b8746f

test "show table in a not existing namespace"

4c0d2b8

test "show an existing table in V1 catalog"

f1aa46c

Move "ShowTables: using v2 catalog" to v2/ShowTablesSuite

3a85ac3

Move v2 pattern tests

0534480

Move more v1 tests

7c12702

Move more v2 tests

327179e

Move "namespace not specified and default v2 catalog not set - fallba…

c07e08b

…ck to v1"

Move "SHOW TABLE EXTENDED valid v1"

7738473

Move "SHOW TABLE EXTENDED not valid v1 database"

91521ce

Introduce ShowRow

79b0fd0

Move "show tables with a pattern" to common trait

e282bb5

Remove duplicate tests

70a578d

Move "SHOW TABLE EXTENDED for default" to the common trait

d21dd5e

Change default USING in v2

48509fb

Add a comment for "show tables in nested namespaces"

b5a6187

Add comment for "show table in a not existing namespace"

bdc249a

Remove duplicate test from v1

ef2b536

Move SHOW VIEWS test back to DataSourceV2SQLSuite

122faa5

Add comment for the test "using v2 catalog with empty namespace"

3016f69

Add comment for the test "namespace is not specified and default v2 c…

53d0c77

…atalog is set"

Add comment for the test "SHOW TABLE EXTENDED not valid v1 database"

ebfa670

Minor: fix test titles

dbf8e76

Move a show view test back to DataSourceV2SQLSuite

1829685

Fix test title

7d3b4b2

Refactoring

80eb4a8

Minor changes

f0152e6

Using foo -> Using _

8e55686

Remove beforeAll()

4364a02

cloud-fan closed this in 1e2eeda Nov 11, 2020

MaxGekk mentioned this pull request Nov 11, 2020

[SPARK-33426][SQL][TESTS] Unify Hive SHOW TABLES tests #30340

Closed

MaxGekk mentioned this pull request Nov 14, 2020

[SPARK-33453][SQL][TESTS] Unify v1 and v2 SHOW PARTITIONS tests #30377

Closed

MaxGekk mentioned this pull request Nov 25, 2020

[WIP][SPARK-33558][SQL][TESTS] Unify v1 and v2 ALTER TABLE .. PARTITION tests #30499

Closed

imback82 mentioned this pull request Jan 5, 2021

[SPARK-34001][SQL][TESTS] Remove unused runShowTablesSql() in DataSourceV2SQLSuite.scala #31022

Closed

MaxGekk deleted the unify-dsv1_v2-tests branch February 19, 2021 15:03

imback82 mentioned this pull request Oct 17, 2021

[SPARK-37031][SQL][TESTS] Unify v1 and v2 DESCRIBE NAMESPACE tests #34305

Closed

Peng-Lei mentioned this pull request Nov 3, 2021

[SPARK-37195][SQL][TESTS] Unify v1 and v2 SHOW TBLPROPERTIES tests #34476

Closed

imback82 mentioned this pull request Nov 15, 2021

[SPARK-34332][SQL][TEST] Unify v1 and v2 ALTER NAMESPACE .. SET LOCATION tests #34610

Closed

Peng-Lei mentioned this pull request Nov 26, 2021

[SPARK-37381][SQL] Unify v1 and v2 SHOW CREATE TABLE tests #34719

Closed

dchvn mentioned this pull request Dec 6, 2021

[SPARK-37478][SQL][TESTS] Unify v1 and v2 DROP NAMESPACE tests #34819

Closed

imback82 mentioned this pull request Dec 9, 2021

[SPARK-37590][SQL][TEST] Unify v1 and v2 ALTER TABLE .. SET PROPERTIES tests #34842

Closed

imback82 mentioned this pull request Jan 4, 2022

[SPARK-37804][SQL][TEST] Unify v1 and v2 CREATE NAMESPACE tests #35093

Closed

imback82 mentioned this pull request Jan 21, 2022

[WIP][SPARK-37888][SQL][TESTS] Unify v1 and v2 DESCRIBE TABLE tests #35265

Closed

MaxGekk mentioned this pull request May 25, 2022

[WIP][SPARK-37888][SQL][TESTS] Unify v1 and v2 DESCRIBE TABLE tests #36671

Closed

MaxGekk mentioned this pull request Jun 20, 2022

[SPARK-37888][SQL][TESTS] Unify v1 and v2 DESCRIBE TABLE tests #36912

Closed

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Uh oh!

[SPARK-33382][SQL][TESTS] Unify datasource v1 and v2 SHOW TABLES tests #30287

[SPARK-33382][SQL][TESTS] Unify datasource v1 and v2 SHOW TABLES tests #30287

Uh oh!

MaxGekk commented Nov 7, 2020 •

edited

Loading

Uh oh!

cloud-fan commented Nov 11, 2020

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

4 participants

[SPARK-33382][SQL][TESTS] Unify datasource v1 and v2 SHOW TABLES tests #30287

[SPARK-33382][SQL][TESTS] Unify datasource v1 and v2 SHOW TABLES tests #30287

Uh oh!

Conversation

MaxGekk commented Nov 7, 2020 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

What changes were proposed in this pull request?

Why are the changes needed?

Does this PR introduce any user-facing change?

How was this patch tested?

Uh oh!

cloud-fan commented Nov 11, 2020

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

4 participants

MaxGekk commented Nov 7, 2020 •

edited

Loading