Skip to content

[SPARK-33426][SQL][TESTS] Unify Hive SHOW TABLES tests #30340

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Closed
MaxGekk wants to merge 7 commits into from
Closed

[SPARK-33426][SQL][TESTS] Unify Hive SHOW TABLES tests #30340

Show file tree
Hide file tree
Changes from 2 commits
Commits
Show all changes
7 commits
Select commit Hold shift + click to select a range
65fd9c3
Add Hive ShowTablesSuite
MaxGekk Nov 11, 2020
4fb6ddc
Set the default namespace
MaxGekk Nov 11, 2020
8dfd9c7
Move Hive tests to the common trait
MaxGekk Nov 12, 2020
c856b89
Revert back ns2 in a test
MaxGekk Nov 12, 2020
fb74566
Use checkAnswer() in runShowTablesSql()
MaxGekk Nov 12, 2020
bd0f785
Avoid the hask with overriding the beforeAll() method
MaxGekk Nov 12, 2020
7eab687
Refactoring: rename classes/traits
MaxGekk Nov 12, 2020
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
1 change: 1 addition & 0 deletions sql/core/src/test/scala/org/apache/spark/sql/execution/command/ShowTablesSuite.scala
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -116,6 +116,7 @@ trait ShowTablesSuite extends SharedSparkSession {
// Update the current namespace to match "ns.tbl".
sql(s"USE $catalog.ns")
runShowTablesSql("SHOW TABLES", Seq(ShowRow("ns", "table", false)))
defaultNamespace.headOption.foreach(ns => sql(s"USE $ns"))
}
}
}
Expand Down
22 changes: 0 additions & 22 deletions sql/hive/src/test/scala/org/apache/spark/sql/hive/execution/HiveCommandSuite.scala
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -95,28 +95,6 @@ class HiveCommandSuite extends QueryTest with SQLTestUtils with TestHiveSingleto
}
}

test("show tables") {
withTable("show1a", "show2b") {
sql("CREATE TABLE show1a(c1 int)")
sql("CREATE TABLE show2b(c2 int)")
checkAnswer(
sql("SHOW TABLES IN default 'show1*'"),
Row("default", "show1a", false) :: Nil)
checkAnswer(
sql("SHOW TABLES IN default 'show1*|show2*'"),
Row("default", "show1a", false) ::
Row("default", "show2b", false) :: Nil)
checkAnswer(
sql("SHOW TABLES 'show1*|show2*'"),
Row("default", "show1a", false) ::
Row("default", "show2b", false) :: Nil)
assert(
sql("SHOW TABLES").count() >= 2)
assert(
sql("SHOW TABLES IN default").count() >= 2)
}
}

test("show views") {
withView("show1a", "show2b", "global_temp.temp1", "temp2") {
sql("CREATE VIEW show1a AS SELECT 1 AS id")
Expand Down
49 changes: 49 additions & 0 deletions sql/hive/src/test/scala/org/apache/spark/sql/hive/execution/command/ShowTablesSuite.scala
View file
Open in desktop
Original file line number Diff line number Diff line change
@@ -0,0 +1,49 @@
/*
* Licensed to the Apache Software Foundation (ASF) under one or more
* contributor license agreements. See the NOTICE file distributed with
* this work for additional information regarding copyright ownership.
* The ASF licenses this file to You under the Apache License, Version 2.0
* (the "License"); you may not use this file except in compliance with
* the License. You may obtain a copy of the License at
*
* http://www.apache.org/licenses/LICENSE-2.0
*
* Unless required by applicable law or agreed to in writing, software
* distributed under the License is distributed on an "AS IS" BASIS,
* WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
* See the License for the specific language governing permissions and
* limitations under the License.
*/

package org.apache.spark.sql.hive.execution.command

import org.apache.spark.sql.SparkSession
import org.apache.spark.sql.execution.command.v1
import org.apache.spark.sql.hive.test.TestHive

class ShowTablesSuite extends v1.ShowTablesSuite {
override def version: String = "Hive V1"
override def defaultUsing: String = "USING HIVE"
override protected val spark: SparkSession = TestHive.sparkSession
Copy link
Member Author

@MaxGekk MaxGekk Nov 12, 2020

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

We could take this from TestHiveSingleton but when I mix the trait:

[error] /Users/maximgekk/proj/show-tables-hive-tests/sql/hive/src/test/scala/org/apache/spark/sql/hive/execution/command/ShowTablesSuite.scala:24:7: class ShowTablesSuite inherits conflicting members:
[error]   method spark in trait SharedSparkSessionBase of type => org.apache.spark.sql.SparkSession  and
[error]   value spark in trait TestHiveSingleton of type org.apache.spark.sql.SparkSession
[error] (Note: this can be resolved by declaring an override in class ShowTablesSuite.)
[error] class ShowTablesSuite extends v1.ShowTablesSuite with TestHiveSingleton {
[error]       ^
[error] one error found

protected override def beforeAll(): Unit = {}
Copy link
Member Author

@MaxGekk MaxGekk Nov 12, 2020

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I had to override this otherwise tests failed with:

 SHOW TABLES Hive V1: change current catalog and namespace with USE statements *** FAILED *** (17 milliseconds)
[info]   java.lang.IllegalStateException: LiveListenerBus is stopped.
[info]   at org.apache.spark.scheduler.LiveListenerBus.addToQueue(LiveListenerBus.scala:97)
[info]   at org.apache.spark.scheduler.LiveListenerBus.addToStatusQueue(LiveListenerBus.scala:80)
...
[info]   Cause: java.lang.IllegalStateException: LiveListenerBus is stopped.
[info]   at org.apache.spark.scheduler.LiveListenerBus.addToQueue(LiveListenerBus.scala:97)
[info]   at org.apache.spark.scheduler.LiveListenerBus.addToStatusQueue(LiveListenerBus.scala:80)
[info]   at org.apache.spark.sql.internal.SharedState.<init>(SharedState.scala:103)
[info]   at org.apache.spark.sql.hive.test.TestHiveSharedState.<init>(TestHive.scala:99)
[info]   at org.apache.spark.sql.hive.test.TestHiveSparkSession.$anonfun$sharedState$1(TestHive.scala:222)
[info]   at scala.Option.getOrElse(Option.scala:189)
[info]   at org.apache.spark.sql.hive.test.TestHiveSparkSession.sharedState$lzycompute(TestHive.scala:222)
[info]   at org.apache.spark.sql.hive.test.TestHiveSparkSession.sharedState(TestHive.scala:221)
[info]   at org.apache.spark.sql.hive.test.TestHiveSparkSession.sharedState(TestHive.scala:174)
[info]   at org.apache.spark.sql.internal.BaseSessionStateBuilder.build(BaseSessionStateBuilder.scala:333)
[info]   at org.apache.spark.sql.hive.test.TestHiveSparkSession.sessionState$lzycompute(TestHive.scala:227)
[info]   at org.apache.spark.sql.hive.test.TestHiveSparkSession.sessionState(TestHive.scala:226)
[info]   at org.apache.spark.sql.hive.test.TestHiveSparkSession.$anonfun$sql$1(TestHive.scala:240)
[info]   at org.apache.spark.sql.SparkSession.withActive(SparkSession.scala:768)
[info]   at org.apache.spark.sql.hive.test.TestHiveSparkSession.sql(TestHive.scala:239)


test("show Hive tables") {
Copy link
Contributor

@cloud-fan cloud-fan Nov 12, 2020

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

is it specific to hive tables? if not we should put it in the base trait.

Copy link
Member Author

@MaxGekk MaxGekk Nov 12, 2020

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

They are not. The difference is probably in omitting explicit USING HIVE.

Copy link
Contributor

@cloud-fan cloud-fan Nov 12, 2020

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

that's not something SHOW TABLES should care about. Let's simply add a test in the base trait for complex patterns and this suite just extends the base trait.

withTable("show1a", "show2b") {
sql("CREATE TABLE show1a(c1 int)")
sql("CREATE TABLE show2b(c2 int)")
runShowTablesSql(
"SHOW TABLES IN default 'show1*'",
ShowRow("default", "show1a", false) :: Nil)
runShowTablesSql(
"SHOW TABLES IN default 'show1*|show2*'",
Copy link
Contributor

@cloud-fan cloud-fan Nov 12, 2020

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I think this is the only place that improves test coverage. Maybe we should just add one more test in the base trait to test complicated patterns.

Copy link
Member Author

@MaxGekk MaxGekk Nov 12, 2020

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Yeh, it seems only this test case brings some benefits. Let me move it the common trait, and remove all those duplicate checks.

ShowRow("default", "show1a", false) ::
ShowRow("default", "show2b", false) :: Nil)
runShowTablesSql(
"SHOW TABLES 'show1*|show2*'",
ShowRow("default", "show1a", false) ::
ShowRow("default", "show2b", false) :: Nil)
assert(sql("SHOW TABLES").count() >= 2)
assert(sql("SHOW TABLES IN default").count() >= 2)
}
}
}