Skip to content

[SPARK-29379][SQL]SHOW FUNCTIONS show '!=', '<>' , 'between', 'case' #26053

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Closed
AngersZhuuuu wants to merge 14 commits into from
Closed

[SPARK-29379][SQL]SHOW FUNCTIONS show '!=', '<>' , 'between', 'case' #26053

Show file tree
Hide file tree
Changes from 4 commits
Commits
Show all changes
14 commits
Select commit Hold shift + click to select a range
911007a
fix show functions
AngersZhuuuu Oct 8, 2019
ba22b62
fix bug
AngersZhuuuu Oct 8, 2019
074ee0e
fit SYSTEM/USER
AngersZhuuuu Oct 8, 2019
9b8d63e
fix UT
AngersZhuuuu Oct 8, 2019
b066088
follow comment
AngersZhuuuu Oct 9, 2019
3d6c85d
follow comment
AngersZhuuuu Oct 16, 2019
85556a6
follow comment
AngersZhuuuu Oct 16, 2019
22b3487
follow comment
AngersZhuuuu Oct 16, 2019
60cd8a8
Update SQLQuerySuite.scala
AngersZhuuuu Oct 16, 2019
522193c
Revert "Update SQLQuerySuite.scala"
AngersZhuuuu Oct 16, 2019
a285290
Update SQLQuerySuite.scala
AngersZhuuuu Oct 16, 2019
9f68ead
remove empty line
AngersZhuuuu Oct 16, 2019
bc04f99
follow commment
AngersZhuuuu Oct 17, 2019
7ac4d16
scalastyle
AngersZhuuuu Oct 17, 2019
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
10 changes: 9 additions & 1 deletion sql/core/src/main/scala/org/apache/spark/sql/execution/command/functions.scala
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -24,6 +24,7 @@ import org.apache.spark.sql.catalyst.FunctionIdentifier
import org.apache.spark.sql.catalyst.analysis.{FunctionRegistry, NoSuchFunctionException}
import org.apache.spark.sql.catalyst.catalog.{CatalogFunction, FunctionResource}
import org.apache.spark.sql.catalyst.expressions.{Attribute, ExpressionInfo}
import org.apache.spark.sql.catalyst.util.StringUtils
import org.apache.spark.sql.types.{StringType, StructField, StructType}


Expand Down Expand Up @@ -222,6 +223,13 @@ case class ShowFunctionsCommand(
case (f, "USER") if showUserFunctions => f.unquotedString
case (f, "SYSTEM") if showSystemFunctions => f.unquotedString
}
functionNames.sorted.map(Row(_))
if (showSystemFunctions) {
(functionNames ++
StringUtils.filterPattern(Seq("!=", "<>", "between", "case"), pattern.getOrElse("*")))

@cloud-fan cloud-fan Oct 16, 2019

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

shall we add comment like

spark/sql/core/src/main/scala/org/apache/spark/sql/execution/command/functions.scala

Line 117 in 9b8d63e

// Hard code "<>", "!=", "between", and "case" for now as there is no corresponding functions.

.sorted.map(Row(_))
} else {
functionNames.sorted.map(Row(_))
}

}
}
7 changes: 5 additions & 2 deletions sql/core/src/test/scala/org/apache/spark/sql/SQLQuerySuite.scala
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -80,13 +80,16 @@ class SQLQuerySuite extends QueryTest with SharedSparkSession {

createFunction(functions)

checkAnswer(sql("SHOW functions"), getFunctions("*"))
checkAnswer(sql("SHOW functions"), (getFunctions("*") ++
Seq(Row("!="), Row("<>"), Row("between"), Row("case"))))

@cloud-fan cloud-fan Oct 8, 2019

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

nit: shall we put this code in getFunctions?

@AngersZhuuuu AngersZhuuuu Oct 8, 2019

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

nit: shall we put this code in getFunctions?

Good ideal, I will try this.

assert(sql("SHOW functions").collect().size > 200)

Seq("^c*", "*e$", "log*", "*date*").foreach { pattern =>
// For the pattern part, only '*' and '|' are allowed as wildcards.
// For '*', we need to replace it to '.*'.
checkAnswer(sql(s"SHOW FUNCTIONS '$pattern'"), getFunctions(pattern))
checkAnswer(sql(s"SHOW FUNCTIONS '$pattern'"),
getFunctions(pattern) ++
StringUtils.filterPattern(Seq("!=", "<>", "between", "case"), pattern).map(Row(_)))
}
dropFunction(functions)
}
Expand Down
12 changes: 6 additions & 6 deletions sql/core/src/test/scala/org/apache/spark/sql/execution/command/DDLSuite.scala
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -2065,14 +2065,14 @@ abstract class DDLSuite extends QueryTest with SQLTestUtils {
test("show functions") {
withUserDefinedFunction("add_one" -> true) {
val numFunctions = FunctionRegistry.functionSet.size.toLong

@cloud-fan cloud-fan Oct 8, 2019

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

shall we update it with val numFunctions = ... + 4?

assert(sql("show functions").count() === numFunctions)
assert(sql("show system functions").count() === numFunctions)
assert(sql("show all functions").count() === numFunctions)
assert(sql("show functions").count() === numFunctions + 4L)
assert(sql("show system functions").count() === numFunctions + 4L)
assert(sql("show all functions").count() === numFunctions + 4L)
assert(sql("show user functions").count() === 0L)
spark.udf.register("add_one", (x: Long) => x + 1)
assert(sql("show functions").count() === numFunctions + 1L)
assert(sql("show system functions").count() === numFunctions)
assert(sql("show all functions").count() === numFunctions + 1L)
assert(sql("show functions").count() === numFunctions + 5L)
assert(sql("show system functions").count() === numFunctions + 4L)
assert(sql("show all functions").count() === numFunctions + 5L)
assert(sql("show user functions").count() === 1L)
}
}
Expand Down
2 changes: 1 addition & 1 deletion sql/hive/src/test/scala/org/apache/spark/sql/hive/execution/HiveUDFSuite.scala
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -563,7 +563,7 @@ class HiveUDFSuite extends QueryTest with TestHiveSingleton with SQLTestUtils {
checkAnswer(
sql("SELECT testUDFToListInt(s) FROM inputTable"),
Seq(Row(Seq(1, 2, 3))))
assert(sql("show functions").count() == numFunc + 1)
assert(sql("show functions").count() == numFunc + 5)

@cloud-fan cloud-fan Oct 17, 2019

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

ditto

@AngersZhuuuu AngersZhuuuu Oct 18, 2019

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

ditto

Could you help to trigger retest ?

assert(spark.catalog.listFunctions().count() == numFunc + 1)
}
}
Expand Down
5 changes: 5 additions & 0 deletions sql/hive/src/test/scala/org/apache/spark/sql/hive/execution/SQLQuerySuite.scala
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -192,6 +192,11 @@ class SQLQuerySuite extends QueryTest with SQLTestUtils with TestHiveSingleton {
allBuiltinFunctions.foreach { f =>
assert(allFunctions.contains(f))
}

Seq("!=", "<>", "between", "case").foreach { f =>
assert(allFunctions.contains(f))
}

withTempDatabase { db =>
def createFunction(names: Seq[String]): Unit = {
names.foreach { name =>
Expand Down