[SPARK-30352][SQL] DataSourceV2: Add CURRENT_CATALOG function #27006

yaooqinn · 2019-12-25T09:13:08Z

What changes were proposed in this pull request?

As we support multiple catalogs with DataSourceV2, we may need the CURRENT_CATALOG value expression from the SQL standard.

CURRENT_CATALOG is a general value specification in the SQL Standard, described as:

The value specified by CURRENT_CATALOG is the character string that represents the current default catalog name.

Why are the changes needed?

improve catalog v2 with ANSI SQL standard.

Does this PR introduce any user-facing change?

yes, add a new function current_catalog() to point the current active catalog

How was this patch tested?

add ut

SparkQA · 2019-12-25T11:04:52Z

Test build #115761 has finished for PR 27006 at commit 180fe0a.

This patch fails Spark unit tests.
This patch merges cleanly.
This patch adds the following public classes (experimental):
case class CurrentCatalog() extends LeafExpression with Unevaluable
case class GetCurrentCatalog(catalogManager: CatalogManager) extends Rule[LogicalPlan]

yaooqinn · 2019-12-25T11:07:00Z

retest this please

SparkQA · 2019-12-25T13:08:43Z

Test build #115763 has finished for PR 27006 at commit 180fe0a.

This patch fails Spark unit tests.
This patch merges cleanly.
This patch adds the following public classes (experimental):
case class CurrentCatalog() extends LeafExpression with Unevaluable
case class GetCurrentCatalog(catalogManager: CatalogManager) extends Rule[LogicalPlan]

HyukjinKwon · 2019-12-27T03:58:15Z

retest this please

SparkQA · 2019-12-27T08:05:01Z

Test build #115840 has finished for PR 27006 at commit 180fe0a.

This patch fails due to an unknown error code, -9.
This patch merges cleanly.
This patch adds the following public classes (experimental):
case class CurrentCatalog() extends LeafExpression with Unevaluable
case class GetCurrentCatalog(catalogManager: CatalogManager) extends Rule[LogicalPlan]

yaooqinn · 2019-12-27T08:14:59Z

retest this please, and thanks @HyukjinKwon

SparkQA · 2019-12-27T12:12:26Z

Test build #115850 has finished for PR 27006 at commit 180fe0a.

This patch passes all tests.
This patch merges cleanly.
This patch adds the following public classes (experimental):
case class CurrentCatalog() extends LeafExpression with Unevaluable
case class GetCurrentCatalog(catalogManager: CatalogManager) extends Rule[LogicalPlan]

SparkQA · 2020-04-27T07:05:02Z

Test build #121870 has finished for PR 27006 at commit 180fe0a.

This patch fails due to an unknown error code, -9.
This patch merges cleanly.
This patch adds the following public classes (experimental):
case class CurrentCatalog() extends LeafExpression with Unevaluable
case class GetCurrentCatalog(catalogManager: CatalogManager) extends Rule[LogicalPlan]

HyukjinKwon · 2020-05-19T09:05:35Z

cc @cloud-fan FYI

cloud-fan · 2020-05-20T07:48:28Z

retest this please

cloud-fan · 2020-05-20T07:49:09Z

sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/optimizer/Optimizer.scala

      ReplaceExpressions,
      ComputeCurrentTime,
      GetCurrentDatabase(catalogManager),
+      GetCurrentCatalog(catalogManager),


can we merge the above 2 rules into one?

SparkQA · 2020-05-20T09:38:09Z

Test build #122878 has finished for PR 27006 at commit 180fe0a.

This patch fails Spark unit tests.
This patch merges cleanly.
This patch adds the following public classes (experimental):
case class CurrentCatalog() extends LeafExpression with Unevaluable
case class GetCurrentCatalog(catalogManager: CatalogManager) extends Rule[LogicalPlan]

SparkQA · 2020-05-20T15:07:36Z

Test build #122889 has finished for PR 27006 at commit c7b82d5.

This patch passes all tests.
This patch merges cleanly.
This patch adds the following public classes (experimental):
case class GetCurrentDatabaseOrCatalog(catalogManager: CatalogManager) extends Rule[LogicalPlan]

cloud-fan · 2020-05-22T09:15:27Z

sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/expressions/misc.scala

+      > SELECT _FUNC_();
+       spark_catalog
+  """,
+  since = "3.0.0")


it's too late for 3.0 now. let's change it to 3.1.0

cloud-fan · 2020-05-22T09:15:53Z

sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/optimizer/finishAnalysis.scala

+ * Replaces the expression of CurrentDatabase with the current database name.
+ * Replaces the expression of CurrentCatalog with the current catalog name.
+ */
+case class GetCurrentDatabaseOrCatalog(catalogManager: CatalogManager) extends Rule[LogicalPlan] {


GetCurrentDatebaseAndCatalog

cloud-fan · 2020-05-22T09:16:58Z

sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/optimizer/finishAnalysis.scala


    plan transformAllExpressions {
      case CurrentDatabase() =>
+        val currentNamespace = catalogManager.currentNamespace.quoted


it's better to get the value before transformAllExpressions

cloud-fan · 2020-05-22T09:17:08Z

sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/optimizer/finishAnalysis.scala

+        val currentNamespace = catalogManager.currentNamespace.quoted
        Literal.create(currentNamespace, StringType)
+      case CurrentCatalog() =>
+        val currentCatalog = catalogManager.currentCatalog.name()


cloud-fan · 2020-05-22T09:17:39Z

sql/core/src/test/resources/sql-tests/inputs/misc-functions.sql

 select typeof(array(1, 2)), typeof(map(1, 2)), typeof(named_struct('a', 1, 'b', 'spark'));
+
+-- get current_catalog
+select current_catalog();


do we have a sql testing file for current_database?

checked and not found

how about we add a current_database_catalog.sql and test both of them there?

I am okay with it.

SparkQA · 2020-05-22T14:55:15Z

Test build #122991 has finished for PR 27006 at commit 2c32f98.

This patch passes all tests.
This patch merges cleanly.
This patch adds the following public classes (experimental):
case class GetCurrentDatabaseAndCatalog(catalogManager: CatalogManager) extends Rule[LogicalPlan]

cloud-fan · 2020-05-22T15:14:48Z

sorry it conflicts, can you fix it?

SparkQA · 2020-05-22T16:49:22Z

Test build #122996 has finished for PR 27006 at commit fa520d1.

This patch passes all tests.
This patch merges cleanly.
This patch adds no public classes.

SparkQA · 2020-05-22T19:50:34Z

Test build #123007 has finished for PR 27006 at commit f951b5e.

This patch fails Spark unit tests.
This patch merges cleanly.
This patch adds no public classes.

yaooqinn · 2020-05-22T19:52:43Z

retest this please

yaooqinn · 2020-05-23T05:28:01Z

retest this please

SparkQA · 2020-05-23T07:05:02Z

Test build #123024 has finished for PR 27006 at commit f951b5e.

This patch fails due to an unknown error code, -9.
This patch merges cleanly.
This patch adds no public classes.

yaooqinn · 2020-05-23T07:16:07Z

retest this please

SparkQA · 2020-05-23T11:41:58Z

Test build #123032 has finished for PR 27006 at commit f951b5e.

This patch passes all tests.
This patch merges cleanly.
This patch adds no public classes.

cloud-fan · 2020-05-25T14:27:38Z

thanks, merging to master!

[SPARK-30352][SQL] DataSourceV2: Add CURRENT_CATALOG function

180fe0a

dongjoon-hyun added the SQL label Feb 5, 2020

cloud-fan reviewed May 20, 2020

View reviewed changes

yaooqinn added 2 commits May 20, 2020 17:40

Merge branch 'master' into SPARK-30352

dd9f417

address comments

c7b82d5

cloud-fan reviewed May 22, 2020

View reviewed changes

yaooqinn added 3 commits May 22, 2020 17:37

address comments

2c32f98

mv tests

f4f8fbe

mv tests

fa520d1

cloud-fan approved these changes May 22, 2020

View reviewed changes

Merge branch 'master' into SPARK-30352

f951b5e

cloud-fan closed this in 0df8dd6 May 25, 2020

[SPARK-30352][SQL] DataSourceV2: Add CURRENT_CATALOG function #27006

[SPARK-30352][SQL] DataSourceV2: Add CURRENT_CATALOG function #27006

Uh oh!

Conversation

yaooqinn commented Dec 25, 2019

What changes were proposed in this pull request?

Why are the changes needed?

Does this PR introduce any user-facing change?

How was this patch tested?

Uh oh!

SparkQA commented Dec 25, 2019

Uh oh!

yaooqinn commented Dec 25, 2019

Uh oh!

SparkQA commented Dec 25, 2019

Uh oh!

HyukjinKwon commented Dec 27, 2019

Uh oh!

SparkQA commented Dec 27, 2019

Uh oh!

yaooqinn commented Dec 27, 2019

Uh oh!

SparkQA commented Dec 27, 2019

Uh oh!

SparkQA commented Apr 27, 2020

Uh oh!

HyukjinKwon commented May 19, 2020

Uh oh!

cloud-fan commented May 20, 2020

Uh oh!

Choose a reason for hiding this comment

Uh oh!

SparkQA commented May 20, 2020

Uh oh!

SparkQA commented May 20, 2020

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

SparkQA commented May 22, 2020

Uh oh!

cloud-fan commented May 22, 2020

Uh oh!

SparkQA commented May 22, 2020

Uh oh!

SparkQA commented May 22, 2020

Uh oh!

yaooqinn commented May 22, 2020

Uh oh!

yaooqinn commented May 23, 2020

Uh oh!

SparkQA commented May 23, 2020

Uh oh!

yaooqinn commented May 23, 2020

Uh oh!

SparkQA commented May 23, 2020

Uh oh!

cloud-fan commented May 25, 2020

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

5 participants