[SPARK-28856][SQL] Implement SHOW DATABASES for Data Source V2 Tables #25601
There are no files selected for viewing
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| Original file line number | Diff line number | Diff line change |
|---|---|---|
| @@ -0,0 +1,24 @@ | ||
| /* | ||
| * Licensed to the Apache Software Foundation (ASF) under one or more | ||
| * contributor license agreements. See the NOTICE file distributed with | ||
| * this work for additional information regarding copyright ownership. | ||
| * The ASF licenses this file to You under the Apache License, Version 2.0 | ||
| * (the "License"); you may not use this file except in compliance with | ||
| * the License. You may obtain a copy of the License at | ||
| * | ||
| * http://www.apache.org/licenses/LICENSE-2.0 | ||
| * | ||
| * Unless required by applicable law or agreed to in writing, software | ||
| * distributed under the License is distributed on an "AS IS" BASIS, | ||
| * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. | ||
| * See the License for the specific language governing permissions and | ||
| * limitations under the License. | ||
| */ | ||
|
|
||
| package org.apache.spark.sql.catalyst.plans.logical.sql | ||
|
|
||
| /** | ||
| * A SHOW NAMESPACES statement, as parsed from SQL. | ||
| */ | ||
| case class ShowNamespacesStatement(namespace: Option[Seq[String]], pattern: Option[String]) | ||
|
Contributor
Author
There was a problem hiding this comment. Yes, this is catalog + namespace, but I followed the same convention as other statements - i.e., |
||
| extends ParsedStatement | ||
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| Original file line number | Diff line number | Diff line change |
|---|---|---|
|
|
@@ -20,15 +20,15 @@ package org.apache.spark.sql.execution.datasources | |
| import scala.collection.mutable | ||
|
|
||
| import org.apache.spark.sql.{AnalysisException, SaveMode} | ||
| import org.apache.spark.sql.catalog.v2.{CatalogManager, Identifier, LookupCatalog, SupportsNamespaces, TableCatalog} | ||
| import org.apache.spark.sql.catalog.v2.expressions.Transform | ||
| import org.apache.spark.sql.catalyst.TableIdentifier | ||
| import org.apache.spark.sql.catalyst.analysis.{CastSupport, UnresolvedRelation} | ||
| import org.apache.spark.sql.catalyst.catalog.{BucketSpec, CatalogTable, CatalogTableType, CatalogUtils, UnresolvedCatalogRelation} | ||
| import org.apache.spark.sql.catalyst.plans.logical.{CreateTableAsSelect, CreateV2Table, DeleteFromTable, DropTable, Filter, LogicalPlan, ReplaceTable, ReplaceTableAsSelect, ShowNamespaces, ShowTables, SubqueryAlias} | ||
| import org.apache.spark.sql.catalyst.plans.logical.sql.{AlterTableAddColumnsStatement, AlterTableSetLocationStatement, AlterTableSetPropertiesStatement, AlterTableUnsetPropertiesStatement, AlterViewSetPropertiesStatement, AlterViewUnsetPropertiesStatement, CreateTableAsSelectStatement, CreateTableStatement, DeleteFromStatement, DescribeColumnStatement, DescribeTableStatement, DropTableStatement, DropViewStatement, QualifiedColType, ReplaceTableAsSelectStatement, ReplaceTableStatement, ShowNamespacesStatement, ShowTablesStatement} | ||
| import org.apache.spark.sql.catalyst.rules.Rule | ||
| import org.apache.spark.sql.execution.command.{AlterTableAddColumnsCommand, AlterTableSetLocationCommand, AlterTableSetPropertiesCommand, AlterTableUnsetPropertiesCommand, DescribeColumnCommand, DescribeTableCommand, DropTableCommand, ShowDatabasesCommand, ShowTablesCommand} | ||
| import org.apache.spark.sql.execution.datasources.v2.FileDataSourceV2 | ||
| import org.apache.spark.sql.internal.SQLConf | ||
| import org.apache.spark.sql.types.{HIVE_TYPE_STRING, HiveStringType, MetadataBuilder, StructField, StructType} | ||
|
|
@@ -169,6 +169,24 @@ case class DataSourceResolution( | |
| val aliased = delete.tableAlias.map(SubqueryAlias(_, relation)).getOrElse(relation) | ||
| DeleteFromTable(aliased, delete.condition) | ||
|
|
||
| case ShowNamespacesStatement(None, pattern) => | ||
| defaultCatalog match { | ||
|
Contributor
There was a problem hiding this comment. I think this should be
Contributor
There was a problem hiding this comment. Let's implement switching the current catalog first, otherwise we are not able to test it.
Contributor
Author
There was a problem hiding this comment. yes, I am working on
Contributor
Author
There was a problem hiding this comment. I should be able to send out the PR sometime tomorrow. |
||
| case Some(catalog) => | ||
| ShowNamespaces(catalog.asNamespaceCatalog, None, pattern) | ||
| case None => | ||
| throw new AnalysisException("No default v2 catalog is set.") | ||
| } | ||
|
|
||
| case ShowNamespacesStatement(Some(namespace), pattern) => | ||
| val CatalogNamespace(maybeCatalog, ns) = namespace | ||
| maybeCatalog match { | ||
| case Some(catalog) => | ||
| ShowNamespaces(catalog.asNamespaceCatalog, Some(ns), pattern) | ||
| case None => | ||
| throw new AnalysisException( | ||
|
Contributor
There was a problem hiding this comment. I think this needs to distinguish between the case where the catalog is None and the catalog does not support namespaces. For the second case, this should report that the catalog doesn't support namespaces. You can also add a conversion method,
Contributor
Author
There was a problem hiding this comment. Using
Contributor
There was a problem hiding this comment. Why not use the current catalog instead of failing?
Contributor
There was a problem hiding this comment. If the catalog name is specified, but catalog doesn't support namespace, I think we should fail instead of falling back to the current catalog. It's similar to: if the catalog name is specified, but doesn't contain the table we need, we should fail instead of falling back to the current catalog. |
||
| s"No v2 catalog is available for ${namespace.quoted}") | ||
| } | ||
|
|
||
| case ShowTablesStatement(None, pattern) => | ||
| defaultCatalog match { | ||
| case Some(catalog) => | ||
|
|
||
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| Original file line number | Diff line number | Diff line change |
|---|---|---|
| @@ -0,0 +1,63 @@ | ||
| /* | ||
| * Licensed to the Apache Software Foundation (ASF) under one or more | ||
| * contributor license agreements. See the NOTICE file distributed with | ||
| * this work for additional information regarding copyright ownership. | ||
| * The ASF licenses this file to You under the Apache License, Version 2.0 | ||
| * (the "License"); you may not use this file except in compliance with | ||
| * the License. You may obtain a copy of the License at | ||
| * | ||
| * http://www.apache.org/licenses/LICENSE-2.0 | ||
| * | ||
| * Unless required by applicable law or agreed to in writing, software | ||
| * distributed under the License is distributed on an "AS IS" BASIS, | ||
| * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. | ||
| * See the License for the specific language governing permissions and | ||
| * limitations under the License. | ||
| */ | ||
|
|
||
| package org.apache.spark.sql.execution.datasources.v2 | ||
|
|
||
| import scala.collection.mutable.ArrayBuffer | ||
|
|
||
| import org.apache.spark.rdd.RDD | ||
| import org.apache.spark.sql.catalog.v2.CatalogV2Implicits.NamespaceHelper | ||
| import org.apache.spark.sql.catalog.v2.SupportsNamespaces | ||
| import org.apache.spark.sql.catalyst.InternalRow | ||
| import org.apache.spark.sql.catalyst.encoders.RowEncoder | ||
| import org.apache.spark.sql.catalyst.expressions.{Attribute, GenericRowWithSchema} | ||
| import org.apache.spark.sql.catalyst.util.StringUtils | ||
| import org.apache.spark.sql.execution.LeafExecNode | ||
|
|
||
| /** | ||
| * Physical plan node for showing namespaces. | ||
| */ | ||
| case class ShowNamespacesExec( | ||
| output: Seq[Attribute], | ||
| catalog: SupportsNamespaces, | ||
| namespace: Option[Seq[String]], | ||
| pattern: Option[String]) | ||
| extends LeafExecNode { | ||
| override protected def doExecute(): RDD[InternalRow] = { | ||
| val namespaces = namespace.map { ns => | ||
| if (ns.nonEmpty) { | ||
|
Contributor
Author
There was a problem hiding this comment. @rdblue @cloud-fan this is for handling the case
Contributor
There was a problem hiding this comment.
I think these 2 are the same?
Contributor
Author
There was a problem hiding this comment. From the SPIP, I see the following: Since the behavior of
Contributor
There was a problem hiding this comment. Calling |
||
| catalog.listNamespaces(ns.toArray) | ||
| } else { | ||
| catalog.listNamespaces() | ||
| } | ||
| } | ||
| .getOrElse(catalog.listNamespaces()) | ||
|
|
||
| val rows = new ArrayBuffer[InternalRow]() | ||
| val encoder = RowEncoder(schema).resolveAndBind() | ||
|
|
||
| namespaces.map(_.quoted).map { ns => | ||
| if (pattern.map(StringUtils.filterPattern(Seq(ns), _).nonEmpty).getOrElse(true)) { | ||
| rows += encoder | ||
| .toRow(new GenericRowWithSchema(Array(ns), schema)) | ||
| .copy() | ||
| } | ||
| } | ||
|
|
||
| sparkContext.parallelize(rows, 1) | ||
| } | ||
| } | ||
Oops, something went wrong.
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
cc @xianyinxin , we should also add DELETE and UPDATE. Can you open a PR to do it?
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
ok, will open a pr.
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
DELETE is already there. UPDATE is included in #25626