[SPARK-31357][SQL][WIP] Catalog API for view metadata

[jzhuge]

What changes were proposed in this pull request?

• View catalog interface
• View resolution rules
• View SQL DDL implementations

These view SQL DDLs will be done in follow-up PRs:

• SHOW TBLPROPERTIES
• SHOW VIEWS

Most changes are pretty surgical. Please review the view substitution rule.

View substitution

Propose a new ViewSubstitution rule that takes a different approach than the current AnalysisContext. Here are some highlights:

• The rule is added to the “Substitution” batch, same as “CTESubstitution”.
• Temp views are processed first.
• Identifiers for “SQLOnFile” are excluded.
• In the plan parsed from view sql text, substitute CTEs first, then fully qualify all identifiers with the currentCatalog and currentNamespace stored in the view metadata.
• Validate currentCatalog is still registered. Other than that, we have to assume registered catalogs have not undergone any breaking change between view creation and usage.

TODO: viewQueryColumnNames

Propose viewQueryColumnNames to be stored in view properties just as it is currently in Hive metastore table properties. If not in view properties, we can derive from schema field names.

Why are the changes needed?

Support view catalog in DataSourceV2. Details in SPIP.

Does this PR introduce any user-facing change?

• View SQL DDLs
• DataFrame.table("catalog.db.view")

How was this patch tested?

TODO: new unit tests

Regression

• DDLParserSuite
• PlanResolutionSuite
• DataSourceV2SQLSuite

[SparkQA]

Test build #120941 has finished for PR 28147 at commit b253954.

• This patch fails Spark unit tests.
• This patch merges cleanly.
• This patch adds the following public classes (experimental):
• public final class ViewReservedProperties
• public static class Updater
• case class ViewSubstitution(sqlParser: ParserInterface) extends Rule[LogicalPlan]
• trait ViewDescription
• case class CatalogViewDescription(metadata: CatalogTable) extends ViewDescription
• case class V2ViewDescription(
• case class CreateView(
• case class DropView(
• case class AlterView(
• case class RenameView(
• case class DescribeView(desc: V2ViewDescription, isExtended: Boolean) extends Command
• case class ShowCreateView(desc: V2ViewDescription) extends Command
• case class ShowViewProperties(
• case class CreateViewAnalysis(
• case class AlterViewExec(
• trait CatalystRowHelper
• case class CreateViewExec(
• case class DescribeViewExec(
• case class DropViewExec(
• case class RenameViewExec(
• case class ShowCreateViewExec(output: Seq[Attribute], desc: V2ViewDescription)
• case class ShowViewPropertiesExec(

[SparkQA]

Test build #120954 has finished for PR 28147 at commit cd00319.

• This patch fails Spark unit tests.
• This patch merges cleanly.
• This patch adds the following public classes (experimental):
• case class ShowTablePropertiesStatement(

[SparkQA]

Test build #120987 has finished for PR 28147 at commit 32ec893.

• This patch passes all tests.
• This patch merges cleanly.
• This patch adds no public classes.

[SparkQA]

Test build #120997 has finished for PR 28147 at commit 3435379.

• This patch fails due to an unknown error code, -9.
• This patch merges cleanly.
• This patch adds no public classes.

[SparkQA]

Test build #121096 has finished for PR 28147 at commit 44678b1.

• This patch passes all tests.
• This patch merges cleanly.
• This patch adds no public classes.

[jzhuge]

@cloud-fan @dongjoon-hyun @brkyvz Could you comment on this WIP PR? Go or no-go? Need more clarifications? Even though it is labeled as WIP, the code is pretty stable and in production for over 6 months.

[github-actions]

We're closing this PR because it hasn't been updated in a while. This isn't a judgement on the merit of the PR in any way. It's just a way of keeping the PR queue manageable.
If you'd like to revive this PR, please reopen it and ask a committer to remove the Stale tag!