[SPARK-39800][SQL][WIP] DataSourceV2: View Support #39796

jzhuge · 2023-01-30T07:20:59Z

What changes were proposed in this pull request?

This PR adds support to load, create, alter, and drop views in DataSource V2 catalogs.

View substitution rule
Create view DDL
View SQL DDLs

Why are the changes needed?

Support views stored in DataSourceV2 catalogs. Details in SPIP.

Does this PR introduce any user-facing change?

Support views from DataSource V2 catalogs in SQL
Support views from DataSource V2 catalogs in DataFrame API

How was this patch tested?

New unit tests

TODO

Regression

DDLParserSuite
PlanResolutionSuite
DataSourceV2SQLSuite

mridulm · 2023-01-30T07:44:51Z

+CC @shardulm94, @wmoustafa

jzhuge · 2023-01-30T20:09:42Z

TODOs and comments:

More unit tests
Conform to “v2 command framework” (SPARK-36586). ResolveCatalogs seem to have too much extra view code, but that cleanup might require migrating more SQL commands to “v2 command framework”. Maybe a good opportunity to split PR.
Further cleanup/simplify/unify code, especially around ResolvedV2View.

jzhuge · 2023-01-30T20:14:36Z

Thanks @amogh-jahagirdar for driving this PR!

jzhuge · 2023-01-30T20:18:58Z

cc @holdenk (Shepherd) @cloud-fan @imback82 @huaxingao @xkrogen

cloud-fan · 2023-01-31T10:27:49Z

sql/core/src/main/scala/org/apache/spark/sql/internal/BaseSessionStateBuilder.scala

  protected def analyzer: Analyzer = new Analyzer(catalogManager) {
+
+    override val extendedSubstitutionRules: Seq[Rule[LogicalPlan]] =
+      Seq(ViewSubstitution(sqlParser))


why does it need to be a substitution rule? The v1 view is resolved in the main resolution batch.

ViewSubstitution had to be done in the same batch as CTESubstitution to handle nested cte or view in each other.

However, since there have been many changes in master since I originally wrote the code, this requirement may no longer be necessary. Let me revisit. It'd be much better to move the rule to Resolution batch if possible!

cloud-fan · 2023-01-31T10:29:15Z

shall we have the first PR only support creating and reading v2 views? Then we can add alter view commands (which can be done in parallel) later.

jzhuge · 2023-01-31T17:42:38Z

shall we have the first PR only support creating and reading v2 views? Then we can add alter view commands (which can be done in parallel) later.

Yes it is exactly what we planed. Amogh will split the PR.

amogh-jahagirdar · 2023-01-31T17:48:27Z

Thanks @jzhuge, happy to help ! Yes I plan on splitting this PR, I will get more time later this week to look into this.

MaxGekk · 2023-02-16T12:30:58Z

sql/core/src/test/scala/org/apache/spark/sql/connector/DataSourceV2SQLSuite.scala

  }

-  test("View commands are not supported in v2 catalogs") {
+  ignore("View commands are not supported in v2 catalogs") {


Could you remove the test if you supported all those commands.

MaxGekk · 2023-02-16T12:39:11Z

sql/core/src/main/scala/org/apache/spark/sql/execution/datasources/v2/ShowCreateViewExec.scala

+
+  override protected def run(): Seq[InternalRow] = {
+    val schema = desc.schema.map(_.name).mkString("(", ", ", ")")
+    val create = s"CREATE VIEW ${desc.identifier} $schema AS\n${desc.query}\n"


Just in case, shouldn't ${desc.identifier} be quoted?

MaxGekk · 2023-02-16T12:44:42Z

sql/core/src/main/scala/org/apache/spark/sql/execution/datasources/v2/DropViewExec.scala

+
+  override protected def run(): Seq[InternalRow] = {
+    val exists = try {
+      catalog.viewExists(ident)


viewExists() doesn't throw the NoSuchViewException exception. Even the default implementation catches it and returns a boolean, see

spark/sql/catalyst/src/main/java/org/apache/spark/sql/connector/catalog/ViewCatalog.java

Lines 108 to 114 in d12abff

default boolean viewExists(Identifier ident) {

try {

return loadView(ident) != null;

} catch (NoSuchViewException e) {

return false;

}

}

MaxGekk · 2023-02-16T12:58:32Z

sql/core/src/main/scala/org/apache/spark/sql/execution/datasources/v2/AlterViewExec.scala

+    } catch {
+      case e: IllegalArgumentException =>
+        throw new SparkException(s"Invalid view change: ${e.getMessage}", e)
+      case e: UnsupportedOperationException =>


alterView doesn't throw UnsupportedOperationException but throws NoSuchViewException. Could you adjust the comment of alterView according to status quo.

MaxGekk · 2023-02-16T12:59:19Z

sql/core/src/main/scala/org/apache/spark/sql/execution/datasources/v2/AlterViewExec.scala

+      catalog.alterView(ident, changes: _*)
+    } catch {
+      case e: IllegalArgumentException =>
+        throw new SparkException(s"Invalid view change: ${e.getMessage}", e)


Please, introduce an error class in error-classes.json if such class doesn't exist.

@MaxGekk I see you manually reviewing and noticing when PRs introduce new cases of errors that don't go through the error-classes, I wonder if we can capture this with a linter rule to save you some trouble? 🙂

MaxGekk · 2023-02-16T12:59:45Z

sql/core/src/main/scala/org/apache/spark/sql/execution/datasources/v2/AlterViewExec.scala

+      case e: IllegalArgumentException =>
+        throw new SparkException(s"Invalid view change: ${e.getMessage}", e)
+      case e: UnsupportedOperationException =>
+        throw new SparkException(s"Unsupported view change: ${e.getMessage}", e)


the same, please, use an error class.

github-actions · 2023-06-03T00:21:11Z

We're closing this PR because it hasn't been updated in a while. This isn't a judgement on the merit of the PR in any way. It's just a way of keeping the PR queue manageable.
If you'd like to revive this PR, please reopen it and ask a committer to remove the Stale tag!

jzhuge · 2023-12-04T18:19:21Z

Current status:

Incorporate all @MaxGekk comments (took care of some)
Add unit tests

Follow up:

Support user specified column names
Support viewSQLConfigs
Conform to “v2 command framework” (SPARK-36586). - ResolveCatalogs seem to have too much extra view code, but that cleanup might require migrating more SQL commands to “v2 command framework”.

holdenk · 2023-12-05T17:59:22Z

It won't let me re-open, can you re-create the PR?

arpitporwal2293 · 2025-09-29T20:23:06Z

Hey @jzhuge, since this pull request is not merged. Is DSv2 support for Views missing in spark? I am not able to find any follow up pull requests which adds that functionality. Would appreciate the help.

jzhuge · 2025-10-01T17:51:11Z

Could you share your use case? Specifically what type of view backend?

Along with Iceberg community, we added view support to Iceberg 1.5. Spark components for view support were added to the Iceberg SQL extension.

If desirable, I will be happy to restart the effort for a Spark PR.

sunchao · 2025-12-19T21:27:47Z

@jzhuge we're also very interested in this feature. Currently we are using Delta Lake with DSv2 (we have a custom v2 catalog impl), and it'd be great if the view support is in Spark itself. Happy to help review & push the Spark PR too!

[SPARK-39800][SQL][WIP] DataSourceV2: View Support

48de5ec

github-actions bot added the SQL label Jan 30, 2023

cloud-fan reviewed Jan 31, 2023

View reviewed changes

MaxGekk reviewed Feb 16, 2023

View reviewed changes

github-actions bot added the Stale label Jun 3, 2023

github-actions bot closed this Jun 4, 2023

nastra mentioned this pull request Oct 24, 2023

Add View Support to Spark apache/iceberg#7938

Closed

jzhuge mentioned this pull request Dec 4, 2023

[SPARK-46050][SQL]: Allow additional rules for the Substitution batch #43952

Closed

holdenk removed the Stale label Dec 5, 2023

jzhuge mentioned this pull request Dec 6, 2023

[SPARK-39800][SQL][WIP] DataSourceV2: View Support #44197

Closed

	default boolean viewExists(Identifier ident) {
	try {
	return loadView(ident) != null;
	} catch (NoSuchViewException e) {
	return false;
	}
	}

[SPARK-39800][SQL][WIP] DataSourceV2: View Support #39796

[SPARK-39800][SQL][WIP] DataSourceV2: View Support #39796

Uh oh!

Conversation

jzhuge commented Jan 30, 2023

What changes were proposed in this pull request?

Why are the changes needed?

Does this PR introduce any user-facing change?

How was this patch tested?

Uh oh!

mridulm commented Jan 30, 2023

Uh oh!

jzhuge commented Jan 30, 2023 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

jzhuge commented Jan 30, 2023

Uh oh!

jzhuge commented Jan 30, 2023

Uh oh!

Choose a reason for hiding this comment

Uh oh!

jzhuge Jan 31, 2023 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

cloud-fan commented Jan 31, 2023

Uh oh!

jzhuge commented Jan 31, 2023

Uh oh!

amogh-jahagirdar commented Jan 31, 2023

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

github-actions bot commented Jun 3, 2023

Uh oh!

jzhuge commented Dec 4, 2023

Uh oh!

holdenk commented Dec 5, 2023

Uh oh!

arpitporwal2293 commented Sep 29, 2025

Uh oh!

jzhuge commented Oct 1, 2025

Uh oh!

sunchao commented Dec 19, 2025

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

9 participants

jzhuge commented Jan 30, 2023 •

edited

Loading

jzhuge Jan 31, 2023 •

edited

Loading