[SPARK-33095][SQL] Support ALTER TABLE in JDBC v2 Table Catalog: add, update type and nullability of columns (MySQL dialect) #30025

ScrapCodes · 2020-10-13T09:18:39Z

What changes were proposed in this pull request?

Override the default SQL strings for:
ALTER TABLE UPDATE COLUMN TYPE
ALTER TABLE UPDATE COLUMN NULLABILITY
in the following MySQL JDBC dialect according to official documentation.
Write MySQL integration tests for JDBC.

Why are the changes needed?

Improved code coverage and support mysql dialect for jdbc.

Does this PR introduce any user-facing change?

Yes, Support ALTER TABLE in JDBC v2 Table Catalog: add, update type and nullability of columns (MySQL dialect)

How was this patch tested?

Added tests.

SparkQA · 2020-10-13T10:19:28Z

Kubernetes integration test starting
URL: https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder-K8s/34345/

SparkQA · 2020-10-13T10:37:09Z

Kubernetes integration test status success
URL: https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder-K8s/34345/

SparkQA · 2020-10-13T12:21:05Z

Test build #129739 has finished for PR 30025 at commit d495611.

This patch passes all tests.
This patch merges cleanly.
This patch adds no public classes.

SparkQA · 2020-10-14T20:46:23Z

Kubernetes integration test starting
URL: https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder-K8s/34356/

SparkQA · 2020-10-14T21:03:57Z

Kubernetes integration test status success
URL: https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder-K8s/34356/

SparkQA · 2020-10-14T23:32:04Z

Test build #129750 has finished for PR 30025 at commit ba7ebdb.

This patch passes all tests.
This patch merges cleanly.
This patch adds no public classes.

SparkQA · 2020-10-15T09:42:53Z

Kubernetes integration test starting
URL: https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder-K8s/34428/

SparkQA · 2020-10-15T10:07:19Z

Kubernetes integration test status success
URL: https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder-K8s/34428/

SparkQA · 2020-10-15T11:57:20Z

Test build #129823 has finished for PR 30025 at commit 1ee7cd6.

This patch passes all tests.
This patch merges cleanly.
This patch adds no public classes.

huaxingao · 2020-10-15T21:32:53Z

...er-integration-tests/src/test/scala/org/apache/spark/sql/jdbc/v2/MySQLIntegrationSuite.scala

+
+import org.apache.spark.SparkConf
+import org.apache.spark.sql.AnalysisException
+import org.apache.spark.sql.catalyst.parser.ParseException


Seems this is not needed?

huaxingao · 2020-10-15T21:33:00Z

...er-integration-tests/src/test/scala/org/apache/spark/sql/jdbc/v2/MySQLIntegrationSuite.scala

+import org.apache.spark.sql.catalyst.parser.ParseException
+import org.apache.spark.sql.execution.datasources.v2.jdbc.JDBCTableCatalog
+import org.apache.spark.sql.jdbc.{DatabaseOnDocker, DockerJDBCIntegrationSuite}
+import org.apache.spark.sql.test.SharedSparkSession


Not needed?

huaxingao · 2020-10-15T21:33:16Z

sql/core/src/main/scala/org/apache/spark/sql/jdbc/MySQLDialect.scala

+      tableName: String,
+      columnName: String,
+      newDataType: String): String = {
+    s"ALTER TABLE $tableName MODIFY COLUMN $columnName $newDataType"


$columnName -> ${quoteIdentifier(columnName)}
please see #30041

huaxingao · 2020-10-15T21:35:31Z

also cc @cloud-fan @MaxGekk

SparkQA · 2020-10-16T07:05:01Z

Test build #129880 has finished for PR 30025 at commit dfd6d4b.

This patch fails due to an unknown error code, -9.
This patch merges cleanly.
This patch adds no public classes.

sql/core/src/main/scala/org/apache/spark/sql/jdbc/MySQLDialect.scala

SparkQA · 2020-10-16T07:43:37Z

Kubernetes integration test starting
URL: https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder-K8s/34485/

SparkQA · 2020-10-16T08:01:21Z

Kubernetes integration test status success
URL: https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder-K8s/34485/

ScrapCodes · 2020-10-21T10:02:16Z

I have updated. First getting schema from the table and taking the type information from the schema and then passing type to alter table update nullability.

Hi @huaxingao, thanks for the help ! Please take a look again.

@cloud-fan thank you for the suggestion, Please take a look again !

SparkQA · 2020-10-21T10:20:20Z

Kubernetes integration test starting
URL: https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder-K8s/34695/

SparkQA · 2020-10-21T10:43:27Z

Kubernetes integration test status success
URL: https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder-K8s/34695/

…te type and nullability of columns (MySQL dialect)

SparkQA · 2020-10-21T11:39:59Z

Kubernetes integration test starting
URL: https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder-K8s/34700/

SparkQA · 2020-10-21T12:02:29Z

Test build #130086 has finished for PR 30025 at commit c7f4113.

This patch passes all tests.
This patch merges cleanly.
This patch adds no public classes.

SparkQA · 2020-10-21T12:02:52Z

Kubernetes integration test status success
URL: https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder-K8s/34700/

SparkQA · 2020-10-21T12:55:15Z

Test build #130091 has finished for PR 30025 at commit 230fed8.

This patch fails Spark unit tests.
This patch merges cleanly.
This patch adds no public classes.

cloud-fan · 2020-10-22T04:54:02Z

...ore/src/main/scala/org/apache/spark/sql/execution/datasources/v2/jdbc/JDBCTableCatalog.scala

-        JdbcUtils.alterTable(conn, getTableName(ident), changes, options)
+        val optionsWithTableName = new JDBCOptions(
+          options.parameters + (JDBCOptions.JDBC_TABLE_NAME -> getTableName(ident)))
+        val tableSchema: StructType = JdbcUtils.getSchemaOption(conn, optionsWithTableName).get


ah now I get what you mean. The AlterTable logical plan does have the table schema, but the catalog API doesn't pass it in. It's not possible to change the catalog API at this point, and it's also not worthy to add an extra table lookup here just to support update nullability in MySQL.

I think the first version is fine. Sorry for the back and forth!

Thanks will do so ! 👍

This reverts commit 230fed8.

SparkQA · 2020-10-22T08:44:03Z

Kubernetes integration test starting
URL: https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder-K8s/34745/

SparkQA · 2020-10-22T09:06:47Z

Kubernetes integration test status success
URL: https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder-K8s/34745/

SparkQA · 2020-10-22T11:05:30Z

Test build #130138 has finished for PR 30025 at commit 3fc4132.

This patch passes all tests.
This patch merges cleanly.
This patch adds no public classes.

cloud-fan · 2020-10-22T13:51:37Z

thanks, merging to master!

ScrapCodes · 2020-10-22T14:03:48Z

Thanks a lot @cloud-fan :)

ScrapCodes changed the title ~~[SPARK-33095] Support ALTER TABLE in JDBC v2 Table Catalog: add, update type and nullability of columns (MySQL dialect)~~ [SPARK-33095][SQL] Support ALTER TABLE in JDBC v2 Table Catalog: add, update type and nullability of columns (MySQL dialect) Oct 13, 2020

ScrapCodes force-pushed the mysql-dialect branch 2 times, most recently from 78e33ec to ba7ebdb Compare October 14, 2020 07:55

dongjoon-hyun added the SQL label Oct 15, 2020

huaxingao reviewed Oct 15, 2020

View reviewed changes

ScrapCodes force-pushed the mysql-dialect branch from 1ee7cd6 to dfd6d4b Compare October 16, 2020 06:51

cloud-fan reviewed Oct 16, 2020

View reviewed changes

sql/core/src/main/scala/org/apache/spark/sql/jdbc/MySQLDialect.scala Show resolved Hide resolved

ScrapCodes added 5 commits October 21, 2020 16:18

[SPARK-33095] Support ALTER TABLE in JDBC v2 Table Catalog: add, upda…

4854728

…te type and nullability of columns (MySQL dialect)

Rebased to latest, and refactored as per the code changes from pr 29972.

dee8534

fix formatting

cbce070

review feedback: quote strings and optimize imports.

ac3600e

Pass type information to alter table for nullability change.

230fed8

ScrapCodes force-pushed the mysql-dialect branch from c7f4113 to 230fed8 Compare October 21, 2020 10:49

cloud-fan reviewed Oct 22, 2020

View reviewed changes

Revert "Pass type information to alter table for nullability change."

3fc4132

This reverts commit 230fed8.

cloud-fan approved these changes Oct 22, 2020

View reviewed changes

cloud-fan closed this in 8cae7f8 Oct 22, 2020

[SPARK-33095][SQL] Support ALTER TABLE in JDBC v2 Table Catalog: add, update type and nullability of columns (MySQL dialect) #30025

[SPARK-33095][SQL] Support ALTER TABLE in JDBC v2 Table Catalog: add, update type and nullability of columns (MySQL dialect) #30025

Uh oh!

Conversation

ScrapCodes commented Oct 13, 2020 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

What changes were proposed in this pull request?

Why are the changes needed?

Does this PR introduce any user-facing change?

How was this patch tested?

Uh oh!

SparkQA commented Oct 13, 2020

Uh oh!

SparkQA commented Oct 13, 2020

Uh oh!

SparkQA commented Oct 13, 2020

Uh oh!

SparkQA commented Oct 14, 2020

Uh oh!

SparkQA commented Oct 14, 2020

Uh oh!

SparkQA commented Oct 14, 2020

Uh oh!

SparkQA commented Oct 15, 2020

Uh oh!

SparkQA commented Oct 15, 2020

Uh oh!

SparkQA commented Oct 15, 2020

Uh oh!

huaxingao Oct 15, 2020

Choose a reason for hiding this comment

Uh oh!

huaxingao Oct 15, 2020

Choose a reason for hiding this comment

Uh oh!

huaxingao Oct 15, 2020

Choose a reason for hiding this comment

Uh oh!

huaxingao commented Oct 15, 2020

Uh oh!

SparkQA commented Oct 16, 2020

Uh oh!

Uh oh!

SparkQA commented Oct 16, 2020

Uh oh!

SparkQA commented Oct 16, 2020

Uh oh!

ScrapCodes commented Oct 21, 2020

Uh oh!

SparkQA commented Oct 21, 2020

Uh oh!

SparkQA commented Oct 21, 2020

Uh oh!

SparkQA commented Oct 21, 2020

Uh oh!

SparkQA commented Oct 21, 2020

Uh oh!

SparkQA commented Oct 21, 2020

Uh oh!

SparkQA commented Oct 21, 2020

Uh oh!

cloud-fan Oct 22, 2020

Choose a reason for hiding this comment

Uh oh!

ScrapCodes Oct 22, 2020

Choose a reason for hiding this comment

Uh oh!

SparkQA commented Oct 22, 2020

Uh oh!

SparkQA commented Oct 22, 2020

Uh oh!

SparkQA commented Oct 22, 2020

Uh oh!

cloud-fan commented Oct 22, 2020

Uh oh!

ScrapCodes commented Oct 22, 2020

Uh oh!

Reviewers

Assignees

Labels

ScrapCodes commented Oct 13, 2020 •

edited

Loading