[SPARK-33130][SQL] Support ALTER TABLE in JDBC v2 Table Catalog: add, update type and nullability of columns (MsSqlServer dialect) #30038

ScrapCodes · 2020-10-14T12:48:01Z

What changes were proposed in this pull request?

Override the default SQL strings for:
ALTER TABLE RENAME COLUMN
ALTER TABLE UPDATE COLUMN NULLABILITY
in the following MsSQLServer JDBC dialect according to official documentation.
Write MsSqlServer integration tests for JDBC.

Why are the changes needed?

To add the support for alter table when interacting with MSSql Server.

Does this PR introduce any user-facing change?

How was this patch tested?

added tests

SparkQA · 2020-10-14T19:46:45Z

Kubernetes integration test starting
URL: https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder-K8s/34352/

SparkQA · 2020-10-14T20:04:34Z

Kubernetes integration test status success
URL: https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder-K8s/34352/

SparkQA · 2020-10-14T21:45:43Z

Test build #129746 has finished for PR 30038 at commit 0159909.

This patch fails Spark unit tests.
This patch merges cleanly.
This patch adds no public classes.

SparkQA · 2020-10-15T09:37:49Z

Kubernetes integration test starting
URL: https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder-K8s/34426/

SparkQA · 2020-10-15T09:57:29Z

Kubernetes integration test status failure
URL: https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder-K8s/34426/

SparkQA · 2020-10-15T11:36:14Z

Test build #129821 has finished for PR 30038 at commit c1605b0.

This patch passes all tests.
This patch merges cleanly.
This patch adds no public classes.

huaxingao · 2020-10-15T21:36:00Z

Please rebase to get the latest change for JdbcDialects.scala.

...egration-tests/src/test/scala/org/apache/spark/sql/jdbc/v2/MsSqlServerIntegrationSuite.scala

sql/core/src/main/scala/org/apache/spark/sql/jdbc/MsSqlServerDialect.scala

SparkQA · 2020-10-23T10:51:35Z

Kubernetes integration test starting
URL: https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder-K8s/34801/

SparkQA · 2020-10-23T11:13:03Z

Kubernetes integration test status success
URL: https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder-K8s/34801/

SparkQA · 2020-10-23T12:44:52Z

Test build #130200 has finished for PR 30038 at commit a119c99.

This patch passes all tests.
This patch merges cleanly.
This patch adds no public classes.

ScrapCodes · 2020-10-23T14:05:42Z

This needs more work, while testing this, I found rename column needs some work for mysql dialect. I am currently working on a patch for the same.

…r MsSql Server. Added unsupported feature exception for switching nullability using alter table.

ScrapCodes · 2020-11-02T06:59:20Z

Hi @huaxingao and @cloud-fan , This is ready for review !

SparkQA · 2020-11-02T07:15:02Z

Kubernetes integration test starting
URL: https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder-K8s/35116/

SparkQA · 2020-11-02T07:36:27Z

Kubernetes integration test starting
URL: https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder-K8s/35117/

SparkQA · 2020-11-02T07:36:45Z

Kubernetes integration test status success
URL: https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder-K8s/35116/

SparkQA · 2020-11-02T07:58:12Z

Kubernetes integration test status success
URL: https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder-K8s/35117/

SparkQA · 2020-11-02T08:05:01Z

Test build #130517 has finished for PR 30038 at commit 1306944.

This patch fails due to an unknown error code, -9.
This patch merges cleanly.
This patch adds no public classes.

SparkQA · 2020-11-02T08:05:02Z

Test build #130516 has finished for PR 30038 at commit d9c6916.

This patch fails due to an unknown error code, -9.
This patch merges cleanly.
This patch adds no public classes.

...egration-tests/src/test/scala/org/apache/spark/sql/jdbc/v2/MsSqlServerIntegrationSuite.scala

SparkQA · 2020-11-03T12:34:08Z

Kubernetes integration test starting
URL: https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder-K8s/35167/

SparkQA · 2020-11-03T12:56:26Z

Kubernetes integration test status success
URL: https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder-K8s/35167/

SparkQA · 2020-11-03T13:39:58Z

Test build #130566 has finished for PR 30038 at commit 449a7ad.

This patch fails Spark unit tests.
This patch merges cleanly.
This patch adds no public classes.

huaxingao · 2020-11-05T22:08:54Z

retest this please

SparkQA · 2020-11-05T23:01:02Z

Kubernetes integration test starting
URL: https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder-K8s/35282/

SparkQA · 2020-11-05T23:24:49Z

Kubernetes integration test status failure
URL: https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder-K8s/35282/

SparkQA · 2020-11-06T00:55:52Z

Test build #130671 has finished for PR 30038 at commit 449a7ad.

This patch passes all tests.
This patch merges cleanly.
This patch adds no public classes.

cloud-fan · 2020-11-06T05:46:36Z

thanks, merging to master!

ScrapCodes · 2020-11-06T09:37:20Z

Thanks !!

ScrapCodes changed the title ~~[WIP]Support ALTER TABLE in JDBC v2 Table Catalog: add, update type and nullability of columns (MsSqlServer dialect)~~ [WIP][SPARK-33130][SQL] Support ALTER TABLE in JDBC v2 Table Catalog: add, update type and nullability of columns (MsSqlServer dialect) Oct 14, 2020

ScrapCodes force-pushed the mssql-dialect branch from f01a096 to c1605b0 Compare October 15, 2020 08:25

dongjoon-hyun added the SQL label Oct 15, 2020

huaxingao reviewed Oct 15, 2020

View reviewed changes

...egration-tests/src/test/scala/org/apache/spark/sql/jdbc/v2/MsSqlServerIntegrationSuite.scala Outdated Show resolved Hide resolved

huaxingao reviewed Oct 15, 2020

View reviewed changes

...egration-tests/src/test/scala/org/apache/spark/sql/jdbc/v2/MsSqlServerIntegrationSuite.scala Outdated Show resolved Hide resolved

huaxingao reviewed Oct 15, 2020

View reviewed changes

...egration-tests/src/test/scala/org/apache/spark/sql/jdbc/v2/MsSqlServerIntegrationSuite.scala Outdated Show resolved Hide resolved

huaxingao reviewed Oct 15, 2020

View reviewed changes

sql/core/src/main/scala/org/apache/spark/sql/jdbc/MsSqlServerDialect.scala Outdated Show resolved Hide resolved

huaxingao reviewed Oct 15, 2020

View reviewed changes

sql/core/src/main/scala/org/apache/spark/sql/jdbc/MsSqlServerDialect.scala Outdated Show resolved Hide resolved

ScrapCodes force-pushed the mssql-dialect branch from c1605b0 to a119c99 Compare October 23, 2020 10:07

ScrapCodes mentioned this pull request Oct 23, 2020

[SPARK-33095] Follow up, support alter table column rename. #30142

Closed

ScrapCodes force-pushed the mssql-dialect branch from a119c99 to d9c6916 Compare November 2, 2020 06:27

Add dialect support for alter table and a docker integration suite fo…

1306944

…r MsSql Server. Added unsupported feature exception for switching nullability using alter table.

ScrapCodes force-pushed the mssql-dialect branch from d9c6916 to 1306944 Compare November 2, 2020 06:45

ScrapCodes requested a review from huaxingao November 2, 2020 06:57

huaxingao reviewed Nov 2, 2020

View reviewed changes

...egration-tests/src/test/scala/org/apache/spark/sql/jdbc/v2/MsSqlServerIntegrationSuite.scala Outdated Show resolved Hide resolved

huaxingao reviewed Nov 2, 2020

View reviewed changes

...egration-tests/src/test/scala/org/apache/spark/sql/jdbc/v2/MsSqlServerIntegrationSuite.scala Show resolved Hide resolved

huaxingao reviewed Nov 2, 2020

View reviewed changes

...egration-tests/src/test/scala/org/apache/spark/sql/jdbc/v2/MsSqlServerIntegrationSuite.scala Outdated Show resolved Hide resolved

huaxingao reviewed Nov 2, 2020

View reviewed changes

...egration-tests/src/test/scala/org/apache/spark/sql/jdbc/v2/MsSqlServerIntegrationSuite.scala Outdated Show resolved Hide resolved

feedback

449a7ad

ScrapCodes requested a review from huaxingao November 3, 2020 12:00

cloud-fan approved these changes Nov 6, 2020

View reviewed changes

cloud-fan closed this in 733a468 Nov 6, 2020

ScrapCodes deleted the mssql-dialect branch November 6, 2020 09:37

[SPARK-33130][SQL] Support ALTER TABLE in JDBC v2 Table Catalog: add, update type and nullability of columns (MsSqlServer dialect) #30038

[SPARK-33130][SQL] Support ALTER TABLE in JDBC v2 Table Catalog: add, update type and nullability of columns (MsSqlServer dialect) #30038

Uh oh!

Conversation

ScrapCodes commented Oct 14, 2020

What changes were proposed in this pull request?

Why are the changes needed?

Does this PR introduce any user-facing change?

How was this patch tested?

Uh oh!

SparkQA commented Oct 14, 2020

Uh oh!

SparkQA commented Oct 14, 2020

Uh oh!

SparkQA commented Oct 14, 2020

Uh oh!

SparkQA commented Oct 15, 2020

Uh oh!

SparkQA commented Oct 15, 2020

Uh oh!

SparkQA commented Oct 15, 2020

Uh oh!

huaxingao commented Oct 15, 2020

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

SparkQA commented Oct 23, 2020

Uh oh!

SparkQA commented Oct 23, 2020

Uh oh!

SparkQA commented Oct 23, 2020

Uh oh!

ScrapCodes commented Oct 23, 2020

Uh oh!

ScrapCodes commented Nov 2, 2020

Uh oh!

SparkQA commented Nov 2, 2020

Uh oh!

SparkQA commented Nov 2, 2020

Uh oh!

SparkQA commented Nov 2, 2020

Uh oh!

SparkQA commented Nov 2, 2020

Uh oh!

SparkQA commented Nov 2, 2020

Uh oh!

SparkQA commented Nov 2, 2020

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

SparkQA commented Nov 3, 2020

Uh oh!

SparkQA commented Nov 3, 2020

Uh oh!

SparkQA commented Nov 3, 2020

Uh oh!

huaxingao commented Nov 5, 2020

Uh oh!

SparkQA commented Nov 5, 2020

Uh oh!

SparkQA commented Nov 5, 2020

Uh oh!

SparkQA commented Nov 6, 2020

Uh oh!

cloud-fan commented Nov 6, 2020

Uh oh!

ScrapCodes commented Nov 6, 2020

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!