Skip to content

[SPARK-37330][SQL] Migrate ReplaceTableStatement to v2 command #34764

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom
Closed
dchvn wants to merge 5 commits into from
Closed

[SPARK-37330][SQL] Migrate ReplaceTableStatement to v2 command #34764

dchvn wants to merge 5 commits into from

Conversation

@dchvn
Copy link
Contributor

@dchvn dchvn commented Dec 1, 2021

What changes were proposed in this pull request?

This PR migrates ReplaceTableStatement to the v2 command

Why are the changes needed?

Migrate to the standard V2 framework

Does this PR introduce any user-facing change?

No

How was this patch tested?

existing tests

@github-actions github-actions bot added the SQL label Dec 1, 2021
@SparkQA
Copy link

SparkQA commented Dec 1, 2021

Kubernetes integration test starting
URL: https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder-K8s/50270/

@SparkQA
Copy link

SparkQA commented Dec 1, 2021

Kubernetes integration test status failure
URL: https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder-K8s/50270/

@Peng-Lei Peng-Lei mentioned this pull request Dec 1, 2021
Closed
@SparkQA
Copy link

SparkQA commented Dec 1, 2021

Test build #145795 has finished for PR 34764 at commit 9947a2e.

  • This patch passes all tests.
  • This patch merges cleanly.
  • This patch adds no public classes.

@dchvn dchvn force-pushed the migrate-replacetable branch from 9947a2e to 66a2aaf Compare December 2, 2021 02:45
@SparkQA
Copy link

SparkQA commented Dec 2, 2021

Kubernetes integration test starting
URL: https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder-K8s/50311/

@SparkQA
Copy link

SparkQA commented Dec 2, 2021

Kubernetes integration test status failure
URL: https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder-K8s/50311/

@SparkQA
Copy link

SparkQA commented Dec 2, 2021

Test build #145836 has finished for PR 34764 at commit 66a2aaf.

  • This patch fails Spark unit tests.
  • This patch merges cleanly.
  • This patch adds no public classes.

@dchvn dchvn force-pushed the migrate-replacetable branch from 66a2aaf to ec966bf Compare December 2, 2021 07:27
@SparkQA
Copy link

SparkQA commented Dec 2, 2021

Kubernetes integration test starting
URL: https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder-K8s/50323/

@dchvn
Copy link
Contributor Author

dchvn commented Dec 2, 2021

CC @cloud-fan @Peng-Lei @huaxingao , Thanks!

@SparkQA
Copy link

SparkQA commented Dec 2, 2021

Kubernetes integration test status failure
URL: https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder-K8s/50323/

@SparkQA
Copy link

SparkQA commented Dec 2, 2021

Test build #145848 has finished for PR 34764 at commit ec966bf.

  • This patch passes all tests.
  • This patch merges cleanly.
  • This patch adds no public classes.

@Peng-Lei
Copy link
Contributor

Peng-Lei commented Dec 2, 2021

LGTM, But I just think @huaxingao maybe is planning to do this issue. because it is very similar with CRETE TABLE #34060.

huaxingao
huaxingao reviewed Dec 2, 2021
View reviewed changes
sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/parser/AstBuilder.scala Outdated
Copy link
Contributor

@huaxingao huaxingao Dec 2, 2021

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

You may want to move this out of the pattern matching block so it can be used by both ReplaceTableAsSelect and ReplaceTable.

Copy link
Contributor Author

@dchvn dchvn Dec 3, 2021

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

updated, thanks!

@SparkQA
Copy link

SparkQA commented Dec 3, 2021

Kubernetes integration test starting
URL: https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder-K8s/50346/

@SparkQA
Copy link

SparkQA commented Dec 3, 2021

Kubernetes integration test status failure
URL: https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder-K8s/50346/

huaxingao
huaxingao reviewed Dec 3, 2021
View reviewed changes
sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/parser/AstBuilder.scala Outdated
Copy link
Contributor

@huaxingao huaxingao Dec 3, 2021

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I think external is always false. https://github.com/apache/spark/blob/master/sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/parser/AstBuilder.scala#L3526

Copy link
Contributor

@Peng-Lei Peng-Lei Dec 3, 2021

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I also think external is outdated. Today Spark have no key word EXTERNAL for CREATE/REPLACE TABLE. We use LOCATION to diff the external table and managed table to unify the syntax for create data souce table. So maybe #Line need to be removed.

Copy link
Contributor Author

@dchvn dchvn Dec 3, 2021

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Thank you all,
I think remove external checking code should be in other PR, right?

@SparkQA
Copy link

SparkQA commented Dec 3, 2021

Kubernetes integration test starting
URL: https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder-K8s/50357/

@SparkQA
Copy link

SparkQA commented Dec 3, 2021

Kubernetes integration test status failure
URL: https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder-K8s/50357/

@SparkQA
Copy link

SparkQA commented Dec 3, 2021

Test build #145871 has finished for PR 34764 at commit 7b358d9.

  • This patch passes all tests.
  • This patch merges cleanly.
  • This patch adds no public classes.

@SparkQA
Copy link

SparkQA commented Dec 3, 2021

Test build #145882 has finished for PR 34764 at commit f3cca49.

  • This patch passes all tests.
  • This patch merges cleanly.
  • This patch adds no public classes.

@cloud-fan
Copy link
Contributor

cloud-fan commented Dec 3, 2021

sorry there are code conflicts now

@dchvn dchvn force-pushed the migrate-replacetable branch from f3cca49 to 05d5a41 Compare December 3, 2021 14:29
@SparkQA
Copy link

SparkQA commented Dec 3, 2021

Test build #145904 has finished for PR 34764 at commit 05d5a41.

  • This patch fails to build.
  • This patch merges cleanly.
  • This patch adds no public classes.

@SparkQA
Copy link

SparkQA commented Dec 3, 2021

Kubernetes integration test unable to build dist.

exiting with code: 1
URL: https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder-K8s/50379/

cloud-fan
cloud-fan reviewed Dec 3, 2021
View reviewed changes
...core/src/main/scala/org/apache/spark/sql/execution/datasources/v2/DataSourceV2Strategy.scala
}.getOrElse(props)
val propsWithOwner = CatalogV2Util.withDefaultOwnership(newProps)
case ReplaceTable(ResolvedDBObjectName(catalog, ident), schema, parts, tableSpec, orCreate) =>
val qualifiedLocation = tableSpec.location.map(makeQualifiedDBObjectPath(_))
Copy link
Contributor

@cloud-fan cloud-fan Dec 3, 2021

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

not related to this PR: seems path is not qualified in CreateTableAsSelect? cc @imback82

Copy link
Contributor

@imback82 imback82 Dec 3, 2021

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Right, it doesn't seem to be qualified. I will create a PR for that.

@SparkQA
Copy link

SparkQA commented Dec 3, 2021

Kubernetes integration test starting
URL: https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder-K8s/50383/

@SparkQA
Copy link

SparkQA commented Dec 3, 2021

Kubernetes integration test status failure
URL: https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder-K8s/50383/

@SparkQA
Copy link

SparkQA commented Dec 3, 2021

Test build #145908 has finished for PR 34764 at commit 31cfd01.

  • This patch passes all tests.
  • This patch merges cleanly.
  • This patch adds no public classes.

@huaxingao
Copy link
Contributor

huaxingao commented Dec 3, 2021

@dchvn Could you rebase? I know the Git Action failures are not related to this PR, but with so many failures, I don't feel comfortable to merge the PR.

@dchvn
Copy link
Contributor Author

dchvn commented Dec 4, 2021

Ping @huaxingao , GitHub action is good now, thanks

@huaxingao huaxingao closed this in c411d26 Dec 4, 2021
@huaxingao
Copy link
Contributor

huaxingao commented Dec 4, 2021

Merged to master. Thanks!

@dchvn
Copy link
Contributor Author

dchvn commented Dec 4, 2021

many thanks! @huaxingao

@dchvn dchvn deleted the migrate-replacetable branch December 4, 2021 02:41
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Reviewers

@cloud-fan cloud-fan cloud-fan approved these changes

@huaxingao huaxingao huaxingao approved these changes

+2 more reviewers

@imback82 imback82 imback82 left review comments

@Peng-Lei Peng-Lei Peng-Lei left review comments

Reviewers whose approvals may not affect merge requirements

Assignees

No one assigned

Labels

SQL

Projects

None yet

Milestone

No milestone

Development

Successfully merging this pull request may close these issues.

6 participants

@dchvn @SparkQA @Peng-Lei @cloud-fan @huaxingao @imback82