Skip to content

Conversation

@viirya
Copy link
Member

@viirya viirya commented Oct 19, 2019

What changes were proposed in this pull request?

Add TruncateTableStatement and make TRUNCATE TABLE go through the same catalog/table resolution framework of v2 commands.

Why are the changes needed?

It's important to make all the commands have the same table resolution behavior, to avoid confusing end-users. e.g.

USE my_catalog
DESC t // success and describe the table t from my_catalog
TRUNCATE TABLE t // report table not found as there is no table t in the session catalog

Does this PR introduce any user-facing change?

yes. When running TRUNCATE TABLE, Spark fails the command if the current catalog is set to a v2 catalog, or the table name specified a v2 catalog.

How was this patch tested?

Unit tests.

@SparkQA
Copy link

SparkQA commented Oct 19, 2019

Test build #112312 has finished for PR 26174 at commit ae50eec.

  • This patch passes all tests.
  • This patch merges cleanly.
  • This patch adds the following public classes (experimental):
  • case class TruncateTableStatement(

@viirya
Copy link
Member Author

viirya commented Oct 19, 2019

cc @cloud-fan @rdblue @imback82

Copy link
Contributor

@imback82 imback82 left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM

v1TableName.asTableIdentifier,
"MSCK REPAIR TABLE")

case TruncateTableStatement(tableName, partitionSpec) =>
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

shall we add a truncateTable method to TableCatalog? cc @rdblue

Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

No. This is not a catalog operation, it is a table operation. We already have 2 ways to truncate a table: if the table implements SupportsTruncate or SupportsOverwrite. A truncate command should build a write and commit without any commit messages.

We can also add a short-cut trait on Table to avoid the builder.

@rdblue
Copy link
Contributor

rdblue commented Oct 21, 2019

Looks good to me if we don't want to add the implementation in the same PR.

@viirya
Copy link
Member Author

viirya commented Oct 21, 2019

I can add the V2 part in this PR or separate one. @cloud-fan

@cloud-fan
Copy link
Contributor

Since it's unclear how to implement a v2 truncate table, let's leave it first.

@cloud-fan
Copy link
Contributor

thanks, merging to master!

@cloud-fan cloud-fan closed this in b4844ee Oct 22, 2019
@rdblue
Copy link
Contributor

rdblue commented Oct 22, 2019

Since it's unclear how to implement a v2 truncate table, let's leave it first.

How is this unclear? There is a truncate API in v2.

@cloud-fan
Copy link
Contributor

I was referring to We can also add a short-cut trait on Table to avoid the builder.

It's unclear if we should submit a job to do truncate, or add a short-cut trait.

@rdblue
Copy link
Contributor

rdblue commented Oct 22, 2019

Even if we add a short-cut trait, Spark should fall back the the builder method if the table supports it.

@viirya viirya deleted the SPARK-29517 branch December 27, 2023 18:37
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

Projects

None yet

Development

Successfully merging this pull request may close these issues.

6 participants