[SPARK-14922][SPARK-17732][SPARK-23866][SQL] Support partition filter in ALTER TABLE DROP PARTITION and batch dropping PARTITIONS #26280

weixiuli · 2019-10-28T09:23:11Z

What changes were proposed in this pull request?

Spark only can drop partitions by exact values. For instance, Spark doesn't support:

ALTER TABLE mytable DROP PARTITION(mydate < '2018-04-06')

The PR adds the support to this syntax.

The PR takes input from the effort in #19691 by @DazhuangSu , in #20999 by @mgaido91 ,
, there are some related PRs: #15987,#16036,#19691,#15704.
some related issues:
https://issues.apache.org/jira/browse/SPARK-14922
https://issues.apache.org/jira/browse/SPARK-17732
https://issues.apache.org/jira/browse/SPARK-23866

How was this patch tested?

Add unittests.

mgaido91 · 2019-10-28T13:03:29Z

please fix all failures

weixiuli · 2019-10-29T05:44:07Z

Ping @cloud-fan @mgaido91 @maropu @DazhuangSu Kindly review, thanks.

maropu · 2019-10-29T06:46:21Z

ok to test

SparkQA · 2019-10-29T07:05:02Z

Test build #112830 has finished for PR 26280 at commit ce453ed.

This patch fails due to an unknown error code, -9.
This patch merges cleanly.
This patch adds the following public classes (experimental):
case class PartitioningAttribute(
public class ExpressionBuilder
public final class Utilities

weixiuli · 2019-10-29T07:28:06Z

please retry test

weixiuli · 2019-10-29T07:48:46Z

jenkins, retest this, please

maropu · 2019-10-29T10:27:49Z

retest this please

SparkQA · 2019-10-29T10:55:30Z

Test build #112845 has finished for PR 26280 at commit ce453ed.

This patch fails Spark unit tests.
This patch merges cleanly.
This patch adds the following public classes (experimental):
case class PartitioningAttribute(
public class ExpressionBuilder
public final class Utilities

weixiuli · 2019-10-29T11:04:58Z

retest this please

wangyum · 2019-10-30T11:04:25Z

retest this please

SparkQA · 2019-10-30T13:06:40Z

Test build #112919 has finished for PR 26280 at commit ce453ed.

This patch fails Spark unit tests.
This patch merges cleanly.
This patch adds the following public classes (experimental):
case class PartitioningAttribute(
public class ExpressionBuilder
public final class Utilities

SparkQA · 2019-10-31T07:05:02Z

Test build #113000 has finished for PR 26280 at commit a0c42fd.

This patch fails due to an unknown error code, -9.
This patch merges cleanly.
This patch adds the following public classes (experimental):
case class PartitioningAttribute(
public class ExpressionBuilder
public final class Utilities

weixiuli · 2019-10-31T07:46:29Z

retest this please

maropu · 2019-10-31T11:16:33Z

retest this please

SparkQA · 2019-10-31T13:17:34Z

Test build #113022 has finished for PR 26280 at commit a0c42fd.

This patch fails Spark unit tests.
This patch merges cleanly.
This patch adds the following public classes (experimental):
case class PartitioningAttribute(
public class ExpressionBuilder
public final class Utilities

SparkQA · 2019-11-01T12:17:16Z

Test build #113084 has finished for PR 26280 at commit 1bd2cd3.

This patch fails Spark unit tests.
This patch merges cleanly.
This patch adds the following public classes (experimental):
case class PartitioningAttribute(
public class ExpressionBuilder
public final class Utilities

SparkQA · 2019-11-14T04:00:05Z

Test build #113747 has finished for PR 26280 at commit e27f13c.

This patch fails to generate documentation.
This patch merges cleanly.
This patch adds the following public classes (experimental):
case class PartitioningAttribute(
case class ExpressionBuilder(val table: MetaStoreApiTable,

weixiuli · 2019-11-14T06:34:29Z

retest this please

SparkQA · 2019-11-14T07:00:15Z

Test build #113757 has finished for PR 26280 at commit 21cd780.

This patch fails to generate documentation.
This patch merges cleanly.
This patch adds the following public classes (experimental):
case class PartitioningAttribute(
case class ExpressionBuilder(val table: MetaStoreApiTable,

SparkQA · 2019-11-14T12:19:41Z

Test build #113775 has finished for PR 26280 at commit 0d5bc03.

This patch fails Spark unit tests.
This patch merges cleanly.
This patch adds the following public classes (experimental):
case class PartitioningAttribute(
case class ExpressionBuilder(val table: MetaStoreApiTable,

SparkQA · 2019-11-15T05:13:56Z

Test build #113834 has finished for PR 26280 at commit 11b4cb0.

This patch fails Spark unit tests.
This patch merges cleanly.
This patch adds the following public classes (experimental):
case class PartitioningAttribute(
case class ExpressionBuilder(val table: MetaStoreApiTable,

SparkQA · 2019-11-15T08:05:01Z

Test build #113852 has finished for PR 26280 at commit 9d3ede5.

This patch fails due to an unknown error code, -9.
This patch merges cleanly.
This patch adds the following public classes (experimental):
case class PartitioningAttribute(
case class ExpressionBuilder(val table: MetaStoreApiTable,

weixiuli · 2019-11-18T01:37:59Z

retest this please.
@cloud-fan @mgaido91 @maropu @DazhuangSu Kindly review, thanks.

…s in ALTER TABLE DROP PARTITION and batch dropping PARTITIONS - Support partition filters in ALTER TABLE DROP PARTITION. - Support batch dropping PARTITIONS by 'ExpressionBuilder'.

SparkQA · 2019-11-19T08:05:02Z

Test build #114062 has finished for PR 26280 at commit c2639d0.

This patch fails due to an unknown error code, -9.
This patch merges cleanly.
This patch adds the following public classes (experimental):
case class PartitioningAttribute(
case class ExpressionBuilder(val table: MetaStoreApiTable,

weixiuli · 2019-11-19T09:36:02Z

@cloud-fan Could you review this PR , thanks.

wangshisan · 2019-12-19T09:51:04Z

sql/catalyst/src/main/antlr4/org/apache/spark/sql/catalyst/parser/SqlBase.g4

+    ;
+
+dropPartitionVal
+    : identifier (comparisonOperator constant)?


One suggestion, can we replace constant with expression? In our PROD environment, there are some requirements like 'ALTER TABLE xxx DROP PARTITION (col < CURRENT_DATE)'.

wangshisan · 2019-12-19T10:08:58Z

This is also a requirement in our PROD environment, could anyone help review?

github-actions · 2020-03-29T00:11:00Z

We're closing this PR because it hasn't been updated in a while. This isn't a judgement on the merit of the PR in any way. It's just a way of keeping the PR queue manageable.
If you'd like to revive this PR, please reopen it and ask a committer to remove the Stale tag!

weixiuli mentioned this pull request Oct 28, 2019

[SPARK-14922][SPARK-17732][SPARK-23866][SQL] Support partition filters in ALTER TABLE DROP PARTITION #20999

Closed

weixiuli force-pushed the issues/SPARK-14922 branch 2 times, most recently from a4e67c3 to ce453ed Compare October 29, 2019 03:18

dongjoon-hyun added the SQL label Oct 30, 2019

weixiuli force-pushed the issues/SPARK-14922 branch from ce453ed to a0c42fd Compare October 31, 2019 06:51

weixiuli force-pushed the issues/SPARK-14922 branch from a0c42fd to 1bd2cd3 Compare November 1, 2019 10:16

weixiuli force-pushed the issues/SPARK-14922 branch from 1bd2cd3 to e27f13c Compare November 14, 2019 03:39

weixiuli force-pushed the issues/SPARK-14922 branch from e27f13c to 21cd780 Compare November 14, 2019 06:39

weixiuli force-pushed the issues/SPARK-14922 branch from 21cd780 to 0d5bc03 Compare November 14, 2019 09:46

weixiuli force-pushed the issues/SPARK-14922 branch from 0d5bc03 to 11b4cb0 Compare November 15, 2019 03:39

weixiuli force-pushed the issues/SPARK-14922 branch from 11b4cb0 to 9d3ede5 Compare November 15, 2019 06:40

[SPARK-14922][SPARK-17732][SPARK-23866][SQL] Support partition filter…

c2639d0

…s in ALTER TABLE DROP PARTITION and batch dropping PARTITIONS - Support partition filters in ALTER TABLE DROP PARTITION. - Support batch dropping PARTITIONS by 'ExpressionBuilder'.

weixiuli force-pushed the issues/SPARK-14922 branch from 9d3ede5 to c2639d0 Compare November 19, 2019 05:57

wangshisan reviewed Dec 19, 2019

View reviewed changes

github-actions bot added the Stale label Mar 29, 2020

github-actions bot closed this Mar 30, 2020

weixiuli mentioned this pull request Dec 13, 2020

SPARK-33691: Support partition filters in ALTER TABLE DROP PARTITION #30733

Closed

[SPARK-14922][SPARK-17732][SPARK-23866][SQL] Support partition filter in ALTER TABLE DROP PARTITION and batch dropping PARTITIONS #26280

[SPARK-14922][SPARK-17732][SPARK-23866][SQL] Support partition filter in ALTER TABLE DROP PARTITION and batch dropping PARTITIONS #26280

Uh oh!

Conversation

weixiuli commented Oct 28, 2019 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

What changes were proposed in this pull request?

How was this patch tested?

Uh oh!

mgaido91 commented Oct 28, 2019

Uh oh!

weixiuli commented Oct 29, 2019

Uh oh!

maropu commented Oct 29, 2019

Uh oh!

SparkQA commented Oct 29, 2019

Uh oh!

weixiuli commented Oct 29, 2019

Uh oh!

weixiuli commented Oct 29, 2019

Uh oh!

maropu commented Oct 29, 2019

Uh oh!

SparkQA commented Oct 29, 2019

Uh oh!

weixiuli commented Oct 29, 2019

Uh oh!

wangyum commented Oct 30, 2019

Uh oh!

SparkQA commented Oct 30, 2019

Uh oh!

SparkQA commented Oct 31, 2019

Uh oh!

weixiuli commented Oct 31, 2019

Uh oh!

maropu commented Oct 31, 2019

Uh oh!

SparkQA commented Oct 31, 2019

Uh oh!

SparkQA commented Nov 1, 2019

Uh oh!

SparkQA commented Nov 14, 2019

Uh oh!

weixiuli commented Nov 14, 2019

Uh oh!

SparkQA commented Nov 14, 2019

Uh oh!

SparkQA commented Nov 14, 2019

Uh oh!

SparkQA commented Nov 15, 2019

Uh oh!

SparkQA commented Nov 15, 2019

Uh oh!

weixiuli commented Nov 18, 2019 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

SparkQA commented Nov 19, 2019

Uh oh!

weixiuli commented Nov 19, 2019

Uh oh!

wangshisan Dec 19, 2019

Choose a reason for hiding this comment

Uh oh!

wangshisan commented Dec 19, 2019

Uh oh!

github-actions bot commented Mar 29, 2020

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

7 participants

weixiuli commented Oct 28, 2019 •

edited

Loading

weixiuli commented Nov 18, 2019 •

edited

Loading