[SPARK-33809][SQL] Support all_columns_except complex column expression

[AngersZhuuuu]

What changes were proposed in this pull request?

We are easy to drop some column when use DF/DS api drop but difficult when use SQL when there are so many columns.
Add a function to support this:

 usage = """
    _FUNC_(expr*) - Returns the columns except columns given in function parameters.
  """,
  examples = """
    Examples:
      > select * FROM TBL;
       A B C
       1 2 3
      > SELECT _FUNC_(a, b) FROM TBL;
       C
       3

Idea from a Chinese blog
https://mp.weixin.qq.com/s?src=11&timestamp=1608729276&ver=2784&signature=IMj38*tURJObyQv8z84ELR-Vfkw2becwRfzQ5*9p0X7ilIp-AJVmIFUGOHOr6DcEF5syqfEhiQHa87ZAdzQJHWmy7FjjvTYdEldLkc9GsqrOELpLQrQ5CVG2KdK0ZCgP&new=1

Why are the changes needed?

Provide a way to handle drop column in SQL

Does this PR introduce any user-facing change?

Yes, user can use all_columns_except func in projection list to handle complex drop column scenario

How was this patch tested?

Added UT

[SparkQA]

Test build #132893 has finished for PR 30805 at commit 7f98b93.

• This patch fails to build.
• This patch merges cleanly.
• This patch adds the following public classes (experimental):
• case class AllColumnsExcept(exclude: Expression*) extends Expression with CodegenFallback

[SparkQA]

Kubernetes integration test starting
URL: https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder-K8s/37495/

[SparkQA]

Kubernetes integration test status failure
URL: https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder-K8s/37495/

[HyukjinKwon]

@AngersZhuuuu, do you have any reference of all_columns_except function in other DBMSes?

[AngersZhuuuu]

@AngersZhuuuu, do you have any reference of all_columns_except function in other DBMSes?

https://stackoverflow.com/questions/729197/sql-exclude-a-column-using-select-except-columna-from-tablea
https://dataschool.com/learn-sql/exclude-a-column/
https://www.postgresonline.com/journal/archives/41-How-to-SELECT--ALL-EXCEPT-some-columns-in-a-table.html

Didn't find any DBMS support such feature yet, but I think many people need this feature. And it make our user convenient to write sql easier and convert DF/DS api to SQL.

[wangyum]

How about?

set spark.sql.parser.quotedRegexColumnNames=true;

EXPLAIN SELECT `(c_customer_sk|c_customer_id|c_current_cdemo_sk|c_current_hdemo_sk|c_current_addr_sk|c_first_shipto_date_sk|c_first_sales_date_sk)?+.+` FROM customer;

== Physical Plan ==
AdaptiveSparkPlan(isFinalPlan=false)
+- Project [c_salutation#1604, c_first_name#1605, c_last_name#1606, c_preferred_cust_flag#1607, c_birth_day#1608, c_birth_month#1609, c_birth_year#1610, c_birth_country#1611, c_login#1612, c_email_address#1613, c_last_review_date#1614]
   +- FileScan parquet carmel_tpcds5t.customer

EXPLAIN SELECT * FROM customer;
== Physical Plan ==
AdaptiveSparkPlan(isFinalPlan=false)
+- FileScan parquet carmel_tpcds5t.customer[c_customer_sk#1597, c_customer_id#1598, c_current_cdemo_sk#1599, c_current_hdemo_sk#1600, c_current_addr_sk#1601, c_first_shipto_date_sk#1602, c_first_sales_date_sk#1603, c_salutation#1604, c_first_name#1605, c_last_name#1606, c_preferred_cust_flag#1607, c_birth_day#1608, c_birth_month#1609, c_birth_year#1610, c_birth_country#1611, c_login#1612, c_email_address#1613, c_last_review_date#1614] Batched: true, DataFilters: [], Format: Parquet,

[wangyum]

Can we add the usage of spark.sql.parser.quotedRegexColumnNames to document?

[AngersZhuuuu]

quotedRegexColumnNames

I think it's necessary. Very useful

[SparkQA]

Test build #132913 has finished for PR 30805 at commit aedf260.

• This patch fails Spark unit tests.
• This patch merges cleanly.
• This patch adds no public classes.

[da-tubi]

@AngersZhuuuu
It would be better if you mention about the post where the original idea of all_columns_except comes from:

https://mp.weixin.qq.com/s?src=11&timestamp=1608729276&ver=2784&signature=IMj38*tURJObyQv8z84ELR-Vfkw2becwRfzQ5*9p0X7ilIp-AJVmIFUGOHOr6DcEF5syqfEhiQHa87ZAdzQJHWmy7FjjvTYdEldLkc9GsqrOELpLQrQ5CVG2KdK0ZCgP&new=1

Anyway, thanks for your work to bring it to Apache Spark.

@wangyum
It's OK to use quotedRegexColumnNames, but the syntax is a bit weird!

[AngersZhuuuu]

@AngersZhuuuu
It would be better if you mention about the post where the original idea of all_columns_except comes from:

https://mp.weixin.qq.com/s?src=11&timestamp=1608729276&ver=2784&signature=IMj38*tURJObyQv8z84ELR-Vfkw2becwRfzQ5*9p0X7ilIp-AJVmIFUGOHOr6DcEF5syqfEhiQHa87ZAdzQJHWmy7FjjvTYdEldLkc9GsqrOELpLQrQ5CVG2KdK0ZCgP&new=1

Anyway, thanks for your work to bring it to Apache Spark.

@wangyum
It's OK to use quotedRegexColumnNames, but the syntax is a bit weird!

Since it's a Chinese blog, so I didn't put it in pr desc. Updated to the pr desc