[SPARK-33758][SQL] Prune unrequired partitionings from AliasAwareOutputPartitionings when some columns are dropped from projection

[prakharjain09]

What changes were proposed in this pull request?

This PR tries to prune the unrequired output partitionings in cases when the columns are dropped from Project/Aggregates etc.

Why are the changes needed?

Consider this query:
select t1.id from t1 JOIN t2 on t1.id = t2.id

This query will have top level Project node which will just project t1.id. But the outputPartitioning of this project node will be: PartitioningCollection(HashPartitioning(t1.id), HashPartitioning(t2.id)).

But since we are not propagating t2.id column, so we can drop HashPartitioning(t2.id) from the output partitioning of Project node.

Does this PR introduce any user-facing change?

No

How was this patch tested?

Added UTs.

[prakharjain09]

cc - @maropu @cloud-fan

[cloud-fan]

ok to test

[cloud-fan]

add to whitelist

[SparkQA]

Kubernetes integration test starting
URL: https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder-K8s/37407/

[SparkQA]

Kubernetes integration test status failure
URL: https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder-K8s/37407/

[SparkQA]

Kubernetes integration test starting
URL: https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder-K8s/37410/

[SparkQA]

Kubernetes integration test status failure
URL: https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder-K8s/37410/

[prakharjain09]

Thanks @cloud-fan for the review. I have addressed your comments.

[SparkQA]

Test build #132805 has finished for PR 30762 at commit c33d418.

• This patch fails SparkR unit tests.
• This patch merges cleanly.
• This patch adds no public classes.

[SparkQA]

Test build #132808 has finished for PR 30762 at commit 09b4673.

• This patch passes all tests.
• This patch merges cleanly.
• This patch adds the following public classes (experimental):
• case class SparkPod(pod: Pod, container: Container)
• trait KubernetesFeatureConfigStep
• public class Distributions
• case class AlterTableRecoverPartitions(child: LogicalPlan) extends Command

[cloud-fan]

thanks, merging to master!

[cloud-fan]

BTW, shall we do the same to output ordering? We can prune SortOrder.sameOrderExpressions

[prakharjain09]

BTW, shall we do the same to output ordering? We can prune SortOrder.sameOrderExpressions

@cloud-fan Sure. Will make changes for it and raise PR in couple of days.

[maropu]

late lgtm. Thanks for fixing it, @prakharjain09