Spark 3.4: Backport: Add procedure and action to compute partition stats.

[slfan1989]

backport #12450, #13480 to Spark3.4

[slfan1989]

@huaxingao While working on the backport of #13956 (Add Support for PartitionStatistics Files in RewriteTablePath), I noticed that the compute partition stats functionality is incomplete in the Spark 3.4 branch.

This issue surfaced because our unit tests rely on methods that compute partition statistics.

This functionality has already been implemented in Spark 4.0 and Spark 3.5, but the Spark 3.4 branch is still missing two key PRs: #12450 and #13480.

To address this, I have created a corresponding backport PR to bring this functionality into Spark 3.4.

Could you please take a look at this PR? Thank you very much!

[huaxingao]

cc @ajantha-bhat Are there any reason why partition stats stored procedure is not supported in 3.4?

[slfan1989]

@huaxingao Thanks for your comment. I'd like to add some additional context: In #12450, it was mentioned that the change needs to be backported to Spark 3.4, but it seems that hasn't been done yet—perhaps the original author may have overlooked it. Additionally, since this functionality is part of an Iceberg Core improvement, it should theoretically be compatible across different Spark versions.

ajantha-bhat commented on May 15

[huaxingao]

@slfan1989 I reviewed the changes: they look like a clean back-port. I just want to double check with you whether there are any version specific changes to be aware of.

[slfan1989]

@slfan1989 I reviewed the changes: they look like a clean back-port. I just want to double check with you whether there are any version specific changes to be aware of.

@huaxingao Thank you very much for the review! I also believe this backport is fine. When creating this backport PR, we can directly cherry-pick the code from the Spark 3.5 branch without any additional modifications. All related unit tests have passed and the results are as expected.

[slfan1989]

@huaxingao Can we continue moving forward with this PR?

[huaxingao]

LGTM

[huaxingao]

Thanks @slfan1989 for the PR!

[slfan1989]

Thanks @slfan1989 for the PR!

@huaxingao Thank you very much for reviewing the code!

[kevinjqliu]

hi @slfan1989
we're gathering feedback on removing spark 3.4 in the upcoming 1.11 release.
could you chime in on the slack thread?

[slfan1989]

hi @slfan1989
we're gathering feedback on removing spark 3.4 in the upcoming 1.11 release.
could you chime in on the slack thread?

@kevinjqliu Thank you for inviting me to participate in this discussion. I agree with removing Spark 3.4 in the next release and have already replied on Slack.