[WIP][SPARK-35954] Upgrade Apache Curator Dependency to 4.2.0

[nirousseau]

WIP / DESCRIPTION TBD

Issue : https://issues.apache.org/jira/browse/SPARK-35954

[dongjoon-hyun]

cc @sunchao since @nirousseau wrote the following in the JIRA.

Spark 3.2.0 aims to support Hadoop 3.3.1 ; this Hadoop release bundles the following shared libraries :
curator-client-4.2.0.jar
curator-framework-4.2.0.jar
curator-recipes-4.2.0.jar
zookeeper-3.5.6.jar
zookeeper-jute-3.5.6.jar

[dongjoon-hyun]

ok to test

[sunchao]

Thanks @dongjoon-hyun for pinging. Yes I think it's a good idea to keep the curator (and zookeeper?) version in sync with what come from Hadoop, similar to #28897.

[dongjoon-hyun]

Thank you for the confirmation, @sunchao .

[SparkQA]

Kubernetes integration test starting
URL: https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder-K8s/44995/

[SparkQA]

Kubernetes integration test status success
URL: https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder-K8s/44995/

[SparkQA]

Test build #140483 has finished for PR 33157 at commit 8a274c4.

• This patch fails Spark unit tests.
• This patch merges cleanly.
• This patch adds no public classes.

[AmplabJenkins]

Can one of the admins verify this patch?

[nirousseau]

as expected, it breaks tests, I will try to setup a local dev env and fix library changes asap.

[dongjoon-hyun]

Thank you for working on this, @nirousseau .

as expected, it breaks tests, I will try to setup a local dev env and fix library changes asap.

[dongjoon-hyun]

I'll close this PR for now, @nirousseau . Feel free to reopen when the PR is ready. Thanks.