[SPARK-28921][K8S][FOLLOWUP] Also bump K8S client version in integration-tests

[srowen]

What changes were proposed in this pull request?

Per #25640 (comment) also bump K8S client version in integration-tests module.

Why are the changes needed?

Harmonize the version as intended.

Does this PR introduce any user-facing change?

No

How was this patch tested?

Existing tests.

[srowen]

@andygrove @shaneknapp @skonto not sure if you all know what to make of this test failure?

lsof: WARNING: can't stat() aufs file system /var/lib/docker/aufs/mnt/04f77d5c3184798b175981cb930d3537efa8ab1dac04588c7cbd418972f0688c
lsof: WARNING: can't stat() aufs file system /home/docker/aufs/mnt/04f77d5c3184798b175981cb930d3537efa8ab1dac04588c7cbd418972f0688c
lsof: WARNING: can't stat() tmpfs file system /var/lib/docker/containers/805a48533fb8d67174d0a268446a7886192a6b21fca4760c199bc9293273489c/mounts/shm
lsof: WARNING: can't stat() tmpfs file system /home/docker/containers/805a48533fb8d67174d0a268446a7886192a6b21fca4760c199bc9293273489c/mounts/shm

Seems more env-related than anything?

[andygrove]

@srowen yes, I agree, this is something env-related

[SparkQA]

Test build #110048 has finished for PR 25664 at commit bf41b94.

• This patch passes all tests.
• This patch merges cleanly.
• This patch adds no public classes.

[srowen]

Hm, it didn't post the link to the failed test:
https://amplab.cs.berkeley.edu/jenkins//job/testing-k8s-prb-make-spark-distribution-unified/15067/

(The pass above is just for the rest of Spark which should not be affected.)

I know we need to harmonize this version though. I'm tempted to just merge it as we don't have a reason to believe it's a legit failure, and the update is important.

[skonto]

@srowen we should trigger the K8s integrations tests just in case here... would also be good to verify once more https://issues.apache.org/jira/browse/SPARK-28953 is not happening upstream.
This should also be backported to 2.4.3, 2.4.4 according to the related PRs.

[srowen]

How do I retrigger it?
Are you saying the issue you cite might cause this?
BTW see #25599

[shaneknapp]

@andygrove @shaneknapp @skonto not sure if you all know what to make of this test failure?

lsof: WARNING: can't stat() aufs file system /var/lib/docker/aufs/mnt/04f77d5c3184798b175981cb930d3537efa8ab1dac04588c7cbd418972f0688c
lsof: WARNING: can't stat() aufs file system /home/docker/aufs/mnt/04f77d5c3184798b175981cb930d3537efa8ab1dac04588c7cbd418972f0688c
lsof: WARNING: can't stat() tmpfs file system /var/lib/docker/containers/805a48533fb8d67174d0a268446a7886192a6b21fca4760c199bc9293273489c/mounts/shm
lsof: WARNING: can't stat() tmpfs file system /home/docker/containers/805a48533fb8d67174d0a268446a7886192a6b21fca4760c199bc9293273489c/mounts/shm

Seems more env-related than anything?

you can safely ignore these...

[shaneknapp]

test this please

[shaneknapp]

k8s integration tests running here: https://amplab.cs.berkeley.edu/jenkins/job/testing-k8s-prb-make-spark-distribution-unified/15147/

[SparkQA]

Kubernetes integration test starting
URL: https://amplab.cs.berkeley.edu/jenkins/job/testing-k8s-prb-make-spark-distribution-unified/15147/

[SparkQA]

Kubernetes integration test status success
URL: https://amplab.cs.berkeley.edu/jenkins/job/testing-k8s-prb-make-spark-distribution-unified/15147/

[SparkQA]

Test build #110138 has finished for PR 25664 at commit bf41b94.

• This patch passes all tests.
• This patch merges cleanly.
• This patch adds no public classes.

[srowen]

Merged to master/2.4