Skip to content

[SPARK-27812][K8S] Bump K8S client version to 4.6.1 #26093

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Closed
igorcalabria wants to merge 6 commits into from
Closed

[SPARK-27812][K8S] Bump K8S client version to 4.6.1 #26093

Show file tree
Hide file tree
Changes from 1 commit
Commits
Show all changes
6 commits
Select commit Hold shift + click to select a range
7f15bac
Bump kubernetes client version
igorcalabria Oct 11, 2019
7148bc8
bump k8s client version in integration tests
igorcalabria Oct 11, 2019
5d6bcd7
bump okhttp version to 3.12
igorcalabria Oct 14, 2019
b31b375
update dep manifests
igorcalabria Oct 14, 2019
d1e1b52
bump kubernetes-client to 4.6.1
igorcalabria Oct 17, 2019
a5eb702
remove explict okhttp version
igorcalabria Oct 17, 2019
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
7 changes: 3 additions & 4 deletions dev/deps/spark-deps-hadoop-2.7
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -106,7 +106,6 @@ jakarta.ws.rs-api-2.1.5.jar
jakarta.xml.bind-api-2.3.2.jar
janino-3.0.15.jar
javassist-3.22.0-CR2.jar
javax.el-3.0.1-b11.jar
javax.inject-1.jar
javax.servlet-api-3.1.0.jar
javolution-5.5.1.jar
Expand Down Expand Up @@ -137,9 +136,9 @@ jsr305-3.0.0.jar
jta-1.1.jar
jul-to-slf4j-1.7.16.jar
kryo-shaded-4.0.2.jar
kubernetes-client-4.4.2.jar
kubernetes-model-4.4.2.jar
kubernetes-model-common-4.4.2.jar
kubernetes-client-4.6.0.jar
kubernetes-model-4.6.0.jar
kubernetes-model-common-4.6.0.jar
leveldbjni-all-1.8.jar
libfb303-0.9.3.jar
libthrift-0.12.0.jar
Expand Down
7 changes: 3 additions & 4 deletions dev/deps/spark-deps-hadoop-3.2
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -106,7 +106,6 @@ jakarta.ws.rs-api-2.1.5.jar
jakarta.xml.bind-api-2.3.2.jar
janino-3.0.15.jar
javassist-3.22.0-CR2.jar
javax.el-3.0.1-b11.jar
javax.inject-1.jar
javax.servlet-api-3.1.0.jar
javolution-5.5.1.jar
Expand Down Expand Up @@ -152,9 +151,9 @@ kerby-pkix-1.0.1.jar
kerby-util-1.0.1.jar
kerby-xdr-1.0.1.jar
kryo-shaded-4.0.2.jar
kubernetes-client-4.4.2.jar
kubernetes-model-4.4.2.jar
kubernetes-model-common-4.4.2.jar
kubernetes-client-4.6.0.jar
kubernetes-model-4.6.0.jar
kubernetes-model-common-4.6.0.jar
leveldbjni-all-1.8.jar
libfb303-0.9.3.jar
libthrift-0.12.0.jar
Expand Down
2 changes: 1 addition & 1 deletion resource-managers/kubernetes/core/pom.xml
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -29,7 +29,7 @@
<name>Spark Project Kubernetes</name>
<properties>
<sbt.project.name>kubernetes</sbt.project.name>
<kubernetes.client.version>4.4.2</kubernetes.client.version>
<kubernetes.client.version>4.6.0</kubernetes.client.version>

@ifilonenko ifilonenko Oct 11, 2019

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

You need to change the kubernetes.client.version in resource-managers/kubernetes/integration-tests/pom.xml as well.

and bump:

    <dependency>
      <groupId>com.squareup.okhttp3</groupId>
      <artifactId>okhttp</artifactId>
      - <version>3.8.1</version>
      + <version>3.12.0</version>
    </dependency>

as I believe there is a dependency here https://mvnrepository.com/artifact/io.fabric8/kubernetes-client/4.6.0

@igorcalabria igorcalabria Oct 11, 2019

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Ok. Is there a reason why okhttp is added as an explicit dependency(kubernetes-client already brings it)? If we really need a specific version, shouldn't we at least exclude it from kubernetes-client? I'm saying this because it's very easy to end up with a unexpected version on the classpath. kubernetes-client uses okhttp's version 3.12 since version 4.1.2 but we're only updating it now.

@srowen srowen Oct 14, 2019

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

It may have been to make sure that another dependency on okhttp didn't win out and override to a lower version. It can just be updated here, or you can try removing it and examining the transitive dependencies afterwards. If it's already at 3.12.0, then it can likely be omitted.

@ifilonenko ifilonenko Oct 14, 2019

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

That is a good point, @mccheah do we have an opinion on this? I am okay with removing the explicit dependency since kubernetes-client already brings it in. If there was a reason, I can't seem to remember, to explicitly state, let's exclude it?

@dongjoon-hyun dongjoon-hyun Oct 15, 2019

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This one looks okay to me.

@erikerlandson erikerlandson Oct 15, 2019

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

agreed we should remove explicit dep to avoid a mismatch failure in the future

</properties>

<dependencies>
Expand Down