[SPARK-32448][K8S][TESTS] Use single version for exec-maven-plugin/scalatest-maven-plugin #29248

dongjoon-hyun · 2020-07-26T22:28:32Z

What changes were proposed in this pull request?

Two different versions are used for the same artifacts, exec-maven-plugin and scalatest-maven-plugin. This PR aims to use the same versions for exec-maven-plugin and scalatest-maven-plugin. In addition, this PR removes scala-maven-plugin.version from K8s integration suite because it's unused.

Why are the changes needed?

This will prevent the mistake which upgrades only one place and forgets the others.

Does this PR introduce any user-facing change?

No.

How was this patch tested?

Pass the Jenkins K8S IT.

…alatest-maven-plugin

dongjoon-hyun · 2020-07-26T22:32:07Z

resource-managers/kubernetes/integration-tests/pom.xml

-    <exec-maven-plugin.version>1.4.0</exec-maven-plugin.version>
    <extraScalaTestArgs></extraScalaTestArgs>
    <kubernetes-client.version>4.9.2</kubernetes-client.version>
-    <scala-maven-plugin.version>3.2.2</scala-maven-plugin.version>


This is unused.

SparkQA · 2020-07-26T23:11:38Z

Kubernetes integration test starting
URL: https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder-K8s/31208/

SparkQA · 2020-07-26T23:34:30Z

Kubernetes integration test status failure
URL: https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder-K8s/31208/

dongjoon-hyun · 2020-07-26T23:37:22Z

Actually, this PR's change passed Jenkins K8s IT because it triggers all tests successfully. The one test case failure is irrelevant and flaky one.

KubernetesSuite:
- Run SparkPi with no resources *** FAILED ***
  The code passed to eventually never returned normally. Attempted 190 times over 3.0022143221666666 minutes. Last failure message: false was not true. (KubernetesSuite.scala:388)
- Run SparkPi with a very long application name.
- Use SparkLauncher.NO_RESOURCE
- Run SparkPi with a master URL without a scheme.
- Run SparkPi with an argument.
- Run SparkPi with custom labels, annotations, and environment variables.
- All pods have the same service account by default
- Run extraJVMOptions check on driver
- Run SparkRemoteFileTest using a remote data file
- Run SparkPi with env and mount secrets.
- Run PySpark on simple pi.py example
- Run PySpark with Python3 to test a pyfiles example
- Run PySpark with memory customization
- Run in client mode.
- Start pod creation from template
- PVs with local storage
- Launcher client dependencies
- Test basic decommissioning
Run completed in 15 minutes, 37 seconds.
Total number of tests run: 18
Suites: completed 2, aborted 0
Tests: succeeded 17, failed 1, canceled 0, ignored 0, pending 0
*** 1 TEST FAILED ***

dongjoon-hyun · 2020-07-26T23:37:48Z

Retest this please.

dongjoon-hyun · 2020-07-26T23:44:46Z

resource-managers/kubernetes/integration-tests/pom.xml

  <artifactId>spark-kubernetes-integration-tests_2.12</artifactId>
  <properties>
    <download-maven-plugin.version>1.3.0</download-maven-plugin.version>
-    <exec-maven-plugin.version>1.4.0</exec-maven-plugin.version>


This is outdated.

dongjoon-hyun · 2020-07-26T23:44:58Z

resource-managers/kubernetes/integration-tests/pom.xml

    <extraScalaTestArgs></extraScalaTestArgs>
    <kubernetes-client.version>4.9.2</kubernetes-client.version>
-    <scala-maven-plugin.version>3.2.2</scala-maven-plugin.version>
-    <scalatest-maven-plugin.version>1.0</scalatest-maven-plugin.version>


This is outdated.

SparkQA · 2020-07-27T00:23:01Z

Kubernetes integration test starting
URL: https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder-K8s/31213/

SparkQA · 2020-07-27T00:23:16Z

Kubernetes integration test status failure
URL: https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder-K8s/31213/

dongjoon-hyun · 2020-07-27T00:25:37Z

GitHub Actions passed (except SparkR due to [SPARK-32451][R] Support Apache Arrow 1.0.0 #29252 ).
K8s IT already passed at the first attempt. The second K8s IT attempt failed with the following minikube issue.

stderr: error execution phase certs/etcd-server: failed to write certificate "etcd-server": failure loading etcd/server certificate: failed to load certificate: the certificate has expired
: Process exited with status 1

dongjoon-hyun · 2020-07-27T00:28:31Z

Hi, @liyinan926 . Could you review this K8s IT dependency change PR? This PR upgrades the old versions and refactors them in order to avoid being outdate again.

dongjoon-hyun · 2020-07-27T01:54:21Z

Retest this please.

dongjoon-hyun · 2020-07-27T02:21:06Z

Hi, @HyukjinKwon and @viirya . Could you review this PR? This is simply in order to update the dependency by removing the duplications. And, it's tested already but it's retriggered due to the flakiness in K8s minikube installation in Jenkins.

dongjoon-hyun · 2020-07-27T02:25:17Z

Thank you so much, @HyukjinKwon .
Merged to master/3.0

dongjoon-hyun · 2020-07-27T02:26:16Z

Oh, Thank you, @viirya , too!

SparkQA · 2020-07-27T02:44:37Z

Kubernetes integration test starting
URL: https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder-K8s/31222/

SparkQA · 2020-07-27T02:44:53Z

Kubernetes integration test status failure
URL: https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder-K8s/31222/

dongjoon-hyun · 2020-07-27T04:25:11Z

For a record, the following is the K8s minikube starts failure log.

+ minikube --vm-driver=kvm2 start --memory 6000 --cpus 8
There is a newer version of minikube available (v1.12.1).  Download it here:
https://github.com/kubernetes/minikube/releases/tag/v1.12.1

To disable this notification, run the following:
minikube config set WantUpdateNotification false
o   minikube v0.34.1 on linux (amd64)
i   Tip: Use 'minikube start -p <name>' to create a new cluster, or 'minikube delete' to delete this one.
:   Re-using the currently running kvm2 VM for "minikube" ...
:   Waiting for SSH access ...
-   "minikube" IP address is 192.168.39.69
-   Configuring Docker as the container runtime ...
-   Preparing Kubernetes environment ...
-   Pulling images required by Kubernetes v1.13.3 ...
:   Relaunching Kubernetes v1.13.3 using kubeadm ... 
!   Error restarting cluster: running cmd: sudo kubeadm init phase certs all --config /var/lib/kubeadm.yaml: command failed: sudo kubeadm init phase certs all --config /var/lib/kubeadm.yaml
stdout: [certs] Using certificateDir folder "/var/lib/minikube/certs/"
[certs] Using existing ca certificate authority
[certs] Using existing apiserver certificate and key on disk

stderr: error execution phase certs/apiserver-kubelet-client: failed to write certificate "apiserver-kubelet-client": failure loading apiserver-kubelet-client certificate: failed to load certificate: the certificate has expired
: Process exited with status 1

*   Sorry that minikube crashed. If this was unexpected, we would love to hear from you:
-   https://github.com/kubernetes/minikube/issues/new

…alatest-maven-plugin ### What changes were proposed in this pull request? Two different versions are used for the same artifacts, `exec-maven-plugin` and `scalatest-maven-plugin`. This PR aims to use the same versions for `exec-maven-plugin` and `scalatest-maven-plugin`. In addition, this PR removes `scala-maven-plugin.version` from `K8s` integration suite because it's unused. ### Why are the changes needed? This will prevent the mistake which upgrades only one place and forgets the others. ### Does this PR introduce _any_ user-facing change? No. ### How was this patch tested? Pass the Jenkins K8S IT. Closes #29248 from dongjoon-hyun/SPARK-32448. Authored-by: Dongjoon Hyun <[email protected]> Signed-off-by: Dongjoon Hyun <[email protected]> (cherry picked from commit 13c64c2) Signed-off-by: Dongjoon Hyun <[email protected]>

SparkQA · 2020-07-27T06:31:41Z

Test build #126585 has finished for PR 29248 at commit 39021ee.

This patch passes all tests.
This patch merges cleanly.
This patch adds no public classes.

SparkQA · 2020-07-27T07:05:03Z

Test build #126590 has finished for PR 29248 at commit 39021ee.

This patch fails due to an unknown error code, -9.
This patch merges cleanly.
This patch adds no public classes.

SparkQA · 2020-07-27T07:05:03Z

Test build #126600 has finished for PR 29248 at commit 39021ee.

This patch fails due to an unknown error code, -9.
This patch merges cleanly.
This patch adds no public classes.

[SPARK-32448][K8S][TESTS] Use single version for exec-maven-plugin/sc…

39021ee

…alatest-maven-plugin

probot-autolabeler bot added BUILD KUBERNETES labels Jul 26, 2020

dongjoon-hyun commented Jul 26, 2020

View reviewed changes

HyukjinKwon approved these changes Jul 27, 2020

View reviewed changes

dongjoon-hyun closed this in 13c64c2 Jul 27, 2020

viirya approved these changes Jul 27, 2020

View reviewed changes

dongjoon-hyun deleted the SPARK-32448 branch July 27, 2020 05:33

[SPARK-32448][K8S][TESTS] Use single version for exec-maven-plugin/scalatest-maven-plugin #29248

[SPARK-32448][K8S][TESTS] Use single version for exec-maven-plugin/scalatest-maven-plugin #29248

Uh oh!

Conversation

dongjoon-hyun commented Jul 26, 2020 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

What changes were proposed in this pull request?

Why are the changes needed?

Does this PR introduce any user-facing change?

How was this patch tested?

Uh oh!

dongjoon-hyun Jul 26, 2020 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

SparkQA commented Jul 26, 2020

Uh oh!

SparkQA commented Jul 26, 2020

Uh oh!

dongjoon-hyun commented Jul 26, 2020

Uh oh!

dongjoon-hyun commented Jul 26, 2020

Uh oh!

dongjoon-hyun Jul 26, 2020

Choose a reason for hiding this comment

Uh oh!

dongjoon-hyun Jul 26, 2020

Choose a reason for hiding this comment

Uh oh!

SparkQA commented Jul 27, 2020

Uh oh!

SparkQA commented Jul 27, 2020

Uh oh!

dongjoon-hyun commented Jul 27, 2020

Uh oh!

dongjoon-hyun commented Jul 27, 2020

Uh oh!

dongjoon-hyun commented Jul 27, 2020

Uh oh!

dongjoon-hyun commented Jul 27, 2020 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

dongjoon-hyun commented Jul 27, 2020 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

dongjoon-hyun commented Jul 27, 2020

Uh oh!

SparkQA commented Jul 27, 2020

Uh oh!

SparkQA commented Jul 27, 2020

Uh oh!

dongjoon-hyun commented Jul 27, 2020

Uh oh!

SparkQA commented Jul 27, 2020

Uh oh!

SparkQA commented Jul 27, 2020

Uh oh!

SparkQA commented Jul 27, 2020

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

4 participants

dongjoon-hyun commented Jul 26, 2020 •

edited

Loading

dongjoon-hyun Jul 26, 2020 •

edited

Loading

dongjoon-hyun commented Jul 27, 2020 •

edited

Loading

dongjoon-hyun commented Jul 27, 2020 •

edited

Loading