[SPARK-23984][K8S][Test] Added Integration Tests for PySpark on Kubernetes #21583

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Closed

ifilonenko wants to merge 5 commits into apache:master from ifilonenko:master

Contributor

ifilonenko commented Jun 18, 2018

What changes were proposed in this pull request?

I added integration tests for PySpark ( + checking JVM options + RemoteFileTest) which wasn't properly merged in the initial integration test PR.

How was this patch tested?

I tested this with integration tests using:

dev/dev-run-integration-tests.sh --spark-tgz spark-2.4.0-SNAPSHOT-bin-2.7.3.tgz


          Fixed Remote File tests and added PySpark tests

2707dee

Contributor Author

ifilonenko commented Jun 18, 2018 •

edited

Loading

@ssuchter @holdenk @mccheah @shaneknapp for review

Contributor Author

ifilonenko commented Jun 18, 2018

I don't know if Jenkins build a distribution with --pip. TBD based on success. But locally, this worked when I ran:
dev/make-distribution.sh --pip --tgz -Phadoop-2.7 -Pkubernetes

SparkQA commented Jun 18, 2018

Kubernetes integration test starting
URL: https://amplab.cs.berkeley.edu/jenkins/job/testing-k8s-prb-make-spark-distribution-unified/265/

SparkQA commented Jun 18, 2018

Kubernetes integration test status failure
URL: https://amplab.cs.berkeley.edu/jenkins/job/testing-k8s-prb-make-spark-distribution-unified/265/

SparkQA commented Jun 18, 2018

Test build #92012 has finished for PR 21583 at commit 2707dee.

This patch passes all tests.
This patch merges cleanly.
This patch adds no public classes.

Contributor Author

ifilonenko commented Jun 18, 2018

retest this please

1 similar comment

Contributor Author

ifilonenko commented Jun 18, 2018

retest this please

SparkQA commented Jun 18, 2018

Test build #92013 has finished for PR 21583 at commit 2707dee.

This patch passes all tests.
This patch merges cleanly.
This patch adds no public classes.

SparkQA commented Jun 18, 2018

Kubernetes integration test starting
URL: https://amplab.cs.berkeley.edu/jenkins/job/testing-k8s-prb-make-spark-distribution-unified/268/

SparkQA commented Jun 18, 2018

Kubernetes integration test status failure
URL: https://amplab.cs.berkeley.edu/jenkins/job/testing-k8s-prb-make-spark-distribution-unified/268/

SparkQA commented Jun 18, 2018

Test build #92014 has finished for PR 21583 at commit 2707dee.

This patch passes all tests.
This patch merges cleanly.
This patch adds no public classes.

Contributor Author

ifilonenko commented Jun 18, 2018

retest this please

SparkQA commented Jun 18, 2018

Kubernetes integration test starting
URL: https://amplab.cs.berkeley.edu/jenkins/job/testing-k8s-prb-make-spark-distribution-unified/283/

SparkQA commented Jun 18, 2018

Test build #92035 has finished for PR 21583 at commit 2707dee.

This patch passes all tests.
This patch merges cleanly.
This patch adds no public classes.

SparkQA commented Jun 18, 2018

Kubernetes integration test status failure
URL: https://amplab.cs.berkeley.edu/jenkins/job/testing-k8s-prb-make-spark-distribution-unified/283/

Contributor Author

ifilonenko commented Jun 18, 2018 •

edited

Loading

This PR is now just waiting on appropriate changes to bin/docker-image-tool.sh which are contained in the following PR: #21555 . Will wait until that is merged.

Contributor Author

ifilonenko commented Jun 21, 2018

retest this please

SparkQA commented Jun 21, 2018

Kubernetes integration test starting
URL: https://amplab.cs.berkeley.edu/jenkins/job/testing-k8s-prb-make-spark-distribution-unified/385/

SparkQA commented Jun 21, 2018

Kubernetes integration test status failure
URL: https://amplab.cs.berkeley.edu/jenkins/job/testing-k8s-prb-make-spark-distribution-unified/385/

SparkQA commented Jun 21, 2018

Test build #92182 has finished for PR 21583 at commit 2707dee.

This patch passes all tests.
This patch merges cleanly.
This patch adds no public classes.

Contributor Author

ifilonenko commented Jun 25, 2018

retest this please


          Merge branch 'master' of https://github.com/apache/spark

44ada92

SparkQA commented Jun 25, 2018

Kubernetes integration test starting
URL: https://amplab.cs.berkeley.edu/jenkins/job/testing-k8s-prb-make-spark-distribution-unified/461/

SparkQA commented Jun 25, 2018

Test build #92304 has finished for PR 21583 at commit 2707dee.

This patch passes all tests.
This patch merges cleanly.
This patch adds no public classes.

SparkQA commented Jun 25, 2018

Kubernetes integration test status failure
URL: https://amplab.cs.berkeley.edu/jenkins/job/testing-k8s-prb-make-spark-distribution-unified/461/

Contributor

foxish commented Jun 25, 2018

@ifilonenko, is this ready to go?


          modification to build args

dfd78d6

SparkQA commented Jun 29, 2018

Kubernetes integration test starting
URL: https://amplab.cs.berkeley.edu/jenkins/job/testing-k8s-prb-make-spark-distribution-unified/584/

SparkQA commented Jun 29, 2018

Kubernetes integration test status failure
URL: https://amplab.cs.berkeley.edu/jenkins/job/testing-k8s-prb-make-spark-distribution-unified/584/

SparkQA commented Jul 12, 2018

Kubernetes integration test status failure
URL: https://amplab.cs.berkeley.edu/jenkins/job/testing-k8s-prb-make-spark-distribution-unified/910/

SparkQA commented Jul 12, 2018

Test build #92952 has finished for PR 21583 at commit c0b5927.

This patch passes all tests.
This patch merges cleanly.
This patch adds no public classes.

Contributor

shaneknapp commented Jul 12, 2018

test this please

SparkQA commented Jul 12, 2018

Kubernetes integration test starting
URL: https://amplab.cs.berkeley.edu/jenkins/job/testing-k8s-prb-make-spark-distribution-unified/912/

SparkQA commented Jul 12, 2018

Kubernetes integration test status failure
URL: https://amplab.cs.berkeley.edu/jenkins/job/testing-k8s-prb-make-spark-distribution-unified/912/

SparkQA commented Jul 12, 2018

Test build #92955 has finished for PR 21583 at commit c0b5927.

This patch passes all tests.
This patch merges cleanly.
This patch adds no public classes.

Contributor

shaneknapp commented Jul 12, 2018

test this please

SparkQA commented Jul 12, 2018

Kubernetes integration test starting
URL: https://amplab.cs.berkeley.edu/jenkins/job/testing-k8s-prb-make-spark-distribution-unified/913/

SparkQA commented Jul 12, 2018

Kubernetes integration test status failure
URL: https://amplab.cs.berkeley.edu/jenkins/job/testing-k8s-prb-make-spark-distribution-unified/913/

Contributor

shaneknapp commented Jul 13, 2018

test this please

SparkQA commented Jul 13, 2018

Test build #92956 has finished for PR 21583 at commit c0b5927.

This patch passes all tests.
This patch merges cleanly.
This patch adds no public classes.

SparkQA commented Jul 13, 2018

Kubernetes integration test starting
URL: https://amplab.cs.berkeley.edu/jenkins/job/testing-k8s-prb-make-spark-distribution-unified/914/

SparkQA commented Jul 13, 2018

Kubernetes integration test status failure
URL: https://amplab.cs.berkeley.edu/jenkins/job/testing-k8s-prb-make-spark-distribution-unified/914/

SparkQA commented Jul 13, 2018

Test build #92957 has finished for PR 21583 at commit c0b5927.

This patch passes all tests.
This patch merges cleanly.
This patch adds no public classes.

Contributor

shaneknapp commented Jul 13, 2018

test this please

SparkQA commented Jul 13, 2018

Test build #92980 has finished for PR 21583 at commit c0b5927.

This patch passes all tests.
This patch merges cleanly.
This patch adds no public classes.

SparkQA commented Jul 13, 2018

Kubernetes integration test starting
URL: https://amplab.cs.berkeley.edu/jenkins/job/testing-k8s-prb-make-spark-distribution-unified/933/

SparkQA commented Jul 13, 2018

Kubernetes integration test status failure
URL: https://amplab.cs.berkeley.edu/jenkins/job/testing-k8s-prb-make-spark-distribution-unified/933/

Contributor

shaneknapp commented Jul 13, 2018

test this please

SparkQA commented Jul 13, 2018

Test build #92983 has finished for PR 21583 at commit c0b5927.

This patch passes all tests.
This patch merges cleanly.
This patch adds no public classes.

SparkQA commented Jul 13, 2018

Kubernetes integration test starting
URL: https://amplab.cs.berkeley.edu/jenkins/job/testing-k8s-prb-make-spark-distribution-unified/935/

SparkQA commented Jul 13, 2018

Kubernetes integration test status failure
URL: https://amplab.cs.berkeley.edu/jenkins/job/testing-k8s-prb-make-spark-distribution-unified/935/

Contributor

shaneknapp commented Jul 13, 2018

test this please

1 similar comment

Contributor

shaneknapp commented Jul 13, 2018

test this please

SparkQA commented Jul 13, 2018

Kubernetes integration test starting
URL: https://amplab.cs.berkeley.edu/jenkins/job/testing-k8s-prb-make-spark-distribution-unified/938/

SparkQA commented Jul 13, 2018

Test build #92987 has finished for PR 21583 at commit c0b5927.

This patch passes all tests.
This patch merges cleanly.
This patch adds no public classes.

SparkQA commented Jul 13, 2018

Kubernetes integration test status success
URL: https://amplab.cs.berkeley.edu/jenkins/job/testing-k8s-prb-make-spark-distribution-unified/938/

Contributor Author

ifilonenko commented Jul 13, 2018

@mccheah
@foxish
Can we merge?

Contributor

mccheah commented Jul 14, 2018

Yeah let's merge this - think there might be some work to clean this up a bit later.

asfgit closed this in

f1a99ad

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet