[WIP] Support client-mode tests

[echarles]

Changes to test client mode upon the cluster mode.

[SparkQA]

Kubernetes integration test (test of tests) starting
URL: https://amplab.cs.berkeley.edu/jenkins/job/testing-k8s-prb-test-integration-tests/5/

[echarles]

The changes aim to run for each test the command with --deploy-mode=client and --deploy-mode=cluster.

I run with docker images built from apache/spark#20451

./dev/dev-run-integration-tests.sh \
  --spark-tgz $DLAHOME/packages/spark-2.4.0-SNAPSHOT-bin-hdfs-2.9.0.tgz \
  --image-repo localhost:5000 \
  --image-tag 2.4.0

For now, client mode fails with:

- Run SparkPi with no resources *** FAILED ***
  java.lang.AssertionError: assertion failed: Failed to execute /home/datalayer/src/datalayer/datalayer/repos/spark-integration/target/spark-dist-unpacked/bin/spark-submit --deploy-mode client --class org.apache.spark.examples.SparkPi --master k8s://https://192.168.99.100:8443/ --conf spark.testing=false --conf spark.kubernetes.driver.pod.name=spark-test-app-d4fbf209577b4d23b5f0df87fcc17cbc --conf spark.kubernetes.driver.label.spark-app-locator=38d685fc2b7d49c5bdf6aecc94db3087 --conf spark.executor.memory=500m --conf spark.kubernetes.executor.label.spark-app-locator=38d685fc2b7d49c5bdf6aecc94db3087 --conf spark.kubernetes.submission.waitAppCompletion=false --conf spark.kubernetes.namespace=1eb9c919964848d7b1aee68ff001cf81 --conf spark.kubernetes.authenticate.driver.serviceAccountName=default --conf spark.app.name=spark-test-app --conf spark.ui.enabled=true --conf spark.executors.instances=1 --conf spark.kubernetes.container.image=localhost:5000/spark:2.4.0 --conf spark.master=k8s://https://192.168.99.100:8443/ --conf spark.executor.cores=1 local:///opt/spark/examples/jars/spark-examples_2.11-2.4.0-SNAPSHOT.jar
  at scala.Predef$.assert(Predef.scala:170)
  at org.apache.spark.deploy.k8s.integrationtest.ProcessUtils$.executeProcess(ProcessUtils.scala:41)
  at org.apache.spark.deploy.k8s.integrationtest.SparkAppLauncher$.launch(KubernetesTestComponents.scala:114)
  at org.apache.spark.deploy.k8s.integrationtest.KubernetesSuite.runSparkApplicationAndVerifyCompletion(KubernetesSuite.scala:292)
  at org.apache.spark.deploy.k8s.integrationtest.KubernetesSuite.org$apache$spark$deploy$k8s$integrationtest$KubernetesSuite$$runSparkPiAndVerifyCompletion(KubernetesSuite.scala:216)
  at org.apache.spark.deploy.k8s.integrationtest.KubernetesSuite$$anonfun$2.apply$mcV$sp(KubernetesSuite.scala:101)
  at org.apache.spark.deploy.k8s.integrationtest.KubernetesSuite$$anonfun$2.apply(KubernetesSuite.scala:101)
  at org.apache.spark.deploy.k8s.integrationtest.KubernetesSuite$$anonfun$2.apply(KubernetesSuite.scala:101)
  at org.scalatest.Transformer$$anonfun$apply$1.apply$mcV$sp(Transformer.scala:22)
  at org.scalatest.OutcomeOf$class.outcomeOf(OutcomeOf.scala:85)

If I take relaunch the printed command, I receive: 'Failure executing: POST at: https://192.168.99.100:8443/api/v1/namespaces/1eb9c919964848d7b1aee68ff001cf81/pods. Message: namespaces "1eb9c919964848d7b1aee68ff001cf81" not found.'

If I change in the command 1eb9c919964848d7b1aee68ff001cf81 to default, command is successful.

[SparkQA]

Kubernetes integration test (test of tests) status failure
URL: https://amplab.cs.berkeley.edu/jenkins/job/testing-k8s-prb-test-integration-tests/5/

[AmplabJenkins]

Merged build finished. Test FAILed.

[AmplabJenkins]

Test FAILed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/testing-k8s-prb-test-integration-tests/5/
Test FAILed.