[SPARK-42785][K8S][Core] When spark submit without --deploy-mode, avoid facing NPE in Kubernetes Case

[zwangsheng]

What changes were proposed in this pull request?

After #37880 when user spark submit without --deploy-mode XXX or –conf spark.submit.deployMode=XXXX, may face NPE with this code.

Why are the changes needed?

https://github.com/apache/spark/blob/master/core/src/main/scala/org/apache/spark/deploy/SparkSubmit.scala#164

args.deployMode.equals("client") &&

Of course, submit without deployMode is not allowed and will throw an exception and terminate the application, but we should leave it to the later logic to give the appropriate hint instead of giving a NPE.

Does this PR introduce any user-facing change?

No

How was this patch tested?

[zwangsheng]

As last PR mentioned, @yaooqinn @shrprasa

[yaooqinn]

Please use the jira-style prefix [SPARK-42785], not a GitHub issue-like one [SPARK #42785] like apache/kyuubi. Please also answer the rest of the questions in the PR description.

[shrprasa]

@zwangsheng @yaooqinn There won't be any NPE if you try to actually submit a job without the deploy-mode because if no value is provided, it defaults to "client". Please refer to org.apache.spark.internal.config/package.scala.

  private[spark] val SUBMIT_DEPLOY_MODE = ConfigBuilder("spark.submit.deployMode")
    .version("1.5.0")
    .stringConf
    .createWithDefault("client")

[zwangsheng]

@zwangsheng @yaooqinn There won't be any NPE if you try to actually submit a job without the deploy-mode because if no value is provided, it defaults to "client". Please refer to org.apache.spark.internal.config/package.scala.

  private[spark] val SUBMIT_DEPLOY_MODE = ConfigBuilder("spark.submit.deployMode")
    .version("1.5.0")
    .stringConf
    .createWithDefault("client")

@shrprasa Thanks for reminder

I saw this default value, you can do local test with writing following code

The default value client you mean, will be set in prepareSubmitEnvironment function, which be called in runMain() after check

args.deployMode.equals("client") &&

[zwangsheng]

Please use the jira-style prefix [SPARK-42785], not a GitHub issue-like one [SPARK #42785] like apache/kyuubi

Sure. XD

[shrprasa]

I was able to reproduce the issue. Thanks @zwangsheng for fixing it.

[dongjoon-hyun]

+1, LGTM. Thank you, @zwangsheng and all.
Merged to master/3.4/3.3/3.2.