[SPARK-38597][K8S][INFRA] Enable Spark on K8S integration tests

[Yikun]

What changes were proposed in this pull request?

Enable Spark on K8S integration tests in Github Action based on minikube:

• The K8S IT will always triggered in user fork repo and apache/spark merged commits to master branch

• This PR does NOT contains Volcano related test due to limited resource of github action.

• minikube installation is allowed by Apache Infra: INFRA-23000

• Why setting driver 0.5 cpu, executor 0.2 cpu?

  • Github-hosted runner hardware limited: 2U7G, so cpu resource is very limited.
  • IT Job available CPU = 2U - 0.85U (K8S deploy) = 1.15U
  • There are 1.15 cpu left after k8s installation, to meet the requirement of K8S tests (one driver + max to 3 executors).
  • For memory: 6947 is maximum (Otherwise raise Exiting due to RSRC_OVER_ALLOC_MEM: Requested memory allocation 7168MB is more than your system limit 6947MB.), but this is not integer multiple of 1024, so I just set this to 6144 for better resource statistic.

• Time cost info:

  • 14 mins to compile related code.
  • 3 mins to build docker images.
  • 20-30 mins to test
  • Total: about 30-40 mins

Why are the changes needed?

This will also improve the efficiency of K8S development and guarantee the quality of spark on K8S and spark docker image in some level.

Does this PR introduce any user-facing change?

No, dev only.

How was this patch tested?

CI passed

Closes #35830

[Yikun]

cc @dongjoon-hyun @HyukjinKwon

[dongjoon-hyun]

Thank you for your patience and reviving this, @Yikun .

[dongjoon-hyun]

I added Closes at the end of your PR description. It will close your PR automatically when this is going to be merged.

[dongjoon-hyun]

I believe we should enable k8s test always instead of this. core module changes will affect many scheduling tests too.

[dongjoon-hyun]

FYI, historically, log4j2 change also causes K8s integration test failure due to log4j2.properties file changes.

[Yikun]

OK, I will change this soon.

[dongjoon-hyun]

+1, LGTM. Thank you, @Yikun . The SQL UT failure is irrelevant to this GitHub Action.
Merged to master for Apache Spark 3.4.

[Yikun]

@dongjoon-hyun Thanks! I will monitor it closely for the next few days.

[dongjoon-hyun]

Thank you for your consistent participation, @Yikun .