Support to get spark app url with pattern http://{{SPARK_DRIVER_POD_IP}}:{{SPARK_UI_PORT}}

[turboFei]

Why are the changes needed?

We are using virtual-kubelet for spark on kubernetes, and spark kubernetes pods would be allocated across kubernetes clusters.

And we use the driver POD ip as driver host, see apache/spark#40392, which is supported since spark-3.5.

The kubernetes context and namespace are virtual and we can not build the app URL by spark driver svc.

And the spark driver pod IP is accessible for our use case, so raise this PR to build the spark app url by spark driver pod id and spark ui port.

How was this patch tested?

UT.

Was this patch authored or co-authored using generative AI tooling?

No.

[codecov-commenter]

Codecov Report

❌ Patch coverage is 0% with 50 lines in your changes missing coverage. Please review.
✅ Project coverage is 0.00%. Comparing base (a1a08e7) to head (1277952).
⚠️ Report is 15 commits behind head on master.

Files with missing lines	Patch %	Lines
...kyuubi/engine/KubernetesApplicationOperation.scala	0.00%	47 Missing ⚠️
...kyuubi/engine/KubernetesApplicationUrlSource.scala	0.00%	2 Missing ⚠️
...in/scala/org/apache/kyuubi/config/KyuubiConf.scala	0.00%	1 Missing ⚠️

Additional details and impacted files

@@          Coverage Diff           @@
##           master   #7141   +/-   ##
======================================
  Coverage    0.00%   0.00%           
======================================
  Files         700     701    +1     
  Lines       43454   43500   +46     
  Branches     5883    5895   +12     
======================================
- Misses      43454   43500   +46     

☔ View full report in Codecov by Sentry.
📢 Have feedback on the report? Share it here.

🚀 New features to boost your workflow:

• ❄️ Test Analytics: Detect flaky tests, report on failures, and find test suite problems.
• 📦 JS Bundle Analysis: Save yourself from yourself by tracking and limiting bundle sizes in JS merges.

[pan3793]

LGTM, one question, SPARK-42769 is only available since Spark 3.5, do we need to write something in docs for these requirements?

[turboFei]

LGTM, one question, SPARK-42769 is only available since Spark 3.5, do we need to write something in docs for these requirements?

Not necessary, this PR can also work with any spark version that supports spark on kubernetes.
the podIp and pod container port are available in the pod.

SPARK-42769 is only for executors to connect to driver via IP.

[turboFei]

Have updated the PR desc.

And we use the driver POD ip as driver host, see https://github.com/apache/spark/pull/40392, which is supported since spark-3.5.

[pan3793]

I see, Pod IP is retrieved by K8s client