[SPARK-45497][K8S] Add a symbolic link file `spark-examples.jar` in K8s Docker images #43324

dongjoon-hyun · 2023-10-11T04:40:50Z

What changes were proposed in this pull request?

This PR aims to add a symbolic link file, spark-examples.jar, in the example jar directory.

$ docker run -it --rm spark:latest ls -al /opt/spark/examples/jars  | tail -n6
total 1620
drwxr-xr-x 1 root root    4096 Oct 11 04:37 .
drwxr-xr-x 1 root root    4096 Sep  9 02:08 ..
-rw-r--r-- 1 root root   78803 Sep  9 02:08 scopt_2.12-3.7.1.jar
-rw-r--r-- 1 root root 1564255 Sep  9 02:08 spark-examples_2.12-3.5.0.jar
lrwxrwxrwx 1 root root      29 Oct 11 04:37 spark-examples.jar -> spark-examples_2.12-3.5.0.jar

Why are the changes needed?

Like PySpark example (pi.py), we can submit the examples without considering the version numbers which was painful before.

bin/spark-submit \
--master k8s://$K8S_MASTER \
--deploy-mode cluster \
...
--class org.apache.spark.examples.SparkPi \
local:///opt/spark/examples/jars/spark-examples.jar 10000

The following is the driver pod log.

+ exec /usr/bin/tini -s -- /opt/spark/bin/spark-submit ...
--deploy-mode client
--properties-file /opt/spark/conf/spark.properties 
--class org.apache.spark.examples.SparkPi
local:///opt/spark/examples/jars/spark-examples.jar 10000
Files  local:///opt/spark/examples/jars/spark-examples.jar from /opt/spark/examples/jars/spark-examples.jar to /opt/spark/work-dir/./spark-examples.jar

Does this PR introduce any user-facing change?

No, this is an additional file.

How was this patch tested?

Manually build the docker image and do ls.

Was this patch authored or co-authored using generative AI tooling?

No.

…K8s Docker images

dongjoon-hyun · 2023-10-11T05:09:37Z

Could you review this PR when you have some time, @viirya ?

viirya · 2023-10-11T05:52:11Z

resource-managers/kubernetes/docker/src/main/dockerfiles/spark/Dockerfile

 COPY kubernetes/dockerfiles/spark/entrypoint.sh /opt/
 COPY kubernetes/dockerfiles/spark/decom.sh /opt/
 COPY examples /opt/spark/examples
+RUN ln -s $(basename $(ls /opt/spark/examples/jars/spark-examples_*.jar)) /opt/spark/examples/jars/spark-examples.jar


This does ln -s spark-examples_2.12-3.5.0.jar /opt/spark/examples/jars/spark-examples.jar, but is spark-examples_2.12-3.5.0.jar under current path?

No~ The symbolic link file is created at jar directory.

$ docker run -it --rm spark:latest ls -al /opt/spark/examples/jars | tail -n6 total 1620 drwxr-xr-x 1 root root 4096 Oct 11 04:37 . drwxr-xr-x 1 root root 4096 Sep 9 02:08 .. -rw-r--r-- 1 root root 78803 Sep 9 02:08 scopt_2.12-3.7.1.jar -rw-r--r-- 1 root root 1564255 Sep 9 02:08 spark-examples_2.12-3.5.0.jar lrwxrwxrwx 1 root root 29 Oct 11 04:37 spark-examples.jar -> spark-examples_2.12-3.5.0.jar

The first argument is a link source and the second argument is the location of newly created symbolic link. Since we don't use directory in the symbolic link, this relation is maintained even during copying the whole Spark directory.

It's not important for ln command. Only the first argument is used as a link location for the newly generate symbolic file.

is spark-examples_2.12-3.5.0.jar under current path?

Hmm, yea, spark-examples_2.12-3.5.0.jar is the link source. My question is, does the source exist under the current path of running ln command?

No~ The symbolic link file is created at jar directory.

Does ln command run under jar directory? I don't see there is command changing to jar directory before ln.

Do I miss anything here?

😄 I understand why you are confused. ln command doesn't need to switch directory. You can do that in your mac.

$ ls examples/jars scopt_2.12-3.7.1.jar spark-examples_2.12-3.5.0.jar $ ln -s spark-examples_2.12-3.5.0.jar examples/jars/spark-examples.jar $ ls -al examples/jars total 3216 drwxr-xr-x 5 dongjoon staff 160 Oct 10 23:41 . drwxr-xr-x 4 dongjoon staff 128 Sep 8 19:08 .. -rw-r--r-- 1 dongjoon staff 78803 Sep 8 19:08 scopt_2.12-3.7.1.jar lrwxr-xr-x 1 dongjoon staff 29 Oct 10 23:41 spark-examples.jar -> spark-examples_2.12-3.5.0.jar -rw-r--r-- 1 dongjoon staff 1564255 Sep 8 19:08 spark-examples_2.12-3.5.0.jar

BTW, this is tested in the cluster already, @viirya ~

It seems that we are repeating the same questions and answers. Maybe, are you asking because something is not working, @viirya ? Does it fail in your environment?

No, I asked this because I always run ln command with link source under current path (or it is absolute path). I don't know that you can run ln with a source in different path. Interesting. 😄

If you test it, then it should be okay.

You can do that in your mac.

Yea, I just tested it locally. It works. 👍

dongjoon-hyun · 2023-10-11T07:35:19Z

Thank you so much for your patience, @viirya ! I must be clear about the ln usage (which I used) from the beginning.

dongjoon-hyun · 2023-10-11T07:36:01Z

Merged to master for Apache Spark 4.0.0.

yaooqinn · 2023-10-11T07:47:43Z

Late LGTM and looks useful.

viirya · 2023-10-11T16:13:26Z

Thank you @dongjoon-hyun for clarifying my confusion!

…8s Docker images ### What changes were proposed in this pull request? This PR aims to add a symbolic link file, `spark-examples.jar`, in the example jar directory. ``` $ docker run -it --rm spark:latest ls -al /opt/spark/examples/jars | tail -n6 total 1620 drwxr-xr-x 1 root root 4096 Oct 11 04:37 . drwxr-xr-x 1 root root 4096 Sep 9 02:08 .. -rw-r--r-- 1 root root 78803 Sep 9 02:08 scopt_2.12-3.7.1.jar -rw-r--r-- 1 root root 1564255 Sep 9 02:08 spark-examples_2.12-3.5.0.jar lrwxrwxrwx 1 root root 29 Oct 11 04:37 spark-examples.jar -> spark-examples_2.12-3.5.0.jar ``` ### Why are the changes needed? Like PySpark example (`pi.py`), we can submit the examples without considering the version numbers which was painful before. ``` bin/spark-submit \ --master k8s://$K8S_MASTER \ --deploy-mode cluster \ ... --class org.apache.spark.examples.SparkPi \ local:///opt/spark/examples/jars/spark-examples.jar 10000 ``` The following is the driver pod log. ``` + exec /usr/bin/tini -s -- /opt/spark/bin/spark-submit ... --deploy-mode client --properties-file /opt/spark/conf/spark.properties --class org.apache.spark.examples.SparkPi local:///opt/spark/examples/jars/spark-examples.jar 10000 Files local:///opt/spark/examples/jars/spark-examples.jar from /opt/spark/examples/jars/spark-examples.jar to /opt/spark/work-dir/./spark-examples.jar ``` ### Does this PR introduce _any_ user-facing change? No, this is an additional file. ### How was this patch tested? Manually build the docker image and do `ls`. ### Was this patch authored or co-authored using generative AI tooling? No. Closes apache#43324 from dongjoon-hyun/SPARK-45497. Authored-by: Dongjoon Hyun <[email protected]> Signed-off-by: Dongjoon Hyun <[email protected]>

…cker images ### What changes were proposed in this pull request? This PR aims to add a symbolic link file, `spark-examples.jar`, in the example jar directory. Apache Spark repository is updated already via - apache/spark#43324 ``` $ docker run -it --rm spark:latest ls -al /opt/spark/examples/jars | tail -n6 total 1620 drwxr-xr-x 1 root root 4096 Oct 11 04:37 . drwxr-xr-x 1 root root 4096 Sep 9 02:08 .. -rw-r--r-- 1 root root 78803 Sep 9 02:08 scopt_2.12-3.7.1.jar -rw-r--r-- 1 root root 1564255 Sep 9 02:08 spark-examples_2.12-3.5.0.jar lrwxrwxrwx 1 root root 29 Oct 11 04:37 spark-examples.jar -> spark-examples_2.12-3.5.0.jar ``` ### Why are the changes needed? Like PySpark example (`pi.py`), we can submit the examples without considering the version numbers which was painful before. ``` bin/spark-submit \ --master k8s://$K8S_MASTER \ --deploy-mode cluster \ ... --class org.apache.spark.examples.SparkPi \ local:///opt/spark/examples/jars/spark-examples.jar 10000 ``` The following is the driver pod log. ``` + exec /usr/bin/tini -s -- /opt/spark/bin/spark-submit ... --deploy-mode client --properties-file /opt/spark/conf/spark.properties --class org.apache.spark.examples.SparkPi local:///opt/spark/examples/jars/spark-examples.jar 10000 Files local:///opt/spark/examples/jars/spark-examples.jar from /opt/spark/examples/jars/spark-examples.jar to /opt/spark/work-dir/./spark-examples.jar ``` ### Does this PR introduce _any_ user-facing change? No, this is an additional file. ### How was this patch tested? Manually build the docker image and do `ls`. ### Was this patch authored or co-authored using generative AI tooling? No. Closes #67 from dongjoon-hyun/SPARK-45497. Authored-by: Dongjoon Hyun <[email protected]> Signed-off-by: Dongjoon Hyun <[email protected]>

…3-4.0.0-preview1.jar` ### What changes were proposed in this pull request? This PR aims to use `spark-examples.jar` instead of `spark-examples_2.13-4.0.0-preview1.jar`. ### Why are the changes needed? To simplify the examples for Apache Spark 4+ via SPARK-45497. - apache/spark#43324 ### Does this PR introduce _any_ user-facing change? Yes, but only example images. ### How was this patch tested? Pass the CIs. ### Was this patch authored or co-authored using generative AI tooling? No. Closes #127 from dongjoon-hyun/SPARK-49705. Authored-by: Dongjoon Hyun <[email protected]> Signed-off-by: Dongjoon Hyun <[email protected]>

[SPARK-45497][K8S] Add an symbolic link file spark-examples.jar in …

1bb6fa6

…K8s Docker images

github-actions bot added the KUBERNETES label Oct 11, 2023

dongjoon-hyun changed the title ~~[SPARK-45497][K8S] Add an symbolic link file spark-examples.jar in K8s Docker images~~ [SPARK-45497][K8S] Add a symbolic link file spark-examples.jar in K8s Docker images Oct 11, 2023

dongjoon-hyun marked this pull request as draft October 11, 2023 04:51

dongjoon-hyun marked this pull request as ready for review October 11, 2023 05:03

viirya reviewed Oct 11, 2023

View reviewed changes

viirya approved these changes Oct 11, 2023

View reviewed changes

dongjoon-hyun closed this in 64ac59d Oct 11, 2023

dongjoon-hyun deleted the SPARK-45497 branch October 11, 2023 07:36

dongjoon-hyun mentioned this pull request Sep 18, 2024

[SPARK-45497] Add a symbolic link file spark-examples.jar in K8s Docker images apache/spark-docker#67

Closed

dongjoon-hyun mentioned this pull request Sep 19, 2024

[SPARK-49705] Use spark-examples.jar instead of spark-examples_2.13-4.0.0-preview1.jar apache/spark-kubernetes-operator#127

Closed

[SPARK-45497][K8S] Add a symbolic link file spark-examples.jar in K8s Docker images #43324

[SPARK-45497][K8S] Add a symbolic link file spark-examples.jar in K8s Docker images #43324

Uh oh!

Conversation

dongjoon-hyun commented Oct 11, 2023 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

What changes were proposed in this pull request?

Why are the changes needed?

Does this PR introduce any user-facing change?

How was this patch tested?

Was this patch authored or co-authored using generative AI tooling?

Uh oh!

dongjoon-hyun commented Oct 11, 2023

Uh oh!

viirya Oct 11, 2023

Choose a reason for hiding this comment

Uh oh!

dongjoon-hyun Oct 11, 2023

Choose a reason for hiding this comment

Uh oh!

dongjoon-hyun Oct 11, 2023

Choose a reason for hiding this comment

Uh oh!

dongjoon-hyun Oct 11, 2023

Choose a reason for hiding this comment

Uh oh!

viirya Oct 11, 2023

Choose a reason for hiding this comment

Uh oh!

dongjoon-hyun Oct 11, 2023 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

dongjoon-hyun Oct 11, 2023 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

viirya Oct 11, 2023 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

viirya Oct 11, 2023

Choose a reason for hiding this comment

Uh oh!

dongjoon-hyun commented Oct 11, 2023

Uh oh!

dongjoon-hyun commented Oct 11, 2023

Uh oh!

yaooqinn commented Oct 11, 2023

Uh oh!

viirya commented Oct 11, 2023

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

3 participants

[SPARK-45497][K8S] Add a symbolic link file `spark-examples.jar` in K8s Docker images #43324

[SPARK-45497][K8S] Add a symbolic link file `spark-examples.jar` in K8s Docker images #43324

dongjoon-hyun commented Oct 11, 2023 •

edited

Loading

dongjoon-hyun Oct 11, 2023 •

edited

Loading

dongjoon-hyun Oct 11, 2023 •

edited

Loading

viirya Oct 11, 2023 •

edited

Loading