Skip to content

[SPARK-45497][K8S] Add a symbolic link file spark-examples.jar in K8s Docker images #43324

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Closed
dongjoon-hyun wants to merge 1 commit into from
Closed

[SPARK-45497][K8S] Add a symbolic link file spark-examples.jar in K8s Docker images #43324

Changes from all commits
Commits
Show all changes
1 commit
Select commit
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
1 change: 1 addition & 0 deletions resource-managers/kubernetes/docker/src/main/dockerfiles/spark/Dockerfile
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -49,6 +49,7 @@ COPY sbin /opt/spark/sbin
COPY kubernetes/dockerfiles/spark/entrypoint.sh /opt/
COPY kubernetes/dockerfiles/spark/decom.sh /opt/
COPY examples /opt/spark/examples
RUN ln -s $(basename $(ls /opt/spark/examples/jars/spark-examples_*.jar)) /opt/spark/examples/jars/spark-examples.jar
Copy link
Member

@viirya viirya Oct 11, 2023

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This does ln -s spark-examples_2.12-3.5.0.jar /opt/spark/examples/jars/spark-examples.jar, but is spark-examples_2.12-3.5.0.jar under current path?

Copy link
Member Author

@dongjoon-hyun dongjoon-hyun Oct 11, 2023

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

No~ The symbolic link file is created at jar directory.

$ docker run -it --rm spark:latest ls -al /opt/spark/examples/jars  | tail -n6
total 1620
drwxr-xr-x 1 root root    4096 Oct 11 04:37 .
drwxr-xr-x 1 root root    4096 Sep  9 02:08 ..
-rw-r--r-- 1 root root   78803 Sep  9 02:08 scopt_2.12-3.7.1.jar
-rw-r--r-- 1 root root 1564255 Sep  9 02:08 spark-examples_2.12-3.5.0.jar
lrwxrwxrwx 1 root root      29 Oct 11 04:37 spark-examples.jar -> spark-examples_2.12-3.5.0.jar

Copy link
Member Author

@dongjoon-hyun dongjoon-hyun Oct 11, 2023

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

The first argument is a link source and the second argument is the location of newly created symbolic link. Since we don't use directory in the symbolic link, this relation is maintained even during copying the whole Spark directory.

Copy link
Member Author

@dongjoon-hyun dongjoon-hyun Oct 11, 2023

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

It's not important for ln command. Only the first argument is used as a link location for the newly generate symbolic file.

is spark-examples_2.12-3.5.0.jar under current path?

Copy link
Member

@viirya viirya Oct 11, 2023

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Hmm, yea, spark-examples_2.12-3.5.0.jar is the link source. My question is, does the source exist under the current path of running ln command?

No~ The symbolic link file is created at jar directory.

Does ln command run under jar directory? I don't see there is command changing to jar directory before ln.

Do I miss anything here?

Copy link
Member Author

@dongjoon-hyun dongjoon-hyun Oct 11, 2023 •
edited
Loading

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

😄 I understand why you are confused. ln command doesn't need to switch directory. You can do that in your mac.

$ ls examples/jars
scopt_2.12-3.7.1.jar          spark-examples_2.12-3.5.0.jar

$ ln -s spark-examples_2.12-3.5.0.jar examples/jars/spark-examples.jar

$ ls -al examples/jars
total 3216
drwxr-xr-x  5 dongjoon  staff      160 Oct 10 23:41 .
drwxr-xr-x  4 dongjoon  staff      128 Sep  8 19:08 ..
-rw-r--r--  1 dongjoon  staff    78803 Sep  8 19:08 scopt_2.12-3.7.1.jar
lrwxr-xr-x  1 dongjoon  staff       29 Oct 10 23:41 spark-examples.jar -> spark-examples_2.12-3.5.0.jar
-rw-r--r--  1 dongjoon  staff  1564255 Sep  8 19:08 spark-examples_2.12-3.5.0.jar

BTW, this is tested in the cluster already, @viirya ~

Copy link
Member Author

@dongjoon-hyun dongjoon-hyun Oct 11, 2023 •
edited
Loading

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

It seems that we are repeating the same questions and answers. Maybe, are you asking because something is not working, @viirya ? Does it fail in your environment?

Copy link
Member

@viirya viirya Oct 11, 2023 •
edited
Loading

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

No, I asked this because I always run ln command with link source under current path (or it is absolute path). I don't know that you can run ln with a source in different path. Interesting. 😄

If you test it, then it should be okay.

Copy link
Member

@viirya viirya Oct 11, 2023

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

You can do that in your mac.

Yea, I just tested it locally. It works. 👍

COPY kubernetes/tests /opt/spark/tests
COPY data /opt/spark/data

Expand Down