Skip to content

[docs] Expose dashboard url when deploying on Yarn using Skein #57793

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Merged
edoakes merged 2 commits into from
Oct 22, 2025
Merged

[docs] Expose dashboard url when deploying on Yarn using Skein #57793

Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
2 commits
Select commit Hold shift + click to select a range
0ecc1f8
[docs] Expose dashboard url when deploying on Yarn using Skein
Zakelly Oct 16, 2025
67cf157
Merge branch 'master' into yarn_dashboard
edoakes Oct 21, 2025
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
18 changes: 18 additions & 0 deletions doc/source/cluster/doc_code/yarn/dashboard.py
View file
Open in desktop
Original file line number Diff line number Diff line change
@@ -0,0 +1,18 @@
import skein
import sys
from urllib.parse import urlparse

if __name__ == "__main__":
if len(sys.argv) < 2:
print("Usage: python dashboard.py <dashboard-address>")
sys.exit(1)
Comment on lines +6 to +8
Copy link
Collaborator

@edoakes edoakes Oct 17, 2025

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

nit: use argparse instead of hand rolling this

Copy link
Contributor Author

@Zakelly Zakelly Oct 17, 2025

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

As there is only one required argument, I'd keep it simple as is.

address = sys.argv[1]
# Check if the address is a valid URL
result = urlparse(address)
if not all([result.scheme, result.netloc]):
print("Error: Invalid dashboard address. Please provide a valid URL.")
sys.exit(1)

print("Registering dashboard " + address + " on skein.")
app = skein.ApplicationClient.from_current()
app.ui.add_page("ray-dashboard", address, "Ray Dashboard")
12 changes: 10 additions & 2 deletions doc/source/cluster/doc_code/yarn/ray-skein.yaml
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -12,6 +12,8 @@ services:
files:
# ray/doc/source/cluster/doc_code/yarn/example.py
example.py: example.py
# ray/doc/source/cluster/doc_code/yarn/dashboard.py
dashboard.py: dashboard.py
# # A packaged python environment using `conda-pack`. Note that Skein
# # doesn't require any specific way of distributing files, but this
# # is a good one for python projects. This is optional.
Expand All @@ -21,14 +23,20 @@ services:
# Activate the packaged conda environment
# - source environment/bin/activate

# This gets the IP address of the head node.
RAY_HEAD_ADDRESS=$(hostname -i)

# This stores the Ray head address in the Skein key-value store so that the workers can retrieve it later.
skein kv put current --key=RAY_HEAD_ADDRESS --value=$(hostname -i)
skein kv put current --key=RAY_HEAD_ADDRESS --value=$RAY_HEAD_ADDRESS

# This command starts all the processes needed on the ray head node.
# By default, we set object store memory and heap memory to roughly 200 MB. This is conservative
# and should be set according to application needs.
#
ray start --head --port=6379 --object-store-memory=200000000 --memory 200000000 --num-cpus=1
ray start --head --port=6379 --object-store-memory=200000000 --memory 200000000 --num-cpus=1 --dashboard-host=$RAY_HEAD_ADDRESS

# This registers the Ray dashboard on Skein, which can be accessed on Skein's web UI.
python dashboard.py "http://$RAY_HEAD_ADDRESS:8265"

# This executes the user script.
python example.py
Expand Down
Binary file added doc/source/cluster/images/yarn-job-dashboard.png
View file
Open in desktop
Loading
Sorry, something went wrong. Reload?
Sorry, we cannot display this file.
Sorry, this file is invalid so it cannot be displayed.
12 changes: 12 additions & 0 deletions doc/source/cluster/vms/user-guides/community/yarn.rst
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -83,6 +83,8 @@ Use the ``files`` option to specify files that will be copied into the YARN cont
files:
# ray/doc/yarn/example.py
example.py: example.py
# ray/doc/yarn/dashboard.py
dashboard.py: dashboard.py
# # A packaged python environment using `conda-pack`. Note that Skein
# # doesn't require any specific way of distributing files, but this
# # is a good one for python projects. This is optional.
Expand Down Expand Up @@ -116,6 +118,12 @@ and heap memory to roughly 200 MB. This is conservative and should be set accord

ray start --head --port=6379 --object-store-memory=200000000 --memory 200000000 --num-cpus=1

Register the ray dashboard to Skein. This exposes the dashboard link on the Skein application page.

.. code-block:: bash

python dashboard.py "http://$(hostname -i):8265"

Execute the user script containing the Ray program.

.. code-block:: bash
Expand Down Expand Up @@ -177,6 +185,10 @@ Once it has been submitted, you can see the job running on the YARN dashboard.

.. image:: /cluster/images/yarn-job.png

If you have registered the Ray dashboard address in the Skein as shown above, you can retrieve it on Skein's application page:

.. image:: /cluster/images/yarn-job-dashboard.png

Cleaning Up
-----------

Expand Down
18 changes: 18 additions & 0 deletions doc/yarn/dashboard.py
View file
Open in desktop
Original file line number Diff line number Diff line change
@@ -0,0 +1,18 @@
import skein
Copy link
Collaborator

@edoakes edoakes Oct 17, 2025

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

this seems to be the same file from doc_code/yard/dashboard.py -- is that intentional?

Copy link
Contributor Author

@Zakelly Zakelly Oct 17, 2025

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Yes. I noticed there are same files under doc/cluster/doc_code/yarn/ and doc/yarn, such as example.py and ray-skein.yaml, so I add same changes under two folder.

import sys
from urllib.parse import urlparse

if __name__ == "__main__":
if len(sys.argv) < 2:
print("Usage: python dashboard.py <dashboard-address>")
sys.exit(1)
address = sys.argv[1]
# Check if the address is a valid URL
result = urlparse(address)
if not all([result.scheme, result.netloc]):
print("Error: Invalid dashboard address. Please provide a valid URL.")
sys.exit(1)

print("Registering dashboard " + address + " on skein.")
app = skein.ApplicationClient.from_current()
app.ui.add_page("ray-dashboard", address, "Ray Dashboard")
12 changes: 10 additions & 2 deletions doc/yarn/ray-skein.yaml
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -12,6 +12,8 @@ services:
files:
# ray/doc/yarn/example.py
example.py: example.py
# ray/doc/yarn/dashboard.py
dashboard.py: dashboard.py
# # A packaged python environment using `conda-pack`. Note that Skein
# # doesn't require any specific way of distributing files, but this
# # is a good one for python projects. This is optional.
Expand All @@ -21,14 +23,20 @@ services:
# Activate the packaged conda environment
# - source environment/bin/activate

# This gets the IP address of the head node.
RAY_HEAD_ADDRESS=$(hostname -i)

# This stores the Ray head address in the Skein key-value store so that the workers can retrieve it later.
skein kv put current --key=RAY_HEAD_ADDRESS --value=$(hostname -i)
skein kv put current --key=RAY_HEAD_ADDRESS --value=$RAY_HEAD_ADDRESS

# This command starts all the processes needed on the ray head node.
# By default, we set object store memory and heap memory to roughly 200 MB. This is conservative
# and should be set according to application needs.
#
ray start --head --port=6379 --object-store-memory=200000000 --memory 200000000 --num-cpus=1
ray start --head --port=6379 --object-store-memory=200000000 --memory 200000000 --num-cpus=1 --dashboard-host=$RAY_HEAD_ADDRESS

# This registers the Ray dashboard on Skein, which can be accessed on Skein's web UI.
python dashboard.py "http://$RAY_HEAD_ADDRESS:8265"

# This executes the user script.
python example.py
Expand Down