Skip to content

[deps][ci] Creating depset for docs & updating pydantic doc ver #57947

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Merged
aslonnie merged 23 commits into from
Oct 21, 2025
Merged

[deps][ci] Creating depset for docs & updating pydantic doc ver #57947

Show file tree
Hide file tree
Changes from 20 commits
Commits
Show all changes
23 commits
Select commit Hold shift + click to select a range
1937054
including lock files for doc builds
elliot-barn Oct 17, 2025
d42ea10
updating p39 to py310
elliot-barn Oct 17, 2025
34b078d
adding lock file
elliot-barn Oct 17, 2025
34b7430
including base build for py3.10
elliot-barn Oct 17, 2025
4594163
merging master
elliot-barn Oct 17, 2025
bbf4906
including 3.10 base test image
elliot-barn Oct 17, 2025
2d514a7
reverting copy command
elliot-barn Oct 17, 2025
a71114b
renaming depset lock file
elliot-barn Oct 17, 2025
16adde9
updating pip command
elliot-barn Oct 17, 2025
0f977d9
removing quotes from docker build arg
elliot-barn Oct 17, 2025
ab72f73
echoing pip requirements
elliot-barn Oct 18, 2025
1132033
expanding ray img depset and updating docbuild requirements file
elliot-barn Oct 18, 2025
7d80ff4
Merge branch 'master' into elliot-barn/depset-for-docbuild
aslonnie Oct 18, 2025
eea5391
using relative path to lock file
elliot-barn Oct 20, 2025
30733c7
Merge branch 'elliot-barn/depset-for-docbuild' of github.com:ray-proj…
elliot-barn Oct 20, 2025
d8db64d
running on python 3.9
elliot-barn Oct 20, 2025
e63d4bb
building 3.9 docbuild
elliot-barn Oct 21, 2025
06d9534
lockfile for python 3.9
elliot-barn Oct 21, 2025
8df0368
installing ray without deps
elliot-barn Oct 21, 2025
47d1fa7
updgrading pydantic constraint
elliot-barn Oct 21, 2025
6a9601a
removing python 310 builds
elliot-barn Oct 21, 2025
ab2ec89
removing install ray
elliot-barn Oct 21, 2025
c72eded
removing markdown-it-py pin
elliot-barn Oct 21, 2025
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
3 changes: 3 additions & 0 deletions .buildkite/base.rayci.yml
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -8,6 +8,7 @@ steps:
wanda: ci/docker/base.test.wanda.yaml
matrix:
- "3.9"
- "3.10"
- "3.11"
- "3.12"
env:
Expand All @@ -23,6 +24,7 @@ steps:
wanda: ci/docker/base.build.wanda.yaml
matrix:
- "3.9"
- "3.10"
- "3.11"
- "3.12"
env:
Expand Down Expand Up @@ -86,6 +88,7 @@ steps:
- "3.12"
env:
PYTHON: "{{matrix}}"
REQUIREMENTS_FILE: "python/deplocks/docs/docbuild_depset_py{{matrix}}.lock"
tags: cibase

- name: docgpubuild
Expand Down
9 changes: 9 additions & 0 deletions .buildkite/dependencies.rayci.yml
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -47,3 +47,12 @@ steps:
- bazel run //ci/raydepsets:raydepsets -- build ci/raydepsets/configs/release_multimodal_inference_benchmarks_tests.depsets.yaml --check
job_env: manylinux
depends_on: manylinux

- label: ":tapioca: build: raydepsets: compile doc dependencies"
Copy link
Collaborator

@aslonnie aslonnie Oct 21, 2025

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

we should think about merging some of those into one single test job. those are rather fast checks, they do not justify the overhead of spinning up separate test job instances.

Copy link
Contributor Author

@elliot-barn elliot-barn Oct 21, 2025

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

sounds good to me

key: raydepsets_compile_doc_dependencies
tags: always
instance_type: medium
commands:
- bazel run //ci/raydepsets:raydepsets -- build ci/raydepsets/configs/docs.depsets.yaml --check
job_env: manylinux
depends_on: manylinux
2 changes: 1 addition & 1 deletion .readthedocs.yaml
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -20,4 +20,4 @@ sphinx:
# https://docs.readthedocs.io/en/stable/guides/reproducible-builds.html
python:
install:
- requirements: doc/requirements-doc.txt
- requirements: python/deplocks/docs/docbuild_depset_py3.12.lock
Copy link
Collaborator

@aslonnie aslonnie Oct 21, 2025

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

beautiful!

8 changes: 5 additions & 3 deletions ci/docker/doc.build.Dockerfile
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -7,10 +7,12 @@ FROM $DOCKER_IMAGE_RAY_DASHBOARD AS ray_dashboard

FROM $DOCKER_IMAGE_BASE_BUILD

SHELL ["/bin/bash", "-ice"]

COPY . .

RUN mv python/deplocks/docs/docbuild_depset_py${PYTHON}.lock python_depset.lock
Copy link
Collaborator

@aslonnie aslonnie Oct 21, 2025

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

why do you need to make the move here? why not just do the pip install directly with the file name down at line 29?

it is a little bit confusing on where the python_depset.lock file comes from if just reading line 29.

Copy link
Contributor Author

@elliot-barn elliot-barn Oct 21, 2025

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

this follows the pattern we use for byod. All images will eventually install python dependencies from python_depset.lock

Copy link
Contributor Author

@elliot-barn elliot-barn Oct 21, 2025

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

updated to just install the lock file directly

Copy link
Collaborator

@aslonnie aslonnie Oct 21, 2025

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

yeah, this is a CI base file, it is different from the release test byod images.


SHELL ["/bin/bash", "-ice"]

RUN --mount=type=bind,from=ray_core,target=/mnt/ray-core \
--mount=type=bind,from=ray_dashboard,target=/mnt/ray-dashboard \
<<EOF
Expand All @@ -24,6 +26,6 @@ cp /mnt/ray-core/ray_pkg.zip /opt/ray-build/ray_pkg.zip
cp /mnt/ray-core/ray_py_proto.zip /opt/ray-build/ray_py_proto.zip
cp /mnt/ray-dashboard/dashboard.tar.gz /opt/ray-build/dashboard.tar.gz

pip install -r doc/requirements-doc.txt
pip install -r python_depset.lock
Copy link
Collaborator

@aslonnie aslonnie Oct 21, 2025

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

could you check if readthedocs is also using the lock file? let's make sure that it is consistent with how CI builds the doc.

Copy link
Contributor Author

@elliot-barn elliot-barn Oct 21, 2025

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Updated here: https://github.com/ray-project/ray/pull/57947/files#diff-03efc769b870804394632e45d7885272b44c16939517fb31c9d7c614d2ffae57R23


EOF
2 changes: 1 addition & 1 deletion ci/docker/doc.build.wanda.yaml
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -5,7 +5,7 @@ froms:
- cr.ray.io/rayproject/ray-dashboard
dockerfile: ci/docker/doc.build.Dockerfile
srcs:
- doc/requirements-doc.txt
- "$REQUIREMENTS_FILE"
build_args:
- DOCKER_IMAGE_BASE_BUILD=cr.ray.io/rayproject/oss-ci-base_build-py$PYTHON
- DOCKER_IMAGE_RAY_CORE=cr.ray.io/rayproject/ray-core-py$PYTHON
Expand Down
18 changes: 15 additions & 3 deletions ci/lint/lint.sh
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -110,15 +110,27 @@ _install_ray() {
unzip -o -q /opt/ray-build/ray_py_proto.zip -d python
mkdir -p python/ray/dashboard/client/build
tar -xzf /opt/ray-build/dashboard.tar.gz -C python/ray/dashboard/client/build
SKIP_BAZEL_BUILD=1 pip install -e "python[all]"
SKIP_BAZEL_BUILD=1 pip install -e "python[all]" --no-deps
else
RAY_DISABLE_EXTRA_CPP=1 pip install -e "python[all]"
fi
}

_install_ray_no_deps() {
Copy link
Collaborator

@aslonnie aslonnie Oct 21, 2025

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

you changed the only two places that were using _install_ray to run _install_ray_no_deps, maybe just remove the _install_ray function?

if [[ -d /opt/ray-build ]]; then
unzip -o -q /opt/ray-build/ray_pkg.zip -d python
unzip -o -q /opt/ray-build/ray_py_proto.zip -d python
mkdir -p python/ray/dashboard/client/build
tar -xzf /opt/ray-build/dashboard.tar.gz -C python/ray/dashboard/client/build
SKIP_BAZEL_BUILD=1 pip install -e "python[all]" --no-deps
else
RAY_DISABLE_EXTRA_CPP=1 pip install -e "python[all]" --no-deps
fi
}
Copy link

@cursor cursor bot Oct 21, 2025

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Bug: Ray Installation Path Dependency Inconsistency

The _install_ray() function now installs Ray with --no-deps in one path but resolves dependencies in another, creating inconsistent behavior. This also makes its --no-deps path redundant with the new _install_ray_no_deps() function, which consistently installs Ray without dependencies.

Fix in Cursor Fix in Web


api_annotations() {
echo "--- Install Ray"
_install_ray
_install_ray_no_deps

echo "--- Check API annotations"
./ci/lint/check_api_annotations.py
Expand All @@ -130,7 +142,7 @@ api_policy_check() {
make -C doc/ html

echo "--- Install Ray"
_install_ray
_install_ray_no_deps

echo "--- Check API/doc consistency"
bazel run //ci/ray_ci/doc:cmd_check_api_discrepancy -- /ray "$@"
Expand Down
28 changes: 28 additions & 0 deletions ci/raydepsets/configs/docs.depsets.yaml
View file
Open in desktop
Original file line number Diff line number Diff line change
@@ -0,0 +1,28 @@
build_arg_sets:
py39:
PYTHON_VERSION: "3.9"
PYTHON_SHORT: "39"
py310:
PYTHON_VERSION: "3.10"
PYTHON_SHORT: "310"
py312:
PYTHON_VERSION: "3.12"
PYTHON_SHORT: "312"


depsets:
- name: docbuild_depset_${PYTHON_SHORT}
operation: compile
depsets:
- ray_img_depset_${PYTHON_SHORT}
requirements:
- doc/requirements-doc.txt
output: python/deplocks/docs/docbuild_depset_py${PYTHON_VERSION}.lock
append_flags:
- --python-version=${PYTHON_VERSION}
- --python-platform=linux
- --unsafe-package ray
build_arg_sets:
- py39
- py310
- py312
4 changes: 2 additions & 2 deletions doc/requirements-doc.txt
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -22,12 +22,12 @@ autodoc_pydantic==2.2.0
appnope
sphinx-docsearch==0.0.7

pydantic!=2.0.*,!=2.1.*,!=2.2.*,!=2.3.*,!=2.4.*,<3
pydantic==2.5.0

# MyST
myst-parser==2.0.0 # Needed to parse markdown
myst-nb==1.0.0rc0 # Most recent version of myst-nb; pin when new release is made

markdown-it-py==3.0.0
# Jupyter conversion
jupytext==1.15.2

Expand Down
Loading