Skip to content

[release] release data batch inference tests #57287

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Merged
aslonnie merged 5 commits into from
Oct 25, 2025
Merged

[release] release data batch inference tests #57287

Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
5 commits
Select commit Hold shift + click to select a range
08d9e2c
upgrading data release tests to py3.10
elliot-barn Oct 7, 2025
be6a14a
upgrading to python 3.10
elliot-barn Oct 7, 2025
65e5946
undoing python version for image_embedding_from_jsonl
elliot-barn Oct 8, 2025
2a96b83
Merge remote-tracking branch 'origin/master' into elliot-barn/release…
elliot-barn Oct 8, 2025
9c4aaf2
merging master
elliot-barn Oct 23, 2025
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
4 changes: 4 additions & 0 deletions release/release_data_tests.yaml
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -491,6 +491,7 @@
# 300 GB image classification parquet data up to 10 GPUs
# 10 g4dn.12xlarge.
- name: "image_classification_{{scaling}}"
python: "3.10"
Copy link
Contributor

@gemini-code-assist gemini-code-assist bot Oct 7, 2025

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

medium

To improve maintainability and reduce repetition, you can define a YAML anchor for the Python version here. This allows you to reuse the value in other test configurations within this file by using an alias, making future updates to the Python version much easier and less error-prone.

  python: &py310 "3.10"

group: data-batch-inference

cluster:
Expand All @@ -507,6 +508,7 @@
--data-directory 300G-image-data-synthetic-raw-parquet --data-format parquet

- name: image_classification_chaos
python: "3.10"
Copy link
Contributor

@gemini-code-assist gemini-code-assist bot Oct 7, 2025

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

medium

Please use the YAML alias *py310 defined earlier. This will make the configuration more maintainable.

  python: *py310

stable: False
# Don't use 'nightly_tests/dataset' as the working directory because we need to run
# the 'setup_chaos.py' script.
Expand All @@ -525,6 +527,7 @@


- name: image_embedding_from_uris_{{case}}
python: "3.10"
Copy link
Contributor

@gemini-code-assist gemini-code-assist bot Oct 7, 2025

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

medium

Using the *py310 alias here will help with future maintenance.

  python: *py310

frequency: weekly
group: data-batch-inference

Expand Down Expand Up @@ -594,6 +597,7 @@
script: python image_embedding_from_jsonl/main.py {{args}}

- name: text_embedding_{{case}}
python: "3.10"
Copy link
Contributor

@gemini-code-assist gemini-code-assist bot Oct 7, 2025

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

medium

And here as well, please use the *py310 alias.

  python: *py310

frequency: weekly
group: data-batch-inference

Expand Down
2 changes: 2 additions & 0 deletions release/release_tests.yaml
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -3367,6 +3367,7 @@
- __suffix__: aws

- name: chaos_dask_on_ray_large_scale_test_no_spilling
python: "3.10"
Copy link
Contributor

@gemini-code-assist gemini-code-assist bot Oct 7, 2025

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

medium

To improve maintainability and reduce repetition, you can define a YAML anchor for the Python version here. This allows you to reuse the value in the subsequent test configuration within this file by using an alias, making future updates to the Python version easier.

  python: &py310 "3.10"

group: data-tests
working_dir: nightly_tests

Expand Down Expand Up @@ -3397,6 +3398,7 @@
cluster_compute: dask_on_ray/dask_on_ray_stress_compute_gce.yaml

- name: chaos_dask_on_ray_large_scale_test_spilling
python: "3.10"
Copy link
Contributor

@gemini-code-assist gemini-code-assist bot Oct 7, 2025

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

medium

To maintain consistency and improve maintainability, please use the YAML alias *py310 here, as defined in the chaos_dask_on_ray_large_scale_test_no_spilling test.

  python: *py310

group: data-tests
working_dir: nightly_tests

Expand Down