Skip to content

[None][chore] Use cluster service discover in disagg CI tests #11242

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Merged
pcastonguay merged 24 commits into from
Mar 5, 2026
Merged

[None][chore] Use cluster service discover in disagg CI tests #11242

Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
24 commits
Select commit Hold shift + click to select a range
b334655
working test_disaggregated_single_gpu_with_mpirun test with service d…
Feb 3, 2026
39a2039
implemented service for more single-gpu tests (untested)
Feb 4, 2026
688274b
all tests in test_disaggregated.py, test_disaggregated_etcd.py, test_…
Feb 7, 2026
171b71d
misc pr fixes to test_disaggregated.py; added service discovery to ac…
Feb 10, 2026
1e348e8
helper function for disagg sd setup, multiple devices per worker for …
Feb 12, 2026
4491727
fixed perf_metric, kv cache time output, and gen only tests. other te…
Feb 13, 2026
9602857
remove etcd from accuracy disagg test
Feb 13, 2026
fcf68c6
Merge branch 'main' into disagg_cluster_tests
Feb 17, 2026
6c26d1f
added model_path to deepseek disagg tests
Feb 17, 2026
bee65fb
untested service discovery for test_workers.py
Feb 17, 2026
6f70cdf
fixed fatal issue with device_ids assignment. added type hints. added…
Feb 18, 2026
f6d2dba
bug fixes and cleanup
Feb 19, 2026
ec4f1e0
Merge branch 'main' into disagg_cluster_tests
pcastonguay Feb 19, 2026
832ab7b
fixed auto scaling test failures
Feb 25, 2026
399cca9
multigpu deepseek model path test fail fix (untested)
Feb 25, 2026
49be4ea
fixed config bug for accuracy disagg test (untested)
Feb 25, 2026
23633b4
Merge branch 'main' into disagg_cluster_tests. Did some cleanup
Feb 26, 2026
dcb3f04
fixing test failures in auto scaling, accuracy tests, and test_disagg…
Feb 27, 2026
6977925
fix test failures
Feb 28, 2026
2d2c72c
revert accidental merge changes
Mar 2, 2026
8f137f2
Merge branch 'main' into disagg_cluster_tests
ekou24 Mar 2, 2026
b302fca
updated files to match test rename
Mar 2, 2026
b73ae11
added ray skip, deleted conftest
Mar 3, 2026
de38205
fixed yaml content
Mar 3, 2026
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
10 changes: 5 additions & 5 deletions tests/README.md
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -185,7 +185,7 @@ l0_a10:
linux_distribution_name: ubuntu*
tests:
# ------------- PyTorch tests ---------------
- disaggregated/test_disaggregated.py::test_disaggregated_single_gpu_with_mpirun[TinyLlama-1.1B-Chat-v1.0]
- disaggregated/test_disaggregated.py::test_disaggregated_single_gpu[TinyLlama-1.1B-Chat-v1.0]
- disaggregated/test_disaggregated.py::test_disaggregated_cuda_graph[TinyLlama-1.1B-Chat-v1.0]
- disaggregated/test_disaggregated.py::test_disaggregated_mixed[TinyLlama-1.1B-Chat-v1.0]
- disaggregated/test_disaggregated.py::test_disaggregated_overlap[TinyLlama-1.1B-Chat-v1.0]
Expand All @@ -200,7 +200,7 @@ l0_a10:
2. Copy all items in `tests` field to a text file, for example, `a10_list.txt`. Don't forget to remove extra characters like comments and the dash marks.

```
disaggregated/test_disaggregated.py::test_disaggregated_single_gpu_with_mpirun[TinyLlama-1.1B-Chat-v1.0]
disaggregated/test_disaggregated.py::test_disaggregated_single_gpu[TinyLlama-1.1B-Chat-v1.0]
disaggregated/test_disaggregated.py::test_disaggregated_cuda_graph[TinyLlama-1.1B-Chat-v1.0]
disaggregated/test_disaggregated.py::test_disaggregated_mixed[TinyLlama-1.1B-Chat-v1.0]
disaggregated/test_disaggregated.py::test_disaggregated_overlap[TinyLlama-1.1B-Chat-v1.0]
Expand All @@ -225,15 +225,15 @@ To set a timeout for specific long-running test cases, follow these steps:
1. Locate the test case line in the corresponding test-db YAML file (e.g., `tests/integration/test_lists/test-db/l0_a10.yml`).
2. Append `TIMEOUT (...)` to the test case line, as shown below:
```yaml
- disaggregated/test_disaggregated.py::test_disaggregated_single_gpu_with_mpirun[TinyLlama-1.1B-Chat-v1.0] TIMEOUT (30)
- disaggregated/test_disaggregated.py::test_disaggregated_single_gpu[TinyLlama-1.1B-Chat-v1.0] TIMEOUT (30)
```
- Ensure there is **at least one space** before and after the `TIMEOUT` keyword.
- The time value inside the parentheses `()` must be a **number** representing the timeout in **minutes**.

### For Local Testing (TXT files):
1. If you are running the tests locally using a prepared `.txt` file (e.g., `a10_list.txt`), append the `TIMEOUT` setting to the test case line in the same way:
```
disaggregated/test_disaggregated.py::test_disaggregated_single_gpu_with_mpirun[TinyLlama-1.1B-Chat-v1.0] TIMEOUT (30)
disaggregated/test_disaggregated.py::test_disaggregated_single_gpu[TinyLlama-1.1B-Chat-v1.0] TIMEOUT (30)
```

## 6. Set isolated execution for cases individually
Expand All @@ -251,7 +251,7 @@ Add `ISOLATION` to the test case line with proper spacing:

**For CI (test-db YAML files):**
```yaml
- disaggregated/test_disaggregated.py::test_disaggregated_single_gpu_with_mpirun[TinyLlama-1.1B-Chat-v1.0] ISOLATION
- disaggregated/test_disaggregated.py::test_disaggregated_single_gpu[TinyLlama-1.1B-Chat-v1.0] ISOLATION
```

## 7. Combining test markers
Expand Down
6 changes: 3 additions & 3 deletions tests/integration/defs/.test_durations
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -340,10 +340,10 @@
"disaggregated/test_disaggregated.py::test_disaggregated_kv_cache_time_output[TinyLlama-1.1B-Chat-v1.0]": 52.78952780482359,
"disaggregated/test_disaggregated.py::test_disaggregated_load_balance[TinyLlama-1.1B-Chat-v1.0]": 73.48997121001594,
"disaggregated/test_disaggregated.py::test_disaggregated_mixed[TinyLlama-1.1B-Chat-v1.0]": 67.3897166326642,
"disaggregated/test_disaggregated.py::test_disaggregated_multi_gpu_with_mpirun[TinyLlama-1.1B-Chat-v1.0]": 54.22262764698826,
"disaggregated/test_disaggregated.py::test_disaggregated_multi_gpu[TinyLlama-1.1B-Chat-v1.0]": 54.22262764698826,
"disaggregated/test_disaggregated.py::test_disaggregated_overlap[TinyLlama-1.1B-Chat-v1.0]": 98.97588296607137,
"disaggregated/test_disaggregated.py::test_disaggregated_single_gpu_with_mpirun[TinyLlama-1.1B-Chat-v1.0]": 67.9668476767838,
"disaggregated/test_disaggregated.py::test_disaggregated_single_gpu_with_mpirun_trt_backend[TinyLlama-1.1B-Chat-v1.0]": 82.28277984517626,
"disaggregated/test_disaggregated.py::test_disaggregated_single_gpu[TinyLlama-1.1B-Chat-v1.0]": 67.9668476767838,
"disaggregated/test_disaggregated.py::test_disaggregated_single_gpu_trt_backend[TinyLlama-1.1B-Chat-v1.0]": 82.28277984517626,
"disaggregated/test_disaggregated.py::test_disaggregated_trtllm_sampler[TinyLlama-1.1B-Chat-v1.0]": 62.51559329708107,
"disaggregated/test_disaggregated_single_gpu.py::test_disaggregated_llama_context_capacity[False-False-DeepSeek-V3-Lite-fp8/fp8]": 238.76137515995651,
"disaggregated/test_disaggregated_single_gpu.py::test_disaggregated_simple_deepseek[False-False-DeepSeek-V3-Lite-fp8/fp8]": 78.98068026197143,
Expand Down
Loading