Skip to content

fix(test): cap RL worker GPU memory so test_rl_worker_discovery survives parallel bin-packing #10602

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Merged
nv-tusharma merged 4 commits into from
Jun 11, 2026
Merged

fix(test): cap RL worker GPU memory so test_rl_worker_discovery survives parallel bin-packing #10602

Changes from all commits
Commits
Show all changes
4 commits
Select commit Hold shift + click to select a range
d68d60d
fix(test): cap RL worker GPU memory so it survives parallel bin-packing
nv-tusharma Jun 11, 2026
c0b5e6d
test: TEMP run RL worker discovery in pre-merge to validate GPU-mem fix
nv-tusharma Jun 11, 2026
72e010e
Revert "test: TEMP run RL worker discovery in pre-merge to validate G…
nv-tusharma Jun 11, 2026
338199d
Merge branch 'main' into tushar/fix-rl-worker-discovery-gpu-mem-binpack
nv-tusharma Jun 11, 2026
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
11 changes: 11 additions & 0 deletions tests/rl/test_worker_discovery.py
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -14,6 +14,7 @@
import requests

from tests.utils.constants import QWEN
from tests.utils.gpu_args import build_gpu_mem_args
from tests.utils.managed_process import DynamoFrontendProcess, ManagedProcess
from tests.utils.payloads import check_models_api
from tests.utils.port_utils import ServicePorts, allocate_port, deallocate_port
Expand Down Expand Up @@ -100,6 +101,15 @@ def _prepare_log_dir(request: pytest.FixtureRequest, suffix: str) -> str:
return str(log_dir)


def _vllm_gpu_mem_args(default_utilization: str) -> list[str]:
# Honor the GPU scheduler's per-worker KV-cache budget under bin-packing;
# fall back to a conservative utilization for serial runs.
return build_gpu_mem_args("build_vllm_gpu_mem_args") or [
"--gpu-memory-utilization",
default_utilization,
]


class RLVllmWorkerProcess(ManagedProcess):
def __init__(
self,
Expand All @@ -122,6 +132,7 @@ def __init__(
"--model",
TEST_MODEL,
"--enforce-eager",
*_vllm_gpu_mem_args("0.4"),
"--max-model-len",
"2048",
"--max-num-seqs",
Expand Down
Loading