Repository navigation
Support Ray Serve through InferenceModelConfig and InferenceServer #1541
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
Changes from 16 commits
76fe56c
04f02b3
051dac2
32e0291
d66a849
8dd7d98
28019fb
1b5f8eb
45df4f4
be55a1f
e304ee4
2677415
a56ee1b
c539e75
834e367
a66aa1d
450a8a3
60cd857
69d07b4
4f8fcbe
ef72ea8
761339b
51c9678
8be76d3
3a8eba8
a8b3b57
17fcac1
b3fe641
File filter
Filter by extension
Conversations
Jump to
Diff view
Diff view
There are no files selected for viewing
Large diffs are not rendered by default.
| Original file line number | Diff line number | Diff line change |
|---|---|---|
|
|
@@ -189,9 +189,29 @@ def run(self, executor: BaseExecutor | None = None, initial_tasks: list[Task] | | |
| list[Task] | None: List of tasks | ||
| """ | ||
| self.build() | ||
| if executor is None: | ||
| from nemo_curator.backends.xenna import XennaExecutor | ||
|
|
||
| from nemo_curator.backends.xenna import XennaExecutor | ||
|
praateekmahajan marked this conversation as resolved.
Resolved
|
||
|
|
||
| if executor is None: | ||
| executor = XennaExecutor() | ||
|
Contributor
There was a problem hiding this comment. Choose a reason for hiding this commentThe reason will be displayed to describe this comment to others. Learn more. Unconditional The import of Before this PR, users who provided a custom executor (e.g., A minimal fix is to guard the self.build()
if executor is None:
from nemo_curator.backends.xenna import XennaExecutor
executor = XennaExecutor()
from nemo_curator.core.serve import is_ray_serve_active
if is_ray_serve_active():
gpu_stages = [s for s in self.stages if s.resources.requires_gpu]
if gpu_stages:
names = ", ".join(s.name for s in gpu_stages)
try:
from nemo_curator.backends.xenna import XennaExecutor as _XennaExecutor
is_xenna = isinstance(executor, _XennaExecutor)
except ImportError:
is_xenna = False
if is_xenna:
... |
||
|
|
||
| from nemo_curator.core.serve import is_ray_serve_active | ||
|
|
||
| if is_ray_serve_active(): | ||
| gpu_stages = [s for s in self.stages if s.resources.requires_gpu] | ||
| if gpu_stages: | ||
| names = ", ".join(s.name for s in gpu_stages) | ||
| if isinstance(executor, XennaExecutor): | ||
| msg = ( | ||
| f"Cannot run XennaExecutor with GPU stages [{names}] while Ray Serve is active. " | ||
| "Xenna manages GPU assignment independently of Ray's resource scheduler, " | ||
| "which causes GPU contention with served models. " | ||
| "Use RayDataExecutor or RayActorPoolExecutor instead." | ||
|
praateekmahajan marked this conversation as resolved.
Resolved
|
||
| ) | ||
| raise RuntimeError(msg) | ||
|
Comment on lines
+196
to
+209
Contributor
There was a problem hiding this comment. Choose a reason for hiding this commentThe reason will be displayed to describe this comment to others. Learn more. XennaExecutor import at line 200 breaks environments without xenna when Ray Serve is active The import Before this PR, the Fix by wrapping the isinstance check in a try/except on the import: if is_ray_serve_active():
gpu_stages = [s for s in self.stages if s.resources.requires_gpu]
if gpu_stages:
names = ", ".join(s.name for s in gpu_stages)
try:
from nemo_curator.backends.xenna import XennaExecutor as _XennaExecutor
is_xenna = isinstance(executor, _XennaExecutor)
except ImportError:
is_xenna = False
if is_xenna:
msg = (...)
raise RuntimeError(msg)
logger.info(...) |
||
| logger.info( | ||
| f"Ray Serve is active and pipeline has GPU stages: [{names}]. " | ||
|
Contributor
There was a problem hiding this comment. Choose a reason for hiding this commentThe reason will be displayed to describe this comment to others. Learn more. Do we know if Xenna handles contention since it has its own allocation logic? If not for Xenna jobs we might want to disallow GPU stages completely.
Contributor
Author
There was a problem hiding this comment. Choose a reason for hiding this commentThe reason will be displayed to describe this comment to others. Learn more. Good point, I know xenna==0.1.2 definitely just allocates the stage to all available GPUs and in this case it gets stuck in a deadlock. I'll add an exception in that case. |
||
| "The executor will schedule GPU stages on GPUs not held by Serve." | ||
| ) | ||
|
|
||
| return executor.execute(self.stages, initial_tasks) | ||
| Original file line number | Diff line number | Diff line change |
|---|---|---|
|
|
@@ -64,12 +64,18 @@ dependencies = [ | |
| "pandas>=2.1.0", | ||
| "pyarrow", | ||
| "ray[default,data]>=2.54", | ||
| "boto3", # Get rid once https://github.com/ray-project/ray/issues/61269 is fixed | ||
|
Contributor
There was a problem hiding this comment. Choose a reason for hiding this commentThe reason will be displayed to describe this comment to others. Learn more.
# before
"boto3", # Get rid once https://github.com/ray-project/ray/issues/61269 is fixed
# after — move into the cuda12/serve optional group
cuda12 = [
...
"boto3", # workaround for ray-project/ray#61269; remove once fixed
"ray[serve,llm]>=2.54"
] |
||
| "torch", | ||
| "transformers", | ||
| ] | ||
|
|
||
| [project.optional-dependencies] | ||
| cuda12 = ["gpustat", "nvidia-ml-py"] | ||
| cuda12 = [ | ||
| "gpustat", | ||
| "nvidia-ml-py", | ||
| "nixl-cu12>=0.10.0; (platform_machine == 'x86_64' and platform_system != 'Darwin')", | ||
| "ray[serve,llm]>=2.54" | ||
| ] | ||
| vllm = ["vllm>=0.14.1; (platform_machine == 'x86_64' and platform_system != 'Darwin')"] | ||
|
|
||
| # Installs CPU + GPU text curation modules | ||
|
|
@@ -258,6 +264,8 @@ torchvision = [ | |
| { index = "pytorch", marker = "platform_machine == 'x86_64' and sys_platform != 'darwin'" }, | ||
| { index = "pypi", marker = "platform_machine != 'x86_64' or sys_platform == 'darwin'" }, | ||
| ] | ||
| nixl = { index = "pypi" } | ||
|
praateekmahajan marked this conversation as resolved.
Resolved
|
||
| nixl-cu12 = { index = "pypi" } | ||
|
|
||
| [tool.coverage.paths] | ||
| source = ["./nemo_curator", "/workspace/nemo_curator", "/home/runner/_work/Curator/Curator/NeMo-Curator/nemo_curator"] | ||
|
|
||
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
extra_kwargssilently overwrites core generation parametersThe
create_kwargs.update(generation_config.extra_kwargs)call allows any key inextra_kwargsto silently overwrite already-set parameters such asmodel,messages,temperature,max_tokens,timeout, etc. A caller who putstemperature=0.5inextra_kwargsexpecting it to be an extension will silently override the value fromgeneration_config.temperature, leading to subtle bugs.Consider guarding against collisions:
The same pattern applies to
AsyncOpenAIClient._query_model_implat lines 142–143.