Skip to content
Merged
Show file tree
Hide file tree
Changes from 16 commits
Commits
Show all changes
28 commits
Select commit Hold shift + click to select a range
76fe56c
first commit
praateekmahajan Feb 24, 2026
04f02b3
verbose handling; cleaner serve.stop
praateekmahajan Feb 26, 2026
051dac2
improve restart and tests
praateekmahajan Feb 27, 2026
32e0291
update pyproject.toml
praateekmahajan Feb 27, 2026
d66a849
add serve,llm to cuda12
praateekmahajan Feb 27, 2026
8dd7d98
switch model server to singleton
praateekmahajan Mar 3, 2026
28019fb
remove logging
praateekmahajan Mar 4, 2026
1b5f8eb
update uv.lock + pr review comments
praateekmahajan Mar 6, 2026
45df4f4
remove logging from start too and add TODO
praateekmahajan Mar 6, 2026
be55a1f
Rename to InferenceModelConfig and InferenceServer and remove tests t…
praateekmahajan Mar 9, 2026
e304ee4
add boto3 comment in pyproject.toml
praateekmahajan Mar 9, 2026
2677415
Merge branch 'main' into praateek/ray-serve
praateekmahajan Mar 9, 2026
a56ee1b
greptile reviews
praateekmahajan Mar 9, 2026
c539e75
Merge branch 'praateek/ray-serve' of github.com:praateekmahajan/Curat…
praateekmahajan Mar 9, 2026
834e367
Add handling for ray.init() and ray.shutdown()
praateekmahajan Mar 10, 2026
a66aa1d
resolve uv lock conflict
praateekmahajan Mar 10, 2026
450a8a3
move to a separate pyproject.toml group
praateekmahajan Mar 10, 2026
60cd857
add boto3 also in inference_server group
praateekmahajan Mar 10, 2026
69d07b4
Update tutorial and benchmark
praateekmahajan Mar 10, 2026
4f8fcbe
That's not a secret
praateekmahajan Mar 10, 2026
ef72ea8
greptile reviews
praateekmahajan Mar 10, 2026
761339b
not a secret and let ruff respect that
praateekmahajan Mar 10, 2026
51c9678
PR Reviews
praateekmahajan Mar 10, 2026
8be76d3
Merge remote-tracking branch 'upstream/main' into praateek/ray-serve
praateekmahajan Mar 10, 2026
3a8eba8
Merge branch 'main' into praateek/ray-serve
praateekmahajan Mar 11, 2026
a8b3b57
PR Reviews + Move inference_server to sdg_cuda12
praateekmahajan Mar 11, 2026
17fcac1
update pyproject.toml for pip
praateekmahajan Mar 11, 2026
b3fe641
remove sdg_cuda12 due to slow pip
praateekmahajan Mar 11, 2026
File filter

Filter by extension

Filter by extension


Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
3 changes: 3 additions & 0 deletions nemo_curator/core/constants.py
Original file line number Diff line number Diff line change
Expand Up @@ -25,3 +25,6 @@
DEFAULT_RAY_MIN_WORKER_PORT = 10002
DEFAULT_RAY_MAX_WORKER_PORT = 19999
RAY_CLUSTER_START_VERIFICATION_TIMEOUT = 300

DEFAULT_SERVE_PORT = 8000
DEFAULT_SERVE_HEALTH_TIMEOUT_S = 300
397 changes: 397 additions & 0 deletions nemo_curator/core/serve.py

Large diffs are not rendered by default.

1 change: 1 addition & 0 deletions nemo_curator/models/client/llm_client.py
Original file line number Diff line number Diff line change
Expand Up @@ -45,6 +45,7 @@ class GenerationConfig:
temperature: float | None = 0.0
top_k: int | None = None
top_p: float | None = 0.95
extra_kwargs: dict | None = None


class LLMClient(ABC):
Expand Down
63 changes: 39 additions & 24 deletions nemo_curator/models/client/openai_client.py
Original file line number Diff line number Diff line change
Expand Up @@ -15,6 +15,7 @@
import warnings
from collections.abc import Iterable

from loguru import logger
from openai import AsyncOpenAI, OpenAI

from nemo_curator.models.client.llm_client import AsyncLLMClient, ConversationFormatter, GenerationConfig, LLMClient
Expand Down Expand Up @@ -56,18 +57,25 @@ def query_model(
if generation_config.top_k is not None:
warnings.warn("top_k is not used in an OpenAIClient", stacklevel=2)

response = self.client.chat.completions.create(
messages=messages,
model=model,
max_tokens=generation_config.max_tokens,
n=generation_config.n,
seed=generation_config.seed,
stop=generation_config.stop,
stream=generation_config.stream,
temperature=generation_config.temperature,
top_p=generation_config.top_p,
timeout=self.timeout,
)
create_kwargs = {
"messages": messages,
"model": model,
"max_tokens": generation_config.max_tokens,
"n": generation_config.n,
"seed": generation_config.seed,
"stop": generation_config.stop,
"stream": generation_config.stream,
"temperature": generation_config.temperature,
"top_p": generation_config.top_p,
"timeout": self.timeout,
}
if generation_config.extra_kwargs:
overlapping = set(generation_config.extra_kwargs) & set(create_kwargs)
if overlapping:
logger.warning(f"extra_kwargs will overwrite existing parameter(s): {overlapping}")
create_kwargs.update(generation_config.extra_kwargs)
Comment on lines +72 to +76

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

extra_kwargs silently overwrites core generation parameters

The create_kwargs.update(generation_config.extra_kwargs) call allows any key in extra_kwargs to silently overwrite already-set parameters such as model, messages, temperature, max_tokens, timeout, etc. A caller who puts temperature=0.5 in extra_kwargs expecting it to be an extension will silently override the value from generation_config.temperature, leading to subtle bugs.

Consider guarding against collisions:

if generation_config.extra_kwargs:
    overlap = set(generation_config.extra_kwargs) & set(create_kwargs)
    if overlap:
        warnings.warn(
            f"extra_kwargs keys {overlap} overlap with standard GenerationConfig fields and will override them.",
            stacklevel=2,
        )
    create_kwargs.update(generation_config.extra_kwargs)

The same pattern applies to AsyncOpenAIClient._query_model_impl at lines 142–143.


response = self.client.chat.completions.create(**create_kwargs)

return [choice.message.content for choice in response.choices]

Expand Down Expand Up @@ -123,17 +131,24 @@ async def _query_model_impl(
if generation_config.top_k is not None:
warnings.warn("top_k is not used in an AsyncOpenAIClient", stacklevel=2)

response = await self.client.chat.completions.create(
messages=messages,
model=model,
max_tokens=generation_config.max_tokens,
n=generation_config.n,
seed=generation_config.seed,
stop=generation_config.stop,
stream=generation_config.stream,
temperature=generation_config.temperature,
top_p=generation_config.top_p,
timeout=self.timeout,
)
create_kwargs = {
"messages": messages,
"model": model,
"max_tokens": generation_config.max_tokens,
"n": generation_config.n,
"seed": generation_config.seed,
"stop": generation_config.stop,
"stream": generation_config.stream,
"temperature": generation_config.temperature,
"top_p": generation_config.top_p,
"timeout": self.timeout,
}
if generation_config.extra_kwargs:
overlapping = set(generation_config.extra_kwargs) & set(create_kwargs)
if overlapping:
logger.warning(f"extra_kwargs will overwrite existing parameter(s): {overlapping}")
create_kwargs.update(generation_config.extra_kwargs)

response = await self.client.chat.completions.create(**create_kwargs)

return [choice.message.content for choice in response.choices]
24 changes: 22 additions & 2 deletions nemo_curator/pipeline/pipeline.py
Original file line number Diff line number Diff line change
Expand Up @@ -189,9 +189,29 @@ def run(self, executor: BaseExecutor | None = None, initial_tasks: list[Task] |
list[Task] | None: List of tasks
"""
self.build()
if executor is None:
from nemo_curator.backends.xenna import XennaExecutor

from nemo_curator.backends.xenna import XennaExecutor
Comment thread
praateekmahajan marked this conversation as resolved.
Resolved

if executor is None:
executor = XennaExecutor()

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Unconditional XennaExecutor import breaks non-xenna deployments

The import of XennaExecutor was moved out of the if executor is None guard so that isinstance(executor, XennaExecutor) can reference it. However, this means every call to pipeline.run() now eagerly imports XennaExecutor, even when the caller explicitly provides their own executor.

Before this PR, users who provided a custom executor (e.g., RayDataExecutor) could run pipeline.run(executor=my_executor) without needing the xenna backend installed. After this change, if nemo_curator.backends.xenna is not importable in their environment, they will get an ImportError even though XennaExecutor is never actually used.

A minimal fix is to guard the isinstance check with a try/except on the import:

self.build()

if executor is None:
    from nemo_curator.backends.xenna import XennaExecutor
    executor = XennaExecutor()

from nemo_curator.core.serve import is_ray_serve_active

if is_ray_serve_active():
    gpu_stages = [s for s in self.stages if s.resources.requires_gpu]
    if gpu_stages:
        names = ", ".join(s.name for s in gpu_stages)
        try:
            from nemo_curator.backends.xenna import XennaExecutor as _XennaExecutor
            is_xenna = isinstance(executor, _XennaExecutor)
        except ImportError:
            is_xenna = False
        if is_xenna:
            ...


from nemo_curator.core.serve import is_ray_serve_active

if is_ray_serve_active():
gpu_stages = [s for s in self.stages if s.resources.requires_gpu]
if gpu_stages:
names = ", ".join(s.name for s in gpu_stages)
if isinstance(executor, XennaExecutor):
msg = (
f"Cannot run XennaExecutor with GPU stages [{names}] while Ray Serve is active. "
"Xenna manages GPU assignment independently of Ray's resource scheduler, "
"which causes GPU contention with served models. "
"Use RayDataExecutor or RayActorPoolExecutor instead."
Comment thread
praateekmahajan marked this conversation as resolved.
Resolved
)
raise RuntimeError(msg)
Comment on lines +196 to +209

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

XennaExecutor import at line 200 breaks environments without xenna when Ray Serve is active

The import from nemo_curator.backends.xenna import XennaExecutor at line 200 is inside if gpu_stages: (which itself is inside if is_ray_serve_active():), but it runs whenever the caller provides their own executor (non-None) and Ray Serve happens to be active with GPU stages present.

Before this PR, the XennaExecutor import was only reached when executor is None (line 189-192). After this PR, a user who passes executor=RayDataExecutor() explicitly and doesn't have the xenna backend installed will get an ImportError at line 200 purely because is_ray_serve_active() and gpu_stages happen to be truthy — even though XennaExecutor is never used.

Fix by wrapping the isinstance check in a try/except on the import:

if is_ray_serve_active():
    gpu_stages = [s for s in self.stages if s.resources.requires_gpu]
    if gpu_stages:
        names = ", ".join(s.name for s in gpu_stages)
        try:
            from nemo_curator.backends.xenna import XennaExecutor as _XennaExecutor
            is_xenna = isinstance(executor, _XennaExecutor)
        except ImportError:
            is_xenna = False
        if is_xenna:
            msg = (...)
            raise RuntimeError(msg)
        logger.info(...)

logger.info(
f"Ray Serve is active and pipeline has GPU stages: [{names}]. "

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Do we know if Xenna handles contention since it has its own allocation logic? If not for Xenna jobs we might want to disallow GPU stages completely.

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Good point, I know xenna==0.1.2 definitely just allocates the stage to all available GPUs and in this case it gets stuck in a deadlock. I'll add an exception in that case.

"The executor will schedule GPU stages on GPUs not held by Serve."
)

return executor.execute(self.stages, initial_tasks)
10 changes: 9 additions & 1 deletion pyproject.toml
Original file line number Diff line number Diff line change
Expand Up @@ -64,12 +64,18 @@ dependencies = [
"pandas>=2.1.0",
"pyarrow",
"ray[default,data]>=2.54",
"boto3", # Get rid once https://github.com/ray-project/ray/issues/61269 is fixed

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

boto3 added as a mandatory dependency without a version constraint

boto3 is added as a top-level (unconditional) core dependency to work around a Ray issue (ray-project/ray#61269). This is problematic for two reasons:

  1. No version constraint — boto3 will float to whatever the latest version is at install time. Given that boto3 and botocore pin each other very tightly and are commonly pinned by downstream tools (e.g., aiobotocore, AWS CLI), an unconstrained boto3 can cause dependency-resolution conflicts.
  2. Mandatory for all users — this is an AWS SDK that most NeMo Curator users (non-Ray-Serve, non-S3 workflows) don't need. It would be cleaner to add it to the cuda12 (or a new serve) optional extras group alongside ray[serve,llm], which is the actual feature that needs it.
# before
"boto3", # Get rid once https://github.com/ray-project/ray/issues/61269 is fixed

# after — move into the cuda12/serve optional group
cuda12 = [
    ...
    "boto3",  # workaround for ray-project/ray#61269; remove once fixed
    "ray[serve,llm]>=2.54"
]

"torch",
"transformers",
]

[project.optional-dependencies]
cuda12 = ["gpustat", "nvidia-ml-py"]
cuda12 = [
"gpustat",
"nvidia-ml-py",
"nixl-cu12>=0.10.0; (platform_machine == 'x86_64' and platform_system != 'Darwin')",
"ray[serve,llm]>=2.54"
]
vllm = ["vllm>=0.14.1; (platform_machine == 'x86_64' and platform_system != 'Darwin')"]

# Installs CPU + GPU text curation modules
Expand Down Expand Up @@ -258,6 +264,8 @@ torchvision = [
{ index = "pytorch", marker = "platform_machine == 'x86_64' and sys_platform != 'darwin'" },
{ index = "pypi", marker = "platform_machine != 'x86_64' or sys_platform == 'darwin'" },
]
nixl = { index = "pypi" }
Comment thread
praateekmahajan marked this conversation as resolved.
Resolved
nixl-cu12 = { index = "pypi" }

[tool.coverage.paths]
source = ["./nemo_curator", "/workspace/nemo_curator", "/home/runner/_work/Curator/Curator/NeMo-Curator/nemo_curator"]
Expand Down
Loading
Loading