Skip to content
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension


Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
4 changes: 2 additions & 2 deletions .github/wheel-constraints/nemo-platform-services.txt
Original file line number Diff line number Diff line change
Expand Up @@ -17,7 +17,7 @@ boto3==1.40.61
botocore==1.40.61
clickhouse-connect==0.15.1
cloudpickle==3.1.2
data-designer==0.6.1
data-designer==0.7.0
datasets==4.3.0
distro==1.9.0
docker==7.2.0
Expand All @@ -42,7 +42,7 @@ langchain-openai==1.3.5
langchain==1.3.13
lark==1.3.1
litellm<1.92 # 1.92.0 native build has no py3.14 wheel
nemo-anonymizer==0.2.1
nemo-anonymizer==0.3.0
nemo-safe-synthesizer==0.1.7
nemoguardrails==0.23.0
ngcsdk==4.20.1
Expand Down
6 changes: 3 additions & 3 deletions docs/data-designer/index.mdx
Original file line number Diff line number Diff line change
Expand Up @@ -10,7 +10,7 @@ Data Designer on NeMo Platform enables high-quality synthetic data generation th

Data Designer is a framework for orchestrating complex synthetic data generation workflows. It coordinates LLM calls, manages dependencies between data fields, handles batching and parallelization, and validates generated data against specifications.

The plugin is built on the open-source [NVIDIA NeMo Data Designer library](https://docs.nvidia.com/nemo/datadesigner/v0.6.1/getting-started/welcome) ([GitHub](https://github.com/NVIDIA-NeMo/DataDesigner)). The library provides the configuration and generation engine; the plugin provides CLI, SDK, Data Designer API, Jobs, Files API, Secrets API, and Inference Gateway API integration.
The plugin is built on the open-source [NVIDIA NeMo Data Designer library](https://docs.nvidia.com/nemo/datadesigner/v0.7.0/getting-started/welcome) ([GitHub](https://github.com/NVIDIA-NeMo/DataDesigner)). The library provides the configuration and generation engine; the plugin provides CLI, SDK, Data Designer API, Jobs, Files API, Secrets API, and Inference Gateway API integration.
Comment thread
mikeknep marked this conversation as resolved.

## How It Works

Expand Down Expand Up @@ -46,7 +46,7 @@ config_builder.add_column(dd.LLMTextColumnConfig(...))

Configuration code describes the dataset schema, columns, dependencies, constraints, seed data, processors, profilers, and inference settings.

**Learn more**: See the [library documentation](https://docs.nvidia.com/nemo/datadesigner/v0.6.1/getting-started/welcome) for comprehensive guides on column types, samplers, constraints, and advanced features.
**Learn more**: See the [library documentation](https://docs.nvidia.com/nemo/datadesigner/v0.7.0/getting-started/welcome) for comprehensive guides on column types, samplers, constraints, and advanced features.

### 2. Choose Where to Execute

Expand Down Expand Up @@ -99,7 +99,7 @@ Learn through examples: basics, seeding, and more.
Move configurations between local CLI and NeMo Services execution.

</Card>
<Card title="Library Documentation" href="https://docs.nvidia.com/nemo/datadesigner/v0.6.1/getting-started/welcome">
<Card title="Library Documentation" href="https://docs.nvidia.com/nemo/datadesigner/v0.7.0/getting-started/welcome">

Comprehensive guides on column types, constraints, and advanced features.

Expand Down
2 changes: 1 addition & 1 deletion docs/data-designer/migration.mdx
Original file line number Diff line number Diff line change
Expand Up @@ -83,4 +83,4 @@ Before switching execution modes, verify:
- **Execution Modes:** See [Execution Modes](/documentation/design-synthetic-data/execution-modes) for the conceptual model.
- **CLI:** See [Data Designer CLI](/documentation/design-synthetic-data/cli) for `run`, `submit`, and persona commands.
- **Tutorials:** Follow the [tutorials](/documentation/design-synthetic-data/tutorials) for hands-on examples.
- **Library Docs:** Refer to the [open-source library documentation](https://docs.nvidia.com/nemo/datadesigner/v0.6.1/getting-started/welcome) for configuration details.
- **Library Docs:** Refer to the [open-source library documentation](https://docs.nvidia.com/nemo/datadesigner/v0.7.0/getting-started/welcome) for configuration details.
8 changes: 4 additions & 4 deletions docs/data-designer/tutorials/basics.mdx
Original file line number Diff line number Diff line change
Expand Up @@ -6,7 +6,7 @@ description: ""

This tutorial demonstrates the fundamentals of Data Designer by generating a product review dataset.

For more detail about column behavior, see the [open-source library's version](https://docs.nvidia.com/nemo/datadesigner/v0.6.1/tutorials/the-basics) of this tutorial.
For more detail about column behavior, see the [open-source library's version](https://docs.nvidia.com/nemo/datadesigner/v0.7.0/tutorials/the-basics) of this tutorial.

## Prerequisites

Expand Down Expand Up @@ -47,7 +47,7 @@ config_builder = dd.DataDesignerConfigBuilder(model_configs)

### Add Columns

Define the columns for your dataset. The [library documentation](https://docs.nvidia.com/nemo/datadesigner/v0.6.1/tutorials/the-basics) explains these column types in detail.
Define the columns for your dataset. The [library documentation](https://docs.nvidia.com/nemo/datadesigner/v0.7.0/tutorials/the-basics) explains these column types in detail.


```python
Expand Down Expand Up @@ -305,5 +305,5 @@ When you use CLI `submit` or the SDK today:

- **Seed data:** Learn how to use external datasets in the [seeding tutorial](/documentation/design-synthetic-data/tutorials/seeding-with-external-datasets)
- **Execution modes:** Learn more about local and NeMo Services execution in [Execution Modes](/documentation/design-synthetic-data/execution-modes)
- **Column types:** Explore all available column types in the [library documentation](https://docs.nvidia.com/nemo/datadesigner/v0.6.1/concepts/columns)
- **Advanced features:** Learn about [processors](https://docs.nvidia.com/nemo/datadesigner/v0.6.1/concepts/processors) and [validation](https://docs.nvidia.com/nemo/datadesigner/v0.6.1/concepts/validators)
- **Column types:** Explore all available column types in the [library documentation](https://docs.nvidia.com/nemo/datadesigner/v0.7.0/concepts/columns)
- **Advanced features:** Learn about [processors](https://docs.nvidia.com/nemo/datadesigner/v0.7.0/concepts/processors) and [validation](https://docs.nvidia.com/nemo/datadesigner/v0.7.0/concepts/validators)
2 changes: 1 addition & 1 deletion docs/data-designer/tutorials/index.mdx
Original file line number Diff line number Diff line change
Expand Up @@ -18,7 +18,7 @@ Data Designer separates **configuration** (building dataset schemas) from **exec

**Part 1: Build Configs (Library)**

Use `data_designer.config` to define your dataset. See the [library documentation](https://docs.nvidia.com/nemo/datadesigner/v0.6.1/getting-started/welcome) for comprehensive guides on column types, constraints, and processors.
Use `data_designer.config` to define your dataset. See the [library documentation](https://docs.nvidia.com/nemo/datadesigner/v0.7.0/getting-started/welcome) for comprehensive guides on column types, constraints, and processors.

```python
import data_designer.config as dd
Expand Down
6 changes: 3 additions & 3 deletions docs/data-designer/tutorials/seeding.mdx
Original file line number Diff line number Diff line change
Expand Up @@ -6,7 +6,7 @@ description: ""

This tutorial demonstrates how to use external datasets as seed data for synthetic data generation in Data Designer.

For more detail about seed dataset behavior, see the [open-source library's version](https://docs.nvidia.com/nemo/datadesigner/v0.6.1/tutorials/seeding-with-an-external-dataset) of this tutorial.
For more detail about seed dataset behavior, see the [open-source library's version](https://docs.nvidia.com/nemo/datadesigner/v0.7.0/tutorials/seeding-with-an-external-dataset) of this tutorial.

## Seed Sources by Execution Mode

Expand Down Expand Up @@ -344,5 +344,5 @@ When you configure a seed dataset:
## Next Steps

- **Execution modes:** Learn more about local and NeMo Services execution in [Execution Modes](/documentation/design-synthetic-data/execution-modes)
- **Column types:** Explore all available column types in the [library documentation](https://docs.nvidia.com/nemo/datadesigner/v0.6.1/concepts/columns)
- **Processors:** Transform your data with processors in the [library documentation](https://docs.nvidia.com/nemo/datadesigner/v0.6.1/concepts/processors)
- **Column types:** Explore all available column types in the [library documentation](https://docs.nvidia.com/nemo/datadesigner/v0.7.0/concepts/columns)
- **Processors:** Transform your data with processors in the [library documentation](https://docs.nvidia.com/nemo/datadesigner/v0.7.0/concepts/processors)
2 changes: 1 addition & 1 deletion packages/data_designer_nemo/pyproject.toml
Original file line number Diff line number Diff line change
Expand Up @@ -6,7 +6,7 @@ requires-python = ">=3.11,<3.15"
authors = [{ name = "NVIDIA", email = "nemo@nvidia.com" }]
dependencies = [
"nemo-platform",
"data-designer==0.6.1",
"data-designer==0.7.0",
"anyio>=4.0",
"duckdb",
"pydantic>=2",
Expand Down
Original file line number Diff line number Diff line change
Expand Up @@ -54,7 +54,6 @@ def make_null_registry() -> ModelProviderRegistry:
# is semantically valid. The library requires a non-empty ModelProviderRegistry, so in this scenario
# we can provide this dummy null registry.
return ModelProviderRegistry(
default=_NO_OP,
providers=[make_noop_provider()],
)

Expand All @@ -74,12 +73,6 @@ async def make_local_first_model_provider_registry(
if len(model_configs) == 0:
return None

missing_providers = [model_config for model_config in model_configs if model_config.provider is None]
if len(missing_providers) > 0:
raise NDDInvalidConfigError(
f"Error: following model configs do not have an explicit provider defined: {missing_providers}"
)

logger.info("Building model provider registry. First checking locally-defined providers.")

local_registry = _make_local_model_provider_registry()
Expand Down Expand Up @@ -118,10 +111,7 @@ async def make_local_first_model_provider_registry(

all_providers = local_providers + igw_registry.providers

return ModelProviderRegistry(
default=all_providers[0].name,
providers=all_providers,
)
return ModelProviderRegistry(providers=all_providers)


async def _get_igw_model_provider_registry(
Expand Down Expand Up @@ -249,11 +239,7 @@ def get_model_provider_registry(self) -> ModelProviderRegistry | None:

if len(self.providers.values()) > 0:
registry_providers = [providers_tuple[0] for providers_tuple in self.providers.values()]
default = registry_providers[0].name
return ModelProviderRegistry(
default=default,
providers=registry_providers,
)
return ModelProviderRegistry(providers=registry_providers)


async def make_model_provider_registry(
Expand Down
Original file line number Diff line number Diff line change
Expand Up @@ -6,7 +6,6 @@
import pytest
from data_designer_nemo.fileset_file_seed_reader import FilesetFileSeedReader, workspace_cvar
from data_designer_nemo.fileset_file_seed_source import FilesetFileSeedSource
from nemo_platform_plugin.files.client import FilesClient


def test_dataset_uri_with_workspace() -> None:
Expand Down Expand Up @@ -40,18 +39,8 @@ def test_create_duckdb_connection_requires_injected_sdk() -> None:
def test_create_duckdb_connection_uses_injected_sdk() -> None:
sdk = Mock()
conn = Mock()
mock_files_client = Mock()

with (
patch("data_designer_nemo.fileset_file_seed_reader.duckdb.connect", return_value=conn),
patch("data_designer_nemo.fileset_file_seed_reader.FilesetFileSystem") as fileset_file_system,
patch(
"data_designer_nemo.fileset_file_seed_reader.client_from_platform",
return_value=mock_files_client,
) as mock_adapter,
):

with patch("data_designer_nemo.fileset_file_seed_reader.duckdb.connect", return_value=conn):
assert FilesetFileSeedReader(sdk).create_duckdb_connection() is conn

mock_adapter.assert_called_once_with(sdk, FilesClient)
fileset_file_system.assert_called_once_with(client=mock_files_client)
conn.register_filesystem.assert_called_once_with(fileset_file_system.return_value)
conn.register_filesystem.assert_called_once_with(sdk.files.fsspec)
Original file line number Diff line number Diff line change
Expand Up @@ -11,6 +11,7 @@ def _make_model_config(alias: str) -> dd.ModelConfig:
return dd.ModelConfig(
alias=alias,
model="nvidia/nemotron-3",
provider="default/nvidia",
)


Expand Down
8 changes: 4 additions & 4 deletions packages/nemo_platform/pyproject.toml
Original file line number Diff line number Diff line change
Expand Up @@ -87,7 +87,7 @@ core-service = [

# Generated from [tool.bundle-package]; do not edit by hand.
data-designer-nemo = [
"data-designer==0.6.1",
"data-designer==0.7.0",
"anyio>=4.0",
"duckdb",
"pydantic>=2",
Expand Down Expand Up @@ -242,8 +242,8 @@ nemo-agents-plugin = [
# Generated from [tool.bundle-package]; do not edit by hand.
nemo-anonymizer-plugin = [
"nemo-platform-plugin",
"nemo-anonymizer>=0.2.1",
"data-designer==0.6.1",
"nemo-anonymizer>=0.3.0",
"data-designer==0.7.0",
"data-designer-nemo",
"httpx>=0.27",
"fastapi",
Expand All @@ -264,7 +264,7 @@ nemo-auditor-plugin = [
# Generated from [tool.bundle-package]; do not edit by hand.
nemo-data-designer-plugin = [
"nemo-platform-plugin",
"data-designer==0.6.1",
"data-designer==0.7.0",
"data-designer-nemo",
"httpx>=0.27",
"fastapi",
Expand Down
4 changes: 2 additions & 2 deletions plugins/nemo-anonymizer/pyproject.toml
Original file line number Diff line number Diff line change
Expand Up @@ -7,8 +7,8 @@ authors = [{ name = "NVIDIA", email = "nemo@nvidia.com" }]
dependencies = [
"nemo-platform-plugin",
"nemo-platform",
"nemo-anonymizer>=0.2.1",
"data-designer==0.6.1",
"nemo-anonymizer>=0.3.0",
"data-designer==0.7.0",
"data-designer-nemo",
"httpx>=0.27",
"fastapi",
Expand Down
4 changes: 3 additions & 1 deletion plugins/nemo-anonymizer/tests/unit/test_run_job.py
Original file line number Diff line number Diff line change
Expand Up @@ -143,6 +143,7 @@ async def test_run_local_allows_missing_model_configs(
is_local=True,
)

assert isinstance(step_config, AnonymizerStepConfig)
assert step_config.model_configs_yaml == ""
assert step_config.dd_model_providers == []
round_tripped = AnonymizerStepConfig.model_validate(step_config.model_dump())
Expand All @@ -157,7 +158,6 @@ async def test_run_local_model_configs_uses_injected_async_sdk(
csv = tmp_path / "input.csv"
csv.write_text("text\nhello\n")
local_first_registry = ModelProviderRegistry(
default="local-provider",
providers=[NDDModelProvider(name="local-provider", endpoint="http://localhost:8000")],
)
local_first_lookup = AsyncMock(return_value=local_first_registry)
Expand All @@ -179,7 +179,9 @@ async def test_run_local_model_configs_uses_injected_async_sdk(
)

local_first_lookup.assert_awaited_once()
assert local_first_lookup.await_args is not None
assert local_first_lookup.await_args.kwargs["sdk"] is async_sdk
assert isinstance(step_config, AnonymizerStepConfig)
assert len(step_config.dd_model_providers) == 1
assert step_config.dd_model_providers[0]["name"] == "local-provider"

Expand Down
Loading
Loading