docs: add NeMo Data Designer (NDD) integration docs for 26.04

fern/versions/v26.04.yml

@@ -276,6 +276,9 @@ navigation:
               - page: Inference Server
                 path: ./v26.04/pages/curate-text/synthetic/inference-server.mdx
                 slug: inference-server
+              - page: NeMo Data Designer
+                path: ./v26.04/pages/curate-text/synthetic/nemo-data-designer.mdx
+                slug: nemo-data-designer
               - page: Multilingual Q&A
                 path: ./v26.04/pages/curate-text/synthetic/multilingual-qa.mdx
                 slug: multilingual-qa

fern/versions/v26.04/pages/about/release-notes/index.mdx

@@ -146,6 +146,17 @@ Standardized return type for all deduplication workflows:
 
 Added tqdm progress bars to `RayActorPoolExecutor` for real-time visibility into task completion during stage processing and shuffle inserts. Progress bars are enabled by default and can be configured with `show_progress` and `progress_interval` parameters. This is particularly useful for long-running deduplication jobs where progress is not otherwise apparent.
 
+### NeMo Data Designer Integration
+
+Integrated the NeMo Data Designer (NDD) client with NeMo Curator for declarative synthetic data generation at scale:
+
+- **`DataDesignerStage`**: New processing stage that wraps NDD's `DataDesigner.preview()` to generate structured synthetic data within Curator pipelines. Accepts a `DataDesignerConfigBuilder` or YAML config file and supports sampler columns (Faker names, UUIDs, dates), expression columns (Jinja templates), and LLM text columns.
+- **NDD-backed Nemotron-CC stages**: Drop-in replacements for the five Nemotron-CC stages (`WikipediaParaphrasingStage`, `DiverseQAStage`, `DistillStage`, `ExtractKnowledgeStage`, `KnowledgeListStage`) that route generation through NDD instead of `AsyncOpenAIClient`. Import from `nemo_curator.stages.synthetic.nemotron_cc.nemo_data_designer.nemotron_cc`.
+- **Token metric collection**: `DataDesignerStage` automatically reports `input_tokens_median_per_record` and `output_tokens_median_per_record` from NDD's analysis, enabling throughput tracking without manual instrumentation.
+- **Local and remote inference**: Supports both the built-in `InferenceServer` (Ray Serve + vLLM) via custom `ModelProvider` and remote endpoints (NVIDIA NIM).
+
+Learn more in the [NeMo Data Designer](/curate-text/synthetic/nemo-data-designer) documentation.
+
 ### S3 Transport for CommonCrawlWARCReader (PR #1604)
 
 Added S3 as an alternative transport for `CommonCrawlWARCReader`, which fetches individual WARC records using byte-range requests:

fern/versions/v26.04/pages/curate-text/synthetic/index.mdx

@@ -37,6 +37,10 @@ Improve or restructure existing data using LLM capabilities. The Nemotron-CC sta
 - Condensed knowledge distillations
 - Extracted factual content
 
+### Declarative Mode (NeMo Data Designer)
+
+Define data generation pipelines declaratively using [NeMo Data Designer](/curate-text/synthetic/nemo-data-designer) (NDD). Instead of writing imperative LLM call logic, you configure structured column generation (samplers, expressions, LLM text columns) through a builder API or YAML file. NDD handles execution, batching, and token metric collection. This mode supports both standalone generation and NDD-backed versions of Nemotron-CC stages.
+
 ## Architecture
 
 The following diagram shows how SDG pipelines process data through preprocessing, LLM generation, and postprocessing stages:
@@ -95,6 +99,7 @@ Nemotron-CC pipelines use the `transformers` library for tokenization, which is
 | `DistillStage` | Create condensed, information-dense paraphrases | Document text |
 | `ExtractKnowledgeStage` | Extract knowledge as textbook-style passages | Document text |
 | `KnowledgeListStage` | Extract structured fact lists | Document text |
+| `DataDesignerStage` | Declarative generation via NeMo Data Designer | Seed data (any schema) |
 
 ---
 
@@ -120,6 +125,12 @@ quickstart
 tutorial
 </Card>
 
+<Card title="NeMo Data Designer" href="/curate-text/synthetic/nemo-data-designer">
+Declarative data generation with structured columns and NDD-backed Nemotron-CC stages
+ndd
+declarative
+</Card>
+
 <Card title="Nemotron-CC Pipelines" href="/curate-text/synthetic/nemotron-cc">
 Advanced text transformation and knowledge extraction workflows
 advanced

fern/versions/v26.04/pages/curate-text/synthetic/nemo-data-designer.mdx

@@ -0,0 +1,316 @@
+---
+description: "Scale synthetic data generation using NeMo Data Designer (NDD) with declarative configuration and structured column generation"
+categories: ["how-to-guides"]
+tags: ["nemo-data-designer", "ndd", "synthetic-data", "data-designer", "declarative-config", "structured-generation"]
+personas: ["data-scientist-focused", "mle-focused"]
+difficulty: "intermediate"
+content_type: "how-to"
+modality: "text-only"
+---
+
+# NeMo Data Designer Integration
+
+[NeMo Data Designer (NDD)](https://nvidia-nemo.github.io/DataDesigner/latest/) is a declarative data generation framework that integrates with NeMo Curator to scale synthetic data pipelines. Instead of writing imperative LLM call logic, you define a configuration that describes what columns to generate, how to sample structured fields, and which LLM to use. NDD handles execution, batching, and token metric collection automatically.
+
+## How It Works
+
+NeMo Curator wraps NDD through the `DataDesignerStage`, which accepts a `DataDesignerConfigBuilder` or a YAML config file. The stage:
+
+1. Takes input records from a `DocumentBatch`
+2. Passes them to NDD as a seed dataset
+3. Calls `DataDesigner.preview()` to generate new columns (samplers, expressions, LLM text)
+4. Returns the enriched dataset as a new `DocumentBatch` with token usage metrics
+
+```mermaid
+flowchart LR
+    A["Seed Data<br/>(JSONL/Parquet)"] --> B["DataDesignerStage"]
+    B --> C["NDD Engine<br/>(Samplers, Expressions,<br/>LLM Generation)"]
+    C --> D["Enriched Output<br/>(JSONL/Parquet)"]
+
+    E["LLM Endpoint<br/>(Local InferenceServer<br/>or NVIDIA NIM)"] -.->|"API Calls"| C
+
+    classDef stage fill:#e3f2fd,stroke:#1976d2,stroke-width:2px,color:#000
+    classDef infra fill:#f3e5f5,stroke:#7b1fa2,stroke-width:2px,color:#000
+    classDef output fill:#e8f5e9,stroke:#2e7d32,stroke-width:3px,color:#000
+
+    class A,B stage
+    class D output
+    class C,E infra
+```
+
+## Prerequisites
+
+Install the NDD dependency:
+
+```bash
+uv pip install --extra-index-url https://pypi.nvidia.com nemo-curator[text_cuda12]
+```
+
+The `data-designer` package is included in the text extras. For local model serving, also install:
+
+```bash
+uv pip install nemo-curator[inference_server]
+```
+
+## DataDesignerStage
+
+The `DataDesignerStage` is the core integration point between NeMo Curator and NDD.
+
+### Parameters
+
+| Parameter | Type | Default | Description |
+| --- | --- | --- | --- |
+| `config_builder` | `DataDesignerConfigBuilder` | None | NDD configuration builder. Mutually exclusive with `data_designer_config_file`. |
+| `data_designer_config_file` | str | None | Path to a YAML config file. Mutually exclusive with `config_builder`. |
+| `model_providers` | list | None | Custom `ModelProvider` instances for local or test endpoints. If None, NDD uses its default providers. |
+| `verbose` | bool | False | When True, show full NDD log output. |
+
+### Metrics
+
+`DataDesignerStage` automatically collects and reports:
+
+- `ndd_running_time`: Wall-clock time for the NDD `preview()` call
+- `num_input_records` / `num_output_records`: Record counts before and after generation
+- `input_tokens_median_per_record` / `output_tokens_median_per_record`: Median token counts across all LLM columns
+
+## Building a Configuration
+
+NDD configurations use a builder pattern. You add columns of three types:
+
+<Tip>
+For full documentation for building NDD configuration, see the [NDD config builder reference](https://nvidia-nemo.github.io/DataDesigner/latest/code_reference/config_builder/).
+</Tip>
+
+### Sampler Columns
+
+Generate structured data using built-in samplers (Faker names, UUIDs, dates):
+
+```python
+import data_designer.config as dd
+
+config_builder = dd.DataDesignerConfigBuilder(model_configs=[model_config])
+
+config_builder.add_column(
+    dd.SamplerColumnConfig(
+        name="patient_name",
+        sampler_type=dd.SamplerType.PERSON_FROM_FAKER,
+        params=dd.PersonFromFakerSamplerParams(),
+    )
+)
+
+config_builder.add_column(
+    dd.SamplerColumnConfig(
+        name="patient_id",
+        sampler_type=dd.SamplerType.UUID,
+        params=dd.UUIDSamplerParams(prefix="PT-", short_form=True, uppercase=True),
+    )
+)
+```
+
+### Expression Columns
+
+Derive values from other columns using Jinja templates:
+
+```python
+config_builder.add_column(
+    dd.ExpressionColumnConfig(
+        name="first_name",
+        expr="{{ patient_name.first_name }}",
+    )
+)
+```
+
+### LLM Text Columns
+
+Generate text using an LLM with prompts that reference other columns:
+
+```python
+config_builder.add_column(
+    dd.LLMTextColumnConfig(
+        name="physician_notes",
+        prompt="""\
+You are a primary-care physician who just had an appointment with {{ first_name }}.
+{{ patient_summary }}
+Write careful notes about your visit. Respond with only the notes.
+""",
+        model_alias="local-llm",
+    )
+)
+```
+
+## End-to-End Example
+
+This example generates synthetic medical notes from seed symptom data using a local `InferenceServer`:
+
+```python
+import data_designer.config as dd
+
+from nemo_curator.backends.ray_data import RayDataExecutor
+from nemo_curator.core.client import RayClient
+from nemo_curator.core.serve import InferenceModelConfig, InferenceServer
+from nemo_curator.pipeline import Pipeline
+from nemo_curator.stages.synthetic.nemo_data_designer.data_designer import DataDesignerStage
+from nemo_curator.stages.text.io.reader.jsonl import JsonlReader
+from nemo_curator.stages.text.io.writer.jsonl import JsonlWriter
+
+# Start Ray cluster
+client = RayClient(num_cpus=16, num_gpus=4)
+client.start()
+
+# Start local inference server
+server_config = InferenceModelConfig(
+    model_identifier="google/gemma-3-27b-it",
+    deployment_config={"autoscaling_config": {"min_replicas": 1, "max_replicas": 1}},
+    engine_kwargs={"tensor_parallel_size": 4},
+)
+inference_server = InferenceServer(models=[server_config])
+inference_server.start()
+
+# Configure NDD model
+model_config = dd.ModelConfig(
+    alias="local-llm",
+    model="google/gemma-3-27b-it",
+    provider="local",
+    skip_health_check=True,
+    inference_parameters=dd.ChatCompletionInferenceParams(
+        temperature=1.0, top_p=1.0, max_tokens=2048,
+    ),
+)
+
+model_provider = dd.ModelProvider(
+    name="local",
+    endpoint=inference_server.endpoint,
+    api_key="unused",
+)
+
+# Build config with sampler and LLM columns
+config_builder = dd.DataDesignerConfigBuilder(model_configs=[model_config])
+
+config_builder.add_column(
+    dd.SamplerColumnConfig(
+        name="patient_name",
+        sampler_type=dd.SamplerType.PERSON_FROM_FAKER,
+        params=dd.PersonFromFakerSamplerParams(),
+    )
+)
+
+config_builder.add_column(
+    dd.LLMTextColumnConfig(
+        name="physician_notes",
+        prompt="You are a physician. Write notes for {{ patient_name.first_name }} "
+               "who has {{ diagnosis }}. {{ patient_summary }}",
+        model_alias="local-llm",
+    )
+)
+
+# Build and run pipeline
+pipeline = Pipeline(name="ndd_medical_notes")
+pipeline.add_stage(JsonlReader(file_paths="seed_data/*.jsonl", fields=["diagnosis", "patient_summary"]))
+pipeline.add_stage(DataDesignerStage(config_builder=config_builder, model_providers=[model_provider]))
+pipeline.add_stage(JsonlWriter(path="./synthetic_output"))
+
+pipeline.run(executor=RayDataExecutor())
+
+inference_server.stop()
+client.stop()
+```
+
+## Using a Remote Provider
+
+To use NVIDIA NIM or another hosted endpoint instead of a local server, configure the `ModelProvider` with the remote URL and API key:
+
+```python
+import os
+
+import data_designer.config as dd
+
+from nemo_curator.stages.synthetic.nemo_data_designer.data_designer import DataDesignerStage
+
+model_config = dd.ModelConfig(
+    alias="nim-llm",
+    model="meta/llama-3.3-70b-instruct",
+    provider="nvidia",
+    inference_parameters=dd.ChatCompletionInferenceParams(
+        temperature=0.5, top_p=0.9, max_tokens=1600,
+    ),
+)
+
+model_provider = dd.ModelProvider(
+    name="nvidia",
+    endpoint="https://integrate.api.nvidia.com/v1",
+    provider_type="openai",
+    api_key=os.environ["NVIDIA_API_KEY"],
+)
+
+config_builder = dd.DataDesignerConfigBuilder(model_configs=[model_config])
+# Add columns as needed...
+
+stage = DataDesignerStage(
+    config_builder=config_builder,
+    model_providers=[model_provider],
+)
+```
+
+## NDD-Backed Nemotron-CC Stages
+
+The Nemotron-CC synthetic data stages have NDD-backed equivalents that replace the `AsyncOpenAIClient` with NDD execution. These stages accept the same `input_field`, `output_field`, and prompt parameters, but route generation through `DataDesignerStage` internally.
+
+| Stage | Import Path | Output Field |
+| --- | --- | --- |
+| `WikipediaParaphrasingStage` | `nemo_curator.stages.synthetic.nemotron_cc.nemo_data_designer.nemotron_cc` | `rephrased` |
+| `DiverseQAStage` | `nemo_curator.stages.synthetic.nemotron_cc.nemo_data_designer.nemotron_cc` | `diverse_qa` |
+| `DistillStage` | `nemo_curator.stages.synthetic.nemotron_cc.nemo_data_designer.nemotron_cc` | `distill` |
+| `ExtractKnowledgeStage` | `nemo_curator.stages.synthetic.nemotron_cc.nemo_data_designer.nemotron_cc` | `extract_knowledge` |
+| `KnowledgeListStage` | `nemo_curator.stages.synthetic.nemotron_cc.nemo_data_designer.nemotron_cc` | `knowledge_list` |
+
+These stages inherit from `NDDBaseSyntheticStage`, which auto-builds an NDD config from the prompt fields. You configure the LLM through `model_configs` and `model_providers` instead of an `AsyncOpenAIClient`:
+
+```python
+import os
+
+import data_designer.config as dd
+
+from nemo_curator.stages.synthetic.nemotron_cc.nemo_data_designer.nemotron_cc import DiverseQAStage
+
+model_config = dd.ModelConfig(
+    alias="meta/llama-3.3-70b-instruct",
+    model="meta/llama-3.3-70b-instruct",
+    provider="nvidia",
+    inference_parameters=dd.ChatCompletionInferenceParams(
+        temperature=0.5, top_p=0.9, max_tokens=1600,
+    ),
+)
+
+model_provider = dd.ModelProvider(
+    name="nvidia",
+    endpoint="https://integrate.api.nvidia.com/v1",
+    provider_type="openai",
+    api_key=os.environ["NVIDIA_API_KEY"],
+)
+
+stage = DiverseQAStage(
+    input_field="text",
+    output_field="diverse_qa",
+    model_alias="meta/llama-3.3-70b-instruct",
+    model_configs=[model_config],
+    model_providers=[model_provider],
+)
+```
+
+## YAML Configuration
+
+Instead of building configs in Python, you can define the entire NDD configuration in a YAML file and pass it to `DataDesignerStage`:
+
+```python
+stage = DataDesignerStage(data_designer_config_file="config.yaml")
+```
+
+This is useful for reproducible pipelines where the generation config is versioned alongside data artifacts.
+
+---
+
+## Next Steps
+
+- [Inference Server](/curate-text/synthetic/inference-server): Co-locate model serving with your pipeline
+- [Nemotron-CC Pipelines](/curate-text/synthetic/nemotron-cc): Advanced text transformation tasks
+- [Synthetic Data Generation](/curate-text/synthetic): Overview of all SDG capabilities