Add stable retrieval SDG recipe APIs

[shan-nvidia]

What

• Add stable Python generation APIs with typed GenerationRunConfig, GenerationPipelineConfig, and immutable result metadata.
• Make the CLI delegate generation and preview execution to the same public API.
• Standardize new runs on Data Designer native resume and one exported JSONL artifact.
• Return conversion output paths and counts from run_conversion().
• Align conversion behavior for unambiguous input discovery, corpus-relative source IDs, deterministic splits, checked group matching, and Parquet normalization.
• Apply the canonical Nemotron generation and embedding defaults and normalize custom-provider conflict handling.
• Validate the complete query/reasoning distributions against num_pairs and use seven pairs by default.
• Redact provider credentials from serialized run configuration and CLI output.

Why

The Nemotron embedding recipe currently consumes vendored retrieval SDG source and a separate conversion script. Replacing those copies with the released plugin requires stable package-level generation and conversion contracts, predictable artifacts, and reviewed conversion semantics. This change establishes that API and behavioral baseline before the recipe installs a released wheel.

Usage

from pathlib import Path

from data_designer_retrieval_sdg import (
    DocumentChunkerSeedSource,
    GenerationPipelineConfig,
    GenerationRunConfig,
    run_conversion,
    run_generation,
)

result = run_generation(
    GenerationRunConfig(
        seed_source=DocumentChunkerSeedSource(path="./docs"),
        output_dir=Path("./generated"),
        dataset_name="retrieval_run",
        pipeline=GenerationPipelineConfig(num_pairs=7),
    )
)
conversion = run_conversion(
    input_path=str(result.output_path),
    corpus_id="example",
)

How

The new public runner owns Data Designer setup, native resume, export, and stable result metadata. Explicit Pydantic-backed run and pipeline models reject unknown fields and replace the previous untyped pipeline_kwargs boundary. Existing low-level pipeline defaults now share the same constants as the stable runner and CLI.

Conversion remains backward-compatible with JSON, JSONL, Parquet, and legacy batch inputs, while ambiguous mixed-format directories fail with an actionable error. Identifier and seeded-split fixes prevent basename collisions and process-dependent split membership.

File-based config loading, default YAML files, resolved run snapshots, provenance, and resume fingerprints are intentionally reserved for a following PR. This PR does not add a plugin llama profile; recipe-specific profile mapping remains owned by Nemotron.

Validation

• make test-plugin PLUGIN=data-designer-retrieval-sdg (106 passed)
• make lint
• make validate
• make check
• git diff --check

[oliverholworthy]

Edge case: I wonder if it makes sense to use result.count_records() rather than the requested num_records? Is there a scenario where Data Designer would produce fewer rows than num_records. If it happens, the GenerationResult could disagree with the exported JSONL?

we could record both, with num_records as results.count_records() and add an extra field for the requested num_records?

[shan-nvidia]

Generation reports actual and requested record counts separately.

[oliverholworthy]

do we need an upper bound le here, or is it valid to have any complexity of 1 or above?

[shan-nvidia]

Added le=5 here

[oliverholworthy]

Might be worth adding a validator for query_counts and reasoning_counts to make sure the string keys provided match the expected set?

[shan-nvidia]

Added validation for query/reasoning keys.

[oliverholworthy]

Is there an assumption or requirement that num_pairs matches the sum of query_counts? It looks like it might based on the prompts. if so, maybe worth adding a validator to check this

[shan-nvidia]

Default num_pairs is now 7. and add validation of the sum.

[oliverholworthy]

for slightly ealier failure it could be worth moving this before data_designer.create so we fail early if the output directory fails to be created for some reason (e.g. invalid, or inaccessible for some reason). we'd learn that before the run starts

[shan-nvidia]

Output directory validation happens before generation starts.

[oliverholworthy]

The overall change looks like great refactor. I added a few minor suggestions that might improve config validation.