Skip to content
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
19 commits
Select commit Hold shift + click to select a range
5bfc9a9
refactor(config): centralize sparse config patch validation
binaryaaron Jun 24, 2026
b02a025
fix: avoid deprecated-field warnings in parameter lookup
binaryaaron Jun 24, 2026
d7d88d9
fix: reject ambiguous config parameter lookups
binaryaaron Jun 24, 2026
499e84d
fix: make from_params overrides order independent
binaryaaron Jun 24, 2026
56bd01a
chore: type from_params kwargs
binaryaaron Jun 24, 2026
262bfd3
fix(sdk): preserve sparse typed config overrides
binaryaaron Jun 25, 2026
93a4276
fix(config): preserve structured generation aliases
binaryaaron Jun 26, 2026
b02f402
fix(config): address review findings
binaryaaron Jun 29, 2026
22f4786
refactor(config): centralize parameter path resolution
binaryaaron Jun 30, 2026
76b1f29
refactor(config): compile schema-aware config patches
binaryaaron Jun 30, 2026
4ecb92e
refactor(config): centralize model config normalization
binaryaaron Jun 30, 2026
0a72e0c
docs(config): clarify patch semantics
binaryaaron Jun 30, 2026
b725fef
fix(config): address review feedback
binaryaaron Jun 30, 2026
4d3aef5
fix(config): mark parameter path string override
binaryaaron Jun 30, 2026
596c793
refactor(config): clarify patch ownership
binaryaaron Jul 1, 2026
0cd8b6a
fix(config): make parameter resolution exhaustive
binaryaaron Jul 1, 2026
e250c5e
refactor(config): centralize parameter-path separator and dedupe reso…
binaryaaron Jul 2, 2026
b4e0d80
refactor(config): unify parameter-tree walk and use match for dispatch
binaryaaron Jul 2, 2026
c427ecc
refactor(config): dedupe patch helpers and tidy control flow
binaryaaron Jul 2, 2026
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
16 changes: 5 additions & 11 deletions STYLE_GUIDE.md
Original file line number Diff line number Diff line change
Expand Up @@ -443,23 +443,17 @@ def teardown(self) -> None:
Tier 2 -- moderate. Summary + `Args:` / `Returns:` / `Raises:` blocks:

```python
def _resolve_config(self, values: ParamDict | NSSParameters | None, cls: type[ParamT], **kwargs) -> ParamT:
"""Resolve configuration from various input types.

Merges caller-supplied overrides on top of a base config. Accepts Pydantic models
(copied with updates), plain dicts (validated then updated), or None (built from
overrides alone).
def load_config(path: Path) -> SafeSynthesizerParameters:
"""Load and validate a Safe Synthesizer YAML configuration.

Args:
values: Base configuration -- a Pydantic model, a dict, or None.
cls: The Pydantic model class to validate against.
**kwargs: Field-level overrides applied on top of the base.
path: YAML configuration path.

Returns:
An instance of `cls` with all overrides applied.
The validated pipeline configuration.

Raises:
TypeError: If `values` is not a BaseModel, dict, or None.
FileNotFoundError: If ``path`` does not exist.
"""
```

Expand Down
72 changes: 56 additions & 16 deletions docs/user-guide/configuration.md
Original file line number Diff line number Diff line change
Expand Up @@ -58,38 +58,78 @@ for more detail on combining config files with runtime overrides.

### Python Parameter Construction

The Python SDK accepts both fully nested config objects and compatibility
shortcuts for fields on top-level parameter sections:
`SafeSynthesizerParameters.from_params()` accepts four forms of keyword name:

- top-level fields such as `generation` or `replace_pii`;
- canonical dotted paths such as `generation.num_records`;
- bare nested names such as `num_records`, when that name is unique in the
configuration schema; and
- legacy structured-generation aliases, retained for compatibility.

Python syntax requires dotted names to be passed through `**` expansion:

```python
from nemo_safe_synthesizer.config import SafeSynthesizerParameters

config = SafeSynthesizerParameters.from_params(
num_records=2000, # generation.num_records
dp_enabled=True, # privacy.dp_enabled
structured_generation={"enabled": True}, # generation.structured_generation.enabled
generation={"temperature": 0.8}, # top-level section
num_records=2000, # unique bare leaf
**{"generation.structured_generation.enabled": True}, # dotted path
)
```

Flat keyword arguments are matched by field name against the top-level parameter
sections. Use the nested shape for fields inside nested subobjects, especially
when a generic field name could appear in multiple places:
An ambiguous bare name raises an error and lists the accepted dotted paths. For
example, `enabled` appears in more than one section, so specify the intended
path:

```python
# Preferred: unambiguous nested form.
SafeSynthesizerParameters.from_params(
structured_generation={"enabled": True},
**{"generation.structured_generation.enabled": True}
)
```

# Also valid: fully nested generation section.
SafeSynthesizerParameters.from_params(
generation={"structured_generation": {"enabled": True}},
)
Legacy aliases such as `use_structured_generation` and
`structured_generation_backend` remain accepted. New code should use the
canonical nested shape or dotted path.

### Sparse Sources and Explicit Values

Absence, explicit `None`, and an explicit value equal to the model default are
different inputs. An absent field inherits the next lower-precedence source or
its model default. `None` is applied when the field accepts it, and an explicitly
supplied default value still counts as an override.

# Avoid: this configures evaluation.enabled, not structured generation.
SafeSynthesizerParameters.from_params(enabled=True)
SDK section methods accept a sparse model or mapping as their source. Keyword
arguments have higher precedence than that source, while omitted source fields
retain the current lower-precedence configuration values:

```python
synthesizer.with_generate({"temperature": 0.8}, num_records=2000)
```

A mapping is a branch only when its schema field is another Pydantic model.
Mapping-valued leaf fields, such as free-form dictionaries, are replaced as one
atomic value rather than recursively merged.

Persistence with `exclude_unset=True` follows Pydantic's explicit-field
metadata. Sparse model sources also inspect nested explicit fields recursively,
so an in-place mutation such as
`source.validation.group_by_fix_unordered_records = True` is captured when that
model is used as a patch input. Unrelated defaults remain implicit.

Raw mapping sources retain the established behavior of ignoring unknown extra
keys. After a name has been resolved to a canonical path, however, that path is
strict: unknown paths and paths that descend through an atomic leaf raise an
error.

### Resume-Time Overrides

When generation resumes from a saved training run, runtime configuration may
override only `generation`, `evaluation`, and `emit_telemetry`. Telemetry is
overridden only when the runtime input explicitly sets it. Saved `training`,
`data`, `privacy`, PII replacement, time-series, and preflight settings remain
unchanged.

---

## Training
Expand Down
19 changes: 9 additions & 10 deletions src/nemo_safe_synthesizer/cli/utils.py
Original file line number Diff line number Diff line change
Expand Up @@ -24,6 +24,7 @@
from pydantic import ValidationError

from ..config import SafeSynthesizerParameters
from ..config.parameters import ConfigPatch
from ..defaults import DEFAULT_ARTIFACTS_PATH
from ..observability import configure_logging_from_workdir, get_logger, initialize_observability
from ..utils import merge_dicts
Expand Down Expand Up @@ -431,26 +432,24 @@ def _initialize_logging_for_cli_from_settings(
return run_logger


def merge_overrides(config_path: str | Path | None, overrides: dict) -> SafeSynthesizerParameters:
"""Merge overrides into a SafeSynthesizerParameters object.
def merge_overrides(config_path: str | Path | None, overrides: ConfigPatch) -> SafeSynthesizerParameters:
"""Apply schema-aware overrides to a ``SafeSynthesizerParameters`` object.

If config_path is None, use the overrides to create a new SafeSynthesizerParameters object.
Otherwise, merge the overrides into the config file.
If ``config_path`` is ``None``, validate the sparse overrides as a new
configuration. Otherwise, apply them on top of the loaded YAML config.

Args:
config_path: Path to config file (YAML)
overrides: Dictionary of override values
overrides: Sparse nested override values.

Returns:
Merged SafeSynthesizerParameters
Validated parameters with the overrides applied.
"""
try:
if config_path is None:
my_config = SafeSynthesizerParameters.model_validate(overrides)
my_config = SafeSynthesizerParameters.from_config_patch(overrides)
else:
file_config = SafeSynthesizerParameters.from_yaml(config_path).model_dump(exclude_unset=True)
params = merge_dicts(file_config, overrides)
my_config = SafeSynthesizerParameters.model_validate(params)
my_config = SafeSynthesizerParameters.from_yaml(config_path).with_config_patch(overrides)
except ValidationError as e:
click.echo(f"{config_path} is invalid:\n{e}")
sys.exit(1)
Expand Down
10 changes: 4 additions & 6 deletions src/nemo_safe_synthesizer/config/autoconfig.py
Original file line number Diff line number Diff line change
Expand Up @@ -20,8 +20,7 @@
from ..defaults import DEFAULT_MAX_SEQ_LENGTH, MAX_ROPE_SCALING_FACTOR
from ..llm.metadata import ModelMetadata
from ..observability import get_logger
from ..utils import merge_dicts
from .parameters import SafeSynthesizerParameters
from .parameters import ConfigPatch, SafeSynthesizerParameters
from .types import AUTO_STR

if TYPE_CHECKING:
Expand Down Expand Up @@ -304,14 +303,13 @@ def _build_updated_params(
Returns:
The validated SafeSynthesizerParameters.
"""
new_params = {
new_params: ConfigPatch = {
"training": training_params,
"data": data_params,
"privacy": privacy_params,
}
updated_params = merge_dicts(self._config.model_dump(exclude_unset=True), new_params)
logger.debug(f"params to update: {updated_params}")
my_config = SafeSynthesizerParameters.model_validate(updated_params)
logger.debug(f"params to update: {new_params}")
my_config = self._config.with_config_patch(new_params)
logger.debug(f"auto-updated config: {my_config.model_dump(exclude_unset=True)}")
return my_config

Expand Down
42 changes: 9 additions & 33 deletions src/nemo_safe_synthesizer/config/generate.py
Original file line number Diff line number Diff line change
Expand Up @@ -4,6 +4,7 @@
from __future__ import annotations

import warnings
from collections.abc import Mapping
from typing import Annotated, Any, ClassVar, Literal, Self

from pydantic import (
Expand All @@ -12,6 +13,7 @@
model_validator,
)

from ..configurator.parameter_paths import ParameterSchema
from ..configurator.parameters import (
Parameters,
)
Expand Down Expand Up @@ -270,45 +272,19 @@ class GenerateParameters(Parameters, BaseModel):
),
] = "auto"

_STRUCTURED_GENERATION_LEGACY_FIELDS: ClassVar[dict[str, str]] = {
"use_structured_generation": "enabled",
"structured_generation_backend": "backend",
"structured_generation_schema_method": "schema_method",
"structured_generation_use_single_sequence": "use_single_sequence",
parameter_aliases: ClassVar[Mapping[str, str]] = {
"use_structured_generation": "structured_generation.enabled",
"structured_generation_backend": "structured_generation.backend",
"structured_generation_schema_method": "structured_generation.schema_method",
"structured_generation_use_single_sequence": "structured_generation.use_single_sequence",
}

@model_validator(mode="before")
@classmethod
def _migrate_legacy_structured_generation_fields(cls, data: Any) -> Any:
if not isinstance(data, dict):
if not isinstance(data, Mapping):
return data

values = dict(data)
legacy = {
new_name: values.pop(old_name)
for old_name, new_name in cls._STRUCTURED_GENERATION_LEGACY_FIELDS.items()
if old_name in values
}
if not legacy:
return values

structured_generation = values.get("structured_generation")
match structured_generation:
case StructuredGenerationParameters() as params:
structured_values = params.model_dump()
case BaseModel() as model:
structured_values = model.model_dump()
case dict() as mapping:
structured_values = dict(mapping)
case None:
structured_values = {}
case _:
return values

# Legacy flat keys are treated as explicit overrides for migration
# paths such as ``from_params(generation={...}, structured_generation_backend=...)``.
values["structured_generation"] = structured_values | legacy
return values
return ParameterSchema.from_model(cls).normalize_aliases(data)

@property
def use_structured_generation(self) -> bool:
Expand Down
Loading
Loading