Skip to content

feat(config): migration tool for legacy dataset identifiers → source: (M6c) - #1639

Closed
wprazuch wants to merge 30 commits into
mainfrom
wprazuch/dataset-source-migration
Closed

feat(config): migration tool for legacy dataset identifiers → source: (M6c)#1639
wprazuch wants to merge 30 commits into
mainfrom
wprazuch/dataset-source-migration

Conversation

@wprazuch

Copy link
Copy Markdown
Contributor

What

Adds nemo_gym.dataset_source_migration — a tool that rewrites the deprecated
gitlab_identifier: / huggingface_identifier: blocks to the unified source: block
introduced in #1637. Completes the migration-script deliverable of M6c.

# preview (no writes; exits non-zero if anything would change)
python -m nemo_gym.dataset_source_migration --dry-run resources_servers responses_api_agents

# rewrite in place
python -m nemo_gym.dataset_source_migration resources_servers responses_api_agents

How

A line-based transform, not a YAML round-trip: it matches a bare block-style key
line and rewrites only that line — gitlab_identifier:source: + type: gitlab.
The already-nested fields (dataset_name/version/artifact_fpath, or repo_id) sit at
exactly the indent source:'s children need, so they're left untouched. Result:

  • comments, key ordering, and formatting are preserved byte-for-byte;
  • diffs are minimal (two changed lines per block);
  • no new dependency (ruamel isn't available in the env).

Flow-style mappings and key lines with trailing comments are deliberately not matched
(rare here; rewriting them blindly risks corruption) — they're left as-is and reported as
0 migrations.

Why python -m, not an ng_* console script

The ng_* scripts run under the Hydra harness (and test_pyproject_scripts calls each
with +h=true expecting SystemExit), which a path-taking argparse migration tool doesn't
fit. A python -m entrypoint is the right shape and keeps this off the unified-CLI surface.

Validation

  • Unit tests (100% coverage of the module): both backends, multi-block files, no-op files,
    flow-style/trailing-comment guards, dry-run vs in-place, and a round-trip test proving the
    migrated YAML validates as a DatasetConfig with no DeprecationWarning.
  • Dry-run on the live tree reports 109 blocks across 59 configs — those can be migrated in
    a follow-up sweep PR (kept separate to avoid a 59-file diff colliding with in-flight branches).

Part of the configuration-friction epic (#1205), milestone M6c. Targets
wprazuch/dataset-source since it produces the source: schema from that PR.

marta-sd added 13 commits June 12, 2026 13:40
Signed-off-by: Marta Stepniewska-Dziubinska <martas@nvidia.com>
Signed-off-by: Marta Stepniewska-Dziubinska <martas@nvidia.com>
Signed-off-by: Marta Stepniewska-Dziubinska <martas@nvidia.com>
Signed-off-by: Marta Stepniewska-Dziubinska <martas@nvidia.com>
Signed-off-by: Marta Stepniewska-Dziubinska <martas@nvidia.com>
Signed-off-by: Marta Stepniewska-Dziubinska <martas@nvidia.com>
Signed-off-by: Marta Stepniewska-Dziubinska <martas@nvidia.com>
Signed-off-by: Marta Stepniewska-Dziubinska <martas@nvidia.com>
Signed-off-by: Marta Stepniewska-Dziubinska <martas@nvidia.com>
Signed-off-by: Marta Stepniewska-Dziubinska <martas@nvidia.com>
Signed-off-by: Marta Stepniewska-Dziubinska <martas@nvidia.com>
…nt deprecation note

Signed-off-by: Marta Stepniewska-Dziubinska <martas@nvidia.com>
Signed-off-by: Marta Stepniewska-Dziubinska <martas@nvidia.com>
@copy-pr-bot

copy-pr-bot Bot commented Jun 17, 2026

Copy link
Copy Markdown

This pull request requires additional validation before any workflows can run on NVIDIA's runners.

Pull request vetters can view their responsibilities here.

Contributors can view more details about this message here.

marta-sd added 8 commits June 17, 2026 13:57
Signed-off-by: Marta Stepniewska-Dziubinska <martas@nvidia.com>
… to stderr)

Signed-off-by: Marta Stepniewska-Dziubinska <martas@nvidia.com>
Signed-off-by: Marta Stepniewska-Dziubinska <martas@nvidia.com>
Signed-off-by: Marta Stepniewska-Dziubinska <martas@nvidia.com>
Signed-off-by: Marta Stepniewska-Dziubinska <martas@nvidia.com>
Signed-off-by: Marta Stepniewska-Dziubinska <martas@nvidia.com>
…a single --model flag

Signed-off-by: Marta Stepniewska-Dziubinska <martas@nvidia.com>
Signed-off-by: Marta Stepniewska-Dziubinska <martas@nvidia.com>
@wprazuch
wprazuch force-pushed the wprazuch/dataset-source branch from db8b03c to 695207e Compare June 22, 2026 13:45
@wprazuch
wprazuch requested a review from a team as a code owner June 22, 2026 13:45
@wprazuch
wprazuch force-pushed the wprazuch/dataset-source-migration branch from 85e21c8 to 2bc5571 Compare June 22, 2026 13:45
marta-sd added 5 commits June 23, 2026 13:08
Signed-off-by: Marta Stepniewska-Dziubinska <martas@nvidia.com>
Signed-off-by: Marta Stepniewska-Dziubinska <martas@nvidia.com>
Signed-off-by: Marta Stepniewska-Dziubinska <martas@nvidia.com>
…or benchmark preparation

Signed-off-by: Marta Stepniewska-Dziubinska <martas@nvidia.com>
Signed-off-by: Marta Stepniewska-Dziubinska <martas@nvidia.com>
@wprazuch
wprazuch force-pushed the wprazuch/dataset-source branch from 695207e to b024b01 Compare June 24, 2026 07:01
@wprazuch
wprazuch force-pushed the wprazuch/dataset-source-migration branch from 2bc5571 to aebb3fd Compare June 24, 2026 07:05
@wprazuch
wprazuch force-pushed the wprazuch/dataset-source branch 2 times, most recently from cdcbd27 to 2905987 Compare June 24, 2026 07:18
@wprazuch
wprazuch force-pushed the wprazuch/dataset-source-migration branch from aebb3fd to 120a9a3 Compare June 24, 2026 07:20
Replace the parallel gitlab_identifier / huggingface_identifier fields on
DatasetConfig with a single self-describing source: block (type selects the
backend). Legacy fields keep working: a legacy identifier is mirrored into
source (with a DeprecationWarning), and a source: is back-filled into the
matching legacy field so existing consumers that read the *_identifier fields
are unaffected. Specifying both is rejected.

Addresses FEP-1025 (reduce configuration friction, epic #1205).

Signed-off-by: Wojciech Prazuch <wprazuch@nvidia.com>
@wprazuch
wprazuch force-pushed the wprazuch/dataset-source branch from 2905987 to 6ca5c48 Compare June 24, 2026 07:21
Add nemo_gym.dataset_source_migration: a line-based YAML rewriter that converts
the deprecated gitlab_identifier/huggingface_identifier blocks to the unified
source: block. It only renames the mapping key and injects the type discriminator,
so comments, ordering, and formatting are preserved byte-for-byte and diffs stay
minimal. Run via 'python -m nemo_gym.dataset_source_migration <paths> [--dry-run]'.

Completes the migration-script deliverable of M6c (epic #1205). Dry-run currently
reports 109 blocks across 59 in-repo configs.

Signed-off-by: Wojciech Prazuch <wprazuch@nvidia.com>
@wprazuch
wprazuch force-pushed the wprazuch/dataset-source-migration branch from 120a9a3 to a8f9ac9 Compare June 24, 2026 07:22
@wprazuch
wprazuch marked this pull request as draft June 24, 2026 07:59
@wprazuch
wprazuch force-pushed the wprazuch/dataset-source branch from 35864ee to f96315d Compare June 25, 2026 11:31
Base automatically changed from wprazuch/dataset-source to main June 25, 2026 11:40
@wprazuch wprazuch closed this Jul 7, 2026
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants