feat(config): migration tool for legacy dataset identifiers → source: (M6c) - #1639
Closed
wprazuch wants to merge 30 commits into
Closed
feat(config): migration tool for legacy dataset identifiers → source: (M6c)#1639wprazuch wants to merge 30 commits into
source: (M6c)#1639wprazuch wants to merge 30 commits into
Conversation
Signed-off-by: Marta Stepniewska-Dziubinska <martas@nvidia.com>
Signed-off-by: Marta Stepniewska-Dziubinska <martas@nvidia.com>
Signed-off-by: Marta Stepniewska-Dziubinska <martas@nvidia.com>
Signed-off-by: Marta Stepniewska-Dziubinska <martas@nvidia.com>
Signed-off-by: Marta Stepniewska-Dziubinska <martas@nvidia.com>
Signed-off-by: Marta Stepniewska-Dziubinska <martas@nvidia.com>
Signed-off-by: Marta Stepniewska-Dziubinska <martas@nvidia.com>
Signed-off-by: Marta Stepniewska-Dziubinska <martas@nvidia.com>
Signed-off-by: Marta Stepniewska-Dziubinska <martas@nvidia.com>
Signed-off-by: Marta Stepniewska-Dziubinska <martas@nvidia.com>
Signed-off-by: Marta Stepniewska-Dziubinska <martas@nvidia.com>
…nt deprecation note Signed-off-by: Marta Stepniewska-Dziubinska <martas@nvidia.com>
Signed-off-by: Marta Stepniewska-Dziubinska <martas@nvidia.com>
Signed-off-by: Marta Stepniewska-Dziubinska <martas@nvidia.com>
… to stderr) Signed-off-by: Marta Stepniewska-Dziubinska <martas@nvidia.com>
Signed-off-by: Marta Stepniewska-Dziubinska <martas@nvidia.com>
Signed-off-by: Marta Stepniewska-Dziubinska <martas@nvidia.com>
Signed-off-by: Marta Stepniewska-Dziubinska <martas@nvidia.com>
Signed-off-by: Marta Stepniewska-Dziubinska <martas@nvidia.com>
…a single --model flag Signed-off-by: Marta Stepniewska-Dziubinska <martas@nvidia.com>
Signed-off-by: Marta Stepniewska-Dziubinska <martas@nvidia.com>
wprazuch
force-pushed
the
wprazuch/dataset-source
branch
from
June 22, 2026 13:45
db8b03c to
695207e
Compare
wprazuch
force-pushed
the
wprazuch/dataset-source-migration
branch
from
June 22, 2026 13:45
85e21c8 to
2bc5571
Compare
Signed-off-by: Marta Stepniewska-Dziubinska <martas@nvidia.com>
Signed-off-by: Marta Stepniewska-Dziubinska <martas@nvidia.com>
Signed-off-by: Marta Stepniewska-Dziubinska <martas@nvidia.com>
…or benchmark preparation Signed-off-by: Marta Stepniewska-Dziubinska <martas@nvidia.com>
Signed-off-by: Marta Stepniewska-Dziubinska <martas@nvidia.com>
wprazuch
force-pushed
the
wprazuch/dataset-source
branch
from
June 24, 2026 07:01
695207e to
b024b01
Compare
wprazuch
force-pushed
the
wprazuch/dataset-source-migration
branch
from
June 24, 2026 07:05
2bc5571 to
aebb3fd
Compare
wprazuch
force-pushed
the
wprazuch/dataset-source
branch
2 times, most recently
from
June 24, 2026 07:18
cdcbd27 to
2905987
Compare
wprazuch
force-pushed
the
wprazuch/dataset-source-migration
branch
from
June 24, 2026 07:20
aebb3fd to
120a9a3
Compare
Replace the parallel gitlab_identifier / huggingface_identifier fields on DatasetConfig with a single self-describing source: block (type selects the backend). Legacy fields keep working: a legacy identifier is mirrored into source (with a DeprecationWarning), and a source: is back-filled into the matching legacy field so existing consumers that read the *_identifier fields are unaffected. Specifying both is rejected. Addresses FEP-1025 (reduce configuration friction, epic #1205). Signed-off-by: Wojciech Prazuch <wprazuch@nvidia.com>
wprazuch
force-pushed
the
wprazuch/dataset-source
branch
from
June 24, 2026 07:21
2905987 to
6ca5c48
Compare
Add nemo_gym.dataset_source_migration: a line-based YAML rewriter that converts the deprecated gitlab_identifier/huggingface_identifier blocks to the unified source: block. It only renames the mapping key and injects the type discriminator, so comments, ordering, and formatting are preserved byte-for-byte and diffs stay minimal. Run via 'python -m nemo_gym.dataset_source_migration <paths> [--dry-run]'. Completes the migration-script deliverable of M6c (epic #1205). Dry-run currently reports 109 blocks across 59 in-repo configs. Signed-off-by: Wojciech Prazuch <wprazuch@nvidia.com>
wprazuch
force-pushed
the
wprazuch/dataset-source-migration
branch
from
June 24, 2026 07:22
120a9a3 to
a8f9ac9
Compare
wprazuch
marked this pull request as draft
June 24, 2026 07:59
wprazuch
force-pushed
the
wprazuch/dataset-source
branch
from
June 25, 2026 11:31
35864ee to
f96315d
Compare
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
What
Adds
nemo_gym.dataset_source_migration— a tool that rewrites the deprecatedgitlab_identifier:/huggingface_identifier:blocks to the unifiedsource:blockintroduced in #1637. Completes the migration-script deliverable of M6c.
How
A line-based transform, not a YAML round-trip: it matches a bare block-style key
line and rewrites only that line —
gitlab_identifier:→source:+type: gitlab.The already-nested fields (
dataset_name/version/artifact_fpath, orrepo_id) sit atexactly the indent
source:'s children need, so they're left untouched. Result:Flow-style mappings and key lines with trailing comments are deliberately not matched
(rare here; rewriting them blindly risks corruption) — they're left as-is and reported as
0 migrations.
Why
python -m, not anng_*console scriptThe
ng_*scripts run under the Hydra harness (andtest_pyproject_scriptscalls eachwith
+h=trueexpectingSystemExit), which a path-taking argparse migration tool doesn'tfit. A
python -mentrypoint is the right shape and keeps this off the unified-CLI surface.Validation
flow-style/trailing-comment guards, dry-run vs in-place, and a round-trip test proving the
migrated YAML validates as a
DatasetConfigwith noDeprecationWarning.a follow-up sweep PR (kept separate to avoid a 59-file diff colliding with in-flight branches).
Part of the configuration-friction epic (#1205), milestone M6c. Targets
wprazuch/dataset-sourcesince it produces thesource:schema from that PR.