feat(cli): judge / auxiliary-model resources-server scaffold template (M6g) - #1644
Draft
wprazuch wants to merge 30 commits into
Draft
feat(cli): judge / auxiliary-model resources-server scaffold template (M6g)#1644wprazuch wants to merge 30 commits into
wprazuch wants to merge 30 commits into
Conversation
Signed-off-by: Marta Stepniewska-Dziubinska <martas@nvidia.com>
Signed-off-by: Marta Stepniewska-Dziubinska <martas@nvidia.com>
Signed-off-by: Marta Stepniewska-Dziubinska <martas@nvidia.com>
Signed-off-by: Marta Stepniewska-Dziubinska <martas@nvidia.com>
Signed-off-by: Marta Stepniewska-Dziubinska <martas@nvidia.com>
Signed-off-by: Marta Stepniewska-Dziubinska <martas@nvidia.com>
Signed-off-by: Marta Stepniewska-Dziubinska <martas@nvidia.com>
Signed-off-by: Marta Stepniewska-Dziubinska <martas@nvidia.com>
Signed-off-by: Marta Stepniewska-Dziubinska <martas@nvidia.com>
Signed-off-by: Marta Stepniewska-Dziubinska <martas@nvidia.com>
Signed-off-by: Marta Stepniewska-Dziubinska <martas@nvidia.com>
…nt deprecation note Signed-off-by: Marta Stepniewska-Dziubinska <martas@nvidia.com>
Signed-off-by: Marta Stepniewska-Dziubinska <martas@nvidia.com>
Signed-off-by: Marta Stepniewska-Dziubinska <martas@nvidia.com>
… to stderr) Signed-off-by: Marta Stepniewska-Dziubinska <martas@nvidia.com>
Signed-off-by: Marta Stepniewska-Dziubinska <martas@nvidia.com>
Signed-off-by: Marta Stepniewska-Dziubinska <martas@nvidia.com>
Signed-off-by: Marta Stepniewska-Dziubinska <martas@nvidia.com>
Signed-off-by: Marta Stepniewska-Dziubinska <martas@nvidia.com>
…a single --model flag Signed-off-by: Marta Stepniewska-Dziubinska <martas@nvidia.com>
Ports #1596 onto the unified gym CLI (#1434). Adds +max_concurrency to ng_test_all: each module still runs in its own isolated subprocess/venv, but up to N run concurrently via a ThreadPoolExecutor, so local ng_test_all and CI both speed up (no matrix sharding). run_command gains an additive capture= param so concurrent module output is collected and printed atomically. Workflow runs on ${{ vars.TEST_RUNNER || 'ubuntu-latest' }} with TEST_CONCURRENCY (default 8). Targets martas/1434. Signed-off-by: Wojciech Prazuch <wprazuch@nvidia.com>
Shared CI fixes for the martas/1434-stacked CLI work: pin uv (0.11.20 drops pinned deps -> 7 servers fail; = #1576) and pull main's graphwalks example_rollouts.jsonl (fixes its data validation). This branch is the base for the ng_validate (#1599) and config-error (#1609) PRs so the fixes live in one place. Drop when martas/1434 rebases on main. Signed-off-by: Wojciech Prazuch <wprazuch@nvidia.com>
Replace the parallel gitlab_identifier / huggingface_identifier fields on DatasetConfig with a single self-describing source: block (type selects the backend). Legacy fields keep working: a legacy identifier is mirrored into source (with a DeprecationWarning), and a source: is back-filled into the matching legacy field so existing consumers that read the *_identifier fields are unaffected. Specifying both is rejected. Addresses FEP-1025 (reduce configuration friction, epic #1205). Signed-off-by: Wojciech Prazuch <wprazuch@nvidia.com>
Annotate the config emitted by ng_init_resources_server with inline comments explaining each non-obvious field (domain, resources_server, policy_model magic name, datasets/source), addressing friction #7 (no inline documentation in generated configs). Also modernize the scaffold to emit the canonical `source:` dataset block instead of the now-deprecated `gitlab_identifier`, so a freshly created server starts on the recommended schema. Addresses M6a (epic #1205). Signed-off-by: Wojciech Prazuch <wprazuch@nvidia.com>
…M6g) Add a reusable LLM-as-judge resources-server template and a --template flag to ng_init_resources_server. `+template=judge` scaffolds a verifier whose verify() calls another model (judge / reward model / subagent) via the server client, wiring judge_model_server + judge_responses_create_params in the generated config, with a bounded judge-call semaphore and a clamped 0..1 reward parser. This is additive: the default 'basic' template is unchanged and the existing LLM-as-judge servers are left untouched (consolidating them is a separate, owner-sensitive change). New template files live under nemo_gym/resources/ and are omitted from coverage (copied verbatim into generated servers; exercised by the bundled test template inside a scaffolded server). Implements M6g (epic #1205, friction #2/#7). Signed-off-by: Wojciech Prazuch <wprazuch@nvidia.com>
Ports #1596 onto the unified gym CLI (#1434). Adds +max_concurrency to ng_test_all: each module still runs in its own isolated subprocess/venv, but up to N run concurrently via a ThreadPoolExecutor, so local ng_test_all and CI both speed up (no matrix sharding). run_command gains an additive capture= param so concurrent module output is collected and printed atomically. Workflow runs on ${{ vars.TEST_RUNNER || 'ubuntu-latest' }} with TEST_CONCURRENCY (default 8). Targets martas/1434. Signed-off-by: Wojciech Prazuch <wprazuch@nvidia.com>
Shared CI fixes for the martas/1434-stacked CLI work: pin uv (0.11.20 drops pinned deps -> 7 servers fail; = #1576) and pull main's graphwalks example_rollouts.jsonl (fixes its data validation). This branch is the base for the ng_validate (#1599) and config-error (#1609) PRs so the fixes live in one place. Drop when martas/1434 rebases on main. Signed-off-by: Wojciech Prazuch <wprazuch@nvidia.com>
Replace the parallel gitlab_identifier / huggingface_identifier fields on DatasetConfig with a single self-describing source: block (type selects the backend). Legacy fields keep working: a legacy identifier is mirrored into source (with a DeprecationWarning), and a source: is back-filled into the matching legacy field so existing consumers that read the *_identifier fields are unaffected. Specifying both is rejected. Addresses FEP-1025 (reduce configuration friction, epic #1205). Signed-off-by: Wojciech Prazuch <wprazuch@nvidia.com>
Annotate the config emitted by ng_init_resources_server with inline comments explaining each non-obvious field (domain, resources_server, policy_model magic name, datasets/source), addressing friction #7 (no inline documentation in generated configs). Also modernize the scaffold to emit the canonical `source:` dataset block instead of the now-deprecated `gitlab_identifier`, so a freshly created server starts on the recommended schema. Addresses M6a (epic #1205). Signed-off-by: Wojciech Prazuch <wprazuch@nvidia.com>
wprazuch
force-pushed
the
wprazuch/judge-scaffold-template
branch
from
June 22, 2026 13:45
266872a to
0b2bec2
Compare
wprazuch
force-pushed
the
wprazuch/init-config-docs
branch
from
June 22, 2026 13:45
5e9d480 to
9bbd7f1
Compare
…wprazuch/judge-scaffold-template # Conflicts: # nemo_gym/cli/env.py
wprazuch
marked this pull request as draft
June 24, 2026 08:03
wprazuch
force-pushed
the
wprazuch/init-config-docs
branch
from
June 24, 2026 08:10
9bbd7f1 to
b6933fb
Compare
Base automatically changed from
wprazuch/init-config-docs
to
wprazuch/dataset-source
June 24, 2026 08:15
wprazuch
force-pushed
the
wprazuch/dataset-source
branch
from
June 25, 2026 11:31
35864ee to
f96315d
Compare
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
What
Adds a reusable LLM-as-judge / auxiliary-model resources-server template and a
--templateoption to
ng_init_resources_server. Today the four LLM-as-judge servers each re-implement thesame "call another model from
verify()" wiring by hand; this gives new servers one sharedstarting point.
The judge scaffold generates:
judge_model_server(referenced by name, likepolicy_model) andjudge_responses_create_paramsinto the resources server;app.pywhoseverify()calls the auxiliary model viaserver_client.post(... /v1/responses),with a bounded concurrency semaphore and a clamped
0..1reward parser (_parse_reward,meant to be swapped for label/JSON parsing);
test_app.pycovering construction and reward parsing.Scope
Additive only. The default
basictemplate is unchanged, and the existing judge servers(
equivalence_llm_judge,arena_judge,aalcr, …) are not refactored onto this base —consolidating them is a separate, owner-sensitive change. This PR just makes the shared pattern
available to new servers (and to a future
ng init envwizard / M6h).Tests
test_init_resources_server_judge_template— asserts+template=judgescaffolds the judgeconfig (
judge_model_server/judge_responses_create_params) and the judge app + test templates.test_app.py(shipped asjudge_resources_server_test_template.py) covers theapp logic (construction,
_parse_rewardnumber-extraction and zero fallback) inside a generatedserver.
omit— they're copied verbatim into generatedservers and run there, not imported in-process (same class as the already-omitted
environments/*/prepare.py).Part of the configuration-friction epic (#1205), friction #2/#7. Targets
wprazuch/init-config-docssince it extends the sameinit_resources_serverflow (and reuses itsinline-docs +
source:config style).