Skip to content

feat(cli): judge / auxiliary-model resources-server scaffold template (M6g) - #1644

Draft
wprazuch wants to merge 30 commits into
mainfrom
wprazuch/judge-scaffold-template
Draft

feat(cli): judge / auxiliary-model resources-server scaffold template (M6g)#1644
wprazuch wants to merge 30 commits into
mainfrom
wprazuch/judge-scaffold-template

Conversation

@wprazuch

Copy link
Copy Markdown
Contributor

What

Adds a reusable LLM-as-judge / auxiliary-model resources-server template and a --template
option to ng_init_resources_server. Today the four LLM-as-judge servers each re-implement the
same "call another model from verify()" wiring by hand; this gives new servers one shared
starting point.

# default basic verifier (unchanged)
ng_init_resources_server +entrypoint=resources_servers/my_server

# verifier that calls a judge / reward model / subagent
ng_init_resources_server +entrypoint=resources_servers/my_judge +template=judge

The judge scaffold generates:

  • a config block wiring judge_model_server (referenced by name, like policy_model) and
    judge_responses_create_params into the resources server;
  • an app.py whose verify() calls the auxiliary model via server_client.post(... /v1/responses),
    with a bounded concurrency semaphore and a clamped 0..1 reward parser (_parse_reward,
    meant to be swapped for label/JSON parsing);
  • a matching test_app.py covering construction and reward parsing.

Scope

Additive only. The default basic template is unchanged, and the existing judge servers
(equivalence_llm_judge, arena_judge, aalcr, …) are not refactored onto this base —
consolidating them is a separate, owner-sensitive change. This PR just makes the shared pattern
available to new servers (and to a future ng init env wizard / M6h).

Tests

  • test_init_resources_server_judge_template — asserts +template=judge scaffolds the judge
    config (judge_model_server/judge_responses_create_params) and the judge app + test templates.
  • The scaffolded test_app.py (shipped as judge_resources_server_test_template.py) covers the
    app logic (construction, _parse_reward number-extraction and zero fallback) inside a generated
    server.
  • The two template files are added to coverage omit — they're copied verbatim into generated
    servers and run there, not imported in-process (same class as the already-omitted
    environments/*/prepare.py).

Part of the configuration-friction epic (#1205), friction #2/#7. Targets
wprazuch/init-config-docs since it extends the same init_resources_server flow (and reuses its
inline-docs + source: config style).

marta-sd added 20 commits June 12, 2026 13:40
Signed-off-by: Marta Stepniewska-Dziubinska <martas@nvidia.com>
Signed-off-by: Marta Stepniewska-Dziubinska <martas@nvidia.com>
Signed-off-by: Marta Stepniewska-Dziubinska <martas@nvidia.com>
Signed-off-by: Marta Stepniewska-Dziubinska <martas@nvidia.com>
Signed-off-by: Marta Stepniewska-Dziubinska <martas@nvidia.com>
Signed-off-by: Marta Stepniewska-Dziubinska <martas@nvidia.com>
Signed-off-by: Marta Stepniewska-Dziubinska <martas@nvidia.com>
Signed-off-by: Marta Stepniewska-Dziubinska <martas@nvidia.com>
Signed-off-by: Marta Stepniewska-Dziubinska <martas@nvidia.com>
Signed-off-by: Marta Stepniewska-Dziubinska <martas@nvidia.com>
Signed-off-by: Marta Stepniewska-Dziubinska <martas@nvidia.com>
…nt deprecation note

Signed-off-by: Marta Stepniewska-Dziubinska <martas@nvidia.com>
Signed-off-by: Marta Stepniewska-Dziubinska <martas@nvidia.com>
Signed-off-by: Marta Stepniewska-Dziubinska <martas@nvidia.com>
… to stderr)

Signed-off-by: Marta Stepniewska-Dziubinska <martas@nvidia.com>
Signed-off-by: Marta Stepniewska-Dziubinska <martas@nvidia.com>
Signed-off-by: Marta Stepniewska-Dziubinska <martas@nvidia.com>
Signed-off-by: Marta Stepniewska-Dziubinska <martas@nvidia.com>
Signed-off-by: Marta Stepniewska-Dziubinska <martas@nvidia.com>
…a single --model flag

Signed-off-by: Marta Stepniewska-Dziubinska <martas@nvidia.com>
@wprazuch
wprazuch requested a review from a team as a code owner June 22, 2026 09:20
@copy-pr-bot

copy-pr-bot Bot commented Jun 22, 2026

Copy link
Copy Markdown

This pull request requires additional validation before any workflows can run on NVIDIA's runners.

Pull request vetters can view their responsibilities here.

Contributors can view more details about this message here.

wprazuch added 8 commits June 22, 2026 15:45
Ports #1596 onto the unified gym CLI (#1434). Adds +max_concurrency to
ng_test_all: each module still runs in its own isolated subprocess/venv, but
up to N run concurrently via a ThreadPoolExecutor, so local ng_test_all and CI
both speed up (no matrix sharding). run_command gains an additive capture=
param so concurrent module output is collected and printed atomically.

Workflow runs on ${{ vars.TEST_RUNNER || 'ubuntu-latest' }} with
TEST_CONCURRENCY (default 8). Targets martas/1434.

Signed-off-by: Wojciech Prazuch <wprazuch@nvidia.com>
Shared CI fixes for the martas/1434-stacked CLI work: pin uv (0.11.20 drops
pinned deps -> 7 servers fail; = #1576) and pull main's graphwalks
example_rollouts.jsonl (fixes its data validation). This branch is the base
for the ng_validate (#1599) and config-error (#1609) PRs so the fixes live in
one place. Drop when martas/1434 rebases on main.

Signed-off-by: Wojciech Prazuch <wprazuch@nvidia.com>
Replace the parallel gitlab_identifier / huggingface_identifier fields on
DatasetConfig with a single self-describing source: block (type selects the
backend). Legacy fields keep working: a legacy identifier is mirrored into
source (with a DeprecationWarning), and a source: is back-filled into the
matching legacy field so existing consumers that read the *_identifier fields
are unaffected. Specifying both is rejected.

Addresses FEP-1025 (reduce configuration friction, epic #1205).

Signed-off-by: Wojciech Prazuch <wprazuch@nvidia.com>
Annotate the config emitted by ng_init_resources_server with inline comments
explaining each non-obvious field (domain, resources_server, policy_model magic
name, datasets/source), addressing friction #7 (no inline documentation in
generated configs). Also modernize the scaffold to emit the canonical `source:`
dataset block instead of the now-deprecated `gitlab_identifier`, so a freshly
created server starts on the recommended schema.

Addresses M6a (epic #1205).

Signed-off-by: Wojciech Prazuch <wprazuch@nvidia.com>
…M6g)

Add a reusable LLM-as-judge resources-server template and a --template flag to
ng_init_resources_server. `+template=judge` scaffolds a verifier whose verify()
calls another model (judge / reward model / subagent) via the server client,
wiring judge_model_server + judge_responses_create_params in the generated config,
with a bounded judge-call semaphore and a clamped 0..1 reward parser.

This is additive: the default 'basic' template is unchanged and the existing
LLM-as-judge servers are left untouched (consolidating them is a separate,
owner-sensitive change). New template files live under nemo_gym/resources/ and
are omitted from coverage (copied verbatim into generated servers; exercised by
the bundled test template inside a scaffolded server).

Implements M6g (epic #1205, friction #2/#7).

Signed-off-by: Wojciech Prazuch <wprazuch@nvidia.com>
Ports #1596 onto the unified gym CLI (#1434). Adds +max_concurrency to
ng_test_all: each module still runs in its own isolated subprocess/venv, but
up to N run concurrently via a ThreadPoolExecutor, so local ng_test_all and CI
both speed up (no matrix sharding). run_command gains an additive capture=
param so concurrent module output is collected and printed atomically.

Workflow runs on ${{ vars.TEST_RUNNER || 'ubuntu-latest' }} with
TEST_CONCURRENCY (default 8). Targets martas/1434.

Signed-off-by: Wojciech Prazuch <wprazuch@nvidia.com>
Shared CI fixes for the martas/1434-stacked CLI work: pin uv (0.11.20 drops
pinned deps -> 7 servers fail; = #1576) and pull main's graphwalks
example_rollouts.jsonl (fixes its data validation). This branch is the base
for the ng_validate (#1599) and config-error (#1609) PRs so the fixes live in
one place. Drop when martas/1434 rebases on main.

Signed-off-by: Wojciech Prazuch <wprazuch@nvidia.com>
Replace the parallel gitlab_identifier / huggingface_identifier fields on
DatasetConfig with a single self-describing source: block (type selects the
backend). Legacy fields keep working: a legacy identifier is mirrored into
source (with a DeprecationWarning), and a source: is back-filled into the
matching legacy field so existing consumers that read the *_identifier fields
are unaffected. Specifying both is rejected.

Addresses FEP-1025 (reduce configuration friction, epic #1205).

Signed-off-by: Wojciech Prazuch <wprazuch@nvidia.com>
Annotate the config emitted by ng_init_resources_server with inline comments
explaining each non-obvious field (domain, resources_server, policy_model magic
name, datasets/source), addressing friction #7 (no inline documentation in
generated configs). Also modernize the scaffold to emit the canonical `source:`
dataset block instead of the now-deprecated `gitlab_identifier`, so a freshly
created server starts on the recommended schema.

Addresses M6a (epic #1205).

Signed-off-by: Wojciech Prazuch <wprazuch@nvidia.com>
@wprazuch
wprazuch force-pushed the wprazuch/judge-scaffold-template branch from 266872a to 0b2bec2 Compare June 22, 2026 13:45
@wprazuch
wprazuch force-pushed the wprazuch/init-config-docs branch from 5e9d480 to 9bbd7f1 Compare June 22, 2026 13:45
…wprazuch/judge-scaffold-template

# Conflicts:
#	nemo_gym/cli/env.py
@wprazuch
wprazuch marked this pull request as draft June 24, 2026 08:03
@wprazuch
wprazuch force-pushed the wprazuch/init-config-docs branch from 9bbd7f1 to b6933fb Compare June 24, 2026 08:10
Base automatically changed from wprazuch/init-config-docs to wprazuch/dataset-source June 24, 2026 08:15
@wprazuch
wprazuch force-pushed the wprazuch/dataset-source branch from 35864ee to f96315d Compare June 25, 2026 11:31
Base automatically changed from wprazuch/dataset-source to main June 25, 2026 11:40
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants