feat: document `--agent-type` CLI flag and agent swapping mechanism by marta-sd · Pull Request #2793 · NVIDIA-NeMo/Gym

marta-sd · 2026-08-26T13:23:04Z

This PR documents the new --agent CLI flag and agent swapping mechanism. It should be merge as the last as it documents features that are not yet in themain branch.

The entire PR stack will introduce the following changes:

config composition
--agent-type flag in the CLI for selecting the agent
safeguard preventing unsupported env <> agent pairings
aligning with dataset changes introduced in epic epic: dataset preparation #1338
docs update (this PR)

copy-pr-bot · 2026-08-26T13:23:08Z

This pull request requires additional validation before any workflows can run on NVIDIA's runners.

Pull request vetters can view their responsibilities here.

Contributors can view more details about this message here.

github-actions · 2026-08-26T13:24:26Z

🌿 Preview your docs: https://nvidia-preview-martas-agent-selector-docs.docs.buildwithfern.com/nemo/gym

Here are the markdown pages you've updated:

sephmard

Approving. I also think that we need to include an update in the gym/evaluation tab. I'd suggest that this is its own new subtab under Evaluate: Harness.

The merge-base changed after approval.

copy-pr-bot · 2026-08-28T10:29:32Z

Auto-sync is disabled for draft pull requests in this repository. Workflows must be run manually.

Contributors can view more details about this message here.

marta-sd · 2026-08-28T10:54:17Z

Approving. I also think that we need to include an update in the gym/evaluation tab. I'd suggest that this is its own new subtab under Evaluate: Harness.

@sephmard I added a dedicated page in 3a4c2b3

sephmard

This page is about agentic evaluation, but "The composed instance is renamed" example (gpqa_mcqa_simple_agent/hermes_agent) isn't agentic — gpqa / gsm8k uses the mcqa resources server, which is plain multiple-choice verification, no tools.

Please swap it for working agentic example — the intro's own terminal_bench_2_1/opencode + terminus_2_agent composition works directly here.

sephmard · 2026-09-02T13:45:09Z

+# Run GPQA with the Hermes harness instead of the one its config names
+gym eval run --agent-type hermes_agent --benchmark gpqa --model-type vllm_model


GPQA isnt the best example, as its non-agentic

sephmard · 2026-09-02T13:58:55Z

+## The composed instance is renamed
+
+Composition replaces the environment's agent and renames the server instance after the harness that
+runs it, so metrics and `gym env status` report what actually ran. Composing `hermes_agent` onto
+`gpqa_mcqa_simple_agent` produces `gpqa_mcqa_hermes_agent`.
+
+That name is what you use to route rollouts when the servers are already up:
+
+```bash
+gym env start --agent-type hermes_agent --benchmark gpqa --model-type vllm_model
+gym eval run --no-serve --agent gpqa_mcqa_hermes_agent --input rows.jsonl
+```
+
+End-to-end rollouts collection (without the `--no-serve` flag) keeps working — rows stamped with the
+pre-swap instance are re-routed for you under the hood.


"The composed instance is renamed" example (gpqa_mcqa_simple_agent/hermes_agent) isn't agentic (plain MCQA, no tools); suggest swapping to the intro's own terminal_bench_2_1/opencode + terminus_2_agent composition, both for accuracy-to-theme and to cut the page down to one running example instead of three.

Replacing it requires introducing changes to the configs - otherwise the composed name was completely unreadable. This change requires approval from eng team (ideally @bxyu-nvidia) because it touches configs that are in the critical path.

Ok, thanks for the context

marta-sd · 2026-09-03T07:03:45Z

This page is about agentic evaluation, but "The composed instance is renamed" example (gpqa_mcqa_simple_agent/hermes_agent) isn't agentic — gpqa / gsm8k uses the mcqa resources server, which is plain multiple-choice verification, no tools.

Please swap it for working agentic example — the intro's own terminal_bench_2_1/opencode + terminus_2_agent composition works directly here.

Due to what's in configs the name would be "opencode_sandboxed_agent_terminal_bench_2_1_terminus_2_sandboxed_agent". I need to update configs for this to make sense.

Signed-off-by: Marta Stepniewska-Dziubinska <martas@nvidia.com>

Co-authored-by: Seph Mard <seph.mard@gmail.com> Signed-off-by: Marta Stepniewska-Dziubinska <marta-sd@users.noreply.github.com>

Signed-off-by: Marta Stepniewska-Dziubinska <martas@nvidia.com>

Co-authored-by: Seph Mard <seph.mard@gmail.com> Signed-off-by: Marta Stepniewska-Dziubinska <marta-sd@users.noreply.github.com>

Server instances are named <benchmark>_<implementation>, but the sandbox benchmarks led with the harness. That also broke the composed instance name: the rename strips the agent type only as a suffix, so swapping a harness in appended the new type instead of replacing the old one. Signed-off-by: Marta Stepniewska-Dziubinska <martas@nvidia.com>

The rename and pairing examples used gpqa/gsm8k, which score multiple choice with no tools, on a page about agentic evaluation. All the sample output is now copied from real runs of the intro's own swap. Signed-off-by: Marta Stepniewska-Dziubinska <martas@nvidia.com>

sephmard

approving docs changes - @bxyu-nvidia reviewed the config changes

marta-sd requested review from lbliii and sephmard August 26, 2026 13:29

marta-sd linked an issue Aug 27, 2026 that may be closed by this pull request

Add docs for the --agent-type CLI flag #2656

Closed

sephmard previously approved these changes Aug 27, 2026

View reviewed changes

marta-sd changed the title ~~martas/agent selector docs~~ feat: document --agent-type CLI flag and agent swapping mechanism Aug 28, 2026

marta-sd force-pushed the martas/agent-selector-docs branch from 9ffc9b8 to 5e8ecfc Compare August 28, 2026 10:29

marta-sd marked this pull request as ready for review August 28, 2026 10:29

copy-pr-bot Bot temporarily deployed to public August 28, 2026 10:30 Inactive

copy-pr-bot Bot temporarily deployed to public August 28, 2026 10:32 Inactive

copy-pr-bot Bot temporarily deployed to public August 28, 2026 10:53 Inactive

copy-pr-bot Bot temporarily deployed to public August 28, 2026 10:55 Inactive

copy-pr-bot Bot temporarily deployed to public August 28, 2026 10:56 Inactive

github-actions Bot added the sla:review-overdue Review response is over the one-business-day SLA label Aug 28, 2026

github-actions Bot mentioned this pull request Aug 28, 2026

[Tracker] Pull request handoff SLA #2067

Open

Glorf previously approved these changes Aug 28, 2026

View reviewed changes

marta-sd dismissed Glorf’s stale review via b301c4c August 28, 2026 12:15

marta-sd force-pushed the martas/agent-selector-docs branch from 3a4c2b3 to b301c4c Compare August 28, 2026 12:15

copy-pr-bot Bot temporarily deployed to public August 28, 2026 12:15 Inactive

copy-pr-bot Bot temporarily deployed to public August 28, 2026 12:17 Inactive

marta-sd added the r0.6.0 Auto-cherrypick to release branch. Apply before merge; cherrypick happens after merge. label Sep 1, 2026

marta-sd dismissed agronskiy’s stale review via f4200e9 September 1, 2026 09:13

copy-pr-bot Bot temporarily deployed to public September 1, 2026 09:14 Inactive

copy-pr-bot Bot temporarily deployed to public September 1, 2026 09:15 Inactive

copy-pr-bot Bot temporarily deployed to public September 1, 2026 09:16 Inactive

copy-pr-bot Bot temporarily deployed to public September 1, 2026 11:30 Inactive

copy-pr-bot Bot temporarily deployed to public September 1, 2026 11:31 Inactive

copy-pr-bot Bot temporarily deployed to public September 1, 2026 11:32 Inactive

sephmard requested changes Sep 2, 2026

View reviewed changes

copy-pr-bot Bot temporarily deployed to public September 3, 2026 07:09 Inactive

copy-pr-bot Bot temporarily deployed to public September 3, 2026 07:10 Inactive

copy-pr-bot Bot temporarily deployed to public September 3, 2026 07:11 Inactive

marta-sd and others added 9 commits September 3, 2026 09:40

document --agent flag and swapping mechanism

96a8aef

Signed-off-by: Marta Stepniewska-Dziubinska <martas@nvidia.com>

update docs to reflect reality

4d9c0c9

Signed-off-by: Marta Stepniewska-Dziubinska <martas@nvidia.com>

add page on harness comparison

64151a3

Signed-off-by: Marta Stepniewska-Dziubinska <martas@nvidia.com>

fix: point the tau2 comment at the renamed config

6b3d0c8

Signed-off-by: Marta Stepniewska-Dziubinska <martas@nvidia.com>

Apply batched suggestions from code review

6dc14fc

Co-authored-by: Seph Mard <seph.mard@gmail.com> Signed-off-by: Marta Stepniewska-Dziubinska <marta-sd@users.noreply.github.com>

docs: use an agentic benchmark in the harness swap example

e2916c8

Signed-off-by: Marta Stepniewska-Dziubinska <martas@nvidia.com>

Apply batched suggestions from code review

c763dbb

Co-authored-by: Seph Mard <seph.mard@gmail.com> Signed-off-by: Marta Stepniewska-Dziubinska <marta-sd@users.noreply.github.com>

marta-sd force-pushed the martas/agent-selector-docs branch from ee41d5d to 46f5ce8 Compare September 3, 2026 08:33

copy-pr-bot Bot temporarily deployed to public September 3, 2026 08:33 Inactive

copy-pr-bot Bot temporarily deployed to public September 3, 2026 08:35 Inactive

bxyu-nvidia approved these changes Sep 3, 2026

View reviewed changes

sephmard approved these changes Sep 3, 2026

View reviewed changes

Navigation Menu

Uh oh!

feat: document `--agent-type` CLI flag and agent swapping mechanism - #2793

feat: document `--agent-type` CLI flag and agent swapping mechanism#2793
marta-sd merged 9 commits into
mainfrom
martas/agent-selector-docs

marta-sd commented Aug 26, 2026 •

edited

Loading

Uh oh!

copy-pr-bot Bot commented Aug 26, 2026

Uh oh!

github-actions Bot commented Aug 26, 2026 •

edited

Loading

Uh oh!

sephmard left a comment

Uh oh!

copy-pr-bot Bot commented Aug 28, 2026

Uh oh!

marta-sd commented Aug 28, 2026

Uh oh!

sephmard left a comment

Uh oh!

sephmard Sep 2, 2026

Uh oh!

Uh oh!

Uh oh!

sephmard Sep 2, 2026

Uh oh!

marta-sd Sep 3, 2026

Uh oh!

sephmard Sep 3, 2026

Uh oh!

marta-sd commented Sep 3, 2026

Uh oh!

sephmard left a comment

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

5 participants

		# Run GPQA with the Hermes harness instead of the one its config names
		gym eval run --agent-type hermes_agent --benchmark gpqa --model-type vllm_model

Uh oh!

Conversation

marta-sd commented Aug 26, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

copy-pr-bot Bot commented Aug 26, 2026

Uh oh!

github-actions Bot commented Aug 26, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

sephmard left a comment

Choose a reason for hiding this comment

Uh oh!

copy-pr-bot Bot commented Aug 28, 2026

Uh oh!

marta-sd commented Aug 28, 2026

Uh oh!

sephmard left a comment

Choose a reason for hiding this comment

Uh oh!

sephmard Sep 2, 2026

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

sephmard Sep 2, 2026

Choose a reason for hiding this comment

Uh oh!

marta-sd Sep 3, 2026

Choose a reason for hiding this comment

Uh oh!

sephmard Sep 3, 2026

Choose a reason for hiding this comment

Uh oh!

marta-sd commented Sep 3, 2026

Uh oh!

sephmard left a comment

Choose a reason for hiding this comment

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

5 participants

marta-sd commented Aug 26, 2026 •

edited

Loading

github-actions Bot commented Aug 26, 2026 •

edited

Loading