Dataset viewer simple aggregations by fsiino-nvidia · Pull Request #9 · NVIDIA-NeMo/Gym

fsiino-nvidia · 2025-09-05T21:28:28Z

Migrated over from gitlab:

Display aggregate metrics
Aggregate generic keys using multineedle
Display other dynamic aggregations
Count string totals and unique values
Remove TrainDataProcessor dependency, add test
Remove dupe file read, fix arg types hints

copy-pr-bot · 2025-09-05T21:28:32Z

This pull request requires additional validation before any workflows can run on NVIDIA's runners.

Pull request vetters can view their responsibilities here.

Contributors can view more details about this message here.

- Display aggregate metrics - Aggregate generic keys using multineedle - Display other dynamic aggregations - Count string totals and unique values - Remove TrainDataProcessor dependency, add test - Remove dupe file read, fix arg types hints Signed-off-by: Frankie Siino <fsiino@nvidia.com>

Signed-off-by: Frankie Siino <fsiino@nvidia.com>

…thub

Migrated over from gitlab: - Display aggregate metrics - Aggregate generic keys using multineedle - Display other dynamic aggregations - Count string totals and unique values - Remove TrainDataProcessor dependency, add test - Remove dupe file read, fix arg types hints --------- Signed-off-by: Frankie Siino <fsiino@nvidia.com> Signed-off-by: soares-f <soarescmsa@gmail.com>

Migrated over from gitlab: - Display aggregate metrics - Aggregate generic keys using multineedle - Display other dynamic aggregations - Count string totals and unique values - Remove TrainDataProcessor dependency, add test - Remove dupe file read, fix arg types hints --------- Signed-off-by: Frankie Siino <fsiino@nvidia.com> Signed-off-by: Abhibha Gupta <abhibhag@nvidia.com>

Migrated over from gitlab: - Display aggregate metrics - Aggregate generic keys using multineedle - Display other dynamic aggregations - Count string totals and unique values - Remove TrainDataProcessor dependency, add test - Remove dupe file read, fix arg types hints --------- Signed-off-by: Frankie Siino <fsiino@nvidia.com>

TESTPLAN NVIDIA-NeMo#9 / Q5: re-run the concurrency sweep (1..131072, 64k body) with the synthetic model drawing per-call latency from a Pareto distribution (~1.5s median, heavy tail capped at 2 min) instead of the near-zero proxy, so we can see whether the throughput ceiling and saturation knee shift under realistic per-rollout latency. Latency params are a single retunable knob (REAL_LATENCY_*) to lock against a real-vLLM calibration later. Add burst_repro and realistic_latency to the suite submitter. Signed-off-by: Ananth Subramaniam <ansubramania@nvidia.com>

Rename the local vLLM *server* launcher from `local_vllm_model` to `vllm_server` (RFC M6b / friction #9). The directory `responses_api_models/local_vllm_model` is moved to `responses_api_models/vllm_server`, and textual references are rewritten via `nemo_gym.model_server_rename`: - config_paths directory paths (`responses_api_models/<old>` -> `<new>`) - the `responses_api_models:` config key in functional in-tree configs (aalcr, browsecomp, nemotron_3_ultra endpoints, indirect_prompt_injection, math_with_judge, xstest) and the moved server's own configs. Python module-path imports `responses_api_models.local_vllm_model.*` are updated to `responses_api_models.vllm_server.*` in genrm_model/app.py, the moved app.py, and the moved tests/test_app.py. Class names (`LocalVLLMModel*`) and the `local_vllm_model_actor.py` filename are kept to minimize churn. A backward-compat shim is left at the old path (`responses_api_models/local_vllm_model/{__init__,app,local_vllm_model_actor}.py`) that re-exports from the new location and emits a DeprecationWarning, so external `import responses_api_models.local_vllm_model.*` keeps working. The 15 model configs are NOT duplicated: external config-PATH pins must migrate to `responses_api_models/vllm_server/configs/...`. Signed-off-by: Wojciech Prazuch <wprazuch@nvidia.com>

…erver Complete the M6b rename (friction #9) for both vLLM model servers, with backward-compat import shims: - git mv the two server dirs; rename canonical configs to <new>.yaml; rewrite the inner responses_api_models config key, _delete_key directives, config_paths, and ~150 doc/config references across the tree (via nemo_gym.model_server_rename, now also _delete_key-aware). - Update Python module-path imports (responses_api_models.<old> -> .<new>) in the consumers of VLLMConverter/VLLMModel (genrm_model, azure_openai_model, local_vllm_model_proxy, browsecomp/swe/tau2/harbor agents) and the renamed servers themselves; class names kept. - Backward-compat shims at the old dir paths (app.py/client.py re-export from the new module + DeprecationWarning) so 'import responses_api_models.<old>.*' keeps working. vllm_model is a remote vLLM *endpoint* client; local_vllm_model launches a local vLLM *server* — the new names say which. Frozen fern v0.2.1/v0.3.0 docs untouched. NOTE: external config-PATH pins (e.g. EFB) to responses_api_models/<old>/configs/... are not covered by the import shim and must migrate; tracked for the deprecation cycle. Part of epic #1205 (M6b / friction #9). Signed-off-by: Wojciech Prazuch <wprazuch@nvidia.com>

…ples) Epic #1205 friction #9 (FEP-1023 / M6d): add a docstring to the Domain enum describing what each value is for, with a recognizable example benchmark, plus guidance to prefer the most specific domain and use 'other' only as a catch-all. Surfaces on IDE hover / when reading config_types. A test enforces every value stays documented. Stacks on the unified-CLI base (martas/1434). Signed-off-by: Wojciech Prazuch <wprazuch@nvidia.com>

Epic #1205 friction #9 (FEP-1023 / M6e), issue #395 ('Resources Server' is nebulous): the key-terminology glossary defined Verifier/Policy Model but not the three FastAPI server types. Add clear entries for Resources Server (= the environment + verifier service), Agent Server, and Model Server, tying each to its directory. Mirrored into latest + v0.3.0. Signed-off-by: Wojciech Prazuch <wprazuch@nvidia.com>

…ples) Epic #1205 friction #9 (FEP-1023 / M6d): add a docstring to the Domain enum describing what each value is for, with a recognizable example benchmark, plus guidance to prefer the most specific domain and use 'other' only as a catch-all. Surfaces on IDE hover / when reading config_types. A test enforces every value stays documented. Stacks on the unified-CLI base (martas/1434). Signed-off-by: Wojciech Prazuch <wprazuch@nvidia.com>

Epic #1205 friction #9 (FEP-1023 / M6e), issue #395 ('Resources Server' is nebulous): the key-terminology glossary defined Verifier/Policy Model but not the three FastAPI server types. Add clear entries for Resources Server (= the environment + verifier service), Agent Server, and Model Server, tying each to its directory. Mirrored into latest + v0.3.0. Signed-off-by: Wojciech Prazuch <wprazuch@nvidia.com>

…ples) Epic #1205 friction #9 (FEP-1023 / M6d): add a docstring to the Domain enum describing what each value is for, with a recognizable example benchmark, plus guidance to prefer the most specific domain and use 'other' only as a catch-all. Surfaces on IDE hover / when reading config_types. A test enforces every value stays documented. Stacks on the unified-CLI base (martas/1434). Signed-off-by: Wojciech Prazuch <wprazuch@nvidia.com>

Epic #1205 friction #9 (FEP-1023 / M6e), issue #395 ('Resources Server' is nebulous): the key-terminology glossary defined Verifier/Policy Model but not the three FastAPI server types. Add clear entries for Resources Server (= the environment + verifier service), Agent Server, and Model Server, tying each to its directory. Mirrored into latest + v0.3.0. Signed-off-by: Wojciech Prazuch <wprazuch@nvidia.com>

…ples) Epic #1205 friction #9 (FEP-1023 / M6d): add a docstring to the Domain enum describing what each value is for, with a recognizable example benchmark, plus guidance to prefer the most specific domain and use 'other' only as a catch-all. Surfaces on IDE hover / when reading config_types. A test enforces every value stays documented. Stacks on the unified-CLI base (martas/1434). Signed-off-by: Wojciech Prazuch <wprazuch@nvidia.com>

Epic #1205 friction #9 (FEP-1023 / M6e), issue #395 ('Resources Server' is nebulous): the key-terminology glossary defined Verifier/Policy Model but not the three FastAPI server types. Add clear entries for Resources Server (= the environment + verifier service), Agent Server, and Model Server, tying each to its directory. Mirrored into latest + v0.3.0. Signed-off-by: Wojciech Prazuch <wprazuch@nvidia.com>

…ples) Epic #1205 friction #9 (FEP-1023 / M6d): add a docstring to the Domain enum describing what each value is for, with a recognizable example benchmark, plus guidance to prefer the most specific domain and use 'other' only as a catch-all. Surfaces on IDE hover / when reading config_types. A test enforces every value stays documented. Stacks on the unified-CLI base (martas/1434). Signed-off-by: Wojciech Prazuch <wprazuch@nvidia.com>

…#1633) ## What Adds a docstring to the `Domain` enum in `config_types.py` — a one-line description per value with a recognizable example benchmark, plus guidance to **pick the most specific domain** and use `other` only as a catch-all (addresses the [#600](#600) 'agent is too broad / no guidance' concern **without** redefining the taxonomy). ``` math — mathematical problem solving (e.g. AIME, MATH, GSM8K). coding — code generation, repair, or execution (e.g. SWE-bench, LiveCodeBench). agent — multi-step, tool-using tasks (e.g. tau2, workplace_assistant). Prefer a more specific value when it fits. ... (knowledge, instruction_following, long_context, safety, games, translation, e2e, rlhf, other) ``` ## Why Epic [#1205](#1205) friction 9 (naming confusion). Surfaces on IDE hover and when reading `config_types`; complements the inline `domain` comment added in #1597. ## Tests `test_domain_enum_documents_every_value` asserts every `Domain` value appears in the docstring, so a newly-added domain can't ship undocumented. --------- Signed-off-by: Wojciech Prazuch <wprazuch@nvidia.com>

Epic #1205 friction #9 (FEP-1023 / M6e), issue #395 ('Resources Server' is nebulous): the key-terminology glossary defined Verifier/Policy Model but not the three FastAPI server types. Add clear entries for Resources Server (= the environment + verifier service), Agent Server, and Model Server, tying each to its directory. Mirrored into latest + v0.3.0. Signed-off-by: Wojciech Prazuch <wprazuch@nvidia.com>

…tion #9, #395) (#1634) ## What The key-terminology glossary (`fern/.../about/concepts/key-terminology.mdx`) defined `Verifier`, `Policy Model`, etc. but **not** the three FastAPI server types — so "Resources Server" was never actually defined ([#395](#395): the name is nebulous). Added clear Architecture-Terms entries: - **Resources Server** — *is* the environment: per-task state + tools + `verify()` → reward. "Read it as the environment + verifier service." - **Agent Server (Responses API Agent)** — the harness that drives the model through the task. - **Model Server (Responses API Model)** — wraps an LLM endpoint behind the Responses API. Each ties the term to its directory (`resources_servers/`, `responses_api_agents/`, `responses_api_models/`). Mirrored into `latest` and `v0.3.0`. ## Why Epic [#1205](#1205) friction 9 (naming confusion), RFC milestone M6e. Docs-only. --------- Signed-off-by: Wojciech Prazuch <wprazuch@nvidia.com>

…#1633) ## What Adds a docstring to the `Domain` enum in `config_types.py` — a one-line description per value with a recognizable example benchmark, plus guidance to **pick the most specific domain** and use `other` only as a catch-all (addresses the [#600](#600) 'agent is too broad / no guidance' concern **without** redefining the taxonomy). ``` math — mathematical problem solving (e.g. AIME, MATH, GSM8K). coding — code generation, repair, or execution (e.g. SWE-bench, LiveCodeBench). agent — multi-step, tool-using tasks (e.g. tau2, workplace_assistant). Prefer a more specific value when it fits. ... (knowledge, instruction_following, long_context, safety, games, translation, e2e, rlhf, other) ``` ## Why Epic [#1205](#1205) friction 9 (naming confusion). Surfaces on IDE hover and when reading `config_types`; complements the inline `domain` comment added in #1597. ## Tests `test_domain_enum_documents_every_value` asserts every `Domain` value appears in the docstring, so a newly-added domain can't ship undocumented. --------- Signed-off-by: Wojciech Prazuch <wprazuch@nvidia.com> Signed-off-by: Rita Fernandes Neves <rfernandesne@nvidia.com>

…tion #9, #395) (#1634) ## What The key-terminology glossary (`fern/.../about/concepts/key-terminology.mdx`) defined `Verifier`, `Policy Model`, etc. but **not** the three FastAPI server types — so "Resources Server" was never actually defined ([#395](#395): the name is nebulous). Added clear Architecture-Terms entries: - **Resources Server** — *is* the environment: per-task state + tools + `verify()` → reward. "Read it as the environment + verifier service." - **Agent Server (Responses API Agent)** — the harness that drives the model through the task. - **Model Server (Responses API Model)** — wraps an LLM endpoint behind the Responses API. Each ties the term to its directory (`resources_servers/`, `responses_api_agents/`, `responses_api_models/`). Mirrored into `latest` and `v0.3.0`. ## Why Epic [#1205](#1205) friction 9 (naming confusion), RFC milestone M6e. Docs-only. --------- Signed-off-by: Wojciech Prazuch <wprazuch@nvidia.com> Signed-off-by: Rita Fernandes Neves <rfernandesne@nvidia.com>

…audit) - (#4) Cache the static mcp_config (read once, reuse) instead of reading it from disk every rollout. - (NVIDIA-NeMo#5) Document that the per-rollout Gym entry overwrites a same-named static mcp_config server. - (NVIDIA-NeMo#6) Warn when seed MCP metadata has no headers (tool calls would otherwise be unauthenticated). - (NVIDIA-NeMo#7) Add an agent test asserting the session cookie is threaded seed_session -> verify. - (#8a) Move the example task ground truth (expected_city) under verifier_metadata, per convention. - (#8b) Use the message-list input format in the example dataset, like the other examples. - (NVIDIA-NeMo#9) Remove the dead 'url' branch in the rollout mcp_config (MCPServerMetadata has no 'url'); audited that every metadata field accessed (server_name/url_path/transport/headers) actually exists. - (#2v) Add a test that a token-less MCP tool call surfaces as a clean isError (HTTP 200), not a 401. Also patch only ensure_claude_code (not all of model_post_init) in the claude_code_agent test helper, so the model's private attributes initialize. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> Signed-off-by: Codex <codex@openai.com>

fsiino-nvidia changed the title ~~Dataset viewer simple aggregations:~~ Dataset viewer simple aggregations Sep 5, 2025

fsiino-nvidia added 4 commits September 5, 2025 15:35

Restore copyright/license, move test

65a7de1

Signed-off-by: Frankie Siino <fsiino@nvidia.com>

Format

3b09058

Signed-off-by: Frankie Siino <fsiino@nvidia.com>

Precommit fix

99afdd0

Signed-off-by: Frankie Siino <fsiino@nvidia.com>

fsiino-nvidia force-pushed the fsiino/dataset-viewer-simple-aggregations-github branch from 958deb5 to 99afdd0 Compare September 5, 2025 22:36

fsiino-nvidia marked this pull request as ready for review September 5, 2025 22:39

fsiino-nvidia added 2 commits September 5, 2025 16:09

Fix test

5f644b7

Signed-off-by: Frankie Siino <fsiino@nvidia.com>

Merge branch 'main' into fsiino/dataset-viewer-simple-aggregations-gi…

de718dd

…thub

fsiino-nvidia requested a review from bxyu-nvidia September 6, 2025 00:12

bxyu-nvidia approved these changes Sep 6, 2025

View reviewed changes

bxyu-nvidia merged commit bb1a3ba into main Sep 6, 2025
5 checks passed

bxyu-nvidia deleted the fsiino/dataset-viewer-simple-aggregations-github branch September 6, 2025 00:16

ritaneves mentioned this pull request Jun 25, 2026

epic: reduce configuration friction for environment authors and users #1205

Closed

5 tasks

Navigation Menu

Uh oh!

Dataset viewer simple aggregations - #9

Dataset viewer simple aggregations#9
bxyu-nvidia merged 6 commits into
mainfrom
fsiino/dataset-viewer-simple-aggregations-github

fsiino-nvidia commented Sep 5, 2025

Uh oh!

copy-pr-bot Bot commented Sep 5, 2025

Uh oh!

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

2 participants

Uh oh!

Conversation

fsiino-nvidia commented Sep 5, 2025

Uh oh!

copy-pr-bot Bot commented Sep 5, 2025

Uh oh!

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

2 participants