Add EnterpriseOps-Gym benchmark: resources server, benchmark registration, and per-turn telemetry agent

[mcuevas-nvidia]

Summary

This PR integrates ServiceNow's EnterpriseOps-Gym (EOG) into NeMo Gym: a 649-task benchmark of stateful, multi-step enterprise tool use across 8 domains (calendar, CSM, drive, email, HR, ITSM, teams, and cross-domain hybrid), where an agent operates 512 tools against live MCP servers backed by SQL databases and is scored on final database state. The integration is eval-complete and RL-ready (token-ID capture, fractional reward mode, and a validated GRPO rollout-collection recipe), with no changes to NeMo Gym core.

What's included (46 files)

Component	Path	What it does
Resources server	resources_servers/enterpriseops_gym/	Seeds a per-session database on the external EOG MCP containers, proxies tool calls (catch-all /{tool_name} route, pooled aiohttp, per-session x-database-id), runs verifiers concurrently, cleans up idempotently (TTL janitor + delete-on-verify). Supports replica pools (gym_url_pools), per-domain metrics, per-tool latency capture, and a strict_verifiers fractional-reward mode for RL.
Verifier engine	.../verifier_engine.py	Line-for-line port of EOG's scoring semantics, pinned by golden fixtures generated from the original implementation (33 cases, byte-identical).
Task converter	.../convert_tasks.py + benchmarks/enterpriseops/	Converts the public HF dataset (ServiceNow-AI/EnterpriseOps-Gym) to Responses-API task rows at prepare.py time; benchmark registered with prompt_config: null (pre-baked rows). Dataset files are not committed.
Per-turn telemetry agent	responses_api_agents/turn_logging_agent/	SimpleAgent subclass with an identical loop that records per-turn timestamps, durations, input/output/cached/reasoning tokens, and tool names, attaching turns to the verify response. Generic — not EOG-specific.
Tests	44 total (42 + 2)	Fully offline: a sqlite-backed stub MCP gym (tests/stub_gym.py) plus golden parity fixtures. No containers or network needed to run CI.
Docs	PARITY.md, PERF.md, RLPILOT.md	Full parity evidence, throughput study, and RL rollout-collection pilot (methodology + measured numbers below).

Verification logic

Each task carries verifier_metadata with a list of verifiers. database_state verifiers
run a SQL query against the session's final database state (via the gym containers'
/api/sql-runner), extract a value, and compare it to the expected value using EOG's
comparison semantics; response_check verifiers score the agent's final message with an
LLM judge (defaults to the policy model, temperature pinned to 0.0, matching EOG). The
public oracle split is 100% database_state (3,496/3,496 verifiers), so scoring there is
fully deterministic given a final DB state. Reward = 1.0 iff all (name-collapsed) verifiers
pass, matching the upstream leaderboard; strict_verifiers: true switches to
every-verifier-counts and a fractional strict_pass_rate for RL shaping.

Scoring fidelity (PARITY.md)

The port is validated bug-for-bug against the upstream harness, preserving its quirks (verifier name-collapse, unknown-gym skips, loose comparison semantics) for leaderboard comparability:

• Unit level: golden fixtures generated by running the original EOG engine — byte-identical extraction/comparison behavior.
• Task level: 12/12 identical outcomes and verifier structures on live containers.
• Full split (649 tasks): port 16.4% vs native 16.8% macro (gpt-4.1-mini, temp 0); per-task agreement 90.4%, disagreements symmetric (McNemar exact p = 0.90).
• k=5 × both harnesses (6,480 rollouts): Δ −0.23 ± 0.7 pp, per-task preference exactly 67:67.
• Cross-stack replication (k=5 on dedicated vLLM, Nemotron 3 Nano): formally equivalent within ±2 pp (TOST, α = 0.05).

Performance (PERF.md)

Scale-tested end-to-end at five client concurrencies (c = 8, 16, 32, 64, 128) with a full
649-task pass per level per harness (10 passes, 4×H100 vLLM, identical endpoint): the port
completed 649/649 tasks at every level with zero retries, with success rates flat
across levels. At matched concurrency the port is 1.14–1.71× faster (largest at low
concurrency, nearest the native harness's documented defaults). The gap narrows by design:
both harnesses converge toward the same GPU throughput floor — and reaching it is the key
result. The port saturates the hardware at c=64; the native harness never reaches the
floor in the tested range and needs ~4× the client concurrency for equal throughput.
Net cost: a full-split eval is 1h27m of 4×H100 time (port) vs 4h40m at native's documented
settings — 3.2× GPU-hours. Mechanism: pooled connections, persistent MCP sessions, and
concurrent verifiers keep vLLM's continuous batch fed. (Since the gap is client dead time
relative to GPU service time, it is expected to widen on faster serving hardware, where
saturation demands even more effective concurrency.)

RL readiness (RLPILOT.md)

A config-only pilot (zero code changes) validated GRPO rollout collection end-to-end: 100% token-ID/logprob coverage via return_token_id_information, 15/20 task groups with mixed binary reward at k=8 (mean within-group std 0.341) on a curriculum selected from repeat-run data, plus a measured deployment-sizing guide (sequence-length distribution, memory budget, recommended node shapes).

How to run

# Start the stack (external EOG MCP containers must be running; see resources server README)
ng_run "+config_paths=[resources_servers/enterpriseops_gym/configs/enterpriseops_gym.yaml,responses_api_models/openai_model/configs/openai_model.yaml]" \
  "++enterpriseops_gym.resources_servers.enterpriseops_gym.seed_sql_root=/path/to/EnterpriseOps-Gym"

# Prepare the benchmark data (downloads the public HF dataset) and collect
ng_prepare_benchmark +benchmark_name=enterpriseops
ng_collect_rollouts +agent_name=enterpriseops_gym_simple_agent \
  +input_jsonl_fpath=benchmarks/enterpriseops/data/enterpriseops_oracle_benchmark.jsonl \
  +output_jsonl_fpath=results/enterpriseops.jsonl \
  +responses_create_params.temperature=0.0 +responses_create_params.max_output_tokens=16384

# Tests (offline, no containers needed)
gym env test --resources-server enterpriseops_gym
ng_test +entrypoint=responses_api_agents/turn_logging_agent

Validation on this exact branch state

• 42/42 + 2/2 tests green after rebase onto current main
• ruff check + ruff format --check clean; README environment table regenerated via scripts/update_env_list.py
• Config stack boots via ng_run dry run
• Live smoke on real MCP containers + gateway model: 5/5 rollouts completed across both agents, all verifiers scored, turn telemetry attached
• All commits DCO signed-off

Contribution-guide compliance

Mapped to the environment /
benchmark guides:

• Required files — all present: app.py, configs/*.yaml (valid domain: agent),
  tests/test_app.py (42 tests), data/example.jsonl (5 tasks),
  data/example_rollouts.jsonl (5 pre-generated rollouts against live containers; note
  these samples are CSM-domain, the benchmark's hardest — near-zero rewards on them are
  expected and consistent with the full-split CSM rate of ~4%), requirements.txt,
  README.md with licensing information.
• Reward profiling — run on a closed model (gpt-4.1-mini: 16.4% macro) and an open
  thinking model (Nemotron 3 Nano, reasoning on: 22–25%), i.e. the instruct+thinking
  mixture the guide asks for. Scores sit inside the official leaderboard's published range
  (Qwen3-4B 13.6% … GPT-5-Mini 22.0%) with a coherent domain pattern (email easiest, CSM
  hardest, matching the leaderboard). This benchmark is legitimately hard — no public model
  reaches 30%.
• Variance — k=5 repeat runs on both stacks; mean@5 resolves ±0.7 pp (< 1% as required).
  Calibration guidance for users is included in PARITY.md.
• Failure-case analysis — performed extensively across parity, perf, and RL-pilot runs
  (documented in the three reports).
• Original-repo reproduction — instead of reproducing a leaderboard model's published
  number (leaderboard models weren't available on our serving), we ran the original EOG
  harness side-by-side on identical models, containers, and endpoints: per-task agreement
  with symmetric disagreements (McNemar p = 0.90) and formal TOST equivalence within ±2 pp.
  This isolates harness fidelity even more directly; happy to additionally run a listed
  leaderboard model if reviewers want the published-number check.
• Stacked PRs — per the development-setup guide, happy to restack this as layers
  (resources server + tests → benchmark registration → turn-logging agent → reports) if
  reviewers prefer; presented as one PR first since the layers are tightly coupled by the
  parity evidence.

Design decisions & notes for reviewers

• Hand-rolled MCP client instead of the official SDK (rationale in mcp_client.py docstring): the SDK's transport is httpx-based (banned for async here), EOG needs session-level and per-call x-database-id headers, and half the surface is non-MCP REST (/api/seed-database etc.).
• Upstream EOG quirks intentionally preserved and documented (PARITY.md §1) rather than fixed, to keep leaderboard comparability; strict_verifiers: true opts into every-verifier-counts scoring for RL.
• turn_logging_agent is separable — it's a general-purpose agent; happy to split it into its own PR if preferred.
• verified: false per convention for new resources servers.
• External dependency: the benchmark requires the EOG Docker containers (from the upstream EOG repo) at eval time; unit tests do not.
• Operational findings that affect large runs (MCP container fd leak ~5/task, upstream runner's silent task drops) are documented in PARITY.md §6; we plan to file these upstream against EOG.

Data provenance & licensing

EnterpriseOps-Gym is Apache 2.0 (code) with a public HF dataset (ServiceNow-AI/EnterpriseOps-Gym). This PR commits only: tool-schema snapshots captured from the public EOG containers (7 JSON files, 512 tools), 13 sample tasks derived from the EOG repo's task files, 5 example rollouts generated against live containers, one synthetic hybrid task hand-authored for tests (written with LLM assistance against live container schemas — disclosed per the synthetic-data guideline), and golden verifier fixtures generated by running the EOG engine. The full benchmark dataset is downloaded at prepare.py time and gitignored.

[copy-pr-bot]

This pull request requires additional validation before any workflows can run on NVIDIA's runners.

Pull request vetters can view their responsibilities here.

Contributors can view more details about this message here.

[cmunley1]

can these long files be pulled from somewhere like huggingface instead of committed?

[cmunley1]

/claude review

[claude]

SHIP WITH CARE — no BLOCKERs. The verifier port and reward aggregation are the highest-risk surface here and they're the most carefully done part of the PR: EOG parity quirks (name-collapse, unknown-gym skips, loose comparisons) are documented and pinned by golden fixtures, strict_verifiers cleanly separates leaderboard-parity reward from RL reward, verify is idempotent and deletes DBs in a finally, the MCP client correctly routes all async HTTP through nemo_gym.server_utils.request() (no httpx, no ray.get()), and per-call x-database-id isolation is the right call over the SDK's session-level headers. No core changes, tests are fully offline, deps declared. Good work.

Two non-blocking findings, both inline:

1. RISK (turn_logging_agent/app.py): the agent hardcodes bare /v1/responses where simple_agent uses url_path_for_request/url_path_for_run, dropping the /ng-rollout/<id> capture-correlation prefix. No-op when observability is off, so eval/reward and inline return_token_id_information are unaffected — but this agent's whole purpose is RL telemetry, which is where capture gets enabled. The docstring even claims step-for-step parity; this is the divergence.

2. RISK (app.py verify): all([]) is True, so a fully empty or all-skipped verifier set scores reward 1.0. A gym_name/pool misconfig silently converts every task to a false pass — inflates eval macro-rates and feeds spurious positive reward into GRPO. Guard the strict path (bool(strict_passed) and all(...)) and at least log.warning on num_verifiers_scored == 0.

Neither blocks merge; #2 is the one I'd want resolved before this feeds a training run.

[claude]

RISK — drops the /ng-rollout/<id> capture-correlation prefix that simple_agent applies.

The docstring says this loop "mirrors simple_agent step for step... Keep it in sync," but it has already diverged here: simple_agent.responses() uses url_path_for_request("/v1/responses", request) (line 89) and simple_agent.run() uses url_path_for_run("/v1/responses", body) (line 190). This agent hardcodes bare /v1/responses at both sites (here and line 273).

WHAT BREAKS: when model-call capture / observability is enabled, url_path_for_* prefixes the outgoing call with /ng-rollout/<rollout_id> so the model server can key captured calls to the rollout. Without it, this agent's captured model calls carry no correlation id — the exact per-rollout observability that a telemetry agent exists to provide is silently lost.

BLAST RADIUS: only when observability is on (no-op prefix otherwise), so eval/reward and return_token_id_information (inline in the response body) are unaffected — RLPILOT numbers stand. But this agent targets RL rollout collection, which is where capture is most likely enabled.

FIX: url_path=self.url_path_for_request("/v1/responses", request) here, and self.url_path_for_run("/v1/responses", body) in run(). Nearly every contributed agent (browsecomp, critpt, cvdp, non_executing_simple, gymnasium…) uses these helpers.

[mcuevas-nvidia]

89ba342 for the correlation fix

[mcuevas-nvidia]

Both call sites now use url_path_for_request / url_path_for_run, matching simple_agent, with a regression test that drives the /ng-rollout//v1/responses route and asserts the prefix survives onto the model call.

[claude]

RISK — an empty or all-skipped verifier set scores reward 1.0.

all([]) is True, so:

• If every verifier is skipped (all reference an unknown gym_name), verification_results is empty → overall_success = all([]) = True → parity reward 1.0.
• If a task defines zero verifiers, strict_passed = [] → strict_success = all([]) = True → strict reward 1.0.

WHAT BREAKS: a misconfigured gym_url_pools/dataset gym_name mismatch (plausible when pointing at replicas for an RL run) silently turns every task into a false pass instead of a hard failure. That corrupts eval macro-rates upward and injects spurious positive reward into GRPO training — the worst-case silent-corruption path.

Note the skipped-verifier→fail guarantee already holds for strict mode when some verifiers survive (test_tool_execution_verifier_and_skipped_gym); the gap is only the fully-empty set.

FIX (at minimum for the strict/RL path, where empty=pass is never intended):
strict_success = bool(strict_passed) and all(strict_passed).
For the parity path, confirm upstream EOG also awards 1.0 on an all-skipped task; if it does, keep it but consider a log.warning when num_verifiers_scored == 0 so silent all-skips are visible in large runs.

[mcuevas-nvidia]

b0494d2 for the verifier guard

[mcuevas-nvidia]

Fixed for the strict/RL path in b0494d2 (bool(strict_passed) and all(strict_passed)), with a regression test.

The collapsed metric keeps upstream EOG's all([]) behavior on purpose: that path is a bug-for-bug port of the upstream scorer, which is what keeps our numbers comparable to the published EOG leaderboard. The rationale is documented at the code site, and num_verifiers_scored exposes the empty-set case to callers.

[mcuevas-nvidia]

Good question. Quick context on what these are: the data/tools/*.json files are snapshots of the tool schemas advertised by EnterpriseOps-Gym's MCP containers (512 tools across 7 domains). They can't come from HuggingFace as-is because the upstream HF dataset only contains tasks; the schemas exist only inside the running Docker containers, and they're needed at prepare.py time to build task rows. Pinned copies are also what guarantees byte-level prompt parity with the upstream harness (see PARITY.md). I agree they're bulky. Two options: 1. Host them on HF (a small companion dataset, or the existing dataset repo if ServiceNow will take them, which would need upstreaming) and have prepare.py download them, keeping only a checksum in-repo. Downside: prepare gains a network dependency for data that changes only when upstream ships new containers. 2. Keep them committed as the reproducibility anchor. They're versioned exactly like golden test fixtures. My mild preference is (2) for reproducibility, but (1) works if repo size is the priority. From: Christian Munley ***@***.***> Date: Monday, July 27, 2026 at 10:13 AM To: NVIDIA-NeMo/Gym ***@***.***> Cc: Marc Cuevas ***@***.***>; Author ***@***.***> Subject: Re: [NVIDIA-NeMo/Gym] Add EnterpriseOps-Gym benchmark: resources server, benchmark registration, and per-turn telemetry agent (PR #2142) @cmunley1 commented on this pull request.

________________________________ In resources_servers/enterpriseops_gym/data/tools/calendar.json<#2142 (comment)>:

@@ -0,0 +1,3436 @@

+{ can these long files be pulled from somewhere like huggingface instead of committed? — Reply to this email directly, view it on GitHub<#2142?email_source=notifications&email_token=B55WDQREOH2VD5KOXZ7ALYD5G6ERPA5CNFSNUABKM5UWIORPF5TWS5BNNB2WEL2QOVWGYUTFOF2WK43UKJSXM2LFO4XTINZYHE3DENBRGMYKM4TFMFZW63VGMF2XI2DPOKSWK5TFNZ2KYZTPN52GK4S7MNWGSY3L#pullrequestreview-4789624130>, or unsubscribe<https://github.com/notifications/unsubscribe-auth/B55WDQVTNXPZ77HYPHRFJJD5G6ERPAVCNFSNUABGKJSXA33TNF2G64TZHMYTANBUGU2TIOBUGA5US43TOVSTWNBZG4ZTENRWHEYTTILWAI>. Triage notifications, keep track of coding agent tasks and review pull requests on the go with GitHub Mobile for iOS<https://github.com/notifications/mobile/ios/B55WDQVOGQBHPOBD2ZQZR735G6ERPA5CNFSNUABKM5UWIORPF5TWS5BNNB2WEL2QOVWGYUTFOF2WK43UKJSXM2LFO4XTINZYHE3DENBRGMYKM4TFMFZW63VGMF2XI2DPOKSWK5TFNZ2KUZTPN52GK4S7NFXXG> and Android<https://github.com/notifications/mobile/android/B55WDQR4JTS3JH3KU26HLPL5G6ERPA5CNFSNUABKM5UWIORPF5TWS5BNNB2WEL2QOVWGYUTFOF2WK43UKJSXM2LFO4XTINZYHE3DENBRGMYKM4TFMFZW63VGMF2XI2DPOKSWK5TFNZ2K4ZTPN52GK4S7MFXGI4TPNFSA>. Download it today! You are receiving this because you authored the thread.