add eval/{env}/failed_rollouts metric

[rasdani]

log silently excluded failed rollouts during online-eval to understand how skewed eval results are.

---

Note

Low Risk
Low risk: changes are limited to evaluation/monitoring instrumentation and warning logs, without altering rollout generation/scoring behavior.

Overview
Adds a new eval/{env}/failed_rollouts metric by comparing requested eval inputs vs returned outputs, and logs it both in normal eval metrics and when all rollouts fail (so the failure rate is still visible).

Improves visibility into generation failures by warning when some rollout groups fail inside vf_utils.generate() (previously failures were only logged per-group).

^{Written by Cursor Bugbot for commit b0afdb6. This will update automatically on new commits. Configure here.}

[cursor]

Cursor Bugbot has reviewed your changes and found 2 potential issues.

Bugbot Autofix prepared fixes for both issues found in the latest run.

• ✅ Fixed: Double call to _get_eval_inputs risks inconsistent metric
  • Modified evaluate_env to call _get_eval_inputs once and pass inputs directly to evaluate function, eliminating the duplicate call and ensuring consistent metrics.
• ✅ Fixed: Failed rollouts metric not logged when all fail
  • Added logging of failed_rollouts metric in the early return path when all rollouts fail, ensuring the metric is tracked even in complete failure scenarios.

Or push these changes by commenting:

@cursor push b3b89f2cc1

Preview (b3b89f2cc1)

diff --git a/src/prime_rl/orchestrator/eval_utils.py b/src/prime_rl/orchestrator/eval_utils.py
--- a/src/prime_rl/orchestrator/eval_utils.py
+++ b/src/prime_rl/orchestrator/eval_utils.py
@@ -103,13 +103,13 @@
     logger = get_logger()
     logger.info(f"Evaluating {env_name} ({num_examples=}, {rollouts_per_example=})")
     eval_start_time = time.perf_counter()
-    total_inputs = len(env._get_eval_inputs(num_examples, rollouts_per_example))
+    inputs = env._get_eval_inputs(num_examples, rollouts_per_example)
+    total_inputs = len(inputs)
     outputs = await evaluate(
         env=env,
         model_name=model_name,
         sampling_args=sampling_args,
-        num_examples=num_examples,
-        rollouts_per_example=rollouts_per_example,
+        inputs=inputs,
         get_client=get_client,
         max_retries=max_retries,
     )
@@ -118,6 +118,16 @@
 
     if not outputs:
         logger.warning(f"All rollouts failed for {env_name}, skipping metrics")
+        monitor = get_monitor()
+        monitor.log(
+            {
+                f"eval/{env_name}/failed_rollouts": failed_rollouts,
+                f"eval/{env_name}/time": eval_time,
+                "progress/ckpt_step": ckpt_step,
+                "step": step,
+            },
+            step=step,
+        )
         return
 
     rows = []

diff --git a/src/prime_rl/orchestrator/vf_utils.py b/src/prime_rl/orchestrator/vf_utils.py
--- a/src/prime_rl/orchestrator/vf_utils.py
+++ b/src/prime_rl/orchestrator/vf_utils.py
@@ -218,8 +218,7 @@
     env: vf.Environment,
     model_name: str,
     sampling_args: dict,
-    num_examples: int,
-    rollouts_per_example: int,
+    inputs: list,
     clients: list[vf.ClientConfig] | None = None,
     get_client: Callable[[], Awaitable[vf.ClientConfig]] | None = None,
     max_retries: int = DEFAULT_RETRIES,
@@ -232,7 +231,6 @@
           Instead, we use our generate() wrapper which round-robins clients.
 
     """
-    inputs = env._get_eval_inputs(num_examples, rollouts_per_example)
     return await generate(
         env=env,
         clients=clients,

_{This Bugbot Autofix run was free. To enable autofix for future PRs, go to the Cursor dashboard.}