fix: rescue lingering finished SWE episodes and mask infra failures

[tdene]

This PR contains two fixes.

For the first, the harness will commonly be left running despite the model having finished the episode. This usually happens due to lingering commands. Currently this leads to a long timeout and failure. Rescuing these episodes from the harness allows them to complete early and retrieve rewards for what would otherwise be marked as a failure.

For the second, infra failures are now tagged so that the training backend can choose to mask out their gradients instead of training with them as 0 reward.

[tdene]

/claude review

[claude]

SHIP — solid, well-tested reliability fix. One NOTE inline; nothing blocking.

The core change is correct: _finish_container_command now polls the OpenHands process instead of a single blocking communicate(), so a finished-but-lingering container (a shell command the model left running) is rescued and scored on its valid output files rather than burning the full timeout and being reported as a failure. What I verified:

• Timeout semantics preserved. The deadline-based loop and the remaining <= 0 → raise TimeoutError reproduce the old wall-clock behavior; the min(_CONTAINER_COMMAND_POLL_INTERVAL_S, remaining) bound prevents overshooting the deadline. Non-agent modes fall through to continue unchanged.
• Rescue path is safe. terminated_after_completion correctly suppresses the SIGKILL-induced non-zero return code at the returncode != 0 check (2556), so a rescued episode isn't misreported as a command failure. Output completeness is gated on full-file json.loads, so a mid-flush partial file waits out the grace window instead of being killed early.
• Fail-loud on empty output. Marker-seen-but-no-output after the grace raises rather than hanging — the right call.
• Masking plumbing is consistent. mask_sample/failure_reason are set on the infra-failure path and OR'd into the downstream mask decision in responses(), then persisted via metrics_to_update. Reward stays 0.0 and mask_sample propagates into SWEBenchVerifyResponse (covered by the updated wrapper test).
• Tests exercise all three branches (missing/complete/late output) with monkeypatched intervals — good coverage of the new logic.

No async-HTTP, ray.get-in-async, or public-API concerns. The one NOTE (inline) is that this new infra-failure path stamps failure_reason = "agent_command_failure" even for timeouts/OOMs, and the downstream if not failure_reason: guard then won't refine it — so the more specific agent_timeout/agent_oom reasons this PR adds are shadowed for those cases. Telemetry granularity only; masking is unaffected.

[claude]

NOTE — failure_reason taxonomy is coarsened for timeouts/OOMs that flow through this path.

WHAT: This except block catches every agent-command failure, including wall-clock timeouts (ValueError("Command timed out")) and watchdog OOMs (the OOM RuntimeError at ~2550). It unconditionally sets metrics.failure_reason = "agent_command_failure" and persists it. Downstream in responses() (~3653), the specific-reason logic is guarded by if not failure_reason:, so it never overwrites — a genuine agent timeout now reports agent_command_failure instead of agent_timeout, and an agent OOM instead of agent_oom.

BLAST RADIUS: Observability only — mask_sample is still correctly True, and agent_timed_out/oom_killed bools remain set, so masking and the boolean signals are unaffected. But the failure_reason taxonomy this PR introduces is partly self-defeating for the two most common failure modes.

FIX: Either set the reason more specifically here (compute after agent_timed_out/watchdog stats), or set it to None/leave unset and let the downstream if not failure_reason: block classify it. Your call — telemetry granularity, not correctness.

[tdene]

Thanks for catching this! Fixed.

[tdene]

/ok to test bbc003e