[gultekinhasancan79]

The eval CLI can wait indefinitely in client.run() when its spawned env-server process exits after the startup health check. A ZMQ peer disappearing does not fail the pending receive. The runner now watches the process it owns and raises an error with the exit code and log path, allowing gather_rollouts() to cancel and drain the remaining work.

Each request races against one shared process watcher. A completed reply keeps precedence; healthy rollouts remain untimed. Cancelled requests are awaited so the existing wire cancellation and client teardown still run.

This addresses the eval-side process-liveness part of #2571. It complements #2131: this change detects exit of the server/broker process owned by the eval, while worker recovery and startup readiness remain separate.

Validation uses a spawned local ZMQ ROUTER and the production _server, EnvClient, and gather_rollouts paths. It makes no model calls. On base f9121b8e4ef8ec284c07961b499f646bcf92fac6, 6 scenarios time out and 3 controls pass; with this patch all 9 pass. Coverage includes exit codes 0 and 23, multiple in-flight requests, semaphore-queued work, completed episodes, ordinary request errors, and caller cancellation.

The focused probe is temporary, following AGENTS.md. A minimal standalone reproduction is included below.
Contributor checks (Linux, Python 3.12.13):

• UV_NO_SYNC=1 uv run pytest tests/: 86 passed, 77 skipped (live tests require PRIME_API_KEY).
• UV_NO_SYNC=1 uv run ruff check --fix .: passed.
• UV_NO_SYNC=1 uv run ty check verifiers: passed.
• UV_NO_SYNC=1 uv run pre-commit run --all-files: passed.

Dependencies came from the unchanged uv.lock, exported with uv export --frozen and installed with pip's hash verification after uv downloads failed in the test container. UV_NO_SYNC=1 preserves that installed environment. No dependency or lockfile changes are included.

Model-free reproduction

Save as repro_eval_server_exit.py outside the checkout, then run it from an installed checkout with uv run --no-sync python /path/to/repro_eval_server_exit.py.

Base: the outer 5-second safety timeout reports BUG. Patched: a RuntimeError reports the server's exit code 23 and the script reports PASS. Only the spawned server target is replaced with a small local ROUTER; the eval runner and client are production code.

"""uv run --frozen python /path/to/repro_eval_server_exit.py

No model, API key, or external service is used. Only the spawned server target
is replaced; _server, RunSlot, and EnvClient execute their production code.
"""

import asyncio
import os
from unittest.mock import patch

import msgpack
import zmq

from verifiers.v1.cli.eval.runner import _server
from verifiers.v1.configs.cli.eval import EvalConfig
from verifiers.v1.env import RunSlot
from verifiers.v1.task import Task, TaskData


def serve_until_run(*, address_queue, **kwargs):
    context = zmq.Context()
    socket = context.socket(zmq.ROUTER)
    socket.bind("tcp://127.0.0.1:0")
    address_queue.put(socket.getsockopt_string(zmq.LAST_ENDPOINT))
    while True:
        client, request, method, payload = socket.recv_multipart()
        if method == b"run":
            # Terminate only this deliberately spawned test server.
            os._exit(23)
        socket.send_multipart(
            [client, request, msgpack.packb({"success": True})]
        )


async def main():
    config = EvalConfig(push=False, rich=None)

    async def completed(episode):
        raise AssertionError("a dead server must not publish a completed episode")

    with patch("verifiers.v1.serve.serve_env", serve_until_run):
        async with _server(config, config.serve, None, completed) as run:
            try:
                await asyncio.wait_for(run(RunSlot(Task(TaskData(prompt="probe")))), 5)
            except TimeoutError:
                print("BUG: the server exited with code 23; the run stayed pending for 5s")
                return 1
            except RuntimeError as error:
                if "exit code 23" not in str(error):
                    raise
                print(f"PASS: {error}")
                return 0
    raise AssertionError("the run unexpectedly succeeded")


if __name__ == "__main__":
    raise SystemExit(asyncio.run(main()))

Note

Fail served rollouts when environment-server process exits

• Adds a wait_for_exit coroutine that polls the spawned environment-server process and completes when it terminates.
• The rollout runner now races the client request against this monitor; if the server exits first, it raises a runtime error with the exit code and log path.
• A rollout response that completes concurrently with termination is still accepted.
• Behavioral Change: previously, an unexpected server-process exit could leave rollout requests unresolved indefinitely; runner.py _server.run now cancels and awaits the request task on all exit paths.

^{Macroscope summarized bc2cf6f.}

[gultekinhasancan79]

Closing this: #2512 made vf-eval in-process only and removed _server, so the code this patched no longer exists on main. The worker-death part of #2571 is still open, and #2131 proposes a broker-side fix for it.

One related gap: prime-rl's new uv run eval launcher starts its env servers but doesn't watch them the way uv run rl and uv run sft do. EnvClient.run has no timeout, so if an env-server process exits mid-run the eval waits indefinitely. If it's useful, I'm happy to open a small draft PR there that fails the eval with the server's exit code and log path. It would only catch the server process exiting, not a pool worker dying behind a live broker.