[eligotts]

Problem

An env-server process can fail during startup or a pool worker can die while serving. Previously, the parent could keep waiting for an address or a reply until a long timeout, leaving the orchestrator stuck at Loading training environments.

Changes

• Report single-server and pooled-server readiness only after serving resources are ready and a worker answers a health probe.
• Poll the child exit code while waiting for its address and include the server log tail in startup errors.
• Reap dead workers, preserve already-queued replies, and fail only their orphaned in-flight requests.
• Keep loading workers out of request dispatch until their health probe succeeds.
• Replace dead workers through a non-fatal spawn path with an independent per-slot recovery budget that resets after readiness.
• Retry missing slots once per reap interval and back off failed elastic scale-ups independently.

Review follow-ups

• Addressed Macroscope's replacement-spawn, queued-reply, and elastic-capacity findings.
• Addressed Bugbot's readiness, failed-spawn, dead-before-reap, recovery-budget, and static-pool findings.
• Kept the request/reply protocol unchanged.

Verification

• uv run pytest tests/test_env_server.py tests/v1/test_configs.py tests/test_imports.py — 29 passed
• uv run ruff check .
• uv run ruff format --check .
• uv run ty check verifiers
• uv run pre-commit run --all-files
• Spawned-process smoke coverage for early child death, single-server readiness, pooled readiness, log-tail propagation, live/ready-only dispatch, and failed scale-up accounting

Companion orchestrator integration: PrimeIntellect-ai/prime-rl#3139.

---

Note

Medium Risk
Changes broker startup, worker lifecycle, and eval server spawn semantics on a critical training/eval path; wire protocol is unchanged but readiness timing and failure modes differ from before.

Overview
Fixes cases where a crashed or still-loading env server left clients and orchestrators waiting indefinitely instead of surfacing an error.

Readiness waits for a real worker. Pooled brokers no longer publish their address at socket bind; they wait until a worker answers a startup health probe (and single-process EnvServer reports after serving() is up). Eval’s server path uses shared wait_for_address, which polls the child exit code and includes log tail on failure, replacing blind address_queue.get plus redundant wait_for_server_startup.

Dead workers are handled in the broker. The serve loop periodically reaps exited workers, drains queued replies before retiring sockets, fails orphaned in-flight requests with BaseResponse(success=False, ...), and replaces workers up to MAX_WORKER_RESTARTS. Mid-serve spawns go through _try_spawn_worker so a failed replacement does not take down the broker; requests with no live workers fail immediately.

^{Reviewed by Cursor Bugbot for commit bb764b6. Bugbot is set up for automated code reviews on this repo. Configure here.}

Note

Fix env-server worker death detection to fail fast instead of hanging

• EnvServerPool now publishes its address only after at least one worker has responded to a health probe, ensuring clients never connect to an unready pool.
• Dead workers are detected by polling exit codes; their in-flight requests receive explicit error replies instead of hanging indefinitely, and replacement workers are spawned up to MAX_WORKER_RESTARTS times with exponential backoff.
• wait_for_address in pool.py polls the process exit code concurrently with the address queue, so startup failures surface immediately with a log tail rather than blocking until timeout.
• server.py delays putting the address on address_queue until the server is fully ready to serve, aligning with the new readiness semantics.
• Behavioral Change: run_eval_server in runner.py replaces a plain address_queue.get(timeout=600) + wait_for_server_startup with a single await wait_for_address(proc, address_queue) that fails fast on worker death.

^{Macroscope summarized 40154d6.}

[macroscopeapp]

Approvability

Verdict: Needs human review

This PR adds substantial worker death detection and replacement infrastructure with new state management and lifecycle logic. An unresolved review comment identifies a medium-severity bug where exhausted restart budgets permanently reduce pool capacity. Both the scope of changes and the open finding warrant human review.

^{You can customize Macroscope's approvability policy. Learn more.}

[cursor]

Cursor Bugbot has reviewed your changes and found 1 potential issue.

^{❌ Bugbot Autofix is OFF. To automatically fix reported issues with cloud agents, enable autofix in the Cursor dashboard.}

^{Reviewed by Cursor Bugbot for commit 40154d6. Configure here.}

[cursor]

Exhausted slots block elastic recovery

Medium Severity

Exhausted restart entries stay in _replacement_queue forever and are counted in _maybe_scale_up via slots. That permanently lowers effective capacity after transient spawn failures, and with an empty pool _on_request returns before scale-up, so even max_workers=None cannot add a fresh worker once budgets are spent.

Additional Locations (2)

• verifiers/v1/serve/pool.py#L296-L318
• verifiers/v1/serve/pool.py#L369-L403

 

^{Reviewed by Cursor Bugbot for commit 40154d6. Configure here.}