[morluto]

Description

Fixes #1991.

The v1 env-server broker now registers each worker's process sentinel with its existing PyZMQ poller. When a worker exits, the broker:

• drains replies already queued by that worker;
• returns an explicit failure for its remaining requests;
• releases their rollout-slot accounting;
• closes the dead worker's broker-side pipe, socket, process handle, and IPC path;
• starts a replacement in the same worker slot.

Requests are failed rather than transparently replayed because a rollout may have made model calls or other external changes before its worker exited.

Hard-shutdown ownership of worker subprocess trees is tracked separately in #1992. This change does not claim to reclaim descendants or remote resources after SIGKILL.

Type of Change

• Bug fix (non-breaking change which fixes an issue)
• New feature (non-breaking change which adds functionality)
• Breaking change (fix or feature that would cause existing functionality to not work as expected)
• Documentation update
• Test improvement

Testing

• All existing tests pass when running uv run pytest locally.
• New tests have been added to cover the changes

Validation performed:

• uv run ruff check --fix . — passed
• focused worker-death proof — the base timed out; this branch returned env server worker exited unexpectedly (exit code -9), replaced the worker, served a subsequent request, and kept the broker alive
• uv run pytest tests/v1/test_configs.py tests/v1/test_taskset.py tests/test_env_server.py — 32 passed
• uv run pytest tests/ — 956 passed, 148 skipped, 6 unrelated failures: four from the incomplete environments/hello_mcp_harbor directory and two environment-load checks requiring OpenAI credentials
• uv run pre-commit run --all-files — Ruff passed; the generated-guide check reported pre-existing stale environments/AGENTS.md outputs
• focused cleanup-cancellation proof — PR head 7d07537a8 lost dead-worker cleanup ownership when cancellation interrupted the final-reply relay; this branch kept cleanup under _shutdown ownership and restored a replacement before an interrupted failure send
• autoreview against origin/main — clean, with no accepted or actionable findings

Checklist

• My code follows the style guidelines of this project as outlined in AGENTS.md
• I have performed a self-review of my own code
• I have commented my code, particularly in hard-to-understand areas
• I have made corresponding changes to the documentation
• My changes generate no new warnings
• Any dependent changes have been merged and published

Additional Notes

Suggested review order:

1. _spawn_worker() sentinel registration
2. _drain_worker_replies() and _worker_died() lifecycle handling
3. run() ordering for replies, worker exits, and new frontend requests

---

Note

Medium Risk
Changes broker request lifecycle and client-visible errors on worker crash; behavior is intentional but affects rollout reliability and in-flight accounting in the env-server pool.

Overview
The v1 EnvServerPool broker now watches each worker’s process sentinel on the existing ZMQ poller and recovers when a worker dies instead of leaving in-flight rollouts stuck.

On exit it drains any replies already queued on that worker’s DEALER, fails remaining requests with an explicit BaseResponse error (no silent replay), releases rollout-slot accounting, tears down IPC/socket/pipe state, and spawns a replacement in the same worker index. The main loop handles dealer replies before sentinel exits so a final successful reply is not treated as a failure.

_spawn_worker accepts an optional index for slot-preserving restarts; workers register both dealer and sentinel with the poller.

^{Reviewed by Cursor Bugbot for commit 7d07537. Bugbot is set up for automated code reviews on this repo. Configure here.}

Note

Recover env pool after worker exit in EnvServerPool broker

• Adds process sentinel tracking to _spawn_worker so the broker can detect when a worker process dies via poller events.
• Adds _drain_worker_replies to non-blockingly flush queued replies from a dead worker's DEALER socket before failing remaining requests.
• Adds _worker_died to join the dead process, drain remaining replies, send BaseResponse(success=False) failure responses to all pending clients, and immediately respawn a replacement worker at the same pool index.
• Reworks the broker poll loop in pool.py to drain replies first, then handle sentinel-triggered worker exits.
• Risk: pending requests owned by a dead worker receive a synthetic failure response; any in-flight work that was not yet queued on the socket is lost.

^{Macroscope summarized 767add9.}

[chatgpt-codex-connector]

You have reached your Codex usage limits for code reviews. You can see your limits in the Codex usage dashboard.

[cursor]

Cursor Bugbot has reviewed your changes and found 1 potential issue.

^{❌ Bugbot Autofix is OFF. To automatically fix reported issues with cloud agents, enable autofix in the Cursor dashboard.}

Want reviews to match your repository better? Bugbot Learning can learn team-specific rules from PR activity. A team admin can enable Learning in the Cursor dashboard.

^{Reviewed by Cursor Bugbot for commit 7d07537. Configure here.}

[xeophon]

superseded by #2518