Fix OpenCodeEnv client timeout and retry transient sandbox-API polls - #1005
Merged
burtenshaw merged 3 commits intoJul 29, 2026
Merged
Conversation
|
The docs for this PR live here. All of your documentation changes will be reflected on that endpoint. The docs are available until 30 days after the last update. |
![cursor[bot]](https://avatars.githubusercontent.com/in/1210556?s=60&v=4){kind=small}
sergiopaniego
force-pushed
the
fix-hf-sandbox-client-timeout-and-retry
branch
from
July 23, 2026 13:57
017dedc to
cf2e876
Compare
OpenCodeEnv inherited MCPToolClient's 60s message_timeout_s default, but a rollout runs up to 900s server-side, so run_rollout timed out client-side mid-rollout. Add __init__ defaulting message_timeout_s to 1800s, matching PiEnv. HFBgJob.wait() polls Sandbox.processes(), which occasionally drops a single poll with httpx.TransportError (RemoteProtocolError, read timeout). Retry on the next tick and give up only after a run of consecutive failures, so a transient disconnect no longer aborts the rollout.
HFBgJob.wait() retried httpx.TransportError from processes() without checking the deadline, so a finite timeout could be exceeded by up to _MAX_TRANSIENT_POLL_ERRORS * _WAIT_POLL_INTERVAL_S and a blip near the limit raised TransportError instead of the TimeoutError callers rely on. Fall through to the single deadline check + sleep on both the normal poll and the retry.
burtenshaw
force-pushed
the
fix-hf-sandbox-client-timeout-and-retry
branch
from
July 29, 2026 10:23
00e89ac to
cbfe8cd
Compare
Contributor
There was a problem hiding this comment.
Cursor Bugbot has reviewed your changes using default effort and found 1 potential issue.
❌ Bugbot Autofix is OFF. To automatically fix reported issues with cloud agents, enable autofix in the Cursor dashboard.
Reviewed by Cursor Bugbot for commit cbfe8cd. Configure here.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Follow-up to #998, which landed
HFSandboxBackend. Two small robustness fixes surfaced while validating loop-owning training on remote HF sandboxes.1. OpenCodeEnv client timeout
OpenCodeEnvinheritedMCPToolClient's 60smessage_timeout_sdefault, but a rollout runs up to 900s server-side (_RUN_ROLLOUT_TIMEOUT_S). Sorun_rollout()timed out client-side mid-rollout. Adds an__init__defaultingmessage_timeout_sto 1800s, matching the fix already inPiEnv(#999).2. Retry transient sandbox-API polls
HFBgJob.wait()pollsSandbox.processes(), which occasionally drops a single poll withhttpx.TransportError(observedRemoteProtocolError: Server disconnected without sending a response, also read timeouts). Before this, one blip aborted the whole rollout. Now it retries on the next poll tick and gives up only after a run of consecutive failures, so a transient disconnect no longer kills a training step.wait()lives inopencode_env/sandbox/hf.py, shared by both opencode_env and pi_env, so this covers both.Tests
Two new unit tests in
test_opencode_hf_sandbox.py: retry recovers from a few transient poll errors, and a persistent outage re-raises instead of looping forever. Full suite green.Validated end-to-end: loop-owning AsyncGRPO training on HF Jobs (h200x2, opencode on remote HF sandboxes) completes cleanly with the transient poll errors now absorbed.
Note
Low Risk
Operational robustness only—timeout defaults and retry logic on sandbox polling—with unit tests and no changes to auth, rewards, or rollout semantics.
Overview
OpenCodeEnv now defaults
message_timeout_sto 1800s (via an explicit constructor that forwards timeouts toMCPToolClient), so longrun_rolloutcalls are not cut off by the previous ~60s client default while the server allows runs up to ~900s.HFBgJob.wait()in the HF sandbox backend retriesSandbox.processes()when polls fail withhttpx.TransportError, up to 10 consecutive failures, and still raisesTimeoutErrorwhen a finite deadline expires. That shared wait path applies to remote HF sandbox rollouts for both opencode and pi envs.Tests cover session client timeout forwarding and transient vs persistent poll failures.
Reviewed by Cursor Bugbot for commit 7556826. Bugbot is set up for automated code reviews on this repo. Configure here.