opencode_env: run setup before the agent, skip verify on non-zero agent exit - #1023
Conversation
|
The docs for this PR live here. All of your documentation changes will be reflected on that endpoint. The docs are available until 30 days after the last update. |
![cursor[bot]](https://avatars.githubusercontent.com/in/1210556?s=60&v=4){kind=small}
9975794 to
e7aadc8
Compare
There was a problem hiding this comment.
Cursor Bugbot has reviewed your changes using default effort and found 1 potential issue.
There are 2 total unresolved issues (including 1 from previous review).
❌ Bugbot Autofix is OFF. To automatically fix reported issues with cloud agents, enable autofix in the Cursor dashboard.
Reviewed by Cursor Bugbot for commit e7aadc8. Configure here.
| _emit(f"setup FAILED at [{i}]: exit={cr.exit_code}") | ||
| break | ||
|
|
||
| # Block until the agent is done (or setup already failed). |
There was a problem hiding this comment.
Timeouts ignore sequential setup cost
Medium Severity
Deferring the agent until after setup makes wall-clock time setup + agent instead of overlapping them, but sandbox lifetime stays agent_timeout_s + 300 and the MCP tool cap stays 900s. With default agent_timeout_s=600 and SETUP_TIMEOUT_S=300, longer setups can kill the sandbox or trip the tool timeout before the agent finishes.
Additional Locations (2)
Reviewed by Cursor Bugbot for commit e7aadc8. Configure here.
run_rollout ran setup after create() had already started the agent, so setup raced the agent over the workspace. create() now takes start_agent (default True, so the loop-owning training path is unchanged), the server passes start_agent=False, runs setup, then calls session.start_agent(), giving the correct setup then agent then verify order. A non-zero agent exit is now treated like a timeout, so a crashed agent no longer runs verify or earns a reward. AI-assisted.
Without it the run pipeline reports tee's exit code (0), so a non-zero opencode exit is masked and the failed-agent guard never fires. Matches the pi runtime, which already sets pipefail.
988f672 to
800c922
Compare
There was a problem hiding this comment.
Pull request overview
Copilot reviewed 4 out of 4 changed files in this pull request and generated no new comments.
Suppressed comments (1)
envs/opencode_env/harness.py:234
- The create() docstring still states that session creation "starts the proxy + agent", but this is no longer always true when start_agent=False (and the proxy itself is mode-dependent). This makes the docstring misleading for callers relying on deferred agent startup.
Session creation spins up a sandbox, installs opencode, and starts the
proxy + agent, the flakiest step in a rollout. Each failed attempt tears
its own sandbox down (see :meth:`_create_once`), so a retry never leaks.
What does this PR do?
Two usability fixes to
opencode_env'srun_rollout, from review on #999.run_rolloutran the setup commands aftercreate(), which had already started the agent, so setup and the agent raced over the workspace.create()now takesstart_agent(defaultTrue, so the loop-owning training path that callscreate()directly is unchanged). The server passesstart_agent=False, runs setup, then callssession.start_agent(), giving the correct setup then agent then verify order.agent_exit_codeis now treated like a timeout (setsresult.error), so verify and reward are skipped for a crashed agent.The same fix for
pi_envis on #999.AI-assisted.
Note
Medium Risk
Changes rollout ordering and reward gating in the MCP server path; default create() behavior is preserved but agent exit handling now affects whether verify runs.
Overview
run_rolloutnow runs task setup before the OpenCode agent starts, instead of racing setup against an agent thatcreate()had already launched.OpenCodeSessionFactory.create()gains an optionalstart_agentflag (default true, so direct training/harness callers are unchanged); the MCP server passesstart_agent=False, runs each setup command, then callssession.start_agent().Failed agent runs are no longer scored. A non-zero
agent_exit_codesetsresult.error(like a timeout), so verify commands and reward computation are skipped on a crashed agent.build_run_cmdaddsset -o pipefailso a failingopencodeinside the| teepipeline surfaces as a non-zero exit code. Factory lifecycle tests were updated for the newstart_agentparameter.Reviewed by Cursor Bugbot for commit 800c922. Bugbot is set up for automated code reviews on this repo. Configure here.