fix: validate an OpenHands setup before using it - #2387
Conversation
|
/ok to test 406b66a |
|
/claude review |
| "assert Version(version('pyjwt')) >= Version('2.9'); " | ||
| "assert Version(version('sqlalchemy')) >= Version('2.0.40'); " | ||
| "assert Version(version('flask')) >= Version('2.2'); " | ||
| "import datasets, wandb" |
There was a problem hiding this comment.
NOTE: the probe's version floors (jinja2>=3.1.3, pyjwt>=2.9, sqlalchemy>=2.0.40, flask>=2.2) are decoupled from what a rebuild actually installs. openhands.sh never pins these — they come from whatever OpenHands' poetry lock resolves. If the lockfile ever resolves any of them below a floor, _existing_setup_is_reusable returns False on every server startup, so the reuse fast-path is silently disabled and each process pays a full rebuild (which resolves the same sub-floor versions and would fail the probe again next time). No infinite loop (setup() doesn't re-probe after rebuild) and no data corruption — the cost is a permanent, silent loss of the reuse optimization. If these floors encode a known-bad OpenHands version, consider pinning them in openhands.sh alongside the wandb/datasets install so the rebuild actually satisfies what the probe demands.
|
SHIP — startup-only reuse gate, well-tested, off the scoring/async hot paths. This adds a runtime probe to the OpenHands harness Blast-radius check: One non-blocking NOTE inline: the probe's version floors are decoupled from what a rebuild installs — if OpenHands' lock ever resolves a probed package below a floor, the reuse fast-path silently disables itself on every startup (permanent rebuild cost, no corruption). Consider pinning those floors in |
| "assert Version(version('pyjwt')) >= Version('2.9'); " | ||
| "assert Version(version('sqlalchemy')) >= Version('2.0.40'); " | ||
| "assert Version(version('flask')) >= Version('2.2'); " | ||
| "import datasets, wandb" |
There was a problem hiding this comment.
NOTE: the probe's version floors (jinja2>=3.1.3, pyjwt>=2.9, sqlalchemy>=2.0.40, flask>=2.2) are decoupled from what a rebuild actually installs. openhands.sh never pins these — they come from whatever OpenHands' poetry lock resolves. If the lockfile ever resolves any of them below a floor, _existing_setup_is_reusable returns False on every server startup, so the reuse fast-path is silently disabled and each process pays a full rebuild (which then resolves the same sub-floor versions and would fail the probe again). No infinite loop (setup() doesn't re-probe after rebuild) and no data corruption — the cost is a permanent, silent loss of the reuse optimization. If these floors encode a known-bad OpenHands version, consider pinning them in openhands.sh alongside the wandb/datasets install so the rebuild actually satisfies what the probe demands.
406b66a to
f389df0
Compare
f389df0 to
bb6fba8
Compare
bb6fba8 to
858a7ad
Compare
Signed-off-by: Teodor-Dumitru Ene <teodord.ene@gmail.com>
858a7ad to
e5178fa
Compare
The reuse gate for a pre-existing OpenHands setup was 'the venv interpreter file exists', which conflates two very different states with a healthy environment: an interrupted build (a python binary with no packages behind it) and a venv whose packages an episode has since downgraded (episodes run pip inside the sandbox).
Attempting to use pre-existing venv's in either case causes all subsequent episodes to fail and mistakenly return 0 reward.
Nothing ever rebuilds the env on
mainbecause the gate always passes.