Skip to content

fix: validate an OpenHands setup before using it - #2387

Open
tdene wants to merge 1 commit into
NVIDIA-NeMo:mainfrom
tdene:tde/openhands_env_validation
Open

fix: validate an OpenHands setup before using it#2387
tdene wants to merge 1 commit into
NVIDIA-NeMo:mainfrom
tdene:tde/openhands_env_validation

Conversation

@tdene

@tdene tdene commented Aug 6, 2026

Copy link
Copy Markdown
Contributor

The reuse gate for a pre-existing OpenHands setup was 'the venv interpreter file exists', which conflates two very different states with a healthy environment: an interrupted build (a python binary with no packages behind it) and a venv whose packages an episode has since downgraded (episodes run pip inside the sandbox).

Attempting to use pre-existing venv's in either case causes all subsequent episodes to fail and mistakenly return 0 reward.

Nothing ever rebuilds the env on main because the gate always passes.

@tdene

tdene commented Aug 6, 2026

Copy link
Copy Markdown
Contributor Author

/ok to test 406b66a

@github-actions github-actions Bot added the sla:triage-overdue Review assignment is over the one-business-day SLA label Aug 7, 2026
@tdene

tdene commented Aug 7, 2026

Copy link
Copy Markdown
Contributor Author

/claude review

"assert Version(version('pyjwt')) >= Version('2.9'); "
"assert Version(version('sqlalchemy')) >= Version('2.0.40'); "
"assert Version(version('flask')) >= Version('2.2'); "
"import datasets, wandb"

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

NOTE: the probe's version floors (jinja2>=3.1.3, pyjwt>=2.9, sqlalchemy>=2.0.40, flask>=2.2) are decoupled from what a rebuild actually installs. openhands.sh never pins these — they come from whatever OpenHands' poetry lock resolves. If the lockfile ever resolves any of them below a floor, _existing_setup_is_reusable returns False on every server startup, so the reuse fast-path is silently disabled and each process pays a full rebuild (which resolves the same sub-floor versions and would fail the probe again next time). No infinite loop (setup() doesn't re-probe after rebuild) and no data corruption — the cost is a permanent, silent loss of the reuse optimization. If these floors encode a known-bad OpenHands version, consider pinning them in openhands.sh alongside the wandb/datasets install so the rebuild actually satisfies what the probe demands.

@claude

claude Bot commented Aug 7, 2026

Copy link
Copy Markdown
Contributor

SHIP — startup-only reuse gate, well-tested, off the scoring/async hot paths.

This adds a runtime probe to the OpenHands harness setup() so a stale .venv (missing wandb, or with dep versions below required floors) is rebuilt instead of reused — previously a corpse venv would be republished and fail every subsequent episode after the agent had done its real work. Good defensive change.

Blast-radius check: _existing_setup_is_reusable/_probe_openhands_venv run once per server process in model_post_initsetup(), under file_lock — not per episode, not on verify() or any score/reward path. Worst case is a wasted rebuild at startup, never silent score corruption or an async hang. subprocess_run here is the same synchronous helper used elsewhere in this file's setup paths. The empty-stderr fallback (" ".join(...[-1:])""f"exit code {...}") is handled. Tests cover the three-way gate (no setup / healthy / failing probe) and the stderr-reporting path.

One non-blocking NOTE inline: the probe's version floors are decoupled from what a rebuild installs — if OpenHands' lock ever resolves a probed package below a floor, the reuse fast-path silently disables itself on every startup (permanent rebuild cost, no corruption). Consider pinning those floors in openhands.sh if they encode a known-bad version.

"assert Version(version('pyjwt')) >= Version('2.9'); "
"assert Version(version('sqlalchemy')) >= Version('2.0.40'); "
"assert Version(version('flask')) >= Version('2.2'); "
"import datasets, wandb"

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

NOTE: the probe's version floors (jinja2>=3.1.3, pyjwt>=2.9, sqlalchemy>=2.0.40, flask>=2.2) are decoupled from what a rebuild actually installs. openhands.sh never pins these — they come from whatever OpenHands' poetry lock resolves. If the lockfile ever resolves any of them below a floor, _existing_setup_is_reusable returns False on every server startup, so the reuse fast-path is silently disabled and each process pays a full rebuild (which then resolves the same sub-floor versions and would fail the probe again). No infinite loop (setup() doesn't re-probe after rebuild) and no data corruption — the cost is a permanent, silent loss of the reuse optimization. If these floors encode a known-bad OpenHands version, consider pinning them in openhands.sh alongside the wandb/datasets install so the rebuild actually satisfies what the probe demands.

@github-actions github-actions Bot removed the sla:triage-overdue Review assignment is over the one-business-day SLA label Aug 7, 2026
@tdene
tdene force-pushed the tde/openhands_env_validation branch from 406b66a to f389df0 Compare August 18, 2026 10:59
@tdene
tdene force-pushed the tde/openhands_env_validation branch from f389df0 to bb6fba8 Compare August 21, 2026 15:16
@tdene
tdene force-pushed the tde/openhands_env_validation branch from bb6fba8 to 858a7ad Compare August 25, 2026 08:56
Signed-off-by: Teodor-Dumitru Ene <teodord.ene@gmail.com>
@tdene
tdene force-pushed the tde/openhands_env_validation branch from 858a7ad to e5178fa Compare September 1, 2026 11:32
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant