openenv: raise the tbench2_env contract floor to OpenEnv#1025 - #2811
Merged
Conversation
huggingface/OpenEnv#1025 (38b2a3135) makes a canonical test.sh run that never writes reward.txt a scoring error instead of reward 0.0. Before it, that reply carried the harness marker and looked exactly like tests failing, so the per-episode guard in multi_turn could not drop it; the launcher's source probe is the only place that can catch such an install. - launcher preflight also requires `_require_canonical_verdict` in the installed server source; error message points at #1025 - README / module docstrings: floor moved from #1012 (04d259ea6) to #1025 (38b2a3135) - multi_turn guard comment lists the no-verdict case under the reward=None / error branch The guard itself is unchanged: the new server behavior lands in its existing `reward is None or error` branch, and the harness-marker branch still covers old installs and the pytest fallback. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
![claude[bot]](https://avatars.githubusercontent.com/in/1236702?s=60&v=4){kind=small}
There was a problem hiding this comment.
Claude Code Review
This repository is configured for manual code reviews. Comment @claude review for a one-time review, or @claude review always to subscribe this PR to a review on every future push.
Tip: disable this comment in your organization's Code Review settings.
Contributor
|
@claude review always |
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
2 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Why
huggingface/OpenEnv#1025 (merged 2026-08-29,
38b2a3135) closes the gap @Shi-Dong spotted: a canonicaltests/test.shrun whose verifier never wrotereward.txtused to come back asreward=0.0with theharness="tests/test.sh"marker — byte-for-byte a genuine test failure. Now the server reports it throughobservation.error/reward=None.Nothing on the miles side worked around that bug (nothing could — the reply looked valid), so there is nothing to delete. The fix lands in the existing guard's
reward is None or eval_errorbranch and, in #2802, in theeval_error→no_verdict_reasonmapping; neither changes. Theharness != "tests/test.sh"branch stays: it covers pre-#1012 installs and the pytest fallback, which #1025 did not touch.What
openenv_launch_common.py): the source probe of the installedtbench2_envnow also requires_require_canonical_verdict. A pre-Security: Remote code execution risk fromtrust_remote_code=Truein model bridge loading #1025 install is the one case the per-episode marker guard cannot catch (the fake 0 carries the marker), so the install-time probe is the only place to reject it. All three sandbox backends bake this install into their task images, so one probe covers them.#1012 (04d259ea6)→#1025 (38b2a3135).multi_turn: thereward=None / errorbullet now names "test.sh never wrote reward.txt" alongside toolkit timeout / staging I/O.Not in this PR
--openenv-env-urlcannot be preflighted; it needs a redeploy from a ≥ Security: Remote code execution risk fromtrust_remote_code=Truein model bridge loading #1025 checkout to actually stop emitting the fake 0.trust_remote_code=Truein model bridge loading #1025 routes both "uvx download failed" (infra, droppable) and "in-shelltimeoutfired" (possibly policy-caused) through the same error path, distinguishable only by the log tail. agent functions: one failure contract for the OpenEnv, NeMo Gym and Harbor legs #2802 currently maps everyeval_errortoVerifierError/0; whether miles should split that by error text is a agent functions: one failure contract for the OpenEnv, NeMo Gym and Harbor legs #2802 decision.Validation
No test changes: the guard and its tests are unchanged; the preflight has no unit test today (it imports the real
tbench2_env). Fast tests run in CI (no torch locally).🤖 Generated with Claude Code