chore: bump verifiers for the unbounded model-call timeout - #3227
Merged
Conversation
Bumps deps/verifiers 29e3a0f7 -> a298bcfe, picking up two commits: a298bcfe fix(v1): restore the unbounded model-call timeout (#2304) 9ca7f5d6 Fix plain HTTP CONNECT for Docker framework routes (#2298) verifiers#2218 had consolidated client construction into build_async_openai/DEFAULT_TIMEOUT and, in doing so, replaced the relay client's explicit `timeout=None` with a 600s read timeout mirroring the OAI SDK default. That cap reached every model-call path, since DEFAULT_TIMEOUT feeds clients/eval.py as well as build_async_openai, which judge.py and clients/train.py use. 600s is fine for chat-shaped evals and wrong for RL: a single agentic turn on a long SWE trajectory routinely runs past ten minutes, and when it does the rollout dies as `ProviderError: Request timed out` even though the model was still producing. Because it truncates the longest trajectories specifically, the loss is biased rather than uniform. Seen on the laguna-s SWE RL run here. #2304 restores unbounded read/write/pool and keeps connect at 5.0 so an unreachable endpoint still fails fast. No lockfile change: verifiers is an editable path dependency (`source = { editable = "deps/verifiers" }`), and neither commit touches its pyproject, so only the submodule pointer moves. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
mikasenghaas
marked this pull request as ready for review
August 9, 2026 04:40
samsja
approved these changes
Aug 9, 2026
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Bumps
deps/verifiers29e3a0f7→a298bcfe, picking up two commits:a298bcfe9ca7f5d6Why
verifiers#2218 consolidated client construction into
build_async_openai/DEFAULT_TIMEOUTand, in doing so, replaced the relay client's explicit no-timeout with a 600s read timeout "mirroring the OAI SDK defaults":That cap reached every model-call path, since
DEFAULT_TIMEOUTfeedsclients/eval.pyas well asbuild_async_openai, whichjudge.pyandclients/train.pyuse.600s is fine for chat-shaped evals and wrong for RL. A single agentic turn on a long SWE trajectory routinely runs past ten minutes; when it does,
httpxraisesTimeoutException,clients/eval.pymaps it to a 504, and the rollout dies asProviderError: Request timed out— even though the model was still producing. Because it truncates the longest trajectories specifically, the loss is biased rather than uniform.Observed on the laguna-s SWE RL run on this cluster. verifiers#2304 restores unbounded
read/write/pooland keepsconnectat 5.0 so an unreachable endpoint still fails fast.Scope
No lockfile change. verifiers is an editable path dependency (
source = { editable = "deps/verifiers" }), and neither commit touches itspyproject.toml, so only the submodule pointer moves.9ca7f5d6touches a single file,verifiers/v1/runtimes/docker/egress.py.🤖 Generated with Claude Code
Note
Medium Risk
Touches every model-call path used in eval, judge, and train clients; the timeout change directly affects RL rollout reliability but fixes a biased failure mode rather than adding new logic.
Overview
Advances the
deps/verifierssubmodule (29e3a0f7→a298bcfe) so this repo picks up two upstream fixes. There is nouv.lockchange—verifiers stays an editable path dependency and only the submodule pointer moves.The main behavioral change is restoring unbounded read/write/pool timeouts on model HTTP clients (with 5s connect kept for fast failure on dead endpoints). That reverses a regression where a shared 600s cap on all OpenAI-style clients was aborting long agentic turns during RL—showing up as
TimeoutException/ProviderError: Request timed outon the longest SWE trajectories even while the model was still streaming.Also included: a small Docker runtime fix for plain HTTP CONNECT on framework egress routes (
verifiers/v1/runtimes/docker/egress.py).Reviewed by Cursor Bugbot for commit 166a0ba. Bugbot is set up for automated code reviews on this repo. Configure here.