Skip to content

chore: bump verifiers for the unbounded model-call timeout - #3227

Merged
mikasenghaas merged 1 commit into
mainfrom
chore/bump-verifiers-model-timeout
Aug 9, 2026
Merged

chore: bump verifiers for the unbounded model-call timeout#3227
mikasenghaas merged 1 commit into
mainfrom
chore/bump-verifiers-model-timeout

Conversation

@mikasenghaas

@mikasenghaas mikasenghaas commented Aug 9, 2026

Copy link
Copy Markdown
Member

Bumps deps/verifiers 29e3a0f7a298bcfe, picking up two commits:

commit
a298bcfe fix(v1): restore the unbounded model-call timeout (verifiers#2304)
9ca7f5d6 Fix plain HTTP CONNECT for Docker framework routes (verifiers#2298)

Why

verifiers#2218 consolidated client construction into build_async_openai / DEFAULT_TIMEOUT and, in doing so, replaced the relay client's explicit no-timeout with a 600s read timeout "mirroring the OAI SDK defaults":

-        # No timeout: agentic completions are slow and the rollout timeout is the real backstop.
-            timeout=None,
+DEFAULT_TIMEOUT = httpx.Timeout(connect=5.0, read=600.0, write=600.0, pool=600.0)

That cap reached every model-call path, since DEFAULT_TIMEOUT feeds clients/eval.py as well as build_async_openai, which judge.py and clients/train.py use.

600s is fine for chat-shaped evals and wrong for RL. A single agentic turn on a long SWE trajectory routinely runs past ten minutes; when it does, httpx raises TimeoutException, clients/eval.py maps it to a 504, and the rollout dies as ProviderError: Request timed out — even though the model was still producing. Because it truncates the longest trajectories specifically, the loss is biased rather than uniform.

Observed on the laguna-s SWE RL run on this cluster. verifiers#2304 restores unbounded read/write/pool and keeps connect at 5.0 so an unreachable endpoint still fails fast.

Scope

No lockfile change. verifiers is an editable path dependency (source = { editable = "deps/verifiers" }), and neither commit touches its pyproject.toml, so only the submodule pointer moves. 9ca7f5d6 touches a single file, verifiers/v1/runtimes/docker/egress.py.

🤖 Generated with Claude Code


Note

Medium Risk
Touches every model-call path used in eval, judge, and train clients; the timeout change directly affects RL rollout reliability but fixes a biased failure mode rather than adding new logic.

Overview
Advances the deps/verifiers submodule (29e3a0f7a298bcfe) so this repo picks up two upstream fixes. There is no uv.lock change—verifiers stays an editable path dependency and only the submodule pointer moves.

The main behavioral change is restoring unbounded read/write/pool timeouts on model HTTP clients (with 5s connect kept for fast failure on dead endpoints). That reverses a regression where a shared 600s cap on all OpenAI-style clients was aborting long agentic turns during RL—showing up as TimeoutException / ProviderError: Request timed out on the longest SWE trajectories even while the model was still streaming.

Also included: a small Docker runtime fix for plain HTTP CONNECT on framework egress routes (verifiers/v1/runtimes/docker/egress.py).

Reviewed by Cursor Bugbot for commit 166a0ba. Bugbot is set up for automated code reviews on this repo. Configure here.

Bumps deps/verifiers 29e3a0f7 -> a298bcfe, picking up two commits:

  a298bcfe fix(v1): restore the unbounded model-call timeout (#2304)
  9ca7f5d6 Fix plain HTTP CONNECT for Docker framework routes (#2298)

verifiers#2218 had consolidated client construction into
build_async_openai/DEFAULT_TIMEOUT and, in doing so, replaced the relay
client's explicit `timeout=None` with a 600s read timeout mirroring the OAI
SDK default. That cap reached every model-call path, since DEFAULT_TIMEOUT
feeds clients/eval.py as well as build_async_openai, which judge.py and
clients/train.py use.

600s is fine for chat-shaped evals and wrong for RL: a single agentic turn on
a long SWE trajectory routinely runs past ten minutes, and when it does the
rollout dies as `ProviderError: Request timed out` even though the model was
still producing. Because it truncates the longest trajectories specifically,
the loss is biased rather than uniform. Seen on the laguna-s SWE RL run here.
#2304 restores unbounded read/write/pool and keeps connect at 5.0 so an
unreachable endpoint still fails fast.

No lockfile change: verifiers is an editable path dependency
(`source = { editable = "deps/verifiers" }`), and neither commit touches its
pyproject, so only the submodule pointer moves.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
@mikasenghaas
mikasenghaas marked this pull request as ready for review August 9, 2026 04:40
@mikasenghaas
mikasenghaas requested a review from samsja August 9, 2026 04:40
@mikasenghaas
mikasenghaas merged commit 6e33f3f into main Aug 9, 2026
21 checks passed
@mikasenghaas
mikasenghaas deleted the chore/bump-verifiers-model-timeout branch August 9, 2026 04:41
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants