Skip to content

feat(gym): make per-agent timeout configurable (GYM_AGENT_TIMEOUT) #9791

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Merged
michaelneale merged 1 commit into from
Jun 17, 2026
Merged

feat(gym): make per-agent timeout configurable (GYM_AGENT_TIMEOUT) #9791

Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
1 commit
Select commit
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
4 changes: 4 additions & 0 deletions evals/open-model-gym/README.md
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -283,6 +283,10 @@ npx tsx src/runner.ts --run-count=5

# Don't auto-open browser
npx tsx src/runner.ts --no-open

# Raise the per-agent timeout (seconds) for slow local models on heavy
# scenarios. Default 300s; also settable via GYM_AGENT_TIMEOUT.
npx tsx src/runner.ts --agent-timeout=1200
```

## Output
Expand Down
31 changes: 26 additions & 5 deletions evals/open-model-gym/suite/src/runner.ts
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -66,6 +66,7 @@ interface CacheInputs {
runnerHash: string;
binaryHash: string;
mcpHarnessHash: string;
timeoutMs: number;
}

interface CacheEntry {
Expand All @@ -87,6 +88,21 @@ interface CacheIndex {
entries: Record<string, CacheEntry>;
}

// =============================================================================
// Agent timeout
// =============================================================================
// Per-invocation timeout for an agent run, in milliseconds. Larger local models
// can exceed the old fixed 5-minute cap on heavier scenarios and get killed
// mid-task (recorded as a failure rather than a timeout). Override the cap with
// GYM_AGENT_TIMEOUT (seconds) or --agent-timeout=<seconds>; default 300s.
const AGENT_TIMEOUT_MS = (() => {
const flag = process.argv
.find((a) => a.startsWith("--agent-timeout="))
?.split("=")[1];
const secs = parseInt(flag ?? process.env.GYM_AGENT_TIMEOUT ?? "", 10);
return (Number.isFinite(secs) && secs > 0 ? secs : 300) * 1000;
Comment on lines +102 to +103

@chatgpt-codex-connector chatgpt-codex-connector Bot Jun 14, 2026

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

P2 Badge Include timeout in cache keys

When a slow model first times out at the default 300s, that failed result is stored under the same cache key that will be used after rerunning with GYM_AGENT_TIMEOUT=1200 or --agent-timeout=1200, because computeCacheKey still hashes only scenario/model/runner/binary/MCP inputs. In that cached-failure scenario the new timeout is never exercised unless the user also knows to clear or bypass the cache, so the advertised fix for previously timed-out runs can silently keep returning the old failure.

Useful? React with 👍 / 👎.

})();

// =============================================================================
// Cache Utilities
// =============================================================================
Expand Down Expand Up @@ -180,10 +196,15 @@ function computeCacheKey(pair: TestPair, binaryHashes: Map<string, string>, mcpH
runnerHash,
binaryHash,
mcpHarnessHash,
timeoutMs: AGENT_TIMEOUT_MS,
};

// Combine all into single key
const key = sha256(scenarioHash + modelKey + runnerHash + binaryHash + mcpHarnessHash);
// Combine all into single key. The timeout is included so a result cached
// under a short timeout (e.g. a 300s ETIMEDOUT) isn't reused when the run is
// retried with a larger GYM_AGENT_TIMEOUT.
const key = sha256(
scenarioHash + modelKey + runnerHash + binaryHash + mcpHarnessHash + AGENT_TIMEOUT_MS,
);

return { key, inputs };
}
Expand Down Expand Up @@ -385,7 +406,7 @@ async function runGooseAgent(
GOOSE_PATH_ROOT: GOOSE_ROOT,
MCP_HARNESS_LOG: join(workdir, "tool-calls.log"),
},
timeout: 5 * 60 * 1000,
timeout: AGENT_TIMEOUT_MS,
encoding: "utf-8",
});

Expand Down Expand Up @@ -479,7 +500,7 @@ async function runOpenCodeAgent(
XDG_CONFIG_HOME: OPENCODE_ROOT,
XDG_DATA_HOME: OPENCODE_ROOT,
},
timeout: 5 * 60 * 1000,
timeout: AGENT_TIMEOUT_MS,
encoding: "utf-8",
shell: "/bin/bash",
});
Expand Down Expand Up @@ -644,7 +665,7 @@ async function runPiAgent(
PI_CODING_AGENT_DIR: PI_CONFIG_DIR, // Use isolated config dir
MCP_HARNESS_LOG: join(workdir, "tool-calls.log"),
},
timeout: 5 * 60 * 1000,
timeout: AGENT_TIMEOUT_MS,
encoding: "utf-8",
shell: "/bin/bash",
});
Expand Down