feat: add Harbor eval runner

[Raptors65]

Summary

This is another attempt at adding a way to run Terminal-Bench 2.0 on goose locally.

The approach in #9092 mostly worked, but was a little flawed in that goose still kinda thought it was running outside the benchmark container; even though we gave it tools that took action within the container, its system prompt would still think it was running outside the container (i.e., the "current directory" would be one outside the container). Also, the idea of needing to create custom MCP servers for every file tool that we want to give goose as part of a benchmark doesn't seem like the best long-term approach; ideally it should just work, without needing any additional work for new tools.

So here's another approach. I wouldn't say it's strictly better than the previous one, but the nice thing is that we're not doing anything janky to get tool calls to work within the container. Here, we just take in an already built goose Linux executable as input to a script, and use it to run any Harbor task, which includes Terminal-Bench 2.0. The main downside is that this requires a build step.

For now, this is a script that's separate from the CLI, in the evals folder, more in the style of the existing "open model gym". If/when we do want to add this to the CLI as a goose bench command, that should be fairly easy (only thing is we'll need to bundle the Python files into the executable using include_str! or something). Whether that should be done is a different question, which is why I haven't done this as part of this PR. (For example, one could argue benchmarking is purely a dev tool that shouldn't be exposed to end users through the CLI.)

Testing

I'm currently doing a run of Terminal-Bench using Qwen3.6-27B as the model (via OpenRouter), and the tasks are running successfully (the failures that I do see seem to be model failures and not an issue with how we're running the benchmark)

To test this yourself, the following should work:

cd evals/harbor

caffeinate -dimsu uv run ./run \
  --goose-binary <path-to-linux-executable> \
  --goose-profile <path-to-goose-config-with-credentials> \
  --dataset terminal-bench/terminal-bench-2 \
  --model <model> \
  --trials 1 \
  --concurrency 1 \
  --job-name <some-memorable-job-name>

Related Issues

N/A

Screenshots/Demos (for UX changes)

N/A

[Raptors65]

• feat: add Harbor eval runner #9138 👈 (View in Graphite)
• main

This stack of pull requests is managed by Graphite. Learn more about stacking.

[chatgpt-codex-connector]

💡 Codex Review

Here are some automated review suggestions for this pull request.

Reviewed commit: 7f316b345e

ℹ️ About Codex in GitHub

Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you

• Open a pull request for review
• Mark a draft as ready
• Comment "@codex review".

If Codex has suggestions, it will comment; otherwise it will react with 👍.

Codex can also answer questions or update the PR. Try commenting "@codex address that feedback".

[DOsinga]

Hey! The core idea here is solid — the agent adapter in goose_binary.py is the real substance and handles the container setup well (profile routing, non-root ownership, secrets handling).

The main feedback: config.py and its tests feel like generated blubber. You have a 156-line module whose entire job is building a dict and writing it as JSON, with a dataclass that duplicates argparse, single-use 3-line helpers extracted into named functions, a separate validator, a separate writer — all called exactly once. Then 130 lines of tests asserting that the mapping function maps correctly.

Just inline it into runner.py. Parse args, build the dict, write it. That's 30 lines, not a module. Kill the tests that are just asserting the code is implemented the way it's implemented (validation raises ValueError, static method returns a string, write_text writes a file). Keep the tests that cover actual logic: resume detection, mismatch handling, the chown fix, the heredoc avoidance.

Ask yourself for each abstraction: is this called from more than one place? Does it have a branch worth testing? If no to both, inline it.

[chatgpt-codex-connector]

💡 Codex Review

Here are some automated review suggestions for this pull request.

Reviewed commit: aee0a9541e

ℹ️ About Codex in GitHub

Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you

• Open a pull request for review
• Mark a draft as ready
• Comment "@codex review".

If Codex has suggestions, it will comment; otherwise it will react with 👍.

Codex can also answer questions or update the PR. Try commenting "@codex address that feedback".

[chatgpt-codex-connector]

💡 Codex Review

Here are some automated review suggestions for this pull request.

Reviewed commit: 31c1d91ba7

ℹ️ About Codex in GitHub

Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you

• Open a pull request for review
• Mark a draft as ready
• Comment "@codex review".

If Codex has suggestions, it will comment; otherwise it will react with 👍.

Codex can also answer questions or update the PR. Try commenting "@codex address that feedback".

[chatgpt-codex-connector]

💡 Codex Review

Here are some automated review suggestions for this pull request.

Reviewed commit: 8f0c80f83f

ℹ️ About Codex in GitHub

Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you

• Open a pull request for review
• Mark a draft as ready
• Comment "@codex review".

If Codex has suggestions, it will comment; otherwise it will react with 👍.

Codex can also answer questions or update the PR. Try commenting "@codex address that feedback".

[chatgpt-codex-connector]

💡 Codex Review

Here are some automated review suggestions for this pull request.

Reviewed commit: 2ec561a2ff

ℹ️ About Codex in GitHub

Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you

• Open a pull request for review
• Mark a draft as ready
• Comment "@codex review".

If Codex has suggestions, it will comment; otherwise it will react with 👍.

Codex can also answer questions or update the PR. Try commenting "@codex address that feedback".

[chatgpt-codex-connector]

💡 Codex Review

Here are some automated review suggestions for this pull request.

Reviewed commit: f1c3f83e3a

ℹ️ About Codex in GitHub

Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you

• Open a pull request for review
• Mark a draft as ready
• Comment "@codex review".

If Codex has suggestions, it will comment; otherwise it will react with 👍.

Codex can also answer questions or update the PR. Try commenting "@codex address that feedback".

[chatgpt-codex-connector]

💡 Codex Review

Here are some automated review suggestions for this pull request.

Reviewed commit: 245883d186

ℹ️ About Codex in GitHub

Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you

• Open a pull request for review
• Mark a draft as ready
• Comment "@codex review".

If Codex has suggestions, it will comment; otherwise it will react with 👍.

Codex can also answer questions or update the PR. Try commenting "@codex address that feedback".