diff --git a/evals/harbor/.agents/skills/compare_tasks/SKILL.md b/evals/harbor/.agents/skills/compare_tasks/SKILL.md index 15e186d95dcb..49877caf775a 100644 --- a/evals/harbor/.agents/skills/compare_tasks/SKILL.md +++ b/evals/harbor/.agents/skills/compare_tasks/SKILL.md @@ -40,12 +40,23 @@ jq '{task_name, trial_name}' "$TRIAL_A_DIR/result.json" ### 2. Headline facts -Pull these fields from each trial's `result.json`. The actual shape (harbor -0.8 `TrialResult`): +The fastest path is to let `cmd.py task` do it for you — it already prints +status, reward, duration, tokens, turns, cost, error class, and the tail of +the verifier stdout: + +```bash +./evals/harbor/cmd.py task "$RUN_A" "$TASK" +./evals/harbor/cmd.py task "$RUN_B" "$TASK" +``` + +Only drop to raw `jq` against `result.json` if you need a field `cmd.py task` +doesn't print. The actual shape (harbor 0.8 `TrialResult`): ```bash jq '{ - reward: (.verifier_result.rewards.reward // null), + reward: (.verifier_result.rewards.reward + // (.verifier_result.rewards | to_entries | .[0].value) + // null), rewards_all: .verifier_result.rewards, duration_seconds: ((.finished_at | fromdateiso8601) - (.started_at | fromdateiso8601)), input_tokens: .agent_result.n_input_tokens, @@ -57,6 +68,10 @@ jq '{ }' "$TRIAL_A_DIR/result.json" ``` +The `reward` fallback mirrors `reporter.trial_reward`: if the verifier +didn't use the conventional `reward` key, take the first value in the +`rewards` map. + Derive status from those: - `pass` if `reward >= 1.0` @@ -72,31 +87,62 @@ timed out during teardown, or it timed out after writing the correct answer). If we got points, count them. See `reporter.trial_status` for the canonical rule. -Several `agent_result` fields are commonly `null` for older `GooseBinaryAgent` -runs (notably `n_cache_tokens`, `n_output_tokens`, `cost_usd`). Don't treat -that as a failure — just omit those facts from the comparison if missing on -either side. The reporter has fallbacks that read goose's `complete` event -from `agent/goose.txt`; you don't normally need to replicate them here. +Several `agent_result` fields can be `null` depending on the harness +(notably `n_cache_tokens`, `n_output_tokens`, `cost_usd` on some goose +runs). Don't treat that as a failure — just omit those facts from the +comparison if missing on either side. `cmd.py task` already applies +harbor's fallbacks (reading goose's `complete` event from `agent/goose.txt` +when the structured field is null), so its numbers are the right ones to +report. ### 3. Read the task spec The task definitions are NOT in the harbor Python package. They are plain -text files on disk, in harbor's dataset cache. Do not run `find /` or +text files on disk, in harbor's task cache. Do not run `find /` or `pip show harbor` — that is the wrong direction. -Find the task directory (works on Linux and macOS): +Harbor caches under `~/.cache/harbor/` on every platform (it uses +`Path("~/.cache/harbor").expanduser()` unconditionally — there is no +`~/Library/Caches/harbor` on macOS, despite what you might expect). + +The on-disk layout for package-backed tasks (the common case — everything +in `terminal-bench/terminal-bench-2` lands here) is: + +``` +~/.cache/harbor/tasks/packages//// +``` + +Note: no dataset name in the path. Tasks are keyed by org + task name + +content digest, not by which dataset pulled them. The `` segment +changes when the task is republished, so discover the dir rather than +hardcoding: ```bash -TASK_DIR=$( - ls -d ~/.cache/harbor/datasets/terminal-bench__terminal-bench-2__*/tasks/"$TASK"/ 2>/dev/null \ - || ls -d ~/Library/Caches/harbor/datasets/terminal-bench__terminal-bench-2__*/tasks/"$TASK"/ 2>/dev/null -) +TASK_DIR=$(ls -d ~/.cache/harbor/tasks/packages/terminal-bench/"$TASK"/*/ 2>/dev/null | head -1) echo "$TASK_DIR" ls "$TASK_DIR" ``` -If both lookups return empty, the dataset hasn't been downloaded yet — bail -out and report that, rather than guessing. +If that's empty, the task could be from a different org or a git source — +broaden the search. `find` returns the parent (one level above the +digest), so descend one more level. Guard against `$PARENT` being empty, +otherwise the glob expands to `/*/` and matches the filesystem root: + +```bash +PARENT=$(find ~/.cache/harbor/tasks -type d -name "$TASK" 2>/dev/null | head -1) +if [ -n "$PARENT" ]; then + TASK_DIR=$(ls -d "$PARENT"/*/ 2>/dev/null | head -1) +fi +``` + +If both lookups come up empty, the task hasn't been downloaded on this +machine — bail out and report that, rather than guessing. (Runs sync via +`cmd.py pull` but the task cache does not, so a machine that only inspects +results may never have the spec locally.) + +`~/.cache/harbor/datasets/` exists too but holds dataset-level metadata, +not the per-task `instruction.md` / `tests/` / `solution/` files — not +what you want here. Inside, you care about three files: @@ -115,11 +161,12 @@ Two sources, prefer the first when present: - `$TRIAL_DIR/agent/trajectory.json` — harbor's ATIF format, one entry per agent step. `jq '.steps[] | {step_id, source, message, tool_calls: [.tool_calls[]?.function_name]}'` - gives a compact view. Recent goose runs (after the populate_context_post_run - fix) have this; older `GooseBinaryAgent` runs may not. + gives a compact view. Most current runs have it; some older harness + versions may not. - `$TRIAL_DIR/agent/.txt` — raw stream-json or log. The filename - matches the harness: `goose.txt`, `pi.txt`, `opencode.txt`, - `claude-code.txt`. `ls "$TRIAL_DIR/agent/"` to find it. + matches the harness (commonly `goose.txt` or `pi.txt`; other harnesses + use their own name). Don't guess — run `ls "$TRIAL_DIR/agent/"` and use + whatever `.txt` file is there. Skim, don't quote in full. For each agent identify: @@ -170,10 +217,11 @@ Output markdown with these sections in order: ## Tools you'll need +- `./evals/harbor/cmd.py task ` for the headline numbers - `ls -d` to discover the `__` trial directories -- `jq` for `result.json` +- `jq` for any `result.json` field `cmd.py task` doesn't print - file reads against `$TRIAL_DIR/agent/` and `$TRIAL_DIR/verifier/` -- file reads against the dataset cache (`~/.cache/harbor/datasets/...`) +- `find ~/.cache/harbor/tasks` to locate the task spec No Python imports, no `harbor` package required. Everything you need is on disk as JSON / text files. diff --git a/evals/harbor/recipes/analyze_bench_failure.yaml b/evals/harbor/recipes/analyze_bench_failure.yaml new file mode 100644 index 000000000000..803ced0f9cc2 --- /dev/null +++ b/evals/harbor/recipes/analyze_bench_failure.yaml @@ -0,0 +1,188 @@ +version: 1.0.0 +title: analyze a single harbor benchmark failure +description: compare one task across two runs, theorize why the target failed, propose what could change +author: + contact: douwe@block.xyz + +parameters: + - key: target + input_type: string + requirement: required + description: "the run we want to improve (typically a goose run)" + - key: reference + input_type: string + requirement: required + description: "the run that succeeded on this task" + - key: task + input_type: string + requirement: required + description: "bare task name, e.g. extract-elf (not terminal-bench/extract-elf)" + +extensions: + - type: builtin + name: developer + display_name: Developer + timeout: 600 + bundled: true + description: Core tool for file operations, shell commands, and code analysis + +instructions: analyze why goose (the target run) failed a task that the reference run passed, and suggest what might change in goose to fix it + +prompt: | + you are analyzing a single harbor benchmark task where the reference run + succeeded and the target run (typically goose) failed. the goal is to + form a theory about *why* target failed and suggest what we could change + in goose to fix it. this is analysis, not implementation — no code + changes, no worktrees. + + target run (the one that failed): {{ target }} + reference run (the one that passed): {{ reference }} + task: {{ task }} + + this recipe assumes it is launched from the root of the goose repo + (the current working directory contains `evals/harbor/`). all paths + below are relative to that. + + ## step 1: headline facts + + cmd.py task prints status, reward, duration, tokens, turns, cost, error, + and a tail of the verifier output. start there for both runs: + + ``` + ./evals/harbor/cmd.py task {{ reference }} {{ task }} + ./evals/harbor/cmd.py task {{ target }} {{ task }} + ``` + + ## step 2: find the trial directories + + harbor 0.8 names trial dirs `__`. discover them + from disk — don't guess the suffix: + + ``` + TARGET_DIR=$(ls -d evals/harbor/runs/{{ target }}/{{ task }}__*/ 2>/dev/null | head -1) + REF_DIR=$(ls -d evals/harbor/runs/{{ reference }}/{{ task }}__*/ 2>/dev/null | head -1) + echo "target: $TARGET_DIR" + echo "ref: $REF_DIR" + ``` + + if either is empty the run didn't include this task — stop and report. + + ## step 3: read the task spec + + the task definition lives in harbor's task cache. package-backed tasks + (the common case, including all of terminal-bench-2) land under + `~/.cache/harbor/tasks/packages////`. the digest is + per task version and changes when the task is republished, so discover + the directory rather than guessing: + + ``` + TASK_DIR=$(ls -d ~/.cache/harbor/tasks/packages/terminal-bench/{{ task }}/*/ 2>/dev/null | head -1) + echo "$TASK_DIR" + ls "$TASK_DIR" + ``` + + if that's empty, fall back to a broader search in case the task came + from a git source or a different org. note that `find` returns the + parent (one level above the digest), so descend one more level. guard + against `$PARENT` being empty — otherwise the glob expands to `/*/` and + matches the filesystem root: + + ``` + PARENT=$(find ~/.cache/harbor/tasks -type d -name "{{ task }}" 2>/dev/null | head -1) + if [ -n "$PARENT" ]; then + TASK_DIR=$(ls -d "$PARENT"/*/ 2>/dev/null | head -1) + fi + ``` + + if both come up empty the task isn't cached locally — say so and continue + with what you can learn from the trial dirs alone (the verifier stdout + often reveals what was being checked). + + read these three when present: + + - `instruction.md` — what the agent was asked to do + - `tests/test_outputs.py` or `run-tests.sh` — what the verifier checks + - `solution/solution.sh` — the reference correct answer + + when describing a failure later, **quote the assertion that failed** + rather than paraphrasing — paraphrase is where wrong conclusions sneak in. + + ## step 4: read each agent's trajectory + + two sources per trial, prefer the first: + + - `$TRIAL_DIR/agent/trajectory.json` — harbor's ATIF format, one entry + per agent step. compact view: + `jq '.steps[] | {step_id, source, message, tool_calls: [.tool_calls[]?.function_name]}' "$TRIAL_DIR/agent/trajectory.json"` + - `$TRIAL_DIR/agent/.txt` — raw log. filename varies by harness + (commonly `goose.txt` or `pi.txt`). don't guess; run + `ls "$TRIAL_DIR/agent/"` and use whatever .txt is there. + + for each side identify: + + - the approach the agent took + - the final artifacts it left in the container (files created / modified) + - for the target (the failure), the failure mode — pick one: + - misread the spec (wrong assumption about input/output) + - right approach, shallow bug (off-by-one, wrong encoding, wrong path) + - ran out of clock — but note whether it was making real progress or + thrashing. a thrashing timeout is really a logic failure. + - diverged into an unproductive thread (debugging a non-issue) + - the verifier expected something the spec didn't telegraph + + ## step 5: read the verifier output + + `$TRIAL_DIR/verifier/test-stdout.txt` is usually the most diagnostic + file — it shows exactly which assertion failed and what the agent's + output looked like at that point. + + ``` + tail -80 "$TARGET_DIR/verifier/test-stdout.txt" + ``` + + ## step 6: look at goose source for a theory + + the target is (typically) goose. once you have a failure mode, dig into + the goose source (the current working directory) to see if there's + something there that could plausibly be improved. relevant places + depending on what you saw: + + - `crates/goose/src/agents/` — agent loop, tool-call handling, + context management + - `crates/goose/src/providers/` — provider-specific quirks (prompt + shape, streaming, tool-call format) + - `crates/goose-mcp/src/developer/` — the developer extension, where + most shell/file tools live + - `crates/goose/src/prompts/` and any system-prompt strings — what + we're telling the model about how to behave + - `crates/goose-cli/src/` — cli-side behavior (less likely to matter + for bench) + + use `rg` to search; don't grep the world. if the reference run used a + different harness (e.g. pi, opencode, claude-code), think about what + that harness does differently — sometimes it's just a prompt difference, + sometimes it's a tool-shape difference, sometimes it's a timeout or + retry policy. + + ## step 7: write up the analysis + + produce markdown with these sections: + + - **task** — one-line restatement of what the task wanted + - **outcome** — reference vs target headline (status, reward, duration, + turns) and which assertion the target failed on (quote it) + - **what reference did** — 2–4 sentences on the winning approach + - **what target did** — 2–4 sentences on the losing approach, with the + failure mode named + - **theory** — why target failed in mechanism terms, not vibes. "the + developer extension's text_editor truncates files >2MB and the task + output was 3MB" beats "goose got confused". + - **what we might change in goose** — concrete, but open-ended. could be + a prompt tweak, a tool behavior change, a default config, a new + capability, or "this is a one-off task quirk and not worth chasing". + cite the source files you looked at. it's fine to list more than one + candidate, and fine to say "not sure, would want to look at more + failures with this shape first". + + stop there. no code changes, no PRs, no issues filed. the user will + triage the suggestions across all the tabs once everything has run. diff --git a/evals/harbor/recipes/compare_bench_run.yaml b/evals/harbor/recipes/compare_bench_run.yaml new file mode 100644 index 000000000000..e8f4bb509e83 --- /dev/null +++ b/evals/harbor/recipes/compare_bench_run.yaml @@ -0,0 +1,98 @@ +version: 1.0.0 +title: compare harbor benchmark runs +description: find tasks where one run succeeded but another failed, and fan out per-task analysis +author: + contact: douwe@block.xyz + +parameters: + - key: target + input_type: string + requirement: required + description: "the run we want to improve (typically a goose run)" + - key: reference + input_type: string + requirement: required + description: "the run to learn from (the one that did better)" + +extensions: + - type: builtin + name: developer + display_name: Developer + timeout: 600 + bundled: true + description: Core tool for file operations, shell commands, and code analysis + +instructions: orchestrate per-task analysis of a benchmark regression + +prompt: | + you are comparing two harbor benchmark runs to find tasks where the + `reference` run passed but the `target` run did not, then fanning out + one agent per such task to analyze why. + + target run (the one we want to improve): {{ target }} + reference run (the one to learn from): {{ reference }} + + this recipe assumes it is launched from the root of the goose repo + (i.e. the current working directory contains `evals/harbor/`). all paths + below are relative to that. + + step 1: locate the runs. + + confirm both run directories exist: + + ``` + ls evals/harbor/runs/{{ target }}/ + ls evals/harbor/runs/{{ reference }}/ + ``` + + if either is missing, stop and tell the user. + + step 2: find the divergent tasks. + + use cmd.py compare with -v to get the per-task breakdown: + + ``` + ./evals/harbor/cmd.py compare {{ reference }} {{ target }} -v + ``` + + note the argument order: we pass `reference` as A and `target` as B, so + the "Only A solved" section is exactly the list we want — tasks the + reference passed and the target did not. + + do NOT filter out timeouts. a timeout often masks a real failure — the + agent kept going down a wrong path until the clock ran out. the per-task + analyzer will call out timeouts that are genuinely "right approach, ran + out of clock" versus ones that are really logic failures wearing a + timeout costume. + + step 3: show the user the list and ask for confirmation. + + print the list of tasks that will be analyzed, with the count, and ask + for explicit agreement before launching. mention that each task will get + its own iTerm tab running `analyze_bench_failure.yaml`. + + step 4: on agreement, launch one tab per task. + + for each task in the list, run (substituting `$PWD` and ``): + + ``` + REPO=$PWD + osascript <" + end tell + end tell + APPLESCRIPT + ``` + + the `cd $REPO` matters: new iTerm tabs open in the user's home directory + by default, but the analyzer recipe expects to run from the repo root. + + put a 1 second sleep between launches, otherwise keystrokes may go to + the wrong tab. use the bare task name (e.g. `extract-elf`), not the + qualified form (`terminal-bench/extract-elf`) — the analyzer recipe and + cmd.py both expect the bare name. + + after launching, print a one-line summary of how many tabs you opened.