Skip to content
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension


Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
92 changes: 25 additions & 67 deletions .github/workflows/_harness-e2e.yml
Original file line number Diff line number Diff line change
Expand Up @@ -49,8 +49,8 @@ jobs:
timeout-minutes: 45
outputs:
scenarios: ${{ steps.scenarios.outputs.json }}
engine_binary: ${{ steps.lock.outputs.binary }}
engine_revision: ${{ steps.lock.outputs.revision }}
engine_binary: ${{ steps.engine.outputs.binary }}
engine_revision: ${{ steps.engine.outputs.version }}
steps:
- uses: actions/checkout@v5

Expand Down Expand Up @@ -86,78 +86,34 @@ jobs:
[[ -n "$JUDGE_MODEL" ]]
[[ "$JUDGE_PROVIDER" =~ ^[A-Za-z0-9_-]+$ ]]

- name: Read and validate engine.lock
id: lock
- name: Install published iii release
id: engine
env:
GITHUB_TOKEN: ${{ github.token }}
III_INSTALL_URL: https://install.iii.dev/iii/main/install.sh
III_INSTALL_BIN_DIR: ${{ github.workspace }}/target/iii/bin
run: |
set -euo pipefail
lock=harness/tests/integration/engine.lock
read_field() {
local field=$1 values
values=$(sed -n "s/^${field} = \"\\([^\"]*\\)\"$/\\1/p" "$lock")
[[ $(printf '%s\n' "$values" | sed '/^$/d' | wc -l) -eq 1 ]] || {
echo "engine.lock: $field must appear exactly once"
exit 3
}
printf '%s' "$values"
}
invalid=$(sed \
-e '/^[[:space:]]*$/d' \
-e '/^[[:space:]]*#/d' \
-e '/^\(repository\|revision\|package\|binary\) = "[^"]*"$/d' \
"$lock")
[[ -z "$invalid" ]] || {
echo "engine.lock: unsupported or malformed entry"
printf '%s\n' "$invalid"
exit 3
}
repository=$(read_field repository)
revision=$(read_field revision)
package=$(read_field package)
binary=$(read_field binary)
[[ "$repository" =~ ^[A-Za-z0-9_.-]+/[A-Za-z0-9_.-]+$ ]] || {
echo "engine.lock: repository must be an owner/name slug"
exit 3
}
[[ "$revision" =~ ^[0-9a-f]{40}$ ]] || {
echo "engine.lock: revision must be 40-hex"
exit 3
}
[[ "$package" =~ ^[A-Za-z0-9_-]+$ ]] || {
echo "engine.lock: package contains unsupported characters"
exit 3
}
[[ "$binary" =~ ^target/(debug|release)/[A-Za-z0-9_.-]+$ ]] || {
echo "engine.lock: binary must be a target profile path"
exit 3
}
installer="$RUNNER_TEMP/install-iii.sh"
curl -fsSL --retry 3 --retry-all-errors \
"$III_INSTALL_URL" \
-o "$installer"
BIN_DIR="$III_INSTALL_BIN_DIR" sh "$installer"

for binary in iii iii-init iii-worker; do
test -x "$III_INSTALL_BIN_DIR/$binary"
done
version=$("$III_INSTALL_BIN_DIR/iii" --version | awk '{print $NF}')
test -n "$version"
{
echo "repository=$repository"
echo "revision=$revision"
echo "package=$package"
echo "binary=$binary"
echo "binary=target/iii/bin/iii"
echo "version=$version"
} >> "$GITHUB_OUTPUT"
Comment on lines +103 to 111

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

🎯 Functional Correctness | 🟡 Minor | ⚡ Quick win

Hardcoded output path duplicates III_INSTALL_BIN_DIR.

echo "binary=target/iii/bin/iii" re-encodes the same path already defined by III_INSTALL_BIN_DIR (line 94). If that directory ever changes, this literal silently diverges and engine_binary/III_BIN (line 255) will point at the wrong location.

🔧 Derive the output path from the existing variable
       env:
         GITHUB_TOKEN: ${{ github.token }}
         III_INSTALL_URL: https://install.iii.dev/iii/main/install.sh
-        III_INSTALL_BIN_DIR: ${{ github.workspace }}/target/iii/bin
+        III_INSTALL_BIN_DIR: ${{ github.workspace }}/target/iii/bin
+        III_INSTALL_BIN_SUBDIR: target/iii/bin
       run: |
         ...
         {
-          echo "binary=target/iii/bin/iii"
+          echo "binary=$III_INSTALL_BIN_SUBDIR/iii"
           echo "version=$version"
         } >> "$GITHUB_OUTPUT"
🤖 Prompt for AI Agents
Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.

In @.github/workflows/_harness-e2e.yml around lines 103 - 111, Update the output
assignment in the workflow’s binary/version output block to derive the binary
path from the existing III_INSTALL_BIN_DIR variable instead of hardcoding
target/iii/bin/iii. Preserve the existing iii executable name and version output
behavior so downstream engine_binary/III_BIN values remain correct when the
install directory changes.


- name: Checkout pinned engine source
uses: actions/checkout@v5
with:
repository: ${{ steps.lock.outputs.repository }}
ref: ${{ steps.lock.outputs.revision }}
path: target/integration-engine-src

- uses: dtolnay/rust-toolchain@stable
with:
components: clippy

- name: Restore engine build
uses: actions/cache/restore@v4
with:
path: target/integration-engine-src/target
key: integration-engine-${{ runner.os }}-${{ runner.arch }}-${{ steps.lock.outputs.revision }}-${{ hashFiles('target/integration-engine-src/Cargo.lock') }}

- name: Build pinned engine
working-directory: target/integration-engine-src
run: cargo build --locked --release -p ${{ steps.lock.outputs.package }}

- name: Restore E2E Rust cache
uses: Swatinem/rust-cache@v2
with:
Expand Down Expand Up @@ -226,7 +182,9 @@ jobs:
run: |
set -euo pipefail
paths=(
"target/integration-engine-src/${{ steps.lock.outputs.binary }}"
target/iii/bin/iii
target/iii/bin/iii-init
target/iii/bin/iii-worker
database/target/release/database
state/target/release/state
queue/target/release/queue
Expand Down Expand Up @@ -294,7 +252,7 @@ jobs:
ANTHROPIC_API_KEY: ${{ secrets.anthropic_api_key }}
OPENAI_API_KEY: ${{ secrets.openai_api_key }}
ZAI_API_KEY: ${{ secrets.zai_api_key }}
III_BIN: ${{ github.workspace }}/target/integration-engine-src/${{ needs.build.outputs.engine_binary }}
III_BIN: ${{ github.workspace }}/${{ needs.build.outputs.engine_binary }}
HARNESS_E2E_BIN: ${{ github.workspace }}/harness/target/release/harness-e2e
HARNESS_E2E_ARTIFACTS_DIR: ${{ github.workspace }}/target/harness-e2e
HARNESS_E2E_SCENARIO: ${{ matrix.scenario }}
Expand Down
16 changes: 9 additions & 7 deletions harness/tests/e2e/README.md
Original file line number Diff line number Diff line change
Expand Up @@ -158,7 +158,7 @@ The runner writes `results.json` with:

- exact catalog-resolved subject and judge model identity and capabilities;
- effective scenario execution policy;
- judge protocol and pinned engine revision when CI supplies it;
- judge protocol and installed engine version when CI supplies it;
- prompt, transcript, and `harness::metrics` for every run;
- hard-gate results and per-criterion points;
- judge attempts, token usage, and failures grouped by phase;
Expand All @@ -180,12 +180,14 @@ zero.
| Harness E2E Main | Relevant push to `main` | 1 per subject/scenario | Score advisory; hard gates and technical failures blocking |
| Harness E2E Nightly | New `main` revision on schedule, or manual dispatch on `main` | 3 per subject/scenario | Median score, two-of-three pass policy, hard gates and technical failures blocking |

The reusable workflow itself is guarded to run only from `main`. It builds the
pinned engine, the SQLite-backed database worker, and only the provider workers
required by the subject matrix and fixed judge. Scenario ids come directly from
`harness-e2e list`. Each subject/scenario pair receives a fresh stack, and
repetitions run sequentially inside that job with unique table, session, and
state namespaces. At most two matrix jobs make live-model calls concurrently.
The reusable workflow itself is guarded to run only from `main`. It installs
the latest published stable `iii` release with its `iii-init` and `iii-worker`
companions, then builds the SQLite-backed database worker and only the provider
workers required by the subject matrix and fixed judge. Scenario ids come
directly from `harness-e2e list`. Each subject/scenario pair receives a fresh
stack, and repetitions run sequentially inside that job with unique table,
session, and state namespaces. At most two matrix jobs make live-model calls
concurrently.

The scheduled nightly skips the live suite when the current `main` SHA already
has a successful full nightly result. A manual dispatch bypasses this
Expand Down
8 changes: 8 additions & 0 deletions harness/tests/e2e/run-ci.sh
Original file line number Diff line number Diff line change
Expand Up @@ -11,6 +11,14 @@ set -Eeuo pipefail
script_dir=$(cd -- "$(dirname -- "${BASH_SOURCE[0]}")" && pwd)
harness_root=$(cd -- "$script_dir/../.." && pwd)
repo_root=$(cd -- "$harness_root/.." && pwd)
iii_bin_dir=$(cd -- "$(dirname -- "$III_BIN")" && pwd)
for binary in iii iii-init iii-worker; do
[[ -x "$iii_bin_dir/$binary" ]] || {
echo "installed iii companion is not executable: $iii_bin_dir/$binary" >&2
exit 2
}
done
export PATH="$iii_bin_dir:$PATH"
artifacts_dir=${HARNESS_E2E_ARTIFACTS_DIR:-"$repo_root/target/harness-e2e"}
e2e_bin=${HARNESS_E2E_BIN:-"$harness_root/target/release/harness-e2e"}
runs=${HARNESS_E2E_RUNS:-1}
Expand Down
1 change: 1 addition & 0 deletions harness/tests/e2e/stack-config/engine.yaml
Original file line number Diff line number Diff line change
@@ -1,4 +1,5 @@
workers:
- name: iii-worker-ops
- name: iii-worker-manager
config:
host: 127.0.0.1
Expand Down
Loading