Skip to content

ci: dynamically select eval agents from changed files #145

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Closed
ralphbean wants to merge 3 commits into from
Closed

ci: dynamically select eval agents from changed files #145

Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
3 commits
Select commit Hold shift + click to select a range
b340292
test(eval): add first happy-path functional test for review agent
ralphbean Jul 10, 2026
3cfdf2b
ci: run review eval in functional-tests workflow
ralphbean Jul 10, 2026
7b39f3f
ci: dynamically select eval agents from changed files
ralphbean Jul 13, 2026
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
310 changes: 310 additions & 0 deletions .github/scripts/select-eval-agents-test.sh
View file
Open in desktop
Original file line number Diff line number Diff line change
@@ -0,0 +1,310 @@
#!/usr/bin/env bash
# Tests for select-eval-agents.sh
set -euo pipefail

SCRIPT_DIR="$(cd "$(dirname "$0")" && pwd)"
SELECT_SCRIPT="${SCRIPT_DIR}/select-eval-agents.sh"
FAILURES=0
TESTS=0

fail() {
echo "FAIL: $1"
FAILURES=$((FAILURES + 1))
}

pass() {
echo "PASS: $1"
}

run_test() {
TESTS=$((TESTS + 1))
}

Comment on lines +1 to +22

@qodo-code-review qodo-code-review Bot Jul 13, 2026

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Remediation recommended

6. New tests not in ci 🐞 Bug ⚙ Maintainability

The PR adds .github/scripts/select-eval-agents-test.sh but it is not invoked by the repo’s existing
make script-test target used in CI. As a result, regressions in select-eval-agents.sh can merge
without the stated unit tests running.
Agent Prompt 
## Issue description
A new unit test script was added but is not wired into the standard script-test target, so CI will not execute it.

## Issue Context
`.github/workflows/script-test.yml` runs `make script-test`; the Makefile target currently doesn’t include the new test.

## Fix Focus Areas
- Makefile[18-28]
- .github/workflows/script-test.yml[35-35]

## Suggested fix
- Add a line to `script-test:`:
  - `$(call run-timed,bash .github/scripts/select-eval-agents-test.sh)`
- Ensure any dependencies the test needs (notably `yq`) are already installed in the script-test workflow (they are today).

ⓘ Copy this prompt and use it to remediate the issue with your preferred AI generation tools

# Create a temporary repo-like structure for each test
setup_fixture() {
local tmpdir
tmpdir="$(mktemp -d)"

# Minimal harness files referencing known paths
mkdir -p "$tmpdir/harness" "$tmpdir/eval/triage/cases" "$tmpdir/eval/review/cases"

cat > "$tmpdir/harness/triage.yaml" << 'YAML'
agent: agents/triage.md
doc: docs/triage.md
policy: policies/triage.yaml
pre_script: scripts/pre-triage.sh
post_script: scripts/post-triage.sh
validation_loop:
script: scripts/validate-output-schema.sh
schema: schemas/triage-result.schema.json
host_files:
- src: common/env/gcp-vertex.env
dest: /sandbox/workspace/.env.d/gcp-vertex.env
- src: env/triage.env
dest: /sandbox/workspace/.env.d/triage.env
- src: ${GOOGLE_APPLICATION_CREDENTIALS}
dest: /tmp/.gcp-credentials.json
skills:
- skills/issue-labels
YAML

cat > "$tmpdir/harness/review.yaml" << 'YAML'
agent: agents/review.md
doc: docs/review.md
policy: policies/review.yaml
pre_script: scripts/pre-review.sh
post_script: scripts/post-review.sh
validation_loop:
script: scripts/validate-output-schema.sh
schema: schemas/review-result.schema.json
host_files:
- src: common/env/gcp-vertex.env
dest: /sandbox/workspace/.env.d/gcp-vertex.env
- src: env/review.env
dest: /sandbox/workspace/.env.d/review.env
- src: ${GOOGLE_APPLICATION_CREDENTIALS}
dest: /tmp/.gcp-credentials.json
skills:
- skills/pr-review
- skills/code-review
plugins:
- plugins/gopls-lsp
YAML

# Agent with no eval config — should never be selected
cat > "$tmpdir/harness/code.yaml" << 'YAML'
agent: agents/code.md
doc: docs/code.md
policy: policies/code.yaml
pre_script: scripts/pre-code.sh
post_script: scripts/post-code.sh
host_files:
- src: env/code.env
dest: /sandbox/workspace/.env.d/code.env
YAML

# Minimal eval configs (just need to exist)
echo "dataset: {}" > "$tmpdir/eval/triage/eval.yaml"
echo "dataset: {}" > "$tmpdir/eval/review/eval.yaml"
# No eval/code/eval.yaml — intentionally missing

echo "$tmpdir"
}

cleanup_fixture() {
rm -rf "$1"
}

# ---------------------------------------------------------------------------
# Test: modifying a harness file selects that agent
# ---------------------------------------------------------------------------
run_test
FIXTURE="$(setup_fixture)"
RESULT=$(echo "harness/triage.yaml" | "$SELECT_SCRIPT" --repo-root "$FIXTURE")
if [[ "$RESULT" == "triage" ]]; then
pass "harness file change selects agent"
else
fail "harness file change selects agent (got: '$RESULT')"
fi
cleanup_fixture "$FIXTURE"

# ---------------------------------------------------------------------------
# Test: modifying env file selects agent that references it
# ---------------------------------------------------------------------------
run_test
FIXTURE="$(setup_fixture)"
RESULT=$(echo "env/triage.env" | "$SELECT_SCRIPT" --repo-root "$FIXTURE")
if [[ "$RESULT" == "triage" ]]; then
pass "env file change selects agent via harness reference"
else
fail "env file change selects agent via harness reference (got: '$RESULT')"
fi
cleanup_fixture "$FIXTURE"

# ---------------------------------------------------------------------------
# Test: modifying shared file selects all agents referencing it
# ---------------------------------------------------------------------------
run_test
FIXTURE="$(setup_fixture)"
RESULT=$(echo "common/env/gcp-vertex.env" | "$SELECT_SCRIPT" --repo-root "$FIXTURE" | sort)
EXPECTED=$(printf "review\ntriage")
if [[ "$RESULT" == "$EXPECTED" ]]; then
pass "shared file change selects all referencing agents"
else
fail "shared file change selects all referencing agents (got: '$RESULT', expected: '$EXPECTED')"
fi
cleanup_fixture "$FIXTURE"

# ---------------------------------------------------------------------------
# Test: modifying agent prompt selects that agent
# ---------------------------------------------------------------------------
run_test
FIXTURE="$(setup_fixture)"
RESULT=$(echo "agents/review.md" | "$SELECT_SCRIPT" --repo-root "$FIXTURE")
if [[ "$RESULT" == "review" ]]; then
pass "agent prompt change selects agent"
else
fail "agent prompt change selects agent (got: '$RESULT')"
fi
cleanup_fixture "$FIXTURE"

# ---------------------------------------------------------------------------
# Test: modifying eval case selects that agent
# ---------------------------------------------------------------------------
run_test
FIXTURE="$(setup_fixture)"
RESULT=$(echo "eval/triage/cases/happy-path.yaml" | "$SELECT_SCRIPT" --repo-root "$FIXTURE")
if [[ "$RESULT" == "triage" ]]; then
pass "eval case change selects agent"
else
fail "eval case change selects agent (got: '$RESULT')"
fi
cleanup_fixture "$FIXTURE"

# ---------------------------------------------------------------------------
# Test: modifying eval config selects that agent
# ---------------------------------------------------------------------------
run_test
FIXTURE="$(setup_fixture)"
RESULT=$(echo "eval/review/eval.yaml" | "$SELECT_SCRIPT" --repo-root "$FIXTURE")
if [[ "$RESULT" == "review" ]]; then
pass "eval config change selects agent"
else
fail "eval config change selects agent (got: '$RESULT')"
fi
cleanup_fixture "$FIXTURE"

# ---------------------------------------------------------------------------
# Test: modifying a file under a skill directory selects the agent
# ---------------------------------------------------------------------------
run_test
FIXTURE="$(setup_fixture)"
RESULT=$(echo "skills/issue-labels/README.md" | "$SELECT_SCRIPT" --repo-root "$FIXTURE")
if [[ "$RESULT" == "triage" ]]; then
pass "skill subpath change selects agent"
else
fail "skill subpath change selects agent (got: '$RESULT')"
fi
cleanup_fixture "$FIXTURE"

# ---------------------------------------------------------------------------
# Test: modifying a plugin directory file selects the agent
# ---------------------------------------------------------------------------
run_test
FIXTURE="$(setup_fixture)"
RESULT=$(echo "plugins/gopls-lsp/init.sh" | "$SELECT_SCRIPT" --repo-root "$FIXTURE")
if [[ "$RESULT" == "review" ]]; then
pass "plugin subpath change selects agent"
else
fail "plugin subpath change selects agent (got: '$RESULT')"
fi
cleanup_fixture "$FIXTURE"

# ---------------------------------------------------------------------------
# Test: modifying files for both agents selects both
# ---------------------------------------------------------------------------
run_test
FIXTURE="$(setup_fixture)"
RESULT=$(printf "env/triage.env\nagents/review.md\n" | "$SELECT_SCRIPT" --repo-root "$FIXTURE" | sort)
EXPECTED=$(printf "review\ntriage")
if [[ "$RESULT" == "$EXPECTED" ]]; then
pass "multiple agents selected from mixed changes"
else
fail "multiple agents selected from mixed changes (got: '$RESULT', expected: '$EXPECTED')"
fi
cleanup_fixture "$FIXTURE"

# ---------------------------------------------------------------------------
# Test: agent without eval config is NOT selected
# ---------------------------------------------------------------------------
run_test
FIXTURE="$(setup_fixture)"
RESULT=$(echo "agents/code.md" | "$SELECT_SCRIPT" --repo-root "$FIXTURE")
if [[ -z "$RESULT" ]]; then
pass "agent without eval config is not selected"
else
fail "agent without eval config is not selected (got: '$RESULT')"
fi
cleanup_fixture "$FIXTURE"

# ---------------------------------------------------------------------------
# Test: unrelated file selects nothing
# ---------------------------------------------------------------------------
run_test
FIXTURE="$(setup_fixture)"
RESULT=$(echo "README.md" | "$SELECT_SCRIPT" --repo-root "$FIXTURE")
if [[ -z "$RESULT" ]]; then
pass "unrelated file selects nothing"
else
fail "unrelated file selects nothing (got: '$RESULT')"
fi
cleanup_fixture "$FIXTURE"

# ---------------------------------------------------------------------------
# Test: shared script referenced by multiple harness files
# ---------------------------------------------------------------------------
run_test
FIXTURE="$(setup_fixture)"
RESULT=$(echo "scripts/validate-output-schema.sh" | "$SELECT_SCRIPT" --repo-root "$FIXTURE" | sort)
EXPECTED=$(printf "review\ntriage")
if [[ "$RESULT" == "$EXPECTED" ]]; then
pass "shared script selects all referencing agents"
else
fail "shared script selects all referencing agents (got: '$RESULT', expected: '$EXPECTED')"
fi
cleanup_fixture "$FIXTURE"

# ---------------------------------------------------------------------------
# Test: forge-level pre/post scripts are also tracked
# ---------------------------------------------------------------------------
run_test
FIXTURE="$(setup_fixture)"
# Add forge scripts to the triage harness
cat >> "$FIXTURE/harness/triage.yaml" << 'YAML'
forge:
github:
pre_script: scripts/forge-pre-triage.sh
post_script: scripts/forge-post-triage.sh
YAML
RESULT=$(echo "scripts/forge-pre-triage.sh" | "$SELECT_SCRIPT" --repo-root "$FIXTURE")
if [[ "$RESULT" == "triage" ]]; then
pass "forge script change selects agent"
else
fail "forge script change selects agent (got: '$RESULT')"
fi
cleanup_fixture "$FIXTURE"

# ---------------------------------------------------------------------------
# Test: schema file change selects agent
# ---------------------------------------------------------------------------
run_test
FIXTURE="$(setup_fixture)"
RESULT=$(echo "schemas/triage-result.schema.json" | "$SELECT_SCRIPT" --repo-root "$FIXTURE")
if [[ "$RESULT" == "triage" ]]; then
pass "schema file change selects agent"
else
fail "schema file change selects agent (got: '$RESULT')"
fi
cleanup_fixture "$FIXTURE"

# ---------------------------------------------------------------------------
# Test: variable-expanded host_files (${VAR}) are ignored
# ---------------------------------------------------------------------------
run_test
FIXTURE="$(setup_fixture)"
RESULT=$(echo '${GOOGLE_APPLICATION_CREDENTIALS}' | "$SELECT_SCRIPT" --repo-root "$FIXTURE")
if [[ -z "$RESULT" ]]; then
pass "variable host_file paths are ignored"
else
fail "variable host_file paths are ignored (got: '$RESULT')"
fi
cleanup_fixture "$FIXTURE"

# ---------------------------------------------------------------------------
# Summary
# ---------------------------------------------------------------------------
echo ""
echo "=== $TESTS tests, $FAILURES failures ==="
if [[ $FAILURES -gt 0 ]]; then
exit 1
fi
Loading
Loading