Skip to content
Closed
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension


Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
21 changes: 21 additions & 0 deletions .github/workflows/eval-regression.yaml
Original file line number Diff line number Diff line change
Expand Up @@ -62,6 +62,27 @@ jobs:
GENERATE_REGRESSIONS_FILE: "true"
run: |
poetry run pytest --no-cov tests/llm/test_ask_holmes.py tests/llm/test_investigate.py -s -n10 -m 'llm and easy and not no-cicd'
- name: Add manual rerun instructions
if: steps.check-tests.outputs.should-run == 'true'
shell: bash
run: |
cat <<'EOF' >> evals_report.md
\--- # separator (escaped to avoid YAML document start)

Want to rerun evals with custom filters or models? Comment on this PR with:

```
/run-evals models=gpt-4o markers="llm and easy" keyword=""
```

Defaults: models=gpt-4o, markers="llm and easy", keyword="", iterations=1, workers=6, classifier_model=gpt-4o.

Examples:
- Focus a test: `/run-evals keyword=80_pvc_storage_class_mismatch iterations=2`
- Change models/markers: `/run-evals models="gpt-4o,anthropic/claude-sonnet-4-20250514" markers="llm and medium" workers=4`

See docs/development/evaluations/running-evals.md for details.
EOF
- name: Post evaluation results
if: always()
uses: ./.github/actions/post-eval-comment
Expand Down
261 changes: 261 additions & 0 deletions .github/workflows/manual-evals.yaml
Original file line number Diff line number Diff line change
@@ -0,0 +1,261 @@
name: Run manual evals from PR comments

on:
issue_comment:
types: [created]

permissions:
contents: read
issues: write
pull-requests: write

jobs:
manual-evals:
if: github.event.issue.pull_request != ''
runs-on: ubuntu-latest

steps:
- name: Parse /run-evals command
id: parse
env:
COMMENT_BODY: ${{ github.event.comment.body }}
COMMENT_AUTHOR_ASSOCIATION: ${{ github.event.comment.author_association }}
run: |
python - <<'PY'
import json
import os
import shlex

body = os.environ["COMMENT_BODY"]
association = os.environ["COMMENT_AUTHOR_ASSOCIATION"]
output_path = os.environ["GITHUB_OUTPUT"]

response_message = ""
should_run = False

allowed_associations = {"COLLABORATOR", "MEMBER", "OWNER"}

def set_output(name: str, value: str) -> None:
with open(output_path, "a", encoding="utf-8") as f:
f.write(f"{name}={value}\n")

if not body.lstrip().startswith("/run-evals"):
# Not a trigger comment
set_output("should-run", "false")
set_output("response-message", response_message)
exit(0)

if association.upper() not in allowed_associations:
response_message = (
"Skipping evals: /run-evals is only available to repository members and collaborators."
)
set_output("should-run", "false")
set_output("response-message", response_message)
exit(0)

tokens = shlex.split(body)

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

⚠️ Potential issue | 🟡 Minor

Wrap shlex.split() in a try-except to handle malformed input gracefully.

If a user posts a comment with unclosed quotes (e.g., /run-evals models="gpt-4o), shlex.split() will raise a ValueError, causing the step to fail with an unclear error. Consider catching this and posting a helpful response.

Suggested fix
-          tokens = shlex.split(body)
+          try:
+              tokens = shlex.split(body)
+          except ValueError as e:
+              response_message = f"⚠️ Could not parse command: {e}. Check for unclosed quotes."
+              set_output("should-run", "false")
+              set_output("response-message", response_message)
+              exit(0)
📝 Committable suggestion

‼️ IMPORTANT
Carefully review the code before committing. Ensure that it accurately replaces the highlighted code, contains no missing lines, and has no issues with indentation. Thoroughly test & benchmark the code to ensure it meets the requirements.

Suggested change
tokens = shlex.split(body)
try:
tokens = shlex.split(body)
except ValueError as e:
response_message = f"⚠️ Could not parse command: {e}. Check for unclosed quotes."
set_output("should-run", "false")
set_output("response-message", response_message)
exit(0)
🤖 Prompt for AI Agents
In .github/workflows/manual-evals.yaml around line 56, the call tokens =
shlex.split(body) can raise ValueError for malformed input (e.g., unclosed
quotes); wrap this call in a try/except that catches ValueError, handle it by
posting a clear user-facing response (or logging) explaining the parse error and
how to format the command, and ensure the workflow step exits gracefully (do not
let the exception propagate and fail the job).

defaults = {
"models": "gpt-4o",
"markers": "llm and easy",
"keyword": "",
"iterations": "1",
"test_path": "tests/llm/",
"workers": "6",
"classifier_model": "gpt-4o",
"extra_pytest_args": "",
"upload_dataset": "true",
"run_live": "true",
}

params = defaults.copy()

for token in tokens[1:]:
if "=" not in token:
continue
key, value = token.split("=", 1)
key = key.lower()
if key in params:
params[key] = value

should_run = True
params["comment_command"] = body.strip()
params["extra_pytest_args_json"] = json.dumps(shlex.split(params["extra_pytest_args"]))

set_output("should-run", "true")
for key, value in params.items():
set_output(key, value)
set_output("response-message", response_message)

# Optional debug output
print(f"parsed /run-evals with params: {params}")
PY

- name: Acknowledge non-trigger or unauthorized comment
if: steps.parse.outputs.should-run != 'true' && steps.parse.outputs.response-message != ''
uses: actions/github-script@v7
with:
github-token: ${{ secrets.GITHUB_TOKEN }}
script: |
await github.rest.issues.createComment({
owner: context.repo.owner,
repo: context.repo.repo,
issue_number: context.issue.number,
body: process.env.RESPONSE_MESSAGE,
});
env:
RESPONSE_MESSAGE: ${{ steps.parse.outputs.response-message }}

- name: Checkout
if: steps.parse.outputs.should-run == 'true'
uses: actions/checkout@v4

- name: Announce manual eval start
if: steps.parse.outputs.should-run == 'true'
uses: actions/github-script@v7
with:
github-token: ${{ secrets.GITHUB_TOKEN }}
script: |
const command = process.env.COMMENT_COMMAND || '/run-evals';
await github.rest.issues.createComment({
owner: context.repo.owner,
repo: context.repo.repo,
issue_number: context.issue.number,
body: `<!-- Manual HolmesGPT evals: starting -->\n🚀 Kicking off manual evals for command:\n\`${command}\`\n\nI'll post the full results when the run finishes.`,
});
env:
COMMENT_COMMAND: ${{ steps.parse.outputs.comment_command }}

- name: Setup HolmesGPT environment
if: steps.parse.outputs.should-run == 'true'
uses: ./.github/actions/setup-holmes-env
with:
python-version: '3.12'
install-kubectl: 'true'

- name: Setup KIND cluster
if: steps.parse.outputs.should-run == 'true'
uses: ./.github/actions/setup-kind-cluster
with:
cluster-name: 'kind'
wait-for-ready: 'true'

- name: Run manual evals
if: steps.parse.outputs.should-run == 'true'
id: run-evals
env:
OPENAI_API_KEY: ${{ secrets.OPENAI_API_KEY }}
ANTHROPIC_API_KEY: ${{ secrets.ANTHROPIC_API_KEY }}
AZURE_API_BASE: ${{ secrets.AZURE_API_BASE }}
AZURE_API_KEY: ${{ secrets.AZURE_API_KEY }}
AZURE_API_VERSION: ${{ secrets.AZURE_API_VERSION }}
BRAINTRUST_API_KEY: ${{ secrets.BRAINTRUST_API_KEY }}
MODEL: ${{ steps.parse.outputs.models }}
CLASSIFIER_MODEL: ${{ steps.parse.outputs.classifier_model }}
ITERATIONS: ${{ steps.parse.outputs.iterations }}
RUN_LIVE: ${{ steps.parse.outputs.run_live }}
UPLOAD_DATASET: ${{ steps.parse.outputs.upload_dataset }}
TEST_PATH: ${{ steps.parse.outputs.test_path }}
MARKERS: ${{ steps.parse.outputs.markers }}
KEYWORD: ${{ steps.parse.outputs.keyword }}
WORKERS: ${{ steps.parse.outputs.workers }}
EXTRA_PYTEST_ARGS_JSON: ${{ steps.parse.outputs.extra_pytest_args_json }}
COMMENT_COMMAND: ${{ steps.parse.outputs.comment_command }}
shell: bash
run: |
set -o pipefail
START_TIME="$(date -u +"%Y-%m-%dT%H:%M:%SZ")"
START_EPOCH="$(date -u +%s)"

mapfile -t EXTRA_ARGS < <(python - <<'PY'
import json
import os
raw = os.environ.get("EXTRA_PYTEST_ARGS_JSON") or "[]"
for arg in json.loads(raw):
print(arg)
PY
)

PYTEST_CMD=(poetry run pytest "${TEST_PATH}")
if [[ -n "${KEYWORD}" ]]; then
PYTEST_CMD+=(-k "${KEYWORD}")
fi
if [[ -n "${MARKERS}" ]]; then
PYTEST_CMD+=(-m "${MARKERS}")
fi
PYTEST_CMD+=(--no-cov --tb=short --strict-setup-mode --strict-setup-exceptions=22_high_latency_dbi_down)
if [[ -n "${WORKERS}" && "${WORKERS}" != "0" ]]; then
PYTEST_CMD+=(-n "${WORKERS}")
fi
PYTEST_CMD+=("${EXTRA_ARGS[@]}")

CMD_PRETTY=$(printf '%q ' "${PYTEST_CMD[@]}")

{
"${PYTEST_CMD[@]}"
} 2>&1 | tee manual_eval_output.log
EXIT_CODE=${PIPESTATUS[0]}

END_TIME="$(date -u +"%Y-%m-%dT%H:%M:%SZ")"
END_EPOCH="$(date -u +%s)"
DURATION=$((END_EPOCH - START_EPOCH))

HOURS=$((DURATION/3600))
MINS=$(( (DURATION%3600)/60 ))
SECS=$((DURATION%60))
HUMAN_DURATION=$(printf "%02d:%02d:%02d" "$HOURS" "$MINS" "$SECS")

{
echo "command=${CMD_PRETTY}"
echo "exit-code=${EXIT_CODE}"
echo "start-time=${START_TIME}"
echo "end-time=${END_TIME}"
echo "duration-human=${HUMAN_DURATION}"
echo "duration-seconds=${DURATION}"
} >> "$GITHUB_OUTPUT"

- name: Upload eval log
if: steps.parse.outputs.should-run == 'true'
uses: actions/upload-artifact@v4
with:
name: manual-eval-log-${{ github.run_id }}
path: manual_eval_output.log

- name: Generate manual eval report
if: steps.parse.outputs.should-run == 'true'
run: |
cat > manual_evals_report.md <<EOF
## Manual HolmesGPT evals

- Triggered by: @${{ github.actor }}
- Command: \`${{ steps.parse.outputs.comment_command }}\`
- Test path: \`${{ steps.parse.outputs.test_path }}\`
- Models: \`${{ steps.parse.outputs.models }}\`
- Markers: \`${{ steps.parse.outputs.markers }}\`
- Keyword filter: \`${{ steps.parse.outputs.keyword || '""' }}\`
- Iterations: \`${{ steps.parse.outputs.iterations }}\`
- Workers: \`${{ steps.parse.outputs.workers }}\`
- Classifier model: \`${{ steps.parse.outputs.classifier_model }}\`
- Extra pytest args: \`${{ steps.parse.outputs.extra_pytest_args }}\`
- Started (UTC): ${{ steps.run-evals.outputs.start-time }}
- Finished (UTC): ${{ steps.run-evals.outputs.end-time }}
- Duration: ${{ steps.run-evals.outputs.duration-human }} (${{ steps.run-evals.outputs.duration-seconds }}s)
- Result: ${{ steps.run-evals.outputs.exit-code == '0' && '✅ Success' || '❌ Failed' }}

**Pytest command**
\`\`\`bash
${{ steps.run-evals.outputs.command }}
\`\`\`

**Logs**
- [Download run log](https://github.com/${{ github.repository }}/actions/runs/${{ github.run_id }})

EOF

- name: Post manual eval comment
if: steps.parse.outputs.should-run == 'true'
uses: ./.github/actions/post-eval-comment
with:
github-token: ${{ secrets.GITHUB_TOKEN }}
report-file: 'manual_evals_report.md'
comment-identifier: 'Manual HolmesGPT evals'
delete-previous: 'false'
42 changes: 42 additions & 0 deletions docs/development/evaluations/running-evals.md
Original file line number Diff line number Diff line change
Expand Up @@ -6,6 +6,8 @@ They are used to both catch regressions and measure the impact of new features.

[Example: pod crashloop eval](https://github.com/HolmesGPT/holmesgpt/tree/master/tests/llm/fixtures/test_ask_holmes/09_crashpod).

In CI, a regression slice of the eval suite runs automatically on every pull request to surface failures early.

## Eval Tags

Evals are tagged and grouped into categories. Two common tags are `easy` and `medium`:
Expand Down Expand Up @@ -189,6 +191,46 @@ Speed up test runs with parallel workers:
RUN_LIVE=true ITERATIONS=10 poetry run pytest tests/llm/ -n 10
```

## Triggering evals from PR comments

Every pull request automatically runs a regression slice of the eval suite (see `.github/workflows/eval-regression.yaml`). Use the manual trigger below when you want to re-run evals with different models, markers, or test focus without pushing another commit.

You can launch live evals directly from a pull request using a slash command. This is useful when you want reviewers to see results without pushing extra commits.

**Who can run:** repository members and collaborators (command is ignored for others).

**Command format (defaults in parentheses)**

```
/run-evals models=<comma-separated models> \ # default: gpt-4o
markers="<pytest -m expression>" \ # default: "llm and easy"
keyword="<pytest -k filter>" \ # default: ""
iterations=<N> \ # default: 1
workers=<N> \ # default: 6
classifier_model=<model> \ # default: gpt-4o
extra_pytest_args="--maxfail=1 --disable-warnings" # default: ""
```

Only include the parameters you need—defaults are used for everything else.

**Examples**

```text
# Quick regression sweep
/run-evals models=gpt-4o markers="llm and easy"

# Focus on a specific scenario
/run-evals models="gpt-4o,anthropic/claude-sonnet-4-20250514" keyword=80_pvc_storage_class_mismatch iterations=3 workers=4
```

Each manual run posts a fresh PR comment (no deletions) that includes:

- The exact pytest command used
- Models, markers, and other parameters
- Start/end timestamps and total duration
- Whether the run succeeded or failed
- A link to download the full logs

### Debugging Failed Tests

When tests fail, use these techniques to investigate:
Expand Down
Loading