ci: stop running evals automatically on every commit

[aantn]

Evals were running on every PR commit (eval-regression.yaml on
pull_request synchronize) and every push to master (eval-master.yaml),
which was the main CI eval cost driver even when no one looked at the
results. Switch both to on-demand:

• eval-regression.yaml: trigger only on the labeled pull_request event
  instead of opened/synchronize/reopened/labeled, and skip by default in
  the automatic path unless the PR carries an evals-* label
  (evals-tag-, evals-id-, evals-model-*). /eval comments and
  workflow_dispatch are unaffected, so on-demand runs still work.
• eval-master.yaml: drop the push-to-master trigger; refresh the baseline
  on demand via workflow_dispatch.

The weekly benchmark cron (eval-benchmarks.yaml) is unchanged.

Co-Authored-By: Claude Opus 4.8 noreply@anthropic.com
Claude-Session: https://claude.ai/code/session_01BzLq3Ubqo1TTonzkUASL3X
Signed-off-by: Claude noreply@anthropic.com

Summary by CodeRabbit

• Chores
  • Evaluation workflows changed from automatic execution on every change to opt-in via manual triggers and pull request labels.
  • Baseline evaluation refresh now requires explicit manual workflow dispatch instead of automatically triggering on master branch changes.
  • Pull request evaluation runs now default to skipped, requiring explicit label opt-in or manual dispatch to execute.

[coderabbitai]

Walkthrough

The master baseline eval workflow removes its automatic push-to-master trigger, leaving only workflow_dispatch. The PR regression eval workflow narrows its pull_request trigger to only the labeled event, sets skipEval=true as the default for automatic runs, and refactors label parsing so that evals-tag-*, evals-id-*, and evals-model-* labels explicitly set skipEval=false and configure eval parameters.

Changes

Eval Workflow Opt-in Triggering

Layer / File(s)	Summary
Remove push trigger from master baseline .github/workflows/eval-master.yaml	Deletes the push-to-master trigger (with paths-ignore); the workflow now runs only on workflow_dispatch. Comments updated to reflect on-demand baseline refresh.
Restrict PR trigger and default skipEval to true .github/workflows/eval-regression.yaml	pull_request trigger types narrowed to labeled only. Automatic runs initialize skipEval=true. Label parsing for evals-tag-*, evals-id-*, evals-skip-default, and evals-model-* refactored so each combination explicitly sets skipEval=false and configures markers, filter, and model. The prior standalone evals-skip-default skip branch is removed since skipping is now the default.

Estimated code review effort

🎯 2 (Simple) | ⏱️ ~10 minutes

Possibly related PRs

• HolmesGPT/holmesgpt#1258: Directly modifies eval-regression.yaml eval-triggering and skipEval/opt-in parameter computation logic.
• HolmesGPT/holmesgpt#1599: Modifies eval-regression.yaml to drive eval selection from evals-tag-*/evals-id-* labels and aligns pull_request triggering around the labeled event.
• HolmesGPT/holmesgpt#1821: Modifies eval-regression.yaml eval parameter computation around skipEval/marker construction and evals-skip-default label logic.

Suggested reviewers

• arikalon1

🚥 Pre-merge checks | ✅ 5

✅ Passed checks (5 passed)

Check name	Status	Explanation
Description Check	✅ Passed	Check skipped - CodeRabbit’s high-level summary is enabled.
Title check	✅ Passed	The title clearly and concisely describes the primary change: disabling automatic evaluation workflow runs to reduce CI costs, which aligns with the main objectives and file-level changes in the changeset.
Docstring Coverage	✅ Passed	No functions found in the changed files to evaluate docstring coverage. Skipping docstring coverage check.
Linked Issues check	✅ Passed	Check skipped because no linked issues were found for this pull request.
Out of Scope Changes check	✅ Passed	Check skipped because no linked issues were found for this pull request.

_{✏️ Tip: You can configure your own custom pre-merge checks in the settings.}

---

Thanks for using CodeRabbit! It's free for OSS, and your support helps us grow. If you like it, consider giving us a shout-out.

❤️ Share

• X
• Mastodon
• Reddit
• LinkedIn

_{Comment @coderabbitai help to get the list of available commands.}

[github-actions]

✅ Docker images ready for aa58252b6 (built in 6m 3s)

• us-central1-docker.pkg.dev/robusta-development/temporary-builds/holmes:aa58252b6
• us-central1-docker.pkg.dev/robusta-development/temporary-builds/holmes-operator:aa58252b6

⚠️ Warning: does not support ARM (ARM images are built on release only - not on every PR)

Use these tags to pull the images for testing.

📋 Copy commands

⚠️ Temporary images are deleted after 30 days. Copy to a permanent registry before using them:

gcloud auth configure-docker us-central1-docker.pkg.dev
docker pull us-central1-docker.pkg.dev/robusta-development/temporary-builds/holmes:aa58252b6
docker tag us-central1-docker.pkg.dev/robusta-development/temporary-builds/holmes:aa58252b6 me-west1-docker.pkg.dev/robusta-development/development/holmes-dev:aa58252b6
docker push me-west1-docker.pkg.dev/robusta-development/development/holmes-dev:aa58252b6
docker pull us-central1-docker.pkg.dev/robusta-development/temporary-builds/holmes-operator:aa58252b6
docker tag us-central1-docker.pkg.dev/robusta-development/temporary-builds/holmes-operator:aa58252b6 me-west1-docker.pkg.dev/robusta-development/development/holmes-operator-dev:aa58252b6
docker push me-west1-docker.pkg.dev/robusta-development/development/holmes-operator-dev:aa58252b6

Patch Helm values in one line (choose the chart you use):

HolmesGPT chart:

helm upgrade --install holmesgpt ./helm/holmes \
  --set registry=me-west1-docker.pkg.dev/robusta-development/development \
  --set image=holmes-dev:aa58252b6 \
  --set operator.registry=me-west1-docker.pkg.dev/robusta-development/development \
  --set operator.image=holmes-operator-dev:aa58252b6

Robusta wrapper chart:

helm upgrade --install robusta robusta/robusta \
  --reuse-values \
  --set holmes.registry=me-west1-docker.pkg.dev/robusta-development/development \
  --set holmes.image=holmes-dev:aa58252b6 \
  --set holmes.operator.registry=me-west1-docker.pkg.dev/robusta-development/development \
  --set holmes.operator.image=holmes-operator-dev:aa58252b6

[netlify]

✅ Deploy Preview for holmes-docs ready!

Name	Link
🔨 Latest commit	193db28
🔍 Latest deploy log	https://app.netlify.com/projects/holmes-docs/deploys/6a3a084e57d4230008222b89
😎 Deploy Preview	https://deploy-preview-2218--holmes-docs.netlify.app
📱 Preview on mobile	Toggle QR Code... Use your smartphone camera to open QR code link.

---

To edit notification comments on pull requests, go to your Netlify project configuration.

[coderabbitai]

Actionable comments posted: 1

🤖 Prompt for all review comments with AI agents

Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.

Inline comments:
In @.github/workflows/eval-regression.yaml:
- Around line 7-9: The current workflow triggers on any label being added to a
PR due to the `types: [labeled]` configuration. To prevent unnecessary CI runs
when unrelated labels are added to PRs that already have `evals-*` labels, add a
conditional check that verifies the specific label being added matches the
evaluation label pattern. Use the GitHub Actions `if` condition to check
`github.event.label.name` against the relevant `evals-*` label names, so the
workflow only triggers when the appropriate evaluation label is specifically
added, not on any arbitrary label addition.

🪄 Autofix (Beta)

Fix all unresolved CodeRabbit comments on this PR:

• Push a commit to this branch (recommended)
• Create a new PR with the fixes

---

ℹ️ Review info

⚙️ Run configuration

Configuration used: Organization UI

Review profile: CHILL

Plan: Pro

Run ID: bd7e0c36-0349-4980-b097-e424df309bcb

📥 Commits

Reviewing files that changed from the base of the PR and between 71a253c and 193db28.

📒 Files selected for processing (2)

• .github/workflows/eval-master.yaml
• .github/workflows/eval-regression.yaml