[ROB-321] Restore regression tag on ES multi-cluster evals + wire ES secrets in… - #2179
Merged
Merged
Conversation
…to master workflow The multi-cluster Elasticsearch evals (254/259/260) had their `regression` tag removed in cbaef1a because they aborted eval runs under --strict-setup-mode when ELASTICSEARCH_URL/ELASTICSEARCH_API_KEY were unset. The PR-triggered regression workflow (eval-regression.yaml) had the ES secrets, so the tests passed there. The benchmark workflow got them in cbaef1a. But eval-master.yaml ("Run eval regression on master") still lacked them, so these tests failed setup and aborted the whole master run. - Re-add `regression` tag to 254/259/260. - Add ELASTICSEARCH_URL/ELASTICSEARCH_API_KEY to eval-master.yaml, mirroring eval-regression.yaml and eval-benchmarks.yaml. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> Signed-off-by: avi@robusta.dev <avi@robusta.dev>
Contributor
✅ Results of HolmesGPT evalsAutomatically triggered by commit b87b849 on branch Results of HolmesGPT evals
Benchmark Comparison DetailsMaster baseline: latest master-* experiment (post-merge regression eval)
Benchmark baseline: latest ci-benchmark experiment on master
Time comparison (seconds):
Cost comparison:
Total tokens comparison:
Cached tokens comparison:
Turns comparison:
Tool calls comparison:
Comparison indicators:
📖 Legend
🔄 Re-run evals manually
Option 1: Comment on this PR with Or with more options (one per line): Run evals on a different branch (e.g., master) for comparison:
Quick re-run: Use Option 2: Trigger via GitHub Actions UI → "Run workflow" Option 3: Add PR labels to include extra evals (applies to both automatic runs and
Examples: 🏷️ Valid tags
🤖 Valid models
Commands: CLI: |
Contributor
|
No actionable comments were generated in the recent review. 🎉 ℹ️ Recent review info⚙️ Run configurationConfiguration used: Organization UI Review profile: CHILL Plan: Pro Run ID: 📒 Files selected for processing (4)
WalkthroughThe PR configures Elasticsearch credential environment variables in the master branch regression evaluation workflow and tags three specific Elasticsearch-dependent test fixtures as regression tests to enable automated regression testing with Elasticsearch. ChangesAdd Elasticsearch regression tests
Estimated code review effort🎯 1 (Trivial) | ⏱️ ~5 minutes Possibly related PRs
Suggested reviewers
🚥 Pre-merge checks | ✅ 5✅ Passed checks (5 passed)
✏️ Tip: You can configure your own custom pre-merge checks in the settings. Thanks for using CodeRabbit! It's free for OSS, and your support helps us grow. If you like it, consider giving us a shout-out. Comment |
Contributor
|
✅ Docker images ready for
Use these tags to pull the images for testing. 📋 Copy commandsgcloud auth configure-docker us-central1-docker.pkg.dev
docker pull us-central1-docker.pkg.dev/robusta-development/temporary-builds/holmes:79989c0de
docker tag us-central1-docker.pkg.dev/robusta-development/temporary-builds/holmes:79989c0de me-west1-docker.pkg.dev/robusta-development/development/holmes-dev:79989c0de
docker push me-west1-docker.pkg.dev/robusta-development/development/holmes-dev:79989c0de
docker pull us-central1-docker.pkg.dev/robusta-development/temporary-builds/holmes-operator:79989c0de
docker tag us-central1-docker.pkg.dev/robusta-development/temporary-builds/holmes-operator:79989c0de me-west1-docker.pkg.dev/robusta-development/development/holmes-operator-dev:79989c0de
docker push me-west1-docker.pkg.dev/robusta-development/development/holmes-operator-dev:79989c0dePatch Helm values in one line (choose the chart you use): HolmesGPT chart: helm upgrade --install holmesgpt ./helm/holmes \
--set registry=me-west1-docker.pkg.dev/robusta-development/development \
--set image=holmes-dev:79989c0de \
--set operator.registry=me-west1-docker.pkg.dev/robusta-development/development \
--set operator.image=holmes-operator-dev:79989c0deRobusta wrapper chart: helm upgrade --install robusta robusta/robusta \
--reuse-values \
--set holmes.registry=me-west1-docker.pkg.dev/robusta-development/development \
--set holmes.image=holmes-dev:79989c0de \
--set holmes.operator.registry=me-west1-docker.pkg.dev/robusta-development/development \
--set holmes.operator.image=holmes-operator-dev:79989c0de |
Sheeproid
approved these changes
Jun 11, 2026
✅ Deploy Preview for holmes-docs ready!
To edit notification comments on pull requests, go to your Netlify project configuration. |
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
2 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Fixing regression tests
regressiontag to 254/259/260.Summary by CodeRabbit
Chores
Tests