Add Coralogix API keys to eval regression workflow

[aantn]

Summary

Added Coralogix API credentials to the eval regression GitHub Actions workflow to enable integration with Coralogix logging and monitoring services.

Changes

• Added CORALOGIX_API_KEY secret to the workflow environment
• Added CORALOGIX_SEND_API_KEY secret to the workflow environment

Details

These environment variables are now available during the eval regression job execution, allowing the regression tests to authenticate with Coralogix services for logging and data transmission. The secrets are already configured in the repository settings and are now being passed through to the workflow.

https://claude.ai/code/session_01XaJtNp4ywbsovnw2cr3hoz

Summary by CodeRabbit

• Chores
  • Updated automated testing infrastructure configuration to support additional external services during evaluation runs.

_{✏️ Tip: You can customize this high-level summary in your review settings.}

[netlify]

✅ Deploy Preview for holmes-docs ready!

Name	Link
🔨 Latest commit	6f2277d
🔍 Latest deploy log	https://app.netlify.com/projects/holmes-docs/deploys/6979fe41730ab30009d73e93
😎 Deploy Preview	https://deploy-preview-1442--holmes-docs.netlify.app
📱 Preview on mobile	Toggle QR Code... Use your smartphone camera to open QR code link.

---

To edit notification comments on pull requests, go to your Netlify project configuration.

[linux-foundation-easycla]

The committers listed above are authorized under a signed CLA.

• ✅ login: aantn / name: Natan Yellin (6f2277d)

[github-actions]

📂 Previous Runs

📜 Run @ 749ea9f (#21436364576)

✅ Results of HolmesGPT evals

Automatically triggered by commit 749ea9f on branch claude/add-coralogix-api-key-TfwLo

View workflow logs

Results of HolmesGPT evals

• ask_holmes: 9/9 test cases were successful, 0 regressions

Status	Test case	Time	Turns	Tools	Cost
✅	09_crashpod	29.1s ±0%	5	10	$0.2173
✅	101_loki_historical_logs_pod_deleted	53.5s ±0%	7	15	$0.3074
✅	111_pod_names_contain_service	23.3s ±0%	4	7	$0.1787
✅	12_job_crashing	35.9s ±0%	5	12	$0.2418
✅	162_get_runbooks	36.5s ±0%	6	11	$0.2643
✅	176_network_policy_blocking_traffic_no_runbooks	50.1s ↑11%	7	18	$0.3032
✅	24_misconfigured_pvc	29.7s ↓18%	5	12	$0.2169
✅	43_current_datetime_from_prompt	3.7s ±0%	1	—	$0.1019
✅	61_exact_match_counting	16.2s ±0%	4	4	$0.1592
	Total	30.9s avg	4.9 avg	11.1 avg	$1.9908

Time/Cost columns show % change vs historical average (↑slower/costlier, ↓faster/cheaper). Changes under 10% shown as ±0%.

Historical Comparison Details

Filter: excluding branch 'claude/add-coralogix-api-key-TfwLo'

Status: Success - 59 test/model combinations loaded

Experiments compared (30):

• github-21430741137.2607.1 (branch: automated/benchmark-20260127_161122)
• github-21429864375.2606.1 (branch: master)
• github-21429729741.2605.1 (branch: deepseek-benchmark)
• ...and 27 more

Comparison indicators:

• ±0% — diff under 10% (within noise threshold)
• ↑N%/↓N% — diff 10-25%
• ↑N%/↓N% — diff over 25% (significant)

---

✅ Results of HolmesGPT evals

Automatically triggered by commit 6f2277d on branch claude/add-coralogix-api-key-TfwLo

View workflow logs

Results of HolmesGPT evals

• ask_holmes: 9/9 test cases were successful, 0 regressions

Status	Test case	Time	Turns	Tools	Cost
✅	09_crashpod	36.1s ↑22%	6	12	$0.2487
✅	101_loki_historical_logs_pod_deleted	50.7s ↓12%	7	13	$0.2974
✅	111_pod_names_contain_service	23.6s ±0%	4	7	$0.1805
✅	12_job_crashing	40.5s ↑12%	6	16	$0.2788
✅	162_get_runbooks	41.4s ±0%	6	11	$0.2899
✅	176_network_policy_blocking_traffic_no_runbooks	49.1s ±0%	7	19	$0.3153
✅	24_misconfigured_pvc	35.3s ±0%	6	13	$0.2371
✅	43_current_datetime_from_prompt	3.6s ↓15%	1	—	$0.1019
✅	61_exact_match_counting	16.1s ±0%	4	4	$0.1589
	Total	32.9s avg	5.2 avg	11.9 avg	$2.1084

Time/Cost columns show % change vs historical average (↑slower/costlier, ↓faster/cheaper). Changes under 10% shown as ±0%.

Historical Comparison Details

Filter: excluding branch 'claude/add-coralogix-api-key-TfwLo'

Status: Success - 45 test/model combinations loaded

Experiments compared (30):

• github-21437619376.2631.1 (branch: claude/clarify-gcp-docs-xN8Vr)
• github-21437403982.2626.1 (branch: claude/conditional-docs-build-LUNOZ)
• github-21437380212.2624.1 (branch: claude/clarify-gcp-docs-xN8Vr)
• ...and 27 more

Comparison indicators:

• ±0% — diff under 10% (within noise threshold)
• ↑N%/↓N% — diff 10-25%
• ↑N%/↓N% — diff over 25% (significant)

📖 Legend

Icon	Meaning
✅	The test was successful
➖	The test was skipped
⚠️	The test failed but is known to be flaky or known to fail
🚧	The test had a setup failure (not a code regression)
🔧	The test failed due to mock data issues (not a code regression)
🚫	The test was throttled by API rate limits/overload
❌	The test failed and should be fixed before merging the PR

🔄 Re-run evals manually

⚠️ Warning: /eval comments always run using the workflow from master, not from this PR branch. If you modified the GitHub Action (e.g., added secrets or env vars), those changes won't take effect.

To test workflow changes, use the GitHub CLI or Actions UI instead:

gh workflow run eval-regression.yaml --repo HolmesGPT/holmesgpt --ref claude/add-coralogix-api-key-TfwLo -f markers=regression -f filter=

---

Option 1: Comment on this PR with /eval:

/eval
markers: regression

Or with more options (one per line):

/eval
model: gpt-4o
markers: regression
filter: 09_crashpod
iterations: 5

Run evals on a different branch (e.g., master) for comparison:

/eval
branch: master
markers: regression

Option	Description
model	Model(s) to test (default: same as automatic runs)
markers	Pytest markers (no default - runs all tests!)
filter	Pytest -k filter (use /list to see valid eval names)
iterations	Number of runs, max 10
branch	Run evals on a different branch (for cross-branch comparison)

Quick re-run: Use /rerun to re-run the most recent /eval on this PR with the same parameters.

Option 2: Trigger via GitHub Actions UI → "Run workflow"

🏷️ Valid markers

benchmark, chain-of-causation, compaction, confluence, context_window, coralogix, counting, database, datadog, datetime, easy, elasticsearch, embeds, frontend, grafana-dashboard, hard, integration, kafka, kubernetes, leaked-information, logs, loki, medium, metrics, network, newrelic, no-cicd, numerical, one-test, port-forward, prometheus, question-answer, regression, runbooks, slackbot, storage, toolset-limitation, traces, transparency

---

Commands: /eval · /rerun · /list

CLI: gh workflow run eval-regression.yaml --repo HolmesGPT/holmesgpt --ref claude/add-coralogix-api-key-TfwLo -f markers=regression -f filter=

[github-actions]

✅ Docker image ready for a41e5a4 (built in 37s)

• us-central1-docker.pkg.dev/robusta-development/temporary-builds/holmes:a41e5a4

⚠️ Warning: does not support ARM (ARM images are built on release only - not on every PR)

Use this tag to pull the image for testing.

📋 Copy commands

⚠️ Temporary images are deleted after 30 days. Copy to a permanent registry before using them:

gcloud auth configure-docker us-central1-docker.pkg.dev
docker pull us-central1-docker.pkg.dev/robusta-development/temporary-builds/holmes:a41e5a4
docker tag us-central1-docker.pkg.dev/robusta-development/temporary-builds/holmes:a41e5a4 me-west1-docker.pkg.dev/robusta-development/development/holmes-dev:a41e5a4
docker push me-west1-docker.pkg.dev/robusta-development/development/holmes-dev:a41e5a4

Patch Helm values in one line (choose the chart you use):

HolmesGPT chart:

helm upgrade --install holmesgpt ./helm/holmes \
  --set registry=me-west1-docker.pkg.dev/robusta-development/development \
  --set image=holmes-dev:a41e5a4

Robusta wrapper chart:

helm upgrade --install robusta robusta/robusta \
  --reuse-values \
  --set holmes.registry=me-west1-docker.pkg.dev/robusta-development/development \
  --set holmes.image=holmes-dev:a41e5a4

[coderabbitai]

Walkthrough

Two new environment secrets, CORALOGIX_API_KEY and CORALOGIX_SEND_API_KEY, are being added to the evaluation regression workflow's test execution step, sourced from GitHub repository secrets.

Changes

Cohort / File(s)	Summary
GitHub Actions Workflow Configuration .github/workflows/eval-regression.yaml	Added two Coralogix-related API key secrets to the "Run tests" step environment variables.

Estimated code review effort

🎯 1 (Trivial) | ⏱️ ~2 minutes

Possibly related PRs

• Pass azure env vars to eval gh workflow #610 — Adds environment secrets/variables to the same eval GitHub Actions "Run tests" step, extending workflow credentials.

Suggested reviewers

• Sheeproid
• Avi-Robusta

🚥 Pre-merge checks | ✅ 3

✅ Passed checks (3 passed)

Check name	Status	Explanation
Description Check	✅ Passed	Check skipped - CodeRabbit’s high-level summary is enabled.
Title check	✅ Passed	The title accurately and specifically describes the main change: adding Coralogix API keys to the eval regression workflow. It is concise, clear, and directly reflects the core purpose of the changeset.
Docstring Coverage	✅ Passed	No functions found in the changed files to evaluate docstring coverage. Skipping docstring coverage check.

_{✏️ Tip: You can configure your own custom pre-merge checks in the settings.}

---

Thanks for using CodeRabbit! It's free for OSS, and your support helps us grow. If you like it, consider giving us a shout-out.

❤️ Share

• X
• Mastodon
• Reddit
• LinkedIn

_{Comment @coderabbitai help to get the list of available commands and usage tips.}