Add Datadog API keys to eval regression workflow

[aantn]

Add DATADOG_API_KEY and DATADOG_APP_KEY environment variables to the
"Run tests" step for Datadog integration evals.

https://claude.ai/code/session_01WzLxLvmEovB7Kb9rYtau6w
Signed-off-by: Claude noreply@anthropic.com

Summary by CodeRabbit

• Chores
  • Updated evaluation regression workflow configuration.

_{✏️ Tip: You can customize this high-level summary in your review settings.}

[linux-foundation-easycla]

• ❌ - login: @claude / name: Claude . The commit (41589dd) is not authorized under a signed CLA. Please click here to be authorized. For further assistance with EasyCLA, please submit a support request ticket.

[netlify]

✅ Deploy Preview for holmes-docs ready!

Name	Link
🔨 Latest commit	41589dd
🔍 Latest deploy log	https://app.netlify.com/projects/holmes-docs/deploys/697c7d194de77f0008854145
😎 Deploy Preview	https://deploy-preview-1455--holmes-docs.netlify.app
📱 Preview on mobile	Toggle QR Code... Use your smartphone camera to open QR code link.

---

To edit notification comments on pull requests, go to your Netlify project configuration.

[github-actions]

✅ Results of HolmesGPT evals

Automatically triggered by commit 41589dd on branch claude/add-datadog-api-keys-BDFKe

View workflow logs

Results of HolmesGPT evals

• ask_holmes: 9/9 test cases were successful, 0 regressions

Status	Test case	Time	Turns	Tools	Cost
✅	09_crashpod	33.9s ↑26%	5	11	$0.2300
✅	101_loki_historical_logs_pod_deleted	54.3s ↑54%	7	13	$0.2977
✅	111_pod_names_contain_service	33.7s ↑23%	5	11	$0.2246
✅	12_job_crashing	37.1s ↑22%	5	12	$0.2486
✅	162_get_runbooks	49.7s ↑12%	7	14	$0.3120
✅	176_network_policy_blocking_traffic_no_runbooks	44.9s ↑14%	6	15	$0.2919
✅	24_misconfigured_pvc	42.1s ↑43%	7	15	$0.2631
✅	43_current_datetime_from_prompt	5.1s ±0%	1	—	$0.1047
✅	61_exact_match_counting	16.7s ↑62%	4	4	$0.1597
	Total	35.3s avg	5.2 avg	11.9 avg	$2.1323

Time/Cost columns show % change vs historical average (↑slower/costlier, ↓faster/cheaper). Changes under 10% shown as ±0%.

Historical Comparison Details

Filter: excluding branch 'claude/add-datadog-api-keys-BDFKe'

Status: Success - 11 test/model combinations loaded

Experiments compared (30):

• github-21493450512.2704.1-07ed1bc7 (branch: master)
• github-21493450512.2704.1 (branch: master)
• github-21493001516.2698.1 (branch: master)
• ...and 27 more

Comparison indicators:

• ±0% — diff under 10% (within noise threshold)
• ↑N%/↓N% — diff 10-25%
• ↑N%/↓N% — diff over 25% (significant)

📖 Legend

Icon	Meaning
✅	The test was successful
➖	The test was skipped
⚠️	The test failed but is known to be flaky or known to fail
🚧	The test had a setup failure (not a code regression)
🔧	The test failed due to mock data issues (not a code regression)
🚫	The test was throttled by API rate limits/overload
❌	The test failed and should be fixed before merging the PR

🔄 Re-run evals manually

⚠️ Warning: /eval comments always run using the workflow from master, not from this PR branch. If you modified the GitHub Action (e.g., added secrets or env vars), those changes won't take effect.

To test workflow changes, use the GitHub CLI or Actions UI instead:

gh workflow run eval-regression.yaml --repo HolmesGPT/holmesgpt --ref claude/add-datadog-api-keys-BDFKe -f markers=regression -f filter=

---

Option 1: Comment on this PR with /eval:

/eval
markers: regression

Or with more options (one per line):

/eval
model: gpt-4o
markers: regression
filter: 09_crashpod
iterations: 5

Run evals on a different branch (e.g., master) for comparison:

/eval
branch: master
markers: regression

Option	Description
model	Model(s) to test (default: same as automatic runs)
markers	Pytest markers (no default - runs all tests!)
filter	Pytest -k filter (use /list to see valid eval names)
iterations	Number of runs, max 10
branch	Run evals on a different branch (for cross-branch comparison)

Quick re-run: Use /rerun to re-run the most recent /eval on this PR with the same parameters.

Option 2: Trigger via GitHub Actions UI → "Run workflow"

🏷️ Valid markers

benchmark, chain-of-causation, compaction, confluence, context_window, coralogix, counting, database, datadog, datetime, easy, elasticsearch, embeds, frontend, grafana-dashboard, hard, integration, kafka, kubernetes, leaked-information, logs, loki, medium, metrics, network, newrelic, no-cicd, numerical, one-test, port-forward, prometheus, question-answer, regression, runbooks, slackbot, storage, toolset-limitation, traces, transparency

---

Commands: /eval · /rerun · /list

CLI: gh workflow run eval-regression.yaml --repo HolmesGPT/holmesgpt --ref claude/add-datadog-api-keys-BDFKe -f markers=regression -f filter=

[github-actions]

✅ Docker image ready for 8eb4216 (built in 4m 33s)

• us-central1-docker.pkg.dev/robusta-development/temporary-builds/holmes:8eb4216

⚠️ Warning: does not support ARM (ARM images are built on release only - not on every PR)

Use this tag to pull the image for testing.

📋 Copy commands

⚠️ Temporary images are deleted after 30 days. Copy to a permanent registry before using them:

gcloud auth configure-docker us-central1-docker.pkg.dev
docker pull us-central1-docker.pkg.dev/robusta-development/temporary-builds/holmes:8eb4216
docker tag us-central1-docker.pkg.dev/robusta-development/temporary-builds/holmes:8eb4216 me-west1-docker.pkg.dev/robusta-development/development/holmes-dev:8eb4216
docker push me-west1-docker.pkg.dev/robusta-development/development/holmes-dev:8eb4216

Patch Helm values in one line (choose the chart you use):

HolmesGPT chart:

helm upgrade --install holmesgpt ./helm/holmes \
  --set registry=me-west1-docker.pkg.dev/robusta-development/development \
  --set image=holmes-dev:8eb4216

Robusta wrapper chart:

helm upgrade --install robusta robusta/robusta \
  --reuse-values \
  --set holmes.registry=me-west1-docker.pkg.dev/robusta-development/development \
  --set holmes.image=holmes-dev:8eb4216

[coderabbitai]

Walkthrough

Added two new secret references (DATADOG_API_KEY and DATADOG_APP_KEY) to the evaluation regression GitHub Actions workflow's test execution step.

Changes

Cohort / File(s)	Summary
Workflow Configuration .github/workflows/eval-regression.yaml	Added DATADOG_API_KEY and DATADOG_APP_KEY secret environment variables to the Run tests step.

Estimated code review effort

🎯 1 (Trivial) | ⏱️ ~3 minutes

Possibly related PRs

• Add Coralogix API keys to eval regression workflow #1442: Modifies the same workflow file and adds secret environment variables to the identical "Run tests" step.

Suggested reviewers

• Sheeproid

🚥 Pre-merge checks | ✅ 3

✅ Passed checks (3 passed)

Check name	Status	Explanation
Description Check	✅ Passed	Check skipped - CodeRabbit’s high-level summary is enabled.
Title check	✅ Passed	The title accurately summarizes the main change: adding Datadog API keys to the eval regression workflow, which matches the changeset modification.
Docstring Coverage	✅ Passed	No functions found in the changed files to evaluate docstring coverage. Skipping docstring coverage check.

_{✏️ Tip: You can configure your own custom pre-merge checks in the settings.}

Tip

🧪 Unit Test Generation v2 is now available!

We have significantly improved our unit test generation capabilities.

To enable: Add this to your .coderabbit.yaml configuration:

reviews:
  finishing_touches:
    unit_tests:
      enabled: true

Try it out by using the @coderabbitai generate unit tests command on your code files or under ✨ Finishing Touches on the walkthrough!

Have feedback? Share your thoughts on our Discord thread!

---

Thanks for using CodeRabbit! It's free for OSS, and your support helps us grow. If you like it, consider giving us a shout-out.

❤️ Share

• X
• Mastodon
• Reddit
• LinkedIn

_{Comment @coderabbitai help to get the list of available commands and usage tips.}