Improve datadog evals

[aantn]

Two fixes for datadog evals:

1. Make several datadog metric evals stricter so that the following does not pass!

2. Loosen the requirements around mock_data in eval 93 so it works even when the llm modifies the default time frame (or leaves it out altogether)

[coderabbitai]

Caution

Review failed

The pull request is closed.

Walkthrough

Test fixtures updated: three LLM expected_output entries tightened wording for random_key to require a "random-looking" value; two Datadog fetch_pod_logs fixtures had start_time removed from match_params while leaving start_time in the response params. No code, API, or control-flow changes.

Changes

Cohort / File(s)	Change summary
LLM expected_output random_key wording tests/llm/fixtures/test_ask_holmes/91_cpu_graph_robusta_runner/test_case.yaml, tests/llm/fixtures/test_ask_holmes/92_cpu_graph_conversation/test_case.yaml, tests/llm/fixtures/test_ask_holmes/110_cpu_graph_robusta_runner/test_case.yaml	Tightened expected_output wording: random_key must be "random-looking"; other fields (type, tool_name) unchanged. Two locations updated in the 110 fixture.
Datadog fetch_pod_logs match_params tests/llm/fixtures/test_ask_holmes/93_calling_datadog/fetch_pod_logssock-shop_catalogue-db-c948fd796-w4vzl_-3600.txt, tests/llm/fixtures/test_ask_holmes/93_calling_datadog/fetch_pod_logssock-shop_catalogue-f7687cb4-zsngc_-3600.txt	Removed start_time from the request match_params; start_time remains present inside the response params. Other content and structure unchanged.

Estimated code review effort

🎯 2 (Simple) | ⏱️ ~8 minutes

Possibly related PRs

• Rob 1573 fix evals mocks #562 — Related edits to fetch_pod_logs fixtures and handling of start_time in test data.
• ROB-1873-validate-datadog-logs-query-working-as-expected #780 — Overlapping changes to the same fixtures, including start_time/match_params and random_key wording.

Suggested reviewers

• moshemorad
• arikalon1

---

📜 Recent review details

Configuration used: CodeRabbit UI
Review profile: CHILL
Plan: Pro

📥 Commits

Reviewing files that changed from the base of the PR and between b3eacc2 and d4d07e1.

📒 Files selected for processing (5)

• tests/llm/fixtures/test_ask_holmes/110_cpu_graph_robusta_runner/test_case.yaml (1 hunks)
• tests/llm/fixtures/test_ask_holmes/91_cpu_graph_robusta_runner/test_case.yaml (1 hunks)
• tests/llm/fixtures/test_ask_holmes/92_cpu_graph_conversation/test_case.yaml (1 hunks)
• tests/llm/fixtures/test_ask_holmes/93_calling_datadog/fetch_pod_logssock-shop_catalogue-db-c948fd796-w4vzl_-3600.txt (1 hunks)
• tests/llm/fixtures/test_ask_holmes/93_calling_datadog/fetch_pod_logssock-shop_catalogue-f7687cb4-zsngc_-3600.txt (1 hunks)

✨ Finishing Touches

🧪 Generate unit tests

• Create PR with unit tests
• Post copyable unit tests in a comment
• Commit unit tests in branch improve-datadog-evals

---

Thanks for using CodeRabbit! It's free for OSS, and your support helps us grow. If you like it, consider giving us a shout-out.

❤️ Share

• X
• Mastodon
• Reddit
• LinkedIn

🪧 Tips

Chat

There are 3 ways to chat with CodeRabbit:

• Review comments: Directly reply to a review comment made by CodeRabbit. Example:
  • I pushed a fix in commit <commit_id>, please review it.
  • Explain this complex logic.
  • Open a follow-up GitHub issue for this discussion.
• Files and specific lines of code (under the "Files changed" tab): Tag @coderabbitai in a new review comment at the desired location with your query. Examples:
  • @coderabbitai explain this code block.
• PR comments: Tag @coderabbitai in a new PR comment to ask questions about the PR branch. For the best results, please provide a very specific query, as very limited context is provided in this mode. Examples:
  • @coderabbitai gather interesting stats about this repository and render them as a table. Additionally, render a pie chart showing the language distribution in the codebase.
  • @coderabbitai read src/utils.ts and explain its main purpose.
  • @coderabbitai read the files in the src/scheduler package and generate a class diagram using mermaid and a README in the markdown format.

Support

Need help? Create a ticket on our support page for assistance with any issues or questions.

CodeRabbit Commands (Invoked using PR comments)

• @coderabbitai pause to pause the reviews on a PR.
• @coderabbitai resume to resume the paused reviews.
• @coderabbitai review to trigger an incremental review. This is useful when automatic reviews are disabled for the repository.
• @coderabbitai full review to do a full review from scratch and review all the files again.
• @coderabbitai summary to regenerate the summary of the PR.
• @coderabbitai generate docstrings to generate docstrings for this PR.
• @coderabbitai generate sequence diagram to generate a sequence diagram of the changes in this PR.
• @coderabbitai generate unit tests to generate unit tests for this PR.
• @coderabbitai resolve resolve all the CodeRabbit review comments.
• @coderabbitai configuration to show the current CodeRabbit configuration for the repository.
• @coderabbitai help to get help.

Other keywords and placeholders

• Add @coderabbitai ignore anywhere in the PR description to prevent this PR from being reviewed.
• Add @coderabbitai summary to generate the high-level summary at a specific location in the PR description.
• Add @coderabbitai anywhere in the PR title to generate the title automatically.

CodeRabbit Configuration File (.coderabbit.yaml)

• You can programmatically configure CodeRabbit by adding a .coderabbit.yaml file to the root of your repository.
• Please see the configuration documentation for more information.
• If your editor has YAML language server enabled, you can add the path at the top of this file to enable auto-completion and validation: # yaml-language-server: $schema=https://coderabbit.ai/integrations/schema.v2.json

Documentation and Community

• Visit our Documentation for detailed information on how to use CodeRabbit.
• Join our Discord Community to get help, request features, and share feedback.
• Follow us on X/Twitter for updates and announcements.

[github-actions]

Results of HolmesGPT evals

• ask_holmes: 25/43 test cases were successful, 0 regressions, 1 skipped, 17 mock failures

Test suite	Test case	Status
ask	01_how_many_pods	✅
ask	02_what_is_wrong_with_pod	🔧
ask	03_what_is_the_command_to_port_forward	🔧
ask	04_related_k8s_events	↪️
ask	05_image_version	🔧
ask	09_crashpod	✅
ask	10_image_pull_backoff	🔧
ask	11_init_containers	✅
ask	14_pending_resources	✅
ask	15_failed_readiness_probe	🔧
ask	17_oom_kill	✅
ask	18_crash_looping_v2	✅
ask	19_detect_missing_app_details	🔧
ask	24_misconfigured_pvc	🔧
ask	28_permissions_error	✅
ask	29_events_from_alert_manager	🔧
ask	39_failed_toolset	🔧
ask	41_setup_argo	✅
ask	42_dns_issues_steps_new_tools	🔧
ask	43_current_datetime_from_prompt	✅
ask	45_fetch_deployment_logs_simple	✅
ask	51_logs_summarize_errors	🔧
ask	53_logs_find_term	✅
ask	54_not_truncated_when_getting_pods	✅
ask	59_label_based_counting	✅
ask	60_count_less_than	🔧
ask	61_exact_match_counting	✅
ask	63_fetch_error_logs_no_errors	✅
ask	77_liveness_probe_misconfiguration	🔧
ask	79_configmap_mount_issue	🔧
ask	83_secret_not_found	🔧
ask	86_configmap_like_but_secret	🔧
ask	88_affinity_like_but_taints	🔧
ask	89_runbook_missing_cloudwatch	🔧
ask	90_runbook_basic_selection	🔧
ask	93_calling_datadog	✅
ask	93_calling_datadog	✅
ask	93_calling_datadog	✅
ask	97_logs_clarification_needed	✅
ask	100_historical_logs	🔧
ask	110_k8s_events_image_pull	🔧
ask	24a_misconfigured_pvc_basic	🔧
ask	13a_pending_node_selector_basic	🔧

Legend

• ✅ the test was successful
• ↪️ the test was skipped
• ⚠️ the test failed but is known to be flaky or known to fail
• 🔧 the test failed due to mock data issues (not a code regression)
• ❌ the test failed and should be fixed before merging the PR