Improve HolmesGPT accuracy on questions about itself, configuring tools, and using runbooks by aantn · Pull Request #729 · HolmesGPT/holmesgpt

aantn · 2025-07-27T17:20:06Z

No description provided.

coderabbitai · 2025-07-27T17:20:16Z

Walkthrough

This change set updates HolmesGPT prompt templates for runbook usage, toolset configuration, and permission error handling. It enhances runbook fetching to support multiple search paths, enriches returned runbook content with usage instructions, and adds new troubleshooting runbooks and test fixtures. Documentation and test configurations were also refined.

Changes

Cohort / File(s)	Change Summary
Prompt templates `holmes/plugins/prompts/_general_instructions.jinja2`, `holmes/plugins/prompts/_runbook_instructions.jinja2`, `holmes/plugins/prompts/_toolsets_instructions.jinja2`, `holmes/plugins/prompts/generic_ask.jinja2`	Expanded and clarified prompt instructions for runbook usage, toolset setup, and permission error handling. Added inclusion of current date/time and externalized permission error instructions. Improved control flow and user guidance.
New permission error prompt `holmes/plugins/prompts/_permission_errors.jinja2`	Added a new prompt template providing a structured three-step process to analyze and report permission errors with links to documentation.
Runbook path resolution `holmes/plugins/runbooks/__init__.py`	Enhanced `get_runbook_by_path` to accept multiple search directories, returning the first existing runbook path or `None` if not found, with updated function signature and docstring.
Runbook fetcher toolset `holmes/plugins/toolsets/runbook/runbook_fetcher.py`	Updated `RunbookFetcher` to search runbooks in multiple directories, improved error reporting with detailed search paths, and appended structured usage instructions to runbook content. Modified `RunbookToolset` constructor to accept additional search paths configuration.
Mock toolset test utility `tests/llm/utils/mock_toolset.py`	Modified mock toolset manager to replace the "runbook" toolset with a new instance including the current test case folder as an additional runbook search path, preserving existing status and configuration.
Test case metadata and comments `tests/llm/fixtures/test_ask_holmes/40_disabled_toolset/test_case.yaml`	Changed test tag from "easy" to "medium" and added detailed comments on test history and breakage due to prompt changes.
Pod manifests and test timing `tests/llm/fixtures/test_ask_holmes/60_count_less_than/manifests.yaml`, `tests/llm/fixtures/test_ask_holmes/60_count_less_than/test_case.yaml`	Standardized pod sleep durations to 1 second and removed trailing blank line in test script.
New AWS Lambda runbook and test `tests/llm/fixtures/test_ask_holmes/89_runbook_missing_cloudwatch/lambda_performance_troubleshooting.md`, `tests/llm/fixtures/test_ask_holmes/89_runbook_missing_cloudwatch/test_case.yaml`	Added AWS Lambda high latency troubleshooting runbook and corresponding test case simulating missing CloudWatch integration.
New Application Gateway runbook and test `tests/llm/fixtures/test_ask_holmes/90_runbook_basic_selection/application_gateway_troubleshooting.md`, `tests/llm/fixtures/test_ask_holmes/90_runbook_basic_selection/test_case.yaml`	Added Application Gateway troubleshooting runbook and test case for basic runbook selection scenario.
Test marker documentation update `CLAUDE.md`	Replaced explicit list of pytest test markers with a note directing users to `pyproject.toml` for valid markers, removing detailed enumeration from documentation.
Minor pyproject.toml formatting `pyproject.toml`	Added a single blank line after last dependency entry; no functional changes.

Sequence Diagram(s)

sequenceDiagram
    participant User
    participant HolmesGPT
    participant RunbookToolset
    participant RunbookFetcher

    User->>HolmesGPT: Reports operational issue (e.g., high CPU, error)
    HolmesGPT->>RunbookToolset: Search for relevant runbook (with additional search paths)
    RunbookToolset->>RunbookFetcher: fetch_runbook(runbook_path, search_paths)
    RunbookFetcher->>RunbookToolset: get_runbook_by_path(runbook_path, search_paths)
    alt Runbook found
        RunbookFetcher-->>HolmesGPT: Return runbook content + usage instructions
        HolmesGPT-->>User: Presents runbook steps, follows instructions, reports findings
    else Runbook not found
        RunbookFetcher-->>HolmesGPT: Return error with attempted search paths
        HolmesGPT-->>User: Informs user runbook not found, suggests next steps
    end

Estimated code review effort

🎯 3 (Moderate) | ⏱️ ~18 minutes

Possibly related PRs

fix: runbook instruction should list all items clearly #587: Both PRs modify _runbook_instructions.jinja2 to improve runbook instruction clarity and structure.
feat(runbook): introduce runbook toolset to fetch internal runbooks #547: This PR builds upon runbook toolset and catalog infrastructure introduced in feat(runbook): introduce runbook toolset to fetch internal runbooks #547, directly extending runbook handling.
further pytest improvements to improve DX #578: Documentation changes here relate to test marker configuration changes introduced in further pytest improvements to improve DX #578.

Suggested reviewers

arikalon1
moshemorad

📜 Recent review details

Configuration used: CodeRabbit UI
Review profile: CHILL
Plan: Pro

📥 Commits

Reviewing files that changed from the base of the PR and between 657255c and 0eb9f37.

📒 Files selected for processing (2)

holmes/plugins/prompts/generic_ask.jinja2 (1 hunks)
tests/llm/fixtures/test_ask_holmes/60_count_less_than/test_case.yaml (0 hunks)

💤 Files with no reviewable changes (1)

tests/llm/fixtures/test_ask_holmes/60_count_less_than/test_case.yaml

✅ Files skipped from review due to trivial changes (1)

holmes/plugins/prompts/generic_ask.jinja2

⏰ Context from checks skipped due to timeout of 90000ms. You can increase the timeout in your CodeRabbit configuration to a maximum of 15 minutes (900000ms). (3)

GitHub Check: llm_evals
GitHub Check: Pre-commit checks
GitHub Check: Pre-commit checks

✨ Finishing Touches

🧪 Generate unit tests

Create PR with unit tests
Post copyable unit tests in a comment
Commit unit tests in branch improve-transparency

Thanks for using CodeRabbit! It's free for OSS, and your support helps us grow. If you like it, consider giving us a shout-out.

❤️ Share

🪧 Tips

Chat

There are 3 ways to chat with CodeRabbit:

Review comments: Directly reply to a review comment made by CodeRabbit. Example:
- I pushed a fix in commit <commit_id>, please review it.
- Explain this complex logic.
- Open a follow-up GitHub issue for this discussion.
Files and specific lines of code (under the "Files changed" tab): Tag @coderabbitai in a new review comment at the desired location with your query. Examples:
- @coderabbitai explain this code block.
- @coderabbitai modularize this function.
PR comments: Tag @coderabbitai in a new PR comment to ask questions about the PR branch. For the best results, please provide a very specific query, as very limited context is provided in this mode. Examples:
- @coderabbitai gather interesting stats about this repository and render them as a table. Additionally, render a pie chart showing the language distribution in the codebase.
- @coderabbitai read src/utils.ts and explain its main purpose.
- @coderabbitai read the files in the src/scheduler package and generate a class diagram using mermaid and a README in the markdown format.
- @coderabbitai help me debug CodeRabbit configuration file.

Support

Need help? Create a ticket on our support page for assistance with any issues or questions.

Note: Be mindful of the bot's finite context window. It's strongly recommended to break down tasks such as reading entire modules into smaller chunks. For a focused discussion, use review comments to chat about specific files and their changes, instead of using the PR comments.

CodeRabbit Commands (Invoked using PR comments)

@coderabbitai pause to pause the reviews on a PR.
@coderabbitai resume to resume the paused reviews.
@coderabbitai review to trigger an incremental review. This is useful when automatic reviews are disabled for the repository.
@coderabbitai full review to do a full review from scratch and review all the files again.
@coderabbitai summary to regenerate the summary of the PR.
@coderabbitai generate docstrings to generate docstrings for this PR.
@coderabbitai generate sequence diagram to generate a sequence diagram of the changes in this PR.
@coderabbitai generate unit tests to generate unit tests for this PR.
@coderabbitai resolve resolve all the CodeRabbit review comments.
@coderabbitai configuration to show the current CodeRabbit configuration for the repository.
@coderabbitai help to get help.

Other keywords and placeholders

Add @coderabbitai ignore anywhere in the PR description to prevent this PR from being reviewed.
Add @coderabbitai summary to generate the high-level summary at a specific location in the PR description.
Add @coderabbitai anywhere in the PR title to generate the title automatically.

CodeRabbit Configuration File (`.coderabbit.yaml`)

You can programmatically configure CodeRabbit by adding a .coderabbit.yaml file to the root of your repository.
Please see the configuration documentation for more information.
If your editor has YAML language server enabled, you can add the path at the top of this file to enable auto-completion and validation: # yaml-language-server: $schema=https://coderabbit.ai/integrations/schema.v2.json

Documentation and Community

Visit our Documentation for detailed information on how to use CodeRabbit.
Join our Discord Community to get help, request features, and share feedback.
Follow us on X/Twitter for updates and announcements.

coderabbitai

Actionable comments posted: 2

🧹 Nitpick comments (4)

holmes/core/tools_utils/tool_executor.py (1)
18-18: Replace direct attribute access with a read-only property

The TODO hints at hiding toolsets.
A minimal, backwards-compatible step is:
 class ToolExecutor:
     def __init__(self, toolsets: List[Toolset]):
-        # TODO: expose function for this instead of callers accessing directly
-        self.toolsets = toolsets
+        # Keep private; expose via property for read-only access
+        self._toolsets = toolsets
+
+    @property
+    def toolsets(self) -> list[Toolset]:
+        """Return all configured toolsets (read-only)."""
+        return self._toolsets
This avoids external mutation while requiring no call-site changes.
tests/llm/utils/test_case_utils.py (1)
290-327: Function implementation is correct but could improve type safety.

The message building logic is sound and follows proper patterns for prompt construction. The temporary nature is appropriately documented.

Consider improving the type annotation for better type safety:
-    tool_executor: Any,  # ToolExecutor type
+    tool_executor: "ToolExecutor",  # Forward reference if needed to avoid circular imports
The function name suffix "2" suggests this might be replacing an existing function - ensure the original is removed once the referenced PR is merged.
tests/llm/test_workload_health.py (1)
62-72: Consider removing commented code

The dataset upload logic has been commented out. If this functionality is no longer needed, consider removing the commented code entirely to improve code cleanliness.
 def get_test_cases():
     mh = MockHelper(TEST_CASES_FOLDER)
-    # dataset_name = braintrust_util.get_dataset_name("health_check")
-    # if os.environ.get("UPLOAD_DATASET") and os.environ.get("BRAINTRUST_API_KEY"):
-    #     bt_helper = braintrust_util.BraintrustEvalHelper(
-    #         project_name=BRAINTRUST_PROJECT, dataset_name=dataset_name
-    #     )
-    #     bt_helper.upload_test_cases(mh.load_test_cases())

     test_cases = mh.load_workload_health_test_cases()
     iterations = int(os.environ.get("ITERATIONS", "1"))
     return [add_tags_to_eval(test_case) for test_case in test_cases] * iterations
tests/llm/test_ask_holmes.py (1)

52-58: Good identification of code duplication

The TODO correctly identifies code duplication across test files. Consider extracting the common test case loading logic into a shared utility function.

Would you like me to help create a shared utility function to eliminate this duplication across the test files?

📜 Review details

Configuration used: CodeRabbit UI
Review profile: CHILL
Plan: Pro

📥 Commits

Reviewing files that changed from the base of the PR and between 30fbb25 and 7934989.

📒 Files selected for processing (26)

.github/workflows/llm-evaluation.yaml (1 hunks)
CLAUDE.md (4 hunks)
conftest.py (2 hunks)
docs/development/evals/index.md (1 hunks)
docs/development/evals/writing.md (1 hunks)
holmes/core/tools_utils/tool_executor.py (1 hunks)
holmes/core/tracing.py (4 hunks)
holmes/plugins/prompts/_general_instructions.jinja2 (2 hunks)
holmes/plugins/prompts/_permission_errors.jinja2 (1 hunks)
holmes/plugins/prompts/_runbook_instructions.jinja2 (1 hunks)
holmes/plugins/prompts/_toolsets_instructions.jinja2 (3 hunks)
holmes/plugins/prompts/generic_ask.jinja2 (1 hunks)
tests/llm/conftest.py (6 hunks)
tests/llm/fixtures/test_ask_holmes/56_kafka_runbook_no_tool/test_case.yaml (1 hunks)
tests/llm/test_ask_holmes.py (7 hunks)
tests/llm/test_investigate.py (4 hunks)
tests/llm/test_workload_health.py (5 hunks)
tests/llm/utils/braintrust.py (4 hunks)
tests/llm/utils/constants.py (0 hunks)
tests/llm/utils/mock_toolset.py (3 hunks)
tests/llm/utils/property_manager.py (1 hunks)
tests/llm/utils/reporting/terminal_reporter.py (3 hunks)
tests/llm/utils/tags.py (1 hunks)
tests/llm/utils/test_case_utils.py (4 hunks)
tests/llm/utils/test_mock_toolset.py (8 hunks)
tests/llm/utils/test_results.py (1 hunks)

💤 Files with no reviewable changes (1)

tests/llm/utils/constants.py

🧰 Additional context used

📓 Path-based instructions (5)

**/*.py

📄 CodeRabbit Inference Engine (CLAUDE.md)

**/*.py: Use Ruff for formatting and linting (configured in pyproject.toml)
Type hints required (mypy configuration in pyproject.toml)

Files:

holmes/core/tools_utils/tool_executor.py
tests/llm/utils/test_results.py
tests/llm/utils/property_manager.py
tests/llm/utils/tags.py
tests/llm/utils/reporting/terminal_reporter.py
tests/llm/utils/braintrust.py
conftest.py
holmes/core/tracing.py
tests/llm/utils/test_case_utils.py
tests/llm/test_investigate.py
tests/llm/test_ask_holmes.py
tests/llm/utils/test_mock_toolset.py
tests/llm/test_workload_health.py
tests/llm/utils/mock_toolset.py
tests/llm/conftest.py

holmes/plugins/prompts/**/*.jinja2

📄 CodeRabbit Inference Engine (CLAUDE.md)

Prompts: holmes/plugins/prompts/{name}.jinja2

Files:

holmes/plugins/prompts/generic_ask.jinja2
holmes/plugins/prompts/_general_instructions.jinja2
holmes/plugins/prompts/_permission_errors.jinja2
holmes/plugins/prompts/_runbook_instructions.jinja2
holmes/plugins/prompts/_toolsets_instructions.jinja2

tests/**/*.py

📄 CodeRabbit Inference Engine (CLAUDE.md)

tests/**/*.py: All new features require unit tests
New toolsets require integration tests with mocks
Tests: Match source structure under tests/

Files:

tests/llm/utils/test_results.py
tests/llm/utils/property_manager.py
tests/llm/utils/tags.py
tests/llm/utils/reporting/terminal_reporter.py
tests/llm/utils/braintrust.py
tests/llm/utils/test_case_utils.py
tests/llm/test_investigate.py
tests/llm/test_ask_holmes.py
tests/llm/utils/test_mock_toolset.py
tests/llm/test_workload_health.py
tests/llm/utils/mock_toolset.py
tests/llm/conftest.py

tests/llm/**/*.py

📄 CodeRabbit Inference Engine (CLAUDE.md)

tests/llm/**/*.py: Complex investigations should have LLM evaluation tests
Each LLM test must use a dedicated namespace app- (e.g., app-01, app-02) to prevent conflicts when tests run simultaneously

Files:

tests/llm/utils/test_results.py
tests/llm/utils/property_manager.py
tests/llm/utils/tags.py
tests/llm/utils/reporting/terminal_reporter.py
tests/llm/utils/braintrust.py
tests/llm/utils/test_case_utils.py
tests/llm/test_investigate.py
tests/llm/test_ask_holmes.py
tests/llm/utils/test_mock_toolset.py
tests/llm/test_workload_health.py
tests/llm/utils/mock_toolset.py
tests/llm/conftest.py

tests/llm/fixtures/**/*

📄 CodeRabbit Inference Engine (CLAUDE.md)

tests/llm/fixtures/**/*: Mock data: tests/llm/fixtures/{test_name}/
All pod names must be unique across tests (e.g., giant-narwhal, blue-whale, sea-turtle) - never reuse pod names between tests
Never use resource names that hint at the problem or expected behavior (e.g., avoid broken-pod, test-project-that-does-not-exist, crashloop-app). Use neutral names that don't give away what the LLM should discover

Files:

tests/llm/fixtures/test_ask_holmes/56_kafka_runbook_no_tool/test_case.yaml

🧠 Learnings (21)

📓 Common learnings

Learnt from: Sheeproid
PR: robusta-dev/holmesgpt#586
File: tests/llm/fixtures/test_ask_holmes/03_what_is_the_command_to_port_forward/test_case.yaml:4-4
Timestamp: 2025-07-02T10:27:17.231Z
Learning: In LLM-as-judge test cases for HolmesGPT, expected outputs should be descriptive rather than prescriptive when testing for flexible responses like port numbers. Using specific values in expected outputs can cause unnecessary test failures when the AI generates different but equally valid responses.

Learnt from: nherment
PR: robusta-dev/holmesgpt#610
File: .github/workflows/llm-evaluation.yaml:39-42
Timestamp: 2025-07-08T08:45:41.069Z
Learning: When suggesting improvements to environment variable handling in robusta-dev/holmesgpt, check first if validation logic already exists rather than reimplementing it.

Learnt from: CR
PR: robusta-dev/holmesgpt#0
File: CLAUDE.md:0-0
Timestamp: 2025-07-27T14:10:31.571Z
Learning: LLM evaluation tests run automatically in CI

Learnt from: CR
PR: robusta-dev/holmesgpt#0
File: CLAUDE.md:0-0
Timestamp: 2025-07-27T14:10:31.571Z
Learning: Applies to holmes/plugins/prompts/**/*.jinja2 : Prompts: holmes/plugins/prompts/{name}.jinja2

Learnt from: nherment
PR: robusta-dev/holmesgpt#610
File: .github/workflows/llm-evaluation.yaml:39-42
Timestamp: 2025-07-08T08:45:41.069Z
Learning: The robusta-dev/holmesgpt codebase has comprehensive existing validation for Azure environment variables (AZURE_API_BASE, AZURE_API_KEY, AZURE_API_VERSION) and MODEL in tests/llm/utils/classifiers.py, tests/llm/conftest.py, and holmes/core/llm.py. Don't suggest adding redundant validation logic.

Learnt from: CR
PR: robusta-dev/holmesgpt#0
File: CLAUDE.md:0-0
Timestamp: 2025-07-27T14:10:31.571Z
Learning: Applies to tests/**/*.py : New toolsets require integration tests with mocks

Learnt from: CR
PR: robusta-dev/holmesgpt#0
File: CLAUDE.md:0-0
Timestamp: 2025-07-27T14:10:31.571Z
Learning: Applies to tests/llm/**/*.py : Complex investigations should have LLM evaluation tests

Learnt from: CR
PR: robusta-dev/holmesgpt#0
File: CLAUDE.md:0-0
Timestamp: 2025-07-27T14:10:31.571Z
Learning: Applies to holmes/plugins/toolsets/**/*.yaml : Toolsets: holmes/plugins/toolsets/{name}.yaml or {name}/

holmes/core/tools_utils/tool_executor.py (1)

Learnt from: nherment
PR: #535
File: holmes/plugins/toolsets/bash/bash_toolset.py:207-209
Timestamp: 2025-06-24T05:51:04.543Z
Learning: The init_config method in toolsets should be idempotent - safely callable multiple times without errors. self.config should maintain consistent typing (not alternate between dict and config object types) throughout the object lifecycle.

holmes/plugins/prompts/generic_ask.jinja2 (1)

Learnt from: CR
PR: robusta-dev/holmesgpt#0
File: CLAUDE.md:0-0
Timestamp: 2025-07-27T14:10:31.571Z
Learning: Applies to holmes/plugins/prompts/**/*.jinja2 : Prompts: holmes/plugins/prompts/{name}.jinja2

docs/development/evals/index.md (3)

Learnt from: nherment
PR: #610
File: .github/workflows/llm-evaluation.yaml:39-42
Timestamp: 2025-07-08T08:45:41.069Z
Learning: When suggesting improvements to environment variable handling in robusta-dev/holmesgpt, check first if validation logic already exists rather than reimplementing it.

Learnt from: nherment
PR: #610
File: .github/workflows/llm-evaluation.yaml:39-42
Timestamp: 2025-07-08T08:45:41.069Z
Learning: The robusta-dev/holmesgpt codebase has comprehensive existing validation for Azure environment variables (AZURE_API_BASE, AZURE_API_KEY, AZURE_API_VERSION) and MODEL in tests/llm/utils/classifiers.py, tests/llm/conftest.py, and holmes/core/llm.py. Don't suggest adding redundant validation logic.

Learnt from: Sheeproid
PR: #586
File: tests/llm/fixtures/test_ask_holmes/03_what_is_the_command_to_port_forward/test_case.yaml:4-4
Timestamp: 2025-07-02T10:27:17.231Z
Learning: In LLM-as-judge test cases for HolmesGPT, expected outputs should be descriptive rather than prescriptive when testing for flexible responses like port numbers. Using specific values in expected outputs can cause unnecessary test failures when the AI generates different but equally valid responses.

holmes/plugins/prompts/_general_instructions.jinja2 (2)

Learnt from: CR
PR: robusta-dev/holmesgpt#0
File: CLAUDE.md:0-0
Timestamp: 2025-07-27T14:10:31.571Z
Learning: Applies to holmes/plugins/prompts/**/*.jinja2 : Prompts: holmes/plugins/prompts/{name}.jinja2

Learnt from: nherment
PR: #408
File: holmes/plugins/toolsets/kubernetes_logs.py:90-97
Timestamp: 2025-05-15T05:13:43.169Z
Learning: In the Kubernetes logs toolset for Holmes, both current and previous logs are intentionally fetched and combined for each pod, even though this requires more API calls. This design ensures all logs are captured even when pods restart but retain their name, providing complete diagnostic information.

holmes/plugins/prompts/_permission_errors.jinja2 (1)

Learnt from: CR
PR: robusta-dev/holmesgpt#0
File: CLAUDE.md:0-0
Timestamp: 2025-07-27T14:10:31.571Z
Learning: Applies to holmes/plugins/prompts/**/*.jinja2 : Prompts: holmes/plugins/prompts/{name}.jinja2

tests/llm/fixtures/test_ask_holmes/56_kafka_runbook_no_tool/test_case.yaml (3)

Learnt from: Sheeproid
PR: #586
File: tests/llm/fixtures/test_ask_holmes/03_what_is_the_command_to_port_forward/test_case.yaml:4-4
Timestamp: 2025-07-02T10:27:17.231Z
Learning: In LLM-as-judge test cases for HolmesGPT, expected outputs should be descriptive rather than prescriptive when testing for flexible responses like port numbers. Using specific values in expected outputs can cause unnecessary test failures when the AI generates different but equally valid responses.

Learnt from: CR
PR: robusta-dev/holmesgpt#0
File: CLAUDE.md:0-0
Timestamp: 2025-07-27T14:10:31.571Z
Learning: Applies to holmes/plugins/toolsets/**/*.yaml : Toolsets: holmes/plugins/toolsets/{name}.yaml or {name}/

Learnt from: CR
PR: robusta-dev/holmesgpt#0
File: CLAUDE.md:0-0
Timestamp: 2025-07-27T14:10:31.571Z
Learning: Applies to tests/llm/fixtures/**/* : Mock data: tests/llm/fixtures/{test_name}/

docs/development/evals/writing.md (3)

Learnt from: CR
PR: robusta-dev/holmesgpt#0
File: CLAUDE.md:0-0
Timestamp: 2025-07-27T14:10:31.571Z
Learning: Applies to tests/llm/fixtures/**/* : Mock data: tests/llm/fixtures/{test_name}/

Learnt from: CR
PR: robusta-dev/holmesgpt#0
File: CLAUDE.md:0-0
Timestamp: 2025-07-27T14:10:31.571Z
Learning: Applies to tests/**/*.py : New toolsets require integration tests with mocks

Learnt from: CR
PR: robusta-dev/holmesgpt#0
File: CLAUDE.md:0-0
Timestamp: 2025-07-27T14:10:31.571Z
Learning: Applies to tests/llm/fixtures/**/* : Never use resource names that hint at the problem or expected behavior (e.g., avoid broken-pod, test-project-that-does-not-exist, crashloop-app). Use neutral names that don't give away what the LLM should discover

.github/workflows/llm-evaluation.yaml (3)

Learnt from: CR
PR: robusta-dev/holmesgpt#0
File: CLAUDE.md:0-0
Timestamp: 2025-07-27T14:10:31.571Z
Learning: LLM evaluation tests run automatically in CI

Learnt from: CR
PR: robusta-dev/holmesgpt#0
File: CLAUDE.md:0-0
Timestamp: 2025-07-27T14:10:31.571Z
Learning: Applies to tests/llm/**/*.py : Complex investigations should have LLM evaluation tests

Learnt from: CR
PR: robusta-dev/holmesgpt#0
File: CLAUDE.md:0-0
Timestamp: 2025-07-27T14:10:31.571Z
Learning: Applies to tests/llm/**/*.py : Each LLM test must use a dedicated namespace app- (e.g., app-01, app-02) to prevent conflicts when tests run simultaneously

holmes/plugins/prompts/_runbook_instructions.jinja2 (1)

Learnt from: CR
PR: robusta-dev/holmesgpt#0
File: CLAUDE.md:0-0
Timestamp: 2025-07-27T14:10:31.571Z
Learning: Applies to holmes/plugins/prompts/**/*.jinja2 : Prompts: holmes/plugins/prompts/{name}.jinja2

tests/llm/utils/braintrust.py (3)

Learnt from: nherment
PR: #610
File: .github/workflows/llm-evaluation.yaml:39-42
Timestamp: 2025-07-08T08:45:41.069Z
Learning: The robusta-dev/holmesgpt codebase has comprehensive existing validation for Azure environment variables (AZURE_API_BASE, AZURE_API_KEY, AZURE_API_VERSION) and MODEL in tests/llm/utils/classifiers.py, tests/llm/conftest.py, and holmes/core/llm.py. Don't suggest adding redundant validation logic.

Learnt from: nherment
PR: #610
File: .github/workflows/llm-evaluation.yaml:39-42
Timestamp: 2025-07-08T08:45:41.069Z
Learning: When suggesting improvements to environment variable handling in robusta-dev/holmesgpt, check first if validation logic already exists rather than reimplementing it.

Learnt from: nherment
PR: #436
File: tests/llm/utils/mock_utils.py:240-249
Timestamp: 2025-06-05T12:23:27.634Z
Learning: The holmesgpt project uses Python >= 3.10 and prefers modern type hint syntax like list[str], dict[str, int] over importing equivalent types from the typing module like List[str], Dict[str, int].

holmes/core/tracing.py (2)

Learnt from: nherment
PR: #610
File: .github/workflows/llm-evaluation.yaml:39-42
Timestamp: 2025-07-08T08:45:41.069Z
Learning: The robusta-dev/holmesgpt codebase has comprehensive existing validation for Azure environment variables (AZURE_API_BASE, AZURE_API_KEY, AZURE_API_VERSION) and MODEL in tests/llm/utils/classifiers.py, tests/llm/conftest.py, and holmes/core/llm.py. Don't suggest adding redundant validation logic.

Learnt from: nherment
PR: #436
File: tests/llm/utils/mock_utils.py:240-249
Timestamp: 2025-06-05T12:23:27.634Z
Learning: The holmesgpt project uses Python >= 3.10 and prefers modern type hint syntax like list[str], dict[str, int] over importing equivalent types from the typing module like List[str], Dict[str, int].

tests/llm/utils/test_case_utils.py (3)

Learnt from: CR
PR: robusta-dev/holmesgpt#0
File: CLAUDE.md:0-0
Timestamp: 2025-07-27T14:10:31.571Z
Learning: Applies to tests/**/*.py : New toolsets require integration tests with mocks

Learnt from: CR
PR: robusta-dev/holmesgpt#0
File: CLAUDE.md:0-0
Timestamp: 2025-07-27T14:10:31.571Z
Learning: Applies to holmes/plugins/prompts/**/*.jinja2 : Prompts: holmes/plugins/prompts/{name}.jinja2

Learnt from: CR
PR: robusta-dev/holmesgpt#0
File: CLAUDE.md:0-0
Timestamp: 2025-07-27T14:10:31.571Z
Learning: Applies to tests/llm/**/*.py : Complex investigations should have LLM evaluation tests

tests/llm/test_investigate.py (5)

Learnt from: CR
PR: robusta-dev/holmesgpt#0
File: CLAUDE.md:0-0
Timestamp: 2025-07-27T14:10:31.571Z
Learning: Applies to tests/**/*.py : New toolsets require integration tests with mocks

Learnt from: CR
PR: robusta-dev/holmesgpt#0
File: CLAUDE.md:0-0
Timestamp: 2025-07-27T14:10:31.571Z
Learning: Applies to tests/llm/**/*.py : Complex investigations should have LLM evaluation tests

Learnt from: CR
PR: robusta-dev/holmesgpt#0
File: CLAUDE.md:0-0
Timestamp: 2025-07-27T14:10:31.571Z
Learning: Applies to tests/llm/fixtures/**/* : Mock data: tests/llm/fixtures/{test_name}/

Learnt from: CR
PR: robusta-dev/holmesgpt#0
File: CLAUDE.md:0-0
Timestamp: 2025-07-27T14:10:31.571Z
Learning: Applies to tests/llm/**/*.py : Each LLM test must use a dedicated namespace app- (e.g., app-01, app-02) to prevent conflicts when tests run simultaneously

Learnt from: CR
PR: robusta-dev/holmesgpt#0
File: CLAUDE.md:0-0
Timestamp: 2025-07-27T14:10:31.571Z
Learning: LLM evaluation tests run automatically in CI

CLAUDE.md (16)