Yiqi/fix bugs by xiaoqi-7 · Pull Request #1332 · HolmesGPT/holmesgpt

xiaoqi-7 · 2026-01-05T23:05:05Z

Summary by CodeRabbit

Release Notes

New Features
- Remote model metadata support for improved context window sizing and token optimization on hosted vLLM models.
- Enhanced conversation compaction with intelligent message handling and context window management.
- Added "now" keyword support for timestamp operations.
Changes
- Removed mandatory runbook enforcement from investigation procedures.
- Added AKS toolset integrations (disabled by default).

_{✏️ Tip: You can customize this high-level summary in your review settings.}

linux-foundation-easycla · 2026-01-05T23:05:12Z

❌ - login: @yiqimicrosoft / name: YiqiWang . The commit (456b434, 5090546, 56e363c, 5dfbf14, b7363a6, be6f855, f2386bb) is not authorized under a signed CLA. Please click here to be authorized. For further assistance with EasyCLA, please submit a support request ticket.

CLAassistant · 2026-01-05T23:05:15Z

Thank you for your submission! We really appreciate it. Like many open source projects, we ask that you sign our Contributor License Agreement before we can accept your contribution.
_{You have signed the CLA already but the status is still pending? Let us recheck it.}

coderabbitai · 2026-01-05T23:05:18Z

Walkthrough

This PR introduces remote model metadata support for hosted vLLM models, comprehensively refactors the message compaction workflow with context-window error handling and retry logic, removes runbook enforcement from prompt templates, enhances utilities (e.g., to_unix handles "now" string), and removes numerous test fixtures for Loki, Datadog, electricity-bidding, and other scenarios. Includes a critical syntax error in test utilities.

Changes

Cohort / File(s)	Summary
LLM Core: Remote Model Metadata Support `holmes/core/llm.py`	Extended DefaultLLM with HTTP URL utilities, remote metadata fetching for OpenAI-compatible hosts, and integration into context window and output token sizing; adds new private methods and fields for thread-safe remote metadata caching.
Message Truncation & Compaction Refactoring `holmes/core/truncation/message_truncation.py`, `holmes/core/truncation/compaction.py`, `holmes/core/truncation/input_context_window_limiter.py`	New message_truncation.py module implements tool-message truncation with metadata; compaction.py adds comprehensive error detection, pre-shrinking, and retry logic on context-window errors; input_context_window_limiter.py adjusts imports.
Prompt Template Simplification `holmes/plugins/prompts/base_user_prompt.jinja2`, `holmes/plugins/prompts/investigation_procedure.jinja2`	Removed runbook conditional inclusion and enforcement blocks; investigation procedure now omits mandatory runbook preconditions.
Utilities & Script Updates `holmes/plugins/toolsets/utils.py`, `scripts/run_eval_setup.py`	to_unix now handles "now" string (case-insensitive) and uses tz-aware imports; run_eval_setup.py adds explicit bash executor to subprocess.run.
Test Utilities: OpenAI Client Compatibility Shims `tests/llm/utils/classifiers.py`	Added proxy classes (_CompletionsProxy, _ChatProxy, _MaxTokensCompatClient) to handle max_completion_tokens fallback on BadRequestError; create_llm_client now wraps client unconditionally.
Test Utilities: Critical Syntax Error `tests/llm/utils/commands.py`	⚠️ Syntax Error: _invoke_command subprocess.run call contains duplicate `executable=` keyword argument, breaking test execution.
Test Utilities: Toolset Configuration `tests/llm/utils/default_toolsets.yaml`	Disabled runbook, robusta, and connectivity_check toolsets; added disabled aks/core and aks/node-health entries.
Test Fixture Removals (Loki) `tests/llm/fixtures/test_ask_holmes/100a_loki_historical_logs/*`	Removed entire Loki test fixture: app-direct-push.py (LogBatcher, HealthHandler), payment-api deployment manifests, test_case.yaml, toolsets.yaml.
Test Fixture Removals (Electricity Bidding) `tests/llm/fixtures/test_ask_holmes/160_electricity_market_bidding_bug/`, `tests/llm/fixtures/test_ask_holmes/161_bidding_version_performance/`	Removed Prometheus, bidder-app, HPA, k6 load-test manifests, run-bidder-check.sh, wait-for-scaling.sh, and all associated test_case.yaml and toolsets.yaml files.
Test Fixture Removals (Datadog Traces) `tests/llm/fixtures/test_ask_holmes/164_datadog_traces_coupon_code/*`	Removed all-in-one Kubernetes manifest, Prometheus config, Datadog traces toolset, validate_promo_performance.py, and test_case.yaml.
Test Fixture Removals (Database/RDS) `tests/llm/fixtures/test_ask_holmes/22_high_latency_dbi_down/*`	Removed customer-orders deployment and service manifests, test_case.yaml, and toolsets.yaml.
Test Fixture Removals (Datadog Metrics & Logs) `tests/llm/fixtures/test_ask_holmes/91{a,b,c,d,e,f,g,h,i}_datadog_/`	Removed Datadog metrics/logs test fixtures across 9 directories: Kubernetes manifests, test_case.yaml files, send_custom_metrics.sh, send_datadog_logs.py, and toolsets.yaml entries.

Sequence Diagram(s)

sequenceDiagram
    participant App as Application
    participant Compact as compact_conversation_history()
    participant Shrink as _shrink_messages_to_fit_compaction_window()
    participant LLM as LLM (Compaction)
    participant ErrorHandler as Error Detection

    App->>Compact: (messages, llm)
    activate Compact
    
    rect rgb(220, 240, 255)
    Note over Compact,Shrink: Phase 1: Prepare & Shrink
    Compact->>Compact: _prepare_messages_for_compaction()
    Compact->>Shrink: shrink messages to fit context window
    activate Shrink
    Shrink->>Shrink: compute reserved output tokens
    Shrink->>Shrink: truncate_messages_to_fit_context()
    Shrink->>Shrink: drop oldest / hard-truncate largest
    Shrink-->>Compact: shrunk messages
    deactivate Shrink
    end
    
    rect rgb(240, 255, 240)
    Note over Compact,LLM: Phase 2: Compaction & Retry Loop
    Compact->>Compact: render compaction prompt
    Compact->>LLM: call LLM with shrunk messages
    activate LLM
    LLM-->>Compact: result or error
    deactivate LLM
    end
    
    Compact->>ErrorHandler: _is_context_window_exceeded_error()?
    activate ErrorHandler
    alt Context Window Exceeded
        ErrorHandler-->>Compact: true
        rect rgb(255, 240, 240)
        Note over Compact,Shrink: Phase 3: Retry with Aggressive Shrinking
        Compact->>Shrink: re-shrink with increased reserved tokens
        Shrink-->>Compact: more aggressively shrunk messages
        Compact->>LLM: retry compaction
        LLM-->>Compact: result
        end
    else No Context Window Error
        ErrorHandler-->>Compact: false
        Note over Compact: Return result as-is
    end
    deactivate ErrorHandler
    
    Compact-->>App: compacted conversation
    deactivate Compact

Estimated code review effort

🎯 4 (Complex) | ⏱️ ~50 minutes

Rationale: The diff exhibits high heterogeneity and logic density. Key complexity drivers include: (1) a critical syntax error in tests/llm/utils/commands.py (duplicate keyword argument) that breaks test execution; (2) significant message compaction refactoring with multi-phase error detection and retry logic requiring careful state management review; (3) new remote model metadata support with threading, caching, and fallback logic in holmes/core/llm.py; (4) extensive test fixture removals (though repetitive, they still require verification of completeness and absence of unintended side effects). The syntax error alone elevates priority, and the compaction logic requires tracing through multiple helper functions and error paths.

Possibly related PRs

#958 — Overlaps with tool-message truncation metadata instrumentation and TruncationMetadata usage patterns.
#805 — Introduces MAX_OUTPUT_TOKEN_RESERVATION constant that this PR's new truncation logic directly depends on.
#831 — Has direct file-level conflicts in test fixtures (22_high_latency_dbi_down manifest.yaml and test_case.yaml).

Suggested reviewers

moshemorad

Pre-merge checks

❌ Failed checks (1 warning, 1 inconclusive)

Check name	Status	Explanation	Resolution
Docstring Coverage	⚠️ Warning	Docstring coverage is 18.75% which is insufficient. The required threshold is 80.00%.	You can run `@coderabbitai generate docstrings` to improve docstring coverage.
Title check	❓ Inconclusive	The title 'Yiqi/fix bugs' is vague and generic, using non-descriptive terms that do not convey meaningful information about the substantial changes in this changeset.	Replace with a specific, descriptive title that captures the main changes, such as: 'Add remote model metadata support and message compaction workflow' or a more focused alternative reflecting the primary objective.

✅ Passed checks (1 passed)

Check name	Status	Explanation
Description Check	✅ Passed	Check skipped - CodeRabbit’s high-level summary is enabled.

Thanks for using CodeRabbit! It's free for OSS, and your support helps us grow. If you like it, consider giving us a shout-out.

❤️ Share

_{Comment @coderabbitai help to get the list of available commands and usage tips.}

netlify · 2026-01-05T23:06:05Z

✅ Deploy Preview for holmes-docs ready!

Name	Link
🔨 Latest commit	`456b434`
🔍 Latest deploy log	https://app.netlify.com/projects/holmes-docs/deploys/695c43a3eccf1000082e5f71
😎 Deploy Preview	https://deploy-preview-1332--holmes-docs.netlify.app
📱 Preview on mobile	Toggle QR Code... Use your smartphone camera to open QR code link.

To edit notification comments on pull requests, go to your Netlify project configuration.

coderabbitai

Actionable comments posted: 6

Fix all issues with AI Agents 🤖

In @holmes/core/truncation/message_truncation.py:
- Around line 11-40: _truncate_tool_message may pass None into
TruncationMetadata because msg.get("tool_call_id") and msg.get("name") can
return None; fix by coercing these to strings or providing defaults before
constructing TruncationMetadata (e.g., tool_call_id =
str(msg.get("tool_call_id") or "") and tool_name = str(msg.get("name") or "")),
use these safe variables in the logging message and in the
TruncationMetadata(...) call so Pydantic validation won't fail at runtime.

In @holmes/plugins/toolsets/utils.py:
- Around line 32-41: The function to_unix contains redundant local imports of
datetime; remove the inline "from datetime import datetime, timezone" and
"import datetime as dtmod" and instead use the module-level datetime import
already present at top of the file; update references inside to_unix to use the
top-level names (e.g., datetime.now(datetime.timezone.utc) and
dt.replace(tzinfo=datetime.timezone.utc)) so no local imports remain.

In @tests/llm/utils/classifiers.py:
- Around line 49-56: Add type hints to class _ChatProxy: annotate the __init__
parameter chat (e.g., chat: Any or a more specific Chat type), set self._chat:
Any, and type self.completions as _CompletionsProxy; annotate __getattr__ as def
__getattr__(self, name: str) -> Any. Also import typing.Any (or the concrete
Chat type) at the top. This keeps the delegation pattern but makes type
annotations explicit for _ChatProxy, __init__, self.completions, and
__getattr__.
- Around line 59-66: Add missing type hints to the _MaxTokensCompatClient proxy:
annotate the class constructor parameter (client) with the appropriate client
type or typing.Any, type the self._client attribute, annotate the chat attribute
as _ChatProxy, and add a return type of Any (or the client type) to __getattr__
as def __getattr__(self, name: str) -> Any; ensure necessary typing imports are
added.
- Around line 23-46: The proxy classes lack type annotations; import typing.Any
and add type hints: annotate _CompletionsProxy.__init__(self, completions: Any)
-> None, _CompletionsProxy.create(self, *args: Any, **kwargs: Any) -> Any, and
_CompletionsProxy.__getattr__(self, name: str) -> Any, and apply the same
pattern to the _ChatProxy and _MaxTokensCompatClient classes so all proxy
constructors, create/handler methods, varargs, kwargs and __getattr__ signatures
use Any and return Any (or None for __init__).

Caution

Some comments are outside the diff and can’t be posted inline due to platform limitations.

⚠️ Outside diff range comments (2)

tests/llm/utils/commands.py (1)
156-167: Critical: Duplicate keyword argument executable causes SyntaxError.

The subprocess.run call specifies executable="/bin/bash" twice (lines 159 and 162). This is a Python syntax error that will prevent the module from loading.
🔎 Proposed fix
         result = subprocess.run(
             command,
             shell=True,
             executable="/bin/bash",  # Force bash instead of default /bin/sh
             capture_output=True,
             text=True,
-            executable="/bin/bash",
             check=True,
             stdin=subprocess.DEVNULL,
             cwd=cwd,
             timeout=actual_timeout,
         )
holmes/core/truncation/input_context_window_limiter.py (1)

61-133: Critical: Duplicate function definition shadows the import, and MAX_OUTPUT_TOKEN_RESERVATION is undefined.

The function truncate_messages_to_fit_context is imported from message_truncation on line 17, but then redefined locally on lines 61-133. This shadows the import and creates code duplication. Additionally, line 87 references MAX_OUTPUT_TOKEN_RESERVATION which is not imported in this file—it's only imported in message_truncation.py. This will cause a NameError at runtime.

The same duplication issue applies to _truncate_tool_message (lines 24-53), which also exists in message_truncation.py.

Remove both duplicated functions and use the imported versions instead. The import statement on line 17 is already correct and should be the single source of truth.

🧹 Nitpick comments (2)

scripts/run_eval_setup.py (1)
44-44: Consider removing shell=True to address the security concern flagged by Ruff.

While adding executable="/bin/bash" makes the shell explicit, Ruff flags the use of shell=True as a potential security issue (S602). Although this is a developer-facing script and the YAML files are trusted repository content, using shell=True with externally loaded data degrades security posture.

If the commands in test_case.yaml don't require shell features (pipes, wildcards, etc.), consider refactoring to use shell=False with a list of arguments. Alternatively, add explicit validation that data[section] contains expected content before execution.
🔎 Example refactor without shell=True (if commands are simple)
-            result = subprocess.run(data[section], shell=True, executable="/bin/bash")
+            import shlex
+            result = subprocess.run(shlex.split(data[section]))
Note: This only works if commands don't use shell-specific features. If shell features are required, the current approach is acceptable given this is a trusted dev script.
holmes/core/truncation/compaction.py (1)
114-115: Consider logging the swallowed exception for debugging.

The try-except-pass pattern silently swallows truncation errors, which could make debugging difficult if truncation consistently fails.
🔎 Proposed fix
     try:
         messages = truncate_messages_to_fit_context(
             messages=messages,
             max_context_size=max_context_size,
             maximum_output_token=reserved_output_tokens,
             count_tokens_fn=llm.count_tokens,
         ).truncated_messages
         if fits(messages):
             return messages
-    except Exception:
-        pass
+    except Exception as e:
+        logging.debug("Truncation pass failed during shrinking, falling back to message dropping: %s", e)

📜 Review details

Configuration used: Organization UI

Review profile: CHILL

Plan: Pro

📥 Commits

Reviewing files that changed from the base of the PR and between 8a4c86a and 456b434.

📒 Files selected for processing (63)

holmes/core/llm.py
holmes/core/truncation/compaction.py
holmes/core/truncation/input_context_window_limiter.py
holmes/core/truncation/message_truncation.py
holmes/plugins/prompts/base_user_prompt.jinja2
holmes/plugins/prompts/investigation_procedure.jinja2
holmes/plugins/toolsets/utils.py
scripts/run_eval_setup.py
tests/llm/fixtures/test_ask_holmes/100a_loki_historical_logs/app-direct-push.py
tests/llm/fixtures/test_ask_holmes/100a_loki_historical_logs/payment-api-direct.yaml
tests/llm/fixtures/test_ask_holmes/100a_loki_historical_logs/test_case.yaml
tests/llm/fixtures/test_ask_holmes/100a_loki_historical_logs/toolsets.yaml
tests/llm/fixtures/test_ask_holmes/160_electricity_market_bidding_bug/bidder-app.yaml
tests/llm/fixtures/test_ask_holmes/160_electricity_market_bidding_bug/bidding_system.md
tests/llm/fixtures/test_ask_holmes/160_electricity_market_bidding_bug/prometheus-config.yaml
tests/llm/fixtures/test_ask_holmes/160_electricity_market_bidding_bug/run-bidder-check.sh
tests/llm/fixtures/test_ask_holmes/160_electricity_market_bidding_bug/test_case.yaml
tests/llm/fixtures/test_ask_holmes/160_electricity_market_bidding_bug/toolsets.yaml
tests/llm/fixtures/test_ask_holmes/161_bidding_version_performance/bidder-v1.yaml
tests/llm/fixtures/test_ask_holmes/161_bidding_version_performance/bidder-v2.yaml
tests/llm/fixtures/test_ask_holmes/161_bidding_version_performance/hpa.yaml
tests/llm/fixtures/test_ask_holmes/161_bidding_version_performance/k6-v1-traffic.yaml
tests/llm/fixtures/test_ask_holmes/161_bidding_version_performance/k6-v2-traffic.yaml
tests/llm/fixtures/test_ask_holmes/161_bidding_version_performance/prometheus-config.yaml
tests/llm/fixtures/test_ask_holmes/161_bidding_version_performance/test_case.yaml
tests/llm/fixtures/test_ask_holmes/161_bidding_version_performance/toolsets.yaml
tests/llm/fixtures/test_ask_holmes/161_bidding_version_performance/wait-for-scaling.sh
tests/llm/fixtures/test_ask_holmes/164_datadog_traces_coupon_code/holmes-all-in-one-fast.yaml
tests/llm/fixtures/test_ask_holmes/164_datadog_traces_coupon_code/prometheus-config.yaml
tests/llm/fixtures/test_ask_holmes/164_datadog_traces_coupon_code/test_case.yaml
tests/llm/fixtures/test_ask_holmes/164_datadog_traces_coupon_code/toolsets.yaml
tests/llm/fixtures/test_ask_holmes/164_datadog_traces_coupon_code/validate_promo_performance.py
tests/llm/fixtures/test_ask_holmes/22_high_latency_dbi_down/manifest.yaml
tests/llm/fixtures/test_ask_holmes/22_high_latency_dbi_down/test_case.yaml
tests/llm/fixtures/test_ask_holmes/22_high_latency_dbi_down/toolsets.yaml
tests/llm/fixtures/test_ask_holmes/91a_datadog_metrics_missing_namespace/test_case.yaml
tests/llm/fixtures/test_ask_holmes/91a_datadog_metrics_missing_namespace/toolsets.yaml
tests/llm/fixtures/test_ask_holmes/91b_datadog_metrics_pod_exists/manifest.yaml
tests/llm/fixtures/test_ask_holmes/91b_datadog_metrics_pod_exists/test_case.yaml
tests/llm/fixtures/test_ask_holmes/91b_datadog_metrics_pod_exists/toolsets.yaml
tests/llm/fixtures/test_ask_holmes/91c_datadog_metrics_deployment/manifest.yaml
tests/llm/fixtures/test_ask_holmes/91c_datadog_metrics_deployment/test_case.yaml
tests/llm/fixtures/test_ask_holmes/91c_datadog_metrics_deployment/toolsets.yaml
tests/llm/fixtures/test_ask_holmes/91d_datadog_metrics_historical_pod/manifest.yaml
tests/llm/fixtures/test_ask_holmes/91d_datadog_metrics_historical_pod/test_case.yaml
tests/llm/fixtures/test_ask_holmes/91d_datadog_metrics_historical_pod/toolsets.yaml
tests/llm/fixtures/test_ask_holmes/91e_datadog_custom_metrics/manifest.yaml
tests/llm/fixtures/test_ask_holmes/91e_datadog_custom_metrics/send_custom_metrics.sh
tests/llm/fixtures/test_ask_holmes/91e_datadog_custom_metrics/test_case.yaml
tests/llm/fixtures/test_ask_holmes/91e_datadog_custom_metrics/toolsets.yaml
tests/llm/fixtures/test_ask_holmes/91f_datadog_logs_historical_pod/send_datadog_logs.py
tests/llm/fixtures/test_ask_holmes/91f_datadog_logs_historical_pod/test_case.yaml
tests/llm/fixtures/test_ask_holmes/91f_datadog_logs_historical_pod/toolsets.yaml
tests/llm/fixtures/test_ask_holmes/91g_datadog_metrics_mismatched_pod/manifest.yaml
tests/llm/fixtures/test_ask_holmes/91g_datadog_metrics_mismatched_pod/test_case.yaml
tests/llm/fixtures/test_ask_holmes/91g_datadog_metrics_mismatched_pod/toolsets.yaml
tests/llm/fixtures/test_ask_holmes/91h_datadog_logs_empty_query_with_url/test_case.yaml
tests/llm/fixtures/test_ask_holmes/91h_datadog_logs_empty_query_with_url/toolsets.yaml
tests/llm/fixtures/test_ask_holmes/91i_datadog_metrics_empty_query_with_url/test_case.yaml
tests/llm/fixtures/test_ask_holmes/91i_datadog_metrics_empty_query_with_url/toolsets.yaml
tests/llm/utils/classifiers.py
tests/llm/utils/commands.py
tests/llm/utils/default_toolsets.yaml

💤 Files with no reviewable changes (54)

tests/llm/fixtures/test_ask_holmes/91a_datadog_metrics_missing_namespace/toolsets.yaml
tests/llm/fixtures/test_ask_holmes/91e_datadog_custom_metrics/send_custom_metrics.sh
tests/llm/fixtures/test_ask_holmes/91c_datadog_metrics_deployment/manifest.yaml
tests/llm/fixtures/test_ask_holmes/91g_datadog_metrics_mismatched_pod/manifest.yaml
tests/llm/fixtures/test_ask_holmes/91c_datadog_metrics_deployment/test_case.yaml
tests/llm/fixtures/test_ask_holmes/160_electricity_market_bidding_bug/test_case.yaml
tests/llm/fixtures/test_ask_holmes/100a_loki_historical_logs/test_case.yaml
tests/llm/fixtures/test_ask_holmes/91d_datadog_metrics_historical_pod/test_case.yaml
tests/llm/fixtures/test_ask_holmes/164_datadog_traces_coupon_code/holmes-all-in-one-fast.yaml
tests/llm/fixtures/test_ask_holmes/161_bidding_version_performance/bidder-v1.yaml
tests/llm/fixtures/test_ask_holmes/91f_datadog_logs_historical_pod/toolsets.yaml
tests/llm/fixtures/test_ask_holmes/164_datadog_traces_coupon_code/prometheus-config.yaml
tests/llm/fixtures/test_ask_holmes/161_bidding_version_performance/wait-for-scaling.sh
tests/llm/fixtures/test_ask_holmes/160_electricity_market_bidding_bug/bidder-app.yaml
tests/llm/fixtures/test_ask_holmes/91b_datadog_metrics_pod_exists/manifest.yaml
tests/llm/fixtures/test_ask_holmes/161_bidding_version_performance/prometheus-config.yaml
tests/llm/fixtures/test_ask_holmes/91h_datadog_logs_empty_query_with_url/test_case.yaml
tests/llm/fixtures/test_ask_holmes/91h_datadog_logs_empty_query_with_url/toolsets.yaml
tests/llm/fixtures/test_ask_holmes/91e_datadog_custom_metrics/test_case.yaml
tests/llm/fixtures/test_ask_holmes/161_bidding_version_performance/k6-v1-traffic.yaml
tests/llm/fixtures/test_ask_holmes/164_datadog_traces_coupon_code/toolsets.yaml
tests/llm/fixtures/test_ask_holmes/91f_datadog_logs_historical_pod/test_case.yaml
tests/llm/fixtures/test_ask_holmes/161_bidding_version_performance/toolsets.yaml
tests/llm/fixtures/test_ask_holmes/22_high_latency_dbi_down/test_case.yaml
tests/llm/fixtures/test_ask_holmes/91b_datadog_metrics_pod_exists/test_case.yaml
holmes/plugins/prompts/base_user_prompt.jinja2
tests/llm/fixtures/test_ask_holmes/160_electricity_market_bidding_bug/bidding_system.md
tests/llm/fixtures/test_ask_holmes/91e_datadog_custom_metrics/manifest.yaml
tests/llm/fixtures/test_ask_holmes/91c_datadog_metrics_deployment/toolsets.yaml
tests/llm/fixtures/test_ask_holmes/161_bidding_version_performance/hpa.yaml
tests/llm/fixtures/test_ask_holmes/164_datadog_traces_coupon_code/test_case.yaml
tests/llm/fixtures/test_ask_holmes/91e_datadog_custom_metrics/toolsets.yaml
tests/llm/fixtures/test_ask_holmes/161_bidding_version_performance/test_case.yaml
tests/llm/fixtures/test_ask_holmes/91g_datadog_metrics_mismatched_pod/toolsets.yaml
tests/llm/fixtures/test_ask_holmes/160_electricity_market_bidding_bug/prometheus-config.yaml
tests/llm/fixtures/test_ask_holmes/91i_datadog_metrics_empty_query_with_url/toolsets.yaml
tests/llm/fixtures/test_ask_holmes/160_electricity_market_bidding_bug/run-bidder-check.sh
tests/llm/fixtures/test_ask_holmes/91a_datadog_metrics_missing_namespace/test_case.yaml
holmes/plugins/prompts/investigation_procedure.jinja2
tests/llm/fixtures/test_ask_holmes/91f_datadog_logs_historical_pod/send_datadog_logs.py
tests/llm/fixtures/test_ask_holmes/161_bidding_version_performance/bidder-v2.yaml
tests/llm/fixtures/test_ask_holmes/91d_datadog_metrics_historical_pod/toolsets.yaml
tests/llm/fixtures/test_ask_holmes/91i_datadog_metrics_empty_query_with_url/test_case.yaml
tests/llm/fixtures/test_ask_holmes/91g_datadog_metrics_mismatched_pod/test_case.yaml
tests/llm/fixtures/test_ask_holmes/22_high_latency_dbi_down/toolsets.yaml
tests/llm/fixtures/test_ask_holmes/161_bidding_version_performance/k6-v2-traffic.yaml
tests/llm/fixtures/test_ask_holmes/100a_loki_historical_logs/payment-api-direct.yaml
tests/llm/fixtures/test_ask_holmes/100a_loki_historical_logs/toolsets.yaml
tests/llm/fixtures/test_ask_holmes/91b_datadog_metrics_pod_exists/toolsets.yaml
tests/llm/fixtures/test_ask_holmes/160_electricity_market_bidding_bug/toolsets.yaml
tests/llm/fixtures/test_ask_holmes/91d_datadog_metrics_historical_pod/manifest.yaml
tests/llm/fixtures/test_ask_holmes/22_high_latency_dbi_down/manifest.yaml
tests/llm/fixtures/test_ask_holmes/164_datadog_traces_coupon_code/validate_promo_performance.py
tests/llm/fixtures/test_ask_holmes/100a_loki_historical_logs/app-direct-push.py

🧰 Additional context used

📓 Path-based instructions (7)

tests/llm/**/*.{py,yaml}