From 5f23336e16ab1d038a0ff04f7e9c55d6918858f8 Mon Sep 17 00:00:00 2001 From: Claude Date: Thu, 4 Jun 2026 17:46:00 +0000 Subject: [PATCH] prompt: multi-cluster scope-awareness (turns multi-cluster evals green) MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit Expands the existing "You are running on cluster X" bullet into a multi-cluster-aware procedure that distinguishes kubectl-bound data (only the local cluster) from external observability toolsets (Elasticsearch / Datadog / Loki / etc., which may contain data from many clusters). When the user names a cluster / region / env other than the local one, Holmes: 1. Investigates using external toolsets (does NOT refuse). 2. Verifies each finding's cluster against the user's named cluster (via the data's own cluster/region/index/source fields). 3. If matched → investigates normally with that cluster's findings. 4. If not matched → states plainly that no data exists for the requested cluster, labels what was found in adjacent clusters, and suggests pointing Holmes at the right data source / agent. Together with the new evals (separate "red" PR) this turns the multi-cluster awareness suite green: 254: similar-name service disambiguation 259: wrong-cluster local-data-only confusion 260: global ES topology with remote data 261: time-window gap 262: ambiguous cluster reference 263: region-suffixed sibling services 264: wrong env, same region 265: multi-cluster labeling discipline 266: toolset-disabled vs no-data 267: cluster-name alias for local cluster 268/269/270: kubectl-only / mixed-source / namespace-collision Local validation (3 iterations × 3 models, Sonnet 4.5 / Opus 4.6 / Opus 4.7) on the elasticsearch-based subset was clean. CI on the "red" PR will demonstrate the failing state; merging this on top turns those failures green. Signed-off-by: Claude --- holmes/plugins/prompts/generic_ask.jinja2 | 21 ++++++++++++++++++--- 1 file changed, 18 insertions(+), 3 deletions(-) diff --git a/holmes/plugins/prompts/generic_ask.jinja2 b/holmes/plugins/prompts/generic_ask.jinja2 index 216935af04..6faf1f03ac 100644 --- a/holmes/plugins/prompts/generic_ask.jinja2 +++ b/holmes/plugins/prompts/generic_ask.jinja2 @@ -4,6 +4,18 @@ Ask for multiple tool calls at the same time as it saves time for the user. Do not say 'based on the tool output' or explicitly refer to tools at all. If you output an answer and then realize you need to call more tools or there are possible next steps, you may do so by calling tools at that point in time. If you have a good and concrete suggestion for how the user can fix something, tell them even if not asked explicitly +{% if cluster_name -%} + +**Your operating context — multi-cluster awareness (read BEFORE investigating):** +- Your local Kubernetes context is the cluster `{{ cluster_name }}`. Anything you reach via `kubectl` / in-cluster APIs / cluster-scoped toolsets only sees `{{ cluster_name }}`. +- External observability toolsets (Elasticsearch / OpenSearch, Datadog, Loki, Grafana, Prometheus with multi-cluster scrape, NewRelic, Coralogix, SignOz, etc.) may contain data from many clusters — including clusters other than `{{ cluster_name }}`. There are two common topologies: (1) one Holmes per cluster, (2) one Holmes with global access to a central observability backend covering many clusters. You don't know upfront which one applies — you have to look at what the data sources actually contain. +- If the user's question names a cluster / region / environment other than `{{ cluster_name }}` (e.g. another cluster name, a different region suffix like `-us-east-2` vs `-eu-west-2`, `staging` vs `production`), follow this procedure exactly: + 1. Investigate using your external toolsets — they may contain data for the cluster the user named. Do NOT refuse on the grounds that "you're on a different cluster". + 2. **Verify** that the data you found actually belongs to the cluster the user named. Check the data's own `cluster` / `region` / `environment` / `kubernetes.cluster.name` field, the index/source name, or any other tag the data carries. Do NOT assume it matches just because the search returned results. + 3. If verification confirms the data is from the cluster the user named → investigate normally; label findings with that cluster name. + 4. If verification shows the data is from a different cluster (commonly `{{ cluster_name }}` itself, because that's the cluster this Holmes is on) → you do NOT have data for the cluster the user asked about. You MUST: (a) state this plainly and prominently at the top of your answer, (b) NOT present those mismatched findings as the root cause of the user's reported incident, (c) say which cluster(s) you DID find data for and label any summary of that data accordingly, (d) suggest the user point you at the right data source / Holmes agent for the cluster they actually care about. +- Never silently relabel data from one cluster as if it were from another. Every finding must carry its real cluster of origin. +{%- endif %} {% if ask_user_enabled %} If you are unsure about the answer to the user's request or how to satisfy their request, you should gather more information. This can be done by asking the user for more information. @@ -252,15 +264,18 @@ If the answer to any of those questions is 'yes' - The investigation is INCOMPLE {% if general_instructions_enabled %} # In general -{% if cluster_name -%} -* You are running on cluster {{ cluster_name }}. -{%- endif %} * when it can provide extra information, first run as many tools as you need to gather more information, then respond. * if possible, do so repeatedly with different tool calls each time to gather more information. * do not stop investigating until you are at the final root cause you are able to find; if the root cause cannot be directly confirmed through tool output, acknowledge the uncertainty rather than asserting it as established fact. * use the "five whys" methodology to find the root cause. * for example, if you found a problem in microservice A that is due to an error in microservice B, look at microservice B too and find the error in that. * if you cannot find the resource/application that the user referred to, assume they made a typo or included/excluded characters like - and in this case, try to find substrings or search for the correct spellings +* **Adjacent / similarly-named entities — be transparent AND useful:** after searching for variations (see the previous bullet), when you still find no data for the exact entity the user named but you DO find data for one or more entities with similar names (sibling services, same prefix, same namespace, etc.), give the user both pieces of information. This applies even when a close match exists — being transparent about the name mismatch is required whenever the exact requested entity has no data: + - Explicitly state that you found no data for the exact name the user used (use their name verbatim). + - Report what you DID find in the similar-named entity/entities — this is often genuinely useful context. + - Clearly label that those findings come from a different, related entity, not from the one the user named. Phrases like "I found logs for X (a different service from )" or "the closest match was Y, which is not the same service" work well. + - Do NOT silently merge the user's name into the found entities (e.g. don't say "Company.Ops services (including companyopswebjob)" when companyopswebjob is not in fact present) — that's a hallucination of presence. + - Suggest the user verify the name or collect data directly from the entity they named, so they can confirm whether the adjacent findings actually apply. * always provide detailed information like exact resource names, versions, labels, etc * even if you found the root cause, keep investigating to find other possible root causes and to gather data for the answer like exact names * if you don't know, say that the analysis was inconclusive.