Some missing code by RobertSzefler · Pull Request #1 · HolmesGPT/holmesgpt

RobertSzefler · 2024-05-30T13:49:11Z

Some code we overlooked when switching repos

When importing holmes and leverage RichHandler to handle log, we need to remove existing log handlers to not let existing log handler break the outpout format of holmes CLI. As a result, the log may not be printed out as expected, for example, ``` Welcome to HolmesGPT: Type '/exit' to exit, '/help' for commands. User: what's my cluster info Thinking... ╭─ AI Response ─────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────╮ │ │ │ Cluster: AKS, region southeastasia, 7 nodes. │ │ │ │ • Nodepools: containerd2 (Azure Linux), nodepool1 (Ubuntu), win (Windows) │ │ • Kubernetes version: v1.33.0 │ │ • Network policy: calico │ │ • All nodes Ready, VMSS-backed, Premium_LRS storage. │ │ │ ╰───────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────╯ User: ``` The force arg is explained here: ``` force If this keyword is specified as true, any existing handlers attached to the root logger are removed and closed, before carrying out the configuration as specified by the other arguments. ``` https://github.com/python/cpython/blob/e41c1ce585827f92dab9b7a7fc3df2bda2f817fe/Lib/logging/__init__.py#L2038-L2041 After the change: ``` ✅ Toolset kubernetes/live-metrics ✅ Toolset kubernetes/kube-prometheus-stack ✅ Toolset internet ✅ Toolset runbook ✅ Toolset kubernetes/logs ✅ Toolset docker/core ✅ Toolset helm/core ✅ Toolset kubernetes/core discovered service with label-selector: `app=kube-prometheus-stack-prometheus` at url: `http://prometheus-kube-prometheus-prometheus.default.svc.cluster.local:9090` Prometheus auto discovered at url http://prometheus-kube-prometheus-prometheus.default.svc.cluster.local:9090 ❌ Toolset prometheus/metrics: Failed to initialize using url=http://prometheus-kube-prometheus-prometheus.default.svc.cluster.local:9090/-/healthy Using 36 datasources (toolsets). To refresh: `holmes toolset refresh` Welcome to HolmesGPT: Type '/exit' to exit, '/help' for commands. User: what's my cluster info Thinking... Running tool #1 kubectl_get_by_kind_in_cluster: kubectl get -A --show-labels -o wide node Finished #1 in 0.94s, output length: 10,264 characters - /show to view contents ╭─ AI Response ─────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────╮ │ │ │ Cluster: MC_qinhao_credential-provider-win_southeastasia Region: southeastasia Kubernetes version: v1.33.0 Nodes: 7 (6 Linux, 1 Windows) Nodepools: │ │ containerd2 (Azure Linux), nodepool1 (Ubuntu), win (Windows) Network policy: calico VM type: Standard_DS2_v2 (Linux), Standard_D2ls_v5 (Windows) │ │ All nodes Ready. │ │ │ ╰───────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────╯ ```

This is an example of what is added to the beginning of the user prompt for the runbooks if there are global instructions, runbooks, or issue/workload instructions (the sections are created dynamically in the prompt in order based on what the user has configured) ``` # Runbook Selection You (HolmesGPT) have access to runbooks with step-by-step troubleshooting instructions. If one of the following runbooks relates to the user's issue, you MUST fetch it with the fetch_runbook tool. You (HolmesGPT) must follow runbook sources in this priority order: 1) Runbook Catalog (priority #1) 2) Global Instructions (priority #2) ## Runbook Catalog (priority #1) Here are MD runbooks: * networking/dns_troubleshooting_instructions.md | description: Runbook to investigate DNS resolution issue in Kubernetes clusters * upgrade/upgrade_troubleshooting_instructions.md | description: Runbook to troubleshoot upgrade issues in Azure Kubernetes Service clusters Here are Robusta runbooks: * id='8fe8e24d-6b53-47a6-92a5-b389938fa823' | title='Kafka Topic and Application Mapping' | symptom='Kafka lag issues, consumer falling behind' * id='40296b5c-2cb5-41df-b1f5-93441f894c44' | title='Pod Crashlooping Debugging Runbook' | symptom='Pod continuously crashing and restarting' * id='b2ffd311-f339-46a2-b379-1e53eb0bc1ed' | title='Signup Service Debugging Runbook' | symptom='Signup service failing, users can't register' ## Global Instructions (priority #2) * Always check service dependencies first * Verify environment variables are set correctly * Check logs for error patterns before investigating further * Contact the interlock team on all issues that could be related to STRIPE or the Stripe API * On all other issues contact the capricorn team If a runbook might match the user's issue, you MUST: 1. Fetch the runbook with the `fetch_runbook` tool. 2. Decide based on the runbook's contents if it is relevant or not. 3. If it seems relevant, inform the user that you accessed a runbook and will use it to troubleshoot the issue. 4. To the maximum extent possible, follow the runbook instructions step-by-step. 5. Provide a detailed report of the steps you performed, including any findings or errors encountered. 6. If a runbook step requires tools or integrations you don't have access to, tell the user that you cannot perform that step due to missing tools. ``` This is an example of how it looks fetching a a robusta runbook ``` id: b2ffd311-f339-46a2-b379-1e53eb0bc1ed symptom: Signup service failing, users can't register title: Signup Service Debugging Runbook instruction: |- # Signup Service Debugging Runbook ## Overview When the signup service is experiencing issues, the problem is often related to the payments service dependency. This runbook provides systematic debugging steps. ## Step 1: Check Signup Service Status ```bash kubectl get pods -l app=signup-service kubectl describe pod <signup-pod-name> kubectl logs <signup-pod-name> ``` Look for: - Pod status and restart count - Error messages in logs ... ```

In build stage of dockerfile, the virtual venv is installed under /app, since poetry installs python executables like poetry with the interpreter "/app/venv/bin/python", but we copied venv from builder to /venv in the docker image, leading to the failure of executing the python tool: `bash: /venv/bin/poetry: cannot execute: required file not found` ``` cat /venv/bin/poetry #!/app/venv/bin/python import sys from poetry.console.application import main if __name__ == '__main__': if sys.argv[0].endswith('.exe'): sys.argv[0] = sys.argv[0][:-4] sys.exit(main()) ``` To make /app folder clean with only holmes related files, I keep the venv under /, so the change is mainly under builder by changing the workdir to / ## Tests - run poetry ``` root@43f694155286:/app# poetry --version Poetry (version 2.2.1) ``` - run holmes cli.py ``` root@43f694155286:/app# python holmes_cli.py ask "what's cluster info" --model azure/gpt-4.1 Loaded models: ['azure/gpt-4.1'] Refreshing available datasources (toolsets) Running without kube-config! e=Invalid kube-config file. No configuration found. ...... Toolset statuses are cached to /root/.holmes/toolsets_status.json ✅ Toolset kubernetes/kube-prometheus-stack ✅ Toolset core_investigation ✅ Toolset internet ✅ Toolset datadog/rds ✅ Toolset bash ✅ Toolset runbook ✅ Toolset kubernetes/logs ✅ Toolset kubernetes/core ✅ Toolset helm/core Using selected model: azure/gpt-4.1 Using model: azure/gpt-4.1 (1,047,576 total tokens, 32,768 output tokens) Welcome to HolmesGPT: Type '/exit' to exit, '/help' for commands. User: what's cluster info Thinking... The AI requested 1 tool call(s). Running tool #1 TodoWrite: Update investigation tasks Task List: +----+--------------------------------------------------------+-----------------+ | ID | Content | Status | +----+--------------------------------------------------------+-----------------+ | 1 | Define what 'cluster info' means in Kubernetes context | [~] in_progress | | 2 | List what is typically included in cluster info output | [ ] pending | | 3 | Verify answer completeness and accuracy | [ ] pending | +----+--------------------------------------------------------+-----------------+ Finished #1 in 0.00s, output length: 518 characters (11 lines) - /show 1 to view contents ``` Signed-off-by: Qingchuan Hao <qingchuan.hao@microsoft.com>

- Rename OTEL_AWS_SERVICE → HOLMES_AWS_OSIS_SERVICE with backwards-compat fallback (Comment HolmesGPT#10, svrnm) - Align OTEL_DEBUG with OTEL spec OTEL_LOG_LEVEL=debug with backwards-compat fallback (Comment HolmesGPT#9, svrnm) - Add ml-commons AgentTracer.java GitHub permalink (Comment HolmesGPT#3, kylehounslow) - Document needs_aws_auth() as single source of truth with consumer list (Comment HolmesGPT#4, kylehounslow) - Clarify otel_logging.py: logs NOT exported via OTLP, naming avoids shadowing builtin logging (Comment HolmesGPT#5, kylehounslow) - Explain experimental/ placement: API evolving, removable via try/except no-op fallbacks (Comment HolmesGPT#6, kylehounslow) - Clarify server.py middleware: tracing/metrics init in init_otel() above, middleware only handles per-request spans (Comment HolmesGPT#7, kylehounslow) - Split env var docs into Standard OTEL / Holmes-Specific tables, add missing vars, add OSIS hyperlink (Comments HolmesGPT#1, HolmesGPT#9, HolmesGPT#10) - Fix no-op tracer fallback in server-agui.py for when OTEL is unavailable Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com> Signed-off-by: Megha Goyal <goyamegh@amazon.com>

Replace _render_stream_event (which printed one line per event) with AgenticProgressRenderer that uses Rich Live(transient=True) to show a compact, updating display while tools run. During tool execution: Running #1 fetch_runbook: Fetch Runbook k8s-problems.md (2s) Running #2 kubectl_get: Get pods in default namespace (1s) When the batch completes, the live display vanishes and a summary prints: Ran 2 tool(s): #1 fetch_runbook: Fetch Runbook k8s-problems.md (0.0s, 0 chars) - /show 1 #2 kubectl_get: Get pods in default namespace (1.2s, 4,230 chars) - /show 2 Key behaviors: - Tool names shown immediately from START_TOOL event data - Elapsed times tick every second via background timer - AI messages stop the Live display and print immediately - flush() cleans up on interrupt or stream end https://claude.ai/code/session_01DZDQDQ93Mau8yVVPpPY4qM Signed-off-by: Claude <noreply@anthropic.com>

- Embed timestamp as hidden HTML comment in buildBody() - Extract and display date in "Mar 21, 14:32 UTC" format in run summaries - Add sequential run numbers (#1, #2, ...) to previous runs (newest = highest) - Strip existing run numbers on re-parse to avoid double-numbering - Bold commit SHA with __underscores__ for visual clarity - Handle multiline header matching for timestamp comment prefix Example: 📜 #3 · Run @ __8d93be9__ (#23379683267) — Mar 21, 14:32 UTC https://claude.ai/code/session_01UoX9TpBtKvD4cjogGmXFrt Signed-off-by: Claude <noreply@anthropic.com>

CI run #1 failed all 14 evals with 'Claude Code returned an error result' — the CLI's tool-use requests to the proxy errored. Root cause: the CI model_list routes Anthropic models as openai/anthropic/... (OpenAI-compat, needed by the autoevals classifier), and LiteLLM's openai-compat path mishandles Anthropic tool-use round-trips over /v1/messages. My working local runs used the openrouter/ provider, which handles tool-use correctly. Fix: when building the proxy config, rewrite openai/<vendor>/<model> entries backed by openrouter.ai to openrouter/<vendor>/<model> (LiteLLM's native OpenRouter provider). Plain openai/<model> and non-OpenRouter entries are untouched; the classifier still uses the original model_list. Also harden the bootstrap probe: it now sends a tool-use request and requires a tool_use block in the response, so a broken route fails the session fast with a clear error instead of silently failing every eval. Signed-off-by: Claude <noreply@anthropic.com>

Replaces the soft "don't over-research" line with an enforced cap: read at most 2 doc pages, and the moment the URL/transport/auth are known, stop and produce the guide - explicitly no GitHub repos, no sub-page crawling, no MCP-registry/OpenAPI browsing once the product docs have the details. Mirrors the frontend builder edit (custom-data-source-setup-prompt.ts). Baseline showed the ~235s is sequential LLM turns driven by how many pages Holmes reads, not fetch speed, so capping reads is the main latency lever. Also adds a MODEL override to the builder so the fixture can be regenerated without a live relay token. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_013u97uKXJXNnf4XXEFRsjkP Signed-off-by: Claude <noreply@anthropic.com>

5-iteration staging comparison (Sentry MCP, alon-elish-cluster, Opus 4.7): the research-budget cap cut mean end-to-end time from 234.6s to 113.1s (-52%) and eliminated the GitHub-repo detours, with the correct config (mcp.sentry.dev, streamable-http, bearer/env) produced in all 5 runs - no correctness regression. See COMPARISON.md. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_013u97uKXJXNnf4XXEFRsjkP Signed-off-by: Claude <noreply@anthropic.com>

…o-registry 3 iterations each against staging with the capped + registry-free prompt. No correctness downside: Sentry (mcp.sentry.dev, 87s) and Stripe (mcp.stripe.com, 82s) produce correct hosted-MCP config; Grafana (no vendor-hosted MCP, docs in a GitHub README) correctly produces a self-hosted mcp-grafana + service-account-token setup with no hallucinated endpoint - just slower (152s) because it is genuinely harder. The "no GitHub" line is a soft steer the model overrides when it truly needs the source, so the prompt degrades gracefully rather than emitting wrong config. See COMPARISON.md. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_013u97uKXJXNnf4XXEFRsjkP Signed-off-by: Claude <noreply@anthropic.com>

Production repro (Opus 4.6, baseline without these instructions) showed the model flailing ~87s on call_aws: repeated inline-JSON --filter attempts with stripped quotes, then echo/mkdir file workarounds (no writable fs), then a single-quote attempt that still failed, before finally falling back to a dimension query. So "just single-quote the JSON" is necessary-but-not- sufficient guidance. Rewrite the AWS MCP llm_instructions to lead with the robust path: 1. Prefer shorthand that needs no JSON (--group-by Type=DIMENSION,Key=SERVICE) and filter/aggregate client-side — sidesteps the quoting problem entirely. 2. Do NOT attempt filesystem workarounds (mkdir/echo/redirect/file://) — the MCP sandbox is read-only; that was pure wasted steps in the repro. 3. Only if a JSON arg is truly needed, single-quote the whole value (no double quotes, no backslash-escaping); if it still fails, fall back to #1 instead of retrying variations. 4. Use suggest_aws_commands when unsure. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01FonHqnDVjnkM9qoa7gdM9v Signed-off-by: Claude <noreply@anthropic.com>

… path (cut ~87s of retries) (#2270) ## Problem In a real Ask-Holmes investigation ("show me graphs of AWS Bedrock spend mom…"), the model burned **~87 s (54% of the turn's LLM time)** fighting the `call_aws` tool before it could retrieve any AWS Cost Explorer data. **Reproduced live on production (Opus 4.6) — see the trace below.** The flailing sequence (baseline, without this change): 1. `aws ce get-cost-and-usage … --filter <inline JSON>` → error (⚠️) 2. retry → error 3. `echo '{...}' > /tmp/aws-api-mcp/workdir/bedrock.json` → error (no writable filesystem) 4. `mkdir -p /tmp/aws-api-mcp/workdir && echo …` → error 5. another `--filter` attempt → error; reasoning: *"the tool is stripping double quotes from the JSON"* 6. a single-quote attempt → still error 7. eventually falls back to a dimension/`--query` approach and gets data Root cause: the AWS API MCP server (`aws-api-mcp-server`, upstream `awslabs.aws-api-mcp-server`) tokenizes the command with `shlex.split()`. Unquoted JSON loses its double quotes. But the repro shows single-quoting alone isn't a dependable fix in practice — the model also wastes steps on filesystem workarounds and quote-variation retries. Not an awslabs code bug (standard shell tokenization) — a usage/guidance problem. ## Fix Rewrite the AWS MCP default `llm_instructions` (`helm/holmes/templates/mcp-servers/aws/_helpers.tpl`) to steer toward the **reliable no-JSON path**, in priority order: 1. **Prefer shorthand that needs no JSON** — `--group-by Type=DIMENSION,Key=SERVICE` and filter/aggregate client-side. Sidesteps the quoting problem entirely; this is what the model eventually fell back to anyway. 2. **Do NOT attempt filesystem workarounds** (`mkdir`/`echo`/redirect/`file://`) — the MCP sandbox is read-only. (Directly kills steps 3–4 above.) 3. **Only if a JSON arg is truly needed**, single-quote the whole value (no double quotes, no backslash-escaping); if it still fails, fall back to #1 instead of retrying variations. 4. Use `suggest_aws_commands` when unsure. No MCP server code change — model guidance only. ## Validation status (honest) The behavior is **confirmed reproduced on production** (baseline). A controlled sandbox harness under-reproduced it (current Opus single-quotes fine in a minimal setup), so the reliable test for this change is **deploying it and re-running the same prompt** — the shorthand-first guidance should eliminate the flailing. Not yet re-run with the change applied. ## Context Linear: ROB-589. One of two independent slowdowns found in one slow chat: ~87 s AWS-tool friction (this PR) and a ~30–60 s visualization retry loop (fixed in robusta-frontend#3369 / FRO-175, which has a deterministic client-side repair). ## Follow-up (upstream DX, out of scope) `awslabs/mcp` `call_aws` could add a single-quoting hint to its tool description or return an actionable error on bare-JSON tokens. Draft note prepared. 🤖 Generated with [Claude Code](https://claude.com/claude-code)  ## Summary by CodeRabbit * **Documentation** * Added a new AWS MCP instruction block with guidance for using JSON arguments reliably, including required single-quote wrapping rules and avoidance of double-quotes/backslash escaping. * Provided an example using a JSON-based `--filter`. * Clarified a safer decision flow: prefer shorthand options and client-side filtering/aggregation, use command suggestions first when unsure, and only then execute via `call_aws`.  --------- Signed-off-by: Claude <noreply@anthropic.com> Co-authored-by: Claude <noreply@anthropic.com>

… path (cut ~87s of retries) (#2270) ## Problem In a real Ask-Holmes investigation ("show me graphs of AWS Bedrock spend mom…"), the model burned **~87 s (54% of the turn's LLM time)** fighting the `call_aws` tool before it could retrieve any AWS Cost Explorer data. **Reproduced live on production (Opus 4.6) — see the trace below.** The flailing sequence (baseline, without this change): 1. `aws ce get-cost-and-usage … --filter <inline JSON>` → error (⚠️) 2. retry → error 3. `echo '{...}' > /tmp/aws-api-mcp/workdir/bedrock.json` → error (no writable filesystem) 4. `mkdir -p /tmp/aws-api-mcp/workdir && echo …` → error 5. another `--filter` attempt → error; reasoning: *"the tool is stripping double quotes from the JSON"* 6. a single-quote attempt → still error 7. eventually falls back to a dimension/`--query` approach and gets data Root cause: the AWS API MCP server (`aws-api-mcp-server`, upstream `awslabs.aws-api-mcp-server`) tokenizes the command with `shlex.split()`. Unquoted JSON loses its double quotes. But the repro shows single-quoting alone isn't a dependable fix in practice — the model also wastes steps on filesystem workarounds and quote-variation retries. Not an awslabs code bug (standard shell tokenization) — a usage/guidance problem. ## Fix Rewrite the AWS MCP default `llm_instructions` (`helm/holmes/templates/mcp-servers/aws/_helpers.tpl`) to steer toward the **reliable no-JSON path**, in priority order: 1. **Prefer shorthand that needs no JSON** — `--group-by Type=DIMENSION,Key=SERVICE` and filter/aggregate client-side. Sidesteps the quoting problem entirely; this is what the model eventually fell back to anyway. 2. **Do NOT attempt filesystem workarounds** (`mkdir`/`echo`/redirect/`file://`) — the MCP sandbox is read-only. (Directly kills steps 3–4 above.) 3. **Only if a JSON arg is truly needed**, single-quote the whole value (no double quotes, no backslash-escaping); if it still fails, fall back to #1 instead of retrying variations. 4. Use `suggest_aws_commands` when unsure. No MCP server code change — model guidance only. ## Validation status (honest) The behavior is **confirmed reproduced on production** (baseline). A controlled sandbox harness under-reproduced it (current Opus single-quotes fine in a minimal setup), so the reliable test for this change is **deploying it and re-running the same prompt** — the shorthand-first guidance should eliminate the flailing. Not yet re-run with the change applied. ## Context Linear: ROB-589. One of two independent slowdowns found in one slow chat: ~87 s AWS-tool friction (this PR) and a ~30–60 s visualization retry loop (fixed in robusta-frontend#3369 / FRO-175, which has a deterministic client-side repair). ## Follow-up (upstream DX, out of scope) `awslabs/mcp` `call_aws` could add a single-quoting hint to its tool description or return an actionable error on bare-JSON tokens. Draft note prepared. 🤖 Generated with [Claude Code](https://claude.com/claude-code)  ## Summary by CodeRabbit * **Documentation** * Added a new AWS MCP instruction block with guidance for using JSON arguments reliably, including required single-quote wrapping rules and avoidance of double-quotes/backslash escaping. * Provided an example using a JSON-based `--filter`. * Clarified a safer decision flow: prefer shorthand options and client-side filtering/aggregation, use command suggestions first when unsure, and only then execute via `call_aws`.  --------- Signed-off-by: Claude <noreply@anthropic.com> Co-authored-by: Claude <noreply@anthropic.com> Signed-off-by: Arik Alon <alon.arik@gmail.com>

…60% fewer prompt tokens) (#2301) ## What Condenses the Holmes prompts for Claude 4.x-class models while keeping every behavioral contract. Newer models over-trigger on repetition, ALL-CAPS emphasis and consequence language ("VIOLATION CONSEQUENCES", "FAILURE = INCOMPLETE INVESTIGATION"), so this cuts the scaffolding and keeps the domain facts, output contracts and non-obvious constraints. **Sizes (cl100k tokens, representative server config with core_investigation enabled):** - System prompt: **~15.3K → ~6.1K (−60%)**; full request payload incl. tool schemas: **−45%** - `generic_ask.jinja2` TodoWrite/phases section: 2,536 → 331 tokens - `investigator_instructions.jinja2`: 13.9KB → 1.7KB (was a verbatim coding-assistant prompt with dark-mode/npm examples and a "4 lines max" rule conflicting with the style guide) - `conversation_history_compaction.jinja2`: ~1,200 → ~390 tokens - Disabled-toolsets listing: one line per toolset; docs-URL pattern stated once, only exceptions listed - bash prefix guidance ~40% smaller; prometheus/robusta llm-instructions deduped (rules stated 2-3× → once); `fetch_pod_logs` + kubernetes.yaml tool schemas tightened; per-message TodoWrite reminder ~120 → ~45 tokens ## Verification (live evals via OpenRouter, k3s + Elasticsearch Cloud) Baselined every suite on the old prompts first, then re-ran per change group. Models chosen by production traffic distribution (opus-4.6 #1, opus-4.7 #2, haiku-4.5 #3, sonnet-4.6). | Suite | opus-4.6 | opus-4.7 | haiku-4.5 | sonnet-4.6 | |---|---|---|---|---| | multicluster/transparency ES (13) | 12/13 baseline → 12/13 (same pre-existing 235 failure) | 13/13 | 9/13 baseline → 7–9/13 (within its 7–11 variance band, n=7 runs) | 10/13 | | k8s regression (13) | 12/13 baseline → 12–13/13 (`24_misconfigured_pvc` baseline failure now passes) | — | — | — | | ES quirks/skills (6) | 6/6 → 6/6 | 6/7 | n/a (0/6 at baseline, pre-existing) | 6/7 | | bash prefix rules (9) | 9/9 | 8/9 (205 flaky ~1/3 runs on all configs) | — | — | | compaction (7) | 7/7 baseline → 7/7 | — | — | — | A more aggressive rewrite (extra ~480 tokens saved) was built and **rejected by evals**: haiku-4.5 averaged 6.75/13 with it vs 8.6/13 (n=9 runs), while opus was unaffected — weaker models need the extra structure, so this PR keeps the version that's safe across the whole prod model mix. **Not eval-verified here** (no creds/infra in the dev sandbox): datadog/newrelic/coralogix instructions (untouched), OpenSearch PPL query-assist docs (untouched — at 56KB it's the biggest remaining candidate, needs a PPL eval first), prometheus eval fixtures can't run in the sandbox so the prometheus edits are strictly dedup-only. The `/eval` comment on this PR covers these areas in CI. **Known-flaky tests observed during this work** (fail ~1/3 runs on baseline AND reduced prompts): `254_elasticsearch_dr_test_log_check`, `205_bash_deployment_logs_all_pods`; `235_elasticsearch_cluster_mismatch` fails consistently on every config/model except opus-4.7. Worth hardening separately. ## Test changes - `test_toolsets_instructions.py`, `test_fetch_logs.py`: string pins updated to the new wording (same semantics). 🤖 Generated with [Claude Code](https://claude.com/claude-code) https://claude.ai/code/session_01XgDuG2hb6jmSqYpD7iGUkd --- _Generated by [Claude Code](https://claude.ai/code/session_01XgDuG2hb6jmSqYpD7iGUkd)_  ## Summary by CodeRabbit * **Improvements** * Strengthened TodoWrite multi-step workflows, task state handling, and evidence-backed completion with clearer blocker follow-ups. * Refined user-facing prompt and tool instructions for log retrieval, including consistent summaries, pod/namespace identification, and timeframe behavior. * Updated guidance for Bash command approval/allow-lists, Kubernetes/Prometheus/Robusta investigations, permissions-focused troubleshooting, and conversation-history compaction. * Improved messaging when toolsets are disabled or unavailable, including clearer next actions. * **Tests** * Updated prompt wording assertions and added a new TodoWrite multi-step audit regression fixture.  --------- Signed-off-by: Claude <noreply@anthropic.com> Co-authored-by: Claude <noreply@anthropic.com>

balloon72

noop

RobertSzefler requested a review from aantn May 30, 2024 13:49

Some missing code

6f88720

RobertSzefler force-pushed the fix/missing-code branch from 05146fe to 6f88720 Compare May 30, 2024 13:51

aantn approved these changes May 30, 2024

View reviewed changes

aantn merged commit 7c55711 into master May 30, 2024

aantn deleted the fix/missing-code branch May 30, 2024 14:44

moshemorad added a commit that referenced this pull request May 29, 2025

Revert test ask holmes #1

068bc48

dotanalter mentioned this pull request Aug 14, 2025

feat: Add --quiet/-q option to ask command to suppress verbose output #847

Closed

github-actions Bot mentioned this pull request Feb 14, 2026

Unwrap ExceptionGroup to surface root MCP errors #1561

Merged

shuaiganzi mentioned this pull request Mar 6, 2026

An error occurred when configuring Prometheus：TypeError: '>' not supported between instances of 'str' and 'int' #1677

Closed

claude Bot mentioned this pull request Mar 21, 2026

Add support for image content in MCP tool results #1803

Merged

alonelish mentioned this pull request Jul 8, 2026

ROB-589 Steer call_aws JSON-argument commands to the reliable no-JSON path (cut ~87s of retries) #2270

Merged

naomi-robusta mentioned this pull request Jul 20, 2026

ROB-635 ROB-674 Reduce system prompt and tool description sizes (~50-60% fewer prompt tokens) #2301

Merged

github-actions Bot mentioned this pull request Jul 22, 2026

Add deterministic tool-result eviction (context editing) #2313

Closed

balloon72 reviewed Aug 3, 2026

View reviewed changes

github-actions Bot mentioned this pull request Aug 21, 2026

Add eval 289: CPU throttling caused by cache-key mismatch bug #2310

Open

Navigation Menu

Some missing code - #1

Some missing code#1
aantn merged 1 commit into
masterfrom
fix/missing-code

RobertSzefler commented May 30, 2024

Uh oh!

balloon72 left a comment

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

3 participants

Conversation

RobertSzefler commented May 30, 2024

Uh oh!

balloon72 left a comment

Choose a reason for hiding this comment

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

3 participants