ROB-589 Steer call_aws JSON-argument commands to the reliable no-JSON path (cut ~87s of retries)

[alonelish]

Problem

In a real Ask-Holmes investigation ("show me graphs of AWS Bedrock spend mom…"), the model burned ~87 s (54% of the turn's LLM time) fighting the call_aws tool before it could retrieve any AWS Cost Explorer data. Reproduced live on production (Opus 4.6) — see the trace below.

The flailing sequence (baseline, without this change):

1. aws ce get-cost-and-usage … --filter <inline JSON> → error (⚠️)
2. retry → error
3. echo '{...}' > /tmp/aws-api-mcp/workdir/bedrock.json → error (no writable filesystem)
4. mkdir -p /tmp/aws-api-mcp/workdir && echo … → error
5. another --filter attempt → error; reasoning: "the tool is stripping double quotes from the JSON"
6. a single-quote attempt → still error
7. eventually falls back to a dimension/--query approach and gets data

Root cause: the AWS API MCP server (aws-api-mcp-server, upstream awslabs.aws-api-mcp-server) tokenizes the command with shlex.split(). Unquoted JSON loses its double quotes. But the repro shows single-quoting alone isn't a dependable fix in practice — the model also wastes steps on filesystem workarounds and quote-variation retries. Not an awslabs code bug (standard shell tokenization) — a usage/guidance problem.

Fix

Rewrite the AWS MCP default llm_instructions (helm/holmes/templates/mcp-servers/aws/_helpers.tpl) to steer toward the reliable no-JSON path, in priority order:

1. Prefer shorthand that needs no JSON — --group-by Type=DIMENSION,Key=SERVICE and filter/aggregate client-side. Sidesteps the quoting problem entirely; this is what the model eventually fell back to anyway.
2. Do NOT attempt filesystem workarounds (mkdir/echo/redirect/file://) — the MCP sandbox is read-only. (Directly kills steps 3–4 above.)
3. Only if a JSON arg is truly needed, single-quote the whole value (no double quotes, no backslash-escaping); if it still fails, fall back to Some missing code #1 instead of retrying variations.
4. Use suggest_aws_commands when unsure.

No MCP server code change — model guidance only.

Validation status (honest)

The behavior is confirmed reproduced on production (baseline). A controlled sandbox harness under-reproduced it (current Opus single-quotes fine in a minimal setup), so the reliable test for this change is deploying it and re-running the same prompt — the shorthand-first guidance should eliminate the flailing. Not yet re-run with the change applied.

Context

Linear: ROB-589. One of two independent slowdowns found in one slow chat: ~87 s AWS-tool friction (this PR) and a ~30–60 s visualization retry loop (fixed in robusta-frontend#3369 / FRO-175, which has a deterministic client-side repair).

Follow-up (upstream DX, out of scope)

awslabs/mcp call_aws could add a single-quoting hint to its tool description or return an actionable error on bare-JSON tokens. Draft note prepared.

🤖 Generated with Claude Code

Summary by CodeRabbit

• Documentation
  • Added a new AWS MCP instruction block with guidance for using JSON arguments reliably, including required single-quote wrapping rules and avoidance of double-quotes/backslash escaping.
  • Provided an example using a JSON-based --filter.
  • Clarified a safer decision flow: prefer shorthand options and client-side filtering/aggregation, use command suggestions first when unsure, and only then execute via call_aws.

[coderabbitai]

No actionable comments were generated in the recent review. 🎉

ℹ️ Recent review info

⚙️ Run configuration

Configuration used: Organization UI

Review profile: CHILL

Plan: Pro

Run ID: 0a1fca4e-6a82-4dc3-b732-dc8b28742f39

📥 Commits

Reviewing files that changed from the base of the PR and between c9aa752 and 50c12ee.

📒 Files selected for processing (1)

• helm/holmes/templates/mcp-servers/aws/_helpers.tpl

✅ Files skipped from review due to trivial changes (1)

• helm/holmes/templates/mcp-servers/aws/_helpers.tpl

---

Walkthrough

This change adds an AWS MCP instruction block in the Helm template that explains how to pass JSON arguments to call_aws, including quoting rules, an example, and fallback guidance when command syntax is unclear.

Changes

AWS MCP call_aws Instructions

Layer / File(s)	Summary
JSON argument quoting guidance helm/holmes/templates/mcp-servers/aws/_helpers.tpl	Adds a “Cost Explorer and other JSON-argument commands (READ THIS FIRST)” section with single-quote JSON handling, a --filter example, an avoidance-first workflow using shorthand options, and direction to use suggest_aws_commands when needed.

Estimated code review effort: 1 (Trivial) | ~3 minutes

🚥 Pre-merge checks | ✅ 5

✅ Passed checks (5 passed)

Check name	Status	Explanation
Description Check	✅ Passed	Check skipped - CodeRabbit’s high-level summary is enabled.
Title check	✅ Passed	The title clearly reflects the main change: steering call_aws away from JSON arguments toward the no-JSON path to reduce retries.
Docstring Coverage	✅ Passed	No functions found in the changed files to evaluate docstring coverage. Skipping docstring coverage check.
Linked Issues check	✅ Passed	Check skipped because no linked issues were found for this pull request.
Out of Scope Changes check	✅ Passed	Check skipped because no linked issues were found for this pull request.

---

Thanks for using CodeRabbit! It's free for OSS, and your support helps us grow. If you like it, consider giving us a shout-out.

❤️ Share

• X
• Mastodon
• Reddit
• LinkedIn

_{Comment @coderabbitai help to get the list of available commands.}

[github-actions]

✅ Docker images ready for 8e888cda0 (built in 55s)

• us-central1-docker.pkg.dev/robusta-development/temporary-builds/holmes:8e888cda0
• us-central1-docker.pkg.dev/robusta-development/temporary-builds/holmes-operator:8e888cda0

⚠️ Warning: does not support ARM (ARM images are built on release only - not on every PR)

Use these tags to pull the images for testing.

📋 Copy commands

⚠️ Temporary images are deleted after 30 days. Copy to a permanent registry before using them:

gcloud auth configure-docker us-central1-docker.pkg.dev
docker pull us-central1-docker.pkg.dev/robusta-development/temporary-builds/holmes:8e888cda0
docker tag us-central1-docker.pkg.dev/robusta-development/temporary-builds/holmes:8e888cda0 me-west1-docker.pkg.dev/robusta-development/development/holmes-dev:8e888cda0
docker push me-west1-docker.pkg.dev/robusta-development/development/holmes-dev:8e888cda0
docker pull us-central1-docker.pkg.dev/robusta-development/temporary-builds/holmes-operator:8e888cda0
docker tag us-central1-docker.pkg.dev/robusta-development/temporary-builds/holmes-operator:8e888cda0 me-west1-docker.pkg.dev/robusta-development/development/holmes-operator-dev:8e888cda0
docker push me-west1-docker.pkg.dev/robusta-development/development/holmes-operator-dev:8e888cda0

Patch Helm values in one line (choose the chart you use):

HolmesGPT chart:

helm upgrade --install holmesgpt ./helm/holmes \
  --set registry=me-west1-docker.pkg.dev/robusta-development/development \
  --set image=holmes-dev:8e888cda0 \
  --set operator.registry=me-west1-docker.pkg.dev/robusta-development/development \
  --set operator.image=holmes-operator-dev:8e888cda0

Robusta wrapper chart:

helm upgrade --install robusta robusta/robusta \
  --reuse-values \
  --set holmes.registry=me-west1-docker.pkg.dev/robusta-development/development \
  --set holmes.image=holmes-dev:8e888cda0 \
  --set holmes.operator.registry=me-west1-docker.pkg.dev/robusta-development/development \
  --set holmes.operator.image=holmes-operator-dev:8e888cda0

[netlify]

✅ Deploy Preview for holmes-docs ready!

Name	Link
🔨 Latest commit	50c12ee
🔍 Latest deploy log	https://app.netlify.com/projects/holmes-docs/deploys/6a4e1a7eaa78090008ee5351
😎 Deploy Preview	https://deploy-preview-2270--holmes-docs.netlify.app
📱 Preview on mobile	Toggle QR Code... Use your smartphone camera to open QR code link.

---

To edit notification comments on pull requests, go to your Netlify project configuration.

[coderabbitai]

🧹 Nitpick comments (1)

helm/holmes/templates/mcp-servers/aws/_helpers.tpl (1)

17-17: 📐 Maintainability & Code Quality | 🔵 Trivial | ⚡ Quick win

Use placeholders instead of fixed dates in the example.

The hard-coded 2026-01-01 / 2026-07-08 dates will age quickly and make the guidance stale. A placeholder or relative-date example would keep the instruction accurate.

♻️ Proposed fix

-  `aws ce get-cost-and-usage --time-period Start=2026-01-01,End=2026-07-08 --granularity MONTHLY --metrics UnblendedCost --filter '{"Dimensions":{"Key":"SERVICE","Values":["Amazon Bedrock"]}}'`
+  `aws ce get-cost-and-usage --time-period Start=<START_DATE>,End=<END_DATE> --granularity MONTHLY --metrics UnblendedCost --filter '{"Dimensions":{"Key":"SERVICE","Values":["Amazon Bedrock"]}}'`

🤖 Prompt for AI Agents

Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.

In `@helm/holmes/templates/mcp-servers/aws/_helpers.tpl` at line 17, Replace the
hard-coded dates in the AWS Cost Explorer example inside the helper template
with placeholders or a relative-date example so the guidance stays current.
Update the example command in the AWS helper content to use variable or generic
date values instead of fixed values, keeping the rest of the `aws ce
get-cost-and-usage` example unchanged.

🤖 Prompt for all review comments with AI agents

Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.

Nitpick comments:
In `@helm/holmes/templates/mcp-servers/aws/_helpers.tpl`:
- Line 17: Replace the hard-coded dates in the AWS Cost Explorer example inside
the helper template with placeholders or a relative-date example so the guidance
stays current. Update the example command in the AWS helper content to use
variable or generic date values instead of fixed values, keeping the rest of the
`aws ce get-cost-and-usage` example unchanged.

---

ℹ️ Review info

⚙️ Run configuration

Configuration used: Organization UI

Review profile: CHILL

Plan: Pro

Run ID: c3dfe149-66b9-49f4-9aab-7bd3b7df847c

📥 Commits

Reviewing files that changed from the base of the PR and between 84cb39c and 86ec60a.

📒 Files selected for processing (1)

• helm/holmes/templates/mcp-servers/aws/_helpers.tpl