Skip to content

Prometheus docs improvements - #1398

Open
aantn wants to merge 13 commits into
masterfrom
claude/concise-prometheus-docs-6Cf2q
Open

aantn wants to merge 13 commits into
masterfrom
claude/concise-prometheus-docs-6Cf2q

Conversation

@aantn

@aantn aantn commented Jan 21, 2026 •

Copy link
Copy Markdown
Collaborator
  • Add jump links to different Prometheus flavors at top of page
  • Show all advanced options in YAML examples (commented out)
  • Add collapsible section with AI prompt to help users find Prometheus URL
  • Consolidate config explanations into inline comments
  • Add missing query_response_size_limit_pct option
  • Remove redundant sections and verbose explanations
  • Convert troubleshooting to concise table format
  • Fact-check defaults against source code

Summary by CodeRabbit

  • Documentation
    • Reworked Prometheus docs into a unified PromQL-focused toolset with a Jump-to navigation for provider sections.
    • Replaced provider-specific setup with generic prompts, an interactive flow to provide/auto-detect Prometheus URL, and a dedicated "Finding your Prometheus URL" section.
    • Added a Validation section with example queries and testing commands.
    • Expanded inline configuration hints, troubleshooting notes, and environment-variable alternatives (PROMETHEUS_URL, PROMETHEUS_AUTH_HEADER).

- Add jump links to different Prometheus flavors at top of page
- Show all advanced options in YAML examples (commented out)
- Add collapsible section with AI prompt to help users find Prometheus URL
- Consolidate config explanations into inline comments
- Add missing `query_response_size_limit_pct` option
- Remove redundant sections and verbose explanations
- Convert troubleshooting to concise table format
- Fact-check defaults against source code

Signed-off-by: Claude <noreply@anthropic.com>
@netlify

netlify Bot commented Jan 21, 2026 •

Copy link
Copy Markdown

✅ Deploy Preview for holmes-docs ready!

Name Link
🔨 Latest commit fe265c0
🔍 Latest deploy log https://app.netlify.com/projects/holmes-docs/deploys/69aa7a3dd05a0f0008873bd1
😎 Deploy Preview https://deploy-preview-1398--holmes-docs.netlify.app
📱 Preview on mobile
Toggle QR Code...

QR Code

Use your smartphone camera to open QR code link.

To edit notification comments on pull requests, go to your Netlify project configuration.

@linux-foundation-easycla

linux-foundation-easycla Bot commented Jan 21, 2026 •

Copy link
Copy Markdown

CLA Signed

The committers listed above are authorized under a signed CLA.

  • ✅ login: claude / name: Claude (fe265c0)

@coderabbitai

coderabbitai Bot commented Jan 21, 2026 •

Copy link
Copy Markdown
Contributor

Note

Reviews paused

It looks like this branch is under active development. To avoid overwhelming you with review comments due to an influx of new commits, CodeRabbit has automatically paused this review. You can configure this behavior by changing the reviews.auto_review.auto_pause_after_reviewed_commits setting.

Use the following commands to manage reviews:

  • @coderabbitai resume to resume automatic reviews.
  • @coderabbitai review to trigger a single review.

Use the checkboxes below for quick actions:

  • ▶️ Resume reviews
  • 🔍 Trigger review

Walkthrough

Reworks Prometheus documentation into a unified, PromQL-focused toolset guide, replacing provider-specific blocks with a consolidated configuration and prompts, adding environment-variable alternatives, validation and “Finding your Prometheus URL” guidance, expanded options (headers, timeouts, cache, SSL, token limits), and reorganized provider guidance.

Changes

Cohort / File(s) Summary
Prometheus docs
docs/data-sources/builtin-toolsets/prometheus.md
Comprehensive documentation rewrite: unified PromQL-centric toolset config, replaced provider-specific blocks with generic prompts and consolidated toolset syntax; added navigation, validation examples, "Finding your Prometheus URL" section (port-forward and DNS methods); expanded configuration options (additional_headers, timeouts, caches, ssl_verify, response/token limits), introduced PROMETHEUS_URL / PROMETHEUS_AUTH_HEADER env alternatives, reorganized provider guidance for Coralogix, AWS AMP, Azure, Google Managed, Grafana Cloud; updated troubleshooting and capability notes.

Estimated code review effort

🎯 3 (Moderate) | ⏱️ ~20 minutes

Possibly related PRs

Suggested reviewers

  • moshemorad
  • arikalon1
🚥 Pre-merge checks | ✅ 3
✅ Passed checks (3 passed)
Check name Status Explanation
Description Check ✅ Passed Check skipped - CodeRabbit’s high-level summary is enabled.
Title check ✅ Passed The title 'Prometheus docs improvements' accurately describes the main change—a comprehensive rework of Prometheus documentation including restructuring, configuration enhancements, and content consolidation.
Docstring Coverage ✅ Passed No functions found in the changed files to evaluate docstring coverage. Skipping docstring coverage check.

✏️ Tip: You can configure your own custom pre-merge checks in the settings.


Thanks for using CodeRabbit! It's free for OSS, and your support helps us grow. If you like it, consider giving us a shout-out.

❤️ Share

Comment @coderabbitai help to get the list of available commands and usage tips.

@coderabbitai coderabbitai Bot left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Actionable comments posted: 1

🤖 Fix all issues with AI agents
In `@docs/data-sources/builtin-toolsets/prometheus.md`:
- Around line 45-53: Replace the current indented prompt block (the lines
starting with "I need to configure HolmesGPT to connect to Prometheus..." and
the "Run: kubectl get svc -A | grep -i prom" line) with an explicit fenced code
block using triple backticks and a language tag (e.g., ```text) so the entire
prompt and the kubectl command are enclosed between the opening and closing
fences to satisfy markdownlint.
🧹 Nitpick comments (1)
docs/data-sources/builtin-toolsets/prometheus.md (1)

30-33: Add blank line between bold label and list to satisfy MkDocs rendering.

MkDocs won’t render lists correctly without a blank line after bold text. Please insert a blank line after each label before the list.

✅ Suggested edits
-**Environment variables** (alternative to config file):
-
-- `PROMETHEUS_URL`: Prometheus server URL
-- `PROMETHEUS_AUTH_HEADER`: Authorization header value (e.g., `Bearer token123`)
+**Environment variables** (alternative to config file):
+
+- `PROMETHEUS_URL`: Prometheus server URL
+- `PROMETHEUS_AUTH_HEADER`: Authorization header value (e.g., `Bearer token123`)

-**Setup:**
-
-1. Find your [regional PromQL endpoint](https://coralogix.com/docs/integrations/coralogix-endpoints/#promql)
+**Setup:**
+
+1. Find your [regional PromQL endpoint](https://coralogix.com/docs/integrations/coralogix-endpoints/#promql)

-**Notes:**
-
-- Automatically uses SigV4 authentication when `aws_region` is present
+**Notes:**
+
+- Automatically uses SigV4 authentication when `aws_region` is present

-**Prerequisites:**
-
-- Google Managed Prometheus enabled
+**Prerequisites:**
+
+- Google Managed Prometheus enabled

-**Setup:**
-
-1. Create service account token in Grafana Cloud (Administration → Service accounts)
+**Setup:**
+
+1. Create service account token in Grafana Cloud (Administration → Service accounts)

As per coding guidelines, add a blank line between bold text and a list in MkDocs docs.

Also applies to: 99-103, 126-130, 144-148, 165-169

Comment thread docs/data-sources/builtin-toolsets/prometheus.md Outdated
- Resolve merge conflicts with master
- Add new Azure Managed Prometheus section in concise format
- Update jump links to include Azure
- Add SSL error troubleshooting entry

Signed-off-by: Claude <noreply@anthropic.com>
@github-actions

github-actions Bot commented Jan 21, 2026 •

Copy link
Copy Markdown
Contributor

✅ Docker image ready for 8c55aa7 (built in 4m 21s)

⚠️ Warning: does not support ARM (ARM images are built on release only - not on every PR)

Use this tag to pull the image for testing.

📋 Copy commands

⚠️ Temporary images are deleted after 30 days. Copy to a permanent registry before using them:

gcloud auth configure-docker us-central1-docker.pkg.dev
docker pull us-central1-docker.pkg.dev/robusta-development/temporary-builds/holmes:8c55aa7
docker tag us-central1-docker.pkg.dev/robusta-development/temporary-builds/holmes:8c55aa7 me-west1-docker.pkg.dev/robusta-development/development/holmes-dev:8c55aa7
docker push me-west1-docker.pkg.dev/robusta-development/development/holmes-dev:8c55aa7

Patch Helm values in one line (choose the chart you use):

HolmesGPT chart:

helm upgrade --install holmesgpt ./helm/holmes \
  --set registry=me-west1-docker.pkg.dev/robusta-development/development \
  --set image=holmes-dev:8c55aa7

Robusta wrapper chart:

helm upgrade --install robusta robusta/robusta \
  --reuse-values \
  --set holmes.registry=me-west1-docker.pkg.dev/robusta-development/development \
  --set holmes.image=holmes-dev:8c55aa7

@github-actions

github-actions Bot commented Jan 21, 2026 •

Copy link
Copy Markdown
Contributor

📂 Previous Runs

📜 Run @ 2dfe3b9 (#21562639115)

✅ Results of HolmesGPT evals

Automatically triggered by commit 2dfe3b9 on branch claude/concise-prometheus-docs-6Cf2q

View workflow logs

Results of HolmesGPT evals

  • ask_holmes: 9/9 test cases were successful, 0 regressions
Status Test case Time Turns Tools Cost
✅ 09_crashpod 33.1s ±0% 6 11 $0.2431
✅ 101_loki_historical_logs_pod_deleted 53.3s ↑27% 7 13 $0.3002
✅ 111_pod_names_contain_service 30.5s ±0% 5 11 $0.2175
✅ 12_job_crashing 31.9s ±0% 5 11 $0.2337
✅ 162_get_runbooks 40.3s ±0% 6 11 $0.2860
✅ 176_network_policy_blocking_traffic_no_runbooks 45.5s ±0% 6 18 $0.2883
✅ 24_misconfigured_pvc 31.5s ↓10% 5 14 $0.2380
✅ 43_current_datetime_from_prompt 5.4s ±0% 1 — $0.1051
✅ 61_exact_match_counting 15.9s ±0% 4 4 $0.1587
Total 31.9s avg 5.0 avg 11.6 avg $2.0705

Time/Cost columns show % change vs historical average (↑slower/costlier, ↓faster/cheaper). Changes under 10% shown as ±0%.

Historical Comparison Details

Filter: excluding branch 'claude/concise-prometheus-docs-6Cf2q'

Status: Success - 38 test/model combinations loaded

Experiments compared (30):

Comparison indicators:

  • ±0% — diff under 10% (within noise threshold)
  • ↑N%/↓N% — diff 10-25%
  • ↑N%/↓N% — diff over 25% (significant)
📜 Run @ 0de642b (#21492914216)

✅ Results of HolmesGPT evals

Automatically triggered by commit 0de642b on branch claude/concise-prometheus-docs-6Cf2q

View workflow logs

⚠️ No eval report was generated.

📜 Run @ eda15df (#21485759258)

✅ Results of HolmesGPT evals

Automatically triggered by commit eda15df on branch claude/concise-prometheus-docs-6Cf2q

View workflow logs

⚠️ No eval report was generated.

📜 Run @ 7d57365 (#21213417080)

✅ Results of HolmesGPT evals

Automatically triggered by commit 7d57365 on branch claude/concise-prometheus-docs-6Cf2q

View workflow logs

Results of HolmesGPT evals

  • ask_holmes: 9/9 test cases were successful, 0 regressions
Status Test case Time Turns Tools Cost
✅ 09_crashpod 33.4s ±0% 6 13 $0.1139
✅ 101_loki_historical_logs_pod_deleted 44.7s ↓27% 7 15 $0.1530
✅ 111_pod_names_contain_service 43.5s ±0% 8 19 $0.1463
✅ 12_job_crashing 43.8s ±0% 8 17 $0.1473
✅ 162_get_runbooks 43.4s ±0% 7 13 $0.1397
✅ 176_network_policy_blocking_traffic_no_runbooks 38.4s ±0% 7 15 $0.1288
✅ 24_misconfigured_pvc 34.7s ±0% 6 17 $0.1148
✅ 43_current_datetime_from_prompt 3.2s ±0% 1 — $0.0085
✅ 61_exact_match_counting 10.7s ±0% 3 3 $0.0326
Total 32.9s avg 5.9 avg 14.0 avg $0.9848

Time/Cost columns show % change vs historical average (↑slower/costlier, ↓faster/cheaper). Changes under 10% shown as ±0%.

Historical Comparison Details

Filter: excluding branch 'claude/concise-prometheus-docs-6Cf2q'

Status: Success - 25 test/model combinations loaded

Experiments compared (30):

Comparison indicators:

  • ±0% — diff under 10% (within noise threshold)
  • ↑N%/↓N% — diff 10-25%
  • ↑N%/↓N% — diff over 25% (significant)

✅ Results of HolmesGPT evals

Automatically triggered by commit 4657f46 on branch claude/concise-prometheus-docs-6Cf2q

View workflow logs

Results of HolmesGPT evals

  • ask_holmes: 9/9 test cases were successful, 0 regressions
Status Test case Time Turns Tools Cost
✅ 09_crashpod 34.7s 5 11 $0.2453
✅ 101_loki_historical_logs_pod_deleted 40.7s 6 9 $0.2505
✅ 111_pod_names_contain_service 36.8s 5 12 $0.2362
✅ 112_find_pvcs_by_uuid 38.1s 7 8 $0.2656
✅ 12_job_crashing 52.1s 7 14 $0.2898
✅ 176_network_policy_blocking_traffic_no_runbooks 53.4s 7 17 $0.3090
✅ 24_misconfigured_pvc 41.7s 7 15 $0.2694
✅ 43_current_datetime_from_prompt 5.3s 1 — $0.1113
✅ 61_exact_match_counting 13.6s 3 2 $0.1489
Total 35.2s avg 5.3 avg 11.0 avg $2.1260
📖 Legend
Icon Meaning
✅ The test was successful
➖ The test was skipped
⚠️ The test failed but is known to be flaky or known to fail
🚧 The test had a setup failure (not a code regression)
🔧 The test failed due to mock data issues (not a code regression)
🚫 The test was throttled by API rate limits/overload
❌ The test failed and should be fixed before merging the PR
🔄 Re-run evals manually

⚠️ Warning: /eval comments always run using the workflow from master, not from this PR branch. If you modified the GitHub Action (e.g., added secrets or env vars), those changes won't take effect.

To test workflow changes, use the GitHub CLI or Actions UI instead:

gh workflow run eval-regression.yaml --repo HolmesGPT/holmesgpt --ref claude/concise-prometheus-docs-6Cf2q -f markers=regression -f filter=

Option 1: Comment on this PR with /eval:

/eval
markers: regression

Or with more options (one per line):

/eval
model: gpt-4o
markers: regression
filter: 09_crashpod
iterations: 5

Run evals on a different branch (e.g., master) for comparison:

/eval
branch: master
markers: regression
Option Description
model Model(s) to test (default: same as automatic runs)
markers Pytest markers (no default - runs all tests!)
filter Pytest -k filter (use /list to see valid eval names)
iterations Number of runs, max 10
branch Run evals on a different branch (for cross-branch comparison)

Quick re-run: Use /rerun to re-run the most recent /eval on this PR with the same parameters.

Option 2: Trigger via GitHub Actions UI → "Run workflow"

🏷️ Valid markers

benchmark, chain-of-causation, compaction, confluence, context_window, coralogix, counting, database, datadog, datetime, easy, elasticsearch, embeds, fast, frontend, grafana-dashboard, hard, integration, kafka, kubernetes, leaked-information, logs, loki, medium, metrics, network, newrelic, no-cicd, numerical, one-test, port-forward, prometheus, question-answer, regression, runbooks, slackbot, storage, toolset-limitation, traces, transparency


Commands: /eval · /rerun · /list

CLI: gh workflow run eval-regression.yaml --repo HolmesGPT/holmesgpt --ref claude/concise-prometheus-docs-6Cf2q -f markers=regression -f filter=

@coderabbitai coderabbitai Bot left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Actionable comments posted: 2

🤖 Fix all issues with AI agents
In `@docs/data-sources/builtin-toolsets/prometheus.md`:
- Around line 57-58: Add a blank line between the bold text line "**Port-forward
for testing:**" and the following code block delimiter "```bash" so the Markdown
renders correctly in MkDocs; update the section around that bold heading (the
bold line and the subsequent code fence) to include one empty line between them.
- Around line 64-65: Add a single blank line between the bold heading "**Get
internal cluster URL:**" and the following code block fence (```bash) so the
Markdown renderer (MkDocs) correctly treats the code block; update the section
in docs/data-sources/builtin-toolsets/prometheus.md by inserting an empty line
after the "**Get internal cluster URL:**" line.
🧹 Nitpick comments (1)
docs/data-sources/builtin-toolsets/prometheus.md (1)

72-72: Consider adjusting table pipe spacing for consistency.

Markdownlint flags inconsistent spacing around table pipes. While this doesn't affect functionality, adding consistent spaces improves formatting consistency.

📋 Example fix for line 72
-|------|-------------|
+| ----- | ----------- |

Apply similar spacing adjustments to line 212.

Also applies to: 212-212

Comment thread docs/data-sources/builtin-toolsets/prometheus.md Outdated
Comment thread docs/data-sources/builtin-toolsets/prometheus.md Outdated
claude and others added 5 commits January 29, 2026 20:03
…nal_headers

- Restore yaml-toolset-config code fence for main config block
- Use renamed additional_headers field (not headers)
- Keep all concise improvements: jump links, commented-out advanced
  options, collapsible URL finder, AI prompt, troubleshooting table
- Add missing query_response_size_limit_pct option

Signed-off-by: Claude <noreply@anthropic.com>
- Keep original Prerequisites and Finding your Prometheus URL sections
- Add Option 3 (Ask Holmes) to URL discovery
- Use yaml-toolset-config fence and additional_headers throughout
- Keep concise provider sections with commented-out advanced options

Signed-off-by: Claude <noreply@anthropic.com>
@github-actions

Copy link
Copy Markdown
Contributor

🔬 CLI Performance Benchmark

🟡 Startup Time (no LLM)

Measures holmes version execution time (imports + initialization)

Metric PR Master Change
Cold Start 10.98s 10.73s +2.4%
Warm Mean 5.09s 4.73s +7.5%
Warm Min 5.05s 4.69s
Warm Max 5.13s 4.77s

🟡 Full CLI with LLM

Measures holmes ask execution time (OpenRouter + Haiku 4.5)

Metric PR Master Change
Cold Start 28.93s 36.37s -20.5%
Warm Mean 7.92s 7.66s +3.5%
Warm Min 7.14s 7.02s
Warm Max 8.36s 8.28s

PR: 8c55aa7b | Master: abb2c746 | Iterations: 5

@coderabbitai coderabbitai Bot left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

🤖 Prompt for all review comments with AI agents
Verify each finding against the current code and only fix it if needed.

Duplicate comments:
In `@docs/data-sources/builtin-toolsets/prometheus.md`:
- Around line 71-78: The fenced code block starting with ``` that contains the
kubectl command needs a language specifier to satisfy MD040: replace the opening
fence of the block (the triple backticks before "I need to configure
HolmesGPT...") with a language-tagged fence such as ```text; keep the block
content and closing ``` unchanged so the command and prompt remain intact.

The prompt can be pasted into any AI coding assistant (Claude Code,
Cursor, etc.) without assuming a specific tool.

Signed-off-by: Claude <noreply@anthropic.com>
Generates 3 tabs (Holmes CLI, Holmes Helm Chart, Robusta Helm Chart)
for every provider config block, not just the main one.

Signed-off-by: Claude <noreply@anthropic.com>

@coderabbitai coderabbitai Bot left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

🤖 Prompt for all review comments with AI agents
Verify each finding against the current code and only fix it if needed.

Duplicate comments:
In `@docs/data-sources/builtin-toolsets/prometheus.md`:
- Around line 69-82: The fenced code block inside the "??? tip \"Prompt for AI
agent\"" admonition is being treated as an indented code block (MD046); add an
explicit language tag to the opening fence to make it a fenced block (e.g.,
change the opening "```" to "```text" or "```bash") so the snippet under the
admonition in prometheus.md is recognized as fenced rather than indented; update
the opening fence for the block that begins with "I need to find my Prometheus
server URL..." accordingly.

Match master's pattern where setup instructions come before the
YAML config block for Coralogix, Azure, Google, and Grafana Cloud.

Signed-off-by: Claude <noreply@anthropic.com>

@coderabbitai coderabbitai Bot left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

🧹 Nitpick comments (1)
docs/data-sources/builtin-toolsets/prometheus.md (1)

38-42: Optional: replace ## Validation header with bold text.

## Validation is a ##-level section for a single one-liner command. Per coding guidelines, minor sections like test steps or examples should use bold text rather than Markdown headers.

♻️ Proposed change
-## Validation
-
-```bash
-holmes ask "Show me CPU usage for the last hour"
-```
+**Validation:**
+
+```bash
+holmes ask "Show me CPU usage for the last hour"
+```

As per coding guidelines: "don't create a header for every small section. For minor sections like test steps or examples, use bold text."

🤖 Prompt for AI Agents
Verify each finding against the current code and only fix it if needed.

In `@docs/data-sources/builtin-toolsets/prometheus.md` around lines 38 - 42,
Replace the "## Validation" header with a bold inline label and adjust the
following code block formatting: change the "## Validation" heading to
"**Validation:**" and ensure the Bash example remains inside a fenced code block
(```bash ... ```) as shown in the diff; update the markdown in
docs/data-sources/builtin-toolsets/prometheus.md around the "Validation" section
accordingly so it's a bold label plus the existing code example instead of a
level-2 header.
🤖 Prompt for all review comments with AI agents
Verify each finding against the current code and only fix it if needed.

Duplicate comments:
In `@docs/data-sources/builtin-toolsets/prometheus.md`:
- Around line 69-82: The fenced code block inside the "Prompt for AI agent" tip
is missing a language tag, triggering markdownlint MD046; update the opening
fence (the triple backticks) to include a language identifier (e.g., ```text or
```bash) so the block is treated as a fenced code block; locate the
triple-backtick fence in the tip admonition (the block that begins with "I need
to find my Prometheus server URL...") and change the opening ``` to ```text (or
another appropriate language) and leave the closing ``` unchanged.

---

Nitpick comments:
In `@docs/data-sources/builtin-toolsets/prometheus.md`:
- Around line 38-42: Replace the "## Validation" header with a bold inline label
and adjust the following code block formatting: change the "## Validation"
heading to "**Validation:**" and ensure the Bash example remains inside a fenced
code block (```bash ... ```) as shown in the diff; update the markdown in
docs/data-sources/builtin-toolsets/prometheus.md around the "Validation" section
accordingly so it's a bold label plus the existing code example instead of a
level-2 header.

Signed-off-by: Claude <noreply@anthropic.com>

This branch has not been deployed

No deployments
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants