Skip to content

Add metadata to HolmesStatus - #1591

Merged
moshemorad merged 6 commits into
masterfrom
ROB-3336-report-is-robusta-ai-enabled-to-holmes-status
Feb 22, 2026
Merged

moshemorad merged 6 commits into
masterfrom
ROB-3336-report-is-robusta-ai-enabled-to-holmes-status

Conversation

@moshemorad

@moshemorad moshemorad commented Feb 19, 2026 •

Copy link
Copy Markdown
Collaborator

Summary by CodeRabbit

  • New Features
    • Holmes status updates now include a metadata field indicating AI feature availability (e.g., Robusta AI enablement) and whether additional system prompts are supported, improving visibility into runtime capabilities and configuration.
    • Metadata is attached to status payloads so downstream tools can make informed decisions based on current feature support.

@netlify

netlify Bot commented Feb 19, 2026 •

Copy link
Copy Markdown

✅ Deploy Preview for holmes-docs ready!

Name Link
🔨 Latest commit 9faef2a
🔍 Latest deploy log https://app.netlify.com/projects/holmes-docs/deploys/699aca7af3e18400082e9521
😎 Deploy Preview https://deploy-preview-1591--holmes-docs.netlify.app
📱 Preview on mobile
Toggle QR Code...

QR Code

Use your smartphone camera to open QR code link.

To edit notification comments on pull requests, go to your Netlify project configuration.

@coderabbitai

coderabbitai Bot commented Feb 19, 2026 •

Copy link
Copy Markdown
Contributor

Walkthrough

Added a new public dataclass HolmesMetadata and included a serialized metadata field in the upsert payload inside update_holmes_status_in_db(), built from config.should_try_robusta_ai. Function signatures were not changed.

Changes

Cohort / File(s) Summary
Metadata Integration
holmes/utils/holmes_status.py
Added HolmesMetadata dataclass (is_robusta_ai_enabled: bool, supports_additional_system_prompt: bool = True); imported asdict, dataclass; created metadata = asdict(HolmesMetadata(...)) from config.should_try_robusta_ai and added it to the upsert payload.

Estimated code review effort

🎯 2 (Simple) | ⏱️ ~10 minutes

🚥 Pre-merge checks | ✅ 2 | ❌ 1

❌ Failed checks (1 warning)

Check name Status Explanation Resolution
Docstring Coverage ⚠️ Warning Docstring coverage is 0.00% which is insufficient. The required threshold is 80.00%. Write docstrings for the functions missing them to satisfy the coverage threshold.
✅ Passed checks (2 passed)
Check name Status Explanation
Description Check ✅ Passed Check skipped - CodeRabbit’s high-level summary is enabled.
Title check ✅ Passed The title 'Add metadata to HolmesStatus' directly and accurately describes the main change: adding a metadata field to HolmesStatus updates.

✏️ Tip: You can configure your own custom pre-merge checks in the settings.


Thanks for using CodeRabbit! It's free for OSS, and your support helps us grow. If you like it, consider giving us a shout-out.

❤️ Share

Comment @coderabbitai help to get the list of available commands and usage tips.

@moshemorad
moshemorad force-pushed the ROB-3336-report-is-robusta-ai-enabled-to-holmes-status branch from 8a56202 to 016369f Compare February 19, 2026 13:47
@github-actions

github-actions Bot commented Feb 19, 2026 •

Copy link
Copy Markdown
Contributor

✅ Docker images ready for baadda2a (built in 5m 32s)

⚠️ Warning: does not support ARM (ARM images are built on release only - not on every PR)

Use these tags to pull the images for testing.

📋 Copy commands

⚠️ Temporary images are deleted after 30 days. Copy to a permanent registry before using them:

gcloud auth configure-docker us-central1-docker.pkg.dev
docker pull us-central1-docker.pkg.dev/robusta-development/temporary-builds/holmes:baadda2a
docker tag us-central1-docker.pkg.dev/robusta-development/temporary-builds/holmes:baadda2a me-west1-docker.pkg.dev/robusta-development/development/holmes-dev:baadda2a
docker push me-west1-docker.pkg.dev/robusta-development/development/holmes-dev:baadda2a
docker pull us-central1-docker.pkg.dev/robusta-development/temporary-builds/holmes-operator:baadda2a
docker tag us-central1-docker.pkg.dev/robusta-development/temporary-builds/holmes-operator:baadda2a me-west1-docker.pkg.dev/robusta-development/development/holmes-operator-dev:baadda2a
docker push me-west1-docker.pkg.dev/robusta-development/development/holmes-operator-dev:baadda2a

Patch Helm values in one line (choose the chart you use):

HolmesGPT chart:

helm upgrade --install holmesgpt ./helm/holmes \
  --set registry=me-west1-docker.pkg.dev/robusta-development/development \
  --set image=holmes-dev:baadda2a \
  --set operator.registry=me-west1-docker.pkg.dev/robusta-development/development \
  --set operator.image=holmes-operator-dev:baadda2a

Robusta wrapper chart:

helm upgrade --install robusta robusta/robusta \
  --reuse-values \
  --set holmes.registry=me-west1-docker.pkg.dev/robusta-development/development \
  --set holmes.image=holmes-dev:baadda2a \
  --set holmes.operator.registry=me-west1-docker.pkg.dev/robusta-development/development \
  --set holmes.operator.image=holmes-operator-dev:baadda2a

@github-actions

github-actions Bot commented Feb 19, 2026 •

Copy link
Copy Markdown
Contributor

📂 Previous Runs

📜 Run @ 6c3cc5f (#22273537865)

✅ Results of HolmesGPT evals

Automatically triggered by commit 6c3cc5f on branch ROB-3336-report-is-robusta-ai-enabled-to-holmes-status

View workflow logs

Results of HolmesGPT evals

  • ask_holmes: 9/9 test cases were successful, 0 regressions
Status Test case Time Turns Tools Cost
✅ 09_crashpod 31.5s 5 12 $0.2436
✅ 101_loki_historical_logs_pod_deleted 42.5s 6 10 $0.2617
✅ 111_pod_names_contain_service 30.0s 5 10 $0.2273
✅ 112_find_pvcs_by_uuid 31.2s 6 8 $0.2444
✅ 12_job_crashing 24.3s 4 7 $0.2031
✅ 176_network_policy_blocking_traffic_no_runbooks 43.6s 7 16 $0.3105
✅ 24_misconfigured_pvc 36.7s 6 16 $0.2666
✅ 43_current_datetime_from_prompt 5.0s 1 — $0.1119
✅ 61_exact_match_counting 15.6s 4 4 $0.1672
Total 28.9s avg 4.9 avg 10.4 avg $2.0364
📜 Run @ bdc7950 (#22218462748)

✅ Results of HolmesGPT evals

Automatically triggered by commit bdc7950 on branch ROB-3336-report-is-robusta-ai-enabled-to-holmes-status

View workflow logs

Results of HolmesGPT evals

  • ask_holmes: 9/9 test cases were successful, 0 regressions
Status Test case Time Turns Tools Cost
✅ 09_crashpod 31.5s 5 10 $0.2313
✅ 101_loki_historical_logs_pod_deleted 49.5s 7 10 $0.2693
✅ 111_pod_names_contain_service 33.0s 5 11 $0.2355
✅ 112_find_pvcs_by_uuid 37.6s 7 9 $0.2668
✅ 12_job_crashing 32.9s 5 11 $0.2433
✅ 176_network_policy_blocking_traffic_no_runbooks 47.4s 6 16 $0.2882
✅ 24_misconfigured_pvc 32.4s 5 12 $0.2277
✅ 43_current_datetime_from_prompt 5.1s 1 — $0.0120
✅ 61_exact_match_counting 13.6s 3 2 $0.1502
Total 31.4s avg 4.9 avg 10.1 avg $1.9244
📜 Run @ 570fb5b (#22217985877)

✅ Results of HolmesGPT evals

Automatically triggered by commit 570fb5b on branch ROB-3336-report-is-robusta-ai-enabled-to-holmes-status

View workflow logs

Results of HolmesGPT evals

  • ask_holmes: 9/9 test cases were successful, 0 regressions
Status Test case Time Turns Tools Cost
✅ 09_crashpod 32.6s 5 11 $0.2329
✅ 101_loki_historical_logs_pod_deleted 45.1s 5 10 $0.2645
✅ 111_pod_names_contain_service 36.3s 5 12 $0.2397
✅ 112_find_pvcs_by_uuid 31.5s 5 7 $0.2299
✅ 12_job_crashing 32.8s 5 11 $0.2408
✅ 176_network_policy_blocking_traffic_no_runbooks 44.6s 6 17 $0.2856
✅ 24_misconfigured_pvc 37.3s 6 14 $0.2512
✅ 43_current_datetime_from_prompt 5.2s 1 — $0.1110
✅ 61_exact_match_counting 19.3s 4 4 $0.1698
Total 31.6s avg 4.7 avg 10.8 avg $2.0254
📜 Run @ 98761d5 (#22189752327)

✅ Results of HolmesGPT evals

Automatically triggered by commit 98761d5 on branch ROB-3336-report-is-robusta-ai-enabled-to-holmes-status

View workflow logs

Results of HolmesGPT evals

  • ask_holmes: 9/9 test cases were successful, 0 regressions
Status Test case Time Turns Tools Cost
✅ 09_crashpod 37.1s 6 12 $0.2560
✅ 101_loki_historical_logs_pod_deleted 50.5s 7 10 $0.3009
✅ 111_pod_names_contain_service 34.8s 5 12 $0.2359
✅ 112_find_pvcs_by_uuid 23.8s 4 5 $0.1988
✅ 12_job_crashing 33.3s 5 12 $0.2477
✅ 176_network_policy_blocking_traffic_no_runbooks 46.1s 6 17 $0.2918
✅ 24_misconfigured_pvc 33.0s 5 13 $0.2334
✅ 43_current_datetime_from_prompt 5.3s 1 — $0.0118
✅ 61_exact_match_counting 12.5s 3 2 $0.1486
Total 30.7s avg 4.7 avg 10.4 avg $1.9249
📜 Run @ 433fb92 (#22189591032)

✅ Results of HolmesGPT evals

Automatically triggered by commit 433fb92 on branch ROB-3336-report-is-robusta-ai-enabled-to-holmes-status

View workflow logs

Results of HolmesGPT evals

  • ask_holmes: 9/9 test cases were successful, 0 regressions
Status Test case Time Turns Tools Cost
✅ 09_crashpod 39.7s 6 13 $0.2756
✅ 101_loki_historical_logs_pod_deleted 45.1s 5 10 $0.2569
✅ 111_pod_names_contain_service 35.0s 5 12 $0.2317
✅ 112_find_pvcs_by_uuid 26.7s 4 5 $0.2033
✅ 12_job_crashing 32.2s 5 11 $0.2391
✅ 176_network_policy_blocking_traffic_no_runbooks 56.1s 8 18 $0.3229
✅ 24_misconfigured_pvc 39.2s 6 15 $0.2546
✅ 43_current_datetime_from_prompt 6.4s 1 — $0.1127
✅ 61_exact_match_counting 16.4s 3 2 $0.1528
Total 33.0s avg 4.8 avg 10.8 avg $2.0496

✅ Results of HolmesGPT evals

Automatically triggered by commit 9faef2a on branch ROB-3336-report-is-robusta-ai-enabled-to-holmes-status

View workflow logs

Results of HolmesGPT evals

  • ask_holmes: 9/9 test cases were successful, 0 regressions
Status Test case Time Turns Tools Cost
✅ 09_crashpod 33.0s 5 12 $0.2523
✅ 101_loki_historical_logs_pod_deleted 48.1s 7 10 $0.2883
✅ 111_pod_names_contain_service 32.6s 5 12 $0.2351
✅ 112_find_pvcs_by_uuid 27.0s 5 5 $0.2092
✅ 12_job_crashing 41.5s 7 14 $0.2901
✅ 176_network_policy_blocking_traffic_no_runbooks 44.7s 6 18 $0.2962
✅ 24_misconfigured_pvc 32.8s 5 13 $0.2383
✅ 43_current_datetime_from_prompt 4.5s 1 — $0.1110
✅ 61_exact_match_counting 16.3s 4 4 $0.1677
Total 31.2s avg 5.0 avg 11.0 avg $2.0883
📖 Legend
Icon Meaning
✅ The test was successful
➖ The test was skipped
⚠️ The test failed but is known to be flaky or known to fail
🚧 The test had a setup failure (not a code regression)
🔧 The test failed due to mock data issues (not a code regression)
🚫 The test was throttled by API rate limits/overload
❌ The test failed and should be fixed before merging the PR
🔄 Re-run evals manually

⚠️ Warning: /eval comments always run using the workflow from master, not from this PR branch. If you modified the GitHub Action (e.g., added secrets or env vars), those changes won't take effect.

To test workflow changes, use the GitHub CLI or Actions UI instead:

gh workflow run eval-regression.yaml --repo HolmesGPT/holmesgpt --ref ROB-3336-report-is-robusta-ai-enabled-to-holmes-status -f markers=regression -f filter=

Option 1: Comment on this PR with /eval:

/eval
tags: regression

Or with more options (one per line):

/eval
model: gpt-4o
tags: regression
filter: 09_crashpod
iterations: 5

Run evals on a different branch (e.g., master) for comparison:

/eval
branch: master
tags: regression
Option Description
model Model(s) to test (default: same as automatic runs)
tags Pytest tags / markers (no default - runs all tests!)
filter Pytest -k filter (use /list to see valid eval names)
iterations Number of runs, max 10
branch Run evals on a different branch (for cross-branch comparison)

Quick re-run: Use /rerun to re-run the most recent /eval on this PR with the same parameters.

Option 2: Trigger via GitHub Actions UI → "Run workflow"

Option 3: Add PR labels to include extra evals in automatic regression runs:

Label Effect
evals-tag-<name> Run tests with tag <name> alongside regression
evals-id-<name> Run a specific eval by test ID

Examples: evals-tag-easy, evals-id-09_crashpod

🏷️ Valid tags

benchmark, chain-of-causation, compaction, confluence, context_window, coralogix, counting, database, datadog, datetime, easy, elasticsearch, embeds, fast, frontend, grafana-dashboard, hard, integration, kafka, kubernetes, leaked-information, logs, loki, medium, metrics, network, newrelic, no-cicd, numerical, one-test, port-forward, prometheus, question-answer, regression, runbooks, slackbot, storage, toolset-limitation, traces, transparency


Commands: /eval · /rerun · /list

CLI: gh workflow run eval-regression.yaml --repo HolmesGPT/holmesgpt --ref ROB-3336-report-is-robusta-ai-enabled-to-holmes-status -f markers=regression -f filter=

@github-actions

github-actions Bot commented Feb 19, 2026 •

Copy link
Copy Markdown
Contributor

🔬 CLI Performance Benchmark

🟡 Startup Time (no LLM)

Measures holmes version execution time (imports + initialization)

Metric PR Master Change
Cold Start 10.41s 10.45s -0.4%
Warm Mean 4.56s 4.58s -0.6%
Warm Min 4.55s 4.56s
Warm Max 4.57s 4.61s

🟡 Full CLI with LLM

Measures holmes ask execution time (OpenRouter + Haiku 4.5)

Metric PR Master Change
Cold Start 28.15s 34.57s -18.6%
Warm Mean 7.59s 7.82s -2.9%
Warm Min 7.19s 7.13s
Warm Max 8.12s 8.86s

PR: baadda2a | Master: 5f48a2d8 | Iterations: 5

@coderabbitai coderabbitai Bot left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

🧹 Nitpick comments (1)
holmes/utils/holmes_status.py (1)

9-11: Consider adding a class-level docstring to HolmesMetadata.

The field name is self-explanatory, but a brief docstring clarifying the purpose of this metadata object (e.g., what it represents in the context of the Holmes status DB entry) would align with the guideline of explaining why rather than what.

✏️ Suggested addition
 `@dataclass`
 class HolmesMetadata:
+    """Metadata attached to each Holmes status DB entry, describing cluster-level feature flags."""
     is_robusta_ai_enabled: bool

As per coding guidelines: "Write clear, concise comments that explain 'why' rather than 'what'."

🤖 Prompt for AI Agents
Verify each finding against the current code and only fix it if needed.

In `@holmes/utils/holmes_status.py` around lines 9 - 11, Add a concise class-level
docstring to the dataclass HolmesMetadata that explains why this metadata exists
and how it is used in the Holmes status DB entry (e.g., it represents status
flags for integrations/features such as whether Robusta AI is enabled), and
mention the meaning of the is_robusta_ai_enabled field for clarity; place the
docstring immediately under the class declaration so it’s discoverable by tools
and developers.
🤖 Prompt for all review comments with AI agents
Verify each finding against the current code and only fix it if needed.

Nitpick comments:
In `@holmes/utils/holmes_status.py`:
- Around line 9-11: Add a concise class-level docstring to the dataclass
HolmesMetadata that explains why this metadata exists and how it is used in the
Holmes status DB entry (e.g., it represents status flags for
integrations/features such as whether Robusta AI is enabled), and mention the
meaning of the is_robusta_ai_enabled field for clarity; place the docstring
immediately under the class declaration so it’s discoverable by tools and
developers.

Signed-off-by: Mohse Morad <moshemorad12340@gmail.com>
Signed-off-by: Mohse Morad <moshemorad12340@gmail.com>
@moshemorad
moshemorad force-pushed the ROB-3336-report-is-robusta-ai-enabled-to-holmes-status branch from 433fb92 to 98761d5 Compare February 19, 2026 16:10

@coderabbitai coderabbitai Bot left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

🧹 Nitpick comments (1)
holmes/utils/holmes_status.py (1)

29-31: Pass native Python objects instead of JSON strings for the model and metadata fields

When using Supabase's Python client with jsonb columns, pass dict/list objects directly rather than json.dumps() strings. Using json.dumps() stores the values as JSONB string scalars instead of JSONB objects, which breaks JSON operators and queries. Both config.get_models_list() (returns List[str]) and asdict(metadata) (returns dict) should be passed directly to the upsert payload.

♻️ Suggested change
-            "model": json.dumps(config.get_models_list()),
+            "model": config.get_models_list(),
             "version": get_version(),
-            "metadata": json.dumps(asdict(metadata)),
+            "metadata": asdict(metadata),
🤖 Prompt for AI Agents
Verify each finding against the current code and only fix it if needed.

In `@holmes/utils/holmes_status.py` around lines 29 - 31, The upsert payload is
JSON-encoding values for the "model" and "metadata" fields which causes Supabase
jsonb columns to store string scalars; instead, stop calling json.dumps and pass
the native Python objects returned by config.get_models_list() (List[str]) and
asdict(metadata) (dict) directly in holmes_status.py where the payload is built
(the dict with keys "model", "version", "metadata"); keep get_version()
unchanged but replace "model": json.dumps(config.get_models_list()) and
"metadata": json.dumps(asdict(metadata)) with the raw values so Supabase stores
proper jsonb objects/arrays.
🤖 Prompt for all review comments with AI agents
Verify each finding against the current code and only fix it if needed.

Nitpick comments:
In `@holmes/utils/holmes_status.py`:
- Around line 29-31: The upsert payload is JSON-encoding values for the "model"
and "metadata" fields which causes Supabase jsonb columns to store string
scalars; instead, stop calling json.dumps and pass the native Python objects
returned by config.get_models_list() (List[str]) and asdict(metadata) (dict)
directly in holmes_status.py where the payload is built (the dict with keys
"model", "version", "metadata"); keep get_version() unchanged but replace
"model": json.dumps(config.get_models_list()) and "metadata":
json.dumps(asdict(metadata)) with the raw values so Supabase stores proper jsonb
objects/arrays.

@moshemorad
moshemorad merged commit 22f5483 into master Feb 22, 2026
20 of 21 checks passed
@moshemorad
moshemorad deleted the ROB-3336-report-is-robusta-ai-enabled-to-holmes-status branch February 22, 2026 09:27
moshemorad added a commit that referenced this pull request Feb 22, 2026
<!-- This is an auto-generated comment: release notes by coderabbit.ai
-->
## Summary by CodeRabbit

* **New Features**
* Holmes status updates now include a metadata field indicating AI
feature availability (e.g., Robusta AI enablement) and whether
additional system prompts are supported, improving visibility into
runtime capabilities and configuration.
* Metadata is attached to status payloads so downstream tools can make
informed decisions based on current feature support.
<!-- end of auto-generated comment: release notes by coderabbit.ai -->

---------

Signed-off-by: Mohse Morad <moshemorad12340@gmail.com>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants