Repository navigation
Conversation
Signed-off-by: Claude <noreply@anthropic.com>
Contributor
📂 Previous Runs📜 #2 · Run @ __d6a7544__ (#26001252455) — May 17, 20:09 UTC✅ Results of HolmesGPT evalsAutomatically triggered by commit d6a7544 on branch Results of HolmesGPT evals
Benchmark Comparison DetailsMaster baseline: latest master-* experiment (post-merge regression eval)
Benchmark baseline: latest ci-benchmark experiment on master
Time comparison (seconds):
Cost comparison:
Total tokens comparison:
Cached tokens comparison:
Turns comparison:
Tool calls comparison:
Comparison indicators:
|
| Status | Test case | Time | Turns | Tools | Cost | Total tokens | Input | Max input | Output | Max output | Cached | Non-cached | Reasoning | Compactions | Src |
|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|
| ✅ | 09_crashpod | 31.9s | 4 | 10 | $0.2591 | 84,811 | 82,719 | 23,925 | 2,092 | 1,070 | 57,834 | 24,885 | 199 | — | src |
| ✅ | 101_loki_historical_logs_pod_deleted | 75.7s | 7 | 20 | $0.4512 | 175,881 | 170,885 | 31,551 | 4,996 | 916 | 135,250 | 35,635 | 925 | — | src |
| ✅ | 112_find_pvcs_by_uuid | 18.7s | 3 | 4 | $0.2032 | 60,691 | 59,532 | 21,707 | 1,159 | 589 | 37,569 | 21,963 | 259 | — | src |
| ✅ | 12_job_crashing | 42.2s | 6 | 13 | $0.2991 | 132,222 | 129,757 | 24,511 | 2,465 | 578 | 104,256 | 25,501 | 233 | — | src |
| ✅ | 176_network_policy_blocking_traffic_no_skills | 53.3s | 6 | 14 | $0.3370 | 135,511 | 132,416 | 25,955 | 3,095 | 759 | 103,447 | 28,969 | 645 | — | src |
| ✅ | 227_count_configmaps_per_namespace[0] | 20.3s | 4 | 9 | $0.2038 | 76,294 | 75,172 | 20,562 | 1,122 | 591 | 54,292 | 20,880 | 53 | — | src |
| ✅ | 243_pod_names_contain_service | 29.7s | 4 | 8 | $0.2314 | 78,996 | 77,225 | 21,720 | 1,771 | 887 | 54,927 | 22,298 | 181 | — | src |
| ✅ | 24_misconfigured_pvc | 40.2s | 5 | 14 | $0.2905 | 106,707 | 103,885 | 23,617 | 2,822 | 1,072 | 78,988 | 24,897 | 392 | — | src |
| ✅ | 43_current_datetime_from_prompt | 5.0s | 1 | — | $0.1183 | 16,900 | 16,798 | 16,798 | 102 | 102 | 0 | 16,798 | 61 | — | src |
| ✅ | 51_logs_summarize_errors | 22.2s | 4 | 5 | $0.2081 | 77,726 | 76,588 | 21,294 | 1,138 | 423 | 55,289 | 21,299 | 76 | — | src |
| ✅ | 61_exact_match_counting | 11.9s | 3 | 3 | $0.1510 | 52,418 | 52,056 | 17,771 | 362 | 215 | 34,281 | 17,775 | 31 | — | src |
| Total | 31.9s avg | 4.3 avg | 10.0 avg | $2.7527 | 998,157 | 977,033 | 31,551 | 21,124 | 1,072 | 716,133 | 260,900 | 3,055 | — |
Benchmark Comparison Details
Master baseline: latest master-* experiment (post-merge regression eval)
Status: 11 test/model combinations loaded
- master-25989042412 (created: 2026-05-17)
Benchmark baseline: latest ci-benchmark experiment on master
Status: 147 test/model combinations loaded
- ci-benchmark-25268492423 (created: 2026-05-03)
Time comparison (seconds):
| Test case | This branch | master (9h ago) | Δ vs master | benchmark (14d ago) | Δ vs benchmark |
|---|---|---|---|---|---|
| 09_crashpod (opus-4.6) 📄 | 31.9s | 34.3s | ±0% | 32.5s | ±0% |
| 101_loki_historical_logs_pod_deleted (opus-4.6) 📄 | 75.7s | 63.8s | ↑19% | 51.7s | ↑47% |
| 112_find_pvcs_by_uuid (opus-4.6) 📄 | 18.7s | 19.7s | ±0% | 18.1s | ±0% |
| 12_job_crashing (opus-4.6) 📄 | 42.2s | 42.2s | ±0% | 42.4s | ±0% |
| 176_network_policy_blocking_traffic_no_skills (opus-4.6) 📄 | 53.3s | 42.3s | ↑26% | 35.7s | ↑49% |
| 227_count_configmaps_per_namespace[0] (opus-4.6) 📄 | 20.3s | 18.6s | ±0% | 19.7s | ±0% |
| 243_pod_names_contain_service (opus-4.6) 📄 | 29.7s | 38.3s | ↓22% | 27.4s | ±0% |
| 24_misconfigured_pvc (opus-4.6) 📄 | 40.2s | 40.5s | ±0% | 35.8s | ↑12% |
| 43_current_datetime_from_prompt (opus-4.6) 📄 | 5.0s | 3.1s | ↑61% | — | — |
| 51_logs_summarize_errors (opus-4.6) 📄 | 22.2s | 21.4s | ±0% | 23.1s | ±0% |
| 61_exact_match_counting (opus-4.6) 📄 | 11.9s | 9.3s | ↑29% | 10.8s | ↑10% |
| Total (all, n=11) | 31.9s | 30.3s | — | 29.7s | — |
| Comparable (m=11, b=10) | 31.9s | 30.3s | ±0% | 29.7s | ↑16% |
Cost comparison:
| Test case | This branch | master (9h ago) | Δ vs master | benchmark (14d ago) | Δ vs benchmark |
|---|---|---|---|---|---|
| 09_crashpod (opus-4.6) 📄 | $0.2591 | $0.2729 | ±0% | $0.2616 | ±0% |
| 101_loki_historical_logs_pod_deleted (opus-4.6) 📄 | $0.4512 | $0.3876 | ↑16% | $0.3371 | ↑34% |
| 112_find_pvcs_by_uuid (opus-4.6) 📄 | $0.2032 | $0.2086 | ±0% | $0.2014 | ±0% |
| 12_job_crashing (opus-4.6) 📄 | $0.2991 | $0.3169 | ±0% | $0.3076 | ±0% |
| 176_network_policy_blocking_traffic_no_skills (opus-4.6) 📄 | $0.3370 | $0.3282 | ±0% | $0.2914 | ↑16% |
| 227_count_configmaps_per_namespace[0] (opus-4.6) 📄 | $0.2038 | $0.2069 | ±0% | $0.2059 | ±0% |
| 243_pod_names_contain_service (opus-4.6) 📄 | $0.2314 | $0.2881 | ↓20% | $0.2280 | ±0% |
| 24_misconfigured_pvc (opus-4.6) 📄 | $0.2905 | $0.3109 | ±0% | $0.2831 | ±0% |
| 43_current_datetime_from_prompt (opus-4.6) 📄 | $0.1183 | $0.0120 | ↑885% | — | — |
| 51_logs_summarize_errors (opus-4.6) 📄 | $0.2081 | $0.2070 | ±0% | $0.2072 | ±0% |
| 61_exact_match_counting (opus-4.6) 📄 | $0.1510 | $0.1510 | ±0% | $0.1522 | ±0% |
| Total (all, n=11) | $0.2502 | $0.2446 | — | $0.2475 | — |
| Comparable (m=11, b=10) | $0.2502 | $0.2446 | ±0% | $0.2475 | ±0% |
Total tokens comparison:
| Test case | This branch | master (9h ago) | Δ vs master | benchmark (14d ago) | Δ vs benchmark |
|---|---|---|---|---|---|
| 09_crashpod (opus-4.6) 📄 | 84,811 | 104,084 | ↓19% | 103,497 | ↓18% |
| 101_loki_historical_logs_pod_deleted (opus-4.6) 📄 | 175,881 | 167,778 | ±0% | 138,670 | ↑27% |
| 112_find_pvcs_by_uuid (opus-4.6) 📄 | 60,691 | 61,070 | ±0% | 61,169 | ±0% |
| 12_job_crashing (opus-4.6) 📄 | 132,222 | 135,709 | ±0% | 133,893 | ±0% |
| 176_network_policy_blocking_traffic_no_skills (opus-4.6) 📄 | 135,511 | 113,278 | ↑20% | 111,145 | ↑22% |
| 227_count_configmaps_per_namespace[0] (opus-4.6) 📄 | 76,294 | 76,279 | ±0% | 76,945 | ±0% |
| 243_pod_names_contain_service (opus-4.6) 📄 | 78,996 | 124,965 | ↓37% | 79,525 | ±0% |
| 24_misconfigured_pvc (opus-4.6) 📄 | 106,707 | 131,123 | ↓19% | 108,047 | ±0% |
| 43_current_datetime_from_prompt (opus-4.6) 📄 | 16,900 | 16,898 | ±0% | — | — |
| 51_logs_summarize_errors (opus-4.6) 📄 | 77,726 | 77,246 | ±0% | 77,707 | ±0% |
| 61_exact_match_counting (opus-4.6) 📄 | 52,418 | 52,428 | ±0% | 52,942 | ±0% |
| Total (all, n=11) | 90,742 | 96,442 | — | 94,354 | — |
| Comparable (m=11, b=10) | 90,742 | 96,442 | ±0% | 94,354 | ±0% |
Cached tokens comparison:
| Test case | This branch | master (9h ago) | Δ vs master | benchmark (14d ago) | Δ vs benchmark |
|---|---|---|---|---|---|
| 09_crashpod (opus-4.6) 📄 | 57,834 | 77,473 | ↓25% | 77,391 | ↓25% |
| 101_loki_historical_logs_pod_deleted (opus-4.6) 📄 | 135,250 | 134,689 | ±0% | 106,565 | ↑27% |
| 112_find_pvcs_by_uuid (opus-4.6) 📄 | 37,569 | 37,651 | ±0% | 38,002 | ±0% |
| 12_job_crashing (opus-4.6) 📄 | 104,256 | 106,020 | ±0% | 104,761 | ±0% |
| 176_network_policy_blocking_traffic_no_skills (opus-4.6) 📄 | 103,447 | 79,473 | ↑30% | 81,519 | ↑27% |
| 227_count_configmaps_per_namespace[0] (opus-4.6) 📄 | 54,292 | 53,666 | ±0% | 54,503 | ±0% |
| 243_pod_names_contain_service (opus-4.6) 📄 | 54,927 | 98,220 | ↓44% | 55,513 | ±0% |
| 24_misconfigured_pvc (opus-4.6) 📄 | 78,988 | 102,442 | ↓23% | 80,270 | ±0% |
| 43_current_datetime_from_prompt (opus-4.6) 📄 | — | 16,795 | — | — | — |
| 51_logs_summarize_errors (opus-4.6) 📄 | 55,289 | 55,037 | ±0% | 55,443 | ±0% |
| 61_exact_match_counting (opus-4.6) 📄 | 34,281 | 34,285 | ±0% | 34,632 | ±0% |
| Total (all, n=11) | 65,103 | 72,341 | — | 68,860 | — |
| Comparable (m=10, b=10) | 71,613 | 77,896 | ±0% | 68,860 | ±0% |
Turns comparison:
| Test case | This branch | master (9h ago) | Δ vs master | benchmark (14d ago) | Δ vs benchmark |
|---|---|---|---|---|---|
| 09_crashpod (opus-4.6) 📄 | 4 | 5 | ↓20% | — | — |
| 101_loki_historical_logs_pod_deleted (opus-4.6) 📄 | 7 | 7 | ±0% | — | — |
| 112_find_pvcs_by_uuid (opus-4.6) 📄 | 3 | 3 | ±0% | — | — |
| 12_job_crashing (opus-4.6) 📄 | 6 | 6 | ±0% | — | — |
| 176_network_policy_blocking_traffic_no_skills (opus-4.6) 📄 | 6 | 5 | ↑20% | — | — |
| 227_count_configmaps_per_namespace[0] (opus-4.6) 📄 | 4 | 4 | ±0% | — | — |
| 243_pod_names_contain_service (opus-4.6) 📄 | 4 | 6 | ↓33% | — | — |
| 24_misconfigured_pvc (opus-4.6) 📄 | 5 | 6 | ↓17% | — | — |
| 43_current_datetime_from_prompt (opus-4.6) 📄 | 1 | 1 | ±0% | — | — |
| 51_logs_summarize_errors (opus-4.6) 📄 | 4 | 4 | ±0% | — | — |
| 61_exact_match_counting (opus-4.6) 📄 | 3 | 3 | ±0% | — | — |
| Total (all, n=11) | 4.3 | 4.5 | — | — | — |
| Comparable (m=11, b=0) | 4.3 | 4.5 | ±0% | — | — |
Tool calls comparison:
| Test case | This branch | master (9h ago) | Δ vs master | benchmark (14d ago) | Δ vs benchmark |
|---|---|---|---|---|---|
| 09_crashpod (opus-4.6) 📄 | 10 | 12 | ↓17% | 10 | ±0% |
| 101_loki_historical_logs_pod_deleted (opus-4.6) 📄 | 20 | 16 | ↑25% | 14 | ↑43% |
| 112_find_pvcs_by_uuid (opus-4.6) 📄 | 4 | 4 | ±0% | 4 | ±0% |
| 12_job_crashing (opus-4.6) 📄 | 13 | 15 | ↓13% | 14 | ±0% |
| 176_network_policy_blocking_traffic_no_skills (opus-4.6) 📄 | 14 | 14 | ±0% | 13 | ±0% |
| 227_count_configmaps_per_namespace[0] (opus-4.6) 📄 | 9 | 9 | ±0% | 9 | ±0% |
| 243_pod_names_contain_service (opus-4.6) 📄 | 8 | 10 | ↓20% | 8 | ±0% |
| 24_misconfigured_pvc (opus-4.6) 📄 | 14 | 15 | ±0% | 14 | ±0% |
| 43_current_datetime_from_prompt (opus-4.6) 📄 | — | — | — | — | — |
| 51_logs_summarize_errors (opus-4.6) 📄 | 5 | 5 | ±0% | 5 | ±0% |
| 61_exact_match_counting (opus-4.6) 📄 | 3 | 3 | ±0% | 3 | ±0% |
| Total (all, n=11) | 9.1 | 10.3 | — | 9.4 | — |
| Comparable (m=10, b=10) | 10.0 | 10.3 | ±0% | 9.4 | ±0% |
Comparison indicators:
±0%— diff under 10% (within noise threshold)↑N%/↓N%— diff 10-25%↑N%/↓N%— diff over 25% (significant)
📖 Legend
| Icon | Meaning |
|---|---|
| ✅ | The test was successful |
| ➖ | The test was skipped |
| The test failed but is known to be flaky or known to fail | |
| 🚧 | The test had a setup failure (not a code regression) |
| 🔧 | The test failed due to mock data issues (not a code regression) |
| 🚫 | The test was throttled by API rate limits/overload |
| ❌ | The test failed and should be fixed before merging the PR |
🔄 Re-run evals manually
⚠️ Warning:/evalcomments always run using the workflow from master, not from this PR branch. If you modified the GitHub Action (e.g., added secrets or env vars), those changes won't take effect.To test workflow changes, use the GitHub CLI or Actions UI instead:
gh workflow run eval-regression.yaml --repo HolmesGPT/holmesgpt --ref claude/debug-benchmark-failures-gPkry -f markers=regression -f filter=
Option 1: Comment on this PR with /eval:
/eval
tags: regression
Or with more options (one per line):
/eval
model: gpt-4o
tags: regression
id: 09_crashpod
iterations: 5
Run evals on a different branch (e.g., master) for comparison:
/eval
branch: master
tags: regression
| Option | Description |
|---|---|
model |
Model(s) to test (default: same as automatic runs) |
tags |
Pytest tags / markers (no default - runs all tests!) |
id |
Eval ID / pytest -k filter (use /list to see valid eval names) |
iterations |
Number of runs, max 10 |
branch |
Run evals on a different branch (for cross-branch comparison) |
Quick re-run: Use /rerun to re-run the most recent /eval on this PR with the same parameters.
Option 2: Trigger via GitHub Actions UI → "Run workflow"
Option 3: Add PR labels to include extra evals (applies to both automatic runs and /eval comments):
| Label | Effect |
|---|---|
evals-tag-<name> |
Run tests with tag <name> alongside regression |
evals-id-<name> |
Run a specific eval by test ID |
evals-model-<name> |
Override the model (use model list name, e.g. sonnet-4.5) |
Examples: evals-tag-easy, evals-id-09_crashpod, evals-model-sonnet-4.5
🏷️ Valid tags
benchmark, chain-of-causation, compaction, confluence, context_window, conversation_worker, coralogix, counting, database, datadog, datetime, db-connectors, easy, elasticsearch, embeds, fast, frontend, grafana, hard, images, integration, kafka, kubernetes, leaked-information, logs, loki, manual, mcp, medium, metrics, network, newrelic, no-cicd, numerical, one-test, port-forward, prometheus, question-answer, regression, skills, slackbot, storage, token-limit, toolset-limitation, traces, transparency, victorialogs
🤖 Valid models
deepseek-chat, deepseek-r1-reasoner, deepseek-reasoner, deepseek-v3.2-chat, gemini-3-flash-preview, gemini-3-pro-preview, gemini-3.1-pro-preview, gpt-4.1, gpt-5.2-high-reasoning, gpt-5.3-codex, gpt-5.4, haiku-4.5, kimi-2.5, kimi-2.5-openrouter, opus-4.5, opus-4.6, opus-4.7, qwen-next-80B-instruct, qwen-next-80B-thinking, sonnet-4.5, sonnet-4.6
Commands: /eval · /rerun · /list
CLI: gh workflow run eval-regression.yaml --repo HolmesGPT/holmesgpt --ref claude/debug-benchmark-failures-gPkry -f markers=regression -f filter=
Contributor
|
✅ Docker images ready for
Use these tags to pull the images for testing. 📋 Copy commandsgcloud auth configure-docker us-central1-docker.pkg.dev
docker pull us-central1-docker.pkg.dev/robusta-development/temporary-builds/holmes:d1de13f2
docker tag us-central1-docker.pkg.dev/robusta-development/temporary-builds/holmes:d1de13f2 me-west1-docker.pkg.dev/robusta-development/development/holmes-dev:d1de13f2
docker push me-west1-docker.pkg.dev/robusta-development/development/holmes-dev:d1de13f2
docker pull us-central1-docker.pkg.dev/robusta-development/temporary-builds/holmes-operator:d1de13f2
docker tag us-central1-docker.pkg.dev/robusta-development/temporary-builds/holmes-operator:d1de13f2 me-west1-docker.pkg.dev/robusta-development/development/holmes-operator-dev:d1de13f2
docker push me-west1-docker.pkg.dev/robusta-development/development/holmes-operator-dev:d1de13f2Patch Helm values in one line (choose the chart you use): HolmesGPT chart: helm upgrade --install holmesgpt ./helm/holmes \
--set registry=me-west1-docker.pkg.dev/robusta-development/development \
--set image=holmes-dev:d1de13f2 \
--set operator.registry=me-west1-docker.pkg.dev/robusta-development/development \
--set operator.image=holmes-operator-dev:d1de13f2Robusta wrapper chart: helm upgrade --install robusta robusta/robusta \
--reuse-values \
--set holmes.registry=me-west1-docker.pkg.dev/robusta-development/development \
--set holmes.image=holmes-dev:d1de13f2 \
--set holmes.operator.registry=me-west1-docker.pkg.dev/robusta-development/development \
--set holmes.operator.image=holmes-operator-dev:d1de13f2 |
Contributor
|
No actionable comments were generated in the recent review. 🎉 ℹ️ Recent review info⚙️ Run configurationConfiguration used: Organization UI Review profile: CHILL Plan: Pro Run ID: 📒 Files selected for processing (1)
🚧 Files skipped from review as they are similar to previous changes (1)
WalkthroughAdded a blank line after the Python shebang in the CLI benchmark script and revamped PR/master GitHub Actions benchmark steps to log environment info, capture per-run exit codes, tee outputs to logs, write step-summary tails, conditionally set JSON outputs, and fail the job if runs fail. ChangesCLI Performance Benchmarks
Sequence Diagram(s)sequenceDiagram
participant ActionsRunner as GitHub Actions Runner
participant WorkflowStep as benchmark shell step
participant Startup as startup benchmark
participant LLM as LLM/e2e benchmark
participant Logs as startup-output.log / llm-output.log
ActionsRunner->>WorkflowStep: invoke step (tracing + env)
WorkflowStep->>Startup: run startup benchmark (set +e, pipe to tee)
Startup->>Logs: write stdout/stderr
WorkflowStep->>WorkflowStep: capture STARTUP_EXIT from PIPESTATUS
WorkflowStep->>LLM: optionally run LLM benchmark (if OPENROUTER_API_KEY)
LLM->>Logs: write stdout/stderr
WorkflowStep->>WorkflowStep: capture LLM_EXIT from PIPESTATUS
WorkflowStep->>ActionsRunner: append log tails and exit codes to $GITHUB_STEP_SUMMARY
WorkflowStep->>ActionsRunner: write startup_json/llm_json only when respective exit == 0
WorkflowStep->>ActionsRunner: exit 1 if STARTUP_EXIT or LLM_EXIT != 0
Estimated code review effort🎯 3 (Moderate) | ⏱️ ~20 minutes Possibly related PRs
Suggested reviewers
🚥 Pre-merge checks | ✅ 5✅ Passed checks (5 passed)
✏️ Tip: You can configure your own custom pre-merge checks in the settings. Thanks for using CodeRabbit! It's free for OSS, and your support helps us grow. If you like it, consider giving us a shout-out. Comment |
✅ Deploy Preview for holmes-docs ready!
To edit notification comments on pull requests, go to your Netlify project configuration. |
Signed-off-by: Claude <noreply@anthropic.com>
Signed-off-by: Claude <noreply@anthropic.com>
Contributor
🔍 Benchmark Master diagnostic dumpstartup-output.log (tail)llm-output.log (tail) |
Contributor
🔍 Benchmark PR diagnostic dumpstartup-output.log (tail)llm-output.log (tail) |
Contributor
🔍 Benchmark PR diagnostic dumpstartup-output.log (tail)llm-output.log (tail) |
Contributor
🔍 Benchmark Master diagnostic dumpstartup-output.log (tail)llm-output.log (tail) |
This branch has not been deployed
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
2 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Trivial whitespace change to fire the cli-performance workflow so we can capture the failure logs and diagnose why "Benchmark PR" and "Benchmark Master" have been failing since ~May 15.
Will close without merging.
Generated by Claude Code
Summary by CodeRabbit
Style
Tests