Repository navigation
Conversation
…ilable fetch_pod_logs previously trimmed logs from the top until they fit the single-tool token budget, irrecoverably dropping everything above the limit. This was designed for older models; with spill-to-disk available the model can read the full output back with bash. - Expose tool_results_dir on ToolInvokeContext so tools can tell whether oversized results will be spilled to disk (storage enabled + bash toolset available) and skip their own lossy truncation. - PodLoggingTool now returns full logs when spilling is available and only falls back to in-place truncation when it is not. - The spill preview now shows the beginning AND the end of the output so chronological data like logs keeps its most recent lines visible in the conversation. https://claude.ai/code/session_01BwJeGAGBLoby5rShhQADwq Signed-off-by: Claude <noreply@anthropic.com>
Contributor
📂 Previous Runs
|
| Status | Test case | Time | Turns | Tools | Cost | Total tokens | Input | Max input | Output | Max output | Cached | Non-cached | Reasoning | Compactions | Denied commands | Src |
|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|
| ✅ | 09_crashpod | 30.0s | 3 | 7 | $0.1918 | 50,759 | 49,122 | 18,867 | 1,637 | 665 | 29,694 | 19,428 | 261 | — | — | src |
| ✅ | 101_loki_historical_logs_pod_deleted | 67.5s | 6 | 11 | $0.2715 | 110,631 | 107,771 | 20,549 | 2,860 | 756 | 86,496 | 21,275 | 352 | — | — | src |
| ✅ | 112_find_pvcs_by_uuid | 23.1s | 2 | 2 | $0.1564 | 33,193 | 32,266 | 17,944 | 927 | 482 | 14,319 | 17,947 | 270 | — | — | src |
| ✅ | 12_job_crashing | 32.2s | 4 | 9 | $0.2296 | 73,325 | 71,544 | 20,704 | 1,781 | 571 | 48,643 | 22,901 | 125 | — | — | src |
| ✅ | 176_network_policy_blocking_traffic_no_skills | 31.1s | 4 | 12 | $0.2198 | 72,082 | 70,421 | 21,230 | 1,661 | 456 | 48,836 | 21,585 | 135 | — | — | src |
| ✅ | 227_count_configmaps_per_namespace[0] | 22.5s | 3 | 6 | $0.1484 | 46,359 | 45,644 | 16,654 | 715 | 432 | 28,986 | 16,658 | 36 | — | — | src |
| ✅ | 243_pod_names_contain_service | 31.0s | 3 | 7 | $0.1855 | 49,329 | 47,749 | 18,037 | 1,580 | 649 | 29,236 | 18,513 | 248 | — | — | src |
| ✅ | 24_misconfigured_pvc | 31.6s | 4 | 11 | $0.2151 | 69,432 | 67,618 | 19,326 | 1,814 | 589 | 47,065 | 20,553 | 128 | — | — | src |
| ✅ | 254_elasticsearch_dr_test_log_check | 64.0s | 9 | 12 | $0.2793 | 121,375 | 117,621 | 17,362 | 3,754 | 918 | 99,895 | 17,726 | 219 | — | — | src |
| ✅ | 259_wrong_cluster_logs_confusion | 64.6s | 6 | 11 | $0.2859 | 90,124 | 85,906 | 18,370 | 4,218 | 1,358 | 66,225 | 19,681 | 595 | — | — | src |
| ✅ | 260_global_es_remote_cluster_logs | 93.5s | 8 | 15 | $0.3354 | 123,221 | 118,172 | 19,974 | 5,049 | 1,609 | 97,133 | 21,039 | 785 | — | — | src |
| ✅ | 43_current_datetime_from_prompt | 6.3s | 1 | — | $0.1017 | 14,428 | 14,306 | 14,306 | 122 | 122 | 0 | 14,306 | 78 | — | — | src |
| ✅ | 51_logs_summarize_errors | 19.9s | 3 | 2 | $0.1591 | 47,010 | 46,091 | 17,155 | 919 | 534 | 28,932 | 17,159 | 40 | — | — | src |
| ✅ | 61_exact_match_counting | 13.1s | 2 | 1 | $0.1148 | 29,162 | 28,935 | 14,649 | 227 | 158 | 14,283 | 14,652 | 39 | — | — | src |
| Total | 37.9s avg | 4.1 avg | 8.2 avg | $2.8944 | 930,430 | 903,166 | 21,230 | 27,264 | 1,609 | 639,743 | 263,423 | 3,311 | — | — |
Benchmark Comparison Details
Master baseline: latest master-* experiment (post-merge regression eval)
Status: 11 test/model combinations loaded
- master-27089078550 (created: 2026-06-07)
Benchmark baseline: latest ci-benchmark experiment on master
Status: 34 test/model combinations loaded
- ci-benchmark-27257681111 (created: 2026-06-10)
Time comparison (seconds):
| Test case | This branch | master (2d ago) | Δ vs master | benchmark (1h ago) | Δ vs benchmark |
|---|---|---|---|---|---|
| 09_crashpod (opus-4.6) 📄 | 30.0s | 26.2s | ↑15% | — | — |
| 101_loki_historical_logs_pod_deleted (opus-4.6) 📄 | 67.5s | 47.6s | ↑42% | — | — |
| 112_find_pvcs_by_uuid (opus-4.6) 📄 | 23.1s | 10.6s | ↑119% | — | — |
| 12_job_crashing (opus-4.6) 📄 | 32.2s | 27.6s | ↑17% | — | — |
| 176_network_policy_blocking_traffic_no_skills (opus-4.6) 📄 | 31.1s | 32.6s | ±0% | — | — |
| 227_count_configmaps_per_namespace[0] (opus-4.6) 📄 | 22.5s | 13.2s | ↑70% | — | — |
| 243_pod_names_contain_service (opus-4.6) 📄 | 31.0s | 28.8s | ±0% | — | — |
| 24_misconfigured_pvc (opus-4.6) 📄 | 31.6s | 31.4s | ±0% | — | — |
| 254_elasticsearch_dr_test_log_check (opus-4.6) 📄 | 64.0s | — | — | — | — |
| 259_wrong_cluster_logs_confusion (opus-4.6) 📄 | 64.6s | — | — | — | — |
| 260_global_es_remote_cluster_logs (opus-4.6) 📄 | 93.5s | — | — | — | — |
| 43_current_datetime_from_prompt (opus-4.6) 📄 | 6.3s | 3.1s | ↑103% | — | — |
| 51_logs_summarize_errors (opus-4.6) 📄 | 19.9s | 18.8s | ±0% | — | — |
| 61_exact_match_counting (opus-4.6) 📄 | 13.1s | 7.1s | ↑85% | — | — |
| Total (all, n=14) | 37.9s | 22.5s | — | — | — |
| Comparable (m=11, b=0) | 28.0s | 22.5s | ↑25% | — | — |
Cost comparison:
| Test case | This branch | master (2d ago) | Δ vs master | benchmark (1h ago) | Δ vs benchmark |
|---|---|---|---|---|---|
| 09_crashpod (opus-4.6) 📄 | $0.1918 | $0.2032 | ±0% | — | — |
| 101_loki_historical_logs_pod_deleted (opus-4.6) 📄 | $0.2715 | $0.2642 | ±0% | — | — |
| 112_find_pvcs_by_uuid (opus-4.6) 📄 | $0.1564 | $0.1368 | ↑14% | — | — |
| 12_job_crashing (opus-4.6) 📄 | $0.2296 | $0.2173 | ±0% | — | — |
| 176_network_policy_blocking_traffic_no_skills (opus-4.6) 📄 | $0.2198 | $0.2524 | ↓13% | — | — |
| 227_count_configmaps_per_namespace[0] (opus-4.6) 📄 | $0.1484 | $0.1464 | ±0% | — | — |
| 243_pod_names_contain_service (opus-4.6) 📄 | $0.1855 | $0.1872 | ±0% | — | — |
| 24_misconfigured_pvc (opus-4.6) 📄 | $0.2151 | $0.2240 | ±0% | — | — |
| 254_elasticsearch_dr_test_log_check (opus-4.6) 📄 | $0.2793 | — | — | — | — |
| 259_wrong_cluster_logs_confusion (opus-4.6) 📄 | $0.2859 | — | — | — | — |
| 260_global_es_remote_cluster_logs (opus-4.6) 📄 | $0.3354 | — | — | — | — |
| 43_current_datetime_from_prompt (opus-4.6) 📄 | $0.1017 | $0.0110 | ↑826% | — | — |
| 51_logs_summarize_errors (opus-4.6) 📄 | $0.1591 | $0.1488 | ±0% | — | — |
| 61_exact_match_counting (opus-4.6) 📄 | $0.1148 | $0.1112 | ±0% | — | — |
| Total (all, n=14) | $0.2067 | $0.1730 | — | — | — |
| Comparable (m=11, b=0) | $0.1813 | $0.1730 | ±0% | — | — |
Total tokens comparison:
| Test case | This branch | master (2d ago) | Δ vs master | benchmark (1h ago) | Δ vs benchmark |
|---|---|---|---|---|---|
| 09_crashpod (opus-4.6) 📄 | 50,759 | 67,778 | ↓25% | — | — |
| 101_loki_historical_logs_pod_deleted (opus-4.6) 📄 | 110,631 | 88,307 | ↑25% | — | — |
| 112_find_pvcs_by_uuid (opus-4.6) 📄 | 33,193 | 30,686 | ±0% | — | — |
| 12_job_crashing (opus-4.6) 📄 | 73,325 | 69,487 | ±0% | — | — |
| 176_network_policy_blocking_traffic_no_skills (opus-4.6) 📄 | 72,082 | 108,726 | ↓34% | — | — |
| 227_count_configmaps_per_namespace[0] (opus-4.6) 📄 | 46,359 | 44,988 | ±0% | — | — |
| 243_pod_names_contain_service (opus-4.6) 📄 | 49,329 | 48,612 | ±0% | — | — |
| 24_misconfigured_pvc (opus-4.6) 📄 | 69,432 | 68,923 | ±0% | — | — |
| 254_elasticsearch_dr_test_log_check (opus-4.6) 📄 | 121,375 | — | — | — | — |
| 259_wrong_cluster_logs_confusion (opus-4.6) 📄 | 90,124 | — | — | — | — |
| 260_global_es_remote_cluster_logs (opus-4.6) 📄 | 123,221 | — | — | — | — |
| 43_current_datetime_from_prompt (opus-4.6) 📄 | 14,428 | 13,970 | ±0% | — | — |
| 51_logs_summarize_errors (opus-4.6) 📄 | 47,010 | 45,163 | ±0% | — | — |
| 61_exact_match_counting (opus-4.6) 📄 | 29,162 | 28,231 | ±0% | — | — |
| Total (all, n=14) | 66,459 | 55,897 | — | — | — |
| Comparable (m=11, b=0) | 54,155 | 55,897 | ±0% | — | — |
Cached tokens comparison:
| Test case | This branch | master (2d ago) | Δ vs master | benchmark (1h ago) | Δ vs benchmark |
|---|---|---|---|---|---|
| 09_crashpod (opus-4.6) 📄 | 29,694 | 46,700 | ↓36% | — | — |
| 101_loki_historical_logs_pod_deleted (opus-4.6) 📄 | 86,496 | 64,209 | ↑35% | — | — |
| 112_find_pvcs_by_uuid (opus-4.6) 📄 | 14,319 | 13,861 | ±0% | — | — |
| 12_job_crashing (opus-4.6) 📄 | 48,643 | 46,249 | ±0% | — | — |
| 176_network_policy_blocking_traffic_no_skills (opus-4.6) 📄 | 48,836 | 84,274 | ↓42% | — | — |
| 227_count_configmaps_per_namespace[0] (opus-4.6) 📄 | 28,986 | 28,072 | ±0% | — | — |
| 243_pod_names_contain_service (opus-4.6) 📄 | 29,236 | 28,690 | ±0% | — | — |
| 24_misconfigured_pvc (opus-4.6) 📄 | 47,065 | 46,110 | ±0% | — | — |
| 254_elasticsearch_dr_test_log_check (opus-4.6) 📄 | 99,895 | — | — | — | — |
| 259_wrong_cluster_logs_confusion (opus-4.6) 📄 | 66,225 | — | — | — | — |
| 260_global_es_remote_cluster_logs (opus-4.6) 📄 | 97,133 | — | — | — | — |
| 43_current_datetime_from_prompt (opus-4.6) 📄 | — | 13,845 | — | — | — |
| 51_logs_summarize_errors (opus-4.6) 📄 | 28,932 | 28,012 | ±0% | — | — |
| 61_exact_match_counting (opus-4.6) 📄 | 14,283 | 13,825 | ±0% | — | — |
| Total (all, n=14) | 45,696 | 37,622 | — | — | — |
| Comparable (m=10, b=0) | 37,649 | 40,000 | ±0% | — | — |
Turns comparison:
| Test case | This branch | master (2d ago) | Δ vs master | benchmark (1h ago) | Δ vs benchmark |
|---|---|---|---|---|---|
| 09_crashpod (opus-4.6) 📄 | 3 | 4 | ↓25% | — | — |
| 101_loki_historical_logs_pod_deleted (opus-4.6) 📄 | 6 | 5 | ↑20% | — | — |
| 112_find_pvcs_by_uuid (opus-4.6) 📄 | 2 | 2 | ±0% | — | — |
| 12_job_crashing (opus-4.6) 📄 | 4 | 4 | ±0% | — | — |
| 176_network_policy_blocking_traffic_no_skills (opus-4.6) 📄 | 4 | 6 | ↓33% | — | — |
| 227_count_configmaps_per_namespace[0] (opus-4.6) 📄 | 3 | 3 | ±0% | — | — |
| 243_pod_names_contain_service (opus-4.6) 📄 | 3 | 3 | ±0% | — | — |
| 24_misconfigured_pvc (opus-4.6) 📄 | 4 | 4 | ±0% | — | — |
| 254_elasticsearch_dr_test_log_check (opus-4.6) 📄 | 9 | — | — | — | — |
| 259_wrong_cluster_logs_confusion (opus-4.6) 📄 | 6 | — | — | — | — |
| 260_global_es_remote_cluster_logs (opus-4.6) 📄 | 8 | — | — | — | — |
| 43_current_datetime_from_prompt (opus-4.6) 📄 | 1 | 1 | ±0% | — | — |
| 51_logs_summarize_errors (opus-4.6) 📄 | 3 | 3 | ±0% | — | — |
| 61_exact_match_counting (opus-4.6) 📄 | 2 | 2 | ±0% | — | — |
| Total (all, n=14) | 4.1 | 3.4 | — | — | — |
| Comparable (m=11, b=0) | 3.2 | 3.4 | ±0% | — | — |
Tool calls comparison:
| Test case | This branch | master (2d ago) | Δ vs master | benchmark (1h ago) | Δ vs benchmark |
|---|---|---|---|---|---|
| 09_crashpod (opus-4.6) 📄 | 7 | 8 | ↓12% | — | — |
| 101_loki_historical_logs_pod_deleted (opus-4.6) 📄 | 11 | 11 | ±0% | — | — |
| 112_find_pvcs_by_uuid (opus-4.6) 📄 | 2 | 2 | ±0% | — | — |
| 12_job_crashing (opus-4.6) 📄 | 9 | 9 | ±0% | — | — |
| 176_network_policy_blocking_traffic_no_skills (opus-4.6) 📄 | 12 | 12 | ±0% | — | — |
| 227_count_configmaps_per_namespace[0] (opus-4.6) 📄 | 6 | 6 | ±0% | — | — |
| 243_pod_names_contain_service (opus-4.6) 📄 | 7 | 7 | ±0% | — | — |
| 24_misconfigured_pvc (opus-4.6) 📄 | 11 | 13 | ↓15% | — | — |
| 254_elasticsearch_dr_test_log_check (opus-4.6) 📄 | 12 | — | — | — | — |
| 259_wrong_cluster_logs_confusion (opus-4.6) 📄 | 11 | — | — | — | — |
| 260_global_es_remote_cluster_logs (opus-4.6) 📄 | 15 | — | — | — | — |
| 43_current_datetime_from_prompt (opus-4.6) 📄 | — | — | — | — | — |
| 51_logs_summarize_errors (opus-4.6) 📄 | 2 | 2 | ±0% | — | — |
| 61_exact_match_counting (opus-4.6) 📄 | 1 | 1 | ±0% | — | — |
| Total (all, n=14) | 7.6 | 7.1 | — | — | — |
| Comparable (m=10, b=0) | 6.8 | 7.1 | ±0% | — | — |
Comparison indicators:
±0%— diff under 10% (within noise threshold)↑N%/↓N%— diff 10-25%↑N%/↓N%— diff over 25% (significant)
📖 Legend
| Icon | Meaning |
|---|---|
| ✅ | The test was successful |
| ➖ | The test was skipped |
| The test failed but is known to be flaky or known to fail | |
| 🚧 | The test had a setup failure (not a code regression) |
| 🔧 | The test failed due to mock data issues (not a code regression) |
| 🚫 | The test was throttled by API rate limits/overload |
| ❌ | The test failed and should be fixed before merging the PR |
🔄 Re-run evals manually
⚠️ Warning:/evalcomments always run using the workflow from master, not from this PR branch. If you modified the GitHub Action (e.g., added secrets or env vars), those changes won't take effect.To test workflow changes, use the GitHub CLI or Actions UI instead:
gh workflow run eval-regression.yaml --repo HolmesGPT/holmesgpt --ref claude/tool-limitations-logs-spill -f markers=regression -f filter=
Option 1: Comment on this PR with /eval:
/eval
tags: regression
Or with more options (one per line):
/eval
model: gpt-4o
tags: regression
id: 09_crashpod
iterations: 5
Run evals on a different branch (e.g., master) for comparison:
/eval
branch: master
tags: regression
| Option | Description |
|---|---|
model |
Model(s) to test (default: same as automatic runs) |
tags |
Pytest tags / markers (no default - runs all tests!) |
id |
Eval ID / pytest -k filter (use /list to see valid eval names) |
iterations |
Number of runs, max 10 |
branch |
Run evals on a different branch (for cross-branch comparison) |
Quick re-run: Use /rerun to re-run the most recent /eval on this PR with the same parameters.
Option 2: Trigger via GitHub Actions UI → "Run workflow"
Option 3: Add PR labels to include extra evals (applies to both automatic runs and /eval comments):
| Label | Effect |
|---|---|
evals-tag-<name> |
Run tests with tag <name> alongside regression |
evals-id-<name> |
Run a specific eval by test ID |
evals-model-<name> |
Override the model (use model list name, e.g. sonnet-4.5) |
Examples: evals-tag-easy, evals-id-09_crashpod, evals-model-sonnet-4.5
🏷️ Valid tags
benchmark, chain-of-causation, compaction, confluence, context_window, conversation_worker, coralogix, counting, database, datadog, datetime, db-connectors, easy, elasticsearch, embeds, fast, frontend, grafana, hard, images, integration, kafka, kubernetes, leaked-information, logs, loki, manual, mcp, medium, metrics, multi-cluster, network, newrelic, no-cicd, numerical, one-test, port-forward, prometheus, question-answer, regression, skills, slackbot, storage, token-limit, toolset-limitation, traces, transparency, victorialogs
🤖 Valid models
deepseek-chat, deepseek-r1-reasoner, deepseek-reasoner, deepseek-v3.2-chat, gemini-3-flash-preview, gemini-3-pro-preview, gemini-3.1-pro-preview, gpt-4.1, gpt-5.2-high-reasoning, gpt-5.3-codex, gpt-5.4, gpt-5.5, haiku-4.5, kimi-2.5, kimi-2.5-openrouter, opus-4.5, opus-4.6, opus-4.7, opus-4.8, qwen-next-80B-instruct, qwen-next-80B-thinking, sonnet-4.5, sonnet-4.6
Commands: /eval · /rerun · /list
CLI: gh workflow run eval-regression.yaml --repo HolmesGPT/holmesgpt --ref claude/tool-limitations-logs-spill -f markers=regression -f filter=
Contributor
|
No actionable comments were generated in the recent review. 🎉 ℹ️ Recent review info⚙️ Run configurationConfiguration used: Organization UI Review profile: CHILL Plan: Pro Run ID: 📒 Files selected for processing (2)
WalkthroughAdds disk-spill support and previewing for oversized tool results: introduces ChangesTool spill-to-disk context and selective truncation
Estimated code review effort🎯 3 (Moderate) | ⏱️ ~20 minutes Suggested reviewers
🚥 Pre-merge checks | ✅ 4 | ❌ 1❌ Failed checks (1 warning)
✅ Passed checks (4 passed)
✏️ Tip: You can configure your own custom pre-merge checks in the settings. Thanks for using CodeRabbit! It's free for OSS, and your support helps us grow. If you like it, consider giving us a shout-out. Comment |
Contributor
|
✅ Docker images ready for
Use these tags to pull the images for testing. 📋 Copy commandsgcloud auth configure-docker us-central1-docker.pkg.dev
docker pull us-central1-docker.pkg.dev/robusta-development/temporary-builds/holmes:fffa2b829
docker tag us-central1-docker.pkg.dev/robusta-development/temporary-builds/holmes:fffa2b829 me-west1-docker.pkg.dev/robusta-development/development/holmes-dev:fffa2b829
docker push me-west1-docker.pkg.dev/robusta-development/development/holmes-dev:fffa2b829
docker pull us-central1-docker.pkg.dev/robusta-development/temporary-builds/holmes-operator:fffa2b829
docker tag us-central1-docker.pkg.dev/robusta-development/temporary-builds/holmes-operator:fffa2b829 me-west1-docker.pkg.dev/robusta-development/development/holmes-operator-dev:fffa2b829
docker push me-west1-docker.pkg.dev/robusta-development/development/holmes-operator-dev:fffa2b829Patch Helm values in one line (choose the chart you use): HolmesGPT chart: helm upgrade --install holmesgpt ./helm/holmes \
--set registry=me-west1-docker.pkg.dev/robusta-development/development \
--set image=holmes-dev:fffa2b829 \
--set operator.registry=me-west1-docker.pkg.dev/robusta-development/development \
--set operator.image=holmes-operator-dev:fffa2b829Robusta wrapper chart: helm upgrade --install robusta robusta/robusta \
--reuse-values \
--set holmes.registry=me-west1-docker.pkg.dev/robusta-development/development \
--set holmes.image=holmes-dev:fffa2b829 \
--set holmes.operator.registry=me-west1-docker.pkg.dev/robusta-development/development \
--set holmes.operator.image=holmes-operator-dev:fffa2b829 |
✅ Deploy Preview for holmes-docs ready!
To edit notification comments on pull requests, go to your Netlify project configuration. |
The multi-backend 'logging backend' abstraction was removed a while ago: each log toolset (Datadog, Loki, VictoriaLogs) now defines its own tools, and logging_api.py only serves shared constants plus the kubernetes/logs fetch_pod_logs implementation. Make that explicit in a module docstring so nobody builds new backends on top of it, and delete what is dead: - process_time_parameters(): zero callers anywhere in the repo - logging_utils/types.py: empty file https://claude.ai/code/session_01BwJeGAGBLoby5rShhQADwq Signed-off-by: Claude <noreply@anthropic.com>
https://claude.ai/code/session_01BwJeGAGBLoby5rShhQADwq Signed-off-by: Claude <noreply@anthropic.com>
aantn
added a commit
that referenced
this pull request
Jun 10, 2026
Documentation-only change, no behavior change. The `llm_summarize` transformer predates the spill-to-disk mechanism, is disabled by default, and never worked well in practice: summarization is lossy (the original tool output is unrecoverable afterwards) and it adds latency and cost to every large tool call. Modern models do better working from the full data spilled to disk. This marks it as legacy in the module/class docstrings and adds a warning admonition to `docs/development/transformers.md`, so future contributors don't build on it and users don't enable it expecting good results. Kept for backwards compatibility with existing configs that reference it. Part of a series of targeted fixes removing old-model data limitations from built-in tools (see #2167, #2168, #2169). https://claude.ai/code/session_01BwJeGAGBLoby5rShhQADwq --- _Generated by [Claude Code](https://claude.ai/code/session_01BwJeGAGBLoby5rShhQADwq)_ <!-- This is an auto-generated comment: release notes by coderabbit.ai --> ## Summary by CodeRabbit * **Documentation** * Added notices marking the `llm_summarize` transformer as a legacy feature that is disabled by default. This feature is not recommended for new configurations due to lossy summarization, increased latency, and associated costs. Documentation has been updated to recommend the spill-to-disk mechanism as the preferred alternative for handling oversized tool results. <!-- end of auto-generated comment: release notes by coderabbit.ai --> --------- Signed-off-by: Claude <noreply@anthropic.com> Co-authored-by: Claude <noreply@anthropic.com>
This branch has not been deployed
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
2 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Problem
fetch_pod_logstrims logs from the top until they fit the single-tool token budget (truncate_logs()inlogging_utils/logging_api.py), prepending[... PREVIOUS LOGS ABOVE THIS LINE HAVE BEEN TRUNCATED]. Everything above the limit is irrecoverably dropped — and because the result is trimmed to just under the budget, the core spill-to-disk mechanism (spill_oversized_tool_result) never triggers, so the full logs are never saved anywhere the model could read them back.This design predates spill-to-disk and was aimed at older models. With modern models + the bash toolset, the better behavior is: save the full output to disk and let the model
cat/grepit.Changes
ToolInvokeContext.tool_results_dir(new field): set during tool invocation when spilling is actually available (storage dir present,HOLMES_TOOL_RESULT_STORAGE_ENABLED, and bash toolset enabled so the model can read files back). Tools can use this to decide between returning full data vs. lossy in-tool truncation. The same condition is now computed in one place (ToolCallingLLM._spill_results_dir()) and used for both the invoke context and the spill call.PodLoggingTool: skips in-tool truncation whentool_results_diris set — full logs flow to the core spill, which saves them to disk and returns a preview + file path. Falls back to the old truncation when spilling is unavailable (no behavior change for such deployments).Testing
Part of a series of targeted fixes removing old-model data limitations from built-in tools (see also the follow-up PRs for Prometheus and limit handling).
https://claude.ai/code/session_01BwJeGAGBLoby5rShhQADwq
Generated by Claude Code
Summary by CodeRabbit
Documentation
New Features
Tests