Repository navigation
ROB-3754 - Disable todowrite behavior controls in scheduled prompts - #1959
Conversation
Disable TodoWrite instructions/reminder for scheduled prompt runs. Without this, the model writes the report and then issues a final TodoWrite call to mark its last todo complete, which produces one extra ANSWER_END turn whose content is just a stub acknowledgement — that stub becomes ChatResponse.analysis while the actual report is buried in conversation_history. https://claude.ai/code/session_01DXoK9Pyd2v6CsowRikjNbZ Signed-off-by: Claude <noreply@anthropic.com>
https://claude.ai/code/session_01DXoK9Pyd2v6CsowRikjNbZ Signed-off-by: Claude <noreply@anthropic.com>
![claude[bot]](https://avatars.githubusercontent.com/in/1236702?s=60&v=4){kind=small}
There was a problem hiding this comment.
Claude Code Review
This repository is configured for manual code reviews. Comment @claude review to trigger a review and subscribe this PR to future pushes, or @claude review once for a one-time review.
Tip: disable this comment in your organization's Code Review settings.
📂 Previous Runs📜 #5 · Run @ __f468773__ (#25106706845) — Apr 29, 11:47 UTC✅ Results of HolmesGPT evalsAutomatically triggered by commit f468773 on branch Results of HolmesGPT evals
Benchmark Comparison DetailsBaseline: latest ci-benchmark experiment on master Status: Success - 74 test/model combinations loaded Benchmark experiment:
No benchmark data available for comparison. Benchmark has no cost, total tokens, cached tokens data. Will appear after the next weekly benchmark run. Comparison indicators:
📜 #4 · Run @ __988e8d1__ (#25060639736) — Apr 28, 15:19 UTC✅ Results of HolmesGPT evalsAutomatically triggered by commit 988e8d1 on branch Results of HolmesGPT evals
Benchmark Comparison DetailsBaseline: latest ci-benchmark experiment on master Status: Success - 74 test/model combinations loaded Benchmark experiment:
No benchmark data available for comparison. Benchmark has no cost, total tokens, cached tokens data. Will appear after the next weekly benchmark run. Comparison indicators:
📜 #3 · Run @ __3dddaef__ (#25060534071) — Apr 28, 15:10 UTC✅ Results of HolmesGPT evalsAutomatically triggered by commit 3dddaef on branch Results of HolmesGPT evals
Benchmark Comparison DetailsBaseline: latest ci-benchmark experiment on master Status: Success - 55 test/model combinations loaded Benchmark experiment:
No benchmark data available for comparison. Benchmark has no cost, total tokens, cached tokens data. Will appear after the next weekly benchmark run. Comparison indicators:
📜 #2 · Run @ __ee11424__ (#25041569760) — Apr 28, 08:17 UTC✅ Results of HolmesGPT evalsAutomatically triggered by commit ee11424 on branch Results of HolmesGPT evals
Benchmark Comparison DetailsBaseline: latest ci-benchmark experiment on master Status: Success - 74 test/model combinations loaded Benchmark experiment:
No benchmark data available for comparison. Benchmark has no cost, total tokens, cached tokens data. Will appear after the next weekly benchmark run. Comparison indicators:
📜 #1 · Run @ __52c38e4__ (#25038540696) — Apr 28, 07:12 UTC✅ Results of HolmesGPT evalsAutomatically triggered by commit 52c38e4 on branch Results of HolmesGPT evals
Benchmark Comparison DetailsBaseline: latest ci-benchmark experiment on master Status: Success - 74 test/model combinations loaded Benchmark experiment:
No benchmark data available for comparison. Benchmark has no cost, total tokens, cached tokens data. Will appear after the next weekly benchmark run. Comparison indicators:
✅ Results of HolmesGPT evalsAutomatically triggered by commit c00c38f on branch Results of HolmesGPT evals
Benchmark Comparison DetailsBaseline: latest ci-benchmark experiment on master Status: Success - 74 test/model combinations loaded Benchmark experiment:
No benchmark data available for comparison. Benchmark has no cost, total tokens, cached tokens data. Will appear after the next weekly benchmark run. Comparison indicators:
📖 Legend
🔄 Re-run evals manually
Option 1: Comment on this PR with Or with more options (one per line): Run evals on a different branch (e.g., master) for comparison:
Quick re-run: Use Option 2: Trigger via GitHub Actions UI → "Run workflow" Option 3: Add PR labels to include extra evals (applies to both automatic runs and
Examples: 🏷️ Valid tags
🤖 Valid models
Commands: CLI: |
|
No actionable comments were generated in the recent review. 🎉 ℹ️ Recent review info⚙️ Run configurationConfiguration used: Organization UI Review profile: CHILL Plan: Pro Run ID: 📒 Files selected for processing (1)
WalkthroughScheduled prompt execution now conditionally builds and passes a Changes
Estimated code review effort🎯 2 (Simple) | ⏱️ ~8 minutes Possibly related PRs
Suggested reviewers
🚥 Pre-merge checks | ✅ 4 | ❌ 1❌ Failed checks (1 warning)
✅ Passed checks (4 passed)
✏️ Tip: You can configure your own custom pre-merge checks in the settings. Thanks for using CodeRabbit! It's free for OSS, and your support helps us grow. If you like it, consider giving us a shout-out. Review rate limit: 5/8 reviews remaining, refill in 21 minutes and 40 seconds.Comment |
|
✅ Docker images ready for
Use these tags to pull the images for testing. 📋 Copy commandsgcloud auth configure-docker us-central1-docker.pkg.dev
docker pull us-central1-docker.pkg.dev/robusta-development/temporary-builds/holmes:33f4dc22
docker tag us-central1-docker.pkg.dev/robusta-development/temporary-builds/holmes:33f4dc22 me-west1-docker.pkg.dev/robusta-development/development/holmes-dev:33f4dc22
docker push me-west1-docker.pkg.dev/robusta-development/development/holmes-dev:33f4dc22
docker pull us-central1-docker.pkg.dev/robusta-development/temporary-builds/holmes-operator:33f4dc22
docker tag us-central1-docker.pkg.dev/robusta-development/temporary-builds/holmes-operator:33f4dc22 me-west1-docker.pkg.dev/robusta-development/development/holmes-operator-dev:33f4dc22
docker push me-west1-docker.pkg.dev/robusta-development/development/holmes-operator-dev:33f4dc22Patch Helm values in one line (choose the chart you use): HolmesGPT chart: helm upgrade --install holmesgpt ./helm/holmes \
--set registry=me-west1-docker.pkg.dev/robusta-development/development \
--set image=holmes-dev:33f4dc22 \
--set operator.registry=me-west1-docker.pkg.dev/robusta-development/development \
--set operator.image=holmes-operator-dev:33f4dc22Robusta wrapper chart: helm upgrade --install robusta robusta/robusta \
--reuse-values \
--set holmes.registry=me-west1-docker.pkg.dev/robusta-development/development \
--set holmes.image=holmes-dev:33f4dc22 \
--set holmes.operator.registry=me-west1-docker.pkg.dev/robusta-development/development \
--set holmes.operator.image=holmes-operator-dev:33f4dc22 |
✅ Deploy Preview for holmes-docs ready!
To edit notification comments on pull requests, go to your Netlify project configuration. |
naomi-robusta
changed the title
✅ Deploy Preview for holmes-docs ready!
To edit notification comments on pull requests, go to your Netlify project configuration. |
🔬 CLI Performance Benchmark🟡 Startup Time (no LLM)Measures
🟡 Full CLI with LLMMeasures
PR: |
Defaults to true so the report lands in ChatResponse.analysis. Can be flipped off to restore TodoWrite planning for runs that need it. https://claude.ai/code/session_01DXoK9Pyd2v6CsowRikjNbZ Signed-off-by: Claude <noreply@anthropic.com>
…//127.0.0.1:26381/git/HolmesGPT/holmesgpt into claude/scheduled-prompt-output-location-rVB1J
Plural form matches the surrounding ENABLED_SCHEDULED_PROMPTS and SCHEDULED_PROMPTS_* env vars. https://claude.ai/code/session_01DXoK9Pyd2v6CsowRikjNbZ Signed-off-by: Claude <noreply@anthropic.com>
![coderabbitai[bot]](https://avatars.githubusercontent.com/in/347564?s=60&v=4){kind=small}
There was a problem hiding this comment.
🧹 Nitpick comments (1)
holmes/common/env_vars.py (1)
173-175: Consider renamingENABLE_SCHEDULED_PROMPTS_FAST_MODEto a behavior-specific flag.The flag disables TodoWrite behavior, but “FAST_MODE” reads like a performance toggle. A descriptive name will reduce config mistakes.
♻️ Backward-compatible rename proposal
-# Disables TodoWrite for scheduled prompts so the report ends up in ChatResponse.analysis -# rather than being buried in conversation_history behind a trailing TodoWrite call. -ENABLE_SCHEDULED_PROMPTS_FAST_MODE = load_bool("ENABLE_SCHEDULED_PROMPTS_FAST_MODE", True) +# Disables TodoWrite in scheduled prompts so output stays in ChatResponse.analysis. +# Keep legacy env var as fallback for backward compatibility. +DISABLE_TODOWRITE_IN_SCHEDULED_PROMPTS = load_bool( + "DISABLE_TODOWRITE_IN_SCHEDULED_PROMPTS", + load_bool("ENABLE_SCHEDULED_PROMPTS_FAST_MODE", True), +)As per coding guidelines:
Use semantic, descriptive names for variables, functions, and components.🤖 Prompt for AI Agents
Verify each finding against the current code and only fix it if needed. In `@holmes/common/env_vars.py` around lines 173 - 175, Rename the poorly named ENABLE_SCHEDULED_PROMPTS_FAST_MODE to a descriptive, behavior-specific constant (e.g., DISABLE_TODOWRITE_FOR_SCHEDULED_PROMPTS) and implement a backward-compatible fallback so existing configs still work: create the new constant using load_bool("DISABLE_TODOWRITE_FOR_SCHEDULED_PROMPTS", None) and if that is None, fall back to load_bool("ENABLE_SCHEDULED_PROMPTS_FAST_MODE", True); update any references to use the new constant name (search for ENABLE_SCHEDULED_PROMPTS_FAST_MODE) and add a brief comment noting the old env var is deprecated.
🤖 Prompt for all review comments with AI agents
Verify each finding against the current code and only fix it if needed.
Nitpick comments:
In `@holmes/common/env_vars.py`:
- Around line 173-175: Rename the poorly named
ENABLE_SCHEDULED_PROMPTS_FAST_MODE to a descriptive, behavior-specific constant
(e.g., DISABLE_TODOWRITE_FOR_SCHEDULED_PROMPTS) and implement a
backward-compatible fallback so existing configs still work: create the new
constant using load_bool("DISABLE_TODOWRITE_FOR_SCHEDULED_PROMPTS", None) and if
that is None, fall back to load_bool("ENABLE_SCHEDULED_PROMPTS_FAST_MODE",
True); update any references to use the new constant name (search for
ENABLE_SCHEDULED_PROMPTS_FAST_MODE) and add a brief comment noting the old env
var is deprecated.
ℹ️ Review info
⚙️ Run configuration
Configuration used: Organization UI
Review profile: CHILL
Plan: Pro
Run ID: 20a90b80-426f-44fa-8c49-c9ba2cc89745
📒 Files selected for processing (2)
holmes/common/env_vars.pyholmes/core/scheduled_prompts/executor.py
🚧 Files skipped from review as they are similar to previous changes (1)
- holmes/core/scheduled_prompts/executor.py
|
us-central1-docker.pkg.dev/genuine-flight-317411/devel/holmes:scheduled-prompts-fix |
3 participants
Summary
Updated the scheduled prompts executor to disable todowrite-related behavior controls when creating chat requests for prompt execution.
Key Changes
behavior_controlsdictionary toChatRequestinitialization in_execute_promptmethodtodowrite_instructionsandtodowrite_reminderbehavior controls for scheduled prompt executionImplementation Details
The behavior controls are explicitly set to
Falseto prevent todowrite functionality from being triggered during scheduled prompt execution. This ensures that scheduled prompts execute with consistent behavior and don't generate unwanted todowrite instructions or reminders.https://claude.ai/code/session_01DXoK9Pyd2v6CsowRikjNbZ
Summary by CodeRabbit
New Features
Bug Fixes