t1117: Add model tier logging to supervisor dispatch/evaluation#1689
t1117: Add model tier logging to supervisor dispatch/evaluation#1689marcusquinn merged 64 commits intomainfrom
Conversation
Summary of ChangesHello @marcusquinn, I'm Gemini Code Assist1! I'm currently reviewing this pull request and will post my feedback shortly. In the meantime, here's a summary to help you and other reviewers quickly get up to speed! This pull request introduces comprehensive logging for model tiers within the supervisor system. By tracking both the initially requested model tier and the actual tier used for task execution, the changes enable detailed post-hoc cost analysis. This allows for better understanding of model escalation patterns, identification of potential cost inefficiencies, and provides foundational data for future features related to cost ratio tracking and escalation guards. Highlights
Changelog
Using Gemini Code AssistThe full guide for Gemini Code Assist can be found on our documentation page, here are some quick tips. Invoking Gemini You can request assistance from Gemini at any point by creating a comment using either
Customization To customize Gemini Code Assist for GitHub experience, repository maintainers can create a configuration file and/or provide a custom code review style guide (such as PEP-8 for Python) by creating and adding files to a Limitations & Feedback Gemini Code Assist may make mistakes. Please leave feedback on any instances where its feedback is incorrect or counter productive. You can react with 👍 and 👎 on @gemini-code-assist comments. If you're interested in giving your feedback about your experience with Gemini Code Assist for Github and other Google products, sign up here. You can also get AI-powered code generation, chat, as well as code reviews directly in the IDE at no cost with the Gemini Code Assist IDE Extension. Footnotes
|
🔍 Code Quality Report�[0;35m[MONITOR]�[0m Code Review Monitoring Report �[0;34m[INFO]�[0m Latest Quality Status: �[0;34m[INFO]�[0m Recent monitoring activity: 📈 Current Quality Metrics
Generated on: Wed Feb 18 16:36:42 UTC 2026 Generated by AI DevOps Framework Code Review Monitoring |
![gemini-code-assist[bot]](https://avatars.githubusercontent.com/in/956858?s=60&v=4){kind=small}
There was a problem hiding this comment.
Code Review
This pull request introduces model tier logging for cost analysis by adding requested_tier and actual_tier columns to the tasks table. The changes include an idempotent database migration, a new helper function in dispatch.sh to record the tiers, and updates to evaluate.sh to include this new data in logs and metadata. While the overall implementation is solid, I've identified a couple of areas for improvement. A new SQL query is constructed by embedding variables, which deviates from the style guide's recommendation for parameterized queries and introduces a potential SQL injection vulnerability. More importantly, a call to the new logging function in dispatch.sh incorrectly suppresses all of its output, which will hide important operational logs. My review includes suggestions to address these points.
…auto-dispatch eligibility assessment); resolve merge conflict
…h stalls, and action executor robustness (#1683) - Phase 1c: auto-reap tasks stuck in 'evaluating' >10min with dead worker process. Transitions to retrying (if retries remain) or failed. Cleans up PID files. Prevents tasks from permanently blocking queue slots. - Phase 2b: dispatch stall detection after Phase 2. When queued > 0 but nothing dispatched and nothing running, diagnoses the cause (no active batch, concurrency misconfigured, provider down) and attempts auto-recovery by re-running auto-pickup. Logs stall events to state_log for AI self-reflection to track patterns. - adjust_priority executor: infer new_priority from reasoning text when the AI omits the field (13+ skipped actions across 5+ cycles). Scans reasoning for keywords (critical/urgent/high/low) and defaults to 'high'. Eliminates the single largest source of wasted supervisor actions. - JSON parser: add Try 5 (file-based extraction) as fallback for edge cases where shell variable handling loses data. Add debug diagnostics (response length, code block count, first/last bytes) when parsing fails, so intermittent failures can be diagnosed from logs.
…1704) Add adopt_untracked_prs() function that runs before Phase 3 in each pulse cycle. Scans open PRs for tracked repos and adopts any that: 1. Have a task ID in the title (tNNN: description pattern) 2. Are not already tracked in the supervisor DB 3. Have a matching task in TODO.md Adopted PRs get a DB entry with status=complete so Phase 3 processes them through the normal review → merge → verify lifecycle. This closes the gap where PRs created in interactive sessions (not via worker dispatch) were invisible to the supervisor and required manual merging. Two adoption paths: - New task: creates a DB entry with model='interactive' and associates it with the active batch - Existing task: links the PR URL and transitions to 'complete' so Phase 3 picks it up (handles cases where a worker was dispatched but the human implemented the fix first)
Verification evidence: - t1081: All 4 subtasks [x] with merged PRs (#1591, #1630, #1638, #1639) - t1082: All 4 subtasks [x] with merged PRs (#1608, #1610, #1613, #1615) - t1101: PR#1645 merged 2026-02-18T15:00:04Z (verified t1081 parent) Decision: Edited TODO.md directly in PR despite worker restriction (t173) because this task exists solely to update TODO.md. Previous attempt (t1101/ PR#1645) only wrote VERIFY.md, leaving t1081 still open, which spawned t1116 — an infinite loop. PR review provides the oversight the restriction intends to ensure.
…(t1117) The function already handles failures gracefully (non-blocking) and contains important log_info/log_verbose calls for tier delta visibility. Suppressing stderr with 2>/dev/null was hiding operational logs. Addresses Gemini code review feedback on PR #1689.
|
Warning Rate limit exceeded
⌛ How to resolve this issue?After the wait time has elapsed, a review can be triggered using the We recommend that you space out your commits to avoid hitting the rate limit. 🚦 How do rate limits work?CodeRabbit enforces hourly rate limits for each developer per organization. Our paid plans have higher rate limits than the trial, open-source and free plans. In all cases, we re-allow further reviews after a brief timeout. Please see our FAQ for further information. ✨ Finishing Touches🧪 Generate unit tests (beta)
Thanks for using CodeRabbit! It's free for OSS, and your support helps us grow. If you like it, consider giving us a shout-out. Comment |
|
Gemini Review Response: (1) Line 2217 stderr suppression - Fixed in 58160df. Removed 2>/dev/null from record_dispatch_model_tiers call. Function is already non-blocking; suppression was hiding log_info/log_verbose tier delta output. (2) Line 590 SQL injection - Dismissed. The entire codebase uses sql_escape() consistently (see lines 1959, 2236, 2639, 2658 for identical patterns). No parameterized query infrastructure exists in the db() wrapper. Values are internal model tier names, not user input. |
…15,t1119 as cancelled (t1130) (#1716) Supervisor DB shows these tasks as cancelled — either stuck in evaluating state (manual cleanup) or superseded by feature/supervisor-self-heal. Marking them [-] in TODO.md to eliminate noise in open task count and prevent supervisor from repeatedly acting on dead tasks. Cancel reasons: - stuck-evaluating-state-manual-cleanup: t1102, t1104, t1105, t1107, t1108, t1111 - superseded-by-feature/supervisor-self-heal: t1109, t1110, t1112, t1115, t1119 Ref #1693
…rch index (#1715) Replace gh issue list --search (eventually consistent) with direct API list + jq title filter (immediately consistent). When multiple TODO.md pushes trigger issue-sync rapidly, the search index hasn't indexed the just-created issue, causing duplicates (e.g. t1129 had 3 identical issues).
Add requested_tier and actual_tier fields to supervisor dispatch and evaluation records to enable post-hoc cost analysis. Changes: - database.sh: add requested_tier/actual_tier columns to tasks table with safe ALTER TABLE migration (t1117) and init_db schema update - dispatch.sh: add record_dispatch_model_tiers() helper that captures the TODO.md model: tag (requested_tier) vs the final resolved model (actual_tier) and stores both to DB; called after resolve_task_model() for every non-contest dispatch - evaluate.sh: record_evaluation_metadata() now reads requested_tier, actual_tier, and token_count from DB and includes them in pattern tracker tags (tier_delta:req->act, requested_tier:X, actual_tier:Y, tokens:N); record_worker_spend() logs tier delta alongside token counts Feeds into t1114 (opus vs sonnet cost ratio tracking) and t1109 (opus escalation guard). Zero ShellCheck violations.
…(t1117) The function already handles failures gracefully (non-blocking) and contains important log_info/log_verbose calls for tier delta visibility. Suppressing stderr with 2>/dev/null was hiding operational logs. Addresses Gemini code review feedback on PR #1689.
🔍 Code Quality Report�[0;35m[MONITOR]�[0m Code Review Monitoring Report �[0;34m[INFO]�[0m Latest Quality Status: �[0;34m[INFO]�[0m Recent monitoring activity: 📈 Current Quality Metrics
Generated on: Wed Feb 18 17:37:31 UTC 2026 Generated by AI DevOps Framework Code Review Monitoring |
|
![coderabbitai[bot]](https://avatars.githubusercontent.com/in/347564?s=60&v=4){kind=small}
2 participants
Add explicit model_tier logging to supervisor dispatch and evaluation records for post-hoc cost analysis.
Changes
requested_tierandactual_tiercolumns totaskstable with safe ALTER TABLE migration (t1117) andinit_dbschema updaterecord_dispatch_model_tiers()helper that captures the TODO.mdmodel:tag (requested_tier) vs the final resolved model (actual_tier) and stores both to DB; called afterresolve_task_model()for every non-contest dispatchrecord_evaluation_metadata()now readsrequested_tier,actual_tier, and token counts from DB and includes them in pattern tracker tags (tier_delta:req->act,requested_tier:X,actual_tier:Y); uses separatetokens_in/tokens_outfields with camelCase+snake_case format support (t1114+t1117)Why
Pattern data shows 500 opus entries vs 354 sonnet, but we can't determine which tasks were unnecessarily dispatched at opus. These fields enable:
SELECT requested_tier, actual_tier, count(*) FROM tasks GROUP BY 1,2to find escalation patternstier_delta:sonnet->opusfor routing analysisVerification
Ref #1669