Repository navigation
fix(OMN-13947): suppress runner drift alert noise - #2243
Conversation
|
Warning Review limit reachedYou’ve reached a temporary PR review limit under our Fair Usage Limits Policy. Next review available in: 11 minutes Enable usage-based reviews in Billing to review now. Otherwise, wait until the next included review is available. How can I continue?After more reviews become available, a review can be triggered using the To avoid repeated limits, reduce automatic review volume by pausing incremental auto-reviews earlier, using label-based review opt-in, excluding WIP or generated PR titles, or requesting reviews manually when the PR is ready. If your team needs uninterrupted high-volume reviews, an organization admin can enable usage-based reviews. How do review limits work?CodeRabbit enforces per-developer PR review limits for each organization. Most developers receive the normal plan review availability. For paid Pro and Pro+ PR reviews, CodeRabbit uses adaptive limits for sustained high-volume activity. When a developer's recent PR review activity reaches the 95th percentile or higher among CodeRabbit users, additional reviews become available more gradually as earlier reviews age out of the rolling window. Please refer docs for additional details. Review details⚙️ Run configurationConfiguration used: defaults Review profile: CHILL Plan: Pro Run ID: 📒 Files selected for processing (2)
📝 WalkthroughWalkthroughThis PR updates runner monitor health classification and alert state handling, then adds tests for the new alert suppression behavior. It also disables UV cache reuse in the url-authority-gate workflow job. ChangesRunner Monitor Alert Logic
Estimated code review effort: 3 (Moderate) | ~25 minutes Sequence Diagram(s)sequenceDiagram
participant ComponentA
participant ComponentB
ComponentA->>ComponentB: observable interaction
Possibly related PRs
🚥 Pre-merge checks | ✅ 5✅ Passed checks (5 passed)
✨ Finishing Touches🧪 Generate unit tests (beta)
Comment |
![coderabbitai[bot]](https://avatars.githubusercontent.com/in/347564?s=60&v=4){kind=small}
There was a problem hiding this comment.
Caution
Some comments are outside the diff and can’t be posted inline due to platform limitations.
⚠️ Outside diff range comments (1)
docker/runners/runner-monitor.sh (1)
841-886: 🩺 Stability & Availability | 🟠 Major | ⚡ Quick winKeep retrying remediation when the actionable count is unchanged.
With
current_alert_count > 0,prev_alert_count > 0, and the same count, execution falls to the finalelse, logsOK, and never callsauto_bounce. That changes remediation behavior for persistent or swapped same-count targets and can leave actionable runners unrepaired after a skipped/failed bounce.Proposed fix
elif [[ $current_alert_count -gt 0 ]] && [[ $prev_alert_count -gt 0 ]] && [[ $current_alert_count -ne $prev_alert_count ]]; then # Transition: actionable issue count changed. Raw GitHub/Docker drift count # can churn under saturation; do not page on that noise. detail=$(printf '%s\n' "${unhealthy_list[@]}") @@ log "UPDATE: ${current_alert_count} actionable issue(s) (was ${prev_alert_count}), ${current_unhealthy_count} raw unhealthy. wedge=${wedge_count} crashloop=${crashloop_count} stuck_created=${stuck_created_count} offline_idle_recreate=${offline_idle_recreate_count}. Slack notified." auto_bounce "${remediation_targets}" +elif [[ $current_alert_count -gt 0 ]]; then + # Ongoing actionable issue with unchanged count: suppress Slack, but keep + # remediation behavior active. + log "ALERT ongoing: ${current_alert_count} actionable issue(s), ${current_unhealthy_count} raw unhealthy. wedge=${wedge_count} crashloop=${crashloop_count} stuck_created=${stuck_created_count} offline_idle_recreate=${offline_idle_recreate_count}. Slack unchanged." + auto_bounce "${remediation_targets}" + elif [[ $current_alert_count -eq 0 ]] && [[ $prev_alert_count -gt 0 ]]; then🤖 Prompt for AI Agents
Verify each finding against current code. Fix only still-valid issues, skip the rest with a brief reason, keep changes minimal, and validate. In `@docker/runners/runner-monitor.sh` around lines 841 - 886, The runner remediation flow in runner-monitor.sh skips auto_bounce when current_alert_count stays the same, because the same-count case falls through to the final “OK” branch. Update the alert-state handling around the current_alert_count/prev_alert_count checks so persistent actionable issues still trigger remediation even when the count is unchanged. Keep the Slack/update logging behavior, but ensure auto_bounce(remediation_targets) runs for unchanged nonzero actionable alerts, especially in the branch that currently only handles count changes.
🤖 Prompt for all review comments with AI agents
Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.
Outside diff comments:
In `@docker/runners/runner-monitor.sh`:
- Around line 841-886: The runner remediation flow in runner-monitor.sh skips
auto_bounce when current_alert_count stays the same, because the same-count case
falls through to the final “OK” branch. Update the alert-state handling around
the current_alert_count/prev_alert_count checks so persistent actionable issues
still trigger remediation even when the count is unchanged. Keep the
Slack/update logging behavior, but ensure auto_bounce(remediation_targets) runs
for unchanged nonzero actionable alerts, especially in the branch that currently
only handles count changes.
ℹ️ Review info
⚙️ Run configuration
Configuration used: defaults
Review profile: CHILL
Plan: Pro
Run ID: 583d6de2-6b07-4628-ba32-2eccf2a597e5
📒 Files selected for processing (2)
docker/runners/runner-monitor.shtests/unit/observability/runner_health/test_runner_monitor_wedge_detection.py
1 participant
Summary
Verification
Live note
Patched /home/jonah/.omnibase/runners/docker/runners/runner-monitor.sh on .201 using temp file + bash -n + backup. No runner containers were restarted or reset.
Summary by CodeRabbit
Evidence-Source: OCC#3759
Evidence-Ticket: OMN-13947