Skip to content

t1273: Add supervisor sanity-check for self-healing queue stalls #2015

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Merged
marcusquinn merged 4 commits into from
Feb 20, 2026
Merged

t1273: Add supervisor sanity-check for self-healing queue stalls #2015

Show file tree
Hide file tree
Changes from 2 commits
Commits
Show all changes
4 commits
Select commit Hold shift + click to select a range
8ea0c8f
feat: add supervisor sanity-check for self-healing queue stalls (t1273)
marcusquinn Feb 20, 2026
8fd10d8
refactor: extract sanity-check helper function and improve sed robust…
marcusquinn Feb 20, 2026
ef42b00
fix: address CodeRabbit review — stdout leakage, COALESCE, regex esca…
marcusquinn Feb 20, 2026
bee152b
fix: escape task_id dots in grep, eliminate IFS mutation, fix fallbac…
marcusquinn Feb 20, 2026
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
1 change: 1 addition & 0 deletions .agents/scripts/supervisor-helper.sh
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -232,6 +232,7 @@ source "${SUPERVISOR_MODULE_DIR}/ai-reason.sh"
source "${SUPERVISOR_MODULE_DIR}/ai-actions.sh"
source "${SUPERVISOR_MODULE_DIR}/issue-audit.sh"
source "${SUPERVISOR_MODULE_DIR}/routine-scheduler.sh"
source "${SUPERVISOR_MODULE_DIR}/sanity-check.sh"

# Valid states for the state machine
# shellcheck disable=SC2034 # Used by supervisor/state.sh
Expand Down
44 changes: 44 additions & 0 deletions .agents/scripts/supervisor/pulse.sh
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -612,6 +612,31 @@ get_task_timeout() {
return 0
}

#######################################
# Phase 0.9 helper: run sanity check for a single repo
# Extracted to avoid duplicating the open-task check + sanity-check + auto-pickup
# sequence for the multi-repo and single-repo code paths in cmd_pulse.
#######################################
_run_sanity_check_for_repo() {
local repo_path="$1"
local todo_file="$repo_path/TODO.md"

if [[ ! -f "$todo_file" ]]; then
return
fi

local open_count
open_count=$(grep -cE '^\s*- \[ \] t[0-9]+' "$todo_file" 2>/dev/null || echo 0)
if [[ "$open_count" -gt 0 ]]; then
local sanity_fixed
sanity_fixed=$(run_sanity_check "$repo_path" 2>>"$SUPERVISOR_LOG")
if [[ "${sanity_fixed:-0}" -gt 0 ]]; then
log_info "Phase 0.9: Sanity check fixed $sanity_fixed issue(s) in $repo_path — re-running auto-pickup"
cmd_auto_pickup --repo "$repo_path" 2>>"$SUPERVISOR_LOG" || true
fi
fi
}

#######################################
# Supervisor pulse - stateless check and dispatch cycle
# Designed to run via cron every 5 minutes
Expand Down Expand Up @@ -1938,6 +1963,25 @@ cmd_pulse() {
fi
fi

# Phase 0.9: Sanity check — question assumptions when queue appears empty
# Runs after all recovery phases (0.5–0.8, 1–1d) and before dispatch (Phase 2).
# When the queue has zero dispatchable tasks but open tasks exist in TODO.md,
# cross-references DB state, TODO.md state, and system state to find
# contradictions that cause silent stalls. Fixes: stale claims on DB-failed
# tasks, failed blocker cascades, missing #auto-dispatch tags, DB orphans.
local queued_before_sanity
queued_before_sanity=$(db "$SUPERVISOR_DB" "SELECT COUNT(*) FROM tasks WHERE status = 'queued';" 2>/dev/null || echo 0)

@gemini-code-assist gemini-code-assist Bot Feb 20, 2026

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

high

Suppressing stderr with 2>/dev/null can hide important underlying errors from db (like SQL syntax errors or connection issues) and violates general repository rules. For SELECT operations, 2>/dev/null should generally be avoided to ensure diagnostic information is visible for debugging. The || echo 0 guard is sufficient to handle cases where no rows are returned or the command fails gracefully. Please remove 2>/dev/null to allow genuine errors to be visible for debugging. This applies to other similar calls in this file (lines 1955, 1971).

Suggested change
queued_before_sanity=$(db "$SUPERVISOR_DB" "SELECT COUNT(*) FROM tasks WHERE status = 'queued';" 2>/dev/null || echo 0)
queued_before_sanity=$(db "$SUPERVISOR_DB" "SELECT COUNT(*) FROM tasks WHERE status = 'queued';" || echo 0)
References
  1. Do not suppress stderr with '2>/dev/null' so that actual syntax or system errors remain visible for debugging. This rule applies generally, with specific exceptions for non-blocking SQLite writes that may fail due to DB lock or missing columns, which is not the case here for a SELECT operation.

if [[ "$queued_before_sanity" -eq 0 ]]; then
# No queued tasks — check if there are open tasks in TODO.md that should be
if [[ -n "$all_repos" ]]; then
while IFS= read -r repo_path; do
_run_sanity_check_for_repo "$repo_path"
done <<<"$all_repos"
else
_run_sanity_check_for_repo "$(pwd)"
fi
fi
Comment on lines +1974 to +1983

@gemini-code-assist gemini-code-assist Bot Feb 20, 2026

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

medium

The logic inside the if [[ -n "$all_repos" ]] block and the else block is nearly identical. This duplication can be avoided by refactoring the common logic into a helper function. This would make the code more maintainable and easier to read.

_run_sanity_check_for_repo() {
    local repo_path="$1"
    local todo_file="$repo_path/TODO.md"
    if [[ ! -f "$todo_file" ]]; then
        return
    fi

    local open_count
    open_count=$(grep -cE '^\s*- \[ \] t[0-9]+' "$todo_file" || echo 0)
    if [[ "$open_count" -gt 0 ]]; then
        local sanity_fixed
        sanity_fixed=$(run_sanity_check "$repo_path" 2>>"$SUPERVISOR_LOG")
        if [[ "${sanity_fixed:-0}" -gt 0 ]]; then
            log_info "Phase 0.9: Sanity check fixed $sanity_fixed issue(s) in $repo_path — re-running auto-pickup"
            cmd_auto_pickup --repo "$repo_path" 2>>"$SUPERVISOR_LOG" || true
        fi
    fi
}

if [[ "$queued_before_sanity" -eq 0 ]]; then
    # No queued tasks — check if there are open tasks in TODO.md that should be
    if [[ -n "$all_repos" ]]; then
        while IFS= read -r repo_path; do
            _run_sanity_check_for_repo "$repo_path"
        done <<<"$all_repos"
    else
        _run_sanity_check_for_repo "$(pwd)"
    fi
fi
References
  1. In shell scripts, extract repeated logic into an internal helper function to improve maintainability. This applies even for standalone scripts where external source dependencies are avoided.

Comment thread
coderabbitai[bot] marked this conversation as resolved.

# Phase 2: Dispatch queued tasks up to concurrency limit

if [[ -n "$batch_id" ]]; then
Expand Down
Loading
Loading