feat: implement issue #1410 — [Phase 4] Add a stall canary + rollback trigger for the Class-2 timer changes, before narrowing them

[don-petry]

User description

Closes #1410

Implemented by dev-lead agent. Please review.

Summary by CodeRabbit

• New Features

  • Added automated detection and reporting of stalled, review-ready pull requests.
  • Added release-tag management for creating and moving channel tags.
  • Added telemetry for LSP startup performance.

• Bug Fixes

  • Improved authentication checks and fallback handling for review automation.
  • Added safeguards for incomplete data, invalid thresholds, and unavailable services.

• Documentation

  • Documented stalled-review monitoring, rollback procedures, and scheduled review automation.
  • Expanded guidance for CI fixes and review verification.

• Tests

  • Added coverage for stalled-review detection, regressions, gating rules, and report generation.

---

CodeAnt-AI Description

Add a safety net for PRs that stop receiving automated reviews and improve recovery when review engines or fix workflows fail

What Changed

• Daily health checks now report open, CI-green PRs that still require review, have not been reviewed at the current commit, and have been idle beyond the configured threshold.
• Intentional holds such as needs-human-review, dev-lead:hands-off, and initiative:hold are excluded from stall reports; detection does not modify PRs.
• Automated review runs now handle model fallback, rate-limit exhaustion, missing engines, timeout reporting, token usage, and MCP failures with clearer outcomes.
• Dev-Lead fixes now work in isolated PR worktrees, avoid overwriting concurrent changes, prevent self-cancelling fixes from being pushed, and preserve redacted session output for troubleshooting.
• Review triage includes current advisory bot feedback and can retry stale CHANGES_REQUESTED reviews after new commits.
• Added regression coverage and documented stall thresholds, rollback triggers, and the monitoring process.

Impact

✅ Earlier detection of silent review stalls
✅ Fewer repeated calls to exhausted review engines
✅ Safer automated fixes without clobbering concurrent PR changes

💡 Usage Guide

Checking Your Pull Request

Every time you make a pull request, our system automatically looks through it. We check for security issues, mistakes in how you're setting up your infrastructure, and common code problems. We do this to make sure your changes are solid and won't cause any trouble later.

Talking to CodeAnt AI

Got a question or need a hand with something in your pull request? You can easily get in touch with CodeAnt AI right here. Just type the following in a comment on your pull request, and replace "Your question here" with whatever you want to ask:

@codeant-ai ask: Your question here

This lets you have a chat with CodeAnt AI about your pull request, making it easier to understand and improve your code.

Example

@codeant-ai ask: Can you suggest a safer alternative to storing this secret?

Preserve Org Learnings with CodeAnt

You can record team preferences so CodeAnt AI applies them in future reviews. Reply directly to the specific CodeAnt AI suggestion (in the same thread) and replace "Your feedback here" with your input:

@codeant-ai: Your feedback here

This helps CodeAnt AI learn and adapt to your team's coding style and standards.

Example

@codeant-ai: Do not flag unused imports.

Retrigger review

Ask CodeAnt AI to review the PR again, by typing:

@codeant-ai: review

Check Your Repository Health

To analyze the health of your code repository, visit our dashboard at https://app.codeant.ai. This tool helps you identify potential issues and areas for improvement in your codebase, ensuring your repository maintains high standards of code health.

[codeant-ai]

🤖 CodeAnt AI — Review Status

Status	Commit	Started (UTC)	Finished (UTC)
✅ Incremental review completed	99d3d05	Aug 02, 2026 · 08:14	08:15
✅ Incremental review completed	ec4a29e	Aug 02, 2026 · 07:04	07:05
✅ Incremental review completed	7cdf267	Aug 02, 2026 · 06:00	06:01
✅ Reviewed your PR	36b4d72	Aug 02, 2026 · 02:55	02:57

[chatgpt-codex-connector]

You have reached your Codex usage limits for code reviews. You can see your limits in the Codex usage dashboard.

[coderabbitai]

📝 Walkthrough

Walkthrough

Adds detect-only stalled-PR monitoring, daily issue reporting, regression coverage, workflow authentication checks, release-tag helpers, engine updates, review-flow changes, and documentation updates.

Changes

Stalled PR detection and reporting

Layer / File(s)	Summary
Detection rules and validation scripts/lib/pr-stall-detect.sh, tests/test_pr_stall_detect.bats, tests/test_pr_stall_regression.bats	Defines stall criteria, human-gated exclusions, timestamp handling, report rendering, and regression coverage.
Open PR scan and state collection scripts/pr_stall_scan.sh	Scans open PR metadata, CI state, reviews, labels, and activity, then exports scan results.
Health workflow issue reporting .github/workflows/daily-pr-review-health.yml, .github/workflows/lint.yml, docs/agentic-interaction-model.md	Runs the scan, truncates reports, manages automated issues, registers tests, and documents rollback and monitoring procedures.

Workflow, release, and runtime updates

Layer / File(s)	Summary
Workflow guards and authentication .github/workflows/actions-fleet-monitor.yml, .github/workflows/pr-review-sweep.yml, .github/workflows/pr-review.yml, README.md	Adds token guards, canonical PAT handling, authentication-scope checks, Claude CLI caching, and scheduled automation documentation.
Release tag API helpers scripts/cut-release.sh	Adds ref normalization, SHA resolution, annotated tag creation, and channel-tag movement.
Engine execution and telemetry scripts/engine.sh, scripts/lib/token-metrics.sh, tests/dev-lead/unit/*	Reorganizes engine helpers, enables Claude LSP capture, emits cold-start records, and adjusts test environment setup.
Review fallback and validation paths scripts/review-batch.sh, scripts/review-one-pr.sh, scripts/validate-engines.sh, scripts/list-prs.sh	Adds fallback handling, current-head review gating, advisory preparation, workspace-trust validation, and documentation cleanup.

Review remediation flow updates

Layer / File(s)	Summary
Dev-lead review execution scripts/dev-lead-fix-reviews.sh	Adds duplicated checkout, marker, commit, rate-limit, no-op, and dispatch implementations.
Post-review item cleanup scripts/post-pr-review.sh	Adds a duplicate obsolete-agent-item cleanup function.
Remediation prompt procedures prompts/dev-lead/fix-ci.md, prompts/dev-lead/fix-reviews.md	Adds targeted-fix, verification, thread-handling, and re-verification procedures.

Documentation and validation content

Layer / File(s)	Summary
Initiative pipeline documentation docs/initiatives/idea-to-initiative-pipeline.md	Documents enhancement, filtering, marker continuity, backfill, and dry-run behavior.
Validation and review templates evals/validate-cases.py, prompts/cascade-action.md, prompts/triage.md	Adds duplicate validation helpers and duplicate review-template sections.

Estimated code review effort: 5 (Critical) | ~120 minutes

Possibly related issues

• [Phase 4] Conflicted-PR / queue-wide deadlock has no detection or resolution path — auto-rebase fires only on push:main, and #1410's stall detector keys on REVIEW_REQUIRED #1424 — Extends the same daily PR health path with complementary conflicted or unmergeable PR stall detection.

Possibly related PRs

• petry-projects/.github-private#1032 — Adds related daily health detection, reporting, workflow integration, and Bats coverage.
• petry-projects/.github-private#1022 — Overlaps in scripts/engine.sh and engine fallback tests.
• petry-projects/.github-private#443 — Overlaps in workflow authentication and fine-grained PAT scope validation.

Suggested labels: needs-human-review

🚥 Pre-merge checks | ✅ 3 | ❌ 2

❌ Failed checks (2 warnings)

Check name	Status	Explanation	Resolution
Out of Scope Changes check	⚠️ Warning	The pull request includes unrelated duplicate logic, secret-authentication changes, release-tag helpers, prompt edits, and broad engine modifications.	Remove unrelated changes and duplicate definitions, then limit the pull request to the stall detector, health reporting, tests, and rollback documentation.
Description check	⚠️ Warning	The description explains the main changes but omits the required Interaction contract and Checklist sections.	Add the required Interaction contract details and complete the Checklist, including shellcheck and documentation confirmations.

✅ Passed checks (3 passed)

Check name	Status	Explanation
Linked Issues check	✅ Passed	The detector, exclusions, regression tests, daily reporting, rollback procedure, and monitoring window address issue #1410.
Docstring Coverage	✅ Passed	Docstring coverage is 82.28% which is sufficient. The required threshold is 80.00%.
Title check	✅ Passed	The title clearly identifies the stall canary and rollback trigger for the Class-2 timer changes.

✨ Finishing Touches

📝 Generate docstrings

• Create stacked PR
• Commit on current branch

🧪 Generate unit tests (beta)

• Create PR with unit tests
• Commit unit tests in branch dev-lead/issue-1410-20260802-0239

---

Thanks for using CodeRabbit! It's free for OSS, and your support helps us grow. If you like it, consider giving us a shout-out.

❤️ Share

• X
• Mastodon
• Reddit
• LinkedIn

_{Comment @coderabbitai help to get the list of available commands.}

[qodo-code-review]

PR Summary by Qodo

Add stalled-PR canary scan + documented rollback trigger for timer narrowing

✨ Enhancement 🧪 Tests 📝 Documentation ⚙️ Configuration changes 🕐 40+ Minutes

AI Description

• Add a detect-only daily scan for PRs stuck CI-green + REVIEW_REQUIRED with no activity.
• Surface stalled-PR candidates via the existing health-check automated-report issue channel.
• Document rollback triggers/procedure and add unit/regression tests for the stall detector.

Diagram

graph TD
  A["daily-pr-review-health.yml"] --> B["pr_stall_scan.sh"] --> C{{"GitHub API"}}
  B --> D["pr-stall-detect lib"]
  B --> E["pr_stall_report.md"] --> F{{"Health-check issue"}}

Loading

High-Level Assessment

The following are alternative approaches to this PR:

1. GitHub Search/GraphQL-driven detection

• ➕ Potentially fewer per-PR API calls than looping gh pr view for every open PR
• ➕ Can express richer predicates server-side (reviewDecision + updatedAt + labels)
• ➖ Harder to precisely replicate current “reviewed at head” marker checks and budget-marker logic
• ➖ More complex to maintain/debug than the existing gh+json snapshot flow; may require GraphQL pagination and schema maintenance

2. Automatic remediation instead of detect-only (label/comment/re-dispatch)

• ➕ Could resolve some stalls without human intervention
• ➕ Fewer manual follow-ups once a stall is detected
• ➖ Contradicts the stated Phase 4 goal (canary + instrument before narrowing) and increases risk of unintended PR mutation
• ➖ Higher chance of feedback loops / runaway automation; needs additional safety controls and auditing

Recommendation: Keep the current detect-only approach wired into the existing daily health workflow. It minimizes operational risk while creating the pushed signal needed before narrowing Class-2 backstops, and it deliberately reuses canonical gating/budget logic to avoid false positives on intentional stops. Revisit Search/GraphQL optimization only if API rate limits become a practical problem.

Files changed (7) +844 / -0

Enhancement (2) +369 / -0

pr-stall-detect.shAdd pure stall-detection library with gating and report rendering +191/-0

Add pure stall-detection library with gating and report rendering

• Introduces a bash library of pure functions to classify stalled PRs based on CI status, review decision, reviewed-at-head marker, idle time, and human gating. Reuses canonical budget/escalation logic and provides markdown report generation for health reporting.

scripts/lib/pr-stall-detect.sh

pr_stall_scan.shAdd GH API-based scanner to detect stalled PRs and generate a daily report +178/-0

Add GH API-based scanner to detect stalled PRs and generate a daily report

• Implements a repo-wide scan over open PRs using gh + jq, computing CI rollups, reviewed-at-head status, human gating, and idle minutes via shared event gathering. Produces pr_stall_report.md and exports HAS_STALL/STALL_COUNT for the workflow to surface via the existing issue channel without mutating PRs.

scripts/pr_stall_scan.sh

Tests (2) +331 / -0

test_pr_stall_detect.batsAdd unit tests for stall detection rules and gating behavior +222/-0

Add unit tests for stall detection rules and gating behavior

• Adds Bats unit tests covering the core stall predicate (strict > threshold), env overrides, defensive handling of malformed inputs, human-gate exclusions, and markdown report rendering.

tests/test_pr_stall_detect.bats

test_pr_stall_regression.batsAdd regression tests for the 'un-eventable green' stall scenario +109/-0

Add regression tests for the 'un-eventable green' stall scenario

• Pins the key safety property: a SonarCloud-only SUCCESS check rollup is classified as CI-passing and, when combined with REVIEW_REQUIRED + idleness, is detected as a stall unless intentionally gated. Confirms the stall signal clears once reviewed-at-head is true.

tests/test_pr_stall_regression.bats

Documentation (1) +79 / -0

agentic-interaction-model.mdDocument stall detector, rollback trigger, and monitoring window for timer narrowing +79/-0

Document stall detector, rollback trigger, and monitoring window for timer narrowing

• Adds a new section specifying why the stall detector exists, what it detects, and the explicit rollback thresholds and revert procedure for future Class-2 timer narrowing. Defines a 14-day monitoring window and makes the daily health scan the pushed watcher.

docs/agentic-interaction-model.md

Other (2) +65 / -0

daily-pr-review-health.ymlRun daily stalled-PR scan and open an automated health-check issue +63/-0

Run daily stalled-PR scan and open an automated health-check issue

• Adds a new step to run scripts/pr_stall_scan.sh during the existing daily health workflow. Truncates the generated markdown report if oversized, and creates a date-stamped health-check/automated-report issue when HAS_STALL is set.

.github/workflows/daily-pr-review-health.yml

lint.ymlInclude stall-detector Bats suites in CI lint/test matrix +2/-0

Include stall-detector Bats suites in CI lint/test matrix

• Registers the new stall detection unit and regression Bats tests so they run in the workflow lint job.

.github/workflows/lint.yml

[gemini-code-assist]

Code Review

This pull request introduces a stall detection mechanism for pull requests, including documentation, a detection library, a daily scan script, and comprehensive unit and regression tests. The review feedback highlights critical shell scripting issues where short-circuit && operators under set -e could cause premature script termination, a date-parsing bug in the test suite, and an opportunity to improve test isolation by using $BATS_TEST_TMPDIR.

[qodo-code-review]

Code Review by Qodo

🐞 Bugs (0) 📘 Rule violations (1) 📜 Skill insights (0)

Context used

✅ Compliance rules (platform): 48 rules

1. GITHUB_TOKEN suppresses downstream triggers ✓ Resolved 🐞 Bug ☼ Reliability

Description

The new stall-candidate issue is created using ${{ github.token }}, but this repo’s documented
“GITHUB_TOKEN event-boundary rule” says events emitted by GITHUB_TOKEN do not trigger other
workflows; this can prevent issue-driven automation (e.g. add-to-project.yml) from running for the
stall alert issue.
That weakens the intended pushed-signal channel by making the alert less visible/less automatically
routed.

Code

.github/workflows/daily-pr-review-health.yml[R207-212]

+      - name: Create stall-candidate issue
+        if: env.HAS_STALL == 'true' && hashFiles('pr_stall_report.md') != ''
+        uses: actions/github-script@3a2844b7e9c422d3c10d287c895573f7108da1b3  # v9.0.0
+        with:
+          github-token: ${{ github.token }}
+          script: |

Relevance

●●● Strong

Team previously accepted switching issue/label creation off github.token to ensure downstream
issues workflows trigger.

PR-#231

ⓘ Recommendations generated based on similar findings in past PRs

Evidence

The stall issue creation step explicitly uses github.token. The repo docs state that
webhooks/events created by GITHUB_TOKEN cannot trigger other workflows, and add-to-project.yml
is explicitly an issues-triggered workflow that would be impacted by that boundary.

.github/workflows/daily-pr-review-health.yml[207-212]
docs/agentic-interaction-model.md[232-239]
.github/workflows/add-to-project.yml[32-36]
PR-#231

Agent prompt

The issue below was found during a code review. Follow the provided context and guidance below and implement a solution

### Issue description
The stall-candidate issue is created via `actions/github-script` using `github-token: ${{ github.token }}`. Per the repo’s documented rule, `GITHUB_TOKEN`-authored events don’t trigger downstream workflows, so issue-based automations (like auto-add-to-project) may not run for the stall alert.

### Issue Context
This workflow already uses a PAT (`secrets.DON_PETRY_BOT_GH_PAT`) for the scan step. Using the same PAT (or another appropriate bot/App token) for issue creation will allow downstream issue-triggered workflows to run.

### Fix Focus Areas
- .github/workflows/daily-pr-review-health.yml[207-212]

ⓘ Copy this prompt and use it to remediate the issue with your preferred AI generation tools

---

2. Partial scan false all-clear ✓ Resolved 🐞 Bug ≡ Correctness

Description

scripts/pr_stall_scan.sh skips a PR when gh pr view fails (including rate limits) and continues,
but it still exports HAS_STALL=false if no candidates were found among the remaining PRs.
This can produce a false “no stalled PRs” result even though the scan did not actually evaluate
every open PR.

Code

scripts/pr_stall_scan.sh[R94-98]

+  if ! snapshot=$(gh pr view "$pr" --repo "$REPO" \
+        --json headRefOid,statusCheckRollup,reviewDecision,reviews,comments,labels,title,url,updatedAt 2>/dev/null); then
+    echo "  skip PR #${pr} — could not fetch (deleted, no access, or rate-limited)"
+    continue
+  fi

Relevance

●●● Strong

Similar gh/API partial-scan risks were fixed before; team prefers fail-closed/track errors over
silent continues.

PR-#176
PR-#303
PR-#471

ⓘ Recommendations generated based on similar findings in past PRs

Evidence

The scan’s per-PR fetch failure path does a continue without tracking an error, and later sets
HAS_STALL=false purely from the candidate count. Elsewhere in the repo, GitHub API rate limits are
detected and handled explicitly to avoid proceeding with partial/unknown data.

scripts/pr_stall_scan.sh[92-98]
scripts/pr_stall_scan.sh[149-173]
scripts/review-one-pr.sh[572-578]
PR-#176

Agent prompt

The issue below was found during a code review. Follow the provided context and guidance below and implement a solution

### Issue description
The stall scan can silently skip PRs on `gh pr view` failures and then still write `HAS_STALL=false`, which is an incorrect all-clear when coverage is incomplete.

### Issue Context
This job is intended to be a safety-net detector; if the scan is incomplete, the workflow should not claim the fleet is clear. The repo already treats GitHub API rate limits as a special case elsewhere (skip/fail explicitly), rather than silently degrading.

### Fix Focus Areas
- scripts/pr_stall_scan.sh[92-98]
- scripts/pr_stall_scan.sh[149-173]

ⓘ Copy this prompt and use it to remediate the issue with your preferred AI generation tools

---

3. Noncompliant secret DON_PETRY_BOT_GH_PAT 📘 Rule violation § Compliance

Description

daily-pr-review-health.yml sets GH_TOKEN from secrets.DON_PETRY_BOT_GH_PAT, which does not
follow the required GH_PAT_<ACCOUNT>[_<QUALIFIER>] naming pattern for PAT secrets. This undermines
consistent token-audit and secret hygiene expectations.

Code

.github/workflows/daily-pr-review-health.yml[R191-195]

+      - name: Scan for stalled PRs
+        env:
+          GH_TOKEN: ${{ secrets.DON_PETRY_BOT_GH_PAT }}
+          REPO: ${{ github.repository }}
+        run: bash scripts/pr_stall_scan.sh

Relevance

● Weak

Repo intentionally uses DON_PETRY_BOT_GH_PAT (renamed from GH_PAT_WORKFLOWS); pattern not enforced
historically.

PR-#99
PR-#426

ⓘ Recommendations generated based on similar findings in past PRs

Evidence

PR Compliance ID 2238698 requires PAT secret names to follow GH_PAT_<ACCOUNT>[_<QUALIFIER>]. The
added workflow step sets GH_TOKEN using secrets.DON_PETRY_BOT_GH_PAT, which does not match that
required prefix/pattern.

Rule 2238698: Enforce PAT secret naming pattern for GitHub tokens
.github/workflows/daily-pr-review-health.yml[191-195]

Agent prompt

The issue below was found during a code review. Follow the provided context and guidance below and implement a solution

## Issue description
The workflow sets `GH_TOKEN` from `secrets.DON_PETRY_BOT_GH_PAT`, but PAT secret names must match `GH_PAT_<ACCOUNT>[_<QUALIFIER>]`.

## Issue Context
This PR adds a new `GH_TOKEN` assignment in `.github/workflows/daily-pr-review-health.yml`; the referenced secret name is not compliant.

## Fix Focus Areas
- .github/workflows/daily-pr-review-health.yml[191-195]

ⓘ Copy this prompt and use it to remediate the issue with your preferred AI generation tools

---

_{To customize comments, go to the Qodo configuration screen, or learn more in the docs.}

[don-petry]

Dev-Lead — review-changes (applied)

Changes committed and pushed.

[donpetry-bot]

CI checks on this PR are still running. Once they complete, re-mention @donpetry-bot to trigger a fresh review.

Posted by the donpetry-bot PR-review cascade.

[donpetry-bot]

CI checks on this PR are still running. Once they complete, re-mention @donpetry-bot to trigger a fresh review.

Posted by the donpetry-bot PR-review cascade.

[don-petry]

Dev-Lead — review-changes (applied)

Changes committed and pushed.

[donpetry-bot]

CI checks on this PR are still running. Once they complete, re-mention @donpetry-bot to trigger a fresh review.

Posted by the donpetry-bot PR-review cascade.

[don-petry]

Dev-Lead — review-changes (applied)

Changes committed and pushed.

[don-petry]

Review — PR #1420 (#1410 stall canary + rollback)

This is the strongest PR of the epic so far, and it answers the QA Lead escalation properly rather than nominally. Verified by running, not reading:

• 25/25 test_pr_stall_detect.bats pass; all test_pr_stall_regression.bats pass.
• AC#3 is genuinely tested — the regression suite proves the un-eventable green transition is caught and still resolves, and that it clears once a review lands at head.
• AC#2's discriminator is tested the right way: "each human-gated halt on the SAME green state is fail-quiet" holds the state constant and varies only the gate. That is the assertion shape that can actually fail if the logic is wrong — most test suites get this wrong by varying both.
• AC#4 holds: wired into the existing daily-pr-review-health job, and I confirmed no new cron anywhere in the diff.
• Human-gate exclusions reuse the canonical pr_has_escalation_label rather than re-deriving the marker set — correct, and what keeps this consistent with the budget breaker.

§11.2's risk framing is right and worth keeping verbatim: "the cost of a false rollback is a return to the known noisy state (fully recoverable), while the cost of tolerating a silent stall is the #860 blindness. When in doubt, revert."

Two findings, neither blocking.

1. (should fix) STALL_MIN_AGE_MINUTES is derived from a cadence that the next story changes

§11.1 justifies the 30-minute default as "double the sweep's ≤15-min backstop cadence". But #1408 — which this story blocks — exists precisely to narrow pr-review-sweep's cron scope. The moment that lands, the stated derivation is stale, and nothing connects them: the threshold is a hardcoded 30 (scripts/lib/pr-stall-detect.sh:49) whose rationale lives only in prose.

That is a live hazard, not a theoretical one, because the sequencing guarantees the cadence changes after this merges. Suggest either:

• deriving the threshold from the sweep's actual cron at scan time (so it self-adjusts), or
• stating in §11.1 that #1408 must re-derive STALL_MIN_AGE_MINUTES when it changes the cadence, and adding that as an explicit task on #1408.

The second is cheaper and probably sufficient — but it should be written down somewhere a person implementing #1408 will actually see, not left implicit.

2. (nit) Broken cross-reference

docs/agentic-interaction-model.md:524 — §11.2 says "during the monitoring window (§11.3)", but §11.3 is The revert procedure and §11.4 is The monitoring window. Off-by-one; should be §11.4.

Worth noting the §4 classification table is now CI-verified while internal cross-references are not — this one would have been caught by nothing. Not asking for a link checker in this PR, just flagging the asymmetry.

---

Otherwise this is ready from my side. The detector is detect-only, the rollback trigger has concrete numeric conditions rather than "if it looks bad", the revert procedure names actual commands, and the monitoring window is defined. That is exactly what the escalation asked for.

[don-petry]

Dev-Lead — review-changes (applied)

Changes committed and pushed.