Fix auto-pilot stall when belt dispatcher is cancelled#1665
Merged
Conversation
GitHub Actions ::warning:: commands truncate/mangle multi-line content. Emit a short annotation message and print full npm stderr in a collapsible ::group:: instead, so logs stay readable. https://claude.ai/code/session_01JhCWWDJG8PqwaSbVPCGfm6
…ation fixes Mirror the main setup-api-client changes into the consumer-repo template to prevent template drift: - Exponential backoff retry (3 attempts, 5s/10s) for transient npm errors - --legacy-peer-deps fallback on first failure - Short ::warning:: annotations with full stderr in collapsible ::group:: - Pin lru-cache@10.4.3 (was ^10.0.0) https://claude.ai/code/session_01JhCWWDJG8PqwaSbVPCGfm6
…feguard Three changes to reusable-codex-run.yml to prevent work loss on timeout: 1. Pre-timeout watchdog: A background timer fires 5 minutes before max_runtime_minutes, committing and pushing any uncommitted work so it survives the job cancellation. Killed automatically if Codex finishes before the timer fires. 2. Robust parser import: Replace sys.path-based import of codex_jsonl_parser with importlib.util.spec_from_file_location. Consumer repos (e.g. Counter_Risk) have their own tools/ package with __init__.py that shadows the Workflows tools/ on sys.path, causing "No module named 'tools.codex_jsonl_parser'". 3. Commit step always runs: Add if: always() to the "Commit and push changes" step so uncommitted work is captured even on non-zero exit codes (the watchdog handles timeout, this handles failures). https://claude.ai/code/session_01JhCWWDJG8PqwaSbVPCGfm6
parseCheckboxStates() and mergeCheckboxStates() only matched top-level checkboxes (^- \[), ignoring indented sub-tasks ( - \[). When PR Meta regenerated the PR body from the issue, auto-reconciled sub-task checkboxes were silently reverted to unchecked. This caused the keepalive loop to stall with rounds_without_task_completion: 8 despite the agent completing real work — PR #256 had 5 tasks auto-checked then immediately un-checked on every push. https://claude.ai/code/session_01JhCWWDJG8PqwaSbVPCGfm6
- P1: Add fetch/rebase before watchdog push to avoid non-fast-forward rejection when another workflow updates the branch during the run. Includes one retry with re-fetch/rebase and merge fallback. - P2: Export watchdog-saved in on.workflow_call.outputs so callers of the reusable workflow can observe the signal. - Copilot: Add git fetch before checking FETCH_HEAD to ensure it exists and is current (actions/checkout doesn't set FETCH_HEAD). - Copilot: Initialize watchdog-saved=false before background subshell so downstream consumers always get a defined value. https://claude.ai/code/session_01JhCWWDJG8PqwaSbVPCGfm6
Update WORKFLOW_OUTPUTS.md to include the new watchdog-saved output from reusable-codex-run.yml, fixing the test_reusable_workflow_outputs_documented test. https://claude.ai/code/session_01JhCWWDJG8PqwaSbVPCGfm6
The body scan in extractIssueNumberFromPull was treating patterns like "Run #2615" as issue references, causing the Upsert PR body sections check to fail with a 404 when trying to fetch non-existent issues. Add a preceding-word filter to skip #NNN when preceded by common non-issue words (run, attempt, step, job, check, task, version, v). Add 12 unit tests covering the extraction logic. https://claude.ai/code/session_01JhCWWDJG8PqwaSbVPCGfm6
… to Claude runner Closes the three remaining feature gaps between the Claude and Codex runners identified in issue #1646: 1. **Session analysis (LLM-powered)**: Reuses analyze_codex_session.py which auto-detects Claude's plain-text session log (data_source=summary) and feeds it through the same LLM analysis pipeline for structured task completion assessment. Outputs feed into the keepalive loop. 2. **Completion checkpoint comment**: Posts a PR comment summarizing completed tasks and acceptance criteria using the shared post_completion_comment.js script. Supports both claude-prompt*.md and codex-prompt*.md file names. 3. **Error diagnostics**: Adds GITHUB_STEP_SUMMARY with error table, creates a diagnostics artifact (JSON + agent output), and posts a structured PR comment on non-transient failures with recovery guidance and log links. Uses a distinct <!-- claude-failure-notification --> marker. https://claude.ai/code/session_01JhCWWDJG8PqwaSbVPCGfm6
Claude runner (reusable-claude-run.yml):
- Fix shell quoting of completed-tasks JSON by using env vars instead
of inline ${{ }} expansion which breaks on apostrophes in task names
- Declare OPENAI_API_KEY and CLAUDE_API_STRANSKE in workflow_call.secrets
so callers can pass them (matches Codex runner)
- Use printf instead of echo when writing PR body to disk to avoid
mangling of -n/-e prefixes or backslashes
- Add info log when falling back to codex-prompt file
Codex runner (reusable-codex-run.yml):
- Gate watchdog-saved=true on actual push success instead of emitting
it unconditionally after push attempts that may have both failed
- Use a fired-flag file so the watchdog kill only terminates the
background process if it's still sleeping (hasn't started its
commit/push work yet)
https://claude.ai/code/session_01JhCWWDJG8PqwaSbVPCGfm6
All four conflicts were in reusable-codex-run.yml watchdog code where our branch has the fired-flag and push-success-gating improvements vs the unchanged main version. Kept our (HEAD) version for all. https://claude.ai/code/session_01JhCWWDJG8PqwaSbVPCGfm6
- Remove "task" from the non-issue prefix filter in extractIssueNumberFromPull so "Task #123" is correctly treated as an issue reference (flagged by Codex on PAEM sync PR) - Make --legacy-peer-deps retry conditional on ERESOLVE/peer-dep errors instead of only firing on the first attempt (flagged by Copilot on TMP sync PR) - Add test for "Task #N" being treated as a valid issue ref https://claude.ai/code/session_01JhCWWDJG8PqwaSbVPCGfm6
The label sync workflow (maint-69-sync-labels.yml) has been failing since Feb 2 because npm install -g js-yaml installs to the global prefix which actions/github-script can't resolve. Install locally so Node's module resolution finds it in node_modules/. https://claude.ai/code/session_01JhCWWDJG8PqwaSbVPCGfm6
Two changes to prevent the issue stranske/Counter_Risk#34 scenario where a single cancelled belt dispatcher run strands an issue: 1. Capability-check step: dispatch the belt up to 3 times with a 15s verification window after each attempt. If the dispatched run is not queued/in_progress, retry. This catches silent cancellations before the auto-pilot moves on. 2. Branch-check loop: on the 2nd+ backoff iteration, check whether any belt dispatcher run is still active. If not, re-dispatch the belt before sleeping. This makes the loop self-healing instead of passively waiting for a run that was already cancelled. https://claude.ai/code/session_01JhCWWDJG8PqwaSbVPCGfm6
stranske
temporarily deployed
to
agent-high-privilege
GitHub Actions
Inactive
Contributor
Contributor
Automated Status SummaryHead SHA: 7e006e6
Coverage Overview
Coverage Trend
Top Coverage Hotspots (lowest coverage)
Updated automatically; will refresh on subsequent CI/Docker completions. Keepalive checklistScopeNo scope information available Tasks
Acceptance criteria
|
Contributor
🤖 Keepalive Loop StatusPR #1665 | Agent: Codex | Iteration 0/5 Current State
🔍 Failure Classification| Error type | infrastructure | |
Contributor
Keepalive Work Log (click to expand)
|
![chatgpt-codex-connector[bot]](https://avatars.githubusercontent.com/in/1144995?s=60&v=4){kind=small}
There was a problem hiding this comment.
💡 Codex Review
Here are some automated review suggestions for this pull request.
Reviewed commit: 7a11836372
ℹ️ About Codex in GitHub
Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you
- Open a pull request for review
- Mark a draft as ready
- Comment "@codex review".
If Codex has suggestions, it will comment; otherwise it will react with 👍.
Codex can also answer questions or update the PR. Try commenting "@codex address that feedback".
templates/consumer-repo/.github/workflows/agents-auto-pilot.yml
Outdated
Show resolved
Hide resolved
templates/consumer-repo/.github/workflows/agents-auto-pilot.yml
Outdated
Show resolved
Hide resolved
Contributor
There was a problem hiding this comment.
Pull request overview
This PR addresses auto-pilot stalls caused by silently cancelled belt dispatcher runs. It adds retry logic with verification to the capability-check dispatch step, and implements self-healing re-dispatch in the branch-check backoff loop.
Changes:
- Add 3-attempt retry loop with 15-second verification for belt dispatcher in capability-check step
- Add re-dispatch logic in branch-check backoff loop to recover from cancelled dispatcher runs
- Improve npm peer dependency handling in setup-api-client to detect specific error patterns
- Remove "task" from issue number extraction skip list to allow "Task #N" references
- Enhance Claude runner with LLM analysis, completion checkpoints, and error diagnostics
- Improve Codex watchdog to avoid race conditions when killing watchdog processes
- Fix npm install in maint-69-sync-labels.yml to use local instead of global installation
Reviewed changes
Copilot reviewed 9 out of 9 changed files in this pull request and generated 6 comments.
Show a summary per file
| File | Description |
|---|---|
| templates/consumer-repo/.github/workflows/agents-auto-pilot.yml | Adds retry/verification for capability-check dispatch and re-dispatch in branch-check loop |
| .github/workflows/reusable-codex-run.yml | Improves watchdog flag file handling to prevent race conditions during timeout |
| .github/workflows/reusable-claude-run.yml | Adds LLM task analysis, completion checkpoint comments, and enhanced error reporting |
| .github/actions/setup-api-client/action.yml | Smarter peer dependency conflict detection using error pattern matching |
| templates/consumer-repo/.github/actions/setup-api-client/action.yml | Same peer dependency handling improvement (synced) |
| .github/scripts/agents_pr_meta_keepalive.js | Removes "task" from skip list to allow "Task #N" issue references |
| templates/consumer-repo/.github/scripts/agents_pr_meta_keepalive.js | Same change (synced) |
| .github/scripts/tests/agents-pr-meta-keepalive.test.js | Adds test verifying "Task #N" is treated as valid issue reference |
| .github/workflows/maint-69-sync-labels.yml | Changes npm install from global to local installation |
templates/consumer-repo/.github/workflows/agents-auto-pilot.yml
Outdated
Show resolved
Hide resolved
templates/consumer-repo/.github/workflows/agents-auto-pilot.yml
Outdated
Show resolved
Hide resolved
templates/consumer-repo/.github/workflows/agents-auto-pilot.yml
Outdated
Show resolved
Hide resolved
templates/consumer-repo/.github/workflows/agents-auto-pilot.yml
Outdated
Show resolved
Hide resolved
After a progress-review action, rounds_without_task_completion was never reset. The next keepalive trigger would re-evaluate, find the counter still above threshold, enter review again, increment the counter, and repeat — permanently trapping the loop in review mode with no agent work ever running again. This affected all 4 agent PRs (#266, #267, #268, #269) which each stalled at progress-review-N with uncompleted tasks. Fix: 1. keepalive_loop.js summary function: reset rounds_without_task_completion to 0 after a review action, so the next evaluate triggers a run instead of another review. The review already provided course-correction feedback — the agent needs a chance to act on it. 2. agents-keepalive-loop.yml: add progress-review as a dependency of the summary job so the state update waits for the review to complete before persisting the reset counter. https://claude.ai/code/session_01JhCWWDJG8PqwaSbVPCGfm6
stranske
temporarily deployed
to
agent-high-privilege
GitHub Actions
Inactive
Fixes from inline code review on PR #1665: 1. Scope dispatcher run verification to the current dispatch by filtering runs created after the dispatch timestamp (dispatchedAt). Previously an old successful run for a different issue would falsely satisfy the check, causing retries to stop early. 2. Verify all dispatch attempts including the final one. Previously the last attempt assumed success without checking, creating a false-positive path when the last run was also cancelled. 3. On verification errors (catch block), continue to the next retry attempt instead of optimistically breaking out of the loop. Transient API errors no longer mask failed dispatches. 4. Scope the branch-check re-dispatch to recent runs (last 30 minutes) instead of any active run. An unrelated dispatcher run for a different issue no longer suppresses re-dispatch. 5. Apply all auto-pilot changes to both .github/workflows/ and templates/consumer-repo/.github/workflows/ per sync conventions. 6. Use --no-save --no-package-lock for npm install in maint-69-sync-labels.yml per repo conventions. https://claude.ai/code/session_01JhCWWDJG8PqwaSbVPCGfm6
Keep --no-save --no-package-lock flags on npm install per repo conventions. https://claude.ai/code/session_01JhCWWDJG8PqwaSbVPCGfm6
stranske
temporarily deployed
to
agent-high-privilege
GitHub Actions
Inactive
This was referenced Feb 26, 2026
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
Context
Observed on
stranske/Counter_Risk#34: the auto-pilot dispatched the belt, but the GitHub Actions run was silently cancelled before receiving a runner (empty logs, nullcancel_actor). No retry existed, so the keepalive loop waited through 5 exponential-backoff cycles (~30 min total), then stalled withneeds-human.Changes
templates/consumer-repo/.github/workflows/agents-auto-pilot.yml:createWorkflowDispatchcall replaced with a 3-attempt loop. After each dispatch, waits 15s and checkslistWorkflowRunsfor aqueued/in_progressrun. Retries if the run was cancelled.Test plan
yaml.safe_load)node --checkin async wrapper)https://claude.ai/code/session_01JhCWWDJG8PqwaSbVPCGfm6