chore: sync workflow templates - #130
Conversation
Automated sync from stranske/Workflows Template hash: 28e1781f3ab5 Changes synced from sync-manifest.yml
stranske
added
sync
GitHub Actions
Inactive
GitHub Actions
Inactive
🤖 Keepalive Loop StatusPR #130 | Agent: Codex | Iteration 0/5 Current State
🔍 Failure Classification| Error type | infrastructure | |
|
Status | ✅ no new diagnostics |
|
Autofix updated these files:
|
There was a problem hiding this comment.
Pull request overview
This PR syncs workflow templates from the central Workflows repository, introducing a significantly enhanced version of the agents-verify-to-issue.yml workflow that extracts comprehensive data from PR verification feedback to create well-structured follow-up issues.
Key changes:
- Enhanced data extraction including provider comparison reports, keepalive state analysis, and structural problem detection
- Intelligent filtering of non-actionable tasks (code snippets, type hints) from actual work items
- Multi-provider concern consolidation with deduplication logic
- Structured issue generation with agents:formatted label for immediate agent readiness
| sources: [provider.name], | ||
| priority: provider.concerns.indexOf(concern) === 0 ? 'high' : 'normal' | ||
| }); | ||
| } else { | ||
| allConcerns.get(key).sources.push(provider.name); | ||
| allConcerns.get(key).priority = 'high'; // Multiple providers = high priority |
There was a problem hiding this comment.
The priority assignment logic sets priority to 'high' when multiple providers report the same concern, but the 'normal' priority initially assigned based on position never gets used in the output. This priority field is calculated but never referenced in the issue body generation, making it dead code.
| sources: [provider.name], | |
| priority: provider.concerns.indexOf(concern) === 0 ? 'high' : 'normal' | |
| }); | |
| } else { | |
| allConcerns.get(key).sources.push(provider.name); | |
| allConcerns.get(key).priority = 'high'; // Multiple providers = high priority | |
| sources: [provider.name] | |
| }); | |
| } else { | |
| allConcerns.get(key).sources.push(provider.name); |
| const task = m[1].trim(); | ||
| // Filter out code snippets that aren't real tasks | ||
| if (!task.match(/^```|^class |^def |^\w+:\s*\w+\s*$|^#|^\s*$/) && | ||
| task.length > 10 && task.length < 200) { |
There was a problem hiding this comment.
The task length filters (task.length > 10 and task.length < 150 on line 168, then task.length > 10 and task.length < 200 on line 257) use different thresholds without explanation. This inconsistency could lead to confusion. Consider using consistent thresholds or documenting why different limits are needed for different contexts.
| task.length > 10 && task.length < 200) { | |
| task.length > 10 && task.length < 150) { |
| const issueBody = sections.join('\n'); | ||
|
|
||
| // Create the issue with agents:formatted label so it's ready for agent work | ||
| const { data: issue } = await github.rest.issues.create({ |
There was a problem hiding this comment.
The variable declaration uses object destructuring with const { data: issue } but the result is used as issue.data.number on line 528. This will fail because issue refers to the data property itself, not the parent object. Either remove the destructuring and use const issue = await ... then access issue.data.number, or keep the destructuring and use issue.number on line 528.
| labels: ['follow-up', 'agents:formatted'] | ||
| }); | ||
|
|
||
| core.info('Created issue #' + issue.data.number); |
There was a problem hiding this comment.
Following up from the previous issue, this line references issue.data.number but issue was destructured from the response as data: issue, so it should be issue.number instead.
| /#### (\w+(?:-\w+)?)\n/g | ||
| )]; | ||
|
|
||
| // For each provider header found, extract their details | ||
| for (const headerMatch of providerMatches) { | ||
| const providerName = headerMatch[1]; | ||
| // Build pattern to extract this provider's section | ||
| const detailPattern = '#### ' + providerName + | ||
| '\\n- \\*\\*Model:\\*\\* ([^\\n]+)' + | ||
| '\\n- \\*\\*Verdict:\\*\\* (\\w+)' + | ||
| '\\n- \\*\\*Confidence:\\*\\* (\\d+)%'; |
There was a problem hiding this comment.
The regex pattern for extracting provider details is fragile. The pattern expects exact whitespace formatting (\n- \*\*) which will break if GitHub's comment rendering or any formatter changes whitespace. Consider using more flexible patterns like \s*-\s*\*\* that allow for variations in whitespace.
| /#### (\w+(?:-\w+)?)\n/g | |
| )]; | |
| // For each provider header found, extract their details | |
| for (const headerMatch of providerMatches) { | |
| const providerName = headerMatch[1]; | |
| // Build pattern to extract this provider's section | |
| const detailPattern = '#### ' + providerName + | |
| '\\n- \\*\\*Model:\\*\\* ([^\\n]+)' + | |
| '\\n- \\*\\*Verdict:\\*\\* (\\w+)' + | |
| '\\n- \\*\\*Confidence:\\*\\* (\\d+)%'; | |
| /^####\s+(\w+(?:-\w+)?)/gm | |
| )]; | |
| // For each provider header found, extract their details | |
| for (const headerMatch of providerMatches) { | |
| const providerName = headerMatch[1]; | |
| // Build pattern to extract this provider's section, allowing flexible whitespace | |
| const detailPattern = '####\\s+' + providerName + | |
| '\\s*-\\s*\\*\\*Model:\\*\\*\\s*([^\\n]+)' + | |
| '\\s*-\\s*\\*\\*Verdict:\\*\\*\\s*(\\w+)' + | |
| '\\s*-\\s*\\*\\*Confidence:\\*\\*\\s*(\\d+)%'; |
| const taskMatches = [...prBody.matchAll(/^- \[ \]\s+(.+)$/gm)]; | ||
| for (const m of taskMatches) { | ||
| const task = m[1].trim(); | ||
| // Filter out code snippets that aren't real tasks | ||
| if (!task.match(/^```|^class |^def |^\w+:\s*\w+\s*$|^#|^\s*$/) && | ||
| task.length > 10 && task.length < 200) { |
There was a problem hiding this comment.
The regex patterns on lines 256 and 252 for filtering unchecked tasks are inconsistent. Line 252 uses ^- \[ \] (with spaces inside brackets) while line 256 uses different patterns. Additionally, the pattern on line 256 checking for ^\w+:\s*\w+\s*$ may be too broad and could filter out legitimate tasks like "Update: fix the config". Consider consolidating these patterns into the same isNonActionable function used earlier for consistency.
| const taskMatches = [...prBody.matchAll(/^- \[ \]\s+(.+)$/gm)]; | |
| for (const m of taskMatches) { | |
| const task = m[1].trim(); | |
| // Filter out code snippets that aren't real tasks | |
| if (!task.match(/^```|^class |^def |^\w+:\s*\w+\s*$|^#|^\s*$/) && | |
| task.length > 10 && task.length < 200) { | |
| const taskMatches = [...prBody.matchAll(/^- \[\s*\]\s+(.+)$/gm)]; | |
| const isNonActionableTask = (task) => { | |
| // Filter out code snippets, signatures, comments, and empty lines | |
| if (task.match(/^```|^class |^def |^#|^\s*$/)) { | |
| return true; | |
| } | |
| // Ignore tasks that are implausibly short or long | |
| if (task.length <= 10 || task.length >= 200) { | |
| return true; | |
| } | |
| return false; | |
| }; | |
| for (const m of taskMatches) { | |
| const task = m[1].trim(); | |
| if (!isNonActionableTask(task)) { |
| let linkedIssueNumber = null; | ||
| const issueMatch = prBody.match(/Issue #(\d+)|meta:issue:(\d+)/); | ||
| if (issueMatch) { | ||
| linkedIssueNumber = issueMatch[1] || issueMatch[2]; | ||
| } |
There was a problem hiding this comment.
The code extracts a linked issue number from the PR body but never uses it except to optionally display it in the output. If the linked issue reference is important for tracking the relationship between issues, consider linking the newly created issue back to the original one using GitHub's issue references or adding it to a project board.
| const verdictMatch = comment.match(/Verdict:\s*\*?\*?(\w+)\*?\*?/i); | ||
| const verdict = verdictMatch ? verdictMatch[1] : 'Unknown'; | ||
| // Check for high task count (often indicates code snippets parsed as tasks) | ||
| if (keepaliveData.totalTasks > 20) { |
There was a problem hiding this comment.
The magic number 20 for task count threshold lacks context. Consider extracting this to a constant at the beginning of the script with a descriptive name like MAX_REASONABLE_TASK_COUNT or documenting why 20 is the appropriate threshold for detecting structural problems.
| examples: nonActionableTasks.slice(0, 3).map(t => t.task) | ||
| }); | ||
| } | ||
| if (lowScores.length > 0) { | ||
| summary += '### Scores Below 7/10\n\n' + lowScores.join('\n') + '\n\n'; | ||
|
|
||
| // Check for unproductive iterations | ||
| if (keepaliveData.failureReason === 'max-iterations-unproductive') { | ||
| structuralProblems.push({ | ||
| problem: 'Agent hit max unproductive iterations', | ||
| cause: 'Tasks may have been unclear, blocked, or not ' + | ||
| 'achievable in isolation', | ||
| fix: 'Break complex tasks into smaller, independently ' + | ||
| 'verifiable steps' | ||
| }); | ||
| } | ||
| if (!summary) { | ||
| summary = 'No specific concerns extracted from verification report.'; | ||
| summary += '\n\nPlease review the original verification comment for details.'; | ||
|
|
||
| // Check for high unchecked ratio | ||
| const uncheckedRatio = keepaliveData.totalTasks > 0 | ||
| ? keepaliveData.uncheckedCount / keepaliveData.totalTasks : 0; | ||
| if (uncheckedRatio > 0.5) { | ||
| structuralProblems.push({ | ||
| problem: 'Over 50% of tasks remain unchecked after ' + | ||
| keepaliveData.iteration + ' iterations', | ||
| cause: 'Tasks may be too broad, ambiguous, or have ' + | ||
| 'hidden dependencies', | ||
| fix: 'Prioritize the 3-5 most critical tasks; defer ' + | ||
| 'others to follow-up issues' | ||
| }); | ||
| } | ||
|
|
||
| // Set outputs using environment file (handles multi-line content) | ||
| const fs = require('fs'); | ||
| // ======================================== | ||
| // 3. EXTRACT UNCHECKED TASKS FROM PR BODY | ||
| // ======================================== | ||
| const uncheckedTasks = []; | ||
| const taskMatches = [...prBody.matchAll(/^- \[ \]\s+(.+)$/gm)]; | ||
| for (const m of taskMatches) { | ||
| const task = m[1].trim(); | ||
| // Filter out code snippets that aren't real tasks | ||
| if (!task.match(/^```|^class |^def |^\w+:\s*\w+\s*$|^#|^\s*$/) && | ||
| task.length > 10 && task.length < 200) { | ||
| uncheckedTasks.push(task); | ||
| } | ||
| } | ||
|
|
||
| // ======================================== | ||
| // 4. EXTRACT LINKED ISSUE INFO | ||
| // ======================================== | ||
| let linkedIssueNumber = null; | ||
| const issueMatch = prBody.match(/Issue #(\d+)|meta:issue:(\d+)/); | ||
| if (issueMatch) { | ||
| linkedIssueNumber = issueMatch[1] || issueMatch[2]; | ||
| } | ||
|
|
||
| // ======================================== | ||
| // 5. BUILD CONSOLIDATED CONCERNS LIST | ||
| // ======================================== | ||
| const allConcerns = new Map(); // Use map to dedupe | ||
|
|
||
| // Add concerns from all providers | ||
| for (const provider of providerData.providers) { | ||
| for (const concern of provider.concerns) { | ||
| // Normalize for deduplication | ||
| const key = concern.toLowerCase().replace(/[^a-z0-9]/g, '').substring(0, 50); | ||
| if (!allConcerns.has(key)) { | ||
| allConcerns.set(key, { | ||
| text: concern, | ||
| sources: [provider.name], | ||
| priority: provider.concerns.indexOf(concern) === 0 ? 'high' : 'normal' | ||
| }); | ||
| } else { | ||
| allConcerns.get(key).sources.push(provider.name); | ||
| allConcerns.get(key).priority = 'high'; // Multiple providers = high priority | ||
| } | ||
| } | ||
| } | ||
|
|
||
| // ======================================== | ||
| // OUTPUT RESULTS | ||
| // ======================================== | ||
| const envFile = process.env.GITHUB_OUTPUT; | ||
| const writeMultiline = (name, value) => { | ||
| const delim = 'EOF_' + Math.random().toString(36).substring(2); | ||
| fs.appendFileSync(envFile, name + '<<' + delim + '\n' + value + '\n' + delim + '\n'); | ||
| }; | ||
|
|
||
| // Provider summary | ||
| const providerSummary = providerData.providers.map(p => | ||
| '- **' + p.name + '** (' + p.model + '): ' + | ||
| p.verdict + ' @ ' + p.confidence + '%' | ||
| ).join('\n') || 'No provider comparison found.'; | ||
| writeMultiline('provider_summary', providerSummary); | ||
|
|
||
| // Consolidated concerns as tasks | ||
| const concernTasks = [...allConcerns.values()] | ||
| .sort((a, b) => (b.sources.length - a.sources.length)) | ||
| .map(c => { | ||
| const sourceTag = c.sources.length > 1 | ||
| ? ' *(agreed by both providers)*' : ''; | ||
| return '- [ ] ' + c.text + sourceTag; | ||
| }) | ||
| .join('\n'); | ||
| writeMultiline('concern_tasks', | ||
| concernTasks || 'No specific concerns identified.'); | ||
|
|
||
| // Unchecked tasks from PR | ||
| const uncheckedTasksStr = uncheckedTasks | ||
| .map(t => '- [ ] ' + t).join('\n'); | ||
| writeMultiline('unchecked_tasks', | ||
| uncheckedTasksStr || 'All tasks appear complete.'); | ||
|
|
||
| // Keepalive analysis with structural problem detection | ||
| let keepaliveAnalysis = ''; | ||
| if (keepaliveData.iteration > 0) { | ||
| keepaliveAnalysis = '**Agent ran ' + keepaliveData.iteration + | ||
| ' iterations**\n'; | ||
| if (keepaliveData.failureReason) { | ||
| keepaliveAnalysis += '- Stop reason: `' + | ||
| keepaliveData.failureReason + '`\n'; | ||
| } | ||
| if (keepaliveData.uncheckedCount > 0) { | ||
| keepaliveAnalysis += '- Remaining unchecked items: ' + | ||
| keepaliveData.uncheckedCount + ' of ' + | ||
| keepaliveData.totalTasks + '\n'; | ||
| } | ||
|
|
||
| // Show non-actionable tasks the agent tried | ||
| const nonActionable = keepaliveData.attemptedTasks | ||
| .filter(t => t.isNonActionable); | ||
| if (nonActionable.length > 0) { | ||
| keepaliveAnalysis += | ||
| '\n**⚠️ Non-actionable items agent attempted:**\n'; | ||
| for (const t of nonActionable.slice(0, 5)) { | ||
| keepaliveAnalysis += | ||
| '- `' + t.task + '` (iteration ' + t.iteration + ')\n'; | ||
| } |
There was a problem hiding this comment.
The hard-coded limits like slice(0, 3) and slice(0, 5) should be extracted as constants. These magic numbers control how many examples are shown in the issue but are scattered throughout the code, making it difficult to adjust the verbosity level consistently.
| for (const provider of providerData.providers) { | ||
| for (const concern of provider.concerns) { | ||
| // Normalize for deduplication | ||
| const key = concern.toLowerCase().replace(/[^a-z0-9]/g, '').substring(0, 50); |
There was a problem hiding this comment.
The concern deduplication logic uses only the first 50 characters of a normalized string as the key. This could lead to false positives where different concerns with similar beginnings are incorrectly merged. Consider using the full normalized string or a hash of it for more accurate deduplication.
| const key = concern.toLowerCase().replace(/[^a-z0-9]/g, '').substring(0, 50); | |
| const key = concern.toLowerCase().replace(/[^a-z0-9]/g, ''); |
Sync Summary
Files Updated
Files Skipped
Review Checklist
Source: stranske/Workflows
Manifest:
.github/sync-manifest.yml