fix: improve verifier follow-up issue format by stranske · Pull Request #310 · stranske/Workflows

stranske · 2025-12-30T06:59:44Z

Automated Status Summary

Scope

Scope section missing from source issue.

Tasks

Tasks section missing from source issue.

Acceptance criteria

Acceptance criteria section missing from source issue.

Head SHA: 9334b84
Latest Runs: ❔ in progress — Gate
Required: gate: ❔ in progress

Workflow / Job	Result	Logs
Agents PR meta manager	❔ in progress	View run
CI Autofix Loop	✅ success	View run
Copilot code review	❔ in progress	View run
Gate	❔ in progress	View run
Health 40 Sweep	✅ success	View run
Health 44 Gate Branch Protection	✅ success	View run
Health 45 Agents Guard	✅ success	View run
Health 50 Security Scan	❔ in progress	View run
Maint 52 Validate Workflows	✅ success	View run
PR 11 - Minimal invariant CI	✅ success	View run
Selftest CI	✅ success	View run
Validate Sync Manifest	✅ success	View run

1. Remove confusing 'Satisfy:' prefix from tasks - it's redundant since the task list already implies items need to be completed 2. Filter out markdown section headers (## Related, etc.) that were incorrectly being captured as acceptance criteria 3. Filter out PR/Issue references that look like list items but aren't actual acceptance criteria This makes follow-up issues clearer and removes noise from the task list.

agents-workflows-bot · 2025-12-30T07:00:38Z

⚠️ Action Required: Unable to determine source issue for PR #310. The PR title, branch name, or body must contain the issue number (e.g. #123, branch: issue-123, or the hidden marker ).

github-actions · 2025-12-30T07:01:16Z

Automated Status Summary

Head SHA: b8957a8
Latest Runs: ⏳ pending — Gate
Required contexts: Gate / gate, Health 45 Agents Guard / Enforce agents workflow protections
Required: core tests (3.11): ⏳ pending, core tests (3.12): ⏳ pending, docker smoke: ⏳ pending, gate: ⏳ pending

Workflow / Job	Result	Logs
(no jobs reported)	⏳ pending	—

Coverage Overview

Coverage history entries: 1

Coverage Trend

Metric	Value
Current	0.00%
Baseline	85.00%
Delta	-85.00%
Minimum	70.00%
Status	❌ Below minimum

Top Coverage Hotspots (lowest coverage)

File	Coverage	Missing
`src/trend_analysis/_autofix_trigger_sample.py`	0.0%	10
`src/trend_analysis/_autofix_violation_case2.py`	0.0%	15
`src/trend_analysis/_autofix_violation_case3.py`	0.0%	13
`src/trend_analysis/_ci_probe_faults.py`	0.0%	18
`src/trend_analysis/automation_multifailure.py`	0.0%	4
`src/trend_analysis/constants.py`	0.0%	1
`src/trend_analysis/selector.py`	0.0%	12
`src/trend_analysis/weighting.py`	0.0%	11
`src/trend_analysis/__init__.py`	100.0%	0

Low Coverage Files (<50.0%)

File	Coverage	Missing
`src/trend_analysis/_autofix_trigger_sample.py`	0.0%	10
`src/trend_analysis/_autofix_violation_case2.py`	0.0%	15
`src/trend_analysis/_autofix_violation_case3.py`	0.0%	13
`src/trend_analysis/_ci_probe_faults.py`	0.0%	18
`src/trend_analysis/automation_multifailure.py`	0.0%	4
`src/trend_analysis/constants.py`	0.0%	1
`src/trend_analysis/selector.py`	0.0%	12
`src/trend_analysis/weighting.py`	0.0%	11

Updated automatically; will refresh on subsequent CI/Docker completions.

Keepalive checklist

Scope

No scope information available

Tasks

No tasks defined

Acceptance criteria

No acceptance criteria defined

github-actions · 2025-12-30T07:01:38Z

🤖 Keepalive Loop Status

PR #310 | Agent: Codex | Iteration 0/5

Current State

Metric	Value
Iteration progress	[----------] 0/5
Action	wait (missing-agent-label)
Gate	success
Tasks	0/2 complete
Keepalive	❌ disabled
Autofix	❌ disabled

🔍 Failure Classification

Copilot

Pull request overview

This PR improves the format of verifier follow-up issues by removing redundant prefixes and filtering out incorrectly captured markdown content from task lists, addressing user feedback about confusing issue #306 formatting.

Key Changes:

Removes "Satisfy:" prefix from task items generated from unmet criteria
Adds filtering to prevent markdown section headers from being captured as acceptance criteria
Adds filtering to prevent PR/Issue reference lines from being captured as task items

💡 Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.

.github/scripts/verifier_issue_formatter.js

Copilot · 2025-12-30T07:02:15Z

.github/scripts/verifier_issue_formatter.js

      newTasks = findings.gaps.map((gap) => `Address: ${gap}`);
    } else if (findings.unmetCriteria.length > 0) {
-      // Use verifier's specific unmet criteria
-      newTasks = findings.unmetCriteria.map((criterion) => `Satisfy: ${criterion}`);
+      // Use verifier's specific unmet criteria (no prefix - it's clear these need work)
+      newTasks = findings.unmetCriteria;
    } else {
      // Fall back to creating tasks from unmet acceptance criteria
-      newTasks = refinedUnmetCriteria.map((criterion) => `Satisfy: ${criterion}`);
+      newTasks = refinedUnmetCriteria;


There's an inconsistency in how tasks are generated. When gaps are identified (line 362), tasks are prefixed with "Address:", but when unmet criteria or refined unmet criteria are used (lines 365, 368), no prefix is added. This creates inconsistent task formatting in the follow-up issue.

For consistency, consider either:

Removing the "Address:" prefix on line 362 to match the other cases (recommended based on the PR's rationale)

Adding appropriate prefixes to lines 365 and 368 to match line 362

The first option aligns with the PR description's goal of removing redundant prefixes since task list items already imply they need work.

Copilot · 2025-12-30T07:02:15Z

.github/scripts/verifier_issue_formatter.js

+      // Skip items that look like markdown links to sections (e.g., "- PR #123")
+      if (/^[-–]\s*(PR|Issue)\s*#\d+/i.test(text)) {


The regex pattern is checking for text that starts with a hyphen/dash followed by "PR" or "Issue", but at this point text has already been extracted from the checkbox content (via match[1].trim()). The leading - or * was consumed by the checkbox pattern on line 184, so text will never start with - or –.

For example, if the input line is - [ ] - PR #296 - Description, then match[1] captures - PR #296 - Description, which will correctly match this pattern. However, if the input is - [ ] PR #296 - Description (without a leading dash after the checkbox), it won't match.

Consider simplifying the pattern to /^(PR|Issue)\s*#\d+/i to match text that starts directly with "PR" or "Issue" followed by a number. Alternatively, if you want to handle both cases, use /^[-–]?\s*(PR|Issue)\s*#\d+/i to make the leading dash optional.

Suggested change

// Skip items that look like markdown links to sections (e.g., "- PR #123")

if (/^[-–]\s*(PR|Issue)\s*#\d+/i.test(text)) {

// Skip items that look like markdown links to sections (e.g., "- PR #123" or "PR #123")

if (/^[-–]?\s*(PR|Issue)\s*#\d+/i.test(text)) {

Copilot · 2025-12-30T07:02:15Z

.github/scripts/verifier_issue_formatter.js

-      items.push(match[1].trim());
+      const text = match[1].trim();
+      // Skip markdown section headers that were incorrectly captured as criteria
+      if (text.startsWith('#') || text.startsWith('##')) {


The check for startsWith('##') is redundant since any string starting with '##' will also start with '#'. The first condition will always catch both cases. Consider removing the || text.startsWith('##') portion to simplify the logic without changing behavior.

Suggested change

if (text.startsWith('#') || text.startsWith('##')) {

if (text.startsWith('#')) {

stranske · 2025-12-30T07:17:24Z

@codex fix comments

chatgpt-codex-connector · 2025-12-30T07:20:05Z

Summary

Simplified unchecked-item filtering to skip markdown headers and optional-dash PR/Issue references before adding tasks. .github/scripts/verifier_issue_formatter.jsL182-L195
Removed the redundant “Address:” prefix when generating tasks from gap findings to align formatting with other task sources. .github/scripts/verifier_issue_formatter.jsL350-L369
Added test coverage to ensure markdown headers and PR/Issue references are filtered while normal tasks remain captured. .github/scripts/tests/verifier-issue-formatter.test.jsL142-L182

Testing

✅ node --test .github/scripts/__tests__/verifier-issue-formatter.test.js

View task →

1. Add helper functions to detect placeholder and garbage content: - isPlaceholderContent(): Detects "section missing from source issue" text - looksLikeSectionHeader(): Filters markdown headers like "## Related" - looksLikeReferenceLink(): Filters PR/Issue reference links 2. Update extractUncheckedItems() to use the new filters, removing: - Placeholder text from bot-generated PR templates - Markdown section headers incorrectly captured as criteria - PR/Issue reference links that aren't actual acceptance criteria 3. Add hasSubstantiveContent flag to formatFollowUpIssue() return: - true when there are real tasks/criteria/gaps to address - false when all content is placeholders or empty 4. Update both verifier workflows to skip issue creation when hasSubstantiveContent is false Fixes issues like #313 (contentless) and #306 (garbage content). Tested: - PR #304 scenario: Now correctly filters out "## Related" and "- PR #N" - PR #310 scenario: Now returns hasSubstantiveContent=false - All 39 existing tests pass

* docs: update SystemEvaluation.md - parallel testing is implemented The documentation incorrectly stated that pytest-xdist was missing and tests run sequentially. In fact: - pytest-xdist is already in pyproject.toml (version 3.8.0) - reusable-10-ci-python.yml installs pytest-xdist automatically - The workflow uses '-n auto --dist=loadgroup' when xdist is detected Updated the Performance section and Recommendations to reflect the actual implemented state. * fix: prevent contentless follow-up issues from verifier 1. Add helper functions to detect placeholder and garbage content: - isPlaceholderContent(): Detects "section missing from source issue" text - looksLikeSectionHeader(): Filters markdown headers like "## Related" - looksLikeReferenceLink(): Filters PR/Issue reference links 2. Update extractUncheckedItems() to use the new filters, removing: - Placeholder text from bot-generated PR templates - Markdown section headers incorrectly captured as criteria - PR/Issue reference links that aren't actual acceptance criteria 3. Add hasSubstantiveContent flag to formatFollowUpIssue() return: - true when there are real tasks/criteria/gaps to address - false when all content is placeholders or empty 4. Update both verifier workflows to skip issue creation when hasSubstantiveContent is false Fixes issues like #313 (contentless) and #306 (garbage content). Tested: - PR #304 scenario: Now correctly filters out "## Related" and "- PR #N" - PR #310 scenario: Now returns hasSubstantiveContent=false - All 39 existing tests pass * Update .github/scripts/verifier_issue_formatter.js Co-authored-by: Copilot <175728472+Copilot@users.noreply.github.com> * Update .github/scripts/verifier_issue_formatter.js Co-authored-by: Copilot <175728472+Copilot@users.noreply.github.com> --------- Co-authored-by: Copilot <175728472+Copilot@users.noreply.github.com>

* docs: update SystemEvaluation.md - parallel testing is implemented The documentation incorrectly stated that pytest-xdist was missing and tests run sequentially. In fact: - pytest-xdist is already in pyproject.toml (version 3.8.0) - reusable-10-ci-python.yml installs pytest-xdist automatically - The workflow uses '-n auto --dist=loadgroup' when xdist is detected Updated the Performance section and Recommendations to reflect the actual implemented state. * fix: prevent contentless follow-up issues from verifier 1. Add helper functions to detect placeholder and garbage content: - isPlaceholderContent(): Detects "section missing from source issue" text - looksLikeSectionHeader(): Filters markdown headers like "## Related" - looksLikeReferenceLink(): Filters PR/Issue reference links 2. Update extractUncheckedItems() to use the new filters, removing: - Placeholder text from bot-generated PR templates - Markdown section headers incorrectly captured as criteria - PR/Issue reference links that aren't actual acceptance criteria 3. Add hasSubstantiveContent flag to formatFollowUpIssue() return: - true when there are real tasks/criteria/gaps to address - false when all content is placeholders or empty 4. Update both verifier workflows to skip issue creation when hasSubstantiveContent is false Fixes issues like #313 (contentless) and #306 (garbage content). Tested: - PR #304 scenario: Now correctly filters out "## Related" and "- PR #N" - PR #310 scenario: Now returns hasSubstantiveContent=false - All 39 existing tests pass * Update .github/scripts/verifier_issue_formatter.js Co-authored-by: Copilot <175728472+Copilot@users.noreply.github.com> * Update .github/scripts/verifier_issue_formatter.js Co-authored-by: Copilot <175728472+Copilot@users.noreply.github.com> * Initial plan * Fix misleading comment in formatSimpleFollowUpIssue The comment claimed "Simple format always has substantive content" but the code correctly checks for it. Updated comment to accurately reflect that we verify content rather than assume it exists. Co-authored-by: stranske <23046322+stranske@users.noreply.github.com> --------- Co-authored-by: stranske <stranske@gmail.com> Co-authored-by: Copilot <175728472+Copilot@users.noreply.github.com> Co-authored-by: copilot-swe-agent[bot] <198982749+Copilot@users.noreply.github.com> Co-authored-by: stranske <23046322+stranske@users.noreply.github.com>

Copilot AI review requested due to automatic review settings December 30, 2025 06:59

stranske temporarily deployed to agent-standard December 30, 2025 06:59 — with GitHub Actions Inactive

Copilot started reviewing on behalf of stranske December 30, 2025 07:00 View session

Copilot AI reviewed Dec 30, 2025

View reviewed changes

stranske merged commit 46938e1 into main Dec 30, 2025
71 checks passed

stranske deleted the fix/verifier-issue-format branch December 30, 2025 07:20

github-actions bot mentioned this pull request Dec 30, 2025

[Follow-up] Unmet criteria from PR #310 #313

Closed

3 tasks

stranske mentioned this pull request Dec 30, 2025

fix: prevent contentless follow-up issues from verifier #316

Merged

20 tasks

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

fix: improve verifier follow-up issue format#310

fix: improve verifier follow-up issue format#310
stranske merged 1 commit intomainfrom
fix/verifier-issue-format

stranske commented Dec 30, 2025 •

edited by agents-workflows-bot bot

Loading

Uh oh!

agents-workflows-bot bot commented Dec 30, 2025

Uh oh!

github-actions bot commented Dec 30, 2025

Uh oh!

github-actions bot commented Dec 30, 2025

Uh oh!

Copilot AI left a comment

Uh oh!

Uh oh!

Copilot AI Dec 30, 2025

Uh oh!

Copilot AI Dec 30, 2025

Uh oh!

Copilot AI Dec 30, 2025

Uh oh!

stranske commented Dec 30, 2025

Uh oh!

chatgpt-codex-connector bot commented Dec 30, 2025

Uh oh!

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

2 participants

		// Skip items that look like markdown links to sections (e.g., "- PR #123")
		if (/^[-–]\s(PR\|Issue)\s#\d+/i.test(text)) {

	if (text.startsWith('#') \|\| text.startsWith('##')) {
	if (text.startsWith('#')) {

Conversation

stranske commented Dec 30, 2025 • edited by agents-workflows-bot bot Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Automated Status Summary

Scope

Tasks

Acceptance criteria

Uh oh!

agents-workflows-bot bot commented Dec 30, 2025

Uh oh!

github-actions bot commented Dec 30, 2025

Automated Status Summary

Coverage Overview

Coverage Trend

Top Coverage Hotspots (lowest coverage)

Low Coverage Files (<50.0%)

Keepalive checklist

Scope

Tasks

Acceptance criteria

Uh oh!

github-actions bot commented Dec 30, 2025

🤖 Keepalive Loop Status

Current State

🔍 Failure Classification

Uh oh!

Copilot AI left a comment

Choose a reason for hiding this comment

Pull request overview

Uh oh!

Uh oh!

Copilot AI Dec 30, 2025

Choose a reason for hiding this comment

Uh oh!

Copilot AI Dec 30, 2025

Choose a reason for hiding this comment

Uh oh!

Copilot AI Dec 30, 2025

Choose a reason for hiding this comment

Uh oh!

stranske commented Dec 30, 2025

Uh oh!

chatgpt-codex-connector bot commented Dec 30, 2025

Uh oh!

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

2 participants

stranske commented Dec 30, 2025 •

edited by agents-workflows-bot bot

Loading