fix: prevent contentless follow-up issues from verifier#316
Merged
Conversation
The documentation incorrectly stated that pytest-xdist was missing and tests run sequentially. In fact: - pytest-xdist is already in pyproject.toml (version 3.8.0) - reusable-10-ci-python.yml installs pytest-xdist automatically - The workflow uses '-n auto --dist=loadgroup' when xdist is detected Updated the Performance section and Recommendations to reflect the actual implemented state.
1. Add helper functions to detect placeholder and garbage content: - isPlaceholderContent(): Detects "section missing from source issue" text - looksLikeSectionHeader(): Filters markdown headers like "## Related" - looksLikeReferenceLink(): Filters PR/Issue reference links 2. Update extractUncheckedItems() to use the new filters, removing: - Placeholder text from bot-generated PR templates - Markdown section headers incorrectly captured as criteria - PR/Issue reference links that aren't actual acceptance criteria 3. Add hasSubstantiveContent flag to formatFollowUpIssue() return: - true when there are real tasks/criteria/gaps to address - false when all content is placeholders or empty 4. Update both verifier workflows to skip issue creation when hasSubstantiveContent is false Fixes issues like #313 (contentless) and #306 (garbage content). Tested: - PR #304 scenario: Now correctly filters out "## Related" and "- PR #N" - PR #310 scenario: Now returns hasSubstantiveContent=false - All 39 existing tests pass
stranske
temporarily deployed
to
agent-high-privilege
GitHub Actions
Inactive
Contributor
Contributor
Automated Status SummaryHead SHA: b75ecde
Coverage Overview
Coverage Trend
Top Coverage Hotspots (lowest coverage)
Low Coverage Files (<50.0%)
Updated automatically; will refresh on subsequent CI/Docker completions. Keepalive checklistScope
Tasks
Acceptance criteria
|
Contributor
🤖 Keepalive Loop StatusPR #316 | Agent: Codex | Iteration 0/5 Current State
🔍 Failure Classification| Error type | infrastructure | |
Contributor
There was a problem hiding this comment.
Pull request overview
This PR prevents the verifier from creating contentless follow-up issues by introducing content filtering and a substantive content check. The changes address issues where placeholder text, markdown section headers, or PR reference links were mistakenly treated as actionable acceptance criteria or tasks.
Key changes:
- Added three helper functions to identify and filter out non-substantive content (placeholders, section headers, and PR/issue references)
- Introduced
hasSubstantiveContentflag in issue formatters to signal whether an issue warrants creation - Updated both verifier workflows to skip issue creation when
hasSubstantiveContentis false
Reviewed changes
Copilot reviewed 4 out of 4 changed files in this pull request and generated 6 comments.
| File | Description |
|---|---|
.github/scripts/verifier_issue_formatter.js |
Adds helper functions for content filtering, updates extractUncheckedItems() to filter placeholders/headers/references, and adds hasSubstantiveContent flag to both issue formatter functions |
.github/workflows/agents-verifier.yml |
Adds check to skip issue creation when hasSubstantiveContent is false and sets appropriate output flags |
.github/workflows/reusable-agents-verifier.yml |
Mirrors the same skip logic as agents-verifier.yml for consistent behavior across workflows |
docs/workflows/SystemEvaluation.md |
Updates documentation to reflect completed parallel testing implementation (unrelated to verifier changes) |
💡 Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.
Contributor
54 tasks
Co-authored-by: Copilot <175728472+Copilot@users.noreply.github.com>
Co-authored-by: Copilot <175728472+Copilot@users.noreply.github.com>
37 tasks
Contributor
stranske
temporarily deployed
to
agent-high-privilege
GitHub Actions
Inactive
Contributor
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Automated Status Summary
Scope
Tasks
Acceptance criteria
Head SHA: c5f36b5
Latest Runs: ❔ in progress — Gate
Required: gate: ❔ in progress