fix: skip verifier when no acceptance criteria exist#193
Merged
Conversation
The verifier was running and potentially creating follow-up issues even when there were no acceptance criteria to verify. Now it skips with a notice when acceptanceCount == 0, avoiding unnecessary issue creation.
Contributor
stranske
temporarily deployed
to
agent-standard
GitHub Actions
Inactive
Contributor
Automated Status SummaryHead SHA: 4a787cf
Coverage Overview
Coverage Trend
Updated automatically; will refresh on subsequent CI/Docker completions. Keepalive checklistScopeNo scope information available Tasks
Acceptance criteria
|
Contributor
🤖 Keepalive Loop StatusPR #193 | Agent: Codex | Iteration 0/5 Current State
🔍 Failure Classification| Error type | infrastructure | |
Contributor
There was a problem hiding this comment.
Pull request overview
This PR adds logic to skip the verifier when no acceptance criteria are found in the PR or linked issues, preventing unnecessary issue creation for PRs without criteria to verify.
Key Changes:
- Added early return logic to skip verifier execution when
acceptanceCount === 0 - Sets appropriate outputs and skip reason when no acceptance criteria exist
- Returns skip context with
shouldRun: falseto prevent verifier from running
💡 Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.
![chatgpt-codex-connector[bot]](https://avatars.githubusercontent.com/in/1144995?s=60&v=4){kind=small}
There was a problem hiding this comment.
💡 Codex Review
Here are some automated review suggestions for this pull request.
ℹ️ About Codex in GitHub
Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you
- Open a pull request for review
- Mark a draft as ready
- Comment "@codex review".
If Codex has suggestions, it will comment; otherwise it will react with 👍.
Codex can also answer questions or update the PR. Try commenting "@codex address that feedback".
- Fix P1: Use hasAcceptanceContent to check for any acceptance content, not just checkboxes. This prevents skipping PRs with plain-text acceptance criteria that aren't formatted as checkboxes. - Optimize: Move skip check before file write to avoid unnecessary I/O when there's nothing to verify. - Consistency: Simplify return object to match other skip paths (shouldRun, reason, ciResults only).
stranske
temporarily deployed
to
agent-standard
GitHub Actions
Inactive
This was referenced Dec 26, 2025
stranske
added a commit
that referenced
this pull request
Jan 7, 2026
* Add Phase 3 integration plan with testing cycle for Manager-Database Phase 3: Pre-Agent Intelligence (4 capabilities) - 3A: Capability Check - supplements agents:optimize with feasibility gate - Runs BEFORE agent assignment on Issues (not after) - Adds needs-human label when agent cannot proceed - 3B: Task Decomposition - auto-split large issues - 3C: Duplicate Detection - comment-only mode, track false positives - 3D: Semantic Labeling - auto-suggest/apply labels Testing Plan: - Test repo: Manager-Database - ~11 test issues across 4 capabilities - False positive tracking for dedup (target: <5%) - Metrics dashboard for validation Also updates: - Mark Collab-Admin PR #113 as merged (7/7 repos now synced) - All immediate tasks completed - Phase 3 ready to begin * Add Phase 3 test issues for Manager-Database Created 3 test issues: - #193: Stripe integration (should FAIL capability check) - #194: Health monitoring (should trigger task decomposition) - #196: Manager list API (should detect as duplicate of #133) Updated testing metrics dashboard to track progress. * Add Phase 4: Full Automation & Cleanup plan Phase 4 includes 5 initiatives: - 4A: Label Cleanup - Remove bloat labels, standardize across 7 repos - 4B: User Guide - Operational documentation for label system (sync to consumers) - 4C: Auto-Pilot Label - End-to-end issue-to-merged-PR automation - 4D: Conflict Resolution - Automated merge conflict handling in keepalive - 4E: Verify-to-Issue - Create follow-up issues from verification feedback Key decisions: - Auto-pilot uses workflow_dispatch between steps (not chained labels) - Conflict detection added to keepalive loop (not separate workflow) - Verify-to-issue is user-triggered (not automatic, avoids false positives) Also identifies 7 additional automation opportunities for future phases. Testing plan defined for Manager-Database. * Correct label analysis after codebase search + expand Phase 5 Label Analysis Corrections: - agents:pause/paused ARE functional (keepalive_gate.js, keepalive-runner.js) - agents:activated IS functional (agents_pr_meta_keepalive.js) - from:codex/copilot ARE functional (merge_manager.js) - automerge IS functional (merge_manager.js, agents_belt_scan.js) - agents (bare) IS functional (agent_task.yml template) - risk:low, ci:green, codex-ready ARE functional (merge_manager.js, issue templates) Only 5-6 labels confirmed as bloat: - codex (bare) - redundant with agent:codex - ai:agent - zero matches - auto-merge-audit - zero matches - automerge:ok - zero matches - architecture, backend, cli, etc. - repo-specific, not synced Phase 5 Analysis: - 5A: Auto-labeling - label_matcher.py EXISTS, ready for workflow - 5B: Coverage check - maint-coverage-guard.yml EXISTS, add soft PR check - 5C: Stale PR cleanup - not needed - 5D: Dependabot - partial (auto-label exists, add auto-merge) - 5E: Issue lint - soft warning approach - 5F: Cross-repo linking - weekly scan with semantic_matcher.py - 5G: Metrics - hybrid LangSmith (LLM) + custom (workflow) * Consolidate agents:pause to agents:paused + expand Phase 4-5 plans Label consolidation: - Replace agents:pause with agents:paused in all source files - Update keepalive_gate.js PAUSE_LABEL constant - Update keepalive_orchestrator_gate_runner.js hardcoded check - Update test to use agents:paused - Update documentation in README, CLAUDE.md, GoalsAndPlumbing.md Phase 4 updates: - 4A: Add idiosyncratic repo bloat cleanup strategy (per-repo audit) - 4B: Add optional issue creation feature to user guide (deferred) - 4D: Full conflict resolution implementation with code examples - 4E: Complete verify-to-issue workflow implementation Phase 5 updates: - 5F: Marked as SKIPPED (not needed per user decision) - 5G: Full LangSmith integration plan + custom metrics All keepalive tests pass (8/8).
stranske
added a commit
that referenced
this pull request
Jan 7, 2026
* Add Phase 3 integration plan with testing cycle for Manager-Database Phase 3: Pre-Agent Intelligence (4 capabilities) - 3A: Capability Check - supplements agents:optimize with feasibility gate - Runs BEFORE agent assignment on Issues (not after) - Adds needs-human label when agent cannot proceed - 3B: Task Decomposition - auto-split large issues - 3C: Duplicate Detection - comment-only mode, track false positives - 3D: Semantic Labeling - auto-suggest/apply labels Testing Plan: - Test repo: Manager-Database - ~11 test issues across 4 capabilities - False positive tracking for dedup (target: <5%) - Metrics dashboard for validation Also updates: - Mark Collab-Admin PR #113 as merged (7/7 repos now synced) - All immediate tasks completed - Phase 3 ready to begin * Add Phase 3 test issues for Manager-Database Created 3 test issues: - #193: Stripe integration (should FAIL capability check) - #194: Health monitoring (should trigger task decomposition) - #196: Manager list API (should detect as duplicate of #133) Updated testing metrics dashboard to track progress. * Add Phase 4: Full Automation & Cleanup plan Phase 4 includes 5 initiatives: - 4A: Label Cleanup - Remove bloat labels, standardize across 7 repos - 4B: User Guide - Operational documentation for label system (sync to consumers) - 4C: Auto-Pilot Label - End-to-end issue-to-merged-PR automation - 4D: Conflict Resolution - Automated merge conflict handling in keepalive - 4E: Verify-to-Issue - Create follow-up issues from verification feedback Key decisions: - Auto-pilot uses workflow_dispatch between steps (not chained labels) - Conflict detection added to keepalive loop (not separate workflow) - Verify-to-issue is user-triggered (not automatic, avoids false positives) Also identifies 7 additional automation opportunities for future phases. Testing plan defined for Manager-Database. * Correct label analysis after codebase search + expand Phase 5 Label Analysis Corrections: - agents:pause/paused ARE functional (keepalive_gate.js, keepalive-runner.js) - agents:activated IS functional (agents_pr_meta_keepalive.js) - from:codex/copilot ARE functional (merge_manager.js) - automerge IS functional (merge_manager.js, agents_belt_scan.js) - agents (bare) IS functional (agent_task.yml template) - risk:low, ci:green, codex-ready ARE functional (merge_manager.js, issue templates) Only 5-6 labels confirmed as bloat: - codex (bare) - redundant with agent:codex - ai:agent - zero matches - auto-merge-audit - zero matches - automerge:ok - zero matches - architecture, backend, cli, etc. - repo-specific, not synced Phase 5 Analysis: - 5A: Auto-labeling - label_matcher.py EXISTS, ready for workflow - 5B: Coverage check - maint-coverage-guard.yml EXISTS, add soft PR check - 5C: Stale PR cleanup - not needed - 5D: Dependabot - partial (auto-label exists, add auto-merge) - 5E: Issue lint - soft warning approach - 5F: Cross-repo linking - weekly scan with semantic_matcher.py - 5G: Metrics - hybrid LangSmith (LLM) + custom (workflow) * Consolidate agents:pause to agents:paused + expand Phase 4-5 plans Label consolidation: - Replace agents:pause with agents:paused in all source files - Update keepalive_gate.js PAUSE_LABEL constant - Update keepalive_orchestrator_gate_runner.js hardcoded check - Update test to use agents:paused - Update documentation in README, CLAUDE.md, GoalsAndPlumbing.md Phase 4 updates: - 4A: Add idiosyncratic repo bloat cleanup strategy (per-repo audit) - 4B: Add optional issue creation feature to user guide (deferred) - 4D: Full conflict resolution implementation with code examples - 4E: Complete verify-to-issue workflow implementation Phase 5 updates: - 5F: Marked as SKIPPED (not needed per user decision) - 5G: Full LangSmith integration plan + custom metrics All keepalive tests pass (8/8). * feat: Implement Phase 4-5 automation features Phase 4 implementations: - 4A: Add scripts/cleanup_labels.py for label auditing - Classifies labels as functional/bloat/idiosyncratic - Requires --confirm flag for actual deletion - Reports audit results with recommendations - 4D: Add conflict detection for keepalive pipeline - .github/scripts/conflict_detector.js module - Detects conflicts from GitHub API, CI logs, PR comments - templates/consumer-repo/.github/codex/prompts/fix_merge_conflicts.md - 4E: Add agents-verify-to-issue.yml workflow - Creates follow-up issues from verification feedback - User-triggered via verify:create-issue label - Extracts concerns and low scores automatically Phase 5 implementations: - 5A: Add agents-auto-label.yml workflow - Semantic label matching for new issues - 90% threshold for auto-apply, 75% for suggestions - Uses existing label_matcher.py script - 5G: Add LangSmith tracing to tools/llm_provider.py - _setup_langsmith_tracing() function - Auto-configures when LANGSMITH_API_KEY present Also: - Update .github/sync-manifest.yml with new sync entries - Update docs/LABELS.md with new label documentation * fix: Address Copilot review comments on PR #650 Code quality improvements based on automated code review: 1. tools/llm_provider.py: - Fix LangSmith API key env var (LANGSMITH_API_KEY vs LANGCHAIN_API_KEY) - Improve f-string formatting for logging - Add usage comment for LANGSMITH_ENABLED constant 2. .github/scripts/conflict_detector.js: - Add debug logging in catch blocks instead of silent failures - Makes debugging easier when log downloads fail 3. .github/workflows/agents-verify-to-issue.yml: - Replace /tmp file usage with GitHub Actions environment files - Use heredoc delimiter for multi-line output - Consolidate find and extract steps for cleaner flow 4. .github/workflows/agents-auto-label.yml: - Make Workflows repo checkout configurable (not hardcoded) - Use github.paginate() for label retrieval (handles >100 labels) 5. templates/consumer-repo/.github/codex/prompts/fix_merge_conflicts.md: - Replace hardcoded 'main' with {{base_branch}} template variable - Make verification steps language-agnostic (not Python-specific) - Add note about checking project README for test commands * fix: Fix CI test failures 1. Fix test_integration_template_installs_and_tests - The test used --user pip install flag which fails in virtualenvs - Added _in_virtualenv() helper to detect virtualenv environment - Only use --user flag when NOT in a virtualenv 2. Add new workflows to expected names mapping - agents-auto-label.yml: 'Auto-Label Issues' - agents-verify-to-issue.yml: 'Create Issue from Verification' 3. Update workflow documentation - docs/ci/WORKFLOWS.md: Added bullet points for new workflows - docs/ci/WORKFLOW_SYSTEM.md: Added table rows for new workflows All 1120 tests now pass. * fix: Remove duplicate env key in agents-auto-label.yml actionlint was failing because the Match labels step had two env blocks. Merged ISSUE_TITLE and ISSUE_BODY into the main env block.
stranske
added a commit
that referenced
this pull request
Jan 8, 2026
…#653) * Add Phase 3 integration plan with testing cycle for Manager-Database Phase 3: Pre-Agent Intelligence (4 capabilities) - 3A: Capability Check - supplements agents:optimize with feasibility gate - Runs BEFORE agent assignment on Issues (not after) - Adds needs-human label when agent cannot proceed - 3B: Task Decomposition - auto-split large issues - 3C: Duplicate Detection - comment-only mode, track false positives - 3D: Semantic Labeling - auto-suggest/apply labels Testing Plan: - Test repo: Manager-Database - ~11 test issues across 4 capabilities - False positive tracking for dedup (target: <5%) - Metrics dashboard for validation Also updates: - Mark Collab-Admin PR #113 as merged (7/7 repos now synced) - All immediate tasks completed - Phase 3 ready to begin * Add Phase 3 test issues for Manager-Database Created 3 test issues: - #193: Stripe integration (should FAIL capability check) - #194: Health monitoring (should trigger task decomposition) - #196: Manager list API (should detect as duplicate of #133) Updated testing metrics dashboard to track progress. * Add Phase 4: Full Automation & Cleanup plan Phase 4 includes 5 initiatives: - 4A: Label Cleanup - Remove bloat labels, standardize across 7 repos - 4B: User Guide - Operational documentation for label system (sync to consumers) - 4C: Auto-Pilot Label - End-to-end issue-to-merged-PR automation - 4D: Conflict Resolution - Automated merge conflict handling in keepalive - 4E: Verify-to-Issue - Create follow-up issues from verification feedback Key decisions: - Auto-pilot uses workflow_dispatch between steps (not chained labels) - Conflict detection added to keepalive loop (not separate workflow) - Verify-to-issue is user-triggered (not automatic, avoids false positives) Also identifies 7 additional automation opportunities for future phases. Testing plan defined for Manager-Database. * Correct label analysis after codebase search + expand Phase 5 Label Analysis Corrections: - agents:pause/paused ARE functional (keepalive_gate.js, keepalive-runner.js) - agents:activated IS functional (agents_pr_meta_keepalive.js) - from:codex/copilot ARE functional (merge_manager.js) - automerge IS functional (merge_manager.js, agents_belt_scan.js) - agents (bare) IS functional (agent_task.yml template) - risk:low, ci:green, codex-ready ARE functional (merge_manager.js, issue templates) Only 5-6 labels confirmed as bloat: - codex (bare) - redundant with agent:codex - ai:agent - zero matches - auto-merge-audit - zero matches - automerge:ok - zero matches - architecture, backend, cli, etc. - repo-specific, not synced Phase 5 Analysis: - 5A: Auto-labeling - label_matcher.py EXISTS, ready for workflow - 5B: Coverage check - maint-coverage-guard.yml EXISTS, add soft PR check - 5C: Stale PR cleanup - not needed - 5D: Dependabot - partial (auto-label exists, add auto-merge) - 5E: Issue lint - soft warning approach - 5F: Cross-repo linking - weekly scan with semantic_matcher.py - 5G: Metrics - hybrid LangSmith (LLM) + custom (workflow) * Consolidate agents:pause to agents:paused + expand Phase 4-5 plans Label consolidation: - Replace agents:pause with agents:paused in all source files - Update keepalive_gate.js PAUSE_LABEL constant - Update keepalive_orchestrator_gate_runner.js hardcoded check - Update test to use agents:paused - Update documentation in README, CLAUDE.md, GoalsAndPlumbing.md Phase 4 updates: - 4A: Add idiosyncratic repo bloat cleanup strategy (per-repo audit) - 4B: Add optional issue creation feature to user guide (deferred) - 4D: Full conflict resolution implementation with code examples - 4E: Complete verify-to-issue workflow implementation Phase 5 updates: - 5F: Marked as SKIPPED (not needed per user decision) - 5G: Full LangSmith integration plan + custom metrics All keepalive tests pass (8/8). * feat: Implement Phase 4-5 automation features Phase 4 implementations: - 4A: Add scripts/cleanup_labels.py for label auditing - Classifies labels as functional/bloat/idiosyncratic - Requires --confirm flag for actual deletion - Reports audit results with recommendations - 4D: Add conflict detection for keepalive pipeline - .github/scripts/conflict_detector.js module - Detects conflicts from GitHub API, CI logs, PR comments - templates/consumer-repo/.github/codex/prompts/fix_merge_conflicts.md - 4E: Add agents-verify-to-issue.yml workflow - Creates follow-up issues from verification feedback - User-triggered via verify:create-issue label - Extracts concerns and low scores automatically Phase 5 implementations: - 5A: Add agents-auto-label.yml workflow - Semantic label matching for new issues - 90% threshold for auto-apply, 75% for suggestions - Uses existing label_matcher.py script - 5G: Add LangSmith tracing to tools/llm_provider.py - _setup_langsmith_tracing() function - Auto-configures when LANGSMITH_API_KEY present Also: - Update .github/sync-manifest.yml with new sync entries - Update docs/LABELS.md with new label documentation * fix: Address Copilot review comments on PR #650 Code quality improvements based on automated code review: 1. tools/llm_provider.py: - Fix LangSmith API key env var (LANGSMITH_API_KEY vs LANGCHAIN_API_KEY) - Improve f-string formatting for logging - Add usage comment for LANGSMITH_ENABLED constant 2. .github/scripts/conflict_detector.js: - Add debug logging in catch blocks instead of silent failures - Makes debugging easier when log downloads fail 3. .github/workflows/agents-verify-to-issue.yml: - Replace /tmp file usage with GitHub Actions environment files - Use heredoc delimiter for multi-line output - Consolidate find and extract steps for cleaner flow 4. .github/workflows/agents-auto-label.yml: - Make Workflows repo checkout configurable (not hardcoded) - Use github.paginate() for label retrieval (handles >100 labels) 5. templates/consumer-repo/.github/codex/prompts/fix_merge_conflicts.md: - Replace hardcoded 'main' with {{base_branch}} template variable - Make verification steps language-agnostic (not Python-specific) - Add note about checking project README for test commands * fix: Fix CI test failures 1. Fix test_integration_template_installs_and_tests - The test used --user pip install flag which fails in virtualenvs - Added _in_virtualenv() helper to detect virtualenv environment - Only use --user flag when NOT in a virtualenv 2. Add new workflows to expected names mapping - agents-auto-label.yml: 'Auto-Label Issues' - agents-verify-to-issue.yml: 'Create Issue from Verification' 3. Update workflow documentation - docs/ci/WORKFLOWS.md: Added bullet points for new workflows - docs/ci/WORKFLOW_SYSTEM.md: Added table rows for new workflows All 1120 tests now pass. * fix: Remove duplicate env key in agents-auto-label.yml actionlint was failing because the Match labels step had two env blocks. Merged ISSUE_TITLE and ISSUE_BODY into the main env block. * feat: Add Phase 3 workflows and sync configuration Phase 3 Pre-Agent Intelligence workflows: - agents-capability-check.yml: Pre-flight agent feasibility gate - agents-decompose.yml: Task decomposition for large issues - agents-dedup.yml: Duplicate detection using embeddings - agents-auto-label.yml: Semantic label matching Also includes: - agents-verify-to-issue.yml: Create follow-up issues from verification (Phase 4E) - Updated sync-manifest.yml with all new workflow entries - pr_verifier.py: Auth error fallback for LLM provider resilience - Tests for fallback behavior All Phase 3 scripts have 129 tests passing. * docs: Add comprehensive Phase 3 testing plan - Mark all Phase 3 implementation tasks as complete - Add detailed test suite with 12 specific test cases: - Suite A: Capability Check (3 tests) - Suite B: Task Decomposition (3 tests) - Suite C: Duplicate Detection (4 tests) - Suite D: Auto-Label (2 tests) - Include pre-testing checklist and execution tracking table - Add rollback plan and success criteria - Include sample issue bodies for reproducible tests * docs: Add deployment verification plan for cross-repo testing Addresses known issue: verify:compare works on Travel-Plan-Permission but fails on Trend_Model_Project PR #4249. New deployment verification plan includes: - Phase 1: Sync deployment tracking across all 7 repos - Phase 2: Existing workflow verification (investigate failures) - Phase 3: New workflow verification with specific test cases - Phase 4: Troubleshooting guide for common issues - Cross-repo verification summary with minimum pass criteria Separates deployment verification from functional regression testing. * docs: Resolve verify:compare investigation - PR not merged (expected behavior) Investigation findings for Trend_Model_Project PR #4249: - Root cause: PR is OPEN, not merged - Verifier correctly skipped (designed for merged PRs only) - verify:* labels missing in most repos (only Travel-Plan-Permission has them) - Added label prerequisite checklist to deployment plan - Updated verification summary with resolved status
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
The verifier was running and potentially creating follow-up issues even when there were no acceptance criteria to verify. This led to noisy issue creation.
Changes
acceptanceCount == 0Testing
shouldRun: falsewith appropriate skip reason when count is zero