Skip to content

feat(verifier): add acceptance_count output for metrics#133

Merged
stranske merged 3 commits intomainfrom
fix/verifier-acceptance-count
Dec 25, 2025
Merged

feat(verifier): add acceptance_count output for metrics#133
stranske merged 3 commits intomainfrom
fix/verifier-acceptance-count

Conversation

@stranske
Copy link
Copy Markdown
Owner

@stranske stranske commented Dec 25, 2025 •
edited by agents-workflows-bot bot
Loading

Summary

Cherry-picks the unique valuable content from PR #100 - acceptance criteria counting.

Changes

  • Add countCheckboxes() function to count acceptance criteria in content
  • Import parseScopeTasksAcceptanceSections for checkbox extraction
  • Output acceptance_count from buildVerifierContext for verifier metrics
  • Count checkboxes from both PR body and linked issues

Background

PR #100 was stale and conflicting badly with main:

This PR extracts that feature cleanly onto current main.

Supersedes #100

Automated Status Summary

Scope

  • Scope section missing from source issue.

Tasks

  • Tasks section missing from source issue.

Acceptance criteria

  • Acceptance criteria section missing from source issue.

Head SHA: 20d2eff
Latest Runs: ✅ success — Gate
Required: gate: ✅ success

Workflow / Job Result Logs
Agents PR meta manager ❔ in progress View run
CI Autofix Loop ✅ success View run
Gate ✅ success View run
Health 40 Sweep ✅ success View run
Health 44 Gate Branch Protection ✅ success View run
Health 45 Agents Guard ✅ success View run
Health 50 Security Scan ✅ success View run
Maint 52 Validate Workflows ✅ success View run
PR 11 - Minimal invariant CI ✅ success View run
Selftest CI ✅ success View run

@stranske
feat(verifier): add acceptance_count output for metrics
2a7bb65 
Cherry-picked from PR #100. Counts acceptance criteria checkboxes in
the PR body and linked issues, outputting acceptance_count for use
in verifier metrics.

This was the unique remaining value from PR #100 - the rest of its
changes (PRs #97, #98) were already merged.

Closes #100
Copilot AI review requested due to automatic review settings December 25, 2025 06:37
@stranske stranske mentioned this pull request Dec 25, 2025
Closed
@github-actions
Copy link
Copy Markdown
Contributor

github-actions bot commented Dec 25, 2025 •
edited
Loading

Automated Status Summary

Head SHA: 409a17b
Latest Runs: ⏳ pending — Gate
Required contexts: Gate / gate, Health 45 Agents Guard / Enforce agents workflow protections
Required: core tests (3.11): ⏳ pending, core tests (3.12): ⏳ pending, docker smoke: ⏳ pending, gate: ⏳ pending

Workflow / Job Result Logs
(no jobs reported) ⏳ pending —

Coverage Overview

  • Coverage history entries: 1

Coverage Trend

Metric Value
Current 77.97%
Baseline 0.00%
Delta +77.97%
Minimum 70.00%
Status ✅ Pass

Updated automatically; will refresh on subsequent CI/Docker completions.

Keepalive checklist

Scope

  • Scope section missing from source issue.

Tasks

  • Tasks section missing from source issue.

Acceptance criteria

  • Acceptance criteria section missing from source issue.

@github-actions
Copy link
Copy Markdown
Contributor

github-actions bot commented Dec 25, 2025 •
edited
Loading

🤖 Keepalive Loop Status

PR #133 | Agent: Codex | Iteration 0/5

Current State

Metric Value
Iteration progress [----------] 0/5
Action wait (missing-agent-label-repeat)
Gate success
Tasks 0/2 complete
Keepalive ❌ disabled
Autofix ❌ disabled

⚠️ Failure Tracking

| Consecutive failures | 3/3 |
| Reason | missing-agent-label |

🛑 Paused – Human Attention Required

The keepalive loop has paused due to repeated failures.

To resume:

  1. Investigate the failure reason above
  2. Fix any issues in the code or prompt
  3. Remove the needs-human label from this PR
  4. The next Gate pass will restart the loop

Or manually edit this comment to reset failure: {} in the state below.

Copilot AI reviewed Dec 25, 2025
View reviewed changes
Copy link
Copy Markdown
Contributor

Copilot AI left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Pull request overview

This PR adds acceptance criteria counting functionality to the verifier context builder, extracting metrics on the number of acceptance criteria checkboxes from both PR body and linked issues.

Key Changes:

  • Introduces countCheckboxes() helper function to count checkbox items using regex pattern matching
  • Integrates acceptance counting into the verifier workflow by parsing acceptance sections and accumulating counts
  • Outputs acceptance_count metric for downstream consumption

💡 Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.

@stranske
Update .github/scripts/agents_verifier_context.js
2414ae6 
Co-authored-by: Copilot <175728472+Copilot@users.noreply.github.com>
@stranske
Copy link
Copy Markdown
Owner Author

stranske commented Dec 25, 2025

@codex fix comments

@chatgpt-codex-connector
Copy link
Copy Markdown

chatgpt-codex-connector bot commented Dec 25, 2025

Summary

Testing

  • ✅ node --test .github/scripts/__tests__/agents-verifier-context.test.js

View task →

@github-actions github-actions bot added the needs-human Requires human intervention or review label Dec 25, 2025
@stranske stranske merged commit 8cf1543 into main Dec 25, 2025
114 checks passed
@stranske stranske deleted the fix/verifier-acceptance-count branch December 25, 2025 06:51
stranske added a commit that referenced this pull request Jan 7, 2026
@stranske
Add Phase 3 test issues for Manager-Database
ef6fcec 
Created 3 test issues:
- #193: Stripe integration (should FAIL capability check)
- #194: Health monitoring (should trigger task decomposition)
- #196: Manager list API (should detect as duplicate of #133)

Updated testing metrics dashboard to track progress.
stranske added a commit that referenced this pull request Jan 7, 2026
@stranske
Add Phase 3 integration plan with testing cycle (#645)
d4e89a0 
* Add Phase 3 integration plan with testing cycle for Manager-Database

Phase 3: Pre-Agent Intelligence (4 capabilities)
- 3A: Capability Check - supplements agents:optimize with feasibility gate
  - Runs BEFORE agent assignment on Issues (not after)
  - Adds needs-human label when agent cannot proceed
- 3B: Task Decomposition - auto-split large issues
- 3C: Duplicate Detection - comment-only mode, track false positives
- 3D: Semantic Labeling - auto-suggest/apply labels

Testing Plan:
- Test repo: Manager-Database
- ~11 test issues across 4 capabilities
- False positive tracking for dedup (target: <5%)
- Metrics dashboard for validation

Also updates:
- Mark Collab-Admin PR #113 as merged (7/7 repos now synced)
- All immediate tasks completed
- Phase 3 ready to begin

* Add Phase 3 test issues for Manager-Database

Created 3 test issues:
- #193: Stripe integration (should FAIL capability check)
- #194: Health monitoring (should trigger task decomposition)
- #196: Manager list API (should detect as duplicate of #133)

Updated testing metrics dashboard to track progress.

* Add Phase 4: Full Automation & Cleanup plan

Phase 4 includes 5 initiatives:
- 4A: Label Cleanup - Remove bloat labels, standardize across 7 repos
- 4B: User Guide - Operational documentation for label system (sync to consumers)
- 4C: Auto-Pilot Label - End-to-end issue-to-merged-PR automation
- 4D: Conflict Resolution - Automated merge conflict handling in keepalive
- 4E: Verify-to-Issue - Create follow-up issues from verification feedback

Key decisions:
- Auto-pilot uses workflow_dispatch between steps (not chained labels)
- Conflict detection added to keepalive loop (not separate workflow)
- Verify-to-issue is user-triggered (not automatic, avoids false positives)

Also identifies 7 additional automation opportunities for future phases.

Testing plan defined for Manager-Database.

* Correct label analysis after codebase search + expand Phase 5

Label Analysis Corrections:
- agents:pause/paused ARE functional (keepalive_gate.js, keepalive-runner.js)
- agents:activated IS functional (agents_pr_meta_keepalive.js)
- from:codex/copilot ARE functional (merge_manager.js)
- automerge IS functional (merge_manager.js, agents_belt_scan.js)
- agents (bare) IS functional (agent_task.yml template)
- risk:low, ci:green, codex-ready ARE functional (merge_manager.js, issue templates)

Only 5-6 labels confirmed as bloat:
- codex (bare) - redundant with agent:codex
- ai:agent - zero matches
- auto-merge-audit - zero matches
- automerge:ok - zero matches
- architecture, backend, cli, etc. - repo-specific, not synced

Phase 5 Analysis:
- 5A: Auto-labeling - label_matcher.py EXISTS, ready for workflow
- 5B: Coverage check - maint-coverage-guard.yml EXISTS, add soft PR check
- 5C: Stale PR cleanup - not needed
- 5D: Dependabot - partial (auto-label exists, add auto-merge)
- 5E: Issue lint - soft warning approach
- 5F: Cross-repo linking - weekly scan with semantic_matcher.py
- 5G: Metrics - hybrid LangSmith (LLM) + custom (workflow)

* Consolidate agents:pause to agents:paused + expand Phase 4-5 plans

Label consolidation:
- Replace agents:pause with agents:paused in all source files
- Update keepalive_gate.js PAUSE_LABEL constant
- Update keepalive_orchestrator_gate_runner.js hardcoded check
- Update test to use agents:paused
- Update documentation in README, CLAUDE.md, GoalsAndPlumbing.md

Phase 4 updates:
- 4A: Add idiosyncratic repo bloat cleanup strategy (per-repo audit)
- 4B: Add optional issue creation feature to user guide (deferred)
- 4D: Full conflict resolution implementation with code examples
- 4E: Complete verify-to-issue workflow implementation

Phase 5 updates:
- 5F: Marked as SKIPPED (not needed per user decision)
- 5G: Full LangSmith integration plan + custom metrics

All keepalive tests pass (8/8).
stranske added a commit that referenced this pull request Jan 7, 2026
@stranske
feat: Implement Phase 4-5 automation features (#650)
1a15b48 
* Add Phase 3 integration plan with testing cycle for Manager-Database

Phase 3: Pre-Agent Intelligence (4 capabilities)
- 3A: Capability Check - supplements agents:optimize with feasibility gate
  - Runs BEFORE agent assignment on Issues (not after)
  - Adds needs-human label when agent cannot proceed
- 3B: Task Decomposition - auto-split large issues
- 3C: Duplicate Detection - comment-only mode, track false positives
- 3D: Semantic Labeling - auto-suggest/apply labels

Testing Plan:
- Test repo: Manager-Database
- ~11 test issues across 4 capabilities
- False positive tracking for dedup (target: <5%)
- Metrics dashboard for validation

Also updates:
- Mark Collab-Admin PR #113 as merged (7/7 repos now synced)
- All immediate tasks completed
- Phase 3 ready to begin

* Add Phase 3 test issues for Manager-Database

Created 3 test issues:
- #193: Stripe integration (should FAIL capability check)
- #194: Health monitoring (should trigger task decomposition)
- #196: Manager list API (should detect as duplicate of #133)

Updated testing metrics dashboard to track progress.

* Add Phase 4: Full Automation & Cleanup plan

Phase 4 includes 5 initiatives:
- 4A: Label Cleanup - Remove bloat labels, standardize across 7 repos
- 4B: User Guide - Operational documentation for label system (sync to consumers)
- 4C: Auto-Pilot Label - End-to-end issue-to-merged-PR automation
- 4D: Conflict Resolution - Automated merge conflict handling in keepalive
- 4E: Verify-to-Issue - Create follow-up issues from verification feedback

Key decisions:
- Auto-pilot uses workflow_dispatch between steps (not chained labels)
- Conflict detection added to keepalive loop (not separate workflow)
- Verify-to-issue is user-triggered (not automatic, avoids false positives)

Also identifies 7 additional automation opportunities for future phases.

Testing plan defined for Manager-Database.

* Correct label analysis after codebase search + expand Phase 5

Label Analysis Corrections:
- agents:pause/paused ARE functional (keepalive_gate.js, keepalive-runner.js)
- agents:activated IS functional (agents_pr_meta_keepalive.js)
- from:codex/copilot ARE functional (merge_manager.js)
- automerge IS functional (merge_manager.js, agents_belt_scan.js)
- agents (bare) IS functional (agent_task.yml template)
- risk:low, ci:green, codex-ready ARE functional (merge_manager.js, issue templates)

Only 5-6 labels confirmed as bloat:
- codex (bare) - redundant with agent:codex
- ai:agent - zero matches
- auto-merge-audit - zero matches
- automerge:ok - zero matches
- architecture, backend, cli, etc. - repo-specific, not synced

Phase 5 Analysis:
- 5A: Auto-labeling - label_matcher.py EXISTS, ready for workflow
- 5B: Coverage check - maint-coverage-guard.yml EXISTS, add soft PR check
- 5C: Stale PR cleanup - not needed
- 5D: Dependabot - partial (auto-label exists, add auto-merge)
- 5E: Issue lint - soft warning approach
- 5F: Cross-repo linking - weekly scan with semantic_matcher.py
- 5G: Metrics - hybrid LangSmith (LLM) + custom (workflow)

* Consolidate agents:pause to agents:paused + expand Phase 4-5 plans

Label consolidation:
- Replace agents:pause with agents:paused in all source files
- Update keepalive_gate.js PAUSE_LABEL constant
- Update keepalive_orchestrator_gate_runner.js hardcoded check
- Update test to use agents:paused
- Update documentation in README, CLAUDE.md, GoalsAndPlumbing.md

Phase 4 updates:
- 4A: Add idiosyncratic repo bloat cleanup strategy (per-repo audit)
- 4B: Add optional issue creation feature to user guide (deferred)
- 4D: Full conflict resolution implementation with code examples
- 4E: Complete verify-to-issue workflow implementation

Phase 5 updates:
- 5F: Marked as SKIPPED (not needed per user decision)
- 5G: Full LangSmith integration plan + custom metrics

All keepalive tests pass (8/8).

* feat: Implement Phase 4-5 automation features

Phase 4 implementations:
- 4A: Add scripts/cleanup_labels.py for label auditing
  - Classifies labels as functional/bloat/idiosyncratic
  - Requires --confirm flag for actual deletion
  - Reports audit results with recommendations

- 4D: Add conflict detection for keepalive pipeline
  - .github/scripts/conflict_detector.js module
  - Detects conflicts from GitHub API, CI logs, PR comments
  - templates/consumer-repo/.github/codex/prompts/fix_merge_conflicts.md

- 4E: Add agents-verify-to-issue.yml workflow
  - Creates follow-up issues from verification feedback
  - User-triggered via verify:create-issue label
  - Extracts concerns and low scores automatically

Phase 5 implementations:
- 5A: Add agents-auto-label.yml workflow
  - Semantic label matching for new issues
  - 90% threshold for auto-apply, 75% for suggestions
  - Uses existing label_matcher.py script

- 5G: Add LangSmith tracing to tools/llm_provider.py
  - _setup_langsmith_tracing() function
  - Auto-configures when LANGSMITH_API_KEY present

Also:
- Update .github/sync-manifest.yml with new sync entries
- Update docs/LABELS.md with new label documentation

* fix: Address Copilot review comments on PR #650

Code quality improvements based on automated code review:

1. tools/llm_provider.py:
   - Fix LangSmith API key env var (LANGSMITH_API_KEY vs LANGCHAIN_API_KEY)
   - Improve f-string formatting for logging
   - Add usage comment for LANGSMITH_ENABLED constant

2. .github/scripts/conflict_detector.js:
   - Add debug logging in catch blocks instead of silent failures
   - Makes debugging easier when log downloads fail

3. .github/workflows/agents-verify-to-issue.yml:
   - Replace /tmp file usage with GitHub Actions environment files
   - Use heredoc delimiter for multi-line output
   - Consolidate find and extract steps for cleaner flow

4. .github/workflows/agents-auto-label.yml:
   - Make Workflows repo checkout configurable (not hardcoded)
   - Use github.paginate() for label retrieval (handles >100 labels)

5. templates/consumer-repo/.github/codex/prompts/fix_merge_conflicts.md:
   - Replace hardcoded 'main' with {{base_branch}} template variable
   - Make verification steps language-agnostic (not Python-specific)
   - Add note about checking project README for test commands

* fix: Fix CI test failures

1. Fix test_integration_template_installs_and_tests
   - The test used --user pip install flag which fails in virtualenvs
   - Added _in_virtualenv() helper to detect virtualenv environment
   - Only use --user flag when NOT in a virtualenv

2. Add new workflows to expected names mapping
   - agents-auto-label.yml: 'Auto-Label Issues'
   - agents-verify-to-issue.yml: 'Create Issue from Verification'

3. Update workflow documentation
   - docs/ci/WORKFLOWS.md: Added bullet points for new workflows
   - docs/ci/WORKFLOW_SYSTEM.md: Added table rows for new workflows

All 1120 tests now pass.

* fix: Remove duplicate env key in agents-auto-label.yml

actionlint was failing because the Match labels step had two env blocks.
Merged ISSUE_TITLE and ISSUE_BODY into the main env block.
stranske added a commit that referenced this pull request Jan 8, 2026
@stranske
feat(phase3): Add Phase 3 agent workflows and deployment verification (…
5e6336d 
…#653)

* Add Phase 3 integration plan with testing cycle for Manager-Database

Phase 3: Pre-Agent Intelligence (4 capabilities)
- 3A: Capability Check - supplements agents:optimize with feasibility gate
  - Runs BEFORE agent assignment on Issues (not after)
  - Adds needs-human label when agent cannot proceed
- 3B: Task Decomposition - auto-split large issues
- 3C: Duplicate Detection - comment-only mode, track false positives
- 3D: Semantic Labeling - auto-suggest/apply labels

Testing Plan:
- Test repo: Manager-Database
- ~11 test issues across 4 capabilities
- False positive tracking for dedup (target: <5%)
- Metrics dashboard for validation

Also updates:
- Mark Collab-Admin PR #113 as merged (7/7 repos now synced)
- All immediate tasks completed
- Phase 3 ready to begin

* Add Phase 3 test issues for Manager-Database

Created 3 test issues:
- #193: Stripe integration (should FAIL capability check)
- #194: Health monitoring (should trigger task decomposition)
- #196: Manager list API (should detect as duplicate of #133)

Updated testing metrics dashboard to track progress.

* Add Phase 4: Full Automation & Cleanup plan

Phase 4 includes 5 initiatives:
- 4A: Label Cleanup - Remove bloat labels, standardize across 7 repos
- 4B: User Guide - Operational documentation for label system (sync to consumers)
- 4C: Auto-Pilot Label - End-to-end issue-to-merged-PR automation
- 4D: Conflict Resolution - Automated merge conflict handling in keepalive
- 4E: Verify-to-Issue - Create follow-up issues from verification feedback

Key decisions:
- Auto-pilot uses workflow_dispatch between steps (not chained labels)
- Conflict detection added to keepalive loop (not separate workflow)
- Verify-to-issue is user-triggered (not automatic, avoids false positives)

Also identifies 7 additional automation opportunities for future phases.

Testing plan defined for Manager-Database.

* Correct label analysis after codebase search + expand Phase 5

Label Analysis Corrections:
- agents:pause/paused ARE functional (keepalive_gate.js, keepalive-runner.js)
- agents:activated IS functional (agents_pr_meta_keepalive.js)
- from:codex/copilot ARE functional (merge_manager.js)
- automerge IS functional (merge_manager.js, agents_belt_scan.js)
- agents (bare) IS functional (agent_task.yml template)
- risk:low, ci:green, codex-ready ARE functional (merge_manager.js, issue templates)

Only 5-6 labels confirmed as bloat:
- codex (bare) - redundant with agent:codex
- ai:agent - zero matches
- auto-merge-audit - zero matches
- automerge:ok - zero matches
- architecture, backend, cli, etc. - repo-specific, not synced

Phase 5 Analysis:
- 5A: Auto-labeling - label_matcher.py EXISTS, ready for workflow
- 5B: Coverage check - maint-coverage-guard.yml EXISTS, add soft PR check
- 5C: Stale PR cleanup - not needed
- 5D: Dependabot - partial (auto-label exists, add auto-merge)
- 5E: Issue lint - soft warning approach
- 5F: Cross-repo linking - weekly scan with semantic_matcher.py
- 5G: Metrics - hybrid LangSmith (LLM) + custom (workflow)

* Consolidate agents:pause to agents:paused + expand Phase 4-5 plans

Label consolidation:
- Replace agents:pause with agents:paused in all source files
- Update keepalive_gate.js PAUSE_LABEL constant
- Update keepalive_orchestrator_gate_runner.js hardcoded check
- Update test to use agents:paused
- Update documentation in README, CLAUDE.md, GoalsAndPlumbing.md

Phase 4 updates:
- 4A: Add idiosyncratic repo bloat cleanup strategy (per-repo audit)
- 4B: Add optional issue creation feature to user guide (deferred)
- 4D: Full conflict resolution implementation with code examples
- 4E: Complete verify-to-issue workflow implementation

Phase 5 updates:
- 5F: Marked as SKIPPED (not needed per user decision)
- 5G: Full LangSmith integration plan + custom metrics

All keepalive tests pass (8/8).

* feat: Implement Phase 4-5 automation features

Phase 4 implementations:
- 4A: Add scripts/cleanup_labels.py for label auditing
  - Classifies labels as functional/bloat/idiosyncratic
  - Requires --confirm flag for actual deletion
  - Reports audit results with recommendations

- 4D: Add conflict detection for keepalive pipeline
  - .github/scripts/conflict_detector.js module
  - Detects conflicts from GitHub API, CI logs, PR comments
  - templates/consumer-repo/.github/codex/prompts/fix_merge_conflicts.md

- 4E: Add agents-verify-to-issue.yml workflow
  - Creates follow-up issues from verification feedback
  - User-triggered via verify:create-issue label
  - Extracts concerns and low scores automatically

Phase 5 implementations:
- 5A: Add agents-auto-label.yml workflow
  - Semantic label matching for new issues
  - 90% threshold for auto-apply, 75% for suggestions
  - Uses existing label_matcher.py script

- 5G: Add LangSmith tracing to tools/llm_provider.py
  - _setup_langsmith_tracing() function
  - Auto-configures when LANGSMITH_API_KEY present

Also:
- Update .github/sync-manifest.yml with new sync entries
- Update docs/LABELS.md with new label documentation

* fix: Address Copilot review comments on PR #650

Code quality improvements based on automated code review:

1. tools/llm_provider.py:
   - Fix LangSmith API key env var (LANGSMITH_API_KEY vs LANGCHAIN_API_KEY)
   - Improve f-string formatting for logging
   - Add usage comment for LANGSMITH_ENABLED constant

2. .github/scripts/conflict_detector.js:
   - Add debug logging in catch blocks instead of silent failures
   - Makes debugging easier when log downloads fail

3. .github/workflows/agents-verify-to-issue.yml:
   - Replace /tmp file usage with GitHub Actions environment files
   - Use heredoc delimiter for multi-line output
   - Consolidate find and extract steps for cleaner flow

4. .github/workflows/agents-auto-label.yml:
   - Make Workflows repo checkout configurable (not hardcoded)
   - Use github.paginate() for label retrieval (handles >100 labels)

5. templates/consumer-repo/.github/codex/prompts/fix_merge_conflicts.md:
   - Replace hardcoded 'main' with {{base_branch}} template variable
   - Make verification steps language-agnostic (not Python-specific)
   - Add note about checking project README for test commands

* fix: Fix CI test failures

1. Fix test_integration_template_installs_and_tests
   - The test used --user pip install flag which fails in virtualenvs
   - Added _in_virtualenv() helper to detect virtualenv environment
   - Only use --user flag when NOT in a virtualenv

2. Add new workflows to expected names mapping
   - agents-auto-label.yml: 'Auto-Label Issues'
   - agents-verify-to-issue.yml: 'Create Issue from Verification'

3. Update workflow documentation
   - docs/ci/WORKFLOWS.md: Added bullet points for new workflows
   - docs/ci/WORKFLOW_SYSTEM.md: Added table rows for new workflows

All 1120 tests now pass.

* fix: Remove duplicate env key in agents-auto-label.yml

actionlint was failing because the Match labels step had two env blocks.
Merged ISSUE_TITLE and ISSUE_BODY into the main env block.

* feat: Add Phase 3 workflows and sync configuration

Phase 3 Pre-Agent Intelligence workflows:
- agents-capability-check.yml: Pre-flight agent feasibility gate
- agents-decompose.yml: Task decomposition for large issues
- agents-dedup.yml: Duplicate detection using embeddings
- agents-auto-label.yml: Semantic label matching

Also includes:
- agents-verify-to-issue.yml: Create follow-up issues from verification (Phase 4E)
- Updated sync-manifest.yml with all new workflow entries
- pr_verifier.py: Auth error fallback for LLM provider resilience
- Tests for fallback behavior

All Phase 3 scripts have 129 tests passing.

* docs: Add comprehensive Phase 3 testing plan

- Mark all Phase 3 implementation tasks as complete
- Add detailed test suite with 12 specific test cases:
  - Suite A: Capability Check (3 tests)
  - Suite B: Task Decomposition (3 tests)
  - Suite C: Duplicate Detection (4 tests)
  - Suite D: Auto-Label (2 tests)
- Include pre-testing checklist and execution tracking table
- Add rollback plan and success criteria
- Include sample issue bodies for reproducible tests

* docs: Add deployment verification plan for cross-repo testing

Addresses known issue: verify:compare works on Travel-Plan-Permission
but fails on Trend_Model_Project PR #4249.

New deployment verification plan includes:
- Phase 1: Sync deployment tracking across all 7 repos
- Phase 2: Existing workflow verification (investigate failures)
- Phase 3: New workflow verification with specific test cases
- Phase 4: Troubleshooting guide for common issues
- Cross-repo verification summary with minimum pass criteria

Separates deployment verification from functional regression testing.

* docs: Resolve verify:compare investigation - PR not merged (expected behavior)

Investigation findings for Trend_Model_Project PR #4249:
- Root cause: PR is OPEN, not merged
- Verifier correctly skipped (designed for merged PRs only)
- verify:* labels missing in most repos (only Travel-Plan-Permission has them)
- Added label prerequisite checklist to deployment plan
- Updated verification summary with resolved status
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Reviewers

Copilot code review Copilot Copilot left review comments

Assignees

No one assigned

Labels

needs-human Requires human intervention or review

Projects

None yet

Milestone

No milestone

Development

Successfully merging this pull request may close these issues.

2 participants

@stranske