Add test coverage for hasSubstantiveContent property by Copilot · Pull Request #318 · stranske/Workflows

Copilot · 2025-12-30T15:47:48Z

Automated Status Summary

Scope

Scope section missing from source issue.

Tasks

Tasks section missing from source issue.

Acceptance criteria

Head SHA: 50345d8
Latest Runs: ⏳ queued — Gate
Required: gate: ⏳ queued

Workflow / Job	Result	Logs
Agents PR meta manager	❔ in progress	View run
CI Autofix Loop	✅ success	View run
Gate	⏳ queued	View run
Health 40 Sweep	✅ success	View run
Health 44 Gate Branch Protection	❔ in progress	View run
Health 45 Agents Guard	✅ success	View run
Health 50 Security Scan	❔ in progress	View run
Maint 52 Validate Workflows	✅ success	View run
PR 11 - Minimal invariant CI	✅ success	View run
Selftest CI	❔ in progress	View run
Validate Sync Manifest	✅ success	View run

agents-workflows-bot · 2025-12-30T15:50:13Z

⚠️ Action Required: Unable to determine source issue for PR #318. The PR title, branch name, or body must contain the issue number (e.g. #123, branch: issue-123, or the hidden marker ).

Copilot

Pull request overview

This PR adds comprehensive test coverage for the hasSubstantiveContent property introduced in #316 to prevent creating follow-up issues with only placeholder text. The PR also includes an implementation fix that wraps hasSubstantiveContent calculations in Boolean() to ensure boolean return values in edge cases.

Key Changes

Added 11 new test cases (6 for formatFollowUpIssue, 5 for formatSimpleFollowUpIssue) to verify hasSubstantiveContent behavior with various content combinations
Wrapped hasSubstantiveContent calculations in both formatters with Boolean() to prevent returning empty strings
Updated workflow files to skip issue creation when hasSubstantiveContent is false, with proper output handling

Reviewed changes

Copilot reviewed 5 out of 5 changed files in this pull request and generated 4 comments.

Show a summary per file

File	Description
`docs/workflows/SystemEvaluation.md`	Updated documentation to reflect completed parallel testing implementation
`.github/workflows/reusable-agents-verifier.yml`	Added skip logic to prevent issue creation when no substantive content exists; sets `skipped` output
`.github/workflows/agents-verifier.yml`	Same skip logic as reusable workflow to maintain consistency across verifier implementations
`.github/scripts/verifier_issue_formatter.js`	Added three helper functions for detecting placeholder content; added `hasSubstantiveContent` property to both formatters with `Boolean()` wrapper; exported helpers for testing
`.github/scripts/__tests__/verifier-issue-formatter.test.js`	Added comprehensive test suite for `hasSubstantiveContent` covering placeholders, real content, verifier gaps, and edge cases

💡 Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.

.github/scripts/verifier_issue_formatter.js

.github/scripts/__tests__/verifier-issue-formatter.test.js

.github/scripts/verifier_issue_formatter.js

Co-authored-by: stranske <23046322+stranske@users.noreply.github.com>

github-actions · 2025-12-30T16:21:47Z

Automated Status Summary

Head SHA: 7c273e5
Latest Runs: ⏳ pending — Gate
Required contexts: Gate / gate, Health 45 Agents Guard / Enforce agents workflow protections
Required: core tests (3.11): ⏳ pending, core tests (3.12): ⏳ pending, docker smoke: ⏳ pending, gate: ⏳ pending

Workflow / Job	Result	Logs
(no jobs reported)	⏳ pending	—

Coverage Overview

Coverage history entries: 1

Coverage Trend

Metric	Value
Current	0.00%
Baseline	85.00%
Delta	-85.00%
Minimum	70.00%
Status	❌ Below minimum

Top Coverage Hotspots (lowest coverage)

File	Coverage	Missing
`src/trend_analysis/_autofix_trigger_sample.py`	0.0%	10
`src/trend_analysis/_autofix_violation_case2.py`	0.0%	15
`src/trend_analysis/_autofix_violation_case3.py`	0.0%	13
`src/trend_analysis/_ci_probe_faults.py`	0.0%	18
`src/trend_analysis/automation_multifailure.py`	0.0%	4
`src/trend_analysis/constants.py`	0.0%	1
`src/trend_analysis/selector.py`	0.0%	12
`src/trend_analysis/weighting.py`	0.0%	11
`src/trend_analysis/__init__.py`	100.0%	0

Low Coverage Files (<50.0%)

File	Coverage	Missing
`src/trend_analysis/_autofix_trigger_sample.py`	0.0%	10
`src/trend_analysis/_autofix_violation_case2.py`	0.0%	15
`src/trend_analysis/_autofix_violation_case3.py`	0.0%	13
`src/trend_analysis/_ci_probe_faults.py`	0.0%	18
`src/trend_analysis/automation_multifailure.py`	0.0%	4
`src/trend_analysis/constants.py`	0.0%	1
`src/trend_analysis/selector.py`	0.0%	12
`src/trend_analysis/weighting.py`	0.0%	11

Updated automatically; will refresh on subsequent CI/Docker completions.

Keepalive checklist

Scope

No scope information available

Tasks

No tasks defined

Acceptance criteria

No acceptance criteria defined

github-actions · 2025-12-30T16:22:10Z

🤖 Keepalive Loop Status

PR #318 | Agent: Codex | Iteration 0/5

Current State

Metric	Value
Iteration progress	[----------] 0/5
Action	wait (missing-agent-label)
Gate	success
Tasks	0/36 complete
Keepalive	❌ disabled
Autofix	❌ disabled

🔍 Failure Classification

…I access - Add 'models: read' permission to reusable-agents-verifier.yml - Add 'models: read' permission to agents-verifier.yml - Fixes 401 authentication errors when using GitHub Models provider - Templates already have this permission configured Resolves GitHub Models authentication issue identified in Travel-Plan-Permission PR #318 test

* feat: add model name to comparison report output Display the model name used by each provider in verify:compare reports. Changes: - Add 'model' field to EvaluationResult to track which model was used - Update _get_llm_clients to return tuples with (client, provider, model) - Add 'Model' column to Provider Summary table - Display model name in expandable Full Provider Details section - Update _fallback_evaluation to accept and store model parameter Example output: | Provider | Model | Verdict | Confidence | Summary | | github-models | gpt-4o | PASS | 85% | ... | | openai | gpt-5.2 | CONCERNS | 72% | ... | This helps users understand which models were used for evaluation, especially when using model1/model2 parameters in compare mode. * fix: update tests to match new 3-tuple format with model name - Add model parameter to test client tuples in test_pr_verifier_compare.py - Update table header assertion to include Model column in test_pr_verifier_comparison_report.py - Addresses failing checks and bot review comments in PR #643 * feat: disable automatic follow-up issue creation by agent verifier The agent verifier will no longer automatically create issues after PR evaluations, regardless of CONCERNS or FAIL verdicts. This addresses user feedback that automatic issue creation was creating unwanted noise. Changes: - Modified _should_create_issue() to always return False - Updated test to verify that issue creation is disabled - Workflow will still have --create-issue flag but it will have no effect * fix: add models permission to verifier workflows for GitHub Models API access - Add 'models: read' permission to reusable-agents-verifier.yml - Add 'models: read' permission to agents-verifier.yml - Fixes 401 authentication errors when using GitHub Models provider - Templates already have this permission configured Resolves GitHub Models authentication issue identified in Travel-Plan-Permission PR #318 test

Copilot AI assigned Copilot and stranske Dec 30, 2025

Copilot AI mentioned this pull request Dec 30, 2025

fix: prevent contentless follow-up issues from verifier #316

Merged

20 tasks

Copilot started work on behalf of stranske December 30, 2025 15:48 View session

Base automatically changed from fix/verifier-contentless-issues to main December 30, 2025 15:50

Copilot AI changed the title ~~[WIP] Address feedback on preventing contentless follow-up issues~~ Add test coverage for hasSubstantiveContent property Dec 30, 2025

Copilot AI requested a review from stranske December 30, 2025 15:54

Copilot finished work on behalf of stranske December 30, 2025 15:54

stranske approved these changes Dec 30, 2025

View reviewed changes

stranske marked this pull request as ready for review December 30, 2025 15:58

Copilot AI review requested due to automatic review settings December 30, 2025 15:58

Copilot started reviewing on behalf of stranske December 30, 2025 15:58 View session

Copilot AI reviewed Dec 30, 2025

View reviewed changes

Copilot AI and others added 2 commits December 30, 2025 16:19

Initial plan

8321162

Add test coverage for hasSubstantiveContent property

50345d8

Co-authored-by: stranske <23046322+stranske@users.noreply.github.com>

stranske force-pushed the copilot/sub-pr-316-again branch from e2dd488 to 50345d8 Compare December 30, 2025 16:20

stranske temporarily deployed to agent-standard December 30, 2025 16:20 — with GitHub Actions Inactive

stranske merged commit 0c608ed into main Dec 30, 2025
51 checks passed

stranske deleted the copilot/sub-pr-316-again branch December 30, 2025 16:22

github-actions bot mentioned this pull request Dec 30, 2025

[Follow-up] Unmet criteria from PR #318 #323

Closed

3 tasks

Copilot AI mentioned this pull request Jan 7, 2026

Add model name to comparison #644

Merged

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Add test coverage for hasSubstantiveContent property#318

Add test coverage for hasSubstantiveContent property#318
stranske merged 2 commits intomainfrom
copilot/sub-pr-316-again

Copilot AI commented Dec 30, 2025 •

edited by agents-workflows-bot bot

Loading

Uh oh!

agents-workflows-bot bot commented Dec 30, 2025

Uh oh!

Copilot AI left a comment

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

github-actions bot commented Dec 30, 2025

Uh oh!

Uh oh!

github-actions bot commented Dec 30, 2025

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

3 participants

Conversation

Copilot AI commented Dec 30, 2025 • edited by agents-workflows-bot bot Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Automated Status Summary

Scope

Tasks

Acceptance criteria

Uh oh!

agents-workflows-bot bot commented Dec 30, 2025

Uh oh!

Copilot AI left a comment

Choose a reason for hiding this comment

Pull request overview

Key Changes

Reviewed changes

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

github-actions bot commented Dec 30, 2025

Automated Status Summary

Coverage Overview

Coverage Trend

Top Coverage Hotspots (lowest coverage)

Low Coverage Files (<50.0%)

Keepalive checklist

Scope

Tasks

Acceptance criteria

Uh oh!

Uh oh!

github-actions bot commented Dec 30, 2025

🤖 Keepalive Loop Status

Current State

🔍 Failure Classification

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

3 participants

Copilot AI commented Dec 30, 2025 •

edited by agents-workflows-bot bot

Loading