Skip to content

fix(agents): fail closed format guard revalidation - #2987

Merged
stranske merged 3 commits into
mainfrom
automation/sync-format-guard-review-round2
Aug 8, 2026
Merged

fix(agents): fail closed format guard revalidation#2987
stranske merged 3 commits into
mainfrom
automation/sync-format-guard-review-round2

Conversation

@stranske

@stranske stranske commented Aug 8, 2026

Copy link
Copy Markdown
Owner

Summary

  • replace dispatch-only inputs references in issue-format guard and optimizer concurrency keys with event-safe inputs
  • fail closed when live issue revalidation hits an unexpected validator runtime error
  • preserve consumer template coverage and add focused regression assertions

Validation

  • python -m pytest -q tests/workflows/test_agents_issue_optimizer_format_trigger.py tests/scripts/test_issue_format.py (40 passed)
  • git diff --check

The full template-drift check still reports the pre-existing optimizer source/template divergence; this change updates both respective concurrency surfaces without collapsing that intentionally different workflow implementation.

Summary by CodeRabbit

  • Bug Fixes

    • Improved handling of manually dispatched issue workflow runs.
    • Issue validation now distinguishes correctly formatted content, formatting problems, and unexpected errors.
    • Prevented unnecessary follow-up actions when content already meets formatting requirements.
    • Improved issue identification and fallback handling for workflow runs.
  • Tests

    • Added regression coverage for manual issue selection, fallback behavior, validation errors, and dispatch decisions.

Copilot AI lite review requested due to automatic review settings August 8, 2026 17:35
@cursor

cursor Bot commented Aug 8, 2026

Copy link
Copy Markdown

Bugbot is not enabled for your account, so this pull request was not reviewed.

Enable Bugbot in the Cursor dashboard to get automatic reviews on future PRs.

@coderabbitai

coderabbitai Bot commented Aug 8, 2026

Copy link
Copy Markdown

Review Change Stack

No actionable comments were generated in the recent review. 🎉

ℹ️ Recent review info
⚙️ Run configuration

Configuration used: Path: .coderabbit.yaml

Review profile: ASSERTIVE

Plan: Pro

Run ID: 280257c1-d945-4413-9bd4-ede283c053b1

📥 Commits

Reviewing files that changed from the base of the PR and between 979021d and 3ecb0c8.

📒 Files selected for processing (1)
  • tests/workflows/test_agents_issue_optimizer_format_trigger.py

📝 Walkthrough

Walkthrough

Changes

The workflows now use event-scoped manual issue inputs. Format revalidation distinguishes valid content, ordinary format failures, and unexpected validator errors. Regression coverage and template-drift fingerprints were updated.

Issue workflow routing

Layer / File(s) Summary
Format-guard input and validation routing
.github/workflows/agents-issue-format-guard.yml, templates/consumer-repo/.github/workflows/agents-issue-format-guard.yml, tests/workflows/test_agents_issue_optimizer_format_trigger.py
Both format guards resolve manual issue numbers from github.event.inputs.issue_number. Revalidation skips dispatch for valid content, dispatches for ordinary format failures, and fails on unexpected validator errors. Tests cover these outcomes.
Optimizer concurrency and template baseline
.github/workflows/agents-issue-optimizer.yml, templates/consumer-repo/.github/workflows/agents-issue-optimizer.yml, config/template-drift-allowlist.txt, tests/workflows/test_agents_issue_optimizer_format_trigger.py
Both optimizer workflows use the event-scoped issue-number fallback. The allowlist stores updated workflow fingerprints. Tests cover optimizer issue-number fallback behavior.

Estimated code review effort: 3 (Moderate) | ~20 minutes

Sequence Diagram(s)

sequenceDiagram
  participant ManualDispatch
  participant FormatGuard
  participant Validator
  participant IssueOptimizer
  ManualDispatch->>FormatGuard: Provide issue_number through github.event.inputs
  FormatGuard->>Validator: Revalidate issue format
  Validator-->>FormatGuard: Return exit code and stderr
  FormatGuard->>IssueOptimizer: Dispatch for ordinary format failure
Loading

Possibly related PRs

Suggested labels: verify:compare

🚥 Pre-merge checks | ✅ 4 | ❌ 1

❌ Failed checks (1 warning)

Check name Status Explanation Resolution
Docstring Coverage ⚠️ Warning Docstring coverage is 0.00% which is insufficient. The required threshold is 80.00%. Write docstrings for the functions missing them to satisfy the coverage threshold.
✅ Passed checks (4 passed)
Check name Status Explanation
Description Check ✅ Passed Check skipped - CodeRabbit’s high-level summary is enabled.
Title check ✅ Passed The title clearly identifies the main change: making format guard revalidation fail closed.
Linked Issues check ✅ Passed Check skipped because no linked issues were found for this pull request.
Out of Scope Changes check ✅ Passed Check skipped because no linked issues were found for this pull request.
✨ Finishing Touches 💡 1
📝 Generate docstrings 💡
  • Create stacked PR
  • Commit on current branch
🧪 Generate unit tests (beta)
  • Create PR with unit tests
  • Commit unit tests in branch automation/sync-format-guard-review-round2

Comment @coderabbitai help to get the list of available commands.

@stranske
stranske temporarily deployed to agent-high-privilege August 8, 2026 17:35 — with GitHub Actions Inactive
@agents-workflows-bot

Copy link
Copy Markdown
Contributor

Workflow source needed

PR #2987 needs either a linked GitHub issue or one valid non-issue Workflow Source before PR metadata automation can manage it safely.

Please do one of:

  • Add <!-- meta:issue:123 --> or a normal Closes #123 / Related to #123 line.
  • Check one Workflow Source option in the PR body.
  • Add a hidden marker such as <!-- workflow-source:local_request -->, <!-- workflow-source:manual_remote -->, <!-- workflow-source:review_followup -->, <!-- workflow-source:sync_campaign -->, or <!-- workflow-source:dependabot -->.
  • Add a workflow source label such as workflow:source-direct-pr, workflow:source-local-request, workflow:source-review-followup, workflow:source-sync, or workflow:no-automation.

Once a valid source is present, this warning will not be reposted.

@chatgpt-codex-connector chatgpt-codex-connector Bot left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

💡 Codex Review

Here are some automated review suggestions for this pull request.

Reviewed commit: 0ce6dcf870

ℹ️ About Codex in GitHub

Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you

  • Open a pull request for review
  • Mark a draft as ready
  • Comment "@codex review".

If Codex has suggestions, it will comment; otherwise it will react with 👍.

Codex can also answer questions or update the PR. Try commenting "@codex address that feedback".

Comment thread .github/workflows/agents-issue-format-guard.yml
@stranske
stranske force-pushed the automation/sync-format-guard-review-round2 branch from 16e4c96 to b04ce02 Compare August 8, 2026 17:37

Copilot AI left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Pull request overview

Updates the agents issue format guard / optimizer workflows to use event-safe github.event.inputs references (avoiding dispatch-only inputs in concurrency keys), and aims to fail closed when live issue revalidation encounters unexpected validator failures. Changes are applied to both the repo workflows and the consumer template workflows, with regression assertions added to enforce the new patterns.

Changes:

  • Replace inputs.issue_number usage in concurrency group keys with github.event.inputs.issue_number (with github.run_id fallback).
  • Update the issue-format guard to capture and act on validator exit codes during live revalidation.
  • Add regression tests asserting event-safe inputs usage and the intended revalidation behavior across source + template workflows.

Reviewed changes

Copilot reviewed 6 out of 6 changed files in this pull request and generated 3 comments.

Show a summary per file
File Description
tests/workflows/test_agents_issue_optimizer_format_trigger.py Adds regression assertions for event-safe inputs usage and revalidation behavior across source/template workflows.
templates/consumer-repo/.github/workflows/agents-issue-optimizer.yml Updates concurrency group key to use github.event.inputs.issue_number fallback.
templates/consumer-repo/.github/workflows/agents-issue-format-guard.yml Updates concurrency group key and revalidation handling in the consumer template workflow.
.github/workflows/agents-issue-optimizer.yml Updates concurrency group key to use github.event.inputs.issue_number fallback.
.github/workflows/agents-issue-format-guard.yml Updates concurrency group key and revalidation handling in the source workflow.

💡 Add a code-review agent skill or configure MCP servers for context-aware, tailored reviews. Learn more in the docs.

Comment thread .github/workflows/agents-issue-format-guard.yml
Comment on lines +102 to +106
assert "github.event.inputs.issue_number ||\n github.run_id" in text
assert "github.event.issue.number || github.event.inputs.issue_number" in text
assert "revalidate_rc=$?" in text
assert 'if [[ "$revalidate_rc" -ne 1 ]]; then' in text

@stranske-keepalive

stranske-keepalive Bot commented Aug 8, 2026

Copy link
Copy Markdown
Contributor

Automated Status Summary

Head SHA: bd8a762
Latest Runs: ⏳ pending — Gate
Required contexts: summary
Required: core tests (3.12): ⏳ pending, core tests (3.13): ⏳ pending, docker smoke: ⏳ pending, gate: ⏳ pending

Workflow / Job Result Logs
(no jobs reported) ⏳ pending

Coverage Overview

  • Coverage history entries: 1

Coverage Trend

Metric Value
Current 76.21%
Baseline 85.00%
Delta -8.79%
Minimum 70.00%
Status ✅ Pass

Top Coverage Hotspots (lowest coverage)

File Coverage Missing
scripts/issue_dedup_smoke.py 0.0% 4
scripts/runner_lib/__main__.py 0.0% 3
scripts/validate_template_sync.py 0.0% 81
scripts/langchain/topic_splitter.py 19.1% 57
tools/codex_log_analyzer.py 19.6% 140
scripts/repo_review_round2_runner.py 25.1% 344
scripts/prune_agent_stubs.py 39.7% 26
scripts/repo_review_round1_runner.py 40.7% 133
tools/ensure_workflow_timeout_variables.py 42.1% 74
scripts/sync_label_docs.py 42.9% 64
tools/discover_model_catalog.py 44.8% 55
scripts/repo_review_backlog_scan.py 45.3% 116
scripts/repo_review_body_writer.py 46.5% 86
tools/codex_session_analyzer.py 47.9% 59
scripts/create_verifier_labels.py 48.3% 58

Low Coverage Files (<50.0%)

File Coverage Missing
scripts/issue_dedup_smoke.py 0.0% 4
scripts/runner_lib/__main__.py 0.0% 3
scripts/validate_template_sync.py 0.0% 81
scripts/langchain/topic_splitter.py 19.1% 57
tools/codex_log_analyzer.py 19.6% 140
scripts/repo_review_round2_runner.py 25.1% 344
scripts/prune_agent_stubs.py 39.7% 26
scripts/repo_review_round1_runner.py 40.7% 133
tools/ensure_workflow_timeout_variables.py 42.1% 74
scripts/sync_label_docs.py 42.9% 64
tools/discover_model_catalog.py 44.8% 55
scripts/repo_review_backlog_scan.py 45.3% 116
scripts/repo_review_body_writer.py 46.5% 86
tools/codex_session_analyzer.py 47.9% 59
scripts/create_verifier_labels.py 48.3% 58

Updated automatically; will refresh on subsequent CI/Docker completions.


Keepalive checklist

Scope

No scope information available

Tasks

  • No tasks defined

Acceptance criteria

  • No acceptance criteria defined

@stranske
stranske force-pushed the automation/sync-format-guard-review-round2 branch from b04ce02 to 03f7532 Compare August 8, 2026 19:32
@stranske
stranske temporarily deployed to agent-high-privilege August 8, 2026 19:33 — with GitHub Actions Inactive

@coderabbitai coderabbitai Bot left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Actionable comments posted: 2

🤖 Prompt for all review comments with AI agents
Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.

Inline comments:
In @.github/workflows/agents-issue-format-guard.yml:
- Around line 138-149: Update both revalidation blocks in
.github/workflows/agents-issue-format-guard.yml (lines 138-149) and
templates/consumer-repo/.github/workflows/agents-issue-format-guard.yml (lines
138-149) to capture validator stderr alongside the exit status. Treat exit code
1 with non-empty stderr as a validator runtime error, log it, and stop before
fingerprint calculation or optimizer dispatch; preserve success handling and
propagate all other non-zero exit codes.

In `@tests/workflows/test_agents_issue_optimizer_format_trigger.py`:
- Around line 107-121: Strengthen
test_format_guard_uses_event_inputs_and_fails_closed_on_revalidation_errors to
assert the complete revalidation routing branches: exit 0 skips dispatch, exit 1
continues, and any other exit code stops the step. Require the full fallback
expression, including github.event.issue.number, in each optimizer concurrency
configuration, rather than checking only partial fragments.
🪄 Autofix

Fix all unresolved CodeRabbit comments on this PR:

  • Push a commit to this branch (recommended)
  • Create a new PR with the fixes

ℹ️ Review info
⚙️ Run configuration

Configuration used: Path: .coderabbit.yaml

Review profile: ASSERTIVE

Plan: Pro

Run ID: 3fc6497d-33eb-4336-b70b-a7b0fc0c290b

📥 Commits

Reviewing files that changed from the base of the PR and between b389712 and 03f7532.

📒 Files selected for processing (6)
  • .github/workflows/agents-issue-format-guard.yml
  • .github/workflows/agents-issue-optimizer.yml
  • config/template-drift-allowlist.txt
  • templates/consumer-repo/.github/workflows/agents-issue-format-guard.yml
  • templates/consumer-repo/.github/workflows/agents-issue-optimizer.yml
  • tests/workflows/test_agents_issue_optimizer_format_trigger.py

Comment thread .github/workflows/agents-issue-format-guard.yml
Comment thread tests/workflows/test_agents_issue_optimizer_format_trigger.py Outdated
Treat exit 1 with non-empty stderr as a validator crash and fail closed
before fingerprinting or optimizer dispatch, matching the initial Validate
step. Strengthen regression assertions for the routing branches.

Co-authored-by: Cursor <cursoragent@cursor.com>
@stranske

stranske commented Aug 8, 2026

Copy link
Copy Markdown
Owner Author

Closer review recovery (head 979021da)

Addressed the active fail-closed revalidation threads (Codex/Copilot/CodeRabbit) that shared one root cause: exit code 1 is used for both non-conformance and Python crashes, so revalidation without stderr capture still dispatched the optimizer on a validator traceback.

Change (source + consumer template):

  • Capture validator stderr into $error_file during live-body revalidation
  • rc==0 → skip optimizer (body now conforms)
  • rc==1 + non-empty stderr → fail closed before fingerprint/dispatch
  • rc==1 + empty stderr → continue (normal non-conformance)
  • other non-zero → propagate after logging stderr

Validation: python -m pytest -q tests/workflows/test_agents_issue_optimizer_format_trigger.py tests/scripts/test_issue_format.py → 40 passed; scripts/check_template_drift.py → 0 unallowlisted; git diff --check clean.

Next: wait ≥7 minutes post-push review window on exact head 979021da, re-check unresolved threads + required checks, then merge when clean. PR still has missing-issue-warning (no linked issue); verify:* after merge remains unauthorized until an issue is linked or a non-issue Workflow Source is declared.

@stranske
stranske temporarily deployed to agent-high-privilege August 8, 2026 20:40 — with GitHub Actions Inactive
@stranske

stranske commented Aug 8, 2026

Copy link
Copy Markdown
Owner Author

Runner dispatch state for autofix on PR #2987. Do not edit.

@github-actions

github-actions Bot commented Aug 8, 2026

Copy link
Copy Markdown
Contributor

Runner dispatch state for codex on PR #2987. Do not edit.

@stranske
stranske temporarily deployed to agent-high-privilege August 8, 2026 20:44 — with GitHub Actions Inactive
@github-actions github-actions Bot added the autofix Opt-in automated formatting & lint remediation label Aug 8, 2026
@github-actions

github-actions Bot commented Aug 8, 2026

Copy link
Copy Markdown
Contributor

Autofix updated these files:

  • tests/workflows/test_agents_issue_optimizer_format_trigger.py

@agents-workflows-bot
agents-workflows-bot Bot temporarily deployed to agent-high-privilege August 8, 2026 20:45 Inactive
@stranske

stranske commented Aug 8, 2026

Copy link
Copy Markdown
Owner Author

Reviewed current head 3ecb0c812d1a98d45a646a7577cb493874b83371: both source and consumer revalidation blocks capture stderr, fail closed for rc==1 plus non-empty stderr before fingerprint/dispatch, and propagate other unexpected non-zero exits. Focused regression assertions cover this contract. Resolving the duplicate still-open review threads; Gate / gate and CodeRabbit are successful on this exact head.

@stranske
stranske merged commit fdc561a into main Aug 8, 2026
61 checks passed
@stranske
stranske deleted the automation/sync-format-guard-review-round2 branch August 8, 2026 21:24
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

autofix:escalated autofix:patch autofix Opt-in automated formatting & lint remediation

Projects

None yet

Development

Successfully merging this pull request may close these issues.

3 participants