Skip to content

fix(agents): address sync review debt - #3005

Merged
stranske merged 11 commits into
mainfrom
agent/sync-review-debt-102
Aug 9, 2026
Merged

fix(agents): address sync review debt#3005
stranske merged 11 commits into
mainfrom
agent/sync-review-debt-102

Conversation

@stranske

@stranske stranske commented Aug 9, 2026

Copy link
Copy Markdown
Owner

Addresses shared active sync-review findings in the source of truth.

Changes:

  • align formatter verify hints with the canonical validator and ignore literal details tags inside fenced original-issue payloads
  • prevent skipped optimizer runs from releasing a format lease
  • keep manual issue-intake dispatch independent of issue payload labels
  • reject fenced examples as Tasks/Acceptance evidence

Validation:

  • python -m pytest -q tests/scripts/test_issue_formatter.py tests/workflows/test_agents_issue_optimizer_format_trigger.py tests/workflows/test_workflow_templates.py (47 passed)
  • git diff --check

Known validation debt: scripts/check_template_drift.py --allowlist config/template-drift-allowlist.txt still reports the pre-existing root/template divergence for issue-intake and issue-optimizer. This PR applies the targeted semantic changes to both surfaces; it intentionally does not overwrite consumer-specific workflow wiring.

Summary by CodeRabbit

  • Bug Fixes

    • Issue validation now ignores task checkboxes, acceptance criteria, headings, and details sections inside fenced Markdown examples.
    • Markdown fences with trailing text are handled correctly.
    • Issue intake workflows can run through manual dispatch without being blocked by the auto-pilot label.
  • Improvements

    • Verification supports additional unittest commands and GitHub CLI commands requiring arguments.
    • Added regression coverage for Markdown formatting and command validation scenarios.

Copilot AI lite review requested due to automatic review settings August 9, 2026 11:32
@cursor

cursor Bot commented Aug 9, 2026

Copy link
Copy Markdown

Bugbot is not enabled for your account, so this pull request was not reviewed.

Enable Bugbot in the Cursor dashboard to get automatic reviews on future PRs.

@stranske
stranske temporarily deployed to agent-high-privilege August 9, 2026 11:33 — with GitHub Actions Inactive
@coderabbitai

coderabbitai Bot commented Aug 9, 2026

Copy link
Copy Markdown

Review Change Stack

Warning

Review limit reached

You’ve reached a temporary PR review limit under our Fair Usage Limits Policy.

Your recent review volume is higher than typical usage, so adaptive limits are currently applied.

Next review available in: 3 minutes

Your organization has reached its usage spending cap. Adjust your spending cap in the billing tab.

How can I continue?

After more reviews become available, a review can be triggered using the @coderabbitai review command as a PR comment. Alternatively, push new commits to this PR.

To avoid repeated limits, reduce automatic review volume by pausing incremental auto-reviews earlier, using label-based review opt-in, excluding WIP or generated PR titles, or requesting reviews manually when the PR is ready. If your team needs uninterrupted high-volume reviews, an organization admin can enable usage-based reviews.

How do review limits work?

CodeRabbit enforces per-developer PR review limits for each organization. Most developers receive the normal plan review availability.

For paid Pro and Pro+ PR reviews, CodeRabbit uses adaptive limits for sustained high-volume activity. When a developer's recent PR review activity reaches the 95th percentile or higher among CodeRabbit users, additional reviews become available more gradually as earlier reviews age out of the rolling window.

Please refer docs for additional details.

Review details
⚙️ Run configuration

Configuration used: Path: .coderabbit.yaml

Review profile: ASSERTIVE

Plan: Pro

Run ID: d6e91560-d0e7-499c-8f77-a62af3b5263a

📥 Commits

Reviewing files that changed from the base of the PR and between 234722e and a849db5.

📒 Files selected for processing (3)
  • langsmith-fleet-worker-attempt.json
  • scripts/langchain/issue_formatter.py
  • tests/scripts/test_issue_formatter.py
📝 Walkthrough

Walkthrough

The PR excludes fenced Markdown from issue validation, expands safe verification-command matching, ignores fenced HTML during Original-Issue parsing, and limits auto-pilot label filtering to issue events. Tests cover fence parsing and command validation.

Changes

Issue validation and workflow gating

Layer / File(s) Summary
Fenced content validation
.github/scripts/issue_format.py, templates/consumer-repo/.github/scripts/issue_format.py, tests/scripts/test_issue_format.py
Fenced backtick and tilde content is removed before task checkbox and acceptance-criteria validation. Tests cover marker-only fence closures.
Formatter parsing and verification
scripts/langchain/issue_formatter.py, tests/scripts/test_issue_formatter.py
Safe verification matching supports unittest and argument-bearing gh commands. Original-Issue parsing ignores fenced <details> tags and requires matching marker-only closing fences.
Workflow trigger and metadata updates
.github/workflows/agents-63-issue-intake.yml, templates/consumer-repo/.github/workflows/agents-issue-intake.yml, config/template-drift-allowlist.txt, langsmith-fleet-worker-attempt.json
Auto-pilot exclusion applies only to issue events. The drift allowlist and worker-attempt record reflect the updated state.

Estimated code review effort: 3 (Moderate) | ~20 minutes

Possibly related PRs

Suggested labels: verify:compare

🚥 Pre-merge checks | ✅ 4 | ❌ 1

❌ Failed checks (1 warning)

Check name Status Explanation Resolution
Docstring Coverage ⚠️ Warning Docstring coverage is 56.25% which is insufficient. The required threshold is 80.00%. Write docstrings for the functions missing them to satisfy the coverage threshold.
✅ Passed checks (4 passed)
Check name Status Explanation
Description Check ✅ Passed Check skipped - CodeRabbit’s high-level summary is enabled.
Title check ✅ Passed The title accurately describes the pull request as a fix for synchronization-related review debt, although it does not list the specific behavior changes.
Linked Issues check ✅ Passed Check skipped because no linked issues were found for this pull request.
Out of Scope Changes check ✅ Passed Check skipped because no linked issues were found for this pull request.
✨ Finishing Touches
🧪 Generate unit tests (beta)
  • Create PR with unit tests
  • Commit unit tests in branch agent/sync-review-debt-102

Comment @coderabbitai help to get the list of available commands.

@coderabbitai coderabbitai Bot left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Actionable comments posted: 2

🤖 Prompt for all review comments with AI agents
Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.

Inline comments:
In @.github/scripts/issue_format.py:
- Around line 202-217: Update _without_fenced_code in
.github/scripts/issue_format.py (lines 202-217) and the corresponding fence
parser in scripts/langchain/issue_formatter.py (lines 480-496) so closing fences
are recognized only when the line contains the required marker characters with
optional surrounding whitespace and no trailing content; apply the same rule to
backtick and tilde fences, and add shared regression coverage for both marker
types with trailing text.

In @.github/workflows/agents-issue-optimizer.yml:
- Line 736: The cleanup conditions at
.github/workflows/agents-issue-optimizer.yml:736-736 and
templates/consumer-repo/.github/workflows/agents-issue-optimizer.yml:589-589
must detect format triggers independently of steps.check.outputs.should_run, so
agents:format is released when trigger validation fails or the workflow is
cancelled. Update only the agents:format cleanup logic in both workflows; leave
agents:auto-pilot and agents:auto-pilot-pause unchanged.
🪄 Autofix

Fix all unresolved CodeRabbit comments on this PR:

  • Push a commit to this branch (recommended)
  • Create a new PR with the fixes

ℹ️ Review info
⚙️ Run configuration

Configuration used: Path: .coderabbit.yaml

Review profile: ASSERTIVE

Plan: Pro

Run ID: 9c0f4aa7-c9c4-4c8b-bc4d-cd189212aa4c

📥 Commits

Reviewing files that changed from the base of the PR and between 102f1fb and c412301.

📒 Files selected for processing (6)
  • .github/scripts/issue_format.py
  • .github/workflows/agents-63-issue-intake.yml
  • .github/workflows/agents-issue-optimizer.yml
  • scripts/langchain/issue_formatter.py
  • templates/consumer-repo/.github/workflows/agents-issue-intake.yml
  • templates/consumer-repo/.github/workflows/agents-issue-optimizer.yml

Comment thread .github/scripts/issue_format.py
Comment thread .github/workflows/agents-issue-optimizer.yml Outdated

@chatgpt-codex-connector chatgpt-codex-connector Bot left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

💡 Codex Review

Here are some automated review suggestions for this pull request.

Reviewed commit: c412301cfd

ℹ️ About Codex in GitHub

Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you

  • Open a pull request for review
  • Mark a draft as ready
  • Comment "@codex review".

If Codex has suggestions, it will comment; otherwise it will react with 👍.

Codex can also answer questions or update the PR. Try commenting "@codex address that feedback".

Comment thread .github/scripts/issue_format.py Outdated

Copilot AI left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Pull request overview

This PR addresses shared sync-review findings across the Workflows “source of truth” and the consumer template surfaces, tightening issue-format validation, making issue-intake manual dispatch behavior consistent, and preventing unintended optimizer lease release behavior.

Changes:

  • Update issue-format validation to ignore Markdown fenced examples when evaluating Tasks and Acceptance Criteria.
  • Align issue_formatter verify-hint parsing with the canonical validator and make Original-Issue stripping robust to literal <details> tags inside fenced payloads.
  • Prevent skipped optimizer runs from releasing the agents:format lease; keep manual issue-intake dispatch independent of issue payload labels.

Reviewed changes

Copilot reviewed 6 out of 6 changed files in this pull request and generated 3 comments.

Show a summary per file
File Description
templates/consumer-repo/.github/workflows/agents-issue-optimizer.yml Only release a failed format lease when the optimizer check indicates the run should execute.
templates/consumer-repo/.github/workflows/agents-issue-intake.yml Ensure the agents:auto-pilot skip condition doesn’t block manual dispatch (non-issues events).
scripts/langchain/issue_formatter.py Expand/strengthen safe verify-hint matching and ignore <details> tags inside fenced Original-Issue payloads.
.github/workflows/agents-issue-optimizer.yml Mirror the optimizer lease-release gating in the source workflow.
.github/workflows/agents-63-issue-intake.yml Mirror the issue-intake manual-dispatch independence change in the source workflow.
.github/scripts/issue_format.py Reject fenced examples as evidence for Tasks/Acceptance by stripping fenced code before validation.

💡 Add a code-review agent skill or configure MCP servers for context-aware, tailored reviews. Learn more in the docs.

Comment thread .github/scripts/issue_format.py
Comment thread scripts/langchain/issue_formatter.py
Comment thread scripts/langchain/issue_formatter.py
…take drift

Align fence parsers with the heading scanner so language tags and trailing junk
cannot end a fence, keep cancel-safe format-lease release independent of
should_run, and refresh the intake drift allowlist for the dispatch-label change.

Co-authored-by: Cursor <cursoragent@cursor.com>
@stranske stranske added agent:codex Agent-created issues from Codex agents:keepalive Use to initiate keepalive functionality with agents autofix Opt-in automated formatting & lint remediation labels Aug 9, 2026
@stranske

stranske commented Aug 9, 2026

Copy link
Copy Markdown
Owner Author

Runner dispatch state for autofix on PR #3005. Do not edit.

@github-actions

github-actions Bot commented Aug 9, 2026

Copy link
Copy Markdown
Contributor

Runner dispatch state for codex on PR #3005. Do not edit.

@stranske-keepalive

stranske-keepalive Bot commented Aug 9, 2026

Copy link
Copy Markdown
Contributor

🤖 Keepalive Loop Status

PR #3005 | Agent: Codex | Iteration 0/12

Current State

Metric Value
Iteration progress [----------] 0/12
Action run (agent-run-failed)
Agent status ❌ AGENT FAILED
Gate success
Tasks 0/6 complete
Timeout 45 min (default)
Timeout usage 12m elapsed (27%, 33m remaining)
Keepalive ✅ enabled
Autofix ❌ disabled

Last Codex Run

Result Value
Status ❌ AGENT FAILED
Reason agent-run-failed
Exit code unknown
Failures 1/3 before pause

To retry immediately:

  • Add the agent:retry label to this PR

Or wait for the next successful Gate run to automatically retry.

🔍 Failure Classification

| Error type | infrastructure |
| Error category | transient |
| Suggested recovery | Capture logs and context; retry once and escalate if the issue persists. |

⚠️ Failure Tracking

| Consecutive failures | 1/3 |
| Reason | agent-run-failed |

@stranske-keepalive

stranske-keepalive Bot commented Aug 9, 2026

Copy link
Copy Markdown
Contributor
Keepalive Work Log (click to expand)
# Time (UTC) Agent Action Result Files Tasks Progress Commit Gate
0 2026-08-09 11:41:09 Codex wait (gate-cancelled-transient-transient) skipped 0 0/6 cancelled
0 2026-08-09 11:43:49 Codex run (agent-run-failed) failure 2 file(s) 0 0/6 38163e4 cancelled
0 2026-08-09 11:44:28 Codex wait (gate-cancelled-transient-transient) skipped 0 0/6 cancelled
0 2026-08-09 11:49:38 Codex run (agent-run-failed) failure 2 file(s) 0 0/6 85ac8b5 cancelled
0 2026-08-09 11:53:27 Codex run (agent-run-failed) failure 2 file(s) 0 0/6 234722e cancelled
0 2026-08-09 11:56:49 Codex run (agent-run-failed-repeat) failure 2 file(s) 0 0/6 29d6906 cancelled
0 2026-08-09 11:57:36 Codex skip (needs-human) skipped 0 0/6
0 2026-08-09 11:58:15 Codex skip (needs-human) skipped 0 0/6 cancelled
0 2026-08-09 12:01:53 Codex skip (needs-human) skipped 0 0/6 success
0 2026-08-09 12:31:29 Codex run (agent-run-failed-repeat) failure 2 file(s) 0 0/6 bc47c34 success
0 2026-08-09 12:32:15 Codex skip (needs-human) skipped 0 0/6 cancelled
0 2026-08-09 12:32:52 Codex skip (needs-human) skipped 0 0/6 cancelled
0 2026-08-09 12:33:56 Codex skip (needs-human) skipped 0 0/6 cancelled
0 2026-08-09 12:39:50 Codex run (agent-run-failed-repeat) retry failure 2 file(s) 0 0/6 a849db5
0 2026-08-09 12:40:31 Codex skip (needs-human) skipped 0 0/6
0 2026-08-09 12:41:09 Codex skip (needs-human) skipped 0 0/6 cancelled
0 2026-08-09 12:42:01 Codex skip (needs-human) skipped 0 0/6 cancelled
0 2026-08-09 12:45:35 Codex skip (needs-human) skipped 0 0/6 success
0 2026-08-11 09:42:02 Codex wait (gate-pending-transient) skipped 0 0/6
0 2026-08-11 09:53:41 Codex run (agent-run-failed) failure 1 file(s) 0 0/6 dc0b760 success

@stranske
stranske had a problem deploying to agent-high-privilege August 9, 2026 11:41 — with GitHub Actions Error
codex-automation and others added 2 commits August 9, 2026 06:41
Add regression coverage for unittest/gh verify hints so SAFE_VERIFY_COMMAND_RE
edits cannot silently widen or narrow the accepted command surface.

Co-authored-by: Cursor <cursoragent@cursor.com>

@coderabbitai coderabbitai Bot left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Actionable comments posted: 1

🤖 Prompt for all review comments with AI agents
Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.

Inline comments:
In `@scripts/langchain/issue_formatter.py`:
- Around line 492-501: Update the fence-state logic in
scripts/langchain/issue_formatter.py lines 492-501 to scan through tag.end()
before evaluating whether the line is a marker-only closing fence, so same-line
HTML such as </details> prevents closing the active fence. Add the requested
parameterized regression case in tests/scripts/test_issue_formatter.py lines
706-740 using f"{marker}</details>", followed by a real closing fence and
details tag.
🪄 Autofix

Fix all unresolved CodeRabbit comments on this PR:

  • Push a commit to this branch (recommended)
  • Create a new PR with the fixes

ℹ️ Review info
⚙️ Run configuration

Configuration used: Path: .coderabbit.yaml

Review profile: ASSERTIVE

Plan: Pro

Run ID: fd7b178c-5ec0-494b-a28d-d534a3bb3788

📥 Commits

Reviewing files that changed from the base of the PR and between c412301 and 38163e4.

📒 Files selected for processing (7)
  • .github/scripts/issue_format.py
  • config/template-drift-allowlist.txt
  • langsmith-fleet-worker-attempt.json
  • scripts/langchain/issue_formatter.py
  • templates/consumer-repo/.github/scripts/issue_format.py
  • tests/scripts/test_issue_format.py
  • tests/scripts/test_issue_formatter.py

Comment thread scripts/langchain/issue_formatter.py
@agents-workflows-bot
agents-workflows-bot Bot temporarily deployed to agent-high-privilege August 9, 2026 11:47 Inactive
@agents-workflows-bot
agents-workflows-bot Bot temporarily deployed to agent-high-privilege August 9, 2026 11:50 Inactive
@agents-workflows-bot
agents-workflows-bot Bot temporarily deployed to agent-high-privilege August 9, 2026 11:54 Inactive
@agents-workflows-bot agents-workflows-bot Bot added the agent:needs-attention Agent needs human review or intervention label Aug 9, 2026
@agents-workflows-bot
agents-workflows-bot Bot temporarily deployed to agent-high-privilege August 9, 2026 12:31 Inactive
@stranske-keepalive stranske-keepalive Bot added agent:needs-attention Agent needs human review or intervention needs-human Requires human intervention or review labels Aug 9, 2026
@stranske stranske removed needs-human Requires human intervention or review agent:needs-attention Agent needs human review or intervention labels Aug 9, 2026
@stranske

stranske commented Aug 9, 2026

Copy link
Copy Markdown
Owner Author

Closer audit (cursor, 2026-08-09T12:36Z):

  • Head advanced by keepalive/autofix after the fence fix: 5bd559f (autofix only touched langsmith-fleet-worker-attempt.json).
  • Active non-outdated unresolved review threads: 0.
  • Required Gate status was stale-ERROR from cancelled run on prior SHA bc47c348; fresh Gate run 31313570310 is pending on exact current head.
  • needs-human / agent:needs-attention reappeared from the cancelled-Gate / automation loop, not a documented human decision — removed again. Retained agent:codex, agents:keepalive, autofix.
  • Post-push review window for this head started 2026-08-09T12:32:59Z (merge not before 12:39:59Z) plus green Gate on the same OID.

@agents-workflows-bot
agents-workflows-bot Bot temporarily deployed to agent-high-privilege August 9, 2026 12:37 Inactive
@stranske-keepalive stranske-keepalive Bot added agent:needs-attention Agent needs human review or intervention needs-human Requires human intervention or review labels Aug 9, 2026
@stranske-keepalive
stranske-keepalive Bot temporarily deployed to agent-high-privilege August 9, 2026 12:40 Inactive
@stranske stranske removed needs-human Requires human intervention or review agent:needs-attention Agent needs human review or intervention labels Aug 9, 2026
@stranske
stranske merged commit 4b20bd3 into main Aug 9, 2026
82 of 84 checks passed
@stranske
stranske deleted the agent/sync-review-debt-102 branch August 9, 2026 13:22
@stranske stranske added the verify:compare Compare multiple LLM evaluations label Aug 11, 2026
@stranske
stranske temporarily deployed to agent-high-privilege August 11, 2026 09:41 — with GitHub Actions Inactive
@github-actions

Copy link
Copy Markdown
Contributor

Provider Comparison Report

Provider Summary

Provider Model Verdict Confidence Summary
openai gpt-5.6-terra CONCERNS 84% The issue-format validation changes are well-structured: fenced Markdown content is removed before Tasks and Acceptance Criteria evidence is evaluated, preventing examples from satisfying requireme...
anthropic claude-sonnet-5 CONCERNS N/A Review the PR manually or re-run once LLM credentials are available.
📋 Full Provider Details (click to expand)

openai

  • Model: gpt-5.6-terra
  • Verdict: CONCERNS
  • Confidence: 84%
  • Scores:
    • Correctness: 7.0/10
    • Completeness: 7.0/10
    • Quality: 8.0/10
    • Testing: 8.0/10
    • Risks: 8.0/10
  • Summary: The issue-format validation changes are well-structured: fenced Markdown content is removed before Tasks and Acceptance Criteria evidence is evaluated, preventing examples from satisfying requirements. The implementation correctly tracks backtick/tilde fence marker type and length, and the root/template validator copies are kept aligned. The intake workflow condition also correctly scopes the auto-pilot-label exclusion to issue events, making workflow_dispatch independent of an issue payload. Formatter-focused tests were added and the template-drift allowlist was rebased consistently. However, the PR diff does not show a corresponding optimizer format-trigger or lease-release change, leaving one explicit task unaddressed or at least unverifiable from the merged code changes.
  • Concerns:
    • The stated requirement to prevent skipped optimizer runs from releasing a format lease is not represented by any changed optimizer/format-trigger workflow or lease-management implementation. The changed workflow files only alter issue-intake label handling, and the changed Python files are issue-format validators/formatters. Based on the supplied changed-file list, this requirement is not demonstrably implemented.
    • No modified test under tests/workflows/ is included for the skipped-optimizer/format-lease behavior. The acceptance test path is referenced, but the PR's added tests are limited to issue-format and issue-formatter behavior, so there is no new regression coverage demonstrating the lease requirement.

anthropic

  • Model: claude-sonnet-5
  • Verdict: CONCERNS
  • Confidence: N/A
  • Summary: Review the PR manually or re-run once LLM credentials are available.
  • Concerns:
    • LLM evaluation could not run.
  • Error: LLM invocation failed: Error code: 400 - {'type': 'error', 'error': {'type': 'invalid_request_error', 'message': 'Your credit balance is too low to access the Anthropic API. Please go to Plans & Billing to upgrade or purchase credits.'}, 'request_id': 'req_011CdvkYLDtqHRNWcSCxyPLc'}

Agreement

  • Verdict: CONCERNS (all providers)

Disagreement

No major disagreements detected.

Unique Insights

  • openai: The stated requirement to prevent skipped optimizer runs from releasing a format lease is not represented by any changed optimizer/format-trigger workflow or lease-management implementation. The changed workflow files only alter issue-intake label handling, and the changed Python files are issue-format validators/formatters. Based on the supplied changed-file list, this requirement is not demonstrably implemented.; No modified test under tests/workflows/ is included for the skipped-optimizer/format-lease behavior. The acceptance test path is referenced, but the PR's added tests are limited to issue-format and issue-formatter behavior, so there is no new regression coverage demonstrating the lease requirement.
  • anthropic: LLM evaluation could not run.

🔍 LangSmith Traces

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

agent:codex Agent-created issues from Codex agents:keepalive Use to initiate keepalive functionality with agents autofix Opt-in automated formatting & lint remediation verify:compare Compare multiple LLM evaluations

Projects

None yet

Development

Successfully merging this pull request may close these issues.

3 participants