fix(issues): harden formatter contract handling - #3004
Conversation
|
Bugbot is not enabled for your account, so this pull request was not reviewed. Enable Bugbot in the Cursor dashboard to get automatic reviews on future PRs. |
|
Note Reviews pausedIt looks like this branch is under active development. To avoid overwhelming you with review comments due to an influx of new commits, CodeRabbit has automatically paused this review. You can configure this behavior by changing the Use the following commands to manage reviews:
Use the checkboxes below for quick actions:
📝 WalkthroughWalkthroughThe PR tightens task and verification command validation, supports attributed ChangesIssue formatting validation
Estimated code review effort: 3 (Moderate) | ~25 minutes Sequence Diagram(s)sequenceDiagram
participant issue_formatter.py
participant agents-auto-pilot.yml
participant GitHubIssue
issue_formatter.py->>agents-auto-pilot.yml: Return needs_refinement and reason
agents-auto-pilot.yml->>GitHubIssue: Apply pause labels
agents-auto-pilot.yml->>GitHubIssue: Post refinement comment
agents-auto-pilot.yml-->>GitHubIssue: Skip body update and formatted marker
Possibly related PRs
Suggested labels: 🚥 Pre-merge checks | ✅ 4 | ❌ 1❌ Failed checks (1 warning)
✅ Passed checks (4 passed)
✨ Finishing Touches🧪 Generate unit tests (beta)
Comment |
GitHub Actions
Inactive
Workflow source neededPR #3004 needs either a linked GitHub issue or one valid non-issue Workflow Source before PR metadata automation can manage it safely. Please do one of:
Once a valid source is present, this warning will not be reposted. |
![chatgpt-codex-connector[bot]](https://avatars.githubusercontent.com/in/1144995?s=60&v=4){kind=small}
There was a problem hiding this comment.
💡 Codex Review
Here are some automated review suggestions for this pull request.
Reviewed commit: 134d91f1d1
ℹ️ About Codex in GitHub
Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you
- Open a pull request for review
- Mark a draft as ready
- Comment "@codex review".
If Codex has suggestions, it will comment; otherwise it will react with 👍.
Codex can also answer questions or update the PR. Try commenting "@codex address that feedback".
There was a problem hiding this comment.
Pull request overview
Hardens the issue formatting pipeline so that automated formatting doesn’t incorrectly “green light” output that fails the canonical issue-format contract, and tightens the validator so vague tasks aren’t treated as concrete.
Changes:
- Tighten the canonical issue validator to require command-shaped invocations (not just command words in prose) for task concreteness.
- Improve formatter fallback handling: recover pre-existing “Original Issue” wrappers, and only promote verify hints when they match a safe/contract-compliant shape.
- Propagate a final
needs_refinementsignal and pause auto-pilot (label + comment) instead of publishing non-conforming formatted output.
Reviewed changes
Copilot reviewed 5 out of 5 changed files in this pull request and generated 3 comments.
Show a summary per file
| File | Description |
|---|---|
| tests/scripts/test_issue_formatter.py | Adds regression tests for safe verify-hint promotion and for recovering attributed Original Issue <details> wrappers. |
| tests/scripts/test_issue_format.py | Adds coverage ensuring command words in prose aren’t treated as concrete task targets. |
| scripts/langchain/issue_formatter.py | Adds safe verify-hint gating, improves Original Issue wrapper recovery, and computes/returns needs_refinement from the final output validity. |
| .github/workflows/agents-auto-pilot.yml | Pauses auto-pilot when the formatter reports non-conforming final output (needs_refinement). |
| .github/scripts/issue_format.py | Strengthens task-target detection by requiring command-shaped invocations (and expands the command patterns). |
💡 Add a code-review agent skill or configure MCP servers for context-aware, tailored reviews. Learn more in the docs.
Automated Status SummaryHead SHA: 2e2cf1a
Coverage Overview
Coverage Trend
Top Coverage Hotspots (lowest coverage)
Low Coverage Files (<50.0%)
Updated automatically; will refresh on subsequent CI/Docker completions. Keepalive checklistScopeNo scope information available Tasks
Acceptance criteria
|
|
Runner dispatch state for autofix on PR #3004. Do not edit. |
GitHub Actions
Inactive
![coderabbitai[bot]](https://avatars.githubusercontent.com/in/347564?s=60&v=4){kind=small}
There was a problem hiding this comment.
Actionable comments posted: 2
🤖 Prompt for all review comments with AI agents
Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.
Inline comments:
In `@scripts/langchain/issue_formatter.py`:
- Line 772: Add regression tests covering the needs_refinement value produced by
the issue formatting workflow for invalid finalized output, valid finalized
output, and --json output. Verify invalid output sets it true, valid output sets
it false, and JSON output preserves the same contract without defaulting or
inverting the field; anchor the tests to _formatted_output_valid and the
relevant formatter entry point.
In `@tests/scripts/test_issue_formatter.py`:
- Around line 103-113: Update
test_format_issue_fallback_does_not_promote_shell_verify_hint to assert that the
extracted Acceptance Criteria section does not contain the generated “Run
`python3 -m pytest tests/test_x.py`” criterion, ensuring the entire
shell-bearing verify hint is rejected rather than partially promoted.
🪄 Autofix
Fix all unresolved CodeRabbit comments on this PR:
- Push a commit to this branch (recommended)
- Create a new PR with the fixes
ℹ️ Review info
⚙️ Run configuration
Configuration used: Path: .coderabbit.yaml
Review profile: ASSERTIVE
Plan: Pro
Run ID: d96072df-f942-4c5f-ba12-fe3161b6eb33
📒 Files selected for processing (6)
.github/scripts/issue_format.py.github/workflows/agents-auto-pilot.ymllangsmith-fleet-worker-attempt.jsonscripts/langchain/issue_formatter.pytests/scripts/test_issue_format.pytests/scripts/test_issue_formatter.py
Validate reused bodies before defaulting needs_refinement, stop auto-pilot redispatch after refinement pauses, require curl args for safe verify hints, accept backticked command targets, and mirror the refinement path in the consumer auto-pilot template to clear template drift.
stranske
added
agent:codex
Closer review recovery (cursor)Pushed exact head
Validation: focused pytest 3 passed (reuse needs_refinement, bare curl, backticked make test); earlier filtered suite 17 passed. Mandatory post-push 7-minute review window applies before merge. |
🤖 Keepalive Loop StatusPR #3004 | Agent: Codex | Iteration 0/12 Current State
🔍 Failure Classification| Error type | infrastructure |
|
GitHub Actions
Failure
|
Addressed the active CodeRabbit test-specificity finding in |
stranske
removed
needs-human
GitHub Actions
Inactive
stranske-keepalive
Bot
added
agent:needs-attention
Closer disposition — outdated Copilot curl threadResolved outdated unresolved thread Claim: Also clearing automation-loop |
stranske
removed
needs-human
stranske-keepalive
Bot
added
agent:needs-attention
Provider Comparison ReportProvider Summary
📋 Full Provider Details (click to expand)openai
anthropic
Agreement
Disagreement
Unique Insights
🔍 LangSmith Traces |
Fixes the shared source for the active consumer-sync review debt.
Validation:
python -m pytest -q tests/scripts/test_issue_formatter.py tests/scripts/test_issue_format.py tests/workflows/test_agents_issue_optimizer_format_trigger.pypython scripts/sync_tool_versions.py --checkpython scripts/sync_manifest_compiler.py --manifest .github/sync-manifest.yml --output-json /tmp/consumer-sync-plan.jsongit diff --checkSummary by CodeRabbit
Bug Fixes
Workflow Improvements
Tests