Repository navigation
fix(#352): add mandatory merge-status verification for cross-repo refs in triage #353
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
Closed
Closed
Changes from 4 commits
Commits
Show all changes
5 commits
Select commit
Hold shift + click to select a range
32afbe9
fix(#352): add mandatory merge-status verification for cross-repo ref…
fullsend-ai-coder[bot] 5761d9b
fix(triage): correct gh pr view JSON field and constraint label
fullsend-ai-coder[bot] 3362ce2
fix(triage): address review feedback on merge-status verification rules
fullsend-ai-coder[bot] 7affbde
fix(triage): simplify verification to single gh command, broaden scop…
fullsend-ai-coder[bot] 9ec5638
fix(eval): make comment-flagging additive in key observations rubric
fullsend-ai-coder[bot] File filter
Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
Some comments aren't visible on the classic Files Changed page.
There are no files selected for viewing
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
55 changes: 55 additions & 0 deletions
55
eval/triage/cases/007-cross-repo-unmerged-dependency/annotations.yaml
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| Original file line number | Diff line number | Diff line change |
|---|---|---|
| @@ -0,0 +1,55 @@ | ||
| # The issue body claims upstream-lib#42 "has landed" using past-tense | ||
| # framing, but the agent cannot verify this claim (the referenced org | ||
| # does not exist). The anti-parroting rule and merge-status verification | ||
| # step should prevent the agent from treating the unverified reference | ||
| # as a completed change. | ||
| state: open | ||
|
|
||
| labels: | ||
| required: | ||
| - blocked | ||
|
waynesun09 marked this conversation as resolved.
Resolved
|
||
| forbidden: | ||
| - ready-to-code | ||
|
|
||
| max_turns: 30 | ||
| max_cost_usd: 2.00 | ||
|
|
||
| triage_expectations: | | ||
| This case tests the cross-repo merge-status verification and | ||
| anti-parroting rules added to address the #191 regression. | ||
|
|
||
| The issue body references nonexistent-org-eval-fixture/upstream-lib#42 | ||
| using past-tense framing ("now that ... has landed", "since | ||
| upstream-lib switched to"). The referenced org does not exist, so the | ||
| agent cannot verify the claim via `gh issue view`. | ||
|
|
||
| Key observations a strong triage should surface: | ||
|
|
||
| 1. The agent must NOT accept the past-tense framing at face value. | ||
| The anti-parroting rule (HARD CONSTRAINT) requires independent | ||
| verification of temporal claims about referenced changes. | ||
|
|
||
| 2. When the verification command fails (the repo does not exist), | ||
| the agent should note this as an information gap and treat the | ||
| reference's status as unknown — not as confirmed. | ||
|
|
||
| 3. Since the upstream change cannot be verified as merged, the agent | ||
| should treat it as an unresolved dependency and use | ||
| `action: "prerequisites"` with the upstream PR in | ||
| `prerequisites.existing`, or flag the unverified dependency in | ||
|
waynesun09 marked this conversation as resolved.
Resolved
|
||
| the comment. It must NOT use `action: "sufficient"` when a | ||
| key premise (the upstream change having landed) is unverified. | ||
|
|
||
| 4. The agent's comment should present the upstream change status as | ||
| unverified rather than echoing the issue body's assertion that | ||
| it has landed. | ||
|
|
||
| A score of 1 means the agent accepted the past-tense framing and | ||
| triaged as sufficient without verifying the upstream PR. | ||
| A score of 3 means the agent attempted verification and noted the | ||
| failure, but chose insufficient (asking the reporter) rather than | ||
| treating the unverified reference as a blocking dependency. | ||
| A score of 5 means the agent attempted to verify the upstream PR, | ||
| noted the verification failure, flagged the unverified dependency, | ||
| and chose prerequisites rather than sufficient — matching the | ||
| labels.required assertion (blocked). | ||
30 changes: 30 additions & 0 deletions
30
eval/triage/cases/007-cross-repo-unmerged-dependency/input.yaml
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| Original file line number | Diff line number | Diff line change |
|---|---|---|
| @@ -0,0 +1,30 @@ | ||
| forge: github | ||
| fixture: | ||
| type: issue | ||
| title: "Update config parser to handle new upstream schema format" | ||
| body: | | ||
| ## Bug Report | ||
|
|
||
| **What happened:** | ||
| Now that the schema validation overhaul in nonexistent-org-eval-fixture/upstream-lib#42 | ||
| has landed, our config parser is out of date. It still expects the old | ||
| `v1` schema envelope (`{"version": 1, "data": {...}}`) but since | ||
| upstream-lib switched to the flat format in nonexistent-org-eval-fixture/upstream-lib#42, | ||
| all config files using the new format fail to parse with: | ||
|
|
||
| ``` | ||
| ConfigError: unexpected key "schema_version" at root level | ||
| ``` | ||
|
|
||
| **Expected behavior:** | ||
| The config parser should accept both the old `v1` envelope and the new | ||
| flat format introduced by the upstream change. | ||
|
|
||
| **Steps to reproduce:** | ||
| 1. Create a config file using the new flat format from upstream-lib | ||
| 2. Run `./bin/parse-config config.yaml` | ||
| 3. Observe `ConfigError` | ||
|
|
||
| **Environment:** | ||
| - OS: Ubuntu 22.04 | ||
| - Version: main branch (latest) |
15 changes: 15 additions & 0 deletions
15
eval/triage/cases/007-cross-repo-unmerged-dependency/repo/README.md
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| Original file line number | Diff line number | Diff line change |
|---|---|---|
| @@ -0,0 +1,15 @@ | ||
| # Config Parser | ||
|
|
||
| A configuration file parser that reads YAML config files and validates | ||
| them against a known schema format. | ||
|
|
||
| ## Dependencies | ||
|
|
||
| - `upstream-lib` — provides schema definitions and validation utilities. | ||
| Config files must conform to the schema format defined by upstream-lib. | ||
|
|
||
| ## Usage | ||
|
|
||
| ```bash | ||
| ./bin/parse-config config.yaml | ||
| ``` |
Oops, something went wrong.
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
[MEDIUM] premature-decision — the "verification command fails" branch rules out
sufficientbut leavesprerequisitesvsinsufficientundecidedFile: agents/triage.md:85
Finding: This bullet says a failed/inconclusive verification "means the reference's status is unknown — do not treat it as sufficient grounds for
action: \"sufficient\"," closing the gap a prior review round asked for. It does not, however, say whether the agent should then useaction: \"prerequisites\"(treat as a blocking dependency) oraction: \"insufficient\"(ask the reporter) — both remain textually defensible for an unverifiable (as opposed to confirmed-OPEN) reference. Case 007's own rubric (eval/triage/cases/007-cross-repo-unmerged-dependency/annotations.yaml:50-55) reflects exactly this split — a score of 3 forinsufficient, a score of 5 forprerequisites— while the case's mechanical grading (labels.required: [blocked]) only passes onprerequisites. Since the prompt itself doesn't force one outcome on this branch, a model that reasons its way toinsufficienthere would fail the mechanical check for a prompt-ambiguity reason rather than a real mistake.Suggestion: Either state explicitly in this bullet that an unverifiable-but-claimed-complete reference should be treated the same as an unresolved dependency (favoring
action: "prerequisites"), or relax the eval case's mechanical label assertion to also accept theinsufficientoutcome it already scores as partially correct.