Skip to content

[codex] Fix sync review source regressions #1909

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Merged
stranske merged 1 commit into from
Apr 26, 2026
Merged

[codex] Fix sync review source regressions #1909

Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
1 commit
Select commit
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
24 changes: 12 additions & 12 deletions .github/workflows/agents-verify-to-new-pr.yml
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -32,7 +32,7 @@ jobs:
steps:
- name: Check PR is merged
id: check-merged
uses: actions/github-script@v9
uses: actions/github-script@3a2844b7e9c422d3c10d287c895573f7108da1b3 # v9

Copilot AI Apr 26, 2026

Copy link

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This workflow now pins some actions by commit SHA, but actions/setup-python@v6 is still referenced by a floating tag later in the job. The repo’s action-pinning policy calls for SHA pinning to avoid tag retags changing workflow behavior (docs/ci/ACTIONS_PINNING.md). Please pin setup-python to a commit SHA as well for consistency and supply-chain integrity.

Copilot uses AI. Check for mistakes.
with:
github-token: ${{ github.token }}
script: |
Expand Down Expand Up @@ -66,7 +66,7 @@ jobs:

- name: Checkout repository
if: steps.check-merged.outputs.merged == 'true'
uses: actions/checkout@v6
uses: actions/checkout@de0fac2e4500dabe0009e67214ff5f5447ce83dd # v6
with:
repository: stranske/Workflows
token: ${{ steps.select-token.outputs.token }}
Expand Down Expand Up @@ -103,7 +103,7 @@ jobs:
- name: Collect verification and original issue data
id: collect
if: steps.check-merged.outputs.merged == 'true'
uses: actions/github-script@v9
uses: actions/github-script@3a2844b7e9c422d3c10d287c895573f7108da1b3 # v9
with:
github-token: ${{ steps.select-token.outputs.token }}
script: |
Expand Down Expand Up @@ -248,7 +248,7 @@ jobs:
- name: Check chain depth limit
id: chain-check
if: steps.check-merged.outputs.merged == 'true'
uses: actions/github-script@v9
uses: actions/github-script@3a2844b7e9c422d3c10d287c895573f7108da1b3 # v9
env:
CHAIN_DEPTH: ${{ steps.collect.outputs.chain_depth }}
MAX_CHAIN_DEPTH: '2'
Expand Down Expand Up @@ -389,7 +389,7 @@ jobs:
steps.check-merged.outputs.merged == 'true' &&
steps.chain-check.outputs.exceeded != 'true' &&
steps.extract-verdict.outputs.needs_human == 'true'
uses: actions/github-script@v9
uses: actions/github-script@3a2844b7e9c422d3c10d287c895573f7108da1b3 # v9
env:
LINKED_ISSUE: ${{ steps.collect.outputs.original_issue_number }}
NEEDS_HUMAN_REASON: ${{ steps.extract-verdict.outputs.needs_human_reason }}
Expand Down Expand Up @@ -505,7 +505,7 @@ jobs:
steps.check-merged.outputs.merged == 'true' &&
steps.chain-check.outputs.exceeded != 'true' &&
steps.extract-verdict.outputs.needs_human != 'true'
uses: actions/github-script@v9
uses: actions/github-script@3a2844b7e9c422d3c10d287c895573f7108da1b3 # v9
env:
FOLLOW_UP_DEPTH: ${{ steps.chain-check.outputs.next_depth }}
EXTRACTED_VERDICT: ${{ steps.extract-verdict.outputs.verdict }}
Expand Down Expand Up @@ -677,7 +677,7 @@ jobs:
steps.check-merged.outputs.merged == 'true' &&
steps.chain-check.outputs.exceeded != 'true' &&
steps.extract-verdict.outputs.needs_human != 'true'
uses: actions/github-script@v9
uses: actions/github-script@3a2844b7e9c422d3c10d287c895573f7108da1b3 # v9
env:
ISSUE_TITLE: >-
${{ steps.generate.outputs.issue_title ||
Expand Down Expand Up @@ -779,7 +779,7 @@ jobs:
steps.chain-check.outputs.exceeded != 'true' &&
steps.extract-verdict.outputs.needs_human != 'true' &&
steps.create-issue.outputs.issue_number != ''
uses: actions/github-script@v9
uses: actions/github-script@3a2844b7e9c422d3c10d287c895573f7108da1b3 # v9
env:
ISSUE_NUMBER: ${{ steps.create-issue.outputs.issue_number }}
DEFAULT_BRANCH: ${{ github.event.repository.default_branch }}
Expand Down Expand Up @@ -816,7 +816,7 @@ jobs:
steps.check-merged.outputs.merged == 'true' &&
steps.chain-check.outputs.exceeded != 'true' &&
steps.extract-verdict.outputs.needs_human != 'true'
uses: actions/github-script@v9
uses: actions/github-script@3a2844b7e9c422d3c10d287c895573f7108da1b3 # v9
env:
ISSUE_NUMBER: ${{ steps.create-issue.outputs.issue_number }}
ISSUE_URL: ${{ steps.create-issue.outputs.issue_url }}
Expand Down Expand Up @@ -870,7 +870,7 @@ jobs:
FOLLOWUP_ISSUE_NUMBER: ${{ steps.create-issue.outputs.issue_number }}
FOLLOWUP_ISSUE_URL: ${{ steps.create-issue.outputs.issue_url }}
DISPATCH_AUTOPILOT_OUTCOME: ${{ steps.dispatch-autopilot.outcome }}
uses: actions/github-script@v9
uses: actions/github-script@3a2844b7e9c422d3c10d287c895573f7108da1b3 # v9
with:
github-token: ${{ steps.select-token.outputs.token }}
script: |
Expand Down Expand Up @@ -1006,7 +1006,7 @@ jobs:

- name: Upload terminal disposition artifact
if: always() && steps.check-merged.outputs.merged == 'true'
uses: actions/upload-artifact@v7
uses: actions/upload-artifact@043fb46d1a93c77aae656e7c1c64a875d1fc6a0a # v7
with:
name: verifier-terminal-disposition-${{ github.run_id }}
path: |
Expand All @@ -1017,7 +1017,7 @@ jobs:

- name: Remove trigger label
if: steps.check-merged.outputs.merged == 'true'
uses: actions/github-script@v9
uses: actions/github-script@3a2844b7e9c422d3c10d287c895573f7108da1b3 # v9
continue-on-error: true
with:
github-token: ${{ steps.select-token.outputs.token }}
Expand Down
12 changes: 11 additions & 1 deletion scripts/aggregate_agent_metrics.py
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -194,7 +194,17 @@ def _append_parse_error_detail(
details[index] = replace(existing, count=existing.count + detail.count)
return

details.append(replace(detail, line=None))
for index, existing in enumerate(details):
if (
existing.path == detail.path
and existing.artifact == detail.artifact
and existing.artifact_family == detail.artifact_family
and existing.reason == detail.reason
):
details[index] = replace(existing, line=None, count=existing.count + detail.count)
return

details[-1] = replace(detail, line=None, count=details[-1].count + detail.count)

@chatgpt-codex-connector chatgpt-codex-connector Bot Apr 26, 2026

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

P2 Badge Preserve counts when compaction bucket is missing

When the detail buffer is full and a new error reason appears that is not already compacted, this branch overwrites details[-1] with the new reason while adding the overwritten row's count. That reassigns existing counts to the wrong reason/path bucket instead of just accounting for the new overflow event. A concrete case is many invalid-json rows followed by legacy-json-fallback-buffer-limit: the code can shift one invalid-json count into the legacy reason, which corrupts by_reason/detail reporting.

Useful? React with 👍 / 👎.

Copilot AI Apr 26, 2026

Copy link

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

When the parse-error detail list is at the storage limit and the incoming detail does not match any existing entry, the fallback details[-1] = replace(detail, ..., count=details[-1].count + detail.count) reassigns the previous last entry’s count to the new detail’s (path/artifact/reason). This can corrupt the by-reason/by-artifact counters (e.g., a large invalid-json count can be shifted into legacy-json-fallback-buffer-limit just because it was appended last). Consider aggregating the overflow count into an existing entry without changing its key, or using a dedicated sentinel/overflow bucket, so existing counts aren’t reclassified when the limit is hit.

Suggested change
details[-1] = replace(detail, line=None, count=details[-1].count + detail.count)
details[-1] = replace(details[-1], line=None, count=details[-1].count + detail.count)

Copilot uses AI. Check for mistakes.


def _parse_error_count(parse_error_details: list[ParseErrorDetail]) -> int:
Expand Down
6 changes: 1 addition & 5 deletions scripts/repo_review_evaluator.py
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -470,11 +470,7 @@ def build_review_execution(state: dict[str, Any]) -> dict[str, Any]:
repo_path = Path(state["local_path"])
tracked_files = tracked_repo_files(repo_path)
evidence_files = [path for path in tracked_files if not is_evidence_noise_file(path)]
implementation_files = [
path
for path in evidence_files
if is_implementation_file(path)
]
implementation_files = [path for path in evidence_files if is_implementation_file(path)]
test_files = [path for path in evidence_files if is_test_file(path)]
implementation_scan_files = implementation_files[:REVIEW_SCAN_FILE_LIMIT]
test_scan_files = test_files[:REVIEW_SCAN_FILE_LIMIT]
Expand Down
24 changes: 12 additions & 12 deletions templates/consumer-repo/.github/workflows/agents-verify-to-new-pr.yml
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -32,7 +32,7 @@ jobs:
steps:
- name: Check PR is merged
id: check-merged
uses: actions/github-script@v9
uses: actions/github-script@3a2844b7e9c422d3c10d287c895573f7108da1b3 # v9

Copilot AI Apr 26, 2026

Copy link

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This workflow now pins some actions by commit SHA, but actions/setup-python@v6 is still referenced by a floating tag later in the job. The repo’s action-pinning policy calls for SHA pinning to avoid tag retags changing workflow behavior (docs/ci/ACTIONS_PINNING.md). Please pin setup-python to a commit SHA as well for consistency and supply-chain integrity.

Copilot uses AI. Check for mistakes.
with:
github-token: ${{ github.token }}
script: |
Expand Down Expand Up @@ -66,7 +66,7 @@ jobs:

- name: Checkout repository
if: steps.check-merged.outputs.merged == 'true'
uses: actions/checkout@v6
uses: actions/checkout@de0fac2e4500dabe0009e67214ff5f5447ce83dd # v6
with:
repository: stranske/Workflows
token: ${{ steps.select-token.outputs.token }}
Expand Down Expand Up @@ -103,7 +103,7 @@ jobs:
- name: Collect verification and original issue data
id: collect
if: steps.check-merged.outputs.merged == 'true'
uses: actions/github-script@v9
uses: actions/github-script@3a2844b7e9c422d3c10d287c895573f7108da1b3 # v9
with:
github-token: ${{ steps.select-token.outputs.token }}
script: |
Expand Down Expand Up @@ -248,7 +248,7 @@ jobs:
- name: Check chain depth limit
id: chain-check
if: steps.check-merged.outputs.merged == 'true'
uses: actions/github-script@v9
uses: actions/github-script@3a2844b7e9c422d3c10d287c895573f7108da1b3 # v9
env:
CHAIN_DEPTH: ${{ steps.collect.outputs.chain_depth }}
MAX_CHAIN_DEPTH: '2'
Expand Down Expand Up @@ -389,7 +389,7 @@ jobs:
steps.check-merged.outputs.merged == 'true' &&
steps.chain-check.outputs.exceeded != 'true' &&
steps.extract-verdict.outputs.needs_human == 'true'
uses: actions/github-script@v9
uses: actions/github-script@3a2844b7e9c422d3c10d287c895573f7108da1b3 # v9
env:
LINKED_ISSUE: ${{ steps.collect.outputs.original_issue_number }}
NEEDS_HUMAN_REASON: ${{ steps.extract-verdict.outputs.needs_human_reason }}
Expand Down Expand Up @@ -505,7 +505,7 @@ jobs:
steps.check-merged.outputs.merged == 'true' &&
steps.chain-check.outputs.exceeded != 'true' &&
steps.extract-verdict.outputs.needs_human != 'true'
uses: actions/github-script@v9
uses: actions/github-script@3a2844b7e9c422d3c10d287c895573f7108da1b3 # v9
env:
FOLLOW_UP_DEPTH: ${{ steps.chain-check.outputs.next_depth }}
EXTRACTED_VERDICT: ${{ steps.extract-verdict.outputs.verdict }}
Expand Down Expand Up @@ -677,7 +677,7 @@ jobs:
steps.check-merged.outputs.merged == 'true' &&
steps.chain-check.outputs.exceeded != 'true' &&
steps.extract-verdict.outputs.needs_human != 'true'
uses: actions/github-script@v9
uses: actions/github-script@3a2844b7e9c422d3c10d287c895573f7108da1b3 # v9
env:
ISSUE_TITLE: >-
${{ steps.generate.outputs.issue_title ||
Expand Down Expand Up @@ -779,7 +779,7 @@ jobs:
steps.chain-check.outputs.exceeded != 'true' &&
steps.extract-verdict.outputs.needs_human != 'true' &&
steps.create-issue.outputs.issue_number != ''
uses: actions/github-script@v9
uses: actions/github-script@3a2844b7e9c422d3c10d287c895573f7108da1b3 # v9
env:
ISSUE_NUMBER: ${{ steps.create-issue.outputs.issue_number }}
DEFAULT_BRANCH: ${{ github.event.repository.default_branch }}
Expand Down Expand Up @@ -816,7 +816,7 @@ jobs:
steps.check-merged.outputs.merged == 'true' &&
steps.chain-check.outputs.exceeded != 'true' &&
steps.extract-verdict.outputs.needs_human != 'true'
uses: actions/github-script@v9
uses: actions/github-script@3a2844b7e9c422d3c10d287c895573f7108da1b3 # v9
env:
ISSUE_NUMBER: ${{ steps.create-issue.outputs.issue_number }}
ISSUE_URL: ${{ steps.create-issue.outputs.issue_url }}
Expand Down Expand Up @@ -870,7 +870,7 @@ jobs:
FOLLOWUP_ISSUE_NUMBER: ${{ steps.create-issue.outputs.issue_number }}
FOLLOWUP_ISSUE_URL: ${{ steps.create-issue.outputs.issue_url }}
DISPATCH_AUTOPILOT_OUTCOME: ${{ steps.dispatch-autopilot.outcome }}
uses: actions/github-script@v9
uses: actions/github-script@3a2844b7e9c422d3c10d287c895573f7108da1b3 # v9
with:
github-token: ${{ steps.select-token.outputs.token }}
script: |
Expand Down Expand Up @@ -1006,7 +1006,7 @@ jobs:

- name: Upload terminal disposition artifact
if: always() && steps.check-merged.outputs.merged == 'true'
uses: actions/upload-artifact@v7
uses: actions/upload-artifact@043fb46d1a93c77aae656e7c1c64a875d1fc6a0a # v7
with:
name: verifier-terminal-disposition-${{ github.run_id }}
path: |
Expand All @@ -1017,7 +1017,7 @@ jobs:

- name: Remove trigger label
if: steps.check-merged.outputs.merged == 'true'
uses: actions/github-script@v9
uses: actions/github-script@3a2844b7e9c422d3c10d287c895573f7108da1b3 # v9
continue-on-error: true
with:
github-token: ${{ steps.select-token.outputs.token }}
Expand Down
12 changes: 11 additions & 1 deletion templates/consumer-repo/scripts/aggregate_agent_metrics.py
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -194,7 +194,17 @@ def _append_parse_error_detail(
details[index] = replace(existing, count=existing.count + detail.count)
return

details.append(replace(detail, line=None))
for index, existing in enumerate(details):
if (
existing.path == detail.path
and existing.artifact == detail.artifact
and existing.artifact_family == detail.artifact_family
and existing.reason == detail.reason
):
details[index] = replace(existing, line=None, count=existing.count + detail.count)
return

details[-1] = replace(detail, line=None, count=details[-1].count + detail.count)

Copilot AI Apr 26, 2026

Copy link

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

When the parse-error detail list is at the storage limit and the incoming detail does not match any existing entry, the fallback details[-1] = replace(detail, ..., count=details[-1].count + detail.count) reassigns the previous last entry’s count to the new detail’s (path/artifact/reason). This can corrupt the by-reason/by-artifact counters (e.g., a large invalid-json count can be shifted into legacy-json-fallback-buffer-limit just because it was appended last). Consider aggregating the overflow count into an existing entry without changing its key, or using a dedicated sentinel/overflow bucket, so existing counts aren’t reclassified when the limit is hit.

Suggested change
details[-1] = replace(detail, line=None, count=details[-1].count + detail.count)
details[-1] = replace(details[-1], line=None, count=details[-1].count + detail.count)

Copilot uses AI. Check for mistakes.


def _parse_error_count(parse_error_details: list[ParseErrorDetail]) -> int:
Expand Down
40 changes: 36 additions & 4 deletions tests/scripts/test_aggregate_agent_metrics.py
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -416,6 +416,37 @@ def test_parse_error_contract_counts_compacted_details() -> None:
assert "| artifact | artifact | metrics.ndjson | n/a | invalid-json | 7 |" in lines


def test_append_parse_error_detail_compacts_at_storage_limit() -> None:
details = [
aggregate_agent_metrics.ParseErrorDetail(
path="metrics.ndjson",
artifact="artifact",
artifact_family="artifact",
line=index,
reason="invalid-json",
)
for index in range(aggregate_agent_metrics._MAX_STORED_PARSE_ERROR_DETAILS)
]

aggregate_agent_metrics._append_parse_error_detail(
details,
aggregate_agent_metrics.ParseErrorDetail(
path="metrics.ndjson",
artifact="artifact",
artifact_family="artifact",
line=999,
reason="invalid-json",
),
)

assert len(details) == aggregate_agent_metrics._MAX_STORED_PARSE_ERROR_DETAILS
assert sum(detail.count for detail in details) == (
aggregate_agent_metrics._MAX_STORED_PARSE_ERROR_DETAILS + 1
)
assert details[0].line is None
assert details[0].count == 2


def test_read_ndjson_preserves_artifact_name_with_id_extraction_dir(tmp_path: Path) -> None:
metrics_dir = (
tmp_path
Expand Down Expand Up @@ -475,13 +506,14 @@ def test_read_ndjson_bounds_legacy_json_fallback_buffer(tmp_path: Path) -> None:
entries, errors = aggregate_agent_metrics._read_ndjson([path])

assert entries == []
assert len(errors) <= aggregate_agent_metrics._MAX_STORED_PARSE_ERROR_DETAILS + 2
assert len(errors) <= aggregate_agent_metrics._MAX_STORED_PARSE_ERROR_DETAILS
assert sum(error.count for error in errors) == (
aggregate_agent_metrics._MAX_LEGACY_JSON_FALLBACK_LINES + 2
)
assert errors[-2].line is None
assert errors[-2].reason == "invalid-json"
assert errors[-2].count == (
compacted_invalid_json = next(
error for error in errors if error.reason == "invalid-json" and error.line is None
)
assert compacted_invalid_json.count >= (
aggregate_agent_metrics._MAX_LEGACY_JSON_FALLBACK_LINES
+ 1
- aggregate_agent_metrics._MAX_STORED_PARSE_ERROR_DETAILS
Expand Down
Loading