Skip to content

fix(#1527): warn when automated review misses initial PR window - #1531

Merged
rh-hemartin merged 1 commit into
mainfrom
agent/1527-review-dispatch-observability
Jun 8, 2026
Merged

rh-hemartin merged 1 commit into
mainfrom
agent/1527-review-dispatch-observability

Conversation

@fullsend-ai-coder

Copy link
Copy Markdown
Contributor

When the review agent's pre-fetch step finds no prior automated review, check whether human reviewers have already submitted formal PR reviews. If so, emit a ::warning:: annotation to surface cases where the automated review dispatch was delayed (e.g. by a transient webhook delivery failure) and a human had to review first. This provides observability into the gap described in the issue without changing dispatch behavior.

Note: make lint could not run due to a Go toolchain permission error in the sandbox (not related to this change). The script passes bash -n syntax validation.


Closes #1527

Post-script verification

  • Branch is not main/master (agent/1527-review-dispatch-observability)
  • Secret scan passed (gitleaks — f6c8d3509682ca18940bad7e0c9cef510572fe4b..HEAD)
  • Pre-commit hooks passed (authoritative run on runner)
  • Tests ran inside sandbox

When the review agent's pre-fetch step finds no prior automated
review, check whether human reviewers have already submitted
formal PR reviews. If so, emit a ::warning:: annotation to
surface cases where the automated review dispatch was delayed
(e.g. by a transient webhook delivery failure) and a human had
to review first. This provides observability into the gap
described in the issue without changing dispatch behavior.

Note: make lint could not run due to a Go toolchain permission
error in the sandbox (not related to this change). The script
passes bash -n syntax validation.

Closes #1527
@github-actions

Copy link
Copy Markdown

Site preview

Preview: https://9aaf1556-site.fullsend-ai.workers.dev

Commit: 8a863b268b6cefe158fee496bc1971741e74c571

@fullsend-ai-review

Copy link
Copy Markdown

Review

Findings

Low

  • [correctness] internal/scaffold/fullsend-repo/scripts/pre-fetch-prior-review.sh:30 — The --paginate --jq pattern applies the jq expression per-page, not across all pages. If reviews span multiple pages (30+ reviews), the command outputs one integer per page (e.g., 3\n2), and the [[ "${HUMAN_REVIEW_COUNT}" -gt 0 ]] comparison silently fails on the multiline string — suppressing the warning even when human reviews exist. The existing code in this same script (lines 18–23) correctly handles pagination by using --jq '.[]' | jq -s '...'. Extremely unlikely in practice but inconsistent with the established pattern.
    Remediation: Follow the existing pagination pattern: --jq '.[]' | jq -s '[.[] | select(.user.type != "Bot")] | length'


# Observability: warn if human reviewers already commented before the
# first automated review dispatch. This surfaces cases where the
# automated review missed the initial PR-creation window (e.g. due to

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

[low] correctness

The --paginate --jq pattern applies the jq expression per-page. If reviews span multiple pages (30+), the command outputs one integer per page and the -gt comparison silently fails on the multiline string, suppressing the warning. The existing code in this same script (lines 18-23) correctly handles pagination by piping --jq .[] to jq -s.

Suggested fix: Follow the existing pagination pattern: --jq '.[]' | jq -s '[.[] | select(.user.type != "Bot")] | length'

@fullsend-ai-review fullsend-ai-review Bot added the ready-for-merge All reviewers approved — ready to merge label May 26, 2026
@rh-hemartin
rh-hemartin added this pull request to the merge queue Jun 8, 2026
@rh-hemartin rh-hemartin self-assigned this Jun 8, 2026
@github-merge-queue
github-merge-queue Bot removed this pull request from the merge queue due to failed status checks Jun 8, 2026
@rh-hemartin
rh-hemartin added this pull request to the merge queue Jun 8, 2026
Merged via the queue into main with commit 2ef42dd Jun 8, 2026
10 of 11 checks passed
@rh-hemartin
rh-hemartin deleted the agent/1527-review-dispatch-observability branch June 8, 2026 09:16
@fullsend-ai-retro

Copy link
Copy Markdown

Retro: PR #1531 — Review dispatch observability warning

Timeline: Issue #1527 was filed by the retro agent on 2026-05-26 about a 6-hour gap where automated review missed PR #1487's initial window. The code agent created PR #1531 within 12 minutes, adding a ::warning:: annotation to pre-fetch-prior-review.sh when human reviews exist but no prior automated review was found. The review agent approved with one low-severity correctness finding about an incorrect --paginate --jq pattern. No fix agent ran. Human reviewer approved 13 days later on 2026-06-08. The PR merged with the bug unfixed.

What went well: Fast code agent turnaround (12 min from issue to PR). Review agent correctly identified the pagination bug. Clean single-file change.

What fell through: The review agent's finding about --paginate --jq applying the jq expression per-page (not across all pages) is a real correctness bug — on PRs with 30+ reviews, the warning would be silently suppressed. Despite finding this bug, the review agent approved, no fix agent was triggered, and the human merged without addressing it. The correct pagination pattern (--jq '.[]' | jq -s '...') existed just 10 lines above in the same file.

Existing issue coverage (skipped proposals):

  • Review agent approving despite correctness findings: covered by #1144, #1476, recently closed #1942
  • Fix agent not triggering on low findings: covered by #870
  • Code agent pattern consistency: partially covered by #871, #1289

1 proposal filed for the concrete shipped bug that has no existing issue tracking it.

Proposals filed

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

ready-for-merge All reviewers approved — ready to merge

Projects

None yet

Development

Successfully merging this pull request may close these issues.

Investigate missing review dispatch on initial PR creation

1 participant