feat(harness): report categorized post-script failures on issue/PR - #38
Conversation
Port fullsend-ai/fullsend#2921: add shared post-failure-report.sh with categorized failure comments, output sanitization, and workflow run links. post-code.sh posts detailed failures on the issue; post-fix.sh posts equivalent comments on the PR. Extend post-code-test.sh and post-fix-test.sh. Signed-off-by: Barak Korren <bkorren@redhat.com> Co-authored-by: Cursor <cursoragent@cursor.com>
|
🤖 Finished Review · ✅ Success · Started 7:39 AM UTC · Completed 7:54 AM UTC |
PR Summary by QodoReport categorized post-script failures on issues/PRs (shared failure reporter)
AI Description
Diagram
High-Level Assessment
Files changed (5)
|
Code Review by Qodo
Context used✅ Compliance rules (platform):
55 rules 1.
|
ReviewRe-review of 3552d6b → 975dd1b. Prior review provenance: app-verified. Prior low finding (bundler sed pipeline ordering for quoted paths with inline comments) addressed — comment-stripping sed expression now runs first, confirmed by FindingsHigh
Low
Previous runReviewRe-review of 72e1a96 → 3552d6b (fix(scripts): address fullsend-ai-review low findings). Prior review provenance: app-verified. All three prior low-severity findings addressed:
FindingsLow
Previous run (2)ReviewRe-review of 679e1ca → 72e1a96 (10372ec review feedback, 9ade211 shellcheck, 72e1a96 cross-retry dedup). Prior review provenance: app-verified. All prior findings either resolved or unchanged. Three new commits address round-2 reviewer feedback: awk-based literal token redaction (replacing glob-unsafe bash substitution), case-insensitive PEM detection, symlink resolution in the bundler, check-bundle drift test refactored to avoid modifying repo files, and new tests for each change. Prior review findings — status
New changes verified correct
FindingsLow
Previous run (3)ReviewVerdict: comment — all prior findings resolved or unchanged. One low-severity follow-up remains. Protected paths require human approval. Re-review of 04f52c7 → 679e1ca (1e45ee8 review feedback, 267a3d9 lib-deps + CI hooks, 5a4d522 merge upstream, 1790d27 rebuild post-merge, 679e1ca gitlint cleanup). Prior review provenance: app-verified. This PR extracts failure reporting into a shared library ( Prior review findings — resolvedBoth prior findings have been addressed:
New changes verified correct
Findings1. Bundler source-line regex does not handle inline comments — LowFile: Severity anchored from prior review (04f52c7). The regex
Previous run (4)ReviewVerdict: approve — all prior medium-severity findings resolved. Two low-severity follow-ups remain. No blocking issues. Re-review of f9542d5 → 04f52c7 (603b401 GHA sanitization + cross-dedup test, 04f52c7 end-of-file fix, plus rebase correcting commit type). Prior review provenance: app-verified. This PR extracts failure reporting into a shared library ( Prior review findings — resolvedAll four prior findings have been addressed:
Findings1.
|
ralphbean
left a comment
There was a problem hiding this comment.
I think we learned in the fix for the prioritize agent's CSMA lib fix that the post scripts of agents cannot reach a lib script. Right?
Bring in script-test CI, post-scribe-test, and other mainline changes before adding the script bundling build stage. Signed-off-by: Barak Korren <bkorren@redhat.com>
|
Addressed @ralphbean's review with a script bundling build stage: |
|
🤖 Finished Review · ✅ Success · Started 12:45 PM UTC · Completed 12:59 PM UTC |
|
🤖 Review · |
|
🤖 Review · ❌ Terminated · Started 1:10 PM UTC · Ended 1:27 PM UTC |
|
CI is green on
@ralphbean — the runtime lib sourcing concern should be addressed by the bundling build stage ( |
|
🤖 Finished Review · ✅ Success · Started 1:10 PM UTC · Completed 1:27 PM UTC |
ifireball
left a comment
There was a problem hiding this comment.
@ralphbean Re: runtime lib sourcing — agreed, and addressed in 7529160+.
Post-scripts are now maintained as *.src.sh and bundled via make script-build (scripts/bundle-sh.sh inlines scripts/lib/*.lib.sh into the committed *.sh artifacts the harness fetches). No runtime source of sibling lib files.
post-prioritize follows the same pattern (CSMA lib re-extracted to github-api-csma.lib.sh and bundled). CI runs make check-bundle plus script-test in both source and bundled modes.
Happy to walk through the layout if helpful — otherwise a re-review/approval when you have a moment would unblock merge.
…cripts Introduce bundle-sh.sh to inline scripts/lib/*.lib.sh into .src.sh sources, producing committed .sh artifacts for harness deployment. Add script-build and check-bundle Makefile targets, split lib tests into post-failure-report-test, and run script-test in source and bundled modes in CI. Signed-off-by: Barak Korren <bkorren@redhat.com> Co-authored-by: Cursor <cursoragent@cursor.com>
Sanitize captured command output before echoing to GHA logs, restore org-mode URL routing tests, fix bundle-sh SC2094, and document POST_FAILURE_DETAIL_MAX_LINES in post-script headers. Signed-off-by: Barak Korren <bkorren@redhat.com> Co-authored-by: Cursor <cursoragent@cursor.com>
Signed-off-by: Barak Korren <bkorren@redhat.com> Co-authored-by: Cursor <cursoragent@cursor.com>
…test Add gha_echo() for sanitised ::warning::/::error::/::notice:: output, cover cross-library bundle dedup in bundle-sh-test, and extend review follow-ups from fullsend-ai-review. Signed-off-by: Barak Korren <bkorren@redhat.com> Co-authored-by: Cursor <cursoragent@cursor.com>
|
Merge queue ejection root cause — The merge queue creates a temporary commit merging PR head into This is a one-time bootstrap problem: the first PR that introduces the bundler will always hit this because Possible fixes:
Job log: https://github.com/fullsend-ai/agents/actions/runs/29539949315/job/87759861026 |
…ilure-reporting Signed-off-by: Barak Korren <bkorren@redhat.com>
…merge Merge of upstream/main textually combined the old hand-written post-code.sh with the bundled artifact. Port the pr_body + gitleaks scan logic into post-code.src.sh and regenerate the bundle so check-bundle matches. Signed-off-by: Barak Korren <bkorren@redhat.com> Co-authored-by: Cursor <cursoragent@cursor.com>
|
🤖 Finished Review · ✅ Success · Started 6:30 AM UTC · Completed 6:37 AM UTC |
|
Review skipped — this PR is already merged. The Posted by fullsend post-review check |
Retro: PR #38 — Categorized post-script failure reportingPR: fullsend-ai/agents#38 by ifireball Timeline
Key findingsReview quality gap: The review agent's security sub-agent ran 9 successful passes using the Safety net worked: The post-review.sh protected-path gate correctly downgraded the approve verdict to comment-only for this Cancellation waste: 9/18 review runs cancelled, with 6 in a 20-minute window during fix iterations on Jul 13. This is consistent with existing debounce proposals (fullsend#4960, #5139, #5210) and provides additional quantitative evidence. No false positives: Both the review agent and human reviewer had zero false positives. The agent's findings were valid but surface-level; the human's were deeper and security-critical. ProposalsTwo proposals filed targeting the security and correctness sub-agent prompts in Proposals filed
|
|
🤖 Finished Retro · ✅ Success · Started 6:36 AM UTC · Completed 7:01 AM UTC |
Summary
Ports fullsend-ai/fullsend#2921: categorized post-script failure reporting for
post-code.shandpost-fix.sh.Failure reporting
scripts/lib/post-failure-report.lib.shwith categorized failure comments, output sanitization (tokens/PEM redaction, GHA workflow-command stripping), and workflow run linkspost-code.shposts detailed failure comments on the issue (push rejected, pre-commit blocked, secret scan, PR creation failed, etc.)post-fix.shposts equivalent failure comments on the PR (previously silent on many post-script failures)post-code-test.sh/post-fix-test.shand addpost-failure-report-test.shfor comment building, sanitization, org-mode URL routing, and push categorizationScript bundling (added during review)
The harness fetches each runner script as an isolated blob, so post-scripts cannot
sourcefiles fromscripts/lib/at runtime (same constraint that motivated the CSMA lib fix forpost-prioritize.sh). To keep shared failure-reporting logic DRY without breaking runtime, this PR adds a small build stage:scripts/lib/*.lib.shscripts/*.src.shsourcecallsscripts/*.sh(from.src.sh)Build tooling:
scripts/bundle-sh.sh— inlinesscripts/lib/*.lib.shinto self-contained.shartifacts (nested sources, deduped)make script-build/make check-bundle— regenerate and verify committed bundlesscript-test.yml) runscheck-bundle, thenscript-testin bothsourceandbundledmodesScripts converted to the source→bundle workflow:
post-code.src.sh→post-code.shpost-fix.src.sh→post-fix.shpost-prioritize.src.sh→post-prioritize.sh(re-extractsgithub-api-csma.lib.shfrom the prior inline copy)After editing a
.src.shor.lib.shfile, runmake script-buildand commit source + bundled files together. See README “Script bundling” section.Test plan
make script-build && make check-bundlemake script-test(source mode)make script-test SCRIPT_TEST_TARGET=bundledbash scripts/post-failure-report-test.shf9542d5