Repository navigation
fix(#2305): treat 401/403 comment-posting errors as non-fatal in post-retro.sh #2306
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
Changes from 2 commits
d2d2428
a84bddf
773df28
File filter
Filter by extension
Conversations
Jump to
Diff view
Diff view
There are no files selected for viewing
| Original file line number | Diff line number | Diff line change |
|---|---|---|
| @@ -0,0 +1,271 @@ | ||
| #!/usr/bin/env bash | ||
| # post-retro-test.sh — Test post-retro.sh with fixture JSON inputs. | ||
| # | ||
| # Uses a mock gh command to capture calls without hitting GitHub. | ||
| # Run from the repo root: bash internal/scaffold/fullsend-repo/scripts/post-retro-test.sh | ||
|
|
||
| set -euo pipefail | ||
|
|
||
| SCRIPT_DIR="$(cd "$(dirname "${BASH_SOURCE[0]}")" && pwd)" | ||
| POST_SCRIPT="${SCRIPT_DIR}/post-retro.sh" | ||
| FAILURES=0 | ||
|
|
||
| # Create a temp directory for test fixtures and mock state. | ||
| TMPDIR="$(mktemp -d)" | ||
| trap 'rm -rf "${TMPDIR}"' EXIT | ||
|
|
||
| # --- Mock gh --- | ||
| # GH_MOCK_COMMENT_FAIL controls how the mock responds to the comment-posting | ||
| # gh api call: | ||
| # "" (empty/unset) — succeed (exit 0) | ||
| # "403" — fail with HTTP 403 | ||
| # "401" — fail with HTTP 401 | ||
| # "500" — fail with HTTP 500 | ||
| # "422" — fail with HTTP 422 | ||
| GH_LOG="${TMPDIR}/gh-calls.log" | ||
| MOCK_BIN="${TMPDIR}/bin" | ||
| mkdir -p "${MOCK_BIN}" | ||
| cat > "${MOCK_BIN}/gh" <<'MOCKEOF' | ||
| #!/usr/bin/env bash | ||
| # Consume stdin if --input - is passed, to avoid SIGPIPE under pipefail. | ||
| for arg in "$@"; do | ||
| if [[ "${arg}" == "--input" ]]; then | ||
| cat > /dev/null | ||
| break | ||
| fi | ||
| done | ||
|
|
||
| echo "gh $*" >> "${GH_LOG}" | ||
|
There was a problem hiding this comment. Choose a reason for hiding this commentThe reason will be displayed to describe this comment to others. Learn more. [low] test coverage gap The happy-path-one-proposal test does not assert that gh issue create was called — it only checks the comment-posting endpoint appeared in the log. Suggested fix: Add an assertion that verifies gh issue create appears in the GH_LOG with expected arguments. There was a problem hiding this comment. Choose a reason for hiding this commentThe reason will be displayed to describe this comment to others. Learn more. [low] test coverage gap The happy-path-one-proposal test name implies it validates the full happy path but only checks the comment-posting endpoint. A separate test covers gh issue create, but the test naming is misleading. Suggested fix: Rename to happy-path-comment-posted, or add the gh issue create assertion. |
||
|
|
||
| # Issue creation calls — return a fake issue URL. | ||
| if [[ "$1" == "issue" && "$2" == "create" ]]; then | ||
| echo "https://github.com/test-org/target-repo/issues/99" | ||
| exit 0 | ||
| fi | ||
|
|
||
| # Comment posting via gh api — controlled by GH_MOCK_COMMENT_FAIL. | ||
| if [[ "$1" == "api" && "$2" == *"/comments" ]]; then | ||
| case "${GH_MOCK_COMMENT_FAIL:-}" in | ||
| 403) | ||
| echo "HTTP 403: Resource not accessible by integration" >&2 | ||
| exit 1 | ||
| ;; | ||
| 401) | ||
| echo "HTTP 401: Unauthorized" >&2 | ||
| exit 1 | ||
| ;; | ||
| 500) | ||
| echo "HTTP 500: Internal Server Error" >&2 | ||
| exit 1 | ||
| ;; | ||
| 422) | ||
| echo "HTTP 422: Unprocessable Entity" >&2 | ||
| exit 1 | ||
| ;; | ||
| *) | ||
| echo '{"id": 1, "html_url": "https://github.com/test-org/test-repo/pull/10#issuecomment-1"}' | ||
| exit 0 | ||
| ;; | ||
| esac | ||
| fi | ||
|
|
||
| # Default: succeed silently. | ||
| exit 0 | ||
| MOCKEOF | ||
| chmod +x "${MOCK_BIN}/gh" | ||
|
|
||
| # Mock jq is not needed — we use the real jq. | ||
| # Mock sed is not needed — we use the real sed. | ||
|
|
||
| export PATH="${MOCK_BIN}:${PATH}" | ||
| export GH_LOG="${GH_LOG}" | ||
| export ORIGINATING_URL="https://github.com/test-org/test-repo/pull/10" | ||
| export GH_TOKEN="fake-token" | ||
|
|
||
| # Fixture: a valid agent result with one proposal. | ||
| FIXTURE_ONE_PROPOSAL='{ | ||
| "summary": "The retro analysis found one improvement opportunity.", | ||
| "proposals": [ | ||
| { | ||
| "target_repo": "test-org/target-repo", | ||
| "title": "Improve error handling in widget service", | ||
| "what_happened": "The widget service crashed on empty input.", | ||
| "what_could_go_better": "Input validation should reject empty payloads.", | ||
| "proposed_change": "Add a nil check at the entry point.", | ||
| "validation_criteria": "Widget service returns 400 on empty input." | ||
| } | ||
| ] | ||
| }' | ||
|
|
||
| # Fixture: a valid agent result with no proposals. | ||
| FIXTURE_NO_PROPOSALS='{ | ||
| "summary": "The retro analysis found no actionable improvements.", | ||
| "proposals": [] | ||
| }' | ||
|
|
||
| run_test() { | ||
| local test_name="$1" | ||
| local json_content="$2" | ||
| local expected_pattern="$3" | ||
| local expect_failure="${4:-false}" | ||
| local comment_fail="${5:-}" | ||
|
|
||
| # Create iteration output structure. | ||
| local run_dir="${TMPDIR}/run-${test_name}" | ||
| mkdir -p "${run_dir}/iteration-1/output" | ||
| echo "${json_content}" > "${run_dir}/iteration-1/output/agent-result.json" | ||
|
|
||
| # Clear gh call log. | ||
| : > "${GH_LOG}" | ||
| export GH_MOCK_COMMENT_FAIL="${comment_fail}" | ||
|
|
||
| # Run the post-script. | ||
| local exit_code=0 | ||
| (cd "${run_dir}" && bash "${POST_SCRIPT}") > "${TMPDIR}/stdout.log" 2>&1 || exit_code=$? | ||
|
|
||
| if [[ "${expect_failure}" == "true" ]]; then | ||
| if [[ ${exit_code} -eq 0 ]]; then | ||
| echo "FAIL: ${test_name} — expected failure but got success" | ||
| FAILURES=$((FAILURES + 1)) | ||
| return | ||
| fi | ||
| echo "PASS: ${test_name} (expected failure, got exit code ${exit_code})" | ||
| return | ||
| fi | ||
|
|
||
| if [[ ${exit_code} -ne 0 ]]; then | ||
| echo "FAIL: ${test_name} — exit code ${exit_code}" | ||
| cat "${TMPDIR}/stdout.log" | ||
| FAILURES=$((FAILURES + 1)) | ||
| return | ||
| fi | ||
|
|
||
| if [[ -n "${expected_pattern}" ]] && ! grep -qF "${expected_pattern}" "${GH_LOG}"; then | ||
| echo "FAIL: ${test_name} — expected gh call pattern '${expected_pattern}' not found" | ||
| echo "Actual calls:" | ||
| cat "${GH_LOG}" | ||
| FAILURES=$((FAILURES + 1)) | ||
| return | ||
| fi | ||
|
|
||
| echo "PASS: ${test_name}" | ||
| } | ||
|
|
||
| run_test_stdout() { | ||
| local test_name="$1" | ||
| local json_content="$2" | ||
| local expected_stdout="$3" | ||
| local expect_failure="${4:-false}" | ||
| local comment_fail="${5:-}" | ||
|
|
||
| local run_dir="${TMPDIR}/run-${test_name}" | ||
| mkdir -p "${run_dir}/iteration-1/output" | ||
| echo "${json_content}" > "${run_dir}/iteration-1/output/agent-result.json" | ||
| : > "${GH_LOG}" | ||
| export GH_MOCK_COMMENT_FAIL="${comment_fail}" | ||
|
|
||
| local exit_code=0 | ||
| (cd "${run_dir}" && bash "${POST_SCRIPT}") > "${TMPDIR}/stdout.log" 2>&1 || exit_code=$? | ||
|
|
||
| if [[ "${expect_failure}" == "true" ]]; then | ||
| if [[ ${exit_code} -eq 0 ]]; then | ||
| echo "FAIL: ${test_name} — expected failure but got success" | ||
| FAILURES=$((FAILURES + 1)) | ||
| return | ||
| fi | ||
| if [[ -n "${expected_stdout}" ]] && ! grep -qF "${expected_stdout}" "${TMPDIR}/stdout.log"; then | ||
| echo "FAIL: ${test_name} — expected stdout pattern '${expected_stdout}' not found" | ||
| echo "Actual stdout:" | ||
| cat "${TMPDIR}/stdout.log" | ||
| FAILURES=$((FAILURES + 1)) | ||
| return | ||
| fi | ||
| echo "PASS: ${test_name} (expected failure)" | ||
| return | ||
| fi | ||
|
|
||
| if [[ ${exit_code} -ne 0 ]]; then | ||
| echo "FAIL: ${test_name} — exit code ${exit_code}" | ||
| cat "${TMPDIR}/stdout.log" | ||
| FAILURES=$((FAILURES + 1)) | ||
| return | ||
| fi | ||
|
|
||
| if ! grep -qF "${expected_stdout}" "${TMPDIR}/stdout.log"; then | ||
| echo "FAIL: ${test_name} — expected stdout pattern '${expected_stdout}' not found" | ||
| echo "Actual stdout:" | ||
| cat "${TMPDIR}/stdout.log" | ||
| FAILURES=$((FAILURES + 1)) | ||
| return | ||
| fi | ||
|
|
||
| echo "PASS: ${test_name}" | ||
| } | ||
|
|
||
| # --- Test cases --- | ||
|
|
||
| # Happy path: one proposal filed, comment posted successfully. | ||
| run_test "happy-path-one-proposal" \ | ||
| "${FIXTURE_ONE_PROPOSAL}" \ | ||
| "repos/test-org/test-repo/issues/10/comments" | ||
|
|
||
| # Verify that the happy-path also called gh issue create. | ||
| run_test "happy-path-issue-created" \ | ||
| "${FIXTURE_ONE_PROPOSAL}" \ | ||
| "gh issue create" | ||
|
|
||
| # Happy path: no proposals, comment posted successfully. | ||
| run_test "happy-path-no-proposals" \ | ||
| "${FIXTURE_NO_PROPOSALS}" \ | ||
| "repos/test-org/test-repo/issues/10/comments" | ||
|
|
||
| # 403 on comment posting is non-fatal — script should exit 0 with a warning. | ||
| run_test_stdout "comment-403-non-fatal" \ | ||
| "${FIXTURE_ONE_PROPOSAL}" \ | ||
| "::warning::Could not post summary comment" \ | ||
| "false" \ | ||
| "403" | ||
|
|
||
| # 401 on comment posting is non-fatal — script should exit 0 with a warning. | ||
| run_test_stdout "comment-401-non-fatal" \ | ||
| "${FIXTURE_ONE_PROPOSAL}" \ | ||
| "::warning::Could not post summary comment" \ | ||
| "false" \ | ||
| "401" | ||
|
|
||
| # 500 on comment posting remains fatal. | ||
| run_test_stdout "comment-500-fatal" \ | ||
| "${FIXTURE_ONE_PROPOSAL}" \ | ||
| "ERROR: failed to post summary comment" \ | ||
| "true" \ | ||
| "500" | ||
|
|
||
| # 422 on comment posting remains fatal. | ||
| run_test_stdout "comment-422-fatal" \ | ||
| "${FIXTURE_ONE_PROPOSAL}" \ | ||
| "ERROR: failed to post summary comment" \ | ||
| "true" \ | ||
| "422" | ||
|
|
||
| # 403 with no proposals — still non-fatal. | ||
| run_test_stdout "comment-403-no-proposals" \ | ||
| "${FIXTURE_NO_PROPOSALS}" \ | ||
| "::warning::Could not post summary comment" \ | ||
| "false" \ | ||
| "403" | ||
|
|
||
| # Post-retro complete should appear on successful runs. | ||
| run_test_stdout "complete-message" \ | ||
| "${FIXTURE_ONE_PROPOSAL}" \ | ||
| "Post-retro complete." | ||
|
|
||
| # --- Results --- | ||
|
|
||
| if [[ ${FAILURES} -gt 0 ]]; then | ||
| echo "" | ||
| echo "${FAILURES} test(s) failed." | ||
| exit 1 | ||
| fi | ||
|
|
||
| echo "" | ||
| echo "All post-retro tests passed." | ||
| Original file line number | Diff line number | Diff line change |
|---|---|---|
|
|
@@ -124,8 +124,34 @@ else | |
| fi | ||
|
There was a problem hiding this comment. Choose a reason for hiding this commentThe reason will be displayed to describe this comment to others. Learn more. [low] architecture-coherence The inline 401/403 handling introduces a third distinct error-handling pattern for GitHub API permission errors across post-* scripts. The inline comment explains the design rationale. |
||
|
|
||
| echo "Posting summary comment on ${ORIGINATING_REPO}#${ORIGINATING_NUMBER}" | ||
| jq -nc --arg body "${COMMENT}" '{body: $body}' | gh api \ | ||
| # Note: we handle 401/403 inline rather than relying on github-api-csma.sh | ||
|
There was a problem hiding this comment. Choose a reason for hiding this commentThe reason will be displayed to describe this comment to others. Learn more. [low] architectural-consistency The inline 401/403 handling bypasses the existing lib/github-api-csma.sh infrastructure. CSMA already distinguishes retryable rate-limit 403s from bare permission 403s, but does not distinguish between rate-limit exhaustion and non-retryable permission error in its return code. The inline approach is simpler but diverges from post-prioritize.sh pattern. |
||
| # because the intent is different. CSMA retries rate-limited requests; here | ||
| # we want graceful degradation when the token permanently lacks permission | ||
| # to comment on a specific repo. Retrying a 403 permission error is futile. | ||
| COMMENT_OUTPUT="" | ||
| COMMENT_EXIT=0 | ||
| COMMENT_OUTPUT=$(jq -nc --arg body "${COMMENT}" '{body: $body}' | gh api \ | ||
| "repos/${ORIGINATING_REPO}/issues/${ORIGINATING_NUMBER}/comments" \ | ||
| --input - | ||
| --input - 2>&1) || COMMENT_EXIT=$? | ||
|
|
||
| if [[ ${COMMENT_EXIT} -ne 0 ]]; then | ||
| # Treat 401/403 as non-fatal — the token lacks permission to comment on | ||
| # this repo, but the core deliverables (analysis + proposal issues) are | ||
|
There was a problem hiding this comment. Choose a reason for hiding this commentThe reason will be displayed to describe this comment to others. Learn more. [low] error handling fragility The grep pattern grep -qE "HTTP (401|403)" depends on the exact error format emitted by the gh CLI. If a future gh version changes the format, the match would fail and the error would be treated as fatal. There was a problem hiding this comment. Choose a reason for hiding this commentThe reason will be displayed to describe this comment to others. Learn more. [low] duplication The sanitization code (5 lines replacing ::, %0A, %0a, %0D, %0d) is duplicated identically in both the if and else branches. Suggested fix: Hoist the sanitization outside the conditional — perform it once on COMMENT_OUTPUT before the if statement. |
||
| # already complete. See #2305. | ||
|
There was a problem hiding this comment. Choose a reason for hiding this commentThe reason will be displayed to describe this comment to others. Learn more. [low] injection COMMENT_RESPONSE is interpolated unsanitized into the ::warning:: GHA workflow command. Although the dangerous commands (::set-env, ::add-path) were disabled by GitHub in 2020 and attacker control is limited, remaining injectable commands (::set-output, ::save-state) could still be exploited. Suggested fix: Sanitize COMMENT_RESPONSE before interpolating into the ::warning:: directive — strip :: sequences and %0A/%0D URL-encoded newlines. |
||
| # The grep pattern matches gh CLI's "HTTP 4xx" error format. If a future | ||
| # gh version changes the format, the match will fail-closed (treating the | ||
| # error as fatal), which is the safer default. | ||
| if echo "${COMMENT_OUTPUT}" | grep -qE "HTTP (401|403)"; then | ||
| # Sanitize before interpolating into GHA workflow command to prevent | ||
| # injecting ::set-output or ::save-state directives via crafted responses. | ||
| SAFE_OUTPUT="${COMMENT_OUTPUT//::/}" | ||
|
There was a problem hiding this comment. Choose a reason for hiding this commentThe reason will be displayed to describe this comment to others. Learn more. [low] sanitization-completeness The sanitization handles only uppercase %0A and %0D. The established pattern in extract-transcript-error.sh handles both cases (%0A/%0a, %0D/%0d). Suggested fix: Add lowercase variant handling: SAFE_OUTPUT="${SAFE_OUTPUT//%0a/}" and SAFE_OUTPUT="${SAFE_OUTPUT//%0d/}" |
||
| SAFE_OUTPUT="${SAFE_OUTPUT//%0A/}" | ||
|
There was a problem hiding this comment. Choose a reason for hiding this commentThe reason will be displayed to describe this comment to others. Learn more. [low] error handling fragility The grep -qE "HTTP (401|403)" pattern depends on the exact error format emitted by the gh CLI. The inline comment acknowledges this fail-closed behavior. There was a problem hiding this comment. Choose a reason for hiding this commentThe reason will be displayed to describe this comment to others. Learn more. [low] error-handling-consistency The fatal error path uses plain echo ERROR: while the warning path uses GHA annotation syntax (::warning::). Other post-scripts use ::error:: for fatal errors. Pre-existing inconsistency in this file. Suggested fix: Change to echo ::error::failed to post summary comment on ... for consistency with post-code.sh. |
||
| SAFE_OUTPUT="${SAFE_OUTPUT//%0D/}" | ||
| echo "::warning::Could not post summary comment to ${ORIGINATING_REPO}#${ORIGINATING_NUMBER}: insufficient permissions (${SAFE_OUTPUT}). Skipping." | ||
| else | ||
| echo "ERROR: failed to post summary comment on ${ORIGINATING_REPO}#${ORIGINATING_NUMBER}: ${COMMENT_OUTPUT}" | ||
|
There was a problem hiding this comment. Choose a reason for hiding this commentThe reason will be displayed to describe this comment to others. Learn more. [low] injection In the else branch (non-401/403 errors), COMMENT_OUTPUT is interpolated unsanitized into a plain echo statement. In GitHub Actions, stdout lines matching ::command:: patterns are processed as workflow commands. Although exit 1 follows immediately, an attacker could inject ::save-state:: or ::set-output:: directives that execute before the exit. Suggested fix: Apply the same SAFE_OUTPUT sanitization before echoing in the else branch. |
||
| exit 1 | ||
| fi | ||
| fi | ||
|
|
||
| echo "Post-retro complete." | ||
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
[low] test coverage gap
The happy-path-one-proposal test does not assert that gh issue create was called — it only checks the comment-posting endpoint appeared in the log.