Skip to content
Merged
Show file tree
Hide file tree
Changes from 2 commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
271 changes: 271 additions & 0 deletions internal/scaffold/fullsend-repo/scripts/post-retro-test.sh
Original file line number Diff line number Diff line change
@@ -0,0 +1,271 @@
#!/usr/bin/env bash
# post-retro-test.sh — Test post-retro.sh with fixture JSON inputs.
#
# Uses a mock gh command to capture calls without hitting GitHub.
# Run from the repo root: bash internal/scaffold/fullsend-repo/scripts/post-retro-test.sh

set -euo pipefail

SCRIPT_DIR="$(cd "$(dirname "${BASH_SOURCE[0]}")" && pwd)"
POST_SCRIPT="${SCRIPT_DIR}/post-retro.sh"
FAILURES=0

# Create a temp directory for test fixtures and mock state.
TMPDIR="$(mktemp -d)"
trap 'rm -rf "${TMPDIR}"' EXIT

# --- Mock gh ---
# GH_MOCK_COMMENT_FAIL controls how the mock responds to the comment-posting
# gh api call:
# "" (empty/unset) — succeed (exit 0)
# "403" — fail with HTTP 403
# "401" — fail with HTTP 401
# "500" — fail with HTTP 500
# "422" — fail with HTTP 422
GH_LOG="${TMPDIR}/gh-calls.log"
MOCK_BIN="${TMPDIR}/bin"
mkdir -p "${MOCK_BIN}"
cat > "${MOCK_BIN}/gh" <<'MOCKEOF'
#!/usr/bin/env bash
# Consume stdin if --input - is passed, to avoid SIGPIPE under pipefail.
for arg in "$@"; do
if [[ "${arg}" == "--input" ]]; then
cat > /dev/null
break
fi
done

echo "gh $*" >> "${GH_LOG}"

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

[low] test coverage gap

The happy-path-one-proposal test does not assert that gh issue create was called — it only checks the comment-posting endpoint appeared in the log.

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

[low] test coverage gap

The happy-path-one-proposal test does not assert that gh issue create was called — it only checks the comment-posting endpoint appeared in the log.

Suggested fix: Add an assertion that verifies gh issue create appears in the GH_LOG with expected arguments.

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

[low] test coverage gap

The happy-path-one-proposal test name implies it validates the full happy path but only checks the comment-posting endpoint. A separate test covers gh issue create, but the test naming is misleading.

Suggested fix: Rename to happy-path-comment-posted, or add the gh issue create assertion.


# Issue creation calls — return a fake issue URL.
if [[ "$1" == "issue" && "$2" == "create" ]]; then
echo "https://github.com/test-org/target-repo/issues/99"
exit 0
fi

# Comment posting via gh api — controlled by GH_MOCK_COMMENT_FAIL.
if [[ "$1" == "api" && "$2" == *"/comments" ]]; then
case "${GH_MOCK_COMMENT_FAIL:-}" in
403)
echo "HTTP 403: Resource not accessible by integration" >&2
exit 1
;;
401)
echo "HTTP 401: Unauthorized" >&2
exit 1
;;
500)
echo "HTTP 500: Internal Server Error" >&2
exit 1
;;
422)
echo "HTTP 422: Unprocessable Entity" >&2
exit 1
;;
*)
echo '{"id": 1, "html_url": "https://github.com/test-org/test-repo/pull/10#issuecomment-1"}'
exit 0
;;
esac
fi

# Default: succeed silently.
exit 0
MOCKEOF
chmod +x "${MOCK_BIN}/gh"

# Mock jq is not needed — we use the real jq.
# Mock sed is not needed — we use the real sed.

export PATH="${MOCK_BIN}:${PATH}"
export GH_LOG="${GH_LOG}"
export ORIGINATING_URL="https://github.com/test-org/test-repo/pull/10"
export GH_TOKEN="fake-token"

# Fixture: a valid agent result with one proposal.
FIXTURE_ONE_PROPOSAL='{
"summary": "The retro analysis found one improvement opportunity.",
"proposals": [
{
"target_repo": "test-org/target-repo",
"title": "Improve error handling in widget service",
"what_happened": "The widget service crashed on empty input.",
"what_could_go_better": "Input validation should reject empty payloads.",
"proposed_change": "Add a nil check at the entry point.",
"validation_criteria": "Widget service returns 400 on empty input."
}
]
}'

# Fixture: a valid agent result with no proposals.
FIXTURE_NO_PROPOSALS='{
"summary": "The retro analysis found no actionable improvements.",
"proposals": []
}'

run_test() {
local test_name="$1"
local json_content="$2"
local expected_pattern="$3"
local expect_failure="${4:-false}"
local comment_fail="${5:-}"

# Create iteration output structure.
local run_dir="${TMPDIR}/run-${test_name}"
mkdir -p "${run_dir}/iteration-1/output"
echo "${json_content}" > "${run_dir}/iteration-1/output/agent-result.json"

# Clear gh call log.
: > "${GH_LOG}"
export GH_MOCK_COMMENT_FAIL="${comment_fail}"

# Run the post-script.
local exit_code=0
(cd "${run_dir}" && bash "${POST_SCRIPT}") > "${TMPDIR}/stdout.log" 2>&1 || exit_code=$?

if [[ "${expect_failure}" == "true" ]]; then
if [[ ${exit_code} -eq 0 ]]; then
echo "FAIL: ${test_name} — expected failure but got success"
FAILURES=$((FAILURES + 1))
return
fi
echo "PASS: ${test_name} (expected failure, got exit code ${exit_code})"
return
fi

if [[ ${exit_code} -ne 0 ]]; then
echo "FAIL: ${test_name} — exit code ${exit_code}"
cat "${TMPDIR}/stdout.log"
FAILURES=$((FAILURES + 1))
return
fi

if [[ -n "${expected_pattern}" ]] && ! grep -qF "${expected_pattern}" "${GH_LOG}"; then
echo "FAIL: ${test_name} — expected gh call pattern '${expected_pattern}' not found"
echo "Actual calls:"
cat "${GH_LOG}"
FAILURES=$((FAILURES + 1))
return
fi

echo "PASS: ${test_name}"
}

run_test_stdout() {
local test_name="$1"
local json_content="$2"
local expected_stdout="$3"
local expect_failure="${4:-false}"
local comment_fail="${5:-}"

local run_dir="${TMPDIR}/run-${test_name}"
mkdir -p "${run_dir}/iteration-1/output"
echo "${json_content}" > "${run_dir}/iteration-1/output/agent-result.json"
: > "${GH_LOG}"
export GH_MOCK_COMMENT_FAIL="${comment_fail}"

local exit_code=0
(cd "${run_dir}" && bash "${POST_SCRIPT}") > "${TMPDIR}/stdout.log" 2>&1 || exit_code=$?

if [[ "${expect_failure}" == "true" ]]; then
if [[ ${exit_code} -eq 0 ]]; then
echo "FAIL: ${test_name} — expected failure but got success"
FAILURES=$((FAILURES + 1))
return
fi
if [[ -n "${expected_stdout}" ]] && ! grep -qF "${expected_stdout}" "${TMPDIR}/stdout.log"; then
echo "FAIL: ${test_name} — expected stdout pattern '${expected_stdout}' not found"
echo "Actual stdout:"
cat "${TMPDIR}/stdout.log"
FAILURES=$((FAILURES + 1))
return
fi
echo "PASS: ${test_name} (expected failure)"
return
fi

if [[ ${exit_code} -ne 0 ]]; then
echo "FAIL: ${test_name} — exit code ${exit_code}"
cat "${TMPDIR}/stdout.log"
FAILURES=$((FAILURES + 1))
return
fi

if ! grep -qF "${expected_stdout}" "${TMPDIR}/stdout.log"; then
echo "FAIL: ${test_name} — expected stdout pattern '${expected_stdout}' not found"
echo "Actual stdout:"
cat "${TMPDIR}/stdout.log"
FAILURES=$((FAILURES + 1))
return
fi

echo "PASS: ${test_name}"
}

# --- Test cases ---

# Happy path: one proposal filed, comment posted successfully.
run_test "happy-path-one-proposal" \
"${FIXTURE_ONE_PROPOSAL}" \
"repos/test-org/test-repo/issues/10/comments"

# Verify that the happy-path also called gh issue create.
run_test "happy-path-issue-created" \
"${FIXTURE_ONE_PROPOSAL}" \
"gh issue create"

# Happy path: no proposals, comment posted successfully.
run_test "happy-path-no-proposals" \
"${FIXTURE_NO_PROPOSALS}" \
"repos/test-org/test-repo/issues/10/comments"

# 403 on comment posting is non-fatal — script should exit 0 with a warning.
run_test_stdout "comment-403-non-fatal" \
"${FIXTURE_ONE_PROPOSAL}" \
"::warning::Could not post summary comment" \
"false" \
"403"

# 401 on comment posting is non-fatal — script should exit 0 with a warning.
run_test_stdout "comment-401-non-fatal" \
"${FIXTURE_ONE_PROPOSAL}" \
"::warning::Could not post summary comment" \
"false" \
"401"

# 500 on comment posting remains fatal.
run_test_stdout "comment-500-fatal" \
"${FIXTURE_ONE_PROPOSAL}" \
"ERROR: failed to post summary comment" \
"true" \
"500"

# 422 on comment posting remains fatal.
run_test_stdout "comment-422-fatal" \
"${FIXTURE_ONE_PROPOSAL}" \
"ERROR: failed to post summary comment" \
"true" \
"422"

# 403 with no proposals — still non-fatal.
run_test_stdout "comment-403-no-proposals" \
"${FIXTURE_NO_PROPOSALS}" \
"::warning::Could not post summary comment" \
"false" \
"403"

# Post-retro complete should appear on successful runs.
run_test_stdout "complete-message" \
"${FIXTURE_ONE_PROPOSAL}" \
"Post-retro complete."

# --- Results ---

if [[ ${FAILURES} -gt 0 ]]; then
echo ""
echo "${FAILURES} test(s) failed."
exit 1
fi

echo ""
echo "All post-retro tests passed."
30 changes: 28 additions & 2 deletions internal/scaffold/fullsend-repo/scripts/post-retro.sh
Original file line number Diff line number Diff line change
Expand Up @@ -124,8 +124,34 @@ else
fi

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

[low] architecture-coherence

The inline 401/403 handling introduces a third distinct error-handling pattern for GitHub API permission errors across post-* scripts. The inline comment explains the design rationale.


echo "Posting summary comment on ${ORIGINATING_REPO}#${ORIGINATING_NUMBER}"
jq -nc --arg body "${COMMENT}" '{body: $body}' | gh api \
# Note: we handle 401/403 inline rather than relying on github-api-csma.sh

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

[low] architectural-consistency

The inline 401/403 handling bypasses the existing lib/github-api-csma.sh infrastructure. CSMA already distinguishes retryable rate-limit 403s from bare permission 403s, but does not distinguish between rate-limit exhaustion and non-retryable permission error in its return code. The inline approach is simpler but diverges from post-prioritize.sh pattern.

# because the intent is different. CSMA retries rate-limited requests; here
# we want graceful degradation when the token permanently lacks permission
# to comment on a specific repo. Retrying a 403 permission error is futile.
COMMENT_OUTPUT=""
COMMENT_EXIT=0
COMMENT_OUTPUT=$(jq -nc --arg body "${COMMENT}" '{body: $body}' | gh api \
"repos/${ORIGINATING_REPO}/issues/${ORIGINATING_NUMBER}/comments" \
--input -
--input - 2>&1) || COMMENT_EXIT=$?

if [[ ${COMMENT_EXIT} -ne 0 ]]; then
# Treat 401/403 as non-fatal — the token lacks permission to comment on
# this repo, but the core deliverables (analysis + proposal issues) are

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

[low] error handling fragility

The grep pattern grep -qE "HTTP (401|403)" depends on the exact error format emitted by the gh CLI. If a future gh version changes the format, the match would fail and the error would be treated as fatal.

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

[low] duplication

The sanitization code (5 lines replacing ::, %0A, %0a, %0D, %0d) is duplicated identically in both the if and else branches.

Suggested fix: Hoist the sanitization outside the conditional — perform it once on COMMENT_OUTPUT before the if statement.

# already complete. See #2305.

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

[low] injection

COMMENT_RESPONSE is interpolated unsanitized into the ::warning:: GHA workflow command. Although the dangerous commands (::set-env, ::add-path) were disabled by GitHub in 2020 and attacker control is limited, remaining injectable commands (::set-output, ::save-state) could still be exploited.

Suggested fix: Sanitize COMMENT_RESPONSE before interpolating into the ::warning:: directive — strip :: sequences and %0A/%0D URL-encoded newlines.

# The grep pattern matches gh CLI's "HTTP 4xx" error format. If a future
# gh version changes the format, the match will fail-closed (treating the
# error as fatal), which is the safer default.
if echo "${COMMENT_OUTPUT}" | grep -qE "HTTP (401|403)"; then
# Sanitize before interpolating into GHA workflow command to prevent
# injecting ::set-output or ::save-state directives via crafted responses.
SAFE_OUTPUT="${COMMENT_OUTPUT//::/}"

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

[low] sanitization-completeness

The sanitization handles only uppercase %0A and %0D. The established pattern in extract-transcript-error.sh handles both cases (%0A/%0a, %0D/%0d).

Suggested fix: Add lowercase variant handling: SAFE_OUTPUT="${SAFE_OUTPUT//%0a/}" and SAFE_OUTPUT="${SAFE_OUTPUT//%0d/}"

SAFE_OUTPUT="${SAFE_OUTPUT//%0A/}"

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

[low] error handling fragility

The grep -qE "HTTP (401|403)" pattern depends on the exact error format emitted by the gh CLI. The inline comment acknowledges this fail-closed behavior.

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

[low] error-handling-consistency

The fatal error path uses plain echo ERROR: while the warning path uses GHA annotation syntax (::warning::). Other post-scripts use ::error:: for fatal errors. Pre-existing inconsistency in this file.

Suggested fix: Change to echo ::error::failed to post summary comment on ... for consistency with post-code.sh.

SAFE_OUTPUT="${SAFE_OUTPUT//%0D/}"
echo "::warning::Could not post summary comment to ${ORIGINATING_REPO}#${ORIGINATING_NUMBER}: insufficient permissions (${SAFE_OUTPUT}). Skipping."
else
echo "ERROR: failed to post summary comment on ${ORIGINATING_REPO}#${ORIGINATING_NUMBER}: ${COMMENT_OUTPUT}"

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

[low] injection

In the else branch (non-401/403 errors), COMMENT_OUTPUT is interpolated unsanitized into a plain echo statement. In GitHub Actions, stdout lines matching ::command:: patterns are processed as workflow commands. Although exit 1 follows immediately, an attacker could inject ::save-state:: or ::set-output:: directives that execute before the exit.

Suggested fix: Apply the same SAFE_OUTPUT sanitization before echoing in the else branch.

exit 1
fi
fi

echo "Post-retro complete."
Loading