Skip to content
Closed
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension


Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
98 changes: 98 additions & 0 deletions .github/actions/check-e2e-authorization/action.yml
Original file line number Diff line number Diff line change
@@ -0,0 +1,98 @@
name: Check E2E Authorization
description: >-
Authorize PR-triggered e2e runs for trusted authors or a fresh ok-to-test label.
Removes stale ok-to-test labels and posts a sticky PR comment when unauthorized.

inputs:
pr_number:
description: Pull request number to authorize
required: true
repository:
description: Repository in owner/name form
required: true
docs_url:
description: Link to e2e testing documentation
required: false
default: https://github.com/fullsend-ai/fullsend/blob/main/docs/guides/dev/e2e-testing.md

outputs:
authorized:
description: Whether e2e tests may run for this PR
value: ${{ steps.check.outputs.authorized }}
reason:
description: Authorization outcome reason code
value: ${{ steps.check.outputs.reason }}
label_removed:
description: Whether a stale ok-to-test label was removed
value: ${{ steps.check.outputs.label_removed }}

runs:
using: composite
steps:
- name: Check authorization
id: check
shell: bash
env:
GH_TOKEN: ${{ github.token }}
PR_NUMBER: ${{ inputs.pr_number }}
REPOSITORY: ${{ inputs.repository }}
run: bash "${GITHUB_WORKSPACE}/scripts/check-e2e-authorization.sh" "${PR_NUMBER}" "${REPOSITORY}"

- name: Post gate comment
if: steps.check.outputs.authorized != 'true'
shell: bash
env:
GH_TOKEN: ${{ github.token }}
Comment thread
ifireball marked this conversation as resolved.
Comment thread
ifireball marked this conversation as resolved.
REPOSITORY: ${{ inputs.repository }}
PR_NUMBER: ${{ inputs.pr_number }}
REASON: ${{ steps.check.outputs.reason }}
LABEL_REMOVED: ${{ steps.check.outputs.label_removed }}
DOCS_URL: ${{ inputs.docs_url }}
run: |
set -euo pipefail

marker='<!-- e2e-gate -->'
owner="${REPOSITORY%%/*}"
repo="${REPOSITORY#*/}"

body="${marker}
## E2E tests did not run

"

case "${REASON}" in
stale_ok_to_test)
body+="The \`ok-to-test\` label was removed because new commits landed after it was applied. A maintainer must re-apply \`ok-to-test\` after reviewing the latest changes.
"
;;
error)
body+="The authorization check failed (GitHub API error). Re-run the workflow when the API is available; if this persists, contact a maintainer.
"
;;
*)
body+="E2E tests run automatically for org/repo **members and collaborators** on pull requests.

For other contributors, a maintainer must add the \`ok-to-test\` label **after** the latest push.
"
;;
esac

if [[ "${LABEL_REMOVED}" == "true" ]]; then
body+="
> **Note:** \`ok-to-test\` was cleared due to new commits.
"
fi

body+="
See [E2E testing guide](${DOCS_URL}) for details."

existing_id="$(gh api "repos/${owner}/${repo}/issues/${PR_NUMBER}/comments" --paginate \
--jq ".[] | select(.body | contains(\"${marker}\")) | .id" | head -n1 || true)"

if [[ -n "${existing_id}" ]]; then
jq -n --arg body "${body}" '{body: $body}' | \
gh api -X PATCH "repos/${owner}/${repo}/issues/comments/${existing_id}" --input - >/dev/null
else
jq -n --arg body "${body}" '{body: $body}' | \
gh api -X POST "repos/${owner}/${repo}/issues/${PR_NUMBER}/comments" --input - >/dev/null
fi
84 changes: 63 additions & 21 deletions .github/workflows/e2e.yml
Original file line number Diff line number Diff line change
@@ -1,10 +1,24 @@
name: E2E Tests

permissions:
contents: read
id-token: write
# PRs invoke this workflow via e2e_shim.yml (workflow_call @ refs/heads/main).
# See docs/ADRs/0043-e2e-wif-shim-and-pr-authorization-gate.md

on:
Comment thread
ifireball marked this conversation as resolved.
workflow_call:
inputs:
pr_number:
description: Pull request number to authorize and test
required: true
type: number
head_sha:
description: PR head SHA to checkout for tests
required: true
type: string
pr_author:
description: PR author login (informational; auth uses API)
required: false
type: string
default: ""
push:
branches: [main]
paths:
Expand All @@ -18,30 +32,58 @@ on:
- 'internal/sentencetoken/english.json'
- 'Makefile'
- '.github/workflows/e2e.yml'
pull_request:
paths:
- '**/*.go'
- 'go.mod'
- 'go.sum'
- 'e2e/**'
- 'internal/scaffold/fullsend-repo/**'
- 'internal/security/hooks/**'
- 'internal/dispatch/gcf/mintsrc/**'
- 'internal/sentencetoken/english.json'
- 'Makefile'
- '.github/workflows/e2e.yml'
- '.github/workflows/e2e_shim.yml'
- '.github/actions/check-e2e-authorization/**'
- 'scripts/check-e2e-authorization.sh'
- 'scripts/check-e2e-authorization_test.sh'
workflow_dispatch:

permissions: {}

concurrency:
group: ${{ github.workflow }}-${{ github.ref }}
group: e2e-${{ github.event_name == 'workflow_call' && inputs.pr_number || github.ref }}
cancel-in-progress: ${{ github.ref != 'refs/heads/main' }}

jobs:
gate:
if: github.event_name == 'workflow_call'
runs-on: ubuntu-latest
permissions:
contents: read
pull-requests: write
outputs:
authorized: ${{ steps.auth.outputs.authorized }}
steps:
- uses: actions/checkout@v4
with:
ref: refs/heads/main

- name: Check PR authorization
id: auth
uses: ./.github/actions/check-e2e-authorization
with:
pr_number: ${{ inputs.pr_number }}
repository: ${{ github.repository }}

e2e:
needs: gate
if: >-

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This branch predates 6aef818, which added merge_group: and a Check for e2e-relevant changes step. That step lets us require e2e in branch protection — the workflow succeeds on PRs that don't touch e2e files instead of skipping.

After rebasing, it'd be worth thinking through how that fits with the shim. The shim's paths: filter prevents the workflow from firing on non-matching PRs, so we can't make e2e a required check. If we want that option, the shim could fire unconditionally and let the relevance check inside e2e.yml decide.

Copy link
Copy Markdown
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I started working on this with the wrong assumptions and ended up with a solution that is far more comlex and cumbersome then what we actually needed. Opened PR #2106 to redo this with fresh approach

!cancelled() &&
(github.event_name != 'workflow_call' || needs.gate.result == 'skipped' || needs.gate.outputs.authorized == 'true')
runs-on: ubuntu-latest
timeout-minutes: 30
permissions:
contents: read
id-token: write
env:
# Shared e2e GCP defaults when repository secrets are unset (see ADR 0043).
E2E_GCP_PROJECT_ID: ${{ secrets.E2E_GCP_PROJECT_ID != '' && secrets.E2E_GCP_PROJECT_ID || 'it-gcp-konflux-e2e-fullsend' }}
E2E_GCP_SERVICE_ACCOUNT: ${{ secrets.E2E_GCP_SERVICE_ACCOUNT != '' && secrets.E2E_GCP_SERVICE_ACCOUNT || 'fullsend-e2e@it-gcp-konflux-e2e-fullsend.iam.gserviceaccount.com' }}
E2E_GCP_WIF_PROVIDER: ${{ secrets.E2E_GCP_WIF_PROVIDER != '' && secrets.E2E_GCP_WIF_PROVIDER || 'projects/208332380190/locations/global/workloadIdentityPools/fullsend-e2e-pool/providers/github-oidc' }}
steps:
- uses: actions/checkout@v4
with:
ref: ${{ github.event_name == 'workflow_call' && inputs.head_sha || github.sha }}

- uses: actions/setup-go@v5
with:
Expand Down Expand Up @@ -73,10 +115,11 @@ jobs:

- name: Authenticate to GCP
if: steps.secrets-check.outputs.available == 'true'
uses: google-github-actions/auth@v2
uses: google-github-actions/auth@v3
with:
workload_identity_provider: ${{ secrets.E2E_GCP_WIF_PROVIDER }}
service_account: ${{ secrets.E2E_GCP_SERVICE_ACCOUNT }}
workload_identity_provider: ${{ env.E2E_GCP_WIF_PROVIDER }}
service_account: ${{ env.E2E_GCP_SERVICE_ACCOUNT }}
project_id: ${{ env.E2E_GCP_PROJECT_ID }}

- name: Run e2e tests
if: steps.secrets-check.outputs.available == 'true'
Expand All @@ -86,13 +129,12 @@ jobs:
E2E_GITHUB_PASSWORD: ${{ secrets.E2E_GITHUB_PASSWORD }}
E2E_GITHUB_TOTP_SECRET: ${{ secrets.E2E_GITHUB_TOTP_SECRET }}
E2E_MINT_URL: ${{ secrets.E2E_MINT_URL }}
E2E_GCP_PROJECT_ID: ${{ secrets.E2E_GCP_PROJECT_ID }}

- name: Upload debug screenshots
if: always() && steps.secrets-check.outputs.available == 'true'
uses: actions/upload-artifact@v4
with:
name: e2e-screenshots
name: e2e-screenshots-${{ github.event_name == 'workflow_call' && inputs.pr_number || github.run_id }}
path: ${{ runner.temp }}/e2e-screenshots/
if-no-files-found: ignore
retention-days: 5
47 changes: 47 additions & 0 deletions .github/workflows/e2e_shim.yml
Original file line number Diff line number Diff line change
@@ -0,0 +1,47 @@
name: E2E Shim

# Lightweight PR shim: calls trusted e2e.yml@main without checking out PR code.
# See docs/ADRs/0043-e2e-wif-shim-and-pr-authorization-gate.md
#
# Security: pull_request_target runs the BASE branch version of this workflow,
# preventing PRs from modifying it to exfiltrate credentials. This shim never
# checks out PR code, so it is not vulnerable to "pwn request" attacks.

on:
pull_request_target:
types: [opened, synchronize, reopened, labeled]
paths:
- '**/*.go'
- 'go.mod'
- 'go.sum'
- 'e2e/**'
- 'internal/scaffold/fullsend-repo/**'
- 'internal/security/hooks/**'
- 'internal/dispatch/gcf/mintsrc/**'
- 'internal/sentencetoken/english.json'
- 'Makefile'
- '.github/workflows/e2e.yml'
- '.github/workflows/e2e_shim.yml'
- '.github/actions/check-e2e-authorization/**'
- 'scripts/check-e2e-authorization.sh'
- 'scripts/check-e2e-authorization_test.sh'
Comment thread
ifireball marked this conversation as resolved.

concurrency:
group: e2e-${{ github.event.pull_request.number }}
cancel-in-progress: true

jobs:
call-e2e:
if: >-
github.event.action != 'labeled' ||
github.event.label.name == 'ok-to-test'
permissions:
contents: read
Comment thread
ifireball marked this conversation as resolved.
Comment thread
ifireball marked this conversation as resolved.
id-token: write
pull-requests: write
uses: fullsend-ai/fullsend/.github/workflows/e2e.yml@refs/heads/main
with:
pr_number: ${{ github.event.pull_request.number }}
head_sha: ${{ github.event.pull_request.head.sha }}
pr_author: ${{ github.event.pull_request.user.login }}
secrets: inherit
1 change: 1 addition & 0 deletions CLAUDE.md
Original file line number Diff line number Diff line change
Expand Up @@ -13,6 +13,7 @@ Fullsend is a platform for fully autonomous agentic development for GitHub-hoste
- Always run `make lint` before submitting changes and fix any failures.
- Use [Conventional Commits](https://www.conventionalcommits.org/) for all commit messages. See [CONTRIBUTING.md](CONTRIBUTING.md#commit-messages) for the full specification. This is critical — GoReleaser uses commit prefixes to generate release notes.
- Never commit secrets (tokens, API keys, PEM keys, gcloud credentials) or sensitive data (GCP project names, service account identifiers, Model Armor template names, internal hostnames). Use environment variables with no defaults for sensitive values.
- **Exception — shared e2e test infrastructure:** The dedicated CI e2e GCP project ID (`it-gcp-konflux-e2e-fullsend`), e2e service account name (`fullsend-e2e`), and WIF pool/provider IDs (`fullsend-e2e-pool`, `github-oidc`) may appear in workflow defaults, tests, and docs. Access is bound by WIF attribute conditions (`repository`, `job_workflow_ref`) — not by hiding these identifiers. See [ADR 0043](docs/ADRs/0043-e2e-wif-shim-and-pr-authorization-gate.md) and [E2E GCP setup](docs/guides/infrastructure/e2e-gcp-setup.md).

## Go code

Expand Down
3 changes: 2 additions & 1 deletion Makefile
Original file line number Diff line number Diff line change
Expand Up @@ -25,7 +25,7 @@ help:
@echo " go-vet - Run go vet"
@echo " go-tidy - Run go mod tidy"
@echo " lint-md-links - Check markdown files for broken in-repo links and anchors"
@echo " script-test - Run shell script tests (post-triage, post-code, post-review, pre-fetch-prior-review, reconcile-repos, validate-output-schema)"
@echo " script-test - Run shell script tests (post-triage, post-code, post-review, pre-fetch-prior-review, reconcile-repos, validate-output-schema, check-e2e-authorization)"
@echo " test - Run all checks: lint-all, go-test, script-test"
@echo " e2e-test - Run admin e2e tests (requires E2E_GITHUB_SESSION_FILE or E2E_GITHUB_USERNAME + E2E_GITHUB_PASSWORD)"
@echo " e2e-export-session - Login to GitHub and export a Playwright session file"
Expand Down Expand Up @@ -116,6 +116,7 @@ script-test:
bash internal/scaffold/fullsend-repo/scripts/pre-fetch-prior-review-test.sh
python3 internal/scaffold/fullsend-repo/scripts/process-fix-result-test.py
python3 skills/topissues/scripts/topissues_test.py
bash scripts/check-e2e-authorization_test.sh

test: lint-all go-test script-test

Expand Down
2 changes: 2 additions & 0 deletions docs/ADRs/0010-stored-session-for-e2e-browser-auth.md
Original file line number Diff line number Diff line change
Expand Up @@ -146,3 +146,5 @@ it does expire, a developer runs `make e2e-upload-session` to refresh it.
every two weeks (and less often on active repos).
- Locally, developers can use username/password directly — `make e2e-test`
auto-generates a session file when credentials are available.
- CI credential gating and WIF for GCP access are described in
[ADR 0043](0043-e2e-wif-shim-and-pr-authorization-gate.md).
1 change: 1 addition & 0 deletions docs/ADRs/0040-org-pool-for-parallel-e2e-tests.md
Original file line number Diff line number Diff line change
Expand Up @@ -72,3 +72,4 @@ slice in the test code. No architectural changes are needed.
across all orgs; session export and PAT creation remain per-run.
- Pool expansion is an operational task (provision org, update one slice
literal), not an architectural change.
- CI authorization and GCP WIF for e2e workflows: [ADR 0043](0043-e2e-wif-shim-and-pr-authorization-gate.md).
Loading
Loading