ci(#2495): add fullsend bot actors to e2e trusted bot list by fullsend-ai-coder[bot] · Pull Request #5429 · fullsend-ai/fullsend

fullsend-ai-coder · 2026-07-22T00:16:58Z

Summary

Add fullsend-ai-coder[bot] and fullsend-ai-fix[bot] to the trusted bot list in the e2e authorization gate script, so bot-authored PRs touching e2e paths automatically get e2e test execution without requiring manual ok-to-test label application.

Related Issue

Closes #2495

Changes

Add fullsend-ai-coder[bot] and fullsend-ai-fix[bot] to TRUSTED_BOT_LOGINS in scripts/check-e2e-authorization.sh
Add test cases in scripts/check-e2e-authorization-test.sh verifying both new bot actors are authorized via the trusted bot path

Testing

All 33 check-e2e-authorization-test.sh tests pass, including 2 new tests for the fullsend bot actors
Existing tests unaffected — renovate bot and unknown bot tests still pass
make lint could not run (sandbox network restriction); post-script runs pre-commit authoritatively

Closes #2495

Post-script verification

Branch is not main/master (agent/2495-trusted-bot-e2e-auth)
Secret scan passed (gitleaks — 6ef9152c619fe81daeb47b3df138b2f72626d456..HEAD)
PR body secret scan passed (gitleaks — no-git)
Pre-commit hooks passed (authoritative run on runner)
Tests ran inside sandbox

The e2e authorization gate (check-e2e-authorization.sh) only recognized renovate-fullsend[bot] as a trusted bot. PRs authored by fullsend-ai-coder[bot] and fullsend-ai-fix[bot] had author_association=NONE and required manual ok-to-test label application before e2e tests would run, meaning e2e-modifying changes could merge without e2e validation. Add fullsend-ai-coder[bot] and fullsend-ai-fix[bot] to the TRUSTED_BOT_LOGINS list so the is_trusted_bot() check authorizes them for e2e test execution. Add corresponding test cases. Note: pre-commit could not run in the sandbox (git fetch blocked by network policy). The post-script runs pre-commit authoritatively on the runner. Closes #2495

fullsend-ai-review · 2026-07-22T00:18:29Z

🤖 Finished Review · ✅ Success · Started 12:18 AM UTC · Completed 12:27 AM UTC
Commit: 09717e6 · View workflow run →

github-actions · 2026-07-22T00:19:07Z

Site preview

Preview: https://bdf50570-site.fullsend-ai.workers.dev

Commit: 4c76e2940554454a983cb66c8ad51f67af12a928

fullsend-ai-review · 2026-07-22T00:27:22Z

Review

Findings

Medium

[protected-path] scripts/check-e2e-authorization.sh, scripts/check-e2e-authorization-test.sh — This PR modifies files under protected paths (scripts/). The change is authorized by linked issue Auto-apply ok-to-test label for trusted bot-authored PRs #2495 and the PR description explains the rationale. Human approval is always required for protected-path changes, regardless of context.

Labels: PR modifies CI e2e authorization scripts

codecov · 2026-07-22T00:33:35Z

Codecov Report

✅ All modified and coverable lines are covered by tests.

📢 Thoughts on this report? Let us know!

ifireball · 2026-07-22T06:07:12Z

/fs-fix

fullsend-ai-fix[bot] is not a real GitHub App identity — the fix agent reuses the coder app (slug: fullsend-ai-coder in harness/fix.yaml), so PRs from the fix agent are authored as fullsend-ai-coder[bot]. Drop fullsend-ai-fix[bot] from TRUSTED_BOT_LOGINS and from the corresponding test case in scripts/check-e2e-authorization-test.sh; keep only renovate-fullsend[bot] and fullsend-ai-coder[bot].

fullsend-ai-coder · 2026-07-22T06:08:43Z

🤖 Finished Fix · ✅ Success · Started 6:08 AM UTC · Completed 6:13 AM UTC
Commit: 7ada4e0 · View workflow run →

fullsend-ai-fix[bot] is not a real GitHub App identity — the fix agent reuses the coder app (slug: fullsend-ai-coder), so its PRs are authored as fullsend-ai-coder[bot]. Remove the phantom identity from TRUSTED_BOT_LOGINS and its corresponding test case. Addresses review feedback on #5429

fullsend-ai-coder · 2026-07-22T06:13:22Z

🔧 Fix agent — iteration 1 (human-triggered)

Removed the phantom fullsend-ai-fix[bot] identity from TRUSTED_BOT_LOGINS and its test case per human instruction. The fix agent uses fullsend-ai-coder[bot] (slug: fullsend-ai-coder), so only renovate-fullsend[bot] and fullsend-ai-coder[bot] remain. All 32 tests pass.

Fixed (2):

Remove fullsend-ai-fix[bot] from TRUSTED_BOT_LOGINS (scripts/check-e2e-authorization.sh): Removed fullsend-ai-fix[bot] from TRUSTED_BOT_LOGINS since the fix agent reuses the coder app identity (fullsend-ai-coder[bot]). Kept renovate-fullsend[bot] and fullsend-ai-coder[bot].
Remove fullsend-ai-fix[bot] test case (scripts/check-e2e-authorization-test.sh): Removed the 3-line test block for fullsend-ai-fix[bot] from the trusted bot test section. All 32 remaining tests pass.

Tests: passed

_{Updated by fullsend fix agent}

github-actions · 2026-07-22T06:13:31Z

E2E tests did not run

E2E tests run automatically for org/repo members and collaborators on pull requests.

For other contributors, a maintainer must add the ok-to-test label after the latest push.

See E2E testing guide for details.

github-actions · 2026-07-22T06:13:31Z

E2E tests did not run

E2E tests run automatically for org/repo members and collaborators on pull requests.

For other contributors, a maintainer must add the ok-to-test label after the latest push.

See E2E testing guide for details.

fullsend-ai-retro · 2026-07-22T06:40:25Z

🤖 Finished Retro · ✅ Success · Started 6:40 AM UTC · Completed 6:52 AM UTC
Commit: 4c76e29 · View workflow run →

fullsend-ai-retro · 2026-07-22T06:52:29Z

Retro: PR #5429 — Add fullsend bot actors to e2e trusted bot list

Timeline

2026-06-22 — The retro agent filed issue #2495 after analyzing PR fix(#2490): add retry logic for flaky e2e TestAdminInstallUninstall #2492, proposing that bot-authored PRs touching e2e paths should get automatic e2e test execution. The issue body mentioned both fullsend-ai-coder[bot] and fullsend-ai-fix[bot] as trusted bot actors.
2026-06-22 — Triage agent triaged #2495, repeating both bot identities in its summary.
2026-07-21 23:53 UTC — Human (ifireball) provided implementation direction: modify check-e2e-authorization.sh directly rather than using harness-level customizations. Triggered /fs-triage for an updated summary.
2026-07-22 00:10 UTC — Human triggered /fs-code. Code agent run produced a commit adding both fullsend-ai-coder[bot] and fullsend-ai-fix[bot] to TRUSTED_BOT_LOGINS with test cases for each.
2026-07-22 00:17–00:27 UTC — Review agent ran four sub-agents (correctness/opus, security/opus, intent-coherence/sonnet, style-conventions/sonnet). Only finding: medium protected-path flag for scripts/. None caught that fullsend-ai-fix[bot] is not a real GitHub App identity.
2026-07-22 05:52 UTC — Human approved the first commit.
2026-07-22 06:07 UTC — Human identified the phantom identity and triggered /fs-fix with detailed instructions: the fix agent reuses the coder app (slug: fullsend-ai-coder), so fullsend-ai-fix[bot] does not exist.
2026-07-22 06:08–06:13 UTC — Fix agent cleanly removed fullsend-ai-fix[bot] from both the script and test file. All 32 tests passed.
2026-07-22 06:37 UTC — PR merged.

Key Finding: Multi-Agent Misinformation Cascade

The phantom fullsend-ai-fix[bot] identity originated in the retro agent's issue body and propagated through every subsequent agent stage uncorrected: retro → triage → code → (review missed it) → human caught it. No agent had access to ground truth about which GitHub App identities actually exist in the org. The code agent faithfully implemented what the triage summary and issue description specified, and the review agent verified the implementation matched the issue — which it did. The error was in the premise, not the implementation.

Workflow Quality

Rework: 1 cycle (human-triggered /fs-fix), cleanly resolved
Token cost: Reasonable — review ran 4 sub-agents, code and fix each completed in ~6 minutes
Time to resolution: ~6.5 hours wall-clock (mostly human think time between midnight and early morning)
Fix agent quality: Excellent — correctly scoped the removal and verified tests

Related Existing Issues (not proposing duplicates)

#1890 — Review agent should challenge factual premises of agent-generated PRs. This retro provides supporting evidence: the review correctness sub-agent verified implementation quality but not premise correctness.
#4037 — Code agent should validate enumerated filter lists. Same pattern: adding names to an enumerated list without verifying they're valid.
#2643 — Retro agent should verify codebase-internal references. The retro agent that created Auto-apply ok-to-test label for trusted bot-authored PRs #2495 included an unverified claim about fullsend-ai-fix[bot] existing as a separate identity.
#488 — Explores implications of agents validating each other's hallucinations. This retro is a concrete instance of that pattern.

Proposals filed

Document bot GitHub App identities in AGENTS.md to prevent phantom identity references (in fullsend-ai/fullsend)

fullsend-ai-coder Bot requested a review from a team as a code owner July 22, 2026 00:16

fullsend-ai-coder Bot added the ready-for-review Triggers review agent dispatch label Jul 22, 2026

fullsend-ai-coder Bot assigned ifireball Jul 22, 2026

github-actions Bot deployed to site-preview July 22, 2026 00:19 View deployment

fullsend-ai-review Bot added requires-manual-review Review requires human judgment component/ci CI pipelines and checks component/e2e End-to-end tests labels Jul 22, 2026

ifireball approved these changes Jul 22, 2026

View reviewed changes

github-actions Bot deployed to site-preview July 22, 2026 06:15 View deployment

ifireball added this pull request to the merge queue Jul 22, 2026

Merged via the queue into main with commit 675086c Jul 22, 2026
13 checks passed

ifireball deleted the agent/2495-trusted-bot-e2e-auth branch July 22, 2026 06:37

This was referenced Jul 22, 2026

Document bot GitHub App identities in AGENTS.md to prevent phantom identity references #5434

Closed

docs(#5434): add bot identities section to AGENTS.md #5435

Merged

ifireball mentioned this pull request Jul 22, 2026

e2e: non-ok-to-test label events cancel in-progress PR e2e runs #5455

Closed

qodo-code-review Bot mentioned this pull request Jul 23, 2026

chore(eval): add rework rate tracking script for agent PRs (#5516) #5517

Closed

2 tasks

Navigation Menu

Uh oh!

ci(#2495): add fullsend bot actors to e2e trusted bot list - #5429

ci(#2495): add fullsend bot actors to e2e trusted bot list#5429
ifireball merged 2 commits into
mainfrom
agent/2495-trusted-bot-e2e-auth

fullsend-ai-coder Bot commented Jul 22, 2026

Uh oh!

fullsend-ai-review Bot commented Jul 22, 2026 •

edited

Loading

Uh oh!

github-actions Bot commented Jul 22, 2026 •

edited

Loading

Uh oh!

fullsend-ai-review Bot commented Jul 22, 2026

Uh oh!

codecov Bot commented Jul 22, 2026

Uh oh!

ifireball commented Jul 22, 2026

Uh oh!

fullsend-ai-coder Bot commented Jul 22, 2026 •

edited

Loading

Uh oh!

fullsend-ai-coder Bot commented Jul 22, 2026

Uh oh!

github-actions Bot commented Jul 22, 2026

Uh oh!

github-actions Bot commented Jul 22, 2026

Uh oh!

Uh oh!

fullsend-ai-retro Bot commented Jul 22, 2026 •

edited

Loading

Uh oh!

fullsend-ai-retro Bot commented Jul 22, 2026

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

1 participant

Uh oh!

Conversation

fullsend-ai-coder Bot commented Jul 22, 2026

Summary

Related Issue

Changes

Testing

Post-script verification

Uh oh!

fullsend-ai-review Bot commented Jul 22, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

github-actions Bot commented Jul 22, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Site preview

Uh oh!

fullsend-ai-review Bot commented Jul 22, 2026

Review

Findings

Medium

Uh oh!

codecov Bot commented Jul 22, 2026

Codecov Report

Uh oh!

ifireball commented Jul 22, 2026

Uh oh!

fullsend-ai-coder Bot commented Jul 22, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

fullsend-ai-coder Bot commented Jul 22, 2026

🔧 Fix agent — iteration 1 (human-triggered)

Uh oh!

github-actions Bot commented Jul 22, 2026

E2E tests did not run

Uh oh!

github-actions Bot commented Jul 22, 2026

E2E tests did not run

Uh oh!

Uh oh!

fullsend-ai-retro Bot commented Jul 22, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

fullsend-ai-retro Bot commented Jul 22, 2026

Retro: PR #5429 — Add fullsend bot actors to e2e trusted bot list

Timeline

Key Finding: Multi-Agent Misinformation Cascade

Workflow Quality

Related Existing Issues (not proposing duplicates)

Proposals filed

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

1 participant

fullsend-ai-review Bot commented Jul 22, 2026 •

edited

Loading

github-actions Bot commented Jul 22, 2026 •

edited

Loading

fullsend-ai-coder Bot commented Jul 22, 2026 •

edited

Loading

fullsend-ai-retro Bot commented Jul 22, 2026 •

edited

Loading