Skip to content

feat: add fullsend admin CLI with install/uninstall/analyze - #142

Merged
ralphbean merged 15 commits into
mainfrom
agent-admin-cli-clean-room
Apr 2, 2026
Merged

feat: add fullsend admin CLI with install/uninstall/analyze#142
ralphbean merged 15 commits into
mainfrom
agent-admin-cli-clean-room

Conversation

@ralphbean

@ralphbean ralphbean commented Apr 2, 2026

Copy link
Copy Markdown
Member

Summary

Implements the fullsend admin CLI to deliver the installation and bootstrap workflow described in #124. An org administrator runs a single command to onboard their organization: the CLI creates a .fullsend config repo with safe defaults, sets up CI workflows and CODEOWNERS, stores agent app credentials as secrets, and opens enrollment PRs on enabled repos — nothing changes in any repo until those PRs are reviewed and merged.

Two architectural choices shape the implementation:

  1. Forge abstraction via interfaces — All git forge operations go through a forge.Client interface, making it straightforward to add GitLab and Forgejo support alongside the current GitHub implementation.

  2. Layered install/uninstall/analyze model — Installation is decomposed into ordered layers (config-repo → workflows → secrets → enrollment), each implementing Install, Uninstall, and Analyze methods. This enables the analyze subcommand to report exactly what's installed, what's missing, and what install would do.

CLI Structure

fullsend admin install <org>    # Set up fullsend for an org
fullsend admin uninstall <org>  # Tear down fullsend from an org
fullsend admin analyze <org>    # Assess installation status

Demo

demo-reduced.mp4

Architecture

Forge Abstraction (internal/forge/)

  • forge.Client interface with 17 methods covering repos, files, branches, PRs, secrets, workflows, and app installations
  • forge.FakeClient — thread-safe test double with error injection and call recording
  • forge/github.LiveClient — GitHub REST API implementation with NaCl secret encryption

Layer Model (internal/layers/)

  • Layer interface: Name(), Install(), Uninstall(), Analyze()
  • Stack processes layers in order (install) or reverse order (uninstall)
  • ConfigRepoLayer — creates/manages the .fullsend config repository
  • WorkflowsLayer — manages CI workflow files and CODEOWNERS
  • SecretsLayer — stores agent app credentials as repo secrets/variables
  • EnrollmentLayer — creates enrollment PRs for enabled repos

Other Packages

  • internal/config/ — OrgConfig types, YAML marshal/unmarshal, validation
  • internal/appsetup/ — GitHub App manifest flow for per-role agent apps
  • internal/ui/ — Styled terminal output via lipgloss
  • internal/cli/ — Cobra CLI with admin subcommand group

Test Coverage

All packages have unit tests:

  • forge — 100% coverage (fake client with race detector)
  • config — 94.3% coverage
  • layers — 91.0% coverage (all 5 layer implementations tested)
  • forge/github — 80.5% coverage (httptest-based API tests)
  • appsetup — 41.5% (manifest flow partially tested due to browser interaction)
  • ui — 100% coverage

Verification

  • go test ./... -count=1 -cover -race — all pass
  • go vet ./... — clean
  • go build -o bin/fullsend ./cmd/fullsend/ — builds successfully

ralphbean added a commit that referenced this pull request Apr 2, 2026
Critical fixes:
- Fix XSS vulnerability in app manifest HTML form (html.EscapeString)
- Add retry with backoff on rate-limited responses (429 + 403 w/ Retry-After)
- Fix exchangeManifestCode to use context-aware HTTP client with timeout
- Add GetRepo to forge.Client for O(1) repo existence checks

Important fixes:
- Add forge.IsNotFound/ErrNotFound for structured error discrimination
- Fix runUninstall to return error when uninstall operations fail
- Secrets layer Analyze now checks both secrets and variables
- Move DefaultAgentRoles to config package to decouple CLI from forge/github
- Fix FakeClient embedding by pointer in enrollment test (mutex safety)
- Add shared forge.ConfigRepoName constant used across all packages
- Fix go.mod to use go 1.25.8

Minor fixes:
- Use errors.As in isNotFound instead of manual unwrap loop
- Add pagination to ListRepoPullRequests
- Delete pointless TestCompileTimeInterfaceCheck
- Add bin/ to .gitignore

Assisted-by: OpenCode claude-opus-4-6@default
ralphbean added 15 commits April 2, 2026 18:53
Assisted-by: OpenCode claude-opus-4-6@default
Assisted-by: OpenCode claude-opus-4-6@default
Add the ui.Printer type with lipgloss-styled methods for consistent
terminal output across the CLI: banner, headers, step indicators,
key-value pairs, summary/error boxes, and PR links. Includes full
test coverage for all 12 methods.

Assisted-by: OpenCode claude-opus-4-6@default
The forge.Client interface abstracts all git forge operations, enabling
future support for GitHub, GitLab, and Forgejo.

Assisted-by: OpenCode claude-opus-4-6@default
Implements all forge.Client methods against the GitHub REST API including
repo management, file operations, secret encryption, and workflow queries.
Includes GitHub-specific types for App configuration with role-based presets.

Assisted-by: OpenCode claude-opus-4-6@default
Handles OrgConfig types, YAML marshal/unmarshal, validation, and
helper methods for accessing enabled repos and agent slugs.

Assisted-by: OpenCode claude-opus-4-6@default
Layers represent discrete installation concerns processed in order for
install, reverse order for uninstall, and assessed individually for analyze.

Assisted-by: OpenCode claude-opus-4-6@default
Handles creation, configuration, and teardown of the org-level
.fullsend configuration repository. The layer creates the repo
(private or public based on org capability), writes config.yaml,
and provides analysis of existing installation state.

Assisted-by: OpenCode claude-opus-4-6@default
Manages reusable agent dispatch workflow, onboarding workflow, and
CODEOWNERS in the .fullsend config repo.

Assisted-by: OpenCode claude-opus-4-6@default
Stores agent app private keys as repo secrets and app IDs as repo
variables in the .fullsend config repo.

Assisted-by: OpenCode claude-opus-4-6@default
Creates enrollment PRs with shim workflow files for enabled repos
that are not yet connected to the fullsend agent pipeline.

Assisted-by: OpenCode claude-opus-4-6@default
Handles creating and installing per-role GitHub Apps using the
manifest flow, with support for reusing existing apps.

Assisted-by: OpenCode claude-opus-4-6@default
Implements fullsend admin {install,uninstall,analyze} <org> with
layer-based installation model and forge-agnostic client interface.

- Root command with Cobra, version support, and silence flags
- Admin subcommand grouping install, uninstall, and analyze
- Install: app setup, repo discovery, layer stack creation and execution
- Uninstall: confirmation prompt, layer teardown, manual cleanup hints
- Analyze: layer-by-layer status assessment with actionable reporting
- Token resolution from GH_TOKEN, GITHUB_TOKEN, or gh CLI
- Org name validation
- Dry-run mode for install preview

Assisted-by: OpenCode claude-opus-4-6@default
Critical fixes:
- Fix XSS vulnerability in app manifest HTML form (html.EscapeString)
- Add retry with backoff on rate-limited responses (429 + 403 w/ Retry-After)
- Fix exchangeManifestCode to use context-aware HTTP client with timeout
- Add GetRepo to forge.Client for O(1) repo existence checks

Important fixes:
- Add forge.IsNotFound/ErrNotFound for structured error discrimination
- Fix runUninstall to return error when uninstall operations fail
- Secrets layer Analyze now checks both secrets and variables
- Move DefaultAgentRoles to config package to decouple CLI from forge/github
- Fix FakeClient embedding by pointer in enrollment test (mutex safety)
- Add shared forge.ConfigRepoName constant used across all packages
- Fix go.mod to use go 1.25.8

Minor fixes:
- Use errors.As in isNotFound instead of manual unwrap loop
- Add pagination to ListRepoPullRequests
- Delete pointless TestCompileTimeInterfaceCheck
- Add bin/ to .gitignore

Assisted-by: OpenCode claude-opus-4-6@default
- Resolve merge conflicts with main (Makefile, .gitignore)
- Fix detect-private-key pre-commit hook failure by constructing PEM
  headers at runtime in secrets_test.go
- Fix GetFileContent base64 decoding: strip newlines from GitHub API's
  line-wrapped base64 content before decoding
- Fix enrollment layer silently swallowing API errors: now properly
  distinguishes not-found from other errors in enrollRepo and Analyze
- Fix CreateOrUpdateFile: handle non-200/404 GET responses instead of
  silently proceeding without a SHA
- Fix XSS in appsetup callback page: escape creds.Name in HTML output
- Add actions/setup-go step to CI workflow so go vet works reliably
- Fix trailing whitespace in plan doc

Assisted-by: OpenCode claude-opus-4-6@default
@ralphbean
ralphbean force-pushed the agent-admin-cli-clean-room branch from c19ff32 to d77fe3f Compare April 2, 2026 19:03
@ralphbean
ralphbean marked this pull request as ready for review April 2, 2026 19:47
@ralphbean

Copy link
Copy Markdown
Member Author

Somehow, I totally forgot about the vertex api credentials. We're going to need those.

At this point, if you're all okay with the outline of this, I'd prefer to get it merged and follow up with that detail in a patch PR on top.

@ralphbean
ralphbean added this pull request to the merge queue Apr 2, 2026
Merged via the queue into main with commit 3eae7d3 Apr 2, 2026
1 check passed
@waynesun09

Copy link
Copy Markdown
Member

Demo / code mismatch, credential scoping, and architectural concerns

1. Demo video shows v6 branch behavior, not the merged code

The demo video demonstrates workflow_dispatch + FULLSEND_DISPATCH_TOKEN (PAT) from the unmerged agent-admin-cli-clean-room-v6 branch. The merged code uses workflow_call:

Merged code (main) Demo video (v6)
Dispatch mechanism workflow_call (reusable workflow) workflow_dispatch (via gh workflow run)
Shim references secrets.FULLSEND_FULLSEND_APP_PRIVATE_KEY secrets.FULLSEND_DISPATCH_TOKEN
Agent runs in Caller's context (target repo) .fullsend repo
Shim trigger protection pull_request pull_request_target (fork-safe)
Secret type App PEM passed cross-repo Fine-grained PAT (Actions:write on .fullsend only)

The demo action run at ralphflux/test-repo/actions/runs/23916426574 shows the v6 shim dispatching to .fullsend, with the real agent workload at ralphflux/.fullsend/actions/runs/23916429888. This two-hop flow doesn't exist in the merged code.

2. Merged code: PEM stored in .fullsend repo secrets is inaccessible from target repo workflows

The merged SecretsLayer stores App PEMs as repo-level secrets on .fullsend:

// internal/layers/secrets.go
s.client.CreateRepoSecret(ctx, s.org, ".fullsend", sName, agent.PEM)

But with workflow_call, the agent workload runs in the caller's context (the target repo). The shim in acme/integration-service references:

secrets:
  APP_PRIVATE_KEY: ${{ secrets.FULLSEND_FULLSEND_APP_PRIVATE_KEY }}

This expression resolves in acme/integration-service's secret scope — it cannot see .fullsend's repo secrets. The secret will be empty at runtime.

Fix: Use CreateOrgSecret with visibility: "selected" scoped to enrolled repo IDs. The GitHub REST API supports this via PUT /orgs/{org}/actions/secrets/{name} with selected_repository_ids. The v6 branch already added CreateOrgSecret / SetOrgSecretRepos to the forge interface — those methods would apply here, just storing the PEM instead of the dispatch token.

3. v6 architecture: centralizing agent workload in .fullsend creates observability friction

The v6 workflow_dispatch model runs all agent execution in .fullsend. While this solves credential isolation (PEMs stay as .fullsend repo secrets):

  • Observability friction — target repo maintainers must navigate to .fullsend's Actions tab to see agent execution details. The GitHub Actions UI has no filter for workflow_dispatch input values, so runs from all enrolled repos are interleaved. This is mitigable with run-name: "[${{ inputs.source_repo }}] ${{ inputs.event_type }}" to make source repos visible in the list, but maintainers still need .fullsend repo access and awareness to find their runs.
  • No target-repo runner access — the agent runs on .fullsend's runners, not the target repo's self-hosted runners or environment. Running the target repo's test suite requires cloning from .fullsend's runner context, losing any repo-specific runner configuration.

4. v6 dispatch token (PAT) has lifecycle concerns

The v6 branch requires an org admin to manually create a fine-grained PAT (promptDispatchToken opens a browser for token creation):

  • Tied to a human account — if that person leaves the org or their account is disabled, the dispatch token dies and all enrolled repos stop dispatching.
  • Manual rotation — fine-grained PATs expire (max 1 year). No automated rotation path.
  • Scope creep risk — the CLI pre-fills actions=write scoped to .fullsend, but nothing prevents the admin from creating a PAT with broader permissions.
  • Audit trail — PAT usage shows up as the human user's actions in audit logs, making agent-triggered dispatches indistinguishable from human actions.

Alternative worth considering: use the GitHub App installation token (already available from the per-role Apps) to trigger the dispatch, eliminating the need for a stored PAT entirely.

ralphbean added a commit that referenced this pull request Apr 3, 2026
…n-room"

This reverts commit 3eae7d3, reversing
changes made to 3519563.
ralphbean added a commit that referenced this pull request Apr 3, 2026
…n CLI implementation

Merges the correct admin CLI implementation from PR #160 after reverting
the incorrectly merged PR #142. The v6 branch includes:
- Dispatch token layer for org-level cross-repo dispatch
- Preflight scope checks and auto-reuse of existing apps
- Idempotent layer operations
- ADRs for forge abstraction, layer model, app model, dispatch security
- GoReleaser release workflow
- Numerous bug fixes and improvements

Assisted-by: OpenCode claude-opus-4-6@default
ralphbean added a commit that referenced this pull request Apr 3, 2026
…yer)

The e2e tests were written against the simpler admin CLI from PR #142 but
need to work with the v6 admin CLI from PR #160 which adds:

- DispatchTokenLayer for org-level cross-repo dispatch secrets
- Preflight scope checks
- Additional forge.Client interface methods (org secrets, dispatch)

Changes:
- Add DispatchTokenLayer to buildTestLayerStack, runUninstall,
  runUninstallAllowNotFound, and verifyNotInstalled
- Add dispatch token verification (OrgSecretExists) to verifyInstalled
  and verifyNotInstalled
- Add E2E_DISPATCH_TOKEN env var for non-interactive dispatch token
  provisioning
- Collect enrolled repo IDs during install for dispatch token scoping
- Add cleanup of stale FULLSEND_DISPATCH_TOKEN org secret
- Second install uses empty dispatch token to exercise reuse path

Assisted-by: OpenCode claude-opus-4-6@default
ralphbean added a commit that referenced this pull request Apr 3, 2026
…n-room"

This reverts commit 3eae7d3, reversing
changes made to 3519563.
ralphbean added a commit that referenced this pull request Apr 3, 2026
Critical fixes:
- Fix XSS vulnerability in app manifest HTML form (html.EscapeString)
- Add retry with backoff on rate-limited responses (429 + 403 w/ Retry-After)
- Fix exchangeManifestCode to use context-aware HTTP client with timeout
- Add GetRepo to forge.Client for O(1) repo existence checks

Important fixes:
- Add forge.IsNotFound/ErrNotFound for structured error discrimination
- Fix runUninstall to return error when uninstall operations fail
- Secrets layer Analyze now checks both secrets and variables
- Move DefaultAgentRoles to config package to decouple CLI from forge/github
- Fix FakeClient embedding by pointer in enrollment test (mutex safety)
- Add shared forge.ConfigRepoName constant used across all packages
- Fix go.mod to use go 1.25.8

Minor fixes:
- Use errors.As in isNotFound instead of manual unwrap loop
- Add pagination to ListRepoPullRequests
- Delete pointless TestCompileTimeInterfaceCheck
- Add bin/ to .gitignore

Assisted-by: OpenCode claude-opus-4-6@default
github-merge-queue Bot pushed a commit that referenced this pull request Apr 6, 2026
…2e-clean

fix: revert PR #142, merge correct admin CLI (PR #160), add e2e tests
@ben-alkov
ben-alkov deleted the agent-admin-cli-clean-room branch April 23, 2026 16:57
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

3 participants