From e7641f38889ee16f553651db4cbc2f3f121b26cb Mon Sep 17 00:00:00 2001 From: Wayne Sun Date: Thu, 3 Sep 2026 07:13:32 -0400 Subject: [PATCH 1/3] docs(mint): add the App permission rollout runbook and the contributor rule MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit Adding a permission to a mint role is an operational rollout, not a deploy: GitHub rejects the whole installation-token request when any requested permission is ungranted, and existing installations grant nothing until an org owner Accepts the App update. The v0.40.0 packages:read rollout exercised the full flow — contributor, App owner, mint admin, installation owners, CLI users — so write it down as the standard procedure. - New guide docs/guides/infrastructure/app-permission-rollout.md: roles at a glance, the five rules that explain the ordering, one step-by-step section per persona with exact UI paths, commands and the log lines to watch, a state -> log -> token -> setup-output table, how to finish the rollout, and troubleshooting for the failure modes seen in practice. - docs/contributing/mintcore.md: "Adding a permission to a role" — the declaration sites that change together, the parity tests, the backward-compatibility rule (new permissions go in optionalRolePermissions for the rollout window), and the BREAKING CHANGE commit convention. - infrastructure-reference.md: the rollout section becomes a short mechanism paragraph plus a link; operator steps and test-pool specifics move out of the reference page. - Sidebar entry, and pointers from mint-administration and operations. Refs #6649, #6945. Assisted-by: Claude Signed-off-by: Wayne Sun --- docs/.vitepress/config.ts | 4 + docs/contributing/mintcore.md | 50 +++ docs/guides/getting-started/operations.md | 6 + .../infrastructure/app-permission-rollout.md | 286 ++++++++++++++++++ .../infrastructure-reference.md | 85 ++---- .../infrastructure/mint-administration.md | 2 + 6 files changed, 368 insertions(+), 65 deletions(-) create mode 100644 docs/guides/infrastructure/app-permission-rollout.md diff --git a/docs/.vitepress/config.ts b/docs/.vitepress/config.ts index 516d56a927..10270d8c09 100644 --- a/docs/.vitepress/config.ts +++ b/docs/.vitepress/config.ts @@ -327,6 +327,10 @@ export default defineConfig({ link: "/guides/infrastructure/infrastructure-reference", }, { text: "Mint Administration", link: "/guides/infrastructure/mint-administration" }, + { + text: "App Permission Rollout", + link: "/guides/infrastructure/app-permission-rollout", + }, { text: "Standalone Mint", link: "/guides/infrastructure/standalone-mint" }, { text: "Private Repositories", link: "/guides/infrastructure/private-repositories" }, { text: "Tracing Reference", link: "/guides/infrastructure/distributed-tracing" }, diff --git a/docs/contributing/mintcore.md b/docs/contributing/mintcore.md index f8bb71de85..4cd9c1beb8 100644 --- a/docs/contributing/mintcore.md +++ b/docs/contributing/mintcore.md @@ -252,6 +252,56 @@ boundary. Each load site constructs the appropriate accessor and passes it to `NewHandler`. Tests can pass any `PEMAccessor` implementation. +## Adding a permission to a role + +A role's GitHub App permissions are declared in three places that must change +together in one PR: + +| Declaration | File | Note | +|-------------|------|------| +| Mint role map | `canonicalRolePermissions` in `internal/mintcore/github.go` | What the mint requests when minting a token | +| GCF embed copy | `internal/dispatch/gcf/mintsrc/mintcore/github.go.embed` | Byte-identical copy; `hack/lint-mint-embed-sync` enforces it | +| App manifest | `AgentAppConfig` in `internal/forge/github/types.go` | What `fullsend github setup` asks GitHub to create the App with | + +The parity tests in `internal/forge/github/types_test.go` +(`TestAgentAppConfig_CoderMatchesMintcorePermissions` and its siblings — today +`e2e`, `scribe`, `coder` and `fix` have one) fail if the manifest and the mint +role map disagree, so the edits land together or CI is red. If the role you are +changing has no parity test yet, add one in the same PR. + +### The backward-compatibility rule + +A permission that existing installations have not granted **must** be added to +`optionalRolePermissions` in the same PR. GitHub rejects the entire +installation-token request with `422` when any requested permission is +ungranted — there is no partial downscope — so without the optional entry every +mint for that role fails on the day the mint deploys, for every installation +that has not accepted the App update yet. + +`effectiveInstallationPermissions` applies the distinction just before the token +POST: it compares the requested map with the installation's granted map, drops +ungranted permissions listed in `optionalRolePermissions` (logging +`installation permissions not granted: … dropped=…`), and returns +`ErrRequiredPermissionsMissing` for any other ungranted permission, which the +handler surfaces as `422`. When the installation lookup carries no `permissions` +map at all, the full requested set is sent and GitHub validates it. + +The optional entry is **temporary**. Once the mint logs stop showing drops for +the permission, remove it in a follow-up PR; the permission then behaves like +every other one in the role's map. The entry stays keyed by role — today +`optionalRolePermissions` has entries for `coder` and `fix`, both for +`packages`. + +### Commit conventions + +Adding a permission is a `BREAKING CHANGE` for users: App owners must update the +App registration before installations can grant it. The commit subject and PR +title carry the `!` suffix (see `COMMITS.md` at the repository root). + +The operator side of the change — updating the App registration, deploying the +mint, chasing installation owners to accept — is +[Rolling out a GitHub App permission](../guides/infrastructure/app-permission-rollout.md). + ## Interfaces vs accessors Mintcore uses two patterns for dependency injection: diff --git a/docs/guides/getting-started/operations.md b/docs/guides/getting-started/operations.md index 4e2e7d8b55..7c00df2d38 100644 --- a/docs/guides/getting-started/operations.md +++ b/docs/guides/getting-started/operations.md @@ -26,6 +26,12 @@ fullsend github set "$OWNER/$REPO" FULLSEND_GCP_REGION global | `FULLSEND_GCP_PROJECT_ID` | Repo secret | GCP project ID where Agent Platform is enabled | `my-gcp-project` | | `FULLSEND_GCP_WIF_PROVIDER` | Repo secret | Full WIF provider resource name for OIDC authentication | `projects/123456789/locations/global/...` | +### Accepting a GitHub App permission update + +GitHub emails organization owners when a fullsend App requests updated permissions — for example when the coder App starts asking for `packages:read`. Go to your org → **Settings** → **Third-party Access** → **GitHub Apps** (`https://github.com/organizations//settings/installations`), click **Configure** on the App, and use the **Review request** banner to accept. Only org **owners** see the banner; members see nothing. + +Agents keep working while the request is pending: during a rollout the mint mints tokens without the pending permission and logs the gap, so nothing breaks before you accept — accepting is what unlocks the capability the permission covers. See [the installation-owner step](../infrastructure/app-permission-rollout.md#installation-owner-accept-the-update) of the rollout runbook. + ### GitLab For GitLab repos, re-run `repos install` with updated values to converge configuration: diff --git a/docs/guides/infrastructure/app-permission-rollout.md b/docs/guides/infrastructure/app-permission-rollout.md new file mode 100644 index 0000000000..49d692df7f --- /dev/null +++ b/docs/guides/infrastructure/app-permission-rollout.md @@ -0,0 +1,286 @@ +# Rolling out a GitHub App permission + +Adding a permission to an agent role is not a code change you can just deploy. The mint asks GitHub +for the role's permissions every time it mints a token, and GitHub only hands out permissions the +**installation** has already accepted. Existing installations accept nothing automatically: an owner +of each installing organization has to click Accept. This page is the procedure that gets a new +permission from a pull request to every installation without breaking the installations that have +not caught up yet. + +The running example is `packages:read` for the `coder` role, shipped in fullsend v0.40.0 so agents +can pull from GitHub Packages. Substitute your own permission and role throughout. + +## When you need this + +Follow this page whenever a role in the mint's permission map needs a permission it did not request +before — a new GitHub API surface for an agent, a read scope for a package registry, a write scope +for a new automation. You do not need it to change *which repositories* a token covers, to add a new +role, or to change anything outside the GitHub App permission set; see +[Mint administration](mint-administration.md) for those. + +Five roles are involved, usually five different people: + +| Role | What they do | +|------|--------------| +| **Contributor** | Lands one PR that adds the permission to the mint's role map, the embedded copy of the mint, and the App manifest, and marks it optional for the duration of the rollout. | +| **App owner** | Adds the permission on the GitHub App registration, which puts a pending update in front of every installation. | +| **Mint admin** | Deploys the mint that understands the new map, confirms the deploy landed, and watches the logs for installations that have not accepted. | +| **Installation owner** | An **owner** of an organization the App is installed on. Accepts the pending update for that organization. | +| **CLI user / repo admin** | Upgrades the fullsend CLI and re-runs `fullsend github setup ` for their repository. | + +Nobody in this list blocks anybody else. That is the whole point of the mechanism: the permission +rolls out over days or weeks while every agent keeps working. + +## The rules + +These five facts explain every step below. If a step ever looks optional, come back here. + +1. **There is no partial downscope.** When the mint POSTs for an installation token, GitHub rejects + the *entire* request with `422` if any requested permission is not granted on that installation. + It does not quietly return a smaller token. A role map that asks for one ungranted permission + therefore breaks every token for that role, not just the new capability. +2. **The installation object lists accepted permissions only.** The mint looks the installation up + (`GET /orgs//installation`, `GET /repos///installation`) and reads the + `permissions` map. A pending update never appears there — only what the organization has already + accepted. This is why the App registration can be updated before the mint is redeployed without + breaking anything, and why the mint can decide what to request *before* it POSTs. +3. **Saving the registration is what starts the rollout.** When the App owner saves a new permission + on the App's Permissions & events page, GitHub marks every installation of that App as having a + pending update and emails the owners of each installing organization. New installations of an + already-updated App get the permission at install time. An existing installation's `permissions` + map flips only when an **owner** of that organization accepts. +4. **Optional permissions are dropped; everything else fails fast.** Before the token POST, the mint + compares the role's requested permissions with the installation's granted map. A permission + listed in `optionalRolePermissions` (today: `packages` for the `coder` role and for the `fix` + role) is dropped when ungranted, and the drop is logged. Any *other* ungranted permission fails + immediately with `422` and no token. The CLI mirrors this: a missing optional permission is a + warning and setup continues; a missing required permission is a setup error, exactly as it was + before this mechanism existed. If the installation lookup carries no `permissions` field at all, + the mint sends the full requested set and lets GitHub validate it. +5. **The order is fixed:** code merged → App registration updated → mint redeployed → CLI released → + ongoing outreach to installation owners. The registration comes before the mint so that + organizations which accept quickly already have the permission on the first mint that asks for + it. The CLI comes after the mint because `fullsend github setup` hard-fails on required gaps and + only downgrades a gap to a warning for the optional set — a CLI that treats a permission as + optional should not be in users' hands before the mint that agrees with it. **Never gate a deploy + on every installation having accepted.** One inactive or unreachable installation would stall the + platform forever; the mint's `dropped=` log lines are the outreach signal, not a release gate. + +## Step by step + +The subsections below are in rollout order. Each one is a different person's job. + +### Contributor: land the change + +One pull request changes the mint's role map, the embedded copy of the mint used by the Cloud +Function, and the GitHub App manifest the CLI creates apps from — and adds the permission to +`optionalRolePermissions` so installations that have not accepted keep minting tokens. The exact +files, the parity tests, and the commit conventions are in +[Adding a permission to a role](../../contributing/mintcore.md#adding-a-permission-to-a-role). This +is a breaking change for users, because App owners must update their registration, so the commit +subject and PR title carry `!`. + +Nothing happens to any installation when this merges. The mint does not use the new map until a mint +admin deploys it. + +### App owner: update the App registration + +Do this once per App set you own. For the hosted apps the coder App slug is `fullsend-ai-coder`; for +your own app set it is `-coder`. + +1. Open `https://github.com/organizations//settings/apps//permissions`. +2. Under **Repository permissions**, set the permission — for the worked example, **Packages** to + **Read-only**. +3. Fill in the optional note to users. GitHub shows it to every organization owner alongside the + request; one sentence saying what the agents need the permission for makes acceptance much + faster. +4. **Save changes**. + +What you should see: the page confirms the save, and every installation of the App now carries a +pending update. GitHub emails the owners of each organization the App is installed on. + +Repeat for any **test** App set you run. A test app set left un-updated sits on permission-drop +warnings forever and makes the rollout logs harder to read. + +The `fix` and `code` dispatch stages both mint the `coder` role and share the coder App. There is no +separate `fix` App to update. + +### Mint admin: deploy the mint and watch the logs + +Deploy the mint that carries the new role map: + +```bash +fullsend mint deploy --project= --region= +``` + +Add `--public` if the mint is a public mint. Deploy rejects mode conversion in either direction, so +redeploy with the same mode the mint already has — omitting `--public` on a public mint fails with +`existing mint is in public mode (PER_REPO_WIF_REPOS=*); redeploy with --public`. + +Then confirm the deploy actually took traffic. A green deploy is not proof: + +```bash +curl -s /health +``` + +The response carries a `commit` field, and it must be the commit you just deployed. If it still +shows the old one, see [Troubleshooting](#troubleshooting) — a deploy can create a new revision +while traffic stays pinned to an older one. + +From then on, three log lines tell you the state of every mint. The first is the rollout working as +intended — the organization has not accepted yet, so the mint dropped the optional permission and +minted a token without it: + +```text +installation permissions not granted: org="" installation_id= role="coder" dropped=packages:read; if the App already requests these permissions, Accept the pending update at https://github.com/organizations//settings/installations/; otherwise the App owner must add them first +``` + +The second is a successful mint. The `permissions` map is exactly what the token carries — check +whether the new permission is in it: + +```text +granted scope: repos=[…] permissions=map[…] repo_selection=… +``` + +The third is a hard failure for a permission that is *not* in the optional list. The same text is +returned to the caller as the body of a `422`. Expect some of these right after the deploy for roles +you did not touch; see [Troubleshooting](#troubleshooting): + +```text +failed to mint token: org= target_org= role=coder err=required permissions missing for role "coder": ; if the App already requests these permissions, Accept the pending update at https://github.com/organizations//settings/installations/; otherwise the App owner must add them first +``` + +To find lagging organizations, read the logs for the drop line. On GCP the filter shape is: + +```bash +gcloud logging read \ + 'resource.type=cloud_run_revision AND textPayload:"permissions not granted"' \ + --project= --limit=50 --freshness=7d +``` + +Swap `"permissions not granted"` for `"required permissions missing"` to find hard failures. Each +line names the organization, so the two filters give you the outreach list and the breakage list. + +### Installation owner: accept the update + +You get an email from GitHub saying the App is requesting updated permissions. To act on it: + +1. Go to your organization → **Settings** → **Third-party Access** → **GitHub Apps**, or straight to + `https://github.com/organizations//settings/installations`. +2. Click **Configure** next to the App (for the worked example, the coder App). +3. A banner at the top of the page says the App is requesting an update to its permissions. Click + **Review request**. +4. Review the listed permission change and accept it. + +**Only organization owners see this.** Members see the App page with no banner and no request — if +someone reports that the Review request is missing, check whether they are an owner. If the App is +not listed at all, it is not installed on that organization and there is nothing to accept. + +To verify from the command line, before and after: + +```bash +gh api orgs//installations \ + --jq '.installations[] | select(.app_slug=="") | .permissions' +``` + +Before accepting, the pending permission is absent from the map — the request does not show up +anywhere in the API. After accepting, it is present (for the worked example, `"packages": "read"`). +The endpoint needs organization-admin access. + +### CLI user: upgrade the CLI and re-run setup + +Upgrade the fullsend CLI to a release cut *after* the mint was deployed, then re-run setup for your +repository: + +```bash +fullsend github setup +``` + +Re-running setup with no flags is the supported update path: it refreshes the shim workflows and +reuses your existing secrets, and it leaves `.fullsend/config.yaml` alone unless you pass +`--runtime`, `--agents`, `--mint-url` or an `--inference-*` flag. + +What you see depends on the state of your organization's installation: + +- Update pending and the permission is optional — a warning, and setup finishes normally: + + ```text + app pending rollout permissions (setup continues): packages:read — if the App already requests these permissions, Accept the pending update at https://github.com/organizations//settings/installations/; otherwise the App owner must add them first + ``` + +- Already accepted — nothing about permissions is printed. +- A **required** permission is missing — a warning, then a failed setup: + + ```text + app missing permissions: — if the App already requests these permissions, Accept the pending update at https://github.com/organizations//settings/installations/; otherwise the App owner must add them first + ``` + + and the run ends with `apps have stale permissions:` listing each App. Fix it by accepting the + pending update (or having the App owner add the permission), then re-run setup. +- GitHub returned no permission data for the installation — `app : permissions not + available, skipping check`. Setup continues, and GitHub validates at mint time. + +## What you will see at run time + +| State | Mint log | Token contents | `fullsend github setup` | +|-------|----------|----------------|-------------------------| +| Update pending, permission is **optional** | `installation permissions not granted: … dropped=packages:read; …`, then `granted scope: …` | Role's permissions **minus** the pending one | `pending rollout permissions (setup continues): packages:read — …`, setup succeeds | +| Update accepted | `granted scope: … permissions=map[… packages:read …]`, no drop line | Full role permission set | Nothing printed about permissions | +| A **required** permission is ungranted | `failed to mint token: … err=required permissions missing for role "": ; …` | No token — the mint returns `422` with that message | `missing permissions: — …`, then setup fails with `apps have stale permissions:` | +| Installation lookup returned no `permissions` map | No preflight line; GitHub validates the POST | Full requested set, or GitHub's `422` | `permissions not available, skipping check` | + +## Finishing the rollout + +The `dropped=` lines are the to-do list. Each one names an organization that has not accepted; +contact its owners with the Accept URL from the log line. When the drop lines stop appearing over a +full activity cycle — long enough that quiet installations have minted at least once — the rollout +is done. + +Then land a follow-up PR that removes the permission from `optionalRolePermissions` (see +[Adding a permission to a role](../../contributing/mintcore.md#adding-a-permission-to-a-role)). It +stays in the role map; only the optional marking goes away. From that point the permission behaves +like every other one: an installation that has not granted it gets a fast, loud `422` instead of a +silently smaller token. Leaving the optional entry in place indefinitely is the failure mode to +avoid — it turns a permanent misconfiguration into a warning nobody reads. + +## Troubleshooting + +**The coder mints fine, but the agent gets a `403` from GitHub Packages later in the run.** Expected +while the rollout is in progress: the token was minted without `packages:read` because the +organization has not accepted yet. Confirm with the mint log — there will be a `dropped=packages:read` +line for that organization — and get an owner to accept. This is the one trade-off of the optional +mechanism: the failure moves from mint time to use time. + +**Right after a mint deploy, `required permissions missing` appears for roles you did not touch.** +That is old debt becoming visible, not a regression from your change. The preflight fails fast on +*any* ungranted required permission, so an App update an organization never accepted — possibly +months ago — now surfaces as a clear error instead of an opaque `422` from GitHub. Do not roll back +the deploy. Send the organization's owner to the installation page; accepting fixes it. + +**An owner says there is no Review request on the App page.** Three causes, in order of likelihood: +they are an organization member rather than an owner (only owners see the banner); the App is not +installed on that organization at all (it will not be listed under Third-party Access → GitHub +Apps); or the update was already accepted. Check with +`gh api orgs//installations --jq '.installations[] | select(.app_slug=="") | .permissions'` +— if the permission is already in the map, there is nothing left to accept. + +**`fullsend mint deploy` reported success but `/health` still shows the old commit.** On GCP a deploy +can build the image and create a new Cloud Run revision while traffic stays pinned to an older +revision, and still report success. Always `curl /health` after a deploy and compare the +`commit` field; if it is stale, inspect the service's revisions and traffic split, move traffic to +the new revision, and re-check. Tracked in +[fullsend-ai/fullsend#6945](https://github.com/fullsend-ai/fullsend/issues/6945). + +**Setup fails on a permission you expected to be optional.** Either the permission is not in +`optionalRolePermissions` for *that* role — the list is per role, and today only `coder` and `fix` +have an entry — or the CLI predates the mint that introduced the optional set. Upgrade the CLI, and +check the role named in the error against the map. If the permission genuinely needs to be optional +during a rollout, that is a code change: see +[Adding a permission to a role](../../contributing/mintcore.md#adding-a-permission-to-a-role). + +## Related + +- [Adding a permission to a role](../../contributing/mintcore.md#adding-a-permission-to-a-role) — the contributor side of this procedure. +- [Mint administration](mint-administration.md) — deploying the mint, managing roles and app sets. +- [Infrastructure reference](infrastructure-reference.md#role-permissions-matrix) — the permissions each role requests. +- [Operations](../getting-started/operations.md) — day-to-day repository and organization administration. diff --git a/docs/guides/infrastructure/infrastructure-reference.md b/docs/guides/infrastructure/infrastructure-reference.md index 409167658d..33b369ff06 100644 --- a/docs/guides/infrastructure/infrastructure-reference.md +++ b/docs/guides/infrastructure/infrastructure-reference.md @@ -124,71 +124,26 @@ App registration control the code/fix rollout for normal dispatches. ### Roll Out a GitHub App Permission -Use this sequence for any new role permission; the current example is -`packages:read` for the `coder` role (which covers both code and fix stages). -Changing the mint's role map does not update existing GitHub App installations. -GitHub rejects the entire installation-token request (`422`) when mint asks for a -permission the installation has not approved yet — there is no partial downscope. - -For shared hosted Apps (for example `fullsend-ai-coder`), the App owner adds the -permission once on the App registration; each installing org's owners must then -[Accept the update](https://docs.github.com/en/apps/using-github-apps/approving-updated-permissions-for-a-github-app). -New installations of an already-updated App receive the new permission at install -time. Self-managed App owners update their own App registration, then Accept on -their installation. - -The implementation sequence is: update `canonicalRolePermissions`, the GCF -embedded mint source, and `AgentAppConfig` together; have mint intersect the -requested role map with the installation's granted `permissions`; and have CLI -`checkPermissions` warn with the installing org's Accept URL instead of failing -**for optional permissions only**. Only permissions explicitly listed in `optionalRolePermissions` -(currently `packages` for `coder` and direct `fix`-role callers) may be omitted when ungranted — all -other permissions remain required and fail before the token POST, preserving -the pre-existing behavior where GitHub's `422` surfaced immediately. Dropped -optional permissions are logged with `org=` and `installation_id=`. The -preflight avoids the two token-creation POSTs that the earlier -packages-specific retry would incur for each lagging installation. - -When an installation lookup omits the `permissions` field, mint preserves the -requested map for compatibility with older or incomplete GitHub responses and -lets GitHub validate it at token creation time; the granted-set preflight -applies only when that map is present. - -This opt-in degradation means a caller that needs the omitted optional -permission may receive a later GitHub `403`; it does not silently drop any -other permission. Missing non-optional permissions fail once with a `422`, the -missing scopes, and guidance covering both App registration and installation -approval. - -Recommended operator order for adding **`packages:read`** to `coder` (code / fix): - -1. Add **Packages: Read-only** on the GitHub App's **Permissions & events** page - (hosted: `https://github.com/organizations/fullsend-ai/settings/apps//permissions`). - Optionally include a short note to users explaining why. -2. Update the App used by the pool installations as well, and have each - `halfsend-01` … `halfsend-12` installation owner Accept its pending update. - For the test-app setup, that is `fullsend-test-coder`; for pools using the - shared hosted App, update `fullsend-ai-coder`. Neither app set should be - left permanently on permission-drop warnings. -3. Deploy mint. Lagging installations keep authenticating; they simply omit - `packages:read` until they Accept. The preflight avoids the two-POST retry - volume that the old rollout path incurred. -4. Release the CLI after the App registration and mint change. `fullsend github setup` reports - pending **optional** permissions — those listed in `optionalRolePermissions`, - currently `packages:read` — as warnings with the installing org's Accept URL - and does not block, so a CLI release is not blocked on every installation - accepting at once. Any other missing permission is still a setup error, - exactly as before the rollout mechanism existed. -5. Tell installation owners to Accept the pending permission update (GitHub also - emails org owners), and use the mint permission logs to find lagging installs. - -Do **not** block mint or CLI deploy on every installation reporting -`packages:read` — inactive or unreachable installs would stall the platform. -Permission-drop logs and setup warnings are outreach signals during rollout, -not deploy gates. To add another permission in the future, add it to -`canonicalRolePermissions`, the matching App config and GCF embed; add it to -`optionalRolePermissions` only when it is explicitly safe to omit during -rollout. Remove that optional entry once all installations have accepted. +Changing the mint's role permission map does not update existing GitHub App +installations, and GitHub rejects the entire installation-token request (`422`) +when the mint asks for a permission an installation has not accepted — there is +no partial downscope. To keep a rollout from breaking installations that have not +accepted yet, the mint compares the role's requested permissions with the +installation's granted map before the token POST: permissions listed in +`optionalRolePermissions` (`internal/mintcore/github.go`, currently `packages` +for the `coder` and `fix` roles) are dropped when ungranted and logged as +`installation permissions not granted: … dropped=…`, while any other ungranted +permission fails immediately with `required permissions missing for role …`. +When an installation lookup omits the `permissions` field entirely, the mint +sends the full requested set and lets GitHub validate it at token creation time. +`fullsend github setup` mirrors the same split: a pending optional permission is +a warning that does not block setup, and any other missing permission is an +error. + +For the operator procedure — updating the App registration, deploying the mint, +getting installation owners to accept, and removing the optional entry once the +rollout is finished — see +[Rolling out a GitHub App permission](app-permission-rollout.md). ### Mint Security Controls diff --git a/docs/guides/infrastructure/mint-administration.md b/docs/guides/infrastructure/mint-administration.md index 728121bdce..1a8c874837 100644 --- a/docs/guides/infrastructure/mint-administration.md +++ b/docs/guides/infrastructure/mint-administration.md @@ -232,6 +232,8 @@ Opens the GitHub App manifest flow in your browser, stores the PEM in Secret Man The `fix` and `code` dispatch stages both mint the `coder` role and reuse the `coder` app — add role `coder` instead; there is no separate `fix` app or mint role to add. +Adding or changing a **permission** on an existing role is a different procedure: the App registration must be updated and each installing org's owners must accept it before the mint can request it. See [Rolling out a GitHub App permission](app-permission-rollout.md). + ### Removing a role `fullsend mint remove-role` removes a role from `ROLE_APP_IDS` and `ALLOWED_ROLES`. By default it also deletes the PEM secret from Secret Manager. Use `--keep-pem` to retain the secret for later re-registration. From 9e5f113c4d62a3c9057565cd674d83add10db6c5 Mon Sep 17 00:00:00 2001 From: Wayne Sun Date: Thu, 3 Sep 2026 07:22:41 -0400 Subject: [PATCH 2/3] docs(mint): separate the 422 body from the log line and tighten the runbook MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit Review follow-ups on the rollout runbook: show the HTTP 422 body (`{"error": …}`, message only) separately from the mint log line that carries the `failed to mint token: org=…` prefix; scope the "nobody blocks anybody" claim to acceptance and CLI upgrade, since the first three steps are ordered; make the CLI step explicitly optional for the permission itself; note that an empty permissions map is treated like a missing one; mention the Organization permissions section, personal account installations, Cloudflare Worker logs, and the --openai-* flags; promote the operations note to its own section; add the guide to the guides index. Assisted-by: Claude Signed-off-by: Wayne Sun --- docs/contributing/mintcore.md | 2 +- docs/guides/README.md | 1 + docs/guides/getting-started/operations.md | 11 ++--- .../infrastructure/app-permission-rollout.md | 44 +++++++++++++------ .../infrastructure-reference.md | 2 +- 5 files changed, 39 insertions(+), 21 deletions(-) diff --git a/docs/contributing/mintcore.md b/docs/contributing/mintcore.md index 4cd9c1beb8..b7b2cbc119 100644 --- a/docs/contributing/mintcore.md +++ b/docs/contributing/mintcore.md @@ -284,7 +284,7 @@ ungranted permissions listed in `optionalRolePermissions` (logging `installation permissions not granted: … dropped=…`), and returns `ErrRequiredPermissionsMissing` for any other ungranted permission, which the handler surfaces as `422`. When the installation lookup carries no `permissions` -map at all, the full requested set is sent and GitHub validates it. +map at all (or an empty one), the full requested set is sent and GitHub validates it. The optional entry is **temporary**. Once the mint logs stop showing drops for the permission, remove it in a follow-up PR; the permission then behaves like diff --git a/docs/guides/README.md b/docs/guides/README.md index 7415770d66..af89afdc29 100644 --- a/docs/guides/README.md +++ b/docs/guides/README.md @@ -24,6 +24,7 @@ Guides for organization owners and repository administrators who manage fullsend Advanced guides for platform operators who deploy and manage the GCP-side infrastructure (token mint, WIF, secrets). - [Mint service administration](infrastructure/mint-administration.md) — Deploying and managing the token mint (GCP or Cloudflare) +- [Rolling out a GitHub App permission](infrastructure/app-permission-rollout.md) — Adding a permission to a mint role without breaking installations that have not accepted it yet - [Standalone mint](infrastructure/standalone-mint.md) — Running the token mint as a standalone HTTP server without GCP - [Infrastructure reference](infrastructure/infrastructure-reference.md) — Token mint, WIF, and secrets deployment details - [Enabling fullsend on private repositories](infrastructure/private-repositories.md) — Additional guardrails and configuration for private repos diff --git a/docs/guides/getting-started/operations.md b/docs/guides/getting-started/operations.md index 7c00df2d38..7ebef766ba 100644 --- a/docs/guides/getting-started/operations.md +++ b/docs/guides/getting-started/operations.md @@ -26,11 +26,6 @@ fullsend github set "$OWNER/$REPO" FULLSEND_GCP_REGION global | `FULLSEND_GCP_PROJECT_ID` | Repo secret | GCP project ID where Agent Platform is enabled | `my-gcp-project` | | `FULLSEND_GCP_WIF_PROVIDER` | Repo secret | Full WIF provider resource name for OIDC authentication | `projects/123456789/locations/global/...` | -### Accepting a GitHub App permission update - -GitHub emails organization owners when a fullsend App requests updated permissions — for example when the coder App starts asking for `packages:read`. Go to your org → **Settings** → **Third-party Access** → **GitHub Apps** (`https://github.com/organizations//settings/installations`), click **Configure** on the App, and use the **Review request** banner to accept. Only org **owners** see the banner; members see nothing. - -Agents keep working while the request is pending: during a rollout the mint mints tokens without the pending permission and logs the gap, so nothing breaks before you accept — accepting is what unlocks the capability the permission covers. See [the installation-owner step](../infrastructure/app-permission-rollout.md#installation-owner-accept-the-update) of the rollout runbook. ### GitLab @@ -47,6 +42,12 @@ fullsend repos install -f repos.yaml "$OWNER/$REPO" \ | `FULLSEND_GCP_PROJECT_ID` | CI/CD secret | GCP project ID for inference | `my-gcp-project` | | `FULLSEND_GCP_WIF_PROVIDER` | CI/CD secret | WIF provider resource name for inference | `projects/123456789/locations/global/...` | +## Accepting a GitHub App permission update + +GitHub emails organization owners when a fullsend App requests updated permissions — for example when the coder App starts asking for `packages:read`. Go to your org → **Settings** → **Third-party Access** → **GitHub Apps** (`https://github.com/organizations//settings/installations`), click **Configure** on the App, and use the **Review request** banner to accept. Only org **owners** see the banner; members see nothing. + +Agents keep working while the request is pending: during a rollout the mint mints tokens without the pending permission and logs the gap, so nothing breaks before you accept — accepting is what unlocks the capability the permission covers. See [the installation-owner step](../infrastructure/app-permission-rollout.md#installation-owner-accept-the-update) of the rollout runbook. + ## Syncing workflow templates After upgrading the fullsend CLI, re-run `github setup` to update the workflow file for a single repo: diff --git a/docs/guides/infrastructure/app-permission-rollout.md b/docs/guides/infrastructure/app-permission-rollout.md index 49d692df7f..fa0ae1236f 100644 --- a/docs/guides/infrastructure/app-permission-rollout.md +++ b/docs/guides/infrastructure/app-permission-rollout.md @@ -28,8 +28,10 @@ Five roles are involved, usually five different people: | **Installation owner** | An **owner** of an organization the App is installed on. Accepts the pending update for that organization. | | **CLI user / repo admin** | Upgrades the fullsend CLI and re-runs `fullsend github setup ` for their repository. | -Nobody in this list blocks anybody else. That is the whole point of the mechanism: the permission -rolls out over days or weeks while every agent keeps working. +The first three steps happen in a fixed order (code, then App registration, then mint deploy). After +that, nobody waits for anybody: installation owners accept whenever they get to it, CLI users upgrade +whenever they like, and every agent keeps working in the meantime. That is the whole point of the +mechanism — the permission rolls out over days or weeks without a flag day. ## The rules @@ -55,8 +57,8 @@ These five facts explain every step below. If a step ever looks optional, come b role) is dropped when ungranted, and the drop is logged. Any *other* ungranted permission fails immediately with `422` and no token. The CLI mirrors this: a missing optional permission is a warning and setup continues; a missing required permission is a setup error, exactly as it was - before this mechanism existed. If the installation lookup carries no `permissions` field at all, - the mint sends the full requested set and lets GitHub validate it. + before this mechanism existed. If the installation lookup carries no `permissions` field at all + (or an empty one), the mint sends the full requested set and lets GitHub validate it. 5. **The order is fixed:** code merged → App registration updated → mint redeployed → CLI released → ongoing outreach to installation owners. The registration comes before the mint so that organizations which accept quickly already have the permission on the first mint that asks for @@ -89,8 +91,9 @@ Do this once per App set you own. For the hosted apps the coder App slug is `ful your own app set it is `-coder`. 1. Open `https://github.com/organizations//settings/apps//permissions`. -2. Under **Repository permissions**, set the permission — for the worked example, **Packages** to - **Read-only**. +2. Find the permission in the section GitHub files it under — **Repository permissions** for most + (the worked example: **Packages** → **Read-only**), **Organization permissions** for org-level + ones such as organization projects — and set the level. 3. Fill in the optional note to users. GitHub shows it to every organization owner alongside the request; one sentence saying what the agents need the permission for makes acceptance much faster. @@ -142,14 +145,21 @@ whether the new permission is in it: granted scope: repos=[…] permissions=map[…] repo_selection=… ``` -The third is a hard failure for a permission that is *not* in the optional list. The same text is -returned to the caller as the body of a `422`. Expect some of these right after the deploy for roles -you did not touch; see [Troubleshooting](#troubleshooting): +The third is a hard failure for a permission that is *not* in the optional list. Expect some of these +right after the deploy for roles you did not touch; see [Troubleshooting](#troubleshooting). In the +mint log it looks like this: ```text failed to mint token: org= target_org= role=coder err=required permissions missing for role "coder": ; if the App already requests these permissions, Accept the pending update at https://github.com/organizations//settings/installations/; otherwise the App owner must add them first ``` +The caller receives HTTP `422` with only the message part as the `error` field — no +`failed to mint token:` prefix and no `org=` fields: + +```json +{"error": "required permissions missing for role \"coder\": ; if the App already requests these permissions, Accept the pending update at https://github.com/organizations//settings/installations/; otherwise the App owner must add them first"} +``` + To find lagging organizations, read the logs for the drop line. On GCP the filter shape is: ```bash @@ -160,6 +170,8 @@ gcloud logging read \ Swap `"permissions not granted"` for `"required permissions missing"` to find hard failures. Each line names the organization, so the two filters give you the outreach list and the breakage list. +For a Cloudflare Worker mint, the same lines appear in the Worker logs (`npx wrangler tail +` or the dashboard's Logs view); grep for the same two phrases. ### Installation owner: accept the update @@ -174,7 +186,9 @@ You get an email from GitHub saying the App is requesting updated permissions. T **Only organization owners see this.** Members see the App page with no banner and no request — if someone reports that the Review request is missing, check whether they are an owner. If the App is -not listed at all, it is not installed on that organization and there is nothing to accept. +not listed at all, it is not installed on that organization and there is nothing to accept. For an +App installed on a personal account rather than an organization, the equivalent page is +`https://github.com/settings/installations`. To verify from the command line, before and after: @@ -189,8 +203,10 @@ The endpoint needs organization-admin access. ### CLI user: upgrade the CLI and re-run setup -Upgrade the fullsend CLI to a release cut *after* the mint was deployed, then re-run setup for your -repository: +Nothing in this step is needed for tokens to gain the permission — that happens the moment your +organization's owner accepts. This step refreshes the repository's shim workflows to the release that +knows about the new permission and shows you where your installation stands. Upgrade the fullsend CLI +to a release cut *after* the mint was deployed, then re-run setup for your repository: ```bash fullsend github setup @@ -198,7 +214,7 @@ fullsend github setup Re-running setup with no flags is the supported update path: it refreshes the shim workflows and reuses your existing secrets, and it leaves `.fullsend/config.yaml` alone unless you pass -`--runtime`, `--agents`, `--mint-url` or an `--inference-*` flag. +`--runtime`, `--agents`, `--mint-url`, an `--inference-*` or an `--openai-*` flag. What you see depends on the state of your organization's installation: @@ -226,7 +242,7 @@ What you see depends on the state of your organization's installation: |-------|----------|----------------|-------------------------| | Update pending, permission is **optional** | `installation permissions not granted: … dropped=packages:read; …`, then `granted scope: …` | Role's permissions **minus** the pending one | `pending rollout permissions (setup continues): packages:read — …`, setup succeeds | | Update accepted | `granted scope: … permissions=map[… packages:read …]`, no drop line | Full role permission set | Nothing printed about permissions | -| A **required** permission is ungranted | `failed to mint token: … err=required permissions missing for role "": ; …` | No token — the mint returns `422` with that message | `missing permissions: — …`, then setup fails with `apps have stale permissions:` | +| A **required** permission is ungranted | `failed to mint token: … err=required permissions missing for role "": ; …` | No token — the mint returns `422` whose `error` field is the message after `err=` | `missing permissions: — …`, then setup fails with `apps have stale permissions:` | | Installation lookup returned no `permissions` map | No preflight line; GitHub validates the POST | Full requested set, or GitHub's `422` | `permissions not available, skipping check` | ## Finishing the rollout diff --git a/docs/guides/infrastructure/infrastructure-reference.md b/docs/guides/infrastructure/infrastructure-reference.md index 33b369ff06..045fc72317 100644 --- a/docs/guides/infrastructure/infrastructure-reference.md +++ b/docs/guides/infrastructure/infrastructure-reference.md @@ -134,7 +134,7 @@ installation's granted map before the token POST: permissions listed in for the `coder` and `fix` roles) are dropped when ungranted and logged as `installation permissions not granted: … dropped=…`, while any other ungranted permission fails immediately with `required permissions missing for role …`. -When an installation lookup omits the `permissions` field entirely, the mint +When an installation lookup omits the `permissions` field entirely (or returns it empty), the mint sends the full requested set and lets GitHub validate it at token creation time. `fullsend github setup` mirrors the same split: a pending optional permission is a warning that does not block setup, and any other missing permission is an From dfdf5f2659b5508c25955aa0d69e1227f54a106e Mon Sep 17 00:00:00 2001 From: Wayne Sun Date: Thu, 3 Sep 2026 07:23:42 -0400 Subject: [PATCH 3/3] docs(mint): add a Before-you-start list to the rollout runbook State the access and tooling each persona needs before their step, and tell readers to jump to their own section. Assisted-by: Claude Signed-off-by: Wayne Sun --- .../infrastructure/app-permission-rollout.md | 16 ++++++++++++++++ 1 file changed, 16 insertions(+) diff --git a/docs/guides/infrastructure/app-permission-rollout.md b/docs/guides/infrastructure/app-permission-rollout.md index fa0ae1236f..718639deca 100644 --- a/docs/guides/infrastructure/app-permission-rollout.md +++ b/docs/guides/infrastructure/app-permission-rollout.md @@ -28,6 +28,22 @@ Five roles are involved, usually five different people: | **Installation owner** | An **owner** of an organization the App is installed on. Accepts the pending update for that organization. | | **CLI user / repo admin** | Upgrades the fullsend CLI and re-runs `fullsend github setup ` for their repository. | +Each step-by-step section below is written for one of these people; find your row and skip to that +section. + +### Before you start + +- **Contributor**: a fullsend checkout with Go and `make lint` working; write access to open a PR. +- **App owner**: an **owner** of the GitHub organization that owns the App registration (for the + hosted set, `fullsend-ai`; for a self-managed set, your own org). +- **Mint admin**: the fullsend CLI at the release that carries the change, deploy rights on the + mint's GCP project or Cloudflare account (see [Mint administration](mint-administration.md)), and + read access to its logs. +- **Installation owner**: an **owner** of each organization the App is installed on. Members + cannot see or accept permission requests. +- **CLI user / repo admin**: admin on the repository, `gh` authenticated, and the fullsend CLI at a + release cut after the mint deploy. + The first three steps happen in a fixed order (code, then App registration, then mint deploy). After that, nobody waits for anybody: installation owners accept whenever they get to it, CLI users upgrade whenever they like, and every agent keeps working in the meantime. That is the whole point of the