From ca1952054237c7f93128c12267e6572808248652 Mon Sep 17 00:00:00 2001 From: Barak Korren Date: Mon, 25 May 2026 13:08:05 +0300 Subject: [PATCH 1/8] docs: add public community mint ADR and hosting spike ADR 0040 records deployment, security, monitoring, scaling, and operations for the community mint profile; the spike holds platform trade-offs from issue #915. Co-authored-by: Cursor Signed-off-by: Barak Korren --- ...0040-public-community-mint-architecture.md | 89 ++++++++++++++ ...ommunity-mint-hosting-gcp-vs-cloudflare.md | 114 ++++++++++++++++++ 2 files changed, 203 insertions(+) create mode 100644 docs/ADRs/0040-public-community-mint-architecture.md create mode 100644 docs/spikes/2026-05-25-community-mint-hosting-gcp-vs-cloudflare.md diff --git a/docs/ADRs/0040-public-community-mint-architecture.md b/docs/ADRs/0040-public-community-mint-architecture.md new file mode 100644 index 0000000000..85c0328ac0 --- /dev/null +++ b/docs/ADRs/0040-public-community-mint-architecture.md @@ -0,0 +1,89 @@ +--- +title: "40. Public community mint architecture" +status: Proposed +relates_to: + - agent-infrastructure + - security-threat-model + - operational-observability +topics: + - identity + - oidc + - github-apps + - deployment + - operations +--- + +# 40. Public community mint architecture + +Date: 2026-05-25 + +## Status + +Proposed + + + +## Context + +[ADR 0029](0029-central-token-mint-secretless-fullsend.md) defines a **community deployment profile**: public (unlisted) shared GitHub Apps per role, keys held only at a **central mint**, and adopters that trust a stable `FULLSEND_MINT_URL` instead of bespoke per-org Apps. [#914](https://github.com/fullsend-ai/fullsend/issues/914) covers shared App registration and install UX; this ADR records **how the public mint is built, secured, operated, and scaled**. + +Hosting and platform trade-offs (GCP vs Cloudflare, cost, dual-console ops, JWKS port) are documented in the spike [Community token mint hosting (GCP vs Cloudflare)](../spikes/2026-05-25-community-mint-hosting-gcp-vs-cloudflare.md) ([#915](https://github.com/fullsend-ai/fullsend/issues/915)). This ADR states the **architecture**; the spike remains the reference for **platform choice rationale and phasing**. + +## Options + +- **Defer a dedicated public profile** and reuse an internal Red Hat mint project — rejected: violates isolation, blast-radius, and community trust boundaries. +- **Imperative CLI deploy** (`fullsend admin install` / GCF provisioner) for the public mint — rejected for production: no enforced review or audit trail ([#1263](https://github.com/fullsend-ai/fullsend/issues/1263)). +- **Long-term GCP + Cloudflare split edge** (GCF origin + proxied hostname) as steady state — rejected as default ongoing ops model; see spike (dual dashboards, $0 budget). Acceptable only as a **short bridge** if funded. + +## Decision + +The **public community mint** is a **stateless, internet-facing** HTTPS service that implements the existing `POST /v1/token` contract ([mint-token action](../../.github/actions/mint-token/action.yml), [infrastructure reference](../guides/admin/infrastructure-reference.md)). Adopters configure only **`FULLSEND_MINT_URL`** and OIDC audience; **hosting location is opaque** to consumers. + +### Deployment + +1. **Launch posture:** GCP Cloud Function (`fullsend-mint`), Go implementation in `internal/mint`, deployed only via **GitOps** ([#1263](https://github.com/fullsend-ai/fullsend/issues/1263))—not the tenant CLI provisioner. +2. **Dedicated GCP project** for the community profile: mint function, WIF pool/providers, and Secret Manager for shared App PEMs. **No** Vertex inference, LLM credentials, or internal Red Hat mint resources in that project. +3. **Install modes:** **Org-level** (default WIF provider, `.fullsend` / upstream `fullsend-ai/fullsend` workflow refs) and **per-repo** (`PER_REPO_WIF_REPOS`, repo-scoped providers)—both on the same public mint. +4. **Steady-state hardening (target):** port mint to **Cloudflare Workers** with **JWKS**-based GitHub OIDC validation and the same claim policy, via pluggable `TokenValidator` / `PEMAccessor`—one operational surface for edge + compute. Details and phasing: [hosting spike](../spikes/2026-05-25-community-mint-hosting-gcp-vs-cloudflare.md). +5. **Tenant-operated mints** remain separate: self-managed orgs may use the CLI/GCF provisioner on GCP today; Cloudflare may become an optional **tenant** target when the Worker implementation exists. Tenant hosting choices do not affect public consumers. + +### Security + +1. **Trust model ([ADR 0029](0029-central-token-mint-secretless-fullsend.md)):** Callers present GitHub Actions OIDC JWTs; the mint returns **short-lived, org-scoped** installation tokens with **role minimum permissions**—never the App’s full grant. +2. **Launch validation path:** GitHub JWT → **GCP STS** exchange against **Workload Identity Federation** (CEL-bound org/repo/workflow) plus in-function **fail-closed** checks (`iss`, `aud`, `job_workflow_ref`, `ALLOWED_ORGS`, `ALLOWED_WORKFLOW_FILES`, per-repo provider routing). +3. **No auth proxy** in front of the mint; abuse defense at launch is **allowlists** and monitoring. **WAF / rate limits** arrive with the Worker steady state (or a **time-boxed** CF bridge only if budget and ops capacity exist—see spike). +4. **Secrets:** Shared community App PEMs **only** in Secret Manager for the community project (layout per [#914](https://github.com/fullsend-ai/fullsend/issues/914)); never in adopters’ `.fullsend` repos. +5. **Blast radius:** Compromise of the public mint affects **all orgs** on that profile; mitigations include GitOps-only deploys, monitoring, key rotation, narrow App installations, and human branch protections—not repo-stored PEMs. + +### Monitoring + +1. **SLO owner:** **Red Hat Fullsend Bootstrap** until community operations assumes on-call. +2. **Targets:** **99.5%** monthly availability for `POST /v1/token` (excluding GitHub OIDC/API outages); **p95 < 2s** mint latency. +3. **Signals (GCP launch):** GCF 5xx and latency, STS/Secret Manager errors, synthetic unauthenticated `POST /v1/token` (expect 401), GitOps audit for allowlist/env changes. Optional deploy notifications ([#1262](https://github.com/fullsend-ai/fullsend/issues/1262)). +4. **Signals (Worker steady state):** Worker errors, WAF blocks, same synthetic on the public URL—single-console triage per spike. +5. **Runbook:** mint health → edge (when present) → GitHub OIDC status; prefer **managed** WAF rules over custom policy authoring. + +### Scaling + +1. **Workload shape:** Stateless request/response; roughly **one mint call per agent job** that needs forge tokens—low QPS at community scale, bursty with Actions concurrency. +2. **Horizontal scaling:** Cloud Functions and Workers scale instances automatically; no mint-side session store. +3. **Limits:** Enforce existing caps (e.g. `repos` list size, request body size); tune **rate limits** at the edge when Scenario 3 (or a funded bridge) is live. +4. **Cost:** Community budget is **~$0**; launch stays on GCP **free tiers** without Cloud Armor + external HTTPS LB; steady-state Workers **Free** tier at expected volume unless traffic or bundle limits require Paid—see spike. + +### Operations + +1. **Change control:** All production changes through **GitOps** ([#1263](https://github.com/fullsend-ai/fullsend/issues/1263)); no individual `gcloud functions deploy` to prod. +2. **Incident response:** Revoke or rotate shared App keys, tighten `ALLOWED_ORGS` / workflow allowlists, disable profile in enrollment docs; Bootstrap team pages until community handoff. +3. **Portability:** Maintain **multiple mint hosts** long-term; second `TokenValidator` (JWKS) and parity tests (STS ≡ JWKS) are strategic work, not launch blockers for [#914](https://github.com/fullsend-ai/fullsend/issues/914). +4. **Documentation:** Platform comparison, OIDC STS vs JWKS, cost, and dashboard trade-offs live in the [hosting spike](../spikes/2026-05-25-community-mint-hosting-gcp-vs-cloudflare.md)—update the spike when platform facts change; update this ADR only when the **architectural commitments** change. + +## Consequences + +- Community adopters get a **stable, reviewed, isolated** mint path aligned with ADR 0029 without depending on Red Hat-internal infrastructure. +- Bootstrap team carries **SLO and incident** responsibility with **single-console** ops at launch; Worker migration reduces edge cost and dual-dashboard risk when executed. +- **~$0 budget** forbids LB+Armor as the primary hardening path; security at launch leans on **OIDC binding and allowlists** until Workers or funded edge exists. +- GitOps and a dedicated project add **operational overhead** up front but improve auditability versus CLI deploys. +- Normative follow-ons remain: shared App definitions ([#914](https://github.com/fullsend-ai/fullsend/issues/914)), GitOps repo layout ([#1263](https://github.com/fullsend-ai/fullsend/issues/1263)), JWKS parity CI, and criteria for transferring SLO ownership to community ops. diff --git a/docs/spikes/2026-05-25-community-mint-hosting-gcp-vs-cloudflare.md b/docs/spikes/2026-05-25-community-mint-hosting-gcp-vs-cloudflare.md new file mode 100644 index 0000000000..31eaf2bc0a --- /dev/null +++ b/docs/spikes/2026-05-25-community-mint-hosting-gcp-vs-cloudflare.md @@ -0,0 +1,114 @@ +# SPIKE: Community token mint hosting (GCP vs Cloudflare) + +**Issue:** [#915](https://github.com/fullsend-ai/fullsend/issues/915) · **Parent:** [#914](https://github.com/fullsend-ai/fullsend/issues/914) · **Epic:** [#912](https://github.com/fullsend-ai/fullsend/issues/912) +**ADR:** [0029](../ADRs/0029-central-token-mint-secretless-fullsend.md) · [0040](../ADRs/0040-public-community-mint-architecture.md) (architecture) · **GitOps:** [#1263](https://github.com/fullsend-ai/fullsend/issues/1263) · **Date:** 2026-05-25 + +## Decision + +**Launch the public mint on Scenario 1** (100% GCP, dedicated project, [GitOps](#public-mint-operations) per [#1263](https://github.com/fullsend-ai/fullsend/issues/1263)), with **org and per-repo** modes unchanged. + +**Do not treat Scenario 2 (GCP + Cloudflare) as the long-term default.** Two consoles (GCP origin + Cloudflare edge) are ongoing human cost for the **Red Hat Fullsend Bootstrap** team (no dedicated SRE). That cost is **not** offset by saving a one-time port—re-platforming is largely AI-driven; **monitoring and incidents are human-driven**. + +**Steady-state target for a hardened public edge: Scenario 3** (100% Cloudflare Workers), not Scenario 2—**one operational surface** (Workers + WAF + alerts) while preserving the same `POST /v1/token` contract. Scenario 2 remains a **short bridge** only if WAF/rate limits are needed before the Worker port lands. + +Public mint consumers only use **`FULLSEND_MINT_URL`**; hosting is opaque. **Self-managed tenant** mints stay on GCP via the CLI provisioner today; Cloudflare becomes an optional tenant target when Scenario 3 exists. + +## Scenarios + +| # | Posture | Operator sees | +|---|---------|---------------| +| **1** | GCF + WIF + Secret Manager in a **mint-only** GCP project | **One** primary stack (Cloud Monitoring / Logging); edge via Cloud Armor + external HTTPS LB if required | +| **2** | Same GCF origin; public URL is **Cloudflare-proxied** | **Two** stacks: GCP (origin health, STS, SM) + Cloudflare (WAF, blocks, 5xx at edge) | +| **3** | Mint on **Workers**; OIDC via JWKS + same claim rules; PEMs in Worker secrets (or external vault) | **One** primary stack (Cloudflare); GitOps via Wrangler | + +Today’s code: native Go GCF (`internal/mint`), pluggable `TokenValidator` / `PEMAccessor`, tenant deploy via `internal/dispatch/gcf`, contract in [mint-token](../../.github/actions/mint-token/action.yml) and [infrastructure reference](../guides/admin/infrastructure-reference.md). + +### OIDC trust: STS/WIF (today) vs JWKS (Scenario 3) + +GitHub Actions sends the mint a short-lived **OIDC JWT** in the `Authorization` header. The mint must prove the token is genuine and matches policy (`job_workflow_ref`, allowed orgs/workflows, etc.). + +| Approach | Used in | What it means | +|----------|---------|----------------| +| **STS + WIF** | Scenario 1 (today) | The mint sends the JWT to **GCP Security Token Service**, which validates it against a **Workload Identity Federation** pool (CEL rules on repo/org). GCP returns a federated token; the mint also decodes and checks claims in Go. Trust is anchored in **GCP**. | +| **JWKS** | Scenario 3 (target) | **JWKS** = *JSON Web Key Set*: the public signing keys GitHub publishes (e.g. `https://token.actions.githubusercontent.com/.well-known/jwks`). The mint **verifies the JWT signature** against those keys locally (no GCP STS call), then runs the **same** claim checks in application code. Trust is anchored in **GitHub’s keys + mint logic**. | + +A Scenario 3 port is often called “STS→JWKS” because the **authorization outcome** should match; only the **validation backend** changes. Per-repo WIF provider selection becomes equivalent routing in mint config, not a second GCP pool lookup at request time. + +## Integrated evaluation + +All requirements below apply together—not as a separate “constraints” checklist. + +| Factor | 1 — GCP | 2 — GCP + CF | 3 — CF | +|--------|---------|--------------|--------| +| **Launch speed** | ● Shipping binary + GitOps | ● + DNS/WAF rules | ◐ Worker port + parity tests | +| **Public vs tenant deploy** | GitOps ([#1263](https://github.com/fullsend-ai/fullsend/issues/1263)); CLI for tenants | Same | GitOps (Wrangler); CLI for tenants later | +| **Isolated from inference/LLM** | ● Dedicated GCP project only | ● Same project; CF zone for mint host | ● No Vertex in mint project; don’t colocate inference Workers | +| **Org + per-repo WIF** | ● Today | ● Same origin | ◐ JWKS path must keep per-repo provider logic | +| **ADR 0029 trust** | ● STS + WIF + handler claims | ● Unchanged at origin | ◐ Prove JWKS path ≡ STS path in CI | +| **Internet-facing abuse** | ◐ Armor+LB setup (still one vendor) | ● Easy WAF/RL | ● WAF/RL in same console as compute | +| **Ongoing human ops** | ● **Single dashboard**; paging from GCP | ○ **Dual dashboard**; split incident triage | ● **Single dashboard**; paging from CF | +| **One-time engineering** | Low | Low–medium | Medium (AI-friendly port behind existing interfaces) | +| **Long-term multi-host** | ◐ GCP-centric | ◐ Split | ● Adds non-GCP option for tenants/public | +| **Blocks [#914](https://github.com/fullsend-ai/fullsend/issues/914)** | No | No | No if launch on 1 first | +| **Cost (~$0 community budget)** | ● Bare GCF+WIF+SM | ○ Free CF = weak edge; paid CF breaks budget | ● Workers Free at community volume | + +### Cost (~$0 community budget) + +Community mint traffic is **low** (roughly one `POST /v1/token` per agent job batch), so **compute** is not the cost driver—**fixed-price edge SKUs** are. + +| Posture | Typical spend at community scale | Fits ~$0? | +|---------|----------------------------------|-----------| +| **Scenario 1 — bare GCF + WIF + SM** | Stays within GCP free tiers for invocations, federation, and SM access at expected volume | **Yes** — best match for launch | +| **Scenario 1 — Cloud Armor + external HTTPS LB** | LB has **baseline monthly cost** even at zero mint traffic | **No** — rules out “hardened GCP edge” on a $0 budget | +| **Scenario 2 — GCF + Cloudflare** | GCF ~$0; **meaningful** WAF/rate limits usually need **paid** CF (Pro/Business), not Free | **Poor value on $0**: dual dashboards + thin Free-tier rules | +| **Scenario 3 — Workers** | Workers **Free** tier is sufficient at community QPS; **Workers Paid** only if volume or bundle limits grow later | **Yes** for steady state without GCP LB fees | + +**Effect on the decision (with ops and security):** + +- **Reinforces** launch on **Scenario 1** without Armor/LB—allowlists and GCP alerts are the $0 edge story until something else is funded or Scenario 3 ships. +- **Discourages** a **long-lived Scenario 2** bridge unless a CF plan is donated; otherwise you pay in **operator time** (two consoles) without buying real protection. +- **Keeps Scenario 3** as the **$0 steady-state** path for WAF/rate limits **plus** one dashboard once the JWKS port is done—it does **not** force skipping Scenario 1 at launch (both can be ~$0 for compute; the port is engineering time, not subscription). + +### Does two dashboards push toward 1 or 3? + +**Toward 1 or 3—not 2.** + +- **Scenario 2** optimizes **edge convenience** at the price of **permanent split-brain ops**: Bootstrap must correlate GCF 5xx/latency with Cloudflare origin errors and WAF blocks; synthetic checks should hit the public hostname *and* the origin; runbooks always have two hops. Alert fan-in to one pager helps but does not remove the second UI for tuning and incidents. +- **Scenario 1** keeps **one vendor console** for mint health. The tradeoff is weaker **default** edge on a bare `cloudfunctions.net` URL—mitigate at launch with fail-closed env allowlists (`ALLOWED_ORGS`, `ALLOWED_WORKFLOW_FILES`) and GCP alerts. **Cloud Armor + LB** is the one-dashboard GCP hardening option but **not** on a ~$0 budget (see [Cost](#cost-0-community-budget)); funded abuse response or **Scenario 3** is the realistic hardening path. +- **Scenario 3** is the way to get **strong edge + single dashboard** without Scenario 2’s ops tax. The STS→JWKS port is a **one-time** cost; Bootstrap’s recurring load is CF-only. Prefer this over staying on Scenario 2 indefinitely. + +**Scenario 2 is justified only as a time-boxed bridge** (weeks, not years): public hostname needs WAF before Workers ship, and Bootstrap accepts dual-console overhead temporarily. + +## Public mint operations + +- **Deploy:** GitOps ([#1263](https://github.com/fullsend-ai/fullsend/issues/1263))—no prod `gcloud functions deploy` by individuals; optional [#1262](https://github.com/fullsend-ai/fullsend/issues/1262) deploy notifications. +- **Project:** Community mint **only**—no shared GCP project with Vertex/inference or internal Red Hat mint resources. +- **SLO owner:** Red Hat Fullsend Bootstrap until community ops exists. Target: **99.5%** availability for `POST /v1/token` (excl. GitHub outages), **p95 < 2s**. +- **Signals (Scenario 1):** GCF 5xx/latency, STS/SM errors, synthetic `POST /v1/token` (expect 401), allowlist-change audit via GitOps. +- **Signals (Scenario 3):** Worker errors, WAF blocks, same synthetic on public URL—no origin correlation. +- **Portability:** Second `TokenValidator` (JWKS) and `PEMAccessor`; parity tests STS vs JWKS—strategic, not launch-blocking. + +## Phasing + +| Phase | Choice | Rationale | +|-------|--------|-----------| +| **Launch** | **Scenario 1** + GitOps + isolated project | Fastest; one console; unblocks [#914](https://github.com/fullsend-ai/fullsend/issues/914) | +| **Bridge (optional)** | **Scenario 2** | Only if abuse/WAF needed before Workers *and* CF budget exists; poor fit on $0 + dual console | +| **Hardened steady state** | **Scenario 3** | WAF + rate limits + **one** ops surface; avoids chronic dual-dashboard | +| **Parallel** | JWKS/port work | AI-heavy; does not block launch | +| **Tenants** | GCP CLI path now; CF when Scenario 3 matures | Tenants choose; public URL stays opaque | + +## Open follow-ups + +1. GitOps layout for mint + WIF/SM without PEMs in git ([#1263](https://github.com/fullsend-ai/fullsend/issues/1263)). +2. Allowlist-only edge until Scenario 3 (Armor+LB excluded on ~$0 budget unless funding appears). +3. Rate-limit thresholds for public hostname (when edge exists). +4. Criteria to transfer SLO ownership from Bootstrap to community ops. + +## References + +- [Infrastructure reference — Token mint](../guides/admin/infrastructure-reference.md) +- [ADR 0029](../ADRs/0029-central-token-mint-secretless-fullsend.md) +- [ADR 0040 — Public community mint architecture](../ADRs/0040-public-community-mint-architecture.md) +- [#1263](https://github.com/fullsend-ai/fullsend/issues/1263) · [#1262](https://github.com/fullsend-ai/fullsend/issues/1262) +- [#915](https://github.com/fullsend-ai/fullsend/issues/915) discussion (2026-05-25) From 5e9cca5a9c52b62f68badc53cf8075fd483c0ed9 Mon Sep 17 00:00:00 2001 From: Barak Korren Date: Mon, 25 May 2026 15:25:19 +0300 Subject: [PATCH 2/8] docs(adr-0040): refine public mint architecture decision Focus Decision on steady-state Cloudflare Workers, defer automated PEM rotation to a future ADR, and align options with ADR 0029 goal. Co-authored-by: Cursor Signed-off-by: Barak Korren --- ...0040-public-community-mint-architecture.md | 87 +++++++++++-------- 1 file changed, 53 insertions(+), 34 deletions(-) diff --git a/docs/ADRs/0040-public-community-mint-architecture.md b/docs/ADRs/0040-public-community-mint-architecture.md index 85c0328ac0..f5fcf3c809 100644 --- a/docs/ADRs/0040-public-community-mint-architecture.md +++ b/docs/ADRs/0040-public-community-mint-architecture.md @@ -11,6 +11,7 @@ topics: - github-apps - deployment - operations + - key-rotation --- # 40. Public community mint architecture @@ -28,62 +29,80 @@ Proposed ## Context -[ADR 0029](0029-central-token-mint-secretless-fullsend.md) defines a **community deployment profile**: public (unlisted) shared GitHub Apps per role, keys held only at a **central mint**, and adopters that trust a stable `FULLSEND_MINT_URL` instead of bespoke per-org Apps. [#914](https://github.com/fullsend-ai/fullsend/issues/914) covers shared App registration and install UX; this ADR records **how the public mint is built, secured, operated, and scaled**. +[ADR 0029](0029-central-token-mint-secretless-fullsend.md) already establishes the **goal**: a **community deployment profile** with public (unlisted) shared GitHub Apps per role, App keys held **only** at a centrally operated mint, and routine adopters trusting a stable **`FULLSEND_MINT_URL`** instead of bespoke per-org Apps and dispatch PATs. Self-managed mints remain a separate, coexisting path; they are not a substitute for delivering the **public** mint. -Hosting and platform trade-offs (GCP vs Cloudflare, cost, dual-console ops, JWKS port) are documented in the spike [Community token mint hosting (GCP vs Cloudflare)](../spikes/2026-05-25-community-mint-hosting-gcp-vs-cloudflare.md) ([#915](https://github.com/fullsend-ai/fullsend/issues/915)). This ADR states the **architecture**; the spike remains the reference for **platform choice rationale and phasing**. +This ADR does **not** revisit whether to build that mint. It records **how** the community-operated public mint is **deployed, secured, monitored, scaled, and run** once shared Apps and enrollment exist ([#914](https://github.com/fullsend-ai/fullsend/issues/914)). The open work is **implementation and operations**, not product direction. + +Platform and phasing choices (interim GCP vs steady-state Workers, cost, operations consoles) are analyzed in the spike [Community token mint hosting (GCP vs Cloudflare)](../spikes/2026-05-25-community-mint-hosting-gcp-vs-cloudflare.md) ([#915](https://github.com/fullsend-ai/fullsend/issues/915)). This ADR defines the **steady-state** architecture; the spike holds **interim paths, comparisons, and rejected postures**. ## Options -- **Defer a dedicated public profile** and reuse an internal Red Hat mint project — rejected: violates isolation, blast-radius, and community trust boundaries. -- **Imperative CLI deploy** (`fullsend admin install` / GCF provisioner) for the public mint — rejected for production: no enforced review or audit trail ([#1263](https://github.com/fullsend-ai/fullsend/issues/1263)). -- **Long-term GCP + Cloudflare split edge** (GCF origin + proxied hostname) as steady state — rejected as default ongoing ops model; see spike (dual dashboards, $0 budget). Acceptable only as a **short bridge** if funded. +Ways to **achieve** the ADR 0029 community mint (same `POST /v1/token` contract, org + per-repo install modes, opaque URL to consumers): + +| Option | Summary | +|--------|---------| +| **A. Dedicated community mint on GCP (interim)** | Go Cloud Function in a **mint-only** GCP project; OIDC via **STS + WIF**; PEMs in Secret Manager; prod deploy via **GitOps** ([#1263](https://github.com/fullsend-ai/fullsend/issues/1263)). | +| **B. Reuse the internal Red Hat mint** for community adopters | Single mint endpoint and project for internal and public tenants. | +| **C. Tenant-style CLI provisioner** for the public mint | Same imperative `fullsend admin install` / GCF path self-managed orgs use. | +| **D. GCP origin + Cloudflare edge (steady state)** | Keep GCF; public hostname proxied for WAF/rate limits long term. | +| **E. Cloudflare Workers (steady state)** | Port mint to Workers; OIDC via **JWKS**; edge and compute in one ops surface. | +| **F. GCP + Cloud Armor + external HTTPS LB** | Harden edge entirely in GCP without Cloudflare. | + +**B** is rejected: shared infrastructure with internal workloads breaks **isolation** and community **trust boundaries**. **C** is rejected for production: no enforced review or deploy audit ([#1263](https://github.com/fullsend-ai/fullsend/issues/1263)). **D** is rejected as the **long-term** default (dual dashboards, poor fit for ~$0 budget on meaningful WAF—see spike); acceptable only as a **short bridge**. **F** is rejected on ~$0 community budget (LB baseline cost); spike documents trade-offs. + +**Chosen:** **E** defines the architecture below. **A** is an acceptable **interim** implementation until **E** is ready; **D** only as a short bridge if edge is urgent before **E** (spike). ## Decision -The **public community mint** is a **stateless, internet-facing** HTTPS service that implements the existing `POST /v1/token` contract ([mint-token action](../../.github/actions/mint-token/action.yml), [infrastructure reference](../guides/admin/infrastructure-reference.md)). Adopters configure only **`FULLSEND_MINT_URL`** and OIDC audience; **hosting location is opaque** to consumers. +Fullsend **will operate** a **public community mint** as required by [ADR 0029](0029-central-token-mint-secretless-fullsend.md). The **steady-state** design (option **E**) is a **stateless, internet-facing** mint on **Cloudflare Workers**, exposing the existing `POST /v1/token` contract ([mint-token action](../../.github/actions/mint-token/action.yml), [infrastructure reference](../guides/admin/infrastructure-reference.md)). Adopters set **`FULLSEND_MINT_URL`** and OIDC audience only; hosting is opaque to them. + +Until that Worker implementation is production-ready, the same contract and trust bar may run temporarily on **GCP Cloud Function** (option **A**, STS + WIF)—see the [hosting spike](../spikes/2026-05-25-community-mint-hosting-gcp-vs-cloudflare.md) for interim phasing, cost, and rejected postures. **Self-managed tenant** mints stay on separate paths (CLI/GCF today); they are not described here. ### Deployment -1. **Launch posture:** GCP Cloud Function (`fullsend-mint`), Go implementation in `internal/mint`, deployed only via **GitOps** ([#1263](https://github.com/fullsend-ai/fullsend/issues/1263))—not the tenant CLI provisioner. -2. **Dedicated GCP project** for the community profile: mint function, WIF pool/providers, and Secret Manager for shared App PEMs. **No** Vertex inference, LLM credentials, or internal Red Hat mint resources in that project. -3. **Install modes:** **Org-level** (default WIF provider, `.fullsend` / upstream `fullsend-ai/fullsend` workflow refs) and **per-repo** (`PER_REPO_WIF_REPOS`, repo-scoped providers)—both on the same public mint. -4. **Steady-state hardening (target):** port mint to **Cloudflare Workers** with **JWKS**-based GitHub OIDC validation and the same claim policy, via pluggable `TokenValidator` / `PEMAccessor`—one operational surface for edge + compute. Details and phasing: [hosting spike](../spikes/2026-05-25-community-mint-hosting-gcp-vs-cloudflare.md). -5. **Tenant-operated mints** remain separate: self-managed orgs may use the CLI/GCF provisioner on GCP today; Cloudflare may become an optional **tenant** target when the Worker implementation exists. Tenant hosting choices do not affect public consumers. +1. **Runtime:** Mint logic on **Cloudflare Workers**, implemented via the shared `internal/mint` handler with a **JWKS** `TokenValidator` and pluggable `PEMAccessor` (port from today’s GCF code). +2. **Release:** Production deploys only through **GitOps** ([#1263](https://github.com/fullsend-ai/fullsend/issues/1263)) (e.g. Wrangler + reviewed merges)—not the tenant CLI provisioner. +3. **Isolation:** Community mint **must not share** infrastructure with Vertex/inference, internal Red Hat mints, or unrelated Workers (e.g. docs/admin). PEMs and mint configuration live in a **dedicated** trust domain (Worker secrets and/or a vault scoped to this profile—layout per [#914](https://github.com/fullsend-ai/fullsend/issues/914)). +4. **Public URL:** Stable **`FULLSEND_MINT_URL`** on a community hostname; TLS and edge policy colocated with the Worker. +5. **Install modes:** **Org-level** (`.fullsend` / upstream `fullsend-ai/fullsend` workflow refs) and **per-repo** (equivalent of today’s per-repo trust routing)—both supported on the **same** public mint. ### Security -1. **Trust model ([ADR 0029](0029-central-token-mint-secretless-fullsend.md)):** Callers present GitHub Actions OIDC JWTs; the mint returns **short-lived, org-scoped** installation tokens with **role minimum permissions**—never the App’s full grant. -2. **Launch validation path:** GitHub JWT → **GCP STS** exchange against **Workload Identity Federation** (CEL-bound org/repo/workflow) plus in-function **fail-closed** checks (`iss`, `aud`, `job_workflow_ref`, `ALLOWED_ORGS`, `ALLOWED_WORKFLOW_FILES`, per-repo provider routing). -3. **No auth proxy** in front of the mint; abuse defense at launch is **allowlists** and monitoring. **WAF / rate limits** arrive with the Worker steady state (or a **time-boxed** CF bridge only if budget and ops capacity exist—see spike). -4. **Secrets:** Shared community App PEMs **only** in Secret Manager for the community project (layout per [#914](https://github.com/fullsend-ai/fullsend/issues/914)); never in adopters’ `.fullsend` repos. -5. **Blast radius:** Compromise of the public mint affects **all orgs** on that profile; mitigations include GitOps-only deploys, monitoring, key rotation, narrow App installations, and human branch protections—not repo-stored PEMs. +1. **Trust model ([ADR 0029](0029-central-token-mint-secretless-fullsend.md)):** GitHub Actions OIDC JWT in, **short-lived, org-scoped** installation token out; **role minimum permissions** enforced in mint logic. +2. **OIDC validation:** Verify JWT signature against GitHub’s **JWKS** (`token.actions.githubusercontent.com`), then **fail-closed** application checks: `iss`, `aud`, `job_workflow_ref`, `ALLOWED_ORGS`, `ALLOWED_WORKFLOW_FILES`, and per-repo policy routing (parity with today’s GCF behavior—prove in CI). +3. **Edge:** **Managed WAF** and **rate limits** on `POST /v1/token` in the same Cloudflare surface as the Worker; no separate origin dashboard for abuse control. +4. **No auth proxy** in front of callers; the mint remains directly reachable with Bearer OIDC only. +5. **Secrets:** Shared community App PEMs **only** at the mint operator boundary—never in adopters’ `.fullsend` repos. +6. **PEM rotation:** **Automated** rotation of shared App private keys (generate, install on GitHub Apps, update mint material, retire old keys without service disruption) is **necessary** for long-running community operations but **deferred** to a **future ADR** that defines ceremony, tooling, and rollout. Until then, rotation is **manual** or GitOps-assisted under operator runbooks. +7. **Blast radius:** One compromised public mint affects **all orgs** on the profile; mitigate with GitOps-only changes, monitoring, timely PEM rotation, narrow App installations, and forge branch protections. ### Monitoring -1. **SLO owner:** **Red Hat Fullsend Bootstrap** until community operations assumes on-call. -2. **Targets:** **99.5%** monthly availability for `POST /v1/token` (excluding GitHub OIDC/API outages); **p95 < 2s** mint latency. -3. **Signals (GCP launch):** GCF 5xx and latency, STS/Secret Manager errors, synthetic unauthenticated `POST /v1/token` (expect 401), GitOps audit for allowlist/env changes. Optional deploy notifications ([#1262](https://github.com/fullsend-ai/fullsend/issues/1262)). -4. **Signals (Worker steady state):** Worker errors, WAF blocks, same synthetic on the public URL—single-console triage per spike. -5. **Runbook:** mint health → edge (when present) → GitHub OIDC status; prefer **managed** WAF rules over custom policy authoring. +1. **Owner:** **Red Hat Fullsend Bootstrap** until community operations assumes on-call. +2. **SLOs:** **99.5%** monthly availability for `POST /v1/token` (excluding GitHub OIDC/API outages); **p95 < 2s** latency. +3. **Signals:** Worker errors and latency, WAF block/challenge rates, synthetic `POST /v1/token` without token (expect 401), GitOps/deploy audit trail; optional deploy notifications ([#1262](https://github.com/fullsend-ai/fullsend/issues/1262)). +4. **Triage:** Single console (Cloudflare)—Worker health, then WAF, then external GitHub status; prefer **managed** rulesets over bespoke policy. ### Scaling -1. **Workload shape:** Stateless request/response; roughly **one mint call per agent job** that needs forge tokens—low QPS at community scale, bursty with Actions concurrency. -2. **Horizontal scaling:** Cloud Functions and Workers scale instances automatically; no mint-side session store. -3. **Limits:** Enforce existing caps (e.g. `repos` list size, request body size); tune **rate limits** at the edge when Scenario 3 (or a funded bridge) is live. -4. **Cost:** Community budget is **~$0**; launch stays on GCP **free tiers** without Cloud Armor + external HTTPS LB; steady-state Workers **Free** tier at expected volume unless traffic or bundle limits require Paid—see spike. +1. **Shape:** Stateless request/response; ~one mint per agent job needing forge access—low baseline QPS, bursty with Actions. +2. **Capacity:** Workers scale automatically; no mint-side session store. +3. **Limits:** Keep request/body/`repos` caps; tune **edge rate limits** as adoption grows. +4. **Cost:** Community budget **~$0** at expected volume—Workers **Free** tier where sufficient; Paid tier only if traffic or bundle limits require it ([spike](../spikes/2026-05-25-community-mint-hosting-gcp-vs-cloudflare.md)). ### Operations -1. **Change control:** All production changes through **GitOps** ([#1263](https://github.com/fullsend-ai/fullsend/issues/1263)); no individual `gcloud functions deploy` to prod. -2. **Incident response:** Revoke or rotate shared App keys, tighten `ALLOWED_ORGS` / workflow allowlists, disable profile in enrollment docs; Bootstrap team pages until community handoff. -3. **Portability:** Maintain **multiple mint hosts** long-term; second `TokenValidator` (JWKS) and parity tests (STS ≡ JWKS) are strategic work, not launch blockers for [#914](https://github.com/fullsend-ai/fullsend/issues/914). -4. **Documentation:** Platform comparison, OIDC STS vs JWKS, cost, and dashboard trade-offs live in the [hosting spike](../spikes/2026-05-25-community-mint-hosting-gcp-vs-cloudflare.md)—update the spike when platform facts change; update this ADR only when the **architectural commitments** change. +1. **Change control:** GitOps-only production changes ([#1263](https://github.com/fullsend-ai/fullsend/issues/1263)). +2. **Incidents:** Rotate shared App keys (manual procedure until automated rotation ADR), tighten allowlists, update enrollment guidance; Bootstrap pages until community handoff. +3. **Key lifecycle:** Follow-on ADR required for **automated PEM rotation** (see Security)—not a launch blocker for [#914](https://github.com/fullsend-ai/fullsend/issues/914). +4. **Evolution:** Hosting comparisons and interim GCP details stay in the [hosting spike](../spikes/2026-05-25-community-mint-hosting-gcp-vs-cloudflare.md); change this ADR when **steady-state** commitments change. ## Consequences -- Community adopters get a **stable, reviewed, isolated** mint path aligned with ADR 0029 without depending on Red Hat-internal infrastructure. -- Bootstrap team carries **SLO and incident** responsibility with **single-console** ops at launch; Worker migration reduces edge cost and dual-dashboard risk when executed. -- **~$0 budget** forbids LB+Armor as the primary hardening path; security at launch leans on **OIDC binding and allowlists** until Workers or funded edge exists. -- GitOps and a dedicated project add **operational overhead** up front but improve auditability versus CLI deploys. -- Normative follow-ons remain: shared App definitions ([#914](https://github.com/fullsend-ai/fullsend/issues/914)), GitOps repo layout ([#1263](https://github.com/fullsend-ai/fullsend/issues/1263)), JWKS parity CI, and criteria for transferring SLO ownership to community ops. +- Delivers the **ADR 0029 community profile** in operable form: shared mint endpoint, isolated from internal Red Hat mint and inference infrastructure. +- **Launch (option A)** unblocks [#914](https://github.com/fullsend-ai/fullsend/issues/914) enrollment against a real `FULLSEND_MINT_URL` without waiting for the Worker port. +- **Steady state (option E)** improves edge posture and **single-console** ops versus a permanent GCP+CF split (option D). +- Bootstrap team owns **SLOs and incidents** until community ops exists; GitOps adds process overhead but matches the security bar for a shared token issuer. +- **~$0 budget** keeps launch on GCP free tiers; LB+Armor (option F) and paid CF edge (option D long term) stay off the critical path unless funding appears. +- Remaining work is **execution**: shared Apps ([#914](https://github.com/fullsend-ai/fullsend/issues/914)), GitOps layout ([#1263](https://github.com/fullsend-ai/fullsend/issues/1263)), JWKS parity CI, SLO handoff criteria—not a replan of whether the public mint should exist. +- **Automated PEM rotation** must be specified in a **future ADR**; operating without it increases reliance on manual ceremony and incident-time rotation discipline. From 386d3cdbac75abc9942e8fd50a9cb0d17eb4743f Mon Sep 17 00:00:00 2001 From: Barak Korren Date: Tue, 30 Jun 2026 13:30:46 +0300 Subject: [PATCH 3/8] docs(adr-0061): rebase public mint ADR on main and address review Renumber to ADR 0061 (0040 taken by org-pool ADR), merge upstream/main, cross-reference ADR 0059/0060/0044, and fold in trust-policy decisions that supersede the original OIDC/enrollment wording. Update spike, README, and architecture.md; annotate ADR 0059 deferred items now covered here. Signed-off-by: Barak Korren Co-authored-by: Cursor --- README.md | 1 + ...0040-public-community-mint-architecture.md | 108 --------------- ...blic-mint-mode-with-wildcard-allowlists.md | 10 +- ...0061-public-community-mint-architecture.md | 126 ++++++++++++++++++ docs/architecture.md | 1 + ...ommunity-mint-hosting-gcp-vs-cloudflare.md | 42 +++--- 6 files changed, 157 insertions(+), 131 deletions(-) delete mode 100644 docs/ADRs/0040-public-community-mint-architecture.md create mode 100644 docs/ADRs/0061-public-community-mint-architecture.md diff --git a/README.md b/README.md index 8abb67a70d..54371b97d2 100644 --- a/README.md +++ b/README.md @@ -55,6 +55,7 @@ This is not a product spec. It's an evolving exploration of a hard problem space - [Repos Management](docs/plans/repos-management.md) — Implementation plan for declarative multi-repo management - [Repos Init](docs/plans/repos-init.md) — Implementation plan for `fullsend repos init` manifest bootstrapping - **[docs/guides/](docs/guides/)** — Practical how-to documentation for administrators and developers (see [ADR 0023](docs/ADRs/0023-user-documentation-structure.md)) +- **[docs/spikes/](docs/spikes/)** — Time-boxed investigations and trade-off analyses that inform ADRs (e.g. community mint hosting) - **[docs/ADRs/](docs/ADRs/)** — Architecture Decision Records for crystallizing specific decisions (see [ADR 0001](docs/ADRs/0001-use-adrs-for-decision-making.md)) - **[web/](web/)** — Browser-delivered assets for the public site (document graph today; future Vite app here). Cloudflare Worker config lives in [`cloudflare_site/`](cloudflare_site/) ([ADR 0019](docs/ADRs/0019-web-source-and-cloudflare-site-layout.md)). - **[docs/landscape.md](docs/landscape.md)** — Survey of AI code review tools, orchestration patterns, and connectivity gateways; how they relate to our goals (time-sensitive — check the date) diff --git a/docs/ADRs/0040-public-community-mint-architecture.md b/docs/ADRs/0040-public-community-mint-architecture.md deleted file mode 100644 index f5fcf3c809..0000000000 --- a/docs/ADRs/0040-public-community-mint-architecture.md +++ /dev/null @@ -1,108 +0,0 @@ ---- -title: "40. Public community mint architecture" -status: Proposed -relates_to: - - agent-infrastructure - - security-threat-model - - operational-observability -topics: - - identity - - oidc - - github-apps - - deployment - - operations - - key-rotation ---- - -# 40. Public community mint architecture - -Date: 2026-05-25 - -## Status - -Proposed - - - -## Context - -[ADR 0029](0029-central-token-mint-secretless-fullsend.md) already establishes the **goal**: a **community deployment profile** with public (unlisted) shared GitHub Apps per role, App keys held **only** at a centrally operated mint, and routine adopters trusting a stable **`FULLSEND_MINT_URL`** instead of bespoke per-org Apps and dispatch PATs. Self-managed mints remain a separate, coexisting path; they are not a substitute for delivering the **public** mint. - -This ADR does **not** revisit whether to build that mint. It records **how** the community-operated public mint is **deployed, secured, monitored, scaled, and run** once shared Apps and enrollment exist ([#914](https://github.com/fullsend-ai/fullsend/issues/914)). The open work is **implementation and operations**, not product direction. - -Platform and phasing choices (interim GCP vs steady-state Workers, cost, operations consoles) are analyzed in the spike [Community token mint hosting (GCP vs Cloudflare)](../spikes/2026-05-25-community-mint-hosting-gcp-vs-cloudflare.md) ([#915](https://github.com/fullsend-ai/fullsend/issues/915)). This ADR defines the **steady-state** architecture; the spike holds **interim paths, comparisons, and rejected postures**. - -## Options - -Ways to **achieve** the ADR 0029 community mint (same `POST /v1/token` contract, org + per-repo install modes, opaque URL to consumers): - -| Option | Summary | -|--------|---------| -| **A. Dedicated community mint on GCP (interim)** | Go Cloud Function in a **mint-only** GCP project; OIDC via **STS + WIF**; PEMs in Secret Manager; prod deploy via **GitOps** ([#1263](https://github.com/fullsend-ai/fullsend/issues/1263)). | -| **B. Reuse the internal Red Hat mint** for community adopters | Single mint endpoint and project for internal and public tenants. | -| **C. Tenant-style CLI provisioner** for the public mint | Same imperative `fullsend admin install` / GCF path self-managed orgs use. | -| **D. GCP origin + Cloudflare edge (steady state)** | Keep GCF; public hostname proxied for WAF/rate limits long term. | -| **E. Cloudflare Workers (steady state)** | Port mint to Workers; OIDC via **JWKS**; edge and compute in one ops surface. | -| **F. GCP + Cloud Armor + external HTTPS LB** | Harden edge entirely in GCP without Cloudflare. | - -**B** is rejected: shared infrastructure with internal workloads breaks **isolation** and community **trust boundaries**. **C** is rejected for production: no enforced review or deploy audit ([#1263](https://github.com/fullsend-ai/fullsend/issues/1263)). **D** is rejected as the **long-term** default (dual dashboards, poor fit for ~$0 budget on meaningful WAF—see spike); acceptable only as a **short bridge**. **F** is rejected on ~$0 community budget (LB baseline cost); spike documents trade-offs. - -**Chosen:** **E** defines the architecture below. **A** is an acceptable **interim** implementation until **E** is ready; **D** only as a short bridge if edge is urgent before **E** (spike). - -## Decision - -Fullsend **will operate** a **public community mint** as required by [ADR 0029](0029-central-token-mint-secretless-fullsend.md). The **steady-state** design (option **E**) is a **stateless, internet-facing** mint on **Cloudflare Workers**, exposing the existing `POST /v1/token` contract ([mint-token action](../../.github/actions/mint-token/action.yml), [infrastructure reference](../guides/admin/infrastructure-reference.md)). Adopters set **`FULLSEND_MINT_URL`** and OIDC audience only; hosting is opaque to them. - -Until that Worker implementation is production-ready, the same contract and trust bar may run temporarily on **GCP Cloud Function** (option **A**, STS + WIF)—see the [hosting spike](../spikes/2026-05-25-community-mint-hosting-gcp-vs-cloudflare.md) for interim phasing, cost, and rejected postures. **Self-managed tenant** mints stay on separate paths (CLI/GCF today); they are not described here. - -### Deployment - -1. **Runtime:** Mint logic on **Cloudflare Workers**, implemented via the shared `internal/mint` handler with a **JWKS** `TokenValidator` and pluggable `PEMAccessor` (port from today’s GCF code). -2. **Release:** Production deploys only through **GitOps** ([#1263](https://github.com/fullsend-ai/fullsend/issues/1263)) (e.g. Wrangler + reviewed merges)—not the tenant CLI provisioner. -3. **Isolation:** Community mint **must not share** infrastructure with Vertex/inference, internal Red Hat mints, or unrelated Workers (e.g. docs/admin). PEMs and mint configuration live in a **dedicated** trust domain (Worker secrets and/or a vault scoped to this profile—layout per [#914](https://github.com/fullsend-ai/fullsend/issues/914)). -4. **Public URL:** Stable **`FULLSEND_MINT_URL`** on a community hostname; TLS and edge policy colocated with the Worker. -5. **Install modes:** **Org-level** (`.fullsend` / upstream `fullsend-ai/fullsend` workflow refs) and **per-repo** (equivalent of today’s per-repo trust routing)—both supported on the **same** public mint. - -### Security - -1. **Trust model ([ADR 0029](0029-central-token-mint-secretless-fullsend.md)):** GitHub Actions OIDC JWT in, **short-lived, org-scoped** installation token out; **role minimum permissions** enforced in mint logic. -2. **OIDC validation:** Verify JWT signature against GitHub’s **JWKS** (`token.actions.githubusercontent.com`), then **fail-closed** application checks: `iss`, `aud`, `job_workflow_ref`, `ALLOWED_ORGS`, `ALLOWED_WORKFLOW_FILES`, and per-repo policy routing (parity with today’s GCF behavior—prove in CI). -3. **Edge:** **Managed WAF** and **rate limits** on `POST /v1/token` in the same Cloudflare surface as the Worker; no separate origin dashboard for abuse control. -4. **No auth proxy** in front of callers; the mint remains directly reachable with Bearer OIDC only. -5. **Secrets:** Shared community App PEMs **only** at the mint operator boundary—never in adopters’ `.fullsend` repos. -6. **PEM rotation:** **Automated** rotation of shared App private keys (generate, install on GitHub Apps, update mint material, retire old keys without service disruption) is **necessary** for long-running community operations but **deferred** to a **future ADR** that defines ceremony, tooling, and rollout. Until then, rotation is **manual** or GitOps-assisted under operator runbooks. -7. **Blast radius:** One compromised public mint affects **all orgs** on the profile; mitigate with GitOps-only changes, monitoring, timely PEM rotation, narrow App installations, and forge branch protections. - -### Monitoring - -1. **Owner:** **Red Hat Fullsend Bootstrap** until community operations assumes on-call. -2. **SLOs:** **99.5%** monthly availability for `POST /v1/token` (excluding GitHub OIDC/API outages); **p95 < 2s** latency. -3. **Signals:** Worker errors and latency, WAF block/challenge rates, synthetic `POST /v1/token` without token (expect 401), GitOps/deploy audit trail; optional deploy notifications ([#1262](https://github.com/fullsend-ai/fullsend/issues/1262)). -4. **Triage:** Single console (Cloudflare)—Worker health, then WAF, then external GitHub status; prefer **managed** rulesets over bespoke policy. - -### Scaling - -1. **Shape:** Stateless request/response; ~one mint per agent job needing forge access—low baseline QPS, bursty with Actions. -2. **Capacity:** Workers scale automatically; no mint-side session store. -3. **Limits:** Keep request/body/`repos` caps; tune **edge rate limits** as adoption grows. -4. **Cost:** Community budget **~$0** at expected volume—Workers **Free** tier where sufficient; Paid tier only if traffic or bundle limits require it ([spike](../spikes/2026-05-25-community-mint-hosting-gcp-vs-cloudflare.md)). - -### Operations - -1. **Change control:** GitOps-only production changes ([#1263](https://github.com/fullsend-ai/fullsend/issues/1263)). -2. **Incidents:** Rotate shared App keys (manual procedure until automated rotation ADR), tighten allowlists, update enrollment guidance; Bootstrap pages until community handoff. -3. **Key lifecycle:** Follow-on ADR required for **automated PEM rotation** (see Security)—not a launch blocker for [#914](https://github.com/fullsend-ai/fullsend/issues/914). -4. **Evolution:** Hosting comparisons and interim GCP details stay in the [hosting spike](../spikes/2026-05-25-community-mint-hosting-gcp-vs-cloudflare.md); change this ADR when **steady-state** commitments change. - -## Consequences - -- Delivers the **ADR 0029 community profile** in operable form: shared mint endpoint, isolated from internal Red Hat mint and inference infrastructure. -- **Launch (option A)** unblocks [#914](https://github.com/fullsend-ai/fullsend/issues/914) enrollment against a real `FULLSEND_MINT_URL` without waiting for the Worker port. -- **Steady state (option E)** improves edge posture and **single-console** ops versus a permanent GCP+CF split (option D). -- Bootstrap team owns **SLOs and incidents** until community ops exists; GitOps adds process overhead but matches the security bar for a shared token issuer. -- **~$0 budget** keeps launch on GCP free tiers; LB+Armor (option F) and paid CF edge (option D long term) stay off the critical path unless funding appears. -- Remaining work is **execution**: shared Apps ([#914](https://github.com/fullsend-ai/fullsend/issues/914)), GitOps layout ([#1263](https://github.com/fullsend-ai/fullsend/issues/1263)), JWKS parity CI, SLO handoff criteria—not a replan of whether the public mint should exist. -- **Automated PEM rotation** must be specified in a **future ADR**; operating without it increases reliance on manual ceremony and incident-time rotation discipline. diff --git a/docs/ADRs/0059-public-mint-mode-with-wildcard-allowlists.md b/docs/ADRs/0059-public-mint-mode-with-wildcard-allowlists.md index ea3ccf33b2..72c0bfbf22 100644 --- a/docs/ADRs/0059-public-mint-mode-with-wildcard-allowlists.md +++ b/docs/ADRs/0059-public-mint-mode-with-wildcard-allowlists.md @@ -92,15 +92,15 @@ Introduce a **public mint mode** when `ALLOWED_ORGS` contains `*`, with `job_wor 11. **Deferred to future ADRs (mint-only):** - Dedicated reusable workflow(s) for custom agent **stages** beyond the built-in set. - **Prioritize** on `workflow_dispatch`: whether manual prioritize mints via an upstream `reusable-*.yml` or remains tight-mint-only until wired ([ADR 0041](0041-synchronous-workflow-call-event-dispatch.md) keeps non-event `workflow_dispatch` entry points). - - Mint infrastructure: WIF pool/provider provisioning, Cloud Function deployment, CEL definitions, abuse controls, WAF, and monitoring. - - Hosted mint enrollment policy: whether the fullsend-operated hosted mint adopts public mode (`ALLOWED_ORGS=*`) or remains tight with explicit enrollment. + - **Mint infrastructure** (WIF pool/provider provisioning, Cloud Function deployment, CEL definitions, abuse controls, WAF, monitoring): see [ADR 0061](0061-public-community-mint-architecture.md). + - **Hosted mint enrollment:** resolved — public mode (`ALLOWED_ORGS=*`) + shared App install per §1 and [ADR 0061](0061-public-community-mint-architecture.md). 12. **Normative specs (`docs/normative/`).** Not required for this decision. The reference mint is implemented in `internal/mintcore/` and configured via documented env vars; this ADR plus [ADR 0029](0029-central-token-mint-secretless-fullsend.md) are the contract for that implementation. A versioned normative spec is **out of scope** until there are multiple independent mint implementations that must interoperate on the same byte-level env and claim rules. ## Consequences - Public mode: `ALLOWED_ORGS=*`, permissive provider id in `WIF_PROVIDER_NAME`, `PER_REPO_WIF_REPOS` empty. Tight mode: explicit `ALLOWED_ORGS`, org-merged provider in `WIF_PROVIDER_NAME`, optional `PER_REPO_WIF_REPOS` list. -- Provider provisioning and IAM are deferred to a mint infrastructure ADR; this ADR does not specify how providers are created. +- Provider provisioning and IAM for the **hosted** public mint are specified in [ADR 0061](0061-public-community-mint-architecture.md); tight/self-managed deployments remain operator-defined. - Custom agents remain configuration-only unless a new shared App, mint role, and upstream workflow file are added. - Per-repo **public** mint assumes event-driven runs mint from `fullsend-ai/fullsend` reusables per [ADR 0041](0041-synchronous-workflow-call-event-dispatch.md) and [ADR 0033](0033-per-repo-installation-mode.md). Legacy `{org}/.fullsend/` provenance is not supported in public mode and is deprecated by [ADR 0044](0044-deprecate-per-org-installation-mode.md). - Ref-any on `fullsend-ai/fullsend` at the mint layer trades pinning rigor for simpler rollout; caller-level SHA pinning per [ADR 0031](0031-reusable-workflows-for-action-installed-distribution.md) remains the recommended org/repo control. @@ -126,7 +126,7 @@ Introduce a **public mint mode** when `ALLOWED_ORGS` contains `*`, with `job_wor |-------|-------------| | Custom-agent reusable workflow | Future ADR | | Prioritize mint path under public mode | Resolve when non-event workflows are wired to reusables ([ADR 0041](0041-synchronous-workflow-call-event-dispatch.md)) | -| Mint WIF providers, IAM, deployment, abuse/WAF | Future ADR (all provisioning) | -| Hosted mint public-mode enrollment | Resolve when moving hosted mint to `ALLOWED_ORGS=*` | +| Mint WIF providers, IAM, deployment, abuse/WAF | [ADR 0061](0061-public-community-mint-architecture.md) | +| Hosted mint public-mode enrollment | [ADR 0061](0061-public-community-mint-architecture.md) + §1 here | | Normative mint contract | Deferred until multiple implementations; not needed now | | Stricter `@ref` allowlist on upstream workflows | Optional later hardening | diff --git a/docs/ADRs/0061-public-community-mint-architecture.md b/docs/ADRs/0061-public-community-mint-architecture.md new file mode 100644 index 0000000000..a552519d8c --- /dev/null +++ b/docs/ADRs/0061-public-community-mint-architecture.md @@ -0,0 +1,126 @@ +--- +title: "61. Public community mint architecture" +status: Accepted +relates_to: + - agent-infrastructure + - security-threat-model + - operational-observability +topics: + - identity + - oidc + - github-apps + - deployment + - operations + - key-rotation +--- + +# 61. Public community mint architecture + +Date: 2026-05-25 + +## Status + +Accepted + + + +## Context + +[ADR 0029](0029-central-token-mint-secretless-fullsend.md) establishes the **goal**: a **community deployment profile** with public (unlisted) shared GitHub Apps per role, App keys held **only** at a centrally operated mint, and routine adopters trusting a stable **`FULLSEND_MINT_URL`** instead of bespoke per-org Apps and dispatch PATs. + +Since this ADR was drafted, related decisions landed on `main`: + +- **[ADR 0059](0059-public-mint-mode-with-wildcard-allowlists.md)** (Accepted) defines **public mint trust policy**: `ALLOWED_ORGS=*`, upstream-only `job_workflow_ref` under `fullsend-ai/fullsend/.github/workflows/`, global per-role `ROLE_APP_IDS` and PEM secrets, and enrollment via **installing the shared Apps** (no per-org mint env churn). Custom per-repo workflow provenance is **tight-mode only** via `PER_REPO_WIF_REPOS`. +- **[ADR 0060](0060-cross-org-mint-authorization-via-org-variables.md)** (Accepted) adds optional `target_org` minting for workloads like the e2e pool ([ADR 0040](0040-org-pool-for-parallel-e2e-tests.md)). +- **[ADR 0044](0044-deprecate-per-org-installation-mode.md)** (Accepted) deprecates per-org `.fullsend` installs; the public profile targets **per-repo** installs calling upstream reusables ([ADR 0033](0033-per-repo-installation-mode.md), [ADR 0031](0031-reusable-workflows-for-action-installed-distribution.md)). +- Mint logic now lives in **`internal/mintcore/`** (shared by GCF `internal/mint/` and standalone **`cmd/mint/`**, which already uses **JWKS** verification). + +This ADR records **how** the community-operated public mint is **deployed, secured at the edge, monitored, scaled, and run**. **OIDC claim rules and enrollment policy** are normative in [ADR 0059](0059-public-mint-mode-with-wildcard-allowlists.md); this ADR fulfills the **mint infrastructure** item deferred there (WIF/provider layout, deployment, WAF, monitoring). + +Platform and phasing choices (interim GCP vs steady-state Workers, cost, operations consoles) are analyzed in the [hosting spike](../spikes/2026-05-25-community-mint-hosting-gcp-vs-cloudflare.md) ([#915](https://github.com/fullsend-ai/fullsend/issues/915)). [#1145](https://github.com/fullsend-ai/fullsend/issues/1145) depends on this architecture for zero-GCP installs against the hosted mint. + +## Options + +Ways to **achieve** the ADR 0029 community mint (same `POST /v1/token` contract, opaque URL to consumers): + +| Option | Summary | +|--------|---------| +| **A. Dedicated community mint on GCP (interim)** | Go Cloud Function (`internal/mint/` + `mintcore`) in a **mint-only** GCP project; OIDC via **STS + WIF**; PEMs in Secret Manager; prod deploy via **GitOps** ([#1263](https://github.com/fullsend-ai/fullsend/issues/1263)). | +| **B. Reuse the internal Red Hat mint** for community adopters | Single mint endpoint and project for internal and public tenants. | +| **C. Tenant-style CLI provisioner** for the public mint | Same imperative `fullsend admin install` / GCF path self-managed orgs use. | +| **D. GCP origin + Cloudflare edge (steady state)** | Keep GCF; public hostname proxied for WAF/rate limits long term. | +| **E. Cloudflare Workers (steady state)** | Port mint to Workers; OIDC via **JWKS** (`mintcore.JWKSVerifier`); edge and compute in one ops surface. | +| **F. GCP + Cloud Armor + external HTTPS LB** | Harden edge entirely in GCP without Cloudflare. | + +**B** is rejected: shared infrastructure with internal workloads breaks **isolation** and community **trust boundaries**. **C** is rejected for production: no enforced review or deploy audit ([#1263](https://github.com/fullsend-ai/fullsend/issues/1263)). **D** is rejected as the **long-term** default (dual dashboards, poor fit for ~$0 budget on meaningful WAF—see spike); acceptable only as a **short bridge**. **F** is rejected on ~$0 community budget (LB baseline cost). + +**Chosen:** **E** defines the steady-state architecture below. **A** is an acceptable **interim** implementation until **E** is ready; **D** only as a short bridge if edge is urgent before **E** (spike). + +## Decision + +Fullsend **will operate** a **public community mint** as required by [ADR 0029](0029-central-token-mint-secretless-fullsend.md). The **steady-state** design (option **E**) is a **stateless, internet-facing** mint on **Cloudflare Workers**, exposing the existing `POST /v1/token` contract ([mint-token action](../../.github/actions/mint-token/action.yml), [infrastructure reference](../guides/infrastructure/infrastructure-reference.md)). Adopters set **`FULLSEND_MINT_URL`** and OIDC audience only; hosting is opaque. + +Until the Worker implementation is production-ready, the same contract and trust bar may run temporarily on **GCP Cloud Function** (option **A**, STS + WIF). **Self-managed tenant** mints stay on separate paths (CLI/GCF or `cmd/mint/`); they are not described here. + +### Trust and enrollment (hosted profile) + +The hosted public mint **will use public mint mode** per [ADR 0059](0059-public-mint-mode-with-wildcard-allowlists.md): + +- **`ALLOWED_ORGS=*`** — any org may request tokens after other checks pass; **installing the shared role Apps is enrollment** ([#914](https://github.com/fullsend-ai/fullsend/issues/914), [#1145](https://github.com/fullsend-ai/fullsend/issues/1145)). No `EnsureOrgInMint` / per-org `ALLOWED_ORGS` updates for new adopters. +- **`job_workflow_ref`** — **upstream reusables only** (`fullsend-ai/fullsend/.github/workflows/`). Legacy `{org}/.fullsend/` and custom `{owner}/{repo}/` workflow paths are **not** supported on the public profile ([ADR 0044](0044-deprecate-per-org-installation-mode.md)). +- **`PER_REPO_WIF_REPOS`** — **unset/empty** on the hosted mint. Per-repo custom workflow provenance remains a **tight-mode** feature for self-managed mints only. +- **Shared credentials** — `ROLE_APP_IDS` and PEM secrets are **global per role** (`fullsend-{role}-app-pem`), not keyed by org ([ADR 0059](0059-public-mint-mode-with-wildcard-allowlists.md) §8). Cross-org isolation uses `repository_owner` + installation lookup, not separate PEMs per org. +- **`aud` validation** — enforced in **`mintcore`** application code (`OIDC_AUDIENCE`) on both STS and JWKS paths; it is **not** a WIF/STS responsibility and carries over unchanged on Workers. +- **Abuse complement** — [ADR 0054](0054-require-authorization-on-all-agent-dispatch-paths.md) dispatch authorization limits who can trigger agent runs; mint openness does not bypass write checks at dispatch. +- **Cross-org** — [ADR 0060](0060-cross-org-mint-authorization-via-org-variables.md) applies on the same hosted endpoint (e.g. e2e pool). + +### Deployment + +1. **Runtime:** **`mintcore`** handler on **Cloudflare Workers** with **`JWKSVerifier`** and pluggable `PEMAccessor` (parity with `cmd/mint/` today). +2. **Interim (option A):** GCF wrapper in `internal/mint/` with **`STSVerifier`**; public-mode env per [ADR 0059](0059-public-mint-mode-with-wildcard-allowlists.md) §2 (permissive `WIF_PROVIDER_NAME`, empty `PER_REPO_WIF_REPOS`). +3. **Release:** Production deploys only through **GitOps** ([#1263](https://github.com/fullsend-ai/fullsend/issues/1263))—not the tenant CLI provisioner. +4. **Isolation:** Community mint **must not share** infrastructure with Vertex/inference, internal Red Hat mints, or unrelated Workers. PEMs and mint configuration live in a **dedicated** trust domain. +5. **Public URL:** Stable **`FULLSEND_MINT_URL`** on a community hostname; TLS and edge policy colocated with the Worker. + +### Security (edge and operations) + +1. **Trust model ([ADR 0029](0029-central-token-mint-secretless-fullsend.md)):** OIDC JWT in, **short-lived, org-scoped** installation token out; **role minimum permissions** in mint logic. +2. **OIDC validation:** JWKS signature verification (steady state) or STS exchange (interim), then the **same** `mintcore` claim checks as today—including `iss`, `aud`, org allowlist, and workflow provenance per [ADR 0059](0059-public-mint-mode-with-wildcard-allowlists.md). Prove STS ≡ JWKS in CI before cutover. +3. **Edge:** **Managed WAF** and **rate limits** on `POST /v1/token` in the same Cloudflare surface as the Worker. +4. **No auth proxy** in front of callers; Bearer OIDC only. +5. **Secrets:** Shared community App PEMs **only** at the mint operator boundary. +6. **PEM rotation:** **Automated** rotation is **necessary** but **deferred** to a **future ADR**; until then, rotation is manual or GitOps-assisted. +7. **Blast radius:** One compromised public mint affects **all orgs** on the profile; mitigate with GitOps-only changes, monitoring, timely PEM rotation, narrow App installations, and forge branch protections. + +### Monitoring + +1. **Owner:** **Red Hat Fullsend Bootstrap** until community operations assumes on-call. +2. **SLOs:** **99.5%** monthly availability for `POST /v1/token` (excluding GitHub OIDC/API outages); **p95 < 2s** latency. +3. **Signals:** Worker errors and latency, WAF block/challenge rates, synthetic `POST /v1/token` without token (expect 401), GitOps/deploy audit trail ([#1262](https://github.com/fullsend-ai/fullsend/issues/1262)). +4. **Triage:** Single console (Cloudflare)—Worker health, then WAF, then external GitHub status. + +### Scaling + +1. **Shape:** Stateless request/response; low baseline QPS, bursty with Actions. +2. **Capacity:** Workers scale automatically; no mint-side session store. +3. **Limits:** Keep request/body/`repos` caps; tune **edge rate limits** as adoption grows. +4. **Cost:** Community budget **~$0** at expected volume ([spike](../spikes/2026-05-25-community-mint-hosting-gcp-vs-cloudflare.md)). + +### Operations + +1. **Change control:** GitOps-only production changes ([#1263](https://github.com/fullsend-ai/fullsend/issues/1263)). +2. **Incidents:** Rotate shared App keys (manual until automated rotation ADR), tighten allowlists if needed, update enrollment guidance. +3. **Evolution:** Hosting comparisons and interim GCP details stay in the [hosting spike](../spikes/2026-05-25-community-mint-hosting-gcp-vs-cloudflare.md). + +## Consequences + +- Delivers the **ADR 0029 community profile** in operable form, with **trust policy** in [ADR 0059](0059-public-mint-mode-with-wildcard-allowlists.md) and **ops/deployment** here. +- **Launch (option A)** unblocks [#914](https://github.com/fullsend-ai/fullsend/issues/914) and [#1145](https://github.com/fullsend-ai/fullsend/issues/1145) without waiting for the Worker port. +- **Steady state (option E)** improves edge posture and **single-console** ops versus a permanent GCP+CF split (option D). +- Bootstrap owns **SLOs and incidents** until community ops exists. +- **~$0 budget** keeps launch on GCP free tiers; LB+Armor (option F) and paid CF edge (option D long term) stay off the critical path unless funding appears. +- Remaining work: shared Apps ([#914](https://github.com/fullsend-ai/fullsend/issues/914)), GitOps layout ([#1263](https://github.com/fullsend-ai/fullsend/issues/1263)), JWKS parity CI, public-mode implementation in `mintcore`, SLO handoff criteria. +- **Automated PEM rotation** must be specified in a **future ADR**. diff --git a/docs/architecture.md b/docs/architecture.md index a81a8fdd9b..7d73e35bd5 100644 --- a/docs/architecture.md +++ b/docs/architecture.md @@ -144,6 +144,7 @@ Identity is not the same as trust. An agent's identity lets it authenticate to e - Per-role GitHub Apps with manifest-based creation. Each agent role gets its own app with scoped permissions. PEMs stored in Secret Manager as `fullsend-{role}-app-pem` — one secret per role, shared across orgs on a mint. `ROLE_APP_IDS` uses the same shared-per-role model (`coder` → app ID). Org isolation is enforced via `ALLOWED_ORGS`, WIF conditions, and installation verification ([ADR 0007](ADRs/0007-per-role-github-apps.md), [ADR 0033](ADRs/0033-per-repo-installation-mode.md)). Public multi-tenant mint (`ALLOWED_ORGS=*`) with upstream-only workflow provenance is defined in [ADR 0059](ADRs/0059-public-mint-mode-with-wildcard-allowlists.md); upstream-only provenance limits which workflows can call the mint, complementing [ADR 0029](ADRs/0029-central-token-mint-secretless-fullsend.md) multi-tenant blast-radius concerns. - Cross-org mint authorization: workflows may request tokens for a different org via optional `target_org` when the target org installs the role App and sets `FULLSEND_FOREIGN__REPOS`. Empty `repos` yields installation-wide tokens on either path; cross-org adds FOREIGN gating, same-org relies on WIF/OIDC enrollment ([ADR 0060](ADRs/0060-cross-org-mint-authorization-via-org-variables.md)). - Standalone mint deployment: `cmd/mint/` provides a self-contained HTTP server that uses direct JWKS verification and filesystem PEM storage instead of GCP infrastructure. It shares the `internal/mintcore/` library with the GCF mint and adds support for custom role permissions and a fallback proxy to an upstream mint. Custom role permissions live in mintcore (not `cmd/mint/`) so that `RolePermissionsFor`, `HasRole`, and `CreateInstallationToken` return a unified view without callers needing to distinguish built-in from custom roles. The GCF mint never calls `RegisterCustomRolePermissions`, so the code is inert there. See the [standalone mint guide](guides/infrastructure/standalone-mint.md). +- Hosted public community mint: steady-state deployment on Cloudflare Workers (JWKS + WAF + single ops console), with interim GCP Cloud Function acceptable until the Worker port is production-ready. Trust policy (`ALLOWED_ORGS=*`, upstream-only workflow provenance) is in [ADR 0059](ADRs/0059-public-mint-mode-with-wildcard-allowlists.md); deployment, edge security, monitoring, and phasing are in [ADR 0061](ADRs/0061-public-community-mint-architecture.md). Enrollment is installing the shared Apps—no per-org mint env registration ([#1145](https://github.com/fullsend-ai/fullsend/issues/1145)). One concrete implementation option is [`oidcx`](https://github.com/oxidecomputer/oidcx): a service that accepts OIDC identity tokens and exchanges them for short-lived access tokens. It can mint tokens scoped to selected GitHub repositories and permissions, or to selected Oxide silos and permissions, and it also ships with a GitHub Action wrapper. In a Fullsend deployment, this can be used by the sandbox entrypoint to narrow a broad GitHub App identity down to only the specific permissions an agent needs for the current run. diff --git a/docs/spikes/2026-05-25-community-mint-hosting-gcp-vs-cloudflare.md b/docs/spikes/2026-05-25-community-mint-hosting-gcp-vs-cloudflare.md index 31eaf2bc0a..59299389ae 100644 --- a/docs/spikes/2026-05-25-community-mint-hosting-gcp-vs-cloudflare.md +++ b/docs/spikes/2026-05-25-community-mint-hosting-gcp-vs-cloudflare.md @@ -1,17 +1,17 @@ # SPIKE: Community token mint hosting (GCP vs Cloudflare) -**Issue:** [#915](https://github.com/fullsend-ai/fullsend/issues/915) · **Parent:** [#914](https://github.com/fullsend-ai/fullsend/issues/914) · **Epic:** [#912](https://github.com/fullsend-ai/fullsend/issues/912) -**ADR:** [0029](../ADRs/0029-central-token-mint-secretless-fullsend.md) · [0040](../ADRs/0040-public-community-mint-architecture.md) (architecture) · **GitOps:** [#1263](https://github.com/fullsend-ai/fullsend/issues/1263) · **Date:** 2026-05-25 +**Issue:** [#915](https://github.com/fullsend-ai/fullsend/issues/915) · **Parent:** [#914](https://github.com/fullsend-ai/fullsend/issues/914) · **Epic:** [#912](https://github.com/fullsend-ai/fullsend/issues/912) +**ADR:** [0029](../ADRs/0029-central-token-mint-secretless-fullsend.md) · [0059](../ADRs/0059-public-mint-mode-with-wildcard-allowlists.md) (trust) · [0061](../ADRs/0061-public-community-mint-architecture.md) (ops/deployment) · **GitOps:** [#1263](https://github.com/fullsend-ai/fullsend/issues/1263) · **Date:** 2026-05-25 ## Decision -**Launch the public mint on Scenario 1** (100% GCP, dedicated project, [GitOps](#public-mint-operations) per [#1263](https://github.com/fullsend-ai/fullsend/issues/1263)), with **org and per-repo** modes unchanged. +**Launch the public mint on Scenario 1** (100% GCP, dedicated project, [GitOps](#public-mint-operations) per [#1263](https://github.com/fullsend-ai/fullsend/issues/1263)), with **per-repo** installs calling upstream reusables ([ADR 0033](../ADRs/0033-per-repo-installation-mode.md), [ADR 0059](../ADRs/0059-public-mint-mode-with-wildcard-allowlists.md)). **Do not treat Scenario 2 (GCP + Cloudflare) as the long-term default.** Two consoles (GCP origin + Cloudflare edge) are ongoing human cost for the **Red Hat Fullsend Bootstrap** team (no dedicated SRE). That cost is **not** offset by saving a one-time port—re-platforming is largely AI-driven; **monitoring and incidents are human-driven**. **Steady-state target for a hardened public edge: Scenario 3** (100% Cloudflare Workers), not Scenario 2—**one operational surface** (Workers + WAF + alerts) while preserving the same `POST /v1/token` contract. Scenario 2 remains a **short bridge** only if WAF/rate limits are needed before the Worker port lands. -Public mint consumers only use **`FULLSEND_MINT_URL`**; hosting is opaque. **Self-managed tenant** mints stay on GCP via the CLI provisioner today; Cloudflare becomes an optional tenant target when Scenario 3 exists. +Public mint consumers only use **`FULLSEND_MINT_URL`**; hosting is opaque. **Self-managed tenant** mints stay on GCP via the CLI provisioner or **`cmd/mint/`** today; Cloudflare becomes an optional tenant target when Scenario 3 exists. ## Scenarios @@ -21,7 +21,7 @@ Public mint consumers only use **`FULLSEND_MINT_URL`**; hosting is opaque. **Sel | **2** | Same GCF origin; public URL is **Cloudflare-proxied** | **Two** stacks: GCP (origin health, STS, SM) + Cloudflare (WAF, blocks, 5xx at edge) | | **3** | Mint on **Workers**; OIDC via JWKS + same claim rules; PEMs in Worker secrets (or external vault) | **One** primary stack (Cloudflare); GitOps via Wrangler | -Today’s code: native Go GCF (`internal/mint`), pluggable `TokenValidator` / `PEMAccessor`, tenant deploy via `internal/dispatch/gcf`, contract in [mint-token](../../.github/actions/mint-token/action.yml) and [infrastructure reference](../guides/admin/infrastructure-reference.md). +Today's code: **`internal/mintcore/`** (shared library), GCF entrypoint (`internal/mint/`), standalone JWKS server (`cmd/mint/`), tenant deploy via `internal/dispatch/gcf`, contract in [mint-token](../../.github/actions/mint-token/action.yml) and [infrastructure reference](../guides/infrastructure/infrastructure-reference.md). ### OIDC trust: STS/WIF (today) vs JWKS (Scenario 3) @@ -30,9 +30,13 @@ GitHub Actions sends the mint a short-lived **OIDC JWT** in the `Authorization` | Approach | Used in | What it means | |----------|---------|----------------| | **STS + WIF** | Scenario 1 (today) | The mint sends the JWT to **GCP Security Token Service**, which validates it against a **Workload Identity Federation** pool (CEL rules on repo/org). GCP returns a federated token; the mint also decodes and checks claims in Go. Trust is anchored in **GCP**. | -| **JWKS** | Scenario 3 (target) | **JWKS** = *JSON Web Key Set*: the public signing keys GitHub publishes (e.g. `https://token.actions.githubusercontent.com/.well-known/jwks`). The mint **verifies the JWT signature** against those keys locally (no GCP STS call), then runs the **same** claim checks in application code. Trust is anchored in **GitHub’s keys + mint logic**. | +| **JWKS** | Scenario 3 (target) | **JWKS** = *JSON Web Key Set*: the public signing keys GitHub publishes (e.g. `https://token.actions.githubusercontent.com/.well-known/jwks`). The mint **verifies the JWT signature** against those keys locally (no GCP STS call), then runs the **same** claim checks in **`mintcore`** (no GCP STS call). Trust is anchored in **GitHub’s keys + mint logic**. | -A Scenario 3 port is often called “STS→JWKS” because the **authorization outcome** should match; only the **validation backend** changes. Per-repo WIF provider selection becomes equivalent routing in mint config, not a second GCP pool lookup at request time. +A Scenario 3 port is often called “STS→JWKS” because the **authorization outcome** should match; only the **validation backend** changes. + +**Public mint profile** ([ADR 0059](../ADRs/0059-public-mint-mode-with-wildcard-allowlists.md)): `ALLOWED_ORGS=*`, permissive default WIF provider, **empty** `PER_REPO_WIF_REPOS`, and **upstream-only** `job_workflow_ref`. Per-repo WIF provider routing is a **tight-mode** concern only. + +**`aud` (audience):** Validated in **`mintcore`** via `OIDC_AUDIENCE` on both STS and JWKS paths—not by WIF/STS. The JWKS port does not change audience rules. ## Integrated evaluation @@ -43,11 +47,11 @@ All requirements below apply together—not as a separate “constraints” chec | **Launch speed** | ● Shipping binary + GitOps | ● + DNS/WAF rules | ◐ Worker port + parity tests | | **Public vs tenant deploy** | GitOps ([#1263](https://github.com/fullsend-ai/fullsend/issues/1263)); CLI for tenants | Same | GitOps (Wrangler); CLI for tenants later | | **Isolated from inference/LLM** | ● Dedicated GCP project only | ● Same project; CF zone for mint host | ● No Vertex in mint project; don’t colocate inference Workers | -| **Org + per-repo WIF** | ● Today | ● Same origin | ◐ JWKS path must keep per-repo provider logic | +| **Public mint trust ([ADR 0059](../ADRs/0059-public-mint-mode-with-wildcard-allowlists.md))** | ● STS + permissive WIF + mintcore claims | ● Unchanged at origin | ◐ Prove JWKS path ≡ STS path in CI | | **ADR 0029 trust** | ● STS + WIF + handler claims | ● Unchanged at origin | ◐ Prove JWKS path ≡ STS path in CI | | **Internet-facing abuse** | ◐ Armor+LB setup (still one vendor) | ● Easy WAF/RL | ● WAF/RL in same console as compute | | **Ongoing human ops** | ● **Single dashboard**; paging from GCP | ○ **Dual dashboard**; split incident triage | ● **Single dashboard**; paging from CF | -| **One-time engineering** | Low | Low–medium | Medium (AI-friendly port behind existing interfaces) | +| **One-time engineering** | Low | Low–medium | Medium (port behind existing `mintcore` interfaces) | | **Long-term multi-host** | ◐ GCP-centric | ◐ Split | ● Adds non-GCP option for tenants/public | | **Blocks [#914](https://github.com/fullsend-ai/fullsend/issues/914)** | No | No | No if launch on 1 first | | **Cost (~$0 community budget)** | ● Bare GCF+WIF+SM | ○ Free CF = weak edge; paid CF breaks budget | ● Workers Free at community volume | @@ -65,16 +69,16 @@ Community mint traffic is **low** (roughly one `POST /v1/token` per agent job ba **Effect on the decision (with ops and security):** -- **Reinforces** launch on **Scenario 1** without Armor/LB—allowlists and GCP alerts are the $0 edge story until something else is funded or Scenario 3 ships. +- **Reinforces** launch on **Scenario 1** without Armor/LB—upstream-only provenance ([ADR 0059](../ADRs/0059-public-mint-mode-with-wildcard-allowlists.md)) and GCP alerts are the $0 edge story until Scenario 3 ships. - **Discourages** a **long-lived Scenario 2** bridge unless a CF plan is donated; otherwise you pay in **operator time** (two consoles) without buying real protection. -- **Keeps Scenario 3** as the **$0 steady-state** path for WAF/rate limits **plus** one dashboard once the JWKS port is done—it does **not** force skipping Scenario 1 at launch (both can be ~$0 for compute; the port is engineering time, not subscription). +- **Keeps Scenario 3** as the **$0 steady-state** path for WAF/rate limits **plus** one dashboard once the JWKS port is done. ### Does two dashboards push toward 1 or 3? **Toward 1 or 3—not 2.** - **Scenario 2** optimizes **edge convenience** at the price of **permanent split-brain ops**: Bootstrap must correlate GCF 5xx/latency with Cloudflare origin errors and WAF blocks; synthetic checks should hit the public hostname *and* the origin; runbooks always have two hops. Alert fan-in to one pager helps but does not remove the second UI for tuning and incidents. -- **Scenario 1** keeps **one vendor console** for mint health. The tradeoff is weaker **default** edge on a bare `cloudfunctions.net` URL—mitigate at launch with fail-closed env allowlists (`ALLOWED_ORGS`, `ALLOWED_WORKFLOW_FILES`) and GCP alerts. **Cloud Armor + LB** is the one-dashboard GCP hardening option but **not** on a ~$0 budget (see [Cost](#cost-0-community-budget)); funded abuse response or **Scenario 3** is the realistic hardening path. +- **Scenario 1** keeps **one vendor console** for mint health. The tradeoff is weaker **default** edge on a bare `cloudfunctions.net` URL—mitigate at launch with fail-closed env allowlists per [ADR 0059](../ADRs/0059-public-mint-mode-with-wildcard-allowlists.md) and GCP alerts. **Cloud Armor + LB** is the one-dashboard GCP hardening option but **not** on a ~$0 budget (see [Cost](#cost-0-community-budget)); funded abuse response or **Scenario 3** is the realistic hardening path. - **Scenario 3** is the way to get **strong edge + single dashboard** without Scenario 2’s ops tax. The STS→JWKS port is a **one-time** cost; Bootstrap’s recurring load is CF-only. Prefer this over staying on Scenario 2 indefinitely. **Scenario 2 is justified only as a time-boxed bridge** (weeks, not years): public hostname needs WAF before Workers ship, and Bootstrap accepts dual-console overhead temporarily. @@ -86,17 +90,17 @@ Community mint traffic is **low** (roughly one `POST /v1/token` per agent job ba - **SLO owner:** Red Hat Fullsend Bootstrap until community ops exists. Target: **99.5%** availability for `POST /v1/token` (excl. GitHub outages), **p95 < 2s**. - **Signals (Scenario 1):** GCF 5xx/latency, STS/SM errors, synthetic `POST /v1/token` (expect 401), allowlist-change audit via GitOps. - **Signals (Scenario 3):** Worker errors, WAF blocks, same synthetic on public URL—no origin correlation. -- **Portability:** Second `TokenValidator` (JWKS) and `PEMAccessor`; parity tests STS vs JWKS—strategic, not launch-blocking. +- **Portability:** `JWKSVerifier` in `mintcore` (used by `cmd/mint/`); parity tests STS vs JWKS—strategic, not launch-blocking. ## Phasing | Phase | Choice | Rationale | |-------|--------|-----------| -| **Launch** | **Scenario 1** + GitOps + isolated project | Fastest; one console; unblocks [#914](https://github.com/fullsend-ai/fullsend/issues/914) | +| **Launch** | **Scenario 1** + GitOps + isolated project | Fastest; one console; unblocks [#914](https://github.com/fullsend-ai/fullsend/issues/914) / [#1145](https://github.com/fullsend-ai/fullsend/issues/1145) | | **Bridge (optional)** | **Scenario 2** | Only if abuse/WAF needed before Workers *and* CF budget exists; poor fit on $0 + dual console | | **Hardened steady state** | **Scenario 3** | WAF + rate limits + **one** ops surface; avoids chronic dual-dashboard | -| **Parallel** | JWKS/port work | AI-heavy; does not block launch | -| **Tenants** | GCP CLI path now; CF when Scenario 3 matures | Tenants choose; public URL stays opaque | +| **Parallel** | JWKS/port work | `cmd/mint/` proves JWKS path; does not block launch | +| **Tenants** | GCP CLI path or `cmd/mint/` now; CF when Scenario 3 matures | Tenants choose; public URL stays opaque | ## Open follow-ups @@ -104,11 +108,13 @@ Community mint traffic is **low** (roughly one `POST /v1/token` per agent job ba 2. Allowlist-only edge until Scenario 3 (Armor+LB excluded on ~$0 budget unless funding appears). 3. Rate-limit thresholds for public hostname (when edge exists). 4. Criteria to transfer SLO ownership from Bootstrap to community ops. +5. Implement `ALLOWED_ORGS=*` and upstream-only workflow validation in `mintcore` per [ADR 0059](../ADRs/0059-public-mint-mode-with-wildcard-allowlists.md). ## References -- [Infrastructure reference — Token mint](../guides/admin/infrastructure-reference.md) +- [Infrastructure reference — Token mint](../guides/infrastructure/infrastructure-reference.md) - [ADR 0029](../ADRs/0029-central-token-mint-secretless-fullsend.md) -- [ADR 0040 — Public community mint architecture](../ADRs/0040-public-community-mint-architecture.md) +- [ADR 0059](../ADRs/0059-public-mint-mode-with-wildcard-allowlists.md) +- [ADR 0061 — Public community mint architecture](../ADRs/0061-public-community-mint-architecture.md) - [#1263](https://github.com/fullsend-ai/fullsend/issues/1263) · [#1262](https://github.com/fullsend-ai/fullsend/issues/1262) - [#915](https://github.com/fullsend-ai/fullsend/issues/915) discussion (2026-05-25) From 4dddd6d73301d8d6e667fe20d07e410ba96db982 Mon Sep 17 00:00:00 2001 From: Barak Korren Date: Wed, 1 Jul 2026 08:36:20 +0300 Subject: [PATCH 4/8] chore: merge main and renumber public mint ADR to 0062 ADR 0061 was taken by harness-cel-dispatch on main. Merge latest upstream/main and renumber the community mint architecture ADR. Signed-off-by: Barak Korren Co-authored-by: Cursor --- .../0059-public-mint-mode-with-wildcard-allowlists.md | 10 +++++----- ...e.md => 0062-public-community-mint-architecture.md} | 4 ++-- docs/architecture.md | 2 +- ...6-05-25-community-mint-hosting-gcp-vs-cloudflare.md | 4 ++-- 4 files changed, 10 insertions(+), 10 deletions(-) rename docs/ADRs/{0061-public-community-mint-architecture.md => 0062-public-community-mint-architecture.md} (99%) diff --git a/docs/ADRs/0059-public-mint-mode-with-wildcard-allowlists.md b/docs/ADRs/0059-public-mint-mode-with-wildcard-allowlists.md index 72c0bfbf22..12eb83c5d8 100644 --- a/docs/ADRs/0059-public-mint-mode-with-wildcard-allowlists.md +++ b/docs/ADRs/0059-public-mint-mode-with-wildcard-allowlists.md @@ -92,15 +92,15 @@ Introduce a **public mint mode** when `ALLOWED_ORGS` contains `*`, with `job_wor 11. **Deferred to future ADRs (mint-only):** - Dedicated reusable workflow(s) for custom agent **stages** beyond the built-in set. - **Prioritize** on `workflow_dispatch`: whether manual prioritize mints via an upstream `reusable-*.yml` or remains tight-mint-only until wired ([ADR 0041](0041-synchronous-workflow-call-event-dispatch.md) keeps non-event `workflow_dispatch` entry points). - - **Mint infrastructure** (WIF pool/provider provisioning, Cloud Function deployment, CEL definitions, abuse controls, WAF, monitoring): see [ADR 0061](0061-public-community-mint-architecture.md). - - **Hosted mint enrollment:** resolved — public mode (`ALLOWED_ORGS=*`) + shared App install per §1 and [ADR 0061](0061-public-community-mint-architecture.md). + - **Mint infrastructure** (WIF pool/provider provisioning, Cloud Function deployment, CEL definitions, abuse controls, WAF, monitoring): see [ADR 0062](0062-public-community-mint-architecture.md). + - **Hosted mint enrollment:** resolved — public mode (`ALLOWED_ORGS=*`) + shared App install per §1 and [ADR 0062](0062-public-community-mint-architecture.md). 12. **Normative specs (`docs/normative/`).** Not required for this decision. The reference mint is implemented in `internal/mintcore/` and configured via documented env vars; this ADR plus [ADR 0029](0029-central-token-mint-secretless-fullsend.md) are the contract for that implementation. A versioned normative spec is **out of scope** until there are multiple independent mint implementations that must interoperate on the same byte-level env and claim rules. ## Consequences - Public mode: `ALLOWED_ORGS=*`, permissive provider id in `WIF_PROVIDER_NAME`, `PER_REPO_WIF_REPOS` empty. Tight mode: explicit `ALLOWED_ORGS`, org-merged provider in `WIF_PROVIDER_NAME`, optional `PER_REPO_WIF_REPOS` list. -- Provider provisioning and IAM for the **hosted** public mint are specified in [ADR 0061](0061-public-community-mint-architecture.md); tight/self-managed deployments remain operator-defined. +- Provider provisioning and IAM for the **hosted** public mint are specified in [ADR 0062](0062-public-community-mint-architecture.md); tight/self-managed deployments remain operator-defined. - Custom agents remain configuration-only unless a new shared App, mint role, and upstream workflow file are added. - Per-repo **public** mint assumes event-driven runs mint from `fullsend-ai/fullsend` reusables per [ADR 0041](0041-synchronous-workflow-call-event-dispatch.md) and [ADR 0033](0033-per-repo-installation-mode.md). Legacy `{org}/.fullsend/` provenance is not supported in public mode and is deprecated by [ADR 0044](0044-deprecate-per-org-installation-mode.md). - Ref-any on `fullsend-ai/fullsend` at the mint layer trades pinning rigor for simpler rollout; caller-level SHA pinning per [ADR 0031](0031-reusable-workflows-for-action-installed-distribution.md) remains the recommended org/repo control. @@ -126,7 +126,7 @@ Introduce a **public mint mode** when `ALLOWED_ORGS` contains `*`, with `job_wor |-------|-------------| | Custom-agent reusable workflow | Future ADR | | Prioritize mint path under public mode | Resolve when non-event workflows are wired to reusables ([ADR 0041](0041-synchronous-workflow-call-event-dispatch.md)) | -| Mint WIF providers, IAM, deployment, abuse/WAF | [ADR 0061](0061-public-community-mint-architecture.md) | -| Hosted mint public-mode enrollment | [ADR 0061](0061-public-community-mint-architecture.md) + §1 here | +| Mint WIF providers, IAM, deployment, abuse/WAF | [ADR 0062](0062-public-community-mint-architecture.md) | +| Hosted mint public-mode enrollment | [ADR 0062](0062-public-community-mint-architecture.md) + §1 here | | Normative mint contract | Deferred until multiple implementations; not needed now | | Stricter `@ref` allowlist on upstream workflows | Optional later hardening | diff --git a/docs/ADRs/0061-public-community-mint-architecture.md b/docs/ADRs/0062-public-community-mint-architecture.md similarity index 99% rename from docs/ADRs/0061-public-community-mint-architecture.md rename to docs/ADRs/0062-public-community-mint-architecture.md index a552519d8c..a886a89ccc 100644 --- a/docs/ADRs/0061-public-community-mint-architecture.md +++ b/docs/ADRs/0062-public-community-mint-architecture.md @@ -1,5 +1,5 @@ --- -title: "61. Public community mint architecture" +title: "62. Public community mint architecture" status: Accepted relates_to: - agent-infrastructure @@ -14,7 +14,7 @@ topics: - key-rotation --- -# 61. Public community mint architecture +# 62. Public community mint architecture Date: 2026-05-25 diff --git a/docs/architecture.md b/docs/architecture.md index e9802b8963..c0ec00d8e8 100644 --- a/docs/architecture.md +++ b/docs/architecture.md @@ -148,7 +148,7 @@ Identity is not the same as trust. An agent's identity lets it authenticate to e - Per-role GitHub Apps with manifest-based creation. Each agent role gets its own app with scoped permissions. PEMs stored in Secret Manager as `fullsend-{role}-app-pem` — one secret per role, shared across orgs on a mint. `ROLE_APP_IDS` uses the same shared-per-role model (`coder` → app ID). Org isolation is enforced via `ALLOWED_ORGS`, WIF conditions, and installation verification ([ADR 0007](ADRs/0007-per-role-github-apps.md), [ADR 0033](ADRs/0033-per-repo-installation-mode.md)). Public multi-tenant mint (`ALLOWED_ORGS=*`) with upstream-only workflow provenance is defined in [ADR 0059](ADRs/0059-public-mint-mode-with-wildcard-allowlists.md); upstream-only provenance limits which workflows can call the mint, complementing [ADR 0029](ADRs/0029-central-token-mint-secretless-fullsend.md) multi-tenant blast-radius concerns. - Cross-org mint authorization: workflows may request tokens for a different org via optional `target_org` when the target org installs the role App and sets `FULLSEND_FOREIGN__REPOS`. Empty `repos` yields installation-wide tokens on either path; cross-org adds FOREIGN gating, same-org relies on WIF/OIDC enrollment ([ADR 0060](ADRs/0060-cross-org-mint-authorization-via-org-variables.md)). - Standalone mint deployment: `cmd/mint/` provides a self-contained HTTP server that uses direct JWKS verification and filesystem PEM storage instead of GCP infrastructure. It shares the `internal/mintcore/` library with the GCF mint and adds support for custom role permissions and a fallback proxy to an upstream mint. Custom role permissions live in mintcore (not `cmd/mint/`) so that `RolePermissionsFor`, `HasRole`, and `CreateInstallationToken` return a unified view without callers needing to distinguish built-in from custom roles. The GCF mint never calls `RegisterCustomRolePermissions`, so the code is inert there. See the [standalone mint guide](guides/infrastructure/standalone-mint.md). -- Hosted public community mint: steady-state deployment on Cloudflare Workers (JWKS + WAF + single ops console), with interim GCP Cloud Function acceptable until the Worker port is production-ready. Trust policy (`ALLOWED_ORGS=*`, upstream-only workflow provenance) is in [ADR 0059](ADRs/0059-public-mint-mode-with-wildcard-allowlists.md); deployment, edge security, monitoring, and phasing are in [ADR 0061](ADRs/0061-public-community-mint-architecture.md). Enrollment is installing the shared Apps—no per-org mint env registration ([#1145](https://github.com/fullsend-ai/fullsend/issues/1145)). +- Hosted public community mint: steady-state deployment on Cloudflare Workers (JWKS + WAF + single ops console), with interim GCP Cloud Function acceptable until the Worker port is production-ready. Trust policy (`ALLOWED_ORGS=*`, upstream-only workflow provenance) is in [ADR 0059](ADRs/0059-public-mint-mode-with-wildcard-allowlists.md); deployment, edge security, monitoring, and phasing are in [ADR 0061](ADRs/0062-public-community-mint-architecture.md). Enrollment is installing the shared Apps—no per-org mint env registration ([#1145](https://github.com/fullsend-ai/fullsend/issues/1145)). One concrete implementation option is [`oidcx`](https://github.com/oxidecomputer/oidcx): a service that accepts OIDC identity tokens and exchanges them for short-lived access tokens. It can mint tokens scoped to selected GitHub repositories and permissions, or to selected Oxide silos and permissions, and it also ships with a GitHub Action wrapper. In a Fullsend deployment, this can be used by the sandbox entrypoint to narrow a broad GitHub App identity down to only the specific permissions an agent needs for the current run. diff --git a/docs/spikes/2026-05-25-community-mint-hosting-gcp-vs-cloudflare.md b/docs/spikes/2026-05-25-community-mint-hosting-gcp-vs-cloudflare.md index 59299389ae..0073c42cb0 100644 --- a/docs/spikes/2026-05-25-community-mint-hosting-gcp-vs-cloudflare.md +++ b/docs/spikes/2026-05-25-community-mint-hosting-gcp-vs-cloudflare.md @@ -1,7 +1,7 @@ # SPIKE: Community token mint hosting (GCP vs Cloudflare) **Issue:** [#915](https://github.com/fullsend-ai/fullsend/issues/915) · **Parent:** [#914](https://github.com/fullsend-ai/fullsend/issues/914) · **Epic:** [#912](https://github.com/fullsend-ai/fullsend/issues/912) -**ADR:** [0029](../ADRs/0029-central-token-mint-secretless-fullsend.md) · [0059](../ADRs/0059-public-mint-mode-with-wildcard-allowlists.md) (trust) · [0061](../ADRs/0061-public-community-mint-architecture.md) (ops/deployment) · **GitOps:** [#1263](https://github.com/fullsend-ai/fullsend/issues/1263) · **Date:** 2026-05-25 +**ADR:** [0029](../ADRs/0029-central-token-mint-secretless-fullsend.md) · [0059](../ADRs/0059-public-mint-mode-with-wildcard-allowlists.md) (trust) · [0062](../ADRs/0062-public-community-mint-architecture.md) (ops/deployment) · **GitOps:** [#1263](https://github.com/fullsend-ai/fullsend/issues/1263) · **Date:** 2026-05-25 ## Decision @@ -115,6 +115,6 @@ Community mint traffic is **low** (roughly one `POST /v1/token` per agent job ba - [Infrastructure reference — Token mint](../guides/infrastructure/infrastructure-reference.md) - [ADR 0029](../ADRs/0029-central-token-mint-secretless-fullsend.md) - [ADR 0059](../ADRs/0059-public-mint-mode-with-wildcard-allowlists.md) -- [ADR 0061 — Public community mint architecture](../ADRs/0061-public-community-mint-architecture.md) +- [ADR 0061 — Public community mint architecture](../ADRs/0062-public-community-mint-architecture.md) - [#1263](https://github.com/fullsend-ai/fullsend/issues/1263) · [#1262](https://github.com/fullsend-ai/fullsend/issues/1262) - [#915](https://github.com/fullsend-ai/fullsend/issues/915) discussion (2026-05-25) From 6ec9eaf3e97c268e93f1dab88389ae5fac7a8ae8 Mon Sep 17 00:00:00 2001 From: Barak Korren Date: Thu, 2 Jul 2026 10:07:42 +0300 Subject: [PATCH 5/8] docs: fix ADR 0062 link text in architecture and spike Correct stale "ADR 0061" display text pointing at the community mint architecture file (0062), which review flagged as misleading. Signed-off-by: Barak Korren Co-authored-by: Cursor --- docs/architecture.md | 2 +- .../2026-05-25-community-mint-hosting-gcp-vs-cloudflare.md | 2 +- 2 files changed, 2 insertions(+), 2 deletions(-) diff --git a/docs/architecture.md b/docs/architecture.md index c0ec00d8e8..c20cc805bc 100644 --- a/docs/architecture.md +++ b/docs/architecture.md @@ -148,7 +148,7 @@ Identity is not the same as trust. An agent's identity lets it authenticate to e - Per-role GitHub Apps with manifest-based creation. Each agent role gets its own app with scoped permissions. PEMs stored in Secret Manager as `fullsend-{role}-app-pem` — one secret per role, shared across orgs on a mint. `ROLE_APP_IDS` uses the same shared-per-role model (`coder` → app ID). Org isolation is enforced via `ALLOWED_ORGS`, WIF conditions, and installation verification ([ADR 0007](ADRs/0007-per-role-github-apps.md), [ADR 0033](ADRs/0033-per-repo-installation-mode.md)). Public multi-tenant mint (`ALLOWED_ORGS=*`) with upstream-only workflow provenance is defined in [ADR 0059](ADRs/0059-public-mint-mode-with-wildcard-allowlists.md); upstream-only provenance limits which workflows can call the mint, complementing [ADR 0029](ADRs/0029-central-token-mint-secretless-fullsend.md) multi-tenant blast-radius concerns. - Cross-org mint authorization: workflows may request tokens for a different org via optional `target_org` when the target org installs the role App and sets `FULLSEND_FOREIGN__REPOS`. Empty `repos` yields installation-wide tokens on either path; cross-org adds FOREIGN gating, same-org relies on WIF/OIDC enrollment ([ADR 0060](ADRs/0060-cross-org-mint-authorization-via-org-variables.md)). - Standalone mint deployment: `cmd/mint/` provides a self-contained HTTP server that uses direct JWKS verification and filesystem PEM storage instead of GCP infrastructure. It shares the `internal/mintcore/` library with the GCF mint and adds support for custom role permissions and a fallback proxy to an upstream mint. Custom role permissions live in mintcore (not `cmd/mint/`) so that `RolePermissionsFor`, `HasRole`, and `CreateInstallationToken` return a unified view without callers needing to distinguish built-in from custom roles. The GCF mint never calls `RegisterCustomRolePermissions`, so the code is inert there. See the [standalone mint guide](guides/infrastructure/standalone-mint.md). -- Hosted public community mint: steady-state deployment on Cloudflare Workers (JWKS + WAF + single ops console), with interim GCP Cloud Function acceptable until the Worker port is production-ready. Trust policy (`ALLOWED_ORGS=*`, upstream-only workflow provenance) is in [ADR 0059](ADRs/0059-public-mint-mode-with-wildcard-allowlists.md); deployment, edge security, monitoring, and phasing are in [ADR 0061](ADRs/0062-public-community-mint-architecture.md). Enrollment is installing the shared Apps—no per-org mint env registration ([#1145](https://github.com/fullsend-ai/fullsend/issues/1145)). +- Hosted public community mint: steady-state deployment on Cloudflare Workers (JWKS + WAF + single ops console), with interim GCP Cloud Function acceptable until the Worker port is production-ready. Trust policy (`ALLOWED_ORGS=*`, upstream-only workflow provenance) is in [ADR 0059](ADRs/0059-public-mint-mode-with-wildcard-allowlists.md); deployment, edge security, monitoring, and phasing are in [ADR 0062](ADRs/0062-public-community-mint-architecture.md). Enrollment is installing the shared Apps—no per-org mint env registration ([#1145](https://github.com/fullsend-ai/fullsend/issues/1145)). One concrete implementation option is [`oidcx`](https://github.com/oxidecomputer/oidcx): a service that accepts OIDC identity tokens and exchanges them for short-lived access tokens. It can mint tokens scoped to selected GitHub repositories and permissions, or to selected Oxide silos and permissions, and it also ships with a GitHub Action wrapper. In a Fullsend deployment, this can be used by the sandbox entrypoint to narrow a broad GitHub App identity down to only the specific permissions an agent needs for the current run. diff --git a/docs/spikes/2026-05-25-community-mint-hosting-gcp-vs-cloudflare.md b/docs/spikes/2026-05-25-community-mint-hosting-gcp-vs-cloudflare.md index 0073c42cb0..2bb39664cb 100644 --- a/docs/spikes/2026-05-25-community-mint-hosting-gcp-vs-cloudflare.md +++ b/docs/spikes/2026-05-25-community-mint-hosting-gcp-vs-cloudflare.md @@ -115,6 +115,6 @@ Community mint traffic is **low** (roughly one `POST /v1/token` per agent job ba - [Infrastructure reference — Token mint](../guides/infrastructure/infrastructure-reference.md) - [ADR 0029](../ADRs/0029-central-token-mint-secretless-fullsend.md) - [ADR 0059](../ADRs/0059-public-mint-mode-with-wildcard-allowlists.md) -- [ADR 0061 — Public community mint architecture](../ADRs/0062-public-community-mint-architecture.md) +- [ADR 0062 — Public community mint architecture](../ADRs/0062-public-community-mint-architecture.md) - [#1263](https://github.com/fullsend-ai/fullsend/issues/1263) · [#1262](https://github.com/fullsend-ai/fullsend/issues/1262) - [#915](https://github.com/fullsend-ai/fullsend/issues/915) discussion (2026-05-25) From ff9df42951060911504c82dc678be1ffc41173f0 Mon Sep 17 00:00:00 2001 From: Barak Korren Date: Sun, 5 Jul 2026 09:45:08 +0300 Subject: [PATCH 6/8] chore: merge main, renumber community mint ADR to 0063 ADR 0062 is dispatch-version-skew on main. Renumber the public community mint architecture ADR to 0063, update cross-references, and add docs/spikes to the VitePress sidebar. Signed-off-by: Barak Korren Co-authored-by: Cursor --- .../0059-public-mint-mode-with-wildcard-allowlists.md | 10 +++++----- ...e.md => 0063-public-community-mint-architecture.md} | 4 ++-- docs/architecture.md | 2 +- ...6-05-25-community-mint-hosting-gcp-vs-cloudflare.md | 4 ++-- website/.vitepress/config.ts | 5 +++++ 5 files changed, 15 insertions(+), 10 deletions(-) rename docs/ADRs/{0062-public-community-mint-architecture.md => 0063-public-community-mint-architecture.md} (99%) diff --git a/docs/ADRs/0059-public-mint-mode-with-wildcard-allowlists.md b/docs/ADRs/0059-public-mint-mode-with-wildcard-allowlists.md index 12eb83c5d8..fc55dc4335 100644 --- a/docs/ADRs/0059-public-mint-mode-with-wildcard-allowlists.md +++ b/docs/ADRs/0059-public-mint-mode-with-wildcard-allowlists.md @@ -92,15 +92,15 @@ Introduce a **public mint mode** when `ALLOWED_ORGS` contains `*`, with `job_wor 11. **Deferred to future ADRs (mint-only):** - Dedicated reusable workflow(s) for custom agent **stages** beyond the built-in set. - **Prioritize** on `workflow_dispatch`: whether manual prioritize mints via an upstream `reusable-*.yml` or remains tight-mint-only until wired ([ADR 0041](0041-synchronous-workflow-call-event-dispatch.md) keeps non-event `workflow_dispatch` entry points). - - **Mint infrastructure** (WIF pool/provider provisioning, Cloud Function deployment, CEL definitions, abuse controls, WAF, monitoring): see [ADR 0062](0062-public-community-mint-architecture.md). - - **Hosted mint enrollment:** resolved — public mode (`ALLOWED_ORGS=*`) + shared App install per §1 and [ADR 0062](0062-public-community-mint-architecture.md). + - **Mint infrastructure** (WIF pool/provider provisioning, Cloud Function deployment, CEL definitions, abuse controls, WAF, monitoring): see [ADR 0063](0063-public-community-mint-architecture.md). + - **Hosted mint enrollment:** resolved — public mode (`ALLOWED_ORGS=*`) + shared App install per §1 and [ADR 0063](0063-public-community-mint-architecture.md). 12. **Normative specs (`docs/normative/`).** Not required for this decision. The reference mint is implemented in `internal/mintcore/` and configured via documented env vars; this ADR plus [ADR 0029](0029-central-token-mint-secretless-fullsend.md) are the contract for that implementation. A versioned normative spec is **out of scope** until there are multiple independent mint implementations that must interoperate on the same byte-level env and claim rules. ## Consequences - Public mode: `ALLOWED_ORGS=*`, permissive provider id in `WIF_PROVIDER_NAME`, `PER_REPO_WIF_REPOS` empty. Tight mode: explicit `ALLOWED_ORGS`, org-merged provider in `WIF_PROVIDER_NAME`, optional `PER_REPO_WIF_REPOS` list. -- Provider provisioning and IAM for the **hosted** public mint are specified in [ADR 0062](0062-public-community-mint-architecture.md); tight/self-managed deployments remain operator-defined. +- Provider provisioning and IAM for the **hosted** public mint are specified in [ADR 0063](0063-public-community-mint-architecture.md); tight/self-managed deployments remain operator-defined. - Custom agents remain configuration-only unless a new shared App, mint role, and upstream workflow file are added. - Per-repo **public** mint assumes event-driven runs mint from `fullsend-ai/fullsend` reusables per [ADR 0041](0041-synchronous-workflow-call-event-dispatch.md) and [ADR 0033](0033-per-repo-installation-mode.md). Legacy `{org}/.fullsend/` provenance is not supported in public mode and is deprecated by [ADR 0044](0044-deprecate-per-org-installation-mode.md). - Ref-any on `fullsend-ai/fullsend` at the mint layer trades pinning rigor for simpler rollout; caller-level SHA pinning per [ADR 0031](0031-reusable-workflows-for-action-installed-distribution.md) remains the recommended org/repo control. @@ -126,7 +126,7 @@ Introduce a **public mint mode** when `ALLOWED_ORGS` contains `*`, with `job_wor |-------|-------------| | Custom-agent reusable workflow | Future ADR | | Prioritize mint path under public mode | Resolve when non-event workflows are wired to reusables ([ADR 0041](0041-synchronous-workflow-call-event-dispatch.md)) | -| Mint WIF providers, IAM, deployment, abuse/WAF | [ADR 0062](0062-public-community-mint-architecture.md) | -| Hosted mint public-mode enrollment | [ADR 0062](0062-public-community-mint-architecture.md) + §1 here | +| Mint WIF providers, IAM, deployment, abuse/WAF | [ADR 0063](0063-public-community-mint-architecture.md) | +| Hosted mint public-mode enrollment | [ADR 0063](0063-public-community-mint-architecture.md) + §1 here | | Normative mint contract | Deferred until multiple implementations; not needed now | | Stricter `@ref` allowlist on upstream workflows | Optional later hardening | diff --git a/docs/ADRs/0062-public-community-mint-architecture.md b/docs/ADRs/0063-public-community-mint-architecture.md similarity index 99% rename from docs/ADRs/0062-public-community-mint-architecture.md rename to docs/ADRs/0063-public-community-mint-architecture.md index a886a89ccc..be770bb733 100644 --- a/docs/ADRs/0062-public-community-mint-architecture.md +++ b/docs/ADRs/0063-public-community-mint-architecture.md @@ -1,5 +1,5 @@ --- -title: "62. Public community mint architecture" +title: "63. Public community mint architecture" status: Accepted relates_to: - agent-infrastructure @@ -14,7 +14,7 @@ topics: - key-rotation --- -# 62. Public community mint architecture +# 63. Public community mint architecture Date: 2026-05-25 diff --git a/docs/architecture.md b/docs/architecture.md index 8fba1cc373..0fd2c3efcc 100644 --- a/docs/architecture.md +++ b/docs/architecture.md @@ -149,7 +149,7 @@ Identity is not the same as trust. An agent's identity lets it authenticate to e - Per-role GitHub Apps with manifest-based creation. Each agent role gets its own app with scoped permissions. PEMs stored in Secret Manager as `fullsend-{role}-app-pem` — one secret per role, shared across orgs on a mint. `ROLE_APP_IDS` uses the same shared-per-role model (`coder` → app ID). Org isolation is enforced via `ALLOWED_ORGS`, WIF conditions, and installation verification ([ADR 0007](ADRs/0007-per-role-github-apps.md), [ADR 0033](ADRs/0033-per-repo-installation-mode.md)). Public multi-tenant mint (`ALLOWED_ORGS=*`) with upstream-only workflow provenance is defined in [ADR 0059](ADRs/0059-public-mint-mode-with-wildcard-allowlists.md); upstream-only provenance limits which workflows can call the mint, complementing [ADR 0029](ADRs/0029-central-token-mint-secretless-fullsend.md) multi-tenant blast-radius concerns. - Cross-org mint authorization: workflows may request tokens for a different org via optional `target_org` when the target org installs the role App and sets `FULLSEND_FOREIGN__REPOS`. Empty `repos` yields installation-wide tokens on either path; cross-org adds FOREIGN gating, same-org relies on WIF/OIDC enrollment ([ADR 0060](ADRs/0060-cross-org-mint-authorization-via-org-variables.md)). - Standalone mint deployment: `cmd/mint/` provides a self-contained HTTP server that uses direct JWKS verification and filesystem PEM storage instead of GCP infrastructure. It shares the `internal/mintcore/` library with the GCF mint and adds support for custom role permissions and a fallback proxy to an upstream mint. Custom role permissions live in mintcore (not `cmd/mint/`) so that `RolePermissionsFor`, `HasRole`, and `CreateInstallationToken` return a unified view without callers needing to distinguish built-in from custom roles. The GCF mint never calls `RegisterCustomRolePermissions`, so the code is inert there. See the [standalone mint guide](guides/infrastructure/standalone-mint.md). -- Hosted public community mint: steady-state deployment on Cloudflare Workers (JWKS + WAF + single ops console), with interim GCP Cloud Function acceptable until the Worker port is production-ready. Trust policy (`ALLOWED_ORGS=*`, upstream-only workflow provenance) is in [ADR 0059](ADRs/0059-public-mint-mode-with-wildcard-allowlists.md); deployment, edge security, monitoring, and phasing are in [ADR 0062](ADRs/0062-public-community-mint-architecture.md). Enrollment is installing the shared Apps—no per-org mint env registration ([#1145](https://github.com/fullsend-ai/fullsend/issues/1145)). +- Hosted public community mint: steady-state deployment on Cloudflare Workers (JWKS + WAF + single ops console), with interim GCP Cloud Function acceptable until the Worker port is production-ready. Trust policy (`ALLOWED_ORGS=*`, upstream-only workflow provenance) is in [ADR 0059](ADRs/0059-public-mint-mode-with-wildcard-allowlists.md); deployment, edge security, monitoring, and phasing are in [ADR 0063](ADRs/0063-public-community-mint-architecture.md). Enrollment is installing the shared Apps—no per-org mint env registration ([#1145](https://github.com/fullsend-ai/fullsend/issues/1145)). One concrete implementation option is [`oidcx`](https://github.com/oxidecomputer/oidcx): a service that accepts OIDC identity tokens and exchanges them for short-lived access tokens. It can mint tokens scoped to selected GitHub repositories and permissions, or to selected Oxide silos and permissions, and it also ships with a GitHub Action wrapper. In a Fullsend deployment, this can be used by the sandbox entrypoint to narrow a broad GitHub App identity down to only the specific permissions an agent needs for the current run. diff --git a/docs/spikes/2026-05-25-community-mint-hosting-gcp-vs-cloudflare.md b/docs/spikes/2026-05-25-community-mint-hosting-gcp-vs-cloudflare.md index 2bb39664cb..d509e41c75 100644 --- a/docs/spikes/2026-05-25-community-mint-hosting-gcp-vs-cloudflare.md +++ b/docs/spikes/2026-05-25-community-mint-hosting-gcp-vs-cloudflare.md @@ -1,7 +1,7 @@ # SPIKE: Community token mint hosting (GCP vs Cloudflare) **Issue:** [#915](https://github.com/fullsend-ai/fullsend/issues/915) · **Parent:** [#914](https://github.com/fullsend-ai/fullsend/issues/914) · **Epic:** [#912](https://github.com/fullsend-ai/fullsend/issues/912) -**ADR:** [0029](../ADRs/0029-central-token-mint-secretless-fullsend.md) · [0059](../ADRs/0059-public-mint-mode-with-wildcard-allowlists.md) (trust) · [0062](../ADRs/0062-public-community-mint-architecture.md) (ops/deployment) · **GitOps:** [#1263](https://github.com/fullsend-ai/fullsend/issues/1263) · **Date:** 2026-05-25 +**ADR:** [0029](../ADRs/0029-central-token-mint-secretless-fullsend.md) · [0059](../ADRs/0059-public-mint-mode-with-wildcard-allowlists.md) (trust) · [0063](../ADRs/0063-public-community-mint-architecture.md) (ops/deployment) · **GitOps:** [#1263](https://github.com/fullsend-ai/fullsend/issues/1263) · **Date:** 2026-05-25 ## Decision @@ -115,6 +115,6 @@ Community mint traffic is **low** (roughly one `POST /v1/token` per agent job ba - [Infrastructure reference — Token mint](../guides/infrastructure/infrastructure-reference.md) - [ADR 0029](../ADRs/0029-central-token-mint-secretless-fullsend.md) - [ADR 0059](../ADRs/0059-public-mint-mode-with-wildcard-allowlists.md) -- [ADR 0062 — Public community mint architecture](../ADRs/0062-public-community-mint-architecture.md) +- [ADR 0063 — Public community mint architecture](../ADRs/0063-public-community-mint-architecture.md) - [#1263](https://github.com/fullsend-ai/fullsend/issues/1263) · [#1262](https://github.com/fullsend-ai/fullsend/issues/1262) - [#915](https://github.com/fullsend-ai/fullsend/issues/915) discussion (2026-05-25) diff --git a/website/.vitepress/config.ts b/website/.vitepress/config.ts index 9e6a4b9005..488775f4c4 100644 --- a/website/.vitepress/config.ts +++ b/website/.vitepress/config.ts @@ -246,6 +246,11 @@ export default defineConfig({ collapsed: true, items: getMarkdownFiles('problems', 'problems'), }, + { + text: 'Spikes', + collapsed: true, + items: getMarkdownFiles('spikes', 'spikes'), + }, { text: 'Experiments (Exploratory)', collapsed: true, From ccdb32a0467da8bc3def5817437c28b772ed639f Mon Sep 17 00:00:00 2001 From: Barak Korren Date: Sun, 12 Jul 2026 09:54:44 +0300 Subject: [PATCH 7/8] docs(adr): address Jul 7 review on community mint ADR Revert frozen ADR 0059 sections to main. Document Workers PEM secret size (~1,675-byte PEMs fit the 5 KB limit), link automated rotation to #4175, and warn in mintcore when PEMs exceed the Workers limit. Signed-off-by: Barak Korren Co-authored-by: Cursor --- cmd/mint/main.go | 3 + ...blic-mint-mode-with-wildcard-allowlists.md | 10 +-- ...0063-public-community-mint-architecture.md | 8 +-- .../gcf/mintsrc/mintcore/file_pem.go.embed | 1 + .../gcf/mintsrc/mintcore/gcp_pem.go.embed | 1 + .../gcf/mintsrc/mintcore/pem_size.go.embed | 59 ++++++++++++++++++ internal/mintcore/file_pem.go | 1 + internal/mintcore/gcp_pem.go | 1 + internal/mintcore/pem_size.go | 59 ++++++++++++++++++ internal/mintcore/pem_size_test.go | 61 +++++++++++++++++++ 10 files changed, 195 insertions(+), 9 deletions(-) create mode 100644 internal/dispatch/gcf/mintsrc/mintcore/pem_size.go.embed create mode 100644 internal/mintcore/pem_size.go create mode 100644 internal/mintcore/pem_size_test.go diff --git a/cmd/mint/main.go b/cmd/mint/main.go index 674fe15406..92d3929e11 100644 --- a/cmd/mint/main.go +++ b/cmd/mint/main.go @@ -50,6 +50,9 @@ func buildHandler() (http.Handler, error) { if err != nil { return nil, fmt.Errorf("initializing PEM accessor: %w", err) } + if err := mintcore.WarnAllPEMsInDir(os.Getenv("PEM_DIR")); err != nil { + log.Printf("warning: scanning PEM directory: %v", err) + } handler, err := mintcore.NewHandler(pemAccessor, verifier) if err != nil { diff --git a/docs/ADRs/0059-public-mint-mode-with-wildcard-allowlists.md b/docs/ADRs/0059-public-mint-mode-with-wildcard-allowlists.md index fc55dc4335..ea3ccf33b2 100644 --- a/docs/ADRs/0059-public-mint-mode-with-wildcard-allowlists.md +++ b/docs/ADRs/0059-public-mint-mode-with-wildcard-allowlists.md @@ -92,15 +92,15 @@ Introduce a **public mint mode** when `ALLOWED_ORGS` contains `*`, with `job_wor 11. **Deferred to future ADRs (mint-only):** - Dedicated reusable workflow(s) for custom agent **stages** beyond the built-in set. - **Prioritize** on `workflow_dispatch`: whether manual prioritize mints via an upstream `reusable-*.yml` or remains tight-mint-only until wired ([ADR 0041](0041-synchronous-workflow-call-event-dispatch.md) keeps non-event `workflow_dispatch` entry points). - - **Mint infrastructure** (WIF pool/provider provisioning, Cloud Function deployment, CEL definitions, abuse controls, WAF, monitoring): see [ADR 0063](0063-public-community-mint-architecture.md). - - **Hosted mint enrollment:** resolved — public mode (`ALLOWED_ORGS=*`) + shared App install per §1 and [ADR 0063](0063-public-community-mint-architecture.md). + - Mint infrastructure: WIF pool/provider provisioning, Cloud Function deployment, CEL definitions, abuse controls, WAF, and monitoring. + - Hosted mint enrollment policy: whether the fullsend-operated hosted mint adopts public mode (`ALLOWED_ORGS=*`) or remains tight with explicit enrollment. 12. **Normative specs (`docs/normative/`).** Not required for this decision. The reference mint is implemented in `internal/mintcore/` and configured via documented env vars; this ADR plus [ADR 0029](0029-central-token-mint-secretless-fullsend.md) are the contract for that implementation. A versioned normative spec is **out of scope** until there are multiple independent mint implementations that must interoperate on the same byte-level env and claim rules. ## Consequences - Public mode: `ALLOWED_ORGS=*`, permissive provider id in `WIF_PROVIDER_NAME`, `PER_REPO_WIF_REPOS` empty. Tight mode: explicit `ALLOWED_ORGS`, org-merged provider in `WIF_PROVIDER_NAME`, optional `PER_REPO_WIF_REPOS` list. -- Provider provisioning and IAM for the **hosted** public mint are specified in [ADR 0063](0063-public-community-mint-architecture.md); tight/self-managed deployments remain operator-defined. +- Provider provisioning and IAM are deferred to a mint infrastructure ADR; this ADR does not specify how providers are created. - Custom agents remain configuration-only unless a new shared App, mint role, and upstream workflow file are added. - Per-repo **public** mint assumes event-driven runs mint from `fullsend-ai/fullsend` reusables per [ADR 0041](0041-synchronous-workflow-call-event-dispatch.md) and [ADR 0033](0033-per-repo-installation-mode.md). Legacy `{org}/.fullsend/` provenance is not supported in public mode and is deprecated by [ADR 0044](0044-deprecate-per-org-installation-mode.md). - Ref-any on `fullsend-ai/fullsend` at the mint layer trades pinning rigor for simpler rollout; caller-level SHA pinning per [ADR 0031](0031-reusable-workflows-for-action-installed-distribution.md) remains the recommended org/repo control. @@ -126,7 +126,7 @@ Introduce a **public mint mode** when `ALLOWED_ORGS` contains `*`, with `job_wor |-------|-------------| | Custom-agent reusable workflow | Future ADR | | Prioritize mint path under public mode | Resolve when non-event workflows are wired to reusables ([ADR 0041](0041-synchronous-workflow-call-event-dispatch.md)) | -| Mint WIF providers, IAM, deployment, abuse/WAF | [ADR 0063](0063-public-community-mint-architecture.md) | -| Hosted mint public-mode enrollment | [ADR 0063](0063-public-community-mint-architecture.md) + §1 here | +| Mint WIF providers, IAM, deployment, abuse/WAF | Future ADR (all provisioning) | +| Hosted mint public-mode enrollment | Resolve when moving hosted mint to `ALLOWED_ORGS=*` | | Normative mint contract | Deferred until multiple implementations; not needed now | | Stricter `@ref` allowlist on upstream workflows | Optional later hardening | diff --git a/docs/ADRs/0063-public-community-mint-architecture.md b/docs/ADRs/0063-public-community-mint-architecture.md index be770bb733..77f315eaa5 100644 --- a/docs/ADRs/0063-public-community-mint-architecture.md +++ b/docs/ADRs/0063-public-community-mint-architecture.md @@ -91,8 +91,8 @@ The hosted public mint **will use public mint mode** per [ADR 0059](0059-public- 2. **OIDC validation:** JWKS signature verification (steady state) or STS exchange (interim), then the **same** `mintcore` claim checks as today—including `iss`, `aud`, org allowlist, and workflow provenance per [ADR 0059](0059-public-mint-mode-with-wildcard-allowlists.md). Prove STS ≡ JWKS in CI before cutover. 3. **Edge:** **Managed WAF** and **rate limits** on `POST /v1/token` in the same Cloudflare surface as the Worker. 4. **No auth proxy** in front of callers; Bearer OIDC only. -5. **Secrets:** Shared community App PEMs **only** at the mint operator boundary. -6. **PEM rotation:** **Automated** rotation is **necessary** but **deferred** to a **future ADR**; until then, rotation is manual or GitOps-assisted. +5. **Secrets:** Shared community App PEMs **only** at the mint operator boundary. Steady-state Workers deployment stores each role PEM as a **Worker secret** (5 KB per secret). Current shared App PEMs are **~1,675 bytes** each—well within that limit—so secrets are stored directly in Workers for now. `mintcore` logs a warning when a PEM exceeds the Workers limit at load time (filesystem or GCP Secret Manager). Larger PEMs require an external vault or a follow-on ADR. +6. **PEM rotation:** **Automated** rotation is **necessary** but **deferred**; track design and implementation in [#4175](https://github.com/fullsend-ai/fullsend/issues/4175). Until then, rotation is manual or GitOps-assisted and must occur before GA or after **N** community adopters (threshold TBD in the tracking issue). 7. **Blast radius:** One compromised public mint affects **all orgs** on the profile; mitigate with GitOps-only changes, monitoring, timely PEM rotation, narrow App installations, and forge branch protections. ### Monitoring @@ -112,7 +112,7 @@ The hosted public mint **will use public mint mode** per [ADR 0059](0059-public- ### Operations 1. **Change control:** GitOps-only production changes ([#1263](https://github.com/fullsend-ai/fullsend/issues/1263)). -2. **Incidents:** Rotate shared App keys (manual until automated rotation ADR), tighten allowlists if needed, update enrollment guidance. +2. **Incidents:** Rotate shared App keys (manual until automated rotation is implemented), tighten allowlists if needed, update enrollment guidance. 3. **Evolution:** Hosting comparisons and interim GCP details stay in the [hosting spike](../spikes/2026-05-25-community-mint-hosting-gcp-vs-cloudflare.md). ## Consequences @@ -123,4 +123,4 @@ The hosted public mint **will use public mint mode** per [ADR 0059](0059-public- - Bootstrap owns **SLOs and incidents** until community ops exists. - **~$0 budget** keeps launch on GCP free tiers; LB+Armor (option F) and paid CF edge (option D long term) stay off the critical path unless funding appears. - Remaining work: shared Apps ([#914](https://github.com/fullsend-ai/fullsend/issues/914)), GitOps layout ([#1263](https://github.com/fullsend-ai/fullsend/issues/1263)), JWKS parity CI, public-mode implementation in `mintcore`, SLO handoff criteria. -- **Automated PEM rotation** must be specified in a **future ADR**. +- **Automated PEM rotation** tracked in [#4175](https://github.com/fullsend-ai/fullsend/issues/4175) (future ADR or implementation plan). diff --git a/internal/dispatch/gcf/mintsrc/mintcore/file_pem.go.embed b/internal/dispatch/gcf/mintsrc/mintcore/file_pem.go.embed index 577679cf1f..18287fb323 100644 --- a/internal/dispatch/gcf/mintsrc/mintcore/file_pem.go.embed +++ b/internal/dispatch/gcf/mintsrc/mintcore/file_pem.go.embed @@ -36,5 +36,6 @@ func (f *FilesystemPEMAccessor) AccessPEM(_ context.Context, role string) ([]byt if err != nil { return nil, fmt.Errorf("reading PEM for role %q: %w", role, err) } + WarnWorkersPEMSize(secretRole, data) return data, nil } diff --git a/internal/dispatch/gcf/mintsrc/mintcore/gcp_pem.go.embed b/internal/dispatch/gcf/mintsrc/mintcore/gcp_pem.go.embed index cd7796272c..864d25a54e 100644 --- a/internal/dispatch/gcf/mintsrc/mintcore/gcp_pem.go.embed +++ b/internal/dispatch/gcf/mintsrc/mintcore/gcp_pem.go.embed @@ -77,6 +77,7 @@ func (s *GCPSecretPEMAccessor) AccessPEM(ctx context.Context, role string) ([]by if err != nil { return nil, fmt.Errorf("decoding secret data: %w", err) } + WarnWorkersPEMSize(secretRole, data) return data, nil } diff --git a/internal/dispatch/gcf/mintsrc/mintcore/pem_size.go.embed b/internal/dispatch/gcf/mintsrc/mintcore/pem_size.go.embed new file mode 100644 index 0000000000..22b14cd3ca --- /dev/null +++ b/internal/dispatch/gcf/mintsrc/mintcore/pem_size.go.embed @@ -0,0 +1,59 @@ +package mintcore + +import ( + "fmt" + "log" + "os" + "path/filepath" + "strings" + "sync" +) + +// CloudflareWorkersSecretMaxBytes is the per-secret size limit for Cloudflare +// Workers (see Cloudflare Workers secrets documentation). Community mint +// steady-state (ADR 0063) stores shared App PEMs as Worker secrets. +const CloudflareWorkersSecretMaxBytes = 5120 + +var workersPEMWarnOnce sync.Map + +// ValidatePEMSize returns an error when pem exceeds maxBytes. +func ValidatePEMSize(pem []byte, maxBytes int) error { + if len(pem) > maxBytes { + return fmt.Errorf("PEM size %d bytes exceeds limit %d", len(pem), maxBytes) + } + return nil +} + +// WarnWorkersPEMSize logs once per role when a PEM exceeds the Workers secret +// limit. GCF and standalone mint deployments may still use larger PEMs; Workers +// steady-state cannot. +func WarnWorkersPEMSize(role string, pem []byte) { + if len(pem) <= CloudflareWorkersSecretMaxBytes { + return + } + if _, loaded := workersPEMWarnOnce.LoadOrStore(role, struct{}{}); loaded { + return + } + log.Printf("warning: PEM for role %q is %d bytes, exceeding Cloudflare Workers secret limit (%d bytes); hosted Worker deployment will not accept this secret", + role, len(pem), CloudflareWorkersSecretMaxBytes) +} + +// WarnAllPEMsInDir scans *.pem files in pemDir and logs Workers size warnings. +func WarnAllPEMsInDir(pemDir string) error { + entries, err := os.ReadDir(pemDir) + if err != nil { + return fmt.Errorf("reading PEM directory %q: %w", pemDir, err) + } + for _, entry := range entries { + if entry.IsDir() || !strings.HasSuffix(entry.Name(), ".pem") { + continue + } + role := strings.TrimSuffix(entry.Name(), ".pem") + data, err := os.ReadFile(filepath.Join(pemDir, entry.Name())) + if err != nil { + continue + } + WarnWorkersPEMSize(role, data) + } + return nil +} diff --git a/internal/mintcore/file_pem.go b/internal/mintcore/file_pem.go index 577679cf1f..18287fb323 100644 --- a/internal/mintcore/file_pem.go +++ b/internal/mintcore/file_pem.go @@ -36,5 +36,6 @@ func (f *FilesystemPEMAccessor) AccessPEM(_ context.Context, role string) ([]byt if err != nil { return nil, fmt.Errorf("reading PEM for role %q: %w", role, err) } + WarnWorkersPEMSize(secretRole, data) return data, nil } diff --git a/internal/mintcore/gcp_pem.go b/internal/mintcore/gcp_pem.go index cd7796272c..864d25a54e 100644 --- a/internal/mintcore/gcp_pem.go +++ b/internal/mintcore/gcp_pem.go @@ -77,6 +77,7 @@ func (s *GCPSecretPEMAccessor) AccessPEM(ctx context.Context, role string) ([]by if err != nil { return nil, fmt.Errorf("decoding secret data: %w", err) } + WarnWorkersPEMSize(secretRole, data) return data, nil } diff --git a/internal/mintcore/pem_size.go b/internal/mintcore/pem_size.go new file mode 100644 index 0000000000..22b14cd3ca --- /dev/null +++ b/internal/mintcore/pem_size.go @@ -0,0 +1,59 @@ +package mintcore + +import ( + "fmt" + "log" + "os" + "path/filepath" + "strings" + "sync" +) + +// CloudflareWorkersSecretMaxBytes is the per-secret size limit for Cloudflare +// Workers (see Cloudflare Workers secrets documentation). Community mint +// steady-state (ADR 0063) stores shared App PEMs as Worker secrets. +const CloudflareWorkersSecretMaxBytes = 5120 + +var workersPEMWarnOnce sync.Map + +// ValidatePEMSize returns an error when pem exceeds maxBytes. +func ValidatePEMSize(pem []byte, maxBytes int) error { + if len(pem) > maxBytes { + return fmt.Errorf("PEM size %d bytes exceeds limit %d", len(pem), maxBytes) + } + return nil +} + +// WarnWorkersPEMSize logs once per role when a PEM exceeds the Workers secret +// limit. GCF and standalone mint deployments may still use larger PEMs; Workers +// steady-state cannot. +func WarnWorkersPEMSize(role string, pem []byte) { + if len(pem) <= CloudflareWorkersSecretMaxBytes { + return + } + if _, loaded := workersPEMWarnOnce.LoadOrStore(role, struct{}{}); loaded { + return + } + log.Printf("warning: PEM for role %q is %d bytes, exceeding Cloudflare Workers secret limit (%d bytes); hosted Worker deployment will not accept this secret", + role, len(pem), CloudflareWorkersSecretMaxBytes) +} + +// WarnAllPEMsInDir scans *.pem files in pemDir and logs Workers size warnings. +func WarnAllPEMsInDir(pemDir string) error { + entries, err := os.ReadDir(pemDir) + if err != nil { + return fmt.Errorf("reading PEM directory %q: %w", pemDir, err) + } + for _, entry := range entries { + if entry.IsDir() || !strings.HasSuffix(entry.Name(), ".pem") { + continue + } + role := strings.TrimSuffix(entry.Name(), ".pem") + data, err := os.ReadFile(filepath.Join(pemDir, entry.Name())) + if err != nil { + continue + } + WarnWorkersPEMSize(role, data) + } + return nil +} diff --git a/internal/mintcore/pem_size_test.go b/internal/mintcore/pem_size_test.go new file mode 100644 index 0000000000..dc6c5c8494 --- /dev/null +++ b/internal/mintcore/pem_size_test.go @@ -0,0 +1,61 @@ +package mintcore + +import ( + "bytes" + "log" + "os" + "path/filepath" + "strings" + "testing" +) + +func TestValidatePEMSize(t *testing.T) { + t.Parallel() + if err := ValidatePEMSize(make([]byte, 5120), CloudflareWorkersSecretMaxBytes); err != nil { + t.Fatalf("at limit: %v", err) + } + if err := ValidatePEMSize(make([]byte, 5121), CloudflareWorkersSecretMaxBytes); err == nil { + t.Fatal("expected error above limit") + } +} + +func TestWarnWorkersPEMSizeLogsOnce(t *testing.T) { + role := "oversized-test-role" + workersPEMWarnOnce.Delete(role) + + var buf bytes.Buffer + old := log.Writer() + log.SetOutput(&buf) + t.Cleanup(func() { log.SetOutput(old) }) + + oversized := make([]byte, CloudflareWorkersSecretMaxBytes+1) + WarnWorkersPEMSize(role, oversized) + WarnWorkersPEMSize(role, oversized) + + if !strings.Contains(buf.String(), role) { + t.Fatalf("expected warning log, got: %q", buf.String()) + } + if strings.Count(buf.String(), "warning:") != 1 { + t.Fatalf("expected one warning, got: %q", buf.String()) + } +} + +func TestWarnAllPEMsInDir(t *testing.T) { + t.Parallel() + dir := t.TempDir() + if err := os.WriteFile(filepath.Join(dir, "coder.pem"), make([]byte, CloudflareWorkersSecretMaxBytes+10), 0o600); err != nil { + t.Fatal(err) + } + + var buf bytes.Buffer + old := log.Writer() + log.SetOutput(&buf) + t.Cleanup(func() { log.SetOutput(old) }) + + if err := WarnAllPEMsInDir(dir); err != nil { + t.Fatal(err) + } + if !strings.Contains(buf.String(), "coder") { + t.Fatalf("expected warning for oversized PEM, got: %q", buf.String()) + } +} From 2e3d9cf698d9b43740dc20f48b1083407689b8d9 Mon Sep 17 00:00:00 2001 From: Barak Korren Date: Mon, 13 Jul 2026 10:46:44 +0300 Subject: [PATCH 8/8] docs(adr): drop PEM size code; keep ADR documentation only Revert mintcore/cmd/mint PEM size warnings added in ccdb32a0. Workers 5 KB limit and current ~1,675-byte PEMs stay documented in ADR 0068. Signed-off-by: Barak Korren Co-authored-by: Cursor --- cmd/mint/main.go | 3 - ...0068-public-community-mint-architecture.md | 2 +- .../gcf/mintsrc/mintcore/file_pem.go.embed | 1 - .../gcf/mintsrc/mintcore/gcp_pem.go.embed | 1 - .../gcf/mintsrc/mintcore/pem_size.go.embed | 59 ------------------ internal/mintcore/file_pem.go | 1 - internal/mintcore/gcp_pem.go | 1 - internal/mintcore/pem_size.go | 59 ------------------ internal/mintcore/pem_size_test.go | 61 ------------------- 9 files changed, 1 insertion(+), 187 deletions(-) delete mode 100644 internal/dispatch/gcf/mintsrc/mintcore/pem_size.go.embed delete mode 100644 internal/mintcore/pem_size.go delete mode 100644 internal/mintcore/pem_size_test.go diff --git a/cmd/mint/main.go b/cmd/mint/main.go index 92d3929e11..674fe15406 100644 --- a/cmd/mint/main.go +++ b/cmd/mint/main.go @@ -50,9 +50,6 @@ func buildHandler() (http.Handler, error) { if err != nil { return nil, fmt.Errorf("initializing PEM accessor: %w", err) } - if err := mintcore.WarnAllPEMsInDir(os.Getenv("PEM_DIR")); err != nil { - log.Printf("warning: scanning PEM directory: %v", err) - } handler, err := mintcore.NewHandler(pemAccessor, verifier) if err != nil { diff --git a/docs/ADRs/0068-public-community-mint-architecture.md b/docs/ADRs/0068-public-community-mint-architecture.md index 265db4721e..004626aef1 100644 --- a/docs/ADRs/0068-public-community-mint-architecture.md +++ b/docs/ADRs/0068-public-community-mint-architecture.md @@ -91,7 +91,7 @@ The hosted public mint **will use public mint mode** per [ADR 0059](0059-public- 2. **OIDC validation:** JWKS signature verification (steady state) or STS exchange (interim), then the **same** `mintcore` claim checks as today—including `iss`, `aud`, org allowlist, and workflow provenance per [ADR 0059](0059-public-mint-mode-with-wildcard-allowlists.md). Prove STS ≡ JWKS in CI before cutover. 3. **Edge:** **Managed WAF** and **rate limits** on `POST /v1/token` in the same Cloudflare surface as the Worker. 4. **No auth proxy** in front of callers; Bearer OIDC only. -5. **Secrets:** Shared community App PEMs **only** at the mint operator boundary. Steady-state Workers deployment stores each role PEM as a **Worker secret** (5 KB per secret). Current shared App PEMs are **~1,675 bytes** each—well within that limit—so secrets are stored directly in Workers for now. `mintcore` logs a warning when a PEM exceeds the Workers limit at load time (filesystem or GCP Secret Manager). Larger PEMs require an external vault or a follow-on ADR. +5. **Secrets:** Shared community App PEMs **only** at the mint operator boundary. Steady-state Workers deployment stores each role PEM as a **Worker secret** (5 KB per secret). Current shared App PEMs are **~1,675 bytes** each—well within that limit—so secrets are stored directly in Workers for now. Operators must validate PEM size before deploying to Workers; larger PEMs require an external vault or a follow-on ADR. 6. **PEM rotation:** **Automated** rotation is **necessary** but **deferred**; track design and implementation in [#4175](https://github.com/fullsend-ai/fullsend/issues/4175). Until then, rotation is manual or GitOps-assisted and must occur before GA or after **N** community adopters (threshold TBD in the tracking issue). 7. **Blast radius:** One compromised public mint affects **all orgs** on the profile; mitigate with GitOps-only changes, monitoring, timely PEM rotation, narrow App installations, and forge branch protections. diff --git a/internal/dispatch/gcf/mintsrc/mintcore/file_pem.go.embed b/internal/dispatch/gcf/mintsrc/mintcore/file_pem.go.embed index 18287fb323..577679cf1f 100644 --- a/internal/dispatch/gcf/mintsrc/mintcore/file_pem.go.embed +++ b/internal/dispatch/gcf/mintsrc/mintcore/file_pem.go.embed @@ -36,6 +36,5 @@ func (f *FilesystemPEMAccessor) AccessPEM(_ context.Context, role string) ([]byt if err != nil { return nil, fmt.Errorf("reading PEM for role %q: %w", role, err) } - WarnWorkersPEMSize(secretRole, data) return data, nil } diff --git a/internal/dispatch/gcf/mintsrc/mintcore/gcp_pem.go.embed b/internal/dispatch/gcf/mintsrc/mintcore/gcp_pem.go.embed index 864d25a54e..cd7796272c 100644 --- a/internal/dispatch/gcf/mintsrc/mintcore/gcp_pem.go.embed +++ b/internal/dispatch/gcf/mintsrc/mintcore/gcp_pem.go.embed @@ -77,7 +77,6 @@ func (s *GCPSecretPEMAccessor) AccessPEM(ctx context.Context, role string) ([]by if err != nil { return nil, fmt.Errorf("decoding secret data: %w", err) } - WarnWorkersPEMSize(secretRole, data) return data, nil } diff --git a/internal/dispatch/gcf/mintsrc/mintcore/pem_size.go.embed b/internal/dispatch/gcf/mintsrc/mintcore/pem_size.go.embed deleted file mode 100644 index 9e13afddfd..0000000000 --- a/internal/dispatch/gcf/mintsrc/mintcore/pem_size.go.embed +++ /dev/null @@ -1,59 +0,0 @@ -package mintcore - -import ( - "fmt" - "log" - "os" - "path/filepath" - "strings" - "sync" -) - -// CloudflareWorkersSecretMaxBytes is the per-secret size limit for Cloudflare -// Workers (see Cloudflare Workers secrets documentation). Community mint -// steady-state (ADR 0068) stores shared App PEMs as Worker secrets. -const CloudflareWorkersSecretMaxBytes = 5120 - -var workersPEMWarnOnce sync.Map - -// ValidatePEMSize returns an error when pem exceeds maxBytes. -func ValidatePEMSize(pem []byte, maxBytes int) error { - if len(pem) > maxBytes { - return fmt.Errorf("PEM size %d bytes exceeds limit %d", len(pem), maxBytes) - } - return nil -} - -// WarnWorkersPEMSize logs once per role when a PEM exceeds the Workers secret -// limit. GCF and standalone mint deployments may still use larger PEMs; Workers -// steady-state cannot. -func WarnWorkersPEMSize(role string, pem []byte) { - if len(pem) <= CloudflareWorkersSecretMaxBytes { - return - } - if _, loaded := workersPEMWarnOnce.LoadOrStore(role, struct{}{}); loaded { - return - } - log.Printf("warning: PEM for role %q is %d bytes, exceeding Cloudflare Workers secret limit (%d bytes); hosted Worker deployment will not accept this secret", - role, len(pem), CloudflareWorkersSecretMaxBytes) -} - -// WarnAllPEMsInDir scans *.pem files in pemDir and logs Workers size warnings. -func WarnAllPEMsInDir(pemDir string) error { - entries, err := os.ReadDir(pemDir) - if err != nil { - return fmt.Errorf("reading PEM directory %q: %w", pemDir, err) - } - for _, entry := range entries { - if entry.IsDir() || !strings.HasSuffix(entry.Name(), ".pem") { - continue - } - role := strings.TrimSuffix(entry.Name(), ".pem") - data, err := os.ReadFile(filepath.Join(pemDir, entry.Name())) - if err != nil { - continue - } - WarnWorkersPEMSize(role, data) - } - return nil -} diff --git a/internal/mintcore/file_pem.go b/internal/mintcore/file_pem.go index 18287fb323..577679cf1f 100644 --- a/internal/mintcore/file_pem.go +++ b/internal/mintcore/file_pem.go @@ -36,6 +36,5 @@ func (f *FilesystemPEMAccessor) AccessPEM(_ context.Context, role string) ([]byt if err != nil { return nil, fmt.Errorf("reading PEM for role %q: %w", role, err) } - WarnWorkersPEMSize(secretRole, data) return data, nil } diff --git a/internal/mintcore/gcp_pem.go b/internal/mintcore/gcp_pem.go index 864d25a54e..cd7796272c 100644 --- a/internal/mintcore/gcp_pem.go +++ b/internal/mintcore/gcp_pem.go @@ -77,7 +77,6 @@ func (s *GCPSecretPEMAccessor) AccessPEM(ctx context.Context, role string) ([]by if err != nil { return nil, fmt.Errorf("decoding secret data: %w", err) } - WarnWorkersPEMSize(secretRole, data) return data, nil } diff --git a/internal/mintcore/pem_size.go b/internal/mintcore/pem_size.go deleted file mode 100644 index 9e13afddfd..0000000000 --- a/internal/mintcore/pem_size.go +++ /dev/null @@ -1,59 +0,0 @@ -package mintcore - -import ( - "fmt" - "log" - "os" - "path/filepath" - "strings" - "sync" -) - -// CloudflareWorkersSecretMaxBytes is the per-secret size limit for Cloudflare -// Workers (see Cloudflare Workers secrets documentation). Community mint -// steady-state (ADR 0068) stores shared App PEMs as Worker secrets. -const CloudflareWorkersSecretMaxBytes = 5120 - -var workersPEMWarnOnce sync.Map - -// ValidatePEMSize returns an error when pem exceeds maxBytes. -func ValidatePEMSize(pem []byte, maxBytes int) error { - if len(pem) > maxBytes { - return fmt.Errorf("PEM size %d bytes exceeds limit %d", len(pem), maxBytes) - } - return nil -} - -// WarnWorkersPEMSize logs once per role when a PEM exceeds the Workers secret -// limit. GCF and standalone mint deployments may still use larger PEMs; Workers -// steady-state cannot. -func WarnWorkersPEMSize(role string, pem []byte) { - if len(pem) <= CloudflareWorkersSecretMaxBytes { - return - } - if _, loaded := workersPEMWarnOnce.LoadOrStore(role, struct{}{}); loaded { - return - } - log.Printf("warning: PEM for role %q is %d bytes, exceeding Cloudflare Workers secret limit (%d bytes); hosted Worker deployment will not accept this secret", - role, len(pem), CloudflareWorkersSecretMaxBytes) -} - -// WarnAllPEMsInDir scans *.pem files in pemDir and logs Workers size warnings. -func WarnAllPEMsInDir(pemDir string) error { - entries, err := os.ReadDir(pemDir) - if err != nil { - return fmt.Errorf("reading PEM directory %q: %w", pemDir, err) - } - for _, entry := range entries { - if entry.IsDir() || !strings.HasSuffix(entry.Name(), ".pem") { - continue - } - role := strings.TrimSuffix(entry.Name(), ".pem") - data, err := os.ReadFile(filepath.Join(pemDir, entry.Name())) - if err != nil { - continue - } - WarnWorkersPEMSize(role, data) - } - return nil -} diff --git a/internal/mintcore/pem_size_test.go b/internal/mintcore/pem_size_test.go deleted file mode 100644 index dc6c5c8494..0000000000 --- a/internal/mintcore/pem_size_test.go +++ /dev/null @@ -1,61 +0,0 @@ -package mintcore - -import ( - "bytes" - "log" - "os" - "path/filepath" - "strings" - "testing" -) - -func TestValidatePEMSize(t *testing.T) { - t.Parallel() - if err := ValidatePEMSize(make([]byte, 5120), CloudflareWorkersSecretMaxBytes); err != nil { - t.Fatalf("at limit: %v", err) - } - if err := ValidatePEMSize(make([]byte, 5121), CloudflareWorkersSecretMaxBytes); err == nil { - t.Fatal("expected error above limit") - } -} - -func TestWarnWorkersPEMSizeLogsOnce(t *testing.T) { - role := "oversized-test-role" - workersPEMWarnOnce.Delete(role) - - var buf bytes.Buffer - old := log.Writer() - log.SetOutput(&buf) - t.Cleanup(func() { log.SetOutput(old) }) - - oversized := make([]byte, CloudflareWorkersSecretMaxBytes+1) - WarnWorkersPEMSize(role, oversized) - WarnWorkersPEMSize(role, oversized) - - if !strings.Contains(buf.String(), role) { - t.Fatalf("expected warning log, got: %q", buf.String()) - } - if strings.Count(buf.String(), "warning:") != 1 { - t.Fatalf("expected one warning, got: %q", buf.String()) - } -} - -func TestWarnAllPEMsInDir(t *testing.T) { - t.Parallel() - dir := t.TempDir() - if err := os.WriteFile(filepath.Join(dir, "coder.pem"), make([]byte, CloudflareWorkersSecretMaxBytes+10), 0o600); err != nil { - t.Fatal(err) - } - - var buf bytes.Buffer - old := log.Writer() - log.SetOutput(&buf) - t.Cleanup(func() { log.SetOutput(old) }) - - if err := WarnAllPEMsInDir(dir); err != nil { - t.Fatal(err) - } - if !strings.Contains(buf.String(), "coder") { - t.Fatalf("expected warning for oversized PEM, got: %q", buf.String()) - } -}