diff --git a/docs/ADRs/0068-public-community-mint-architecture.md b/docs/ADRs/0068-public-community-mint-architecture.md new file mode 100644 index 0000000000..004626aef1 --- /dev/null +++ b/docs/ADRs/0068-public-community-mint-architecture.md @@ -0,0 +1,126 @@ +--- +title: "68. Public community mint architecture" +status: Accepted +relates_to: + - agent-infrastructure + - security-threat-model + - operational-observability +topics: + - identity + - oidc + - github-apps + - deployment + - operations + - key-rotation +--- + +# 68. Public community mint architecture + +Date: 2026-05-25 + +## Status + +Accepted + + + +## Context + +[ADR 0029](0029-central-token-mint-secretless-fullsend.md) establishes the **goal**: a **community deployment profile** with public (unlisted) shared GitHub Apps per role, App keys held **only** at a centrally operated mint, and routine adopters trusting a stable **`FULLSEND_MINT_URL`** instead of bespoke per-org Apps and dispatch PATs. + +Since this ADR was drafted, related decisions landed on `main`: + +- **[ADR 0059](0059-public-mint-mode-with-wildcard-allowlists.md)** (Accepted) defines **public mint trust policy**: `ALLOWED_ORGS=*`, upstream-only `job_workflow_ref` under `fullsend-ai/fullsend/.github/workflows/`, global per-role `ROLE_APP_IDS` and PEM secrets, and enrollment via **installing the shared Apps** (no per-org mint env churn). Custom per-repo workflow provenance is **tight-mode only** via `PER_REPO_WIF_REPOS`. +- **[ADR 0060](0060-cross-org-mint-authorization-via-org-variables.md)** (Accepted) adds optional `target_org` minting for workloads like the e2e pool ([ADR 0040](0040-org-pool-for-parallel-e2e-tests.md)). +- **[ADR 0044](0044-deprecate-per-org-installation-mode.md)** (Accepted) deprecates per-org `.fullsend` installs; the public profile targets **per-repo** installs calling upstream reusables ([ADR 0033](0033-per-repo-installation-mode.md), [ADR 0031](0031-reusable-workflows-for-action-installed-distribution.md)). +- Mint logic now lives in **`internal/mintcore/`** (shared by GCF `internal/mint/` and standalone **`cmd/mint/`**, which already uses **JWKS** verification). + +This ADR records **how** the community-operated public mint is **deployed, secured at the edge, monitored, scaled, and run**. **OIDC claim rules and enrollment policy** are normative in [ADR 0059](0059-public-mint-mode-with-wildcard-allowlists.md); this ADR fulfills the **mint infrastructure** item deferred there (WIF/provider layout, deployment, WAF, monitoring). + +Platform and phasing choices (interim GCP vs steady-state Workers, cost, operations consoles) are analyzed in the [hosting spike](../spikes/2026-05-25-community-mint-hosting-gcp-vs-cloudflare.md) ([#915](https://github.com/fullsend-ai/fullsend/issues/915)). [#1145](https://github.com/fullsend-ai/fullsend/issues/1145) depends on this architecture for zero-GCP installs against the hosted mint. + +## Options + +Ways to **achieve** the ADR 0029 community mint (same `POST /v1/token` contract, opaque URL to consumers): + +| Option | Summary | +|--------|---------| +| **A. Dedicated community mint on GCP (interim)** | Go Cloud Function (`internal/mint/` + `mintcore`) in a **mint-only** GCP project; OIDC via **STS + WIF**; PEMs in Secret Manager; prod deploy via **GitOps** ([#1263](https://github.com/fullsend-ai/fullsend/issues/1263)). | +| **B. Reuse the internal Red Hat mint** for community adopters | Single mint endpoint and project for internal and public tenants. | +| **C. Tenant-style CLI provisioner** for the public mint | Same imperative `fullsend admin install` / GCF path self-managed orgs use. | +| **D. GCP origin + Cloudflare edge (steady state)** | Keep GCF; public hostname proxied for WAF/rate limits long term. | +| **E. Cloudflare Workers (steady state)** | Port mint to Workers; OIDC via **JWKS** (`mintcore.JWKSVerifier`); edge and compute in one ops surface. | +| **F. GCP + Cloud Armor + external HTTPS LB** | Harden edge entirely in GCP without Cloudflare. | + +**B** is rejected: shared infrastructure with internal workloads breaks **isolation** and community **trust boundaries**. **C** is rejected for production: no enforced review or deploy audit ([#1263](https://github.com/fullsend-ai/fullsend/issues/1263)). **D** is rejected as the **long-term** default (dual dashboards, poor fit for ~$0 budget on meaningful WAF—see spike); acceptable only as a **short bridge**. **F** is rejected on ~$0 community budget (LB baseline cost). + +**Chosen:** **E** defines the steady-state architecture below. **A** is an acceptable **interim** implementation until **E** is ready; **D** only as a short bridge if edge is urgent before **E** (spike). + +## Decision + +Fullsend **will operate** a **public community mint** as required by [ADR 0029](0029-central-token-mint-secretless-fullsend.md). The **steady-state** design (option **E**) is a **stateless, internet-facing** mint on **Cloudflare Workers**, exposing the existing `POST /v1/token` contract ([mint-token action](../../.github/actions/mint-token/action.yml), [infrastructure reference](../guides/infrastructure/infrastructure-reference.md)). Adopters set **`FULLSEND_MINT_URL`** and OIDC audience only; hosting is opaque. + +Until the Worker implementation is production-ready, the same contract and trust bar may run temporarily on **GCP Cloud Function** (option **A**, STS + WIF). **Self-managed tenant** mints stay on separate paths (CLI/GCF or `cmd/mint/`); they are not described here. + +### Trust and enrollment (hosted profile) + +The hosted public mint **will use public mint mode** per [ADR 0059](0059-public-mint-mode-with-wildcard-allowlists.md): + +- **`ALLOWED_ORGS=*`** — any org may request tokens after other checks pass; **installing the shared role Apps is enrollment** ([#914](https://github.com/fullsend-ai/fullsend/issues/914), [#1145](https://github.com/fullsend-ai/fullsend/issues/1145)). No `EnsureOrgInMint` / per-org `ALLOWED_ORGS` updates for new adopters. +- **`job_workflow_ref`** — **upstream reusables only** (`fullsend-ai/fullsend/.github/workflows/`). Legacy `{org}/.fullsend/` and custom `{owner}/{repo}/` workflow paths are **not** supported on the public profile ([ADR 0044](0044-deprecate-per-org-installation-mode.md)). +- **`PER_REPO_WIF_REPOS`** — **unset/empty** on the hosted mint. Per-repo custom workflow provenance remains a **tight-mode** feature for self-managed mints only. +- **Shared credentials** — `ROLE_APP_IDS` and PEM secrets are **global per role** (`fullsend-{role}-app-pem`), not keyed by org ([ADR 0059](0059-public-mint-mode-with-wildcard-allowlists.md) §8). Cross-org isolation uses `repository_owner` + installation lookup, not separate PEMs per org. +- **`aud` validation** — enforced in **`mintcore`** application code (`OIDC_AUDIENCE`) on both STS and JWKS paths; it is **not** a WIF/STS responsibility and carries over unchanged on Workers. +- **Abuse complement** — [ADR 0054](0054-require-authorization-on-all-agent-dispatch-paths.md) dispatch authorization limits who can trigger agent runs; mint openness does not bypass write checks at dispatch. +- **Cross-org** — [ADR 0060](0060-cross-org-mint-authorization-via-org-variables.md) applies on the same hosted endpoint (e.g. e2e pool). + +### Deployment + +1. **Runtime:** **`mintcore`** handler on **Cloudflare Workers** with **`JWKSVerifier`** and pluggable `PEMAccessor` (parity with `cmd/mint/` today). +2. **Interim (option A):** GCF wrapper in `internal/mint/` with **`STSVerifier`**; public-mode env per [ADR 0059](0059-public-mint-mode-with-wildcard-allowlists.md) §2 (permissive `WIF_PROVIDER_NAME`, empty `PER_REPO_WIF_REPOS`). +3. **Release:** Production deploys only through **GitOps** ([#1263](https://github.com/fullsend-ai/fullsend/issues/1263))—not the tenant CLI provisioner. +4. **Isolation:** Community mint **must not share** infrastructure with Vertex/inference, internal Red Hat mints, or unrelated Workers. PEMs and mint configuration live in a **dedicated** trust domain. +5. **Public URL:** Stable **`FULLSEND_MINT_URL`** on a community hostname; TLS and edge policy colocated with the Worker. + +### Security (edge and operations) + +1. **Trust model ([ADR 0029](0029-central-token-mint-secretless-fullsend.md)):** OIDC JWT in, **short-lived, org-scoped** installation token out; **role minimum permissions** in mint logic. +2. **OIDC validation:** JWKS signature verification (steady state) or STS exchange (interim), then the **same** `mintcore` claim checks as today—including `iss`, `aud`, org allowlist, and workflow provenance per [ADR 0059](0059-public-mint-mode-with-wildcard-allowlists.md). Prove STS ≡ JWKS in CI before cutover. +3. **Edge:** **Managed WAF** and **rate limits** on `POST /v1/token` in the same Cloudflare surface as the Worker. +4. **No auth proxy** in front of callers; Bearer OIDC only. +5. **Secrets:** Shared community App PEMs **only** at the mint operator boundary. Steady-state Workers deployment stores each role PEM as a **Worker secret** (5 KB per secret). Current shared App PEMs are **~1,675 bytes** each—well within that limit—so secrets are stored directly in Workers for now. Operators must validate PEM size before deploying to Workers; larger PEMs require an external vault or a follow-on ADR. +6. **PEM rotation:** **Automated** rotation is **necessary** but **deferred**; track design and implementation in [#4175](https://github.com/fullsend-ai/fullsend/issues/4175). Until then, rotation is manual or GitOps-assisted and must occur before GA or after **N** community adopters (threshold TBD in the tracking issue). +7. **Blast radius:** One compromised public mint affects **all orgs** on the profile; mitigate with GitOps-only changes, monitoring, timely PEM rotation, narrow App installations, and forge branch protections. + +### Monitoring + +1. **Owner:** **Red Hat Fullsend Bootstrap** until community operations assumes on-call. +2. **SLOs:** **99.5%** monthly availability for `POST /v1/token` (excluding GitHub OIDC/API outages); **p95 < 2s** latency. +3. **Signals:** Worker errors and latency, WAF block/challenge rates, synthetic `POST /v1/token` without token (expect 401), GitOps/deploy audit trail ([#1262](https://github.com/fullsend-ai/fullsend/issues/1262)). +4. **Triage:** Single console (Cloudflare)—Worker health, then WAF, then external GitHub status. + +### Scaling + +1. **Shape:** Stateless request/response; low baseline QPS, bursty with Actions. +2. **Capacity:** Workers scale automatically; no mint-side session store. +3. **Limits:** Keep request/body/`repos` caps; tune **edge rate limits** as adoption grows. +4. **Cost:** Community budget **~$0** at expected volume ([spike](../spikes/2026-05-25-community-mint-hosting-gcp-vs-cloudflare.md)). + +### Operations + +1. **Change control:** GitOps-only production changes ([#1263](https://github.com/fullsend-ai/fullsend/issues/1263)). +2. **Incidents:** Rotate shared App keys (manual until automated rotation is implemented), tighten allowlists if needed, update enrollment guidance. +3. **Evolution:** Hosting comparisons and interim GCP details stay in the [hosting spike](../spikes/2026-05-25-community-mint-hosting-gcp-vs-cloudflare.md). + +## Consequences + +- Delivers the **ADR 0029 community profile** in operable form, with **trust policy** in [ADR 0059](0059-public-mint-mode-with-wildcard-allowlists.md) and **ops/deployment** here. +- **Launch (option A)** unblocks [#914](https://github.com/fullsend-ai/fullsend/issues/914) and [#1145](https://github.com/fullsend-ai/fullsend/issues/1145) without waiting for the Worker port. +- **Steady state (option E)** improves edge posture and **single-console** ops versus a permanent GCP+CF split (option D). +- Bootstrap owns **SLOs and incidents** until community ops exists. +- **~$0 budget** keeps launch on GCP free tiers; LB+Armor (option F) and paid CF edge (option D long term) stay off the critical path unless funding appears. +- Remaining work: shared Apps ([#914](https://github.com/fullsend-ai/fullsend/issues/914)), GitOps layout ([#1263](https://github.com/fullsend-ai/fullsend/issues/1263)), JWKS parity CI, public-mode implementation in `mintcore`, SLO handoff criteria. +- **Automated PEM rotation** tracked in [#4175](https://github.com/fullsend-ai/fullsend/issues/4175) (future ADR or implementation plan). diff --git a/docs/architecture.md b/docs/architecture.md index da49d6b730..6cf1d3ccec 100644 --- a/docs/architecture.md +++ b/docs/architecture.md @@ -158,6 +158,7 @@ Identity is not the same as trust. An agent's identity lets it authenticate to e - Per-role GitHub Apps with manifest-based creation. Each agent role gets its own app with scoped permissions. PEMs stored in Secret Manager as `fullsend-{role}-app-pem` — one secret per role, shared across orgs on a mint. `ROLE_APP_IDS` uses the same shared-per-role model (`coder` → app ID). Org isolation is enforced via `ALLOWED_ORGS`, WIF conditions, and installation verification ([ADR 0007](ADRs/0007-per-role-github-apps.md), [ADR 0033](ADRs/0033-per-repo-installation-mode.md)). Public multi-tenant mint (`ALLOWED_ORGS=*`) with upstream-only workflow provenance is defined in [ADR 0059](ADRs/0059-public-mint-mode-with-wildcard-allowlists.md); upstream-only provenance limits which workflows can call the mint, complementing [ADR 0029](ADRs/0029-central-token-mint-secretless-fullsend.md) multi-tenant blast-radius concerns. - Cross-org mint authorization: workflows may request tokens for a different org via optional `target_org` when the target org installs the role App and sets `FULLSEND_FOREIGN__REPOS`. Empty `repos` yields installation-wide tokens on either path; cross-org adds FOREIGN gating, same-org relies on WIF/OIDC enrollment ([ADR 0060](ADRs/0060-cross-org-mint-authorization-via-org-variables.md)). - Standalone mint deployment: `cmd/mint/` provides a self-contained HTTP server that uses direct JWKS verification and filesystem PEM storage instead of GCP infrastructure. It shares the `internal/mintcore/` library with the GCF mint and adds support for custom role permissions and a fallback proxy to an upstream mint. Custom role permissions live in mintcore (not `cmd/mint/`) so that `RolePermissionsFor`, `HasRole`, and `CreateInstallationToken` return a unified view without callers needing to distinguish built-in from custom roles. The GCF mint never calls `RegisterCustomRolePermissions`, so the code is inert there. See the [standalone mint guide](guides/infrastructure/standalone-mint.md). +- Hosted public community mint: steady-state deployment on Cloudflare Workers (JWKS + WAF + single ops console), with interim GCP Cloud Function acceptable until the Worker port is production-ready. Trust policy (`ALLOWED_ORGS=*`, upstream-only workflow provenance) is in [ADR 0059](ADRs/0059-public-mint-mode-with-wildcard-allowlists.md); deployment, edge security, monitoring, and phasing are in [ADR 0068](ADRs/0068-public-community-mint-architecture.md). Enrollment is installing the shared Apps—no per-org mint env registration ([#1145](https://github.com/fullsend-ai/fullsend/issues/1145)). One concrete implementation option is [`oidcx`](https://github.com/oxidecomputer/oidcx): a service that accepts OIDC identity tokens and exchanges them for short-lived access tokens. It can mint tokens scoped to selected GitHub repositories and permissions, or to selected Oxide silos and permissions, and it also ships with a GitHub Action wrapper. In a Fullsend deployment, this can be used by the sandbox entrypoint to narrow a broad GitHub App identity down to only the specific permissions an agent needs for the current run. diff --git a/docs/spikes/2026-05-25-community-mint-hosting-gcp-vs-cloudflare.md b/docs/spikes/2026-05-25-community-mint-hosting-gcp-vs-cloudflare.md new file mode 100644 index 0000000000..d4d45e7b30 --- /dev/null +++ b/docs/spikes/2026-05-25-community-mint-hosting-gcp-vs-cloudflare.md @@ -0,0 +1,120 @@ +# SPIKE: Community token mint hosting (GCP vs Cloudflare) + +**Issue:** [#915](https://github.com/fullsend-ai/fullsend/issues/915) · **Parent:** [#914](https://github.com/fullsend-ai/fullsend/issues/914) · **Epic:** [#912](https://github.com/fullsend-ai/fullsend/issues/912) +**ADR:** [0029](../ADRs/0029-central-token-mint-secretless-fullsend.md) · [0059](../ADRs/0059-public-mint-mode-with-wildcard-allowlists.md) (trust) · [0068](../ADRs/0068-public-community-mint-architecture.md) (ops/deployment) · **GitOps:** [#1263](https://github.com/fullsend-ai/fullsend/issues/1263) · **Date:** 2026-05-25 + +## Decision + +**Launch the public mint on Scenario 1** (100% GCP, dedicated project, [GitOps](#public-mint-operations) per [#1263](https://github.com/fullsend-ai/fullsend/issues/1263)), with **per-repo** installs calling upstream reusables ([ADR 0033](../ADRs/0033-per-repo-installation-mode.md), [ADR 0059](../ADRs/0059-public-mint-mode-with-wildcard-allowlists.md)). + +**Do not treat Scenario 2 (GCP + Cloudflare) as the long-term default.** Two consoles (GCP origin + Cloudflare edge) are ongoing human cost for the **Red Hat Fullsend Bootstrap** team (no dedicated SRE). That cost is **not** offset by saving a one-time port—re-platforming is largely AI-driven; **monitoring and incidents are human-driven**. + +**Steady-state target for a hardened public edge: Scenario 3** (100% Cloudflare Workers), not Scenario 2—**one operational surface** (Workers + WAF + alerts) while preserving the same `POST /v1/token` contract. Scenario 2 remains a **short bridge** only if WAF/rate limits are needed before the Worker port lands. + +Public mint consumers only use **`FULLSEND_MINT_URL`**; hosting is opaque. **Self-managed tenant** mints stay on GCP via the CLI provisioner or **`cmd/mint/`** today; Cloudflare becomes an optional tenant target when Scenario 3 exists. + +## Scenarios + +| # | Posture | Operator sees | +|---|---------|---------------| +| **1** | GCF + WIF + Secret Manager in a **mint-only** GCP project | **One** primary stack (Cloud Monitoring / Logging); edge via Cloud Armor + external HTTPS LB if required | +| **2** | Same GCF origin; public URL is **Cloudflare-proxied** | **Two** stacks: GCP (origin health, STS, SM) + Cloudflare (WAF, blocks, 5xx at edge) | +| **3** | Mint on **Workers**; OIDC via JWKS + same claim rules; PEMs in Worker secrets (or external vault) | **One** primary stack (Cloudflare); GitOps via Wrangler | + +Today's code: **`internal/mintcore/`** (shared library), GCF entrypoint (`internal/mint/`), standalone JWKS server (`cmd/mint/`), tenant deploy via `internal/dispatch/gcf`, contract in [mint-token](../../.github/actions/mint-token/action.yml) and [infrastructure reference](../guides/infrastructure/infrastructure-reference.md). + +### OIDC trust: STS/WIF (today) vs JWKS (Scenario 3) + +GitHub Actions sends the mint a short-lived **OIDC JWT** in the `Authorization` header. The mint must prove the token is genuine and matches policy (`job_workflow_ref`, allowed orgs/workflows, etc.). + +| Approach | Used in | What it means | +|----------|---------|----------------| +| **STS + WIF** | Scenario 1 (today) | The mint sends the JWT to **GCP Security Token Service**, which validates it against a **Workload Identity Federation** pool (CEL rules on repo/org). GCP returns a federated token; the mint also decodes and checks claims in Go. Trust is anchored in **GCP**. | +| **JWKS** | Scenario 3 (target) | **JWKS** = *JSON Web Key Set*: the public signing keys GitHub publishes (e.g. `https://token.actions.githubusercontent.com/.well-known/jwks`). The mint **verifies the JWT signature** against those keys locally (no GCP STS call), then runs the **same** claim checks in **`mintcore`** (no GCP STS call). Trust is anchored in **GitHub’s keys + mint logic**. | + +A Scenario 3 port is often called “STS→JWKS” because the **authorization outcome** should match; only the **validation backend** changes. + +**Public mint profile** ([ADR 0059](../ADRs/0059-public-mint-mode-with-wildcard-allowlists.md)): `ALLOWED_ORGS=*`, permissive default WIF provider, **empty** `PER_REPO_WIF_REPOS`, and **upstream-only** `job_workflow_ref`. Per-repo WIF provider routing is a **tight-mode** concern only. + +**`aud` (audience):** Validated in **`mintcore`** via `OIDC_AUDIENCE` on both STS and JWKS paths—not by WIF/STS. The JWKS port does not change audience rules. + +## Integrated evaluation + +All requirements below apply together—not as a separate “constraints” checklist. + +| Factor | 1 — GCP | 2 — GCP + CF | 3 — CF | +|--------|---------|--------------|--------| +| **Launch speed** | ● Shipping binary + GitOps | ● + DNS/WAF rules | ◐ Worker port + parity tests | +| **Public vs tenant deploy** | GitOps ([#1263](https://github.com/fullsend-ai/fullsend/issues/1263)); CLI for tenants | Same | GitOps (Wrangler); CLI for tenants later | +| **Isolated from inference/LLM** | ● Dedicated GCP project only | ● Same project; CF zone for mint host | ● No Vertex in mint project; don’t colocate inference Workers | +| **Public mint trust ([ADR 0059](../ADRs/0059-public-mint-mode-with-wildcard-allowlists.md))** | ● STS + permissive WIF + mintcore claims | ● Unchanged at origin | ◐ Prove JWKS path ≡ STS path in CI | +| **ADR 0029 trust** | ● STS + WIF + handler claims | ● Unchanged at origin | ◐ Prove JWKS path ≡ STS path in CI | +| **Internet-facing abuse** | ◐ Armor+LB setup (still one vendor) | ● Easy WAF/RL | ● WAF/RL in same console as compute | +| **Ongoing human ops** | ● **Single dashboard**; paging from GCP | ○ **Dual dashboard**; split incident triage | ● **Single dashboard**; paging from CF | +| **One-time engineering** | Low | Low–medium | Medium (port behind existing `mintcore` interfaces) | +| **Long-term multi-host** | ◐ GCP-centric | ◐ Split | ● Adds non-GCP option for tenants/public | +| **Blocks [#914](https://github.com/fullsend-ai/fullsend/issues/914)** | No | No | No if launch on 1 first | +| **Cost (~$0 community budget)** | ● Bare GCF+WIF+SM | ○ Free CF = weak edge; paid CF breaks budget | ● Workers Free at community volume | + +### Cost (~$0 community budget) + +Community mint traffic is **low** (roughly one `POST /v1/token` per agent job batch), so **compute** is not the cost driver—**fixed-price edge SKUs** are. + +| Posture | Typical spend at community scale | Fits ~$0? | +|---------|----------------------------------|-----------| +| **Scenario 1 — bare GCF + WIF + SM** | Stays within GCP free tiers for invocations, federation, and SM access at expected volume | **Yes** — best match for launch | +| **Scenario 1 — Cloud Armor + external HTTPS LB** | LB has **baseline monthly cost** even at zero mint traffic | **No** — rules out “hardened GCP edge” on a $0 budget | +| **Scenario 2 — GCF + Cloudflare** | GCF ~$0; **meaningful** WAF/rate limits usually need **paid** CF (Pro/Business), not Free | **Poor value on $0**: dual dashboards + thin Free-tier rules | +| **Scenario 3 — Workers** | Workers **Free** tier is sufficient at community QPS; **Workers Paid** only if volume or bundle limits grow later | **Yes** for steady state without GCP LB fees | + +**Effect on the decision (with ops and security):** + +- **Reinforces** launch on **Scenario 1** without Armor/LB—upstream-only provenance ([ADR 0059](../ADRs/0059-public-mint-mode-with-wildcard-allowlists.md)) and GCP alerts are the $0 edge story until Scenario 3 ships. +- **Discourages** a **long-lived Scenario 2** bridge unless a CF plan is donated; otherwise you pay in **operator time** (two consoles) without buying real protection. +- **Keeps Scenario 3** as the **$0 steady-state** path for WAF/rate limits **plus** one dashboard once the JWKS port is done. + +### Does two dashboards push toward 1 or 3? + +**Toward 1 or 3—not 2.** + +- **Scenario 2** optimizes **edge convenience** at the price of **permanent split-brain ops**: Bootstrap must correlate GCF 5xx/latency with Cloudflare origin errors and WAF blocks; synthetic checks should hit the public hostname *and* the origin; runbooks always have two hops. Alert fan-in to one pager helps but does not remove the second UI for tuning and incidents. +- **Scenario 1** keeps **one vendor console** for mint health. The tradeoff is weaker **default** edge on a bare `cloudfunctions.net` URL—mitigate at launch with fail-closed env allowlists per [ADR 0059](../ADRs/0059-public-mint-mode-with-wildcard-allowlists.md) and GCP alerts. **Cloud Armor + LB** is the one-dashboard GCP hardening option but **not** on a ~$0 budget (see [Cost](#cost-0-community-budget)); funded abuse response or **Scenario 3** is the realistic hardening path. +- **Scenario 3** is the way to get **strong edge + single dashboard** without Scenario 2’s ops tax. The STS→JWKS port is a **one-time** cost; Bootstrap’s recurring load is CF-only. Prefer this over staying on Scenario 2 indefinitely. + +**Scenario 2 is justified only as a time-boxed bridge** (weeks, not years): public hostname needs WAF before Workers ship, and Bootstrap accepts dual-console overhead temporarily. + +## Public mint operations + +- **Deploy:** GitOps ([#1263](https://github.com/fullsend-ai/fullsend/issues/1263))—no prod `gcloud functions deploy` by individuals; optional [#1262](https://github.com/fullsend-ai/fullsend/issues/1262) deploy notifications. +- **Project:** Community mint **only**—no shared GCP project with Vertex/inference or internal Red Hat mint resources. +- **SLO owner:** Red Hat Fullsend Bootstrap until community ops exists. Target: **99.5%** availability for `POST /v1/token` (excl. GitHub outages), **p95 < 2s**. +- **Signals (Scenario 1):** GCF 5xx/latency, STS/SM errors, synthetic `POST /v1/token` (expect 401), allowlist-change audit via GitOps. +- **Signals (Scenario 3):** Worker errors, WAF blocks, same synthetic on public URL—no origin correlation. +- **Portability:** `JWKSVerifier` in `mintcore` (used by `cmd/mint/`); parity tests STS vs JWKS—strategic, not launch-blocking. + +## Phasing + +| Phase | Choice | Rationale | +|-------|--------|-----------| +| **Launch** | **Scenario 1** + GitOps + isolated project | Fastest; one console; unblocks [#914](https://github.com/fullsend-ai/fullsend/issues/914) / [#1145](https://github.com/fullsend-ai/fullsend/issues/1145) | +| **Bridge (optional)** | **Scenario 2** | Only if abuse/WAF needed before Workers *and* CF budget exists; poor fit on $0 + dual console | +| **Hardened steady state** | **Scenario 3** | WAF + rate limits + **one** ops surface; avoids chronic dual-dashboard | +| **Parallel** | JWKS/port work | `cmd/mint/` proves JWKS path; does not block launch | +| **Tenants** | GCP CLI path or `cmd/mint/` now; CF when Scenario 3 matures | Tenants choose; public URL stays opaque | + +## Open follow-ups + +1. GitOps layout for mint + WIF/SM without PEMs in git ([#1263](https://github.com/fullsend-ai/fullsend/issues/1263)). +2. Allowlist-only edge until Scenario 3 (Armor+LB excluded on ~$0 budget unless funding appears). +3. Rate-limit thresholds for public hostname (when edge exists). +4. Criteria to transfer SLO ownership from Bootstrap to community ops. +5. Implement `ALLOWED_ORGS=*` and upstream-only workflow validation in `mintcore` per [ADR 0059](../ADRs/0059-public-mint-mode-with-wildcard-allowlists.md). + +## References + +- [Infrastructure reference — Token mint](../guides/infrastructure/infrastructure-reference.md) +- [ADR 0029](../ADRs/0029-central-token-mint-secretless-fullsend.md) +- [ADR 0059](../ADRs/0059-public-mint-mode-with-wildcard-allowlists.md) +- [ADR 0068 — Public community mint architecture](../ADRs/0068-public-community-mint-architecture.md) +- [#1263](https://github.com/fullsend-ai/fullsend/issues/1263) · [#1262](https://github.com/fullsend-ai/fullsend/issues/1262) +- [#915](https://github.com/fullsend-ai/fullsend/issues/915) discussion (2026-05-25) diff --git a/website/.vitepress/config.ts b/website/.vitepress/config.ts index 7447b9a5e6..f4e556f3c3 100644 --- a/website/.vitepress/config.ts +++ b/website/.vitepress/config.ts @@ -283,6 +283,11 @@ export default defineConfig({ collapsed: true, items: getMarkdownFiles("problems", "problems"), }, + { + text: "Spikes", + collapsed: true, + items: getMarkdownFiles("spikes", "spikes"), + }, { text: "Experiments (Exploratory)", collapsed: true,