diff --git a/launchpad/deploy/DEPLOYMENT-CHANGES.md b/launchpad/deploy/DEPLOYMENT-CHANGES.md new file mode 100644 index 00000000000..16688a20443 --- /dev/null +++ b/launchpad/deploy/DEPLOYMENT-CHANGES.md @@ -0,0 +1,234 @@ +# Launchpad relay deployment mapping changes + +## Audience and purpose + +This document is for human operators and automation agents from any provider. +It records exactly what changed in the Launchpad relay image publication and VPS +deployment path, and why each change was necessary. + +For the procedure that operators should run, use +[`VPS-DEPLOYMENT-RUNBOOK.md`](VPS-DEPLOYMENT-RUNBOOK.md). The material under +[`archived/`](archived/) is historical evidence from a failed deployment method; +it is not an alternative runbook. + +## Problem being corrected + +The fork inherited an internally consistent upstream deployment path: + +```text +block/buzz source + -> .github/workflows/docker.yml on main + -> ghcr.io/block/buzz + -> BUZZ_IMAGE, defaulting to ghcr.io/block/buzz:main + -> deploy/compose/run.sh + -> relay container +``` + +That mapping was unsafe in `launchpad-26/buzz`. A VPS checkout did not build the +root `Dockerfile`; it consumed the prebuilt image selected by `BUZZ_IMAGE`. +Because Compose supplied `ghcr.io/block/buzz:main` when `BUZZ_IMAGE` was absent, +a Launchpad deployment could complete successfully while running upstream Block +code. Repository identity on disk therefore did not prove runtime identity. + +The corrected path is: + +```text +launchpad-26/buzz commit on launchpad + -> .github/workflows/docker.yml + -> root Dockerfile, runtime target + -> ghcr.io/launchpad-26/buzz + -> sha- tag and manifest digest + -> deploy/compose/.env BUZZ_IMAGE + -> launchpad/deploy/run.sh policy checks + -> deploy/compose/run.sh orchestration + -> deploy/compose/compose.yml relay service + -> relay container +``` + +Every transition is now explicit. Normal VPS deployment still consumes a +prebuilt image and does not build the root `Dockerfile` locally. + +## Exact file changes and rationale + +### `.github/workflows/docker.yml` + + + +| Before | After | Why | +|---|---|---| +| Branch publication listened to `main`. | Branch publication listens to `launchpad`. | The fork's canonical source branch is `launchpad`; the workflow must build the commits that Launchpad actually merges. | +| `IMAGE_NAME` could fall back to `ghcr.io/block/buzz`. | `IMAGE_NAME` is fixed to `ghcr.io/launchpad-26/buzz`. | A missing repository variable must not redirect a Launchpad build into the upstream package namespace. | +| Commit tags used the short Git SHA. | Commit tags use `sha-` plus the full 40-character Git SHA. | The complete source revision is unambiguous and directly usable as a production pin. | +| OCI metadata did not explicitly identify the Launchpad source and revision. | Metadata sets `org.opencontainers.image.source=https://github.com/launchpad-26/buzz` and `org.opencontainers.image.revision=${{ github.sha }}`. | A pulled image can identify its source repository and exact Git commit without relying only on a moving tag. | +| Provenance examples verified ownership against `block`. | Examples verify against `launchpad-26`. | Attestation verification must use the organization that publishes the corrected image. | +| The inherited APNs push-gateway jobs still targeted `ghcr.io/block/buzz-push-gateway`. | Both gateway jobs run only when `github.repository == 'block/buzz'`. | Launchpad does not operate that separate service, and silently retargeting it would make an unsupported infrastructure decision. Keeping the inherited jobs conditional also reduces future upstream merge divergence. | + + + +The relay build remains multi-architecture (`linux/amd64` and `linux/arm64`). +Each architecture is pushed by digest, the merge job creates the multi-arch +manifest, and GitHub Actions attaches build-provenance attestations. + +The resulting publication behavior is: + +- a push to `launchpad` publishes `:launchpad`, + `:sha-`, and corresponding `debug-` tags; +- a `relay-v*` tag retains the inherited semver tag family and stable-release + `:latest` behavior; +- a pull request builds but does not publish an image; +- Block push-gateway jobs are skipped in `launchpad-26/buzz`. + +The moving `:launchpad` tag is supplementary. It is not the recommended +production value. + +### `Dockerfile` + +The header and OCI source, URL, and documentation labels now point to +`https://github.com/launchpad-26/buzz` instead of `block/buzz`. + +This change is metadata-only; the build stages and runtime contents are not +forked. The labels matter because they make image ownership and provenance +inspectable and associate the GHCR artifact with the repository that built it. +They do not remove upstream Apache-2.0 attribution or change runtime +dependencies. + +### `deploy/compose/compose.yml` + +The relay image expression changed from: + +```yaml +image: ${BUZZ_IMAGE:-ghcr.io/block/buzz:main} +``` + +to: + +```yaml +image: ${BUZZ_IMAGE:?BUZZ_IMAGE must be set to an immutable relay image} +``` + +The previous expression silently executed upstream code when the variable was +missing. The required-variable expression makes image selection fail closed and +uses Compose interpolation supported by the project's required Docker Compose +V2 version. It deliberately does not choose a default Launchpad tag either: +production image selection is an operator decision and must be reviewable. + +### `deploy/compose/.env.example` + +The example image changed from the moving upstream value +`ghcr.io/block/buzz:main` to the clearly incomplete Launchpad form: + +```text +ghcr.io/launchpad-26/buzz:sha-CHANGE_ME_FULL_40_CHARACTER_GIT_COMMIT +``` + +The example teaches the correct namespace and immutable commit-tag format +without checking a real environment value or credential into Git. The local +`deploy/compose/.env` remains ignored and must contain the real deployment +configuration. + +### `deploy/compose/README.md` + +The canonical Compose documentation now explains that GitHub Actions builds the +image, the VPS consumes it through `BUZZ_IMAGE`, and normal Launchpad operations +start through the policy guard. It also records the distinct pull behavior of +`start`, `upgrade`, and `restart`, plus traceability and rollback commands. + +This documentation was necessary because the old quick start made the upstream +image default look authoritative and did not expose the source-to-runtime gap. + +### `launchpad/deploy/run.sh` + +This is a new, thin policy guard. Before delegating, it: + +1. requires Docker and Docker Compose V2; +2. requires Compose version 2.24.4 or newer; +3. requires the local, untracked `deploy/compose/.env`; +4. requires exactly one non-empty `BUZZ_IMAGE` assignment in that file; +5. rejects `ghcr.io/block/buzz`, including tags and digests; +6. rejects every namespace except `ghcr.io/launchpad-26/buzz`; +7. accepts a digest or full 40-character `sha-...` tag as immutable; +8. warns and rejects a floating tag unless + `BUZZ_ALLOW_FLOATING_IMAGE=true` is deliberately set for development or + testing; +9. exports the reviewed `.env` image so an ambient shell variable cannot + replace it; and +10. delegates the requested operation to `deploy/compose/run.sh`. + +The `check` command performs the policy checks and renders the canonical Compose +configuration without changing services. The override for floating tags never +permits an upstream Block image. + +The guard contains no copied Compose lifecycle logic and owns no parallel +stack. That isolation gives Launchpad a strict image policy without maintaining +a fork of upstream orchestration. + +### `launchpad/deploy/README.md` and `launchpad/deploy/AGENTS.md` + +These files identify `launchpad/deploy/run.sh` as the active guard, preserve +`deploy/compose/run.sh` as the canonical runner, and mark everything under +`archived/` as non-executable historical material. + +This boundary is explicit because the former experiment was based on the wrong +image-flow model. Retaining it as an apparent alternative would invite humans +or agents to reintroduce the same upstream-image failure. + +## What intentionally did not change + +- `deploy/compose/run.sh` remains the only Compose lifecycle implementation. +- `deploy/compose/compose.caddy.yml`, `compose.dev.yml`, and `Caddyfile` retain + their upstream-compatible roles. +- The VPS still pulls a prebuilt image; it does not build the relay locally. +- Release-tag semantics remain inherited except that artifacts publish in the + Launchpad namespace with full-SHA traceability. +- The Block APNs push-gateway implementation remains in the shared workflow but + cannot execute in the Launchpad repository. +- Historical files under `launchpad/deploy/archived/` were not converted into + a supported deployment path. + +These choices minimize the conflict surface when merging future changes from +`block/buzz`. Likely conflict points are the four shared upstream files changed +for correctness: `.github/workflows/docker.yml`, `Dockerfile`, +`deploy/compose/compose.yml`, and `deploy/compose/README.md`. Launchpad policy +and operator guidance remain isolated under `launchpad/deploy/`. + +## Failure behavior after the change + + + +| Scenario | Result | +|---|---| +| `BUZZ_IMAGE` missing from `.env` | Guard aborts; Compose also rejects the missing variable. | +| `BUZZ_IMAGE` empty or assigned more than once | Guard aborts. | +| `ghcr.io/block/buzz:main` or another Block relay reference | Guard aborts even if the floating-image override is set. | +| Full Launchpad commit tag | Accepted, for example `ghcr.io/launchpad-26/buzz:sha-<40-hex-commit>`. | +| Launchpad manifest digest | Accepted, for example `ghcr.io/launchpad-26/buzz@sha256:<64-hex-digest>`. | +| Moving Launchpad tag such as `:launchpad` | Warning followed by rejection in normal use; accepted only with the explicit development/testing override. | +| Direct use of canonical Compose with no `BUZZ_IMAGE` | Compose aborts instead of selecting Block Buzz. | + + + +## Source-to-runtime traceability + +For a concrete source commit `GIT_COMMIT`, the workflow publishes: + +```text +ghcr.io/launchpad-26/buzz:sha-GIT_COMMIT +``` + +The multi-arch manifest also has a `sha256:` digest. Production should place +the digest reference in `deploy/compose/.env`; the full-SHA tag is the fallback +immutable choice. The running image exposes +`org.opencontainers.image.revision=GIT_COMMIT`, while the workflow attestation +binds the manifest digest to the GitHub build. The runbook contains the exact +inspection commands. + +## Validation boundary + +Static validation can prove the workflow configuration, Compose failure mode, +guard behavior, YAML structure, shell syntax, and source-to-image mapping. It +cannot prove that GHCR accepted a publication or that a real VPS can pull the +package. After this change reaches `launchpad`, an operator must confirm a +successful Docker workflow run, the expected full-SHA tag and digest in +`ghcr.io/launchpad-26/buzz`, package visibility or authentication, attestation +verification, and a fresh VPS deployment before treating the live chain as +proven. diff --git a/launchpad/deploy/VPS-DEPLOYMENT-RUNBOOK.md b/launchpad/deploy/VPS-DEPLOYMENT-RUNBOOK.md new file mode 100644 index 00000000000..d74f7ce2a42 --- /dev/null +++ b/launchpad/deploy/VPS-DEPLOYMENT-RUNBOOK.md @@ -0,0 +1,352 @@ +# Launchpad Buzz VPS deployment runbook + +## Audience and authority + +This runbook is for human operators and automation agents from any provider. +The same commands and safety boundaries apply regardless of who performs the +deployment. + +The supported entry point is: + +```text +launchpad/deploy/run.sh +``` + +It validates Launchpad image policy and delegates orchestration to +`deploy/compose/run.sh`. Do not execute, copy, or repair deployment material +under `launchpad/deploy/archived/`; it is a record of a failed method. + +Agents must also obey the repository instructions in `launchpad/AGENTS.md` and +`launchpad/deploy/AGENTS.md`. An agent must not start, stop, upgrade, or roll +back a production VPS unless the human request explicitly authorizes that +state change. + +## Required inputs + +Before changing a VPS, identify all of the following: + +- the exact approved 40-character Git commit from the `launchpad` branch; +- a successful `Docker image` workflow run for that commit; +- the corresponding `ghcr.io/launchpad-26/buzz` full-SHA tag or, preferably, + manifest digest; +- the VPS public domain and DNS record if Caddy will provide HTTPS; +- the relay owner public key; +- stable relay, database, Redis, S3, and git-hook secrets; and +- a rollback image plus compatible pre-change data backups for an upgrade. + +Stop if the approved commit, successful workflow run, image identity, package +access, production secrets, backup state, or migration compatibility is +unknown. Do not substitute `:launchpad`, `:latest`, `:main`, or any +`ghcr.io/block/buzz` image. + +Never paste `deploy/compose/.env`, tokens, private keys, or secret-bearing +command output into an issue, pull request, chat, agent log, or terminal capture. + +## VPS prerequisites + +Use a supported Ubuntu VPS with: + +- SSH and `sudo` access; +- Docker Engine; +- Docker Compose V2 version 2.24.4 or newer; +- Git; +- outbound HTTPS access to GitHub and GHCR; and +- ports 80 and 443 allowed when using the Caddy TLS configuration. + +Install Docker Engine and the Compose plugin through Docker's supported Ubuntu +installation method. The project does not maintain a separate Docker installer. +Confirm the installed tools before continuing: + +```bash +docker --version +docker compose version +git --version +``` + +If `ghcr.io/launchpad-26/buzz` is private, authenticate Docker to GHCR using a +least-privileged credential with package-read access. Supply the credential +through a secure interactive or secret-management channel; do not place it in +the repository or this runbook. + +## Fresh VPS deployment + +### 1. Check out the canonical deployment files + +Run as the intended non-root deployment user: + +```bash +git clone --branch launchpad --single-branch https://github.com/launchpad-26/buzz.git +cd buzz +git status --short --branch +``` + +The checked-out repository supplies the deployment files. It does not cause the +VPS to build the relay image. + +### 2. Select and verify an exact source commit + +Set `SOURCE_COMMIT` to the approved, full 40-character commit from the +successful workflow run, then confirm that the commit exists in the checkout: + +```bash +SOURCE_COMMIT='' +git show --no-patch --format='commit=%H subject=%s' "$SOURCE_COMMIT" +IMAGE_TAG="ghcr.io/launchpad-26/buzz:sha-${SOURCE_COMMIT}" +printf 'candidate_image=%s\n' "$IMAGE_TAG" +``` + +`` is an instruction marker, not a real +value. Do not continue until it has been replaced and the output shows the same +commit that the operator approved. + +If GitHub CLI is available, confirm the workflow conclusion without exposing +credentials: + +```bash +gh run list \ + --repo launchpad-26/buzz \ + --workflow docker.yml \ + --commit "$SOURCE_COMMIT" \ + --json databaseId,status,conclusion,headSha,url +``` + +Continue only when the relevant non-PR publication run completed successfully. +A pull-request build does not publish an image. + +### 3. Pull the full-SHA tag and resolve its digest + +```bash +docker pull "$IMAGE_TAG" +IMAGE_REF=$(docker image inspect \ + --format '{{index .RepoDigests 0}}' \ + "$IMAGE_TAG") +printf 'immutable_image=%s\n' "$IMAGE_REF" +``` + +The printed value must start with: + +```text +ghcr.io/launchpad-26/buzz@sha256: +``` + +Stop if it points anywhere else. The digest reference is the preferred +production value because it cannot move after publication. + +When GitHub CLI supports attestation verification on the host, verify the +artifact before deployment: + +```bash +gh attestation verify "oci://${IMAGE_REF}" --owner launchpad-26 +``` + +If attestation verification is unavailable, record that fact for the reviewer; +do not claim provenance verification was performed. + +### 4. Create the local production configuration + +```bash +cp deploy/compose/.env.example deploy/compose/.env +chmod 600 deploy/compose/.env +${EDITOR:?Set EDITOR to a trusted terminal editor} deploy/compose/.env +``` + +In the editor: + +1. replace every `CHANGE_ME` value; +2. set `BUZZ_IMAGE` to the exact value printed as `immutable_image`; +3. set the public domain and URL values consistently; +4. set `RELAY_OWNER_PUBKEY` to the intended 64-character hex public key; +5. generate and store stable secrets through the operator's approved secret + process; and +6. decide migration behavior deliberately before the first start. + +`BUZZ_AUTO_MIGRATE=true` lets the relay apply embedded migrations on startup. +For an established production database, migration and backup compatibility must +be reviewed before an image change. Do not rotate stable secrets during a +restart or upgrade. + +The `.env` file is ignored by Git. Confirm only its permissions and ignore +status; do not print its contents: + +```bash +stat -c '%a %n' deploy/compose/.env +git check-ignore deploy/compose/.env +``` + +### 5. Validate without starting services + +```bash +./launchpad/deploy/run.sh check +./launchpad/deploy/run.sh backup-hint +``` + +The guard prints the selected public image reference and Compose version. It +must report that the deployment configuration is valid. It must not print +secrets. + +### 6. Start the stack + +For a public VPS where Caddy terminates HTTPS: + +```bash +export BUZZ_COMPOSE_TLS=true +./launchpad/deploy/run.sh start +./launchpad/deploy/run.sh status +``` + +For an intentionally non-TLS development environment, omit the exported TLS +switch. Do not use the non-TLS form for a public production relay without a +separate, reviewed TLS terminator. + +`start` runs `docker compose up -d --wait`. It does not run an explicit pull. +Compose may fetch an absent image, but it may reuse an image already present on +the VPS. Pulling and resolving the immutable image in step 3 removes that +ambiguity for the first deployment. + +### 7. Verify health and running identity + +Read the public domain without displaying the rest of `.env`, then check the +relay and inspect the running image: + + + +```bash +BUZZ_DOMAIN=$(sed -n 's/^BUZZ_DOMAIN=//p' deploy/compose/.env) +curl --fail --silent --show-error "https://${BUZZ_DOMAIN}/_liveness" + +container_id=$(docker compose \ + --env-file deploy/compose/.env \ + -f deploy/compose/compose.yml \ + ps -q relay) + +docker inspect \ + --format 'configured={{.Config.Image}} image_id={{.Image}}' \ + "$container_id" + +image_id=$(docker inspect --format '{{.Image}}' "$container_id") +docker image inspect \ + --format 'repo_digests={{json .RepoDigests}} revision={{index .Config.Labels "org.opencontainers.image.revision"}} source={{index .Config.Labels "org.opencontainers.image.source"}}' \ + "$image_id" +``` + + + +Confirm all four facts: + +1. `configured=` equals the reviewed `BUZZ_IMAGE` digest; +2. `repo_digests` includes the same Launchpad digest; +3. `revision=` equals `SOURCE_COMMIT`; and +4. `source=` is `https://github.com/launchpad-26/buzz`. + +If any value differs, stop and investigate before declaring the deployment +successful. + +## Command behavior + +All supported operations should go through `launchpad/deploy/run.sh` so image +policy is checked first. + + + +| Command | Canonical behavior | Pulls images? | +|---|---|---| +| `check` | Renders and validates the Compose configuration. | No. | +| `start` | Runs `compose up -d --wait`. | No explicit pull; Compose may fetch a missing image. | +| `pull` | Runs `compose pull`. | Yes, for configured service images. | +| `upgrade` | Runs `compose pull`, then `compose up -d --wait`, then prints backup reminders. | Yes, for all configured service images, not only the relay. | +| `restart` | Runs `compose up -d --wait --force-recreate relay`. | No. | +| `stop` | Runs `compose down` without `--volumes`. | No; named volumes remain. | +| `status` | Runs `compose ps`. | No. | +| `logs [service]` | Follows logs; defaults to `relay`. | No. | + + + +Because `upgrade` pulls all configured service images, review the whole Compose +configuration and backups before production upgrades. A relay digest pin does +not make the other service tags immutable. + +## Intentional upgrade + +1. Approve a new full source commit and confirm its non-PR Docker workflow run + succeeded. +2. Pull its full-SHA tag, resolve the manifest digest, and verify provenance as + in fresh-deployment steps 2 and 3. +3. Record the currently running `BUZZ_IMAGE`, Git revision, and image digest. +4. Run the backup checklist and create coordinated Postgres, MinIO/media, and + git-data backups from the same maintenance window. +5. Review database migrations between the running and target commits. +6. Edit only `BUZZ_IMAGE` in the local `.env` unless the release explicitly + requires another reviewed configuration change. +7. Validate, upgrade, and inspect the result: + +```bash +export BUZZ_COMPOSE_TLS=true +./launchpad/deploy/run.sh check +./launchpad/deploy/run.sh backup-hint +./launchpad/deploy/run.sh upgrade +./launchpad/deploy/run.sh status +``` + +Repeat the health and identity checks from fresh-deployment step 7. Do not use +`restart` as an upgrade command: it does not pull the target image. + +## Rollback + +An image rollback is appropriate only when the previous relay version can use +the current database and stored data. If the upgrade applied an incompatible +migration, restore the matching pre-upgrade database, object/media, and git-data +backups as a coordinated recovery instead of rolling back only the container. + +For an image-compatible rollback: + +1. confirm the previous Launchpad digest and its source commit from the recorded + pre-upgrade state; +2. ensure that exact digest is still pullable; +3. replace only `BUZZ_IMAGE` in `deploy/compose/.env` with the previous digest; +4. run: + +```bash +export BUZZ_COMPOSE_TLS=true +./launchpad/deploy/run.sh check +./launchpad/deploy/run.sh upgrade +./launchpad/deploy/run.sh status +``` + +1. repeat the health and running-identity checks; and +2. record why the rollback occurred and which immutable digest is now running. + +Do not roll back to a moving tag. Do not restore only one member of a +Postgres/object/git backup set when those components must represent the same +maintenance window. + +## Development-only floating image override + +The guard rejects moving tags. If a development or test environment +intentionally needs one, the operator may run: + +```bash +BUZZ_ALLOW_FLOATING_IMAGE=true ./launchpad/deploy/run.sh check +``` + +Set the override on the actual operation as well. This is not a production +procedure. The override never permits `ghcr.io/block/buzz` or a non-Launchpad +namespace. + +## Failure response + +- Missing or duplicate `BUZZ_IMAGE`: correct the local `.env`; do not bypass + the guard. +- Block or non-Launchpad image: stop and resolve the correct Launchpad workflow + artifact. +- Floating image rejection: select a digest or full commit tag; do not set the + override for production. +- GHCR pull or attestation failure: confirm workflow success, package access, + and authentication without exposing credentials. +- Compose version rejection: install a supported Compose V2 release; do not + remove the minimum-version check. +- Health or identity mismatch: preserve logs and inspection output that contain + no secrets, stop further rollout, and escalate to the human operator. + +The deployment is complete only when the configured digest, running digest, +OCI revision label, approved Git commit, workflow run, and Launchpad source URL +all agree.