From 7a3ada9f47026baaafd07fa6bf2ad2eb61ceabf4 Mon Sep 17 00:00:00 2001 From: Konstantin Tursunov Date: Fri, 7 Aug 2026 11:25:36 +0800 Subject: [PATCH 01/59] chore(studio): sync Constructor Studio config - add the sdlc kit tool_risk_fingerprint to core.toml - add the @cf:root-agents block to CLAUDE.md - normalize a dash in the config README Signed-off-by: Konstantin Tursunov --- .cf-studio/config/README.md | 2 +- .cf-studio/config/core.toml | 1 + .gitignore | 2 -- CLAUDE.md | 8 ++++++++ 4 files changed, 10 insertions(+), 3 deletions(-) diff --git a/.cf-studio/config/README.md b/.cf-studio/config/README.md index 5020a6f28..a664dfd30 100644 --- a/.cf-studio/config/README.md +++ b/.cf-studio/config/README.md @@ -1,4 +1,4 @@ -# config — User Configuration +# config -- User Configuration This directory contains **user-editable** configuration files. diff --git a/.cf-studio/config/core.toml b/.cf-studio/config/core.toml index 69d3a4c71..c4744560e 100644 --- a/.cf-studio/config/core.toml +++ b/.cf-studio/config/core.toml @@ -11,6 +11,7 @@ version = "v1.2.1" source = "github:constructorfabric/studio-kit-sdlc" install_mode = "copy" tracking = "tracked" +tool_risk_fingerprint = "3bdcd30d3029ad80e74c76eb0b57d80ad617a51191eb9d9307b2921c65f5dc16" [kits.sdlc.resources] [kits.sdlc.resources.adr_template] diff --git a/.gitignore b/.gitignore index d7ddf951c..0fbf02f74 100644 --- a/.gitignore +++ b/.gitignore @@ -86,8 +86,6 @@ coverage-raw/ # Files matched here are owned by Constructor Studio and may be overwritten. .cf-studio/.core/ .cf-studio/.gen/ -# Constructor Studio plan files — local working state (plan-first + cf-plan). -.cf-studio/.plans/ .agents/skills/cf-analyze/SKILL.md .agents/skills/cf-auto-config/SKILL.md .agents/skills/cf-brainstorm/SKILL.md diff --git a/CLAUDE.md b/CLAUDE.md index 43c994c2d..46099ee2f 100644 --- a/CLAUDE.md +++ b/CLAUDE.md @@ -1 +1,9 @@ + +```toml +cf-studio-path = ".cf-studio" +``` + +ALWAYS resolve and enforce prerequisites of skills/workflows/commands BEFORE applying user intent. + + @AGENTS.md From 1257fbe20ca90b38f0c9bdf9ed30f5f3595f269a Mon Sep 17 00:00:00 2001 From: Konstantin Tursunov Date: Sun, 9 Aug 2026 12:12:42 +0800 Subject: [PATCH 02/59] feat(gitops): add the test-stand environment, CI emulation and deploy credential The umbrella-only gitops environment the post-merge test-stand deploy installs through, plus the two operator tools that make it usable without CI. `environments/test-stand/` deliberately manages the umbrella and nothing else: `bootstrap.*` and `system.*` are all false because the stand's datastores are operator-managed in their own namespaces and its edge is Envoy Gateway with HTTPRoutes the chart does not render. Its values reconcile the reference deployment repo against what the release actually carries, and keep `credentials.deploymentMode: helm` / `autoGenerate: true` to match: the chart's `lookup` finds the pre-created, label-less `insight-db-creds` and stands down, so an upgrade cannot rotate live datastore passwords. A server-side dry run renders exactly the release's object set, adding and removing nothing. `emulate-ci-deploy.sh` runs the workflow's three stages from a laptop through an explicit kubeconfig, read-only unless `--apply` plus a typed confirmation token. It asserts the target cluster before anything runs, and prints every command it is about to execute so the local path and the CI path can be diffed rather than assumed. `--allow-dirty` swaps `make diff` for a bare `helm template` when the tree is not clean, which is the only way to rehearse changes that are not committed yet; it says so loudly rather than passing quietly. `provision-ci-deployer.sh` mints the namespace-scoped ServiceAccount the deploy runs as, and asserts its containment before handing over a kubeconfig. A token is deletable; a client certificate needs a CA rotation, which is why an admin kubeconfig is never the CI credential. Refs #2244 Signed-off-by: Konstantin Tursunov --- .../gitops/environments/test-stand/README.md | 282 ++++ .../environments/test-stand/inventory.yaml | 199 +++ .../manifests/httproute-keycloak.yaml | 49 + .../test-stand/manifests/httproute.yaml | 63 + .../environments/test-stand/values.yaml | 531 ++++++++ deploy/gitops/scripts/emulate-ci-deploy.sh | 1200 +++++++++++++++++ .../gitops/scripts/provision-ci-deployer.sh | 907 +++++++++++++ .../deployment/gitops/ci-emulation.md | 260 ++++ .../specs/sop/credentials-runbook.md | 490 +++++++ 9 files changed, 3981 insertions(+) create mode 100644 deploy/gitops/environments/test-stand/README.md create mode 100644 deploy/gitops/environments/test-stand/inventory.yaml create mode 100644 deploy/gitops/environments/test-stand/manifests/httproute-keycloak.yaml create mode 100644 deploy/gitops/environments/test-stand/manifests/httproute.yaml create mode 100644 deploy/gitops/environments/test-stand/values.yaml create mode 100755 deploy/gitops/scripts/emulate-ci-deploy.sh create mode 100755 deploy/gitops/scripts/provision-ci-deployer.sh create mode 100644 docs/components/deployment/gitops/ci-emulation.md create mode 100644 docs/components/deployment/specs/sop/credentials-runbook.md diff --git a/deploy/gitops/environments/test-stand/README.md b/deploy/gitops/environments/test-stand/README.md new file mode 100644 index 000000000..3017c7c10 --- /dev/null +++ b/deploy/gitops/environments/test-stand/README.md @@ -0,0 +1,282 @@ +# `test-stand` — the published test stand + +The gitops environment for **insight-test.cfabric.org**: the cluster that CI +upgrades to the umbrella chart it has just published, then seeds and +smoke-tests, on every merge to `main` +([constructorfabric/insight#2244](https://github.com/constructorfabric/insight/issues/2244)). + +Its whole job is to answer one question automatically: *does the chart we +just published actually install, hold data, and let a person log in and see +that data?* + +> **Read this before touching anything here.** This environment is shaped +> differently from every other environment in `deploy/gitops/`. The usual +> `make bootstrap` → `make system-*` → `make deploy` sequence is **not** the +> deploy path for this stand, and two of those three targets would damage it. +> The [Why not `make deploy`](#why-not-make-deploy) section is not optional +> reading. + +## Contents + +```text +environments/test-stand/ +├── README.md # this file +├── inventory.yaml # cluster address + "this env manages the umbrella only" +├── values.yaml # the umbrella overlay — the file the deploy passes to helm +└── manifests/ + ├── httproute.yaml # public hostname -> insight-gateway (the chart renders none) + └── httproute-keycloak.yaml # /kc on the same hostname -> the bundled Keycloak +``` + +There is deliberately no `sealed-secrets/` directory, no `keycloak/realms/` +directory and no `-values.yaml` here. Each absence is a decision, and +each one is explained below. + +## The ownership boundary + +This tree now owns the **stand's application configuration**. It does not own +the cluster. + +| Layer | What | Owned by | Changed how | +|---|---|---|---| +| L0 | Cluster prereqs: cert-manager and its ClusterIssuer, Envoy Gateway and the `Gateway` object, the namespaces themselves | the deployment repository (outside this repo) | a human, deliberately | +| L2 | Datastores — ClickHouse, MariaDB, Redis, Redpanda — each under its own operator in its own namespace; plus Airbyte and Argo Workflows | the deployment repository | a human, deliberately | +| L2 | Generate-once Secrets: `insight-db-creds`, `insight-authenticator-signing-keys`, `insight-oidc`, `insight-keycloak-admin`, `insight-keycloak-config` | the deployment repository | a human, deliberately | +| L2 | The Keycloak realm content (the `insight-keycloak-config-realms` ConfigMap) | the deployment repository | a human, deliberately | +| **L3** | **The umbrella Helm release `insight` in namespace `insight` — every value in `values.yaml`** | **this directory** | **CI, on every merge to `main`** | +| L3 | The two `HTTPRoute`s in `manifests/` | source of truth here; applied by the deployment repository | a human, from the files here | +| L3 | The `argo-workflow` ServiceAccount + Role + RoleBinding that the chart's WorkflowTemplates pin but do not create | the deployment repository | a human, deliberately | + +Two consequences worth stating plainly: + +* **Nothing in this directory creates or rotates a credential.** Every Secret + the release consumes already exists, was generated once, and is referenced + by name. That is why `inventory.yaml` lists all of them with + `enabled: false` — see the long comment there for what each one would break + if it were re-materialised. +* **A change merged here reaches a published stand at merge speed.** There is + no staging step between this file and the cluster. Review accordingly. + +### What this env does not own, but depends on + +Four objects live outside the Helm release and outside this directory. If any +of them disappears, the release still installs and reports `deployed`, and the +stand is broken anyway: + +| Object | Symptom if missing | +|---|---| +| `HTTPRoute/insight-gateway` | the public URL answers nothing; every smoke check fails at the first request | +| `HTTPRoute/insight-keycloak` | `/kc` is unreachable, so OIDC discovery fails and nobody can log in | +| `ServiceAccount/argo-workflow` (+ its Role/RoleBinding) | every scheduled transform and data-quality run fails with `serviceaccount "argo-workflow" not found`, while all app pods stay healthy — nothing surfaces until a scheduled run | +| The Secrets in the table above | services fail to start, or start with blank configuration | + +The two routes are committed here (`manifests/`) because the acceptance +criteria travel through them. They are **verified, not applied**, by the +deploy — see the header comment in `manifests/httproute.yaml` for why re-applying +would create two writers on one object. The Argo RBAC is not copied here: it +is Argo plumbing rather than application configuration, and it sits outside the +deploy/seed/smoke scope this environment was created for. + +## Deploying by hand + +Everything below is read-only until the `helm upgrade` line, and each step is +worth running on its own the first time. + +**0. Point at the stand.** The `kubeContext` in `inventory.yaml` is an +assertion, not a lookup — `make` refuses to act unless +`kubectl config current-context` equals it. Either rename your context to +match, or pass `KUBE_CTX=` on every make command line: + +```bash +export KUBECONFIG=/path/to/your/test-stand.kubeconfig # outside this repo's working tree +kubectl config rename-context insight-test-stand +kubectl config use-context insight-test-stand +``` + +Keep the kubeconfig **outside the repository working tree**. The Makefile's +`sync-clean` prerequisite fails on any dirty file, so a kubeconfig written next +to these files blocks every make target that touches the cluster. + +**1. Confirm you are on the right cluster.** The context-name check above +proves only that a file says so. Assert the API server as well — this is the +one mechanical guard against an upgrade landing somewhere it should not: + +```bash +kubectl config view --minify -o jsonpath='{.clusters[0].cluster.server}' +``` + +Compare it against the address recorded for this stand in the deployment +repository. It is deliberately not written down here: this repository is +public, and a cluster API endpoint is not something to publish. + +**2. See what would change.** Read-only, and it renders through the same +values file the upgrade will use: + +```bash +make diff ENV=test-stand INSIGHT_VERSION= +``` + +`make diff` works for this env unchanged. It renders offline and never +contacts the cluster, so it needs no context; its prerequisites are +`sync-clean`, `values-present` and `chart-present` — none of which trips over +the sealed-secrets problem described below. `sync-clean` is why step 0 insists +the kubeconfig lives outside the working tree. + +**3. Upgrade.** Called directly, not through `make deploy`: + +```bash +helm upgrade --install insight \ + oci://ghcr.io/constructorfabric/charts/insight \ + --version "" \ + --namespace insight \ + --values deploy/gitops/environments/test-stand/values.yaml \ + --set-string authenticator.oidc.clientSecret="$(kubectl -n insight \ + get secret insight-oidc -o jsonpath='{.data.client-secret}' | base64 -d)" \ + --wait --timeout 10m --history-max 10 +``` + +Three things about that command are load-bearing: + +* **`--set-string authenticator.oidc.clientSecret=…` is mandatory.** The + values file leaves it empty because this repository is public. The chart + writes whatever it is given straight into the authenticator's config Secret, + so an upgrade without this flag produces a confidential OIDC client with a + blank secret: pods Ready, release `deployed`, every login broken. Read it + from the cluster Secret, as above, so there is one source of truth rather + than a second copy to rotate. +* **`--wait` but deliberately no `--atomic`.** A failed upgrade is left in + place. Rolling back automatically destroys the evidence of *why* it failed, + and this stand exists to produce that evidence. Recovery is the next merge, + or a human running `make rollback` (below). +* **`--timeout 10m`**, not the Makefile's 30m default. A deploy that is going + to fail should say so inside the CI budget. + +**4. Restart what the chart cannot know to restart.** Each subchart's +`checksum/config` annotation hashes its own ConfigMap. It does **not** cover +the umbrella-rendered `insight-*-config` Secrets, which the pods consume with +`envFrom` and therefore read exactly once, at container start. So a changed +datastore host, tenant, or client secret updates the Secret, leaves the pod +spec identical, and never reaches a running process — a `deployed` release +with all-Ready pods running stale configuration: + +```bash +kubectl -n insight rollout restart \ + deploy/insight-authenticator deploy/insight-analytics deploy/insight-identity-resolution +kubectl -n insight rollout status --timeout=5m \ + deploy/insight-authenticator deploy/insight-analytics deploy/insight-identity-resolution +``` + +All three are listed on purpose. Each of the three consumes its config Secret +with `envFrom`, so all three are subject to the same staleness — a restart +list of two would leave one service holding old configuration. + +**5. Check the edge is still routing.** The chart renders no route, so a +successful upgrade tells you nothing about whether the stand is reachable: + +```bash +kubectl -n insight get httproute insight-gateway insight-keycloak \ + -o custom-columns='NAME:.metadata.name,ACCEPTED:.status.parents[0].conditions[?(@.type=="Accepted")].status' +``` + +**6. Seed and smoke.** Seeding reuses the seeder verbatim — no test-stand +variant, no per-stand flags beyond the ones below (it discovers the tenant, +the datastore coordinates and the IdP source type from the cluster itself): + +```bash +src/ingestion/tools/seed/seed-stand.sh -n insight --email
--days 730 +``` + +The seeder's manifest — the list of personas, fixtures, the tenant and the +data window that the smoke suite reads — is **printed to the seed Job's +stdout** and written to a path inside a pod whose filesystem is discarded. +Capture it from the Job log before the Job's TTL reaps it. Treat that JSON as +run-internal: it carries persona addresses, UUIDs and in-cluster service URLs, +so it must not become a CI artifact on a public repository. + +### Rollback + +```bash +make status ENV=test-stand +make rollback ENV=test-stand +``` + +Both work for this env unchanged (they take the context from +`inventory.yaml` and assert it against your current one, so step 0 is a +prerequisite for both). `rollback` is a human action by design: an +automated rollback on failure is exactly what `--atomic` would have done, and +what step 3 deliberately does not do. + +## Why not `make deploy` + +`make deploy ENV=test-stand` does not work against this stand, and would not +be the right tool even if it did. Three separate reasons, in the order you +would hit them: + +1. **It aborts on the sealed-secrets prerequisite.** `deploy-insight` depends + on `apply-app-secrets`, which requires at least one + `*-sealedsecret.yaml` under `environments//sealed-secrets/insight/` + and `kubectl apply`s every file it finds. This env has none, on purpose, + and the cluster has neither the sealed-secrets controller nor the + `SealedSecret` CRD. There is no documented skip switch. + +2. **If that hurdle were removed, the target would then do damage.** + `apply-app-secrets` goes on to run `compose-app-secrets.sh`, which + overwrites the three `insight-*-config` Secrets with locally composed + content and reads the OIDC client secret from a key name this stand does + not use — writing a blank client secret into the authenticator's config. + In `helm` credentials mode the same-run `helm upgrade` rewrites them back, + so the visible result is churn plus a broken-login window; if the helm step + fails, the broken state is what is left. Sealing `insight-db-creds` into + this directory would be worse still: the controller would overwrite the + live Secret composed from the operators' own credentials, and every service + would lose its datastore login. + +3. **Its exit status cannot gate anything.** The deploy recipe is a single + backslash-continued shell line whose last command ends in `|| true`, and + the helm call is piped into `tee` with no `pipefail`. A failed upgrade + reports success. Acceptance criterion (2) — the workflow result is gated by + the deploy, seed and smoke — cannot be built on that. `--atomic` is also + hard-coded with no override, which contradicts the leave-it-failed + disposition in step 3 above. + +Making `make deploy` usable here means changing the Makefile: an optional +sealed-manifest glob, `set -o pipefail` plus a status-bearing final command, +and a way to opt out of `--atomic`. That is worth doing, and it is a separate +change from introducing this environment. Until then, the `helm upgrade` in +step 3 **is** the deploy path, and CI runs that same command. + +## What CI does + +The deploy runs as a reusable workflow called from the chart-publishing +workflow's final job, on `main` only, with the chart version passed in from +the publish job's output rather than read from `.insight-version` (which is +only committed at the very end of publishing, so a checkout of the trigger +commit would read the previous version). + +* Credentials come from the `insight-test-stand` GitHub environment, + restricted to `main`. The CI credential is a namespace-scoped + ServiceAccount in `insight` — admin kubeconfigs stay human-only, and no + credential for this stand lives in the repository. +* Runs are coalesced, never cancelled: an upgrade already in flight is allowed + to finish. +* Three named stages — deploy, seed, smoke — and smoke never runs after a + failed seed. +* On failure the run publishes a curated, redacted set of diagnostics only. + This repository is public and so are its run logs, so there are no + `describe` dumps, no environment dumps and no log artifacts. +* A red run belongs to the author of the merge that produced it. There is no + freeze: fix forward, or revert. + +## Known gaps + +* **Scripted login.** The stand's realm federates login to an external OAuth + provider and has no local password users, so a username+password login + cannot be scripted against it as configured. Resolving that is a decision + about who owns the realm and what credential CI is allowed to hold — see the + `keycloakConfig` comment in `values.yaml` for the ownership half of it. +* **`authenticator.overrideEnabled: true`** is carried forward from the + installed release. It is a standing impersonation primitive on an + internet-reachable stand, gated on that flag alone. Tracked separately; + see the comment on the key in `values.yaml`. +* **The identity CronJobs** can un-seed logins once their input table stops + being empty. See the comment at the bottom of `values.yaml`. diff --git a/deploy/gitops/environments/test-stand/inventory.yaml b/deploy/gitops/environments/test-stand/inventory.yaml new file mode 100644 index 000000000..7d5a14fc5 --- /dev/null +++ b/deploy/gitops/environments/test-stand/inventory.yaml @@ -0,0 +1,199 @@ +## +## environments/test-stand/inventory.yaml +## +## The published test stand (insight-test.cfabric.org) — the cluster CI +## upgrades to the umbrella chart it just published on merge to `main` +## (constructorfabric/insight#2244), then seeds and smoke-tests. +## +## ───────────────────────────────────────────────────────────────────────── +## THIS ENV MANAGES THE UMBRELLA AND NOTHING ELSE +## ───────────────────────────────────────────────────────────────────────── +## Every other environment in this tree assumes the gitops layer owns the +## whole stack: `make bootstrap` installs ingress-nginx / cert-manager / +## sealed-secrets, `make system-*` installs the datastores as Bitnami +## releases into `insight-infra`, and `make deploy` puts the umbrella on +## top. This stand is NOT built that way, and pretending otherwise would +## be actively destructive: +## +## what this tree's default model the test stand +## ───────────────── ───────────────────────────── ──────────────────────── +## datastores Bitnami charts, one release Kubernetes OPERATORS, +## each, all in `insight-infra` each in its OWN +## (`make system-`) namespace, installed and +## owned outside this repo +## edge ingress-nginx + Ingress Envoy Gateway + Gateway +## objects rendered by the API HTTPRoute; the chart +## chart renders no HTTPRoute, so +## the route is a committed +## manifest (manifests/) +## secrets sealed-secrets controller + plain Secrets, pre-created +## `make seal` + committed once and composed from the +## SealedSecret manifests datastore namespaces by the +## L0/L2 bring-up scripts +## namespaces `insight` + `insight-infra` `insight` only; there is +## no `insight-infra` +## +## So: every `bootstrap.*` and every `system.*` toggle below is false, and +## nothing is sealed. This env is a values overlay plus a cluster address — +## the L0 (cluster prereqs) and L2 (datastores, Envoy Gateway, Airbyte, +## Argo) layers are brought up and owned elsewhere. See README.md in this +## directory for the ownership boundary and the exact deploy command. +## + +# ─── Cluster targeting ───────────────────────────────────────────────── +# +# This name is an ASSERTION, not a lookup: `make`'s `kube-ctx` target +# refuses to run unless `kubectl config current-context` equals it. That +# check is the only thing standing between a mistyped KUBECONFIG and an +# upgrade landing on a cluster that is not this one, so the value is +# deliberately specific rather than something generic like `default` +# (a name several kubeconfigs use, which would make the guard pass by +# accident). +# +# Consequence: whoever produces a kubeconfig for this stand must NAME the +# context this. CI writes its kubeconfig from the `insight-test-stand` +# GitHub environment and renames the context to match. A human holding an +# admin kubeconfig whose context is called something else either renames +# it (`kubectl config rename-context insight-test-stand`) or passes +# `KUBE_CTX=` on the make command line — KUBE_CTX is a `?=` +# assignment, so the command line and the shell environment both win over +# this file. +kubeContext: insight-test-stand + +# TRUE, deliberately: this stand is published on the public internet and +# people look at it. `make deploy` therefore refuses to run unless the +# operator repeats the env name back: +# +# make deploy ENV=test-stand CONFIRM=yes-deploy-test-stand +# +# The token is `yes-deploy-`, built by the Makefile's `confirm` +# target — for this env, exactly `yes-deploy-test-stand`. CI is not +# exempt: the deploy workflow passes that literal string. That is not a +# security control (anything running with these credentials could deploy +# anyway), it is a typo control — it makes `ENV=test-stand` impossible to +# reach by tab-completion or by copying a command meant for another env. +protected: true + +# ─── Namespaces ──────────────────────────────────────────────────────── +namespaces: + # The umbrella release namespace. Everything this env touches lives + # here, which is also why a namespace-scoped ServiceAccount is enough + # for CI to deploy — no cluster-scoped grant is needed or wanted. + services: insight + + # Read the warning before changing this. + # + # THERE IS NO INFRA NAMESPACE ON THIS STAND. The datastores run under + # operators in their own namespaces, none of which is called this. The + # key is kept because the Makefile reads `.namespaces.infra` on the + # deploy path and hands it to scripts/push-deploy-log.sh, which tries a + # port-forward to a Loki Service in it. That Service does not exist + # here, the port-forward times out in a few seconds, and the Makefile + # appends `|| true` — so the deploy-log push is a no-op for this env by + # construction. Nothing else on the deploy path reads it. + # + # It is spelled with the default value rather than something invented so + # that a reader diffing this file against another env sees "same as + # everywhere else, and unused here" instead of wondering what a novel + # namespace name means. + infra: insight-infra + +# ─── L3 helm release ─────────────────────────────────────────────────── +# Must match the release already installed in `insight`. The chart hard-codes +# `insight-`-prefixed in-cluster DNS in several places (the keycloakConfig +# URL below among them), so this is not freely renameable. +release: insight + +# ─── L0 — cluster prereqs (drives `make bootstrap`) ──────────────────── +# +# ALL FALSE. `make bootstrap ENV=test-stand` is a no-op that prints four +# skip lines, and that is the intended behaviour — running any of these +# against this stand would install a second, competing copy of something +# the cluster already has: +# +# namespaces — `insight` exists; the CI ServiceAccount is scoped to +# it and cannot create namespaces anyway. `helm upgrade +# --install --create-namespace` covers the only case +# this toggle would have handled. +# ingressNginx — the edge is Envoy Gateway. Installing ingress-nginx +# would put a second LoadBalancer in front of the same +# hostname. +# certManager — already installed and owning the ClusterIssuer this +# env's `authenticator.tlsDiscovery.issuerRef` names. +# sealedSecrets — not installed, and must not be: see `secrets:` below. +bootstrap: + namespaces: false + ingressNginx: false + certManager: false + sealedSecrets: false + +# ─── L2 — shared infra (drives `make system`) ────────────────────────── +# +# ALL FALSE, for the same reason: every one of these is already running, +# installed by the deployment repo, and in a different shape than +# `make system-` would produce. `make system-clickhouse` would +# install a Bitnami ClickHouse release into `insight-infra` alongside the +# operator-managed one — two ClickHouses, one of them empty, and the +# umbrella still pointed at neither by accident. +# +# The umbrella does not need any of them installed by us; it only needs +# their addresses, which are the `.host` values in values.yaml. +system: + mariadb: false + clickhouse: false + redis: false + redpanda: false + redpandaConsole: false + airbyte: false + argoWorkflows: false + # Observability: this stand exports nothing over OTLP + # (`observability.otlp.endpoint` is left at the chart's empty default in + # values.yaml, i.e. structured JSON to stdout only). Flipping these on + # would self-host an LGTM stack that nothing points at. + loki: false + alloy: false + grafana: false + +# ─── Secrets (drives `make seal`) ────────────────────────────────────── +# +# NOTHING IS SEALED FOR THIS ENV, and nothing may become sealed without a +# deliberate decision, because every Secret this release consumes already +# exists in the cluster with generate-once semantics: +# +# insight-db-creds composed at bring-up from the +# credentials the ClickHouse / MariaDB / +# Redis OPERATORS generated in their own +# namespaces. Sealing a copy here and +# letting a controller apply it would +# overwrite those with values the +# datastores have never heard of, and +# every service loses its database. +# insight-authenticator-signing-keys the ES256 key the authenticator signs +# gateway JWTs with. Generated once. A new +# key invalidates every session and every +# JWT already in flight. +# insight-oidc the confidential OIDC client secret, +# shared with the realm. Re-materialising +# it breaks the token exchange until the +# realm is updated to match. +# insight-keycloak-admin bootstrap admin for the bundled Keycloak. +# insight-keycloak-config keycloak-config-cli login + the realm's +# external-IdP passthrough values. +# +# There is also no sealed-secrets CONTROLLER and no SealedSecret CRD on this +# cluster, so the manifests would not reconcile even if they were committed. +# +# This is more than documentation: `make deploy`'s `apply-app-secrets` +# prerequisite hard-requires at least one *-sealedsecret.yaml under +# `sealed-secrets/insight/` and aborts without one. That abort is why this +# env does not use `make deploy` today — see README.md, "Why not `make +# deploy`". +secrets: + # Empty on purpose, not omitted: there is no infra namespace to seal into. + infra: [] + services: + - {name: insight-db-creds, enabled: false} + - {name: insight-authenticator-signing-keys, enabled: false} + - {name: insight-oidc, enabled: false} + - {name: insight-keycloak-admin, enabled: false} + - {name: insight-keycloak-config, enabled: false} diff --git a/deploy/gitops/environments/test-stand/manifests/httproute-keycloak.yaml b/deploy/gitops/environments/test-stand/manifests/httproute-keycloak.yaml new file mode 100644 index 000000000..e7a0b097b --- /dev/null +++ b/deploy/gitops/environments/test-stand/manifests/httproute-keycloak.yaml @@ -0,0 +1,49 @@ +# /kc on the shared Gateway -> the chart's bundled Keycloak, for the test stand. +# +# The companion to httproute.yaml; read that file's header first — the same +# reasoning about why the route is hand-written, why it is committed, and why +# the deploy verifies rather than applies it holds here unchanged. +# +# WHY THE PREFIX WORKS WITHOUT A REWRITE +# Keycloak serves under /kc natively (the subchart passes +# --http-relative-path), and the same prefix is part of `keycloak.hostname` +# in ../values.yaml, which is what the discovery document advertises. So +# there is deliberately NO URLRewrite filter — adding one would strip a +# prefix the server expects and break discovery. +# +# WHY IT SHARES THE APPLICATION HOSTNAME +# Keycloak has no DNS record of its own. It rides the application's hostname +# and takes /kc from it, which is only safe because Gateway API gives the +# longest matching PathPrefix priority — the application route's "/" cannot +# swallow /kc. +# +# ORDERING NOTE for a recreate: Gateway API awards a contested path to the +# OLDEST route, so if a stale route ever claims /kc it must be deleted before +# this one is applied, not after. +# +# kubectl apply -f deploy/gitops/environments/test-stand/manifests/httproute-keycloak.yaml +# kubectl -n insight wait --for=condition=Accepted httproute/insight-keycloak +# +# Lives in the `insight` namespace so its backendRef stays namespace-local. +apiVersion: gateway.networking.k8s.io/v1 +kind: HTTPRoute +metadata: + name: insight-keycloak + namespace: insight +spec: + parentRefs: + - name: insight + namespace: envoy-gateway-system + sectionName: https + hostnames: + # Same hostname as the application route, on purpose — see above. + - insight-test.cfabric.org + rules: + - matches: + - path: + type: PathPrefix + value: /kc + backendRefs: + # The insight-keycloak subchart's Service, in this namespace. + - name: insight-keycloak + port: 8085 diff --git a/deploy/gitops/environments/test-stand/manifests/httproute.yaml b/deploy/gitops/environments/test-stand/manifests/httproute.yaml new file mode 100644 index 000000000..b387d44b3 --- /dev/null +++ b/deploy/gitops/environments/test-stand/manifests/httproute.yaml @@ -0,0 +1,63 @@ +# Cluster edge -> the Insight gateway, for the test stand. +# +# WHY THIS FILE EXISTS AT ALL +# The umbrella chart renders no HTTPRoute template. It renders an Ingress, +# which this stand cannot use: the edge is Envoy Gateway, which serves +# Gateway API routes and ignores Ingress objects. So `gateway.ingress.enabled` +# is false in ../values.yaml and the route has to be a hand-written object. +# If a later chart release learns Gateway API, prefer its template and retire +# this file rather than keeping two sources for one route. +# +# WHY IT IS COMMITTED HERE RATHER THAN LEFT ONLY IN THE BRING-UP REPO +# Everything the acceptance gate proves — a person logs in through the public +# URL and sees data — travels through this object. A values file that +# configures a gateway nothing routes to is not a deployable description of +# the stand. Committing it makes the request path reviewable in one place. +# +# WHY THE DEPLOY DOES NOT APPLY IT +# The route already exists on the stand and is Accepted, and it is applied +# today by the L0/L2 bring-up outside this repository. Two writers on one +# object is how a manual fix and an automated deploy start reverting each +# other. So the deploy VERIFIES this route (exists, Accepted, ResolvedRefs) +# and fails the run if it does not — it does not re-apply it. This file is +# the reviewed source for the day the route has to be recreated: +# +# kubectl apply -f deploy/gitops/environments/test-stand/manifests/httproute.yaml +# kubectl -n insight wait --for=condition=Accepted httproute/insight-gateway +# +# See ../README.md, "What this env does not own", for the boundary. +# +# ROUTING NOTE +# The Insight gateway is the single edge proxy for the application: it runs +# the cookie-to-JWT exchange against the authenticator and fans out to /api/* +# and the SPA. So this route sends everything ("/") to it and path routing +# happens inside the gateway, not here. The one exception is /kc, claimed by +# httproute-keycloak.yaml on the same shared Gateway — longest PathPrefix +# wins by Gateway API spec, so Keycloak traffic never reaches this backend. +# +# Lives in the `insight` namespace so the backendRef stays namespace-local +# (and so a namespace-scoped CI credential could re-apply it if the team ever +# decides the deploy should own it). +apiVersion: gateway.networking.k8s.io/v1 +kind: HTTPRoute +metadata: + name: insight-gateway + namespace: insight +spec: + parentRefs: + - name: insight + namespace: envoy-gateway-system + sectionName: https + hostnames: + # Must match `authenticator.oidc.redirectUri`, `authenticator.csrfOrigins` + # and `keycloak.hostname` in ../values.yaml. A mismatch here does not fail + # the deploy — it fails login, after the release reports success. + - insight-test.cfabric.org + rules: + - matches: + - path: + type: PathPrefix + value: / + backendRefs: + - name: insight-gateway + port: 8080 diff --git a/deploy/gitops/environments/test-stand/values.yaml b/deploy/gitops/environments/test-stand/values.yaml new file mode 100644 index 000000000..8e1365658 --- /dev/null +++ b/deploy/gitops/environments/test-stand/values.yaml @@ -0,0 +1,531 @@ +## +## Insight umbrella — test-stand overlay (insight-test.cfabric.org). +## +## The values file CI passes to `helm upgrade --install` after publishing a +## new umbrella chart from `main` (constructorfabric/insight#2244). It is +## also the file a human passes for a manual upgrade — there is no second, +## hidden copy anywhere. +## +## ───────────────────────────────────────────────────────────────────────── +## THE CONTRACT THIS FILE HAS TO KEEP +## ───────────────────────────────────────────────────────────────────────── +## An upgrade with this file must be semantically identical to the release +## already installed on the stand, EXCEPT for the chart version. That is not +## a nice-to-have: the first automated deploy runs against a stand that has +## real state (a Keycloak realm, generated signing keys, database +## credentials the operators own, a seeded tenant), and anything this file +## says differently from the live release is a change that lands the moment +## CI first fires — unreviewed, at merge speed, on a published stand. +## +## Every value below was reconciled against the running release's +## user-supplied values (`helm get values insight -n insight`). Where the +## originating deployment repo and the live release disagreed, the LIVE +## value wins and the difference is called out in a comment. +## +## Verify before changing anything here: +## helm get values insight -n insight -o yaml # what runs now +## make diff ENV=test-stand INSIGHT_VERSION= # render-vs-render +## +## ───────────────────────────────────────────────────────────────────────── +## WHAT THIS FILE DOES NOT CONTAIN +## ───────────────────────────────────────────────────────────────────────── +## No secrets, and no way to reconstruct one. The repository is public. +## Exactly one secret-shaped key exists in the chart's surface that this +## stand needs — `authenticator.oidc.clientSecret` — and it is left empty +## here on purpose; see the long note on it below for how the deploy +## supplies it. Everything else the release consumes arrives through +## `existingSecret` / `passwordSecret` references to Secrets that already +## live in the cluster. +## +## In-cluster Service DNS names and the tenant UUID DO appear below. They +## are configuration, not credentials: none of them is reachable or +## meaningful outside this cluster, and the chart cannot dial the +## datastores without them. +## + +# ═══════════════════════════════════════════════════════════════════════════ +# Credentials mode — deliberately NOT the gitops mode +# ═══════════════════════════════════════════════════════════════════════════ +# +# Every other environment in this tree runs `deploymentMode: gitops` + +# `autoGenerate: false`, because the gitops model pre-creates +# `insight-db-creds` from sealed secrets and composes the per-service config +# Secrets with scripts/compose-app-secrets.sh. +# +# This env keeps `helm` + `autoGenerate: true`, matching the installed +# release. Three independent reasons, any one of which is sufficient: +# +# 1. The chart wraps `insight-analytics-config`, +# `insight-authenticator-config` and `insight-identity-resolution-config` +# in `{{- if .Values.credentials.autoGenerate }}`. Flipping to gitops +# mode makes the chart stop emitting all three, so they are present in +# the previous release manifest and absent from the new one — i.e. Helm +# DELETES them mid-upgrade, and the services that mount them via +# `envFrom` fail to start. Recovering needs a `helm.sh/resource-policy: +# keep` annotation applied to all three BEFORE the upgrade. That is a +# migration, not a values change. +# +# 2. `autoGenerate: true` does not generate anything here. The chart reuses +# an existing `insight-db-creds` through `lookup`, which works because +# this stand is upgraded with `helm upgrade` (a live API server) and not +# rendered by a reconcile loop. The rotation hazard the chart's own +# values.yaml warns about is specific to `helm template`, where `lookup` +# returns nil — it does not apply to this deploy path. The chart's +# validator enforces the pairing in the other direction anyway: it +# refuses `gitops` + `autoGenerate: true`. +# +# 3. Under gitops mode the three config Secrets would instead be composed +# by scripts/compose-app-secrets.sh, which reads the OIDC client secret +# from `insight-oidc` under the key `oidc-client-secret`. The Secret on +# this stand carries that value under the key `client-secret`. The +# script's lookup would miss, fall back to the (correctly empty) +# `authenticator.oidc.clientSecret` below, and write a BLANK client +# secret into the authenticator's config — a confidential OIDC client +# with no secret, i.e. login broken on the next pod restart, with every +# pod still reporting healthy. +# +# Switch to gitops mode only as its own reviewed change, with the keep +# annotations applied first and the key-name mismatch resolved. +credentials: + deploymentMode: helm + autoGenerate: true + +# ═══════════════════════════════════════════════════════════════════════════ +# keycloak — the chart's bundled IdP (insight-keycloak subchart) +# ═══════════════════════════════════════════════════════════════════════════ +# +# The stack's own Keycloak (ADR-0003): a stateless `start` Deployment whose +# entire state lives in the MariaDB `keycloak` database — created and +# granted by the chart's `mariadb-init-svcdbs` hook Job precisely because +# `deploy: true` here — with realm content coming exclusively from the +# keycloakConfig Job below. It is published at /kc by +# manifests/httproute-keycloak.yaml in this directory, not by an Ingress. +keycloak: + deploy: true + # Advertised issuer base — the URL the browser AND the authenticator pod + # both resolve. MUST end in /kc: Keycloak 26's hostname-v2 does not fold + # `--http-relative-path` into the advertised issuer, so the prefix has to + # be part of the hostname the subchart passes. + hostname: "https://insight-test.cfabric.org/kc" + admin: + # Generate-once bootstrap admin, created by the L0/L2 bring-up outside + # this repo. Not sealed, not templated, not rotated by a deploy. + existingSecret: "insight-keycloak-admin" + database: + # The operator's single-writer Service, for the same reason as + # `mariadb.host` below — see that comment. + host: "mariadb-primary.mariadb.svc.cluster.local" + port: 3306 + name: "keycloak" + # The existing application user. The chart's init Job grants it the + # `keycloak` database, so no separate MariaDB user or credential exists + # for Keycloak — the password is the same `mariadb-password` already in + # `insight-db-creds`. + username: "insight" + passwordSecret: {name: "insight-db-creds", key: "mariadb-password"} + +# ═══════════════════════════════════════════════════════════════════════════ +# keycloakConfig — realm as code (ADR-0003) +# ═══════════════════════════════════════════════════════════════════════════ +# +# The chart's keycloak-config-cli post-install/post-upgrade hook Job applies +# whatever realm files are in the ConfigMap `-keycloak-config-realms` +# against the server above, with the import cache OFF — so the realm is +# re-imposed on every deploy and admin-console edits do not survive. +# +# OWNERSHIP, and why this env does not ship a `keycloak/realms/` directory: +# +# That ConfigMap is created OUTSIDE the Helm release by the deployment +# repo's bring-up script; the chart only reads it. Keeping it that way means +# an upgrade through this file re-applies the realm content that is already +# on the stand and changes nothing about login. +# +# The gitops Makefile has a competing mechanism — the `keycloak-broker-realms` +# target packs `environments//keycloak/realms/*` into that same +# ConfigMap name, with no placeholder rendering. Adding such a directory here +# would make two writers fight over one object: a manual bring-up and a CI +# deploy would each revert the other's realm, and whoever ran last would +# decide whether anyone can log in. Moving realm ownership into this tree is +# a deliberate follow-up (it needs a fully literal realm file and a decision +# about the external IdP's client credentials), not a side effect of adding +# this environment. +keycloakConfig: + enabled: true + # The subchart's in-release Service over plain HTTP; the /kc relative path + # is part of the URL. `allowInsecureUrl` is required for a non-https URL + # and is acceptable only because this hop never leaves the cluster network + # — TLS terminates at the edge for the public name. + url: "http://insight-keycloak:8085/kc" + allowInsecureUrl: true + # Composed at bring-up from Secrets in this namespace: the config-cli + # login (KEYCLOAK_USER / KEYCLOAK_PASSWORD), the confidential client + # secret, and the external IdP's OAuth credentials that the realm + # federates login to. The Job injects the tenant id itself from + # `global.tenantDefaultId`. + existingSecret: "insight-keycloak-config" + +# ═══════════════════════════════════════════════════════════════════════════ +# Global +# ═══════════════════════════════════════════════════════════════════════════ +global: + # The stand's single tenant. Not a credential — an identifier the deploy + # cannot work without: the chart's default is empty, and an empty tenant + # fails closed in the analytics metric-catalog resolver and in + # identity-resolution's bootstrap-admin seed. + # + # This UUID has to stay equal to THREE other places or the stand + # silently half-works: `ingestion.reconcile.tenantId` below, + # `authenticator.oidc.defaultTenantId` below, and the value the seeder + # discovers from the composed `insight-identity-resolution-config` Secret + # at seed time (src/ingestion/tools/seed/seed-stand.sh reads it from the + # cluster rather than taking a flag, so a drift here silently seeds a + # tenant nobody can log into). + tenantDefaultId: "3f1d8f4e-6c2a-4a9b-91d7-8e5c0b2a7f36" + # Not set, and worth knowing why: `observability.otlp.endpoint` is left at + # the chart's empty default, which means every service emits structured + # JSON to stdout and exports no OTLP. There is no collector on this stand + # to export to (all `system.*` toggles in inventory.yaml are false), and a + # non-empty endpoint pointed at nothing costs every pod a retry loop. + +# ═══════════════════════════════════════════════════════════════════════════ +# Datastores — all external, all operator-managed, none deployed by us +# ═══════════════════════════════════════════════════════════════════════════ +# +# The umbrella never runs a datastore; these four blocks are pure wiring — +# the host/port/database/username strings the app services and the chart's +# init Jobs dial. Passwords are NOT here: each subchart defaults its +# `passwordSecret` to a key of `insight-db-creds`, which the L0/L2 bring-up +# composed from the credentials the operators generated. +# +# There is no `deploy: false` on these blocks. The chart has no deploy +# switch for any datastore, so such a key would read like an off switch +# someone could flip and would in fact be ignored. + +clickhouse: + # A PER-POD Service, not the cluster-wide one, and that distinction is + # load-bearing. + # + # The chart is not cluster-aware: no template emits `ON CLUSTER`, there is + # no cluster-name value, and the objects it creates are + # ReplacingMergeTree / MergeTree / View rather than Replicated*, so nothing + # is synchronised between servers. Pointed at a Service that fans out + # across replicas, each new HTTP connection can land on a different server, + # so the post-install migration's DDL statements are distributed across + # independent servers that then disagree about which tables exist — the + # failure surfaces as ClickHouse's UNKNOWN_TABLE from whichever server + # answers a later statement. Airbyte writes and analytics reads would + # scatter the same way. + # + # `chi----` is the Altinity operator's + # per-pod Service, so this name always resolves to exactly one server. It + # is used even where the installation currently runs a single replica: the + # guarantee then lives in this file rather than in a replica count, so + # scaling out cannot silently reintroduce round-robin DDL. + # + # Revisit if the chart learns `ON CLUSTER` DDL and Replicated engines. + host: chi-clickhouse-clickhouse-0-0.clickhouse.svc.cluster.local + port: 8123 + database: insight + username: insight + +mariadb: + # The operator's `-primary` Service, not the all-members one. + # + # MariaDB runs here as a Galera cluster, which certifies every write + # across all members. Concurrent writers arriving on different members + # through a round-robin Service produce certification conflicts, which a + # schema-migration Job — many statements, one transaction each, no retry — + # surfaces as lock-wait/deadlock errors and a failed upgrade. The operator + # keeps `-primary` pointed at a single member and moves it on failover, so + # this address gives a single writer without pinning a pod name. + host: mariadb-primary.mariadb.svc.cluster.local + port: 3306 + database: insight + username: insight + +redis: + # The replication group's primary Service — NOT a Redis Cluster endpoint. + # + # The authenticator's Redis client is compiled without cluster support + # (src/backend/Cargo.toml enables only tokio-comp and connection-manager; + # the Rust `redis` crate feature-gates cluster behind "cluster"), and it + # opens a plain standalone connection. A clustered Redis answers MOVED for + # any key the contacted node does not own, and that client cannot follow + # the redirect — so every session operation whose key hashes to a slot + # owned by another node fails, and /auth/login returns 500. Sessions are + # not optional: a Redis the authenticator cannot address means nobody can + # log in, which is exactly what the smoke gate is there to catch. + # + # docs/components/backend/specs/DESIGN.md states the same contract + # ("Redis: Single instance; Sentinel for HA"), and every other environment + # in this tree points at a `redis-master` Service. + # + # `-master` selects on the operator's redis-role=master label, so it + # follows a failover without the client noticing. + host: redis-master.redis.svc.cluster.local + port: 6379 + +redpanda: + # ONE comma-separated `host:PORT` bootstrap string, not a host/port pair + # like the datastores above — the chart hands this to a Kafka client + # verbatim, so a bare hostname silently becomes host:9092 and never + # connects. 9093 is the internal Kafka API listener of the redpanda chart. + # Confirm the port on a stand with: + # kubectl -n redpanda get svc redpanda \ + # -o jsonpath='{range .spec.ports[*]}{.name}={.port}{"\n"}{end}' + brokers: redpanda.redpanda.svc.cluster.local:9093 + +# ═══════════════════════════════════════════════════════════════════════════ +# Ingestion +# ═══════════════════════════════════════════════════════════════════════════ +ingestion: + templates: + # The umbrella's Argo WorkflowTemplates / CronWorkflows. On this stand + # Argo Workflows is installed (outside this repo) and configured to watch + # this namespace, so the templates are useful here — unlike the local + # sandbox, where they are off because the controller only watches its own + # namespace. + # + # Note what this implies for a namespace rebuild: the templates pin + # `serviceAccountName: argo-workflow`, a ServiceAccount the chart + # references but does not create. It exists on the stand today, applied + # outside the release. See README.md, "What this env does not own". + enabled: true + reconcile: + # Must equal `global.tenantDefaultId`. Two copies of one UUID is a drift + # risk the chart does not check for us. + tenantId: "3f1d8f4e-6c2a-4a9b-91d7-8e5c0b2a7f36" + destinationName: clickhouse-bronze + # `-` — the instance id the Argo + # controller filters Workflows on. A wrong value here produces Workflows + # that are created and never picked up. + argoInstanceId: "argo-workflows-argo" + airbyteSync: + # How long a connector may sit idle before the sync loop treats it as + # stalled, in seconds. Chart default carried forward. + idleThresholdSeconds: 3600 + +airbyte: + # `namespace` does two things and only one of them is the URL: it also + # decides where the chart renders the Role/RoleBinding that lets the + # reconcile loop read Airbyte's own auth Secret. Left empty it defaults to + # the release namespace, so the RBAC would land in `insight`, where that + # Secret does not exist — and connector provisioning then fails at run time + # while every pod still looks healthy. + namespace: airbyte + # Deliberately empty: the chart computes + # http://-airbyte-server-svc..svc.cluster.local:8001 + # from the release name and the namespace above, which is the address on + # this stand. Set it only for a non-standard URL. + apiUrl: "" + +# ═══════════════════════════════════════════════════════════════════════════ +# Application services +# ═══════════════════════════════════════════════════════════════════════════ +# +# No image overrides anywhere in this file, on purpose. Each subchart renders +# `image.tag | default .Chart.AppVersion`, so a published chart version +# already names a coherent, tested set of images — which is the whole premise +# of deploying by chart version. Pinning one service's tag here would hold it +# on a different build from everything around it and make the deployed state +# unreadable from the chart version alone. + +analytics: + replicaCount: 1 # chart default is 2; one is enough for a test stand + resources: + requests: {cpu: 100m, memory: 128Mi} + limits: {cpu: 500m, memory: 512Mi} + +frontend: # the web UI (dashboard) + replicaCount: 1 + ingress: + # Correct, and also the chart's own default: the UI is never published + # directly. The gateway owns the only public route and proxies "/" to + # this Service, so a second entry point here would bypass the auth edge + # entirely. + enabled: false + +gateway: + replicaCount: 1 + ingress: + # OFF because the edge is Envoy Gateway, which serves Gateway API routes + # and not Ingress objects — and the umbrella renders no HTTPRoute + # template. Enabling this would render an Ingress that nothing serves. + # The route lives in manifests/httproute.yaml in this directory; the + # hostname pinned there must match `authenticator.oidc.redirectUri` + # below. + enabled: false + resources: + requests: {cpu: 25m, memory: 32Mi} + limits: {cpu: 500m, memory: 128Mi} + # No `gateway.routes` override. Overriding a list REPLACES the chart's + # table rather than merging into it, so any route a later chart release + # adds would silently stop being served — which is the opposite of what a + # deploy-the-newest-chart stand wants. Re-add only to CHANGE a route, never + # to restate the defaults, and diff against the gateway subchart's + # values.yaml first. ("/" and "/auth/*" are not in that table at all: the + # gateway generates them from its frontUrl and authenticatorUrl, which is + # why the UI and login work with no route entry.) + +# ═══════════════════════════════════════════════════════════════════════════ +# Authenticator — OIDC login, Redis sessions, gateway-JWT mint +# ═══════════════════════════════════════════════════════════════════════════ +authenticator: + replicaCount: 1 + + # Honour `/auth/login?__override=` — the session is minted for that + # person instead of the authenticated one. + # + # TRUE here because it is true on the installed release, and this file's + # job is to be equivalent to what runs. It is NOT endorsed: the chart's own + # values.yaml says this is for dev/demo environments only and "MUST stay + # false anywhere real users log in", and the service gates it on this flag + # alone — no role check, and it resolves an email regardless of tenant. On + # a stand reachable from the internet that is a standing impersonation + # primitive available to anyone who can complete a login. + # + # Changing it to false is a behaviour change to a published stand and does + # not belong in the change that introduces this environment; it is filed as + # a follow-up. If the smoke suite ends up depending on this flag to reach + # multiple personas, that dependency must be stated out loud rather than + # inherited quietly. + overrideEnabled: true + + # Origins accepted for state-changing requests. Exactly the public origin — + # the same host as `oidc.redirectUri` and the HTTPRoute hostname. + csrfOrigins: + - "https://insight-test.cfabric.org" + + tlsDiscovery: + enabled: true + issuerRef: + # Must name the ClusterIssuer the cluster's cert-manager actually has; + # the bring-up outside this repo validates that the two agree. A typo + # here leaves the authn-tls sidecar without a certificate and the + # authenticator unreachable from the gateway. + name: insight-ca + kind: ClusterIssuer + + oidc: + # The bundled Keycloak's broker realm, served at /kc on the public + # hostname. The authenticator resolves this URL ITSELF at startup for + # discovery, so the name has to be reachable from inside the cluster as + # well as from the browser — which is why it is the public name and not + # an in-cluster Service address. + issuerUrl: "https://insight-test.cfabric.org/kc/realms/insight-broker" + clientId: "insight-authenticator" + + # ───────────────────────────────────────────────────────────────────── + # EMPTY ON PURPOSE — and the deploy MUST supply it. + # ───────────────────────────────────────────────────────────────────── + # This is the confidential client's secret. The repository is public, so + # it cannot live here, and it must not be reconstructible from anything + # here. + # + # It is NOT optional: with `credentials.autoGenerate: true` the chart + # writes this value straight into the authenticator's config Secret. An + # upgrade that leaves it empty produces a confidential OIDC client with a + # blank secret — the pods stay Ready, the release reports `deployed`, and + # the authorization-code exchange fails for every login. + # + # The deploy passes it at apply time, from the Secret that already holds + # it in this namespace: + # + # --set-string authenticator.oidc.clientSecret="$(kubectl -n insight \ + # get secret insight-oidc -o jsonpath='{.data.client-secret}' \ + # | base64 -d)" + # + # Reading it out of the cluster rather than out of a GitHub secret is + # deliberate: the cluster copy is the one the realm was configured with, + # so there is exactly one source of truth and no second copy to rotate. + # It costs the CI credential a `get` on this one Secret. If that grant is + # unacceptable, the alternative is a value in the `insight-test-stand` + # GitHub environment — but then rotating the realm's client secret means + # rotating two places, and forgetting one breaks login silently. + clientSecret: "" + + redirectUri: "https://insight-test.cfabric.org/auth/callback" + + # `openid` alone is correct here and is not an oversight: the broker + # client is `fullScopeAllowed: false` with a fixed set of default scopes, + # so the claims the login bootstrap needs arrive without being requested. + scopes: ["openid"] + + # How a logged-in principal is resolved to a person row. The login + # bootstrap calls + # GET /internal/persons/by-external-id?source_type= + # &external_id= + # and fails CLOSED when there is no match — there is no email fallback, + # by design. + # + # These two values are GLOBAL to the authenticator: the per-host issuer + # map (ADR-0003) can override only issuerUrl / clientId / clientSecret / + # redirectUri / defaultTenantId, never these. So every realm this + # deployment serves must key its people the same way, which is the + # constraint any change to how CI logs in has to satisfy. + sourceType: "github" + externalIdClaim: "idp_sub" + + # Fallback tenant for an id_token without a tenant claim. Must equal + # `global.tenantDefaultId`; empty would fail closed. + defaultTenantId: "3f1d8f4e-6c2a-4a9b-91d7-8e5c0b2a7f36" + +# ═══════════════════════════════════════════════════════════════════════════ +# Identity Resolution — the only identity service, plus its two CronJobs +# ═══════════════════════════════════════════════════════════════════════════ +# +# There is no `identity:` block anywhere in this file. The .NET identity +# service was removed from the umbrella; `identity.*` keys are no longer read +# by anything, so one written here would be silently ignored rather than +# rejected. +identityResolution: + deploy: true + databaseName: "identity" + + # Rebuilds the identity org projection — `persons`, `account_person_map`, + # `org_chart` — in MariaDB from the observation history in ClickHouse. + seed: + enabled: true + schedule: "30 6 * * *" # chart default; after the overnight syncs + # Trimmed from the chart's 100m/128Mi request: these are transient Jobs + # and two of them fire within fifteen minutes of each other. Limits stay + # at the chart's ceiling so a real run can burst. + resources: + requests: {cpu: 50m, memory: 128Mi} + limits: {cpu: 500m, memory: 512Mi} + # No `tenantDefaultId` here: for an umbrella install the composed + # identity-resolution config Secret already carries the tenant from + # `global.tenantDefaultId`. Setting it here would add a fourth copy of + # the same UUID that can drift; the key exists for standalone installs + # whose Secret does not carry one. + + # Publishes that MariaDB persons log into ClickHouse, where the metrics + # models resolve email -> person_id against it. Deliberately 15 minutes + # after the seed: the seed rewrites the log, the sync publishes it. Each + # run is a full snapshot plus an atomic swap, so the ordering is a + # freshness concern, never a correctness one. + sync: + enabled: true + schedule: "45 6 * * *" # chart default; 15 min after seed + resources: + requests: {cpu: 50m, memory: 128Mi} + limits: {cpu: 500m, memory: 512Mi} + + # Both Jobs guard their input and exit non-zero rather than publishing an + # empty projection: seed aborts on an empty observation read or a tenant + # mismatch, sync on an empty persons log. On a stand with no connector data + # they therefore fail by design, and a failing seed Job is the guard + # working rather than a broken install. Force a run by hand with: + # kubectl -n insight create job --from=cronjob/insight-identity-resolution-seed seed-manual + # kubectl -n insight create job --from=cronjob/insight-identity-resolution-sync sync-manual + # + # OPEN RISK for a CI-seeded stand, stated here because this is where + # somebody will look: the seed CronJob rebuilds `persons` from ClickHouse + # observations, and the fixture seeder does not write those observations. + # While the observation table is empty the Job's guard fires and the + # seeded rows survive. Once anything populates that table, the rebuild + # would replace `persons` and drop the seeder's login-bootstrap rows — + # un-seeding logins between merges, overnight, with no signal until the + # next smoke run. Suspending these two CronJobs on this stand is a values + # change (`enabled: false`) and a deliberate decision, not something to do + # by reflex. diff --git a/deploy/gitops/scripts/emulate-ci-deploy.sh b/deploy/gitops/scripts/emulate-ci-deploy.sh new file mode 100755 index 000000000..dc1f509bd --- /dev/null +++ b/deploy/gitops/scripts/emulate-ci-deploy.sh @@ -0,0 +1,1200 @@ +#!/usr/bin/env bash +# +# emulate-ci-deploy.sh — run, from a laptop, the same three stages that +# .github/workflows/deploy-test-stand.yml runs after the umbrella chart is +# published: deploy -> seed -> smoke. +# +# WHY THIS EXISTS +# +# The test-stand workflow only ever fires on a merge to main. That makes it the +# worst possible place to find out that the gitops environment is wrong: the +# feedback loop is "merge, wait, read a redacted public log, guess, merge +# again", and every iteration leaves a red X on somebody's merge commit. This +# script closes that loop before the workflow is ever enabled — and keeps it +# closed afterwards, because when CI does go red the first question is always +# "does it reproduce by hand?" and the only answer worth anything is one +# produced by the SAME commands. +# +# So the contract of this file is narrow and deliberate: it does not invent a +# deploy path. Each stage runs the invocation the workflow's matching step runs, +# prints it before running it, and refuses to run any of them against the wrong +# cluster. Everything else — richer output, retries, cleanup, convenience +# defaults CI does not have — was considered and rejected: every divergence +# between this script and the workflow is a way for a green local run to lie. +# +# THE THREE STAGES, AND WHERE THEY COME FROM +# +# 1. deploy `helm upgrade --install` spelled out, with the OIDC client +# secret read out of the cluster, `--wait`, `--timeout 10m`, and +# deliberately no `--atomic`; then three checks the upgrade's exit +# status cannot answer — that the live release is the chart this +# run asked for, that the three envFrom-configured services were +# restarted, and that the two edge routes are still Accepted. +# NOT `make deploy`: `deploy-insight` chains apply-app-secrets, +# which hard-requires a sealed manifest this stand has no +# controller for and then rewrites the chart's config Secrets from +# a key name this stand does not use, and it hardcodes `--atomic` +# after $(HELM_UPGRADE_FLAGS) so no knob turns the rollback off. +# `make diff` DOES work here unchanged, which is why it is this +# stage's read-only form. +# 2. seed `seed-stand.sh -n insight --context … --email … --days 730`, +# verbatim. No wrapper: the seeder discovers the tenant, the +# datastore coordinates and the IdP source type (there is no +# --auth-mode any more — it was folded into --idp-source-type, +# which is itself read off the release) +# from the cluster, and a value supplied from outside is a value +# that can be wrong while looking right. +# 3. smoke `uv run --project tests --frozen pytest tests/stand/smoke +# --stand-manifest ` against the public URL. Real DNS, +# real TLS, real IdP redirect — a stand that works only from +# inside the cluster is a stand nobody can use. +# +# READ-ONLY BY DEFAULT +# +# The target is a published stand that other people look at. A script that +# deploys by default is a script that deploys by accident, so the default mode +# renders and reports and touches nothing: +# +# deploy -> `make diff` (helm template plus a diff against the last render; +# contacts the OCI registry and the local git tree, never the +# cluster), then read-only checks of the two objects the upgrade +# depends on and does not own. +# seed -> `seed-stand.sh --dry-run`, which performs the SAME cluster +# discovery the real run performs and prints the Job it would +# apply. Read-only, and a genuine RBAC probe. +# smoke -> one unauthenticated GET of the public URL, then +# `pytest --collect-only`. +# +# Mutating the stand requires BOTH `--apply` AND `--i-know-this-deploys +# yes-deploy-`. Two flags rather than one because a single `--apply` is +# exactly the kind of thing that ends up in a shell history and gets recalled +# with the wrong `--kubeconfig` still on the line. The token spells out the +# environment, so recalling that line for a different stand fails closed. It is +# deliberately the same string the Makefile's protected-environment CONFIRM= +# uses, so there is one token to remember rather than two. +# +# THE CLUSTER GUARD +# +# `--kubeconfig` and `--expect-cluster` are both required, in every mode, +# including dry-run. The guard is the whole point of the file: this +# organisation runs more than one Insight stand, at least one of which must +# never be touched by this tooling, and a kube-context name is a local alias +# anybody can typo into agreement. So before anything runs we assert, in order, +# all read-only: +# +# 1. the gitops inventory's `kubeContext` equals the kubeconfig's +# current-context — the same assertion the workflow makes. It matters more +# than it looks: nothing on this path passes helm an explicit +# `--kube-context`, so the ambient context IS the target; +# 2. the cluster ENTRY that context points at is named exactly +# `--expect-cluster`; +# 3. optionally, that the sha256 of that cluster's API server URL equals +# `--expect-api-server-sha256`. +# +# (3) is optional but is the only check that binds to the cluster rather than to +# names in a file, which is why `--apply` without it prints a warning naming the +# residual risk. It takes a digest rather than the URL itself so the value is +# safe to keep in a GitHub environment variable or a team note: this repo is +# public and a cluster API endpoint is not ours to publish. +# +# WHAT THIS SCRIPT DELIBERATELY DOES NOT DO +# +# * It does not create, seal, rotate, or print any credential. The OIDC client +# secret is read out of the cluster into a variable that exists for the +# length of one helm call; the printed command shows the substitution +# expression, never its result. Every SMOKE_* login variable arrives from the +# caller's environment untouched, and the smoke suite validates them itself. +# * It does not roll back, retry, or clean up. A failed deploy is left exactly +# where it failed so the evidence survives; recovery is the next merge or a +# deliberate `make rollback ENV=test-stand`. +# * It does not apply the HTTPRoutes. They are owned outside this repository; +# applying them from here would make a second writer on one object. +# * It does not print its own failure diagnostics. Those belong to the +# workflow's stand-diagnostics.sh, a curated redacted allowlist, and reusing +# it by path is what keeps a laptop run from showing evidence the red CI run +# will not have. +# * It does not pipe stage output through redact-stand-log.py. CI must, because +# its console is public forever; a laptop's is not, and redacting the one copy +# you are debugging from removes the detail you are debugging. This is one of +# the deliberate differences from CI — all of them are enumerated in +# docs/components/deployment/gitops/ci-emulation.md. +# +# Usage: emulate-ci-deploy.sh --kubeconfig PATH --expect-cluster NAME [options] +# Run with --help for the option list, or --print-commands to read the whole +# plan without running anything. + +set -euo pipefail + +SCRIPT_DIR="$(cd "$(dirname "${BASH_SOURCE[0]}")" && pwd)" +GITOPS_DIR="$(cd "$SCRIPT_DIR/.." && pwd)" +REPO_ROOT="$(cd "$GITOPS_DIR/../.." && pwd)" + +# The gitops environment this harness drives. Overridable so the same script can +# rehearse a second stand later, but the default is the one the workflow +# hardcodes — a harness whose default target differs from CI's rehearses the +# wrong thing. +GITOPS_ENV="test-stand" + +WORKFLOW_REL=".github/workflows/deploy-test-stand.yml" +WORKFLOW_FILE="$REPO_ROOT/$WORKFLOW_REL" +SEED_REL="./src/ingestion/tools/seed/seed-stand.sh" +SEED_SCRIPT="$REPO_ROOT/src/ingestion/tools/seed/seed-stand.sh" +SMOKE_SUITE="tests/stand/smoke" + +# The umbrella chart, spelled exactly as the workflow's CHART_REF spells it. +CHART_REF="oci://ghcr.io/constructorfabric/charts/insight" + +# Failure diagnostics belong to the workflow. Overridable with --diagnostics or +# $INSIGHT_STAND_DIAGNOSTICS, but there is one canonical home and probing for +# others would only hide a rename. +DIAGNOSTICS_DEFAULT_REL=".github/workflows/scripts/stand-diagnostics.sh" + +# Artefacts land under the gitops .deploy/ directory, which deploy/gitops/ +# .gitignore already excludes. That is not tidiness: `make diff` depends on +# `sync-clean`, which fails on ANY file `git status --porcelain` reports, so a +# harness that wrote its captured seed log into the working tree would break the +# very stage it is trying to rehearse. The same reasoning is why --kubeconfig is +# refused when it points inside the tree. +ARTIFACT_DIR="$GITOPS_DIR/.deploy/ci-emulation" + +# The dev-lead persona the seeder binds `--email` to. In CI this arrives from the +# environment secret TEST_STAND_SEED_EMAIL; locally the seeder's own committed +# canonical value (src/ingestion/tools/seed/PROFILE.md), on a deliberately +# non-routable domain, is a safe default that addresses the person the roster +# describes. +SEED_DEV_EMAIL_DEFAULT="email_development_lead@company.nonpresent" + +# The seed window. `--days`, NOT `--window-days`: seed-stand.sh's argument parser +# has no such flag and refuses unknown arguments, so the wrong spelling aborts +# the run before a single row is written. +SEED_WINDOW_DAYS="730" + +# helm's own budget for the upgrade, matching the workflow. Not the Makefile's +# 30m default: a deploy that is going to fail should say so inside the stage +# budget. +DEPLOY_TIMEOUT="10m" + +# The three deployments whose configuration arrives through `envFrom` and is +# therefore read exactly once, at container start. Each subchart's +# `checksum/config` annotation hashes its OWN ConfigMap and not the +# umbrella-rendered insight-*-config Secrets, so without the restart a changed +# host, tenant or client secret produces a `deployed` release with all-Ready pods +# running stale configuration. All three on purpose: a list of two leaves one +# service holding yesterday's configuration with nothing to show for it. +RESTART_TARGETS="deploy/insight-authenticator deploy/insight-analytics deploy/insight-identity-resolution" + +# The edge objects the chart does not render and this release does not own, but +# every acceptance criterion travels through. +ROUTE_NAMES="insight-gateway insight-keycloak" + +KUBECONFIG_PATH="" +EXPECT_CLUSTER="" +ALLOW_DIRTY=0 +EXPECT_API_SHA="" +AS_USER="" +STAGE="all" +CHART_VERSION="" +BASE_URL="" +SEED_DEV_EMAIL="" +MANIFEST_PATH="" +DIAGNOSTICS_SCRIPT="${INSIGHT_STAND_DIAGNOSTICS:-}" +APPLY=0 +CONFIRM_TOKEN="" +PRINT_ONLY=0 + +usage() { + cat <<'USAGE' +Usage: emulate-ci-deploy.sh --kubeconfig PATH --expect-cluster NAME [options] + +Runs the three stages of .github/workflows/deploy-test-stand.yml from a laptop, +against a stand, through an explicit kubeconfig. Read-only unless --apply. + +Required: + --kubeconfig kubeconfig FILE to act through. Deliberately not the + ambient $KUBECONFIG: the guard rests on this being a + value you typed for this run. Must live OUTSIDE the + repository working tree — `make diff` depends on + sync-clean. + --expect-cluster + the kubeconfig CLUSTER entry the current context + must point at. Refuses loudly on mismatch. + +Safety: + --apply actually mutate the stand. Without it, every stage + runs its read-only form. + --allow-dirty read-only deploy stage only: when the working tree is + dirty, render with `helm template` through the same + chart/version/values instead of `make diff`, which + refuses any uncommitted file. For rehearsing changes + that are not committed yet — a real deploy always + renders from a clean checkout, and the run says so. + --i-know-this-deploys + required with --apply. Must be exactly + yes-deploy- (default: yes-deploy-test-stand). + --expect-api-server-sha256 + optional, recommended with --apply: sha256 of the + cluster's API server URL. The only check that binds + to the cluster rather than to names in a file. + --as-user run the read-only RBAC probe as this user (e.g. the + CI ServiceAccount) instead of as you. Probe only: + the stages always run as the kubeconfig's identity. + +Selection: + --stage deploy | seed | smoke | all [default: all] + --env gitops environment directory [default: test-stand] + --chart-version umbrella version to deploy — the value CI hands over + from publish-chart. + [default: deploy/gitops/.insight-version] + --timeout helm --timeout for the upgrade [default: 10m] + +Seed and smoke inputs: + --seed-email persona the seeder binds the dev-lead login to. + [default: $TEST_STAND_SEED_EMAIL, else the seeder's + committed canonical dev-lead address] + --manifest seed manifest for pytest. [default: captured out of + the seed stage into the artefact directory] + --base-url the stand's public URL. [default: $SMOKE_BASE_URL, + else derived from the committed values.yaml] + +Other: + --diagnostics failure-diagnostics script to reuse. + [default: .github/workflows/scripts/stand-diagnostics.sh] + --print-commands print the whole plan and exit, running nothing — not + even the cluster guard. + -h, --help this text + +Credentials this script never supplies and never prints. Export them yourself; +in CI they come from the insight-test-stand GitHub environment: + SMOKE_LOGIN_MODE password | override + SMOKE_PERSONA_PASSWORD password mode + (SMOKE_PERSONA_PASSWORD__ overrides one persona) + SMOKE_BOOTSTRAP_EMAIL + SMOKE_BOOTSTRAP_PASSWORD override mode +The smoke suite resolves and validates these itself and names the missing one. + +Examples: + # read the plan without touching anything + emulate-ci-deploy.sh --print-commands + + # read-only rehearsal of everything against the live stand + emulate-ci-deploy.sh --kubeconfig ~/.kube/stand.yaml --expect-cluster my-cluster + + # rehearse the seed stage's discovery as the CI ServiceAccount + emulate-ci-deploy.sh --kubeconfig ~/.kube/stand.yaml --expect-cluster my-cluster \ + --stage seed --as-user system:serviceaccount:insight:ci-deployer + + # the real thing, as CI would run it for a freshly published chart + emulate-ci-deploy.sh --kubeconfig ~/.kube/stand.yaml --expect-cluster my-cluster \ + --chart-version 0.5.101 --apply --i-know-this-deploys yes-deploy-test-stand +USAGE +} + +die() { + printf 'ERROR: %s\n' "$*" >&2 + exit 1 +} + +# Distinct from die() on purpose: a guard rejection is not a bug in the run, it +# is the guard doing its job, and it must be impossible to miss in a scrollback +# full of helm output. +refuse() { + printf '\n' >&2 + printf '%s\n' '================================================================' >&2 + printf 'REFUSING TO ACT: %s\n' "$1" >&2 + shift + while [ $# -gt 0 ]; do + printf ' %s\n' "$1" >&2 + shift + done + printf '%s\n' '================================================================' >&2 + exit 1 +} + +need() { + command -v "$1" >/dev/null 2>&1 \ + || die "$1 is required but not on PATH (see: make -C deploy/gitops doctor)." +} + +note() { printf ' %s\n' "$*"; } +head1() { printf '\n%s\n' "== $* =="; } +head2() { printf '\n%s\n' "-- $* --"; } + +# Print a command exactly as it will be run, in a form that can be pasted back +# into a shell. This is half the value of the whole script: the printed line is +# the evidence that the local path and the CI path are the same commands, and it +# is what a reviewer diffs against the workflow YAML. +show_cmd() { + local prefix="$1" + shift + printf ' $ %s%s\n' "$prefix" "$(printf '%q ' "$@")" +} + +sha256_of() { + if command -v sha256sum >/dev/null 2>&1; then + printf '%s' "$1" | sha256sum | awk '{print $1}' + elif command -v shasum >/dev/null 2>&1; then + printf '%s' "$1" | shasum -a 256 | awk '{print $1}' + else + die "neither sha256sum nor shasum is on PATH; --expect-api-server-sha256 cannot be checked." + fi +} + +while [ $# -gt 0 ]; do + case "$1" in + --kubeconfig) KUBECONFIG_PATH="${2:?--kubeconfig needs a value}"; shift 2 ;; + --expect-cluster) EXPECT_CLUSTER="${2:?--expect-cluster needs a value}"; shift 2 ;; + --allow-dirty) ALLOW_DIRTY=1; shift ;; + --expect-api-server-sha256) + EXPECT_API_SHA="${2:?--expect-api-server-sha256 needs a value}"; shift 2 ;; + --as-user) AS_USER="${2:?--as-user needs a value}"; shift 2 ;; + --stage) STAGE="${2:?--stage needs a value}"; shift 2 ;; + --env) GITOPS_ENV="${2:?--env needs a value}"; shift 2 ;; + --chart-version) CHART_VERSION="${2:?--chart-version needs a value}"; shift 2 ;; + --timeout) DEPLOY_TIMEOUT="${2:?--timeout needs a value}"; shift 2 ;; + --base-url) BASE_URL="${2:?--base-url needs a value}"; shift 2 ;; + --seed-email) SEED_DEV_EMAIL="${2:?--seed-email needs a value}"; shift 2 ;; + --manifest) MANIFEST_PATH="${2:?--manifest needs a value}"; shift 2 ;; + --diagnostics) DIAGNOSTICS_SCRIPT="${2:?--diagnostics needs a value}"; shift 2 ;; + --apply) APPLY=1; shift ;; + --i-know-this-deploys) CONFIRM_TOKEN="${2:?--i-know-this-deploys needs a value}"; shift 2 ;; + --print-commands) PRINT_ONLY=1; shift ;; + -h|--help) usage; exit 0 ;; + *) usage >&2; die "unknown argument: $1" ;; + esac +done + +case "$STAGE" in + deploy|seed|smoke|all) ;; + *) die "--stage must be one of deploy, seed, smoke, all (got '$STAGE')." ;; +esac + +CONFIRM_EXPECTED="yes-deploy-${GITOPS_ENV}" +ENV_REL="deploy/gitops/environments/$GITOPS_ENV" +ENV_DIR="$REPO_ROOT/$ENV_REL" +INVENTORY="$ENV_DIR/inventory.yaml" +VALUES_REL="$ENV_REL/values.yaml" +VALUES="$REPO_ROOT/$VALUES_REL" + +need yq +need helm +need kubectl +need jq +need make + +[ -n "$SEED_DEV_EMAIL" ] || SEED_DEV_EMAIL="${TEST_STAND_SEED_EMAIL:-$SEED_DEV_EMAIL_DEFAULT}" + +# ── The environment the harness drives ────────────────────────────────────── +# Read from the committed gitops environment rather than taken as flags: the +# stand's configuration lives in the repo, and a harness that accepted the +# namespace and release on the command line could rehearse a topology the +# workflow will never deploy. The workflow pins STAND_NAMESPACE / STAND_RELEASE +# to the same values and reads kubeContext out of the same inventory. +# +# --print-commands is the one mode that tolerates a missing environment: it +# exists so a reviewer can read the contract before the environment lands, and +# it substitutes obviously-fake placeholders rather than plausible ones. +if [ -f "$INVENTORY" ]; then + KUBE_CTX="$(yq -r '.kubeContext // ""' "$INVENTORY")" + NS_APP="$(yq -r '.namespaces.services // "insight"' "$INVENTORY")" + RELEASE="$(yq -r '.release // "insight"' "$INVENTORY")" + [ -n "$KUBE_CTX" ] && [ "$KUBE_CTX" != "null" ] \ + || die "$INVENTORY has no kubeContext; every guard in this script, in the workflow and in the Makefile keys off it." +elif [ "$PRINT_ONLY" -eq 1 ]; then + KUBE_CTX="" + NS_APP="" + RELEASE="" +else + die "no gitops inventory at $INVENTORY — the '$GITOPS_ENV' environment has not landed in this checkout yet." +fi + +if [ ! -f "$VALUES" ] && [ "$PRINT_ONLY" -eq 0 ]; then + die "no values file at $VALUES — the '$GITOPS_ENV' environment is incomplete." +fi + +if [ -z "$CHART_VERSION" ]; then + CHART_VERSION="$(cat "$GITOPS_DIR/.insight-version" 2>/dev/null || true)" + [ -n "$CHART_VERSION" ] \ + || die "no --chart-version, and deploy/gitops/.insight-version is empty. In CI this value is the publish-chart job's output; locally you must name it." +fi + +# ── Where the stand answers ───────────────────────────────────────────────── +# Derived from the committed values file so a local smoke run addresses the same +# origin the deployed authenticator redirects back to. Deriving beats a hardcoded +# constant twice over: the URL is never written into this script, and a values +# file that changes hosts moves the smoke target with it instead of silently +# testing the old one. (CI takes it from a repository variable instead — the same +# address arriving by a different route.) +resolve_base_url() { + local candidate + [ -f "$VALUES" ] || return 1 + + # The authenticator's registered redirect is by definition an absolute URL on + # the stand's public origin, and always ends in the callback path. + candidate="$(yq -r '.authenticator.oidc.redirectUri // ""' "$VALUES")" + if [ -n "$candidate" ] && [ "$candidate" != "null" ]; then + candidate="${candidate%/auth/callback}" + printf '%s' "${candidate%/}" + return 0 + fi + + # Fallback: the bundled Keycloak's hostname is the same origin with the realm + # prefix appended. + candidate="$(yq -r '.keycloak.hostname // ""' "$VALUES")" + if [ -n "$candidate" ] && [ "$candidate" != "null" ]; then + candidate="${candidate%/kc}" + printf '%s' "${candidate%/}" + return 0 + fi + + return 1 +} + +[ -n "$BASE_URL" ] || BASE_URL="${SMOKE_BASE_URL:-}" +[ -n "$BASE_URL" ] || BASE_URL="$(resolve_base_url || true)" + +mkdir -p "$ARTIFACT_DIR" +ARTIFACT_REL="${ARTIFACT_DIR#"$REPO_ROOT"/}" +RUN_STAMP="$(date -u +%Y%m%d-%H%M%S)" +SEED_LOG="$ARTIFACT_DIR/seed-$RUN_STAMP.log" +SEED_LOG_REL="${SEED_LOG#"$REPO_ROOT"/}" +[ -n "$MANIFEST_PATH" ] || MANIFEST_PATH="$ARTIFACT_DIR/seed-manifest.json" +MANIFEST_REL="${MANIFEST_PATH#"$REPO_ROOT"/}" + +# ── Command definitions ───────────────────────────────────────────────────── +# One array per command, defined once, printed before use and never rebuilt +# inline — so what the script announces and what the script runs cannot drift, +# and so --print-commands can show the whole plan without executing it. +# +# The one command NOT fully materialised here is the helm upgrade: its +# --set-string carries the OIDC client secret, which is read out of the cluster +# at exec time and must never reach the terminal. The display form shows the +# substitution expression instead — which is also what documents where the value +# comes from. + +restart_args=() +for _target in $RESTART_TARGETS; do + restart_args+=("$_target") +done +unset _target + +route_args=() +for _route in $ROUTE_NAMES; do + route_args+=("$_route") +done +unset _route + +deploy_helm_cmd=( + helm upgrade --install "$RELEASE" "$CHART_REF" + --version "$CHART_VERSION" + --namespace "$NS_APP" + --values "$VALUES_REL" + --wait --timeout "$DEPLOY_TIMEOUT" + --history-max 10 +) + +# shellcheck disable=SC2016 # the command substitution is displayed, never run +deploy_helm_secret_display='--set-string authenticator.oidc.clientSecret="$(kubectl -n '"$NS_APP"' get secret insight-oidc -o '"'"'jsonpath={.data.client-secret}'"'"' | base64 --decode)"' + +deploy_verify_cmd=( + helm list -n "$NS_APP" + --deployed --failed --pending --uninstalling + --filter "^${RELEASE}\$" -o json +) + +deploy_restart_cmd=(kubectl -n "$NS_APP" rollout restart "${restart_args[@]}") +deploy_rollout_cmd=(kubectl -n "$NS_APP" rollout status --timeout=5m "${restart_args[@]}") + +route_table_cmd=( + kubectl -n "$NS_APP" get httproute "${route_args[@]}" + -o 'custom-columns=NAME:.metadata.name,ACCEPTED:.status.parents[0].conditions[?(@.type=="Accepted")].status' +) +route_status_cmd=( + kubectl -n "$NS_APP" get httproute "${route_args[@]}" + -o 'jsonpath={range .items[*]}{.metadata.name}{"="}{.status.parents[0].conditions[?(@.type=="Accepted")].status}{"\n"}{end}' +) + +deploy_cmd_dry=(make diff "ENV=$GITOPS_ENV" "INSIGHT_VERSION=$CHART_VERSION") + +# The --allow-dirty render. Same chart, same version, same values file as the +# upgrade above; no cluster contact, no clean-tree assertion. +deploy_render_cmd=( + helm template "$RELEASE" "$CHART_REF" + --version "$CHART_VERSION" + --namespace "$NS_APP" + --values "$VALUES_REL" +) + +seed_cmd_apply=( + "$SEED_REL" + -n "$NS_APP" + --context "$KUBE_CTX" + --email "$SEED_DEV_EMAIL" + --days "$SEED_WINDOW_DAYS" +) + +seed_cmd_dry=("${seed_cmd_apply[@]}" --dry-run) + +smoke_cmd_apply=( + uv run --project tests --frozen + pytest "$SMOKE_SUITE" --stand-manifest "$MANIFEST_PATH" +) + +smoke_cmd_dry=("${smoke_cmd_apply[@]}" --collect-only -q) + +print_plan() { + head1 "stage commands" + note "Diff these against $WORKFLOW_REL; they must match." + note "Stage 1's dry-run form runs from deploy/gitops; everything else runs" + note "from the repository root." + + head2 "stage 1 - deploy" + note "apply:" + show_cmd "KUBECONFIG= " "${deploy_helm_cmd[@]}" + printf ' %s\n' "$deploy_helm_secret_display" + note "then, because a release can be 'deployed' and still be the wrong chart:" + show_cmd "KUBECONFIG= " "${deploy_verify_cmd[@]}" + note "then, because envFrom configuration is read once at container start:" + show_cmd "KUBECONFIG= " "${deploy_restart_cmd[@]}" + show_cmd "KUBECONFIG= " "${deploy_rollout_cmd[@]}" + note "then, because the chart renders no route at all:" + show_cmd "KUBECONFIG= " "${route_table_cmd[@]}" + note "dry-run substitute for the upgrade (offline render and diff):" + show_cmd "" "${deploy_cmd_dry[@]}" + + head2 "stage 2 - seed" + note "apply:" + show_cmd "KUBECONFIG= " "${seed_cmd_apply[@]}" + note "dry-run:" + show_cmd "KUBECONFIG= " "${seed_cmd_dry[@]}" + + head2 "stage 3 - smoke" + note "apply:" + show_cmd "SMOKE_BASE_URL=${BASE_URL:-<--base-url>} " "${smoke_cmd_apply[@]}" + note "dry-run:" + show_cmd "SMOKE_BASE_URL=${BASE_URL:-<--base-url>} " "${smoke_cmd_dry[@]}" + printf '\n' +} + +if [ "$PRINT_ONLY" -eq 1 ]; then + print_plan + exit 0 +fi + +# ── Argument validation that only matters for a run ───────────────────────── + +[ -n "$KUBECONFIG_PATH" ] || { usage >&2; die "--kubeconfig is required."; } +[ -n "$EXPECT_CLUSTER" ] || { usage >&2; die "--expect-cluster is required. It is the guard that stops this run reaching a stand it was not aimed at."; } +[ -f "$KUBECONFIG_PATH" ] || die "kubeconfig not found at $KUBECONFIG_PATH." + +case "$KUBECONFIG_PATH" in + "$REPO_ROOT"/*) + die "the kubeconfig is inside the repository working tree ($KUBECONFIG_PATH). \`make diff\` depends on sync-clean, which fails on any file git reports — keep the kubeconfig elsewhere." ;; +esac + +if [ "$APPLY" -eq 1 ] && [ "$CONFIRM_TOKEN" != "$CONFIRM_EXPECTED" ]; then + refuse "--apply was given without the matching confirmation token." \ + "This run would change a published stand that other people look at." \ + "" \ + "Re-run with: --i-know-this-deploys $CONFIRM_EXPECTED" +fi +if [ "$APPLY" -eq 0 ] && [ -n "$CONFIRM_TOKEN" ]; then + die "--i-know-this-deploys was given without --apply. Both are required to mutate; neither alone does anything." +fi + +# ── Cluster guard ─────────────────────────────────────────────────────────── +# Everything below is either a `kubectl config` read (a local file) or a +# `get`/`auth can-i` (read-only), and it all runs before any stage, in every +# mode — including the dry run, which for the seed stage does reach the cluster. + +kc() { kubectl --kubeconfig "$KUBECONFIG_PATH" "$@"; } + +guard_cluster() { + head1 "cluster guard" + + local current cluster server actual_sha + local as_args=() + local probe verb resource why answer + + current="$(kc config current-context 2>/dev/null || true)" + [ -n "$current" ] || refuse "the kubeconfig has no current-context." \ + "File: $KUBECONFIG_PATH" \ + "Set one with: kubectl --kubeconfig $KUBECONFIG_PATH config use-context " + + # CLUSTER IDENTITY IS CHECKED FIRST, AND IT IS THE ONLY CHECK THAT IS FATAL. + # + # The context NAME is a label inside a file; the cluster entry is what decides + # which API server gets written to. Checking the name first (as this did + # originally) refuses runs that are perfectly safe — an admin kubeconfig + # downloaded from a provider is routinely called `default` — while proving + # nothing about the target. So: verify the cluster, THEN reconcile the name. + cluster="$(kc config view -o "jsonpath={.contexts[?(@.name==\"$current\")].context.cluster}" 2>/dev/null || true)" + [ -n "$cluster" ] || refuse "the current context names no cluster entry." \ + "This kubeconfig is malformed; nothing further can be verified about the target." + + if [ "$cluster" != "$EXPECT_CLUSTER" ]; then + refuse "the target cluster is not the one you said you expected." \ + "--expect-cluster : $EXPECT_CLUSTER" \ + "kubeconfig says : $cluster" \ + "" \ + "Nothing has been run. If the kubeconfig is right, fix the flag; if the" \ + "flag is right, you have the wrong kubeconfig on the command line." + fi + note "cluster entry matches --expect-cluster" + + # Nothing on this path passes helm an explicit --kube-context, and the gitops + # Makefile's own `kube-ctx` guard asserts `kubectl config current-context` == + # inventory.kubeContext. In CI that holds by construction: provision-ci- + # deployer.sh writes the generated kubeconfig with the context already named + # after the environment. A human rehearsing with an admin kubeconfig has + # whatever name the provider chose, so rather than refuse, derive a normalised + # COPY with the context renamed and run everything through that. The original + # file is never modified, and the rename is only reached after the cluster + # entry above has already been proven to be the intended one. + if [ "$current" != "$KUBE_CTX" ]; then + local normalised="$ARTIFACT_DIR/kubeconfig-$KUBE_CTX.yaml" + ( umask 077; kc config view --raw --minify > "$normalised" ) \ + || refuse "could not derive a context-normalised kubeconfig." \ + "Source: $KUBECONFIG_PATH" + kubectl --kubeconfig "$normalised" config rename-context "$current" "$KUBE_CTX" >/dev/null 2>&1 \ + || refuse "could not rename the context in the derived kubeconfig." \ + "Derived file: $normalised" + kubectl --kubeconfig "$normalised" config use-context "$KUBE_CTX" >/dev/null 2>&1 \ + || refuse "could not select the renamed context in the derived kubeconfig." \ + "Derived file: $normalised" + chmod 600 "$normalised" 2>/dev/null || true + KUBECONFIG_PATH="$normalised" + note "context '$current' renamed to '$KUBE_CTX' in a derived copy (original untouched)" + note " derived: ${normalised#"$REPO_ROOT"/}" + else + note "current-context matches inventory.kubeContext" + fi + + server="$(kc config view -o "jsonpath={.clusters[?(@.name==\"$cluster\")].cluster.server}" 2>/dev/null || true)" + [ -n "$server" ] || refuse "the cluster entry has no server URL." \ + "Nothing about the target can be verified; refusing rather than guessing." + + if [ -n "$EXPECT_API_SHA" ]; then + actual_sha="$(sha256_of "$server")" + if [ "$actual_sha" != "$EXPECT_API_SHA" ]; then + refuse "the API server fingerprint does not match." \ + "expected sha256 : $EXPECT_API_SHA" \ + "actual sha256 : $actual_sha" \ + "" \ + "Names in a kubeconfig can be edited into agreement; this cannot. The" \ + "server URL itself is deliberately not printed — this repo is public" \ + "and terminal output gets pasted into issues." + fi + note "API server fingerprint matches" + else + note "no --expect-api-server-sha256 given" + note " Context and cluster names are LOCAL ALIASES: a kubeconfig for a" + note " different stand whose entries happen to carry these names passes" + note " every check above. Pin the fingerprint before using --apply:" + note " kubectl --kubeconfig config view --minify \\" + note " -o 'jsonpath={.clusters[0].cluster.server}' | shasum -a 256" + if [ "$APPLY" -eq 1 ]; then + note " WARNING: mutating a published stand with the weaker guard." + fi + fi + + # Reachability, and the namespace the whole run addresses. `get namespace` + # rather than `cluster-info`: a namespace-scoped token — which is exactly what + # the CI deployer credential is — can fail cluster-info while being perfectly + # able to do its job, and a guard that rejects the real CI credential is a + # guard nobody will run. + if ! kc --request-timeout=10s get namespace "$NS_APP" -o name >/dev/null 2>&1; then + refuse "cannot read namespace '$NS_APP' on the target cluster." \ + "Either the cluster is unreachable (VPN?) or this credential cannot see it." \ + "Nothing has been run." + fi + note "namespace $NS_APP is reachable" + + # An RBAC rehearsal, not a gate: it answers "would the CI ServiceAccount get + # through?" before a merge has to. Printed, never enforced — the verbs a stage + # needs are the stage's business, and a can-i list that disagreed with reality + # would be one more thing to keep in sync. + head2 "RBAC probe${AS_USER:+ (as $AS_USER)}" + [ -n "$AS_USER" ] && as_args=(--as "$AS_USER") + for probe in \ + "get:secret:read insight-oidc for the client secret" \ + "get:configmap:read the platform coordinates (seed discovery)" \ + "patch:deployment:helm upgrade, and the rollout restart after it" \ + "create:job:apply the seed Job" \ + "get:httproute:confirm the edge still routes"; do + verb="${probe%%:*}" + resource="${probe#*:}" + why="${resource#*:}" + resource="${resource%%:*}" + answer="$(kc ${as_args[@]+"${as_args[@]}"} auth can-i "$verb" "$resource" -n "$NS_APP" 2>/dev/null || true)" + printf ' %-7s %-11s %-5s %s\n' "$verb" "$resource" "${answer:-?}" "$why" + done +} + +# ── Parity report ─────────────────────────────────────────────────────────── +# Cheap, never fatal, printed on every run. It cannot prove the two paths are +# identical — only a human diff of the printed commands against the workflow can +# do that — but it does catch the failure mode that matters in practice: the +# workflow and this harness quietly growing two ways to do one thing. The named +# checks at the end are for divergences already known to break a run; each is an +# assertion about the workflow's TEXT, so it goes quiet the moment CI is fixed. + +parity_report() { + head1 "parity with $WORKFLOW_REL" + if [ ! -f "$WORKFLOW_FILE" ]; then + note "the workflow does not exist in this checkout — nothing to compare against." + note "When it lands, every command this script prints must appear in it verbatim." + return 0 + fi + local anchor + for anchor in \ + "helm upgrade --install" \ + "$CHART_REF" \ + "$VALUES_REL" \ + "authenticator.oidc.clientSecret" \ + "--wait --timeout $DEPLOY_TIMEOUT" \ + "helm list" \ + "--deployed --failed --pending --uninstalling" \ + "rollout restart" \ + "get httproute" \ + "seed-stand.sh" \ + "--days $SEED_WINDOW_DAYS" \ + "--stand-manifest" \ + "SMOKE_BASE_URL" \ + "$SMOKE_SUITE" \ + "stand-diagnostics.sh"; do + if grep -qF -- "$anchor" "$WORKFLOW_FILE"; then + printf ' %-6s %s\n' "found" "$anchor" + else + printf ' %-6s %s\n' "ABSENT" "$anchor" + fi + done + + # Known-wrong spelling: seed-stand.sh has no --window-days flag and refuses + # unknown arguments, so a workflow carrying it fails the seed stage every run. + if grep -qF -- "--window-days" "$WORKFLOW_FILE"; then + printf ' %-6s %s\n' "BUG" "the workflow passes --window-days; seed-stand.sh only accepts --days" + fi + + # The smoke suite is aimed by $SMOKE_BASE_URL and by nothing else: its conftest + # raises pytest.UsageError when the command line names the directory and that + # variable is unset, so a workflow setting only the shared INSIGHT_STAND_* + # names never reaches a single check. + if grep -qF -- "INSIGHT_STAND_BASE_URL" "$WORKFLOW_FILE" \ + && ! grep -qF -- "SMOKE_BASE_URL" "$WORKFLOW_FILE"; then + printf ' %-6s %s\n' "BUG" "the workflow sets INSIGHT_STAND_BASE_URL; tests/stand/smoke reads SMOKE_BASE_URL" + fi + if grep -qF -- "INSIGHT_STAND_PERSONA_PASSWORD" "$WORKFLOW_FILE" \ + && ! grep -qF -- "SMOKE_PERSONA_PASSWORD" "$WORKFLOW_FILE"; then + printf ' %-6s %s\n' "BUG" "the workflow sets INSIGHT_STAND_PERSONA_PASSWORD; tests/stand/smoke/login.py reads SMOKE_PERSONA_PASSWORD" + fi + + note "" + note "ABSENT means the workflow does something this harness does not emulate," + note "or the reverse. BUG means the two disagree in a way that fails a run." + note "Reconcile either way, or the local rehearsal proves the wrong thing." +} + +# ── Failure diagnostics ───────────────────────────────────────────────────── +# The workflow's own script, invoked with its documented positional interface +# ( ) and reading the ambient kubeconfig, exactly as CI +# invokes it. Never allowed to change the exit status: the stage's failure is the +# verdict, and a diagnostics script that died would otherwise mask it. + +emit_diagnostics() { + local stage="$1" + local script="$DIAGNOSTICS_SCRIPT" + head1 "diagnostics after a failed '$stage' stage" + + [ -n "$script" ] || script="$REPO_ROOT/$DIAGNOSTICS_DEFAULT_REL" + if [ ! -f "$script" ]; then + note "no diagnostics script at ${script#"$REPO_ROOT"/}." + note "It belongs to the workflow (a curated, redacted allowlist — a public" + note "repo's run logs are public). Point at it with --diagnostics, or set" + note "INSIGHT_STAND_DIAGNOSTICS." + note "" + note "Deliberately NOT falling back to an ad-hoc dump here: output this" + note "harness produced but CI never would is exactly what teaches an" + note "engineer to expect evidence the red run will not have." + return 0 + fi + + # It emits GitHub workflow commands (::group::, ::error::). On a laptop those + # show as literal text; that is cosmetic and is left alone, because teaching it + # a second output mode would make the local and the CI evidence differ in a way + # nobody could then compare. + show_cmd "KUBECONFIG=$KUBECONFIG_PATH " bash "${script#"$REPO_ROOT"/}" "$NS_APP" "$RELEASE" + KUBECONFIG="$KUBECONFIG_PATH" bash "$script" "$NS_APP" "$RELEASE" \ + || note "(the diagnostics script itself exited non-zero; the stage failure above is still the verdict)" +} + +fail_stage() { + local stage="$1" + local code="$2" + emit_diagnostics "$stage" + printf '\n%s\n' "== stage '$stage' FAILED (exit $code) ==" + note "Nothing has been rolled back. That is the intended disposition: the" + note "failed state is the evidence, which is also why the upgrade runs with" + note "--wait and deliberately without --atomic. Recovery is the next deploy," + note "or a deliberate 'make -C deploy/gitops rollback ENV=$GITOPS_ENV'." + exit "$code" +} + +# ── Shared checks ─────────────────────────────────────────────────────────── + +# Presence of the KEY, never its value: nothing here captures the secret into a +# variable, and the pipeline's only consumer is `grep -q`. +check_oidc_secret() { + head2 "the OIDC client secret the upgrade injects" + show_cmd "KUBECONFIG=$KUBECONFIG_PATH " kubectl -n "$NS_APP" get secret insight-oidc \ + -o 'jsonpath={.data.client-secret}' + if kc -n "$NS_APP" get secret insight-oidc -o "jsonpath={.data.client-secret}" 2>/dev/null \ + | grep -q .; then + note "present (presence only — no value is read into this shell or printed)" + return 0 + fi + note "ABSENT: Secret insight-oidc has no non-empty 'client-secret' key." + note "An upgrade without it writes a BLANK client secret into the" + note "authenticator's config: release deployed, pods Ready, every login" + note "broken at the confidential-client token exchange." + return 1 +} + +# The chart renders NO HTTPRoute, so a successful upgrade says nothing about +# whether the stand is reachable. Read, never applied: the files under +# environments//manifests/ are the source of truth and a human owns them. +check_routes() { + head2 "the edge routes (verified, never applied)" + show_cmd "KUBECONFIG=$KUBECONFIG_PATH " "${route_table_cmd[@]}" + if ! kc "${route_table_cmd[@]:1}" 2>&1; then + note "could not read the routes at all." + return 1 + fi + local not_accepted + not_accepted="$(kc "${route_status_cmd[@]:1}" 2>/dev/null | grep -v '=True$' || true)" + if [ -n "$not_accepted" ]; then + note "an edge route is not Accepted:" + printf '%s\n' "$not_accepted" + note "The release may be perfectly healthy; the stand is still unreachable," + note "and the smoke would fail at its first request with a much less useful" + note "message. Fix the route, not the release." + return 1 + fi + return 0 +} + +# ── Stage 1 · deploy ──────────────────────────────────────────────────────── + +stage_deploy() { + head1 "stage 1 - deploy (chart $CHART_VERSION -> release $RELEASE in $NS_APP)" + + local rc=0 + + if [ "$APPLY" -eq 0 ]; then + note "read-only. CI would run:" + show_cmd "KUBECONFIG= " "${deploy_helm_cmd[@]}" + printf ' %s\n' "$deploy_helm_secret_display" + note "running the Makefile's own offline render and diff instead — it goes" + note "through the same values file the upgrade would, and never contacts" + note "the cluster:" + show_cmd "" "${deploy_cmd_dry[@]}" + note "It depends on sync-clean, so a dirty working tree fails it exactly as" + note "it would fail a real deploy from a clean CI checkout." + + # The chicken-and-egg this flag exists for: `make diff` refuses on ANY file + # `git status --porcelain` reports, which is correct for a deploy (CI always + # renders from a clean checkout) and impossible while the environment, the + # workflow and this script are themselves uncommitted work in progress. The + # strict path stays the default; --allow-dirty swaps in a bare `helm + # template` through the SAME chart, version and values file, so the render is + # still the real one — only the clean-tree assertion is skipped, and the + # output says so rather than quietly passing. + if [ "$ALLOW_DIRTY" -eq 1 ] && ! ( cd "$REPO_ROOT" && git diff --quiet HEAD -- . 2>/dev/null && [ -z "$(git status --porcelain)" ] ); then + note "" + note "--allow-dirty: the tree is dirty, so 'make diff' would refuse. Rendering" + note "the same chart+values directly instead. THIS SKIPS THE CLEAN-TREE GATE" + note "that a real deploy enforces — commit before trusting a green rehearsal." + show_cmd "" "${deploy_render_cmd[@]}" + ( cd "$REPO_ROOT" && "${deploy_render_cmd[@]}" > "$ARTIFACT_DIR/render-$RUN_STAMP.yaml" ) || rc=$? + if [ "$rc" -eq 0 ]; then + note "rendered $(grep -c '^kind:' "$ARTIFACT_DIR/render-$RUN_STAMP.yaml" 2>/dev/null || echo '?') objects -> ${ARTIFACT_DIR#"$REPO_ROOT"/}/render-$RUN_STAMP.yaml" + fi + else + ( cd "$GITOPS_DIR" && "${deploy_cmd_dry[@]}" ) || rc=$? + fi + [ "$rc" -eq 0 ] || fail_stage deploy "$rc" + + # Reported, not fatal: a rehearsal that stopped at the first missing + # prerequisite would hide the second, and finding both in one pass is the + # point of running this before a merge. + check_oidc_secret || note "(reported, not failed: the dry run changes nothing)" + check_routes || note "(reported, not failed: the dry run changes nothing)" + return 0 + fi + + # Refuse before touching the release rather than after: an upgrade that blanks + # the client secret leaves a healthy-looking, unusable stand. + check_oidc_secret || fail_stage deploy 1 + + head2 "upgrade" + show_cmd "KUBECONFIG=$KUBECONFIG_PATH " "${deploy_helm_cmd[@]}" + printf ' %s\n' "$deploy_helm_secret_display" + + # The secret lives in a variable for the length of one helm call and is never + # echoed. `set -x` is not used anywhere in this script for exactly this reason. + local oidc_client_secret + oidc_client_secret="$(kc -n "$NS_APP" get secret insight-oidc \ + -o "jsonpath={.data.client-secret}" 2>/dev/null | base64 --decode)" + [ -n "$oidc_client_secret" ] || fail_stage deploy 1 + + ( cd "$REPO_ROOT" \ + && KUBECONFIG="$KUBECONFIG_PATH" "${deploy_helm_cmd[@]}" \ + --set-string "authenticator.oidc.clientSecret=$oidc_client_secret" ) || rc=$? + oidc_client_secret="" + [ "$rc" -eq 0 ] || fail_stage deploy "$rc" + + # A release can be `deployed` and still be the wrong chart — a resumed run, a + # hand-deploy that raced this one, an input that did not say what it meant. + # Checked separately from the upgrade's exit status because they answer + # different questions. The status flags are enumerated rather than `--all`, + # which Helm 4 removed: the interesting failures are the ones the default + # listing hides — a `pending-upgrade` release must be reported as itself, not + # as "absent" — but `--all` errors out on a v4 client and reports exactly that + # false "absent". + head2 "verify the release is the chart this run asked for" + show_cmd "KUBECONFIG=$KUBECONFIG_PATH " "${deploy_verify_cmd[@]}" + + local listed status chart revision + listed="$( KUBECONFIG="$KUBECONFIG_PATH" "${deploy_verify_cmd[@]}" 2>/dev/null || printf '[]' )" + status="$(printf '%s' "$listed" | jq -r '.[0].status // "absent"')" + chart="$(printf '%s' "$listed" | jq -r '.[0].chart // "absent"')" + revision="$(printf '%s' "$listed" | jq -r '.[0].revision // "?"')" + note "release $RELEASE: status=$status chart=$chart revision=$revision" + + if [ "$status" != "deployed" ]; then + note "release '$RELEASE' is '$status', not 'deployed'. It has been left in" + note "that state on purpose — read the diagnostics below, then fix forward" + note "with another deploy or roll back by hand." + fail_stage deploy 1 + fi + if [ "$chart" != "insight-$CHART_VERSION" ]; then + note "the stand is running '$chart', not 'insight-$CHART_VERSION'. helm" + note "reported success while installing something else — treat this as a" + note "problem with the chart reference or the version input, not the stand." + fail_stage deploy 1 + fi + + head2 "restart what the chart cannot know to restart" + show_cmd "KUBECONFIG=$KUBECONFIG_PATH " "${deploy_restart_cmd[@]}" + ( KUBECONFIG="$KUBECONFIG_PATH" "${deploy_restart_cmd[@]}" ) || rc=$? + [ "$rc" -eq 0 ] || fail_stage deploy "$rc" + show_cmd "KUBECONFIG=$KUBECONFIG_PATH " "${deploy_rollout_cmd[@]}" + ( KUBECONFIG="$KUBECONFIG_PATH" "${deploy_rollout_cmd[@]}" ) || rc=$? + [ "$rc" -eq 0 ] || fail_stage deploy "$rc" + + check_routes || fail_stage deploy 1 + note "deploy stage complete" +} + +# ── Stage 2 · seed ────────────────────────────────────────────────────────── + +stage_seed() { + head1 "stage 2 - seed" + [ -f "$SEED_SCRIPT" ] || die "seeder not found at $SEED_SCRIPT." + + local rc=0 + + if [ "$APPLY" -eq 0 ]; then + note "read-only. CI would run:" + show_cmd "KUBECONFIG=$KUBECONFIG_PATH " "${seed_cmd_apply[@]}" + note "running the seeder's own dry run instead — it performs the SAME" + note "cluster discovery (ConfigMap and Secret reads) and prints the Job it" + note "would apply, which is the honest read-only rehearsal of this stage:" + show_cmd "KUBECONFIG=$KUBECONFIG_PATH " "${seed_cmd_dry[@]}" + ( cd "$REPO_ROOT" && KUBECONFIG="$KUBECONFIG_PATH" "${seed_cmd_dry[@]}" ) || rc=$? + [ "$rc" -eq 0 ] || fail_stage seed "$rc" + note "" + note "A dry run produces no manifest: the seeder emits one only after a real" + note "run completes. Stage 3's dry run therefore has nothing to read unless" + note "--manifest names one from an earlier real seed." + return 0 + fi + + show_cmd "KUBECONFIG=$KUBECONFIG_PATH " "${seed_cmd_apply[@]}" + note "raw log -> $SEED_LOG_REL" + + # The seed runs as a Job whose filesystem is discarded with the pod, so the + # manifest exists only in the log the seeder streams back. Capturing it is the + # same plumbing the workflow's "capture the seed manifest" step performs, from + # the same raw stream: without it stage 3 has no manifest and pytest aborts at + # collection. CI additionally pipes the console copy through the redactor; + # locally the console is the engineer's own, so it is left intact. + # + # `set -o pipefail` is in force, so the pipeline's status is the seeder's and + # not tee's. + ( cd "$REPO_ROOT" && KUBECONFIG="$KUBECONFIG_PATH" "${seed_cmd_apply[@]}" ) 2>&1 \ + | tee "$SEED_LOG" || rc=$? + [ "$rc" -eq 0 ] || fail_stage seed "$rc" + + extract_manifest || fail_stage seed 1 + head2 "manifest" + note "written to $MANIFEST_REL" + note "manifest_version: $(jq -r '.manifest_version // "?"' "$MANIFEST_PATH")" + note "anchor_date: $(jq -r '.anchor_date // "?"' "$MANIFEST_PATH")" + note "personas: $(jq -r '(.personas // []) | length' "$MANIFEST_PATH")" + note "fixtures: $(jq -r '(.fixtures // {}) | keys | join(\", \")' "$MANIFEST_PATH")" + note "" + note "A summary, not the document. Fixture NAMES are contract; the file itself" + note "also carries persona addresses, UUIDs, the tenant and in-cluster service" + note "URLs, and stays under .deploy/ (gitignored). Never attach it to an" + note "issue, a PR, or a CI artifact." +} + +# The manifest is a pretty-printed JSON object with sorted keys, so its opening +# brace is alone on a line at column 0 and its closing brace is the next line at +# column 0; nested braces are indented and cannot be confused for either. The +# LAST such block wins, matching the workflow's extractor: a run that printed +# more than one is a run whose later document supersedes the earlier. The result +# is parsed before it is trusted, so a stream that interleaved badly fails here +# with a message rather than three stages later as an opaque pytest error. +extract_manifest() { + local start end + start="$(grep -n '^{$' "$SEED_LOG" | tail -n1 | cut -d: -f1 || true)" + if [ -z "$start" ]; then + note "no seed manifest in the seed log — the seeder did not reach the end of" + note "its run. Read $SEED_LOG_REL." + return 1 + fi + end="$(awk -v s="$start" 'NR>=s && /^}$/ {print NR; exit}' "$SEED_LOG")" + if [ -z "$end" ]; then + note "the manifest block in the seed log is not closed — the stream was truncated." + return 1 + fi + awk -v s="$start" -v e="$end" 'NR>=s && NR<=e' "$SEED_LOG" > "$MANIFEST_PATH" + if ! jq -e 'has("manifest_version")' "$MANIFEST_PATH" >/dev/null 2>&1; then + note "the extracted block is not a seed manifest (invalid JSON, or no" + note "manifest_version key) — the seed log interleaved. Recover it by hand" + note "from $SEED_LOG_REL and pass --manifest." + return 1 + fi + return 0 +} + +# ── Stage 3 · smoke ───────────────────────────────────────────────────────── + +stage_smoke() { + head1 "stage 3 - smoke" + need uv + need curl + [ -d "$REPO_ROOT/$SMOKE_SUITE" ] \ + || die "no smoke suite at $SMOKE_SUITE — it is what turns this from a deploy into a gate." + [ -n "$BASE_URL" ] \ + || die "no base URL. Pass --base-url, export SMOKE_BASE_URL, or set authenticator.oidc.redirectUri in $VALUES_REL." + + local rc=0 + local code + + # Through the public URL, no port-forwards. Prove the origin answers before + # spending a suite's worth of time discovering that it does not. + head2 "reachability" + show_cmd "" curl -sS -o /dev/null -w '%{http_code}' "$BASE_URL/" + code="$(curl -sS -o /dev/null -w '%{http_code}' --max-time 20 "$BASE_URL/" 2>/dev/null || printf '000')" + note "GET / -> HTTP $code" + if [ "$code" = "000" ]; then + note "the stand's public URL did not answer at all. Check the edge routes" + note "the deploy stage reported before looking at the release." + fail_stage smoke 1 + fi + + if [ "$APPLY" -eq 0 ]; then + note "read-only. CI would run:" + show_cmd "SMOKE_BASE_URL=$BASE_URL " "${smoke_cmd_apply[@]}" + if [ ! -f "$MANIFEST_PATH" ]; then + note "" + note "Collection needs a seed manifest and there is none at $MANIFEST_REL." + note "The suite resolves its personas from that document at COLLECTION" + note "time, so this is a hard stop rather than a skip. Run the seed stage" + note "for real once, or pass --manifest from a previous run." + return 0 + fi + show_cmd "SMOKE_BASE_URL=$BASE_URL " "${smoke_cmd_dry[@]}" + ( cd "$REPO_ROOT" \ + && SMOKE_BASE_URL="$BASE_URL" \ + INSIGHT_STAND_ARTIFACT_DIR="$ARTIFACT_DIR" \ + INSIGHT_STAND_MANIFEST="$MANIFEST_PATH" \ + "${smoke_cmd_dry[@]}" ) || rc=$? + [ "$rc" -eq 0 ] || fail_stage smoke "$rc" + return 0 + fi + + [ -f "$MANIFEST_PATH" ] \ + || die "no seed manifest at $MANIFEST_REL. Run --stage seed --apply first, or pass --manifest." + + # The suite's own credential resolution names the exact variable missing for + # the chosen SMOKE_LOGIN_MODE, so this is a pointer rather than a second copy + # of that rule — duplicating it here would be one more thing to keep in sync + # with the login code, and it is the login code that has to be right. + head2 "credentials" + note "mode = ${SMOKE_LOGIN_MODE:-password [the suite default]}" + note "Read from the environment by the suite itself, which fails naming the" + note "missing variable. Nothing is defaulted, bridged, or printed here." + + show_cmd "SMOKE_BASE_URL=$BASE_URL " "${smoke_cmd_apply[@]}" + # SMOKE_BASE_URL is what aims the suite — its conftest copies the value into + # the shared stand resolution and refuses an explicit request without it. + # INSIGHT_STAND_* are set explicitly because the library computes its defaults + # from its own file location, and neither a runner nor an arbitrary working + # directory is a developer's checkout. + ( cd "$REPO_ROOT" \ + && SMOKE_BASE_URL="$BASE_URL" \ + INSIGHT_STAND_ARTIFACT_DIR="$ARTIFACT_DIR" \ + INSIGHT_STAND_MANIFEST="$MANIFEST_PATH" \ + "${smoke_cmd_apply[@]}" ) || rc=$? + [ "$rc" -eq 0 ] || fail_stage smoke "$rc" +} + +# ── Run ───────────────────────────────────────────────────────────────────── + +head1 "emulating $WORKFLOW_REL" +if [ "$APPLY" -eq 1 ]; then + printf ' %-18s %s\n' "mode" "APPLY — this run changes the stand" +else + printf ' %-18s %s\n' "mode" "read-only (dry run)" +fi +printf ' %-18s %s\n' "stage(s)" "$STAGE" +printf ' %-18s %s\n' "gitops env" "$GITOPS_ENV" +printf ' %-18s %s\n' "chart version" "$CHART_VERSION" +printf ' %-18s %s\n' "release / ns" "$RELEASE / $NS_APP" +printf ' %-18s %s\n' "kubeconfig" "$KUBECONFIG_PATH" +printf ' %-18s %s\n' "base url" "${BASE_URL:-}" +printf ' %-18s %s\n' "artefacts" "$ARTIFACT_REL" + +parity_report +guard_cluster + +case "$STAGE" in + deploy) stage_deploy ;; + seed) stage_seed ;; + smoke) stage_smoke ;; + all) + # Sequential, and each stage function exits the script on failure rather + # than returning — so this ordering IS the gate that keeps smoke from ever + # running after a failed seed. The workflow gets the same guarantee from + # steps without `if:`. + stage_deploy + stage_seed + stage_smoke + ;; +esac + +head1 "done" +if [ "$APPLY" -eq 1 ]; then + note "stage(s) '$STAGE' completed against $EXPECT_CLUSTER." +else + note "read-only rehearsal of stage(s) '$STAGE' completed. Nothing was changed." +fi +note "" +note "This output names your cluster, context and kubeconfig path, and is NOT" +note "redacted. The repo is public — scrub it before pasting any of it into an" +note "issue or a PR, or re-run the stage piping through" +note " python3 .github/workflows/scripts/redact-stand-log.py" +note "to see what CI would have published." diff --git a/deploy/gitops/scripts/provision-ci-deployer.sh b/deploy/gitops/scripts/provision-ci-deployer.sh new file mode 100755 index 000000000..46c035d44 --- /dev/null +++ b/deploy/gitops/scripts/provision-ci-deployer.sh @@ -0,0 +1,907 @@ +#!/usr/bin/env bash +# +# provision-ci-deployer.sh — mint, rotate, verify and revoke the Kubernetes +# credential that CI uses to deploy the Insight umbrella chart onto the test +# stand (constructorfabric/insight#2244, decision D5). +# +# ═══════════════════════════════════════════════════════════════════════════ +# WHAT IT CREATES +# ═══════════════════════════════════════════════════════════════════════════ +# ServiceAccount /ci-deployer +# RoleBinding /ci-deployer-admin → ClusterRole/admin +# Role /ci-deployer-crd-supplement +# RoleBinding /ci-deployer-crd-supplement → the Role above +# Secret /ci-deployer-token (service-account-token) +# <--out> a kubeconfig carrying server + CA + that token, mode 0600 +# +# Every object is namespaced. Nothing cluster-scoped is created and nothing +# cluster-scoped is granted. The verification section at the end of a run +# PROVES that with `kubectl auth can-i` instead of asserting it in prose — +# a reviewer should be able to read the assertions rather than trust the +# author's reading of the RBAC bootstrap policy. +# +# ═══════════════════════════════════════════════════════════════════════════ +# WHY A SERVICEACCOUNT TOKEN AND NOT A COPY OF SOMEONE'S ADMIN KUBECONFIG +# ═══════════════════════════════════════════════════════════════════════════ +# 1. REVOCABILITY — the decisive reason. +# A ServiceAccount token can be destroyed. `--revoke` deletes the token +# Secret and the credential stops authenticating on the next request, no +# control-plane restart and no coordination with anyone. Deleting the +# ServiceAccount itself is even more final: the SA's UID is embedded in +# every token it ever issued and is re-checked on every request, so a new +# ServiceAccount of the same name does not resurrect an old token. (Both +# checks depend on the API server running with --service-account-lookup, +# which has been the default since 1.7 and which `--verify-only` will show +# you the truth about after a revoke: the credential either authenticates +# or it does not.) +# +# An admin kubeconfig is, in almost every distribution, a CLIENT +# CERTIFICATE. Kubernetes has no certificate revocation at all: the API +# server consults no CRL and no OCSP responder. Once such a kubeconfig +# leaks, the ONLY way to invalidate it is to rotate the cluster CA — which +# invalidates every other client certificate on the cluster at the same +# time and requires restarting the control plane. "Rotate the CI +# credential" must not be a cluster outage. +# +# 2. BLAST RADIUS. An admin kubeconfig authenticates as a cluster-admin +# subject: every namespace, every CRD, every node. This credential is +# bound with a RoleBinding, so even though the roleRef names a +# ClusterRole, the grant stops dead at the namespace edge. +# +# 3. ATTRIBUTION. Audit entries read `system:serviceaccount::ci-deployer` +# — a subject that exists for exactly one purpose. A shared human +# kubeconfig makes every automated action look like a person, and makes +# "who deleted the release" unanswerable. +# +# ─── Alternatives considered and rejected ───────────────────────────────── +# * `kubectl create token ci-deployer --duration=…` (the TokenRequest API, +# and the modern recommendation). Rejected HERE only because a GitHub +# Actions environment secret is a static string: a projected/bound token +# expires (one hour by default, and the API server caps the ceiling with +# --service-account-max-token-expiration), and nothing in the merge path +# would refresh it. The genuinely correct long-term answer is federation +# — GitHub's OIDC id_token exchanged for a short-lived cluster credential +# — which needs the API server to trust GitHub as an OIDC issuer, or a +# cloud IAM broker in front of it. That is a follow-up, not a +# prerequisite for #2244. A long-lived token is the deliberate trade, and +# it is affordable precisely because it is namespace-scoped and instantly +# revocable. +# * ClusterRoleBinding → admin. Rejected: identical permissions in EVERY +# namespace, which is the whole thing we are trying not to do. +# * RoleBinding → cluster-admin. Works, and a RoleBinding does scope it to +# one namespace — but `cluster-admin` carries `escalate` and `bind` and +# matches every API group that will ever exist on the cluster. `admin` is +# the intended "owns this namespace" role and deliberately lacks both. +# * A hand-written Role enumerating every kind the umbrella renders. +# Rejected after writing it out: the rendered set changes whenever a +# subchart adds a kind, and a stale bespoke list fails closed during a +# deploy rather than during review. `admin` plus a small, explicitly +# justified supplement for custom resources is the maintainable middle. +# +# ═══════════════════════════════════════════════════════════════════════════ +# WHY THE SUPPLEMENTAL ROLE IS NOT OPTIONAL +# ═══════════════════════════════════════════════════════════════════════════ +# The built-in `admin` ClusterRole is an AGGREGATE. It picks up rules only +# from ClusterRoles labelled `rbac.authorization.k8s.io/aggregate-to-admin`, +# so a custom resource is covered only if its provider ships such a +# ClusterRole and its chart has that switch enabled. The upstream Gateway API +# CRD bundle ships no aggregation ClusterRoles at all; Argo Workflows and +# cert-manager ship them but behind chart values. Relying on that is how you +# get a credential that works on one cluster and 403s on the next. +# +# There is a second, sharper reason. The umbrella chart itself renders a Role +# (charts/insight/templates/ingestion/reconcile-rbac.yaml, gated on +# `ingestion.templates.enabled`) that grants `argoproj.io` and +# `onepassword.com` verbs. RBAC escalation prevention refuses to let a +# subject CREATE a Role containing permissions the subject does not itself +# hold, unless it holds the `escalate` verb — which `admin` does not. Without +# the supplement, `helm upgrade` fails on that one manifest with +# "attempt to grant extra privileges", after having already applied half the +# release. The supplement is therefore a superset of what the chart's own +# Roles grant, on purpose. +# +# ═══════════════════════════════════════════════════════════════════════════ +# KNOWN LIMITATION — helm's --create-namespace +# ═══════════════════════════════════════════════════════════════════════════ +# `helm upgrade --install … --create-namespace` (which deploy/gitops/Makefile +# hardcodes) POSTs a Namespace object at CLUSTER scope when, and only when, +# the release does not yet exist. This credential cannot create namespaces — +# that is the point — so a FIRST install into an empty namespace fails with a +# 403 before any chart resource is applied. Upgrades of an existing release +# never reach that code path, so day-2 CI is unaffected. +# Disposition: the target namespace is created once, by a human, with a +# human credential. If CI ever has to bootstrap a namespace from nothing, +# fix it by pre-creating the namespace in the same human step — NOT by +# granting namespace-create to CI. +# +# ═══════════════════════════════════════════════════════════════════════════ +# WHAT THIS DELIBERATELY DOES NOT GRANT +# ═══════════════════════════════════════════════════════════════════════════ +# * Anything cluster-scoped: namespaces, nodes, CRDs, ClusterRoles, +# ClusterRoleBindings, PersistentVolumes, StorageClasses. +# * Any other namespace. In particular the datastore namespaces, whose +# Secrets are the stand's crown jewels, stay out of reach. +# * The chart's `airbyte-auth-rbac.yaml` renders a Role into +# `airbyte.namespace` when that value is non-empty. A namespace-scoped +# credential cannot create it. Keep `airbyte.namespace: ""` (same +# namespace as the release) in the CI-driven environment's values, or +# provision a second, equally narrow supplement there by hand. +# +# ═══════════════════════════════════════════════════════════════════════════ +# SAFETY PROPERTIES OF THE SCRIPT ITSELF +# ═══════════════════════════════════════════════════════════════════════════ +# * DRY-RUN BY DEFAULT. Nothing is created, deleted or written without an +# explicit `--apply`. The default run prints the exact manifests and the +# exact verification commands and exits 0. +# * EXPLICIT TARGET. `--kubeconfig` and `--expect-cluster` are both +# required, and the script refuses loudly when the kubeconfig's context +# resolves to a different cluster. There is no "current context" default +# and no way to omit the expectation: a deploy credential minted on the +# wrong cluster is a security incident, not a typo. +# * NO SECRET EVER REACHES STDOUT. The token is read into a shell variable +# (never an argv, which `ps` exposes to every local user), written into +# the output kubeconfig under `umask 077`, and unset. The assembled +# kubeconfig is never cat'ed, diffed or echoed by this script. +# * NO INFRA HOSTNAME REACHES STDOUT. The API server URL is printed with +# its host redacted unless `--show-server` is passed. Operator terminals +# get pasted into pull requests, and this repository is public. +# +# ═══════════════════════════════════════════════════════════════════════════ +# USAGE +# ═══════════════════════════════════════════════════════════════════════════ +# Plan (default — reads the cluster, writes nothing): +# ./provision-ci-deployer.sh --kubeconfig ~/.kube/stand.yaml \ +# --expect-cluster +# +# Apply, then assemble the CI kubeconfig: +# ./provision-ci-deployer.sh --kubeconfig ~/.kube/stand.yaml \ +# --expect-cluster --apply +# +# Re-check the scope of a kubeconfig you already have: +# ./provision-ci-deployer.sh --kubeconfig ~/.kube/stand.yaml \ +# --expect-cluster --verify-only +# +# Rotate / revoke: add --rotate or --revoke (still needs --apply). +# +# The operator runbook — GitHub environment creation, secret names, rotation +# cadence, cleanup — lives at +# docs/components/deployment/specs/sop/credentials-runbook.md. + +set -euo pipefail + +# ─── Presentation ───────────────────────────────────────────────────────── +# Same tput-with-fallback shape as scripts/doctor.sh so the two read alike +# when they scroll past each other in one terminal. +C_RED=$(tput setaf 1 2>/dev/null || echo "") +C_GRN=$(tput setaf 2 2>/dev/null || echo "") +C_YEL=$(tput setaf 3 2>/dev/null || echo "") +C_CYA=$(tput setaf 6 2>/dev/null || echo "") +C_RST=$(tput sgr0 2>/dev/null || echo "") + +note() { printf '%s\n' "$*"; } +ok() { printf '%sOK%s %s\n' "$C_GRN" "$C_RST" "$*"; } +warn() { printf '%sNOTE%s %s\n' "$C_YEL" "$C_RST" "$*"; } +bad() { printf '%sFAIL%s %s\n' "$C_RED" "$C_RST" "$*"; } +hdr() { printf '\n%s══ %s ══%s\n' "$C_CYA" "$*" "$C_RST"; } +die() { + printf '%sERROR%s: %s\n' "$C_RED" "$C_RST" "$*" >&2 + exit 1 +} + +# ─── Defaults ───────────────────────────────────────────────────────────── +KUBECONFIG_IN="" +EXPECT_CLUSTER="" +EXPECT_SERVER="" +SRC_CONTEXT="" +NAMESPACE="insight" +SA_NAME="ci-deployer" +TOKEN_SECRET="" +# Written into the GENERATED kubeconfig as the context name. The gitops +# Makefile's `kube-ctx` target refuses to act unless +# `kubectl config current-context` equals inventory `.kubeContext`, and +# deploy/gitops/README.md documents the convention `insight-`. Naming +# the generated context after the environment therefore makes +# `KUBECONFIG= make deploy ENV=test-stand` work with no KUBE_CTX +# override. Change it with --context-name if the inventory disagrees. +OUT_CONTEXT="insight-test-stand" +OUT_PATH="${HOME}/.kube/insight-test-stand-ci-deployer.kubeconfig" +CA_FILE="" +APPLY=0 +MODE="provision" # provision | rotate | revoke | purge | verify +SHOW_SERVER=0 +WITH_SUPPLEMENT=1 +TOKEN_WAIT_S=60 + +usage() { + cat <<'USAGE' +provision-ci-deployer.sh — provision the namespace-scoped CI deploy credential. + +Required: + --kubeconfig PATH Admin kubeconfig for the target cluster. Read only; + this script never modifies it. + --expect-cluster NAME The cluster name the kubeconfig's context MUST + resolve to. The script refuses if it does not. + +Target selection: + --context NAME Context inside --kubeconfig (default: its + current-context). + --expect-server URL Optional second guard: the API server URL must + match this string exactly. + --namespace NAME Namespace to scope the credential to + (default: insight). + --serviceaccount NAME ServiceAccount name (default: ci-deployer). + --token-secret NAME Token Secret name (default: -token). + +Output: + --out PATH Where to write the assembled kubeconfig + (default: ~/.kube/insight-test-stand-ci-deployer.kubeconfig). + Its parent directory must already exist. Refused if + the path is inside a git work tree and not ignored. + --context-name NAME Context/cluster name INSIDE the generated + kubeconfig (default: insight-test-stand). + --ca-file PATH PEM CA bundle to embed, if neither the token Secret + nor the admin kubeconfig carries one. + +Modes (at most one; default provisions): + --rotate Replace ONLY the token Secret and rewrite --out. + --revoke Delete the token Secret. The credential stops + authenticating; the ServiceAccount and RBAC stay. + --purge Delete token Secret, both RoleBindings, the Role + and the ServiceAccount. + --verify-only Run the scope assertions against an existing --out. + +Behaviour: + --apply Actually mutate. WITHOUT THIS THE SCRIPT ONLY + PRINTS THE PLAN. + --no-supplement Skip the Gateway API / Argo / cert-manager + supplemental Role. Only for a cluster whose CRD + providers ship aggregate-to-admin ClusterRoles. + --show-server Print the API server URL unredacted. + --token-wait SECONDS How long to wait for the token controller to fill + the Secret (default: 60). + -h, --help This text. +USAGE +} + +# ─── Argument parsing ───────────────────────────────────────────────────── +set_mode() { + [ "$MODE" = "provision" ] || die "modes are mutually exclusive: already in '$MODE', cannot also do '$1'" + MODE="$1" +} + +while [ $# -gt 0 ]; do + case "$1" in + --kubeconfig) + KUBECONFIG_IN="${2:?--kubeconfig needs a value}" + shift 2 + ;; + --expect-cluster) + EXPECT_CLUSTER="${2:?--expect-cluster needs a value}" + shift 2 + ;; + --expect-server) + EXPECT_SERVER="${2:?--expect-server needs a value}" + shift 2 + ;; + --context) + SRC_CONTEXT="${2:?--context needs a value}" + shift 2 + ;; + --namespace) + NAMESPACE="${2:?--namespace needs a value}" + shift 2 + ;; + --serviceaccount) + SA_NAME="${2:?--serviceaccount needs a value}" + shift 2 + ;; + --token-secret) + TOKEN_SECRET="${2:?--token-secret needs a value}" + shift 2 + ;; + --out) + OUT_PATH="${2:?--out needs a value}" + shift 2 + ;; + --context-name) + OUT_CONTEXT="${2:?--context-name needs a value}" + shift 2 + ;; + --ca-file) + CA_FILE="${2:?--ca-file needs a value}" + shift 2 + ;; + --token-wait) + TOKEN_WAIT_S="${2:?--token-wait needs a value}" + shift 2 + ;; + --rotate) + set_mode rotate + shift + ;; + --revoke) + set_mode revoke + shift + ;; + --purge) + set_mode purge + shift + ;; + --verify-only) + set_mode verify + shift + ;; + --apply) + APPLY=1 + shift + ;; + --no-supplement) + WITH_SUPPLEMENT=0 + shift + ;; + --show-server) + SHOW_SERVER=1 + shift + ;; + -h | --help) + usage + exit 0 + ;; + *) die "unknown argument: $1 (try --help)" ;; + esac +done + +[ -n "$KUBECONFIG_IN" ] || { + usage >&2 + die "--kubeconfig is required. There is no current-context default on purpose." +} +[ -n "$EXPECT_CLUSTER" ] || { + usage >&2 + die "--expect-cluster is required. Minting a deploy credential on the wrong cluster is not a recoverable typo." +} +[ -r "$KUBECONFIG_IN" ] || die "cannot read kubeconfig at $KUBECONFIG_IN" +[ -n "$TOKEN_SECRET" ] || TOKEN_SECRET="${SA_NAME}-token" +case "$TOKEN_WAIT_S" in +'' | *[!0-9]*) die "--token-wait must be an integer number of seconds" ;; +esac + +command -v kubectl >/dev/null 2>&1 || die "kubectl is required (brew install kubectl)" + +ROLE_NAME="${SA_NAME}-crd-supplement" +RB_ADMIN="${SA_NAME}-admin" +RB_SUPPLEMENT="${SA_NAME}-crd-supplement" + +# `kubectl` against the ADMIN kubeconfig. Every call in this script that uses +# it is either read-only or gated behind $APPLY. +kadm() { kubectl --kubeconfig "$KUBECONFIG_IN" --context "$SRC_CONTEXT" "$@"; } + +# ─── Output path guard ──────────────────────────────────────────────────── +# Checked before anything touches the network, so a bad --out costs a second +# rather than a round trip. The assembled kubeconfig contains a bearer token: +# if it lands inside the work tree of this PUBLIC repository and is not +# ignored, one `git add -A` publishes it. Refuse rather than rely on review +# catching it. +OUT_DIR="$(dirname "$OUT_PATH")" +[ -d "$OUT_DIR" ] || die "the parent directory of --out does not exist: $OUT_DIR (mkdir -p it first)" +OUT_ABS="$(cd "$OUT_DIR" && pwd)/$(basename "$OUT_PATH")" + +if git -C "$OUT_DIR" rev-parse --show-toplevel >/dev/null 2>&1; then + if ! git -C "$OUT_DIR" check-ignore -q "$OUT_ABS" 2>/dev/null; then + die "refusing to write a bearer token to '$OUT_ABS': it is inside a git work tree and not gitignored. Choose a path outside the repo (e.g. ~/.kube/…)." + fi + warn "--out is inside a git work tree but gitignored — allowed, still prefer a path outside the repo" +fi + +# ─── Target guard ───────────────────────────────────────────────────────── +# The single most important thing this script does. `kubectl` happily follows +# whatever current-context it finds; a credential provisioned against the +# wrong cluster is a silent, standing grant nobody is looking for. +hdr "target" + +if [ -z "$SRC_CONTEXT" ]; then + SRC_CONTEXT="$(kubectl --kubeconfig "$KUBECONFIG_IN" config current-context 2>/dev/null || true)" + [ -n "$SRC_CONTEXT" ] || die "no current-context in $KUBECONFIG_IN and --context was not given" +fi + +ACTUAL_CLUSTER="$(kubectl --kubeconfig "$KUBECONFIG_IN" config view \ + -o "jsonpath={.contexts[?(@.name==\"${SRC_CONTEXT}\")].context.cluster}" 2>/dev/null || true)" +[ -n "$ACTUAL_CLUSTER" ] || die "context '$SRC_CONTEXT' is not present in $KUBECONFIG_IN" + +if [ "$ACTUAL_CLUSTER" != "$EXPECT_CLUSTER" ]; then + printf '%s\n' "$C_RED" >&2 + printf ' ┌──────────────────────────────────────────────────────────────┐\n' >&2 + printf ' │ REFUSING TO ACT — CLUSTER MISMATCH │\n' >&2 + printf ' └──────────────────────────────────────────────────────────────┘%s\n' "$C_RST" >&2 + printf ' kubeconfig : %s\n' "$KUBECONFIG_IN" >&2 + printf ' context : %s\n' "$SRC_CONTEXT" >&2 + printf ' resolves to: %s\n' "$ACTUAL_CLUSTER" >&2 + printf ' --expect-cluster says: %s\n\n' "$EXPECT_CLUSTER" >&2 + printf ' Nothing was created, deleted or written. Re-run with the right\n' >&2 + printf ' --kubeconfig/--context, or correct --expect-cluster if you are\n' >&2 + printf ' certain which cluster you mean.\n' >&2 + exit 2 +fi + +SERVER="$(kubectl --kubeconfig "$KUBECONFIG_IN" config view \ + -o "jsonpath={.clusters[?(@.name==\"${ACTUAL_CLUSTER}\")].cluster.server}" 2>/dev/null || true)" +[ -n "$SERVER" ] || die "cluster '$ACTUAL_CLUSTER' has no server URL in $KUBECONFIG_IN" + +if [ -n "$EXPECT_SERVER" ] && [ "$SERVER" != "$EXPECT_SERVER" ]; then + die "API server does not match --expect-server (compared without printing either; pass --show-server to see them)" +fi + +# Redact the host by default: this repo is public and operator terminals end +# up in pull requests and issue comments. The scheme and port are enough to +# tell "yes, that's an API server endpoint" apart from "that's the public +# ingress URL". +redact_url() { + if [ "$SHOW_SERVER" = "1" ]; then + printf '%s' "$1" + else + printf '%s' "$1" | sed -E 's#^([a-zA-Z][a-zA-Z0-9+.-]*://)[^/:]+#\1#' + fi +} + +ok "cluster '$ACTUAL_CLUSTER' matches --expect-cluster" +note " context : $SRC_CONTEXT" +note " api server : $(redact_url "$SERVER")" +note " namespace : $NAMESPACE" + +# Liveness + "is this really a cluster I can act on" check. Read-only. +kadm version --request-timeout=15s -o json >/dev/null 2>&1 || + die "cannot reach the API server for context '$SRC_CONTEXT' (VPN down? kubeconfig expired?)" +ok "API server reachable" + +if kadm get namespace "$NAMESPACE" >/dev/null 2>&1; then + ok "namespace '$NAMESPACE' exists" +else + warn "namespace '$NAMESPACE' does not exist yet — create it with a human credential first" + warn " (see the 'helm --create-namespace' limitation in this script's header)" +fi + +# ═══════════════════════════════════════════════════════════════════════════ +# MANIFESTS +# ═══════════════════════════════════════════════════════════════════════════ +# Built as one document so the plan the operator reads and the bytes that get +# applied are literally the same string. +render_manifests() { + cat <- + GitHub Actions deploy credential for the Insight test stand. + Provisioned by deploy/gitops/scripts/provision-ci-deployer.sh. + Rotate or revoke with that script; see + docs/components/deployment/specs/sop/credentials-runbook.md. +automountServiceAccountToken: false +--- +# RoleBinding — NOT ClusterRoleBinding. roleRef names the built-in cluster +# role \`admin\`, but a RoleBinding applies its rules only inside this +# namespace. That is the whole containment story in four lines of YAML. +apiVersion: rbac.authorization.k8s.io/v1 +kind: RoleBinding +metadata: + name: ${RB_ADMIN} + namespace: ${NAMESPACE} + labels: + app.kubernetes.io/name: ${SA_NAME} + app.kubernetes.io/component: ci-credential + app.kubernetes.io/managed-by: provision-ci-deployer.sh +roleRef: + apiGroup: rbac.authorization.k8s.io + kind: ClusterRole + name: admin +subjects: + - kind: ServiceAccount + name: ${SA_NAME} + namespace: ${NAMESPACE} +EOF + + if [ "$WITH_SUPPLEMENT" = "1" ]; then + cat </dev/null || true)" + got="${got%%$'\n'*}" + got="${got%% *}" + [ -n "$got" ] || got="(no answer)" + if [ "$got" = "$want" ]; then + ok "$(printf '%-4s %s' "$got" "$desc")" + else + bad "$(printf 'want %-3s got %-3s %s' "$want" "$got" "$desc")" + SCOPE_FAILURES=$((SCOPE_FAILURES + 1)) + fi +} + +print_verification_plan() { + cat </dev/null || true)" + if [ -n "$who" ]; then + if [ "$who" = "system:serviceaccount:${NAMESPACE}:${SA_NAME}" ]; then + ok "authenticates as $who" + else + bad "authenticates as '$who', expected system:serviceaccount:${NAMESPACE}:${SA_NAME}" + SCOPE_FAILURES=$((SCOPE_FAILURES + 1)) + fi + else + warn "\`kubectl auth whoami\` unavailable (needs kubectl 1.27+ / apiserver 1.28+) — identity not asserted" + fi + + expect_can yes "create jobs in ${NAMESPACE} (the seed stage renders a Job)" create jobs -n "$NAMESPACE" + expect_can yes "get pods/log in ${NAMESPACE} (seed + curated diagnostics)" get pods/log -n "$NAMESPACE" + expect_can yes "create secrets in ${NAMESPACE} (helm release storage)" create secrets -n "$NAMESPACE" + expect_can yes "update deployments in ${NAMESPACE} (helm upgrade, rollout restart)" update deployments.apps -n "$NAMESPACE" + expect_can yes "create Roles in ${NAMESPACE} (the chart renders its own RBAC)" create roles.rbac.authorization.k8s.io -n "$NAMESPACE" + + if [ "$WITH_SUPPLEMENT" = "1" ]; then + expect_can yes "create WorkflowTemplates in ${NAMESPACE}" create workflowtemplates.argoproj.io -n "$NAMESPACE" + expect_can yes "update HTTPRoutes in ${NAMESPACE}" update httproutes.gateway.networking.k8s.io -n "$NAMESPACE" + fi + + expect_can no "anything, anywhere (the blanket check)" '*' '*' --all-namespaces + expect_can no "list namespaces at cluster scope" list namespaces --all-namespaces + expect_can no "create namespaces (hence the --create-namespace limitation)" create namespaces --all-namespaces + expect_can no "read Secrets across all namespaces" get secrets --all-namespaces + expect_can no "list nodes" list nodes --all-namespaces + expect_can no "create ClusterRoleBindings" create clusterrolebindings.rbac.authorization.k8s.io --all-namespaces + expect_can no "read pods in kube-system" get pods -n kube-system + + if [ "$SCOPE_FAILURES" -gt 0 ]; then + die "$SCOPE_FAILURES scope assertion(s) failed — do NOT put this kubeconfig in a GitHub environment" + fi + ok "all scope assertions hold" +} + +# ═══════════════════════════════════════════════════════════════════════════ +# KUBECONFIG ASSEMBLY +# ═══════════════════════════════════════════════════════════════════════════ +# Written by hand rather than with `kubectl config set-credentials --token=…` +# on purpose: anything passed as an argv is visible in `ps` to every local +# user for the lifetime of the call. The token only ever exists in a shell +# variable and in the 0600 file. +assemble_kubeconfig() { + local token ca_b64 + + # The go-template form decodes in-process; `base64 -d` vs `-D` differs + # between GNU and BSD and is not worth the portability shim. + token="$(kadm -n "$NAMESPACE" get secret "$TOKEN_SECRET" \ + -o go-template='{{ if .data.token }}{{ .data.token | base64decode }}{{ end }}' 2>/dev/null || true)" + [ -n "$token" ] || die "the token Secret '$TOKEN_SECRET' carries no token" + + # certificate-authority-data in a kubeconfig is base64(PEM), and the token + # Secret's ca.crt field is already base64(PEM) — copy it across verbatim. + ca_b64="$(kadm -n "$NAMESPACE" get secret "$TOKEN_SECRET" \ + -o "jsonpath={.data.ca\.crt}" 2>/dev/null || true)" + + if [ -z "$ca_b64" ] && [ -n "$CA_FILE" ]; then + [ -r "$CA_FILE" ] || die "cannot read --ca-file at $CA_FILE" + ca_b64="$(base64 <"$CA_FILE" | tr -d '\n')" + fi + if [ -z "$ca_b64" ]; then + ca_b64="$(kubectl --kubeconfig "$KUBECONFIG_IN" config view --raw \ + -o "jsonpath={.clusters[?(@.name==\"${ACTUAL_CLUSTER}\")].cluster.certificate-authority-data}" 2>/dev/null || true)" + fi + if [ -z "$ca_b64" ]; then + local ca_path + ca_path="$(kubectl --kubeconfig "$KUBECONFIG_IN" config view --raw \ + -o "jsonpath={.clusters[?(@.name==\"${ACTUAL_CLUSTER}\")].cluster.certificate-authority}" 2>/dev/null || true)" + if [ -n "$ca_path" ] && [ -r "$ca_path" ]; then + ca_b64="$(base64 <"$ca_path" | tr -d '\n')" + fi + fi + [ -n "$ca_b64" ] || die "could not resolve a cluster CA — pass --ca-file . (Refusing to emit insecure-skip-tls-verify: a CI credential that does not pin its server can be handed to anything that answers on that address.)" + + # umask BEFORE creating the file: a chmod after the write leaves a window + # in which the token is world-readable. + local old_umask + old_umask="$(umask)" + umask 077 + # The cluster entry is named after the generated CONTEXT, not after the + # real cluster, so nothing about the target cluster's naming leaks into a + # file that might get copied around. + cat >"$OUT_PATH" </dev/null || true)" ]; then + ok "token controller populated $TOKEN_SECRET after ${i}s" + return 0 + fi + sleep 1 + done + die "the token controller never populated '$TOKEN_SECRET' within ${TOKEN_WAIT_S}s. On clusters that disable the legacy token controller, mint with TokenRequest instead and accept the expiry — see the runbook." +} + +# ═══════════════════════════════════════════════════════════════════════════ +# MODES +# ═══════════════════════════════════════════════════════════════════════════ +if [ "$MODE" = "verify" ]; then + run_verification + exit 0 +fi + +hdr "plan (mode: $MODE)" + +case "$MODE" in +provision | rotate) + if [ "$MODE" = "rotate" ]; then + note "delete secret/${TOKEN_SECRET} -n ${NAMESPACE} (invalidates the current CI token)" + note "re-apply the manifests below, then rewrite ${OUT_ABS}" + else + note "apply the manifests below, then write ${OUT_ABS} (mode 0600)" + fi + note "" + render_manifests + note "" + note "then, against the assembled kubeconfig:" + print_verification_plan + ;; +revoke) + note "delete secret/${TOKEN_SECRET} -n ${NAMESPACE}" + note "" + note "The ServiceAccount, Role and both RoleBindings are LEFT IN PLACE: they" + note "grant nothing without a token, and keeping them makes re-issuing a" + note "credential a one-command operation. Use --purge to remove them too." + ;; +purge) + note "delete secret/${TOKEN_SECRET} -n ${NAMESPACE}" + note "delete rolebinding/${RB_ADMIN} -n ${NAMESPACE}" + note "delete rolebinding/${RB_SUPPLEMENT} -n ${NAMESPACE}" + note "delete role/${ROLE_NAME} -n ${NAMESPACE}" + note "delete serviceaccount/${SA_NAME} -n ${NAMESPACE}" + note "" + note "Deleting the ServiceAccount is the definitive revocation: its UID is" + note "embedded in every token it ever issued and re-checked on every request." + ;; +esac + +if [ "$APPLY" != "1" ]; then + hdr "dry run" + note "Nothing was created, deleted or written." + note "Re-run with ${C_CYA}--apply${C_RST} to execute the plan above." + exit 0 +fi + +# ─── Execute ────────────────────────────────────────────────────────────── +hdr "applying" + +case "$MODE" in +provision | rotate) + if [ "$MODE" = "rotate" ]; then + kadm -n "$NAMESPACE" delete secret "$TOKEN_SECRET" --ignore-not-found + ok "deleted the previous token Secret — the old CI token is now dead" + fi + render_manifests | kadm apply -f - + wait_for_token + assemble_kubeconfig + run_verification + hdr "next" + note "1. Load it into the GitHub environment, base64-encoded on a single" + note " line (that is the form the deploy workflow decodes), never into" + note " the repo:" + note " base64 < \"$OUT_ABS\" | tr -d '\\n' \\" + note " | gh secret set TEST_STAND_KUBECONFIG --env insight-test-stand" + note "2. Then remove the local copy, or keep it at 0600 in ~/.kube only." + note " Full procedure: docs/components/deployment/specs/sop/credentials-runbook.md" + ;; +revoke) + kadm -n "$NAMESPACE" delete secret "$TOKEN_SECRET" --ignore-not-found + ok "token Secret deleted — the credential no longer authenticates" + warn "the GitHub environment still holds the dead kubeconfig; replace or delete it" + ;; +purge) + kadm -n "$NAMESPACE" delete secret "$TOKEN_SECRET" --ignore-not-found + kadm -n "$NAMESPACE" delete rolebinding "$RB_ADMIN" --ignore-not-found + kadm -n "$NAMESPACE" delete rolebinding "$RB_SUPPLEMENT" --ignore-not-found + kadm -n "$NAMESPACE" delete role "$ROLE_NAME" --ignore-not-found + kadm -n "$NAMESPACE" delete serviceaccount "$SA_NAME" --ignore-not-found + ok "ServiceAccount and RBAC removed" + warn "the GitHub environment still holds the dead kubeconfig; replace or delete it" + ;; +esac diff --git a/docs/components/deployment/gitops/ci-emulation.md b/docs/components/deployment/gitops/ci-emulation.md new file mode 100644 index 000000000..df461edd7 --- /dev/null +++ b/docs/components/deployment/gitops/ci-emulation.md @@ -0,0 +1,260 @@ +--- +status: proposed +date: 2026-08-09 +--- + +# Emulating the test-stand workflow locally + +`deploy/gitops/scripts/emulate-ci-deploy.sh` runs, from a laptop, the same three stages that `.github/workflows/deploy-test-stand.yml` runs after the umbrella chart is published: **deploy → seed → smoke**. This page explains what the emulation is worth — which parts are byte-identical to CI, which parts cannot be and why — and gives the exact sequence an engineer runs to gain confidence in the whole path *before the workflow is ever enabled*, including a read-only rehearsal against the live stand. + +It is written for the person who is about to turn that workflow on for the first time, and for the person who has to answer "does it reproduce by hand?" the first time it goes red. + +## Table of Contents + +- [1. Why a local harness at all](#1-why-a-local-harness-at-all) +- [2. The emulation model](#2-the-emulation-model) + - [2.1 Byte-identical to CI](#21-byte-identical-to-ci) + - [2.2 Necessarily different](#22-necessarily-different) + - [2.3 Present locally, absent in CI — on purpose](#23-present-locally-absent-in-ci--on-purpose) + - [2.4 How drift is detected rather than assumed](#24-how-drift-is-detected-rather-than-assumed) +- [3. Prerequisites](#3-prerequisites) +- [4. The rehearsal sequence](#4-the-rehearsal-sequence) + - [Step 0 — tooling](#step-0--tooling) + - [Step 1 — read the plan without a cluster](#step-1--read-the-plan-without-a-cluster) + - [Step 2 — pin the API server fingerprint](#step-2--pin-the-api-server-fingerprint) + - [Step 3 — read-only rehearsal against the live stand](#step-3--read-only-rehearsal-against-the-live-stand) + - [Step 4 — rehearse as the CI credential](#step-4--rehearse-as-the-ci-credential) + - [Step 5 — the first real run, one stage at a time](#step-5--the-first-real-run-one-stage-at-a-time) + - [Step 6 — only now, enable the workflow](#step-6--only-now-enable-the-workflow) +- [5. Reading a failure](#5-reading-a-failure) +- [6. Safety rules that are not negotiable](#6-safety-rules-that-are-not-negotiable) +- [7. Known gaps and things to reconcile](#7-known-gaps-and-things-to-reconcile) +- [Traceability](#traceability) + +## 1. Why a local harness at all + +The test-stand workflow fires on a merge to `main` and on nothing else. That makes it the worst possible place to discover that the gitops environment is wrong: the feedback loop is *merge, wait, read a redacted public log, guess, merge again*, and every iteration leaves a red X on somebody's merge commit and a published stand in an unknown state. + +Two failure modes make that loop especially expensive here: + +- **Most of what can go wrong is configuration, not code.** A missing Secret key, a context name that does not match the inventory, a ServiceAccount without `patch` on deployments, an edge route nobody re-applied. None of it needs a chart to be published to be discoverable, and all of it is discoverable in seconds from a laptop. +- **The stand is published.** Failures are not private. A deploy that half-lands is visible to whoever looks at the stand next, and the run log that explains it is public forever. + +So the harness exists to move every one of those discoveries to *before* the merge. Its design constraint follows directly: it must not invent a deploy path. A local script that did the same thing a slightly different way would give a green run that proves nothing about the red one. + +## 2. The emulation model + +### 2.1 Byte-identical to CI + +The following commands are the same commands, in the same order, with the same flags. The harness prints each one before it runs it, so the terminal output doubles as the proof. + +| Stage | Command | Workflow step | +|---|---|---| +| deploy | `helm upgrade --install oci://ghcr.io/constructorfabric/charts/insight --version --namespace --values deploy/gitops/environments/test-stand/values.yaml --set-string authenticator.oidc.clientSecret=… --wait --timeout 10m --history-max 10` | `Stage 1/3 — upgrade the release` | +| deploy | `helm list -n --deployed --failed --pending --uninstalling --filter '^$' -o json`, then assert `status == deployed` and `chart == insight-` | `Stage 1/3 — verify the release…` | +| deploy | `kubectl -n rollout restart ` and `rollout status --timeout=5m` on the same list | `Stage 1/3 — restart what the chart cannot know to restart` | +| deploy | `kubectl -n get httproute ` and fail unless every one is `Accepted=True` | `Stage 1/3 — confirm the edge still routes` | +| seed | `./src/ingestion/tools/seed/seed-stand.sh -n --context --email --days 730` | `Stage 2/3 — seed the stand` | +| seed | slice the last column-0 `{ … }` block out of the raw seed log, parse it, write it as the stand manifest | `Stage 2/3 — capture the seed manifest` | +| smoke | `uv run --project tests --frozen pytest tests/stand/smoke --stand-manifest `, with `SMOKE_BASE_URL` set and nothing else aiming it | `Stage 3/3 — smoke the stand…` | +| any failure | `bash .github/workflows/scripts/stand-diagnostics.sh ` | `Failure diagnostics (curated, redacted)` | + +Three properties of that list are load-bearing and worth stating rather than leaving implied: + +- **The verification half of stage 1 is not belt-and-braces.** `helm upgrade` succeeding and *the release being the chart this run asked for* are different questions, and a resumed run, a hand-deploy that raced CI, or a mistyped version input answers the first `yes` and the second `no`. +- **`make deploy` is deliberately not used, in either place.** `deploy-insight` chains `apply-app-secrets`, which hard-requires a sealed manifest this stand has no controller for and then rewrites the chart's config Secrets from a key name this stand does not use; and it hardcodes `--atomic` after `$(HELM_UPGRADE_FLAGS)`, so no existing knob turns the rollback off. `make diff`, `make status` and `make rollback` *do* work for this environment unchanged, which is why `make diff` is the deploy stage's read-only form. +- **Smoke never runs after a failed seed.** In the workflow that is a property of steps without `if:`. In the harness it is a property of each stage function exiting the script rather than returning. Same guarantee, two mechanisms — which is itself something to keep in mind if either file is restructured. + +### 2.2 Necessarily different + +These differences cannot be removed. Each one is a place where a green local run does *not* prove the CI run will be green, so read this list as the residual risk the rehearsal leaves behind. + +| Aspect | CI | Local | Why it cannot be closed | +|---|---|---|---| +| **Runner OS** | `ubuntu-latest`, GNU coreutils | your laptop, often BSD/macOS coreutils | The harness sticks to spellings both accept (`base64 --decode`, POSIX `awk`, no GNU-only `grep` flags), but a laptop is not a runner and never will be. A stage that passes locally and fails in CI on a text-processing difference is the one class of bug this harness cannot rule out. | +| **Credential source** | the `insight-test-stand` GitHub environment: a base64 kubeconfig in `TEST_STAND_KUBECONFIG`, decoded to `$RUNNER_TEMP` under `umask 077` | a kubeconfig file you name with `--kubeconfig` | Admin kubeconfigs stay human-only by design; CI gets a namespace-scoped ServiceAccount. The two credentials have different rights, which is exactly what `--as-user` exists to rehearse. | +| **Identity on the cluster** | the namespace-scoped `ci-deployer` ServiceAccount | whoever your kubeconfig is, usually far more privileged | A rehearsal run as yourself can succeed at a step CI will be refused. Run [Step 4](#step-4--rehearse-as-the-ci-credential). | +| **Checkout** | the merge commit on `main`, fresh clone, `persist-credentials: false`, tree guaranteed clean | your working tree, with whatever you are mid-way through | `make diff` depends on `sync-clean`, which fails on any file `git status --porcelain` reports. That is not the harness being fussy: the real deploy has the same prerequisite. It is also why the harness refuses a `--kubeconfig` that lives inside the repository. | +| **Console redaction** | every stage is piped through `redact-stand-log.py`; the raw stream is teed to `$RUNNER_TEMP` | not redacted | CI's console is public forever; yours is not, and redacting the one copy you are debugging from removes the detail you are debugging. To see what CI *would* have published, re-run a stage piping through `python3 .github/workflows/scripts/redact-stand-log.py`. | +| **Seed manifest transport** | `$RUNNER_TEMP`, discarded with the runner, never uploaded | `deploy/gitops/.deploy/ci-emulation/seed-manifest.json`, gitignored, persists | Locally it has to persist so stage 3 can be re-run without re-seeding. It carries persona addresses, UUIDs, the tenant and in-cluster service URLs — treat it as run-internal and never attach it to anything. | +| **Chart version** | the `publish-chart` job's output — the version that was just published | `--chart-version`, defaulting to the committed `deploy/gitops/.insight-version` | `.insight-version` is written back to the branch by the publishing job, so a checkout always reads a version from *before* the run you are emulating. Always pass `--chart-version` explicitly when you mean a specific one. | +| **Stand address** | the `TEST_STAND_BASE_URL` repository variable | derived from `authenticator.oidc.redirectUri` in the committed values file, or `--base-url` | Same address, different route to it. If the two ever disagree, the committed values file is the one the deployed authenticator will actually redirect to. | +| **Seed persona address** | the `TEST_STAND_SEED_EMAIL` environment secret | `--seed-email`, defaulting to the seeder's committed canonical dev-lead address | The address is a configuration value CI keeps out of the repository; the local default addresses the same person the roster describes. | +| **Step budgets** | per-step `timeout-minutes` (deploy 14, seed 65, smoke 5) | none | helm's own `--timeout 10m` and the seed Job's `activeDeadlineSeconds` are identical in both, and those are the ones that produce a clean diagnosable failure. The GitHub step budget only exists to stop a wedged runner, and a laptop has you instead. | +| **Concurrency** | `group: test-stand-deploy`, `cancel-in-progress: false` | nothing | Two people running the harness at once against the same stand will interleave. Say so in chat before you use `--apply`. | + +### 2.3 Present locally, absent in CI — on purpose + +The harness does four things the workflow does not. All four are read-only, all four are labelled in the output, and none of them changes what the stages run. + +1. **The cluster guard.** `--kubeconfig` and `--expect-cluster` are both mandatory in every mode. Before anything runs it asserts that the kubeconfig's current-context equals the committed inventory's `kubeContext`, that the cluster entry that context points at is named exactly `--expect-cluster`, and — when `--expect-api-server-sha256` is given — that the sha256 of the cluster's API server URL matches. CI needs less because its credential arrives from a branch-restricted environment; a laptop has every kubeconfig you have ever been given. +2. **Read-only stage forms.** `make diff` for deploy, `seed-stand.sh --dry-run` for seed, `pytest --collect-only` plus one unauthenticated `GET /` for smoke. The seed dry run is the most valuable of the three: it performs the *same* cluster discovery the real run performs, so it is simultaneously a rehearsal and an RBAC probe. +3. **Prerequisite checks.** Presence of the `client-secret` key in the `insight-oidc` Secret (presence only — the value is never read into the shell), and the `Accepted` status of the two edge routes. Both are objects the upgrade depends on and does not own; each converts a failure that would otherwise cost a full deploy into a message in the first ten seconds. +4. **The parity report.** Described next. + +### 2.4 How drift is detected rather than assumed + +Two files that must run the same commands will not stay that way by good intentions. Every harness run greps the workflow for the literal command fragments it rehearses and prints `found` or `ABSENT` for each, plus named `BUG` lines for divergences already known to break a run: + +- the workflow passing `--window-days`, which `seed-stand.sh` does not accept and refuses; +- the workflow setting `INSIGHT_STAND_BASE_URL` without `SMOKE_BASE_URL`, which the smoke conftest treats as "not aimed" and refuses; +- the workflow setting `INSIGHT_STAND_PERSONA_PASSWORD` without `SMOKE_PERSONA_PASSWORD`, which `tests/stand/smoke/login.py` never reads. + +Each `BUG` check is an assertion about the workflow's *text*, so it goes quiet the moment CI is fixed. The report is never fatal — a grep cannot prove two paths are identical, and a harness that refused to run because of one would just get bypassed. It is a prompt to go and diff the printed commands against the workflow by eye, which is the only check that actually proves anything. + +The workflow spells `VALUES_FILE`, `CHART_REF` and the rest out in full rather than composing them from `ENV_NAME` partly so those literals stay greppable. If you shorten them, update the anchors in the harness in the same change. + +## 3. Prerequisites + +- `helm`, `kubectl`, `yq`, `jq`, `make`, `uv`, `curl`. `make -C deploy/gitops doctor` checks the first five with version floors and installation hints. +- A kubeconfig for the stand, **outside the repository working tree**, whose current-context is named exactly what `deploy/gitops/environments/test-stand/inventory.yaml` says. Rename yours if it is not: + ```bash + kubectl --kubeconfig config rename-context + kubectl --kubeconfig config use-context + ``` +- A clean working tree if you intend to run the deploy stage, because `make diff` inherits `sync-clean`. +- For the smoke stage in `--apply` mode, the `SMOKE_*` credentials in your environment. The harness never defaults, bridges or prints them; the suite resolves them itself and fails naming the one that is missing. + +## 4. The rehearsal sequence + +This is the order to run things in. Steps 1–4 change nothing anywhere and can be run today, against the live stand, with no coordination. + +### Step 0 — tooling + +```bash +make -C deploy/gitops doctor +uv sync --project tests --frozen +``` + +### Step 1 — read the plan without a cluster + +```bash +./deploy/gitops/scripts/emulate-ci-deploy.sh --print-commands +``` + +Nothing runs — not even the cluster guard. You get every stage command in both its apply and dry-run form, quoted so it can be pasted back into a shell. Open `.github/workflows/deploy-test-stand.yml` beside it and read the two together. This is the step that actually establishes that the local path and the CI path are the same path; everything after it is checking the environment, not the equivalence. + +### Step 2 — pin the API server fingerprint + +Context and cluster names are local aliases. A kubeconfig for a *different* cluster whose entries happen to carry the same names passes every name check. The fingerprint does not: + +```bash +kubectl --kubeconfig config view --minify \ + -o 'jsonpath={.clusters[0].cluster.server}' | shasum -a 256 +``` + +Keep the digest wherever the team keeps stand facts and pass it as `--expect-api-server-sha256` from now on. A digest is safe to write down; the URL it is a digest of is not — this repository is public and a cluster API endpoint is not ours to publish. + +### Step 3 — read-only rehearsal against the live stand + +Run the stages individually, in order, and read the output of each before moving on. + +```bash +KC= +CL= +SHA= + +./deploy/gitops/scripts/emulate-ci-deploy.sh \ + --kubeconfig "$KC" --expect-cluster "$CL" --expect-api-server-sha256 "$SHA" \ + --stage deploy --chart-version +``` + +Renders the chart offline through the committed values file, diffs it against the last render, and then reports on the OIDC client secret and the two edge routes. It never contacts the cluster for the render itself, so a failure here is a repository problem, not a stand problem. + +```bash +./deploy/gitops/scripts/emulate-ci-deploy.sh \ + --kubeconfig "$KC" --expect-cluster "$CL" --expect-api-server-sha256 "$SHA" \ + --stage seed +``` + +Runs the seeder's own `--dry-run`. It discovers the tenant, the datastore coordinates, the seed image and the IdP source type from the cluster and prints the Job it *would* apply. This is the single highest-value read-only step: if discovery works, the real seed's inputs are correct, and if it does not, the message names the flag that fixes it. + +```bash +./deploy/gitops/scripts/emulate-ci-deploy.sh \ + --kubeconfig "$KC" --expect-cluster "$CL" --expect-api-server-sha256 "$SHA" \ + --stage smoke +``` + +Confirms the public URL answers, then collects the suite. Collection needs a seed manifest, which a dry run cannot produce — so on a first pass this step stops with a message saying exactly that. That is the expected outcome, not a failure; it becomes meaningful after Step 5's seed. + +### Step 4 — rehearse as the CI credential + +Everything above ran as you. CI runs as a namespace-scoped ServiceAccount, which is the thing that will actually be refused: + +```bash +./deploy/gitops/scripts/emulate-ci-deploy.sh \ + --kubeconfig "$KC" --expect-cluster "$CL" --expect-api-server-sha256 "$SHA" \ + --stage seed --as-user system:serviceaccount::ci-deployer +``` + +The RBAC probe prints `yes`/`no` per verb for the five permissions the three stages need, with the reason each one is needed. Every `no` is a `helm upgrade` or a seed that will fail on a merge. Note that the probe uses impersonation and therefore reports what the ServiceAccount *may* do; the stages themselves still run as your kubeconfig. + +### Step 5 — the first real run, one stage at a time + +Only now, and only with a deliberate choice to change a published stand. Say so wherever the team coordinates first — the workflow's concurrency group has no local equivalent, and two people applying at once will interleave. + +```bash +./deploy/gitops/scripts/emulate-ci-deploy.sh \ + --kubeconfig "$KC" --expect-cluster "$CL" --expect-api-server-sha256 "$SHA" \ + --chart-version \ + --stage deploy --apply --i-know-this-deploys yes-deploy-test-stand +``` + +Then `--stage seed --apply …`, which writes the manifest into the artefact directory, and then `--stage smoke --apply …`, which reads it. Running them separately the first time is worth the extra typing: each stage's failure mode is different, and a combined `--stage all` run makes the second failure harder to attribute than the first. + +`--apply` without the token is refused. The token is deliberately the same string the Makefile's protected-environment `CONFIRM=` uses, so there is one string to remember; and because it names the environment, a command line recalled from history for a different stand fails closed instead of deploying. + +Once each stage has passed individually, run the whole thing end to end — that is the run that most closely resembles what CI will do: + +```bash +./deploy/gitops/scripts/emulate-ci-deploy.sh \ + --kubeconfig "$KC" --expect-cluster "$CL" --expect-api-server-sha256 "$SHA" \ + --chart-version \ + --apply --i-know-this-deploys yes-deploy-test-stand +``` + +### Step 6 — only now, enable the workflow + +By this point you know that the values file renders, that the release upgrades and reports the chart it was asked for, that the three envFrom services restart cleanly, that the routes are Accepted, that the seeder can discover everything it needs, that the manifest capture produces a document the suite can read, and that the CI ServiceAccount holds every permission the path uses. What is left for the first CI run to discover is confined to [§2.2](#22-necessarily-different): the runner's OS, the environment's secrets, and the checkout. + +Populate the `insight-test-stand` GitHub environment, restrict it to `main`, and merge the workflow. Re-run `--print-commands` and read the parity report after any later edit to either file. + +## 5. Reading a failure + +The harness runs the workflow's own `stand-diagnostics.sh` on any stage failure, with the same positional interface (` `) and the same ambient-kubeconfig assumption. You therefore see exactly the evidence a red CI run would publish — no more, which is the point. It emits GitHub workflow commands (`::group::`, `::error::`); on a laptop those appear as literal text, and that is left alone deliberately, because a second output mode would make the local and the CI evidence hard to compare. + +Nothing is rolled back. That is the intended disposition and the reason the upgrade runs with `--wait` and deliberately without `--atomic`: a rollback deletes the pods that hold the reason, leaving a red run and a healthy-looking stand, which is the one combination nobody can diagnose. Recovery is the next deploy, or: + +```bash +make -C deploy/gitops status ENV=test-stand +make -C deploy/gitops rollback ENV=test-stand +``` + +Both work for this environment unchanged, and both assert the current context against the inventory first. + +If the harness produced output you want to share, remember it names your cluster, your context and your kubeconfig path, and is not redacted. Pipe the stage through `python3 .github/workflows/scripts/redact-stand-log.py` to see the CI-safe form, and scrub the rest by hand. + +## 6. Safety rules that are not negotiable + +- **Read-only is the default, and it is a real default.** No stage mutates anything without both `--apply` and the typed token. +- **The guard runs in every mode**, including dry runs — because the seed dry run does reach the cluster. +- **Nothing secret is printed.** The OIDC client secret lives in a variable for the length of one `helm upgrade` and is never echoed; the printed command shows the substitution expression instead. `set -x` is not used anywhere in the script, for that reason. +- **The seed manifest is run-internal.** It stays under the gitignored artefact directory. It is never an artifact, never an attachment, never a paste. +- **The kubeconfig lives outside the working tree.** Enforced, not advised: `make diff` fails on any file git reports, so a kubeconfig next to the values file breaks the stage it is supposed to help you rehearse. + +## 7. Known gaps and things to reconcile + +- **The environment's hand-deploy runbook and the workflow have drifted apart at least once.** `deploy/gitops/environments/test-stand/README.md` and the workflow are two descriptions of the same sequence, maintained separately. When they disagree, the workflow is what runs on a merge and the harness follows the workflow. Treat a disagreement as a bug in the README, and fix it in the same change. +- **The Makefile still hardcodes `--atomic`.** It does not affect this path today, because neither the workflow nor the harness uses `make deploy`. It does mean `make deploy ENV=test-stand` remains the wrong tool for this stand, and it is worth the one-line `ATOMIC ?= --atomic` change so that stops being true. +- **A scripted password login depends on the stand's realm.** If the realm federates to an external provider and holds no local users, no amount of test code makes the smoke's login work; the suite fails saying so rather than skipping. That is a stand-configuration decision, not a harness one, and it is the one precondition Steps 1–4 cannot rehearse. +- **`.insight-version` is not the version CI just published.** It is written back to the branch at the end of the publishing job, so any checkout reads the previous one. Pass `--chart-version` explicitly whenever the version matters. +- **The harness has no concurrency control.** CI coalesces runs on `test-stand-deploy` with `cancel-in-progress: false`; two engineers with `--apply` have nothing but each other. + +## Traceability + +- Harness: `deploy/gitops/scripts/emulate-ci-deploy.sh` +- Workflow: `.github/workflows/deploy-test-stand.yml` +- Environment: `deploy/gitops/environments/test-stand/` (and its `README.md` for the hand-deploy runbook) +- Diagnostics: `.github/workflows/scripts/stand-diagnostics.sh`, `.github/workflows/scripts/redact-stand-log.py` +- Seeder: `src/ingestion/tools/seed/seed-stand.sh`, `src/ingestion/tools/seed/PROFILE.md` +- Smoke suite: `tests/stand/smoke/` diff --git a/docs/components/deployment/specs/sop/credentials-runbook.md b/docs/components/deployment/specs/sop/credentials-runbook.md new file mode 100644 index 000000000..5f0aba12a --- /dev/null +++ b/docs/components/deployment/specs/sop/credentials-runbook.md @@ -0,0 +1,490 @@ +# SOP — Test-Stand CI Credentials + +**Audience**: platform engineers with an admin kubeconfig for the test-stand cluster, and repo admins on `constructorfabric/insight`. +**Covers**: provisioning, verifying, storing, rotating, revoking and cleaning up the credential CI uses to deploy the Insight umbrella chart onto the test stand (issue #2244, decision D5). +**Last verified**: 2026-08-09. + +> Every value in this document is a placeholder. Substitute your own and never +> paste a real one back in — this repository is public, and so are its issues, +> pull requests and workflow run logs. + +--- + +## 0. What exists, and where it lives + +| Thing | Where it lives | Who can read it | +|---|---|---| +| Admin kubeconfig for the stand | An operator's laptop / password manager | Humans only. **Never** goes into GitHub. | +| `ServiceAccount insight/ci-deployer` + its RBAC | The cluster | Anyone with cluster read access | +| The CI kubeconfig (server + CA + SA token), base64-encoded | GitHub environment `insight-test-stand`, secret `TEST_STAND_KUBECONFIG` | Jobs that declare that environment, on `main` only | +| Persona email / password, OIDC client secret | Same GitHub environment | Same | +| Public stand URL | Same environment, as **variable** `TEST_STAND_BASE_URL` (not a secret) | Same, and it is public anyway | + +The split is the whole point of D5: **admin kubeconfigs stay human-only.** CI gets +a credential that is namespace-scoped and destroyable, and nothing else. + +The provisioning tool is +[`deploy/gitops/scripts/provision-ci-deployer.sh`](../../../../../deploy/gitops/scripts/provision-ci-deployer.sh). +Its header explains *why* a ServiceAccount token beats a copied admin kubeconfig +(one-line answer: a token can be revoked; a client certificate cannot be revoked +without rotating the cluster CA). Read it once before your first run. + +--- + +## 1. Prerequisites + +1. `kubectl` ≥ 1.27 and `git` on PATH — `make doctor` from `deploy/gitops/` covers + the wider toolchain. +2. An admin kubeconfig for the target cluster, readable at a path you know. + This runbook writes it as `~/.kube/-admin.kubeconfig`. +3. The **exact cluster name** that kubeconfig's context resolves to. Find it + without acting on anything: + + ```bash + kubectl --kubeconfig ~/.kube/-admin.kubeconfig config current-context + kubectl --kubeconfig ~/.kube/-admin.kubeconfig config view -o json \ + | jq -r '.contexts[] | "\(.name) -> \(.context.cluster)"' + ``` + + You will pass that cluster name as `--expect-cluster`. The script refuses to + do anything if the kubeconfig resolves to a different cluster, which is the + mechanical guard against provisioning a standing grant on the wrong stand. +4. The application namespace already exists (`insight` by default). See + §7 "First install into an empty namespace" if it does not. +5. `gh` authenticated with admin rights on the repository, for §4. + +--- + +## 2. Provision + +### 2.1 Read the plan first + +The script is **dry-run by default**. It reads the cluster, prints the exact +manifests it would apply and the exact assertions it would run, and exits +without writing anything: + +```bash +cd deploy/gitops +./scripts/provision-ci-deployer.sh \ + --kubeconfig ~/.kube/-admin.kubeconfig \ + --expect-cluster '' +``` + +Read the printed manifests. You are looking for four things: + +- the binding is a `RoleBinding`, not a `ClusterRoleBinding`; +- its `roleRef` is `ClusterRole/admin` — a cluster role bound namespace-wide, which + is not the same as a cluster-wide grant; +- the supplemental `Role` lists only `gateway.networking.k8s.io`, `argoproj.io`, + `onepassword.com` and `cert-manager.io`; +- the token `Secret` carries no `data:` block (the token controller fills it). + +If the cluster name does not match, the script prints a refusal banner and exits +2 having touched nothing. That is the expected outcome of a typo — do not "fix" +it by changing `--expect-cluster` until you are certain which cluster you mean. + +### 2.2 Apply + +```bash +./scripts/provision-ci-deployer.sh \ + --kubeconfig ~/.kube/-admin.kubeconfig \ + --expect-cluster '' \ + --out ~/.kube/insight-test-stand-ci-deployer.kubeconfig \ + --apply +``` + +What happens, in order: + +1. the manifests are applied (idempotent — re-running is safe and is the + supported way to repair drift); +2. the script waits for the token controller to populate the Secret; +3. the kubeconfig is assembled at `--out` with **mode 0600** and its contents are + never printed; +4. the scope assertions run, and a single failure aborts with a non-zero exit and + a "do not put this kubeconfig in a GitHub environment" message. + +`--out` defaults to `~/.kube/insight-test-stand-ci-deployer.kubeconfig`. The +script refuses any path inside a git work tree unless that path is gitignored — +a bearer token one `git add -A` away from a public repository is not a risk worth +carrying. + +### 2.3 Context name + +The generated kubeconfig's context is named `insight-test-stand` by default, +matching the gitops convention `insight-` documented in +`deploy/gitops/README.md`. That matters because the Makefile's `kube-ctx` target +refuses to act unless `kubectl config current-context` equals the environment +inventory's `kubeContext`. If your inventory says something else, either fix the +inventory or pass `--context-name ` when provisioning. + +--- + +## 3. Verify the scope + +The apply run verifies automatically. Re-verify at any time — after a cluster +upgrade, after someone edits the Role, before you trust a run — without +re-provisioning: + +```bash +./scripts/provision-ci-deployer.sh \ + --kubeconfig ~/.kube/-admin.kubeconfig \ + --expect-cluster '' \ + --out ~/.kube/insight-test-stand-ci-deployer.kubeconfig \ + --verify-only +``` + +### 3.1 What it asserts + +**Must be `yes`** — the deploy and seed stages genuinely need these: + +| Check | Why the deploy needs it | +|---|---| +| `create jobs -n insight` | the seed stage renders and applies a Job | +| `get pods/log -n insight` | seed follows the Job's log; diagnostics tail failed containers | +| `create secrets -n insight` | helm stores release state as Secrets in the namespace | +| `update deployments.apps -n insight` | `helm upgrade`, and the post-upgrade rollout restart | +| `create roles.rbac.authorization.k8s.io -n insight` | the chart renders its own reconcile RBAC | +| `create workflowtemplates.argoproj.io -n insight` | the chart renders WorkflowTemplates and CronWorkflows | +| `update httproutes.gateway.networking.k8s.io -n insight` | the edge routes are applied alongside the release | + +**Must be `no`** — this is the containment claim: + +| Check | | +|---|---| +| `'*' '*' --all-namespaces` | the blanket check | +| `list namespaces --all-namespaces` | cannot enumerate the cluster | +| `create namespaces --all-namespaces` | see §7 | +| `get secrets --all-namespaces` | the datastore namespaces' credentials stay out of reach | +| `list nodes --all-namespaces` | no cluster-scoped reads | +| `create clusterrolebindings… --all-namespaces` | cannot widen its own grant | +| `get pods -n kube-system` | no other namespace, control plane included | + +### 3.2 One RBAC subtlety worth knowing + +`kubectl get namespace insight` may succeed while `kubectl get namespaces` +(the list) is forbidden, and both are correct. A request naming a single +namespace is evaluated *as a request in that namespace*, so a RoleBinding there +can satisfy it; a list is a cluster-scoped request that only a +ClusterRoleBinding can satisfy. Every "must be `no`" assertion above passes +`--all-namespaces`, which sends an empty namespace, so the assertions test what +they claim to test. Do not "fix" a `yes` from `kubectl get ns insight` — it is +not a scope leak. + +### 3.3 Manual spot check + +```bash +export KUBECONFIG=~/.kube/insight-test-stand-ci-deployer.kubeconfig +kubectl auth whoami # system:serviceaccount:insight:ci-deployer +kubectl get pods # works — the context defaults to the namespace +kubectl get ns # Error from server (Forbidden) +kubectl -n kube-system get pods # Error from server (Forbidden) +unset KUBECONFIG +``` + +--- + +## 4. Create the GitHub environment and load the secrets + +Set the repo once for the commands below: + +```bash +REPO=constructorfabric/insight +ENVIRONMENT=insight-test-stand +``` + +### 4.1 Create the environment, restricted to `main` + +```bash +# Create it, and declare that it uses a custom branch allow-list. +gh api --method PUT "repos/$REPO/environments/$ENVIRONMENT" --input - <<'JSON' +{ + "deployment_branch_policy": { + "protected_branches": false, + "custom_branch_policies": true + } +} +JSON + +# Then allow exactly one branch. +gh api --method POST \ + "repos/$REPO/environments/$ENVIRONMENT/deployment-branch-policies" \ + -f name='main' +``` + +`protected_branches: true` would have been shorter, but it allows *every* +protected branch — this repository also protects `release-**`, and a +release-branch build must not reach the test stand. The custom allow-list says +`main` and means `main`. + +Confirm: + +```bash +gh api "repos/$REPO/environments/$ENVIRONMENT/deployment-branch-policies" \ + --jq '.branch_policies[].name' # -> main +``` + +### 4.2 Load the secrets + +| Secret | Value | Notes | +|---|---|---| +| `TEST_STAND_KUBECONFIG` | **base64 of** the file from §2.2 | the only credential CI needs for the cluster | +| `TEST_STAND_SEED_EMAIL` | the address `seed-stand.sh --email` is given, e.g. `email_development_lead@company.nonpresent` | kept as a secret rather than a variable purely so it is masked in run logs | +| `TEST_STAND_PERSONA_PASSWORD` | the password the smoke stage logs in with | reaches pytest as `INSIGHT_STAND_PERSONA_PASSWORD` | +| `TEST_STAND_OIDC_CLIENT_SECRET` | the confidential OIDC client secret | **only if** the deploy has to inject it; see §4.3 | + +The kubeconfig goes in **base64-encoded**, single-line. The workflow decodes it +with `base64 -d` before writing it to the runner's temp directory. A raw +multi-line YAML secret survives `gh secret set` but is fragile in transit +(trailing-newline handling, and GitHub's log masker only masks the value as a +whole, so a multi-line secret is effectively unmasked line by line). + +```bash +# From a file, no shell history exposure. `tr -d '\n'` makes the line-wrapping +# behaviour identical on macOS and Linux — GNU's `base64 -w0` is not portable. +base64 < ~/.kube/insight-test-stand-ci-deployer.kubeconfig | tr -d '\n' \ + | gh secret set TEST_STAND_KUBECONFIG --env "$ENVIRONMENT" --repo "$REPO" + +# Typed values: omit --body and let gh prompt, so the value never enters +# the shell history or the process table. +gh secret set TEST_STAND_SEED_EMAIL --env "$ENVIRONMENT" --repo "$REPO" +gh secret set TEST_STAND_PERSONA_PASSWORD --env "$ENVIRONMENT" --repo "$REPO" + +gh secret list --env "$ENVIRONMENT" --repo "$REPO" +``` + +Never `echo '' | gh secret set …` and never `--body ''`: both put +the cleartext into shell history and into `ps` output for the duration of the +call. + +The public stand URL is not a secret and should not be one — a masked value in a +log is harder to debug and buys nothing. The workflow reads it as a **variable**: + +```bash +gh variable set TEST_STAND_BASE_URL --env "$ENVIRONMENT" --repo "$REPO" \ + --body 'https://' + +gh variable list --env "$ENVIRONMENT" --repo "$REPO" +``` + +### 4.3 When `TEST_STAND_OIDC_CLIENT_SECRET` is needed + +Only when the deploy path is the one that materialises the authenticator's +config Secret. The gitops path reads the client secret from the in-cluster +Secret `insight-oidc` under the key `oidc-client-secret`. A stand whose Secret +uses a different key name has the secret present but invisible to that lookup, +and the composed config silently lands with an **empty** client secret — the +confidential-client token exchange then fails and login breaks. + +Preferred fix: add the expected key to the existing in-cluster Secret, once, by +hand, with a human credential. Then this GitHub secret is unnecessary and should +not exist. Add it only if you consciously choose to inject the value from CI +instead, and record that choice in the workflow header. + +### 4.4 Why the secrets may look missing to a job + +Environment secrets are readable only by a job that declares +`environment: `. GitHub Actions does **not** allow `environment:` on a job +that calls a reusable workflow with `uses:` — the declaration has to go on the +jobs *inside* the called workflow. A job whose `${{ secrets.TEST_STAND_* }}` is +empty is almost always this, not a missing secret. Branch policies still +evaluate against the caller's ref, so the `main`-only restriction holds either +way. + +--- + +## 5. Rotate + +**Cadence**: every 90 days, and immediately on any of — an operator with a copy +leaves the team, the value is pasted anywhere it should not be, a runner or +laptop that held it is compromised, or the cluster CA is rotated (which +invalidates the embedded CA bundle even though the token itself survives). + +Rotation deletes the old token first, so there is a window in which the secret +stored in GitHub is dead. Do it when no deploy is in flight, and re-upload +immediately. + +```bash +# 1. Confirm nothing is deploying: the workflow's concurrency group is +# `test-stand-deploy`, and it coalesces rather than cancels. +gh run list --repo "$REPO" --workflow build-images.yml --limit 5 + +# 2. Plan, then rotate. +./scripts/provision-ci-deployer.sh \ + --kubeconfig ~/.kube/-admin.kubeconfig \ + --expect-cluster '' \ + --out ~/.kube/insight-test-stand-ci-deployer.kubeconfig \ + --rotate # prints the plan and exits + +./scripts/provision-ci-deployer.sh \ + --kubeconfig ~/.kube/-admin.kubeconfig \ + --expect-cluster '' \ + --out ~/.kube/insight-test-stand-ci-deployer.kubeconfig \ + --rotate --apply # deletes the old token, mints a new one, + # rewrites --out, re-runs the assertions + +# 3. Upload the new one (base64, single line — see §4.2). +base64 < ~/.kube/insight-test-stand-ci-deployer.kubeconfig | tr -d '\n' \ + | gh secret set TEST_STAND_KUBECONFIG --env "$ENVIRONMENT" --repo "$REPO" + +# 4. Prove it end to end before you walk away. +gh workflow run build-images.yml --repo "$REPO" --ref main +``` + +Then do §8 (cleanup). + +Rotating the persona password or the seed email is just `gh secret set` again — +no cluster action — but a persona password also has to change wherever the +identity provider holds it, and the two must be changed in the same sitting or +the smoke stage fails on the next merge. + +--- + +## 6. Revoke + +Two levels. Both are immediate: the API server re-validates a ServiceAccount +token against the Secret and against the ServiceAccount's UID on every request, +so the credential stops working as soon as the delete propagates. (That +re-validation depends on `--service-account-lookup`, on by default since 1.7 — +the confirmation step below is what actually proves it on *your* cluster, so do +not skip it.) + +**Level 1 — kill the token, keep the identity.** Use when the credential leaked +but you still want CI to work after re-issuing. + +```bash +./scripts/provision-ci-deployer.sh \ + --kubeconfig ~/.kube/-admin.kubeconfig \ + --expect-cluster '' \ + --revoke --apply +``` + +**Level 2 — remove the identity entirely.** Use when decommissioning the stand +or the CI integration. + +```bash +./scripts/provision-ci-deployer.sh \ + --kubeconfig ~/.kube/-admin.kubeconfig \ + --expect-cluster '' \ + --purge --apply +``` + +Either way, clear the stored copy too — a dead credential in a secret store is a +false sense of coverage and an obstacle to the next person debugging: + +```bash +gh secret delete TEST_STAND_KUBECONFIG --env "$ENVIRONMENT" --repo "$REPO" +# Decommissioning the whole integration: +gh api --method DELETE "repos/$REPO/environments/$ENVIRONMENT" +``` + +Confirm the credential is dead: + +```bash +KUBECONFIG=~/.kube/insight-test-stand-ci-deployer.kubeconfig kubectl get pods +# -> error: You must be logged in to the server (Unauthorized) +``` + +> This is exactly the operation that is **impossible** with a copied admin +> kubeconfig. Those authenticate with a client certificate, and Kubernetes +> consults no CRL and no OCSP responder — the only revocation is rotating the +> cluster CA, which invalidates every other client certificate at the same time +> and needs a control-plane restart. That asymmetry is the reason this whole +> runbook exists. + +--- + +## 7. Troubleshooting + +**`the token controller never populated '' within Ns`** +The cluster runs without the legacy token controller (some hardened +distributions disable it). There is no long-lived token to mint. Either enable +it, or accept a bound token from `kubectl create token` plus a refresh mechanism +— and note that a static GitHub secret cannot hold an expiring token, so this +becomes a workflow change, not a secret change. + +**`attempt to grant extra privileges` during `helm upgrade`** +RBAC escalation prevention: a subject cannot create a Role granting permissions +it does not itself hold, and `admin` does not carry `escalate`. The chart renders +its own reconcile Role with `argoproj.io` and `onepassword.com` rules. You almost +certainly ran with `--no-supplement`, or someone trimmed the supplemental Role. +Re-run provisioning without `--no-supplement`. + +**`namespaces is forbidden` on a first install** +`helm upgrade --install --create-namespace` POSTs a Namespace at cluster scope, +but only when the release does not yet exist. This credential cannot create +namespaces, by design. Create the namespace once, by hand, with a human +credential: + +```bash +kubectl --kubeconfig ~/.kube/-admin.kubeconfig create namespace insight +``` + +Do **not** grant namespace-create to CI to make this go away. Upgrades of an +existing release never reach that code path. + +**`Error: context "…" does not exist` from the gitops Makefile** +The generated kubeconfig's context name and the environment inventory's +`kubeContext` disagree. Re-provision with `--context-name `, or +pass `KUBE_CTX=` on the make invocation. + +**A Gateway or an object in another namespace is unreadable** +Expected. The gateway controller's own namespace is out of scope for this +credential. A preflight that inspects the `Gateway` object needs either a human +credential or a separate, equally narrow read-only grant in that namespace — +file it as a follow-up rather than widening this one. + +**The workflow reports empty secrets** +See §4.4. + +--- + +## 8. Cleanup — do this every time + +The local copy of the CI kubeconfig is a live credential. Treat it like one. + +```bash +# 1. Nothing sensitive is sitting in the work tree. +git -C status --porcelain + +# 2. Every local kubeconfig is owner-only. Do this for the admin one too. +chmod 600 ~/.kube/insight-test-stand-ci-deployer.kubeconfig +chmod 600 ~/.kube/-admin.kubeconfig +chmod 700 ~/.kube +ls -l ~/.kube/*.kubeconfig # expect -rw------- on each + +# 3. Remove throwaway copies. Prefer an overwriting delete where you have one. +# macOS: rm -P Linux (coreutils): shred -u +rm -P /tmp/ci-deployer.* 2>/dev/null || true + +# 4. If you typed a secret inline despite §4.2, drop it from history now. +history | tail -40 # find the offending entries +# zsh: fc -W after editing ~/.zsh_history bash: edit ~/.bash_history +``` + +Decide deliberately whether to keep the local CI kubeconfig at all: + +- **Keep it** (mode 0600, in `~/.kube`, never in the repo) if you expect to + re-verify the scope with `--verify-only`. +- **Delete it** once it is in the GitHub environment. Re-issuing is one + `--rotate --apply` away, and a credential that does not exist on your laptop + cannot leak from it. This is the recommended default. + +Finally, confirm nothing sensitive reached the repository. The gitops tree ships +`.gitleaks.toml` for the pre-commit hook; run it explicitly if you have it: + +```bash +gitleaks detect --source --config /deploy/gitops/.gitleaks.toml +``` + +--- + +## 9. Done when + +- `--verify-only` passes every assertion against the credential CI will use. +- `gh api "repos/$REPO/environments/$ENVIRONMENT/deployment-branch-policies"` + lists exactly `main`. +- `gh secret list --env "$ENVIRONMENT"` shows the kubeconfig, the seed email and + the persona password (and the OIDC client secret only if §4.3 applies). +- One full deploy run on `main` is green. +- No kubeconfig, token or password exists anywhere in the repository work tree, + and every local kubeconfig is mode 0600. From 6071508c0e4d6c938f7cb3faf0e481362a89477e Mon Sep 17 00:00:00 2001 From: Konstantin Tursunov Date: Sun, 9 Aug 2026 12:13:35 +0800 Subject: [PATCH 03/59] ci(stand): deploy, seed and smoke the published chart on the test stand MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit Closes the gap between "images built, chart published" and "a release anyone can actually use". On merge to main, `build-images.yml` now hands the version it just published to a reusable workflow that installs that exact chart on the shared test stand, seeds it, and proves a login and a metric through the public URL. This is an after-merge alarm, not a gate, and it cannot be anything else: the artefact it installs does not exist until publish-chart has run. Merge blocking stays with the merge-queue lanes; a red run here means main is broken, and the merge author owns it — fix forward if the cause is small, revert if it is not. Three separately-named stages with their own timeouts, so a red run says which one died without opening a log: deploy (no `--atomic`, because a rollback destroys the evidence and leaves the stand silently running an older chart than main), seed, then smoke — which never runs after a failed seed. The repo is public, so every run log is public forever. `stand-diagnostics.sh` is therefore a curated allowlist — release status, a pod table, warning reasons without messages, and the tail of failed containers only — piped through `redact-stand-log.py`. No describes, no environment dumps, no uploaded artefacts. Anything deeper needs the kubeconfig, deliberately. The diagnostics enumerate `--deployed --failed --pending --uninstalling` rather than `helm list --all`: `--all` is a Helm 3 flag that Helm 4 removed, and a diagnostics script runs precisely when something is already broken, so it can never be the thing that breaks. Refs #2244 Signed-off-by: Konstantin Tursunov --- .github/workflows/build-images.yml | 58 ++ .github/workflows/deploy-test-stand.yml | 792 ++++++++++++++++++ .github/workflows/scripts/redact-stand-log.py | 210 +++++ .../workflows/scripts/stand-diagnostics.sh | 236 ++++++ 4 files changed, 1296 insertions(+) create mode 100644 .github/workflows/deploy-test-stand.yml create mode 100755 .github/workflows/scripts/redact-stand-log.py create mode 100755 .github/workflows/scripts/stand-diagnostics.sh diff --git a/.github/workflows/build-images.yml b/.github/workflows/build-images.yml index 74b5f54a6..5bc9960f7 100644 --- a/.github/workflows/build-images.yml +++ b/.github/workflows/build-images.yml @@ -1784,6 +1784,16 @@ jobs: || needs.changes.outputs.umbrella == 'true' ) runs-on: ubuntu-latest + # The deploy contract. `chart_version` is what a downstream job installs, and + # it must come from here rather than from deploy/gitops/.insight-version in a + # checkout: that file is written by the `Bump umbrella version` step but only + # reaches the branch in the [skip ci] commit at the END of this job, so any + # checkout of the trigger SHA reads the PREVIOUS release. `chart_digest` is + # the identity the SLSA attestation was made against — carried so a consumer + # can verify what it is about to install rather than trust the tag. + outputs: + chart_version: ${{ steps.umbrella.outputs.version }} + chart_digest: ${{ steps.helm_push.outputs.digest }} steps: # Mint a short-lived (1h) installation token from the # insight-chart-release-bot GitHub App so the auto-commit push at the @@ -2039,3 +2049,51 @@ jobs: echo "recompute the bump against the new tip." >&2 exit 1 fi + + # ─── deploy-test-stand ──────────────────────────────────────────────────── + # The chart this run just published, put on the shared test stand, seeded, and + # smoked through its public URL. Issue #2244. + # + # This is an AFTER-MERGE ALARM, not a gate. It cannot be a gate: the artefact + # it installs does not exist until publish-chart has run, and publish-chart + # runs on push to main. What guards the merge is the merge-queue lanes; what + # this guards is the claim that the artefact those lanes approved installs and + # works. A red run here means main is broken — the merge author owns it, fixes + # forward if the cause is small, and reverts if it is not. + # + # `needs: publish-chart` waits for the WHOLE job, so there is no race with the + # version bump this job commits back to the branch. The hazard runs the other + # way, and the guard below is deliberate about it: publish-chart can push and + # attest the chart and STILL end in `failure`, because its final + # `git push origin HEAD:main` has no rebase-retry and exits 1 when the tip + # moved under it. In that case a chart exists in the registry that this deploy + # will not install. That is the right disposition — the branch and the stand + # must not disagree about what is deployed — but it is a choice, not an + # accident, and re-running the failed publish-chart job is the recovery. + # + # Do NOT rewrite this as `always()` plus a non-empty version check. + # `steps.umbrella.outputs.version` is set BEFORE `helm push`, so on a push + # failure the output holds a version that was never published, and the deploy + # would spend ten minutes failing to pull a chart that does not exist. + deploy-test-stand: + needs: [publish-chart] + # Re-asserts main even though publish-chart is upstream: publish-chart's own + # guard also permits `release-**` branches and `workflow_dispatch`, and + # inheriting it silently would deploy a release-branch chart to the test + # stand. A manual redeploy is done by dispatching deploy-test-stand.yml + # directly with the version — that is what its `workflow_dispatch` trigger + # is for — rather than by widening this condition. + if: | + needs.publish-chart.result == 'success' + && github.event_name == 'push' + && github.ref == 'refs/heads/main' + # Narrower than the workflow-level block (contents: write + packages: write + + # id-token: write + attestations: write), following the per-job narrowing + # this file already does at the CVE-gate jobs. The deploy reads the repo and + # talks to a cluster; it publishes nothing. + permissions: + contents: read + uses: ./.github/workflows/deploy-test-stand.yml + with: + chart_version: ${{ needs.publish-chart.outputs.chart_version }} + secrets: inherit diff --git a/.github/workflows/deploy-test-stand.yml b/.github/workflows/deploy-test-stand.yml new file mode 100644 index 000000000..962614f9b --- /dev/null +++ b/.github/workflows/deploy-test-stand.yml @@ -0,0 +1,792 @@ +name: Deploy test stand + +# Take the umbrella chart that build-images.yml has just published, put it on the +# shared test stand, seed it, and prove a person can sign in and see data. Issue +# #2244's three acceptance criteria, and nothing else. +# +# ── WHY THIS IS AN ALARM AND NOT A GATE ───────────────────────────────────── +# +# The thing under test does not exist until the merge has happened. `publish-chart` +# runs on push to main; the chart version this workflow deploys is minted by that +# job, so there is no point before the merge at which a deploy of it could be +# required. That is not a limitation to be engineered around — it is the shape of +# the problem. What guards the merge is the existing merge-queue lanes (ci.yml, +# e2e-stand.yml, e2e-bronze-to-api.yml); what this workflow guards is the claim +# that the artefact those lanes approved actually installs and works. +# +# So the failure mode is an ALARM: a red X on the merge commit. There is no +# freeze, no automatic revert, no rollback. A red run means the stand is broken +# and, by construction, so is main. +# +# RED-RUN OWNERSHIP. The author of the merge commit owns the red run. Fix +# forward if the cause is obvious and small; revert if it is not. "Someone will +# look at it" is how a permanently red stand happens, and a permanently red +# stand is worth less than no stand at all — it stops being read. +# +# The stand's state is CI's: every merge redeploys and reseeds it. Do not put +# hand-made data on it, do not point a demo at it expecting it to be stable, and +# do not `helm upgrade` it by hand except to recover from a failure this workflow +# left in place on purpose (see "deploy" below). +# +# ── WHY ONE JOB AND NOT THREE ─────────────────────────────────────────────── +# +# The three stages are steps in one job, each with its own `timeout-minutes`, +# rather than three jobs. Three jobs would be tidier to read in the UI and would +# cost two things that matter more: +# +# * The seed manifest would have to travel between jobs, and the only transport +# is an artifact. That manifest names every seeded persona — emails, UUIDs, +# the tenant. On a public repository an uploaded artifact is a downloadable +# copy of that for anyone who wants one. Kept inside one job it lives in +# $RUNNER_TEMP and dies with the runner. +# * The kubeconfig would have to be re-materialised in each job, tripling the +# number of places a credential is written to disk. +# +# Steps also give "smoke never runs after a failed seed" for free: a step without +# `if:` does not run once an earlier step has failed. Nothing enforces that +# ordering except the absence of `if: always()`, which is why every `if:` in this +# file is deliberate and worth a second look in review. +# +# ── WHAT THIS DELIBERATELY DOES NOT DO ────────────────────────────────────── +# +# Scope is exactly the three acceptance criteria. Everything below is a known gap +# with a follow-up issue, NOT an oversight: +# +# * No alerting. The red X is the whole notification story. +# * No rollback, and no `--atomic`. A failed upgrade is LEFT failed so the next +# person to look has the evidence (see the deploy stage). +# * No infrastructure bootstrap. Datastores, the gateway, cert-manager and the +# IdP realm are somebody else's lifecycle; this workflow installs the +# application chart and nothing beneath it. +# * No `kubectl rollout restart` after the upgrade. The composed *-config +# Secrets reach pods through `envFrom`, which is read once at container +# start, so a changed coordinate can produce a healthy-looking release with +# stale configuration. Adding the restart is a real improvement and a real +# behaviour change; it belongs in its own change with its own reasoning. +# * No exact-value assertions on seeded data. The smoke proves shape and +# non-emptiness. Reading a number off a running stand and asserting it back +# is a test of nothing. +# * No second environment. `ENV=test-stand` is hardcoded; a second stand gets a +# second gitops environment and an input, not a copy of this file. +# +# ── EVERY LINE THIS PRINTS IS PUBLIC ──────────────────────────────────────── +# +# This repository is public, so run logs are public, forever, to everyone. The +# rules that follow from that are not negotiable and are enforced structurally +# rather than by care: +# +# * All cluster and seeder output goes through +# .github/workflows/scripts/redact-stand-log.py before it reaches the +# console. The raw copy stays in $RUNNER_TEMP. +# * Failure diagnostics are an allowlist — +# .github/workflows/scripts/stand-diagnostics.sh — not "dump everything and +# hope". No `kubectl describe`, no `-o yaml`, no environment dumps, no +# artifact uploads. +# * Every credential arrives from the `insight-test-stand` GitHub environment. +# Nothing about the stand's identity, addresses or accounts is committed here. +# +# ── WHY THE HELM CALL IS SPELLED OUT AND NOT `make deploy` ────────────────── +# +# Reusing `deploy/gitops`'s own entry point was the first design and it is the +# wrong one FOR THIS STAND. `make deploy` → `deploy-insight` depends on +# `apply-app-secrets`, which: +# +# * hard-requires at least one *-sealedsecret.yaml under +# environments//sealed-secrets/insight/ and `kubectl apply`s it. This +# cluster runs no sealed-secrets controller and has no SealedSecret CRD, and +# the target has no skip switch; +# * then runs compose-app-secrets.sh, which overwrites the three chart-owned +# *-config Secrets and reads the OIDC client secret from a key name this +# stand's Secret does not use — writing a BLANK client secret and breaking +# every login while the release still reports `deployed`. +# +# and `deploy-insight` additionally hardcodes `--atomic` after +# $(HELM_UPGRADE_FLAGS), so no existing variable can turn it off (a later +# `--atomic` re-sets what an earlier `--atomic=false` cleared) — which would roll +# a failed upgrade back and destroy the evidence this stand exists to produce. +# +# So the deploy stage runs the invocation written down in +# deploy/gitops/environments/test-stand/README.md, step by step. "CI and humans +# run the same code path" is preserved by that README plus +# deploy/gitops/scripts/emulate-ci-deploy.sh, which runs these same commands from +# a laptop and prints a parity report against THIS FILE. If you change a command +# here, run that harness: it greps for each command's anchor and tells you which +# one drifted. `make diff`, `make status` and `make rollback` do still work for +# this environment and are the right tools for their jobs. +# +# If that harness's parity report says `make deploy` and `ATOMIC=` are ABSENT +# from this file: that is this decision, not a regression. The authority is +# deploy/gitops/environments/test-stand/README.md, "Why not `make deploy`" — +# the sealed-secrets prerequisite alone makes that target unable to run against +# this stand at all, with or without an `--atomic` knob. +# +# ── PRECONDITIONS THIS WORKFLOW ASSERTS RATHER THAN ASSUMES ───────────────── +# +# Each is checked with a named error rather than left to fail obscurely: +# +# 1. `deploy/gitops/environments/test-stand/` exists and its `kubeContext` +# matches the kubeconfig in the environment secret. That comparison is the +# mechanical guard that a deploy cannot land on a cluster nobody meant to +# touch — the credential alone is not a guarantee, because a credential is +# whatever someone last pasted into the settings page. +# 2. The in-cluster Secret `insight-oidc` carries a non-empty `client-secret`. +# The committed values file leaves `authenticator.oidc.clientSecret` empty +# because this repository is public, and the chart writes whatever it is +# given straight into the authenticator's config. An upgrade without that +# value produces a confidential OIDC client with a blank secret: pods Ready, +# release `deployed`, every login broken. +# 3. The stand's IdP can complete the login the smoke stage attempts. At the +# time of writing the stand's realm federates to an external provider and +# has no local users, which no amount of test code can work around; the +# smoke suite carries two modes and fails — loudly, naming the step it got +# stuck at — rather than pretending. See tests/stand/smoke/README.md. +# +# ── COST ──────────────────────────────────────────────────────────────────── +# +# This runs INSIDE the caller's workflow run, so a merge's image build and this +# deploy share one `build-images-refs/heads/main` concurrency slot. The next +# merge's build queues behind this run's stand work. That is a deliberate trade: +# coalescing deploys (`cancel-in-progress: false`) never kills a running upgrade, +# and an upgrade killed halfway leaves a release in `pending-upgrade`, which is +# the one state neither this workflow nor a human can diagnose from a log. + +on: + # The normal path: build-images.yml calls this as its final job, once the + # chart is published. See .cf-studio/.plans/test-stand-ci/build-images-hook.patch.md + # for the calling block and the reasoning behind its guards. + workflow_call: + inputs: + chart_version: + description: >- + Exact umbrella chart version to install, as published to oci://ghcr.io/constructorfabric/charts/insight — e.g. 0.5.101. Comes from publish-chart's output, never from deploy/gitops/.insight-version: that file is only committed at the END of publish-chart, so a checkout of the trigger SHA reads the PREVIOUS release. + required: true + type: string + # The human path: redeploy a specific version by hand, e.g. after fixing the + # stand by hand and wanting CI's view of it back. Same input, same code path — + # there is no manual-only branch anywhere below. + workflow_dispatch: + inputs: + chart_version: + description: "Umbrella chart version to install (e.g. 0.5.101). Must already be published." + required: true + type: string + +# One stand, one deploy at a time. `cancel-in-progress: false` because the thing +# being cancelled would be a `helm upgrade` mid-flight: killing it leaves the +# release in `pending-upgrade`, which blocks the next upgrade until a human +# intervenes. Waiting is cheap; an interrupted upgrade is not. +# +# The group lives HERE, in the called workflow, so what it serialises is the +# deploy work inside each caller run. A burst of merges therefore deploys in +# order, each run installing the chart its own run published, rather than two +# upgrades racing on one release. If that ever needs to become "newest wins, +# drop the rest", the change is a group on the calling job AND removing this one +# — never both, see the note below. +# +# NOTE for the caller: do NOT also put `concurrency:` with this group on the job +# that `uses:` this workflow. The calling job would hold the group while the +# called workflow's job waits for it, and the run would sit there until it timed +# out. +concurrency: + group: test-stand-deploy + cancel-in-progress: false + +# Narrower than the caller's. This workflow reads the repository and talks to a +# cluster; it publishes nothing, writes nothing back to git, and needs no +# registry token — the umbrella chart is pullable anonymously from GHCR, which +# was verified rather than assumed. +permissions: + contents: read + +jobs: + test-stand: + name: deploy → seed → smoke + runs-on: ubuntu-latest + # The environment is declared HERE and not on the calling job: `environment:` + # is not a legal key on a job that `uses:` a reusable workflow. Its + # deployment-branch policy (main only) still evaluates against the caller's + # ref, so restricting the environment to main restricts this whole path to + # main — including a manual dispatch from a topic branch, which is refused + # again below with a readable message. + environment: insight-test-stand + # The ceiling, not the expectation. Sum of the stage budgets plus room for a + # cold runner: deploy 14 + seed 65 + smoke 5 + toolchain and preflight. + timeout-minutes: 95 + env: + # The gitops environment this deploys. One stand, one name; a second stand + # gets its own directory and an input, not a fork of this file. + # + # The two paths are spelled out in full rather than composed from + # ENV_NAME. Composing them would read fine and cost two things: a reader + # would have to resolve a variable to learn which file the upgrade + # actually passes to helm, and deploy/gitops/scripts/emulate-ci-deploy.sh's + # parity report — which greps this file for the literal commands it + # rehearses — would stop finding them. + ENV_NAME: test-stand + VALUES_FILE: deploy/gitops/environments/test-stand/values.yaml + INVENTORY_FILE: deploy/gitops/environments/test-stand/inventory.yaml + CHART_REF: oci://ghcr.io/constructorfabric/charts/insight + STAND_NAMESPACE: insight + STAND_RELEASE: insight + SMOKE_SUITE: tests/stand/smoke + # The three deployments whose configuration arrives through `envFrom` and + # is therefore read exactly once, at container start. See stage 1's restart + # step for why a list of two would be a silent bug. + RESTART_TARGETS: deploy/insight-authenticator deploy/insight-analytics deploy/insight-identity-resolution + # The edge objects the chart does not render and this release does not own, + # but every acceptance criterion travels through. + ROUTE_NAMES: insight-gateway insight-keycloak + CHART_VERSION: ${{ inputs.chart_version }} + REDACT: .github/workflows/scripts/redact-stand-log.py + + steps: + - uses: actions/checkout@3d3c42e5aac5ba805825da76410c181273ba90b1 # v7.0.1 + with: + persist-credentials: false # don't leave the token in .git/config + + # ── Guards ──────────────────────────────────────────────────────────── + # Everything that can be refused without touching the cluster is refused + # here, in order of how cheap it is to check. A deploy that fails after + # `helm upgrade` has started costs a stand; one that fails now costs + # nothing. + - name: Refuse anything that is not main, and anything that is not a version + env: + # Through the environment rather than interpolated into the script + # body: `chart_version` is a free-text `workflow_dispatch` input, and a + # string spliced into a shell line is a script-injection sink. It is + # also passed to `make` further down, which is the reason the shape + # check below is a refusal and not a warning. + REF_NAME: ${{ github.ref_name }} + EVENT: ${{ github.event_name }} + run: | + set -euo pipefail + if [ "$REF_NAME" != "main" ]; then + echo "::error::this workflow deploys main and only main (ref was '$REF_NAME', event '$EVENT')." + echo "The insight-test-stand environment's deployment-branch policy is the real control;" + echo "this check exists so the refusal is readable instead of a missing-secret failure." + exit 1 + fi + if ! printf '%s' "$CHART_VERSION" | grep -Eq '^[0-9]+\.[0-9]+\.[0-9]+([-+][0-9A-Za-z.-]+)?$'; then + echo "::error::chart_version '$CHART_VERSION' is not a semver. Refusing to pass it to make." + exit 1 + fi + echo "deploying umbrella chart $CHART_VERSION to ENV=$ENV_NAME" + + - name: Confirm the environment is wired up + env: + # Presence only. Values are never printed; GitHub masks the secrets + # anyway, but a workflow that relies on masking to keep a secret out of + # a log is one `base64` away from not doing so. + # + # The names are the ones in + # docs/components/deployment/specs/sop/credentials-runbook.md — that + # runbook is how they get set, and a rename has to happen in both. + HAVE_KUBECONFIG: ${{ secrets.TEST_STAND_KUBECONFIG != '' }} + HAVE_SEED_EMAIL: ${{ secrets.TEST_STAND_SEED_EMAIL != '' }} + HAVE_PERSONA_PASSWORD: ${{ secrets.TEST_STAND_PERSONA_PASSWORD != '' }} + HAVE_BOOTSTRAP_EMAIL: ${{ secrets.TEST_STAND_BOOTSTRAP_EMAIL != '' }} + HAVE_BOOTSTRAP_PASSWORD: ${{ secrets.TEST_STAND_BOOTSTRAP_PASSWORD != '' }} + HAVE_BASE_URL: ${{ vars.TEST_STAND_BASE_URL != '' }} + # `password` (every persona authenticates as themselves) or `override` + # (one principal authenticates and every persona session is minted from + # it). Unset means the suite's own default, which is `password`. + LOGIN_MODE: ${{ vars.TEST_STAND_SMOKE_LOGIN_MODE }} + run: | + set -euo pipefail + missing="" + [ "$HAVE_KUBECONFIG" = "true" ] || missing="$missing secrets.TEST_STAND_KUBECONFIG" + [ "$HAVE_SEED_EMAIL" = "true" ] || missing="$missing secrets.TEST_STAND_SEED_EMAIL" + [ "$HAVE_BASE_URL" = "true" ] || missing="$missing vars.TEST_STAND_BASE_URL" + + # Checked HERE rather than left to the smoke suite, which validates the + # same thing far better than this does — but only after a deploy and a + # seed have already run. Twenty minutes is too long to wait to be told + # a password is missing. + case "${LOGIN_MODE:-password}" in + override) + [ "$HAVE_BOOTSTRAP_EMAIL" = "true" ] || missing="$missing secrets.TEST_STAND_BOOTSTRAP_EMAIL" + [ "$HAVE_BOOTSTRAP_PASSWORD" = "true" ] || missing="$missing secrets.TEST_STAND_BOOTSTRAP_PASSWORD" ;; + password) + [ "$HAVE_PERSONA_PASSWORD" = "true" ] || missing="$missing secrets.TEST_STAND_PERSONA_PASSWORD" ;; + *) + echo "::error::vars.TEST_STAND_SMOKE_LOGIN_MODE is '$LOGIN_MODE'; it must be 'password' or 'override'" + exit 1 ;; + esac + + if [ -n "$missing" ]; then + echo "::error::the insight-test-stand environment is missing:$missing" + echo "" + echo "What each one is (full procedure: docs/components/deployment/specs/sop/credentials-runbook.md):" + echo " TEST_STAND_KUBECONFIG base64 of the kubeconfig for the namespace-scoped" + echo " 'ci-deployer' ServiceAccount, as produced by" + echo " deploy/gitops/scripts/provision-ci-deployer.sh." + echo " Admin kubeconfigs stay with humans." + echo " TEST_STAND_SEED_EMAIL address the seeded dev-lead persona resolves to." + echo " A secret rather than a variable purely so it is" + echo " masked in this public log." + echo " TEST_STAND_PERSONA_PASSWORD 'password' mode: the credential every persona" + echo " signs in with." + echo " TEST_STAND_BOOTSTRAP_EMAIL 'override' mode: the one principal that really" + echo " TEST_STAND_BOOTSTRAP_PASSWORD authenticates; persona sessions are minted from it." + echo " TEST_STAND_BASE_URL the stand's public HTTPS origin, scheme and host," + echo " no trailing slash. A variable, not a secret: the" + echo " smoke drives it as a browser would." + exit 1 + fi + echo "smoke login mode: ${LOGIN_MODE:-password (suite default)}" + + # ── Toolchain ───────────────────────────────────────────────────────── + # All of it before anything touches the cluster: a broken lockfile or a + # missing binary should not be discovered after an upgrade has landed. + # Every pin is one this repository already uses elsewhere. + - name: Set up Helm + uses: azure/setup-helm@9bc31f4ebc9c6b171d7bfbaa5d006ae7abdb4310 # v5.0.1 + with: + # Same pin as publish-chart, so the chart is installed by the version + # of helm that packaged it. Also sidesteps the helm 4.2.1 `--wait` + # regression that deploy/gitops/scripts/doctor.sh refuses. + version: '3.14.0' + + - name: Set up kubectl + uses: azure/setup-kubectl@829323503d1be3d00ca8346e5391ca0b07a9ab0d # v5.1.0 + + - uses: astral-sh/setup-uv@c771a70e6277c0a99b617c7a806ffedaca235ff9 # v9.0.0 + with: + cache-suffix: test-stand-smoke + + - name: Check the tools this workflow and the seeder shell out to + run: | + set -euo pipefail + # The inventory is read with yq; seed-stand.sh renders its Job with + # envsubst; the release check and the diagnostics select fields with + # jq. All three are on the GitHub-hosted image today, and all three + # would otherwise fail somewhere much less obvious than here. + missing="" + for tool in helm kubectl yq jq envsubst python3; do + command -v "$tool" >/dev/null 2>&1 || missing="$missing $tool" + done + if [ -n "$missing" ]; then + echo "::error::the runner image no longer carries:$missing — install it in this job" + exit 1 + fi + helm version --short + kubectl version --client -o yaml | head -n 3 + + - name: Resolve the smoke suite's dependencies + run: | + set -euo pipefail + # --frozen: the lockfile is the contract. A run that silently resolves + # a different dependency set is testing a different suite. + uv sync --project tests --frozen + + # ── Cluster identity ────────────────────────────────────────────────── + - name: Write the stand kubeconfig + env: + KUBECONFIG_B64: ${{ secrets.TEST_STAND_KUBECONFIG }} + run: | + set -euo pipefail + target="$RUNNER_TEMP/test-stand.kubeconfig" + # umask BEFORE the file exists, chmod after: the file must never be + # readable by another process on the runner, not even for the instant + # between creation and chmod. + ( + umask 077 + printf '%s' "$KUBECONFIG_B64" | tr -d '[:space:]' | base64 -d > "$target" + ) + chmod 600 "$target" + # Never `cat`, never `kubectl config view` without a redirect: the only + # thing this prints is whether the file parses at all. + if ! KUBECONFIG="$target" kubectl config view --minify >/dev/null 2>&1; then + echo "::error::TEST_STAND_KUBECONFIG did not decode into a usable kubeconfig." + echo "It must be base64 of a complete kubeconfig with a current-context set —" + echo "'base64 -w0 < kubeconfig' on Linux, 'base64 < kubeconfig | tr -d \\\\n' on macOS." + exit 1 + fi + echo "KUBECONFIG=$target" >> "$GITHUB_ENV" + + - name: Confirm the credential points at the stand this repo describes + run: | + set -euo pipefail + if [ ! -f "$INVENTORY_FILE" ]; then + echo "::error::$INVENTORY_FILE does not exist." + echo "The committed gitops environment is what makes this deploy reviewable:" + echo "the cluster it targets, the namespaces, and which infrastructure is" + echo "externally managed all live there. Add it before wiring up the deploy." + exit 1 + fi + expected="$(yq -r '.kubeContext // ""' "$INVENTORY_FILE")" + actual="$(kubectl config current-context 2>/dev/null || echo "")" + if [ -z "$expected" ]; then + echo "::error::$INVENTORY_FILE has no kubeContext, so there is nothing to check the credential against." + exit 1 + fi + if [ "$expected" != "$actual" ]; then + # Deliberately does NOT echo the context the kubeconfig actually + # carries. This branch is exactly the case where the credential points + # somewhere it should not, and naming that somewhere in a public log is + # the thing the check exists to prevent. + echo "::error::the kubeconfig's current context is not the one $INVENTORY_FILE names ('$expected')." + echo "Either the environment secret holds the wrong cluster's credential, or the" + echo "inventory was changed without rotating it. Both are human decisions; CI will not guess." + exit 1 + fi + # `get namespace ` rather than `cluster-info` or a workload list: + # the CI credential is namespace-scoped and cannot do anything + # cluster-wide, so a reachability probe that needs cluster scope would + # reject the very credential it is meant to validate. A request for a + # Namespace object is authorised within that namespace, which a + # RoleBinding can satisfy. + if ! kubectl --request-timeout=20s get namespace "$STAND_NAMESPACE" -o name >/dev/null 2>&1; then + echo "::error::cannot read namespace '$STAND_NAMESPACE' with this credential." + echo "Either the cluster is unreachable from a GitHub-hosted runner, or the" + echo "ci-deployer ServiceAccount and its RoleBinding are not in place. Re-issue with" + echo "deploy/gitops/scripts/provision-ci-deployer.sh." + exit 1 + fi + echo "context matches the committed inventory; namespace $STAND_NAMESPACE is reachable" + + - name: Confirm the chart this run is asked to install exists + run: | + set -euo pipefail + # Read-only, no cluster, a couple of seconds — and it converts "helm + # spent ten minutes failing to pull a chart" into an immediate, + # specific failure. The registry is anonymous-readable, so no token is + # needed here and the job asks for no `packages:` permission. + if ! helm show chart "$CHART_REF" --version "$CHART_VERSION" >/dev/null 2>&1; then + echo "::error::$CHART_REF:$CHART_VERSION is not in the registry." + echo "publish-chart is what puts it there; if that job failed after pushing but" + echo "before committing the version bump, re-run it rather than deploying by hand." + exit 1 + fi + echo "$CHART_REF:$CHART_VERSION is published" + + # ── Stage 1 of 3: deploy ────────────────────────────────────────────── + # `timeout-minutes` is 14 against helm's own `--timeout 10m` on purpose, + # with room for the chart pull and the render that precede it. The step + # budget must never be the one that fires: a step timeout SIGKILLs helm and + # leaves the release in `pending-upgrade`, while helm's own timeout leaves + # a clean `failed` release that `helm history` explains. + - name: 'Stage 1/3 — upgrade the release' + timeout-minutes: 14 + run: | + set -euo pipefail + # The OIDC client secret is read out of the cluster at deploy time and + # injected, because the committed values file leaves it empty — this + # repository is public. The chart writes whatever it is given straight + # into the authenticator's config Secret, so an upgrade without this + # produces a confidential client with a blank secret: pods Ready, + # release `deployed`, every login broken. + # + # Read from the cluster rather than held as a GitHub secret so there is + # ONE copy of the value — the one the realm already agrees with. A + # second copy in CI is a second thing to rotate and a second way for + # the two to disagree. + oidc_client_secret="$( + kubectl -n "$STAND_NAMESPACE" get secret insight-oidc \ + -o 'jsonpath={.data.client-secret}' | base64 --decode + )" + if [ -z "$oidc_client_secret" ]; then + echo "::error::Secret insight-oidc in namespace $STAND_NAMESPACE has no non-empty 'client-secret' key." + echo "Refusing to deploy: the upgrade would succeed and leave the authenticator with a" + echo "blank client secret, which looks healthy and fails every login." + exit 1 + fi + # Belt and braces. The value is never echoed below, but a masked value + # cannot be leaked by a future edit either. + echo "::add-mask::$oidc_client_secret" + + # Deliberately NO --atomic: a failed upgrade is LEFT failed so the pods + # holding the reason are still there for the diagnostics step. Recovery + # is the next merge, or a human running `make rollback ENV=test-stand`. + # + # Deliberately NO --create-namespace: the namespace exists, and the CI + # credential is namespace-scoped and could not create one anyway — + # asking would turn a working deploy into an RBAC failure. + # + # --timeout 10m rather than the gitops Makefile's 30m default: a deploy + # that is going to fail should say so inside the CI budget. + helm upgrade --install "$STAND_RELEASE" "$CHART_REF" \ + --version "$CHART_VERSION" \ + --namespace "$STAND_NAMESPACE" \ + --values "$VALUES_FILE" \ + --set-string authenticator.oidc.clientSecret="$oidc_client_secret" \ + --wait --timeout 10m \ + --history-max 10 \ + 2>&1 | python3 "$REDACT" + + - name: 'Stage 1/3 — verify the release is the chart this run published' + run: | + set -euo pipefail + # A release can be `deployed` and still be the wrong chart — a resumed + # run, a hand-deploy that raced this one, an input that did not say what + # it meant. Checked separately from the upgrade's exit status because + # they answer different questions. + # + # The status flags are enumerated instead of `--all` because `--all` is + # a Helm 3 flag that Helm 4 removed: on a runner carrying v4 it exits + # `Error: unknown flag: --all` and this verification silently reports + # every release as "absent". The explicit set means the same thing on + # both majors. The intent it preserves: the interesting failures are + # the ones the default listing hides — a `pending-upgrade` release (an + # upgrade that was killed rather than one that failed) must be reported + # as itself, not as "absent". + listed="$(helm list -n "$STAND_NAMESPACE" \ + --deployed --failed --pending --uninstalling \ + --filter "^${STAND_RELEASE}\$" -o json)" + status="$(printf '%s' "$listed" | jq -r '.[0].status // "absent"')" + chart="$(printf '%s' "$listed" | jq -r '.[0].chart // "absent"')" + revision="$(printf '%s' "$listed" | jq -r '.[0].revision // "?"')" + echo "release $STAND_RELEASE: status=$status chart=$chart revision=$revision" + if [ "$status" != "deployed" ]; then + echo "::error::release '$STAND_RELEASE' is '$status', not 'deployed'." + echo "It has been left in that state on purpose — see the failure diagnostics below," + echo "then either fix forward with another merge or roll back by hand." + exit 1 + fi + if [ "$chart" != "insight-$CHART_VERSION" ]; then + echo "::error::the stand is running '$chart', not 'insight-$CHART_VERSION'." + echo "helm reported success while installing something else — treat this as a" + echo "problem with the chart reference or the version input, not with the stand." + exit 1 + fi + + - name: 'Stage 1/3 — restart what the chart cannot know to restart' + timeout-minutes: 8 + run: | + set -euo pipefail + # Each subchart's `checksum/config` annotation hashes its OWN ConfigMap. + # It does not cover the umbrella-rendered insight-*-config Secrets, + # which these pods consume with `envFrom` and therefore read exactly + # once, at container start. So a changed datastore host, tenant or + # client secret updates the Secret, leaves the pod spec byte-identical, + # and never reaches a running process — a `deployed` release with + # all-Ready pods running stale configuration. + # + # All three services are listed on purpose: each consumes its config + # the same way, and a restart list of two would leave one of them + # holding yesterday's configuration with nothing to show for it. + # shellcheck disable=SC2086 # RESTART_TARGETS is a deliberate word list + kubectl -n "$STAND_NAMESPACE" rollout restart $RESTART_TARGETS + # shellcheck disable=SC2086 + kubectl -n "$STAND_NAMESPACE" rollout status --timeout=5m $RESTART_TARGETS + + - name: 'Stage 1/3 — confirm the edge still routes' + run: | + set -euo pipefail + # The chart renders NO HTTPRoute. These two objects live outside the + # release, are owned by the deployment repository, and every acceptance + # criterion travels through them — so a successful upgrade says nothing + # about whether the stand is reachable. + # + # Read, never applied: the files under + # deploy/gitops/environments/test-stand/manifests/ are the source of + # truth, and applying them from here would make CI a second writer on an + # object a human owns. + # shellcheck disable=SC2086 # ROUTE_NAMES is a deliberate word list + kubectl -n "$STAND_NAMESPACE" get httproute $ROUTE_NAMES \ + -o 'custom-columns=NAME:.metadata.name,ACCEPTED:.status.parents[0].conditions[?(@.type=="Accepted")].status' + # shellcheck disable=SC2086 + not_accepted="$( + kubectl -n "$STAND_NAMESPACE" get httproute $ROUTE_NAMES \ + -o 'jsonpath={range .items[*]}{.metadata.name}{"="}{.status.parents[0].conditions[?(@.type=="Accepted")].status}{"\n"}{end}' \ + | grep -v '=True$' || true + )" + if [ -n "$not_accepted" ]; then + echo "::error::an edge route is not Accepted:" + printf '%s\n' "$not_accepted" + echo "The release may be perfectly healthy; the stand is still unreachable, and the" + echo "smoke below would fail at its first request with a much less useful message." + exit 1 + fi + + # ── Stage 2 of 3: seed ──────────────────────────────────────────────── + # 65 minutes is a CEILING chosen against the seed Job's own + # activeDeadlineSeconds of 3600s, not a measurement: deliberately just + # ABOVE it, so the pod's deadline is what fires first. A step timeout that + # fired first would kill the log follow while leaving the Job running in + # the cluster, and the next run would meet it half-finished. + # + # PENDING: replace this with a measured value once a handful of runs have + # reported wall-clock. If the real figure is (say) 12 minutes, the Job's + # --deadline should come down with the step budget, in one change, with the + # measurement quoted. + - name: 'Stage 2/3 — seed the stand' + timeout-minutes: 65 + env: + SEED_EMAIL: ${{ secrets.TEST_STAND_SEED_EMAIL }} + run: | + set -euo pipefail + # seed-stand.sh verbatim — no wrapper, no re-implemented Job manifest. + # It discovers every coordinate it needs (datastore hosts, the tenant, + # the seed image the release pins) from the cluster, which is precisely + # why CI must not hand it any: a value CI supplied would be a value CI + # could get wrong, and a stand seeded against the wrong tenant looks + # seeded. + # + # `--context` is passed even though the ambient kubeconfig already + # points there. The script prints the context it resolved before it + # writes anything, and a run whose target is stated rather than + # inherited is one a reader of this log can check. + # + # `--days`, spelled exactly like that: the seeder rejects unknown + # arguments, so a plausible-looking synonym fails the stage every time. + # + # `tee` keeps the raw stream in $RUNNER_TEMP — it is the only copy of + # the seed manifest, which the next step reads — while only the redacted + # form reaches this public log. `pipefail` (set above) is what makes + # the seeder's exit status, rather than python's, the status of this + # step; without it a failed seed would be reported as a clean run. + kube_context="$(yq -r '.kubeContext' "$INVENTORY_FILE")" + ./src/ingestion/tools/seed/seed-stand.sh \ + -n "$STAND_NAMESPACE" \ + --context "$kube_context" \ + --email "$SEED_EMAIL" \ + --days 730 \ + 2>&1 | tee "$RUNNER_TEMP/seed.log" | python3 "$REDACT" + + - name: 'Stage 2/3 — capture the seed manifest' + run: | + set -euo pipefail + # A cluster seed Job's filesystem dies with its pod, so the seeder + # prints the manifest to stdout before writing it. That log is the only + # record, and the smoke suite needs the document to know which personas + # exist. Extracted here rather than uploaded anywhere: it names every + # persona and the tenant. + python3 - "$RUNNER_TEMP/seed.log" "$RUNNER_TEMP/stand-manifest.json" <<'PY' + import json + import sys + + source, target = sys.argv[1], sys.argv[2] + lines = open(source, encoding="utf-8", errors="replace").read().splitlines() + + # The document is rendered with json.dumps(indent=2), so its opening + # brace is alone on a line at column 0 and its closing brace is the next + # line at column 0. Nested braces are indented and cannot be confused + # for either. The LAST such block wins: a run that printed more than one + # is a run whose later document supersedes the earlier. + block, start = None, None + for index, line in enumerate(lines): + if line == "{": + start = index + elif line == "}" and start is not None: + block = "\n".join(lines[start : index + 1]) + start = None + + if block is None: + print("::error::no seed manifest in the seed log — the seeder did not reach the end of its run") + raise SystemExit(1) + + try: + doc = json.loads(block) + except json.JSONDecodeError as exc: + print(f"::error::the seed log's manifest block is not valid JSON: {exc}") + raise SystemExit(1) from exc + + with open(target, "w", encoding="utf-8") as handle: + json.dump(doc, handle, indent=2, sort_keys=True) + handle.write("\n") + + # A summary, not the document. Fixture NAMES are contract; persona + # emails, UUIDs and the tenant are not printed anywhere. + personas = doc.get("personas") or [] + fixtures = sorted((doc.get("fixtures") or {}).keys()) + print(f"manifest_version: {doc.get('manifest_version')}") + print(f"anchor_date: {doc.get('anchor_date')}") + print(f"data_window: {doc.get('data_window')}") + print(f"seed_revision: {doc.get('seed_revision')}") + print(f"personas: {len(personas)}") + print(f"fixtures: {', '.join(fixtures) if fixtures else '(none)'}") + PY + echo "INSIGHT_STAND_MANIFEST=$RUNNER_TEMP/stand-manifest.json" >> "$GITHUB_ENV" + + # ── Stage 3 of 3: smoke ─────────────────────────────────────────────── + # Never reached after a failed seed: this step has no `if:`, so a failure + # above skips it. That is the whole mechanism — asserting against an + # unseeded stand would produce a failure that looks like a product bug. + - name: 'Stage 3/3 — smoke the stand through its public URL' + timeout-minutes: 5 + env: + # The public origin, driven exactly as a browser would drive it: real + # DNS, real TLS, real IdP redirect. No port-forward, no in-cluster + # address — a stand that works only from inside the cluster is a stand + # nobody can use. + # + # SMOKE_BASE_URL and nothing else aims the suite: its conftest copies + # the value into the shared stand resolution itself, so passing + # `--base-url` as well would be a second spelling of one address. + SMOKE_BASE_URL: ${{ vars.TEST_STAND_BASE_URL }} + SMOKE_LOGIN_MODE: ${{ vars.TEST_STAND_SMOKE_LOGIN_MODE }} + SMOKE_PERSONA_PASSWORD: ${{ secrets.TEST_STAND_PERSONA_PASSWORD }} + SMOKE_BOOTSTRAP_EMAIL: ${{ secrets.TEST_STAND_BOOTSTRAP_EMAIL }} + SMOKE_BOOTSTRAP_PASSWORD: ${{ secrets.TEST_STAND_BOOTSTRAP_PASSWORD }} + run: | + set -euo pipefail + if [ ! -d "$SMOKE_SUITE" ]; then + echo "::error::$SMOKE_SUITE does not exist." + echo "The smoke module is what turns this workflow from a deploy into a gate." + echo "It belongs there so it inherits tests/stand/conftest.py — the manifest, the" + echo "persona factory, the API client — without inheriting tests/stand/api's" + echo "scratch-row policy." + exit 1 + fi + # Both spellings, on purpose. INSIGHT_STAND_MANIFEST is what the shared + # library reads; --stand-manifest is what the suite's own option reads; + # they point at the same file, and setting only one of them makes the + # run depend on which layer happens to resolve first. + # + # INSIGHT_STAND_ARTIFACT_DIR is explicit because the library computes + # its default from its own location on disk, and a runner is not a + # developer's checkout. + export INSIGHT_STAND_ARTIFACT_DIR="$RUNNER_TEMP/stand-artifacts" + mkdir -p "$INSIGHT_STAND_ARTIFACT_DIR" + # Redacted like everything else: the suite's assertion messages quote + # response bodies, and a response body is not a place anyone controls + # what appears. `pipefail` (set above) keeps pytest's exit status as + # the step's. + # The suite path is spelled literally here, not as $SMOKE_SUITE, for the + # same reason the values file is: this line is one of the anchors + # deploy/gitops/scripts/emulate-ci-deploy.sh greps for when it reports + # whether a laptop rehearsal is still running what CI runs. + uv run --project tests --frozen \ + pytest tests/stand/smoke -ra --stand-manifest "$INSIGHT_STAND_MANIFEST" \ + 2>&1 | python3 "$REDACT" + + # ── Diagnostics ─────────────────────────────────────────────────────── + # One step, only on failure, and everything it can print is decided by a + # committed script rather than by this file — so widening what a red run + # publishes is a reviewable diff and not an edit to a `run:` block. + - name: Failure diagnostics (curated, redacted) + if: failure() + timeout-minutes: 6 + run: | + set -uo pipefail + if [ -z "${KUBECONFIG:-}" ] || [ ! -f "$KUBECONFIG" ]; then + echo "the run failed before a cluster credential existed — nothing to collect" + exit 0 + fi + bash .github/workflows/scripts/stand-diagnostics.sh "$STAND_NAMESPACE" "$STAND_RELEASE" + + - name: Summarise the run + if: always() + env: + OUTCOME: ${{ job.status }} + run: | + set -euo pipefail + { + echo "### Test stand — \`$CHART_VERSION\`" + echo "" + echo "| | |" + echo "|---|---|" + echo "| chart | \`insight-$CHART_VERSION\` |" + echo "| environment | \`$ENV_NAME\` (namespace \`$STAND_NAMESPACE\`, release \`$STAND_RELEASE\`) |" + echo "| outcome | **$OUTCOME** |" + echo "" + if [ "$OUTCOME" != "success" ]; then + echo "This is an after-merge alarm, not a merge gate: the chart under test did not" + echo "exist until the merge happened. **The author of the merge commit owns this run.**" + echo "Fix forward if the cause is small and obvious, revert if it is not." + echo "" + echo "A failed \`helm upgrade\` is left in place on purpose. Recovery is the next" + echo "merge, or a deliberate \`helm rollback\` by a human." + fi + } >> "$GITHUB_STEP_SUMMARY" diff --git a/.github/workflows/scripts/redact-stand-log.py b/.github/workflows/scripts/redact-stand-log.py new file mode 100755 index 000000000..2c800e3c8 --- /dev/null +++ b/.github/workflows/scripts/redact-stand-log.py @@ -0,0 +1,210 @@ +#!/usr/bin/env python3 +"""Mask credentials out of deployed-stand output before it reaches a public run log. + +This repository is PUBLIC, and so is every line a workflow prints. The test-stand +deploy talks to a real cluster with a real credential, seeds real personas and +then signs them in — three activities whose natural output carries session +cookies, bearer tokens, connection strings and people's email addresses. Nothing +in that chain was written with a public audience in mind: `seed-stand.sh` prints +the seed manifest so a discarded pod's work is not lost, and `kubectl logs` +prints whatever the container felt like printing. + +So the workflow never pipes cluster or seeder output straight to the console. +Everything goes through this filter, and the filter is the only thing standing +between a container that decided to log its DSN and a permanent public record of +it. + +**Fail closed, per line.** A line this script cannot prove it has cleaned is +replaced wholesale with a marker rather than passed through. The verification +pass at the end of `_clean` re-scans the *result*: redaction that claims to have +worked is not the same as redaction that worked, and the cheap way to tell them +apart is to look at the bytes about to be printed. An exception anywhere aborts +the stream — a truncated diagnostic is recoverable, a published secret is not. + +What is masked, and why each rule exists: + +* **URLs carrying credentials** (`scheme://user:pass@host`) — the shape a DSN + takes in a connection error. Run first, because the `pass@host.example` tail + also matches the email rule and the URL form is the more informative mask. +* **Email addresses** — replaced by a short, stable digest rather than a flat + marker. Two lines about the same person still visibly concern the same person, + which is most of what an address is worth in a diagnostic, and the digest is + one-way. Seeded personas live on a synthetic domain, but the `--email` the + seeder is pointed at is a real one. +* **`Bearer` values, JWTs and `__Host-sid` cookies** — the credentials the smoke + stage handles. A JWT is recognised by its `eyJ` header prefix, so an + unsigned or truncated one is caught too. +* **`key: value` pairs whose KEY names a secret** — `password:`, `token:`, + `client-key-data:` and friends. This is what a kubeconfig, a Secret dump or a + helm values render looks like, and the key name is the only reliable signal. +* **Long unbroken base64/hex runs** — the catch-all for a secret whose shape + nobody anticipated. Container image digests are exempted (they are + diagnostics, not credentials, and losing them makes a pull failure + unreadable); everything else of 40+ characters is assumed to be material. +* **IPv4 literals** — not credentials, but infrastructure detail this repo does + not publish. Chart versions and other three-part numbers are untouched: the + rule needs four octets. +* **Over-long lines** — truncated. A 40 KB line is either a heap dump or a blob, + and neither belongs in a run log; truncating it also bounds what an unknown + secret shape can leak past the rules above. + +What this does NOT do, stated plainly: it does not understand structure. A +secret spread across two lines, or one that looks like an ordinary English word, +survives. The rule this implements is "credential-shaped text does not reach the +console"; the workflow's own discipline — no `kubectl describe`, no environment +dumps, no artifact uploads — is what covers the rest. + +Usage: + | redact-stand-log.py # stdin -> stdout, line at a time + redact-stand-log.py FILE [FILE ...] # named files -> stdout + redact-stand-log.py --max-line 400 FILE # tighter truncation + +Exit status is 0 when the whole stream was cleaned and emitted, 1 when the +filter aborted. A caller that pipes into this script should run under +`set -o pipefail` so an abort is not mistaken for a clean run. +""" + +from __future__ import annotations + +import hashlib +import re +import sys +from collections.abc import Iterable, Iterator +from pathlib import Path + +#: What an unrecoverable line becomes. Deliberately loud: a reader must be able +#: to tell "this line was removed" from "nothing was logged here". +LINE_MARKER = "[line withheld by CI — redaction could not be verified]" + +#: Default ceiling on a single emitted line. Long enough for a helm error or a +#: Rust panic with a backtrace frame, short enough that a blob cannot ride out. +DEFAULT_MAX_LINE = 1000 + +#: Placeholder standing in for an image digest while the long-blob rule runs. +#: Restored verbatim afterwards — see `_clean`. +_DIGEST_SLOT = "\x00digest{}\x00" + +_URL_CREDENTIALS = re.compile(r"(?P[A-Za-z][A-Za-z0-9+.\-]*://)[^\s/@:]+:[^\s/@]+@") +_EMAIL = re.compile(r"[A-Za-z0-9._%+\-]+@[A-Za-z0-9.\-]+\.[A-Za-z]{2,}") +_JWT = re.compile(r"\beyJ[A-Za-z0-9_\-]{4,}\.[A-Za-z0-9_\-]{4,}(?:\.[A-Za-z0-9_\-]+)?") +_BEARER = re.compile(r"(?i)\b(bearer|basic)\s+[A-Za-z0-9._~+/=\-]{8,}") +_SESSION_COOKIE = re.compile(r"(?i)(__Host-sid|__Secure-sid|sid)=[^\s;,\"']{8,}") +_PRIVATE_KEY_HEADER = re.compile(r"-----BEGIN [A-Z ]*PRIVATE KEY-----") + +#: Keys whose value is material wherever they appear — kubeconfig, Secret dumps, +#: rendered values, connection errors. Matched case-insensitively against the +#: token immediately left of a `:` or `=`. +_SECRET_KEY = re.compile( + r"(?i)\b(" + r"password|passwd|pwd|secret|secret[_\-]?key|client[_\-]?secret|" + r"token|access[_\-]?token|refresh[_\-]?token|id[_\-]?token|bearer[_\-]?token|" + r"api[_\-]?key|private[_\-]?key|signing[_\-]?key|" + r"client-key-data|client-certificate-data|certificate-authority-data" + r")([\"']?\s*[:=]\s*)(?!\s*$)\S+" +) + +#: An image digest: kept, because "which image failed to pull" is the answer to +#: half the seed failures there are. +_DIGEST = re.compile(r"\bsha(?:256|512):[0-9a-fA-F]{32,128}\b") + +#: The catch-all. 40 characters is above every Kubernetes object name and image +#: tag in this stack (all of which carry `-`, `.` or `/`) and below every key, +#: token and certificate body. +_LONG_BLOB = re.compile(r"(? str: + """A stable one-way handle for one address. + + Six hex characters: enough that two personas in the same log are told apart, + far too few to attack, and short enough to keep a table readable. + """ + digest = hashlib.sha256(match.group(0).lower().encode("utf-8")).hexdigest()[:6] + return f"[email:{digest}]" + + +def _clean(line: str, *, max_line: int) -> str: + """Return `line` with every credential-shaped run replaced. + + Order is load-bearing. URLs go first so a DSN's password is masked as a URL + credential rather than half-caught by the email rule; digests are parked + before the long-blob sweep and restored after it; the verification pass runs + last, over the bytes that are actually about to be printed. + """ + if _PRIVATE_KEY_HEADER.search(line): + # The header line names the key type and nothing else useful, and the + # body that follows is caught by the blob rule. Drop the whole thing — + # a PEM in a CI log is never a diagnostic. + return "[private key material withheld by CI]" + + out = _URL_CREDENTIALS.sub(r"\g[credentials redacted]@", line) + out = _JWT.sub("[jwt redacted]", out) + out = _BEARER.sub(r"\1 [redacted]", out) + out = _SESSION_COOKIE.sub(r"\1=[redacted]", out) + out = _SECRET_KEY.sub(r"\1\2[redacted]", out) + out = _EMAIL.sub(_email_slot, out) + + digests: list[str] = [] + + def _park(match: re.Match[str]) -> str: + digests.append(match.group(0)) + return _DIGEST_SLOT.format(len(digests) - 1) + + out = _DIGEST.sub(_park, out) + out = _LONG_BLOB.sub("[blob redacted]", out) + for index, digest in enumerate(digests): + out = out.replace(_DIGEST_SLOT.format(index), digest) + + out = _IPV4.sub("[ip redacted]", out) + + if len(out) > max_line: + out = out[:max_line] + f" …[truncated at {max_line} chars by CI]" + + # The result, not the input, is what gets published — so the check is on the + # result. Anything credential-shaped that survived every rule above means a + # rule has a hole, and the safe reading of a hole is "do not print this". + if _EMAIL.search(out) or _JWT.search(out) or _PRIVATE_KEY_HEADER.search(out): + return LINE_MARKER + return out + + +def _stream(lines: Iterable[str], *, max_line: int) -> Iterator[str]: + for line in lines: + yield _clean(line.rstrip("\n"), max_line=max_line) + + +def main(argv: list[str]) -> int: + args = argv[1:] + max_line = DEFAULT_MAX_LINE + if args[:1] == ["--max-line"]: + if len(args) < 2 or not args[1].isdigit() or int(args[1]) < 80: + print("--max-line needs a whole number of at least 80", file=sys.stderr) # noqa: T201 + return 2 + max_line = int(args[1]) + args = args[2:] + + try: + if not args: + for cleaned in _stream(sys.stdin, max_line=max_line): + print(cleaned, flush=True) # noqa: T201 + return 0 + for path in args: + with Path(path).open(encoding="utf-8", errors="replace") as handle: + for cleaned in _stream(handle, max_line=max_line): + print(cleaned, flush=True) # noqa: T201 + except BrokenPipeError: + # The consumer went away (a `head`, or a cancelled step). Not a + # redaction failure, and not worth a red X. + return 0 + except Exception as exc: # noqa: BLE001 — any failure means "stop printing" + # Whatever was emitted before this point was cleaned; what follows was + # not, so nothing follows. + print(f"::error::redaction aborted, output truncated: {type(exc).__name__}: {exc}", file=sys.stderr) # noqa: T201 + return 1 + return 0 + + +if __name__ == "__main__": + raise SystemExit(main(sys.argv)) diff --git a/.github/workflows/scripts/stand-diagnostics.sh b/.github/workflows/scripts/stand-diagnostics.sh new file mode 100755 index 000000000..41c8d2e41 --- /dev/null +++ b/.github/workflows/scripts/stand-diagnostics.sh @@ -0,0 +1,236 @@ +#!/usr/bin/env bash +# Curated, redacted evidence from a deployed stand after something went wrong. +# +# This is the whole of what a failed test-stand run publishes about the cluster. +# It is an allowlist, not a filter over "everything": each section below was +# chosen because it answers a question a red run actually raises, and anything +# not on the list is absent by decision rather than by omission. +# +# release which chart version is installed, whether helm thinks the release +# is deployed or failed, and the last few revisions with the +# description helm wrote — that description is where "timed out +# waiting for the condition" lives. +# pods name, phase, ready, restart count, and the waiting/terminated +# reason. Enough to tell CrashLoopBackOff from ImagePullBackOff from +# a pod that never got scheduled. +# warnings Warning events as (reason, kind, name, count). REASONS ONLY — the +# message body is where a scheduler or a kubelet quotes back +# whatever it was handed, including image references, node names and +# occasionally the content of a Secret volume it could not mount. +# logs the last 50 lines of containers that are not healthy, and the +# previous container's tail when one has restarted. Everything goes +# through redact-stand-log.py. +# +# What is deliberately NOT here, and must stay that way: +# +# * `kubectl describe` — it prints the pod spec, which prints every +# environment variable name AND the Secret/ConfigMap each one is bound to, +# plus node names, image digests and volume paths. It is the single most +# common way a public CI log becomes an infrastructure map. +# * `kubectl get ... -o wide` / `-o yaml` / `-o json` dumps — same reason, plus +# node IPs. +# * anything reading a Secret or a ConfigMap, even by name-only listing. +# * artifact uploads. A run log is public but ephemeral in attention; an +# uploaded archive is a downloadable copy of whatever slipped through. +# * `env`, `printenv`, `set -x` over a step that carries credentials. +# +# Every section is best-effort and this script ALWAYS exits 0. It runs after a +# stage has already failed; a diagnostic that fails is a missing paragraph, not +# a second failure, and turning it into one would relabel every red run with the +# wrong cause. +# +# Usage: +# stand-diagnostics.sh [max-containers] +# +# Reads the ambient kubeconfig ($KUBECONFIG or ~/.kube/config) — the same +# context the deploy itself acted on. Runnable by hand: a human debugging the +# stand gets exactly the view CI got, which is the point of it being a committed +# script rather than an inline block in the workflow. + +set -uo pipefail + +SCRIPT_DIR="$(cd "$(dirname "${BASH_SOURCE[0]}")" && pwd)" +REDACT="$SCRIPT_DIR/redact-stand-log.py" + +NAMESPACE="${1:-}" +RELEASE="${2:-}" +# A stand with a dozen sick pods produces a wall of text nobody reads, and the +# first few are almost always the same failure. Bounded, and the bound is +# reported when it bites. +MAX_CONTAINERS="${3:-8}" + +if [[ -z "$NAMESPACE" || -z "$RELEASE" ]]; then + echo "usage: stand-diagnostics.sh [max-containers]" >&2 + exit 0 +fi + +if [[ ! -f "$REDACT" ]]; then + echo "::warning::redaction filter missing at $REDACT — refusing to print cluster output" >&2 + exit 0 +fi + +# Nothing reaches the console except through this. Written as a function so a +# new section cannot accidentally forget the pipe: every `kubectl`/`helm` call +# below ends in `| redact`. +redact() { + python3 "$REDACT" +} + +# A short timeout everywhere: this runs when the stand is already unwell, and an +# apiserver that has stopped answering must not hold the runner for ten minutes. +kube() { + kubectl --request-timeout=20s -n "$NAMESPACE" "$@" 2>&1 +} + +section() { + echo "::group::$1" +} + +endsection() { + echo "::endgroup::" +} + +if ! kubectl --request-timeout=10s version --output=json >/dev/null 2>&1; then + echo "::warning::no reachable cluster — diagnostics skipped (the failure is upstream of the deploy)" + exit 0 +fi + +# ── release ───────────────────────────────────────────────────────────────── +# `helm list` and `helm history`, deliberately NOT `helm status`: status's +# document also carries `.info.notes` (the rendered NOTES.txt, which quotes +# hostnames and sign-in URLs) and the rendered manifest. The two commands used +# here have small, stable documents with nothing in them but identity and +# outcome — and `history`'s `description` is where helm records "Upgrade +# \"insight\" failed: timed out waiting for the condition", which is the single +# most useful line a failed deploy produces. +# +# The status flags are spelled out one by one rather than using `--all`, and +# stderr is NOT folded into the pipe. Both were real bugs: +# +# * `helm list --all` is a Helm 3 flag. Helm 4 removed it, so on a runner (or +# a laptop) carrying helm v4 the command exits with `Error: unknown flag: +# --all` and prints nothing useful. The explicit set below means the same +# thing and parses on both majors, which is what a diagnostics script has to +# do — it runs precisely when something is already wrong, so it can never be +# the thing that fails. +# * `... 2>&1 | jq` fed helm's error text INTO jq, which then died with +# `parse error: Invalid numeric literal` and swallowed the actual message. +# A diagnostic that hides the diagnosis is worse than no diagnostic; stderr +# now reaches the log on its own. +# +# The intent the flags preserve: the default listing hides a release that is +# neither deployed nor failed, and `pending-upgrade` — the state a killed +# upgrade leaves behind — is exactly the one worth seeing. +section "helm release" +if command -v jq >/dev/null 2>&1; then + helm list -n "$NAMESPACE" \ + --deployed --failed --pending --uninstalling \ + --filter "^${RELEASE}\$" -o json | + jq -r ' + if type == "array" and length > 0 then + (.[0] | "release: \(.name)", + "namespace: \(.namespace)", + "revision: \(.revision)", + "status: \(.status)", + "chart: \(.chart)", + "appVersion: \(.app_version)", + "updated: \(.updated)") + else + "no release named \($rel) in this namespace" + end' --arg rel "$RELEASE" | redact +else + echo "jq is not on PATH — release summary skipped" | redact +fi +endsection + +section "helm history (last 5 revisions)" +if command -v jq >/dev/null 2>&1; then + helm history "$RELEASE" -n "$NAMESPACE" -o json | + jq -r ' + if type == "array" then + (sort_by(.revision) | reverse | .[0:5][] + | "rev \(.revision) \(.status) \(.chart) \(.updated) \(.description // "")") + else + "helm history returned no revisions" + end' | redact +else + echo "jq is not on PATH — history skipped" | redact +fi +endsection + +# ── pods ──────────────────────────────────────────────────────────────────── +# custom-columns rather than `-o wide`: wide adds the node name and the pod IP, +# which are infrastructure detail this repo does not publish, and adds nothing +# to "which container is unhappy". +section "pods" +kube get pods \ + -o 'custom-columns=NAME:.metadata.name,PHASE:.status.phase,READY:.status.containerStatuses[*].ready,RESTARTS:.status.containerStatuses[*].restartCount,WAITING:.status.containerStatuses[*].state.waiting.reason,TERMINATED:.status.containerStatuses[*].state.terminated.reason' | + redact +endsection + +# ── warning events ────────────────────────────────────────────────────────── +# Reason and involved object only. The MESSAGE column is deliberately absent — +# see the header. Sorted oldest-first by the apiserver, so the tail is the +# recent end. +section "warning events (reasons only, no messages)" +kube get events --field-selector type=Warning --sort-by=.lastTimestamp \ + -o 'custom-columns=LAST:.lastTimestamp,REASON:.reason,KIND:.involvedObject.kind,OBJECT:.involvedObject.name,COUNT:.count' | + tail -n 25 | redact +endsection + +# ── logs of unhealthy containers ──────────────────────────────────────────── +# "Unhealthy" is: not ready, or restarted at least once, or currently waiting, +# or terminated with a non-zero exit code. Init containers count — a stand that +# fails its migration hook never reaches its app containers at all. +section "tail of unhealthy containers" +if ! command -v jq >/dev/null 2>&1; then + echo "jq is not on PATH — container selection skipped" | redact + endsection + exit 0 +fi + +targets="$( + kube get pods -o json 2>/dev/null | + jq -r ' + (.items // [])[] + | .metadata.name as $pod + | ((.status.containerStatuses // []) + (.status.initContainerStatuses // []))[] + | select( + (.ready != true) + or ((.restartCount // 0) > 0) + or (.state.waiting != null) + or ((.state.terminated.exitCode // 0) != 0) + ) + | "\($pod) \(.name) \(.restartCount // 0)" + ' 2>/dev/null +)" + +if [[ -z "$targets" ]]; then + echo "every container is ready with no restarts — the failure is not in a pod's log" | redact + endsection + exit 0 +fi + +printed=0 +while read -r pod container restarts; do + [[ -n "$pod" ]] || continue + if [[ "$printed" -ge "$MAX_CONTAINERS" ]]; then + echo "…more unhealthy containers than the $MAX_CONTAINERS this prints; the rest are in the pod table above" | redact + break + fi + printed=$((printed + 1)) + + echo "----- $pod/$container (restarts: $restarts) -----" | redact + # --limit-bytes as well as --tail: a container logging one enormous line + # would otherwise defeat the line count, and the point of the cap is to bound + # what reaches a public log rather than to bound the line count as such. + kube logs "$pod" -c "$container" --tail=50 --limit-bytes=20000 | redact + + if [[ "${restarts:-0}" -gt 0 ]]; then + echo "----- $pod/$container (previous container) -----" | redact + kube logs "$pod" -c "$container" --previous --tail=50 --limit-bytes=20000 | redact + fi +done <<<"$targets" +endsection + +exit 0 From 410427aa4c72a4a514ace05b1a983f8c79e9659a Mon Sep 17 00:00:00 2001 From: Konstantin Tursunov Date: Sun, 9 Aug 2026 12:14:14 +0800 Subject: [PATCH 04/59] test(stand): add the post-deploy smoke gate for a deployed stand MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit Four checks against a real deployed stand through its public URL, in the order a person meets them: `/auth/login` redirects to the IdP, a seeded persona signs in, `/auth/me` returns that persona, and a metric returns data for the seeded window. Together they are the "a user can log in and see data" half of #2244 — the part a green `helm upgrade` cannot tell you. Everything runs through the public hostname. No port-forwarding, no in-cluster shortcut: DNS, TLS, the gateway and the IdP redirect are exactly the parts that work in a cluster and fail from a browser, so testing around them would test the wrong thing. Personas come from the seeder's manifest at collection time rather than a hardcoded list, so the suite follows the roster instead of drifting from it, and a missing manifest is a hard stop with an actionable message rather than a silent skip. Assertions are on shape and non-emptiness — series present, at least one point, no null values — never on exact numbers: reading a figure off a running stand and asserting it back proves nothing and breaks on every legitimate fixture change. `SMOKE_BASE_URL` and the credentials are required environment variables with no defaults, so the suite cannot quietly aim at somebody's local stack. The `stand_smoke` marker keeps it out of the compose lanes (`-m 'not stand_smoke'`). Refs #2244 Signed-off-by: Konstantin Tursunov --- tests/pyproject.toml | 1 + tests/stand/smoke/README.md | 177 +++++++++ tests/stand/smoke/__init__.py | 28 ++ tests/stand/smoke/conftest.py | 354 +++++++++++++++++ tests/stand/smoke/login.py | 447 +++++++++++++++++++++ tests/stand/smoke/test_deploy_smoke.py | 513 +++++++++++++++++++++++++ 6 files changed, 1520 insertions(+) create mode 100644 tests/stand/smoke/README.md create mode 100644 tests/stand/smoke/__init__.py create mode 100644 tests/stand/smoke/conftest.py create mode 100644 tests/stand/smoke/login.py create mode 100644 tests/stand/smoke/test_deploy_smoke.py diff --git a/tests/pyproject.toml b/tests/pyproject.toml index 5662c3f1c..ed21d0c22 100644 --- a/tests/pyproject.toml +++ b/tests/pyproject.toml @@ -81,6 +81,7 @@ markers = [ "requires_ingestion: needs a stand whose manifest declares the 'ingestion' capability; skipped with a reason when it does not", "requires_catalogue(*parts): needs rows the analytics seed writes ('table_columns', 'definition_override'); skipped with a reason on a stand seeded without that step", "requires_service_principal: needs a stand whose authenticator token listener this runner can reach, so a service principal can be obtained; skipped with a reason when it cannot", + "stand_smoke: the post-deploy gate in tests/stand/smoke — addresses a DEPLOYED stand at $SMOKE_BASE_URL rather than the compose stand, so a lane that means to run only the compose suite selects with `-m 'not stand_smoke'`", ] [tool.ruff] diff --git a/tests/stand/smoke/README.md b/tests/stand/smoke/README.md new file mode 100644 index 000000000..4f8dc26c7 --- /dev/null +++ b/tests/stand/smoke/README.md @@ -0,0 +1,177 @@ +# The deployed-stand smoke gate + +Four checks that answer one question about a **deployed** stand: *can a user log +in and see data?* They are the gate a post-merge deployment run is graded on — +CI publishes an umbrella chart, upgrades the test stand to that exact version, +re-seeds it, and then runs this directory against the stand's **public URL**. + +Everything else under `tests/stand/` targets the local compose stand. This +directory does not, and that difference is the reason it is its own package: + +| Directory | Targets | Proves | +|---|---|---| +| `api/` | compose | the HTTP contract, per operation and per status code | +| `ui/` | compose | the rendered product, in a real browser | +| `smoke/` | a **deployed** stand, at `$SMOKE_BASE_URL` | that this deployment works at all | + +It reuses the suite's shared library (`tests/lib/insight_stand`) for everything +below the credential: the manifest reader, `LoginSession`'s real +authorization-code + PKCE chain, and `ApiClient`. Nothing here writes a new HTTP +client, mints a token, forges a cookie, or talks to Keycloak's admin API. + +## What each check proves + +Definition order is the diagnosis — each check narrows the previous one's +answer, so the **first** failure names the layer that broke. + +1. **`test_the_login_route_redirects_to_an_oidc_authorize_endpoint`** + `GET /auth/login` redirects to an OIDC authorize endpoint carrying + `response_type=code`, PKCE (`code_challenge_method=S256`), the `openid` + scope, and a `redirect_uri` whose path is `/auth/callback`. + *Proves:* the public URL resolves, the edge routes `/auth/*`, the + authenticator has an issuer configured for this host, and it could reach its + login-state store. The **target host is never asserted** — only the shape — + so the stand's IdP hostname stays in the environment and out of this + repository. + +2. **`test_each_seeded_persona_can_log_in`** (per persona) + The whole chain through the public URL: `/auth/login` → the IdP's real login + page → the form submit → `/auth/callback` → a `__Host-sid` session. + *Proves:* a credential this stand accepts exists, and the deployed OIDC flow + completes. Several personas rather than one, because a single login cannot + tell a working realm from one that happens to work for one user. + +3. **`test_auth_me_names_the_authenticated_persona`** (per persona) + `GET /auth/me` is 200 and reports that persona's email, their manifest person + id, and the stand's tenant. + *Proves:* the session belongs to the person who logged in. Check 2 only + proves the IdP accepted a credential; a stand where every login resolves to + the same person, or to the wrong tenant, passes check 2 and fails here. + +4. **`test_a_seeded_metric_answers_over_the_seeded_window`** + `POST /api/analytics/v1/metric-results` for the lead persona over the + manifest's own `data_window`, probing each key in + `SEED_GUARANTEED_METRIC_KEYS` in its own request. + *Proves:* the seeded data reaches the API as a number. Asserts a period value + present and non-null, a timeseries with at least one series, at least one + point, and at least one non-null point value — **never a number**. The seed's + golden set is empty by design, so asserting a value read back off a running + stand would prove only that the code which produced it produced it. + +## Running it locally + +```bash +export SMOKE_BASE_URL="https://" +export SMOKE_PERSONA_PASSWORD="" + +uv sync --project tests +uv run --project tests --frozen pytest tests/stand/smoke \ + --stand-manifest /path/to/manifest.json +``` + +`--stand-manifest` is how the run learns who was seeded. On a cluster the seed +Job writes its manifest inside a pod whose filesystem is discarded and **echoes +the whole document to its log**, so the caller captures it from +`kubectl logs job/` and hands the file here (or sets +`$INSIGHT_STAND_MANIFEST`). Without a manifest the session aborts: a defaulted +manifest would turn "this stand was never seeded" into a green suite. + +Naming `tests/stand/smoke` on the command line is what marks the run as +*aimed*. If `$SMOKE_BASE_URL` is unset the run aborts immediately with a +`UsageError`. If some broader collection sweeps this directory up — the compose +lane runs `pytest tests/stand --ignore=tests/stand/ui` and would otherwise +collect it — the checks **skip with a printed reason** instead, because a deploy +gate has no business turning a compose lane red. Select against that explicitly +with `-m "not stand_smoke"` if you would rather not collect them at all. + +## Environment variables + +Nothing here has a default. A deploy gate that guesses an address or a +credential is worse than one that refuses to start, so every missing value is +reported by name before a single request is made. + +| Variable | Required | Meaning | +|---|---|---| +| `SMOKE_BASE_URL` | always | The stand's public address, the one a human would type. Also what aims the run. | +| `SMOKE_LOGIN_MODE` | no (default `password`) | `password` or `override` — see below. | +| `SMOKE_PERSONA_PASSWORD` | in `password` mode | One secret shared by every persona. | +| `SMOKE_PERSONA_PASSWORD__` | no | Overrides the shared value for one persona. `` is the manifest fixture name upper-cased (`SMOKE_PERSONA_PASSWORD__DEV_LEAD`). | +| `SMOKE_BOOTSTRAP_EMAIL` | in `override` mode | The one principal that authenticates for real. | +| `SMOKE_BOOTSTRAP_PASSWORD` | in `override` mode | That principal's IdP password. | +| `INSIGHT_STAND_MANIFEST` | one of these two | Path to the manifest the seed run wrote… | +| `--stand-manifest` | …or the flag | …same thing, on the command line. | + +In CI every secret above comes from the `insight-test-stand` GitHub environment, +never from the repository. + +## The login blocker + +**A scripted username/password login only works if the stand's realm serves a +password form.** A Keycloak realm that brokers login to an external OAuth +provider does not: its browser flow is `auth-cookie` OR +`identity-provider-redirector`, so the authorize endpoint answers a redirect +straight to the provider and there is nothing for a script to submit. +`LoginSession` stops there deliberately — it requires a 200 carrying a +`login-actions/authenticate` form, and it refuses to post credentials to any +origin but the IdP's. + +This suite does not paper over that. It implements the two configurations that +can actually work and makes the operator pick one; when neither is in place the +checks **fail**, and the failure message names the realm, the step the login +stopped at, and both options. + +### `SMOKE_LOGIN_MODE=password` (default) + +Every persona authenticates as themselves. + +*Requires of the stand:* the realm carries a **local user per persona** — one +whose username/email is the persona's, with a password credential — and a +browser flow that reaches a forms step. Because the authenticator resolves a +person by `(source_type, external_id)` and the seeder writes +`external_id = ` for the whole roster whenever the +release's `--idp-source-type` resolves to a real realm, such a user must carry +the claim the deployment reads as its external id set to that same UUID. + +*Proves the most:* N independent real logins, each with its own credential, +through the real redirect. + +### `SMOKE_LOGIN_MODE=override` + +**One** bootstrap principal authenticates, and each persona session is minted +through the product's own view-as path: `GET /auth/login?__override=`, +which the authenticator resolves **by email** against the same +`identity.persons` rows the seeder writes. + +*Requires of the stand:* the authenticator running with `override_enabled`. When +it is off the parameter is ignored (and logged), which shows up here as check 3 +failing on `impersonator_email` rather than as a false pass. + +*Proves less, and says so:* every session is flagged as an impersonation, so +what is exercised is one credential plus the product's own person resolution, +not N independent authentications. Use it when the stand cannot be given local +users; prefer `password` otherwise. + +### Ruled out, so nobody re-proposes it + +A service-account / client-credentials / token-exchange token **cannot** stand +in for a persona here. The gateway reads `__Host-sid` and overwrites the +`Authorization` header with the JWT it fetches for that session, so a token +minted anywhere else never reaches `/api/*` through the public URL — it would +test nothing this gate is for. + +## House rules this directory follows + +* **No writes.** Every request is a read; the gate runs against a stand CI is + about to hand to people. +* **No exact metric values.** See check 4 and + `src/ingestion/tools/seed/golden_metrics.py`. +* **No generated-model validation.** Body validation against the OpenAPI models + is `api/`'s job and it is a contract test. A deploy gate that went red because + a generated model gained a field would be crying wolf about the one thing it + is supposed to be trusted on, so the shape checks here are hand-written and + narrow. +* **No hardcoded persona list.** The roster is resolved from the manifest's + `fixtures{}` catalog by realm role, so it keeps meaning "one person at each + authority level this stand actually seeded" through a roster reshuffle. +* **No skips for a broken stand.** The only skip in this directory is "this run + was not aimed at a deployed stand", and it prints its reason. diff --git a/tests/stand/smoke/__init__.py b/tests/stand/smoke/__init__.py new file mode 100644 index 000000000..e4b0335c2 --- /dev/null +++ b/tests/stand/smoke/__init__.py @@ -0,0 +1,28 @@ +"""Post-deploy smoke checks against a DEPLOYED (cluster) stand. + +Four checks, in narrowing order — the edge answers, a person can authenticate, +the session names that person, and the seeded data is actually queryable. They +exist to gate a deployment: after CI publishes an umbrella chart, upgrades the +test stand to that exact version and re-seeds it, this directory is what says +"a user can log in and see data" or names what broke instead. + +Why a sibling of `api/` and `ui/` rather than a module inside them: + +* `api/` is a CONTRACT suite. It asserts status codes per operation, validates + every body against the generated OpenAPI models, feeds the per-operation + coverage gate, and carries a scratch-mutation policy with a session-scoped + leak sweep. A deploy gate must not inherit any of that: it has to stay green + when a model gains a field, and it must never write to the stand it is + smoke-testing. +* `ui/` needs a browser. A deploy gate should fail in seconds on an HTTP + answer, not minutes later on a screenshot; the browser journeys already + cover the rendered half against the compose stand. + +Everything else is shared. The manifest reader, `LoginSession`, `ApiClient` and +the persona helpers all come from `tests/lib/insight_stand`; nothing here +re-implements a transport, a login or a credential lookup. + +Still a package, all the way down — pytest imports test modules by bare +basename without `__init__.py`, and this directory's module names would collide +with the ones under `api/` and `ui/`. +""" diff --git a/tests/stand/smoke/conftest.py b/tests/stand/smoke/conftest.py new file mode 100644 index 000000000..0f5750106 --- /dev/null +++ b/tests/stand/smoke/conftest.py @@ -0,0 +1,354 @@ +"""Wiring for the deployed-stand smoke: aiming it, and logging each persona in once. + +Three jobs, and the first one is the one that needs explaining. + + +1. Aiming the run +----------------- + +Every other directory under `tests/stand/` targets the local compose stand, and +`tests/stand/conftest.py` resolves that address from `$INSIGHT_STAND_BASE_URL` +or the `GATEWAY_PORT` in `.env.compose.test-stand`. This directory targets a +DEPLOYED stand through its public URL, named by `$SMOKE_BASE_URL` and by nothing +else. + +That creates one problem worth stating plainly: `tests/stand/smoke/` is +collected by anything that collects `tests/stand/`, including the existing +compose lane in `.github/workflows/e2e-stand.yml`, which narrows only with +`--ignore=tests/stand/ui`. A deploy gate that hard-failed there would turn a +lane red for a suite it was never aimed at. So the rule is: + +* the command line NAMES this directory and `$SMOKE_BASE_URL` is unset + → `pytest.UsageError`. You asked for the smoke; you forgot to aim it. +* the directory was merely swept up by a broader collection and + `$SMOKE_BASE_URL` is unset → every check SKIPS, with a reason naming the + variable. `-ra` is in `addopts`, so the reason is printed; it is not silent. +* `$SMOKE_BASE_URL` is set → everything else required is mandatory, and a + missing value aborts the session naming the variable. Nothing after this + point skips, and in particular a login that cannot work FAILS (see + `login.py`'s "the login blocker"). + +When this conftest is an INITIAL conftest — which it is whenever the command +line names a path inside this directory — its `pytest_configure` runs before the +parent's. That is what lets it hand `$SMOKE_BASE_URL` to the parent's stand +resolution, so a smoke-only run against a cluster does not abort in a fixture +that was looking for a compose env file. Verified behaviour, not an assumption: +pytest loads initial conftests during pre-parse and calls conftest hooks +deepest-first. + + +2. Choosing the personas +------------------------ + +D13 wants MULTIPLE personas, and the manifest's `fixtures{}` catalog is the only +canonical source of who exists — `PROFILE.md` is generated documentation and no +code reads it. The roster is therefore resolved BY REALM ROLE +(`resolve_by_realm_role`) rather than by hardcoded fixture names, so it keeps +meaning "one person at each authority level this stand actually seeded" through +a roster reshuffle. + +The admin OPERATOR is included when the stand seeded one, because it is the +account an operator would log in with and it authenticates through a different +row than the org roles do. It is deliberately excluded from the metric probe: +it sits outside the org chart, so it has no activity of its own. + + +3. Logging in once per persona +------------------------------ + +`smoke_login` is a session-scoped FACTORY with its own cache, mirroring +`session_for` in the parent conftest. A session-scoped fixture cannot depend on +a function-scoped parameter, and a function-scoped login fixture would +re-authenticate for every check — which on a rate-limited public stand is a +self-inflicted failure. +""" + +from __future__ import annotations + +import os +from collections.abc import Callable, Mapping +from pathlib import Path +from typing import Final + +import pytest +from insight_stand import ( + ADMIN_OPERATOR_FIXTURE, + ADMIN_ROLE, + LEAD_ROLE, + MEMBER_ROLE, + Manifest, + ManifestError, + PersonaError, + default_manifest_path, + resolve_by_realm_role, +) +from insight_stand import ( + BASE_URL_ENV as STAND_BASE_URL_ENV, +) + +from .login import ( + BASE_URL_ENV, + SmokeCredentials, + SmokeLogin, + open_smoke_session, + resolve_credentials, +) + +#: This directory, as it appears on a pytest command line. Used only to answer +#: "did the operator ask for the smoke, or merely collect it". +_SMOKE_DIR: Final[str] = "tests/stand/smoke" + +#: Role label -> how to find that persona in the manifest. Ordered, because the +#: parametrization ids and the login order follow it, and a deploy gate reads +#: better when it walks authority from the top down. +#: +#: `admin` resolves to an ORG MEMBER holding `insight-admin` (in practice the +#: CEO), never the operator — `resolve_by_realm_role` skips operator accounts +#: for exactly that reason. `lead` excludes admins so the two rows cannot +#: collapse onto the same person and make the roster one persona shorter than it +#: claims to be. +_ROLE_LOOKUPS: Final[tuple[tuple[str, str, str | None], ...]] = ( + ("admin", ADMIN_ROLE, None), + ("lead", LEAD_ROLE, ADMIN_ROLE), + ("member", MEMBER_ROLE, None), +) + +#: The role label the metric probe asks its question as. A lead has both their +#: own seeded activity and a subtree, so a 403 from the visibility gate would be +#: a real defect rather than an artefact of asking as the wrong person. +METRIC_PROBE_ROLE: Final[str] = "lead" + +_MANIFEST: Manifest | None = None +_CREDENTIALS: SmokeCredentials | None = None + + +# --------------------------------------------------------------------------- +# Aiming +# --------------------------------------------------------------------------- + + +def _aimed() -> bool: + return bool((os.environ.get(BASE_URL_ENV) or "").strip()) + + +def _explicitly_selected(config: pytest.Config) -> bool: + """Did the command line name a path inside this directory? + + `config.invocation_params.args` is the raw argument list, before pytest + normalises anything, which is what makes it the right thing to read: a + `tests/stand` sweep and a `tests/stand/smoke` request are two different + intents and only the raw arguments still tell them apart. + """ + return any( + _SMOKE_DIR in str(argument).replace(os.sep, "/") + for argument in config.invocation_params.args + ) + + +def _not_aimed_reason() -> str: + return ( + f"${BASE_URL_ENV} is not set, so this run is not aimed at a deployed stand. " + "These checks address a cluster stand through its public URL and are skipped " + "when a broader collection sweeps them up (see tests/stand/smoke/README.md)." + ) + + +def pytest_configure(config: pytest.Config) -> None: + """Refuse an unaimed explicit request, and hand the address to the parent. + + The alias is the whole reason this hook exists. `tests/stand/conftest.py` + resolves the stand's address in ITS `pytest_configure` and raises + `pytest.UsageError` when it cannot — correct for a compose run, and fatal + for a smoke-only run whose address lives in a variable that conftest has + never heard of. Copying the value across is a smaller, more honest fix than + teaching the shared conftest about a directory-specific variable, and it + only ever fires when nothing else has aimed the run. + """ + if _explicitly_selected(config) and not _aimed(): + raise pytest.UsageError( + f"${BASE_URL_ENV} is not set, but the command line asked for {_SMOKE_DIR}.\n" + " It is the deployed stand's public address, e.g. " + f"{BASE_URL_ENV}=https:// pytest {_SMOKE_DIR}\n" + " Every variable this suite reads is listed in tests/stand/smoke/README.md; " + "none of them has a default." + ) + + aimed_at = (os.environ.get(BASE_URL_ENV) or "").strip() + already_aimed = ( + config.getoption("base_url", default=None) + or config.getini("base_url") + or (os.environ.get(STAND_BASE_URL_ENV) or "").strip() + ) + if aimed_at and not already_aimed: + os.environ[STAND_BASE_URL_ENV] = aimed_at.rstrip("/") + + +# --------------------------------------------------------------------------- +# The roster, resolved at collection time +# --------------------------------------------------------------------------- + + +def _manifest(config: pytest.Config) -> Manifest: + """The stand's self-description, loaded the way the parent conftest loads it. + + Loaded again here rather than borrowed, because the parent keeps its copy in + a module-private global and the parametrization below needs the roster + BEFORE any fixture can run. Both readers are read-only and both honour + `--stand-manifest` then `$INSIGHT_STAND_MANIFEST`, so they cannot disagree + about which document describes the stand. + """ + global _MANIFEST + if _MANIFEST is None: + chosen = config.getoption("--stand-manifest") + _MANIFEST = Manifest.load(Path(str(chosen)) if chosen else default_manifest_path()) + return _MANIFEST + + +def smoke_roster(manifest: Manifest) -> Mapping[str, str]: + """Role label -> manifest fixture name, for the personas this suite drives. + + Raises `PersonaError` when a stand seeded no persona at some authority + level. That is the right disposition: a deploy gate that quietly tested one + persona because the other two could not be found would still be green while + proving a third of what it claims. + """ + roster = { + label: resolve_by_realm_role(manifest, role, excluding=excluding) + for label, role, excluding in _ROLE_LOOKUPS + } + # Present on any stand the seeder wrote; absent only on a stand seeded by + # something else, in which case there is nothing to log in as and nothing to + # report — the org roles above already cover D13's "multiple personas". + if ADMIN_OPERATOR_FIXTURE in manifest.seeded_names: + roster["operator"] = ADMIN_OPERATOR_FIXTURE + return roster + + +def pytest_generate_tests(metafunc: pytest.Metafunc) -> None: + """Parametrize the per-persona checks over the roster this stand actually has. + + The ids are the manifest fixture names rather than the role labels, so a + failing check names the person a reader can look up in the manifest. + """ + if "persona_name" not in metafunc.fixturenames: + return + + try: + names = tuple(smoke_roster(_manifest(metafunc.config)).values()) + except (ManifestError, PersonaError) as exc: + if not _aimed(): + # Not our stand and not our problem: the checks are about to be + # skipped anyway, and a placeholder keeps that skip visible instead + # of turning an unaimed sweep into a collection error. + metafunc.parametrize("persona_name", [""], ids=["unresolved"]) + return + raise pytest.UsageError( + f"cannot choose the smoke personas from the stand's manifest: {exc}\n" + " The roster is resolved by realm role, so this means the manifest " + "describes no persona at one of the levels this suite drives.\n" + " Re-seed the stand and point the run at the manifest that seed wrote " + "(--stand-manifest / $INSIGHT_STAND_MANIFEST)." + ) from exc + + metafunc.parametrize("persona_name", names, ids=names) + + +def _credentials() -> SmokeCredentials: + """Resolve the environment once per session, raising `PersonaError` on a gap.""" + global _CREDENTIALS + if _CREDENTIALS is None: + _CREDENTIALS = resolve_credentials() + return _CREDENTIALS + + +def pytest_collection_modifyitems(config: pytest.Config, items: list[pytest.Item]) -> None: + """Skip this directory when the run was not aimed; validate the config when it was. + + Both halves act only on THIS directory's items — `items` is the whole + session's collection, and a conftest that reached outside its own tree would + be a trap. + + The configuration check lives here rather than in the `smoke_credentials` + fixture because a `UsageError` raised in a fixture is not a session abort: it + becomes one ERROR per test, so a single missing password would be reported + ten identical times and the actual sentence would scroll past. Raised from a + collection hook it stops the session once, before any request is made — the + same contract the parent conftest gives `requires_seed`. + """ + del config + here = Path(__file__).parent + mine = [item for item in items if here in Path(str(item.path)).parents] + if not mine: + return + + if not _aimed(): + reason = _not_aimed_reason() + for item in mine: + item.add_marker(pytest.mark.skip(reason=reason)) + return + + try: + _credentials() + except PersonaError as exc: + raise pytest.UsageError( + f"the deployed-stand smoke is aimed at {os.environ.get(BASE_URL_ENV)} but its " + f"configuration is incomplete:\n{exc}" + ) from exc + + +# --------------------------------------------------------------------------- +# Fixtures +# --------------------------------------------------------------------------- + + +@pytest.fixture(scope="session") +def smoke_credentials() -> SmokeCredentials: + """The stand's address and the way in, from the environment. Never defaulted. + + Already validated at collection time (above), so by the time a test asks for + it this cannot fail — the `except` is here for a run that reaches the fixture + by some other route, not as the primary report. + """ + try: + return _credentials() + except PersonaError as exc: + raise pytest.UsageError(f"smoke configuration is incomplete:\n{exc}") from exc + + +@pytest.fixture(scope="session") +def smoke_base_url(smoke_credentials: SmokeCredentials) -> str: + """The deployed stand's public address — `$SMOKE_BASE_URL`, and only that. + + Deliberately NOT pytest-base-url's `base_url` fixture. That one is resolved + by the parent conftest and, in a whole-suite run, points at the compose + stand; addressing it from here would smoke-test the wrong deployment and + pass. + """ + return smoke_credentials.base_url + + +@pytest.fixture(scope="session") +def smoke_personas(stand_manifest: Manifest) -> Mapping[str, str]: + """Role label -> manifest fixture name, for the personas under test.""" + return smoke_roster(stand_manifest) + + +@pytest.fixture(scope="session") +def smoke_login( + stand_manifest: Manifest, smoke_credentials: SmokeCredentials +) -> Callable[[str], SmokeLogin]: + """`smoke_login("dev_lead")` → that persona's login attempt, made once. + + Cached per session, so a persona authenticates at the IdP once no matter how + many checks ask about them. The attempt is returned whether or not it + succeeded — see `SmokeLogin`; the login check is what turns a failure into a + readable assertion. + """ + cache: dict[str, SmokeLogin] = {} + + def _login(name: str) -> SmokeLogin: + if name not in cache: + cache[name] = open_smoke_session(name, stand_manifest, smoke_credentials) + return cache[name] + + return _login diff --git a/tests/stand/smoke/login.py b/tests/stand/smoke/login.py new file mode 100644 index 000000000..d962acee9 --- /dev/null +++ b/tests/stand/smoke/login.py @@ -0,0 +1,447 @@ +"""Winning a session on a DEPLOYED stand, where the compose credential story does not hold. + +`insight_stand.personas.open_session` is the suite's normal way in, and this +module deliberately does NOT use it — for exactly two reasons, both about where +the credential comes from rather than about how the login works: + +* `persona_password()` resolves ONE variable (`INSIGHT_STAND_PERSONA_PASSWORD`) + shared by every persona, and otherwise falls back to + `deploy/compose/keycloak/realm-insight.generated.json`. That file does not + exist for a cluster stand, and a deploy gate wants the option of a distinct + secret per persona. +* A cluster stand may not be able to serve a password form at all (see + "the login blocker" below), in which case the only honest way in is the + product's own view-as path. That is a different `/auth/login` request, not a + different transport. + +Everything BELOW the credential is reused verbatim: `LoginSession` drives the +real authorization-code+PKCE chain through the public URL, and `ApiClient` is +the only thing that issues requests. Nothing here mints a token, forges a +cookie, or talks to Keycloak's admin API. + + +The login blocker +----------------- + +A Keycloak realm that brokers login to an external OAuth provider renders no +username/password form: its browser flow is `auth-cookie` OR +`identity-provider-redirector`, so the authorize endpoint answers a redirect +straight to the provider and there is nothing for a script to submit. +`LoginSession._fetch_login_form` fails there by design — it requires a 200 +carrying a `login-actions/authenticate` form, and it refuses to post credentials +to any origin but the IdP's. No amount of test code works around that; the realm +has to present a password form, or the login has to be a different login. + +So this module implements BOTH supported configurations and makes the operator +choose one explicitly, with `$SMOKE_LOGIN_MODE`: + +`password` (default) + Every persona authenticates as themselves. Requires the stand's realm to + carry a LOCAL user per persona, with a password supplied by the environment. + +`override` + ONE bootstrap principal authenticates, and each persona session is minted + through the product's own view-as path — `GET /auth/login?__override=` + — which the authenticator resolves by email against the same + `identity.persons` rows the seeder writes. Requires the authenticator to run + with `override_enabled`. + +What is NOT implemented, and will not be: exchanging a client-credentials or +service token for a persona session. The gateway reads `__Host-sid` and +OVERWRITES the `Authorization` header with the JWT it fetches for that session, +so a token minted anywhere else cannot reach `/api/*` through the public URL — +it would test nothing this suite is for. + +When neither mode can complete, the failure is loud and names the realm: see +`describe_login_failure`. It is never a skip. +""" + +from __future__ import annotations + +import os +from collections.abc import Mapping, Sequence +from dataclasses import dataclass, field +from enum import StrEnum +from typing import Final +from urllib.parse import quote, urlsplit + +import httpx +from insight_stand import ( + ApiClient, + LoginNotCompletedError, + LoginSession, + Manifest, + Person, + PersonaError, + PersonaSession, + verify_realm_roles, +) +from insight_stand.session import LOGIN_PATH + +#: The stand's PUBLIC address. Required, and deliberately without a default of +#: any kind: this suite is aimed at a deployed stand from the outside, and a +#: default would either be a real host committed to a public repository or a +#: localhost guess that silently smoke-tests the wrong thing. +BASE_URL_ENV: Final[str] = "SMOKE_BASE_URL" + +#: `password` | `override` — see the module docstring. Defaulted rather than +#: required because `password` is the configuration we want stands to have; a +#: stand that needs `override` is opting out of that and should say so. +LOGIN_MODE_ENV: Final[str] = "SMOKE_LOGIN_MODE" + +#: One secret shared by every persona, which is what a stand provisioned from a +#: single CI secret has. +PERSONA_PASSWORD_ENV: Final[str] = "SMOKE_PERSONA_PASSWORD" + +#: `SMOKE_PERSONA_PASSWORD__DEV_LEAD` overrides the shared value for the +#: `dev_lead` manifest fixture. The suffix is the fixture name upper-cased with +#: `-` mapped to `_`, so it is a legal environment-variable name and still reads +#: as the fixture it belongs to. +PERSONA_PASSWORD_PREFIX: Final[str] = "SMOKE_PERSONA_PASSWORD__" + +#: `override` mode only: the one principal that can actually authenticate. +BOOTSTRAP_EMAIL_ENV: Final[str] = "SMOKE_BOOTSTRAP_EMAIL" +BOOTSTRAP_PASSWORD_ENV: Final[str] = "SMOKE_BOOTSTRAP_PASSWORD" + +#: The view-as query parameter the authenticator reads (`LoginParams.__override`). +#: Honoured only when the deployment sets `override_enabled`; otherwise the +#: authenticator logs the attempt and ignores the parameter, which shows up here +#: as a session belonging to the bootstrap principal rather than the persona — +#: caught by the `/auth/me` check, not papered over. +OVERRIDE_PARAM: Final[str] = "__override" + + +class LoginMode(StrEnum): + """How a persona session is obtained on this stand.""" + + PASSWORD = "password" + OVERRIDE = "override" + + +def _required(environ: Mapping[str, str], name: str, why: str) -> str: + value = (environ.get(name) or "").strip() + if not value: + raise PersonaError( + f"${name} is not set. {why}\n" + f" See tests/stand/smoke/README.md for the full variable list. " + f"Nothing here has a default — a deploy gate that guesses an address " + f"or a credential is worse than one that refuses to start." + ) + return value + + +@dataclass(frozen=True) +class SmokeCredentials: + """Everything the smoke needs to authenticate, resolved from the environment. + + The secrets carry `repr=False` for the same reason `LoginSession.password` + does: pytest prints fixture reprs in a traceback, and this suite's output is + a public CI log. + """ + + mode: LoginMode + base_url: str + bootstrap_email: str = "" + bootstrap_password: str = field(default="", repr=False) + shared_password: str = field(default="", repr=False) + persona_passwords: Mapping[str, str] = field(default_factory=dict, repr=False) + + def password_for(self, fixture_name: str) -> str: + """The credential this persona logs in with, per-persona value first.""" + specific = self.persona_passwords.get(fixture_name, "") + if specific: + return specific + if self.shared_password: + return self.shared_password + raise PersonaError( + f"no password for the {fixture_name!r} persona: neither " + f"${PERSONA_PASSWORD_PREFIX}{_env_suffix(fixture_name)} nor " + f"${PERSONA_PASSWORD_ENV} is set" + ) + + +def _env_suffix(fixture_name: str) -> str: + return fixture_name.upper().replace("-", "_") + + +def resolve_credentials(environ: Mapping[str, str] | None = None) -> SmokeCredentials: + """Read the smoke's configuration, or raise `PersonaError` naming what is missing. + + Raising beats defaulting at every branch here. A missing base URL, a missing + password and an unknown mode are all operator mistakes that a deploy gate + must report before it touches the stand, not discover halfway through a + parametrized run. + """ + env = os.environ if environ is None else environ + + base_url = _required( + env, + BASE_URL_ENV, + "It is the stand's public address, the one a human would type.", + ).rstrip("/") + + raw_mode = (env.get(LOGIN_MODE_ENV) or LoginMode.PASSWORD.value).strip().lower() + try: + mode = LoginMode(raw_mode) + except ValueError: + modes = ", ".join(sorted(m.value for m in LoginMode)) + raise PersonaError( + f"${LOGIN_MODE_ENV}={raw_mode!r} is not a login mode; use one of: {modes}. " + "See tests/stand/smoke/README.md for what each one requires of the stand." + ) from None + + per_persona = { + key[len(PERSONA_PASSWORD_PREFIX) :].lower(): value.strip() + for key, value in env.items() + if key.startswith(PERSONA_PASSWORD_PREFIX) and value.strip() + } + + if mode is LoginMode.OVERRIDE: + return SmokeCredentials( + mode=mode, + base_url=base_url, + bootstrap_email=_required( + env, + BOOTSTRAP_EMAIL_ENV, + f"In {LoginMode.OVERRIDE.value!r} mode one principal authenticates for real " + "and every persona session is minted from it.", + ), + bootstrap_password=_required( + env, + BOOTSTRAP_PASSWORD_ENV, + f"It is the bootstrap principal's IdP password ({BOOTSTRAP_EMAIL_ENV}).", + ), + ) + + return SmokeCredentials( + mode=mode, + base_url=base_url, + shared_password=_required( + env, + PERSONA_PASSWORD_ENV, + f"In {LoginMode.PASSWORD.value!r} mode every persona authenticates as themselves. " + f"One shared value is enough; ${PERSONA_PASSWORD_PREFIX} overrides it " + "for a single persona.", + ), + persona_passwords=per_persona, + ) + + +def override_login_path(email: str) -> str: + """`/auth/login?__override=` — the product's own view-as entry point.""" + return f"{LOGIN_PATH}?{OVERRIDE_PARAM}={quote(email, safe='@')}" + + +def _redact(url: str | None) -> str: + """Scheme, host and path — never the query string. + + An IdP authorize URL carries `state`, `nonce`, `code_challenge` and the + client id, and everything this module writes lands in a CI log on a public + repository. The path alone is what makes a failure diagnosable; the query is + only noise that happens to be sensitive. + """ + if not url: + return "" + parts = urlsplit(url) + if not parts.scheme or not parts.netloc: + return parts.path or "" + return f"{parts.scheme}://{parts.netloc}{parts.path}" + + +def describe_login_failure( + *, + name: str, + email: str, + manifest: Manifest, + credentials: SmokeCredentials, + reason: str, + stopped_at: str | None, +) -> str: + """The actionable message a failed login reports, with the realm named. + + Long on purpose. The overwhelmingly likely cause is a stand-configuration + fact — the realm has no password form, or view-as is switched off — and + whoever reads this in a CI log is not the person who wrote the suite. A + one-line "login failed" would send them to the wrong place every time. + """ + realm = manifest.realm + issuer = _redact(realm.issuer) if realm.issuer else "" + return "\n".join( + [ + f"persona {name!r} ({email}) could not obtain a session at " + f"{credentials.base_url} in {credentials.mode.value!r} mode.", + f" stopped at: {_redact(stopped_at)}", + f" reason: {reason}", + f" realm: {realm.name!r} (issuer {issuer}), per {manifest.source_path}", + "", + "The usual cause is the stand's realm, not the product. A Keycloak realm", + "that brokers login to an external OAuth provider renders NO username/password", + "form — its browser flow redirects straight to the provider — so a scripted", + "login has nothing to submit. insight_stand's LoginSession stops there", + "deliberately rather than inventing a credential path the product does not have.", + "", + "Two supported configurations, and this suite implements both:", + "", + f" {LOGIN_MODE_ENV}={LoginMode.PASSWORD.value}", + " The stand's realm carries a LOCAL user per persona, with a password form", + f" in its browser flow. Credentials come from ${PERSONA_PASSWORD_ENV}", + f" (or ${PERSONA_PASSWORD_PREFIX} per persona).", + "", + f" {LOGIN_MODE_ENV}={LoginMode.OVERRIDE.value}", + " The authenticator runs with override_enabled, ONE principal authenticates", + f" (${BOOTSTRAP_EMAIL_ENV} / ${BOOTSTRAP_PASSWORD_ENV}), and every persona", + " session is minted through GET /auth/login?__override=, which", + " resolves by email against the persons rows the seeder writes.", + "", + "Neither is something a test can arrange for itself: both are stand", + "configuration. tests/stand/smoke/README.md states what has to be true.", + ] + ) + + +@dataclass(frozen=True) +class SmokeLogin: + """One persona's login attempt — the session, or why there is none. + + A failed login is CARRIED rather than raised so the login check can report + it as a failing assertion naming the persona, instead of every test that + touches that persona erroring out in fixture setup with the same traceback. + """ + + name: str + person: Person + mode: LoginMode + persona: PersonaSession | None = None + failure: str | None = None + + def require(self) -> PersonaSession: + """The session, or an `AssertionError` carrying the whole diagnosis. + + The only accessor, deliberately: an `ok` predicate beside it would invite + `assert attempt.ok, attempt.failure`, and pytest's assertion rewriting + then appends a truncated `SmokeLogin(...)` repr underneath the sentence + that was supposed to be the whole message. + """ + if self.persona is None: + raise AssertionError(self.failure or f"no session for persona {self.name!r}") + return self.persona + + +def open_smoke_session( + name: str, + manifest: Manifest, + credentials: SmokeCredentials, + *, + timeout_s: float = 30.0, +) -> SmokeLogin: + """Log a manifest fixture in, capturing any failure instead of raising it. + + `name` is a key in the manifest's `fixtures{}` catalog — never an email and + never a UUID — so a roster reshuffle moves the person without touching a + test. The email that reaches the IdP (and, in override mode, the email in + the `__override` parameter) is read off that fixture. + """ + person = manifest.fixture(name) + + try: + # Cheap, and it is the one thing that catches a seed whose roster and + # role mapping disagree. On a cluster the realm export is absent, so + # only the manifest-vs-roster half runs — which is the half that would + # otherwise let a persona carry quietly wrong authority. + verify_realm_roles(person) + password = ( + credentials.bootstrap_password + if credentials.mode is LoginMode.OVERRIDE + else credentials.password_for(name) + ) + except PersonaError as exc: + return SmokeLogin( + name=name, + person=person, + mode=credentials.mode, + failure=( + f"persona {name!r} ({person.email}) is not usable on this stand " + f"before any login was attempted: {exc}" + ), + ) + + if credentials.mode is LoginMode.OVERRIDE: + session = LoginSession( + base_url=credentials.base_url, + email=credentials.bootstrap_email, + password=password, + login_path=override_login_path(person.email), + timeout_s=timeout_s, + ) + else: + session = LoginSession( + base_url=credentials.base_url, + email=person.email, + password=password, + timeout_s=timeout_s, + ) + + try: + session.login() + except LoginNotCompletedError as exc: + return SmokeLogin( + name=name, + person=person, + mode=credentials.mode, + failure=describe_login_failure( + name=name, + email=person.email, + manifest=manifest, + credentials=credentials, + reason=str(exc), + stopped_at=exc.stopped_at, + ), + ) + except httpx.HTTPError as exc: + # A transport failure part-way through the chain — DNS, TLS, a timeout, + # a reset. Distinguished from the protocol failure above because it says + # something completely different about the deployment, and because the + # long realm diagnosis would be actively misleading here. + return SmokeLogin( + name=name, + person=person, + mode=credentials.mode, + failure=( + f"the login chain for persona {name!r} ({person.email}) did not complete " + f"because a request failed in transport: {type(exc).__name__}: {exc}\n" + f" stand: {credentials.base_url}\n" + f" This is the stand or the network, not the realm — the IdP hop leaves " + f"this runner, so a stand reachable at its own address can still fail here " + f"when the IdP's hostname does not resolve from CI." + ), + ) + + return SmokeLogin( + name=name, + person=person, + mode=credentials.mode, + persona=PersonaSession( + name=name, + person=person, + session=session, + client=ApiClient(base_url=credentials.base_url, session=session, timeout_s=timeout_s), + ), + ) + + +__all__: Sequence[str] = ( + "BASE_URL_ENV", + "BOOTSTRAP_EMAIL_ENV", + "BOOTSTRAP_PASSWORD_ENV", + "LOGIN_MODE_ENV", + "OVERRIDE_PARAM", + "PERSONA_PASSWORD_ENV", + "PERSONA_PASSWORD_PREFIX", + "LoginMode", + "SmokeCredentials", + "SmokeLogin", + "describe_login_failure", + "open_smoke_session", + "override_login_path", + "resolve_credentials", +) diff --git a/tests/stand/smoke/test_deploy_smoke.py b/tests/stand/smoke/test_deploy_smoke.py new file mode 100644 index 000000000..e2576ebe3 --- /dev/null +++ b/tests/stand/smoke/test_deploy_smoke.py @@ -0,0 +1,513 @@ +"""The post-deploy gate: the edge answers, people log in, and the data is there. + + GET /auth/login 302 to the IdP's authorize endpoint + GET /auth/login (per persona) the whole OIDC chain, to a session cookie + GET /auth/me 200, naming the authenticated persona + POST /api/analytics/v1/metric-results 200, with a value over the seeded window + +Four checks, one per contract, in DEFINITION ORDER — pytest runs a module's +tests in the order they appear, and that order is the point. Each check narrows +the previous one's answer, so the FIRST failure is the diagnosis: + +1. the edge is up and the authenticator is wired to an IdP; +2. a credential this stand accepts exists, and the OIDC chain completes; +3. the session that came back belongs to the person who logged in; +4. the seeded data reaches the API as a number, for the window the seed wrote. + +A stand where check 1 passes and check 2 fails has an IdP problem. One where +2 passes and 3 fails has an identity-resolution problem. One where 3 passes and +4 fails deployed and authenticated fine but has no data — three genuinely +different pages, and this ordering is what tells them apart without reading a +log. + +What this module deliberately does NOT do: + +* **Assert a metric's value.** The seed's golden set is empty by design, and + reading a number off a running stand to assert it back proves only that the + code which produced it produced it. Check 4 asserts SHAPE and NON-NULLNESS + over the manifest's own `data_window`, never a number. +* **Validate bodies against the generated OpenAPI models.** That is `api/`'s + job and it is a contract test. A deploy gate that went red because a + generated model gained a field would be crying wolf about the one thing it is + supposed to be trusted on, so the shape checks here are hand-written, narrow, + and about the fields a human would notice missing. +* **Write anything.** Every request is a read. The gate runs against a stand CI + is about to hand to people. +* **Skip when it cannot log in.** See `login.py` — an unusable realm is a + configuration failure with a named cause, not an absent capability. +""" + +from __future__ import annotations + +import math +from collections.abc import Callable, Mapping, Sequence +from dataclasses import dataclass +from typing import Final +from urllib.parse import parse_qs, urlsplit + +import httpx +import pytest +from insight_stand import ApiClient, ApiResponse, Manifest, analytics_path +from insight_stand.api import JsonValue +from insight_stand.session import CALLBACK_PATH, LOGIN_PATH, SESSION_COOKIE_NAME + +from .conftest import METRIC_PROBE_ROLE +from .login import BASE_URL_ENV, LoginMode, SmokeCredentials, SmokeLogin + +pytestmark = pytest.mark.stand_smoke + +ME_PATH: Final[str] = "/auth/me" +METRIC_RESULTS: Final[str] = analytics_path("/v1/metric-results") + +#: Query parameters an OpenID Connect authorization request carries. Asserted by +#: NAME and never by value, because every value in here is stand-specific: the +#: client id, the redirect host and the PKCE challenge all differ per +#: deployment, and pinning any of them would make this check a configuration +#: snapshot rather than a statement about the protocol. +_AUTHORIZE_PARAMS: Final[tuple[str, ...]] = ( + "response_type", + "client_id", + "redirect_uri", + "state", + "scope", + "code_challenge", + "code_challenge_method", +) + +#: Metric keys the seeder's generators write data behind, one per generator +#: family (`generators/git.py`, `generators/task.py`, `generators/collab.py`). +#: +#: Three rather than one so a single generator regressing cannot make this check +#: vacuous, and three rather than the whole catalogue so the gate stays a few +#: seconds long. Each is probed in its OWN request: an unknown `metric_key` +#: makes the whole batch 400, so batching them would let one retired key hide +#: the answer from the other two. +#: +#: Only ONE has to answer. A metric being retired from the registry is a product +#: decision that should not turn a deploy gate red — the failure message names +#: every key and what it did, so a shrinking list is visible rather than silent. +SEED_GUARANTEED_METRIC_KEYS: Final[tuple[str, ...]] = ( + "git.commits", + "tasks.closed", + "collab.messages_sent", +) + + +# --------------------------------------------------------------------------- +# 1. The edge answers and the authenticator is wired to an IdP +# --------------------------------------------------------------------------- + + +def test_the_login_route_redirects_to_an_oidc_authorize_endpoint(smoke_base_url: str) -> None: + """`GET /auth/login`, unauthenticated, starts a real authorization-code flow. + + The cheapest possible statement that the deployment is alive end to end: the + public URL resolves, the edge routes `/auth/*` to the authenticator, the + authenticator has an issuer configured for this host, and it can reach its + login-state store — a 429 or a 500 here all mean something different from a + 302. + + Asserted by SHAPE, never by host. The stand's IdP hostname is deployment + configuration and belongs in the environment, not in a test in a public + repository; what the product actually promises is an absolute redirect to an + OIDC authorize endpoint carrying PKCE, and that is what is checked. + """ + try: + response = ApiClient(base_url=smoke_base_url).get(LOGIN_PATH) + except httpx.HTTPError as exc: + # The first request of the run, so this is where "the deploy never + # became reachable" lands. Turned into a stated failure rather than left + # as a transport traceback: the useful facts are the address and where + # it came from, and neither of those is in an httpx stack trace. + pytest.fail( + f"the stand did not answer at all: {type(exc).__name__}: {exc}\n" + f" address: {smoke_base_url} (from ${BASE_URL_ENV})\n" + f" Nothing below this can run. Either the deployment never became " + f"reachable at that address, or ${BASE_URL_ENV} names the wrong one." + ) + + assert response.status_code in (301, 302, 303, 307, 308), ( + f"GET {LOGIN_PATH} answered {response.status_code} instead of redirecting to the IdP. " + f"A 404 means the edge does not route /auth/*; a 5xx means the authenticator is up " + f"but could not start a login; a 200 means something other than the authenticator " + f"answered. Body: {response.text[:300]}" + ) + + location = response.headers.get("location", "") + assert location, ( + f"GET {LOGIN_PATH} answered {response.status_code} with no Location header — " + f"there is nothing for a browser to follow. Headers: {sorted(response.headers)}" + ) + + target = urlsplit(location) + assert target.scheme in ("http", "https") and target.netloc, ( + f"the login redirect target is not an absolute URL: {location!r}. The authenticator " + f"builds it from the issuer it discovered, so a relative or empty target means the " + f"issuer is misconfigured for this host." + ) + + query = parse_qs(target.query) + missing = [name for name in _AUTHORIZE_PARAMS if not query.get(name, [""])[0]] + assert not missing, ( + f"the login redirect is missing {missing} — that is not an OIDC authorization " + f"request. Redirected to {target.scheme}://{target.netloc}{target.path} with " + f"parameters {sorted(query)}." + ) + assert query["response_type"][0] == "code", ( + f"the authenticator asked for response_type={query['response_type'][0]!r}; this " + f"product only implements authorization code + PKCE." + ) + assert query["code_challenge_method"][0] == "S256", ( + f"PKCE challenge method is {query['code_challenge_method'][0]!r}, not S256 — " + f"the flow started without a usable proof key." + ) + assert "openid" in query["scope"][0].split(), ( + f"the authorization request asked for scope {query['scope'][0]!r}, which does not " + f"include 'openid', so no id_token would come back and no person could be resolved." + ) + + callback = urlsplit(query["redirect_uri"][0]) + assert callback.path == CALLBACK_PATH, ( + f"the IdP is told to send the code to {callback.path!r} rather than {CALLBACK_PATH!r}; " + f"the authenticator's configured redirect_uri does not match the route it serves, so " + f"every login would dead-end after the IdP." + ) + + +# --------------------------------------------------------------------------- +# 2. A seeded persona can actually log in +# --------------------------------------------------------------------------- + + +def test_each_seeded_persona_can_log_in( + smoke_login: Callable[[str], SmokeLogin], persona_name: str +) -> None: + """The whole OIDC chain, once per persona, through the public URL. + + Nothing is stubbed and nothing is minted: `/auth/login` → the IdP's real + login page → the form submit → `/auth/callback` → `__Host-sid`. Several + personas rather than one because a single login only proves that ONE + credential works — a realm that granted roles to one user, an identity + resolution that resolved one person, and a tenant claim that happened to + match are all indistinguishable from a working stand until a second person + tries. + + The whole diagnosis lives in the assertion message, including which stand + configuration is missing when the IdP cannot serve a password form at all. + `require()` rather than `assert attempt.ok, attempt.failure`: the bare assert + makes pytest append its own rewritten `where False = SmokeLogin(...)` line, + which repeats the diagnosis inside a truncated dataclass repr and buries it. + """ + persona = smoke_login(persona_name).require() + + assert persona.session.is_authenticated(), ( + f"persona {persona_name!r} completed the login chain but holds no live " + f"{SESSION_COOKIE_NAME} session — the callback answered without setting the cookie, " + f"or it expired between being set and being read." + ) + + +# --------------------------------------------------------------------------- +# 3. The session belongs to the person who logged in +# --------------------------------------------------------------------------- + + +def test_auth_me_names_the_authenticated_persona( + smoke_login: Callable[[str], SmokeLogin], + persona_name: str, + stand_manifest: Manifest, + smoke_credentials: SmokeCredentials, +) -> None: + """`/auth/me` reports the persona's own identity, not just *an* identity. + + This is the check that makes check 2 mean something. A session cookie proves + the IdP accepted a credential; it does not prove the authenticator resolved + the right person, put the right tenant on the session, or that identity + holds a row for them at all. A stand where every login silently resolves to + the same person, or to the wrong tenant, passes check 2 and fails here. + + The person id is asserted rather than only the email because it is the key + every person-scoped route takes — the same value check 4 asks the metric + about. + """ + persona = smoke_login(persona_name).require() + response = persona.client.get(ME_PATH) + + assert response.status_code == 200, ( + f"GET {ME_PATH} answered {response.status_code} for {persona.email} while carrying " + f"a session cookie. A 401 means the authenticator no longer recognises the session " + f"it just issued (a session store that lost it, or a TTL shorter than this run); " + f"a 5xx means it could not read it. Body: {response.text[:300]}" + ) + + body = _json_object(response, f"{ME_PATH} for {persona.email}") + + reported_email = str(body.get("email", "")) + assert reported_email.casefold() == persona.email.casefold(), ( + f"logged in as {persona.email!r} and {ME_PATH} reports {reported_email!r}. The " + f"session was minted for a different person than the one who authenticated — " + f"compare against the manifest at {stand_manifest.source_path}." + ) + assert str(body.get("user", "")) == persona.person.uuid, ( + f"{ME_PATH} resolved {persona.email} to person id {body.get('user')!r}, but the " + f"manifest says {persona.person.uuid!r}. Identity resolved the login to the wrong " + f"row, so every person-scoped query in this session would be about somebody else." + ) + assert str(body.get("tenant_id", "")) == stand_manifest.tenant, ( + f"{ME_PATH} put tenant {body.get('tenant_id')!r} on {persona.email}'s session, but " + f"the stand was seeded for {stand_manifest.tenant!r}. Every tenant-scoped query " + f"would come back empty and look like missing data." + ) + + if smoke_credentials.mode is LoginMode.OVERRIDE: + # In view-as mode the session is deliberately an impersonation, and the + # authenticator says so. Asserting it keeps the two modes honestly + # different: a run that BELIEVES it is impersonating but is really just + # logged in as the bootstrap principal would otherwise pass every check + # above with the wrong person's data. + assert str(body.get("impersonator_email", "")).casefold() == ( + smoke_credentials.bootstrap_email.casefold() + ), ( + f"{LoginMode.OVERRIDE.value!r} mode expects {ME_PATH} to name the real principal " + f"behind the view-as session as {smoke_credentials.bootstrap_email!r}, and it " + f"reports {body.get('impersonator_email')!r}. Either override_enabled is off on " + f"this stand (the authenticator then ignores __override and logs the attempt), " + f"or the session is not the one this run thinks it is." + ) + + +# --------------------------------------------------------------------------- +# 4. The seeded data reaches the API +# --------------------------------------------------------------------------- + + +def test_a_seeded_metric_answers_over_the_seeded_window( + smoke_login: Callable[[str], SmokeLogin], + smoke_personas: Mapping[str, str], + stand_manifest: Manifest, +) -> None: + """A real number comes back for a real person over the window the seed wrote. + + The end of the chain this gate exists for. Everything before it can pass on + a stand whose ClickHouse is empty, whose gold models were never rebuilt, or + whose tenant on the session does not match the tenant the rows carry — and a + user would see a dashboard of dashes. This is the check that notices. + + The period is the manifest's own `data_window`, so the request asks for the + range the stand was actually seeded over rather than a guess, and the anchor + moves with the seed instead of being pinned to a date in a test. + + Values are asserted NON-NULL and FINITE and nothing more. Not the number: + the seed's golden set is empty by design (see + `src/ingestion/tools/seed/golden_metrics.py`) and asserting a value read + back off a running stand proves only that the code which produced it + produced it. Not even non-negativity, tempting as it is for three counters: + that is a fact about the metric definitions, it belongs to the dbt gold + tests that already assert it (`assert_ic_kpis_bounds`, + `assert_collab_messaging_bounds`), and encoding it here would make a + definition change look like a broken deployment. + """ + persona = smoke_login(smoke_personas[METRIC_PROBE_ROLE]).require() + start, _, end = stand_manifest.data_window.partition("..") + assert start and end, ( + f"the manifest at {stand_manifest.source_path} carries data_window " + f"{stand_manifest.data_window!r}, which is not a `from..to` range — there is no " + f"period to ask about." + ) + + probes = tuple( + _probe(persona.client, persona.person.uuid, start, end, metric_key) + for metric_key in SEED_GUARANTEED_METRIC_KEYS + ) + report = "\n".join(f" {probe.summary()}" for probe in probes) + + # Bound to a name rather than left as `any(... for ...)`: pytest's assertion + # rewriting appends what it evaluated, and a bare generator prints as + # `where False = any()`, which is noise directly under + # the sentence that matters. + answered = [probe for probe in probes if probe.plausible] + assert answered, ( + f"no seed-guaranteed metric answered with data for {persona.email} over the seeded " + f"window {stand_manifest.data_window}:\n{report}\n" + f" Every probe reached the API through the public URL with a real session, so this " + f"is not an auth failure. The usual causes, in order: the stand was deployed but " + f"never seeded; the seed wrote silver but the gold models were not rebuilt after it; " + f"the rows carry a different tenant than the session does " + f"({stand_manifest.tenant}); or all three metric keys have been retired from the " + f"registry, which the statuses above would show as 400." + ) + + malformed = [probe for probe in probes if probe.status == 200 and probe.note] + assert not malformed, ( + "a metric answered 200 with a body this gate could not read:\n" + + "\n".join(f" {probe.summary()}" for probe in malformed) + + "\n The shape checked here is only what a dashboard needs — a period value and a " + "timeseries with points — so a failure means the response is genuinely not that." + ) + + +# --------------------------------------------------------------------------- +# Reading the answers +# --------------------------------------------------------------------------- + + +def _json_object(response: ApiResponse, what: str) -> Mapping[str, JsonValue]: + """The body as a JSON object, or a failure naming what was asked for.""" + body = response.json() + if not isinstance(body, dict): + raise AssertionError( + f"{what} did not answer with a JSON object (content-type " + f"{response.content_type or ''}): {response.text[:300]}" + ) + return body + + +def _number(value: JsonValue) -> float | None: + """A finite number, or None for anything else — including a JSON `true`.""" + if isinstance(value, bool) or not isinstance(value, int | float): + return None + return float(value) if math.isfinite(value) else None + + +def _objects(value: JsonValue) -> list[dict[str, JsonValue]]: + """The JSON objects in a list, or nothing. + + Walking the body through this rather than indexing it keeps `_probe` free of + `isinstance` ladders and makes "the field was there but held the wrong kind + of thing" behave the same as "the field was absent" — both are reported as a + missing part of the shape, which is what a reader of the failure needs. + """ + if not isinstance(value, list): + return [] + return [item for item in value if isinstance(item, dict)] + + +@dataclass(frozen=True) +class MetricProbe: + """What one metric key did when asked about one person over one period.""" + + metric_key: str + status: int + #: Empty when the body was the shape a dashboard needs; otherwise the first + #: thing about it that was not. + note: str = "" + period_values: tuple[float | None, ...] = () + series: int = 0 + points: int = 0 + non_null_points: int = 0 + + @property + def plausible(self) -> bool: + """Answered 200, in shape, with at least one real number on both views.""" + return ( + self.status == 200 + and not self.note + and any(value is not None for value in self.period_values) + and self.series >= 1 + and self.points >= 1 + and self.non_null_points >= 1 + ) + + def summary(self) -> str: + detail = self.note or ( + f"period values {list(self.period_values)}, {self.series} series, " + f"{self.points} points ({self.non_null_points} non-null)" + ) + return f"{self.metric_key}: HTTP {self.status} — {detail}" + + +def _probe(client: ApiClient, person_id: str, start: str, end: str, metric_key: str) -> MetricProbe: + """Ask one metric for one person over one period, and describe the answer. + + Shape problems are RECORDED rather than raised, so the check above can + report every key at once. A gate that stopped at the first unreadable body + would hide the two keys that might have answered perfectly well. + """ + response = client.post( + METRIC_RESULTS, + json_body={ + "entity": {"type": "person", "ids": [person_id]}, + "period": {"from": start, "to": end}, + "metrics": [ + { + "metric_key": metric_key, + # Both views, because they fail differently: a period value + # can be non-null while the timeseries is empty (a bucketing + # or window bug), and a timeseries can carry points while the + # period scalar is null (an aggregation bug). A dashboard + # renders both. + "views": [{"view": "period"}, {"view": "timeseries"}], + } + ], + }, + ) + if response.status_code != 200: + return MetricProbe( + metric_key=metric_key, + status=response.status_code, + note=f"not answered: {response.text[:200]}", + ) + + body = response.json() + if not isinstance(body, dict): + return MetricProbe( + metric_key=metric_key, + status=200, + note=f"the body is not a JSON object: {response.text[:200]}", + ) + + metrics = _objects(body.get("metrics")) + answered = [str(entry.get("metric_key", "")) for entry in metrics] + if answered != [metric_key]: + return MetricProbe( + metric_key=metric_key, + status=200, + note=( + f"asked for one metric and the response answered for {answered}: " + f"{response.text[:200]}" + ), + ) + + by_view = {str(view.get("view", "")): view for view in _objects(metrics[0].get("views"))} + + period = by_view.get("period") + if period is None: + return MetricProbe( + metric_key=metric_key, + status=200, + note=f"no period view came back; views present: {sorted(by_view)}", + ) + period_values = tuple( + _number(entry.get("value")) + for entry in _objects(period.get("values")) + if str(entry.get("entity_id", "")) == person_id + ) + if not period_values: + return MetricProbe( + metric_key=metric_key, + status=200, + note=f"the period view carried no value for the person asked about ({person_id})", + ) + + timeseries = by_view.get("timeseries") + if timeseries is None: + return MetricProbe( + metric_key=metric_key, + status=200, + note=f"no timeseries view came back; views present: {sorted(by_view)}", + period_values=period_values, + ) + series = _objects(timeseries.get("series")) + points = [point for entry in series for point in _objects(entry.get("points"))] + + return MetricProbe( + metric_key=metric_key, + status=200, + period_values=period_values, + series=len(series), + points=len(points), + non_null_points=sum(1 for point in points if _number(point.get("value")) is not None), + ) + + +__all__: Sequence[str] = ("SEED_GUARANTEED_METRIC_KEYS",) From 6d043a0cf788e783a7c37ff1ccc48301f148e841 Mon Sep 17 00:00:00 2001 From: Konstantin Tursunov Date: Sun, 9 Aug 2026 12:18:35 +0800 Subject: [PATCH 05/59] ci(stand): do not inherit every repository secret into the deploy MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit `secrets: inherit` passes the caller's ENTIRE secret set to the reusable workflow, including the GitHub App key that is on `main`'s branch-protection bypass list. The deploy needs none of them: its credentials live in the `insight-test-stand` environment, and environment secrets are resolved by the job that declares `environment:` — which is inside the called workflow. Semgrep's yaml.github-actions.security.secrets-inherit rule blocks this, and it is right to. Not inheriting fails loudly rather than silently: the first cluster-touching step refuses with a named error naming the missing secret. Refs #2244 Signed-off-by: Konstantin Tursunov --- .github/workflows/build-images.yml | 18 +++++++++++++++++- 1 file changed, 17 insertions(+), 1 deletion(-) diff --git a/.github/workflows/build-images.yml b/.github/workflows/build-images.yml index 5bc9960f7..748a28515 100644 --- a/.github/workflows/build-images.yml +++ b/.github/workflows/build-images.yml @@ -2096,4 +2096,20 @@ jobs: uses: ./.github/workflows/deploy-test-stand.yml with: chart_version: ${{ needs.publish-chart.outputs.chart_version }} - secrets: inherit + # No `secrets:` key, deliberately — not an omission. + # + # `deploy-test-stand.yml` reads no caller secrets. Every credential it uses + # lives in the `insight-test-stand` GitHub environment, and environment + # secrets are resolved by the job that declares `environment:` — which is + # inside the called workflow, not here. + # + # `secrets: inherit` would hand it EVERY repository secret, including the + # GitHub App key that sits on `main`'s branch-protection bypass list. Giving + # a repository-write bypass credential to a job whose purpose is to run + # tooling against a cluster is a gratuitous blast-radius increase on a public + # repo, which is why semgrep's yaml.github-actions.security.secrets-inherit + # rule blocks it. The rule is right. + # + # The failure mode of not inheriting is loud: the called workflow's first + # cluster-touching step refuses with a named error listing exactly which + # environment secret is missing. From 0aca4e79610664141aacdbab9adcc4ef60ddefdf Mon Sep 17 00:00:00 2001 From: Konstantin Tursunov Date: Sun, 9 Aug 2026 12:32:48 +0800 Subject: [PATCH 06/59] fix(stand): redact underscore-joined secret keys, and grant the cross-namespace RBAC the chart needs MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit Two defects found by review, both of which only show up in production. REDACTION. `_SECRET_KEY` began with ``, which does not match between `_` and a letter — underscore is a word character. Every underscore-joined key therefore passed through the filter untouched, which is the dominant naming convention in this stack: MARIADB_PASSWORD=… leaked kubectl_token=… leaked APP__gears__authenticator__config__idp__client_secret: leaked while a bare `password:` was correctly redacted, so the filter looked like it worked. On a public repository those lines are published forever. The boundary is now `(?-airbyte-auth-reader` — a Role granting `get` on one Secret, plus its RoleBinding — into the namespace named by `airbyte.namespace`, which this environment must set: leaving it empty puts the RBAC where Airbyte's `airbyte-auth-secrets` does not exist and connector provisioning fails at run time while every pod looks healthy. A credential scoped only to the release namespace cannot write those two objects, so the FIRST `helm upgrade` would have been refused — CI red before it deployed anything. Verified against the stand: the live release owns both objects in that namespace, and `can-i update rolebindings` there answers `no`. The supplement is deliberately tiny: rights over `roles`/`rolebindings`, plus `get` restricted by `resourceNames` to the single Secret the chart's own Role names — the latter only because escalation prevention forbids creating a Role granting a permission the creator does not hold. Removing it does not shrink the credential, it breaks the deploy. Pods, other Secrets and workloads in that namespace stay unreachable, and the script now asserts all three facts. Refs #2244 Signed-off-by: Konstantin Tursunov --- .github/workflows/scripts/redact-stand-log.py | 19 ++- .../gitops/scripts/provision-ci-deployer.sh | 110 +++++++++++++++++- 2 files changed, 122 insertions(+), 7 deletions(-) diff --git a/.github/workflows/scripts/redact-stand-log.py b/.github/workflows/scripts/redact-stand-log.py index 2c800e3c8..b641a4046 100755 --- a/.github/workflows/scripts/redact-stand-log.py +++ b/.github/workflows/scripts/redact-stand-log.py @@ -94,13 +94,28 @@ #: Keys whose value is material wherever they appear — kubeconfig, Secret dumps, #: rendered values, connection errors. Matched case-insensitively against the #: token immediately left of a `:` or `=`. +#: +#: The leading boundary is `(?-airbyte-auth-reader` into `airbyte.namespace` whenever that +# value is non-empty — and the test-stand environment must set it, because +# leaving it empty puts the RBAC in the release namespace where Airbyte's +# `airbyte-auth-secrets` does not exist, and connector provisioning then +# fails at run time while every pod still looks healthy. A credential scoped +# only to the release namespace cannot write those two objects, so the FIRST +# `helm upgrade` is refused outright. This script therefore provisions a +# second, deliberately tiny supplement in that namespace: rights over +# `roles`/`rolebindings`, plus `get` on the one Secret name the chart's Role +# grants — the latter only because RBAC escalation prevention forbids +# creating a Role granting a permission the creator does not itself hold. +# It grants nothing else there; Airbyte's own workloads and its other +# Secrets stay out of reach. # # ═══════════════════════════════════════════════════════════════════════════ # SAFETY PROPERTIES OF THE SCRIPT ITSELF @@ -210,6 +224,10 @@ APPLY=0 MODE="provision" # provision | rotate | revoke | purge | verify SHOW_SERVER=0 WITH_SUPPLEMENT=1 +# The chart renders its airbyte-auth-reader Role/RoleBinding into this +# namespace whenever `airbyte.namespace` is non-empty in the values. Empty +# string disables the second supplement entirely. +AIRBYTE_NAMESPACE="airbyte" TOKEN_WAIT_S=60 usage() { @@ -256,6 +274,11 @@ Behaviour: --no-supplement Skip the Gateway API / Argo / cert-manager supplemental Role. Only for a cluster whose CRD providers ship aggregate-to-admin ClusterRoles. + --airbyte-namespace NAME The namespace the chart renders its + airbyte-auth-reader Role into, i.e. the values' + `airbyte.namespace` (default: airbyte). Empty + string skips that second supplement — correct only + when the values leave `airbyte.namespace` unset. --show-server Print the API server URL unredacted. --token-wait SECONDS How long to wait for the token controller to fill the Secret (default: 60). @@ -335,6 +358,10 @@ while [ $# -gt 0 ]; do APPLY=1 shift ;; + --airbyte-namespace) + AIRBYTE_NAMESPACE="${2-}" + shift + ;; --no-supplement) WITH_SUPPLEMENT=0 shift @@ -370,6 +397,8 @@ command -v kubectl >/dev/null 2>&1 || die "kubectl is required (brew install kub ROLE_NAME="${SA_NAME}-crd-supplement" RB_ADMIN="${SA_NAME}-admin" RB_SUPPLEMENT="${SA_NAME}-crd-supplement" +AIRBYTE_ROLE_NAME="${SA_NAME}-airbyte-rbac" +AIRBYTE_RB_NAME="${SA_NAME}-airbyte-rbac" # `kubectl` against the ADMIN kubeconfig. Every call in this script that uses # it is either read-only or gated behind $APPLY. @@ -583,6 +612,62 @@ subjects: name: ${SA_NAME} namespace: ${NAMESPACE} EOF + + if [ -n "$AIRBYTE_NAMESPACE" ]; then + cat <-airbyte-auth-reader\` — a Role granting \`get\` on the single Secret +# \`airbyte-auth-secrets\`, plus its RoleBinding — into the namespace named by +# the values' \`airbyte.namespace\`. Without rights over those two objects there, +# the FIRST \`helm upgrade\` is refused and CI is red before it has deployed +# anything. +# +# The \`secrets\` rule below looks like a widening and is the opposite: it is +# restricted by \`resourceNames\` to the exact Secret the chart's own Role names, +# and it exists only because RBAC escalation prevention forbids creating a Role +# that grants a permission the creator does not hold. Removing it does not make +# the credential smaller; it makes the deploy fail. +# +# Nothing else in this namespace is reachable: no pods, no other Secrets, no +# workloads, no exec. +apiVersion: rbac.authorization.k8s.io/v1 +kind: Role +metadata: + name: ${AIRBYTE_ROLE_NAME} + namespace: ${AIRBYTE_NAMESPACE} + labels: + app.kubernetes.io/name: ${SA_NAME} + app.kubernetes.io/component: ci-credential + app.kubernetes.io/managed-by: provision-ci-deployer.sh +rules: + - apiGroups: ["rbac.authorization.k8s.io"] + resources: [roles, rolebindings] + verbs: [get, list, watch, create, update, patch, delete] + - apiGroups: [""] + resources: [secrets] + resourceNames: [airbyte-auth-secrets] + verbs: [get] +--- +apiVersion: rbac.authorization.k8s.io/v1 +kind: RoleBinding +metadata: + name: ${AIRBYTE_RB_NAME} + namespace: ${AIRBYTE_NAMESPACE} + labels: + app.kubernetes.io/name: ${SA_NAME} + app.kubernetes.io/component: ci-credential + app.kubernetes.io/managed-by: provision-ci-deployer.sh +roleRef: + apiGroup: rbac.authorization.k8s.io + kind: Role + name: ${AIRBYTE_ROLE_NAME} +subjects: + - kind: ServiceAccount + name: ${SA_NAME} + namespace: ${NAMESPACE} +EOF + fi fi cat < Date: Mon, 10 Aug 2026 09:34:32 +0800 Subject: [PATCH 07/59] ci(stand): let a dispatch run deploy and seed without the smoke credentials MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit The smoke stage needs a login credential the deploy and seed stages do not, and the preflight demands every secret before the first stage runs. That is right for the merge path — being told twenty minutes in that a password is missing is worse than being told immediately — but it made "does this chart install and seed on this stand" unanswerable until an unrelated question about IdP credentials was settled. `stages` is a PREFIX of deploy -> seed -> smoke, never a set: each stage is the next one's precondition, and independent checkboxes would let someone smoke yesterday's data against today's chart and call it a pass. The preflight now requires the smoke credentials only when smoke will actually run, and still validates the login-mode string either way so a typo is caught on the run that introduces it. A partial run is a rehearsal, not a green light, and the workflow says so three times rather than trusting anyone to remember: a warning annotation, a stages row plus a blockquote in the job summary, and the skipped steps in the UI. The ref guard is now absolute for `workflow_call` and advisory for `workflow_dispatch`. A call arrives from build-images.yml after a publish, which only happens on main, so a call from any other ref means the caller's guard broke. A dispatch is a person asking on purpose — usually to rehearse this workflow on the branch that changes it, which cannot be done from main by definition. What decides whether a dispatch may touch the stand is the environment's deployment-branch policy, enforced before the job starts. Refs #2244 Signed-off-by: Konstantin Tursunov --- .github/workflows/deploy-test-stand.yml | 115 +++++++++++++++++++++++- 1 file changed, 111 insertions(+), 4 deletions(-) diff --git a/.github/workflows/deploy-test-stand.yml b/.github/workflows/deploy-test-stand.yml index 962614f9b..dcf5d369b 100644 --- a/.github/workflows/deploy-test-stand.yml +++ b/.github/workflows/deploy-test-stand.yml @@ -161,6 +161,12 @@ on: Exact umbrella chart version to install, as published to oci://ghcr.io/constructorfabric/charts/insight — e.g. 0.5.101. Comes from publish-chart's output, never from deploy/gitops/.insight-version: that file is only committed at the END of publish-chart, so a checkout of the trigger SHA reads the PREVIOUS release. required: true type: string + stages: + description: >- + Which stages to run, as a prefix of deploy -> seed -> smoke: `all`, `deploy+seed`, or `deploy`. The caller has no reason to pass anything but the default; it exists on this trigger only so the value has one definition rather than two. A partial run is never a pass — the summary and the run title both say which stages were skipped. + required: false + default: all + type: string # The human path: redeploy a specific version by hand, e.g. after fixing the # stand by hand and wanting CI's view of it back. Same input, same code path — # there is no manual-only branch anywhere below. @@ -170,6 +176,15 @@ on: description: "Umbrella chart version to install (e.g. 0.5.101). Must already be published." required: true type: string + stages: + description: "Stages to run. A prefix only: you cannot smoke without seeding, or seed without deploying." + required: false + default: all + type: choice + options: + - all + - deploy+seed + - deploy # One stand, one deploy at a time. `cancel-in-progress: false` because the thing # being cancelled would be a `helm upgrade` mid-flight: killing it leaves the @@ -260,11 +275,29 @@ jobs: EVENT: ${{ github.event_name }} run: | set -euo pipefail + # Absolute for workflow_call, advisory for workflow_dispatch, and the + # asymmetry is deliberate. A CALL comes from build-images.yml after a + # publish, and a publish only happens on main: a call carrying any + # other ref means the caller's guard has broken, which is a bug rather + # than a request. A DISPATCH is a person with write access asking for a + # specific version on purpose — typically to rehearse this workflow on + # the branch that changes it, which cannot be done from main by + # definition. What actually decides whether a dispatch may touch the + # stand is the environment's deployment-branch policy, which is + # enforced by GitHub before this job starts and cannot be talked out of + # by anything written here. So dispatch reports the ref and continues. if [ "$REF_NAME" != "main" ]; then - echo "::error::this workflow deploys main and only main (ref was '$REF_NAME', event '$EVENT')." - echo "The insight-test-stand environment's deployment-branch policy is the real control;" - echo "this check exists so the refusal is readable instead of a missing-secret failure." - exit 1 + if [ "$EVENT" = "workflow_dispatch" ]; then + echo "::warning::dispatched from '$REF_NAME', not main. Allowed because the" \ + "insight-test-stand environment's deployment-branch policy already" \ + "decided this ref may deploy — but what lands on the stand is that" \ + "branch's idea of the deploy, not main's." + else + echo "::error::this workflow deploys main and only main (ref was '$REF_NAME', event '$EVENT')." + echo "A workflow_call arrives from build-images.yml after a publish, which only" + echo "happens on main, so a call from anywhere else means the caller's guard broke." + exit 1 + fi fi if ! printf '%s' "$CHART_VERSION" | grep -Eq '^[0-9]+\.[0-9]+\.[0-9]+([-+][0-9A-Za-z.-]+)?$'; then echo "::error::chart_version '$CHART_VERSION' is not a semver. Refusing to pass it to make." @@ -272,6 +305,45 @@ jobs: fi echo "deploying umbrella chart $CHART_VERSION to ENV=$ENV_NAME" + # ── Which stages run ────────────────────────────────────────────────── + # `stages` is a PREFIX of deploy -> seed -> smoke, never a set. You cannot + # smoke without seeding or seed without deploying, because each stage is + # the previous one's precondition: the smoke reads personas the seed + # writes, and the seed reads coordinates the deploy settles. Offering the + # stages as independent checkboxes would let someone smoke yesterday's data + # against today's chart and call it a pass. + # + # A partial run is a REHEARSAL, not a green light, and this workflow says + # so in three places rather than trusting anyone to remember: a warning + # annotation here, the per-stage lines in the summary, and the skipped + # steps in the UI. The reason it exists at all is that the smoke needs a + # credential the deploy does not, so "does this chart install and seed on + # this stand" is a question worth being able to answer on its own. + - name: Resolve which stages run + id: stages + env: + STAGES: ${{ inputs.stages }} + run: | + set -euo pipefail + case "${STAGES:-all}" in + all) deploy=true seed=true smoke=true ;; + deploy+seed) deploy=true seed=true smoke=false ;; + deploy) deploy=true seed=false smoke=false ;; + *) + echo "::error::stages '$STAGES' is not one of: all, deploy+seed, deploy" + exit 1 ;; + esac + { + echo "run_deploy=$deploy" + echo "run_seed=$seed" + echo "run_smoke=$smoke" + } >> "$GITHUB_OUTPUT" + echo "stages: ${STAGES:-all} (deploy=$deploy seed=$seed smoke=$smoke)" + if [ "$smoke" != "true" ]; then + echo "::warning::partial run — stages='${STAGES:-all}'. The smoke gate did NOT run," \ + "so this run says the chart installs, not that the stand works." + fi + - name: Confirm the environment is wired up env: # Presence only. Values are never printed; GitHub masks the secrets @@ -291,6 +363,7 @@ jobs: # (one principal authenticates and every persona session is minted from # it). Unset means the suite's own default, which is `password`. LOGIN_MODE: ${{ vars.TEST_STAND_SMOKE_LOGIN_MODE }} + RUN_SMOKE: ${{ steps.stages.outputs.run_smoke }} run: | set -euo pipefail missing="" @@ -302,6 +375,22 @@ jobs: # same thing far better than this does — but only after a deploy and a # seed have already run. Twenty minutes is too long to wait to be told # a password is missing. + # + # Skipped entirely when the smoke stage is not running: demanding a + # credential for work that will not happen turns a deliberate partial + # rehearsal into an unrunnable one, which is exactly the trap this + # input exists to avoid. The mode string is still validated, because a + # typo there should be caught on the run that introduces it rather than + # on the first full run afterwards. + if [ "$RUN_SMOKE" != "true" ]; then + case "${LOGIN_MODE:-password}" in + password|override) ;; + *) + echo "::error::vars.TEST_STAND_SMOKE_LOGIN_MODE is '$LOGIN_MODE'; it must be 'password' or 'override'" + exit 1 ;; + esac + echo "smoke stage not selected — its credentials are not required for this run" + else case "${LOGIN_MODE:-password}" in override) [ "$HAVE_BOOTSTRAP_EMAIL" = "true" ] || missing="$missing secrets.TEST_STAND_BOOTSTRAP_EMAIL" @@ -312,6 +401,7 @@ jobs: echo "::error::vars.TEST_STAND_SMOKE_LOGIN_MODE is '$LOGIN_MODE'; it must be 'password' or 'override'" exit 1 ;; esac + fi if [ -n "$missing" ]; then echo "::error::the insight-test-stand environment is missing:$missing" @@ -611,6 +701,7 @@ jobs: # --deadline should come down with the step budget, in one change, with the # measurement quoted. - name: 'Stage 2/3 — seed the stand' + if: steps.stages.outputs.run_seed == 'true' timeout-minutes: 65 env: SEED_EMAIL: ${{ secrets.TEST_STAND_SEED_EMAIL }} @@ -645,6 +736,7 @@ jobs: 2>&1 | tee "$RUNNER_TEMP/seed.log" | python3 "$REDACT" - name: 'Stage 2/3 — capture the seed manifest' + if: steps.stages.outputs.run_seed == 'true' run: | set -euo pipefail # A cluster seed Job's filesystem dies with its pod, so the seeder @@ -704,6 +796,7 @@ jobs: # above skips it. That is the whole mechanism — asserting against an # unseeded stand would produce a failure that looks like a product bug. - name: 'Stage 3/3 — smoke the stand through its public URL' + if: steps.stages.outputs.run_smoke == 'true' timeout-minutes: 5 env: # The public origin, driven exactly as a browser would drive it: real @@ -770,6 +863,9 @@ jobs: if: always() env: OUTCOME: ${{ job.status }} + RUN_SEED: ${{ steps.stages.outputs.run_seed }} + RUN_SMOKE: ${{ steps.stages.outputs.run_smoke }} + STAGES: ${{ inputs.stages }} run: | set -euo pipefail { @@ -779,8 +875,19 @@ jobs: echo "|---|---|" echo "| chart | \`insight-$CHART_VERSION\` |" echo "| environment | \`$ENV_NAME\` (namespace \`$STAND_NAMESPACE\`, release \`$STAND_RELEASE\`) |" + echo "| stages | \`${STAGES:-all}\` — deploy ✓, seed $([ "$RUN_SEED" = "true" ] && echo '✓' || echo '— skipped'), smoke $([ "$RUN_SMOKE" = "true" ] && echo '✓' || echo '— skipped') |" echo "| outcome | **$OUTCOME** |" echo "" + # A partial run that says nothing about being partial is worse than + # no run: the next person reads a green tick and believes the stand + # was proven. Said here as well as in the warning annotation because + # this table is what gets screenshotted into a thread. + if [ "$RUN_SMOKE" != "true" ]; then + echo "> **Partial run — this is a rehearsal, not a pass.** The smoke gate did not" + echo "> run, so nothing here claims a person can sign in or that any metric returns" + echo "> data. It says the chart installs$([ "$RUN_SEED" = "true" ] && echo ' and the stand seeds' || echo ''), and no more." + echo "" + fi if [ "$OUTCOME" != "success" ]; then echo "This is an after-merge alarm, not a merge gate: the chart under test did not" echo "exist until the merge happened. **The author of the merge commit owns this run.**" From edbcaefe5662568c90879b267adaf2f6354649dd Mon Sep 17 00:00:00 2001 From: Konstantin Tursunov Date: Mon, 10 Aug 2026 09:38:27 +0800 Subject: [PATCH 08/59] docs(stand): say why the branch allow-list is the control, and that the token is a stopgap MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit Two things the runbook set up correctly but never argued, which is how correct setup gets undone by a well-meaning later edit. The allow-list reads like bookkeeping. It is the security control. No secret store protects a value from the people who can run jobs that read it: an environment secret is readable by any job declaring that environment on an allowed branch, and someone with write access can add a step that prints it. What bounds this credential is "allow-list says main" AND "main is branch-protected" — together, a change that reads the secret must survive review first. Widening the list does not degrade that property, it removes it. Recorded alongside the two consequences already in the tree: why fork PRs cannot read it, and why `secrets: inherit` had to go. Also recorded: a rehearsal environment on a personal fork may legitimately allow a feature branch, and that is precisely why it must stay on the fork and never be copied into the org repo. And the honest framing of what this credential is. A GitHub environment secret is a static string with no refresh path, so the token is long-lived by necessity rather than preference — an accepted trade given the asserted blast radius and one-command revocation, but one whose missing property is expiry. The 90-day cadence in §5 is therefore an obligation, not a suggestion: nothing else ages this credential out. The durable answer, noted so it is a decision rather than a gap, is trusting GitHub's OIDC issuer from the API server and minting a short-lived token per run — a cluster-operator change, which is the only reason it is not the starting point. Refs #2244 Signed-off-by: Konstantin Tursunov --- .../specs/sop/credentials-runbook.md | 46 +++++++++++++++++++ 1 file changed, 46 insertions(+) diff --git a/docs/components/deployment/specs/sop/credentials-runbook.md b/docs/components/deployment/specs/sop/credentials-runbook.md index 5f0aba12a..1759b4452 100644 --- a/docs/components/deployment/specs/sop/credentials-runbook.md +++ b/docs/components/deployment/specs/sop/credentials-runbook.md @@ -223,6 +223,52 @@ gh api "repos/$REPO/environments/$ENVIRONMENT/deployment-branch-policies" \ --jq '.branch_policies[].name' # -> main ``` +#### The allow-list is the security control, not a tidiness preference + +Worth being explicit, because it is easy to read the list above as bookkeeping. +No secret store protects a value from the people who can run jobs that read it: +an environment secret is readable by **any** job that declares +`environment: insight-test-stand` on an allowed branch, and someone with write +access can add a step that prints it. Encryption at rest does nothing about +that. + +So what actually bounds this credential is the pair "allow-list says `main`" and +"`main` is branch-protected". Together they mean a change that reads the secret +has to survive review before it can run. Widen the allow-list and that property +is gone — not degraded, gone — which is why: + +* a pull request from a fork cannot read it (GitHub withholds environment + secrets from fork PRs, and the allow-list would refuse anyway); +* `secrets: inherit` was removed from the calling job. Inheriting would have + handed this job every repository secret, including the App key that bypasses + branch protection on `main` — a worse exposure than the kubeconfig, and one + that would have undone the control described here; +* a **rehearsal** environment on a personal fork may legitimately allow a + feature branch, and that widening is exactly why it must stay on the fork. + Never copy a rehearsal allow-list into `constructorfabric/insight`. + +#### This credential is a deliberate stopgap + +A GitHub environment secret is a static string with no refresh path, so this is +a long-lived, non-expiring token by necessity rather than by preference. That is +an accepted trade, not an oversight: the token is namespace-scoped, its blast +radius is asserted on every provisioning run, and revoking it is one command +with no cluster-wide consequence. + +The property it cannot have is expiry. Treat the 90-day cadence in §5 as a real +obligation rather than a suggestion — nothing else ages this credential out. +Kubernetes' legacy-token cleanup only reaps tokens unused for a year, and a +token used on every merge never qualifies. + +The durable answer is a credential that is never stored: Kubernetes 1.30+ +structured authentication config can trust GitHub's OIDC issuer directly, so the +workflow presents a short-lived token minted per run and the cluster maps its +claims (`repository`, `ref`, `environment`) onto the same namespace-scoped RBAC +this script already creates. Trust then reads "runs from this repository, on this +ref, in this environment" instead of "whoever holds this string", and there is +nothing in GitHub left to leak. It is a cluster-operator change rather than a +repository one, which is the only reason it is not the starting point. + ### 4.2 Load the secrets | Secret | Value | Notes | From 78d12253e5d8132b84ad983d7a1effb8d17c407c Mon Sep 17 00:00:00 2001 From: Konstantin Tursunov Date: Mon, 10 Aug 2026 16:48:14 +0800 Subject: [PATCH 09/59] feat(gitops): re-sync the test-stand environment to the redeployed stand MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit The stand was rebuilt from the reference deployment repository, and two things this environment asserted stopped being true. THE EDGE. The umbrella grew native Gateway API templates, so `gateway.route`, `keycloak.route` and `frontend.route` replace the `ingress` keys and the release renders and owns both HTTPRoutes. The two manifests this directory carried are therefore deleted rather than kept: a hand-applied copy beside a release that renders the same object is a second writer, and the check that guarded them has not disappeared but inverted — `helm --wait` does not wait on HTTPRoute status, so the deploy now asserts `Accepted` on what it has just written. THE IdP. The stand runs `INSIGHT_LOGIN_MODE=seeded`: realm `insight`, generated from the demo seed's own roster with local password users, not the GitHub-federated `insight-broker`. So `sourceType` is `keycloak` and the external-IdP prose is gone from the Secret and scope comments — there is no external IdP here to describe. `externalIdClaim` stays `idp_sub` and now says why it is not `sub`: the generator sets each user's id to the roster uuid, which would land in `sub` on a realm import, but the chart applies realms with keycloak-config-cli, which creates users through the admin REST API where Keycloak assigns its own id and discards the document's. A "fix" to `sub` produces a stand where every persona authenticates and is then denied, against a fully populated identity projection, with nothing pointing back at that line. A server-side dry run of chart 0.5.111 with this values file renders 38 objects and `helm get manifest` of the live release is 38 objects — nothing added, nothing removed. The comment corrections changed no value: parsed YAML is identical to the previous revision. INFRA.md is new, and answers the question this tree could not: what has to exist beneath the umbrella. Layer model, deploy order with its causal edges, a version-pinned component inventory, and — the part worth the file — 29 load-bearing settings each written as the setting, what breaks without it, and how the breakage presents. It opens with `maxUserConnections: 100`, because at the operator's default of 10 that one account cannot back four connection pools and the symptom is not a recognisable connection limit: two services crash-loop while every other pod stays Ready, and the upgrade dies blaming a client rate limiter. Refs #2244 Signed-off-by: Konstantin Tursunov --- .../gitops/environments/test-stand/INFRA.md | 1010 +++++++++++++++++ .../gitops/environments/test-stand/README.md | 201 +++- .../environments/test-stand/inventory.yaml | 26 +- .../manifests/httproute-keycloak.yaml | 49 - .../test-stand/manifests/httproute.yaml | 63 - .../environments/test-stand/values.yaml | 238 +++- 6 files changed, 1408 insertions(+), 179 deletions(-) create mode 100644 deploy/gitops/environments/test-stand/INFRA.md delete mode 100644 deploy/gitops/environments/test-stand/manifests/httproute-keycloak.yaml delete mode 100644 deploy/gitops/environments/test-stand/manifests/httproute.yaml diff --git a/deploy/gitops/environments/test-stand/INFRA.md b/deploy/gitops/environments/test-stand/INFRA.md new file mode 100644 index 000000000..bf44c57f3 --- /dev/null +++ b/deploy/gitops/environments/test-stand/INFRA.md @@ -0,0 +1,1010 @@ +# `test-stand` — what has to exist beneath the umbrella + +[`README.md`](README.md) describes the one thing this directory deploys: the +umbrella Helm release `insight`, from [`values.yaml`](values.yaml). This file +describes everything the release lands *on top of* — the cluster, the eight +infrastructure components under it, and the settings among them that are not +obvious and not recoverable by guesswork. + +It exists because the release cannot tell you any of it. `helm upgrade --wait` +reports Ready pods and a `deployed` release; it does not report a Redis the +authenticator's client cannot address, an Argo controller that will never look +at the CronWorkflows it was just given, or a database account four services +are about to exhaust. Most of the failures collected below share one signature: +**every pod Ready, the release `deployed`, and the thing that matters silently +not happening.** That sentence is the most useful one in this document. + +> **Scope.** This is the L0/L2 half of the ownership boundary in +> [`README.md`](README.md#the-ownership-boundary). Nothing here is deployed by +> anything in this repository, and nothing here is deployed by CI. It is +> written so that a stand of this shape can be rebuilt, and so that a reviewer +> of a change to `values.yaml` can see what that value is coupled to. + +## What this file is, and what it is not + +**This file is the specification. The deployment repository is the executable +form.** That repository holds one `deploy-.sh` and one `validate-.sh` +per component, a `deploy-all.sh` that runs them in a fixed order, per-stand +overlays under `stands/`, and the manifests and values files they apply. Running +it is how a stand actually gets built. Reading this file is how you find out +what it is doing and why, without a checkout of it and without cluster access. + +That split is deliberate, and it is the same split +[`inventory.yaml`](inventory.yaml) already makes for the L0/L2 layers: this +environment is a values overlay plus a cluster address, and the layers beneath +it are brought up and owned elsewhere. What was missing until now is any record +*in this repository* of what those layers are. A CI job that deploys a chart +into a cluster it did not build, and gates a merge on the result, needs that +record — otherwise the first unexplained red run is an archaeology exercise +against a repository the reader may not have. + +**Two consequences follow, and both need stating out loud.** + +*This file can drift from the executable form.* There is no generation step and +no test that compares the two. A change made in the deployment repository — a +version bump, a new required setting, a namespace rename — reaches the stand +without touching anything here. Treat every version and every name below as +"true when it was verified against a live stand", not as "true now". + +*Drift is cheap to detect, and the checks are read-only.* Against an admin +kubeconfig: + +```bash +helm list -A # against the inventory table below +kubectl get sc # `cinder`, and it must be the default +kubectl get clusterissuer # insight-selfsigned + insight-ca, both Ready +kubectl -n mariadb get user insight -o jsonpath='{.spec.maxUserConnections}{"\n"}' +kubectl -n envoy-gateway-system get gateway insight \ + -o jsonpath='{range .spec.listeners[*]}{.name}={.protocol}/{.port}{"\n"}{end}' +kubectl -n insight get sa argo-workflow +kubectl -n airbyte get role insight-airbyte-auth-reader +``` + +None of those is available to CI. The deploy credential is a namespace-scoped +ServiceAccount in `insight` that is granted nothing cluster-scoped and nothing +in another namespace beyond one Role/RoleBinding pair in `airbyte` (see +[`../../scripts/provision-ci-deployer.sh`](../../scripts/provision-ci-deployer.sh)). +It cannot `helm list -A`, cannot read `mariadb/`, cannot see the Gateway it +attaches routes to. That is the right shape for a deploy credential, and it is +also why this file has to be maintained by hand rather than generated from the +cluster on every run. + +### Why there is no `infra-versions.yaml` + +A machine-readable companion pinning the versions was considered and rejected. +Nothing in this repository would read it: the Makefile reads +`inventory.yaml` and `values.yaml` and nothing else from this directory, the +workflow reads neither, and the credential that runs the workflow cannot +enumerate the releases it would be compared against. It would be a second copy +of the numbers in the table below, kept in step by nobody — the exact failure +mode this tree spends a page of `values.yaml` warning about for a single tenant +UUID. If a checker is ever written, it belongs next to `doctor.sh` and it should +parse the table here rather than a duplicate of it. + +## The layer model + +| Layer | What | Deployed by | This repository's relationship to it | +|---|---|---|---| +| **L-1** | The cluster itself: nodes, a CNI, the OpenStack Cinder CSI driver registered as `csidriver/cinder.csi.openstack.org`, and a public DNS record for the hostname | the cloud platform, outside every repository | assumed; five deploy scripts hard-fail without the CSI driver, and the Insight step is gated on the DNS record resolving | +| **L0** | Edge and PKI: Envoy Gateway (`GatewayClass`, `Gateway`, `EnvoyProxy`), cert-manager and the two-step `ClusterIssuer` chain, the `cinder` StorageClass | the deployment repository | named in `values.yaml` (`gateway.route.parentRef`, `authenticator.tlsDiscovery.issuerRef`); never created here | +| **L2** | Datastores under their own operators — ClickHouse, MariaDB, Redis, Redpanda — plus Airbyte and Argo Workflows, each in its own namespace | the deployment repository | addressed in `values.yaml` by in-cluster Service DNS; never created here | +| **L2** | The generate-once Secrets and the realm ConfigMap the release consumes by name | the deployment repository | referenced by name in `values.yaml`; listed with `enabled: false` in `inventory.yaml` | +| **L3** | **The umbrella release `insight` in namespace `insight` — every value in `values.yaml`** | **this directory** | **owned, and changed by CI on every merge to `main`** | + +The single most important structural fact: **the release renders its own edge +routes but not the Gateway they attach to.** Since chart 0.5.107 the umbrella +carries native Gateway API templates (`gateway.route`, `keycloak.route`, +`frontend.route`), so `HTTPRoute/insight-gateway` and `HTTPRoute/insight-keycloak` +arrive with the release and carry its Helm ownership metadata. What they attach +*to* — `Gateway/insight` in `envoy-gateway-system`, its `EnvoyProxy`, its +certificate — is L0 and stays L0. An HTTPRoute whose `parentRef` names a Gateway +that does not exist is created successfully and serves nothing. + +## The deploy order + +Nine steps, and they are not interchangeable. `deploy-all.sh` runs each +`deploy-.sh` followed immediately by its `validate-.sh`, stopping at +the first failure: + +```text +envoy-gateway → cert-manager → clickhouse → mariadb → redis → redpanda + → airbyte → argo-workflows → insight ( → seed, deliberately separate ) +``` + +Two edges make that an order rather than a list, and both are worth knowing +before an attempt in the wrong order wastes an afternoon: + +* **envoy-gateway must be first**, because its chart installs the Gateway API + CRDs. cert-manager is installed with `config.enableGatewayAPI=true`, and its + gateway-shim controller refuses to start without + `gateways.gateway.networking.k8s.io`. The cert-manager deploy script fails + fast on the missing CRD rather than letting the controller crash-loop, which + is the difference between a five-second error and a twenty-minute one. +* **cert-manager back-fills what envoy-gateway deliberately left undone.** On a + fresh cluster there is no `ClusterIssuer` yet, so the Gateway's `https` + listener comes up *unprogrammed* — it has no certificate. That is expected, + not a failure. The cert-manager step completes the handshake by waiting for + `Certificate/insight-origin-tls` in the gateway namespace to go Ready. + Neither script is complete on its own, and reading either one's output in + isolation is misleading. + +The seed step is **not** in this list, and its absence is a decision: a seeded +stand is deployed and seeded by two different actions, because seeding is what +gives the stand its people and the deploy is what gives it its services. See +[Between deploy and seed](#between-deploy-and-seed). + +## Component inventory + +Versions below were cross-checked against `helm list -A` on a live stand of +this shape. They are pinned, not floating — every deploy script passes an +explicit `--version`. + +| # | Layer | Component | Namespace | Release | Chart @ version (source) | What a later step takes from it | +|---|---|---|---|---|---|---| +| — | L-1 | CNI | `kube-system` | `cilium` | `cilium` 1.16.3 | pod networking; its pod CIDR has to fall inside the ClickHouse user's IP allow-list (see §20) | +| — | L-1 | Block storage | — | — | `csidriver/cinder.csi.openstack.org` | every PVC below; five deploy scripts hard-fail if it is absent | +| — | L-1 | StorageClass | — | — | `cinder`, provisioner `cinder.csi.openstack.org`, `WaitForFirstConsumer`, `allowVolumeExpansion`, **annotated as the cluster default** | named explicitly by every datastore; taken implicitly by Airbyte's bundled PostgreSQL and MinIO (see §21) | +| 1 | L0 | Envoy Gateway | `envoy-gateway-system` | `eg` | `gateway-helm` v1.8.3 (`oci://docker.io/envoyproxy/gateway-helm`) | the Gateway API CRDs; `GatewayClass/envoy`; `Gateway/insight`; the LoadBalancer Service holding the stand's floating address | +| 2 | L0 | cert-manager | `cert-manager` | `cert-manager` | `cert-manager` v1.21.1 (`https://charts.jetstack.io`) | `ClusterIssuer/insight-ca` — the issuer `authenticator.tlsDiscovery.issuerRef` names — and, via the gateway-shim, the Gateway's origin certificate | +| 3 | L2 | ClickHouse operator | `clickhouse-operator` | `clickhouse-operator` | `altinity-clickhouse-operator` 0.27.1 (`https://helm.altinity.com`) | watches `clickhouse` only | +| 3 | L2 | ClickHouse + Keeper | `clickhouse` | — (CRs) | `ClickHouseKeeperInstallation/clickhouse-keeper`, `ClickHouseInstallation/clickhouse` | Secrets `clickhouse-default-credentials` / `clickhouse-insight-credentials`; the `insight` user; the **per-pod** Service `chi-clickhouse-clickhouse-0-0` | +| 4 | L2 | MariaDB operator | `mariadb-operator` | `mariadb-operator-crds`, `mariadb-operator` | both 26.6.0 (`oci://ghcr.io/mariadb-operator/charts/…`) | the `MariaDB`/`Database`/`User`/`Grant` CRDs | +| 4 | L2 | MariaDB (Galera ×3) | `mariadb` | — (CRs) | `MariaDB/mariadb`, image `mariadb:11.8.8` | Secrets `mariadb-root` / `mariadb-insight-credentials`; the `insight` database, user and grant; Service `mariadb-primary` | +| 5 | L2 | Redis operator | `redis-operator` | `redis-operator` | `redis-operator` 0.25.0 (`https://ot-container-kit.github.io/helm-charts`) | watches `redis` only; cluster-scoped RBAC | +| 5 | L2 | Redis (replication ×3) | `redis` | `redis` | `redis-replication` 0.17.0 (same repo) | Secret `redis-auth`; Service `redis-master`, label-selected on the operator's `redis-role=master` | +| 6 | L2 | Redpanda | `redpanda` | `redpanda` | `redpanda` 26.1.9 (`https://charts.redpanda.com`) | Service `redpanda` with the **internal** Kafka listener on **9093** | +| 7 | L2 | Airbyte | `airbyte` | `airbyte` | `airbyte` 1.8.5 (`https://airbytehq.github.io/helm-charts`) | the `airbyte` namespace itself (a deploy preflight); Secret `airbyte-auth-secrets`; the server the reconcile loop calls | +| 8 | L2 | Argo Workflows | `argo` | `argo-workflows` | `argo-workflows` 1.0.22 (`https://argoproj.github.io/argo-helm`) | the `workflowtemplates` / `cronworkflows` CRDs — **without which the umbrella cannot render at all** — and the cluster-scoped controller RBAC | +| 9 | L3 | **Insight umbrella** | `insight` | `insight` | `insight` 0.5.111 (`oci://ghcr.io/constructorfabric/charts/insight`) | the stand | + +Persistent volumes, for capacity planning: ClickHouse data 100Gi, Keeper 10Gi, +MariaDB 100Gi data plus a separate 1Gi Galera-config volume per member, Redis +10Gi per member, Redpanda 50Gi per broker, Airbyte 10Gi each for the +pre-created PostgreSQL and MinIO claims. Retention is deliberately +conservative: ClickHouse and Keeper use `reclaimPolicy: Retain`, MariaDB +retains on both `whenDeleted` and `whenScaled`, and Redis does the same. + +Node shape is a hard requirement, not a recommendation. Redpanda and MariaDB +Galera both use **hard** pod anti-affinity on `kubernetes.io/hostname` for their +three members, so a cluster with fewer than three schedulable workers leaves +both StatefulSets permanently unconverged. Redis uses **soft** anti-affinity on +purpose: a required spread across three workers deadlocks against the +operator's own master/replica anti-affinity. The reference profile is three +control-plane nodes plus three workers on Kubernetes 1.31, with workers sized +to hold the datastore requests above with headroom. + +## The load-bearing settings + +Each entry is the same three-part contract: **the setting**, **what breaks +without it**, and **how the breakage presents**. The third part is the one that +earns the section — almost every item here fails in a way that names something +other than itself. + +Ordered by how badly the failure lies to you. + +### 1. `maxUserConnections: 100` on the MariaDB `User/insight` + +**The setting.** A field on the `k8s.mariadb.com/v1alpha1` `User` object in the +`mariadb` namespace — not a chart value, and not something `values.yaml` can +influence. The operator's default is **10**. + +**What breaks without it.** That one account backs **four** connection pools: +Analytics, Identity Resolution, the chart's `migrate`/`init` Jobs, and — because +[`values.yaml`](values.yaml) points `keycloak.database.username` at the same +user rather than provisioning a second one — the bundled Keycloak, whose own +pool defaults to 100 on its own. Ten is not enough for that, and the server's +global ceiling is 151, so 100 leaves headroom for `root` and the operator's +agent. + +**How it presents.** Not as a connection-limit error anywhere you would look +first. Identity Resolution and the Analytics `migrate` initContainer crash-loop +at startup with + +```text +1226 (42000): User 'insight' has exceeded the max_user_connections resource +``` + +while every other service stays Ready and the MariaDB cluster itself reports +healthy. The deploy does not surface that message at all: `helm upgrade --wait` +sits until its timeout and then dies with + +```text +client rate limiter Wait returned an error: context deadline exceeded +``` + +which reads like an API-server or throttling problem and is not one. If a +deploy of this stack times out and the only error mentions a rate limiter, +read the pod logs of `insight-identity-resolution` before reading anything +else. + +### 2. `clickhouse.host` is the **per-pod** Service, never the shared one + +**The setting.** `chi-clickhouse-clickhouse-0-0.clickhouse.svc.cluster.local` — +the Altinity operator's `chi----` Service, +which always resolves to exactly one server. Not `clickhouse-clickhouse`, the +all-pod Service one word away. + +**What breaks without it.** The chart is not cluster-aware: no template emits +`ON CLUSTER`, there is no cluster-name value, and every table it creates is +`ReplacingMergeTree` / `MergeTree` / `View` rather than `Replicated*`. Nothing +synchronises between servers, so DDL sent through a fan-out Service is applied +to whichever server answered that connection. + +**How it presents.** The post-install migration scatters its statements across +independent servers that then disagree about which tables exist, and fails part +way through with ClickHouse's `UNKNOWN_TABLE` surfaced as `HTTP Error 404` — +i.e. the server that answered did not have the table another server had just +been given. Airbyte writes and Analytics reads scatter the same way afterwards. +The per-pod name is used **even at one replica** so that raising the replica +count cannot silently reintroduce round-robin DDL; see the comment on the key +in [`values.yaml`](values.yaml). + +### 3. `redis.host` is a replication primary, not a Redis Cluster endpoint + +**The setting.** `redis-master.redis.svc.cluster.local`, backed by a +three-member `redis-replication` release. Redis **Cluster** is the wrong +topology here, and Sentinel is deliberately not deployed either. + +**What breaks without it.** The authenticator's Rust Redis client is compiled +with only `tokio-comp` and `connection-manager`; the crate feature-gates cluster +support behind `cluster`, so `ClusterClient` is not in the binary and it opens a +plain standalone connection. A clustered Redis answers `MOVED` for any key whose +slot the contacted node does not own, and that client cannot follow the +redirect. + +**How it presents.** `/auth/login` returns HTTP 500 for every visitor, while +every pod stays Ready and the release reports `deployed`. Sessions are not +optional, so the symptom is "nobody can log in" with nothing in the deploy +output to suggest why. `-master` selects on the operator's `redis-role=master` +label, so it follows a failover with no client change — which is the reason the +Service name and not a pod name is the right address. + +### 4. Argo `singleNamespace: false` **and** a matching `instanceID` + +**The setting.** Two values that only work as a pair. +`singleNamespace: false` in the Argo release, and +`controller.instanceID.explicitID` equal to the umbrella's +`ingestion.reconcile.argoInstanceId` — both `argo-workflows-argo`, i.e. +`-`. + +**What breaks without either.** With `singleNamespace: true` the chart passes +`--namespaced` and the controller watches only the `argo` namespace, while every +CronWorkflow and WorkflowTemplate the umbrella installs lives in `insight`. And +the umbrella stamps each of those objects with +`workflows.argoproj.io/controller-instanceid: `; Argo's matching rule is +exact, and a controller with no instanceID processes **only** objects carrying +no such label — so a labelled CronWorkflow is invisible to it. + +**How it presents.** Identically, and silently, for both causes: the +CronWorkflows exist, are unsuspended, carry a valid schedule, and their +`status` stays `{}` forever with zero `Workflow` objects ever created. It reads +as "nothing has been triggered yet". The controller logs `instanceID=""` at +startup and then never mentions the objects again. + +### 5. Argo `controller.workflowNamespaces` stays `[argo]` — do not add `insight` + +**The setting.** A list that looks like it should name the namespace the +workflows run in, and must not. + +**What breaks if you extend it.** That list controls only where the *chart* +creates the workflow ServiceAccount and its Role/RoleBinding. `insight` already +has an `argo-workflow` SA + Role + RoleBinding applied with `kubectl` (see +[Bootstrap §5](#5-the-argo-workflow-serviceaccount-in-insight)), and Helm cannot +adopt objects it did not create. + +**How it presents.** The Argo upgrade fails outright with `invalid ownership +metadata`. What actually lets the controller *watch* other namespaces is the +ClusterRole/ClusterRoleBinding that `singleNamespace: false` renders — not this +list. + +### 6. `airbyte.namespace: airbyte` + +**The setting.** A chart value that does two things, only one of which is the +API URL. + +**What breaks without it.** It also decides which namespace the chart renders +`Role`/`RoleBinding` `insight-airbyte-auth-reader` into — the RBAC that lets the +ingestion reconcile loop read Airbyte's own `airbyte-auth-secrets`. Left empty +it defaults to the **release** namespace. + +**How it presents.** The RBAC lands in `insight`, where that Secret does not +exist, and connector provisioning fails at run time while every pod looks +healthy and the deploy reports success. Verify with +`kubectl -n airbyte get role insight-airbyte-auth-reader` — in the *airbyte* +namespace, not in `insight`. + +### 7. Airbyte `global.auth.enabled: true` + +**The setting.** An Airbyte chart value, and it is not about protecting the UI. + +**What breaks without it.** `charts/server` only sets +`API_AUTHORIZATION_ENABLED` when the edition is pro/enterprise **or** +(community **and** `global.auth.enabled`). Without that variable the server +never mounts the authorization endpoints, so `POST /api/v1/applications/token` +— the token mint the ingestion reconcile loop uses — answers 404. + +**How it presents.** `reconcile` logs `applications/token failed: … 404 / +Object not found` for every connector and exits non-zero, while +`/api/v1/health` returns 200 and the unauthenticated public API at `/v1/*` +still answers — so the server looks healthy and only half of it is. The +consequence of `true` is the one-time instance setup in +[Bootstrap §7](#7-the-airbyte-instance-setup-call). + +### 8. `mariadb.host` is the operator's `-primary` Service + +**The setting.** `mariadb-primary.mariadb.svc.cluster.local`, not the +round-robin `mariadb` Service that exists beside it. + +**What breaks without it.** MariaDB runs as Galera, which certifies every write +cluster-wide. Concurrent writers arriving on different members through a +round-robin Service produce certification conflicts. + +**How it presents.** The Analytics `migrate` Job — many statements, one +transaction each, no retry — fails with `1213` deadlocks, and the upgrade fails +with it. `-primary` is kept pointed at a single member by the operator and moves +on failover, so it gives a single writer without pinning a pod name. The same +applies to `keycloak.database.host`, which must be the same address for the same +reason. + +### 9. `redpanda.brokers` is one `host:PORT` string, and the port is 9093 + +**The setting.** A single comma-separated bootstrap string — +`redpanda.redpanda.svc.cluster.local:9093` — not the host/port pair every other +datastore in `values.yaml` takes. 9093 is the redpanda chart's *internal* Kafka +listener; there is no 9092 listener on the Service at all. + +**What breaks without the port.** The chart hands the value to a Kafka client +verbatim, so a bare hostname silently becomes `host:9092`. + +**How it presents.** It never connects, and every pod stays healthy. The +deployment repository's preflight parses this value and refuses any entry +without a colon-port precisely because nothing downstream will tell you. Confirm +the listener on a stand with +`kubectl -n redpanda get svc redpanda -o jsonpath='{range .spec.ports[*]}{.name}={.port}{"\n"}{end}'`. + +### 10. `authenticator.tlsDiscovery.enabled: true` with a Ready `ClusterIssuer` + +**The setting.** `issuerRef: {name: insight-ca, kind: ClusterIssuer}`. This is +not optional hardening. + +**What breaks without it.** The chart hardcodes +`gateway_issuer: https://insight-authenticator.insight.svc.cluster.local:8443` +in its secrets template and mounts `insight-authenticator-authn-tls-cert` as a +**non-optional** Secret volume in both Analytics and Identity Resolution. +Turning `tlsDiscovery` off does not remove the dependency; it removes the thing +that satisfies it. An issuer that exists but cannot sign has the same effect. + +**How it presents.** `helm --wait` does not wait on `Certificate` objects, so +Analytics and Identity Resolution sit in `ContainerCreating` until the entire +20-minute timeout expires, and the failure message is a timeout rather than a +missing certificate. Check the issuer's `Ready` condition, not just its +existence. + +### 11. `authenticator.oidc.externalIdClaim: idp_sub` — **not** `sub` + +**The setting.** The claim the login bootstrap reads as the external id. On a +seeded stand it is `idp_sub`, a custom claim, even though the realm generator +sets each realm user's `id` to that person's roster UUID and its own comments +say `sub` carries it. + +**What breaks with `sub`.** That would hold for a Keycloak *realm import*. The +chart applies realms with keycloak-config-cli, which creates users one at a +time through the admin REST API — and Keycloak assigns its own id on +`POST /users`, silently discarding the document's. So `sub` is not the roster +UUID. The deployment repository post-processes the generated realm to copy the +roster UUID into an `idp_sub` user attribute and adds an +`oidc-usermodel-attribute-mapper` on every client that emits it. + +**How it presents.** Late, and as a data problem. Every user is created, the +password login succeeds, and the callback answers `login denied: no matching +person in Identity` with `event="login_denied_unknown_person"` against a fully +populated `identity.persons`. + +### 12. The generated realm's declarative user profile — `unmanagedAttributePolicy: ENABLED` + +**The setting.** A property of the realm document the deployment repository +generates and packs, not of anything in this directory. + +**What breaks without it.** Keycloak 26 always runs the declarative user +profile, and its default policy **discards** every attribute the profile does +not declare. The realm sets three per user (`tenant_id`, `org_unit`, +`idp_sub`). + +**How it presents.** None of the three survives import: `USER_ATTRIBUTE` comes +back empty for every user while the protocol mappers are created correctly, so +the mappers emit nothing and login fails at the callback with `id_token carries +no non-empty idp_sub claim`. Nothing warns, and the realm looks right in the +admin console — the attributes were never rejected, only dropped. + +### 13. `keycloakConfig.filesLocations: "/config/*.json"` + +**The setting.** In [`values.yaml`](values.yaml). The chart's default pattern is +`"/config/*.yaml"` only. + +**What breaks without it.** The generated roster realm is JSON, and in seeded +mode the broker realm's YAML is not packed at all — so the default pattern +matches zero files, and keycloak-config-cli **errors** on a glob that matches +nothing rather than treating it as a no-op. + +**How it presents.** The post-install/post-upgrade hook Job fails, which fails +the whole upgrade. Loud and correctly attributed, which is why it is this far +down the list — but it costs a deploy to find out if the ConfigMap's one key is +ever renamed. + +### 14. `keycloak.hostname` must end in `/kc` + +**The setting.** The advertised issuer base, in [`values.yaml`](values.yaml). + +**What breaks without the suffix.** Keycloak 26's hostname-v2 does not fold +`--http-relative-path` into the advertised issuer, so the subchart passes both +and the path has to be part of the hostname. + +**How it presents.** The server advertises an issuer the browser cannot reach. +Discovery fails in a way that reads like a TLS or routing problem. The matching +in-cluster value is `keycloakConfig.url: http://insight-keycloak:8085/kc` with +`allowInsecureUrl: true` — required for a non-https URL and acceptable only +because that hop never leaves the cluster. + +### 15. `authenticator.oidc.sourceType` must equal what the seeder writes + +**The setting.** `keycloak` on a seeded stand. The seeder writes +`(insight_source_type='keycloak', value_type='id', value_id=)`, and +`seed-stand.sh` reads the source type back out of `insight-authenticator-config` +rather than taking it as a flag — so there is one writer, not two copies. + +**What breaks if they disagree.** Nothing, visibly. Both halves succeed. + +**How it presents.** A login that authenticates and then resolves to nobody, +against a projection that is fully seeded. Identical symptom to §11, different +cause — which is why both are worth knowing separately. + +### 16. `frontend.route.enabled: false`, and the creation-timestamp tie-break + +**The setting.** The gateway owns the only `/` route and proxies to the frontend +Service via its `frontUrl`. The frontend must not publish itself. + +**What breaks with a second route.** Two HTTPRoutes claiming the same path on +the same host are resolved by Gateway API using **creation timestamp**, oldest +wins — and both report `Accepted`. + +**How it presents.** Depending on deploy order, the edge either bypasses the +auth gateway entirely or the frontend route does nothing, with no error +anywhere. The same rule makes the deployment repository's optional maintenance +page a trap: it claims `PathPrefix: /` on the same hostname with no `hostnames:` +of its own, so being older it keeps serving after Insight is deployed. Removing +it is a required step, not a cleanup. + +### 17. Application routes attach with `sectionName: https` + +**The setting.** `gateway.route.parentRef` and `keycloak.route.parentRef` name +`{name: insight, namespace: envoy-gateway-system, sectionName: https}` — the +**named** listener. + +**What breaks with the wrong section.** The Gateway also carries a +hostname-less HTTPS listener on 443 reserved for bare-address and unknown-Host +visitors. A route can narrow a listener's hostname but never widen it, which is +why the maintenance route names no hostname and every application route does. +Attaching an application route to the catch-all mixes the two populations. + +**How it presents.** As a routing oddity rather than an error — both listeners +are legal on one port because an absent hostname means `*` and envoy picks by +SNI. The deployment repository's preflight additionally asserts that +`gateway.route.parentRef` names *this* Gateway, because a values file +re-pointed at the chart's upstream default would render routes no Gateway here +ever serves, with everything reporting healthy. + +### 18. `insight-db-creds` must exist **without** Helm's ownership label + +**The setting.** A plain Secret in `insight`, created outside any release, with +keys `clickhouse-password`, `mariadb-password`, `mariadb-root-password`, +`redis-password`. + +**What breaks otherwise.** The chart detects "bring your own" by the *absence* +of `app.kubernetes.io/managed-by=Helm` and then skips emitting its own copy; +`credentials.autoGenerate: true` composes the per-service config Secrets from +it. + +**How it presents.** Anything that has claimed the name for Helm makes the +install abort with `invalid ownership metadata`. Note the dry-run artefact: +Helm skips `lookup` on a dry run, so rendered output shows the chart emitting +its own copy — a real install finds the pre-created one. Do not "fix" that. + +### 19. A DSN charset constraint on all four datastore passwords + +**The setting.** The chart composes every DSN by string interpolation, and its +helpers **reject** any of `@ : / ? # %` in *every* key of `insight-db-creds`, +including `mariadb-root-password`, failing the install rather than warning. + +**What breaks without the discipline.** The deployment repository's generators +emit hex for exactly this reason, and the MariaDB root password is +repository-owned (`generate: false`) rather than operator-generated because the +operator's mixed-character value fails often enough to matter. + +**How it presents.** A hard install failure naming the key — correctly +attributed, but only after a datastore has already been provisioned with the +offending value. One caveat is recorded in the manifest and worth repeating: +`generate: false` only decides the **bootstrap** password. On a cluster that has +already run, editing the Secret does not change the live server — rotate with +`ALTER USER` on both `root@%` and `root@localhost` in the same breath, or the +operator loses its own SQL access. + +### 20. MariaDB `log_bin_trust_function_creators=1` + +**The setting.** In the `MariaDB` CR's `myCnf`. + +**What breaks without it.** The Insight analytics migration creates stored +functions as the unprivileged `insight` user, and MariaDB refuses with +`1419 (HY000): You do not have the SUPER privilege and binary logging is +enabled`. Galera raises this even with binary logging off, because wsrep +replicates DDL by statement. + +**How it presents.** The Analytics Deployment sits in `Init:CrashLoopBackOff` +until `helm --wait` gives up with "Progress deadline exceeded". The alternative +— granting `SUPER` to the application user — is worse on every axis. + +### 21. The ClickHouse `insight` user carries a network allow-list + +**The setting.** `insight/networks/ip` on the `ClickHouseInstallation` — a +single private-range block, written in the deployment repository's +`clickhouse/clickhouse.yaml`. The literal is not repeated here; read it from +that manifest, or from `kubectl -n clickhouse get chi clickhouse -o yaml`. + +**What breaks without a matching pod network.** The allow-list silently assumes +the cluster's pod CIDR falls inside that one block. A cluster built with a pod +network outside it authenticates nothing from any pod. + +**How it presents.** ClickHouse itself is perfectly healthy and every +application connection is refused. Nothing in the manifest checks the +assumption — a cluster template whose pod CIDRs happen to fall inside the range +makes it invisible. Compare the allow-list against +`kubectl get nodes -o jsonpath='{range .items[*]}{.spec.podCIDR}{"\n"}{end}'` +before reusing the manifest on a differently-provisioned cluster. + +### 22. `cinder` must be the cluster **default** StorageClass + +**The setting.** The `storageclass.kubernetes.io/is-default-class: "true"` +annotation, not just the class's existence. + +**What breaks without it.** Every other datastore names `storageClassName: +cinder` explicitly, but the Airbyte chart's bundled PostgreSQL and MinIO take +the **default**. + +**How it presents.** The Airbyte deploy script reads the annotation and +hard-fails unless the default is exactly `cinder` — so this one is caught. Left +unchecked it would surface as PVCs that never bind, long after the rest of the +stack is up. + +### 23. Non-default security contexts on three workloads + +**The setting.** MariaDB runs with `runAsNonRoot: false` / `runAsUser: 0` / +`fsGroup: 999` / `fsGroupChangePolicy: Always`; Redis runs with +`runAsUser`/`runAsGroup`/`fsGroup` all 0; ClickHouse and Keeper set +`fsGroup: 101`; and Airbyte's bundled PostgreSQL and MinIO PVCs are +**pre-created** and ownership-initialised by two throwaway Jobs before the chart +is installed. + +**What breaks without them.** This Cinder deployment does not apply `fsGroup` +ownership to new volumes, so each workload's entrypoint has to chown its own +data directory before dropping privileges. + +**How it presents.** Permission-denied crash loops on first start of a +freshly-provisioned volume. These look like sloppiness if copied without the +reason, which is exactly why the reason is written here — they are not +candidates for tightening without first fixing the volume-ownership behaviour. + +### 24. The edge Service: pinned address plus `keep-floatingip`, and `externalTrafficPolicy: Cluster` + +**The setting.** `EnvoyProxy/insight` sets a pinned `loadBalancerIP` **and** the +annotation `loadbalancer.openstack.org/keep-floatingip: "true"`, with +`externalTrafficPolicy: Cluster`. + +**What breaks without each.** Without `keep-floatingip`, a teardown releases the +address back to the pool and the DNS record dangles — that annotation is the +entire reason a rebuild keeps the same public name. With an address pinned that +is still attached to something else, the cloud controller refuses to steal it. +And `externalTrafficPolicy: Local` — the Envoy Gateway default — only routes +through nodes actually running an envoy pod, depending on the load balancer's +health monitors to notice which those are. + +**How it presents.** A contended pinned address means the Gateway never gets an +address and the deploy times out after ten minutes rather than failing fast. +`Local` costs reachability and buys nothing here: client IPs are lost either +way, because Cloudflare fronts the origin and the real client is in +`CF-Connecting-IP`, not the TCP peer. **For a brand-new stand the correct +setting is `auto`**, which drops the `loadBalancerIP` line entirely and lets the +cloud controller allocate; the deploy script then prints the allocated address +to pin. + +### 25. After every upgrade, Analytics, Authenticator and Identity Resolution must be restarted + +**The setting.** Not a value — a step. See +[`README.md` step 4](README.md#deploying-by-hand). + +**What breaks without it.** The chart injects `insight--config` with +`envFrom.secretRef`, and environment variables are read once at container start. +Each subchart's `checksum/config` annotation covers its **gears ConfigMap**, not +that Secret. + +**How it presents.** `helm upgrade` rewrites datastore hosts and passwords, the +pod spec is identical so nothing restarts, and the running pods keep serving the +old configuration. Moving Redis from one endpoint shape to another is the +canonical case: the release reported `deployed`, every pod stayed Ready, and the +authenticator went on dialling a Service that no longer existed. Note a +disagreement worth resolving: the deployment repository rolls two Deployments +(authenticator, analytics) while this repository's README rolls three — +`identity-resolution` also consumes its config Secret with `envFrom` and is +subject to the identical staleness, so three is the correct list. `gateway` and +`frontend` are excluded deliberately: neither reads that Secret, and restarting +them drops live traffic for nothing. + +### 26. `helm --wait` does not wait on HTTPRoute status + +**The setting.** Not a value — the post-condition the deploy has to add itself. + +**What breaks without the check.** A route the Gateway rejects leaves the +release `deployed` and the site dark. Both conditions matter: `Accepted` alone +still 503s when the `backendRef` names a Service that does not exist. + +**How it presents.** A green deploy and an unreachable stand. This is why the +deploy step in [`README.md`](README.md) and the workflow both check `Accepted` +after the upgrade even though the release now owns the routes — the check's +meaning changed from "somebody else's object survived" to "the object this +upgrade just wrote was accepted", and it earns its place either way. + +### 27. Helm 4 apply-mode coupling + +**The setting.** `--server-side=true --force-conflicts`, named explicitly rather +than inherited. Helm 3 takes neither flag. + +**What breaks without naming it.** Helm 4.0/4.1 defaulted to server-side apply, +so `--force-conflicts` alone sufficed. Helm 4.2+ made `--server-side` default to +`auto` — meaning "whatever the previous release used" — so a release whose last +revision went client-side resolves to client-side. + +**How it presents.** `invalid client update option(s): forceConflicts enabled +when serverSideApply disabled`, inherited rather than chosen: a stand starts +failing because of how somebody ran the *previous* upgrade. + +### 28. Sizing values that must move together + +**The setting.** Three couplings, each of which is silent when broken. +ClickHouse `replicasCount` and Keeper `replicasCount` must be raised in lockstep +with the replica counts the deploy and validate scripts size their waits from — +and ClickHouse must not be scaled at all until the chart issues cluster-wide DDL +(§2). `MARIADB_BUFFER_POOL` must move **with** the MariaDB memory request, +because InnoDB never uses a byte more than the buffer pool, so raising container +memory alone reserves capacity the database cannot touch. And the anti-affinity +asymmetry described under [Component inventory](#component-inventory) — hard for +Redpanda and Galera, soft for Redis — is a decision, not an inconsistency. + +**How it presents.** A raised replica count that the deploy script never waits +for, a database that ignores the memory it was given, or a StatefulSet that +never converges with no event explaining why. + +### 29. Template rendering discipline + +**The setting.** Several manifests in the deployment repository are +**templates** carrying `__INSIGHT_HOST__` / `__ENVOY_GATEWAY_LB_IP__` / +`__CLICKHOUSE_*__` / `__KEEPER_*__` / `__MARIADB_*__` placeholders. They must +never be `kubectl apply -f`'d directly, only through their deploy script. + +**What breaks otherwise.** A literal `__FOO__` is valid YAML. + +**How it presents.** It reaches the API server as a hostname or a quantity and +fails at admission with a message naming neither the stand nor the file. The +repository asserts in both directions — it fails a deploy on any leftover +placeholder, and it fails if a literal value creeps back **into** a template, +which would silently re-pin every stand to one host or address. + +## Bootstrap steps that are not a `helm install` + +These are the steps that create state the release consumes by name. They are +invisible in `helm get manifest`, so anyone reconstructing the stand from the +rendered release alone will miss all of them. + +### 1. Databases and grants — created by two different layers + +The deployment repository creates only the MariaDB `insight` +Database + User + Grant (operator CRs) and the ClickHouse `default` and +`insight` users. **The rest are created in-server by the umbrella's own +pre-install/pre-upgrade hook Jobs:** + +* `insight-mariadb-init-svcdbs` (hook-weight 5) connects as MariaDB **root**, + using `insight-db-creds`/`mariadb-root-password`, and runs + `CREATE DATABASE IF NOT EXISTS` for `identity` and `keycloak` (utf8mb4 / + utf8mb4_unicode_ci) plus `GRANT ALL` to `insight@%`. +* `insight-clickhouse-init-svcdbs` (hook-weight 5) creates the ClickHouse + databases `insight` and `presentation` as the `insight` user over the HTTP + API. + +That is **why the root password has to be in `insight-db-creds`** even though no +DSN interpolates it, and why `identity` and `keycloak` do not appear as MariaDB +`Database` CRs. The other two hooks are +`insight-keycloak-config` (post-install/post-upgrade, weight 100 — the +keycloak-config-cli Job, drift-reverting, so admin-console edits do not survive +a deploy) and `insight-clickhouse-migrate` (weight 200 — the ClickHouse schema +migration, whose failure fails the whole upgrade). Every hook Job carries +`ttlSecondsAfterFinished: 600`, so Kubernetes deletes it and its logs ten +minutes after it finishes: **capture hook logs immediately on a failure.** + +### 2. The Secrets the release expects to already exist + +In `insight`, all created outside the release: + +| Secret | Keys | Semantics | +|---|---|---| +| `insight-db-creds` | `clickhouse-password`, `mariadb-password`, `mariadb-root-password`, `redis-password` | **Recomposed on every run** by reading the four operator-owned Secrets below. Applied without a Helm label (§18). | +| `insight-authenticator-signing-keys` | `current.pem` | **Generate once, reuse forever.** See §3 below. | +| `insight-keycloak-admin` | `username`, `password` | Bootstrap admin for the bundled Keycloak. Generated once; regenerating strands the server's stored admin user. | +| `insight-oidc` | `client-secret` | The confidential client secret. Generated once; the keycloak-config Job pushes the same value into the realm's client on every deploy so the two cannot drift. Rotation is "delete the Secret, re-run the deploy". | +| `insight-keycloak-config` | config-cli login + client secret | **Recomposed on every run** from the two above. On a seeded stand it carries nothing else — the external-IdP OAuth passthrough values exist only in the GitHub login mode. | + +Required in **other** namespaces before the Insight step will run: +`clickhouse/clickhouse-insight-credentials`, `mariadb/mariadb-insight-credentials`, +`mariadb/mariadb-root`, `redis/redis-auth`. Separately, +`airbyte/airbyte-auth-secrets` is created by Airbyte on first boot and is warned +about rather than failed on — the reconcile loop cannot authenticate without it. + +A seeded stand needs **no GitHub OAuth App**; requiring one would be asking for +a registration nothing ever reads. + +### 3. The ES256 signing key + +```text +openssl ecparam -name prime256v1 -genkey -noout | openssl pkcs8 -topk8 -nocrypt +``` + +under `umask 077`, stored as key `current.pem`. The chart mounts it +non-optionally and never generates it. Generate-once semantics here are +absolute: a new key silently invalidates every issued gateway JWT and every +token the gateway has cached, so "re-run the bootstrap to be safe" logs +everybody out. + +### 4. The realm ConfigMap pack + +`-keycloak-config-realms` in `insight`, holding **exactly one key** — +`realm-insight.json` on a seeded stand. It is written with +`create configmap --dry-run=client | apply`, which rewrites the whole data map, +so switching login modes *prunes* the other mode's realm file rather than +leaving two for the config Job's glob to find. The key name is load-bearing +because `keycloakConfig.filesLocations` globs by extension (§13). + +On a seeded stand the realm is **generated, not checked in**, from the seeder's +own module, so the realm and `identity.persons` stay two projections of one +roster. Two guards run before generation: the tenant must be non-empty (a realm +minting users with no tenant claim would be invisible to every login), and the +generator's password constant must match the copy the validator signs in with, +or the deploy refuses. The generated document is then post-processed to add the +`idp_sub` attribute and mapper (§11) and the declarative user profile (§12). + +### 5. The `argo-workflow` ServiceAccount in `insight` + +A ServiceAccount, a Role granting `create` and `patch` on +`workflowtaskresults.argoproj.io`, and a RoleBinding — applied with `kubectl`, +**outside** the Helm release so `helm uninstall` cannot take the account away +from workflows that are still queued. + +The umbrella pins every WorkflowTemplate and CronWorkflow it ships to that +account and lists it as a subject of its own `insight-airbyte-auth-reader` +RoleBinding, but does not create it; the Argo release's copy lives in the `argo` +namespace, and a workflow pod runs in the namespace of its Workflow. Without it +every dbt transform and data-quality check fails with +`serviceaccount "argo-workflow" not found` while all app pods stay healthy — +nothing surfaces until a scheduled run. The `workflowtaskresults` Role is +separately load-bearing: Argo 3.4+ has each step report its outcome through the +pod's own ServiceAccount, so a step without those two verbs dies with exit 64 +and `workflowtaskresults.argoproj.io is forbidden` before the user container +starts. + +### 6. The Gateway and the ClusterIssuer chain + +Applied as manifests, not as chart values: + +* `GatewayClass/envoy` (controller + `gateway.envoyproxy.io/gatewayclass-controller`), `Gateway/insight` in + `envoy-gateway-system` with `infrastructure.parametersRef` → + `EnvoyProxy/insight` and the annotation + `cert-manager.io/cluster-issuer: insight-ca`, plus + `HTTPRoute/redirect-http-to-https` issuing a 301 from the `http` listener. +* A **two-step** self-signed chain, not one issuer: + `ClusterIssuer/insight-selfsigned` (bootstrap only) signs + `Certificate/insight-ca` in the `cert-manager` namespace (isCA, ECDSA-256, + long-lived, secret `insight-ca-key-pair`), and `ClusterIssuer/insight-ca` + built on that key pair is the one the umbrella names. A bare selfSigned + issuer would also publish a `ca.crt`, but every leaf would be its own + unrelated root, so downstream trust could not be pinned to one CA. The + Certificate's namespace is fixed: a `ca` ClusterIssuer only ever reads its key + pair from the cert-manager release namespace. + +The listener's TLS Secret `insight-origin-tls` is **minted, never created by +hand** — cert-manager's gateway-shim reads the Gateway annotation and mints a +Certificate per HTTPS listener hostname. The chain is self-signed on purpose: +Cloudflare terminates the public TLS and its "Full" origin-pull accepts a +self-signed origin. **The zone must stay on Full** — "Full (strict)" would +reject the origin certificate, and "Flexible" would fetch over port 80 and turn +the http→https redirect into an infinite loop. + +### 7. The Airbyte instance setup call + +`POST /api/v1/instance_configuration/setup` with `initialSetupComplete: true`, +an email and an organisation name, authenticated by a token minted from +`POST /api/v1/applications/token` using the instance-admin credentials in +`airbyte-auth-secrets`. It is the consequence of load-bearing setting §7: with +auth enabled a fresh instance boots into a setup wizard and stays +half-initialised until this call is made, and the chart does not make it. Run from a throwaway in-cluster +pod rather than a port-forward, so it exercises the same Service DNS the +ingestion workflows resolve. Idempotent by construction, so it runs on every +deploy rather than guessing. + +### 8. Outside the cluster entirely: the DNS record + +A public A record for the stand hostname pointing at the edge address, +Cloudflare-proxied with SSL mode Full. The Insight step is **gated** on that +record resolving and refuses to install without it, because the authenticator +resolves `issuerUrl` at startup *and* the browser is redirected to that +hostname — installing before the record exists produces a stand nobody can log +in to. + +### 9. Route adoption — migration only + +For a stand that was deployed **before** chart 0.5.107, the deploy labels and +annotates any pre-existing `insight-gateway` / `insight-keycloak` HTTPRoute for +Helm to adopt in place; without that the upgrade dies with `invalid ownership +metadata … missing key "app.kubernetes.io/managed-by"`. Idempotent, and a fresh +stand needs nothing. The chart's route names were chosen to match the old ones +exactly, which makes the adoption an update-in-place with no traffic blip and no +creation-timestamp tie (§16). + +## Between deploy and seed + +**A seeded stand is not usable between `deploy` and `seed`, and that is not a +fault.** Logins authenticate against Keycloak and then resolve against the +`identity.persons` rows the seeder writes, so an unseeded stand authenticates a +user and immediately denies them. Two consequences follow that get misread as +breakage: + +* A login test run before seeding fails on a login that authenticates and + resolves to nobody. That is true, and it is not a deploy failure — which is + why the deployment repository's deep validation deliberately does not run it + as part of the deploy. +* The `identity-resolution` **seed** CronJob is *expected* to fail on a fresh + stand. It guards an empty `identity_inputs` read and exits non-zero rather + than publishing an empty projection. A failing seed Job here is the guard + working. See the comment at the bottom of [`values.yaml`](values.yaml) for the + converse risk once that table stops being empty. + +Seeding itself is a thin wrapper around the application repository's own +`src/ingestion/tools/seed/seed-stand.sh`. It discovers everything from the +cluster — datastore hosts from `insight-platform`, tenant and identity database +from `insight-identity-resolution-config`, the seeder image from the chart's +`ingestion.seedImage`, the login source type from `insight-authenticator-config` +— so nothing is copied from the deployment repository and nothing can drift. + +## Stand-specific versus stand-shape + +The split below is **enforced, not merely documented**: several names are +pinned by hard assertions in the deploy scripts, so an override is a startup +error rather than a silent divergence. + +### What a second stand of this shape changes + +| Thing | Notes | +|---|---| +| The kubeconfig and the API server the scripts are allowed to act against | The guard is the server URL, not the context name, because a context name can repeat across kubeconfigs. Neither belongs in this repository. | +| The public hostname | One value; every manifest that names it carries a placeholder. Also the DNS record, the Gateway listener hostname, `gateway.route.host`, `keycloak.route.host`, `issuerUrl`, `redirectUri`, `csrfOrigins`. | +| The edge address | `auto` on a brand-new stand (§24). Never written here. | +| The login mode — `github` or `seeded` | Everything downstream follows from it: the realm name, `sourceType`, `externalIdClaim`, `filesLocations`, whether a GitHub OAuth App is needed at all, and whether seeding is mandatory. | +| The tenant UUID | Technically per-stand, but written in **four coupled places** that must be identical: `global.tenantDefaultId`, `ingestion.reconcile.tenantId`, `authenticator.oidc.defaultTenantId`, and the value handed to the realm generator (which reads it back out of the values file rather than repeating it). Rows written under a tenant the stand does not use are invisible to every login while deploy and seed both report success. `identityResolution.seed.tenantDefaultId` is deliberately **not** set — the composed config Secret already carries it, and setting it adds a fifth copy. | +| The resource profile | Placeholder-rendered sizing for the CR-based datastores; per-stand Helm overlays for the chart-based ones. | + +### What a second stand copies verbatim + +The deploy order. Every namespace name (`envoy-gateway-system`, `cert-manager`, +`clickhouse-operator`, `clickhouse`, `mariadb-operator`, `mariadb`, +`redis-operator`, `redis`, `redpanda`, `airbyte`, `argo`, `insight`). Every +release name (`eg`, `cert-manager`, `clickhouse-operator`, `mariadb-operator` +and `mariadb-operator-crds`, `redis-operator`, `redis`, `redpanda`, `airbyte`, +`argo-workflows`, `insight`). Every chart repository and pinned version. The +StorageClass name `cinder` and its default-class annotation. The issuer names +`insight-selfsigned` / `insight-ca` and the secret name `insight-ca-key-pair`. +The Gateway name `insight`, GatewayClass `envoy`, EnvoyProxy `insight`, TLS +Secret `insight-origin-tls`, and the listener names. Every in-cluster Service +DNS name in `values.yaml`. Every Secret and ConfigMap name. The ServiceAccount +name `argo-workflow`. And **every one of the load-bearing settings above.** + +## Recreating a stand of this shape + +Top to bottom. Steps 1–11 are the deployment repository's job; step 12 is where +this repository takes over. + +1. **Provision the cluster.** Kubernetes with a CNI, at least three schedulable + workers (hard anti-affinity, above), and the OpenStack Cinder CSI driver + registered as `csidriver/cinder.csi.openstack.org`. Verify with + `kubectl get csidriver` before going further — five later steps hard-fail on + its absence and one of them takes ten minutes to do so. +2. **Create the `cinder` StorageClass and make it the cluster default.** + Idempotently applied by five of the deploy scripts, each of which also + accepts a skip flag. +3. **Choose the per-stand values**: hostname, login mode, tenant UUID, resource + profile, and `auto` for the edge address on a brand-new stand. +4. **Deploy Envoy Gateway** (step 1 of the order). Its `https` listener will + finish **unprogrammed** — expected. Note the allocated edge address the + script prints, and pin it for subsequent runs. +5. **Create the public DNS record** pointing at that address, Cloudflare-proxied + with SSL mode **Full**. Do this now: the Insight step is gated on it + resolving, and DNS propagation is the one thing in this list that cannot be + hurried. +6. **Deploy cert-manager** (step 2), which creates the two-step issuer chain and + completes the Gateway's certificate. Confirm both ClusterIssuers report + `Ready=True` before continuing — the Insight preflight will refuse without it, + but finding out here is cheaper. +7. **Deploy the four datastores in order**: clickhouse, mariadb, redis, redpanda + (steps 3–6). Each deploy script waits past pod-Ready for the thing that + actually matters — ClickHouse until the `insight` user authenticates over the + pod network and `system.clusters` reports the expected topology; Redis until + exactly one member reports `role:master`, both replicas report + `master_link_status:up`, **and** the `redis-master` Service has a ready + endpoint. Do not shortcut those waits: a StatefulSet can report complete + while the Service the application dials has no endpoint at all. +8. **Confirm `maxUserConnections: 100` on the MariaDB `User/insight`** (§1) + before anything connects to it. This is the one item on the list that is + cheap now and expensive later. +9. **Deploy Airbyte** (step 7), including its pre-created PVCs, its + ownership-init Jobs and the one-time instance setup call + ([Bootstrap §7](#7-the-airbyte-instance-setup-call)). +10. **Deploy Argo Workflows** (step 8) with `crds.install=true` and + `crds.keep=true`. Verify `singleNamespace: false` and that the controller's + `instanceID` matches `ingestion.reconcile.argoInstanceId` (§4) — both are + silent when wrong. +11. **Run the Insight bootstrap that is not a helm install**: compose + `insight-db-creds` without a Helm label, generate the ES256 signing key, + generate the keycloak admin and OIDC client secrets, compose + `insight-keycloak-config`, generate and pack the realm ConfigMap, and apply + the `argo-workflow` ServiceAccount + Role + RoleBinding + ([Bootstrap §§2–5](#2-the-secrets-the-release-expects-to-already-exist)). + Remove the maintenance route if one was applied (§16). +12. **Hand over to this repository.** The umbrella upgrade is the `helm upgrade` + spelled out in [`README.md` step 3](README.md#deploying-by-hand) — the same + command CI runs — followed by the restart in step 4 (§25) and the route + check in step 5 (§26). Rehearse it read-only first with + [`../../scripts/emulate-ci-deploy.sh`](../../scripts/emulate-ci-deploy.sh), + which runs the workflow's own invocations and changes nothing by default. + Note that `make deploy ENV=test-stand` is **not** the deploy path for this + environment and would do damage; see + [Why not `make deploy`](README.md#why-not-make-deploy). +13. **Seed.** The stand is not usable until this runs + ([Between deploy and seed](#between-deploy-and-seed)). Capture the seeder's + manifest from the Job log before its TTL reaps it, and treat that JSON as + run-internal — it carries persona addresses, UUIDs and in-cluster URLs. + +Teardown, for completeness, is the exact reverse: insight, argo-workflows, +airbyte, redpanda, redis, mariadb, clickhouse, envoy-gateway, cert-manager. +Envoy Gateway must come **after** insight because the application HTTPRoutes +attach to its Gateway, and cert-manager must be **last** because both the +authenticator's Certificate and the Gateway's origin certificate must be gone +before their issuer is, and the gateway-shim crashes if the Gateway API CRDs +disappear while it is still running. CRDs and the shared StorageClass are +cluster-scoped and are never removed unless asked for by name. The edge address +survives by design (§24), which is what keeps DNS valid across a rebuild. + +## Known drift, and what not to copy + +* **The Gateway's listener set.** The reference manifest defines three listeners + — `https`, `https-default`, `http` — while a stand built before the + hostname-less catch-all was added carries only `https` and `http` and has + never been re-applied. A rebuild from the reference will therefore produce a + three-listener Gateway that differs from an older live one. That is an + expected difference, not a fault; the behaviour that changes is what a + bare-address or unknown-Host visitor gets (a 404 from envoy, versus the + catch-all listener's response). +* **Retired components in the reference repository.** A Patroni/PostgreSQL + triple, an operator-managed Keycloak triple that depended on it, a Dex + alternative, and an older seed mechanism all still exist as files and are + **not** deployed. None appears in the deploy order. They keep literal + hostnames rather than placeholders precisely so that parameterising them + cannot imply they are live. Do not copy any of them into a rebuild. +* **Reference-repository prose that predates the current shape.** Its README + still describes a PostgreSQL layer and a six-member Redis *Cluster*; the live + shape is no PostgreSQL at all and a three-member Redis *replication* group. + Trust the manifests and `helm list -A` over the prose, in that repository and + in this one. diff --git a/deploy/gitops/environments/test-stand/README.md b/deploy/gitops/environments/test-stand/README.md index 3017c7c10..5ad6f939d 100644 --- a/deploy/gitops/environments/test-stand/README.md +++ b/deploy/gitops/environments/test-stand/README.md @@ -21,17 +21,23 @@ that data?* ```text environments/test-stand/ ├── README.md # this file +├── INFRA.md # what has to exist BENEATH the umbrella, and in what order ├── inventory.yaml # cluster address + "this env manages the umbrella only" -├── values.yaml # the umbrella overlay — the file the deploy passes to helm -└── manifests/ - ├── httproute.yaml # public hostname -> insight-gateway (the chart renders none) - └── httproute-keycloak.yaml # /kc on the same hostname -> the bundled Keycloak +└── values.yaml # the umbrella overlay — the file the deploy passes to helm ``` There is deliberately no `sealed-secrets/` directory, no `keycloak/realms/` -directory and no `-values.yaml` here. Each absence is a decision, and +directory, no `-values.yaml` and — since the chart grew native Gateway API +templates — no `manifests/` directory either. Each absence is a decision, and each one is explained below. +[`INFRA.md`](INFRA.md) answers the question this file does not: *how do I get a +cluster to run any of this against?* It is the L0/L2 capture — the operators, +the datastores, the edge, the order they have to come up in, and the handful of +settings whose absence produces a release that reports `deployed` while the +stand silently does not work. Read it before building a second stand; read this +file to change the one that exists. + ## The ownership boundary This tree now owns the **stand's application configuration**. It does not own @@ -42,9 +48,9 @@ the cluster. | L0 | Cluster prereqs: cert-manager and its ClusterIssuer, Envoy Gateway and the `Gateway` object, the namespaces themselves | the deployment repository (outside this repo) | a human, deliberately | | L2 | Datastores — ClickHouse, MariaDB, Redis, Redpanda — each under its own operator in its own namespace; plus Airbyte and Argo Workflows | the deployment repository | a human, deliberately | | L2 | Generate-once Secrets: `insight-db-creds`, `insight-authenticator-signing-keys`, `insight-oidc`, `insight-keycloak-admin`, `insight-keycloak-config` | the deployment repository | a human, deliberately | -| L2 | The Keycloak realm content (the `insight-keycloak-config-realms` ConfigMap) | the deployment repository | a human, deliberately | +| L2 | The Keycloak realm content (the `insight-keycloak-config-realms` ConfigMap) — on this stand, the roster realm generated from the seeder's own organisation | the deployment repository | a human, deliberately | | **L3** | **The umbrella Helm release `insight` in namespace `insight` — every value in `values.yaml`** | **this directory** | **CI, on every merge to `main`** | -| L3 | The two `HTTPRoute`s in `manifests/` | source of truth here; applied by the deployment repository | a human, from the files here | +| L3 | The two `HTTPRoute`s, rendered by the release from `gateway.route` and `keycloak.route` in `values.yaml` | **this directory** | **CI, on every merge to `main`** | | L3 | The `argo-workflow` ServiceAccount + Role + RoleBinding that the chart's WorkflowTemplates pin but do not create | the deployment repository | a human, deliberately | Two consequences worth stating plainly: @@ -59,23 +65,33 @@ Two consequences worth stating plainly: ### What this env does not own, but depends on -Four objects live outside the Helm release and outside this directory. If any +Five things live outside the Helm release and outside this directory. If any of them disappears, the release still installs and reports `deployed`, and the stand is broken anyway: | Object | Symptom if missing | |---|---| -| `HTTPRoute/insight-gateway` | the public URL answers nothing; every smoke check fails at the first request | -| `HTTPRoute/insight-keycloak` | `/kc` is unreachable, so OIDC discovery fails and nobody can log in | +| `Gateway/insight` in the gateway controller's namespace — what both routes `parentRef` | the release renders its two `HTTPRoute`s and neither ever attaches: `Accepted` stays false because the parent resolves to nothing. The public URL answers nothing; every smoke check fails at the first request | +| `ClusterIssuer/insight-ca` — what `authenticator.tlsDiscovery.issuerRef` names | the authenticator's authn-TLS Certificate never issues, and the chart mounts the Secret it would have produced **non-optionally** into analytics and identity-resolution. Both sit in `ContainerCreating` until `--wait` gives up, and nothing in that failure names an issuer | +| `ConfigMap/insight-keycloak-config-realms` — the realm content | the chart's `keycloak-config` hook Job has nothing to apply and fails the whole upgrade. Worse if it is present but wrong: a realm that lost its `idp_sub` mapper authenticates every persona and then denies them, with a fully populated identity projection | | `ServiceAccount/argo-workflow` (+ its Role/RoleBinding) | every scheduled transform and data-quality run fails with `serviceaccount "argo-workflow" not found`, while all app pods stay healthy — nothing surfaces until a scheduled run | | The Secrets in the table above | services fail to start, or start with blank configuration | -The two routes are committed here (`manifests/`) because the acceptance -criteria travel through them. They are **verified, not applied**, by the -deploy — see the header comment in `manifests/httproute.yaml` for why re-applying -would create two writers on one object. The Argo RBAC is not copied here: it -is Argo plumbing rather than application configuration, and it sits outside the -deploy/seed/smoke scope this environment was created for. +The two routes used to be committed here, as manifests, because the chart +rendered none and every acceptance criterion travels through them. The umbrella +has since grown native Gateway API templates — `gateway.route` and +`keycloak.route` replaced the old `ingress` blocks — so the release renders both +`HTTPRoute`s and owns them, and the manifests are gone. The check that used to +guard them has not disappeared, it has inverted: `helm --wait` does not wait on +HTTPRoute *status*, so a route helm wrote successfully can still be refused by +the Gateway (a `parentRef` naming a Gateway that is not there, a `sectionName` +naming a listener that is not, a `backendRef` pointed at a Service a chart +upgrade renamed), and every one of those leaves the release `deployed` and the +stand dark. Step 5 below therefore asserts `Accepted` on what the upgrade has +just written — a post-condition of the deploy, not a check on somebody else's +object. The Argo RBAC is still not copied here: it is Argo plumbing rather than +application configuration, and it sits outside the deploy/seed/smoke scope this +environment was created for. ## Deploying by hand @@ -131,7 +147,7 @@ helm upgrade --install insight \ --namespace insight \ --values deploy/gitops/environments/test-stand/values.yaml \ --set-string authenticator.oidc.clientSecret="$(kubectl -n insight \ - get secret insight-oidc -o jsonpath='{.data.client-secret}' | base64 -d)" \ + get secret insight-oidc -o jsonpath='{.data.client-secret}' | base64 --decode)" \ --wait --timeout 10m --history-max 10 ``` @@ -151,6 +167,21 @@ Three things about that command are load-bearing: * **`--timeout 10m`**, not the Makefile's 30m default. A deploy that is going to fail should say so inside the CI budget. +Then ask the second question, which the upgrade's exit status does not answer: +*is the release the chart you meant?* A resumed run, a hand-deploy that raced +CI, or a mistyped version answers "did it succeed?" with yes and this with no. +CI runs the same check as its own step: + +```bash +helm list -n insight --deployed --failed --pending --uninstalling \ + --filter '^insight$' -o json | jq -r '.[] | "\(.status) \(.chart) rev=\(.revision)"' +``` + +Expect `deployed insight- rev=`. The status flags are +enumerated rather than passed as `--all`, which Helm 4 removed — on a runner or +laptop carrying v4, `--all` exits `unknown flag` and the check silently reports +every release as absent. + **4. Restart what the chart cannot know to restart.** Each subchart's `checksum/config` annotation hashes its own ConfigMap. It does **not** cover the umbrella-rendered `insight-*-config` Secrets, which the pods consume with @@ -170,22 +201,44 @@ All three are listed on purpose. Each of the three consumes its config Secret with `envFrom`, so all three are subject to the same staleness — a restart list of two would leave one service holding old configuration. -**5. Check the edge is still routing.** The chart renders no route, so a -successful upgrade tells you nothing about whether the stand is reachable: +**5. Check the edge accepted the routes the upgrade just wrote.** The release +renders both of them, but `helm --wait` does not wait on their status — so a +route the Gateway refuses leaves the upgrade green and the stand dark. This is +the command CI runs: ```bash kubectl -n insight get httproute insight-gateway insight-keycloak \ -o custom-columns='NAME:.metadata.name,ACCEPTED:.status.parents[0].conditions[?(@.type=="Accepted")].status' ``` -**6. Seed and smoke.** Seeding reuses the seeder verbatim — no test-stand -variant, no per-stand flags beyond the ones below (it discovers the tenant, -the datastore coordinates and the IdP source type from the cluster itself): +Both must read `True`. When one does not, read `ResolvedRefs` in the same +breath — `Accepted` says the Gateway took the attachment, `ResolvedRefs` says +its `backendRef` resolves, and a route that is accepted with unresolved refs +serves 503s rather than nothing: ```bash -src/ingestion/tools/seed/seed-stand.sh -n insight --email
--days 730 +kubectl -n insight get httproute insight-gateway insight-keycloak \ + -o 'jsonpath={range .items[*]}{.metadata.name}{"\t"}{range .status.parents[0].conditions[*]}{.type}{"="}{.status}{" "}{end}{"\n"}{end}' ``` +**6. Seed.** Seeding reuses the seeder verbatim — no test-stand variant, no +per-stand flags beyond the ones below (it discovers the tenant, the datastore +coordinates, the seed image and the IdP source type from the cluster itself): + +```bash +./src/ingestion/tools/seed/seed-stand.sh \ + -n insight \ + --context insight-test-stand \ + --email
\ + --days 730 +``` + +`--context` is passed even though the ambient kubeconfig already points there: +the script prints the context it resolved before it writes anything, and a run +whose target is stated rather than inherited is one a reader of the log can +check. `--days`, spelled exactly like that — the seeder rejects unknown +arguments, so a plausible-looking synonym fails every time. + The seeder's manifest — the list of personas, fixtures, the tenant and the data window that the smoke suite reads — is **printed to the seed Job's stdout** and written to a path inside a pod whose filesystem is discarded. @@ -193,6 +246,23 @@ Capture it from the Job log before the Job's TTL reaps it. Treat that JSON as run-internal: it carries persona addresses, UUIDs and in-cluster service URLs, so it must not become a CI artifact on a public repository. +**7. Smoke.** Driven through the public URL exactly as a browser would drive +it — real DNS, real TLS, real IdP redirect. `SMOKE_BASE_URL` is what aims the +suite; nothing else does: + +```bash +export SMOKE_BASE_URL=https://insight-test.cfabric.org +export SMOKE_LOGIN_MODE=password +export SMOKE_PERSONA_PASSWORD= + +uv run --project tests --frozen \ + pytest tests/stand/smoke -ra --stand-manifest +``` + +Nothing in the suite has a default: a missing value is reported by name before +a single request is made. See `tests/stand/smoke/README.md` for the full +variable table. + ### Rollback ```bash @@ -206,6 +276,65 @@ prerequisite for both). `rollback` is a human action by design: an automated rollback on failure is exactly what `--atomic` would have done, and what step 3 deliberately does not do. +## How login works here — the seeded realm + +The stand's IdP is the bundled Keycloak, published at `/kc` on the same public +hostname, and the realm it serves is `insight`: not the GitHub-federated +`insight-broker` realm the deployment repository's other login mode uses, but a +realm **generated from the demo seed's own organisation** — one local user per +person on the roster, each with a password credential. That single decision is +what makes an automated multi-persona login possible here at all, and it is why +`password` is the correct mode for the smoke suite on this stand. + +Four consequences, roughly in the order they bite: + +* **The stand is useless between deploy and seed.** Authentication and + authorisation are two systems here. Keycloak authenticates; the login + bootstrap then resolves the principal to a row in `identity.persons` by + `(source_type, external_id)` and **fails closed** when there is no match — + there is no email fallback, by design. So a freshly deployed, unseeded stand + accepts a correct password and *then* denies the person, which reads like a + broken login and is really an empty projection. Deploy → seed → smoke is a + sequence, not a convenience, and the identity-resolution seed CronJob failing + on a stand with no data yet is the documented healthy state rather than a + fault. +* **The external id is `idp_sub`, not `sub`.** The realm generator sets each + realm user's id to that person's roster UUID, which would be enough for a + realm *import* — but the chart applies realms with keycloak-config-cli, which + creates users one at a time through the admin REST API, where Keycloak assigns + its own id and silently discards the document's. The bring-up outside this + repository therefore copies the roster UUID into an `idp_sub` user attribute + and adds the mapper that emits it as a claim, and + `authenticator.oidc.externalIdClaim` names that claim. Point it at `sub` and + every login authenticates and is then denied, against a fully populated + identity projection — the most expensive way this stand can fail, because + nothing about it looks like a configuration error. [`INFRA.md`](INFRA.md#11-authenticatoroidcexternalidclaim-idp_sub--not-sub) + carries the full contract the generated realm has to satisfy, including the + declarative user profile that otherwise discards the attribute before the + mapper ever sees it. +* **`sourceType` and the seeder have to agree.** `authenticator.oidc.sourceType` + is `keycloak` here, and the seeder writes + `(source_type='keycloak', value_type='id', value_id=)`. + Neither value is repeated in CI: `seed-stand.sh` reads the source type back + out of `insight-authenticator-config` rather than being told. Changing one + without the other produces exactly the same authenticate-then-deny. +* **The persona password is a shared constant, not a per-stand secret.** Every + user the realm generator emits carries its `DEV_PASSWORD` constant + (`insight_seed.keycloak_realm`), so `SMOKE_PERSONA_PASSWORD` — and its CI + spelling `TEST_STAND_PERSONA_PASSWORD` — is one value derived from the + checkout rather than minted per stand. The per-persona + `SMOKE_PERSONA_PASSWORD__` override therefore cannot do anything on a + realm the seeder generated: all of its users share the one value. It is stored + as a GitHub environment secret so it stays masked in a public run log, but it + is not a secret in the sense the word usually carries. See Known gaps. + +For CI that means `vars.TEST_STAND_SMOKE_LOGIN_MODE` is `password`. The suite's +`override` mode — one principal authenticates and every persona session is +minted from it through `/auth/login?__override=` — still works and is +kept for a stand whose realm federates to an external provider and can serve no +password form at all. This stand is not that stand, and nothing in the gate +depends on `authenticator.overrideEnabled` any more. + ## Why not `make deploy` `make deploy ENV=test-stand` does not work against this stand, and would not @@ -269,14 +398,26 @@ commit would read the previous version). ## Known gaps -* **Scripted login.** The stand's realm federates login to an external OAuth - provider and has no local password users, so a username+password login - cannot be scripted against it as configured. Resolving that is a decision - about who owns the realm and what credential CI is allowed to hold — see the - `keycloakConfig` comment in `values.yaml` for the ownership half of it. +* **The persona password is a published constant.** Every user in the generated + realm carries the seeder's `DEV_PASSWORD`, so an internet-reachable stand + serves local accounts on a value anybody can read out of this repository. + Nothing in CI is blocked by it — the gate signs in as its personas with it, + which is the point — but it is not the "no well-known passwords on a public + stand" posture this environment was specified with. Closing it means teaching + the realm generator to take a password instead of embedding one, which is a + change to the seeder rather than to this directory. +* **Realm ownership still sits outside this repository.** The + `insight-keycloak-config-realms` ConfigMap is written by the bring-up, not by + this tree, and the deploy only reads it. That is deliberate — see the + `keycloakConfig` comment in `values.yaml` — but it does mean a login failure + can have a cause no file here can show you. * **`authenticator.overrideEnabled: true`** is carried forward from the installed release. It is a standing impersonation primitive on an - internet-reachable stand, gated on that flag alone. Tracked separately; - see the comment on the key in `values.yaml`. + internet-reachable stand, gated on that flag alone. The conditional that used + to hang over it has resolved in the good direction: the smoke suite reaches + every persona through a real password login and never sends `__override`, so + turning the flag off is now an unblocked change rather than one that would + take the gate with it. Tracked separately; see the comment on the key in + `values.yaml`. * **The identity CronJobs** can un-seed logins once their input table stops being empty. See the comment at the bottom of `values.yaml`. diff --git a/deploy/gitops/environments/test-stand/inventory.yaml b/deploy/gitops/environments/test-stand/inventory.yaml index 7d5a14fc5..24dcb8d76 100644 --- a/deploy/gitops/environments/test-stand/inventory.yaml +++ b/deploy/gitops/environments/test-stand/inventory.yaml @@ -22,10 +22,13 @@ ## (`make system-`) namespace, installed and ## owned outside this repo ## edge ingress-nginx + Ingress Envoy Gateway + Gateway -## objects rendered by the API HTTPRoute; the chart -## chart renders no HTTPRoute, so -## the route is a committed -## manifest (manifests/) +## objects rendered by the API HTTPRoutes, RENDERED +## chart BY THE CHART from +## `gateway.route` and +## `keycloak.route`, and +## attached to a shared +## Gateway this repo does +## not own ## secrets sealed-secrets controller + plain Secrets, pre-created ## `make seal` + committed once and composed from the ## SealedSecret manifests datastore namespaces by the @@ -37,7 +40,8 @@ ## nothing is sealed. This env is a values overlay plus a cluster address — ## the L0 (cluster prereqs) and L2 (datastores, Envoy Gateway, Airbyte, ## Argo) layers are brought up and owned elsewhere. See README.md in this -## directory for the ownership boundary and the exact deploy command. +## directory for the ownership boundary and the exact deploy command, and +## INFRA.md for what those layers must be, component by component. ## # ─── Cluster targeting ───────────────────────────────────────────────── @@ -177,8 +181,16 @@ system: # it breaks the token exchange until the # realm is updated to match. # insight-keycloak-admin bootstrap admin for the bundled Keycloak. -# insight-keycloak-config keycloak-config-cli login + the realm's -# external-IdP passthrough values. +# insight-keycloak-config the keycloak-config-cli login, plus the +# confidential client secret the realm +# document substitutes in. Nothing more on +# this stand: the realm is seeded from the +# demo organisation with LOCAL password +# users, so there is no external IdP and +# nothing to broker to. External-IdP +# passthrough values (an upstream +# provider's client id/secret) exist only +# on a federated stand shape. # # There is also no sealed-secrets CONTROLLER and no SealedSecret CRD on this # cluster, so the manifests would not reconcile even if they were committed. diff --git a/deploy/gitops/environments/test-stand/manifests/httproute-keycloak.yaml b/deploy/gitops/environments/test-stand/manifests/httproute-keycloak.yaml deleted file mode 100644 index e7a0b097b..000000000 --- a/deploy/gitops/environments/test-stand/manifests/httproute-keycloak.yaml +++ /dev/null @@ -1,49 +0,0 @@ -# /kc on the shared Gateway -> the chart's bundled Keycloak, for the test stand. -# -# The companion to httproute.yaml; read that file's header first — the same -# reasoning about why the route is hand-written, why it is committed, and why -# the deploy verifies rather than applies it holds here unchanged. -# -# WHY THE PREFIX WORKS WITHOUT A REWRITE -# Keycloak serves under /kc natively (the subchart passes -# --http-relative-path), and the same prefix is part of `keycloak.hostname` -# in ../values.yaml, which is what the discovery document advertises. So -# there is deliberately NO URLRewrite filter — adding one would strip a -# prefix the server expects and break discovery. -# -# WHY IT SHARES THE APPLICATION HOSTNAME -# Keycloak has no DNS record of its own. It rides the application's hostname -# and takes /kc from it, which is only safe because Gateway API gives the -# longest matching PathPrefix priority — the application route's "/" cannot -# swallow /kc. -# -# ORDERING NOTE for a recreate: Gateway API awards a contested path to the -# OLDEST route, so if a stale route ever claims /kc it must be deleted before -# this one is applied, not after. -# -# kubectl apply -f deploy/gitops/environments/test-stand/manifests/httproute-keycloak.yaml -# kubectl -n insight wait --for=condition=Accepted httproute/insight-keycloak -# -# Lives in the `insight` namespace so its backendRef stays namespace-local. -apiVersion: gateway.networking.k8s.io/v1 -kind: HTTPRoute -metadata: - name: insight-keycloak - namespace: insight -spec: - parentRefs: - - name: insight - namespace: envoy-gateway-system - sectionName: https - hostnames: - # Same hostname as the application route, on purpose — see above. - - insight-test.cfabric.org - rules: - - matches: - - path: - type: PathPrefix - value: /kc - backendRefs: - # The insight-keycloak subchart's Service, in this namespace. - - name: insight-keycloak - port: 8085 diff --git a/deploy/gitops/environments/test-stand/manifests/httproute.yaml b/deploy/gitops/environments/test-stand/manifests/httproute.yaml deleted file mode 100644 index b387d44b3..000000000 --- a/deploy/gitops/environments/test-stand/manifests/httproute.yaml +++ /dev/null @@ -1,63 +0,0 @@ -# Cluster edge -> the Insight gateway, for the test stand. -# -# WHY THIS FILE EXISTS AT ALL -# The umbrella chart renders no HTTPRoute template. It renders an Ingress, -# which this stand cannot use: the edge is Envoy Gateway, which serves -# Gateway API routes and ignores Ingress objects. So `gateway.ingress.enabled` -# is false in ../values.yaml and the route has to be a hand-written object. -# If a later chart release learns Gateway API, prefer its template and retire -# this file rather than keeping two sources for one route. -# -# WHY IT IS COMMITTED HERE RATHER THAN LEFT ONLY IN THE BRING-UP REPO -# Everything the acceptance gate proves — a person logs in through the public -# URL and sees data — travels through this object. A values file that -# configures a gateway nothing routes to is not a deployable description of -# the stand. Committing it makes the request path reviewable in one place. -# -# WHY THE DEPLOY DOES NOT APPLY IT -# The route already exists on the stand and is Accepted, and it is applied -# today by the L0/L2 bring-up outside this repository. Two writers on one -# object is how a manual fix and an automated deploy start reverting each -# other. So the deploy VERIFIES this route (exists, Accepted, ResolvedRefs) -# and fails the run if it does not — it does not re-apply it. This file is -# the reviewed source for the day the route has to be recreated: -# -# kubectl apply -f deploy/gitops/environments/test-stand/manifests/httproute.yaml -# kubectl -n insight wait --for=condition=Accepted httproute/insight-gateway -# -# See ../README.md, "What this env does not own", for the boundary. -# -# ROUTING NOTE -# The Insight gateway is the single edge proxy for the application: it runs -# the cookie-to-JWT exchange against the authenticator and fans out to /api/* -# and the SPA. So this route sends everything ("/") to it and path routing -# happens inside the gateway, not here. The one exception is /kc, claimed by -# httproute-keycloak.yaml on the same shared Gateway — longest PathPrefix -# wins by Gateway API spec, so Keycloak traffic never reaches this backend. -# -# Lives in the `insight` namespace so the backendRef stays namespace-local -# (and so a namespace-scoped CI credential could re-apply it if the team ever -# decides the deploy should own it). -apiVersion: gateway.networking.k8s.io/v1 -kind: HTTPRoute -metadata: - name: insight-gateway - namespace: insight -spec: - parentRefs: - - name: insight - namespace: envoy-gateway-system - sectionName: https - hostnames: - # Must match `authenticator.oidc.redirectUri`, `authenticator.csrfOrigins` - # and `keycloak.hostname` in ../values.yaml. A mismatch here does not fail - # the deploy — it fails login, after the release reports success. - - insight-test.cfabric.org - rules: - - matches: - - path: - type: PathPrefix - value: / - backendRefs: - - name: insight-gateway - port: 8080 diff --git a/deploy/gitops/environments/test-stand/values.yaml b/deploy/gitops/environments/test-stand/values.yaml index 8e1365658..99ea1bb20 100644 --- a/deploy/gitops/environments/test-stand/values.yaml +++ b/deploy/gitops/environments/test-stand/values.yaml @@ -98,8 +98,16 @@ credentials: # entire state lives in the MariaDB `keycloak` database — created and # granted by the chart's `mariadb-init-svcdbs` hook Job precisely because # `deploy: true` here — with realm content coming exclusively from the -# keycloakConfig Job below. It is published at /kc by -# manifests/httproute-keycloak.yaml in this directory, not by an Ingress. +# keycloakConfig Job below. +# +# It is published at /kc by the `keycloak.route` block below — still inside +# this `keycloak:` key, past the database wiring — which the umbrella renders +# as an HTTPRoute the RELEASE owns. Not an Ingress: the chart's `ingress` keys +# are gone, and one written here would be silently ignored. And no longer a +# manifest applied beside the release either — this directory used to carry +# manifests/httproute-keycloak.yaml and does not any more. See that block for +# why the hand-applied copy had to go rather than stay on as a belt-and-braces +# duplicate. keycloak: deploy: true # Advertised issuer base — the URL the browser AND the authenticator pod @@ -124,6 +132,37 @@ keycloak: username: "insight" passwordSecret: {name: "insight-db-creds", key: "mariadb-password"} + # ─────────────────────────────────────────────────────────────────────── + # keycloak.route — the /kc edge, and the ONLY thing that publishes it + # ─────────────────────────────────────────────────────────────────────── + # The umbrella renders this block as an HTTPRoute owned by the release, so + # `helm upgrade` creates it and patches it on every deploy. That is exactly + # why the hand-written manifests/httproute-keycloak.yaml this directory used + # to carry was DELETED rather than kept alongside as a second copy: two + # writers on one object means whoever ran last decides where /kc points, and + # neither writer reports the conflict. The old manifest's reasoning did not + # disappear with it — it is restated here and in the `route:` keys below. + # + # Keycloak rides the APPLICATION's hostname because it has no DNS record of + # its own. That is safe only because Gateway API gives the longest matching + # PathPrefix priority, so the gateway route's "/" cannot swallow /kc. + # + # Ordering note for a namespace rebuild: Gateway API awards a contested path + # to the OLDEST route. A stale route still claiming /kc has to be deleted + # BEFORE this one is rendered, not after — otherwise the upgrade succeeds, + # the HTTPRoute is `Accepted`, and requests keep going to the stale backend. + route: + # /kc on the SAME public hostname as the application. Keycloak serves + # under /kc natively (--http-relative-path), so there is no rewrite here + # and the prefix is part of the advertised issuer above — strip it and + # discovery breaks in a way that looks like a TLS problem. + enabled: true + host: insight-test.cfabric.org + parentRef: + name: insight + namespace: envoy-gateway-system + sectionName: https + # ═══════════════════════════════════════════════════════════════════════════ # keycloakConfig — realm as code (ADR-0003) # ═══════════════════════════════════════════════════════════════════════════ @@ -133,6 +172,12 @@ keycloak: # against the server above, with the import cache OFF — so the realm is # re-imposed on every deploy and admin-console edits do not survive. # +# Keep this banner anchored immediately above `keycloakConfig:`. It used to +# sit higher up, between `keycloak:` and `keycloak.route`, where the +# indentation invited every reader to attribute the whole essay to the route — +# and none of it is about the route. It documents the key directly below it +# and nothing else. +# # OWNERSHIP, and why this env does not ship a `keycloak/realms/` directory: # # That ConfigMap is created OUTSIDE the Helm release by the deployment @@ -145,10 +190,20 @@ keycloak: # ConfigMap name, with no placeholder rendering. Adding such a directory here # would make two writers fight over one object: a manual bring-up and a CI # deploy would each revert the other's realm, and whoever ran last would -# decide whether anyone can log in. Moving realm ownership into this tree is -# a deliberate follow-up (it needs a fully literal realm file and a decision -# about the external IdP's client credentials), not a side effect of adding -# this environment. +# decide whether anyone can log in. +# +# Note what a checked-in realm would have to be on THIS stand, which is not +# what that Makefile target was built for. The realm here is `insight` — the +# roster realm the seeder GENERATES at deploy time from the same roster module +# that writes `identity.persons`, so the two stay projections of one source. +# Checking a realm into this tree would mean a literal document restating +# every roster uuid and local password, maintained by hand against a generator +# that can change — and a realm and a person table that disagree produce a +# login that authenticates and then resolves to nobody. (It would NOT mean +# deciding what to do about an external IdP's client credentials: this stand +# federates to nothing. That question belongs to the federated `insight-broker` +# shape, not here.) Moving realm ownership into this tree is therefore a +# deliberate follow-up, not a side effect of adding this environment. keycloakConfig: enabled: true # The subchart's in-release Service over plain HTTP; the /kc relative path @@ -157,12 +212,45 @@ keycloakConfig: # — TLS terminates at the edge for the public name. url: "http://insight-keycloak:8085/kc" allowInsecureUrl: true - # Composed at bring-up from Secrets in this namespace: the config-cli - # login (KEYCLOAK_USER / KEYCLOAK_PASSWORD), the confidential client - # secret, and the external IdP's OAuth credentials that the realm - # federates login to. The Job injects the tenant id itself from - # `global.tenantDefaultId`. + # The Job takes this Secret by `envFrom`, so every key in it becomes an env + # var — both the config-cli login and whatever the realm's `$(env:VAR)` + # placeholders resolve against. `keycloakConfig.extraEnv` is left unset here, + # so this Secret is the only source of those, and its KEY NAMES are the + # contract rather than an incidental detail. + # + # It is recomposed at bring-up from two Secrets that already live in this + # namespace, and on a seeded stand it carries exactly two things: + # + # * the config-cli login, keyed KEYCLOAK_USER / KEYCLOAK_PASSWORD — the + # bootstrap admin from `insight-keycloak-admin`, which is who the Job + # authenticates to the server above as; and + # * the confidential client secret, from `insight-oidc`, which the Job + # pushes into the realm's `insight-authenticator` client on every deploy + # so the realm's copy and the authenticator's copy cannot drift. It is + # the same value the `--set-string` for `authenticator.oidc.clientSecret` + # below reads, out of that same `insight-oidc` Secret — one source of + # truth, not two copies to rotate. + # + # It does NOT carry the tenant id, and must not be made to. The Job sets + # INSIGHT_TENANT_ID in its own `env:` block from `global.tenantDefaultId`, so + # storing it here would add another copy of that UUID to the list already + # enumerated at `global.tenantDefaultId` — one that drifts silently because + # the Job's own value would keep winning. + # + # And it does not carry any external IdP's OAuth credentials. An earlier + # version of this comment said it did; that describes the OTHER stand shape — + # the federated `insight-broker` realm, where Keycloak brokers login to an + # upstream provider and needs that provider's client id and secret to do it. + # This stand runs the seeded `insight` realm with local password users and + # federates to nothing, so there is no upstream, no OAuth app, and nothing to + # register one with. Adapting this file for a federated stand means ADDING + # those keys here; their absence is not something that has gone missing. existingSecret: "insight-keycloak-config" + # The generated roster realm is JSON, and the chart's default pattern is + # "/config/*.yaml" only. config-cli errors on a glob that matches nothing, + # so this names exactly the pattern this stand's ConfigMap contains — in + # seeded mode the broker realm's YAML is not packed at all. + filesLocations: "/config/*.json" # ═══════════════════════════════════════════════════════════════════════════ # Global @@ -338,23 +426,41 @@ analytics: frontend: # the web UI (dashboard) replicaCount: 1 - ingress: + route: # Correct, and also the chart's own default: the UI is never published # directly. The gateway owns the only public route and proxies "/" to # this Service, so a second entry point here would bypass the auth edge # entirely. + # + # `route`, not `ingress`: the chart grew native Gateway API HTTPRoute + # templates, so the edge is expressed in values rather than in manifests + # applied beside the release. The `ingress` keys are gone; leaving one + # here would be silently ignored, which is the failure mode that keeps + # dead keys alive in values files for months. enabled: false gateway: replicaCount: 1 - ingress: - # OFF because the edge is Envoy Gateway, which serves Gateway API routes - # and not Ingress objects — and the umbrella renders no HTTPRoute - # template. Enabling this would render an Ingress that nothing serves. - # The route lives in manifests/httproute.yaml in this directory; the - # hostname pinned there must match `authenticator.oidc.redirectUri` - # below. - enabled: false + route: + # The public entry point, and now rendered BY THE CHART. This replaces the + # pair of HTTPRoute manifests this directory used to carry: the umbrella + # gained native Gateway API templates, so the release owns the routes, and + # a hand-applied copy beside it would be a second writer on one object. + # + # `host` must match `authenticator.oidc.redirectUri` and the `issuerUrl` + # below — the browser is redirected to all three, and a mismatch shows up + # as a login that loops rather than as a routing error. + enabled: true + host: insight-test.cfabric.org + parentRef: + # The shared Envoy Gateway, which lives in the controller's own + # namespace — a namespace this deploy's credential cannot see at all. + # It does not need to: attaching to a Gateway is a reference, and the + # `Accepted` condition the deploy checks afterwards is written onto the + # HTTPRoute in THIS namespace. + name: insight + namespace: envoy-gateway-system + sectionName: https resources: requests: {cpu: 25m, memory: 32Mi} limits: {cpu: 500m, memory: 128Mi} @@ -407,12 +513,22 @@ authenticator: kind: ClusterIssuer oidc: - # The bundled Keycloak's broker realm, served at /kc on the public - # hostname. The authenticator resolves this URL ITSELF at startup for - # discovery, so the name has to be reachable from inside the cluster as - # well as from the browser — which is why it is the public name and not - # an in-cluster Service address. - issuerUrl: "https://insight-test.cfabric.org/kc/realms/insight-broker" + # Realm `insight` — the roster realm generated from the demo seed's own + # organisation — NOT `insight-broker`, which is the GitHub-federated one. + # This stand is the CI stand: no GitHub login, no OAuth app, nobody + # signing in by hand, so its people are the seeded roster and they carry + # local passwords. That single decision is what makes an automated + # multi-persona login possible here at all. + # + # Consequence worth stating: the stand is only useful once the seeder has + # run. Logins resolve against `identity.persons` rows the seeder writes, + # so an unseeded stand authenticates a user and then denies them. + # + # Served at /kc on the public hostname. The authenticator resolves this + # URL ITSELF at startup for discovery, so the name has to be reachable + # from inside the cluster as well as from the browser — which is why it is + # the public name and not an in-cluster Service address. + issuerUrl: "https://insight-test.cfabric.org/kc/realms/insight" clientId: "insight-authenticator" # ───────────────────────────────────────────────────────────────────── @@ -446,9 +562,35 @@ authenticator: redirectUri: "https://insight-test.cfabric.org/auth/callback" - # `openid` alone is correct here and is not an oversight: the broker - # client is `fullScopeAllowed: false` with a fixed set of default scopes, - # so the claims the login bootstrap needs arrive without being requested. + # `openid` alone is correct here and is not an oversight. An earlier + # version of this comment argued it from the broker client's + # `fullScopeAllowed: false` and its fixed default scopes — that is the + # federated realm, which this stand no longer uses. The reason on the + # seeded realm is different and simpler. + # + # The generated realm attaches its claim mappers to the CLIENTS directly, + # not to a client scope. In + # src/ingestion/tools/seed/insight_seed/keycloak_realm.py, + # `_protocol_mappers()` builds five mappers — `tenant_id` and `org_unit` + # (oidc-usermodel-attribute-mapper), `groups`, `roles`, and the + # `aud-insight` audience mapper — and the SAME list is handed to both + # clients as `protocolMappers`, in `_client_insight()` and + # `_client_insight_authenticator()`. `build_realm()` emits `realm`, + # `roles`, `groups`, `users` and `clients` and nothing else: the document + # declares no `clientScopes`, no `defaultClientScopes` and no + # `optionalClientScopes` at all. A client's own dedicated mappers run on + # every token that client issues, unconditionally — they are not gated on + # a scope being requested — so the claims the login bootstrap needs are + # already in the id_token with `openid` as the only scope. `idp_sub` + # arrives by the same mechanism: the deploy post-processes the generated + # document to add one more attribute mapper on each client (see + # `externalIdClaim` below). + # + # Consequence for a future edit: adding a scope name here cannot ADD a + # claim, because nothing here is scope-gated — but a scope the realm does + # not have configured makes the authorization request fail outright with + # `invalid_scope`, i.e. login stops working to gain nothing. A new claim is + # added in the generator as a mapper, never here as a scope. scopes: ["openid"] # How a logged-in principal is resolved to a person row. The login @@ -463,7 +605,43 @@ authenticator: # redirectUri / defaultTenantId, never these. So every realm this # deployment serves must key its people the same way, which is the # constraint any change to how CI logs in has to satisfy. - sourceType: "github" + # + # `keycloak`, matching the seeded realm above. The seeder writes + # (insight_source_type='keycloak', value_type='id', value_id=) + # and the realm generator pins each realm user to that same uuid, so the + # login-bootstrap lookup finds the person. + # + # The seed Job's IDP_SOURCE_TYPE must equal this. It is not repeated + # anywhere: seed-stand.sh reads it back out of insight-authenticator-config. + # + # ───────────────────────────────────────────────────────────────────── + # `idp_sub` — and specifically NOT `sub`. Read this before "fixing" it. + # ───────────────────────────────────────────────────────────────────── + # `sub` is the obvious answer and it is wrong here, in a way that looks + # like a data problem rather than a configuration one. + # + # The realm generator sets each realm user's `id` to that person's roster + # uuid (`_user()` in keycloak_realm.py returns `"id": person.uuid`), and + # on a Keycloak REALM IMPORT that id is what `sub` carries — which is what + # makes `sub` look correct. But the chart does not import the realm: the + # keycloakConfig Job runs keycloak-config-cli, which creates users one at + # a time through the admin REST API, and Keycloak assigns its OWN id on + # `POST /users` and silently discards the document's. So on this stand + # `sub` is a Keycloak-internal uuid that matches nothing the seeder ever + # wrote to `identity.persons`. + # + # The deploy therefore copies the roster uuid into an `idp_sub` user + # attribute and adds an `oidc-usermodel-attribute-mapper` that emits it — + # a custom claim carrying the id `sub` would have carried under an import. + # (Keycloak 26's declarative user profile must also allow that attribute + # through, or it is dropped on creation and the mapper emits nothing.) + # + # Change this to `sub` and every persona still authenticates — password + # accepted, tokens issued, pods healthy, release `deployed` — and is then + # denied at the callback with `login denied: no matching person in + # Identity`, against a fully populated `identity.persons`. Nothing in that + # signal points back at this line. INFRA.md §11 has the same warning. + sourceType: "keycloak" externalIdClaim: "idp_sub" # Fallback tenant for an id_token without a tenant claim. Must equal From 54f3a14f37757c4d785eec35a2c3e50996cfe889 Mon Sep 17 00:00:00 2001 From: Konstantin Tursunov Date: Mon, 10 Aug 2026 16:49:13 +0800 Subject: [PATCH 10/59] fix(stand): recover the seed manifest from an interleaved log, and re-sync the rest to the stand MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit THE BUG. The manifest capture scanned for a `{` at column 0, took the next `}` at column 0, and parsed everything between. The seeder prints the manifest to stdout while its logging handler writes to stderr, and both `kubectl logs` and the shell redirection that captures a seed run merge the two — drained independently, so a twelve-kilobyte document does not arrive in one piece and a log line written mid-drain lands BETWEEN two of the document's lines. The seeder logs immediately after printing it, which makes this the ordinary case. One interleaved line and `json.loads` raised, the step failed, and the smoke stage aborted for want of a manifest — on a run whose deploy and seed had both succeeded, presenting as a seeding problem. The fix drops lines matching the logger's shape (the SHAPE, so a persona's display name is never filtered out of the document), then tries the surviving column-0 spans latest-open-first and accepts only one that parses AND carries `manifest_version` — which covers what the filter cannot, like a traceback or a helper writing on the container's own descriptors. It lives in a committed script because two callers need it. They were two implementations; when the CI one was fixed the harness's was not, and its comment went on claiming the two matched — a parity report asserting a parity it no longer had, which is worse than not checking. One algorithm now, invoked by both, so neither can drift without deleting the file. THE REST is re-synced to the redeployed stand. The route-check comments said the chart renders no HTTPRoute and that CI would be a second writer; both are false now that the release owns them, and the CI credential's Gateway API grant is more necessary for it, not less. The workflow header claimed it does no `kubectl rollout restart` while carrying that very step. The smoke suite was written for a stand where password login was impossible; the seeded roster realm serves a password form, so `password` is the mode it defaults to and the long explanation of why it could not work is replaced by what is now true. Refs #2244 Signed-off-by: Konstantin Tursunov --- .github/workflows/deploy-test-stand.yml | 169 ++++++----- .../scripts/extract-seed-manifest.py | 153 ++++++++++ deploy/gitops/scripts/emulate-ci-deploy.sh | 126 +++++--- .../gitops/scripts/provision-ci-deployer.sh | 40 ++- .../deployment/gitops/ci-emulation.md | 14 +- .../specs/sop/credentials-runbook.md | 119 ++++++-- tests/stand/smoke/README.md | 139 +++++++-- tests/stand/smoke/conftest.py | 5 +- tests/stand/smoke/login.py | 271 ++++++++++++++---- tests/stand/smoke/test_deploy_smoke.py | 6 +- 10 files changed, 784 insertions(+), 258 deletions(-) create mode 100755 .github/workflows/scripts/extract-seed-manifest.py diff --git a/.github/workflows/deploy-test-stand.yml b/.github/workflows/deploy-test-stand.yml index dcf5d369b..e6731eff4 100644 --- a/.github/workflows/deploy-test-stand.yml +++ b/.github/workflows/deploy-test-stand.yml @@ -55,14 +55,13 @@ name: Deploy test stand # * No alerting. The red X is the whole notification story. # * No rollback, and no `--atomic`. A failed upgrade is LEFT failed so the next # person to look has the evidence (see the deploy stage). -# * No infrastructure bootstrap. Datastores, the gateway, cert-manager and the -# IdP realm are somebody else's lifecycle; this workflow installs the -# application chart and nothing beneath it. -# * No `kubectl rollout restart` after the upgrade. The composed *-config -# Secrets reach pods through `envFrom`, which is read once at container -# start, so a changed coordinate can produce a healthy-looking release with -# stale configuration. Adding the restart is a real improvement and a real -# behaviour change; it belongs in its own change with its own reasoning. +# * No infrastructure bootstrap. Datastores, the shared Envoy Gateway, +# cert-manager and the realm ConfigMap the bundled IdP is configured from +# are somebody else's lifecycle; this workflow installs the application +# chart and nothing beneath it. One thing that used to be on that list has +# moved: the two edge HTTPRoutes are rendered BY the chart, from this +# environment's `gateway.route` and `keycloak.route`, so the release owns +# them. What stays external is the Gateway they attach to. # * No exact-value assertions on seeded data. The smoke proves shape and # non-emptiness. Reading a number off a running stand and asserting it back # is a test of nothing. @@ -135,11 +134,19 @@ name: Deploy test stand # given straight into the authenticator's config. An upgrade without that # value produces a confidential OIDC client with a blank secret: pods Ready, # release `deployed`, every login broken. -# 3. The stand's IdP can complete the login the smoke stage attempts. At the -# time of writing the stand's realm federates to an external provider and -# has no local users, which no amount of test code can work around; the -# smoke suite carries two modes and fails — loudly, naming the step it got -# stuck at — rather than pretending. See tests/stand/smoke/README.md. +# 3. The stand's IdP can complete the login the smoke stage attempts. This +# stand's realm is `insight` — the roster realm generated from the demo +# seed's own organisation, carrying a local password user per seeded +# person — so a scripted username+password login is the supported path +# here and the smoke's `password` mode is the one that fits. What the +# precondition still buys is the half nobody expects: AUTHENTICATING IS +# NOT RESOLVING. The callback looks the signed-in principal up in +# `identity.persons` by the id the seeder wrote, and fails closed when +# there is no match, so a deployed-but-unseeded stand signs a persona in +# and then denies them. That is why the smoke stage runs after the seed +# and never instead of it, and why the suite fails — loudly, naming the +# step it got stuck at — rather than pretending. See +# tests/stand/smoke/README.md. # # ── COST ──────────────────────────────────────────────────────────────────── # @@ -248,8 +255,11 @@ jobs: # is therefore read exactly once, at container start. See stage 1's restart # step for why a list of two would be a silent bug. RESTART_TARGETS: deploy/insight-authenticator deploy/insight-analytics deploy/insight-identity-resolution - # The edge objects the chart does not render and this release does not own, - # but every acceptance criterion travels through. + # The two edge routes the chart renders from this environment's + # `gateway.route` and `keycloak.route`. The release owns them, so the + # upgrade below is what writes them; what the check after it asks is + # whether the Gateway ACCEPTED them, which helm does not wait for. Every + # acceptance criterion travels through both. ROUTE_NAMES: insight-gateway insight-keycloak CHART_VERSION: ${{ inputs.chart_version }} REDACT: .github/workflows/scripts/redact-stand-log.py @@ -362,6 +372,23 @@ jobs: # `password` (every persona authenticates as themselves) or `override` # (one principal authenticates and every persona session is minted from # it). Unset means the suite's own default, which is `password`. + # + # `password` is the mode THIS stand serves. Its realm carries a local + # password user per seeded person, so each persona signs in as itself + # and a green run is evidence of a login rather than of an + # impersonation. `override` is for a stand whose realm federates to an + # external provider and therefore serves no password form at all: it + # proves strictly less, and it additionally depends on the + # authenticator's `overrideEnabled` staying on — a flag the chart's own + # values say must be false anywhere real users log in. + # + # Still read from a repository variable rather than pinned in this + # file, because which mode fits is a fact about the stand's realm and + # not about this workflow: a stand can be redeployed into the other + # shape without a line here changing. Setting it explicitly to + # `password` is better than leaving it unset — an unset variable and a + # stale `override` look identical in the settings page, and only one of + # them quietly keeps the gate in impersonation mode. LOGIN_MODE: ${{ vars.TEST_STAND_SMOKE_LOGIN_MODE }} RUN_SMOKE: ${{ steps.stages.outputs.run_smoke }} run: | @@ -414,10 +441,13 @@ jobs: echo " TEST_STAND_SEED_EMAIL address the seeded dev-lead persona resolves to." echo " A secret rather than a variable purely so it is" echo " masked in this public log." - echo " TEST_STAND_PERSONA_PASSWORD 'password' mode: the credential every persona" - echo " signs in with." - echo " TEST_STAND_BOOTSTRAP_EMAIL 'override' mode: the one principal that really" - echo " TEST_STAND_BOOTSTRAP_PASSWORD authenticates; persona sessions are minted from it." + echo " TEST_STAND_PERSONA_PASSWORD 'password' mode — the mode this stand's realm" + echo " serves: the credential every persona signs in" + echo " with. The runbook says where its value comes from." + echo " TEST_STAND_BOOTSTRAP_EMAIL 'override' mode, for a stand whose realm serves no" + echo " TEST_STAND_BOOTSTRAP_PASSWORD password form at all: the one principal that really" + echo " authenticates; persona sessions are minted from it." + echo " Not needed here while the mode is 'password'." echo " TEST_STAND_BASE_URL the stand's public HTTPS origin, scheme and host," echo " no trailing slash. A variable, not a secret: the" echo " smoke drives it as a browser would." @@ -663,15 +693,24 @@ jobs: - name: 'Stage 1/3 — confirm the edge still routes' run: | set -euo pipefail - # The chart renders NO HTTPRoute. These two objects live outside the - # release, are owned by the deployment repository, and every acceptance - # criterion travels through them — so a successful upgrade says nothing - # about whether the stand is reachable. + # The chart renders both of these, from this environment's + # `gateway.route` and `keycloak.route`, so the release owns them and + # the upgrade above is what wrote them. This step is therefore a + # POST-CONDITION on the deploy — "did what I just write take effect" — + # and not, as it once was, a check on an object somebody else applied. # - # Read, never applied: the files under - # deploy/gitops/environments/test-stand/manifests/ are the source of - # truth, and applying them from here would make CI a second writer on an - # object a human owns. + # It still earns its place, because `helm --wait` waits on workloads + # and not on HTTPRoute status. A route helm stored successfully can be + # refused by the Gateway it names: a parentRef pointing at a Gateway + # that is not there, a sectionName naming a listener that does not + # exist, a hostname the listener will not widen to. Every one of those + # leaves the release reporting `deployed` with all pods Ready and the + # stand unreachable, and `Accepted=False` on the route is the only + # place that failure is written down. + # + # Nothing is applied from here, and there is nothing to apply: the + # routes arrive with the release, and the Gateway they attach to lives + # in a namespace this credential cannot see at all. # shellcheck disable=SC2086 # ROUTE_NAMES is a deliberate word list kubectl -n "$STAND_NAMESPACE" get httproute $ROUTE_NAMES \ -o 'custom-columns=NAME:.metadata.name,ACCEPTED:.status.parents[0].conditions[?(@.type=="Accepted")].status' @@ -684,8 +723,11 @@ jobs: if [ -n "$not_accepted" ]; then echo "::error::an edge route is not Accepted:" printf '%s\n' "$not_accepted" - echo "The release may be perfectly healthy; the stand is still unreachable, and the" - echo "smoke below would fail at its first request with a much less useful message." + echo "The release is 'deployed' and its pods are Ready; the stand is still unreachable," + echo "and the smoke below would fail at its first request with a much less useful" + echo "message. The route came out of this deploy, so start in the values file that" + echo "produced it — $VALUES_FILE, keys gateway.route and keycloak.route — and check the" + echo "parentRef they name against the Gateway the cluster actually has." exit 1 fi @@ -744,57 +786,26 @@ jobs: # record, and the smoke suite needs the document to know which personas # exist. Extracted here rather than uploaded anywhere: it names every # persona and the tenant. - python3 - "$RUNNER_TEMP/seed.log" "$RUNNER_TEMP/stand-manifest.json" <<'PY' - import json - import sys - - source, target = sys.argv[1], sys.argv[2] - lines = open(source, encoding="utf-8", errors="replace").read().splitlines() - - # The document is rendered with json.dumps(indent=2), so its opening - # brace is alone on a line at column 0 and its closing brace is the next - # line at column 0. Nested braces are indented and cannot be confused - # for either. The LAST such block wins: a run that printed more than one - # is a run whose later document supersedes the earlier. - block, start = None, None - for index, line in enumerate(lines): - if line == "{": - start = index - elif line == "}" and start is not None: - block = "\n".join(lines[start : index + 1]) - start = None - - if block is None: - print("::error::no seed manifest in the seed log — the seeder did not reach the end of its run") - raise SystemExit(1) - - try: - doc = json.loads(block) - except json.JSONDecodeError as exc: - print(f"::error::the seed log's manifest block is not valid JSON: {exc}") - raise SystemExit(1) from exc - - with open(target, "w", encoding="utf-8") as handle: - json.dump(doc, handle, indent=2, sort_keys=True) - handle.write("\n") - - # A summary, not the document. Fixture NAMES are contract; persona - # emails, UUIDs and the tenant are not printed anywhere. - personas = doc.get("personas") or [] - fixtures = sorted((doc.get("fixtures") or {}).keys()) - print(f"manifest_version: {doc.get('manifest_version')}") - print(f"anchor_date: {doc.get('anchor_date')}") - print(f"data_window: {doc.get('data_window')}") - print(f"seed_revision: {doc.get('seed_revision')}") - print(f"personas: {len(personas)}") - print(f"fixtures: {', '.join(fixtures) if fixtures else '(none)'}") - PY - echo "INSIGHT_STAND_MANIFEST=$RUNNER_TEMP/stand-manifest.json" >> "$GITHUB_ENV" + # + # The algorithm lives in a committed script rather than in this heredoc + # because deploy/gitops/scripts/emulate-ci-deploy.sh needs the same one, + # and when this step was fixed for interleaved log lines that harness's + # copy was not — while its comment went on claiming the two matched. One + # file, two callers, no drift. Read that script's header for why a naive + # brace scan cannot work: the manifest is stdout and the seeder's log is + # stderr, and this log is the two merged and interleaved. + # + # Diagnostics arrive on stderr in plain text; the `::error::` framing is + # this caller's, which is why the script does not bake it in. + if ! python3 .github/workflows/scripts/extract-seed-manifest.py \ + "$RUNNER_TEMP/seed.log" "$RUNNER_TEMP/stand-manifest.json" 2> "$RUNNER_TEMP/extract.err"; then + echo "::error::could not recover the seed manifest from the seed log" + python3 "$REDACT" < "$RUNNER_TEMP/extract.err" + echo "The seed stage's redacted output is immediately above — the document is in it," + echo "along with whatever landed inside it." + exit 1 + fi - # ── Stage 3 of 3: smoke ─────────────────────────────────────────────── - # Never reached after a failed seed: this step has no `if:`, so a failure - # above skips it. That is the whole mechanism — asserting against an - # unseeded stand would produce a failure that looks like a product bug. - name: 'Stage 3/3 — smoke the stand through its public URL' if: steps.stages.outputs.run_smoke == 'true' timeout-minutes: 5 diff --git a/.github/workflows/scripts/extract-seed-manifest.py b/.github/workflows/scripts/extract-seed-manifest.py new file mode 100755 index 000000000..a268a0a88 --- /dev/null +++ b/.github/workflows/scripts/extract-seed-manifest.py @@ -0,0 +1,153 @@ +#!/usr/bin/env python3 +"""Recover the seed manifest from a seed run's merged log. + +Usage: extract-seed-manifest.py + +WHY THIS FILE EXISTS AT ALL, RATHER THAN INLINE IN ITS TWO CALLERS +----------------------------------------------------------------- +Two things read a seed log for its manifest: the `Stage 2/3 — capture the seed +manifest` step in .github/workflows/deploy-test-stand.yml, and +`extract_manifest()` in deploy/gitops/scripts/emulate-ci-deploy.sh. That harness +exists to prove the laptop path and the CI path run the SAME commands, and it +prints a parity report saying so. + +They were two implementations. When the CI one was fixed for interleaved log +lines the shell one was not, and its comment went on claiming it matched — the +harness asserting a parity it no longer had, which is worse than not checking. +Extracting the logic here makes the parity structural instead of asserted: there +is one algorithm and both callers invoke it, so neither can drift from the other +without deleting this file. + +WHY THE SCAN IS NOT "the last `{` at column 0 through the next `}` at column 0" +------------------------------------------------------------------------------ +That is what both callers used to do, and it is wrong for a reason no amount of +care about brace nesting can fix: THE DOCUMENT AND THE LOG LINES ARE TWO +DIFFERENT STREAMS. The seeder prints the manifest to stdout, while the logging +handler it installs writes to stderr. Both `kubectl logs` and the shell +redirection that captures a seed run merge the two, and the container runtime +that recorded them drained the pipes independently — so a twelve-kilobyte +document does not arrive in one piece, and a log line written while it is being +drained is recorded BETWEEN two of the document's lines. The seeder logs +immediately after printing the manifest, which makes this the ordinary case +rather than an exotic one. + +The old scan joined every line of the span, so one interleaved line made +`json.loads` raise, the step failed, and the smoke stage aborted for want of a +manifest — on a run whose deploy and seed had both succeeded. Please do not +simplify it back. + +Two defences, in order: + + 1. Drop the interlopers. The seeder's handler opens every line with a + timestamp, a level and a dotted logger name. No line of a + `json.dumps(indent=2)` document can match that: they are `{`, `}`, or + indented by at least two spaces. Matching the logger's SHAPE rather than a + keyword list is what keeps a persona's display name from being filtered out + of the document. + 2. Try the surviving spans instead of trusting one, and accept only a span + that parses AND carries `manifest_version`. That covers what rule 1 cannot: + a traceback, a log record with an embedded newline, or some other column-0 + `{`…`}` block in the stream — the seed steps shell out to helpers on the + container's own file descriptors, and their output is not the seeder's to + shape. + +Diagnostics go to stderr in plain text and the exit status is what callers +branch on; neither caller's annotation style is baked in here. +""" + +from __future__ import annotations + +import json +import re +import sys +from itertools import islice +from pathlib import Path + +#: The seeder's log-line shape: "