From 3fed82454e1f7f7935030871ec655e665916416d Mon Sep 17 00:00:00 2001 From: Konstantin Tursunov Date: Sat, 25 Jul 2026 00:48:10 +0300 Subject: [PATCH 01/14] docs(deploy): rewrite HELM_DEPLOY.md for umbrella chart 0.4.31 The runbook described a pre-split architecture that no longer exists. Conform it to chart 0.4.31: - Overview: 5 first-party services (gateway, authenticator, analytics, identity, frontend) + request/JWT diagram; authenticator is a separate pod. - Prerequisites: add cert-manager + ClusterIssuer + ingress controller. - Step 1 values: real gateway:/authenticator:/identity:/frontend: blocks; drop phantom apiGateway/authDisabled/oidc.existingSecret and inert *.deploy infra flags; image.tag documented as optional (appVersion fallback). - Step 2: drop fictional insight-oidc; add insight-authenticator-signing-keys (ES256 current.pem). - Steps 4-5: document the clickhouse-migrate hook Job; correct pod/CronWorkflow names. - Remove the fictional non-idempotent-migration note (the gold-view migration is idempotent) and correct the gateway auth description (cached Lua cosocket to /internal/authz, not nginx auth_request). - Connectors: 25 (drop unimplemented bitbucket-server); mark hubspot as a baked-url_base CDK connector. - Note fakeidp/keycloak dev images are not on public GHCR. Validated live: all 5 services install and reach Running following the runbook; helm template renders with no schema errors; the migrate hook is idempotent across two consecutive helm upgrades. Also gitignore .cf-studio/.plans/ (local Constructor Studio plan working state). Signed-off-by: Konstantin Tursunov --- .gitignore | 2 + deploy/HELM_DEPLOY.md | 298 +++++++++++++++++++++++++----------------- 2 files changed, 180 insertions(+), 120 deletions(-) diff --git a/.gitignore b/.gitignore index bf27f21e8..790bec6b7 100644 --- a/.gitignore +++ b/.gitignore @@ -79,6 +79,8 @@ coverage-raw/ # Files matched here are owned by Constructor Studio and may be overwritten. .cf-studio/.core/ .cf-studio/.gen/ +# Constructor Studio plan files — local working state (plan-first + cf-plan). +.cf-studio/.plans/ .agents/skills/cf-analyze/SKILL.md .agents/skills/cf-auto-config/SKILL.md .agents/skills/cf-brainstorm/SKILL.md diff --git a/deploy/HELM_DEPLOY.md b/deploy/HELM_DEPLOY.md index 4abfb2ce2..ba92a55c7 100644 --- a/deploy/HELM_DEPLOY.md +++ b/deploy/HELM_DEPLOY.md @@ -4,22 +4,59 @@ This runbook shows a platform or DevOps engineer how to install the Insight busi ## Contents + + +- [Contents](#contents) - [Overview](#overview) - [Prerequisites](#prerequisites) + - [Cluster and CLI tools](#cluster-and-cli-tools) + - [Cluster-level dependencies](#cluster-level-dependencies) + - [Running external infrastructure](#running-external-infrastructure) - [Step 1 — Configure values/umbrella.yaml](#step-1--configure-valuesumbrellayaml) -- [Step 2 — Fill the two secret files](#step-2--fill-the-two-secret-files) -- [Step 3 — Create namespace and apply secrets](#step-3--create-namespace-and-apply-secrets) +- [Step 2 — Fill the secret files](#step-2--fill-the-secret-files) + - [secrets/insight-db-creds.yaml](#secretsinsight-db-credsyaml) + - [secrets/insight-authenticator-signing-keys.yaml](#secretsinsight-authenticator-signing-keysyaml) +- [Step 3 — Create namespace, apply secrets, mirror Airbyte auth](#step-3--create-namespace-apply-secrets-mirror-airbyte-auth) - [Step 4 — Install with Helm](#step-4--install-with-helm) -- [Step 5 — Verify the installation](#step-5--verify-the-install) +- [Step 5 — Verify the install](#step-5--verify-the-install) - [Step 6 — Configure connectors (optional)](#step-6--configure-connectors-optional) + - [Anatomy of a connector Secret](#anatomy-of-a-connector-secret) + - [The 25 available connectors](#the-25-available-connectors) + - [Example Secret for every connector](#example-secret-for-every-connector) - [Troubleshooting](#troubleshooting) - [Appendix — Reference](#appendix--reference) + - [values/umbrella.yaml placeholders](#valuesumbrellayaml-placeholders) + - [secrets/insight-db-creds.yaml keys](#secretsinsight-db-credsyaml-keys) + - [secrets/insight-authenticator-signing-keys.yaml keys](#secretsinsight-authenticator-signing-keysyaml-keys) + - [values/umbrella.orbstack.yaml (local variant)](#valuesumbrellaorbstackyaml-local-variant) + + ## Overview -Insight reads engineering and collaboration data from your tools (Jira, Slack, GitHub, and so on), pipelines it through ClickHouse, and serves metrics to a dashboard behind an OIDC login. It installs as four services in one Helm "umbrella" chart — bundled sub-charts, so a single `helm install` deploys everything — published at `oci://ghcr.io/constructorfabric/charts/insight`. The four are the Router (`insight-api-gateway`, public ingress + OIDC), Analytic (`insight-analytics`, metrics), Identity (`insight-identity`, person resolution), and the Frontend (`insight-frontend`, the web UI). +Insight reads engineering and collaboration data from your tools (Jira, Slack, GitHub, and so on), pipelines it through ClickHouse, and serves metrics to a dashboard behind an OIDC (OpenID Connect, the login protocol) login. It installs as five first-party services in one Helm "umbrella" chart — bundled sub-charts, so a single `helm install` deploys everything — published at `oci://ghcr.io/constructorfabric/charts/insight` (the chart source also lives in-repo at `charts/insight`, which you can install from directly with `helm install insight ./charts/insight` instead of the OCI form). The five are: + +- **Gateway** (`insight-gateway`, alias `gateway`) — the OpenResty edge. It owns the public ingress and is the single entrance to the cluster: it routes `/*` to the Frontend and `/api/*` to Analytics/Identity, performing a cached cookie-to-JWT exchange against the Authenticator's `/internal/authz` endpoint (a per-pod Lua cosocket lookup, not nginx's `auth_request`) and injecting the resulting gateway JWT into upstream requests. +- **Authenticator** (`insight-authenticator`, alias `authenticator`) — a separate pod that performs the OIDC login with your IdP, keeps Redis-backed sessions, and mints the ES256 gateway JWT the Gateway injects downstream. +- **Analytics** (`insight-analytics`, alias `analytics`) — serves metrics from the ClickHouse Gold layer. +- **Identity** (`insight-identity`, alias `identity`) — resolves people and org data from MariaDB; optional (`identity.deploy`, default `false`). +- **Frontend** (`insight-frontend`, alias `frontend`) — the web UI (dashboard); optional (`frontend.deploy`, default `true`). + +Two more subcharts exist purely for local/dev use and are off by default: `fakeidp` (alias `fakeidp`, condition `fakeidp.deploy`) and `keycloak` (alias `keycloak`, condition `keycloak.deploy`) — both are bundled OIDC providers for a cluster with no real IdP available. `fakeidp` is the one this runbook documents; `keycloak` is a heavier bundled alternative not covered here. Neither is appropriate for a real environment. + +```mermaid +flowchart LR + B[Browser] --> ING[Ingress: host] + ING --> GW[insight-gateway] + GW -->|"/*"| FE[insight-frontend] + GW -->|"/api/*"| AN[insight-analytics / insight-identity] + GW -->|"cookie→JWT exchange
(cached, cosocket)"| AUTH[insight-authenticator] + AUTH -->|OIDC login| IDP[External IdP] + AUTH -->|mints gateway JWT
signed w/ signing keys| GW + AN -->|verifies JWT via JWKS
over cert-manager CA| AUTH +``` -This path assumes your data infrastructure (ClickHouse, MariaDB, Redis, Redpanda, Airbyte, Argo Workflows) already runs and is reachable from the cluster, in another namespace or external. The chart doesn't stand it up; it only wires the services to it. You supply one values file, two secret files, and optionally one Secret per connector. No GitOps repo, CI, or auto-reconciliation — you run the commands yourself. +This path assumes your data infrastructure (ClickHouse, MariaDB, Redis, Redpanda, Airbyte, Argo Workflows) already runs and is reachable from the cluster, in another namespace or external. The chart doesn't stand it up; it only wires the services to it. You supply one values file, secret files, and optionally one Secret per connector. No GitOps repo, CI, or auto-reconciliation — you run the commands yourself. ## Prerequisites @@ -28,22 +65,31 @@ This path assumes your data infrastructure (ClickHouse, MariaDB, Redis, Redpanda - A Kubernetes cluster you can already reach with `kubectl`, with permission to create namespaces, Secrets, workloads, and Roles/RoleBindings — including in the Airbyte namespace when it differs from the app's (the chart installs a Role there that lets its jobs read Airbyte's auth Secret). That namespace must exist before the umbrella install. - `helm` ≥ 3.8 (OCI registry support is stable from 3.8 onward, since the chart is pulled as an OCI artifact). - `kubectl`. +- `jq`, used to mirror the Airbyte auth Secret in Step 3. +- `openssl`, used to generate the authenticator's signing key in Step 2. - `base64`, used when copying existing datastore passwords in Step 2 (most systems ship this by default). +### Cluster-level dependencies + +Two things must already be installed in the cluster before you install this chart — neither is bundled by it: + +- **An ingress controller.** The Gateway and Frontend ingress blocks are hardcoded to `className: nginx`; install an ingress-nginx controller (or override `gateway.ingress.className` / `frontend.ingress.className` to match whatever you run). +- **cert-manager**, with a working `ClusterIssuer`. The authenticator's TLS-discovery sidecar (`authenticator.tlsDiscovery.enabled: true` by default) creates a `cert-manager.io/v1` `Certificate`, so cert-manager's CRDs must be present. Analytics and Identity trust that cert-manager-issued CA to verify the authenticator's JWKS over HTTPS — this is load-bearing, not optional. The chart's default `issuerRef.name` is `local-ca`; either provision a `ClusterIssuer` with that name, or override `authenticator.tlsDiscovery.issuerRef.name` to point at your own. + ### Running external infrastructure -All six systems below must be deployed and reachable from the cluster before you start. For the four datastores — ClickHouse, MariaDB, Redis, and Redpanda — the chart's `deploy: false` settings (see Step 1) tell it to dial these systems, not install them. Airbyte and Argo Workflows have no `deploy` key; the chart instead points at them via `airbyte.namespace` (or an explicit `airbyte.apiUrl`) and `ingestion.reconcile.argoInstanceId`. +All six systems below must be deployed and reachable from the cluster before you start. The chart never installs any of them — ClickHouse, MariaDB, Redis, and Redpanda are wired in purely by host/credentials (Step 1); Airbyte and Argo Workflows are wired in via `airbyte.apiUrl` and `ingestion.reconcile.argoInstanceId`. | System | Used for | |--------|----------| -| ClickHouse | Stores the Bronze (raw ingested data), Silver (cleaned/conformed), and Gold (query-ready) data layers; Analytic reads the Gold layer to serve metrics | +| ClickHouse | Stores the Bronze (raw ingested data), Silver (cleaned/conformed), and Gold (query-ready) data layers; Analytics reads the Gold layer to serve metrics | | MariaDB | Owns the `identity` database that Identity uses to resolve people and org data | -| Redis | Caching layer used by Analytic | +| Redis | Caching layer used by Analytics and session storage used by the Authenticator | | Redpanda | Event-streaming backbone (Kafka-compatible) used by the ingestion pipeline | | Airbyte | Runs the data connectors (Jira, Slack, GitHub, and so on) that load raw data into ClickHouse Bronze | | Argo Workflows | Runs the dbt transform workflows that turn Bronze into Silver and Gold, and runs the sync workflows Airbyte connections trigger | -Run all commands here from the directory holding your `values/`, `secrets/`, and `connectors/` files. This document is self-contained: it shows the full `values/umbrella.yaml` skeleton, both secret files, and an example Secret for every connector, so you can assemble all three directories from what follows. +Run all commands here from the directory holding your `values/`, `secrets/`, and `connectors/` files. This document is self-contained: it shows the full `values/umbrella.yaml` skeleton, the secret files, and an example Secret for every connector, so you can assemble all three directories from what follows. ## Step 1 — Configure values/umbrella.yaml @@ -57,35 +103,32 @@ credentials: autoGenerate: true # BYO compose; won't overwrite a labelless insight-db-creds global: - tenantDefaultId: "" # default tenant UUID; must equal ingestion.reconcile.tenantId + tenantDefaultId: "" # single-tenant seed UUID; must equal ingestion.reconcile.tenantId # storageClass: "" # "" = cluster default; e.g. "local-path" locally # imagePullSecrets: [] # [{name: my-regcred}] for a private registry -# Datastore wiring — deploy:false = dial existing infra, don't install it. +# Datastore wiring — every dep is external; the chart only dials it. clickhouse: - deploy: false host: # e.g. clickhouse..svc.cluster.local port: 8123 database: insight username: insight mariadb: - deploy: false host: port: 3306 database: insight username: insight redis: - deploy: false host: port: 6379 redpanda: - deploy: false brokers: "" # e.g. redpanda..svc.cluster.local:9093 # Ingestion — point at existing Airbyte + Argo; install the dbt WorkflowTemplates. ingestion: templates: enabled: true + toolboxImage: "" # e.g. ghcr.io/constructorfabric/insight-toolbox: reconcile: tenantId: "" destinationName: clickhouse-bronze @@ -96,15 +139,16 @@ airbyte: analytics: replicaCount: 1 # chart default 2; bump for HA + image: + tag: "" # optional — falls back to the chart's appVersion resources: requests: { cpu: 100m, memory: 128Mi } limits: { cpu: 500m, memory: 512Mi } -apiGateway: +gateway: replicaCount: 1 - authDisabled: false # true = NO auth — LOCAL DEV ONLY - oidc: - existingSecret: "insight-oidc" + image: + tag: "" # optional — falls back to the chart's appVersion ingress: enabled: true className: nginx @@ -116,25 +160,50 @@ apiGateway: requests: { cpu: 100m, memory: 128Mi } limits: { cpu: 500m, memory: 256Mi } +authenticator: + replicaCount: 1 + image: + tag: "" + # ES256 signing keys — see Step 2. MUST already exist as a Secret before install. + signingKeysSecret: "insight-authenticator-signing-keys" + # cert-manager Certificate for the JWKS-discovery sidecar. Override + # issuerRef.name only if your cluster's ClusterIssuer isn't named `local-ca`. + tlsDiscovery: + enabled: true + issuerRef: + name: local-ca + oidc: + issuerUrl: "" # MUST be set — your IdP's issuer URL + clientId: "" + clientSecret: "" + redirectUri: "https:///auth/callback" # MUST be set — browser-facing callback through the gateway + scopes: ["openid", "profile", "email"] + identity: deploy: true # MUST be true (chart default false) replicaCount: 1 + image: + tag: "" databaseName: "identity" - tenantDefaultId: "" resources: requests: { cpu: 50m, memory: 96Mi } limits: { cpu: 250m, memory: 384Mi } frontend: # the web UI (dashboard) + deploy: true replicaCount: 1 + image: + tag: "" ingress: enabled: true # WITHOUT this the UI pod runs but is never exposed className: nginx - host: # same FQDN as apiGateway.ingress.host; /api/* → Router, /* → UI + host: # same FQDN as gateway.ingress.host; /api/* → Gateway routes, /* → UI oidc: # public values; the browser starts the login here - issuer: "" # same IdP as the Router + issuer: "" # same IdP as the authenticator clientId: "" scopes: "openid profile email" # IdP-specific + +# fakeidp: {deploy: false} # local/dev-only alternative to a real IdP — see note below ``` If you do not already have this file, you can generate the chart's default values as a starting point instead of typing the skeleton by hand: @@ -147,33 +216,34 @@ The placeholder table below explains every `<...>` value in the skeleton: | Placeholder | What it should be | |-------------|--------------------| -| `` | Your Insight tenant UUID. Must be the same value in all three of `global.tenantDefaultId`, `ingestion.reconcile.tenantId`, and `identity.tenantDefaultId` | +| `` | Your Insight tenant UUID/slug. Must be the same value in `global.tenantDefaultId` and `ingestion.reconcile.tenantId` | | `` | ClickHouse HTTP host, in `host:8123` form | | `` | MariaDB host, in `host:3306` form | | `` | Redis host, in `host:6379` form | | `` | Redpanda broker(s), in `host:9093` form | -| `` | Namespace of the Airbyte release, for example `insight-infra`. Leave `""` if Airbyte shares the app namespace | +| `` | The ingestion toolbox image reference (drives the WorkflowTemplates and the ClickHouse gold-view migration Job, Step 4/5) | +| `` | Airbyte server API URL, for example `http://host:8001` | | `` | Your Argo controller's instance ID, for example `argo-workflows-insight-infra` | -| `` | Public FQDN for the ingress, shared by the Router and Frontend, for example `insight.example.com` | +| `` | The Insight product image tag for each service. All five `image.tag` fields are optional — each falls back to that subchart's `Chart.yaml` appVersion (pinned by the release pipeline). Set them explicitly (recommended) so every service lands on the exact same product build; leaving them blank is safe only when all five subcharts' appVersion are in lockstep in the chart release you install | +| `` | Public FQDN for the ingress, shared by the Gateway and Frontend, for example `insight.example.com` | | `` | Name of the Kubernetes TLS Secret that covers that domain | - -The `frontend.oidc` block also uses `` and `` — the same public values you put in the `insight-oidc` Secret (Step 2). The browser needs them to start the login; the Router validates the resulting token from its Secret. +| `` | Your IdP's issuer URL. Its `/.well-known/openid-configuration` document must resolve from inside the cluster | +| `` / `` | Your OIDC client / application registration credentials | For infrastructure running in the same cluster, use the in-cluster DNS form `..svc.cluster.local`. Any resolvable host or IP address also works. -Three settings deserve a closer look before you install: - -- **`clickhouse.deploy`, `mariadb.deploy`, `redis.deploy`, `redpanda.deploy` are all `false`.** This tells the chart to connect to your existing datastores rather than install its own. Do not flip these to `true` unless you actually want the chart to provision new infrastructure — that is a different install path and is out of scope for this guide. -- **`identity.deploy` must be `true`.** The chart's own default is `false`, so this block requires an explicit override. Without it, the Identity service (and person resolution for the whole app) will not deploy. -- **`apiGateway.authDisabled` must stay `false` in any real environment.** Setting it to `true` disables authentication on the Router entirely — this is a local-dev-only escape hatch, never appropriate for a shared or production cluster. +A few settings deserve a closer look before you install: -A local/OrbStack variant of this file, `values/umbrella.orbstack.yaml`, is available as a starting point for local development against a k3s cluster with in-cluster infra under `insight-infra`. Treat it as a reference for local testing, not as a template for a real install — see the Appendix for a summary of how it differs. +- **`identity.deploy` must be `true`.** The chart's own default is `false`, so this block requires an explicit override. Without it, the Identity service (and person resolution for the whole app) does not deploy. +- **`authenticator.oidc.issuerUrl` and `authenticator.oidc.redirectUri` are hard requirements.** `charts/insight/templates/secrets.yaml` wraps both in Helm's `required` function — the chart refuses to render without them. There is no auth-off escape hatch: OIDC is mandatory in every environment. +- **No dummy-IdP values file exists for this chart.** If you need a working install without wiring up a real external IdP (local/dev only), enable the bundled fake provider instead: set `fakeidp.deploy: true`, point `authenticator.oidc.issuerUrl` at the in-cluster fakeidp FQDN it exposes, and leave `clientSecret` empty. Never do this in a shared or production cluster. Note: the `fakeidp` (and `keycloak`) images are dev-only and are **not** published to public GHCR — building them locally and loading them into the cluster (or supplying an `imagePullSecret` with access) is required, otherwise the fakeidp pod fails with `ImagePullBackOff` (403). This does not affect a real install, which uses your own IdP and never deploys fakeidp. +- **`authenticator.signingKeysSecret` must already exist.** It is not auto-generated by the chart — create it in Step 2 before installing. -## Step 2 — Fill the two secret files +## Step 2 — Fill the secret files ### secrets/insight-db-creds.yaml -This Secret holds the four datastore passwords used by Analytic and Identity. All four keys are required — the chart fails fast if any is missing. Values must match the passwords your datastores were deployed with. +This Secret holds the four datastore passwords used by Analytics and Identity. All four keys are required — the chart fails fast if any is missing. Values must match the passwords your datastores were deployed with. ```yaml apiVersion: v1 @@ -181,10 +251,10 @@ kind: Secret metadata: { name: insight-db-creds, namespace: insight } type: Opaque stringData: - clickhouse-password: "CHANGE_ME" # ClickHouse admin password -> Analytic - mariadb-password: "CHANGE_ME" # MariaDB app-user password -> Analytic + Identity + clickhouse-password: "CHANGE_ME" # ClickHouse admin password -> Analytics + mariadb-password: "CHANGE_ME" # MariaDB app-user password -> Analytics + Identity mariadb-root-password: "CHANGE_ME" # MariaDB root password (identity-DB init hook) -> Identity - redis-password: "CHANGE_ME" # Redis password -> Analytic + redis-password: "CHANGE_ME" # Redis password -> Analytics + Authenticator ``` If your existing datastores already have these passwords stored in Secrets in your infrastructure namespace, copy them across instead of retyping them: @@ -201,47 +271,26 @@ Paste the decoded output into the matching `clickhouse-password` / `mariadb-pass > **Do not add an `app.kubernetes.io/managed-by: Helm` label to this Secret.** The chart reads that label's *absence* as "bring your own" credentials. With the label, it assumes ownership and may overwrite your passwords with generated ones. Without it, the chart keeps your values and composes `insight-analytics-config` and `insight-identity-config` from them. -### secrets/insight-oidc.yaml +### secrets/insight-authenticator-signing-keys.yaml -This Secret carries the OIDC (OpenID Connect, the login protocol) configuration consumed by the Router. It works with any standards-compliant OIDC identity provider — Entra, Okta, Auth0, Keycloak, or Dex. +The authenticator mints the gateway JWT using an ES256 (EC P-256) key pair. This Secret is **not** auto-generated by the chart — you must create it yourself before `helm install`. Generate a PKCS#8 private key and load it under the required `current.pem` key: -```yaml -apiVersion: v1 -kind: Secret -metadata: { name: insight-oidc, namespace: insight } -type: Opaque -stringData: - APP__gears__oidc-authn-plugin__config__issuer_url: "" - APP__gears__oidc-authn-plugin__config__audience: "" - APP__gears__oidc-authn-plugin__config__jwks_url: "" - APP__gears__auth-info__config__issuer_url: "" - APP__gears__auth-info__config__client_id: "" - APP__gears__auth-info__config__redirect_uri: "/callback" - APP__gears__auth-info__config__scopes: "openid profile email" +```sh +openssl ecparam -name prime256v1 -genkey -noout | openssl pkcs8 -topk8 -nocrypt -out current.pem +kubectl -n insight create secret generic insight-authenticator-signing-keys --from-file=current.pem ``` -The `APP__gears__...` key names follow the app's environment-variable-style config convention (double underscores separate config sections); leave them exactly as shown and only fill the values. +During a key rotation, add a `previous.pem` (the outgoing key) alongside the new `current.pem` for at least the JWT TTL plus downstream JWKS-cache age (roughly 65 minutes), then roll the authenticator pods: -| Placeholder | What it should be | -|-------------|--------------------| -| `` | Your IdP's issuer URL. Its `/.well-known/openid-configuration` document must resolve | -| `` | The `jwks_uri` value from that same well-known discovery document | -| `` | Your OIDC client / application registration ID | -| `` | The Router's public URL, the same value as `https://` from Step 1 | - -Example issuer URLs by provider: - -| IdP | Example `` | -|-----|---------------------------| -| Entra | `https://login.microsoftonline.com//v2.0` | -| Okta | `https://.okta.com/oauth2/default` | -| Auth0 | `https://.auth0.com/` | -| Keycloak | `https:///realms/` | -| Dex | `https:///dex` | +```sh +kubectl -n insight create secret generic insight-authenticator-signing-keys \ + --from-file=current.pem --from-file=previous.pem \ + --dry-run=client -o yaml | kubectl apply -f - +``` ## Step 3 — Create namespace and apply secrets -Create the `insight` namespace and apply both secret files: +Create the `insight` namespace and apply the secret files: ```sh # create the namespace and apply all secrets @@ -249,10 +298,20 @@ kubectl create namespace insight kubectl -n insight apply -f secrets/ # verify -kubectl -n insight get secret insight-db-creds insight-oidc # expect 4 keys / 7 keys +kubectl -n insight get secret insight-db-creds insight-authenticator-signing-keys # expect 4 keys / 1-2 keys (current.pem [+ previous.pem]) ``` -The Analytic service also needs Airbyte's own auth credentials (`airbyte-auth-secrets`, created by the Airbyte chart in the infrastructure namespace) to talk to the Airbyte API. Do **not** copy that Secret into `insight` — a copy silently breaks when Airbyte regenerates its credentials on reinstall. Instead, set `airbyte.namespace` in your values (Step 1); the chart reads the Secret from that namespace at run time and installs a Role/RoleBinding there that grants its jobs `get` on that one Secret. +The Analytics service also needs Airbyte's own auth credentials to talk to the Airbyte API. Mirror that Secret from your infrastructure namespace into `insight`: + +```sh +# mirror the Airbyte auth secret from your infra namespace +NS_INFRA= +kubectl -n $NS_INFRA get secret airbyte-auth-secrets -o json \ + | jq 'del(.metadata.uid,.metadata.resourceVersion,.metadata.creationTimestamp,.metadata.ownerReferences,.metadata.annotations,.metadata.labels) | .metadata.namespace="insight"' \ + | kubectl -n insight apply -f - +``` + +The `jq` step strips the original Secret's identity fields (UID, resource version, owner references) and retargets it to the `insight` namespace, so Kubernetes accepts it as a new object. ## Step 4 — Install with Helm @@ -265,20 +324,29 @@ helm upgrade --install insight oci://ghcr.io/constructorfabric/charts/insight \ Omit `--version` to install the latest published chart, or add `--version ` to pin a specific release. `--wait --timeout 15m` blocks the command until all resources report ready, or until 15 minutes pass, whichever comes first — this gives you a clear pass/fail signal instead of a detached background rollout. +This command also runs a post-install/post-upgrade Helm hook Job, `insight-clickhouse-migrate`, which applies the ClickHouse gold-view migrations (`src/ingestion/scripts/migrations/*.sql`) against your external ClickHouse using `ingestion.toolboxImage`. It runs on **every** `helm upgrade`, not just the first install — this is gated by `clickhouse.runMigrations` (default `true`). `helm upgrade` blocks on this Job the same way it blocks on any other resource; a failing migration fails the whole upgrade. The migration script drops-and-recreates every gold object on each run, so a genuine migrate-Job failure usually points to a schema/data problem in the referenced Bronze/Silver tables, not a stale-object conflict. + ## Step 5 — Verify the install -Confirm all three service pods are running: +Confirm all pods are running (Identity only appears when `identity.deploy: true`; fakeidp/keycloak only when their `deploy` flag is set): ```sh kubectl -n insight get pods - # expect: insight-api-gateway, insight-analytics, insight-identity, insight-frontend (all Running) + # expect: insight-gateway, insight-authenticator, insight-analytics, insight-identity, insight-frontend (all Running) +``` + +Confirm the chart composed the per-service config Secrets from `insight-db-creds`: + +```sh +kubectl -n insight get secret insight-analytics-config insight-authenticator-config insight-identity-config + # chart composed these from insight-db-creds (insight-identity-config only exists when identity.deploy=true) ``` -Confirm the chart composed the two config Secrets from `insight-db-creds`: +Inspect the ClickHouse gold-view migration hook Job and confirm it completed: ```sh -kubectl -n insight get secret insight-analytics-config insight-identity-config - # chart composed these from insight-db-creds +kubectl -n insight get jobs -l app.kubernetes.io/component=clickhouse-migrate +kubectl -n insight logs job/insight-clickhouse-migrate ``` Confirm the reconcile loop's scheduled workflow exists — this is the job that discovers connector Secrets and provisions Airbyte sources and connections automatically: @@ -319,16 +387,16 @@ stringData: jira_api_token: "ATATT-CHANGE_ME" ``` -### The 26 available connectors +### The 25 available connectors Replace `CHANGE_ME` (and any other placeholder) values in whichever connector files you need, under `connectors/`: -`jira`, `slack`, `github-v2`, `gitlab`, `m365`, `salesforce`, `zoom`, `confluence`, `youtrack`, `zendesk`, `workday`, `bamboohr`, `ms-entra`, `figma`, `outline`, `hubspot`, `cursor`, `openai`, `chatgpt-team`, `claude-team`, `claude-admin`, `claude-enterprise`, `github-copilot`, `bitbucket-cloud`, `bitbucket-server`, `zulip-proxy`. +`jira`, `slack`, `github-v2`, `gitlab`, `m365`, `salesforce`, `zoom`, `confluence`, `youtrack`, `zendesk`, `workday`, `bamboohr`, `ms-entra`, `figma`, `outline`, `hubspot`, `cursor`, `openai`, `chatgpt-team`, `claude-team`, `claude-admin`, `claude-enterprise`, `github-copilot`, `bitbucket-cloud`, `zulip-proxy`. Apply all of them at once, or one at a time: ```sh -kubectl -n insight apply -f connectors/ # all 26 connectors at once +kubectl -n insight apply -f connectors/ # all 25 connectors at once # or one at a time: kubectl -n insight apply -f connectors/jira.yaml ``` @@ -341,10 +409,9 @@ The reconcile loop scans the `insight` namespace about every 15 minutes. On a ne Each block below is a complete, copy-paste-ready Secret for one connector. Fill in the `CHANGE_ME` (and any other placeholder) values, save it under `connectors/.yaml`, and apply it as shown above. -Two things to know before you copy these: +One thing to know before you copy these: - Connectors marked ⚠ are CDK connectors (built on Airbyte's Connector Development Kit). They bake their own `url_base` into the connector image, so they cannot be repointed at a mock or self-hosted endpoint. -- `bitbucket-server` intentionally has an empty `stringData: {}` — its URL and credentials come from the connector manifest, not from this Secret. #### AI & coding assistants @@ -498,19 +565,6 @@ stringData: bitbucket_workspaces: "workspace-a,workspace-b" ``` -```yaml -# required_fields: [] — URL/creds come from the connector manifest, not this Secret -apiVersion: v1 -kind: Secret -metadata: - name: insight-bitbucket-server-main - namespace: insight - labels: { app.kubernetes.io/part-of: insight } - annotations: { insight.cyberfabric.com/connector: bitbucket-server, insight.cyberfabric.com/source-id: bitbucket-server-main } -type: Opaque -stringData: {} -``` - #### Issue tracking & docs ```yaml @@ -717,6 +771,7 @@ stringData: ``` ```yaml +# ⚠ CDK connector; baked url_base (api.hubapi.com) apiVersion: v1 kind: Secret metadata: @@ -750,9 +805,12 @@ stringData: | Problem | What to check | |---------|-----------------| | `insight-analytics` / `insight-identity` stuck in `CreateContainerConfigError` | The chart could not compose the `*-config` Secrets. Confirm `insight-db-creds` has all four keys and carries **no** `app.kubernetes.io/managed-by: Helm` label: `kubectl -n insight get secret insight-db-creds -o yaml \| grep managed-by` should return nothing | -| Dashboards show "no peer data" (the benchmark/comparison panel is empty) | After Gold-layer data has loaded, restart Analytic: `kubectl -n insight rollout restart deploy/insight-analytics` | -| Login breaks after changing the host | Update `insight-oidc` (issuer and redirect URI), then restart the Router: `kubectl -n insight rollout restart deploy/insight-api-gateway` | -| Connectors are not syncing | Check reconcile logs: `kubectl -n insight logs deploy/insight-analytics \| grep -i reconcile`. Confirm `airbyte.namespace` in your values points at the namespace that holds `airbyte-auth-secrets`, and that the `*-airbyte-auth-reader` RoleBinding exists there | +| `helm install`/`upgrade` fails with `signingKeysSecret is required` or the authenticator pod won't mount its keys | The Secret named in `authenticator.signingKeysSecret` (default `insight-authenticator-signing-keys`) doesn't exist or is missing `current.pem`. Create it as shown in Step 2 before installing | +| `helm install`/`upgrade` fails with `tlsDiscovery.issuerRef.name is required` or the `insight-authenticator-authn-tls` Certificate never turns `Ready` | cert-manager isn't installed, or the `ClusterIssuer` named in `authenticator.tlsDiscovery.issuerRef.name` (default `local-ca`) doesn't exist. Confirm with `kubectl get clusterissuer local-ca` and `kubectl -n insight describe certificate insight-authenticator-authn-tls` | +| `helm install`/`upgrade` fails with `authenticator.oidc.issuerUrl is required` / `redirectUri is required` | Both fields are mandatory (`charts/insight/templates/secrets.yaml` wraps them in `required`). Set real values, or for local/dev only, set `fakeidp.deploy: true` and point `issuerUrl` at the in-cluster fakeidp — never disable auth | +| Dashboards show "no peer data" (the benchmark/comparison panel is empty) | After Gold-layer data has loaded, restart Analytics: `kubectl -n insight rollout restart deploy/insight-analytics` | +| Login breaks after changing the host | Update `authenticator.oidc.redirectUri` (and `frontend.oidc.issuer`/`clientId` if the IdP changed) in values, `helm upgrade`, then restart the gateway: `kubectl -n insight rollout restart deploy/insight-gateway` | +| Connectors are not syncing | Check reconcile logs: `kubectl -n insight logs deploy/insight-analytics \| grep -i reconcile`. Confirm `airbyte-auth-secrets` was mirrored into the `insight` namespace (Step 3) | ## Appendix — Reference @@ -760,45 +818,45 @@ stringData: | Placeholder | Field(s) | Notes | |-------------|----------|-------| -| `` | `global.tenantDefaultId`, `ingestion.reconcile.tenantId`, `identity.tenantDefaultId` | Must be identical across all three | -| `` | `clickhouse.host` | `deploy: false`; port fixed at `8123` in the file | -| `` | `mariadb.host` | `deploy: false`; port fixed at `3306` | -| `` | `redis.host` | `deploy: false`; port fixed at `6379` | -| `` | `redpanda.brokers` | `deploy: false`; include port, e.g. `:9093` | -| `` | `airbyte.namespace` | Namespace of the Airbyte release; `""` = app namespace | +| `` | `global.tenantDefaultId`, `ingestion.reconcile.tenantId` | Must be identical across both | +| `` | `clickhouse.host` | Always external; port fixed at `8123` in the file | +| `` | `mariadb.host` | Always external; port fixed at `3306` | +| `` | `redis.host` | Always external; port fixed at `6379` | +| `` | `redpanda.brokers` | Always external; include port, e.g. `:9093` | +| `` | `ingestion.toolboxImage` | Drives the ingestion WorkflowTemplates and the ClickHouse gold-view migrate Job | +| `` | `airbyte.apiUrl` | e.g. `http://host:8001` | | `` | `ingestion.reconcile.argoInstanceId` | Your Argo controller's instance ID | -| `` | `apiGateway.ingress.host`, `frontend.ingress.host` | Public FQDN, shared by Router and Frontend (`/api/*` → Router, `/*` → UI) | -| `` | `apiGateway.ingress.tls.secretName` | Kubernetes TLS Secret name | +| `` | `gateway.image.tag`, `authenticator.image.tag`, `analytics.image.tag`, `identity.image.tag`, `frontend.image.tag` | All five are optional — each falls back to that subchart's `Chart.yaml` appVersion (pinned by the release pipeline). Set them explicitly (recommended) so every service lands on the exact same product build; leaving them blank is safe only when all five subcharts' appVersion are in lockstep in the chart release you install | +| `` | `gateway.ingress.host`, `frontend.ingress.host` | Public FQDN, shared by the Gateway and Frontend (`/*` → UI, `/api/*` → Gateway routes to Analytics/Identity) | +| `` | `gateway.ingress.tls.secretName` | Kubernetes TLS Secret name | +| `` | `authenticator.oidc.issuerUrl`, `frontend.oidc.issuer` | Your IdP's issuer URL | +| `` / `` | `authenticator.oidc.clientId`/`clientSecret`, `frontend.oidc.clientId` | Your OIDC client / application registration credentials | Other notable (non-placeholder) settings in this file: - `credentials.deploymentMode: helm` and `credentials.autoGenerate: true` — this enables the "bring your own" credentials path, where the chart keeps a labelless `insight-db-creds` Secret instead of generating random passwords. - `identity.deploy: true` — required override; the chart's own default is `false`. -- `apiGateway.authDisabled: false` — keep `false`; `true` disables all auth and is local-dev only. +- `authenticator.tlsDiscovery.issuerRef.name: local-ca` — the cert-manager `ClusterIssuer` name the JWKS-discovery Certificate is issued from; override to match your cluster's issuer. +- There is no auth-off toggle anywhere in this chart. `authenticator.oidc.issuerUrl` and `authenticator.oidc.redirectUri` are hard `required` fields — the simplest no-real-IdP path is `fakeidp.deploy: true`, local/dev only; `keycloak.deploy: true` is a heavier bundled alternative (not documented here). ### secrets/insight-db-creds.yaml keys | Key | Meaning | Consumed by | |-----|---------|--------------| -| `clickhouse-password` | ClickHouse admin password | Analytic | -| `mariadb-password` | MariaDB app-user password | Analytic + Identity | +| `clickhouse-password` | ClickHouse admin password | Analytics | +| `mariadb-password` | MariaDB app-user password | Analytics + Identity | | `mariadb-root-password` | MariaDB root password, used by the identity-DB init hook | Identity | -| `redis-password` | Redis password | Analytic | +| `redis-password` | Redis password | Analytics + Authenticator | Recall: this Secret must never carry an `app.kubernetes.io/managed-by: Helm` label. -### secrets/insight-oidc.yaml keys +### secrets/insight-authenticator-signing-keys.yaml keys | Key | Meaning | |-----|---------| -| `APP__gears__oidc-authn-plugin__config__issuer_url` | IdP issuer URL | -| `APP__gears__oidc-authn-plugin__config__audience` | OIDC client/application ID | -| `APP__gears__oidc-authn-plugin__config__jwks_url` | `jwks_uri` from the IdP discovery document | -| `APP__gears__auth-info__config__issuer_url` | IdP issuer URL (same value as above) | -| `APP__gears__auth-info__config__client_id` | OIDC client/application ID (same value as above) | -| `APP__gears__auth-info__config__redirect_uri` | `/callback` | -| `APP__gears__auth-info__config__scopes` | `openid profile email` | +| `current.pem` | The active ES256 (EC P-256) signing key, PKCS#8 PEM, unencrypted. Required. | +| `previous.pem` | The outgoing key during a rotation window. Optional; keep it alongside `current.pem` until the JWT TTL plus downstream JWKS-cache age has elapsed | ### values/umbrella.orbstack.yaml (local variant) -This file is a pre-filled variant of the values file for local development on OrbStack's bundled k3s cluster, with all infrastructure running in an `insight-infra` namespace. It sets a fixed tenant UUID, in-cluster DNS hosts, an empty (host-less) ingress that matches any `Host` header, and disabled TLS. Use it only as a reference for local testing — do not reuse its host-less ingress or disabled TLS settings on a shared or production cluster. +This file is a pre-filled variant of the values file for local development on OrbStack's bundled k3s cluster, with all infrastructure running in an `insight-infra` namespace. It sets a fixed tenant UUID, in-cluster DNS hosts, an empty (host-less) ingress that matches any `Host` header, disabled TLS, and points the authenticator at the in-cluster `fakeidp`/`keycloak` provider rather than a real IdP. Use it only as a reference for local testing — do not reuse its host-less ingress, disabled TLS, or fake-IdP settings on a shared or production cluster. From 3858424eb1808fd0a64fc4d3170051be8466f991 Mon Sep 17 00:00:00 2001 From: Konstantin Tursunov Date: Sat, 25 Jul 2026 19:17:37 +0800 Subject: [PATCH 02/14] docs(deploy): split connector config out of HELM_DEPLOY.md into CONNECTORS.md MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit Deploying the app and configuring data connectors are distinct operations, so separate the docs: - New deploy/CONNECTORS.md: connector Secret anatomy, the 25-connector catalog (a copy-paste example Secret per connector, CDK/url_base markers intact), reconcile-loop behavior, and a connector-syncing Troubleshooting section. - HELM_DEPLOY.md: Step 6 reduced to a short cross-linked pointer; Overview/Prerequisites updated to send connector examples to CONNECTORS.md; connector-syncing troubleshooting relocated there. - Fix a pre-existing wrong reconcile-log command: reconcile runs as the insight-reconcile-loop Argo CronWorkflow, not the analytics Deployment — point at its Workflow pods. - Mark github-copilot github_start_date as optional (spec.json requires only github_token + github_org). Signed-off-by: Konstantin Tursunov --- deploy/CONNECTORS.md | 472 ++++++++++++++++++++++++++++++++++++++++++ deploy/HELM_DEPLOY.md | 450 +--------------------------------------- 2 files changed, 478 insertions(+), 444 deletions(-) create mode 100644 deploy/CONNECTORS.md diff --git a/deploy/CONNECTORS.md b/deploy/CONNECTORS.md new file mode 100644 index 000000000..7492de93f --- /dev/null +++ b/deploy/CONNECTORS.md @@ -0,0 +1,472 @@ +# Configuring Insight Connectors + +Connectors pull data from your tools — Jira issues, Slack messages, GitHub pull requests, and so on — into ClickHouse Bronze via Airbyte. Each connector is a single Kubernetes Secret; the reconcile loop discovers it automatically and provisions the matching Airbyte source and connection, no further steps required. + +## Prerequisites + +- A completed Insight install per the deployment runbook: [HELM_DEPLOY.md](./HELM_DEPLOY.md). +- The `insight-reconcile-loop` CronWorkflow present in the `insight` namespace (installed as part of that runbook). +- The `airbyte-auth-secrets` Secret mirrored into the `insight` namespace — done in Step 3 of the deployment runbook. + +## Contents + + + +- [Prerequisites](#prerequisites) +- [Contents](#contents) +- [Anatomy of a connector Secret](#anatomy-of-a-connector-secret) +- [The 25 available connectors](#the-25-available-connectors) +- [Example Secret for every connector](#example-secret-for-every-connector) + - [AI & coding assistants](#ai--coding-assistants) + - [Source control & CI](#source-control--ci) + - [Issue tracking & docs](#issue-tracking--docs) + - [Communication & meetings](#communication--meetings) + - [HR & identity](#hr--identity) + - [CRM & support](#crm--support) +- [Troubleshooting](#troubleshooting) + + + +## Anatomy of a connector Secret + +Every connector Secret needs three things for the reconcile loop to discover and wire it up: + +- **A label**, `app.kubernetes.io/part-of: insight` — the selector the reconcile loop uses to find connector Secrets. +- **Two annotations**: `insight.cyberfabric.com/connector: ` identifies which connector definition to use, and `insight.cyberfabric.com/source-id: ` names this specific source instance (the convention is `-main`). +- **`stringData`** holding the connector's required fields — credentials, base URLs, and similar settings specific to that tool. + +For example, the Jira connector Secret (`connectors/jira.yaml`) looks like this: + +```yaml +apiVersion: v1 +kind: Secret +metadata: + name: insight-jira-main + namespace: insight + labels: { app.kubernetes.io/part-of: insight } + annotations: { insight.cyberfabric.com/connector: jira, insight.cyberfabric.com/source-id: jira-main } +type: Opaque +stringData: + jira_instance_url: "https://your-org.atlassian.net" + jira_email: "svc@your-org.com" + jira_api_token: "ATATT-CHANGE_ME" +``` + +## The 25 available connectors + +Replace `CHANGE_ME` (and any other placeholder) values in whichever connector files you need, under `connectors/`: + +`jira`, `slack`, `github-v2`, `gitlab`, `m365`, `salesforce`, `zoom`, `confluence`, `youtrack`, `zendesk`, `workday`, `bamboohr`, `ms-entra`, `figma`, `outline`, `hubspot`, `cursor`, `openai`, `chatgpt-team`, `claude-team`, `claude-admin`, `claude-enterprise`, `github-copilot`, `bitbucket-cloud`, `zulip-proxy`. + +Apply all of them at once, or one at a time: + +```sh +kubectl -n insight apply -f connectors/ # all 25 connectors at once +# or one at a time: +kubectl -n insight apply -f connectors/jira.yaml +``` + +You only need to create Secrets for the tools you actually use — an unused connector file can be left unfilled and simply not applied. + +The reconcile loop scans the `insight` namespace about every 15 minutes. On a new or changed connector Secret, it provisions the matching Airbyte source and connection and starts syncing into Bronze automatically — no further steps once the Secret is applied and filled in correctly. + +## Example Secret for every connector + +Each block below is a complete, copy-paste-ready Secret for one connector. Fill in the `CHANGE_ME` (and any other placeholder) values, save it under `connectors/.yaml`, and apply it as shown above. + +One thing to know before you copy these: + +- Connectors marked ⚠ are CDK connectors (built on Airbyte's Connector Development Kit). They bake their own `url_base` into the connector image, so they cannot be repointed at a mock or self-hosted endpoint. + +### AI & coding assistants + +```yaml +apiVersion: v1 +kind: Secret +metadata: + name: insight-chatgpt-team-main + namespace: insight + labels: { app.kubernetes.io/part-of: insight } + annotations: { insight.cyberfabric.com/connector: chatgpt-team, insight.cyberfabric.com/source-id: chatgpt-team-main } +type: Opaque +stringData: + chatgpt_account_id: "CHANGE_ME" + proxy_url: "CHANGE_ME" # your ChatGPT admin-proxy base URL + proxy_auth_token: "CHANGE_ME" + # chatgpt_org_id: "CHANGE_ME" # optional (subscription streams) + # start_date: "2026-01-01" # optional +``` + +```yaml +apiVersion: v1 +kind: Secret +metadata: + name: insight-claude-team-main + namespace: insight + labels: { app.kubernetes.io/part-of: insight } + annotations: { insight.cyberfabric.com/connector: claude-team, insight.cyberfabric.com/source-id: claude-team-main } +type: Opaque +stringData: + claude_org_id: "CHANGE_ME" + proxy_url: "CHANGE_ME" + proxy_auth_token: "CHANGE_ME" +``` + +```yaml +apiVersion: v1 +kind: Secret +metadata: + name: insight-claude-admin-main + namespace: insight + labels: { app.kubernetes.io/part-of: insight } + annotations: { insight.cyberfabric.com/connector: claude-admin, insight.cyberfabric.com/source-id: claude-admin-main } +type: Opaque +stringData: + admin_api_key: "CHANGE_ME" +``` + +```yaml +apiVersion: v1 +kind: Secret +metadata: + name: insight-claude-enterprise-main + namespace: insight + labels: { app.kubernetes.io/part-of: insight } + annotations: { insight.cyberfabric.com/connector: claude-enterprise, insight.cyberfabric.com/source-id: claude-enterprise-main } +type: Opaque +stringData: + analytics_api_key: "CHANGE_ME" +``` + +```yaml +# ⚠ CDK connector; org-scoped GitHub PAT +apiVersion: v1 +kind: Secret +metadata: + name: insight-github-copilot-main + namespace: insight + labels: { app.kubernetes.io/part-of: insight } + annotations: { insight.cyberfabric.com/connector: github-copilot, insight.cyberfabric.com/source-id: github-copilot-main } +type: Opaque +stringData: + github_token: "CHANGE_ME" # PAT with Copilot org metrics scope + github_org: "CHANGE_ME" + # github_start_date: "2026-01-01" # optional; default = 90 days ago +``` + +```yaml +apiVersion: v1 +kind: Secret +metadata: + name: insight-cursor-main + namespace: insight + labels: { app.kubernetes.io/part-of: insight } + annotations: { insight.cyberfabric.com/connector: cursor, insight.cyberfabric.com/source-id: cursor-main } +type: Opaque +stringData: + cursor_api_key: "CHANGE_ME" +``` + +```yaml +apiVersion: v1 +kind: Secret +metadata: + name: insight-openai-main + namespace: insight + labels: { app.kubernetes.io/part-of: insight } + annotations: { insight.cyberfabric.com/connector: openai, insight.cyberfabric.com/source-id: openai-main } +type: Opaque +stringData: + openai_admin_api_key: "CHANGE_ME" + openai_start_date: "2026-01-01" +``` + +### Source control & CI + +```yaml +# ⚠ CDK connector; supersedes `github` +apiVersion: v1 +kind: Secret +metadata: + name: insight-github-v2-main + namespace: insight + labels: { app.kubernetes.io/part-of: insight } + annotations: { insight.cyberfabric.com/connector: github-v2, insight.cyberfabric.com/source-id: github-v2-main } +type: Opaque +stringData: + github_token: "CHANGE_ME" + github_organizations: "org-a,org-b" + github_start_date: "2026-01-01" + github_skip_archived: "true" + github_skip_forks: "true" +``` + +```yaml +apiVersion: v1 +kind: Secret +metadata: + name: insight-gitlab-main + namespace: insight + labels: { app.kubernetes.io/part-of: insight } + annotations: { insight.cyberfabric.com/connector: gitlab, insight.cyberfabric.com/source-id: gitlab-main } +type: Opaque +stringData: + gitlab_url: "https://gitlab.com" + gitlab_token: "CHANGE_ME" +``` + +```yaml +# ⚠ CDK connector; baked url_base +apiVersion: v1 +kind: Secret +metadata: + name: insight-bitbucket-cloud-main + namespace: insight + labels: { app.kubernetes.io/part-of: insight } + annotations: { insight.cyberfabric.com/connector: bitbucket-cloud, insight.cyberfabric.com/source-id: bitbucket-cloud-main } +type: Opaque +stringData: + bitbucket_token: "CHANGE_ME" # Atlassian ATCTT access token (NOT an ATATT API token) + bitbucket_workspaces: "workspace-a,workspace-b" +``` + +### Issue tracking & docs + +```yaml +apiVersion: v1 +kind: Secret +metadata: + name: insight-jira-main + namespace: insight + labels: { app.kubernetes.io/part-of: insight } + annotations: { insight.cyberfabric.com/connector: jira, insight.cyberfabric.com/source-id: jira-main } +type: Opaque +stringData: + jira_instance_url: "https://your-org.atlassian.net" + jira_email: "svc@your-org.com" + jira_api_token: "ATATT-CHANGE_ME" +``` + +```yaml +apiVersion: v1 +kind: Secret +metadata: + name: insight-youtrack-main + namespace: insight + labels: { app.kubernetes.io/part-of: insight } + annotations: { insight.cyberfabric.com/connector: youtrack, insight.cyberfabric.com/source-id: youtrack-main } +type: Opaque +stringData: + youtrack_base_url: "https://your-org.youtrack.cloud/api" + youtrack_token: "perm-CHANGE_ME" + # youtrack_page_size: "100" # optional +``` + +```yaml +apiVersion: v1 +kind: Secret +metadata: + name: insight-confluence-main + namespace: insight + labels: { app.kubernetes.io/part-of: insight } + annotations: { insight.cyberfabric.com/connector: confluence, insight.cyberfabric.com/source-id: confluence-main } +type: Opaque +stringData: + confluence_instance_url: "https://your-org.atlassian.net/wiki" + confluence_email: "svc@your-org.com" + confluence_api_token: "ATATT-CHANGE_ME" + confluence_start_date: "2026-01-01" +``` + +```yaml +apiVersion: v1 +kind: Secret +metadata: + name: insight-outline-main + namespace: insight + labels: { app.kubernetes.io/part-of: insight } + annotations: { insight.cyberfabric.com/connector: outline, insight.cyberfabric.com/source-id: outline-main } +type: Opaque +stringData: + outline_instance_url: "https://your-outline-host" + outline_api_token: "CHANGE_ME" + outline_start_date: "2026-01-01" +``` + +```yaml +apiVersion: v1 +kind: Secret +metadata: + name: insight-figma-main + namespace: insight + labels: { app.kubernetes.io/part-of: insight } + annotations: { insight.cyberfabric.com/connector: figma, insight.cyberfabric.com/source-id: figma-main } +type: Opaque +stringData: + figma_token: "figd_CHANGE_ME" + figma_team_ids: "1234567890,0987654321" + figma_start_date: "2026-01-01" +``` + +### Communication & meetings + +```yaml +apiVersion: v1 +kind: Secret +metadata: + name: insight-slack-main + namespace: insight + labels: { app.kubernetes.io/part-of: insight } + annotations: { insight.cyberfabric.com/connector: slack, insight.cyberfabric.com/source-id: slack-main } +type: Opaque +stringData: + slack_bot_token: "xoxb-CHANGE_ME" + slack_start_date: "2026-01-01" +``` + +```yaml +apiVersion: v1 +kind: Secret +metadata: + name: insight-zoom-main + namespace: insight + labels: { app.kubernetes.io/part-of: insight } + annotations: { insight.cyberfabric.com/connector: zoom, insight.cyberfabric.com/source-id: zoom-main } +type: Opaque +stringData: + zoom_account_id: "CHANGE_ME" + zoom_client_id: "CHANGE_ME" + zoom_client_secret: "CHANGE_ME" +``` + +```yaml +apiVersion: v1 +kind: Secret +metadata: + name: insight-m365-main + namespace: insight + labels: { app.kubernetes.io/part-of: insight } + annotations: { insight.cyberfabric.com/connector: m365, insight.cyberfabric.com/source-id: m365-main } +type: Opaque +stringData: + azure_tenant_id: "CHANGE_ME" + azure_client_id: "CHANGE_ME" + azure_client_secret: "CHANGE_ME" +``` + +```yaml +apiVersion: v1 +kind: Secret +metadata: + name: insight-zulip-proxy-main + namespace: insight + labels: { app.kubernetes.io/part-of: insight } + annotations: { insight.cyberfabric.com/connector: zulip-proxy, insight.cyberfabric.com/source-id: zulip-proxy-main } +type: Opaque +stringData: + zulip_proxy_base_url: "CHANGE_ME" + zulip_proxy_api_key: "CHANGE_ME" + zulip_proxy_start_date: "2026-01-01" +``` + +### HR & identity + +```yaml +apiVersion: v1 +kind: Secret +metadata: + name: insight-bamboohr-main + namespace: insight + labels: { app.kubernetes.io/part-of: insight } + annotations: { insight.cyberfabric.com/connector: bamboohr, insight.cyberfabric.com/source-id: bamboohr-main } +type: Opaque +stringData: + bamboohr_api_key: "CHANGE_ME" + bamboohr_domain: "your-company" # the in .bamboohr.com +``` + +```yaml +apiVersion: v1 +kind: Secret +metadata: + name: insight-workday-main + namespace: insight + labels: { app.kubernetes.io/part-of: insight } + annotations: { insight.cyberfabric.com/connector: workday, insight.cyberfabric.com/source-id: workday-main } +type: Opaque +stringData: + workday_base_url: "https://wd2-impl-services1.workday.com" + workday_isu_username: "CHANGE_ME" + workday_isu_password: "CHANGE_ME" + workday_workers_report_path: "/ccx/service/customreport2/.../Workers" + workday_leave_report_path: "/ccx/service/customreport2/.../Leave" +``` + +```yaml +apiVersion: v1 +kind: Secret +metadata: + name: insight-ms-entra-main + namespace: insight + labels: { app.kubernetes.io/part-of: insight } + annotations: { insight.cyberfabric.com/connector: ms-entra, insight.cyberfabric.com/source-id: ms-entra-main } +type: Opaque +stringData: + azure_tenant_id: "CHANGE_ME" + azure_client_id: "CHANGE_ME" + azure_client_secret: "CHANGE_ME" +``` + +### CRM & support + +```yaml +apiVersion: v1 +kind: Secret +metadata: + name: insight-salesforce-main + namespace: insight + labels: { app.kubernetes.io/part-of: insight } + annotations: { insight.cyberfabric.com/connector: salesforce, insight.cyberfabric.com/source-id: salesforce-main } +type: Opaque +stringData: + salesforce_instance_url: "https://your-org.my.salesforce.com" + salesforce_client_id: "CHANGE_ME" + salesforce_client_secret: "CHANGE_ME" + salesforce_start_date: "2026-01-01" +``` + +```yaml +# ⚠ CDK connector; baked url_base (api.hubapi.com) +apiVersion: v1 +kind: Secret +metadata: + name: insight-hubspot-main + namespace: insight + labels: { app.kubernetes.io/part-of: insight } + annotations: { insight.cyberfabric.com/connector: hubspot, insight.cyberfabric.com/source-id: hubspot-main } +type: Opaque +stringData: + hubspot_access_token: "CHANGE_ME" +``` + +```yaml +apiVersion: v1 +kind: Secret +metadata: + name: insight-zendesk-main + namespace: insight + labels: { app.kubernetes.io/part-of: insight } + annotations: { insight.cyberfabric.com/connector: zendesk, insight.cyberfabric.com/source-id: zendesk-main } +type: Opaque +stringData: + zendesk_subdomain: "your-subdomain" # .zendesk.com + zendesk_email: "agent@your-org.com" + zendesk_api_token: "CHANGE_ME" + # start_date: "2026-01-01" # optional +``` + +## Troubleshooting + +| Problem | What to check | +|---------|-----------------| +| Connectors are not syncing | Confirm `airbyte-auth-secrets` was mirrored into the `insight` namespace (Step 3 of the deployment runbook, [HELM_DEPLOY.md](./HELM_DEPLOY.md)). The reconcile loop runs as an Argo `CronWorkflow` named `insight-reconcile-loop` — **not** the analytics pod — so inspect the Workflow pods it spawns: `kubectl -n insight get pods -l workflows.argoproj.io/cron-workflow=insight-reconcile-loop`, then `kubectl -n insight logs ` (or `argo logs @latest -n insight` if the Argo CLI is available) | diff --git a/deploy/HELM_DEPLOY.md b/deploy/HELM_DEPLOY.md index ba92a55c7..cac24f26b 100644 --- a/deploy/HELM_DEPLOY.md +++ b/deploy/HELM_DEPLOY.md @@ -20,9 +20,6 @@ This runbook shows a platform or DevOps engineer how to install the Insight busi - [Step 4 — Install with Helm](#step-4--install-with-helm) - [Step 5 — Verify the install](#step-5--verify-the-install) - [Step 6 — Configure connectors (optional)](#step-6--configure-connectors-optional) - - [Anatomy of a connector Secret](#anatomy-of-a-connector-secret) - - [The 25 available connectors](#the-25-available-connectors) - - [Example Secret for every connector](#example-secret-for-every-connector) - [Troubleshooting](#troubleshooting) - [Appendix — Reference](#appendix--reference) - [values/umbrella.yaml placeholders](#valuesumbrellayaml-placeholders) @@ -56,7 +53,7 @@ flowchart LR AN -->|verifies JWT via JWKS
over cert-manager CA| AUTH ``` -This path assumes your data infrastructure (ClickHouse, MariaDB, Redis, Redpanda, Airbyte, Argo Workflows) already runs and is reachable from the cluster, in another namespace or external. The chart doesn't stand it up; it only wires the services to it. You supply one values file, secret files, and optionally one Secret per connector. No GitOps repo, CI, or auto-reconciliation — you run the commands yourself. +This path assumes your data infrastructure (ClickHouse, MariaDB, Redis, Redpanda, Airbyte, Argo Workflows) already runs and is reachable from the cluster, in another namespace or external. The chart doesn't stand it up; it only wires the services to it. You supply one values file, secret files, and optionally one Secret per connector (see [deploy/CONNECTORS.md](./CONNECTORS.md)). No GitOps repo, CI, or auto-reconciliation — you run the commands yourself. ## Prerequisites @@ -89,7 +86,7 @@ All six systems below must be deployed and reachable from the cluster before you | Airbyte | Runs the data connectors (Jira, Slack, GitHub, and so on) that load raw data into ClickHouse Bronze | | Argo Workflows | Runs the dbt transform workflows that turn Bronze into Silver and Gold, and runs the sync workflows Airbyte connections trigger | -Run all commands here from the directory holding your `values/`, `secrets/`, and `connectors/` files. This document is self-contained: it shows the full `values/umbrella.yaml` skeleton, the secret files, and an example Secret for every connector, so you can assemble all three directories from what follows. +Run all commands here from the directory holding your `values/` and `secrets/` files. This document shows the full `values/umbrella.yaml` skeleton and the secret files, so you can assemble both directories from what follows. Connector configuration (the `connectors/` directory) is a separate, later step — see [deploy/CONNECTORS.md](./CONNECTORS.md). ## Step 1 — Configure values/umbrella.yaml @@ -360,445 +357,9 @@ Finally, open `https://` in a browser (the host you set in Step 1) and con ## Step 6 — Configure connectors (optional) -Connectors pull data from your tools — Jira issues, Slack messages, GitHub pull requests, and so on. Each is one Kubernetes Secret that both configures and enables a single Airbyte data source. Fill in the ones you need, then apply them. +Configuring connectors is a separate operation from installing the app, done once the app is up and running. There are 25 available connectors, each a single Kubernetes Secret that the `insight-reconcile-loop` CronWorkflow discovers and auto-provisions as an Airbyte source — no further steps once it's applied and filled in correctly. -### Anatomy of a connector Secret - -Every connector Secret needs three things for the reconcile loop to discover and wire it up: - -- **A label**, `app.kubernetes.io/part-of: insight` — the selector the reconcile loop uses to find connector Secrets. -- **Two annotations**: `insight.cyberfabric.com/connector: ` identifies which connector definition to use, and `insight.cyberfabric.com/source-id: ` names this specific source instance (the convention is `-main`). -- **`stringData`** holding the connector's required fields — credentials, base URLs, and similar settings specific to that tool. - -For example, the Jira connector Secret (`connectors/jira.yaml`) looks like this: - -```yaml -apiVersion: v1 -kind: Secret -metadata: - name: insight-jira-main - namespace: insight - labels: { app.kubernetes.io/part-of: insight } - annotations: { insight.cyberfabric.com/connector: jira, insight.cyberfabric.com/source-id: jira-main } -type: Opaque -stringData: - jira_instance_url: "https://your-org.atlassian.net" - jira_email: "svc@your-org.com" - jira_api_token: "ATATT-CHANGE_ME" -``` - -### The 25 available connectors - -Replace `CHANGE_ME` (and any other placeholder) values in whichever connector files you need, under `connectors/`: - -`jira`, `slack`, `github-v2`, `gitlab`, `m365`, `salesforce`, `zoom`, `confluence`, `youtrack`, `zendesk`, `workday`, `bamboohr`, `ms-entra`, `figma`, `outline`, `hubspot`, `cursor`, `openai`, `chatgpt-team`, `claude-team`, `claude-admin`, `claude-enterprise`, `github-copilot`, `bitbucket-cloud`, `zulip-proxy`. - -Apply all of them at once, or one at a time: - -```sh -kubectl -n insight apply -f connectors/ # all 25 connectors at once -# or one at a time: -kubectl -n insight apply -f connectors/jira.yaml -``` - -You only need to create Secrets for the tools you actually use — an unused connector file can be left unfilled and simply not applied. - -The reconcile loop scans the `insight` namespace about every 15 minutes. On a new or changed connector Secret, it provisions the matching Airbyte source and connection and starts syncing into Bronze automatically — no further steps once the Secret is applied and filled in correctly. - -### Example Secret for every connector - -Each block below is a complete, copy-paste-ready Secret for one connector. Fill in the `CHANGE_ME` (and any other placeholder) values, save it under `connectors/.yaml`, and apply it as shown above. - -One thing to know before you copy these: - -- Connectors marked ⚠ are CDK connectors (built on Airbyte's Connector Development Kit). They bake their own `url_base` into the connector image, so they cannot be repointed at a mock or self-hosted endpoint. - -#### AI & coding assistants - -```yaml -apiVersion: v1 -kind: Secret -metadata: - name: insight-chatgpt-team-main - namespace: insight - labels: { app.kubernetes.io/part-of: insight } - annotations: { insight.cyberfabric.com/connector: chatgpt-team, insight.cyberfabric.com/source-id: chatgpt-team-main } -type: Opaque -stringData: - chatgpt_account_id: "CHANGE_ME" - proxy_url: "CHANGE_ME" # your ChatGPT admin-proxy base URL - proxy_auth_token: "CHANGE_ME" - # chatgpt_org_id: "CHANGE_ME" # optional (subscription streams) - # start_date: "2026-01-01" # optional -``` - -```yaml -apiVersion: v1 -kind: Secret -metadata: - name: insight-claude-team-main - namespace: insight - labels: { app.kubernetes.io/part-of: insight } - annotations: { insight.cyberfabric.com/connector: claude-team, insight.cyberfabric.com/source-id: claude-team-main } -type: Opaque -stringData: - claude_org_id: "CHANGE_ME" - proxy_url: "CHANGE_ME" - proxy_auth_token: "CHANGE_ME" -``` - -```yaml -apiVersion: v1 -kind: Secret -metadata: - name: insight-claude-admin-main - namespace: insight - labels: { app.kubernetes.io/part-of: insight } - annotations: { insight.cyberfabric.com/connector: claude-admin, insight.cyberfabric.com/source-id: claude-admin-main } -type: Opaque -stringData: - admin_api_key: "CHANGE_ME" -``` - -```yaml -apiVersion: v1 -kind: Secret -metadata: - name: insight-claude-enterprise-main - namespace: insight - labels: { app.kubernetes.io/part-of: insight } - annotations: { insight.cyberfabric.com/connector: claude-enterprise, insight.cyberfabric.com/source-id: claude-enterprise-main } -type: Opaque -stringData: - analytics_api_key: "CHANGE_ME" -``` - -```yaml -# ⚠ CDK connector; org-scoped GitHub PAT -apiVersion: v1 -kind: Secret -metadata: - name: insight-github-copilot-main - namespace: insight - labels: { app.kubernetes.io/part-of: insight } - annotations: { insight.cyberfabric.com/connector: github-copilot, insight.cyberfabric.com/source-id: github-copilot-main } -type: Opaque -stringData: - github_token: "CHANGE_ME" # PAT with Copilot org metrics scope - github_org: "CHANGE_ME" - github_start_date: "2026-01-01" -``` - -```yaml -apiVersion: v1 -kind: Secret -metadata: - name: insight-cursor-main - namespace: insight - labels: { app.kubernetes.io/part-of: insight } - annotations: { insight.cyberfabric.com/connector: cursor, insight.cyberfabric.com/source-id: cursor-main } -type: Opaque -stringData: - cursor_api_key: "CHANGE_ME" -``` - -```yaml -apiVersion: v1 -kind: Secret -metadata: - name: insight-openai-main - namespace: insight - labels: { app.kubernetes.io/part-of: insight } - annotations: { insight.cyberfabric.com/connector: openai, insight.cyberfabric.com/source-id: openai-main } -type: Opaque -stringData: - openai_admin_api_key: "CHANGE_ME" - openai_start_date: "2026-01-01" -``` - -#### Source control & CI - -```yaml -# ⚠ CDK connector; supersedes `github` -apiVersion: v1 -kind: Secret -metadata: - name: insight-github-v2-main - namespace: insight - labels: { app.kubernetes.io/part-of: insight } - annotations: { insight.cyberfabric.com/connector: github-v2, insight.cyberfabric.com/source-id: github-v2-main } -type: Opaque -stringData: - github_token: "CHANGE_ME" - github_organizations: "org-a,org-b" - github_start_date: "2026-01-01" - github_skip_archived: "true" - github_skip_forks: "true" -``` - -```yaml -apiVersion: v1 -kind: Secret -metadata: - name: insight-gitlab-main - namespace: insight - labels: { app.kubernetes.io/part-of: insight } - annotations: { insight.cyberfabric.com/connector: gitlab, insight.cyberfabric.com/source-id: gitlab-main } -type: Opaque -stringData: - gitlab_url: "https://gitlab.com" - gitlab_token: "CHANGE_ME" -``` - -```yaml -# ⚠ CDK connector; baked url_base -apiVersion: v1 -kind: Secret -metadata: - name: insight-bitbucket-cloud-main - namespace: insight - labels: { app.kubernetes.io/part-of: insight } - annotations: { insight.cyberfabric.com/connector: bitbucket-cloud, insight.cyberfabric.com/source-id: bitbucket-cloud-main } -type: Opaque -stringData: - bitbucket_token: "CHANGE_ME" # Atlassian ATCTT access token (NOT an ATATT API token) - bitbucket_workspaces: "workspace-a,workspace-b" -``` - -#### Issue tracking & docs - -```yaml -apiVersion: v1 -kind: Secret -metadata: - name: insight-jira-main - namespace: insight - labels: { app.kubernetes.io/part-of: insight } - annotations: { insight.cyberfabric.com/connector: jira, insight.cyberfabric.com/source-id: jira-main } -type: Opaque -stringData: - jira_instance_url: "https://your-org.atlassian.net" - jira_email: "svc@your-org.com" - jira_api_token: "ATATT-CHANGE_ME" -``` - -```yaml -apiVersion: v1 -kind: Secret -metadata: - name: insight-youtrack-main - namespace: insight - labels: { app.kubernetes.io/part-of: insight } - annotations: { insight.cyberfabric.com/connector: youtrack, insight.cyberfabric.com/source-id: youtrack-main } -type: Opaque -stringData: - youtrack_base_url: "https://your-org.youtrack.cloud/api" - youtrack_token: "perm-CHANGE_ME" - # youtrack_page_size: "100" # optional -``` - -```yaml -apiVersion: v1 -kind: Secret -metadata: - name: insight-confluence-main - namespace: insight - labels: { app.kubernetes.io/part-of: insight } - annotations: { insight.cyberfabric.com/connector: confluence, insight.cyberfabric.com/source-id: confluence-main } -type: Opaque -stringData: - confluence_instance_url: "https://your-org.atlassian.net/wiki" - confluence_email: "svc@your-org.com" - confluence_api_token: "ATATT-CHANGE_ME" - confluence_start_date: "2026-01-01" -``` - -```yaml -apiVersion: v1 -kind: Secret -metadata: - name: insight-outline-main - namespace: insight - labels: { app.kubernetes.io/part-of: insight } - annotations: { insight.cyberfabric.com/connector: outline, insight.cyberfabric.com/source-id: outline-main } -type: Opaque -stringData: - outline_instance_url: "https://your-outline-host" - outline_api_token: "CHANGE_ME" - outline_start_date: "2026-01-01" -``` - -```yaml -apiVersion: v1 -kind: Secret -metadata: - name: insight-figma-main - namespace: insight - labels: { app.kubernetes.io/part-of: insight } - annotations: { insight.cyberfabric.com/connector: figma, insight.cyberfabric.com/source-id: figma-main } -type: Opaque -stringData: - figma_token: "figd_CHANGE_ME" - figma_team_ids: "1234567890,0987654321" - figma_start_date: "2026-01-01" -``` - -#### Communication & meetings - -```yaml -apiVersion: v1 -kind: Secret -metadata: - name: insight-slack-main - namespace: insight - labels: { app.kubernetes.io/part-of: insight } - annotations: { insight.cyberfabric.com/connector: slack, insight.cyberfabric.com/source-id: slack-main } -type: Opaque -stringData: - slack_bot_token: "xoxb-CHANGE_ME" - slack_start_date: "2026-01-01" -``` - -```yaml -apiVersion: v1 -kind: Secret -metadata: - name: insight-zoom-main - namespace: insight - labels: { app.kubernetes.io/part-of: insight } - annotations: { insight.cyberfabric.com/connector: zoom, insight.cyberfabric.com/source-id: zoom-main } -type: Opaque -stringData: - zoom_account_id: "CHANGE_ME" - zoom_client_id: "CHANGE_ME" - zoom_client_secret: "CHANGE_ME" -``` - -```yaml -apiVersion: v1 -kind: Secret -metadata: - name: insight-m365-main - namespace: insight - labels: { app.kubernetes.io/part-of: insight } - annotations: { insight.cyberfabric.com/connector: m365, insight.cyberfabric.com/source-id: m365-main } -type: Opaque -stringData: - azure_tenant_id: "CHANGE_ME" - azure_client_id: "CHANGE_ME" - azure_client_secret: "CHANGE_ME" -``` - -```yaml -apiVersion: v1 -kind: Secret -metadata: - name: insight-zulip-proxy-main - namespace: insight - labels: { app.kubernetes.io/part-of: insight } - annotations: { insight.cyberfabric.com/connector: zulip-proxy, insight.cyberfabric.com/source-id: zulip-proxy-main } -type: Opaque -stringData: - zulip_proxy_base_url: "CHANGE_ME" - zulip_proxy_api_key: "CHANGE_ME" - zulip_proxy_start_date: "2026-01-01" -``` - -#### HR & identity - -```yaml -apiVersion: v1 -kind: Secret -metadata: - name: insight-bamboohr-main - namespace: insight - labels: { app.kubernetes.io/part-of: insight } - annotations: { insight.cyberfabric.com/connector: bamboohr, insight.cyberfabric.com/source-id: bamboohr-main } -type: Opaque -stringData: - bamboohr_api_key: "CHANGE_ME" - bamboohr_domain: "your-company" # the in .bamboohr.com -``` - -```yaml -apiVersion: v1 -kind: Secret -metadata: - name: insight-workday-main - namespace: insight - labels: { app.kubernetes.io/part-of: insight } - annotations: { insight.cyberfabric.com/connector: workday, insight.cyberfabric.com/source-id: workday-main } -type: Opaque -stringData: - workday_base_url: "https://wd2-impl-services1.workday.com" - workday_isu_username: "CHANGE_ME" - workday_isu_password: "CHANGE_ME" - workday_workers_report_path: "/ccx/service/customreport2/.../Workers" - workday_leave_report_path: "/ccx/service/customreport2/.../Leave" -``` - -```yaml -apiVersion: v1 -kind: Secret -metadata: - name: insight-ms-entra-main - namespace: insight - labels: { app.kubernetes.io/part-of: insight } - annotations: { insight.cyberfabric.com/connector: ms-entra, insight.cyberfabric.com/source-id: ms-entra-main } -type: Opaque -stringData: - azure_tenant_id: "CHANGE_ME" - azure_client_id: "CHANGE_ME" - azure_client_secret: "CHANGE_ME" -``` - -#### CRM & support - -```yaml -apiVersion: v1 -kind: Secret -metadata: - name: insight-salesforce-main - namespace: insight - labels: { app.kubernetes.io/part-of: insight } - annotations: { insight.cyberfabric.com/connector: salesforce, insight.cyberfabric.com/source-id: salesforce-main } -type: Opaque -stringData: - salesforce_instance_url: "https://your-org.my.salesforce.com" - salesforce_client_id: "CHANGE_ME" - salesforce_client_secret: "CHANGE_ME" - salesforce_start_date: "2026-01-01" -``` - -```yaml -# ⚠ CDK connector; baked url_base (api.hubapi.com) -apiVersion: v1 -kind: Secret -metadata: - name: insight-hubspot-main - namespace: insight - labels: { app.kubernetes.io/part-of: insight } - annotations: { insight.cyberfabric.com/connector: hubspot, insight.cyberfabric.com/source-id: hubspot-main } -type: Opaque -stringData: - hubspot_access_token: "CHANGE_ME" -``` - -```yaml -apiVersion: v1 -kind: Secret -metadata: - name: insight-zendesk-main - namespace: insight - labels: { app.kubernetes.io/part-of: insight } - annotations: { insight.cyberfabric.com/connector: zendesk, insight.cyberfabric.com/source-id: zendesk-main } -type: Opaque -stringData: - zendesk_subdomain: "your-subdomain" # .zendesk.com - zendesk_email: "agent@your-org.com" - zendesk_api_token: "CHANGE_ME" - # start_date: "2026-01-01" # optional -``` +See [deploy/CONNECTORS.md](./CONNECTORS.md) for the full list of connectors and a copy-paste-ready example Secret for each. ## Troubleshooting @@ -810,7 +371,8 @@ stringData: | `helm install`/`upgrade` fails with `authenticator.oidc.issuerUrl is required` / `redirectUri is required` | Both fields are mandatory (`charts/insight/templates/secrets.yaml` wraps them in `required`). Set real values, or for local/dev only, set `fakeidp.deploy: true` and point `issuerUrl` at the in-cluster fakeidp — never disable auth | | Dashboards show "no peer data" (the benchmark/comparison panel is empty) | After Gold-layer data has loaded, restart Analytics: `kubectl -n insight rollout restart deploy/insight-analytics` | | Login breaks after changing the host | Update `authenticator.oidc.redirectUri` (and `frontend.oidc.issuer`/`clientId` if the IdP changed) in values, `helm upgrade`, then restart the gateway: `kubectl -n insight rollout restart deploy/insight-gateway` | -| Connectors are not syncing | Check reconcile logs: `kubectl -n insight logs deploy/insight-analytics \| grep -i reconcile`. Confirm `airbyte-auth-secrets` was mirrored into the `insight` namespace (Step 3) | + +For connector-syncing problems, see the Troubleshooting section of [deploy/CONNECTORS.md](./CONNECTORS.md). ## Appendix — Reference From 3996e334267ab8cd3e50f40075e0b33202b73d44 Mon Sep 17 00:00:00 2001 From: Konstantin Tursunov Date: Mon, 27 Jul 2026 10:49:45 +0800 Subject: [PATCH 03/14] docs(deploy): add tenant-ID Step 0 to HELM_DEPLOY.md and tighten the steps MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit The runbook had operators inventing a tenant ID inline in Step 1, described it as a "UUID/slug", and carried an architecture diagram plus step prose that read as explanation rather than instruction. - New Step 0: generate the tenant UUID in one command before anything else. It must be a lowercase UUID — the identity tables type insight_tenant_id as ClickHouse UUID (20260408000000_init-identity.sql), and the silver models pass the connector's string through verbatim with no toUUID cast, so a slug fails to insert and mixed case stops comparing equal. - Document why global.tenantDefaultId and ingestion.reconcile.tenantId must match: the first resolves the tenant for the app, the second is exported as INSIGHT_TENANT_ID and injected into every Airbyte source config as insight_tenant_id, so it is what lands in bronze. Warn that changing it after the first sync orphans ingested data, and point local/dev at the existing 00000000-df51-5b42-9538-d2b56b7ee953 sentinel. - Add uuidgen (or python3) to the CLI prerequisites; Step 1 skeleton and both placeholder tables now reference Step 0 and say lowercase UUID. - Rewrite Steps 1-6 as instructions: gotcha paragraphs become imperative one-liners, the Step 4 migrate-hook prose becomes three bullets, Step 5's four checks merge into one block with expectations as inline comments, and redundant code comments and duplicated notes are dropped. - Remove the mermaid request-flow diagram. No commands, placeholders, or constraints were lost — the reference tables, Troubleshooting, and Appendix are unchanged. Signed-off-by: Konstantin Tursunov --- deploy/HELM_DEPLOY.md | 111 +++++++++++++++++++----------------------- 1 file changed, 51 insertions(+), 60 deletions(-) diff --git a/deploy/HELM_DEPLOY.md b/deploy/HELM_DEPLOY.md index cac24f26b..f4acaf886 100644 --- a/deploy/HELM_DEPLOY.md +++ b/deploy/HELM_DEPLOY.md @@ -12,6 +12,7 @@ This runbook shows a platform or DevOps engineer how to install the Insight busi - [Cluster and CLI tools](#cluster-and-cli-tools) - [Cluster-level dependencies](#cluster-level-dependencies) - [Running external infrastructure](#running-external-infrastructure) +- [Step 0 — Generate the tenant ID](#step-0--generate-the-tenant-id) - [Step 1 — Configure values/umbrella.yaml](#step-1--configure-valuesumbrellayaml) - [Step 2 — Fill the secret files](#step-2--fill-the-secret-files) - [secrets/insight-db-creds.yaml](#secretsinsight-db-credsyaml) @@ -41,18 +42,6 @@ Insight reads engineering and collaboration data from your tools (Jira, Slack, G Two more subcharts exist purely for local/dev use and are off by default: `fakeidp` (alias `fakeidp`, condition `fakeidp.deploy`) and `keycloak` (alias `keycloak`, condition `keycloak.deploy`) — both are bundled OIDC providers for a cluster with no real IdP available. `fakeidp` is the one this runbook documents; `keycloak` is a heavier bundled alternative not covered here. Neither is appropriate for a real environment. -```mermaid -flowchart LR - B[Browser] --> ING[Ingress: host] - ING --> GW[insight-gateway] - GW -->|"/*"| FE[insight-frontend] - GW -->|"/api/*"| AN[insight-analytics / insight-identity] - GW -->|"cookie→JWT exchange
(cached, cosocket)"| AUTH[insight-authenticator] - AUTH -->|OIDC login| IDP[External IdP] - AUTH -->|mints gateway JWT
signed w/ signing keys| GW - AN -->|verifies JWT via JWKS
over cert-manager CA| AUTH -``` - This path assumes your data infrastructure (ClickHouse, MariaDB, Redis, Redpanda, Airbyte, Argo Workflows) already runs and is reachable from the cluster, in another namespace or external. The chart doesn't stand it up; it only wires the services to it. You supply one values file, secret files, and optionally one Secret per connector (see [deploy/CONNECTORS.md](./CONNECTORS.md)). No GitOps repo, CI, or auto-reconciliation — you run the commands yourself. ## Prerequisites @@ -64,6 +53,7 @@ This path assumes your data infrastructure (ClickHouse, MariaDB, Redis, Redpanda - `kubectl`. - `jq`, used to mirror the Airbyte auth Secret in Step 3. - `openssl`, used to generate the authenticator's signing key in Step 2. +- `uuidgen` (or `python3`), used to generate the tenant ID in Step 0. - `base64`, used when copying existing datastore passwords in Step 2 (most systems ship this by default). ### Cluster-level dependencies @@ -86,11 +76,24 @@ All six systems below must be deployed and reachable from the cluster before you | Airbyte | Runs the data connectors (Jira, Slack, GitHub, and so on) that load raw data into ClickHouse Bronze | | Argo Workflows | Runs the dbt transform workflows that turn Bronze into Silver and Gold, and runs the sync workflows Airbyte connections trigger | -Run all commands here from the directory holding your `values/` and `secrets/` files. This document shows the full `values/umbrella.yaml` skeleton and the secret files, so you can assemble both directories from what follows. Connector configuration (the `connectors/` directory) is a separate, later step — see [deploy/CONNECTORS.md](./CONNECTORS.md). +Run every command below from the directory holding your `values/` and `secrets/` files — Steps 1 and 2 give you the full contents of both. Connector configuration (the `connectors/` directory) comes later, in [deploy/CONNECTORS.md](./CONNECTORS.md). + +## Step 0 — Generate the tenant ID + +Generate the tenant UUID and record it — you paste it into two fields in Step 1: + +```sh +uuidgen | tr '[:upper:]' '[:lower:]' # no uuidgen? python3 -c 'import uuid; print(uuid.uuid4())' +``` + +- It must be a lowercase UUID. The identity tables type the column `UUID`, and the Silver models pass the string through verbatim (`tenant_id AS insight_tenant_id`), so case has to stay consistent. +- Use the same value for `global.tenantDefaultId` (how the app resolves the tenant) and `ingestion.reconcile.tenantId` (stamped into every ingested row as `insight_tenant_id`). If they diverge, dashboards read one tenant while the pipeline writes another. +- Never change it after the first sync — ingested data is keyed by it. +- Local/dev against the compose wizard, the seed generators, or `fakeidp`: use their fixed tenant `00000000-df51-5b42-9538-d2b56b7ee953` instead of generating one. ## Step 1 — Configure values/umbrella.yaml -Create `values/umbrella.yaml` with the skeleton below, then replace every `<...>` placeholder with your infrastructure's real addresses. Passwords never go here — they live in the secret files from Step 2. +Create `values/umbrella.yaml` from the skeleton below and replace every `<...>` placeholder. No passwords here — they go in the Step 2 secret files. ```yaml ## values/umbrella.yaml — the only values file you need. @@ -100,7 +103,7 @@ credentials: autoGenerate: true # BYO compose; won't overwrite a labelless insight-db-creds global: - tenantDefaultId: "" # single-tenant seed UUID; must equal ingestion.reconcile.tenantId + tenantDefaultId: "" # the UUID from Step 0; must equal ingestion.reconcile.tenantId # storageClass: "" # "" = cluster default; e.g. "local-path" locally # imagePullSecrets: [] # [{name: my-regcred}] for a private registry @@ -203,17 +206,17 @@ frontend: # the web UI (dashboard) # fakeidp: {deploy: false} # local/dev-only alternative to a real IdP — see note below ``` -If you do not already have this file, you can generate the chart's default values as a starting point instead of typing the skeleton by hand: +To start from the chart's full defaults instead of typing the skeleton: ```sh helm show values oci://ghcr.io/constructorfabric/charts/insight > values/umbrella.yaml ``` -The placeholder table below explains every `<...>` value in the skeleton: +Fill each placeholder: | Placeholder | What it should be | |-------------|--------------------| -| `` | Your Insight tenant UUID/slug. Must be the same value in `global.tenantDefaultId` and `ingestion.reconcile.tenantId` | +| `` | The tenant UUID you generated in Step 0. Must be the same value in `global.tenantDefaultId` and `ingestion.reconcile.tenantId` | | `` | ClickHouse HTTP host, in `host:8123` form | | `` | MariaDB host, in `host:3306` form | | `` | Redis host, in `host:6379` form | @@ -221,26 +224,26 @@ The placeholder table below explains every `<...>` value in the skeleton: | `` | The ingestion toolbox image reference (drives the WorkflowTemplates and the ClickHouse gold-view migration Job, Step 4/5) | | `` | Airbyte server API URL, for example `http://host:8001` | | `` | Your Argo controller's instance ID, for example `argo-workflows-insight-infra` | -| `` | The Insight product image tag for each service. All five `image.tag` fields are optional — each falls back to that subchart's `Chart.yaml` appVersion (pinned by the release pipeline). Set them explicitly (recommended) so every service lands on the exact same product build; leaving them blank is safe only when all five subcharts' appVersion are in lockstep in the chart release you install | +| `` | The Insight product image tag. Optional on all five services — each falls back to its subchart's `Chart.yaml` appVersion — but set them explicitly so every service lands on the same build (see the Appendix) | | `` | Public FQDN for the ingress, shared by the Gateway and Frontend, for example `insight.example.com` | | `` | Name of the Kubernetes TLS Secret that covers that domain | | `` | Your IdP's issuer URL. Its `/.well-known/openid-configuration` document must resolve from inside the cluster | | `` / `` | Your OIDC client / application registration credentials | -For infrastructure running in the same cluster, use the in-cluster DNS form `..svc.cluster.local`. Any resolvable host or IP address also works. +For infrastructure in the same cluster, use `..svc.cluster.local`. Any resolvable host or IP also works. -A few settings deserve a closer look before you install: +Check these four before installing: -- **`identity.deploy` must be `true`.** The chart's own default is `false`, so this block requires an explicit override. Without it, the Identity service (and person resolution for the whole app) does not deploy. -- **`authenticator.oidc.issuerUrl` and `authenticator.oidc.redirectUri` are hard requirements.** `charts/insight/templates/secrets.yaml` wraps both in Helm's `required` function — the chart refuses to render without them. There is no auth-off escape hatch: OIDC is mandatory in every environment. -- **No dummy-IdP values file exists for this chart.** If you need a working install without wiring up a real external IdP (local/dev only), enable the bundled fake provider instead: set `fakeidp.deploy: true`, point `authenticator.oidc.issuerUrl` at the in-cluster fakeidp FQDN it exposes, and leave `clientSecret` empty. Never do this in a shared or production cluster. Note: the `fakeidp` (and `keycloak`) images are dev-only and are **not** published to public GHCR — building them locally and loading them into the cluster (or supplying an `imagePullSecret` with access) is required, otherwise the fakeidp pod fails with `ImagePullBackOff` (403). This does not affect a real install, which uses your own IdP and never deploys fakeidp. -- **`authenticator.signingKeysSecret` must already exist.** It is not auto-generated by the chart — create it in Step 2 before installing. +- Set `identity.deploy: true`. The chart default is `false`, and without the override Identity — and person resolution for the whole app — never deploys. +- Set real values for `authenticator.oidc.issuerUrl` and `redirectUri`. The chart wraps both in Helm's `required`, and there is no auth-off switch. +- Create the Secret named in `authenticator.signingKeysSecret` before installing (Step 2). The chart does not generate it. +- No real IdP, local/dev only: set `fakeidp.deploy: true`, point `issuerUrl` at the in-cluster fakeidp FQDN, leave `clientSecret` empty. Its image is not on public GHCR, so build and load it locally (or supply an `imagePullSecret`) or the pod hits `ImagePullBackOff`. Never in a shared cluster. ## Step 2 — Fill the secret files ### secrets/insight-db-creds.yaml -This Secret holds the four datastore passwords used by Analytics and Identity. All four keys are required — the chart fails fast if any is missing. Values must match the passwords your datastores were deployed with. +Create this Secret with all four datastore passwords — the chart fails fast if any key is missing. Use the passwords your datastores already run with. ```yaml apiVersion: v1 @@ -254,7 +257,7 @@ stringData: redis-password: "CHANGE_ME" # Redis password -> Analytics + Authenticator ``` -If your existing datastores already have these passwords stored in Secrets in your infrastructure namespace, copy them across instead of retyping them: +If those passwords already live in Secrets in your infrastructure namespace, copy them across instead of retyping them: ```sh NS_INFRA= # where your L2 services run @@ -264,20 +267,20 @@ kubectl -n $NS_INFRA get secret -o jsonpath='{.data. -o jsonpath='{.data.}' | base64 -d; echo # redis-password ``` -Paste the decoded output into the matching `clickhouse-password` / `mariadb-password` / `mariadb-root-password` / `redis-password` field. +Paste each decoded value into the matching field. -> **Do not add an `app.kubernetes.io/managed-by: Helm` label to this Secret.** The chart reads that label's *absence* as "bring your own" credentials. With the label, it assumes ownership and may overwrite your passwords with generated ones. Without it, the chart keeps your values and composes `insight-analytics-config` and `insight-identity-config` from them. +> **Never label this Secret `app.kubernetes.io/managed-by: Helm`.** The chart reads the label's *absence* as "bring your own" and composes `insight-analytics-config` and `insight-identity-config` from your values; with the label it takes ownership and may overwrite them with generated passwords. ### secrets/insight-authenticator-signing-keys.yaml -The authenticator mints the gateway JWT using an ES256 (EC P-256) key pair. This Secret is **not** auto-generated by the chart — you must create it yourself before `helm install`. Generate a PKCS#8 private key and load it under the required `current.pem` key: +Generate the authenticator's ES256 (EC P-256) gateway-JWT key as PKCS#8 and create the Secret — the chart does not generate it: ```sh openssl ecparam -name prime256v1 -genkey -noout | openssl pkcs8 -topk8 -nocrypt -out current.pem kubectl -n insight create secret generic insight-authenticator-signing-keys --from-file=current.pem ``` -During a key rotation, add a `previous.pem` (the outgoing key) alongside the new `current.pem` for at least the JWT TTL plus downstream JWKS-cache age (roughly 65 minutes), then roll the authenticator pods: +To rotate, keep the outgoing key as `previous.pem` beside the new `current.pem` for at least the JWT TTL plus downstream JWKS-cache age (~65 minutes), then roll the authenticator pods: ```sh kubectl -n insight create secret generic insight-authenticator-signing-keys \ @@ -287,10 +290,9 @@ kubectl -n insight create secret generic insight-authenticator-signing-keys \ ## Step 3 — Create namespace and apply secrets -Create the `insight` namespace and apply the secret files: +Create the namespace and apply the secret files: ```sh -# create the namespace and apply all secrets kubectl create namespace insight kubectl -n insight apply -f secrets/ @@ -298,68 +300,57 @@ kubectl -n insight apply -f secrets/ kubectl -n insight get secret insight-db-creds insight-authenticator-signing-keys # expect 4 keys / 1-2 keys (current.pem [+ previous.pem]) ``` -The Analytics service also needs Airbyte's own auth credentials to talk to the Airbyte API. Mirror that Secret from your infrastructure namespace into `insight`: +Mirror Airbyte's auth Secret into `insight` — Analytics needs it to call the Airbyte API: ```sh -# mirror the Airbyte auth secret from your infra namespace NS_INFRA= kubectl -n $NS_INFRA get secret airbyte-auth-secrets -o json \ | jq 'del(.metadata.uid,.metadata.resourceVersion,.metadata.creationTimestamp,.metadata.ownerReferences,.metadata.annotations,.metadata.labels) | .metadata.namespace="insight"' \ | kubectl -n insight apply -f - ``` -The `jq` step strips the original Secret's identity fields (UID, resource version, owner references) and retargets it to the `insight` namespace, so Kubernetes accepts it as a new object. +The `jq` filter strips the source object's identity fields (UID, resource version, owner references) and retargets the namespace, so Kubernetes accepts it as a new object. ## Step 4 — Install with Helm -Run the umbrella chart install, pointing it at your filled-in values file: +Install the umbrella chart against your values file: ```sh helm upgrade --install insight oci://ghcr.io/constructorfabric/charts/insight \ -n insight -f values/umbrella.yaml --wait --timeout 15m ``` -Omit `--version` to install the latest published chart, or add `--version ` to pin a specific release. `--wait --timeout 15m` blocks the command until all resources report ready, or until 15 minutes pass, whichever comes first — this gives you a clear pass/fail signal instead of a detached background rollout. - -This command also runs a post-install/post-upgrade Helm hook Job, `insight-clickhouse-migrate`, which applies the ClickHouse gold-view migrations (`src/ingestion/scripts/migrations/*.sql`) against your external ClickHouse using `ingestion.toolboxImage`. It runs on **every** `helm upgrade`, not just the first install — this is gated by `clickhouse.runMigrations` (default `true`). `helm upgrade` blocks on this Job the same way it blocks on any other resource; a failing migration fails the whole upgrade. The migration script drops-and-recreates every gold object on each run, so a genuine migrate-Job failure usually points to a schema/data problem in the referenced Bronze/Silver tables, not a stale-object conflict. +- Add `--version ` to pin a chart release; omit it for the latest published one. +- `--wait --timeout 15m` blocks until every resource is ready, giving a pass/fail signal instead of a detached rollout. +- The install also runs the `insight-clickhouse-migrate` hook Job, which applies the ClickHouse gold-view migrations (`src/ingestion/scripts/migrations/*.sql`) with `ingestion.toolboxImage`. It fires on **every** upgrade, not just the first install (gated by `clickhouse.runMigrations`, default `true`), and a failing migration fails the whole upgrade. It drops and recreates every gold object each run, so a failure points at Bronze/Silver schema or data, not a stale-object conflict. ## Step 5 — Verify the install -Confirm all pods are running (Identity only appears when `identity.deploy: true`; fakeidp/keycloak only when their `deploy` flag is set): +Run all four checks: ```sh kubectl -n insight get pods - # expect: insight-gateway, insight-authenticator, insight-analytics, insight-identity, insight-frontend (all Running) -``` - -Confirm the chart composed the per-service config Secrets from `insight-db-creds`: + # expect insight-gateway, -authenticator, -analytics, -identity, -frontend all Running + # (Identity only with identity.deploy: true; fakeidp/keycloak only with their deploy flag) -```sh kubectl -n insight get secret insight-analytics-config insight-authenticator-config insight-identity-config - # chart composed these from insight-db-creds (insight-identity-config only exists when identity.deploy=true) -``` - -Inspect the ClickHouse gold-view migration hook Job and confirm it completed: + # the chart composes these from insight-db-creds (the identity one only when identity.deploy=true) -```sh kubectl -n insight get jobs -l app.kubernetes.io/component=clickhouse-migrate kubectl -n insight logs job/insight-clickhouse-migrate -``` + # the gold-view migration Job must be Complete -Confirm the reconcile loop's scheduled workflow exists — this is the job that discovers connector Secrets and provisions Airbyte sources and connections automatically: - -```sh kubectl -n insight get cronworkflow - # expect: insight-reconcile-loop (provisions Airbyte sources/connections) + # expect insight-reconcile-loop (provisions Airbyte sources/connections) ``` -Finally, open `https://` in a browser (the host you set in Step 1) and confirm the login redirect to your OIDC provider works. +Then open `https://` — the host from Step 1 — and confirm the login redirect to your OIDC provider. ## Step 6 — Configure connectors (optional) -Configuring connectors is a separate operation from installing the app, done once the app is up and running. There are 25 available connectors, each a single Kubernetes Secret that the `insight-reconcile-loop` CronWorkflow discovers and auto-provisions as an Airbyte source — no further steps once it's applied and filled in correctly. +Configure connectors after the app is up. Each of the 25 connectors is a single Kubernetes Secret; the `insight-reconcile-loop` CronWorkflow discovers it and provisions the Airbyte source automatically, so there is nothing else to run. -See [deploy/CONNECTORS.md](./CONNECTORS.md) for the full list of connectors and a copy-paste-ready example Secret for each. +See [deploy/CONNECTORS.md](./CONNECTORS.md) for the connector list and a copy-paste Secret for each. ## Troubleshooting @@ -380,7 +371,7 @@ For connector-syncing problems, see the Troubleshooting section of [deploy/CONNE | Placeholder | Field(s) | Notes | |-------------|----------|-------| -| `` | `global.tenantDefaultId`, `ingestion.reconcile.tenantId` | Must be identical across both | +| `` | `global.tenantDefaultId`, `ingestion.reconcile.tenantId` | Generated in Step 0; a lowercase UUID, identical across both | | `` | `clickhouse.host` | Always external; port fixed at `8123` in the file | | `` | `mariadb.host` | Always external; port fixed at `3306` | | `` | `redis.host` | Always external; port fixed at `6379` | From 5f4e2b9286686cf7020bedd49600138437f076b0 Mon Sep 17 00:00:00 2001 From: Konstantin Tursunov Date: Mon, 27 Jul 2026 10:52:54 +0800 Subject: [PATCH 04/14] docs(deploy): restyle cluster-level dependencies as instructions MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit - Rewrite the two bullets in the runbook's imperative style and add a verify block (ingressclass / cert-manager CRD / clusterissuer) so the prerequisite is checkable instead of assumed, matching the check blocks in Steps 3 and 5. - Drop the claim that the ingress blocks are "hardcoded" to className: nginx — it is a values default (values.yaml:318 gateway, :529 frontend), which the same sentence then contradicted by telling the reader to override it. Signed-off-by: Konstantin Tursunov --- deploy/HELM_DEPLOY.md | 14 +++++++++++--- 1 file changed, 11 insertions(+), 3 deletions(-) diff --git a/deploy/HELM_DEPLOY.md b/deploy/HELM_DEPLOY.md index f4acaf886..d890a0b6d 100644 --- a/deploy/HELM_DEPLOY.md +++ b/deploy/HELM_DEPLOY.md @@ -58,10 +58,18 @@ This path assumes your data infrastructure (ClickHouse, MariaDB, Redis, Redpanda ### Cluster-level dependencies -Two things must already be installed in the cluster before you install this chart — neither is bundled by it: +Install both of these before the chart — it bundles neither: -- **An ingress controller.** The Gateway and Frontend ingress blocks are hardcoded to `className: nginx`; install an ingress-nginx controller (or override `gateway.ingress.className` / `frontend.ingress.className` to match whatever you run). -- **cert-manager**, with a working `ClusterIssuer`. The authenticator's TLS-discovery sidecar (`authenticator.tlsDiscovery.enabled: true` by default) creates a `cert-manager.io/v1` `Certificate`, so cert-manager's CRDs must be present. Analytics and Identity trust that cert-manager-issued CA to verify the authenticator's JWKS over HTTPS — this is load-bearing, not optional. The chart's default `issuerRef.name` is `local-ca`; either provision a `ClusterIssuer` with that name, or override `authenticator.tlsDiscovery.issuerRef.name` to point at your own. +- **An ingress controller.** Install ingress-nginx, or override `gateway.ingress.className` / `frontend.ingress.className` to match what you run. Both default to `className: nginx`. +- **cert-manager, with a working `ClusterIssuer`.** The authenticator's TLS-discovery sidecar (`authenticator.tlsDiscovery.enabled`, default `true`) creates a `cert-manager.io/v1` `Certificate`, and Analytics and Identity verify the authenticator's JWKS over HTTPS against that CA — load-bearing, not optional. Provision a `ClusterIssuer` named `local-ca` (the chart default) or point `authenticator.tlsDiscovery.issuerRef.name` at your own. + +Confirm both are in place: + +```sh +kubectl get ingressclass nginx # the className both ingress blocks use +kubectl get crd certificates.cert-manager.io # cert-manager CRDs installed +kubectl get clusterissuer local-ca # the issuer the chart defaults to +``` ### Running external infrastructure From 72f1a400c273c4440bd1354942124c61ae27fc40 Mon Sep 17 00:00:00 2001 From: Konstantin Tursunov Date: Mon, 27 Jul 2026 11:29:43 +0800 Subject: [PATCH 05/14] docs(deploy): widen Step 0 to collecting every Step 1 value MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit Step 0 only generated the tenant ID, leaving operators to guess where the external service addresses come from — and was documented as "host:9093 form", which hides both its shape and how to find it. - Retitle Step 0 "Collect the values Step 1 needs" and split it into three subsections: generate the tenant ID, look up the external service addresses, compose the Redpanda brokers string. Every datastore is external, so the addresses are looked up, not created. - Address lookup: kubectl get svc in the infra namespace, with the ..svc.cluster.local: form, the ports already fixed in the skeleton, and where AIRBYTE_API_URL comes from. - Redpanda gets its own subsection because it is the exception: one comma-separated host:port bootstrap string aimed at the internal Kafka API listener, not the host/port pair the other datastores take. Commands find the Service and print its ports; notes cover reading the port instead of assuming 9093 (the redpanda/redpanda chart's internal listener; the compose stack uses 9092), single-broker bootstrap, and the unroutable-placeholder escape hatch for clusters with no Redpanda (functional-ci's redpanda-disabled:9093), since redpanda.brokers is a required field. - Update both table rows to the real shape, add the new subsections to the ToC, and point the external-infrastructure prerequisite at Step 0. Signed-off-by: Konstantin Tursunov --- deploy/HELM_DEPLOY.md | 46 +++++++++++++++++++++++++++++++++++++------ 1 file changed, 40 insertions(+), 6 deletions(-) diff --git a/deploy/HELM_DEPLOY.md b/deploy/HELM_DEPLOY.md index d890a0b6d..fc83df3a1 100644 --- a/deploy/HELM_DEPLOY.md +++ b/deploy/HELM_DEPLOY.md @@ -12,7 +12,10 @@ This runbook shows a platform or DevOps engineer how to install the Insight busi - [Cluster and CLI tools](#cluster-and-cli-tools) - [Cluster-level dependencies](#cluster-level-dependencies) - [Running external infrastructure](#running-external-infrastructure) -- [Step 0 — Generate the tenant ID](#step-0--generate-the-tenant-id) +- [Step 0 — Collect the values Step 1 needs](#step-0--collect-the-values-step-1-needs) + - [Generate the tenant ID](#generate-the-tenant-id) + - [Look up the external service addresses](#look-up-the-external-service-addresses) + - [Compose the Redpanda brokers string](#compose-the-redpanda-brokers-string) - [Step 1 — Configure values/umbrella.yaml](#step-1--configure-valuesumbrellayaml) - [Step 2 — Fill the secret files](#step-2--fill-the-secret-files) - [secrets/insight-db-creds.yaml](#secretsinsight-db-credsyaml) @@ -73,7 +76,7 @@ kubectl get clusterissuer local-ca # the issuer the chart defaults ### Running external infrastructure -All six systems below must be deployed and reachable from the cluster before you start. The chart never installs any of them — ClickHouse, MariaDB, Redis, and Redpanda are wired in purely by host/credentials (Step 1); Airbyte and Argo Workflows are wired in via `airbyte.apiUrl` and `ingestion.reconcile.argoInstanceId`. +All six systems below must be deployed and reachable from the cluster before you start. The chart never installs any of them — ClickHouse, MariaDB, Redis, and Redpanda are wired in purely by host/credentials; Airbyte and Argo Workflows via `airbyte.apiUrl` and `ingestion.reconcile.argoInstanceId`. Step 0 shows how to read those addresses off your cluster. | System | Used for | |--------|----------| @@ -86,9 +89,11 @@ All six systems below must be deployed and reachable from the cluster before you Run every command below from the directory holding your `values/` and `secrets/` files — Steps 1 and 2 give you the full contents of both. Connector configuration (the `connectors/` directory) comes later, in [deploy/CONNECTORS.md](./CONNECTORS.md). -## Step 0 — Generate the tenant ID +## Step 0 — Collect the values Step 1 needs -Generate the tenant UUID and record it — you paste it into two fields in Step 1: +Step 1 asks for two kinds of value: the tenant ID, which you generate, and the addresses of the external services, which you look up. Every datastore is external — the chart only dials it — so nothing here creates infrastructure. + +### Generate the tenant ID ```sh uuidgen | tr '[:upper:]' '[:lower:]' # no uuidgen? python3 -c 'import uuid; print(uuid.uuid4())' @@ -99,6 +104,35 @@ uuidgen | tr '[:upper:]' '[:lower:]' # no uuidgen? python3 -c 'import uuid; p - Never change it after the first sync — ingested data is keyed by it. - Local/dev against the compose wizard, the seed generators, or `fakeidp`: use their fixed tenant `00000000-df51-5b42-9538-d2b56b7ee953` instead of generating one. +### Look up the external service addresses + +List the Services in the namespace your infrastructure runs in and read each host and port off it: + +```sh +NS_INFRA= +kubectl -n $NS_INFRA get svc + # every address in Step 1 is .$NS_INFRA.svc.cluster.local: + # ClickHouse 8123, MariaDB 3306, Redis 6379 are already fixed in the skeleton — you only supply the host + # Airbyte: is the server Service, e.g. http://airbyte-airbyte-server-svc.$NS_INFRA.svc.cluster.local:8001 +``` + +Infrastructure outside the cluster works the same way — use any resolvable host or IP instead of the in-cluster DNS name. + +### Compose the Redpanda brokers string + +`redpanda.brokers` is the exception: one comma-separated `host:port` bootstrap string rather than the host/port pair the other datastores take, and it must point at Redpanda's internal Kafka API listener. + +```sh +kubectl -n $NS_INFRA get svc -l app.kubernetes.io/name=redpanda +kubectl -n $NS_INFRA get svc redpanda -o jsonpath='{range .spec.ports[*]}{.name}={.port}{"\n"}{end}' + # compose .$NS_INFRA.svc.cluster.local: + # e.g. redpanda.insight-infra.svc.cluster.local:9093 +``` + +- Read the port instead of assuming it. `9093` is the internal listener of the official `redpanda/redpanda` chart; other setups differ — this repo's compose stack runs it on `9092`. +- One reachable broker bootstraps the client, which then discovers the rest from cluster metadata. Comma-separate more only for resilience. +- The field is `required`, so the chart will not render without it. If you have no Redpanda and are not exercising the authenticator's audit stream, point it at an unroutable placeholder the way the functional-CI overlay does (`brokers: "redpanda-disabled:9093"`). + ## Step 1 — Configure values/umbrella.yaml Create `values/umbrella.yaml` from the skeleton below and replace every `<...>` placeholder. No passwords here — they go in the Step 2 secret files. @@ -228,7 +262,7 @@ Fill each placeholder: | `` | ClickHouse HTTP host, in `host:8123` form | | `` | MariaDB host, in `host:3306` form | | `` | Redis host, in `host:6379` form | -| `` | Redpanda broker(s), in `host:9093` form | +| `` | The bootstrap string you composed in Step 0 — comma-separated `host:port` pointing at the internal Kafka API listener | | `` | The ingestion toolbox image reference (drives the WorkflowTemplates and the ClickHouse gold-view migration Job, Step 4/5) | | `` | Airbyte server API URL, for example `http://host:8001` | | `` | Your Argo controller's instance ID, for example `argo-workflows-insight-infra` | @@ -383,7 +417,7 @@ For connector-syncing problems, see the Troubleshooting section of [deploy/CONNE | `` | `clickhouse.host` | Always external; port fixed at `8123` in the file | | `` | `mariadb.host` | Always external; port fixed at `3306` | | `` | `redis.host` | Always external; port fixed at `6379` | -| `` | `redpanda.brokers` | Always external; include port, e.g. `:9093` | +| `` | `redpanda.brokers` | Always external; a single comma-separated `host:port` string, not a host/port pair. `9093` for the `redpanda/redpanda` chart's internal listener — read yours in Step 0 | | `` | `ingestion.toolboxImage` | Drives the ingestion WorkflowTemplates and the ClickHouse gold-view migrate Job | | `` | `airbyte.apiUrl` | e.g. `http://host:8001` | | `` | `ingestion.reconcile.argoInstanceId` | Your Argo controller's instance ID | From 0eac00f6e0119244c42d500b9751247e98779e40 Mon Sep 17 00:00:00 2001 From: Konstantin Tursunov Date: Mon, 27 Jul 2026 12:21:23 +0800 Subject: [PATCH 06/14] docs(deploy): mark ingestion.toolboxImage optional (chart-default appVersion) The values skeleton presented as a required fill-in, but the umbrella chart already ships a real default for ingestion.toolboxImage pinned to the chart appVersion, and the ClickHouse gold-view migrate Job / ingestion WorkflowTemplates inherit it when unset. Comment the field out in the skeleton and note it as optional in both placeholder tables. Signed-off-by: Konstantin Tursunov --- deploy/HELM_DEPLOY.md | 6 +++--- 1 file changed, 3 insertions(+), 3 deletions(-) diff --git a/deploy/HELM_DEPLOY.md b/deploy/HELM_DEPLOY.md index fc83df3a1..ed4af7745 100644 --- a/deploy/HELM_DEPLOY.md +++ b/deploy/HELM_DEPLOY.md @@ -170,7 +170,7 @@ redpanda: ingestion: templates: enabled: true - toolboxImage: "" # e.g. ghcr.io/constructorfabric/insight-toolbox: + # toolboxImage: "" # optional — defaults to the chart's appVersion-pinned toolbox; only set to override reconcile: tenantId: "" destinationName: clickhouse-bronze @@ -263,7 +263,7 @@ Fill each placeholder: | `` | MariaDB host, in `host:3306` form | | `` | Redis host, in `host:6379` form | | `` | The bootstrap string you composed in Step 0 — comma-separated `host:port` pointing at the internal Kafka API listener | -| `` | The ingestion toolbox image reference (drives the WorkflowTemplates and the ClickHouse gold-view migration Job, Step 4/5) | +| `` | Optional. The ingestion toolbox image (drives the WorkflowTemplates and the ClickHouse gold-view migration Job, Step 4/5). Omit to inherit the chart's default, which is pinned to the chart appVersion; set only to override | | `` | Airbyte server API URL, for example `http://host:8001` | | `` | Your Argo controller's instance ID, for example `argo-workflows-insight-infra` | | `` | The Insight product image tag. Optional on all five services — each falls back to its subchart's `Chart.yaml` appVersion — but set them explicitly so every service lands on the same build (see the Appendix) | @@ -418,7 +418,7 @@ For connector-syncing problems, see the Troubleshooting section of [deploy/CONNE | `` | `mariadb.host` | Always external; port fixed at `3306` | | `` | `redis.host` | Always external; port fixed at `6379` | | `` | `redpanda.brokers` | Always external; a single comma-separated `host:port` string, not a host/port pair. `9093` for the `redpanda/redpanda` chart's internal listener — read yours in Step 0 | -| `` | `ingestion.toolboxImage` | Drives the ingestion WorkflowTemplates and the ClickHouse gold-view migrate Job | +| `` | `ingestion.toolboxImage` | Optional — defaults to the chart appVersion. Drives the ingestion WorkflowTemplates and the ClickHouse gold-view migrate Job | | `` | `airbyte.apiUrl` | e.g. `http://host:8001` | | `` | `ingestion.reconcile.argoInstanceId` | Your Argo controller's instance ID | | `` | `gateway.image.tag`, `authenticator.image.tag`, `analytics.image.tag`, `identity.image.tag`, `frontend.image.tag` | All five are optional — each falls back to that subchart's `Chart.yaml` appVersion (pinned by the release pipeline). Set them explicitly (recommended) so every service lands on the exact same product build; leaving them blank is safe only when all five subcharts' appVersion are in lockstep in the chart release you install | From 4131474bedf5f9ef11c2a1623dfe7f9d6a9388bd Mon Sep 17 00:00:00 2001 From: Konstantin Tursunov Date: Mon, 27 Jul 2026 12:34:08 +0800 Subject: [PATCH 07/14] docs(deploy): add Step 0 instruction to read the Argo instance ID from infra MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit was presented with only a guessed example and no way to obtain the real value. Add a Step 0 subsection that reads instanceID off the workflow-controller config map (covering both the nested config: key and the legacy top-level key), explains that an unpinned controller — the common case — takes an empty argoInstanceId, and cross-checks via the controller args. Wire it into the TOC, the values-skeleton comment, and both placeholder tables. Signed-off-by: Konstantin Tursunov --- deploy/HELM_DEPLOY.md | 27 ++++++++++++++++++++++++--- 1 file changed, 24 insertions(+), 3 deletions(-) diff --git a/deploy/HELM_DEPLOY.md b/deploy/HELM_DEPLOY.md index ed4af7745..c5114f8df 100644 --- a/deploy/HELM_DEPLOY.md +++ b/deploy/HELM_DEPLOY.md @@ -16,6 +16,7 @@ This runbook shows a platform or DevOps engineer how to install the Insight busi - [Generate the tenant ID](#generate-the-tenant-id) - [Look up the external service addresses](#look-up-the-external-service-addresses) - [Compose the Redpanda brokers string](#compose-the-redpanda-brokers-string) + - [Read the Argo workflow-controller instance ID](#read-the-argo-workflow-controller-instance-id) - [Step 1 — Configure values/umbrella.yaml](#step-1--configure-valuesumbrellayaml) - [Step 2 — Fill the secret files](#step-2--fill-the-secret-files) - [secrets/insight-db-creds.yaml](#secretsinsight-db-credsyaml) @@ -133,6 +134,26 @@ kubectl -n $NS_INFRA get svc redpanda -o jsonpath='{range .spec.ports[*]}{.name} - One reachable broker bootstraps the client, which then discovers the rest from cluster metadata. Comma-separate more only for resilience. - The field is `required`, so the chart will not render without it. If you have no Redpanda and are not exercising the authenticator's audit stream, point it at an unroutable placeholder the way the functional-CI overlay does (`brokers: "redpanda-disabled:9093"`). +### Read the Argo workflow-controller instance ID + +`ingestion.reconcile.argoInstanceId` must match the `instanceID` the cluster's Argo workflow controller runs with. It stamps the `workflows.argoproj.io/controller-instanceid` label onto the workflows reconcile submits, so a controller pinned to that instance ID picks them up. **If the controller has no `instanceID` configured — the common case — leave this empty** (the label is omitted and an unpinned controller accepts the workflows anyway). Only set it when the controller is pinned. + +Read it off the controller's config map, which is the authoritative source: + +```sh +# find the controller config map (name varies by chart, e.g. argo-workflows-workflow-controller-configmap) +kubectl -n $NS_INFRA get cm | grep workflow-controller + +CM= +# newer charts nest all controller config under a single `config:` YAML key … +kubectl -n $NS_INFRA get cm $CM -o jsonpath='{.data.config}' | grep -i instanceID +# … older ones expose it as a top-level data key: +kubectl -n $NS_INFRA get cm $CM -o jsonpath='{.data.instanceID}{"\n"}' +``` + +- A non-empty match (e.g. `instanceID: argo-workflows-insight`) → set `argoInstanceId` to that exact string. +- No `instanceID` line / empty output → leave `argoInstanceId` empty (comment the line out). Verify the controller is unpinned by confirming its flags carry no `--instanceid`: `kubectl -n $NS_INFRA get deploy -l app.kubernetes.io/component=workflow-controller -o jsonpath='{.items[0].spec.template.spec.containers[0].args}'`. + ## Step 1 — Configure values/umbrella.yaml Create `values/umbrella.yaml` from the skeleton below and replace every `<...>` placeholder. No passwords here — they go in the Step 2 secret files. @@ -174,7 +195,7 @@ ingestion: reconcile: tenantId: "" destinationName: clickhouse-bronze - argoInstanceId: "" # e.g. argo-workflows- + argoInstanceId: "" # match the controller's instanceID (Step 0); leave "" if unpinned airbyte: namespace: "" # namespace of the Airbyte release, e.g. ; "" = same as the app apiUrl: "" # "" = computed from airbyte.releaseName + airbyte.namespace; set only for a non-standard URL @@ -265,7 +286,7 @@ Fill each placeholder: | `` | The bootstrap string you composed in Step 0 — comma-separated `host:port` pointing at the internal Kafka API listener | | `` | Optional. The ingestion toolbox image (drives the WorkflowTemplates and the ClickHouse gold-view migration Job, Step 4/5). Omit to inherit the chart's default, which is pinned to the chart appVersion; set only to override | | `` | Airbyte server API URL, for example `http://host:8001` | -| `` | Your Argo controller's instance ID, for example `argo-workflows-insight-infra` | +| `` | The `instanceID` your Argo workflow controller is pinned to — read it off the controller config map in Step 0. Leave empty (`""`) if the controller is unpinned, the common case | | `` | The Insight product image tag. Optional on all five services — each falls back to its subchart's `Chart.yaml` appVersion — but set them explicitly so every service lands on the same build (see the Appendix) | | `` | Public FQDN for the ingress, shared by the Gateway and Frontend, for example `insight.example.com` | | `` | Name of the Kubernetes TLS Secret that covers that domain | @@ -420,7 +441,7 @@ For connector-syncing problems, see the Troubleshooting section of [deploy/CONNE | `` | `redpanda.brokers` | Always external; a single comma-separated `host:port` string, not a host/port pair. `9093` for the `redpanda/redpanda` chart's internal listener — read yours in Step 0 | | `` | `ingestion.toolboxImage` | Optional — defaults to the chart appVersion. Drives the ingestion WorkflowTemplates and the ClickHouse gold-view migrate Job | | `` | `airbyte.apiUrl` | e.g. `http://host:8001` | -| `` | `ingestion.reconcile.argoInstanceId` | Your Argo controller's instance ID | +| `` | `ingestion.reconcile.argoInstanceId` | Match the controller's configured `instanceID` (Step 0); empty if unpinned | | `` | `gateway.image.tag`, `authenticator.image.tag`, `analytics.image.tag`, `identity.image.tag`, `frontend.image.tag` | All five are optional — each falls back to that subchart's `Chart.yaml` appVersion (pinned by the release pipeline). Set them explicitly (recommended) so every service lands on the exact same product build; leaving them blank is safe only when all five subcharts' appVersion are in lockstep in the chart release you install | | `` | `gateway.ingress.host`, `frontend.ingress.host` | Public FQDN, shared by the Gateway and Frontend (`/*` → UI, `/api/*` → Gateway routes to Analytics/Identity) | | `` | `gateway.ingress.tls.secretName` | Kubernetes TLS Secret name | From d99688e54329c9b073a3dbdb3288e0c8a2b6d41d Mon Sep 17 00:00:00 2001 From: Konstantin Tursunov Date: Mon, 27 Jul 2026 14:01:40 +0800 Subject: [PATCH 08/14] docs(deploy): trim Step 0 to sentence+command and fix stand-specific defaults MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit Step 0 had grown bullet lists under each item, and three values were documented from the local sandbox's point of view rather than a real cluster's. - Every Step 0 item is now one sentence plus one example command: tenant ID, external service addresses, Airbyte API URL, Redpanda brokers, Argo instance ID. The dropped rationale already lives in the placeholder tables and Troubleshooting. - Airbyte gets its own subsection, and apiUrl is no longer presented as unconditionally required: insight.airbyte.url falls back to http://-airbyte-server-svc.<.Release.Namespace>:8001 (_helpers.tpl:113-118), which interpolates the umbrella's own namespace, so the fallback only resolves when Airbyte is a release in `insight` and must be set explicitly otherwise. - Stop telling operators to provision a ClusterIssuer named local-ca. That issuer is the self-signed root deploy/gitops/bootstrap/local/selfsigned-issuer.yaml creates for the local k3s sandbox via `make bootstrap-cert-manager ENV=local`; no other cluster has it. issuerRef.name is now a placeholder with rows in both tables, and the prerequisite notes the cert is internal-only — trusted via the mounted CA, unrelated to the public ingress cert — so any issuer, self-signed included, is fine. - Drop the five blocks and both placeholder rows: subcharts render `image.tag | default .Chart.AppVersion` and the release pipeline pins appVersions in lockstep, so a chart release already carries a coherent image set. The Appendix records why they are absent and how to override one service. - Stop assuming a single infra namespace anywhere: services are located with `kubectl get svc -A`, and the Step 2 password copy, the Step 3 Airbyte mirror, the Redpanda lookup, and the Argo lookup each scope to the namespace that dependency actually runs in. Signed-off-by: Konstantin Tursunov --- deploy/HELM_DEPLOY.md | 110 +++++++++++++++++------------------------- 1 file changed, 43 insertions(+), 67 deletions(-) diff --git a/deploy/HELM_DEPLOY.md b/deploy/HELM_DEPLOY.md index c5114f8df..f88b043b2 100644 --- a/deploy/HELM_DEPLOY.md +++ b/deploy/HELM_DEPLOY.md @@ -15,6 +15,7 @@ This runbook shows a platform or DevOps engineer how to install the Insight busi - [Step 0 — Collect the values Step 1 needs](#step-0--collect-the-values-step-1-needs) - [Generate the tenant ID](#generate-the-tenant-id) - [Look up the external service addresses](#look-up-the-external-service-addresses) + - [Find the Airbyte API URL](#find-the-airbyte-api-url) - [Compose the Redpanda brokers string](#compose-the-redpanda-brokers-string) - [Read the Argo workflow-controller instance ID](#read-the-argo-workflow-controller-instance-id) - [Step 1 — Configure values/umbrella.yaml](#step-1--configure-valuesumbrellayaml) @@ -65,14 +66,14 @@ This path assumes your data infrastructure (ClickHouse, MariaDB, Redis, Redpanda Install both of these before the chart — it bundles neither: - **An ingress controller.** Install ingress-nginx, or override `gateway.ingress.className` / `frontend.ingress.className` to match what you run. Both default to `className: nginx`. -- **cert-manager, with a working `ClusterIssuer`.** The authenticator's TLS-discovery sidecar (`authenticator.tlsDiscovery.enabled`, default `true`) creates a `cert-manager.io/v1` `Certificate`, and Analytics and Identity verify the authenticator's JWKS over HTTPS against that CA — load-bearing, not optional. Provision a `ClusterIssuer` named `local-ca` (the chart default) or point `authenticator.tlsDiscovery.issuerRef.name` at your own. +- **cert-manager, plus a `ClusterIssuer` of your own.** The authenticator's TLS-discovery sidecar (`authenticator.tlsDiscovery.enabled`, default `true`) creates a `cert-manager.io/v1` `Certificate`, and Analytics and Identity verify the authenticator's JWKS over HTTPS against that CA — load-bearing, not optional. Set `authenticator.tlsDiscovery.issuerRef.name` to whatever your cluster issues from. Do not expect the chart's `local-ca` default to resolve: that issuer belongs to this repo's local k3s sandbox (`deploy/gitops/bootstrap/local/selfsigned-issuer.yaml`, applied by `make bootstrap-cert-manager ENV=local`) and will not exist in a cluster you did not bootstrap that way. Any issuer will do — this certificate is internal-only, trusted through the CA the services mount, so it needs no public chain and is unrelated to the public ingress certificate in ``. Confirm both are in place: ```sh kubectl get ingressclass nginx # the className both ingress blocks use kubectl get crd certificates.cert-manager.io # cert-manager CRDs installed -kubectl get clusterissuer local-ca # the issuer the chart defaults to +kubectl get clusterissuer # pick one for tlsDiscovery.issuerRef.name ``` ### Running external infrastructure @@ -92,68 +93,53 @@ Run every command below from the directory holding your `values/` and `secrets/` ## Step 0 — Collect the values Step 1 needs -Step 1 asks for two kinds of value: the tenant ID, which you generate, and the addresses of the external services, which you look up. Every datastore is external — the chart only dials it — so nothing here creates infrastructure. +Generate the tenant ID, then read the rest off the cluster — every dependency is external, and each may sit in its own namespace or outside the cluster entirely. ### Generate the tenant ID +A lowercase UUID, used verbatim for both `global.tenantDefaultId` and `ingestion.reconcile.tenantId`, and never changed after the first sync (local/dev against the compose wizard, the seed generators, or `fakeidp` instead reuses their fixed `00000000-df51-5b42-9538-d2b56b7ee953`). + ```sh uuidgen | tr '[:upper:]' '[:lower:]' # no uuidgen? python3 -c 'import uuid; print(uuid.uuid4())' ``` -- It must be a lowercase UUID. The identity tables type the column `UUID`, and the Silver models pass the string through verbatim (`tenant_id AS insight_tenant_id`), so case has to stay consistent. -- Use the same value for `global.tenantDefaultId` (how the app resolves the tenant) and `ingestion.reconcile.tenantId` (stamped into every ingested row as `insight_tenant_id`). If they diverge, dashboards read one tenant while the pipeline writes another. -- Never change it after the first sync — ingested data is keyed by it. -- Local/dev against the compose wizard, the seed generators, or `fakeidp`: use their fixed tenant `00000000-df51-5b42-9538-d2b56b7ee953` instead of generating one. - ### Look up the external service addresses -List the Services in the namespace your infrastructure runs in and read each host and port off it: +Every host is `..svc.cluster.local` — or any resolvable host or IP for off-cluster infrastructure — and the ClickHouse, MariaDB, and Redis ports are already fixed in the skeleton, so you supply only the host. ```sh -NS_INFRA= -kubectl -n $NS_INFRA get svc - # every address in Step 1 is .$NS_INFRA.svc.cluster.local: - # ClickHouse 8123, MariaDB 3306, Redis 6379 are already fixed in the skeleton — you only supply the host - # Airbyte: is the server Service, e.g. http://airbyte-airbyte-server-svc.$NS_INFRA.svc.cluster.local:8001 +kubectl get svc -A | grep -Ei 'clickhouse|mariadb|redis|redpanda|airbyte' ``` -Infrastructure outside the cluster works the same way — use any resolvable host or IP instead of the in-cluster DNS name. +### Find the Airbyte API URL + +`airbyte.apiUrl` is the Airbyte **server** Service on its HTTP port, and you must set it whenever Airbyte runs anywhere other than the `insight` namespace — left empty, the chart computes `http://-airbyte-server-svc..svc.cluster.local:8001`, which only resolves when Airbyte is a release in that same namespace. + +```sh +kubectl -n get svc | grep server + # e.g. http://airbyte-airbyte-server-svc..svc.cluster.local:8001 +``` ### Compose the Redpanda brokers string -`redpanda.brokers` is the exception: one comma-separated `host:port` bootstrap string rather than the host/port pair the other datastores take, and it must point at Redpanda's internal Kafka API listener. +`redpanda.brokers` takes one comma-separated `host:port` bootstrap string — not the host/port pair the other datastores take — aimed at Redpanda's internal Kafka API listener, so read the port rather than assuming `9093` (that is the `redpanda/redpanda` chart's default; this repo's compose stack uses `9092`). ```sh -kubectl -n $NS_INFRA get svc -l app.kubernetes.io/name=redpanda -kubectl -n $NS_INFRA get svc redpanda -o jsonpath='{range .spec.ports[*]}{.name}={.port}{"\n"}{end}' - # compose .$NS_INFRA.svc.cluster.local: +kubectl -n get svc -o jsonpath='{range .spec.ports[*]}{.name}={.port}{"\n"}{end}' + # compose ..svc.cluster.local: # e.g. redpanda.insight-infra.svc.cluster.local:9093 ``` -- Read the port instead of assuming it. `9093` is the internal listener of the official `redpanda/redpanda` chart; other setups differ — this repo's compose stack runs it on `9092`. -- One reachable broker bootstraps the client, which then discovers the rest from cluster metadata. Comma-separate more only for resilience. -- The field is `required`, so the chart will not render without it. If you have no Redpanda and are not exercising the authenticator's audit stream, point it at an unroutable placeholder the way the functional-CI overlay does (`brokers: "redpanda-disabled:9093"`). - ### Read the Argo workflow-controller instance ID -`ingestion.reconcile.argoInstanceId` must match the `instanceID` the cluster's Argo workflow controller runs with. It stamps the `workflows.argoproj.io/controller-instanceid` label onto the workflows reconcile submits, so a controller pinned to that instance ID picks them up. **If the controller has no `instanceID` configured — the common case — leave this empty** (the label is omitted and an unpinned controller accepts the workflows anyway). Only set it when the controller is pinned. - -Read it off the controller's config map, which is the authoritative source: +Set `ingestion.reconcile.argoInstanceId` to the controller's configured `instanceID` only when it is pinned to one; no match means leave it empty — the common case, where the reconcile workflows go unlabelled and any controller accepts them. ```sh -# find the controller config map (name varies by chart, e.g. argo-workflows-workflow-controller-configmap) -kubectl -n $NS_INFRA get cm | grep workflow-controller - -CM= -# newer charts nest all controller config under a single `config:` YAML key … -kubectl -n $NS_INFRA get cm $CM -o jsonpath='{.data.config}' | grep -i instanceID -# … older ones expose it as a top-level data key: -kubectl -n $NS_INFRA get cm $CM -o jsonpath='{.data.instanceID}{"\n"}' +kubectl -n get cm | grep workflow-controller # name varies by chart version +kubectl -n get cm -o jsonpath='{.data.config}' | grep -i instanceID # newer charts nest it under `config:` +kubectl -n get cm -o jsonpath='{.data.instanceID}{"\n"}' # older ones use a top-level key ``` -- A non-empty match (e.g. `instanceID: argo-workflows-insight`) → set `argoInstanceId` to that exact string. -- No `instanceID` line / empty output → leave `argoInstanceId` empty (comment the line out). Verify the controller is unpinned by confirming its flags carry no `--instanceid`: `kubectl -n $NS_INFRA get deploy -l app.kubernetes.io/component=workflow-controller -o jsonpath='{.items[0].spec.template.spec.containers[0].args}'`. - ## Step 1 — Configure values/umbrella.yaml Create `values/umbrella.yaml` from the skeleton below and replace every `<...>` placeholder. No passwords here — they go in the Step 2 secret files. @@ -172,7 +158,7 @@ global: # Datastore wiring — every dep is external; the chart only dials it. clickhouse: - host: # e.g. clickhouse..svc.cluster.local + host: # e.g. clickhouse..svc.cluster.local port: 8123 database: insight username: insight @@ -185,7 +171,7 @@ redis: host: port: 6379 redpanda: - brokers: "" # e.g. redpanda..svc.cluster.local:9093 + brokers: "" # e.g. redpanda..svc.cluster.local:9093 # Ingestion — point at existing Airbyte + Argo; install the dbt WorkflowTemplates. ingestion: @@ -197,21 +183,16 @@ ingestion: destinationName: clickhouse-bronze argoInstanceId: "" # match the controller's instanceID (Step 0); leave "" if unpinned airbyte: - namespace: "" # namespace of the Airbyte release, e.g. ; "" = same as the app - apiUrl: "" # "" = computed from airbyte.releaseName + airbyte.namespace; set only for a non-standard URL + apiUrl: "" # required unless Airbyte runs in the `insight` namespace (Step 0) analytics: replicaCount: 1 # chart default 2; bump for HA - image: - tag: "" # optional — falls back to the chart's appVersion resources: requests: { cpu: 100m, memory: 128Mi } limits: { cpu: 500m, memory: 512Mi } gateway: replicaCount: 1 - image: - tag: "" # optional — falls back to the chart's appVersion ingress: enabled: true className: nginx @@ -225,16 +206,14 @@ gateway: authenticator: replicaCount: 1 - image: - tag: "" # ES256 signing keys — see Step 2. MUST already exist as a Secret before install. signingKeysSecret: "insight-authenticator-signing-keys" - # cert-manager Certificate for the JWKS-discovery sidecar. Override - # issuerRef.name only if your cluster's ClusterIssuer isn't named `local-ca`. + # cert-manager Certificate for the JWKS-discovery sidecar (internal TLS only). tlsDiscovery: enabled: true issuerRef: - name: local-ca + name: # your cluster's ClusterIssuer; the chart's `local-ca` + # default only exists in the local k3s sandbox oidc: issuerUrl: "" # MUST be set — your IdP's issuer URL clientId: "" @@ -245,8 +224,6 @@ authenticator: identity: deploy: true # MUST be true (chart default false) replicaCount: 1 - image: - tag: "" databaseName: "identity" resources: requests: { cpu: 50m, memory: 96Mi } @@ -255,8 +232,6 @@ identity: frontend: # the web UI (dashboard) deploy: true replicaCount: 1 - image: - tag: "" ingress: enabled: true # WITHOUT this the UI pod runs but is never exposed className: nginx @@ -285,11 +260,11 @@ Fill each placeholder: | `` | Redis host, in `host:6379` form | | `` | The bootstrap string you composed in Step 0 — comma-separated `host:port` pointing at the internal Kafka API listener | | `` | Optional. The ingestion toolbox image (drives the WorkflowTemplates and the ClickHouse gold-view migration Job, Step 4/5). Omit to inherit the chart's default, which is pinned to the chart appVersion; set only to override | -| `` | Airbyte server API URL, for example `http://host:8001` | +| `` | The Airbyte server Service URL from Step 0, for example `http://host:8001`. Omit only when Airbyte is a release in the `insight` namespace — the chart then computes it from `airbyte.releaseName` | | `` | The `instanceID` your Argo workflow controller is pinned to — read it off the controller config map in Step 0. Leave empty (`""`) if the controller is unpinned, the common case | -| `` | The Insight product image tag. Optional on all five services — each falls back to its subchart's `Chart.yaml` appVersion — but set them explicitly so every service lands on the same build (see the Appendix) | | `` | Public FQDN for the ingress, shared by the Gateway and Frontend, for example `insight.example.com` | | `` | Name of the Kubernetes TLS Secret that covers that domain | +| `` | A cert-manager `ClusterIssuer` in your cluster, for the authenticator's internal JWKS certificate. Self-signed is fine; the chart's `local-ca` default exists only in this repo's local sandbox | | `` | Your IdP's issuer URL. Its `/.well-known/openid-configuration` document must resolve from inside the cluster | | `` / `` | Your OIDC client / application registration credentials | @@ -323,11 +298,11 @@ stringData: If those passwords already live in Secrets in your infrastructure namespace, copy them across instead of retyping them: ```sh -NS_INFRA= # where your L2 services run -kubectl -n $NS_INFRA get secret -o jsonpath='{.data.}' | base64 -d; echo # clickhouse-password -kubectl -n $NS_INFRA get secret -o jsonpath='{.data.}' | base64 -d; echo # mariadb-password (app user) -kubectl -n $NS_INFRA get secret -o jsonpath='{.data.}' | base64 -d; echo # mariadb-root-password -kubectl -n $NS_INFRA get secret -o jsonpath='{.data.}' | base64 -d; echo # redis-password +# each password lives in a Secret in its own datastore's namespace — they need not be the same namespace +kubectl -n get secret -o jsonpath='{.data.}' | base64 -d; echo # clickhouse-password +kubectl -n get secret -o jsonpath='{.data.}' | base64 -d; echo # mariadb-password (app user) +kubectl -n get secret -o jsonpath='{.data.}' | base64 -d; echo # mariadb-root-password +kubectl -n get secret -o jsonpath='{.data.}' | base64 -d; echo # redis-password ``` Paste each decoded value into the matching field. @@ -366,8 +341,8 @@ kubectl -n insight get secret insight-db-creds insight-authenticator-signing-key Mirror Airbyte's auth Secret into `insight` — Analytics needs it to call the Airbyte API: ```sh -NS_INFRA= -kubectl -n $NS_INFRA get secret airbyte-auth-secrets -o json \ +NS_AIRBYTE= +kubectl -n $NS_AIRBYTE get secret airbyte-auth-secrets -o json \ | jq 'del(.metadata.uid,.metadata.resourceVersion,.metadata.creationTimestamp,.metadata.ownerReferences,.metadata.annotations,.metadata.labels) | .metadata.namespace="insight"' \ | kubectl -n insight apply -f - ``` @@ -421,7 +396,7 @@ See [deploy/CONNECTORS.md](./CONNECTORS.md) for the connector list and a copy-pa |---------|-----------------| | `insight-analytics` / `insight-identity` stuck in `CreateContainerConfigError` | The chart could not compose the `*-config` Secrets. Confirm `insight-db-creds` has all four keys and carries **no** `app.kubernetes.io/managed-by: Helm` label: `kubectl -n insight get secret insight-db-creds -o yaml \| grep managed-by` should return nothing | | `helm install`/`upgrade` fails with `signingKeysSecret is required` or the authenticator pod won't mount its keys | The Secret named in `authenticator.signingKeysSecret` (default `insight-authenticator-signing-keys`) doesn't exist or is missing `current.pem`. Create it as shown in Step 2 before installing | -| `helm install`/`upgrade` fails with `tlsDiscovery.issuerRef.name is required` or the `insight-authenticator-authn-tls` Certificate never turns `Ready` | cert-manager isn't installed, or the `ClusterIssuer` named in `authenticator.tlsDiscovery.issuerRef.name` (default `local-ca`) doesn't exist. Confirm with `kubectl get clusterissuer local-ca` and `kubectl -n insight describe certificate insight-authenticator-authn-tls` | +| `helm install`/`upgrade` fails with `tlsDiscovery.issuerRef.name is required` or the `insight-authenticator-authn-tls` Certificate never turns `Ready` | cert-manager isn't installed, or the `ClusterIssuer` named in `authenticator.tlsDiscovery.issuerRef.name` doesn't exist in this cluster — the usual cause is leaving the chart's local-sandbox `local-ca` default in place. Confirm with `kubectl get clusterissuer` and `kubectl -n insight describe certificate insight-authenticator-authn-tls` | | `helm install`/`upgrade` fails with `authenticator.oidc.issuerUrl is required` / `redirectUri is required` | Both fields are mandatory (`charts/insight/templates/secrets.yaml` wraps them in `required`). Set real values, or for local/dev only, set `fakeidp.deploy: true` and point `issuerUrl` at the in-cluster fakeidp — never disable auth | | Dashboards show "no peer data" (the benchmark/comparison panel is empty) | After Gold-layer data has loaded, restart Analytics: `kubectl -n insight rollout restart deploy/insight-analytics` | | Login breaks after changing the host | Update `authenticator.oidc.redirectUri` (and `frontend.oidc.issuer`/`clientId` if the IdP changed) in values, `helm upgrade`, then restart the gateway: `kubectl -n insight rollout restart deploy/insight-gateway` | @@ -440,19 +415,20 @@ For connector-syncing problems, see the Troubleshooting section of [deploy/CONNE | `` | `redis.host` | Always external; port fixed at `6379` | | `` | `redpanda.brokers` | Always external; a single comma-separated `host:port` string, not a host/port pair. `9093` for the `redpanda/redpanda` chart's internal listener — read yours in Step 0 | | `` | `ingestion.toolboxImage` | Optional — defaults to the chart appVersion. Drives the ingestion WorkflowTemplates and the ClickHouse gold-view migrate Job | -| `` | `airbyte.apiUrl` | e.g. `http://host:8001` | +| `` | `airbyte.apiUrl` | e.g. `http://host:8001`. Empty falls back to `http://-airbyte-server-svc.:8001`, so it is only safe to omit when Airbyte shares the `insight` namespace | | `` | `ingestion.reconcile.argoInstanceId` | Match the controller's configured `instanceID` (Step 0); empty if unpinned | -| `` | `gateway.image.tag`, `authenticator.image.tag`, `analytics.image.tag`, `identity.image.tag`, `frontend.image.tag` | All five are optional — each falls back to that subchart's `Chart.yaml` appVersion (pinned by the release pipeline). Set them explicitly (recommended) so every service lands on the exact same product build; leaving them blank is safe only when all five subcharts' appVersion are in lockstep in the chart release you install | | `` | `gateway.ingress.host`, `frontend.ingress.host` | Public FQDN, shared by the Gateway and Frontend (`/*` → UI, `/api/*` → Gateway routes to Analytics/Identity) | | `` | `gateway.ingress.tls.secretName` | Kubernetes TLS Secret name | +| `` | `authenticator.tlsDiscovery.issuerRef.name` | A cert-manager `ClusterIssuer` that exists in your cluster; internal cert, so self-signed is fine | | `` | `authenticator.oidc.issuerUrl`, `frontend.oidc.issuer` | Your IdP's issuer URL | | `` / `` | `authenticator.oidc.clientId`/`clientSecret`, `frontend.oidc.clientId` | Your OIDC client / application registration credentials | Other notable (non-placeholder) settings in this file: +- Image tags are omitted deliberately. Each subchart renders `image.tag | default .Chart.AppVersion`, and the release pipeline pins those appVersions in lockstep, so a chart release already carries a coherent set of product images. Set `.image.tag` only to pin one service to a different build. - `credentials.deploymentMode: helm` and `credentials.autoGenerate: true` — this enables the "bring your own" credentials path, where the chart keeps a labelless `insight-db-creds` Secret instead of generating random passwords. - `identity.deploy: true` — required override; the chart's own default is `false`. -- `authenticator.tlsDiscovery.issuerRef.name: local-ca` — the cert-manager `ClusterIssuer` name the JWKS-discovery Certificate is issued from; override to match your cluster's issuer. +- `authenticator.tlsDiscovery.issuerRef.name` — the cert-manager `ClusterIssuer` the JWKS-discovery Certificate is issued from. Always set this: the chart ships `local-ca`, which is the self-signed root that `make bootstrap-cert-manager ENV=local` creates for the local k3s sandbox, not anything a real cluster has. - There is no auth-off toggle anywhere in this chart. `authenticator.oidc.issuerUrl` and `authenticator.oidc.redirectUri` are hard `required` fields — the simplest no-real-IdP path is `fakeidp.deploy: true`, local/dev only; `keycloak.deploy: true` is a heavier bundled alternative (not documented here). ### secrets/insight-db-creds.yaml keys From d6cc3bf4a7ecf44e3bdcae6b0d267b24bee0ad84 Mon Sep 17 00:00:00 2001 From: Konstantin Tursunov Date: Mon, 27 Jul 2026 14:23:19 +0800 Subject: [PATCH 09/14] docs(deploy): make a real IdP a prerequisite and drop the toolbox-image placeholder MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit The runbook treated the bundled dev IdP subcharts as the fallback for a stand with no identity provider, and still asked operators to fill in an image placeholder the chart defaults correctly. - Add a real OIDC identity provider to the cluster-level prerequisites, alongside the ingress controller and cert-manager: OIDC is mandatory in every environment, so the authenticator needs something to log in against. Where a stand has none, the suggested route is installing Keycloak as its own release like any other external dependency — real OIDC, public image, admin console for the realm and confidential client — not the bundled subcharts, which are dev-mode servers on an embedded database with known passwords and are now described that way in the Overview, the Step 1 checklist, the Appendix, and Troubleshooting. - New Step 0 subsection collects the OIDC client details from that IdP (confidential client, redirect https:///auth/callback, issuer/clientId/clientSecret; on Keycloak the issuer is /realms/) and verifies discovery from inside the cluster with an oidc-probe pod, since that is where the authenticator resolves it. - Generalise the issuer troubleshooting row off the bundled chart's values onto any IdP, naming the split-horizon DNS trap (public hostname outside, Service DNS inside) that breaks `iss` validation. - Remove everywhere — skeleton line and both placeholder rows — for the same reason went: ingestion.toolboxImage defaults to the chart's appVersion-pinned toolbox. The Step 4 migrate-hook bullet now refers to that pinned image rather than the removed placeholder. Signed-off-by: Konstantin Tursunov --- deploy/HELM_DEPLOY.md | 32 ++++++++++++++++++++------------ 1 file changed, 20 insertions(+), 12 deletions(-) diff --git a/deploy/HELM_DEPLOY.md b/deploy/HELM_DEPLOY.md index f88b043b2..27b8ae710 100644 --- a/deploy/HELM_DEPLOY.md +++ b/deploy/HELM_DEPLOY.md @@ -17,6 +17,7 @@ This runbook shows a platform or DevOps engineer how to install the Insight busi - [Look up the external service addresses](#look-up-the-external-service-addresses) - [Find the Airbyte API URL](#find-the-airbyte-api-url) - [Compose the Redpanda brokers string](#compose-the-redpanda-brokers-string) + - [Get the OIDC client details](#get-the-oidc-client-details) - [Read the Argo workflow-controller instance ID](#read-the-argo-workflow-controller-instance-id) - [Step 1 — Configure values/umbrella.yaml](#step-1--configure-valuesumbrellayaml) - [Step 2 — Fill the secret files](#step-2--fill-the-secret-files) @@ -45,7 +46,7 @@ Insight reads engineering and collaboration data from your tools (Jira, Slack, G - **Identity** (`insight-identity`, alias `identity`) — resolves people and org data from MariaDB; optional (`identity.deploy`, default `false`). - **Frontend** (`insight-frontend`, alias `frontend`) — the web UI (dashboard); optional (`frontend.deploy`, default `true`). -Two more subcharts exist purely for local/dev use and are off by default: `fakeidp` (alias `fakeidp`, condition `fakeidp.deploy`) and `keycloak` (alias `keycloak`, condition `keycloak.deploy`) — both are bundled OIDC providers for a cluster with no real IdP available. `fakeidp` is the one this runbook documents; `keycloak` is a heavier bundled alternative not covered here. Neither is appropriate for a real environment. +Two more subcharts exist purely for local development and are off by default: `keycloak` (alias `keycloak`, condition `keycloak.deploy`) and `fakeidp` (alias `fakeidp`, condition `fakeidp.deploy`) — dev-mode OIDC servers with an embedded database and known passwords. Neither is covered here, and neither is the IdP for a stand: this runbook expects the real one from [Prerequisites](#cluster-level-dependencies). This path assumes your data infrastructure (ClickHouse, MariaDB, Redis, Redpanda, Airbyte, Argo Workflows) already runs and is reachable from the cluster, in another namespace or external. The chart doesn't stand it up; it only wires the services to it. You supply one values file, secret files, and optionally one Secret per connector (see [deploy/CONNECTORS.md](./CONNECTORS.md)). No GitOps repo, CI, or auto-reconciliation — you run the commands yourself. @@ -63,12 +64,13 @@ This path assumes your data infrastructure (ClickHouse, MariaDB, Redis, Redpanda ### Cluster-level dependencies -Install both of these before the chart — it bundles neither: +Install all three of these before the chart — it bundles none of them: - **An ingress controller.** Install ingress-nginx, or override `gateway.ingress.className` / `frontend.ingress.className` to match what you run. Both default to `className: nginx`. +- **A real OIDC identity provider.** OIDC is mandatory in every environment — there is no auth-off switch — so the authenticator needs an IdP to log in against: Entra ID, Okta, Auth0, or your own. **No IdP on the stand? Install Keycloak as its own release and treat it like any other external dependency** — it is the straightforward choice: a real OIDC implementation, a public image, and an admin console for creating the realm and the confidential client. Give it a hostname the browser *and* the authenticator pod both resolve to the same URL, then read its issuer, client ID, and client secret in Step 0. The chart's bundled `keycloak`/`fakeidp` subcharts are not this: they run dev-mode servers with an embedded database and known passwords, for local development only. - **cert-manager, plus a `ClusterIssuer` of your own.** The authenticator's TLS-discovery sidecar (`authenticator.tlsDiscovery.enabled`, default `true`) creates a `cert-manager.io/v1` `Certificate`, and Analytics and Identity verify the authenticator's JWKS over HTTPS against that CA — load-bearing, not optional. Set `authenticator.tlsDiscovery.issuerRef.name` to whatever your cluster issues from. Do not expect the chart's `local-ca` default to resolve: that issuer belongs to this repo's local k3s sandbox (`deploy/gitops/bootstrap/local/selfsigned-issuer.yaml`, applied by `make bootstrap-cert-manager ENV=local`) and will not exist in a cluster you did not bootstrap that way. Any issuer will do — this certificate is internal-only, trusted through the CA the services mount, so it needs no public chain and is unrelated to the public ingress certificate in ``. -Confirm both are in place: +Confirm the cluster-side pieces (the IdP gets verified in Step 0, once you have its issuer URL): ```sh kubectl get ingressclass nginx # the className both ingress blocks use @@ -130,6 +132,16 @@ kubectl -n get svc -o jsonpath='{range .spec.ports[ # e.g. redpanda.insight-infra.svc.cluster.local:9093 ``` +### Get the OIDC client details + +In the IdP from Prerequisites, register a **confidential** client whose redirect URI is `https:///auth/callback`, and collect its issuer URL, client ID, and client secret; on Keycloak the issuer is `/realms/`, and the client's *Credentials* tab holds the secret. + +```sh +# the issuer must be the SAME URL the browser and the authenticator pod resolve, or `iss` won't validate +kubectl run oidc-probe --rm -i --restart=Never --image=curlimages/curl -- \ + curl -sS /.well-known/openid-configuration | head -c 200 +``` + ### Read the Argo workflow-controller instance ID Set `ingestion.reconcile.argoInstanceId` to the controller's configured `instanceID` only when it is pinned to one; no match means leave it empty — the common case, where the reconcile workflows go unlabelled and any controller accepts them. @@ -177,7 +189,6 @@ redpanda: ingestion: templates: enabled: true - # toolboxImage: "" # optional — defaults to the chart's appVersion-pinned toolbox; only set to override reconcile: tenantId: "" destinationName: clickhouse-bronze @@ -240,8 +251,6 @@ frontend: # the web UI (dashboard) issuer: "" # same IdP as the authenticator clientId: "" scopes: "openid profile email" # IdP-specific - -# fakeidp: {deploy: false} # local/dev-only alternative to a real IdP — see note below ``` To start from the chart's full defaults instead of typing the skeleton: @@ -259,7 +268,6 @@ Fill each placeholder: | `` | MariaDB host, in `host:3306` form | | `` | Redis host, in `host:6379` form | | `` | The bootstrap string you composed in Step 0 — comma-separated `host:port` pointing at the internal Kafka API listener | -| `` | Optional. The ingestion toolbox image (drives the WorkflowTemplates and the ClickHouse gold-view migration Job, Step 4/5). Omit to inherit the chart's default, which is pinned to the chart appVersion; set only to override | | `` | The Airbyte server Service URL from Step 0, for example `http://host:8001`. Omit only when Airbyte is a release in the `insight` namespace — the chart then computes it from `airbyte.releaseName` | | `` | The `instanceID` your Argo workflow controller is pinned to — read it off the controller config map in Step 0. Leave empty (`""`) if the controller is unpinned, the common case | | `` | Public FQDN for the ingress, shared by the Gateway and Frontend, for example `insight.example.com` | @@ -275,7 +283,7 @@ Check these four before installing: - Set `identity.deploy: true`. The chart default is `false`, and without the override Identity — and person resolution for the whole app — never deploys. - Set real values for `authenticator.oidc.issuerUrl` and `redirectUri`. The chart wraps both in Helm's `required`, and there is no auth-off switch. - Create the Secret named in `authenticator.signingKeysSecret` before installing (Step 2). The chart does not generate it. -- No real IdP, local/dev only: set `fakeidp.deploy: true`, point `issuerUrl` at the in-cluster fakeidp FQDN, leave `clientSecret` empty. Its image is not on public GHCR, so build and load it locally (or supply an `imagePullSecret`) or the pod hits `ImagePullBackOff`. Never in a shared cluster. +- Point the OIDC fields at the real IdP from Prerequisites. The bundled `keycloak`/`fakeidp` subcharts are dev-mode servers for local development, not a stand's IdP. ## Step 2 — Fill the secret files @@ -360,7 +368,7 @@ helm upgrade --install insight oci://ghcr.io/constructorfabric/charts/insight \ - Add `--version ` to pin a chart release; omit it for the latest published one. - `--wait --timeout 15m` blocks until every resource is ready, giving a pass/fail signal instead of a detached rollout. -- The install also runs the `insight-clickhouse-migrate` hook Job, which applies the ClickHouse gold-view migrations (`src/ingestion/scripts/migrations/*.sql`) with `ingestion.toolboxImage`. It fires on **every** upgrade, not just the first install (gated by `clickhouse.runMigrations`, default `true`), and a failing migration fails the whole upgrade. It drops and recreates every gold object each run, so a failure points at Bronze/Silver schema or data, not a stale-object conflict. +- The install also runs the `insight-clickhouse-migrate` hook Job, which applies the ClickHouse gold-view migrations (`src/ingestion/scripts/migrations/*.sql`) using the chart's pinned toolbox image. It fires on **every** upgrade, not just the first install (gated by `clickhouse.runMigrations`, default `true`), and a failing migration fails the whole upgrade. It drops and recreates every gold object each run, so a failure points at Bronze/Silver schema or data, not a stale-object conflict. ## Step 5 — Verify the install @@ -397,7 +405,8 @@ See [deploy/CONNECTORS.md](./CONNECTORS.md) for the connector list and a copy-pa | `insight-analytics` / `insight-identity` stuck in `CreateContainerConfigError` | The chart could not compose the `*-config` Secrets. Confirm `insight-db-creds` has all four keys and carries **no** `app.kubernetes.io/managed-by: Helm` label: `kubectl -n insight get secret insight-db-creds -o yaml \| grep managed-by` should return nothing | | `helm install`/`upgrade` fails with `signingKeysSecret is required` or the authenticator pod won't mount its keys | The Secret named in `authenticator.signingKeysSecret` (default `insight-authenticator-signing-keys`) doesn't exist or is missing `current.pem`. Create it as shown in Step 2 before installing | | `helm install`/`upgrade` fails with `tlsDiscovery.issuerRef.name is required` or the `insight-authenticator-authn-tls` Certificate never turns `Ready` | cert-manager isn't installed, or the `ClusterIssuer` named in `authenticator.tlsDiscovery.issuerRef.name` doesn't exist in this cluster — the usual cause is leaving the chart's local-sandbox `local-ca` default in place. Confirm with `kubectl get clusterissuer` and `kubectl -n insight describe certificate insight-authenticator-authn-tls` | -| `helm install`/`upgrade` fails with `authenticator.oidc.issuerUrl is required` / `redirectUri is required` | Both fields are mandatory (`charts/insight/templates/secrets.yaml` wraps them in `required`). Set real values, or for local/dev only, set `fakeidp.deploy: true` and point `issuerUrl` at the in-cluster fakeidp — never disable auth | +| `helm install`/`upgrade` fails with `authenticator.oidc.issuerUrl is required` / `redirectUri is required` | Both fields are mandatory (`charts/insight/templates/secrets.yaml` wraps them in `required`). Set them to your IdP's real values — a real IdP is a prerequisite, and there is no way to disable auth | +| Login fails with an issuer/`iss` mismatch, or discovery 404s | `authenticator.oidc.issuerUrl` must be the issuer the IdP actually advertises, resolving to the *same* URL from the browser and from the authenticator pod — a split-horizon setup (public hostname outside, Service DNS inside) breaks `iss` validation. On Keycloak the issuer is `/realms/`, and the base URL must match what the server advertises. Check with the `oidc-probe` command from Step 0 | | Dashboards show "no peer data" (the benchmark/comparison panel is empty) | After Gold-layer data has loaded, restart Analytics: `kubectl -n insight rollout restart deploy/insight-analytics` | | Login breaks after changing the host | Update `authenticator.oidc.redirectUri` (and `frontend.oidc.issuer`/`clientId` if the IdP changed) in values, `helm upgrade`, then restart the gateway: `kubectl -n insight rollout restart deploy/insight-gateway` | @@ -414,7 +423,6 @@ For connector-syncing problems, see the Troubleshooting section of [deploy/CONNE | `` | `mariadb.host` | Always external; port fixed at `3306` | | `` | `redis.host` | Always external; port fixed at `6379` | | `` | `redpanda.brokers` | Always external; a single comma-separated `host:port` string, not a host/port pair. `9093` for the `redpanda/redpanda` chart's internal listener — read yours in Step 0 | -| `` | `ingestion.toolboxImage` | Optional — defaults to the chart appVersion. Drives the ingestion WorkflowTemplates and the ClickHouse gold-view migrate Job | | `` | `airbyte.apiUrl` | e.g. `http://host:8001`. Empty falls back to `http://-airbyte-server-svc.:8001`, so it is only safe to omit when Airbyte shares the `insight` namespace | | `` | `ingestion.reconcile.argoInstanceId` | Match the controller's configured `instanceID` (Step 0); empty if unpinned | | `` | `gateway.ingress.host`, `frontend.ingress.host` | Public FQDN, shared by the Gateway and Frontend (`/*` → UI, `/api/*` → Gateway routes to Analytics/Identity) | @@ -429,7 +437,7 @@ Other notable (non-placeholder) settings in this file: - `credentials.deploymentMode: helm` and `credentials.autoGenerate: true` — this enables the "bring your own" credentials path, where the chart keeps a labelless `insight-db-creds` Secret instead of generating random passwords. - `identity.deploy: true` — required override; the chart's own default is `false`. - `authenticator.tlsDiscovery.issuerRef.name` — the cert-manager `ClusterIssuer` the JWKS-discovery Certificate is issued from. Always set this: the chart ships `local-ca`, which is the self-signed root that `make bootstrap-cert-manager ENV=local` creates for the local k3s sandbox, not anything a real cluster has. -- There is no auth-off toggle anywhere in this chart. `authenticator.oidc.issuerUrl` and `authenticator.oidc.redirectUri` are hard `required` fields — the simplest no-real-IdP path is `fakeidp.deploy: true`, local/dev only; `keycloak.deploy: true` is a heavier bundled alternative (not documented here). +- There is no auth-off toggle anywhere in this chart. `authenticator.oidc.issuerUrl` and `authenticator.oidc.redirectUri` are hard `required` fields, so a real IdP is a prerequisite; install Keycloak as a separate release if the stand has none. The bundled `keycloak`/`fakeidp` subcharts are local-development servers (embedded database, known passwords) and not a substitute. ### secrets/insight-db-creds.yaml keys From 5ce659739a6b24bb6a0e119086794fed4d60615b Mon Sep 17 00:00:00 2001 From: Konstantin Tursunov Date: Mon, 27 Jul 2026 14:24:24 +0800 Subject: [PATCH 10/14] docs(deploy): drop key-rotation guidance and the orbstack values appendix MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit Both covered situations the runbook does not need to carry right now. - Remove the previous.pem rotation window: the Step 2 rotation command, the appendix key row, and the Step 3 verify comment's "1-2 keys" expectation, which now reads "1 key (current.pem)". The signing-keys appendix is a single sentence rather than a one-row table. - Remove the values/umbrella.orbstack.yaml appendix section and its ToC entry. It described a local OrbStack/k3s variant built on host-less ingress, disabled TLS, and the bundled dev IdP — none of which this runbook points operators at any more. Signed-off-by: Konstantin Tursunov --- deploy/HELM_DEPLOY.md | 22 +++------------------- 1 file changed, 3 insertions(+), 19 deletions(-) diff --git a/deploy/HELM_DEPLOY.md b/deploy/HELM_DEPLOY.md index 27b8ae710..bff6274be 100644 --- a/deploy/HELM_DEPLOY.md +++ b/deploy/HELM_DEPLOY.md @@ -32,7 +32,6 @@ This runbook shows a platform or DevOps engineer how to install the Insight busi - [values/umbrella.yaml placeholders](#valuesumbrellayaml-placeholders) - [secrets/insight-db-creds.yaml keys](#secretsinsight-db-credsyaml-keys) - [secrets/insight-authenticator-signing-keys.yaml keys](#secretsinsight-authenticator-signing-keysyaml-keys) - - [values/umbrella.orbstack.yaml (local variant)](#valuesumbrellaorbstackyaml-local-variant) @@ -326,15 +325,7 @@ openssl ecparam -name prime256v1 -genkey -noout | openssl pkcs8 -topk8 -nocrypt kubectl -n insight create secret generic insight-authenticator-signing-keys --from-file=current.pem ``` -To rotate, keep the outgoing key as `previous.pem` beside the new `current.pem` for at least the JWT TTL plus downstream JWKS-cache age (~65 minutes), then roll the authenticator pods: - -```sh -kubectl -n insight create secret generic insight-authenticator-signing-keys \ - --from-file=current.pem --from-file=previous.pem \ - --dry-run=client -o yaml | kubectl apply -f - -``` - -## Step 3 — Create namespace and apply secrets +## Step 3 — Create namespace, apply secrets, mirror Airbyte auth Create the namespace and apply the secret files: @@ -343,7 +334,7 @@ kubectl create namespace insight kubectl -n insight apply -f secrets/ # verify -kubectl -n insight get secret insight-db-creds insight-authenticator-signing-keys # expect 4 keys / 1-2 keys (current.pem [+ previous.pem]) +kubectl -n insight get secret insight-db-creds insight-authenticator-signing-keys # expect 4 keys / 1 key (current.pem) ``` Mirror Airbyte's auth Secret into `insight` — Analytics needs it to call the Airbyte API: @@ -452,11 +443,4 @@ Recall: this Secret must never carry an `app.kubernetes.io/managed-by: Helm` lab ### secrets/insight-authenticator-signing-keys.yaml keys -| Key | Meaning | -|-----|---------| -| `current.pem` | The active ES256 (EC P-256) signing key, PKCS#8 PEM, unencrypted. Required. | -| `previous.pem` | The outgoing key during a rotation window. Optional; keep it alongside `current.pem` until the JWT TTL plus downstream JWKS-cache age has elapsed | - -### values/umbrella.orbstack.yaml (local variant) - -This file is a pre-filled variant of the values file for local development on OrbStack's bundled k3s cluster, with all infrastructure running in an `insight-infra` namespace. It sets a fixed tenant UUID, in-cluster DNS hosts, an empty (host-less) ingress that matches any `Host` header, disabled TLS, and points the authenticator at the in-cluster `fakeidp`/`keycloak` provider rather than a real IdP. Use it only as a reference for local testing — do not reuse its host-less ingress, disabled TLS, or fake-IdP settings on a shared or production cluster. +One required key, `current.pem`: the active ES256 (EC P-256) signing key, unencrypted PKCS#8 PEM. From 20ffa7df6fe7fd792d439c3debffb47e0976e533 Mon Sep 17 00:00:00 2001 From: Konstantin Tursunov Date: Mon, 27 Jul 2026 14:31:41 +0800 Subject: [PATCH 11/14] docs(deploy): remove the Troubleshooting section MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit Drop the section, its seven symptom rows, the pointer to CONNECTORS.md's own troubleshooting, and the ToC entry. The constraints behind the fail-fast rows still live where they are configured — the managed-by label warning in Step 2, the issuer guidance in Step 0, the ClusterIssuer caveat in Prerequisites — so what goes away is the symptom-to-cause mapping, not the requirements themselves. Signed-off-by: Konstantin Tursunov --- deploy/HELM_DEPLOY.md | 15 --------------- 1 file changed, 15 deletions(-) diff --git a/deploy/HELM_DEPLOY.md b/deploy/HELM_DEPLOY.md index bff6274be..24b5b90b2 100644 --- a/deploy/HELM_DEPLOY.md +++ b/deploy/HELM_DEPLOY.md @@ -27,7 +27,6 @@ This runbook shows a platform or DevOps engineer how to install the Insight busi - [Step 4 — Install with Helm](#step-4--install-with-helm) - [Step 5 — Verify the install](#step-5--verify-the-install) - [Step 6 — Configure connectors (optional)](#step-6--configure-connectors-optional) -- [Troubleshooting](#troubleshooting) - [Appendix — Reference](#appendix--reference) - [values/umbrella.yaml placeholders](#valuesumbrellayaml-placeholders) - [secrets/insight-db-creds.yaml keys](#secretsinsight-db-credsyaml-keys) @@ -389,20 +388,6 @@ Configure connectors after the app is up. Each of the 25 connectors is a single See [deploy/CONNECTORS.md](./CONNECTORS.md) for the connector list and a copy-paste Secret for each. -## Troubleshooting - -| Problem | What to check | -|---------|-----------------| -| `insight-analytics` / `insight-identity` stuck in `CreateContainerConfigError` | The chart could not compose the `*-config` Secrets. Confirm `insight-db-creds` has all four keys and carries **no** `app.kubernetes.io/managed-by: Helm` label: `kubectl -n insight get secret insight-db-creds -o yaml \| grep managed-by` should return nothing | -| `helm install`/`upgrade` fails with `signingKeysSecret is required` or the authenticator pod won't mount its keys | The Secret named in `authenticator.signingKeysSecret` (default `insight-authenticator-signing-keys`) doesn't exist or is missing `current.pem`. Create it as shown in Step 2 before installing | -| `helm install`/`upgrade` fails with `tlsDiscovery.issuerRef.name is required` or the `insight-authenticator-authn-tls` Certificate never turns `Ready` | cert-manager isn't installed, or the `ClusterIssuer` named in `authenticator.tlsDiscovery.issuerRef.name` doesn't exist in this cluster — the usual cause is leaving the chart's local-sandbox `local-ca` default in place. Confirm with `kubectl get clusterissuer` and `kubectl -n insight describe certificate insight-authenticator-authn-tls` | -| `helm install`/`upgrade` fails with `authenticator.oidc.issuerUrl is required` / `redirectUri is required` | Both fields are mandatory (`charts/insight/templates/secrets.yaml` wraps them in `required`). Set them to your IdP's real values — a real IdP is a prerequisite, and there is no way to disable auth | -| Login fails with an issuer/`iss` mismatch, or discovery 404s | `authenticator.oidc.issuerUrl` must be the issuer the IdP actually advertises, resolving to the *same* URL from the browser and from the authenticator pod — a split-horizon setup (public hostname outside, Service DNS inside) breaks `iss` validation. On Keycloak the issuer is `/realms/`, and the base URL must match what the server advertises. Check with the `oidc-probe` command from Step 0 | -| Dashboards show "no peer data" (the benchmark/comparison panel is empty) | After Gold-layer data has loaded, restart Analytics: `kubectl -n insight rollout restart deploy/insight-analytics` | -| Login breaks after changing the host | Update `authenticator.oidc.redirectUri` (and `frontend.oidc.issuer`/`clientId` if the IdP changed) in values, `helm upgrade`, then restart the gateway: `kubectl -n insight rollout restart deploy/insight-gateway` | - -For connector-syncing problems, see the Troubleshooting section of [deploy/CONNECTORS.md](./CONNECTORS.md). - ## Appendix — Reference ### values/umbrella.yaml placeholders From 57598962dabb45e156bcab7b5424c174c38357ff Mon Sep 17 00:00:00 2001 From: Konstantin Tursunov Date: Mon, 27 Jul 2026 18:56:34 +0800 Subject: [PATCH 12/14] docs(deploy): correct the runbook against published chart 0.4.43 MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit Validated every claim in HELM_DEPLOY.md against the published umbrella chart 0.4.43 (pulled from ghcr, not the working tree, which is behind main) and against the services' own code, then applied the confirmed fixes. The doc's own values skeleton, extracted mechanically and rendered against 0.4.43, produces 34 objects with exit 0 after every edit here. Wrong instructions, in order of what they cost an operator: - The host placeholder rows told operators to fill `host:8123` form, but the chart appends the separate `port:` key itself, so every composed DSN got a doubled port (`http://host:8123:8123`) — render exits 0 and it only breaks inside pods. Hosts are now documented as host-only. - Nothing said the MariaDB `insight` database and login must pre-exist; the chart's hook creates only Identity's `identity`, and Analytics' blocking migrate initContainer needs the other one, so `--wait` burned its whole timeout and failed. Added to Prerequisites with the SQL. Filed as constructorfabric/insight#1945 — the chart should create it, like it does for identity. - Step 2 created a Secret in the `insight` namespace one step before Step 3 created the namespace, and named a `secrets/` file it never wrote. The signing key is now written to that file with `--dry-run=client -o yaml`, so `apply -f secrets/` creates it, the heading is true, and a re-run is idempotent. - Step 5 said the ClickHouse migrate Job "must be Complete", but Helm deletes that hook Job on success, so the check failed on every healthy install. Replaced with `helm history` and the real pass signal. - `frontend.ingress` set a `host:` and `tls:` the frontend subchart's template does not read, publishing a hostless TLS-less catch-all next to the gateway Ingress. Dropped; the gateway owns the only Ingress (render goes from 2 Ingresses to 1). - `frontend.oidc` documented the pre-BFF login model ADR-0001 retired; the authenticator is the only OIDC client. Dropped, with both Appendix rows. - `` and `` removed: subcharts render `image.tag | default .Chart.AppVersion`. The Appendix no longer claims those appVersions move in lockstep — 0.4.43 bundles three different build stamps. - `local-ca` was presented as an issuer to provision; it only exists in this repo's local k3s sandbox. Now a `` placeholder, with `issuerRef.kind` documented. - Corrected consumer attributions: Identity fetches JWKS over plain HTTP with no CA (only Analytics verifies against it), the Airbyte auth Secret is read by the reconcile loop and airbyte-sync rather than Analytics, `clickhouse-password` belongs to `clickhouse.username` rather than an admin, `mariadb-root-password` is the account literally named `root`, and the managed-by label aborts the install rather than overwriting passwords. Undocumented prerequisites and constraints now stated: a real OIDC IdP, the `argo-workflow` ServiceAccount and executor RBAC, the Argo CRDs at >= 3.5, the ClickHouse CREATE DATABASE grant, the URL-reserved-character rule on all four passwords, `clickhouse.nativePort`, the `authn-tls` Certificate that blocks both pods, `csrfOrigins`, and that `global.tenantDefaultId` is a claim-absent fallback. `offline_access` is now requested by default. Then a simplification pass over the prose the corrections had bloated, verified by a two-lens fact diff against the previous commit: prerequisites became a three-item "yours to create" list, the CLI tools collapsed from seven bullets to three, the six infra-table cells and Step 0's four subsections were re-cut, and the duplicated "your infrastructure already runs" paragraph came out. That check caught one bug of its own — a truncated `tlsDiscovery.enabled` that named no real values path — now restored to `authenticator.tlsDiscovery.enabled`. Signed-off-by: Konstantin Tursunov --- deploy/HELM_DEPLOY.md | 140 +++++++++++++++++++++++------------------- 1 file changed, 77 insertions(+), 63 deletions(-) diff --git a/deploy/HELM_DEPLOY.md b/deploy/HELM_DEPLOY.md index 24b5b90b2..50b3fbcc0 100644 --- a/deploy/HELM_DEPLOY.md +++ b/deploy/HELM_DEPLOY.md @@ -36,7 +36,7 @@ This runbook shows a platform or DevOps engineer how to install the Insight busi ## Overview -Insight reads engineering and collaboration data from your tools (Jira, Slack, GitHub, and so on), pipelines it through ClickHouse, and serves metrics to a dashboard behind an OIDC (OpenID Connect, the login protocol) login. It installs as five first-party services in one Helm "umbrella" chart — bundled sub-charts, so a single `helm install` deploys everything — published at `oci://ghcr.io/constructorfabric/charts/insight` (the chart source also lives in-repo at `charts/insight`, which you can install from directly with `helm install insight ./charts/insight` instead of the OCI form). The five are: +Insight reads engineering and collaboration data from your tools (Jira, Slack, GitHub, and so on), pipelines it through ClickHouse, and serves metrics to a dashboard behind an OIDC (OpenID Connect, the login protocol) login. It installs as five first-party services in one Helm "umbrella" chart — bundled sub-charts, so a single `helm install` deploys everything — published at `oci://ghcr.io/constructorfabric/charts/insight`. The five are: - **Gateway** (`insight-gateway`, alias `gateway`) — the OpenResty edge. It owns the public ingress and is the single entrance to the cluster: it routes `/*` to the Frontend and `/api/*` to Analytics/Identity, performing a cached cookie-to-JWT exchange against the Authenticator's `/internal/authz` endpoint (a per-pod Lua cosocket lookup, not nginx's `auth_request`) and injecting the resulting gateway JWT into upstream requests. - **Authenticator** (`insight-authenticator`, alias `authenticator`) — a separate pod that performs the OIDC login with your IdP, keeps Redis-backed sessions, and mints the ES256 gateway JWT the Gateway injects downstream. @@ -44,60 +44,74 @@ Insight reads engineering and collaboration data from your tools (Jira, Slack, G - **Identity** (`insight-identity`, alias `identity`) — resolves people and org data from MariaDB; optional (`identity.deploy`, default `false`). - **Frontend** (`insight-frontend`, alias `frontend`) — the web UI (dashboard); optional (`frontend.deploy`, default `true`). -Two more subcharts exist purely for local development and are off by default: `keycloak` (alias `keycloak`, condition `keycloak.deploy`) and `fakeidp` (alias `fakeidp`, condition `fakeidp.deploy`) — dev-mode OIDC servers with an embedded database and known passwords. Neither is covered here, and neither is the IdP for a stand: this runbook expects the real one from [Prerequisites](#cluster-level-dependencies). +A sixth first-party subchart, `insight-identity-resolution` (alias `identityResolution`), is bundled but off by default and is not part of this install — leave `identityResolution.deploy: false`. -This path assumes your data infrastructure (ClickHouse, MariaDB, Redis, Redpanda, Airbyte, Argo Workflows) already runs and is reachable from the cluster, in another namespace or external. The chart doesn't stand it up; it only wires the services to it. You supply one values file, secret files, and optionally one Secret per connector (see [deploy/CONNECTORS.md](./CONNECTORS.md)). No GitOps repo, CI, or auto-reconciliation — you run the commands yourself. +Two more subcharts are bundled for local development only, off by default: `keycloak` (dev mode, embedded database, known admin login) and `fakeidp` (a stateless stub). Neither is a stand's IdP — this runbook expects the real one from [Prerequisites](#cluster-level-dependencies). + +You supply one values file, secret files, and optionally one Secret per connector (see [deploy/CONNECTORS.md](./CONNECTORS.md)) — no GitOps repo, CI or auto-reconciliation. The data infrastructure is your side of the contract; see [Prerequisites](#running-external-infrastructure). ## Prerequisites ### Cluster and CLI tools -- A Kubernetes cluster you can already reach with `kubectl`, with permission to create namespaces, Secrets, workloads, and Roles/RoleBindings — including in the Airbyte namespace when it differs from the app's (the chart installs a Role there that lets its jobs read Airbyte's auth Secret). That namespace must exist before the umbrella install. -- `helm` ≥ 3.8 (OCI registry support is stable from 3.8 onward, since the chart is pulled as an OCI artifact). -- `kubectl`. -- `jq`, used to mirror the Airbyte auth Secret in Step 3. -- `openssl`, used to generate the authenticator's signing key in Step 2. -- `uuidgen` (or `python3`), used to generate the tenant ID in Step 0. -- `base64`, used when copying existing datastore passwords in Step 2 (most systems ship this by default). +- A Kubernetes cluster you can already reach with `kubectl`, with permission to create namespaces, Secrets, and workloads. +- `helm` ≥ 3.8 — the chart is pulled as an OCI artifact, and OCI support is stable from 3.8 onward. +- `kubectl`, plus `jq`, `openssl`, `uuidgen` (or `python3`) and `base64` for the commands in Steps 0–3. ### Cluster-level dependencies Install all three of these before the chart — it bundles none of them: -- **An ingress controller.** Install ingress-nginx, or override `gateway.ingress.className` / `frontend.ingress.className` to match what you run. Both default to `className: nginx`. -- **A real OIDC identity provider.** OIDC is mandatory in every environment — there is no auth-off switch — so the authenticator needs an IdP to log in against: Entra ID, Okta, Auth0, or your own. **No IdP on the stand? Install Keycloak as its own release and treat it like any other external dependency** — it is the straightforward choice: a real OIDC implementation, a public image, and an admin console for creating the realm and the confidential client. Give it a hostname the browser *and* the authenticator pod both resolve to the same URL, then read its issuer, client ID, and client secret in Step 0. The chart's bundled `keycloak`/`fakeidp` subcharts are not this: they run dev-mode servers with an embedded database and known passwords, for local development only. -- **cert-manager, plus a `ClusterIssuer` of your own.** The authenticator's TLS-discovery sidecar (`authenticator.tlsDiscovery.enabled`, default `true`) creates a `cert-manager.io/v1` `Certificate`, and Analytics and Identity verify the authenticator's JWKS over HTTPS against that CA — load-bearing, not optional. Set `authenticator.tlsDiscovery.issuerRef.name` to whatever your cluster issues from. Do not expect the chart's `local-ca` default to resolve: that issuer belongs to this repo's local k3s sandbox (`deploy/gitops/bootstrap/local/selfsigned-issuer.yaml`, applied by `make bootstrap-cert-manager ENV=local`) and will not exist in a cluster you did not bootstrap that way. Any issuer will do — this certificate is internal-only, trusted through the CA the services mount, so it needs no public chain and is unrelated to the public ingress certificate in ``. +- **An ingress controller.** Install ingress-nginx, or point `gateway.ingress.className` at what you run (default `nginx`). The gateway owns the only Ingress: UI at `/`, APIs under `/api/`. +- **A real OIDC identity provider** — Entra ID, Okta, Auth0, or your own. OIDC is mandatory and there is no auth-off switch. **No IdP on the stand? Install Keycloak as its own release**, on a hostname the browser and the authenticator pod resolve identically, then read its issuer, client ID and client secret in Step 0. The bundled `keycloak`/`fakeidp` subcharts are dev-mode servers for local development, not this. +- **cert-manager, plus a `ClusterIssuer` of your own.** The authenticator's TLS-discovery sidecar (`authenticator.tlsDiscovery.enabled`, default `true`) creates a `cert-manager.io/v1` `Certificate`, and Analytics verifies the authenticator's JWKS against that CA — load-bearing, not optional. Point `authenticator.tlsDiscovery.issuerRef.name` at an issuer your cluster actually has: the chart's `local-ca` default exists only in this repo's local k3s sandbox (`deploy/gitops/bootstrap/local/selfsigned-issuer.yaml`). Any issuer works, self-signed included — the certificate is internal-only and unrelated to the ingress certificate in ``. (Identity fetches JWKS over plain HTTP and trusts no CA, so an Identity token rejection is never a CA problem.) Confirm the cluster-side pieces (the IdP gets verified in Step 0, once you have its issuer URL): ```sh -kubectl get ingressclass nginx # the className both ingress blocks use +kubectl get ingressclass nginx # the className the gateway Ingress uses kubectl get crd certificates.cert-manager.io # cert-manager CRDs installed kubectl get clusterissuer # pick one for tlsDiscovery.issuerRef.name ``` ### Running external infrastructure -All six systems below must be deployed and reachable from the cluster before you start. The chart never installs any of them — ClickHouse, MariaDB, Redis, and Redpanda are wired in purely by host/credentials; Airbyte and Argo Workflows via `airbyte.apiUrl` and `ingestion.reconcile.argoInstanceId`. Step 0 shows how to read those addresses off your cluster. +All six systems below must already run and be reachable from the cluster. The chart installs none of them: ClickHouse, MariaDB, Redis and Redpanda are wired in by host/credentials, Airbyte and Argo via `airbyte.apiUrl` and `ingestion.reconcile.argoInstanceId` — Step 0 reads those addresses off your cluster. + +Argo is the exception to "only wired in": the chart installs WorkflowTemplates and CronWorkflows into `insight`, so its CRDs must be present, at >= 3.5 for the plural `schedules:` field. Without Argo, set `ingestion.templates.enabled: false` or the install fails with `no matches for kind "WorkflowTemplate"`. | System | Used for | |--------|----------| -| ClickHouse | Stores the Bronze (raw ingested data), Silver (cleaned/conformed), and Gold (query-ready) data layers; Analytics reads the Gold layer to serve metrics | -| MariaDB | Owns the `identity` database that Identity uses to resolve people and org data | -| Redis | Caching layer used by Analytics and session storage used by the Authenticator | -| Redpanda | Event-streaming backbone (Kafka-compatible) used by the ingestion pipeline | -| Airbyte | Runs the data connectors (Jira, Slack, GitHub, and so on) that load raw data into ClickHouse Bronze | -| Argo Workflows | Runs the dbt transform workflows that turn Bronze into Silver and Gold, and runs the sync workflows Airbyte connections trigger | +| ClickHouse | The Bronze (raw), Silver (conformed) and Gold (query-ready) layers; Analytics serves metrics from Gold | +| MariaDB | The `identity` database Identity resolves people and org data from | +| Redis | Analytics' cache and the authenticator's session store | +| Redpanda | Kafka-compatible event stream | +| Airbyte | Runs the connectors (Jira, Slack, GitHub, …) that load Bronze | +| Argo Workflows | Runs the dbt transforms Bronze → Silver → Gold, and the Airbyte sync workflows | + +Three things on those systems are yours to create: + +- **MariaDB: the `insight` database and login.** The pre-install hook creates only Identity's `identity`, and Analytics' blocking `migrate` initContainer needs this one — without it `--wait` burns its whole timeout and fails. + + ```sql + CREATE DATABASE IF NOT EXISTS `insight` + CHARACTER SET utf8mb4 COLLATE utf8mb4_unicode_ci; + CREATE USER IF NOT EXISTS `insight`@`%` IDENTIFIED BY ''; + GRANT ALL ON `insight`.* TO `insight`@`%`; + ``` -Run every command below from the directory holding your `values/` and `secrets/` files — Steps 1 and 2 give you the full contents of both. Connector configuration (the `connectors/` directory) comes later, in [deploy/CONNECTORS.md](./CONNECTORS.md). +- **ClickHouse: the `clickhouse.username` account, holding CREATE DATABASE.** The pre-install hook creates the Bronze/Silver/Gold databases as that user; the chart has no separate admin account. +- **Argo: an `argo-workflow` ServiceAccount in `insight`,** plus a Role granting create/patch on `workflowtaskresults.argoproj.io`. The chart pins every ingestion workflow to it and ships neither, so without them dbt transforms and data-quality checks fail with `serviceaccount "argo-workflow" not found` (connector provisioning still works). + +Run every command below from the directory holding your `values/` and `secrets/` files; Steps 1 and 2 give you their full contents. ## Step 0 — Collect the values Step 1 needs -Generate the tenant ID, then read the rest off the cluster — every dependency is external, and each may sit in its own namespace or outside the cluster entirely. +Generate the tenant ID, then read the rest off the cluster. Each dependency may sit in its own namespace, or outside the cluster entirely. ### Generate the tenant ID -A lowercase UUID, used verbatim for both `global.tenantDefaultId` and `ingestion.reconcile.tenantId`, and never changed after the first sync (local/dev against the compose wizard, the seed generators, or `fakeidp` instead reuses their fixed `00000000-df51-5b42-9538-d2b56b7ee953`). +A lowercase UUID, used verbatim for both `global.tenantDefaultId` and `ingestion.reconcile.tenantId`, and never changed after the first sync. (Local/dev against the compose wizard, the seed generators or `fakeidp` reuses their fixed `00000000-df51-5b42-9538-d2b56b7ee953`.) `global.tenantDefaultId` is only a fallback — the request tenant comes from the id_token claim named by `authenticator.oidc.tenantClaim`, default `tenant_id` — so if your IdP asserts that claim, give it this same UUID. ```sh uuidgen | tr '[:upper:]' '[:lower:]' # no uuidgen? python3 -c 'import uuid; print(uuid.uuid4())' @@ -105,7 +119,7 @@ uuidgen | tr '[:upper:]' '[:lower:]' # no uuidgen? python3 -c 'import uuid; p ### Look up the external service addresses -Every host is `..svc.cluster.local` — or any resolvable host or IP for off-cluster infrastructure — and the ClickHouse, MariaDB, and Redis ports are already fixed in the skeleton, so you supply only the host. +Every host is `..svc.cluster.local`, or any resolvable host or IP off-cluster. The ClickHouse, MariaDB and Redis ports are already in the skeleton, so you supply only the host. ```sh kubectl get svc -A | grep -Ei 'clickhouse|mariadb|redis|redpanda|airbyte' @@ -113,7 +127,7 @@ kubectl get svc -A | grep -Ei 'clickhouse|mariadb|redis|redpanda|airbyte' ### Find the Airbyte API URL -`airbyte.apiUrl` is the Airbyte **server** Service on its HTTP port, and you must set it whenever Airbyte runs anywhere other than the `insight` namespace — left empty, the chart computes `http://-airbyte-server-svc..svc.cluster.local:8001`, which only resolves when Airbyte is a release in that same namespace. +`airbyte.apiUrl` is the Airbyte **server** Service on its HTTP port. Set it whenever Airbyte runs outside the `insight` namespace: left empty, the chart computes `http://-airbyte-server-svc..svc.cluster.local:8001`, which resolves only for a release in that same namespace. ```sh kubectl -n get svc | grep server @@ -122,7 +136,7 @@ kubectl -n get svc | grep server ### Compose the Redpanda brokers string -`redpanda.brokers` takes one comma-separated `host:port` bootstrap string — not the host/port pair the other datastores take — aimed at Redpanda's internal Kafka API listener, so read the port rather than assuming `9093` (that is the `redpanda/redpanda` chart's default; this repo's compose stack uses `9092`). +`redpanda.brokers` takes one comma-separated `host:port` bootstrap string, not the host/port pair the other datastores take, aimed at the internal Kafka API listener. Read the port rather than assuming `9093` — that is the `redpanda/redpanda` chart's default, while this repo's compose stack uses `9092`. ```sh kubectl -n get svc -o jsonpath='{range .spec.ports[*]}{.name}={.port}{"\n"}{end}' @@ -132,7 +146,7 @@ kubectl -n get svc -o jsonpath='{range .spec.ports[ ### Get the OIDC client details -In the IdP from Prerequisites, register a **confidential** client whose redirect URI is `https:///auth/callback`, and collect its issuer URL, client ID, and client secret; on Keycloak the issuer is `/realms/`, and the client's *Credentials* tab holds the secret. +In the IdP from Prerequisites, register a **confidential** client with redirect URI `https:///auth/callback` and collect its issuer URL, client ID and client secret. On Keycloak the issuer is `/realms/` and the secret is on the client's *Credentials* tab. ```sh # the issuer must be the SAME URL the browser and the authenticator pod resolve, or `iss` won't validate @@ -163,13 +177,12 @@ credentials: global: tenantDefaultId: "" # the UUID from Step 0; must equal ingestion.reconcile.tenantId - # storageClass: "" # "" = cluster default; e.g. "local-path" locally # imagePullSecrets: [] # [{name: my-regcred}] for a private registry # Datastore wiring — every dep is external; the chart only dials it. clickhouse: host: # e.g. clickhouse..svc.cluster.local - port: 8123 + port: 8123 # reachable from the insight namespace database: insight username: insight mariadb: @@ -223,12 +236,15 @@ authenticator: issuerRef: name: # your cluster's ClusterIssuer; the chart's `local-ca` # default only exists in the local k3s sandbox + kind: ClusterIssuer # or Issuer, if yours is namespaced into `insight` oidc: issuerUrl: "" # MUST be set — your IdP's issuer URL clientId: "" clientSecret: "" redirectUri: "https:///auth/callback" # MUST be set — browser-facing callback through the gateway - scopes: ["openid", "profile", "email"] + scopes: ["openid", "profile", "email", "offline_access"] + # csrfOrigins: ["https://"] # fail-closed by default: if the UI's POST /auth/logout, + # /auth/refresh or DELETE /auth/sessions return 403, set this identity: deploy: true # MUST be true (chart default false) @@ -239,16 +255,8 @@ identity: limits: { cpu: 250m, memory: 384Mi } frontend: # the web UI (dashboard) - deploy: true + deploy: true # served through the gateway at / — no ingress of its own replicaCount: 1 - ingress: - enabled: true # WITHOUT this the UI pod runs but is never exposed - className: nginx - host: # same FQDN as gateway.ingress.host; /api/* → Gateway routes, /* → UI - oidc: # public values; the browser starts the login here - issuer: "" # same IdP as the authenticator - clientId: "" - scopes: "openid profile email" # IdP-specific ``` To start from the chart's full defaults instead of typing the skeleton: @@ -262,14 +270,14 @@ Fill each placeholder: | Placeholder | What it should be | |-------------|--------------------| | `` | The tenant UUID you generated in Step 0. Must be the same value in `global.tenantDefaultId` and `ingestion.reconcile.tenantId` | -| `` | ClickHouse HTTP host, in `host:8123` form | -| `` | MariaDB host, in `host:3306` form | -| `` | Redis host, in `host:6379` form | +| `` | ClickHouse host only — no port, no scheme; the skeleton's `port: 8123` supplies the port | +| `` | MariaDB host only — no port; the skeleton's `port: 3306` supplies it | +| `` | Redis host only — no port; the skeleton's `port: 6379` supplies it | | `` | The bootstrap string you composed in Step 0 — comma-separated `host:port` pointing at the internal Kafka API listener | | `` | The Airbyte server Service URL from Step 0, for example `http://host:8001`. Omit only when Airbyte is a release in the `insight` namespace — the chart then computes it from `airbyte.releaseName` | | `` | The `instanceID` your Argo workflow controller is pinned to — read it off the controller config map in Step 0. Leave empty (`""`) if the controller is unpinned, the common case | -| `` | Public FQDN for the ingress, shared by the Gateway and Frontend, for example `insight.example.com` | -| `` | Name of the Kubernetes TLS Secret that covers that domain | +| `` | Public FQDN for the gateway ingress — the single entrance for both the UI and the APIs, for example `insight.example.com` | +| `` | TLS Secret for that domain. The chart only references it: pre-create it, or add `gateway.ingress.annotations: {cert-manager.io/cluster-issuer: }`. Missing, ingress-nginx quietly serves its own fake certificate | | `` | A cert-manager `ClusterIssuer` in your cluster, for the authenticator's internal JWKS certificate. Self-signed is fine; the chart's `local-ca` default exists only in this repo's local sandbox | | `` | Your IdP's issuer URL. Its `/.well-known/openid-configuration` document must resolve from inside the cluster | | `` / `` | Your OIDC client / application registration credentials | @@ -287,7 +295,7 @@ Check these four before installing: ### secrets/insight-db-creds.yaml -Create this Secret with all four datastore passwords — the chart fails fast if any key is missing. Use the passwords your datastores already run with. +Create this Secret with all four datastore passwords — the chart fails fast if any key is missing. Use the passwords your datastores already run with, but check them first: the chart composes DSNs by string interpolation and rejects any password containing `@ : / ? # %`, so those must be rotated to `[A-Za-z0-9._~-]` before you install. ```yaml apiVersion: v1 @@ -295,9 +303,9 @@ kind: Secret metadata: { name: insight-db-creds, namespace: insight } type: Opaque stringData: - clickhouse-password: "CHANGE_ME" # ClickHouse admin password -> Analytics + clickhouse-password: "CHANGE_ME" # password of clickhouse.username -> Analytics + hooks mariadb-password: "CHANGE_ME" # MariaDB app-user password -> Analytics + Identity - mariadb-root-password: "CHANGE_ME" # MariaDB root password (identity-DB init hook) -> Identity + mariadb-root-password: "CHANGE_ME" # password of the account literally named `root` -> identity-DB init hook redis-password: "CHANGE_ME" # Redis password -> Analytics + Authenticator ``` @@ -313,7 +321,7 @@ kubectl -n get secret -o jsonpath='{.data. **Never label this Secret `app.kubernetes.io/managed-by: Helm`.** The chart reads the label's *absence* as "bring your own" and composes `insight-analytics-config` and `insight-identity-config` from your values; with the label it takes ownership and may overwrite them with generated passwords. +> **Never label this Secret `app.kubernetes.io/managed-by: Helm`.** The chart reads the label's *absence* as "bring your own" and composes `insight-analytics-config` and `insight-identity-config` from your values; with the label Helm claims ownership of a Secret it did not create and the install aborts with `invalid ownership metadata`. ### secrets/insight-authenticator-signing-keys.yaml @@ -321,7 +329,9 @@ Generate the authenticator's ES256 (EC P-256) gateway-JWT key as PKCS#8 and crea ```sh openssl ecparam -name prime256v1 -genkey -noout | openssl pkcs8 -topk8 -nocrypt -out current.pem -kubectl -n insight create secret generic insight-authenticator-signing-keys --from-file=current.pem +kubectl create secret generic insight-authenticator-signing-keys \ + --namespace insight --from-file=current.pem \ + --dry-run=client -o yaml > secrets/insight-authenticator-signing-keys.yaml ``` ## Step 3 — Create namespace, apply secrets, mirror Airbyte auth @@ -336,7 +346,7 @@ kubectl -n insight apply -f secrets/ kubectl -n insight get secret insight-db-creds insight-authenticator-signing-keys # expect 4 keys / 1 key (current.pem) ``` -Mirror Airbyte's auth Secret into `insight` — Analytics needs it to call the Airbyte API: +Mirror Airbyte's auth Secret into `insight` — the reconcile loop and the airbyte-sync workflows read it to call the Airbyte API: ```sh NS_AIRBYTE= @@ -358,7 +368,8 @@ helm upgrade --install insight oci://ghcr.io/constructorfabric/charts/insight \ - Add `--version ` to pin a chart release; omit it for the latest published one. - `--wait --timeout 15m` blocks until every resource is ready, giving a pass/fail signal instead of a detached rollout. -- The install also runs the `insight-clickhouse-migrate` hook Job, which applies the ClickHouse gold-view migrations (`src/ingestion/scripts/migrations/*.sql`) using the chart's pinned toolbox image. It fires on **every** upgrade, not just the first install (gated by `clickhouse.runMigrations`, default `true`), and a failing migration fails the whole upgrade. It drops and recreates every gold object each run, so a failure points at Bronze/Silver schema or data, not a stale-object conflict. +- The install also runs the `insight-clickhouse-migrate` hook Job. It creates the staging/silver/app databases, seeds bronze placeholders, ALTERs bronze/silver tables, applies `src/ingestion/scripts/migrations/*.sql`, then rebuilds the gold models with dbt — that last part dominates on a cluster with real data, so raise `--timeout` if it runs close. It fires on **every** upgrade (gated by `clickhouse.runMigrations`, default `true`) and a failure fails the upgrade. Because it drops and recreates every gold object each run, a failure points at Bronze/Silver schema or data, not a stale-object conflict. +- If `--wait` stalls with the authenticator and analytics in `ContainerCreating`, the cert-manager Certificate `insight-authenticator-authn-tls` has not issued — `--wait` does not wait on Certificates. Check `kubectl -n insight describe certificate insight-authenticator-authn-tls`. ## Step 5 — Verify the install @@ -372,12 +383,15 @@ kubectl -n insight get pods kubectl -n insight get secret insight-analytics-config insight-authenticator-config insight-identity-config # the chart composes these from insight-db-creds (the identity one only when identity.deploy=true) -kubectl -n insight get jobs -l app.kubernetes.io/component=clickhouse-migrate -kubectl -n insight logs job/insight-clickhouse-migrate - # the gold-view migration Job must be Complete +helm -n insight history insight + # the ClickHouse migration runs as a post-install/post-upgrade hook Job; Helm deletes it + # on success (hook-delete-policy: hook-succeeded), so "no jobs found" is the healthy + # state — Step 4 exiting 0 is the pass signal. On failure the Job survives: + # kubectl -n insight logs job/insight-clickhouse-migrate kubectl -n insight get cronworkflow - # expect insight-reconcile-loop (provisions Airbyte sources/connections) + # expect two: insight-reconcile-loop (provisions Airbyte sources/connections) + # and insight-data-quality (ingestion.dataQuality.enabled, default true) ``` Then open `https://` — the host from Step 1 — and confirm the login redirect to your OIDC provider. @@ -401,15 +415,15 @@ See [deploy/CONNECTORS.md](./CONNECTORS.md) for the connector list and a copy-pa | `` | `redpanda.brokers` | Always external; a single comma-separated `host:port` string, not a host/port pair. `9093` for the `redpanda/redpanda` chart's internal listener — read yours in Step 0 | | `` | `airbyte.apiUrl` | e.g. `http://host:8001`. Empty falls back to `http://-airbyte-server-svc.:8001`, so it is only safe to omit when Airbyte shares the `insight` namespace | | `` | `ingestion.reconcile.argoInstanceId` | Match the controller's configured `instanceID` (Step 0); empty if unpinned | -| `` | `gateway.ingress.host`, `frontend.ingress.host` | Public FQDN, shared by the Gateway and Frontend (`/*` → UI, `/api/*` → Gateway routes to Analytics/Identity) | -| `` | `gateway.ingress.tls.secretName` | Kubernetes TLS Secret name | +| `` | `gateway.ingress.host` | Public FQDN on the gateway's Ingress, the only one the chart publishes (`/` → UI, `/api/*` → Analytics/Identity) | +| `` | `gateway.ingress.tls.secretName` | Kubernetes TLS Secret name; referenced only — the chart never creates it | | `` | `authenticator.tlsDiscovery.issuerRef.name` | A cert-manager `ClusterIssuer` that exists in your cluster; internal cert, so self-signed is fine | -| `` | `authenticator.oidc.issuerUrl`, `frontend.oidc.issuer` | Your IdP's issuer URL | -| `` / `` | `authenticator.oidc.clientId`/`clientSecret`, `frontend.oidc.clientId` | Your OIDC client / application registration credentials | +| `` | `authenticator.oidc.issuerUrl` | Your IdP's issuer URL | +| `` / `` | `authenticator.oidc.clientId`/`clientSecret` | Your OIDC client / application registration credentials. The authenticator is the only OIDC client — the frontend does not register one | Other notable (non-placeholder) settings in this file: -- Image tags are omitted deliberately. Each subchart renders `image.tag | default .Chart.AppVersion`, and the release pipeline pins those appVersions in lockstep, so a chart release already carries a coherent set of product images. Set `.image.tag` only to pin one service to a different build. +- Image tags are omitted deliberately. Each subchart renders `image.tag | default .Chart.AppVersion`, so a chart release already carries a tested set of product images. Set `.image.tag` only to pin one service to a different build. - `credentials.deploymentMode: helm` and `credentials.autoGenerate: true` — this enables the "bring your own" credentials path, where the chart keeps a labelless `insight-db-creds` Secret instead of generating random passwords. - `identity.deploy: true` — required override; the chart's own default is `false`. - `authenticator.tlsDiscovery.issuerRef.name` — the cert-manager `ClusterIssuer` the JWKS-discovery Certificate is issued from. Always set this: the chart ships `local-ca`, which is the self-signed root that `make bootstrap-cert-manager ENV=local` creates for the local k3s sandbox, not anything a real cluster has. @@ -419,9 +433,9 @@ Other notable (non-placeholder) settings in this file: | Key | Meaning | Consumed by | |-----|---------|--------------| -| `clickhouse-password` | ClickHouse admin password | Analytics | +| `clickhouse-password` | Password of the `clickhouse.username` account | Analytics, Identity, and the init/migrate hooks | | `mariadb-password` | MariaDB app-user password | Analytics + Identity | -| `mariadb-root-password` | MariaDB root password, used by the identity-DB init hook | Identity | +| `mariadb-root-password` | Password of the account literally named `root` — the hook runs `mariadb -uroot` and no values key renames it. A wrong one surfaces as `MariaDB did not become reachable within 2 minutes`, not as an auth error | identity-DB init hook | | `redis-password` | Redis password | Analytics + Authenticator | Recall: this Secret must never carry an `app.kubernetes.io/managed-by: Helm` label. From eb9a22d0221ffa1daa89e6bd2b18f9ffc52d1ef2 Mon Sep 17 00:00:00 2001 From: Konstantin Tursunov Date: Mon, 27 Jul 2026 19:11:30 +0800 Subject: [PATCH 13/14] docs(deploy): note in the skeleton that the gateway Ingress carries all routing MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit The gateway.ingress block gave no hint where traffic goes after the edge, so the skeleton read as if a route table were missing. It is not: insight-gateway's own values template every upstream through `tpl` off the release name — frontUrl to `-frontend`, /api/analytics to `-analytics:8081`, /api/identity to `-identity:8082` — verified by rendering under two different release names. Nothing for the operator to set, so the comment just says so. Signed-off-by: Konstantin Tursunov --- deploy/HELM_DEPLOY.md | 6 +++--- 1 file changed, 3 insertions(+), 3 deletions(-) diff --git a/deploy/HELM_DEPLOY.md b/deploy/HELM_DEPLOY.md index 50b3fbcc0..066e76945 100644 --- a/deploy/HELM_DEPLOY.md +++ b/deploy/HELM_DEPLOY.md @@ -215,9 +215,9 @@ analytics: gateway: replicaCount: 1 - ingress: - enabled: true - className: nginx + ingress: # the only Ingress the chart publishes; the gateway + enabled: true # itself routes / to the UI and /api/* to Analytics + className: nginx # and Identity, from subchart defaults you need not set host: tls: enabled: true From 1132b62c8e7014740524b4edd08b6bafddcffc37 Mon Sep 17 00:00:00 2001 From: Konstantin Tursunov Date: Mon, 27 Jul 2026 19:35:41 +0800 Subject: [PATCH 14/14] docs(deploy): adopt airbyte.namespace after rebase onto chart 0.4.47 MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit Rebasing onto main brought in #1935, which retires the manual Airbyte auth Secret mirror and reads the Secret from Airbyte's own namespace instead. The rebase resolved HELM_DEPLOY.md in favour of this branch's rewrite, so that change had to be re-applied on top rather than inherited. - Step 3 drops the kubectl|jq mirror and says plainly not to copy `airbyte-auth-secrets`: the reconcile loop and airbyte-sync read it from Airbyte's namespace at run time, and a copy freezes credentials Airbyte regenerates on reinstall. Step 3's heading and the ToC follow. - The skeleton sets `airbyte.namespace` and leaves `apiUrl` empty: `insight.airbyte.url` now composes the URL from `releaseName` + that namespace (_helpers.tpl:113-123), so an explicit apiUrl is only for a non-standard URL. - Step 0's Airbyte subsection asks for the namespace rather than the URL, and both placeholder tables swap `` for ``. - Prerequisites now require permission to create Roles/RoleBindings, including in Airbyte's namespace, which must exist before the install — the chart renders `insight-airbyte-auth-reader` there. - `jq` leaves the CLI tool list; nothing else in the runbook uses it. Verified against published 0.4.47: the doc's own skeleton renders 36 objects, exit 0, one Ingress, no doubled ports, the computed URL resolves to airbyte-airbyte-server-svc in the namespace given, and the Role lands in that namespace scoped to `get` on `airbyte-auth-secrets` alone. Signed-off-by: Konstantin Tursunov --- deploy/HELM_DEPLOY.md | 41 ++++++++++++++++------------------------- 1 file changed, 16 insertions(+), 25 deletions(-) diff --git a/deploy/HELM_DEPLOY.md b/deploy/HELM_DEPLOY.md index 066e76945..6a1b08117 100644 --- a/deploy/HELM_DEPLOY.md +++ b/deploy/HELM_DEPLOY.md @@ -15,7 +15,7 @@ This runbook shows a platform or DevOps engineer how to install the Insight busi - [Step 0 — Collect the values Step 1 needs](#step-0--collect-the-values-step-1-needs) - [Generate the tenant ID](#generate-the-tenant-id) - [Look up the external service addresses](#look-up-the-external-service-addresses) - - [Find the Airbyte API URL](#find-the-airbyte-api-url) + - [Find the Airbyte namespace](#find-the-airbyte-namespace) - [Compose the Redpanda brokers string](#compose-the-redpanda-brokers-string) - [Get the OIDC client details](#get-the-oidc-client-details) - [Read the Argo workflow-controller instance ID](#read-the-argo-workflow-controller-instance-id) @@ -23,7 +23,7 @@ This runbook shows a platform or DevOps engineer how to install the Insight busi - [Step 2 — Fill the secret files](#step-2--fill-the-secret-files) - [secrets/insight-db-creds.yaml](#secretsinsight-db-credsyaml) - [secrets/insight-authenticator-signing-keys.yaml](#secretsinsight-authenticator-signing-keysyaml) -- [Step 3 — Create namespace, apply secrets, mirror Airbyte auth](#step-3--create-namespace-apply-secrets-mirror-airbyte-auth) +- [Step 3 — Create the namespace and apply the secrets](#step-3--create-the-namespace-and-apply-the-secrets) - [Step 4 — Install with Helm](#step-4--install-with-helm) - [Step 5 — Verify the install](#step-5--verify-the-install) - [Step 6 — Configure connectors (optional)](#step-6--configure-connectors-optional) @@ -54,9 +54,9 @@ You supply one values file, secret files, and optionally one Secret per connecto ### Cluster and CLI tools -- A Kubernetes cluster you can already reach with `kubectl`, with permission to create namespaces, Secrets, and workloads. +- A Kubernetes cluster you can already reach with `kubectl`, with permission to create namespaces, Secrets, workloads, and Roles/RoleBindings — including in Airbyte's namespace when it differs from the app's, where the chart installs a Role letting its jobs read Airbyte's auth Secret. That namespace must exist before you install. - `helm` ≥ 3.8 — the chart is pulled as an OCI artifact, and OCI support is stable from 3.8 onward. -- `kubectl`, plus `jq`, `openssl`, `uuidgen` (or `python3`) and `base64` for the commands in Steps 0–3. +- `kubectl`, plus `openssl`, `uuidgen` (or `python3`) and `base64` for the commands in Steps 0–3. ### Cluster-level dependencies @@ -76,7 +76,7 @@ kubectl get clusterissuer # pick one for tlsDiscovery.issu ### Running external infrastructure -All six systems below must already run and be reachable from the cluster. The chart installs none of them: ClickHouse, MariaDB, Redis and Redpanda are wired in by host/credentials, Airbyte and Argo via `airbyte.apiUrl` and `ingestion.reconcile.argoInstanceId` — Step 0 reads those addresses off your cluster. +All six systems below must already run and be reachable from the cluster. The chart installs none of them: ClickHouse, MariaDB, Redis and Redpanda are wired in by host/credentials, Airbyte and Argo via `airbyte.namespace` and `ingestion.reconcile.argoInstanceId` — Step 0 reads those off your cluster. Argo is the exception to "only wired in": the chart installs WorkflowTemplates and CronWorkflows into `insight`, so its CRDs must be present, at >= 3.5 for the plural `schedules:` field. Without Argo, set `ingestion.templates.enabled: false` or the install fails with `no matches for kind "WorkflowTemplate"`. @@ -125,13 +125,14 @@ Every host is `..svc.cluster.local`, or any resolvable h kubectl get svc -A | grep -Ei 'clickhouse|mariadb|redis|redpanda|airbyte' ``` -### Find the Airbyte API URL +### Find the Airbyte namespace -`airbyte.apiUrl` is the Airbyte **server** Service on its HTTP port. Set it whenever Airbyte runs outside the `insight` namespace: left empty, the chart computes `http://-airbyte-server-svc..svc.cluster.local:8001`, which resolves only for a release in that same namespace. +Set `airbyte.namespace` to the namespace the Airbyte release runs in. It drives both the computed API URL and the namespace the jobs read `airbyte-auth-secrets` from, so `airbyte.apiUrl` stays empty unless your Airbyte sits behind a non-standard URL. ```sh -kubectl -n get svc | grep server - # e.g. http://airbyte-airbyte-server-svc..svc.cluster.local:8001 +kubectl get svc -A | grep airbyte-server + # the namespace in that row is airbyte.namespace + # computed URL: http://-airbyte-server-svc..svc.cluster.local:8001 ``` ### Compose the Redpanda brokers string @@ -205,7 +206,8 @@ ingestion: destinationName: clickhouse-bronze argoInstanceId: "" # match the controller's instanceID (Step 0); leave "" if unpinned airbyte: - apiUrl: "" # required unless Airbyte runs in the `insight` namespace (Step 0) + namespace: "" # where the Airbyte release runs; "" = the app namespace + apiUrl: "" # "" = computed from releaseName + namespace; set only for a non-standard URL analytics: replicaCount: 1 # chart default 2; bump for HA @@ -274,7 +276,7 @@ Fill each placeholder: | `` | MariaDB host only — no port; the skeleton's `port: 3306` supplies it | | `` | Redis host only — no port; the skeleton's `port: 6379` supplies it | | `` | The bootstrap string you composed in Step 0 — comma-separated `host:port` pointing at the internal Kafka API listener | -| `` | The Airbyte server Service URL from Step 0, for example `http://host:8001`. Omit only when Airbyte is a release in the `insight` namespace — the chart then computes it from `airbyte.releaseName` | +| `` | Namespace of the Airbyte release from Step 0, for example `insight-infra`. Leave `""` if Airbyte shares the app namespace | | `` | The `instanceID` your Argo workflow controller is pinned to — read it off the controller config map in Step 0. Leave empty (`""`) if the controller is unpinned, the common case | | `` | Public FQDN for the gateway ingress — the single entrance for both the UI and the APIs, for example `insight.example.com` | | `` | TLS Secret for that domain. The chart only references it: pre-create it, or add `gateway.ingress.annotations: {cert-manager.io/cluster-issuer: }`. Missing, ingress-nginx quietly serves its own fake certificate | @@ -334,9 +336,7 @@ kubectl create secret generic insight-authenticator-signing-keys \ --dry-run=client -o yaml > secrets/insight-authenticator-signing-keys.yaml ``` -## Step 3 — Create namespace, apply secrets, mirror Airbyte auth - -Create the namespace and apply the secret files: +## Step 3 — Create the namespace and apply the secrets ```sh kubectl create namespace insight @@ -346,16 +346,7 @@ kubectl -n insight apply -f secrets/ kubectl -n insight get secret insight-db-creds insight-authenticator-signing-keys # expect 4 keys / 1 key (current.pem) ``` -Mirror Airbyte's auth Secret into `insight` — the reconcile loop and the airbyte-sync workflows read it to call the Airbyte API: - -```sh -NS_AIRBYTE= -kubectl -n $NS_AIRBYTE get secret airbyte-auth-secrets -o json \ - | jq 'del(.metadata.uid,.metadata.resourceVersion,.metadata.creationTimestamp,.metadata.ownerReferences,.metadata.annotations,.metadata.labels) | .metadata.namespace="insight"' \ - | kubectl -n insight apply -f - -``` - -The `jq` filter strips the source object's identity fields (UID, resource version, owner references) and retargets the namespace, so Kubernetes accepts it as a new object. +Do **not** copy Airbyte's own `airbyte-auth-secrets` into `insight`. The reconcile loop and the airbyte-sync workflows read it from Airbyte's namespace at run time — that is what `airbyte.namespace` in Step 1 is for, and the chart renders a Role/RoleBinding there granting `get` on that one Secret. A copy would freeze credentials Airbyte regenerates on reinstall. ## Step 4 — Install with Helm @@ -413,7 +404,7 @@ See [deploy/CONNECTORS.md](./CONNECTORS.md) for the connector list and a copy-pa | `` | `mariadb.host` | Always external; port fixed at `3306` | | `` | `redis.host` | Always external; port fixed at `6379` | | `` | `redpanda.brokers` | Always external; a single comma-separated `host:port` string, not a host/port pair. `9093` for the `redpanda/redpanda` chart's internal listener — read yours in Step 0 | -| `` | `airbyte.apiUrl` | e.g. `http://host:8001`. Empty falls back to `http://-airbyte-server-svc.:8001`, so it is only safe to omit when Airbyte shares the `insight` namespace | +| `` | `airbyte.namespace` | Namespace of the Airbyte release; `""` = app namespace. Drives the computed `apiUrl` and where the jobs read `airbyte-auth-secrets` | | `` | `ingestion.reconcile.argoInstanceId` | Match the controller's configured `instanceID` (Step 0); empty if unpinned | | `` | `gateway.ingress.host` | Public FQDN on the gateway's Ingress, the only one the chart publishes (`/` → UI, `/api/*` → Analytics/Identity) | | `` | `gateway.ingress.tls.secretName` | Kubernetes TLS Secret name; referenced only — the chart never creates it |