diff --git a/README.md b/README.md index 479ba3093..08c975c76 100644 --- a/README.md +++ b/README.md @@ -22,7 +22,7 @@ This repository is the **monorepo** for the Insight product. It contains: - [Connector Coverage](#connector-coverage) - [Key Concepts](#key-concepts) - [Quick Start](#quick-start) - - [Local development (`dev-up.sh`)](#local-development-dev-upsh) + - [Local development (Docker Compose)](#local-development-docker-compose) - [Cluster deployment](#cluster-deployment) - [Configure connectors](#configure-connectors) - [Services and ports](#services-and-ports) @@ -114,13 +114,12 @@ The solution consists of five main components: ### Root scripts ``` -./dev-up.sh ← Create Kind cluster + deploy all services -./init.sh ← Apply secrets + initialize ingestion stack -./dev-down.sh ← Stop services (data preserved) -./cleanup.sh ← Delete cluster and all data -k8s/kind-config.yaml ← Kind cluster configuration +./dev-compose.sh ← Docker Compose dev stack (default laptop path) +deploy/gitops/ ← Kubernetes path: `make deploy ENV=` ``` +See [CONTRIBUTING.md](CONTRIBUTING.md) for both paths in full. + ### `src/` Source code for all platform components. @@ -226,43 +225,26 @@ This repo uses [Cypilot](https://github.com/cyberfabric/cyber-pilot) — an AI a ## Quick Start -Two ways to bring up Insight: +Two supported paths: -- **Local development** — [`./dev-up.sh`](./dev-up.sh) builds a local Kind cluster from source and runs the full stack with hot reload. -- **Cluster deployment** — Cyberfabric engineers use the private `infra/insight-gitops` repository (Makefile-driven, OCI-pinned chart). External consumers of the umbrella Helm chart use it directly via `helm`, ArgoCD, Flux, or whatever orchestrator they already have; the chart contract lives at [`charts/insight/README.md`](charts/insight/README.md). +- **Local development (Docker Compose)** — [`./dev-compose.sh up`](./dev-compose.sh) runs the full stack on a developer laptop with only Docker. Default for day-to-day backend / frontend work. +- **Cluster deployment** — Cyberfabric engineers use the private `infra/insight-gitops` repository (Makefile-driven, OCI-pinned chart); the same path runs locally on a Kind/OrbStack cluster via `cd deploy/gitops && make deploy ENV=local` when you need Airbyte / Argo Workflows or the real cluster shape. External consumers of the umbrella Helm chart use it directly via `helm`, ArgoCD, Flux, or whatever orchestrator they already have; the chart contract lives at [`charts/insight/README.md`](charts/insight/README.md). -Common prerequisites: `kubectl` ≥ 1.27, `helm` ≥ 3.13, `kubeconfig` for the target cluster. Local development also needs `kind` and `docker`. +The two paths share a single first-run wizard, so the MariaDB / ClickHouse / tenant / dev-email answers are identical across them. The full guide for both is [CONTRIBUTING.md](CONTRIBUTING.md). -### Local development (`dev-up.sh`) +### Local development (Docker Compose) -For laptop development. Builds all backend + toolbox images from `src/`, loads them into Kind, deploys the stack with single-replica defaults, and opens stable port-forwards. +For laptop development. No Rust / .NET / Node on the host — every build runs in a builder container; the only prerequisite is Docker (Engine 24+, compose v2). ```bash -brew install kind kubectl helm docker - -cp .env.local.example .env.local -$EDITOR .env.local # AUTH_DISABLED=true OR fill OIDC_* - -./dev-up.sh # creates Kind, builds images, installs everything +git clone https://github.com/constructorfabric/insight.git +cd insight +./dev-compose.sh up # first-run wizard, then builds + seeds the stack ``` -What runs under the hood: -- Creates Kind cluster `insight` if missing. -- Installs `ingress-nginx` (skipped when present). -- Builds backend images (`api-gateway`, `analytics-api`, `identity`, `toolbox`) from local source and `kind load`s them — no registry needed. -- Generates a temporary Helm overlay from `.env.local` and runs `install-airbyte.sh` → `install-argo.sh` → `install-insight.sh` against the Kind cluster (single namespace `insight`). -- Starts port-forwards for every service on stable local ports (see [Services and ports](#services-and-ports)). +The wizard prompts for local-vs-external MariaDB / ClickHouse, a dev-user email, and the frontend mode (defaults pull the published `insight-front` image). First `up` auto-seeds a demo dataset; open . -Lifecycle commands (run from repo root): - -| Command | Description | -|---|---| -| `./dev-up.sh` | Full bring-up (idempotent on re-run) | -| `./dev-up.sh ingestion` | Only Airbyte/Argo/ClickHouse | -| `./dev-up.sh app` | Only application services | -| `./dev-up.sh frontend` | Only frontend (skip backend rebuild) | -| `./dev-down.sh` | Stop port-forwards (Kind cluster + data preserved) | -| `./cleanup.sh` | Delete Kind cluster and all volumes | +The compose stack does **not** ship Airbyte or Argo Workflows — for ingestion work use the Kubernetes path below. See [CONTRIBUTING.md](CONTRIBUTING.md) for the edit-build loop, frontend modes, seeding, and the `.env.compose` settings reference. ### Cluster deployment @@ -315,35 +297,25 @@ kubectl -n insight get workflows -l tenant=default --watch ### Services and ports -For local development (`dev-up.sh`) all services have stable local port-forwards: +For local Docker Compose development every web service publishes a host port (override any `*_PORT` in `.env.compose` on conflict): | Service | URL | Notes | |---|---|---| -| Frontend | http://localhost:8003 | SPA | -| API Gateway | http://localhost:8080 | `/api/v1`; auth disabled when `AUTH_DISABLED=true` | -| Airbyte UI/API | http://localhost:8001 | UI and API on the same port (Airbyte ≥ 1.5) | -| Argo Workflows UI | http://localhost:2746 | `--auth-mode=server` (no Bearer) when `DEV_MODE=1` | +| Frontend | http://localhost:3000 | SPA | +| API Gateway | http://localhost:8080 | `/api/v1`; auth disabled in the `no-auth` config | +| Analytics API | http://localhost:8081 | | +| Identity | http://localhost:8082 | .NET 9 | | ClickHouse HTTP | http://localhost:8123 | `/play` for browser SQL | | MariaDB | localhost:3306 | | | Redis | localhost:6379 | | -For cluster deployments services are reached via the configured ingress hostname (or set up port-forwards manually). +The compose stack does not run Airbyte or Argo Workflows — those live on the Kubernetes path. For cluster deployments services are reached via the configured ingress hostname (or `kubectl port-forward`). ### Image configuration The chart fails fast if any image tag is empty — there are **no `:latest` defaults** anywhere. -For local development (`dev-up.sh`) tags come from `.env.`: - -| `.env` variable | Default | Description | -|---|---|---| -| `IMAGE_REGISTRY` | _(empty)_ | Registry prefix (e.g. `ghcr.io/constructorfabric`); empty for local-only Kind builds | -| `IMAGE_TAG` | `local` | Tag applied to all locally-built images | -| `_IMAGE_TAG` | `$IMAGE_TAG` | Per-service override (`API_GATEWAY_IMAGE_TAG`, …) | -| `IMAGE_PULL_POLICY` | `IfNotPresent` | Kubernetes pullPolicy | -| `BUILD_IMAGES` | `true` | Build images locally before deploy | -| `BUILD_AND_PUSH` | `false` | Push built images to `$IMAGE_REGISTRY` | -| `IMAGE_PLATFORM` | _(empty)_ | Cross-build platform, e.g. `linux/amd64` | +For local Docker Compose development each backend service is built locally by default; skip the build for any service and pull its published image instead by setting `_IMAGE` in `.env.compose` (e.g. `API_GATEWAY_IMAGE=ghcr.io/constructorfabric/insight-api-gateway:latest`) or with `./dev-compose.sh up --from-ghcr=`. See [CONTRIBUTING.md](CONTRIBUTING.md) for the build targets and frontend modes. For cluster deployments image tags flow through automatically: the umbrella chart's CI bumps each subchart's `appVersion` on every merge to `main`, and the subchart templates default `image.tag` to `.Chart.AppVersion`. Env overlays only need to pin a tag explicitly for a hotfix scenario (testing one service at a different tag than the one bundled in the umbrella version). Image source repos: diff --git a/docker-compose.yml b/docker-compose.yml index aebf9dae6..fd141ed6e 100644 --- a/docker-compose.yml +++ b/docker-compose.yml @@ -1,8 +1,8 @@ # Insight platform — local docker-compose dev stack. # -# This is the docker-compose path for working on Insight on a developer -# laptop. The Kind/helmfile path (dev-up.sh) is being retired in favour -# of this; see CONTRIBUTING.md. +# This is the default path for working on Insight on a developer laptop. +# For the Kubernetes path (Airbyte / Argo Workflows, real cluster shape) +# see CONTRIBUTING.md and deploy/gitops/. # # Quick start: # cp .env.compose.example .env.compose diff --git a/docs/components/backend/identity-resolution/identity/README.md b/docs/components/backend/identity-resolution/identity/README.md index 482a38096..aeb076125 100644 --- a/docs/components/backend/identity-resolution/identity/README.md +++ b/docs/components/backend/identity-resolution/identity/README.md @@ -15,7 +15,8 @@ and the (forthcoming) reconciliation service. | Path | Command | |---|---| -| Dev (local kind) | `./dev-up.sh --env local backend` — builds `insight-identity:local`, loads into kind, installs umbrella with `identity.deploy=true`. | +| Dev (Docker Compose, default) | `./dev-compose.sh up` runs the identity service in a container alongside MariaDB etc. Build the service image with `./dev-compose.sh build identity`. No Kind, no umbrella chart. | +| Dev (Kubernetes via gitops) | `cd deploy/gitops && make deploy ENV=local` on a local Kind/OrbStack cluster installs the umbrella chart, which includes identity-resolution when `identity.deploy=true`. | | Production / staging | Standard umbrella install. Override `identity.deploy=true` and `identity.image.tag=` in your values overlay. | | Standalone (no umbrella) | `helm install identity ./src/backend/services/identity/helm` with a pre-created `insight-identity-config` Secret. | diff --git a/docs/components/deployment/gitops/README.md b/docs/components/deployment/gitops/README.md index a5a11c5fa..a376b6b20 100644 --- a/docs/components/deployment/gitops/README.md +++ b/docs/components/deployment/gitops/README.md @@ -154,24 +154,24 @@ For one cluster carrying environment ``: | `insight-infra` | L2 | mariadb, clickhouse, redis, redpanda, redpanda-console, airbyte, argo-workflows (each as its own Helm release) | | `insight` | L3 | the umbrella chart (api-gateway, analytics-api, identity-resolution, frontend) | -Each cluster hosts exactly one Insight install. The cluster's identity (which env it represents) lives in the kube-context name (`insight-`) and the gitops repo's `environments//` directory — not in the namespace. Operationally this keeps the two well-known namespace names (`insight`, `insight-infra`) the same across every install, matching the `dev-up.sh` local-Kind convention and any external chart consumer's expectation of a single `insight` release. +Each cluster hosts exactly one Insight install. The cluster's identity (which env it represents) lives in the kube-context name (`insight-`) and the gitops repo's `environments//` directory — not in the namespace. Operationally this keeps the two well-known namespace names (`insight`, `insight-infra`) the same across every install — including the local cluster (`ENV=local`) — and matches any external chart consumer's expectation of a single `insight` release. #### Dual-purpose umbrella: `.deploy` toggles -The umbrella chart in `constructorfabric/insight` keeps its infrastructure subcharts (`clickhouse`, `mariadb`, `redis`, `redpanda`) **gated by per-service `.deploy: true|false` flags**. The flag is the dev-vs-prod switch: +The umbrella chart in `constructorfabric/insight` keeps its infrastructure subcharts (`clickhouse`, `mariadb`, `redis`, `redpanda`) **gated by per-service `.deploy: true|false` flags**. The flag selects the install shape: | Caller | `.deploy` | Result | |--------|----------------|--------| -| `dev-up.sh` (Kind / OrbStack local) | `true` for all infra subcharts | Single fat Helm release in the `insight` namespace; the umbrella renders MariaDB, ClickHouse, Redis, Redpanda **and** the app services together. Convenient for one-command local bring-up. | -| Gitops production (any cluster managed by this repo) | `false` for every infra subchart | Umbrella renders the app services only, into the `insight` namespace. L2 services come from one of: (a) `make system-` Helm releases in `insight-infra` per [§3.5](#35-engineer-provisions-the-system-layer-l2); (b) managed external endpoints (RDS, MSK, …); (c) a separate team's infra namespace. App values point at the actual host. | +| External consumer wanting a single-namespace install | `true` for all infra subcharts | Single fat Helm release in the `insight` namespace; the umbrella renders MariaDB, ClickHouse, Redis, Redpanda **and** the app services together. Self-contained, no separate L2. | +| Gitops (any cluster managed by this repo — production and local `ENV=local`) | `false` for every infra subchart | Umbrella renders the app services only, into the `insight` namespace. L2 services come from one of: (a) `make system-` Helm releases in `insight-infra` per [§3.5](#35-engineer-provisions-the-system-layer-l2); (b) managed external endpoints (RDS, MSK, …); (c) a separate team's infra namespace. App values point at the actual host. | Why dual-purpose instead of two charts: -- **One chart shape** — `dev-up.sh` exercises the same templates that production renders. Bugs in app rendering are caught locally. +- **One chart shape** — the local gitops cluster (`make deploy ENV=local`) exercises the same templates and layered shape that production renders. Bugs in app rendering are caught locally. - **Customers can choose** — an external chart consumer who is fine running everything in one namespace can flip the toggles `true` and get a self-contained install. A consumer with managed infra flips them `false`. - **No flag conflicts** — the existing `.deploy: false` path already wires the app to look up `.host` / `.port` from values, so cross-namespace DNS (`.insight-infra.svc.cluster.local`) is just one well-formed hostname away. -Airbyte and Argo Workflows are **not** subcharts of the umbrella in either mode — they are always installed as separate Helm releases (see `deploy/scripts/install-{airbyte,argo}.sh` for dev, `make system-{airbyte,argo}` for production). The same release goes to `insight` (dev) or `insight-infra` (prod) depending on the caller; no chart change required. +Airbyte and Argo Workflows are **not** subcharts of the umbrella in either mode — on every gitops cluster (production and local) they are installed as separate Helm releases via `make system-{airbyte,argo}` into `insight-infra`; no chart change required. ## 2. Tagging & Versioning @@ -857,7 +857,7 @@ These are accepted gaps that do not block the MVP but must be tracked. - **Audit log of deploys.** `make deploy` writes a local log file; there is no central audit. A trivial follow-up posts the log to a `#deploys` Slack channel via the poller's bot token; deferred until the team needs it. - **Rollback-by-tag.** `make rollback` calls `helm rollback` to the previous revision. Rolling back to an arbitrary historical state is `git checkout && make deploy`, which works but has not been rehearsed. - **Cross-namespace defaults in the umbrella.** The umbrella keeps its infra subcharts gated by `.deploy: true|false` (see [§1.5 dual-purpose umbrella](#15-layer-model)). For the gitops production case (`.deploy: false`), the app's connection helpers must default the host to `.insight-infra.svc.cluster.local` when no explicit `.host` is supplied — so a values file that only says `.deploy: false` "just works" against `insight-infra`. Verify the helpers do this; if not, a small chart-template change is needed. Also document the dual-purpose intent in `charts/insight/README.md` so external chart consumers understand the toggle. -- **dev-up.sh Airbyte/Argo namespace.** `dev-up.sh` installs Airbyte and Argo Workflows into the same namespace as the umbrella (`insight` for local). Production gitops puts them in `insight-infra`. The chart values surface for both is identical (Airbyte API URL, Argo SA name) — confirm by render. If anything still hard-codes the `insight` namespace in templates, parameterise it. +- **Airbyte/Argo namespace assumptions.** On every gitops cluster (production and local `ENV=local`) Airbyte and Argo Workflows run as L2 releases in `insight-infra`, while the umbrella runs in `insight`. The chart's helpers must therefore resolve the Airbyte API URL and Argo SA name across namespaces — confirm by render. If anything still hard-codes the `insight` namespace for these in templates, parameterise it. - **L2 chart-pin policy.** System service chart versions (`MARIADB_VERSION`, `CLICKHOUSE_VERSION`, etc.) are Makefile constants today; bumping is a deliberate PR. A future enhancement: split each service's pin into its own `system//.version` file (mirroring `.insight-version`) so a poller could pre-flight version compatibility against published Bitnami / Redpanda / Airbyte releases. Out of scope for v0. - **Per-cluster L2 inventory.** When a cluster swaps a self-hosted service for a managed endpoint (e.g. a managed database instead of `system/mariadb`), the gitops repo currently has no machine-readable record of "this cluster runs MariaDB on-cluster vs. external." A small `environments//inventory.yaml` listing which `system-*` targets to run on this cluster would make `make doctor` able to validate that the cluster matches the expected inventory, and would make it possible to render a per-customer install runbook from the repo. Captured. diff --git a/docs/components/deployment/specs/DESIGN.md b/docs/components/deployment/specs/DESIGN.md index 2b60b39a9..e65fc462f 100644 --- a/docs/components/deployment/specs/DESIGN.md +++ b/docs/components/deployment/specs/DESIGN.md @@ -29,16 +29,16 @@ date: 2026-05-12 ### 1.1 Architectural Vision -The Deployment subsystem is a **two-layer** distribution pipeline. Layer one is the artifact: a single Helm umbrella chart at `charts/insight/`, built and published per merge to `main` of `constructorfabric/insight` as `oci://ghcr.io/constructorfabric/charts/insight:` by the Chart Publishing CI workflow, alongside per-service application images at `ghcr.io/constructorfabric/insight-:`. Layer two is the set of consumers: the private `infra/insight-gitops` repository (driving every Cyberfabric-operated cluster — internal `dev`/`test`/`stage` and each customer-named production cluster), the `dev-up.sh` wrapper (driving a local Kind/OrbStack cluster), and any external Helm consumer (Constructor Platform tenants, evaluators, partners) who pulls the chart and runs it with their own tooling. The architectural rationale for publishing the chart this way is captured in [ADR-0001](./ADR/0001-chart-publishing-on-merge.md); the operational contract for the gitops consumer is captured in the [gitops SPEC](../gitops/README.md). +The Deployment subsystem is a **two-layer** distribution pipeline. Layer one is the artifact: a single Helm umbrella chart at `charts/insight/`, built and published per merge to `main` of `constructorfabric/insight` as `oci://ghcr.io/constructorfabric/charts/insight:` by the Chart Publishing CI workflow, alongside per-service application images at `ghcr.io/constructorfabric/insight-:`. Layer two is the set of consumers: the private `infra/insight-gitops` repository (driving every Cyberfabric-operated cluster — internal `dev`/`test`/`stage` and each customer-named production cluster, and the same path run locally against a Kind/OrbStack cluster via `make deploy ENV=local`), and any external Helm consumer (Constructor Platform tenants, evaluators, partners) who pulls the chart and runs it with their own tooling. Day-to-day backend / frontend development uses a separate Docker Compose stack (`dev-compose.sh`) that does not consume the chart. The architectural rationale for publishing the chart this way is captured in [ADR-0001](./ADR/0001-chart-publishing-on-merge.md); the operational contract for the gitops consumer is captured in the [gitops SPEC](../gitops/README.md). The artifact is deliberately thin: the umbrella owns the chart-shape contract but does not own controllers or CRDs. It orchestrates eight subcharts (four infra + four app services), emits one bridge object (`{release}-platform` ConfigMap), emits Argo `WorkflowTemplate` objects when gated, and runs a fail-fast `insight.validate` template at render time. Every infra subchart is pluggable via the `.deploy: true|false` toggle plus the same flat `.host` / `.port` / `.passwordSecret` shape — read identically whether the subchart is bundled or external. That dual-purpose toggle is what makes one chart serve two install shapes. The chart serves two install shapes selected by the consumer: -- **Single-namespace fat install** (`.deploy: true` for every infra block). The umbrella renders MariaDB, ClickHouse, Redis, Redpanda **and** the app services into a single namespace. `dev-up.sh` runs this shape locally. An external consumer who wants everything in one namespace can also flip the toggles `true`. +- **Single-namespace fat install** (`.deploy: true` for every infra block). The umbrella renders MariaDB, ClickHouse, Redis, Redpanda **and** the app services into a single namespace. An external consumer who wants everything in one namespace flips the toggles `true`. - **Layered app-only install** (`.deploy: false` for every infra block). The umbrella renders the app services only into the `insight` namespace; infra comes from L2 (gitops production: `insight-infra` namespace) or from managed external endpoints / a separate team's namespace (Constructor Platform, external customers). This shape is driven by the gitops Makefile per the [L0/L2/L3 model](../gitops/README.md#15-layer-model). -Tenant separation across customers is at the **cluster boundary** for gitops production — one Insight install per cluster, the customer's identity in the kube-context name (`insight-`) and in the gitops repo's `environments//` directory, never in the namespace. The two well-known namespaces (`insight-infra`, `insight`) are the same on every cluster. Tenant separation on a shared cluster (`dev-up.sh` parallel branches, Constructor Platform tenants) is at the namespace boundary, with `controller.instanceID` scoping Argo workflows. +Tenant separation across customers is at the **cluster boundary** for gitops production — one Insight install per cluster, the customer's identity in the kube-context name (`insight-`) and in the gitops repo's `environments//` directory, never in the namespace. The two well-known namespaces (`insight-infra`, `insight`) are the same on every cluster — including the local `ENV=local` cluster. Tenant separation on a shared cluster (Constructor Platform tenants) is at the namespace boundary, with `controller.instanceID` scoping Argo workflows. ### 1.2 Architecture Drivers @@ -63,12 +63,12 @@ Tenant separation across customers is at the **cluster boundary** for gitops pro | `cpt-insightspec-fr-dep-dual-purpose-toggle` | `.deploy: true|false` toggles on the four infra subcharts. Same chart, two install shapes; cross-namespace wiring uses the same `.host` / `.port` shape as Constructor Platform external mode. Documented in the chart README and the gitops SPEC §1.5. | | `cpt-insightspec-fr-dep-layered-architecture` | The gitops SPEC §1.5 layer model (L0 Bootstrap / L2 System / L3 App) is the consumer-side contract; this DESIGN documents how the chart's dual-purpose toggle makes the layered shape possible (app services in `insight`, infra elsewhere by L2 or managed). | | `cpt-insightspec-fr-dep-customer-named-envs` | Implemented in the gitops Makefile (`PROTECTED_ENVS` + `CONFIRM=yes-deploy-` token), documented in the [gitops SPEC §6.3](../gitops/README.md#63-pre-flight-safety-checks); the chart itself is environment-agnostic. | -| `cpt-insightspec-fr-dep-namespace-convention` | Chart assumes release name `insight`; cross-namespace helpers default L2 host to `.insight-infra.svc.cluster.local` for the layered shape. `dev-up.sh` uses the same release name in the same `insight` namespace. | -| `cpt-insightspec-fr-dep-dev-wrapper` | `dev-up.sh` scripts the Kind/OrbStack bootstrap, backend image builds + `kind load`, frontend build from `insight-front_symlink` with arch-aware fallback, applies `deploy/values-dev.yaml` (which flips `.deploy: true`), installs Airbyte + Argo via dev-helper scripts, and opens the documented port-forwards. | -| `cpt-insightspec-fr-dep-dev-namespace-param` | `dev-up.sh`, `dev-down.sh` and `init.sh` read `INSIGHT_NAMESPACE` (default `insight`). | +| `cpt-insightspec-fr-dep-namespace-convention` | Chart assumes release name `insight`; cross-namespace helpers default L2 host to `.insight-infra.svc.cluster.local` for the layered shape. The local gitops cluster (`ENV=local`) uses the same release name in the same `insight` namespace. | +| `cpt-insightspec-fr-dep-dev-wrapper` | The Docker Compose dev stack (`dev-compose.sh` + `docker-compose.yml`) builds the backend services and frontend from source in builder containers (or pulls published images), runs them with bundled MariaDB / ClickHouse / Redis / Redpanda containers, auto-reloads on rebuild, and auto-seeds a demo dataset. It does not consume the umbrella chart. | +| `cpt-insightspec-fr-dep-dev-namespace-param` | The compose stack's published host ports and frontend / backend image sources are overridable via `.env.compose`, so multiple stacks (or a stack pointed at external DBs) can coexist on one host. | | `cpt-insightspec-fr-dep-tenant-isolation-boundary` | Cluster-per-customer for gitops production (kube-context `insight-`); namespace-per-tenant on shared clusters with `controller.instanceID.explicitID=$RELEASE-$NAMESPACE` and `controller.workflowNamespaces[0]=$NAMESPACE`. No ClusterRole / ClusterRoleBinding from L3. | | `cpt-insightspec-fr-dep-empty-credentials-default` | `charts/insight/values.yaml` ships no inline passwords. With `credentials.autoGenerate=true` (default) the umbrella creates `insight-db-creds` on first install via `lookup` + `randAlphaNum 24` and reuses it on every upgrade. BYO mode: an operator-supplied `insight-db-creds` is auto-detected via absence of the `app.kubernetes.io/managed-by=Helm` label, in which case the chart skips its own Secret-template emission so Helm does not attempt ownership transfer. With `autoGenerate=false` and no pre-existing Secret the install fails fast. OIDC fields are empty and the validator refuses any render that doesn't either set `apiGateway.oidc.existingSecret` or all three of `issuer`/`clientId`/`redirectUri`. | -| `cpt-insightspec-fr-dep-dev-overlay-isolation` | Eval credentials live only in `deploy/values-dev.yaml`; applied by `dev-up.sh` exclusively. Production credentials reach the cluster through sealed secrets (gitops Passbolt → SealedSecret) or operator-managed Secrets, never via committed values files. | +| `cpt-insightspec-fr-dep-dev-overlay-isolation` | Eval credentials live only in local-only artifacts — `.env.compose` (compose, gitignored) or wizard-generated values (local gitops). Production credentials reach the cluster through sealed secrets (gitops Passbolt → SealedSecret) or operator-managed Secrets, never via committed values files. | #### NFR Allocation @@ -92,20 +92,20 @@ Tenant separation across customers is at the **cluster boundary** for gitops pro │ : │ └────────────────────────┬─────────────────────┘ │ pull oci://… by tag - ┌─────────────────────────────────┼──────────────────────────────┐ - │ │ │ - ▼ ▼ ▼ -┌───────────────────┐ ┌────────────────────────┐ ┌──────────────────┐ -│ Cyberfabric SRE │ │ Platform Developer │ │ External chart │ -│ (gitops repo) │ │ (dev-up.sh) │ │ consumer │ -│ │ │ │ │ (helm / ArgoCD / │ -│ make deploy ENV │ │ Kind / OrbStack │ │ Flux / …) │ -│ L0 / L2 / L3 │ │ .deploy: true │ │ │ -│ insight-infra + │ │ single-namespace fat │ │ Operator picks │ -│ insight ns │ │ in insight ns │ │ own tooling │ -└────────┬──────────┘ └───────────┬────────────┘ └────────┬─────────┘ - │ │ │ - ▼ ▼ ▼ + ┌──────────────────────────┴──────────────────────────┐ + │ │ + ▼ ▼ +┌──────────────────────────────┐ ┌──────────────────┐ +│ gitops (Cyberfabric SRE + │ │ External chart │ +│ local dev `ENV=local`) │ │ consumer │ +│ │ │ (helm / ArgoCD / │ +│ make deploy ENV= │ │ Flux / …) │ +│ L0 / L2 / L3 │ │ │ +│ .deploy: false (layered)│ │ Operator picks │ +│ insight-infra + insight ns │ │ own tooling │ +└────────────┬──────────────────┘ └────────┬─────────┘ + │ │ + ▼ ▼ ┌─────────────────────────────────────────────────────┐ │ Insight umbrella chart │ │ - subcharts: clickhouse, mariadb, redis, redpanda │ @@ -122,8 +122,8 @@ Tenant separation across customers is at the **cluster boundary** for gitops pro |-------|---------------|------------| | Artifact (CI) | Build images, bump subchart `appVersion`s, patch-bump umbrella, package, push to GHCR, auto-commit version bumps. | GitHub Actions, Docker buildx, Helm 3.14+, GHCR. | | Artifact (chart) | Aggregate subcharts, emit `{release}-platform` ConfigMap, emit Argo `WorkflowTemplate` objects, run fail-fast validation, bridge bundled/external mode via helpers. | Helm chart (apiVersion v2), Go-template helpers. | -| Consumer (gitops) | Pull chart from OCI pinned to `.insight-version`; drive L0 bootstrap + L2 system services + L3 app deploy per env via the Makefile. | Private `infra/insight-gitops` repo on GitLab; bash Makefile; Passbolt + sealed-secrets-controller. | -| Consumer (dev) | Bootstrap Kind/OrbStack, build images from source, install the umbrella with `.deploy: true` plus Airbyte + Argo via dev-helper scripts, open port-forwards. | bash, Kind/OrbStack, Helm, kubectl. | +| Consumer (gitops) | Pull chart from OCI pinned to `.insight-version`; drive L0 bootstrap + L2 system services + L3 app deploy per env via the Makefile — on production clusters and on a local Kind/OrbStack cluster (`make deploy ENV=local`). | Private `infra/insight-gitops` repo on GitLab; bash Makefile; Passbolt + sealed-secrets-controller. | +| Local development | Day-to-day backend / frontend work runs the Docker Compose stack (`dev-compose.sh`), which does **not** consume the chart; chart and ingestion validation runs the gitops path locally (`make deploy ENV=local`). | Docker Compose; Kind/OrbStack for the local gitops cluster. | | Consumer (external) | Pull chart from OCI by tag, install with their own values + tooling (helm, ArgoCD, Flux, Terraform Helm provider, …). | Consumer's choice. | | Subcharts | Ship the actual Kubernetes workloads (StatefulSets, Deployments, Services, HPAs). | Bitnami / Bitnamilegacy (MariaDB, Redis), upstream Redpanda, local wrapper for ClickHouse, in-repo charts for app services. | @@ -143,7 +143,7 @@ The product is the umbrella chart. Everything that ships together is declared as - [ ] `p1` - **ID**: `cpt-insightspec-principle-dep-dual-purpose-chart` -The chart's `.deploy: true|false` toggle is the dev-vs-prod switch. `dev-up.sh` flips it `true` for a single-namespace fat install; gitops production flips it `false` for an app-only layered install with infra in `insight-infra` (L2) or managed externally. Both shapes render through the same templates and the same helpers, so a bug in app rendering surfaces on the developer's laptop before it reaches a customer cluster. +The chart's `.deploy: true|false` toggle selects the install shape. An external consumer who wants everything in one namespace flips it `true` for a single-namespace fat install; gitops (production and local) flips it `false` for an app-only layered install with infra in `insight-infra` (L2) or managed externally. Both shapes render through the same templates and the same helpers, so a bug in app rendering surfaces on a developer's local `make deploy ENV=local` cluster before it reaches a customer cluster. **ADRs**: none. @@ -151,7 +151,7 @@ The chart's `.deploy: true|false` toggle is the dev-vs-prod switch. `dev-up - [ ] `p1` - **ID**: `cpt-insightspec-principle-dep-single-artifact-ref` -Every consumer addresses the chart by the same OCI URL (`oci://ghcr.io/constructorfabric/charts/insight`) and a semver tag. No sibling-checkout dependency, no curl-from-GitHub at deploy time, no per-consumer publishing flow. Cyberfabric SRE, Constructor Platform tenants, external customers and `dev-up.sh` (for testing the published artifact) all pull from the same place. +Every consumer addresses the chart by the same OCI URL (`oci://ghcr.io/constructorfabric/charts/insight`) and a semver tag. No sibling-checkout dependency, no curl-from-GitHub at deploy time, no per-consumer publishing flow. Cyberfabric SRE (production and local `ENV=local`), Constructor Platform tenants and external customers all pull from the same place. **ADRs**: [ADR-0001](./ADR/0001-chart-publishing-on-merge.md). @@ -183,7 +183,7 @@ The gitops repo has no generic "prod". Internal envs (`dev`, `test`, `stage`) an - [ ] `p2` - **ID**: `cpt-insightspec-principle-dep-convergent-paths` -The developer wrapper and the gitops production path share the umbrella chart artifact. The dev shape is one set of values overlays (`.deploy: true`, eval creds, port-forwards); the production shape is another (`.deploy: false`, sealed creds, ingress). A change that breaks production rendering surfaces in `helm template` on the developer path first. +The local gitops path (`make deploy ENV=local`) and the gitops production path are the same Makefile and the same umbrella chart artifact — they differ only in the per-env values overlay (local: wizard-generated creds, dev-friendly toggles; production: sealed creds, ingress). A change that breaks production rendering surfaces in `helm template` on the local path first. **ADRs**: none. @@ -201,7 +201,7 @@ The umbrella chart declares `kubeVersion: ">=1.27.0-0"`. Consumers on older Kube - [ ] `p1` - **ID**: `cpt-insightspec-constraint-dep-helm-version` -OCI chart pulls require Helm 3.14+. This applies to every consumer (gitops Makefile, `dev-up.sh`, ArgoCD/Flux instances rendering the chart, customer-side helm installs). +OCI chart pulls require Helm 3.14+. This applies to every chart consumer (gitops Makefile on production and local clusters, ArgoCD/Flux instances rendering the chart, customer-side helm installs). **ADRs**: [ADR-0001](./ADR/0001-chart-publishing-on-merge.md). @@ -225,7 +225,7 @@ Airbyte chart pinned to 1.8.5+ (app 1.8.5+) at the consumer side. Chart 1.9.x wa - [ ] `p3` - **ID**: `cpt-insightspec-constraint-dep-frontend-amd64` -The published `ghcr.io/constructorfabric/insight-front` image ships only a linux/amd64 manifest. The dev wrapper works around this by rebuilding from the sibling `insight-front` checkout on arm64 hosts. Production installs on amd64 clusters are unaffected; the publish workflow does not bump frontend image tags automatically (the frontend source lives in a separate repo). +The published `ghcr.io/constructorfabric/insight-front` image ships only a linux/amd64 manifest. The Docker Compose dev stack works around this on arm64 hosts by rebuilding from the sibling `insight-front` checkout (`FRONTEND_MODE=dev` / `built`); the default `ghcr` mode runs the amd64 image under QEMU. Production installs on amd64 clusters are unaffected; the publish workflow does not bump frontend image tags automatically (the frontend source lives in a separate repo). **ADRs**: none. @@ -246,7 +246,7 @@ Deployment has no runtime domain model — it neither stores nor serves data. Th - **Chart artifact**: a packaged Helm chart (`insight-.tgz`) addressable by `(oci://ghcr.io/constructorfabric/charts/insight, version)`. One produced per merge to `main`. Immutable once published. - **Service image**: a container image at `ghcr.io/constructorfabric/insight-:`. Referenced by exactly one subchart's `appVersion`. - **Subchart**: a Helm subchart aggregated under the umbrella, identified by `name`, optionally `alias`, `version`, `repository`, and (for infra) `condition: .deploy`. -- **Release**: a Helm release applied by a consumer (the L3 umbrella release, the L2 per-service releases in the gitops shape, the Airbyte/Argo releases in `dev-up.sh`). Identified by `(namespace, release-name)`. +- **Release**: a Helm release applied by a consumer (the L3 umbrella release, the L2 per-service releases in the gitops shape including Airbyte/Argo). Identified by `(namespace, release-name)`. - **Values file**: a YAML file that parameterises a release. The chart's `values.yaml` is the canonical reference; consumer overlays compose on top. - **Platform ConfigMap**: the single bridge object emitted by the umbrella. Maps resolved infra coordinates into environment variables for every pod in the namespace. - **Infra contract**: a single flat `` block (`deploy`, `host`, `port`, `database`, `username`, `passwordSecret`) — same shape whether the umbrella runs the dep itself or consumes an externally-provided one. @@ -269,8 +269,7 @@ graph TB end subgraph Consumers["Consumers"] - Gitops[infra/insight-gitops Makefile] - DevUp[dev-up.sh] + Gitops[infra/insight-gitops Makefile
production + local ENV=local] External[External Helm consumer] end @@ -293,10 +292,8 @@ graph TB Publish --> GHCR GHCR -.-> Gitops - GHCR -.-> DevUp GHCR -.-> External Gitops -->|helm upgrade --install| Umbrella - DevUp -->|helm upgrade --install| Umbrella External -->|helm/argocd/flux| Umbrella Umbrella --> Helpers Umbrella --> PlatformCM @@ -329,10 +326,10 @@ Every consumer of Insight — Cyberfabric SRE pinning one version per cluster, C ##### Responsibility boundaries - Does not ship CRDs or controllers. -- Does not install Airbyte or Argo Workflows — those are separate Helm releases driven by the consumer (`dev-up.sh` for dev, `make system-airbyte` / `make system-argo` in the gitops repo for production). +- Does not install Airbyte or Argo Workflows — those are separate Helm releases driven by the consumer (`make system-airbyte` / `make system-argo` in the gitops repo, on production and local clusters; external consumers by their own means). - Does not create ClusterRoles, ClusterRoleBindings or any cross-namespace resources. - Does not own runtime behaviour of the subcharts; their configuration lives in their own values blocks exposed here. -- Is not tied to any one consumer; the chart is the same artifact in OCI whether pulled by the gitops Makefile, by `dev-up.sh`, by ArgoCD/Flux, or by `helm install` directly. +- Is not tied to any one consumer; the chart is the same artifact in OCI whether pulled by the gitops Makefile, by ArgoCD/Flux, or by `helm install` directly. ##### Related components (by ID) @@ -446,33 +443,32 @@ Every pod in the namespace needs the same set of resolved coordinates. Pushing t - `cpt-insightspec-component-dep-service-resolution-helpers` — depends on. - `cpt-insightspec-component-dep-umbrella-chart` — owned by. -#### Dev Wrapper (`dev-up.sh`) +#### Docker Compose Dev Stack (`dev-compose.sh`) - [ ] `p3` - **ID**: `cpt-insightspec-component-dep-dev-wrapper` ##### Why this component exists -Developers iterate faster when the platform bring-up is one command. The wrapper adds what production consumers do not need — cluster bootstrap, image builds from source, Kind image loading, port-forwards — while installing the *same* umbrella chart that production consumes, just with the `.deploy: true` toggles flipped on for a single-namespace fat install. That keeps dev and production paths convergent. +Day-to-day backend / frontend work needs a fast loop with no Kubernetes overhead — no cluster bootstrap, no image registry, no chart rendering. The Docker Compose stack brings the application services up from source (or from published images) alongside their bundled databases on a developer laptop with only Docker installed. It deliberately does **not** consume the umbrella chart; chart-shape and ingestion validation is done on the local gitops cluster (`make deploy ENV=local`), which exercises the same artifact production consumes. ##### Responsibility scope -- Bootstraps a Kind/OrbStack cluster named `insight` if absent. The cluster config (`k8s/kind-config.yaml`) pins a specific `kindest/node` image so the dev path is independent of the host's installed `kind` binary version. Recent kind defaults (`v1.32+`) require cgroup v2, which Docker Desktop on Windows and several Linux distros do not provide; pinning to a cgroup-v1-compatible image keeps the bring-up working on stock developer hosts. Bumping the pin requires validating Argo / Airbyte / bitnami subchart compatibility against the new node image. -- Builds backend images and loads them with `kind load docker-image`. -- Builds the frontend image from the sibling `insight-front` checkout with native-arch try + `linux/amd64` fallback; on pull-only paths uses `docker pull --platform`. -- Installs Airbyte and Argo Workflows into the same local `insight` namespace via `deploy/scripts/install-airbyte.sh` and `deploy/scripts/install-argo.sh` (internal helpers; not a public deploy path). -- Installs the umbrella chart with `deploy/values-dev.yaml` (`.deploy: true` for the infra subcharts plus eval credentials). -- Opens port-forwards for Frontend :8003, API Gateway :8080, Airbyte UI :8002, Airbyte API :8001, Argo UI :2746, ClickHouse HTTP :8123. -- `dev-down.sh` tears the cluster down; `init.sh` bootstraps `.env.*` defaults. +- Runs `docker-compose.yml`: api-gateway + analytics-api (Rust) + identity (.NET 9) + frontend, plus bundled MariaDB / ClickHouse / Redis / Redpanda containers. +- Builds the backend services and (optionally) the frontend in builder containers — no Rust / .NET / Node toolchain on the host. Per-service `_IMAGE` overrides in `.env.compose` (or `--from-ghcr=`) pull a published image instead of building. +- Auto-reloads each backend service in ~1 second on rebuild via `watchexec` when `ENABLE_AUTO_RELOAD=true` (compose-only; never set in a Kubernetes manifest). +- Auto-seeds a demo dataset (identity + silver) on first `up`, tracked via `SEEDED_LOCAL_*` markers in `.env.compose`. +- A first-run wizard generates `.env.compose`, capturing local-vs-external MariaDB / ClickHouse, the dev-user email, the tenant id, and the frontend mode. Re-run by deleting `.env.compose` or `./dev-compose.sh prune`. +- Publishes the web services on configurable host ports (Frontend :3000, API Gateway :8080, Analytics API :8081, Identity :8082, ClickHouse :8123, MariaDB :3306, Redis :6379). ##### Responsibility boundaries -- Not for production use. `dev-up.sh` is explicitly local-only; production consumers either go through the gitops repo or pull the chart from OCI directly. -- The `deploy/scripts/install-*.sh` helpers under `dev-up.sh` are internal to this wrapper, not a documented public deploy path. They exist to install Airbyte / Argo into the local cluster — production consumers install Airbyte / Argo by whatever means matches their environment (the gitops Makefile's `make system-airbyte` / `make system-argo` for Cyberfabric clusters; the consumer's own choice for external installs). -- Does not persist state outside the Kind/OrbStack cluster; nothing leaks to the host. +- Not for production use; explicitly local-only. +- Does not consume the umbrella chart and does not ship Airbyte or Argo Workflows — ingestion work that needs them runs on the local gitops cluster (`make deploy ENV=local`). +- Eval credentials live in `.env.compose` (gitignored); nothing leaks into the canonical chart values or any published artifact. ##### Related components (by ID) -- `cpt-insightspec-component-dep-umbrella-chart` — consumes. +- None — the compose stack is independent of the umbrella chart. Chart consumers are documented under `cpt-insightspec-component-dep-umbrella-chart`. ### 3.3 API Contracts @@ -522,22 +518,26 @@ The dual-purpose intent of the four `.deploy` toggles is documented in [` Per-tag artifacts are immutable; the Chart Publishing CI does not overwrite. GHCR retention may delete old tags — consumers pinning a specific version SHOULD mirror to their own registry for long-term reproducibility (tracked in the gitops SPEC §8 open items). -#### Dev wrapper environment contract +#### Compose dev stack settings contract - [ ] `p2` - **ID**: `cpt-insightspec-interface-dep-dev-wrapper-env` -- **Technology**: shell environment variables read by the dev wrapper. -- **Location**: [dev-up.sh](../../../../dev-up.sh), [dev-down.sh](../../../../dev-down.sh), [init.sh](../../../../init.sh). +- **Technology**: dotenv settings read by `dev-compose.sh` and `docker-compose.yml`, generated by the first-run wizard. +- **Location**: [dev-compose.sh](../../../../dev-compose.sh), [docker-compose.yml](../../../../docker-compose.yml), [.env.compose.example](../../../../.env.compose.example). **Endpoints Overview**: -| Variable | Used by | Description | Stability | -|----------|---------|-------------|-----------| -| `INSIGHT_NAMESPACE` | dev-up.sh, dev-down.sh, init.sh | Target Kubernetes namespace (default `insight`). | stable | -| `DEV_MODE` | dev-up.sh helpers | When `1`, merges `deploy/argo/values-dev.yaml` for the local Argo install (auth-mode=server for eval). | stable | -| `INSIGHT_VALUES_FILES` | dev-up.sh | Colon-separated list of `-f` files for the local umbrella install. `dev-up.sh` defaults this to include `deploy/values-dev.yaml`. | stable | +| Variable | Description | Stability | +|----------|-------------|-----------| +| `ENABLE_AUTO_RELOAD` | Wraps each backend entrypoint in `watchexec --restart` for ~1s reload. Compose-only — never set in a Kubernetes manifest. | stable | +| `FRONTEND_MODE` | `ghcr` (published image, default), `dev` (Vite HMR from `INSIGHT_FRONT_PATH`), or `built` (host-built dist). | stable | +| `_IMAGE` | Pull a published image for a backend service instead of building it (e.g. `API_GATEWAY_IMAGE`). | stable | +| `*_PORT` | Host port for each published service (Frontend :3000, gateway :8080, …); override on conflict. | stable | +| `MARIADB_EXTERNAL` / `_HOST` / `_INTERNAL_PORT`, ClickHouse equivalents | Point the stack at an external DB instead of the bundled container. | stable | +| `TENANT_DEFAULT_ID` | Tenant UUID used by the seed and the dev caller context. | stable | +| `SEEDED_LOCAL_MARIA` / `SEEDED_LOCAL_CH` | First-run seed bookkeeping; clear to force a re-seed on next `up`. | stable | -The dev wrapper's internal `deploy/scripts/install-*.sh` helpers are not a documented public surface; they exist solely to install Airbyte and Argo into the local cluster. Production consumers do not use them. +`.env.compose.example` documents the full settings contract. The stack is local-only; none of these settings reach the canonical chart values or any published artifact. ### 3.4 Internal Dependencies @@ -550,7 +550,6 @@ The dev wrapper's internal `deploy/scripts/install-*.sh` helpers are not a docum | `helmfile/charts/clickhouse` | Helm subchart (local wrapper) | ClickHouse OLAP store. | | `charts/insight/templates/ingestion/*.yaml` | First-class Helm templates | Ingestion WorkflowTemplate sources, gated by `ingestion.templates.enabled`; consume umbrella helpers directly via `include`. | | `.github/workflows/build-images.yml` (`publish-chart` job) | GitHub Actions workflow | Chart Publishing CI — produces the published umbrella artifact per merge to `main`. | -| `helmfile.yaml.gotmpl` | Repo-root helmfile (legacy, EXPERIMENTAL) | Pre-consolidation alternative install path. Structurally incompatible with the post-consolidation umbrella + bitnami-subcharts + Secret-emission pattern. Header warning in the file documents the issues; retained for reference only and may be removed entirely in a follow-up cleanup. **Not on any supported install path.** | | `src/ingestion/airbyte-toolkit/lib/env.sh` | Read `AIRBYTE_API_URL` from the platform ConfigMap | Ingestion scripts consume Airbyte coordinates from the ConfigMap rather than hard-coding. | **Dependency Rules**: @@ -565,13 +564,13 @@ The dev wrapper's internal `deploy/scripts/install-*.sh` helpers are not a docum | Dependency Module | Interface Used | Purpose | |-------------------|----------------|---------| -| `airbyte/airbyte` chart 1.8.5+ | Helm release | Data extraction engine; installed as a separate release. Lives in `insight-infra` (gitops production L2) or in the local `insight` namespace (`dev-up.sh`). | +| `airbyte/airbyte` chart 1.8.5+ | Helm release | Data extraction engine; installed as a separate L2 release in `insight-infra` by the gitops path, on production and local (`ENV=local`) clusters. | #### Argo Workflows | Dependency Module | Interface Used | Purpose | |-------------------|----------------|---------| -| `argo/argo-workflows` chart 0.45.x | Helm release | Workflow engine for ingestion pipelines; installed as a separate release. Lives in `insight-infra` (gitops production L2) or in the local `insight` namespace (`dev-up.sh`). | +| `argo/argo-workflows` chart 0.45.x | Helm release | Workflow engine for ingestion pipelines; installed as a separate L2 release in `insight-infra` by the gitops path, on production and local (`ENV=local`) clusters. | #### Bitnami Helm charts (MariaDB, Redis) @@ -705,47 +704,43 @@ sequenceDiagram **Use cases**: `cpt-insightspec-usecase-dep-eval-install`, `cpt-insightspec-usecase-dep-dev-inner-loop`. -**Actors**: `cpt-insightspec-actor-platform-developer`, `cpt-insightspec-actor-kubernetes`. +**Actors**: `cpt-insightspec-actor-platform-developer`. ```mermaid sequenceDiagram actor Dev as Platform Developer - participant DevUp as dev-up.sh - participant Kind as Kind / OrbStack - participant Helm as Helm - participant K8s as Cluster (insight ns) - - Dev->>DevUp: ./dev-up.sh --env local - DevUp->>Kind: kind create cluster --name insight (if absent) - DevUp->>DevUp: docker build backend + frontend - DevUp->>Kind: kind load docker-image - DevUp->>Helm: helm upgrade --install airbyte airbyte/airbyte -n insight - DevUp->>Helm: helm upgrade --install argo-workflows argo/argo-workflows -n insight (DEV_MODE) - DevUp->>Helm: helm upgrade --install insight charts/insight -n insight -f deploy/values-dev.yaml - Note over Helm: values-dev flips .deploy: true for the four infra subcharts - Helm->>K8s: render single-namespace fat install - K8s-->>DevUp: Ready - DevUp->>DevUp: open port-forwards (UI :8003, API :8080, Argo :2746, ...) - DevUp-->>Dev: ready; open http://localhost:8003 + participant CLI as dev-compose.sh + participant Compose as Docker Compose + participant Watch as watchexec (in container) + + Dev->>CLI: ./dev-compose.sh up + Note over CLI: first run only — wizard generates .env.compose + CLI->>CLI: build backend (+ frontend) in builder containers + CLI->>Compose: docker compose up -d (backend + frontend + MariaDB/ClickHouse/Redis/Redpanda) + Compose-->>CLI: containers healthy + CLI->>CLI: auto-seed demo dataset (first run) + CLI-->>Dev: ready; open http://localhost:3000 + Dev->>CLI: edit code; ./dev-compose.sh build + CLI->>Watch: bind-mounted binary changes + Watch->>Compose: SIGTERM + respawn (~1s) ``` -**Description**: The dev wrapper installs the same umbrella chart that production consumers pull from OCI, just with `.deploy: true` overlays for a single-namespace fat install. Airbyte and Argo are installed via internal `dev-up.sh` helpers into the same `insight` namespace (local-only convenience; production deploys put them in `insight-infra` via the gitops Makefile, but the chart values surface — `airbyte.apiUrl`, Argo SA name — is identical). +**Description**: The Docker Compose stack runs the application services and bundled databases as containers on the laptop — it does not consume the umbrella chart. After the first-run wizard generates `.env.compose`, `up` builds from source (or pulls published images), seeds a demo dataset, and exposes the frontend on :3000. The edit-build loop relies on `watchexec` restarting the bind-mounted binary in ~1 second. Chart-shape and ingestion validation against the real cluster topology is a separate flow — `make deploy ENV=local` — covered by the "Gitops deploy from OCI pin" sequence above. ### 3.7 Database schemas & tables -Not applicable. The Deployment subsystem stores no data; it produces a chart artifact and a dev wrapper. Runtime data stores (ClickHouse, MariaDB, Redpanda) are introduced by subcharts but their schemas are owned elsewhere (Backend DESIGN for MariaDB; Ingestion Layer DESIGN and Connector DESIGNs for ClickHouse). +Not applicable. The Deployment subsystem stores no data; it produces a chart artifact and a Docker Compose dev stack. Runtime data stores (ClickHouse, MariaDB, Redpanda) are introduced by subcharts but their schemas are owned elsewhere (Backend DESIGN for MariaDB; Ingestion Layer DESIGN and Connector DESIGNs for ClickHouse). ## 4. Additional context -**Lessons learned from the first-run debugging cycle.** The `DEVLOG.md` at the repo root captures the twelve issues discovered during the first end-to-end run of `dev-up.sh` on a fresh Apple-Silicon laptop. The ones that fed back into the design: +**Lessons that fed into the chart design.** Early end-to-end runs on a fresh Apple-Silicon laptop surfaced several issues that shaped the current chart and deploy model: -1. Airbyte auth was off in the curated values. Now `global.auth.enabled: true`; the chart generates a random admin password on first install into `airbyte-auth-secrets/instance-admin-password`. The dev wrapper picks it up. -2. The Airbyte webapp port-forward was missing from `dev-up.sh`. Now port 8002 is opened. -3. DB passwords are not auto-generated. Canonical `values.yaml` intentionally leaves infra credentials empty; dev bring-up supplies eval values via `deploy/values-dev.yaml`. This is a deliberate choice over `randAlphaNum` hooks because reproducibility across dev runs beats marginal "security" on a throwaway cluster. -4. Frontend image is linux/amd64 only. The dev wrapper builds from the sibling `insight-front` checkout on arm64 hosts with a `docker pull --platform linux/amd64` fallback; follow-up is to publish multi-arch images (infra team). -5. Argo's chart expects `controller.instanceID` as a sub-object (`enabled` + `explicitID`), not a plain string. The dev-up Argo install uses the dotted-key form. -6. Argo's supplemental RBAC requires a namespace-scoped Role + Binding for the workflow service account. Shipped as `deploy/argo/rbac.yaml` for `dev-up.sh`; gitops production ships `bootstrap/argo-rbac.yaml.tmpl` with `${NAMESPACE}` / `${WORKFLOW_SA}` placeholders substituted via `envsubst`. -7. The umbrella chart's internal DNS references are keyed off `.Release.Name`; a non-default release name breaks inline URLs in `values.yaml`. Acknowledged as tech debt; the planned follow-up migrates all app services to read the platform ConfigMap via `envFrom` so inline URLs disappear from values. +1. Airbyte auth was off in the curated values. Now `global.auth.enabled: true`; the Airbyte L2 release generates a random admin password on first install into `airbyte-auth-secrets/instance-admin-password`, which consumers read from that Secret. +2. DB passwords are not auto-generated for local use. Canonical `values.yaml` intentionally leaves infra credentials empty; local development supplies eval values out-of-band (`.env.compose` for the compose stack; wizard-generated values for a local gitops cluster). This is a deliberate choice over `randAlphaNum` hooks because reproducibility across local runs beats marginal "security" on a throwaway environment. +3. Frontend image is linux/amd64 only. The Docker Compose stack can build the frontend from the sibling `insight-front` checkout on arm64 hosts (`FRONTEND_MODE=dev` / `built`); the default `ghcr` mode runs the amd64 image under QEMU. Follow-up is to publish multi-arch images (infra team). +4. Argo's chart expects `controller.instanceID` as a sub-object (`enabled` + `explicitID`), not a plain string. The gitops Argo (L2) install uses the dotted-key form. +5. Argo's supplemental RBAC requires a namespace-scoped Role + Binding for the workflow service account. The gitops path ships `bootstrap/argo-rbac.yaml.tmpl` with `${NAMESPACE}` / `${WORKFLOW_SA}` placeholders substituted via `envsubst`. +6. The umbrella chart's internal DNS references are keyed off `.Release.Name`; a non-default release name breaks inline URLs in `values.yaml`. Acknowledged as tech debt; the planned follow-up migrates all app services to read the platform ConfigMap via `envFrom` so inline URLs disappear from values. **Known gaps (post-consolidation)**: diff --git a/docs/components/deployment/specs/PRD.md b/docs/components/deployment/specs/PRD.md index 536dccdbf..0de6e8ea7 100644 --- a/docs/components/deployment/specs/PRD.md +++ b/docs/components/deployment/specs/PRD.md @@ -47,24 +47,24 @@ date: 2026-05-12 ### 1.1 Purpose -The Deployment subsystem produces **one releasable artifact** for the Insight platform — the `insight` umbrella Helm chart, published per-merge to `oci://ghcr.io/constructorfabric/charts/insight:`. That single artifact is consumed by two distinct paths: the private `infra/insight-gitops` repository, which drives every Cyberfabric-operated cluster (internal `dev` / `test` / `stage` and each customer-named production cluster — `acme`, `globex`, …), and any external Helm-aware consumer that wants to run Insight on their own terms (helm, ArgoCD, Flux, kustomize render). The same chart also powers the developer bring-up wrapper `dev-up.sh`, which targets a local Kind/OrbStack cluster. +The Deployment subsystem produces **one releasable artifact** for the Insight platform — the `insight` umbrella Helm chart, published per-merge to `oci://ghcr.io/constructorfabric/charts/insight:`. That single artifact is consumed by two distinct paths: the private `infra/insight-gitops` repository, which drives every Cyberfabric-operated cluster (internal `dev` / `test` / `stage` and each customer-named production cluster — `acme`, `globex`, …), and any external Helm-aware consumer that wants to run Insight on their own terms (helm, ArgoCD, Flux, kustomize render). The gitops path also runs locally against a Kind/OrbStack cluster (`make deploy ENV=local`) so developers can exercise the published chart on a real cluster shape. Day-to-day backend / frontend development uses a separate Docker Compose stack (`dev-compose.sh`) that does **not** consume the chart. -The subsystem does not ship product functionality on its own — it composes the application services (API Gateway, Analytics API, Frontend, optional Identity Resolution) with their required infrastructure (ClickHouse, MariaDB, Redis, Redpanda, Airbyte, Argo Workflows) into a versioned chart, enforces the contracts between them (single-namespace dev mode, layered L0/L2/L3 gitops mode, external-mode infra contracts, fail-fast validation, mandatory OIDC in production) and provides the dev wrapper. Orchestration of *customer* installs that are not Cyberfabric-operated is explicitly out of scope: external chart consumers pick their own tooling. +The subsystem does not ship product functionality on its own — it composes the application services (API Gateway, Analytics API, Frontend, optional Identity Resolution) with their required infrastructure (ClickHouse, MariaDB, Redis, Redpanda, Airbyte, Argo Workflows) into a versioned chart, enforces the contracts between them (single-namespace fat mode, layered L0/L2/L3 gitops mode, external-mode infra contracts, fail-fast validation, mandatory OIDC in production) and provides the Docker Compose dev stack. Orchestration of *customer* installs that are not Cyberfabric-operated is explicitly out of scope: external chart consumers pick their own tooling. ### 1.2 Background / Problem Statement -Before the consolidation captured in [ADR-0001](./ADR/0001-chart-publishing-on-merge.md), the Insight stack was distributed through three overlapping paths: a canonical shell installer (`deploy/scripts/install.sh`), an ArgoCD App-of-Apps shipped in the public repo (`deploy/gitops/`), and the developer wrapper. Each path had its own assumptions about where the chart came from, how images were tagged, and what the operator had to clone. The result was structural drift — engineers overriding `image.tag` per service in environment values while a chart-shape change shipped on an independent cadence — and three documentation surfaces for what should have been one product. +Before the consolidation captured in [ADR-0001](./ADR/0001-chart-publishing-on-merge.md), the Insight stack was distributed through several overlapping paths: a shell installer chain, an ArgoCD App-of-Apps shipped in the public repo, and a Kind-based developer bring-up wrapper. Each path had its own assumptions about where the chart came from, how images were tagged, and what the operator had to clone. The result was structural drift — engineers overriding `image.tag` per service in environment values while a chart-shape change shipped on an independent cadence — and multiple documentation surfaces for what should have been one product. Two concrete pain points drove the rewrite. First, Cyberfabric SRE needs to ship the same chart to a growing roster of customer-named production clusters from a private settings-only repo, without forcing every operator workstation to clone the public application repo or to hand-pick image tags. Second, external chart consumers (Constructor Platform, enterprise evaluators) need a stable artifact reference they can pin in their own tooling — not a recipe for assembling an install from a sibling checkout. Both demand a single versioned artifact published per merge with the chart shape and the images coming from the same CI run on the same commit. -The third driver is reproducibility for the development team itself: a developer joining the project should be able to clone the repo, run one script, and end up with a live stack that mirrors the production topology — so that layout bugs are caught in dev rather than in a customer environment. The dev wrapper therefore reuses the same chart, in single-namespace mode via per-service `.deploy: true` toggles, instead of a parallel dev-only chart. +The third driver is reproducibility for the development team itself: a developer joining the project should be able to clone the repo, run one command, and end up with a live stack. Day-to-day work uses the Docker Compose stack (`dev-compose.sh up`), which needs only Docker on the host. When a change has to be validated against the real cluster shape — chart edits, ingestion work that needs Airbyte / Argo — the same gitops path that drives production runs locally (`make deploy ENV=local`) so layout bugs are caught before they reach a customer environment, against the very chart artifact customers consume. ### 1.3 Goals (Business Outcomes) - Eliminate chart-vs-image drift structurally: every published umbrella version bundles a known set of image tags (each subchart's `appVersion`), so pinning the umbrella semver in a gitops repo pins both chart shape and images atomically. Measured by absence of out-of-band `image.tag` overrides in env values files after the migration. - Reduce the time from "Cyberfabric SRE merges a fix" to "the fix is running on `dev`" to under one hour, measured end-to-end from PR merge to pod rollout, by automating chart publish + poller-driven `.insight-version` bump. - Enable Constructor Platform onboarding by allowing each infra dependency to be flipped from bundled to external via a single `.deploy: false` toggle plus the same flat `host` / `port` / `passwordSecret` fields the bundled mode reads, so a shared-platform tenant install reuses the platform's ClickHouse / MariaDB / Redpanda without code changes. -- Keep developer inner-loop under 10 minutes from `dev-up.sh` to a usable cluster with locally built images, so platform changes can be tested against a realistic topology before review. +- Keep the developer inner-loop fast: `dev-compose.sh up` brings up a usable stack from source on a laptop, and `make deploy ENV=local` reproduces the production cluster topology so platform changes can be tested against a realistic shape before review. - Prevent accidental shipping of default passwords or placeholder secrets by failing `helm install` fast when credentials are empty and no external Secret is declared. ### 1.4 Glossary @@ -73,16 +73,16 @@ The third driver is reproducibility for the development team itself: a developer |------|------------| | Umbrella chart | The `charts/insight/` Helm chart that aggregates all Insight subcharts (infra + app services + ingestion templates) via Chart.yaml dependencies. Published per merge to `oci://ghcr.io/constructorfabric/charts/insight:`. | | Chart Publishing CI | The GitHub Actions workflow in `constructorfabric/insight` that, on every merge to `main`, builds the changed service images, bumps the affected subcharts' `appVersion` to the build tag, patch-bumps the umbrella's `version`, sets the umbrella `appVersion` to the build tag, packages the chart, and pushes it to GHCR. | -| Dev wrapper | `dev-up.sh` (and `dev-down.sh`) — bring-up scripts that build images from source, create a local Kind/OrbStack cluster, and install the same umbrella chart with all `.deploy: true` for a single-namespace fat install. | +| Compose dev stack | `dev-compose.sh` — the Docker Compose path for laptop development. Builds the backend services and frontend from source (or pulls published images), runs them alongside bundled MariaDB / ClickHouse / Redis / Redpanda containers. Does not consume the umbrella chart and does not ship Airbyte / Argo Workflows. | | Gitops repo | Private `infra/insight-gitops` settings-only repository on internal GitLab that drives every Cyberfabric-operated cluster. Pins exactly one umbrella semver per environment via `.insight-version` and pulls the chart from OCI at deploy time; does **not** vendor the chart. | | L0 / L2 / L3 | Three deploy layers used by gitops production: **L0 Bootstrap** (cluster prerequisites — sealed-secrets-controller, ingress-nginx, cert-manager — cluster-scoped, runs once per cluster); **L2 System** (shared stateful infra — MariaDB, ClickHouse, Redis, Redpanda + Console, Airbyte, Argo Workflows — one Helm release per service in the `insight-infra` namespace, each replaceable by a managed external endpoint); **L3 App** (the umbrella chart, app services only, in the `insight` namespace). There is no L1 — that number is reserved for cluster provisioning, which is out of scope. | | Customer-named env | A production environment named after the customer that owns it (`acme`, `globex`, …). The gitops repo has no generic "prod"; each customer install is its own env directory and its own `kubectl` context (`insight-`). | -| Dual-purpose umbrella | One chart, two install shapes selected by `.deploy` toggles. `dev-up.sh` flips them all `true` (single fat `insight` namespace with bundled infra); gitops production flips them all `false` (app services only in `insight`, infra elsewhere). | +| Dual-purpose umbrella | One chart, two install shapes selected by `.deploy` toggles. An external consumer who wants everything in one namespace flips them all `true` (single fat `insight` namespace with bundled infra); gitops (production and local) flips them all `false` (app services only in `insight`, infra elsewhere). | | External mode | State of an infra dependency where `.deploy: false`. The umbrella does not run the bundled subchart; consumers read the same flat `.host`, `.port` and `.passwordSecret` fields and the Secret is provided by the operator (or platform). The same shape used by gitops production cross-namespace wiring (`.insight-infra.svc.cluster.local`) and by Constructor Platform tenant installs (platform-issued endpoints). | | Constructor Platform | Shared multi-product infrastructure fabric operated by the vendor. It provides ClickHouse, MariaDB, Redpanda and identity services that tenant products consume via external-mode contracts. | | Platform ConfigMap | The single `{release}-platform` ConfigMap emitted by the umbrella that contains resolved infra coordinates (CLICKHOUSE_URL, MARIADB_HOST, AIRBYTE_API_URL, …). Pods consume it via `envFrom`. | | `.insight-version` | One-line file at the root of `infra/insight-gitops` containing the umbrella semver currently pinned for the repo. The poller bumps it on auto-bumped envs; engineers bump it via merge request for customer-named envs. | -| Eval credentials | Throwaway passwords in `deploy/values-dev.yaml` used only by dev bring-up and short-lived eval clusters; never shipped to production. | +| Eval credentials | Throwaway passwords used only by local development — in `.env.compose` for the Docker Compose stack, or wizard-generated for a local gitops cluster; never shipped to production. | ## 2. Actors @@ -114,7 +114,7 @@ The third driver is reproducibility for the development team itself: a developer **ID**: `cpt-insightspec-actor-platform-developer` **Role**: Engineer on the Insight team iterating on the services, charts or ingestion code. -**Needs**: A single dev wrapper that builds images locally, bootstraps a Kind cluster, applies dev overlays, port-forwards the relevant services, and exercises the same umbrella chart that customers run (with `.deploy: true` flipped on for in-cluster bundled infra) so bugs show up before release. +**Needs**: A fast laptop loop for backend / frontend work (the Docker Compose stack, `dev-compose.sh`, with build-from-source and auto-reload) and a way to exercise the same umbrella chart that customers run on a real cluster shape (the gitops path locally, `make deploy ENV=local`) so chart and ingestion bugs show up before release. ### 2.2 System Actors @@ -128,7 +128,7 @@ The third driver is reproducibility for the development team itself: a developer **ID**: `cpt-insightspec-actor-helm` -**Role**: Package manager used by every consumer path (the gitops Makefile, `dev-up.sh`, and any external consumer). The umbrella ships as a Helm chart at `oci://ghcr.io/constructorfabric/charts/insight`; Airbyte and Argo Workflows are their upstream Helm charts pinned per L2 system release (gitops) or installed via dev-helper scripts (`dev-up.sh`). +**Role**: Package manager used by every chart consumer path (the gitops Makefile and any external consumer). The umbrella ships as a Helm chart at `oci://ghcr.io/constructorfabric/charts/insight`; Airbyte and Argo Workflows are their upstream Helm charts pinned per L2 system release (gitops, including the local `ENV=local` cluster). #### Chart Publishing CI @@ -140,13 +140,13 @@ The third driver is reproducibility for the development team itself: a developer **ID**: `cpt-insightspec-actor-argo-workflows` -**Role**: Engine that executes the ingestion `WorkflowTemplates` emitted by the umbrella. Installed as a separate Helm release — in `insight-infra` for gitops production (L2), in the `insight` namespace for `dev-up.sh` local — scoped to the install via `controller.instanceID` and `controller.workflowNamespaces`. +**Role**: Engine that executes the ingestion `WorkflowTemplates` emitted by the umbrella. Installed as a separate Helm release in `insight-infra` (L2) by the gitops path, on both production and local (`ENV=local`) clusters — scoped to the install via `controller.instanceID` and `controller.workflowNamespaces`. #### Airbyte Engine **ID**: `cpt-insightspec-actor-airbyte-engine` -**Role**: Data extraction engine. Installed as a separate Helm release in `insight-infra` (gitops production) or the `insight` namespace (`dev-up.sh` local), pinned to chart 1.8.5+ / app 1.8.5+ per release. Post-install setup-wizard automation completes Airbyte's one-time setup via its REST API so the UI is usable on first visit. +**Role**: Data extraction engine. Installed as a separate Helm release in `insight-infra` (L2) by the gitops path, on both production and local (`ENV=local`) clusters, pinned to chart 1.8.5+ / app 1.8.5+ per release. Post-install setup-wizard automation completes Airbyte's one-time setup via its REST API so the UI is usable on first visit. #### OCI Artifact Registry (GHCR) @@ -160,12 +160,12 @@ The third driver is reproducibility for the development team itself: a developer - Target Kubernetes version: 1.27 or newer (declared via `kubeVersion: ">=1.27.0-0"` in the umbrella `Chart.yaml`). - Helm 3.14+ required for OCI chart pulls and registry authentication. -- Docker Desktop / Rancher Desktop / OrbStack / Podman with working containerd image load required on the developer path (the dev wrapper uses `kind load docker-image`). +- Docker (Engine 24+, compose v2) is required for the Docker Compose dev stack. The local gitops path additionally needs a local Kubernetes cluster (Kind / OrbStack / k3d / minikube) with working containerd image load. - Bitnami chart dependencies (MariaDB, Redis) are pinned to the `bitnamilegacy` registry variants with `global.security.allowInsecureImages: true`, because Bitnami moved free images off `docker.io/bitnami/*` in 2025. -- Frontend image is currently published as `linux/amd64` only. On Apple Silicon hosts the dev wrapper rebuilds the frontend from the sibling `insight-front` checkout rather than pulling the upstream image; production installs rely on Docker Desktop's QEMU emulation or a customer-side multi-arch mirror. +- Frontend image is currently published as `linux/amd64` only. On Apple Silicon hosts the Docker Compose stack can rebuild the frontend from the sibling `insight-front` checkout (`FRONTEND_MODE=dev` / `built`) instead of pulling the upstream image; the default `ghcr` mode and production installs rely on QEMU emulation or a customer-side multi-arch mirror. - The umbrella chart assumes release name `insight` for its internal DNS references inside `values.yaml`. Using a non-default release name requires overriding the affected URL fields. - For gitops production, each cluster carries exactly one Insight install; the cluster's identity (which env it represents) lives in the kube-context name (`insight-`) and the gitops repo's `environments//` directory — not in the namespace. The two well-known namespaces (`insight-infra` for L2, `insight` for L3) are the same across every install. -- The dev wrapper targets Kind 0.22+ / OrbStack; cluster bootstrapping uses a fixed cluster name `insight` to match hard-coded port mappings. +- The local gitops path (`make deploy ENV=local`) targets a developer-supplied Kind 0.22+ / OrbStack cluster; the operator brings their own cluster and `$KUBECONFIG`. ## 4. Scope @@ -175,20 +175,19 @@ The third driver is reproducibility for the development team itself: a developer - The service-resolution helper library (`templates/_helpers.tpl`) that returns the same values whether a dependency is bundled or external, and the `insight.validate` template that fails rendering on missing required fields. - The single `{release}-platform` ConfigMap that exposes resolved infra coordinates to every pod in the namespace via `envFrom`. - Argo `WorkflowTemplate` emission as first-class Helm templates under `charts/insight/templates/ingestion/*.yaml`, gated by `ingestion.templates.enabled` and consuming umbrella helpers (`insight.clickhouse.fqdn`, `insight.airbyte.url`, …) directly via `include`. -- The dual-purpose `.deploy: true|false` toggle: same chart powers `dev-up.sh` single-namespace fat installs and gitops production layered installs. +- The dual-purpose `.deploy: true|false` toggle: the same chart serves single-namespace fat installs (external consumers who want everything in one namespace) and gitops layered installs (production and local). - The Chart Publishing CI workflow that publishes the umbrella to `oci://ghcr.io/constructorfabric/charts/insight:` per merge to `main`; the per-subchart `appVersion = image tag` contract; the umbrella semver versioning rules; `.insight-version` as the single gitops pin. -- Developer bring-up wrappers `dev-up.sh` / `dev-down.sh` / `init.sh`, parameterised by `INSIGHT_NAMESPACE`, with Kind bootstrap, image build + `kind load`, and port-forwards for the common UIs. -- Dev-only credential overlay `deploy/values-dev.yaml` (throwaway passwords) applied automatically by `dev-up.sh`. -- The DEVLOG.md that records the first-run debugging narrative so future developers and dev-up users can resolve the same twelve issues without rediscovering them. +- The Docker Compose dev stack (`dev-compose.sh` + `docker-compose.yml`) for laptop development: builds backend services and frontend from source (or pulls published images), runs bundled MariaDB / ClickHouse / Redis / Redpanda containers, auto-reloads on rebuild, and auto-seeds a demo dataset on first `up`. +- Dev / eval credentials confined to local-only artifacts (`.env.compose` for compose; wizard-generated values for a local gitops cluster); never present in the canonical chart values or any published artifact. - The chart README (`charts/insight/README.md`) as the values contract for every consumer. ### 4.2 Out of Scope - **Orchestration of customer / external installs.** Anyone consuming the chart from OCI picks their own tooling (helm, ArgoCD, Flux, Terraform Helm provider, kustomize render, custom GitOps). The Deployment subsystem does not ship a customer-facing installer script; it ships the chart and documents the values contract. -- **The `infra/insight-gitops` repository content itself.** That repo is owned operationally by Cyberfabric SRE and lives on private GitLab. Its design and operational contract are captured in the [gitops SPEC](../gitops/README.md); the public deployment PRD/DESIGN documents the chart-as-artifact and the dev wrapper, not the gitops repo's internals. +- **The `infra/insight-gitops` repository content itself.** That repo is owned operationally by Cyberfabric SRE and lives on private GitLab. Its design and operational contract are captured in the [gitops SPEC](../gitops/README.md); the public deployment PRD/DESIGN documents the chart-as-artifact and the Docker Compose dev stack, not the gitops repo's internals. - Multi-architecture (linux/arm64) frontend image publication. - Bidirectional sync between the umbrella-managed `insight-db-creds` Secret and a customer-supplied secret-management system (Vault, AWS Secrets Manager, External Secrets Operator). Customers integrating with such systems pre-create `insight-db-creds` themselves (the chart auto-detects BYO via absence of the `app.kubernetes.io/managed-by=Helm` label and skips its own Secret-template emission); or they accept the auto-generated values and mirror them outwards by their own means. -- Cluster provisioning (creating the customer's Kubernetes cluster, setting up a StorageClass, installing ingress-nginx on a production cluster). The dev wrapper bootstraps Kind for local work; gitops production handles L0 in `infra/insight-gitops` (out of this repo's scope); external chart consumers bring their own cluster. +- Cluster provisioning (creating the customer's Kubernetes cluster, setting up a StorageClass, installing ingress-nginx on a production cluster). For local work the developer brings their own Kind/OrbStack cluster; the gitops path handles L0 bootstrap on it; external chart consumers bring their own cluster. - Backup, restore, and disaster-recovery workflows for the bundled stateful services (ClickHouse, MariaDB). Mentioned in the Backend PRD; not owned by Deployment. - Identity Provider (OIDC) provisioning. The deployment contract requires OIDC credentials as input; standing up an IdP is the consumer's responsibility. - Customer-facing documentation portal. Internal README files, the chart README, the gitops SPEC and DEVLOG.md are in scope; hosted docs are not. @@ -329,12 +328,12 @@ The umbrella's `Chart.yaml` `version` **MUST** follow semver with patch-bump per The umbrella chart **MUST** keep its infrastructure subcharts (`clickhouse`, `mariadb`, `redis`, `redpanda`) gated by per-service `.deploy: true|false` toggles so the same chart serves two install shapes: -- **Single-namespace fat install** (`.deploy: true` for every infra, used by `dev-up.sh` and by any external consumer who is fine running everything in one namespace): the umbrella renders MariaDB, ClickHouse, Redis, Redpanda **and** the app services together in the `insight` namespace. +- **Single-namespace fat install** (`.deploy: true` for every infra, used by any external consumer who is fine running everything in one namespace): the umbrella renders MariaDB, ClickHouse, Redis, Redpanda **and** the app services together in the `insight` namespace. - **Layered app-only install** (`.deploy: false` for every infra, used by gitops production): the umbrella renders the app services only into `insight`; infra services come from L2 in `insight-infra` (gitops Cyberfabric clusters) or from managed external endpoints / a separate team's namespace (Constructor Platform, external customers). The same chart shape **MUST** render under both configurations; cross-namespace wiring uses the same `.host` / `.port` shape as Constructor Platform external mode. -**Rationale**: One chart, two operating modes — same templates exercise both, so a bug in app rendering is caught locally on `dev-up.sh` before it reaches a production cluster. +**Rationale**: One chart, two operating modes — same templates exercise both. The layered shape that production runs is the same one a developer renders locally via `make deploy ENV=local`, so a bug in app rendering is caught before it reaches a production cluster. **Actors**: `cpt-insightspec-actor-platform-developer`, `cpt-insightspec-actor-cyberfabric-sre`, `cpt-insightspec-actor-customer-sre` @@ -368,7 +367,7 @@ The gitops repo **MUST** name every production environment after the customer th - [ ] `p2` - **ID**: `cpt-insightspec-fr-dep-namespace-convention` -Every cluster targeted by gitops production **MUST** carry exactly two Insight-owned namespaces: `insight-infra` for L2 shared services and `insight` for the L3 umbrella release. The cluster's environment identity **MUST** live in the kube-context name and the gitops repo directory — not in the namespace. This matches the `dev-up.sh` local convention (everything in `insight`) and the external chart consumer's expectation of a single `insight` release name. +Every cluster targeted by gitops production **MUST** carry exactly two Insight-owned namespaces: `insight-infra` for L2 shared services and `insight` for the L3 umbrella release. The cluster's environment identity **MUST** live in the kube-context name and the gitops repo directory — not in the namespace. This holds identically on the local gitops cluster (`ENV=local`) and matches the external chart consumer's expectation of a single `insight` release name. **Rationale**: Two well-known namespace names across every install make tooling, runbooks and `kubectl` commands reproducible across customer environments; the env identity stays out of the namespace string so the chart shape does not vary by env. @@ -376,23 +375,23 @@ Every cluster targeted by gitops production **MUST** carry exactly two Insight-o ### 5.5 Developer Workflow -#### Dev wrapper for local bring-up +#### Docker Compose dev stack for local bring-up - [ ] `p2` - **ID**: `cpt-insightspec-fr-dep-dev-wrapper` -The system **MUST** ship `dev-up.sh` (renamed from the legacy `up.sh`) that bootstraps a Kind/OrbStack cluster, builds backend images from source and loads them into the cluster, builds the frontend image from the sibling `insight-front` checkout (with `docker pull --platform` fallback) to avoid Apple Silicon arm64/amd64 manifest mismatches, applies `deploy/values-dev.yaml` automatically, installs Airbyte and Argo Workflows into the local `insight` namespace via dev-helper scripts, installs the umbrella chart with `.deploy: true` for the infra subcharts, and opens port-forwards for the common UIs (Frontend :8003, API Gateway :8080, Airbyte UI :8002, Airbyte API :8001, Argo UI :2746, ClickHouse HTTP :8123). +The system **MUST** ship a Docker Compose dev stack (`dev-compose.sh` + `docker-compose.yml`) that, with only Docker on the host, builds the backend services (Rust + .NET) and the frontend from source in builder containers — or pulls their published images on demand — runs them alongside bundled MariaDB / ClickHouse / Redis / Redpanda containers, auto-reloads each backend service on rebuild, auto-seeds a demo dataset on first `up`, and publishes the web services on configurable host ports (Frontend :3000, API Gateway :8080, Analytics API :8081, Identity :8082, ClickHouse :8123, MariaDB :3306, Redis :6379). A first-run wizard captures the MariaDB / ClickHouse / tenant / dev-email choices. The stack does not consume the umbrella chart and does not ship Airbyte / Argo Workflows; ingestion work uses the local gitops path instead. -**Rationale**: The dev path must exercise the same umbrella chart as production so layout bugs are caught in dev; image build + Kind loading + dev overlays must be invisible to the developer. +**Rationale**: The day-to-day backend / frontend loop must be fast and require no Kubernetes; chart-shape and ingestion validation belongs on the local gitops cluster (`make deploy ENV=local`), which exercises the same artifact production consumes. **Actors**: `cpt-insightspec-actor-platform-developer` -#### Namespace parameterisation +#### Configurable local stack for parallel work - [ ] `p2` - **ID**: `cpt-insightspec-fr-dep-dev-namespace-param` -`dev-up.sh`, `dev-down.sh` and `init.sh` **MUST** honour an `INSIGHT_NAMESPACE` environment variable defaulting to `insight`, so that multiple concurrent dev environments can share a single Kind cluster by choosing distinct namespaces. +The Docker Compose dev stack **MUST** allow every published host port and the frontend / backend image sources to be overridden via `.env.compose` (and equivalent `dev-compose.sh` flags), so that more than one stack — or a stack pointed at external MariaDB / ClickHouse — can run on a single host without collisions. -**Rationale**: Two parallel feature branches on the same cluster is a common developer need; hard-coded namespaces block that. +**Rationale**: Working on two branches at once, or running against shared external databases, is a common developer need; hard-coded ports and image sources block that. **Actors**: `cpt-insightspec-actor-platform-developer` @@ -402,7 +401,7 @@ The system **MUST** ship `dev-up.sh` (renamed from the legacy `up.sh`) that boot - [ ] `p1` - **ID**: `cpt-insightspec-fr-dep-tenant-isolation-boundary` -Each Cyberfabric-operated cluster **MUST** host exactly one Insight install (one customer per cluster) — tenant separation across customers is at the cluster boundary. Two installs on a single shared cluster (Constructor Platform tenants, dev wrapper concurrent runs) **MUST** be isolated by distinct namespaces, with Argo Workflows scoped via `controller.workflowNamespaces` and `controller.instanceID`. No ClusterRole or ClusterRoleBinding is created by L3 deploys; cross-namespace DNS is the only cross-namespace coupling. +Each Cyberfabric-operated cluster **MUST** host exactly one Insight install (one customer per cluster) — tenant separation across customers is at the cluster boundary. Two installs on a single shared cluster (Constructor Platform tenants) **MUST** be isolated by distinct namespaces, with Argo Workflows scoped via `controller.workflowNamespaces` and `controller.instanceID`. No ClusterRole or ClusterRoleBinding is created by L3 deploys; cross-namespace DNS is the only cross-namespace coupling. **Rationale**: Cluster-per-customer is the gitops production model; namespace-per-tenant is the shared-cluster model. Both axes need to work without changing the chart shape. @@ -424,7 +423,7 @@ The canonical `charts/insight/values.yaml` **MUST** leave all credential fields - [ ] `p2` - **ID**: `cpt-insightspec-fr-dep-dev-overlay-isolation` -Eval / dev credentials **MUST** live in a separate file `deploy/values-dev.yaml` that is applied only by the dev wrapper, and **MUST NOT** appear anywhere in the canonical chart values or in any published artifact. Production credentials reach the cluster through sealed secrets fed from a corporate secret store (Passbolt in the gitops model), or through customer-owned Secrets pre-created in the namespace, never via committed values files. +Eval / dev credentials **MUST** be confined to local-only artifacts — `.env.compose` for the Docker Compose stack (gitignored dotenv), or wizard-generated values for a local gitops cluster — and **MUST NOT** appear anywhere in the canonical chart values or in any published artifact. Production credentials reach the cluster through sealed secrets fed from a corporate secret store (Passbolt in the gitops model), or through customer-owned Secrets pre-created in the namespace, never via committed values files. **Rationale**: Keeps throwaway eval passwords out of the production code path by construction; keeps real credentials out of git by construction. @@ -442,7 +441,7 @@ Two Insight installs on the same Kubernetes cluster in different namespaces **MU **Threshold**: no cross-namespace RBAC binding created by an L3 deploy; Argo controllers scoped via `controller.workflowNamespaces` and `controller.instanceID`. -**Rationale**: Constructor Platform operates as a shared fabric and the dev wrapper supports parallel branches per developer — cross-tenant leakage would be a platform-level incident. +**Rationale**: Constructor Platform operates as a shared fabric with multiple tenant installs per cluster — cross-tenant leakage would be a platform-level incident. #### Fail-fast on misconfiguration @@ -466,8 +465,8 @@ Every merge to `main` of `constructorfabric/insight` **MUST** publish a new umbr ### 6.2 NFR Exclusions -- **Install time target**: removed in this revision. The Deployment subsystem no longer ships an opinionated installer; install duration depends on the consumer's choice of tooling, the cluster's image pull bandwidth and the layered model's pre-existing L2 state. Time-to-Ready measurements live with the consumer (gitops Makefile timing for Cyberfabric SRE; `dev-up.sh` self-times for developers). -- **Availability target (REL-PRD-001)**: Not applicable because the Deployment subsystem produces a chart artifact and a dev wrapper, not a running service. The availability SLO of the running platform is defined in the Backend PRD. +- **Install time target**: removed in this revision. The Deployment subsystem no longer ships an opinionated installer; install duration depends on the consumer's choice of tooling, the cluster's image pull bandwidth and the layered model's pre-existing L2 state. Time-to-Ready measurements live with the consumer (gitops Makefile timing for Cyberfabric SRE and local clusters; the Docker Compose stack reports its own build / seed time for developers). +- **Availability target (REL-PRD-001)**: Not applicable because the Deployment subsystem produces a chart artifact and a dev stack, not a running service. The availability SLO of the running platform is defined in the Backend PRD. - **Recovery targets RPO/RTO (REL-PRD-002)**: Not applicable because Deployment does not persist runtime state. Backup/restore of the data stores is defined separately; see Backend PRD and the Ingestion Layer PRD. - **Performance response-time expectations (PERF-PRD-001)**: Not applicable because no user-facing request path lives inside the Deployment subsystem. - **Accessibility (UX-PRD-002)**: Not applicable because this subsystem has no end-user UI; it is operator-facing CLI and YAML. @@ -475,7 +474,7 @@ Every merge to `main` of `constructorfabric/insight` **MUST** publish a new umbr - **Offline capability (UX-PRD-004)**: Not applicable because chart distribution inherently requires registry connectivity; offline / air-gapped installs (image + chart pre-loaded into a customer registry) are a future consideration. - **Inclusivity (UX-PRD-005)**: Not applicable because the audience is a narrow technical one — SREs and platform engineers. - **Regulatory compliance (COMPL-PRD-001)**: Not applicable at this layer because the Deployment subsystem does not process personal data; regulatory obligations apply to the running platform and are captured in the Backend PRD. -- **Privacy by Design (SEC-PRD-005)**: Not applicable — no personal data flows through the chart artifact or the dev wrapper. +- **Privacy by Design (SEC-PRD-005)**: Not applicable — no personal data flows through the chart artifact or the dev stack. - **Safety (SAFE-PRD-001/002)**: Not applicable — software-only artifact pipeline with no physical side effects. ## 7. Public Library Interfaces @@ -490,7 +489,7 @@ Every merge to `main` of `constructorfabric/insight` **MUST** publish a new umbr **Stability**: unstable (pre-1.0 while the chart is at `version: 0.1.x`). -**Description**: The values contract that every consumer (Cyberfabric SRE in gitops, Constructor Platform, external customers, `dev-up.sh`) targets. It covers the `credentials` block (`autoGenerate`), the `global` block, the four infra blocks (ClickHouse, MariaDB, Redis, Redpanda) each with the unified flat shape (`deploy`, `host`, `port`, `database`, `username`, `passwordSecret`), the three mandatory app-service blocks (apiGateway, analyticsApi, frontend) plus the optional `identityResolution` (`deploy`-gated), and the `airbyte` + `ingestion.templates` blocks. The `.deploy` toggles drive the dual-purpose dev-vs-prod split documented in [§5.4](#54-layered-deploy-model-and-customer-envs). +**Description**: The values contract that every chart consumer (Cyberfabric SRE in gitops — production and local, Constructor Platform, external customers) targets. It covers the `credentials` block (`autoGenerate`), the `global` block, the four infra blocks (ClickHouse, MariaDB, Redis, Redpanda) each with the unified flat shape (`deploy`, `host`, `port`, `database`, `username`, `passwordSecret`), the three mandatory app-service blocks (apiGateway, analyticsApi, frontend) plus the optional `identityResolution` (`deploy`-gated), and the `airbyte` + `ingestion.templates` blocks. The `.deploy` toggles drive the dual-purpose dev-vs-prod split documented in [§5.4](#54-layered-deploy-model-and-customer-envs). **Breaking Change Policy**: minor version bump on the umbrella for additive fields; major version bump for removed or renamed values keys; the validator output must name any newly required field. @@ -514,7 +513,7 @@ Every merge to `main` of `constructorfabric/insight` **MUST** publish a new umbr **Direction**: required from client (Insight consumes Airbyte's API). -**Protocol/Format**: HTTP/JSON on the Airbyte REST API; bearer token obtained via OAuth2 client_credentials at `/api/v1/applications/token` using `instance-admin-client-id` / `instance-admin-client-secret` from the `airbyte-auth-secrets` Secret created by the Airbyte chart. No JWT signing on our side. In gitops production, Airbyte runs as an L2 release in `insight-infra`; in `dev-up.sh` it runs in the local `insight` namespace; the chart reads `airbyte.apiUrl` either way. +**Protocol/Format**: HTTP/JSON on the Airbyte REST API; bearer token obtained via OAuth2 client_credentials at `/api/v1/applications/token` using `instance-admin-client-id` / `instance-admin-client-secret` from the `airbyte-auth-secrets` Secret created by the Airbyte chart. No JWT signing on our side. Airbyte runs as an L2 release in `insight-infra` on every gitops cluster (production and local); the chart reads `airbyte.apiUrl` to reach it. **Compatibility**: pinned to Airbyte chart 1.8.5+ / app 1.8.5+ at the consumer side. Chart 1.9.x was intentionally skipped while its bundled app was 2.0.x-alpha. Version bumps happen in dedicated PRs with regression tests. @@ -546,22 +545,22 @@ Every merge to `main` of `constructorfabric/insight` **MUST** publish a new umbr **Actors**: `cpt-insightspec-actor-platform-developer`, `cpt-insightspec-actor-customer-sre` -**Preconditions**: Docker Desktop or equivalent is running; kubectl and helm 3.14+ are installed; no Insight stack is running. +**Preconditions**: Docker (Engine 24+, compose v2) is running; no Insight stack is running. No Rust / .NET / Node / kubectl / helm needed. **Main Flow**: -1. Operator clones the repository and copies `.env.local.example` to `.env.local`. -2. Operator runs `./dev-up.sh --env local`. -3. Dev wrapper creates a Kind/OrbStack cluster, builds backend images, loads them, applies `deploy/values-dev.yaml` (which flips `.deploy: true` for the infra subcharts), and installs Airbyte and Argo Workflows in the local `insight` namespace via the dev-helper scripts. -4. Dev wrapper installs the umbrella chart in single-namespace fat mode; pods reach Ready. -5. Port-forwards open for Frontend, API Gateway, Airbyte UI, Argo UI, ClickHouse HTTP. -6. Operator opens http://localhost:8003 and sees the Insight UI. +1. Operator clones the repository and runs `./dev-compose.sh up`. +2. The first-run wizard prompts for local-vs-external MariaDB / ClickHouse, a dev-user email, and the frontend mode (default pulls the published `insight-front` image). +3. The script builds the backend services in a builder container, brings up the compose stack (backend + frontend + bundled MariaDB / ClickHouse / Redis / Redpanda), and auto-seeds a demo dataset. +4. Containers reach healthy; the seed completes. +5. Operator opens http://localhost:3000 and sees the Insight UI populated with demo data. -**Postconditions**: all pods are Ready in namespace `insight`; eval credentials are in effect; Airbyte setup wizard is complete. +**Postconditions**: all containers are healthy; eval credentials live in `.env.compose`; the demo dataset is seeded. **Alternative Flows**: -- **Apple Silicon host**: dev-up detects arm64, falls back to `docker pull --platform linux/amd64` for the frontend image, Docker Desktop's QEMU emulation runs it. +- **Apple Silicon host**: the default `ghcr` frontend mode pulls the published linux/amd64 image and runs it under QEMU; for active frontend work the operator sets `FRONTEND_MODE=dev` to build from the sibling `insight-front` checkout natively. +- **Cluster-shape eval**: to evaluate the real Kubernetes topology (Airbyte / Argo / the umbrella chart) instead of compose, the operator runs `cd deploy/gitops && make deploy ENV=local` against a local Kind/OrbStack cluster. ### 8.2 Constructor Platform tenant install @@ -590,28 +589,29 @@ Every merge to `main` of `constructorfabric/insight` **MUST** publish a new umbr **Actors**: `cpt-insightspec-actor-platform-developer` -**Preconditions**: developer has a checked-out repo, a working Kind/OrbStack cluster (or is about to create one via `dev-up.sh`), and is iterating on a backend service. +**Preconditions**: developer has a checked-out repo, the Docker Compose stack already up (`./dev-compose.sh up`), and is iterating on a backend service. **Main Flow**: 1. Developer makes a code change in `src/backend/...`. -2. Developer runs `./dev-up.sh app` (or full `./dev-up.sh`), which rebuilds the affected image and loads it into Kind. -3. Helm upgrade runs against the local umbrella; the pod is rolled. -4. Developer re-opens the Frontend port-forward and exercises the change. -5. When done, developer runs `./dev-down.sh` to tear everything down. +2. Developer runs `./dev-compose.sh build `, which rebuilds the binary in the builder container. +3. The bind-mounted binary changes; the service container's `watchexec` restarts it in ~1 second (`ENABLE_AUTO_RELOAD=true`). +4. Developer exercises the change at http://localhost:3000 (or curls the gateway on :8080). +5. When done, developer runs `./dev-compose.sh down` (containers stop, volumes preserved) or `down --volumes` for a clean slate. -**Postconditions**: clean cluster state at the end of the session; no leftover resources. +**Postconditions**: the changed service runs the new build; stack state is clean at session end. ## 9. Acceptance Criteria - [ ] `helm template insight oci://ghcr.io/constructorfabric/charts/insight --version ` with no overlay aborts with a readable message because OIDC and credentials are empty — zero successful renders of a misconfigured install. -- [ ] `helm template insight charts/insight -f deploy/values-dev.yaml` renders cleanly and produces every required Kubernetes object, including the three Argo `WorkflowTemplate` objects. +- [ ] `helm template insight charts/insight -f deploy/gitops/environments/local/values.yaml.template` renders cleanly and produces every required app-service Kubernetes object, including the three Argo `WorkflowTemplate` objects. - [ ] On a merge to `main` of `constructorfabric/insight` that changes one service, the publish-chart workflow builds the image, bumps that subchart's `appVersion`, patch-bumps the umbrella, packages and pushes `oci://ghcr.io/constructorfabric/charts/insight:`, and auto-commits the version bumps back to `main`. - [ ] `helm template` of the pulled chart confirms `image.tag` for the changed service equals the new build tag, others equal their previous tags. -- [ ] `./dev-up.sh` on a fresh laptop installs Airbyte, Argo and the umbrella into the local `insight` namespace; all pods reach Ready without manual intervention. +- [ ] `./dev-compose.sh up` on a fresh laptop brings the compose stack up and auto-seeds demo data without manual intervention; the frontend at http://localhost:3000 shows it. +- [ ] `cd deploy/gitops && make deploy ENV=local` on a local Kind/OrbStack cluster installs the L2 system services (Airbyte, Argo, infra in `insight-infra`) and the L3 umbrella (app services in `insight`); all pods reach Ready. - [ ] Two concurrent installs in namespaces `insight-a` and `insight-b` on the same Kind cluster do not observe each other's Workflow objects. - [ ] With `clickhouse.deploy: false` + a complete `clickhouse.host` / `.port` / `.passwordSecret` block, the resulting pods read from that external ClickHouse via the platform ConfigMap without modification to any subchart. -- [ ] `dev-up.sh` on Apple Silicon succeeds end-to-end without manual `docker pull --platform` calls; the `DEVLOG.md`-documented first-run failures do not regress. +- [ ] `./dev-compose.sh up` on Apple Silicon succeeds end-to-end with the default `ghcr` frontend mode (linux/amd64 image under QEMU) and builder-container backend builds — no manual architecture flags. ## 10. Dependencies @@ -622,8 +622,8 @@ Every merge to `main` of `constructorfabric/insight` **MUST** publish a new umbr | Airbyte chart 1.8.5+ | Data extraction engine; installed as a separate Helm release by every consumer that needs it. | p1 | | Argo Workflows chart 0.45.x | Workflow engine for ingestion pipelines. | p1 | | GHCR (`oci://ghcr.io/constructorfabric/charts/insight`) | Distribution target for the umbrella chart; written by Chart Publishing CI, read by every consumer. | p1 | -| Kind 0.22+ / OrbStack (dev only) | Local Kubernetes for the developer inner loop. | p2 | -| Docker Desktop / containerd with `kind load` support (dev only) | Image ingestion into Kind. | p2 | +| Docker (Engine 24+, compose v2) | Runs the Docker Compose dev stack; also the container runtime for the local Kubernetes cluster. | p2 | +| Kind 0.22+ / OrbStack (local gitops only) | Local Kubernetes for the `make deploy ENV=local` path; image ingestion into the cluster via `kind load`. | p2 | | Bitnami Helm subcharts (MariaDB, Redis) via `bitnamilegacy` | Bundled-infra images still free after Bitnami's 2025 registry change. | p2 | | Consumer-managed OIDC issuer | Required for any non-dev install (fail-fast validator enforces). | p1 | @@ -631,7 +631,7 @@ Every merge to `main` of `constructorfabric/insight` **MUST** publish a new umbr - Cluster operators (Cyberfabric SRE for gitops, customer SREs for external installs) provide a working default StorageClass and an ingress controller; the Deployment subsystem does not provision either. - Operators consuming the chart are comfortable with Helm values files, kubectl, and at least one of (helm, ArgoCD, Flux, Terraform Helm provider); the chart is not targeted at non-technical operators. -- The sibling repository `insight-front` (symlinked as `insight-front_symlink`) is present on developer machines for the dev wrapper's frontend build step. +- The sibling repository `insight-front` is present on developer machines only when the Docker Compose stack runs with `FRONTEND_MODE=dev` or `built`; the default `ghcr` mode pulls the published frontend image, so a fresh laptop with only Docker can run the full stack. - The bundled Airbyte and Argo Workflows versions remain viable for the next release cycle; upgrades to newer minors are handled in dedicated PRs with regression tests over ingestion workflows. - On a shared cluster, tenant isolation is acceptable at the Kubernetes namespace boundary — workloads within a tenant namespace are mutually trusted. On a Cyberfabric-operated cluster, tenant isolation is at the cluster boundary (one customer per cluster). - The Constructor Platform provides stable Secret resource references; tenants receive them out-of-band (not created by the consumer's chart install). @@ -643,7 +643,7 @@ Every merge to `main` of `constructorfabric/insight` **MUST** publish a new umbr |------|--------|------------| | Chart Publishing CI auto-commit-back fails on branch protection. | A merge that should publish a new umbrella tag publishes the chart but fails to bump `Chart.yaml` on `main`, leaving the repo state out of sync with what was published. | Track in repo settings: fine-grained PAT in `RELEASE_PUSH_PAT` with bypass on protected branch, or a GitHub App with bypass rights. Until either is in place, the auto-commit step is replayed manually after merge. | | Inline infra passwords previously had to be duplicated into app-service DSNs. | Drift between infra password and DSN produced a silently-broken install. | Resolved: `credentials.autoGenerate=true` writes `insight-db-creds` once and the umbrella derives all app-service Secrets (`insight-analytics-api-config`, `insight-identity-resolution-config`) from the same passwords. BYO mode reads the customer-supplied `insight-db-creds` instead. | -| Frontend image is `linux/amd64` only — Apple Silicon hosts rely on QEMU emulation or local rebuild. | Slow first pull and occasional emulation bugs on dev machines. | Dev wrapper builds the frontend from source as a workaround; infra team to publish multi-arch images. | +| Frontend image is `linux/amd64` only — Apple Silicon hosts rely on QEMU emulation or local rebuild. | Slow first pull and occasional emulation bugs on dev machines. | The Docker Compose stack can build the frontend from source (`FRONTEND_MODE=dev` / `built`) as a workaround; infra team to publish multi-arch images. | | Identity Resolution subchart ships as MVP stub that crashloops on empty bronze. | If operator flips `identityResolution.deploy: true` before any BambooHR sync, the release looks broken. | Keep default `identityResolution.deploy: false`; document the prerequisite in README; surface a clearer error message in the service itself (Backend concern). | | Airbyte chart 1.9.x was deliberately skipped because its bundled app 2.0.x-alpha is not production-grade. | Consumer asking for 1.9 gets a "no". | Document the policy in the Airbyte README; revisit when 2.0 GA ships. | | Bitnami's late-2025 registry change means the MariaDB / Redis subcharts rely on `bitnamilegacy` + `global.security.allowInsecureImages`. | If Bitnami deprecates `bitnamilegacy`, both subcharts break. | Monitor Bitnami's policy; plan a migration to a vendored or self-hosted registry; enterprise customers are expected to use their own internal registry. | diff --git a/docs/domain/bronze-to-api-e2e/specs/PRD.md b/docs/domain/bronze-to-api-e2e/specs/PRD.md index 24ad81538..77a832973 100644 --- a/docs/domain/bronze-to-api-e2e/specs/PRD.md +++ b/docs/domain/bronze-to-api-e2e/specs/PRD.md @@ -51,7 +51,7 @@ It exists so that a developer changing any layer of the pipeline (dbt model, mig ### 1.2 Background / Problem Statement -Today the only automated coverage of Insight's transformation pipeline is dbt's own generic tests (`unique`, `not_null`) plus a small set of hand-written assertion tests under `src/ingestion/dbt/tests/`. They catch dbt-level invariants on silver tables but they don't observe the gold views (created by ClickHouse migrations in `src/ingestion/scripts/migrations/`) and they don't observe the analytics-api response shape that the UI actually reads. A typical regression — a renamed column in a dbt model, a changed `argMax` in a migration view, a tightened OData filter parser in the Rust service — sneaks past CI and gets caught either by a developer in `dev-up.sh` or by a tenant in production. +Today the only automated coverage of Insight's transformation pipeline is dbt's own generic tests (`unique`, `not_null`) plus a small set of hand-written assertion tests under `src/ingestion/dbt/tests/`. They catch dbt-level invariants on silver tables but they don't observe the gold views (created by ClickHouse migrations in `src/ingestion/scripts/migrations/`) and they don't observe the analytics-api response shape that the UI actually reads. A typical regression — a renamed column in a dbt model, a changed `argMax` in a migration view, a tightened OData filter parser in the Rust service — sneaks past CI and gets caught either by a developer running the stack locally or by a tenant in production. The transformation chain has four authoring surfaces: @@ -162,7 +162,7 @@ The related "is the data correct?" question (whether a metric value is semantica - Requires Python ≥ 3.12 (matches `dbt` runtime in the repo) - Requires `cargo` toolchain to build the `analytics-api` binary once per session - Requires ClickHouse and MariaDB versions pinned to production parity — the framework MUST NOT silently downgrade to an older container image -- Tests cannot run inside K8s — the framework is local-host only; for K8s integration use `dev-up.sh` +- Tests cannot run inside K8s — the framework is local-host only; for K8s integration use the local gitops deploy (`cd deploy/gitops && make deploy ENV=local`) ## 4. Scope diff --git a/docs/domain/connector/specs/PRD.md b/docs/domain/connector/specs/PRD.md index cd7bbc060..60d00fd3d 100644 --- a/docs/domain/connector/specs/PRD.md +++ b/docs/domain/connector/specs/PRD.md @@ -419,7 +419,7 @@ The system MUST enforce distinct permission boundaries for the three authorship - [ ] `p1` - **ID**: `cpt-insightspec-fr-cn-rls` -ClickHouse ROW POLICYs **MUST** be applied to all Bronze and Silver tables to enforce tenant-level data isolation at the database level. RLS policies **MUST** filter on `tenant_id` column and restrict each tenant role to only their data. Policies **MUST** survive table recreation (DROP + CREATE) — they are managed separately via `apply-rls.sh` and a declarative RLS config. The RLS config **MUST** be applied automatically during `./dev-up.sh` initialization and **MUST** be re-applicable without data loss. +ClickHouse ROW POLICYs **MUST** be applied to all Bronze and Silver tables to enforce tenant-level data isolation at the database level. RLS policies **MUST** filter on `tenant_id` column and restrict each tenant role to only their data. Policies **MUST** survive table recreation (DROP + CREATE) — they are managed separately from table DDL via a declarative RLS config. The RLS config **MUST** be applied automatically as part of the Kubernetes ingestion initialization step (the local gitops deploy and the ingestion init) and **MUST** be re-applicable without data loss. **Rationale**: Application-level `WHERE tenant_id = ...` is insufficient as the sole isolation mechanism. Database-level RLS provides defense-in-depth and prevents accidental cross-tenant data exposure via ad-hoc queries or BI tools connecting directly to ClickHouse. diff --git a/docs/domain/identity-resolution/specs/DESIGN.md b/docs/domain/identity-resolution/specs/DESIGN.md index 2db2705cb..bd356f85a 100644 --- a/docs/domain/identity-resolution/specs/DESIGN.md +++ b/docs/domain/identity-resolution/specs/DESIGN.md @@ -903,7 +903,7 @@ See [ADR-0002](ADR/0002-stable-person-id-via-persons-observations.md) for the fu **Prerequisites and ordering** (end-to-end bootstrap): 1. Connector secrets applied (`./secrets/apply.sh`). -2. `./dev-up.sh` (or the canonical `deploy/scripts/install.sh`) — installs Airbyte + Argo Workflows + the Insight umbrella chart. The umbrella's `identity-db-init-job` Helm pre-install Job provisions the `identity` MariaDB database and grants. The identity-resolution pod then starts and applies its sea-orm migrations (including the `persons` table) at startup via `run_migrations(&db)` in `main.rs`. +2. Kubernetes deploy (`cd deploy/gitops && make deploy ENV=`) — installs Airbyte + Argo Workflows (L2 system) and the Insight umbrella chart (L3 app). The umbrella's `identity-db-init-job` Helm pre-install Job provisions the `identity` MariaDB database and grants. The identity-resolution pod then starts and applies its sea-orm migrations (including the `persons` table) at startup via `run_migrations(&db)` in `main.rs`. 3. `./src/ingestion/run-init.sh` — runs ClickHouse migrations, registers connectors, creates Airbyte connections, syncs Argo flows. 4. Airbyte sync produces Bronze data (`./sync-all.sh` + wait). 5. dbt models run to populate `identity.identity_inputs` (`dbt run --select +identity_inputs`). diff --git a/docs/domain/ingestion/README.md b/docs/domain/ingestion/README.md index 813d125d7..c42a35cf9 100644 --- a/docs/domain/ingestion/README.md +++ b/docs/domain/ingestion/README.md @@ -4,34 +4,38 @@ End-to-end data pipeline from external source APIs to unified Silver tables. Bui ## Quick Start +Ingestion needs Airbyte, Argo Workflows, and ClickHouse, which the Docker Compose +dev path does not provide. The local-dev path for ingestion is therefore the +Kubernetes (gitops) path, which installs Airbyte + Argo Workflows and the umbrella chart. + ```bash -cd src/ingestion +# 1. Bring up the full stack on a local Kind/OrbStack cluster +cd deploy/gitops && make deploy ENV=local -# 1. Copy and fill tenant credentials +# 2. Copy and fill tenant credentials +cd src/ingestion cp connections/example-tenant.yaml.example connections/my-tenant.yaml # Edit: fill in real API keys -# 2. Start everything -./dev-up.sh - # 3. Run a sync manually ./run-sync.sh m365 my-tenant ``` ## Prerequisites -- Docker Desktop -- `kubectl`, `helm`, `kind` (`brew install kubectl helm kind`) +- Docker Desktop (or OrbStack) +- `kubectl`, `helm`, `kind`, `make` (`brew install kubectl helm kind make`) ## Commands ### Lifecycle +The stack lifecycle is managed by the gitops deploy from `deploy/gitops` (see +[deploy/gitops SPEC](../../../deploy/gitops/README.md)). + | Command | What it does | |---------|-------------| -| `./dev-up.sh` | Start all services. Idempotent — safe to re-run | -| `./dev-down.sh` | Stop all services. Data preserved | -| `./cleanup.sh` | Delete cluster and all data. Asks for confirmation | +| `cd deploy/gitops && make deploy ENV=local` | Install/upgrade all services on the local cluster. Idempotent — safe to re-run | ### Operations @@ -46,7 +50,8 @@ cp connections/example-tenant.yaml.example connections/my-tenant.yaml ```bash # Full setup from scratch -./dev-up.sh +cd deploy/gitops && make deploy ENV=local +cd src/ingestion # Update M365 connector manifest after editing connector.yaml ./update-connectors.sh @@ -60,17 +65,21 @@ cp connections/example-tenant.yaml.example connections/my-tenant.yaml # Run M365 sync for example-tenant right now ./run-sync.sh m365 example_tenant -# Monitor in Argo UI -open http://localhost:30500 +# Monitor in Argo UI (port-forward, then open the printed URL) +kubectl -n insight port-forward svc/argo-server 2746:2746 ``` ## Services -| Service | URL | Credentials | -|---------|-----|-------------| -| Airbyte | http://localhost:8000 | Printed by `dev-up.sh` | -| Argo UI | http://localhost:30500 | No auth (local) | -| ClickHouse | http://localhost:30123 | user: `default`, password: `clickhouse` | +On the gitops path these are exposed via cluster ingress or reached with +`kubectl port-forward`. See the [deploy/gitops SPEC](../../../deploy/gitops/README.md) +for ingress hostnames and the services/ports it provisions. + +| Service | Access | Credentials | +|---------|--------|-------------| +| Airbyte | ingress or `kubectl -n insight port-forward svc/airbyte-webapp-svc 8000:80` | from gitops secrets | +| Argo UI | ingress or `kubectl -n insight port-forward svc/argo-server 2746:2746` | No auth (local) | +| ClickHouse | ingress or `kubectl -n insight port-forward svc/insight-clickhouse 8123:8123` | user: `default`, password: `clickhouse` | ## Configuration diff --git a/docs/domain/ingestion/specs/DECOMPOSITION.md b/docs/domain/ingestion/specs/DECOMPOSITION.md index 2fb57043d..78246e5f5 100644 --- a/docs/domain/ingestion/specs/DECOMPOSITION.md +++ b/docs/domain/ingestion/specs/DECOMPOSITION.md @@ -38,7 +38,7 @@ The Ingestion Layer DESIGN is decomposed into seven features organized around de - Per-tenant Argo CronWorkflows generated from connector `descriptor.yaml` - Kind K8s cluster for local development (same Helm charts as production) - `insight-toolbox` container runs all management scripts inside the cluster -- Auto-initialization on `./dev-up.sh` — no manual setup +- Auto-initialization on `cd deploy/gitops && make deploy ENV=local` — no manual setup ## 2. Entries @@ -50,7 +50,7 @@ The Ingestion Layer DESIGN is decomposed into seven features organized around de - [ ] `p1` - **ID**: `cpt-insightspec-feature-local-infra` -- **Purpose**: Provide a fully automated local development environment via Kind K8s cluster that mirrors production topology. Running `./dev-up.sh` creates a working instance with ClickHouse, Airbyte, Argo Workflows, and all initialization — no manual configuration required. +- **Purpose**: Provide a fully automated local development environment via Kind K8s cluster that mirrors production topology. Running `cd deploy/gitops && make deploy ENV=local` creates a working instance with ClickHouse, Airbyte, Argo Workflows, and all initialization — no manual configuration required. - **Depends On**: None diff --git a/docs/domain/ingestion/specs/DESIGN.md b/docs/domain/ingestion/specs/DESIGN.md index 56d9a2aca..f39a93d1f 100644 --- a/docs/domain/ingestion/specs/DESIGN.md +++ b/docs/domain/ingestion/specs/DESIGN.md @@ -668,20 +668,19 @@ Key deployment decisions: - **Single-namespace model**: Airbyte, Argo and the Insight umbrella share the release namespace; cross-release service references resolve via plain DNS (e.g. `airbyte-airbyte-server-svc.insight.svc.cluster.local:8001`). Lifecycle is still independent — each release upgrades on its own schedule. - Argo Workflows stores state in K8s etcd — no external database required. - Airbyte ships its own bundled PostgreSQL for connector metadata (managed by the `airbyte/airbyte` chart, not by the umbrella). -- Helm charts: Airbyte via `airbyte/airbyte`, Argo via `argo/argo-workflows`, the Insight platform via `charts/insight` in this repo. The canonical installer chain `deploy/scripts/install.sh` runs all three in order; `dev-up.sh` is the developer wrapper that builds local images and delegates to it. +- Helm charts: Airbyte via `airbyte/airbyte`, Argo via `argo/argo-workflows`, the Insight platform via `charts/insight` in this repo. On Cyberfabric-operated and local clusters all three are installed by the gitops Makefile (`cd deploy/gitops && make deploy ENV=` — Airbyte/Argo as L2 system releases, the umbrella as the L3 app release); external consumers install them with their own tooling. - ClickHouse, MariaDB, Redis and Redpanda are all umbrella subcharts under `charts/insight/Chart.yaml` (`.deploy: true|false`). The unified shape `.host / .port / .database / .username / .passwordSecret` works whether the dep is bundled or external — see `charts/insight/values.yaml`. - ClickHouse is a StatefulSet (`insight-clickhouse`) created by `helmfile/charts/clickhouse`; access is via Service `insight-clickhouse:8123` inside the cluster. Health probes use HTTP GET `/ping` (not `clickhouse-client` exec — avoids CLI flag parsing issues with auto-generated passwords). - MariaDB per-service databases are provisioned via the bundled bitnami `mariadb.initdbScriptsConfigMap` (see `charts/insight/templates/mariadb-initdb-scripts.yaml`) — bitnami runs every script in that ConfigMap on the FIRST MariaDB pod boot, mounted at `/docker-entrypoint-initdb.d`. The data lives in the PVC after that, so restarts and helm upgrades are no-ops. Each owning service then runs its own SeaORM migrations at startup — see §4.4.2 and [ADR-0006](ADR/0006-service-owned-migrations.md). - CDK connector build script (`airbyte-toolkit/build-connector.sh`) uses `CLUSTER_NAME` env var (default `insight`) for Kind image loading — not hardcoded. - Airbyte port-forward uses `nohup ... & disown` to avoid blocking the terminal. - Argo `dbt-run` WorkflowTemplate uses locally-built `insight-toolbox:local` image (with `imagePullPolicy: IfNotPresent`) — not `ghcr.io/constructorfabric/insight-toolbox:latest`. Local builds via `tools/toolbox/build.sh` pick up dbt model changes without requiring a registry push. Template also accepts `full_refresh` parameter (pass `--full-refresh` to recreate tables from scratch). -- CoreDNS is patched to use public DNS upstream (`8.8.8.8`, `8.8.4.4`) — WSL's `/etc/resolv.conf` points to an internal WSL nameserver that cannot reliably resolve external domains (e.g. `login.microsoftonline.com`). Patch lives in `scripts/dev/patch-coredns-wsl.sh` (idempotent, opt-out via `SKIP_COREDNS_PATCH=1`); invoked from `dev-up.sh` after Kind bootstrap and re-applied by `dev-restart.sh`. -- `dev-restart.sh` also cleans up stale Airbyte replication-job pods (from previous syncs that did not exit cleanly). +- CoreDNS is patched to use public DNS upstream (`8.8.8.8`, `8.8.4.4`) — WSL's `/etc/resolv.conf` points to an internal WSL nameserver that cannot reliably resolve external domains (e.g. `login.microsoftonline.com`). Patch lives in `scripts/dev/patch-coredns-wsl.sh` (idempotent, opt-out via `SKIP_COREDNS_PATCH=1`); Windows/WSL+Kind operators run it manually against their cluster after bootstrap. - Gold views migration (`20260422000000_gold-views.sql`) references bronze tables from optional connectors (jira, m365, zoom). When the corresponding bronze table does not yet exist (no real connector data ingested yet), `scripts/create-bronze-placeholders.sh` creates empty placeholder tables with a minimal compatible schema so the gold-views migration succeeds on a partial install. - **Placeholder handoff caveat**: Airbyte ClickHouse destination v2.0.8+ throws an error on the first sync if the target bronze table exists with a schema that does not match the destination's expected schema for that stream. The placeholder schemas in `create-bronze-placeholders.sh` are intentionally minimal (only the columns referenced by gold views) — they are **not** a drop-in replacement for a native Airbyte-generated table. Before enabling a previously-placeholdered connector, the operator should manually `DROP TABLE` the placeholder(s) in ClickHouse so Airbyte can create them fresh on its first sync. - - Script runs automatically via `init.sh` before migrations. + - Script runs automatically via `src/ingestion/run-init.sh` before migrations. - All credentials managed via Kubernetes Secrets (see §4.1.1). -- Service access via Ingress (PR #224): the umbrella chart configures `ingress-nginx` routes for Frontend, API Gateway, Airbyte UI and Argo UI; in `dev-up.sh --env local` the Kind cluster maps host ports 80/443 to the ingress controller. Direct port-forwards (`kubectl port-forward`) remain available for debugging. +- Service access via Ingress (PR #224): the umbrella chart configures `ingress-nginx` routes for Frontend, API Gateway, Airbyte UI and Argo UI; on a local Kubernetes cluster the Kind config maps host ports 80/443 to the ingress controller. Direct port-forwards (`kubectl port-forward`) remain available for debugging. #### 4.1.1 Infrastructure Secrets @@ -722,37 +721,40 @@ All Insight components share the release namespace (default `insight`); override - `full_refresh` streams → `overwrite` (each sync replaces all data — no duplicate accumulation) - `incremental` streams → `append_dedup` (appends new records, deduplicates by primary key) -### 4.2 Local Development (Kind K8s Cluster) +### 4.2 Local Development (local Kubernetes cluster) -All services run inside a single Kind K8s cluster (`insight`), in a single namespace, using the same three-Helm-release model as production: -- Airbyte installed via Helm chart (`airbyte/airbyte`) -- Argo Workflows installed via Helm chart (`argo/argo-workflows`) -- Insight platform installed via the umbrella chart `charts/insight` — bundles ClickHouse, MariaDB, Redis, Redpanda, Analytics API, Identity Resolution, API Gateway and Frontend as subcharts. Image tags for the Rust services are filled by `dev-up.sh` from locally-built images (`kind load docker-image`). +Ingestion work needs Airbyte and Argo Workflows, which the docker-compose dev +stack does not ship — so the ingestion inner loop runs against a local +Kubernetes cluster (Kind / OrbStack / k3d / minikube) brought up by the gitops +path. (For backend / frontend work that does not touch ingestion, the +docker-compose stack — `./dev-compose.sh up` — is the lighter default; see +[CONTRIBUTING.md](../../../../CONTRIBUTING.md).) + +The gitops Makefile with `ENV=local` brings up the whole stack: +- Airbyte installed via Helm chart (`airbyte/airbyte`) as a separate Helm release +- Argo Workflows installed via Helm chart (`argo/argo-workflows`) as a separate Helm release +- Insight platform installed via the umbrella chart `charts/insight` — the app services (Analytics API, Identity Resolution, API Gateway, Frontend) plus, depending on the `.deploy` toggles, the bundled ClickHouse / MariaDB / Redis / Redpanda subcharts or wiring to externally-provided instances. - dbt runs as Argo container steps (`insight-toolbox`) -KUBECONFIG: `~/.kube/insight.kubeconfig` +`ENV=local` runs the wizard once and then the `bootstrap → system → deploy-app` +chain. The cluster topology, layer model and namespace conventions are owned by +the [gitops SPEC](../../components/deployment/gitops/README.md). #### Startup scripts | Script | Purpose | |--------|---------| -| `./dev-up.sh` | Dev wrapper: builds Docker images from `src/`, creates a local Kind cluster (or targets a dev-owned remote cluster), loads images into the cluster, and delegates to `deploy/scripts/install*.sh` to install Airbyte, Argo Workflows, and the Insight umbrella chart. Uses `.env.` for configuration. Idempotent. | -| `./dev-down.sh` | Graceful stop: scales all `insight`-namespace deployments to 0, stops the Kind container. Data preserved — `./dev-restart.sh` brings everything back. | -| `./dev-restart.sh` | Quick restart after WSL/Docker crash or `./dev-down.sh`: restarts the Kind container, scales `insight`-namespace pods back to 1, re-patches CoreDNS, restores port-forwards. Falls back to `./dev-up.sh` if the cluster is gone. Lightweight — no image builds, no helm upgrade. | -| `deploy/scripts/install.sh` | Production-style installer (canonical path): chains `install-airbyte.sh` → `install-argo.sh` → `install-insight.sh` against the current kubeconfig. Used by `dev-up.sh` and end-user installs from published chart artifacts. | +| `cd deploy/gitops && make deploy ENV=local` | Brings up the full local stack: bootstrap (L0), Airbyte + Argo Workflows + the L2 system services, then the Insight umbrella chart (L3). Idempotent — re-running reconciles. Honours `$KUBECONFIG`. | | `src/ingestion/run-init.sh` | Post-deploy init: verifies secrets, runs ClickHouse migrations, registers connectors, applies Airbyte connections, syncs Argo flows. (MariaDB schema is applied per-service by each backend service's own sea-orm `Migrator` at startup — see §4.4 and [ADR-0006](ADR/0006-service-owned-migrations.md). Per-service databases beyond the umbrella default are provisioned by `charts/insight/templates/mariadb-initdb-scripts.yaml` on the first MariaDB pod boot.) | | `src/ingestion/sync-all.sh` | Trigger Airbyte sync for all connections. Reads connection IDs from state, calls Airbyte API. Use after `run-init.sh` to start first data load, or anytime to re-sync all sources. | **First-time setup**: -1. Copy `.env.local.example` → `.env.local` -2. Copy connector secret examples → fill credentials (`src/ingestion/secrets/connectors/`) -3. `./dev-up.sh` — full stack deployment -4. `./src/ingestion/run-init.sh` — databases, connectors, connections -5. `cd src/ingestion && ./sync-all.sh` — trigger first Airbyte sync for all connections - -**After WSL/Docker crash**: `./dev-restart.sh` — one command, restores full state. +1. Copy connector secret examples → fill credentials (`src/ingestion/secrets/connectors/`) +2. `cd deploy/gitops && make deploy ENV=local` — full stack deployment (answer the wizard prompts on first run) +3. `./src/ingestion/run-init.sh` — databases, connectors, connections +4. `cd src/ingestion && ./sync-all.sh` — trigger first Airbyte sync for all connections -**Environment support**: `./dev-up.sh --env ` loads `.env.` config. Supports `local` (Kind) and remote (external kubeconfig) modes. +**Re-running**: `make deploy ENV=local` is idempotent — re-running on a converged cluster reconciles the stack in place. This enables: - Testing connector registration and sync execution