constructorfabric · cyberantonz · Jun 22, 2026 · Jun 22, 2026 · Jun 22, 2026
@@ -22,7 +22,7 @@ This repository is the **monorepo** for the Insight product. It contains:
 - [Connector Coverage](#connector-coverage)
 - [Key Concepts](#key-concepts)
 - [Quick Start](#quick-start)
-  - [Local development (`dev-up.sh`)](#local-development-dev-upsh)
+  - [Local development (Docker Compose)](#local-development-docker-compose)
   - [Cluster deployment](#cluster-deployment)
   - [Configure connectors](#configure-connectors)
   - [Services and ports](#services-and-ports)
@@ -114,13 +114,12 @@ The solution consists of five main components:
 ### Root scripts
 
 ```
-./dev-up.sh               ← Create Kind cluster + deploy all services
-./init.sh             ← Apply secrets + initialize ingestion stack
-./dev-down.sh             ← Stop services (data preserved)
-./cleanup.sh          ← Delete cluster and all data
-k8s/kind-config.yaml  ← Kind cluster configuration
+./dev-compose.sh      ← Docker Compose dev stack (default laptop path)
+deploy/gitops/        ← Kubernetes path: `make deploy ENV=<env>`
 ```
 
+See [CONTRIBUTING.md](CONTRIBUTING.md) for both paths in full.
+
 ### `src/`
 
 Source code for all platform components.
@@ -226,43 +225,26 @@ This repo uses [Cypilot](https://github.com/cyberfabric/cyber-pilot) — an AI a
 
 ## Quick Start
 
-Two ways to bring up Insight:
+Two supported paths:
 
-- **Local development** — [`./dev-up.sh`](./dev-up.sh) builds a local Kind cluster from source and runs the full stack with hot reload.
-- **Cluster deployment** — Cyberfabric engineers use the private `infra/insight-gitops` repository (Makefile-driven, OCI-pinned chart). External consumers of the umbrella Helm chart use it directly via `helm`, ArgoCD, Flux, or whatever orchestrator they already have; the chart contract lives at [`charts/insight/README.md`](charts/insight/README.md).
+- **Local development (Docker Compose)** — [`./dev-compose.sh up`](./dev-compose.sh) runs the full stack on a developer laptop with only Docker. Default for day-to-day backend / frontend work.
+- **Cluster deployment** — Cyberfabric engineers use the private `infra/insight-gitops` repository (Makefile-driven, OCI-pinned chart); the same path runs locally on a Kind/OrbStack cluster via `cd deploy/gitops && make deploy ENV=local` when you need Airbyte / Argo Workflows or the real cluster shape. External consumers of the umbrella Helm chart use it directly via `helm`, ArgoCD, Flux, or whatever orchestrator they already have; the chart contract lives at [`charts/insight/README.md`](charts/insight/README.md).
 
-Common prerequisites: `kubectl` ≥ 1.27, `helm` ≥ 3.13, `kubeconfig` for the target cluster. Local development also needs `kind` and `docker`.
+The two paths share a single first-run wizard, so the MariaDB / ClickHouse / tenant / dev-email answers are identical across them. The full guide for both is [CONTRIBUTING.md](CONTRIBUTING.md).
 
-### Local development (`dev-up.sh`)
+### Local development (Docker Compose)
 
-For laptop development. Builds all backend + toolbox images from `src/`, loads them into Kind, deploys the stack with single-replica defaults, and opens stable port-forwards.
+For laptop development. No Rust / .NET / Node on the host — every build runs in a builder container; the only prerequisite is Docker (Engine 24+, compose v2).
 
 ```bash
-brew install kind kubectl helm docker
-
-cp .env.local.example .env.local
-$EDITOR .env.local         # AUTH_DISABLED=true OR fill OIDC_*
-
-./dev-up.sh                # creates Kind, builds images, installs everything
+git clone https://github.com/constructorfabric/insight.git
+cd insight
+./dev-compose.sh up        # first-run wizard, then builds + seeds the stack
 ```
 
-What runs under the hood:
-- Creates Kind cluster `insight` if missing.
-- Installs `ingress-nginx` (skipped when present).
-- Builds backend images (`api-gateway`, `analytics-api`, `identity`, `toolbox`) from local source and `kind load`s them — no registry needed.
-- Generates a temporary Helm overlay from `.env.local` and runs `install-airbyte.sh` → `install-argo.sh` → `install-insight.sh` against the Kind cluster (single namespace `insight`).
-- Starts port-forwards for every service on stable local ports (see [Services and ports](#services-and-ports)).
+The wizard prompts for local-vs-external MariaDB / ClickHouse, a dev-user email, and the frontend mode (defaults pull the published `insight-front` image). First `up` auto-seeds a demo dataset; open <http://localhost:3000>.
 
-Lifecycle commands (run from repo root):
-
-| Command | Description |
-|---|---|
-| `./dev-up.sh` | Full bring-up (idempotent on re-run) |
-| `./dev-up.sh ingestion` | Only Airbyte/Argo/ClickHouse |
-| `./dev-up.sh app` | Only application services |
-| `./dev-up.sh frontend` | Only frontend (skip backend rebuild) |
-| `./dev-down.sh` | Stop port-forwards (Kind cluster + data preserved) |
-| `./cleanup.sh` | Delete Kind cluster and all volumes |
+The compose stack does **not** ship Airbyte or Argo Workflows — for ingestion work use the Kubernetes path below. See [CONTRIBUTING.md](CONTRIBUTING.md) for the edit-build loop, frontend modes, seeding, and the `.env.compose` settings reference.
 
 ### Cluster deployment
 
@@ -315,35 +297,25 @@ kubectl -n insight get workflows -l tenant=default --watch
 
 ### Services and ports
 
-For local development (`dev-up.sh`) all services have stable local port-forwards:
+For local Docker Compose development every web service publishes a host port (override any `*_PORT` in `.env.compose` on conflict):
 
 | Service | URL | Notes |
 |---|---|---|
-| Frontend | http://localhost:8003 | SPA |
-| API Gateway | http://localhost:8080 | `/api/v1`; auth disabled when `AUTH_DISABLED=true` |
-| Airbyte UI/API | http://localhost:8001 | UI and API on the same port (Airbyte ≥ 1.5) |
-| Argo Workflows UI | http://localhost:2746 | `--auth-mode=server` (no Bearer) when `DEV_MODE=1` |
+| Frontend | http://localhost:3000 | SPA |
+| API Gateway | http://localhost:8080 | `/api/v1`; auth disabled in the `no-auth` config |
+| Analytics API | http://localhost:8081 | |
+| Identity | http://localhost:8082 | .NET 9 |
 | ClickHouse HTTP | http://localhost:8123 | `/play` for browser SQL |
 | MariaDB | localhost:3306 | |
 | Redis | localhost:6379 | |
 
-For cluster deployments services are reached via the configured ingress hostname (or set up port-forwards manually).
+The compose stack does not run Airbyte or Argo Workflows — those live on the Kubernetes path. For cluster deployments services are reached via the configured ingress hostname (or `kubectl port-forward`).
 
 ### Image configuration
 
 The chart fails fast if any image tag is empty — there are **no `:latest` defaults** anywhere.
 
-For local development (`dev-up.sh`) tags come from `.env.<name>`:
-
-| `.env` variable | Default | Description |
-|---|---|---|
-| `IMAGE_REGISTRY` | _(empty)_ | Registry prefix (e.g. `ghcr.io/constructorfabric`); empty for local-only Kind builds |
-| `IMAGE_TAG` | `local` | Tag applied to all locally-built images |
-| `<SVC>_IMAGE_TAG` | `$IMAGE_TAG` | Per-service override (`API_GATEWAY_IMAGE_TAG`, …) |
-| `IMAGE_PULL_POLICY` | `IfNotPresent` | Kubernetes pullPolicy |
-| `BUILD_IMAGES` | `true` | Build images locally before deploy |
-| `BUILD_AND_PUSH` | `false` | Push built images to `$IMAGE_REGISTRY` |
-| `IMAGE_PLATFORM` | _(empty)_ | Cross-build platform, e.g. `linux/amd64` |
+For local Docker Compose development each backend service is built locally by default; skip the build for any service and pull its published image instead by setting `<SVC>_IMAGE` in `.env.compose` (e.g. `API_GATEWAY_IMAGE=ghcr.io/constructorfabric/insight-api-gateway:latest`) or with `./dev-compose.sh up --from-ghcr=<svc>`. See [CONTRIBUTING.md](CONTRIBUTING.md) for the build targets and frontend modes.
 
 For cluster deployments image tags flow through automatically: the umbrella chart's CI bumps each subchart's `appVersion` on every merge to `main`, and the subchart templates default `image.tag` to `.Chart.AppVersion`. Env overlays only need to pin a tag explicitly for a hotfix scenario (testing one service at a different tag than the one bundled in the umbrella version). Image source repos:
 

@@ -1,8 +1,8 @@
 # Insight platform — local docker-compose dev stack.
 #
-# This is the docker-compose path for working on Insight on a developer
-# laptop. The Kind/helmfile path (dev-up.sh) is being retired in favour
-# of this; see CONTRIBUTING.md.
+# This is the default path for working on Insight on a developer laptop.
+# For the Kubernetes path (Airbyte / Argo Workflows, real cluster shape)
+# see CONTRIBUTING.md and deploy/gitops/.
 #
 # Quick start:
 #   cp .env.compose.example .env.compose

@@ -15,7 +15,8 @@ and the (forthcoming) reconciliation service.
 
 | Path | Command |
 |---|---|
-| Dev (local kind) | `./dev-up.sh --env local backend` — builds `insight-identity:local`, loads into kind, installs umbrella with `identity.deploy=true`. |
+| Dev (Docker Compose, default) | `./dev-compose.sh up` runs the identity service in a container alongside MariaDB etc. Build the service image with `./dev-compose.sh build identity`. No Kind, no umbrella chart. |
+| Dev (Kubernetes via gitops) | `cd deploy/gitops && make deploy ENV=local` on a local Kind/OrbStack cluster installs the umbrella chart, which includes identity-resolution when `identity.deploy=true`. |
 | Production / staging | Standard umbrella install. Override `identity.deploy=true` and `identity.image.tag=<release>` in your values overlay. |
 | Standalone (no umbrella) | `helm install identity ./src/backend/services/identity/helm` with a pre-created `insight-identity-config` Secret. |
 

@@ -154,24 +154,24 @@ For one cluster carrying environment `<env>`:
 | `insight-infra` | L2 | mariadb, clickhouse, redis, redpanda, redpanda-console, airbyte, argo-workflows (each as its own Helm release) |
 | `insight` | L3 | the umbrella chart (api-gateway, analytics-api, identity-resolution, frontend) |
 
-Each cluster hosts exactly one Insight install. The cluster's identity (which env it represents) lives in the kube-context name (`insight-<env>`) and the gitops repo's `environments/<env>/` directory — not in the namespace. Operationally this keeps the two well-known namespace names (`insight`, `insight-infra`) the same across every install, matching the `dev-up.sh` local-Kind convention and any external chart consumer's expectation of a single `insight` release.
+Each cluster hosts exactly one Insight install. The cluster's identity (which env it represents) lives in the kube-context name (`insight-<env>`) and the gitops repo's `environments/<env>/` directory — not in the namespace. Operationally this keeps the two well-known namespace names (`insight`, `insight-infra`) the same across every install — including the local cluster (`ENV=local`) — and matches any external chart consumer's expectation of a single `insight` release.
 
 #### Dual-purpose umbrella: `<service>.deploy` toggles
 
-The umbrella chart in `constructorfabric/insight` keeps its infrastructure subcharts (`clickhouse`, `mariadb`, `redis`, `redpanda`) **gated by per-service `<service>.deploy: true|false` flags**. The flag is the dev-vs-prod switch:
+The umbrella chart in `constructorfabric/insight` keeps its infrastructure subcharts (`clickhouse`, `mariadb`, `redis`, `redpanda`) **gated by per-service `<service>.deploy: true|false` flags**. The flag selects the install shape:
 
 | Caller | `<svc>.deploy` | Result |
 |--------|----------------|--------|
-| `dev-up.sh` (Kind / OrbStack local) | `true` for all infra subcharts | Single fat Helm release in the `insight` namespace; the umbrella renders MariaDB, ClickHouse, Redis, Redpanda **and** the app services together. Convenient for one-command local bring-up. |
-| Gitops production (any cluster managed by this repo) | `false` for every infra subchart | Umbrella renders the app services only, into the `insight` namespace. L2 services come from one of: (a) `make system-<service>` Helm releases in `insight-infra` per [§3.5](#35-engineer-provisions-the-system-layer-l2); (b) managed external endpoints (RDS, MSK, …); (c) a separate team's infra namespace. App values point at the actual host. |
+| External consumer wanting a single-namespace install | `true` for all infra subcharts | Single fat Helm release in the `insight` namespace; the umbrella renders MariaDB, ClickHouse, Redis, Redpanda **and** the app services together. Self-contained, no separate L2. |
+| Gitops (any cluster managed by this repo — production and local `ENV=local`) | `false` for every infra subchart | Umbrella renders the app services only, into the `insight` namespace. L2 services come from one of: (a) `make system-<service>` Helm releases in `insight-infra` per [§3.5](#35-engineer-provisions-the-system-layer-l2); (b) managed external endpoints (RDS, MSK, …); (c) a separate team's infra namespace. App values point at the actual host. |
 
 Why dual-purpose instead of two charts:
 
-- **One chart shape** — `dev-up.sh` exercises the same templates that production renders. Bugs in app rendering are caught locally.
+- **One chart shape** — the local gitops cluster (`make deploy ENV=local`) exercises the same templates and layered shape that production renders. Bugs in app rendering are caught locally.
 - **Customers can choose** — an external chart consumer who is fine running everything in one namespace can flip the toggles `true` and get a self-contained install. A consumer with managed infra flips them `false`.
 - **No flag conflicts** — the existing `<service>.deploy: false` path already wires the app to look up `<service>.host` / `<service>.port` from values, so cross-namespace DNS (`<release>.insight-infra.svc.cluster.local`) is just one well-formed hostname away.
 
-Airbyte and Argo Workflows are **not** subcharts of the umbrella in either mode — they are always installed as separate Helm releases (see `deploy/scripts/install-{airbyte,argo}.sh` for dev, `make system-{airbyte,argo}` for production). The same release goes to `insight` (dev) or `insight-infra` (prod) depending on the caller; no chart change required.
+Airbyte and Argo Workflows are **not** subcharts of the umbrella in either mode — on every gitops cluster (production and local) they are installed as separate Helm releases via `make system-{airbyte,argo}` into `insight-infra`; no chart change required.
 
 ## 2. Tagging & Versioning
 
@@ -857,7 +857,7 @@ These are accepted gaps that do not block the MVP but must be tracked.
 - **Audit log of deploys.** `make deploy` writes a local log file; there is no central audit. A trivial follow-up posts the log to a `#deploys` Slack channel via the poller's bot token; deferred until the team needs it.
 - **Rollback-by-tag.** `make rollback` calls `helm rollback` to the previous revision. Rolling back to an arbitrary historical state is `git checkout <deploy-tag> && make deploy`, which works but has not been rehearsed.
 - **Cross-namespace defaults in the umbrella.** The umbrella keeps its infra subcharts gated by `<service>.deploy: true|false` (see [§1.5 dual-purpose umbrella](#15-layer-model)). For the gitops production case (`.deploy: false`), the app's connection helpers must default the host to `<release>.insight-infra.svc.cluster.local` when no explicit `<service>.host` is supplied — so a values file that only says `<service>.deploy: false` "just works" against `insight-infra`. Verify the helpers do this; if not, a small chart-template change is needed. Also document the dual-purpose intent in `charts/insight/README.md` so external chart consumers understand the toggle.
-- **dev-up.sh Airbyte/Argo namespace.** `dev-up.sh` installs Airbyte and Argo Workflows into the same namespace as the umbrella (`insight` for local). Production gitops puts them in `insight-infra`. The chart values surface for both is identical (Airbyte API URL, Argo SA name) — confirm by render. If anything still hard-codes the `insight` namespace in templates, parameterise it.
+- **Airbyte/Argo namespace assumptions.** On every gitops cluster (production and local `ENV=local`) Airbyte and Argo Workflows run as L2 releases in `insight-infra`, while the umbrella runs in `insight`. The chart's helpers must therefore resolve the Airbyte API URL and Argo SA name across namespaces — confirm by render. If anything still hard-codes the `insight` namespace for these in templates, parameterise it.
 - **L2 chart-pin policy.** System service chart versions (`MARIADB_VERSION`, `CLICKHOUSE_VERSION`, etc.) are Makefile constants today; bumping is a deliberate PR. A future enhancement: split each service's pin into its own `system/<service>/.version` file (mirroring `.insight-version`) so a poller could pre-flight version compatibility against published Bitnami / Redpanda / Airbyte releases. Out of scope for v0.
 - **Per-cluster L2 inventory.** When a cluster swaps a self-hosted service for a managed endpoint (e.g. a managed database instead of `system/mariadb`), the gitops repo currently has no machine-readable record of "this cluster runs MariaDB on-cluster vs. external." A small `environments/<env>/inventory.yaml` listing which `system-*` targets to run on this cluster would make `make doctor` able to validate that the cluster matches the expected inventory, and would make it possible to render a per-customer install runbook from the repo. Captured.