Skip to content
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
16 changes: 2 additions & 14 deletions charts/insight/Chart.lock
Original file line number Diff line number Diff line change
@@ -1,16 +1,4 @@
dependencies:
- name: clickhouse
repository: file://../../helmfile/charts/clickhouse
version: 0.1.0
- name: mariadb
repository: oci://registry-1.docker.io/bitnamicharts
version: 20.0.0
- name: redis
repository: oci://registry-1.docker.io/bitnamicharts
version: 21.0.3
- name: redpanda
repository: https://charts.redpanda.com
version: 5.0.10
- name: insight-api-gateway
repository: file://../../src/backend/services/api-gateway/helm
version: 0.1.0
Expand All @@ -23,5 +11,5 @@ dependencies:
- name: insight-frontend
repository: file://../../src/frontend/helm
version: 0.1.0
digest: sha256:c459b435deac85d6312a9ac5ad4022c3e4f2c9d62d972b237a24b8b7fa915b69
generated: "2026-05-12T13:07:43.7606569+02:00"
digest: sha256:7556fa96a2d4a77c01f410c944290f88f8bf0d701a7ffe55dbcc7da1cfe58c38
generated: "2026-06-22T13:27:58.875272+02:00"
42 changes: 12 additions & 30 deletions charts/insight/Chart.yaml
Original file line number Diff line number Diff line change
Expand Up @@ -5,10 +5,12 @@
## Emits almost nothing on its own — it orchestrates subcharts.
##
## What is INCLUDED here:
## - Infrastructure: ClickHouse, MariaDB, Redis, Redpanda
## - Application: api-gateway, analytics-api, identity, frontend
## - Application: api-gateway, analytics-api, identity, frontend
##
## What is NOT included (deployed separately):
## - L2 infra (ClickHouse, MariaDB, Redis, Redpanda) — deployed as
## separate releases in `insight-infra` via gitops `make system-*`.
## The umbrella only dials them through `<dep>.host` wiring.
## - Airbyte — see scripts/install-airbyte.sh
## - Argo Workflows — see scripts/install-argo.sh
##
Expand All @@ -22,7 +24,7 @@ type: application
# version — packaging version of the chart. Bump on EVERY chart change.
# appVersion — product version. Matches the image tag of all insight-* images.
# Release pipeline: git tag v0.1.0 → version=0.1.0, appVersion="0.1.0".
version: 0.1.84
version: 0.2.0
appVersion: "2026.06.19.10.39-e006ef2"
kubeVersion: ">=1.27.0-0"
home: https://github.com/constructorfabric/insight
Expand All @@ -33,10 +35,15 @@ maintainers:
url: https://github.com/constructorfabric
# Dependencies — the heart of the umbrella.
#
# This chart bundles ONLY the first-party application services. The L2
# infrastructure (ClickHouse, MariaDB, Redis, Redpanda) is deployed
# separately in `insight-infra` via gitops `make system-*`; the umbrella
# dials those out-of-chart hosts through `<dep>.host` wiring in values.yaml.
#
# Fields:
# name — dependency chart name (must match the dependency's Chart.yaml name)
# version — strict pin for first-party charts, ~X.Y for bitnami/redpanda
# repository — file:// for local charts, oci:// / https:// for remote
# version — strict pin for first-party charts
# repository — file:// for local app-service charts
# condition — toggle in values (path to a boolean field)
# alias — short key name in values.yaml (otherwise the full chart name is needed)
#
Expand All @@ -45,31 +52,6 @@ maintainers:
# 2. add a matching block in values.yaml under the same key (or alias)
#
dependencies:
# ─── Infrastructure ─────────────────────────────────────────────────────

# ClickHouse — OLAP store. Local wrapper chart so we keep control over
# keeper/shards/macros configuration.
- name: clickhouse
version: "0.1.0"
repository: "file://../../helmfile/charts/clickhouse"
condition: clickhouse.deploy
# MariaDB — operational DB (tenants, plugin installs, configs).
# Bitnami chart, OCI registry. Semver ~20.0.0 = 20.x security patches.
- name: mariadb
version: "~20.0.0"
repository: "oci://registry-1.docker.io/bitnamicharts"
condition: mariadb.deploy
# Redis — query response cache for analytics-api.
- name: redis
version: "~21.0.0"
repository: "oci://registry-1.docker.io/bitnamicharts"
condition: redis.deploy
# Redpanda — event bus (notifications, audit log, future).
# Kafka-compatible, simpler to operate.
- name: redpanda
version: "~5.0.0"
repository: "https://charts.redpanda.com"
condition: redpanda.deploy
# ─── Application services ──────────────────────────────────────────────
# All aliases are camelCase so values.yaml stays clean.
# Service names inside the cluster stay kebab-case: `insight-api-gateway`.
Expand Down
93 changes: 20 additions & 73 deletions charts/insight/README.md
Original file line number Diff line number Diff line change
Expand Up @@ -8,12 +8,11 @@ Single canonical unit of delivery for the Insight platform.

## What it contains

The umbrella bundles ONLY the first-party application services. Each is a
local `file://` subchart:

| Component | Kind | Source | Toggle |
|-----------------------|----------------------|----------------------------------------------|---------------------------------|
| ClickHouse | infra | `helmfile/charts/clickhouse` (local wrapper) | `clickhouse.deploy` |
| MariaDB | infra | bitnami/mariadb ~20 | `mariadb.deploy` |
| Redis | infra | bitnami/redis ~21 | `redis.deploy` |
| Redpanda | infra | redpanda/redpanda ~5 | `redpanda.deploy` |
| API Gateway | app service (req'd) | `src/backend/services/api-gateway/helm` | mandatory (no flag) |
| Analytics API | app service (req'd) | `src/backend/services/analytics-api/helm` | mandatory (no flag) |
| Frontend (SPA) | app service (req'd) | `src/frontend/helm` | mandatory (no flag) |
Expand All @@ -23,19 +22,20 @@ Single canonical unit of delivery for the Insight platform.

## What it does NOT contain

| Component | Why separate | How to install |
|------------------|-------------------------------------------------------|--------------------------------|
| Airbyte | Heavy (10+ pods), its own release cadence | Separate helm release |
| Argo Workflows | Cluster-scoped infra, often shared across products | Separate helm release |
| Plugins | Runtime-managed via UI (not Helm — see architecture) | Through platform API |
| Component | Why separate | How to install |
|------------------------------------|-------------------------------------------------------|---------------------------------------------|
| ClickHouse / MariaDB / Redis / Redpanda (L2 infra) | Operated independently; shared lifecycle / managed services | Separate releases in `insight-infra` (gitops `make system-*`); the umbrella dials them via `<dep>.host` |
| Airbyte | Heavy (10+ pods), its own release cadence | Separate helm release |
| Argo Workflows | Cluster-scoped infra, often shared across products | Separate helm release |
| Plugins | Runtime-managed via UI (not Helm — see architecture) | Through platform API |

See [`docs/distribution/README.md`](../../docs/distribution/README.md) for the full distribution model.

## Release name convention

**This chart assumes release name = `insight`.**

Internal DNS references (e.g. `http://insight-analytics-api:8081`, `http://insight-clickhouse:8123`) are hardcoded in `values.yaml` with the `insight-` prefix. Helm subcharts use `{{ .Release.Name }}-{chart-suffix}` for service naming, which produces these exact names when the release is `insight`.
Internal DNS references between app services (e.g. `http://insight-analytics-api:8081`, `http://insight-identity:8082`) are templated with the `insight-` prefix. Helm subcharts use `{{ .Release.Name }}-{chart-suffix}` for service naming, which produces these exact names when the release is `insight`. (External L2 infra is reached via the explicit `<dep>.host` wiring, not the release-name convention.)

If you install under a different name, override all cross-service URLs in your own values.yaml. Prefer sticking to the convention.

Expand Down Expand Up @@ -67,26 +67,19 @@ Before going to prod:
- [ ] Set OIDC via `apiGateway.oidc.existingSecret` (preferred) or all three of `issuer` + `clientId` + `redirectUri` together. Never inline secrets.
- [ ] Enable ingress + TLS: `apiGateway.ingress`, `frontend.ingress`
- [ ] Bump resources where needed (default `requests` are conservative)
- [ ] `redpanda.tls.enabled: true`, `redpanda.auth.sasl.enabled: true`
- [ ] Point MariaDB/ClickHouse/Redis/Redpanda to external managed services if running inside Constructor Platform — set `<dep>.deploy: false` and fill `<dep>.host` / `.port` / `.passwordSecret`. App-service URLs follow automatically (resolved by helpers).
- [ ] Provision the L2 infra (ClickHouse / MariaDB / Redis / Redpanda) out-of-chart and fill `<dep>.host` / `.port` / `.passwordSecret`. App-service URLs follow automatically (resolved by helpers).
- [ ] Set `global.imagePullSecrets` if pulling from a private registry

## Integration modes

The chart uses ONE unified shape per infra dependency (ClickHouse, MariaDB, Redis, Redpanda). The `deploy` flag toggles whether the umbrella runs the subchart; everything else (host, port, credentials) is the same data the consumers read in either case.
## Infra wiring

**Standalone** (eval, on-prem single-tenant, dev):
- `<dep>.deploy: true` — the umbrella runs the subchart.
- `<dep>.host: ""` — defaults to `{release}-<dep>` (internal in-cluster service).
- `<dep>.passwordSecret` points at `insight-db-creds`, which the umbrella auto-generates on first install (or you pre-create for BYO).
L2 infra (ClickHouse, MariaDB, Redis, Redpanda) is always **external** — deployed out-of-chart as separate releases in `insight-infra` (gitops `make system-*`), or pointed at managed services. The umbrella only carries the wiring it needs to dial them:

**Constructor Platform component** (Insight ships inside the platform):
- `<dep>.deploy: false` — the umbrella does NOT run the subchart.
- `<dep>.host` is required (validator fails fast otherwise).
- `<dep>.passwordSecret` points at a Secret the platform created in the namespace.
- App-service URLs are computed by helpers from the same `<dep>.host` / `.port`, so no extra overrides are needed.
- `<dep>.host` is required (the validator / helpers fail fast otherwise). Redpanda uses `<dep>.brokers`.
- `<dep>.port` / `.database` / `.username` as applicable.
- `<dep>.passwordSecret` points at a Secret in the namespace (e.g. `insight-db-creds`) — auto-generated by the umbrella, mirrored by a platform operator, or pre-created (BYO).
- App-service URLs are computed by helpers from `<dep>.host` / `.port`, so no extra overrides are needed.

The umbrella validator (`templates/_helpers.tpl` → `insight.validate`) fails fast on the typical typos: `deploy: false` without `host`, OIDC enabled without `existingSecret` or all inline fields, missing `passwordSecret.{name,key}`.
The umbrella validator (`templates/_helpers.tpl` → `insight.validate`) fails fast on the typical typos: missing `<dep>.host` / `.brokers`, OIDC enabled without `existingSecret` or all inline fields, missing `passwordSecret.{name,key}`.

## Values reference

Expand All @@ -95,39 +88,14 @@ See comments in [`values.yaml`](./values.yaml) — every block is documented inl
Key groups:

- `credentials.autoGenerate` — toggle umbrella-managed `insight-db-creds`
- `global.*` — cluster-wide defaults (pull secrets, storage class, bitnami image policy)
- `<dep>.deploy` / `<dep>.host` / `<dep>.port` / `<dep>.passwordSecret`unified shape for ClickHouse, MariaDB, Redis, Redpanda
- `global.*` — cluster-wide defaults (pull secrets, storage class)
- `<dep>.host` / `<dep>.port` / `<dep>.passwordSecret` (Redpanda: `<dep>.brokers`)external-infra wiring for ClickHouse, MariaDB, Redis, Redpanda
- `apiGateway` / `analyticsApi` / `frontend` — **mandatory** app services (no deploy-flag; the gateway is the single entrance and the product is one unit)
- `identity.deploy` — **optional** .NET identity service (off by default; not an OIDC provider)
- `apiGateway.oidc` — OIDC configuration (prefer `existingSecret`; inline requires `issuer` + `clientId` + `redirectUri` together)
- `apiGateway.proxy.routes` — reverse-proxy config to downstream services
- `ingestion.templates.enabled` — whether to ship Argo WorkflowTemplates; requires Argo CRDs to be present in the cluster

## Bitnami Legacy images — maintenance model

In late 2025 Bitnami removed free image distribution from `docker.io/bitnami/*` ([bitnami/charts#30850](https://github.com/bitnami/charts/issues/30850)) and moved unsupported tags to a `docker.io/bitnamilegacy/*` namespace. The umbrella points the bundled MariaDB and Redis subcharts at `bitnamilegacy` so the eval / on-prem path keeps working without a paid Bitnami subscription. This is documented inline in `values.yaml` (`mariadb.image.repository`, `redis.image.repository`).

**Ownership and cadence.** Insight maintainers own the upgrade cadence for these images. The chart `~20.0.0` / `~21.0.0` constraints allow patch-level (CVE) bumps, but **minor releases require an explicit chart edit** — minors can carry breaking changes that need verification.

- **CVE-driven bumps**: tracked via Renovate against `bitnamilegacy/mariadb` and `bitnamilegacy/redis` tags; a critical CVE in either image opens a PR within 24h.
- **Routine bumps**: scheduled monthly review of the `~MAJOR.0.0` constraint window. Minor-version bumps (e.g. `mariadb 20.x → 21.x`) ship in a dedicated PR with regression tests.
- **Upstream deprecation risk**: if Bitnami deprecates `bitnamilegacy/*` (no announced timeline as of 2026-04), Insight will mirror the last-good tags into a self-hosted registry and update `image.repository` / `image.registry` in `values.yaml`.

**Enterprise customers** with a Bitnami subscription or an internal mirror should override the registry once in their values overlay:

```yaml
mariadb:
image:
registry: registry.internal.example.com
repository: my-mirror/mariadb
redis:
image:
registry: registry.internal.example.com
repository: my-mirror/redis
```

…and unset `global.security.allowInsecureImages` (the Bitnami chart's `secure-images` allowlist will accept your registry once images come from a non-`bitnamilegacy` path).

## Operations

```bash
Expand All @@ -146,27 +114,6 @@ helm -n insight uninstall insight
kubectl -n insight delete pvc -l app.kubernetes.io/part-of=insight
```

## Subchart prerequisites (done as part of this change)

To make the umbrella compose cleanly, two subcharts were patched:

- `helmfile/charts/clickhouse` — added `clickhouse.fullname` helper so the Service is named `<release>-clickhouse`, not just `<release>`.
- `src/frontend/helm` — changed `insight-frontend.fullname` to append `-frontend`, so it doesn't collide with other resources that use bare `{release}`.

Both charts remain compatible with the existing Helmfile (`helmfile sync`) — they add a suffix that wasn't there before, so service names under Helmfile become `clickhouse-clickhouse` / `frontend-frontend`. If Helmfile references old names anywhere, update those too.

## Relationship to `helmfile.yaml.gotmpl`

| Concern | `helmfile` (dev) | umbrella (distribution) |
|-------------------|------------------------------------|------------------------------------|
| Audience | Developers | Customers / GitOps |
| Invocation | `helmfile -e local sync` | `helm install insight charts/insight` |
| Templating | gotmpl DSL | Pure Helm + YAML |
| Secret injection | `.env.local` → helmfile vars | `existingSecret` references |
| Publishing | Not published | OCI registry (`helm push`) |

They coexist. Devs keep using Helmfile locally for fast iteration; distribution goes through the umbrella.

## Publishing (release workflow — not wired up yet)

```bash
Expand Down
10 changes: 5 additions & 5 deletions charts/insight/templates/NOTES.txt
Original file line number Diff line number Diff line change
Expand Up @@ -7,11 +7,11 @@
Release: {{ .Release.Name }}
Revision: {{ .Release.Revision }}

Resolved infra endpoints (see ConfigMap {{ .Release.Name }}-platform):
ClickHouse: {{ include "insight.clickhouse.url" . }} {{ if not .Values.clickhouse.deploy }}[external]{{ end }}
MariaDB: {{ include "insight.mariadb.host" . }}:{{ include "insight.mariadb.port" . }} {{ if not .Values.mariadb.deploy }}[external]{{ end }}
Redis: {{ include "insight.redis.url" . }} {{ if not .Values.redis.deploy }}[external]{{ end }}
Redpanda: {{ include "insight.redpanda.brokers" . }} {{ if not .Values.redpanda.deploy }}[external]{{ end }}
Resolved infra endpoints (external L2, see ConfigMap {{ .Release.Name }}-platform):
ClickHouse: {{ include "insight.clickhouse.url" . }}
MariaDB: {{ include "insight.mariadb.host" . }}:{{ include "insight.mariadb.port" . }}
Redis: {{ include "insight.redis.url" . }}
Redpanda: {{ include "insight.redpanda.brokers" . }}
Airbyte: {{ include "insight.airbyte.url" . }}

Application services (always-on):
Expand Down
Loading
Loading