From 960ef7d38659a4767948df2f6677c748c99b9e9b Mon Sep 17 00:00:00 2001 From: Konstantin Tursunov Date: Thu, 6 Aug 2026 11:21:58 +0800 Subject: [PATCH 01/10] feat(seed): run the demo seeder on Kubernetes stands MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit The seeder was compose-only, so a chart-deployed stand starts empty: every login is refused and every dashboard reads "No data". Make populating one a first-class path. Relocate the package to src/ingestion/tools/seed. The toolbox image's build context is already src/ingestion and its CI paths-filter is src/ingestion/**, so the published image now carries the seeder with no build-context, workflow or .dockerignore change, and a seeder edit rebuilds that image on its own. The tree splits into the `insight_seed` package and its `tests/`, with one entry point — `python3 -m insight_seed ` — shared by the compose image, the cluster Job and a local run. Give it an environment contract that fails loudly. TENANT_DEFAULT_ID and MARIADB_ANALYTICS_DB lose their defaults: rows written under the wrong tenant are invisible to every login while the run still reports success, and which database holds the catalogue tables is a per-stand fact — the compose stack keeps them in a database of their own, a chart-deployed stand in `mariadb.database`. A preflight module answers every such question before anything is written and reports the whole list at once instead of the first failure. Refuse rather than damage. The identity step is additive and marks every row it writes with a `reason` in its own namespace, so a tenant already holding other rows is refused. The silver step is NOT additive — it TRUNCATEs every table it writes, across all tenants — so a stand holding another tenant's rows anywhere in that reset surface is refused too. The surface is a single registry that `truncate` itself enforces, kept in step with the generator call sites by a test. SEED_FORCE=1 overrides either refusal, deliberately. Add seed-stand.sh, which reads a stand's coordinates from its own ConfigMap and Secrets, renders seed-job.yaml.tpl into a one-shot Job and follows it. Credentials never pass through the shell — the Job references the release's own Secret by key and runs as the application MariaDB user, which the chart already grants everything the seed writes. It is a plain Job rather than a chart hook, so a failed seed never gates or rolls back a release, and --dry-run prints the manifest instead of applying it. Along the way: gate the second-tenant fixture, which aborts identity-resolution's scheduled projection on a cluster and is only wanted on compose; read the activity window from one place so the manifest cannot report a window the rows do not sit in; make identity re-runs idempotent (seed_persons and seed_person_names still relied on an INSERT IGNORE collision that migration 004 removed, so each run appended duplicate email and name observations); and keep host build artefacts out of the toolbox image. Two things the tool cannot do for you, both documented: create the IdP user whose email anchors the dev-lead persona, and stand in for the chart's own ClickHouse migration hook. Refs #2243 Signed-off-by: Konstantin Tursunov --- .env.compose.example | 3 +- .github/dependabot.yml | 4 +- .github/workflows/e2e-stand.yml | 2 +- .gitignore | 10 +- CONTRIBUTING.md | 82 ++-- deploy/HELM_DEPLOY.md | 15 + deploy/compose/insight-init.sh | 25 +- deploy/compose/keycloak/README.md | 2 +- deploy/compose/keycloak/gen-realm.py | 29 +- deploy/compose/ui-tests.Dockerfile | 3 +- deploy/seed/README.md | 76 --- dev-compose.sh | 16 +- docker-compose.yml | 18 +- docs/TESTING.md | 2 +- src/ingestion/.dockerignore | 13 + .../ingestion/tools}/seed/Dockerfile | 26 +- .../ingestion/tools}/seed/PROFILE.md | 16 +- src/ingestion/tools/seed/README.md | 179 +++++++ .../tools/seed/insight_seed/__init__.py | 7 + .../tools/seed/insight_seed/__main__.py | 54 ++- .../tools/seed/insight_seed}/analytics.py | 22 +- .../tools/seed/insight_seed/config.py | 222 +++++++++ .../seed/insight_seed}/generators/__init__.py | 0 .../tools/seed/insight_seed}/generators/ai.py | 4 +- .../seed/insight_seed}/generators/base.py | 60 ++- .../seed/insight_seed}/generators/collab.py | 4 +- .../seed/insight_seed}/generators/crm.py | 4 +- .../seed/insight_seed}/generators/git.py | 4 +- .../tools/seed/insight_seed}/generators/hr.py | 40 +- .../seed/insight_seed}/generators/people.py | 4 +- .../seed/insight_seed}/generators/support.py | 4 +- .../seed/insight_seed}/generators/task.py | 4 +- .../seed/insight_seed}/golden_metrics.py | 2 +- .../tools/seed/insight_seed}/identity.py | 211 ++++++--- .../tools/seed/insight_seed}/manifest.py | 61 ++- .../tools/seed/insight_seed/preflight.py | 371 +++++++++++++++ .../tools/seed/insight_seed}/profile_md.py | 17 +- .../tools/seed/insight_seed}/profiles.py | 0 .../seed/insight_seed}/render_profile.py | 14 +- .../tools/seed/insight_seed}/silver.py | 51 +- .../ingestion/tools}/seed/pyproject.toml | 31 +- src/ingestion/tools/seed/seed-job.yaml.tpl | 145 ++++++ src/ingestion/tools/seed/seed-stand.sh | 447 ++++++++++++++++++ src/ingestion/tools/seed/tests/__init__.py | 1 + src/ingestion/tools/seed/tests/conftest.py | 39 ++ .../tools/seed/tests}/test_identity.py | 60 +-- .../tools/seed/tests/test_preflight.py | 307 ++++++++++++ src/ingestion/tools/toolbox/Dockerfile | 15 +- tests/generate_schemas.py | 2 +- tests/lib/insight_stand/__init__.py | 2 +- tests/lib/insight_stand/manifest.py | 19 +- tests/lib/insight_stand/personas.py | 2 +- tests/pyproject.toml | 12 +- tests/stand/README.md | 19 +- tests/stand/api/identity/test_internal.py | 13 +- tests/stand/conftest.py | 4 +- tests/versions.yaml | 2 +- 57 files changed, 2332 insertions(+), 469 deletions(-) delete mode 100644 deploy/seed/README.md rename {deploy => src/ingestion/tools}/seed/Dockerfile (59%) rename {deploy => src/ingestion/tools}/seed/PROFILE.md (94%) create mode 100644 src/ingestion/tools/seed/README.md create mode 100644 src/ingestion/tools/seed/insight_seed/__init__.py rename deploy/seed/seed.py => src/ingestion/tools/seed/insight_seed/__main__.py (58%) rename {deploy/seed => src/ingestion/tools/seed/insight_seed}/analytics.py (95%) create mode 100644 src/ingestion/tools/seed/insight_seed/config.py rename {deploy/seed => src/ingestion/tools/seed/insight_seed}/generators/__init__.py (100%) rename {deploy/seed => src/ingestion/tools/seed/insight_seed}/generators/ai.py (99%) rename {deploy/seed => src/ingestion/tools/seed/insight_seed}/generators/base.py (84%) rename {deploy/seed => src/ingestion/tools/seed/insight_seed}/generators/collab.py (99%) rename {deploy/seed => src/ingestion/tools/seed/insight_seed}/generators/crm.py (98%) rename {deploy/seed => src/ingestion/tools/seed/insight_seed}/generators/git.py (99%) rename {deploy/seed => src/ingestion/tools/seed/insight_seed}/generators/hr.py (75%) rename {deploy/seed => src/ingestion/tools/seed/insight_seed}/generators/people.py (98%) rename {deploy/seed => src/ingestion/tools/seed/insight_seed}/generators/support.py (97%) rename {deploy/seed => src/ingestion/tools/seed/insight_seed}/generators/task.py (99%) rename {deploy/seed => src/ingestion/tools/seed/insight_seed}/golden_metrics.py (97%) rename {deploy/seed => src/ingestion/tools/seed/insight_seed}/identity.py (68%) rename {deploy/seed => src/ingestion/tools/seed/insight_seed}/manifest.py (86%) create mode 100644 src/ingestion/tools/seed/insight_seed/preflight.py rename {deploy/seed => src/ingestion/tools/seed/insight_seed}/profile_md.py (92%) rename {deploy/seed => src/ingestion/tools/seed/insight_seed}/profiles.py (100%) rename {deploy/seed => src/ingestion/tools/seed/insight_seed}/render_profile.py (88%) rename {deploy/seed => src/ingestion/tools/seed/insight_seed}/silver.py (83%) rename {deploy => src/ingestion/tools}/seed/pyproject.toml (65%) create mode 100644 src/ingestion/tools/seed/seed-job.yaml.tpl create mode 100755 src/ingestion/tools/seed/seed-stand.sh create mode 100644 src/ingestion/tools/seed/tests/__init__.py create mode 100644 src/ingestion/tools/seed/tests/conftest.py rename {deploy/seed => src/ingestion/tools/seed/tests}/test_identity.py (67%) create mode 100644 src/ingestion/tools/seed/tests/test_preflight.py diff --git a/.env.compose.example b/.env.compose.example index 880485b85..7b93d313f 100644 --- a/.env.compose.example +++ b/.env.compose.example @@ -166,7 +166,8 @@ AUTHENTICATOR_IDENTITY_URL=http://identity-resolution:8082 # ── Local login identity ────────────────────────────────────────────── # The Keycloak realm anchors its dev-lead persona on this person, and the # seeder creates the matching development-team lead in Identity. Keep them -# in sync or re-seed after changing this. See deploy/seed/profiles.py. +# in sync or re-seed after changing this. See +# src/ingestion/tools/seed/insight_seed/profiles.py. DEV_USER_EMAIL=dev@company.nonpresent # ── Frontend security headers ───────────────────────────────────────── diff --git a/.github/dependabot.yml b/.github/dependabot.yml index a26877936..8151e271b 100644 --- a/.github/dependabot.yml +++ b/.github/dependabot.yml @@ -36,7 +36,7 @@ updates: patterns: ["*"] ignore: # dbt's dependency chain (dbt-common -> mashumaro) does not import on - # Python 3.14 yet; deploy/seed stays on 3.13 until it does. + # Python 3.14 yet, and the toolbox image the seeder ships in is 3.12. - dependency-name: python versions: [">=3.14"] @@ -60,7 +60,7 @@ updates: versions: [">=0.13"] - package-ecosystem: pip - directory: /deploy/seed + directory: /src/ingestion/tools/seed schedule: interval: daily time: "21:30" diff --git a/.github/workflows/e2e-stand.yml b/.github/workflows/e2e-stand.yml index c372166f6..fb95571f6 100644 --- a/.github/workflows/e2e-stand.yml +++ b/.github/workflows/e2e-stand.yml @@ -117,7 +117,7 @@ jobs: fi changed="$(git diff --name-only "$BASE" "$HEAD")" echo "$changed" | sed 's/^/ /' - if echo "$changed" | grep -qE '^(tests/(stand|lib)/|tests/pyproject\.toml|tests/uv\.lock|deploy/(seed|compose)/|docker-compose\.yml|dev-compose\.sh|src/backend/|src/frontend/helm/|docs/components/backend/.*/openapi\.json|\.github/workflows/e2e-stand\.yml|\.github/workflows/scripts/redact-playwright-trace\.py)'; then + if echo "$changed" | grep -qE '^(tests/(stand|lib)/|tests/pyproject\.toml|tests/uv\.lock|deploy/compose/|src/ingestion/tools/seed/|docker-compose\.yml|dev-compose\.sh|src/backend/|src/frontend/helm/|docs/components/backend/.*/openapi\.json|\.github/workflows/e2e-stand\.yml|\.github/workflows/scripts/redact-playwright-trace\.py)'; then echo "relevant=true" >> "$GITHUB_OUTPUT" else echo "relevant=false" >> "$GITHUB_OUTPUT" diff --git a/.gitignore b/.gitignore index 43a433e81..4bf6d5386 100644 --- a/.gitignore +++ b/.gitignore @@ -10,10 +10,10 @@ up-orb.sh /deploy/compose/build/ /deploy/compose/override.generated.yml /deploy/compose/keycloak/realm-insight.generated.json -/deploy/seed/.venv/ -/deploy/seed/__pycache__/ -/deploy/seed/.mypy_cache/ -/deploy/seed/.ruff_cache/ +/src/ingestion/tools/seed/.venv/ +/src/ingestion/tools/seed/__pycache__/ +/src/ingestion/tools/seed/.mypy_cache/ +/src/ingestion/tools/seed/.ruff_cache/ __pycache__/ *.pyc .venv/ @@ -582,7 +582,7 @@ trivy-image.sarif trufflehog-findings.jsonl # Seed manifest — per-stand runtime output (PROFILE.md is the committed view). -deploy/seed/manifest.json +src/ingestion/tools/seed/manifest.json # pytest-playwright's artefact directory: traces, screenshots and video from a # stand run. Written by `./dev-compose.sh test-stand test`, uploaded by CI. diff --git a/CONTRIBUTING.md b/CONTRIBUTING.md index 24720b804..803c7b59d 100644 --- a/CONTRIBUTING.md +++ b/CONTRIBUTING.md @@ -157,8 +157,8 @@ Then the chain runs: `bootstrap → fetch-cert → seal → system → deploy-app`. Subsequent runs skip the wizard and reconcile the stack. K8s and compose can coexist — disjoint host ports by default. Demo-data -seeding on k8s is manual (wizard output prints the port-forward + -`deploy/seed/` recipe). +seeding on k8s is one command +(`src/ingestion/tools/seed/seed-stand.sh`, printed by the wizard). ### Kubernetes — non-interactive (CI) @@ -456,9 +456,11 @@ for the per-environment IdP selection rationale. ## Seeding -The seed package lives in [`deploy/seed/`](deploy/seed/) — its -README documents the ruff / mypy / venv setup. Both deploy paths use -the same package; only how it's invoked differs. +The seeder lives in [`src/ingestion/tools/seed/`](src/ingestion/tools/seed/): +the `insight_seed` package, its `tests/`, and the artifacts it writes. Its +README documents the layout and the ruff / mypy / venv setup. Both deploy paths +run the same package (`python3 -m insight_seed `); only how it is invoked +differs. **Identity content (after `seed identity`):** CEO, your `DEV_USER_EMAIL` person (leads the dev team), 4 team leads (dev / @@ -469,10 +471,10 @@ the whole tree. **Silver content (after `seed silver`):** bronze + silver placeholder tables, every `src/ingestion/scripts/migrations/*.sql` applied -(produces the `insight.*` gold views), ~24k rows across 16 silver +(produces the `insight.*` gold views), ~29k rows across 19 silver tables profile-typed per team (`class_git_*` for devs, `class_crm_*` for sales, …). The full per-team activity table is in -[`deploy/seed/profiles.py`](deploy/seed/profiles.py). analytics's +[`src/ingestion/tools/seed/profiles.py`](src/ingestion/tools/seed/profiles.py). analytics's schema validator flips from "80 metrics error" to "80 ok". ### Compose @@ -492,50 +494,36 @@ To force auto-seed on next `up`, clear the `SEEDED_LOCAL_*` markers in ### Kubernetes -No auto-seed. The chart doesn't ship a `seed` Job, so you point the -same Python package at port-forwarded L2 services from the host. One -recipe per re-seed: +No auto-seed, but one command. The seeder ships inside the toolbox +image the release already pins, so nothing is built or port-forwarded: ```bash -# 1. Port-forward MariaDB + ClickHouse in the background. -KUBECONFIG=/path/to/config.yaml kubectl -n insight-infra \ - port-forward svc/mariadb 3306:3306 & -KUBECONFIG=/path/to/config.yaml kubectl -n insight-infra \ - port-forward svc/clickhouse 8123:8123 & - -# 2. Run the seed package against them. First time only: bootstrap a venv. -cd deploy/seed -python3 -m venv .venv && .venv/bin/pip install -r requirements.txt - -# Identity + silver. Drop `all` and pass `identity` / `silver` for partial. -# Schema inputs (placeholders script + gold-view migrations) are -# auto-located: the container bind-mount when present, otherwise -# repo-relative to deploy/seed. No path env vars needed. -MARIADB_HOST=127.0.0.1 MARIADB_PORT=3306 \ -MARIADB_USER=insight MARIADB_PASSWORD=insight-local \ -CLICKHOUSE_HOST=127.0.0.1 CLICKHOUSE_HTTP_PORT=8123 \ -CLICKHOUSE_USER=insight CLICKHOUSE_PASSWORD=insight-local \ -DEV_USER_EMAIL=dev@company.nonpresent \ - .venv/bin/python seed.py all - -# 3. Kick analytics so its schema validator re-runs against the -# now-populated silver tables. Without this, schema_status stays -# cached at boot-time 'table_not_found' and the FE shows "no peer -# data" everywhere (cf/insight#1307). -KUBECONFIG=/path/to/config.yaml kubectl -n insight \ - rollout restart deploy/insight-analytics - -# 4. Stop the port-forwards. -kill %1 %2 +export KUBECONFIG=/path/to/config.yaml +./src/ingestion/tools/seed/seed-stand.sh -n insight --email you@example.com + +# Kick analytics so its schema validator re-runs against the +# now-populated silver tables. Without this, schema_status stays +# cached at boot-time 'table_not_found' and the FE shows "no peer +# data" everywhere (cf/insight#1307). +kubectl -n insight rollout restart deploy/insight-analytics ``` -Use the real cluster credentials in place of `insight-local` if you -switched to external DBs at wizard time — the values are whatever the -operator stored in `secrets-store.yaml` and `make seal` baked into the -cluster's `mariadb-creds` / `clickhouse-creds` Secrets. - -The seeded `DEV_USER_EMAIL` must match FakeIdP's login identity so the -authenticator can resolve it to a person row. +The script reads the stand's coordinates from its own ConfigMap and +Secrets — hosts, databases, tenant, image — renders +[`seed-job.yaml.tpl`](src/ingestion/tools/seed/seed-job.yaml.tpl) into a +one-shot Job and follows it. `--dry-run` prints that Job instead of +applying it; `--step identity|silver|analytics` runs one step. No +credential passes through the shell: the Job reads the release's own +`insight-db-creds` by key, as the application MariaDB user. + +`--email` must name a user that already exists in the stand's IdP — the +authenticator resolves people by the email claim, so a seeded persona +nobody can authenticate as is not reachable. + +Seeding refuses rather than mixing: a tenant already holding `persons` +rows the seeder did not write, or silver tables holding another tenant's +rows (the silver step TRUNCATEs before writing), both stop the run. +`--force` overrides, deliberately. --- diff --git a/deploy/HELM_DEPLOY.md b/deploy/HELM_DEPLOY.md index 54c209b2f..0e2a435cb 100644 --- a/deploy/HELM_DEPLOY.md +++ b/deploy/HELM_DEPLOY.md @@ -27,6 +27,7 @@ This runbook shows a platform or DevOps engineer how to install the Insight busi - [Step 4 — Install with Helm](#step-4--install-with-helm) - [Step 5 — Verify the install](#step-5--verify-the-install) - [Step 6 — Configure connectors (optional)](#step-6--configure-connectors-optional) +- [Step 7 — Seed demo data (test stands only)](#step-7--seed-demo-data-test-stands-only) - [Appendix — Reference](#appendix--reference) - [values/umbrella.yaml placeholders](#valuesumbrellayaml-placeholders) - [secrets/insight-db-creds.yaml keys](#secretsinsight-db-credsyaml-keys) @@ -400,6 +401,20 @@ Configure connectors after the app is up. Each of the 25 connectors is a single See [deploy/CONNECTORS.md](./CONNECTORS.md) for the connector list and a copy-paste Secret for each. +## Step 7 — Seed demo data (test stands only) + +A freshly installed stand holds no people, so every login is refused and every dashboard reads "No data". On a **test** stand you can populate it with a 25-person demo organisation and per-team activity: + +```sh +./src/ingestion/tools/seed/seed-stand.sh -n insight --email you@example.com +``` + +The script reads this stand's own coordinates — infrastructure hosts from the `-platform` ConfigMap, the tenant and identity database from `insight-identity-resolution-config`, the image from `ingestion.toolboxImage` — and runs the seeder as a one-shot Job on that image. Nothing is hand-edited, no credential passes through the shell, and it runs as the application MariaDB user rather than root. `--dry-run` prints the Job it would apply; `--step identity` seeds only the roster. + +Two things it cannot do for you: a user with the `--email` address must already exist in your IdP (the authenticator resolves people by the email claim), and the ClickHouse schema must exist already — that is Step 4's migration hook. + +This is demo data. The seeder refuses a tenant that already holds `persons` rows it did not write, so pointing it at a stand carrying real identity data fails instead of mixing the two. See [the seeder's README](../src/ingestion/tools/seed/README.md) for the full flag list. + ## Appendix — Reference ### values/umbrella.yaml placeholders diff --git a/deploy/compose/insight-init.sh b/deploy/compose/insight-init.sh index 5dc28666e..48f806e73 100755 --- a/deploy/compose/insight-init.sh +++ b/deploy/compose/insight-init.sh @@ -762,19 +762,18 @@ EOF Next: \`make deploy ENV=local\` (already running, if invoked from there) will continue with: bootstrap → fetch-cert → seal → system → deploy-app. -Manual demo-data seeding (the compose stack auto-seeds; the k8s stack -doesn't ship a seed image yet — port-forward and run from the host): - - kubectl -n insight-infra port-forward svc/mariadb 3306:3306 & - kubectl -n insight-infra port-forward svc/clickhouse 8123:8123 & - cd $ROOT_DIR/deploy/seed - python3 -m venv .venv && .venv/bin/pip install -r requirements.txt - MARIADB_HOST=127.0.0.1 CLICKHOUSE_HOST=127.0.0.1 \\ - MARIADB_USER=$MARIADB_USER MARIADB_PASSWORD=$MARIADB_PASSWORD \\ - CLICKHOUSE_USER=$CLICKHOUSE_USER CLICKHOUSE_PASSWORD=$CLICKHOUSE_PASSWORD \\ - .venv/bin/python seed.py all - -See deploy/seed/README.md for the package layout. +Demo-data seeding (the compose stack auto-seeds; a k8s stand is seeded by +one command, which reads the stand's own coordinates and runs the seeder as +a Job on the toolbox image the release already pins): + + $ROOT_DIR/src/ingestion/tools/seed/seed-stand.sh -n insight --email you@example.com + +Add --dry-run to read the Job it would apply, or --step identity to seed only +the roster (no ClickHouse, finishes in seconds). A user with the --email +address must exist in the realm first: the authenticator resolves people by +the email claim. + +See src/ingestion/tools/seed/README.md for the flags and the package layout. EOF } diff --git a/deploy/compose/keycloak/README.md b/deploy/compose/keycloak/README.md index fb35fbc01..97940c756 100644 --- a/deploy/compose/keycloak/README.md +++ b/deploy/compose/keycloak/README.md @@ -82,7 +82,7 @@ one and only one tenant per token. artifact rebuilt from the seed roster on every `up`. To change realm shape, edit the generator's inputs: -- [`deploy/seed/profiles.py`](../../seed/profiles.py) — the roster (`build_roster`), +- [`src/ingestion/tools/seed/profiles.py`](../../../src/ingestion/tools/seed/profiles.py) — the roster (`build_roster`), team/role assignments, dev-lead email resolution. - [`gen-realm.py`](./gen-realm.py) — the realm generator (clients, protocol mappers, role mapping). The `insight-authenticator` client redirect + secret are parameters diff --git a/deploy/compose/keycloak/gen-realm.py b/deploy/compose/keycloak/gen-realm.py index 7c555c66e..5f355df11 100755 --- a/deploy/compose/keycloak/gen-realm.py +++ b/deploy/compose/keycloak/gen-realm.py @@ -2,7 +2,7 @@ """Generate the `insight` Keycloak realm from the seeded 25-person org. Reads the same roster builder the DB seeder uses -(`deploy/seed/profiles.py::build_roster`) so every user in the realm +(`src/ingestion/tools/seed/profiles.py::build_roster`) so every user in the realm matches a row in `identity.persons`, then emits an importable Keycloak realm JSON: 26 users (the 25-person org plus the admin operator), the `insight` + `insight-authenticator` clients, their 5 shared protocol @@ -21,12 +21,19 @@ import sys from pathlib import Path -# `deploy/seed` is a sibling package (not installed), so it has to be put on -# sys.path explicitly to import `profiles` from this script's location. -_SEED_DIR = Path(__file__).resolve().parents[2] / "seed" -sys.path.insert(0, str(_SEED_DIR)) +# The seed package (not installed) has to be put on sys.path explicitly to +# import it from this script's location. parents[3] = repo root; the entry +# added is the package's PARENT, so `insight_seed` imports as itself. +_SEED_TOOL_DIR = Path(__file__).resolve().parents[3] / "src/ingestion/tools/seed" +sys.path.insert(0, str(_SEED_TOOL_DIR)) -from profiles import TENANT_OTHER, Person, build_other_tenant_roster, build_roster, get_dev_user_email # noqa: E402 +from insight_seed.profiles import ( # noqa: E402 + TENANT_OTHER, + Person, + build_other_tenant_roster, + build_roster, + get_dev_user_email, +) REALM_NAME = "insight" DEV_PASSWORD = "insight-dev" @@ -247,10 +254,12 @@ def main() -> None: # Explicit --dev-email wins; otherwise fall back to DEV_USER_EMAIL via # get_dev_user_email() (which fail-fasts if that is also unset). dev_user_email = args.dev_email if args.dev_email else get_dev_user_email() - # Same fallback value as deploy/seed/identity.py's run() — that script's - # own TENANT_DEFAULT_ID lookup carries the identical default, so this - # mirrors (rather than introduces) that convention. - tenant_id = os.environ.get( # RULE-DEFAULTS-OK: mirrors deploy/seed/identity.py's TENANT_DEFAULT_ID default, so the realm's tenant_id claim converges with an un-configured seed run + # The compose stack's tenant. The seeder REQUIRES `TENANT_DEFAULT_ID` and + # keeps no default of its own (rows under the wrong tenant are invisible to + # every login), so this is the compose convention's only remaining home: + # docker-compose.yml passes the same value to the seeder, and both converge + # on it when nobody sets one. + tenant_id = os.environ.get( # RULE-DEFAULTS-OK: the compose tenant, mirrored by docker-compose.yml's TENANT_DEFAULT_ID default so the realm's tenant_id claim matches what the seed writes "TENANT_DEFAULT_ID", "00000000-df51-5b42-9538-d2b56b7ee953" ) diff --git a/deploy/compose/ui-tests.Dockerfile b/deploy/compose/ui-tests.Dockerfile index 877227ab9..097b02ba1 100644 --- a/deploy/compose/ui-tests.Dockerfile +++ b/deploy/compose/ui-tests.Dockerfile @@ -8,7 +8,8 @@ # docker run --rm --network container:insight-gateway \ # -e INSIGHT_STAND_BASE_URL=http://localhost:8080 \ # -e INSIGHT_STAND_PERSONA_PASSWORD=... \ -# -v "$PWD/deploy/seed/manifest.json:/deploy/seed/manifest.json:ro" \ +# -v "$PWD/src/ingestion/tools/seed/manifest.json:/stand/manifest.json:ro" \ +# -e INSIGHT_STAND_MANIFEST=/stand/manifest.json \ # insight-ui-tests:dev # # The network namespace is shared with the gateway rather than joining the diff --git a/deploy/seed/README.md b/deploy/seed/README.md deleted file mode 100644 index 14f50b212..000000000 --- a/deploy/seed/README.md +++ /dev/null @@ -1,76 +0,0 @@ -# Insight sample-data seeder - -Python script that populates the local docker-compose stack with a -25-person demo organisation (4 teams + CEO) and per-team activity in -ClickHouse silver tables. `profiles.py` documents the roster and the -per-team source-type weights; the per-domain generators under -`generators/` document the row shapes they emit. See -[PROFILE.md](PROFILE.md) for what a freshly seeded stand actually contains -— roster, fixtures, populated metrics and capabilities. - -## Run it - -The stack must be up first (`./dev-compose.sh up`). Then: - -```bash -./dev-compose.sh seed # everything -./dev-compose.sh seed identity # just identity -./dev-compose.sh seed silver # just silver -``` - -A successful run writes `manifest.json` next to this README, describing the -stand it just produced (roster, fixtures, data window, capabilities). - -## Reproducing a dataset - -`SEED_ANCHOR_DATE` fixes the last day carrying activity; `SEED_DAYS` sets the -window length. Pin both to reproduce a dataset exactly: - -```bash -SEED_ANCHOR_DATE=2026-06-30 SEED_DAYS=60 ./dev-compose.sh seed -``` - -Unset (or the literal `today`), the anchor is yesterday UTC, so the developer -loop stays populated as the calendar moves. Whichever applied is recorded in -`manifest.json`, so a stand always reports how to recreate it. - -## [PROFILE.md](PROFILE.md) - -[`PROFILE.md`](PROFILE.md) is generated and committed. Regenerate it after any change to the -roster or the manifest builder: - -```bash -python3 deploy/seed/render_profile.py # regenerate -python3 deploy/seed/render_profile.py --check # verify (no database needed) -``` - -## Develop on it - -```bash -cd deploy/seed -python3 -m venv .venv # one-time -.venv/bin/pip install -e '.[dev]' - -.venv/bin/ruff check . -.venv/bin/mypy . -``` - -Deps live in `pyproject.toml`: `[project.dependencies]` for runtime, -`[project.optional-dependencies].dev` for the tooling (ruff, mypy, stubs). - -## Layout - -| File | Role | -|------|------| -| `seed.py` | CLI entry; dispatches subcommands. | -| `profiles.py` | Demo roster + per-team activity weights. | -| `identity.py` | MariaDB seed: persons, org_chart, account_person_map. | -| `silver.py` | ClickHouse silver seed — full implementation: bronze placeholders → 8 domain generators → CH migrations → dependent-MV refresh. | -| `manifest.py` | Builds `manifest.json` — the machine-readable description of a seeded stand. | -| `golden_metrics.py` | The only source for the manifest's `golden_metrics[]`. Hand-curated. | -| `profile_md.py` | Renders `PROFILE.md` from the manifest. | -| `render_profile.py` | Regenerates / verifies `PROFILE.md`. Needs no database. | -| `PROFILE.md` | GENERATED — human-readable stand profile. Do not hand-edit. | -| `manifest.json` | GENERATED at seed time, per-stand (gitignored). | -| `Dockerfile` | One-shot image for the compose `seed-sample` service. | -| `pyproject.toml` | Package metadata, deps (runtime + dev), ruff + mypy config. | diff --git a/dev-compose.sh b/dev-compose.sh index 816d273b7..6e83791b2 100755 --- a/dev-compose.sh +++ b/dev-compose.sh @@ -464,7 +464,7 @@ cmd_up() { echo " Remove AUTH_MODE from $env_file to silence this." >&2 fi AUTH_MODE="keycloak" - # The seed-sample container reads AUTH_MODE too (deploy/seed/profiles.py's + # The seed-sample container reads AUTH_MODE too (src/ingestion/tools/seed/profiles.py's # get_login_id_pairs) to pick which roster personas get a login-id fixture — # export so the child `docker compose` process's env-var interpolation sees it. export AUTH_MODE @@ -655,7 +655,7 @@ YML # id to their OWN roster uuid, so sub IS that uuid — not the fixed # "fakeidp|dev" string fakeidp issues), so it must be seeded/looked-up # under its own source_type, not the fakeidp default (see - # deploy/seed/profiles.py::get_login_id_pairs). + # src/ingestion/tools/seed/profiles.py::get_login_id_pairs). export AUTHENTICATOR_IDP_SOURCE_TYPE="keycloak" echo "authenticator issuer → ${AUTHENTICATOR_OIDC_ISSUER}" @@ -1097,7 +1097,7 @@ Without that bounce, every metric stays cached at the boot-time and section badges read "no peer data" everywhere. Tracking upstream as constructorfabric/insight#1307. -See deploy/seed/README.md for the ruff/mypy/venv setup. +See src/ingestion/tools/seed/README.md for the ruff/mypy/venv setup. EOF } @@ -1475,7 +1475,7 @@ test_stand_write_env() { # # The list is committed rather than derived. It was read off the evidence # models' own sources (src/ingestion/gold/_metric_evidence.sql) against -# what deploy/seed/generators/ writes: +# what src/ingestion/tools/seed/generators/ writes: # # task <- task_issue_state / task_status_spans / task_worklog_flow (task.py) # git <- class_git_{commits,file_changes,pull_requests,…} (git.py) @@ -1623,7 +1623,7 @@ test_stand_test_in_image() { return 1 fi - local manifest="deploy/seed/manifest.json" + local manifest="src/ingestion/tools/seed/manifest.json" [[ -f "$manifest" ]] || { echo "ERROR: $manifest not found — seed the stand first: ./dev-compose.sh test-stand seed" >&2 return 1; } @@ -1648,7 +1648,11 @@ test_stand_test_in_image() { --user "$(id -u):$(id -g)" --network "container:${TEST_STAND_GATEWAY_CONTAINER}" -e "INSIGHT_STAND_BASE_URL=http://localhost:${TEST_STAND_GATEWAY_CONTAINER_PORT}" - -v "$PWD/${manifest}:/deploy/seed/manifest.json:ro" + # Mounted at a stable path and NAMED, rather than reproducing the suite's + # own repo-relative arithmetic inside an image where the tree lives at + # /tests and there is nothing above it. + -v "$PWD/${manifest}:/stand/manifest.json:ro" + -e "INSIGHT_STAND_MANIFEST=/stand/manifest.json" -v "$PWD/${TEST_STAND_ARTIFACT_DIR}:/tests/${TEST_STAND_ARTIFACT_DIR}" # Named, not inferred. The suite otherwise resolves this by walking up from # its own file to the directory holding `tests/` — which is the repo root in diff --git a/docker-compose.yml b/docker-compose.yml index 96d3b797e..cd9b4a9dc 100644 --- a/docker-compose.yml +++ b/docker-compose.yml @@ -384,8 +384,8 @@ services: APP__gears__authenticator__config__idp__client_secret: "${OIDC_CLIENT_SECRET:-}" # Required: the identity-resolution source_type the login-bootstrap # lookup is scoped to. The realm sets each user's `sub` to their own - # roster uuid — the stable external id; deploy/seed/identity.py seeds - # the matching value_type='id' rows under this same source_type. + # roster uuid — the stable external id; src/ingestion/tools/seed/identity.py + # seeds the matching value_type='id' rows under this same source_type. APP__gears__authenticator__config__idp__source_type: "${AUTHENTICATOR_IDP_SOURCE_TYPE:-keycloak}" # Callback rides the SPA's browser origin (Vite :3000) so the __Host-sid # cookie lands where the SPA runs; see .env.compose.example. @@ -701,7 +701,7 @@ services: container_name: ${COMPOSE_PROJECT_NAME:-insight}-seed-sample networks: [insight] build: - context: deploy/seed + context: src/ingestion/tools/seed dockerfile: Dockerfile # The image pins USER 1000. That is unambiguous where it was chosen — as a # k8s Job the seed mounts nothing from a host — but compose bind-mounts the @@ -726,7 +726,7 @@ services: IDP_SOURCE_TYPE: "${AUTHENTICATOR_IDP_SOURCE_TYPE:-keycloak}" # Selects which roster personas get a login-id fixture (fakeidp: dev lead # only, its fixed "fakeidp|dev"; keycloak: the whole roster, each on their - # own uuid — see deploy/seed/profiles.py's get_login_id_pairs). + # own uuid — see src/ingestion/tools/seed/profiles.py's get_login_id_pairs). # dev-compose.sh exports this; keycloak is the only mode it runs. AUTH_MODE: "${AUTH_MODE:-keycloak}" # MariaDB — falls back to the local docker service name/port; the @@ -735,6 +735,14 @@ services: MARIADB_PORT: "${MARIADB_INTERNAL_PORT:-3306}" MARIADB_USER: "${MARIADB_USER:-insight}" MARIADB_PASSWORD: "${MARIADB_PASSWORD:-insight-local}" + # Required by the seeder, with no default in code: which database holds the + # analytics catalogue tables is a per-stand fact. Here it is a database of + # its own; a chart-deployed stand keeps them in `mariadb.database`. + MARIADB_ANALYTICS_DB: "${MARIADB_ANALYTICS_DB:-analytics}" + # The second-tenant refusal fixture. ON here because tests/stand asserts + # cross-tenant refusal against it; a cluster stand turns it off, where a + # second tenant would abort identity-resolution's scheduled projection. + SEED_CROSS_TENANT_FIXTURE: "${SEED_CROSS_TENANT_FIXTURE:-1}" # ClickHouse — same convention as MariaDB above. CLICKHOUSE_HOST: "${CLICKHOUSE_HOST:-clickhouse}" CLICKHOUSE_HTTP_PORT: "${CLICKHOUSE_INTERNAL_HTTP_PORT:-8123}" @@ -766,7 +774,7 @@ services: # NOT read-only: the seed writes manifest.json back into this directory # at the end of a successful run, and downstream test phases read it # from that fixed path. - - ./deploy/seed:/app + - ./src/ingestion/tools/seed:/app # Schema + gold inputs: the SAME scripts the k8s clickhouse-migrate # Hook Job runs — create-bronze-placeholders.sh (+ lib/ch-exec.sh), # apply-ch-migrations.sh, migrations/*.sql — plus the full dbt project diff --git a/docs/TESTING.md b/docs/TESTING.md index 183e9327d..819455167 100644 --- a/docs/TESTING.md +++ b/docs/TESTING.md @@ -113,7 +113,7 @@ cd src/ingestion/tests/e2e - Every user-facing surface **should** have at least one smoke assertion. - A separate **compose-stand suite** (`tests/stand`, documented in `tests/stand/README.md`) drives a real Keycloak login and a set of browser journeys against the SPA, plus an API-contract suite — all against a local - `docker-compose` stand seeded deterministically for tests (`deploy/seed`). Run it with + `docker-compose` stand seeded deterministically for tests (`src/ingestion/tools/seed`). Run it with `./dev-compose.sh test-stand up|test|down`. It asserts no metric VALUE against a declared expectation: the seed's `golden_metrics` is empty by design, and a harness for it is being migrated separately. It does reconcile every metric's drilldown evidence against that metric's own served value, which needs no declared expectation. diff --git a/src/ingestion/.dockerignore b/src/ingestion/.dockerignore index fc4e2bec7..2a162e21a 100644 --- a/src/ingestion/.dockerignore +++ b/src/ingestion/.dockerignore @@ -1,6 +1,19 @@ secrets/ .env.local *.env.local +# Host-only Python artefacts, kept out of the toolbox image the same way the +# repo-root .dockerignore keeps them out of its contexts (Docker reads only the +# one at the context root, so this tree needs its own copy). Load-bearing rather +# than tidy: a host `.venv` under tools/seed carries macOS binaries that are +# useless — and hundreds of megabytes — inside a Linux image, and a stale +# `__pycache__` makes the container report HOST file paths. +**/__pycache__/ +**/*.py[cod] +**/.venv/ +**/.pytest_cache/ +**/.ruff_cache/ +**/.mypy_cache/ +**/*.egg-info/ tools/declarative-connector/ workflows/example-tenant/ workflows/*/ diff --git a/deploy/seed/Dockerfile b/src/ingestion/tools/seed/Dockerfile similarity index 59% rename from deploy/seed/Dockerfile rename to src/ingestion/tools/seed/Dockerfile index b1bdcc95b..b21562c0d 100644 --- a/deploy/seed/Dockerfile +++ b/src/ingestion/tools/seed/Dockerfile @@ -5,10 +5,11 @@ # so iterating on seed.py doesn't require a rebuild; rebuild when # pyproject.toml or the packaged sources change. -# Python 3.13 to match this seeder's pyproject (ruff target-version / -# mypy python_version = 3.13). dbt's dependency chain (dbt-common -> -# mashumaro) does not import on 3.14 yet, and dependabot.yml ignores that -# bump until it does. +# Above the package's >=3.12 floor rather than at it: this image is only the +# compose runner, and the floor exists for the toolbox image (python:3.12-slim) +# that carries the same package into a cluster. dbt's dependency chain +# (dbt-common -> mashumaro) does not import on 3.14 yet, and dependabot.yml +# ignores that bump until it does. FROM python:3.13-slim ENV PYTHONDONTWRITEBYTECODE=1 \ @@ -27,19 +28,18 @@ RUN apt-get update \ && apt-get install -y --no-install-recommends curl bash \ && rm -rf /var/lib/apt/lists/* -# Source is copied before the install so the build backend can package the -# flat modules + generators/. At runtime the source is bind-mounted (see -# docker-compose.yml seed-sample service); these COPY lines are also the -# fallback for `docker build && docker run` without a volume — exercised only -# by ad-hoc inspection, so the `generators/` package must come along or -# `silver.py`'s `from generators import ...` fails at import time. +# Source is copied before the install so the build backend can package it. At +# runtime the tool directory is bind-mounted over /app (see docker-compose.yml +# seed-sample service); this COPY is also the fallback for +# `docker build && docker run` without a volume. COPY pyproject.toml . -COPY *.py ./ -COPY generators ./generators/ +COPY insight_seed ./insight_seed/ RUN pip install . RUN useradd -U -u 1000 -m appuser USER 1000 -ENTRYPOINT ["python", "seed.py"] +# `-m` rather than a path: /app is the package's parent both in the image and +# under the bind mount, so the same invocation works either way. +ENTRYPOINT ["python", "-m", "insight_seed"] CMD ["all"] diff --git a/deploy/seed/PROFILE.md b/src/ingestion/tools/seed/PROFILE.md similarity index 94% rename from deploy/seed/PROFILE.md rename to src/ingestion/tools/seed/PROFILE.md index efc0a264e..ca630a306 100644 --- a/deploy/seed/PROFILE.md +++ b/src/ingestion/tools/seed/PROFILE.md @@ -1,11 +1,11 @@ + Regenerate: python3 -m insight_seed.render_profile + Verify: python3 -m insight_seed.render_profile --check + Content is derived from insight_seed/manifest.py + profiles.py. --> # Seed Profile -What a stand seeded by `deploy/seed` contains. Generated from the same +What a stand seeded by the seeder contains. Generated from the same builder that writes `manifest.json`, so the two cannot disagree. ## Stand summary @@ -17,7 +17,7 @@ builder that writes `manifest.json`, so the two cannot disagree. | realm | `insight` | | anchor_date | `2026-06-30` | | data_window | `2026-05-02..2026-06-30` | -| seed_revision | `35aeb2b31c302e8a` | +| seed_revision | `4d34657f4b488c01` | | manifest_version | 1 | `anchor_date` is the last day carrying seeded activity. It is resolved @@ -96,7 +96,7 @@ renaming one breaks every test that declares it. Rows the product provisions by operator or migration, so no endpoint creates them and no test fixture can either — the suite holds no -database connection. Seeded by `deploy/seed/analytics.py` and named +database connection. Seeded by `insight_seed/analytics.py` and named here so a test reads the name rather than hardcoding one. **No tenant `metric_definitions` override.** Nothing proves the listing @@ -107,11 +107,11 @@ resolves a tenant's label over the product default. **None.** The golden set is empty, and that is a deliberate state rather than an oversight. -> empty: no measured inventory records an exact expected value; see deploy/seed/golden_metrics.py for the criteria to add one +> empty: no measured inventory records an exact expected value; see insight_seed/golden_metrics.py for the criteria to add one A test suite consuming this manifest therefore asserts no metric values. That is a visible gap; a populated-but-guessed set would be a -silent wrong answer. See `deploy/seed/golden_metrics.py` for the +silent wrong answer. See `insight_seed/golden_metrics.py` for the criteria an entry must meet before it is added. ## Capabilities diff --git a/src/ingestion/tools/seed/README.md b/src/ingestion/tools/seed/README.md new file mode 100644 index 000000000..7059f7e0c --- /dev/null +++ b/src/ingestion/tools/seed/README.md @@ -0,0 +1,179 @@ +# Insight sample-data seeder + +Populates a stand with a 25-person demo organisation (4 teams + CEO) and +per-team activity in ClickHouse silver tables. `profiles.py` documents the +roster and the per-team source-type weights; the per-domain generators under +`generators/` document the row shapes they emit. See +[PROFILE.md](PROFILE.md) for what a freshly seeded stand actually contains +— roster, fixtures, populated metrics and capabilities. + +It runs against two kinds of stand, from the same sources: the local +docker-compose stack, and a chart-deployed Kubernetes stand (the package ships +inside the toolbox image, so no separate image or build is involved). + +This package lives inside `src/ingestion` deliberately: the silver step runs the +ingestion tree's own DDL and gold-build scripts, and being in the same tree means +the published toolbox image carries both, at one version, with no chance of the +seeder and the migration SQL drifting apart. + +## Run it on compose + +The stack must be up first (`./dev-compose.sh up`). Then: + +```bash +./dev-compose.sh seed # everything +./dev-compose.sh seed identity # just identity +./dev-compose.sh seed silver # just silver +``` + +A successful run writes `manifest.json` next to this README, describing the +stand it just produced (roster, fixtures, data window, capabilities). + +## Run it on a Kubernetes stand + +```bash +export KUBECONFIG= +./src/ingestion/tools/seed/seed-stand.sh -n --email you@example.com +``` + +That renders [`seed-job.yaml.tpl`](seed-job.yaml.tpl) into a one-shot Job, applies +it, and follows the logs. Every coordinate comes from the stand itself, so there +is no manifest to hand-edit and no tenant UUID to copy: + +| Value | Read from | +|-------|-----------| +| MariaDB + ClickHouse host, port, user | ConfigMap `-platform` | +| database holding the analytics catalogue | ConfigMap `-platform`, `MARIADB_DATABASE` | +| database holding `persons` | Secret `insight-identity-resolution-config`, `…database_url` | +| the stand's tenant | Secret `insight-identity-resolution-config`, `…tenant_default_id` | +| the image to run | `helm get values `, `ingestion.toolboxImage` | +| passwords | never read — the Job references Secret `insight-db-creds` by key | + +Credentials never pass through the script, and the Job runs as the application +MariaDB user rather than root: the umbrella already grants that user everything +the seed writes. + +Useful flags — `--dry-run` prints the rendered Job instead of applying it, +`--step identity|silver|analytics` runs one step (identity alone needs no +ClickHouse and finishes in seconds), `--tenant` seeds a tenant of your choosing, +and `--days` / `--anchor` pin the activity window. `--help` lists the rest. + +Anything the script cannot discover is a hard error naming the flag that supplies +it — it never falls back to a guess. + +### Two prerequisites it cannot satisfy for you + +1. **A user with `--email` must already exist in the stand's IdP.** The + authenticator resolves people by the email claim, so the seeded dev-lead + persona is only reachable by a login that already authenticates. Create the + user in the realm first, or point `--email` at one that exists. +2. **The stand's ClickHouse schema must exist** before `--step silver`, i.e. the + chart's `clickhouse-migrate` hook has run at least once. The step re-applies + the placeholder DDL and rebuilds gold, but it does not stand in for the + release's own migration path. + +### It refuses rather than making a mess + +Preflight runs before anything is written and reports every problem at once: + +- `TENANT_DEFAULT_ID` missing or not a UUID — rows under the wrong tenant are + invisible to every login while the run still reports success; +- the named analytics database does not hold `metric_definitions` — the error + names the database it looked in; +- MariaDB or ClickHouse unreachable, or the ingestion scripts missing; +- the target tenant already holds `persons` rows this seeder did not write + (every row it writes carries a `reason` starting `seed.py `); +- any table the silver step clears holds rows for another tenant — that step + **TRUNCATEs every table it writes**, across all tenants, so those rows would be + destroyed. This is the one genuinely destructive thing the seeder does, and it + is why an occupied stand is refused rather than merged into. The surface it + checks is `generators.base.RESET_TARGETS`, the same list `truncate` itself + enforces — including two inputs outside the silver database (an + identity-projection table and a bronze HR table). Targets carrying no tenant + column at all cannot be attributed to anyone; the run logs them by name + instead of pretending to have judged them. + +Either refusal is overridable with `--force`, which is how you say "yes, clear +it" out loud. + +The Job carries `backoffLimit: 0`: a failed seed is kept for reading rather than +retried, and because it is a plain Job rather than a chart hook, a failure never +touches the release or triggers a rollback. + +## Reproducing a dataset + +`SEED_ANCHOR_DATE` fixes the last day carrying activity; `SEED_DAYS` sets the +window length. Pin both to reproduce a dataset exactly: + +```bash +SEED_ANCHOR_DATE=2026-06-30 SEED_DAYS=60 ./dev-compose.sh seed +``` + +Unset (or the literal `today`), the anchor is yesterday UTC, so the developer +loop stays populated as the calendar moves. Whichever applied is recorded in +`manifest.json`, so a stand always reports how to recreate it. + +## [PROFILE.md](PROFILE.md) + +[`PROFILE.md`](PROFILE.md) is generated and committed. Regenerate it after any change to the +roster or the manifest builder: + +```bash +cd src/ingestion/tools/seed +python3 -m insight_seed.render_profile # regenerate +python3 -m insight_seed.render_profile --check # verify (no database needed) +``` + +## Develop on it + +```bash +cd src/ingestion/tools/seed +python3 -m venv .venv # one-time +.venv/bin/pip install -e '.[dev]' + +.venv/bin/ruff check . # package + tests +.venv/bin/mypy . +python3 -m unittest discover -s tests -t . # stdlib only, no database +``` + +The tests need nothing installed: they stub the database drivers and exercise +the pure half — the environment contract, the SQL a guard issues, and the +messages a refusal carries. + +Deps live in `pyproject.toml`: `[project.dependencies]` for runtime, +`[project.optional-dependencies].dev` for the tooling (ruff, mypy, stubs). + +## Layout + +Code and tests are separate trees, and the artifacts the package produces sit +at the root beside this README — where their readers (the stand suite, the +compose bind mount) name them. + +```text +src/ingestion/tools/seed/ +├── insight_seed/ the package — everything importable +│ ├── __main__.py `python3 -m insight_seed `: the entry point +│ ├── config.py environment contract: required, defaulted, and why +│ ├── preflight.py refuses a stand that cannot take the seed +│ ├── identity.py MariaDB: persons, org_chart, account_person_map +│ ├── silver.py ClickHouse: placeholders → generators → gold build +│ ├── analytics.py the catalogue rows no endpoint can create +│ ├── profiles.py demo roster + per-team activity weights +│ ├── manifest.py builds `manifest.json`, the stand's description +│ ├── golden_metrics.py the only source for the manifest's golden set +│ ├── profile_md.py renders `PROFILE.md` from a manifest +│ ├── render_profile.py regenerates / verifies `PROFILE.md`; no database +│ └── generators/ one module per activity domain, `base.py` shared +├── tests/ stdlib unittest; drivers stubbed in `conftest.py` +├── seed-stand.sh seeds a Kubernetes stand (discover → render → apply) +├── seed-job.yaml.tpl the Job it renders — and the reference manifest +├── Dockerfile the compose `seed-sample` image +├── pyproject.toml package metadata, deps, ruff + mypy config +├── PROFILE.md GENERATED, committed — do not hand-edit +└── manifest.json GENERATED per stand at seed time (gitignored) +``` + +On a cluster the runtime is the toolbox image (`../toolbox/Dockerfile`), which +carries this tree at `/ingestion/tools/seed` together with the migration +scripts the silver step runs — so the Job's command is the same +`python -m insight_seed` you would run locally. diff --git a/src/ingestion/tools/seed/insight_seed/__init__.py b/src/ingestion/tools/seed/insight_seed/__init__.py new file mode 100644 index 000000000..b87cd3656 --- /dev/null +++ b/src/ingestion/tools/seed/insight_seed/__init__.py @@ -0,0 +1,7 @@ +"""Insight sample-data seeder. + +Populates a stand — the compose stack or a chart-deployed Kubernetes stand — +with a demo organisation and its activity. `__main__` is the entry point +(`python -m insight_seed `); `README.md` one level up documents the flags, +the environment contract and what a seeded stand contains. +""" diff --git a/deploy/seed/seed.py b/src/ingestion/tools/seed/insight_seed/__main__.py similarity index 58% rename from deploy/seed/seed.py rename to src/ingestion/tools/seed/insight_seed/__main__.py index a02b82f0a..bc090536e 100644 --- a/deploy/seed/seed.py +++ b/src/ingestion/tools/seed/insight_seed/__main__.py @@ -9,9 +9,13 @@ tenant metric-definition override (MariaDB, analytics database). all Run every step. -See deploy/seed/README.md for the ruff/mypy/venv setup and the -per-domain generators under deploy/seed/generators/ for the data -shape each one emits. +Run as a module from the tool directory: + + python3 -m insight_seed all + +See the README one level up for the flags, the environment contract and the +ruff/mypy/venv setup; the per-domain generators under `generators/` document +the row shape each one emits. """ from __future__ import annotations @@ -21,9 +25,14 @@ LOG = logging.getLogger("seed") +STEPS = ("identity", "silver", "analytics") + def main(argv: list[str] | None = None) -> int: parser = argparse.ArgumentParser( + # Named for how it is invoked, not for the file argparse found itself in + # — `__main__.py` in a usage line tells a reader nothing. + prog="python3 -m insight_seed", description=__doc__, formatter_class=argparse.RawDescriptionHelpFormatter, ) @@ -34,17 +43,31 @@ def main(argv: list[str] | None = None) -> int: sub.add_parser("all", help="run every step") args = parser.parse_args(argv) + logging.basicConfig( + level=logging.INFO, + format="%(asctime)s %(levelname)s %(name)s %(message)s", + ) + + steps = STEPS if args.cmd == "all" else (args.cmd,) + + # Before anything is written: a seed takes minutes and writes to three + # places, so every answerable question is answered first. Raises with the + # whole list of problems rather than the first one. + from .preflight import check as preflight_check + + preflight_check(steps=steps) + seeded: list[str] = [] catalogue: dict[str, object] | None = None if args.cmd in ("identity", "all"): - from identity import run as run_identity + from .identity import run as run_identity run_identity() seeded.append("identity") if args.cmd in ("silver", "all"): - from silver import run as run_silver + from .silver import run as run_silver run_silver() seeded.append("silver") @@ -53,7 +76,7 @@ def main(argv: list[str] | None = None) -> int: # its startup, so the service has to have booted. On `all` that is already # true — the stand starts every service before seeding. if args.cmd in ("analytics", "all"): - from analytics import run as run_analytics + from .analytics import run as run_analytics catalogue = run_analytics() seeded.append("analytics") @@ -64,10 +87,25 @@ def main(argv: list[str] | None = None) -> int: # committed PROFILE.md. import os - from manifest import build_manifest, manifest_path, write_manifest + from .manifest import ( + assert_no_credentials, + build_manifest, + manifest_path, + render_manifest, + write_manifest, + ) + + doc = build_manifest(os.environ, seeded=seeded, catalogue=catalogue) + + # A cluster Job's filesystem dies with the pod, so the log is the only + # record of what a run seeded. Printed before the write, so it survives even + # a run that cannot persist the file — but never before the credential + # guard: a log line is as public as the file it stands in for. + assert_no_credentials(doc) + print(render_manifest(doc)) try: - path = write_manifest(build_manifest(os.environ, seeded=seeded, catalogue=catalogue)) + path = write_manifest(doc) LOG.info("manifest written: %s", path) except OSError as exc: # The seed container has historically mounted /app read-only. Fail diff --git a/deploy/seed/analytics.py b/src/ingestion/tools/seed/insight_seed/analytics.py similarity index 95% rename from deploy/seed/analytics.py rename to src/ingestion/tools/seed/insight_seed/analytics.py index 07a1cac08..459553fba 100644 --- a/deploy/seed/analytics.py +++ b/src/ingestion/tools/seed/insight_seed/analytics.py @@ -29,6 +29,8 @@ import pymysql +from . import config + LOG = logging.getLogger("seed.analytics") #: Deterministic id, so re-seeding an un-torn-down stand replaces its own row @@ -38,7 +40,7 @@ #: The label constant lives in `manifest` rather than here: `PROFILE.md` is #: rendered by a tool that must import no third-party package, and this module #: needs pymysql. The manifest owns the NAMES; this module owns writing the rows. -from manifest import OVERRIDE_LABEL # noqa: E402 +from .manifest import OVERRIDE_LABEL # noqa: E402 def _bin(u: str) -> bytes: @@ -48,14 +50,16 @@ def _bin(u: str) -> bytes: @contextmanager def _connect() -> Iterator[pymysql.connections.Connection]: + # Not MARIADB_DB: that one names the IDENTITY database. Which database holds + # the catalogue tables is a per-stand fact, so `MARIADB_ANALYTICS_DB` is + # required rather than defaulted (see config.parse_analytics_database). + target = config.parse_mariadb(os.environ, database=config.parse_analytics_database(os.environ)) conn = pymysql.connect( - host=os.environ.get("MARIADB_HOST", "mariadb"), - port=int(os.environ.get("MARIADB_PORT", "3306")), - user=os.environ.get("MARIADB_USER", "insight"), - password=os.environ.get("MARIADB_PASSWORD", "insight-local"), - # Not MARIADB_DB: that one names the IDENTITY database. These tables - # belong to analytics, which owns a database of its own. - database=os.environ.get("MARIADB_ANALYTICS_DB", "analytics"), + host=target.host, + port=target.port, + user=target.user, + password=target.password, + database=target.database, autocommit=False, cursorclass=pymysql.cursors.Cursor, ) @@ -278,7 +282,7 @@ def run() -> dict[str, Any]: level=logging.INFO, format="%(asctime)s %(levelname)s %(name)s %(message)s", ) - tenant = os.environ.get("TENANT_DEFAULT_ID", "00000000-df51-5b42-9538-d2b56b7ee953") + tenant = config.parse_tenant_id(os.environ) LOG.info("analytics catalogue seed (tenant %s)", tenant) with _connect() as conn: diff --git a/src/ingestion/tools/seed/insight_seed/config.py b/src/ingestion/tools/seed/insight_seed/config.py new file mode 100644 index 000000000..413e3b220 --- /dev/null +++ b/src/ingestion/tools/seed/insight_seed/config.py @@ -0,0 +1,222 @@ +"""Typed parsing of the seeder's environment contract. + +Two variables have no safe default and are therefore required rather than +guessed: + +* `TENANT_DEFAULT_ID` — rows are tenant-scoped, and rows written under a tenant + the stand does not use are invisible to every login while the seed still + reports success. +* `MARIADB_ANALYTICS_DB` — which database holds the analytics catalogue tables + is a per-stand fact, not a convention. + +Everything else keeps a default, because getting it wrong fails loudly on the +first connection attempt instead of producing a silently useless stand. +""" + +from __future__ import annotations + +import datetime as _dt +import uuid as uuid_mod +from collections.abc import Mapping +from dataclasses import dataclass, replace + +#: Prefix on the `reason` of every identity row this seeder writes. It is what +#: lets a preflight check tell demo rows from rows some other writer owns, so +#: `identity.py` composes every reason from it rather than spelling it out. +SEED_REASON_PREFIX = "seed.py " + +TENANT_ENV = "TENANT_DEFAULT_ID" +ANALYTICS_DB_ENV = "MARIADB_ANALYTICS_DB" +IDENTITY_DB_ENV = "MARIADB_DB" +CROSS_TENANT_FIXTURE_ENV = "SEED_CROSS_TENANT_FIXTURE" +FORCE_ENV = "SEED_FORCE" +ANCHOR_ENV = "SEED_ANCHOR_DATE" +DAYS_ENV = "SEED_DAYS" +DEV_USER_EMAIL_ENV = "DEV_USER_EMAIL" + +#: Length of the seeded activity window when nobody pins one. +DEFAULT_SEED_DAYS = 60 + +_TRUE = frozenset({"1", "true", "yes", "on"}) +_FALSE = frozenset({"0", "false", "no", "off"}) + + +class EnvContractError(Exception): + """The environment cannot describe a stand to seed. + + Carries every problem found, not the first one: an operator wiring up a new + stand should see the whole list in one run. + """ + + def __init__(self, problems: tuple[str, ...]) -> None: + self.problems = problems + super().__init__("\n".join(f" - {p}" for p in problems)) + + +@dataclass(frozen=True) +class MariaDb: + host: str + port: int + user: str + password: str + database: str + + def with_database(self, database: str) -> MariaDb: + return replace(self, database=database) + + +@dataclass(frozen=True) +class ClickHouse: + host: str + http_port: int + user: str + password: str + database: str + + @property + def url(self) -> str: + return f"http://{self.host}:{self.http_port}" + + +def parse_tenant_id(env: Mapping[str, str]) -> str: + """The tenant every seeded row is scoped to. Required, and a real UUID.""" + raw = (env.get(TENANT_ENV) or "").strip() + if not raw: + raise EnvContractError( + ( + f"{TENANT_ENV} is not set. Every seeded row is scoped to a tenant, and " + "rows written under the wrong one are invisible to every login on the " + "stand. Set it to the tenant the stand authenticates against " + "(the chart's global.tenantDefaultId).", + ) + ) + try: + uuid_mod.UUID(raw) + except ValueError as exc: + raise EnvContractError((f"{TENANT_ENV}={raw!r} is not a UUID: {exc}.",)) from exc + return raw + + +def parse_analytics_database(env: Mapping[str, str]) -> str: + """The database holding the analytics catalogue tables. Required. + + The compose stack keeps them in a database of their own; a chart-deployed + stand keeps them in `mariadb.database` alongside the rest of the product. + Neither is a default the other can live with. + """ + raw = (env.get(ANALYTICS_DB_ENV) or "").strip() + if not raw: + raise EnvContractError( + ( + f"{ANALYTICS_DB_ENV} is not set. It names the database holding the " + "analytics catalogue tables (metric_definitions, metric_catalog), which " + "differs per stand: the compose stack uses a separate `analytics` " + "database, a chart-deployed stand keeps them in `mariadb.database`.", + ) + ) + return raw + + +def parse_identity_database(env: Mapping[str, str]) -> str: + return (env.get(IDENTITY_DB_ENV) or "").strip() or "identity" + + +def parse_mariadb(env: Mapping[str, str], *, database: str) -> MariaDb: + return MariaDb( + host=env.get("MARIADB_HOST", "mariadb"), + port=int(env.get("MARIADB_PORT", "3306")), + user=env.get("MARIADB_USER", "insight"), + password=env.get("MARIADB_PASSWORD", "insight-local"), + database=database, + ) + + +def parse_clickhouse(env: Mapping[str, str]) -> ClickHouse: + return ClickHouse( + host=env.get("CLICKHOUSE_HOST", "clickhouse"), + http_port=int(env.get("CLICKHOUSE_HTTP_PORT", "8123")), + user=env.get("CLICKHOUSE_USER", "insight"), + password=env.get("CLICKHOUSE_PASSWORD", "insight-local"), + database=env.get("CLICKHOUSE_DATABASE", "insight"), + ) + + +def parse_flag(env: Mapping[str, str], name: str, *, default: bool) -> bool: + raw = (env.get(name) or "").strip().lower() + if not raw: + return default + if raw in _TRUE: + return True + if raw in _FALSE: + return False + raise EnvContractError( + (f"{name}={raw!r} is not a boolean; use one of {sorted(_TRUE | _FALSE)}.",) + ) + + +def cross_tenant_fixture_enabled(env: Mapping[str, str]) -> bool: + """Whether to write the second tenant's cross-tenant refusal fixture. + + On by default, because the compose stand's test suite asserts against it. A + cluster stand turns it off: the fixture trips identity-resolution's + tenant-mismatch guard, which then aborts every scheduled projection run. + """ + return parse_flag(env, CROSS_TENANT_FIXTURE_ENV, default=True) + + +def force_enabled(env: Mapping[str, str]) -> bool: + """Whether to seed a tenant that already holds rows this seeder did not write.""" + return parse_flag(env, FORCE_ENV, default=False) + + +def parse_anchor_date(env: Mapping[str, str]) -> _dt.date: + """Last day carrying seeded activity. + + Unset, or the literal `today`, means yesterday UTC — the default tracks the + calendar because a fixed past anchor ages one day per day and the surfaces + this data populates are read through a UI whose period is relative to now. + Determinism means "a given anchor always yields the same bytes", and the + anchor a run used is recorded in its manifest. + + Lives here, next to the rest of the environment contract, because both the + row generators and the manifest builder need the same answer: two readers of + the same variable computing `now()` independently disagree across a UTC + midnight, and the manifest would then report a window one day off the rows. + """ + raw = (env.get(ANCHOR_ENV) or "").strip() + if raw and raw.lower() != "today": + try: + return _dt.date.fromisoformat(raw) + except ValueError as exc: + raise EnvContractError( + (f"{ANCHOR_ENV}={raw!r} is not an ISO date (YYYY-MM-DD) or `today`: {exc}.",) + ) from exc + return _dt.datetime.now(_dt.UTC).date() - _dt.timedelta(days=1) + + +def parse_seed_days(env: Mapping[str, str], default: int = DEFAULT_SEED_DAYS) -> int: + """Length of the seeded activity window, in days. Empty means unset.""" + raw = (env.get(DAYS_ENV) or "").strip() + if not raw: + return default + try: + days = int(raw) + except ValueError as exc: + raise EnvContractError((f"{DAYS_ENV}={raw!r} is not a whole number of days.",)) from exc + if days < 1: + raise EnvContractError((f"{DAYS_ENV}={days} must be at least 1.",)) + return days + + +def parse_dev_user_email(env: Mapping[str, str]) -> str: + """The persona the dev-lead login resolves to. Required by every roster build.""" + raw = (env.get(DEV_USER_EMAIL_ENV) or "").strip().lower() + if not raw: + raise EnvContractError( + ( + f"{DEV_USER_EMAIL_ENV} is not set. It names the person who leads the demo dev " + "team, and the roster is built around them — a stand seeded without it has no " + "login that resolves to a person.", + ) + ) + return raw diff --git a/deploy/seed/generators/__init__.py b/src/ingestion/tools/seed/insight_seed/generators/__init__.py similarity index 100% rename from deploy/seed/generators/__init__.py rename to src/ingestion/tools/seed/insight_seed/generators/__init__.py diff --git a/deploy/seed/generators/ai.py b/src/ingestion/tools/seed/insight_seed/generators/ai.py similarity index 99% rename from deploy/seed/generators/ai.py rename to src/ingestion/tools/seed/insight_seed/generators/ai.py index c006f9390..089c10d40 100644 --- a/deploy/seed/generators/ai.py +++ b/src/ingestion/tools/seed/insight_seed/generators/ai.py @@ -12,7 +12,8 @@ from collections.abc import Sequence from typing import TYPE_CHECKING -from generators.base import ( +from ..profiles import TEAM_PROFILES, Person +from .base import ( anchor_datetime, bulk_insert, days_window, @@ -23,7 +24,6 @@ truncate, weekday_multiplier, ) -from profiles import TEAM_PROFILES, Person if TYPE_CHECKING: import clickhouse_connect.driver.client diff --git a/deploy/seed/generators/base.py b/src/ingestion/tools/seed/insight_seed/generators/base.py similarity index 84% rename from deploy/seed/generators/base.py rename to src/ingestion/tools/seed/insight_seed/generators/base.py index 5b14e94e5..7d13e2bcf 100644 --- a/deploy/seed/generators/base.py +++ b/src/ingestion/tools/seed/insight_seed/generators/base.py @@ -17,6 +17,8 @@ import re from typing import TYPE_CHECKING +from .. import config + LOG = logging.getLogger("seed.generators") if TYPE_CHECKING: @@ -26,12 +28,12 @@ UTC = _dt.UTC -DEFAULT_SEED_DAYS = 60 - -# Env knobs. Both are resolved ONCE per process by the helpers below and are -# recorded in the manifest, so a run is reproducible from what it reports. -_ANCHOR_ENV = "SEED_ANCHOR_DATE" -_DAYS_ENV = "SEED_DAYS" +# Env knobs are parsed by `config`, the one reader the manifest builder uses too +# — two copies computing `now()` independently disagree across UTC midnight, and +# the manifest would then report a window the rows do not sit in. Resolved ONCE +# per process by the helpers below, and recorded in the manifest, so a run is +# reproducible from what it reports. +DEFAULT_SEED_DAYS = config.DEFAULT_SEED_DAYS _anchor_cache: _dt.date | None = None @@ -61,11 +63,7 @@ def anchor_date() -> _dt.date: """ global _anchor_cache if _anchor_cache is None: - raw = os.environ.get(_ANCHOR_ENV, "").strip() - if raw and raw.lower() != "today": - _anchor_cache = _dt.date.fromisoformat(raw) - else: - _anchor_cache = _dt.datetime.now(UTC).date() - _dt.timedelta(days=1) + _anchor_cache = config.parse_anchor_date(os.environ) return _anchor_cache @@ -82,8 +80,7 @@ def anchor_datetime() -> _dt.datetime: def seed_days(default: int = DEFAULT_SEED_DAYS) -> int: """Length of the seeded activity window, in days.""" - raw = os.environ.get(_DAYS_ENV, "").strip() - return int(raw) if raw else default + return config.parse_seed_days(os.environ, default) def days_window(days: int, end: _dt.date | None = None) -> list[_dt.date]: @@ -162,12 +159,49 @@ def deterministic_int(*parts: str) -> int: # ─── Insert helpers ────────────────────────────────────────────────────── +#: Every relation the generators clear before writing — the seed's destructive +#: surface, in one place because `preflight` has to refuse a stand whose data +#: sits in exactly these and nowhere else. `truncate` rejects an unregistered +#: target, and `test_preflight.py` scans the call sites to keep the two in step: +#: a new generator that clears a table nobody registered fails the test rather +#: than quietly widening what a seed run destroys. +RESET_TARGETS: tuple[tuple[str, str], ...] = ( + ("bronze_bamboohr", "employees"), + ("identity", "identity_persons"), + ("silver", "class_ai_assistant_usage"), + ("silver", "class_ai_dev_usage"), + ("silver", "class_collab_chat_activity"), + ("silver", "class_collab_email_activity"), + ("silver", "class_collab_meeting_activity"), + ("silver", "class_crm_activities"), + ("silver", "class_crm_deals"), + ("silver", "class_crm_users"), + ("silver", "class_focus_metrics"), + ("silver", "class_git_commits"), + ("silver", "class_git_file_changes"), + ("silver", "class_git_pull_requests"), + ("silver", "class_git_pull_requests_commits"), + ("silver", "class_people"), + ("silver", "class_support_activity"), + ("silver", "class_task_field_history"), + ("silver", "class_task_statuses"), + ("silver", "class_task_users"), + ("silver", "class_task_worklogs"), +) + + def truncate( client: clickhouse_connect.driver.client.Client, schema: str, table: str, ) -> None: """Idempotent reset: TRUNCATE before INSERT.""" + if (schema, table) not in RESET_TARGETS: + raise ValueError( + f"{schema}.{table} is not in RESET_TARGETS. Clearing a relation preflight " + "does not know about would let a seed run destroy data it never warned about " + "— register it there (and in the same commit) instead." + ) client.command(f"TRUNCATE TABLE IF EXISTS `{schema}`.`{table}`") diff --git a/deploy/seed/generators/collab.py b/src/ingestion/tools/seed/insight_seed/generators/collab.py similarity index 99% rename from deploy/seed/generators/collab.py rename to src/ingestion/tools/seed/insight_seed/generators/collab.py index e246f1192..5b8a12f2d 100644 --- a/deploy/seed/generators/collab.py +++ b/src/ingestion/tools/seed/insight_seed/generators/collab.py @@ -9,7 +9,8 @@ from collections.abc import Sequence from typing import TYPE_CHECKING -from generators.base import ( +from ..profiles import TEAM_PROFILES, Person +from .base import ( bulk_insert, days_window, persona_multiplier, @@ -18,7 +19,6 @@ truncate, weekday_multiplier, ) -from profiles import TEAM_PROFILES, Person if TYPE_CHECKING: import clickhouse_connect.driver.client diff --git a/deploy/seed/generators/crm.py b/src/ingestion/tools/seed/insight_seed/generators/crm.py similarity index 98% rename from deploy/seed/generators/crm.py rename to src/ingestion/tools/seed/insight_seed/generators/crm.py index 1dd2edc19..67e417e10 100644 --- a/deploy/seed/generators/crm.py +++ b/src/ingestion/tools/seed/insight_seed/generators/crm.py @@ -11,7 +11,8 @@ from collections.abc import Sequence from typing import TYPE_CHECKING -from generators.base import ( +from ..profiles import TEAM_PROFILES, Person +from .base import ( anchor_date, bulk_insert, days_window, @@ -22,7 +23,6 @@ truncate, weekday_multiplier, ) -from profiles import TEAM_PROFILES, Person if TYPE_CHECKING: import clickhouse_connect.driver.client diff --git a/deploy/seed/generators/git.py b/src/ingestion/tools/seed/insight_seed/generators/git.py similarity index 99% rename from deploy/seed/generators/git.py rename to src/ingestion/tools/seed/insight_seed/generators/git.py index d5ca7ef92..c50078efc 100644 --- a/deploy/seed/generators/git.py +++ b/src/ingestion/tools/seed/insight_seed/generators/git.py @@ -11,7 +11,8 @@ from collections.abc import Sequence from typing import TYPE_CHECKING -from generators.base import ( +from ..profiles import TEAM_PROFILES, Person +from .base import ( bulk_insert, clamp, days_window, @@ -23,7 +24,6 @@ truncate, weekday_multiplier, ) -from profiles import TEAM_PROFILES, Person if TYPE_CHECKING: import clickhouse_connect.driver.client diff --git a/deploy/seed/generators/hr.py b/src/ingestion/tools/seed/insight_seed/generators/hr.py similarity index 75% rename from deploy/seed/generators/hr.py rename to src/ingestion/tools/seed/insight_seed/generators/hr.py index 7e9cad8f3..f2d18963d 100644 --- a/deploy/seed/generators/hr.py +++ b/src/ingestion/tools/seed/insight_seed/generators/hr.py @@ -14,7 +14,8 @@ from collections.abc import Sequence from typing import TYPE_CHECKING -from generators.base import ( +from ..profiles import Person +from .base import ( bulk_insert, clamp, days_window, @@ -24,7 +25,6 @@ truncate, weekday_multiplier, ) -from profiles import Person if TYPE_CHECKING: import clickhouse_connect.driver.client @@ -43,9 +43,16 @@ def seed_focus_metrics( ) -> int: truncate(client, "silver", "class_focus_metrics") cols = [ - "insight_tenant_id", "email", "day", "unique_key", - "meetings_count", "meeting_hours", "working_hours_per_day", - "focus_time_pct", "dev_time_h", "_version", + "insight_tenant_id", + "email", + "day", + "unique_key", + "meetings_count", + "meeting_hours", + "working_hours_per_day", + "focus_time_pct", + "dev_time_h", + "_version", ] rows: list[tuple[object, ...]] = [] version = 1 @@ -63,17 +70,26 @@ def seed_focus_metrics( # heavy day, generally less. m_hours = clamp( rng.gauss(1.5 * persona * weekday_multiplier(d), 0.8), - 0, min(wh * 0.7, 6), + 0, + min(wh * 0.7, 6), ) m_count = round(m_hours * 1.5) focus_pct = 0.0 if wh < 1 else clamp((wh - m_hours) / wh * 100, 0, 100) dev_h = clamp(wh - m_hours, 0, 10) - rows.append(( - tenant_uuid, p.email, d, - deterministic_uuid("focus", p.uuid, d.isoformat()), - m_count, round(m_hours, 2), round(wh, 2), - round(focus_pct, 2), round(dev_h, 2), version, - )) + rows.append( + ( + tenant_uuid, + p.email, + d, + deterministic_uuid("focus", p.uuid, d.isoformat()), + m_count, + round(m_hours, 2), + round(wh, 2), + round(focus_pct, 2), + round(dev_h, 2), + version, + ) + ) return bulk_insert(client, "silver", "class_focus_metrics", cols, rows) diff --git a/deploy/seed/generators/people.py b/src/ingestion/tools/seed/insight_seed/generators/people.py similarity index 98% rename from deploy/seed/generators/people.py rename to src/ingestion/tools/seed/insight_seed/generators/people.py index f4101c7f4..f6b8edf05 100644 --- a/deploy/seed/generators/people.py +++ b/src/ingestion/tools/seed/insight_seed/generators/people.py @@ -33,8 +33,8 @@ from collections.abc import Sequence from typing import TYPE_CHECKING -from generators.base import bulk_insert, deterministic_uuid, truncate -from profiles import Person +from ..profiles import Person +from .base import bulk_insert, deterministic_uuid, truncate if TYPE_CHECKING: import clickhouse_connect.driver.client diff --git a/deploy/seed/generators/support.py b/src/ingestion/tools/seed/insight_seed/generators/support.py similarity index 97% rename from deploy/seed/generators/support.py rename to src/ingestion/tools/seed/insight_seed/generators/support.py index 48fd27a44..a26623129 100644 --- a/deploy/seed/generators/support.py +++ b/src/ingestion/tools/seed/insight_seed/generators/support.py @@ -13,7 +13,8 @@ from collections.abc import Sequence from typing import TYPE_CHECKING -from generators.base import ( +from ..profiles import TEAM_PROFILES, Person +from .base import ( bulk_insert, days_window, persona_multiplier, @@ -22,7 +23,6 @@ truncate, weekday_multiplier, ) -from profiles import TEAM_PROFILES, Person if TYPE_CHECKING: import clickhouse_connect.driver.client diff --git a/deploy/seed/generators/task.py b/src/ingestion/tools/seed/insight_seed/generators/task.py similarity index 99% rename from deploy/seed/generators/task.py rename to src/ingestion/tools/seed/insight_seed/generators/task.py index 6ad12f6df..3572ca69f 100644 --- a/deploy/seed/generators/task.py +++ b/src/ingestion/tools/seed/insight_seed/generators/task.py @@ -22,7 +22,8 @@ from collections.abc import Sequence from typing import TYPE_CHECKING -from generators.base import ( +from ..profiles import TEAM_PROFILES, Person +from .base import ( anchor_date, anchor_datetime, bulk_insert, @@ -34,7 +35,6 @@ truncate, weekday_multiplier, ) -from profiles import TEAM_PROFILES, Person if TYPE_CHECKING: import clickhouse_connect.driver.client diff --git a/deploy/seed/golden_metrics.py b/src/ingestion/tools/seed/insight_seed/golden_metrics.py similarity index 97% rename from deploy/seed/golden_metrics.py rename to src/ingestion/tools/seed/insight_seed/golden_metrics.py index 93bd0d52a..83d2bf936 100644 --- a/deploy/seed/golden_metrics.py +++ b/src/ingestion/tools/seed/insight_seed/golden_metrics.py @@ -53,5 +53,5 @@ # from "measured and genuinely zero". GOLDEN_METRICS_NOTE = ( "empty: no measured inventory records an exact expected value; " - "see deploy/seed/golden_metrics.py for the criteria to add one" + "see insight_seed/golden_metrics.py for the criteria to add one" ) diff --git a/deploy/seed/identity.py b/src/ingestion/tools/seed/insight_seed/identity.py similarity index 68% rename from deploy/seed/identity.py rename to src/ingestion/tools/seed/insight_seed/identity.py index 7aa40b80e..63c869faa 100644 --- a/deploy/seed/identity.py +++ b/src/ingestion/tools/seed/insight_seed/identity.py @@ -16,7 +16,8 @@ import pymysql -from profiles import ( +from . import config +from .profiles import ( ADMIN_ROLE_NAME, AUTHOR_PERSON_UUID, DEV_SEED_SOURCE_ID, @@ -34,6 +35,17 @@ LOG = logging.getLogger("seed.identity") +# Every row this module writes is marked as this seeder's own. Composed from the +# shared prefix rather than spelled out, because `preflight` matches on that +# prefix to tell demo rows from rows another writer owns — a reason that drifted +# out of the namespace would make the whole stand look foreign. +_REASON_ROSTER = f"{config.SEED_REASON_PREFIX}demo roster" +_REASON_LOGIN_ID = f"{config.SEED_REASON_PREFIX}login id" +_REASON_NAMES = f"{config.SEED_REASON_PREFIX}demo names" +_REASON_ORG_CHART = f"{config.SEED_REASON_PREFIX}demo org-chart" +_REASON_ADMIN = f"{config.SEED_REASON_PREFIX}admin operator" +_REASON_ACCOUNT_MAP = f"{config.SEED_REASON_PREFIX}account-person map" + def _bin(u: str) -> bytes: """UUID string → 16 raw bytes, RFC 4122 big-endian.""" @@ -42,17 +54,13 @@ def _bin(u: str) -> bytes: @contextmanager def _connect() -> Iterator[pymysql.connections.Connection]: - host = os.environ.get("MARIADB_HOST", "mariadb") - port = int(os.environ.get("MARIADB_PORT", "3306")) - user = os.environ.get("MARIADB_USER", "insight") - pwd = os.environ.get("MARIADB_PASSWORD", "insight-local") - db = os.environ.get("MARIADB_DB", "identity") + target = config.parse_mariadb(os.environ, database=config.parse_identity_database(os.environ)) conn = pymysql.connect( - host=host, - port=port, - user=user, - password=pwd, - database=db, + host=target.host, + port=target.port, + user=target.user, + password=target.password, + database=target.database, autocommit=False, cursorclass=pymysql.cursors.Cursor, ) @@ -66,19 +74,60 @@ def _connect() -> Iterator[pymysql.connections.Connection]: conn.close() +#: The two columns an observation's value lands in, by kind. `persons` is an EAV +#: log: identifier-shaped values go in `value_id`, free text in `value_full_text`. +_VALUE_COLUMNS = ("value_id", "value_full_text") + + +def _observation_exists( + cur: pymysql.cursors.Cursor, + tenant_uuid: str, + person_uuid: str, + value_type: str, + value_column: str, + value: str, +) -> bool: + """Whether this exact observation is already recorded. + + Checked explicitly rather than left to `INSERT IGNORE`, for the reason + `seed_login_ids` documents at length: since migration 004 the unique key + carries `created_at`, so a re-run's insert never collides and IGNORE stopped + deduplicating anything. The logical key — ignoring `created_at` — is what + makes a re-run a no-op. + """ + if value_column not in _VALUE_COLUMNS: + raise ValueError(f"{value_column!r} is not an observation value column") + cur.execute( + f""" + SELECT 1 FROM persons + WHERE insight_tenant_id = %s + AND person_id = %s + AND insight_source_type = %s + AND insight_source_id = %s + AND value_type = %s + AND `{value_column}` = %s + LIMIT 1 + """, + ( + _bin(tenant_uuid), + _bin(person_uuid), + DEV_SEED_SOURCE_TYPE, + _bin(DEV_SEED_SOURCE_ID), + value_type, + value, + ), + ) + return cur.fetchone() is not None + + def seed_persons( cur: pymysql.cursors.Cursor, tenant_uuid: str, roster: Iterable[Person], ) -> int: - """Insert one observation row per person (value_type='email'). - - The unique key on `persons` is - (tenant, person, source_type, source_id, value_type, value_hash). - INSERT IGNORE absorbs re-runs cleanly. - """ + """Insert one observation row per person (value_type='email').""" sql = """ - INSERT IGNORE INTO persons ( + INSERT INTO persons ( value_type, insight_source_type, insight_source_id, insight_tenant_id, value_id, person_id, author_person_id, reason @@ -86,20 +135,24 @@ def seed_persons( 'email', %s, %s, %s, %s, %s, %s, %s ) """ - rows = [ - ( - DEV_SEED_SOURCE_TYPE, - _bin(DEV_SEED_SOURCE_ID), - _bin(tenant_uuid), - p.email, - _bin(p.uuid), - _bin(AUTHOR_PERSON_UUID), - "seed.py demo roster", + inserted = 0 + for p in roster: + if _observation_exists(cur, tenant_uuid, p.uuid, "email", "value_id", p.email): + continue + cur.execute( + sql, + ( + DEV_SEED_SOURCE_TYPE, + _bin(DEV_SEED_SOURCE_ID), + _bin(tenant_uuid), + p.email, + _bin(p.uuid), + _bin(AUTHOR_PERSON_UUID), + _REASON_ROSTER, + ), ) - for p in roster - ] - cur.executemany(sql, rows) - return cur.rowcount + inserted += cur.rowcount + return inserted def seed_login_ids( @@ -151,24 +204,30 @@ def seed_login_ids( """ inserted = 0 for person_uuid, external_id in get_login_id_pairs(list(roster)): - cur.execute(exists_sql, ( - _bin(tenant_uuid), - _bin(person_uuid), - source_type, - _bin(DEV_SEED_SOURCE_ID), - external_id, - )) + cur.execute( + exists_sql, + ( + _bin(tenant_uuid), + _bin(person_uuid), + source_type, + _bin(DEV_SEED_SOURCE_ID), + external_id, + ), + ) if cur.fetchone() is not None: continue - cur.execute(insert_sql, ( - source_type, - _bin(DEV_SEED_SOURCE_ID), - _bin(tenant_uuid), - external_id, - _bin(person_uuid), - _bin(AUTHOR_PERSON_UUID), - "seed.py login id", - )) + cur.execute( + insert_sql, + ( + source_type, + _bin(DEV_SEED_SOURCE_ID), + _bin(tenant_uuid), + external_id, + _bin(person_uuid), + _bin(AUTHOR_PERSON_UUID), + _REASON_LOGIN_ID, + ), + ) inserted += cur.rowcount return inserted @@ -183,11 +242,10 @@ def seed_person_names( The identity service routes these value_types into `value_full_text` (not `value_id`); see seed-persons-from-identity-input.py's VALUE_TYPES_FOR_VALUE_FULL_TEXT. Without them the persons API returns - empty names and the UI falls back to email. INSERT IGNORE absorbs - re-runs (unique key includes value_type). + empty names and the UI falls back to email. """ sql = """ - INSERT IGNORE INTO persons ( + INSERT INTO persons ( value_type, insight_source_type, insight_source_id, insight_tenant_id, value_full_text, person_id, author_person_id, reason @@ -195,7 +253,7 @@ def seed_person_names( %s, %s, %s, %s, %s, %s, %s, %s ) """ - rows: list[tuple[object, ...]] = [] + inserted = 0 for p in roster: for value_type, value in ( ("display_name", p.display_name), @@ -204,7 +262,10 @@ def seed_person_names( ): if not value: continue - rows.append( + if _observation_exists(cur, tenant_uuid, p.uuid, value_type, "value_full_text", value): + continue + cur.execute( + sql, ( value_type, DEV_SEED_SOURCE_TYPE, @@ -213,11 +274,11 @@ def seed_person_names( value, _bin(p.uuid), _bin(AUTHOR_PERSON_UUID), - "seed.py demo names", - ) + _REASON_NAMES, + ), ) - cur.executemany(sql, rows) - return cur.rowcount + inserted += cur.rowcount + return inserted def seed_org_chart( @@ -243,7 +304,7 @@ def seed_org_chart( _bin(p.uuid), _bin(p.parent_uuid), _bin(AUTHOR_PERSON_UUID), - "seed.py demo org-chart", + _REASON_ORG_CHART, ) for p in roster if p.parent_uuid @@ -296,7 +357,7 @@ def seed_person_roles( _bin(p.uuid), role_id, _bin(AUTHOR_PERSON_UUID), - "seed.py admin operator", + _REASON_ADMIN, ) for p in roster if p.role == "admin" @@ -341,7 +402,7 @@ def seed_account_person_map( p.email, _bin(p.uuid), _bin(AUTHOR_PERSON_UUID), - "seed.py account-person map", + _REASON_ACCOUNT_MAP, ) ) cur.executemany(sql, rows) @@ -354,7 +415,7 @@ def run() -> None: format="%(asctime)s %(levelname)s %(name)s %(message)s", ) - tenant = os.environ.get("TENANT_DEFAULT_ID", "00000000-df51-5b42-9538-d2b56b7ee953") + tenant = config.parse_tenant_id(os.environ) dev_email = get_dev_user_email() roster = build_roster(dev_email) LOG.info( @@ -369,12 +430,23 @@ def run() -> None: # running them again under a different tenant is the whole difference. A # per-person tenant field would have had to thread through five writers and # every generator, to express one row. - other_roster = build_other_tenant_roster() - LOG.info( - "seeding %d person(s) under tenant %s (cross-tenant refusal fixture)", - len(other_roster), - TENANT_OTHER, + # + # Off on a cluster stand: a second tenant makes identity-resolution's + # scheduled projection abort on its tenant-mismatch guard, and the suite + # that reads this fixture only ever runs against compose. `manifest.py` + # reads the same switch, so a stand seeded without it advertises no + # `other_tenant_lead` fixture and the tests that need one skip. + other_roster = ( + build_other_tenant_roster() if config.cross_tenant_fixture_enabled(os.environ) else [] ) + if other_roster: + LOG.info( + "seeding %d person(s) under tenant %s (cross-tenant refusal fixture)", + len(other_roster), + TENANT_OTHER, + ) + else: + LOG.info("cross-tenant refusal fixture disabled (%s)", config.CROSS_TENANT_FIXTURE_ENV) with _connect() as conn: cur = conn.cursor() @@ -388,10 +460,11 @@ def run() -> None: # No org_chart and no person_roles for them: they are a caller, not a # subject. An edge would put them in somebody's subtree, and a role # would make the refusal ambiguous — is it the tenant or the grant? - n_persons += seed_persons(cur, TENANT_OTHER, other_roster) - n_login_id += seed_login_ids(cur, TENANT_OTHER, other_roster) - n_names += seed_person_names(cur, TENANT_OTHER, other_roster) - n_acct += seed_account_person_map(cur, TENANT_OTHER, other_roster) + if other_roster: + n_persons += seed_persons(cur, TENANT_OTHER, other_roster) + n_login_id += seed_login_ids(cur, TENANT_OTHER, other_roster) + n_names += seed_person_names(cur, TENANT_OTHER, other_roster) + n_acct += seed_account_person_map(cur, TENANT_OTHER, other_roster) LOG.info( "DONE: persons=%d (new), login_id=%d (new), names=%d (new), " diff --git a/deploy/seed/manifest.py b/src/ingestion/tools/seed/insight_seed/manifest.py similarity index 86% rename from deploy/seed/manifest.py rename to src/ingestion/tools/seed/insight_seed/manifest.py index 2e2447708..0b9a1ae34 100644 --- a/deploy/seed/manifest.py +++ b/src/ingestion/tools/seed/insight_seed/manifest.py @@ -1,8 +1,9 @@ """Seed manifest — the machine-readable description of a seeded stand. -Written to the fixed path `deploy/seed/manifest.json` (`/app/manifest.json` -in the seed container, the same file through the bind mount). Downstream test -phases read that path; it is frozen and has no env knob. +Written to one fixed path — `manifest.json` at the tool root, beside this +package (`/app/manifest.json` in the compose seed container, the same file +through the bind mount). Downstream test phases read that path; it is frozen +and has no env knob. The builder is a PURE FUNCTION of (roster, supplied env, committed constants). It queries neither MariaDB nor ClickHouse. That is what lets the @@ -25,8 +26,8 @@ from pathlib import Path from typing import Any -import profiles -from golden_metrics import GOLDEN_METRICS, GOLDEN_METRICS_NOTE +from . import config, profiles +from .golden_metrics import GOLDEN_METRICS, GOLDEN_METRICS_NOTE MANIFEST_VERSION = 1 @@ -79,6 +80,9 @@ "SEED_DAYS": "60", "AUTH_MODE": "", "AUTHENTICATOR_OIDC_ISSUER": "", + # The canonical stand carries the cross-tenant refusal fixture, so the + # committed PROFILE.md describes a compose stand — the one the suite reads. + config.CROSS_TENANT_FIXTURE_ENV: "1", } # Literals that must never reach the manifest. Checked before the file is @@ -94,30 +98,36 @@ ) _FORBIDDEN_KEY_SUBSTRINGS = ("password", "secret", "token", "credential", "passwd") +#: The tool directory: the package's home, holding the artifacts it writes +#: (`manifest.json`) and the ones it renders (`PROFILE.md`). +_TOOL_ROOT = Path(__file__).resolve().parents[1] -def manifest_path() -> Path: - """The frozen manifest location. No env knob by design.""" - return Path(__file__).resolve().parent / "manifest.json" +def manifest_path() -> Path: + """The frozen manifest location. No env knob by design. -def _anchor(env: Mapping[str, str]) -> _dt.date: - raw = (env.get("SEED_ANCHOR_DATE") or "").strip() - if raw and raw.lower() != "today": - return _dt.date.fromisoformat(raw) - return _dt.datetime.now(_dt.UTC).date() - _dt.timedelta(days=1) + The tool directory, not the package directory: the manifest is a per-stand + artifact this package produces, and its readers (the stand suite, the + compose bind mount) name that path. + """ + return _TOOL_ROOT / "manifest.json" -def _days(env: Mapping[str, str]) -> int: - raw = (env.get("SEED_DAYS") or "").strip() - return int(raw) if raw else 60 +# The window comes from `config`, the same reader the generators use, so the +# window this document reports and the dates the rows carry cannot disagree — +# two independent `now()` calls straddling UTC midnight is exactly how they +# would. +_anchor = config.parse_anchor_date +_days = config.parse_seed_days def seed_revision() -> str: """Content hash over the seed package's Python sources. - Identifies the generator code that produced a stand, with no git - dependency and no clock. Any edit under deploy/seed changes it, which is - the point: it is what makes a committed PROFILE.md detectably stale. + Identifies the generator code that produced a stand, with no git dependency + and no clock. Any edit to the package changes it, which is the point: it is + what makes a committed PROFILE.md detectably stale. Tests are deliberately + outside the hash — they cannot change what a run writes. """ root = Path(__file__).resolve().parent digest = hashlib.sha256() @@ -256,10 +266,15 @@ def build_manifest( days = _days(env) window_start = anchor - _dt.timedelta(days=days - 1) - personas = [ - _persona(p) - for p in (*profiles.build_roster(dev_email), *profiles.build_other_tenant_roster()) - ] + # The second tenant's person appears here only when the seed run actually + # wrote them (`identity.py` reads the same switch). Advertising a fixture + # whose row does not exist would turn every test that declares + # `requires_seed("other_tenant_lead")` from a skip into a failure. + roster = list(profiles.build_roster(dev_email)) + if config.cross_tenant_fixture_enabled(env): + roster += profiles.build_other_tenant_roster() + + personas = [_persona(p) for p in roster] auth_mode = (env.get("AUTH_MODE") or "").strip().lower() issuer = (env.get("AUTHENTICATOR_OIDC_ISSUER") or "").strip() diff --git a/src/ingestion/tools/seed/insight_seed/preflight.py b/src/ingestion/tools/seed/insight_seed/preflight.py new file mode 100644 index 000000000..9c5847827 --- /dev/null +++ b/src/ingestion/tools/seed/insight_seed/preflight.py @@ -0,0 +1,371 @@ +"""Refuse to seed before anything is written. + +The seed writes to three places and takes minutes to finish. Every failure this +module reports would otherwise surface either as a stack trace halfway through a +run or — worse — as a successful run whose data nobody can see. It answers one +question per check and reports every answer at once. + +Two of the checks are safety gates rather than correctness ones: the seeder is +demo data, it now ships inside the same image a cluster uses for migrations, and +a CLI can point it at any stand. + +* Identity rows are additive, and every one the seeder writes carries a `reason` + starting with `seed.py ` — so a tenant holding person rows with any other + reason is a tenant somebody else's data lives in. +* Silver rows are NOT additive: the generators TRUNCATE each table before + writing, across every tenant, because a partially rewritten silver table + produces metrics that are wrong rather than absent. Rows there for any other + tenant would be destroyed, so their presence is a refusal too. + +Both are overridable with `SEED_FORCE=1`, which is the only way to say "yes, +clear it" out loud. +""" + +from __future__ import annotations + +import logging +import re +import uuid as uuid_mod +from collections.abc import Iterable, Sequence +from pathlib import Path + +import pymysql + +from . import config +from .config import SEED_REASON_PREFIX, ClickHouse, EnvContractError, MariaDb + +LOG = logging.getLogger("seed.preflight") + +STEPS = ("identity", "silver", "analytics") + + +class PreflightError(Exception): + """The stand cannot take this seed. Carries every reason, not the first.""" + + def __init__(self, problems: Sequence[str]) -> None: + self.problems = tuple(problems) + super().__init__( + "preflight refused to seed this stand:\n" + "\n".join(f" - {p}" for p in self.problems) + ) + + +def table_missing_problem(database: str, table: str, *, needed_for: str) -> str: + """The message for a database that does not hold a table the seed writes to. + + Names the database it looked in, because the whole class of bug this catches + is being pointed at the wrong one. + """ + return ( + f"`{database}` has no `{table}` table, so it is not the database that holds " + f"{needed_for}. Point the seed at the database whose migrations created it." + ) + + +def foreign_rows_problem(count: int, tenant: str, database: str) -> str: + return ( + f"tenant {tenant} already holds {count} `{database}.persons` row(s) this seeder " + f"did not write (their `reason` does not start with {SEED_REASON_PREFIX!r}), so " + "this stand carries identity data from somewhere else. Seed a tenant of your own, " + f"or set {config.FORCE_ENV}=1 to add demo people to this one anyway." + ) + + +def _table_exists(cur: pymysql.cursors.Cursor, database: str, table: str) -> bool: + cur.execute( + "SELECT 1 FROM information_schema.tables " + "WHERE table_schema = %s AND table_name = %s LIMIT 1", + (database, table), + ) + return cur.fetchone() is not None + + +def _count_foreign_persons(cur: pymysql.cursors.Cursor, tenant: str) -> int: + cur.execute( + "SELECT COUNT(*) FROM persons " + "WHERE insight_tenant_id = %s AND (reason IS NULL OR reason NOT LIKE %s)", + (uuid_mod.UUID(tenant).bytes, f"{SEED_REASON_PREFIX}%"), + ) + row = cur.fetchone() + return int(row[0]) if row else 0 + + +def _connect(target: MariaDb) -> pymysql.connections.Connection: + return pymysql.connect( + host=target.host, + port=target.port, + user=target.user, + password=target.password, + database=target.database, + autocommit=True, + cursorclass=pymysql.cursors.Cursor, + ) + + +def _check_identity(target: MariaDb, tenant: str, *, force: bool) -> list[str]: + try: + conn = _connect(target) + except pymysql.MySQLError as exc: + return [f"cannot reach MariaDB `{target.database}` at {target.host}:{target.port}: {exc}"] + + try: + cur = conn.cursor() + if not _table_exists(cur, target.database, "persons"): + return [ + table_missing_problem( + target.database, "persons", needed_for="the identity projection" + ) + ] + if force: + LOG.warning("%s=1: skipping the foreign-rows check", config.FORCE_ENV) + return [] + foreign = _count_foreign_persons(cur, tenant) + if foreign: + return [foreign_rows_problem(foreign, tenant, target.database)] + return [] + finally: + conn.close() + + +def _check_analytics(target: MariaDb) -> list[str]: + try: + conn = _connect(target) + except pymysql.MySQLError as exc: + return [f"cannot reach MariaDB `{target.database}` at {target.host}:{target.port}: {exc}"] + + try: + if not _table_exists(conn.cursor(), target.database, "metric_definitions"): + return [ + table_missing_problem( + target.database, + "metric_definitions", + needed_for="the analytics catalogue", + ) + ] + return [] + finally: + conn.close() + + +#: Column names a reset target may carry its tenant in, most specific first. +TENANT_COLUMNS = ("insight_tenant_id", "tenant_id") + +_IDENTIFIER = re.compile(r"^[a-z0-9_]+$") + + +def foreign_silver_problem(rows: int, tables: Sequence[tuple[str, int]], tenant: str) -> str: + """The message for reset targets holding another tenant's rows. + + The silver step TRUNCATEs before it inserts — a reset, not a merge, and not + tenant-scoped, because a partial silver table produces metrics that are + wrong rather than absent. That makes somebody else's rows in those tables a + refusal, not a warning. + """ + worst = ", ".join(f"{name} ({count})" for name, count in tables) + return ( + f"the silver step clears {rows} row(s) belonging to another tenant than {tenant} " + f"({worst}). It TRUNCATEs every table it writes, so those rows would be destroyed. " + f"Seed a stand of your own, or set {config.FORCE_ENV}=1 to clear them deliberately." + ) + + +def _tenant_columns(client: object) -> dict[tuple[str, str], str]: + """Which column carries the tenant, per reset target that has one.""" + from .generators.base import RESET_TARGETS + + schemas = sorted({schema for schema, _ in RESET_TARGETS}) + found = client.query( # type: ignore[attr-defined] + "SELECT database, table, name FROM system.columns " + "WHERE database IN {dbs:Array(String)} AND name IN {cols:Array(String)}", + parameters={"dbs": schemas, "cols": list(TENANT_COLUMNS)}, + ) + + by_target: dict[tuple[str, str], str] = {} + for database, table, column in found.result_rows: + target = (str(database), str(table)) + if target not in RESET_TARGETS: + continue + current = by_target.get(target) + # Most specific wins, so a table carrying both is counted once. + if current is None or TENANT_COLUMNS.index(str(column)) < TENANT_COLUMNS.index(current): + by_target[target] = str(column) + return by_target + + +def _foreign_silver_rows( + client: object, tenant: str, *, limit: int = 4 +) -> tuple[int, list[tuple[str, int]], list[tuple[str, str]]]: + """Foreign rows in the reset surface, plus the targets that cannot be judged. + + Scans exactly what the generators clear — `generators.base.RESET_TARGETS` — + rather than a name pattern: a pattern both misses targets in other databases + and refuses stands over tables the step never touches. + """ + from .generators.base import RESET_TARGETS + + tenant_column = _tenant_columns(client) + unattributable = [t for t in RESET_TARGETS if t not in tenant_column] + + # One statement rather than one per table: the answer is a single number plus + # the worst offenders. + # + # A NULL tenant is NOT foreign. Several generators carry the tenant in + # another column (`class_people` uses `workspace_id`) and leave the tenant + # column unset, so counting NULLs would make this seeder's own rows look like + # somebody else's and refuse every re-seed of a stand it seeded itself. + parts = [ + f"SELECT '{schema}.{table}' AS tbl, count() AS n FROM `{schema}`.`{table}` " + f"WHERE `{column}` IS NOT NULL AND `{column}` != {{tenant:String}}" + for (schema, table), column in sorted(tenant_column.items()) + if _IDENTIFIER.match(schema) and _IDENTIFIER.match(table) and _IDENTIFIER.match(column) + ] + if not parts: + return 0, [], unattributable + + result = client.query( # type: ignore[attr-defined] + f"SELECT tbl, n FROM ({' UNION ALL '.join(parts)}) WHERE n > 0 ORDER BY n DESC", + parameters={"tenant": tenant}, + ) + rows = [(str(row[0]), int(row[1])) for row in result.result_rows] + return sum(count for _, count in rows), rows[:limit], unattributable + + +def _check_clickhouse(target: ClickHouse, scripts: Path, tenant: str, *, force: bool) -> list[str]: + problems: list[str] = [] + + # Imported here, not at module scope: the identity step alone must not need + # a ClickHouse driver installed to run its own preflight. + import clickhouse_connect + + client = None + try: + client = clickhouse_connect.get_client( + host=target.host, + port=target.http_port, + username=target.user, + password=target.password, + ) + client.command("SELECT 1") + # The driver raises a wide family here (network, auth, protocol), and every + # one of them means the same thing to the operator reading this. + except Exception as exc: + problems.append(f"cannot reach ClickHouse at {target.url} as {target.user!r}: {exc}") + + if client is not None: + if force: + LOG.warning("%s=1: skipping the foreign-silver-rows check", config.FORCE_ENV) + else: + try: + rows, worst, unattributable = _foreign_silver_rows(client, tenant) + except Exception as exc: + # A stand with no silver database yet is the normal fresh case, + # and the placeholder script creates it. Anything else here is + # still worth reporting rather than swallowing. + LOG.info("foreign-silver-rows check skipped: %s", exc) + rows, worst, unattributable = 0, [], [] + if rows: + problems.append(foreign_silver_problem(rows, worst, tenant)) + elif unattributable: + # Said out loud rather than silently ignored: these targets are + # cleared too and carry no tenant, so nobody can tell whose rows + # they hold. In practice a stand holding foreign rows here also + # holds them in a tenant-bearing sibling (silver is derived from + # the same bronze), which is what the count above catches. + LOG.warning( + "reset targets with no tenant column, cleared without attribution: %s", + ", ".join(f"{schema}.{table}" for schema, table in unattributable), + ) + + missing = [ + name + for name in ("create-bronze-placeholders.sh", "apply-ch-migrations.sh") + if not (scripts / name).is_file() + ] + if missing: + problems.append( + f"{scripts} does not hold {', '.join(missing)} — the silver step runs the " + "ingestion tree's own DDL and gold-build scripts and cannot substitute for them." + ) + return problems + + +def check(env: dict[str, str] | None = None, steps: Iterable[str] = STEPS) -> None: + """Verify the environment and the stand, or raise with every problem found.""" + import os + + environ = dict(os.environ if env is None else env) + requested = tuple(steps) + problems: list[str] = [] + + try: + config.parse_tenant_id(environ) + except EnvContractError as exc: + problems.extend(exc.problems) + + analytics_db: str | None = None + if "analytics" in requested: + try: + analytics_db = config.parse_analytics_database(environ) + except EnvContractError as exc: + problems.extend(exc.problems) + + # Both steps that build the roster need the persona it is built around, and + # `profiles.get_dev_user_email` only complains once the step is already + # running — which for silver is after it has applied DDL. + if "identity" in requested or "silver" in requested: + try: + config.parse_dev_user_email(environ) + except EnvContractError as exc: + problems.extend(exc.problems) + + # One try per reader: sharing one would let a malformed first value hide the + # second, and reporting the whole list in one run is the point of this module. + for reader in ( + config.cross_tenant_fixture_enabled, + config.force_enabled, + config.parse_anchor_date, + config.parse_seed_days, + ): + try: + reader(environ) + except EnvContractError as exc: + problems.extend(exc.problems) + + if problems: + # Nothing below can run without these, and every one of them is + # answerable without touching a database. + raise PreflightError(problems) + + # Re-read rather than carried down from the try above: past this point it is + # a plain `str`, and nothing has to reason about how it got that way. + tenant = config.parse_tenant_id(environ) + + if "identity" in requested: + problems += _check_identity( + config.parse_mariadb(environ, database=config.parse_identity_database(environ)), + tenant, + force=config.force_enabled(environ), + ) + + if analytics_db is not None: + problems += _check_analytics(config.parse_mariadb(environ, database=analytics_db)) + + if "silver" in requested: + # Imported inside the branch, not at the top: importing `silver` pulls the + # ClickHouse driver, which the identity step must not need. + from .silver import _ingestion_scripts_dir + + problems += _check_clickhouse( + config.parse_clickhouse(environ), + _ingestion_scripts_dir(), + tenant, + force=config.force_enabled(environ), + ) + + if problems: + raise PreflightError(problems) + + LOG.info( + "preflight ok: tenant=%s steps=%s", + tenant, + ",".join(requested), + ) diff --git a/deploy/seed/profile_md.py b/src/ingestion/tools/seed/insight_seed/profile_md.py similarity index 92% rename from deploy/seed/profile_md.py rename to src/ingestion/tools/seed/insight_seed/profile_md.py index bcb99e0b5..7bfbb5c06 100644 --- a/deploy/seed/profile_md.py +++ b/src/ingestion/tools/seed/insight_seed/profile_md.py @@ -1,4 +1,4 @@ -"""Render `deploy/seed/PROFILE.md` from a manifest document. +"""Render the seeder's committed `PROFILE.md` from a manifest document. PROFILE.md is the human-readable companion to `manifest.json`: it tells a person (or an agent planning a test) what a seeded stand actually contains — @@ -19,12 +19,13 @@ from pathlib import Path from typing import Any -REGEN_COMMAND = "python3 deploy/seed/render_profile.py" -CHECK_COMMAND = "python3 deploy/seed/render_profile.py --check" +REGEN_COMMAND = "python3 -m insight_seed.render_profile" +CHECK_COMMAND = "python3 -m insight_seed.render_profile --check" def profile_path() -> Path: - return Path(__file__).resolve().parent / "PROFILE.md" + """The committed profile page, at the tool root beside the README.""" + return Path(__file__).resolve().parents[1] / "PROFILE.md" def _table(headers: list[str], rows: list[list[str]]) -> list[str]: @@ -49,11 +50,11 @@ def render_profile(doc: dict[str, Any]) -> str: "", + " Content is derived from insight_seed/manifest.py + profiles.py. -->", "", "# Seed Profile", "", - "What a stand seeded by `deploy/seed` contains. Generated from the same", + "What a stand seeded by the seeder contains. Generated from the same", "builder that writes `manifest.json`, so the two cannot disagree.", "", "## Stand summary", @@ -144,7 +145,7 @@ def render_profile(doc: dict[str, Any]) -> str: "", "Rows the product provisions by operator or migration, so no endpoint", "creates them and no test fixture can either — the suite holds no", - "database connection. Seeded by `deploy/seed/analytics.py` and named", + "database connection. Seeded by `insight_seed/analytics.py` and named", "here so a test reads the name rather than hardcoding one.", "", ] @@ -185,7 +186,7 @@ def render_profile(doc: dict[str, Any]) -> str: "", "A test suite consuming this manifest therefore asserts no metric", "values. That is a visible gap; a populated-but-guessed set would be a", - "silent wrong answer. See `deploy/seed/golden_metrics.py` for the", + "silent wrong answer. See `insight_seed/golden_metrics.py` for the", "criteria an entry must meet before it is added.", ] diff --git a/deploy/seed/profiles.py b/src/ingestion/tools/seed/insight_seed/profiles.py similarity index 100% rename from deploy/seed/profiles.py rename to src/ingestion/tools/seed/insight_seed/profiles.py diff --git a/deploy/seed/render_profile.py b/src/ingestion/tools/seed/insight_seed/render_profile.py similarity index 88% rename from deploy/seed/render_profile.py rename to src/ingestion/tools/seed/insight_seed/render_profile.py index 9f4b75f2c..2d79d4fd5 100644 --- a/deploy/seed/render_profile.py +++ b/src/ingestion/tools/seed/insight_seed/render_profile.py @@ -1,8 +1,10 @@ -#!/usr/bin/env python3 """Generate (or verify) the committed `PROFILE.md`. - python3 deploy/seed/render_profile.py # write PROFILE.md - python3 deploy/seed/render_profile.py --check # fail if it is stale + python3 -m insight_seed.render_profile # write PROFILE.md + python3 -m insight_seed.render_profile --check # fail if it is stale + +Run from the tool directory (`src/ingestion/tools/seed`), or with it on +`PYTHONPATH`. Both render against `manifest.CANONICAL_ENV`, not the ambient environment, so the committed page is a function of committed bytes: it does not embed one @@ -21,12 +23,8 @@ import argparse import difflib import sys -from pathlib import Path - -sys.path.insert(0, str(Path(__file__).resolve().parent)) -import manifest -import profile_md +from . import manifest, profile_md def _rendered() -> str: diff --git a/deploy/seed/silver.py b/src/ingestion/tools/seed/insight_seed/silver.py similarity index 83% rename from deploy/seed/silver.py rename to src/ingestion/tools/seed/insight_seed/silver.py index cdd5c7058..b3beaff80 100644 --- a/deploy/seed/silver.py +++ b/src/ingestion/tools/seed/insight_seed/silver.py @@ -41,13 +41,13 @@ import clickhouse_connect -from generators import ai, collab, crm, git, hr, people, support, task -from profiles import build_roster, get_dev_user_email +from . import config +from .generators import ai, collab, crm, git, hr, people, support, task +from .generators.base import seed_days +from .profiles import build_roster, get_dev_user_email LOG = logging.getLogger("seed.silver") -DEFAULT_DAYS = 60 - def _ingestion_scripts_dir() -> Path: """Locate src/ingestion/scripts — no env knobs needed. @@ -56,45 +56,40 @@ def _ingestion_scripts_dir() -> Path: at /ingestion (docker-compose.yml `seed-sample.volumes`), mirroring the toolbox image layout the scripts resolve their relative paths against (apply-ch-migrations.sh cd's into ../dbt). Host runs resolve it relative - to this file: deploy/seed lives in the same repo as src/ingestion, two - levels below the root. + to this file: this package lives inside the ingestion tree it seeds. """ mounted = Path("/ingestion/scripts") if mounted.is_dir(): return mounted - # parents[2] = repo root (silver.py -> seed -> deploy -> root). - return Path(__file__).resolve().parents[2] / "src/ingestion/scripts" + # parents[3] = src/ingestion (silver -> insight_seed -> seed -> tools). + return Path(__file__).resolve().parents[3] / "scripts" def _script_env() -> dict[str, str]: """Env for the ingestion shell scripts (create-bronze-placeholders.sh, apply-ch-migrations.sh) — CLICKHOUSE_URL/USER/PASSWORD/DATABASE per lib/ch-exec.sh + apply-ch-migrations.sh's own asserts.""" - host = os.environ.get("CLICKHOUSE_HOST", "clickhouse") - port = os.environ.get("CLICKHOUSE_HTTP_PORT", "8123") + target = config.parse_clickhouse(os.environ) return { **os.environ, - "CLICKHOUSE_URL": f"http://{host}:{port}", - "CLICKHOUSE_USER": os.environ.get("CLICKHOUSE_USER", "insight"), - "CLICKHOUSE_PASSWORD": os.environ.get("CLICKHOUSE_PASSWORD", "insight-local"), - "CLICKHOUSE_DATABASE": os.environ.get("CLICKHOUSE_DATABASE", "insight"), + "CLICKHOUSE_URL": target.url, + "CLICKHOUSE_USER": target.user, + "CLICKHOUSE_PASSWORD": target.password, + "CLICKHOUSE_DATABASE": target.database, } def _ch_client() -> clickhouse_connect.driver.client.Client: - host = os.environ.get("CLICKHOUSE_HOST", "clickhouse") - port = int(os.environ.get("CLICKHOUSE_HTTP_PORT", "8123")) - user = os.environ.get("CLICKHOUSE_USER", "insight") - pwd = os.environ.get("CLICKHOUSE_PASSWORD", "insight-local") + target = config.parse_clickhouse(os.environ) # Views (gold) are created by apply-ch-migrations.sh, not this client; # the compose CH ships join_use_nulls=1 as a profile default # (deploy/compose/clickhouse-user-defaults.xml) so those CREATE VIEWs # type-check server-side. This client only INSERTs silver rows. return clickhouse_connect.get_client( - host=host, - port=port, - username=user, - password=pwd, + host=target.host, + port=target.http_port, + username=target.user, + password=target.password, ) @@ -110,7 +105,7 @@ def apply_create_bronze_placeholders() -> None: raise FileNotFoundError( f"placeholders script not found at {script}. In compose, the " "seed-sample container must mount /ingestion; on a host run, " - "deploy/seed must sit inside the insight repo next to src/ingestion." + "this package must sit inside the ingestion tree (src/ingestion/tools/seed)." ) subprocess.run(["bash", str(script)], env=_script_env(), check=True) LOG.info("placeholders: %s applied", script.name) @@ -130,7 +125,7 @@ def apply_ch_migrations() -> None: raise FileNotFoundError( f"migrations script not found at {script}. In compose, the " "seed-sample container must mount /ingestion; on a host run, " - "deploy/seed must sit inside the insight repo next to src/ingestion." + "this package must sit inside the ingestion tree (src/ingestion/tools/seed)." ) subprocess.run(["bash", str(script)], env=_script_env(), check=True) LOG.info("migrations + gold: %s applied", script.name) @@ -140,10 +135,14 @@ def generate_rows( client: clickhouse_connect.driver.client.Client, ) -> None: """Populate silver tables with per-team activity for the demo roster.""" - tenant_uuid = os.environ.get("TENANT_DEFAULT_ID", "00000000-df51-5b42-9538-d2b56b7ee953") + tenant_uuid = config.parse_tenant_id(os.environ) dev_email = get_dev_user_email() roster = build_roster(dev_email) - days = int(os.environ.get("SEED_DAYS", DEFAULT_DAYS)) + # The generators' own reader, not a second copy of it: they date every row + # from this window, and a `SEED_DAYS` the two disagreed on would put rows + # outside the range this function logs. It also treats an empty value as + # "unset", which a rendered Job manifest passes for a window nobody pinned. + days = seed_days() LOG.info( "generating silver rows: tenant=%s days=%d persons=%d", tenant_uuid, diff --git a/deploy/seed/pyproject.toml b/src/ingestion/tools/seed/pyproject.toml similarity index 65% rename from deploy/seed/pyproject.toml rename to src/ingestion/tools/seed/pyproject.toml index 0ec80006f..e80ade156 100644 --- a/deploy/seed/pyproject.toml +++ b/src/ingestion/tools/seed/pyproject.toml @@ -1,7 +1,7 @@ -# Package metadata, dependencies, and tool config for the Insight -# sample-data seeder. Lives next to the code so `ruff check .` and `mypy .` -# work from the dir, and `pip install .` builds the flat module + generators -# package straight from here. +# Package metadata, dependencies, and tool config for the Insight sample-data +# seeder. Lives at the tool root so `ruff check .` and `mypy .` cover both the +# `insight_seed` package and its `tests/`, and `pip install .` builds the +# package (only the package — tests are not shipped). [build-system] requires = ["setuptools>=61"] @@ -10,8 +10,12 @@ build-backend = "setuptools.build_meta" [project] name = "insight-seed" version = "0.1.0" -description = "Insight sample-data seeder for the local docker-compose stack." -requires-python = ">=3.13" +description = "Insight sample-data seeder for the compose stack and Kubernetes test stands." +# 3.12, not 3.13: this package ships inside the toolbox image +# (tools/toolbox/Dockerfile), which is python:3.12-slim, and the rest of the +# ingestion tree pins the same direction (tests/connectors caps at <3.13). No +# 3.13-only syntax or API is used here, so the floor costs nothing. +requires-python = ">=3.12" dependencies = [ "PyMySQL==1.2.0", "clickhouse-connect==1.6.0", @@ -32,16 +36,19 @@ dev = [ "types-PyMySQL==1.2.0.20260724", ] -# Flat layout: top-level modules plus the generators/ package. -[tool.setuptools] -py-modules = ["seed", "silver", "identity", "profiles"] +# The entry point every runner uses: `python3 -m insight_seed ` from this +# directory, which is also what the container images and the Kubernetes Job run. +# The console script is for an installed copy. +[project.scripts] +insight-seed = "insight_seed.__main__:main" +# One importable package; `tests/` stays out of the distribution. [tool.setuptools.packages.find] -include = ["generators*"] +include = ["insight_seed*"] [tool.ruff] line-length = 100 -target-version = "py313" +target-version = "py312" [tool.ruff.lint] # Pragmatic strict set: pycodestyle, pyflakes, isort, bugbear, comprehensions, @@ -66,7 +73,7 @@ ignore = [ ] [tool.mypy] -python_version = "3.13" +python_version = "3.12" strict = true warn_unused_ignores = true warn_redundant_casts = true diff --git a/src/ingestion/tools/seed/seed-job.yaml.tpl b/src/ingestion/tools/seed/seed-job.yaml.tpl new file mode 100644 index 000000000..559c1031a --- /dev/null +++ b/src/ingestion/tools/seed/seed-job.yaml.tpl @@ -0,0 +1,145 @@ +# One-shot Job that runs this seeder against a Kubernetes stand. +# +# Rendered by `seed-stand.sh` with envsubst and applied to the cluster; that +# script resolves every variable below from the stand itself and refuses to +# render when one is empty, so this file is also the reference manifest — the +# thing that runs IS the thing you read, and the two cannot drift. +# +# Expected variables (all required, all exported by seed-stand.sh): +# SEED_JOB_NAME Job name, unique per run +# SEED_NAMESPACE namespace holding the release and its creds Secret +# SEED_IMAGE toolbox image carrying tools/seed (the chart's own pin) +# SEED_STEP identity | silver | analytics | all +# SEED_DEADLINE_SECONDS wall-clock ceiling for the pod +# SEED_DB_SECRET Secret with mariadb-password + clickhouse-password +# SEED_MARIADB_HOST/_PORT/_USER +# SEED_IDENTITY_DB database holding `persons` +# SEED_ANALYTICS_DB database holding `metric_definitions` +# SEED_CLICKHOUSE_HOST/_HTTP_PORT/_USER/_DATABASE +# SEED_TENANT_ID tenant every seeded row is scoped to +# SEED_DEV_USER_EMAIL persona the dev-lead login resolves to +# SEED_AUTH_MODE keycloak | fakeidp — which personas get login rows +# SEED_IDP_SOURCE_TYPE identity source_type those rows are written under +# SEED_CROSS_TENANT 0 | 1 — write the second-tenant refusal fixture +# SEED_FORCE 0 | 1 — seed a tenant holding foreign person rows +# SEED_WINDOW_DAYS activity-window length, empty for the seeder's own +# SEED_ANCHOR_DATE last day of activity, empty for the seeder's own +# SEED_PULL_SECRETS YAML flow sequence of pull secrets, `[]` for none +apiVersion: batch/v1 +kind: Job +metadata: + name: ${SEED_JOB_NAME} + namespace: ${SEED_NAMESPACE} + labels: + app.kubernetes.io/name: insight-seed + app.kubernetes.io/component: sample-data +spec: + # A failed seed needs reading, not retrying: every failure it can produce is + # deterministic (wrong database, wrong tenant, unreachable dependency), and + # preflight has already refused the ones that are answerable up front. + backoffLimit: 0 + activeDeadlineSeconds: ${SEED_DEADLINE_SECONDS} + # Long enough to read the logs of a finished run, short enough not to litter. + ttlSecondsAfterFinished: 3600 + template: + metadata: + labels: + app.kubernetes.io/name: insight-seed + app.kubernetes.io/component: sample-data + spec: + restartPolicy: Never + # Suppress the legacy `_SERVICE_HOST/PORT` env vars kubelet injects + # for every Service in the namespace — several collide by name with the + # seeder's own variables. + enableServiceLinks: false + # The same secrets the release's own Jobs use for this image; `[]` on a + # stand that pulls it anonymously. Without them a private toolbox image + # leaves the pod in ImagePullBackOff. + imagePullSecrets: ${SEED_PULL_SECRETS} + containers: + - name: seed + image: ${SEED_IMAGE} + # IfNotPresent so a locally built image can be tried on a local + # cluster without pushing it to a registry first. + imagePullPolicy: IfNotPresent + # Run as a module from the tool directory, which puts the package's + # parent on sys.path without installing anything into the image. + command: [bash, -c] + args: + - exec python -m insight_seed ${SEED_STEP} + workingDir: /ingestion/tools/seed + env: + # ── MariaDB ────────────────────────────────────────────────── + # The app user, not root: the umbrella grants it ALL on the + # identity database and it owns the product database, so the seed + # needs no privilege the services do not already hold. + - name: MARIADB_HOST + value: "${SEED_MARIADB_HOST}" + - name: MARIADB_PORT + value: "${SEED_MARIADB_PORT}" + - name: MARIADB_USER + value: "${SEED_MARIADB_USER}" + - name: MARIADB_PASSWORD + valueFrom: + secretKeyRef: + name: ${SEED_DB_SECRET} + key: mariadb-password + - name: MARIADB_DB + value: "${SEED_IDENTITY_DB}" + # Required by the seeder, and NOT the compose convention: a + # chart-deployed stand keeps the catalogue tables in the product + # database rather than one of their own. + - name: MARIADB_ANALYTICS_DB + value: "${SEED_ANALYTICS_DB}" + # ── ClickHouse ─────────────────────────────────────────────── + - name: CLICKHOUSE_HOST + value: "${SEED_CLICKHOUSE_HOST}" + - name: CLICKHOUSE_HTTP_PORT + value: "${SEED_CLICKHOUSE_HTTP_PORT}" + - name: CLICKHOUSE_USER + value: "${SEED_CLICKHOUSE_USER}" + - name: CLICKHOUSE_PASSWORD + valueFrom: + secretKeyRef: + name: ${SEED_DB_SECRET} + key: clickhouse-password + - name: CLICKHOUSE_DATABASE + value: "${SEED_CLICKHOUSE_DATABASE}" + # ── Seed semantics ─────────────────────────────────────────── + - name: TENANT_DEFAULT_ID + value: "${SEED_TENANT_ID}" + - name: DEV_USER_EMAIL + value: "${SEED_DEV_USER_EMAIL}" + - name: AUTH_MODE + value: "${SEED_AUTH_MODE}" + - name: IDP_SOURCE_TYPE + value: "${SEED_IDP_SOURCE_TYPE}" + # Off on a cluster stand: a second tenant makes + # identity-resolution's scheduled projection abort on its + # tenant-mismatch guard. + - name: SEED_CROSS_TENANT_FIXTURE + value: "${SEED_CROSS_TENANT}" + - name: SEED_FORCE + value: "${SEED_FORCE}" + # Empty means "whatever the seeder documents", which is a window + # ending yesterday. Pin both to reproduce a dataset exactly. + - name: SEED_DAYS + value: "${SEED_WINDOW_DAYS}" + - name: SEED_ANCHOR_DATE + value: "${SEED_ANCHOR_DATE}" + # dbt (run by the silver step's gold build) writes target/, logs/ + # and ~/.dbt under whatever it is given; the toolbox image owns + # /ingestion, but keep the scratch paths explicit. + - name: DBT_TARGET_PATH + value: /tmp/dbt-target + - name: DBT_LOG_PATH + value: /tmp/dbt-logs + - name: HOME + value: /tmp + resources: + requests: + cpu: 250m + memory: 512Mi + limits: + cpu: "2" + memory: 2Gi diff --git a/src/ingestion/tools/seed/seed-stand.sh b/src/ingestion/tools/seed/seed-stand.sh new file mode 100755 index 000000000..94e958d05 --- /dev/null +++ b/src/ingestion/tools/seed/seed-stand.sh @@ -0,0 +1,447 @@ +#!/usr/bin/env bash +# Seed a Kubernetes stand with the demo organisation and its activity. +# +# Every coordinate the seeder needs is already in the cluster, so this script +# reads them from the stand rather than asking an operator to copy them: +# +# ConfigMap -platform MariaDB + ClickHouse hosts, ports, +# users, and the product database that +# holds the analytics catalogue +# Secret insight-identity-resolution-* the stand's tenant, and the database +# holding `persons` +# helm get values the toolbox image the release pins +# +# Credentials are never read: the rendered Job references the release's own +# database Secret by key, so nothing sensitive passes through this shell. +# +# Nothing is defaulted. A value that can neither be discovered nor supplied is +# a hard error naming the flag that fixes it. +set -euo pipefail + +SCRIPT_DIR="$(cd "$(dirname "${BASH_SOURCE[0]}")" && pwd)" +JOB_TEMPLATE="$SCRIPT_DIR/seed-job.yaml.tpl" + +# RULE-DEFAULTS-OK: `all` is the seeder's own documented default command and +# every step is idempotent, so seeding everything is the safe reading of "seed +# this stand"; the flag exists to do LESS than that. +STEP="all" +# RULE-DEFAULTS-OK: wall-clock ceiling for a pod, not a config input — the gold +# rebuild dominates a full run and an hour is well beyond it. +DEADLINE_SECONDS="3600" + +NAMESPACE="" +RELEASE="" +DEV_EMAIL="" +TENANT="" +IMAGE="" +ANALYTICS_DB="" +IDENTITY_DB="" +DB_SECRET="" +PULL_SECRETS="" +AUTH_MODE="" +IDP_SOURCE_TYPE="" +WINDOW_DAYS="" +ANCHOR_DATE="" +CROSS_TENANT="0" +FORCE="0" +DRY_RUN=0 +FOLLOW=1 + +usage() { + cat <<'USAGE' +Usage: seed-stand.sh -n --email
[options] + +Runs the demo-data seeder as a one-shot Job on a chart-deployed stand, using the +toolbox image the release already pins. + +Required: + -n, --namespace namespace the Insight release runs in + --email
persona the dev-lead login resolves to. A user with + this email must already exist in the stand's IdP — + the authenticator resolves people by the email claim. + +Discovered from the stand (pass a flag only to override): + --release helm release name [default: same as -n] + --tenant tenant every seeded row is scoped to + --image toolbox image to run + --analytics-db database holding metric_definitions + --identity-db database holding persons + --db-secret Secret holding mariadb-password + clickhouse-password + --pull-secret image-pull Secret (default: the release's own) + --auth-mode keycloak | fakeidp — which personas get login rows + --idp-source-type identity source_type those rows are written under + +Seed options: + --step identity | silver | analytics | all [default: all] + --days activity-window length in days + --anchor last day carrying activity (YYYY-MM-DD) + --cross-tenant also write the second-tenant refusal fixture + --force seed a tenant that already holds foreign person rows + --deadline pod wall-clock ceiling [default: 3600] + +Output: + --dry-run print the rendered Job manifest and exit + --no-follow apply the Job without following its logs + -h, --help this text + +Examples: + seed-stand.sh -n insight --email you@example.com --dry-run + seed-stand.sh -n insight --email you@example.com + seed-stand.sh -n insight --email you@example.com --step identity +USAGE +} + +die() { + echo "ERROR: $*" >&2 + exit 1 +} + +need() { + command -v "$1" >/dev/null 2>&1 || die "$1 is required but not on PATH." +} + +while [[ $# -gt 0 ]]; do + case "$1" in + -n|--namespace) NAMESPACE="${2:?--namespace needs a value}"; shift 2 ;; + --release) RELEASE="${2:?--release needs a value}"; shift 2 ;; + --email) DEV_EMAIL="${2:?--email needs a value}"; shift 2 ;; + --tenant) TENANT="${2:?--tenant needs a value}"; shift 2 ;; + --image) IMAGE="${2:?--image needs a value}"; shift 2 ;; + --analytics-db) ANALYTICS_DB="${2:?--analytics-db needs a value}"; shift 2 ;; + --identity-db) IDENTITY_DB="${2:?--identity-db needs a value}"; shift 2 ;; + --db-secret) DB_SECRET="${2:?--db-secret needs a value}"; shift 2 ;; + --pull-secret) PULL_SECRETS="[{\"name\":\"${2:?--pull-secret needs a value}\"}]"; shift 2 ;; + --auth-mode) AUTH_MODE="${2:?--auth-mode needs a value}"; shift 2 ;; + --idp-source-type) IDP_SOURCE_TYPE="${2:?--idp-source-type needs a value}"; shift 2 ;; + --step) STEP="${2:?--step needs a value}"; shift 2 ;; + --days) WINDOW_DAYS="${2:?--days needs a value}"; shift 2 ;; + --anchor) ANCHOR_DATE="${2:?--anchor needs a value}"; shift 2 ;; + --deadline) DEADLINE_SECONDS="${2:?--deadline needs a value}"; shift 2 ;; + --cross-tenant) CROSS_TENANT="1"; shift ;; + --force) FORCE="1"; shift ;; + --dry-run) DRY_RUN=1; shift ;; + --no-follow) FOLLOW=0; shift ;; + -h|--help) usage; exit 0 ;; + *) usage >&2; die "unknown argument: $1" ;; + esac +done + +need kubectl +need envsubst +[[ -f "$JOB_TEMPLATE" ]] || die "Job template not found at $JOB_TEMPLATE." +[[ -n "$NAMESPACE" ]] || { usage >&2; die "--namespace is required."; } +[[ -n "$DEV_EMAIL" ]] || { usage >&2; die "--email is required."; } +case "$STEP" in + identity|silver|analytics|all) ;; + *) die "--step must be one of identity, silver, analytics, all (got '$STEP')." ;; +esac + +# The release name is the one thing a namespace cannot answer, so the common +# case (release named after its namespace) is assumed and reported, and the flag +# overrides it. Every value read below is verified to exist, so a wrong guess +# fails naming --release rather than seeding something unintended. +if [[ -z "$RELEASE" ]]; then + RELEASE="$NAMESPACE" +fi + +echo "==> stand: namespace=$NAMESPACE release=$RELEASE" + +# ── Discovery ─────────────────────────────────────────────────────────────── +# One ConfigMap read per value keeps each failure attributable; kubectl's +# jsonpath returns empty rather than failing on a missing key, so each result is +# checked against the flag that would supply it. +platform_cm="${RELEASE}-platform" +kubectl -n "$NAMESPACE" get configmap "$platform_cm" >/dev/null 2>&1 || die \ + "ConfigMap $platform_cm not found in namespace $NAMESPACE. It is generated by + the umbrella chart and names every infrastructure coordinate this Job needs. + Check --release (currently '$RELEASE')." + +cm_value() { + kubectl -n "$NAMESPACE" get configmap "$platform_cm" -o "jsonpath={.data.$1}" +} + +secret_value() { + # $1 = secret, $2 = key. Missing secret or key yields an empty string, which + # every caller treats as "not discovered". + kubectl -n "$NAMESPACE" get secret "$1" -o "jsonpath={.data.$2}" 2>/dev/null \ + | { base64 --decode 2>/dev/null || true; } +} + +MARIADB_HOST="$(cm_value MARIADB_HOST)" +MARIADB_PORT="$(cm_value MARIADB_PORT)" +MARIADB_USER="$(cm_value MARIADB_USERNAME)" +CLICKHOUSE_HOST="$(cm_value CLICKHOUSE_HOST)" +CLICKHOUSE_HTTP_PORT="$(cm_value CLICKHOUSE_PORT)" +CLICKHOUSE_USER="$(cm_value CLICKHOUSE_USER)" +CLICKHOUSE_DATABASE="$(cm_value CLICKHOUSE_DATABASE)" + +# The product database is where a chart-deployed stand's analytics migrations +# created the catalogue tables. Preflight verifies that inside the Job. +if [[ -z "$ANALYTICS_DB" ]]; then + ANALYTICS_DB="$(cm_value MARIADB_DATABASE)" +fi + +# The seeder speaks HTTP to ClickHouse (see config.ClickHouse.url), so a stand +# fronting it with TLS would be dialled on the wrong scheme — said now rather +# than as a connection error inside the Job. +platform_ch_url="$(cm_value CLICKHOUSE_URL)" +case "$platform_ch_url" in + https://*) + die "this stand's ClickHouse is $platform_ch_url, and the seeder speaks plain HTTP only. + Point it at an HTTP endpoint with --release/--image overrides, or teach + config.ClickHouse.url a scheme first." ;; +esac + +ir_secret="insight-identity-resolution-config" +# Told apart from a missing key: an unreadable Secret is an access problem, and +# reporting it as "pass --tenant" would send the operator after the wrong thing. +if ! kubectl -n "$NAMESPACE" get secret "$ir_secret" -o name >/dev/null 2>&1; then + echo "WARNING: Secret $ir_secret is absent or not readable in namespace $NAMESPACE;" >&2 + echo " the tenant and identity database cannot be discovered from it." >&2 +fi + +if [[ -z "$TENANT" ]]; then + # The gear name is spelled with hyphens on some chart versions and + # underscores on others; both are the same value. + TENANT="$(secret_value "$ir_secret" 'APP__gears__identity-resolution__config__tenant_default_id')" + if [[ -z "$TENANT" ]]; then + TENANT="$(secret_value "$ir_secret" 'APP__gears__identity_resolution__config__tenant_default_id')" + fi +fi + +if [[ -z "$IDENTITY_DB" ]]; then + for key in 'APP__gears__identity-resolution__config__database_url' \ + 'APP__gears__identity_resolution__config__database_url'; do + url="$(secret_value "$ir_secret" "$key")" + # Database name is the last path segment, minus any query string. The URL + # also carries a password, so it is never echoed — only this segment leaves + # the expansion. + if [[ -n "$url" ]]; then + candidate="${url##*/}" + candidate="${candidate%%\?*}" + # A URL with no path segment leaves the AUTHORITY here — which carries the + # password. Only an identifier-shaped result is a database name; anything + # else is dropped so no credential can reach the rendered manifest. + if [[ "$candidate" =~ ^[A-Za-z0-9_]+$ ]]; then + IDENTITY_DB="$candidate" + break + fi + fi + done +fi + +if [[ -z "$IDP_SOURCE_TYPE" ]]; then + IDP_SOURCE_TYPE="$(secret_value insight-authenticator-config \ + 'APP__gears__authenticator__config__idp__source_type')" +fi + +# Which personas get a login-bootstrap row follows from the IdP: a Keycloak +# realm can authenticate the whole roster, the fakeidp fixture only the dev +# lead. Rows for people a realm does not hold are inert observations. +if [[ -z "$AUTH_MODE" ]]; then + if [[ "$IDP_SOURCE_TYPE" == "fakeidp" ]]; then + AUTH_MODE="fakeidp" + elif [[ -n "$IDP_SOURCE_TYPE" ]]; then + AUTH_MODE="keycloak" + fi +fi + +if [[ -z "$IMAGE" || -z "$PULL_SECRETS" ]]; then + need helm + need jq + # `|| true`: a helm failure (no such release, no permission) must reach the + # missing-values report below naming --image, not kill the script through + # pipefail with nothing said. + release_values="$(helm get values "$RELEASE" -n "$NAMESPACE" -a -o json 2>/dev/null || true)" + if [[ -n "$release_values" ]]; then + if [[ -z "$IMAGE" ]]; then + IMAGE="$(printf '%s' "$release_values" | jq -r '.ingestion.toolboxImage // empty')" + fi + if [[ -z "$PULL_SECRETS" ]]; then + # Rendered as a YAML flow sequence so the template needs no conditional: + # the release's own secrets, or an empty list. + PULL_SECRETS="$(printf '%s' "$release_values" \ + | jq -c '[(.global.imagePullSecrets // [])[] | if type == "string" then {name: .} else . end]')" + fi + fi +fi +[[ -n "$PULL_SECRETS" ]] || PULL_SECRETS="[]" + +if [[ -z "$DB_SECRET" ]]; then + # Discovered by looking for BOTH keys the Job references rather than by + # assuming the name: a Secret carrying only one of them would render a Job + # that fails at pod creation on the missing key. + for candidate in insight-db-creds "${RELEASE}-db-creds"; do + have_maria="$(kubectl -n "$NAMESPACE" get secret "$candidate" \ + -o 'jsonpath={.data.mariadb-password}' 2>/dev/null || true)" + have_ch="$(kubectl -n "$NAMESPACE" get secret "$candidate" \ + -o 'jsonpath={.data.clickhouse-password}' 2>/dev/null || true)" + if [[ -n "$have_maria" && -n "$have_ch" ]]; then + DB_SECRET="$candidate" + break + fi + done +fi + +# ── Every value, or the flag that supplies it ─────────────────────────────── +# Accumulated as newline-delimited text rather than an array: `${#arr[@]}` on an +# empty array is an unbound-variable error under `set -u` in bash 3.2, which is +# what /bin/bash still is on macOS. +missing="" +missing_count=0 +check() { + # $1 = value, $2 = what it is, $3 = flag that overrides it + if [[ -z "$1" ]]; then + missing="${missing} - $2 — pass $3"$'\n' + missing_count=$((missing_count + 1)) + fi +} +check "$MARIADB_HOST" "MariaDB host (ConfigMap $platform_cm, MARIADB_HOST)" "--release" +check "$MARIADB_PORT" "MariaDB port (ConfigMap $platform_cm, MARIADB_PORT)" "--release" +check "$MARIADB_USER" "MariaDB user (ConfigMap $platform_cm, MARIADB_USERNAME)" "--release" +check "$CLICKHOUSE_HOST" "ClickHouse host (ConfigMap $platform_cm)" "--release" +check "$CLICKHOUSE_HTTP_PORT" "ClickHouse HTTP port (ConfigMap $platform_cm)" "--release" +check "$CLICKHOUSE_USER" "ClickHouse user (ConfigMap $platform_cm)" "--release" +check "$CLICKHOUSE_DATABASE" "ClickHouse database (ConfigMap $platform_cm)" "--release" +check "$ANALYTICS_DB" "analytics catalogue database" "--analytics-db" +check "$IDENTITY_DB" "identity database (Secret $ir_secret, database_url)" "--identity-db" +check "$TENANT" "stand tenant (Secret $ir_secret, tenant_default_id)" "--tenant" +check "$IMAGE" "toolbox image (helm values ingestion.toolboxImage)" "--image" +check "$DB_SECRET" "database-credentials Secret" "--db-secret" +check "$IDP_SOURCE_TYPE" \ + "the identity source_type the stand's logins resolve under. Newer charts + publish it as authenticator.oidc.sourceType; a chart that predates that field + resolves logins by email instead, and any stable label (e.g. 'keycloak') is + then correct as long as it matches what the authenticator will use later" \ + "--idp-source-type" +check "$AUTH_MODE" \ + "which personas get a login-bootstrap row ('keycloak' = the whole roster, + 'fakeidp' = the dev lead only). Derived from the source_type above" \ + "--auth-mode" + +if [[ "$missing_count" -gt 0 ]]; then + echo "ERROR: could not resolve $missing_count value(s) from namespace $NAMESPACE:" >&2 + printf '%s' "$missing" >&2 + exit 1 +fi + +echo "==> tenant: $TENANT" +echo "==> databases: identity=$IDENTITY_DB analytics=$ANALYTICS_DB clickhouse=$CLICKHOUSE_DATABASE" +echo "==> image: $IMAGE" +echo "==> idp: auth_mode=$AUTH_MODE source_type=$IDP_SOURCE_TYPE dev_user=$DEV_EMAIL" + +# ── Render ────────────────────────────────────────────────────────────────── +# A name per run: a Job's pod spec is immutable, so reusing one name would make +# a re-run fail on an unrelated conflict instead of seeding. +job_name="insight-seed-${STEP}-$(date -u +%Y%m%d%H%M%S)" + +export SEED_JOB_NAME="$job_name" +export SEED_NAMESPACE="$NAMESPACE" +export SEED_IMAGE="$IMAGE" +export SEED_STEP="$STEP" +export SEED_DEADLINE_SECONDS="$DEADLINE_SECONDS" +export SEED_DB_SECRET="$DB_SECRET" +export SEED_MARIADB_HOST="$MARIADB_HOST" +export SEED_MARIADB_PORT="$MARIADB_PORT" +export SEED_MARIADB_USER="$MARIADB_USER" +export SEED_IDENTITY_DB="$IDENTITY_DB" +export SEED_ANALYTICS_DB="$ANALYTICS_DB" +export SEED_CLICKHOUSE_HOST="$CLICKHOUSE_HOST" +export SEED_CLICKHOUSE_HTTP_PORT="$CLICKHOUSE_HTTP_PORT" +export SEED_CLICKHOUSE_USER="$CLICKHOUSE_USER" +export SEED_CLICKHOUSE_DATABASE="$CLICKHOUSE_DATABASE" +export SEED_TENANT_ID="$TENANT" +export SEED_DEV_USER_EMAIL="$DEV_EMAIL" +export SEED_AUTH_MODE="$AUTH_MODE" +export SEED_IDP_SOURCE_TYPE="$IDP_SOURCE_TYPE" +export SEED_CROSS_TENANT="$CROSS_TENANT" +export SEED_FORCE="$FORCE" +# Deliberately allowed to be empty: the seeder documents its own window, and an +# empty value here means "use it" rather than a second copy of that default. +export SEED_WINDOW_DAYS="$WINDOW_DAYS" +export SEED_ANCHOR_DATE="$ANCHOR_DATE" +export SEED_PULL_SECRETS="$PULL_SECRETS" + +# Only the seed variables are substituted, so a `$HOME` or `$PATH` in the +# template stays literal. +manifest="$(envsubst ' + ${SEED_JOB_NAME} ${SEED_NAMESPACE} ${SEED_IMAGE} ${SEED_STEP} + ${SEED_DEADLINE_SECONDS} ${SEED_DB_SECRET} + ${SEED_MARIADB_HOST} ${SEED_MARIADB_PORT} ${SEED_MARIADB_USER} + ${SEED_IDENTITY_DB} ${SEED_ANALYTICS_DB} + ${SEED_CLICKHOUSE_HOST} ${SEED_CLICKHOUSE_HTTP_PORT} ${SEED_CLICKHOUSE_USER} + ${SEED_CLICKHOUSE_DATABASE} + ${SEED_TENANT_ID} ${SEED_DEV_USER_EMAIL} ${SEED_AUTH_MODE} ${SEED_IDP_SOURCE_TYPE} + ${SEED_CROSS_TENANT} ${SEED_FORCE} ${SEED_WINDOW_DAYS} ${SEED_ANCHOR_DATE} + ${SEED_PULL_SECRETS} +' < "$JOB_TEMPLATE")" + +if [[ "$DRY_RUN" -eq 1 ]]; then + printf '%s\n' "$manifest" + exit 0 +fi + +printf '%s\n' "$manifest" | kubectl apply -f - +echo "==> applied Job $job_name" + +if [[ "$FOLLOW" -eq 0 ]]; then + echo " follow it with: kubectl -n $NAMESPACE logs -f job/$job_name" + exit 0 +fi + +# Wait for the pod's CONTAINER to start, not just for the pod object: `logs -f` +# against a pod still in ContainerCreating fails immediately, and the run would +# then finish with nothing streamed. A pod that cannot start at all is caught by +# the poll loop below, so this wait is bounded and never fatal. +for _ in $(seq 1 60); do + phase="$(kubectl -n "$NAMESPACE" get pod -l "job-name=$job_name" \ + -o 'jsonpath={.items[0].status.phase}' 2>/dev/null || true)" + case "$phase" in + Running|Succeeded|Failed) break ;; + esac + sleep 2 +done + +kubectl -n "$NAMESPACE" logs -f "job/$job_name" || true + +# The log stream ending is not the verdict — read it from the Job. Polled rather +# than `kubectl wait --for=condition=complete`, which only knows how to wait for +# success: a refused seed would sit there until the timeout instead of reporting +# in the second it failed. +deadline=$((SECONDS + DEADLINE_SECONDS)) +while [[ "$SECONDS" -lt "$deadline" ]]; do + # `|| true` on every read: a transient apiserver hiccup inside a loop that may + # run for an hour must not kill the script through `set -e`. + succeeded="$(kubectl -n "$NAMESPACE" get "job/$job_name" \ + -o 'jsonpath={.status.succeeded}' 2>/dev/null || true)" + failed="$(kubectl -n "$NAMESPACE" get "job/$job_name" \ + -o 'jsonpath={.status.failed}' 2>/dev/null || true)" + if [[ "${succeeded:-0}" -ge 1 ]]; then + echo "==> seed complete: $job_name" + exit 0 + fi + if [[ "${failed:-0}" -ge 1 ]]; then + echo "ERROR: Job $job_name failed. Its logs above hold the reason; the Job is kept" >&2 + echo " (backoffLimit 0, no retry) so it can be read again:" >&2 + echo " kubectl -n $NAMESPACE logs job/$job_name" >&2 + exit 1 + fi + + # A pod that cannot start never becomes either, and waiting out the deadline + # for it would be an hour of silence. The common cause is an image the cluster + # cannot pull — including a locally built one on a remote cluster. + waiting="$(kubectl -n "$NAMESPACE" get pod -l "job-name=$job_name" \ + -o 'jsonpath={.items[0].status.containerStatuses[0].state.waiting.reason}' 2>/dev/null || true)" + case "$waiting" in + ImagePullBackOff|ErrImagePull|InvalidImageName) + echo "ERROR: the seed pod cannot start: $waiting for image $IMAGE." >&2 + echo " Push the image somewhere the cluster can pull it, or pass a tag it already has." >&2 + exit 1 ;; + esac + + sleep 3 +done + +echo "ERROR: Job $job_name neither completed nor failed within ${DEADLINE_SECONDS}s:" >&2 +echo " kubectl -n $NAMESPACE describe job/$job_name" >&2 +exit 1 diff --git a/src/ingestion/tools/seed/tests/__init__.py b/src/ingestion/tools/seed/tests/__init__.py new file mode 100644 index 000000000..93b8588de --- /dev/null +++ b/src/ingestion/tools/seed/tests/__init__.py @@ -0,0 +1 @@ +"""Unit tests for the seed package. Run: python3 -m unittest discover -s tests -t .""" diff --git a/src/ingestion/tools/seed/tests/conftest.py b/src/ingestion/tools/seed/tests/conftest.py new file mode 100644 index 000000000..f93b4ee6e --- /dev/null +++ b/src/ingestion/tools/seed/tests/conftest.py @@ -0,0 +1,39 @@ +"""Shared test bootstrap: put the tool directory on `sys.path` and stub the +database drivers. + +Named `conftest.py` for pytest's benefit, but it is plain module-level code and +works the same under `unittest`, which is what this package actually uses (see +`tests/README` in the seeder README). Both entry points import it first: + + python3 -m unittest discover -s tests -t . # from the tool directory + +`pymysql` and `clickhouse_connect` are runtime-only dependencies of the seed +image. Every test here exercises the pure half — env parsing, SQL shapes, +refusal messages — so stand-ins keep the suite runnable with nothing installed. +""" + +from __future__ import annotations + +import sys +import types +from pathlib import Path + +#: The tool directory (parent of `tests/`), so `import insight_seed` resolves +#: whichever checkout this file lives in. +TOOL_ROOT = Path(__file__).resolve().parents[1] +if str(TOOL_ROOT) not in sys.path: + sys.path.insert(0, str(TOOL_ROOT)) + + +def _stub_pymysql() -> None: + if "pymysql" in sys.modules: + return + stub = types.ModuleType("pymysql") + stub.cursors = types.SimpleNamespace(Cursor=object) # type: ignore[attr-defined] + stub.connections = types.SimpleNamespace(Connection=object) # type: ignore[attr-defined] + stub.connect = lambda **_kwargs: None # type: ignore[attr-defined] + stub.MySQLError = Exception # type: ignore[attr-defined] + sys.modules["pymysql"] = stub + + +_stub_pymysql() diff --git a/deploy/seed/test_identity.py b/src/ingestion/tools/seed/tests/test_identity.py similarity index 67% rename from deploy/seed/test_identity.py rename to src/ingestion/tools/seed/tests/test_identity.py index 1df0d1548..2900233dd 100644 --- a/deploy/seed/test_identity.py +++ b/src/ingestion/tools/seed/tests/test_identity.py @@ -1,61 +1,34 @@ """Idempotency + roster-scope tests for `identity.seed_login_ids`. -No pytest / DB fixture in this package (deploy/seed has no existing test -harness) — a stdlib `unittest` test against a minimal fake cursor is enough to -lock two regressions: +A stdlib `unittest` test against a minimal fake cursor, locking two +regressions: 1. Idempotency: migration 004 (`004_persons_relax_constraints.sql`) put `created_at` in `persons`' unique key, so `INSERT IGNORE` alone no longer dedupes a re-run (each insert gets a fresh `created_at`, so the unique key - never collides). `seed_login_ids` must check for an existing row - explicitly, per pair, before inserting. + never collides). Every writer must check for an existing row explicitly. 2. Roster scope: fakeidp only defines a fixed dev-lead identity, but a Keycloak realm seeds the WHOLE roster (gen-realm.py pins every realm user's id to their own roster uuid) — `seed_login_ids` must seed a row per roster member under `AUTH_MODE=keycloak`, not just the dev lead. -`pymysql` isn't installed in every environment this runs in (it's only a -runtime dependency of the seed container image) — a minimal stand-in is -injected into `sys.modules` before importing `identity` so this test has no -external dependency beyond the stdlib. - -Primary, always-works invocation, from anywhere: -`python3 -m unittest deploy/seed/test_identity.py -v`. `python3 -m unittest -discover -s deploy/seed` also works in most setups, but `discover` additionally -requires the start dir to be importable as a top-level package in some -Python/cwd combinations (no `__init__.py` here by design — `deploy/seed` is a -flat-module package, see pyproject.toml's `py-modules`) — if that fails in -your environment, use the primary invocation above. +From the tool directory: + + python3 -m unittest discover -s tests -t . + python3 -m unittest tests.test_identity -v """ from __future__ import annotations import os -import sys -import types import unittest from typing import Any +from . import conftest # noqa: F401 — sys.path + driver stubs, before the imports below + os.environ.setdefault("IDP_SOURCE_TYPE", "fakeidp") -# `identity`/`profiles` are flat modules (no package __init__), so importing -# them by name only works when deploy/seed is on sys.path. unittest's -# file-path invocation (`python3 -m unittest path/to/test_identity.py`) does -# NOT add the file's own directory the way `discover` does — add it -# explicitly so both invocation styles resolve `import identity` the same way. -_THIS_DIR = os.path.dirname(os.path.abspath(__file__)) -if _THIS_DIR not in sys.path: - sys.path.insert(0, _THIS_DIR) - -if "pymysql" not in sys.modules: - _pymysql_stub = types.ModuleType("pymysql") - _pymysql_stub.cursors = types.SimpleNamespace(Cursor=object) # type: ignore[attr-defined] - _pymysql_stub.connections = types.SimpleNamespace(Connection=object) # type: ignore[attr-defined] - _pymysql_stub.connect = lambda **_kwargs: None # type: ignore[attr-defined] - sys.modules["pymysql"] = _pymysql_stub - -import identity # noqa: E402 — stub/env setup above must run first -import profiles # noqa: E402 — stub/env setup above must run first +from insight_seed import identity, profiles _TENANT = "00000000-df51-5b42-9538-d2b56b7ee953" @@ -133,8 +106,9 @@ def test_second_run_does_not_insert_a_duplicate(self) -> None: cur = _FakeCursor() roster = _roster() - first_run_count = identity.seed_login_ids(cur, _TENANT, roster) - second_run_count = identity.seed_login_ids(cur, _TENANT, roster) + # The fake cursor implements only what these writers call. + first_run_count = identity.seed_login_ids(cur, _TENANT, roster) # type: ignore[arg-type] + second_run_count = identity.seed_login_ids(cur, _TENANT, roster) # type: ignore[arg-type] self.assertEqual(first_run_count, 1, "fakeidp seeds only the dev lead") self.assertEqual(second_run_count, 0, "re-run must be a no-op, not a duplicate insert") @@ -145,11 +119,13 @@ def test_keycloak_seeds_the_whole_roster(self) -> None: cur = _FakeCursor() roster = _roster() - first_run_count = identity.seed_login_ids(cur, _TENANT, roster) - second_run_count = identity.seed_login_ids(cur, _TENANT, roster) + # The fake cursor implements only what these writers call. + first_run_count = identity.seed_login_ids(cur, _TENANT, roster) # type: ignore[arg-type] + second_run_count = identity.seed_login_ids(cur, _TENANT, roster) # type: ignore[arg-type] self.assertEqual( - first_run_count, len(roster), + first_run_count, + len(roster), "keycloak seeds every roster persona (gen-realm.py registers all of them)", ) self.assertEqual(second_run_count, 0, "re-run must be a no-op for every pair") diff --git a/src/ingestion/tools/seed/tests/test_preflight.py b/src/ingestion/tools/seed/tests/test_preflight.py new file mode 100644 index 000000000..6fc5bd195 --- /dev/null +++ b/src/ingestion/tools/seed/tests/test_preflight.py @@ -0,0 +1,307 @@ +"""Env-contract and preflight-message tests. + +Same harness as `test_identity.py`: stdlib `unittest`, no database, no +third-party import. Everything under test here is the pure half — env parsing, +the SQL a guard issues, and the messages a refusal carries — because that is +what has to stay true for an operator who reads only the error. + +From the tool directory: + + python3 -m unittest discover -s tests -t . + python3 -m unittest tests.test_preflight -v +""" + +from __future__ import annotations + +import datetime as _dt +import re +import unittest +import uuid as uuid_mod + +from insight_seed import config, identity, preflight + +from . import conftest + +_TENANT = "3f1d8f4e-6c2a-4a9b-91d7-8e5c0b2a7f36" + + +class _CapturingCursor: + """Minimal cursor stand-in: records statements, replays one row.""" + + def __init__(self, result: tuple[object, ...] | None) -> None: + self.executed: list[tuple[str, tuple[object, ...]]] = [] + self._result = result + + def execute(self, sql: str, params: tuple[object, ...] = ()) -> None: + self.executed.append((sql, params)) + + def fetchone(self) -> tuple[object, ...] | None: + return self._result + + +class TenantContractTests(unittest.TestCase): + def test_a_missing_tenant_is_refused_and_names_the_variable(self) -> None: + with self.assertRaises(config.EnvContractError) as caught: + config.parse_tenant_id({}) + self.assertIn(config.TENANT_ENV, str(caught.exception)) + + def test_a_blank_tenant_is_the_same_as_a_missing_one(self) -> None: + for value in ("", " ", "\t"): + with self.subTest(value=value), self.assertRaises(config.EnvContractError): + config.parse_tenant_id({config.TENANT_ENV: value}) + + def test_a_tenant_that_is_not_a_uuid_is_refused(self) -> None: + with self.assertRaises(config.EnvContractError) as caught: + config.parse_tenant_id({config.TENANT_ENV: "the-default-one"}) + self.assertIn("not a UUID", str(caught.exception)) + + def test_a_uuid_tenant_is_returned_stripped(self) -> None: + self.assertEqual(config.parse_tenant_id({config.TENANT_ENV: f" {_TENANT} "}), _TENANT) + + +class AnalyticsDatabaseContractTests(unittest.TestCase): + def test_a_missing_analytics_database_is_refused_and_explains_why(self) -> None: + with self.assertRaises(config.EnvContractError) as caught: + config.parse_analytics_database({}) + message = str(caught.exception) + self.assertIn(config.ANALYTICS_DB_ENV, message) + self.assertIn("metric_definitions", message) + + def test_the_identity_database_keeps_a_default_because_every_stand_agrees(self) -> None: + self.assertEqual(config.parse_identity_database({}), "identity") + self.assertEqual(config.parse_identity_database({config.IDENTITY_DB_ENV: "ident"}), "ident") + + +class FlagContractTests(unittest.TestCase): + def test_the_cross_tenant_fixture_is_on_unless_turned_off(self) -> None: + self.assertTrue(config.cross_tenant_fixture_enabled({})) + for value in ("0", "false", "NO", "off"): + with self.subTest(value=value): + self.assertFalse( + config.cross_tenant_fixture_enabled({config.CROSS_TENANT_FIXTURE_ENV: value}) + ) + + def test_force_is_off_unless_asked_for(self) -> None: + self.assertFalse(config.force_enabled({})) + self.assertTrue(config.force_enabled({config.FORCE_ENV: "1"})) + + def test_a_flag_that_is_neither_true_nor_false_is_refused(self) -> None: + with self.assertRaises(config.EnvContractError): + config.force_enabled({config.FORCE_ENV: "maybe"}) + + +class RefusalMessageTests(unittest.TestCase): + def test_a_wrong_database_refusal_names_the_database_it_looked_in(self) -> None: + message = preflight.table_missing_problem( + "wrong_db", "metric_definitions", needed_for="the analytics catalogue" + ) + self.assertIn("wrong_db", message) + self.assertIn("metric_definitions", message) + + def test_a_foreign_rows_refusal_offers_the_override_by_name(self) -> None: + message = preflight.foreign_rows_problem(7, _TENANT, "identity") + self.assertIn("7", message) + self.assertIn(_TENANT, message) + self.assertIn(config.FORCE_ENV, message) + + def test_a_refusal_reports_every_problem_not_just_the_first(self) -> None: + error = preflight.PreflightError(["first problem", "second problem"]) + self.assertIn("first problem", str(error)) + self.assertIn("second problem", str(error)) + + +class _FakeResult: + def __init__(self, rows: list[tuple[object, ...]]) -> None: + self.result_rows = rows + + +class _FakeClickHouse: + """Replays the column lookup, then the per-target counts.""" + + def __init__( + self, + columns: list[tuple[str, str, str]], + counts: list[tuple[str, int]], + ) -> None: + self._columns = columns + self._counts = counts + self.queries: list[str] = [] + + def query(self, sql: str, parameters: dict[str, object] | None = None) -> _FakeResult: + self.queries.append(sql) + if "system.columns" in sql: + return _FakeResult(list(self._columns)) + return _FakeResult([(name, count) for name, count in self._counts]) + + +class SilverResetGuardTests(unittest.TestCase): + def test_the_scan_covers_exactly_what_the_generators_clear(self) -> None: + """A name pattern would both miss targets in other databases and refuse + stands over tables the seed never touches.""" + from insight_seed.generators.base import RESET_TARGETS + + columns = [(schema, table, "tenant_id") for schema, table in RESET_TARGETS] + client = _FakeClickHouse(columns=columns, counts=[]) + preflight._foreign_silver_rows(client, _TENANT) + + scan = client.queries[-1] + for schema, table in RESET_TARGETS: + with self.subTest(target=f"{schema}.{table}"): + self.assertIn(f"`{schema}`.`{table}`", scan) + self.assertNotIn("class_wiki_activity", scan) + + def test_every_registered_target_is_actually_truncated_by_a_generator(self) -> None: + """The registry is what preflight scans, so a target that no generator + clears would refuse a stand over data the seed leaves alone.""" + from insight_seed.generators.base import RESET_TARGETS + + called: set[tuple[str, str]] = set() + for path in sorted((conftest.TOOL_ROOT / "insight_seed" / "generators").glob("*.py")): + for schema, table in re.findall( + r'truncate\(\s*client,\s*"([a-z0-9_]+)",\s*"([a-z0-9_]+)"', path.read_text() + ): + called.add((schema, table)) + self.assertEqual(called, set(RESET_TARGETS)) + + def test_a_tenant_column_is_found_under_either_name(self) -> None: + client = _FakeClickHouse( + columns=[ + ("silver", "class_focus_metrics", "insight_tenant_id"), + ("silver", "class_people", "tenant_id"), + ("silver", "not_a_target", "tenant_id"), + ], + counts=[], + ) + found = preflight._tenant_columns(client) + self.assertEqual(found[("silver", "class_focus_metrics")], "insight_tenant_id") + self.assertEqual(found[("silver", "class_people")], "tenant_id") + self.assertNotIn(("silver", "not_a_target"), found) + + def test_targets_without_a_tenant_column_are_reported_as_unjudgeable(self) -> None: + from insight_seed.generators.base import RESET_TARGETS + + client = _FakeClickHouse( + columns=[("silver", "class_people", "tenant_id")], + counts=[], + ) + _, _, unattributable = preflight._foreign_silver_rows(client, _TENANT) + self.assertIn(("bronze_bamboohr", "employees"), unattributable) + self.assertNotIn(("silver", "class_people"), unattributable) + self.assertEqual(len(unattributable), len(RESET_TARGETS) - 1) + + def test_foreign_rows_are_summed_and_the_worst_targets_named(self) -> None: + client = _FakeClickHouse( + columns=[ + ("silver", "class_git_commits", "tenant_id"), + ("silver", "class_people", "tenant_id"), + ], + counts=[("silver.class_git_commits", 900), ("silver.class_people", 100)], + ) + total, worst, _ = preflight._foreign_silver_rows(client, _TENANT) + self.assertEqual(total, 1000) + self.assertEqual(worst[0], ("silver.class_git_commits", 900)) + + def test_a_null_tenant_is_this_seeder_s_own_row_not_a_foreign_one(self) -> None: + """`class_people` carries the tenant in `workspace_id`; counting NULLs + would refuse every re-seed of a stand this seeder seeded itself.""" + client = _FakeClickHouse(columns=[("silver", "class_people", "tenant_id")], counts=[]) + preflight._foreign_silver_rows(client, _TENANT) + self.assertIn("IS NOT NULL", client.queries[-1]) + self.assertNotIn("IS NULL OR", client.queries[-1]) + + def test_a_stand_with_none_of_those_tables_reports_nothing(self) -> None: + client = _FakeClickHouse(columns=[], counts=[]) + total, worst, _ = preflight._foreign_silver_rows(client, _TENANT) + self.assertEqual((total, worst), (0, [])) + + def test_the_refusal_says_the_step_truncates_and_offers_the_override(self) -> None: + message = preflight.foreign_silver_problem(1000, [("silver.class_people", 1000)], _TENANT) + self.assertIn("TRUNCATE", message) + self.assertIn("class_people", message) + self.assertIn(config.FORCE_ENV, message) + + +class ResetRegistryTests(unittest.TestCase): + def test_clearing_an_unregistered_relation_is_refused(self) -> None: + from insight_seed.generators import base + + with self.assertRaises(ValueError) as caught: + # The client is never reached: registration is checked first. + base.truncate(object(), "silver", "class_not_registered") # type: ignore[arg-type] + self.assertIn("RESET_TARGETS", str(caught.exception)) + + +class WindowContractTests(unittest.TestCase): + """The window has one reader, so the manifest cannot report a window the rows + are not in — two copies computing `now()` disagree across a UTC midnight.""" + + def test_the_generators_and_the_manifest_read_the_same_window(self) -> None: + from insight_seed import manifest as manifest_mod + from insight_seed.generators import base + + env = {config.ANCHOR_ENV: "2026-06-30", config.DAYS_ENV: "14"} + self.assertEqual(config.parse_anchor_date(env), _dt.date(2026, 6, 30)) + self.assertEqual(config.parse_seed_days(env), 14) + self.assertIs(manifest_mod._anchor, config.parse_anchor_date) + self.assertIs(manifest_mod._days, config.parse_seed_days) + self.assertIs(base.DEFAULT_SEED_DAYS, config.DEFAULT_SEED_DAYS) + + def test_the_literal_today_and_an_unset_anchor_mean_the_same_day(self) -> None: + self.assertEqual( + config.parse_anchor_date({config.ANCHOR_ENV: "today"}), + config.parse_anchor_date({}), + ) + + def test_an_empty_window_means_unset_because_a_rendered_job_passes_empty(self) -> None: + self.assertEqual(config.parse_seed_days({config.DAYS_ENV: ""}), config.DEFAULT_SEED_DAYS) + self.assertEqual( + config.parse_anchor_date({config.ANCHOR_ENV: ""}), config.parse_anchor_date({}) + ) + + def test_a_malformed_window_is_refused_before_anything_is_written(self) -> None: + for env in ( + {config.ANCHOR_ENV: "30-06-2026"}, + {config.DAYS_ENV: "x"}, + {config.DAYS_ENV: "0"}, + ): + with self.subTest(env=env), self.assertRaises(config.EnvContractError): + config.parse_anchor_date( + env + ) if config.ANCHOR_ENV in env else config.parse_seed_days(env) + + def test_a_missing_dev_user_email_is_refused_and_says_what_it_anchors(self) -> None: + with self.assertRaises(config.EnvContractError) as caught: + config.parse_dev_user_email({}) + self.assertIn(config.DEV_USER_EMAIL_ENV, str(caught.exception)) + + +class SeedReasonNamespaceTests(unittest.TestCase): + def test_every_reason_the_identity_seed_writes_carries_the_shared_prefix(self) -> None: + reasons = [ + value + for name, value in vars(identity).items() + if name.startswith("_REASON_") and isinstance(value, str) + ] + self.assertTrue(reasons, "identity.py should define its reasons as _REASON_* constants") + for reason in reasons: + with self.subTest(reason=reason): + self.assertTrue(reason.startswith(config.SEED_REASON_PREFIX)) + + def test_the_foreign_row_query_excludes_exactly_that_prefix(self) -> None: + cursor = _CapturingCursor(result=(3,)) + self.assertEqual(preflight._count_foreign_persons(cursor, _TENANT), 3) # type: ignore[arg-type] + + sql, params = cursor.executed[-1] + self.assertIn("reason NOT LIKE", sql) + self.assertEqual(params[0], uuid_mod.UUID(_TENANT).bytes) + self.assertEqual(params[1], f"{config.SEED_REASON_PREFIX}%") + + def test_a_tenant_with_no_foreign_rows_reads_as_zero(self) -> None: + self.assertEqual( + preflight._count_foreign_persons(_CapturingCursor(result=None), _TENANT), # type: ignore[arg-type] + 0, + ) + + +if __name__ == "__main__": + unittest.main() diff --git a/src/ingestion/tools/toolbox/Dockerfile b/src/ingestion/tools/toolbox/Dockerfile index 7024e8f01..b34caaa9e 100644 --- a/src/ingestion/tools/toolbox/Dockerfile +++ b/src/ingestion/tools/toolbox/Dockerfile @@ -22,10 +22,19 @@ RUN curl -fsSL "https://github.com/mikefarah/yq/releases/latest/download/yq_linu RUN curl -fsSL "https://dl.k8s.io/release/$(curl -fsSL https://dl.k8s.io/release/stable.txt)/bin/linux/$(dpkg --print-architecture)/kubectl" \ -o /usr/local/bin/kubectl && chmod +x /usr/local/bin/kubectl -# dbt-clickhouse -RUN pip install --no-cache-dir dbt-clickhouse +# dbt-clickhouse, plus the two database drivers the sample-data seeder +# (tools/seed, COPYed in below) needs on top of it. Same layer on purpose: the +# seeder is part of what this image is for, not an afterthought bolted on at +# run time. +RUN pip install --no-cache-dir \ + dbt-clickhouse \ + 'PyMySQL==1.2.0' \ + 'clickhouse-connect==1.6.0' -# Copy entire ingestion project +# Copy entire ingestion project. This is also how the seeder ships: it lives at +# tools/seed inside this tree, so `seed-stand.sh` can run it from +# /ingestion/tools/seed against the same scripts/ and dbt/ the migrations use — +# one image, one version, no drift between seeder and migration SQL. COPY . /ingestion WORKDIR /ingestion diff --git a/tests/generate_schemas.py b/tests/generate_schemas.py index 999d73155..d99a5b63d 100644 --- a/tests/generate_schemas.py +++ b/tests/generate_schemas.py @@ -24,7 +24,7 @@ The output is COMMITTED. A test run must never need the generator, which is a dev-only dependency and absent from the ui-tests image; `--check` is what keeps -the committed copy honest, the same arrangement `deploy/seed/render_profile.py` +the committed copy honest, the same arrangement `src/ingestion/tools/seed/render_profile.py` uses for PROFILE.md. Deliberately NOT a pytest case. `tests/stand/` exists to assert things about a diff --git a/tests/lib/insight_stand/__init__.py b/tests/lib/insight_stand/__init__.py index d6e50dbe7..08a4939e4 100644 --- a/tests/lib/insight_stand/__init__.py +++ b/tests/lib/insight_stand/__init__.py @@ -3,7 +3,7 @@ This package holds: * `manifest` — the typed model of the stand's self-description - (`deploy/seed/manifest.json`), the only source of fixture names, capabilities + (`src/ingestion/tools/seed/manifest.json`), the only source of fixture names, capabilities and seeded facts. * `stand` — where the stand is: base-URL resolution for a host-side or in-network runner. diff --git a/tests/lib/insight_stand/manifest.py b/tests/lib/insight_stand/manifest.py index 2d4deee18..efd02f569 100644 --- a/tests/lib/insight_stand/manifest.py +++ b/tests/lib/insight_stand/manifest.py @@ -1,9 +1,9 @@ """Typed model of the seed manifest. The manifest is the stand's self-description: what was seeded, who exists, what -the stand can do. `deploy/seed/seed.py` writes it to one place: +the stand can do. `src/ingestion/tools/seed/seed.py` writes it to one place: - deploy/seed/manifest.json + src/ingestion/tools/seed/manifest.json A reader may be told to look elsewhere — `$INSIGHT_STAND_MANIFEST`, or pytest's `--stand-manifest` — because a runner that does not share the repo's filesystem @@ -38,7 +38,7 @@ # The location the seed writes to, resolved from this file: # tests/lib/insight_stand/manifest.py -> ../../../ _REPO_ROOT: Final[Path] = Path(__file__).resolve().parents[3] -MANIFEST_PATH: Final[Path] = _REPO_ROOT / "deploy" / "seed" / "manifest.json" +MANIFEST_PATH: Final[Path] = _REPO_ROOT / "src" / "ingestion" / "tools" / "seed" / "manifest.json" # Point a runner at a manifest it can actually reach. Named the same way as # $INSIGHT_STAND_BASE_URL and $INSIGHT_STAND_ENV_FILE in stand.py. @@ -52,10 +52,11 @@ def default_manifest_path(environ: Mapping[str, str] | None = None) -> Path: The override earns its place in containers. `MANIFEST_PATH` is derived from THIS FILE's location, so in an image that holds the tree at `/tests` it - resolves to `/deploy/seed/manifest.json` — and a bind mount then has to - reproduce that arithmetic exactly, or the suite reports an unseeded stand. - Naming the file is the honest alternative to guessing where `parents[3]` - landed. + resolves to a path with nothing above it — reproducing that arithmetic in a + bind mount is how a seeded stand gets reported as unseeded. A containerised + runner therefore mounts the file at a stable path and names it through this + variable (`dev-compose.sh` uses `/stand/manifest.json`) instead of guessing + where `parents[3]` landed. """ env = os.environ if environ is None else environ override = (env.get(MANIFEST_PATH_ENV) or "").strip() @@ -187,7 +188,7 @@ class GoldenMetric: """One `golden_metrics[]` entry: an exact, hand-sourced expectation. Currently always absent — `golden_metrics` is `[]` on every stand, by - design. See `deploy/seed/golden_metrics.py` for why, and read + design. See `src/ingestion/tools/seed/golden_metrics.py` for why, and read `Manifest.golden_metrics_note` to tell "none measured yet" apart from "measured and genuinely zero". @@ -259,7 +260,7 @@ def parse(cls, doc: Mapping[str, Any], where: str) -> DefinitionOverride: @dataclass(frozen=True) class Catalogue: - """Rows no endpoint creates, seeded by `deploy/seed/analytics.py`. + """Rows no endpoint creates, seeded by `src/ingestion/tools/seed/analytics.py`. It is optional and a test must treat absence as "cannot assert" rather than as failure: a stand seeded without the `analytics` step is a real state. diff --git a/tests/lib/insight_stand/personas.py b/tests/lib/insight_stand/personas.py index 491d86ea5..d07d6a836 100644 --- a/tests/lib/insight_stand/personas.py +++ b/tests/lib/insight_stand/personas.py @@ -70,7 +70,7 @@ #: The manifest fixture name for the second tenant's only person. Their whole #: purpose is to be a caller the product refuses, so they hold no role, no team -#: and no org-chart edge — see `deploy/seed/profiles.py::build_other_tenant_roster`. +#: and no org-chart edge — see `src/ingestion/tools/seed/profiles.py::build_other_tenant_roster`. OTHER_TENANT_FIXTURE: Final[str] = "other_tenant_lead" diff --git a/tests/pyproject.toml b/tests/pyproject.toml index 5171c4fa3..c9dbfa9ee 100644 --- a/tests/pyproject.toml +++ b/tests/pyproject.toml @@ -21,10 +21,12 @@ build-backend = "setuptools.build_meta" name = "insight-stand-tests" version = "0.1.0" description = "Deployed-stand test suite for Insight: shared library plus API and browser journeys." -# Matches deploy/seed/pyproject.toml, whose manifest and PROFILE renderer this -# suite reads. uv provisions this interpreter everywhere — including inside the -# ui-tests image, whose own system Python is older — so the floor holds in every -# runner rather than being a host-only aspiration. +# This suite's own floor, not inherited from anywhere: it reads the seeder's +# manifest as DATA and imports none of its modules, so the seeder's lower 3.12 +# floor (it ships inside the 3.12 toolbox image) does not apply here. uv +# provisions this interpreter everywhere — including inside the ui-tests image, +# whose own system Python is older — so the floor holds in every runner rather +# than being a host-only aspiration. requires-python = ">=3.13" dependencies = [ "pytest>=8", @@ -91,7 +93,7 @@ target-version = "py313" extend-exclude = ["stand/api/schemas/analytics.py", "stand/api/schemas/authenticator.py"] [tool.ruff.lint] -# Same pragmatic strict set as deploy/seed/pyproject.toml. +# Same pragmatic strict set as src/ingestion/tools/seed/pyproject.toml. select = [ "E", "W", # pycodestyle "F", # pyflakes diff --git a/tests/stand/README.md b/tests/stand/README.md index 44d956d7f..92f23c3a9 100644 --- a/tests/stand/README.md +++ b/tests/stand/README.md @@ -2,7 +2,8 @@ Deployed-stand tests for Insight: a real Keycloak login, browser journeys against the SPA, and an API-contract suite — all run against a local -`docker-compose` stand seeded deterministically for tests (`deploy/seed`). +`docker-compose` stand seeded deterministically for tests +(`src/ingestion/tools/seed`). This suite assumes an **already-running, already-seeded** stand. It never starts compose, applies migrations, or spawns service processes itself — @@ -63,7 +64,7 @@ a stand other than the one it just brought up. ## Reading PROFILE.md before writing a test -[`deploy/seed/PROFILE.md`](../../deploy/seed/PROFILE.md) is generated from +[`src/ingestion/tools/seed/PROFILE.md`](../../src/ingestion/tools/seed/PROFILE.md) is generated from the same builder that writes the stand's `manifest.json`, so the two cannot disagree. Before adding a test, read it for: @@ -71,7 +72,7 @@ disagree. Before adding a test, read it for: role-shaped names (`dev_lead`, `admin_operator`, …) a test may declare against; a raw email or UUID is never a stable target. - **populated / golden metrics** — that table is empty by design (see - `deploy/seed/golden_metrics.py`'s admission criteria: an expectation must be + `src/ingestion/tools/seed/golden_metrics.py`'s admission criteria: an expectation must be computable from the seed inputs, not read back out of the gold layer). No test here asserts a metric's exact value, and none should until the table has entries — reading a number off a running stand and asserting it back @@ -87,7 +88,8 @@ disagree. Before adding a test, read it for: (compose seeds silver/gold directly). A test that needs a capability the stand may lack should carry the matching marker (below), not assume it. -Regenerate it with `python3 deploy/seed/render_profile.py` after changing +Regenerate it with `python3 -m insight_seed.render_profile` (from +`src/ingestion/tools/seed`) after changing the roster or the manifest builder; `--check` verifies it without a database. @@ -161,9 +163,12 @@ it": gate over the committed OpenAPI document); migrating it is a known follow-up. Until it lands, `api/operations.py` is the only catalogue of the surface and it is kept honest by hand. -- **Cross-tenant refusal.** `deploy/seed` provisions a single tenant - (`TENANT_DEFAULT_ID`), so there is no second tenant's caller to be refused - with. +- **Cross-tenant refusal.** Covered on compose, and only there: the second + tenant's caller is a fixture the seed writes when + `SEED_CROSS_TENANT_FIXTURE` is on, which `docker-compose.yml` sets. A cluster + stand turns it off (a second tenant aborts identity-resolution's scheduled + projection), and the seed's manifest then omits `other_tenant_lead`, so tests + declaring `requires_seed("other_tenant_lead")` skip rather than fail. - **JWT verification** — an expired token, a wrong audience, an untrusted issuer, a signature from a key the JWKS never published. Minting tokens is ruled out here by design (see `../lib/insight_stand/session.py`): this suite diff --git a/tests/stand/api/identity/test_internal.py b/tests/stand/api/identity/test_internal.py index 0cea3ae54..af2e62e88 100644 --- a/tests/stand/api/identity/test_internal.py +++ b/tests/stand/api/identity/test_internal.py @@ -50,7 +50,7 @@ from ..schemas import IdentityValue # The dev lead's fixed external id under fakeidp — mirrors -# `deploy/seed/profiles.py::_FAKEIDP_DEV_LEAD_EXTERNAL_ID`. fakeidp's +# `src/ingestion/tools/seed/profiles.py::_FAKEIDP_DEV_LEAD_EXTERNAL_ID`. fakeidp's # users.yaml pins its first user's `sub` to this value regardless of which # email the dev lead persona was seeded under, so it cannot be derived from # the manifest fixture and has to be duplicated here, same as the seed does. @@ -61,7 +61,7 @@ def _dev_lead_login_id(stand_manifest: Manifest) -> tuple[str, str]: """`(source_type, external_id)` the dev lead's login-bootstrap row was seeded under, for whichever IdP this stand runs. - Mirrors `deploy/seed/profiles.py::get_login_id_pairs` / + Mirrors `src/ingestion/tools/seed/profiles.py::get_login_id_pairs` / `get_idp_source_type`: on `keycloak` every persona's external id is their own roster uuid; on `fakeidp` only the dev lead can log in at all, under the fixed id above. `capabilities.idp` (`"keycloak"` | `"fakeidp"`) is @@ -198,10 +198,15 @@ def test_by_external_id_refuses_a_person( @pytest.mark.requires_service_principal -def test_by_external_id_of_an_unknown_id_is_404(stand_manifest: Manifest, service_client: ApiClient) -> None: +def test_by_external_id_of_an_unknown_id_is_404( + stand_manifest: Manifest, service_client: ApiClient +) -> None: response = service_client.get( "/internal/persons/by-external-id", - params={"source_type": stand_manifest.capabilities.idp, "external_id": "nobody-external-id"}, + params={ + "source_type": stand_manifest.capabilities.idp, + "external_id": "nobody-external-id", + }, ) assert response.status_code == 404, ( f"an unknown external id answered {response.status_code} to a service principal: " diff --git a/tests/stand/conftest.py b/tests/stand/conftest.py index fc7227c42..076d56ae2 100644 --- a/tests/stand/conftest.py +++ b/tests/stand/conftest.py @@ -12,7 +12,7 @@ Two rules follow from that split: * The stand must describe itself. Every fixture name, capability and seeded - fact comes from `deploy/seed/manifest.json`. If it is missing or unparseable + fact comes from `src/ingestion/tools/seed/manifest.json`. If it is missing or unparseable the session aborts; nothing here has a default to fall back to. * Unsatisfiable data requirements are a COLLECTION-time abort, not a run-time failure. Finding out on test #47 that the stand was never seeded wastes the @@ -214,7 +214,7 @@ def pytest_collection_modifyitems(config: pytest.Config, items: list[pytest.Item * capability markers — a missing capability is a legitimate property of this stand, not a defect, so it skips that item alone. * `requires_catalogue` — same resolution as a capability, for the rows - `deploy/seed/analytics.py` writes. A stand seeded without that step is a + `src/ingestion/tools/seed/analytics.py` writes. A stand seeded without that step is a real state, and a test that needs those rows must say so rather than assert against an empty universe and pass for the wrong reason. """ diff --git a/tests/versions.yaml b/tests/versions.yaml index 250aa9724..a4e7d5d30 100644 --- a/tests/versions.yaml +++ b/tests/versions.yaml @@ -25,7 +25,7 @@ tests_stand: dependency_manager: "uv" lockfile: "tests/uv.lock" notes: >- - The Python floor matches deploy/seed/pyproject.toml, whose manifest and PROFILE renderer this suite reads. uv provisions that interpreter in every runner, including inside the ui-tests image whose own system Python is older, so the floor holds everywhere rather than being a host-only aspiration. Still outstanding: every GitHub Actions workflow pins python-version "3.12", below this floor — reconciling that is CI enablement's job, not a reason to lower the suite. #magic___^_^___line + This suite owns its Python floor: it reads the seeder's manifest as data and imports none of its modules, so the seeder's lower 3.12 floor (it ships inside the 3.12 toolbox image) does not bind here. uv provisions that interpreter in every runner, including inside the ui-tests image whose own system Python is older, so the floor holds everywhere rather than being a host-only aspiration. Still outstanding: every GitHub Actions workflow pins python-version "3.12", below this floor — reconciling that is CI enablement's job, not a reason to lower the suite. #magic___^_^___line # Browser runner image for tests/stand — deploy/compose/ui-tests.Dockerfile. # Both images pinned by tag AND digest so a rebuild cannot drift. Resolved # 2026-07-31 from the registry tag lists, cross-checked against the From 03f50cac2fa464d1150c675a1d6ad21b1a8467a2 Mon Sep 17 00:00:00 2001 From: Konstantin Tursunov Date: Thu, 6 Aug 2026 12:04:47 +0800 Subject: [PATCH 02/10] refactor(seed): install the seeder as a package instead of running it by path MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit Both images now `pip install` the seeder, so every runner invokes a program on PATH — `insight-seed ` — instead of a module resolved out of a working directory. The Job's command loses its shell wrapper and its `workingDir`, and the toolbox stops listing the database drivers by hand: they come from the package's own metadata, which cannot drift from it. That removes the last two places reaching into the source tree by path: * The Keycloak realm generator moves into the package as `insight_seed.keycloak_realm`. It was building the realm from the seeder's roster through a `sys.path.insert`, which is what a shared roster looks like when the two halves live apart — a realm and a person table that disagree produce a login that authenticates and then resolves to nobody. It ships as a second entry point, `insight-seed-realm`, and `dev-compose.sh` runs it through `uv run --project` rather than reaching for the directory. * The test bootstrap is gone. `tests/` imports `insight_seed` like anything else does, with no `sys.path` surgery and no stubbed driver modules, because the package is installed in the environment the tests run in. dbt moves from a hard dependency to a `silver` extra. No module here imports it: the silver step shells out to the ingestion tree's scripts, and dbt belongs to THAT environment — the toolbox image installs it directly, and the extra covers a host run. As a hard dependency it made every install of this package, including the one a realm generation needs, resolve dbt-core. Installing the package also broke an assumption the generated artifacts were resting on: `manifest.json` and `PROFILE.md` were anchored to the package's own directory, which is wherever pip put it — for the cluster Job, site-packages, where the write failed with EACCES after the seed had already run. Both now resolve against the working directory, with `SEED_MANIFEST_PATH` naming the manifest explicitly where that is not writable (the Job points it at /tmp, and its log carries the document anyway). Compose is unaffected: its working directory is the bind-mounted seeder directory the stand suite reads. Two more, from running it: * `seed-stand.sh` takes `--context` and prints the resolved context in its banner. It inherited the ambient kube context, and an ambient context can change between two runs of a script that applies Jobs to clusters. * The identity existence check no longer formats a column name into its SQL. Two complete statements selected by key: every statement the module executes is now a constant, which is the only form a reader — or a scanner — can confirm at a glance. Signed-off-by: Konstantin Tursunov --- CONTRIBUTING.md | 8 +- deploy/compose/keycloak/README.md | 6 +- dev-compose.sh | 18 ++-- docker-compose.yml | 2 +- src/ingestion/tools/seed/Dockerfile | 12 ++- src/ingestion/tools/seed/PROFILE.md | 2 +- src/ingestion/tools/seed/README.md | 51 +++++++---- .../tools/seed/insight_seed/__main__.py | 17 ++-- .../tools/seed/insight_seed/config.py | 16 ++++ .../tools/seed/insight_seed/identity.py | 50 +++++++---- .../tools/seed/insight_seed/keycloak_realm.py | 86 ++++++++++++------- .../tools/seed/insight_seed/manifest.py | 39 ++++----- .../tools/seed/insight_seed/profile_md.py | 9 +- .../tools/seed/insight_seed/profiles.py | 4 +- src/ingestion/tools/seed/pyproject.toml | 22 +++-- src/ingestion/tools/seed/seed-job.yaml.tpl | 16 ++-- src/ingestion/tools/seed/seed-stand.sh | 59 +++++++++---- src/ingestion/tools/seed/tests/conftest.py | 39 --------- .../tools/seed/tests/test_identity.py | 13 ++- .../tools/seed/tests/test_preflight.py | 44 +++++++--- src/ingestion/tools/toolbox/Dockerfile | 25 +++--- tests/lib/insight_stand/manifest.py | 2 +- tests/lib/insight_stand/personas.py | 2 +- 23 files changed, 326 insertions(+), 216 deletions(-) rename deploy/compose/keycloak/gen-realm.py => src/ingestion/tools/seed/insight_seed/keycloak_realm.py (81%) mode change 100755 => 100644 delete mode 100644 src/ingestion/tools/seed/tests/conftest.py diff --git a/CONTRIBUTING.md b/CONTRIBUTING.md index 803c7b59d..bf23dc1eb 100644 --- a/CONTRIBUTING.md +++ b/CONTRIBUTING.md @@ -458,9 +458,11 @@ for the per-environment IdP selection rationale. The seeder lives in [`src/ingestion/tools/seed/`](src/ingestion/tools/seed/): the `insight_seed` package, its `tests/`, and the artifacts it writes. Its -README documents the layout and the ruff / mypy / venv setup. Both deploy paths -run the same package (`python3 -m insight_seed `); only how it is invoked -differs. +README documents the layout and the uv / ruff / mypy setup. Both deploy paths +install the package and run the same program (`insight-seed `); only how +it is invoked differs. Generating the compose Keycloak realm also runs from that +package (`insight-seed-realm`, via `uv run`), so `uv` is a prerequisite for +`./dev-compose.sh up --auth=keycloak`. **Identity content (after `seed identity`):** CEO, your `DEV_USER_EMAIL` person (leads the dev team), 4 team leads (dev / diff --git a/deploy/compose/keycloak/README.md b/deploy/compose/keycloak/README.md index 97940c756..85ba86bac 100644 --- a/deploy/compose/keycloak/README.md +++ b/deploy/compose/keycloak/README.md @@ -31,7 +31,7 @@ using the pre-seeded `insight-authenticator` client + dev secret). On `up`, `dev-compose.sh`: -- generates the realm from the seed roster (`gen-realm.py`) and starts Keycloak +- generates the realm from the seed roster (`insight_seed.keycloak_realm`) and starts Keycloak (single container, `:8085`, profile `auth-keycloak`); - points the **authenticator** at the realm's `insight-authenticator` confidential client — exporting `KEYCLOAK_HOSTNAME` + `AUTHENTICATOR_OIDC_ISSUER` to the @@ -82,9 +82,9 @@ one and only one tenant per token. artifact rebuilt from the seed roster on every `up`. To change realm shape, edit the generator's inputs: -- [`src/ingestion/tools/seed/profiles.py`](../../../src/ingestion/tools/seed/profiles.py) — the roster (`build_roster`), +- [`insight_seed/profiles.py`](../../../src/ingestion/tools/seed/insight_seed/profiles.py) — the roster (`build_roster`), team/role assignments, dev-lead email resolution. -- [`gen-realm.py`](./gen-realm.py) — the realm generator (clients, protocol mappers, +- [`insight_seed/keycloak_realm.py`](../../../src/ingestion/tools/seed/insight_seed/keycloak_realm.py) — the realm generator (clients, protocol mappers, role mapping). The `insight-authenticator` client redirect + secret are parameters (`--authenticator-redirect`, `--authenticator-secret`) so k8s can seed the same realm with its own ingress callback. diff --git a/dev-compose.sh b/dev-compose.sh index 6e83791b2..7e99ef0b1 100755 --- a/dev-compose.sh +++ b/dev-compose.sh @@ -379,7 +379,7 @@ cmd_up() { local build_only_csv="" local frontend_mode_override="" local instance="$COMPOSE_INSTANCE" - # Repeatable. Empty => gen-realm.py keeps its own defaults untouched. + # Repeatable. Empty => the realm generator keeps its own defaults untouched. local authenticator_redirects="" local skip_build=false local no_frontend=false @@ -618,7 +618,7 @@ YML local kc_base="http://${kc_ip:-localhost}:8085/kc" echo "=== Generating Keycloak realm import (deploy/compose/keycloak/realm-insight.generated.json) ===" - # gen-realm.py's own --authenticator-redirect REPLACES its defaults rather + # The generator's own --authenticator-redirect REPLACES its defaults rather # than appending, so whenever we pass any URI we must re-state the two # defaults too — dropping them would deregister the human login origins # and break `./dev-compose.sh up`. @@ -632,11 +632,19 @@ YML done echo " registering redirect URIs:$redirect_args" fi + # The realm is built from the seeder's roster, so the generator ships in + # that package and runs as an installed program. uv provisions the package + # into its own .venv on first use — the same tool the stand suite already + # requires — instead of this script reaching into the source tree. + command -v uv >/dev/null 2>&1 || { + echo "ERROR: uv is required to generate the Keycloak realm." >&2 + echo " Install it (brew install uv) and re-run; see CONTRIBUTING.md." >&2 + return 1; } # shellcheck disable=SC2086 # redirect_args is a deliberately word-split flag list - python3 deploy/compose/keycloak/gen-realm.py \ + uv run --project "$ROOT_DIR/src/ingestion/tools/seed" insight-seed-realm \ --dev-email "$dev_lead_email" \ $redirect_args \ - --out deploy/compose/keycloak/realm-insight.generated.json + --out "$ROOT_DIR/deploy/compose/keycloak/realm-insight.generated.json" # NGINX_BFF: the AUTHENTICATOR (not the frontend) logs in against Keycloak, # server-side, as the pre-seeded `insight-authenticator` confidential client. @@ -651,7 +659,7 @@ YML export OIDC_CLIENT_ID="${OIDC_CLIENT_ID:-insight-authenticator}" export OIDC_CLIENT_SECRET="${OIDC_CLIENT_SECRET:-insight-authenticator-dev-secret}" # The login-bootstrap resolve is scoped to idp.source_type; keycloak's - # sub differs in KIND from fakeidp's (gen-realm.py sets each realm user's + # sub differs in KIND from fakeidp's (keycloak_realm sets each realm user's # id to their OWN roster uuid, so sub IS that uuid — not the fixed # "fakeidp|dev" string fakeidp issues), so it must be seeded/looked-up # under its own source_type, not the fakeidp default (see diff --git a/docker-compose.yml b/docker-compose.yml index cd9b4a9dc..4c9cb3346 100644 --- a/docker-compose.yml +++ b/docker-compose.yml @@ -468,7 +468,7 @@ services: # # Single container, embedded H2, imports the roster-generated realm at # deploy/compose/keycloak/realm-insight.generated.json (produced by - # deploy/compose/keycloak/gen-realm.py, gitignored). Gated behind + # insight_seed.keycloak_realm, gitignored). Gated behind # --profile auth-keycloak; mutually exclusive with fakeidp's auth-fakeidp # profile. keycloak: diff --git a/src/ingestion/tools/seed/Dockerfile b/src/ingestion/tools/seed/Dockerfile index b21562c0d..5c826ef04 100644 --- a/src/ingestion/tools/seed/Dockerfile +++ b/src/ingestion/tools/seed/Dockerfile @@ -34,12 +34,16 @@ RUN apt-get update \ # `docker build && docker run` without a volume. COPY pyproject.toml . COPY insight_seed ./insight_seed/ -RUN pip install . + +# EDITABLE, unlike the toolbox image's install: compose mounts the working tree +# over /app so seeder edits need no rebuild, and a regular install would keep +# executing the copy baked in above while the developer edited a file that never +# ran. The silver extra brings dbt, which this image's step runs. +RUN pip install -e '.[silver]' RUN useradd -U -u 1000 -m appuser USER 1000 -# `-m` rather than a path: /app is the package's parent both in the image and -# under the bind mount, so the same invocation works either way. -ENTRYPOINT ["python", "-m", "insight_seed"] +# A program on PATH, installed above — same entry point as the cluster Job. +ENTRYPOINT ["insight-seed"] CMD ["all"] diff --git a/src/ingestion/tools/seed/PROFILE.md b/src/ingestion/tools/seed/PROFILE.md index ca630a306..a55502bb0 100644 --- a/src/ingestion/tools/seed/PROFILE.md +++ b/src/ingestion/tools/seed/PROFILE.md @@ -17,7 +17,7 @@ builder that writes `manifest.json`, so the two cannot disagree. | realm | `insight` | | anchor_date | `2026-06-30` | | data_window | `2026-05-02..2026-06-30` | -| seed_revision | `4d34657f4b488c01` | +| seed_revision | `c8aea877df6976da` | | manifest_version | 1 | `anchor_date` is the last day carrying seeded activity. It is resolved diff --git a/src/ingestion/tools/seed/README.md b/src/ingestion/tools/seed/README.md index 7059f7e0c..09a1758c8 100644 --- a/src/ingestion/tools/seed/README.md +++ b/src/ingestion/tools/seed/README.md @@ -26,8 +26,11 @@ The stack must be up first (`./dev-compose.sh up`). Then: ./dev-compose.sh seed silver # just silver ``` -A successful run writes `manifest.json` next to this README, describing the -stand it just produced (roster, fixtures, data window, capabilities). +A successful run writes `manifest.json` describing the stand it just produced +(roster, fixtures, data window, capabilities). It lands in the working +directory — for the compose service that is the bind-mounted seeder directory, +which is where the stand test suite reads it; `SEED_MANIFEST_PATH` names it +explicitly anywhere else. ## Run it on a Kubernetes stand @@ -119,26 +122,35 @@ loop stays populated as the calendar moves. Whichever applied is recorded in roster or the manifest builder: ```bash -cd src/ingestion/tools/seed -python3 -m insight_seed.render_profile # regenerate -python3 -m insight_seed.render_profile --check # verify (no database needed) +cd src/ingestion/tools/seed # the page lives here +uv run python -m insight_seed.render_profile # regenerate +uv run python -m insight_seed.render_profile --check # verify (no database needed) ``` ## Develop on it ```bash cd src/ingestion/tools/seed -python3 -m venv .venv # one-time -.venv/bin/pip install -e '.[dev]' -.venv/bin/ruff check . # package + tests -.venv/bin/mypy . -python3 -m unittest discover -s tests -t . # stdlib only, no database +uv run --extra dev python -m unittest discover -s tests -t . # tests +uv run --extra dev ruff check . # package + tests +uv run --extra dev mypy . ``` -The tests need nothing installed: they stub the database drivers and exercise -the pure half — the environment contract, the SQL a guard issues, and the -messages a refusal carries. +`uv` resolves and installs the package into a local `.venv` on first use, so +the tests import `insight_seed` the same way anything else does — no +`sys.path` juggling and no stubbed modules. A hand-made venv works identically +(`python3 -m venv .venv && .venv/bin/pip install -e '.[dev]'`). + +Both images install the package too, so every runner invokes a program rather +than a module in a directory: `insight-seed ` seeds, and +`insight-seed-realm` generates the compose Keycloak realm from the same roster +(`dev-compose.sh` runs it through `uv run --project`). The extras split what +each caller needs: `silver` adds dbt for the gold build (the toolbox image +installs it separately), `dev` adds ruff, mypy and stubs. + +The tests touch no database: they cover the pure half — the environment +contract, the SQL a guard issues, and the messages a refusal carries. Deps live in `pyproject.toml`: `[project.dependencies]` for runtime, `[project.optional-dependencies].dev` for the tooling (ruff, mypy, stubs). @@ -152,7 +164,7 @@ compose bind mount) name them. ```text src/ingestion/tools/seed/ ├── insight_seed/ the package — everything importable -│ ├── __main__.py `python3 -m insight_seed `: the entry point +│ ├── __main__.py the `insight-seed ` entry point │ ├── config.py environment contract: required, defaulted, and why │ ├── preflight.py refuses a stand that cannot take the seed │ ├── identity.py MariaDB: persons, org_chart, account_person_map @@ -163,8 +175,9 @@ src/ingestion/tools/seed/ │ ├── golden_metrics.py the only source for the manifest's golden set │ ├── profile_md.py renders `PROFILE.md` from a manifest │ ├── render_profile.py regenerates / verifies `PROFILE.md`; no database +│ ├── keycloak_realm.py the `insight-seed-realm` entry point, same roster │ └── generators/ one module per activity domain, `base.py` shared -├── tests/ stdlib unittest; drivers stubbed in `conftest.py` +├── tests/ stdlib unittest against the installed package ├── seed-stand.sh seeds a Kubernetes stand (discover → render → apply) ├── seed-job.yaml.tpl the Job it renders — and the reference manifest ├── Dockerfile the compose `seed-sample` image @@ -173,7 +186,7 @@ src/ingestion/tools/seed/ └── manifest.json GENERATED per stand at seed time (gitignored) ``` -On a cluster the runtime is the toolbox image (`../toolbox/Dockerfile`), which -carries this tree at `/ingestion/tools/seed` together with the migration -scripts the silver step runs — so the Job's command is the same -`python -m insight_seed` you would run locally. +On a cluster the runtime is the toolbox image (`../toolbox/Dockerfile`): it +carries this tree at `/ingestion/tools/seed` together with the migration scripts +the silver step runs, and installs the package, so the Job's command is just +`insight-seed` — no shell, no working directory, no path assumptions. diff --git a/src/ingestion/tools/seed/insight_seed/__main__.py b/src/ingestion/tools/seed/insight_seed/__main__.py index bc090536e..c4eb4914e 100644 --- a/src/ingestion/tools/seed/insight_seed/__main__.py +++ b/src/ingestion/tools/seed/insight_seed/__main__.py @@ -30,9 +30,10 @@ def main(argv: list[str] | None = None) -> int: parser = argparse.ArgumentParser( - # Named for how it is invoked, not for the file argparse found itself in - # — `__main__.py` in a usage line tells a reader nothing. - prog="python3 -m insight_seed", + # The installed console script, not the file argparse found itself in: + # every runner installs this package, and `__main__.py` in a usage line + # tells a reader nothing. + prog="insight-seed", description=__doc__, formatter_class=argparse.RawDescriptionHelpFormatter, ) @@ -108,12 +109,12 @@ def main(argv: list[str] | None = None) -> int: path = write_manifest(doc) LOG.info("manifest written: %s", path) except OSError as exc: - # The seed container has historically mounted /app read-only. Fail - # loudly rather than leaving downstream consumers reading a stale - # manifest from a previous run. + # Loud rather than skipped: a consumer reading a stale manifest from a + # previous run is worse than a failed one. The document is already on + # stdout above, so nothing is lost but the file. raise RuntimeError( - f"could not write {manifest_path()}: {exc}. The seed source mount " - "must be writable — see docker-compose.yml seed-sample.volumes." + f"could not write {manifest_path()}: {exc}. Run from a writable directory, " + "or set SEED_MANIFEST_PATH to somewhere this process can write." ) from exc return 0 diff --git a/src/ingestion/tools/seed/insight_seed/config.py b/src/ingestion/tools/seed/insight_seed/config.py index 413e3b220..883002087 100644 --- a/src/ingestion/tools/seed/insight_seed/config.py +++ b/src/ingestion/tools/seed/insight_seed/config.py @@ -19,6 +19,7 @@ import uuid as uuid_mod from collections.abc import Mapping from dataclasses import dataclass, replace +from pathlib import Path #: Prefix on the `reason` of every identity row this seeder writes. It is what #: lets a preflight check tell demo rows from rows some other writer owns, so @@ -31,6 +32,7 @@ CROSS_TENANT_FIXTURE_ENV = "SEED_CROSS_TENANT_FIXTURE" FORCE_ENV = "SEED_FORCE" ANCHOR_ENV = "SEED_ANCHOR_DATE" +MANIFEST_PATH_ENV = "SEED_MANIFEST_PATH" DAYS_ENV = "SEED_DAYS" DEV_USER_EMAIL_ENV = "DEV_USER_EMAIL" @@ -208,6 +210,20 @@ def parse_seed_days(env: Mapping[str, str], default: int = DEFAULT_SEED_DAYS) -> return days +def parse_manifest_path(env: Mapping[str, str]) -> Path: + """Where a run writes its manifest. + + The working directory by default, NOT a path derived from this module's + location: the package is installed (into the toolbox image, into a venv), so + its own directory is wherever pip put it — writing there means writing into + site-packages. The compose service runs with the seeder's directory as its + working directory, so the default lands exactly where the stand suite reads + it, and a cluster Job points the variable at somewhere writable. + """ + raw = (env.get(MANIFEST_PATH_ENV) or "").strip() + return Path(raw) if raw else Path.cwd() / "manifest.json" + + def parse_dev_user_email(env: Mapping[str, str]) -> str: """The persona the dev-lead login resolves to. Required by every roster build.""" raw = (env.get(DEV_USER_EMAIL_ENV) or "").strip().lower() diff --git a/src/ingestion/tools/seed/insight_seed/identity.py b/src/ingestion/tools/seed/insight_seed/identity.py index 63c869faa..c0078fb13 100644 --- a/src/ingestion/tools/seed/insight_seed/identity.py +++ b/src/ingestion/tools/seed/insight_seed/identity.py @@ -74,9 +74,36 @@ def _connect() -> Iterator[pymysql.connections.Connection]: conn.close() -#: The two columns an observation's value lands in, by kind. `persons` is an EAV -#: log: identifier-shaped values go in `value_id`, free text in `value_full_text`. -_VALUE_COLUMNS = ("value_id", "value_full_text") +#: One complete statement per value column, spelled out rather than composed. +#: `persons` is an EAV log — identifier-shaped values land in `value_id`, free +#: text in `value_full_text` — and a column name cannot be a bound parameter, so +#: the alternative is formatting one into the SQL. Two literals keep every +#: statement this module executes a constant, which is the only version a reader +#: (or a scanner) can confirm at a glance. +_EXISTS_BY_VALUE_ID = """ + SELECT 1 FROM persons + WHERE insight_tenant_id = %s + AND person_id = %s + AND insight_source_type = %s + AND insight_source_id = %s + AND value_type = %s + AND value_id = %s + LIMIT 1 +""" +_EXISTS_BY_VALUE_FULL_TEXT = """ + SELECT 1 FROM persons + WHERE insight_tenant_id = %s + AND person_id = %s + AND insight_source_type = %s + AND insight_source_id = %s + AND value_type = %s + AND value_full_text = %s + LIMIT 1 +""" +_EXISTS_SQL: dict[str, str] = { + "value_id": _EXISTS_BY_VALUE_ID, + "value_full_text": _EXISTS_BY_VALUE_FULL_TEXT, +} def _observation_exists( @@ -94,20 +121,13 @@ def _observation_exists( carries `created_at`, so a re-run's insert never collides and IGNORE stopped deduplicating anything. The logical key — ignoring `created_at` — is what makes a re-run a no-op. + + Indexing `_EXISTS_SQL` rather than validating a name: an unknown column is a + `KeyError` before any statement exists, and the statement that does run was + written out in full above. """ - if value_column not in _VALUE_COLUMNS: - raise ValueError(f"{value_column!r} is not an observation value column") cur.execute( - f""" - SELECT 1 FROM persons - WHERE insight_tenant_id = %s - AND person_id = %s - AND insight_source_type = %s - AND insight_source_id = %s - AND value_type = %s - AND `{value_column}` = %s - LIMIT 1 - """, + _EXISTS_SQL[value_column], ( _bin(tenant_uuid), _bin(person_uuid), diff --git a/deploy/compose/keycloak/gen-realm.py b/src/ingestion/tools/seed/insight_seed/keycloak_realm.py old mode 100755 new mode 100644 similarity index 81% rename from deploy/compose/keycloak/gen-realm.py rename to src/ingestion/tools/seed/insight_seed/keycloak_realm.py index 5f355df11..1f92589e4 --- a/deploy/compose/keycloak/gen-realm.py +++ b/src/ingestion/tools/seed/insight_seed/keycloak_realm.py @@ -1,16 +1,20 @@ -#!/usr/bin/env python3 """Generate the `insight` Keycloak realm from the seeded 25-person org. -Reads the same roster builder the DB seeder uses -(`src/ingestion/tools/seed/profiles.py::build_roster`) so every user in the realm -matches a row in `identity.persons`, then emits an importable Keycloak -realm JSON: 26 users (the 25-person org plus the admin operator), the -`insight` + `insight-authenticator` clients, their 5 shared protocol -mappers, the 4 team groups + `executive` + `operations`, and the 3 +Built from the SAME roster the database seed writes (`profiles.build_roster`), +so every user in the realm matches a row in `identity.persons`, and emits an +importable Keycloak realm JSON: 26 users (the 25-person org plus the admin +operator), the `insight` + `insight-authenticator` clients, their 5 shared +protocol mappers, the 4 team groups + `executive` + `operations`, and the 3 realm roles. -Usage: - python3 gen-realm.py --out deploy/compose/keycloak/realm-insight.generated.json +It lives in this package rather than beside the compose Keycloak assets it +feeds precisely because of that shared roster: a realm and a person table that +disagree produce a login that authenticates and then resolves to nobody. One +module owns the roster; everything derived from it imports it normally. + +Usage, from the tool directory (`src/ingestion/tools/seed`): + + python3 -m insight_seed.keycloak_realm --out /realm-insight.generated.json """ from __future__ import annotations @@ -18,16 +22,10 @@ import argparse import json import os -import sys from pathlib import Path +from typing import Any -# The seed package (not installed) has to be put on sys.path explicitly to -# import it from this script's location. parents[3] = repo root; the entry -# added is the package's PARENT, so `insight_seed` imports as itself. -_SEED_TOOL_DIR = Path(__file__).resolve().parents[3] / "src/ingestion/tools/seed" -sys.path.insert(0, str(_SEED_TOOL_DIR)) - -from insight_seed.profiles import ( # noqa: E402 +from .profiles import ( TENANT_OTHER, Person, build_other_tenant_roster, @@ -72,9 +70,13 @@ def _org_unit(person: Person) -> str: return OPERATOR_ORG_UNIT if person.role == "admin" else "executive" -def _protocol_mappers(tenant_id: str) -> list[dict]: +def _protocol_mappers(tenant_id: str) -> list[dict[str, Any]]: """The 5 shared mappers, identical on both clients (Input table).""" - common = {"id.token.claim": "true", "access.token.claim": "true", "userinfo.token.claim": "true"} + common = { + "id.token.claim": "true", + "access.token.claim": "true", + "userinfo.token.claim": "true", + } return [ { # The authenticator reads this single-string claim (idp.tenant_claim, @@ -95,14 +97,24 @@ def _protocol_mappers(tenant_id: str) -> list[dict]: "protocol": "openid-connect", "protocolMapper": "oidc-usermodel-attribute-mapper", "consentRequired": False, - "config": {**common, "user.attribute": "tenant_id", "claim.name": "tenant_id", "jsonType.label": "String"}, + "config": { + **common, + "user.attribute": "tenant_id", + "claim.name": "tenant_id", + "jsonType.label": "String", + }, }, { "name": "org_unit", "protocol": "openid-connect", "protocolMapper": "oidc-usermodel-attribute-mapper", "consentRequired": False, - "config": {**common, "user.attribute": "org_unit", "claim.name": "org_unit", "jsonType.label": "String"}, + "config": { + **common, + "user.attribute": "org_unit", + "claim.name": "org_unit", + "jsonType.label": "String", + }, }, { "name": "groups", @@ -116,19 +128,28 @@ def _protocol_mappers(tenant_id: str) -> list[dict]: "protocol": "openid-connect", "protocolMapper": "oidc-usermodel-realm-role-mapper", "consentRequired": False, - "config": {**common, "claim.name": "roles", "jsonType.label": "String", "multivalued": "true"}, + "config": { + **common, + "claim.name": "roles", + "jsonType.label": "String", + "multivalued": "true", + }, }, { "name": "aud-insight", "protocol": "openid-connect", "protocolMapper": "oidc-audience-mapper", "consentRequired": False, - "config": {"id.token.claim": "false", "access.token.claim": "true", "included.client.audience": "insight"}, + "config": { + "id.token.claim": "false", + "access.token.claim": "true", + "included.client.audience": "insight", + }, }, ] -def _client_insight(tenant_id: str) -> dict: +def _client_insight(tenant_id: str) -> dict[str, Any]: return { "clientId": "insight", "publicClient": True, @@ -146,7 +167,9 @@ def _client_insight(tenant_id: str) -> dict: } -def _client_insight_authenticator(tenant_id: str, redirect_uris: list[str], secret: str) -> dict: +def _client_insight_authenticator( + tenant_id: str, redirect_uris: list[str], secret: str +) -> dict[str, Any]: return { "clientId": "insight-authenticator", "publicClient": False, @@ -166,7 +189,7 @@ def _client_insight_authenticator(tenant_id: str, redirect_uris: list[str], secr } -def _user(person: Person, tenant_id: str) -> dict: +def _user(person: Person, tenant_id: str) -> dict[str, Any]: org_unit = _org_unit(person) # Refused rather than defaulted: a user whose `tenant_id` attribute is # missing logs in and gets a token with no tenant claim, which fails deep @@ -197,8 +220,11 @@ def _user(person: Person, tenant_id: str) -> dict: def build_realm( - dev_user_email: str, tenant_id: str, authenticator_redirects: list[str], authenticator_secret: str -) -> dict: + dev_user_email: str, + tenant_id: str, + authenticator_redirects: list[str], + authenticator_secret: str, +) -> dict[str, Any]: roster = build_roster(dev_user_email) # The second tenant's single person, in the SAME realm. One realm because # the authenticator is configured with one issuer and one client — a second @@ -229,7 +255,9 @@ def main() -> None: parser.add_argument( "--dev-email", default=None, - help=("Roster-anchor email for the dev-lead persona. Falls back to DEV_USER_EMAIL when omitted."), + help=( + "Roster-anchor email for the dev-lead persona. Falls back to DEV_USER_EMAIL when omitted." + ), ) parser.add_argument( "--authenticator-redirect", diff --git a/src/ingestion/tools/seed/insight_seed/manifest.py b/src/ingestion/tools/seed/insight_seed/manifest.py index 0b9a1ae34..846812be9 100644 --- a/src/ingestion/tools/seed/insight_seed/manifest.py +++ b/src/ingestion/tools/seed/insight_seed/manifest.py @@ -1,9 +1,9 @@ """Seed manifest — the machine-readable description of a seeded stand. -Written to one fixed path — `manifest.json` at the tool root, beside this -package (`/app/manifest.json` in the compose seed container, the same file -through the bind mount). Downstream test phases read that path; it is frozen -and has no env knob. +Written to `manifest.json` in the working directory, which for the compose +seed container is the bind-mounted seeder directory the stand suite reads +(`/app/manifest.json`). `SEED_MANIFEST_PATH` names it explicitly — a cluster +Job does, because its filesystem is discarded and the log is the record. The builder is a PURE FUNCTION of (roster, supplied env, committed constants). It queries neither MariaDB nor ClickHouse. That is what lets the @@ -31,12 +31,12 @@ MANIFEST_VERSION = 1 -# Values copied verbatim from deploy/compose/keycloak/gen-realm.py, which -# builds the Keycloak realm from this same roster. The two must agree exactly -# or a persona will authenticate as someone the API does not recognise. -REALM_NAME = "insight" # gen-realm.py REALM_NAME -EXECUTIVE_ORG_UNIT = "executive" # gen-realm.py _org_unit, teamless -OPERATOR_ORG_UNIT = "operations" # gen-realm.py OPERATOR_ORG_UNIT +# Values shared with `keycloak_realm`, which builds the Keycloak realm from +# this same roster. The two must agree exactly or a persona will authenticate +# as someone the API does not recognise. +REALM_NAME = "insight" # keycloak_realm.REALM_NAME +EXECUTIVE_ORG_UNIT = "executive" # keycloak_realm._org_unit, teamless +OPERATOR_ORG_UNIT = "operations" # keycloak_realm.OPERATOR_ORG_UNIT ROLE_TO_REALM_ROLES: dict[str, list[str]] = { "ceo": ["insight-admin", "insight-lead"], "lead": ["insight-lead"], @@ -90,27 +90,20 @@ # instead of shipping. _FORBIDDEN_LITERALS = frozenset( { - "insight-dev", # gen-realm.py dev password - "insight-authenticator-dev-secret", # gen-realm.py client secret + "insight-dev", # keycloak_realm dev password + "insight-authenticator-dev-secret", # keycloak_realm client secret "insight-local", # MariaDB / ClickHouse dev password "root-local", # MariaDB root password } ) _FORBIDDEN_KEY_SUBSTRINGS = ("password", "secret", "token", "credential", "passwd") -#: The tool directory: the package's home, holding the artifacts it writes -#: (`manifest.json`) and the ones it renders (`PROFILE.md`). -_TOOL_ROOT = Path(__file__).resolve().parents[1] - def manifest_path() -> Path: - """The frozen manifest location. No env knob by design. + """Where this run writes its manifest — see `config.parse_manifest_path`.""" + import os - The tool directory, not the package directory: the manifest is a per-stand - artifact this package produces, and its readers (the stand suite, the - compose bind mount) name that path. - """ - return _TOOL_ROOT / "manifest.json" + return config.parse_manifest_path(os.environ) # The window comes from `config`, the same reader the generators use, so the @@ -145,7 +138,7 @@ def seed_revision() -> str: def _persona(person: profiles.Person) -> dict[str, Any]: """One roster entry. Fields are named explicitly, never spread from the Person object, so a future attribute cannot leak into the document.""" - # Mirrors gen-realm.py's `_org_unit`: teamless people are the CEO + # Mirrors `keycloak_realm._org_unit`: teamless people are the CEO # (executive) and the admin operator (operations, its own unit because it # administers the product rather than belonging to the org). if person.team is not None: diff --git a/src/ingestion/tools/seed/insight_seed/profile_md.py b/src/ingestion/tools/seed/insight_seed/profile_md.py index 7bfbb5c06..90f67fc9a 100644 --- a/src/ingestion/tools/seed/insight_seed/profile_md.py +++ b/src/ingestion/tools/seed/insight_seed/profile_md.py @@ -24,8 +24,13 @@ def profile_path() -> Path: - """The committed profile page, at the tool root beside the README.""" - return Path(__file__).resolve().parents[1] / "PROFILE.md" + """The committed profile page, in the working directory. + + Not derived from this module's location: an installed package lives wherever + pip put it, and this page belongs to the checkout. `render_profile` is a + developer tool run from the seeder's directory (see the README). + """ + return Path.cwd() / "PROFILE.md" def _table(headers: list[str], rows: list[list[str]]) -> list[str]: diff --git a/src/ingestion/tools/seed/insight_seed/profiles.py b/src/ingestion/tools/seed/insight_seed/profiles.py index f79286787..66defff04 100644 --- a/src/ingestion/tools/seed/insight_seed/profiles.py +++ b/src/ingestion/tools/seed/insight_seed/profiles.py @@ -28,7 +28,7 @@ # The dev lead's UUID matches the value the original dev-compose.sh seed # inserts, so re-runs across both scripts converge on the same row. # Default tenant for the demo organisation. Mirrors TENANT_DEFAULT_ID in -# docker-compose.yml and deploy/compose/keycloak/gen-realm.py. +# docker-compose.yml and `keycloak_realm`. TENANT_DEFAULT = "00000000-df51-5b42-9538-d2b56b7ee953" # A SECOND tenant, holding exactly one person and nothing else. @@ -397,7 +397,7 @@ def get_login_id_pairs(roster: list[Person]) -> list[tuple[str, str]]: dev-compose.sh's `--auth` flag (`AUTH_MODE`, forwarded into this container's environment) selects which IdP fixture applies, and the two are NOT symmetric in how many personas can actually log in: - - keycloak: gen-realm.py sets EVERY realm user's Keycloak `id` to that + - keycloak: `keycloak_realm` sets EVERY realm user's Keycloak `id` to that person's OWN roster UUID (`"id": person.uuid`), and Keycloak issues `sub` equal to the user's internal id verbatim — so every seeded persona's external id IS their own roster uuid, and the whole roster diff --git a/src/ingestion/tools/seed/pyproject.toml b/src/ingestion/tools/seed/pyproject.toml index e80ade156..8eaed288b 100644 --- a/src/ingestion/tools/seed/pyproject.toml +++ b/src/ingestion/tools/seed/pyproject.toml @@ -16,15 +16,22 @@ description = "Insight sample-data seeder for the compose stack and Kubernetes t # ingestion tree pins the same direction (tests/connectors caps at <3.13). No # 3.13-only syntax or API is used here, so the floor costs nothing. requires-python = ">=3.12" +# What the package IMPORTS. dbt is deliberately absent: no module here imports +# it — the silver step shells out to the ingestion tree's own scripts, and dbt +# is a requirement of THAT environment (the toolbox image installs it directly, +# and the `silver` extra below covers a host run). Declaring it here made every +# install of this package resolve dbt-core, including the one a realm +# generation needs. dependencies = [ "PyMySQL==1.2.0", "clickhouse-connect==1.6.0", - # Unpinned to match src/ingestion/tools/toolbox/Dockerfile. dbt-core pulls - # in PyYAML, which apply-ch-migrations.sh's profiles.yml generator imports. - "dbt-clickhouse", ] [project.optional-dependencies] +# The silver step's environment for a host run outside the toolbox image: +# `apply-ch-migrations.sh` writes a profiles.yml with PyYAML (which dbt-core +# pulls in) and then runs dbt itself. +silver = ["dbt-clickhouse"] dev = [ # dbt-core 1.12 raised its own cap to pathspec<1.1 (dbt-labs/dbt-core#12385), # so a current mypy resolves again. Below 2.0 it would pull pathspec<1.0 and @@ -36,11 +43,14 @@ dev = [ "types-PyMySQL==1.2.0.20260724", ] -# The entry point every runner uses: `python3 -m insight_seed ` from this -# directory, which is also what the container images and the Kubernetes Job run. -# The console script is for an installed copy. +# The entry points every runner uses. Both images install this package, so the +# compose service, the Kubernetes Job and a developer shell all invoke a program +# on PATH rather than a module in a particular directory. [project.scripts] insight-seed = "insight_seed.__main__:main" +# The realm generator is a second entry point on the same roster, so compose +# bring-up can run it as a program rather than a module in a directory. +insight-seed-realm = "insight_seed.keycloak_realm:main" # One importable package; `tests/` stays out of the distribution. [tool.setuptools.packages.find] diff --git a/src/ingestion/tools/seed/seed-job.yaml.tpl b/src/ingestion/tools/seed/seed-job.yaml.tpl index 559c1031a..0ce453ede 100644 --- a/src/ingestion/tools/seed/seed-job.yaml.tpl +++ b/src/ingestion/tools/seed/seed-job.yaml.tpl @@ -62,12 +62,11 @@ spec: # IfNotPresent so a locally built image can be tried on a local # cluster without pushing it to a registry first. imagePullPolicy: IfNotPresent - # Run as a module from the tool directory, which puts the package's - # parent on sys.path without installing anything into the image. - command: [bash, -c] - args: - - exec python -m insight_seed ${SEED_STEP} - workingDir: /ingestion/tools/seed + # The package is installed in the image, so this is a program on PATH + # — no shell, no working directory, nothing that depends on where the + # source happens to sit. + command: [insight-seed] + args: [${SEED_STEP}] env: # ── MariaDB ────────────────────────────────────────────────── # The app user, not root: the umbrella grants it ALL on the @@ -127,6 +126,11 @@ spec: value: "${SEED_WINDOW_DAYS}" - name: SEED_ANCHOR_DATE value: "${SEED_ANCHOR_DATE}" + # The pod's filesystem dies with it and the manifest is echoed to + # the log, so this only has to be somewhere writable — the package + # is installed in the image and its working directory is not. + - name: SEED_MANIFEST_PATH + value: /tmp/manifest.json # dbt (run by the silver step's gold build) writes target/, logs/ # and ~/.dbt under whatever it is given; the toolbox image owns # /ingestion, but keep the scratch paths explicit. diff --git a/src/ingestion/tools/seed/seed-stand.sh b/src/ingestion/tools/seed/seed-stand.sh index 94e958d05..2fda07334 100755 --- a/src/ingestion/tools/seed/seed-stand.sh +++ b/src/ingestion/tools/seed/seed-stand.sh @@ -30,6 +30,7 @@ STEP="all" DEADLINE_SECONDS="3600" NAMESPACE="" +CONTEXT="" RELEASE="" DEV_EMAIL="" TENANT="" @@ -61,6 +62,7 @@ Required: the authenticator resolves people by the email claim. Discovered from the stand (pass a flag only to override): + --context kube context to act on [default: the current one] --release helm release name [default: same as -n] --tenant tenant every seeded row is scoped to --image toolbox image to run @@ -100,9 +102,27 @@ need() { command -v "$1" >/dev/null 2>&1 || die "$1 is required but not on PATH." } +# Every cluster call goes through these, so the target is whatever --context +# names rather than whatever the shell was last pointed at. An ambient context +# can change between two runs of this script; the cluster it writes to should +# not. +kube() { + if [[ -n "$CONTEXT" ]]; then kubectl --context "$CONTEXT" "$@"; else kubectl "$@"; fi +} + +helm_release() { + if [[ -n "$CONTEXT" ]]; then helm --kube-context "$CONTEXT" "$@"; else helm "$@"; fi +} + +# The copy-pasteable form of the above, for the hints this script prints. +kubectl_hint() { + if [[ -n "$CONTEXT" ]]; then echo "kubectl --context $CONTEXT"; else echo "kubectl"; fi +} + while [[ $# -gt 0 ]]; do case "$1" in -n|--namespace) NAMESPACE="${2:?--namespace needs a value}"; shift 2 ;; + --context) CONTEXT="${2:?--context needs a value}"; shift 2 ;; --release) RELEASE="${2:?--release needs a value}"; shift 2 ;; --email) DEV_EMAIL="${2:?--email needs a value}"; shift 2 ;; --tenant) TENANT="${2:?--tenant needs a value}"; shift 2 ;; @@ -144,26 +164,31 @@ if [[ -z "$RELEASE" ]]; then RELEASE="$NAMESPACE" fi -echo "==> stand: namespace=$NAMESPACE release=$RELEASE" +# Printed, always: the context is ambient unless --context says otherwise, and +# an operator should see which cluster is about to be written to before it is. +resolved_context="$CONTEXT" +[[ -n "$resolved_context" ]] || resolved_context="$(kubectl config current-context 2>/dev/null || true)" +[[ -n "$resolved_context" ]] || die "no kube context is set and --context was not given." +echo "==> stand: context=$resolved_context namespace=$NAMESPACE release=$RELEASE" # ── Discovery ─────────────────────────────────────────────────────────────── # One ConfigMap read per value keeps each failure attributable; kubectl's # jsonpath returns empty rather than failing on a missing key, so each result is # checked against the flag that would supply it. platform_cm="${RELEASE}-platform" -kubectl -n "$NAMESPACE" get configmap "$platform_cm" >/dev/null 2>&1 || die \ +kube -n "$NAMESPACE" get configmap "$platform_cm" >/dev/null 2>&1 || die \ "ConfigMap $platform_cm not found in namespace $NAMESPACE. It is generated by the umbrella chart and names every infrastructure coordinate this Job needs. Check --release (currently '$RELEASE')." cm_value() { - kubectl -n "$NAMESPACE" get configmap "$platform_cm" -o "jsonpath={.data.$1}" + kube -n "$NAMESPACE" get configmap "$platform_cm" -o "jsonpath={.data.$1}" } secret_value() { # $1 = secret, $2 = key. Missing secret or key yields an empty string, which # every caller treats as "not discovered". - kubectl -n "$NAMESPACE" get secret "$1" -o "jsonpath={.data.$2}" 2>/dev/null \ + kube -n "$NAMESPACE" get secret "$1" -o "jsonpath={.data.$2}" 2>/dev/null \ | { base64 --decode 2>/dev/null || true; } } @@ -195,7 +220,7 @@ esac ir_secret="insight-identity-resolution-config" # Told apart from a missing key: an unreadable Secret is an access problem, and # reporting it as "pass --tenant" would send the operator after the wrong thing. -if ! kubectl -n "$NAMESPACE" get secret "$ir_secret" -o name >/dev/null 2>&1; then +if ! kube -n "$NAMESPACE" get secret "$ir_secret" -o name >/dev/null 2>&1; then echo "WARNING: Secret $ir_secret is absent or not readable in namespace $NAMESPACE;" >&2 echo " the tenant and identity database cannot be discovered from it." >&2 fi @@ -252,7 +277,7 @@ if [[ -z "$IMAGE" || -z "$PULL_SECRETS" ]]; then # `|| true`: a helm failure (no such release, no permission) must reach the # missing-values report below naming --image, not kill the script through # pipefail with nothing said. - release_values="$(helm get values "$RELEASE" -n "$NAMESPACE" -a -o json 2>/dev/null || true)" + release_values="$(helm_release get values "$RELEASE" -n "$NAMESPACE" -a -o json 2>/dev/null || true)" if [[ -n "$release_values" ]]; then if [[ -z "$IMAGE" ]]; then IMAGE="$(printf '%s' "$release_values" | jq -r '.ingestion.toolboxImage // empty')" @@ -272,9 +297,9 @@ if [[ -z "$DB_SECRET" ]]; then # assuming the name: a Secret carrying only one of them would render a Job # that fails at pod creation on the missing key. for candidate in insight-db-creds "${RELEASE}-db-creds"; do - have_maria="$(kubectl -n "$NAMESPACE" get secret "$candidate" \ + have_maria="$(kube -n "$NAMESPACE" get secret "$candidate" \ -o 'jsonpath={.data.mariadb-password}' 2>/dev/null || true)" - have_ch="$(kubectl -n "$NAMESPACE" get secret "$candidate" \ + have_ch="$(kube -n "$NAMESPACE" get secret "$candidate" \ -o 'jsonpath={.data.clickhouse-password}' 2>/dev/null || true)" if [[ -n "$have_maria" && -n "$have_ch" ]]; then DB_SECRET="$candidate" @@ -381,11 +406,11 @@ if [[ "$DRY_RUN" -eq 1 ]]; then exit 0 fi -printf '%s\n' "$manifest" | kubectl apply -f - +printf '%s\n' "$manifest" | kube apply -f - echo "==> applied Job $job_name" if [[ "$FOLLOW" -eq 0 ]]; then - echo " follow it with: kubectl -n $NAMESPACE logs -f job/$job_name" + echo " follow it with: $(kubectl_hint) -n $NAMESPACE logs -f job/$job_name" exit 0 fi @@ -394,7 +419,7 @@ fi # then finish with nothing streamed. A pod that cannot start at all is caught by # the poll loop below, so this wait is bounded and never fatal. for _ in $(seq 1 60); do - phase="$(kubectl -n "$NAMESPACE" get pod -l "job-name=$job_name" \ + phase="$(kube -n "$NAMESPACE" get pod -l "job-name=$job_name" \ -o 'jsonpath={.items[0].status.phase}' 2>/dev/null || true)" case "$phase" in Running|Succeeded|Failed) break ;; @@ -402,7 +427,7 @@ for _ in $(seq 1 60); do sleep 2 done -kubectl -n "$NAMESPACE" logs -f "job/$job_name" || true +kube -n "$NAMESPACE" logs -f "job/$job_name" || true # The log stream ending is not the verdict — read it from the Job. Polled rather # than `kubectl wait --for=condition=complete`, which only knows how to wait for @@ -412,9 +437,9 @@ deadline=$((SECONDS + DEADLINE_SECONDS)) while [[ "$SECONDS" -lt "$deadline" ]]; do # `|| true` on every read: a transient apiserver hiccup inside a loop that may # run for an hour must not kill the script through `set -e`. - succeeded="$(kubectl -n "$NAMESPACE" get "job/$job_name" \ + succeeded="$(kube -n "$NAMESPACE" get "job/$job_name" \ -o 'jsonpath={.status.succeeded}' 2>/dev/null || true)" - failed="$(kubectl -n "$NAMESPACE" get "job/$job_name" \ + failed="$(kube -n "$NAMESPACE" get "job/$job_name" \ -o 'jsonpath={.status.failed}' 2>/dev/null || true)" if [[ "${succeeded:-0}" -ge 1 ]]; then echo "==> seed complete: $job_name" @@ -423,14 +448,14 @@ while [[ "$SECONDS" -lt "$deadline" ]]; do if [[ "${failed:-0}" -ge 1 ]]; then echo "ERROR: Job $job_name failed. Its logs above hold the reason; the Job is kept" >&2 echo " (backoffLimit 0, no retry) so it can be read again:" >&2 - echo " kubectl -n $NAMESPACE logs job/$job_name" >&2 + echo " $(kubectl_hint) -n $NAMESPACE logs job/$job_name" >&2 exit 1 fi # A pod that cannot start never becomes either, and waiting out the deadline # for it would be an hour of silence. The common cause is an image the cluster # cannot pull — including a locally built one on a remote cluster. - waiting="$(kubectl -n "$NAMESPACE" get pod -l "job-name=$job_name" \ + waiting="$(kube -n "$NAMESPACE" get pod -l "job-name=$job_name" \ -o 'jsonpath={.items[0].status.containerStatuses[0].state.waiting.reason}' 2>/dev/null || true)" case "$waiting" in ImagePullBackOff|ErrImagePull|InvalidImageName) @@ -443,5 +468,5 @@ while [[ "$SECONDS" -lt "$deadline" ]]; do done echo "ERROR: Job $job_name neither completed nor failed within ${DEADLINE_SECONDS}s:" >&2 -echo " kubectl -n $NAMESPACE describe job/$job_name" >&2 +echo " $(kubectl_hint) -n $NAMESPACE describe job/$job_name" >&2 exit 1 diff --git a/src/ingestion/tools/seed/tests/conftest.py b/src/ingestion/tools/seed/tests/conftest.py deleted file mode 100644 index f93b4ee6e..000000000 --- a/src/ingestion/tools/seed/tests/conftest.py +++ /dev/null @@ -1,39 +0,0 @@ -"""Shared test bootstrap: put the tool directory on `sys.path` and stub the -database drivers. - -Named `conftest.py` for pytest's benefit, but it is plain module-level code and -works the same under `unittest`, which is what this package actually uses (see -`tests/README` in the seeder README). Both entry points import it first: - - python3 -m unittest discover -s tests -t . # from the tool directory - -`pymysql` and `clickhouse_connect` are runtime-only dependencies of the seed -image. Every test here exercises the pure half — env parsing, SQL shapes, -refusal messages — so stand-ins keep the suite runnable with nothing installed. -""" - -from __future__ import annotations - -import sys -import types -from pathlib import Path - -#: The tool directory (parent of `tests/`), so `import insight_seed` resolves -#: whichever checkout this file lives in. -TOOL_ROOT = Path(__file__).resolve().parents[1] -if str(TOOL_ROOT) not in sys.path: - sys.path.insert(0, str(TOOL_ROOT)) - - -def _stub_pymysql() -> None: - if "pymysql" in sys.modules: - return - stub = types.ModuleType("pymysql") - stub.cursors = types.SimpleNamespace(Cursor=object) # type: ignore[attr-defined] - stub.connections = types.SimpleNamespace(Connection=object) # type: ignore[attr-defined] - stub.connect = lambda **_kwargs: None # type: ignore[attr-defined] - stub.MySQLError = Exception # type: ignore[attr-defined] - sys.modules["pymysql"] = stub - - -_stub_pymysql() diff --git a/src/ingestion/tools/seed/tests/test_identity.py b/src/ingestion/tools/seed/tests/test_identity.py index 2900233dd..4825b9208 100644 --- a/src/ingestion/tools/seed/tests/test_identity.py +++ b/src/ingestion/tools/seed/tests/test_identity.py @@ -8,14 +8,13 @@ dedupes a re-run (each insert gets a fresh `created_at`, so the unique key never collides). Every writer must check for an existing row explicitly. 2. Roster scope: fakeidp only defines a fixed dev-lead identity, but a - Keycloak realm seeds the WHOLE roster (gen-realm.py pins every realm + Keycloak realm seeds the WHOLE roster (`keycloak_realm` pins every realm user's id to their own roster uuid) — `seed_login_ids` must seed a row per roster member under `AUTH_MODE=keycloak`, not just the dev lead. -From the tool directory: +Run against the installed package (see the README's develop section): - python3 -m unittest discover -s tests -t . - python3 -m unittest tests.test_identity -v + uv run --extra dev python -m unittest discover -s tests -t . """ from __future__ import annotations @@ -24,8 +23,8 @@ import unittest from typing import Any -from . import conftest # noqa: F401 — sys.path + driver stubs, before the imports below - +# Set before the roster is imported: `profiles` reads it at call time, but a +# test that forgot would otherwise depend on the developer's shell. os.environ.setdefault("IDP_SOURCE_TYPE", "fakeidp") from insight_seed import identity, profiles @@ -126,7 +125,7 @@ def test_keycloak_seeds_the_whole_roster(self) -> None: self.assertEqual( first_run_count, len(roster), - "keycloak seeds every roster persona (gen-realm.py registers all of them)", + "keycloak seeds every roster persona (keycloak_realm registers all of them)", ) self.assertEqual(second_run_count, 0, "re-run must be a no-op for every pair") self.assertEqual(cur.insert_count, len(roster)) diff --git a/src/ingestion/tools/seed/tests/test_preflight.py b/src/ingestion/tools/seed/tests/test_preflight.py index 6fc5bd195..746696f44 100644 --- a/src/ingestion/tools/seed/tests/test_preflight.py +++ b/src/ingestion/tools/seed/tests/test_preflight.py @@ -1,26 +1,24 @@ """Env-contract and preflight-message tests. -Same harness as `test_identity.py`: stdlib `unittest`, no database, no -third-party import. Everything under test here is the pure half — env parsing, -the SQL a guard issues, and the messages a refusal carries — because that is -what has to stay true for an operator who reads only the error. +Stdlib `unittest` against the real package: env parsing, the SQL a guard +issues, and the messages a refusal carries — the half that has to stay true +for an operator who reads only the error. No database is touched. -From the tool directory: +Run against the installed package (see the README's develop section): - python3 -m unittest discover -s tests -t . - python3 -m unittest tests.test_preflight -v + uv run --extra dev python -m unittest discover -s tests -t . """ from __future__ import annotations import datetime as _dt +import pathlib import re import unittest import uuid as uuid_mod from insight_seed import config, identity, preflight - -from . import conftest +from insight_seed.generators import base _TENANT = "3f1d8f4e-6c2a-4a9b-91d7-8e5c0b2a7f36" @@ -156,7 +154,7 @@ def test_every_registered_target_is_actually_truncated_by_a_generator(self) -> N from insight_seed.generators.base import RESET_TARGETS called: set[tuple[str, str]] = set() - for path in sorted((conftest.TOOL_ROOT / "insight_seed" / "generators").glob("*.py")): + for path in sorted(pathlib.Path(base.__file__).parent.glob("*.py")): for schema, table in re.findall( r'truncate\(\s*client,\s*"([a-z0-9_]+)",\s*"([a-z0-9_]+)"', path.read_text() ): @@ -223,8 +221,6 @@ def test_the_refusal_says_the_step_truncates_and_offers_the_override(self) -> No class ResetRegistryTests(unittest.TestCase): def test_clearing_an_unregistered_relation_is_refused(self) -> None: - from insight_seed.generators import base - with self.assertRaises(ValueError) as caught: # The client is never reached: registration is checked first. base.truncate(object(), "silver", "class_not_registered") # type: ignore[arg-type] @@ -237,7 +233,6 @@ class WindowContractTests(unittest.TestCase): def test_the_generators_and_the_manifest_read_the_same_window(self) -> None: from insight_seed import manifest as manifest_mod - from insight_seed.generators import base env = {config.ANCHOR_ENV: "2026-06-30", config.DAYS_ENV: "14"} self.assertEqual(config.parse_anchor_date(env), _dt.date(2026, 6, 30)) @@ -275,6 +270,29 @@ def test_a_missing_dev_user_email_is_refused_and_says_what_it_anchors(self) -> N self.assertIn(config.DEV_USER_EMAIL_ENV, str(caught.exception)) +class ArtifactLocationTests(unittest.TestCase): + """Generated files go where the CALLER is, never where pip put the code — + an installed package's own directory is site-packages.""" + + def test_the_manifest_defaults_to_the_working_directory(self) -> None: + self.assertEqual(config.parse_manifest_path({}), pathlib.Path.cwd() / "manifest.json") + + def test_an_explicit_manifest_path_wins(self) -> None: + self.assertEqual( + config.parse_manifest_path({config.MANIFEST_PATH_ENV: "/tmp/somewhere.json"}), + pathlib.Path("/tmp/somewhere.json"), + ) + + def test_neither_artifact_resolves_inside_the_installed_package(self) -> None: + from insight_seed import manifest as manifest_mod + from insight_seed import profile_md + + package_dir = pathlib.Path(manifest_mod.__file__).resolve().parent + for path in (manifest_mod.manifest_path(), profile_md.profile_path()): + with self.subTest(path=path): + self.assertFalse(path.resolve().is_relative_to(package_dir)) + + class SeedReasonNamespaceTests(unittest.TestCase): def test_every_reason_the_identity_seed_writes_carries_the_shared_prefix(self) -> None: reasons = [ diff --git a/src/ingestion/tools/toolbox/Dockerfile b/src/ingestion/tools/toolbox/Dockerfile index b34caaa9e..a1964fc0b 100644 --- a/src/ingestion/tools/toolbox/Dockerfile +++ b/src/ingestion/tools/toolbox/Dockerfile @@ -22,22 +22,25 @@ RUN curl -fsSL "https://github.com/mikefarah/yq/releases/latest/download/yq_linu RUN curl -fsSL "https://dl.k8s.io/release/$(curl -fsSL https://dl.k8s.io/release/stable.txt)/bin/linux/$(dpkg --print-architecture)/kubectl" \ -o /usr/local/bin/kubectl && chmod +x /usr/local/bin/kubectl -# dbt-clickhouse, plus the two database drivers the sample-data seeder -# (tools/seed, COPYed in below) needs on top of it. Same layer on purpose: the -# seeder is part of what this image is for, not an afterthought bolted on at -# run time. -RUN pip install --no-cache-dir \ - dbt-clickhouse \ - 'PyMySQL==1.2.0' \ - 'clickhouse-connect==1.6.0' +# dbt-clickhouse. Installed before the source is copied so this slow layer +# survives every ingestion change; it is also what the sample-data seeder's +# silver step needs at run time (apply-ch-migrations.sh writes a profiles.yml +# and runs dbt), which is why that package declares it as an extra rather than +# a hard dependency. +RUN pip install --no-cache-dir dbt-clickhouse # Copy entire ingestion project. This is also how the seeder ships: it lives at -# tools/seed inside this tree, so `seed-stand.sh` can run it from -# /ingestion/tools/seed against the same scripts/ and dbt/ the migrations use — -# one image, one version, no drift between seeder and migration SQL. +# tools/seed inside this tree, so it runs against the same scripts/ and dbt/ the +# migrations use — one image, one version, no drift between seeder and migration +# SQL. COPY . /ingestion WORKDIR /ingestion +# Install the seeder as a package, so the Job runs `insight-seed` from PATH +# rather than a module out of a working directory, and its drivers come from its +# own metadata instead of a second list here that could drift from it. +RUN pip install --no-cache-dir /ingestion/tools/seed + # dbt writes logs/ and target/ under the project root, so the tree must belong # to the runtime user. RUN useradd -U -u 1000 -m appuser && chown -R 1000:1000 /ingestion diff --git a/tests/lib/insight_stand/manifest.py b/tests/lib/insight_stand/manifest.py index efd02f569..e24f9fc7e 100644 --- a/tests/lib/insight_stand/manifest.py +++ b/tests/lib/insight_stand/manifest.py @@ -94,7 +94,7 @@ def _optional_str(doc: Mapping[str, Any], key: str, where: str) -> str | None: @dataclass(frozen=True) class Realm: - """`realm` — mirrors deploy/compose/keycloak/gen-realm.py's realm.""" + """`realm` — mirrors the seeder's `keycloak_realm` output.""" name: str issuer: str diff --git a/tests/lib/insight_stand/personas.py b/tests/lib/insight_stand/personas.py index d07d6a836..1d454c17f 100644 --- a/tests/lib/insight_stand/personas.py +++ b/tests/lib/insight_stand/personas.py @@ -40,7 +40,7 @@ # takes precedence over the realm export. PASSWORD_ENV: Final[str] = "INSIGHT_STAND_PERSONA_PASSWORD" -# Mirrors `_ROLE_TO_REALM_ROLES` in deploy/compose/keycloak/gen-realm.py, which +# Mirrors `_ROLE_TO_REALM_ROLES` in insight_seed/keycloak_realm.py, which # is what actually builds the realm. Duplicated rather than imported because # that module belongs to the seed/compose tree; `verify_realm_roles` below is # what keeps the copy honest, by checking it against the realm the stand ran. From 5a895d9d391d02d35cef0e7d6126e7734198cba0 Mon Sep 17 00:00:00 2001 From: Konstantin Tursunov Date: Thu, 6 Aug 2026 12:22:09 +0800 Subject: [PATCH 03/10] ci(e2e-stand): install uv in the ui-journeys job Bringing the stand up now generates the Keycloak realm through the seed package's own entry point, which dev-compose.sh runs with uv. The API job already installs it for the suite; this job needs it for the bring-up itself, and would otherwise fail on a missing tool rather than on anything it tests. Signed-off-by: Konstantin Tursunov --- .cf-studio/config/README.md | 2 +- .cf-studio/version.toml | 2 +- .github/workflows/e2e-stand.yml | 8 ++++++++ 3 files changed, 10 insertions(+), 2 deletions(-) diff --git a/.cf-studio/config/README.md b/.cf-studio/config/README.md index 5020a6f28..a664dfd30 100644 --- a/.cf-studio/config/README.md +++ b/.cf-studio/config/README.md @@ -1,4 +1,4 @@ -# config — User Configuration +# config -- User Configuration This directory contains **user-editable** configuration files. diff --git a/.cf-studio/version.toml b/.cf-studio/version.toml index 5d3408aae..0a8b00224 100644 --- a/.cf-studio/version.toml +++ b/.cf-studio/version.toml @@ -1,7 +1,7 @@ # Constructor Studio pinned cfs version [cfs] version = "v1.6.2" -requested_ref = "latest" +requested_ref = "v1.6.2" source_type = "github" canonical_source = "https://api.github.com/repos/constructorfabric/studio" effective_source = "https://api.github.com/repos/constructorfabric/studio" diff --git a/.github/workflows/e2e-stand.yml b/.github/workflows/e2e-stand.yml index fb95571f6..1c6746870 100644 --- a/.github/workflows/e2e-stand.yml +++ b/.github/workflows/e2e-stand.yml @@ -301,6 +301,14 @@ jobs: with: python-version: "3.12" # runs the redaction script only — stdlib alone + # `test-stand up` forces Keycloak, and the realm is generated by the seed + # package's own entry point (`insight-seed-realm`), which dev-compose.sh + # runs through uv. The API job above installs uv for the suite; this job + # needs it for the bring-up itself. + - uses: astral-sh/setup-uv@c771a70e6277c0a99b617c7a806ffedaca235ff9 # v9.0.0 + with: + cache-suffix: stand-ui + - uses: docker/login-action@dbcb813823bdd20940b903addbd779551569679f # v4.6.0 with: registry: ghcr.io From d15a30f15b3fb64628dc489a7c6f43d9ba89035a Mon Sep 17 00:00:00 2001 From: Konstantin Tursunov Date: Thu, 6 Aug 2026 14:14:23 +0800 Subject: [PATCH 04/10] fix(seed): address the review findings MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit Two that mattered, both introduced by this branch: * The realm generator kept a tenant fallback while the seeder started requiring one. Generate a realm with the variable unset and the users carry a tenant claim the seed never writes rows for — they authenticate and then resolve to nobody, the exact failure a shared roster exists to prevent. It now reads the tenant through the same parser the seed does, so the two cannot disagree. * Preflight's foreign-silver scan caught every exception and read the result as a clean stand, which would then TRUNCATE the reset targets unexamined. A fresh stand needs no such tolerance — `system.columns` answers empty for databases that do not exist — so anything raised there is a scan that failed, and it is now a refusal. Then the small ones: `--deadline` is validated before it becomes both the pod ceiling and this script's polling budget; the tests set `IDP_SOURCE_TYPE` rather than defaulting it, so a value in the developer's shell cannot decide what they assert; and four stale module references and a code span left over from the package move. Not taken: an advisory lock around the identity step. Two concurrent seed runs against one tenant can both pass the existence check and append duplicate observations — true, but the read path resolves by newest observation, the duplicates are inert, and concurrent runs of a one-shot demo-data tool are not a case worth serialising a whole step for. Reconsider it if the seeder ever runs unattended. Signed-off-by: Konstantin Tursunov --- CONTRIBUTING.md | 2 +- docker-compose.yml | 4 ++-- src/ingestion/tools/seed/PROFILE.md | 2 +- src/ingestion/tools/seed/README.md | 3 ++- .../tools/seed/insight_seed/keycloak_realm.py | 16 ++++++++------ .../tools/seed/insight_seed/preflight.py | 14 ++++++++---- src/ingestion/tools/seed/seed-stand.sh | 5 +++++ src/ingestion/tools/seed/tests/__init__.py | 1 - .../tools/seed/tests/test_identity.py | 22 +++++++++++-------- 9 files changed, 43 insertions(+), 26 deletions(-) diff --git a/CONTRIBUTING.md b/CONTRIBUTING.md index bf23dc1eb..1461fd23f 100644 --- a/CONTRIBUTING.md +++ b/CONTRIBUTING.md @@ -476,7 +476,7 @@ tables, every `src/ingestion/scripts/migrations/*.sql` applied (produces the `insight.*` gold views), ~29k rows across 19 silver tables profile-typed per team (`class_git_*` for devs, `class_crm_*` for sales, …). The full per-team activity table is in -[`src/ingestion/tools/seed/profiles.py`](src/ingestion/tools/seed/profiles.py). analytics's +[`insight_seed/profiles.py`](src/ingestion/tools/seed/insight_seed/profiles.py). analytics's schema validator flips from "80 metrics error" to "80 ok". ### Compose diff --git a/docker-compose.yml b/docker-compose.yml index 4c9cb3346..82e5640ba 100644 --- a/docker-compose.yml +++ b/docker-compose.yml @@ -384,8 +384,8 @@ services: APP__gears__authenticator__config__idp__client_secret: "${OIDC_CLIENT_SECRET:-}" # Required: the identity-resolution source_type the login-bootstrap # lookup is scoped to. The realm sets each user's `sub` to their own - # roster uuid — the stable external id; src/ingestion/tools/seed/identity.py - # seeds the matching value_type='id' rows under this same source_type. + # roster uuid — the stable external id; the seeder's identity step seeds + # the matching value_type='id' rows under this same source_type. APP__gears__authenticator__config__idp__source_type: "${AUTHENTICATOR_IDP_SOURCE_TYPE:-keycloak}" # Callback rides the SPA's browser origin (Vite :3000) so the __Host-sid # cookie lands where the SPA runs; see .env.compose.example. diff --git a/src/ingestion/tools/seed/PROFILE.md b/src/ingestion/tools/seed/PROFILE.md index a55502bb0..9ffe46e41 100644 --- a/src/ingestion/tools/seed/PROFILE.md +++ b/src/ingestion/tools/seed/PROFILE.md @@ -17,7 +17,7 @@ builder that writes `manifest.json`, so the two cannot disagree. | realm | `insight` | | anchor_date | `2026-06-30` | | data_window | `2026-05-02..2026-06-30` | -| seed_revision | `c8aea877df6976da` | +| seed_revision | `abc7d57fccf41b41` | | manifest_version | 1 | `anchor_date` is the last day carrying seeded activity. It is resolved diff --git a/src/ingestion/tools/seed/README.md b/src/ingestion/tools/seed/README.md index 09a1758c8..4654000d5 100644 --- a/src/ingestion/tools/seed/README.md +++ b/src/ingestion/tools/seed/README.md @@ -85,7 +85,8 @@ Preflight runs before anything is written and reports every problem at once: names the database it looked in; - MariaDB or ClickHouse unreachable, or the ingestion scripts missing; - the target tenant already holds `persons` rows this seeder did not write - (every row it writes carries a `reason` starting `seed.py `); + (every row it writes carries a `reason` starting `"seed.py "`, trailing space + included); - any table the silver step clears holds rows for another tenant — that step **TRUNCATEs every table it writes**, across all tenants, so those rows would be destroyed. This is the one genuinely destructive thing the seeder does, and it diff --git a/src/ingestion/tools/seed/insight_seed/keycloak_realm.py b/src/ingestion/tools/seed/insight_seed/keycloak_realm.py index 1f92589e4..4a0ceb5bd 100644 --- a/src/ingestion/tools/seed/insight_seed/keycloak_realm.py +++ b/src/ingestion/tools/seed/insight_seed/keycloak_realm.py @@ -25,6 +25,7 @@ from pathlib import Path from typing import Any +from . import config from .profiles import ( TENANT_OTHER, Person, @@ -283,13 +284,14 @@ def main() -> None: # get_dev_user_email() (which fail-fasts if that is also unset). dev_user_email = args.dev_email if args.dev_email else get_dev_user_email() # The compose stack's tenant. The seeder REQUIRES `TENANT_DEFAULT_ID` and - # keeps no default of its own (rows under the wrong tenant are invisible to - # every login), so this is the compose convention's only remaining home: - # docker-compose.yml passes the same value to the seeder, and both converge - # on it when nobody sets one. - tenant_id = os.environ.get( # RULE-DEFAULTS-OK: the compose tenant, mirrored by docker-compose.yml's TENANT_DEFAULT_ID default so the realm's tenant_id claim matches what the seed writes - "TENANT_DEFAULT_ID", "00000000-df51-5b42-9538-d2b56b7ee953" - ) + # Required, exactly as the seeder requires it — and read through the same + # parser. A default here could mint realm users whose tenant claim matches + # nothing the seed then writes: they would authenticate and resolve to + # nobody, which is the failure this whole roster-sharing exists to avoid. + try: + tenant_id = config.parse_tenant_id(os.environ) + except config.EnvContractError as exc: + raise SystemExit(f"ERROR: cannot generate the realm.\n{exc}") from exc redirects = args.authenticator_redirects or DEFAULT_AUTHENTICATOR_REDIRECTS realm = build_realm(dev_user_email, tenant_id, redirects, args.authenticator_secret) diff --git a/src/ingestion/tools/seed/insight_seed/preflight.py b/src/ingestion/tools/seed/insight_seed/preflight.py index 9c5847827..35c0335f9 100644 --- a/src/ingestion/tools/seed/insight_seed/preflight.py +++ b/src/ingestion/tools/seed/insight_seed/preflight.py @@ -257,11 +257,17 @@ def _check_clickhouse(target: ClickHouse, scripts: Path, tenant: str, *, force: try: rows, worst, unattributable = _foreign_silver_rows(client, tenant) except Exception as exc: - # A stand with no silver database yet is the normal fresh case, - # and the placeholder script creates it. Anything else here is - # still worth reporting rather than swallowing. - LOG.info("foreign-silver-rows check skipped: %s", exc) + # A guard that cannot run is not a guard that passed. The fresh + # stand needs no tolerance here — `system.columns` answers with + # an empty result for databases that do not exist yet — so + # anything raised is a scan that failed, and the next step would + # TRUNCATE tables nobody has looked at. rows, worst, unattributable = 0, [], [] + problems.append( + "could not check whether the tables the silver step clears hold another " + f"tenant's rows ({exc}). Refusing rather than clearing them unexamined; " + f"{config.FORCE_ENV}=1 proceeds anyway." + ) if rows: problems.append(foreign_silver_problem(rows, worst, tenant)) elif unattributable: diff --git a/src/ingestion/tools/seed/seed-stand.sh b/src/ingestion/tools/seed/seed-stand.sh index 2fda07334..c9e1e55b8 100755 --- a/src/ingestion/tools/seed/seed-stand.sh +++ b/src/ingestion/tools/seed/seed-stand.sh @@ -155,6 +155,11 @@ case "$STEP" in identity|silver|analytics|all) ;; *) die "--step must be one of identity, silver, analytics, all (got '$STEP')." ;; esac +# Checked here rather than by the apiserver: this value is also the script's own +# polling budget, and a non-numeric one turns the wait loop's arithmetic into a +# silent zero — the run would be abandoned the moment it started. +[[ "$DEADLINE_SECONDS" =~ ^[0-9]+$ && "$DEADLINE_SECONDS" -gt 0 ]] \ + || die "--deadline must be a positive whole number of seconds (got '$DEADLINE_SECONDS')." # The release name is the one thing a namespace cannot answer, so the common # case (release named after its namespace) is assumed and reported, and the flag diff --git a/src/ingestion/tools/seed/tests/__init__.py b/src/ingestion/tools/seed/tests/__init__.py index 93b8588de..e69de29bb 100644 --- a/src/ingestion/tools/seed/tests/__init__.py +++ b/src/ingestion/tools/seed/tests/__init__.py @@ -1 +0,0 @@ -"""Unit tests for the seed package. Run: python3 -m unittest discover -s tests -t .""" diff --git a/src/ingestion/tools/seed/tests/test_identity.py b/src/ingestion/tools/seed/tests/test_identity.py index 4825b9208..1d3c3f865 100644 --- a/src/ingestion/tools/seed/tests/test_identity.py +++ b/src/ingestion/tools/seed/tests/test_identity.py @@ -23,10 +23,6 @@ import unittest from typing import Any -# Set before the roster is imported: `profiles` reads it at call time, but a -# test that forgot would otherwise depend on the developer's shell. -os.environ.setdefault("IDP_SOURCE_TYPE", "fakeidp") - from insight_seed import identity, profiles _TENANT = "00000000-df51-5b42-9538-d2b56b7ee953" @@ -91,14 +87,22 @@ def fetchone(self) -> tuple[int] | None: class SeedLoginIdsTests(unittest.TestCase): + """Both variables are SET, not defaulted: `profiles` reads them at call time, + so a value left over from the developer's shell would otherwise decide which + personas these tests expect.""" + + _ENV = ("AUTH_MODE", "IDP_SOURCE_TYPE") + def setUp(self) -> None: - self._prev_auth_mode = os.environ.get("AUTH_MODE") + self._previous = {name: os.environ.get(name) for name in self._ENV} + os.environ["IDP_SOURCE_TYPE"] = "fakeidp" def tearDown(self) -> None: - if self._prev_auth_mode is None: - os.environ.pop("AUTH_MODE", None) - else: - os.environ["AUTH_MODE"] = self._prev_auth_mode + for name, value in self._previous.items(): + if value is None: + os.environ.pop(name, None) + else: + os.environ[name] = value def test_second_run_does_not_insert_a_duplicate(self) -> None: os.environ["AUTH_MODE"] = "fakeidp" From 0a6ecc13648b6cd90fc7c80add8633d265af6684 Mon Sep 17 00:00:00 2001 From: Konstantin Tursunov Date: Thu, 6 Aug 2026 14:15:12 +0800 Subject: [PATCH 05/10] chore: keep the Studio tooling files out of this branch A local Studio version bump and a punctuation change in its config README were swept into the previous commit by the hook's stash/restore. Neither has anything to do with the seeder; restore them to main so the branch carries only its own subject. Signed-off-by: Konstantin Tursunov --- .cf-studio/config/README.md | 2 +- .cf-studio/version.toml | 2 +- 2 files changed, 2 insertions(+), 2 deletions(-) diff --git a/.cf-studio/config/README.md b/.cf-studio/config/README.md index a664dfd30..5020a6f28 100644 --- a/.cf-studio/config/README.md +++ b/.cf-studio/config/README.md @@ -1,4 +1,4 @@ -# config -- User Configuration +# config — User Configuration This directory contains **user-editable** configuration files. diff --git a/.cf-studio/version.toml b/.cf-studio/version.toml index 0a8b00224..5d3408aae 100644 --- a/.cf-studio/version.toml +++ b/.cf-studio/version.toml @@ -1,7 +1,7 @@ # Constructor Studio pinned cfs version [cfs] version = "v1.6.2" -requested_ref = "v1.6.2" +requested_ref = "latest" source_type = "github" canonical_source = "https://api.github.com/repos/constructorfabric/studio" effective_source = "https://api.github.com/repos/constructorfabric/studio" From b85cd345584e30c69a994cac6a93e0a163bb1fc8 Mon Sep 17 00:00:00 2001 From: Konstantin Tursunov Date: Thu, 6 Aug 2026 15:22:54 +0800 Subject: [PATCH 06/10] fix(seed): close the gaps a second review pass found MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit The gitops `keycloak-realm` target still invoked the generator by its old path, so `make deploy ENV=local` would have failed at realm generation. It now runs the installed console script through `uv run --project`, requires `uv` explicitly, and reads `global.tenantDefaultId` out of the env's values — a missing tenant is an error naming the file, not a realm whose users authenticate and resolve to nobody. Two more holes in the refusals: - The silver step consults the persons signal too. Scanning only for CROSS-tenant silver rows can never fire on a single-tenant stand, which is the stand most likely to be pointed at by mistake, and `--step silver` skipped the persons check entirely while truncating 21 relations. - The anchor date is resolved from the clock once per process. Two callers each reading `now()` disagree across a UTC midnight, and the manifest would then describe a window the rows do not sit in. Tests cover both, plus the reset surface the operator is shown before a truncating step. Signed-off-by: Konstantin Tursunov --- deploy/gitops/Makefile | 20 +++-- .../environments/local/values.yaml.template | 6 +- .../tools/seed/insight_seed/config.py | 13 +++- .../tools/seed/insight_seed/keycloak_realm.py | 1 - .../tools/seed/insight_seed/preflight.py | 72 +++++++++++++++--- .../tools/seed/tests/test_identity.py | 76 +++++++++++++++++++ .../tools/seed/tests/test_preflight.py | 30 ++++++++ 7 files changed, 195 insertions(+), 23 deletions(-) diff --git a/deploy/gitops/Makefile b/deploy/gitops/Makefile index ded00832b..82b44fed5 100644 --- a/deploy/gitops/Makefile +++ b/deploy/gitops/Makefile @@ -337,15 +337,21 @@ local-wizard: bash "$$WIZARD" --target=k8s-local .PHONY: keycloak-realm -# ENV=local sandbox inputs for the in-stack Keycloak: generate the roster -# realm (gen-realm.py) into the env realms dir — keycloak-broker-realms packs -# it with the rest — and ensure the sandbox admin/placeholder Secret. -# No-op unless keycloak.deploy=true. +# ENV=local sandbox inputs for the in-stack Keycloak: generate the roster realm +# into the env realms dir — keycloak-broker-realms packs it with the rest — and +# ensure the sandbox admin/placeholder Secret. No-op unless keycloak.deploy=true. +# +# The generator lives in the seed package (it builds the realm from the same +# roster the database seed writes) and runs as its installed entry point, so +# this recipe needs uv. It also needs the tenant: the generator requires it +# rather than defaulting, because a realm whose tenant claim differs from the +# seeded rows authenticates users that then resolve to nobody. keycloak-realm: kube-ctx values-present @set -e; \ if [ "$$(yq -r '.keycloak.deploy // false' $(VALUES))" != "true" ]; then \ echo "$(C_YEL)skip$(C_RST) keycloak-realm (keycloak.deploy != true)"; exit 0; fi; \ - GEN="$$(cd ../.. && pwd)/deploy/compose/keycloak/gen-realm.py"; \ + SEED_DIR="$$(cd ../.. && pwd)/src/ingestion/tools/seed"; \ + command -v uv >/dev/null 2>&1 || { echo "$(C_RED)uv is required to generate the Keycloak realm$(C_RST)"; exit 1; }; \ DIR=environments/$(ENV)/keycloak/realms; mkdir -p "$$DIR"; \ OUT="$$DIR/realm-insight.generated.json"; \ DEV_EMAIL="$$(yq -r '.fakeidp.devUserEmail // "dev@company.nonpresent"' $(VALUES))"; \ @@ -354,7 +360,9 @@ keycloak-realm: kube-ctx values-present KC_BASE="$$(yq -r '.keycloak.hostname // ""' $(VALUES))"; \ LB_REDIRECT="$${KC_BASE%/kc}/auth/callback"; \ echo "$(C_GRN)→$(C_RST) generating Keycloak realm (dev-lead $$DEV_EMAIL; redirects $$REDIRECT, $$LB_REDIRECT)"; \ - python3 "$$GEN" --dev-email "$$DEV_EMAIL" \ + TENANT_ID="$$(yq -r '.global.tenantDefaultId // ""' $(VALUES))"; \ + [ -n "$$TENANT_ID" ] || { echo "$(C_RED)global.tenantDefaultId is required to generate the realm$(C_RST)"; exit 1; }; \ + TENANT_DEFAULT_ID="$$TENANT_ID" uv run --project "$$SEED_DIR" insight-seed-realm --dev-email "$$DEV_EMAIL" \ --authenticator-redirect "$$REDIRECT" \ --authenticator-redirect "$$LB_REDIRECT" \ --authenticator-secret "$$AUTH_SECRET" \ diff --git a/deploy/gitops/environments/local/values.yaml.template b/deploy/gitops/environments/local/values.yaml.template index f430033dd..6d5119208 100644 --- a/deploy/gitops/environments/local/values.yaml.template +++ b/deploy/gitops/environments/local/values.yaml.template @@ -116,7 +116,7 @@ authenticator: # matches for discovery AND the browser redirect. `redirectUri` stays on # `http://localhost` because the `__Host-sid` session cookie requires a # secure-context origin, and `http://localhost` qualifies (the LB IP does not). - # clientSecret matches the dev secret gen-realm.py bakes into the realm. + # clientSecret matches the dev secret the realm generator bakes in. oidc: issuerUrl: 'http://__INGRESS_LB_IP__/kc/realms/insight' clientId: "insight-authenticator" @@ -128,8 +128,8 @@ authenticator: # exists on identity-resolution (Rust), never the .NET identity twin. sourceType: "keycloak" # id_token claim carrying the stable external user id for sourceType. - # gen-realm.py pins each realm user's `id` to their OWN roster uuid (see - # deploy/compose/keycloak/gen-realm.py), so Keycloak's `sub` (the default) + # The realm generator pins each realm user's `id` to their OWN roster uuid + # (src/ingestion/tools/seed/insight_seed/keycloak_realm.py), so `sub` # already IS that stable id — no Entra-style non-default claim needed here. externalIdClaim: "sub" # Optional tenant sourcing: `tenantClaim` names the id_token claim carrying diff --git a/src/ingestion/tools/seed/insight_seed/config.py b/src/ingestion/tools/seed/insight_seed/config.py index 883002087..f4d58336b 100644 --- a/src/ingestion/tools/seed/insight_seed/config.py +++ b/src/ingestion/tools/seed/insight_seed/config.py @@ -39,6 +39,9 @@ #: Length of the seeded activity window when nobody pins one. DEFAULT_SEED_DAYS = 60 +#: The clock is read once; see `parse_anchor_date`. +_DERIVED_ANCHOR: _dt.date | None = None + _TRUE = frozenset({"1", "true", "yes", "on"}) _FALSE = frozenset({"0", "false", "no", "off"}) @@ -193,7 +196,15 @@ def parse_anchor_date(env: Mapping[str, str]) -> _dt.date: raise EnvContractError( (f"{ANCHOR_ENV}={raw!r} is not an ISO date (YYYY-MM-DD) or `today`: {exc}.",) ) from exc - return _dt.datetime.now(_dt.UTC).date() - _dt.timedelta(days=1) + + # Resolved from the clock ONCE per process. Sharing the reader was not + # enough: two callers each computing `now()` still disagree if the run + # straddles a UTC midnight, and the manifest would then report a window the + # rows do not sit in. + global _DERIVED_ANCHOR + if _DERIVED_ANCHOR is None: + _DERIVED_ANCHOR = _dt.datetime.now(_dt.UTC).date() - _dt.timedelta(days=1) + return _DERIVED_ANCHOR def parse_seed_days(env: Mapping[str, str], default: int = DEFAULT_SEED_DAYS) -> int: diff --git a/src/ingestion/tools/seed/insight_seed/keycloak_realm.py b/src/ingestion/tools/seed/insight_seed/keycloak_realm.py index 4a0ceb5bd..9bea4fc64 100644 --- a/src/ingestion/tools/seed/insight_seed/keycloak_realm.py +++ b/src/ingestion/tools/seed/insight_seed/keycloak_realm.py @@ -283,7 +283,6 @@ def main() -> None: # Explicit --dev-email wins; otherwise fall back to DEV_USER_EMAIL via # get_dev_user_email() (which fail-fasts if that is also unset). dev_user_email = args.dev_email if args.dev_email else get_dev_user_email() - # The compose stack's tenant. The seeder REQUIRES `TENANT_DEFAULT_ID` and # Required, exactly as the seeder requires it — and read through the same # parser. A default here could mint realm users whose tenant claim matches # nothing the seed then writes: they would authenticate and resolve to diff --git a/src/ingestion/tools/seed/insight_seed/preflight.py b/src/ingestion/tools/seed/insight_seed/preflight.py index 35c0335f9..0a7c4cdf5 100644 --- a/src/ingestion/tools/seed/insight_seed/preflight.py +++ b/src/ingestion/tools/seed/insight_seed/preflight.py @@ -9,16 +9,22 @@ demo data, it now ships inside the same image a cluster uses for migrations, and a CLI can point it at any stand. -* Identity rows are additive, and every one the seeder writes carries a `reason` - starting with `seed.py ` — so a tenant holding person rows with any other - reason is a tenant somebody else's data lives in. -* Silver rows are NOT additive: the generators TRUNCATE each table before - writing, across every tenant, because a partially rewritten silver table - produces metrics that are wrong rather than absent. Rows there for any other - tenant would be destroyed, so their presence is a refusal too. - -Both are overridable with `SEED_FORCE=1`, which is the only way to say "yes, -clear it" out loud. +Two signals, because neither sees everything: + +* `persons` rows in the target tenant whose `reason` is outside this seeder's + namespace. Identity rows are additive, so this one is about not mixing demo + people into somebody's directory — but it is also the ONLY signal that works + on a single-tenant stand, so the silver step consults it too. +* Rows in the reset surface belonging to a different tenant. Silver rows are + NOT additive: the generators TRUNCATE each table before writing, across every + tenant, because a partially rewritten silver table produces metrics that are + wrong rather than absent. This one is differential, so it says nothing on a + stand that has only ever had one tenant — hence the first. + +Neither can attribute rows in the targets that carry no tenant column at all; +those are named in the log rather than silently counted as clean. Both refusals +are overridable with `SEED_FORCE=1`, which is the only way to say "yes, clear +it" out loud. """ from __future__ import annotations @@ -229,6 +235,29 @@ def _foreign_silver_rows( return sum(count for _, count in rows), rows[:limit], unattributable +def _reset_surface_rows(client: object) -> int: + """How many rows the silver step would clear, across every reset target. + + Tenant-agnostic on purpose: the refusal above answers "whose rows are + these", this answers "how much is there", and the second question still has + an answer when the first one cannot be told apart on a single-tenant stand. + """ + from .generators.base import RESET_TARGETS + + parts = [ + f"SELECT count() AS n FROM `{schema}`.`{table}`" + for schema, table in RESET_TARGETS + if _IDENTIFIER.match(schema) and _IDENTIFIER.match(table) + ] + try: + result = client.query(f"SELECT sum(n) FROM ({' UNION ALL '.join(parts)})") # type: ignore[attr-defined] + except Exception as exc: + LOG.info("could not size the reset surface: %s", exc) + return 0 + rows = result.result_rows + return int(rows[0][0]) if rows and rows[0][0] is not None else 0 + + def _check_clickhouse(target: ClickHouse, scripts: Path, tenant: str, *, force: bool) -> list[str]: problems: list[str] = [] @@ -270,7 +299,19 @@ def _check_clickhouse(target: ClickHouse, scripts: Path, tenant: str, *, force: ) if rows: problems.append(foreign_silver_problem(rows, worst, tenant)) - elif unattributable: + if not problems: + # Visible even on a clean pass: this is the one step that + # destroys, and an operator should be told what it is about to + # clear rather than reading it out of the generators. + total = _reset_surface_rows(client) + if total: + LOG.warning( + "the silver step clears %d row(s) across the tables it writes; " + "all of them belong to tenant %s or carry no tenant", + total, + tenant, + ) + if unattributable: # Said out loud rather than silently ignored: these targets are # cleared too and carry no tenant, so nobody can tell whose rows # they hold. In practice a stand holding foreign rows here also @@ -345,7 +386,14 @@ def check(env: dict[str, str] | None = None, steps: Iterable[str] = STEPS) -> No # a plain `str`, and nothing has to reason about how it got that way. tenant = config.parse_tenant_id(environ) - if "identity" in requested: + # The persons check runs for the SILVER step too, and it is the more + # important of the two guards there. The silver scan is differential — it + # compares tenants — so on a single-tenant stand, which is the ordinary + # case, it can only ever return zero however much real data the tables + # hold. Foreign rows in `persons` are what says "this stand belongs to + # somebody" when there is no second tenant to compare against, and the + # silver step is the one that TRUNCATEs. + if "identity" in requested or "silver" in requested: problems += _check_identity( config.parse_mariadb(environ, database=config.parse_identity_database(environ)), tenant, diff --git a/src/ingestion/tools/seed/tests/test_identity.py b/src/ingestion/tools/seed/tests/test_identity.py index 1d3c3f865..3126e7819 100644 --- a/src/ingestion/tools/seed/tests/test_identity.py +++ b/src/ingestion/tools/seed/tests/test_identity.py @@ -137,3 +137,79 @@ def test_keycloak_seeds_the_whole_roster(self) -> None: if __name__ == "__main__": unittest.main() + + +class _ObservationCursor: + """Answers the exists-then-insert pair the observation writers issue. + + Keyed on (person, value_type, value) — the logical identity of an + observation, which is exactly what `created_at` stopped enforcing. The two + writers order their parameters differently, so the INSERT key is read from + the statement's own shape. + """ + + def __init__(self) -> None: + self.insert_count = 0 + self.rowcount = 0 + self._seen: set[tuple[Any, Any, Any]] = set() + self._pending: tuple[int] | None = None + + def execute(self, sql: str, params: tuple[Any, ...] = ()) -> None: + head = sql.strip().upper() + if head.startswith("SELECT"): + # _observation_exists: (tenant, person, source_type, source_id, value_type, value) + self._pending = (1,) if (params[1], params[4], params[5]) in self._seen else None + elif head.startswith("INSERT"): + if "value_full_text" in sql: + # (value_type, source_type, source_id, tenant, value, person, author, reason) + key = (params[5], params[0], params[4]) + else: + # value_type is the literal 'email' in the statement: + # (source_type, source_id, tenant, value_id, person, author, reason) + key = (params[4], "email", params[3]) + self._seen.add(key) + self.insert_count += 1 + self.rowcount = 1 + else: + raise AssertionError(f"unexpected SQL: {sql}") + + def fetchone(self) -> tuple[int] | None: + return self._pending + + +class SeedPersonsIdempotencyTests(unittest.TestCase): + """The writers that used to rely on `INSERT IGNORE`. Migration 004 put + `created_at` in the unique key, so a re-run never collided and every run + appended a duplicate observation; each writer now checks first.""" + + def setUp(self) -> None: + self._previous = os.environ.get("IDP_SOURCE_TYPE") + os.environ["IDP_SOURCE_TYPE"] = "fakeidp" + + def tearDown(self) -> None: + if self._previous is None: + os.environ.pop("IDP_SOURCE_TYPE", None) + else: + os.environ["IDP_SOURCE_TYPE"] = self._previous + + def test_seed_persons_inserts_once_across_two_runs(self) -> None: + cur = _ObservationCursor() + roster = _roster() + + first = identity.seed_persons(cur, _TENANT, roster) # type: ignore[arg-type] + second = identity.seed_persons(cur, _TENANT, roster) # type: ignore[arg-type] + + self.assertEqual(first, len(roster), "the first run writes one row per person") + self.assertEqual(second, 0, "a re-run must be a no-op, not a duplicate observation") + self.assertEqual(cur.insert_count, len(roster)) + + def test_seed_person_names_inserts_once_across_two_runs(self) -> None: + cur = _ObservationCursor() + roster = _roster() + + first = identity.seed_person_names(cur, _TENANT, roster) # type: ignore[arg-type] + second = identity.seed_person_names(cur, _TENANT, roster) # type: ignore[arg-type] + + self.assertGreater(first, 0, "names are written on the first run") + self.assertEqual(second, 0, "a re-run must be a no-op, not a duplicate observation") + self.assertEqual(cur.insert_count, first) diff --git a/src/ingestion/tools/seed/tests/test_preflight.py b/src/ingestion/tools/seed/tests/test_preflight.py index 746696f44..335297b2d 100644 --- a/src/ingestion/tools/seed/tests/test_preflight.py +++ b/src/ingestion/tools/seed/tests/test_preflight.py @@ -120,15 +120,19 @@ def __init__( self, columns: list[tuple[str, str, str]], counts: list[tuple[str, int]], + total_rows: int = 0, ) -> None: self._columns = columns self._counts = counts + self._total_rows = total_rows self.queries: list[str] = [] def query(self, sql: str, parameters: dict[str, object] | None = None) -> _FakeResult: self.queries.append(sql) if "system.columns" in sql: return _FakeResult(list(self._columns)) + if sql.startswith("SELECT sum(n)"): + return _FakeResult([(self._total_rows,)]) return _FakeResult([(name, count) for name, count in self._counts]) @@ -293,6 +297,32 @@ def test_neither_artifact_resolves_inside_the_installed_package(self) -> None: self.assertFalse(path.resolve().is_relative_to(package_dir)) +class GuardCoverageTests(unittest.TestCase): + """Which guard runs for which step. The silver scan is differential, so on a + single-tenant stand it returns zero however much real data the tables hold — + the `persons` signal is what covers that case, and the destructive step has + to consult it.""" + + def test_the_silver_step_also_consults_the_persons_signal(self) -> None: + source = pathlib.Path(preflight.__file__).read_text() + gate = '"identity" in requested or "silver" in requested' + self.assertIn(gate, source) + + def test_the_silver_scan_is_differential_and_says_so(self) -> None: + """Locks the reason the gate above exists: a same-tenant row is not + counted, so this scan alone cannot see a single-tenant stand's data.""" + client = _FakeClickHouse( + columns=[("silver", "class_people", "tenant_id")], + counts=[], + ) + preflight._foreign_silver_rows(client, _TENANT) + self.assertIn("!= {tenant:String}", client.queries[-1]) + + def test_the_reset_surface_is_sized_for_the_operator(self) -> None: + client = _FakeClickHouse(columns=[], counts=[], total_rows=4321) + self.assertEqual(preflight._reset_surface_rows(client), 4321) + + class SeedReasonNamespaceTests(unittest.TestCase): def test_every_reason_the_identity_seed_writes_carries_the_shared_prefix(self) -> None: reasons = [ From b91131688ae3b2a6f8a4bd60ecefa929c0e9c455 Mon Sep 17 00:00:00 2001 From: Konstantin Tursunov Date: Thu, 6 Aug 2026 15:23:17 +0800 Subject: [PATCH 07/10] build(seed): publish the seeder as its own image MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit The seeder was going to ride along in the toolbox. That image runs migrations against real stands, and a demo-data generator has no business being installed there — one `docker run` away from writing a fake organisation into a stand that holds a real one. So it gets an image of its own. Same build context (`src/ingestion`, so the DDL and gold-build scripts the silver step shells out to travel with it, at one version), a narrower COPY set, and `insight-seed` as the entry point. The toolbox stops carrying it: `tools/seed/` is excluded there through a per-Dockerfile `Dockerfile.dockerignore`, which applies to that image only, so both can build from the same context and get different content. Published beside the toolbox — per-arch matrix, digest merge, provenance attestation — and pinned into the chart as `ingestion.seedImage`, which is what `seed-stand.sh` now discovers. The chart never renders it; nothing in a deployed release runs the seeder. It is published state, so an operator can ask a stand which seeder matches it. The amd64 leg validates before pushing: the console script resolves, every step module imports (the whole driver set, which is exactly what was missing when the seeder could not run on a stand at all), the package's tests pass, and dbt is present for the silver step's subprocess. The tests run inside the shipped image rather than in ci.yml's Python matrix — they are stdlib `unittest` and touch no database, so what gets tested is what gets published. Compose builds the same Dockerfile locally, and the nightly Trivy matrix scans the published image. Signed-off-by: Konstantin Tursunov --- .github/workflows/build-images.yml | 183 +++++++++++++++++- .github/workflows/trivy-images.yml | 1 + CONTRIBUTING.md | 2 +- README.md | 1 + charts/insight/values.yaml | 7 + deploy/HELM_DEPLOY.md | 2 +- docker-compose.yml | 26 ++- src/ingestion/README.md | 4 + src/ingestion/tools/seed/Dockerfile | 83 ++++---- src/ingestion/tools/seed/README.md | 56 ++++-- src/ingestion/tools/seed/seed-job.yaml.tpl | 10 +- src/ingestion/tools/seed/seed-stand.sh | 42 ++-- src/ingestion/tools/toolbox/Dockerfile | 20 +- .../tools/toolbox/Dockerfile.dockerignore | 27 +++ 14 files changed, 363 insertions(+), 101 deletions(-) create mode 100644 src/ingestion/tools/toolbox/Dockerfile.dockerignore diff --git a/.github/workflows/build-images.yml b/.github/workflows/build-images.yml index c77f8a6b6..09711065b 100644 --- a/.github/workflows/build-images.yml +++ b/.github/workflows/build-images.yml @@ -83,6 +83,7 @@ jobs: identity_resolution: ${{ steps.filter.outputs.identity_resolution }} frontend: ${{ steps.filter.outputs.frontend }} toolbox: ${{ steps.filter.outputs.toolbox }} + seed: ${{ steps.filter.outputs.seed }} ui_tests: ${{ steps.filter.outputs.ui_tests }} umbrella: ${{ steps.filter.outputs.umbrella }} build_tag: ${{ steps.tag.outputs.build_tag }} @@ -139,6 +140,14 @@ jobs: - 'src/frontend/**' toolbox: - 'src/ingestion/**' + # The sample-data seeder's own image. It COPYs a subset of the + # ingestion tree (scripts, dbt, silver, gold, connectors, the tool + # itself), but the filter stays as wide as the toolbox's on + # purpose: a narrower list would have to be kept in step with the + # Dockerfile's COPY lines, and the failure mode of forgetting is a + # silently stale published image. + seed: + - 'src/ingestion/**' # The browser runner for tests/stand (deploy/compose/ui-tests.Dockerfile). # It COPYs tests/lib + tests/stand and installs from the committed # tests/uv.lock, so all four are baked in and any of them changing @@ -1101,6 +1110,162 @@ jobs: run: | echo '- `${{ env.IMAGE_PREFIX }}/insight-toolbox:${{ needs.changes.outputs.build_tag }}`' >> "$GITHUB_STEP_SUMMARY" + # ─── Sample-data seeder build (with import smoke) ────────────────────────── + # The seeder's own image, separate from the toolbox on purpose: the toolbox + # runs migrations against real stands and a demo-data generator has no + # business being installed there. Same build context (src/ingestion), a + # narrower COPY set, and `insight-seed` as the entry point — see + # src/ingestion/tools/seed/Dockerfile. + # + # `seed-stand.sh` runs this image as a one-shot Job; the ref it discovers is + # `ingestion.seedImage`, pinned by publish-chart below. + seed: + needs: changes + if: | + needs.changes.outputs.seed == 'true' + || (github.event_name == 'workflow_dispatch' && !inputs.frontend_only) + strategy: + fail-fast: false + matrix: + include: + - arch: amd64 + runner: ubuntu-latest + platform: linux/amd64 + - arch: arm64 + runner: ubuntu-24.04-arm + platform: linux/arm64 + runs-on: ${{ matrix.runner }} + steps: + - uses: actions/checkout@11d5960a326750d5838078e36cf38b85af677262 # v4 + + - uses: docker/setup-buildx-action@8d2750c68a42422c14e847fe6c8ac0403b4cbd6f # v3 + + - uses: docker/login-action@c94ce9fb468520275223c153574b00df6fe4bcc9 # v3 + if: needs.changes.outputs.should_push == 'true' + with: + registry: ${{ env.REGISTRY }} + username: ${{ github.actor }} + password: ${{ secrets.GITHUB_TOKEN }} + + # Build → validate (amd64 leg only) → push by digest, exactly as the + # toolbox above: the content is arch-agnostic, so one validate is enough + # and the push-build reuses the buildx cache. + - name: Build seed image (load locally, amd64 only) + if: matrix.arch == 'amd64' + uses: docker/build-push-action@263435318d21b8e681c14492fe198d362a7d2c83 # v6.18.0 + with: + context: src/ingestion + file: src/ingestion/tools/seed/Dockerfile + platforms: linux/amd64 + load: true + push: false + tags: insight-seed:ci-validate + cache-from: type=gha,scope=seed-amd64 + cache-to: ${{ needs.changes.outputs.should_push == 'true' && 'type=gha,mode=max,scope=seed-amd64,ignore-error=true' || '' }} + + - name: Validate the seeder (entry point, imports, tests, dbt) + if: matrix.arch == 'amd64' + # The failure this catches is the one that made the seeder unrunnable + # on a stand in the first place: a module whose dependency is not + # installed in the image. Importing every step module exercises the + # whole dependency set (MariaDB and ClickHouse drivers included) + # without touching a database, `--help` proves the console script is on + # PATH, and `dbt --version` proves the silver step's subprocess + # environment exists. + # + # The package's own tests run here rather than in ci.yml's Python + # matrix: they are stdlib `unittest` and touch no database, so the + # shipped image can run them with nothing installed on top — and this + # way what is tested is exactly what is published. + run: | + docker run --rm insight-seed:ci-validate --help + docker run --rm --entrypoint python insight-seed:ci-validate -c \ + 'import insight_seed.identity, insight_seed.silver, insight_seed.analytics, insight_seed.preflight, insight_seed.keycloak_realm' + docker run --rm --entrypoint python insight-seed:ci-validate -m unittest discover -s tests -t . + docker run --rm --entrypoint dbt insight-seed:ci-validate --version + + - name: Build and push seed image (by digest) + id: build + uses: docker/build-push-action@263435318d21b8e681c14492fe198d362a7d2c83 # v6.18.0 + with: + context: src/ingestion + file: src/ingestion/tools/seed/Dockerfile + platforms: ${{ matrix.platform }} + outputs: type=image,name=${{ env.IMAGE_PREFIX }}/insight-seed,push-by-digest=true,name-canonical=true,push=${{ needs.changes.outputs.should_push == 'true' }} + cache-from: type=gha,scope=seed-${{ matrix.arch }} + cache-to: ${{ needs.changes.outputs.should_push == 'true' && format('type=gha,mode=max,scope=seed-{0},ignore-error=true', matrix.arch) || '' }} + + - name: Export digest + if: needs.changes.outputs.should_push == 'true' + run: | + mkdir -p /tmp/digests + digest="${{ steps.build.outputs.digest }}" + touch "/tmp/digests/${digest#sha256:}" + - name: Upload digest + if: needs.changes.outputs.should_push == 'true' + uses: actions/upload-artifact@ea165f8d65b6e75b540449e92b4886f43607fa02 # v4 + with: + name: digests-seed-${{ matrix.arch }} + path: /tmp/digests/* + if-no-files-found: error + retention-days: 1 + + merge-seed: + needs: [changes, seed] + if: | + always() + && needs.seed.result != 'skipped' + && needs.changes.outputs.should_push == 'true' + runs-on: ubuntu-latest + steps: + - name: Require the platform builds to have passed + if: needs.seed.result != 'success' + run: | + echo "::error::seed finished '${{ needs.seed.result }}' — nothing to merge." + exit 1 + - name: Download digests + uses: actions/download-artifact@d3f86a106a0bac45b974a628896c90dbdf5c8093 # v4 + with: + path: /tmp/digests + pattern: digests-seed-* + merge-multiple: true + - uses: docker/setup-buildx-action@8d2750c68a42422c14e847fe6c8ac0403b4cbd6f # v3 + - uses: docker/login-action@c94ce9fb468520275223c153574b00df6fe4bcc9 # v3 + with: + registry: ${{ env.REGISTRY }} + username: ${{ github.actor }} + password: ${{ secrets.GITHUB_TOKEN }} + - uses: docker/metadata-action@c299e40c65443455700f0fdfc63efafe5b349051 # v5 + id: meta + with: + images: ${{ env.IMAGE_PREFIX }}/insight-seed + tags: | + type=raw,value=${{ needs.changes.outputs.build_tag }} + type=raw,value=latest,enable=${{ github.ref == 'refs/heads/main' }} + - name: Create multi-arch manifest and push + working-directory: /tmp/digests + run: | + docker buildx imagetools create \ + $(jq -cr '.tags | map("-t " + .) | join(" ")' <<< "$DOCKER_METADATA_OUTPUT_JSON") \ + $(printf '${{ env.IMAGE_PREFIX }}/insight-seed@sha256:%s ' *) + - name: Inspect manifest digest + id: inspect + run: | + DIGEST=$(docker buildx imagetools inspect \ + ${{ env.IMAGE_PREFIX }}/insight-seed:${{ needs.changes.outputs.build_tag }} \ + --format '{{json .Manifest}}' | jq -r .digest) + echo "digest=$DIGEST" >> "$GITHUB_OUTPUT" + - name: Attest build provenance + uses: actions/attest-build-provenance@e8998f949152b193b063cb0ec769d69d929409be # v2 + with: + subject-name: ${{ env.IMAGE_PREFIX }}/insight-seed + subject-digest: ${{ steps.inspect.outputs.digest }} + push-to-registry: true + # Copy-pastable pushed ref on the run's Summary page (#1994). + - name: Report pushed image + run: | + echo '- `${{ env.IMAGE_PREFIX }}/insight-seed:${{ needs.changes.outputs.build_tag }}`' >> "$GITHUB_STEP_SUMMARY" + # ─── ui-tests browser runner (deployed-stand journeys) ───────────────────── # The Playwright image e2e-stand.yml's `ui-journeys` check pulls. It bakes in # tests/lib + tests/stand and installs from the committed tests/uv.lock, so @@ -1588,6 +1753,7 @@ jobs: - merge-identity-resolution - merge-frontend - merge-toolbox + - merge-seed - merge-image - bump-descriptors if: | @@ -1601,6 +1767,7 @@ jobs: && (needs.merge-identity-resolution.result == 'success' || needs.merge-identity-resolution.result == 'skipped') && (needs.merge-frontend.result == 'success' || needs.merge-frontend.result == 'skipped') && (needs.merge-toolbox.result == 'success' || needs.merge-toolbox.result == 'skipped') + && (needs.merge-seed.result == 'success' || needs.merge-seed.result == 'skipped') && (needs.merge-image.result == 'success' || needs.merge-image.result == 'skipped') && (needs.bump-descriptors.result == 'success' || needs.bump-descriptors.result == 'skipped') && needs.bump-descriptors.outputs.committed != 'true' @@ -1611,6 +1778,7 @@ jobs: || needs.changes.outputs.identity_resolution == 'true' || needs.changes.outputs.frontend == 'true' || needs.changes.outputs.toolbox == 'true' + || needs.changes.outputs.seed == 'true' || needs.changes.outputs.umbrella == 'true' ) runs-on: ubuntu-latest @@ -1688,14 +1856,15 @@ jobs: FRONTEND: ${{ needs.changes.outputs.frontend }} run: .github/workflows/scripts/bump-service-appversions.sh - - name: Bump ingestion toolbox image ref + - name: Bump ingestion image refs env: BUILD_TAG: ${{ needs.changes.outputs.build_tag }} TOOLBOX: ${{ needs.changes.outputs.toolbox }} + SEED: ${{ needs.changes.outputs.seed }} IMAGE_PREFIX: ${{ env.IMAGE_PREFIX }} run: | set -euo pipefail - # toolbox is the only ingestion image still pinned in + # toolbox and seed are the only ingestion images pinned in # charts/insight/values.yaml — connector image refs travel inside # the toolbox image's baked descriptors (ADR-0016). Connector # descriptors are patched by `bump-descriptors` in a separate @@ -1705,6 +1874,14 @@ jobs: echo "Setting ingestion.toolboxImage → $REF" yq -i ".ingestion.toolboxImage = \"$REF\"" charts/insight/values.yaml fi + # seedImage is not deployed by the chart — nothing renders it into a + # pod. It is published state: `seed-stand.sh` reads it back off an + # installed release to learn which seeder image matches that stand. + if [ "$SEED" = "true" ]; then + REF="$IMAGE_PREFIX/insight-seed:$BUILD_TAG" + echo "Setting ingestion.seedImage → $REF" + yq -i ".ingestion.seedImage = \"$REF\"" charts/insight/values.yaml + fi # Umbrella `version` always patch-bumps (semver, per-publish contract). # Umbrella `appVersion` is DISPLAY-ONLY in the deploy path — the actual @@ -1850,7 +2027,7 @@ jobs: # push is expected to fast-forward. Do NOT rebase the bump commit on # failure: the only way the tip moves under us now is a direct human # push to ${GITHUB_REF_NAME}, and rebasing our bump onto it conflicts - # on the values.yaml lines we just edited (toolboxImage) — + # on the values.yaml lines we just edited (toolboxImage, seedImage) — # unresolvable in CI. Fail loud instead; a plain job re-run is # idempotent (it recomputes against the refreshed tip, see # "Re-anchor" above). diff --git a/.github/workflows/trivy-images.yml b/.github/workflows/trivy-images.yml index db239a2a0..1c3e218f0 100644 --- a/.github/workflows/trivy-images.yml +++ b/.github/workflows/trivy-images.yml @@ -44,6 +44,7 @@ jobs: - insight-frontend # Python tooling - insight-toolbox + - insight-seed - insight-jira-enrich # Python connectors - source-active-directory-insight diff --git a/CONTRIBUTING.md b/CONTRIBUTING.md index 1461fd23f..caeae67a7 100644 --- a/CONTRIBUTING.md +++ b/CONTRIBUTING.md @@ -496,7 +496,7 @@ To force auto-seed on next `up`, clear the `SEEDED_LOCAL_*` markers in ### Kubernetes -No auto-seed, but one command. The seeder ships inside the toolbox +No auto-seed, but one command. The seeder runs from the `insight-seed` image the release already pins, so nothing is built or port-forwarded: ```bash diff --git a/README.md b/README.md index 0f0cc3777..240089e9b 100644 --- a/README.md +++ b/README.md @@ -320,6 +320,7 @@ For cluster deployments image tags flow through automatically: the umbrella char | `insight-analytics` | this repo | https://github.com/constructorfabric/insight/pkgs/container/insight-analytics | | `insight-identity-resolution` | this repo | https://github.com/constructorfabric/insight/pkgs/container/insight-identity-resolution | | `insight-toolbox` | this repo | https://github.com/constructorfabric/insight/pkgs/container/insight-toolbox | +| `insight-seed` (test stands only) | this repo | https://github.com/constructorfabric/insight/pkgs/container/insight-seed | | `insight-frontend` | this repo | https://github.com/constructorfabric/insight/pkgs/container/insight-frontend | | `insight-jira-enrich` | **separate** `constructorfabric/insight-jira-enrich` | https://github.com/constructorfabric/insight/pkgs/container/insight-jira-enrich | diff --git a/charts/insight/values.yaml b/charts/insight/values.yaml index 44a0cf986..8a8ce2dd9 100644 --- a/charts/insight/values.yaml +++ b/charts/insight/values.yaml @@ -192,6 +192,13 @@ ingestion: templates: enabled: true toolboxImage: "ghcr.io/constructorfabric/insight-toolbox:2026.08.06.12.26-aef33a6" # e.g. "ghcr.io/constructorfabric/insight-toolbox:2026.04.28.10.34-b08b460" + # The sample-data seeder's image, for TEST stands only. Nothing in the chart + # runs it: `src/ingestion/tools/seed/seed-stand.sh` reads this value to render + # a one-shot Job. Its own image rather than the toolbox on purpose — the + # toolbox runs migrations on real stands and does not carry demo data. Empty + # (the default) means the stand cannot be seeded until an operator names an + # image, either here or with the script's `--image` flag. + seedImage: "" # e.g. "ghcr.io/constructorfabric/insight-seed:2026.04.28.10.34-b08b460" # airbyte-sync poll-job tunables — see templates/ingestion/airbyte-sync.yaml. # The poll loop replaces a wall-clock deadline with a progress watchdog: # it tracks (status, bytesEmitted, recordsEmitted, stateMessagesEmitted) diff --git a/deploy/HELM_DEPLOY.md b/deploy/HELM_DEPLOY.md index 0e2a435cb..a2b3da6b2 100644 --- a/deploy/HELM_DEPLOY.md +++ b/deploy/HELM_DEPLOY.md @@ -409,7 +409,7 @@ A freshly installed stand holds no people, so every login is refused and every d ./src/ingestion/tools/seed/seed-stand.sh -n insight --email you@example.com ``` -The script reads this stand's own coordinates — infrastructure hosts from the `-platform` ConfigMap, the tenant and identity database from `insight-identity-resolution-config`, the image from `ingestion.toolboxImage` — and runs the seeder as a one-shot Job on that image. Nothing is hand-edited, no credential passes through the shell, and it runs as the application MariaDB user rather than root. `--dry-run` prints the Job it would apply; `--step identity` seeds only the roster. +The script reads this stand's own coordinates — infrastructure hosts from the `-platform` ConfigMap, the tenant and identity database from `insight-identity-resolution-config`, the image from `ingestion.seedImage` — and runs the seeder as a one-shot Job on that image. Nothing is hand-edited, no credential passes through the shell, and it runs as the application MariaDB user rather than root. `--dry-run` prints the Job it would apply; `--step identity` seeds only the roster. Two things it cannot do for you: a user with the `--email` address must already exist in your IdP (the authenticator resolves people by the email claim), and the ClickHouse schema must exist already — that is Step 4's migration hook. diff --git a/docker-compose.yml b/docker-compose.yml index 82e5640ba..2b757f020 100644 --- a/docker-compose.yml +++ b/docker-compose.yml @@ -700,9 +700,12 @@ services: profiles: ["seed"] container_name: ${COMPOSE_PROJECT_NAME:-insight}-seed-sample networks: [insight] + # Context is the ingestion tree: the image carries the seeder AND the DDL / + # gold-build scripts its silver step runs. Same image CI publishes for the + # cluster Job — the toolbox deliberately carries neither. build: - context: src/ingestion/tools/seed - dockerfile: Dockerfile + context: src/ingestion + dockerfile: tools/seed/Dockerfile # The image pins USER 1000. That is unambiguous where it was chosen — as a # k8s Job the seed mounts nothing from a host — but compose bind-mounts the # source below and the seed writes manifest.json back into it, so the @@ -770,16 +773,9 @@ services: # records the real IdP instead of assuming the default. AUTHENTICATOR_OIDC_ISSUER: "${AUTHENTICATOR_OIDC_ISSUER:-}" volumes: - # Bind-mount the source so seed.py edits don't need a rebuild. - # NOT read-only: the seed writes manifest.json back into this directory - # at the end of a successful run, and downstream test phases read it - # from that fixed path. - - ./src/ingestion/tools/seed:/app - # Schema + gold inputs: the SAME scripts the k8s clickhouse-migrate - # Hook Job runs — create-bronze-placeholders.sh (+ lib/ch-exec.sh), - # apply-ch-migrations.sh, migrations/*.sql — plus the full dbt project - # (dbt/, silver/, gold/, connectors/ referenced by dbt model-paths) - # so `dbt run --select tag:gold` can build the dbt-owned gold models. - # Mounted at /ingestion to match the toolbox image layout the scripts - # resolve paths against. - - ./src/ingestion:/ingestion:ro + # One mount, writable, over the tree the image already carries: seeder + # edits and script edits both take effect without a rebuild (the package + # is installed editable), and the seed writes manifest.json back into + # tools/seed at the end of a run — the fixed path downstream test phases + # read. Read-only would fail that last step after all the real work. + - ./src/ingestion:/ingestion diff --git a/src/ingestion/README.md b/src/ingestion/README.md index b41f396a4..8cb802957 100644 --- a/src/ingestion/README.md +++ b/src/ingestion/README.md @@ -325,6 +325,10 @@ src/ingestion/ ├── toolbox/ # insight-toolbox Docker image │ ├── Dockerfile # python + dbt + kubectl + yq │ └── build.sh # Build + push to GHCR (or load into Kind) + ├── seed/ # insight-seed Docker image (TEST stands) + │ ├── insight_seed/ # the package: `insight-seed ` + │ ├── seed-stand.sh # seeds a chart-deployed stand as a Job + │ └── README.md # what it writes, and the env contract └── declarative-connector/ # Local connector debugging ├── source.sh # check / discover / read ├── generate-catalog.sh # Render Airbyte catalog from connector.yaml diff --git a/src/ingestion/tools/seed/Dockerfile b/src/ingestion/tools/seed/Dockerfile index 5c826ef04..ad6cb8848 100644 --- a/src/ingestion/tools/seed/Dockerfile +++ b/src/ingestion/tools/seed/Dockerfile @@ -1,49 +1,66 @@ -# One-shot image for the sample-data seed script. +# The sample-data seeder's own image — the only one that carries it. # -# Installs deps + the seeder from pyproject.toml (single source of truth). -# The script is bind-mounted at runtime (see docker-compose.yml seed-sample) -# so iterating on seed.py doesn't require a rebuild; rebuild when -# pyproject.toml or the packaged sources change. - -# Above the package's >=3.12 floor rather than at it: this image is only the -# compose runner, and the floor exists for the toolbox image (python:3.12-slim) -# that carries the same package into a cluster. dbt's dependency chain -# (dbt-common -> mashumaro) does not import on 3.14 yet, and dependabot.yml -# ignores that bump until it does. -FROM python:3.13-slim +# Deliberately NOT the toolbox: that image runs migrations on real stands, and +# a demo-data generator has no business being installed there. This one carries +# the seeder plus exactly what its silver step shells out to — the ingestion +# tree's DDL and gold-build scripts and the dbt project they run — and none of +# the toolbox's operator tooling (node, kubectl, yq). +# +# Build context is the INGESTION TREE, not this directory: +# +# docker build -f src/ingestion/tools/seed/Dockerfile src/ingestion +# +# Both callers use it: compose builds it locally (see docker-compose.yml +# seed-sample), and CI publishes it as `insight-seed` for `seed-stand.sh` to +# run as a Job. + +# 3.12 to match the package's floor and the rest of the ingestion tree; dbt's +# dependency chain does not import on 3.14 yet. +FROM python:3.12-slim ENV PYTHONDONTWRITEBYTECODE=1 \ PYTHONUNBUFFERED=1 \ PIP_DISABLE_PIP_VERSION_CHECK=1 \ - PIP_NO_CACHE_DIR=1 + PIP_NO_CACHE_DIR=1 \ + HOME=/tmp -WORKDIR /app - -# curl: needed by src/ingestion/scripts/{create-bronze-placeholders, -# apply-ch-migrations}.sh (bind-mounted at /ingestion), which silver.py runs -# to create the placeholder tables + build the gold layer — the same scripts -# the k8s clickhouse-migrate Hook Job runs. -# bash: the scripts use bashisms (process substitution, [[ ]]); slim ships dash. +# curl + bash: scripts/create-bronze-placeholders.sh and +# scripts/apply-ch-migrations.sh use both, and slim ships dash and no curl. RUN apt-get update \ && apt-get install -y --no-install-recommends curl bash \ && rm -rf /var/lib/apt/lists/* -# Source is copied before the install so the build backend can package it. At -# runtime the tool directory is bind-mounted over /app (see docker-compose.yml -# seed-sample service); this COPY is also the fallback for -# `docker build && docker run` without a volume. -COPY pyproject.toml . -COPY insight_seed ./insight_seed/ +# The slow layer, before any source is copied so it survives every edit below. +# dbt is the silver step's environment rather than an import of the package, +# which is why the package declares it as an extra and it is installed here. +RUN pip install --no-cache-dir dbt-clickhouse + +WORKDIR /ingestion -# EDITABLE, unlike the toolbox image's install: compose mounts the working tree -# over /app so seeder edits need no rebuild, and a regular install would keep -# executing the copy baked in above while the developer edited a file that never -# ran. The silver extra brings dbt, which this image's step runs. -RUN pip install -e '.[silver]' +# Only what the seed actually runs. `dbt_project.yml` reads its models from +# ../silver, ../gold and ../connectors, so those come along; the connector +# runtimes, reconcile tooling, tests and secrets do not. +COPY scripts ./scripts/ +COPY dbt ./dbt/ +COPY silver ./silver/ +COPY gold ./gold/ +COPY connectors ./connectors/ +COPY tools/seed ./tools/seed/ -RUN useradd -U -u 1000 -m appuser +# Editable, so the compose service can mount the working tree over /ingestion +# and have its edits take effect without a rebuild. The console script it +# installs (`insight-seed`) is the entry point for both callers. +RUN pip install --no-cache-dir -e ./tools/seed + +# dbt writes target/ and logs/ under the project root and the seeder writes its +# manifest beside the package, so the tree belongs to the runtime user. +RUN useradd -U -u 1000 -m appuser && chown -R 1000:1000 /ingestion USER 1000 -# A program on PATH, installed above — same entry point as the cluster Job. +# The manifest lands in the working directory (see config.parse_manifest_path), +# which is the seeder's own directory — the path compose bind-mounts and the +# stand test suite reads. +WORKDIR /ingestion/tools/seed + ENTRYPOINT ["insight-seed"] CMD ["all"] diff --git a/src/ingestion/tools/seed/README.md b/src/ingestion/tools/seed/README.md index 4654000d5..9feb07b33 100644 --- a/src/ingestion/tools/seed/README.md +++ b/src/ingestion/tools/seed/README.md @@ -8,13 +8,16 @@ roster and the per-team source-type weights; the per-domain generators under — roster, fixtures, populated metrics and capabilities. It runs against two kinds of stand, from the same sources: the local -docker-compose stack, and a chart-deployed Kubernetes stand (the package ships -inside the toolbox image, so no separate image or build is involved). +docker-compose stack, and a chart-deployed Kubernetes stand. Both run the same +image, `insight-seed`, built from [`Dockerfile`](Dockerfile) — compose builds it +locally, CI publishes it. This package lives inside `src/ingestion` deliberately: the silver step runs the ingestion tree's own DDL and gold-build scripts, and being in the same tree means -the published toolbox image carries both, at one version, with no chance of the -seeder and the migration SQL drifting apart. +one image carries both, at one version, with no chance of the seeder and the +migration SQL drifting apart. It is deliberately NOT the toolbox image — that one +runs migrations against real stands, and a demo-data generator has no business +being installed there. ## Run it on compose @@ -49,7 +52,7 @@ is no manifest to hand-edit and no tenant UUID to copy: | database holding the analytics catalogue | ConfigMap `-platform`, `MARIADB_DATABASE` | | database holding `persons` | Secret `insight-identity-resolution-config`, `…database_url` | | the stand's tenant | Secret `insight-identity-resolution-config`, `…tenant_default_id` | -| the image to run | `helm get values `, `ingestion.toolboxImage` | +| the image to run | `helm get values `, `ingestion.seedImage` | | passwords | never read — the Job references Secret `insight-db-creds` by key | Credentials never pass through the script, and the Job runs as the application @@ -86,16 +89,19 @@ Preflight runs before anything is written and reports every problem at once: - MariaDB or ClickHouse unreachable, or the ingestion scripts missing; - the target tenant already holds `persons` rows this seeder did not write (every row it writes carries a `reason` starting `"seed.py "`, trailing space - included); + included). The silver step checks this too — on a stand that has only ever had + one tenant it is the only signal that can tell somebody's data from nothing; - any table the silver step clears holds rows for another tenant — that step **TRUNCATEs every table it writes**, across all tenants, so those rows would be destroyed. This is the one genuinely destructive thing the seeder does, and it is why an occupied stand is refused rather than merged into. The surface it checks is `generators.base.RESET_TARGETS`, the same list `truncate` itself enforces — including two inputs outside the silver database (an - identity-projection table and a bronze HR table). Targets carrying no tenant - column at all cannot be attributed to anyone; the run logs them by name - instead of pretending to have judged them. + identity-projection table and a bronze HR table). This check is differential, + so it finds nothing on a single-tenant stand; that is what the `persons` signal + above is for. Targets carrying no tenant column at all cannot be attributed to + anyone; the run logs them by name, and logs how many rows the step is about to + clear in total, instead of pretending to have judged them. Either refusal is overridable with `--force`, which is how you say "yes, clear it" out loud. @@ -143,12 +149,13 @@ the tests import `insight_seed` the same way anything else does — no `sys.path` juggling and no stubbed modules. A hand-made venv works identically (`python3 -m venv .venv && .venv/bin/pip install -e '.[dev]'`). -Both images install the package too, so every runner invokes a program rather +The image installs the package too, so every runner invokes a program rather than a module in a directory: `insight-seed ` seeds, and `insight-seed-realm` generates the compose Keycloak realm from the same roster (`dev-compose.sh` runs it through `uv run --project`). The extras split what -each caller needs: `silver` adds dbt for the gold build (the toolbox image -installs it separately), `dev` adds ruff, mypy and stubs. +each caller needs: `silver` adds dbt for the gold build (the image installs it +in its own layer, before the source, so an edit here does not re-resolve it), +`dev` adds ruff, mypy and stubs. The tests touch no database: they cover the pure half — the environment contract, the SQL a guard issues, and the messages a refusal carries. @@ -181,13 +188,28 @@ src/ingestion/tools/seed/ ├── tests/ stdlib unittest against the installed package ├── seed-stand.sh seeds a Kubernetes stand (discover → render → apply) ├── seed-job.yaml.tpl the Job it renders — and the reference manifest -├── Dockerfile the compose `seed-sample` image +├── Dockerfile the `insight-seed` image, for both callers ├── pyproject.toml package metadata, deps, ruff + mypy config ├── PROFILE.md GENERATED, committed — do not hand-edit └── manifest.json GENERATED per stand at seed time (gitignored) ``` -On a cluster the runtime is the toolbox image (`../toolbox/Dockerfile`): it -carries this tree at `/ingestion/tools/seed` together with the migration scripts -the silver step runs, and installs the package, so the Job's command is just -`insight-seed` — no shell, no working directory, no path assumptions. +## The image + +[`Dockerfile`](Dockerfile) builds from the ingestion tree, not this directory: + +```bash +docker build -f src/ingestion/tools/seed/Dockerfile src/ingestion +``` + +It carries this tree at `/ingestion/tools/seed` together with the DDL and +gold-build scripts the silver step runs and the dbt project they drive, installs +the package, and sets `insight-seed` as the entry point — so the Job's command is +just the step name: no shell, no working directory, no path assumptions. What it +does not carry is the toolbox's operator tooling (node, kubectl, yq); what the +toolbox does not carry is this tree, excluded there by +`../toolbox/Dockerfile.dockerignore`. + +CI builds and publishes it as `insight-seed` beside the toolbox +(`.github/workflows/build-images.yml`) and pins the pushed ref into the chart as +`ingestion.seedImage`, which is the ref `seed-stand.sh` discovers. diff --git a/src/ingestion/tools/seed/seed-job.yaml.tpl b/src/ingestion/tools/seed/seed-job.yaml.tpl index 0ce453ede..ed972daa9 100644 --- a/src/ingestion/tools/seed/seed-job.yaml.tpl +++ b/src/ingestion/tools/seed/seed-job.yaml.tpl @@ -8,7 +8,7 @@ # Expected variables (all required, all exported by seed-stand.sh): # SEED_JOB_NAME Job name, unique per run # SEED_NAMESPACE namespace holding the release and its creds Secret -# SEED_IMAGE toolbox image carrying tools/seed (the chart's own pin) +# SEED_IMAGE the seeder image (the chart's own pin) # SEED_STEP identity | silver | analytics | all # SEED_DEADLINE_SECONDS wall-clock ceiling for the pod # SEED_DB_SECRET Secret with mariadb-password + clickhouse-password @@ -53,8 +53,8 @@ spec: # seeder's own variables. enableServiceLinks: false # The same secrets the release's own Jobs use for this image; `[]` on a - # stand that pulls it anonymously. Without them a private toolbox image - # leaves the pod in ImagePullBackOff. + # stand that pulls it anonymously. Without them a private image leaves + # the pod in ImagePullBackOff. imagePullSecrets: ${SEED_PULL_SECRETS} containers: - name: seed @@ -132,8 +132,8 @@ spec: - name: SEED_MANIFEST_PATH value: /tmp/manifest.json # dbt (run by the silver step's gold build) writes target/, logs/ - # and ~/.dbt under whatever it is given; the toolbox image owns - # /ingestion, but keep the scratch paths explicit. + # and ~/.dbt under whatever it is given; the image owns /ingestion, + # but keep the scratch paths explicit. - name: DBT_TARGET_PATH value: /tmp/dbt-target - name: DBT_LOG_PATH diff --git a/src/ingestion/tools/seed/seed-stand.sh b/src/ingestion/tools/seed/seed-stand.sh index c9e1e55b8..8ba37edac 100755 --- a/src/ingestion/tools/seed/seed-stand.sh +++ b/src/ingestion/tools/seed/seed-stand.sh @@ -9,7 +9,7 @@ # holds the analytics catalogue # Secret insight-identity-resolution-* the stand's tenant, and the database # holding `persons` -# helm get values the toolbox image the release pins +# helm get values the seed image the release pins # # Credentials are never read: the rendered Job references the release's own # database Secret by key, so nothing sensitive passes through this shell. @@ -53,7 +53,7 @@ usage() { Usage: seed-stand.sh -n --email
[options] Runs the demo-data seeder as a one-shot Job on a chart-deployed stand, using the -toolbox image the release already pins. +seeder image the release already pins. Required: -n, --namespace namespace the Insight release runs in @@ -65,7 +65,7 @@ Discovered from the stand (pass a flag only to override): --context kube context to act on [default: the current one] --release helm release name [default: same as -n] --tenant tenant every seeded row is scoped to - --image toolbox image to run + --image seeder image to run (chart: ingestion.seedImage) --analytics-db database holding metric_definitions --identity-db database holding persons --db-secret Secret holding mariadb-password + clickhouse-password @@ -187,14 +187,24 @@ kube -n "$NAMESPACE" get configmap "$platform_cm" >/dev/null 2>&1 || die \ Check --release (currently '$RELEASE')." cm_value() { - kube -n "$NAMESPACE" get configmap "$platform_cm" -o "jsonpath={.data.$1}" + # Same reasoning as secret_value below: a failed read must reach the + # missing-values report, not `set -e`. + kube -n "$NAMESPACE" get configmap "$platform_cm" -o "jsonpath={.data.$1}" 2>/dev/null || true } secret_value() { - # $1 = secret, $2 = key. Missing secret or key yields an empty string, which - # every caller treats as "not discovered". - kube -n "$NAMESPACE" get secret "$1" -o "jsonpath={.data.$2}" 2>/dev/null \ - | { base64 --decode 2>/dev/null || true; } + # $1 = secret, $2 = key. An absent secret, an absent key, or a read this + # caller is not allowed to make all yield an empty string, which every call + # site treats as "not discovered" and reports through the missing-values list. + # + # The `|| true` is on the READ, not on the decode: under `pipefail` a failing + # kubectl at the head of a pipeline sets the whole pipeline's status, and a + # bare `X="$(secret_value …)"` assignment would then take `set -e` with it — + # killing the script before it can name the flag that fixes the problem. + local encoded + encoded="$(kube -n "$NAMESPACE" get secret "$1" -o "jsonpath={.data.$2}" 2>/dev/null || true)" + [[ -n "$encoded" ]] || return 0 + printf '%s' "$encoded" | base64 --decode 2>/dev/null || true } MARIADB_HOST="$(cm_value MARIADB_HOST)" @@ -285,7 +295,10 @@ if [[ -z "$IMAGE" || -z "$PULL_SECRETS" ]]; then release_values="$(helm_release get values "$RELEASE" -n "$NAMESPACE" -a -o json 2>/dev/null || true)" if [[ -n "$release_values" ]]; then if [[ -z "$IMAGE" ]]; then - IMAGE="$(printf '%s' "$release_values" | jq -r '.ingestion.toolboxImage // empty')" + # `ingestion.seedImage`, NOT the toolbox: the seeder ships in an image of + # its own so the operator toolbox carries no demo data. A release that + # never set it cannot be seeded until someone names one. + IMAGE="$(printf '%s' "$release_values" | jq -r '.ingestion.seedImage // empty')" fi if [[ -z "$PULL_SECRETS" ]]; then # Rendered as a YAML flow sequence so the template needs no conditional: @@ -336,7 +349,11 @@ check "$CLICKHOUSE_DATABASE" "ClickHouse database (ConfigMap $platform_cm)" "--r check "$ANALYTICS_DB" "analytics catalogue database" "--analytics-db" check "$IDENTITY_DB" "identity database (Secret $ir_secret, database_url)" "--identity-db" check "$TENANT" "stand tenant (Secret $ir_secret, tenant_default_id)" "--tenant" -check "$IMAGE" "toolbox image (helm values ingestion.toolboxImage)" "--image" +check "$IMAGE" \ + "the seeder's image. The chart carries it as ingestion.seedImage, empty by + default because seeding is a test-stand activity; CI publishes the image as + insight-seed alongside every toolbox build" \ + "--image" check "$DB_SECRET" "database-credentials Secret" "--db-secret" check "$IDP_SOURCE_TYPE" \ "the identity source_type the stand's logins resolve under. Newer charts @@ -451,8 +468,9 @@ while [[ "$SECONDS" -lt "$deadline" ]]; do exit 0 fi if [[ "${failed:-0}" -ge 1 ]]; then - echo "ERROR: Job $job_name failed. Its logs above hold the reason; the Job is kept" >&2 - echo " (backoffLimit 0, no retry) so it can be read again:" >&2 + echo "ERROR: Job $job_name failed. Its logs above hold the reason; it is not" >&2 + echo " retried (backoffLimit 0) and survives for an hour" >&2 + echo " (ttlSecondsAfterFinished), so it can be read again until then:" >&2 echo " $(kubectl_hint) -n $NAMESPACE logs job/$job_name" >&2 exit 1 fi diff --git a/src/ingestion/tools/toolbox/Dockerfile b/src/ingestion/tools/toolbox/Dockerfile index a1964fc0b..e02e279da 100644 --- a/src/ingestion/tools/toolbox/Dockerfile +++ b/src/ingestion/tools/toolbox/Dockerfile @@ -22,25 +22,17 @@ RUN curl -fsSL "https://github.com/mikefarah/yq/releases/latest/download/yq_linu RUN curl -fsSL "https://dl.k8s.io/release/$(curl -fsSL https://dl.k8s.io/release/stable.txt)/bin/linux/$(dpkg --print-architecture)/kubectl" \ -o /usr/local/bin/kubectl && chmod +x /usr/local/bin/kubectl -# dbt-clickhouse. Installed before the source is copied so this slow layer -# survives every ingestion change; it is also what the sample-data seeder's -# silver step needs at run time (apply-ch-migrations.sh writes a profiles.yml -# and runs dbt), which is why that package declares it as an extra rather than -# a hard dependency. +# dbt-clickhouse RUN pip install --no-cache-dir dbt-clickhouse -# Copy entire ingestion project. This is also how the seeder ships: it lives at -# tools/seed inside this tree, so it runs against the same scripts/ and dbt/ the -# migrations use — one image, one version, no drift between seeder and migration -# SQL. +# Copy the ingestion project, minus the sample-data seeder: this image runs +# migrations against real stands, and a demo-data generator does not belong on +# one. The exclusion lives in Dockerfile.dockerignore beside this file, so it +# applies to THIS image only — the seeder's own image (tools/seed/Dockerfile) +# builds from the same context and does carry it. COPY . /ingestion WORKDIR /ingestion -# Install the seeder as a package, so the Job runs `insight-seed` from PATH -# rather than a module out of a working directory, and its drivers come from its -# own metadata instead of a second list here that could drift from it. -RUN pip install --no-cache-dir /ingestion/tools/seed - # dbt writes logs/ and target/ under the project root, so the tree must belong # to the runtime user. RUN useradd -U -u 1000 -m appuser && chown -R 1000:1000 /ingestion diff --git a/src/ingestion/tools/toolbox/Dockerfile.dockerignore b/src/ingestion/tools/toolbox/Dockerfile.dockerignore new file mode 100644 index 000000000..f7b307810 --- /dev/null +++ b/src/ingestion/tools/toolbox/Dockerfile.dockerignore @@ -0,0 +1,27 @@ +# Ignore file for tools/toolbox/Dockerfile ONLY — BuildKit prefers +# `.dockerignore` over the context's `.dockerignore`, which is how +# two images built from the same context can carry different things. +# +# Everything the context-level file excludes, repeated because this file +# replaces it rather than adding to it: +**/__pycache__/ +**/*.py[cod] +**/.venv/ +**/.pytest_cache/ +**/.ruff_cache/ +**/.mypy_cache/ +**/*.egg-info/ +secrets/ +.env.local +*.env.local +tools/declarative-connector/ +workflows/example-tenant/ +workflows/*/ +!workflows/templates/ +!workflows/schedules/ +dbt/target/ +dbt/logs/ + +# And the point of this file: the sample-data seeder ships in its own image +# (tools/seed/Dockerfile), never in the operator toolbox. +tools/seed/ From 75a9f07171a4fcc8beeb84c00e2df7d5ab5102fb Mon Sep 17 00:00:00 2001 From: Konstantin Tursunov Date: Thu, 6 Aug 2026 17:51:47 +0800 Subject: [PATCH 08/10] fix(authenticator): point the e2e rig at the relocated realm generator MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit The rig landed on main while this branch was open and calls the generator by its old path — `python3 deploy/compose/keycloak/gen-realm.py`, a file this branch deletes. It runs through `uv run --project` now, the same way dev-compose.sh and the gitops `keycloak-realm` target invoke it, and the job installs uv for it. The generator also requires `TENANT_DEFAULT_ID` rather than defaulting to a stand's, so the rig names one. Which tenant is immaterial here — person resolution is keyed by email — but every realm user carries the claim and a realm that will not build is the cheaper failure. The workflow's paths filter followed the roster to its new home, so a change to the people the realm is built from still re-runs the suite. Verified: the rig's exact invocation produces the realm it expects — 27 users, both clients, the three rig redirect URIs replacing the compose defaults — and `kc-realm-overlay.py` consumes it unchanged, emitting both realms with the rig's 9 test users layered on. Signed-off-by: Konstantin Tursunov --- .github/workflows/authenticator.yml | 12 ++++++++++-- .../authenticator/tests/kc-realm-overlay.py | 4 ++-- .../services/authenticator/tests/run-e2e.sh | 19 +++++++++++++++++-- 3 files changed, 29 insertions(+), 6 deletions(-) diff --git a/.github/workflows/authenticator.yml b/.github/workflows/authenticator.yml index 8a4e49f08..a85e0649d 100644 --- a/.github/workflows/authenticator.yml +++ b/.github/workflows/authenticator.yml @@ -16,9 +16,10 @@ on: - "src/backend/services/authenticator/**" - "src/backend/libs/authenticator-sdk/**" # The e2e stack imports the generated compose realm (run-e2e.sh), which - # gen-realm.py builds from the seed roster. + # `insight-seed-realm` builds from the seed roster. - "deploy/compose/keycloak/**" - - "deploy/seed/profiles.py" + - "src/ingestion/tools/seed/insight_seed/profiles.py" + - "src/ingestion/tools/seed/insight_seed/keycloak_realm.py" # A workspace-wide dependency bump can break the authenticator build # without touching its sources (reqwest 0.12 -> 0.13 did, via # openidconnect's pinned HTTP-client impls). @@ -61,6 +62,13 @@ jobs: workspaces: src/backend -> target key: authenticator-e2e + # run-e2e.sh generates the realm through `uv run --project`: the + # generator is a console script of the seed package, and uv resolves and + # installs it on first use. The runner has python3 but not uv. + - uses: astral-sh/setup-uv@37802adc94f370d6bfd71619e3f0bf239e1f3b78 # v7 + with: + cache-suffix: authenticator-e2e + - name: Run the authenticator e2e suite # run-e2e.sh builds the authenticator (release), boots the stack # (Redis + Keycloak with the imported generated realm), runs every diff --git a/src/backend/services/authenticator/tests/kc-realm-overlay.py b/src/backend/services/authenticator/tests/kc-realm-overlay.py index c83a839f4..df7ae3d36 100755 --- a/src/backend/services/authenticator/tests/kc-realm-overlay.py +++ b/src/backend/services/authenticator/tests/kc-realm-overlay.py @@ -1,7 +1,7 @@ #!/usr/bin/env python3 """Turn the generated compose realm into the e2e rig's Keycloak import set. -Input is `gen-realm.py` output (the roster realm the compose stack imports). +Input is `insight-seed-realm` output (the roster realm the compose stack imports). The rig needs the same realm plus what only the e2e suites care about: - a short access-token lifespan, so the authenticator's background refresher @@ -67,7 +67,7 @@ def _test_user(email: str, password: str, tenant_id: str) -> dict: def main() -> None: parser = argparse.ArgumentParser(description=__doc__) - parser.add_argument("--realm", required=True, help="gen-realm.py output JSON") + parser.add_argument("--realm", required=True, help="insight-seed-realm output JSON") parser.add_argument("--out-dir", required=True, help="Keycloak import directory") parser.add_argument("--second-realm", required=True, help="Name of the user-less second realm") parser.add_argument("--backchannel-url", required=True, help="insight-authenticator back-channel logout URL") diff --git a/src/backend/services/authenticator/tests/run-e2e.sh b/src/backend/services/authenticator/tests/run-e2e.sh index 738c8086b..924ac652f 100755 --- a/src/backend/services/authenticator/tests/run-e2e.sh +++ b/src/backend/services/authenticator/tests/run-e2e.sh @@ -8,7 +8,8 @@ # /internal/authz returns 401. # # The IdP is a real Keycloak importing the generated compose realm -# (deploy/compose/keycloak/gen-realm.py) with the rig's overlay on top +# (`insight-seed-realm`, from src/ingestion/tools/seed) with the rig's overlay +# on top # (tests/kc-realm-overlay.py: test users, fast token lifespan, back-channel # registration, and a second realm for the host-keyed issuer map). What the # retired fakeidp offered as `/_control/*` hooks the suites now drive through @@ -51,6 +52,15 @@ KC_REALM_B=insight-b KC_ADMIN_USER=admin KC_ADMIN_PASSWORD=admin E2E_USER=dev@company.nonpresent +# The realm generator lives in the seed package and is run through uv, which +# resolves and installs it on first use — the same way dev-compose.sh and the +# gitops `keycloak-realm` target invoke it. +SEED_DIR="$ROOT_DIR/src/ingestion/tools/seed" +# Every realm user carries a tenant claim and the generator requires one rather +# than defaulting to a stand's. Which tenant is immaterial here: the rig +# resolves people by email (idp.external_id_claim=email), so this only has to +# be named, and it is the value the compose stack uses. +KC_TENANT_ID=00000000-df51-5b42-9538-d2b56b7ee953 pids=() cleanup() { @@ -95,7 +105,12 @@ rm -rf "$KC_IMPORT_DIR" && mkdir -p "$KC_IMPORT_DIR" # The redirect URIs are the three authenticator instances below; the compose # defaults would deregister them (--authenticator-redirect REPLACES, not # appends). -python3 "$ROOT_DIR/deploy/compose/keycloak/gen-realm.py" \ +command -v uv >/dev/null 2>&1 || { + echo "uv is required to generate the realm — https://docs.astral.sh/uv/getting-started/installation/" >&2 + exit 1 +} +TENANT_DEFAULT_ID="$KC_TENANT_ID" \ +uv run --project "$SEED_DIR" insight-seed-realm \ --dev-email "$E2E_USER" \ --authenticator-redirect "http://localhost:$AUTH_PORT/auth/callback" \ --authenticator-redirect "http://localhost:$AUTH2_PORT/auth/callback" \ From e1a820478e177e9a0f7751889ae1b60ee8863fd6 Mon Sep 17 00:00:00 2001 From: Konstantin Tursunov Date: Thu, 6 Aug 2026 17:56:10 +0800 Subject: [PATCH 09/10] fix(seed): register the task issue-type relation in the reset surface MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit A generator that truncates a relation the registry does not list is a runtime failure: `truncate` refuses an unregistered target, so the silver step would have died partway through, and preflight would have scanned one relation short of what the step actually clears. `class_task_issuetypes` arrived on main while this branch was open. The registry test is what caught it — which is the reason it enumerates the generators' call sites rather than trusting a name pattern. PROFILE.md regenerated: `seed_revision` is a content hash, so it moves with any change to what the seed writes. Signed-off-by: Konstantin Tursunov --- src/ingestion/tools/seed/PROFILE.md | 2 +- src/ingestion/tools/seed/insight_seed/generators/base.py | 1 + 2 files changed, 2 insertions(+), 1 deletion(-) diff --git a/src/ingestion/tools/seed/PROFILE.md b/src/ingestion/tools/seed/PROFILE.md index 9ffe46e41..b624861a2 100644 --- a/src/ingestion/tools/seed/PROFILE.md +++ b/src/ingestion/tools/seed/PROFILE.md @@ -17,7 +17,7 @@ builder that writes `manifest.json`, so the two cannot disagree. | realm | `insight` | | anchor_date | `2026-06-30` | | data_window | `2026-05-02..2026-06-30` | -| seed_revision | `abc7d57fccf41b41` | +| seed_revision | `8085d4bfcbe92ae4` | | manifest_version | 1 | `anchor_date` is the last day carrying seeded activity. It is resolved diff --git a/src/ingestion/tools/seed/insight_seed/generators/base.py b/src/ingestion/tools/seed/insight_seed/generators/base.py index 7d13e2bcf..669335bce 100644 --- a/src/ingestion/tools/seed/insight_seed/generators/base.py +++ b/src/ingestion/tools/seed/insight_seed/generators/base.py @@ -184,6 +184,7 @@ def deterministic_int(*parts: str) -> int: ("silver", "class_people"), ("silver", "class_support_activity"), ("silver", "class_task_field_history"), + ("silver", "class_task_issuetypes"), ("silver", "class_task_statuses"), ("silver", "class_task_users"), ("silver", "class_task_worklogs"), From eb6f054cfe4334afb569f91ec4fe0df77e329c41 Mon Sep 17 00:00:00 2001 From: Konstantin Tursunov Date: Fri, 7 Aug 2026 09:55:00 +0800 Subject: [PATCH 10/10] ci(seed): match the action pins the actions bump landed on main MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit The seed and merge-seed jobs were written against the pins that were current when this branch opened. The actions group has moved since, so replaying the branch would have reintroduced eight superseded versions in the two jobs nobody else touches — checkout, buildx, login, build-push, upload/download artifact, metadata and attest — plus the realm generator's uv setup in the authenticator lane. Every pin this branch adds now matches what the rest of the file uses. Signed-off-by: Konstantin Tursunov --- .github/workflows/authenticator.yml | 2 +- .github/workflows/build-images.yml | 22 +++++++++++----------- 2 files changed, 12 insertions(+), 12 deletions(-) diff --git a/.github/workflows/authenticator.yml b/.github/workflows/authenticator.yml index a85e0649d..57a6246e7 100644 --- a/.github/workflows/authenticator.yml +++ b/.github/workflows/authenticator.yml @@ -65,7 +65,7 @@ jobs: # run-e2e.sh generates the realm through `uv run --project`: the # generator is a console script of the seed package, and uv resolves and # installs it on first use. The runner has python3 but not uv. - - uses: astral-sh/setup-uv@37802adc94f370d6bfd71619e3f0bf239e1f3b78 # v7 + - uses: astral-sh/setup-uv@c771a70e6277c0a99b617c7a806ffedaca235ff9 # v9.0.0 with: cache-suffix: authenticator-e2e diff --git a/.github/workflows/build-images.yml b/.github/workflows/build-images.yml index 09711065b..a7013e38d 100644 --- a/.github/workflows/build-images.yml +++ b/.github/workflows/build-images.yml @@ -1136,11 +1136,11 @@ jobs: platform: linux/arm64 runs-on: ${{ matrix.runner }} steps: - - uses: actions/checkout@11d5960a326750d5838078e36cf38b85af677262 # v4 + - uses: actions/checkout@3d3c42e5aac5ba805825da76410c181273ba90b1 # v7.0.1 - - uses: docker/setup-buildx-action@8d2750c68a42422c14e847fe6c8ac0403b4cbd6f # v3 + - uses: docker/setup-buildx-action@bb05f3f5519dd87d3ba754cc423b652a5edd6d2c # v4.2.0 - - uses: docker/login-action@c94ce9fb468520275223c153574b00df6fe4bcc9 # v3 + - uses: docker/login-action@dbcb813823bdd20940b903addbd779551569679f # v4.6.0 if: needs.changes.outputs.should_push == 'true' with: registry: ${{ env.REGISTRY }} @@ -1152,7 +1152,7 @@ jobs: # and the push-build reuses the buildx cache. - name: Build seed image (load locally, amd64 only) if: matrix.arch == 'amd64' - uses: docker/build-push-action@263435318d21b8e681c14492fe198d362a7d2c83 # v6.18.0 + uses: docker/build-push-action@53b7df96c91f9c12dcc8a07bcb9ccacbed38856a # v7.3.0 with: context: src/ingestion file: src/ingestion/tools/seed/Dockerfile @@ -1186,7 +1186,7 @@ jobs: - name: Build and push seed image (by digest) id: build - uses: docker/build-push-action@263435318d21b8e681c14492fe198d362a7d2c83 # v6.18.0 + uses: docker/build-push-action@53b7df96c91f9c12dcc8a07bcb9ccacbed38856a # v7.3.0 with: context: src/ingestion file: src/ingestion/tools/seed/Dockerfile @@ -1203,7 +1203,7 @@ jobs: touch "/tmp/digests/${digest#sha256:}" - name: Upload digest if: needs.changes.outputs.should_push == 'true' - uses: actions/upload-artifact@ea165f8d65b6e75b540449e92b4886f43607fa02 # v4 + uses: actions/upload-artifact@043fb46d1a93c77aae656e7c1c64a875d1fc6a0a # v7.0.1 with: name: digests-seed-${{ matrix.arch }} path: /tmp/digests/* @@ -1224,18 +1224,18 @@ jobs: echo "::error::seed finished '${{ needs.seed.result }}' — nothing to merge." exit 1 - name: Download digests - uses: actions/download-artifact@d3f86a106a0bac45b974a628896c90dbdf5c8093 # v4 + uses: actions/download-artifact@3e5f45b2cfb9172054b4087a40e8e0b5a5461e7c # v8.0.1 with: path: /tmp/digests pattern: digests-seed-* merge-multiple: true - - uses: docker/setup-buildx-action@8d2750c68a42422c14e847fe6c8ac0403b4cbd6f # v3 - - uses: docker/login-action@c94ce9fb468520275223c153574b00df6fe4bcc9 # v3 + - uses: docker/setup-buildx-action@bb05f3f5519dd87d3ba754cc423b652a5edd6d2c # v4.2.0 + - uses: docker/login-action@dbcb813823bdd20940b903addbd779551569679f # v4.6.0 with: registry: ${{ env.REGISTRY }} username: ${{ github.actor }} password: ${{ secrets.GITHUB_TOKEN }} - - uses: docker/metadata-action@c299e40c65443455700f0fdfc63efafe5b349051 # v5 + - uses: docker/metadata-action@dc802804100637a589fabce1cb79ff13a1411302 # v6.2.0 id: meta with: images: ${{ env.IMAGE_PREFIX }}/insight-seed @@ -1256,7 +1256,7 @@ jobs: --format '{{json .Manifest}}' | jq -r .digest) echo "digest=$DIGEST" >> "$GITHUB_OUTPUT" - name: Attest build provenance - uses: actions/attest-build-provenance@e8998f949152b193b063cb0ec769d69d929409be # v2 + uses: actions/attest-build-provenance@0f67c3f4856b2e3261c31976d6725780e5e4c373 # v4.1.1 with: subject-name: ${{ env.IMAGE_PREFIX }}/insight-seed subject-digest: ${{ steps.inspect.outputs.digest }}