From be4d786e5da43665a910a6bb9b3f03fd71f1c9ed Mon Sep 17 00:00:00 2001 From: Anton Zelenov Date: Tue, 28 Jul 2026 00:16:40 +0800 Subject: [PATCH] =?UTF-8?q?chore(bootstrap-db):=20address=20review=20?= =?UTF-8?q?=E2=80=94=20exact=20venv=20pins,=20consistent=20docs?= MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit - run-dbt.sh: exact-equality version check on both dbt-core and dbt-clickhouse (substring match accepted e.g. 1.9.6.post1; a DBT_CORE_VERSION-only bump reused a stale venv). - README: local ClickHouse example now uses the LAN IP like the prerequisites; snapshot fallback paths relative to bootstrap-db/. - Reminder workflow: same path fix. Co-Authored-By: Claude Fable 5 Signed-off-by: Anton Zelenov --- .github/workflows/connectors-ddl-reminder.yml | 4 ++-- src/ingestion/scripts/bootstrap-db/README.md | 4 ++-- src/ingestion/scripts/bootstrap-db/run-dbt.sh | 6 ++++-- 3 files changed, 8 insertions(+), 6 deletions(-) diff --git a/.github/workflows/connectors-ddl-reminder.yml b/.github/workflows/connectors-ddl-reminder.yml index c50c5acbb..56bffc08f 100644 --- a/.github/workflows/connectors-ddl-reminder.yml +++ b/.github/workflows/connectors-ddl-reminder.yml @@ -56,8 +56,8 @@ jobs: ' (`host.docker.internal` does not resolve on the macOS host itself)', '- python3.12 or python3.11 on PATH (pinned dbt venv)', '- HubSpot + Salesforce credentials in `.env` — their `discover` calls the', - ' live APIs; without them, apply `scripts/connectors-ddl/{hubspot,salesforce}.sql`', - ' to seed their bronze, then run the dbt step', + ' live APIs; without them, apply `../connectors-ddl/{hubspot,salesforce}.sql`', + ' (relative to bootstrap-db/) to seed their bronze, then run the dbt step', '', '```bash', 'cd src/ingestion/scripts/bootstrap-db', diff --git a/src/ingestion/scripts/bootstrap-db/README.md b/src/ingestion/scripts/bootstrap-db/README.md index aab247c70..2321ffa9a 100644 --- a/src/ingestion/scripts/bootstrap-db/README.md +++ b/src/ingestion/scripts/bootstrap-db/README.md @@ -9,7 +9,7 @@ How it works: for every connector the source image runs `discover` (schemas are - `docker`, `jq`, `yq` (mikefarah v4) - `python3.12` or `python3.11` on `PATH` — `run-dbt.sh` builds a local `.venv` with the pinned dbt from it (parity with the toolbox image; dbt-core 1.10 does not run on newer pythons). A `python -m venv`-capable interpreter is required (uv-managed pythons lack `ensurepip`; with those, pre-build the venv via `uv venv --seed .venv && .venv/bin/pip install dbt-core== dbt-clickhouse==`). - ClickHouse reachable under `CLICKHOUSE_HOST` both from this machine (dbt) and from inside docker containers (destination connector). For a ClickHouse running on this machine use the machine's LAN IP (`ipconfig getifaddr en0`) — `host.docker.internal` resolves inside containers but not on the macOS host itself. -- Real HubSpot and Salesforce credentials in `.env` — their CDK `discover` calls the live APIs, so fake values fail. Without credentials, seed their bronze from the committed snapshot instead: apply `scripts/connectors-ddl/{hubspot,salesforce}.sql`, run `./run-dbt.sh --select hubspot__bronze_promoted salesforce__bronze_promoted`, and continue from the dbt step. +- Real HubSpot and Salesforce credentials in `.env` — their CDK `discover` calls the live APIs, so fake values fail. Without credentials, seed their bronze from the committed snapshot instead: apply `../connectors-ddl/{hubspot,salesforce}.sql` (paths relative to this directory), run `./run-dbt.sh --select hubspot__bronze_promoted salesforce__bronze_promoted`, and continue from the dbt step. ## Local ClickHouse for testing @@ -22,7 +22,7 @@ docker run -d --name bootstrap-db-clickhouse -p 8123:8123 \ "${CLICKHOUSE_SERVER_IMAGE}" ``` -Point `.env` at it: `CLICKHOUSE_HOST=host.docker.internal`, `CLICKHOUSE_PORT=8123`, `CLICKHOUSE_PROTOCOL=http`, user/password/database `insight` — the host name works both for dbt on this machine and for the connector containers. Check what got created: +Point `.env` at it: `CLICKHOUSE_HOST=$(ipconfig getifaddr en0)` (the LAN IP — reachable both for dbt on this machine and for the connector containers; see Prerequisites), `CLICKHOUSE_PORT=8123`, `CLICKHOUSE_PROTOCOL=http`, user/password/database `insight`. Check what got created: ```bash curl -s "http://localhost:8123/" -H "X-ClickHouse-User: insight" -H "X-ClickHouse-Key: insight" \ diff --git a/src/ingestion/scripts/bootstrap-db/run-dbt.sh b/src/ingestion/scripts/bootstrap-db/run-dbt.sh index 3e6e69f84..d888c830f 100755 --- a/src/ingestion/scripts/bootstrap-db/run-dbt.sh +++ b/src/ingestion/scripts/bootstrap-db/run-dbt.sh @@ -18,8 +18,10 @@ VENV_DIR="${SCRIPT_DIR}/.venv" DBT_BIN="${VENV_DIR}/bin/dbt" # No `pip show | grep -q`: grep -q exits at first match, pip dies on EPIPE and # pipefail fails the check for a perfectly good venv (which then gets rm -rf'd). -INSTALLED="$("${VENV_DIR}/bin/pip" show dbt-clickhouse 2>/dev/null || true)" -if [[ ! -x "${DBT_BIN}" ]] || [[ "${INSTALLED}" != *"Version: ${DBT_CLICKHOUSE_VERSION}"* ]]; then +venv_pin_ok() { + [[ "$("${VENV_DIR}/bin/pip" show "$1" 2>/dev/null | awk '/^Version:/{print $2}')" == "$2" ]] +} +if [[ ! -x "${DBT_BIN}" ]] || ! venv_pin_ok dbt-core "${DBT_CORE_VERSION}" || ! venv_pin_ok dbt-clickhouse "${DBT_CLICKHOUSE_VERSION}"; then PYTHON_BIN="$(command -v python3.12 || command -v python3.11)" : "${PYTHON_BIN:?python3.12 or python3.11 is required to run dbt (same major as the toolbox image)}" rm -rf "${VENV_DIR}"