Skip to content

chore(bootstrap-db): address #1940 review — exact venv pins, consistent docs #1954

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Merged
cyberantonz merged 2 commits into from
Jul 28, 2026
+8 −6
Merged

chore(bootstrap-db): address #1940 review — exact venv pins, consistent docs #1954

Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
2 commits
Select commit Hold shift + click to select a range
be4d786
chore(bootstrap-db): address review — exact venv pins, consistent docs
cyberantonz Jul 27, 2026
413e32d
Merge branch 'main' into chore/bootstrap-db-review-fixes
cyberantonz Jul 28, 2026
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
4 changes: 2 additions & 2 deletions .github/workflows/connectors-ddl-reminder.yml
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -56,8 +56,8 @@ jobs:
' (`host.docker.internal` does not resolve on the macOS host itself)',
'- python3.12 or python3.11 on PATH (pinned dbt venv)',
'- HubSpot + Salesforce credentials in `.env` — their `discover` calls the',
' live APIs; without them, apply `scripts/connectors-ddl/{hubspot,salesforce}.sql`',
' to seed their bronze, then run the dbt step',
' live APIs; without them, apply `../connectors-ddl/{hubspot,salesforce}.sql`',
' (relative to bootstrap-db/) to seed their bronze, then run the dbt step',
'',
'```bash',
'cd src/ingestion/scripts/bootstrap-db',
Expand Down
4 changes: 2 additions & 2 deletions src/ingestion/scripts/bootstrap-db/README.md
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -9,7 +9,7 @@ How it works: for every connector the source image runs `discover` (schemas are
- `docker`, `jq`, `yq` (mikefarah v4)
- `python3.12` or `python3.11` on `PATH` — `run-dbt.sh` builds a local `.venv` with the pinned dbt from it (parity with the toolbox image; dbt-core 1.10 does not run on newer pythons). A `python -m venv`-capable interpreter is required (uv-managed pythons lack `ensurepip`; with those, pre-build the venv via `uv venv --seed .venv && .venv/bin/pip install dbt-core==<pin> dbt-clickhouse==<pin>`).
- ClickHouse reachable under `CLICKHOUSE_HOST` both from this machine (dbt) and from inside docker containers (destination connector). For a ClickHouse running on this machine use the machine's LAN IP (`ipconfig getifaddr en0`) — `host.docker.internal` resolves inside containers but not on the macOS host itself.
- Real HubSpot and Salesforce credentials in `.env` — their CDK `discover` calls the live APIs, so fake values fail. Without credentials, seed their bronze from the committed snapshot instead: apply `scripts/connectors-ddl/{hubspot,salesforce}.sql`, run `./run-dbt.sh --select hubspot__bronze_promoted salesforce__bronze_promoted`, and continue from the dbt step.
- Real HubSpot and Salesforce credentials in `.env` — their CDK `discover` calls the live APIs, so fake values fail. Without credentials, seed their bronze from the committed snapshot instead: apply `../connectors-ddl/{hubspot,salesforce}.sql` (paths relative to this directory), run `./run-dbt.sh --select hubspot__bronze_promoted salesforce__bronze_promoted`, and continue from the dbt step.

## Local ClickHouse for testing

Expand All @@ -22,7 +22,7 @@ docker run -d --name bootstrap-db-clickhouse -p 8123:8123 \
"${CLICKHOUSE_SERVER_IMAGE}"
```

Point `.env` at it: `CLICKHOUSE_HOST=host.docker.internal`, `CLICKHOUSE_PORT=8123`, `CLICKHOUSE_PROTOCOL=http`, user/password/database `insight` — the host name works both for dbt on this machine and for the connector containers. Check what got created:
Point `.env` at it: `CLICKHOUSE_HOST=$(ipconfig getifaddr en0)` (the LAN IP — reachable both for dbt on this machine and for the connector containers; see Prerequisites), `CLICKHOUSE_PORT=8123`, `CLICKHOUSE_PROTOCOL=http`, user/password/database `insight`. Check what got created:

```bash
curl -s "http://localhost:8123/" -H "X-ClickHouse-User: insight" -H "X-ClickHouse-Key: insight" \
Expand Down
6 changes: 4 additions & 2 deletions src/ingestion/scripts/bootstrap-db/run-dbt.sh
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -18,8 +18,10 @@ VENV_DIR="${SCRIPT_DIR}/.venv"
DBT_BIN="${VENV_DIR}/bin/dbt"
# No `pip show | grep -q`: grep -q exits at first match, pip dies on EPIPE and
# pipefail fails the check for a perfectly good venv (which then gets rm -rf'd).
INSTALLED="$("${VENV_DIR}/bin/pip" show dbt-clickhouse 2>/dev/null || true)"
if [[ ! -x "${DBT_BIN}" ]] || [[ "${INSTALLED}" != *"Version: ${DBT_CLICKHOUSE_VERSION}"* ]]; then
venv_pin_ok() {
[[ "$("${VENV_DIR}/bin/pip" show "$1" 2>/dev/null | awk '/^Version:/{print $2}')" == "$2" ]]
}
if [[ ! -x "${DBT_BIN}" ]] || ! venv_pin_ok dbt-core "${DBT_CORE_VERSION}" || ! venv_pin_ok dbt-clickhouse "${DBT_CLICKHOUSE_VERSION}"; then
PYTHON_BIN="$(command -v python3.12 || command -v python3.11)"
: "${PYTHON_BIN:?python3.12 or python3.11 is required to run dbt (same major as the toolbox image)}"
rm -rf "${VENV_DIR}"
Expand Down
Loading