Skip to content
Merged
Show file tree
Hide file tree
Changes from 4 commits
Commits
Show all changes
20 commits
Select commit Hold shift + click to select a range
File filter

Filter by extension

Filter by extension


Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
6 changes: 6 additions & 0 deletions .github/workflows/deploy-web-environment.yml
Original file line number Diff line number Diff line change
Expand Up @@ -71,6 +71,11 @@ on:
required: false
type: string
default: ""
cloudbeaver_host_rule:
description: "Traefik Host() rule for CloudBeaver"
required: false
type: string
default: ""
pgadmin_host_rule:
description: "Traefik Host() rule for pgAdmin"
required: false
Expand Down Expand Up @@ -162,6 +167,7 @@ jobs:
portainer_host_rule: ${{ inputs.portainer_host_rule }}
netdata_host_rule: ${{ inputs.netdata_host_rule }}
databasus_host_rule: ${{ inputs.databasus_host_rule }}
cloudbeaver_host_rule: ${{ inputs.cloudbeaver_host_rule }}
pgadmin_host_rule: ${{ inputs.pgadmin_host_rule }}
dispatching_run_id: ${{ github.run_id }}
refresh_materialized_views: ${{ inputs.refresh_materialized_views }}
Expand Down
10 changes: 9 additions & 1 deletion .github/workflows/deploy-web-stack.yml
Original file line number Diff line number Diff line change
Expand Up @@ -60,6 +60,11 @@ on:
required: false
type: string
default: ""
cloudbeaver_host_rule:
description: "Traefik Host() rule for CloudBeaver"
required: false
type: string
default: ""
pgadmin_host_rule:
description: "Traefik Host() rule for pgAdmin"
required: false
Expand Down Expand Up @@ -160,6 +165,7 @@ jobs:
PORTAINER_HOST_RULE_INPUT: ${{ inputs.portainer_host_rule }}
NETDATA_HOST_RULE_INPUT: ${{ inputs.netdata_host_rule }}
DATABASUS_HOST_RULE_INPUT: ${{ inputs.databasus_host_rule }}
CLOUDBEAVER_HOST_RULE_INPUT: ${{ inputs.cloudbeaver_host_rule }}
PGADMIN_HOST_RULE_INPUT: ${{ inputs.pgadmin_host_rule }}
run: |
{
Expand Down Expand Up @@ -188,6 +194,7 @@ jobs:
write_multiline_env PORTAINER_HOST_RULE "$PORTAINER_HOST_RULE_INPUT"
write_multiline_env NETDATA_HOST_RULE "$NETDATA_HOST_RULE_INPUT"
write_multiline_env DATABASUS_HOST_RULE "$DATABASUS_HOST_RULE_INPUT"
write_multiline_env CLOUDBEAVER_HOST_RULE "$CLOUDBEAVER_HOST_RULE_INPUT"
write_multiline_env PGADMIN_HOST_RULE "$PGADMIN_HOST_RULE_INPUT"

- name: Create dotenv command runner
Expand Down Expand Up @@ -261,13 +268,14 @@ jobs:
docker pull "temporalio/auto-setup:1.29"
docker pull "temporalio/admin-tools:1.29"
docker pull "minio/minio:latest@sha256:14cea493d9a34af32f524e538b8346cf79f3321eff8e708c1e2960462bd8936e"
docker pull "dbeaver/cloudbeaver:26.0.3@sha256:3daa6269a49198363af3f0fcd8af58def9a32fa38152b9ab37854dc80c16a4a4"

- name: Validate host bind mount paths
run: |
docker run --rm \
--mount type=bind,source=/mnt/dofek-data,target=/mnt/dofek-data,readonly \
postgres:18-alpine sh -euc '
for path in /mnt/dofek-data/postgres /mnt/dofek-data/clickhouse /mnt/dofek-data/databasus /mnt/dofek-data/redis /mnt/dofek-data/peerdb-catalog /mnt/dofek-data/peerdb-minio; do
for path in /mnt/dofek-data/postgres /mnt/dofek-data/clickhouse /mnt/dofek-data/databasus /mnt/dofek-data/cloudbeaver /mnt/dofek-data/redis /mnt/dofek-data/peerdb-catalog /mnt/dofek-data/peerdb-minio; do
if [ ! -d "$path" ]; then
echo "::error::Required host bind mount path is missing: $path"
exit 1
Expand Down
1 change: 1 addition & 0 deletions .github/workflows/deploy-web.yml
Original file line number Diff line number Diff line change
Expand Up @@ -80,6 +80,7 @@ jobs:
portainer_host_rule: 'Host(`staging-portainer.dofek.asherlc.com`)'
netdata_host_rule: 'Host(`staging-netdata.dofek.asherlc.com`)'
databasus_host_rule: 'Host(`staging-databasus.dofek.asherlc.com`)'
cloudbeaver_host_rule: 'Host(`staging-cloudbeaver.dofek.asherlc.com`)'
pgadmin_host_rule: 'Host(`staging-pgadmin.dofek.asherlc.com`)'
refresh_materialized_views: ${{ github.event_name == 'workflow_dispatch' && inputs.refresh_materialized_views || false }}
secrets: inherit
Expand Down
4 changes: 3 additions & 1 deletion deploy/README.md
Original file line number Diff line number Diff line change
Expand Up @@ -15,6 +15,7 @@ Dofek is deployed as a **single-node Docker Swarm** stack on **Hetzner Cloud** (
- **Stable mount alias**: Terraform maintains `/mnt/dofek-data` as a symlink to the attached Hetzner volume mount path (`/mnt/HC_Volume_<id>`).
- **DB data path**: The `db` service bind-mounts Postgres data to `/mnt/dofek-data/postgres`.
- **Databasus state path**: The `databasus` service bind-mounts its internal state to `/mnt/dofek-data/databasus` so backup schedules and storage config survive Docker volume churn.
- **CloudBeaver state path**: The `cloudbeaver` service bind-mounts its workspace to `/mnt/dofek-data/cloudbeaver`, including the Terraform-synced preconfigured Postgres and ClickHouse datasource file.
- **S3 (R2)**: Cloudflare R2 buckets for training data (`dofek-training-data`), OTA updates (`dofek-ota`), Storybook (`dofek-storybook`), and DB backups (`dofek-db-backups`).
- **Networking**:
- **Firewall**: `hcloud_firewall` allows SSH (port 22) from restricted IPs and HTTP/HTTPS (80/443) from everywhere.
Expand Down Expand Up @@ -43,7 +44,7 @@ Dofek is deployed as a **single-node Docker Swarm** stack on **Hetzner Cloud** (
- `cloud-init.yml`: Installs Docker CE, configures Docker log rotation (10m, 3 files), and idempotently runs `docker swarm init`. No deploy helpers, no Infisical CLI.

### Swarm Stack (`stack.yml`)
- Single file defining all services: `web`, `worker`, `training-export-worker`, `traefik`, `db`, `clickhouse`, `redis`, `collector`, `ota`, `databasus`, `pgadmin`, `portainer`, `netdata`.
- Single file defining all services: `web`, `worker`, `training-export-worker`, `traefik`, `db`, `clickhouse`, `redis`, `collector`, `ota`, `databasus`, `cloudbeaver`, `pgadmin`, `portainer`, `netdata`.
- Traefik consumes both the swarm provider and a bind-mounted dynamic-config directory at `/opt/dofek/traefik-dynamic` so review-app workspaces can add exact PR host routes without modifying the stack for every PR.
- Zero-downtime updates for `web` and `worker` are configured via `deploy.update_config` (`order: start-first`, `failure_action: rollback`, healthcheck-gated `monitor` window).
- The `default` overlay network is declared `attachable: true` so CI can run one-shot migration containers on it from a remote Docker context.
Expand Down Expand Up @@ -285,4 +286,5 @@ If management subdomains return `404 page not found`, use:
- **Portainer**: `https://portainer.dofek.asherlc.com` (Protected by Authentik)
- **Netdata**: `https://netdata.dofek.asherlc.com` (Protected by Authentik)
- **Databasus**: `https://databasus.dofek.asherlc.com` (DB management + backups)
- **CloudBeaver**: `https://cloudbeaver.dofek.asherlc.com` (Postgres + ClickHouse UI)
- **pgAdmin**: `https://pgadmin.dofek.asherlc.com` (Postgres UI)
65 changes: 65 additions & 0 deletions deploy/cloudbeaver-data-sources.json
Original file line number Diff line number Diff line change
@@ -0,0 +1,65 @@
{
"folders": {
"Dofek": {}
},
"connections": {
"postgres-jdbc-dofek-production": {
"provider": "postgresql",
"driver": "postgres-jdbc",
"name": "Dofek Postgres",
"description": "Dofek TimescaleDB/Postgres on the internal swarm network",
"save-password": false,
"show-system-objects": false,
"read-only": true,
"folder": "Dofek",
"configuration": {
"user": "health",
"host": "db",
"port": "5432",
"database": "health",
"url": "jdbc:postgresql://db:5432/health",
Comment thread
Asherlc marked this conversation as resolved.
Resolved
"configurationType": "MANUAL",
"type": "prod",
"provider-properties": {
"@dbeaver-show-non-default-db@": "false",
"@dbeaver-show-template-db@": "false",
"@dbeaver-show-unavailable-db@": "false",
"show-database-statistics": "false"
},
"auth-model": "native"
}
},
"clickhouse-com-dofek-production": {
"provider": "clickhouse",
"driver": "com_clickhouse",
"name": "Dofek ClickHouse",
"description": "Dofek ClickHouse analytics store on the internal swarm network",
"save-password": false,
"show-system-objects": false,
"read-only": true,
"folder": "Dofek",
"configuration": {
"user": "default",
"host": "clickhouse",
"port": "8123",
"database": "fitness",
"url": "jdbc:clickhouse://clickhouse:8123/fitness",
"configurationType": "MANUAL",
Comment thread
Asherlc marked this conversation as resolved.
Resolved
"type": "prod",
"provider-properties": {},
"auth-model": "native"
}
}
},
"connection-types": {
"prod": {
"name": "Production",
"color": "255,0,0",
"description": "Production database",
Comment thread
Asherlc marked this conversation as resolved.
Resolved
"auto-commit": false,
"confirm-execute": true,
"confirm-data-change": true,
"auto-close-transactions": false
}
}
}
18 changes: 18 additions & 0 deletions deploy/dns.tf
Original file line number Diff line number Diff line change
Expand Up @@ -123,6 +123,15 @@ resource "cloudflare_dns_record" "databasus_dofek_asherlc" {
ttl = 1
}

resource "cloudflare_dns_record" "cloudbeaver_dofek_asherlc" {
zone_id = data.cloudflare_zone.asherlc_com.zone_id
type = "A"
name = "cloudbeaver.dofek.asherlc.com"
content = hcloud_server.dofek.ipv4_address
proxied = false
ttl = 1
}

resource "cloudflare_dns_record" "pgadmin_dofek_asherlc" {
zone_id = data.cloudflare_zone.asherlc_com.zone_id
type = "A"
Expand Down Expand Up @@ -177,6 +186,15 @@ resource "cloudflare_dns_record" "staging_databasus_dofek_asherlc" {
ttl = 1
}

resource "cloudflare_dns_record" "staging_cloudbeaver_dofek_asherlc" {
zone_id = data.cloudflare_zone.asherlc_com.zone_id
type = "A"
name = "staging-cloudbeaver.dofek.asherlc.com"
content = hcloud_server.dofek_staging.ipv4_address
proxied = false
ttl = 1
}

resource "cloudflare_dns_record" "staging_pgadmin_dofek_asherlc" {
zone_id = data.cloudflare_zone.asherlc_com.zone_id
type = "A"
Expand Down
66 changes: 62 additions & 4 deletions deploy/server.tf
Original file line number Diff line number Diff line change
Expand Up @@ -128,7 +128,7 @@ resource "terraform_data" "data_volume_mount_alias" {
count = var.data_volume_size_gb > 0 ? 1 : 0

triggers_replace = [
"volume-mount-alias-v6",
"volume-mount-alias-v7",
hcloud_volume.dofek_data[0].id,
]

Expand All @@ -145,7 +145,7 @@ resource "terraform_data" "data_volume_mount_alias" {
"target=/mnt/HC_Volume_${hcloud_volume.dofek_data[0].id}",
"if [ ! -d \"$target\" ]; then echo \"Expected mounted volume path missing: $target\" >&2; exit 1; fi",
"ln -sfn \"$target\" /mnt/dofek-data",
"mkdir -p /mnt/dofek-data/postgres /mnt/dofek-data/clickhouse /mnt/dofek-data/databasus /mnt/dofek-data/redis /mnt/dofek-data/peerdb-catalog /mnt/dofek-data/peerdb-minio",
"mkdir -p /mnt/dofek-data/postgres /mnt/dofek-data/clickhouse /mnt/dofek-data/databasus /mnt/dofek-data/cloudbeaver /mnt/dofek-data/redis /mnt/dofek-data/peerdb-catalog /mnt/dofek-data/peerdb-minio",
"source_path=$(docker volume inspect -f '{{ .Mountpoint }}' dofek_databasus_data 2>/dev/null || true); if [ -n \"$source_path\" ] && [ -d \"$source_path\" ] && [ -z \"$(find /mnt/dofek-data/databasus -mindepth 1 -print -quit)\" ] && [ -n \"$(find \"$source_path\" -mindepth 1 -print -quit)\" ]; then cp -a \"$source_path\"/. /mnt/dofek-data/databasus/; fi",
"source_path=$(docker volume inspect -f '{{ .Mountpoint }}' dofek_redis_data 2>/dev/null || true); if [ -n \"$source_path\" ] && [ -d \"$source_path\" ] && [ -z \"$(find /mnt/dofek-data/redis -mindepth 1 -print -quit)\" ] && [ -n \"$(find \"$source_path\" -mindepth 1 -print -quit)\" ]; then cp -a \"$source_path\"/. /mnt/dofek-data/redis/; fi",
]
Expand All @@ -156,7 +156,7 @@ resource "terraform_data" "staging_data_volume_mount_alias" {
count = var.staging_data_volume_size_gb > 0 ? 1 : 0

triggers_replace = [
"volume-mount-alias-v4",
"volume-mount-alias-v5",
hcloud_volume.dofek_staging_data[0].id,
]

Expand All @@ -173,7 +173,7 @@ resource "terraform_data" "staging_data_volume_mount_alias" {
"target=/mnt/HC_Volume_${hcloud_volume.dofek_staging_data[0].id}",
"if [ ! -d \"$target\" ]; then echo \"Expected mounted volume path missing: $target\" >&2; exit 1; fi",
"ln -sfn \"$target\" /mnt/dofek-data",
"mkdir -p /mnt/dofek-data/postgres /mnt/dofek-data/clickhouse /mnt/dofek-data/databasus /mnt/dofek-data/redis /mnt/dofek-data/peerdb-catalog /mnt/dofek-data/peerdb-minio",
"mkdir -p /mnt/dofek-data/postgres /mnt/dofek-data/clickhouse /mnt/dofek-data/databasus /mnt/dofek-data/cloudbeaver /mnt/dofek-data/redis /mnt/dofek-data/peerdb-catalog /mnt/dofek-data/peerdb-minio",
"source_path=$(docker volume inspect -f '{{ .Mountpoint }}' dofek-staging_redis_data 2>/dev/null || true); if [ -n \"$source_path\" ] && [ -d \"$source_path\" ] && [ -z \"$(find /mnt/dofek-data/redis -mindepth 1 -print -quit)\" ] && [ -n \"$(find \"$source_path\" -mindepth 1 -print -quit)\" ]; then cp -a \"$source_path\"/. /mnt/dofek-data/redis/; fi",
]
}
Expand Down Expand Up @@ -230,6 +230,64 @@ resource "terraform_data" "staging_otel_config_sync" {
}
}

# Sync CloudBeaver preconfigured database connections to the persistent
# workspace used by the CloudBeaver service.
resource "terraform_data" "cloudbeaver_datasources_sync" {
count = var.data_volume_size_gb > 0 ? 1 : 0

triggers_replace = [
filesha256("${path.module}/cloudbeaver-data-sources.json"),
]
Comment thread
coderabbitai[bot] marked this conversation as resolved.
Resolved

connection {
type = "ssh"
host = hcloud_server.dofek.ipv4_address
user = "root"
private_key = var.ssh_private_key
}

provisioner "remote-exec" {
inline = [
"mkdir -p /mnt/dofek-data/cloudbeaver/GlobalConfiguration/.dbeaver",
]
}

provisioner "file" {
source = "${path.module}/cloudbeaver-data-sources.json"
destination = "/mnt/dofek-data/cloudbeaver/GlobalConfiguration/.dbeaver/data-sources.json"
}

depends_on = [terraform_data.data_volume_mount_alias]
}

resource "terraform_data" "staging_cloudbeaver_datasources_sync" {
count = var.staging_data_volume_size_gb > 0 ? 1 : 0

triggers_replace = [
filesha256("${path.module}/cloudbeaver-data-sources.json"),
]

connection {
type = "ssh"
host = hcloud_server.dofek_staging.ipv4_address
user = "root"
private_key = var.ssh_private_key
}

provisioner "remote-exec" {
inline = [
"mkdir -p /mnt/dofek-data/cloudbeaver/GlobalConfiguration/.dbeaver",
]
}

provisioner "file" {
source = "${path.module}/cloudbeaver-data-sources.json"
destination = "/mnt/dofek-data/cloudbeaver/GlobalConfiguration/.dbeaver/data-sources.json"
}

depends_on = [terraform_data.staging_data_volume_mount_alias]
}

# Apply Redis kernel configuration to existing servers
resource "terraform_data" "redis_kernel_config" {
triggers_replace = [
Expand Down
17 changes: 17 additions & 0 deletions deploy/stack.yml
Original file line number Diff line number Diff line change
Expand Up @@ -304,6 +304,23 @@ services:
- traefik.http.routers.databasus.middlewares=databasus-auth
- traefik.http.services.databasus.loadbalancer.server.port=4005

cloudbeaver:
image: dbeaver/cloudbeaver:26.0.3@sha256:3daa6269a49198363af3f0fcd8af58def9a32fa38152b9ab37854dc80c16a4a4
volumes:
- /mnt/dofek-data/cloudbeaver:/opt/cloudbeaver/workspace
deploy:
labels:
- traefik.enable=true
- traefik.http.middlewares.cloudbeaver-auth.forwardAuth.address=https://authentik.asherlc.com/outpost.goauthentik.io/auth/traefik
- traefik.http.middlewares.cloudbeaver-auth.forwardAuth.trustForwardHeader=true
- traefik.http.middlewares.cloudbeaver-auth.forwardAuth.authResponseHeaders=X-authentik-username,X-authentik-groups,X-authentik-email,X-authentik-name,X-authentik-uid
- traefik.http.routers.cloudbeaver.rule=${CLOUDBEAVER_HOST_RULE:-Host(`cloudbeaver.dofek.asherlc.com`)}
- traefik.http.routers.cloudbeaver.entrypoints=websecure
- traefik.http.routers.cloudbeaver.tls=true
- traefik.http.routers.cloudbeaver.tls.certresolver=le
- traefik.http.routers.cloudbeaver.middlewares=cloudbeaver-auth
- traefik.http.services.cloudbeaver.loadbalancer.server.port=8978

pgadmin:
image: dpage/pgadmin4:9.14.0
environment:
Expand Down
56 changes: 32 additions & 24 deletions docs/production-incident-baseline.md
Original file line number Diff line number Diff line change
Expand Up @@ -2117,47 +2117,55 @@ compare row counts and recent-row freshness between Postgres,
`postgres_fitness.metric_stream`, and `peerdb.metric_stream`, then cut
`analytics.deduped_sensor` over in a separate migration.

## 2026-05-01: Production Deploy Migration Timed Out During Metric Stream Backfill
## 2026-05-01: Production Deploy Blocked by ClickHouse Backfill Timeout

### Symptoms

Production deploy run `25237109231` failed in job `74005587158` during the
`Run migrations` step. The migration container exited with code 1 before
`docker stack deploy` ran, so the new web stack image was not released.
Production deploy workflow `25239905048` failed in the `Run migrations` step
before the stack deploy step, so the CloudBeaver service was not released.

### User Impact

The existing production web stack kept running on the previous image, but the
new CloudBeaver production service was unavailable.

### Evidence

The first fatal job line was `Migration failed (exit code 1).` The last
migration log line was:
The first fatal workflow log line was:

```text
error: [migrate] Error: Timeout error.
```

Immediately before the error, Postgres migrations had applied 0 new files and
materialized view sync had completed with `synced=0 skipped=7 refreshed=0`.
The next code path is `runClickHouseMigrations()`.
The migration log showed Postgres migrations and materialized view sync
completed, then ClickHouse migration `0006_backfill_native_metric_stream`
failed before recording its row in `analytics.schema_migrations`. Production
ClickHouse had only `42` completed rows in
`analytics.metric_stream_backfill_chunks`, while production Timescale had `195`
`fitness.metric_stream` chunks. The next uncompleted Timescale chunk,
`2021-04-29 00:00:00+00` to `2021-05-06 00:00:00+00`, contained about
`4,014,020` rows.

### Root Cause

ClickHouse migration `0006_backfill_native_metric_stream` copied each full
Timescale chunk with one `INSERT INTO ... SELECT FROM postgresql(...)` command.
Production chunks were large enough for a ClickHouse HTTP command to exceed the
client's default 30 second request timeout. Because the migration also dropped
`postgres_fitness` and the backfill progress table on every retry, a retry
would restart the native backfill instead of continuing from completed ranges.
ClickHouse migration `0006_backfill_native_metric_stream` backfilled one full
Timescale chunk per ClickHouse insert. Some production weekly chunks are too
large for the ClickHouse HTTP client request timeout, so the migration timed out
while copying historical `metric_stream` rows.

### Fix or Mitigation

The migration now splits Timescale chunk ranges into one-hour ClickHouse
backfill windows and records those bounded windows in
`analytics.metric_stream_backfill_chunks`. It only drops `postgres_fitness` and
the progress table when `postgres_fitness` is still backed by a non-native
database engine; retries against an already-native database preserve completed
backfill windows and continue from the first missing window.
The migration now splits Timescale chunk ranges into six-hour backfill ranges
before checking and recording `analytics.metric_stream_backfill_chunks`. This
keeps each ClickHouse insert bounded while preserving resumability through the
existing range-completion table. The rerun path also treats older broader
completion ranges as covering new subranges and uses an anti-join on existing
ClickHouse `metric_stream` IDs so a timed-out-but-successful insert is not
duplicated.

### Remaining Risk

Very dense one-hour windows can still take longer than expected, but future
failures will now identify the exact window being copied and retries will not
discard completed native backfill progress.
The production backfill still needs to finish before migration `0006` is marked
applied. The deploy should be rerun after the fix reaches the image used by the
workflow, then verified by confirming `0006_backfill_native_metric_stream`
exists in `analytics.schema_migrations` and CloudBeaver is running.
Loading
Loading