Skip to content
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension


Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
8 changes: 4 additions & 4 deletions .claude/skills/metric-test/SKILL.md
Original file line number Diff line number Diff line change
Expand Up @@ -9,7 +9,7 @@ allowed-tools: Bash, Read, Write, Edit, Glob, Grep
# Author a metric test (declarative YAML)

This skill writes and validates `*.test.yaml` fixtures that drive the full
`bronze → dbt silver → gold view → analytics-api` path and assert the result.
`bronze → dbt silver → gold view → analytics` path and assert the result.

## Source of truth (reference — open only if you need the detail)

Expand Down Expand Up @@ -193,7 +193,7 @@ with a dedicated spec (see `metrics/collab_emails_read.test.yaml`):
## Scaffolding a new test

1. **Resolve the metric_id and its shape.** Find it in the seed catalog
(`grep -rn "<label>" src/backend/services/analytics-api/src/migration/*.rs`) and
(`grep -rn "<label>" src/backend/services/analytics/src/migration/*.rs`) and
the live `query_ref` rewrite for that metric. Note whether it returns a bullet
(`metric_key`/`value`/`median`/`range_*`) or per-person rows. For the collaboration
bullets the median/range is **DEPARTMENT/org_unit-scoped for BOTH** the Team
Expand Down Expand Up @@ -316,12 +316,12 @@ connector that isn't there yet:
## Gotchas (rig operations + cross-test impact)

- **Stale binary / your migration didn't run.** Historically the biggest trap:
`./e2e.sh` builds analytics-api into the `cargo-target` Docker volume, and on
`./e2e.sh` builds analytics into the `cargo-target` Docker volume, and on
Docker Desktop (macOS) the mtimes cargo reads through the bind mount don't
reliably advance, so cargo relinked a stale object and the binary silently
lacked new SeaORM migrations (symptoms: `query_ref`/catalog changes have no
effect, a `find` matches 0 rows, `size(items)` off by your new key). FIXED in
`lib/analytics_api.py::build` — it now `touch`es the analytics-api crate
`lib/analytics.py::build` — it now `touch`es the analytics crate
sources before `cargo build`, forcing a recompile every run (~1-2 min, only
that crate). So a plain `./e2e.sh test` picks up new migrations now; you should
NOT need `down -v` for this. If you still suspect a stale binary, confirm by
Expand Down
6 changes: 3 additions & 3 deletions .env.compose.example
Original file line number Diff line number Diff line change
Expand Up @@ -49,13 +49,13 @@ FRONTEND_IMAGE=
#
# Equivalent CLI: `./dev-compose.sh up --from-ghcr=api-gateway,identity`
API_GATEWAY_IMAGE=
ANALYTICS_API_IMAGE=
ANALYTICS_IMAGE=
IDENTITY_IMAGE=

# ── Host port mapping ─────────────────────────────────────────────────
# Change these if 8080/8081/8082/3000/3306/8123/6379 are already in use.
API_GATEWAY_PORT=8080
ANALYTICS_API_PORT=8081
ANALYTICS_PORT=8081
IDENTITY_PORT=8082
FRONTEND_PORT=3000

Expand Down Expand Up @@ -110,7 +110,7 @@ SEEDED_LOCAL_MARIA=
SEEDED_LOCAL_CH=

# ── Tenant / OIDC ─────────────────────────────────────────────────────
# Default tenant for header-less callers (analytics-api + identity).
# Default tenant for header-less callers (analytics + identity).
# Replace with the UUID present in your persons.insight_tenant_id.
TENANT_DEFAULT_ID=00000000-df51-5b42-9538-d2b56b7ee953

Expand Down
60 changes: 30 additions & 30 deletions .github/workflows/build-images.yml
Original file line number Diff line number Diff line change
Expand Up @@ -57,7 +57,7 @@ jobs:
runs-on: ubuntu-latest
outputs:
api_gateway: ${{ steps.filter.outputs.api_gateway }}
analytics_api: ${{ steps.filter.outputs.analytics_api }}
analytics: ${{ steps.filter.outputs.analytics }}
identity: ${{ steps.filter.outputs.identity }}
toolbox: ${{ steps.filter.outputs.toolbox }}
umbrella: ${{ steps.filter.outputs.umbrella }}
Expand All @@ -72,8 +72,8 @@ jobs:
- 'src/backend/services/api-gateway/**'
- 'src/backend/Cargo.toml'
- 'src/backend/Cargo.lock'
analytics_api:
- 'src/backend/services/analytics-api/**'
analytics:
- 'src/backend/services/analytics/**'
- 'src/backend/Cargo.toml'
- 'src/backend/Cargo.lock'
identity:
Expand Down Expand Up @@ -296,10 +296,10 @@ jobs:
subject-digest: ${{ steps.inspect.outputs.digest }}
push-to-registry: true

backend-analytics-api:
backend-analytics:
needs: changes
if: |
needs.changes.outputs.analytics_api == 'true'
needs.changes.outputs.analytics == 'true'
|| (github.event_name == 'workflow_dispatch' && inputs.frontend_tag == '')
strategy:
fail-fast: false
Expand All @@ -325,11 +325,11 @@ jobs:
id: build
with:
context: src/backend
file: src/backend/services/analytics-api/Dockerfile
file: src/backend/services/analytics/Dockerfile
platforms: ${{ matrix.platform }}
outputs: type=image,name=${{ env.IMAGE_PREFIX }}/insight-analytics-api,push-by-digest=true,name-canonical=true,push=${{ github.event_name != 'pull_request' && github.ref == 'refs/heads/main' }}
cache-from: type=gha,scope=analytics-api-${{ matrix.arch }}
cache-to: type=gha,mode=max,scope=analytics-api-${{ matrix.arch }},ignore-error=true
outputs: type=image,name=${{ env.IMAGE_PREFIX }}/insight-analytics,push-by-digest=true,name-canonical=true,push=${{ github.event_name != 'pull_request' && github.ref == 'refs/heads/main' }}
cache-from: type=gha,scope=analytics-${{ matrix.arch }}
cache-to: type=gha,mode=max,scope=analytics-${{ matrix.arch }},ignore-error=true
- name: Export digest
if: github.event_name != 'pull_request' && github.ref == 'refs/heads/main'
run: |
Expand All @@ -340,16 +340,16 @@ jobs:
if: github.event_name != 'pull_request' && github.ref == 'refs/heads/main'
uses: actions/upload-artifact@v4
with:
name: digests-analytics-api-${{ matrix.arch }}
name: digests-analytics-${{ matrix.arch }}
path: /tmp/digests/*
if-no-files-found: error
retention-days: 1

merge-analytics-api:
needs: [changes, backend-analytics-api]
merge-analytics:
needs: [changes, backend-analytics]
if: |
always()
&& needs.backend-analytics-api.result == 'success'
&& needs.backend-analytics.result == 'success'
&& github.event_name != 'pull_request'
&& github.ref == 'refs/heads/main'
runs-on: ubuntu-latest
Expand All @@ -358,7 +358,7 @@ jobs:
uses: actions/download-artifact@v4
with:
path: /tmp/digests
pattern: digests-analytics-api-*
pattern: digests-analytics-*
merge-multiple: true
- uses: docker/setup-buildx-action@v3
- uses: docker/login-action@v3
Expand All @@ -369,7 +369,7 @@ jobs:
- uses: docker/metadata-action@v5
id: meta
with:
images: ${{ env.IMAGE_PREFIX }}/insight-analytics-api
images: ${{ env.IMAGE_PREFIX }}/insight-analytics
tags: |
type=raw,value=${{ needs.changes.outputs.build_tag }}
type=raw,value=latest
Expand All @@ -378,18 +378,18 @@ jobs:
run: |
docker buildx imagetools create \
$(jq -cr '.tags | map("-t " + .) | join(" ")' <<< "$DOCKER_METADATA_OUTPUT_JSON") \
$(printf '${{ env.IMAGE_PREFIX }}/insight-analytics-api@sha256:%s ' *)
$(printf '${{ env.IMAGE_PREFIX }}/insight-analytics@sha256:%s ' *)
- name: Inspect manifest digest
id: inspect
run: |
DIGEST=$(docker buildx imagetools inspect \
${{ env.IMAGE_PREFIX }}/insight-analytics-api:${{ needs.changes.outputs.build_tag }} \
${{ env.IMAGE_PREFIX }}/insight-analytics:${{ needs.changes.outputs.build_tag }} \
--format '{{json .Manifest}}' | jq -r .digest)
echo "digest=$DIGEST" >> "$GITHUB_OUTPUT"
- name: Attest build provenance
uses: actions/attest-build-provenance@v2
with:
subject-name: ${{ env.IMAGE_PREFIX }}/insight-analytics-api
subject-name: ${{ env.IMAGE_PREFIX }}/insight-analytics
subject-digest: ${{ steps.inspect.outputs.digest }}
push-to-registry: true

Expand Down Expand Up @@ -418,7 +418,7 @@ jobs:
registry: ${{ env.REGISTRY }}
username: ${{ github.actor }}
password: ${{ secrets.GITHUB_TOKEN }}
# Context is src/backend/ (uniform with api-gateway + analytics-api)
# Context is src/backend/ (uniform with api-gateway + analytics)
# so the Dockerfile can reach src/backend/docker-entrypoint.sh —
# the shared ENABLE_AUTO_RELOAD-aware entrypoint added in the
# docker-compose dev path. Internal COPY paths inside the
Expand Down Expand Up @@ -792,7 +792,7 @@ jobs:
- changes
- discover-images
- merge-api-gateway
- merge-analytics-api
- merge-analytics
- merge-identity
- merge-toolbox
- merge-image
Expand All @@ -812,7 +812,7 @@ jobs:
&& needs.discover-images.outputs.any == 'true'
&& needs.merge-image.result == 'success'
&& (needs.merge-api-gateway.result == 'success' || needs.merge-api-gateway.result == 'skipped')
&& (needs.merge-analytics-api.result == 'success' || needs.merge-analytics-api.result == 'skipped')
&& (needs.merge-analytics.result == 'success' || needs.merge-analytics.result == 'skipped')
&& (needs.merge-identity.result == 'success' || needs.merge-identity.result == 'skipped')
&& (needs.merge-toolbox.result == 'success' || needs.merge-toolbox.result == 'skipped')
runs-on: ubuntu-latest
Expand Down Expand Up @@ -947,7 +947,7 @@ jobs:
- changes
- discover-images
- merge-api-gateway
- merge-analytics-api
- merge-analytics
- merge-identity
- merge-toolbox
- merge-image
Expand All @@ -957,15 +957,15 @@ jobs:
&& (github.event_name == 'push' || github.event_name == 'workflow_dispatch')
&& github.ref == 'refs/heads/main'
&& (needs.merge-api-gateway.result == 'success' || needs.merge-api-gateway.result == 'skipped')
&& (needs.merge-analytics-api.result == 'success' || needs.merge-analytics-api.result == 'skipped')
&& (needs.merge-analytics.result == 'success' || needs.merge-analytics.result == 'skipped')
&& (needs.merge-identity.result == 'success' || needs.merge-identity.result == 'skipped')
&& (needs.merge-toolbox.result == 'success' || needs.merge-toolbox.result == 'skipped')
&& (needs.merge-image.result == 'success' || needs.merge-image.result == 'skipped')
&& (needs.bump-descriptors.result == 'success' || needs.bump-descriptors.result == 'skipped')
&& needs.bump-descriptors.outputs.committed != 'true'
&& (
needs.changes.outputs.api_gateway == 'true'
|| needs.changes.outputs.analytics_api == 'true'
|| needs.changes.outputs.analytics == 'true'
|| needs.changes.outputs.identity == 'true'
|| needs.changes.outputs.toolbox == 'true'
|| needs.changes.outputs.umbrella == 'true'
Expand Down Expand Up @@ -1033,17 +1033,17 @@ jobs:
env:
BUILD_TAG: ${{ needs.changes.outputs.build_tag }}
API_GATEWAY: ${{ needs.changes.outputs.api_gateway }}
ANALYTICS_API: ${{ needs.changes.outputs.analytics_api }}
ANALYTICS: ${{ needs.changes.outputs.analytics }}
IDENTITY: ${{ needs.changes.outputs.identity }}
run: |
set -euo pipefail
if [ "$API_GATEWAY" = "true" ]; then
echo "Bumping api-gateway subchart appVersion → $BUILD_TAG"
yq -i ".appVersion = \"$BUILD_TAG\"" src/backend/services/api-gateway/helm/Chart.yaml
fi
if [ "$ANALYTICS_API" = "true" ]; then
echo "Bumping analytics-api subchart appVersion → $BUILD_TAG"
yq -i ".appVersion = \"$BUILD_TAG\"" src/backend/services/analytics-api/helm/Chart.yaml
if [ "$ANALYTICS" = "true" ]; then
echo "Bumping analytics subchart appVersion → $BUILD_TAG"
yq -i ".appVersion = \"$BUILD_TAG\"" src/backend/services/analytics/helm/Chart.yaml
fi
if [ "$IDENTITY" = "true" ]; then
echo "Bumping identity subchart appVersion → $BUILD_TAG"
Expand Down Expand Up @@ -1104,7 +1104,7 @@ jobs:
# breaks every install (charts 0.1.48–0.1.55 were affected).
NEW_APP=$(yq -r '.appVersion' \
src/backend/services/api-gateway/helm/Chart.yaml \
src/backend/services/analytics-api/helm/Chart.yaml \
src/backend/services/analytics/helm/Chart.yaml \
src/backend/services/identity/helm/Chart.yaml \
src/frontend/helm/Chart.yaml \
| grep -Ev '^(---|null)$' \
Expand Down Expand Up @@ -1190,7 +1190,7 @@ jobs:
charts/insight/Chart.lock \
charts/insight/values.yaml \
src/backend/services/api-gateway/helm/Chart.yaml \
src/backend/services/analytics-api/helm/Chart.yaml \
src/backend/services/analytics/helm/Chart.yaml \
src/backend/services/identity/helm/Chart.yaml \
src/frontend/helm/Chart.yaml \
deploy/gitops/.insight-version
Expand Down
22 changes: 18 additions & 4 deletions .github/workflows/ci.yml
Original file line number Diff line number Diff line change
Expand Up @@ -134,7 +134,7 @@ jobs:
with:
tool: cargo-llvm-cov

# DB-backed components (analytics-api) run their #[ignore]d live_tests
# DB-backed components (analytics) run their #[ignore]d live_tests
# against a real MariaDB. Gated on live_db so only those entries pay the
# ~15s startup. ClickHouse is intentionally not provisioned — CH-gated
# tests skip themselves (see cf/insight#1564).
Expand Down Expand Up @@ -170,9 +170,10 @@ jobs:
feats=""
if [ "${{ matrix.entry.all_features }}" = "true" ]; then feats="--all-features"; fi
if [ "${{ matrix.entry.live_db }}" = "true" ]; then
# The gear reads its DB URL from APP__gears__<alias>__config__database_url
# (underscore alias bridged to the kebab gear key by fold_gear_env_alias).
APP__gears__analytics_api__config__database_url="$INTEGRATION_TESTS_MARIADB_URL" \
# The gear reads its DB URL from APP__gears__analytics__config__database_url
# (the gear config key is "analytics" — dash-free, so the env var
# survives sh/dash entrypoints and k8s envFrom).
APP__gears__analytics__config__database_url="$INTEGRATION_TESTS_MARIADB_URL" \
cargo llvm-cov run --no-report --package "${{ matrix.entry.package }}" $feats -- \
-c "services/${{ matrix.entry.name }}/config/insight.yaml" migrate
cargo llvm-cov --no-report --package "${{ matrix.entry.package }}" $feats -- --include-ignored
Expand All @@ -189,6 +190,19 @@ jobs:
cargo llvm-cov report --cobertura $ignore \
--output-path "$GITHUB_WORKSPACE/coverage/rust-${{ matrix.entry.name }}.cobertura.xml"

# Changed crates with cover=false (api-gateway — no unit tests yet) still
# run plain `cargo test`: a failure fails this job; only the Cobertura
# collection and the coverage gates are skipped. Lint-only fanout entries
# (test=false) skip this too — coverage/tests stay per-changed-crate.
- name: Test (${{ matrix.entry.name }}, no coverage)
if: ${{ matrix.entry.test && !matrix.entry.cover }}
working-directory: ${{ matrix.entry.root }}
run: |
set -euo pipefail
feats=""
if [ "${{ matrix.entry.all_features }}" = "true" ]; then feats="--all-features"; fi
cargo test -p ${{ matrix.entry.package }} $feats

- name: Upload Cobertura
if: ${{ matrix.entry.cover }}
uses: actions/upload-artifact@v7
Expand Down
14 changes: 7 additions & 7 deletions .github/workflows/e2e-bronze-to-api.yml
Original file line number Diff line number Diff line change
@@ -1,14 +1,14 @@
name: E2E — Bronze to API

# End-to-end transformation tests covering bronze → dbt → ClickHouse
# migration views → analytics-api HTTP response → expect rules.
# migration views → analytics HTTP response → expect rules.
#
# Everything (Python, Rust, dbt, pytest) runs inside the runner image
# defined at src/ingestion/tests/e2e/compose/Dockerfile.runner. The same
# image is used for local development via src/ingestion/tests/e2e/e2e.sh.
#
# Topology: ONE heavy job (`e2e`) boots the stack, runs the suite, and — while
# analytics-api is already up — snapshots the metric catalog into `.artifacts/`,
# analytics is already up — snapshots the metric catalog into `.artifacts/`,
# which it uploads. The metric-coverage-gate job then analyses it as a pure-Python
# check (no Docker, no app boot). See
# src/ingestion/tests/e2e/lib/{metric_coverage,collect_metrics}.py.
Expand Down Expand Up @@ -55,7 +55,7 @@ jobs:
- uses: docker/setup-buildx-action@v3

# ─── Build the runner image ─────────────────────────────────────────
# The runner bakes in analytics-api + each connector's enrich binary, each
# The runner bakes in analytics + each connector's enrich binary, each
# compiled FROM ITS OWN Dockerfile via build-only services wired as
# `additional_contexts` (see compose/docker-compose.runner.yml). Nothing is
# compiled inside the runner at test time.
Expand All @@ -74,14 +74,14 @@ jobs:
# mariadb `depends_on` services — compose pulls their images on demand
# and waits for `service_healthy` — so no explicit image pre-pull is needed.
# Serial on purpose: the session rig is not xdist-safe yet (see
# conftest.py — per-worker analytics-api spawns race SeaORM migrations
# conftest.py — per-worker analytics spawns race SeaORM migrations
# in the shared MariaDB, non-primary workers don't wait for ClickHouse
# migrations, and the shared dbt target/ dir is deleted by whichever
# worker finishes first). Wall-time cost is negligible: the runner image
# build (which compiles analytics-api + enrich) dominates the job.
# build (which compiles analytics + enrich) dominates the job.
#
# The suite ALSO snapshots the metric catalog to .artifacts/catalog_metrics.json
# (lib/collect_metrics.py, while analytics-api is up) — the
# (lib/collect_metrics.py, while analytics is up) — the
# metric-coverage-gate job below analyses it, no second app boot.
- name: Run E2E suite
run: ./e2e.sh test --tb=short -q
Expand Down Expand Up @@ -122,7 +122,7 @@ jobs:

# ─── Gate job — pure-Python analysis of the collected catalog ─────────────
# Runs after `e2e` (needs), downloads `coverage-inputs`, reads the catalog file.
# No Docker, no analytics-api boot. `!cancelled()` so it still reports when the
# No Docker, no analytics boot. `!cancelled()` so it still reports when the
# suite itself failed (the artifact is usually present).

metric-coverage-gate:
Expand Down
Loading
Loading