Skip to content
19 changes: 16 additions & 3 deletions analytics/models/read_models/deduped_activities.sql
Original file line number Diff line number Diff line change
Expand Up @@ -29,7 +29,11 @@ absent_group_members AS (
absent.id AS activity_id,
absent.provider_id AS provider_id,
absent.external_id AS external_id,
absent.provider_absent_at AS provider_absent_at
absent.provider_absent_at AS provider_absent_at,
coalesce(
nullIf(trim(BOTH ' ' FROM JSONExtractString(absent.raw, 'sourceName')), ''),
nullIf(trim(BOTH ' ' FROM absent.source_name), '')
) AS subsource
FROM final_groups
INNER JOIN {{ source('postgres_fitness', 'activity') }} AS absent FINAL
ON absent.id = final_groups.activity_id
Expand All @@ -48,7 +52,8 @@ absent_source_links AS (
'providerId', assumeNotNull(provider_id),
'externalId', assumeNotNull(external_id),
'memberActivityId', toString(activity_id),
'providerAbsentAt', toString(assumeNotNull(provider_absent_at))
'providerAbsentAt', toString(assumeNotNull(provider_absent_at)),
'subsource', coalesce(subsource, '')
),
provider_id IS NOT null
AND external_id IS NOT null
Expand Down Expand Up @@ -104,7 +109,15 @@ merged AS (
maxIf(ranked.source_synced_at, ranked.activity_id IS NOT null) AS source_synced_at,
arraySort(groupUniqArrayIf(ranked.provider_id, ranked.activity_id IS NOT null)) AS source_providers,
groupArrayIf(
map('providerId', ranked.provider_id, 'externalId', ranked.external_id),
map(
'providerId', ranked.provider_id,
'externalId', ranked.external_id,
'memberActivityId', toString(ranked.activity_id),
'subsource', coalesce(
nullIf(trim(BOTH ' ' FROM JSONExtractString(ranked.raw, 'sourceName')), ''),
nullIf(trim(BOTH ' ' FROM ranked.source_name), '')
)
),
ranked.activity_id IS NOT null
AND ranked.external_id IS NOT null
AND ranked.external_id != ''
Expand Down
Original file line number Diff line number Diff line change
Expand Up @@ -130,6 +130,8 @@ describe("production analytics read-model build", () => {
expect(sql).toContain("ref('activity_duplicate_groups')");
expect(sql).toContain("current_deduped_activities AS");
expect(sql).toContain("member_activity_ids");
expect(sql).toContain("'memberActivityId', toString(ranked.activity_id)");
expect(sql).toContain("'subsource', coalesce(");
expect(sql).toContain("assumeNotNull(user_id) AS user_id");
expect(sql).toContain("stale_deduped_activities AS");
expect(sql).toContain("{% if is_incremental() %}");
Expand Down
35 changes: 35 additions & 0 deletions docs/production-incident-baseline.md
Original file line number Diff line number Diff line change
Expand Up @@ -11493,3 +11493,38 @@ new incremental tables are populated.
- **Remaining risk:** PR CI for
[#1456](https://github.com/Asherlc/dofek/pull/1456) was still pending when the
branch was pushed.

## 2026-07-03 — Strong Apple Health workout hidden by grouped source attribution

- **Symptoms:** The July 2, 2026 strength workout detail showed `WHOOP (via
Apple Health), WHOOP (Cloud)` and did not show `Strong`, even though the
expected Strong workout had synced from Apple Health.
- **User impact:** The workout was present in production data and visible as one
deduped activity, but the source attribution was misleading because the Strong
Apple Health member was not named in the activity detail source list.
- **Evidence:** Production ClickHouse `postgres_fitness.activity FINAL` contains
Apple Health member `58ccbfde-7867-4e22-874b-8e54b7106112`,
`external_id = hk:workout:1A5306B6-84B6-4889-90C2-55BE600111BA`,
`raw.sourceName = Strong`, `activity_type = strength_training`, local start
`2026-07-02 07:46:57`, with no provider tombstone or delete marker.
`analytics.deduped_activity_members FINAL` maps that member to canonical
activity `2d9e3c60-6acc-46df-8fdd-5c00928ae27c`, whose group also includes
one WHOOP cloud row and six WHOOP Apple Health rows. CDC health logs reported
`[clickhouse-cdc-health] ok: checked 3 slots and 1 mirror`.
- **Root cause:** Source attribution is modeled at provider granularity plus one
activity-level Apple Health `subsource`. The deduped group stores
`source_providers = {apple_health, whoop}` and multiple Apple Health
`source_external_ids`, but those entries do not carry per-member
`raw.sourceName`. The web source renderer therefore has only one
`apple_health` label and uses the group's single `subsource`, which was
`WHOOP`, so `Strong` was dropped from the displayed attribution.
- **Fix / mitigation:** No data repair was needed. A code fix should preserve
per-member Apple Health source names in source attribution and render distinct
labels such as `Strong (via Apple Health)` and `WHOOP (via Apple Health)`
alongside `WHOOP (Cloud)`.
- **Validation:** Read-only production queries confirmed the Strong row exists
in both the raw ClickHouse mirror and the deduped activity membership table;
no ClickHouse CDC staleness was observed.
- **Remaining risk:** The UI will keep omitting secondary Apple Health app names
for deduped activities that contain multiple Apple Health sources until source
attribution carries per-member subsources.
Original file line number Diff line number Diff line change
@@ -0,0 +1,132 @@
# Activity Source Attribution Domain Model TDD Plan

This plan uses test-first implementation steps. Steps use checkbox (`- [ ]`) syntax for tracking.

**Goal:** Promote activity source attribution from generic provider/id JSON plumbing into a first-class typed read-model contract.

**Behavior:** A deduped activity exposes each member source distinctly, including provider, upstream app/subsource, external id, member activity id, source label, source URL, and removed/absent state, so web and mobile render the same source list without provider-level inference.

**Scope:** Activity source-attribution models, API repository schemas, ClickHouse/dbt and Postgres view projections, and web/mobile source rendering. Non-goals: changing canonical activity storage, changing provider priority, or adding a new ingestion deduplication path.

**Docs:** `docs/production-incident-baseline.md`, `packages/server/src/models/activity-source-attribution.ts`, `packages/server/src/models/activity.ts`, `packages/server/src/repositories/activity-repository.ts`, `packages/server/src/repositories/activities-calendar-repository.ts`, `packages/web/src/pages/ActivityDetailPage.tsx`, `packages/mobile/app/activity/[id].tsx`, `analytics/models/read_models/deduped_activities.sql`, `drizzle/_views/01_v_activity.sql`.

---

## Current Evidence

- The July 2, 2026 Strong strength workout exists in production ClickHouse and is grouped into the same canonical activity as WHOOP cloud and WHOOP Apple Health member rows.
- Current attribution plumbing is split between `source_providers`, `source_external_ids`, `subsource`, and `provider_id`, which encourages clients to collapse multiple Apple Health upstream apps into one provider-level label.
- `ActivitySourceAttribution` already centralizes part of the behavior, but its input is still a generic `source_external_ids` map shape rather than an explicit activity-source read model.
- Before the immediate UI fix, web and mobile source rendering could collapse duplicate provider ids; the longer-term cleanup should make the typed source-attribution contract explicit enough that clients cannot regress to provider-level inference.

## Test Strategy

- Unit: source-attribution model tests cover grouped active sources, duplicate providers with different subsources, provider URL resolution, absent sources, mixed active/absent sources, and deterministic ordering.
- Repository/API: schema tests cover typed activity source rows from Postgres and ClickHouse projections, including `memberActivityId`, `providerId`, `externalId`, `subsource`, and `providerAbsentAt`.
- SQL/read models: SQL text or integration tests verify `deduped_activities` and `v_activity` emit the typed source-attribution shape from member rows.
- UI/mobile/web parity: web and mobile activity detail tests render multiple source entries for Strong via Apple Health, WHOOP via Apple Health, and WHOOP Cloud without collapsing by provider.

## File Structure

- Create: `packages/server/src/models/activity-source.ts` - first-class domain/read-model types for activity member source attribution.
- Modify: `packages/server/src/models/activity-source-attribution.ts` - consume typed source entries instead of generic external-id maps.
- Modify: `packages/server/src/models/activity.ts` - expose the typed source-attribution API shape.
- Modify: `packages/server/src/repositories/activity-repository.ts` - parse typed source-attribution rows with Zod.
- Modify: `packages/server/src/repositories/activities-calendar-repository.ts` - reuse the same typed source-attribution parsing for calendar activity rows.
- Modify: `analytics/models/read_models/deduped_activities.sql` - emit the typed activity source shape for ClickHouse consumers.
- Modify: `drizzle/_views/01_v_activity.sql` and add a migration - emit the typed activity source shape for Postgres consumers.
- Modify: `packages/web/src/pages/ActivityDetailPage.tsx` - render typed source attribution directly.
- Modify: `packages/mobile/app/activity/[id].tsx` - render typed source attribution directly, including duplicate providers with distinct subsources.

## Tasks

### Task 1: Add Failing Domain Model Tests

**Files:**
- Create/modify: `packages/server/src/models/activity-source.test.ts`
- Modify: `packages/server/src/models/activity-source-attribution.test.ts`
- Modify: `packages/server/src/models/activity.test.ts`

- [ ] Define the typed source-attribution behavior through tests before adding the new model.
- [ ] Cover duplicate provider ids with distinct subsources and member ids.
- [ ] Cover active plus absent source behavior without collapsing absent entries incorrectly.
- [ ] Run `rtk pnpm vitest run packages/server/src/models/activity-source.test.ts packages/server/src/models/activity-source-attribution.test.ts packages/server/src/models/activity.test.ts`.
- [ ] Confirm the tests fail for the expected missing typed model or collapsed-source reason.

### Task 2: Add Failing Repository And Read-Model Tests

**Files:**
- Modify: `packages/server/src/repositories/activity-repository.test.ts`
- Modify: `packages/server/src/repositories/activities-calendar-repository.test.ts`
- Modify: `analytics/models/read_models/read_model_microbatch.sql.test.ts`

- [ ] Add repository tests proving rows parse typed source attribution instead of unstructured map entries.
- [ ] Add calendar repository tests for the same source-attribution shape.
- [ ] Add SQL/read-model assertions for `memberActivityId`, `providerId`, `externalId`, `subsource`, and `providerAbsentAt`.
- [ ] Run `rtk pnpm vitest run packages/server/src/repositories/activity-repository.test.ts packages/server/src/repositories/activities-calendar-repository.test.ts analytics/models/read_models/read_model_microbatch.sql.test.ts`.
- [ ] Confirm the tests fail for the expected schema/projection reason.

### Task 3: Add Failing Web And Mobile Parity Tests

**Files:**
- Modify: `packages/web/src/pages/ActivityDetailPage.test.tsx`
- Modify: `packages/mobile/app/activity/[id].test.tsx`

- [ ] Add web activity-detail tests for Strong via Apple Health, WHOOP via Apple Health, and WHOOP Cloud on one deduped activity.
- [ ] Add mobile activity-detail tests for the same source list.
- [ ] Assert duplicate provider ids are rendered as distinct entries when `memberActivityId` or `subsource` differs.
- [ ] Run `rtk pnpm vitest run packages/web/src/pages/ActivityDetailPage.test.tsx`.
- [ ] Run `rtk pnpm test:mobile -- app/activity/[id].test.tsx`.
- [ ] Confirm the tests fail for the expected collapsed-source reason.

### Task 4: Implement The Typed Source-Attribution Contract

**Files:**
- Create: `packages/server/src/models/activity-source.ts`
- Modify: `packages/server/src/models/activity-source-attribution.ts`
- Modify: `packages/server/src/models/activity.ts`
- Modify: `packages/server/src/repositories/activity-repository.ts`
- Modify: `packages/server/src/repositories/activities-calendar-repository.ts`

- [ ] Add a typed activity source entry model with explicit member id, provider id, external id, subsource, absent timestamp, display label, and URL fields.
- [ ] Make repository schemas parse the typed shape with Zod at the runtime boundary.
- [ ] Keep provider label/link resolution in the server domain model so clients only render supplied fields.
- [ ] Keep `sourceProviders` only where a legacy/list-level provider-id summary is still needed.
- [ ] Run the focused server/model/repository tests and confirm they pass.

### Task 5: Update SQL Projections

**Files:**
- Modify: `analytics/models/read_models/deduped_activities.sql`
- Modify: `analytics/models/read_models/read_model_microbatch.sql.test.ts`
- Modify: `drizzle/_views/01_v_activity.sql`
- Create: `drizzle/YYYY_v_activity_source_attribution_model.sql`

- [ ] Emit the typed source-attribution shape from deduped activity member rows.
- [ ] Preserve per-member Apple Health upstream app names from `raw.sourceName` with a provider `source_name` fallback.
- [ ] Include absent source entries with the same typed shape and removed timestamp.
- [ ] Run `rtk pnpm vitest run analytics/models/read_models/read_model_microbatch.sql.test.ts`.
- [ ] Run `rtk pnpm lint:migrations`.

### Task 6: Render Typed Sources On Web And Mobile

**Files:**
- Modify: `packages/web/src/pages/ActivityDetailPage.tsx`
- Modify: `packages/web/src/pages/ActivityDetailPage.stories.tsx`
- Modify: `packages/mobile/app/activity/[id].tsx`

- [ ] Render `activity.sourceLinks` or its successor typed source list directly on web.
- [ ] Render the same typed source list directly on mobile without looking up links by provider id.
- [ ] Keep removed sources visually distinct on both platforms.
- [ ] Update stories for default, multi-source grouped, and removed-source states where the component has stories.
- [ ] Run the focused web and mobile tests and confirm they pass.

### Task 7: Final Verification

- [ ] Run `rtk pnpm vitest run packages/server/src/models/activity-source.test.ts packages/server/src/models/activity-source-attribution.test.ts packages/server/src/models/activity.test.ts packages/server/src/repositories/activity-repository.test.ts packages/server/src/repositories/activities-calendar-repository.test.ts analytics/models/read_models/read_model_microbatch.sql.test.ts packages/web/src/pages/ActivityDetailPage.test.tsx`.
- [ ] Run `rtk pnpm test:mobile -- app/activity/[id].test.tsx`.
- [ ] Run `rtk pnpm lint`.
- [ ] Run `rtk pnpm tsc --noEmit`.
- [ ] From `packages/server`, run `rtk pnpm tsc --noEmit`.
- [ ] From `packages/web`, run `rtk pnpm tsc --noEmit`.
- [ ] From `packages/mobile`, run `rtk pnpm tsc --noEmit`.
Loading
Loading