Skip to content
Merged
58 changes: 58 additions & 0 deletions docs/production-incident-baseline.md
Original file line number Diff line number Diff line change
Expand Up @@ -19500,6 +19500,64 @@ Drizzle schema and runtime Zod schemas. Findings and remediations:
complete production analytics-plus-cache cycle below the unchanged
four-minute ceiling, and only then resolve DOFEK-SERVER-5A.

## 2026-07-27 — HealthKit Observer Replayed Every Sample Type

- **Status:** Direct source fix reproduced and validated locally; merge,
native release, and physical-device production validation pending.
- **Symptoms:** Mobile
[DOFEK-MOBILE-1C](https://east-bay-software.sentry.io/issues/7633118736/)
continued reporting native observer expirations. Before its independent
server-side resolution in
[PR #2232](https://github.com/Asherlc/dofek/pull/2232), downstream
[DOFEK-SERVER-5A](https://east-bay-software.sentry.io/issues/7632766197/)
also recorded the physical-row amplification that made analytics timeouts
more expensive.
- **User impact:** A change to one HealthKit type could replay the complete
previous-day HealthKit window. The native callback could expire before the
upload finished, and each replay created another physical metric-stream
version that made downstream latest-version analytics progressively more
expensive.
- **Evidence:** The exact mobile failure was
`com.dofek.healthkit-observer: Code: 1` after 25,016 ms for
`HKQuantityTypeIdentifierDistanceWalkingRunning`. The exact downstream
failure was ClickHouse code 159 after 240,002 ms while `deduped_sensor`
scanned 88,877,102 physical rows, approximately twice its 44,438,551-row
source. Code inspection showed that the observer event's
`typeIdentifier` was discarded and every delivery invoked all 18 quantity
queries plus workout routes and sleep over the previous-day window.
`queryAnchoredSamples` already existed but was unused, and its returned
anchor was persisted before the server upload succeeded.
- **Root cause:** The background observer path treated a type-specific change
signal as a request for a full-window, all-type replay and advanced unused
anchors before durable server processing, multiplying physical event
versions and keeping observer callbacks open past their deadline.
- **Fix / mitigation:** Preserve and coalesce delivered type identifiers,
register background delivery only for types the pipeline consumes, and run
type-scoped syncs. UUID-addressable quantity types now use two-phase
`HKAnchoredObjectQuery` processing: upload additions and UUID-scoped
tombstones first, then persist the opaque anchor only on success. Initial
anchor bootstrap and aggregate-only types remain bounded to the prior local
day. Explicit foreground/manual sync retains the full-range path. No timeout,
retry, analytics query, or size limit changed. Apple documents observer
queries as change signals and anchored queries as the API for retrieving
additions and deletions:
<https://developer.apple.com/documentation/healthkit/executing-observer-queries>
and
<https://developer.apple.com/documentation/HealthKit/HKAnchoredObjectQuery>.
- **Validation:** Test-first regressions cover delivered-type scoping,
addition/deletion upload, anchor commit only after server success, failed
upload retry behavior, and the restricted native background type set. The
focused mobile and server Vitest suites pass, the Swift package's 75 tests
pass, mobile/server/repository TypeScript checks pass, and an ad-hoc signed
Release simulator build succeeds with the generated Expo application
workspace.
- **Remaining risk / follow-up:** Ship through the normal native release path,
perform the documented physical-iPhone observer acceptance check, confirm
callbacks finish before the unchanged 25-second boundary, and verify that
metric-stream physical-row growth stays stable before resolving
DOFEK-MOBILE-1C. PR #2232 separately records the completed production
remediation and resolution of DOFEK-SERVER-5A.

## 2026-07-27 — Supplement integration fixtures failed canonical foreign keys

- **Status:** Root cause fixed locally on PR #2225; fresh exact-head CI
Expand Down
7 changes: 7 additions & 0 deletions packages/mobile/app/_layout.cleanup.test.tsx
Original file line number Diff line number Diff line change
Expand Up @@ -174,6 +174,12 @@ vi.mock("../modules/health-kit", () => ({
isAvailable: (...args: unknown[]) => mockIsHealthKitAvailable(...args),
hasEverAuthorized: (...args: unknown[]) => mockHasEverAuthorized(...args),
requestPermissions: (...args: unknown[]) => mockRequestPermissions(...args),
completeAnchoredQuery: vi.fn().mockResolvedValue(true),
queryAnchoredSamples: vi.fn().mockResolvedValue({
queryId: null,
samples: [],
deletedUUIDs: [],
}),
queryDailyStatistics: vi.fn().mockResolvedValue([]),
queryQuantitySamples: vi.fn().mockResolvedValue([]),
queryWorkouts: vi.fn().mockResolvedValue([]),
Expand All @@ -187,6 +193,7 @@ vi.mock("./login", () => ({

mockCreateClient.mockImplementation(() => ({
healthKitSync: {
deleteQuantitySamples: { mutate: vi.fn() },
Comment thread
coderabbitai[bot] marked this conversation as resolved.
Resolved
pushQuantitySamples: { mutate: vi.fn() },
pushWorkouts: { mutate: vi.fn() },
pushWorkoutRoutes: { mutate: vi.fn() },
Expand Down
3 changes: 3 additions & 0 deletions packages/mobile/app/_layout.tsx
Original file line number Diff line number Diff line change
Expand Up @@ -236,6 +236,9 @@ function AuthGate() {
if (!user || !trpcClient || !backgroundSyncReady) return;
const syncClient: SyncTrpcClient = {
healthKitSync: {
deleteQuantitySamples: {
mutate: (input) => trpcClient.healthKitSync.deleteQuantitySamples.mutate(input),
},
pushQuantitySamples: {
mutate: (input) => trpcClient.healthKitSync.pushQuantitySamples.mutate(input),
},
Expand Down
6 changes: 6 additions & 0 deletions packages/mobile/app/providers/[id].test.tsx
Original file line number Diff line number Diff line change
Expand Up @@ -316,9 +316,15 @@ vi.mock("../../lib/telemetry", () => ({
}));

vi.mock("../../modules/health-kit", () => ({
completeAnchoredQuery: vi.fn().mockResolvedValue(true),
getRequestStatus: (...args: unknown[]) => mockGetRequestStatus(...args),
hasEverAuthorized: (...args: unknown[]) => mockHasEverAuthorized(...args),
isAvailable: () => true,
queryAnchoredSamples: vi.fn().mockResolvedValue({
queryId: null,
samples: [],
deletedUUIDs: [],
}),
queryDailyStatistics: vi.fn(),
queryQuantitySamples: vi.fn(),
querySleepSamples: vi.fn(),
Expand Down
7 changes: 7 additions & 0 deletions packages/mobile/app/providers/index.test.tsx
Original file line number Diff line number Diff line change
Expand Up @@ -239,6 +239,12 @@ vi.mock("../../modules/health-kit", () => ({
getRequestStatus: (...args: unknown[]) => mockGetRequestStatus(...args),
hasEverAuthorized: (...args: unknown[]) => mockHasEverAuthorized(...args),
requestPermissions: (...args: unknown[]) => mockRequestPermissions(...args),
completeAnchoredQuery: vi.fn().mockResolvedValue(true),
queryAnchoredSamples: vi.fn().mockResolvedValue({
queryId: null,
samples: [],
deletedUUIDs: [],
}),
queryDailyStatistics: vi.fn().mockResolvedValue([]),
queryQuantitySamples: vi.fn().mockResolvedValue([]),
queryWorkouts: vi.fn().mockResolvedValue([]),
Expand Down Expand Up @@ -330,6 +336,7 @@ vi.mock("../../lib/trpc", () => ({
invalidate: mockInvalidate,
client: {
healthKitSync: {
deleteQuantitySamples: { mutate: vi.fn().mockResolvedValue({ deleted: 0 }) },
pushQuantitySamples: { mutate: vi.fn().mockResolvedValue({ inserted: 0, errors: [] }) },
pushWorkouts: { mutate: vi.fn().mockResolvedValue({ inserted: 0 }) },
pushWorkoutRoutes: { mutate: vi.fn().mockResolvedValue({ inserted: 0 }) },
Expand Down
9 changes: 9 additions & 0 deletions packages/mobile/lib/apple-health-provider.test.ts
Original file line number Diff line number Diff line change
Expand Up @@ -3,9 +3,15 @@ import { describe, expect, it, vi } from "vitest";
import type { SyncResult } from "./health-kit-sync";

vi.mock("../modules/health-kit", () => ({
completeAnchoredQuery: vi.fn(async () => true),
getRequestStatus: vi.fn(async () => "shouldRequest"),
hasEverAuthorized: vi.fn(() => false),
isAvailable: vi.fn(() => true),
queryAnchoredSamples: vi.fn(async () => ({
queryId: null,
samples: [],
deletedUUIDs: [],
})),
queryDailyStatistics: vi.fn(async () => []),
queryQuantitySamples: vi.fn(async () => []),
querySleepSamples: vi.fn(async () => []),
Expand Down Expand Up @@ -44,6 +50,9 @@ function createNative(overrides: Partial<AppleHealthAuthorizationNative> = {}) {
function createTrpcClient(): AppleHealthTrpcClient {
return {
healthKitSync: {
deleteQuantitySamples: {
mutate: vi.fn(async () => ({ deleted: 0 })),
},
pushQuantitySamples: {
mutate: vi.fn(async () => ({ inserted: 0, errors: [] })),
},
Expand Down
17 changes: 17 additions & 0 deletions packages/mobile/lib/apple-health-provider.ts
Original file line number Diff line number Diff line change
@@ -1,8 +1,10 @@
import { useCallback, useEffect, useMemo, useRef, useState } from "react";
import {
completeAnchoredQuery,
getRequestStatus,
hasEverAuthorized,
isAvailable,
queryAnchoredSamples,
queryDailyStatistics,
queryQuantitySamples,
querySleepSamples,
Expand All @@ -15,6 +17,7 @@ import {
type SyncOptions,
type SyncResult,
type SyncTrpcClient,
syncHealthKitObserverChanges,
syncHealthKitToServer,
} from "./health-kit-sync";
import { captureException } from "./telemetry";
Expand Down Expand Up @@ -158,6 +161,18 @@ export class AppleHealthSyncService {
onStage: options.onStage,
});
}

async syncObserverChanges(options: {
typeIdentifiers: readonly string[];
onStage?: SyncOptions["onStage"];
}): Promise<SyncResult> {
return syncHealthKitObserverChanges({
trpcClient: this.#trpcClient,
healthKit: this.#healthKit,
typeIdentifiers: options.typeIdentifiers,
onStage: options.onStage,
});
}
}

export class AppleHealthProviderModel {
Expand Down Expand Up @@ -324,6 +339,8 @@ export const defaultAppleHealthAuthorizationNative: AppleHealthAuthorizationNati
};

export const defaultAppleHealthAdapter: HealthKitAdapter = {
completeAnchoredQuery,
queryAnchoredSamples,
queryDailyStatistics,
queryQuantitySamples,
queryWorkouts,
Expand Down
Loading
Loading