Skip to content
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
55 changes: 49 additions & 6 deletions docs/production-incident-baseline.md
Original file line number Diff line number Diff line change
Expand Up @@ -18472,8 +18472,8 @@ Drizzle schema and runtime Zod schemas. Findings and remediations:

## 2026-07-26 — Peloton Performance Summary Type Drift Blocked Sync

- **Status:** Direct source fix reproduced and validated locally; merge and
production deployment pending.
- **Status:** Resolved after the direct source fix merged in PR #2047 and
deployed in release `f5b951f09ae35f3947dfd03fa0459b7d7d3a4596`.
- **Symptoms:** The first post-deploy scheduled Peloton sync reported
[Sentry issue DOFEK-SERVER-5F](https://east-bay-software.sentry.io/issues/DOFEK-SERVER-5F)
while validating performance graphs.
Expand Down Expand Up @@ -18501,10 +18501,13 @@ Drizzle schema and runtime Zod schemas. Findings and remediations:
first with the exact two Zod paths reported by Sentry. After correcting the
canonical schema, 129 focused client/provider unit tests and all 12 Peloton
sync integration tests pass; the root TypeScript typecheck and targeted
Biome checks also pass.
- **Remaining risk / follow-up:** Merge through normal CI, deploy, observe a
successful scheduled Peloton sync on the fixed release, and then resolve
DOFEK-SERVER-5F.
Biome checks also pass. The first scheduled production sync on the fixed
release completed at `2026-07-26T20:00:05Z` with two workouts and 1,261
metric-stream rows. Sentry recorded no event after the previous release's
final failure at `2026-07-26T19:30:04Z`, and DOFEK-SERVER-5F was resolved
with that evidence attached.
- **Remaining risk / follow-up:** None beyond normal scheduled-sync and Sentry
monitoring.

## 2026-07-26 — Reused Analytics CTEs Recomputed and Forced Projection Failed

Expand Down Expand Up @@ -18562,3 +18565,43 @@ Drizzle schema and runtime Zod schemas. Findings and remediations:
changing the four-minute query ceiling, observe a complete production dbt
cycle in which both models succeed, verify downstream cache warming, then
resolve DOFEK-SERVER-5A.

## 2026-07-26 — HealthKit Observer Sync Waited Behind a Background Timer

- **Status:** Direct source fix reproduced and validated locally; merge, OTA
deployment, and physical-device production validation pending.
- **Symptoms:** The iOS native observer coordinator reported
[Sentry issue DOFEK-MOBILE-1C](https://east-bay-software.sentry.io/issues/DOFEK-MOBILE-1C)
because HealthKit observer callbacks remained incomplete past their
25-second native failure boundary.
- **User impact:** Background HealthKit deliveries could expire before their
JavaScript sync began, delaying the affected samples until a later delivery
or foreground catch-up.
- **Evidence:** The latest production breadcrumb trail recorded a burst of
observer updates beginning at `03:17:55.677Z`; every callback logged
`Sample update event received, debouncing`, but `Starting sync` did not
appear until `03:18:25.763Z`, about 30 seconds later and after the native
25-second expiration. The app was backgrounded and the device was locked.
The timestamps, device state, and expired update ID are preserved in
[Sentry issue DOFEK-MOBILE-1C](https://east-bay-software.sentry.io/issues/DOFEK-MOBILE-1C).
- **Root cause:** The observer path deferred its serialized queue behind a
500-millisecond JavaScript `setTimeout`. In the recorded background delivery,
that timer did not execute for about 30 seconds while the native completion
deadline continued advancing.
- **Fix / mitigation:** Remove the timer and enqueue each native delivery
directly into the existing single-flight queue. One sync still runs at a
time; update IDs delivered during it remain pending for the next serialized
sync and are acknowledged only after that sync settles. The native
25-second failure boundary remains unchanged. Apple requires observer
completion only after processing the delivered data:
<https://developer.apple.com/documentation/healthkit/executing-observer-queries>.
- **Validation:** The regression first failed because no sync began without
advancing fake timers. After the direct queue fix, the regression and all 31
background HealthKit orchestration tests pass, including initial catch-up
ordering, updates delivered during an active sync, exact native
acknowledgements, locked-device behavior, and failure telemetry.
- **Remaining risk / follow-up:** Merge through normal CI, deploy the
JavaScript bundle through the production OTA channel, then confirm on a
physical-device observer delivery that `Starting sync` follows the update
immediately and no fixed-update expiration is reported before resolving
DOFEK-MOBILE-1C.
70 changes: 51 additions & 19 deletions packages/mobile/lib/background-health-kit-sync.test.ts
Original file line number Diff line number Diff line change
Expand Up @@ -188,7 +188,7 @@ describe("initBackgroundHealthKitSync", () => {
updateId: "update-1",
});

// Advance past debounce timer
// Allow the serialized observer sync to settle.
await vi.advanceTimersByTimeAsync(5000);
// Let sync promises resolve
await vi.runAllTimersAsync();
Expand Down Expand Up @@ -447,7 +447,7 @@ describe("initBackgroundHealthKitSync", () => {
const firstClient = createMockClient();
const firstSync = createDeferred<{ inserted: number }>();
firstClient.healthKitSync.pushWorkouts.mutate.mockReturnValueOnce(firstSync.promise);
vi.mocked(queryWorkouts).mockResolvedValueOnce([
vi.mocked(queryWorkouts).mockResolvedValue([
{
activityType: 1,
startDate: "2026-03-22T10:00:00Z",
Expand All @@ -462,15 +462,22 @@ describe("initBackgroundHealthKitSync", () => {
const secondClient = createMockClient();
const secondOnSyncComplete = vi.fn();
await initBackgroundHealthKitSync(secondClient, secondOnSyncComplete);
const secondListener = mockAddSampleUpdateListener.mock.calls[1][0];
secondListener({
typeIdentifier: "HKQuantityTypeIdentifierHeartRate",
updateId: "second-context-update",
});
expect(secondClient.healthKitSync.pushWorkouts.mutate).not.toHaveBeenCalled();

firstSync.resolve({ inserted: 1 });
await vi.waitFor(() => {
expect(secondClient.healthKitSync.pushWorkouts.mutate).toHaveBeenCalledTimes(1);
expect(secondClient.healthKitSync.pushWorkouts.mutate).toHaveBeenCalledTimes(2);
});

expect(firstOnSyncComplete).toHaveBeenCalledTimes(1);
expect(secondOnSyncComplete).toHaveBeenCalledTimes(1);
expect(firstClient.healthKitSync.pushWorkouts.mutate).toHaveBeenCalledTimes(1);
expect(firstOnSyncComplete).not.toHaveBeenCalled();
expect(secondOnSyncComplete).toHaveBeenCalledTimes(2);
expect(mockCompleteObserverUpdates).toHaveBeenCalledWith(["second-context-update"], true);
});

it("tears down and reports an observer registration failure", async () => {
Expand Down Expand Up @@ -574,7 +581,35 @@ describe("initBackgroundHealthKitSync", () => {
vi.useRealTimers();
});

it("coalesces debounced observer updates and completes every callback once", async () => {
it("starts observer sync without waiting for a background timer (DOFEK-MOBILE-1C)", async () => {
const client = createMockClient();
await initBackgroundHealthKitSync(client);
await vi.waitFor(() => {
expect(mockLoggerInfo).toHaveBeenCalledWith(
"bg-healthkit-sync",
"Observer processing complete",
expect.any(Object),
);
});
const startingSyncCount = mockLoggerInfo.mock.calls.filter(
([, message]) => message === "Starting sync",
).length;

const listener = mockAddSampleUpdateListener.mock.calls[0][0];
listener({
typeIdentifier: "HKQuantityTypeIdentifierStepCount",
updateId: "update-1",
});

expect(
mockLoggerInfo.mock.calls.filter(([, message]) => message === "Starting sync"),
).toHaveLength(startingSyncCount + 1);
await vi.waitFor(() => {
expect(mockCompleteObserverUpdates).toHaveBeenCalledWith(["update-1"], true);
});
});
Comment thread
qodo-code-review[bot] marked this conversation as resolved.

it("serializes observer updates and completes every callback once", async () => {
vi.useFakeTimers();
const client = createMockClient();
await initBackgroundHealthKitSync(client);
Expand All @@ -594,13 +629,14 @@ describe("initBackgroundHealthKitSync", () => {
await vi.advanceTimersByTimeAsync(500);
await vi.runAllTimersAsync();

expect(mockCompleteObserverUpdates).toHaveBeenCalledTimes(1);
expect(mockCompleteObserverUpdates).toHaveBeenCalledWith(["update-1", "update-2"], true);
expect(client.healthKitSync.pushWorkouts.mutate).toHaveBeenCalledTimes(2);
expect(mockCompleteObserverUpdates).toHaveBeenCalledTimes(2);
expect(mockCompleteObserverUpdates).toHaveBeenNthCalledWith(1, ["update-1"], true);
expect(mockCompleteObserverUpdates).toHaveBeenNthCalledWith(2, ["update-2"], true);
expect(client.healthKitSync.pushWorkouts.mutate).toHaveBeenCalledTimes(3);
vi.useRealTimers();
});

it("debounces an update received while the preceding observer sync is still running", async () => {
it("runs a queued update immediately after the preceding observer sync", async () => {
vi.useFakeTimers();
const client = createMockClient();
await initBackgroundHealthKitSync(client);
Expand Down Expand Up @@ -634,20 +670,16 @@ describe("initBackgroundHealthKitSync", () => {
typeIdentifier: "HKQuantityTypeIdentifierStepCount",
updateId: "update-1",
});
await vi.advanceTimersByTimeAsync(500);
listener({
typeIdentifier: "HKQuantityTypeIdentifierHeartRate",
updateId: "update-2",
});

firstSync.resolve({ inserted: 1 });
await vi.advanceTimersByTimeAsync(499);
expect(mockCompleteObserverUpdates).toHaveBeenCalledTimes(1);
expect(mockCompleteObserverUpdates).toHaveBeenCalledWith(["update-1"], true);

await vi.advanceTimersByTimeAsync(1);
await vi.runAllTimersAsync();
expect(mockCompleteObserverUpdates).toHaveBeenCalledTimes(2);
await vi.waitFor(() => {
expect(mockCompleteObserverUpdates).toHaveBeenCalledTimes(2);
});
expect(mockCompleteObserverUpdates).toHaveBeenNthCalledWith(1, ["update-1"], true);
expect(mockCompleteObserverUpdates).toHaveBeenLastCalledWith(["update-2"], true);
vi.useRealTimers();
});
Expand Down Expand Up @@ -725,7 +757,7 @@ describe("teardownBackgroundHealthKitSync", () => {
teardownBackgroundHealthKitSync();
});

it("removes the listener, clears timers, and drains pending native callbacks", async () => {
it("removes the listener and drains pending native callbacks", async () => {
vi.useFakeTimers();
const mockRemove = vi.fn();
mockAddSampleUpdateListener.mockReturnValue({ remove: mockRemove });
Expand Down
96 changes: 50 additions & 46 deletions packages/mobile/lib/background-health-kit-sync.ts
Original file line number Diff line number Diff line change
Expand Up @@ -11,17 +11,15 @@ import type { HealthKitSyncStage, SyncTrpcClient } from "./health-kit-sync";
import { captureException, logger } from "./telemetry";

const TAG = "bg-healthkit-sync";
const DEBOUNCE_MS = 500;

interface SyncContext {
onSyncComplete?: () => void | Promise<void>;
trpcClient: SyncTrpcClient;
}

let subscription: EventSubscription | null = null;
let debounceTimer: ReturnType<typeof setTimeout> | undefined;
let observerSyncReady: true | undefined;
let pendingCatchUp:
| {
onSyncComplete?: () => void | Promise<void>;
trpcClient: SyncTrpcClient;
}
| undefined;
let currentSyncContext: SyncContext | undefined;
let pendingCatchUp: SyncContext | undefined;
const pendingUpdateIds = new Set<string>();
let syncing: true | undefined;

Expand Down Expand Up @@ -127,56 +125,58 @@ function acknowledgeObserverUpdates(updateIds: string[], succeeded: boolean): vo
}
}

async function drainSyncQueue(
trpcClient: SyncTrpcClient,
onSyncComplete?: () => void | Promise<void>,
): Promise<void> {
if (syncing) {
async function drainSyncQueue(): Promise<void> {
const context = currentSyncContext;
if (syncing || !context) {
return;
}

const catchUp = pendingCatchUp;
const catchUp = pendingCatchUp === context;
if (catchUp) {
pendingCatchUp = undefined;
} else if (!observerSyncReady) {
} else if (pendingUpdateIds.size === 0) {
return;
}

const updateIds = catchUp ? [] : Array.from(pendingUpdateIds);
if (!catchUp) {
pendingUpdateIds.clear();
observerSyncReady = undefined;
}

syncing = true;
Comment thread
Asherlc marked this conversation as resolved.
const succeeded = await performHealthKitSync(
catchUp?.trpcClient ?? trpcClient,
catchUp?.onSyncComplete ?? onSyncComplete,
);
if (updateIds.length > 0) {
acknowledgeObserverUpdates(updateIds, succeeded);
try {
const succeeded = await performHealthKitSync(
context.trpcClient,
context.onSyncComplete
? async () => {
if (currentSyncContext === context) {
await context.onSyncComplete?.();
}
}
: undefined,
);
if (currentSyncContext === context && updateIds.length > 0) {
acknowledgeObserverUpdates(updateIds, succeeded);
}
} catch (error) {
captureException(error, {
source: TAG,
operation: "drainSyncQueue",
});
if (currentSyncContext === context && updateIds.length > 0) {
acknowledgeObserverUpdates(updateIds, false);
}
} finally {
syncing = undefined;
}
Comment thread
qodo-code-review[bot] marked this conversation as resolved.
syncing = undefined;

await drainSyncQueue(trpcClient, onSyncComplete);
}

function scheduleObserverSync(
trpcClient: SyncTrpcClient,
onSyncComplete?: () => void | Promise<void>,
): void {
clearTimeout(debounceTimer);
debounceTimer = setTimeout(() => {
debounceTimer = undefined;
observerSyncReady = true;
void drainSyncQueue(trpcClient, onSyncComplete);
}, DEBOUNCE_MS);
await drainSyncQueue();
}

/**
* Initialize background HealthKit sync.
* Sets up observer queries that fire when new health samples arrive,
* then debounces and syncs the last 24 hours of data to the server.
* then serializes syncs of the last 24 hours of data to the server.
*
* Call this once after authentication is established.
*/
Expand All @@ -196,12 +196,18 @@ export async function initBackgroundHealthKitSync(
teardownBackgroundHealthKitSync();
}

const context: SyncContext = { trpcClient, onSyncComplete };
currentSyncContext = context;

// Listen before registering native observers so an immediate HealthKit
// delivery can never race ahead of the JavaScript callback.
subscription = addSampleUpdateListener((event) => {
logger.info(TAG, "Sample update event received, debouncing");
if (currentSyncContext !== context) {
return;
}
logger.info(TAG, "Sample update event received, queueing sync");
pendingUpdateIds.add(event.updateId);
scheduleObserverSync(trpcClient, onSyncComplete);
void drainSyncQueue();
});

try {
Expand All @@ -215,22 +221,20 @@ export async function initBackgroundHealthKitSync(
throw error;
}
logger.info(TAG, "Background observers registered");
pendingCatchUp = { trpcClient, onSyncComplete };
void drainSyncQueue(trpcClient, onSyncComplete);
pendingCatchUp = context;
void drainSyncQueue();

logger.info(TAG, "Init complete, listening for HealthKit updates");
}

/** Clean up background sync listeners and timers */
/** Clean up background sync listeners and observers. */
export function teardownBackgroundHealthKitSync() {
currentSyncContext = undefined;
if (subscription) {
logger.info(TAG, "Tearing down: removing listener");
subscription.remove();
subscription = null;
}
clearTimeout(debounceTimer);
debounceTimer = undefined;
observerSyncReady = undefined;
pendingCatchUp = undefined;
pendingUpdateIds.clear();
try {
Expand Down
12 changes: 7 additions & 5 deletions packages/mobile/modules/health-kit/README.md
Original file line number Diff line number Diff line change
Expand Up @@ -28,10 +28,12 @@ This module provides the iOS-native HealthKit bridge used by the mobile app to:
## Background Observer Completion

Each native observer delivery receives a unique update ID. The native module
retains HealthKit's completion callback while JavaScript debounces the delivery
for 500 ms, runs one coalesced sync, and reports the batch result through
`completeObserverUpdates`. Re-registration, JavaScript teardown, and Expo module
destruction stop the queries and complete every callback still pending.
retains HealthKit's completion callback while JavaScript immediately places the
delivery in a single-flight sync queue and reports the batch result through
`completeObserverUpdates`. Updates delivered during a running sync remain
pending for the next serialized sync. Re-registration, JavaScript teardown, and
Expo module destruction stop the queries and complete every callback still
pending.
Comment thread
qodo-code-review[bot] marked this conversation as resolved.

A native 25-second expiration completes an update exactly once and reports the
expired update ID, HealthKit sample type, and monotonic callback age to Sentry
Expand Down Expand Up @@ -83,7 +85,7 @@ a physical iPhone with a Release build:
expiration and that no `com.dofek.healthkit-observer` expiration event is
reported.
4. Repeat with multiple sample types delivered together and confirm every
update ID is completed exactly once after the coalesced sync.
update ID is completed exactly once after its queued serialized sync settles.

Apple requires the observer completion handler to run after the app finishes
processing the delivered data:
Expand Down
Loading
Loading