Skip to content

[native] Bound FIT import memory with native streaming - #1646

Merged
Asherlc merged 9 commits into
mainfrom
Asherlc/investigate-sentry-7614315645
Jul 17, 2026
Merged

Asherlc merged 9 commits into
mainfrom
Asherlc/investigate-sentry-7614315645

Conversation

@Asherlc

@Asherlc Asherlc commented Jul 17, 2026

Copy link
Copy Markdown
Owner

Summary

  • Replace whole-file JavaScript FIT parsing with Garmin's official C++ decoder, built through CMake, Ninja, and vcpkg in Docker and CI.
  • Stream validated activity or weight messages through 250-record, 512 KiB backpressured batches while bounding classification telemetry and hashing files without buffering them.
  • Route Garmin dumps and Wahoo, COROS, and Suunto downloads through the canonical FIT import queue with correct worker lifecycle, retry cleanup, progress, and unsupported-file handling.
  • Add native protocol coverage, provider/job regressions, exact incident-file validation, package documentation, and the production RCA.

Summary by cubic

Replaced whole‑file JS FIT parsing with a native C++ streaming decoder to bound memory and backpressure large imports. Also gated sync/upload/webhook work behind worker startup and tightened the import lifecycle to finish in‑process decoding before cleanup.

  • New Features

    • Added native/fit-decoder (CMake + vcpkg, Garmin FIT C++ SDK) emitting NDJSON with per‑batch acks (max 250 messages / 512 KiB); CI job, Docker stage, and a build:fit-decoder script.
    • Unified FIT imports: Garmin dumps and Wahoo, COROS, Suunto now use the fit-file-import workflow via enqueueFitFileImportAndWait with retries, progress, and aggregation.
    • Targeted webhook syncs can start the worker when needed (requiresWorkerForWebhookSync); FIT external IDs can fall back to a streaming SHA‑256 without buffering files.
  • Bug Fixes

    • Prevented partial metric‑stream replacement on decoder errors by writing to a temp scope and swapping on success; validated replacement publishers before mutation.
    • Avoided provider self‑deadlocks by running FIT imports inline in provider workers; uploads still use the queue.
    • Gated enqueues behind worker startup and surfaced failures to callers; start-worker now resolves only after Docker confirms the worker is running.
    • Awaited in‑process decoding before deleting staged files; added tests enforcing worker‑start awaiting and protocol size limits.

Written for commit 73eb113. Summary will update on new commits.

Review in cubic

Summary by CodeRabbit

  • New Features

    • Added native, streaming Garmin FIT decoding with bounded batches, backpressure acknowledgements, and support for both activity and weight files.
    • Routed all provider FIT handling through a unified “fit-file import” workflow with staged replay and consistent aggregated results.
    • FIT external IDs can now be derived directly from file content when needed.
  • Bug Fixes

    • Prevented duplicate background worker startup during targeted webhook sync.
    • Improved reliability by awaiting worker startup in sync and upload flows.
  • Documentation

    • Updated FIT importing documentation and Docker build instructions for the new native decoder pipeline.

Asherlc added 3 commits July 16, 2026 19:47
Replace whole-file JavaScript decoding with Garmin's native C++ decoder and backpressured batches so large and concurrent imports cannot multiply the complete parsed object graph in the Node heap.
Preserve current file-type routing and dropped-field telemetry while deriving counts from capped native metadata instead of reconstructing complete JavaScript message graphs.
Copilot AI review requested due to automatic review settings July 17, 2026 03:02
@qodo-code-review

Copy link
Copy Markdown

Qodo reviews are paused for this user.

Troubleshooting steps vary by plan Learn more →

On a Teams plan?
Reviews resume once this user has a paid seat and their Git account is linked in Qodo.
Link Git account →

Using GitHub Enterprise Server, GitLab Self-Managed, or Bitbucket Data Center?
These require an Enterprise plan - Contact us
Contact us →

@cursor

cursor Bot commented Jul 17, 2026

Copy link
Copy Markdown

Bugbot is not enabled for your account, so this pull request was not reviewed.

Enable Bugbot in the Cursor dashboard to get automatic reviews on future PRs.

@greptile-apps greptile-apps Bot left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Asherlc has reached the 50-credit limit for trial accounts. To continue receiving code reviews, upgrade your plan.

@sourcery-ai sourcery-ai Bot left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Sorry @Asherlc, your pull request is larger than the review limit of 150000 diff characters

Copilot AI left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Copilot was unable to review this pull request because the user who requested the review has reached their quota limit.

@coderabbitai

coderabbitai Bot commented Jul 17, 2026

Copy link
Copy Markdown

Review Change Stack

📝 Walkthrough

Walkthrough

This PR replaces JavaScript worker-thread FIT parsing with a native C++ streaming decoder that applies bounded batch limits and acknowledgement backpressure. FIT data flows through the canonical fit-file-import queue, providers and webhooks coordinate worker startup asynchronously, and Docker/CI/tests validate the new pipeline end-to-end.

Changes

Native FIT streaming decoder and protocol

Layer / File(s) Summary
Native FIT decoder C++ implementation
native/fit-decoder/src/fit-decoder.cpp, native/fit-decoder/src/field-json.cpp, native/fit-decoder/src/field-json.hpp, native/fit-decoder/tests/FieldJsonTest.cpp
Two-pass FIT decode: metadata-only first pass, then record/weight JSON batches capped at 250 messages and 512 KiB with continue acknowledgement waits. FIT field-to-JSON conversion handles timestamps, strings, semicircles, 64-bit precision, and special types like left_right_balance.
Garmin SDK vcpkg overlay port
native/fit-decoder/vcpkg-ports/garmin-fit-sdk/*, native/fit-decoder/vcpkg-configuration.json
Adds overlay port with CMake build, generates FIT enum name lookup tables from fit_profile.hpp, pins Garmin SDK commit via vcpkg manifest-mode with baseline configuration.
CMake build and test
native/fit-decoder/CMakeLists.txt, native/fit-decoder/CMakePresets.json, native/fit-decoder/tests/VerifyProtocol.cmake
CMake 3.24+ build with C++17, release preset, output directory .build/fit-decoder/bin. Protocol validation test enforces 250-message batch limits, requires metadata and end messages, verifies message count consistency, and rejects truncated files.
TypeScript streaming adapter
src/fit/stream-decoder.ts, src/fit/stream-decoder.test.ts
Spawns native decoder, parses newline-delimited JSON protocol, enforces metadata-before-records/weights, applies idle timeout with SIGKILL, sends "continue\n" acknowledgements after consumer callbacks, validates message counts and file-type consistency. Comprehensive test coverage for sequencing, backpressure, enum mapping, failure modes, and protocol violations.
File-based external ID
src/fit/external-id.ts, src/fit/external-id.test.ts
Adds fitExternalIdFromFile() to stream-hash files for stable external IDs without loading full content to memory; replaces buffer-based ID derivation in import jobs.
Remove worker-thread parser
src/fit/parser.ts
Deletes ParsedFitActivity interface, parseFitFile() function, and timeout logic; retains only parseFitRecord/parseFitSession for record normalization.

Queued FIT import job and persistence

Layer / File(s) Summary
Queue results and lifecycle
src/jobs/queues.ts, src/jobs/queues.test.ts
Adds fitFileImportJobResultSchema with recordsSynced and errors; updates queue generics to include result type; adds QueueEvents caching with getFitFileImportQueueEvents() for lifecycle cleanup.
FIT import enqueuing
src/jobs/enqueue-fit-file-import.ts, src/jobs/enqueue-fit-file-import.test.ts
Stages FIT to temp directory, writes input.fit, builds logical filename and activity summary. Routes to either in-process processFitFileImportJob (when metricStreamPublisher provided) or enqueues to queue and waits for completion via QueueEvents.
FIT import job orchestration
src/jobs/process-fit-file-import-job.ts, src/jobs/process-fit-file-import-job.test.ts
Streams FIT via streamFitFile, stages decoded batches to spool files, replays weights or records via scoped metric writers, derives activity metadata and external IDs from stream, reports progress, handles errors (decoder/validation as recoverable), deletes FIT file on completion or unrecoverable failure. Tests validate streaming, staged replay, scope-aware persistence, file deletion, and dropped-field accounting.
Batch aggregation for Garmin fan-out
src/jobs/process-fit-file-import-batch-job.ts, src/jobs/process-garmin-dump-import-job.ts
Aggregates child FIT import job results, normalizes and groups error messages by cause, caps aggregated error output. Updates Garmin checkpoint schema to use unified fitFileImportJobResultSchema instead of local result schema.

Worker startup injection and coordination

Layer / File(s) Summary
Async worker startup
packages/server/src/lib/start-worker.ts, packages/server/src/lib/start-worker.test.ts
Refactors startWorker() from void to Promise<void>, resolves on success or "already started", rejects with composed error messages for failures. Tests verify resolution timing, error classification, and message preference (stderr over err.message).
Sync router factory
packages/server/src/routers/sync.ts, packages/server/src/routers/sync.test.ts
Adds createSyncRouter(startSyncWorker) and createTriggerSyncProcedure(startSyncWorker) factories; triggerSync now awaits startSyncWorker() when providerJobs.length > 0.
Upload router and import injection
packages/server/src/routes/upload.ts, packages/server/src/routes/upload.test.ts
Adds UploadRouteDeps.startWorker; enqueueImport awaits worker startup before returning job id; routes inject deps.startWorker into single-file and chunked upload paths.
App router composition
packages/server/src/index.ts, packages/server/src/router.ts, packages/server/src/router.test.ts
Adds CreateAppOptions.startWorker and createAppRouter(syncRouterOverride) factory to wire injected worker startup through sync/upload routers.
Targeted webhook sync with worker coordination
packages/server/src/routes/webhooks.ts, packages/server/src/routes/webhooks.test.ts
Adds WebhookProvider.requiresWorkerForWebhookSync flag, per-request workerStartRequested guard to start worker only once, dynamically imports and awaits startWorker() before targeted syncWebhookEvent, reports errors via captureException().

Provider FIT route-through to import job

Layer / File(s) Summary
Provider FIT enqueuing
src/providers/coros.ts, src/providers/coros.test.ts, src/providers/suunto.ts, src/providers/suunto.test.ts, src/providers/wahoo/provider.ts, src/providers/wahoo/activity-persister.ts, src/providers/wahoo.test.ts
Routes provider FIT downloads (COROS/Suunto/Wahoo) to enqueueFitFileImportAndWait with activity summary. Removes inline parsing, metric-stream batch writing, and row-count logging. Wahoo also reorders WahooActivityPersister constructor to accept userId before metricStreamPublisher.

Build, CI, Docker, and documentation

Layer / File(s) Summary
Docker multi-stage build
Dockerfile
Adds fit-decoder-build stage that bootstraps vcpkg, builds decoder with CMake release preset, runs ctest, copies binary to server image. Server stage gains libstdc++ runtime dependency.
CI native decoder test
.github/workflows/test.yml
Adds test-fit-decoder job that bootstraps vcpkg, builds/tests with CMake, uploads binary artifact. Integration, mutation, and gate jobs download artifact and verify decoder availability before running tests.
Configuration and scripting
package.json, .dockerignore, knip.json, .github/dependabot.yml
Adds build:fit-decoder npm script; adds .build to .dockerignore; adds cmake to knip.json ignored binaries; configures Dependabot weekly vcpkg updates for native/fit-decoder.
Documentation and incidents
README.md, src/fit/README.md, src/fit/AGENTS.md, native/fit-decoder/README.md, native/fit-decoder/AGENTS.md, docs/production-incident-baseline.md
Documents native streaming architecture, two-pass protocol, 120s idle timeout, Docker layers, FIT contract limits (250 msg/batch, 512 KiB), protocol violations/backpressure, and updates incident records for heap exhaustion mitigation, disk space recovery, and CI workflow fixes. src/fit/AGENTS.md enforces requirements: stream FIT with preserved acks, dedicate processFitFileImportJob for persistence and spool cleanup, populate raw fields, validate in native tests and adapter tests.

Estimated code review effort: 5 (Critical) | ~120 minutes

Possibly related issues

  • Asherlc/dofek#1606: Addresses root cause (heap exhaustion from unbounded FIT import) via bounded streaming decoder, acknowledgement backpressure, and redesigned import orchestration.

Possibly related PRs

  • Asherlc/dofek#1588: Introduced the worker-thread FIT parser path (parser-worker.ts, parser-worker-entry.ts) replaced by this native streaming decoder.
  • Asherlc/dofek#1594: Overlaps on src/jobs/queues.ts and src/fit/external-id.ts FIT import queue infrastructure.
  • Asherlc/dofek#1639: Modifies same src/jobs/process-fit-file-import-job.ts for weight-type schema validation and FIT field handling.

Suggested labels: area/server, area/providers, area/infra, type/refactor

🚥 Pre-merge checks | ✅ 2
✅ Passed checks (2 passed)
Check name Status Explanation
Description Check ✅ Passed Check skipped - CodeRabbit’s high-level summary is enabled.
Title check ✅ Passed The title is concise, imperative, and accurately summarizes the native streaming FIT import change.

Thanks for using CodeRabbit! It's free for OSS, and your support helps us grow. If you like it, consider giving us a shout-out.

❤️ Share

Comment @coderabbitai help to get the list of available commands.

@greptile-apps greptile-apps Bot left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Asherlc has reached the 50-credit limit for trial accounts. To continue receiving code reviews, upgrade your plan.

@github-actions

github-actions Bot commented Jul 17, 2026

Copy link
Copy Markdown
Contributor

Storybook previews for c12b15b2 are ready:

This comment updates automatically on each PR push.

@greptile-apps greptile-apps Bot left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Asherlc has reached the 50-credit limit for trial accounts. To continue receiving code reviews, upgrade your plan.

@coderabbitai coderabbitai Bot left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Actionable comments posted: 11

Caution

Some comments are outside the diff and can’t be posted inline due to platform limitations.

⚠️ Outside diff range comments (1)
src/jobs/process-fit-file-import-job.ts (1)

211-286: 🗄️ Data Integrity & Integration | 🔴 Critical | 🏗️ Heavy lift

Do not mutate committed metrics before the decoder protocol completes.

Callbacks persist weight rows and clear/write activity samples before streamFitFile validates its final exit and message count. A late FitDecoderError therefore leaves partial weight data or erases an activity’s previous samples; the job then becomes unrecoverable and deletes its source file.

Stage each import and atomically replace the committed scope only after successful stream completion. Add a regression where metadata and one batch succeed before the decoder fails.

Also applies to: 331-394, 421-430

🤖 Prompt for AI Agents
Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.

In `@src/jobs/process-fit-file-import-job.ts` around lines 211 - 286, Change the
FIT import flow around writeWeightBatch, beginActivityImport, and streamFitFile
so decoder callbacks stage weight and activity metrics instead of mutating
committed database rows during streaming. Only atomically replace the committed
scope after streamFitFile completes validation of its exit status and message
count; preserve existing data when decoding fails. Add a regression covering
successful metadata and batch callbacks followed by a FitDecoderError.
🤖 Prompt for all review comments with AI agents
Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.

Inline comments:
In @.github/workflows/test.yml:
- Around line 561-563: Restrict the GITHUB_TOKEN permissions for the
test-fit-decoder job by adding a job-level permissions configuration with
minimal read-only access, replacing reliance on default workflow permissions.
Apply this to the job containing the runs-on and VCPKG_ROOT settings, without
changing its existing execution behavior.

In `@docs/production-incident-baseline.md`:
- Around line 13388-13418: Update the incident claims in the documented Fix /
mitigation, Validation, and Remaining risk sections to include primary-source
citations. Link decoder behavior and protocol assertions to the relevant native
decoder README, protocol tests, regression tests, and CI artifacts; cite
authoritative Docker or incident output for disk-exhaustion and operational
measurements. Ensure every newly added implementation, test-result, and
operational claim has an appropriate repository or official-source reference,
including the additional affected sections.

In `@native/fit-decoder/AGENTS.md`:
- Around line 3-4: Update the instructions in AGENTS.md to explicitly require
reading the repository root README.md before changing the package, while
preserving the existing requirement to read the package README.md.

In `@native/fit-decoder/CMakePresets.json`:
- Around line 21-26: Update native/fit-decoder/CMakePresets.json at lines 21-26
to add a release testPresets entry with outputOnFailure enabled and the
execution directory inherited from the release configure preset. Update
.github/workflows/test.yml at line 584 and Dockerfile at line 44 to invoke the
native tests with ctest --preset release instead of explicit arguments.

In `@native/fit-decoder/README.md`:
- Around line 7-8: Update the README’s backpressure description to state that
the decoder waits for a continue acknowledgement only after metadata and batch
output messages, not after the final end message. Align the wording with the
precise contract documented around the metadata, batches, and termination
behavior.
- Around line 3-8: Add authoritative links in the README paragraphs covering
protocol limits, message ordering, acknowledgement behavior, and failure
semantics, pointing to the native implementation and protocol-verification
source files. Keep the existing claims and wording intact while making each
claim traceable to its primary source.

In `@native/fit-decoder/src/fit-decoder.cpp`:
- Around line 118-130: The numeric conversion logic should preserve exact
integer values for the raw parsed record instead of routing every field through
GetFLOAT64Value(). Update this field-value handling to detect integer-valued FIT
fields and serialize their raw values losslessly, while retaining floating-point
conversion for fields requiring FIT scaling such as semicircles; keep the
existing left_right_balance representation unchanged.

In `@packages/server/src/routes/webhooks.test.ts`:
- Around line 611-616: Update the webhook test assertions around mockStartWorker
and syncWebhookEvent to explicitly require that syncWebhookEvent was called
before comparing invocation order. Remove the Number.POSITIVE_INFINITY fallback
so the test fails when the targeted sync never runs, while preserving the
ordering assertion that mockStartWorker occurs first.

In `@packages/server/src/routes/webhooks.ts`:
- Line 162: Update the webhook event-processing loop to use workerStartRequested
as a guard before every startWorker invocation, not only the final fallback. Set
the flag when startup is requested so multi-event payloads trigger at most one
worker start, while preserving the fallback behavior when no targeted event
requests startup.

In `@src/fit/AGENTS.md`:
- Around line 13-14: Add a blank line immediately after the “Testing Strategy”
heading in AGENTS.md, before the Native protocol list item.

In `@src/fit/stream-decoder.ts`:
- Around line 235-295: Reject any protocol message received after the first end
message before the switch dispatches it, so records and weights cannot reach
their consumers after completion. Update the end handling around endMessageCount
to establish this guard and remove the now-unreachable duplicate-end check; add
a regression test covering messages after end, including batches that could
otherwise satisfy the final count check.

---

Outside diff comments:
In `@src/jobs/process-fit-file-import-job.ts`:
- Around line 211-286: Change the FIT import flow around writeWeightBatch,
beginActivityImport, and streamFitFile so decoder callbacks stage weight and
activity metrics instead of mutating committed database rows during streaming.
Only atomically replace the committed scope after streamFitFile completes
validation of its exit status and message count; preserve existing data when
decoding fails. Add a regression covering successful metadata and batch
callbacks followed by a FitDecoderError.
🪄 Autofix (Beta)

Fix all unresolved CodeRabbit comments on this PR:

  • Push a commit to this branch (recommended)
  • Create a new PR with the fixes

ℹ️ Review info
⚙️ Run configuration

Configuration used: Path: .coderabbit.yaml

Review profile: ASSERTIVE

Plan: Pro Plus

Run ID: 50bc8808-ac68-4124-9e65-8b8ba95bf057

📥 Commits

Reviewing files that changed from the base of the PR and between 7fcfc78 and c34298b.

⛔ Files ignored due to path filters (1)
  • pnpm-lock.yaml is excluded by !**/pnpm-lock.yaml
📒 Files selected for processing (56)
  • .dockerignore
  • .github/dependabot.yml
  • .github/workflows/test.yml
  • Dockerfile
  • README.md
  • docs/production-incident-baseline.md
  • native/fit-decoder/AGENTS.md
  • native/fit-decoder/CLAUDE.md
  • native/fit-decoder/CMakeLists.txt
  • native/fit-decoder/CMakePresets.json
  • native/fit-decoder/GEMINI.md
  • native/fit-decoder/README.md
  • native/fit-decoder/src/fit-decoder.cpp
  • native/fit-decoder/tests/VerifyProtocol.cmake
  • native/fit-decoder/vcpkg-configuration.json
  • native/fit-decoder/vcpkg-ports/garmin-fit-sdk/CMakeLists.txt
  • native/fit-decoder/vcpkg-ports/garmin-fit-sdk/cmake/GenerateFitEnumNames.cmake
  • native/fit-decoder/vcpkg-ports/garmin-fit-sdk/portfile.cmake
  • native/fit-decoder/vcpkg-ports/garmin-fit-sdk/vcpkg.json
  • native/fit-decoder/vcpkg.json
  • package.json
  • packages/server/src/routes/webhooks.test.ts
  • packages/server/src/routes/webhooks.ts
  • src/fit/AGENTS.md
  • src/fit/README.md
  • src/fit/external-id.test.ts
  • src/fit/external-id.ts
  • src/fit/parser-edge.test.ts
  • src/fit/parser-worker-entry.test.ts
  • src/fit/parser-worker-entry.ts
  • src/fit/parser-worker.test.ts
  • src/fit/parser-worker.ts
  • src/fit/parser.test.ts
  • src/fit/parser.ts
  • src/fit/stream-decoder.test.ts
  • src/fit/stream-decoder.ts
  • src/fit/test-helpers.ts
  • src/index.test.ts
  • src/index.ts
  • src/jobs/enqueue-fit-file-import.test.ts
  • src/jobs/enqueue-fit-file-import.ts
  • src/jobs/garmin-import-progress.test.ts
  • src/jobs/process-fit-file-import-batch-job.ts
  • src/jobs/process-fit-file-import-job.test.ts
  • src/jobs/process-fit-file-import-job.ts
  • src/jobs/process-garmin-dump-import-job.ts
  • src/jobs/queues.test.ts
  • src/jobs/queues.ts
  • src/providers/coros.test.ts
  • src/providers/coros.ts
  • src/providers/suunto.test.ts
  • src/providers/suunto.ts
  • src/providers/types.ts
  • src/providers/wahoo.test.ts
  • src/providers/wahoo/activity-persister.ts
  • src/providers/wahoo/provider.ts
💤 Files with no reviewable changes (7)
  • src/fit/parser-worker-entry.test.ts
  • src/fit/parser-worker-entry.ts
  • src/fit/parser-worker.test.ts
  • src/fit/parser-edge.test.ts
  • src/fit/test-helpers.ts
  • src/fit/parser-worker.ts
  • src/fit/parser.ts

Comment thread .github/workflows/test.yml Outdated
Comment thread docs/production-incident-baseline.md Outdated
Comment thread native/fit-decoder/AGENTS.md Outdated
Comment thread native/fit-decoder/CMakePresets.json
Comment thread native/fit-decoder/README.md Outdated
Comment thread native/fit-decoder/src/fit-decoder.cpp Outdated
Comment thread packages/server/src/routes/webhooks.test.ts Outdated
Comment thread packages/server/src/routes/webhooks.ts Outdated
Comment thread src/fit/AGENTS.md Outdated
Comment thread src/fit/stream-decoder.ts
Keep failed decoder sessions from partially replacing persisted data and let provider workers process bounded native batches without deadlocking on their own queue.

@greptile-apps greptile-apps Bot left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Asherlc has reached the 50-credit limit for trial accounts. To continue receiving code reviews, upgrade your plan.

@Asherlc

Asherlc commented Jul 17, 2026

Copy link
Copy Markdown
Owner Author

Addressed the outside-diff atomicity finding from CodeRabbit review in commit 2d31b12. The import job now spools every bounded record or weight batch to a private temporary directory while the decoder protocol runs. It performs no activity upsert, sample replacement, or metric publish until metadata, end, process exit, and the final message count all validate; only then does it replay one batch of at most 250 messages at a time. A late FitDecoderError regression proves no mutation occurs and the spool is removed. The process-fit mutation score is 88.85%; focused and full unit suites pass.

@coderabbitai coderabbitai Bot left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Actionable comments posted: 6

Caution

Some comments are outside the diff and can’t be posted inline due to platform limitations.

⚠️ Outside diff range comments (3)
packages/server/src/routes/webhooks.ts (2)

196-200: 🩺 Stability & Availability | 🟠 Major | ⚡ Quick win

Set workerStartRequested before awaiting worker startup.

If startWorker() throws, workerStartRequested remains false, causing the event loop and the fallback block to redundantly attempt worker startup again for subsequent payloads.

Proposed fix
               if (provider.requiresWorkerForWebhookSync && !workerStartRequested) {
+                workerStartRequested = true;
                 const { startWorker } = await import("../lib/start-worker.ts");
                 await startWorker();
-                workerStartRequested = true;
               }
🤖 Prompt for AI Agents
Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.

In `@packages/server/src/routes/webhooks.ts` around lines 196 - 200, Update the
worker startup branch guarded by provider.requiresWorkerForWebhookSync and
!workerStartRequested to set workerStartRequested before awaiting startWorker().
Preserve the existing dynamic import and startup flow while ensuring a failed
start does not leave the flag false for subsequent payloads.

238-243: 📐 Maintainability & Code Quality | 🟡 Minor | ⚡ Quick win

Call captureException() for unexpected errors.

Per the AGENTS.md rules for **/*.ts, "Never silently swallow errors — every unexpected catch must call captureException()."

Proposed fix

Ensure captureException is imported from @sentry/node at the top of the file, then update the catch block:

         } catch (err) {
+          captureException(err);
           logger.warn(`[webhook] Failed to start worker: ${err}`);
         }
🤖 Prompt for AI Agents
Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.

In `@packages/server/src/routes/webhooks.ts` around lines 238 - 243, Update the
catch block surrounding startWorker in the webhook route to call
captureException from `@sentry/node` for the caught unexpected error, while
preserving the existing logger.warn message. Add the captureException import at
the file’s imports.

Source: Path instructions

src/jobs/process-fit-file-import-job.ts (1)

307-325: 🔒 Security & Privacy | 🟠 Major | ⚡ Quick win

Keep FIT paths out of weight metric identifiers. Garmin logical paths may contain account emails, so using originalPath directly persists PII.

  • src/jobs/process-fit-file-import-job.ts#L307-L325: derive an opaque stable path hash for the external ID.
  • src/jobs/process-fit-file-import-job.test.ts#L743-L768: use an email-bearing path and assert the generated rows contain no path or email.
🤖 Prompt for AI Agents
Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.

In `@src/jobs/process-fit-file-import-job.ts` around lines 307 - 325, The weight
external ID built in writeWeightBatch must not persist the potentially
PII-bearing data.originalPath. Derive a stable opaque hash of the original path
and use that hash in each externalId while preserving uniqueness with the
timestamp. In src/jobs/process-fit-file-import-job.test.ts lines 743-768, update
the fixture to include an email-bearing path and assert generated rows contain
neither the path nor the email.
🤖 Prompt for all review comments with AI agents
Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.

Inline comments:
In @.github/workflows/test.yml:
- Around line 591-594: Add explicit name fields to the artifact upload step
using fit-decoder-linux, the corresponding artifact download steps, and the
chmod steps in the referenced CI sections. Use concise names that identify the
artifact or permission stage so workflow failures clearly pinpoint the broken
step.

In `@native/fit-decoder/src/field-json.cpp`:
- Around line 19-20: Update IsFitTimestampField and its callers to require the
field’s declared FIT base type to be FIT_BASE_TYPE_UINT32 in addition to
matching timestamp names before invoking FitTimestampJson(). Preserve normal
serialization for developer fields with those names but different types,
avoiding name-only timestamp coercion.

In `@src/fit/AGENTS.md`:
- Around line 11-17: Add primary-source links to the claims in AGENTS.md: link
the raw-field data-retention statement to its implementation, the native
protocol testing statement to VerifyProtocol.cmake, and the adapter and
normalization testing statements to their referenced test files. Preserve the
existing claims and testing descriptions while adding only the appropriate
repository links.

In `@src/jobs/enqueue-fit-file-import.test.ts`:
- Around line 45-96: The in-process import test around
enqueueFitFileImportAndWait currently resolves immediately and cannot detect
premature cleanup. Make processFitFileImportJob return a deferred promise,
assert rm has not been called while that promise is pending, then resolve it and
verify the import result and cleanup occur afterward.

In `@src/jobs/enqueue-fit-file-import.ts`:
- Around line 65-69: Update the in-process import branch in
processFitFileImportJob to await processFitFileImportJob before the surrounding
finally cleanup executes. In src/jobs/enqueue-fit-file-import.ts lines 65-69,
use an awaited return; in src/jobs/enqueue-fit-file-import.test.ts lines 45-96,
replace the immediately resolved mock with a deferred promise and verify rm runs
only after that promise settles.

In `@src/jobs/process-fit-file-import-job.ts`:
- Around line 382-388: Require or validate that metricStreamPublisher supports
replaceRows before calling replaceMetricStreamBatch in the activity replacement
path, preserving the existing no-publisher behavior only when appropriate. In
src/jobs/process-fit-file-import-job.ts lines 382-388, enforce the
replacement-capable publisher contract before mutation; in
src/jobs/process-fit-file-import-job.test.ts lines 700-740, provide replaceRows
in the mock and test the capability contract instead of bypassing it.

---

Outside diff comments:
In `@packages/server/src/routes/webhooks.ts`:
- Around line 196-200: Update the worker startup branch guarded by
provider.requiresWorkerForWebhookSync and !workerStartRequested to set
workerStartRequested before awaiting startWorker(). Preserve the existing
dynamic import and startup flow while ensuring a failed start does not leave the
flag false for subsequent payloads.
- Around line 238-243: Update the catch block surrounding startWorker in the
webhook route to call captureException from `@sentry/node` for the caught
unexpected error, while preserving the existing logger.warn message. Add the
captureException import at the file’s imports.

In `@src/jobs/process-fit-file-import-job.ts`:
- Around line 307-325: The weight external ID built in writeWeightBatch must not
persist the potentially PII-bearing data.originalPath. Derive a stable opaque
hash of the original path and use that hash in each externalId while preserving
uniqueness with the timestamp. In src/jobs/process-fit-file-import-job.test.ts
lines 743-768, update the fixture to include an email-bearing path and assert
generated rows contain neither the path nor the email.
🪄 Autofix (Beta)

Fix all unresolved CodeRabbit comments on this PR:

  • Push a commit to this branch (recommended)
  • Create a new PR with the fixes

ℹ️ Review info
⚙️ Run configuration

Configuration used: Path: .coderabbit.yaml

Review profile: ASSERTIVE

Plan: Pro Plus

Run ID: 65ab4921-604c-468a-9014-24b3d2590737

📥 Commits

Reviewing files that changed from the base of the PR and between c34298b and 2d31b12.

📒 Files selected for processing (33)
  • .github/workflows/test.yml
  • Dockerfile
  • docs/production-incident-baseline.md
  • knip.json
  • native/fit-decoder/AGENTS.md
  • native/fit-decoder/CMakeLists.txt
  • native/fit-decoder/CMakePresets.json
  • native/fit-decoder/README.md
  • native/fit-decoder/src/field-json.cpp
  • native/fit-decoder/src/field-json.hpp
  • native/fit-decoder/src/fit-decoder.cpp
  • native/fit-decoder/tests/FieldJsonTest.cpp
  • packages/server/src/lib/start-worker.test.ts
  • packages/server/src/lib/start-worker.ts
  • packages/server/src/mcp/tools.ts
  • packages/server/src/routers/sync.ts
  • packages/server/src/routes/upload.ts
  • packages/server/src/routes/webhooks.test.ts
  • packages/server/src/routes/webhooks.ts
  • src/fit/AGENTS.md
  • src/fit/stream-decoder.test.ts
  • src/fit/stream-decoder.ts
  • src/jobs/enqueue-fit-file-import.test.ts
  • src/jobs/enqueue-fit-file-import.ts
  • src/jobs/process-fit-file-import-job.test.ts
  • src/jobs/process-fit-file-import-job.ts
  • src/providers/coros.test.ts
  • src/providers/coros.ts
  • src/providers/suunto.test.ts
  • src/providers/suunto.ts
  • src/providers/wahoo.test.ts
  • src/providers/wahoo/activity-persister.ts
  • src/providers/wahoo/provider.ts

Comment thread .github/workflows/test.yml Outdated
Comment thread native/fit-decoder/src/field-json.cpp Outdated
Comment thread src/fit/AGENTS.md Outdated
Comment thread src/jobs/enqueue-fit-file-import.test.ts
Comment thread src/jobs/enqueue-fit-file-import.ts Outdated
Comment thread src/jobs/process-fit-file-import-job.ts
Await in-process decoding before staged-file cleanup. Validate replacement publishers before mutation and close the remaining native, webhook, privacy, and CI review gaps.

@greptile-apps greptile-apps Bot left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Asherlc has reached the 50-credit limit for trial accounts. To continue receiving code reviews, upgrade your plan.

@Asherlc

Asherlc commented Jul 17, 2026

Copy link
Copy Markdown
Owner Author

Addressed all three outside-diff findings from CodeRabbit review.

  • Worker startup is marked requested before awaiting it, so multi-event webhook payloads do not retry a failed start. Both targeted-sync and fallback startup failures are reported to Sentry, with regression coverage.
  • Weight metric external IDs now use a stable SHA-256 digest of the logical FIT path. The regression uses an email-bearing path and proves neither the path nor email reaches persisted rows.
  • Activity imports validate scoped replacement capability before any activity upsert or sample mutation.

Commit: e7f309e

@greptile-apps greptile-apps Bot left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Asherlc has reached the 50-credit limit for trial accounts. To continue receiving code reviews, upgrade your plan.

@coderabbitai coderabbitai Bot left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Actionable comments posted: 3

Caution

Some comments are outside the diff and can’t be posted inline due to platform limitations.

⚠️ Outside diff range comments (2)
packages/server/src/routes/webhooks.ts (1)

163-163: 🩺 Stability & Availability | 🟠 Major | ⚡ Quick win

A failed startup attempt is incorrectly treated as a running worker.

  • packages/server/src/routes/webhooks.ts#L163-L163: replace the boolean with a cached startup Promise.
  • packages/server/src/routes/webhooks.ts#L197-L203: await that Promise before every worker-dependent targeted sync.
  • packages/server/src/routes/webhooks.ts#L239-L246: use the Promise’s presence to avoid a second startup attempt.
  • packages/server/src/routes/webhooks.test.ts#L713-L743: assert worker-dependent syncWebhookEvent is not called after startup rejects.
🤖 Prompt for AI Agents
Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.

In `@packages/server/src/routes/webhooks.ts` at line 163, Replace
workerStartRequested with a cached startup Promise in
packages/server/src/routes/webhooks.ts:163, await it before each
worker-dependent targeted sync at
packages/server/src/routes/webhooks.ts:197-203, and use its presence to prevent
duplicate startup attempts at packages/server/src/routes/webhooks.ts:239-246.
Update packages/server/src/routes/webhooks.test.ts:713-743 to assert
syncWebhookEvent is not called when startup rejects.

Sources: Coding guidelines, Path instructions

packages/server/src/routes/upload.ts (1)

108-129: 🗄️ Data Integrity & Integration | 🟠 Major | ⚡ Quick win

Worker-start failure leaves a queued import referencing a deleted file.

  • packages/server/src/routes/upload.ts#L108-L129: start the worker before adding the job, or safely remove the job before callers clean up its staged file.
  • packages/server/src/routes/upload.test.ts#L418-L428: assert queue.add is not called when startup fails.
🤖 Prompt for AI Agents
Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.

In `@packages/server/src/routes/upload.ts` around lines 108 - 129, The
enqueueImport flow in packages/server/src/routes/upload.ts lines 108-129 must
start the import worker before calling importQueue.add, so startup failures
prevent a queued job from referencing a file that callers clean up; preserve the
existing job payload and returned ID behavior after successful startup. In
packages/server/src/routes/upload.test.ts lines 418-428, add coverage asserting
queue.add is not called when startImportWorker fails.

Source: Coding guidelines

🤖 Prompt for all review comments with AI agents
Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.

Inline comments:
In `@packages/server/src/routes/upload.test.ts`:
- Around line 418-428: Extend the “returns 500 when the import worker fails to
start” test to mock or access the upload queue and assert its add method was not
called after startWorker rejects. Keep the existing 500 response and startWorker
invocation assertions, ensuring the test reproduces the orphaned-job scenario.

In `@packages/server/src/routes/webhooks.test.ts`:
- Around line 713-743: Update the test “does not retry failed worker startup for
later targeted events” to retain a reference to the provider’s syncWebhookEvent
mock and assert it was never called after mockStartWorker rejects. Ensure the
test fails if either targeted event proceeds into worker-dependent
synchronization.

In `@src/providers/wahoo/provider.ts`:
- Line 294: Update the argument reference in the Wahoo sync flow around
options.metricStreamPublisher to safely access options when it is undefined,
preserving an undefined publisher value so normal workout processing continues
without crashing.

---

Outside diff comments:
In `@packages/server/src/routes/upload.ts`:
- Around line 108-129: The enqueueImport flow in
packages/server/src/routes/upload.ts lines 108-129 must start the import worker
before calling importQueue.add, so startup failures prevent a queued job from
referencing a file that callers clean up; preserve the existing job payload and
returned ID behavior after successful startup. In
packages/server/src/routes/upload.test.ts lines 418-428, add coverage asserting
queue.add is not called when startImportWorker fails.

In `@packages/server/src/routes/webhooks.ts`:
- Line 163: Replace workerStartRequested with a cached startup Promise in
packages/server/src/routes/webhooks.ts:163, await it before each
worker-dependent targeted sync at
packages/server/src/routes/webhooks.ts:197-203, and use its presence to prevent
duplicate startup attempts at packages/server/src/routes/webhooks.ts:239-246.
Update packages/server/src/routes/webhooks.test.ts:713-743 to assert
syncWebhookEvent is not called when startup rejects.
🪄 Autofix (Beta)

Fix all unresolved CodeRabbit comments on this PR:

  • Push a commit to this branch (recommended)
  • Create a new PR with the fixes

ℹ️ Review info
⚙️ Run configuration

Configuration used: Path: .coderabbit.yaml

Review profile: ASSERTIVE

Plan: Pro Plus

Run ID: 397d9364-9f28-47b6-9cbd-ddc0086f801f

📥 Commits

Reviewing files that changed from the base of the PR and between 2d31b12 and 72a21bc.

📒 Files selected for processing (24)
  • .github/workflows/test.yml
  • native/fit-decoder/src/field-json.cpp
  • native/fit-decoder/tests/FieldJsonTest.cpp
  • packages/server/src/index.integration.test.ts
  • packages/server/src/index.test.ts
  • packages/server/src/index.ts
  • packages/server/src/router.test.ts
  • packages/server/src/router.ts
  • packages/server/src/routers/sync.test.ts
  • packages/server/src/routers/sync.ts
  • packages/server/src/routes/upload.integration.test.ts
  • packages/server/src/routes/upload.test.ts
  • packages/server/src/routes/upload.ts
  • packages/server/src/routes/webhooks.test.ts
  • packages/server/src/routes/webhooks.ts
  • packages/server/src/upload-auth.integration.test.ts
  • src/fit/AGENTS.md
  • src/jobs/enqueue-fit-file-import.test.ts
  • src/jobs/enqueue-fit-file-import.ts
  • src/jobs/process-fit-file-import-job.test.ts
  • src/jobs/process-fit-file-import-job.ts
  • src/jobs/queues.test.ts
  • src/providers/wahoo/provider.test.ts
  • src/providers/wahoo/provider.ts

Comment thread packages/server/src/routes/upload.test.ts
Comment thread packages/server/src/routes/webhooks.test.ts
Comment thread src/providers/wahoo/provider.ts

@greptile-apps greptile-apps Bot left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Asherlc has reached the 50-credit limit for trial accounts. To continue receiving code reviews, upgrade your plan.

@Asherlc

Asherlc commented Jul 17, 2026

Copy link
Copy Markdown
Owner Author

Addressed the two outside-diff findings from #1646 (review) in 73eb1130e.

  • Upload imports now await worker startup before queue.add, preventing an orphaned queued job when startup fails.
  • Webhook requests cache one worker startup Promise; every worker-dependent targeted event awaits it, and a rejected startup cannot allow later targeted sync execution or trigger another startup attempt.

Validation: 137 route tests passed, the focused async-ordering mutation run killed all 17 mutants, lint passed, and all 26 typed workspace packages passed.

@Asherlc Asherlc changed the title perf: bound FIT import memory with native streaming [native] Bound FIT import memory with native streaming Jul 17, 2026
@Asherlc
Asherlc merged commit fc5d05a into main Jul 17, 2026
120 checks passed
@Asherlc
Asherlc deleted the Asherlc/investigate-sentry-7614315645 branch July 17, 2026 14:23
@codereviewbot-ai

Copy link
Copy Markdown

🤖 Review skipped: Rate limit exceeded. Free accounts are limited to 2 reviews per 4 hours. Upgrade to a paid plan for unlimited reviews.

6 similar comments
@codereviewbot-ai

Copy link
Copy Markdown

🤖 Review skipped: Rate limit exceeded. Free accounts are limited to 2 reviews per 4 hours. Upgrade to a paid plan for unlimited reviews.

@codereviewbot-ai

Copy link
Copy Markdown

🤖 Review skipped: Rate limit exceeded. Free accounts are limited to 2 reviews per 4 hours. Upgrade to a paid plan for unlimited reviews.

@codereviewbot-ai

Copy link
Copy Markdown

🤖 Review skipped: Rate limit exceeded. Free accounts are limited to 2 reviews per 4 hours. Upgrade to a paid plan for unlimited reviews.

@codereviewbot-ai

Copy link
Copy Markdown

🤖 Review skipped: Rate limit exceeded. Free accounts are limited to 2 reviews per 4 hours. Upgrade to a paid plan for unlimited reviews.

@codereviewbot-ai

Copy link
Copy Markdown

🤖 Review skipped: Rate limit exceeded. Free accounts are limited to 2 reviews per 4 hours. Upgrade to a paid plan for unlimited reviews.

@codereviewbot-ai

Copy link
Copy Markdown

🤖 Review skipped: Rate limit exceeded. Free accounts are limited to 2 reviews per 4 hours. Upgrade to a paid plan for unlimited reviews.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants