Skip to content
Merged
Show file tree
Hide file tree
Changes from 1 commit
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
44 changes: 44 additions & 0 deletions docs/production-incident-baseline.md
Original file line number Diff line number Diff line change
Expand Up @@ -21113,3 +21113,47 @@ Drizzle schema and runtime Zod schemas. Findings and remediations:
- **Remaining risk / follow-up:** Group coupled CodeQL and Slack dependencies
in Dependabot configuration so future updates arrive atomically. Continue to
require a native Release build for Sentry React Native updates.

## 2026-07-30 — Test Env Mutation Re-Enabled Production Classification for Local Runs

- **Status:** Recurrence fix implemented in test/config paths; production
unaffected.
- **Symptoms:** Every exception ingested into error tracking over the preceding
days was a unit-test fixture string (for example `test failure`,
`test worker error`, `invalid FIT`, `unrecoverable sync error`,
`Redis job log unavailable`) rather than a real failure. The events clustered
in a single short window, all attributed to the `dofek-server` distinct id,
with stack frames pointing at `@vitest/runner` and a local Conductor
workspace path.
- **User impact:** No production user impact. The blast radius is the error
tracking surface itself: fixture noise buries genuine production exceptions
and each distinct fixture spawns its own "new issue" alert.
- **Evidence:** `src/jobs/worker.test.ts` assigned
`process.env.DEPLOY_ENVIRONMENT = "prod"` and `process.env.SENTRY_DSN`
directly inside `vi.hoisted()`. Those raw assignments were never torn down,
so the "prod" classification persisted for the whole vitest fork and defeated
the production-only guard in `initProductionSentry()`
([src/lib/sentry.ts](../src/lib/sentry.ts)) for every module that ran in the
same fork. `@sentry/node` is mocked in that test, so Sentry stayed clean, but
any un-mocked telemetry sink wired to the same guard emitted the fixture
Comment thread
Asherlc marked this conversation as resolved.
Resolved
errors as real production events.
Comment thread
qodo-code-review[bot] marked this conversation as resolved.
Resolved
- **Root cause:** A recurrence of the failure mode fixed by commit 7eba92d
("Prevent local errors from reaching production Sentry",
[#1882](https://github.com/Asherlc/dofek/pull/1882)), which added the
`DEPLOY_ENVIRONMENT`-gated guard. The worker test tricked that guard back
open by mutating the shared process environment and never restoring it.
- **Fix / mitigation:** Replace the raw `process.env` assignments with tracked
`vi.stubEnv` calls and restore them with `vi.unstubAllEnvs()` in an
`afterAll`, so the "prod" classification cannot outlive the test file within
a reused fork. Add an explicit non-production `DEPLOY_ENVIRONMENT: "test"`
default to the shared test env in
[vitest.config.ts](../vitest.config.ts) so no test run is classified as a
production deployment unless it opts in. No production code was changed and
no `NODE_ENV === "test"` branch was added.
- **Validation:** `src/jobs/worker.test.ts` and both `sentry.test.ts` suites
pass, the full Docker-free unit tier passes, and lint and TypeScript checks
pass.
- **Remaining risk / follow-up:** When a PostHog (`posthog-node`) error
exporter lands on `main`, mock it in `worker.test.ts` the same way
`@sentry/node` is mocked so no test run can emit real telemetry regardless of
environment classification.
19 changes: 16 additions & 3 deletions src/jobs/worker.test.ts
Original file line number Diff line number Diff line change
@@ -1,12 +1,19 @@
import { describe, expect, it, vi } from "vitest";
import { afterAll, describe, expect, it, vi } from "vitest";

// All mock dependencies live inside vi.hoisted() so they are guaranteed to exist
// before vi.mock() factories resolve and before the static import of worker.ts.
// This satisfies vitest's "related" mode used by Stryker in CI, which requires a
// static import dependency between the test and the source module.
const hoisted = vi.hoisted(() => {
process.env.DEPLOY_ENVIRONMENT = "prod";
process.env.SENTRY_DSN = "https://test@sentry.io/123";
// Classify this fork as a production deployment so worker.ts initializes the
// production Sentry client on import. Use vi.stubEnv (not a raw process.env
// assignment) so the change is tracked and torn down after this file runs —
// otherwise the "prod" classification leaks into every other test module in
// the same vitest fork and defeats the production-only guard in
// initProductionSentry(), which is how local test-fixture errors previously
// reached production error tracking.
vi.stubEnv("DEPLOY_ENVIRONMENT", "prod");
vi.stubEnv("SENTRY_DSN", "https://test@sentry.io/123");

function noOpExit(): never {
throw new Error("process.exit called unexpectedly in test");
Expand Down Expand Up @@ -282,6 +289,12 @@ const {
// test file as related to worker.ts, so mutations in worker.ts are covered.
import "./worker.ts";

// Restore the deployment-environment/DSN stubs so the "prod" classification does
// not outlive this file within a reused vitest fork.
afterAll(() => {
vi.unstubAllEnvs();
});

describe("worker module", () => {
// 2 per-provider workers (strava, garmin) + 1 legacy sync + 1 import + 1 FIT import
// + 1 FIT batch + 1 ZIP extract + 1 export + 1 scheduled-sync + 1 post-sync
Expand Down
8 changes: 8 additions & 0 deletions vitest.config.ts
Original file line number Diff line number Diff line change
Expand Up @@ -45,6 +45,14 @@ const sharedTestEnv = {
CREDENTIAL_ENCRYPTION_KEY_NAMESPACE: "dofek-test",
CREDENTIAL_ENCRYPTION_KEY_NAME: "provider-credentials-test",
PUBLIC_URL: "https://app.example.test",
// Classify the test environment as non-production by default so no test run is
// ever treated as a production deployment. The production-only guard in
// initProductionSentry() keys off DEPLOY_ENVIRONMENT; an explicit non-prod
// default keeps a stray env mutation in one test from letting local
// test-fixture errors reach production error tracking. Tests that need to
// exercise the production path must opt in explicitly via vi.stubEnv and
// restore it afterwards.
DEPLOY_ENVIRONMENT: "test",
};

const configuredClickHouseUrl = process.env.CLICKHOUSE_URL?.trim();
Expand Down
Loading