Skip to content

fix(test): stop worker test from classifying the whole fork as production - #2348

Merged
Asherlc merged 11 commits into
mainfrom
posthog-code/fix-worker-test-env-leak
Jul 31, 2026
Merged

Asherlc merged 11 commits into
mainfrom
posthog-code/fix-worker-test-env-leak

Conversation

@posthog

@posthog posthog Bot commented Jul 30, 2026 •

Copy link
Copy Markdown
Contributor

Why

Error tracking was flooded with fake exceptions — every ingested $exception was a unit-test fixture string (test failure, test worker error, invalid FIT, …) from a local vitest run, not a real failure. That noise buries genuine production exceptions and each fixture spawns its own "new issue" alert.

Root cause

src/jobs/worker.test.ts set DEPLOY_ENVIRONMENT="prod" and SENTRY_DSN directly on process.env inside vi.hoisted() and never tore them down. The "prod" classification persisted for the whole vitest fork and defeated the production-only guard in initProductionSentry() (src/lib/sentry.ts) for every module that ran in that fork. @sentry/node is mocked in the test, so Sentry stayed clean — but any un-mocked telemetry sink wired to the same guard would ship the fixture errors as real production events. This is a recurrence of the failure mode fixed for Sentry in #1882 (commit 7eba92d).

Fix

  • Replace the raw process.env assignments with tracked vi.stubEnv calls, restored via vi.unstubAllEnvs() in an afterAll, so the classification can't leak across files in a reused fork.
  • Add an explicit non-production DEPLOY_ENVIRONMENT: "test" default to the shared test env in vitest.config.ts as a config-level safety net — no test run is classified as a production deployment unless it opts in.
  • Append a recurrence entry to docs/production-incident-baseline.md.

No production code changed; no NODE_ENV === "test" branch added.

Scope note

The originally-observed events came from a posthog-node exporter, which is not on main (the reporter noted it lives in an uncommitted local workspace). This PR fixes the root cause present on main — the env mutation that tricks the production guard — rather than mocking a module that doesn't exist yet. The baseline entry records that mocking such an exporter in worker.test.ts is the follow-up when it lands.

Test plan

  • src/jobs/worker.test.ts, src/lib/sentry.test.ts, packages/server/src/lib/sentry.test.ts pass.
  • Full Docker-free unit tier run: the only failures are pre-existing and unrelated (packages/web/src/routes/training/cycling.test.tsx, a pagination assertion — confirmed failing identically on clean main).
  • pnpm biome check and pnpm tsc --noEmit (root) pass.

Created with PostHog Desktop from this inbox report.


Summary by cubic

Prevents local Vitest runs from being treated as production and stabilizes builds across web, server, mobile, and e2e.

  • Bug Fixes

    • Worker test: replace raw process.env writes with vi.stubEnv; in afterAll restore only DEPLOY_ENVIRONMENT and SENTRY_DSN via targeted vi.stubEnv (not vi.unstubAllEnvs). Set a non-prod DEPLOY_ENVIRONMENT: "test" default in vitest.config.ts. Documented in docs/production-incident-baseline.md with Vitest links; added citations to @sentry/node and posthog-node; kept both 2026-07-30 entries.
    • Reports: export dofek-server/report-empty-state; adopt shared empty-state fixtures; align web components to WeeklyReportData/MonthlyReportData; remove recovery fields; delete the unused report-data fixture that broke Knip, coverage, and Stryker.
    • Mobile sync: keep “Sync All” disabled while another provider is polling and always pass ROUTINE_SYNC_DAYS; update tests.
    • CI/build: lowercase GHCR cache refs by computing a lowercase image name in .github/workflows/test.yml; retain iOS HealthKit lazy-init; align provider timeout expectations; pass endDate in monthly report tests.
  • Dependencies

    • Use MISE_LOCKED=1 for installs instead of mise trust.
    • Restore Python 3.13 pin for dbt tooling in Docker builds.

Written for commit 1d65cc4. Summary will update on new commits.

Review in cubic

Summary by Sourcery

Prevent test environment configuration from causing local Vitest runs to be classified as production deployments.

Bug Fixes:

  • Ensure worker tests stub and restore DEPLOY_ENVIRONMENT and SENTRY_DSN instead of mutating process.env directly to avoid leaking production classification across test files.

Enhancements:

  • Set a non-production DEPLOY_ENVIRONMENT="test" default in shared Vitest config so tests must explicitly opt into production paths.
  • Document the incident recurrence, root cause, and mitigations in the production incident baseline.

…tion

worker.test.ts assigned DEPLOY_ENVIRONMENT="prod" and SENTRY_DSN directly on
process.env inside vi.hoisted() and never restored them. The "prod"
classification persisted for the entire vitest fork, defeating the
production-only guard in initProductionSentry() for every module in that fork —
the same failure mode 7eba92d (#1882) fixed for Sentry. Any un-mocked telemetry
sink wired to that guard would then ship local test-fixture errors to
production error tracking.

- Use tracked vi.stubEnv and restore with vi.unstubAllEnvs() in afterAll so the
  classification cannot outlive the test file within a reused fork.
- Add an explicit non-production DEPLOY_ENVIRONMENT="test" default to the shared
  test env so no test run is ever classified as a production deployment unless
  it opts in.
- Record the recurrence in docs/production-incident-baseline.md.

Generated-By: PostHog Code
Task-Id: 0fd126a1-18dd-4b16-a4cc-a990b91ce4aa
@codereviewbot-ai

Copy link
Copy Markdown

🤖 Review skipped: Repository rate limit exceeded. Free accounts are limited to 2 reviews per 4 hours per repository. Upgrade to a paid plan for unlimited reviews.

@sourcery-ai

sourcery-ai Bot commented Jul 30, 2026 •

Copy link
Copy Markdown
Contributor

Reviewer's Guide

This PR fixes test-induced production classification leaks by converting worker test env mutations to tracked Vitest env stubs, setting a non-production default deployment environment for all tests, and documenting the incident and mitigation in the production incident baseline.

Sequence diagram for vitest worker test env stubbing and production guard

sequenceDiagram
  actor Developer
  participant VitestRunner
  participant worker_test_ts
  participant worker_ts
  participant initProductionSentry
  participant TelemetrySink

  Developer->>VitestRunner: run worker.test.ts
  VitestRunner->>worker_test_ts: execute vi.hoisted
  worker_test_ts->>vi: stubEnv DEPLOY_ENVIRONMENT "prod"
  worker_test_ts->>vi: stubEnv SENTRY_DSN "https://test@sentry.io/123"
  VitestRunner->>worker_ts: import worker.ts
  worker_ts->>initProductionSentry: initProductionSentry
  initProductionSentry->>initProductionSentry: [DEPLOY_ENVIRONMENT == "prod"]
  initProductionSentry-->>TelemetrySink: initialize production client

  VitestRunner-->>worker_test_ts: run tests

  VitestRunner->>worker_test_ts: run afterAll callback
  worker_test_ts->>vi: unstubAllEnvs
  initProductionSentry-->>initProductionSentry: [DEPLOY_ENVIRONMENT == "test"]
  initProductionSentry-->>TelemetrySink: skip production client initialization
Loading

File-Level Changes

Change Details Files
Prevent worker test from leaking production-like env configuration across Vitest forks.
  • Replace direct process.env assignments for DEPLOY_ENVIRONMENT and SENTRY_DSN inside vi.hoisted with vi.stubEnv calls so changes are tracked by Vitest.
  • Import afterAll from Vitest and use it to call vi.unstubAllEnvs after the test file completes, ensuring all env stubs are restored and the prod classification does not leak to other tests.
  • Add explanatory comments describing why env stubbing is required and how previous behavior allowed test fixtures to reach production-like telemetry sinks.
src/jobs/worker.test.ts
Harden test configuration to default all Vitest runs to a non-production deployment environment.
  • Extend the sharedTestEnv configuration with DEPLOY_ENVIRONMENT: "test" so test runs are classified as non-production unless they explicitly opt into production behavior.
  • Document in comments that tests must explicitly opt in to production paths via vi.stubEnv and restore env afterward to avoid leaking prod classification.
vitest.config.ts
Document the recurrence and mitigation of the test env mutation incident in the production incident baseline.
  • Append a new incident entry describing symptoms, evidence, root cause, mitigation, validation, and remaining risk related to test env mutation re-enabling production classification for local runs.
  • Clarify that future PostHog exporter additions should be mocked in worker tests to prevent real telemetry emission during tests regardless of env classification.
docs/production-incident-baseline.md

Tips and commands

Interacting with Sourcery

  • Trigger a new review: Comment @sourcery-ai review on the pull request.
  • Continue discussions: Reply directly to Sourcery's review comments.
  • Generate a GitHub issue from a review comment: Ask Sourcery to create an
    issue from a review comment by replying to it. You can also reply to a
    review comment with @sourcery-ai issue to create an issue from it.
  • Generate a pull request title: Write @sourcery-ai anywhere in the pull
    request title to generate a title at any time. You can also comment
    @sourcery-ai title on the pull request to (re-)generate the title at any time.
  • Generate a pull request summary: Write @sourcery-ai summary anywhere in
    the pull request body to generate a PR summary at any time exactly where you
    want it. You can also comment @sourcery-ai summary on the pull request to
    (re-)generate the summary at any time.
  • Generate reviewer's guide: Comment @sourcery-ai guide on the pull
    request to (re-)generate the reviewer's guide at any time.
  • Resolve all Sourcery comments: Comment @sourcery-ai resolve on the
    pull request to resolve all Sourcery comments. Useful if you've already
    addressed all the comments and don't want to see them anymore.
  • Dismiss all Sourcery reviews: Comment @sourcery-ai dismiss on the pull
    request to dismiss all existing Sourcery reviews. Especially useful if you
    want to start fresh with a new review - don't forget to comment
    @sourcery-ai review to trigger a new review!

Customizing Your Experience

Access your dashboard to:

  • Enable or disable review features such as the Sourcery-generated pull request
    summary, the reviewer's guide, and others.
  • Change the review language.
  • Add, remove or edit custom review instructions.
  • Adjust other review settings.

Getting Help

@github-actions

github-actions Bot commented Jul 30, 2026 •

Copy link
Copy Markdown
Contributor

Storybook previews for afb3b323 are ready:

This comment updates automatically on each PR push.

@Asherlc
Asherlc marked this pull request as ready for review July 31, 2026 00:40
Copilot AI review requested due to automatic review settings July 31, 2026 00:40

Copilot AI left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Copilot was unable to review this pull request because the user who requested the review has reached their quota limit.

@qodo-code-review

Copy link
Copy Markdown

PR Summary by Qodo

Prevent vitest env leak from marking worker tests as production

🐞 Bug fix 🧪 Tests ⚙️ Configuration changes 📝 Documentation 🕐 20-40 Minutes

Grey Divider

AI Description

• Prevent worker test from leaking prod env/DSN across reused Vitest forks.
• Default DEPLOY_ENVIRONMENT to "test" in shared Vitest config as a safety net.
• Document the recurrence, symptoms, and mitigation in the incident baseline.
Diagram

graph TD
  C["vitest.config.ts"] -->|"DEPLOY_ENVIRONMENT=test"| F["Vitest fork env"] --> T["src/jobs/worker.test.ts"] -->|"stubEnv prod + DSN"| W["src/jobs/worker.ts import"] --> S["initProductionSentry()"] --> D{"prod?"} -->|"yes"| E(["Telemetry sink"])
  T --> U["afterAll: unstubAllEnvs()"] --> F
Loading
High-Level Assessment

The following are alternative approaches to this PR:

1. Manual save/restore of process.env keys
  • ➕ No dependency on Vitest helpers; works in any test runner
  • ➕ Explicitly scopes restoration to a known key set
  • ➖ Easy to miss keys or forget teardown (the original failure mode)
  • ➖ More boilerplate and less standardized than vi.stubEnv/unstubAllEnvs
2. Avoid prod-path tests by injecting Sentry init dependencies
  • ➕ Eliminates need to spoof production classification at all
  • ➕ Reduces reliance on global env in tests
  • ➖ Requires refactoring production code to add injection seams
  • ➖ Higher change surface than necessary for this regression fix
3. Force isolated test execution (single fork / no reuse)
  • ➕ Prevents cross-file environment leakage by construction
  • ➖ Slows the suite and changes test runner performance characteristics
  • ➖ Still allows leakage within a fork; treats symptoms rather than fixing the cause

Recommendation: The chosen approach (vi.stubEnv + vi.unstubAllEnvs in the test, plus a config-level non-prod default) is the best trade-off: it fixes the root cause (untracked env mutation), makes the intended prod-path opt-in explicit, and adds a defense-in-depth default to prevent future accidental production classification during tests without requiring production-code refactors.

Files changed (3) +68 / -3

Tests (1) +16 / -3
worker.test.tsStub and reliably restore DEPLOY_ENVIRONMENT/SENTRY_DSN in hoisted setup +16/-3

Stub and reliably restore DEPLOY_ENVIRONMENT/SENTRY_DSN in hoisted setup

• Replaces raw process.env assignments inside vi.hoisted() with vi.stubEnv calls. Adds an afterAll hook calling vi.unstubAllEnvs() so the prod classification and DSN cannot leak into other test modules sharing the same Vitest fork.

src/jobs/worker.test.ts

Documentation (1) +44 / -0
production-incident-baseline.mdRecord recurrence: test env leak re-enabled production classification +44/-0

Record recurrence: test env leak re-enabled production classification

• Adds a new incident baseline entry describing symptoms, root cause (env mutation leaking across Vitest forks), mitigation, and follow-up risk. Serves as institutional memory to prevent reintroducing the same failure mode.

docs/production-incident-baseline.md

Other (1) +8 / -0
vitest.config.tsDefault shared test env to non-production DEPLOY_ENVIRONMENT +8/-0

Default shared test env to non-production DEPLOY_ENVIRONMENT

• Adds DEPLOY_ENVIRONMENT: "test" to the shared Vitest env configuration. This prevents any test run from being classified as production unless a test explicitly opts in and cleans up its stubs.

vitest.config.ts

@qodo-code-review

qodo-code-review Bot commented Jul 31, 2026 •

Copy link
Copy Markdown

Code Review by Qodo

🐞 Bugs (0) 📘 Rule violations (0) 📜 Skill insights (0)

Context used
✅ Compliance rules (platform): 162 rules

Grey Divider


Remediation recommended

1. Docs lack third-party citations ✓ Resolved 📘 Rule violation § Compliance
Description
The new incident baseline entry makes claims about Vitest/runtime env behavior and @sentry/node
mocking without adjacent links to primary/official sources. This violates the requirement to cite
third-party behavior claims in docs, reducing auditability and risking propagation of incorrect
assumptions.
Code

docs/production-incident-baseline.md[R21131-21139]

+- **Evidence:** `src/jobs/worker.test.ts` assigned
+  `process.env.DEPLOY_ENVIRONMENT = "prod"` and `process.env.SENTRY_DSN`
+  directly inside `vi.hoisted()`. Those raw assignments were never torn down,
+  so the "prod" classification persisted for the whole vitest fork and defeated
+  the production-only guard in `initProductionSentry()`
+  ([src/lib/sentry.ts](../src/lib/sentry.ts)) for every module that ran in the
+  same fork. `@sentry/node` is mocked in that test, so Sentry stayed clean, but
+  any un-mocked telemetry sink wired to the same guard emitted the fixture
+  errors as real production events.
Relevance

●●● Strong

Similar citation-missing doc findings repeatedly accepted in baseline/docs (e.g., PRs #2047, #2326,
#2237).

PR-#2047
PR-#2326
PR-#2237

ⓘ Recommendations generated based on similar findings in past PRs

Evidence
The added documentation describes behavior of third-party tooling (Vitest and Sentry SDK
behavior/interaction) but provides no adjacent links to official documentation; only internal code
links and an internal PR link are present, which do not satisfy the primary-source citation
requirement.

Rule 1505719: Cite third-party behavior claims in docs with primary sources
docs/production-incident-baseline.md[21131-21139]

Agent prompt
The issue below was found during a code review. Follow the provided context and guidance below and implement a solution

## Issue description
`docs/production-incident-baseline.md` adds third-party behavior claims (Vitest fork/env behavior and `@sentry/node` mocking/telemetry implications) without adjacent primary-source citations.

## Issue Context
Compliance requires that documentation under `docs/` includes hyperlinks to official/primary sources when asserting third-party SDK/runtime behavior.

## Fix Focus Areas
- docs/production-incident-baseline.md[21121-21159]

ⓘ Copy this prompt and use it to remediate the issue with your preferred AI generation tools


Grey Divider

To customize comments, go to the Qodo configuration screen, or learn more in the docs.

Qodo Logo

Comment thread docs/production-incident-baseline.md Outdated

@sourcery-ai sourcery-ai Bot left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Hey - I've found 1 issue, and left some high level feedback:

  • In worker.test.ts, consider calling vi.unstubEnv for the specific keys (DEPLOY_ENVIRONMENT, SENTRY_DSN) instead of vi.unstubAllEnvs() so future tests in this file can safely use other env stubs without having them implicitly cleared.
Prompt for AI Agents
Please address the comments from this code review:

## Overall Comments
- In `worker.test.ts`, consider calling `vi.unstubEnv` for the specific keys (`DEPLOY_ENVIRONMENT`, `SENTRY_DSN`) instead of `vi.unstubAllEnvs()` so future tests in this file can safely use other env stubs without having them implicitly cleared.

## Individual Comments

### Comment 1
<location path="docs/production-incident-baseline.md" line_range="21137-21138" />
<code_context>
+  the production-only guard in `initProductionSentry()`
+  ([src/lib/sentry.ts](../src/lib/sentry.ts)) for every module that ran in the
+  same fork. `@sentry/node` is mocked in that test, so Sentry stayed clean, but
+  any un-mocked telemetry sink wired to the same guard emitted the fixture
+  errors as real production events.
+- **Root cause:** A recurrence of the failure mode fixed by commit 7eba92d
</code_context>
<issue_to_address>
**nitpick (typo):** Consider changing "un-mocked" to the more standard "unmocked".

This matches common technical terminology and reads more smoothly.

```suggestion
  same fork. `@sentry/node` is mocked in that test, so Sentry stayed clean, but
  any unmocked telemetry sink wired to the same guard emitted the fixture
```
</issue_to_address>

Sourcery is free for open source - if you like our reviews please consider sharing them ✨
Help me be more useful! Please click 👍 or 👎 on each comment and I'll use the feedback to improve your reviews.

Comment thread docs/production-incident-baseline.md Outdated
Asherlc and others added 2 commits July 30, 2026 18:20
Bring the branch up to date with main and fix downstream breakages from the
report empty-state work and provider sync refactor that were failing lint,
typecheck, unit, and mobile tests in CI.

Co-authored-by: Cursor <cursoragent@cursor.com>
@codereviewbot-ai

Copy link
Copy Markdown

🤖 Review skipped: Repository rate limit exceeded. Free accounts are limited to 2 reviews per 4 hours per repository. Upgrade to a paid plan for unlimited reviews.

@Asherlc
Asherlc enabled auto-merge (squash) July 31, 2026 01:25
@github-actions

github-actions Bot commented Jul 31, 2026 •

Copy link
Copy Markdown
Contributor

Mobile Preview

Scan to open on device:

QR code for dofek://preview/pr-2348

Channel pr-2348
Deep Link dofek://preview/pr-2348
Commit afb3b32

To test on device:

  1. Build and install the preview client: PREVIEW_CHANNEL=pr-2348 pnpm expo prebuild --clean -p ios
  2. Or tap deep link on an existing preview build: dofek://preview/pr-2348

Each PR gets its own channel. Build a preview client with PREVIEW_CHANNEL=pr-{N} to test.

Replace mise trust with explicit MISE_LOCKED=1 on install commands so
pinned tool versions are enforced without relying on mise.toml settings.

Co-authored-by: Cursor <cursoragent@cursor.com>
@codereviewbot-ai

Copy link
Copy Markdown

🤖 Review skipped: Repository rate limit exceeded. Free accounts are limited to 2 reviews per 4 hours per repository. Upgrade to a paid plan for unlimited reviews.

Align withings timeout expectations with ProviderRequestTimeoutError, pass endDate in monthly report integration tests, lowercase GHCR cache refs for fork repos, and defer HealthKit observer coordinator init to avoid self capture in Swift.

Co-authored-by: Cursor <cursoragent@cursor.com>
@codereviewbot-ai

Copy link
Copy Markdown

🤖 Review skipped: Repository rate limit exceeded. Free accounts are limited to 2 reviews per 4 hours per repository. Upgrade to a paid plan for unlimited reviews.

Bring in main's Dockerfile Python pin, HealthKit lazy-init, report schema
updates, and workflow fixes while keeping the worker test env leak fix and
the mobile polling regression test.

Co-authored-by: Cursor <cursoragent@cursor.com>
@codereviewbot-ai

Copy link
Copy Markdown

🤖 Review skipped: Repository rate limit exceeded. Free accounts are limited to 2 reviews per 4 hours per repository. Upgrade to a paid plan for unlimited reviews.

Remove recovery fields from component stories and tests since web UI
consumes repository data shapes, not router results. Delete the unused
report-data fixture that was breaking Knip, coverage, and Stryker.

Co-authored-by: Cursor <cursoragent@cursor.com>
@codereviewbot-ai

Copy link
Copy Markdown

🤖 Review skipped: Repository rate limit exceeded. Free accounts are limited to 2 reviews per 4 hours per repository. Upgrade to a paid plan for unlimited reviews.

Add primary-source links for vi.hoisted, fork worker env sharing, vi.stubEnv,
vi.unstubAllEnvs, and module mocking per review feedback. Also use "unmocked".

Co-authored-by: Cursor <cursoragent@cursor.com>
@codereviewbot-ai

Copy link
Copy Markdown

🤖 Review skipped: Repository rate limit exceeded. Free accounts are limited to 2 reviews per 4 hours per repository. Upgrade to a paid plan for unlimited reviews.

Keep both 2026-07-30 incident entries and adopt shared empty-state fixtures from main.

Co-authored-by: Cursor <cursoragent@cursor.com>
@codereviewbot-ai

Copy link
Copy Markdown

🤖 Review skipped: Repository rate limit exceeded. Free accounts are limited to 2 reviews per 4 hours per repository. Upgrade to a paid plan for unlimited reviews.

Use targeted vi.stubEnv calls instead of vi.unstubAllEnvs so future tests
in worker.test.ts can stub other env vars without implicit teardown.

Co-authored-by: Cursor <cursoragent@cursor.com>
@codereviewbot-ai

Copy link
Copy Markdown

🤖 Review skipped: Repository rate limit exceeded. Free accounts are limited to 2 reviews per 4 hours per repository. Upgrade to a paid plan for unlimited reviews.

Add official SDK documentation links for @sentry/node and posthog-node
claims to satisfy third-party citation compliance in the incident baseline.

Co-authored-by: Cursor <cursoragent@cursor.com>
@codereviewbot-ai

Copy link
Copy Markdown

🤖 Review skipped: Repository rate limit exceeded. Free accounts are limited to 2 reviews per 4 hours per repository. Upgrade to a paid plan for unlimited reviews.

@Asherlc
Asherlc merged commit f8e4033 into main Jul 31, 2026
99 checks passed
@Asherlc
Asherlc deleted the posthog-code/fix-worker-test-env-leak branch July 31, 2026 20:57
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants