fix(worker): copy packages/core into worker runtime image by buremba · Pull Request #1035 · lobu-ai/lobu

buremba · 2026-05-25T00:29:45Z

Problem

Every connector feed sync on the worker deployment (summaries-app-lobu-worker) crashes at runtime with:

ENOENT reading "/app/packages/connector-sdk/node_modules/@lobu/core"

Nothing gets ingested. Discovered while wiring the Spotify connector (valid OAuth, all scopes) — the sync runs fail before emitting any events.

Root cause

docker/worker/Dockerfile builds @lobu/core in the builder stage (cd packages/core && bunx tsc, line 54) but the runtime stage COPY block (lines 82-88) copies node_modules, connector-sdk, connectors, embeddings, connector-worker — never packages/core.

At feed-sync time the compiled connector emits a bare import '@lobu/connector-sdk' (the SDK is externalized by the compile pipeline). @lobu/connector-sdk/dist/index.js re-exports logger.js + retry.js, which import { createLogger, retryWithBackoff } from '@lobu/core'. bun install's workspace symlink connector-sdk/node_modules/@lobu/core → ../../../core then dangles because its target (packages/core) was never copied → ENOENT.

docker/app/Dockerfile already copies core (line 139), which is why the app deployment isn't affected.

Not a regression — long-standing latent gap: the worker runtime stage has never copied packages/core. It only surfaces now that the worker deployment is being exercised for connector sync.

Fix

One line in the runtime COPY block (mirrors the app image):

COPY --from=builder --chown=worker:worker /app/packages/core ./packages/core

Validation (red → green)

No Docker on the dev host, so reproduced the exact failure mode at the module-resolution level with the real built artifacts (Node 22), replicating the runtime image's COPY layout:

RED (no packages/core, current behavior): fs.realpathSync on connector-sdk/node_modules/@lobu/core throws ENOENT — the exact prod error.
GREEN (with packages/core + root node_modules, as the image has after the fix): the symlink resolves, Node resolves @lobu/core from connector-sdk's dist, and the module loads exposing createLogger + retryWithBackoff (the functions connector-sdk imports).

Full image build verification happens in CI (the worker image build). After this deploys, re-triggering the Spotify feeds in buremba should ingest cleanly.

🤖 Generated with Claude Code

Summary by CodeRabbit

Bug Fixes
- Fixed runtime errors occurring during feed and synchronization operations caused by missing dependencies, ensuring stable and reliable performance.

The worker Dockerfile builds @lobu/core in the builder stage but never copies packages/core into the runtime stage. bun install's workspace symlink (connector-sdk/node_modules/@lobu/core -> ../../../core) then dangles at runtime, so every connector feed sync crashes with "ENOENT reading .../connector-sdk/node_modules/@lobu/core" when @lobu/connector-sdk loads its @lobu/core-backed logger/retry helpers. Add the missing COPY, mirroring docker/app/Dockerfile which already copies packages/core.

coderabbitai · 2026-05-25T00:29:58Z

No actionable comments were generated in the recent review. 🎉

ℹ️ Recent review info

⚙️ Run configuration

Configuration used: defaults

Review profile: CHILL

Plan: Pro Plus

Run ID: 7aa864ea-616d-4e0e-a25e-42897ce81b5d

📥 Commits

Reviewing files that changed from the base of the PR and between a91baed and 2a16b84.

📒 Files selected for processing (1)

docker/worker/Dockerfile

📝 Walkthrough

Walkthrough

The Docker worker runtime image is updated to explicitly copy packages/core from the builder stage. This ensures the bun workspace symlink for @lobu/core exists in the runtime environment, preventing ENOENT errors when @lobu/connector-sdk reads this dependency during feed and sync operations.

Changes

Docker Worker Runtime Symlink Fix

Layer / File(s)	Summary
Runtime stage packages/core copy `docker/worker/Dockerfile`	Lines 85–89 add an explicit `COPY --from=builder` of `packages/core` into the runtime stage with comments explaining the prevention of a dangling bun workspace symlink for `@lobu/core`, avoiding `ENOENT` errors at runtime.

Estimated code review effort

🎯 2 (Simple) | ⏱️ ~10 minutes

Poem

🐇 A symlink once broken in Docker's deep night,
Now packages/core copies, making things right!
The connector feeds data with no more ENOENT,
A five-line fix settles what workspace can't vent. ✨

🚥 Pre-merge checks | ✅ 4 | ❌ 1

❌ Failed checks (1 warning)

Check name	Status	Explanation	Resolution
Description check	⚠️ Warning	The description provides detailed problem statement, root cause analysis, fix explanation, and validation approach, but lacks explicit Test plan section with checkboxes as specified in the template.	Add a Test plan section documenting validation steps performed (e.g., module-resolution testing, CI verification confirmation) using the template format with checkboxes.

✅ Passed checks (4 passed)

Check name	Status	Explanation
Title check	✅ Passed	The title directly and clearly describes the main fix: copying packages/core into the worker runtime image to resolve ENOENT errors during connector feed syncs.
Docstring Coverage	✅ Passed	No functions found in the changed files to evaluate docstring coverage. Skipping docstring coverage check.
Linked Issues check	✅ Passed	Check skipped because no linked issues were found for this pull request.
Out of Scope Changes check	✅ Passed	Check skipped because no linked issues were found for this pull request.

_{✏️ Tip: You can configure your own custom pre-merge checks in the settings.}

✨ Finishing Touches

🧪 Generate unit tests (beta)

Create PR with unit tests
Commit unit tests in branch fix/worker-copy-core

_{Comment @coderabbitai help to get the list of available commands and usage tips.}

codecov-commenter · 2026-05-25T00:31:05Z

⚠️ Please install the to ensure uploads and comments are reliably processed by Codecov.

Codecov Report

✅ All modified and coverable lines are covered by tests.

📢 Thoughts on this report? Let us know!

connector-worker has no direct @lobu/core dependency — it's consumed transitively by @lobu/connector-sdk (logger/retry). The self-check resolved @lobu/core from connector-worker's own location, which resolves in the hoisted dev workspace but fails in the isolated-linker worker image: a false negative (caught by the gate's own docker leg on #1037). Resolve/import @lobu/core anchored at @lobu/connector-sdk (its real consumer), reproducing the exact prod edge that dangled in #1035: .../connector-sdk/node_modules/@lobu/core.

* ci: add connector runtime parity smoke gate Guards against the packaging/parity drift that broke prod in #1035: the worker Docker image shipped missing `COPY packages/core`, so @lobu/connector-sdk's transitive @lobu/core import dangled and every connector feed sync crashed at runtime — yet all CI was green because the workspace tests resolve @lobu/core and build-images.yml builds + pushes images but never RUNS them. Add a single SHARED runtime self-check (packages/connector-worker/src/self-check) executed from BOTH the built CLI runtime (`lobu connector runtime-self-check`, hidden) and the worker image (`bun src/bin.ts self-check`). Both entrypoints call the SAME runConnectorRuntimeSelfCheck() over the same compile + default SubprocessExecutor path (the parity invariant; only the connector-source discovery roots differ). It asserts artifact/packaging facts only — @lobu/connector-sdk + @lobu/core resolve+import, EXTERNAL_RUNTIME_DEPS (playwright/sharp/jimp) resolve, the bundled connector source dir is discoverable, all discovered connectors instantiate with unique key/name/version, and a synthetic no-op connector compiles + runs through the default SubprocessExecutor. No network/DB/gateway/OAuth; passes under docker run --network=none. CI wiring: - ci.yml PR gate (path-filtered like mac-build-smoke): build worker image amd64 (load, no push), run the image self-check with --network=none, and run the CLI self-check on the host. - build-images.yml main/deploy gate: same amd64 image self-check as a job that build-worker (and transitively build-app) depends on, so a failing smoke blocks BOTH worker and app pushes — Flux can't roll the shared tag to a broken worker. Shares the buildx cache scope with build-worker. * ci: anchor @lobu/core self-check resolution at connector-sdk connector-worker has no direct @lobu/core dependency — it's consumed transitively by @lobu/connector-sdk (logger/retry). The self-check resolved @lobu/core from connector-worker's own location, which resolves in the hoisted dev workspace but fails in the isolated-linker worker image: a false negative (caught by the gate's own docker leg on #1037). Resolve/import @lobu/core anchored at @lobu/connector-sdk (its real consumer), reproducing the exact prod edge that dangled in #1035: .../connector-sdk/node_modules/@lobu/core. * ci: drop duplicate pathToFileURL import in self-check The previous commit added a pathToFileURL import that already existed (line 50), tripping TS2300 in make build-packages. pathToFileURL is already imported; remove the duplicate. * refactor(self-check): consolidate checks via async helper, trim prose Collapse the four hand-written async try/catch IIFEs into one async check() helper (records returned detail or error); fold connector discovery and key-uniqueness into the same pattern; trim duplicated narration while keeping the non-obvious rationale (SDK-anchored core resolution, cwd temp file, dynamic-import probes). No behavior change: 534 -> 413 lines, all 11 checks still pass and the dangling-@lobu/core failure path still exits 1. * ci(self-check): cwd-anchor external dep resolution; fail on unreadable connector subdir Address CodeRabbit review: - Resolve EXTERNAL_RUNTIME_DEPS from a cwd-rooted require, matching how child-runner stages and resolves the compiled bundle, so the probe fails where a real connector would instead of off connector-worker's hoisted tree. - Propagate readdir errors on connector subdirs (an unreadable subdir is a packaging defect for a parity gate) instead of silently skipping. - Document the self-check + child-runner runtime-compiled-bundle imports in the AGENTS.md dynamic-import allow-list. - Add least-privilege contents:read to the connector-parity-smoke job.

buremba enabled auto-merge (squash) May 25, 2026 00:33

buremba merged commit 3d9175c into main May 25, 2026
18 checks passed

buremba deleted the fix/worker-copy-core branch May 25, 2026 00:38

This was referenced May 24, 2026

chore(main): release lobu 9.4.0 #1031

Merged

ci: connector runtime parity smoke gate #1037

Merged

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

fix(worker): copy packages/core into worker runtime image#1035

fix(worker): copy packages/core into worker runtime image#1035
buremba merged 1 commit into
mainfrom
fix/worker-copy-core

buremba commented May 25, 2026 •

edited by coderabbitai Bot

Loading

Uh oh!

coderabbitai Bot commented May 25, 2026 •

edited

Loading

Walkthrough

Changes

Estimated code review effort

Poem

❌ Failed checks (1 warning)

Uh oh!

codecov-commenter commented May 25, 2026

Uh oh!

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

2 participants

Conversation

buremba commented May 25, 2026 • edited by coderabbitai Bot Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Problem

Root cause

Fix

Validation (red → green)

Summary by CodeRabbit

Uh oh!

coderabbitai Bot commented May 25, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Walkthrough

Changes

Estimated code review effort

Poem

❌ Failed checks (1 warning)

Uh oh!

codecov-commenter commented May 25, 2026

Codecov Report

Uh oh!

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

2 participants

buremba commented May 25, 2026 •

edited by coderabbitai Bot

Loading

coderabbitai Bot commented May 25, 2026 •

edited

Loading