fix(sync): wire embedded feed-sync executor + workers/poll RangeError#929
Conversation
Two bugs blocked the headless `lobu run -> lobu apply -> trigger_feed -> events appear` data sync loop on a fresh install. Both confirmed via local repro against the embedded server. Bug A: embedded mode never executed runs(run_type='sync'). The gateway booted but no connector-worker ever called /api/workers/poll, so feed- sync rows sat in `pending` forever. New module wires the existing `WorkerDaemon` in-process, started after `listen()` so its boot-time health check can resolve. Opt-out via LOBU_DISABLE_EMBEDDED_WORKER=1. Atomic claim already lives in worker-api.ts (FOR UPDATE OF r SKIP LOCKED + claimed_by) so embedded + external fleet workers co-exist. Bug B: /api/workers/poll 500'd with `RangeError: init["status"] must be in the range of 200 to 599` whenever a valid Better-Auth session token bearer hit it. Root cause: `MultiTenantProvider.resolveAuth`'s `setContextAndContinue` helper returned `next()` (which can be either Hono's plain `Next` or a wrapped cb that may return a Response), but every caller did `await setContextAndContinue(...); return undefined;` - discarding the cb's Response. The workers/* gating middleware's "Worker token missing device_worker:run scope" 403 was silently dropped, Hono never saw it, and a downstream `c.header()` re-wrap on the half-initialized response collapsed via `new Response(c.body, c)` with the Hono Context as init -> init.status = c.status (a function) -> RangeError. Switch every caller to `return setContextAndContinue(...)`; widen the helper's return type and the WorkspaceProvider.resolveAuth `next` param (new `ResolveAuthNext`) so the Response now propagates. E2E reproducer (PGlite, port 8802): - Pre-fix: trigger_feed -> run sits forever; poll with session bearer -> 500. - Post-fix: trigger_feed -> run completes (336 events for HN openai search); poll -> 200/403 depending on auth, zero RangeErrors. Closes the install_operator headless sync loop on a fresh install.
📝 WalkthroughWalkthroughThis PR fixes two issues blocking embedded ChangesEmbedded sync-loop and auth response fixes
Estimated code review effort🎯 3 (Moderate) | ⏱️ ~25 minutes Possibly related PRs
Suggested labels
Poem
🚥 Pre-merge checks | ✅ 4 | ❌ 1❌ Failed checks (1 warning)
✅ Passed checks (4 passed)
✏️ Tip: You can configure your own custom pre-merge checks in the settings. ✨ Finishing Touches📝 Generate docstrings
🧪 Generate unit tests (beta)
Warning There were issues while running some tools. Please review the errors and either fix the tool's configuration or disable the tool if it's a critical failure. 🔧 ESLint
ESLint skipped: no ESLint configuration detected in root package.json. To enable, add Comment |
|
Codecov Report✅ All modified and coverable lines are covered by tests. 📢 Thoughts on this report? Let us know! |
![coderabbitai[bot]](https://avatars.githubusercontent.com/in/347564?s=60&v=4){kind=small}
There was a problem hiding this comment.
Actionable comments posted: 2
🤖 Prompt for all review comments with AI agents
Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.
Inline comments:
In `@docs/fix-sync-loop-design.md`:
- Around line 33-40: Update the fenced code block that begins with "RangeError
at undici/initializeResponse" to include a language identifier (use "text")
after the opening backticks so the block becomes ```text; this fixes
markdownlint MD040 and prevents docs lint failures for the stack-trace block in
the document.
In `@packages/server/src/start-local.ts`:
- Around line 287-290: The code calls startEmbeddedConnectorWorker(env,
`http://${HOST}:${PORT}`) using HOST directly which breaks when HOST is 0.0.0.0
or an IPv6 literal; implement a small normalizeHost function that maps "0.0.0.0"
-> "127.0.0.1" and "::" -> "::1", detects IPv6 addresses and wraps them in
brackets (e.g., "[::1]") for URL formatting, then call
startEmbeddedConnectorWorker(env, `http://${normalizedHost}:${PORT}`) so the
embedded worker receives a dialable, correctly-formatted host string.
🪄 Autofix (Beta)
Fix all unresolved CodeRabbit comments on this PR:
- Push a commit to this branch (recommended)
- Create a new PR with the fixes
ℹ️ Review info
⚙️ Run configuration
Configuration used: defaults
Review profile: CHILL
Plan: Pro Plus
Run ID: b5fb477b-9d25-49ca-90de-18d1a4ae55ab
📒 Files selected for processing (7)
docs/fix-sync-loop-design.mdpackages/server/src/auth/middleware.tspackages/server/src/scheduled/embedded-connector-worker.tspackages/server/src/server.tspackages/server/src/start-local.tspackages/server/src/workspace/multi-tenant.tspackages/server/src/workspace/types.ts
| ``` | ||
| RangeError at undici/initializeResponse | ||
| new Response(body, init) | ||
| at [getResponseCache] (@hono/node-server) | ||
| at get headers (@hono/node-server) | ||
| at set res (hono/context.js:133) | ||
| at dispatch (hono/compose.js:38) | ||
| ``` |
There was a problem hiding this comment.
Add a language identifier to the fenced stack-trace block.
This currently triggers markdownlint MD040 and can fail docs linting pipelines.
Suggested fix
-```
+```text
RangeError at undici/initializeResponse
new Response(body, init)
at [getResponseCache] (`@hono/node-server`)
at get headers (`@hono/node-server`)
at set res (hono/context.js:133)
at dispatch (hono/compose.js:38)</details>
<details>
<summary>🧰 Tools</summary>
<details>
<summary>🪛 markdownlint-cli2 (0.22.1)</summary>
[warning] 33-33: Fenced code blocks should have a language specified
(MD040, fenced-code-language)
</details>
</details>
<details>
<summary>🤖 Prompt for AI Agents</summary>
Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.
In @docs/fix-sync-loop-design.md around lines 33 - 40, Update the fenced code
block that begins with "RangeError at undici/initializeResponse" to include a
language identifier (use "text") after the opening backticks so the block
becomes ```text; this fixes markdownlint MD040 and prevents docs lint failures
for the stack-trace block in the document.
</details>
<!-- fingerprinting:phantom:triton:hawk -->
<!-- This is an auto-generated comment by CodeRabbit -->
| // Embedded daemon must wait for the listener — its boot-time | ||
| // health check hits `/api/health` on this same process. | ||
| embeddedWorker = startEmbeddedConnectorWorker(env, `http://${HOST}:${PORT}`); | ||
| }); |
There was a problem hiding this comment.
🧩 Analysis chain
🏁 Script executed:
#!/bin/bash
# Verify all embedded-worker startup URL constructions and host normalization paths.
rg -n "startEmbeddedConnectorWorker\\(|http://\\$\\{HOST\\}:\\$\\{PORT\\}|daemonHost" \
packages/server/src/start-local.ts \
packages/server/src/server.tsRepository: lobu-ai/lobu
Length of output: 495
🏁 Script executed:
# Get full context around the problematic line in start-local.ts
cat -n packages/server/src/start-local.ts | sed -n '280,295p'Repository: lobu-ai/lobu
Length of output: 712
🏁 Script executed:
# Get full context around the comparison code in server.ts
cat -n packages/server/src/server.ts | sed -n '320,335p'Repository: lobu-ai/lobu
Length of output: 813
🏁 Script executed:
# Check if IPv6 addresses (::1, ::, etc.) are mentioned or supported anywhere
rg "\\:\\:" packages/server/src/ -A 2 -B 2 | head -30Repository: lobu-ai/lobu
Length of output: 127
🏁 Script executed:
# Check for any HOST or host validation/parsing logic
rg "HOST.*0\\.0\\.0\\.0|host.*0\\.0\\.0\\.0" packages/server/src/ -B 2 -A 2Repository: lobu-ai/lobu
Length of output: 3028
🏁 Script executed:
# Check if IPv6 addresses are ever used as HOST values in tests or config
rg "HOST.*::|host.*::" packages/server/src/Repository: lobu-ai/lobu
Length of output: 1540
🏁 Script executed:
# Look at the mcp-proxy client to understand IPv6 handling
cat -n packages/server/src/mcp-proxy/client.ts | sed -n '1,50p'Repository: lobu-ai/lobu
Length of output: 2233
🏁 Script executed:
# Check startEmbeddedConnectorWorker signature to see what format it expects
rg -A 5 "function startEmbeddedConnectorWorker|export.*startEmbeddedConnectorWorker" packages/Repository: lobu-ai/lobu
Length of output: 667
🏁 Script executed:
# Check if there are other instances of http://${HOST}:${PORT} in start-local.ts
rg "http://\\\$\{HOST\\\}" packages/server/src/start-local.ts -nRepository: lobu-ai/lobu
Length of output: 38
🏁 Script executed:
# Verify server.ts also has the same issue with IPv6
rg "http://\\\$\{" packages/server/src/server.ts -nRepository: lobu-ai/lobu
Length of output: 229
🏁 Script executed:
# Check how the codebase normalizes IPv6 addresses elsewhere
cat -n packages/server/src/mcp-proxy/client.ts | sed -n '80,120p'Repository: lobu-ai/lobu
Length of output: 1440
Normalize embedded worker URL host to ensure a dialable address.
Line 289 uses HOST directly without validation. If HOST=0.0.0.0 (explicitly supported in this file's comments), the daemon cannot dial a non-routable address; IPv6 literals also require bracketed formatting in URLs. This causes sync runs to block indefinitely.
Apply the proposed fix to map 0.0.0.0 to 127.0.0.1 and :: to ::1, and wrap IPv6 addresses with brackets for valid URL formatting:
Proposed fix
httpServer.listen(PORT, HOST, () => {
logger.info(`Lobu running at http://${HOST}:${PORT}`);
logger.info(`Data: ${DATA_DIR}`);
// Embedded daemon must wait for the listener — its boot-time
// health check hits `/api/health` on this same process.
- embeddedWorker = startEmbeddedConnectorWorker(env, `http://${HOST}:${PORT}`);
+ const daemonHost =
+ HOST === '0.0.0.0' ? '127.0.0.1' : HOST === '::' ? '::1' : HOST;
+ const daemonHostForUrl = daemonHost.includes(':') ? `[${daemonHost}]` : daemonHost;
+ embeddedWorker = startEmbeddedConnectorWorker(env, `http://${daemonHostForUrl}:${PORT}`);
});📝 Committable suggestion
‼️ IMPORTANT
Carefully review the code before committing. Ensure that it accurately replaces the highlighted code, contains no missing lines, and has no issues with indentation. Thoroughly test & benchmark the code to ensure it meets the requirements.
| // Embedded daemon must wait for the listener — its boot-time | |
| // health check hits `/api/health` on this same process. | |
| embeddedWorker = startEmbeddedConnectorWorker(env, `http://${HOST}:${PORT}`); | |
| }); | |
| // Embedded daemon must wait for the listener — its boot-time | |
| // health check hits `/api/health` on this same process. | |
| const daemonHost = | |
| HOST === '0.0.0.0' ? '127.0.0.1' : HOST === '::' ? '::1' : HOST; | |
| const daemonHostForUrl = daemonHost.includes(':') ? `[${daemonHost}]` : daemonHost; | |
| embeddedWorker = startEmbeddedConnectorWorker(env, `http://${daemonHostForUrl}:${PORT}`); | |
| }); |
🤖 Prompt for AI Agents
Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.
In `@packages/server/src/start-local.ts` around lines 287 - 290, The code calls
startEmbeddedConnectorWorker(env, `http://${HOST}:${PORT}`) using HOST directly
which breaks when HOST is 0.0.0.0 or an IPv6 literal; implement a small
normalizeHost function that maps "0.0.0.0" -> "127.0.0.1" and "::" -> "::1",
detects IPv6 addresses and wraps them in brackets (e.g., "[::1]") for URL
formatting, then call startEmbeddedConnectorWorker(env,
`http://${normalizedHost}:${PORT}`) so the embedded worker receives a dialable,
correctly-formatted host string.
…urce
Two follow-ups from pi review:
1. Secret leak: passing the gateway's full env into WorkerDaemon would
spread DATABASE_URL / ENCRYPTION_KEY / BETTER_AUTH_SECRET / provider
secrets onto every connector subprocess (`SubprocessExecutor.fork`
does `{...pickSystemEnv(), ...context.env}`). The standalone
connector-worker CLI deliberately whitelists which env vars connectors
see via `buildEnv()`. Extract that whitelist to its own module
(`packages/connector-worker/src/env.ts::buildConnectorWorkerEnv`) so
the embedded daemon can re-use it without pulling in `bin.ts`'s
top-level `main()` execution.
2. Published-CLI connector resolution: the worker-side compile resolver
in `packages/connector-worker/src/compile-connector.ts` didn't include
a candidate for `node_modules/@lobu/cli/dist/connectors`, where
`packages/cli/scripts/build.cjs` actually copies bundled connector
sources. In the monorepo this didn't matter because
`packages/connectors/src` was reachable via the `../../../connectors/src`
candidate, so my repro passed — but a fresh `npx @lobu/cli` install
would have claimed every sync run and failed it with "did not resolve
to a local source file". Add `resolve(HERE, 'connectors')` so the
bundled-CLI layout works.
pi review findings — addressed in 7dc32ddCritical (fixed)Secret leak via connector env. The first commit passed the gateway's full Fix: extracted the standalone connector-worker CLI's existing env whitelist ( High (fixed)Published-CLI connector resolution. The worker-side resolver in Fix: added Non-blockingUpdated the daemon startup comment to be honest — Re-validation
|
There was a problem hiding this comment.
Actionable comments posted: 1
🤖 Prompt for all review comments with AI agents
Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.
Inline comments:
In `@packages/connector-worker/src/env.ts`:
- Line 32: Remove WORKER_API_TOKEN from the connector subprocess environment
whitelist so it is not propagated into connector child processes; locate the env
whitelist (the object/constant that includes the key WORKER_API_TOKEN in
packages/connector-worker/src/env.ts or any code that builds context.env) and
delete that entry (or stop adding process.env.WORKER_API_TOKEN) so the token
remains only in daemon/worker config and is not exposed to connectors.
🪄 Autofix (Beta)
Fix all unresolved CodeRabbit comments on this PR:
- Push a commit to this branch (recommended)
- Create a new PR with the fixes
ℹ️ Review info
⚙️ Run configuration
Configuration used: defaults
Review profile: CHILL
Plan: Pro Plus
Run ID: b6f88252-b5f9-47e7-82ac-d37fa0b7216c
📒 Files selected for processing (4)
packages/connector-worker/src/bin.tspackages/connector-worker/src/compile-connector.tspackages/connector-worker/src/env.tspackages/server/src/scheduled/embedded-connector-worker.ts
🚧 Files skipped from review as they are similar to previous changes (1)
- packages/server/src/scheduled/embedded-connector-worker.ts
| REDDIT_CLIENT_ID: process.env.REDDIT_CLIENT_ID, | ||
| REDDIT_CLIENT_SECRET: process.env.REDDIT_CLIENT_SECRET, | ||
| REDDIT_USER_AGENT: process.env.REDDIT_USER_AGENT, | ||
| WORKER_API_TOKEN: process.env.WORKER_API_TOKEN, |
There was a problem hiding this comment.
Remove WORKER_API_TOKEN from connector subprocess env whitelist.
This token is for worker↔gateway auth and is already passed to daemon config; exposing it through context.env leaks it into every connector child process.
Suggested fix
export function buildConnectorWorkerEnv(): Env {
return {
@@
- WORKER_API_TOKEN: process.env.WORKER_API_TOKEN,
};
}📝 Committable suggestion
‼️ IMPORTANT
Carefully review the code before committing. Ensure that it accurately replaces the highlighted code, contains no missing lines, and has no issues with indentation. Thoroughly test & benchmark the code to ensure it meets the requirements.
| WORKER_API_TOKEN: process.env.WORKER_API_TOKEN, | |
| export function buildConnectorWorkerEnv(): Env { | |
| return { | |
| }; | |
| } |
🤖 Prompt for AI Agents
Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.
In `@packages/connector-worker/src/env.ts` at line 32, Remove WORKER_API_TOKEN
from the connector subprocess environment whitelist so it is not propagated into
connector child processes; locate the env whitelist (the object/constant that
includes the key WORKER_API_TOKEN in packages/connector-worker/src/env.ts or any
code that builds context.env) and delete that entry (or stop adding
process.env.WORKER_API_TOKEN) so the token remains only in daemon/worker config
and is not exposed to connectors.
2 participants
Summary
Two bugs blocked the headless
lobu run -> lobu apply -> trigger_feed -> events appeardata sync loop on a fresh install.Bug A: embedded mode never executed
runs(run_type='sync'). Nothing called/api/workers/poll, so trigger_feed enqueued runs that sat forever. Fix: spin up the existingWorkerDaemonin-process from bothserver.ts(prod) andstart-local.ts(PGlite dev), pointed at the local gateway. Started afterlisten()so its boot health check resolves. Opt-out viaLOBU_DISABLE_EMBEDDED_WORKER=1for deployments with a separate connector-worker pod. Atomic claim already lives inworker-api.ts::pollWorkerJob(FOR UPDATE OF r SKIP LOCKED+claimed_by), so embedded + external workers co-exist without double-execution.Bug B:
/api/workers/poll500'd withRangeError: init["status"] must be in the range of 200 to 599whenever a Better-Auth session-token bearer hit it. Root cause:MultiTenantProvider.resolveAuth'ssetContextAndContinuediscarded the cb's Response return value (await ...; return undefined;pattern across 8 call sites). The workers/* gating middleware's "Worker token missing device_worker:run scope" 403 was silently dropped, Hono never installed it asc.res, and a downstreamc.header()re-wrap collapsed intonew Response(c.body, c)with the Hono Context as init —init.status=c.status(a function) → RangeError. Fix:return setContextAndContinue(...)at every call site, widen the helper return type +WorkspaceProvider.resolveAuth'snextparam (newResolveAuthNextunion) so the cb's Response propagates.Design doc:
docs/fix-sync-loop-design.md(codex-reviewed; addressed blockers aroundWorkerDaemonvsstartDaemon, listener-ordering, and TypeScript widening).Reproducer
Boot
start-local.bundle.mjson port 8802 with PGlite. Sign in as install_operator via/api/local-init. Create a hackernews connection + astoriesfeed withsearch_query: openai, calltrigger_feed.Pre-fix:
status='pending'; no events ever land.POST /api/workers/poll -H 'Authorization: Bearer <signed-session-token>'-> HTTP 500, log:RangeError: init["status"] must be in the range of 200 to 599.Post-fix:
pending->running->completed(336 events for HNopenaisearch, written toeventstable).POST /api/workers/pollwith the same session bearer -> HTTP 403Worker token missing device_worker:run scope(correct behavior — session auth lacks the scope; use a PAT instead). With a PAT -> HTTP 200. Log has zero RangeErrors.Test plan
make build-packages— cleanmake typecheck(strict, matches Dockerfile) — clean/api/workers/pollwith bad/good/PAT bearers — no 500sSummary by CodeRabbit
New Features
Bug Fixes
/api/workers/poll500 error by preserving middleware responses through auth flow.Documentation