Skip to content

feat(connectors): user-declared connector dependencies (npm bundled + nix native)#973

Merged
buremba merged 8 commits into
mainfrom
feat/connector-deps
May 20, 2026
Merged

feat(connectors): user-declared connector dependencies (npm bundled + nix native)#973
buremba merged 8 commits into
mainfrom
feat/connector-deps

Conversation

@buremba
Copy link
Copy Markdown
Member

@buremba buremba commented May 20, 2026

Summary

Lets connectors declare and use dependencies:

  • npm deps → the project's package.json (next to lobu.toml), bundled into the connector by esbuild at compile time. esbuild resolves a connector's imports relative to the connector file's directory, so the project's node_modules is used regardless of where the lobu binary is installed (global / npx / local).
  • native deps → declared as nixpkgs refs in runtime.nix.packages on the connector definition; provisioned on PATH via nix-shell at execution time.
  • Compile moves to the CLI: lobu apply runs bun install in the project, compiles each connector with the project's deps, and uploads the pre-compiled bundle (compiled: true) — the server can't bundle deps it never receives.
  • Lean bundles: @lobu/connector-sdk is externalized (runtime-provided, like Lambda's aws-sdk) instead of bundled, so a connector artifact is its own code + deps (KB) rather than the SDK's full infra graph (Sentry/OTel/grpc/git, ~6+ MB).
  • lobu init scaffolds package.json (with @lobu/connector-sdk devDependency for editor types) + tsconfig.json + connectors/.

Commits

  • feat(connector-worker) — SDK runtime.nix.packages type; compile externalizes the SDK + inline sourcemap; executor wraps the child in nix-shell -p … --run "exec node …" (preserves IPC + kill) with a graceful "nix not installed" error.
  • feat(server) — poll response surfaces nix_packages (read from the existing runtime JSONB — no migration) → daemon → executor.
  • feat(cli) — compile-on-apply + bun install --ignore-scripts (project root resolved via nearest lobu.toml); lobu init scaffolding; AGENTS.md docs.
  • refactor(cli) — lazy-load the connector-compile graph in apply (allow-listed) to keep it off apply-cmd's module-load path.

Validation

  • nix path E2E (real nix 2.34.7): declared package on PATH inside the connector subprocess, IPC + kill survive the wrap. ✅
  • externalize-SDK against the real built SDK graph: user dep bundles + runs; SDK + transitive infra resolve from the runtime. ✅
  • project-dep resolution: real compiler bundles a dep present only in the project's node_modules (not the CLI's). ✅
  • make typecheck (strict, Dockerfile-equivalent): clean. ✅
  • bun test packages/connector-worker: 52 pass / 0 fail. ✅
  • CLI apply suite (packages/cli/.../apply) in isolation: 130 pass / 0 fail. ✅

Test plan

  • CI green (Linux) — covers the unit suites.
  • Human E2E: lobu apply a project whose connector declares an npm dep + runtime.nix.packages, on a gateway, and confirm a run succeeds with the tool on PATH.

Notes

  • bun test packages/cli (full suite) shows 4 AutoCreateWhenRule errors only on macOS/node-26/bun-1.3.5 — a pre-existing bun test-runner quirk on a value+type dual re-export in connector-sdk, reproduced identically on the base commit and unrelated to this PR. Each file passes in isolation; standalone import works; CI (Linux) is unaffected.

Summary by CodeRabbit

  • New Features

    • Init scaffolds connector projects (TS config, connector SDK dep, connectors/ dir).
    • CLI can install project deps, compile local connectors, and upload precompiled bundles; server/workers accept a compiled flag.
    • Connectors may declare native (nix) packages to be provisioned at runtime; workers will use nix-shell when provided.
  • Documentation

    • Expanded connector workflow docs for dependency installation, compilation, and upload behavior.
  • Reliability

    • CLI checks/installs deps and surfaces errors when required tools (bun/nix) are missing.

Review Change Stack

@coderabbitai
Copy link
Copy Markdown

coderabbitai Bot commented May 20, 2026

Caution

Review failed

Pull request was closed or merged during review

No actionable comments were generated in the recent review. 🎉

ℹ️ Recent review info
⚙️ Run configuration

Configuration used: defaults

Review profile: CHILL

Plan: Pro Plus

Run ID: 7d7ef81f-83b5-405f-8509-b7179175ae16

📥 Commits

Reviewing files that changed from the base of the PR and between 203b82f and fe84016.

📒 Files selected for processing (3)
  • packages/cli/src/commands/memory/_lib/schema.ts
  • packages/connector-sdk/package.json
  • packages/pgvector-embedded/src/index.ts
✅ Files skipped from review due to trivial changes (1)
  • packages/cli/src/commands/memory/_lib/schema.ts

📝 Walkthrough

Walkthrough

This PR adds end-to-end native dependency support for connectors: CLI pre-compiles local connector source (ensuring project deps), the compile pipeline externalizes the SDK, connectors may declare nix.packages, the daemon threads packages to executors, and SubprocessExecutor can run connector code inside nix-shell. Server poll includes runtime metadata.

Changes

Connector Native Dependencies: End-to-End Integration

Layer / File(s) Summary
Runtime schema and execution contracts
packages/connector-sdk/src/connector-types.ts, packages/connector-worker/src/executor/interface.ts, packages/connector-worker/src/daemon/client.ts, packages/connector-sdk/package.json
ConnectorRuntimeInfo gains optional nix: { packages: string[] }; ExecutionOptions carries per-run nixPackages; PollResponse adds nix_packages?: string[]; SDK package exports gains ./identity-types subpath.
CLI: ensure deps, compile, and upload
packages/cli/src/commands/_lib/ensure-deps-installed.ts, packages/cli/src/commands/_lib/apply/apply-cmd.ts, packages/cli/src/commands/_lib/apply/client.ts, packages/cli/src/commands/init.ts, packages/cli/src/commands/memory/_lib/schema.ts
CLI finds connector project root, runs bun install --ignore-scripts when stale, lazy-loads compilation helpers, compiles local *.connector.ts when def.sourcePath is present, uploads compiled bundles with compiled: true, accepts compiled in client payloads, scaffolds connector files on lobu init, and updates a test import to the new SDK subpath.
Connector compilation pipeline: SDK externalization
packages/connector-worker/src/compile/index.ts
Introduces an esbuild plugin that externalizes lobu@lobu/connector-sdk, removes alias-based SDK resolution, and changes sourcemap to inline with sourcesContent: false.
Executor runtime: nix-shell wrapping and threading
packages/connector-worker/src/executor/runtime.ts, packages/connector-worker/src/executor/subprocess.ts, packages/connector-worker/src/daemon/executor.ts, packages/connector-worker/src/executor/interface.ts
Daemon threads job.nix_packages to executeCompiledConnector; SyncExecutor.execute accepts options?: ExecutionOptions; SubprocessExecutor conditionally runs connector inside nix-shell -p <packages> or falls back to fork.
Server worker poll: connector runtime metadata delivery
packages/server/src/worker-api.ts
Worker poll query LEFT JOINs connector_definitions to include connector_runtime and returns computed nix_packages (omitted when empty) in the job payload.
Documentation: connector workflows and dynamic-import allow-list
AGENTS.md
Adds a "Connectors" section describing dependency handling, compile/upload flow, lobu init scaffolding, and updates the dynamic import allow-list with the lazy-loaded CLI apply helper.
pgvector-embedded: macOS native lib symlink helper
packages/pgvector-embedded/src/index.ts
Adds ensureNativeLibSonames() to create missing version/unversion symlinks for .dylib files on macOS and calls it during injectPgvector().

Sequence Diagram(s)

sequenceDiagram
  Client->>Server: installConnector({ sourceCode|sourceUrl|compiled })
  Server->>WorkerDaemon: job available via poll (includes nix_packages)
  WorkerDaemon->>Executor: executeCompiledConnector(compiledCode, job, hooks, nixPackages)
  Executor->>SubprocessExecutor: execute(..., options:{nixPackages})
  alt nixPackages present
    SubprocessExecutor->>NixShell: spawn nix-shell -p <packages> --run "exec node <child-runner>"
    NixShell->>ChildProcess: runs connector in environment with native packages
  else no nixPackages
    SubprocessExecutor->>ChildProcess: fork child-runner directly
  end
Loading

Estimated code review effort

🎯 4 (Complex) | ⏱️ ~45 minutes

Possibly related issues

Possibly related PRs

  • lobu-ai/lobu#711: Overlaps in SubprocessExecutor / child-runner execution path changes.
  • lobu-ai/lobu#931: Overlaps in connector compile pipeline and loader refactors.
  • lobu-ai/lobu#932: Related updates to the AGENTS.md dynamic-import allow-list.

Suggested labels

skip-size-check

Poem

A rabbit packed a tiny crate,
With nix and node it stitched its fate,
CLI compiles, the shell encloses,
Native tools snug as clover roses—🐰✨

🚥 Pre-merge checks | ✅ 4 | ❌ 1

❌ Failed checks (1 warning)

Check name Status Explanation Resolution
Docstring Coverage ⚠️ Warning Docstring coverage is 71.43% which is insufficient. The required threshold is 80.00%. Write docstrings for the functions missing them to satisfy the coverage threshold.
✅ Passed checks (4 passed)
Check name Status Explanation
Title check ✅ Passed The title clearly and specifically describes the main change: enabling user-declared connector dependencies (npm bundled + nix native) in connectors, which is the central feature of this PR.
Description check ✅ Passed The pull request description covers all required template sections: Summary (detailed 5-point explanation of the feature), Test plan (with checkboxes for validation steps), and Notes (addressing a pre-existing macOS test quirk and pgvector fix).
Linked Issues check ✅ Passed Check skipped because no linked issues were found for this pull request.
Out of Scope Changes check ✅ Passed Check skipped because no linked issues were found for this pull request.

✏️ Tip: You can configure your own custom pre-merge checks in the settings.

✨ Finishing Touches
📝 Generate docstrings
  • Create stacked PR
  • Commit on current branch
🧪 Generate unit tests (beta)
  • Create PR with unit tests
  • Commit unit tests in branch feat/connector-deps

Warning

There were issues while running some tools. Please review the errors and either fix the tool's configuration or disable the tool if it's a critical failure.

🔧 ESLint

If the error stems from missing dependencies, add them to the package.json file. For unrecoverable errors (e.g., due to private dependencies), disable the tool in the CodeRabbit configuration.

ESLint skipped: no ESLint configuration detected in root package.json. To enable, add eslint to devDependencies.


Comment @coderabbitai help to get the list of available commands and usage tips.

@codecov-commenter
Copy link
Copy Markdown

codecov-commenter commented May 20, 2026

⚠️ Please install the 'codecov app svg image' to ensure uploads and comments are reliably processed by Codecov.

Codecov Report

❌ Patch coverage is 52.50000% with 38 lines in your changes missing coverage. Please review.

Files with missing lines Patch % Lines
packages/cli/src/commands/_lib/apply/apply-cmd.ts 0.00% 32 Missing ⚠️
packages/cli/src/commands/_lib/apply/client.ts 0.00% 6 Missing ⚠️

📢 Thoughts on this report? Let us know!

Copy link
Copy Markdown

@coderabbitai coderabbitai Bot left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Actionable comments posted: 4

🤖 Prompt for all review comments with AI agents
Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.

Inline comments:
In `@AGENTS.md`:
- Line 62: Update the sentence describing what `lobu apply` runs so it
accurately states the exact install command: replace the mention of `bun
install` with `bun install --ignore-scripts` in the paragraph that explains
"Compile happens on the CLI" and the `lobu apply` behavior, ensuring the doc
clearly communicates the safety/behavior contract regarding script execution;
keep the rest of the explanation about bundling and externalizing
`@lobu/connector-sdk` unchanged.

In `@packages/cli/src/commands/_lib/apply/apply-cmd.ts`:
- Around line 380-397: The compile step currently always uses def.sourceFile
which ignores def.sourceCode (set when using source_url) and can compile the
wrong file; update the branch that checks def.sourceCode to prefer compiling
from the in-memory source (def.sourceCode) rather than def.sourceFile — either
by passing def.sourceCode into compileConnectorFromFile (or adding a new
compileConnectorFromSource API) or by writing def.sourceCode to a temporary file
and using that path; also ensure ensureProjectDepsInstalled is invoked with the
correct project entry (use the temp file path or adjust it to accept source
content) before calling client.installConnector so the uploaded bundle matches
the provided source.

In `@packages/cli/src/commands/init.ts`:
- Around line 666-699: The current init logic unconditionally overwrites
package.json and tsconfig.json via writeFile; change it to first check for
existing files (e.g., using fs.existsSync or fs.promises.access) and only
create/overwrite when absent or explicitly forced, and when a file exists read
and merge sensible defaults (for package.json merge devDependencies and preserve
existing fields; for tsconfig.json merge/extend existing compilerOptions and
include) before writing; update the writeFile calls that target join(projectDir,
"package.json") and join(projectDir, "tsconfig.json") to implement this
existence check + merge behavior and keep the mkdir(join(projectDir,
"connectors"), { recursive: true }) and .gitkeep creation unchanged.

In `@packages/server/src/worker-api.ts`:
- Line 492: The join on connector_definitions that supplies connector_runtime is
currently broad and can return nondeterministic rows; update the query that
selects cd.runtime AS connector_runtime (and the related joins at the same area)
to only pick the latest active connector_definition per (key, organization_id) —
e.g., replace the direct join with a subquery or LATERAL join that filters WHERE
status = 'active' and orders by a monotonic column (created_at or version) DESC
LIMIT 1, and use that subquery's runtime as connector_runtime so rows[0] is
deterministic and always returns the latest active runtime.nix.packages.
🪄 Autofix (Beta)

Fix all unresolved CodeRabbit comments on this PR:

  • Push a commit to this branch (recommended)
  • Create a new PR with the fixes

ℹ️ Review info
⚙️ Run configuration

Configuration used: defaults

Review profile: CHILL

Plan: Pro Plus

Run ID: 1cf9c280-0bdd-46de-8727-9563121fb51c

📥 Commits

Reviewing files that changed from the base of the PR and between c8553c1 and 0702f23.

📒 Files selected for processing (13)
  • AGENTS.md
  • packages/cli/src/commands/_lib/apply/apply-cmd.ts
  • packages/cli/src/commands/_lib/apply/client.ts
  • packages/cli/src/commands/_lib/ensure-deps-installed.ts
  • packages/cli/src/commands/init.ts
  • packages/connector-sdk/src/connector-types.ts
  • packages/connector-worker/src/compile/index.ts
  • packages/connector-worker/src/daemon/client.ts
  • packages/connector-worker/src/daemon/executor.ts
  • packages/connector-worker/src/executor/interface.ts
  • packages/connector-worker/src/executor/runtime.ts
  • packages/connector-worker/src/executor/subprocess.ts
  • packages/server/src/worker-api.ts

Comment thread AGENTS.md Outdated
Comment thread packages/cli/src/commands/_lib/apply/apply-cmd.ts Outdated
Comment thread packages/cli/src/commands/init.ts Outdated
conn.config AS connection_config,
conn.device_worker_id AS connection_device_worker_id,
cv.compiled_code,
cd.runtime AS connector_runtime,
Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

⚠️ Potential issue | 🟠 Major | ⚡ Quick win

Use latest active connector_definition when reading runtime metadata.

Line 504 introduces a broad join on connector_definitions without status/order constraints. If multiple rows exist for the same (key, organization_id), rows[0] becomes nondeterministic and can return stale/wrong runtime.nix.packages.

Suggested fix
-      LEFT JOIN connector_definitions cd ON cd.key = r.connector_key
-        AND cd.organization_id = r.organization_id
+      LEFT JOIN LATERAL (
+        SELECT runtime
+        FROM connector_definitions cd
+        WHERE cd.key = r.connector_key
+          AND cd.organization_id = r.organization_id
+          AND cd.status = 'active'
+        ORDER BY cd.updated_at DESC, cd.id DESC
+        LIMIT 1
+      ) cd ON true

Also applies to: 504-506

🤖 Prompt for AI Agents
Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.

In `@packages/server/src/worker-api.ts` at line 492, The join on
connector_definitions that supplies connector_runtime is currently broad and can
return nondeterministic rows; update the query that selects cd.runtime AS
connector_runtime (and the related joins at the same area) to only pick the
latest active connector_definition per (key, organization_id) — e.g., replace
the direct join with a subquery or LATERAL join that filters WHERE status =
'active' and orders by a monotonic column (created_at or version) DESC LIMIT 1,
and use that subquery's runtime as connector_runtime so rows[0] is deterministic
and always returns the latest active runtime.nix.packages.

Copy link
Copy Markdown

@coderabbitai coderabbitai Bot left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

♻️ Duplicate comments (2)
packages/cli/src/commands/init.ts (1)

673-691: ⚠️ Potential issue | 🟠 Major | ⚡ Quick win

Don’t silently clobber an existing package.json on read/parse failures.

The current broad fallback can replace an existing file when parsing fails, which is destructive for --here init. Treat only missing-file as “create new”, and surface parse/read errors for existing files.

Suggested fix
-    let pkgJson: Record<string, unknown>;
-    try {
-      pkgJson = JSON.parse(await readFile(pkgJsonPath, "utf-8")) as Record<
-        string,
-        unknown
-      >;
-    } catch {
-      pkgJson = {
-        name: projectName,
-        version: "0.0.0",
-        private: true,
-        type: "module",
-      };
-    }
+    let pkgJson: Record<string, unknown>;
+    try {
+      const raw = await readFile(pkgJsonPath, "utf-8");
+      pkgJson = JSON.parse(raw) as Record<string, unknown>;
+    } catch (err) {
+      const code = (err as NodeJS.ErrnoException).code;
+      if (code !== "ENOENT") {
+        throw new Error(
+          `Found existing package.json but could not parse/read it. Please fix it and re-run init.`
+        );
+      }
+      pkgJson = {
+        name: projectName,
+        version: "0.0.0",
+        private: true,
+        type: "module",
+      };
+    }
🤖 Prompt for AI Agents
Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.

In `@packages/cli/src/commands/init.ts` around lines 673 - 691, The current
try/catch around reading/parsing pkgJsonPath can silently overwrite an existing
package.json on parse/read failure; change the logic in the init flow that sets
pkgJson (variables: pkgJsonPath, pkgJson, projectName, cliVersion) so that you
only build the default object when the file truly does not exist (e.g., detect
ENOENT via fs.stat/fs.access or check the thrown error.code), but for any other
read or JSON.parse error rethrow or surface the error to the user instead of
falling back to the default; then continue to merge devDependencies and write
the file as before.
packages/server/src/worker-api.ts (1)

504-506: ⚠️ Potential issue | 🟠 Major | ⚡ Quick win

Make connector runtime join deterministic (latest active row).

status = 'active' helps, but this join can still return multiple rows per key/org and make runtime selection nondeterministic.

Suggested fix
-      LEFT JOIN connector_definitions cd ON cd.key = r.connector_key
-        AND cd.organization_id = r.organization_id
-        AND cd.status = 'active'
+      LEFT JOIN LATERAL (
+        SELECT runtime
+        FROM connector_definitions cd
+        WHERE cd.key = r.connector_key
+          AND cd.organization_id = r.organization_id
+          AND cd.status = 'active'
+        ORDER BY cd.updated_at DESC, cd.id DESC
+        LIMIT 1
+      ) cd ON TRUE
🤖 Prompt for AI Agents
Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.

In `@packages/server/src/worker-api.ts` around lines 504 - 506, The current LEFT
JOIN to connector_definitions (alias cd) using cd.key = r.connector_key,
cd.organization_id = r.organization_id and cd.status = 'active' can return
multiple active rows and make runtime selection nondeterministic; change the
join to target a single deterministic "latest" active row by joining cd to a
subquery that selects the single id (or other unique PK) of the latest active
definition per key/org (e.g., ORDER BY created_at or version DESC LIMIT 1) and
join on cd.id = that id so functions referencing connector_definitions always
get the latest active row for r.connector_key and r.organization_id.
🤖 Prompt for all review comments with AI agents
Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.

Duplicate comments:
In `@packages/cli/src/commands/init.ts`:
- Around line 673-691: The current try/catch around reading/parsing pkgJsonPath
can silently overwrite an existing package.json on parse/read failure; change
the logic in the init flow that sets pkgJson (variables: pkgJsonPath, pkgJson,
projectName, cliVersion) so that you only build the default object when the file
truly does not exist (e.g., detect ENOENT via fs.stat/fs.access or check the
thrown error.code), but for any other read or JSON.parse error rethrow or
surface the error to the user instead of falling back to the default; then
continue to merge devDependencies and write the file as before.

In `@packages/server/src/worker-api.ts`:
- Around line 504-506: The current LEFT JOIN to connector_definitions (alias cd)
using cd.key = r.connector_key, cd.organization_id = r.organization_id and
cd.status = 'active' can return multiple active rows and make runtime selection
nondeterministic; change the join to target a single deterministic "latest"
active row by joining cd to a subquery that selects the single id (or other
unique PK) of the latest active definition per key/org (e.g., ORDER BY
created_at or version DESC LIMIT 1) and join on cd.id = that id so functions
referencing connector_definitions always get the latest active row for
r.connector_key and r.organization_id.

ℹ️ Review info
⚙️ Run configuration

Configuration used: defaults

Review profile: CHILL

Plan: Pro Plus

Run ID: fa1fc511-b56f-4ebe-8ef2-8aa4a9905cc5

📥 Commits

Reviewing files that changed from the base of the PR and between 0702f23 and 18e953b.

📒 Files selected for processing (4)
  • AGENTS.md
  • packages/cli/src/commands/_lib/apply/apply-cmd.ts
  • packages/cli/src/commands/init.ts
  • packages/server/src/worker-api.ts
✅ Files skipped from review due to trivial changes (1)
  • AGENTS.md

buremba added 7 commits May 20, 2026 19:37
Foundation for user-declared connector dependencies:

- SDK: add `runtime.nix.packages` to ConnectorRuntimeInfo so a connector
  declares its native (nixpkgs) tools; rides the existing `runtime` JSONB,
  no migration. npm deps are bundled at compile time and don't go here.
- Compile: externalize `@lobu/connector-sdk` (+ `lobu` alias) instead of
  bundling it. The SDK pulls a large infra graph (Sentry/OTel/grpc/git)
  transitively, inflating every connector to multiple MB; the runtime
  provides the SDK (it's a connector-worker dep), so the bundle leaves it
  as a runtime-resolved import — the standard externalize-the-framework
  pattern. Bundles drop from ~MB to the connector's own code + deps.
  Also emit an inline source map (sourcesContent:false) for stack traces.
- Executor: when a connector declares nix packages, wrap the child in
  `nix-shell -p <pkgs> --run "exec node ..."` so the tools are on PATH;
  `exec` preserves the IPC channel and kill semantics. Plain fork() stays
  the path when no packages are declared. Fail with a clear actionable
  error when nix-shell is absent but packages are required.
Threads a connector's declared native packages from storage to the
executor so they're on PATH during a run:

- worker-api poll: join connector_definitions and surface its `runtime`,
  emitting `nix_packages` in the poll response (the `runtime.nix.packages`
  the SDK extraction already persists in the existing runtime JSONB — no
  storage change needed).
- daemon: carry `nix_packages` on PollResponse and pass it through the
  three executeCompiledConnector call sites (sync/action/auth).
- executeCompiledConnector: forward `nixPackages` to executor.execute,
  which wraps the child in nix-shell when non-empty.

Connectors that declare no native deps are unaffected (plain fork path).
…ckaging

Connector npm deps now work in the apply→cloud path:

- apply: project-supplied connectors are compiled on the CLI (the only
  place the project's node_modules exists, so esbuild can bundle the
  connector's declared deps) and uploaded as a pre-compiled bundle
  (`compiled: true`); the server stores the artifact instead of
  recompiling source it can't resolve deps for.
- ensure-deps-installed: runs `bun install --ignore-scripts` in the
  connector's project (resolved via the nearest `lobu.toml`, so a
  connector inside a monorepo never installs the wrong root); no-ops when
  the project declares no package.json.
- client.installConnector gains a `compiled` flag.
- lobu init scaffolds package.json (with @lobu/connector-sdk devDependency
  for editor types) + tsconfig.json + connectors/.
- AGENTS.md documents the convention: npm = bundled at compile, native =
  nix at run time; SDK is runtime-provided/externalized.
Keep esbuild + connector-worker + SDK out of apply-cmd's module-load path
(only `lobu apply` with local *.connector.ts needs it). Documented in the
AGENTS.md dynamic-import allow-list. Matches connector-run-cmd's pattern.
…oin, init merge

- apply: compile `def.sourcePath` (the actual `.ts`) for local connectors, not
  `def.sourceFile` (an error-message label that may point at a `type: connector`
  YAML). `source_url` connectors (source fetched into `sourceCode`, no local
  deps) upload raw for gateway compile instead of being compiled locally.
- worker-api: filter the connector_definitions join to `status = 'active'` so it
  matches the partial unique index `idx_connector_defs_org_key` — archived/draft
  rows share `(key, org)` and made the runtime lookup nondeterministic.
- init: merge into an existing `package.json` (preserve the user's fields, add
  the SDK devDependency) and never overwrite an existing `tsconfig.json`, so
  `--here` into an existing project is non-destructive.
- docs: note `bun install --ignore-scripts` (the actual command).
@buremba buremba force-pushed the feat/connector-deps branch from 18e953b to 203b82f Compare May 20, 2026 18:39
Copy link
Copy Markdown

@coderabbitai coderabbitai Bot left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

🧹 Nitpick comments (1)
packages/cli/src/commands/init.ts (1)

295-336: ⚡ Quick win

Wire the database choice to a CLI flag (or remove the URL branch).

At Line 295, flag is hardcoded to undefined, so databaseChoice can only be 'embedded'/'external' from the select prompt. That makes the URL branch at Line 329 effectively unreachable and blocks non-interactive selection of an external DB in --yes flows.

💡 Suggested minimal patch
 export interface InitOptions {
   yes?: boolean;
   here?: boolean;
+  database?: string;
   port?: string;
   publicUrl?: string;
   network?: string;
@@
   const databaseChoice = await promptOrDefault({
-    flag: undefined,
+    flag: options.database,
     useDefaults,
     defaultValue: "embedded",
🤖 Prompt for AI Agents
Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.

In `@packages/cli/src/commands/init.ts` around lines 295 - 336, The select
prompt's flag is set to undefined so a passed CLI value never populates
databaseChoice, making the URL branch unreachable; set the select's flag (e.g.,
flag: "database") so CLI-supplied values are honored, keep the validate function
that accepts "embedded"|"external"|/^(postgres(ql)?|file):/i so URLs validate,
and ensure subsequent logic uses the resulting databaseChoice to populate
databaseUrl (so the existing URL branch at the databaseUrl assignment becomes
reachable in non-interactive --yes flows).
🤖 Prompt for all review comments with AI agents
Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.

Nitpick comments:
In `@packages/cli/src/commands/init.ts`:
- Around line 295-336: The select prompt's flag is set to undefined so a passed
CLI value never populates databaseChoice, making the URL branch unreachable; set
the select's flag (e.g., flag: "database") so CLI-supplied values are honored,
keep the validate function that accepts
"embedded"|"external"|/^(postgres(ql)?|file):/i so URLs validate, and ensure
subsequent logic uses the resulting databaseChoice to populate databaseUrl (so
the existing URL branch at the databaseUrl assignment becomes reachable in
non-interactive --yes flows).

ℹ️ Review info
⚙️ Run configuration

Configuration used: defaults

Review profile: CHILL

Plan: Pro Plus

Run ID: ff32e53c-0844-4b8c-8a4c-d1ff81197e12

📥 Commits

Reviewing files that changed from the base of the PR and between 18e953b and 203b82f.

📒 Files selected for processing (13)
  • AGENTS.md
  • packages/cli/src/commands/_lib/apply/apply-cmd.ts
  • packages/cli/src/commands/_lib/apply/client.ts
  • packages/cli/src/commands/_lib/ensure-deps-installed.ts
  • packages/cli/src/commands/init.ts
  • packages/connector-sdk/src/connector-types.ts
  • packages/connector-worker/src/compile/index.ts
  • packages/connector-worker/src/daemon/client.ts
  • packages/connector-worker/src/daemon/executor.ts
  • packages/connector-worker/src/executor/interface.ts
  • packages/connector-worker/src/executor/runtime.ts
  • packages/connector-worker/src/executor/subprocess.ts
  • packages/server/src/worker-api.ts
✅ Files skipped from review due to trivial changes (1)
  • AGENTS.md

@buremba
Copy link
Copy Markdown
Member Author

buremba commented May 20, 2026

bug_free 95, simplicity 95, slop 0, bugs 0, 0 blockers

[env] Integration tests failed due to ENOSPC (no space left on device) during postgres DB initialization, unrelated to the diff.

Full verdict JSON
{
  "bug_free_confidence": 95,
  "bugs": 0,
  "slop": 0,
  "simplicity": 95,
  "blockers": [],
  "change_type": "feat",
  "behavior_change_risk": "low",
  "tests_adequate": true,
  "suggested_fixes": [],
  "notes": "[env] Integration tests failed due to ENOSPC (no space left on device) during postgres DB initialization, unrelated to the diff.",
  "categories": {
    "src": 376,
    "tests": 0,
    "docs": 6,
    "config": 0,
    "deps": 0,
    "migrations": 0,
    "ci": 0,
    "generated": 0
  }
}

Local review gate — branch protection can require the pi-review commit status. See docs/REVIEW_SCHEMA.md.

…ed-pg dylib symlinks

Two pre-existing macOS-only blockers that fail `make review` locally (both
already green on CI's Linux runners), surfaced while validating this PR:

- connector-sdk: a barrel re-export of the value+type dual name
  `AutoCreateWhenRule` trips bun's cross-file module lexer in the test runner
  ("Export named ... not found"), failing the CLI test suite on macOS. Add a
  `./identity-types` subpath export and import the value from it in the memory
  CLI's schema (bypasses the barrel). CLI suite: 4 fail → 0 fail (308 pass).
- pgvector-embedded: the @embedded-postgres/darwin-* packages ship only
  fully-versioned `lib*.<major>.<minor>.dylib` ICU libs; postgres/initdb link
  against the `lib*.<major>.dylib` SONAME and the unversioned `lib*.dylib`
  linker name, so initdb aborted with "Library not loaded: libicui18n.dylib".
  injectPgvector now hydrates both symlink levels idempotently (macOS only).
  Embedded Postgres initdb + start now succeed on darwin-arm64.
@buremba buremba merged commit ac2ddbd into main May 20, 2026
14 of 17 checks passed
@buremba buremba deleted the feat/connector-deps branch May 20, 2026 19:21
buremba added a commit that referenced this pull request May 20, 2026
…976) (#984)

* test: harden flaky port allocation + connector-sdk re-export barrel (#976)

Two pre-existing test flakes from #976 (both intermittent, CI green; flake-
reduction, not broken tests):

- Port collisions: embedded-postgres-backend.ts and proxy-hardening.test.ts
  picked random high ports in overlapping ranges (50000-60000 vs 10000-60000)
  with no collision handling, racing to EADDRINUSE under concurrent test load.
  Add a shared free-port helper (bind :0, read assigned port, retry on
  EADDRINUSE) and route both sites through it.

- connector-sdk AutoCreateWhenRule: re-export the identity-types schemas via
  import + a local export list instead of a transitive `export {…} from`
  barrel, so bun's ESM linker stops intermittently failing to resolve the name
  under concurrent load. Preventive — see PR body (could not reproduce on
  current main, likely already mitigated by #973 externalizing the SDK).

* test: re-tag embedded-postgres bind failure as EADDRINUSE so retry fires

Review finding (pi): embedded-postgres rejects start() with `undefined` on any
early exit — a port collision included — so the OS-level EADDRINUSE never
reaches withFreePortRetry's catch, leaving the retry dead for this caller.

Capture stderr via `onLog` and re-tag a bind failure (`could not bind` /
`address already in use`) as an Error with code='EADDRINUSE' so the wrapper
retries; surface any other failure as a real Error instead of the original
`undefined` reject.

Validated red->green with a real PG-vs-PG collision: start() rejects with
`undefined` (raw err type: undefined), onLog captures the bind error, and the
catch re-tags it EADDRINUSE.
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants