Skip to content

feat(server)!: replace PGlite with embedded Postgres; bundle pgvector; earthdistance geo #965

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Merged
buremba merged 18 commits into from
May 20, 2026
Merged

feat(server)!: replace PGlite with embedded Postgres; bundle pgvector; earthdistance geo #965

Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
18 commits
Select commit Hold shift + click to select a range
23080a4
feat(server)!: replace PGlite with embedded Postgres; bundle pgvector…
buremba May 20, 2026
e315039
feat(cli): lobu init asks embedded vs external Postgres; write DATABA…
buremba May 20, 2026
6fba2ec
refactor(server): consolidate the two server entrypoints into one
buremba May 20, 2026
85085de
refactor: remove LOBU_DATA_DIR — DATABASE_URL is the single backend s…
buremba May 20, 2026
45b8e2f
chore: bump owletto pointer — Mac runner uses DATABASE_URL=file:// (d…
buremba May 20, 2026
919d1e6
chore: bump packages/owletto pointer for Keychain ENCRYPTION_KEY pers…
buremba May 20, 2026
c6c3561
fix(pgvector-embedded): stage artifacts from build dir + build portab…
buremba May 20, 2026
8d4c257
refactor(server): drop simpleQuery identity wrapper
buremba May 20, 2026
95bcfe0
chore(pgvector-embedded): add linux-x64 + linux-arm64 prebuilt artifacts
buremba May 20, 2026
aba14f4
fix(docker): wire @lobu/pgvector-embedded into the build, prune embed…
buremba May 20, 2026
81bc467
refactor(server): drop dead applyDevProjectPathDefault, fix start-loc…
buremba May 20, 2026
7e1bbf1
Merge origin/main into feat/embedded-pg-spike
buremba May 20, 2026
100f7c7
fix(ci): build @lobu/pgvector-embedded before the integration suite
buremba May 20, 2026
c117da3
fix(ci): build darwin-x64 pgvector on macos-15-intel, not retired mac…
buremba May 20, 2026
1fe2964
Merge remote-tracking branch 'origin/main' into feat/embedded-pg-spike
buremba May 20, 2026
0c44ed3
chore(pgvector-embedded): add darwin-x64 prebuilt artifact
buremba May 20, 2026
5a54c7d
fix: turn-liveness test uses renamed gateway DB helper; review.sh DAT…
buremba May 20, 2026
481cd59
docs(server): correct entry-point comment — embedded-runtime is stati…
buremba May 20, 2026
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
121 changes: 121 additions & 0 deletions .github/workflows/build-pgvector-embedded.yml
View file
Open in desktop
Original file line number Diff line number Diff line change
@@ -0,0 +1,121 @@
name: Build pgvector-embedded artifacts

# Rebuilds the prebuilt pgvector artifacts vendored in
# packages/pgvector-embedded/prebuilt/<platform>/ for every platform
# embedded-postgres supports. Run on demand (bump pgvector / PG major) or when
# the build script changes. embedded-postgres ships vanilla PG 18 with no
# pgvector, so each cell compiles pgvector against a same-major PostgreSQL (the
# extension ABI is stable within a major) and uploads the result; a final job
# opens a PR with the regenerated artifacts.
#
# Windows (windows-x64) is intentionally NOT built yet — pgvector on Windows
# needs an MSVC/nmake build the bash script doesn't cover. Follow-up.

on:
workflow_dispatch:
inputs:
pgvector_version:
description: pgvector git tag to build
default: v0.8.1
push:
paths:
- packages/pgvector-embedded/scripts/build.sh
- .github/workflows/build-pgvector-embedded.yml

permissions:
contents: write
pull-requests: write
Comment on lines +25 to +27
Copy link
Copy Markdown

@coderabbitai coderabbitai Bot May 20, 2026

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

⚠️ Potential issue | 🟠 Major | ⚡ Quick win

🧩 Analysis chain

🏁 Script executed:

cat .github/workflows/build-pgvector-embedded.yml

Repository: lobu-ai/lobu

Length of output: 4464

Scope permissions per job instead of workflow-wide write access.

The build job only performs checkout and artifact upload, requiring read-only access. The open-pr job requires write permissions for git operations and PR creation. Restrict workflow-level permissions to contents: read and add job-level write permissions only to open-pr.

Suggested hardening diff 
 permissions:
-  contents: write
-  pull-requests: write
+  contents: read

 jobs:
   build:
+    permissions:
+      contents: read
@@
   open-pr:
+    permissions:
+      contents: write
+      pull-requests: write
🧰 Tools
🪛 zizmor (1.25.2)

[error] 26-26: overly broad permissions (excessive-permissions): contents: write is overly broad at the workflow level

(excessive-permissions)

[error] 27-27: overly broad permissions (excessive-permissions): pull-requests: write is overly broad at the workflow level

(excessive-permissions)

🤖 Prompt for AI Agents 
Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.

In @.github/workflows/build-pgvector-embedded.yml around lines 25 - 27, The
workflow currently grants workflow-wide write perms; change the top-level
permissions mapping (the 'permissions' key) to "contents: read" and then scope
write permissions only on the job that needs them by adding a job-level
permissions block to the "open-pr" job with "contents: write" and
"pull-requests: write"; leave the "build" job with default/read-only
permissions. Update the entries for the 'permissions' key and add a per-job
permissions map under the "open-pr" job in the same YAML.


env:
PGVECTOR_VERSION: ${{ inputs.pgvector_version || 'v0.8.1' }}
Copy link
Copy Markdown

@coderabbitai coderabbitai Bot May 20, 2026

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

⚠️ Potential issue | 🟠 Major | ⚡ Quick win

🧩 Analysis chain

🌐 Web query:

For GitHub Actions workflows triggered by both workflow_dispatchandpush, is ${{ inputs.foo }}valid on push events, or should${{ github.event.inputs.foo }} be used for cross-event safety?

💡 Result:

The ${{ inputs.foo }} context is not valid for push events [1][2]. The inputs context is only available in workflows triggered by workflow_dispatch or workflow_call [1]. When a workflow is configured to be triggered by both workflow_dispatch and push, the inputs context will only be populated when the workflow is manually triggered via workflow_dispatch [1][2]. During a push event, any reference to ${{ inputs.foo }} will resolve to an empty string (or null/undefined, depending on usage), as inputs are not defined for the push event [3]. For cross-event safety, you should not rely on ${{ inputs.foo }} for push triggers. If your workflow logic depends on values that need to be provided differently depending on the trigger, you should use conditional logic (e.g., using the if keyword or shell scripts) to handle the absence of these inputs when the workflow is triggered by a push [1][4]. Accessing ${{ github.event.inputs.foo }} is also restricted to workflow_dispatch events and will similarly be empty during a push event [1][2]. In summary: 1. Inputs are not supported for push events [3][5]. 2. ${{ inputs.foo }} and ${{ github.event.inputs.foo }} will be empty/null when the workflow is triggered by push [1][2]. 3. You must design your workflow to account for the absence of these inputs during non-dispatch triggers [4].

Citations:

Design the workflow to handle missing inputs on push events.

Line 30 references ${{ inputs.pgvector_version }}, but the inputs context is only available on workflow_dispatch and workflow_call events, not on push events. Since this workflow triggers on both workflow_dispatch and push, the input will be unavailable during push, causing the expression to resolve to an empty string and fall back to the default 'v0.8.1'. While the fallback prevents failure, this design pattern is fragile if the workflow logic later depends on inputs being defined.

The suggested fix (using github.event.inputs) does not solve this issue—it has the same unavailability on push events. To properly handle cross-event triggers, either:

  • Limit the workflow to workflow_dispatch if inputs are required, or
  • Use conditional jobs or steps based on the trigger event (github.event_name) to handle different input availability.
🤖 Prompt for AI Agents 
Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.

In @.github/workflows/build-pgvector-embedded.yml at line 30, The workflow reads
PGVECTOR_VERSION from the inputs context (PGVECTOR_VERSION: ${{
inputs.pgvector_version || 'v0.8.1' }}) which only exists for
workflow_dispatch/workflow_call and is unavailable on push; change the workflow
to handle cross-event triggers by either restricting the trigger to
workflow_dispatch if an input is required, or guard usage with conditionals that
check github.event_name (e.g., only read inputs when github.event_name ==
'workflow_dispatch') and provide a reliable default for push; update any
jobs/steps that reference inputs or github.event.inputs accordingly so they run
only when inputs are present or fall back to the intended default behavior.


jobs:
build:
strategy:
fail-fast: false
matrix:
include:
- { platform: darwin-arm64, runner: macos-15 }
# macos-15-intel is GitHub's x86_64 macOS runner. The old macos-13
# (Intel) image was retired Dec 2025, so that label queues forever.
# macos-15-intel runs Intel on macOS 15 (available until ~Aug 2027,
# the last x86_64 macOS image GitHub will offer).
- { platform: darwin-x64, runner: macos-15-intel }
- { platform: linux-x64, runner: ubuntu-latest }
- { platform: linux-arm64, runner: ubuntu-24.04-arm }
runs-on: ${{ matrix.runner }}
steps:
- uses: actions/checkout@v4
- uses: actions/setup-node@v4
Comment on lines +48 to +49
Copy link
Copy Markdown

@coderabbitai coderabbitai Bot May 20, 2026

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

⚠️ Potential issue | 🟠 Major | ⚡ Quick win

🧩 Analysis chain

🏁 Script executed:

cat -n .github/workflows/build-pgvector-embedded.yml

Repository: lobu-ai/lobu

Length of output: 5276

🌐 Web query:

GitHub Actions security best practices pin commit SHA versions persist-credentials

💡 Result:

Securing GitHub Actions workflows involves two critical practices: pinning actions to immutable commit SHAs and managing credential persistence during checkout [1][2][3]. Pinning Actions to Commit SHAs Referencing actions by version tags (e.g., v4) is insecure because tags are mutable and can be repointed to malicious code by compromised maintainer accounts [4][5][6]. To ensure supply chain integrity, you should pin all third-party actions to their full-length commit SHA [1][7][3]. - Implementation: Replace tag references (e.g., uses: actions/checkout@v4) with the specific SHA (e.g., uses: actions/checkout@b4ffde6) [5][7]. - Documentation: Always include a comment indicating the version tag the SHA corresponds to for easier maintenance (e.g., # v4.1.1) [4][5]. - Automation: Use tools like Dependabot, Renovate, or specialized scanners (e.g., zizmor, pinact) to automate and manage these updates [4][3]. Managing persist-credentials By default, the actions/checkout action configures Git to persist the GitHub token in the local.git/config file [2][8]. This can expose credentials to subsequent steps in the workflow or leak through artifacts if they are improperly managed [2][8]. - Best Practice: Explicitly set persist-credentials to false in your checkout step unless you specifically require the token for Git operations later in the job [1][2]. - Implementation Example: - name: Checkout uses: actions/checkout@v4 with: persist-credentials: false General Security Recommendations - Least Privilege: Grant the GITHUB_TOKEN only the minimum required permissions (default to read-only at the top level and elevate only where necessary) [9][10][11]. - Avoid Secrets Exposure: Do not store sensitive values in plain text; use GitHub Secrets and leverage OpenID Connect (OIDC) for cloud authentication instead of long-lived static secrets [3][12][10]. - Monitor Risks: Periodically audit workflows for unpinned actions, broad permissions, and insecure trigger usage (e.g., pull_request_target) [1][2][13].

Citations:

Pin actions to commit SHAs and disable checkout credential persistence for supply chain security.

Lines 44, 45, 72, 82, and 83 use mutable tag refs (@v4) instead of commit SHAs, exposing the workflow to supply chain attacks if maintainer accounts are compromised. Additionally, checkout steps at lines 44 and 82 should explicitly set persist-credentials: false to prevent GitHub token exposure in the job context and artifacts.

Also applies to: 72-72, 83-83

🧰 Tools
🪛 zizmor (1.25.2)

[warning] 44-44: credential persistence through GitHub Actions artifacts (artipacked): does not set persist-credentials: false

(artipacked)

[error] 44-44: unpinned action reference (unpinned-uses): action is not pinned to a hash (required by blanket policy)

(unpinned-uses)

[error] 45-45: unpinned action reference (unpinned-uses): action is not pinned to a hash (required by blanket policy)

(unpinned-uses)

🤖 Prompt for AI Agents 
Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.

In @.github/workflows/build-pgvector-embedded.yml around lines 44 - 45, The
workflow uses mutable action tags (uses: actions/checkout@v4 and uses:
actions/setup-node@v4) and omits checkout credential hardening; replace each
mutable tag with the corresponding pinned commit SHA for actions/checkout and
actions/setup-node (every occurrence of uses: actions/checkout@v4 and uses:
actions/setup-node@v4) and add persist-credentials: false to each checkout step
(the steps using actions/checkout) to prevent GitHub token persistence; ensure
you update all occurrences mentioned in the diff so every checkout and
setup-node reference is pinned and every checkout step includes
persist-credentials: false.

with:
node-version: 22

- name: Install PostgreSQL 18 dev (macOS)
if: runner.os == 'macOS'
run: |
brew install postgresql@18
echo "PG_CONFIG=$(brew --prefix postgresql@18)/bin/pg_config" >> "$GITHUB_ENV"

- name: Install PostgreSQL 18 dev (Linux)
if: runner.os == 'Linux'
run: |
# Use the official pgdg setup script rather than a hand-rolled
# curl|gpg --dearmor pipe: the latter intermittently fails on GitHub
# runners with "gpg: cannot open '/dev/tty'". The script adds the
# repo + key non-interactively.
sudo apt-get update
sudo apt-get install -y postgresql-common
sudo /usr/share/postgresql-common/pgdg/apt.postgresql.org.sh -y
sudo apt-get install -y postgresql-server-dev-18 build-essential
echo "PG_CONFIG=/usr/lib/postgresql/18/bin/pg_config" >> "$GITHUB_ENV"

- name: Build pgvector artifact
run: |
chmod +x packages/pgvector-embedded/scripts/build.sh
PLATFORM=${{ matrix.platform }} packages/pgvector-embedded/scripts/build.sh

- uses: actions/upload-artifact@v4
with:
name: pgvector-${{ matrix.platform }}
path: packages/pgvector-embedded/prebuilt/${{ matrix.platform }}
if-no-files-found: error

open-pr:
needs: build
runs-on: ubuntu-latest
steps:
- uses: actions/checkout@v4
- uses: actions/download-artifact@v4
with:
path: /tmp/pgvector-artifacts

- name: Stage regenerated artifacts
run: |
for dir in /tmp/pgvector-artifacts/pgvector-*; do
platform="$(basename "$dir" | sed 's/^pgvector-//')"
dest="packages/pgvector-embedded/prebuilt/${platform}"
rm -rf "$dest"
mkdir -p "$dest"
cp -R "$dir/." "$dest/"
done

- name: Open PR
env:
GH_TOKEN: ${{ github.token }}
run: |
git config user.name "github-actions[bot]"
git config user.email "github-actions[bot]@users.noreply.github.com"
branch="chore/pgvector-embedded-${PGVECTOR_VERSION}-${GITHUB_RUN_ID}"
git switch -c "$branch"
git add packages/pgvector-embedded/prebuilt
if git diff --cached --quiet; then
echo "No artifact changes — nothing to PR."
exit 0
fi
git commit -m "chore(pgvector-embedded): rebuild prebuilt artifacts (${PGVECTOR_VERSION})"
git push -u origin "$branch"
gh pr create \
--base main \
--head "$branch" \
--title "chore(pgvector-embedded): rebuild prebuilt artifacts (${PGVECTOR_VERSION})" \
--body "Regenerated by the build-pgvector-embedded workflow (pgvector ${PGVECTOR_VERSION}, PG 18)."
7 changes: 7 additions & 0 deletions .github/workflows/ci.yml
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -202,11 +202,18 @@ jobs:
# the `import` condition and fails to load if dist is absent —
# which transitively breaks every integration file whose import
# graph touches `queue-helpers` / `worker-api` / `connector-catalog`.
#
# pgvector-embedded must build too: the test backend
# (src/__tests__/setup/embedded-postgres-backend.ts) imports
# `@lobu/pgvector-embedded`, so vite resolves its `import` condition
# (./dist/index.js) at transform time even though the embedded backend
# is never started when DATABASE_URL points at the external Postgres.
run: |
cd packages/core && bun run build && cd ../..
cd packages/connector-sdk && bun run build && cd ../..
cd packages/embeddings && bun run build && cd ../..
cd packages/connector-worker && bun run build && cd ../..
cd packages/pgvector-embedded && bun run build && cd ../..

- name: Verify Postgres health (fail fast if pgvector setup is broken)
run: |
Expand Down
2 changes: 2 additions & 0 deletions AGENTS.md
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -106,6 +106,8 @@ Rules for agents:
- `packages/cli/src/commands/_lib/connector-run-cmd.ts` — `browser-mirror`, `devtools-active-port`, `executeCompiledConnector`.
- `packages/cli/src/commands/_lib/apply/desired-state.ts` — `yaml` (loaded only on YAML inputs).
- `packages/cli/src/commands/memory/_lib/browser-auth-cmd.ts` — `decryptChromeCookiesMacOS`, `playwright/chromium`.
- `packages/server/src/server.ts` — `./embedded-runtime` is statically imported, but `./server-lifecycle` is lazy: its transitive imports read env at module-eval, and the embedded branch only finalises DATABASE_URL during `main()`. Loading the lifecycle eagerly would snapshot a stale env.
- `packages/server/src/embedded-runtime.ts` — `embedded-postgres` + `@lobu/pgvector-embedded` (and `postgres`) are lazy so the external/prod path (postgres:// URL) never resolves or loads the ~145MB embedded-Postgres binary even though it sits in node_modules. Only reached when DATABASE_URL is a path/file://.
- **Tests** — `await import(...)` inside `beforeAll` / `beforeEach` / `test()` is allowed (load after `vi.mock(...)`); this is the vitest pattern, not a production exemption.

## Scope discipline and branch hygiene
Expand Down
2 changes: 1 addition & 1 deletion Makefile
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -34,7 +34,7 @@ typecheck:
# Build all TypeScript packages in dependency order
build-packages:
@echo "📦 Building all TypeScript packages..."
@for pkg in core connector-sdk agent-worker openclaw-plugin embeddings connector-worker promptfoo-provider; do \
@for pkg in core pgvector-embedded connector-sdk agent-worker openclaw-plugin embeddings connector-worker promptfoo-provider; do \
echo " 📦 Building packages/$$pkg..."; \
( cd packages/$$pkg && bun run build ) || exit $$?; \
done
Expand Down
60 changes: 44 additions & 16 deletions bun.lock
View file
Open in desktop

Some generated files are not rendered by default. Learn more about how customized files appear on GitHub.

Loading
Loading