Skip to content

fix(server): flush WAL after /api/embed so search sees new embeddings (#1149)#1359

Merged
magyargergo merged 1 commit into
abhigyanpatwari:mainfrom
azizur100389:fix/embed-checkpoint-flush
May 6, 2026
Merged

fix(server): flush WAL after /api/embed so search sees new embeddings (#1149)#1359
magyargergo merged 1 commit into
abhigyanpatwari:mainfrom
azizur100389:fix/embed-checkpoint-flush

Conversation

@azizur100389

Copy link
Copy Markdown
Contributor

Summary

  • After /api/embed completes, the embedding rows remain in the WAL because the server keeps the DB connection open for other routes.
  • Subsequent /api/search requests return zero semantic results until an unrelated operation forces a WAL replay.
  • Adds an explicit CHECKPOINT inside the withLbugDb callback after the embedding pipeline finishes, matching the pattern already documented in closeLbug().

Root cause

The CLI path (analyze --embeddings) flushes the WAL via closeLbug() during process.exit(). The server path (/api/embed) uses a shared singleton connection that stays open -- withLbugDb only releases the session lock, it never checkpoints or closes. The closeLbug() comment is explicit: "This is especially critical after embedding writes, which generate large amounts of WAL data."

Test plan

  • npx tsc --noEmit -- clean
  • npx vitest run test/unit/rate-limit.test.ts -- 16/16 pass (source-grep confirms api.ts structure)
  • Prettier clean
  • Manual: POST /api/embed -> poll completion -> POST /api/search returns semantic results (reviewer verification)

Closes #1149

…abhigyanpatwari#1149)

The CLI path (analyze --embeddings) flushes the WAL via closeLbug()
during process exit, but /api/embed uses a shared DB connection that
stays open for other routes. Without an explicit CHECKPOINT the
embedding rows remain in the WAL and subsequent /api/search requests
return zero semantic results until an unrelated operation forces a
WAL replay.

Issue a CHECKPOINT inside the withLbugDb callback after the embedding
pipeline completes, matching the pattern already documented in
closeLbug() ("especially critical after embedding writes, which
generate large amounts of WAL data").
@vercel

vercel Bot commented May 5, 2026

Copy link
Copy Markdown

@azizur100389 is attempting to deploy a commit to the NexusCore Team on Vercel.

A member of the Team first needs to authorize it.

@magyargergo

Copy link
Copy Markdown
Collaborator

Nice find! There was one more close. We need to consolidate them. Would you like to work on this?

@magyargergo magyargergo merged commit 46e4c97 into abhigyanpatwari:main May 6, 2026
29 of 30 checks passed
@azizur100389

Copy link
Copy Markdown
Contributor Author

Thanks for the merge! Yes, happy to take this on.

I've had a look — api.ts currently has two closeLbug() calls (DELETE /api/repo before removing files, and the graceful-shutdown handler) plus the inline CHECKPOINT I added in the embed route. The other withLbugDb call sites are all reads so shouldn't need flushing.

Could you point me at which "one more close" you have in mind? Happy to scope the consolidation once I know the starting point.

@magyargergo

Copy link
Copy Markdown
Collaborator

Could you point me at which "one more close" you have in mind? Happy to scope the consolidation once I know the starting point.

I meant the one you found and fixed.

@azizur100389

Copy link
Copy Markdown
Contributor Author

Got it — thanks for clarifying. So the consolidation is about unifying the CHECKPOINT / closeLbug() patterns across api.ts rather than hunting for another missing flush. I'll open an issue to scope it and take it from there.

@magyargergo

Copy link
Copy Markdown
Collaborator

Got it — thanks for clarifying. So the consolidation is about unifying the CHECKPOINT / closeLbug() patterns across api.ts rather than hunting for another missing flush. I'll open an issue to scope it and take it from there.

Feel free to collaborate with @ChrisGVE, I mentioned this here too - #1301 (comment)

@azizur100389

Copy link
Copy Markdown
Contributor Author

Thanks for the pointer! I've opened #1376 and have a branch ready — I extracted a helper that wraps the CHECKPOINT try/catch and wired it into closeLbug, withLbugDb's retry cleanup, and doInitLbug's reconnect path. Will rename it to safeClose per your suggestion and tag @ChrisGVE on the issue.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

2 participants