Skip to content

fix(cli): repair corrupted LanceDB indexes #12914

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Open
lambertjosh wants to merge 1 commit into
base: main
Choose a base branch
from
+43 −6
Open

fix(cli): repair corrupted LanceDB indexes #12914

Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
1 commit
Select commit
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
5 changes: 5 additions & 0 deletions .changeset/repair-missing-lance-files.md
View file
Open in desktop
Original file line number Diff line number Diff line change
@@ -0,0 +1,5 @@
---
"@kilocode/cli": patch
---

Repair code indexes with missing LanceDB data files and prevent startup cleanup from invalidating active index readers
25 changes: 19 additions & 6 deletions packages/kilo-indexing/src/indexing/vector-store/lancedb-vector-store.ts
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -277,6 +277,15 @@ export class LanceDBVectorStore implements IVectorStore {
}

async initialize(): Promise<boolean> {
return this.init(true)
}

private missing(error: unknown): boolean {
const message = error instanceof Error ? error.message : String(error)
return message.includes("Object at location ") && message.includes(".lance not found:")

@kilo-code-bot kilo-code-bot Bot Aug 5, 2026

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

SUGGESTION: Consider noting the LanceDB version this error text is matched against

missing() keys off LanceDB's exact error wording (Object at location ... .lance not found:), which is the only signal available since lancedb doesn't expose structured error codes — the gating is reasonable. The risk is that a future @lancedb/lancedb upgrade rewords this message, at which point self-repair silently stops triggering and init falls back to throwing. A short comment pinning the matched format to the current dependency version (0.26.x) would tell future upgraders to re-verify this string.

Reply with @kilocode-bot fix it to have Kilo Code address this issue.

}

private async init(repair: boolean): Promise<boolean> {
try {
await this.closeConnect()
const db = await this.getDb()
Expand Down Expand Up @@ -321,7 +330,6 @@ export class LanceDBVectorStore implements IVectorStore {
await this._dropTableIfExists(db, this.metadataTableName)
await this._createVectorTable(db)
await this._createMetadataTable(db)
this.optimizeTable()

log.info("LanceDB store reinitialized for embedding profile change", {
workspacePath: this.workspacePath,
Expand All @@ -332,7 +340,6 @@ export class LanceDBVectorStore implements IVectorStore {

return true
}
this.optimizeTable()
log.info("LanceDB store initialized", {
workspacePath: this.workspacePath,
dbPath: this.dbPath,
Expand All @@ -341,6 +348,15 @@ export class LanceDBVectorStore implements IVectorStore {
})
return false
} catch (error) {
if (repair && this.missing(error)) {
log.warn("Rebuilding LanceDB store with missing data files", {
workspacePath: this.workspacePath,
dbPath: this.dbPath,
error,
})
await this.deleteCollection()
return this.init(false)
}
log.error("Failed to initialize LanceDB store", { error })
throw new Error(`Failed to initialize LanceDB store: ${(error as Error).message}`, { cause: error })
}
Expand Down Expand Up @@ -577,10 +593,7 @@ export class LanceDBVectorStore implements IVectorStore {
try {
const table = await this.getTable()

await table.optimize({
cleanupOlderThan: new Date(),
deleteUnverified: false,
})
await table.optimize()

@kilo-code-bot kilo-code-bot Bot Aug 5, 2026

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

SUGGESTION: With startup optimization removed, version cleanup no longer has a periodic caller

Removing the this.optimizeTable() calls from init() makes sense — the old cleanupOlderThan: new Date() could delete data files that concurrent readers still referenced. But optimizeTable() is now only called from clearCollection(), and every upsert/delete creates a new LanceDB on-disk version, so long-lived workspaces that never clear the collection will accumulate versions unboundedly (the doc comment above still says "Should be called periodically to prevent unbounded disk space growth"). Since table.optimize() with default retention (~7 days) is far safer for concurrent readers than the old call, it may be worth keeping a throttled optimize (e.g. after a successful init, or driven by the orchestrator) — or, if running only on clear is the intended design, updating the doc comment to say so.

Reply with @kilocode-bot fix it to have Kilo Code address this issue.

} catch (error) {
log.error("Failed to optimize table", { error })
}
Expand Down
19 changes: 19 additions & 0 deletions packages/kilo-indexing/test/kilocode/indexing/vector-store/lancedb-vector-store.test.ts
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -36,6 +36,7 @@ const mockTable = {
remove: mock(),
createIndex: mock(),
dropIndex: mock(),
optimize: mock().mockResolvedValue(undefined),
indexes: [],
columns: [],
primaryKey: "id",
Expand Down Expand Up @@ -100,6 +101,7 @@ const allMocks = [
mockTable.remove,
mockTable.createIndex,
mockTable.dropIndex,
mockTable.optimize,
mockTable.batch,
mockTable.distanceRange,
mockDb.openTable,
Expand Down Expand Up @@ -132,6 +134,7 @@ function resetAllMocks() {
mockTable.openTable.mockResolvedValue(undefined)
mockTable.search.mockReturnThis()
mockTable.distanceRange.mockReturnThis()
mockTable.optimize.mockResolvedValue(undefined)
mockDb.openTable.mockResolvedValue(mockTable)
mockDb.createTable.mockResolvedValue(mockTable)
mockDb.dropTable.mockResolvedValue(undefined)
Expand Down Expand Up @@ -243,6 +246,7 @@ describe("LocalVectorStore", () => {
store["_getMetadataValue"] = mock().mockResolvedValue("2")
const result = await store.initialize()
expect(result).toBe(false)
expect(mockTable.optimize).not.toHaveBeenCalled()
})

test("recreates an index using the legacy payload schema", async () => {
Expand All @@ -268,6 +272,20 @@ describe("LocalVectorStore", () => {
expect(mockDb.createTable).not.toHaveBeenCalled()
})

test("rebuilds an index whose manifest references a missing data file", async () => {
const error = new Error(
"Failed to get next batch from stream: LanceError(IO): Object at location metadata.lance/data/missing.lance not found: os error 2",
)
spyOn(fs, "existsSync").mockReturnValue(true)
const remove = spyOn(fs, "rmSync").mockImplementation(() => {})
mockDb.tableNames.mockResolvedValueOnce(["vector", "metadata"]).mockResolvedValueOnce([])
store["_getStoredVectorSize"] = mock().mockRejectedValue(error)

expect(await store.initialize()).toBe(true)
expect(remove).toHaveBeenCalledWith(store["dbPath"], { recursive: true, force: true })
expect(mockDb.createTable).toHaveBeenCalledTimes(2)
})

test("does not recreate when profile metadata cannot be read", async () => {
mockTable.countRows.mockResolvedValue(1)
store["_getStoredVectorSize"] = mock().mockResolvedValue(vectorSize)
Expand Down Expand Up @@ -515,6 +533,7 @@ describe("LocalVectorStore", () => {
mockTable.delete.mockResolvedValue(undefined)
await expect(store.clearCollection()).resolves.toBeUndefined()
expect(mockTable.delete).toHaveBeenCalledWith("true")
expect(mockTable.optimize).toHaveBeenCalledWith()
})

test("should warn if metadata table clear fails", async () => {
Expand Down
Loading