Skip to content

fix(etl): flush remaining items before totalProcessed + use ORM for completion status #2413

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Merged
andrew-bierman merged 1 commit into from
May 13, 2026
Merged

fix(etl): flush remaining items before totalProcessed + use ORM for completion status #2413

Changes from all commits
Commits
Show all changes
1 commit
Select commit
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
45 changes: 21 additions & 24 deletions packages/api/src/services/etl/processCatalogEtl.ts
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -130,42 +130,39 @@ export async function processCatalogETL({

console.log(`🔍 [TRACE] Streaming complete - processing remaining batches`);

// Flush remaining items after the stream ends
const remainingItems = validItemsBatch.length + invalidItemsBatch.length;
// Flush remaining items BEFORE updating totalProcessed so that if a flush throws,
// totalProcessed isn't inflated while valid/invalid counts stay null.
const remainingValid = validItemsBatch.length;
const remainingInvalid = invalidItemsBatch.length;

if (remainingValid > 0) {
console.log(`🔍 [TRACE] Processing valid items batch - size: ${remainingValid}`);
await processValidItemsBatch({ jobId, items: validItemsBatch, env });
}

if (remainingInvalid > 0) {
console.log(`🔍 [TRACE] Processing invalid items batch - size: ${remainingInvalid}`);
await processLogsBatch({ jobId, logs: invalidItemsBatch, env });
}

const remainingItems = remainingValid + remainingInvalid;
if (remainingItems > 0) {
await db
.update(etlJobs)
.set({ totalProcessed: sql`COALESCE(${etlJobs.totalProcessed}, 0) + ${remainingItems}` })
.where(eq(etlJobs.id, jobId));
}

if (validItemsBatch.length > 0) {
console.log(`🔍 [TRACE] Processing valid items batch - size: ${validItemsBatch.length}`);
await processValidItemsBatch({
jobId,
items: validItemsBatch,
env,
});
}

if (invalidItemsBatch.length > 0) {
console.log(`🔍 [TRACE] Processing invalid items batch - size: ${invalidItemsBatch.length}`);
await processLogsBatch({
jobId,
logs: invalidItemsBatch,
env,
});
}

const totalRows = rowIndex;

// Use raw SQL to avoid neon-http enum serialization issues with Drizzle ORM.
// Mark completed using Drizzle ORM (same as the failed path below) — avoids the
// silent failure that ::etl_job_status raw SQL casts produced in some Neon HTTP driver versions.
// Isolated try-catch so a transient DB hiccup here doesn't cascade to status='failed'.
try {
await db.execute(
sql`UPDATE etl_jobs SET status = 'completed'::etl_job_status, completed_at = NOW() WHERE id = ${jobId}`,
);
await db
.update(etlJobs)
.set({ status: 'completed', completedAt: new Date() })
.where(eq(etlJobs.id, jobId));
} catch (completionErr) {
console.error(
`[ETL] Failed to mark job ${jobId} completed — will be reset by stuck-job sweep:`,
Expand Down
Loading