From 43f0458ae464e4cab3293ba071797134acf6a50a Mon Sep 17 00:00:00 2001 From: sebi Date: Fri, 14 Aug 2026 13:08:14 -0500 Subject: [PATCH 1/5] =?UTF-8?q?feat(jobs):=20backfill()=20=E2=80=94=20a=20?= =?UTF-8?q?factory=20over=20job(),=20not=20a=20ninth=20primitive?= MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit - new packages/jobs/src/backfill.ts: backfill({ name, source, batch, handle }) returns a JobHandle, so a table sweep inherits .enqueue(), the retry policy, the worker's cancellation, the dead-letter path and its manifest row - the pass iterates entity's inBatches() — one statement per page, keyset — with every page inside its own step.run named batch:, so a killed attempt resumes on the page it stopped at and re-reads none before it - a step persists the CURSOR and a row count, never the page: steps.ts retains a completed step's output for the whole run - the iteration is rebuilt whenever it disagrees with the checkpoints, because retryFromStep re-opens one step in the middle of a finished run - a checkpoint read back is checked, not trusted — step.run replays it through an unchecked cast and an absent cursor would restart the pass at the top - batch size refused at declaration; idempotencyKey is the backfill's name 10 tests, 8 mutations verified red. verify 13/17 (4 honest skips), reference-app gate 10/17 (7 pinned) — both baselines unchanged. Co-Authored-By: Claude --- CHANGELOG.md | 15 ++ CLAUDE.md | 2 + packages/jobs/CLAUDE.md | 15 ++ packages/jobs/README.md | 36 +++ packages/jobs/src/backfill.test.ts | 343 +++++++++++++++++++++++++++++ packages/jobs/src/backfill.ts | 213 ++++++++++++++++++ packages/jobs/src/index.ts | 7 + 7 files changed, 631 insertions(+) create mode 100644 packages/jobs/src/backfill.test.ts create mode 100644 packages/jobs/src/backfill.ts diff --git a/CHANGELOG.md b/CHANGELOG.md index 76b06c5ba..aeb85d26e 100644 --- a/CHANGELOG.md +++ b/CHANGELOG.md @@ -84,6 +84,21 @@ Semver applies from 1.0.0. A breaking change to a documented API needs a major ### Added +- **`backfill()` — one pass over a table, declared as a `job`.** `@ultimat3/jobs` gains a factory, not a ninth primitive: a backfill is durable work with an input schema, a retry policy, an idempotency key and a queue, so `backfill({ name, source, batch, handle })` *returns* a `JobHandle` and inherits `.enqueue()`, the worker's cancellation, the dead-letter path, `x jobs show` and its manifest row with no line of its own. Same rule `llm()` follows: a new capability arrives as a factory over an existing primitive. + + ```ts + export const rewriteSlugs = backfill({ + name: 'rewrite-slugs', + batch: 1_000, + source: () => db.posts.where({ published: true }), + async handle({ rows }) { + await db.posts.upsertAll(rows.map(slugged), { onConflict: ['id'] }); + }, + }); + ``` + + `source` is read through `inBatches()` — one statement per page, keyset, never OFFSET — and every page is handled inside its own `step.run`, named `batch:0`, `batch:1`, … So a run killed mid-pass resumes on the page it stopped at: the completed steps replay from storage without touching the database, and the iteration is opened at the cursor they leave behind. What a step persists is that cursor and a row count, **never the page** — `steps.ts` hands a completed step's output back for the whole run, so checkpointing rows would retain every row already processed until the job ended. The body is handed no `step` for the same reason step names are positional: a name minted inside it would collide with itself on the second batch. `idempotencyKey` is the backfill's name, so re-enqueueing a live pass is the same pass. + - **The destructive-SQL rail: a migration that destroys data must say so, and `x verify` refuses one that does not.** `x db gen` now writes `-- destructive: true` into any migration whose `up` drops a table, drops a column, truncates or retypes; `x verify`'s `drift` step reads the committed files back and fails an unmarked one with the new `X_MIGRATION_DESTRUCTIVE`. The strong-migrations idea, enforced rather than documented — a drop is allowed, an *undeclared* drop is not. ```text diff --git a/CLAUDE.md b/CLAUDE.md index 6c0627ed0..38ed4e879 100644 --- a/CLAUDE.md +++ b/CLAUDE.md @@ -115,6 +115,8 @@ Everything in the framework is one of these. **If a feature doesn't fit one of t **`llm()` is an action factory, not a ninth primitive — decided 2026-08.** A model call is a server-authoritative operation with an input schema, an output schema and a policy, which is the definition of an `action`; so `llm()` ([`packages/ai/src/llm.ts`](packages/ai/src/llm.ts)) *returns* one. That is what gives a model call `.tool()`, `.openapi()`, `.client()`, `.job()` and `.contract()` for free, one authz object across every surface, and a place in the manifest — none of which a ninth primitive would have inherited. The rule generalises: a new capability arrives as a **factory over an existing primitive**, never as a new kind of thing. +**`backfill()` is a job factory — the rule's second instance, decided 2026-08.** A one-pass sweep over a table is durable background work with an input schema, a retry policy, an idempotency key and a queue, which is the definition of a `job`; so `backfill()` ([`packages/jobs/src/backfill.ts`](packages/jobs/src/backfill.ts)) *returns* one, and inherits `.enqueue()`, the worker's cancellation, the dead-letter path, `x jobs show` and its manifest row. The pass is `inBatches()` — one statement per page — with every page in its own `step.run`, so a killed attempt resumes on the page it stopped at. What a step persists is a cursor, never the page. + ## Non-negotiables - **Bun only.** No Node-specific APIs unless via `node:` and unavoidable, and then with a comment saying why. diff --git a/packages/jobs/CLAUDE.md b/packages/jobs/CLAUDE.md index 01de0b90a..0873d8c31 100644 --- a/packages/jobs/CLAUDE.md +++ b/packages/jobs/CLAUDE.md @@ -122,6 +122,20 @@ Tier 3. The `job` + `task` primitives, durable steps, transactional outbox, queu `accept` phase while it runs, handed back in the teardown's `finally` so a `release()` that threw still gives it up, `isLeader` cleared there too because a lock this process could not hand back is never treated as still held. +- **`backfill()` is a FACTORY over `job()`, never a ninth primitive.** Same rule `llm()` follows + in `@ultimat3/ai`: a new capability arrives as a factory over an existing primitive, so a + backfill inherits `.enqueue()`, the retry policy, the cancellation, the dead-letter path and + its manifest row instead of re-declaring them. Three things in `backfill.ts` are load-bearing + and none is an implementation detail: a step's persisted output is the CURSOR and a row count + and never the page, because `steps.ts` retains a completed step's output for the whole run and + a page there is every processed row held until the job ends; step names are positional + (`batch:`) so the next attempt mints the same key the last one wrote, which is also why + `handle` is handed no `step` — a name minted inside a per-batch body collides with itself on + batch 2; and the iteration is rebuilt whenever `batches.cursor` disagrees with the checkpoint, + because `retryFromStep` re-opens ONE step in the middle of a finished run and the cursor then + jumps over a page the live iteration never read. A checkpoint READ back is checked rather than + trusted — `step.run` replays it through an unchecked `as T`, and an absent cursor is not `null`, + so the pass would silently reopen the source at the top and walk the whole table again. - Suspension is control flow: `StepSuspension` -> `nack({ countsAsAttempt: false })`. Never log it as an error, never let it burn an attempt. - Step results are persisted BEFORE the step returns. Keep it that way or replay breaks. @@ -164,6 +178,7 @@ picture from the other side. | File | Owns | |---|---| | `job.ts` | the `job()` primitive + registry + the handle's fluent surface + `registerJob` | +| `backfill.ts` | `backfill()` — a factory over `job()`: the batched pass and its cursor checkpoints | | `register.ts` | `registerJobs`/`registerTasks` over a module namespace + the registrar announcements | | `describe.ts` | the JSON projection one handle emits; `describeJobs()` is a map over it | | `steps.ts` | `StepStore`, `StepApi`, memoized-replay executor, `StepSuspension` | diff --git a/packages/jobs/README.md b/packages/jobs/README.md index c69c9bf8e..f714eb286 100644 --- a/packages/jobs/README.md +++ b/packages/jobs/README.md @@ -103,6 +103,42 @@ Step names are the replay key, so they must be deterministic and unique in a run duplicate is `X_STEP_DUPLICATE`, not a silent overwrite. Suspension is control flow (`StepSuspension`), never a failure: it does not burn a retry attempt. +## Backfills are jobs + +`backfill()` is a **factory over `job()`**, not a ninth primitive — one pass over every row a +chain matches, with the retry policy, the queue, the cancellation and the dead-letter path a job +already has. + +```ts +import { backfill } from '@ultimat3/jobs'; +import { db } from '@postly/db'; + +export const rewriteSlugs = backfill({ + name: 'rewrite-slugs', // REQUIRED: a durable key + batch: 1_000, // rows per statement and per step + source: () => db.posts.where({ published: true }), + async handle({ rows }) { + await db.posts.upsertAll(rows.map(slugged), { onConflict: ['id'] }); + }, +}); + +await rewriteSlugs.enqueue({}); // it is a JobHandle +``` + +The source is read through `inBatches()` — one statement per page, keyset, never OFFSET — and +each page is handled inside its own `step.run`, named `batch:0`, `batch:1`, … A run killed +mid-pass therefore **resumes on the page it stopped at**: completed steps replay from storage +without a statement, and the iteration reopens at the cursor they left behind. + +| Rule | Why | +|---|---| +| the checkpoint is a cursor and a count, never the page | a completed step's output is retained for the whole run — checkpointing rows would hold every processed row until the job ends | +| `handle` is given no `step` | a step name minted inside it collides with itself on batch 2 (`X_STEP_DUPLICATE`) | +| `handle` is given a `signal` | the run's deadline composed with the batch's own ceiling — hand it to whatever the body calls | +| `handle` runs at least once per page | an attempt cancelled between the last row and the checkpoint replays it — write through `upsertAll` / `updateWhere`, never `count + 1` | +| `idempotencyKey` is the backfill's name | re-enqueueing a live pass is the same pass, not a second writer on one table | +| `batch` is refused at declaration | `0`, `1.5` and a `NaN` from an env var fail the build, not the fourth attempt | + ## The deadline cancels A job's `timeout` aborts `ctx.signal` **before** it fails the attempt, because the nack that diff --git a/packages/jobs/src/backfill.test.ts b/packages/jobs/src/backfill.test.ts new file mode 100644 index 000000000..3d8325baf --- /dev/null +++ b/packages/jobs/src/backfill.test.ts @@ -0,0 +1,343 @@ +// `backfill()` is a factory over `job()`, so what needs pinning here is not that a job exists — +// `job.test.ts` owns that — but the pass: every row visited once, a killed attempt resuming on the +// page it stopped at WITHOUT re-reading the ones before it, and a checkpoint that is a cursor +// rather than the rows behind it. The source is a real `@ultimat3/entity` chain over a real +// `memoryRepo`, so the cursors, the keyset paging and `inBatches()`'s own refusals are the shipped +// ones and not a fixture that agrees with the factory by construction. + +import { beforeEach, describe, expect, test } from 'bun:test'; +import type { Ctx } from '@ultimat3/core'; +import { createContext, isUltimateError } from '@ultimat3/core'; +import type { BatchIterator, ReadBuilder, Repo } from '@ultimat3/entity'; +import { entity, memoryRepo, tableFor, text, uuid } from '@ultimat3/entity'; +import { backfill, DEFAULT_BACKFILL_BATCH } from './backfill'; +import { getJob, isJobHandle, resetJobs } from './job'; +import type { StepRecord, StepStore } from './steps'; +import { createMemoryStepStore, createStepRunner } from './steps'; + +const rows = entity('backfill_test_rows', { + columns: { id: uuid().primaryKey(), orgId: uuid(), title: text({ max: 40 }) }, +}); + +type Row = typeof rows.$row; + +const ORG = '00000000-0000-7000-8000-0000000000a1'; +const OTHER = '00000000-0000-7000-8000-0000000000a2'; + +/** Sortable ids, ten apart, so the keyset cursor between two pages is a real position. */ +const id = (index: number): string => + `00000000-0000-7000-8000-0000000001${String(index).padStart(2, '0')}`; + +const row = (index: number, orgId: string = ORG): Row => ({ + id: id(index), + orgId, + title: `row-${index}`, +}); + +/** Ten rows for the org under test, two more no pass may ever touch. */ +const SEED: readonly Row[] = [ + ...Array.from({ length: 10 }, (_unused, index) => row(index * 10)), + row(900, OTHER), + row(910, OTHER), +]; + +/** What the wrappers below record: statements sent, iterations opened, iterations closed. */ +interface Watch { + reads: number; + opens: number; + closes: number; +} + +const newWatch = (): Watch => ({ reads: 0, opens: 0, closes: 0 }); + +/** `cursor` is a getter on the real iterator, so it is forwarded rather than spread-copied. */ +const watchedIterator = (iterator: BatchIterator, watch: Watch): BatchIterator => { + const close = async (): Promise => { + watch.closes += 1; + await iterator.close(); + }; + return { + get cursor(): string | null { + return iterator.cursor; + }, + [Symbol.asyncIterator]: () => iterator[Symbol.asyncIterator](), + close, + [Symbol.asyncDispose]: close, + }; +}; + +/** `after()` returns a NEW chain, so the wrapper has to survive it — the factory calls both. */ +const watchedChain = (chain: ReadBuilder, watch: Watch): ReadBuilder => ({ + ...chain, + after: (cursor) => watchedChain(chain.after(cursor), watch), + inBatches: (size) => { + watch.opens += 1; + return watchedIterator(chain.inBatches(size), watch); + }, +}); + +const countingRepo = (repo: Repo, watch: Watch): Repo => ({ + ...repo, + findMany: async (args) => { + watch.reads += 1; + return repo.findMany(args); + }, +}); + +interface Harness { + readonly watch: Watch; + readonly store: StepStore; + /** Titles handed to the body, one entry per batch it actually ran. */ + readonly seen: readonly (readonly string[])[]; + /** Every batch index the body should reject on. */ + failOn: ReadonlySet; + run(): Promise; + steps(): Promise; +} + +const RUN_ID = 'run-backfill-1'; + +const ctx: Ctx = createContext(); + +/** + * One backfill over `SEED`, driven through a real step runner rather than a worker: this file is + * about the pass and its checkpoints, and `execute.ts` already owns what a driver does with the + * outcome. Every `run()` is a fresh attempt against the SAME store and run id, which is exactly + * what a retry is. + */ +const harness = (options: { batch?: number; name?: string } = {}): Harness => { + const watch = newWatch(); + const store = createMemoryStepStore(); + const seen: string[][] = []; + const table = tableFor(rows, countingRepo(memoryRepo(rows, SEED), watch)); + const state: Harness = { + watch, + store, + seen, + failOn: new Set(), + async run() { + const runner = createStepRunner({ runId: RUN_ID, jobName: 'backfill', store }); + return handle.run({ + input: {}, + step: runner.step, + ctx, + attempt: 1, + jobId: 'job-1', + runId: RUN_ID, + }); + }, + steps: () => store.list(RUN_ID), + }; + const handle = backfill({ + name: options.name ?? 'rewrite-titles', + ...(options.batch === undefined ? {} : { batch: options.batch }), + source: () => watchedChain(table.where({ orgId: ORG }), watch), + handle: ({ rows: page, index }) => { + if (state.failOn.has(index)) throw new Error(`batch ${String(index)} failed`); + seen.push(page.map((entry) => entry.title)); + }, + }); + return state; +}; + +beforeEach(() => { + resetJobs(); +}); + +describe('the factory', () => { + test('returns a registered job handle keyed by the declared name', () => { + const handle = backfill({ + name: 'rewrite-titles', + queue: 'maintenance', + retry: { attempts: 7, backoff: 'fixed' }, + source: () => tableFor(rows, memoryRepo(rows, SEED)).where({ orgId: ORG }), + handle: () => undefined, + }); + + expect(isJobHandle(handle)).toBe(true); + expect(getJob('rewrite-titles')).toBe(handle); + expect(handle.queue).toBe('maintenance'); + expect(handle.retry.attempts).toBe(7); + // One live run per name: a second enqueue while the pass is going is the same pass. + expect(handle.idempotencyKeyFor({})).toBe('rewrite-titles'); + }); + + test('refuses a batch size that is not a whole number of rows, where it was written', () => { + for (const batch of [0, -1, 1.5, Number.NaN]) { + let thrown: unknown; + try { + backfill({ + name: `bad-${String(batch)}`, + batch, + source: () => tableFor(rows, memoryRepo(rows, SEED)).where({ orgId: ORG }), + handle: () => undefined, + }); + } catch (error) { + thrown = error; + } + expect(isUltimateError(thrown)).toBe(true); + expect(isUltimateError(thrown) ? thrown.code : undefined).toBe('X_INVARIANT'); + // Refused before `job()` ran, so the name is still free for the corrected definition. + expect(getJob(`bad-${String(batch)}`)).toBeUndefined(); + } + }); + + test('defaults to one statement per DEFAULT_BACKFILL_BATCH rows', async () => { + const test10 = harness(); + await test10.run(); + // Ten rows, one default-sized batch: the default is what the pass used. + expect(DEFAULT_BACKFILL_BATCH).toBeGreaterThan(10); + expect(test10.seen.length).toBe(1); + expect(test10.watch.reads).toBe(1); + }); +}); + +describe('one pass', () => { + test('visits every matching row once, in key order, one statement per batch', async () => { + const pass = harness({ batch: 3 }); + + const report = await pass.run(); + + expect(pass.seen).toEqual([ + ['row-0', 'row-10', 'row-20'], + ['row-30', 'row-40', 'row-50'], + ['row-60', 'row-70', 'row-80'], + ['row-90'], + ]); + expect(report).toEqual({ name: 'rewrite-titles', batches: 4, rows: 10 }); + // Four batches, four statements: the exhausted source costs no extra read. + expect(pass.watch.reads).toBe(4); + // One iteration for the whole pass, closed once on the way out. + expect(pass.watch.opens).toBe(1); + expect(pass.watch.closes).toBe(1); + }); + + test('an empty source is one step, no body call and a zero report', async () => { + const pass = harness({ batch: 3, name: 'empty-pass' }); + const table = tableFor(rows, memoryRepo(rows, [])); + const empty = backfill({ + name: 'nothing-to-do', + batch: 3, + source: () => table.where({ orgId: ORG }), + handle: () => expect.unreachable('an empty source has no batch to hand over'), + }); + const runner = createStepRunner({ runId: RUN_ID, jobName: 'nothing', store: pass.store }); + + const report = await empty.run({ + input: {}, + step: runner.step, + ctx, + attempt: 1, + jobId: 'job-2', + runId: RUN_ID, + }); + + expect(report).toEqual({ name: 'nothing-to-do', batches: 0, rows: 0 }); + expect((await pass.steps()).map((record) => record.name)).toEqual(['batch:0']); + }); + + test('checkpoints a bounded cursor and a count — never the page', async () => { + const pass = harness({ batch: 3 }); + + await pass.run(); + + const steps = await pass.steps(); + // Deterministic and positional: a replay finds a step by the name the next attempt mints. + expect(steps.map((record) => record.name)).toEqual([ + 'batch:0', + 'batch:1', + 'batch:2', + 'batch:3', + ]); + const checkpoints = steps.map((record) => record.output as Record); + for (const [index, output] of checkpoints.entries()) { + expect(steps[index]?.status).toBe('completed'); + // `steps.ts` retains a completed step's output for the whole run, so a checkpoint carrying + // its page would hold every row the pass has touched until the job ended. + expect(Object.keys(output).sort()).toEqual(['cursor', 'rows']); + expect(typeof output.rows).toBe('number'); + expect(output.cursor === null || typeof output.cursor === 'string').toBe(true); + } + // The pass ends because the source did, and that is what the last checkpoint records. + expect(checkpoints.at(-1)?.cursor).toBeNull(); + }); +}); + +describe('resume', () => { + test('a failed attempt resumes on its own page and re-reads none before it', async () => { + const pass = harness({ batch: 3 }); + pass.failOn = new Set([1]); + + await expect(pass.run()).rejects.toThrow('batch 1 failed'); + + // Two statements: the batch that worked and the one that failed. + expect(pass.watch.reads).toBe(2); + expect(pass.seen).toEqual([['row-0', 'row-10', 'row-20']]); + expect(pass.watch.closes).toBe(1); + + pass.failOn = new Set(); + const report = await pass.run(); + + // Batch 0 came from storage, so the second attempt read three pages, not four. + expect(pass.watch.reads).toBe(5); + expect(pass.seen).toEqual([ + ['row-0', 'row-10', 'row-20'], + ['row-30', 'row-40', 'row-50'], + ['row-60', 'row-70', 'row-80'], + ['row-90'], + ]); + expect(report).toEqual({ name: 'rewrite-titles', batches: 4, rows: 10 }); + }); + + test('a completed pass replays whole: no statement, no body call, the same report', async () => { + const pass = harness({ batch: 3 }); + const first = await pass.run(); + + const readsAfterFirst = pass.watch.reads; + const second = await pass.run(); + + expect(second).toEqual(first); + expect(pass.watch.reads).toBe(readsAfterFirst); + expect(pass.watch.opens).toBe(1); + expect(pass.seen.length).toBe(4); + }); + + test('a step re-opened in the middle rebuilds the iteration at the checkpoint', async () => { + const pass = harness({ batch: 3 }); + await pass.run(); + + // What `retryFromStep` does: drop ONE step so it re-executes, leaving the ones after it + // memoized. The cursor therefore JUMPS over a page the live iteration never read, and an + // iteration trusted to be in step with the checkpoints would hand batch 3 the rows of batch 2. + await pass.store.del(RUN_ID, 'batch:1'); + await pass.store.del(RUN_ID, 'batch:3'); + const readsAfterFirst = pass.watch.reads; + const seenAfterFirst = pass.seen.length; + + const report = await pass.run(); + + expect(pass.seen.slice(seenAfterFirst)).toEqual([['row-30', 'row-40', 'row-50'], ['row-90']]); + // One statement per re-opened step, and a second iteration because the cursor jumped. + expect(pass.watch.reads).toBe(readsAfterFirst + 2); + expect(pass.watch.opens).toBe(3); + expect(pass.watch.closes).toBe(3); + expect(report).toEqual({ name: 'rewrite-titles', batches: 4, rows: 10 }); + }); + + test('a replayed step that is not a checkpoint fails the run instead of restarting it', async () => { + const pass = harness({ batch: 3 }); + await pass.run(); + const first = await pass.store.get(RUN_ID, 'batch:0'); + expect(first).toBeDefined(); + // `steps.ts` returns a completed step's output through an unchecked cast, so this is what a + // run id carrying another job's step under the same name looks like from in here. An absent + // cursor is not `null`: unchecked, the pass would reopen the source at the top and walk the + // whole table again, reporting a clean resume. + if (first !== undefined) await pass.store.put({ ...first, output: { done: true } }); + const readsAfterFirst = pass.watch.reads; + + await expect(pass.run()).rejects.toThrow(/not a backfill checkpoint/); + + expect(pass.watch.reads).toBe(readsAfterFirst); + }); +}); diff --git a/packages/jobs/src/backfill.ts b/packages/jobs/src/backfill.ts new file mode 100644 index 000000000..c22939e51 --- /dev/null +++ b/packages/jobs/src/backfill.ts @@ -0,0 +1,213 @@ +// `backfill()` — one pass over every row a chain matches, declared as a `job` and NOT as a ninth +// primitive. A backfill is durable background work with an input schema, a retry policy, an +// idempotency key and a queue, which is the definition of a `job` — so this file is a FACTORY over +// `job()`, exactly as `llm()` is one over `action()`. That is what gives a backfill `.enqueue()`, +// the worker's cancellation, the dead-letter path, `x jobs show` and a manifest row without a line +// here, and it is why nothing in the framework grows a ninth kind of thing to hold table sweeps. +// +// What the factory adds is the iteration. `inBatches()` reads the source one statement per page, +// and each page is handled inside its own `step.run`, so an attempt killed mid-pass resumes on the +// page it stopped at instead of reading the table again from the top. What a step persists is the +// CURSOR and the row count, never the page: `steps.ts` hands a completed step's output back for the +// whole run, so checkpointing rows would retain every row of every batch already processed until +// the job ended — the leak that turns a backfill of a large table into an OOM. + +import type { Ctx } from '@ultimat3/core'; +import { assert } from '@ultimat3/core'; +import type { BatchIterator, ReadBuilder } from '@ultimat3/entity'; +import { t } from '@ultimat3/schema'; +import type { DurationInput } from './clock'; +import type { JobHandle, JobRunArgs } from './job'; +import { job } from './job'; +import type { RetryPolicy } from './retry'; +import { DEFAULT_RETRY } from './retry'; + +/** + * Rows per statement and per durable step. Not `entity`'s `DEFAULT_PAGE_SIZE`: that number is a + * page somebody scrolls, and a backfill of a million rows at 50 writes twenty thousand step rows + * to move the same data. One statement's worth of work that a worker can still finish inside a + * lease is the size that belongs here. + */ +export const DEFAULT_BACKFILL_BATCH = 1_000; + +/** Every batch is a step, and this is the name it is checkpointed under. See `pass()`. */ +const STEP_PREFIX = 'batch:'; + +export interface BackfillBatch { + /** The page `page()` would have returned here: tenancy, soft delete, projection, preloads. */ + readonly rows: readonly Row[]; + readonly ctx: Ctx; + /** + * The run's cancellation composed with this batch's own ceiling — the same seam `ctx.signal` is + * elsewhere. Past it this step may no longer write, so hand it to whatever the body calls. + */ + readonly signal: AbortSignal; + /** 0-based position in the pass, and the step name this batch is checkpointed under. */ + readonly index: number; +} + +export interface BackfillDefinition { + /** + * REQUIRED, unlike a job's. A backfill's name is a durable key — the queue row, the step trace + * and the ledger of what has already been run all carry it — so it is never left to whichever + * export name a module happened to use. + */ + readonly name: string; + /** + * The rows to visit, as a chain: `() => db.posts.where({ published: true })`, or one narrowed by + * the run's own context — `({ ctx }) => db.posts.where({ orgId: ctx.actor.orgId })`. Read once + * per attempt and never enqueued, so what a run visits cannot drift from what was declared. + * `orderBy` is optional — the driver's own total order ends in the primary key, which is what + * lets every batch resume from the last one's cursor. + */ + source(args: { readonly ctx: Ctx }): ReadBuilder; + /** + * One page, in the batch's own durable step. Deliberately handed no `step`: a step name minted + * inside this body would have to be unique across the whole run, and the natural spelling + * (`step.run('rewrite', …)`) collides with itself on the second batch (`X_STEP_DUPLICATE`). + * + * At least once, like every other job body — an attempt cancelled between the last row and the + * checkpoint replays this page. Write through `upsertAll`, `updateWhere` or an idempotent + * statement; never `count + 1`. + */ + handle(batch: BackfillBatch): Promise | void; + /** Rows per statement and per step. Defaults to `DEFAULT_BACKFILL_BATCH`. */ + readonly batch?: number; + readonly queue?: string; + readonly retry?: RetryPolicy; + /** Per attempt, not per pass: a resumed attempt picks up at the last checkpoint. */ + readonly timeout?: DurationInput; +} + +/** What one completed pass reports — bounded, so `x jobs show` can print it. */ +export interface BackfillReport { + readonly name: string; + /** Batches this pass handled, replayed ones included. */ + readonly batches: number; + readonly rows: number; +} + +/** + * A backfill's payload is its identity and nothing else. What to visit is declared, so a queue row + * carries no parameters that could drift from the definition, and a run enqueued last week resumes + * against today's source rather than a stale copy of it. + */ +export type BackfillInput = Record; + +/** What a step persists: a bounded position, never the page it came from. See the file header. */ +interface Checkpoint { + /** Where the next batch starts; `null` once the pass is over. */ + readonly cursor: string | null; + readonly rows: number; +} + +/** One live iteration and the pull that advances it. Rebuilt when the checkpoints disagree. */ +interface Iteration { + readonly batches: BatchIterator; + readonly pull: AsyncIterator; +} + +const fieldOf = (value: unknown, key: string): unknown => + typeof value === 'object' && value !== null ? (value as Record)[key] : undefined; + +/** + * `steps.ts` hands a completed step's output back through an unchecked `as T`, so a checkpoint + * READ from storage is this shape by claim and never by check. It is checked here because the + * failure it would otherwise cause is silent and expensive: an absent cursor is not `null`, so the + * loop would reopen the source at the top and walk the whole table a second time. + */ +function asCheckpoint(value: unknown, step: string): Checkpoint { + const cursor = fieldOf(value, 'cursor'); + const rows = fieldOf(value, 'rows'); + assert( + (cursor === null || typeof cursor === 'string') && typeof rows === 'number', + `step "${step}" replayed ${JSON.stringify(value)}, which is not a backfill checkpoint`, + `x jobs show --json prints the run's steps — a run id whose "${step}" was written by something other than this backfill has to be retired, not resumed`, + ); + return { cursor, rows }; +} + +export function backfill(definition: BackfillDefinition): JobHandle { + const size = definition.batch ?? DEFAULT_BACKFILL_BATCH; + // Refused where it was written. `inBatches()` refuses the same number one statement in, which + // for a backfill means a dead-lettered job and a stack trace instead of a failing build. + assert( + Number.isSafeInteger(size) && size >= 1, + `backfill "${definition.name}" declares batch: ${String(size)} — a batch is a whole number of rows, at least one`, + `set batch: ${DEFAULT_BACKFILL_BATCH} on backfill("${definition.name}") — the rows one statement reads and one durable step handles`, + ); + + return job({ + name: definition.name, + input: t.object({}), + // One live run per name. A second enqueue while the pass is still going is the same pass, and + // deduping it is what makes "kick it again" safe rather than a second writer on one table. + idempotencyKey: () => definition.name, + retry: definition.retry ?? DEFAULT_RETRY, + ...(definition.queue === undefined ? {} : { queue: definition.queue }), + ...(definition.timeout === undefined ? {} : { timeout: definition.timeout }), + run: (args) => pass(definition, size, args), + }); +} + +/** + * The pass itself: one `step.run` per batch, named by position so a replay finds it. Completed + * steps are served from storage without touching the database, so a resumed attempt walks its own + * history to the last checkpoint and opens the source there. + */ +async function pass( + definition: BackfillDefinition, + size: number, + args: JobRunArgs, +): Promise { + const { ctx, step } = args; + let cursor: string | null = null; + let rows = 0; + let batches = 0; + let live: Iteration | undefined; + + /** + * The iteration positioned at `cursor`, opened on the first batch this attempt actually runs so + * a resumed pass sends no statement for a page it already handled. + * + * `batches.cursor` IS where the next statement starts, so comparing it with the checkpoint's is + * the whole staleness test — and it is not academic: `retryFromStep` re-opens ONE step in the + * middle of a finished run, which moves the checkpoints somewhere this iteration is not. Rebuilt + * from the checkpoint rather than read from wherever the old one was parked. + */ + const iterate = async (): Promise> => { + if (live !== undefined && live.batches.cursor === cursor) return live; + await live?.batches.close(); + const opened = definition.source({ ctx }).after(cursor).inBatches(size); + live = { batches: opened, pull: opened[Symbol.asyncIterator]() }; + return live; + }; + + try { + for (let index = 0; ; index += 1) { + const name = `${STEP_PREFIX}${index}`; + const checkpoint = asCheckpoint( + await step.run(name, async (signal): Promise => { + const iteration = await iterate(); + const next = await iteration.pull.next(); + if (next.done === true) return { cursor: null, rows: 0 }; + await definition.handle({ rows: next.value, ctx, signal, index }); + return { cursor: iteration.batches.cursor, rows: next.value.length }; + }), + name, + ); + rows += checkpoint.rows; + // `inBatches()` never yields an empty batch, so rows is what tells a handled page from the + // one step an exhausted source costs. + if (checkpoint.rows > 0) batches += 1; + cursor = checkpoint.cursor; + if (cursor === null) break; + } + } finally { + // Whatever the iteration holds belongs to this attempt, and an attempt that failed, was + // cancelled or finished is done with it either way. + await live?.batches.close(); + } + + return { name: definition.name, batches, rows }; +} diff --git a/packages/jobs/src/index.ts b/packages/jobs/src/index.ts index 8bc5c998a..211ad2383 100644 --- a/packages/jobs/src/index.ts +++ b/packages/jobs/src/index.ts @@ -10,6 +10,13 @@ import './register'; /** Re-exported so a `job`/`task` file needs one import, not two. Same object as schema's. */ export type { Infer } from '@ultimat3/schema'; export { t } from '@ultimat3/schema'; +export type { + BackfillBatch, + BackfillDefinition, + BackfillInput, + BackfillReport, +} from './backfill'; +export { backfill, DEFAULT_BACKFILL_BATCH } from './backfill'; export type { JobDescriptor } from './describe'; export type { ClaimedJob, From dc80788c46738f8955c8486ff8c1f3bbdf80604f Mon Sep 17 00:00:00 2001 From: sebi Date: Fri, 14 Aug 2026 13:32:39 -0500 Subject: [PATCH 2/5] feat(jobs): the x_backfills ledger, mirroring x_migrations MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit - new `backfill-ledger.ts`: `BackfillLedger`, the definition checksum, the pure `decideBackfill` verdict and a memory implementation - `x_backfills` ships in `SQL_JOBS_TABLE` and hangs off `JobDriver.backfills`, optional like `introspect` — a driver without one runs backfills unrecorded - a completed name re-enqueued is a no-op with a report; `{ force: true }` sweeps again into a NEW row; a moved checksum warns and still does not run - the row is a report, never a resume source — the step checkpoints stay the only answer to "where were we" - `backfill.ts` split: the declaration stays, the pass moves to `backfill-pass.ts`, same shape as `job.ts` / `execute.ts` - `appVersion()` lands in `@ultimat3/core`; `db`'s `runningAppVersion` delegates, so `x_migrations` and `x_backfills` cannot name one build twice Co-Authored-By: Claude --- CHANGELOG.md | 6 + docs/architecture/08-jobs-internals.md | 4 +- packages/core/CLAUDE.md | 1 + packages/core/src/app-version.test.ts | 30 ++ packages/core/src/app-version.ts | 32 ++ packages/core/src/index.ts | 1 + packages/db/CLAUDE.md | 5 + packages/db/src/migrate.ts | 8 +- packages/jobs/CLAUDE.md | 29 +- packages/jobs/README.md | 22 + packages/jobs/src/backfill-ledger.test.ts | 175 ++++++++ packages/jobs/src/backfill-ledger.ts | 165 ++++++++ packages/jobs/src/backfill-pass.test.ts | 489 ++++++++++++++++++++++ packages/jobs/src/backfill-pass.ts | 202 +++++++++ packages/jobs/src/backfill.test.ts | 299 +------------ packages/jobs/src/backfill.ts | 144 ++----- packages/jobs/src/driver-memory.ts | 6 + packages/jobs/src/driver-pg-sql.test.ts | 40 ++ packages/jobs/src/driver-pg-sql.ts | 55 +++ packages/jobs/src/driver-pg.test.ts | 77 ++++ packages/jobs/src/driver-pg.ts | 56 +++ packages/jobs/src/driver.ts | 8 + packages/jobs/src/index.ts | 12 + 23 files changed, 1457 insertions(+), 409 deletions(-) create mode 100644 packages/core/src/app-version.test.ts create mode 100644 packages/core/src/app-version.ts create mode 100644 packages/jobs/src/backfill-ledger.test.ts create mode 100644 packages/jobs/src/backfill-ledger.ts create mode 100644 packages/jobs/src/backfill-pass.test.ts create mode 100644 packages/jobs/src/backfill-pass.ts create mode 100644 packages/jobs/src/driver-pg-sql.test.ts diff --git a/CHANGELOG.md b/CHANGELOG.md index aeb85d26e..54699fecb 100644 --- a/CHANGELOG.md +++ b/CHANGELOG.md @@ -99,6 +99,12 @@ Semver applies from 1.0.0. A breaking change to a documented API needs a major `source` is read through `inBatches()` — one statement per page, keyset, never OFFSET — and every page is handled inside its own `step.run`, named `batch:0`, `batch:1`, … So a run killed mid-pass resumes on the page it stopped at: the completed steps replay from storage without touching the database, and the iteration is opened at the cursor they leave behind. What a step persists is that cursor and a row count, **never the page** — `steps.ts` hands a completed step's output back for the whole run, so checkpointing rows would retain every row already processed until the job ended. The body is handed no `step` for the same reason step names are positional: a name minted inside it would collide with itself on the second batch. `idempotencyKey` is the backfill's name, so re-enqueueing a live pass is the same pass. + **`x_backfills` records what has already been swept**, the twin of `x_migrations` one level up: name, definition checksum, status, app version, rows processed, last cursor, started/completed. It ships in the same DDL as `x_jobs` and `x_job_steps` and hangs off the queue driver as `driver.backfills`, so a ledger a pass cannot write is a queue it could not have been claimed from — and a driver without one runs backfills with no bookkeeping rather than refusing them, exactly as `introspect` already degrades. Re-enqueueing a **completed** name is a no-op with a report (`{ skipped: true, previousRunId }`) — no statement, no body call, no row. `enqueue({ force: true })` sweeps again and writes a **new** row: reruns are history, never an edit of the row they rerun. A definition whose checksum moved since the completed pass **warns** and still does not run, because this checksum is over function source text, which a bundler can move without a line of behaviour changing — where `@ultimat3/db`'s `auditLedger` throws on the same fact, since SQL text is what it applied. + + The row is a **report**, never a resume source: where a resumed pass restarts is decided by the step checkpoints and by nothing else. A retry adopts its own row (`started_at` is when the pass began, not this attempt); an attempt that failed is recorded as `failed` with its cursor kept, because where a pass stopped is the first thing anyone asks about one. + +- **`appVersion()` in `@ultimat3/core`.** One reader for `APP_VERSION`, defaulting to `dev`. It was spelled inside `@ultimat3/db`'s `runningAppVersion`, which now delegates — `x_migrations.app_version` and `x_backfills.app_version` are two durable columns an operator reads side by side, `@ultimat3/jobs` cannot reach `db` for the answer, and two packages defaulting the key their own way would put two names on one build. + - **The destructive-SQL rail: a migration that destroys data must say so, and `x verify` refuses one that does not.** `x db gen` now writes `-- destructive: true` into any migration whose `up` drops a table, drops a column, truncates or retypes; `x verify`'s `drift` step reads the committed files back and fails an unmarked one with the new `X_MIGRATION_DESTRUCTIVE`. The strong-migrations idea, enforced rather than documented — a drop is allowed, an *undeclared* drop is not. ```text diff --git a/docs/architecture/08-jobs-internals.md b/docs/architecture/08-jobs-internals.md index a1d3187b4..8d1afeb12 100644 --- a/docs/architecture/08-jobs-internals.md +++ b/docs/architecture/08-jobs-internals.md @@ -68,10 +68,12 @@ export interface JobDriver { | Driver | State | Trade-off | |---|---|---| -| `pg` (default) | `x_jobs`, `x_job_steps`, `x_outbox`, `x_rate_buckets` | outbox is free (same DB, same tx); `SKIP LOCKED` claiming; zero extra infra | +| `pg` (default) | `x_jobs`, `x_job_steps`, `x_backfills`, `x_outbox`, `x_rate_buckets` | outbox is free (same DB, same tx); `SKIP LOCKED` claiming; zero extra infra | | `redis` | streams + consumer groups, outbox relay in front | high throughput, short jobs; loses "queue state in one backup" | | `nats` | JetStream, outbox relay in front | strongest delivery semantics, most operational surface. `As of 2026-07` `claim` throws `X_NOT_IMPLEMENTED` with `fix: set jobs.driver = "pg" in app.config.ts` | +`x_backfills` is the odd one out: it is not queue state but the ledger of what a `backfill()` pass has already swept, hanging off `JobDriver.backfills` because it ships in the same DDL as `x_jobs` — `As of 2026-08` only the `pg` and `memory` drivers carry one, and a driver without it runs backfills with no bookkeeping rather than refusing them. + Because `saveStep`/`loadSteps` are driver methods, step persistence works identically on all three. Switching is a config line plus `x jobs drain --to redis` for in-flight rows. ## The pg claim loop diff --git a/packages/core/CLAUDE.md b/packages/core/CLAUDE.md index 7fa36e0ab..c3d19fabf 100644 --- a/packages/core/CLAUDE.md +++ b/packages/core/CLAUDE.md @@ -34,6 +34,7 @@ shape against a locally declared sample interface for exactly that reason. |---|---|---| | which deploy this is | `environment.ts` (`ULTIMATE_ENV`) | the twin of `ROLE`; never declare a second env var for it | | what this process does | `roles.ts` (`ROLE`) | | +| which build of the APP this is | `app-version.ts` (`APP_VERSION`) | one reader, `dev` by default: `db` writes it into `x_migrations` and `jobs` into `x_backfills`, and `jobs` cannot reach `db` for the answer | | the values | `env.ts` | `checkEnv().values` holds REAL secrets — anything that prints goes through `maskedEnvValues()` | | `.env.example` | `env-example.ts` | a projection of the schema, never hand-maintained | | loading `.env` | **Bun**, not us | `envFileCandidates()` documents the measured order; there is no `.env.staging` | diff --git a/packages/core/src/app-version.test.ts b/packages/core/src/app-version.test.ts new file mode 100644 index 000000000..94e36065f --- /dev/null +++ b/packages/core/src/app-version.test.ts @@ -0,0 +1,30 @@ +import { describe, expect, test } from 'bun:test'; +import { APP_VERSION_KEY, appVersion, DEFAULT_APP_VERSION } from './app-version'; + +describe('appVersion', () => { + test('reads APP_VERSION', () => { + expect(appVersion({ [APP_VERSION_KEY]: '1.4.2' })).toBe('1.4.2'); + }); + + test('an unset key is the local build', () => { + expect(appVersion({})).toBe(DEFAULT_APP_VERSION); + expect(DEFAULT_APP_VERSION).toBe('dev'); + }); + + test('an exported-but-empty key names no build either', () => { + // A platform that exports the key with no value has said nothing, and `""` written into a + // ledger row is a build id no operator can look up. + expect(appVersion({ [APP_VERSION_KEY]: '' })).toBe(DEFAULT_APP_VERSION); + }); + + test('reads this process environment by default', () => { + const previous = process.env[APP_VERSION_KEY]; + process.env[APP_VERSION_KEY] = 'from-process-env'; + try { + expect(appVersion()).toBe('from-process-env'); + } finally { + if (previous === undefined) delete process.env[APP_VERSION_KEY]; + else process.env[APP_VERSION_KEY] = previous; + } + }); +}); diff --git a/packages/core/src/app-version.ts b/packages/core/src/app-version.ts new file mode 100644 index 000000000..4ee0ae618 --- /dev/null +++ b/packages/core/src/app-version.ts @@ -0,0 +1,32 @@ +// Single responsibility: which build of the APP this process is running. The third key in the +// same family as `environment.ts` (`ULTIMATE_ENV` — which deploy) and `roles.ts` (`ROLE` — what +// this process does): one key, one spelling, one default. +// +// Deliberately NOT `version.ts`. That answers what version of the FRAMEWORK shipped, read from +// `@ultimat3/core`'s own manifest; this answers what the deploy calls itself, and on every release +// that does not bump both they are different strings. + +export const APP_VERSION_KEY = 'APP_VERSION'; + +/** An unset key is a local process, never a build nobody can name. */ +export const DEFAULT_APP_VERSION = 'dev'; + +/** + * `APP_VERSION`, else `dev`. + * + * One reader rather than one per caller because the value is DURABLE: it is written into + * `x_migrations` rows by `@ultimat3/db` and `x_backfills` rows by `@ultimat3/jobs`, both of which + * outlive the process that wrote them. Two packages defaulting it differently would put two names + * on one build, in two tables an operator reads side by side. `jobs` cannot reach `db` for it — + * `db` is tier 1 and off that package's import list — so the shared answer lives here, at tier 0. + */ +export function appVersion( + env: Readonly> = process.env as Record< + string, + string | undefined + >, +): string { + const declared = env[APP_VERSION_KEY]; + // Empty is unset: a platform that exports the key with no value has named no build. + return declared === undefined || declared === '' ? DEFAULT_APP_VERSION : declared; +} diff --git a/packages/core/src/index.ts b/packages/core/src/index.ts index bd081d1d1..d5118ebdd 100644 --- a/packages/core/src/index.ts +++ b/packages/core/src/index.ts @@ -23,6 +23,7 @@ export { userActor, withFacts, } from './actor'; +export { APP_VERSION_KEY, appVersion, DEFAULT_APP_VERSION } from './app-version'; export type { InvariantOptions } from './assert'; export { assert, assertNever, invariant } from './assert'; export type { Clock, FrozenClock } from './clock'; diff --git a/packages/db/CLAUDE.md b/packages/db/CLAUDE.md index 0b0a2dec6..8d6182030 100644 --- a/packages/db/CLAUDE.md +++ b/packages/db/CLAUDE.md @@ -321,6 +321,11 @@ cannot be an identifier at all. rail's wording and the rail reads SQL text: leaving the blanker in `readonly.ts` would have put the error registry, which registers codes at module evaluation, inside an import cycle. +`runningAppVersion()` delegates to `@ultimat3/core`'s `appVersion()` and keeps its explicit +override — `x_migrations.app_version` and `@ultimat3/jobs`' `x_backfills.app_version` are two +durable columns an operator reads side by side, and `jobs` cannot import this package for the +answer, so the key has one reader at tier 0 rather than one per writer. + `checkDrift()` is the **post-migrate verification** and the only drift question that needs a database: the live catalog against the ledger the run just wrote. It is asked where a connection is open — `@ultimat3/cli`'s `runMigrations`, which is `x db migrate`, `x db reset` and `ROLE=migrate` diff --git a/packages/db/src/migrate.ts b/packages/db/src/migrate.ts index 6289c43ac..1242f1bcf 100644 --- a/packages/db/src/migrate.ts +++ b/packages/db/src/migrate.ts @@ -3,6 +3,7 @@ // app-version fence is the `migrate` role's contract — a pod must refuse to migrate a database // another build already owns, because the alternative is two schemas racing during a rollout. +import { appVersion } from '@ultimat3/core'; import { baseClient, type DbClient, type DbConnection, isReservable } from './client'; import { migrationConflict } from './errors'; import { expectedQueryLoop } from './expected-loop'; @@ -68,8 +69,13 @@ export function migrationChecksum(migration: Migration): string { return migration.checksum ?? checksumOf(migration.up); } +/** + * Core's, never a second read of the key: `x_migrations.app_version` and `x_backfills.app_version` + * are two durable columns an operator reads side by side, and a package defaulting `APP_VERSION` + * its own way would put two names on one build. + */ export function runningAppVersion(explicit?: string | undefined): string { - return explicit ?? process.env['APP_VERSION'] ?? 'dev'; + return explicit ?? appVersion(); } export async function ensureLedger(client: DbClient): Promise { diff --git a/packages/jobs/CLAUDE.md b/packages/jobs/CLAUDE.md index 0873d8c31..902fe4c22 100644 --- a/packages/jobs/CLAUDE.md +++ b/packages/jobs/CLAUDE.md @@ -136,12 +136,33 @@ Tier 3. The `job` + `task` primitives, durable steps, transactional outbox, queu jumps over a page the live iteration never read. A checkpoint READ back is checked rather than trusted — `step.run` replays it through an unchecked `as T`, and an absent cursor is not `null`, so the pass would silently reopen the source at the top and walk the whole table again. +- **`x_backfills` is what has already been SWEPT, and the step checkpoints are where a pass + resumes. Never the other way round.** The ledger row (`backfill-ledger.ts`) is a report an + operator reads and the record that a completed name is done; the checkpoints are written in step + with the work. A resume driven off `last_cursor` would be a second answer to "where were we" + against a row that is not transactional with the rows it describes. Keyed by RUN, not by name, + because `force` writes a NEW row rather than editing the one it reruns — history, not an edit of + it. Only `completed` blocks: a `running` row is this pass resuming or one holding the single live + idempotency key, a `failed` one is an attempt the queue is about to retry, and `start()` puts an + adopted row back to `running` keeping the `started_at` the PASS began at. A moved checksum WARNS + and still does not run — it hashes `source` and `handle`'s source text with a `\u0000` between + them (raw concatenation would hash a boundary, not a pair), and a bundler moves that text without + a line of behaviour changing, which is why `@ultimat3/db`'s `auditLedger` may throw on the same + fact and this may not. `force` is the only override, and it rides the input rather than the + idempotency key: one live pass per name, forced or not, or "kick it again" becomes a second + writer on one table. +- **The ledger hangs off the queue driver (`driver.backfills`), optional like `introspect`.** + `x_backfills` ships in `SQL_JOBS_TABLE`, so a ledger a pass cannot write is a queue it could not + have been claimed from — and `dev-queue.ts` applies that one constant, so `x dev` and production + create the same table. A driver that ships none (`driver-redis`, `driver-nats`, a hand-rolled + one) runs backfills with NO bookkeeping rather than refusing them: nothing blocks a completed + name there, and that degradation is the price of one install point. - Suspension is control flow: `StepSuspension` -> `nack({ countsAsAttempt: false })`. Never log it as an error, never let it burn an attempt. - Step results are persisted BEFORE the step returns. Keep it that way or replay breaks. - All time is epoch ms from an injected `Clock`, read via `nowMs()` in `clock.ts`. -- Drivers implement exactly the six `JobDriver` methods plus optional `introspect`. - New capabilities go behind the interface, never as a driver-specific export. +- Drivers implement exactly the six `JobDriver` methods plus optional `introspect` and + `backfills`. New capabilities go behind the interface, never as a driver-specific export. - `inspect.ts` returns plain JSON-serialisable objects — CLI, `/_x` and MCP share them. - `src/index.ts` re-exports `t` from `@ultimat3/schema` **verbatim**, so a job/task file imports one package. Never wrap, spread or re-declare it: `t` delegates to `schemaProvider()` on every @@ -178,7 +199,9 @@ picture from the other side. | File | Owns | |---|---| | `job.ts` | the `job()` primitive + registry + the handle's fluent surface + `registerJob` | -| `backfill.ts` | `backfill()` — a factory over `job()`: the batched pass and its cursor checkpoints | +| `backfill.ts` | `backfill()` — a factory over `job()`: the declaration, its checksum and its input | +| `backfill-pass.ts` | one pass: the batched iteration, its cursor checkpoints and its ledger row | +| `backfill-ledger.ts` | `x_backfills` — the contract, the checksum, the verdict, the memory ledger | | `register.ts` | `registerJobs`/`registerTasks` over a module namespace + the registrar announcements | | `describe.ts` | the JSON projection one handle emits; `describeJobs()` is a map over it | | `steps.ts` | `StepStore`, `StepApi`, memoized-replay executor, `StepSuspension` | diff --git a/packages/jobs/README.md b/packages/jobs/README.md index f714eb286..360076cb0 100644 --- a/packages/jobs/README.md +++ b/packages/jobs/README.md @@ -139,6 +139,28 @@ without a statement, and the iteration reopens at the cursor they left behind. | `idempotencyKey` is the backfill's name | re-enqueueing a live pass is the same pass, not a second writer on one table | | `batch` is refused at declaration | `0`, `1.5` and a `NaN` from an env var fail the build, not the fourth attempt | +### The `x_backfills` ledger + +What has already been swept, the twin of `x_migrations`. It ships in the same DDL as `x_jobs`, +hangs off the queue driver as `driver.backfills`, and carries one row per **pass**: name, +definition checksum, status, app version, rows processed, last cursor, started/completed. + +```ts +await rewriteSlugs.enqueue({}); // completed already? no-op with a report +// → { name: 'rewrite-slugs', batches: 0, rows: 0, skipped: true, previousRunId: '…' } + +await rewriteSlugs.enqueue({ force: true }); // sweeps again, into a NEW row +``` + +| Rule | Why | +|---|---| +| only a **completed** row blocks | a `running` row is this pass resuming, a `failed` one is an attempt the queue is about to retry | +| `force` writes a new row | reruns are history, never an edit of the row they rerun | +| a moved checksum **warns** | it hashes function source text, which a bundler moves without behaviour changing — `@ultimat3/db` throws on the same fact because SQL text is what it applied | +| the row is a report, never a resume source | where a resumed pass restarts is the step checkpoints' answer, and there is only one | +| a retry adopts its own row | `started_at` is when the pass began, not when this attempt did | +| a driver without a ledger runs the pass anyway | the same degradation `introspect` has — no bookkeeping, never a refusal | + ## The deadline cancels A job's `timeout` aborts `ctx.signal` **before** it fails the attempt, because the nack that diff --git a/packages/jobs/src/backfill-ledger.test.ts b/packages/jobs/src/backfill-ledger.test.ts new file mode 100644 index 000000000..18db6a1d9 --- /dev/null +++ b/packages/jobs/src/backfill-ledger.test.ts @@ -0,0 +1,175 @@ +// The ledger in isolation: what a row means, what "the same backfill" means, and what a completed +// row decides. `backfill.test.ts` owns the pass that writes them. + +import { describe, expect, test } from 'bun:test'; +import { frozenClock } from '@ultimat3/core'; +import type { BackfillRun } from './backfill-ledger'; +import { backfillChecksum, createMemoryBackfillLedger, decideBackfill } from './backfill-ledger'; +import { + SQL_BACKFILL_FINISH, + SQL_BACKFILL_LIST, + SQL_BACKFILL_START, + SQL_JOBS_TABLE, +} from './driver-pg-sql'; + +const completed = (fields: Partial = {}): BackfillRun => ({ + runId: 'run-1', + name: 'rewrite-titles', + checksum: 'aaaa', + status: 'completed', + appVersion: '1.2.0', + rows: 10, + cursor: null, + startedAt: 1, + completedAt: 2, + ...fields, +}); + +describe('backfillChecksum', () => { + const source = (): number => 1; + const handle = (): number => 2; + + test('is a stable 32-character hash of the two bodies', () => { + const first = backfillChecksum(source, handle); + expect(first).toMatch(/^[0-9a-f]{32}$/); + expect(backfillChecksum(source, handle)).toBe(first); + }); + + test('moves when either body moves', () => { + const base = backfillChecksum(source, handle); + expect(backfillChecksum((): number => 99, handle)).not.toBe(base); + expect(backfillChecksum(source, (): number => 99)).not.toBe(base); + }); + + test('the two bodies are delimited, never concatenated', () => { + // Raw concatenation hashes a BOUNDARY rather than a pair: `ab` + `c` and `a` + `bc` are one + // string, so a statement moving from `handle` into `source` could leave the checksum where it + // was. Asserted against the naive spelling, because no pair of REAL functions can demonstrate + // the collision — a body cannot be cut anywhere and still parse on both sides. + const naive = new Bun.CryptoHasher('sha256') + .update(`${source.toString()}${handle.toString()}`) + .digest('hex') + .slice(0, 32); + expect(backfillChecksum(source, handle)).not.toBe(naive); + }); +}); + +describe('decideBackfill', () => { + test('a name the ledger has never completed runs', () => { + expect(decideBackfill(undefined, 'aaaa', false)).toEqual({ run: true, changed: false }); + }); + + test('a completed name is a no-op', () => { + const previous = completed(); + expect(decideBackfill(previous, 'aaaa', false)).toEqual({ + run: false, + previous, + changed: false, + }); + }); + + test('force runs it again and still names what it is rerunning', () => { + const previous = completed(); + expect(decideBackfill(previous, 'aaaa', true)).toEqual({ run: true, previous, changed: false }); + }); + + test('a definition that moved warns — it does not run, and it does not refuse', () => { + const previous = completed({ checksum: 'bbbb' }); + const verdict = decideBackfill(previous, 'aaaa', false); + // `changed` is the warning; `run: false` is still the answer. Only force overrides. + expect(verdict).toEqual({ run: false, previous, changed: true }); + }); +}); + +describe('the memory ledger', () => { + const ledger = () => createMemoryBackfillLedger(frozenClock(new Date('2026-08-14T00:00:00Z'))); + const opened = { runId: 'run-1', name: 'sweep', checksum: 'aaaa', appVersion: '1.2.0' }; + + test('start opens a running row at zero', async () => { + const store = ledger(); + await store.start(opened); + expect(await store.list()).toEqual([ + { ...opened, status: 'running', rows: 0, cursor: null, startedAt: Date.parse('2026-08-14') }, + ]); + }); + + test('progress moves the row and completion clears the cursor', async () => { + const store = ledger(); + await store.start(opened); + await store.progress('run-1', { rows: 6, cursor: 'c6' }); + expect((await store.list())[0]).toMatchObject({ rows: 6, cursor: 'c6', status: 'running' }); + + await store.finish('run-1', { status: 'completed', rows: 10 }); + expect((await store.list())[0]).toMatchObject({ rows: 10, cursor: null, status: 'completed' }); + }); + + test('a failure keeps its cursor, and the next attempt adopts the same row', async () => { + const store = ledger(); + await store.start(opened); + await store.progress('run-1', { rows: 6, cursor: 'c6' }); + await store.finish('run-1', { status: 'failed', rows: 6 }); + // Where a pass stopped is the first thing anyone asks about a failed one. + expect((await store.list())[0]).toMatchObject({ status: 'failed', cursor: 'c6', rows: 6 }); + + await store.start(opened); + const adopted = (await store.list())[0]; + expect(adopted).toMatchObject({ status: 'running', cursor: 'c6', rows: 6 }); + // `startedAt` is when the PASS began, not this attempt. + expect(adopted?.startedAt).toBe(Date.parse('2026-08-14')); + expect(await store.list()).toHaveLength(1); + }); + + test('a rerun is a new row, never an edit of the one it reruns', async () => { + const store = ledger(); + await store.start(opened); + await store.finish('run-1', { status: 'completed', rows: 10 }); + await store.start({ ...opened, runId: 'run-2' }); + + const rows = await store.list({ name: 'sweep' }); + expect(rows.map((run) => run.runId)).toEqual(['run-2', 'run-1']); + expect(rows.map((run) => run.status)).toEqual(['running', 'completed']); + }); + + test('list filters by name and status and honours its limit', async () => { + const store = ledger(); + await store.start(opened); + await store.finish('run-1', { status: 'completed', rows: 10 }); + await store.start({ ...opened, runId: 'run-2', name: 'other' }); + + expect(await store.list({ name: 'other' })).toHaveLength(1); + expect((await store.list({ status: 'completed' }))[0]?.runId).toBe('run-1'); + expect(await store.list({ limit: 1 })).toHaveLength(1); + }); + + test('a write to a run nobody opened is a no-op, not a phantom row', async () => { + const store = ledger(); + await store.progress('never-started', { rows: 5, cursor: 'c5' }); + await store.finish('never-started', { status: 'completed', rows: 5 }); + expect(await store.list()).toEqual([]); + }); +}); + +describe('the shipped SQL', () => { + test('x_backfills is created wherever x_jobs is', () => { + // `dev-queue.ts` applies this constant statement by statement, so a ledger declared anywhere + // else would exist in production and not in `x dev`. + expect(SQL_JOBS_TABLE).toContain('create table if not exists x_backfills'); + expect(SQL_JOBS_TABLE).toContain('run_id uuid primary key'); + expect(SQL_JOBS_TABLE).toContain('create index if not exists x_backfills_name_idx'); + }); + + test('start adopts a run rather than opening a second row for it', () => { + expect(SQL_BACKFILL_START).toContain("on conflict (run_id) do update set status = 'running'"); + }); + + test('only a completed pass clears its cursor', () => { + expect(SQL_BACKFILL_FINISH).toContain( + "last_cursor = case when $2 = 'completed' then null else last_cursor end", + ); + }); + + test('list is newest first and reads epoch milliseconds', () => { + expect(SQL_BACKFILL_LIST).toContain('order by started_at desc'); + expect(SQL_BACKFILL_LIST).toContain('(extract(epoch from started_at) * 1000)::bigint'); + }); +}); diff --git a/packages/jobs/src/backfill-ledger.ts b/packages/jobs/src/backfill-ledger.ts new file mode 100644 index 000000000..60ea69adc --- /dev/null +++ b/packages/jobs/src/backfill-ledger.ts @@ -0,0 +1,165 @@ +// `x_backfills` — what has already been swept, and whether this pass runs at all. The twin of +// `@ultimat3/db`'s `x_migrations`, one level up: a migration ledger is keyed by migration id +// because a migration applies once and only once, and this one is keyed by RUN because a backfill +// may legitimately be run again. `force` therefore writes a NEW row rather than overwriting the +// one that says what the last pass did — reruns are history, not an edit of it. +// +// The row is a REPORT, never a resume source. Where a resumed pass restarts is decided by the step +// checkpoints in `backfill-pass.ts` and by nothing else; `cursor` here is what an operator reads +// while a pass is running. A second answer to "where were we" is the one thing this file must not +// become — the checkpoints are transactional with the work, and this row is not. + +import type { Clock } from '@ultimat3/core'; +import { systemClock } from '@ultimat3/core'; +import { nowMs } from './clock'; + +/** + * `failed` is the LAST ATTEMPT's verdict, not the pass's: the queue owns retries, so a row here + * goes back to `running` when the next attempt starts. Only `completed` is terminal, and it is the + * one status that blocks a re-run. + */ +export type BackfillStatus = 'running' | 'completed' | 'failed'; + +export interface BackfillRun { + /** The job run this pass belongs to, and the ledger's primary key. */ + readonly runId: string; + readonly name: string; + readonly checksum: string; + readonly status: BackfillStatus; + /** The build that STARTED the pass — a redeploy mid-pass does not rewrite it. */ + readonly appVersion: string; + readonly rows: number; + /** Where the pass had got to. `null` before the first batch and once it is over. */ + readonly cursor: string | null; + readonly startedAt: number; + readonly completedAt?: number | undefined; +} + +export interface BackfillFilter { + readonly name?: string | undefined; + readonly status?: BackfillStatus | undefined; + readonly limit?: number | undefined; +} + +/** + * Durable, and therefore behind the queue driver: `x_backfills` ships in the same DDL as `x_jobs` + * and `x_job_steps` (`driver-pg-sql.ts`), so a ledger a pass cannot write is a queue it could not + * have been claimed from. Optional on `JobDriver` for the same reason `introspect` is — a driver + * with no ledger runs backfills with no bookkeeping rather than refusing them. + */ +export interface BackfillLedger { + /** Open this run's row, or adopt the one a previous attempt of the SAME run opened. */ + start(run: { + readonly runId: string; + readonly name: string; + readonly checksum: string; + readonly appVersion: string; + }): Promise; + /** Move the row forward. Absolute position, so a replayed batch reports the same number. */ + progress( + runId: string, + at: { readonly rows: number; readonly cursor: string | null }, + ): Promise; + finish( + runId: string, + at: { readonly status: 'completed' | 'failed'; readonly rows: number }, + ): Promise; + /** Newest first. `x db backfill --list` and the verdict below read the same method. */ + list(filter?: BackfillFilter): Promise; +} + +/** Anything callable. `Function` is a banned type and a real signature would pin one definition. */ +type AnyFn = (...args: never[]) => unknown; + +/** + * Between the two bodies, and a byte no source text carries. Concatenating them raw would hash a + * BOUNDARY rather than a pair — `ab` + `c` and `a` + `bc` are one string — so a statement moving + * from `handle` into `source` could leave the checksum where it was. + */ +const CHECKSUM_SEPARATOR = '\u0000'; + +/** + * What "the same backfill" means: the source text of the two functions that decide which rows a + * pass visits and what happens to them, which is the whole of what a completed row claims was + * done. `batch` is deliberately out — paging a sweep differently is a tuning change, and a + * checksum that moved for it would warn on every one. + * + * A code hash is fuzzier than a migration's SQL hash: a bundler that reformats a body moves it + * with no line of behaviour changing. That is exactly why a mismatch WARNS and never refuses, + * where `@ultimat3/db`'s `auditLedger` throws on the same fact — SQL text is what it applied. + */ +export function backfillChecksum(source: AnyFn, handle: AnyFn): string { + return new Bun.CryptoHasher('sha256') + .update(`${source.toString()}${CHECKSUM_SEPARATOR}${handle.toString()}`) + .digest('hex') + .slice(0, 32); +} + +export interface BackfillVerdict { + readonly run: boolean; + /** The completed pass this verdict is about, when the ledger holds one. */ + readonly previous?: BackfillRun | undefined; + /** `previous` completed under a different definition. Warned about, never refused. */ + readonly changed: boolean; +} + +/** + * Pure, so the pass, a test and `x db backfill` all read one decision. Only a COMPLETED row + * blocks: a `running` row is this pass resuming (same run) or a pass another worker holds the one + * live idempotency key for, and a `failed` one is an attempt the queue is about to retry. + */ +export function decideBackfill( + completed: BackfillRun | undefined, + checksum: string, + force: boolean, +): BackfillVerdict { + if (completed === undefined) return { run: true, changed: false }; + return { run: force, previous: completed, changed: completed.checksum !== checksum }; +} + +export function createMemoryBackfillLedger(clock: Clock = systemClock): BackfillLedger { + const runs = new Map(); + const patch = (runId: string, fields: Partial): void => { + const existing = runs.get(runId); + if (existing !== undefined) runs.set(runId, { ...existing, ...fields }); + }; + return { + start(run) { + const existing = runs.get(run.runId); + // A retry adopts its own row: `startedAt` is when the PASS began, not this attempt, and the + // status goes back to `running` so a row a failed attempt marked stops claiming otherwise. + runs.set( + run.runId, + existing === undefined + ? { ...run, status: 'running', rows: 0, cursor: null, startedAt: nowMs(clock) } + : { ...existing, status: 'running' }, + ); + return Promise.resolve(); + }, + progress(runId, at) { + patch(runId, { rows: at.rows, cursor: at.cursor }); + return Promise.resolve(); + }, + finish(runId, at) { + // A failure keeps its cursor — where a pass stopped is the first thing anyone asks. + patch(runId, { + status: at.status, + rows: at.rows, + completedAt: nowMs(clock), + ...(at.status === 'completed' ? { cursor: null } : {}), + }); + return Promise.resolve(); + }, + list(filter = {}) { + // Reversed BEFORE the sort: the test clock is frozen, so two rows share a `startedAt` and a + // stable sort would hand back the oldest of them first under a "newest first" contract. + const rows = [...runs.values()] + .reverse() + .sort((a, b) => b.startedAt - a.startedAt) + .filter((run) => filter.name === undefined || run.name === filter.name) + .filter((run) => filter.status === undefined || run.status === filter.status) + .slice(0, filter.limit ?? 100); + return Promise.resolve(rows); + }, + }; +} diff --git a/packages/jobs/src/backfill-pass.test.ts b/packages/jobs/src/backfill-pass.test.ts new file mode 100644 index 000000000..af9949a10 --- /dev/null +++ b/packages/jobs/src/backfill-pass.test.ts @@ -0,0 +1,489 @@ +// One `backfill()` pass, end to end: every row visited once, a killed attempt resuming on the page +// it stopped at WITHOUT re-reading the ones before it, a checkpoint that is a cursor rather than +// the rows behind it, and the `x_backfills` row that reports all of it. The source is a real +// `@ultimat3/entity` chain over a real `memoryRepo`, so the cursors, the keyset paging and +// `inBatches()`'s own refusals are the shipped ones and not a fixture that agrees by construction. +// `backfill.test.ts` owns the declaration; `backfill-ledger.test.ts` owns the ledger on its own. + +import { afterEach, beforeEach, describe, expect, spyOn, test } from 'bun:test'; +import type { Ctx } from '@ultimat3/core'; +import { createContext, logger } from '@ultimat3/core'; +import type { BatchIterator, ReadBuilder, Repo } from '@ultimat3/entity'; +import { entity, memoryRepo, tableFor, text, uuid } from '@ultimat3/entity'; +import type { BackfillInput } from './backfill'; +import { backfill, DEFAULT_BACKFILL_BATCH } from './backfill'; +import type { BackfillLedger } from './backfill-ledger'; +import { resetJobDriver, setJobDriver } from './driver'; +import { createMemoryDriver } from './driver-memory'; +import { resetJobs } from './job'; +import type { StepRecord, StepStore } from './steps'; +import { createMemoryStepStore, createStepRunner, StepSuspension } from './steps'; + +const rows = entity('backfill_test_rows', { + columns: { id: uuid().primaryKey(), orgId: uuid(), title: text({ max: 40 }) }, +}); + +type Row = typeof rows.$row; + +const ORG = '00000000-0000-7000-8000-0000000000a1'; +const OTHER = '00000000-0000-7000-8000-0000000000a2'; + +/** Sortable ids, ten apart, so the keyset cursor between two pages is a real position. */ +const id = (index: number): string => + `00000000-0000-7000-8000-0000000001${String(index).padStart(2, '0')}`; + +const row = (index: number, orgId: string = ORG): Row => ({ + id: id(index), + orgId, + title: `row-${index}`, +}); + +/** Ten rows for the org under test, two more no pass may ever touch. */ +const SEED: readonly Row[] = [ + ...Array.from({ length: 10 }, (_unused, index) => row(index * 10)), + row(900, OTHER), + row(910, OTHER), +]; + +/** What the wrappers below record: statements sent, iterations opened, iterations closed. */ +interface Watch { + reads: number; + opens: number; + closes: number; +} + +const newWatch = (): Watch => ({ reads: 0, opens: 0, closes: 0 }); + +/** `cursor` is a getter on the real iterator, so it is forwarded rather than spread-copied. */ +const watchedIterator = (iterator: BatchIterator, watch: Watch): BatchIterator => { + const close = async (): Promise => { + watch.closes += 1; + await iterator.close(); + }; + return { + get cursor(): string | null { + return iterator.cursor; + }, + [Symbol.asyncIterator]: () => iterator[Symbol.asyncIterator](), + close, + [Symbol.asyncDispose]: close, + }; +}; + +/** `after()` returns a NEW chain, so the wrapper has to survive it — the factory calls both. */ +const watchedChain = (chain: ReadBuilder, watch: Watch): ReadBuilder => ({ + ...chain, + after: (cursor) => watchedChain(chain.after(cursor), watch), + inBatches: (size) => { + watch.opens += 1; + return watchedIterator(chain.inBatches(size), watch); + }, +}); + +const countingRepo = (repo: Repo, watch: Watch): Repo => ({ + ...repo, + findMany: async (args) => { + watch.reads += 1; + return repo.findMany(args); + }, +}); + +interface Harness { + readonly watch: Watch; + readonly store: StepStore; + /** Titles handed to the body, one entry per batch it actually ran. */ + readonly seen: readonly (readonly string[])[]; + /** Every batch index the body should reject on. */ + failOn: ReadonlySet; + /** A run id of its own is a NEW pass; the default is a retry of the same one. */ + run(options?: { runId?: string; input?: BackfillInput }): Promise; + steps(): Promise; +} + +const RUN_ID = 'run-backfill-1'; + +const ctx: Ctx = createContext(); + +/** + * One backfill over `SEED`, driven through a real step runner rather than a worker: this file is + * about the pass and its checkpoints, and `execute.ts` already owns what a driver does with the + * outcome. Every `run()` is a fresh attempt against the SAME store and run id, which is exactly + * what a retry is. + */ +const harness = (options: { batch?: number; name?: string } = {}): Harness => { + const watch = newWatch(); + const store = createMemoryStepStore(); + const seen: string[][] = []; + const table = tableFor(rows, countingRepo(memoryRepo(rows, SEED), watch)); + const state: Harness = { + watch, + store, + seen, + failOn: new Set(), + async run(runOptions = {}) { + const runId = runOptions.runId ?? RUN_ID; + const runner = createStepRunner({ runId, jobName: 'backfill', store }); + return handle.run({ + input: runOptions.input ?? {}, + step: runner.step, + ctx, + attempt: 1, + jobId: 'job-1', + runId, + }); + }, + steps: () => store.list(RUN_ID), + }; + const handle = backfill({ + name: options.name ?? 'rewrite-titles', + ...(options.batch === undefined ? {} : { batch: options.batch }), + source: () => watchedChain(table.where({ orgId: ORG }), watch), + handle: ({ rows: page, index }) => { + if (state.failOn.has(index)) throw new Error(`batch ${String(index)} failed`); + seen.push(page.map((entry) => entry.title)); + }, + }); + return state; +}; + +/** + * The ledger hangs off the installed queue driver, so a test that wants one installs a driver. + * Every test above this line runs with none — which is the documented degradation, and why the + * pass and its checkpoints are pinned without one. + */ +const installLedger = (): BackfillLedger => { + const driver = createMemoryDriver(); + setJobDriver(driver); + const ledger = driver.backfills; + if (ledger === undefined) throw new Error('the memory driver must ship a backfill ledger'); + return ledger; +}; + +beforeEach(() => { + resetJobs(); +}); + +afterEach(() => { + resetJobDriver(); +}); + +describe('one pass', () => { + test('defaults to one statement per DEFAULT_BACKFILL_BATCH rows', async () => { + const test10 = harness(); + await test10.run(); + // Ten rows, one default-sized batch: the default is what the pass used. + expect(DEFAULT_BACKFILL_BATCH).toBeGreaterThan(10); + expect(test10.seen.length).toBe(1); + expect(test10.watch.reads).toBe(1); + }); + + test('visits every matching row once, in key order, one statement per batch', async () => { + const pass = harness({ batch: 3 }); + + const report = await pass.run(); + + expect(pass.seen).toEqual([ + ['row-0', 'row-10', 'row-20'], + ['row-30', 'row-40', 'row-50'], + ['row-60', 'row-70', 'row-80'], + ['row-90'], + ]); + expect(report).toEqual({ name: 'rewrite-titles', batches: 4, rows: 10, skipped: false }); + // Four batches, four statements: the exhausted source costs no extra read. + expect(pass.watch.reads).toBe(4); + // One iteration for the whole pass, closed once on the way out. + expect(pass.watch.opens).toBe(1); + expect(pass.watch.closes).toBe(1); + }); + + test('an empty source is one step, no body call and a zero report', async () => { + const pass = harness({ batch: 3, name: 'empty-pass' }); + const table = tableFor(rows, memoryRepo(rows, [])); + const empty = backfill({ + name: 'nothing-to-do', + batch: 3, + source: () => table.where({ orgId: ORG }), + handle: () => expect.unreachable('an empty source has no batch to hand over'), + }); + const runner = createStepRunner({ runId: RUN_ID, jobName: 'nothing', store: pass.store }); + + const report = await empty.run({ + input: {}, + step: runner.step, + ctx, + attempt: 1, + jobId: 'job-2', + runId: RUN_ID, + }); + + expect(report).toEqual({ name: 'nothing-to-do', batches: 0, rows: 0, skipped: false }); + expect((await pass.steps()).map((record) => record.name)).toEqual(['batch:0']); + }); + + test('checkpoints a bounded cursor and a count — never the page', async () => { + const pass = harness({ batch: 3 }); + + await pass.run(); + + const steps = await pass.steps(); + // Deterministic and positional: a replay finds a step by the name the next attempt mints. + expect(steps.map((record) => record.name)).toEqual([ + 'batch:0', + 'batch:1', + 'batch:2', + 'batch:3', + ]); + const checkpoints = steps.map((record) => record.output as Record); + for (const [index, output] of checkpoints.entries()) { + expect(steps[index]?.status).toBe('completed'); + // `steps.ts` retains a completed step's output for the whole run, so a checkpoint carrying + // its page would hold every row the pass has touched until the job ended. + expect(Object.keys(output).sort()).toEqual(['cursor', 'rows']); + expect(typeof output.rows).toBe('number'); + expect(output.cursor === null || typeof output.cursor === 'string').toBe(true); + } + // The pass ends because the source did, and that is what the last checkpoint records. + expect(checkpoints.at(-1)?.cursor).toBeNull(); + }); +}); + +describe('resume', () => { + test('a failed attempt resumes on its own page and re-reads none before it', async () => { + const pass = harness({ batch: 3 }); + pass.failOn = new Set([1]); + + await expect(pass.run()).rejects.toThrow('batch 1 failed'); + + // Two statements: the batch that worked and the one that failed. + expect(pass.watch.reads).toBe(2); + expect(pass.seen).toEqual([['row-0', 'row-10', 'row-20']]); + expect(pass.watch.closes).toBe(1); + + pass.failOn = new Set(); + const report = await pass.run(); + + // Batch 0 came from storage, so the second attempt read three pages, not four. + expect(pass.watch.reads).toBe(5); + expect(pass.seen).toEqual([ + ['row-0', 'row-10', 'row-20'], + ['row-30', 'row-40', 'row-50'], + ['row-60', 'row-70', 'row-80'], + ['row-90'], + ]); + expect(report).toEqual({ name: 'rewrite-titles', batches: 4, rows: 10, skipped: false }); + }); + + test('a completed pass replays whole: no statement, no body call, the same report', async () => { + const pass = harness({ batch: 3 }); + const first = await pass.run(); + + const readsAfterFirst = pass.watch.reads; + const second = await pass.run(); + + expect(second).toEqual(first); + expect(pass.watch.reads).toBe(readsAfterFirst); + expect(pass.watch.opens).toBe(1); + expect(pass.seen.length).toBe(4); + }); + + test('a step re-opened in the middle rebuilds the iteration at the checkpoint', async () => { + const pass = harness({ batch: 3 }); + await pass.run(); + + // What `retryFromStep` does: drop ONE step so it re-executes, leaving the ones after it + // memoized. The cursor therefore JUMPS over a page the live iteration never read, and an + // iteration trusted to be in step with the checkpoints would hand batch 3 the rows of batch 2. + await pass.store.del(RUN_ID, 'batch:1'); + await pass.store.del(RUN_ID, 'batch:3'); + const readsAfterFirst = pass.watch.reads; + const seenAfterFirst = pass.seen.length; + + const report = await pass.run(); + + expect(pass.seen.slice(seenAfterFirst)).toEqual([['row-30', 'row-40', 'row-50'], ['row-90']]); + // One statement per re-opened step, and a second iteration because the cursor jumped. + expect(pass.watch.reads).toBe(readsAfterFirst + 2); + expect(pass.watch.opens).toBe(3); + expect(pass.watch.closes).toBe(3); + expect(report).toEqual({ name: 'rewrite-titles', batches: 4, rows: 10, skipped: false }); + }); + + test('a replayed step that is not a checkpoint fails the run instead of restarting it', async () => { + const pass = harness({ batch: 3 }); + await pass.run(); + const first = await pass.store.get(RUN_ID, 'batch:0'); + expect(first).toBeDefined(); + // `steps.ts` returns a completed step's output through an unchecked cast, so this is what a + // run id carrying another job's step under the same name looks like from in here. An absent + // cursor is not `null`: unchecked, the pass would reopen the source at the top and walk the + // whole table again, reporting a clean resume. + if (first !== undefined) await pass.store.put({ ...first, output: { done: true } }); + const readsAfterFirst = pass.watch.reads; + + await expect(pass.run()).rejects.toThrow(/not a backfill checkpoint/); + + expect(pass.watch.reads).toBe(readsAfterFirst); + }); +}); + +describe('the x_backfills ledger', () => { + test('one pass is one row: running while it sweeps, completed with what it swept', async () => { + const ledger = installLedger(); + const pass = harness({ batch: 3 }); + + await pass.run(); + + const runs = await ledger.list({ name: 'rewrite-titles' }); + expect(runs).toHaveLength(1); + expect(runs[0]).toMatchObject({ + runId: RUN_ID, + name: 'rewrite-titles', + status: 'completed', + rows: 10, + cursor: null, + appVersion: 'dev', + }); + // The definition's own hash, so a later edit is decidable rather than guessed at. + expect(runs[0]?.checksum).toMatch(/^[0-9a-f]{32}$/); + }); + + test('progress moves with the pass, batch by batch, and survives the failure that stops it', async () => { + const ledger = installLedger(); + const pass = harness({ batch: 3 }); + pass.failOn = new Set([2]); + + await expect(pass.run()).rejects.toThrow('batch 2 failed'); + + const failed = (await ledger.list())[0]; + // Two batches landed before the third threw, and where it stopped is kept. + expect(failed).toMatchObject({ status: 'failed', rows: 6 }); + expect(typeof failed?.cursor).toBe('string'); + + pass.failOn = new Set(); + await pass.run(); + + // A retry is the SAME pass: it adopts its own row rather than opening a second one. + const runs = await ledger.list(); + expect(runs).toHaveLength(1); + expect(runs[0]).toMatchObject({ status: 'completed', rows: 10, cursor: null }); + expect(runs[0]?.startedAt).toBe(failed?.startedAt ?? -1); + }); + + test('a completed name re-enqueued is a no-op with a report', async () => { + const ledger = installLedger(); + const pass = harness({ batch: 3 }); + await pass.run(); + const readsAfterFirst = pass.watch.reads; + const seenAfterFirst = pass.seen.length; + + const report = await pass.run({ runId: 'run-backfill-2' }); + + expect(report).toEqual({ + name: 'rewrite-titles', + batches: 0, + rows: 0, + skipped: true, + previousRunId: RUN_ID, + }); + // No statement, no body call, and no row for a pass that never opened one. + expect(pass.watch.reads).toBe(readsAfterFirst); + expect(pass.seen.length).toBe(seenAfterFirst); + expect(await ledger.list()).toHaveLength(1); + }); + + test('force sweeps again and writes a NEW row, keeping the one it reruns', async () => { + const ledger = installLedger(); + const pass = harness({ batch: 3 }); + await pass.run(); + + const report = await pass.run({ runId: 'run-backfill-2', input: { force: true } }); + + expect(report).toEqual({ name: 'rewrite-titles', batches: 4, rows: 10, skipped: false }); + expect(pass.seen.length).toBe(8); + const runs = await ledger.list({ name: 'rewrite-titles' }); + expect(runs.map((run) => run.runId)).toEqual(['run-backfill-2', RUN_ID]); + // History, not an edit of it: both passes stay readable. + expect(runs.every((run) => run.status === 'completed')).toBe(true); + }); + + test('a definition that moved since the completed pass warns, and still does not run', async () => { + const ledger = installLedger(); + const first = harness({ batch: 3 }); + await first.run(); + resetJobs(); + + // The same durable name, a different body — what editing a shipped backfill looks like. + const table = tableFor(rows, memoryRepo(rows, SEED)); + const edited = backfill({ + name: 'rewrite-titles', + batch: 3, + source: () => table.where({ orgId: ORG }), + handle: () => expect.unreachable('a completed name is a no-op until it is forced'), + }); + const runner = createStepRunner({ + runId: 'run-edited', + jobName: 'backfill', + store: first.store, + }); + const warn = spyOn(logger, 'warn'); + + const report = await edited.run({ + input: {}, + step: runner.step, + ctx, + attempt: 1, + jobId: 'job-3', + runId: 'run-edited', + }); + + const changed = warn.mock.calls.filter( + (call) => call[0] === 'jobs.backfill.definition-changed', + ); + warn.mockRestore(); + // A warning and never a refusal: this checksum is over source text, which a bundler moves. + expect(changed).toHaveLength(1); + expect(report).toMatchObject({ skipped: true, previousRunId: RUN_ID }); + expect(await ledger.list()).toHaveLength(1); + }); + + test('a suspended pass is parked, not failed', async () => { + const ledger = installLedger(); + const store = createMemoryStepStore(); + const table = tableFor(rows, memoryRepo(rows, SEED)); + const parked = backfill({ + name: 'parks-itself', + batch: 3, + source: () => table.where({ orgId: ORG }), + handle: () => { + throw new StepSuspension({ step: 'batch:0', resumeAt: 1, reason: 'sleep' }); + }, + }); + const runner = createStepRunner({ runId: 'run-parked', jobName: 'backfill', store }); + + await expect( + parked.run({ + input: {}, + step: runner.step, + ctx, + attempt: 1, + jobId: 'job-4', + runId: 'run-parked', + }), + ).rejects.toBeInstanceOf(StepSuspension); + + // Control flow, not a failure: a parked run is coming back to this exact step. + expect((await ledger.list())[0]).toMatchObject({ status: 'running', rows: 0 }); + }); + + test('with no driver installed the pass runs and reports, unblocked and unrecorded', async () => { + // The documented degradation: a driver that ships no ledger runs backfills with no + // bookkeeping rather than refusing them — so a completed name has nothing to block on. + const pass = harness({ batch: 3 }); + await pass.run(); + + const report = await pass.run({ runId: 'run-backfill-2' }); + + expect(report).toEqual({ name: 'rewrite-titles', batches: 4, rows: 10, skipped: false }); + expect(pass.seen.length).toBe(8); + }); +}); diff --git a/packages/jobs/src/backfill-pass.ts b/packages/jobs/src/backfill-pass.ts new file mode 100644 index 000000000..3a2c47122 --- /dev/null +++ b/packages/jobs/src/backfill-pass.ts @@ -0,0 +1,202 @@ +// One `backfill()` pass: the iteration, its durable checkpoints, and the `x_backfills` row that +// reports them. Split out of `backfill.ts` for the reason `execute.ts` is split out of `job.ts` — +// that file declares a backfill, this one runs it. +// +// `inBatches()` reads the source one statement per page, and each page is handled inside its own +// `step.run`, so an attempt killed mid-pass resumes on the page it stopped at instead of reading +// the table again from the top. What a step persists is the CURSOR and the row count, never the +// page: `steps.ts` hands a completed step's output back for the whole run, so checkpointing rows +// would retain every row of every batch already processed until the job ended — the leak that +// turns a backfill of a large table into an OOM. +// +// The ledger is the OTHER half and never a second copy of this one: a step checkpoint is written +// in step with the work and decides where a resumed pass restarts, while the `x_backfills` row is +// a report an operator reads and the record that a completed name has already been swept. + +import { appVersion, assert, logger } from '@ultimat3/core'; +import type { BatchIterator } from '@ultimat3/entity'; +import type { BackfillDefinition, BackfillInput, BackfillReport } from './backfill'; +import type { BackfillLedger, BackfillRun } from './backfill-ledger'; +import { decideBackfill } from './backfill-ledger'; +import { jobDriver } from './driver'; +import type { JobRunArgs } from './job'; +import { isStepSuspension } from './steps'; + +/** Every batch is a step, and this is the name it is checkpointed under. See `backfillPass()`. */ +const STEP_PREFIX = 'batch:'; + +/** What a step persists: a bounded position, never the page it came from. See the file header. */ +interface Checkpoint { + /** Where the next batch starts; `null` once the pass is over. */ + readonly cursor: string | null; + readonly rows: number; +} + +/** One live iteration and the pull that advances it. Rebuilt when the checkpoints disagree. */ +interface Iteration { + readonly batches: BatchIterator; + readonly pull: AsyncIterator; +} + +/** Everything about the backfill that is fixed at declaration, hashed and sized once. */ +export interface BackfillPlan { + readonly definition: BackfillDefinition; + readonly size: number; + readonly checksum: string; +} + +const fieldOf = (value: unknown, key: string): unknown => + typeof value === 'object' && value !== null ? (value as Record)[key] : undefined; + +/** + * `steps.ts` hands a completed step's output back through an unchecked `as T`, so a checkpoint + * READ from storage is this shape by claim and never by check. It is checked here because the + * failure it would otherwise cause is silent and expensive: an absent cursor is not `null`, so the + * loop would reopen the source at the top and walk the whole table a second time. + */ +function asCheckpoint(value: unknown, step: string): Checkpoint { + const cursor = fieldOf(value, 'cursor'); + const rows = fieldOf(value, 'rows'); + assert( + (cursor === null || typeof cursor === 'string') && typeof rows === 'number', + `step "${step}" replayed ${JSON.stringify(value)}, which is not a backfill checkpoint`, + `x jobs show --json prints the run's steps — a run id whose "${step}" was written by something other than this backfill has to be retired, not resumed`, + ); + return { cursor, rows }; +} + +/** + * The newest COMPLETED pass under this name, or nothing. A `running` row is this pass resuming or + * one another worker holds the single live idempotency key for, and a `failed` one is an attempt + * the queue is about to retry — neither is a sweep that has already happened. + */ +async function completedRun( + ledger: BackfillLedger | undefined, + name: string, +): Promise { + if (ledger === undefined) return undefined; + const rows = await ledger.list({ name, status: 'completed', limit: 1 }); + return rows[0]; +} + +/** + * Record the attempt that ended badly and let the original error through untouched: the queue's + * retry decision is made on it, so a bookkeeping failure that replaced it would dead-letter — or + * silently retry — for the wrong reason. Logged rather than swallowed, because a ledger nobody + * can write is worth exactly one line. + */ +async function markFailed( + ledger: BackfillLedger | undefined, + runId: string, + rows: number, +): Promise { + try { + await ledger?.finish(runId, { status: 'failed', rows }); + } catch (error) { + logger.warn('jobs.backfill.ledger-failed', { + runId, + error: error instanceof Error ? error.message : String(error), + }); + } +} + +/** + * The pass itself: one `step.run` per batch, named by position so a replay finds it. Completed + * steps are served from storage without touching the database, so a resumed attempt walks its own + * history to the last checkpoint and opens the source there. + */ +export async function backfillPass( + plan: BackfillPlan, + args: JobRunArgs, +): Promise { + const { definition, size, checksum } = plan; + const { ctx, step, runId } = args; + const name = definition.name; + // Absent on a driver that ships no ledger, which runs the pass with no bookkeeping rather than + // refusing it — the same degradation `introspect` already has. + const ledger = jobDriver()?.backfills; + + const previous = await completedRun(ledger, name); + const verdict = decideBackfill(previous, checksum, args.input.force === true); + if (!verdict.run) { + if (verdict.changed) { + // A warning and never a refusal, unlike `@ultimat3/db`'s `auditLedger`: this checksum is + // over function source text, which a bundler can move without a line of behaviour changing. + logger.warn('jobs.backfill.definition-changed', { + backfill: name, + completedAs: previous?.checksum, + now: checksum, + fix: `enqueue ${name} with { force: true } to sweep again under the new definition`, + }); + } + logger.info('jobs.backfill.skipped', { backfill: name, completedBy: previous?.runId }); + return { + name, + batches: 0, + rows: 0, + skipped: true, + ...(previous === undefined ? {} : { previousRunId: previous.runId }), + }; + } + await ledger?.start({ runId, name, checksum, appVersion: appVersion() }); + + let cursor: string | null = null; + let rows = 0; + let batches = 0; + let live: Iteration | undefined; + + /** + * The iteration positioned at `cursor`, opened on the first batch this attempt actually runs so + * a resumed pass sends no statement for a page it already handled. + * + * `batches.cursor` IS where the next statement starts, so comparing it with the checkpoint's is + * the whole staleness test — and it is not academic: `retryFromStep` re-opens ONE step in the + * middle of a finished run, which moves the checkpoints somewhere this iteration is not. Rebuilt + * from the checkpoint rather than read from wherever the old one was parked. + */ + const iterate = async (): Promise> => { + if (live !== undefined && live.batches.cursor === cursor) return live; + await live?.batches.close(); + const opened = definition.source({ ctx }).after(cursor).inBatches(size); + live = { batches: opened, pull: opened[Symbol.asyncIterator]() }; + return live; + }; + + try { + for (let index = 0; ; index += 1) { + const stepName = `${STEP_PREFIX}${index}`; + const checkpoint = asCheckpoint( + await step.run(stepName, async (signal): Promise => { + const iteration = await iterate(); + const next = await iteration.pull.next(); + if (next.done === true) return { cursor: null, rows: 0 }; + await definition.handle({ rows: next.value, ctx, signal, index }); + return { cursor: iteration.batches.cursor, rows: next.value.length }; + }), + stepName, + ); + rows += checkpoint.rows; + cursor = checkpoint.cursor; + // `inBatches()` never yields an empty batch, so rows is what tells a handled page from the + // one step an exhausted source costs — and what keeps the exhausted step off the ledger, + // whose last write is `finish` either way. + if (checkpoint.rows > 0) { + batches += 1; + // Absolute, so a replayed batch reports the position it reported the first time. + await ledger?.progress(runId, { rows, cursor }); + } + if (cursor === null) break; + } + } catch (error) { + // Control flow, not a failure: a suspended run is parked and will be back on this step. + if (!isStepSuspension(error)) await markFailed(ledger, runId, rows); + throw error; + } finally { + // Whatever the iteration holds belongs to this attempt, and an attempt that failed, was + // cancelled or finished is done with it either way. + await live?.batches.close(); + } + + await ledger?.finish(runId, { status: 'completed', rows }); + return { name, batches, rows, skipped: false }; +} diff --git a/packages/jobs/src/backfill.test.ts b/packages/jobs/src/backfill.test.ts index 3d8325baf..0d844430c 100644 --- a/packages/jobs/src/backfill.test.ts +++ b/packages/jobs/src/backfill.test.ts @@ -1,144 +1,22 @@ -// `backfill()` is a factory over `job()`, so what needs pinning here is not that a job exists — -// `job.test.ts` owns that — but the pass: every row visited once, a killed attempt resuming on the -// page it stopped at WITHOUT re-reading the ones before it, and a checkpoint that is a cursor -// rather than the rows behind it. The source is a real `@ultimat3/entity` chain over a real -// `memoryRepo`, so the cursors, the keyset paging and `inBatches()`'s own refusals are the shipped -// ones and not a fixture that agrees with the factory by construction. +// `backfill()` is a factory over `job()`, so what this file pins is the DECLARATION: the handle it +// returns is a real registered job, keyed by the durable name, and a batch size that is not a whole +// number of rows is refused where it was written. The pass itself is `backfill-pass.test.ts`. import { beforeEach, describe, expect, test } from 'bun:test'; -import type { Ctx } from '@ultimat3/core'; -import { createContext, isUltimateError } from '@ultimat3/core'; -import type { BatchIterator, ReadBuilder, Repo } from '@ultimat3/entity'; +import { isUltimateError } from '@ultimat3/core'; import { entity, memoryRepo, tableFor, text, uuid } from '@ultimat3/entity'; -import { backfill, DEFAULT_BACKFILL_BATCH } from './backfill'; +import { backfill } from './backfill'; import { getJob, isJobHandle, resetJobs } from './job'; -import type { StepRecord, StepStore } from './steps'; -import { createMemoryStepStore, createStepRunner } from './steps'; -const rows = entity('backfill_test_rows', { +const rows = entity('backfill_factory_rows', { columns: { id: uuid().primaryKey(), orgId: uuid(), title: text({ max: 40 }) }, }); type Row = typeof rows.$row; const ORG = '00000000-0000-7000-8000-0000000000a1'; -const OTHER = '00000000-0000-7000-8000-0000000000a2'; -/** Sortable ids, ten apart, so the keyset cursor between two pages is a real position. */ -const id = (index: number): string => - `00000000-0000-7000-8000-0000000001${String(index).padStart(2, '0')}`; - -const row = (index: number, orgId: string = ORG): Row => ({ - id: id(index), - orgId, - title: `row-${index}`, -}); - -/** Ten rows for the org under test, two more no pass may ever touch. */ -const SEED: readonly Row[] = [ - ...Array.from({ length: 10 }, (_unused, index) => row(index * 10)), - row(900, OTHER), - row(910, OTHER), -]; - -/** What the wrappers below record: statements sent, iterations opened, iterations closed. */ -interface Watch { - reads: number; - opens: number; - closes: number; -} - -const newWatch = (): Watch => ({ reads: 0, opens: 0, closes: 0 }); - -/** `cursor` is a getter on the real iterator, so it is forwarded rather than spread-copied. */ -const watchedIterator = (iterator: BatchIterator, watch: Watch): BatchIterator => { - const close = async (): Promise => { - watch.closes += 1; - await iterator.close(); - }; - return { - get cursor(): string | null { - return iterator.cursor; - }, - [Symbol.asyncIterator]: () => iterator[Symbol.asyncIterator](), - close, - [Symbol.asyncDispose]: close, - }; -}; - -/** `after()` returns a NEW chain, so the wrapper has to survive it — the factory calls both. */ -const watchedChain = (chain: ReadBuilder, watch: Watch): ReadBuilder => ({ - ...chain, - after: (cursor) => watchedChain(chain.after(cursor), watch), - inBatches: (size) => { - watch.opens += 1; - return watchedIterator(chain.inBatches(size), watch); - }, -}); - -const countingRepo = (repo: Repo, watch: Watch): Repo => ({ - ...repo, - findMany: async (args) => { - watch.reads += 1; - return repo.findMany(args); - }, -}); - -interface Harness { - readonly watch: Watch; - readonly store: StepStore; - /** Titles handed to the body, one entry per batch it actually ran. */ - readonly seen: readonly (readonly string[])[]; - /** Every batch index the body should reject on. */ - failOn: ReadonlySet; - run(): Promise; - steps(): Promise; -} - -const RUN_ID = 'run-backfill-1'; - -const ctx: Ctx = createContext(); - -/** - * One backfill over `SEED`, driven through a real step runner rather than a worker: this file is - * about the pass and its checkpoints, and `execute.ts` already owns what a driver does with the - * outcome. Every `run()` is a fresh attempt against the SAME store and run id, which is exactly - * what a retry is. - */ -const harness = (options: { batch?: number; name?: string } = {}): Harness => { - const watch = newWatch(); - const store = createMemoryStepStore(); - const seen: string[][] = []; - const table = tableFor(rows, countingRepo(memoryRepo(rows, SEED), watch)); - const state: Harness = { - watch, - store, - seen, - failOn: new Set(), - async run() { - const runner = createStepRunner({ runId: RUN_ID, jobName: 'backfill', store }); - return handle.run({ - input: {}, - step: runner.step, - ctx, - attempt: 1, - jobId: 'job-1', - runId: RUN_ID, - }); - }, - steps: () => store.list(RUN_ID), - }; - const handle = backfill({ - name: options.name ?? 'rewrite-titles', - ...(options.batch === undefined ? {} : { batch: options.batch }), - source: () => watchedChain(table.where({ orgId: ORG }), watch), - handle: ({ rows: page, index }) => { - if (state.failOn.has(index)) throw new Error(`batch ${String(index)} failed`); - seen.push(page.map((entry) => entry.title)); - }, - }); - return state; -}; +const source = () => tableFor(rows, memoryRepo(rows, [])).where({ orgId: ORG }); beforeEach(() => { resetJobs(); @@ -150,7 +28,7 @@ describe('the factory', () => { name: 'rewrite-titles', queue: 'maintenance', retry: { attempts: 7, backoff: 'fixed' }, - source: () => tableFor(rows, memoryRepo(rows, SEED)).where({ orgId: ORG }), + source, handle: () => undefined, }); @@ -169,7 +47,7 @@ describe('the factory', () => { backfill({ name: `bad-${String(batch)}`, batch, - source: () => tableFor(rows, memoryRepo(rows, SEED)).where({ orgId: ORG }), + source, handle: () => undefined, }); } catch (error) { @@ -181,163 +59,4 @@ describe('the factory', () => { expect(getJob(`bad-${String(batch)}`)).toBeUndefined(); } }); - - test('defaults to one statement per DEFAULT_BACKFILL_BATCH rows', async () => { - const test10 = harness(); - await test10.run(); - // Ten rows, one default-sized batch: the default is what the pass used. - expect(DEFAULT_BACKFILL_BATCH).toBeGreaterThan(10); - expect(test10.seen.length).toBe(1); - expect(test10.watch.reads).toBe(1); - }); -}); - -describe('one pass', () => { - test('visits every matching row once, in key order, one statement per batch', async () => { - const pass = harness({ batch: 3 }); - - const report = await pass.run(); - - expect(pass.seen).toEqual([ - ['row-0', 'row-10', 'row-20'], - ['row-30', 'row-40', 'row-50'], - ['row-60', 'row-70', 'row-80'], - ['row-90'], - ]); - expect(report).toEqual({ name: 'rewrite-titles', batches: 4, rows: 10 }); - // Four batches, four statements: the exhausted source costs no extra read. - expect(pass.watch.reads).toBe(4); - // One iteration for the whole pass, closed once on the way out. - expect(pass.watch.opens).toBe(1); - expect(pass.watch.closes).toBe(1); - }); - - test('an empty source is one step, no body call and a zero report', async () => { - const pass = harness({ batch: 3, name: 'empty-pass' }); - const table = tableFor(rows, memoryRepo(rows, [])); - const empty = backfill({ - name: 'nothing-to-do', - batch: 3, - source: () => table.where({ orgId: ORG }), - handle: () => expect.unreachable('an empty source has no batch to hand over'), - }); - const runner = createStepRunner({ runId: RUN_ID, jobName: 'nothing', store: pass.store }); - - const report = await empty.run({ - input: {}, - step: runner.step, - ctx, - attempt: 1, - jobId: 'job-2', - runId: RUN_ID, - }); - - expect(report).toEqual({ name: 'nothing-to-do', batches: 0, rows: 0 }); - expect((await pass.steps()).map((record) => record.name)).toEqual(['batch:0']); - }); - - test('checkpoints a bounded cursor and a count — never the page', async () => { - const pass = harness({ batch: 3 }); - - await pass.run(); - - const steps = await pass.steps(); - // Deterministic and positional: a replay finds a step by the name the next attempt mints. - expect(steps.map((record) => record.name)).toEqual([ - 'batch:0', - 'batch:1', - 'batch:2', - 'batch:3', - ]); - const checkpoints = steps.map((record) => record.output as Record); - for (const [index, output] of checkpoints.entries()) { - expect(steps[index]?.status).toBe('completed'); - // `steps.ts` retains a completed step's output for the whole run, so a checkpoint carrying - // its page would hold every row the pass has touched until the job ended. - expect(Object.keys(output).sort()).toEqual(['cursor', 'rows']); - expect(typeof output.rows).toBe('number'); - expect(output.cursor === null || typeof output.cursor === 'string').toBe(true); - } - // The pass ends because the source did, and that is what the last checkpoint records. - expect(checkpoints.at(-1)?.cursor).toBeNull(); - }); -}); - -describe('resume', () => { - test('a failed attempt resumes on its own page and re-reads none before it', async () => { - const pass = harness({ batch: 3 }); - pass.failOn = new Set([1]); - - await expect(pass.run()).rejects.toThrow('batch 1 failed'); - - // Two statements: the batch that worked and the one that failed. - expect(pass.watch.reads).toBe(2); - expect(pass.seen).toEqual([['row-0', 'row-10', 'row-20']]); - expect(pass.watch.closes).toBe(1); - - pass.failOn = new Set(); - const report = await pass.run(); - - // Batch 0 came from storage, so the second attempt read three pages, not four. - expect(pass.watch.reads).toBe(5); - expect(pass.seen).toEqual([ - ['row-0', 'row-10', 'row-20'], - ['row-30', 'row-40', 'row-50'], - ['row-60', 'row-70', 'row-80'], - ['row-90'], - ]); - expect(report).toEqual({ name: 'rewrite-titles', batches: 4, rows: 10 }); - }); - - test('a completed pass replays whole: no statement, no body call, the same report', async () => { - const pass = harness({ batch: 3 }); - const first = await pass.run(); - - const readsAfterFirst = pass.watch.reads; - const second = await pass.run(); - - expect(second).toEqual(first); - expect(pass.watch.reads).toBe(readsAfterFirst); - expect(pass.watch.opens).toBe(1); - expect(pass.seen.length).toBe(4); - }); - - test('a step re-opened in the middle rebuilds the iteration at the checkpoint', async () => { - const pass = harness({ batch: 3 }); - await pass.run(); - - // What `retryFromStep` does: drop ONE step so it re-executes, leaving the ones after it - // memoized. The cursor therefore JUMPS over a page the live iteration never read, and an - // iteration trusted to be in step with the checkpoints would hand batch 3 the rows of batch 2. - await pass.store.del(RUN_ID, 'batch:1'); - await pass.store.del(RUN_ID, 'batch:3'); - const readsAfterFirst = pass.watch.reads; - const seenAfterFirst = pass.seen.length; - - const report = await pass.run(); - - expect(pass.seen.slice(seenAfterFirst)).toEqual([['row-30', 'row-40', 'row-50'], ['row-90']]); - // One statement per re-opened step, and a second iteration because the cursor jumped. - expect(pass.watch.reads).toBe(readsAfterFirst + 2); - expect(pass.watch.opens).toBe(3); - expect(pass.watch.closes).toBe(3); - expect(report).toEqual({ name: 'rewrite-titles', batches: 4, rows: 10 }); - }); - - test('a replayed step that is not a checkpoint fails the run instead of restarting it', async () => { - const pass = harness({ batch: 3 }); - await pass.run(); - const first = await pass.store.get(RUN_ID, 'batch:0'); - expect(first).toBeDefined(); - // `steps.ts` returns a completed step's output through an unchecked cast, so this is what a - // run id carrying another job's step under the same name looks like from in here. An absent - // cursor is not `null`: unchecked, the pass would reopen the source at the top and walk the - // whole table again, reporting a clean resume. - if (first !== undefined) await pass.store.put({ ...first, output: { done: true } }); - const readsAfterFirst = pass.watch.reads; - - await expect(pass.run()).rejects.toThrow(/not a backfill checkpoint/); - - expect(pass.watch.reads).toBe(readsAfterFirst); - }); }); diff --git a/packages/jobs/src/backfill.ts b/packages/jobs/src/backfill.ts index c22939e51..e0bb3a920 100644 --- a/packages/jobs/src/backfill.ts +++ b/packages/jobs/src/backfill.ts @@ -5,19 +5,18 @@ // the worker's cancellation, the dead-letter path, `x jobs show` and a manifest row without a line // here, and it is why nothing in the framework grows a ninth kind of thing to hold table sweeps. // -// What the factory adds is the iteration. `inBatches()` reads the source one statement per page, -// and each page is handled inside its own `step.run`, so an attempt killed mid-pass resumes on the -// page it stopped at instead of reading the table again from the top. What a step persists is the -// CURSOR and the row count, never the page: `steps.ts` hands a completed step's output back for the -// whole run, so checkpointing rows would retain every row of every batch already processed until -// the job ended — the leak that turns a backfill of a large table into an OOM. +// The declaration lives here and the pass lives in `backfill-pass.ts` — the same split `job.ts` +// and `execute.ts` already have, and the reason the iteration, the checkpoints and the +// `x_backfills` ledger are one file's problem rather than this one's. import type { Ctx } from '@ultimat3/core'; import { assert } from '@ultimat3/core'; -import type { BatchIterator, ReadBuilder } from '@ultimat3/entity'; +import type { ReadBuilder } from '@ultimat3/entity'; import { t } from '@ultimat3/schema'; +import { backfillChecksum } from './backfill-ledger'; +import { backfillPass } from './backfill-pass'; import type { DurationInput } from './clock'; -import type { JobHandle, JobRunArgs } from './job'; +import type { JobHandle } from './job'; import { job } from './job'; import type { RetryPolicy } from './retry'; import { DEFAULT_RETRY } from './retry'; @@ -30,9 +29,6 @@ import { DEFAULT_RETRY } from './retry'; */ export const DEFAULT_BACKFILL_BATCH = 1_000; -/** Every batch is a step, and this is the name it is checkpointed under. See `pass()`. */ -const STEP_PREFIX = 'batch:'; - export interface BackfillBatch { /** The page `page()` would have returned here: tenancy, soft delete, projection, preloads. */ readonly rows: readonly Row[]; @@ -82,49 +78,27 @@ export interface BackfillDefinition { /** What one completed pass reports — bounded, so `x jobs show` can print it. */ export interface BackfillReport { readonly name: string; - /** Batches this pass handled, replayed ones included. */ + /** Batches THIS pass handled, replayed ones included. `0` when it was skipped. */ readonly batches: number; readonly rows: number; + /** True when `x_backfills` already held a completed pass under this name and `force` was not set. */ + readonly skipped: boolean; + /** The completed pass the ledger answered with, when there was one. */ + readonly previousRunId?: string | undefined; } /** - * A backfill's payload is its identity and nothing else. What to visit is declared, so a queue row - * carries no parameters that could drift from the definition, and a run enqueued last week resumes - * against today's source rather than a stale copy of it. - */ -export type BackfillInput = Record; - -/** What a step persists: a bounded position, never the page it came from. See the file header. */ -interface Checkpoint { - /** Where the next batch starts; `null` once the pass is over. */ - readonly cursor: string | null; - readonly rows: number; -} - -/** One live iteration and the pull that advances it. Rebuilt when the checkpoints disagree. */ -interface Iteration { - readonly batches: BatchIterator; - readonly pull: AsyncIterator; -} - -const fieldOf = (value: unknown, key: string): unknown => - typeof value === 'object' && value !== null ? (value as Record)[key] : undefined; - -/** - * `steps.ts` hands a completed step's output back through an unchecked `as T`, so a checkpoint - * READ from storage is this shape by claim and never by check. It is checked here because the - * failure it would otherwise cause is silent and expensive: an absent cursor is not `null`, so the - * loop would reopen the source at the top and walk the whole table a second time. + * A backfill's payload is its identity plus the one decision a queue row is allowed to carry. + * What to visit is declared, so nothing here can drift from the definition; `force` changes only + * whether a name the ledger already records as completed runs a SECOND pass, and never what that + * pass would do. */ -function asCheckpoint(value: unknown, step: string): Checkpoint { - const cursor = fieldOf(value, 'cursor'); - const rows = fieldOf(value, 'rows'); - assert( - (cursor === null || typeof cursor === 'string') && typeof rows === 'number', - `step "${step}" replayed ${JSON.stringify(value)}, which is not a backfill checkpoint`, - `x jobs show --json prints the run's steps — a run id whose "${step}" was written by something other than this backfill has to be retired, not resumed`, - ); - return { cursor, rows }; +export interface BackfillInput { + /** + * Run even though `x_backfills` holds a completed pass under this name. The rerun is a NEW + * ledger row — history is never overwritten, so what each pass swept stays readable. + */ + readonly force?: boolean | undefined; } export function backfill(definition: BackfillDefinition): JobHandle { @@ -136,78 +110,20 @@ export function backfill(definition: BackfillDefinition): JobHandle({ name: definition.name, - input: t.object({}), - // One live run per name. A second enqueue while the pass is still going is the same pass, and - // deduping it is what makes "kick it again" safe rather than a second writer on one table. + input: t.object({ force: t.boolean.optional() }), + // One live run per name, forced or not. A second enqueue while the pass is still going is the + // same pass, and deduping it is what makes "kick it again" safe rather than a second writer on + // one table — which is exactly what a `force` in the key would have allowed. idempotencyKey: () => definition.name, retry: definition.retry ?? DEFAULT_RETRY, ...(definition.queue === undefined ? {} : { queue: definition.queue }), ...(definition.timeout === undefined ? {} : { timeout: definition.timeout }), - run: (args) => pass(definition, size, args), + run: (args) => backfillPass({ definition, size, checksum }, args), }); } - -/** - * The pass itself: one `step.run` per batch, named by position so a replay finds it. Completed - * steps are served from storage without touching the database, so a resumed attempt walks its own - * history to the last checkpoint and opens the source there. - */ -async function pass( - definition: BackfillDefinition, - size: number, - args: JobRunArgs, -): Promise { - const { ctx, step } = args; - let cursor: string | null = null; - let rows = 0; - let batches = 0; - let live: Iteration | undefined; - - /** - * The iteration positioned at `cursor`, opened on the first batch this attempt actually runs so - * a resumed pass sends no statement for a page it already handled. - * - * `batches.cursor` IS where the next statement starts, so comparing it with the checkpoint's is - * the whole staleness test — and it is not academic: `retryFromStep` re-opens ONE step in the - * middle of a finished run, which moves the checkpoints somewhere this iteration is not. Rebuilt - * from the checkpoint rather than read from wherever the old one was parked. - */ - const iterate = async (): Promise> => { - if (live !== undefined && live.batches.cursor === cursor) return live; - await live?.batches.close(); - const opened = definition.source({ ctx }).after(cursor).inBatches(size); - live = { batches: opened, pull: opened[Symbol.asyncIterator]() }; - return live; - }; - - try { - for (let index = 0; ; index += 1) { - const name = `${STEP_PREFIX}${index}`; - const checkpoint = asCheckpoint( - await step.run(name, async (signal): Promise => { - const iteration = await iterate(); - const next = await iteration.pull.next(); - if (next.done === true) return { cursor: null, rows: 0 }; - await definition.handle({ rows: next.value, ctx, signal, index }); - return { cursor: iteration.batches.cursor, rows: next.value.length }; - }), - name, - ); - rows += checkpoint.rows; - // `inBatches()` never yields an empty batch, so rows is what tells a handled page from the - // one step an exhausted source costs. - if (checkpoint.rows > 0) batches += 1; - cursor = checkpoint.cursor; - if (cursor === null) break; - } - } finally { - // Whatever the iteration holds belongs to this attempt, and an attempt that failed, was - // cancelled or finished is done with it either way. - await live?.batches.close(); - } - - return { name: definition.name, batches, rows }; -} diff --git a/packages/jobs/src/driver-memory.ts b/packages/jobs/src/driver-memory.ts index b6db53b67..599af30d9 100644 --- a/packages/jobs/src/driver-memory.ts +++ b/packages/jobs/src/driver-memory.ts @@ -4,6 +4,8 @@ import type { Clock } from '@ultimat3/core'; import { assert, systemClock, uuid } from '@ultimat3/core'; +import type { BackfillLedger } from './backfill-ledger'; +import { createMemoryBackfillLedger } from './backfill-ledger'; import { nowMs } from './clock'; import type { ClaimedJob, @@ -25,6 +27,8 @@ import { createMemoryStepStore } from './steps'; export interface MemoryDriverOptions { readonly clock?: Clock; readonly steps?: StepStore; + /** Injectable for the same reason `steps` is: two drivers in one test sharing one ledger. */ + readonly backfills?: BackfillLedger; } const LIVE_STATES = new Set(['ready', 'delayed', 'running', 'suspended']); @@ -32,6 +36,7 @@ const LIVE_STATES = new Set(['ready', 'delayed', 'running', 'suspended']); export function createMemoryDriver(options: MemoryDriverOptions = {}): JobDriver { const clock = options.clock ?? systemClock; const steps = options.steps ?? createMemoryStepStore(); + const backfills = options.backfills ?? createMemoryBackfillLedger(clock); const jobs = new Map(); const liveByKey = (key: string): JobRecord | undefined => { @@ -88,6 +93,7 @@ export function createMemoryDriver(options: MemoryDriverOptions = {}): JobDriver return { name: 'memory', steps, + backfills, introspect, enqueue(request: EnqueueRequest): Promise { diff --git a/packages/jobs/src/driver-pg-sql.test.ts b/packages/jobs/src/driver-pg-sql.test.ts new file mode 100644 index 000000000..e453fce84 --- /dev/null +++ b/packages/jobs/src/driver-pg-sql.test.ts @@ -0,0 +1,40 @@ +// `dev-queue.ts` applies `SQL_JOBS_TABLE` by splitting it on `;` — PGlite speaks the extended +// protocol, which carries one statement per round trip — and names this file as where that stays +// safe. So this is what it pins: every chunk is a whole DDL statement, and no `;` in the constant +// is data. A `;` inside a string literal or a dollar-quoted body would split a statement in half +// and the queue would come up with half a schema, in `x dev` only. + +import { describe, expect, test } from 'bun:test'; +import { SQL_JOBS_TABLE } from './driver-pg-sql'; + +const statements = SQL_JOBS_TABLE.split(';') + .map((statement) => statement.trim()) + .filter((statement) => statement.length > 0); + +describe('the queue DDL', () => { + test('every chunk the `;` split yields is a whole create statement', () => { + for (const statement of statements) { + // Comments lead several of them, and a chunk is a statement only after they are dropped. + const code = statement + .split('\n') + .filter((line) => !line.trimStart().startsWith('--')) + .join('\n') + .trim(); + expect(code).toMatch(/^create (table|unique index|index) if not exists /); + } + }); + + test('no `;` in the constant is data, so nothing splits inside a literal', () => { + // Every quote pairs up: an odd count would mean a `;` sits inside an open literal somewhere, + // which is exactly the case the split cannot see. + expect((SQL_JOBS_TABLE.match(/'/g) ?? []).length % 2).toBe(0); + expect(SQL_JOBS_TABLE).not.toContain('$$'); + }); + + test('declares the three tables the queue and the backfill ledger need', () => { + const tables = statements + .map((statement) => /create table if not exists (\w+)/.exec(statement)?.[1]) + .filter((name): name is string => name !== undefined); + expect(tables).toEqual(['x_jobs', 'x_job_steps', 'x_backfills']); + }); +}); diff --git a/packages/jobs/src/driver-pg-sql.ts b/packages/jobs/src/driver-pg-sql.ts index 0f6bc4415..903ffb607 100644 --- a/packages/jobs/src/driver-pg-sql.ts +++ b/packages/jobs/src/driver-pg-sql.ts @@ -47,6 +47,22 @@ create table if not exists x_job_steps ( error text, primary key (run_id, name) ); + +-- The backfill ledger. Keyed by RUN, not by name: a completed name blocks a re-run, and a forced +-- one writes a new row, so what each pass swept survives the rerun that followed it. +create table if not exists x_backfills ( + run_id uuid primary key, + name text not null, + checksum text not null, + status text not null default 'running', + app_version text not null, + rows_processed bigint not null default 0, + last_cursor text, + started_at timestamptz not null default now(), + completed_at timestamptz +); + +create index if not exists x_backfills_name_idx on x_backfills (name, started_at desc); `.trim(); export const SQL_ENQUEUE = ` @@ -168,3 +184,42 @@ on conflict (run_id, name) do update completed_at = excluded.completed_at, wake_at = excluded.wake_at, attempts = excluded.attempts, error = excluded.error `.trim(); + +/** + * Open the row, or adopt the one an earlier attempt of the SAME run opened. `do update` rather + * than `do nothing` because a failed attempt left `failed` behind and this one is running: + * `started_at`, `app_version` and `checksum` stay as the pass began, and the progress columns + * stay where the last attempt got to. + */ +export const SQL_BACKFILL_START = ` +insert into x_backfills (run_id, name, checksum, app_version) +values ($1, $2, $3, $4) +on conflict (run_id) do update set status = 'running' +`.trim(); + +export const SQL_BACKFILL_PROGRESS = ` +update x_backfills + set rows_processed = $2, last_cursor = $3 + where run_id = $1 +`.trim(); + +/** A failure KEEPS its cursor: where a pass stopped is the first thing anyone asks about one. */ +export const SQL_BACKFILL_FINISH = ` +update x_backfills + set status = $2, + rows_processed = $3, + completed_at = now(), + last_cursor = case when $2 = 'completed' then null else last_cursor end + where run_id = $1 +`.trim(); + +export const SQL_BACKFILL_LIST = ` +select run_id, name, checksum, status, app_version, rows_processed, last_cursor, + (extract(epoch from started_at) * 1000)::bigint as started_at, + (extract(epoch from completed_at) * 1000)::bigint as completed_at + from x_backfills + where ($1::text is null or name = $1) + and ($2::text is null or status = $2) + order by started_at desc + limit $3 +`.trim(); diff --git a/packages/jobs/src/driver-pg.test.ts b/packages/jobs/src/driver-pg.test.ts index 361f91494..4a7825474 100644 --- a/packages/jobs/src/driver-pg.test.ts +++ b/packages/jobs/src/driver-pg.test.ts @@ -3,6 +3,9 @@ import type { PgExecutor } from './driver-pg'; import { createPgDriver } from './driver-pg'; import { SQL_ACK, + SQL_BACKFILL_FINISH, + SQL_BACKFILL_PROGRESS, + SQL_BACKFILL_START, SQL_CLAIM, SQL_ENQUEUE, SQL_HEARTBEAT, @@ -97,3 +100,77 @@ describe('pg driver', () => { await expect(createPgDriver().stats()).rejects.toThrow(DriverUnavailableError); }); }); + +describe('the pg backfill ledger', () => { + const ledgerOf = (executor: PgExecutor) => { + const ledger = createPgDriver({ executor }).backfills; + if (ledger === undefined) throw new Error('the pg driver must ship a backfill ledger'); + return ledger; + }; + + test('every write binds its parameters in the order the statement declares them', async () => { + const executor = recordingExecutor(); + const ledger = ledgerOf(executor); + + await ledger.start({ + runId: 'run-1', + name: 'sweep', + checksum: 'aaaa', + appVersion: '1.2.0', + }); + await ledger.progress('run-1', { rows: 6, cursor: 'c6' }); + await ledger.finish('run-1', { status: 'completed', rows: 10 }); + + expect(executor.calls.map((call) => call.params)).toEqual([ + ['run-1', 'sweep', 'aaaa', '1.2.0'], + ['run-1', 6, 'c6'], + ['run-1', 'completed', 10], + ]); + expect(executor.calls.map((call) => call.sql)).toEqual([ + SQL_BACKFILL_START, + SQL_BACKFILL_PROGRESS, + SQL_BACKFILL_FINISH, + ]); + }); + + test('a row comes back as epoch milliseconds, a number of rows and a nullable cursor', async () => { + // `rows_processed` is a bigint, which a Postgres client hands back as a string, and the two + // timestamps arrive already extracted by the statement. + const executor = recordingExecutor([ + { + run_id: 'run-1', + name: 'sweep', + checksum: 'aaaa', + status: 'completed', + app_version: '1.2.0', + rows_processed: '4200', + last_cursor: null, + started_at: '1000', + completed_at: '2000', + }, + ]); + + const runs = await ledgerOf(executor).list({ name: 'sweep', status: 'completed', limit: 5 }); + + expect(runs).toEqual([ + { + runId: 'run-1', + name: 'sweep', + checksum: 'aaaa', + status: 'completed', + appVersion: '1.2.0', + rows: 4200, + cursor: null, + startedAt: 1000, + completedAt: 2000, + }, + ]); + expect(executor.calls[0]?.params).toEqual(['sweep', 'completed', 5]); + }); + + test('an unfiltered list is nulls and the default limit, never a missing predicate', async () => { + const executor = recordingExecutor(); + await ledgerOf(executor).list(); + expect(executor.calls[0]?.params).toEqual([null, null, 100]); + }); +}); diff --git a/packages/jobs/src/driver-pg.ts b/packages/jobs/src/driver-pg.ts index 02cc6a2a6..509bd34aa 100644 --- a/packages/jobs/src/driver-pg.ts +++ b/packages/jobs/src/driver-pg.ts @@ -5,6 +5,7 @@ import type { Clock } from '@ultimat3/core'; import { systemClock, uuid } from '@ultimat3/core'; +import type { BackfillLedger, BackfillRun, BackfillStatus } from './backfill-ledger'; import { nowMs } from './clock'; import type { ClaimedJob, @@ -22,6 +23,10 @@ import { DEFAULT_QUEUE } from './driver'; import { SQL_ACK, SQL_ADVISORY_UNLOCK, + SQL_BACKFILL_FINISH, + SQL_BACKFILL_LIST, + SQL_BACKFILL_PROGRESS, + SQL_BACKFILL_START, SQL_CLAIM, SQL_ENQUEUE, SQL_FIND_LIVE_BY_KEY, @@ -59,6 +64,18 @@ interface StepRow { readonly error: string | null; } +interface BackfillRow { + readonly run_id: string; + readonly name: string; + readonly checksum: string; + readonly status: string; + readonly app_version: string; + readonly rows_processed: number | string; + readonly last_cursor: string | null; + readonly started_at: number | string; + readonly completed_at: number | string | null; +} + interface JobRow { readonly id: string; readonly name: string; @@ -171,6 +188,44 @@ function pgStepStore(exec: () => PgExecutor): StepStore { }; } +function toBackfillRun(row: BackfillRow): BackfillRun { + const completedAt = optionalNum(row.completed_at); + return { + runId: row.run_id, + name: row.name, + checksum: row.checksum, + status: row.status as BackfillStatus, + appVersion: row.app_version, + // `rows_processed` is a bigint, which every Postgres client hands back as a string. + rows: num(row.rows_processed), + cursor: row.last_cursor, + startedAt: num(row.started_at), + ...(completedAt === undefined ? {} : { completedAt }), + }; +} + +function pgBackfillLedger(exec: () => PgExecutor): BackfillLedger { + return { + async start(run) { + await exec().query(SQL_BACKFILL_START, [run.runId, run.name, run.checksum, run.appVersion]); + }, + async progress(runId, at) { + await exec().query(SQL_BACKFILL_PROGRESS, [runId, at.rows, at.cursor]); + }, + async finish(runId, at) { + await exec().query(SQL_BACKFILL_FINISH, [runId, at.status, at.rows]); + }, + async list(filter = {}) { + const rows = await exec().query(SQL_BACKFILL_LIST, [ + filter.name ?? null, + filter.status ?? null, + filter.limit ?? 100, + ]); + return rows.map(toBackfillRun); + }, + }; +} + export function createPgDriver(options: PgDriverOptions = {}): JobDriver { const clock = options.clock ?? systemClock; let executor: PgExecutor | undefined; @@ -234,6 +289,7 @@ export function createPgDriver(options: PgDriverOptions = {}): JobDriver { return { name: 'pg', steps: pgStepStore(exec), + backfills: pgBackfillLedger(exec), introspect, async enqueue(request: EnqueueRequest): Promise { diff --git a/packages/jobs/src/driver.ts b/packages/jobs/src/driver.ts index dd49dc21f..92d091a90 100644 --- a/packages/jobs/src/driver.ts +++ b/packages/jobs/src/driver.ts @@ -2,6 +2,7 @@ // switching backends is a config line and ZERO job-code change. Six methods and no more: // claim/ack/nack with a visibility timeout is the smallest set that survives a worker crash. +import type { BackfillLedger } from './backfill-ledger'; import type { StepStore } from './steps'; export type JobState = 'ready' | 'delayed' | 'running' | 'suspended' | 'done' | 'failed' | 'dead'; @@ -115,6 +116,13 @@ export interface JobDriver { /** Extends the lease of a long-running job so it is not double-claimed. */ heartbeat(jobId: string, options: { readonly visibilityTimeoutMs: number }): Promise; stats(): Promise; + /** + * Optional, like `introspect`: `x_backfills` records what a `backfill()` pass has already swept, + * and a driver without one runs backfills with no bookkeeping rather than refusing them. It + * hangs here for the same reason `steps` does — durable state that ships in the queue's own DDL, + * so one install point covers both. + */ + readonly backfills?: BackfillLedger; readonly introspect?: JobIntrospection; close?(): Promise; } diff --git a/packages/jobs/src/index.ts b/packages/jobs/src/index.ts index 211ad2383..0a385e5d4 100644 --- a/packages/jobs/src/index.ts +++ b/packages/jobs/src/index.ts @@ -17,6 +17,14 @@ export type { BackfillReport, } from './backfill'; export { backfill, DEFAULT_BACKFILL_BATCH } from './backfill'; +export type { + BackfillFilter, + BackfillLedger, + BackfillRun, + BackfillStatus, + BackfillVerdict, +} from './backfill-ledger'; +export { backfillChecksum, createMemoryBackfillLedger, decideBackfill } from './backfill-ledger'; export type { JobDescriptor } from './describe'; export type { ClaimedJob, @@ -48,6 +56,10 @@ export { createPgDriver, createPgLeader } from './driver-pg'; export { SQL_ACK, SQL_ADVISORY_UNLOCK, + SQL_BACKFILL_FINISH, + SQL_BACKFILL_LIST, + SQL_BACKFILL_PROGRESS, + SQL_BACKFILL_START, SQL_CLAIM, SQL_ENQUEUE, SQL_HEARTBEAT, From 838e6293a2e5976d48cdf609dae49521716dce7f Mon Sep 17 00:00:00 2001 From: sebi Date: Fri, 14 Aug 2026 14:06:53 -0500 Subject: [PATCH 3/5] feat(jobs): throttle a backfill, and make its progress readable MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit - `rate` (batches/sec, default 5) paces every pass. The wait is spent INSIDE each batch's `step.run`, so a resumed attempt replaying 500 checkpoints re-pays none of their pauses, and it unwinds on the run's cancellation (`X_ABORTED`) instead of sitting in a timer. No unthrottled mode: to sweep faster raise the number. Not part of the definition checksum — pacing is tuning, like `batch`. - `inspectBackfills()` is the ONE projection of the `x_backfills` ledger, read by four surfaces so none can disagree: `x db backfill --list` (`--name`, `--status`, `--limit`, `--json`), `x jobs ls` (the passes in flight, rows and cursor), `x jobs show ` (this run's row, under `backfill`), and `/_x`'s jobs panel (the whole ledger plus a live count). It reads no clock and answers `[]` — never a throw — for a driver with no ledger. - `x db backfill` without `--list` is refused naming the command that works: `x db backfill ` will one day RUN a pass. - `withJobDriver` (`jobs-driver.ts`) is now the one place a CLI command gets hold of the queue; `x jobs` and `x db backfill` share it. - Fixes `SQL_BACKFILL_LIST`: the new run-id predicate needs `$3::uuid`, since `uuid = text` has no operator and would have failed every call, filtered or not. Caught against a real database, pinned in `driver-pg.test.ts`. Co-Authored-By: Claude --- CHANGELOG.md | 4 + packages/admin/src/dev/data.test.ts | 47 +++++ packages/admin/src/dev/data.ts | 27 +++ packages/admin/src/dev/facts.ts | 23 +++ packages/admin/src/dev/index.ts | 1 + packages/admin/src/dev/panel-jobs.test.ts | 77 ++++++++ packages/admin/src/dev/panel-jobs.ts | 15 +- packages/cli/CLAUDE.md | 16 +- packages/cli/src/cmd-db.test.ts | 97 +++++++++- packages/cli/src/cmd-db.ts | 66 ++++++- packages/cli/src/cmd-jobs.test.ts | 60 ++++++- packages/cli/src/cmd-jobs.ts | 45 ++--- packages/cli/src/db-backfill.test.ts | 149 ++++++++++++++++ packages/cli/src/db-backfill.ts | 76 ++++++++ packages/cli/src/jobs-driver.test.ts | 61 +++++++ packages/cli/src/jobs-driver.ts | 33 ++++ packages/cli/src/jobs-json.test.ts | 82 ++++++++- packages/cli/src/jobs-json.ts | 24 +++ packages/cli/src/jobs-report.test.ts | 23 +++ packages/cli/src/jobs-report.ts | 21 ++- packages/cli/src/messages.ts | 7 + packages/jobs/CLAUDE.md | 27 +++ packages/jobs/README.md | 25 +++ packages/jobs/src/backfill-inspect.ts | 76 ++++++++ packages/jobs/src/backfill-ledger.ts | 6 + packages/jobs/src/backfill-pass.test.ts | 9 +- packages/jobs/src/backfill-pass.ts | 15 +- packages/jobs/src/backfill-rate.test.ts | 174 ++++++++++++++++++ packages/jobs/src/backfill-rate.ts | 96 ++++++++++ packages/jobs/src/backfill-throttle.test.ts | 188 ++++++++++++++++++++ packages/jobs/src/backfill.test.ts | 33 ++++ packages/jobs/src/backfill.ts | 23 ++- packages/jobs/src/driver-pg-sql.ts | 9 +- packages/jobs/src/driver-pg.test.ts | 21 ++- packages/jobs/src/driver-pg.ts | 1 + packages/jobs/src/index.ts | 4 + packages/jobs/src/inspect.ts | 10 ++ wiki/CLI-Reference.md | 36 +++- wiki/Jobs-And-Workflows.md | 7 +- 39 files changed, 1653 insertions(+), 61 deletions(-) create mode 100644 packages/admin/src/dev/panel-jobs.test.ts create mode 100644 packages/cli/src/db-backfill.test.ts create mode 100644 packages/cli/src/db-backfill.ts create mode 100644 packages/cli/src/jobs-driver.test.ts create mode 100644 packages/cli/src/jobs-driver.ts create mode 100644 packages/jobs/src/backfill-inspect.ts create mode 100644 packages/jobs/src/backfill-rate.test.ts create mode 100644 packages/jobs/src/backfill-rate.ts create mode 100644 packages/jobs/src/backfill-throttle.test.ts diff --git a/CHANGELOG.md b/CHANGELOG.md index 54699fecb..d72fe0338 100644 --- a/CHANGELOG.md +++ b/CHANGELOG.md @@ -103,6 +103,10 @@ Semver applies from 1.0.0. A breaking change to a documented API needs a major The row is a **report**, never a resume source: where a resumed pass restarts is decided by the step checkpoints and by nothing else. A retry adopts its own row (`started_at` is when the pass began, not this attempt); an attempt that failed is recorded as `failed` with its cursor kept, because where a pass stopped is the first thing anyone asks about one. + **A backfill is throttled, and the default is slow.** `rate` is batches per second and defaults to `DEFAULT_BACKFILL_RATE` (5) — 5,000 rows/sec at the default batch, one statement every 200ms, so the pool spends the rest of each interval serving the requests the app is still taking. There is **no unthrottled mode**: to sweep faster you raise the number, and a rate above what the batches can achieve simply produces no wait. The pause is spent **inside** each batch's `step.run`, which is what stops a resumed attempt from re-paying the throttle of the five hundred batches it is replaying, and it unwinds on the run's cancellation (`X_ABORTED`) rather than sitting in a timer nobody is waiting for. `rate` is not part of the definition checksum, for the reason `batch` is not: pacing is tuning, and changing it does not make a completed sweep a different sweep. + + **Progress is readable from three surfaces, all reading the one ledger.** `x db backfill --list` prints it (`--name`, `--status`, `--limit`, and `--json` carrying the same rows); `x jobs ls` reports the passes **in flight** with rows-so-far and cursor, and `x jobs show ` carries the ledger row for that run under `backfill`; `/_x`'s jobs panel carries the whole ledger plus a live count. One projection behind all of them — `inspectBackfills()` in `@ultimat3/jobs` — so the dashboard renders what `--json` prints. `x db backfill` without `--list` is refused with the command that works, because `x db backfill ` will one day *run* a pass and a bare invocation that quietly listed would be a silent no-op the day it lands. + - **`appVersion()` in `@ultimat3/core`.** One reader for `APP_VERSION`, defaulting to `dev`. It was spelled inside `@ultimat3/db`'s `runningAppVersion`, which now delegates — `x_migrations.app_version` and `x_backfills.app_version` are two durable columns an operator reads side by side, `@ultimat3/jobs` cannot reach `db` for the answer, and two packages defaulting the key their own way would put two names on one build. - **The destructive-SQL rail: a migration that destroys data must say so, and `x verify` refuses one that does not.** `x db gen` now writes `-- destructive: true` into any migration whose `up` drops a table, drops a column, truncates or retypes; `x verify`'s `drift` step reads the committed files back and fails an unmarked one with the new `X_MIGRATION_DESTRUCTIVE`. The strong-migrations idea, enforced rather than documented — a drop is allowed, an *undeclared* drop is not. diff --git a/packages/admin/src/dev/data.test.ts b/packages/admin/src/dev/data.test.ts index 11ba89e66..24c893f13 100644 --- a/packages/admin/src/dev/data.test.ts +++ b/packages/admin/src/dev/data.test.ts @@ -162,6 +162,52 @@ describe('hooks', () => { }); }); +describe('the backfill ledger source', () => { + test('the WHOLE ledger comes back, completed passes included', async () => { + const { createMemoryDriver, resetJobDriver, setJobDriver } = await import('@ultimat3/jobs'); + const driver = createMemoryDriver(); + setJobDriver(driver); + try { + const ledger = driver.backfills; + if (ledger === undefined) throw new Error('the memory driver must ship a backfill ledger'); + await ledger.start({ runId: 'r1', name: 'sweep', checksum: 'aa', appVersion: '1.2.0' }); + await ledger.progress('r1', { rows: 40, cursor: 'p40' }); + await ledger.finish('r1', { status: 'completed', rows: 40 }); + await ledger.start({ runId: 'r2', name: 'recount', checksum: 'bb', appVersion: '1.2.0' }); + await ledger.progress('r2', { rows: 7, cursor: 'p7' }); + + const facts = await defaultDevSources().backfills(); + + // Filtered to `running`, this panel would report a finished sweep as one that never + // happened — which is the question `x jobs ls` deliberately does NOT answer and this one + // does. `completed` clears the cursor; the row count is what survives. + expect(facts.map((fact) => [fact.runId, fact.status, fact.rows])).toEqual([ + ['r2', 'running', 7], + ['r1', 'completed', 40], + ]); + expect(facts[1]?.durationMs).not.toBeNull(); + expect(facts[0]?.durationMs).toBeNull(); + expect(facts[0]?.cursor).toBe('p7'); + } finally { + resetJobDriver(); + await driver.close(); + } + }); + + test('no queue at all refuses, naming the source and the panel', async () => { + // The same line `queues` and `jobRuns` draw: a process with no driver cannot answer "what has + // been swept", and `[]` there would claim nothing ever has. A driver that merely ships no + // ledger is the other case — `inspectBackfills` answers `[]` for it, in `@ultimat3/jobs`. + const caught = (await defaultDevSources() + .backfills() + .catch((error: unknown) => error)) as { cause: string; fix: string }; + + expect(caught).toBeUltimateError('X_NOT_IMPLEMENTED'); + expect(caught.cause).toContain('backfills'); + expect(caught.cause).toContain('jobs'); + }); +}); + describe('staticDevSources', () => { test('still answers empty for everything — the explicit-fixture path must not throw', async () => { const sources = staticDevSources(); @@ -174,6 +220,7 @@ describe('staticDevSources', () => { await expect(sources.jobDefs()).resolves.toEqual([]); await expect(sources.queues()).resolves.toEqual([]); await expect(sources.jobRuns()).resolves.toEqual([]); + await expect(sources.backfills()).resolves.toEqual([]); await expect(sources.tasks()).resolves.toEqual([]); await expect(sources.tables()).resolves.toEqual([]); await expect(sources.drift()).resolves.toEqual([]); diff --git a/packages/admin/src/dev/data.ts b/packages/admin/src/dev/data.ts index 837429a5d..a41005b2a 100644 --- a/packages/admin/src/dev/data.ts +++ b/packages/admin/src/dev/data.ts @@ -11,6 +11,7 @@ import type { JobState, StepStatus } from '@ultimat3/jobs'; import type { AdminActor, AdminAuthz } from '../authz'; import { DevSourceUnavailableError } from '../errors'; import type { + BackfillFact, CacheEdgeFact, DevSources, DriftFact, @@ -46,6 +47,7 @@ export function staticDevSources(facts: Partial = {}): DevSources { jobDefs: facts.jobDefs ?? ((): Promise => empty([])), queues: facts.queues ?? ((): Promise => empty([])), jobRuns: facts.jobRuns ?? ((): Promise => empty([])), + backfills: facts.backfills ?? ((): Promise => empty([])), tasks: facts.tasks ?? ((): Promise => empty([])), tables: facts.tables ?? ((): Promise => empty([])), drift: facts.drift ?? ((): Promise => empty([])), @@ -253,6 +255,31 @@ export function defaultDevSources(opts: DevSourceOptions = {}): DevSources { })); }, + /** + * The whole ledger, newest first — not just the passes in flight. `x jobs ls` reports the live + * queue and says so; a panel is read to answer "has this backfill ever run here, and what did + * it sweep", and the completed rows ARE that answer. + * + * `inspectBackfills` returns `[]` for a driver that ships no ledger, so only a process with no + * queue at all refuses here — which is the same line `queues` and `jobRuns` draw. + */ + async backfills(): Promise { + const { inspectBackfills, jobDriver } = await import('@ultimat3/jobs'); + const driver = jobDriver(); + if (driver === undefined) throw unavailable('backfills', 'jobs'); + return (await inspectBackfills(driver)).map((pass) => ({ + runId: pass.runId, + name: pass.name, + status: pass.status, + rows: pass.rows, + cursor: pass.cursor, + startedAt: pass.startedAt, + completedAt: pass.completedAt, + durationMs: pass.durationMs, + appVersion: pass.appVersion, + })); + }, + async tasks(): Promise { const { inspectManifest } = await import('@ultimat3/jobs'); return inspectManifest().tasks.map((task) => ({ diff --git a/packages/admin/src/dev/facts.ts b/packages/admin/src/dev/facts.ts index f6f7ee06f..edd1aca13 100644 --- a/packages/admin/src/dev/facts.ts +++ b/packages/admin/src/dev/facts.ts @@ -120,6 +120,28 @@ export interface TaskFact { readonly nextRunAt: string | null; } +/** + * One `x_backfills` pass. The queue's own facts answer "is this job moving"; only the ledger + * answers "how much of the table is behind it", which is the one question a sweep is watched for. + * `cursor` is where the pass had got to and `null` before its first batch — never a percentage: + * nothing counted the rows ahead of it, and a made-up denominator is the fact this panel exists + * not to invent. + */ +export interface BackfillFact { + /** The pass, and the run id of the job sweeping it — the join back to `runs` on this panel. */ + readonly runId: string; + readonly name: string; + readonly status: 'running' | 'completed' | 'failed'; + readonly rows: number; + readonly cursor: string | null; + readonly startedAt: string; + readonly completedAt: string | null; + /** How long the pass took. `null` while it is still running. */ + readonly durationMs: number | null; + /** The build that STARTED the pass — a redeploy mid-sweep does not rewrite it. */ + readonly appVersion: string; +} + export interface ColumnFact { readonly name: string; readonly type: string; @@ -192,6 +214,7 @@ export interface DevSources { jobDefs(): Promise; queues(): Promise; jobRuns(): Promise; + backfills(): Promise; tasks(): Promise; tables(): Promise; drift(): Promise; diff --git a/packages/admin/src/dev/index.ts b/packages/admin/src/dev/index.ts index 35f688481..cabce535b 100644 --- a/packages/admin/src/dev/index.ts +++ b/packages/admin/src/dev/index.ts @@ -5,6 +5,7 @@ export { type DevSourceOptions, defaultDevSources, staticDevSources } from './data'; export type { + BackfillFact, CacheEdgeFact, ColumnFact, DevSources, diff --git a/packages/admin/src/dev/panel-jobs.test.ts b/packages/admin/src/dev/panel-jobs.test.ts new file mode 100644 index 000000000..0600f1c50 --- /dev/null +++ b/packages/admin/src/dev/panel-jobs.test.ts @@ -0,0 +1,77 @@ +import { describe, expect, test } from 'bun:test'; +import { staticDevSources } from './data'; +import type { BackfillFact, JobRunFact } from './facts'; +import { jobsPanel } from './panel-jobs'; + +const pass = (fact: Partial & { runId: string }): BackfillFact => ({ + name: 'recount-likes', + status: 'running', + rows: 0, + cursor: null, + startedAt: '2026-08-14T09:00:00.000Z', + completedAt: null, + durationMs: null, + appVersion: '1.2.0', + ...fact, +}); + +const run = (fact: Partial & { id: string }): JobRunFact => ({ + job: 'recount-likes', + queue: 'default', + status: 'running', + attempt: 1, + steps: [], + ...fact, +}); + +describe('the jobs panel reports the backfill ledger', () => { + test('every pass is carried, and only the live ones are counted in flight', async () => { + const data = await jobsPanel.data( + staticDevSources({ + backfills: () => + Promise.resolve([ + pass({ runId: 'r3', rows: 12_500, cursor: 'post_12500' }), + pass({ runId: 'r2', status: 'failed', rows: 400, cursor: 'post_400' }), + pass({ + runId: 'r1', + status: 'completed', + rows: 84_000, + completedAt: '2026-08-14T09:04:00.000Z', + durationMs: 240_000, + }), + ]), + }), + new URLSearchParams(), + ); + + // The whole ledger, because "has this ever run here, and what did it sweep" is the panel's + // question — a filter to `running` would answer it with silence on a finished sweep. + expect(data.backfills.map((entry) => entry.runId)).toEqual(['r3', 'r2', 'r1']); + // A failed pass is an attempt the queue will retry, not a sweep in flight. + expect(data.backfillsInFlight).toBe(1); + expect(data.backfills[0]?.rows).toBe(12_500); + expect(data.backfills[0]?.cursor).toBe('post_12500'); + }); + + test('a ?queue= filter scopes the runs and never the ledger', async () => { + const sources = staticDevSources({ + jobRuns: () => + Promise.resolve([run({ id: 'r3' }), run({ id: 'other', queue: 'mail', job: 'send' })]), + backfills: () => Promise.resolve([pass({ runId: 'r3' })]), + }); + + const data = await jobsPanel.data(sources, new URLSearchParams('queue=mail')); + + // A backfill belongs to a table, not to a queue: scoping the ledger by the queue filter would + // hide the one pass whose job row the same filter just dropped. + expect(data.runs.map((entry) => entry.id)).toEqual(['other']); + expect(data.backfills.map((entry) => entry.runId)).toEqual(['r3']); + expect(data.backfillsInFlight).toBe(1); + }); + + test('an app that has never swept anything reports an empty ledger, not a failure', async () => { + const data = await jobsPanel.data(staticDevSources(), new URLSearchParams()); + expect(data.backfills).toEqual([]); + expect(data.backfillsInFlight).toBe(0); + }); +}); diff --git a/packages/admin/src/dev/panel-jobs.ts b/packages/admin/src/dev/panel-jobs.ts index 410703e0f..ae2a86f6b 100644 --- a/packages/admin/src/dev/panel-jobs.ts +++ b/packages/admin/src/dev/panel-jobs.ts @@ -2,7 +2,7 @@ // Kills: "is the queue moving, and which step failed?" — queue depth, per-run step traces, // the retry-from-step target, and the dead letter. -import type { JobDefFact, JobRunFact, QueueFact, TaskFact } from './facts'; +import type { BackfillFact, JobDefFact, JobRunFact, QueueFact, TaskFact } from './facts'; import type { DevPanel } from './panel'; export interface RetryTarget { @@ -22,6 +22,14 @@ export interface JobsPanelData { readonly deadLetter: readonly JobRunFact[]; readonly retryTargets: readonly RetryTarget[]; readonly totalDepth: number; + /** Every pass the `x_backfills` ledger holds, newest first — see `backfillsInFlight`. */ + readonly backfills: readonly BackfillFact[]; + /** + * The sweeps still going, split out rather than left for the reader to filter: a `?queue=` + * filter scopes the runs on this panel and a backfill is not a queue, so the one number that + * would otherwise be missed is how many passes are moving right now. + */ + readonly backfillsInFlight: number; } const firstFailedStep = (run: JobRunFact): RetryTarget | null => { @@ -41,11 +49,12 @@ export const jobsPanel: DevPanel = { titleKey: 'dev.panel.jobs.title', questionKey: 'dev.panel.jobs.question', async data(sources, params): Promise { - const [queues, jobs, tasks, runs] = await Promise.all([ + const [queues, jobs, tasks, runs, backfills] = await Promise.all([ sources.queues(), sources.jobDefs(), sources.tasks(), sources.jobRuns(), + sources.backfills(), ]); const queueFilter = params.get('queue'); const scoped = queueFilter === null ? runs : runs.filter((run) => run.queue === queueFilter); @@ -61,6 +70,8 @@ export const jobsPanel: DevPanel = { .map(firstFailedStep) .filter((target): target is RetryTarget => target !== null), totalDepth: queues.reduce((sum, queue) => sum + queue.depth, 0), + backfills, + backfillsInFlight: backfills.filter((pass) => pass.status === 'running').length, }; }, }; diff --git a/packages/cli/CLAUDE.md b/packages/cli/CLAUDE.md index cc0d88159..a9736854c 100644 --- a/packages/cli/CLAUDE.md +++ b/packages/cli/CLAUDE.md @@ -58,7 +58,7 @@ them is answered by this table rather than by a second convention: | Command | Files | Reads | |---|---|---| | `x actions` / `x queries` / `x entities` | `cmd-registries.ts` | the three declaration registries | -| `x jobs` | `cmd-jobs.ts`, `jobs-{report,drain,json,table}.ts` | `@ultimat3/jobs`' own introspection | +| `x jobs` | `cmd-jobs.ts`, `jobs-{driver,report,drain,json,table}.ts` | `@ultimat3/jobs`' own introspection | | `x tasks` | `cmd-tasks.ts`, `tasks-facts.ts` | `registeredTasks()` + `@ultimat3/time`'s cron resolution | | `x policy` | `cmd-policy.ts`, `policy-facts.ts` | `@ultimat3/policy`'s `policyMatrix()` over the app's own `Policy` objects | | `x i18n` | `cmd-i18n.ts`, `i18n-audit.ts` | `@ultimat3/i18n`'s `extractFromFiles` + `auditCatalogs` | @@ -196,6 +196,20 @@ regex and `+` is a quantifier — `n1` is what actually selects these tests. | `cmd-db.ts` | the subcommands, and nothing else — `gen` calls `db-generate.ts`, `migrate`/`reset` call `serve.ts`'s `runMigrations` | | `drift.ts` | `checkSourceDrift`: the `.hash` sidecar `x verify`'s `drift` step compares, no database needed | | `db-destructive.ts` | `checkDestructiveMigrations`: the same step's second half — every committed `up` that drops, truncates or retypes must carry `-- destructive: true` | +| `db-backfill.ts` | `x db backfill --list`: the flag parsing, the ledger read and the table | + +`jobs-driver.ts` is the ONE place a CLI command gets hold of the app's queue — `withJobDriver`, +which `x jobs` and `x db backfill` both call. It reuses an ambient `jobDriver()` when a process +already installed one (inside `x dev` or `x mcp serve`, booting a second queue talks to the wrong +database) and otherwise boots `startQueue` and releases it in a `finally`, or a CLI that exits +holding the PGlite lock breaks the next command run against this app. A second copy of that boot +would be two answers to "which queue is this command talking to". + +`x db backfill` requires `--list` and refuses without it, in `X_CLI_BAD_FLAG` naming the exact +invocation. `x db backfill ` will one day RUN a pass; a bare command that quietly printed the +ledger would be a second spelling of one thing today and a silent no-op the day the runner lands. +`X_NOT_IMPLEMENTED` would be wrong for the opposite reason — that code is for a subcommand shipping +nothing, and this one ships the ledger. `x db migrate` and `ROLE=migrate` are the same function call. That is the whole design: until 1.2.0 the CLI shelled out to `bunx drizzle-kit` — a second engine, a second journal, declared in no diff --git a/packages/cli/src/cmd-db.test.ts b/packages/cli/src/cmd-db.test.ts index 100c57043..4d9776bed 100644 --- a/packages/cli/src/cmd-db.test.ts +++ b/packages/cli/src/cmd-db.test.ts @@ -3,14 +3,19 @@ // own function — and nothing anywhere shells out to a second migrator. Its post-condition is one // check too: the live schema against the ledger, which is what `runMigrations` returns. -import { describe, expect, test } from 'bun:test'; +import { afterEach, describe, expect, test } from 'bun:test'; // `node:fs`/`node:os` — Bun has no temp-directory API; `node:path` — no Bun path joiner. import { mkdtempSync, rmSync } from 'node:fs'; import { tmpdir } from 'node:os'; import { join } from 'node:path'; +import type { JobDriver } from '@ultimat3/jobs'; +import { createMemoryDriver, resetJobDriver, setJobDriver } from '@ultimat3/jobs'; import { branchDatabaseName, branchSql, DB_SUBCOMMANDS, dbCommand, driftFindings } from './cmd-db'; import type { CommandContext } from './command'; +import { BadFlagError } from './errors'; import { exec } from './exec'; +import { msg } from './messages'; +import type { CommandResult } from './output'; import { flagBool, parseArgs } from './parse'; import { SPECS } from './registry'; @@ -170,3 +175,93 @@ describe('unit · the post-migrate report renders the pinned contract output', ( expect(driftFindings({ ok: true, differences: [] })).toEqual([]); }); }); + +/** Install the driver `withJobDriver` must reuse, so no test here boots a queue or a database. */ +async function runBackfill(driver: JobDriver, argv: readonly string[]): Promise { + setJobDriver(driver); + const root = await appRoot(); + try { + return await dbCommand.run(ctxFor(argv, root)); + } finally { + rmSync(root, { recursive: true, force: true }); + } +} + +async function seedPass(driver: JobDriver, runId: string, name: string): Promise { + await driver.backfills?.start({ runId, name, checksum: 'abc123', appVersion: '1.2.0' }); + await driver.backfills?.progress(runId, { rows: 250, cursor: 'post_250' }); +} + +afterEach(() => { + resetJobDriver(); +}); + +describe('unit · x db backfill', () => { + test('the subcommand and its three filter flags are declared, so the parser reaches them', () => { + expect(DB_SUBCOMMANDS).toContain('backfill'); + const flags = dbCommand.spec.flags?.map((flag) => flag.name) ?? []; + expect(flags).toContain('list'); + expect(flags).toContain('status'); + expect(flags).toContain('limit'); + // `--name` already meant "migration or branch"; one declaration, widened, never a second. + expect(flags.filter((flag) => flag === 'name')).toHaveLength(1); + }); + + test('without --list it refuses and names the invocation that works', async () => { + const driver = createMemoryDriver(); + const thrown: unknown = await runBackfill(driver, ['db', 'backfill']).then( + () => undefined, + (error: unknown) => error, + ); + // Not X_NOT_IMPLEMENTED: this subcommand ships the ledger. Not a silent default-to-list + // either — running a pass is the shape `x db backfill ` is reserved for. + expect(thrown).toBeInstanceOf(BadFlagError); + expect((thrown as BadFlagError).fix).toBe('x db backfill --list'); + expect((thrown as BadFlagError).cause).toContain('x db backfill --list'); + }); + + test('--list prints the ledger as a table and carries the same rows in data', async () => { + const driver = createMemoryDriver(); + await seedPass(driver, 'run_1', 'reindex-posts'); + + const result = await runBackfill(driver, ['db', 'backfill', '--list']); + const rendered = (result.lines ?? []).join('\n'); + + expect(result.ok).toBe(true); + expect(result.summary).toBe(msg('cli.db.backfill.listed', { count: 1 })); + expect(rendered).toContain('started-at'); + expect(rendered).toContain('reindex-posts'); + expect(rendered).toContain('post_250'); + expect(rendered).not.toContain('⟦'); + expect(result.data).toMatchObject([ + { runId: 'run_1', name: 'reindex-posts', status: 'running', rows: 250, cursor: 'post_250' }, + ]); + }); + + test('an empty ledger is ok: nothing has swept this database yet is an answer', async () => { + const result = await runBackfill(createMemoryDriver(), ['db', 'backfill', '--list']); + + expect(result.ok).toBe(true); + expect(result.summary).toBe(msg('cli.db.backfill.empty')); + expect(result.lines).toEqual([]); + expect(result.data).toEqual([]); + }); + + test('--name and --status filter the ledger the command prints', async () => { + const driver = createMemoryDriver(); + await seedPass(driver, 'run_1', 'reindex-posts'); + await seedPass(driver, 'run_2', 'recount-likes'); + + const named = await runBackfill(driver, [ + 'db', + 'backfill', + '--list', + '--name', + 'recount-likes', + ]); + expect(named.data).toMatchObject([{ runId: 'run_2' }]); + + const done = await runBackfill(driver, ['db', 'backfill', '--list', '--status', 'completed']); + expect(done.data).toEqual([]); + }); +}); diff --git a/packages/cli/src/cmd-db.ts b/packages/cli/src/cmd-db.ts index 60002d486..832451afd 100644 --- a/packages/cli/src/cmd-db.ts +++ b/packages/cli/src/cmd-db.ts @@ -1,6 +1,6 @@ -// `x db gen|migrate|reset|studio|branch` — everything that touches the database, including the -// branch DB that makes destructive work safe. An agent that can clone the database in a second -// can migrate, seed and break things without a human deciding whether to let it. +// `x db gen|migrate|reset|studio|branch|backfill` — everything that touches the database, +// including the branch DB that makes destructive work safe. An agent that can clone the database +// in a second can migrate, seed and break things without a human deciding whether to let it. // // Every subcommand here runs `@ultimat3/db`'s own engine, which is the engine `ROLE=migrate` runs // (`serve.ts`): one `x_migrations` ledger, one checksum rule, one advisory lock, from a laptop to @@ -15,18 +15,21 @@ import { branchPglite, type DriftReport, driftError } from '@ultimat3/db'; import { requireAppRoot } from './app-root'; import { plannedSubcommand } from './cmd-planned'; import type { CliCommand, CommandContext } from './command'; +import { listBackfills, renderBackfillTable } from './db-backfill'; import { generateAppMigration } from './db-generate'; import { resolveServices } from './dev-services'; -import { CliNotImplementedError } from './errors'; +import { BadFlagError, CliNotImplementedError } from './errors'; import type { ExecResult } from './exec'; import { execOutput } from './exec'; +import { withJobDriver } from './jobs-driver'; +import { backfillToJson } from './jobs-json'; import { msg } from './messages'; import type { CommandResult, Finding } from './output'; import { findingFrom, isUltimateErrorShape } from './output'; import { flagBool, flagString } from './parse'; import { runMigrations } from './serve'; -export const DB_SUBCOMMANDS = ['gen', 'migrate', 'reset', 'studio', 'branch'] as const; +export const DB_SUBCOMMANDS = ['gen', 'migrate', 'reset', 'studio', 'branch', 'backfill'] as const; const failure = (result: ExecResult, code: string, fix: string): Finding => ({ code, @@ -121,12 +124,20 @@ async function runBranch( export const dbCommand: CliCommand = { spec: { name: 'db', - summary: 'gen, migrate, reset, studio, branch', - usage: 'x db gen "add publish_at" | migrate | reset | studio | branch ', + summary: 'gen, migrate, reset, studio, branch, backfill', + usage: + 'x db gen "add publish_at" | migrate | reset | studio | branch | backfill --list [--name n] [--status s] [--limit n]', requiresApp: true, subcommands: DB_SUBCOMMANDS, flags: [ - { name: 'name', type: 'string', summary: 'migration or branch name' }, + { name: 'name', type: 'string', summary: 'migration or branch name, or backfill to filter' }, + { name: 'list', type: 'boolean', summary: 'backfill: print the x_backfills ledger' }, + { + name: 'status', + type: 'string', + summary: 'backfill: filter by running, completed or failed', + }, + { name: 'limit', type: 'string', summary: 'backfill: max ledger rows to return' }, // Declared because `X_MIGRATION_IRREVERSIBLE`'s own fix line names it. A `fix:` is copied // and run verbatim, so a flag the parser refuses would make the error unfollowable. { @@ -145,6 +156,7 @@ export const dbCommand: CliCommand = { if (sub === 'migrate') return runMigrate(ctx, root, msg('cli.db.migrate.applied')); if (sub === 'reset') return runReset(ctx, root); if (sub === 'studio') throw plannedSubcommand('db', 'studio'); + if (sub === 'backfill') return runBackfill(ctx, root); return runBranch(ctx, root, argument ?? 'preview'); }, @@ -259,3 +271,41 @@ async function runReset(ctx: CommandContext, root: string): Promise` will one day RUN a pass, so a bare `x db backfill` that quietly + * printed the ledger would be a second spelling of one command today and a silent no-op the day + * the runner lands. Not `X_NOT_IMPLEMENTED` either — that is for a subcommand that ships nothing, + * and this one ships the ledger. The reason names the invocation that works, verbatim. + * + * An empty ledger is `ok: true`. "Nothing has swept this database yet" is an answer to the + * question asked, and a command that failed over it would be unrunnable on a fresh app. + */ +async function runBackfill(ctx: CommandContext, root: string): Promise { + if (!flagBool(ctx.args, 'list')) { + throw new BadFlagError({ + flag: 'list', + command: 'db', + reason: 'x db backfill reports the ledger and runs nothing yet: x db backfill --list', + fix: 'x db backfill --list', + }); + } + return withJobDriver(root, ctx, async (driver) => { + const rows = await listBackfills(driver, { + name: flagString(ctx.args, 'name'), + status: flagString(ctx.args, 'status'), + limit: flagString(ctx.args, 'limit'), + }); + return { + ok: true, + command: 'db', + summary: + rows.length === 0 + ? msg('cli.db.backfill.empty') + : msg('cli.db.backfill.listed', { count: rows.length }), + lines: rows.length === 0 ? [] : renderBackfillTable(rows).map((line) => ` ${line}`), + data: rows.map(backfillToJson), + }; + }); +} diff --git a/packages/cli/src/cmd-jobs.test.ts b/packages/cli/src/cmd-jobs.test.ts index f009d8bf2..4692e17aa 100644 --- a/packages/cli/src/cmd-jobs.test.ts +++ b/packages/cli/src/cmd-jobs.test.ts @@ -1,5 +1,5 @@ // The command surface of `x jobs`: the spec, the `--to` validation, and what `run()` actually -// renders. Driven through an ambient `createMemoryDriver()` so `withDriver` reuses it instead of +// renders. Driven through an ambient `createMemoryDriver()` so `withJobDriver` reuses it instead of // booting a queue — a real driver, real claim/ack semantics, no database and no app to load. import { afterEach, describe, expect, test } from 'bun:test'; @@ -51,7 +51,7 @@ const contextFor = (root: string, options: RunOptions): CommandContext => ({ bunVersion: '1.3.0', }); -/** Install the driver `withDriver` must reuse, so no command under test boots a second queue. */ +/** Install the driver `withJobDriver` must reuse, so no command under test boots a second queue. */ function runJobs(driver: JobDriver, options: RunOptions = {}): Promise { setJobDriver(driver); return jobsCommand.run(contextFor(appRoot(), options)); @@ -119,6 +119,62 @@ describe('unit · x jobs ls rendering', () => { expect(rendered).not.toContain('⟦'); }); + test('a pass in flight is reported with how far it has got, and finished ones are not', async () => { + const driver = createMemoryDriver(); + await driver.backfills?.start({ + runId: 'run_live', + name: 'reindex-posts', + checksum: 'abc123', + appVersion: '1.2.0', + }); + await driver.backfills?.progress('run_live', { rows: 250, cursor: 'post_250' }); + await driver.backfills?.start({ + runId: 'run_old', + name: 'recount-likes', + checksum: 'abc123', + appVersion: '1.2.0', + }); + await driver.backfills?.finish('run_old', { status: 'completed', rows: 900 }); + + const result = await runJobs(driver, { subcommand: 'ls' }); + const rendered = (result.lines ?? []).join('\n'); + + expect(rendered).toContain(msg('cli.jobs.backfills', { count: 1 })); + expect(rendered).toContain( + msg('cli.jobs.backfillRow', { name: 'reindex-posts', rows: 250, cursor: 'post_250' }), + ); + expect(rendered).toContain('run_live'); + // `x jobs ls` is the LIVE queue — a pass that finished is `x db backfill --list`'s answer. + expect(rendered).not.toContain('recount-likes'); + expect(rendered).not.toContain('⟦'); + expect(result.data).toMatchObject({ backfills: [{ runId: 'run_live', status: 'running' }] }); + }); + + test('a pass that has not reached its first batch says so instead of printing null', async () => { + const driver = createMemoryDriver(); + await driver.backfills?.start({ + runId: 'run_new', + name: 'reindex-posts', + checksum: 'abc123', + appVersion: '1.2.0', + }); + + const rendered = ((await runJobs(driver, { subcommand: 'ls' })).lines ?? []).join('\n'); + + expect(rendered).toContain(msg('cli.jobs.backfillNoCursor')); + expect(rendered).not.toContain('null'); + }); + + test('no backfill in flight renders no section at all', async () => { + const driver = createMemoryDriver(); + await enqueue(driver, 'send-email'); + + const result = await runJobs(driver, { subcommand: 'ls' }); + + expect((result.lines ?? []).join('\n')).not.toContain(msg('cli.jobs.backfills', { count: 0 })); + expect(result.data).toMatchObject({ backfills: [] }); + }); + test('a bad --limit fails the command through X_CLI_BAD_FLAG', async () => { const driver = createMemoryDriver(); await expect(runJobs(driver, { subcommand: 'ls', flags: { limit: '0' } })).rejects.toThrow( diff --git a/packages/cli/src/cmd-jobs.ts b/packages/cli/src/cmd-jobs.ts index b2a196eb2..4e50ab986 100644 --- a/packages/cli/src/cmd-jobs.ts +++ b/packages/cli/src/cmd-jobs.ts @@ -1,17 +1,17 @@ // `x jobs ls|show|retry|drain` — introspect and recover the job queue, bound to `@ultimat3/jobs`'s // own introspection so the CLI, `/_x` and MCP report identically. This file is CLI wiring only: // the driver-injected logic is `jobs-report.ts`, the `--json` shapes `jobs-json.ts`, the table -// `jobs-table.ts`. +// `jobs-table.ts`, and getting hold of the queue at all is `jobs-driver.ts` — shared with `x db`. import type { JobDriver } from '@ultimat3/jobs'; -import { createMemoryDriver, createNatsDriver, createRedisDriver, jobDriver } from '@ultimat3/jobs'; +import { createMemoryDriver, createNatsDriver, createRedisDriver } from '@ultimat3/jobs'; import { requireAppRoot } from './app-root'; import type { CliCommand, CommandContext } from './command'; -import { startQueue } from './dev-queue'; -import { resolveServices } from './dev-services'; import { BadFlagError } from './errors'; import { drainJobs } from './jobs-drain'; +import { withJobDriver } from './jobs-driver'; import { + backfillToJson, deadLetterToJson, depthToJson, drainFailureToJson, @@ -29,29 +29,6 @@ export const JOBS_SUBCOMMANDS = ['ls', 'show', 'retry', 'drain'] as const; const DRAIN_TARGETS = ['memory', 'redis', 'nats'] as const; -/** - * `x jobs` needs the app's real driver. Reuse an already-running one first — inside `x dev` or - * `x mcp serve`, `jobDriver()` is already set and booting a second queue on top of it would talk - * to the wrong database. Otherwise boot just the db + jobs half (`startQueue`, not the full - * `startServices`: this command touches no transport, storage or mail) and always release it, or - * a CLI that exits holding the PGlite lock breaks the next command run against this app. - */ -async function withDriver( - root: string, - ctx: CommandContext, - fn: (driver: JobDriver) => Promise, -): Promise { - const ambient = jobDriver(); - if (ambient !== undefined) return fn(ambient); - const services = resolveServices(root, ctx.env); - const queue = await startQueue(services); - try { - return await fn(queue.jobs); - } finally { - await queue.stop(); - } -} - function requireIdPositional(ctx: CommandContext, sub: string): string { const id = ctx.args.positionals[0]; if (id === undefined) { @@ -114,6 +91,17 @@ async function runLs(driver: JobDriver, ctx: CommandContext): Promise 0) { + lines.push(` ${msg('cli.jobs.backfills', { count: result.backfills.length })}`); + for (const pass of result.backfills) { + const cursor = pass.cursor ?? msg('cli.jobs.backfillNoCursor'); + const progress = msg('cli.jobs.backfillRow', { name: pass.name, rows: pass.rows, cursor }); + lines.push(` ${pass.runId} ${progress}`); + } + } return { ok: true, command: 'jobs', @@ -129,6 +117,7 @@ async function runLs(driver: JobDriver, ctx: CommandContext): Promise { const root = requireAppRoot('jobs', ctx.cwd).dir; const sub = ctx.args.subcommand ?? 'ls'; - return withDriver(root, ctx, (driver) => { + return withJobDriver(root, ctx, (driver) => { if (sub === 'show') return runShow(driver, ctx); if (sub === 'retry') return runRetry(driver, ctx); if (sub === 'drain') return runDrain(driver, ctx); diff --git a/packages/cli/src/db-backfill.test.ts b/packages/cli/src/db-backfill.test.ts new file mode 100644 index 000000000..171dca7c9 --- /dev/null +++ b/packages/cli/src/db-backfill.test.ts @@ -0,0 +1,149 @@ +// The ledger projection behind `x db backfill --list`, driven through `createMemoryDriver()` — +// a real `BackfillLedger` with real start/progress/finish semantics, so a row here got its +// `cursor` and `durationMs` the way a pg ledger would. No `ParsedArgs`, no app, no queue boot. + +import { describe, expect, test } from 'bun:test'; +import type { BackfillStatus, JobDriver } from '@ultimat3/jobs'; +import { createMemoryDriver } from '@ultimat3/jobs'; +import { + BACKFILL_STATUSES, + listBackfills, + parseBackfillStatusFlag, + renderBackfillTable, +} from './db-backfill'; +import { BadFlagError } from './errors'; +import { msg } from './messages'; + +interface Seed { + readonly runId: string; + readonly name: string; + readonly rows?: number; + readonly cursor?: string | null; + readonly finish?: 'completed' | 'failed'; +} + +/** Optional chaining, not an assertion: `backfills` is optional on `JobDriver` by design. */ +async function seed(driver: JobDriver, input: Seed): Promise { + await driver.backfills?.start({ + runId: input.runId, + name: input.name, + checksum: 'abc123', + appVersion: '1.2.0', + }); + if (input.rows !== undefined) { + await driver.backfills?.progress(input.runId, { + rows: input.rows, + cursor: input.cursor ?? null, + }); + } + if (input.finish !== undefined) { + await driver.backfills?.finish(input.runId, { status: input.finish, rows: input.rows ?? 0 }); + } +} + +describe('unit · backfill filter parsing', () => { + test('every status the ledger can record is accepted, and nothing else is', () => { + expect(BACKFILL_STATUSES).toEqual(['running', 'completed', 'failed']); + for (const status of BACKFILL_STATUSES) { + expect(parseBackfillStatusFlag(status)).toBe(status); + } + expect(parseBackfillStatusFlag(undefined)).toBeUndefined(); + }); + + test('an unknown --status names the three that exist', () => { + const thrown: unknown = (() => { + try { + return parseBackfillStatusFlag('done'); + } catch (error: unknown) { + return error; + } + })(); + expect(thrown).toBeInstanceOf(BadFlagError); + expect((thrown as BadFlagError).cause).toContain('running, completed, failed'); + // `x db`, not `x jobs`: the flag was typed on this command, so the error names this command. + expect((thrown as BadFlagError).cause).toContain('"x db"'); + }); + + test('a bad --limit is refused as a db flag, not as a jobs one', async () => { + const driver = createMemoryDriver(); + const thrown: unknown = await listBackfills(driver, { limit: '0' }).then( + () => undefined, + (error: unknown) => error, + ); + expect(thrown).toBeInstanceOf(BadFlagError); + expect((thrown as BadFlagError).cause).toContain('--limit on "x db"'); + }); +}); + +describe('unit · listBackfills', () => { + test('an empty ledger is an answer, not a failure', async () => { + expect(await listBackfills(createMemoryDriver())).toEqual([]); + }); + + test('every pass, newest first, with progress and duration as the ledger recorded them', async () => { + const driver = createMemoryDriver(); + await seed(driver, { runId: 'run_1', name: 'recount-likes', rows: 500, finish: 'completed' }); + await seed(driver, { runId: 'run_2', name: 'reindex-posts', rows: 120, cursor: 'post_120' }); + + const rows = await listBackfills(driver); + + expect(rows.map((row) => row.runId)).toEqual(['run_2', 'run_1']); + expect(rows[0]).toMatchObject({ + name: 'reindex-posts', + status: 'running', + rows: 120, + cursor: 'post_120', + completedAt: null, + durationMs: null, // a running pass has no span, and nothing here reads a clock + }); + expect(rows[1]).toMatchObject({ status: 'completed', rows: 500, cursor: null }); + expect(typeof rows[1]?.durationMs).toBe('number'); + }); + + test('--name and --status narrow the ledger, --limit caps it', async () => { + const driver = createMemoryDriver(); + await seed(driver, { runId: 'run_1', name: 'recount-likes', rows: 10, finish: 'completed' }); + await seed(driver, { runId: 'run_2', name: 'recount-likes', rows: 3 }); + await seed(driver, { runId: 'run_3', name: 'reindex-posts', rows: 7, finish: 'failed' }); + + expect((await listBackfills(driver, { name: 'recount-likes' })).map((r) => r.runId)).toEqual([ + 'run_2', + 'run_1', + ]); + const failed: BackfillStatus = 'failed'; + expect((await listBackfills(driver, { status: failed })).map((r) => r.runId)).toEqual([ + 'run_3', + ]); + expect(await listBackfills(driver, { limit: '1' })).toHaveLength(1); + }); +}); + +describe('unit · renderBackfillTable', () => { + test('the header, one padded row per pass, and the ISO start printed verbatim', async () => { + const driver = createMemoryDriver(); + await seed(driver, { runId: 'run_2', name: 'reindex-posts', rows: 120, cursor: 'post_120' }); + const rows = await listBackfills(driver); + + const lines = renderBackfillTable(rows); + + expect(lines[0]).toContain('name'); + expect(lines[0]).toContain('started-at'); + expect(lines[0]).toContain('run-id'); + expect(lines[1]).toContain('reindex-posts'); + expect(lines[1]).toContain('post_120'); + // Printed as the ledger stored it — formatting a date needs a zone this command has none of. + expect(lines[1]).toContain(rows[0]?.startedAt ?? 'missing'); + expect(lines).toHaveLength(2); + }); + + test('a cursor that has not moved and a pass that has not finished render the empty cell', async () => { + const driver = createMemoryDriver(); + await seed(driver, { runId: 'run_9', name: 'recount-likes' }); + const lines = renderBackfillTable(await listBackfills(driver)); + const cells = (lines[1] ?? '').split(/\s{2,}/); + + expect(cells).toContain(msg('cli.db.backfill.none')); + expect(cells.filter((cell) => cell === msg('cli.db.backfill.none'))).toHaveLength(2); + expect(lines.join('\n')).not.toContain('⟦'); + }); +}); diff --git a/packages/cli/src/db-backfill.ts b/packages/cli/src/db-backfill.ts new file mode 100644 index 000000000..065b0aa9a --- /dev/null +++ b/packages/cli/src/db-backfill.ts @@ -0,0 +1,76 @@ +// The `x_backfills` ledger as `x db backfill --list` reports it: the filter flags, the driver +// read and the fixed-width table. A driver plus plain strings in, plain data out — the +// `cmd-jobs.ts` / `jobs-report.ts` split repeated, so every projection here is testable with no +// `ParsedArgs`, no app boot and no queue. `cmd-db.ts` is the CLI wiring and nothing else. + +import type { BackfillProgress, BackfillStatus, JobDriver } from '@ultimat3/jobs'; +import { inspectBackfills } from '@ultimat3/jobs'; +import { BadFlagError } from './errors'; +import { parseLimitFlag } from './jobs-report'; +import { msg } from './messages'; +import { renderTable } from './table'; + +/** Mirrors `BackfillStatus` from `@ultimat3/jobs`, which exports the type but no runtime list. */ +export const BACKFILL_STATUSES: readonly BackfillStatus[] = ['running', 'completed', 'failed']; + +const isBackfillStatus = (value: string): value is BackfillStatus => + (BACKFILL_STATUSES as readonly string[]).includes(value); + +export function parseBackfillStatusFlag(value: string | undefined): BackfillStatus | undefined { + if (value === undefined) return undefined; + if (isBackfillStatus(value)) return value; + throw new BadFlagError({ + flag: 'status', + command: 'db', + reason: `unknown status "${value}" (known: ${BACKFILL_STATUSES.join(', ')})`, + fix: 'x db backfill --list --json', + }); +} + +export interface BackfillListFilter { + readonly name?: string | undefined; + readonly status?: string | undefined; + readonly limit?: string | undefined; +} + +/** + * Every pass the ledger holds, newest first, filtered by the flags as typed. The empty list is an + * ANSWER — a driver with no ledger and an app that has never swept anything are both "nothing has + * run", and `inspectBackfills` already refuses to throw for either. + */ +export async function listBackfills( + driver: JobDriver, + filter: BackfillListFilter = {}, +): Promise { + const status = parseBackfillStatusFlag(filter.status); + const limit = parseLimitFlag(filter.limit, 'db'); + return inspectBackfills(driver, { + ...(filter.name === undefined ? {} : { name: filter.name }), + ...(status === undefined ? {} : { status }), + ...(limit === undefined ? {} : { limit }), + }); +} + +const HEADER = ['name', 'status', 'rows', 'cursor', 'started-at', 'duration-ms', 'run-id'] as const; + +/** + * `started-at` is the ledger's own ISO string, printed verbatim and never re-formatted: the repo + * forbids a date rendered without an explicit IANA `timeZone`, and not formatting at all is the + * one rendering with no zone to get wrong — the same rule `jobs-table.ts` states for `run-at-ms`. + * `--json` carries these exact values, so the two renders of one command stay comparable. + */ +export function renderBackfillTable(rows: readonly BackfillProgress[]): readonly string[] { + const none = msg('cli.db.backfill.none'); + return renderTable( + HEADER, + rows.map((row) => [ + row.name, + row.status, + String(row.rows), + row.cursor ?? none, + row.startedAt, + row.durationMs === null ? none : String(row.durationMs), + row.runId, + ]), + ); +} diff --git a/packages/cli/src/jobs-driver.test.ts b/packages/cli/src/jobs-driver.test.ts new file mode 100644 index 000000000..eb3295271 --- /dev/null +++ b/packages/cli/src/jobs-driver.test.ts @@ -0,0 +1,61 @@ +// The seam `x jobs` and `x db backfill` both reach the queue through. Two rules, and a CLI that +// breaks either is one that talks to the wrong database or leaves the next command locked out. + +import { describe, expect, test } from 'bun:test'; +import { join } from 'node:path'; +import type { JobDriver } from '@ultimat3/jobs'; +import { createMemoryDriver, resetJobDriver, setJobDriver } from '@ultimat3/jobs'; +import type { CommandContext } from './command'; +import { withJobDriver } from './jobs-driver'; + +const contextFor = (): CommandContext => ({ + args: { + command: 'jobs', + positionals: [], + flags: new Map(), + json: false, + help: false, + passthrough: [], + }, + cwd: '/nowhere', + runner: () => + Promise.resolve({ + command: ['true'], + code: 0, + ok: true, + stdout: '', + stderr: '', + durationMs: 0, + }), + env: {}, + bunVersion: '1.3.0', +}); + +describe('unit · withJobDriver', () => { + test('an ambient driver is reused, so no command boots a second queue over it', async () => { + const ambient: JobDriver = createMemoryDriver(); + setJobDriver(ambient); + let seen: JobDriver | undefined; + try { + // `/nowhere` is not an app root: reaching `resolveServices` at all would throw here, which + // is the point — inside `x dev` this command must never touch the boot path. + const result = await withJobDriver('/nowhere', contextFor(), (driver) => { + seen = driver; + return Promise.resolve({ ok: true, command: 'jobs', summary: 'ok' }); + }); + expect(result.ok).toBe(true); + } finally { + resetJobDriver(); + } + expect(seen).toBe(ambient); + }); + + test('a queue it booted itself is always released', async () => { + const source = await Bun.file(join(import.meta.dir, 'jobs-driver.ts')).text(); + // Structural, because the alternative needs a real PGlite boot to observe: the moment the + // release stops being a `finally`, a command that throws exits holding the data-directory + // lock and the next `x` run against this app fails for a reason nobody can see from here. + expect(source).toContain('finally'); + expect(source).toContain('await queue.stop()'); + }); +}); diff --git a/packages/cli/src/jobs-driver.ts b/packages/cli/src/jobs-driver.ts new file mode 100644 index 000000000..fae9f542a --- /dev/null +++ b/packages/cli/src/jobs-driver.ts @@ -0,0 +1,33 @@ +// The one place a CLI command gets hold of the app's job queue. `x jobs` and `x db backfill` +// both need a `JobDriver` and neither owns one — a second copy of this boot would be two answers +// to "which queue is this command talking to", which is the drift axiom 1 refuses. + +import type { JobDriver } from '@ultimat3/jobs'; +import { jobDriver } from '@ultimat3/jobs'; +import type { CommandContext } from './command'; +import { startQueue } from './dev-queue'; +import { resolveServices } from './dev-services'; +import type { CommandResult } from './output'; + +/** + * `x jobs` needs the app's real driver. Reuse an already-running one first — inside `x dev` or + * `x mcp serve`, `jobDriver()` is already set and booting a second queue on top of it would talk + * to the wrong database. Otherwise boot just the db + jobs half (`startQueue`, not the full + * `startServices`: these commands touch no transport, storage or mail) and always release it, or + * a CLI that exits holding the PGlite lock breaks the next command run against this app. + */ +export async function withJobDriver( + root: string, + ctx: CommandContext, + fn: (driver: JobDriver) => Promise, +): Promise { + const ambient = jobDriver(); + if (ambient !== undefined) return fn(ambient); + const services = resolveServices(root, ctx.env); + const queue = await startQueue(services); + try { + return await fn(queue.jobs); + } finally { + await queue.stop(); + } +} diff --git a/packages/cli/src/jobs-json.test.ts b/packages/cli/src/jobs-json.test.ts index 86e7b3689..60ca0dfaa 100644 --- a/packages/cli/src/jobs-json.test.ts +++ b/packages/cli/src/jobs-json.test.ts @@ -4,8 +4,15 @@ import { describe, expect, test } from 'bun:test'; import type { JobRecord } from '@ultimat3/jobs'; -import { createMemoryDriver, inspectDeadLetters, inspectJob, inspectQueues } from '@ultimat3/jobs'; import { + createMemoryDriver, + inspectBackfills, + inspectDeadLetters, + inspectJob, + inspectQueues, +} from '@ultimat3/jobs'; +import { + backfillToJson, deadLetterToJson, depthToJson, drainFailureToJson, @@ -102,6 +109,79 @@ describe('unit · queue projections', () => { }); expect((await listJobs(driver)).rows).toHaveLength(1); }); + + test('a job that is not a backfill carries an explicit null, never a missing key', async () => { + const driver = createMemoryDriver(); + const { id } = await driver.enqueue({ + name: 'send-email', + queue: 'default', + input: {}, + idempotencyKey: 'k3', + maxAttempts: 3, + }); + const trace = await inspectJob(driver, id); + const json = jobTraceToJson(trace as NonNullable); + expect(json).toEqual(roundTrip(json) as never); + expect(Object.keys(json as Record)).toContain('backfill'); + expect(json).toMatchObject({ backfill: null }); + }); + + test("a backfill run's progress rides on its own trace, so --json never drops it", async () => { + const driver = createMemoryDriver(); + const { id, runId } = await driver.enqueue({ + name: 'reindex-posts', + queue: 'default', + input: {}, + idempotencyKey: 'k4', + maxAttempts: 3, + }); + // The ledger is keyed by run id, which is how `x jobs show ` joins the two with no + // second lookup table — write the row against this job's run and it must come back here. + await driver.backfills?.start({ + runId, + name: 'reindex-posts', + checksum: 'abc123', + appVersion: '1.2.0', + }); + await driver.backfills?.progress(runId, { rows: 250, cursor: 'post_250' }); + + const trace = await inspectJob(driver, id); + const json = jobTraceToJson(trace as NonNullable); + + expect(json).toEqual(roundTrip(json) as never); + expect(json).toMatchObject({ + backfill: { runId, name: 'reindex-posts', rows: 250, cursor: 'post_250', status: 'running' }, + }); + }); +}); + +describe('unit · backfillToJson', () => { + test('every absent value is already null at the source, so no key vanishes', async () => { + const driver = createMemoryDriver(); + await driver.backfills?.start({ + runId: 'run_1', + name: 'reindex-posts', + checksum: 'abc123', + appVersion: '1.2.0', + }); + const [progress] = await inspectBackfills(driver); + const json = backfillToJson(progress as NonNullable); + + expect(json).toEqual(roundTrip(json) as never); + expect(json).toMatchObject({ + runId: 'run_1', + name: 'reindex-posts', + status: 'running', + checksum: 'abc123', + appVersion: '1.2.0', + rows: 0, + cursor: null, + completedAt: null, + durationMs: null, + }); + // ISO in, ISO out: the projection re-formats no date, because it has no zone to do it with. + expect(json).toMatchObject({ startedAt: progress?.startedAt ?? 'missing' }); + }); }); describe('unit · drain projections', () => { diff --git a/packages/cli/src/jobs-json.ts b/packages/cli/src/jobs-json.ts index 94b7f5a23..90950f405 100644 --- a/packages/cli/src/jobs-json.ts +++ b/packages/cli/src/jobs-json.ts @@ -3,6 +3,7 @@ // enforceable if the projections sit together where one missing `?? null` is visible. import type { + BackfillProgress, DeadLetterEntry, JobRecord, JobTrace, @@ -26,6 +27,26 @@ function stepTraceToJson(step: StepTrace): JsonValue { }; } +/** + * One `x_backfills` row. Every absent value is already `null` at the source (`toBackfillProgress` + * makes it so), which is what lets this be a straight field list — and it is spelled out rather + * than spread so a field added upstream arrives here as a decision, not as untyped passthrough. + */ +export function backfillToJson(progress: BackfillProgress): JsonValue { + return { + runId: progress.runId, + name: progress.name, + status: progress.status, + checksum: progress.checksum, + appVersion: progress.appVersion, + rows: progress.rows, + cursor: progress.cursor, + startedAt: progress.startedAt, + completedAt: progress.completedAt, + durationMs: progress.durationMs, + }; +} + export function jobTraceToJson(trace: JobTrace): JsonValue { return { id: trace.id, @@ -41,6 +62,9 @@ export function jobTraceToJson(trace: JobTrace): JsonValue { tenantId: trace.tenantId, steps: trace.steps.map(stepTraceToJson), retryDelaysMs: trace.retryDelaysMs.map((ms) => ms), + // `null` for every job that is not a `backfill()` pass. Dropped, `x jobs show --json` + // would answer "how far has it got" with silence for the one job kind that can say. + backfill: trace.backfill === null ? null : backfillToJson(trace.backfill), }; } diff --git a/packages/cli/src/jobs-report.test.ts b/packages/cli/src/jobs-report.test.ts index 2bf4d6a6b..b58602d89 100644 --- a/packages/cli/src/jobs-report.test.ts +++ b/packages/cli/src/jobs-report.test.ts @@ -137,6 +137,29 @@ describe('unit · listJobs', () => { await expect(listJobs(driver, { state: 'exploded' })).rejects.toThrow(BadFlagError); await expect(listJobs(driver, { limit: '3.5' })).rejects.toThrow(BadFlagError); }); + + test('only the backfills still sweeping come back — a finished one is history', async () => { + const driver = createMemoryDriver(); + for (const runId of ['run_live', 'run_old']) { + await driver.backfills?.start({ + runId, + name: `pass-${runId}`, + checksum: 'abc123', + appVersion: '1.2.0', + }); + } + await driver.backfills?.progress('run_live', { rows: 42, cursor: 'post_42' }); + await driver.backfills?.finish('run_old', { status: 'completed', rows: 900 }); + + const result = await listJobs(driver); + + expect(result.backfills.map((pass) => pass.runId)).toEqual(['run_live']); + expect(result.backfills[0]).toMatchObject({ rows: 42, cursor: 'post_42', status: 'running' }); + }); + + test('a queue with no ledger rows reports no backfills rather than failing', async () => { + expect((await listJobs(createMemoryDriver())).backfills).toEqual([]); + }); }); describe('unit · showJob and retryJob', () => { diff --git a/packages/cli/src/jobs-report.ts b/packages/cli/src/jobs-report.ts index 0711b54ee..eea6b3f83 100644 --- a/packages/cli/src/jobs-report.ts +++ b/packages/cli/src/jobs-report.ts @@ -3,6 +3,7 @@ // alone. Drain is `jobs-drain.ts`, the `--json` shapes `jobs-json.ts`, the table `jobs-table.ts`. import type { + BackfillProgress, DeadLetterEntry, JobDriver, JobFilter, @@ -12,6 +13,7 @@ import type { QueueDepthReport, } from '@ultimat3/jobs'; import { + inspectBackfills, inspectDeadLetters, inspectJob, inspectJobList, @@ -48,15 +50,18 @@ export function parseStateFlag(value: string | undefined): JobState | undefined * A digit string is not yet a limit: past `Number.MAX_SAFE_INTEGER` the parse silently lands on a * different integer, and `1e400`-shaped input yields `Infinity`. Either way the driver would be * handed a bound other than the one typed, so the safe-integer check is the flag's real contract. + * + * `command` exists because `x db backfill --list` takes the same `--limit` and must not report it + * as a flag on `x jobs` — one parser, and the error still names the command that was typed. */ -export function parseLimitFlag(value: string | undefined): number | undefined { +export function parseLimitFlag(value: string | undefined, command = 'jobs'): number | undefined { if (value === undefined) return undefined; const digits = value.trim(); const limit = /^\d+$/.test(digits) ? Number(digits) : Number.NaN; if (!Number.isSafeInteger(limit) || limit <= 0) { throw new BadFlagError({ flag: 'limit', - command: 'jobs', + command, reason: `expects an integer from 1 to ${Number.MAX_SAFE_INTEGER}, got "${value}"`, }); } @@ -76,12 +81,19 @@ export interface JobsListResult { readonly depth: QueueDepthReport; readonly rows: readonly JobRecord[]; readonly deadLetters: readonly DeadLetterEntry[]; + /** The sweeps still in flight — see below for why finished ones are not this command's answer. */ + readonly backfills: readonly BackfillProgress[]; } /** * The depth report AND the filtered rows, plus dead letters unconditionally: a dead job that a * `--state ready` filter (or the default 100-row cap) pushes out of view is the exact failure * mode this command exists to prevent. + * + * Backfills are read `running` only. `x jobs ls` is a LIVE view of the queue — a pass that + * finished last week is history, and `x db backfill --list` is where that question is asked and + * answered with the whole ledger. A driver with no ledger answers `[]` rather than throwing, so + * the queue view never fails over a fact nobody asked about. */ export async function listJobs( driver: JobDriver, @@ -95,12 +107,13 @@ export async function listJobs( ...(state === undefined ? {} : { state }), ...(limit === undefined ? {} : { limit }), }; - const [depth, rows, deadLetters] = await Promise.all([ + const [depth, rows, deadLetters, backfills] = await Promise.all([ inspectQueues(driver), inspectJobList(driver, jobFilter), inspectDeadLetters(driver), + inspectBackfills(driver, { status: 'running' }), ]); - return { depth, rows, deadLetters }; + return { depth, rows, deadLetters, backfills }; } // ── show ────────────────────────────────────────────────────────────────── diff --git a/packages/cli/src/messages.ts b/packages/cli/src/messages.ts index a5900262b..5174351b9 100644 --- a/packages/cli/src/messages.ts +++ b/packages/cli/src/messages.ts @@ -24,6 +24,10 @@ const CATALOG = { 'cli.cron.inMonths': 'in {months}', 'cli.cron.onDaysOfMonth': 'on day {days} of the month', 'cli.cron.onWeekdays': 'on {days}', + 'cli.db.backfill.empty': 'no backfill has run against this database yet', + 'cli.db.backfill.listed': '{count} backfill pass(es)', + /** The empty cell in a `x db backfill --list` column — a value, not a column key. */ + 'cli.db.backfill.none': '-', 'cli.db.branch.ready': 'branch {name} ready', 'cli.db.gen.failed': 'migration not generated', 'cli.db.gen.unchanged': 'entities and migrations agree — nothing to generate', @@ -58,6 +62,9 @@ const CATALOG = { 'cli.i18n.ok': '{locales} locale(s), {keys} key(s) used — no gaps', 'cli.i18n.synced': 'synced {locale} from {from} — {added} key(s) added, {total} total', 'cli.i18n.unused': '{count} key(s) defined in {locale} and never used:', + 'cli.jobs.backfillNoCursor': 'no cursor yet', + 'cli.jobs.backfillRow': '{name} — {rows} row(s) so far, cursor {cursor}', + 'cli.jobs.backfills': '{count} backfill(s) in flight:', 'cli.jobs.deadLetters': '{count} dead letter(s):', 'cli.jobs.depth': '{ready} ready · {running} running · {delayed} delayed · {dead} dead across {queues} queue(s)', diff --git a/packages/jobs/CLAUDE.md b/packages/jobs/CLAUDE.md index 902fe4c22..d418836b0 100644 --- a/packages/jobs/CLAUDE.md +++ b/packages/jobs/CLAUDE.md @@ -151,6 +151,26 @@ Tier 3. The `job` + `task` primitives, durable steps, transactional outbox, queu fact and this may not. `force` is the only override, and it rides the input rather than the idempotency key: one live pass per name, forced or not, or "kick it again" becomes a second writer on one table. +- **The throttle is spent INSIDE the batch's own `step.run`, and that is the whole of it.** + `backfill-rate.ts` owns one pacer per declaration (`rate`, batches/sec, default 5); the wait is + the first statement of the step body, never around it. Outside the step, an attempt resuming at + batch 500 would replay 500 completed checkpoints — which run no body and touch no database — and + still pay the full throttle of everything it had already done before reading a new row. The + pacer takes the step's `signal` and rejects `X_ABORTED` both before the wait and after it, for + the reason `execute.ts` cancels before it rejects: the wait's sleeper settles early on abort, so + what it means is "the wait is over", never "the slot arrived". Built once at declaration, like + the checksum: the interval belongs to the table and the pool, not to whichever attempt holds the + run. `rate` is deliberately NOT in the checksum, for the reason `batch` is not. +- **`inspectBackfills()` is the ONE projection of the ledger, and there is no second reader.** + `backfill-inspect.ts` maps a `BackfillRun` to a plain JSON object (epochs to ISO, absent to + `null`) for `x db backfill --list`, `x jobs ls`, `x jobs show` and `/_x`'s jobs panel — four + surfaces that must not disagree about how many rows a pass has swept. It reads no clock: a + running row's elapsed time is a different number in every process that asks, so `durationMs` is + the pass's own completed span or `null`. It answers `[]` — never a throw — for a driver with no + ledger, because `x jobs ls` is asked about the queue and must not fail over a fact nobody asked + for; the surface that IS asking (`x db backfill --list`) says so in its own summary. `JobTrace` + carries `backfill` for the same reason `steps` is on it: a step trace says which batch is next, + the ledger row says what is behind it. - **The ledger hangs off the queue driver (`driver.backfills`), optional like `introspect`.** `x_backfills` ships in `SQL_JOBS_TABLE`, so a ledger a pass cannot write is a queue it could not have been claimed from — and `dev-queue.ts` applies that one constant, so `x dev` and production @@ -202,6 +222,8 @@ picture from the other side. | `backfill.ts` | `backfill()` — a factory over `job()`: the declaration, its checksum and its input | | `backfill-pass.ts` | one pass: the batched iteration, its cursor checkpoints and its ledger row | | `backfill-ledger.ts` | `x_backfills` — the contract, the checksum, the verdict, the memory ledger | +| `backfill-rate.ts` | the `rate` throttle: batches/sec as an interval, and the cancellable wait | +| `backfill-inspect.ts` | the ledger projected for `x db backfill`, `x jobs`, `/_x` and MCP | | `register.ts` | `registerJobs`/`registerTasks` over a module namespace + the registrar announcements | | `describe.ts` | the JSON projection one handle emits; `describeJobs()` is a map over it | | `steps.ts` | `StepStore`, `StepApi`, memoized-replay executor, `StepSuspension` | @@ -220,6 +242,11 @@ picture from the other side. | `events.ts` | stored event bus for `step.waitForEvent` | | `inspect.ts` | `--json` introspection | +`backfill-pass.test.ts` is at **496 of the 500-line ceiling** — the next task that adds to it splits +the `x_backfills` describe into its own file first (it and the iteration tests share `harness()` +and `installLedger()`, so the split is a shared-fixture decision, not a cut-and-paste). The throttle +already left, as `backfill-throttle.test.ts` with a fixture of its own. + `*.job.test.ts` is the opt-in suite the `job` verify step runs: `replay.job.test.ts`, `idempotency.job.test.ts` and `outbox-atomicity.job.test.ts` each prove one named guarantee (step replay, idempotency dedupe, outbox atomicity) through a REAL worker — `start()`/`stop()` and its diff --git a/packages/jobs/README.md b/packages/jobs/README.md index 360076cb0..541c9c14e 100644 --- a/packages/jobs/README.md +++ b/packages/jobs/README.md @@ -116,6 +116,7 @@ import { db } from '@postly/db'; export const rewriteSlugs = backfill({ name: 'rewrite-slugs', // REQUIRED: a durable key batch: 1_000, // rows per statement and per step + rate: 5, // batches/sec — the default source: () => db.posts.where({ published: true }), async handle({ rows }) { await db.posts.upsertAll(rows.map(slugged), { onConflict: ['id'] }); @@ -138,6 +139,20 @@ without a statement, and the iteration reopens at the cursor they left behind. | `handle` runs at least once per page | an attempt cancelled between the last row and the checkpoint replays it — write through `upsertAll` / `updateWhere`, never `count + 1` | | `idempotencyKey` is the backfill's name | re-enqueueing a live pass is the same pass, not a second writer on one table | | `batch` is refused at declaration | `0`, `1.5` and a `NaN` from an env var fail the build, not the fourth attempt | +| `rate` throttles, and there is no way off | a sweep shares its pool with the requests the app is still serving; to go faster raise the number | +| the pause is spent **inside** the step | a resumed attempt replays 500 checkpoints and re-pays none of their pauses | + +### The throttle + +`rate` is batches per second, defaulting to `DEFAULT_BACKFILL_RATE` (5) — 5,000 rows/sec at the +default batch, one statement every 200ms, so the pool spends the rest of each interval on the +app. A rate above what the batches can actually achieve produces no wait, which is why there is +no unthrottled mode to reach for: `rate: 200` is the fast sweep. Fractions are rates too +(`rate: 0.5` is one batch every two seconds), a rate that is not finite and positive is refused +where it was written, and the wait unwinds on the run's cancellation (`X_ABORTED`) rather than +sitting in a timer nobody is waiting for. `rate` is **not** part of the definition checksum, for +the reason `batch` is not: pacing is tuning, and changing it does not make a completed sweep a +different sweep. ### The `x_backfills` ledger @@ -161,6 +176,16 @@ await rewriteSlugs.enqueue({ force: true }); // sweeps again, into a NEW row | a retry adopts its own row | `started_at` is when the pass began, not when this attempt did | | a driver without a ledger runs the pass anyway | the same degradation `introspect` has — no bookkeeping, never a refusal | +Three surfaces read it, all through one projection (`inspectBackfills()`), so none of them can +report a different number: + +| Surface | Shows | +|---|---| +| `x db backfill --list` | the whole ledger, `--name` / `--status` / `--limit`, and `--json` | +| `x jobs ls` | the passes **in flight** — rows so far and cursor, beside the queue depth | +| `x jobs show ` | the ledger row for that run, under `backfill` (`null` for any other job) | +| `/_x` → jobs | the whole ledger plus a live count, alongside the queues and the step traces | + ## The deadline cancels A job's `timeout` aborts `ctx.signal` **before** it fails the attempt, because the nack that diff --git a/packages/jobs/src/backfill-inspect.ts b/packages/jobs/src/backfill-inspect.ts new file mode 100644 index 000000000..4d14203d1 --- /dev/null +++ b/packages/jobs/src/backfill-inspect.ts @@ -0,0 +1,76 @@ +// The `x_backfills` ledger, projected for the surfaces that report it: `x db backfill --list`, +// `x jobs`, `/_x`'s jobs panel and MCP. Plain JSON-serialisable objects, the same contract +// `inspect.ts` holds for the queue — one shape, so the dashboard renders what `--json` prints. +// +// Nothing here reads a clock. A running row's elapsed time would need one, and "how long has this +// been going" computed against the reader's wall clock is a different number in every process +// that asks — so `durationMs` is the pass's own completed span or nothing at all. + +import type { BackfillRun, BackfillStatus } from './backfill-ledger'; +import type { JobDriver } from './driver'; + +/** One ledger row as a surface reports it: epochs become ISO, absent becomes `null`. */ +export interface BackfillProgress { + /** The pass, and the run id of the job that is sweeping — `x jobs show ` joins on it. */ + readonly runId: string; + readonly name: string; + readonly status: BackfillStatus; + readonly checksum: string; + readonly appVersion: string; + /** Rows the pass has handled so far. Absolute, so a replayed batch reports the same number. */ + readonly rows: number; + /** Where the pass had got to. `null` before the first batch and once it is over. */ + readonly cursor: string | null; + readonly startedAt: string; + readonly completedAt: string | null; + /** How long the pass took. `null` while it is still running — see the file header. */ + readonly durationMs: number | null; +} + +export function toBackfillProgress(run: BackfillRun): BackfillProgress { + return { + runId: run.runId, + name: run.name, + status: run.status, + checksum: run.checksum, + appVersion: run.appVersion, + rows: run.rows, + cursor: run.cursor, + startedAt: new Date(run.startedAt).toISOString(), + completedAt: run.completedAt === undefined ? null : new Date(run.completedAt).toISOString(), + durationMs: run.completedAt === undefined ? null : run.completedAt - run.startedAt, + }; +} + +/** + * Every pass the ledger holds, newest first. An EMPTY list on a driver that ships no ledger, and + * deliberately not a throw: `x jobs ls` and the jobs panel report the queue, and a queue that + * answers everything except "no backfills recorded" is a broken command for a fact nobody asked + * about. `x db backfill --list` says so in its own summary instead, where it IS the question. + */ +export async function inspectBackfills( + driver: JobDriver, + filter: { + readonly name?: string | undefined; + readonly status?: BackfillStatus | undefined; + readonly runId?: string | undefined; + readonly limit?: number | undefined; + } = {}, +): Promise { + const ledger = driver.backfills; + if (ledger === undefined) return []; + return (await ledger.list(filter)).map(toBackfillProgress); +} + +/** + * The pass one job run is sweeping, or nothing. Keyed by run rather than by name because `force` + * writes a NEW row for a name that already has one — the row this job wrote is the only one that + * describes this job. + */ +export async function backfillForRun( + driver: JobDriver, + runId: string, +): Promise { + const [row] = await inspectBackfills(driver, { runId, limit: 1 }); + return row; +} diff --git a/packages/jobs/src/backfill-ledger.ts b/packages/jobs/src/backfill-ledger.ts index 60ea69adc..9960d017c 100644 --- a/packages/jobs/src/backfill-ledger.ts +++ b/packages/jobs/src/backfill-ledger.ts @@ -38,6 +38,11 @@ export interface BackfillRun { export interface BackfillFilter { readonly name?: string | undefined; readonly status?: BackfillStatus | undefined; + /** + * The one pass this row belongs to. A backfill's run id IS the queue row's, so `x jobs show + * ` can ask how far the sweep behind that job has got without a second lookup table. + */ + readonly runId?: string | undefined; readonly limit?: number | undefined; } @@ -158,6 +163,7 @@ export function createMemoryBackfillLedger(clock: Clock = systemClock): Backfill .sort((a, b) => b.startedAt - a.startedAt) .filter((run) => filter.name === undefined || run.name === filter.name) .filter((run) => filter.status === undefined || run.status === filter.status) + .filter((run) => filter.runId === undefined || run.runId === filter.runId) .slice(0, filter.limit ?? 100); return Promise.resolve(rows); }, diff --git a/packages/jobs/src/backfill-pass.test.ts b/packages/jobs/src/backfill-pass.test.ts index af9949a10..6a0bdaeb8 100644 --- a/packages/jobs/src/backfill-pass.test.ts +++ b/packages/jobs/src/backfill-pass.test.ts @@ -3,7 +3,8 @@ // the rows behind it, and the `x_backfills` row that reports all of it. The source is a real // `@ultimat3/entity` chain over a real `memoryRepo`, so the cursors, the keyset paging and // `inBatches()`'s own refusals are the shipped ones and not a fixture that agrees by construction. -// `backfill.test.ts` owns the declaration; `backfill-ledger.test.ts` owns the ledger on its own. +// `backfill.test.ts` owns the declaration, `backfill-ledger.test.ts` the ledger on its own, and +// `backfill-throttle.test.ts` the pacing — every harness here declares a rate no timer resolves. import { afterEach, beforeEach, describe, expect, spyOn, test } from 'bun:test'; import type { Ctx } from '@ultimat3/core'; @@ -102,6 +103,9 @@ interface Harness { const RUN_ID = 'run-backfill-1'; +/** Above the millisecond a timer can resolve, so the pacer skips every wait and nothing sleeps. */ +const NO_WAIT_RATE = 100_000; + const ctx: Ctx = createContext(); /** @@ -136,6 +140,9 @@ const harness = (options: { batch?: number; name?: string } = {}): Harness => { }; const handle = backfill({ name: options.name ?? 'rewrite-titles', + // Every test on this harness is about the iteration and not the throttle, so it declares a + // rate no timer can resolve and the pacer skips each wait. `the rate throttle` owns pacing. + rate: NO_WAIT_RATE, ...(options.batch === undefined ? {} : { batch: options.batch }), source: () => watchedChain(table.where({ orgId: ORG }), watch), handle: ({ rows: page, index }) => { diff --git a/packages/jobs/src/backfill-pass.ts b/packages/jobs/src/backfill-pass.ts index 3a2c47122..61d49ceb3 100644 --- a/packages/jobs/src/backfill-pass.ts +++ b/packages/jobs/src/backfill-pass.ts @@ -18,6 +18,7 @@ import type { BatchIterator } from '@ultimat3/entity'; import type { BackfillDefinition, BackfillInput, BackfillReport } from './backfill'; import type { BackfillLedger, BackfillRun } from './backfill-ledger'; import { decideBackfill } from './backfill-ledger'; +import type { Pacer } from './backfill-rate'; import { jobDriver } from './driver'; import type { JobRunArgs } from './job'; import { isStepSuspension } from './steps'; @@ -38,11 +39,13 @@ interface Iteration { readonly pull: AsyncIterator; } -/** Everything about the backfill that is fixed at declaration, hashed and sized once. */ +/** Everything about the backfill that is fixed at declaration, hashed, sized and paced once. */ export interface BackfillPlan { readonly definition: BackfillDefinition; readonly size: number; readonly checksum: string; + /** The declared `rate`, already an interval. Built at declaration for the reason `size` is. */ + readonly pace: Pacer; } const fieldOf = (value: unknown, key: string): unknown => @@ -109,7 +112,7 @@ export async function backfillPass( plan: BackfillPlan, args: JobRunArgs, ): Promise { - const { definition, size, checksum } = plan; + const { definition, size, checksum, pace } = plan; const { ctx, step, runId } = args; const name = definition.name; // Absent on a driver that ships no ledger, which runs the pass with no bookkeeping rather than @@ -167,6 +170,14 @@ export async function backfillPass( const stepName = `${STEP_PREFIX}${index}`; const checkpoint = asCheckpoint( await step.run(stepName, async (signal): Promise => { + // INSIDE the body, which is the whole of it: a completed step is served from storage + // without its body running, so an attempt resuming at batch 500 replays 500 checkpoints + // and pays none of their pauses. Paced outside the step, a resumed pass would spend the + // entire throttle of everything it had already done before touching a new row. + // + // The signal is the run's cancellation composed with this step's ceiling, so a cancelled + // pass unwinds out of the wait instead of sitting in a timer nobody is waiting for. + await pace.wait({ signal, step: stepName }); const iteration = await iterate(); const next = await iteration.pull.next(); if (next.done === true) return { cursor: null, rows: 0 }; diff --git a/packages/jobs/src/backfill-rate.test.ts b/packages/jobs/src/backfill-rate.test.ts new file mode 100644 index 000000000..c9f4596df --- /dev/null +++ b/packages/jobs/src/backfill-rate.test.ts @@ -0,0 +1,174 @@ +// The pacer on its own: what it ASKS for between batches, what the batch's own time buys back, and +// that a cancelled attempt leaves the wait immediately instead of holding a timer it no longer owns. +// The sleeper is injected in most of these, so the pacing is asserted rather than spent — the two +// that keep the real one are there because "does not sit in the timer" cannot be faked. + +import { describe, expect, test } from 'bun:test'; +import { frozenClock, isUltimateError } from '@ultimat3/core'; +import { DEFAULT_BACKFILL_BATCH } from './backfill'; +import { createPacer, DEFAULT_BACKFILL_RATE } from './backfill-rate'; + +const NEVER_ABORTED = new AbortController().signal; + +interface Paced { + /** Every wait, in ms, in the order it was asked for. Waits skipped as too short never appear. */ + readonly asked: readonly number[]; + /** Time the batch itself took, charged against the interval like a real page would be. */ + batchTook(ms: number): void; + wait(signal?: AbortSignal, step?: string): Promise; +} + +/** A pacer whose sleep is bookkeeping: the frozen clock moves by exactly what was asked for. */ +const paced = (rate: number): Paced => { + const clock = frozenClock('2026-08-14T00:00:00.000Z'); + const asked: number[] = []; + const pacer = createPacer({ + rate, + job: 'rewrite-titles', + clock, + sleep: (ms) => { + asked.push(ms); + clock.advance(ms); + return Promise.resolve(); + }, + }); + return { + asked, + batchTook: (ms) => clock.advance(ms), + wait: (signal = NEVER_ABORTED, step) => + pacer.wait({ signal, ...(step === undefined ? {} : { step }) }), + }; +}; + +describe('the interval', () => { + test('lets the first batch through and holds every one after it to the declared rate', async () => { + const pacer = paced(5); + + await pacer.wait(); + await pacer.wait(); + await pacer.wait(); + + // Five batches a second is one every 200ms — and nothing before the first, which has no + // previous batch to be spaced from. + expect(pacer.asked).toEqual([200, 200]); + }); + + test('charges the batch its own time, so only a fast page is held back', async () => { + const pacer = paced(5); + await pacer.wait(); + + pacer.batchTook(150); + await pacer.wait(); + + expect(pacer.asked).toEqual([50]); + }); + + test('a batch slower than the interval waits for nothing at all', async () => { + const pacer = paced(5); + await pacer.wait(); + + pacer.batchTook(500); + await pacer.wait(); + pacer.batchTook(200); + await pacer.wait(); + + // Overshoot is not banked either: the next batch starts from where this one ended. + expect(pacer.asked).toEqual([]); + }); + + test('a rate faster than a timer can resolve degenerates to no wait, not a slower sweep', async () => { + // The documented escape hatch, and the only one: to sweep faster you raise `rate`. At 100k + // batches/sec the remaining 0.01ms has no timer, and waiting a whole one for it would leave + // the pass slower than it asked to be. + const pacer = paced(100_000); + + await pacer.wait(); + await pacer.wait(); + await pacer.wait(); + + expect(pacer.asked).toEqual([]); + }); + + test('a fractional rate is a rate: one batch every two seconds', async () => { + const pacer = paced(0.5); + + await pacer.wait(); + await pacer.wait(); + + expect(pacer.asked).toEqual([2000]); + }); +}); + +describe('cancellation', () => { + test('refuses to start a wait for an attempt that no longer owns the run', async () => { + const pacer = paced(5); + await pacer.wait(); + const cancelled = AbortSignal.abort(); + + const thrown = await pacer.wait(cancelled, 'batch:1').catch((error: unknown) => error); + + expect(isUltimateError(thrown) ? thrown.code : undefined).toBe('X_ABORTED'); + expect(isUltimateError(thrown) ? thrown.cause : '').toContain('batch:1'); + // Never entered: a cancelled attempt must not spend 200ms finding out it was cancelled. + expect(pacer.asked).toEqual([]); + }); + + test('an abort DURING the wait rejects instead of resuming the pass', async () => { + const clock = frozenClock('2026-08-14T00:00:00.000Z'); + const controller = new AbortController(); + const pacer = createPacer({ + rate: 5, + job: 'rewrite-titles', + clock, + // What a real sleeper does when the signal fires: settle early, having waited part of it. + sleep: (ms, signal) => { + clock.advance(ms / 2); + controller.abort(); + expect(signal.aborted).toBe(true); + return Promise.resolve(); + }, + }); + await pacer.wait({ signal: controller.signal }); + + const thrown = await pacer + .wait({ signal: controller.signal, step: 'batch:1' }) + .catch((error: unknown) => error); + + // The sleeper resolving means "the wait is over", never "the slot arrived": read another page + // here and the attempt that replaced this one gets the write. + expect(isUltimateError(thrown) ? thrown.code : undefined).toBe('X_ABORTED'); + }); + + test('the real sleeper wakes on abort instead of sitting the interval out', async () => { + // Real timers on purpose — this is the one claim an injected sleeper cannot make. One batch + // every two seconds, cancelled 5ms in: the wait has to end with the attempt, not with the slot. + const pacer = createPacer({ rate: 0.5, job: 'rewrite-titles' }); + const controller = new AbortController(); + await pacer.wait({ signal: controller.signal }); + setTimeout(() => controller.abort(), 5); + + const startedAt = performance.now(); + const thrown = await pacer + .wait({ signal: controller.signal, step: 'batch:1' }) + .catch((error: unknown) => error); + const elapsed = performance.now() - startedAt; + + expect(isUltimateError(thrown) ? thrown.code : undefined).toBe('X_ABORTED'); + expect(elapsed).toBeLessThan(1_000); + }); +}); + +describe('the default', () => { + test('is a background sweep and not a load test', async () => { + // The number in the doc comment, in the unit an operator cares about. Raising it is allowed; + // raising it past a sweep that leaves the pool to the app is a decision, not a tweak. + expect(DEFAULT_BACKFILL_RATE * DEFAULT_BACKFILL_BATCH).toBeLessThanOrEqual(10_000); + expect(DEFAULT_BACKFILL_RATE).toBeGreaterThan(0); + + const pacer = paced(DEFAULT_BACKFILL_RATE); + await pacer.wait(); + await pacer.wait(); + + expect(pacer.asked).toEqual([1000 / DEFAULT_BACKFILL_RATE]); + }); +}); diff --git a/packages/jobs/src/backfill-rate.ts b/packages/jobs/src/backfill-rate.ts new file mode 100644 index 000000000..8108ad78b --- /dev/null +++ b/packages/jobs/src/backfill-rate.ts @@ -0,0 +1,96 @@ +// The `rate` throttle a `backfill()` sweeps under: batches per second, spent as a wait between one +// batch and the next. A sweep shares its connection pool with the requests the app is still +// serving, so a backfill that runs flat out is an outage of its own making — the pacer is what +// makes "one pass over the table" background work rather than a load test. + +import type { Clock } from '@ultimat3/core'; +import { systemClock } from '@ultimat3/core'; +import { nowMs } from './clock'; +import { JobAbortedError } from './errors'; + +/** + * Batches per second, and deliberately slow: at `DEFAULT_BACKFILL_BATCH` (1,000 rows) this is + * 5,000 rows/sec — one statement every 200ms, so the pool spends the other 199 serving the app, + * and a million-row table is still swept in under four minutes. A sweep that has to go faster + * raises the number; there is no unthrottled mode to reach for instead. + */ +export const DEFAULT_BACKFILL_RATE = 5; + +/** + * Under a millisecond there is no timer to wait on — `setTimeout(0.4)` fires a millisecond later, + * which would leave the pass SLOWER than the rate it declared. So a rate far above what the + * batches can actually achieve degenerates to no wait at all, which is the point: to sweep faster + * you raise `rate`, never turn the pacer off. + */ +const RESOLUTION_MS = 1; + +export interface PacerOptions { + /** Batches per second. A backfill validates it at declaration, where it was written. */ + readonly rate: number; + /** The name a cancellation is reported under — one pacer belongs to exactly one backfill. */ + readonly job: string; + readonly clock?: Clock | undefined; + /** + * The wait itself, injectable for the same reason the clock is: a test asserts what was ASKED + * for instead of spending it. Must settle when `signal` aborts rather than waiting the rest out. + */ + readonly sleep?: ((ms: number, signal: AbortSignal) => Promise) | undefined; +} + +export interface Pacer { + /** Batches per second this pacer was built for. */ + readonly rate: number; + /** + * Hold the caller until this batch's slot comes round. Rejects with `JobAbortedError` when the + * attempt was cancelled — before the wait or during it, never after sitting the timer out. + */ + wait(args: { readonly signal: AbortSignal; readonly step?: string | undefined }): Promise; +} + +/** Real timers, cleared on abort so a cancelled pass leaves nothing pending behind it. */ +function timerSleep(ms: number, signal: AbortSignal): Promise { + return new Promise((resolve) => { + const done = (): void => { + clearTimeout(timer); + signal.removeEventListener('abort', done); + resolve(); + }; + const timer = setTimeout(done, ms); + signal.addEventListener('abort', done, { once: true }); + }); +} + +/** + * Built once, at declaration, and shared by every attempt of that backfill in this process: the + * rate is a property of the table being swept and the pool it is swept through, not of whichever + * attempt happens to hold the run. + */ +export function createPacer(options: PacerOptions): Pacer { + const clock = options.clock ?? systemClock; + const sleep = options.sleep ?? timerSleep; + const intervalMs = 1000 / options.rate; + let lastAt: number | undefined; + + return { + rate: options.rate, + async wait({ signal, step }) { + const aborted = (): JobAbortedError => + new JobAbortedError({ job: options.job, ...(step === undefined ? {} : { step }) }); + if (signal.aborted) throw aborted(); + + const at = nowMs(clock); + // The batch's own time is interval already paid, so a slow page waits for nothing and only a + // fast one is held back. The first batch has no previous one to be spaced from. + const remaining = lastAt === undefined ? 0 : intervalMs - (at - lastAt); + if (remaining < RESOLUTION_MS) { + lastAt = at; + return; + } + await sleep(remaining, signal); + // The sleeper settles early on abort, so what it means is "the wait is over", never "the + // slot arrived" — the attempt no longer owns the run and must not read another page. + if (signal.aborted) throw aborted(); + lastAt = nowMs(clock); + }, + }; +} diff --git a/packages/jobs/src/backfill-throttle.test.ts b/packages/jobs/src/backfill-throttle.test.ts new file mode 100644 index 000000000..235d2ab42 --- /dev/null +++ b/packages/jobs/src/backfill-throttle.test.ts @@ -0,0 +1,188 @@ +// The `rate` throttle, at both ends: the pacer's arithmetic against an injected clock, and a real +// `backfill()` pass proving the declared rate actually reaches the batch loop. Its own file, and +// its own entity, because `backfill-pass.test.ts` is about the iteration and every harness there +// declares a rate no timer can resolve so that nothing in it sleeps. +// +// What the ordering assertions exist for: the wait is spent INSIDE each batch's `step.run`, so a +// completed step replays from storage without re-paying its pause. Paced outside the step, a +// backfill resuming at batch 500 would spend the whole throttle of 500 batches it had already done. + +import { describe, expect, test } from 'bun:test'; +import type { Ctx } from '@ultimat3/core'; +import { createContext, frozenClock } from '@ultimat3/core'; +import { entity, memoryRepo, tableFor, text, uuid } from '@ultimat3/entity'; +import type { BackfillDefinition } from './backfill'; +import { backfill } from './backfill'; +import { backfillPass } from './backfill-pass'; +import { createPacer, DEFAULT_BACKFILL_RATE } from './backfill-rate'; +import { createMemoryStepStore, createStepRunner } from './steps'; + +const rows = entity('backfill_rate_rows', { + columns: { id: uuid().primaryKey(), orgId: uuid(), title: text({ max: 40 }) }, +}); + +type Row = typeof rows.$row; + +const ORG = '00000000-0000-7000-8000-0000000000b1'; + +const id = (index: number): string => `00000000-0000-7000-8000-${String(index).padStart(12, '0')}`; + +const SEED: readonly Row[] = Array.from({ length: 12 }, (_, index) => ({ + id: id(index), + orgId: ORG, + title: `row-${String(index)}`, +})); + +const RUN_ID = 'run-paced-1'; +const ctx: Ctx = createContext(); + +/** What a paced pass records: the waits it asked for, and where they fell among its statements. */ +interface Throttled { + /** Every wait the pass asked for, in ms. A wait the pacer skipped never appears. */ + readonly asked: readonly number[]; + /** Statements and waits in one list, so their ORDER is what the assertion reads. */ + readonly log: readonly string[]; + readonly seen: readonly (readonly string[])[]; + readonly failOn: Set; + run(): Promise; +} + +/** + * A pass whose pacer is one the test can watch: the plan is built here rather than by `backfill()` + * so the sleep is bookkeeping instead of time, and every statement lands in the same list as every + * wait — which is how "the wait comes before the pull" is asserted rather than assumed. + */ +const throttled = (rate: number): Throttled => { + const clock = frozenClock('2026-08-14T00:00:00.000Z'); + const asked: number[] = []; + const log: string[] = []; + const seen: string[][] = []; + const failOn = new Set(); + const store = createMemoryStepStore(); + const base = memoryRepo(rows, SEED); + const table = tableFor(rows, { + ...base, + findMany: async (args) => { + log.push('read'); + return base.findMany(args); + }, + }); + const definition: BackfillDefinition = { + name: 'paced-sweep', + source: () => table.where({ orgId: ORG }), + handle: ({ rows: page, index }) => { + if (failOn.has(index)) throw new Error(`batch ${String(index)} failed`); + seen.push(page.map((entry) => entry.title)); + }, + }; + const pace = createPacer({ + rate, + job: definition.name, + clock, + sleep: (ms) => { + log.push('wait'); + asked.push(ms); + clock.advance(ms); + return Promise.resolve(); + }, + }); + return { + asked, + log, + seen, + failOn, + run: (): Promise => + backfillPass( + { definition, size: 3, checksum: 'paced-checksum', pace }, + { + input: {}, + step: createStepRunner({ runId: RUN_ID, jobName: 'paced', store }).step, + ctx, + attempt: 1, + jobId: 'job-paced', + runId: RUN_ID, + }, + ), + }; +}; + +describe('the rate throttle', () => { + test('spaces the batches at the declared rate, waiting BEFORE each statement', async () => { + const pass = throttled(5); + + await pass.run(); + + // Four batches, three intervals: the first has no previous batch to be spaced from. + expect(pass.asked).toEqual([200, 200, 200]); + expect(pass.log).toEqual(['read', 'wait', 'read', 'wait', 'read', 'wait', 'read']); + expect(pass.seen).toHaveLength(4); + }); + + test('a resumed attempt pays for the batches it runs, never for the ones it replays', async () => { + const pass = throttled(5); + pass.failOn.add(2); + await expect(pass.run()).rejects.toThrow('batch 2 failed'); + expect(pass.asked).toEqual([200, 200]); + + pass.failOn.clear(); + await pass.run(); + + // Batches 0 and 1 came back from storage without their bodies running, so the resumed attempt + // paced only batch 2 and batch 3. Paced outside the step, this pass would have waited six + // times for four batches — and a backfill resuming at batch 500 would re-pay all 500 pauses. + expect(pass.asked).toEqual([200, 200, 200, 200]); + expect(pass.seen).toHaveLength(4); + }); + + test('the default is the one a definition that says nothing sweeps under', () => { + // Pinned as arithmetic rather than as a number: the doc comment promises 5,000 rows/sec at + // `DEFAULT_BACKFILL_BATCH`, and a default raised without that promise being revisited is the + // change this assertion exists to stop. + expect(1000 / DEFAULT_BACKFILL_RATE).toBe(200); + }); + + test('a declared rate throttles the real pass, and cancelling it leaves the wait at once', async () => { + // Real timers, the shipped sleeper, and `backfill()` rather than a hand-built plan: this is the + // end-to-end claim that `rate:` on a definition reaches the batch loop at all. + const store = createMemoryStepStore(); + const seen: string[][] = []; + const table = tableFor(rows, memoryRepo(rows, SEED)); + const slow = backfill({ + name: 'slow-sweep', + batch: 3, + rate: 0.5, + source: () => table.where({ orgId: ORG }), + handle: ({ rows: page }) => { + seen.push(page.map((entry) => entry.title)); + }, + }); + const controller = new AbortController(); + const runner = createStepRunner({ + runId: 'run-slow', + jobName: 'slow-sweep', + store, + signal: controller.signal, + }); + const running = slow.run({ + input: {}, + step: runner.step, + ctx, + attempt: 1, + jobId: 'job-slow', + runId: 'run-slow', + }); + + await Bun.sleep(25); + // Unthrottled this pass is over in microtasks — `memoryRepo` never touches a socket. One batch + // in 25ms is the two-second interval actually being spent. + expect(seen).toHaveLength(1); + + const abortedAt = performance.now(); + controller.abort(); + await expect(running).rejects.toMatchObject({ code: 'X_ABORTED' }); + + // The attempt no longer owns the run, so the wait ends with it and not with the slot. + expect(performance.now() - abortedAt).toBeLessThan(1_000); + expect(seen).toHaveLength(1); + }); +}); diff --git a/packages/jobs/src/backfill.test.ts b/packages/jobs/src/backfill.test.ts index 0d844430c..d52d9606b 100644 --- a/packages/jobs/src/backfill.test.ts +++ b/packages/jobs/src/backfill.test.ts @@ -59,4 +59,37 @@ describe('the factory', () => { expect(getJob(`bad-${String(batch)}`)).toBeUndefined(); } }); + + test('refuses a rate that is not batches per second, where it was written', () => { + for (const rate of [0, -1, Number.NaN, Number.POSITIVE_INFINITY]) { + let thrown: unknown; + try { + backfill({ + name: `unpaced-${String(rate)}`, + rate, + source, + handle: () => undefined, + }); + } catch (error) { + thrown = error; + } + expect(isUltimateError(thrown) ? thrown.code : undefined).toBe('X_INVARIANT'); + // Not one statement into a dead-lettered job: an unpaced sweep saturates the pool, and the + // app finds out before the queue does. + expect(getJob(`unpaced-${String(rate)}`)).toBeUndefined(); + } + }); + + test('a fraction is a rate too — one batch every two seconds', () => { + // Unlike `batch`, which is a whole number of rows: this one is a frequency, and a backfill + // slower than one batch a second is exactly what a hot table wants. + const handle = backfill({ + name: 'slow-and-steady', + rate: 0.5, + source, + handle: () => undefined, + }); + + expect(getJob('slow-and-steady')).toBe(handle); + }); }); diff --git a/packages/jobs/src/backfill.ts b/packages/jobs/src/backfill.ts index e0bb3a920..2db1231ea 100644 --- a/packages/jobs/src/backfill.ts +++ b/packages/jobs/src/backfill.ts @@ -15,6 +15,7 @@ import type { ReadBuilder } from '@ultimat3/entity'; import { t } from '@ultimat3/schema'; import { backfillChecksum } from './backfill-ledger'; import { backfillPass } from './backfill-pass'; +import { createPacer, DEFAULT_BACKFILL_RATE } from './backfill-rate'; import type { DurationInput } from './clock'; import type { JobHandle } from './job'; import { job } from './job'; @@ -69,6 +70,13 @@ export interface BackfillDefinition { handle(batch: BackfillBatch): Promise | void; /** Rows per statement and per step. Defaults to `DEFAULT_BACKFILL_BATCH`. */ readonly batch?: number; + /** + * Batches per second. Defaults to `DEFAULT_BACKFILL_RATE`, which is slow on purpose: this pass + * shares its pool with the requests the app is still serving. Fractions are a rate too — + * `rate: 0.5` is one batch every two seconds. To sweep faster raise it; there is no way to + * turn it off, because a backfill that saturates the pool has no correct value here. + */ + readonly rate?: number; readonly queue?: string; readonly retry?: RetryPolicy; /** Per attempt, not per pass: a resumed attempt picks up at the last checkpoint. */ @@ -110,9 +118,22 @@ export function backfill(definition: BackfillDefinition): JobHandle 0, + `backfill "${definition.name}" declares rate: ${String(rate)} — a rate is batches per second, greater than zero`, + `set rate: ${DEFAULT_BACKFILL_RATE} on backfill("${definition.name}"), or leave it out — to sweep faster raise the number, there is no unthrottled mode`, + ); // Hashed once, at declaration: the definition cannot change while the process runs, and a hash // per attempt would charge every batch of every pass for a fact that is fixed at import. + // `rate` is NOT in it, for the reason `batch` is not: pacing is a tuning knob, and changing one + // does not make a completed sweep a different sweep. const checksum = backfillChecksum(definition.source, definition.handle); + // Built here rather than per attempt: the interval belongs to the table and the pool, not to + // whichever attempt holds the run, so a retrying pass keeps the pace it was declared with. + const pace = createPacer({ rate, job: definition.name }); return job({ name: definition.name, @@ -124,6 +145,6 @@ export function backfill(definition: BackfillDefinition): JobHandle backfillPass({ definition, size, checksum }, args), + run: (args) => backfillPass({ definition, size, checksum, pace }, args), }); } diff --git a/packages/jobs/src/driver-pg-sql.ts b/packages/jobs/src/driver-pg-sql.ts index 903ffb607..bfe0e97e3 100644 --- a/packages/jobs/src/driver-pg-sql.ts +++ b/packages/jobs/src/driver-pg-sql.ts @@ -213,6 +213,12 @@ update x_backfills where run_id = $1 `.trim(); +/** + * `$3::uuid`, where its two neighbours are `::text`, because `run_id` IS a uuid: the cast on the + * null test is what tells Postgres the parameter's type, and `::text` there pins it to text for + * the comparison below it — `uuid = text` has no operator, so every call failed, filtered or not. + * Nothing hand-types this value; it arrives from a job record the same database wrote. + */ export const SQL_BACKFILL_LIST = ` select run_id, name, checksum, status, app_version, rows_processed, last_cursor, (extract(epoch from started_at) * 1000)::bigint as started_at, @@ -220,6 +226,7 @@ select run_id, name, checksum, status, app_version, rows_processed, last_cursor, from x_backfills where ($1::text is null or name = $1) and ($2::text is null or status = $2) + and ($3::uuid is null or run_id = $3) order by started_at desc - limit $3 + limit $4 `.trim(); diff --git a/packages/jobs/src/driver-pg.test.ts b/packages/jobs/src/driver-pg.test.ts index 4a7825474..7e23aeac1 100644 --- a/packages/jobs/src/driver-pg.test.ts +++ b/packages/jobs/src/driver-pg.test.ts @@ -4,6 +4,7 @@ import { createPgDriver } from './driver-pg'; import { SQL_ACK, SQL_BACKFILL_FINISH, + SQL_BACKFILL_LIST, SQL_BACKFILL_PROGRESS, SQL_BACKFILL_START, SQL_CLAIM, @@ -165,12 +166,28 @@ describe('the pg backfill ledger', () => { completedAt: 2000, }, ]); - expect(executor.calls[0]?.params).toEqual(['sweep', 'completed', 5]); + expect(executor.calls[0]?.params).toEqual(['sweep', 'completed', null, 5]); }); test('an unfiltered list is nulls and the default limit, never a missing predicate', async () => { const executor = recordingExecutor(); await ledgerOf(executor).list(); - expect(executor.calls[0]?.params).toEqual([null, null, 100]); + expect(executor.calls[0]?.params).toEqual([null, null, null, 100]); + }); + + test('a run id rides in its own parameter, cast to the uuid the column actually is', async () => { + const executor = recordingExecutor(); + await ledgerOf(executor).list({ runId: '11111111-2222-3333-4444-555555555555', limit: 1 }); + + expect(executor.calls[0]?.params).toEqual([ + null, + null, + '11111111-2222-3333-4444-555555555555', + 1, + ]); + // `uuid = text` has no operator in Postgres, so a `::text` cast here is not a style choice — + // it fails every call, filtered or not. Pinned as text rather than as a substring of the + // statement so a reformat of the SQL does not silently retire the assertion. + expect(SQL_BACKFILL_LIST).toContain('$3::uuid is null or run_id = $3'); }); }); diff --git a/packages/jobs/src/driver-pg.ts b/packages/jobs/src/driver-pg.ts index 509bd34aa..84afc9ce1 100644 --- a/packages/jobs/src/driver-pg.ts +++ b/packages/jobs/src/driver-pg.ts @@ -219,6 +219,7 @@ function pgBackfillLedger(exec: () => PgExecutor): BackfillLedger { const rows = await exec().query(SQL_BACKFILL_LIST, [ filter.name ?? null, filter.status ?? null, + filter.runId ?? null, filter.limit ?? 100, ]); return rows.map(toBackfillRun); diff --git a/packages/jobs/src/index.ts b/packages/jobs/src/index.ts index 0a385e5d4..45f3cb2fb 100644 --- a/packages/jobs/src/index.ts +++ b/packages/jobs/src/index.ts @@ -17,6 +17,8 @@ export type { BackfillReport, } from './backfill'; export { backfill, DEFAULT_BACKFILL_BATCH } from './backfill'; +export type { BackfillProgress } from './backfill-inspect'; +export { backfillForRun, inspectBackfills, toBackfillProgress } from './backfill-inspect'; export type { BackfillFilter, BackfillLedger, @@ -25,6 +27,8 @@ export type { BackfillVerdict, } from './backfill-ledger'; export { backfillChecksum, createMemoryBackfillLedger, decideBackfill } from './backfill-ledger'; +export type { Pacer, PacerOptions } from './backfill-rate'; +export { createPacer, DEFAULT_BACKFILL_RATE } from './backfill-rate'; export type { JobDescriptor } from './describe'; export type { ClaimedJob, diff --git a/packages/jobs/src/inspect.ts b/packages/jobs/src/inspect.ts index 733f05aa3..47396614d 100644 --- a/packages/jobs/src/inspect.ts +++ b/packages/jobs/src/inspect.ts @@ -2,6 +2,8 @@ // object so `x jobs ... --json` and the MCP tool share one shape — an agent debugging a stuck // queue reads exactly what the dashboard renders. +import type { BackfillProgress } from './backfill-inspect'; +import { backfillForRun } from './backfill-inspect'; import type { JobDriver, JobFilter, JobRecord, QueueStats } from './driver'; import { JobsNotImplementedError } from './errors'; import { registeredJobs } from './job'; @@ -70,6 +72,12 @@ export interface JobTrace { readonly steps: readonly StepTrace[]; /** Remaining retry delays in ms, jitter excluded. */ readonly retryDelaysMs: readonly number[]; + /** + * The `x_backfills` row this run wrote, when the job is a `backfill()`. `null` for every other + * job and for a driver with no ledger — a step trace says which batch is next, and this says how + * many rows the pass has actually put behind it. + */ + readonly backfill: BackfillProgress | null; } const iso = (ms: number | undefined): string | null => @@ -103,6 +111,7 @@ export async function inspectJob(driver: JobDriver, jobId: string): Promise candidate.name === record.name); + const backfill = await backfillForRun(driver, record.runId); return { id: record.id, name: record.name, @@ -117,6 +126,7 @@ export async function inspectJob(driver: JobDriver, jobId: string): Promise` | scaffold a monorepo that already runs | shipped | | `x dev` | all roles in one process: embedded services, sub-second reload, `/_x` mounted | shipped | | `x g ` | scaffold a primitive with its test | shipped | -| `x db ` | gen, migrate, reset, studio, branch | shipped | +| `x db ` | gen, migrate, reset, studio, branch, backfill | shipped | | `x verify [--workers N]` | the gate — 17 steps, in this order: typecheck, lint, boundaries, filesize, package-shape, errors, unit, contract, live, job, e2e, eval, drift, contract-diff, budgets, manifest, roadmap | shipped | | `x env [check\|example]` | validate the process env against `envSchema`, or regenerate `.env.example` from it | shipped | | `x secrets ` | the committed encrypted secrets file: show, init, edit, set, rotate | shipped | @@ -208,6 +208,7 @@ Alias: `x generate`. ```bash x db gen "add publish_at" | migrate | reset | studio | branch + | backfill --list [--name n] [--status s] [--limit n] [--json] ``` | Subcommand | Does | Notes | @@ -217,11 +218,34 @@ x db gen "add publish_at" | migrate | reset | studio | branch | `reset` | delete the embedded data directory, then migrate | **embedded database only** — against an external Postgres it exits `X_NOT_IMPLEMENTED` and tells you to drop and recreate it yourself | | `studio` | — | **planned**: exits `X_NOT_IMPLEMENTED` pointing at the `/_x` db panel. It used to shell out to `bunx drizzle-kit studio`; one subcommand is not worth a second schema engine | | `branch ` | `CREATE DATABASE … TEMPLATE` copy-on-write clone (PGlite: a copied data directory) | the isolation an agent should use before migrating | +| `backfill --list` | print the `x_backfills` ledger — one row per `backfill()` pass, newest first | `--list` is required, and its absence is `X_CLI_BAD_FLAG` naming the invocation that works: `x db backfill ` will one day *run* a pass, and a bare command that quietly listed would become a silent no-op the day it does | | Flag | Type | Default | Meaning | |---|---|---|---| -| `--name` | string | — | migration or branch name, when you would rather not pass it positionally | +| `--name` | string | — | migration or branch name, or the backfill to filter the ledger by, when you would rather not pass it positionally | | `--allow-destructive` | boolean | `false` | let `gen` emit a DROP whose `down` cannot restore the rows. `X_MIGRATION_IRREVERSIBLE`'s own `fix:` line names it | +| `--list` | boolean | `false` | `backfill`: print the ledger. The only shape this subcommand ships | +| `--status` | string | — | `backfill`: `running`, `completed` or `failed`. Anything else is `X_CLI_BAD_FLAG` naming the three | +| `--limit` | string | `100` | `backfill`: max ledger rows | + +**The backfill ledger is what has already been SWEPT.** One row per pass — name, status, rows +processed, last cursor, the app version that started it, started/completed — keyed by run rather +than by name, so `enqueue({ force: true })` writes a *new* row instead of editing the one it +reruns. A completed row is what makes re-enqueueing that name a no-op. + +```bash +$ x db backfill --list + name status rows cursor started-at duration-ms run-id + reindex-posts completed 84000 - 2026-08-14T09:00:00.000Z 240000 9c1e… + recount-likes running 12500 post_12500 2026-08-14T09:31:12.004Z - 7f3a… +✓ 2 backfill pass(es) +``` + +`started-at` is the ledger's own ISO string, printed verbatim — this repo formats no date without +an explicit IANA `timeZone`, and not formatting is the one rendering with no zone to get wrong. +An empty ledger exits **0**: "nothing has swept this database yet" is an answer. `--json` carries +the same rows plus the definition checksum. The live half of the same ledger is on `x jobs ls`, +and `/_x`'s jobs panel carries the whole of it. **One migration engine, everywhere.** `gen` calls `@ultimat3/db`'s `generateMigration()`; `migrate` and `reset` call `migrate()` — the same `x_migrations` ledger, the same per-migration @@ -590,13 +614,17 @@ x jobs [ls|show |retry |drain --to ] [--queue q] [--state s] [-- | Subcommand | Does | |---|---| -| `ls` | queue depth, the matching rows, and the dead-letter list — a dead job is never filtered out of view | -| `show ` | state, attempt, every step's result, and the remaining retry delays | +| `ls` | queue depth, the matching rows, the dead-letter list — a dead job is never filtered out of view — and the `backfill()` passes **in flight**, with rows so far and cursor | +| `show ` | state, attempt, every step's result, the remaining retry delays, and the `x_backfills` row for this run when the job is a backfill (`backfill: null` for every other job) | | `retry ` | re-queue; `--from-step ` drops that step so it re-executes while everything before it replays from storage | | `drain --to memory\|redis\|nats` | move every `ready`/`delayed`/`suspended` job onto another driver; `--dry-run` reports the plan and moves nothing | Runs against the app's own driver — the ambient one when a process already installed it, otherwise the same embedded Postgres queue `x dev` boots. `drain` enqueues on the target **before** acking the source: a crash mid-drain duplicates a job, where the idempotency key dedupes it, instead of losing it. +`ls` reports only the sweeps still **running**, because it is a live view of the queue; the whole +ledger, finished passes included, is `x db backfill --list`. A driver that ships no backfill ledger +answers with no passes rather than failing the command — the queue is still the question here. + Errors: `X_JOB_UNKNOWN`, `X_CLI_BAD_FLAG`, and `X_NOT_IMPLEMENTED` from a driver with no introspection. ## x tasks diff --git a/wiki/Jobs-And-Workflows.md b/wiki/Jobs-And-Workflows.md index 25c89f091..bf925384f 100644 --- a/wiki/Jobs-And-Workflows.md +++ b/wiki/Jobs-And-Workflows.md @@ -195,9 +195,10 @@ Draining a worker mid-job is safe: it finishes the current step, persists it, an | Surface | Contents | |---|---| -| `/_x` dev panel | queue depth per queue, in-flight, failed, step timeline per job | -| `x jobs ls --json` | one row per job: state, queue, attempts, `runAt`, idempotency key | -| `x jobs show --json` | machine-readable state, step results, next retry, dead-letter reason | +| `/_x` dev panel | queue depth per queue, in-flight, failed, step timeline per job, and the whole `x_backfills` ledger with a live count | +| `x jobs ls --json` | one row per job: state, queue, attempts, `runAt`, idempotency key — plus the `backfill()` passes **in flight**, with rows so far and cursor | +| `x jobs show --json` | machine-readable state, step results, next retry, dead-letter reason, and this run's ledger row under `backfill` when the job is a backfill | +| `x db backfill --list --json` | the whole ledger: one row per pass, newest first → [CLI reference](CLI-Reference#x-db) | | MCP dev tools | `jobs.inspect` (definitions, retry policy, steps) and `queue.depth` (pending/running/failed per queue) — scope `dev:read`, never reachable in `ROLE=web` | | OpenTelemetry | one span per job, one child span per step, trace linked to the enqueuing request | | Metrics | `queue_depth{queue}`, `jobs_total{queue,outcome}`, `job_leases_lost_total{queue}` → [Observability](Observability) | From 06a568a68f8ef769999fad6d61b057cd2a36fb31 Mon Sep 17 00:00:00 2001 From: sebi Date: Fri, 14 Aug 2026 14:18:47 -0500 Subject: [PATCH 4/5] feat(cli): add backfill generator - `x g backfill ` scaffolds a backfill declaration with a passing test - backfill is a factory over job(), inheriting .enqueue(), retry, and manifest row - template provides a skeleton with the source and handle functions, requiring the user to fill in the database query - fixture test updated to include backfill in the scaffold coverage Co-Authored-By: Claude --- packages/cli/src/cmd-generate.ts | 4 ++ packages/cli/src/scaffold-fixture.ts | 1 + packages/cli/src/templates/backfill.ts | 90 ++++++++++++++++++++++++++ packages/cli/src/templates/index.ts | 1 + 4 files changed, 96 insertions(+) create mode 100644 packages/cli/src/templates/backfill.ts diff --git a/packages/cli/src/cmd-generate.ts b/packages/cli/src/cmd-generate.ts index 581f5a381..34fb136be 100644 --- a/packages/cli/src/cmd-generate.ts +++ b/packages/cli/src/cmd-generate.ts @@ -24,6 +24,7 @@ import { flagBool, flagList, flagString } from './parse'; import type { GeneratedFile, Surface } from './templates'; import { actionFiles, + backfillFiles, CATALOG_ROOT, entityFiles, i18nIndex, @@ -40,6 +41,7 @@ export const GENERATORS = [ 'resource', 'action', 'mutator', + 'backfill', 'job', 'route', 'policy', @@ -167,6 +169,8 @@ export function generate(options: GenerateOptions): readonly GeneratedFile[] { return dedupe(actionFiles(options.name, target)); case 'mutator': return dedupe(actionFiles(options.name, { ...target, mutator: true })); + case 'backfill': + return dedupe(backfillFiles(options.name, target)); case 'entity': return dedupe(entityFiles(options.name, target)); case 'policy': diff --git a/packages/cli/src/scaffold-fixture.ts b/packages/cli/src/scaffold-fixture.ts index 3ec5b0a9b..e2f590e58 100644 --- a/packages/cli/src/scaffold-fixture.ts +++ b/packages/cli/src/scaffold-fixture.ts @@ -28,6 +28,7 @@ export const FIXTURE_GENERATORS: readonly GenerateOptions[] = [ { kind: 'query', name: 'invoice-search', feature: 'invoice' }, { kind: 'query', name: 'invoice-feed', feature: 'invoice', live: true }, { kind: 'job', name: 'sweep-invoices', feature: 'invoice' }, + { kind: 'backfill', name: 'reindex-invoices', feature: 'invoice' }, { kind: 'task', name: 'nightly-sweep', feature: 'invoice' }, { kind: 'route', name: 'pricing', surface: 'site' }, { kind: 'route', name: 'billing', surface: 'app' }, diff --git a/packages/cli/src/templates/backfill.ts b/packages/cli/src/templates/backfill.ts new file mode 100644 index 000000000..0e46889b9 --- /dev/null +++ b/packages/cli/src/templates/backfill.ts @@ -0,0 +1,90 @@ +// `x g backfill` — a one-pass table sweep. The backfill is a factory over job(), not a ninth +// primitive, so it inherits .enqueue(), the retry policy, the cancellation and the manifest row. +// One live run per name: a second enqueue while the pass is going is the same pass. + +import type { FeatureTarget } from './entity'; +import type { GeneratedFile, NameSet } from './naming'; +import { names } from './naming'; + +const backfillSource = ( + name: NameSet, + feature: NameSet, +): string => `// ${name.camel}: one pass over a chain of rows. The backfill is a job factory, not a ninth +// primitive, so it inherits .enqueue(), retry, cancellation and the manifest row. +// \`BackfillBatch\` comes from @ultimat3/jobs, not @ultimat3/schema: a backfill file imports one package. + +import type { ReadBuilder } from '@ultimat3/entity'; +import type { BackfillBatch } from '@ultimat3/jobs'; +import { backfill } from '@ultimat3/jobs'; +import type { ${feature.pascal} } from '../entity'; +import * as repo from '../repo'; + +export const ${name.camel} = backfill({ + name: '${name.kebab}', + source: ({ ctx }): ReadBuilder<${feature.pascal}> => { + // Query the ${feature.kebab}s to backfill, tenanted by ctx.actor.orgId. + // Access the table through your database client or repo functions. + throw new Error('Backfill source not implemented — add your query here'); + }, + handle: async ({ rows, signal }: BackfillBatch<${feature.pascal}>) => { + // One page, in a durable step. Await signal to respect timeout and cancellation. + // Write through upsertAll, updateWhere or an idempotent statement; never count + 1. + if (rows.length === 0) return; + signal.throwIfAborted(); + // Placeholder: process the rows after querying them. + void rows; // silence unused warning; delete this line and use rows + }, + // batch: 1_000, // rows per step, default. Adjust to balance statement size and retry scope. + // rate: 5, // batches per second, default. Raise to sweep faster; there is no unthrottled mode. + // retry: { attempts: 5, backoff: 'exponential' }, +}); +`; + +const backfillTest = ( + name: NameSet, +): string => `import { createMemoryDriver, resetJobDriver, setJobDriver } from '@ultimat3/jobs'; +import { afterAll, beforeAll, expect, jobTest } from '@ultimat3/testing'; +import { ${name.camel} } from './${name.kebab}'; + +// The driver is process-global, so it is installed and released around this file rather than +// left behind for whichever test happens to run next. +beforeAll(() => { + setJobDriver(createMemoryDriver()); +}); +afterAll(resetJobDriver); + +jobTest('${name.camel} declares a durable name and retry policy', () => { + expect(${name.camel}.kind).toBe('job'); + expect(${name.camel}.idempotencyKeyFor({})).toBe(\`${name.kebab}\`); + expect(${name.camel}.retry.attempts).toBeGreaterThan(1); +}); + +jobTest('${name.camel} uses the same idempotency key across attempts', () => { + expect(${name.camel}.idempotencyKeyFor({})).toBe(${name.camel}.idempotencyKeyFor({})); +}); + +jobTest('${name.camel} projects itself into the manifest', () => { + const described = ${name.camel}.describe(); + expect(described.queue).toBe('default'); + expect(described.retry.attempts).toBeGreaterThan(0); +}); + +jobTest('${name.camel} enqueues once, and dedupes the retry', async () => { + // One live run per name, forced or not: a second enqueue while the pass is going is the same pass. + // \`.enqueue()\` is the backfill path — the declared job, queued with no scheduler involved. + const first = await ${name.camel}.enqueue({}); + expect(first.deduped).toBe(false); + const again = await ${name.camel}.enqueue({}); + expect(again.deduped).toBe(true); +}); +`; + +export function backfillFiles(rawName: string, target: FeatureTarget): readonly GeneratedFile[] { + const name = names(rawName); + const feature = names(target.feature); + const dir = `${target.surfaceDir}/${target.feature}/backfills`; + return [ + { path: `${dir}/${name.kebab}.ts`, contents: backfillSource(name, feature) }, + { path: `${dir}/${name.kebab}.test.ts`, contents: backfillTest(name) }, + ]; +} diff --git a/packages/cli/src/templates/index.ts b/packages/cli/src/templates/index.ts index 7a1cf209c..d73fc5bf9 100644 --- a/packages/cli/src/templates/index.ts +++ b/packages/cli/src/templates/index.ts @@ -4,6 +4,7 @@ export type { ActionOptions } from './action'; export { actionFiles } from './action'; export { adminFiles } from './admin'; +export { backfillFiles } from './backfill'; export type { FeatureTarget } from './entity'; export { entityFiles } from './entity'; export { jobFiles, taskFiles } from './job'; From efc2f9fd9c37798719c68e9936ce8674de2e8e7f Mon Sep 17 00:00:00 2001 From: sebi Date: Fri, 14 Aug 2026 15:00:49 -0500 Subject: [PATCH 5/5] =?UTF-8?q?fix(jobs):=20PR=20#74=20review=20=E2=80=94?= =?UTF-8?q?=20ledger=20completed=5Fat,=20pacer=20guard,=20real=20template?= MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit - `SQL_BACKFILL_START` / the memory ledger clear `completed_at` when a retry adopts its row: `finish` stamps it for `failed` too, so a running pass rendered with a completion time in the past on all four surfaces - `BACKFILL_STATUSES` + `isBackfillStatus` declared once in `@ultimat3/jobs`, `BackfillStatus` derived from the list; the CLI's copy is gone - `createPacer` asserts its own rate: `0` makes the interval Infinity, which the timer clamps to ~1ms — an unvalidated zero was "no throttle at all" - `inspectBackfills` takes `BackfillFilter` itself instead of restating it - `x g backfill` emits working source: no bare `throw new Error`, no no-op handler that checkpoints a page it never wrote. The row projection is exported so the generated test asserts the WORK and its idempotency - `handle` is at-least-once: documented in CHANGELOG, CLAUDE.md and the wiki, and pinned by a regression test driving a lost checkpoint - docs/architecture + wiki `JobDriver` samples carry the real interface, `backfills` and the `memory` driver included - `backfill-pass.test.ts` split at the 500-line ceiling into a shared `backfill-pass-fixture.ts` plus `backfill-pass-ledger.test.ts` - headers on three test files, `node:path` justified, misleading `::text` comment corrected, bare `throw` in a test replaced with an assertion Co-Authored-By: Claude --- CHANGELOG.md | 4 +- CLAUDE.md | 2 +- docs/architecture/08-jobs-internals.md | 37 +- packages/admin/src/dev/data.test.ts | 5 +- packages/admin/src/dev/panel-jobs.test.ts | 4 + packages/cli/src/db-backfill.test.ts | 9 +- packages/cli/src/db-backfill.ts | 12 +- packages/cli/src/jobs-driver.test.ts | 2 + packages/cli/src/templates/backfill.ts | 80 +++- packages/core/src/app-version.test.ts | 4 + packages/jobs/CLAUDE.md | 30 +- packages/jobs/src/backfill-inspect.ts | 11 +- packages/jobs/src/backfill-ledger.test.ts | 35 +- packages/jobs/src/backfill-ledger.ts | 16 +- packages/jobs/src/backfill-pass-fixture.ts | 170 ++++++++ .../jobs/src/backfill-pass-ledger.test.ts | 193 +++++++++ packages/jobs/src/backfill-pass.test.ts | 372 +++--------------- packages/jobs/src/backfill-rate.ts | 17 +- packages/jobs/src/backfill-throttle.test.ts | 21 +- packages/jobs/src/driver-pg-sql.ts | 7 +- packages/jobs/src/driver-pg.test.ts | 6 +- packages/jobs/src/index.ts | 8 +- wiki/Jobs-And-Workflows.md | 6 +- 23 files changed, 655 insertions(+), 396 deletions(-) create mode 100644 packages/jobs/src/backfill-pass-fixture.ts create mode 100644 packages/jobs/src/backfill-pass-ledger.test.ts diff --git a/CHANGELOG.md b/CHANGELOG.md index d72fe0338..3b0424498 100644 --- a/CHANGELOG.md +++ b/CHANGELOG.md @@ -99,9 +99,11 @@ Semver applies from 1.0.0. A breaking change to a documented API needs a major `source` is read through `inBatches()` — one statement per page, keyset, never OFFSET — and every page is handled inside its own `step.run`, named `batch:0`, `batch:1`, … So a run killed mid-pass resumes on the page it stopped at: the completed steps replay from storage without touching the database, and the iteration is opened at the cursor they leave behind. What a step persists is that cursor and a row count, **never the page** — `steps.ts` hands a completed step's output back for the whole run, so checkpointing rows would retain every row already processed until the job ended. The body is handed no `step` for the same reason step names are positional: a name minted inside it would collide with itself on the second batch. `idempotencyKey` is the backfill's name, so re-enqueueing a live pass is the same pass. + **`handle` is at least once, and that is a requirement on the handler.** It runs *before* its checkpoint lands, so an attempt killed, cancelled or lease-expired between the last row and the step record hands the same page to the next attempt: write through `upsertAll`, `updateWhere` or a statement whose second run changes nothing, never `count + 1` and never an unguarded external side effect. The order is the only one that is safe — a checkpoint written first would report a page as swept that no attempt ever wrote, and a lost page is unrecoverable where a repeated one is a handler's problem to be idempotent about. + **`x_backfills` records what has already been swept**, the twin of `x_migrations` one level up: name, definition checksum, status, app version, rows processed, last cursor, started/completed. It ships in the same DDL as `x_jobs` and `x_job_steps` and hangs off the queue driver as `driver.backfills`, so a ledger a pass cannot write is a queue it could not have been claimed from — and a driver without one runs backfills with no bookkeeping rather than refusing them, exactly as `introspect` already degrades. Re-enqueueing a **completed** name is a no-op with a report (`{ skipped: true, previousRunId }`) — no statement, no body call, no row. `enqueue({ force: true })` sweeps again and writes a **new** row: reruns are history, never an edit of the row they rerun. A definition whose checksum moved since the completed pass **warns** and still does not run, because this checksum is over function source text, which a bundler can move without a line of behaviour changing — where `@ultimat3/db`'s `auditLedger` throws on the same fact, since SQL text is what it applied. - The row is a **report**, never a resume source: where a resumed pass restarts is decided by the step checkpoints and by nothing else. A retry adopts its own row (`started_at` is when the pass began, not this attempt); an attempt that failed is recorded as `failed` with its cursor kept, because where a pass stopped is the first thing anyone asks about one. + The row is a **report**, never a resume source: where a resumed pass restarts is decided by the step checkpoints and by nothing else. A retry adopts its own row (`started_at` is when the pass began, not this attempt) and clears `completed_at`, which `finish` stamps for `failed` as well as for `completed` — every surface projects that column, so a running pass keeping it would render with a completion time in the past. An attempt that failed is recorded as `failed` with its cursor kept, because where a pass stopped is the first thing anyone asks about one. **A backfill is throttled, and the default is slow.** `rate` is batches per second and defaults to `DEFAULT_BACKFILL_RATE` (5) — 5,000 rows/sec at the default batch, one statement every 200ms, so the pool spends the rest of each interval serving the requests the app is still taking. There is **no unthrottled mode**: to sweep faster you raise the number, and a rate above what the batches can achieve simply produces no wait. The pause is spent **inside** each batch's `step.run`, which is what stops a resumed attempt from re-paying the throttle of the five hundred batches it is replaying, and it unwinds on the run's cancellation (`X_ABORTED`) rather than sitting in a timer nobody is waiting for. `rate` is not part of the definition checksum, for the reason `batch` is not: pacing is tuning, and changing it does not make a completed sweep a different sweep. diff --git a/CLAUDE.md b/CLAUDE.md index 38ed4e879..739340c03 100644 --- a/CLAUDE.md +++ b/CLAUDE.md @@ -115,7 +115,7 @@ Everything in the framework is one of these. **If a feature doesn't fit one of t **`llm()` is an action factory, not a ninth primitive — decided 2026-08.** A model call is a server-authoritative operation with an input schema, an output schema and a policy, which is the definition of an `action`; so `llm()` ([`packages/ai/src/llm.ts`](packages/ai/src/llm.ts)) *returns* one. That is what gives a model call `.tool()`, `.openapi()`, `.client()`, `.job()` and `.contract()` for free, one authz object across every surface, and a place in the manifest — none of which a ninth primitive would have inherited. The rule generalises: a new capability arrives as a **factory over an existing primitive**, never as a new kind of thing. -**`backfill()` is a job factory — the rule's second instance, decided 2026-08.** A one-pass sweep over a table is durable background work with an input schema, a retry policy, an idempotency key and a queue, which is the definition of a `job`; so `backfill()` ([`packages/jobs/src/backfill.ts`](packages/jobs/src/backfill.ts)) *returns* one, and inherits `.enqueue()`, the worker's cancellation, the dead-letter path, `x jobs show` and its manifest row. The pass is `inBatches()` — one statement per page — with every page in its own `step.run`, so a killed attempt resumes on the page it stopped at. What a step persists is a cursor, never the page. +**`backfill()` is a job factory — the rule's second instance, decided 2026-08.** A one-pass sweep over a table is durable background work with an input schema, a retry policy, an idempotency key and a queue, which is the definition of a `job`; so `backfill()` ([`packages/jobs/src/backfill.ts`](packages/jobs/src/backfill.ts)) *returns* one, and inherits `.enqueue()`, the worker's cancellation, the dead-letter path, `x jobs show` and its manifest row. The pass is `inBatches()` — one statement per page — with every page in its own `step.run`, so a killed attempt resumes on the page it stopped at. What a step persists is a cursor, never the page. **`handle` is at least once**: it runs before its checkpoint lands, so an attempt cancelled between the two replays that page — the handler must be idempotent (`upsertAll`, `updateWhere`, a statement whose second run changes nothing), never `count + 1`. ## Non-negotiables diff --git a/docs/architecture/08-jobs-internals.md b/docs/architecture/08-jobs-internals.md index 8d1afeb12..76eb5f53b 100644 --- a/docs/architecture/08-jobs-internals.md +++ b/docs/architecture/08-jobs-internals.md @@ -51,30 +51,37 @@ Worked trace for the canonical job: ## Driver interface -One interface, three implementations. **Job code never changes.** +One interface, four implementations — three for production plus `memory`, which is the one every framework test and `x dev` run against. **Job code never changes.** + +Six required methods, the `steps` store, and three optional members. A driver that ships none of the three is still a working queue: `introspect` absent is `x jobs ls` with nothing to list, `backfills` absent is a `backfill()` pass that runs with no bookkeeping rather than one that is refused, and `close` absent is a driver holding nothing to hand back. ```ts export interface JobDriver { - enqueue(job: JobRef, input: unknown, opts: EnqueueOpts, tx?: Tx): Promise; - claim(queue: string, limit: number): Promise; - heartbeat(id: JobId): Promise; - complete(id: JobId, result: unknown): Promise; - fail(id: JobId, err: SerializedError, retryAt: Date | null): Promise; - saveStep(id: JobId, name: string, result: unknown): Promise; - loadSteps(id: JobId): Promise>; - sleepUntil(id: JobId, at: Date): Promise; + readonly name: string; + /** Step persistence lives with the queue: one store, one transaction boundary. */ + readonly steps: StepStore; + enqueue(request: EnqueueRequest): Promise; + claim(options: ClaimOptions): Promise; + ack(jobId: string): Promise; + nack(jobId: string, options: NackOptions): Promise; + heartbeat(jobId: string, options: { readonly visibilityTimeoutMs: number }): Promise; + stats(): Promise; + readonly backfills?: BackfillLedger; + readonly introspect?: JobIntrospection; + close?(): Promise; } ``` -| Driver | State | Trade-off | -|---|---|---| -| `pg` (default) | `x_jobs`, `x_job_steps`, `x_backfills`, `x_outbox`, `x_rate_buckets` | outbox is free (same DB, same tx); `SKIP LOCKED` claiming; zero extra infra | -| `redis` | streams + consumer groups, outbox relay in front | high throughput, short jobs; loses "queue state in one backup" | -| `nats` | JetStream, outbox relay in front | strongest delivery semantics, most operational surface. `As of 2026-07` `claim` throws `X_NOT_IMPLEMENTED` with `fix: set jobs.driver = "pg" in app.config.ts` | +| Driver | State | `backfills` | Trade-off | +|---|---|---|---| +| `pg` (default) | `x_jobs`, `x_job_steps`, `x_backfills`, `x_outbox`, `x_rate_buckets` | yes | outbox is free (same DB, same tx); `SKIP LOCKED` claiming; zero extra infra | +| `memory` | in-process maps, lost with the process | yes | tests and `x dev` only — nothing survives a restart, so it is never a deployment target | +| `redis` | streams + consumer groups, outbox relay in front | no | high throughput, short jobs; loses "queue state in one backup" | +| `nats` | JetStream, outbox relay in front | no | strongest delivery semantics, most operational surface. `As of 2026-07` `claim` throws `X_NOT_IMPLEMENTED` with `fix: set jobs.driver = "pg" in app.config.ts` | `x_backfills` is the odd one out: it is not queue state but the ledger of what a `backfill()` pass has already swept, hanging off `JobDriver.backfills` because it ships in the same DDL as `x_jobs` — `As of 2026-08` only the `pg` and `memory` drivers carry one, and a driver without it runs backfills with no bookkeeping rather than refusing them. -Because `saveStep`/`loadSteps` are driver methods, step persistence works identically on all three. Switching is a config line plus `x jobs drain --to redis` for in-flight rows. +Because `steps` is a driver member, step persistence works identically on all four. Switching is a config line plus `x jobs drain --to redis` for in-flight rows. ## The pg claim loop diff --git a/packages/admin/src/dev/data.test.ts b/packages/admin/src/dev/data.test.ts index 24c893f13..e846e76b0 100644 --- a/packages/admin/src/dev/data.test.ts +++ b/packages/admin/src/dev/data.test.ts @@ -169,7 +169,10 @@ describe('the backfill ledger source', () => { setJobDriver(driver); try { const ledger = driver.backfills; - if (ledger === undefined) throw new Error('the memory driver must ship a backfill ledger'); + // An assertion rather than a throw: `backfills` is optional on `JobDriver`, so this is a + // claim about the memory driver worth failing on by name — and it narrows the type. + expect(ledger).toBeDefined(); + if (ledger === undefined) return; await ledger.start({ runId: 'r1', name: 'sweep', checksum: 'aa', appVersion: '1.2.0' }); await ledger.progress('r1', { rows: 40, cursor: 'p40' }); await ledger.finish('r1', { status: 'completed', rows: 40 }); diff --git a/packages/admin/src/dev/panel-jobs.test.ts b/packages/admin/src/dev/panel-jobs.test.ts index 0600f1c50..f76bfe8fe 100644 --- a/packages/admin/src/dev/panel-jobs.test.ts +++ b/packages/admin/src/dev/panel-jobs.test.ts @@ -1,3 +1,7 @@ +// The jobs panel carries the WHOLE backfill ledger beside the queue rows, and a panel that +// quietly narrowed it — to `running`, or to the `?queue=` a run row is filtered by — would answer +// "has this ever swept here" with silence. That narrowing is what this file exists to refuse. + import { describe, expect, test } from 'bun:test'; import { staticDevSources } from './data'; import type { BackfillFact, JobRunFact } from './facts'; diff --git a/packages/cli/src/db-backfill.test.ts b/packages/cli/src/db-backfill.test.ts index 171dca7c9..2b0ecbb59 100644 --- a/packages/cli/src/db-backfill.test.ts +++ b/packages/cli/src/db-backfill.test.ts @@ -4,13 +4,8 @@ import { describe, expect, test } from 'bun:test'; import type { BackfillStatus, JobDriver } from '@ultimat3/jobs'; -import { createMemoryDriver } from '@ultimat3/jobs'; -import { - BACKFILL_STATUSES, - listBackfills, - parseBackfillStatusFlag, - renderBackfillTable, -} from './db-backfill'; +import { BACKFILL_STATUSES, createMemoryDriver } from '@ultimat3/jobs'; +import { listBackfills, parseBackfillStatusFlag, renderBackfillTable } from './db-backfill'; import { BadFlagError } from './errors'; import { msg } from './messages'; diff --git a/packages/cli/src/db-backfill.ts b/packages/cli/src/db-backfill.ts index 065b0aa9a..f5a5a8d5d 100644 --- a/packages/cli/src/db-backfill.ts +++ b/packages/cli/src/db-backfill.ts @@ -4,18 +4,16 @@ // `ParsedArgs`, no app boot and no queue. `cmd-db.ts` is the CLI wiring and nothing else. import type { BackfillProgress, BackfillStatus, JobDriver } from '@ultimat3/jobs'; -import { inspectBackfills } from '@ultimat3/jobs'; +import { BACKFILL_STATUSES, inspectBackfills, isBackfillStatus } from '@ultimat3/jobs'; import { BadFlagError } from './errors'; import { parseLimitFlag } from './jobs-report'; import { msg } from './messages'; import { renderTable } from './table'; -/** Mirrors `BackfillStatus` from `@ultimat3/jobs`, which exports the type but no runtime list. */ -export const BACKFILL_STATUSES: readonly BackfillStatus[] = ['running', 'completed', 'failed']; - -const isBackfillStatus = (value: string): value is BackfillStatus => - (BACKFILL_STATUSES as readonly string[]).includes(value); - +/** + * The list and the guard are `@ultimat3/jobs`': a status the ledger can record and this flag + * rejects is exactly the drift a second copy here would produce. + */ export function parseBackfillStatusFlag(value: string | undefined): BackfillStatus | undefined { if (value === undefined) return undefined; if (isBackfillStatus(value)) return value; diff --git a/packages/cli/src/jobs-driver.test.ts b/packages/cli/src/jobs-driver.test.ts index eb3295271..4916d2e60 100644 --- a/packages/cli/src/jobs-driver.test.ts +++ b/packages/cli/src/jobs-driver.test.ts @@ -2,6 +2,8 @@ // breaks either is one that talks to the wrong database or leaves the next command locked out. import { describe, expect, test } from 'bun:test'; +// `node:path` — Bun ships no path joiner, and the boundary assertion below reads this module's +// own source off disk rather than trusting an import graph a bundler could have rewritten. import { join } from 'node:path'; import type { JobDriver } from '@ultimat3/jobs'; import { createMemoryDriver, resetJobDriver, setJobDriver } from '@ultimat3/jobs'; diff --git a/packages/cli/src/templates/backfill.ts b/packages/cli/src/templates/backfill.ts index 0e46889b9..e07553522 100644 --- a/packages/cli/src/templates/backfill.ts +++ b/packages/cli/src/templates/backfill.ts @@ -6,6 +6,12 @@ import type { FeatureTarget } from './entity'; import type { GeneratedFile, NameSet } from './naming'; import { names } from './naming'; +/** + * Working source, never a stub: a generated `throw new Error(…)` carries no `X_*` code and a + * generated no-op handler checkpoints a page it never wrote, which reports swept rows nobody + * touched. The row projection is the one line an author replaces, and it is exported so the + * generated test asserts the WORK rather than only the declaration around it. + */ const backfillSource = ( name: NameSet, feature: NameSet, @@ -13,26 +19,46 @@ const backfillSource = ( // primitive, so it inherits .enqueue(), retry, cancellation and the manifest row. // \`BackfillBatch\` comes from @ultimat3/jobs, not @ultimat3/schema: a backfill file imports one package. +import { assert } from '@ultimat3/core'; import type { ReadBuilder } from '@ultimat3/entity'; +import { postgresRepo, tableFor } from '@ultimat3/entity'; import type { BackfillBatch } from '@ultimat3/jobs'; import { backfill } from '@ultimat3/jobs'; import type { ${feature.pascal} } from '../entity'; -import * as repo from '../repo'; +import { ${feature.camel} } from '../entity'; + +/** The table as a chain — the seam \`database()\` hands an app, so this sweep reads what a query reads. */ +const ${feature.camel}Table = () => tableFor(${feature.camel}, postgresRepo(${feature.camel})); + +/** + * What the sweep writes for one row. Replace the projection with the change this pass exists to + * make, and keep it IDEMPOTENT: a page replays whole when an attempt is cancelled between the last + * row and its checkpoint, so the second run of this function must produce the first run's row. + */ +export const ${name.camel}Row = (row: ${feature.pascal}): ${feature.pascal} => ({ + ...row, + title: row.title.trim(), +}); export const ${name.camel} = backfill({ name: '${name.kebab}', source: ({ ctx }): ReadBuilder<${feature.pascal}> => { - // Query the ${feature.kebab}s to backfill, tenanted by ctx.actor.orgId. - // Access the table through your database client or repo functions. - throw new Error('Backfill source not implemented — add your query here'); + // \`where({ orgId })\` is what satisfies the tenancy guard, and a sweep with no org would visit + // every tenant at once — so the actor this run carries has to name one. + const { orgId } = ctx.actor; + assert( + orgId !== undefined, + '${name.kebab}: the actor running this pass carries no orgId, and a sweep is tenanted', + 'x jobs enqueue ${name.kebab} --actor ', + ); + return ${feature.camel}Table().where({ orgId }); }, handle: async ({ rows, signal }: BackfillBatch<${feature.pascal}>) => { - // One page, in a durable step. Await signal to respect timeout and cancellation. - // Write through upsertAll, updateWhere or an idempotent statement; never count + 1. - if (rows.length === 0) return; + // One page, in its own durable step, at least once. Write through upsertAll, updateWhere or an + // idempotent statement; never count + 1. The signal is the run cancellation composed with this + // batch's ceiling, so a cancelled pass stops here instead of writing past its lease. signal.throwIfAborted(); - // Placeholder: process the rows after querying them. - void rows; // silence unused warning; delete this line and use rows + await ${feature.camel}Table().upsertAll(rows.map(${name.camel}Row), { onConflict: ['id'] }); }, // batch: 1_000, // rows per step, default. Adjust to balance statement size and retry scope. // rate: 5, // batches per second, default. Raise to sweep faster; there is no unthrottled mode. @@ -42,9 +68,15 @@ export const ${name.camel} = backfill({ const backfillTest = ( name: NameSet, -): string => `import { createMemoryDriver, resetJobDriver, setJobDriver } from '@ultimat3/jobs'; + feature: NameSet, +): string => `// ${name.camel} sweeps rows a user never asked for, so the two facts worth failing on are its +// durable identity — one live run per name, retried under the same key — and that the row +// projection it applies is idempotent, because a cancelled attempt replays its page whole. + +import { createMemoryDriver, resetJobDriver, setJobDriver } from '@ultimat3/jobs'; import { afterAll, beforeAll, expect, jobTest } from '@ultimat3/testing'; -import { ${name.camel} } from './${name.kebab}'; +import type { ${feature.pascal} } from '../entity'; +import { ${name.camel}, ${name.camel}Row } from './${name.kebab}'; // The driver is process-global, so it is installed and released around this file rather than // left behind for whichever test happens to run next. @@ -53,9 +85,18 @@ beforeAll(() => { }); afterAll(resetJobDriver); +const row = (over: Partial<${feature.pascal}> = {}): ${feature.pascal} => ({ + id: '00000000-0000-4000-8000-000000000001', + orgId: '00000000-0000-4000-8000-000000000002', + title: ' needs normalising ', + price: { minor: 1000, currency: 'USD' }, + createdAt: new Date(0), + ...over, +}); + jobTest('${name.camel} declares a durable name and retry policy', () => { expect(${name.camel}.kind).toBe('job'); - expect(${name.camel}.idempotencyKeyFor({})).toBe(\`${name.kebab}\`); + expect(${name.camel}.idempotencyKeyFor({})).toBe('${name.kebab}'); expect(${name.camel}.retry.attempts).toBeGreaterThan(1); }); @@ -69,6 +110,19 @@ jobTest('${name.camel} projects itself into the manifest', () => { expect(described.retry.attempts).toBeGreaterThan(0); }); +jobTest('${name.camel} actually rewrites the row it is handed', () => { + // The declaration alone cannot fail this: a handler that returned without writing would still + // enqueue, still checkpoint and still report the page as swept. + expect(${name.camel}Row(row()).title).toBe('needs normalising'); +}); + +jobTest('${name.camel} is idempotent — a replayed page writes what the first pass wrote', () => { + // At least once is the contract: an attempt cancelled between the last row and its checkpoint + // hands this page to the next attempt. Twice through must equal once through. + const once = ${name.camel}Row(row()); + expect(${name.camel}Row(once)).toEqual(once); +}); + jobTest('${name.camel} enqueues once, and dedupes the retry', async () => { // One live run per name, forced or not: a second enqueue while the pass is going is the same pass. // \`.enqueue()\` is the backfill path — the declared job, queued with no scheduler involved. @@ -85,6 +139,6 @@ export function backfillFiles(rawName: string, target: FeatureTarget): readonly const dir = `${target.surfaceDir}/${target.feature}/backfills`; return [ { path: `${dir}/${name.kebab}.ts`, contents: backfillSource(name, feature) }, - { path: `${dir}/${name.kebab}.test.ts`, contents: backfillTest(name) }, + { path: `${dir}/${name.kebab}.test.ts`, contents: backfillTest(name, feature) }, ]; } diff --git a/packages/core/src/app-version.test.ts b/packages/core/src/app-version.test.ts index 94e36065f..91d70fff0 100644 --- a/packages/core/src/app-version.test.ts +++ b/packages/core/src/app-version.test.ts @@ -1,3 +1,7 @@ +// `APP_VERSION` is stamped onto every ledger row and every manifest, so "which build wrote this" +// is only answerable if an unset key resolves to one known value rather than to `undefined`. +// Pinned here because `db` and `jobs` both read this one function and neither may keep a copy. + import { describe, expect, test } from 'bun:test'; import { APP_VERSION_KEY, appVersion, DEFAULT_APP_VERSION } from './app-version'; diff --git a/packages/jobs/CLAUDE.md b/packages/jobs/CLAUDE.md index d418836b0..8aca23241 100644 --- a/packages/jobs/CLAUDE.md +++ b/packages/jobs/CLAUDE.md @@ -136,6 +136,12 @@ Tier 3. The `job` + `task` primitives, durable steps, transactional outbox, queu jumps over a page the live iteration never read. A checkpoint READ back is checked rather than trusted — `step.run` replays it through an unchecked `as T`, and an absent cursor is not `null`, so the pass would silently reopen the source at the top and walk the whole table again. +- **`handle` is AT LEAST ONCE, and the ordering that makes it so is deliberate.** The body runs + inside the step and the record is written after it returns, so an attempt killed, cancelled or + lease-expired between the two hands that page to the next attempt — which is why the doc comment, + the CHANGELOG, `CLAUDE.md` and `x g backfill`'s generated source all say the handler must be + idempotent. Never invert it: checkpointing first would report a page as swept that nobody wrote, + and a lost page is unrecoverable where a repeated one is the handler's problem. - **`x_backfills` is what has already been SWEPT, and the step checkpoints are where a pass resumes. Never the other way round.** The ledger row (`backfill-ledger.ts`) is a report an operator reads and the record that a completed name is done; the checkpoints are written in step @@ -144,7 +150,10 @@ Tier 3. The `job` + `task` primitives, durable steps, transactional outbox, queu because `force` writes a NEW row rather than editing the one it reruns — history, not an edit of it. Only `completed` blocks: a `running` row is this pass resuming or one holding the single live idempotency key, a `failed` one is an attempt the queue is about to retry, and `start()` puts an - adopted row back to `running` keeping the `started_at` the PASS began at. A moved checksum WARNS + adopted row back to `running` keeping the `started_at` the PASS began at and CLEARING + `completed_at` — `finish` stamps that column for `failed` as well as for `completed`, and all + four surfaces project it, so a retried run that kept it renders as running with a completion time + in the past. A moved checksum WARNS and still does not run — it hashes `source` and `handle`'s source text with a `\u0000` between them (raw concatenation would hash a boundary, not a pair), and a bundler moves that text without a line of behaviour changing, which is why `@ultimat3/db`'s `auditLedger` may throw on the same @@ -160,7 +169,10 @@ Tier 3. The `job` + `task` primitives, durable steps, transactional outbox, queu the reason `execute.ts` cancels before it rejects: the wait's sleeper settles early on abort, so what it means is "the wait is over", never "the slot arrived". Built once at declaration, like the checksum: the interval belongs to the table and the pool, not to whichever attempt holds the - run. `rate` is deliberately NOT in the checksum, for the reason `batch` is not. + run. `rate` is deliberately NOT in the checksum, for the reason `batch` is not. `createPacer` is + exported, so it asserts the rate itself rather than trusting `backfill()` to have done it: `0` + makes the interval `Infinity`, which the timer clamps to about a millisecond — an unvalidated + zero is "no throttle at all", the one setting this module exists to make unreachable. - **`inspectBackfills()` is the ONE projection of the ledger, and there is no second reader.** `backfill-inspect.ts` maps a `BackfillRun` to a plain JSON object (epochs to ISO, absent to `null`) for `x db backfill --list`, `x jobs ls`, `x jobs show` and `/_x`'s jobs panel — four @@ -221,7 +233,7 @@ picture from the other side. | `job.ts` | the `job()` primitive + registry + the handle's fluent surface + `registerJob` | | `backfill.ts` | `backfill()` — a factory over `job()`: the declaration, its checksum and its input | | `backfill-pass.ts` | one pass: the batched iteration, its cursor checkpoints and its ledger row | -| `backfill-ledger.ts` | `x_backfills` — the contract, the checksum, the verdict, the memory ledger | +| `backfill-ledger.ts` | `x_backfills` — the contract, `BACKFILL_STATUSES`, the checksum, the verdict, the memory ledger | | `backfill-rate.ts` | the `rate` throttle: batches/sec as an interval, and the cancellable wait | | `backfill-inspect.ts` | the ledger projected for `x db backfill`, `x jobs`, `/_x` and MCP | | `register.ts` | `registerJobs`/`registerTasks` over a module namespace + the registrar announcements | @@ -242,10 +254,14 @@ picture from the other side. | `events.ts` | stored event bus for `step.waitForEvent` | | `inspect.ts` | `--json` introspection | -`backfill-pass.test.ts` is at **496 of the 500-line ceiling** — the next task that adds to it splits -the `x_backfills` describe into its own file first (it and the iteration tests share `harness()` -and `installLedger()`, so the split is a shared-fixture decision, not a cut-and-paste). The throttle -already left, as `backfill-throttle.test.ts` with a fixture of its own. +The pass is pinned across three files off **one** fixture. `backfill-pass-fixture.ts` owns the +harness — a real `@ultimat3/entity` chain over a real `memoryRepo`, wrapped so statements, +iterations and closes are counted, plus `installLedger()`; `backfill-pass.test.ts` drives the +iteration and its checkpoints, `backfill-pass-ledger.test.ts` the `x_backfills` row the same pass +writes. Split at the 500-line ceiling, and shared rather than copied because two harnesses that +drifted would be two different passes agreeing only by construction. The throttle left earlier, as +`backfill-throttle.test.ts` with a fixture of its own (it needs a clock the pacer can be watched +against, which the shared harness deliberately does not have). `*.job.test.ts` is the opt-in suite the `job` verify step runs: `replay.job.test.ts`, `idempotency.job.test.ts` and `outbox-atomicity.job.test.ts` each prove one named guarantee (step diff --git a/packages/jobs/src/backfill-inspect.ts b/packages/jobs/src/backfill-inspect.ts index 4d14203d1..052ecd456 100644 --- a/packages/jobs/src/backfill-inspect.ts +++ b/packages/jobs/src/backfill-inspect.ts @@ -6,7 +6,7 @@ // been going" computed against the reader's wall clock is a different number in every process // that asks — so `durationMs` is the pass's own completed span or nothing at all. -import type { BackfillRun, BackfillStatus } from './backfill-ledger'; +import type { BackfillFilter, BackfillRun, BackfillStatus } from './backfill-ledger'; import type { JobDriver } from './driver'; /** One ledger row as a surface reports it: epochs become ISO, absent becomes `null`. */ @@ -50,12 +50,9 @@ export function toBackfillProgress(run: BackfillRun): BackfillProgress { */ export async function inspectBackfills( driver: JobDriver, - filter: { - readonly name?: string | undefined; - readonly status?: BackfillStatus | undefined; - readonly runId?: string | undefined; - readonly limit?: number | undefined; - } = {}, + // `BackfillFilter` itself, never its fields restated: `ledger.list` takes that type, so a field + // added to it reaches this surface instead of being silently dropped one call short of it. + filter: BackfillFilter = {}, ): Promise { const ledger = driver.backfills; if (ledger === undefined) return []; diff --git a/packages/jobs/src/backfill-ledger.test.ts b/packages/jobs/src/backfill-ledger.test.ts index 18db6a1d9..2be7bfcd1 100644 --- a/packages/jobs/src/backfill-ledger.test.ts +++ b/packages/jobs/src/backfill-ledger.test.ts @@ -4,7 +4,13 @@ import { describe, expect, test } from 'bun:test'; import { frozenClock } from '@ultimat3/core'; import type { BackfillRun } from './backfill-ledger'; -import { backfillChecksum, createMemoryBackfillLedger, decideBackfill } from './backfill-ledger'; +import { + BACKFILL_STATUSES, + backfillChecksum, + createMemoryBackfillLedger, + decideBackfill, + isBackfillStatus, +} from './backfill-ledger'; import { SQL_BACKFILL_FINISH, SQL_BACKFILL_LIST, @@ -25,6 +31,21 @@ const completed = (fields: Partial = {}): BackfillRun => ({ ...fields, }); +describe('BackfillStatus', () => { + test('the runtime list IS the declaration — nothing restates the three', () => { + // `BackfillStatus` is derived from this array, so a status added here is a status + // `x db backfill --status` accepts without a second list to remember to edit. + expect([...BACKFILL_STATUSES]).toEqual(['running', 'completed', 'failed']); + for (const status of BACKFILL_STATUSES) expect(isBackfillStatus(status)).toBe(true); + }); + + test('a string nothing in the ledger can record is refused, not cast', () => { + for (const value of ['done', 'RUNNING', '', 'complete']) { + expect(isBackfillStatus(value)).toBe(false); + } + }); +}); + describe('backfillChecksum', () => { const source = (): number => 1; const handle = (): number => 2; @@ -117,6 +138,10 @@ describe('the memory ledger', () => { // `startedAt` is when the PASS began, not this attempt. expect(adopted?.startedAt).toBe(Date.parse('2026-08-14')); expect(await store.list()).toHaveLength(1); + // `finish` stamps `completedAt` for `failed` too, so the attempt that adopts the row has to + // clear it — a running pass carrying a completion time in the past is what every surface + // reading this row would print. + expect(adopted?.completedAt).toBeUndefined(); }); test('a rerun is a new row, never an edit of the one it reruns', async () => { @@ -162,6 +187,14 @@ describe('the shipped SQL', () => { expect(SQL_BACKFILL_START).toContain("on conflict (run_id) do update set status = 'running'"); }); + test('start clears the completion time the attempt it adopts left behind', () => { + // `SQL_BACKFILL_FINISH` stamps `completed_at` for `failed` as well as `completed`, and + // `SQL_BACKFILL_LIST` returns the column to `x db backfill --list`, `x jobs show` and `/_x`. + // Adopting the row without clearing it renders a running pass with a past completion time. + expect(SQL_BACKFILL_START).toContain('completed_at = null'); + expect(SQL_BACKFILL_FINISH).toContain('completed_at = now()'); + }); + test('only a completed pass clears its cursor', () => { expect(SQL_BACKFILL_FINISH).toContain( "last_cursor = case when $2 = 'completed' then null else last_cursor end", diff --git a/packages/jobs/src/backfill-ledger.ts b/packages/jobs/src/backfill-ledger.ts index 9960d017c..ef796bd28 100644 --- a/packages/jobs/src/backfill-ledger.ts +++ b/packages/jobs/src/backfill-ledger.ts @@ -17,8 +17,18 @@ import { nowMs } from './clock'; * `failed` is the LAST ATTEMPT's verdict, not the pass's: the queue owns retries, so a row here * goes back to `running` when the next attempt starts. Only `completed` is terminal, and it is the * one status that blocks a re-run. + * + * The runtime list is the declaration and `BackfillStatus` is derived from it, the shape + * `PRIMITIVE_KINDS` already has: `x db backfill --status` has to validate a string it was handed, + * and a second list spelled out in the CLI is a status the ledger can record and the flag rejects. */ -export type BackfillStatus = 'running' | 'completed' | 'failed'; +export const BACKFILL_STATUSES = ['running', 'completed', 'failed'] as const; + +export type BackfillStatus = (typeof BACKFILL_STATUSES)[number]; + +/** Narrows a string the CLI, MCP or a URL handed over. Never a cast — the list decides. */ +export const isBackfillStatus = (value: string): value is BackfillStatus => + (BACKFILL_STATUSES as readonly string[]).includes(value); export interface BackfillRun { /** The job run this pass belongs to, and the ledger's primary key. */ @@ -133,11 +143,13 @@ export function createMemoryBackfillLedger(clock: Clock = systemClock): Backfill const existing = runs.get(run.runId); // A retry adopts its own row: `startedAt` is when the PASS began, not this attempt, and the // status goes back to `running` so a row a failed attempt marked stops claiming otherwise. + // `completedAt` goes with it — `finish` stamps one for `failed` too, and a running pass that + // kept it would report a completion time in the past on every surface that reads the row. runs.set( run.runId, existing === undefined ? { ...run, status: 'running', rows: 0, cursor: null, startedAt: nowMs(clock) } - : { ...existing, status: 'running' }, + : { ...existing, status: 'running', completedAt: undefined }, ); return Promise.resolve(); }, diff --git a/packages/jobs/src/backfill-pass-fixture.ts b/packages/jobs/src/backfill-pass-fixture.ts new file mode 100644 index 000000000..c7546c6ea --- /dev/null +++ b/packages/jobs/src/backfill-pass-fixture.ts @@ -0,0 +1,170 @@ +// The fixture two files drive one `backfill()` pass through: a real `@ultimat3/entity` chain over +// a real `memoryRepo`, wrapped so every statement, iteration and close is counted. Shared rather +// than copied because `backfill-pass.test.ts` (the iteration and its checkpoints) and +// `backfill-pass-ledger.test.ts` (the `x_backfills` row it writes) must drive the SAME pass — two +// harnesses that drifted would be two passes agreeing only by construction. +import type { Ctx } from '@ultimat3/core'; +import { assert, createContext } from '@ultimat3/core'; +import type { BatchIterator, ReadBuilder, Repo } from '@ultimat3/entity'; +import { entity, memoryRepo, tableFor, text, uuid } from '@ultimat3/entity'; +import type { BackfillInput } from './backfill'; +import { backfill } from './backfill'; +import type { BackfillLedger } from './backfill-ledger'; +import { setJobDriver } from './driver'; +import { createMemoryDriver } from './driver-memory'; +import type { StepRecord, StepStore } from './steps'; +import { createMemoryStepStore, createStepRunner } from './steps'; + +export const rows = entity('backfill_test_rows', { + columns: { id: uuid().primaryKey(), orgId: uuid(), title: text({ max: 40 }) }, +}); + +export type Row = typeof rows.$row; + +export const ORG = '00000000-0000-7000-8000-0000000000a1'; +const OTHER = '00000000-0000-7000-8000-0000000000a2'; + +/** Sortable ids, ten apart, so the keyset cursor between two pages is a real position. */ +const id = (index: number): string => + `00000000-0000-7000-8000-0000000001${String(index).padStart(2, '0')}`; + +const row = (index: number, orgId: string = ORG): Row => ({ + id: id(index), + orgId, + title: `row-${index}`, +}); + +/** Ten rows for the org under test, two more no pass may ever touch. */ +export const SEED: readonly Row[] = [ + ...Array.from({ length: 10 }, (_unused, index) => row(index * 10)), + row(900, OTHER), + row(910, OTHER), +]; + +/** What the wrappers below record: statements sent, iterations opened, iterations closed. */ +interface Watch { + reads: number; + opens: number; + closes: number; +} + +const newWatch = (): Watch => ({ reads: 0, opens: 0, closes: 0 }); + +/** `cursor` is a getter on the real iterator, so it is forwarded rather than spread-copied. */ +const watchedIterator = (iterator: BatchIterator, watch: Watch): BatchIterator => { + const close = async (): Promise => { + watch.closes += 1; + await iterator.close(); + }; + return { + get cursor(): string | null { + return iterator.cursor; + }, + [Symbol.asyncIterator]: () => iterator[Symbol.asyncIterator](), + close, + [Symbol.asyncDispose]: close, + }; +}; + +/** `after()` returns a NEW chain, so the wrapper has to survive it — the factory calls both. */ +const watchedChain = (chain: ReadBuilder, watch: Watch): ReadBuilder => ({ + ...chain, + after: (cursor) => watchedChain(chain.after(cursor), watch), + inBatches: (size) => { + watch.opens += 1; + return watchedIterator(chain.inBatches(size), watch); + }, +}); + +const countingRepo = (repo: Repo, watch: Watch): Repo => ({ + ...repo, + findMany: async (args) => { + watch.reads += 1; + return repo.findMany(args); + }, +}); + +export interface Harness { + readonly watch: Watch; + readonly store: StepStore; + /** Titles handed to the body, one entry per batch it actually ran. */ + readonly seen: readonly (readonly string[])[]; + /** Every batch index the body should reject on. */ + failOn: ReadonlySet; + /** A run id of its own is a NEW pass; the default is a retry of the same one. */ + run(options?: { runId?: string; input?: BackfillInput }): Promise; + steps(): Promise; +} + +export const RUN_ID = 'run-backfill-1'; + +/** Above the millisecond a timer can resolve, so the pacer skips every wait and nothing sleeps. */ +export const NO_WAIT_RATE = 100_000; + +export const ctx: Ctx = createContext(); + +/** + * One backfill over `SEED`, driven through a real step runner rather than a worker: this file is + * about the pass and its checkpoints, and `execute.ts` already owns what a driver does with the + * outcome. Every `run()` is a fresh attempt against the SAME store and run id, which is exactly + * what a retry is. + */ +export const harness = ( + options: { batch?: number; name?: string; store?: StepStore } = {}, +): Harness => { + const watch = newWatch(); + const store = options.store ?? createMemoryStepStore(); + const seen: string[][] = []; + const table = tableFor(rows, countingRepo(memoryRepo(rows, SEED), watch)); + const state: Harness = { + watch, + store, + seen, + failOn: new Set(), + async run(runOptions = {}) { + const runId = runOptions.runId ?? RUN_ID; + const runner = createStepRunner({ runId, jobName: 'backfill', store }); + return handle.run({ + input: runOptions.input ?? {}, + step: runner.step, + ctx, + attempt: 1, + jobId: 'job-1', + runId, + }); + }, + steps: () => store.list(RUN_ID), + }; + const handle = backfill({ + name: options.name ?? 'rewrite-titles', + // Every test on this harness is about the iteration and not the throttle, so it declares a + // rate no timer can resolve and the pacer skips each wait. `the rate throttle` owns pacing. + rate: NO_WAIT_RATE, + ...(options.batch === undefined ? {} : { batch: options.batch }), + source: () => watchedChain(table.where({ orgId: ORG }), watch), + handle: ({ rows: page, index }) => { + if (state.failOn.has(index)) throw new Error(`batch ${String(index)} failed`); + seen.push(page.map((entry) => entry.title)); + }, + }); + return state; +}; + +/** + * The ledger hangs off the installed queue driver, so a test that wants one installs a driver. + * Every test above this line runs with none — which is the documented degradation, and why the + * pass and its checkpoints are pinned without one. + */ +export const installLedger = (): BackfillLedger => { + const driver = createMemoryDriver(); + setJobDriver(driver); + const ledger = driver.backfills; + // Never a bare `Error`: `backfills` is optional on `JobDriver`, so this is a claim about the + // memory driver, and the fix names the file that would have to change for it to stop holding. + assert( + ledger !== undefined, + 'createMemoryDriver() shipped no backfill ledger, and every test that installs one needs it', + 'restore driver-memory.ts: createMemoryBackfillLedger(clock) on JobDriver.backfills', + ); + return ledger; +}; diff --git a/packages/jobs/src/backfill-pass-ledger.test.ts b/packages/jobs/src/backfill-pass-ledger.test.ts new file mode 100644 index 000000000..40e361346 --- /dev/null +++ b/packages/jobs/src/backfill-pass-ledger.test.ts @@ -0,0 +1,193 @@ +// The `x_backfills` row one `backfill()` pass writes, and the decisions taken off it: a running row +// that becomes a completed one, a failure that keeps its cursor, a retry adopting its own row, a +// completed name that blocks a re-enqueue, `force` writing a NEW row and a moved checksum that +// warns without running. Split from `backfill-pass.test.ts` at the file-size ceiling; both drive +// `backfill-pass-fixture.ts`, so the pass under the row here is the pass pinned there. + +import { afterEach, beforeEach, describe, expect, spyOn, test } from 'bun:test'; +import { logger } from '@ultimat3/core'; +import { memoryRepo, tableFor } from '@ultimat3/entity'; +import { backfill } from './backfill'; +import { + ctx, + harness, + installLedger, + ORG, + type Row, + RUN_ID, + rows, + SEED, +} from './backfill-pass-fixture'; +import { resetJobDriver } from './driver'; +import { resetJobs } from './job'; +import { createMemoryStepStore, createStepRunner, StepSuspension } from './steps'; + +beforeEach(() => { + resetJobs(); +}); + +afterEach(() => { + resetJobDriver(); +}); + +describe('the x_backfills ledger', () => { + test('one pass is one row: running while it sweeps, completed with what it swept', async () => { + const ledger = installLedger(); + const pass = harness({ batch: 3 }); + + await pass.run(); + + const runs = await ledger.list({ name: 'rewrite-titles' }); + expect(runs).toHaveLength(1); + expect(runs[0]).toMatchObject({ + runId: RUN_ID, + name: 'rewrite-titles', + status: 'completed', + rows: 10, + cursor: null, + appVersion: 'dev', + }); + // The definition's own hash, so a later edit is decidable rather than guessed at. + expect(runs[0]?.checksum).toMatch(/^[0-9a-f]{32}$/); + }); + + test('progress moves with the pass, batch by batch, and survives the failure that stops it', async () => { + const ledger = installLedger(); + const pass = harness({ batch: 3 }); + pass.failOn = new Set([2]); + + await expect(pass.run()).rejects.toThrow('batch 2 failed'); + + const failed = (await ledger.list())[0]; + // Two batches landed before the third threw, and where it stopped is kept. + expect(failed).toMatchObject({ status: 'failed', rows: 6 }); + expect(typeof failed?.cursor).toBe('string'); + + pass.failOn = new Set(); + await pass.run(); + + // A retry is the SAME pass: it adopts its own row rather than opening a second one. + const runs = await ledger.list(); + expect(runs).toHaveLength(1); + expect(runs[0]).toMatchObject({ status: 'completed', rows: 10, cursor: null }); + expect(runs[0]?.startedAt).toBe(failed?.startedAt ?? -1); + }); + + test('a completed name re-enqueued is a no-op with a report', async () => { + const ledger = installLedger(); + const pass = harness({ batch: 3 }); + await pass.run(); + const readsAfterFirst = pass.watch.reads; + const seenAfterFirst = pass.seen.length; + + const report = await pass.run({ runId: 'run-backfill-2' }); + + expect(report).toEqual({ + name: 'rewrite-titles', + batches: 0, + rows: 0, + skipped: true, + previousRunId: RUN_ID, + }); + // No statement, no body call, and no row for a pass that never opened one. + expect(pass.watch.reads).toBe(readsAfterFirst); + expect(pass.seen.length).toBe(seenAfterFirst); + expect(await ledger.list()).toHaveLength(1); + }); + + test('force sweeps again and writes a NEW row, keeping the one it reruns', async () => { + const ledger = installLedger(); + const pass = harness({ batch: 3 }); + await pass.run(); + + const report = await pass.run({ runId: 'run-backfill-2', input: { force: true } }); + + expect(report).toEqual({ name: 'rewrite-titles', batches: 4, rows: 10, skipped: false }); + expect(pass.seen.length).toBe(8); + const runs = await ledger.list({ name: 'rewrite-titles' }); + expect(runs.map((run) => run.runId)).toEqual(['run-backfill-2', RUN_ID]); + // History, not an edit of it: both passes stay readable. + expect(runs.every((run) => run.status === 'completed')).toBe(true); + }); + + test('a definition that moved since the completed pass warns, and still does not run', async () => { + const ledger = installLedger(); + const first = harness({ batch: 3 }); + await first.run(); + resetJobs(); + + // The same durable name, a different body — what editing a shipped backfill looks like. + const table = tableFor(rows, memoryRepo(rows, SEED)); + const edited = backfill({ + name: 'rewrite-titles', + batch: 3, + source: () => table.where({ orgId: ORG }), + handle: () => expect.unreachable('a completed name is a no-op until it is forced'), + }); + const runner = createStepRunner({ + runId: 'run-edited', + jobName: 'backfill', + store: first.store, + }); + const warn = spyOn(logger, 'warn'); + + const report = await edited.run({ + input: {}, + step: runner.step, + ctx, + attempt: 1, + jobId: 'job-3', + runId: 'run-edited', + }); + + const changed = warn.mock.calls.filter( + (call) => call[0] === 'jobs.backfill.definition-changed', + ); + warn.mockRestore(); + // A warning and never a refusal: this checksum is over source text, which a bundler moves. + expect(changed).toHaveLength(1); + expect(report).toMatchObject({ skipped: true, previousRunId: RUN_ID }); + expect(await ledger.list()).toHaveLength(1); + }); + + test('a suspended pass is parked, not failed', async () => { + const ledger = installLedger(); + const store = createMemoryStepStore(); + const table = tableFor(rows, memoryRepo(rows, SEED)); + const parked = backfill({ + name: 'parks-itself', + batch: 3, + source: () => table.where({ orgId: ORG }), + handle: () => { + throw new StepSuspension({ step: 'batch:0', resumeAt: 1, reason: 'sleep' }); + }, + }); + const runner = createStepRunner({ runId: 'run-parked', jobName: 'backfill', store }); + + await expect( + parked.run({ + input: {}, + step: runner.step, + ctx, + attempt: 1, + jobId: 'job-4', + runId: 'run-parked', + }), + ).rejects.toBeInstanceOf(StepSuspension); + + // Control flow, not a failure: a parked run is coming back to this exact step. + expect((await ledger.list())[0]).toMatchObject({ status: 'running', rows: 0 }); + }); + + test('with no driver installed the pass runs and reports, unblocked and unrecorded', async () => { + // The documented degradation: a driver that ships no ledger runs backfills with no + // bookkeeping rather than refusing them — so a completed name has nothing to block on. + const pass = harness({ batch: 3 }); + await pass.run(); + + const report = await pass.run({ runId: 'run-backfill-2' }); + + expect(report).toEqual({ name: 'rewrite-titles', batches: 4, rows: 10, skipped: false }); + expect(pass.seen.length).toBe(8); + }); +}); diff --git a/packages/jobs/src/backfill-pass.test.ts b/packages/jobs/src/backfill-pass.test.ts index 6a0bdaeb8..186b86112 100644 --- a/packages/jobs/src/backfill-pass.test.ts +++ b/packages/jobs/src/backfill-pass.test.ts @@ -1,170 +1,17 @@ // One `backfill()` pass, end to end: every row visited once, a killed attempt resuming on the page -// it stopped at WITHOUT re-reading the ones before it, a checkpoint that is a cursor rather than -// the rows behind it, and the `x_backfills` row that reports all of it. The source is a real -// `@ultimat3/entity` chain over a real `memoryRepo`, so the cursors, the keyset paging and -// `inBatches()`'s own refusals are the shipped ones and not a fixture that agrees by construction. -// `backfill.test.ts` owns the declaration, `backfill-ledger.test.ts` the ledger on its own, and -// `backfill-throttle.test.ts` the pacing — every harness here declares a rate no timer resolves. +// it stopped at WITHOUT re-reading the ones before it, and a checkpoint that is a cursor rather +// than the rows behind it. The `x_backfills` row the same pass writes is +// `backfill-pass-ledger.test.ts`, off the SAME harness; `backfill.test.ts` owns the declaration, +// `backfill-ledger.test.ts` the ledger on its own and `backfill-throttle.test.ts` the pacing. -import { afterEach, beforeEach, describe, expect, spyOn, test } from 'bun:test'; -import type { Ctx } from '@ultimat3/core'; -import { createContext, logger } from '@ultimat3/core'; -import type { BatchIterator, ReadBuilder, Repo } from '@ultimat3/entity'; -import { entity, memoryRepo, tableFor, text, uuid } from '@ultimat3/entity'; -import type { BackfillInput } from './backfill'; +import { afterEach, beforeEach, describe, expect, test } from 'bun:test'; +import { memoryRepo, tableFor } from '@ultimat3/entity'; import { backfill, DEFAULT_BACKFILL_BATCH } from './backfill'; -import type { BackfillLedger } from './backfill-ledger'; -import { resetJobDriver, setJobDriver } from './driver'; -import { createMemoryDriver } from './driver-memory'; +import { ctx, harness, ORG, type Row, RUN_ID, rows } from './backfill-pass-fixture'; +import { resetJobDriver } from './driver'; import { resetJobs } from './job'; -import type { StepRecord, StepStore } from './steps'; -import { createMemoryStepStore, createStepRunner, StepSuspension } from './steps'; - -const rows = entity('backfill_test_rows', { - columns: { id: uuid().primaryKey(), orgId: uuid(), title: text({ max: 40 }) }, -}); - -type Row = typeof rows.$row; - -const ORG = '00000000-0000-7000-8000-0000000000a1'; -const OTHER = '00000000-0000-7000-8000-0000000000a2'; - -/** Sortable ids, ten apart, so the keyset cursor between two pages is a real position. */ -const id = (index: number): string => - `00000000-0000-7000-8000-0000000001${String(index).padStart(2, '0')}`; - -const row = (index: number, orgId: string = ORG): Row => ({ - id: id(index), - orgId, - title: `row-${index}`, -}); - -/** Ten rows for the org under test, two more no pass may ever touch. */ -const SEED: readonly Row[] = [ - ...Array.from({ length: 10 }, (_unused, index) => row(index * 10)), - row(900, OTHER), - row(910, OTHER), -]; - -/** What the wrappers below record: statements sent, iterations opened, iterations closed. */ -interface Watch { - reads: number; - opens: number; - closes: number; -} - -const newWatch = (): Watch => ({ reads: 0, opens: 0, closes: 0 }); - -/** `cursor` is a getter on the real iterator, so it is forwarded rather than spread-copied. */ -const watchedIterator = (iterator: BatchIterator, watch: Watch): BatchIterator => { - const close = async (): Promise => { - watch.closes += 1; - await iterator.close(); - }; - return { - get cursor(): string | null { - return iterator.cursor; - }, - [Symbol.asyncIterator]: () => iterator[Symbol.asyncIterator](), - close, - [Symbol.asyncDispose]: close, - }; -}; - -/** `after()` returns a NEW chain, so the wrapper has to survive it — the factory calls both. */ -const watchedChain = (chain: ReadBuilder, watch: Watch): ReadBuilder => ({ - ...chain, - after: (cursor) => watchedChain(chain.after(cursor), watch), - inBatches: (size) => { - watch.opens += 1; - return watchedIterator(chain.inBatches(size), watch); - }, -}); - -const countingRepo = (repo: Repo, watch: Watch): Repo => ({ - ...repo, - findMany: async (args) => { - watch.reads += 1; - return repo.findMany(args); - }, -}); - -interface Harness { - readonly watch: Watch; - readonly store: StepStore; - /** Titles handed to the body, one entry per batch it actually ran. */ - readonly seen: readonly (readonly string[])[]; - /** Every batch index the body should reject on. */ - failOn: ReadonlySet; - /** A run id of its own is a NEW pass; the default is a retry of the same one. */ - run(options?: { runId?: string; input?: BackfillInput }): Promise; - steps(): Promise; -} - -const RUN_ID = 'run-backfill-1'; - -/** Above the millisecond a timer can resolve, so the pacer skips every wait and nothing sleeps. */ -const NO_WAIT_RATE = 100_000; - -const ctx: Ctx = createContext(); - -/** - * One backfill over `SEED`, driven through a real step runner rather than a worker: this file is - * about the pass and its checkpoints, and `execute.ts` already owns what a driver does with the - * outcome. Every `run()` is a fresh attempt against the SAME store and run id, which is exactly - * what a retry is. - */ -const harness = (options: { batch?: number; name?: string } = {}): Harness => { - const watch = newWatch(); - const store = createMemoryStepStore(); - const seen: string[][] = []; - const table = tableFor(rows, countingRepo(memoryRepo(rows, SEED), watch)); - const state: Harness = { - watch, - store, - seen, - failOn: new Set(), - async run(runOptions = {}) { - const runId = runOptions.runId ?? RUN_ID; - const runner = createStepRunner({ runId, jobName: 'backfill', store }); - return handle.run({ - input: runOptions.input ?? {}, - step: runner.step, - ctx, - attempt: 1, - jobId: 'job-1', - runId, - }); - }, - steps: () => store.list(RUN_ID), - }; - const handle = backfill({ - name: options.name ?? 'rewrite-titles', - // Every test on this harness is about the iteration and not the throttle, so it declares a - // rate no timer can resolve and the pacer skips each wait. `the rate throttle` owns pacing. - rate: NO_WAIT_RATE, - ...(options.batch === undefined ? {} : { batch: options.batch }), - source: () => watchedChain(table.where({ orgId: ORG }), watch), - handle: ({ rows: page, index }) => { - if (state.failOn.has(index)) throw new Error(`batch ${String(index)} failed`); - seen.push(page.map((entry) => entry.title)); - }, - }); - return state; -}; - -/** - * The ledger hangs off the installed queue driver, so a test that wants one installs a driver. - * Every test above this line runs with none — which is the documented degradation, and why the - * pass and its checkpoints are pinned without one. - */ -const installLedger = (): BackfillLedger => { - const driver = createMemoryDriver(); - setJobDriver(driver); - const ledger = driver.backfills; - if (ledger === undefined) throw new Error('the memory driver must ship a backfill ledger'); - return ledger; -}; +import type { StepStore } from './steps'; +import { createMemoryStepStore, createStepRunner } from './steps'; beforeEach(() => { resetJobs(); @@ -280,6 +127,43 @@ describe('resume', () => { expect(report).toEqual({ name: 'rewrite-titles', batches: 4, rows: 10, skipped: false }); }); + test('handle is AT LEAST ONCE: a page whose checkpoint never landed is handed over again', async () => { + // The window the contract names, and the reason a handler has to be idempotent: `handle` + // runs INSIDE the step body and the record is written after it returns, so a process killed, + // cancelled or lease-expired between the two leaves a page that was swept and not recorded. + // The order is deliberate — checkpointing first would report a page nobody wrote, and a lost + // page is unrecoverable where a repeated one is the handler's problem. + const inner = createMemoryStepStore(); + let dropped = false; + const losesOneCheckpoint: StepStore = { + ...inner, + put: async (record) => { + if (record.name === 'batch:1' && !dropped) { + dropped = true; + throw new Error('checkpoint lost'); + } + await inner.put(record); + }, + }; + const pass = harness({ batch: 3, store: losesOneCheckpoint }); + + await expect(pass.run()).rejects.toThrow('checkpoint lost'); + const report = await pass.run(); + + // `batch:1` twice, everything else once: the retry found no record of the page the first + // attempt had already handed to the body. + expect(pass.seen).toEqual([ + ['row-0', 'row-10', 'row-20'], + ['row-30', 'row-40', 'row-50'], + ['row-30', 'row-40', 'row-50'], + ['row-60', 'row-70', 'row-80'], + ['row-90'], + ]); + // The report counts checkpoints, never body calls — so the replayed page is not double-counted + // and `rows` is still the ten rows the table holds. + expect(report).toEqual({ name: 'rewrite-titles', batches: 4, rows: 10, skipped: false }); + }); + test('a completed pass replays whole: no statement, no body call, the same report', async () => { const pass = harness({ batch: 3 }); const first = await pass.run(); @@ -332,165 +216,3 @@ describe('resume', () => { expect(pass.watch.reads).toBe(readsAfterFirst); }); }); - -describe('the x_backfills ledger', () => { - test('one pass is one row: running while it sweeps, completed with what it swept', async () => { - const ledger = installLedger(); - const pass = harness({ batch: 3 }); - - await pass.run(); - - const runs = await ledger.list({ name: 'rewrite-titles' }); - expect(runs).toHaveLength(1); - expect(runs[0]).toMatchObject({ - runId: RUN_ID, - name: 'rewrite-titles', - status: 'completed', - rows: 10, - cursor: null, - appVersion: 'dev', - }); - // The definition's own hash, so a later edit is decidable rather than guessed at. - expect(runs[0]?.checksum).toMatch(/^[0-9a-f]{32}$/); - }); - - test('progress moves with the pass, batch by batch, and survives the failure that stops it', async () => { - const ledger = installLedger(); - const pass = harness({ batch: 3 }); - pass.failOn = new Set([2]); - - await expect(pass.run()).rejects.toThrow('batch 2 failed'); - - const failed = (await ledger.list())[0]; - // Two batches landed before the third threw, and where it stopped is kept. - expect(failed).toMatchObject({ status: 'failed', rows: 6 }); - expect(typeof failed?.cursor).toBe('string'); - - pass.failOn = new Set(); - await pass.run(); - - // A retry is the SAME pass: it adopts its own row rather than opening a second one. - const runs = await ledger.list(); - expect(runs).toHaveLength(1); - expect(runs[0]).toMatchObject({ status: 'completed', rows: 10, cursor: null }); - expect(runs[0]?.startedAt).toBe(failed?.startedAt ?? -1); - }); - - test('a completed name re-enqueued is a no-op with a report', async () => { - const ledger = installLedger(); - const pass = harness({ batch: 3 }); - await pass.run(); - const readsAfterFirst = pass.watch.reads; - const seenAfterFirst = pass.seen.length; - - const report = await pass.run({ runId: 'run-backfill-2' }); - - expect(report).toEqual({ - name: 'rewrite-titles', - batches: 0, - rows: 0, - skipped: true, - previousRunId: RUN_ID, - }); - // No statement, no body call, and no row for a pass that never opened one. - expect(pass.watch.reads).toBe(readsAfterFirst); - expect(pass.seen.length).toBe(seenAfterFirst); - expect(await ledger.list()).toHaveLength(1); - }); - - test('force sweeps again and writes a NEW row, keeping the one it reruns', async () => { - const ledger = installLedger(); - const pass = harness({ batch: 3 }); - await pass.run(); - - const report = await pass.run({ runId: 'run-backfill-2', input: { force: true } }); - - expect(report).toEqual({ name: 'rewrite-titles', batches: 4, rows: 10, skipped: false }); - expect(pass.seen.length).toBe(8); - const runs = await ledger.list({ name: 'rewrite-titles' }); - expect(runs.map((run) => run.runId)).toEqual(['run-backfill-2', RUN_ID]); - // History, not an edit of it: both passes stay readable. - expect(runs.every((run) => run.status === 'completed')).toBe(true); - }); - - test('a definition that moved since the completed pass warns, and still does not run', async () => { - const ledger = installLedger(); - const first = harness({ batch: 3 }); - await first.run(); - resetJobs(); - - // The same durable name, a different body — what editing a shipped backfill looks like. - const table = tableFor(rows, memoryRepo(rows, SEED)); - const edited = backfill({ - name: 'rewrite-titles', - batch: 3, - source: () => table.where({ orgId: ORG }), - handle: () => expect.unreachable('a completed name is a no-op until it is forced'), - }); - const runner = createStepRunner({ - runId: 'run-edited', - jobName: 'backfill', - store: first.store, - }); - const warn = spyOn(logger, 'warn'); - - const report = await edited.run({ - input: {}, - step: runner.step, - ctx, - attempt: 1, - jobId: 'job-3', - runId: 'run-edited', - }); - - const changed = warn.mock.calls.filter( - (call) => call[0] === 'jobs.backfill.definition-changed', - ); - warn.mockRestore(); - // A warning and never a refusal: this checksum is over source text, which a bundler moves. - expect(changed).toHaveLength(1); - expect(report).toMatchObject({ skipped: true, previousRunId: RUN_ID }); - expect(await ledger.list()).toHaveLength(1); - }); - - test('a suspended pass is parked, not failed', async () => { - const ledger = installLedger(); - const store = createMemoryStepStore(); - const table = tableFor(rows, memoryRepo(rows, SEED)); - const parked = backfill({ - name: 'parks-itself', - batch: 3, - source: () => table.where({ orgId: ORG }), - handle: () => { - throw new StepSuspension({ step: 'batch:0', resumeAt: 1, reason: 'sleep' }); - }, - }); - const runner = createStepRunner({ runId: 'run-parked', jobName: 'backfill', store }); - - await expect( - parked.run({ - input: {}, - step: runner.step, - ctx, - attempt: 1, - jobId: 'job-4', - runId: 'run-parked', - }), - ).rejects.toBeInstanceOf(StepSuspension); - - // Control flow, not a failure: a parked run is coming back to this exact step. - expect((await ledger.list())[0]).toMatchObject({ status: 'running', rows: 0 }); - }); - - test('with no driver installed the pass runs and reports, unblocked and unrecorded', async () => { - // The documented degradation: a driver that ships no ledger runs backfills with no - // bookkeeping rather than refusing them — so a completed name has nothing to block on. - const pass = harness({ batch: 3 }); - await pass.run(); - - const report = await pass.run({ runId: 'run-backfill-2' }); - - expect(report).toEqual({ name: 'rewrite-titles', batches: 4, rows: 10, skipped: false }); - expect(pass.seen.length).toBe(8); - }); -}); diff --git a/packages/jobs/src/backfill-rate.ts b/packages/jobs/src/backfill-rate.ts index 8108ad78b..bb2201530 100644 --- a/packages/jobs/src/backfill-rate.ts +++ b/packages/jobs/src/backfill-rate.ts @@ -4,7 +4,7 @@ // makes "one pass over the table" background work rather than a load test. import type { Clock } from '@ultimat3/core'; -import { systemClock } from '@ultimat3/core'; +import { assert, systemClock } from '@ultimat3/core'; import { nowMs } from './clock'; import { JobAbortedError } from './errors'; @@ -25,7 +25,11 @@ export const DEFAULT_BACKFILL_RATE = 5; const RESOLUTION_MS = 1; export interface PacerOptions { - /** Batches per second. A backfill validates it at declaration, where it was written. */ + /** + * Batches per second, greater than zero and finite. `backfill()` refuses a bad one at the + * declaration, where it was written and with the definition's name in the message; this is the + * same rule at the constructor, for the callers that do not come through a declaration. + */ readonly rate: number; /** The name a cancellation is reported under — one pacer belongs to exactly one backfill. */ readonly job: string; @@ -66,6 +70,15 @@ function timerSleep(ms: number, signal: AbortSignal): Promise { * attempt happens to hold the run. */ export function createPacer(options: PacerOptions): Pacer { + // Refused HERE and not only at `backfill()`: `rate: 0` makes `intervalMs` Infinity, which the + // timer clamps to about a millisecond — so an unvalidated zero reads as "no throttle at all", + // which is the one setting this module exists to make unreachable. A negative rate is the same + // bug with a negative wait. + assert( + Number.isFinite(options.rate) && options.rate > 0, + `createPacer({ rate: ${String(options.rate)} }) for "${options.job}" — a rate is batches per second, greater than zero`, + `pass rate: ${DEFAULT_BACKFILL_RATE} — to sweep faster raise the number, there is no unthrottled mode`, + ); const clock = options.clock ?? systemClock; const sleep = options.sleep ?? timerSleep; const intervalMs = 1000 / options.rate; diff --git a/packages/jobs/src/backfill-throttle.test.ts b/packages/jobs/src/backfill-throttle.test.ts index 235d2ab42..318a94c00 100644 --- a/packages/jobs/src/backfill-throttle.test.ts +++ b/packages/jobs/src/backfill-throttle.test.ts @@ -9,7 +9,7 @@ import { describe, expect, test } from 'bun:test'; import type { Ctx } from '@ultimat3/core'; -import { createContext, frozenClock } from '@ultimat3/core'; +import { createContext, frozenClock, isUltimateError } from '@ultimat3/core'; import { entity, memoryRepo, tableFor, text, uuid } from '@ultimat3/entity'; import type { BackfillDefinition } from './backfill'; import { backfill } from './backfill'; @@ -134,6 +134,25 @@ describe('the rate throttle', () => { expect(pass.seen).toHaveLength(4); }); + test('createPacer refuses a rate that is no throttle at all, not just backfill() does', () => { + // `createPacer` is exported, so `backfill()` is not the only way in. `rate: 0` makes the + // interval Infinity, which the timer clamps to about a millisecond — an unvalidated zero is + // therefore "sweep flat out", which is the one setting this module exists to make unreachable. + for (const rate of [0, -1, Number.NaN, Number.POSITIVE_INFINITY]) { + let thrown: unknown; + try { + createPacer({ rate, job: 'unthrottled' }); + } catch (error) { + thrown = error; + } + expect(isUltimateError(thrown)).toBe(true); + expect(isUltimateError(thrown) ? thrown.code : undefined).toBe('X_INVARIANT'); + // The message names the pacer this was asked of, so a caller that never wrote a declaration + // still knows which sweep it was building. + expect(isUltimateError(thrown) ? thrown.cause : '').toContain('unthrottled'); + } + }); + test('the default is the one a definition that says nothing sweeps under', () => { // Pinned as arithmetic rather than as a number: the doc comment promises 5,000 rows/sec at // `DEFAULT_BACKFILL_BATCH`, and a default raised without that promise being revisited is the diff --git a/packages/jobs/src/driver-pg-sql.ts b/packages/jobs/src/driver-pg-sql.ts index bfe0e97e3..a2ff76505 100644 --- a/packages/jobs/src/driver-pg-sql.ts +++ b/packages/jobs/src/driver-pg-sql.ts @@ -190,11 +190,16 @@ on conflict (run_id, name) do update * than `do nothing` because a failed attempt left `failed` behind and this one is running: * `started_at`, `app_version` and `checksum` stay as the pass began, and the progress columns * stay where the last attempt got to. + * + * `completed_at` is cleared, and is the one column here that is not preserved: `finish` stamps it + * for `failed` as well as for `completed`, so a retried run that kept it would report a running + * pass with a completion time in the past — on `x db backfill --list`, on `x jobs show` and in + * `/_x`, all of which project that column straight through. */ export const SQL_BACKFILL_START = ` insert into x_backfills (run_id, name, checksum, app_version) values ($1, $2, $3, $4) -on conflict (run_id) do update set status = 'running' +on conflict (run_id) do update set status = 'running', completed_at = null `.trim(); export const SQL_BACKFILL_PROGRESS = ` diff --git a/packages/jobs/src/driver-pg.test.ts b/packages/jobs/src/driver-pg.test.ts index 7e23aeac1..f2d469bb5 100644 --- a/packages/jobs/src/driver-pg.test.ts +++ b/packages/jobs/src/driver-pg.test.ts @@ -185,9 +185,9 @@ describe('the pg backfill ledger', () => { '11111111-2222-3333-4444-555555555555', 1, ]); - // `uuid = text` has no operator in Postgres, so a `::text` cast here is not a style choice — - // it fails every call, filtered or not. Pinned as text rather than as a substring of the - // statement so a reformat of the SQL does not silently retire the assertion. + // `run_id` IS a uuid, and `uuid = text` has no operator in Postgres — so `::uuid` here is not + // a style choice next to its two `::text` neighbours: `::text` fails every call, filtered or + // not. Pinned as text so a reformat of the SQL does not silently retire the assertion. expect(SQL_BACKFILL_LIST).toContain('$3::uuid is null or run_id = $3'); }); }); diff --git a/packages/jobs/src/index.ts b/packages/jobs/src/index.ts index 45f3cb2fb..276dd0e5e 100644 --- a/packages/jobs/src/index.ts +++ b/packages/jobs/src/index.ts @@ -26,7 +26,13 @@ export type { BackfillStatus, BackfillVerdict, } from './backfill-ledger'; -export { backfillChecksum, createMemoryBackfillLedger, decideBackfill } from './backfill-ledger'; +export { + BACKFILL_STATUSES, + backfillChecksum, + createMemoryBackfillLedger, + decideBackfill, + isBackfillStatus, +} from './backfill-ledger'; export type { Pacer, PacerOptions } from './backfill-rate'; export { createPacer, DEFAULT_BACKFILL_RATE } from './backfill-rate'; export type { JobDescriptor } from './describe'; diff --git a/wiki/Jobs-And-Workflows.md b/wiki/Jobs-And-Workflows.md index bf925384f..b6b176105 100644 --- a/wiki/Jobs-And-Workflows.md +++ b/wiki/Jobs-And-Workflows.md @@ -161,11 +161,15 @@ export interface JobDriver { nack(jobId: string, options: NackOptions): Promise; heartbeat(jobId: string, options: { readonly visibilityTimeoutMs: number }): Promise; stats(): Promise; + /** The `x_backfills` ledger, when the driver ships one. `postgres` and `memory` do. */ + readonly backfills?: BackfillLedger; readonly introspect?: JobIntrospection; close?(): Promise; } ``` +The three optional members degrade rather than refuse: no `introspect` is `x jobs ls` with nothing to list, no `backfills` is a `backfill()` pass that runs with no bookkeeping, and no `close` is a driver holding nothing to hand back. + Two implementations ship in 1.0.0. Two more are **v2** — interface-complete stubs, so an app typechecks against them, and every method throws `X_NOT_IMPLEMENTED` with a runnable `fix:` rather than silently dropping a job. | Driver | Status `As of 2026-08` | When | Trade-off | @@ -242,7 +246,7 @@ Asserted by the runner: step replay, idempotency-key dedupe, retry/backoff, conc ## Rules -- Never assume a job runs once. Assume at-least-once. +- Never assume a job runs once. Assume at-least-once. A `backfill()`'s `handle` is the same rule one level down: it runs *before* its checkpoint lands, so an attempt cancelled between the two hands that page to the next one — write through `upsertAll`, `updateWhere` or a statement whose second run changes nothing. - Never put durable business state only in the payload. - Never do slow work inline in an action — enqueue a job. - A job never renders, redirects, or reads headers. Actor and tenant come from `ctx`.