diff --git a/CHANGELOG.md b/CHANGELOG.md index d14c31d67..da3bfce35 100644 --- a/CHANGELOG.md +++ b/CHANGELOG.md @@ -78,6 +78,50 @@ Semver applies from 1.0.0. A breaking change to a documented API needs a major ``` One vocabulary, `preload('')` — no `include`, `join` or `with`, and nothing new to declare: the name is the `references()` already written, resolved by `relationNamed()` when `preload()` is called, so an unknown one is `X_PRELOAD_UNKNOWN_RELATION` on the chain and not a page later. A `belongsTo` attaches the row or `null`, a `hasMany` an array — always present, so "no author" and "nobody preloaded the author" never read the same. Several `preload()` calls resolve concurrently, never one after the other, and naming one relation twice is one statement; keys are chunked at `MAX_IDS_PER_STATEMENT` (500) exactly as a coalesced point read already is, so past 500 keys a relation costs several statements rather than one Postgres refuses for its bind count, and a relation with more rows than one page holds costs another keyset page rather than a silent truncation. The page's own tenant predicate carries onto the related read only when the other entity's tenant column has the same name — a value that scopes one entity is a guess on another, and a guess here is a cross-tenant read — so a differently-named tenant column carries nothing, and carrying nothing is not silent: the related read builds its own plan and `assertScoped` refuses it there as `X_TENANCY_UNSCOPED`. The related read is `findMany`, so `deleted_at is null` applies exactly as it does anywhere else. Attached to a copy of each row — `{ ...row }`, because the in-memory driver hands back the row it stores — after `select()`, which is widened internally with the relation's local key so a projection can drop neither the key a preload reads nor the relation it attaches; `plan().select` reports the widened list. `page()`, `all()` and `one()` preload; `count()` and `plan()` do not, since a count reads no rows. A table reaches only the entities its own `database()` call named, through the driver that call was given, so a preload against memory means what a preload against Postgres means — a relation whose other end is outside the set is `X_INVARIANT_VIOLATED` naming the `database({ … })` call that fixes it, and `tableFor(entity, repo)` built by hand reaches no other table. `preload('author')` returns `ReadBuilder` — `unknown`, not a generated type, because the name is a string resolved at runtime and the row on the other side is parsed by its own entity. +- **`insertAll(rows)` and `upsertAll(rows, { onConflict })` — the write loop, as one statement.** Three bullets above make a read loop stop being N statements; a loop of `insert()` calls still was. Both land on `Repo`, on `Table` and on **both** drivers: + + ```ts + // One `insert into … values (…), (…), (…) returning *`, not three round trips. + await db.tags.insertAll(names.map((name) => ({ orgId, name }))); + + // The like nobody can like twice, without a read to find out first. + await db.likes.upsertAll(rows, { onConflict: ['orgId', 'postId', 'memberId'], onMatch: 'nothing' }); + ``` + + `insertStatement` builds every insert in the framework now, one row or ten thousand, so `insertAll([row])` compiles to the text `insert(row)` always compiled to and there is no second builder for the two to drift apart in. Rows are `Insertable`, parsed by `$parse` exactly as one row is — declared defaults are filled here, not by the caller — and `upsertAll` also stamps `onUpdateNow()` columns through `touch()`, the one place that happens, because an upsert that lands on a stored row *is* an update. The result is the rows the call actually **wrote**: under `onMatch: 'nothing'` a row already stored is skipped and absent, which is what `returning *` says on the server and is therefore how a caller counts what it inserted. A collision overwrites every column in the batch except three closed sets — the conflict target, which is how the stored row was found, the primary key, which is where it lives, and the soft-delete stamp, which is whether the row is there at all; moving either of the first two would move a row nobody asked to move and every foreign key pointing at that id would miss it, and writing the third would bring back a row the app had deleted. A soft-deleted row still occupies its conflict target — the unique index it collides with is not partial — so `excluded."deleted_at"` is the resurrection `update(id, patch)` and `updateWhere` both refuse by carrying `deleted_at is null`, which an `on conflict` clause cannot carry; it is dropped from the set list rather than refused, because `$parse` fills `deletedAt: null` into every row before the plan is built, so that value is the framework's and not the caller's. `insertAll` is untouched: a row that collides with nothing writes the stamp it carries, exactly as `insert` does. Past Postgres's 65535 bind parameters the batch becomes several whole statements rather than one the server refuses, so an all-or-nothing caller wraps the call in `withTransaction`. Four refusals precede the statement, each of them a `42P10`, a cross-tenant write or a silent surprise otherwise: a conflict target no declared unique constraint matches, a target that omits the tenant column under `onMatch: 'update'` (`X_TENANCY_UNSCOPED` — another tenant's row would match and be rewritten, so an updating upsert must be scoped by the constraint itself), a batch that repeats one conflict target under `'update'` (Postgres answers that `ON CONFLICT DO UPDATE command cannot affect row a second time`), and a batch whose rows name different columns under `'update'`, where `excluded.` is that column's default and not "leave it alone". The in-memory driver answers all four the same way and judges the whole batch before storing any of it, so a test that passes against memory still says something about Postgres — including `NULLS DISTINCT`: a null in the conflict target collides with nothing, there and here. +- **`inBatches(size)` — reading a whole table is a terminal on the chain, not a loop around `page()`.** A page is bounded on purpose, so every backfill, export and reindex hand-rolled the cursor loop, and the hand-rolled ones are where an `offset` creeps back in. `ReadBuilder` now ends in one: + + ```ts + // One statement per batch, one page of rows in memory at a time. + for await (const batch of db.posts.where({ orgId }).preload('author').inBatches(500)) { + await search.index(batch); + } + + // Stopping early keeps the position, so a job resumes where it ran out of time. + await using batches = db.posts.where({ orgId }).after(checkpoint).inBatches(500); + for await (const batch of batches) { + await search.index(batch); + if (ctx.clock.now() > deadline) break; + } + await db.checkpoints.update(id, { cursor: batches.cursor }); + ``` + + A batch **is** the page `page()` would have returned at that position — same filters, same tenancy, same soft-delete visibility, same `select()`, same `preload()` — so there is no second read path to learn or to drift, and both drivers inherit it from the one they already share. The handle is the iteration: `break`, `return`, a throw and `await using` all stop the *next* statement (`close()` is the generator's own `return()`, so it is idempotent by construction), a second `for await` continues it rather than re-reading the table from the top, and an empty batch is never yielded. `.cursor` is where the next batch starts and advances before the yield, so a consumer that breaks reads the position it stopped at and `.after(cursor)` resumes it. Three refusals land on the chain instead of one batch later: a size that is not a whole number of rows, a chain that also called `limit()` — one number with two meanings, and honouring it reads a fraction of a batch while dropping it reads the whole table the caller thought they had bounded — and an ordering no cursor can carry, which is the one that matters most: a nullable sort column mints no cursor when the result fits in a single batch, so it would pass every test and fail once the table grew. +- **`countBy(column)` — a `count()` per row is one grouped count.** Recounting a page of posts' likes sent one `select count(*)` per post, and no batch above reaches those: each of them asks a different question. This asks all of them at once, on `Repo`, on the chain and on **both** drivers: + + ```ts + // Before: one statement per post. + for (const id of ids) { + const likeCount = await db.likes.where({ orgId, postId: id }).count(); + await db.posts.update(id, { likeCount }); + } + + // After: one statement for every post in `ids`. + const counts = await db.likes.where({ orgId }).andWhere('postId', 'in', ids).countBy('postId'); + for (const id of ids) await db.posts.update(id, { likeCount: counts.get(id) ?? 0 }); + ``` + + It counts the whole predicate and never the page, exactly as `count()` does: the chain's filters, its tenancy (`X_TENANCY_UNSCOPED` on a tenant-scoped entity with no org predicate) and its `deleted_at is null` visibility are all in the statement, and `limit()`/`after()` bound it no more than they bound a `count()`. **A value nothing matched is absent, never `0`** — that is what `group by` returns, and it is what lets a caller tell "none" from "never asked"; the default is the caller's `?? 0`. **NULL is one group**, keyed `null`, in both drivers: in memory a property the row never carried is `?? null`, so it lands in the group Postgres files its NULL rows under, while `0`, `''` and `false` stay the values they are. Entries come back biggest group first, ties by the value — numbers and bigints numerically, everything else by its text — and `null` last; the order is applied in `count-by.ts` after the rows are in rather than in the statement, because a hash aggregate returns groups in whatever order it built them and a `Map` filled row by row returns insertion order, so an `order by` there would let the two drivers disagree about a result they agree on. Groupable columns are a closed set — `uuid`, `text`, `char`, `boolean`, `integer`, `bigint` — and a `timestamptz`, a `jsonb` or `money` is `X_INVARIANT_VIOLATED` whose `fix` names a column of that entity that *is* groupable: a `Map` compares a non-primitive key by identity, so such a map could only ever answer `undefined`. The number of groups is bounded at `MAX_GROUPS` (1000), and going past it is a **refusal, not a truncation** — the statement asks for one group more than the bound, the trick a page already uses when it reads one row past its limit, and that extra group fires `X_INVARIANT_VIOLATED` spelling `andWhere('', 'in', ).countBy('')`, because a map that silently lost its tail reads exactly like a complete one and a caller recounting from it would write the wrong number to every row it missed. One statement: `select "post_id" as group_value, count(*) as group_count from "likes" where … group by "post_id" limit $n`. Both output names are fixed aliases, so they cannot collide with each other whatever the table declares — an entity is free to have a column called `count` — and the grouped value is re-parsed by the column that declared it, since `int8` arrives as a string and would otherwise key the map by text where the in-memory driver keys it by a `bigint`. No new error code, no `groupBy()` builder, nothing to declare: it is a terminal on the chain that already exists. Pinned in both drivers — in memory, and against the recording client. - **`relationMap()` — the foreign keys an entity already declared, readable at query time.** `ColumnMeta.references` was resolved in exactly one place, `describe.ts`, to spell a DDL constraint; at query time nothing could answer "what is a post's author". `@ultimat3/entity` now derives a named map from the same thunks — `belongsTo` from an entity's own foreign keys, `hasMany` from the inbound ones: ```ts diff --git a/docs/architecture/06-data-layer.md b/docs/architecture/06-data-layer.md index 60f8bab0c..86dbe301d 100644 --- a/docs/architecture/06-data-layer.md +++ b/docs/architecture/06-data-layer.md @@ -51,12 +51,16 @@ Generated per entity. The **only** place SQL lives. |---|---| | `findById(id)` | tenant filter applied automatically; returns `null`, never throws on miss. Inside a request, the calls issued in one microtask are **one** `where "id" in (…)` ([`coalesce.ts`](../../packages/entity/src/coalesce.ts)) — same scope, same soft-delete filter, one round trip. An id that is a foreign key on a page this request read resolves that key for the whole page instead ([`jit-preload.ts`](../../packages/entity/src/jit-preload.ts)), which is what batches a sequential `for … of` loop | | `findMany(args)` | cursor-paginated only (below); leaves its page's foreign key values behind for the preload above | +| `inBatches(size)` | not on `Repo` — a `ReadBuilder` terminal, `for await (const batch of db..where({ … }).inBatches(500))`. `findMany` in a loop over its own cursor ([`batch.ts`](../../packages/entity/src/batch.ts)), so a batch **is** the page `page()` would have returned at that position and there is no second read path. The handle is the iteration: `break`, a throw and `await using` all stop the next statement, and `.cursor` is where it stopped — `.after(cursor)` resumes it | | `preload(name)` | not on `Repo` — a `ReadBuilder` chain method, `db.
.preload('')`, resolved by `page()`/`all()`/`one()`. One more `where in (…)` per named relation, over the page `findMany` already read ([`preload.ts`](../../packages/entity/src/preload.ts)); several relations resolve concurrently, and naming one twice is one statement | | `insert(values)` / `update(id, patch)` | invariants run before the statement | +| `insertAll(rows)` | many rows, one `insert into … values (…), (…) returning *` ([`pg-sql.ts`](../../packages/entity/src/pg-sql.ts) builds *every* insert, so `insertAll([row])` is the text `insert(row)` produced). Resolves with the rows as stored; past 65535 bind parameters it is several statements, so all-or-nothing is `withTransaction`'s | +| `upsertAll(rows, args)` | `insertAll` that resolves a collision — `on conflict (…) do update set …`, or `do nothing`. Resolves with the rows this call **wrote**, so a skipped row is absent. What both drivers must agree on lives in [`bulk-write.ts`](../../packages/entity/src/bulk-write.ts) | | `delete(id)` | soft-deletes when the entity declares `deletedAt`, hard-deletes when it does not | | `deleteWhere(filter)` | delete by equality filter; resolves with the **number of rows removed** | | `updateWhere(filter, patch)` | update by equality filter; resolves with the **number of rows written** | | `count(args)` | the same plan as `findMany`, without the page | +| `countBy(column, args)` | the grouped count — one entry per distinct value of `column`, over exactly the rows `count(args)` counts, in one `group by` statement where a `count()` per row was N. What both drivers must agree on lives in [`count-by.ts`](../../packages/entity/src/count-by.ts) | | `Transactor.run(fn)` | joins the ambient transaction if one exists, opens one otherwise | | custom | added in the feature's `repo.ts`, returning schema-parsed rows | @@ -70,7 +74,36 @@ One family, three members: Reach for `preload()` when the relation is part of what the page *is* — a list rendered with its authors, rows handed to something that will not call back into the repo, or a read a reviewer should see stated rather than inferred from a loop. The other two ask for nothing: a same-microtask fan-out and a sequential `for … of` loop already get them for free. All three read their keys through one file ([`batch-read.ts`](../../packages/entity/src/batch-read.ts)) — one spelling of a key, one 500-id bind cap — so a bound and a key's identity cannot disagree between them. What each may widen is its own: the two implicit paths share a scope key and refuse to widen across a tenant, a soft-delete visibility, a projection or an entity, because they answer a statement the caller already sent; `preload()` sends its own, so it carries the chain's tenant predicate onto it and refuses when it cannot. -The two filtered writes are not conveniences. `delete(id)` and `update(id, patch)` both need a single-column primary key — `singleKeyOf` throws `X_INVARIANT_VIOLATED` on a composite one — so on a join table (`likes`, `blocks`, `participants`) they are the only write paths there are. Without them a composite-key row is create-only: `likes` could be liked and never unliked, and `participants.lastReadAt` could never be marked read. Four properties make them safe to be the only filtered writes: +That family collapses one *lookup* repeated per row. The other repeated statement is an aggregate — one `count()` per row, the shape `recountLikes` had — and no batch reaches it, because each of those statements asks a different question. `countBy(column)` asks all of them at once: + +```ts +// One statement for every post in `ids`, not one `select count(*)` each. +const counts = await db.likes.where({ orgId }).andWhere('postId', 'in', ids).countBy('postId'); +for (const id of ids) await db.posts.update(id, { likeCount: counts.get(id) ?? 0 }); +``` + +| Rule | Mechanism | +|---|---| +| Counts the predicate, never the page | the plan `count(args)` builds, so the filters, the tenant predicate and `deleted_at is null` are in the statement Postgres runs and `limit`/`after` bound nothing | +| A value nothing matched is absent, never `0` | what `group by` returns, and the only way a caller can tell "none" from "never asked". The `?? 0` is the caller's | +| NULL is one group, keyed `null` | the memory driver reads the property as `?? null`, so it lands where Postgres puts its NULL rows. `0`, `''` and `false` stay the values they are | +| Ordered biggest group first | ties by the value, `null` last — applied in [`count-by.ts`](../../packages/entity/src/count-by.ts) *after* the rows are in, because a hash aggregate returns groups in whatever order it built them and a `Map` filled row by row returns insertion order, so an `order by` in the statement would let the two drivers disagree about a result they agree on | +| Groupable columns are a closed set | `uuid`, `text`, `char`, `boolean`, `integer`, `bigint`. A `timestamptz` is a `Date`, a `jsonb` is an object, `money` is two columns — a `Map` compares those by identity, so the map could only ever answer `undefined`. `X_INVARIANT_VIOLATED` naming a column of that entity that is groupable | +| The group count is bounded, and the bound is a refusal | `MAX_GROUPS` (1000). The statement asks for one group past it, exactly as a page reads one row past its limit; the extra group is `X_INVARIANT_VIOLATED` spelling `andWhere('', 'in', )`. A truncated map reads exactly like a complete one, and a caller recounting from it writes the wrong number to every row it missed | +| One statement, fixed output names | `select "post_id" as group_value, count(*) as group_count … group by "post_id" limit $n` ([`pg-sql.ts`](../../packages/entity/src/pg-sql.ts)) — aliased so an entity may still declare a column called `count`, and the grouped value is re-parsed by the column that declared it, since `int8` arrives as a string | + +`upsertAll` refuses four things before a statement exists, each of them otherwise a `42P10`, a cross-tenant write, a `21000` or a silent surprise: + +| Refusal | Why it is not the server's job | +|---|---| +| A conflict target no declared unique constraint matches | Postgres answers `42P10`, which arrives as `X_DB_UNAVAILABLE` and names nothing to edit. The entity already holds `$primaryKey`, `$indexes` and `$invariants`, so the target is checkable at the seam. All three spellings of one constraint count — the key, `unique()`/`indexes:`, and `invariant(name, c.unique([…]))` — or the refusal would ask for a second declaration of a constraint that exists. A partial unique index is not a target on any of them, since `on conflict` would have to repeat its predicate | +| A target omitting the tenant column under `onMatch: 'update'` | `X_TENANCY_UNSCOPED`. `upsertAll` builds no read plan, so nothing else puts an org predicate in the statement: a target that omits the tenant column matches a row stored by another tenant and rewrites it, tenant column included. `'nothing'` stays legal — it writes nothing to a row it does not own | +| A batch repeating one conflict target under `'update'` | `ON CONFLICT DO UPDATE command cannot affect row a second time` (`21000`) on the server, and a silent last-one-wins in memory. The two drivers have to mean one thing | +| An uneven batch under `'update'` | `excluded.` for a row that omitted the column is that column's *default*, not the stored value, so "leave the others alone" is not what runs. `insertAll` and `'nothing'` accept one and render `default` in the missing cell, which is what the same row means on its own | + +A collision overwrites every column the batch writes minus three closed sets — the conflict target, which is how the row was found, the primary key, which is where it lives, and the soft-delete stamp, which is whether the row is there at all. A soft-deleted row still occupies its conflict target, so `excluded."deleted_at"` would clear a delete the app made; excluded from the set list rather than refused, since `$parse` fills that `deletedAt: null` in before the plan is built. A null anywhere in the target collides with nothing, in both drivers, because a Postgres unique index is `NULLS DISTINCT`. + +The two filtered writes are not conveniences. `delete(id)` and `update(id, patch)` both need a single-column primary key — `singleKeyOf` throws `X_INVARIANT_VIOLATED` on a composite one — so on a join table (`likes`, `blocks`, `participants`) they are the only write paths there are. Without them a composite-key row is create-only: `likes` could be liked and never unliked, and `participants.lastReadAt` could never be marked read. They double as the bulk forms of `delete`/`update` in the ordinary case, too — one statement for a loop that would otherwise delete or patch a row at a time, the same role `insertAll`/`upsertAll` play for a per-row insert loop. Four properties make them safe to be the only filtered writes: | Property | Mechanism | |---|---| diff --git a/packages/entity/CLAUDE.md b/packages/entity/CLAUDE.md index 6cc89fa93..b38c197d2 100644 --- a/packages/entity/CLAUDE.md +++ b/packages/entity/CLAUDE.md @@ -20,7 +20,14 @@ Columns + invariants; the row type is derived from the columns. Tier 2. sort order, page size), `cursor.ts` (one codec, values included) and the `Repo` contract, so a test that passes against memory says something about Postgres. A guard, an operator or a sort rule added to one and not the other is the bug this split exists to prevent — `pg-driver.test.ts` - pins the parity. + pins the parity, and every bulk method added since carries the same two files: a + `*-parity.test.ts` seeding identical rows into both drivers and asserting identical output + (`batch-parity.test.ts`, `preload-parity.test.ts`, `count-by-parity.test.ts`, and the + `insertAll`/`upsertAll` cross-driver assertions inside `pg-driver-bulk.test.ts`), and a + `pg-driver-.live.test.ts` proving the same call against a real server + (`pg-driver-batch.live.test.ts`, `pg-driver-preload.live.test.ts`, `pg-driver-count.live.test.ts`, + `pg-driver-bulk.live.test.ts`). A method with only the first is unproven against Postgres itself; + a method with only the second is unproven against memory. Both are the bar, not either one. - **The Postgres driver is proved against a real Postgres, not only against a recording client.** `pg-driver.live.test.ts` runs the whole chain — `entity()` -> `$describe()` -> `generateMigration()` -> a live server -> `postgresDriver()` -> decoded row — and skips when no @@ -96,12 +103,63 @@ Columns + invariants; the row type is derived from the columns. Tier 2. mutates**: a preloaded relation is written onto `{ ...row }`, because the in-memory driver hands back the row it stores and attaching directly would leak the relation into the table itself. **Preloading terminals only**: `page()`, `all()` and `one()` resolve every named - relation; `count()` and `plan()` do not, since neither reads a row to attach one to. + relation; `count()`, `countBy()` and `plan()` do not, since none reads a row to attach one to. - **Cursor pagination only.** OFFSET is wrong under concurrent writes: an insert before the offset shifts every later page, so a client silently skips and repeats rows. No `offset` on `FindManyArgs` or the builder; the primary key is always the last sort key, so the order is total. The cursor carries the sort **values**, not just an id — seeking by an id that was deleted between two requests would restart pagination at the top. +- **`inBatches(size)` is that same page in a loop, and the loop owns it.** `batch.ts` holds no + driver of its own: a batch is the `findMany` the chain would have sent at that position, so + filters, tenancy, soft delete, the projection and every `preload()` mean there what they mean in + `page()` and there is no second read path to drift. Properties, none optional. **The handle is + the iteration**: it is its own iterator, so `break`, `return`, a throw and `await using` all stop + the *next* statement — `close()` is `AsyncGenerator.return()` and therefore idempotent by + construction, never a flag two paths could disagree about — and a second `for await` continues it + instead of re-reading the table from the top. **The position is readable**: `.cursor` is where + the next batch starts, advanced *before* the yield, so a consumer that breaks reads the position + it stopped at and `.after(cursor).inBatches(size)` resumes it; stopping early is then cheap + rather than wasted. **An empty batch is never yielded** — a consumer forced to check + `batch.length` is reading around the iterator. **Three refusals, all on the chain**: a size that + is not a whole number of rows ≥ 1, a chain that also called `limit()` (one number, two meanings — + honouring it reads a fraction of a batch, dropping it reads the whole table the caller thought + they had bounded), and an ordering no cursor can carry. That last one is why + `totalOrder(entity, orderBy)` is exported from `plan.ts` rather than inlined in `planFor`: the + guard has to judge the order the driver will *send*, primary key included, and a result that fits + in one batch mints no cursor — so a nullable sort key would otherwise pass in every test and fail + once the table grew. `State.limit` is `number | undefined` for the same reason: only "the caller + named a page size" can be told apart from the default, which the driver already applies. +- **A grouped count means one thing in both drivers, and `count-by.ts` is where that one thing is + written.** `countBy(column)` is the aggregate a `count()` per row is the N+1 of, so both drivers + call `groupColumnOf` before their statement exists and `countsFrom` after their rows are in — a + rule added to `pg-driver.ts` or to `repo.ts` alone is exactly the drift that file exists to + prevent. **Groupable kinds are a closed set**: `uuid`, `text`, `char`, `boolean`, `integer`, + `bigint`. A `timestamptz` is a `Date`, a `jsonb` is an object and `money` is two physical columns + — a `Map` compares a non-primitive key by identity, so any of those would file rows under a key + no caller can look up again and the result would be a map that only ever answers `undefined`. The + refusal is `X_INVARIANT_VIOLATED` naming a column of *this* entity that is groupable, never + `x entity explain`: what repairs it is one edit to the call, and the entity is the only place the + replacement column lives. **The bound is a refusal, not a truncation.** The statement asks for + `MAX_GROUPS + 1` groups — the trick a page already uses when it reads one row past its limit — and + that extra group is what says the answer was never going to fit, so `countsFrom` throws with the + `andWhere(…, 'in', )` that bounds it. Truncating would hand back a map that reads exactly + like a complete one, and a caller recounting from it would write the wrong number to every row it + missed. **Absent is not `0`**: a value nothing matched has no entry, because that is what + `group by` returns and it is the only way a caller can tell "none" from "never asked" — the + `?? 0` is theirs to write, and inventing it here would answer for keys the table has never seen. + **NULL is one group**, keyed `null`: the memory driver reads the property as `?? null` so it lands + where Postgres puts its NULL rows, while `0`, `''` and `false` stay the values they are. **The + order is applied after the rows are in, never in SQL** — a hash aggregate returns groups in + whatever order it built them and a `Map` filled row by row returns insertion order, so an + `order by` in the statement would let the two drivers disagree about a result they agree on; + sorting groups (never rows) costs nothing at this size and is what puts the largest bucket at the + front. **Both output names are fixed aliases** — `group_value` and `group_count` in + `countByStatement` (`pg-sql.ts`) — because an entity is free to declare a column called `count`, + and the un-aliased form would then return two outputs of one name; the grouped value is re-parsed + by the column that declared it, since `int8` arrives as a string and would otherwise key the map + by text where memory keys it by a `bigint`. **Nothing new to declare**: no `groupBy()` builder and + no error code of its own — it is a terminal on the chain that already exists, over exactly the + rows `count()` counts. - **The codec is `@ultimat3/core`'s, and both drivers reach it through exactly two functions**: `cursorFor(entity, plan, row, id)` and `seekFrom(entity, plan)` in `cursor.ts`. Both call `assertSeekable`, so an ordering that cannot carry a position — a nullable key, an undeclared @@ -146,7 +204,11 @@ Columns + invariants; the row type is derived from the columns. Tier 2. are bounded by construction.** `delete(id)` and `update(id, patch)` need a single-column primary key, so on a composite key — `likes`, `blocks`, `participants`, any join table — the filtered pair is the only write path that exists; without them the entity is create-only and a row can be - written and never unwritten. Properties, none of them optional: + written and never unwritten. They are also the bulk forms of `delete`/`update` for the ordinary + case — one statement for a `for … of` loop that would otherwise delete or patch one row at a + time — the same role `insertAll`/`upsertAll` (below) play for a per-row insert loop; a + write-loop detector's `fix:` names one of these four, never a hand-rolled loop. Properties, none + of them optional: - an empty filter is `X_WRITE_UNFILTERED` and never every row; an empty patch is `X_PATCH_EMPTY` and never a counted no-op. An `undefined` value is dropped *before* either count, so a forgotten variable lands on the error rather than on the table. @@ -166,9 +228,58 @@ Columns + invariants; the row type is derived from the columns. Tier 2. ends up writing a stale `updatedAt`. It returns an empty patch untouched, so whether `X_PATCH_EMPTY` fires depends on the call and not on whether the entity happens to declare the column. +- **A many-row write is one statement, and every refusal it needs happens before that statement + exists.** `insertStatement` (`pg-sql.ts`) builds *every* insert in the framework — one row or ten + thousand — so `insertAll([row])` compiles to exactly the text `insert(row)` always compiled to + and there is no second builder for the two to drift apart in. What both drivers have to agree on + lives in `bulk-write.ts`, decided in **property** space and projected to physical columns for the + SQL: the column list a batch writes (`Object.hasOwn`, exactly as `bindValues` decides it), what a + collision overwrites, the conflict key, and the chunking. Rules, none optional. **A collision + overwrites every column in the batch except three closed sets** — the conflict target, which is + how the stored row was found, the primary key, which is where it lives, and the soft-delete + stamp, which is whether the row is there at all; an upsert that moved either of the first two + would move a row nobody asked to move and every foreign key already pointing at that id would + miss it. **The stamp is the third because a soft-deleted row still occupies its conflict target** + — the index it collides with is not partial — so `excluded."deleted_at"` would clear a delete the + app made and hand the row back holding the batch's values, which is the resurrection + `update(id, patch)` and `updateWhere` refuse by carrying `deleted_at is null` and an + `on conflict` clause cannot carry. Excluded from the set list rather than refused, because + `$parse` fills every declared column before a row reaches `upsertPlan`: that `deletedAt: null` is + the framework's and not the caller's, so refusing it would make `onMatch: 'update'` impossible on + every soft-deleting entity. `insertAll` is untouched — a row colliding with nothing writes the + stamp it carries, exactly as `insert` does. **The conflict target must be a declared unique + constraint** — because a target + Postgres cannot infer an index for is `42P10` wrapped as `X_DB_UNAVAILABLE`, which names nothing + the author can act on. All **three** of this framework's spellings of one count, or the refusal + would tell an author to declare a constraint they already declared and ship two indexes: the + primary key, a non-partial `unique: true` entry in `$indexes` (`unique()` on a column and + `indexes:` both land there), and a `kind: 'unique'` entry in `$invariants` + (`invariant(name, c.unique([…]))`, whose `CREATE UNIQUE INDEX` never touches `$indexes`). A + partial one is deliberately not a target on either list, since its predicate would have to be + repeated in the `on conflict` clause and this layer does not spell one — which is also why a + soft-deleting entity's `c.unique()` invariant, stamped `deleted_at is null` by `bindInvariant`, + is excluded by that same rule. **The tenant column is part of that + constraint or `onMatch: 'update'` is refused** (`X_TENANCY_UNSCOPED`) — this is a security + boundary, not ergonomics: `upsertAll` builds no read plan, so nothing else puts an org predicate + in the statement, and a target that omits the tenant column matches a row stored by another tenant + and rewrites it, tenant column included. `'nothing'` stays legal on such a target because it + writes nothing to a row it does not own. **A batch that repeats one conflict target is refused + under `'update'`** — Postgres answers that statement `ON CONFLICT DO UPDATE command cannot affect + row a second time`, so passing it in memory and failing in production is the exact drift the two + drivers exist to prevent — and **an uneven batch is refused under `'update'`** for the same + reason: `excluded.` for a row that omitted it is that column's *default*, not the stored + value, so "leave it alone" is not what happens. `insertAll` and `'nothing'` accept an uneven batch + and render `default` in the missing cell, which is what the same row means on its own. + **Null is not a value here**: a null anywhere in the conflict target means the row collides with + nothing, in both drivers, because a Postgres unique index is `NULLS DISTINCT`. **The memory + driver judges the whole batch before storing any of it** — `$assert` over every row first — since + Postgres refuses the statement as one and a half-applied batch would make the two disagree about + what one call did. Past `MAX_BIND_PARAMETERS` (65535) the batch is several whole statements, so + atomicity across them is `withTransaction`'s and never one statement's. - **Nothing is interpolated into SQL.** `pg-sql.ts` binds every value through `sql` and resolves every identifier through the entity, so a column name can only be one the entity declared. - `raw()` appears exactly twice, for `asc|desc` and the seek operator — both closed sets. + `raw()` appears exactly three times, for `asc|desc`, the seek operator and the `default` cell of a + many-row `values` list — each a closed set of one word. - **Money is `bigint` minor units + `char(3)` currency.** A float throws. Never one column, never an implied single currency. - **Timestamps are `timestamptz`.** A naive timestamp must stay inexpressible. @@ -223,9 +334,12 @@ Columns + invariants; the row type is derived from the columns. Tier 2. | `query.ts` / `database.ts` | chainable read to a cursor page; `database()` + `Driver` | | `repo.ts` / `tenancy.ts` | `Repo` + `memoryDriver`'s repo, tx rollback; `QueryPlan` + `assertScoped()` | | `plan.ts` / `cursor.ts` | the plan both drivers execute; the one keyset cursor codec | +| `batch.ts` | `inBatches(size)` — the chain's page in a loop, closed by the loop that reads it | | `pg-driver.ts` | `postgresDriver()`, `postgresRepo()`, `postgresTransactor()` | | `coalesce.ts` | one microtask of `findById` calls → one `where id in (…)`, per request | | `batch-read.ts` | what a shared point read is made of — the scope key, `keyOf`, the one `in` statement | +| `bulk-write.ts` | what a many-row write is made of — the column list, the conflict plan, the bind-count chunking | +| `count-by.ts` | what a grouped count is made of — the groupable kinds, the group bound, the key's decoding, the order | | `jit-preload.ts` | a page's foreign key values → one `in` statement for the whole `for … of` loop | | `preload.ts` | the relation `preload()` names → one related-rows statement → attached to the page | | `pg-sql.ts` / `pg-row.ts` | plan → parameterised SQL; physical row ⇄ entity row (money is two columns) | diff --git a/packages/entity/README.md b/packages/entity/README.md index 394891bdc..236cf5706 100644 --- a/packages/entity/README.md +++ b/packages/entity/README.md @@ -118,8 +118,69 @@ these filters, this sort order. A tampered cursor, or one taken from another lis `X_CURSOR_INVALID` rather than a silent page one. The page size is deliberately outside the scope: asking for a bigger next page is the same query. +## Iterating every row + +`As of 2026-08`. A page is bounded on purpose, so reading a whole table is a loop — and the loop is +the terminal, not something the caller writes around `page()`: + +```ts +// One statement per batch, one page of rows in memory at a time. +for await (const batch of db.posts.where({ orgId }).preload('author').inBatches(500)) { + await search.index(batch); +} + +// Stopping early is cheap: the position survives, so the next run resumes where this one stopped. +await using batches = db.posts.where({ orgId }).after(checkpoint).inBatches(500); +for await (const batch of batches) { + await search.index(batch); + if (ctx.clock.now() > deadline) break; +} +await db.checkpoints.update(id, { cursor: batches.cursor }); +``` + +Every batch is the page `page()` would have returned at that position — same filters, same tenancy, +same soft-delete visibility, same `select()`, same `preload()` — so there is no second read path to +learn or to drift. + +| | | +|---|---| +| Statements | one per batch, each asking for one row past it, exactly as `page()` does. An empty batch is never yielded | +| Position | keyset, never OFFSET: a row written mid-iteration cannot make the loop skip or repeat one. `after(cursor)` starts it, `.cursor` is where it stopped, `null` once exhausted | +| Closing | `break`, `return` and a throw all stop the next statement; `await using` is the same guarantee for a handle kept in a variable, and `close()` is idempotent. One handle is one iteration — a second `for await` continues it rather than restarting the table | +| Refusals | on the chain, not one batch later: a size that is not a whole number of rows, a `limit()` on the same chain (one number, two meanings), and an ordering no cursor can carry — a nullable sort column, which a result that fits in one batch would otherwise hide until the table grew | +| Tenancy | the plan's, as everywhere else: an unscoped chain is `X_TENANCY_UNSCOPED` on its first batch | + +## Counting by a column + +`As of 2026-08`. `count()` answers one number, so a screen or a backfill that needs one per row +asks N times. `countBy(column)` is that whole loop as one statement, keyed by the value: + +```ts +// One statement for every post in `ids`, not one `select count(*)` each. +const counts = await db.likes.where({ orgId }).andWhere('postId', 'in', ids).countBy('postId'); +for (const id of ids) await db.posts.update(id, { likeCount: counts.get(id) ?? 0 }); +``` + +`ReadonlyMap`, keyed by the column named — the chain knows the row, so +`counts.get(postId)` is a `number | undefined` and the `undefined` is load-bearing. + +| | | +|---|---| +| Counts | the whole predicate, exactly as `count()` does: the chain's filters, its tenancy and its soft-delete visibility. `limit()` and `after()` bound the page, never the count | +| A value nothing matched | absent, never `0` — that is what `group by` returns, and it is what tells "none" apart from "never asked". The default is the caller's `?? 0` | +| NULL | one group, keyed `null`, in both drivers. `0`, `''` and `false` stay the values they are | +| Order | biggest group first, ties by the value (numbers and bigints numerically, everything else by its text), `null` last — applied after the rows are in, since a hash aggregate and a `Map` filled row by row have no order to inherit | +| Groupable columns | `uuid`, `text`, `char`, `boolean`, `integer`, `bigint`. A timestamp, a `jsonb` or `money` is `X_INVARIANT_VIOLATED` naming one of this entity's columns that is: a `Map` compares a non-primitive key by identity, so such a map could only ever answer `undefined` | +| More than 1000 groups | `X_INVARIANT_VIOLATED`, never a truncated map — the statement asks for one group past the bound, exactly as a page reads one row past its limit. The `fix` spells the `andWhere('', 'in', )` that bounds it | +| Statement | `select "post_id" as group_value, count(*) as group_count … group by "post_id"`. Both names are fixed aliases, so an entity may still declare a column called `count`, and the grouped value is re-parsed by the column that declared it | + ## Writing by filter +`deleteWhere`/`updateWhere` are the bulk forms of `delete`/`update` — the same fix `insertAll` is +for a per-row insert loop, applied to the two write shapes a composite-key entity cannot address +one row at a time: a `for … of` deleting or patching one row per iteration is one statement here, +not `n`. + ```ts db.posts.delete(id); // by a single primary key db.posts.update(id, { title }); // ditto @@ -146,6 +207,42 @@ and never unwritten. | Soft delete | the entity's `deletedAt` column is the same switch `delete(id)` uses. Stamped rows are not matched again by either call, so the original deletion time survives and a deleted row is never patched back into shape | | `onUpdateNow()` | stamped by `touch()`, the same helper `update(id, patch)` uses — one place, so the two can never disagree about `updatedAt` | +## Writing many rows + +```ts +await db.tags.insertAll(names.map((name) => ({ orgId, name }))); // one statement, n rows + +await db.likes.upsertAll(rows, { // insert, or leave what is there + onConflict: ['orgId', 'postId', 'memberId'], + onMatch: 'nothing', +}); + +await db.counters.upsertAll(rows, { onConflict: ['orgId', 'day'] }); // insert, or overwrite +``` + +`insertAll` is `insert` in bulk and nothing else: rows are `Insertable`, each one goes through +`$parse`, so declared defaults are filled here rather than by the caller. `upsertAll` adds the one +thing a per-row loop cannot do without a read first — resolve a collision — and stamps +`onUpdateNow()` columns through the same `touch()` `update(id, patch)` uses, because an upsert that +lands on a stored row *is* an update. + +Both resolve with **the rows this call wrote**, in order. Under `onMatch: 'nothing'` a row already +stored is skipped and absent from the result, exactly as `returning *` reports it — which is how a +caller counts what it actually inserted. + +| | | +|---|---| +| One builder | `insertStatement` compiles every insert in the framework, so `insertAll([row])` is the text `insert(row)` always produced. There is no second insert path to drift | +| What a collision overwrites | every column the batch writes, minus the conflict target (how the row was found), minus the primary key (where it lives) and minus the soft-delete stamp (whether the row is there at all). Moving either of the first two moves a row nobody asked to move, and every foreign key pointing at that id misses it | +| A soft-deleted row it lands on | stays deleted, and takes the batch's other columns. The stamped row still occupies its conflict target — that index is not partial — so `excluded."deleted_at"` would resurrect it; `$parse` fills `deletedAt: null` into every row before the plan is built, so the stamp is dropped from the set list rather than refused. `insertAll` still writes the stamp a new row carries | +| Conflict target | properties of a **declared** unique constraint — the primary key, a `unique()` column, an `indexes: [{ on, unique: true }]` entry, or an `invariant(name, c.unique([…]))`. Anything else is `X_INVARIANT_VIOLATED` here rather than `42P10` from the server | +| Tenancy | on a tenant-scoped entity `onMatch: 'update'` requires the tenant column *in the conflict target*, else `X_TENANCY_UNSCOPED`: a target that omits it matches another tenant's row and rewrites it. `'nothing'` is allowed — it writes nothing to a row it does not own | +| A batch that repeats itself | two rows with one conflict target under `'update'` is refused. Postgres answers that statement `ON CONFLICT DO UPDATE command cannot affect row a second time`, so it cannot pass in memory either | +| Uneven batches | under `'update'` every row must name the same columns: `excluded.` for a row that omitted one is that column's *default*, not "leave it alone". Under `'nothing'` and under `insertAll`, an omitted column is `default` in its cell, which is what the same row means on its own | +| Nulls | a null in the conflict target collides with nothing, in both drivers — a Postgres unique index is `NULLS DISTINCT` | +| Size | past 65535 bind parameters the batch is several statements, never one the server refuses. Wrap the call in `withTransaction` when all-or-nothing matters | +| Filtered writes | `updateWhere` / `deleteWhere` above are the bulk forms of `update` and `delete` — one statement for a loop that would otherwise call either per row; there is no `updateAll` | + ## Relations are the foreign keys, read twice ```ts @@ -206,7 +303,7 @@ page.rows[0].author; // the member row, or null — always present | Shape | `belongsTo` attaches the row or `null`; `hasMany` an array — always present | | Statements | one extra per relation, resolved concurrently; naming one twice is one statement | | Tenancy | carried onto the related read only when the other entity's tenant column shares the name; otherwise `X_TENANCY_UNSCOPED` refuses the related read rather than guess | -| Terminals | `page()`, `all()`, `one()` preload; `count()` and `plan()` don't — neither reads a row | +| Terminals | `page()`, `all()`, `one()` preload; `count()`, `countBy()` and `plan()` don't — none reads a row to attach one to | ## Two drivers, one meaning diff --git a/packages/entity/src/batch-parity.test.ts b/packages/entity/src/batch-parity.test.ts new file mode 100644 index 000000000..0d4c0d46e --- /dev/null +++ b/packages/entity/src/batch-parity.test.ts @@ -0,0 +1,175 @@ +// The memory-vs-Postgres cross-check for `inBatches(size)`. `batch.test.ts` proves memory's own +// boundaries and, separately, the statement text the Postgres driver sends for one page; neither +// proves the two drivers cut the *same* seed into the *same* batches. This file seeds one row set +// into both and walks them side by side, so a boundary rule added to one driver and not the other +// shows up here first. + +import { afterAll, beforeEach, describe, expect, test } from 'bun:test'; +import { createRecordingClient, type RecordingClient } from '@ultimat3/db'; +import type { BatchIterator } from './batch'; +import { text, uuid } from './columns'; +import { entity } from './entity'; +import { postgresRepo } from './pg-driver'; +import { tableFor } from './query'; +import { clearRegistry } from './registry'; +import { memoryRepo } from './repo'; + +const orgs = entity('batch_parity_orgs', { + columns: { id: uuid().primaryKey(), name: text({ max: 40 }) }, +}); + +const posts = entity('batch_parity_posts', { + columns: { + id: uuid().primaryKey(), + orgId: uuid() + .references(() => orgs.id) + .tenant(), + title: text({ max: 80 }), + }, +}); + +type Post = typeof posts.$row; + +const ORG = '00000000-0000-7000-8000-0000000000b1'; + +const id = (index: number): string => + `00000000-0000-7000-8000-0000000002${String(index).padStart(2, '0')}`; + +const post = (index: number): Post => ({ + id: id(index), + orgId: ORG, + title: `Post ${index}`, +}); + +/** Six rows: divisible by 2 with nothing left over, and not divisible by 4 — one seed, two shapes. */ +const SEED: readonly Post[] = Array.from({ length: 6 }, (_, index) => post(index)); + +/** Physical-cased, the same shape `batch.test.ts`'s own Postgres `describe` block builds by hand. */ +const physical = (row: Post): Record => ({ + id: row.id, + org_id: row.orgId, + title: row.title, +}); + +/** + * The windows a real server would answer, call by call: each one asks for `size + 1` rows past the + * last one served — the peek `pg-driver.ts` reads to decide whether a next cursor exists — and only + * `size` of them are ever handed to a caller. Precomputing every window from the seed is what lets a + * canned client answer a live walk without knowing SQL: the driver is real, only the network is not. + */ +const windowsFor = (rows: readonly Post[], size: number): readonly Post[][] => { + const windows: Post[][] = []; + for (let index = 0; index < rows.length; index += size) { + windows.push(rows.slice(index, index + size + 1)); + } + return windows; +}; + +let client: RecordingClient; +let pg: ReturnType>; +let mem: ReturnType>; + +beforeEach(() => { + client = createRecordingClient(); + pg = tableFor(posts, postgresRepo(posts, { client })); + mem = tableFor(posts, memoryRepo(posts, SEED)); +}); + +afterAll(() => { + clearRegistry(); +}); + +/** + * Drains the Postgres side against precomputed windows. `RecordingClient.on()` is "whatever + * matches wins from here on", not a queue, so the next window is armed only once the generator is + * about to ask for it — the same rhythm `batch.test.ts`'s own Postgres `describe` block uses. + */ +const drainPg = async ( + windows: readonly Post[][], + batches: BatchIterator, +): Promise => { + let index = 0; + client.on('select', { rows: (windows[index] ?? []).map(physical) }); + const seen: Post[][] = []; + for await (const batch of batches) { + seen.push([...batch]); + index += 1; + const next = windows[index]; + if (next !== undefined) client.on('select', { rows: next.map(physical) }); + } + return seen; +}; + +const drainMem = async (batches: BatchIterator): Promise => { + const seen: Post[][] = []; + for await (const batch of batches) seen.push([...batch]); + return seen; +}; + +const idsOf = (batches: readonly Post[][]): readonly string[] => + batches.flat().map((row) => row.id); +const sizesOf = (batches: readonly Post[][]): readonly number[] => + batches.map((batch) => batch.length); + +describe('memory and Postgres draw batch boundaries at the same place', () => { + test.each([ + ['divides evenly', 2, [2, 2, 2]], + ['does not divide evenly', 4, [4, 2]], + ] as const)( + 'size %2$p (%1$s): identical sizes and identical row order', + async (_label, size, expectedSizes) => { + const windows = windowsFor(SEED, size); + + const fromMemory = await drainMem(mem.where({ orgId: ORG }).inBatches(size)); + const fromPostgres = await drainPg(windows, pg.where({ orgId: ORG }).inBatches(size)); + + expect(sizesOf(fromMemory)).toEqual([...expectedSizes]); + expect(sizesOf(fromPostgres)).toEqual(sizesOf(fromMemory)); + expect(idsOf(fromPostgres)).toEqual(idsOf(fromMemory)); + expect(idsOf(fromMemory)).toEqual(SEED.map((row) => row.id)); + }, + ); + + test('the cursor after a partial walk resumes to the same row in both drivers', async () => { + const size = 2; + const windows = windowsFor(SEED, size); + + const memBatches = mem.where({ orgId: ORG }).inBatches(size); + const firstMem: Post[] = []; + for await (const batch of memBatches) { + firstMem.push(...batch); + break; + } + + let index = 0; + client.on('select', { rows: (windows[index] ?? []).map(physical) }); + const pgBatches = pg.where({ orgId: ORG }).inBatches(size); + const firstPg: Post[] = []; + for await (const batch of pgBatches) { + firstPg.push(...batch); + index += 1; + const next = windows[index]; + if (next !== undefined) client.on('select', { rows: next.map(physical) }); + break; + } + + // Both codecs are `cursor.ts`'s one implementation, so a cursor minted from the same plan and + // the same last row is not merely equivalent — it is the same string. + expect(pgBatches.cursor).toEqual(memBatches.cursor); + expect(firstPg.map((row) => row.id)).toEqual(firstMem.map((row) => row.id)); + + const restMem = await drainMem( + mem.where({ orgId: ORG }).after(memBatches.cursor).inBatches(size), + ); + const restPg = await drainPg( + windows.slice(index), + pg.where({ orgId: ORG }).after(pgBatches.cursor).inBatches(size), + ); + + expect(idsOf(restPg)).toEqual(idsOf(restMem)); + // Nothing repeated, nothing skipped: the two halves add back up to the whole seed, in order. + expect([...firstMem.map((row) => row.id), ...idsOf(restMem)]).toEqual( + SEED.map((row) => row.id), + ); + }); +}); diff --git a/packages/entity/src/batch.test.ts b/packages/entity/src/batch.test.ts new file mode 100644 index 000000000..1c9749b19 --- /dev/null +++ b/packages/entity/src/batch.test.ts @@ -0,0 +1,383 @@ +// `inBatches(size)` is the only read that hands back a resource rather than rows, so this pins +// three things a page terminal cannot go wrong at: that the batches are the chain's own pages and +// nothing else, that stopping early stops the statements and says where it stopped, and that a +// chain which cannot carry a cursor is refused before the first one goes out rather than after it. + +import { afterAll, beforeEach, describe, expect, test } from 'bun:test'; +import { createRecordingClient, type RecordingClient } from '@ultimat3/db'; +import type { BatchIterator } from './batch'; +import { text, timestamp, uuid } from './columns'; +import { database } from './database'; +import { entity } from './entity'; +import { EntityError } from './errors'; +import { postgresRepo } from './pg-driver'; +import { tableFor } from './query'; +import { clearRegistry } from './registry'; +import { memoryRepo, type Repo } from './repo'; + +const orgs = entity('batch_test_orgs', { + columns: { id: uuid().primaryKey(), name: text({ max: 40 }) }, +}); + +const posts = entity('batch_test_posts', { + columns: { + id: uuid().primaryKey(), + orgId: uuid() + .references(() => orgs.id) + .tenant(), + title: text({ max: 80 }), + /** Nullable on purpose: an ordering no cursor can carry has to be expressible to be refused. */ + publishedAt: timestamp().nullable(), + }, +}); + +type Post = typeof posts.$row; + +const ORG = '00000000-0000-7000-8000-0000000000a1'; +const OTHER = '00000000-0000-7000-8000-0000000000a2'; + +const id = (index: number): string => + `00000000-0000-7000-8000-0000000001${String(index).padStart(2, '0')}`; + +const post = (index: number, orgId: string = ORG): Post => ({ + id: id(index), + orgId, + title: `Post ${index}`, + publishedAt: null, +}); + +/** Ids in tens, so a row written mid-iteration can land between two the loop already read. */ +const ids = (...indexes: readonly number[]): readonly string[] => indexes.map(id); +const SEEDED = [0, 10, 20, 30, 40, 50, 60] as const; + +/** Seven rows for the org under test, two more nobody iterating it may ever see. */ +const seed: readonly Post[] = [ + ...SEEDED.map((index) => post(index)), + post(90, OTHER), + post(91, OTHER), +]; + +/** A repo that counts its reads: "one statement per batch" is a claim about calls, not about rows. */ +interface Counted { + readonly repo: Repo; + reads(): number; +} + +const counting = (inner: Repo): Counted => { + let reads = 0; + return { + repo: { + ...inner, + findMany: (args) => { + reads += 1; + return inner.findMany(args); + }, + }, + reads: () => reads, + }; +}; + +let table = tableFor(posts, memoryRepo(posts, seed)); + +beforeEach(() => { + table = tableFor(posts, memoryRepo(posts, seed)); +}); + +afterAll(() => { + clearRegistry(); +}); + +const feed = () => table.where({ orgId: ORG }); + +const drain = async (batches: BatchIterator): Promise => { + const seen: (readonly Post[])[] = []; + for await (const batch of batches) seen.push(batch); + return seen; +}; + +/** `toBeUltimateError` reads a value, and a chain-shaped refusal throws where it was written. */ +const caught = (run: () => unknown): EntityError => { + try { + run(); + } catch (error) { + if (error instanceof EntityError) return error; + throw error; + } + throw new Error('expected a refusal, got a builder'); +}; + +describe('the batches a chain yields', () => { + test('every matching row, in order, size at a time', async () => { + const batches = await drain(feed().inBatches(3)); + + expect(batches.map((batch) => batch.length)).toEqual([3, 3, 1]); + expect(batches.flat().map((row) => row.id)).toEqual(ids(...SEEDED)); + }); + + test('an empty batch is never yielded', async () => { + expect(await drain(feed().andWhere('title', 'eq', 'nothing').inBatches(3))).toEqual([]); + // Exactly divisible: the page after the last row is where a trailing empty batch would appear. + expect((await drain(feed().andWhere('title', 'like', 'Post %').inBatches(7))).length).toBe(1); + }); + + test('a batch bigger than the result is one batch, not a first page', async () => { + const batches = await drain(feed().inBatches(500)); + + expect(batches).toHaveLength(1); + expect(batches[0]).toHaveLength(7); + }); + + test('the filter, the order and the projection are the chain’s', async () => { + const batches = await drain( + feed() + .andWhere('title', 'like', 'Post %') + .orderBy('title', 'desc') + .select({ id: true }) + .inBatches(4), + ); + const rows = batches.flat(); + + expect(rows[0]).toEqual({ id: id(60) }); + expect(rows.map((row) => row.id)).toEqual(ids(...[...SEEDED].reverse())); + }); + + test('another tenant’s rows are outside every batch', async () => { + const rows = (await drain(feed().inBatches(2))).flat(); + + expect(rows).toHaveLength(7); + expect(rows.some((row) => row.orgId === OTHER)).toBe(false); + }); + + test('a preloaded relation is attached to every batch, not only the first', async () => { + const db = database({ orgs, posts }); + await db.orgs.insert({ id: ORG, name: 'Acme' }); + for (const row of seed.filter((candidate) => candidate.orgId === ORG)) { + await db.posts.insert(row); + } + + const batches = await drain(db.posts.where({ orgId: ORG }).preload('org').inBatches(3)); + + expect(batches).toHaveLength(3); + for (const batch of batches) { + for (const row of batch) expect(row.org).toEqual({ id: ORG, name: 'Acme' }); + } + }); +}); + +describe('one statement per batch, and none once it is closed', () => { + let counted: Counted; + + beforeEach(() => { + counted = counting(memoryRepo(posts, seed)); + table = tableFor(posts, counted.repo); + }); + + test('seven rows in threes is three reads, not seven', async () => { + await drain(feed().inBatches(3)); + + expect(counted.reads()).toBe(3); + }); + + test('breaking out of the loop stops the next statement', async () => { + for await (const batch of feed().inBatches(3)) { + expect(batch).toHaveLength(3); + break; + } + + expect(counted.reads()).toBe(1); + }); + + test('a throw inside the loop closes it too', async () => { + const batches = feed().inBatches(3); + await expect( + (async () => { + for await (const batch of batches) throw new Error(`boom on ${batch.length}`); + })(), + ).rejects.toThrow('boom on 3'); + + expect(counted.reads()).toBe(1); + expect((await batches[Symbol.asyncIterator]().next()).done).toBe(true); + }); + + test('await using closes the handle at the end of its scope', async () => { + let held: BatchIterator | null = null; + { + await using batches = feed().inBatches(3); + held = batches; + const first = await batches[Symbol.asyncIterator]().next(); + expect(first.value).toHaveLength(3); + } + + expect((await held[Symbol.asyncIterator]().next()).done).toBe(true); + expect(counted.reads()).toBe(1); + }); + + test('close() is idempotent, and closing before the first batch reads nothing', async () => { + const batches = feed().inBatches(3); + await batches.close(); + await batches.close(); + + expect(await drain(batches)).toEqual([]); + expect(counted.reads()).toBe(0); + }); + + test('one handle is one iteration — a second loop continues it, never restarts it', async () => { + const batches = feed().inBatches(3); + const first = await drain(batches); + const second = await drain(batches); + + expect(first.flat()).toHaveLength(7); + expect(second).toEqual([]); + expect(counted.reads()).toBe(3); + }); +}); + +describe('the position it stopped at', () => { + test('the chain’s own cursor is where it starts', () => { + expect(feed().inBatches(3).cursor).toBeNull(); + expect(feed().after('c_1').inBatches(3).cursor).toBe('c_1'); + }); + + test('after an exhausted iteration there is no next batch', async () => { + const batches = feed().inBatches(3); + await drain(batches); + + expect(batches.cursor).toBeNull(); + }); + + test('a break leaves the cursor the next batch starts from, and after() resumes it', async () => { + const batches = feed().inBatches(3); + const first: Post[] = []; + for await (const batch of batches) { + first.push(...batch); + break; + } + + expect(batches.cursor).not.toBeNull(); + const rest = (await drain(feed().after(batches.cursor).inBatches(3))).flat(); + + expect(first.map((row) => row.id)).toEqual(ids(0, 10, 20)); + // Every remaining row exactly once: a resumed iteration neither repeats nor skips. + expect(rest.map((row) => row.id)).toEqual(ids(30, 40, 50, 60)); + }); +}); + +describe('keyset, so a concurrent write cannot shift the loop', () => { + test('a row inserted before the cursor is not read twice, and one after it is read', async () => { + const seen: Post[] = []; + for await (const batch of feed().inBatches(3)) { + seen.push(...batch); + if (seen.length === 3) { + // An OFFSET pagination would repeat a row here and skip another; a keyset one reads the + // insert only where the sort order puts it, and never a row it already passed. + await table.insert(post(15)); + await table.insert(post(45)); + } + } + + expect(seen.map((row) => row.id)).toEqual(ids(0, 10, 20, 30, 40, 45, 50, 60)); + }); + + test('deleting rows already consumed does not skip the ones after them', async () => { + const seen: Post[] = []; + for await (const batch of feed().inBatches(3)) { + seen.push(...batch); + // What an OFFSET loop gets wrong: every row deleted shifts the next page back over rows + // nobody has read yet. The seek is a position in the sort order, so nothing moves. + for (const row of batch) await table.delete(row.id, { orgId: ORG }); + } + + expect(seen.map((row) => row.id)).toEqual(ids(...SEEDED)); + }); +}); + +describe('refused on the chain, not one batch later', () => { + let counted: Counted; + + beforeEach(() => { + counted = counting(memoryRepo(posts, seed)); + table = tableFor(posts, counted.repo); + }); + + test.each([0, -1, 2.5, Number.NaN, Number.POSITIVE_INFINITY])( + 'inBatches(%p) is a refusal, not a statement', + (size) => { + expect(caught(() => feed().inBatches(size))).toBeUltimateError('X_INVARIANT_VIOLATED'); + expect(counted.reads()).toBe(0); + }, + ); + + test('a chain that also called limit() has written one number twice', () => { + const error = caught(() => feed().limit(10).inBatches(500)); + + expect(error).toBeUltimateError('X_INVARIANT_VIOLATED'); + // The fix is the caller's own bound as the batch size — the reading that keeps their intent. + expect(error.fix).toContain('inBatches(10)'); + }); + + test('an ordering no cursor can carry is refused even when one batch would have hidden it', () => { + // The whole result fits in one batch, so nothing would ever mint a cursor and the mistake + // would survive until the table grew — which is the trap this eager check exists for. + expect(caught(() => feed().orderBy('publishedAt').inBatches(500))).toBeUltimateError( + 'X_INVARIANT_VIOLATED', + ); + expect(counted.reads()).toBe(0); + }); + + test('tenancy is still the plan’s: an unscoped chain rejects on its first batch', async () => { + const batches = table.inBatches(3); + + await expect(drain(batches)).rejects.toBeUltimateError('X_TENANCY_UNSCOPED'); + }); +}); + +describe('the Postgres driver sends the same statement page() sends', () => { + const physical = (index: number): Record => ({ + id: id(index), + org_id: ORG, + title: `Post ${index}`, + published_at: null, + }); + + let client: RecordingClient; + let pg: ReturnType>; + + beforeEach(() => { + client = createRecordingClient(); + pg = tableFor(posts, postgresRepo(posts, { client })); + }); + + test('one select per batch, each asking for one row past it', async () => { + // Three rows for a batch of two: the row past the page is what says there is a next one. + client.on('select', { rows: [physical(0), physical(1), physical(2)] }); + const seen: string[] = []; + for await (const batch of pg.where({ orgId: ORG }).inBatches(2)) { + seen.push(...batch.map((row) => row.id)); + client.on('select', { rows: [physical(3)] }); + } + + expect(seen).toEqual([id(0), id(1), id(3)]); + expect(client.texts).toHaveLength(2); + expect(client.statements[0]?.values.at(-1)).toBe(3); + }); + + test('the second batch seeks from the last row, never an offset', async () => { + client.on('select', { rows: [physical(0), physical(1), physical(2)] }); + for await (const _batch of pg.where({ orgId: ORG }).inBatches(2)) { + client.on('select', { rows: [] }); + } + const second = client.texts[1] ?? ''; + + expect(second).not.toContain('offset'); + expect(second).toContain('("id" > $2)'); + expect(second).toContain('"org_id" = $1'); + expect(client.statements[1]?.values[1]).toBe(id(1)); + }); + + test('a refused chain sends nothing at all', () => { + expect(caught(() => pg.where({ orgId: ORG }).limit(5).inBatches(5))).toBeUltimateError( + 'X_INVARIANT_VIOLATED', + ); + expect(client.statements).toHaveLength(0); + }); +}); diff --git a/packages/entity/src/batch.ts b/packages/entity/src/batch.ts new file mode 100644 index 000000000..2498a2f17 --- /dev/null +++ b/packages/entity/src/batch.ts @@ -0,0 +1,120 @@ +// Single responsibility: what `inBatches(size)` hands back — the keyset iteration a `for await` +// consumes, one page per statement, holding its own position and closed by the loop that reads it. +// +// Not a second read path. Every batch is the `findMany` the chain would have sent at that +// position, so filters, tenancy, soft delete, the projection and every `preload()` mean here +// exactly what they mean in `page()`. What this file owns is the loop, the refusals that belong on +// the chain rather than one batch later, and what closing means. + +import { assertSeekable } from './cursor'; +import type { EntityCore } from './entity'; +import { EntityError } from './errors'; +import { totalOrder } from './plan'; +import type { Page } from './repo'; +import type { SortKey } from './tenancy'; + +/** + * A batch size is rows, whole and at least one. `0`, a fraction and a `NaN` from a parsed + * environment variable all reach the same statement — `limit 0` reads nothing forever — so they are + * refused where they were written instead of hanging a job. + */ +const badBatchSize = (entityName: string, size: number): EntityError => + new EntityError({ + code: 'X_INVARIANT_VIOLATED', + cause: `${entityName}.inBatches(${size}) — a batch is a whole number of rows, at least one`, + fix: `${entityName}.inBatches(500) # the rows one statement reads`, + }); + +/** + * `limit()` bounds one page; `inBatches()` reads every page. A chain saying both has written one + * number with two meanings, and neither reading is safe to guess: honouring the limit reads a + * fraction of a batch, dropping it reads the whole table the caller thought they had bounded. + */ +const limitedBatches = (entityName: string, limit: number, size: number): EntityError => + new EntityError({ + code: 'X_INVARIANT_VIOLATED', + cause: `${entityName}.limit(${limit}).inBatches(${size}) — limit() bounds one page, inBatches() reads every page`, + fix: `${entityName}.inBatches(${limit}) # drop the limit(): the batch size is the page size`, + }); + +/** + * What the chain has to be before a batch of it exists. Every one of these is the author's own + * text, so all three are refused when `inBatches()` is called rather than one statement in: an + * iteration that reads half a table and then fails has already done half the work twice. + */ +export const assertBatchable = ( + entity: EntityCore, + size: number, + chain: { readonly limit: number | undefined; readonly orderBy: readonly SortKey[] }, +): void => { + if (!Number.isSafeInteger(size) || size < 1) throw badBatchSize(entity.$name, size); + if (chain.limit !== undefined) throw limitedBatches(entity.$name, chain.limit, size); + // The order the driver will sort by, primary key included: the cursor between two batches is + // minted from it, and an ordering that cannot carry one fails on the batch *after* the first — + // where whatever size the caller happened to pass decides whether anyone ever finds out. + assertSeekable(entity, totalOrder(entity, chain.orderBy)); +}; + +/** Where the batches come from: the chain's own starting position, and one page of it at a time. */ +export interface BatchRead { + /** `after(cursor)` on the chain, or `null` for the first batch. */ + readonly from: string | null; + /** The page this chain reads at `cursor` — relations attached, exactly as `page()` returns it. */ + page(cursor: string | null): Promise>; +} + +/** + * One iteration, one handle. `for await` consumes it and closes it on the way out — `break`, + * `return` and a throw all call `return()` on the iterator, which is what stops the next statement + * from going out; `await using` is the same guarantee for a handle kept in a variable. It is its + * own iterator, so a second `for await` continues where the first stopped instead of re-reading + * the table from the top. + */ +export interface BatchIterator extends AsyncIterable, AsyncDisposable { + /** + * Where the next batch starts, `null` once there is no next one. Persist it and + * `.after(cursor).inBatches(size)` resumes the iteration — which is what makes stopping early + * cheap rather than wasted. + */ + readonly cursor: string | null; + /** Ends the iteration. Idempotent, and what `await using` calls. */ + close(): Promise; +} + +/** + * The loop itself. Keyset, never OFFSET: each batch resumes from the cursor the previous one ended + * on, so a row inserted or deleted mid-iteration cannot make the loop skip or repeat one — the + * whole reason this package has no `offset`. + */ +export const batchIterator = (read: BatchRead): BatchIterator => { + let cursor = read.from; + + async function* batches(): AsyncGenerator { + do { + const page = await read.page(cursor); + // Advanced before the yield, so a consumer that breaks reads `.cursor` as the position it + // stopped at rather than the one it just consumed. + cursor = page.nextCursor; + // Never an empty batch: a consumer forced to check `batch.length` is reading around the + // iterator instead of through it. The last page ends the loop by its cursor, not by + // yielding nothing. + if (page.rows.length > 0) yield page.rows; + } while (cursor !== null); + } + + // Created once and never restarted: the generator's own state is what "closed" means, so + // `close()` is idempotent by construction rather than by a flag two paths could disagree about. + const iterator = batches(); + const close = async (): Promise => { + await iterator.return(undefined); + }; + + return { + get cursor(): string | null { + return cursor; + }, + [Symbol.asyncIterator]: () => iterator, + close, + [Symbol.asyncDispose]: close, + }; +}; diff --git a/packages/entity/src/bulk-write.test.ts b/packages/entity/src/bulk-write.test.ts new file mode 100644 index 000000000..3a075a50c --- /dev/null +++ b/packages/entity/src/bulk-write.test.ts @@ -0,0 +1,128 @@ +// Single responsibility: what ONE many-row statement carries, decided in property space — the +// column list a batch writes and how many rows fit inside Postgres's bind count. Which batches an +// upsert refuses, and what a collision overwrites, is `upsert-plan.test.ts`'s subject; both read +// the same `bulk-write.ts`, so a rule missing from either is one the two drivers may disagree on. + +import { afterAll, describe, expect, test } from 'bun:test'; +import { insertChunks, insertColumns, MAX_BIND_PARAMETERS, namedProperties } from './bulk-write'; +import { integer, money, text, uuid } from './columns'; +import { entity } from './entity'; +import { clearRegistry } from './registry'; + +const items = entity('bulk_items', { + columns: { + id: uuid().primaryKey(), + sku: text().unique(), + serial: text().nullable().unique(), + label: text(), + unitPrice: money(), + quantity: integer(), + }, +}); + +type Item = typeof items.$row; + +const ID = '0192f5a0-0000-7000-8000-00000000000a'; + +const item = { id: ID, sku: 'a', label: 'l', quantity: 1 }; + +afterAll(() => { + clearRegistry(); +}); + +describe('the columns a batch writes', () => { + test('names them in declaration order, never in the order the caller happened to write', () => { + // The statement's column list is shared by every row in the chunk, so it cannot depend on + // which row was read first — one row spelling it backwards would bind the wrong values. + expect(namedProperties(items, [{ quantity: 2, sku: 'a' }])).toEqual(['sku', 'quantity']); + expect(Object.keys(items.$columns)).toEqual([ + 'id', + 'sku', + 'serial', + 'label', + 'unitPrice', + 'quantity', + ]); + }); + + test('unions the batch: a column one row names is a column the statement carries', () => { + expect(namedProperties(items, [{ sku: 'a' }, { quantity: 1 }, { sku: 'b' }])).toEqual([ + 'sku', + 'quantity', + ]); + }); + + test('a property present and undefined is a value the caller wrote', () => { + // `Object.hasOwn`, exactly as `bindValues` decides it — dropping the column here would insert + // one the update set then skipped. `exactOptionalPropertyTypes` is why it takes a cast to say. + const written = { label: undefined } as Partial; + expect(namedProperties(items, [written])).toEqual(['label']); + expect(namedProperties(items, [{}])).toEqual([]); + expect(namedProperties(items, [])).toEqual([]); + }); + + test('a key the entity never declared is not a column', () => { + const stray = { sku: 'a', nickname: 'nope' } as Partial; + expect(namedProperties(items, [stray])).toEqual(['sku']); + }); + + test('money is two physical columns, expanded in place and snake_cased', () => { + expect(insertColumns(items, ['id', 'unitPrice', 'quantity'])).toEqual([ + 'id', + 'unit_price_minor', + 'unit_price_currency', + 'quantity', + ]); + }); + + test('the physical width is what the chunker divides, not the property count', () => { + const properties = namedProperties(items, [ + { ...item, serial: null, unitPrice: { minor: 1234n, currency: 'EUR' } }, + ]); + expect(properties).toHaveLength(6); + expect(insertColumns(items, properties)).toHaveLength(7); + }); + + test('a property the entity does not declare contributes no column', () => { + expect(insertColumns(items, ['nope'])).toEqual([]); + }); +}); + +describe('how many rows fit in one statement', () => { + const numbers = (count: number): readonly number[] => Array.from({ length: count }, (_, i) => i); + + test('an empty batch is no statements at all', () => { + expect(insertChunks([], 4)).toEqual([]); + }); + + test('a batch inside the bind budget is one statement', () => { + expect(insertChunks([1, 2, 3], 3)).toEqual([[1, 2, 3]]); + }); + + test('a batch past it is several, in order, none wider than the budget', () => { + const width = 21845; // 65535 / 3, so three rows fit and a fourth does not + const chunks = insertChunks(numbers(7), width); + expect(chunks.map((chunk) => chunk.length)).toEqual([3, 3, 1]); + expect(chunks.flat()).toEqual([...numbers(7)]); + for (const chunk of chunks) { + expect(chunk.length * width).toBeLessThanOrEqual(MAX_BIND_PARAMETERS); + } + }); + + test('the split lands exactly on the bind count, not one row short of it', () => { + // 13107 rows × 5 columns is 65535 binds — the widest statement Postgres accepts, and a + // chunker that rounded down would pay a second round trip for every batch this size. + const chunks = insertChunks(numbers(13108), 5); + expect(chunks.map((chunk) => chunk.length)).toEqual([13107, 1]); + expect(13107 * 5).toBe(MAX_BIND_PARAMETERS); + }); + + test('a row wider than the whole budget still goes, one row per statement', () => { + expect(insertChunks(['a', 'b', 'c'], MAX_BIND_PARAMETERS + 1)).toEqual([['a'], ['b'], ['c']]); + }); + + test('a width nobody could divide by is one statement, never zero-sized chunks', () => { + expect(insertChunks([1, 2], 0)).toEqual([[1, 2]]); + expect(insertChunks([1, 2], -3)).toEqual([[1, 2]]); + }); +}); diff --git a/packages/entity/src/bulk-write.ts b/packages/entity/src/bulk-write.ts new file mode 100644 index 000000000..d6853abe4 --- /dev/null +++ b/packages/entity/src/bulk-write.ts @@ -0,0 +1,284 @@ +// Single responsibility: what a many-row write is made of, decided once in PROPERTY space so both +// drivers write the same thing — the columns one statement carries, which of them a collision +// overwrites, and the chunking that keeps a statement inside Postgres's bind count. The SQL is +// `pg-sql.ts`'s; everything here is the decision that has to precede it, and the in-memory driver +// reads the same answer rather than a second copy of the rule. + +import { keyOf } from './batch-read'; +import { valueAt } from './cursor'; +import { type EntityCore, SOFT_DELETE_COLUMN } from './entity'; +import { EntityError, invariantViolated } from './errors'; +import { columnsOf } from './pg-row'; + +/** + * Postgres binds at most 65535 parameters in one statement and a multi-row insert spends + * rows × columns of them, so a wide batch becomes several statements rather than one the server + * refuses — the same rule `MAX_IDS_PER_STATEMENT` (`batch-read.ts`) applies to a batched read. + */ +export const MAX_BIND_PARAMETERS = 65535; + +/** What an `upsertAll` resolves to. Property names, so the two drivers read one answer. */ +export interface UpsertPlan { + /** The unique constraint a collision is judged against. */ + readonly on: readonly string[]; + /** What a colliding row takes from the incoming one. Empty leaves the stored row alone. */ + readonly set: readonly string[]; +} + +const owns = (row: unknown, property: string): boolean => + typeof row === 'object' && row !== null && Object.hasOwn(row, property); + +/** + * The properties a batch writes: every declared column at least one row names, in declaration + * order. `Object.hasOwn` decides it, exactly as `bindValues` does — a property present and + * `undefined` is a value the caller wrote, and dropping it here would insert a column the update + * set then skipped. + */ +export const namedProperties = ( + entity: EntityCore, + rows: readonly Partial[], +): readonly string[] => + Object.keys(entity.$columns).filter((property) => rows.some((row) => owns(row, property))); + +/** Those properties as physical columns. Money is two columns, so this is not the same list. */ +export const insertColumns = ( + entity: EntityCore, + properties: readonly string[], +): readonly string[] => + properties.flatMap((property) => { + const column = entity.$columns[property]; + return column === undefined ? [] : columnsOf(property, column); + }); + +/** + * Rows per statement: the bind budget divided by one row's width, never fewer than one. Splitting + * can only cost round trips; not splitting would fail a write that succeeded one row at a time. + */ +export const insertChunks = (rows: readonly T[], width: number): readonly (readonly T[])[] => { + if (rows.length === 0) return []; + const size = width <= 0 ? rows.length : Math.max(1, Math.floor(MAX_BIND_PARAMETERS / width)); + const chunks: (readonly T[])[] = []; + for (let from = 0; from < rows.length; from += size) chunks.push(rows.slice(from, from + size)); + return chunks; +}; + +/** + * Not `invariantViolated`: its fix opens `x entity explain`, which describes invariants nobody + * wrote here. What repairs each of these is one edit to the call. + */ +const noConflictTarget = (entityName: string): EntityError => + new EntityError({ + code: 'X_INVARIANT_VIOLATED', + cause: `${entityName}.upsertAll() named no onConflict columns — a collision is judged against a unique constraint, never against "any"`, + fix: `${entityName}.upsertAll(rows, { onConflict: [''] }) # the columns of the unique index a duplicate lands on`, + }); + +const nothingToSet = ( + entityName: string, + on: readonly string[], + spared: readonly string[], +): EntityError => + new EntityError({ + code: 'X_INVARIANT_VIOLATED', + cause: `${entityName}.upsertAll() would write nothing on a collision: every column in the batch is one a collision never moves (${spared.join(', ')})`, + fix: `${entityName}.upsertAll(rows, { onConflict: ['${on.join("', '")}'], onMatch: 'nothing' }) # keep the stored row`, + }); + +const collidesWithItself = ( + entityName: string, + on: readonly string[], + position: number, +): EntityError => + new EntityError({ + code: 'X_INVARIANT_VIOLATED', + cause: `${entityName}.upsertAll() row ${position + 1} repeats a value of (${on.join(', ')}) already in the batch — one statement cannot update the same row twice`, + fix: `${entityName}.upsertAll(rows, { onConflict: ['${on.join("', '")}'], onMatch: 'nothing' }) # or dedupe rows on (${on.join(', ')}) before the call`, + }); + +const noSuchConstraint = ( + entityName: string, + on: readonly string[], + declared: readonly string[], +): EntityError => + new EntityError({ + code: 'X_INVARIANT_VIOLATED', + cause: `${entityName} declares no unique constraint on (${on.join(', ')}); Postgres answers a conflict target it cannot match with 42P10${declared.length === 0 ? '' : ` — it has ${declared.join('; ')}`}`, + fix: `indexes: [{ on: ['${on.join("', '")}'], unique: true }] # in the ${entityName} entity(), then x db gen "unique ${on.join(' ')}"`, + }); + +const unevenBatch = (entityName: string, position: number, missing: string): EntityError => + new EntityError({ + code: 'X_INVARIANT_VIOLATED', + cause: `${entityName}.upsertAll() row ${position + 1} does not name "${missing}", which other rows of the batch do — under onMatch: 'update' a column one row omits would be overwritten with that column's default, not left alone`, + fix: `${entityName}.upsertAll(rows.map((row) => ({ ...row, ${missing}: })), args) # every row of an updating batch names the same columns`, + }); + +/** + * A cross-tenant upsert is not a batching mistake, it is a write into another tenant's row, so it + * gets tenancy's own code. `X_TENANCY_UNSCOPED`'s own factory sends the reader to `{ orgId }`, + * which is not what repairs this: the scope has to be part of the constraint the collision is + * judged against, or the row that comes back was never in scope to begin with. + */ +const crossTenantUpsert = ( + entityName: string, + tenant: string, + on: readonly string[], +): EntityError => + new EntityError({ + code: 'X_TENANCY_UNSCOPED', + cause: `${entityName}.upsertAll() resolves a collision on (${on.join(', ')}), which does not include the tenant column "${tenant}" — a row stored by another tenant would match and be overwritten`, + fix: `${entityName}.upsertAll(rows, { onConflict: ['${tenant}', '${on.join("', '")}'] }) # or onMatch: 'nothing', which writes nothing to a row it does not own`, + }); + +/** + * Every conflict target Postgres could infer an index for. Three sources, because this framework + * has three ways to declare one unique index and a target refused for being declared the "wrong" + * way would send its author to add a SECOND declaration of a constraint they already wrote: + * the primary key, `unique()`/`indexes: [{ unique: true }]` (both land in `$indexes`), and + * `invariant(name, c.unique([…]))`, which emits its `create unique index` out of `$invariants`. + * + * A partial unique index is excluded from all three: its predicate would have to be repeated in + * the `on conflict` clause, which this layer does not spell. `bindInvariant` stamps that `where` + * on a soft-deleting entity, so the same one rule covers both lists. + */ +const uniqueTargets = (entity: EntityCore): readonly (readonly string[])[] => [ + insertColumns(entity, entity.$primaryKey), + ...entity.$indexes.filter((i) => i.unique && i.where === undefined).map((i) => i.columns), + ...entity.$invariants + .filter((i) => i.kind === 'unique' && i.where === undefined) + .map((i) => i.columns), +]; + +const sameColumns = (left: readonly string[], right: readonly string[]): boolean => + left.length === right.length && [...left].sort().join() === [...right].sort().join(); + +/** + * The conflict target, and what a collision overwrites. `'nothing'` overwrites nothing by + * definition; `'update'` takes every column the batch writes except three closed sets — the + * conflict target, which is how the stored row was found, the primary key, which is its address, + * and the soft-delete stamp, which is whether the row is there at all. An upsert that moved one of + * the first two would move a row nobody asked to move, and every foreign key already pointing at + * that id would miss it. + * + * The stamp is the third for a reason no caller can work around: a soft-deleted row still occupies + * its conflict target — the unique index it collides with is not partial — so setting `deleted_at` + * from `excluded` would clear a stamp the app wrote and hand the row back holding this batch's + * values. That is the resurrection `update(id, patch)` and `updateWhere` both refuse by carrying + * `deleted_at is null`, which an `on conflict` clause cannot carry. Excluded rather than refused, + * because `$parse` fills every declared column before a row reaches here: the `deletedAt: null` in + * the batch is the framework's, not the caller's, and refusing it would make `'update'` impossible + * on every soft-deleting entity. `insertAll` is untouched — a row with no stored row to collide + * with writes the stamp it carries, exactly as `insert` does. + * + * Four refusals precede all of that, and each one is a statement Postgres would either reject or, + * worse, accept: a target no declared unique constraint matches (`42P10`), a target that does not + * carry the tenant column under `'update'` (another tenant's row, silently rewritten), a batch + * whose rows name different columns under `'update'` (`excluded.` is the column's default for + * a row that omitted it, so "leave it alone" is not what happens), and a target that leaves + * nothing to write. + */ +export const upsertPlan = ( + entity: EntityCore, + rows: readonly Partial[], + onConflict: readonly string[], + onMatch: 'update' | 'nothing', +): UpsertPlan => { + if (onConflict.length === 0) throw noConflictTarget(entity.$name); + for (const property of onConflict) { + if (entity.$columns[property] === undefined) { + throw invariantViolated( + entity.$name, + 'upsertAll', + `no column "${property}" — pick from: ${Object.keys(entity.$columns).join(', ')}`, + ); + } + } + const targets = uniqueTargets(entity); + const physical = insertColumns(entity, onConflict); + if (!targets.some((target) => sameColumns(target, physical))) { + throw noSuchConstraint( + entity.$name, + onConflict, + targets.map((target) => `(${target.join(', ')})`), + ); + } + if (onMatch === 'nothing') return { on: onConflict, set: [] }; + const tenant = entity.$tenantColumn; + if (tenant !== null && !onConflict.includes(tenant)) { + throw crossTenantUpsert(entity.$name, tenant, onConflict); + } + const properties = namedProperties(entity, rows); + for (const [position, row] of rows.entries()) { + const missing = properties.find((property) => !owns(row, property)); + if (missing !== undefined) throw unevenBatch(entity.$name, position, missing); + } + const spared = new Set([ + ...onConflict, + ...entity.$primaryKey, + ...(entity.$softDelete ? [SOFT_DELETE_COLUMN] : []), + ]); + const set = properties.filter((property) => !spared.has(property)); + // An empty batch sends no statement, so there is nothing to refuse — but the target above is + // checked either way, so a typo in `onConflict` fails on no rows exactly as it does on a page. + if (set.length === 0 && properties.length > 0) { + throw nothingToSet(entity.$name, onConflict, [...spared]); + } + return { on: onConflict, set }; +}; + +/** + * One cell of a conflict target. `keyOf` alone is `batch-read.ts`'s spelling of an *id*, and three + * kinds need more than it: a `Date` stringifies without its milliseconds, an object (money is one) + * stringifies to `[object Object]`, and — the one that changes an answer — a null is not a value + * Postgres compares. A default unique index is `NULLS DISTINCT`, so two rows with a null in the + * target collide with nothing, not with each other. + */ +const cellKey = (kind: string, value: unknown): string | undefined => { + if (value === null || value === undefined) return undefined; + if (value instanceof Date) return `date:${value.getTime()}`; + if (typeof value === 'object') { + return `json:${JSON.stringify(value, (_, part) => (typeof part === 'bigint' ? `${part}n` : part))}`; + } + return `value:${keyOf(kind, value)}`; +}; + +/** + * One row's conflict target as a string, spelled the way a batched read spells a key — a `uuid` + * handed in upper case is the value Postgres matches, so it must be the value matched here. + * `undefined` when any cell is null: that row's target is distinct from every other, as it is in + * the index, so it collides with nothing. + */ +export const conflictKeyOf = ( + entity: EntityCore, + on: readonly string[], + row: Partial, +): string | undefined => { + const cells = on.map((property) => + cellKey(entity.$columns[property]?.$meta.kind ?? '', valueAt(row, property)), + ); + return cells.some((cell) => cell === undefined) ? undefined : JSON.stringify(cells); +}; + +/** + * Every row's conflict key, refusing a batch that collides with itself. Postgres answers two rows + * with one conflict target under `do update` with `ON CONFLICT DO UPDATE command cannot affect row + * a second time`, so it is refused here — in both drivers — rather than passing in memory and + * failing in production. Under `do nothing` the server skips the repeat, and so does the memory + * driver, which is why this only guards the update form. + */ +export const conflictKeys = ( + entity: EntityCore, + plan: UpsertPlan, + rows: readonly Partial[], +): readonly (string | undefined)[] => { + const keys = rows.map((row) => conflictKeyOf(entity, plan.on, row)); + if (plan.set.length === 0) return keys; + const seen = new Set(); + for (const [position, key] of keys.entries()) { + // A null in the target is no key at all, so it repeats nothing — `NULLS DISTINCT` again. + if (key === undefined) continue; + if (seen.has(key)) throw collidesWithItself(entity.$name, plan.on, position); + seen.add(key); + } + return keys; +}; diff --git a/packages/entity/src/count-by-parity.test.ts b/packages/entity/src/count-by-parity.test.ts new file mode 100644 index 000000000..3808d3036 --- /dev/null +++ b/packages/entity/src/count-by-parity.test.ts @@ -0,0 +1,356 @@ +// The two drivers of a grouped count, side by side: the answer they must agree on, the refusals +// they must share, and the bound that refuses rather than truncates. A recording client stands in +// for the server, `memoryRepo` is the other half, and both are compared against rows written out +// by hand so neither can be right only because the other is wrong. The SQL a chain compiles to is +// `count-by-statements.test.ts`'s subject — this file is about the answer, never the text. + +import { afterAll, beforeEach, describe, expect, test } from 'bun:test'; +import { isUltimateError } from '@ultimat3/core'; +import { createRecordingClient, type RecordingClient, setDbClient } from '@ultimat3/db'; +import { column } from './column'; +import { integer, money, text, timestamp, uuid } from './columns'; +import { MAX_GROUPS } from './count-by'; +import { entity } from './entity'; +import { postgresRepo } from './pg-driver'; +import { clearRegistry } from './registry'; +import { memoryRepo } from './repo'; +import type { Column } from './types'; + +/** + * `bigint` and `jsonb` are `ColumnKind` members no builder in `columns.ts` spells — one groupable, + * one refused — so both are built from the same `column()` helper every public builder is made of. + * `count-by.ts` branches on the kind, and a kind with no test is a branch nothing runs. + */ +const bigints = (): Column => + column('bigint', (value) => (typeof value === 'bigint' ? value : BigInt(String(value)))); + +const jsonb = (): Column => column('jsonb', (value) => value); + +const orgs = entity('count_parity_orgs', { + columns: { id: uuid().primaryKey(), slug: text({ max: 40 }).unique() }, +}); + +/** + * One row per reaction — the shape `countBy` exists for: `likes.countBy('postId')` in place of one + * `count()` per post. Every kind the groupable set turns on or refuses is declared once here, and + * `count` is among them because an entity is free to name a column after the aggregate. + */ +const likes = entity('count_parity_likes', { + columns: { + id: uuid().primaryKey(), + orgId: uuid() + .references(() => orgs.id) + .tenant(), + postId: uuid().nullable(), + reaction: text({ max: 20 }), + count: integer(), + weight: bigints(), + payload: jsonb(), + total: money(), + createdAt: timestamp().defaultNow(), + deletedAt: timestamp().nullable(), + }, +}); + +/** + * Every column a `Map` cannot be keyed by and nothing else — a `timestamptz` is a legal primary + * key, so an entity really can offer a grouped count no column of it could answer. The one shape + * that reaches the branch where the refusal has no call to suggest. + */ +const ungroupable = entity('count_parity_ungroupable', { + columns: { at: timestamp().primaryKey(), payload: jsonb(), total: money() }, +}); + +type Like = typeof likes.$row; + +const idAt = (suffix: string): string => `00000000-0000-7000-8000-${suffix.padStart(12, '0')}`; + +const ORG = idAt('a1'); +const OTHER_ORG = idAt('a2'); +const POST_A = idAt('aaa'); +const POST_B = idAt('bbb'); +const POST_C = idAt('ccc'); + +const like = (index: number, over: Partial = {}): Like => ({ + id: idAt(String(index)), + orgId: ORG, + postId: POST_A, + reaction: 'like', + count: 1, + weight: 10n, + payload: {}, + total: { minor: 0n, currency: 'EUR' }, + createdAt: new Date('2026-01-02T03:04:05.000Z'), + deletedAt: null, + ...over, +}); + +/** + * One tenant's rows — three on post A, two each on B, C and on no post at all — then a stamped row + * and another tenant's row, both on post A, so 4 or 5 there is the only reading either leak could + * produce. The group rows below are what a server answers for exactly these, written out rather + * than derived, so the two drivers are compared against a third thing and not against each other. + */ +const SEED: readonly Like[] = [ + ...[POST_A, POST_A, POST_A, POST_B, POST_B, POST_C, POST_C, null, null].map((postId, index) => + like(index, { postId }), + ), + like(90, { deletedAt: new Date('2026-02-02T00:00:00.000Z') }), + like(91, { orgId: OTHER_ORG }), +]; + +/** What Postgres answers for `SEED` under `ORG` — in an order the map must not keep. */ +const VISIBLE_GROUPS = [ + { group_value: POST_C, group_count: '2' }, + { group_value: null, group_count: '2' }, + { group_value: POST_A, group_count: '3' }, + { group_value: POST_B, group_count: '2' }, +]; + +/** The same, with the soft-delete clause dropped: post A gains the stamped row and nothing else. */ +const WITH_DELETED_GROUPS = [ + { group_value: POST_B, group_count: '2' }, + { group_value: POST_A, group_count: '4' }, + { group_value: null, group_count: '2' }, + { group_value: POST_C, group_count: '2' }, +]; + +let client: RecordingClient; + +beforeEach(() => { + client = createRecordingClient(); + setDbClient(client); +}); + +afterAll(() => { + setDbClient(undefined); + clearRegistry(); +}); + +const repo = () => postgresRepo(likes); +const memory = (seed: readonly Like[] = SEED) => memoryRepo(likes, seed); +const lastText = (): string => client.texts.at(-1) ?? ''; +const lastValues = (): readonly unknown[] => client.statements.at(-1)?.values ?? []; + +type Refusal = Record<'code' | 'cause' | 'fix', string>; + +/** What a refusal says. A code is the contract, and a `fix:` that names no edit is not one. */ +const refusal = async (call: Promise): Promise => { + try { + await call; + return { code: 'resolved', cause: '', fix: '' }; + } catch (error) { + if (!isUltimateError(error)) return { code: String(error), cause: '', fix: '' }; + return { code: error.code, cause: error.cause, fix: error.fix }; + } +}; + +/** Distinct groups, in the shape Bun.SQL hands them over: an int8 count is a string. */ +const groupRows = (howMany: number): readonly Record[] => + Array.from({ length: howMany }, (_, index) => ({ + group_value: idAt(String(index)), + group_count: String((index % 7) + 1), + })); + +describe('the bound refuses rather than truncates', () => { + test('one group past the bound is an error, not a map missing its tail', async () => { + client.on('group by', { rows: groupRows(MAX_GROUPS + 1) }); + + await expect(repo().countBy('postId', { orgId: ORG })).rejects.toBeUltimateError( + 'X_INVARIANT_VIOLATED', + ); + // One group more than the bound, on purpose: that extra group is the whole signal, and a + // `limit MAX_GROUPS` would make a complete answer and a truncated one read identically. + expect(lastValues().at(-1)).toBe(MAX_GROUPS + 1); + }); + + test('exactly the bound resolves, whole', async () => { + client.on('group by', { rows: groupRows(MAX_GROUPS) }); + + const counts = await repo().countBy('postId', { orgId: ORG }); + + expect(counts.size).toBe(MAX_GROUPS); + expect([...counts.values()][0]).toBe(7); + expect(counts.get(idAt('0'))).toBe(1); + }); + + test('the in-memory driver refuses at the same bound, from the same function', async () => { + // The bound belongs to `count-by.ts`, not to the statement: 1001 distinct values held in + // memory is the same refusal, and a bound that lived in `limit` alone would let this through. + const many = Array.from({ length: MAX_GROUPS + 1 }, (_, index) => + like(index, { postId: idAt(`p${index}`) }), + ); + + await expect(memory(many).countBy('postId', { orgId: ORG })).rejects.toBeUltimateError( + 'X_INVARIANT_VIOLATED', + ); + // One distinct value fewer and it resolves, so the refusal above is about the bound and not + // about the size of the seed. + expect((await memory(many.slice(0, MAX_GROUPS)).countBy('postId', { orgId: ORG })).size).toBe( + MAX_GROUPS, + ); + }); +}); + +describe('the refusals both drivers share', () => { + const UNGROUPABLE: readonly (readonly [string, string])[] = [ + ['secret', 'no column "secret" on count_parity_likes'], + ['createdAt', 'a timestamptz column is not a key a map can be looked up by'], + ['payload', 'a jsonb column is not a key a map can be looked up by'], + ['total', 'a money column is not a key a map can be looked up by'], + ]; + + test('an ungroupable column is the same refusal in both drivers, before any statement', async () => { + for (const [property, cause] of UNGROUPABLE) { + const fromPostgres = await refusal(repo().countBy(property, { orgId: ORG })); + const fromMemory = await refusal(memory().countBy(property, { orgId: ORG })); + + expect(fromPostgres.code).toBe('X_INVARIANT_VIOLATED'); + // Byte for byte, because ONE function raised both: a driver growing its own copy of this + // rule is the drift `count-by.ts` exists to prevent, and it would surface here first. + expect(fromPostgres).toEqual(fromMemory); + expect(fromPostgres.cause).toContain(`count_parity_likes.countBy('${property}'): ${cause}`); + } + expect(client.statements).toHaveLength(0); + }); + + test('the column is judged before the scope is', async () => { + // `groupColumnOf` runs ahead of `readPlan` in both drivers, so an author who named the wrong + // column is told about the column — not about a tenancy predicate they would then add for + // nothing and still be refused. + expect((await refusal(repo().countBy('createdAt'))).code).toBe('X_INVARIANT_VIOLATED'); + expect((await refusal(memory().countBy('createdAt'))).code).toBe('X_INVARIANT_VIOLATED'); + expect(client.statements).toHaveLength(0); + }); + + test('the refusal names a column of this entity that would have worked', async () => { + const refused = await refusal(repo().countBy('createdAt', { orgId: ORG })); + + // Not `x entity explain`: what repairs this is one edit to the call, and the replacement + // column can only be named by reading the entity. + expect(refused.fix).toBe( + "count_parity_likes.countBy('id') # group by one of: id, orgId, postId, reaction, count, weight", + ); + }); + + test('an entity with nothing to group by is told to read it, since no call would work', async () => { + // The one branch where a `fix:` cannot be a call: every column here would be refused the same + // way, so it names a command that exists (`x entities describe`, which prints the kinds) rather + // than suggesting a `countBy` that fails identically. + const refused = await refusal(memoryRepo(ungroupable).countBy('at')); + + expect(refused.code).toBe('X_INVARIANT_VIOLATED'); + expect(refused.fix).toBe( + 'x entities describe count_parity_ungroupable --json # this entity declares no column a count can be keyed by', + ); + }); +}); + +describe('two drivers, one meaning', () => { + test('the same rows produce the same map, in the same order, in both drivers', async () => { + client.on('group by', { rows: VISIBLE_GROUPS }); + + const fromPostgres = await repo().countBy('postId', { orgId: ORG }); + const fromMemory = await memory().countBy('postId', { orgId: ORG }); + + expect([...fromPostgres]).toEqual([...fromMemory]); + // And both against a third thing, so neither can be right only because the other is wrong. + expect([...fromMemory]).toEqual([ + [POST_A, 3], + [POST_B, 2], + [POST_C, 2], + [null, 2], + ]); + // The stamped row and the other tenant's row both sit on post A. + expect(fromMemory.get(POST_A)).toBe(3); + }); + + test('includeDeleted is the same one extra row in both drivers', async () => { + client.on('group by', { rows: WITH_DELETED_GROUPS }); + + const args = { orgId: ORG, includeDeleted: true }; + const fromPostgres = await repo().countBy('postId', args); + // Postgres does it by dropping a clause, memory by skipping a filter — same answer. + expect(lastText()).not.toContain('"deleted_at" is null'); + + const fromMemory = await memory().countBy('postId', args); + + expect([...fromPostgres]).toEqual([...fromMemory]); + expect(fromMemory.get(POST_A)).toBe(4); + }); + + test('another tenant is a different breakdown, never a shared one', async () => { + const ours = await memory().countBy('postId', { orgId: ORG }); + const theirs = await memory().countBy('postId', { orgId: OTHER_ORG }); + + expect(ours.get(POST_A)).toBe(3); + expect([...theirs]).toEqual([[POST_A, 1]]); + + // The Postgres driver carries that same scope into the statement rather than filtering rows + // it has already read, which is the only way it can mean the same thing. + await repo().countBy('postId', { orgId: OTHER_ORG }); + expect(lastText()).toContain('"org_id" = $1'); + expect(lastValues()[0]).toBe(OTHER_ORG); + }); + + test('a bounded breakdown narrows both drivers the same way', async () => { + // The shape the too-many-groups refusal tells an author to write: bound the values, then + // count them in one statement. + const args = { + orgId: ORG, + where: [{ column: 'postId', op: 'in' as const, value: [POST_B, POST_C] }], + }; + client.on('group by', { + rows: [ + { group_value: POST_C, group_count: '2' }, + { group_value: POST_B, group_count: '2' }, + ], + }); + + const fromPostgres = await repo().countBy('postId', args); + expect(lastText()).toContain('"post_id" in ($1, $2)'); + + const fromMemory = await memory().countBy('postId', args); + + expect([...fromPostgres]).toEqual([...fromMemory]); + expect([...fromMemory]).toEqual([ + [POST_B, 2], + [POST_C, 2], + ]); + }); + + test('a page bounds a page, never a count: limit() and after() reach neither driver', async () => { + // An aggregate covers the predicate — the chain's filters, its tenancy, its soft-delete + // visibility — and never the page. A cursor that reached one driver and not the other would + // make the breakdown change as the caller paged through it. + const args = { orgId: ORG } as const; + const { nextCursor } = await memory().findMany({ ...args, limit: 2 }); + expect(nextCursor).not.toBeNull(); + const paged = { ...args, limit: 2, cursor: nextCursor }; + client.on('group by', { rows: VISIBLE_GROUPS }); + + const fromPostgres = await repo().countBy('postId', paged); + // The only limit in the statement is the group bound, and no seek term made it in. + expect(lastValues().at(-1)).toBe(MAX_GROUPS + 1); + expect(lastText()).not.toContain('>'); + + const fromMemory = await memory().countBy('postId', paged); + + expect([...fromPostgres]).toEqual([...fromMemory]); + expect([...fromMemory]).toEqual([...(await memory().countBy('postId', args))]); + // `count()` is the same rule with one group, and it is the one both are compared against. + expect(await memory().count(paged)).toBe(await memory().count(args)); + }); + + test('a predicate nothing matches is an empty map in both drivers, never a map of zeros', async () => { + const args = { orgId: ORG, where: [{ column: 'reaction', op: 'eq' as const, value: 'nope' }] }; + + const fromPostgres = await repo().countBy('postId', args); + const fromMemory = await memory().countBy('postId', args); + + expect([...fromPostgres]).toEqual([]); + expect([...fromMemory]).toEqual([]); + // Absent, so the caller can still tell "none" from "never asked" — the `?? 0` is theirs. + expect(fromMemory.get(POST_A)).toBeUndefined(); + }); +}); diff --git a/packages/entity/src/count-by-statements.test.ts b/packages/entity/src/count-by-statements.test.ts new file mode 100644 index 000000000..dbd5bed10 --- /dev/null +++ b/packages/entity/src/count-by-statements.test.ts @@ -0,0 +1,240 @@ +// The grouped count against a recording client: the one statement `countBy` compiles to, and what +// an int8, a group key and a NULL decode into on the way back. Statement text is exactly what a +// live server cannot be asked, which is why it is asserted here; the answer the two drivers must +// agree on, their shared refusals and the group bound are `count-by-parity.test.ts`'s subject. + +import { afterAll, beforeEach, describe, expect, test } from 'bun:test'; +import { createRecordingClient, type RecordingClient, setDbClient } from '@ultimat3/db'; +import { column } from './column'; +import { integer, money, text, timestamp, uuid } from './columns'; +import { MAX_GROUPS } from './count-by'; +import { entity } from './entity'; +import { postgresRepo } from './pg-driver'; +import { clearRegistry } from './registry'; +import { memoryRepo } from './repo'; +import type { Column } from './types'; + +/** + * `bigint` and `jsonb` are `ColumnKind` members no builder in `columns.ts` spells — one groupable, + * one refused — so both are built from the same `column()` helper every public builder is made of. + * `count-by.ts` branches on the kind, and a kind with no test is a branch nothing runs. + */ +const bigints = (): Column => + column('bigint', (value) => (typeof value === 'bigint' ? value : BigInt(String(value)))); + +const jsonb = (): Column => column('jsonb', (value) => value); + +const orgs = entity('count_test_orgs', { + columns: { id: uuid().primaryKey(), slug: text({ max: 40 }).unique() }, +}); + +/** + * One row per reaction — the shape `countBy` exists for: `likes.countBy('postId')` in place of one + * `count()` per post. Every kind the groupable set turns on or refuses is declared once here, and + * `count` is among them because an entity is free to name a column after the aggregate. + */ +const likes = entity('count_test_likes', { + columns: { + id: uuid().primaryKey(), + orgId: uuid() + .references(() => orgs.id) + .tenant(), + postId: uuid().nullable(), + reaction: text({ max: 20 }), + count: integer(), + weight: bigints(), + payload: jsonb(), + total: money(), + createdAt: timestamp().defaultNow(), + deletedAt: timestamp().nullable(), + }, +}); + +const idAt = (suffix: string): string => `00000000-0000-7000-8000-${suffix.padStart(12, '0')}`; + +const ORG = idAt('a1'); +const POST_A = idAt('aaa'); +const POST_B = idAt('bbb'); +const POST_C = idAt('ccc'); + +let client: RecordingClient; + +beforeEach(() => { + client = createRecordingClient(); + setDbClient(client); +}); + +afterAll(() => { + setDbClient(undefined); + clearRegistry(); +}); + +const repo = () => postgresRepo(likes); +// Seedless on purpose: the one thing this file asks the in-memory driver is whether a refusal +// lands before a row is read, and a fixture would only suggest the rows had something to do with it. +const memory = () => memoryRepo(likes, []); +const lastText = (): string => client.texts.at(-1) ?? ''; +const lastValues = (): readonly unknown[] => client.statements.at(-1)?.values ?? []; + +/** The predicate half of a statement — what `count()` and `countBy()` have to agree on exactly. */ +const predicateOf = (text: string): string => { + const grouped = text.indexOf(' group by '); + return text.slice(text.indexOf(' where '), grouped === -1 ? undefined : grouped); +}; + +describe('the statement a grouped count compiles to', () => { + test('it is the statement count() sends, one group by more, both outputs aliased', async () => { + await repo().countBy('postId', { orgId: ORG }); + + expect(lastText()).toBe( + 'select "post_id" as group_value, count(*) as group_count from "count_test_likes"' + + ' where "org_id" = $1 and "deleted_at" is null group by "post_id" limit $2', + ); + expect(lastValues()).toEqual([ORG, MAX_GROUPS + 1]); + }); + + test('the predicate is byte-for-byte the one count() would have sent', async () => { + const args = { orgId: ORG, where: [{ column: 'reaction', op: 'eq' as const, value: 'like' }] }; + + await repo().count(args); + const counted = predicateOf(lastText()); + await repo().countBy('postId', args); + + // Compiled by one function over one plan, so a filter, a scope or a visibility rule that + // changes for `count()` and not for `countBy()` is two different answers to one question. + expect(predicateOf(lastText())).toBe(counted); + expect(counted).toBe(' where "reaction" = $1 and "org_id" = $2 and "deleted_at" is null'); + }); + + test('every value is bound, and so is the group bound — nothing is spliced in', async () => { + const evil = "x'; drop table count_test_likes; --"; + + await repo().countBy('postId', { + orgId: ORG, + where: [{ column: 'reaction', op: 'eq', value: evil }], + }); + + expect(lastText()).not.toContain('drop table'); + expect(lastText()).toContain('"reaction" = $1'); + // The bound is a parameter too — the one value in this statement no caller supplied, and so + // the one a builder is tempted to concatenate. + expect(lastText()).toEndWith('limit $3'); + expect(lastText()).not.toContain(String(MAX_GROUPS + 1)); + expect(lastValues()).toEqual([evil, ORG, MAX_GROUPS + 1]); + }); + + test('both outputs are aliased, so a column named count cannot collide with the aggregate', async () => { + client.on('group by "count"', { rows: [{ group_value: 3, group_count: '4' }] }); + + const counts = await repo().countBy('count', { orgId: ORG }); + + expect(lastText()).toStartWith( + 'select "count" as group_value, count(*) as group_count from "count_test_likes"', + ); + expect(lastText()).toContain('group by "count" limit $2'); + // Un-aliased, both outputs would be named `count` and one would have eaten the other. + expect([...counts]).toEqual([[3, 4]]); + }); + + test('soft-deleted rows are out by default and in under includeDeleted', async () => { + await repo().countBy('postId', { orgId: ORG }); + expect(lastText()).toContain('"deleted_at" is null'); + + await repo().countBy('postId', { orgId: ORG, includeDeleted: true }); + expect(lastText()).not.toContain('"deleted_at" is null'); + expect(lastText()).toContain('group by "post_id"'); + }); + + test('an unscoped grouped count refuses before a statement exists, in both drivers', async () => { + await expect(repo().countBy('postId')).rejects.toBeUltimateError('X_TENANCY_UNSCOPED'); + await expect(memory().countBy('postId')).rejects.toBeUltimateError('X_TENANCY_UNSCOPED'); + // The guard every other read answers to, applied where it still costs nothing: an aggregate + // that ran and was then filtered would already have read another tenant's rows. + expect(client.statements).toHaveLength(0); + + await repo().countBy('postId', { orgId: ORG }); + expect(lastText()).toContain('"org_id" = $1'); + expect(lastValues()[0]).toBe(ORG); + }); +}); + +describe('what comes back off the wire', () => { + test('count(*) arrives as an int8 string and lands in the map as a number', async () => { + client.on('group by', { rows: [{ group_value: POST_A, group_count: '3' }] }); + + const counts = await repo().countBy('postId', { orgId: ORG }); + + expect(counts.get(POST_A)).toBe(3); + // The type, not only the value: left as a string, `(counts.get(id) ?? 0) + 1` is '31'. + expect([...counts.values()].map((value) => typeof value)).toEqual(['number']); + }); + + test('the group key is re-parsed by the column that declared it', async () => { + client.on('group by "weight"', { rows: [{ group_value: '900', group_count: '2' }] }); + const byWeight = await repo().countBy('weight', { orgId: ORG }); + + // int8 on the key side: '900' is a key the caller who wrote `900n` can never look up, and a + // map that only answers `undefined` reads exactly like a table with no such row. + expect([...byWeight.keys()]).toEqual([900n]); + expect(byWeight.get('900')).toBeUndefined(); + + client.on('group by "post_id"', { rows: [{ group_value: POST_A, group_count: '2' }] }); + expect([...(await repo().countBy('postId', { orgId: ORG })).keys()]).toEqual([POST_A]); + }); + + test('a null group_value is the one group SQL puts every NULL row in', async () => { + client.on('group by', { + rows: [ + { group_value: null, group_count: '2' }, + { group_value: POST_A, group_count: '1' }, + ], + }); + + const counts = await repo().countBy('postId', { orgId: ORG }); + + expect(counts.get(null)).toBe(2); + expect(counts.size).toBe(2); + }); +}); + +describe('the order the map comes back in', () => { + test('biggest group first; ties by the value, numbers and bigints numerically', async () => { + client.on('group by "count"', { + rows: [ + { group_value: 20, group_count: '5' }, + { group_value: null, group_count: '5' }, + { group_value: 100, group_count: '5' }, + { group_value: 3, group_count: '9' }, + { group_value: 7, group_count: '1' }, + ], + }); + + const counts = await repo().countBy('count', { orgId: ORG }); + + // 20 before 100: compared as text they would come back the other way round. And `null` is + // last of the three that tie, not last of the five — the count still decides the position. + expect([...counts.keys()]).toEqual([3, 20, 100, null, 7]); + expect([...counts.values()]).toEqual([9, 5, 5, 5, 1]); + + client.on('group by "post_id"', { + rows: [ + { group_value: POST_C, group_count: '2' }, + { group_value: null, group_count: '4' }, + { group_value: POST_A, group_count: '2' }, + { group_value: POST_B, group_count: '7' }, + ], + }); + // Text ties by text; `null` sits where its own count of 4 puts it, in the middle. + const byPost = await repo().countBy('postId', { orgId: ORG }); + expect([...byPost.keys()]).toEqual([POST_B, null, POST_A, POST_C]); + + client.on('group by "weight"', { + rows: [ + { group_value: '100', group_count: '2' }, + { group_value: '20', group_count: '2' }, + ], + }); + // Bigints numerically too — as text, '100' would sort before '20'. + expect([...(await repo().countBy('weight', { orgId: ORG })).keys()]).toEqual([20n, 100n]); + }); +}); diff --git a/packages/entity/src/count-by.ts b/packages/entity/src/count-by.ts new file mode 100644 index 000000000..1ffbbdf47 --- /dev/null +++ b/packages/entity/src/count-by.ts @@ -0,0 +1,148 @@ +// Single responsibility: what a grouped count is made of — which columns a count may be keyed by, +// how many groups one statement is allowed to answer with, and the order the map comes back in. +// Both drivers read those three rules from here, so a `countBy` against memory means exactly what +// a `countBy` against Postgres means; a rule added to one driver alone is the drift this file +// exists to prevent. + +import type { EntityCore } from './entity'; +import { EntityError } from './errors'; +import type { AnyColumn, ColumnKind } from './types'; + +/** + * How many groups one call may answer with. A grouped count answers a page's worth of keys, or a + * column with a handful of values; past that it is a report, and a report is paged. The statement + * therefore asks for one group more than this and the extra one is *refused* rather than dropped — + * a map that silently lost its tail reads exactly like a complete one, and a caller recounting + * from it would write the wrong number to every row it missed. + */ +export const MAX_GROUPS = 1000; + +/** + * The kinds a group key can be. A `Map` compares keys by identity for anything that is not a + * primitive, so a `timestamptz` (a `Date`) or a `jsonb` (an object) would file every row under a + * key no caller can look up again — the result would be a map that only ever answers `undefined`. + * `money` is two physical columns, which is not one value to group by at all. + */ +const GROUPABLE: ReadonlySet = new Set([ + 'uuid', + 'text', + 'char', + 'boolean', + 'integer', + 'bigint', +]); + +const groupableColumns = (entity: EntityCore): readonly string[] => + Object.entries(entity.$columns) + .filter(([, column]) => GROUPABLE.has(column.$meta.kind)) + .map(([property]) => property); + +/** + * Not `invariantViolated`: its fix opens `x entity explain`, which describes invariants nobody + * wrote here. What repairs this is one edit to the call — a different column, named in the message + * because the entity is the only place the answer lives. Only when this entity offers no such + * column does the fix become a command, and then it is `x entities describe`, which prints the + * kinds: there is no call to suggest, since every column it declares would be refused the same way. + */ +const notGroupable = ( + entity: EntityCore, + operation: string, + property: string, + reason: string, +): EntityError => { + const [first] = groupableColumns(entity); + return new EntityError({ + code: 'X_INVARIANT_VIOLATED', + cause: `${entity.$name}.${operation}('${property}'): ${reason}`, + fix: + first === undefined + ? `x entities describe ${entity.$name} --json # this entity declares no column a count can be keyed by` + : `${entity.$name}.${operation}('${first}') # group by one of: ${groupableColumns(entity).join(', ')}`, + }); +}; + +/** + * The bound, spelled as the call that stays under it. A grouped count of a foreign key is the + * point of this method, so the fix leads with the `in` predicate that bounds one — the shape a + * page-then-count loop collapses to. + */ +const tooManyGroups = ( + entity: EntityCore, + operation: string, + property: string, +): EntityError => + new EntityError({ + code: 'X_INVARIANT_VIOLATED', + cause: `${entity.$name}.${operation}('${property}') matched more than ${MAX_GROUPS} distinct values — that column is a key, not a grouping`, + fix: `${entity.$name}.andWhere('${property}', 'in', ).${operation}('${property}') # bound the values first; a whole-table breakdown is a report, and a report is paged`, + }); + +/** + * The column a count may be keyed by, or the refusal. Called by both drivers before the statement + * exists, so an ungroupable column is the same error whichever one is installed. + */ +export const groupColumnOf = ( + entity: EntityCore, + property: string, + operation: string, +): AnyColumn => { + const column = entity.$columns[property]; + if (column === undefined) { + throw notGroupable( + entity, + operation, + property, + `no column "${property}" on ${entity.$name} — pick from: ${Object.keys(entity.$columns).join(', ')}`, + ); + } + if (!GROUPABLE.has(column.$meta.kind)) { + throw notGroupable( + entity, + operation, + property, + `a ${column.$meta.kind} column is not a key a map can be looked up by`, + ); + } + return column; +}; + +/** + * The value a group is filed under: re-parsed by the column that declared it, exactly as + * `decodeRow` re-parses a row's own value — `int8` arrives as a string and would otherwise key the + * map by text where the in-memory driver keys it by a `bigint`. An absent value is `null`, which + * is the one group SQL's `group by` puts every NULL row in. + */ +export const groupValue = (column: AnyColumn, value: unknown): unknown => + value === null || value === undefined ? null : column.$parse(value); + +/** Ties: numbers and bigints numerically, everything else by its text, `null` last. */ +const byValue = (left: unknown, right: unknown): number => { + if (left === null) return right === null ? 0 : 1; + if (right === null) return -1; + if (typeof left === 'number' && typeof right === 'number') return left - right; + if (typeof left === 'bigint' && typeof right === 'bigint') { + return left < right ? -1 : left > right ? 1 : 0; + } + const [a, b] = [String(left), String(right)]; + return a < b ? -1 : a > b ? 1 : 0; +}; + +/** + * The map both drivers hand back: the biggest group first, ties by the value itself, `null` last. + * Ordered here rather than in the statement, because there is no order to inherit — a hash + * aggregate returns groups in whatever order it built them and a `Map` filled row by row returns + * them in insertion order, so the two drivers would disagree about a result they agree on. + * Sorting the groups (never the rows) costs nothing at this size and it is what makes + * "the largest bucket" readable off the front. + */ +export const countsFrom = ( + entity: EntityCore, + property: string, + operation: string, + groups: readonly (readonly [unknown, number])[], +): ReadonlyMap => { + if (groups.length > MAX_GROUPS) throw tooManyGroups(entity, operation, property); + return new Map( + [...groups].sort((left, right) => right[1] - left[1] || byValue(left[0], right[0])), + ); +}; diff --git a/packages/entity/src/index.ts b/packages/entity/src/index.ts index d246144c9..963d9b3ca 100644 --- a/packages/entity/src/index.ts +++ b/packages/entity/src/index.ts @@ -3,6 +3,7 @@ /** Re-exported so an `entity` file needs one import, not two. Same object as schema's. */ export type { Infer } from '@ultimat3/schema'; export { t } from '@ultimat3/schema'; +export type { BatchIterator } from './batch'; export type { TextOptions } from './columns'; export { boolean, @@ -66,7 +67,15 @@ export { } from './registry'; export type { EntityRelations, Relation, RelationKind, RelationMap } from './relations'; export { relationMap, relationNamed, relationsFor, relationsOf } from './relations'; -export type { FindManyArgs, Page, Repo, RepoOptions, Transactor, Tx } from './repo'; +export type { + FindManyArgs, + Page, + Repo, + RepoOptions, + Transactor, + Tx, + UpsertArgs, +} from './repo'; export { memoryRepo, memoryTransactor } from './repo'; export type { Seed, SeedContext, SeedOptions } from './seed'; export { defineSeed, seedId } from './seed'; diff --git a/packages/entity/src/pg-driver-batch.live.test.ts b/packages/entity/src/pg-driver-batch.live.test.ts new file mode 100644 index 000000000..1a6f487a4 --- /dev/null +++ b/packages/entity/src/pg-driver-batch.live.test.ts @@ -0,0 +1,198 @@ +// `inBatches(size)` against a real Postgres. `batch.test.ts` proves memory's own boundaries and +// (via a recording client) the statement text the Postgres driver sends; neither proves the server +// accepts that statement, that the seek is a real index scan rather than a scan Postgres refuses, +// or that closing a handle early really stops the next round trip. Skips unless +// `TEST_DATABASE_URL` is set, as every other live file in this package does. + +import { afterAll, beforeAll, describe, expect, test } from 'bun:test'; +import { + createPostgresClient, + generateMigration, + type PostgresClient, + raw, + setDbClient, + setStatementObserver, +} from '@ultimat3/db'; +import { text, uuid } from './columns'; +import { database } from './database'; +import { entity } from './entity'; +import { postgresDriver } from './pg-driver'; +import { clearRegistry } from './registry'; + +// `TEST_DATABASE_URL` only: `beforeAll`/`afterAll` here run `drop table … cascade`, so falling back +// to the app's own `DATABASE_URL` would hand this file whatever database a developer had exported. +const adminUrl = Bun.env['TEST_DATABASE_URL']; +const hasPostgres = typeof adminUrl === 'string' && adminUrl.length > 0; + +const orgs = entity('pg_batch_live_orgs', { + columns: { id: uuid().primaryKey(), slug: text({ max: 40 }).unique() }, +}); + +const posts = entity('pg_batch_live_posts', { + columns: { + id: uuid().primaryKey(), + orgId: uuid() + .references(() => orgs.id) + .tenant(), + title: text({ max: 80 }), + }, +}); + +const DROP = 'drop table if exists "pg_batch_live_posts", "pg_batch_live_orgs" cascade'; + +/** + * One `up` script out, one statement per `execute` in: every value in it is an identifier or a + * CHECK the entity wrote, none of which can contain a semicolon. + */ +const statementsOf = (script: string): readonly string[] => + script + .split(';\n') + .map((statement) => statement.trim()) + .filter((statement) => statement.length > 0); + +describe.skipIf(!hasPostgres)('live · postgres · inBatches', () => { + let client: PostgresClient; + + beforeAll(async () => { + client = createPostgresClient({ url: adminUrl ?? '' }); + setDbClient(client); + await client.execute(raw(DROP)); + const migration = generateMigration({ + // Declaration order is dependency order: the foreign key needs its target first. + entities: [orgs.$describe(), posts.$describe()], + name: 'live inBatches', + now: new Date('2026-08-13T00:00:00.000Z'), + }); + for (const statement of statementsOf(migration.up)) await client.execute(raw(statement)); + }); + + afterAll(async () => { + await client.execute(raw(DROP)); + await client.close(); + setDbClient(undefined); + clearRegistry(); + }); + + const db = () => database({ orgs, posts }, { driver: postgresDriver() }); + + /** A tenant nobody else writes to, so a whole-tenant walk is an assertion and not a race. */ + const newOrg = async (slug: string): Promise => + (await database({ orgs }, { driver: postgresDriver() }).orgs.insert({ slug })).id; + + /** Seven titled rows for `org` — the size the header's `[3, 3, 1]` claim is built on. */ + const seedSeven = async (org: string): Promise => { + const titles = ['A', 'B', 'C', 'D', 'E', 'F', 'G']; + for (const title of titles) await db().posts.insert({ orgId: org, title }); + return titles; + }; + + /** Every statement sent while `work` runs, filtered to the ones matching `match`. */ + const capturing = async (match: string, work: () => Promise): Promise<[T, string[]]> => { + const seen: string[] = []; + setStatementObserver({ + onStatement: (event) => { + if (event.text.includes(match)) seen.push(event.text); + }, + }); + try { + return [await work(), seen]; + } finally { + setStatementObserver(undefined); + } + }; + + test('every row comes back once, in order, batched exactly [3, 3, 1]', async () => { + const org = await newOrg('batch-walk'); + const titles = await seedSeven(org); + + const batches: string[][] = []; + for await (const batch of db() + .posts.where({ orgId: org }) + .orderBy('title', 'asc') + .inBatches(3)) { + batches.push(batch.map((row) => row.title)); + } + + expect(batches.map((batch) => batch.length)).toEqual([3, 3, 1]); + expect(batches.flat()).toEqual(titles); + }); + + test('the seek between batches is a predicate, never an offset', async () => { + const org = await newOrg('batch-seek'); + await seedSeven(org); + + const [, sent] = await capturing('from "pg_batch_live_posts"', async () => { + const seen: unknown[] = []; + for await (const batch of db() + .posts.where({ orgId: org }) + .orderBy('title', 'asc') + .inBatches(3)) { + seen.push(batch); + } + return seen; + }); + + expect(sent.length).toBeGreaterThan(1); + for (const statement of sent) expect(statement.toLowerCase()).not.toContain('offset'); + // The second-and-later statements carry the seek predicate this package writes instead. + expect(sent[1]).toContain('>'); + }); + + test('breaking out of the loop sends no further statement', async () => { + const org = await newOrg('batch-break'); + await seedSeven(org); + + const [, before] = await capturing('from "pg_batch_live_posts"', async () => { + for await (const batch of db() + .posts.where({ orgId: org }) + .orderBy('title', 'asc') + .inBatches(3)) { + expect(batch).toHaveLength(3); + break; + } + }); + expect(before).toHaveLength(1); + + // Confirmed against a second, independent capture rather than a running total: nothing queued + // by the loop's own cleanup fires after the `for await` has already returned control here. + const [, after] = await capturing('from "pg_batch_live_posts"', async () => { + // Deliberately empty — proving silence, not another read. + }); + expect(after).toHaveLength(0); + }); + + test('preload() composed with inBatches() preloads every batch, not only the first', async () => { + const org = await newOrg('batch-preload'); + await seedSeven(org); + + const [batches, orgReads] = await capturing('from "pg_batch_live_orgs"', async () => { + const seen: { title: string; org: { id: string; slug: string } }[][] = []; + for await (const batch of db() + .posts.where({ orgId: org }) + .orderBy('title', 'asc') + .preload('org') + .inBatches(3)) { + seen.push(batch); + } + return seen; + }); + + // Three batches ([3, 3, 1]), one preload statement each — composition, not a reproof of + // `preload()` itself, which is a different file's claim. + expect(batches).toHaveLength(3); + expect(orgReads).toHaveLength(3); + for (const batch of batches) { + for (const row of batch) expect(row.org).toEqual({ id: org, slug: 'batch-preload' }); + } + }); + + test('an unscoped chain refuses on its first batch, against a real statement', async () => { + await expect( + (async () => { + for await (const _batch of db().posts.inBatches(3)) { + // unreachable — the first `page()` call throws before a row is ever yielded. + } + })(), + ).rejects.toBeUltimateError('X_TENANCY_UNSCOPED'); + }); +}); diff --git a/packages/entity/src/pg-driver-bulk.live.test.ts b/packages/entity/src/pg-driver-bulk.live.test.ts new file mode 100644 index 000000000..cde0998b1 --- /dev/null +++ b/packages/entity/src/pg-driver-bulk.live.test.ts @@ -0,0 +1,322 @@ +// The bulk write path against a real Postgres. `pg-driver-bulk.test.ts` asserts the statement text +// a batch compiles to; nothing there proves the server accepts a multi-row `insert`, that `default` +// in a cell resolves to the column's own DEFAULT, or that `on conflict` finds the unique index the +// framework's own migration wrote. Skips unless `TEST_DATABASE_URL` is set, as the live suite does. + +import { afterAll, beforeAll, describe, expect, test } from 'bun:test'; +import { + createPostgresClient, + generateMigration, + type PostgresClient, + raw, + setDbClient, + setStatementObserver, +} from '@ultimat3/db'; +import { boolean, money, text, timestamp, uuid } from './columns'; +import { database } from './database'; +import { entity } from './entity'; +import { postgresDriver, postgresRepo } from './pg-driver'; +import { clearRegistry } from './registry'; + +// `TEST_DATABASE_URL` only: `beforeAll`/`afterAll` here run `drop table … cascade`, so falling back +// to the app's own `DATABASE_URL` would hand this file whatever database a developer had exported. +const adminUrl = Bun.env['TEST_DATABASE_URL']; +const hasPostgres = typeof adminUrl === 'string' && adminUrl.length > 0; + +const orgs = entity('pg_bulk_live_orgs', { + columns: { id: uuid().primaryKey(), slug: text({ max: 40 }).unique() }, +}); + +/** + * `reference` is unique PER TENANT, and that is the shape an updating upsert forces: the conflict + * target must carry the tenant column, so the constraint it is inferred against must too. A global + * `unique()` on `reference` would make the cross-tenant case below unwritable in the first place. + */ +const invoices = entity('pg_bulk_live_invoices', { + columns: { + id: uuid().primaryKey(), + orgId: uuid() + .references(() => orgs.id) + .tenant(), + reference: text({ max: 40 }), + total: money(), + paid: boolean().default(false), + note: text().nullable(), + issuedAt: timestamp().defaultNow(), + deletedAt: timestamp().nullable(), + }, + indexes: [{ on: ['orgId', 'reference'], unique: true }], +}); + +/** The other kind of unique index an upsert can land on: a composite primary key. */ +const tags = entity('pg_bulk_live_tags', { + columns: { + orgId: uuid() + .references(() => orgs.id) + .tenant(), + invoiceId: uuid().references(() => invoices.id, { onDelete: 'cascade' }), + label: text({ max: 40 }), + note: text().nullable(), + }, + primaryKey: ['orgId', 'invoiceId', 'label'], +}); + +type Invoice = typeof invoices.$row; + +const DROP = + 'drop table if exists "pg_bulk_live_tags", "pg_bulk_live_invoices", "pg_bulk_live_orgs" cascade'; + +const MONEY = { minor: 12_500n, currency: 'EUR' }; + +const UNIQUE_INDEX = 'pg_bulk_live_invoices_org_id_reference_key'; + +/** + * The constraint an updating upsert on a tenant-scoped entity has to be inferred against, written + * out. `generateMigration` cannot emit it: `EntityDescription.indexes` carries index NAMES only and + * `parseIndexName` recovers ONE column from `
___key`, so the generated DDL reads + * `("org_id_reference")` and Postgres answers `42703`. Reported, not papered over — this file is + * about the driver, and the driver needs the index to exist. + */ +const COMPOSITE_UNIQUE = `create unique index "${UNIQUE_INDEX}" + on "pg_bulk_live_invoices" ("org_id", "reference")`; + +/** + * The generator returns one `up` script; a driver executes one statement. Splitting on `;\n` is + * enough because every value in a generated clause is an identifier or a CHECK the entity wrote. + */ +const statementsOf = (script: string): readonly string[] => + script + .split(';\n') + .map((statement) => statement.trim()) + .filter((statement) => statement.length > 0); + +/** + * A row that leaves columns out — which is what puts a `default` in its cell. `Repo` is typed for + * whole rows, so the omission takes one assertion here; a batch assembled in JS arrives this way. + */ +const partial = (values: Partial): Invoice => values as Invoice; + +describe.skipIf(!hasPostgres)('live · postgres · bulk writes', () => { + let client: PostgresClient; + let migration: ReturnType; + + beforeAll(async () => { + client = createPostgresClient({ url: adminUrl ?? '' }); + setDbClient(client); + await client.execute(raw(DROP)); + migration = generateMigration({ + // Declaration order is dependency order: the foreign keys need their targets first. + entities: [orgs.$describe(), invoices.$describe(), tags.$describe()], + name: 'live bulk writes', + now: new Date('2026-08-12T00:00:00.000Z'), + }); + let replaced = 0; + for (const statement of statementsOf(migration.up)) { + // See `COMPOSITE_UNIQUE`: the generated form of this one index names a column that does not + // exist, so it is replaced rather than applied. + if (statement.includes(UNIQUE_INDEX)) { + replaced += 1; + continue; + } + await client.execute(raw(statement)); + } + // Counted, because a generator that renames that index would match nothing here and send the + // broken `create unique index` to the server — a `42703` that reads like a driver bug. + expect(replaced).toBe(1); + await client.execute(raw(COMPOSITE_UNIQUE)); + }); + + afterAll(async () => { + await client.execute(raw(DROP)); + await client.close(); + setDbClient(undefined); + clearRegistry(); + }); + + const db = () => database({ orgs, invoices, tags }, { driver: postgresDriver() }); + const repo = () => postgresRepo(invoices); + + /** A tenant nobody else wrote to, so a whole-tenant count is an assertion and not a race. */ + const newOrg = async (slug: string): Promise => + (await database({ orgs }, { driver: postgresDriver() }).orgs.insert({ slug })).id; + + /** What a call produced, and how many statements against one table it took. */ + const counting = async (match: string, work: () => Promise): Promise<[T, number]> => { + const seen: string[] = []; + setStatementObserver({ + onStatement: (event) => { + if (event.text.includes(match)) seen.push(event.text); + }, + }); + try { + return [await work(), seen.length]; + } finally { + setStatementObserver(undefined); + } + }; + + test('the server carries the unique constraints on conflict is inferred against', async () => { + // The premise of every upsert below, asserted against the catalog rather than against a script: + // whatever created them, both constraints are on this server or nothing here means anything. + const indexes = await client.query<{ indexdef: string }>( + raw(`select indexdef from pg_indexes where tablename = 'pg_bulk_live_invoices'`), + ); + expect(indexes.some((index) => index.indexdef.includes('(org_id, reference)'))).toBe(true); + // The generator does get a composite PRIMARY KEY right; it is only `indexes:` it cannot spell. + expect(migration.up).toContain('primary key ("org_id", "invoice_id", "label")'); + }); + + test('a batch round-trips through one multi-row insert', async () => { + const org = await newOrg('bulk-insert'); + const batch = ['BULK-1', 'BULK-2', 'BULK-3'].map((reference) => ({ + orgId: org, + reference, + total: MONEY, + })); + + const [written, sent] = await counting('insert into "pg_bulk_live_invoices"', () => + db().invoices.insertAll(batch), + ); + + // Three rows, one statement — the whole point of the call, against a server that counts. + expect(sent).toBe(1); + expect(written.map((row) => row.reference)).toEqual(['BULK-1', 'BULK-2', 'BULK-3']); + for (const row of written) { + // Every one of these came back from Postgres, not from the object that was passed in. + expect(row.total).toEqual({ minor: 12_500n, currency: 'EUR' }); + expect(row.paid).toBe(false); + expect(row.note).toBeNull(); + expect(row.issuedAt).toBeInstanceOf(Date); + expect(row.id).toMatch(/^[0-9a-f-]{36}$/); + } + expect(await repo().count({ orgId: org })).toBe(3); + }); + + test('a row that omits a column takes the DEFAULT the migration wrote for it', async () => { + const org = await newOrg('bulk-default'); + const stamped = new Date('2020-03-04T05:06:07.000Z'); + const rows = [ + partial({ orgId: org, reference: 'DEF-named', total: MONEY, paid: false, issuedAt: stamped }), + partial({ orgId: org, reference: 'DEF-omitted', total: MONEY, paid: false }), + ]; + + const [named, omitted] = await repo().insertAll(rows); + + // One column list for the statement, so the row that did not name `issued_at` says `default` + // in that cell — and `default` there is the column's own `now()`, resolved by the server. + expect(named?.issuedAt).toEqual(stamped); + expect(omitted?.issuedAt).toBeInstanceOf(Date); + expect(omitted?.issuedAt.getTime()).toBeGreaterThan(stamped.getTime()); + // Neither row named `id`, so the column is absent from the statement altogether and both keys + // are the server's: a batch cannot make a generated key stop being generated. + expect(named?.id).not.toBe(omitted?.id); + expect(omitted?.id).toMatch(/^[0-9a-f-]{36}$/); + }); + + test('a nothing upsert on a composite key inserts the new rows and leaves the stored one', async () => { + const org = await newOrg('bulk-nothing'); + const invoice = await db().invoices.insert({ orgId: org, reference: 'TAG-1', total: MONEY }); + const tag = (label: string, note: string | null) => ({ + orgId: org, + invoiceId: invoice.id, + label, + note, + }); + await db().tags.insertAll([tag('urgent', 'first')]); + + const written = await db().tags.upsertAll( + [tag('urgent', 'overwritten'), tag('archive', null), tag('review', null)], + { onConflict: ['orgId', 'invoiceId', 'label'], onMatch: 'nothing' }, + ); + + // `returning *` names only the rows the server inserted, so the collision is absent from it. + expect(written.map((row) => row.label).sort()).toEqual(['archive', 'review']); + expect(await db().tags.where({ orgId: org }).count()).toBe(3); + expect((await db().tags.where({ orgId: org, label: 'urgent' }).one())?.note).toBe('first'); + }); + + test('an update upsert keeps the stored key and cannot reach another tenant', async () => { + const org = await newOrg('bulk-update'); + const other = await newOrg('bulk-update-other'); + const stored = await db().invoices.insert({ orgId: org, reference: 'UP-1', total: MONEY }); + // The SAME reference under another tenant — legal, because the constraint is per tenant. This + // is the row a target of `['reference']` alone would have matched, overwritten and taken. + const theirs = await db().invoices.insert({ orgId: other, reference: 'UP-1', total: MONEY }); + const incoming = '00000000-0000-7000-8000-0000000000ff'; + + const [written] = await db().invoices.upsertAll( + [ + { + id: incoming, + orgId: org, + reference: 'UP-1', + total: { minor: 900n, currency: 'GBP' }, + note: 'settled', + paid: true, + }, + ], + { onConflict: ['orgId', 'reference'] }, + ); + + // The stored row's address survives the write: every foreign key pointing at it still hits it, + // and the id the caller happened to generate is not where the row moved to. + expect(written?.id).toBe(stored.id); + expect(written?.id).not.toBe(incoming); + expect(written?.note).toBe('settled'); + expect(written?.paid).toBe(true); + expect(written?.total).toEqual({ minor: 900n, currency: 'GBP' }); + // One row, not two: the upsert landed on the stored one rather than beside it. + expect(await repo().count({ orgId: org })).toBe(1); + + // And the other tenant's row is byte-for-byte what it was — same id, same values, still theirs. + expect(await repo().findById(theirs.id, { orgId: other })).toEqual(theirs); + expect(await repo().count({ orgId: other })).toBe(1); + }); + + test('a soft-deleted row still occupies its unique key, so an upsert collides with it', async () => { + const org = await newOrg('bulk-soft'); + const stored = await db().invoices.insert({ orgId: org, reference: 'SOFT-1', total: MONEY }); + await repo().delete(stored.id, { orgId: org }); + + const written = await db().invoices.upsertAll( + [{ orgId: org, reference: 'SOFT-1', total: MONEY }], + { onConflict: ['orgId', 'reference'], onMatch: 'nothing' }, + ); + + // The unique index is not partial, so a hidden row still owns `reference`: the insert collides, + // writes nothing, and never becomes a second row the app can no longer read. + expect(written).toEqual([]); + expect(await repo().findById(stored.id, { orgId: org })).toBeNull(); + + // One row, still stamped: the collision neither resurrected it nor sat down beside it. + const hidden = await repo().findMany({ orgId: org, includeDeleted: true }); + expect(hidden.rows).toHaveLength(1); + expect(hidden.rows[0]?.id).toBe(stored.id); + expect(hidden.rows[0]?.deletedAt).toBeInstanceOf(Date); + }); + + test('an update upsert onto that stamped row writes its columns and never its stamp', async () => { + const org = await newOrg('bulk-soft-update'); + const stored = await db().invoices.insert({ + orgId: org, + reference: 'SOFT-2', + total: MONEY, + note: 'before', + }); + await repo().delete(stored.id, { orgId: org }); + + // The row the caller passes carries `deletedAt: null` whether they wrote it or not — `$parse` + // fills every declared column — so this is the shape that would resurrect it. + const [written] = await db().invoices.upsertAll( + [{ orgId: org, reference: 'SOFT-2', total: MONEY, note: 'after' }], + { onConflict: ['orgId', 'reference'] }, + ); + + // `"deleted_at"` is not in the `do update set` list: the delete the app made stands, and the + // row is still invisible to an ordinary read. + expect(written?.note).toBe('after'); + expect(written?.deletedAt).toBeInstanceOf(Date); + expect(await repo().findById(stored.id, { orgId: org })).toBeNull(); + expect(await repo().count({ orgId: org })).toBe(0); + }); +}); diff --git a/packages/entity/src/pg-driver-bulk.test.ts b/packages/entity/src/pg-driver-bulk.test.ts new file mode 100644 index 000000000..f3ce28b0b --- /dev/null +++ b/packages/entity/src/pg-driver-bulk.test.ts @@ -0,0 +1,415 @@ +// Many rows, ONE statement. `insertAll`/`upsertAll` against a recording client, where the statement +// text is the whole claim: the tuple list, the `default` cell a row naming fewer columns takes, the +// `on conflict` clause each `onMatch` compiles to and the four refusals that precede it, and the +// chunking that keeps a wide batch inside Postgres's bind count. + +import { afterAll, beforeEach, describe, expect, test } from 'bun:test'; +import { createContext, isUltimateError, runWithContext } from '@ultimat3/core'; +import { createRecordingClient, type RecordingClient, setDbClient } from '@ultimat3/db'; +import { MAX_BIND_PARAMETERS } from './bulk-write'; +import { boolean, money, text, timestamp, uuid } from './columns'; +import { entity } from './entity'; +import { postgresRepo } from './pg-driver'; +import { allColumns } from './pg-row'; +import { clearRegistry } from './registry'; +import { memoryRepo, type UpsertArgs } from './repo'; + +const orgs = entity('pg_bulk_orgs', { + columns: { id: uuid().primaryKey(), slug: text({ max: 40 }).unique() }, +}); + +/** + * Two unique constraints, since an upsert may only target one that exists: `reference` alone, which + * a tenant-scoped entity can only resolve with `'nothing'`, and `(orgId, reference)`, which carries + * the tenant column and is therefore the only target `'update'` accepts here. + */ +const invoices = entity('pg_bulk_invoices', { + columns: { + id: uuid().primaryKey(), + orgId: uuid() + .references(() => orgs.id) + .tenant(), + reference: text({ max: 40 }).unique(), + total: money(), + paid: boolean().default(false), + note: text().nullable(), + issuedAt: timestamp().defaultNow(), + deletedAt: timestamp().nullable(), + }, + indexes: [{ on: ['orgId', 'reference'], unique: true }], +}); + +/** + * A composite primary key — the other kind of unique index an upsert lands on. A batch naming only + * the three key columns is the shape `'update'` has nothing to write on: target and key are both + * excluded from the set list by construction. + */ +const likes = entity('pg_bulk_likes', { + columns: { + orgId: uuid().tenant(), + postId: uuid(), + memberId: uuid(), + createdAt: timestamp().defaultNow(), + }, + primaryKey: ['orgId', 'postId', 'memberId'], +}); + +type Invoice = typeof invoices.$row; +type Like = typeof likes.$row; + +const idAt = (index: number): string => + `00000000-0000-7000-8000-${String(index).padStart(12, '0')}`; + +const ORG = idAt(1); +const STAMPED = new Date('2026-01-02T03:04:05.000Z'); + +const ROW: Invoice = { + id: idAt(101), + orgId: ORG, + reference: 'INV-101', + total: { minor: 129900n, currency: 'EUR' }, + paid: false, + note: null, + issuedAt: STAMPED, + deletedAt: null, +}; + +const invoice = (index: number, over: Partial = {}): Invoice => ({ + ...ROW, + id: idAt(index), + reference: `INV-${index}`, + ...over, +}); + +/** A row leaving columns out — a `default` cell. `Repo` is typed for whole rows, hence the cast. */ +const partial = (values: Partial): Invoice => values as Invoice; + +/** One row in three spellings: the physical column, the value bound, what Bun.SQL hands back. */ +const cellsOf = (row: Invoice): readonly (readonly [string, unknown, unknown])[] => [ + ['id', row.id, row.id], + ['org_id', row.orgId, row.orgId], + ['reference', row.reference, row.reference], + ['total_minor', row.total.minor, String(row.total.minor)], + ['total_currency', row.total.currency, row.total.currency], + ['paid', row.paid, row.paid], + ['note', row.note, row.note], + ['issued_at', row.issuedAt, row.issuedAt.toISOString()], + ['deleted_at', row.deletedAt, row.deletedAt?.toISOString() ?? null], +]; + +const physicalOf = (row: Invoice): Record => + Object.fromEntries(cellsOf(row).map(([column, , stored]) => [column, stored])); +const COLUMNS = `(${cellsOf(ROW) + .map(([column]) => `"${column}"`) + .join(', ')})`; + +let client: RecordingClient; + +beforeEach(() => { + client = createRecordingClient(); + setDbClient(client); +}); + +afterAll(() => { + setDbClient(undefined); + clearRegistry(); +}); + +const repo = () => postgresRepo(invoices); +const lastText = (): string => client.texts.at(-1) ?? ''; +const lastValues = (): readonly unknown[] => client.statements.at(-1)?.values ?? []; +const tuplesOf = (text: string): readonly string[] => text.split('), ('); + +/** An `onConflict` `keyof Row & string` refuses; a JS caller reaches the runtime guard for free. */ +const targeting = (...onConflict: string[]): UpsertArgs => + ({ onConflict }) as UpsertArgs; + +type Refusal = Record<'code' | 'cause' | 'fix', string>; + +/** What a refusal says. A code is the contract, and a `fix:` that names no edit is not one. */ +const refusal = async (call: Promise): Promise => { + try { + await call; + return { code: 'resolved', cause: '', fix: '' }; + } catch (error) { + if (!isUltimateError(error)) return { code: String(error), cause: '', fix: '' }; + return { code: error.code, cause: error.cause, fix: error.fix }; + } +}; + +describe('a batch is one statement, and both drivers mean the same by it', () => { + test('three rows are one statement; the same three, one at a time, are three', async () => { + const batch = [invoice(2), invoice(3), invoice(4)]; + client.on('insert into', { rows: batch.map((row) => physicalOf(row)) }); + + const written = await repo().insertAll(batch); + + expect(client.statements).toHaveLength(1); + expect(lastText()).toStartWith(`insert into "pg_bulk_invoices" ${COLUMNS} values (`); + expect(tuplesOf(lastText())).toHaveLength(3); + expect(lastText()).toEndWith('returning *'); + expect(lastValues()).toEqual(batch.flatMap((row) => cellsOf(row).map(([, bind]) => bind))); + expect(written).toEqual(batch); + + // The N+1 this call removes: the loop it replaces is one statement per row. + for (const row of batch) await repo().insert(row); + expect(client.statements).toHaveLength(4); + }); + + test('every value is bound, never spliced into the text', async () => { + const evil = "x'; drop table pg_bulk_invoices; --"; + await repo().insertAll([invoice(2), invoice(3, { reference: evil })]); + + expect(lastText()).not.toContain('drop table'); + expect(lastText()).toEndWith('($10, $11, $12, $13, $14, $15, $16, $17, $18) returning *'); + expect(lastValues()[11]).toBe(evil); + }); + + test('money binds as two parameters, in the positions its two columns occupy', async () => { + await repo().insertAll([invoice(2), invoice(3, { total: { minor: 5n, currency: 'USD' } })]); + + expect(lastText()).toContain('"total_minor", "total_currency"'); + expect(lastValues().slice(3, 5)).toEqual([129900n, 'EUR']); + expect(lastValues().slice(12, 14)).toEqual([5n, 'USD']); + }); + + test('no rows is no statement, and one row is the statement insert always sent', async () => { + expect(await repo().insertAll([])).toEqual([]); + expect(client.statements).toHaveLength(0); + + client.on('insert into', { rows: [physicalOf(ROW)] }); + await repo().insert(ROW); + const alone = client.statements.at(-1); + await repo().insertAll([ROW]); + + expect(client.statements.at(-1)?.text).toBe(alone?.text ?? ''); + expect(client.statements.at(-1)?.values).toEqual(alone?.values ?? []); + }); + + test('a row that names fewer columns takes the column default in its cell', async () => { + const sparse = partial({ orgId: ORG, reference: 'INV-3', total: ROW.total }); + await repo().insertAll([invoice(2), sparse]); + + // One column list for the statement, so the row naming three of them says `default` for the + // rest — which is what makes a row inside a batch mean what it means on its own. + expect(tuplesOf(lastText())[0]).not.toContain('default'); + expect(lastText()).toContain( + '(default, $10, $11, $12, $13, default, default, default, default)', + ); + expect(lastValues()).toHaveLength(13); + }); + + test('past the bind count it is whole statements whose tuples are the batch, in order', async () => { + // Computed, never a magic number: the limit is Postgres's and the width is this entity's. + const width = allColumns(invoices).length; + const perStatement = Math.floor(MAX_BIND_PARAMETERS / width); + const batch = Array.from({ length: perStatement + 2 }, (_, index) => invoice(index + 1000)); + + await repo().insertAll(batch); + + expect(client.statements).toHaveLength(2); + expect(client.statements[0]?.values).toHaveLength(perStatement * width); + expect(client.statements[1]?.values).toHaveLength(2 * width); + // Each one is a whole statement, never a fragment of one Postgres would refuse. + for (const statement of client.statements) { + expect(statement.text).toStartWith(`insert into "pg_bulk_invoices" ${COLUMNS} values (`); + expect(statement.text).toEndWith('returning *'); + } + // The references, in the order the batch named them: nothing dropped, nothing reordered. + const bound = client.statements.flatMap((statement) => [...statement.values]); + expect(bound.filter((_, index) => index % width === 2)).toEqual( + batch.map((row) => row.reference), + ); + }); + + test('decodes, and reports what it wrote, exactly as the in-memory driver does', async () => { + const plain = [invoice(2), invoice(3)]; + client.on('insert into', { rows: plain.map((row) => physicalOf(row)) }); + expect(await repo().insertAll(plain)).toEqual(await memoryRepo(invoices).insertAll(plain)); + + const stored = invoice(2); + const colliding = [invoice(4, { reference: stored.reference }), invoice(3)]; + // `returning *` names only the rows the server inserted: the collision is not one of them. + client.on('insert into', { rows: [physicalOf(invoice(3))] }); + const args = { onConflict: ['reference'] as const, onMatch: 'nothing' as const }; + + const written = await repo().upsertAll(colliding, args); + + expect(written).toEqual(await memoryRepo(invoices, [stored]).upsertAll(colliding, args)); + expect(written.map((row) => row.reference)).toEqual(['INV-3']); + }); +}); + +describe('upsertAll() compiles the conflict clause', () => { + const batch = [invoice(2), invoice(3)]; + + test('the two onMatch modes are do nothing and a set list that spares key and target', async () => { + await repo().upsertAll(batch, { onConflict: ['reference'], onMatch: 'nothing' }); + expect(lastText()).toEndWith(' on conflict ("reference") do nothing returning *'); + + await repo().upsertAll(batch, { onConflict: ['orgId', 'reference'] }); + expect(lastText()).toEndWith( + ' on conflict ("org_id", "reference") do update set' + + ' "total_minor" = excluded."total_minor", "total_currency" = excluded."total_currency",' + + ' "paid" = excluded."paid", "note" = excluded."note", "issued_at" = excluded."issued_at"' + + ' returning *', + ); + // Neither `"id"` nor `"reference"` is in that list: how the row was found and where it lives. + // `"org_id"` is absent too, because the tenant column is part of the target now — so the + // statement can only land on a row of the tenant that sent it, and this is the assertion that + // stops a cross-tenant rewrite coming back as a set-list entry. + expect(lastText()).not.toContain('excluded."org_id"'); + // And `"deleted_at"` is the third: a soft-deleted row still occupies its conflict target, so + // setting the stamp from `excluded` would clear a delete the app made and hand the row back. + // The column is still in the INSERT list above — a row that collides with nothing keeps it. + expect(lastText()).not.toContain('excluded."deleted_at"'); + expect(lastText()).toContain('"deleted_at") values'); + }); + + test('a collision leaves the stamp where it was, in both drivers', async () => { + // The rule lives in `bulk-write.ts`, so the in-memory driver reads the same answer rather than + // a second copy of it: the stored row stays deleted and takes the batch's other columns. + const deleted = invoice(2, { deletedAt: STAMPED, note: 'gone' }); + const incoming = invoice(2, { note: 'back?' }); + const args = { onConflict: ['orgId', 'reference'] as const }; + + const [merged] = await memoryRepo(invoices, [deleted]).upsertAll([incoming], args); + + expect(merged?.deletedAt).toEqual(STAMPED); + expect(merged?.note).toBe('back?'); + }); + + test('a composite conflict target names every column of the index, and sets none of them', async () => { + // The entity's own composite primary key — a constraint that exists, which is the only kind + // there is: an arbitrary pair of columns is `42P10`, with no index to match it. + const like = { orgId: ORG, postId: idAt(7), memberId: idAt(8), createdAt: STAMPED }; + await postgresRepo(likes).upsertAll([like], { onConflict: ['orgId', 'postId', 'memberId'] }); + + expect(lastText()).toEndWith( + ' on conflict ("org_id", "post_id", "member_id")' + + ' do update set "created_at" = excluded."created_at" returning *', + ); + }); + + test('a batch repeating a conflict target is refused under update, allowed under nothing', async () => { + const twice = [invoice(2), invoice(3, { reference: 'INV-2' })]; + const args = { onConflict: ['orgId', 'reference'] as const }; + + expect((await refusal(repo().upsertAll(twice, args))).code).toBe('X_INVARIANT_VIOLATED'); + expect(client.statements).toHaveLength(0); + + // `do nothing` skips the repeat on the server, so there is nothing here to refuse. + await repo().upsertAll(twice, { ...args, onMatch: 'nothing' }); + expect(client.statements).toHaveLength(1); + }); + + test('a target no declared unique constraint matches is refused, naming 42P10', async () => { + // Two ordinary columns: Postgres can infer no index for them, and the refusal has to name the + // declaration that would make the target real — `42P10` alone names nothing to edit. + const refused = await refusal(repo().upsertAll([ROW], targeting('reference', 'note'))); + + expect(refused.code).toBe('X_INVARIANT_VIOLATED'); + expect(refused.cause).toContain('42P10'); + expect(refused.cause).toContain('(org_id, reference)'); + expect(refused.fix).toContain("indexes: [{ on: ['reference', 'note'], unique: true }]"); + expect(client.statements).toHaveLength(0); + }); + + test('an updating target that misses the tenant column is refused; nothing is not', async () => { + // `reference` is genuinely unique, so this is not a bad target — it is one that matches another + // tenant's row. Nothing else scopes the statement: `upsertAll` builds no read plan, and + // `on conflict … do update` carries no `where`. + const refused = await refusal(repo().upsertAll(batch, { onConflict: ['reference'] })); + + expect(refused.code).toBe('X_TENANCY_UNSCOPED'); + expect(refused.cause).toContain( + 'a row stored by another tenant would match and be overwritten', + ); + expect(refused.fix).toContain("onConflict: ['orgId', 'reference']"); + expect(client.statements).toHaveLength(0); + + // `do nothing` writes nothing to a row it does not own, so the same target is legal there. + await repo().upsertAll(batch, { onConflict: ['reference'], onMatch: 'nothing' }); + expect(client.statements).toHaveLength(1); + }); + + test('an uneven batch is refused under update, and keeps its default cell under nothing', async () => { + const uneven = [invoice(2), partial({ orgId: ORG, reference: 'INV-3', total: ROW.total })]; + const args = { onConflict: ['orgId', 'reference'] as const }; + + // Why it cannot be waved through: `excluded."id"` for the row that omitted `id` is that + // column's default, so "leave the stored value alone" is not what the server would do. + const refused = await refusal(repo().upsertAll(uneven, args)); + expect(refused.code).toBe('X_INVARIANT_VIOLATED'); + expect(refused.cause).toContain('does not name "id"'); + expect(client.statements).toHaveLength(0); + + // `do nothing` overwrites nothing, so the cell means what it means in a plain `insertAll`. + await repo().upsertAll(uneven, { ...args, onMatch: 'nothing' }); + expect(lastText()).toContain( + '(default, $10, $11, $12, $13, default, default, default, default)', + ); + }); + + test('an empty target, an undeclared column and nothing-to-set each refuse before any statement', async () => { + // Only the key columns, so nothing this batch names is outside the address `'update'` may not + // move — and a statement overwriting nothing would report rows it had not written. + const keyOnly = { orgId: ORG, postId: idAt(7), memberId: idAt(8) } as Like; + const target = { onConflict: ['orgId', 'postId', 'memberId'] as const }; + + const empty = await refusal(repo().upsertAll([ROW], targeting())); + expect(empty.code).toBe('X_INVARIANT_VIOLATED'); + expect(empty.fix).toContain("pg_bulk_invoices.upsertAll(rows, { onConflict: [''] })"); + + const undeclared = await refusal(repo().upsertAll([ROW], targeting('secret'))); + expect(undeclared.code).toBe('X_INVARIANT_VIOLATED'); + expect(undeclared.fix).toContain('x entity explain pg_bulk_invoices'); + + const nothingToSet = await refusal(postgresRepo(likes).upsertAll([keyOnly], target)); + expect(nothingToSet.code).toBe('X_INVARIANT_VIOLATED'); + expect(nothingToSet.fix).toContain("onMatch: 'nothing'"); + + expect(client.statements).toHaveLength(0); + }); +}); + +describe('a bulk write drops what the request preloaded', () => { + const newOrg = [{ id: idAt(9), slug: 'new' }]; + + /** + * A page of invoices leaves its `orgId` values behind, so the lookup after it costs one widened + * statement. The write in the middle is the only variable: one to `pg_bulk_orgs` goes through + * `writing()`, which drops those rows first, so the second lookup reads again. + */ + const pageLookupWriteLookup = async (write: () => Promise): Promise => { + // Self-contained, so two of these in one test each count their own statements. + client.reset(); + client.on('from "pg_bulk_invoices"', { + rows: [physicalOf(invoice(2)), physicalOf(invoice(3))], + }); + client.on('from "pg_bulk_orgs"', { rows: [{ id: ORG, slug: 'acme' }] }); + client.on('insert into "pg_bulk_orgs"', { rows: newOrg }); + await runWithContext(createContext(), async () => { + await repo().findMany({ orgId: ORG, limit: 2 }); + await postgresRepo(orgs).findById(ORG); + await write(); + await postgresRepo(orgs).findById(ORG); + }); + return client.statements.length; + }; + + test('insertAll and upsertAll both re-read the row a page had already resolved', async () => { + // Page, preload, write, and the read the write forced. + expect(await pageLookupWriteLookup(() => postgresRepo(orgs).insertAll(newOrg))).toBe(4); + const upsert = () => + postgresRepo(orgs).upsertAll(newOrg, { onConflict: ['slug'], onMatch: 'nothing' }); + expect(await pageLookupWriteLookup(upsert)).toBe(4); + }); + + test('a bulk write to another entity leaves the preload intact', async () => { + // Three, not four: the second lookup is memory, so the preload is what is being counted — + // without it every case here would read four and prove nothing. + expect(await pageLookupWriteLookup(() => repo().insertAll([invoice(5)]))).toBe(3); + expect(client.texts[1]).toContain('"id" in ($1)'); + }); +}); diff --git a/packages/entity/src/pg-driver-count.live.test.ts b/packages/entity/src/pg-driver-count.live.test.ts new file mode 100644 index 000000000..f989cafd8 --- /dev/null +++ b/packages/entity/src/pg-driver-count.live.test.ts @@ -0,0 +1,214 @@ +// The grouped count against a real Postgres. `count-by-statements.test.ts` asserts the text a +// `countBy` compiles to; nothing there proves the server accepts `select … as group_value, +// count(*) as group_count … group by … limit $n`, that the numbers are right over real rows, or +// that an int8 count arrives as a `number`. Skips unless `TEST_DATABASE_URL` is set. + +import { afterAll, beforeAll, describe, expect, test } from 'bun:test'; +import { + createPostgresClient, + generateMigration, + type PostgresClient, + raw, + setDbClient, + setStatementObserver, +} from '@ultimat3/db'; +import { integer, text, timestamp, uuid } from './columns'; +import { database } from './database'; +import { entity } from './entity'; +import { postgresDriver, postgresRepo } from './pg-driver'; +import { clearRegistry } from './registry'; + +// `TEST_DATABASE_URL` only: `beforeAll`/`afterAll` run `drop table … cascade`, so falling back to +// the app's own `DATABASE_URL` would hand this file whatever database a developer had exported. +const adminUrl = Bun.env['TEST_DATABASE_URL']; +const hasPostgres = typeof adminUrl === 'string' && adminUrl.length > 0; + +const orgs = entity('pg_count_live_orgs', { + columns: { id: uuid().primaryKey(), slug: text({ max: 40 }).unique() }, +}); + +/** + * One row per reaction — the table a per-post `count()` loop is the N+1 over. `postId` is nullable + * on purpose: SQL files every NULL row under one group, and that group has to survive the trip + * back as the map key `null`, which is the one group value no column ever parses. + */ +const likes = entity('pg_count_live_likes', { + columns: { + id: uuid().primaryKey(), + orgId: uuid() + .references(() => orgs.id) + .tenant(), + postId: uuid().nullable(), + reaction: text({ max: 20 }), + score: integer(), + deletedAt: timestamp().nullable(), + }, +}); + +const DROP = 'drop table if exists "pg_count_live_likes", "pg_count_live_orgs" cascade'; + +const POST_A = '00000000-0000-7000-8000-0000000000aa'; +const POST_B = '00000000-0000-7000-8000-0000000000bb'; + +/** One `up` script out, one statement per `execute` in: every value in it is an identifier. */ +const statementsOf = (script: string): readonly string[] => + script + .split(';\n') + .map((statement) => statement.trim()) + .filter((statement) => statement.length > 0); + +describe.skipIf(!hasPostgres)('live · postgres · grouped counts', () => { + let client: PostgresClient; + + beforeAll(async () => { + client = createPostgresClient({ url: adminUrl ?? '' }); + setDbClient(client); + await client.execute(raw(DROP)); + const migration = generateMigration({ + // Declaration order is dependency order: the foreign key needs its target first. + entities: [orgs.$describe(), likes.$describe()], + name: 'live grouped counts', + now: new Date('2026-08-13T00:00:00.000Z'), + }); + for (const statement of statementsOf(migration.up)) await client.execute(raw(statement)); + }); + + afterAll(async () => { + await client.execute(raw(DROP)); + await client.close(); + setDbClient(undefined); + clearRegistry(); + }); + + const db = () => database({ orgs, likes }, { driver: postgresDriver() }); + const repo = () => postgresRepo(likes); + + /** What a call produced, and every statement it sent that matched — the N+1 claim needs both. */ + const counting = async (match: string, work: () => Promise): Promise<[T, string[]]> => { + const seen: string[] = []; + setStatementObserver({ + onStatement: (event) => { + if (event.text.includes(match)) seen.push(event.text); + }, + }); + try { + return [await work(), seen]; + } finally { + setStatementObserver(undefined); + } + }; + + const newOrg = async (slug: string): Promise => + (await database({ orgs }, { driver: postgresDriver() }).orgs.insert({ slug })).id; + + /** + * A tenant nobody else writes to, so every count below is an assertion and not a race: three + * rows on post A, two on post B, one on no post at all. + */ + const seed = async (slug: string): Promise => { + const org = await newOrg(slug); + await db().likes.insertAll([ + { orgId: org, postId: POST_A, reaction: 'like', score: 1 }, + { orgId: org, postId: POST_A, reaction: 'like', score: 2 }, + { orgId: org, postId: POST_A, reaction: 'wow', score: 2 }, + { orgId: org, postId: POST_B, reaction: 'like', score: 1 }, + { orgId: org, postId: POST_B, reaction: 'wow', score: 3 }, + { orgId: org, postId: null, reaction: 'like', score: 1 }, + ]); + return org; + }; + + test('the server accepts the aliased group statement and answers what count() answers', async () => { + const org = await seed('counts'); + + const [counts, sent] = await counting(' group by ', () => + db().likes.where({ orgId: org }).countBy('postId'), + ); + + // It ran, which is the half no recording client can assert: `as group_value` and + // `count(*) as group_count` are names Postgres either accepts or answers 42601 for. + expect(sent).toHaveLength(1); + expect(sent[0]).toContain('as group_value'); + expect(sent[0]).toContain('count(*) as group_count'); + expect([...counts]).toEqual([ + [POST_A, 3], + [POST_B, 2], + [null, 1], + ]); + // And it agrees with the per-value `count()` loop it stands in for, on the same server. + for (const postId of [POST_A, POST_B]) { + const alone = await repo().count({ + orgId: org, + where: [{ column: 'postId', op: 'eq', value: postId }], + }); + expect(counts.get(postId)).toBe(alone); + } + }); + + test('int8 comes back as a number, not the string Bun.SQL hands over', async () => { + const org = await seed('int8'); + + const counts = await db().likes.where({ orgId: org }).countBy('postId'); + + expect(counts.size).toBe(3); + expect([...counts.values()].every((value) => typeof value === 'number')).toBe(true); + // The mistake the decode prevents, spelled as the arithmetic a caller writes: '3' + 1 is '31'. + expect((counts.get(POST_A) ?? 0) + 1).toBe(4); + }); + + test('a soft-deleted row is not counted, and includeDeleted brings it back', async () => { + const org = await seed('soft'); + // Exactly one row: post A with score 1. Stamped, not removed — the row is still on the table. + expect(await repo().deleteWhere({ orgId: org, postId: POST_A, score: 1 })).toBe(1); + + // The `deleted_at is null` clause is what does this, on the server, over a row still there. + expect((await db().likes.where({ orgId: org }).countBy('postId')).get(POST_A)).toBe(2); + expect((await repo().countBy('postId', { orgId: org, includeDeleted: true })).get(POST_A)).toBe( + 3, + ); + }); + + test('another tenant is a different breakdown, never a shared one', async () => { + const org = await seed('scoped-ours'); + const other = await newOrg('scoped-theirs'); + // The same post under the other tenant, so an unscoped statement reads 4 for both of them. + await db().likes.insert({ orgId: other, postId: POST_A, reaction: 'like', score: 9 }); + + const ours = await db().likes.where({ orgId: org }).countBy('postId'); + const theirs = await db().likes.where({ orgId: other }).countBy('postId'); + + expect(ours.get(POST_A)).toBe(3); + expect([...theirs]).toEqual([[POST_A, 1]]); + }); + + test('a text key and an integer key come back as the values they were stored as', async () => { + const org = await seed('kinds'); + + expect([...(await db().likes.where({ orgId: org }).countBy('reaction'))]).toEqual([ + ['like', 4], + ['wow', 2], + ]); + // int4 out of the server is a number here: the map is keyed by 1, never by '1'. + expect([...(await db().likes.where({ orgId: org }).countBy('score'))]).toEqual([ + [1, 3], + [2, 2], + [3, 1], + ]); + }); + + test('a bounded breakdown is one statement where the loop it replaces is one per value', async () => { + const org = await seed('bounded'); + + const [counts, sent] = await counting('from "pg_count_live_likes"', () => + db().likes.where({ orgId: org }).andWhere('postId', 'in', [POST_A, POST_B]).countBy('postId'), + ); + + // The shape the too-many-groups refusal tells an author to write, and the whole point of the + // call: one statement for a breakdown a `count()` per post would have paid two for. + expect(sent).toHaveLength(1); + expect([...counts]).toEqual([ + [POST_A, 3], + [POST_B, 2], + ]); + }); +}); diff --git a/packages/entity/src/pg-driver-preload.live.test.ts b/packages/entity/src/pg-driver-preload.live.test.ts new file mode 100644 index 000000000..25f3f00b3 --- /dev/null +++ b/packages/entity/src/pg-driver-preload.live.test.ts @@ -0,0 +1,249 @@ +// `preload()` against a real Postgres. `preload-statements.test.ts` proves the statement *shape* +// against a recording client; nothing there proves the server accepts the `in (…)` it renders, +// that a cross-tenant or soft-deleted target really excludes, or that the attached rows round-trip +// through `decodeRow`. Skips unless `TEST_DATABASE_URL` is set. + +import { afterAll, beforeAll, describe, expect, test } from 'bun:test'; +import { + createPostgresClient, + generateMigration, + type PostgresClient, + raw, + setDbClient, + setStatementObserver, +} from '@ultimat3/db'; +import { text, timestamp, uuid } from './columns'; +import { database } from './database'; +import { entity } from './entity'; +import { postgresDriver } from './pg-driver'; +import { clearRegistry } from './registry'; + +// `TEST_DATABASE_URL` only: `beforeAll`/`afterAll` run `drop table … cascade`, so falling back to +// the app's own `DATABASE_URL` would hand this file whatever database a developer had exported. +const adminUrl = Bun.env['TEST_DATABASE_URL']; +const hasPostgres = typeof adminUrl === 'string' && adminUrl.length > 0; + +const orgs = entity('pg_preload_live_orgs', { + columns: { id: uuid().primaryKey(), slug: text({ max: 40 }).unique() }, +}); + +const members = entity('pg_preload_live_members', { + columns: { + id: uuid().primaryKey(), + orgId: uuid() + .references(() => orgs.id) + .tenant(), + email: text({ max: 120 }), + deletedAt: timestamp().nullable(), + }, +}); + +const posts = entity('pg_preload_live_posts', { + columns: { + id: uuid().primaryKey(), + orgId: uuid() + .references(() => orgs.id) + .tenant(), + authorId: uuid().references(() => members.id), + /** Nullable on purpose: the missing-key case a `belongsTo` has to attach `null` for. */ + reviewerId: uuid() + .references(() => members.id) + .nullable(), + title: text({ max: 120 }), + }, +}); + +const DROP = + 'drop table if exists "pg_preload_live_posts", "pg_preload_live_members", ' + + '"pg_preload_live_orgs" cascade'; + +/** One `up` script out, one statement per `execute` in: every value in it is an identifier. */ +const statementsOf = (script: string): readonly string[] => + script + .split(';\n') + .map((statement) => statement.trim()) + .filter((statement) => statement.length > 0); + +/** + * A `hasMany` is named for the entity the rows come from, and both inbound keys want that name — + * so every member of the group takes its long form, exactly as `preload.test.ts` spells it. + */ +const BY_AUTHOR = 'pg_preload_live_postsByAuthor'; +const BY_REVIEWER = 'pg_preload_live_postsByReviewer'; + +const field = (row: unknown, key: string): unknown => (row as Record)[key]; + +describe.skipIf(!hasPostgres)('live · postgres · preload', () => { + let client: PostgresClient; + + beforeAll(async () => { + client = createPostgresClient({ url: adminUrl ?? '' }); + setDbClient(client); + await client.execute(raw(DROP)); + const migration = generateMigration({ + // Declaration order is dependency order: both foreign keys need their targets first. + entities: [orgs.$describe(), members.$describe(), posts.$describe()], + name: 'live preload', + now: new Date('2026-08-13T00:00:00.000Z'), + }); + for (const statement of statementsOf(migration.up)) await client.execute(raw(statement)); + }); + + afterAll(async () => { + await client.execute(raw(DROP)); + await client.close(); + setDbClient(undefined); + clearRegistry(); + }); + + const db = () => database({ orgs, members, posts }, { driver: postgresDriver() }); + + /** What a call produced, and every statement it sent that matched — the count claim needs both. */ + const counting = async (match: string, work: () => Promise): Promise<[T, string[]]> => { + const seen: string[] = []; + setStatementObserver({ + onStatement: (event) => { + if (event.text.includes(match)) seen.push(event.text); + }, + }); + try { + return [await work(), seen]; + } finally { + setStatementObserver(undefined); + } + }; + + const newOrg = async (slug: string): Promise => + (await database({ orgs }, { driver: postgresDriver() }).orgs.insert({ slug })).id; + + /** A tenant nobody else writes to, plus its two members and three cross-linked posts. */ + const seed = async (slug: string) => { + const org = await newOrg(slug); + const ana = (await db().members.insert({ orgId: org, email: `ana-${slug}@example.com` })).id; + const ben = (await db().members.insert({ orgId: org, email: `ben-${slug}@example.com` })).id; + const p1 = (await db().posts.insert({ orgId: org, authorId: ana, title: 'First' })).id; + const p2 = ( + await db().posts.insert({ orgId: org, authorId: ana, reviewerId: ben, title: 'Second' }) + ).id; + const p3 = (await db().posts.insert({ orgId: org, authorId: ben, title: 'Third' })).id; + return { org, ana, ben, p1, p2, p3 }; + }; + + test('a belongsTo attaches the related row, a missing key attaches null', async () => { + const { org, ana, ben } = await seed('belongs-to'); + + const rows = await db().posts.where({ orgId: org }).orderBy('title').preload('author').all(); + + expect(rows).toHaveLength(3); + expect(field(rows[0], 'author')).toMatchObject({ id: ana }); + expect(field(rows[2], 'author')).toMatchObject({ id: ben }); + + const withReviewer = await db() + .posts.where({ orgId: org }) + .orderBy('title') + .preload('reviewer') + .all(); + + // 'First' has no reviewer at all: null, and present, never absent. + expect(field(withReviewer[0], 'reviewer')).toBeNull(); + expect('reviewer' in (withReviewer[0] as object)).toBe(true); + // 'Second' resolves to the row Postgres actually stored for it. + expect(field(withReviewer[1], 'reviewer')).toMatchObject({ id: ben }); + }); + + test('a hasMany attaches an array, empty when the page has none', async () => { + const { org, ana, ben, p1, p2, p3 } = await seed('has-many'); + + const byAuthor = await db() + .members.where({ orgId: org }) + .orderBy('email') + .preload(BY_AUTHOR) + .all(); + const anaRow = byAuthor.find((row) => field(row, 'id') === ana); + const benRow = byAuthor.find((row) => field(row, 'id') === ben); + // The attached array is not sorted here: it arrives in the related read's own total order, + // which is that entity's primary key ascending — and a uuid sorts the same way in Postgres as + // its lower-case text does here. The ids are the server's, hence the sort on the expectation. + expect((field(anaRow, BY_AUTHOR) as readonly unknown[]).map((row) => field(row, 'id'))).toEqual( + [p1, p2].sort(), + ); + expect((field(benRow, BY_AUTHOR) as readonly unknown[]).map((row) => field(row, 'id'))).toEqual( + [p3], + ); + + const byReviewer = await db() + .members.where({ orgId: org }) + .orderBy('email') + .preload(BY_REVIEWER) + .all(); + const anaReviewed = byReviewer.find((row) => field(row, 'id') === ana); + const benReviewed = byReviewer.find((row) => field(row, 'id') === ben); + // Nobody named ana as reviewer on this seed — an empty array, not an absent one. + expect(field(anaReviewed, BY_REVIEWER)).toEqual([]); + expect( + (field(benReviewed, BY_REVIEWER) as readonly unknown[]).map((row) => field(row, 'id')), + ).toEqual([p2]); + }); + + test('exactly one extra statement per relation, over the distinct keys the page carried', async () => { + const { org } = await seed('one-statement'); + + const [rows, sent] = await counting('from "pg_preload_live_members"', () => + db().posts.where({ orgId: org }).orderBy('title').preload('author').preload('reviewer').all(), + ); + + expect(rows).toHaveLength(3); + // Two relations named, two statements sent — never one merged, never one per row. + expect(sent).toHaveLength(2); + // Two authors are distinct across three posts: two binds, not three. One reviewer named across + // three posts: one bind. Counted rather than indexed — the two relations resolve concurrently, + // so arrival order is not the order they were named in and pinning it would flake. + expect(sent.filter((text) => text.includes('in ($1, $2)'))).toHaveLength(1); + expect(sent.filter((text) => text.includes('in ($1)'))).toHaveLength(1); + }); + + test('the page tenant scope carries onto the preload statement, and a cross-tenant target excludes', async () => { + const ours = await seed('scope-ours'); + const theirs = await seed('scope-theirs'); + // A post in our tenant pointing at a member row that lives in the other tenant — the FK is + // satisfiable (both rows exist), but the tenant boundary is not. + const crossed = ( + await db().posts.insert({ orgId: ours.org, authorId: theirs.ana, title: 'Crossed' }) + ).id; + + const [rows, sent] = await counting('from "pg_preload_live_members"', () => + db().posts.where({ orgId: ours.org, id: crossed }).preload('author').all(), + ); + + expect(field(rows[0], 'author')).toBeNull(); + // The tenant predicate really is inside the SQL Postgres ran, not only in this process's memory. + expect(sent).toHaveLength(1); + expect(sent[0]).toContain('"org_id" ='); + }); + + test('a soft-deleted target is as invisible to a preload as it is to any other read', async () => { + const { org, ana } = await seed('soft-delete'); + await db().members.delete(ana, { orgId: org }); + const authored = (await db().posts.insert({ orgId: org, authorId: ana, title: 'Orphaned' })).id; + + const [rows, sent] = await counting('from "pg_preload_live_members"', () => + db().posts.where({ orgId: org, id: authored }).preload('author').all(), + ); + + expect(field(rows[0], 'author')).toBeNull(); + // The clause any other read of `members` carries is inside the preload statement too. + expect(sent).toHaveLength(1); + expect(sent[0]).toContain('"deleted_at" is null'); + }); + + test('count() reads no row, so it sends no extra statement', async () => { + const { org } = await seed('count-no-extra'); + + const [total, sent] = await counting('from "pg_preload_live_members"', () => + db().posts.where({ orgId: org }).preload('author').count(), + ); + + expect(total).toBe(3); + expect(sent).toHaveLength(0); + }); +}); diff --git a/packages/entity/src/pg-driver.ts b/packages/entity/src/pg-driver.ts index 2ea0f866b..48eebfec5 100644 --- a/packages/entity/src/pg-driver.ts +++ b/packages/entity/src/pg-driver.ts @@ -14,8 +14,16 @@ import { type TransactionOptions, withTransaction, } from '@ultimat3/db'; +import { + conflictKeys, + insertChunks, + insertColumns, + namedProperties, + upsertPlan, +} from './bulk-write'; import { coalesceFindById } from './coalesce'; import { snake } from './column'; +import { countsFrom, groupColumnOf, groupValue, MAX_GROUPS } from './count-by'; import { cursorFor, seekFrom, valueAt } from './cursor'; import type { Driver } from './database'; import { type EntityCore, SOFT_DELETE_COLUMN } from './entity'; @@ -23,15 +31,18 @@ import { notFound } from './errors'; import { forgetPreloaded, tagSiblings } from './jit-preload'; import { bindValues, decodeRow, type PhysicalRow } from './pg-row'; import { + type ConflictTarget, + countByStatement, countStatement, deleteStatement, + type GroupRow, insertStatement, type ReadShape, selectStatement, updateStatement, } from './pg-sql'; import { deletePlan, idPlan, readPlan, updatePlan } from './plan'; -import type { FindManyArgs, Repo, Transactor } from './repo'; +import type { FindManyArgs, Repo, Transactor, UpsertArgs } from './repo'; import type { QueryPlan } from './tenancy'; export interface PostgresDriverOptions { @@ -95,6 +106,35 @@ export const postgresRepo = ( return send(); }; + /** + * The one insert path, for one row or ten thousand: the batch's own column list, split into as + * many statements as Postgres's bind count allows, and the rows the server stored. `insert(row)` + * comes through here too, so there is no second builder for the two to drift apart in — and a + * batch wide enough to split is several statements, which is why an all-or-nothing caller wraps + * the call in `withTransaction` rather than trusting one statement's atomicity. + */ + const writeRows = async ( + batch: readonly Row[], + conflict: ConflictTarget | undefined, + ): Promise => { + if (batch.length === 0) return []; + for (const row of batch) entity.$assert(row); + const columns = insertColumns(entity, namedProperties(entity, batch)); + const bound = batch.map((row) => bindValues(entity, row)); + const shape = { columns, ...(conflict === undefined ? {} : { conflict }) }; + const written: Row[] = []; + await writing(async () => { + for (const chunk of insertChunks(bound, columns.length)) { + for (const row of await client().query( + insertStatement(entity, chunk, shape), + )) { + written.push(decodeRow(entity, row)); + } + } + }); + return written; + }; + return { async findById(id, options) { const plan = idPlan(entity, id, options, 'findById'); @@ -129,12 +169,24 @@ export const postgresRepo = ( }, async insert(values) { - entity.$assert(values); - const written = await writing(() => - client().one(insertStatement(entity, bindValues(entity, values))), - ); + const [written] = await writeRows([values], undefined); // `returning *` is the row Postgres actually stored, defaults included. - return written === null ? values : decodeRow(entity, written); + return written ?? values; + }, + + async insertAll(batch) { + return writeRows(batch, undefined); + }, + + async upsertAll(batch, args: UpsertArgs) { + const plan = upsertPlan(entity, batch, args.onConflict, args.onMatch ?? 'update'); + // Refused here, not by the server: a batch that repeats a conflict target is `21000` in + // Postgres and a silent overwrite in memory, and the two drivers have to mean one thing. + conflictKeys(entity, plan, batch); + return writeRows(batch, { + columns: insertColumns(entity, plan.on), + set: insertColumns(entity, plan.set), + }); }, async update(id, patch, options) { @@ -166,6 +218,12 @@ export const postgresRepo = ( // No `X_NOT_FOUND` here: a filter that matches nothing is a fact the caller asked for, not a // failed address. The count is the answer, which is why it is a `number` and not `void`. + // + // `deleteWhere`/`updateWhere` are the bulk forms of `delete(id)`/`update(id, patch)` — same + // role `insertAll`/`upsertAll` play for a per-row insert loop, one statement instead of one + // per row. They are also the *only* filtered writes a composite-key entity has: `likes`, + // `blocks`, `participants`, any join table has no single-column id for `delete`/`update` to + // address, so without this pair such an entity would be create-only. async deleteWhere(filter, options) { // The plan is built before the write is announced: a refused filter never happened. const plan = deletePlan(entity, filter, options, 'deleteWhere'); @@ -193,6 +251,23 @@ export const postgresRepo = ( ); return Number(row?.count ?? 0); }, + + async countBy(column, args = {}) { + const grouped = groupColumnOf(entity, column, 'countBy'); + const plan = readPlan(entity, args, 'countBy'); + // One group past the bound, for the same reason a page reads one row past its limit: the + // presence of that group is what says the answer was never going to fit, and `countsFrom` + // refuses it rather than hand back a breakdown missing its tail. + const rows = await client().query( + countByStatement(entity, plan, shapeOf(args), column, MAX_GROUPS + 1), + ); + return countsFrom( + entity, + column, + 'countBy', + rows.map((row) => [groupValue(grouped, row.group_value), Number(row.group_count)] as const), + ); + }, }; }; diff --git a/packages/entity/src/pg-sql.ts b/packages/entity/src/pg-sql.ts index 5b093e68f..2599e64aa 100644 --- a/packages/entity/src/pg-sql.ts +++ b/packages/entity/src/pg-sql.ts @@ -135,14 +135,86 @@ export const countStatement = ( ): SqlFragment => sql`select count(*) as count from ${identifier(entity.$table)} where ${conditions(entity, plan, shape)}`; +/** What a grouped count comes back as. Both names are fixed, so neither can be a column's. */ +export interface GroupRow { + readonly group_value: unknown; + readonly group_count: unknown; +} + +/** + * The grouped count: one row per distinct value of one column, over exactly the rows + * `countStatement` would have counted — the same predicates, the same soft-delete filter, one + * `group by` more. `limit` bounds the groups, not the rows, which is what turns a whole-table + * breakdown into a refusal instead of a result set nobody sized. + * + * Both output names are aliases and fixed, so they cannot collide with each other whatever the + * table declares: an entity is free to have a column called `count`, and the un-aliased form would + * then return two outputs of one name. + */ +export const countByStatement = ( + entity: EntityCore, + plan: QueryPlan, + shape: ReadShape, + column: string, + limit: number, +): SqlFragment => { + const grouped = columnRef(entity, column); + return sql`select ${grouped} as group_value, count(*) as group_count from ${identifier( + entity.$table, + )} where ${conditions(entity, plan, shape)} group by ${grouped} limit ${limit}`; +}; + +/** `on conflict (…) do update set …`, or `do nothing` when there is nothing to overwrite. */ +export interface ConflictTarget { + /** Physical columns of the unique index a collision is judged against. */ + readonly columns: readonly string[]; + /** Physical columns a colliding row takes from the incoming one. Empty is `do nothing`. */ + readonly set: readonly string[]; +} + +export interface InsertShape { + /** Every physical column written — one list, shared by every row of the statement. */ + readonly columns: readonly string[]; + /** How a collision resolves. Absent, it is the caller's error, exactly as it is for one row. */ + readonly conflict?: ConflictTarget | undefined; +} + +/** + * The cell of a row that did not name this column. `default` is the third and last `raw()` in this + * file and, like the other two, a closed set of one word: it is what makes a row inside a many-row + * `insert` mean what the same row means on its own, where an unnamed column is simply left out. + */ +const DEFAULT_CELL = raw('default'); + +const conflictSql = (conflict: ConflictTarget): SqlFragment => { + const target = join(conflict.columns.map(identifier)); + return conflict.set.length === 0 + ? sql` on conflict (${target}) do nothing` + : sql` on conflict (${target}) do update set ${join( + conflict.set.map((column) => sql`${identifier(column)} = excluded.${identifier(column)}`), + )}`; +}; + +/** + * One statement for any number of rows. A single row compiles to exactly the text it always did, + * which is the point: `insertAll([row])` and `insert(row)` are one code path, so there is no + * second insert builder for the two to drift apart in. + */ export const insertStatement = ( entity: EntityCore, - values: ReadonlyMap, + rows: readonly ReadonlyMap[], + shape: InsertShape, ): SqlFragment => { - const entries = [...values]; + const tuples = rows.map( + (row) => + sql`(${join( + shape.columns.map((column) => (row.has(column) ? sql`${row.get(column)}` : DEFAULT_CELL)), + )})`, + ); + const conflict = shape.conflict === undefined ? sql`` : conflictSql(shape.conflict); return sql`insert into ${identifier(entity.$table)} (${join( - entries.map(([column]) => identifier(column)), - )}) values (${join(entries.map(([, value]) => sql`${value}`))}) returning *`; + shape.columns.map(identifier), + )}) values ${join(tuples)}${conflict} returning *`; }; export const updateStatement = ( diff --git a/packages/entity/src/plan.ts b/packages/entity/src/plan.ts index 053f5cf13..112d7b225 100644 --- a/packages/entity/src/plan.ts +++ b/packages/entity/src/plan.ts @@ -6,7 +6,7 @@ import type { EntityCore } from './entity'; import { invariantViolated, patchEmpty, writeUnfiltered } from './errors'; import type { FindManyArgs, RepoOptions } from './repo'; -import type { Predicate, QueryPlan } from './tenancy'; +import type { Predicate, QueryPlan, SortKey } from './tenancy'; import { assertScoped } from './tenancy'; /** A page is bounded by default; an unbounded read is a production incident waiting for traffic. */ @@ -30,23 +30,34 @@ export const singleKeyOf = (entity: EntityCore, operation: string): st return only; }; +/** + * The sort keys a read actually runs with: the caller's, then whatever the primary key still adds. + * The primary key is always the final key — a cursor needs a total order, or two rows with the same + * sort value straddle a page boundary. + * + * Exported because a chain can be judged before it runs: `inBatches()` refuses an ordering that + * cannot carry a cursor, and it has to be looking at the order the driver will send rather than at + * the one the caller typed. + */ +export const totalOrder = ( + entity: EntityCore, + ordered: readonly SortKey[], +): readonly SortKey[] => [ + ...ordered, + ...entity.$primaryKey + .filter((property) => !ordered.some((entry) => entry.column === property)) + .map((property) => ({ column: property, direction: 'asc' as const })), +]; + export const planFor = (entity: EntityCore, args: FindManyArgs): QueryPlan => { const scoped = args.orgId === undefined || entity.$tenantColumn === null ? [] : [{ column: entity.$tenantColumn, op: 'eq', value: args.orgId } satisfies Predicate]; - const ordered = args.orderBy ?? []; return { entity: entity.$name, where: [...(args.where ?? []), ...scoped], - // The primary key is always the final sort key: a cursor needs a total order, or two - // rows with the same sort value straddle a page boundary. - orderBy: [ - ...ordered, - ...entity.$primaryKey - .filter((property) => !ordered.some((entry) => entry.column === property)) - .map((property) => ({ column: property, direction: 'asc' as const })), - ], + orderBy: totalOrder(entity, args.orderBy ?? []), limit: args.limit ?? DEFAULT_PAGE_SIZE, ...(args.cursor === undefined || args.cursor === null ? {} : { cursor: args.cursor }), ...(args.select === undefined ? {} : { select: args.select }), diff --git a/packages/entity/src/preload-parity.test.ts b/packages/entity/src/preload-parity.test.ts new file mode 100644 index 000000000..16860f108 --- /dev/null +++ b/packages/entity/src/preload-parity.test.ts @@ -0,0 +1,201 @@ +// The one claim neither sibling makes: `preload.test.ts` proves what attaches against memory, +// `preload-statements.test.ts` proves the statement shape against a recording client standing in +// for Postgres — but nothing seeds the same logical rows into both and asserts the attached values +// agree. This file does exactly that: same ids, same foreign keys, one chain run against each +// driver, `toEqual` across the pair. + +import { afterAll, beforeEach, describe, expect, test } from 'bun:test'; +import { createRecordingClient, type RecordingClient, setDbClient } from '@ultimat3/db'; +import { text, uuid } from './columns'; +import { database, memoryDriver } from './database'; +import { entity } from './entity'; +import { postgresDriver } from './pg-driver'; +import { clearRegistry } from './registry'; + +const orgs = entity('preload_parity_orgs', { + columns: { id: uuid().primaryKey(), slug: text({ max: 40 }) }, +}); + +const members = entity('preload_parity_members', { + columns: { + id: uuid().primaryKey(), + orgId: uuid() + .references(() => orgs.id) + .tenant(), + email: text({ max: 120 }), + }, +}); + +const posts = entity('preload_parity_posts', { + columns: { + id: uuid().primaryKey(), + orgId: uuid() + .references(() => orgs.id) + .tenant(), + authorId: uuid().references(() => members.id), + /** Nullable, and pointed at an id no member holds on `P3` — the two shapes `belongsTo` has to + * attach `null` for, and both have to read the same across drivers. */ + reviewerId: uuid() + .references(() => members.id) + .nullable(), + title: text({ max: 120 }), + }, +}); + +const entities = { orgs, members, posts }; +type Db = ReturnType>; + +const idAt = (index: number): string => + `00000000-0000-7000-8000-${String(index).padStart(12, '0')}`; + +/** + * A `hasMany` is named for the entity the rows come from, and both inbound keys want that name — + * so every member of the group takes its long form, exactly as `preload.test.ts` spells it. + */ +const BY_AUTHOR = 'preload_parity_postsByAuthor'; +const BY_REVIEWER = 'preload_parity_postsByReviewer'; + +const ORG = idAt(1); +const ANA = idAt(10); +const BEN = idAt(11); +/** Referenced by a foreign key, never inserted anywhere — the "resolved to nothing" case. */ +const GHOST = idAt(99); +const P1 = idAt(20); +const P2 = idAt(21); +const P3 = idAt(22); + +/** What Bun.SQL hands back: snake_case names, nothing decoded — same convention as + * `preload-statements.test.ts`'s `postRow`/`memberRow`. */ +const postRow = (id: string, authorId: string, reviewerId: string | null, title: string) => ({ + id, + org_id: ORG, + author_id: authorId, + reviewer_id: reviewerId, + title, +}); + +const memberRow = (id: string) => ({ id, org_id: ORG, email: `${id}@example.com` }); + +const field = (row: unknown, key: string): unknown => (row as Record)[key]; + +let mem: Db; +let client: RecordingClient; + +/** The identical logical rows on both sides: same ids, same foreign keys, same tenant. */ +beforeEach(async () => { + mem = database(entities, { driver: memoryDriver() }); + await mem.orgs.insert({ id: ORG, slug: 'ours' }); + await mem.members.insert({ id: ANA, orgId: ORG, email: `${ANA}@example.com` }); + await mem.members.insert({ id: BEN, orgId: ORG, email: `${BEN}@example.com` }); + await mem.posts.insert({ id: P1, orgId: ORG, authorId: ANA, title: 'First' }); + await mem.posts.insert({ id: P2, orgId: ORG, authorId: ANA, reviewerId: BEN, title: 'Second' }); + await mem.posts.insert({ id: P3, orgId: ORG, authorId: BEN, reviewerId: GHOST, title: 'Third' }); + + client = createRecordingClient(); + setDbClient(client); + client.on('preload_parity_posts', { + rows: [ + postRow(P1, ANA, null, 'First'), + postRow(P2, ANA, BEN, 'Second'), + postRow(P3, BEN, GHOST, 'Third'), + ], + }); + client.on('preload_parity_members', { rows: [memberRow(ANA), memberRow(BEN)] }); +}); + +afterAll(() => { + setDbClient(undefined); + clearRegistry(); +}); + +const pg = (): Db => database(entities, { driver: postgresDriver() }); + +describe('memory and postgres attach the same values', () => { + test('a belongsTo attaches the identical row, and a resolved-to-nothing key attaches the identical null', async () => { + const memRows = await mem.posts + .where({ orgId: ORG }) + .orderBy('id') + .preload('author') + .preload('reviewer') + .all(); + const pgRows = await pg() + .posts.where({ orgId: ORG }) + .orderBy('id') + .preload('author') + .preload('reviewer') + .all(); + + expect(pgRows).toEqual(memRows); + // Pinned independently of the parity claim, so a bug shared by both drivers cannot hide here. + expect(field(memRows[0], 'author')).toMatchObject({ id: ANA }); + expect(field(memRows[0], 'reviewer')).toBeNull(); + expect(field(memRows[1], 'reviewer')).toMatchObject({ id: BEN }); + // GHOST is a key that resolved to nothing — null, on both drivers, never an absent property. + expect(field(memRows[2], 'reviewer')).toBeNull(); + expect('reviewer' in (memRows[2] as object)).toBe(true); + expect('reviewer' in (pgRows[2] as object)).toBe(true); + }); + + test('a hasMany attaches the identical array, same order, empty the same way', async () => { + const memRows = await mem.members + .where({ orgId: ORG }) + .orderBy('id') + .preload(BY_AUTHOR) + .preload(BY_REVIEWER) + .all(); + const pgRows = await pg() + .members.where({ orgId: ORG }) + .orderBy('id') + .preload(BY_AUTHOR) + .preload(BY_REVIEWER) + .all(); + + expect(pgRows).toEqual(memRows); + // ANA authored P1 and P2, in that order; BEN reviewed neither post ANA authored — [] . + expect( + (field(memRows[0], BY_AUTHOR) as readonly unknown[]).map((row) => field(row, 'id')), + ).toEqual([P1, P2]); + expect(field(memRows[0], BY_REVIEWER)).toEqual([]); + // BEN authored P3 and reviewed P2 — one of each, never merged into one array. + expect( + (field(memRows[1], BY_AUTHOR) as readonly unknown[]).map((row) => field(row, 'id')), + ).toEqual([P3]); + expect( + (field(memRows[1], BY_REVIEWER) as readonly unknown[]).map((row) => field(row, 'id')), + ).toEqual([P2]); + }); + + test('a belongsTo and a hasMany never trade shapes — object or null one side, array the other', async () => { + const [post] = await mem.posts.where({ orgId: ORG, id: P1 }).preload('author').all(); + const [member] = await mem.members.where({ orgId: ORG, id: ANA }).preload(BY_AUTHOR).all(); + + expect(Array.isArray(field(post, 'author'))).toBe(false); + expect(Array.isArray(field(member, BY_AUTHOR))).toBe(true); + + const [pgPost] = await pg().posts.where({ orgId: ORG, id: P1 }).preload('author').all(); + const [pgMember] = await pg().members.where({ orgId: ORG, id: ANA }).preload(BY_AUTHOR).all(); + + expect(Array.isArray(field(pgPost, 'author'))).toBe(false); + expect(Array.isArray(field(pgMember, BY_AUTHOR))).toBe(true); + }); + + test('attachment survives composition with select() — the same narrowed row on both drivers', async () => { + const memRows = await mem.posts + .where({ orgId: ORG }) + .orderBy('id') + .select({ id: true, title: true }) + .preload('author') + .all(); + const pgRows = await pg() + .posts.where({ orgId: ORG }) + .orderBy('id') + .select({ id: true, title: true }) + .preload('author') + .all(); + + expect(pgRows).toEqual(memRows); + // The projection really did narrow the row: only what was selected plus what was preloaded. + expect(Object.keys(memRows[0] as object).sort()).toEqual(['author', 'id', 'title']); + expect(field(memRows[0], 'author')).toMatchObject({ id: ANA }); + }); +}); diff --git a/packages/entity/src/query.ts b/packages/entity/src/query.ts index e1c251f5e..b097fe973 100644 --- a/packages/entity/src/query.ts +++ b/packages/entity/src/query.ts @@ -1,15 +1,18 @@ // The chainable read. Every chain terminates in a cursor page — `page()` returns rows plus the -// cursor for the next one, and `all()`/`one()` are that page's rows. There is no `offset()` and -// there will not be one: under concurrent writes an insert before the offset shifts every later -// page, so a client silently skips and repeats rows. +// cursor for the next one, `all()`/`one()` are that page's rows, and `inBatches()` is that page +// repeated until the cursor runs out. There is no `offset()` and there will not be one: under +// concurrent writes an insert before the offset shifts every later page, so a client silently +// skips and repeats rows. +import type { BatchIterator } from './batch'; +import { assertBatchable, batchIterator } from './batch'; import type { EntityCore } from './entity'; -import { namedColumns } from './plan'; +import { DEFAULT_PAGE_SIZE, namedColumns } from './plan'; import type { RelatedTables } from './preload'; import { preloaded } from './preload'; import type { Relation } from './relations'; import { relationNamed } from './relations'; -import type { Page, Repo, RepoOptions } from './repo'; +import type { Page, Repo, RepoOptions, UpsertArgs } from './repo'; import type { Operator, Predicate, QueryPlan, SortDirection, SortKey } from './tenancy'; import type { ColumnMap, IdOf, Insertable } from './types'; @@ -43,17 +46,64 @@ export interface ReadBuilder { * `select()` narrows the columns, never the relations. */ preload(relation: Name): ReadBuilder>; + /** + * Every row the chain matches, `size` at a time and one statement per batch — the terminal a + * `for await` consumes instead of holding a whole table in memory. A batch is the page `page()` + * would have returned at that position, so filters, tenancy, soft delete, the projection and + * every `preload()` mean here what they mean there, and an empty batch is never yielded. + * + * Keyset, never OFFSET: each batch resumes from the cursor the previous one ended on, so a row + * written mid-iteration cannot make the loop skip or repeat one. `after(cursor)` is where it + * starts and `.cursor` is where it stopped — persist that and a job resumes the iteration. + * + * The loop closes it: `break`, `return` and a throw all stop the next statement, and + * `await using` does the same for a handle kept in a variable. A chain that cannot carry a + * cursor — a nullable sort column — and a chain that also called `limit()` are refused here, + * not one batch later. + */ + inBatches(size: number): BatchIterator; /** The terminal: one bounded page and the cursor that continues it. */ page(): Promise>; all(): Promise; one(): Promise; count(): Promise; + /** + * The grouped count: one statement, one entry per distinct value of `column`, keyed by that + * value — the aggregate a `count()` per row is the N+1 of. `recount every post's likes` is one + * `likes.andWhere('postId', 'in', ids).countBy('postId')`, not one statement per post. + * + * Counts the whole predicate, exactly as `count()` does: the chain's filters, its tenancy and + * its soft-delete visibility, never its page. A value nothing matched is absent rather than `0`, + * which is what `group by` returns and what lets a caller tell "none" from "never asked". + * Entries come back biggest group first, ties by the value, `null` — the group every row without + * one shares — last. + * + * Refused rather than answered: a column whose values a map cannot be keyed by (a timestamp, a + * jsonb, money), and a chain matching more distinct values than one statement should answer with. + */ + countBy(column: K): Promise>; /** The plan this chain describes. Safe to log — `describePlan()` elides values. */ plan(): QueryPlan; } export interface Table extends ReadBuilder { insert(values: Insertable, options?: RepoOptions): Promise; + /** + * Many rows, one statement — the bulk write a per-row `insert` loop is the N+1 of, and the line + * an N+1 warning on a write loop names. Every row is parsed and asserted exactly as `insert` + * parses one, so declared defaults are filled here and not by the caller. Resolves with the rows + * as stored, in order; `insertAll([])` writes nothing. A collision is an error — `upsertAll` is + * the call that tolerates one. + */ + insertAll(rows: readonly Insertable[], options?: RepoOptions): Promise; + /** + * `insertAll` that resolves a collision instead of failing on it: `on conflict (…) do update`, + * or `do nothing` with `onMatch: 'nothing'`. Resolves with the rows this call actually wrote, so + * a row left alone is absent from the result — which is how a caller counts what it inserted. + * The conflict target and the primary key are never overwritten: they are how the stored row was + * found and where it lives. + */ + upsertAll(rows: readonly Insertable[], args: UpsertArgs): Promise; /** `IdOf`: an entity that declared `uuid()` is addressed by a `PostId` only. */ update(id: IdOf, patch: Partial, options?: RepoOptions): Promise; delete(id: IdOf, options?: RepoOptions): Promise; @@ -75,7 +125,12 @@ export interface Table extends ReadBuilder interface State { readonly where: readonly Predicate[]; readonly orderBy: readonly SortKey[]; - readonly limit: number; + /** + * `undefined` until `limit()` is called, and not the default spelled a second time: the driver + * already defaults an unnamed page to `DEFAULT_PAGE_SIZE`, and only "the caller named a page + * size" tells `inBatches()` it was handed one number for two jobs. + */ + readonly limit: number | undefined; readonly cursor: string | null; readonly select: readonly string[] | undefined; /** Resolved when `preload()` was called, so an unknown name fails at the chain and not a page later. */ @@ -85,7 +140,7 @@ interface State { const EMPTY: State = { where: [], orderBy: [], - limit: 50, + limit: undefined, cursor: null, select: undefined, preload: [], @@ -117,7 +172,9 @@ const builder = ( const args = () => ({ where: state.where, orderBy: state.orderBy, - limit: state.limit, + // Sent only when the caller named one: an unnamed page is the driver's default, and passing + // it from here would be the same number written in two files. + ...(state.limit === undefined ? {} : { limit: state.limit }), cursor: state.cursor, ...(selected === undefined ? {} : { select: selected }), }); @@ -183,6 +240,19 @@ const builder = ( ); }, + inBatches(size) { + assertBatchable(entity, size, state); + return batchIterator({ + from: state.cursor, + // The chain's own arguments with the batch as the page size and the iteration's position + // in place of the chain's: one statement per batch, and the same one `page()` sends. + page: async (cursor) => { + const result = await repo.findMany({ ...args(), limit: size, cursor }); + return { rows: await attach(result.rows), nextCursor: result.nextCursor }; + }, + }); + }, + page: async () => { const result = await repo.findMany(args()); return { rows: await attach(result.rows), nextCursor: result.nextCursor }; @@ -198,11 +268,19 @@ const builder = ( count: () => repo.count(args()), + async countBy(column: K) { + // The one cast on this terminal, and it is the same seam `select()` has: the driver contract + // is row-agnostic because a column name is a runtime string there, while the chain knows + // which property it just named and therefore what the map is keyed by. + return (await repo.countBy(column, args())) as ReadonlyMap; + }, + plan: (): QueryPlan => ({ entity: entity.$name, where: state.where, orderBy: state.orderBy, - limit: state.limit, + // The page that will actually run, so an unnamed one still reads as the bound it has. + limit: state.limit ?? DEFAULT_PAGE_SIZE, ...(state.cursor === null ? {} : { cursor: state.cursor }), // The projection actually sent, preload keys included: a plan that is safe to log is only // useful if it is the plan that ran. @@ -221,7 +299,7 @@ const builder = ( * column, and `X_PATCH_EMPTY` downstream would never see it — so whether the refusal fires would * depend on the schema rather than on the call. */ -const touch = (entity: EntityCore, patch: Partial): Partial => { +const touch = (entity: EntityCore, patch: Patch): Patch => { if (namedColumns(patch).length === 0) return patch; const stamped: Record = {}; for (const [property, column] of Object.entries(entity.$columns)) { @@ -241,6 +319,19 @@ export const tableFor = ( ): Table => ({ ...builder(entity, repo, EMPTY, (row) => row, related), insert: async (values, options) => repo.insert(entity.$parse(values), options), + insertAll: async (rows, options) => + repo.insertAll( + rows.map((row) => entity.$parse(row)), + options, + ), + // `touch` and not `$parse` alone: an upsert that lands on a stored row IS an update, so an + // `onUpdateNow()` column has to move exactly as `update(id, patch)` moves it — and stamping it + // in a second place is how one of the two ends up writing a stale `updatedAt`. + upsertAll: async (rows, args) => + repo.upsertAll( + rows.map((row) => touch(entity, entity.$parse(row))), + args, + ), update: async (id, patch, options) => repo.update(id, touch(entity, patch), options), delete: async (id, options) => repo.delete(id, options), deleteWhere: async (filter, options) => repo.deleteWhere(filter, options), diff --git a/packages/entity/src/repo.test.ts b/packages/entity/src/repo.test.ts index 7d0becf48..d43dd7377 100644 --- a/packages/entity/src/repo.test.ts +++ b/packages/entity/src/repo.test.ts @@ -1,8 +1,9 @@ import { afterAll, describe, expect, test } from 'bun:test'; import { decodeCursor } from '@ultimat3/core'; -import { text, timestamp, uuid } from './columns'; +import { integer, text, timestamp, uuid } from './columns'; import { planScope } from './cursor'; import { entity } from './entity'; +import { invariant } from './invariants'; import { planFor } from './plan'; import { clearRegistry } from './registry'; import type { FindManyArgs } from './repo'; @@ -303,3 +304,162 @@ describe('writes', () => { expect((await repo.findById(ids[0] ?? '', { orgId: org(1) }))?.title).toBe('a'); }); }); + +// A batch is one statement in Postgres and one loop here, and the two have to resolve a collision +// the same way — same conflict target, same columns written, same refusal — or a green test here +// is a production bug there. +describe('many-row writes', () => { + const members = entity('repo_test_bulk_members', { + columns: { + id: uuid().primaryKey(), + email: text().unique(), + // Nullable and unique: the one target whose key can be absent, which is a different + // question from "which row does it match". + handle: text().nullable().unique(), + nickname: text(), + visits: integer(), + }, + invariants: (c) => [invariant('nickname_present', c.nickname.minLength(1))], + }); + type Member = typeof members.$row; + + const memberId = (index: number): string => `00000000-0000-7000-8000-00000000020${index}`; + const member = (index: number, over: Partial = {}): Member => ({ + id: memberId(index), + email: `member-${index}@example.test`, + handle: null, + nickname: `m${index}`, + visits: index, + ...over, + }); + + test('a batch writes every row, in the order it was given', async () => { + const repo = memoryRepo(members); + const written = await repo.insertAll([member(3), member(1), member(2)]); + expect(written.map((row) => row.id)).toEqual([memberId(3), memberId(1), memberId(2)]); + expect(await repo.count()).toBe(3); + // An empty batch writes nothing and is not an error — a caller filtering a list to nothing + // should not have to branch around the call. + expect(await repo.insertAll([])).toEqual([]); + expect(await repo.count()).toBe(3); + }); + + test('an invariant judges every row of the batch, not just the first', async () => { + const repo = memoryRepo(members); + await expect( + repo.insertAll([member(1), member(2, { nickname: '' })]), + ).rejects.toBeUltimateError('X_INVARIANT_VIOLATED'); + // Nothing landed, and that is the rule both drivers hold to: `insertAll` asserts every row + // before its first `write`, `writeRows` before its first statement. Pinned here because an + // untested rule is one a refactor deletes — move `$assert` into `write()` and this batch + // half-applies while every other test stays green. + expect(await repo.count()).toBe(0); + }); + + test('an upsert with nothing to collide with is an insert', async () => { + const repo = memoryRepo(members); + const written = await repo.upsertAll([member(1), member(2)], { onConflict: ['email'] }); + expect(written.map((row) => row.id)).toEqual([memberId(1), memberId(2)]); + expect(await repo.count()).toBe(2); + }); + + test('a collision keeps the stored row in place and writes the rest of the columns', async () => { + const repo = memoryRepo(members, [member(1)]); + const [written] = await repo.upsertAll( + [member(9, { email: member(1).email, nickname: 'renamed', visits: 42 })], + { onConflict: ['email'] }, + ); + // The conflict target is not the primary key, so the incoming row carries an id of its own — + // taking it would move the row every foreign key already points at. + expect(written?.id).toBe(memberId(1)); + expect(written?.email).toBe(member(1).email); + expect(written?.nickname).toBe('renamed'); + expect(written?.visits).toBe(42); + expect(await repo.count()).toBe(1); + expect(await repo.findById(memberId(9))).toBeNull(); + }); + + test('onMatch nothing keeps the stored row exactly, and says so by omitting it', async () => { + const repo = memoryRepo(members, [member(1)]); + const written = await repo.upsertAll( + [member(9, { email: member(1).email, nickname: 'ignored', visits: 99 }), member(2)], + { onConflict: ['email'], onMatch: 'nothing' }, + ); + // `returning *` under `do nothing` names no row, so a skipped row is absent from the result + // rather than present and unchanged: the result is always "the rows this call wrote". + expect(written.map((row) => row.id)).toEqual([memberId(2)]); + expect(await repo.findById(memberId(1))).toEqual(member(1)); + }); + + test('two rows of one batch under nothing insert once — the second sees the first', async () => { + const repo = memoryRepo(members); + const written = await repo.upsertAll([member(1), member(2, { email: member(1).email })], { + onConflict: ['email'], + onMatch: 'nothing', + }); + expect(written.map((row) => row.id)).toEqual([memberId(1)]); + expect(await repo.count()).toBe(1); + }); + + test('two rows of one batch under update are refused before the first is written', async () => { + const repo = memoryRepo(members); + // Postgres answers this with 21000 rather than writing one of the two, so nothing may land + // here either — a partial batch is the outcome neither driver is allowed to produce. + await expect( + repo.upsertAll([member(1), member(2, { email: member(1).email })], { onConflict: ['email'] }), + ).rejects.toBeUltimateError('X_INVARIANT_VIOLATED'); + expect(await repo.count()).toBe(0); + }); + + test('a null conflict target matches nothing, stored or in the batch', async () => { + // A Postgres unique index is `NULLS DISTINCT`, so a null handle is not a value two rows can + // share: neither row lands on the stored one, and neither lands on the other. + const repo = memoryRepo(members, [member(1)]); + const written = await repo.upsertAll([member(2), member(3)], { onConflict: ['handle'] }); + expect(written.map((row) => row.id)).toEqual([memberId(2), memberId(3)]); + expect(await repo.count()).toBe(3); + }); + + test('a conflict target is required, and checked on a batch with no rows in it', async () => { + const repo = memoryRepo(members); + await expect(repo.upsertAll([], { onConflict: [] })).rejects.toBeUltimateError( + 'X_INVARIANT_VIOLATED', + ); + expect(await repo.upsertAll([], { onConflict: ['email'] })).toEqual([]); + }); + + test('a failed transaction undoes the whole batch, insert and upsert alike', async () => { + const repo = memoryRepo(members, [member(1)]); + const attempt = memoryTransactor().run(async (tx) => { + await repo.insertAll([member(2), member(3)], { tx }); + // `UpsertArgs extends RepoOptions`, so the args ARE the options — one bag, and the undo is + // registered by the same write every other path here goes through. + await repo.upsertAll([member(9, { email: member(1).email, visits: 99 })], { + tx, + onConflict: ['email'], + }); + expect(await repo.count({ tx })).toBe(3); + throw new Error('boom'); + }); + await expect(attempt).rejects.toThrow('boom'); + expect((await repo.findMany({ limit: 10 })).rows).toEqual([member(1)]); + }); + + test('a soft-deleted row still occupies its conflict key', async () => { + // The unique index it would collide with in Postgres is not partial, so a deleted row is a + // collision here too. `nothing` is what makes the two answers tell each other apart: indexed, + // the incoming row is skipped and the stamp survives; missed, it would be written on top. + const repo = memoryRepo(notes, seed); + const stamped = ids[0] ?? ''; + await repo.delete(stamped, { orgId: org(1) }); + const written = await repo.upsertAll( + [{ id: stamped, orgId: org(1), title: 'usurper', deletedAt: null }], + { onConflict: ['id'], onMatch: 'nothing', orgId: org(1) }, + ); + expect(written).toEqual([]); + expect(await repo.count({ orgId: org(1) })).toBe(3); + const all = await repo.findMany({ orgId: org(1), includeDeleted: true, limit: 10 }); + expect(all.rows).toHaveLength(4); + expect(all.rows.find((row) => row.id === stamped)?.title).toBe('a'); + }); +}); diff --git a/packages/entity/src/repo.ts b/packages/entity/src/repo.ts index 45f60b218..4a1dac08b 100644 --- a/packages/entity/src/repo.ts +++ b/packages/entity/src/repo.ts @@ -7,6 +7,8 @@ // table silently skips and repeats rows. A keyset cursor is stable because it names a // position in the sort order, not a row count. +import { conflictKeyOf, conflictKeys, upsertPlan } from './bulk-write'; +import { countsFrom, groupColumnOf } from './count-by'; import { cursorFor, seekFrom, valueAt } from './cursor'; import { type EntityCore, SOFT_DELETE_COLUMN } from './entity'; import { notFound } from './errors'; @@ -26,6 +28,22 @@ export interface RepoOptions { readonly orgId?: string; } +/** What `upsertAll` does with a row that lands on one already stored. */ +export interface UpsertArgs extends RepoOptions { + /** + * The unique constraint a collision is judged against, named as entity properties — never a + * constraint name, which is a migration artefact this layer cannot resolve. At least one column: + * "any constraint" is not something a caller can reason about. + */ + readonly onConflict: readonly (keyof T & string)[]; + /** + * `'update'` (the default) overwrites the stored row with the incoming values, except the + * conflict target and the primary key. `'nothing'` leaves the stored row exactly as it is and + * omits it from the result, so the result is always "the rows this call wrote". + */ + readonly onMatch?: 'update' | 'nothing'; +} + export interface FindManyArgs extends RepoOptions { readonly where?: readonly Predicate[]; readonly orderBy?: readonly SortKey[]; @@ -53,6 +71,20 @@ export interface Repo { findById(id: IdOf, options?: RepoOptions): Promise; findMany(args?: FindManyArgs): Promise>; insert(values: T, options?: RepoOptions): Promise; + /** + * Many rows, one statement — the bulk form a per-row `insert` loop is the N+1 of. Resolves with + * the rows as stored, defaults included, in the order given; an empty batch writes nothing and + * resolves with `[]`. Nothing here resolves a collision — `upsertAll` is the call that does. + * Past Postgres's bind count the batch becomes several statements, so wrap it in + * `withTransaction` when all-or-nothing matters. + */ + insertAll(rows: readonly T[], options?: RepoOptions): Promise; + /** + * `insertAll` that resolves a collision instead of failing on it. Resolves with the rows this + * call actually wrote — under `onMatch: 'nothing'` a row already stored is skipped and absent, + * which is what `returning *` says on the Postgres side. + */ + upsertAll(rows: readonly T[], args: UpsertArgs): Promise; update(id: IdOf, patch: Partial, options?: RepoOptions): Promise; delete(id: IdOf, options?: RepoOptions): Promise; /** @@ -70,6 +102,16 @@ export interface Repo { */ updateWhere(filter: Partial, patch: Partial, options?: RepoOptions): Promise; count(args?: FindManyArgs): Promise; + /** + * The grouped count: one statement, one entry per distinct value of `column`, over exactly the + * rows `count(args)` counts — the aggregate a `count()` per row is the N+1 of. A value nothing + * matched is absent rather than `0`: the caller knows which keys they asked about, and a map + * that invents them cannot say which ones the table has never seen. + * + * `column` is a property name, spelled as `select` spells one — the typed form is + * `ReadBuilder.countBy`, which knows the row. Ordered by count, biggest group first. + */ + countBy(column: string, args?: FindManyArgs): Promise>; } export interface Transactor { @@ -259,6 +301,52 @@ export const memoryRepo = (entity: EntityCore, seed: readonly Row[] = return write(values, options); }, + async insertAll(batch, options) { + // The whole batch is judged before any of it lands: Postgres refuses the statement as one, + // so a row an invariant rejects must not leave the rows before it stored here either. + for (const row of batch) entity.$assert(row); + return batch.map((row) => write(row, options)); + }, + + async upsertAll(batch, args) { + for (const row of batch) entity.$assert(row); + const plan = upsertPlan(entity, batch, args.onConflict, args.onMatch ?? 'update'); + const keys = conflictKeys(entity, plan, batch); + // The stored rows under the same key, so "does this collide" is the question the unique + // index answers in Postgres and not a scan per row. A soft-deleted row still occupies its + // key here, because the index it would collide with there is not partial either — and a row + // whose target holds a null occupies none, because the index is `NULLS DISTINCT`. + const stored = new Map(); + for (const row of rows.values()) { + const key = conflictKeyOf(entity, plan.on, row); + if (key !== undefined) stored.set(key, row); + } + const written: Row[] = []; + for (const [position, row] of batch.entries()) { + const key = keys[position]; + const existing = key === undefined ? undefined : stored.get(key); + // `do nothing` writes no row, and `returning *` therefore names none: a skipped row is + // absent from the result rather than present and unchanged. + if (existing !== undefined && plan.set.length === 0) continue; + const merged = + existing === undefined + ? row + : Object.assign( + {}, + existing, + Object.fromEntries(plan.set.map((property) => [property, field(row, property)])), + ); + // `UpsertArgs extends RepoOptions`, so the args ARE the options — one bag, and a `tx` + // passed to an upsert registers its undo exactly as it does for every other write here. + const result = write(merged, args); + // Filed as it lands, so a later row of the same batch collides with an earlier one exactly + // as it would with a row the request stored a moment before it. + if (key !== undefined) stored.set(key, result); + written.push(result); + } + return written; + }, + async update(id, patch, options) { return write(Object.assign({}, addressed(id, options, 'update'), patch), options); }, @@ -305,6 +393,21 @@ export const memoryRepo = (entity: EntityCore, seed: readonly Row[] = async count(args = {}) { return select(args, 'count').found.length; }, + + async countBy(column, args = {}) { + // Refused before a row is read, and by the same function the Postgres driver calls: a column + // a map cannot be keyed by is that mistake in both drivers or in neither. + groupColumnOf(entity, column, 'countBy'); + const { found } = select(args, 'countBy'); + const groups = new Map(); + for (const row of found) { + // `?? null`, so a property this row never carried lands in the same group Postgres puts a + // NULL row in — and `0`, `''` and `false` stay the values they are. + const value = field(row, column) ?? null; + groups.set(value, (groups.get(value) ?? 0) + 1); + } + return countsFrom(entity, column, 'countBy', [...groups]); + }, }; }; diff --git a/packages/entity/src/type-pins.ts b/packages/entity/src/type-pins.ts index cf960be2d..d0e34a503 100644 --- a/packages/entity/src/type-pins.ts +++ b/packages/entity/src/type-pins.ts @@ -112,6 +112,36 @@ type _NullableStillAccepted = Assert< /** A non-nullable column with no default is still required — the pin must not over-relax. */ type _RequiredStaysRequired = Assert<{ optionalByNull: null } extends InsertPin ? false : true>; +// --- The bulk write path keeps every narrowing the single-row one has --------- +// `insertAll`/`upsertAll` are `insert` in bulk, and a bulk signature is exactly where one quietly +// widens: `readonly Row[]` instead of `readonly Insertable[]` would make every batch spell out +// the defaults `insert` fills for one row, and a `readonly string[]` conflict target would push a +// typo the single-row path never had to the runtime guard. + +type InsertPinTable = Table, InsertPinColumns>; +type ConflictTarget = Parameters[1]['onConflict']; + +/** A batch is `Insertable`, so a nullable column stays omissible a hundred rows at a time. */ +type _InsertAllTakesInsertables = Assert< + readonly { required: 'x' }[] extends Parameters[0] ? true : false +>; + +/** What comes back is the stored row, never the insertable the caller handed in. */ +type _InsertAllResolvesWithRows = Assert< + [Awaited>] extends [readonly RowOf[]] + ? true + : false +>; + +type _ConflictTargetTakesADeclaredColumn = Assert< + readonly 'required'[] extends ConflictTarget ? true : false +>; + +/** The narrowing that matters: a misspelled conflict target is a compile error, not a rejection. */ +type _ConflictTargetRejectsATypo = Assert< + readonly 'requiredd'[] extends ConflictTarget ? false : true +>; + // --- A branded id survives the whole type chain ------------------------------ // `uuid()` is where the brand is declared, and every hop after it has to carry it. The // derivation (`TypeOf`, `RowOf`, `Insertable`) always did; the BUILDER hard-coded `Column` @@ -159,6 +189,13 @@ type _FindByIdRejectsAnotherBrand = Assert< [UserId] extends [Parameters['findById']>[0]] ? false : true >; +/** + * `UpsertArgs.onConflict` is `readonly (keyof T & string)[]`, which is `readonly never[]` at + * `T = unknown` — so a typed repository must still satisfy the row-agnostic one `RelatedTable.repo` + * and the generated admin are written against, or narrowing the target breaks the preload seam. + */ +type _TypedRepoIsARowAgnosticRepo = Assert<[Repo] extends [Repo] ? true : false>; + type _TableUpdateTakesTheBrand = Assert< [Parameters['update']>[0]] extends [PostId] ? true : false >; @@ -175,4 +212,22 @@ type _UnbrandedIdStaysAString = Assert< [string] extends [IdOf<{ readonly id: string }>] ? true : false >; +// --- The batch iteration stays consumable both ways -------------------------- +// `inBatches()` is the one read that hands back a resource instead of a value, and both ways of +// consuming it are language features rather than methods a call site would obviously miss: +// `for await` needs `[Symbol.asyncIterator]`, `await using` needs `[Symbol.asyncDispose]`. Losing +// either is a silent regression — every existing call keeps compiling, and only the loop that was +// supposed to stop reading stops stopping. + +type BatchPin = ReturnType['inBatches']>; + +type _BatchIterates = Assert< + [BatchPin] extends [AsyncIterable] ? true : false +>; + +type _BatchDisposes = Assert<[BatchPin] extends [AsyncDisposable] ? true : false>; + +/** Batches, never rows: yielding one row at a time is the loop this call exists to replace. */ +type _BatchYieldsBatches = Assert<[BatchPin] extends [AsyncIterable] ? false : true>; + type _RowAgnosticIdStaysAString = Assert<[string] extends [IdOf] ? true : false>; diff --git a/packages/entity/src/upsert-plan.test.ts b/packages/entity/src/upsert-plan.test.ts new file mode 100644 index 000000000..515afc54e --- /dev/null +++ b/packages/entity/src/upsert-plan.test.ts @@ -0,0 +1,407 @@ +// Single responsibility: which batches an `upsertAll` refuses before any SQL exists, what a +// collision overwrites when it does not, and the key that collision is judged on. The sibling +// `bulk-write.test.ts` covers what one statement carries — the column list and the bind-count +// chunking — and both read the same `bulk-write.ts`, so a rule missing from either is a rule the +// two drivers are free to disagree about. + +import { afterAll, describe, expect, test } from 'bun:test'; +import { keyOf } from './batch-read'; +import { conflictKeyOf, conflictKeys, upsertPlan } from './bulk-write'; +import { integer, money, text, timestamp, uuid } from './columns'; +import { entity } from './entity'; +import { EntityError } from './errors'; +import { invariant } from './invariants'; +import { clearRegistry } from './registry'; + +const items = entity('upsert_items', { + columns: { + id: uuid().primaryKey(), + sku: text().unique(), + serial: text().nullable().unique(), + label: text(), + unitPrice: money(), + quantity: integer(), + }, +}); + +/** A composite key: "the plan drops every primary key property" has to mean more than one. */ +const marks = entity('upsert_marks', { + columns: { postId: uuid(), memberId: uuid(), rank: integer(), note: text().unique() }, + primaryKey: ['postId', 'memberId'], +}); + +/** A tenant column and a partial unique index — the two refusals that need a schema to exist. */ +const drafts = entity('upsert_drafts', { + columns: { + id: uuid().primaryKey(), + orgId: uuid().tenant(), + slug: text(), + title: text(), + deletedAt: timestamp().nullable(), + }, + indexes: [ + { on: ['orgId', 'slug'], unique: true }, + // A partial unique index has to have its predicate repeated in the `on conflict` clause, and + // a bare column list cannot say it — so this is a constraint no conflict target can name. + { on: ['title'], unique: true, where: (c) => c.title.minLength(1) }, + ], + // `bindInvariant` stamps `deleted_at is null` onto a unique invariant of a soft-deleting entity, + // so this one is partial too — the same exclusion, reached down the other declaration path. + invariants: (c) => [invariant('upsert_draft_slug', c.unique(['slug']))], +}); + +/** + * The third spelling of one unique index: `invariant(name, c.unique([…]))` emits its + * `create unique index` out of `$invariants` and never touches `$indexes`. Refusing it would tell + * the author to declare a constraint they already declared — and then ship two of them. + */ +const tickets = entity('upsert_tickets', { + columns: { id: uuid().primaryKey(), queue: text(), slot: integer(), note: text() }, + invariants: (c) => [invariant('upsert_ticket_slot', c.unique(['queue', 'slot']))], +}); + +const ID = '0192f5a0-0000-7000-8000-00000000000a'; +const ORG = '0192f5a0-0000-7000-8000-0000000000b0'; + +const item = { id: ID, sku: 'a', label: 'l', quantity: 1 }; +const mark = { postId: ID, memberId: ORG, rank: 1, note: 'n' }; +const draft = { id: ID, orgId: ORG, slug: 'first', title: 'First', deletedAt: null }; +/** The same row with no stamp named at all — what a batch assembled by hand carries. */ +const live = { id: ID, orgId: ORG, slug: 'first', title: 'First' }; + +/** `toBeUltimateError` reads a value, and every refusal here throws synchronously. */ +const caught = (run: () => unknown): EntityError | undefined => { + try { + run(); + } catch (error) { + return error instanceof EntityError ? error : undefined; + } + return undefined; +}; + +afterAll(() => { + clearRegistry(); +}); + +describe('the constraint a collision is judged against', () => { + test('the primary key and a unique() column are both targets a duplicate can land on', () => { + expect(upsertPlan(items, [item], ['id'], 'nothing')).toEqual({ on: ['id'], set: [] }); + expect(upsertPlan(items, [item], ['sku'], 'nothing')).toEqual({ on: ['sku'], set: [] }); + // Compared as a SET of physical columns, so a composite key named in either order is the + // same constraint — an `on conflict` clause is not ordered either. + expect(upsertPlan(marks, [mark], ['memberId', 'postId'], 'nothing')).toEqual({ + on: ['memberId', 'postId'], + set: [], + }); + }); + + test('a target no unique constraint matches is refused before a statement exists', () => { + // Postgres answers `on conflict (label)` with 42P10 — "there is no unique or exclusion + // constraint matching the ON CONFLICT specification" — whatever the rest of the batch says. + const error = caught(() => upsertPlan(items, [item], ['label'], 'update')); + expect(error).toBeUltimateError('X_INVARIANT_VIOLATED'); + expect(error?.cause).toContain('declares no unique constraint on (label)'); + expect(error?.cause).toContain('42P10'); + expect(error?.fix).toContain("indexes: [{ on: ['label'], unique: true }]"); + }); + + test('two columns are not a constraint just because each of them is a column', () => { + // `(sku)` is unique and `(id)` is the key, and neither makes `(sku, label)` an index. + expect(caught(() => upsertPlan(items, [item], ['sku', 'label'], 'update'))).toBeUltimateError( + 'X_INVARIANT_VIOLATED', + ); + expect(caught(() => upsertPlan(marks, [mark], ['postId'], 'nothing'))).toBeUltimateError( + 'X_INVARIANT_VIOLATED', + ); + // Money is two physical columns and no index covers the pair, so it is refused here rather + // than reaching a driver that would have to guess which half the conflict is on. + expect(caught(() => upsertPlan(items, [item], ['unitPrice'], 'nothing'))).toBeUltimateError( + 'X_INVARIANT_VIOLATED', + ); + }); + + test('a partial unique index is not a target a bare column list can name', () => { + const error = caught(() => upsertPlan(drafts, [draft], ['title'], 'nothing')); + expect(error).toBeUltimateError('X_INVARIANT_VIOLATED'); + expect(error?.cause).toContain('42P10'); + // The list of what it does have is the proof the partial one was left out of it — and `slug` + // is the invariant-declared one, partial for the same reason, so both paths are covered. + expect(error?.cause).toContain('it has (id); (org_id, slug)'); + expect(caught(() => upsertPlan(drafts, [draft], ['slug'], 'nothing'))).toBeUltimateError( + 'X_INVARIANT_VIOLATED', + ); + expect(upsertPlan(drafts, [draft], ['orgId', 'slug'], 'nothing')).toEqual({ + on: ['orgId', 'slug'], + set: [], + }); + }); + + test('a unique() invariant is a constraint too — this framework declares one three ways', () => { + const ticket = { id: ID, queue: 'inbound', slot: 3, note: 'first' }; + // Refusing it would send the author to add `indexes: [{ on: ['queue','slot'], unique: true }]` + // beside the invariant that already emits that exact index — two declarations, two indexes. + expect(upsertPlan(tickets, [ticket], ['queue', 'slot'], 'update')).toEqual({ + on: ['queue', 'slot'], + set: ['note'], + }); + expect( + caught(() => upsertPlan(tickets, [ticket], ['queue', 'note'], 'nothing')), + ).toBeUltimateError('X_INVARIANT_VIOLATED'); + }); + + test('a batch with no conflict target is refused — "any constraint" is not a target', () => { + const error = caught(() => upsertPlan(items, [item], [], 'update')); + expect(error).toBeUltimateError('X_INVARIANT_VIOLATED'); + expect(error?.cause).toContain('named no onConflict columns'); + expect(error?.fix).toContain("onConflict: ['']"); + }); + + test('a conflict target the entity never declared is refused, and the columns are listed', () => { + for (const onMatch of ['update', 'nothing'] as const) { + const error = caught(() => upsertPlan(items, [item], ['nope'], onMatch)); + expect(error).toBeUltimateError('X_INVARIANT_VIOLATED'); + expect(error?.cause).toContain('no column "nope"'); + expect(error?.cause).toContain('sku'); + expect(error?.cause).toContain('unitPrice'); + } + }); +}); + +describe('what a collision overwrites', () => { + test('update takes the batch minus the conflict target and minus the primary key', () => { + expect(upsertPlan(items, [item], ['sku'], 'update')).toEqual({ + on: ['sku'], + set: ['label', 'quantity'], + }); + }); + + test('every primary key property goes, even when none of them is the conflict target', () => { + // The address is the closed set the upsert must not move: a composite key is two properties, + // and dropping only the one named in `onConflict` would leave the other writable. + expect(upsertPlan(marks, [mark], ['note'], 'update')).toEqual({ on: ['note'], set: ['rank'] }); + }); + + test('nothing overwrites nothing, however much the batch names', () => { + expect(upsertPlan(items, [item], ['sku'], 'nothing')).toEqual({ on: ['sku'], set: [] }); + }); + + test('an update that would write nothing is refused, and the fix names the other mode', () => { + // Every column in the batch is the conflict target or the key, so `do update` would set + // nothing — Postgres accepts that statement and the caller learns nothing changed from a row + // count they never see. + const error = caught(() => upsertPlan(items, [{ id: ID, sku: 'a' }], ['sku'], 'update')); + expect(error).toBeUltimateError('X_INVARIANT_VIOLATED'); + expect(error?.cause).toContain('would write nothing on a collision'); + expect(error?.fix).toContain("onMatch: 'nothing'"); + }); + + test('an empty batch has nothing to refuse, but its conflict target is still checked', () => { + expect(upsertPlan(items, [], ['sku'], 'update')).toEqual({ on: ['sku'], set: [] }); + // A typo in `onConflict`, or a target no index matches, fails on no rows exactly as it does + // on a page — the target is a fact about the schema, not about the batch. + for (const target of [[], ['nope'], ['label']]) { + expect(caught(() => upsertPlan(items, [], target, 'update'))).toBeUltimateError( + 'X_INVARIANT_VIOLATED', + ); + } + }); +}); + +describe('the tenant column an updating upsert has to carry', () => { + test('a target that skips it is refused: the collision would land on another tenant', () => { + const error = caught(() => upsertPlan(drafts, [draft], ['id'], 'update')); + expect(error).toBeUltimateError('X_TENANCY_UNSCOPED'); + expect(error?.cause).toContain('"orgId"'); + expect(error?.fix).toContain("onConflict: ['orgId', 'id']"); + }); + + test('nothing writes nothing to a row it does not own, so the same target is allowed', () => { + expect(upsertPlan(drafts, [draft], ['id'], 'nothing')).toEqual({ on: ['id'], set: [] }); + }); + + test('a target carrying the tenant column is the one that resolves a collision', () => { + expect(upsertPlan(drafts, [live], ['orgId', 'slug'], 'update')).toEqual({ + on: ['orgId', 'slug'], + set: ['title'], + }); + }); +}); + +describe('the soft-delete stamp an updating upsert never writes', () => { + test('a batch naming it still does not set it: the collision would bring a deleted row back', () => { + // A stamped row still occupies its conflict target — the index it collides with is not partial + // — so `excluded."deleted_at"` would clear a stamp the app wrote. `update(id, patch)` and + // `updateWhere` refuse the same resurrection by carrying `deleted_at is null`, which an + // `on conflict` clause cannot carry. + expect(upsertPlan(drafts, [draft], ['orgId', 'slug'], 'update')).toEqual({ + on: ['orgId', 'slug'], + set: ['title'], + }); + }); + + test('excluded, not refused: `$parse` writes that null, so refusing it would refuse every row', () => { + // Every declared column is present by the time a row reaches the plan, stamp included — the + // `deletedAt: null` is the framework's, not the caller's. + expect( + upsertPlan(drafts, [{ ...live, deletedAt: undefined }], ['orgId', 'slug'], 'update').set, + ).toEqual(['title']); + }); + + test('a batch of nothing but the stamp writes nothing, and says which columns are spared', () => { + const error = caught(() => + upsertPlan( + drafts, + [{ orgId: ORG, slug: 'first', deletedAt: null }], + ['orgId', 'slug'], + 'update', + ), + ); + expect(error).toBeUltimateError('X_INVARIANT_VIOLATED'); + expect(error?.cause).toContain('deletedAt'); + }); + + test('an entity with no soft-delete column has no such stamp to spare', () => { + // `deletedAt` is only a stamp where the entity declares one: elsewhere it is an ordinary + // column, and sparing it would be a rule read off a property name. + expect(upsertPlan(items, [{ ...item, deletedAt: null }], ['sku'], 'update').set).toEqual([ + 'label', + 'quantity', + ]); + }); +}); + +describe('the rows of an updating batch name the same columns', () => { + const uneven = [{ sku: 'a', label: 'x' }, { sku: 'b' }]; + + test('a row that omits a column the batch writes is refused, naming both', () => { + // `excluded.` for a row that omitted the column is that column's DEFAULT, not "leave the + // stored value alone" — so an uneven batch overwrites with a default nobody asked for. + const error = caught(() => upsertPlan(items, uneven, ['sku'], 'update')); + expect(error).toBeUltimateError('X_INVARIANT_VIOLATED'); + expect(error?.cause).toContain('row 2'); + expect(error?.cause).toContain('"label"'); + expect(error?.fix).toContain('label: '); + }); + + test('naming it and leaving it undefined is naming it — the same Object.hasOwn', () => { + const even = [ + { sku: 'a', label: undefined }, + { sku: 'b', label: 'y' }, + ] as Partial[]; + expect(upsertPlan(items, even, ['sku'], 'update')).toEqual({ on: ['sku'], set: ['label'] }); + }); + + test('an uneven batch under nothing is allowed — there is nothing to overwrite', () => { + expect(upsertPlan(items, uneven, ['sku'], 'nothing')).toEqual({ on: ['sku'], set: [] }); + }); +}); + +describe('the key a collision is judged on', () => { + test('a uuid is spelled the way a batched read spells it — case-insensitively', () => { + // Postgres compares a uuid as a value and prints it lower-cased, so an id handed in upper case + // matches the stored row there and would miss it here. + expect(conflictKeyOf(items, ['id'], { id: ID.toUpperCase() })).toBe( + conflictKeyOf(items, ['id'], { id: ID }), + ); + expect(conflictKeyOf(items, ['id'], { id: ID.toUpperCase() })).toContain(keyOf('uuid', ID)); + }); + + test('every other kind is byte-exact — lower-casing merges rows Postgres separates', () => { + expect(conflictKeyOf(items, ['sku'], { sku: 'Bob' })).not.toBe( + conflictKeyOf(items, ['sku'], { sku: 'bob' }), + ); + expect(conflictKeyOf(items, ['quantity'], { quantity: 1 })).not.toBe( + conflictKeyOf(items, ['quantity'], { quantity: 11 }), + ); + }); + + test('a cell is spelled by its type, never by String()', () => { + // `String(date)` drops the milliseconds and `String(money)` is "[object Object]" — either one + // files two rows Postgres keeps apart under a single key. + const at = (ms: string): Date => new Date(`2026-03-04T05:06:07.${ms}Z`); + expect(conflictKeyOf(drafts, ['deletedAt'], { deletedAt: at('001') })).not.toBe( + conflictKeyOf(drafts, ['deletedAt'], { deletedAt: at('002') }), + ); + const priced = (minor: bigint): Partial => ({ unitPrice: { minor, currency: 'EUR' } }); + expect(conflictKeyOf(items, ['unitPrice'], priced(1n))).not.toBe( + conflictKeyOf(items, ['unitPrice'], priced(2n)), + ); + expect(conflictKeyOf(items, ['unitPrice'], priced(1n))).toBe( + conflictKeyOf(items, ['unitPrice'], priced(1n)), + ); + }); + + test('a null in the target is no key at all — a default unique index is NULLS DISTINCT', () => { + expect(conflictKeyOf(items, ['serial'], { serial: null })).toBeUndefined(); + // Every other cell being present does not save it: the index compares no null to anything. + expect(conflictKeyOf(marks, ['postId', 'memberId'], { postId: ID, memberId: null })).toBe( + undefined, + ); + expect(conflictKeyOf(items, ['serial'], { serial: 'S-1' })).not.toBeUndefined(); + }); + + test('a composite target cannot have its boundary forged by a value carrying one', () => { + // A joined string would spell ('a','b') and ('ab','') identically, and one row would then + // collide with a row Postgres treats as a different one. JSON is why it cannot. + const parts = conflictKeyOf(marks, ['postId', 'memberId'], { postId: 'a', memberId: 'b' }); + expect(parts).not.toBe( + conflictKeyOf(marks, ['postId', 'memberId'], { postId: 'ab', memberId: '' }), + ); + expect(parts).not.toBe( + conflictKeyOf(marks, ['memberId', 'postId'], { postId: 'a', memberId: 'b' }), + ); + }); + + test('one key per row, in the order the batch gave them', () => { + const rows = [ + { sku: 'a', label: 'x' }, + { sku: 'b', label: 'y' }, + { sku: 'c', label: 'z' }, + ]; + const keys = conflictKeys(items, upsertPlan(items, rows, ['sku'], 'update'), rows); + expect(keys).toHaveLength(3); + expect(keys).toEqual(rows.map((row) => conflictKeyOf(items, ['sku'], row))); + }); + + test('a batch that collides with itself under update is refused, naming the row', () => { + // Postgres answers this with 21000, "ON CONFLICT DO UPDATE command cannot affect row a second + // time" — refused in both drivers rather than passing in memory and failing in production. + const rows = [ + { sku: 'a', label: 'x' }, + { sku: 'b', label: 'y' }, + { sku: 'a', label: 'z' }, + ]; + const plan = upsertPlan(items, rows, ['sku'], 'update'); + const error = caught(() => conflictKeys(items, plan, rows)); + expect(error).toBeUltimateError('X_INVARIANT_VIOLATED'); + expect(error?.cause).toContain('row 3'); + expect(error?.cause).toContain('sku'); + expect(error?.fix).toContain("onMatch: 'nothing'"); + }); + + test('the repeat is judged on the key, so two spellings of one uuid are one row', () => { + const rows = [ + { id: ID, label: 'x' }, + { id: ID.toUpperCase(), label: 'y' }, + ]; + const plan = upsertPlan(items, rows, ['id'], 'update'); + expect(caught(() => conflictKeys(items, plan, rows))).toBeUltimateError('X_INVARIANT_VIOLATED'); + }); + + test('two null keys are not a repeat — they are two rows that collide with nothing', () => { + const rows = [ + { serial: null, label: 'x' }, + { serial: null, label: 'y' }, + ]; + const plan = upsertPlan(items, rows, ['serial'], 'update'); + expect(plan.set).toEqual(['label']); + expect(conflictKeys(items, plan, rows)).toEqual([undefined, undefined]); + }); + + test('under nothing the server skips the repeat, so the repeat is kept', () => { + const rows = [{ sku: 'a' }, { sku: 'a' }]; + const keys = conflictKeys(items, upsertPlan(items, rows, ['sku'], 'nothing'), rows); + expect(keys).toHaveLength(2); + expect(keys[0]).toBe(keys[1] ?? ''); + }); +}); diff --git a/wiki/Entities-And-Migrations.md b/wiki/Entities-And-Migrations.md index cac31caa5..a84459908 100644 --- a/wiki/Entities-And-Migrations.md +++ b/wiki/Entities-And-Migrations.md @@ -198,7 +198,7 @@ a page later. |---|---| | Shape | a `belongsTo` attaches the row or `null`; a `hasMany` attaches an array, empty when there are none. Always present, so "no author" and "nobody preloaded the author" cannot read the same | | Statements | one extra statement per named relation, resolved **concurrently** — two `preload()` calls are two statements in flight, never one after the other. Naming one relation twice is one statement | -| Terminals | `page()`, `all()`, `one()` resolve every named relation. `count()` and `plan()` do not — a count reads no rows | +| Terminals | `page()`, `all()`, `one()` resolve every named relation. `count()`, `countBy()` and `plan()` do not — a count reads no row to attach one to | | Projection | attached after `select()`. `select()` is widened internally with the relation's own key, so a projection can drop neither the key the preload reads nor the relation it attaches — `plan().select` reports the widened list, the one that actually ran | | Tenancy | the page's own tenant predicate carries onto the related read only when **both** entities are scoped by a column of that same name — a value that scopes one entity is a guess on another, and a source scoped by `workspaceId` may carry an ordinary `orgId` predicate that is a filter and not its tenancy. Carrying nothing is not silence: the related read builds its own plan, so it is refused as `X_TENANCY_UNSCOPED` rather than answered with a guess | | Soft delete | the related read is `findMany`, so `deleted_at is null` applies exactly as it does to any other read | @@ -211,6 +211,113 @@ stated rather than inferred from a loop. The other two members of the family ask `findById` batches a same-microtask fan-out for itself, and a `for … of` loop over a page is already two statements, above. +## Reading a whole table is a loop of pages + +A page is bounded on purpose, so reading everything is iteration — and the iteration is a terminal +on the same chain, not something written around `page()`, `As of 2026-08`: + +```ts +// One statement per batch, one page of rows in memory at a time. +for await (const batch of db.posts.where({ orgId }).preload('author').inBatches(500)) { + await search.index(batch); +} +``` + +A batch **is** the page `page()` would have returned at that position: same filters, same tenancy, +same soft-delete visibility, same `select()`, same `preload()`. Nothing here is a second read path. + +Stopping early keeps the position, which is what makes a long backfill a job that can run out of +time and pick up where it stopped: + +```ts +await using batches = db.posts.where({ orgId }).after(checkpoint).inBatches(500); +for await (const batch of batches) { + await search.index(batch); + if (ctx.clock.now() > deadline) break; +} +await db.checkpoints.update(id, { cursor: batches.cursor }); // resume with .after(cursor) +``` + +| Property | Behavior | +|---|---| +| Statements | one per batch, each asking for one row past it exactly as a page does. An empty batch is never yielded, so a consumer never checks `batch.length` | +| Position | keyset, never OFFSET — a row written mid-iteration cannot make the loop skip or repeat one. `.after(cursor)` starts it, `.cursor` is where the next batch starts, `null` once exhausted | +| Closing | `break`, `return` and a throw all stop the next statement; `await using` is the same guarantee for a handle held in a variable, and `close()` is idempotent. One handle is one iteration — a second `for await` continues it rather than restarting the table | +| Batch size | a whole number of rows, at least one. A chain that also called `limit()` is `X_INVARIANT_VIOLATED`: one number with two meanings, and neither reading is safe to guess | +| Sort order | an ordering no cursor can carry — a nullable sort column — is refused at `inBatches()`, not one batch later. A result that fits in a single batch mints no cursor, so deferring it would hide the mistake until the table grew | +| Tenancy | the plan's, as everywhere else: an unscoped chain is `X_TENANCY_UNSCOPED` on its first batch | + +## A count per row is one grouped count + +The batching family above collapses a *lookup* repeated per row. A `count()` per row is not a +lookup — each statement asks a different question, so no batch reaches it. One statement asks all of +them, `As of 2026-08`: + +```ts +// One statement for all of `ids`, not one `select count(*)` per post. +const counts = await db.likes.where({ orgId }).andWhere('postId', 'in', ids).countBy('postId'); +for (const id of ids) await db.posts.update(id, { likeCount: counts.get(id) ?? 0 }); +``` + +A `ReadonlyMap` keyed by the column's own values and typed from the row, so `counts.get(postId)` is +a `number | undefined` — and the `undefined` is load-bearing. + +| Property | Behavior | +|---|---| +| Counts | the whole predicate, exactly as `count()` does: the chain's filters, its tenancy and its soft-delete visibility are in the statement. `limit()` and `after()` bound a page, never a count | +| A value nothing matched | absent, never `0`. That is what `group by` returns, and it is what tells "none" apart from "never asked" — the default is the caller's `?? 0` | +| NULL | one group, keyed `null`, in both drivers: a property a row never carried is read as `null`, so it lands where Postgres puts its NULL rows. `0`, `''` and `false` stay the values they are | +| Order | biggest group first, ties by the value — numbers and bigints numerically, everything else by its text — and `null` last. Applied after the rows are in, not in the statement: a hash aggregate returns groups in whatever order it built them, so an `order by` there would let the two drivers disagree about a result they agree on | +| Groupable columns | `uuid`, `text`, `char`, `boolean`, `integer`, `bigint`. A timestamp, a `jsonb` or a `money` column is `X_INVARIANT_VIOLATED`, whose `fix` names a column of that entity that *is* groupable — a `Map` compares a non-primitive key by identity, so such a map could only ever answer `undefined` | +| More than 1000 groups | `X_INVARIANT_VIOLATED`, never a truncated map: the statement asks for one group past the bound, exactly as a page reads one row past its limit, and the `fix` spells the `andWhere('', 'in', )` that bounds it. A map that lost its tail reads like a complete one, and recounting from it writes the wrong number to every row it missed | +| Statement | `select "post_id" as group_value, count(*) as group_count … group by "post_id"` — both names are fixed aliases, so an entity may still declare a column called `count`, and the grouped value is re-parsed by the column that declared it | +| Tenancy | the plan's, as everywhere else: an unscoped chain on a tenant-scoped entity is `X_TENANCY_UNSCOPED`, never a count across tenants | + +There is no `groupBy()` builder and no error code of its own: `countBy` is a terminal on the chain +that already exists, over exactly the rows `count()` counts. + +## Writing many rows is one statement + +The sections above make a read loop stop being N statements. A loop of `insert()` calls is +the same defect on the write side, and `insertAll`/`upsertAll` are its answer, `As of 2026-08`: + +```ts +await db.tags.insertAll(names.map((name) => ({ orgId, name }))); // one statement, n rows + +await db.likes.upsertAll(rows, { // insert, or leave what is there + onConflict: ['orgId', 'postId', 'memberId'], + onMatch: 'nothing', +}); + +await db.counters.upsertAll(rows, { onConflict: ['orgId', 'day'] }); // insert, or overwrite +``` + +Rows are `Insertable` and go through `$parse` exactly as one row does, so declared defaults are +filled here rather than by the caller; `upsertAll` also stamps `onUpdateNow()` columns through the +same `touch()` `update(id, patch)` uses, because an upsert that lands on a stored row *is* an +update. Both resolve with **the rows this call wrote**, in order — under `onMatch: 'nothing'` a row +already stored is skipped and absent, which is how a caller counts what it actually inserted. + +| Property | Behavior | +|---|---| +| One builder | every insert in the framework compiles through the same function, so `insertAll([row])` is the text `insert(row)` always produced. There is no second insert path to drift | +| What a collision writes | every column the batch names, minus the conflict target (how the row was found), minus the primary key (where it lives) and minus the soft-delete stamp (whether the row is there at all). Moving either of the first two moves a row nobody asked to move, and every foreign key pointing at that id misses it | +| A soft-deleted row it lands on | stays deleted, and takes the batch's other columns. A stamped row still occupies its conflict target — that unique index is not partial — so writing `deletedAt` from the incoming row would bring back a row the app deleted, which is the resurrection `update()` and `updateWhere()` refuse by carrying `deleted_at is null`. `insertAll` is unaffected: a row that collides with nothing writes the stamp it carries | +| Conflict target | properties of a **declared** unique constraint — the primary key, a `unique()` column, an `indexes: [{ on, unique: true }]` entry, or an `invariant(name, c.unique([…]))`. Anything else is `X_INVARIANT_VIOLATED` here rather than `42P10` from the server | +| Tenancy | on a tenant-scoped entity, `onMatch: 'update'` requires the tenant column *inside* the conflict target, else `X_TENANCY_UNSCOPED`: an upsert builds no read plan, so a target that omits it matches another tenant's row and rewrites it. `'nothing'` is allowed — it writes nothing to a row it does not own | +| A batch repeating itself | two rows with one conflict target under `'update'` is refused: Postgres answers that statement `ON CONFLICT DO UPDATE command cannot affect row a second time`, so it cannot pass in memory either | +| Uneven batches | under `'update'` every row names the same columns — `excluded.` for a row that omitted one is that column's *default*, not the stored value. `insertAll` and `'nothing'` accept an uneven batch and render `default` in the missing cell | +| Nulls | a null in the conflict target collides with nothing, in both drivers — a Postgres unique index is `NULLS DISTINCT` | +| Size | past 65535 bind parameters the batch is several statements, never one the server refuses. Wrap the call in `withTransaction` when all-or-nothing matters | + +There is no `updateAll`: `updateWhere(filter, patch)` and `deleteWhere(filter)` are already the +bulk forms of `update` and `delete`, and a second spelling of one of them would be a second path. + +One consequence to know before you reach for `onMatch: 'update'` on a tenant-scoped entity: the +composite unique index the tenancy rule requires is one the migration generator still cannot emit — +see the composite-index row in [Known gaps](Known-Gaps). Declare it on the entity as usual and write +that one `create unique index` by hand in the migration until it is fixed. + ## Invariants | Rule | Behavior | diff --git a/wiki/Error-Codes.md b/wiki/Error-Codes.md index 24b21e8c8..48e53b018 100644 --- a/wiki/Error-Codes.md +++ b/wiki/Error-Codes.md @@ -132,7 +132,7 @@ A denial is `X_FORBIDDEN`, above — `@ultimat3/policy` owns it and every surfac |---|---|---|---| | `X_POLICY_MISSING` | an action was declared without a policy | a new action shipped with no `policy:` | add `policy: can(':')`, or `allow('public')` to say so explicitly | | `X_PERMISSION_UNKNOWN` | permission string is not in the permission set | typo, or a permission never declared | add it to `definePermissions([…])` | -| `X_TENANCY_UNSCOPED` | a tenant-scoped query has no org predicate | a repo call that forgot the tenant | pass `{ orgId }`, or wrap the plan with `orgScoped(entity, orgId, plan)` | +| `X_TENANCY_UNSCOPED` | a tenant-scoped query has no org predicate | a repo call that forgot the tenant, or an `upsertAll(rows, { onMatch: 'update' })` whose `onConflict` omits the tenant column — a collision there lands on another tenant's row | pass `{ orgId }`, wrap the plan with `orgScoped(entity, orgId, plan)`, or put the tenant column in `onConflict` | | `X_ACTION_POLICY_MISSING` | an action was registered without a policy | build-time check on the action registry | add `policy: can('')` to the declaration | | `X_QUERY_POLICY_MISSING` | a query was registered without a policy | same, for reads | add `policy: can('')` to the query | | `X_QUERY_NOT_PAGEABLE` | a read returned rows with no id, so a cursor cannot name a position | a projection or aggregate that drops the primary key — the id is the tiebreak that makes the sort order total | return the key from the query's `sql:`, e.g. `db.posts.select({ id: true, … })` | diff --git a/wiki/Queries-And-Live-Queries.md b/wiki/Queries-And-Live-Queries.md index 4a484d26d..fe93f35c6 100644 --- a/wiki/Queries-And-Live-Queries.md +++ b/wiki/Queries-And-Live-Queries.md @@ -144,6 +144,8 @@ Streamed `` holes ([Routes and render modes](Routes-And-Render-Modes)) The memo collapses the *same* read asked twice. Fifty *different* row lookups collapse one layer down, in the repo: `findById` issued across one microtask of a request is one `where "id" in (…)` ([Entities and migrations → Point lookups batch themselves](Entities-And-Migrations#point-lookups-batch-themselves)) — or, named on the chain instead of inferred from a loop, one `preload()` per relation ([Entities and migrations → Preload states a relation the loop would infer](Entities-And-Migrations#preload-states-a-relation-the-loop-would-infer)). +Fifty *counts* collapse neither way — one `count()` per row is fifty different questions, so nothing above the repo can batch them. The repo answers them in one statement instead: `db.likes.where({ orgId }).andWhere('postId', 'in', ids).countBy('postId')` is a map keyed by the column's values, biggest group first, with a value nothing matched absent rather than `0` ([Entities and migrations → A count per row is one grouped count](Entities-And-Migrations#a-count-per-row-is-one-grouped-count)). + ## Every cached query carries a tag The contract. **Not yet a gate** — `As of 2026-08` `X_CACHE_UNTAGGED_QUERY` is reserved: no code path raises it, and `x errors explain X_CACHE_UNTAGGED_QUERY` refuses it ([Error codes → Reserved codes](Error-Codes#reserved-codes)).