diff --git a/docs/CHANGELOG.md b/docs/CHANGELOG.md index a27f3288a..1afe590fc 100644 --- a/docs/CHANGELOG.md +++ b/docs/CHANGELOG.md @@ -18,6 +18,34 @@ entry. See `CONTRIBUTING.md` § Releases & changelog. ## [Unreleased] +### Added — tamper-evident receipt chain: hash chaining + signed checkpoints (#758) + +- **Hash chain (migration `0041`).** Every persisted receipt row now joins a + per-stream chain: `entry_hash = sha256(stream ‖ seq ‖ prev_hash ‖ + canonical(payload))`, appends serialized through a `FOR UPDATE`-locked + stream head so concurrent turns form one linear chain. Editing row *n* + breaks the copy of its hash stored in row *n+1* — visible to every later + entry. Replayed turns roll the whole transaction back (no phantom head + movement). UPDATE on `turn_receipts` is trigger-forbidden (defence in + depth; the chain is the proof); DELETE stays legal for retention, and + deletions show as seq gaps. +- **Ed25519 checkpoints.** On an interval (`AUDIT_CHECKPOINT_INTERVAL_MINUTES`, + default 60) the stream head is signed with a key held ONLY in + env/secret-manager (`AUDIT_SIGNING_KEY` — never in Postgres, or the admin + the chain defends against could re-sign a rewritten chain). Optional + external anchor file (`AUDIT_ANCHOR_PATH`, JSONL) for WORM storage. + Keygen: `node scripts/generate-audit-signing-key.mjs`. Public key + + fingerprint served at `GET /api/v1/operator/provenance/public-key`. +- **Verification foundation** (`verifyChainSegment`) ships with tamper tests + (edit → `hash_mismatch` at the exact seq; delete → `seq_gap`; forged + suffix → `link_mismatch`); the operator-facing verify surface (endpoint, + signed export, offline verifier, UI) is #761 — until it ships, + "cryptographically verifiable" remains a non-claim + (`docs/ai-act-transparency.md`). +- Known limitations, stated: detection not prevention; per-row time is + anchored by checkpoint cadence, not per-row (`created_at` is outside the + hash); pre-chain rows carry NULL chain columns ("pre-chain era"). + ### Added — Privacy Shield: operator deny-lists, miss-report queue, idnum coverage, eval CI gate (#760) - **Operator deny-list.** Two new privacy-plugin setup fields: `custom_terms` diff --git a/docs/ai-act-transparency.md b/docs/ai-act-transparency.md index d9daaa72d..6f91f8f7e 100644 --- a/docs/ai-act-transparency.md +++ b/docs/ai-act-transparency.md @@ -162,10 +162,14 @@ und begründet in #684; jeder Ausfall wird seitdem gezählt und protokolliert. Migration `0039`): der PII-freie Privacy-Receipt jedes abgeschlossenen Turns wird auf dem Postgres-Backend synchron gespeichert — ohne Graph-Sink, ohne User-Cluster-Vorbedingung — und ist unter `/api/v1/operator/receipts` (auth-gated) sowie im Operator-UI abrufbar. -Fehlschläge werden gezählt und protokolliert, nie still verworfen. Der Receipt ist ein -*Record*, aber (noch) nicht manipulationssicher: Hash-Verkettung, Signaturen und -Verifikation sind #758/#761 — bis dahin bleibt „kryptographisch nachweisbar" eine -Nicht-Zusage. +Fehlschläge werden gezählt und protokolliert, nie still verworfen. **Seit #758 ist der +Record hash-verkettet und checkpoint-signiert** (Migration `0041`: `entry_hash` über +`prev_hash` verkettet, Ed25519-Checkpoints mit Schlüssel außerhalb der DB, optionaler +externer Anker) — eine nachträgliche Änderung bricht die Kette sichtbar. Was noch fehlt, +ist die **Verifikations-Fläche** (#761: Verify-Endpoint, signierter Export, +Offline-Verifier, UI). Bis #761 shipped, bleibt „kryptographisch nachweisbar" öffentlich +eine Nicht-Zusage — intern ist der Mechanismus da, aber ein Nachweis, den nur wir führen +können, ist noch kein Nachweis. **C2PA ist offen.** Im Code existiert keine C2PA-Implementierung. Für Bilder wäre das der naheliegende nächste Schritt; heute ist es keine Zusage, sondern ein offener Punkt. diff --git a/docs/middleware-agent-handoff.md b/docs/middleware-agent-handoff.md index 235b3b715..74dbb106e 100644 --- a/docs/middleware-agent-handoff.md +++ b/docs/middleware-agent-handoff.md @@ -1155,6 +1155,25 @@ auth-gated **`GET /api/v1/operator/receipts`** (Liste, Composite-Keyset-Cursor Reaper mit Eager-Boot-Tick, Cutoff auf der DB-Uhr. Tests: `test/turnReceipts.test.ts`, `test/orchestrator/turnReceiptPersistence.test.ts`. +### Receipt-Hash-Kette + signierte Checkpoints (#758) + +`turn_receipts` ist seit Migration `0041` hash-verkettet: `entry_hash = +sha256(stream ‖ seq ‖ prev_hash ‖ canonical(payload))`, Appends serialisiert +über `audit_stream_heads` (FOR UPDATE — eine lineare Kette, keine Forks); +Replay ⇒ kompletter Rollback. UPDATE per Trigger verboten, DELETE bleibt für +Retention erlaubt (Lücken sind detektierbar). Ed25519-Checkpoints +(`src/receipts/checkpoints.ts`): Key NUR in Env (`AUDIT_SIGNING_KEY`, +Keygen `scripts/generate-audit-signing-key.mjs`), Intervall +`AUDIT_CHECKPOINT_INTERVAL_MINUTES` (60), externer Anker `AUDIT_ANCHOR_PATH` +(JSONL). Public Key: **`GET /api/v1/operator/provenance/public-key`**. +Verify-Grundstein `verifyChainSegment` in `src/receipts/chain.ts` (Tamper- +Tests in `test/receiptHashChain.test.ts`); die Operator-Verify-Fläche ist +#761. Zeitanker: Checkpoint-Kadenz, nicht pro Zeile (`created_at` ist +außerhalb des Hashes — bewusst, begründet in `src/receipts/chain.ts` + +`receiptChainPayload` in `store.ts`). ⚠️ #761-Pflicht: Retention-Lücken +gegen die Checkpoint-Zeitachse prüfen (Backdating-Laundering-Kanal, s. +security-architecture §7b). + ## 4. Migration Managed Agents → Lokal ### Warum migriert diff --git a/docs/security-architecture.md b/docs/security-architecture.md index f3a867b0b..da44ae5b8 100644 --- a/docs/security-architecture.md +++ b/docs/security-architecture.md @@ -197,6 +197,33 @@ the connection. A subscription URL is also checked at creation time (`assertOutboundUrlAllowed`), so an operator gets an immediate 400 rather than only discovering the block on the first delivery attempt. +## 7b. Tamper-evident receipt chain (#758) + +The per-turn receipt record (`turn_receipts`, #757) is hash-chained: each +row's `entry_hash` covers its canonical payload plus the previous row's +hash, appends serialized through a locked stream head. Editing a row breaks +the copy of its hash stored in the next row — the chain visibly breaks for +every later entry. Periodic Ed25519 checkpoints sign the head with a key +held **only** in env/secret-manager (`AUDIT_SIGNING_KEY`) — never in +Postgres, or the DB admin the chain defends against could re-sign a +rewritten chain — optionally anchored to an external append-only file +(`AUDIT_ANCHOR_PATH`) for WORM storage. Threat model: **detection, not +prevention** — wholesale destruction shows as sequence gaps and orphaned +checkpoints; per-row timestamps are anchored by checkpoint cadence, not +per-row. UPDATE on the table is trigger-forbidden as defence in depth; +DELETE stays legal for bounded retention. The operator verify surface +(endpoint, signed export, offline verifier) is #761. + +One consequence to state explicitly: because `created_at` sits outside the +hash and DELETE is legal, an admin who drops the trigger could backdate +`created_at` and let the reaper delete a row early — presenting the gap as +legal retention. The mitigation is the checkpoint timeline: a row with +`seq ≤` a checkpoint's seq provably existed by that checkpoint's signed +time, so **the #761 verifier MUST check every retention gap's age against +the checkpoint timeline** (a gap younger than the retention window measured +in checkpoint time is a finding, not retention). Recorded as a hard +requirement on #761. + ## 7a. Conductor approvals: strict semantics, cancellation, and the baton audit (#759) Three properties of the human-approval gate are security decisions, made diff --git a/middleware/.env.example b/middleware/.env.example index f240b5b2b..8ebd620cd 100644 --- a/middleware/.env.example +++ b/middleware/.env.example @@ -242,6 +242,17 @@ DEV_PLATFORM_SUBSCRIPTION_MODE=false # Postgres backend only). turn_id + scope are personal-data linkage, so rows # are reaped after this many days. RECEIPT_RETENTION_DAYS=90 + +# --- Receipt hash chain checkpoints (#758) ---------------------------------- +# Ed25519 private key (base64 PKCS#8 DER) for signing chain checkpoints — +# generate with `node scripts/generate-audit-signing-key.mjs`. Keep it in a +# secret manager / env, NEVER in the database. Absent = chain builds, no +# signed checkpoints (logged loudly at boot). +# AUDIT_SIGNING_KEY= +AUDIT_CHECKPOINT_INTERVAL_MINUTES=60 +# Optional external anchor file (JSONL, append-only) outside the DB — point +# at storage the DB admin cannot rewrite (WORM / synced bucket). +# AUDIT_ANCHOR_PATH=/data/audit-anchors.jsonl # DEV_PLATFORM_SUBSCRIPTION_ACK= # required acknowledgment string when SUBSCRIPTION_MODE=true # --- Conductor generic webhooks (issue #437) -------------------------------- diff --git a/middleware/migrations/0041_receipt_hash_chain.sql b/middleware/migrations/0041_receipt_hash_chain.sql new file mode 100644 index 000000000..aec3c9fef --- /dev/null +++ b/middleware/migrations/0041_receipt_hash_chain.sql @@ -0,0 +1,82 @@ +-- #758 — tamper-evident receipt chain: hash chaining + signed checkpoints. +-- +-- Builds directly on `turn_receipts` (0039, #757). Every receipt row joins a +-- per-stream hash chain: `entry_hash = sha256(canonical(payload) || prev_hash +-- || seq)`. Editing row n breaks the match stored in row n+1 — the chain +-- visibly breaks for every later entry. Periodic Ed25519 checkpoints sign +-- (stream, seq, head_hash) with a key held OUTSIDE the database, so the +-- whole chain cannot be silently rewritten either. +-- +-- Threat model: DETECTION of after-the-fact modification, not prevention. +-- Wholesale destruction shows as seq gaps + orphaned checkpoints. +-- +-- Retention interplay: the reaper (#757) legitimately DELETEs expired rows, +-- so DELETE stays allowed and deletions are detectable (seq gaps below the +-- oldest surviving row are expected exactly up to the retention horizon). +-- UPDATE is never legitimate on this table — blocked by trigger below +-- (defence in depth: an admin can drop the trigger; the chain is the proof). +-- +-- Rows written before this migration (or while chaining was not yet active) +-- have NULL chain columns — the "pre-chain era", which a verifier reports as +-- unverifiable rather than broken. +-- +-- Numbering: 0039 = #757 (turn_receipts), 0040 = #760 (privacy_miss_reports). + +ALTER TABLE turn_receipts + ADD COLUMN IF NOT EXISTS stream_id TEXT, + ADD COLUMN IF NOT EXISTS seq BIGINT, + ADD COLUMN IF NOT EXISTS prev_hash BYTEA, + ADD COLUMN IF NOT EXISTS entry_hash BYTEA, + ADD COLUMN IF NOT EXISTS hash_version SMALLINT; + +CREATE UNIQUE INDEX IF NOT EXISTS turn_receipts_stream_seq + ON turn_receipts (stream_id, seq); + +-- Serialization point for chain appends: one row per stream, locked +-- FOR UPDATE inside the insert transaction so concurrent appends line up +-- into a single linear chain (no forks). +CREATE TABLE IF NOT EXISTS audit_stream_heads ( + stream_id TEXT PRIMARY KEY, + head_seq BIGINT NOT NULL, + head_hash BYTEA NOT NULL, + updated_at TIMESTAMPTZ NOT NULL DEFAULT NOW() +); + +-- Seed the receipts head at genesis (review H1): `SELECT … FOR UPDATE` on a +-- row that does not exist locks NOTHING, so on a fresh deployment two +-- concurrent FIRST appends would both compute seq=1 and the loser's receipt +-- would be permanently lost on the unique index. With the row pre-seeded the +-- lock always has something to grab. head_seq 0 + the genesis hash keep the +-- store's `seq = head_seq + 1` arithmetic identical. +-- The literal is sha256('genesis:receipts') — reproduce with: +-- node -e "console.log(require('node:crypto').createHash('sha256').update('genesis:receipts','utf-8').digest('hex'))" +-- (hard-coded rather than pgcrypto's digest() so the migration needs no extension). +INSERT INTO audit_stream_heads (stream_id, head_seq, head_hash) +VALUES ('receipts', 0, '\xb69452622fd89eb75373337022abd13f81da4da98bdad81955868609bbe42ac2') +ON CONFLICT (stream_id) DO NOTHING; + +-- Signed checkpoints: Ed25519 over (stream_id, seq, head_hash, signed_at). +-- The private key lives in env/secret manager, NEVER in this database — +-- otherwise the admin we defend against could re-sign a rewritten chain. +CREATE TABLE IF NOT EXISTS audit_checkpoints ( + id UUID PRIMARY KEY DEFAULT gen_random_uuid(), + stream_id TEXT NOT NULL, + seq BIGINT NOT NULL, + head_hash BYTEA NOT NULL, + signed_at TIMESTAMPTZ NOT NULL, + signature BYTEA NOT NULL, + public_key_fingerprint TEXT NOT NULL, + UNIQUE (stream_id, seq) +); + +-- Defence in depth, not the proof: UPDATE is never legitimate on receipts. +CREATE OR REPLACE FUNCTION turn_receipts_forbid_update() RETURNS trigger AS $$ +BEGIN + RAISE EXCEPTION 'turn_receipts is append-only: UPDATE is forbidden (#758)'; +END; +$$ LANGUAGE plpgsql; + +DROP TRIGGER IF EXISTS turn_receipts_no_update ON turn_receipts; +CREATE TRIGGER turn_receipts_no_update + BEFORE UPDATE ON turn_receipts + FOR EACH ROW EXECUTE FUNCTION turn_receipts_forbid_update(); diff --git a/middleware/scripts/generate-audit-signing-key.mjs b/middleware/scripts/generate-audit-signing-key.mjs new file mode 100644 index 000000000..38797929e --- /dev/null +++ b/middleware/scripts/generate-audit-signing-key.mjs @@ -0,0 +1,25 @@ +#!/usr/bin/env node +/** + * #758 — generate the Ed25519 checkpoint-signing keypair. + * + * Prints the PRIVATE key as base64 PKCS#8 DER (the AUDIT_SIGNING_KEY value) + * and the PUBLIC key as PEM + fingerprint (hand the public half to auditors + * out-of-band; pin the fingerprint). Store the private key in your secret + * manager / env — NEVER in the database the chain defends. + */ + +import { createHash, generateKeyPairSync } from 'node:crypto'; + +const { privateKey, publicKey } = generateKeyPairSync('ed25519'); + +const privateDer = privateKey.export({ format: 'der', type: 'pkcs8' }); +const publicPem = publicKey.export({ format: 'pem', type: 'spki' }).toString(); +const fingerprint = createHash('sha256') + .update(publicKey.export({ format: 'der', type: 'spki' })) + .digest('hex'); + +console.log('AUDIT_SIGNING_KEY (private — keep in secret manager / env):\n'); +console.log(privateDer.toString('base64')); +console.log('\nPublic key (share with auditors out-of-band):\n'); +console.log(publicPem); +console.log(`Public key fingerprint (sha256 of SPKI DER):\n\n${fingerprint}`); diff --git a/middleware/src/config.ts b/middleware/src/config.ts index 28fe8b406..2a8f8edd9 100644 --- a/middleware/src/config.ts +++ b/middleware/src/config.ts @@ -288,6 +288,18 @@ const ConfigSchema = z.object({ // scope are personal-data linkage, so rows are reaped after this many days. RECEIPT_RETENTION_DAYS: z.coerce.number().int().positive().default(90), + // #758 — Ed25519 checkpoint signing for the receipt hash chain. The + // private key (base64 PKCS#8 DER; generate with + // scripts/generate-audit-signing-key.mjs) lives HERE — env / secret + // manager — never in Postgres: the admin the chain defends against must + // not be able to re-sign a rewritten chain. Absent ⇒ the chain still + // builds, but no signed checkpoints are produced (logged loudly at boot). + AUDIT_SIGNING_KEY: z.string().optional(), + AUDIT_CHECKPOINT_INTERVAL_MINUTES: z.coerce.number().int().positive().default(60), + // Optional external anchor: checkpoint JSONL appended OUTSIDE the DB — + // point it at storage the DB admin cannot rewrite (WORM/S3 sync). + AUDIT_ANCHOR_PATH: z.string().optional(), + // Epic #470 W4 — default per-job LLM cost budget (USD) applied when neither the // job nor its repo sets one (spec §5). Token budgets have NO default: they are // enforced only when explicitly set on the job or repo. diff --git a/middleware/src/index.ts b/middleware/src/index.ts index 525723d15..68a1ed580 100644 --- a/middleware/src/index.ts +++ b/middleware/src/index.ts @@ -35,6 +35,7 @@ import { createMissReportRoutes } from './privacy/missReportRoutes.js'; import { TURN_RECEIPT_STORE_SERVICE_NAME } from '@omadia/plugin-api'; import { PgTurnReceiptStore, startTurnReceiptReaper } from './receipts/store.js'; import { createReceiptRoutes } from './receipts/routes.js'; +import { loadCheckpointSigner, startCheckpointWorker } from './receipts/checkpoints.js'; import { bindingKeyForTurn } from './conductor/principalId.js'; import { createOperatorChannelsRouter } from './routes/operatorChannels.js'; import { createAgentBuilderRouter } from './routes/agentBuilder.js'; @@ -3529,6 +3530,43 @@ async function main(): Promise { `[middleware] turn receipts wired at /api/v1/operator/receipts (auth-gated, retention ${config.RECEIPT_RETENTION_DAYS}d)`, ); + // #758 — signed checkpoints over the receipt hash chain. The chain + // itself always builds (the store appends chained rows unconditionally); + // signing is the layer that needs the operator-held key. Absent key ⇒ + // loud boot log, not a silent no-op. + const checkpointSigner = config.AUDIT_SIGNING_KEY + ? loadCheckpointSigner(config.AUDIT_SIGNING_KEY) + : undefined; + if (checkpointSigner) { + startCheckpointWorker(graphPool, checkpointSigner, { + intervalMs: config.AUDIT_CHECKPOINT_INTERVAL_MINUTES * 60_000, + ...(config.AUDIT_ANCHOR_PATH ? { anchorPath: config.AUDIT_ANCHOR_PATH } : {}), + }); + console.log( + `[middleware] audit checkpoints wired (every ${config.AUDIT_CHECKPOINT_INTERVAL_MINUTES}min, fingerprint ${checkpointSigner.publicKeyFingerprint.slice(0, 16)}…${config.AUDIT_ANCHOR_PATH ? ', external anchor on' : ''})`, + ); + } else { + console.warn( + '[middleware] AUDIT_SIGNING_KEY not set — receipt chain builds WITHOUT signed checkpoints; generate a key with scripts/generate-audit-signing-key.mjs (#758)', + ); + } + // Always-on (review LOW): a keyless deployment answers `configured:false` + // instead of an undifferentiated 404 — #761 tooling can discover the + // posture either way. + app.get('/api/v1/operator/provenance/public-key', requireAuth, (_req, res) => { + res.json({ + configured: Boolean(checkpointSigner), + ...(checkpointSigner + ? { + publicKeyPem: checkpointSigner.publicKeyPem, + fingerprint: checkpointSigner.publicKeyFingerprint, + } + : {}), + checkpointIntervalMinutes: config.AUDIT_CHECKPOINT_INTERVAL_MINUTES, + anchorConfigured: Boolean(config.AUDIT_ANCHOR_PATH), + }); + }); + const userStore = new UserStore(graphPool); const bootstrapResult = await runAuthBootstrap({ diff --git a/middleware/src/receipts/chain.ts b/middleware/src/receipts/chain.ts new file mode 100644 index 000000000..6bc69aa3e --- /dev/null +++ b/middleware/src/receipts/chain.ts @@ -0,0 +1,123 @@ +/** + * #758 — receipt hash chain: canonicalization, entry hashing, and segment + * verification. Pure functions — the transactional append lives in + * `store.ts`, the signing in `checkpoints.ts`. + * + * Mechanism (the sentence for the docs): every entry carries the fingerprint + * of its predecessor; editing entry n changes its hash, which no longer + * matches the copy stored in entry n+1 — the chain visibly breaks for every + * later entry. Signed checkpoints anchored outside the DB mean the whole + * chain cannot be silently rewritten either. Detection, not prevention. + */ + +import { createHash } from 'node:crypto'; + +/** Bump when the canonicalization or hash input layout changes; stored per + * row so old chains stay verifiable under their own rules. */ +export const HASH_VERSION = 1; + +/** The one stream #758 ships. More streams (admin_audit, …) join later. */ +export const RECEIPT_STREAM_ID = 'receipts'; + +/** + * Canonical JSON — the RFC 8785 (JCS) subset this codebase needs: objects + * with lexicographically sorted keys (code-unit order), arrays in order, + * `undefined` object members dropped (JSON.stringify semantics), numbers as + * JSON.stringify renders them. Payloads here are JSONB round-trips — + * strings, bounded numbers, booleans, null, plain objects/arrays — so the + * full-JCS number edge cases (±0, exponents beyond double round-trip) cannot + * arise; `hash_version` exists for the day that changes. + */ +export function canonicalJson(value: unknown): string { + if (value === null || typeof value !== 'object') { + return JSON.stringify(value) ?? 'null'; + } + if (Array.isArray(value)) { + return `[${value.map((v) => canonicalJson(v === undefined ? null : v)).join(',')}]`; + } + const record = value as Record; + const keys = Object.keys(record) + .filter((k) => record[k] !== undefined) + .sort(); + const body = keys.map((k) => `${JSON.stringify(k)}:${canonicalJson(record[k])}`).join(','); + return `{${body}}`; +} + +/** Genesis hash of a stream: sha256 over the stream id itself — a fixed, + * reproducible starting anchor an offline verifier can recompute. */ +export function genesisHash(streamId: string): Buffer { + return createHash('sha256').update(`genesis:${streamId}`, 'utf-8').digest(); +} + +export interface ChainEntryInput { + readonly streamId: string; + readonly seq: number; + readonly prevHash: Buffer; + /** The canonical payload — for receipts: {turnId, sessionScope, channel, + * model, receipt}. Never includes DB-generated values (created_at): + * time is anchored by checkpoint cadence, not per-row (documented). */ + readonly payload: unknown; +} + +/** entry_hash = sha256(streamId \n seq \n hex(prevHash) \n canonical(payload)). + * Newline framing keeps fields from bleeding into each other. */ +export function computeEntryHash(input: ChainEntryInput): Buffer { + return createHash('sha256') + .update( + `${input.streamId}\n${String(input.seq)}\n${input.prevHash.toString('hex')}\n${canonicalJson(input.payload)}`, + 'utf-8', + ) + .digest(); +} + +export interface ChainRow { + readonly seq: number; + readonly prevHash: Buffer; + readonly entryHash: Buffer; + readonly payload: unknown; +} + +export type ChainBreakKind = 'hash_mismatch' | 'link_mismatch' | 'seq_gap'; + +export interface ChainVerdict { + readonly ok: boolean; + readonly checkedEntries: number; + readonly firstBrokenSeq?: number; + readonly breakKind?: ChainBreakKind; +} + +/** + * Verify a contiguous ascending-`seq` segment. `trustedPrevHash` is the hash + * the first row must link to: the genesis hash when the segment starts at + * seq 1, or a hash vouched for out-of-band (a signed checkpoint) when the + * prefix was reaped by retention. Foundation for the #761 verify surface; + * shipped here so the chain is testable (tamper tests) from day one. + */ +export function verifyChainSegment( + streamId: string, + rows: readonly ChainRow[], + trustedPrevHash: Buffer, +): ChainVerdict { + let prev = trustedPrevHash; + let expectedSeq: number | undefined; + for (const row of rows) { + if (expectedSeq !== undefined && row.seq !== expectedSeq) { + return { ok: false, checkedEntries: rows.indexOf(row), firstBrokenSeq: row.seq, breakKind: 'seq_gap' }; + } + if (!row.prevHash.equals(prev)) { + return { ok: false, checkedEntries: rows.indexOf(row), firstBrokenSeq: row.seq, breakKind: 'link_mismatch' }; + } + const recomputed = computeEntryHash({ + streamId, + seq: row.seq, + prevHash: row.prevHash, + payload: row.payload, + }); + if (!recomputed.equals(row.entryHash)) { + return { ok: false, checkedEntries: rows.indexOf(row), firstBrokenSeq: row.seq, breakKind: 'hash_mismatch' }; + } + prev = row.entryHash; + expectedSeq = row.seq + 1; + } + return { ok: true, checkedEntries: rows.length }; +} diff --git a/middleware/src/receipts/checkpoints.ts b/middleware/src/receipts/checkpoints.ts new file mode 100644 index 000000000..ce76aa580 --- /dev/null +++ b/middleware/src/receipts/checkpoints.ts @@ -0,0 +1,181 @@ +/** + * #758 — signed audit checkpoints. On an interval, sign the receipt stream's + * current head `(stream_id, seq, head_hash, signed_at)` with Ed25519 and + * persist the checkpoint — plus, when configured, append it to an external + * anchor file (JSONL) OUTSIDE the database, suitable for shipping to WORM + * storage. The private key comes from the environment / secret manager and + * is NEVER stored in Postgres: the admin the chain defends against must not + * be able to re-sign a rewritten chain. + * + * Key format: base64-encoded PKCS#8 DER Ed25519 private key — generate with + * `node scripts/generate-audit-signing-key.mjs`. + */ + +import { appendFile } from 'node:fs/promises'; +import { + createHash, + createPrivateKey, + createPublicKey, + sign as edSign, + type KeyObject, +} from 'node:crypto'; +import type { Pool } from 'pg'; + +import { RECEIPT_STREAM_ID } from './chain.js'; + +export interface CheckpointSigner { + readonly privateKey: KeyObject; + readonly publicKeyPem: string; + /** sha256 over the SPKI DER of the public key, hex — the stable id an + * offline verifier pins. */ + readonly publicKeyFingerprint: string; +} + +/** Parse the configured signing key. Throws with a actionable message on a + * malformed key — a silently-disabled signer would be the #640 no-op. */ +export function loadCheckpointSigner(privateKeyBase64: string): CheckpointSigner { + let privateKey: KeyObject; + try { + privateKey = createPrivateKey({ + key: Buffer.from(privateKeyBase64, 'base64'), + format: 'der', + type: 'pkcs8', + }); + } catch (err) { + throw new Error( + `AUDIT_SIGNING_KEY is not a base64 PKCS#8 Ed25519 private key (generate one with scripts/generate-audit-signing-key.mjs): ${err instanceof Error ? err.message : String(err)}`, + { cause: err }, + ); + } + if (privateKey.asymmetricKeyType !== 'ed25519') { + throw new Error( + `AUDIT_SIGNING_KEY must be an Ed25519 key, got '${privateKey.asymmetricKeyType ?? 'unknown'}'`, + ); + } + const publicKey = createPublicKey(privateKey); + const spkiDer = publicKey.export({ format: 'der', type: 'spki' }); + return { + privateKey, + publicKeyPem: publicKey.export({ format: 'pem', type: 'spki' }).toString(), + publicKeyFingerprint: createHash('sha256').update(spkiDer).digest('hex'), + }; +} + +/** The exact bytes a checkpoint signature covers. Newline-framed, hex for + * the hash — reproducible by an offline verifier with no library beyond + * node:crypto. */ +export function checkpointSigningInput(input: { + streamId: string; + seq: number; + headHash: Buffer; + signedAtIso: string; +}): Buffer { + return Buffer.from( + `omadia-audit-checkpoint-v1\n${input.streamId}\n${String(input.seq)}\n${input.headHash.toString('hex')}\n${input.signedAtIso}`, + 'utf-8', + ); +} + +export interface CheckpointRecord { + readonly streamId: string; + readonly seq: number; + readonly headHashHex: string; + readonly signedAtIso: string; + readonly signatureBase64: string; + readonly publicKeyFingerprint: string; +} + +/** + * One checkpoint pass: read the stream head; if it advanced past the last + * checkpoint, sign and persist (+ optionally anchor). Exported for tests and + * for an eager boot pass. Returns the record written, or undefined when the + * head has not moved (no pointless duplicate checkpoints). + */ +export async function runCheckpointPass( + pool: Pool, + signer: CheckpointSigner, + opts: { anchorPath?: string; now?: () => Date }, +): Promise { + const head = await pool.query<{ head_seq: string; head_hash: Buffer }>( + `SELECT head_seq, head_hash FROM audit_stream_heads WHERE stream_id = $1`, + [RECEIPT_STREAM_ID], + ); + const row = head.rows[0]; + if (!row) return undefined; // nothing recorded yet + const seq = Number(row.head_seq); + const last = await pool.query<{ seq: string }>( + `SELECT MAX(seq)::text AS seq FROM audit_checkpoints WHERE stream_id = $1`, + [RECEIPT_STREAM_ID], + ); + const lastSeq = last.rows[0]?.seq ? Number(last.rows[0].seq) : 0; + if (seq <= lastSeq) return undefined; + + const signedAt = (opts.now?.() ?? new Date()).toISOString(); + const signature = edSign( + null, // Ed25519: algorithm must be null/undefined + checkpointSigningInput({ + streamId: RECEIPT_STREAM_ID, + seq, + headHash: row.head_hash, + signedAtIso: signedAt, + }), + signer.privateKey, + ); + const insertRes = await pool.query( + `INSERT INTO audit_checkpoints + (stream_id, seq, head_hash, signed_at, signature, public_key_fingerprint) + VALUES ($1, $2, $3, $4, $5, $6) + ON CONFLICT (stream_id, seq) DO NOTHING`, + [RECEIPT_STREAM_ID, seq, row.head_hash, signedAt, signature, signer.publicKeyFingerprint], + ); + // Replica race (review M1): the loser of the (stream, seq) conflict must + // NOT anchor or report its own differently-timestamped signature — the + // anchor file has to correspond to what the DB actually stored, or the + // promised anchor↔DB comparison flags phantom "tampering". + if ((insertRes.rowCount ?? 0) === 0) return undefined; + const record: CheckpointRecord = { + streamId: RECEIPT_STREAM_ID, + seq, + headHashHex: row.head_hash.toString('hex'), + signedAtIso: signedAt, + signatureBase64: signature.toString('base64'), + publicKeyFingerprint: signer.publicKeyFingerprint, + }; + if (opts.anchorPath) { + // External anchor: append-only JSONL outside the DB. Failure is loud but + // non-fatal — the in-DB checkpoint stands, and a missing anchor line is + // itself detectable when comparing the two. + try { + await appendFile(opts.anchorPath, `${JSON.stringify(record)}\n`, 'utf-8'); + } catch (err) { + console.error('[receipts] checkpoint anchor append failed:', err); + } + } + return record; +} + +/** Interval worker — unref'd, eager first pass (the table exists: plugin + * activation applies migrations well before this wiring, same reasoning as + * the retention reaper). */ +export function startCheckpointWorker( + pool: Pool, + signer: CheckpointSigner, + opts: { intervalMs: number; anchorPath?: string }, +): { stop: () => void } { + const tick = async (): Promise => { + try { + const record = await runCheckpointPass(pool, signer, { anchorPath: opts.anchorPath }); + if (record) { + console.log( + `[receipts] checkpoint signed: stream=${record.streamId} seq=${String(record.seq)} fingerprint=${record.publicKeyFingerprint.slice(0, 16)}…`, + ); + } + } catch (err) { + console.error('[receipts] checkpoint pass failed:', err); + } + }; + const timer = setInterval(() => void tick(), opts.intervalMs); + timer.unref(); + void tick(); + return { stop: () => clearInterval(timer) }; +} diff --git a/middleware/src/receipts/store.ts b/middleware/src/receipts/store.ts index 697e5a0b8..338469379 100644 --- a/middleware/src/receipts/store.ts +++ b/middleware/src/receipts/store.ts @@ -20,6 +20,13 @@ import type { TurnReceiptStore, } from '@omadia/plugin-api'; +import { + HASH_VERSION, + RECEIPT_STREAM_ID, + computeEntryHash, + genesisHash, +} from './chain.js'; + /** Process-wide failure counters, exported for /health-style introspection * and asserted in tests. Mirrors `runTraceObservability.ts`'s "count what * would otherwise be silently incomplete" obligation. */ @@ -40,16 +47,59 @@ export function resetTurnReceiptCounters(): void { counters.persistFailures = 0; } +/** #758 — the canonical hash payload of a receipt row. NEVER includes + * DB-generated values (created_at): time is anchored by checkpoint cadence, + * not per-row. Exported so the #761 verifier recomputes the identical shape. + * + * The JSON round-trip is load-bearing (review M3): the verifier recomputes + * from the stored JSONB, which honored `toJSON` at write time — hashing the + * live object would canonicalize e.g. a Date to `{}` while the row stores + * its ISO string, a guaranteed spurious mismatch. Round-tripping here makes + * hash input and stored row see the identical plain-JSON value. */ +export function receiptChainPayload(entry: TurnReceiptRecordInput): unknown { + return JSON.parse( + JSON.stringify({ + turnId: entry.turnId, + sessionScope: entry.sessionScope ?? null, + channel: entry.channel ?? null, + model: entry.model ?? null, + receipt: entry.receipt, + }), + ); +} + export class PgTurnReceiptStore implements TurnReceiptStore { constructor(private readonly pool: Pool) {} async record(entry: TurnReceiptRecordInput): Promise { + const client = await this.pool.connect(); try { - // Idempotent on turn_id: a replayed `done` event (retry, double flush) - // must not duplicate the row; first write wins. - const result = await this.pool.query( - `INSERT INTO turn_receipts (turn_id, session_scope, channel, model, receipt) - VALUES ($1, $2, $3, $4, $5::jsonb) + // #758 — chained append. One transaction: lock the stream head + // (FOR UPDATE serializes concurrent appends into a single linear + // chain — no forks), compute seq/prev, insert, advance the head. + // Idempotence on turn_id is preserved: a replayed `done` event hits + // DO NOTHING, and then the head must NOT advance — the transaction + // rolls back to keep head and rows consistent. + await client.query('BEGIN'); + const headRes = await client.query<{ head_seq: string; head_hash: Buffer }>( + `SELECT head_seq, head_hash FROM audit_stream_heads + WHERE stream_id = $1 FOR UPDATE`, + [RECEIPT_STREAM_ID], + ); + const head = headRes.rows[0]; + const prevHash = head ? head.head_hash : genesisHash(RECEIPT_STREAM_ID); + const seq = head ? Number(head.head_seq) + 1 : 1; + const entryHash = computeEntryHash({ + streamId: RECEIPT_STREAM_ID, + seq, + prevHash, + payload: receiptChainPayload(entry), + }); + const inserted = await client.query( + `INSERT INTO turn_receipts + (turn_id, session_scope, channel, model, receipt, + stream_id, seq, prev_hash, entry_hash, hash_version) + VALUES ($1, $2, $3, $4, $5::jsonb, $6, $7, $8, $9, $10) ON CONFLICT (turn_id) DO NOTHING`, [ entry.turnId, @@ -57,16 +107,37 @@ export class PgTurnReceiptStore implements TurnReceiptStore { entry.channel ?? null, entry.model ?? null, JSON.stringify(entry.receipt), + RECEIPT_STREAM_ID, + seq, + prevHash, + entryHash, + HASH_VERSION, ], ); - // A replayed turn hits DO NOTHING (rowCount 0) — that is not a - // persist, and the counter must not overstate the record. - if ((result.rowCount ?? 0) > 0) { - counters.persisted += 1; + if ((inserted.rowCount ?? 0) === 0) { + // Replayed turn: no row, no head movement, no counter. + await client.query('ROLLBACK'); + return; } + await client.query( + `INSERT INTO audit_stream_heads (stream_id, head_seq, head_hash, updated_at) + VALUES ($1, $2, $3, NOW()) + ON CONFLICT (stream_id) + DO UPDATE SET head_seq = EXCLUDED.head_seq, head_hash = EXCLUDED.head_hash, updated_at = NOW()`, + [RECEIPT_STREAM_ID, seq, entryHash], + ); + await client.query('COMMIT'); + counters.persisted += 1; } catch (err) { + try { + await client.query('ROLLBACK'); + } catch { + /* connection-level failure — nothing further to roll back */ + } counters.persistFailures += 1; throw err; + } finally { + client.release(); } } } diff --git a/middleware/test/receiptHashChain.test.ts b/middleware/test/receiptHashChain.test.ts new file mode 100644 index 000000000..9da162c0a --- /dev/null +++ b/middleware/test/receiptHashChain.test.ts @@ -0,0 +1,293 @@ +/** + * #758 — receipt hash chain: canonicalization, entry hashing, segment + * verification (tamper tests), the transactional chained append, and + * Ed25519 checkpoint signing. All against in-memory fakes; the pg wire + * behaviour (FOR UPDATE serialization) is modelled by a stateful fake pool. + */ + +import { strict as assert } from 'node:assert'; +import { generateKeyPairSync, verify as edVerify, createPublicKey } from 'node:crypto'; +import { mkdtempSync, readFileSync } from 'node:fs'; +import { tmpdir } from 'node:os'; +import { join } from 'node:path'; +import { describe, it } from 'node:test'; +import type { Pool } from 'pg'; + +import { + RECEIPT_STREAM_ID, + canonicalJson, + computeEntryHash, + genesisHash, + verifyChainSegment, + type ChainRow, +} from '../src/receipts/chain.js'; +import { + PgTurnReceiptStore, + receiptChainPayload, + resetTurnReceiptCounters, + turnReceiptCounters, +} from '../src/receipts/store.js'; +import { + checkpointSigningInput, + loadCheckpointSigner, + runCheckpointPass, +} from '../src/receipts/checkpoints.js'; + +const RECEIPT = { + datasetsInterned: 1, + fieldsMasked: 4, + fieldsCleartext: 2, + verbsExecuted: ['v4_sort'], + pseudonymProjectionUsed: false, +}; + +describe('#758 canonicalJson', () => { + it('is key-order independent, drops undefined members, keeps array order', () => { + const a = canonicalJson({ b: 1, a: { y: [3, 1], x: 'ü' }, skip: undefined }); + const b = canonicalJson({ a: { x: 'ü', y: [3, 1] }, b: 1 }); + assert.equal(a, b); + assert.equal(a, '{"a":{"x":"ü","y":[3,1]},"b":1}'); + }); +}); + +describe('#758 computeEntryHash + verifyChainSegment', () => { + function buildChain(payloads: unknown[]): ChainRow[] { + const rows: ChainRow[] = []; + let prev = genesisHash(RECEIPT_STREAM_ID); + payloads.forEach((payload, i) => { + const seq = i + 1; + const entryHash = computeEntryHash({ streamId: RECEIPT_STREAM_ID, seq, prevHash: prev, payload }); + rows.push({ seq, prevHash: prev, entryHash, payload }); + prev = entryHash; + }); + return rows; + } + + it('a well-formed chain verifies end to end', () => { + const rows = buildChain([{ n: 1 }, { n: 2 }, { n: 3 }]); + const verdict = verifyChainSegment(RECEIPT_STREAM_ID, rows, genesisHash(RECEIPT_STREAM_ID)); + assert.deepEqual(verdict, { ok: true, checkedEntries: 3 }); + }); + + it('TAMPER: editing a mid-chain payload reports hash_mismatch at exactly that seq', () => { + const rows = buildChain([{ n: 1 }, { n: 2 }, { n: 3 }]); + const tampered = rows.map((r) => (r.seq === 2 ? { ...r, payload: { n: 99 } } : r)); + const verdict = verifyChainSegment(RECEIPT_STREAM_ID, tampered, genesisHash(RECEIPT_STREAM_ID)); + assert.equal(verdict.ok, false); + assert.equal(verdict.firstBrokenSeq, 2); + assert.equal(verdict.breakKind, 'hash_mismatch'); + }); + + it('TAMPER: deleting a mid-chain row reports seq_gap', () => { + const rows = buildChain([{ n: 1 }, { n: 2 }, { n: 3 }]); + const withGap = [rows[0]!, rows[2]!]; + const verdict = verifyChainSegment(RECEIPT_STREAM_ID, withGap, genesisHash(RECEIPT_STREAM_ID)); + assert.equal(verdict.ok, false); + assert.equal(verdict.firstBrokenSeq, 3); + assert.equal(verdict.breakKind, 'seq_gap'); + }); + + it('TAMPER: a re-written chain suffix still fails against the trusted genesis (link_mismatch)', () => { + const rows = buildChain([{ n: 1 }]); + const forged = buildChain([{ n: 999 }]).map((r) => ({ ...r, prevHash: rows[0]!.entryHash })); + const verdict = verifyChainSegment(RECEIPT_STREAM_ID, forged, genesisHash(RECEIPT_STREAM_ID)); + assert.equal(verdict.ok, false); + assert.equal(verdict.breakKind, 'link_mismatch'); + }); +}); + +// ── stateful fake pool modelling the chained-append transaction ──────────── + +interface FakeRow { + turn_id: string; + seq: number; + prev_hash: Buffer; + entry_hash: Buffer; + payload: unknown; +} + +function chainFakePool(): { + pool: Pool; + rows: FakeRow[]; + head: () => { head_seq: number; head_hash: Buffer } | undefined; +} { + const rows: FakeRow[] = []; + const turnIds = new Set(); + let head: { head_seq: number; head_hash: Buffer } | undefined; + const query = async (sql: string, params: unknown[] = []) => { + if (sql.startsWith('BEGIN') || sql.startsWith('COMMIT') || sql.startsWith('ROLLBACK')) { + // The fake applies writes immediately; ROLLBACK-safety is asserted via + // the head/rows invariants in the replay test below. + return { rows: [], rowCount: 0 }; + } + if (sql.includes('FROM audit_stream_heads') && sql.includes('FOR UPDATE')) { + return { rows: head ? [{ head_seq: String(head.head_seq), head_hash: head.head_hash }] : [], rowCount: head ? 1 : 0 }; + } + if (sql.includes('INSERT INTO turn_receipts')) { + const turnId = params[0] as string; + if (turnIds.has(turnId)) return { rows: [], rowCount: 0 }; // ON CONFLICT DO NOTHING + turnIds.add(turnId); + rows.push({ + turn_id: turnId, + seq: params[6] as number, + prev_hash: params[7] as Buffer, + entry_hash: params[8] as Buffer, + payload: JSON.parse(params[4] as string), + }); + return { rows: [], rowCount: 1 }; + } + if (sql.includes('INSERT INTO audit_stream_heads')) { + head = { head_seq: params[1] as number, head_hash: params[2] as Buffer }; + return { rows: [], rowCount: 1 }; + } + throw new Error(`chainFakePool: unscripted SQL: ${sql.slice(0, 80)}`); + }; + const client = { query, release: () => undefined }; + const pool = { connect: async () => client, query } as unknown as Pool; + return { pool, rows, head: () => head }; +} + +describe('#758 PgTurnReceiptStore chained append', () => { + it('three records form a verifiable linear chain from the genesis hash', async () => { + resetTurnReceiptCounters(); + const { pool, rows } = chainFakePool(); + const store = new PgTurnReceiptStore(pool); + for (const id of ['t-1', 't-2', 't-3']) { + await store.record({ turnId: id, sessionScope: 's', receipt: RECEIPT }); + } + assert.equal(rows.length, 3); + const chainRows: ChainRow[] = rows.map((r) => ({ + seq: r.seq, + prevHash: r.prev_hash, + entryHash: r.entry_hash, + // The verifier recomputes from the SAME payload shape the store hashed. + payload: receiptChainPayload({ turnId: r.turn_id, sessionScope: 's', receipt: r.payload as never }), + })); + const verdict = verifyChainSegment(RECEIPT_STREAM_ID, chainRows, genesisHash(RECEIPT_STREAM_ID)); + assert.deepEqual(verdict, { ok: true, checkedEntries: 3 }); + assert.equal(turnReceiptCounters().persisted, 3); + }); + + it('a migration-seeded head (seq 0, genesis hash) yields the identical first append', async () => { + // Review H1 — 0041 seeds the head row so FOR UPDATE always has a row to + // lock. The seeded state (0, genesis) must produce byte-identical chain + // rows to the pre-seed fallback path. + resetTurnReceiptCounters(); + const { pool, rows } = chainFakePool(); + // Simulate the seed by priming the fake's head before any append. + await pool.query( + 'INSERT INTO audit_stream_heads (stream_id, head_seq, head_hash, updated_at) VALUES ($1,$2,$3,NOW())', + [RECEIPT_STREAM_ID, 0, genesisHash(RECEIPT_STREAM_ID)], + ); + await new PgTurnReceiptStore(pool).record({ turnId: 't-1', sessionScope: 's', receipt: RECEIPT }); + assert.equal(rows.length, 1); + assert.equal(rows[0]!.seq, 1); + assert.ok(rows[0]!.prev_hash.equals(genesisHash(RECEIPT_STREAM_ID))); + }); + + it('a replayed turn advances neither rows, nor head, nor the counter', async () => { + resetTurnReceiptCounters(); + const { pool, rows, head } = chainFakePool(); + const store = new PgTurnReceiptStore(pool); + await store.record({ turnId: 't-1', receipt: RECEIPT }); + const headAfterFirst = head()!.head_hash; + await store.record({ turnId: 't-1', receipt: RECEIPT }); // replay + assert.equal(rows.length, 1); + assert.ok(head()!.head_hash.equals(headAfterFirst), 'head must not move on a replay'); + assert.equal(turnReceiptCounters().persisted, 1); + }); +}); + +describe('#758 checkpoint signing', () => { + function makeKey(): string { + const { privateKey } = generateKeyPairSync('ed25519'); + return privateKey.export({ format: 'der', type: 'pkcs8' }).toString('base64'); + } + + it('loadCheckpointSigner rejects a non-Ed25519 key loudly', () => { + const { privateKey } = generateKeyPairSync('rsa', { modulusLength: 2048 }); + const rsaB64 = privateKey.export({ format: 'der', type: 'pkcs8' }).toString('base64'); + assert.throws(() => loadCheckpointSigner(rsaB64), /must be an Ed25519 key/); + assert.throws(() => loadCheckpointSigner('not-a-key'), /AUDIT_SIGNING_KEY/); + }); + + function checkpointFakePool(headSeq: number | undefined): { pool: Pool; inserted: unknown[][] } { + const inserted: unknown[][] = []; + let lastCheckpointSeq: number | undefined; + const pool = { + query: async (sql: string, params: unknown[] = []) => { + if (sql.includes('FROM audit_stream_heads')) { + return headSeq === undefined + ? { rows: [], rowCount: 0 } + : { rows: [{ head_seq: String(headSeq), head_hash: genesisHash('x') }], rowCount: 1 }; + } + if (sql.includes('MAX(seq)')) { + return { rows: [{ seq: lastCheckpointSeq === undefined ? null : String(lastCheckpointSeq) }], rowCount: 1 }; + } + if (sql.includes('INSERT INTO audit_checkpoints')) { + inserted.push(params); + lastCheckpointSeq = params[1] as number; + return { rows: [], rowCount: 1 }; + } + throw new Error(`checkpointFakePool: unscripted SQL: ${sql.slice(0, 60)}`); + }, + } as unknown as Pool; + return { pool, inserted }; + } + + it('signs the head, is verifiable with the public key, anchors externally, and never duplicates', async () => { + const signer = loadCheckpointSigner(makeKey()); + const { pool, inserted } = checkpointFakePool(7); + const anchorPath = join(mkdtempSync(join(tmpdir(), 'anchor-')), 'anchors.jsonl'); + const record = await runCheckpointPass(pool, signer, { anchorPath }); + assert.ok(record); + assert.equal(record.seq, 7); + // Signature verifies against the exported public key over the documented input. + const ok = edVerify( + null, + checkpointSigningInput({ + streamId: record.streamId, + seq: record.seq, + headHash: Buffer.from(record.headHashHex, 'hex'), + signedAtIso: record.signedAtIso, + }), + createPublicKey(signer.publicKeyPem), + Buffer.from(record.signatureBase64, 'base64'), + ); + assert.equal(ok, true, 'checkpoint signature must verify with the public key'); + // External anchor line landed and round-trips. + const anchored = JSON.parse(readFileSync(anchorPath, 'utf-8').trim()) as { seq: number }; + assert.equal(anchored.seq, 7); + // Head unchanged ⇒ second pass writes nothing (no duplicate checkpoints). + const again = await runCheckpointPass(pool, signer, { anchorPath }); + assert.equal(again, undefined); + assert.equal(inserted.length, 1); + }); + + it('a stream with no head yet produces no checkpoint (nothing to certify)', async () => { + const signer = loadCheckpointSigner(makeKey()); + const { pool, inserted } = checkpointFakePool(undefined); + assert.equal(await runCheckpointPass(pool, signer, {}), undefined); + assert.equal(inserted.length, 0); + }); + + it('the LOSING replica of a checkpoint race neither anchors nor reports (review M1)', async () => { + const signer = loadCheckpointSigner(makeKey()); + // Fake where the checkpoint INSERT loses the (stream, seq) conflict. + const pool = { + query: async (sql: string) => { + if (sql.includes('FROM audit_stream_heads')) { + return { rows: [{ head_seq: '7', head_hash: genesisHash('x') }], rowCount: 1 }; + } + if (sql.includes('MAX(seq)')) return { rows: [{ seq: null }], rowCount: 1 }; + if (sql.includes('INSERT INTO audit_checkpoints')) return { rows: [], rowCount: 0 }; + throw new Error('unscripted'); + }, + } as unknown as Pool; + const anchorPath = join(mkdtempSync(join(tmpdir(), 'anchor-loser-')), 'anchors.jsonl'); + const record = await runCheckpointPass(pool, signer, { anchorPath }); + assert.equal(record, undefined, 'the loser must not report a checkpoint'); + // The anchor file must not exist — nothing was appended. + assert.throws(() => readFileSync(anchorPath, 'utf-8')); + }); +}); diff --git a/middleware/test/turnReceipts.test.ts b/middleware/test/turnReceipts.test.ts index 92ddfdd3d..77e997bc7 100644 --- a/middleware/test/turnReceipts.test.ts +++ b/middleware/test/turnReceipts.test.ts @@ -31,14 +31,16 @@ function fakePool( handler: (sql: string, params: unknown[]) => { rows?: unknown[]; rowCount?: number } | Error, ): { pool: Pool; queries: RecordedQuery[] } { const queries: RecordedQuery[] = []; - const pool = { - query: async (sql: string, params: unknown[] = []) => { - queries.push({ sql, params }); - const result = handler(sql, params); - if (result instanceof Error) throw result; - return { rows: result.rows ?? [], rowCount: result.rowCount ?? (result.rows?.length ?? 0) }; - }, - } as unknown as Pool; + const query = async (sql: string, params: unknown[] = []) => { + queries.push({ sql, params }); + const result = handler(sql, params); + if (result instanceof Error) throw result; + return { rows: result.rows ?? [], rowCount: result.rowCount ?? (result.rows?.length ?? 0) }; + }; + // #758 — the store's chained append runs on a dedicated client (BEGIN … + // COMMIT); route/reaper paths keep using pool.query directly. + const client = { query, release: () => undefined }; + const pool = { query, connect: async () => client } as unknown as Pool; return { pool, queries }; } @@ -62,14 +64,24 @@ describe('#757 PgTurnReceiptStore', () => { model: 'claude-test', receipt: RECEIPT, }); - assert.equal(queries.length, 1); - const q = queries[0]!; - assert.match(q.sql, /INSERT INTO turn_receipts/); + const q = queries.find((x) => x.sql.includes('INSERT INTO turn_receipts'))!; + assert.ok(q, 'a receipt INSERT must run'); // Idempotence is the SQL's job: a replayed done event must hit the // turn_id conflict target, not add a second row. assert.match(q.sql, /ON CONFLICT \(turn_id\) DO NOTHING/); assert.deepEqual(q.params.slice(0, 4), ['t-1', 'sess-1', 'teams', 'claude-test']); assert.deepEqual(JSON.parse(q.params[4] as string), RECEIPT); + // #758 — the row joins the hash chain: stream, seq 1 (genesis append), + // 32-byte prev/entry hashes, hash version. + assert.equal(q.params[5], 'receipts'); + assert.equal(q.params[6], 1); + assert.equal((q.params[7] as Buffer).length, 32); + assert.equal((q.params[8] as Buffer).length, 32); + assert.equal(q.params[9], 1); + assert.ok( + queries.some((x) => x.sql.includes('INSERT INTO audit_stream_heads')), + 'the stream head must advance with the row', + ); assert.equal(turnReceiptCounters().persisted, 1); assert.equal(turnReceiptCounters().persistFailures, 0); }); @@ -90,15 +102,21 @@ describe('#757 PgTurnReceiptStore', () => { resetTurnReceiptCounters(); const { pool, queries } = fakePool(() => ({ rowCount: 1 })); await new PgTurnReceiptStore(pool).record({ turnId: 't-3', receipt: RECEIPT }); - assert.deepEqual(queries[0]!.params.slice(0, 4), ['t-3', null, null, null]); + const q = queries.find((x) => x.sql.includes('INSERT INTO turn_receipts'))!; + assert.deepEqual(q.params.slice(0, 4), ['t-3', null, null, null]); }); - it('a replayed turn (ON CONFLICT no-op) does not inflate the persisted counter', async () => { + it('a replayed turn (ON CONFLICT no-op) rolls back: no counter, no head movement', async () => { resetTurnReceiptCounters(); - const { pool } = fakePool(() => ({ rowCount: 0 })); + const { pool, queries } = fakePool((sql) => + sql.includes('INSERT INTO turn_receipts') ? { rowCount: 0 } : { rowCount: 1, rows: [] }, + ); await new PgTurnReceiptStore(pool).record({ turnId: 't-4', receipt: RECEIPT }); assert.equal(turnReceiptCounters().persisted, 0); assert.equal(turnReceiptCounters().persistFailures, 0); + // #758 — the head must not advance for a row that was never inserted. + assert.ok(!queries.some((x) => x.sql.includes('INSERT INTO audit_stream_heads'))); + assert.ok(queries.some((x) => x.sql.startsWith('ROLLBACK'))); }); });