fix(recovery): prove restore catalog index decoys

docs/adr/0018-postgres-restore-catalog-acceptance.md

@@ -0,0 +1,93 @@
+# ADR 0018: Isolated restore catalog acceptance without executor authority
+
+- **Status:** Proposed
+- **Date:** 2026-08-16
+
+## Context
+
+Protected main can hash a backup artifact and the packaged `schema.sql` resource,
+and it can encode a content-free recovery receipt. Those primitives do not prove
+that an isolated restore target contains the package-owned tables, tenant-status
+indexes, or forced row-level security that a buyer needs before using the
+database again.
+
+Executable `pg_dump` and `pg_restore` remain separate active writers. A catalog
+acceptance probe must not open a package-owned connection, must not execute
+backup or restore, and must not treat a receipt or hash as restorability.
+
+PostgreSQL publishes `relrowsecurity` and `relforcerowsecurity` on `pg_class`
+as the catalog truth for whether row-level security is enabled and whether it
+also applies to the table owner (PostgreSQL Global Development Group, 2026b,
+2026c). Index ownership, key identity, uniqueness, validity, and access method
+are published on `pg_index` and `pg_am`, and `pg_get_indexdef` reconstructs
+each key when a column number is supplied (PostgreSQL Global Development
+Group, 2026d, 2026e, 2026f). Contingency planning requires proving that
+recovered information-system state is usable, not only that a backup command
+returned zero (Swanson et al., 2010).
+
+## Decision
+
+Add `inspect_postgres_restore_catalog(connection)` as a caller-owned catalog
+probe:
+
+1. The caller supplies an already-authorized connection. The package does not
+   parse a DSN, inherit libpq defaults, or create a connection.
+2. The probe reads `pg_class` in `current_schema()` through one parameterized
+   query. Relation names and kinds are bound; they are never concatenated into
+   SQL.
+3. Required packaged-schema tables and the tenant-qualified lifecycle unique
+   index plus tenant-status index must be present. Those indexes must belong to
+   `llm_remote_batch_jobs`, use the packaged key order, stay valid and ready,
+   remain plain btree indexes with default key options, and, for the unique
+   identity, remain a non-deferrable unique constraint. Missing or same-name
+   decoy objects fail closed.
+4. `llm_remote_batch_jobs` must have row-level security enabled and forced.
+   If `llm_result_stream_checkpoints` is present, it must also be forced. A
+   schema-init target that has not applied migration 0007 may omit the
+   checkpoint store.
+5. Evidence is content-free: counts, boolean RLS flags, and the packaged schema
+   SHA-256/size from `inspect_postgres_schema()`. Paths, DSNs, SQL text, and
+   lower-layer diagnostics never enter the evidence or exception text.
+6. Success does not mean a backup is restorable, a live cluster matches every
+   constraint/function body, PITR works, or a stated RPO/RTO is met.
+
+## Consequences
+
+Operators can fail an isolated restore target before they point production
+traffic at it, using the same packaged schema identity already hashed on
+protected main. Backup and restore executors stay on their own lanes. Canonical
+PRD/TRD/CHANGELOG status remains owned by those writers until this probe
+integrates.
+
+## Rollback
+
+Remove `pg_llm_batch/postgres_restore_acceptance.py`, its tests, this ADR, and
+the doctoring record. No schema migration or durable state conversion is
+required.
+
+## References
+
+PostgreSQL Global Development Group. (2026a). *Backup and restore*. In
+*PostgreSQL 18 documentation*. https://www.postgresql.org/docs/18/backup.html
+
+PostgreSQL Global Development Group. (2026b). *pg_class*. In *PostgreSQL 18
+documentation*. https://www.postgresql.org/docs/18/catalog-pg-class.html
+
+PostgreSQL Global Development Group. (2026c). *Row security policies*. In
+*PostgreSQL 18 documentation*.
+https://www.postgresql.org/docs/18/ddl-rowsecurity.html
+
+PostgreSQL Global Development Group. (2026d). *pg_index*. In *PostgreSQL 18
+documentation*. https://www.postgresql.org/docs/18/catalog-pg-index.html
+
+PostgreSQL Global Development Group. (2026e). *pg_am*. In *PostgreSQL 18
+documentation*. https://www.postgresql.org/docs/18/catalog-pg-am.html
+
+PostgreSQL Global Development Group. (2026f). *System information functions and
+operators*. In *PostgreSQL 18 documentation*.
+https://www.postgresql.org/docs/18/functions-info.html
+
+Swanson, M., Bowen, P., Phillips, A., Gallup, D., & Lynes, D. (2010).
+*Contingency planning guide for federal information systems* (NIST Special
+Publication 800-34 Rev. 1). National Institute of Standards and Technology.
+https://doi.org/10.6028/NIST.SP.800-34r1

docs/doctoring/postgres-restore-acceptance.md

@@ -0,0 +1,85 @@
+# Isolated PostgreSQL restore catalog acceptance
+
+## What to do next
+
+After you restore into an isolated libpq target, call
+`inspect_postgres_restore_catalog(connection)` on a connection you already
+opened to that target. Do not point production traffic at the target until the
+probe returns. If it raises, keep the target isolated, do not retry a
+transactional restore into the same service, and compare the packaged schema
+hash on the evidence with the backup you intended to apply.
+
+## Claim boundary
+
+This probe proves that the caller-owned connection can see the required
+package-owned tables, the tenant-qualified lifecycle unique index, the
+tenant-status index, and forced row-level security on
+`llm_remote_batch_jobs`. The required indexes must belong to
+`llm_remote_batch_jobs` and match the packaged key order, uniqueness,
+validity, readiness, btree access method, and default key options. A
+same-name index on another table, with swapped keys, or without the unique
+constraint, is incomplete. If `llm_result_stream_checkpoints` is present, it
+must also be forced.
+
+It does **not** execute `pg_dump` or `pg_restore`, open a package-owned
+connection, prove every constraint or function body, prove live-cluster parity
+with `schema.sql` byte-for-byte, authenticate an operator, or establish
+PITR/RPO/RTO/HA/DR, CSAP, or SOC 2 readiness. A recovery receipt or artifact
+hash is not acceptance.
+
+## Operator contract
+
+1. Restore only into an isolated target whose credentials you already trust.
+2. Open the connection yourself. Pass that connection object. The package never
+   reads a DSN for this probe.
+3. On success, record the returned `expected_schema_sha256` and
+   `expected_schema_size_bytes` next to the backup artifact hash you already
+   have. Those values come from the packaged `schema.sql` resource, not from
+   hashing the live cluster.
+4. Treat `checkpoint_store_present=false` as a schema-init target that has not
+   applied migration 0007. Do not treat it as proof that checkpoints were
+   intentionally empty.
+5. On `PostgresRestoreAcceptanceError`, read only the fixed category. The
+   exception text never contains SQL, DSNs, passwords, or catalog dumps.
+
+## Verification
+
+Focused tests prove a complete isolated catalog is accepted, a missing
+lifecycle table or tenant-status index is incomplete, unforced lifecycle or
+checkpoint RLS fails tenant-isolation checks, hostile name subclasses and
+lower-layer diagnostics stay out of exceptions, and the probe binds
+`current_schema()` with parameters. The catalog SQL is bound to the packaged
+`UNIQUE (tenant_scope, endpoint_alias, remote_batch_id)` and
+`idx_llm_remote_batch_jobs_tenant_status_observed` column order. The
+container smoke creates same-name decoys on the live packaged image and
+requires `inspect_postgres_restore_catalog` to fail closed, then restore the
+packaged indexes and accept again. Coverage for
+`pg_llm_batch.postgres_restore_acceptance` is 100% statement and branch.
+Public docstrings are complete.
+
+## References
+
+PostgreSQL Global Development Group. (2026a). *Backup and restore*. In
+*PostgreSQL 18 documentation*. https://www.postgresql.org/docs/18/backup.html
+
+PostgreSQL Global Development Group. (2026b). *pg_class*. In *PostgreSQL 18
+documentation*. https://www.postgresql.org/docs/18/catalog-pg-class.html
+
+PostgreSQL Global Development Group. (2026c). *Row security policies*. In
+*PostgreSQL 18 documentation*.
+https://www.postgresql.org/docs/18/ddl-rowsecurity.html
+
+PostgreSQL Global Development Group. (2026d). *pg_index*. In *PostgreSQL 18
+documentation*. https://www.postgresql.org/docs/18/catalog-pg-index.html
+
+PostgreSQL Global Development Group. (2026e). *pg_am*. In *PostgreSQL 18
+documentation*. https://www.postgresql.org/docs/18/catalog-pg-am.html
+
+PostgreSQL Global Development Group. (2026f). *System information functions and
+operators*. In *PostgreSQL 18 documentation*.
+https://www.postgresql.org/docs/18/functions-info.html
+
+Swanson, M., Bowen, P., Phillips, A., Gallup, D., & Lynes, D. (2010).
+*Contingency planning guide for federal information systems* (NIST Special
+Publication 800-34 Rev. 1). National Institute of Standards and Technology.
+https://doi.org/10.6028/NIST.SP.800-34r1

pg_llm_batch/postgres_restore_acceptance.py

@@ -0,0 +1,213 @@
+# SPDX-License-Identifier: Apache-2.0
+"""Inspect a caller-owned PostgreSQL catalog after an isolated restore."""
+
+from __future__ import annotations
+
+from dataclasses import dataclass
+
+from pg_llm_batch.postgres_schema_evidence import inspect_postgres_schema
+
+
+_REQUIRED_TABLES = (
+    "com_config",
+    "com_secrets",
+    "llm_queues",
+    "llm_batches",
+    "llm_remote_batch_jobs",
+    "llm_batch_file_payloads",
+    "llm_batch_files",
+    "llm_requests",
+    "llm_jsonl_lines",
+    "llm_endpoints",
+    "llm_endpoint_models",
+)
+_REQUIRED_INDEXES = (
+    "idx_llm_remote_batch_jobs_tenant_status_observed",
+    "uq_llm_remote_batch_jobs_tenant_endpoint_id",
+)
+_CHECKPOINT_TABLE = "llm_result_stream_checkpoints"
+_ALLOWED_NAMES = frozenset(_REQUIRED_TABLES + _REQUIRED_INDEXES + (_CHECKPOINT_TABLE,))
+_RELATION_KINDS = frozenset({"r", "i"})
+_MAX_CATALOG_ROWS = 16
+_CATALOG_SQL = """
+SELECT c.relname, c.relkind, c.relrowsecurity, c.relforcerowsecurity
+FROM pg_catalog.pg_class AS c
+INNER JOIN pg_catalog.pg_namespace AS n ON n.oid = c.relnamespace
+LEFT JOIN pg_catalog.pg_index AS idx ON idx.indexrelid = c.oid
+LEFT JOIN pg_catalog.pg_class AS indexed_table ON indexed_table.oid = idx.indrelid
+WHERE n.nspname = current_schema()
+  AND c.relkind = ANY(%s)
+  AND c.relname = ANY(%s)
+  AND (
+      c.relkind = 'r'
+      OR (
+          c.relkind = 'i'
+          AND indexed_table.relname = 'llm_remote_batch_jobs'
+          AND indexed_table.relnamespace = n.oid
+          AND idx.indisvalid
+          AND idx.indisready
+          AND idx.indpred IS NULL
+          AND idx.indexprs IS NULL
+          AND idx.indnkeyatts = 3
+          AND idx.indnatts = 3
+          AND idx.indoption = '0 0 0'::pg_catalog.int2vector
+          AND EXISTS (
+              SELECT 1
+              FROM pg_catalog.pg_am AS access_method
+              WHERE access_method.oid = c.relam
+                AND access_method.amname = 'btree'
+          )
+          AND pg_catalog.pg_get_indexdef(c.oid, 1, TRUE) = 'tenant_scope'
+          AND (
+              (
+                  c.relname = 'idx_llm_remote_batch_jobs_tenant_status_observed'
+                  AND NOT idx.indisunique
+                  AND pg_catalog.pg_get_indexdef(c.oid, 2, TRUE) = 'batch_status'
+                  AND pg_catalog.pg_get_indexdef(c.oid, 3, TRUE) = 'last_observed_at'
+              )
+              OR (
+                  c.relname = 'uq_llm_remote_batch_jobs_tenant_endpoint_id'
+                  AND idx.indisunique
+                  AND pg_catalog.pg_get_indexdef(c.oid, 2, TRUE) = 'endpoint_alias'
+                  AND pg_catalog.pg_get_indexdef(c.oid, 3, TRUE) = 'remote_batch_id'
+                  AND EXISTS (
+                      SELECT 1
+                      FROM pg_catalog.pg_constraint AS constraint_row
+                      WHERE constraint_row.conindid = c.oid
+                        AND constraint_row.conrelid = indexed_table.oid
+                        AND constraint_row.contype = 'u'
+                        AND NOT constraint_row.condeferrable
+                  )
+              )
+          )
+      )
+  )
+""".strip()
+
+
+class PostgresRestoreAcceptanceError(ValueError):
+    """Report a fail-closed isolated restore catalog acceptance violation."""
+
+
+@dataclass(frozen=True, slots=True)
+class PostgresRestoreCatalogEvidence:
+    """Represent content-free catalog acceptance for one isolated restore target."""
+
+    required_table_count: int
+    required_index_count: int
+    lifecycle_rls_enabled: bool
+    lifecycle_rls_forced: bool
+    checkpoint_store_present: bool
+    checkpoint_store_rls_forced: bool
+    expected_schema_sha256: str
+    expected_schema_size_bytes: int
+
+    def as_dict(self) -> dict[str, object]:
+        """Return the stable machine-readable restore-catalog evidence schema."""
+        return {
+            "required_table_count": self.required_table_count,
+            "required_index_count": self.required_index_count,
+            "lifecycle_rls_enabled": self.lifecycle_rls_enabled,
+            "lifecycle_rls_forced": self.lifecycle_rls_forced,
+            "checkpoint_store_present": self.checkpoint_store_present,
+            "checkpoint_store_rls_forced": self.checkpoint_store_rls_forced,
+            "expected_schema_sha256": self.expected_schema_sha256,
+            "expected_schema_size_bytes": self.expected_schema_size_bytes,
+        }
+
+
+def _invalid_catalog() -> None:
+    """Reject malformed catalog evidence without reflecting row contents."""
+    raise PostgresRestoreAcceptanceError(
+        "PostgreSQL restore catalog evidence is invalid"
+    )
+
+
+def _evaluate_catalog_rows(
+    rows: object,
+) -> PostgresRestoreCatalogEvidence:
+    """Validate one finite catalog snapshot against the packaged restore contract."""
+    if type(rows) is not list or len(rows) > _MAX_CATALOG_ROWS:
+        _invalid_catalog()
+    tables: dict[str, tuple[bool, bool]] = {}
+    indexes: dict[str, bool] = {}
+    for row in rows:
+        if type(row) is not tuple or len(row) != 4:
+            _invalid_catalog()
+        relation_name, relation_kind, row_security, force_row_security = row
+        if (
+            type(relation_name) is not str
+            or relation_name not in _ALLOWED_NAMES
+            or type(relation_kind) is not str
+            or relation_kind not in _RELATION_KINDS
+            or type(row_security) is not bool
+            or type(force_row_security) is not bool
+        ):
+            _invalid_catalog()
+        if relation_kind == "r":
+            if relation_name in tables:
+                _invalid_catalog()
+            tables[relation_name] = (row_security, force_row_security)
+        else:
+            if relation_name in indexes:
+                _invalid_catalog()
+            indexes[relation_name] = True
+    if any(name not in tables for name in _REQUIRED_TABLES):
+        raise PostgresRestoreAcceptanceError(
+            "PostgreSQL restore catalog is incomplete"
+        )
+    if any(name not in indexes for name in _REQUIRED_INDEXES):
+        raise PostgresRestoreAcceptanceError(
+            "PostgreSQL restore catalog is incomplete"
+        )
+    lifecycle_security = tables["llm_remote_batch_jobs"]
+    if not lifecycle_security[0] or not lifecycle_security[1]:
+        raise PostgresRestoreAcceptanceError(
+            "PostgreSQL restore catalog failed tenant isolation checks"
+        )
+    checkpoint_present = _CHECKPOINT_TABLE in tables
+    checkpoint_security = tables.get(_CHECKPOINT_TABLE, (False, False))
+    if checkpoint_present and (not checkpoint_security[0] or not checkpoint_security[1]):
+        raise PostgresRestoreAcceptanceError(
+            "PostgreSQL restore catalog failed tenant isolation checks"
+        )
+    schema = inspect_postgres_schema()
+    return PostgresRestoreCatalogEvidence(
+        required_table_count=len(_REQUIRED_TABLES),
+        required_index_count=len(_REQUIRED_INDEXES),
+        lifecycle_rls_enabled=lifecycle_security[0],
+        lifecycle_rls_forced=lifecycle_security[1],
+        checkpoint_store_present=checkpoint_present,
+        checkpoint_store_rls_forced=checkpoint_security[1],
+        expected_schema_sha256=schema.sha256,
+        expected_schema_size_bytes=schema.size_bytes,
+    )
+
+
+def inspect_postgres_restore_catalog(
+    connection: object,
+) -> PostgresRestoreCatalogEvidence:
+    """Prove required package catalog objects on a caller-owned restore target.
+
+    The callable inspects ``pg_class`` through the caller-owned connection. It
+    does not execute ``pg_dump`` or ``pg_restore``, open a package-owned
+    connection, or claim that a backup artifact is restorable. Missing required
+    tables or tenant-status indexes fail closed. Lifecycle row-level security
+    must be enabled and forced. A present checkpoint store must also be forced.
+    Required lifecycle indexes must belong to that lifecycle table and match the
+    packaged key order, uniqueness, validity, readiness, btree access method,
+    default key options, and plain-index shape.
+    """
+    relation_names = list(_REQUIRED_TABLES + _REQUIRED_INDEXES + (_CHECKPOINT_TABLE,))
+    try:
+        with connection.cursor() as cursor:
+            cursor.execute(
+                _CATALOG_SQL,
+                (["r", "i"], relation_names),
+            )
+            rows = cursor.fetchall()
+    except Exception:
+        raise PostgresRestoreAcceptanceError(
+            "PostgreSQL restore catalog could not be inspected"
+        ) from None
+    return _evaluate_catalog_rows(rows)