Skip to content
Closed
Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
17 commits
Select commit Hold shift + click to select a range
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
30 changes: 30 additions & 0 deletions AGENTS.md
Original file line number Diff line number Diff line change
Expand Up @@ -237,3 +237,33 @@ add CODEOWNERS-based merge gates until multiple independent maintainers exist.
deterministic transaction, event-time, tenant, validation, bounded-read,
migration, rollback, immutability-trigger, compatibility, and documentation
tests.

## Checkpoint audit export pagination contract

- Keep pagination opt-in and preserve `list_audit_events()` behavior. Do not
replace the existing bounded convenience read with an unbounded iterator.
- Use `checkpoint_audit_event_id` as a strict positive signed-PostgreSQL-`BIGINT`
keyset cursor. Do not use `OFFSET` for multi-page audit traversal.
- Query at most the validated public limit plus one lookahead row and expose at
most the requested 1..1,000 events. A driver returning more than the SQL bound
fails closed.
- Continuations must use `checkpoint_audit_event_id < before_audit_event_id` in
exact newest-first order. When more rows exist, the next cursor is exactly the
final returned event identity.
- Revalidate every database row through `CheckpointAuditEvent`, compare its
tenant/consumer/endpoint/batch key to the trusted request, and require strict
descending identities before exposing a page.
- Treat the cursor as navigation state only. It is not proof of completeness,
chronology, authenticity, delivery, or non-repudiation; identity allocation
can contain gaps and commit ordering can differ from allocation ordering.
- Package-owned page calls do not promise a single historic snapshot. Hosts
requiring one export snapshot must begin their own PostgreSQL `REPEATABLE READ`
or stricter transaction before the first query and repeatedly call
`list_audit_event_page_in_transaction()` on the same transaction.
- Keep export destinations, credentials, retention policy, immutable/WORM
storage, delivery receipts, cryptographic manifests, and reconciliation in the
host/operator boundary. This package primitive adds no network exporter or
write authority.
- Maintain 100% production statement, branch, and public-docstring coverage with
strict cursor, page-shape, lookahead, keyset SQL, ordering, row-key,
malformed-driver, concurrency-semantics, and transaction-ownership tests.
51 changes: 47 additions & 4 deletions ARCHITECTURE.md
Original file line number Diff line number Diff line change
Expand Up @@ -261,6 +261,43 @@ existing volumes require explicit reviewed migration because Docker entrypoint
initialization is not an upgrade mechanism. Audit retention, export, legal hold,
and disposal remain host/operator policy.

## Bounded checkpoint-audit export boundary

`CheckpointAuditPage`, `list_audit_event_page()`, and
`list_audit_event_page_in_transaction()` add an opt-in bounded traversal layer
without changing the existing one-page audit read or database schema. The page
identity remains the exact tenant, consumer, endpoint, and remote-batch key.

Pagination is primary-key keyset traversal rather than offset traversal. The
first page orders by `checkpoint_audit_event_id DESC`; a continuation adds
`checkpoint_audit_event_id < before_audit_event_id` with the same ordering. The
cursor is strict, non-coercive, positive, and bounded by PostgreSQL signed
`BIGINT`. Each query requests at most the validated public limit plus one
lookahead row and exposes at most 1,000 events. The lookahead is used only to
decide whether an older continuation exists.

Rows are not trusted merely because PostgreSQL returned them. Every row is
reconstructed through the public event validator, compared to the exact trusted
request key, and required to be strictly descending by audit identity. A malformed
collection, impossible driver overrun, cross-key row, duplicate identity, or
ascending identity fails closed before a page is exposed.

Keyset traversal prevents later higher-identity inserts from shifting an older
continuation window, but it is not a multi-page database snapshot. Package-owned
page calls may observe different committed database states. A host that requires
one snapshot for a long export pass must begin a caller-owned PostgreSQL
`REPEATABLE READ` or stricter transaction before the first query and repeatedly
call the in-transaction method on that same transaction. The package never
silently changes caller isolation.

Audit identities are database navigation keys, not cryptographic chronology.
Identity sequences may contain gaps and allocation order can differ from commit
order. A cursor therefore does not prove completeness, delivery, authenticity,
retention, or non-repudiation. Hosts still own external immutable/WORM storage,
export receipts, cryptographic manifests where required, reconciliation, legal
hold, and disposal. This boundary exists so those controls can consume retained
evidence without unbounded package memory or offset drift.

## Checkpoint OpenTelemetry observability boundary

`OpenTelemetryCheckpointStore` is an opt-in wrapper around a durable checkpoint
Expand Down Expand Up @@ -327,7 +364,10 @@ place local PostgreSQL effects and `save_in_transaction()` on the same caller
cursor; cross-system effects remain host-owned recovery boundaries. Hosts that
choose the audited store gain transaction-coupled accepted-save evidence but
still own identity authorization, retention, export, and stronger tamper-proof
controls where required.
controls where required. Hosts may use bounded audit-page keyset traversal as a
standalone export primitive or embed it into a larger MSA retention workflow;
package pagination itself does not require `contextual-orchestrator`, `naruon`,
or a network export service.

## Verification boundary

Expand Down Expand Up @@ -367,9 +407,12 @@ insert-time wall-clock semantics, rejected-save no-success evidence, strict even
revalidation, tenant-qualified bounded reads, package/container migration
identity and idempotent timestamp-default repair, forced RLS, UPDATE/DELETE and
TRUNCATE rejection, fresh-image installation order, fail-closed non-empty
rollback, and explicit administrator-tamper exclusions. Checkpoint telemetry
tests additionally prove exact delegation, fixed low-cardinality signal
attributes, storage-agnostic operation spans, seconds-based nonnegative duration,
rollback, explicit administrator-tamper exclusions, strict PostgreSQL-BIGINT
identity compatibility, immutable page shape, bounded lookahead, keyset cursor
validation, trusted-key row revalidation, strict descending order, malformed
driver output, and package-owned no-commit page reads. Checkpoint telemetry tests
additionally prove exact delegation, fixed low-cardinality signal attributes,
storage-agnostic operation spans, seconds-based nonnegative duration,
confidential failure classification, explicit Error status without descriptions,
Unset success status, disabled exception recording, and preservation of
application results and exception identity during ordinary tracer, meter, span,
Expand Down
12 changes: 12 additions & 0 deletions CHANGELOG.md
Original file line number Diff line number Diff line change
Expand Up @@ -9,6 +9,18 @@ and this project adheres to [Semantic Versioning](https://semver.org/spec/v2.0.0

### Added

- Stable bounded checkpoint-audit export pages through `CheckpointAuditPage`,
`list_audit_event_page()`, and `list_audit_event_page_in_transaction()`. The
cursor is a strict positive PostgreSQL `BIGINT`; continuation uses
`checkpoint_audit_event_id < before_audit_event_id` rather than `OFFSET`, the
SQL query requests at most one bounded lookahead row, and returned rows are
revalidated against the exact trusted audit key and strict descending order.
Each package-owned page uses its own database transaction; the in-transaction
method never commits or rolls back the caller's transaction. Hosts that need
one database snapshot across pages must own a `REPEATABLE READ` or stricter
transaction. The cursor is navigation state rather than completeness,
authenticity, or non-repudiation evidence. No migration, exporter credential,
version bump, or release is included.
- Optional append-only checkpoint accepted-save audit evidence through
`AuditedPostgresBatchResultCheckpointStore` and
`llm_result_checkpoint_audit_events`. Successful save calls append a fixed
Expand Down
31 changes: 31 additions & 0 deletions CLAUDE.md
Original file line number Diff line number Diff line change
Expand Up @@ -214,3 +214,34 @@
- Maintain test-first transaction, event-time, validation, bounded-read,
migration, rollback, trigger, compatibility, documentation, and 100%
production statement/branch/public-docstring coverage.

## Checkpoint audit export pagination invariants

- Preserve `list_audit_events()` and make multi-page export explicit through
`CheckpointAuditPage` and the two `list_audit_event_page*` methods.
- Accept only `None` or a positive signed-PostgreSQL-`BIGINT` continuation
identity. Never coerce strings, floats, booleans, or oversized integers.
- Use strict newest-first keyset pagination with
`checkpoint_audit_event_id < before_audit_event_id`; never introduce
`OFFSET` into the export path.
- Fetch only `limit + 1` rows to determine continuation and return at most the
validated 1..1,000 public events. Treat a driver that returns more as an
integrity failure.
- Revalidate each row through `CheckpointAuditEvent`, require the exact trusted
tenant/consumer/endpoint/batch key, and reject duplicate or non-descending
identities before constructing the immutable page.
- Set `next_before_audit_event_id` only when a lookahead row proves an older
continuation, and bind it exactly to the last returned identity.
- Do not call cursor traversal a snapshot, chronology proof, delivery receipt,
completeness proof, or tamper evidence. Identity sequences can contain gaps
and commit order can differ from identity allocation order.
- Leave transaction isolation under caller ownership. A host requiring one
export snapshot must establish `REPEATABLE READ` or stronger before its first
query and reuse one caller-owned transaction with the in-transaction method.
- Keep external immutable/WORM retention, exporter credentials, manifests,
receipts, and reconciliation outside the package. Pagination adds no network
client or database write authority.
- Maintain deterministic tests for cursor validation, immutable page shape,
bounded lookahead, SQL keyset semantics, key/order revalidation, malformed
adapter output, owned-read commit behavior, and 100% production
statement/branch/public-docstring coverage.
121 changes: 121 additions & 0 deletions docs/adr/0010-bounded-checkpoint-audit-export-pagination.md
Original file line number Diff line number Diff line change
@@ -0,0 +1,121 @@
# ADR 0010: Bounded checkpoint-audit export pagination

- **Status:** Accepted for the stacked implementation
- **Date:** 2026-08-07
- **Decision owners:** ContextualWisdomLab
- **Depends on:** ADR 0009 and the append-only checkpoint audit trail

## Context

The package-owned audit store previously exposed only one newest-first bounded
read. That is appropriate for an operator console, but not sufficient for a
host that must move a longer retained audit history to separately governed
storage. Leaving multi-page traversal to every embedding product would produce
incompatible cursor semantics and would encourage `OFFSET` pagination, which can
skip or duplicate rows when the visible set changes between requests.

The existing audit identity is a PostgreSQL `BIGINT GENERATED ALWAYS AS
IDENTITY` primary key. The compound checkpoint-key index already ends in
`checkpoint_audit_event_id DESC`, so the table can support bounded keyset
pagination without a schema migration or a second ordering column.

## Decision

Add an opt-in `CheckpointAuditPage` contract and two audit-store methods:

- `list_audit_event_page()` for package-owned page reads; and
- `list_audit_event_page_in_transaction()` for hosts that own the surrounding
PostgreSQL transaction.

Each request validates a strict 1..1,000 page size and an optional positive
signed-`BIGINT` `before_audit_event_id`. The first page orders by
`checkpoint_audit_event_id DESC`. Continuations add the strict predicate
`checkpoint_audit_event_id < before_audit_event_id` and preserve the same order.
The SQL query requests only `limit + 1` rows. At most `limit` rows are returned;
the lookahead row only proves that an older continuation exists. When a
continuation exists, `next_before_audit_event_id` is exactly the identity of the
last returned event.

Every returned row is revalidated as a `CheckpointAuditEvent`, rechecked against
the trusted tenant/consumer/endpoint/batch key, and required to be strictly
descending. A database adapter that returns a non-sequence or more than the
bounded query size fails closed.

## Concurrency boundary

Keyset traversal solves offset drift, not snapshot isolation. Rows committed
after page one with identities greater than the cursor cannot move or duplicate
older rows already traversed. However, package-owned calls execute as separate
transactions and therefore do not promise one historic database snapshot.

A host that requires one snapshot for an export pass must own a PostgreSQL
`REPEATABLE READ` or stricter transaction and call the in-transaction method on
one cursor. The package does not silently alter transaction isolation because
that would change caller-owned database semantics.

Identity allocation order is not a cryptographic chronology and may contain
gaps. A transaction can allocate an identity before another transaction and
commit later. The cursor is therefore a database navigation key, not a statement
that all real-world events before or after a wall-clock instant have been
captured.

## Security and acquisition boundary

The page remains tenant-qualified and inherits forced RLS and ordinary-role
append-only controls from ADR 0009. It does not create a new database object,
credential, network destination, export worker, background process, or write
permission. It never accepts provider or model output as tenant or consumer
identity.

The feature is intended to make bounded export to separately governed immutable
or write-once retention storage practical. It does not itself provide immutable
external storage, cryptographic non-repudiation, signed completeness evidence,
administrator-proof tamper detection, or delivery acknowledgement. Those remain
host/operator controls. A later cryptographically protected export manifest can
be layered on this stable pagination contract without changing database
navigation semantics.

## Alternatives rejected

### SQL OFFSET/LIMIT

Rejected because additions or visibility changes before an offset can shift the
subsequent window, producing duplicates or omissions during a long-running
export.

### Unbounded iterator or fetch-all export

Rejected because acquisition-grade audit volume is not bounded by a single
operator interaction. Materializing arbitrary retained history would weaken the
package's existing memory-safety policy.

### Implicit REPEATABLE READ in the package-owned API

Rejected because changing isolation belongs to the owner of the transaction and
must occur before the first transaction query. The in-transaction API gives a
host an explicit way to obtain one snapshot without the package surprising
other database work.

### Timestamp cursor

Rejected because timestamps are not unique and database wall-clock values are
not a total order. The existing primary-key identity is already indexed and
provides an unambiguous strict continuation boundary.

## Verification

Permanent deterministic tests require strict cursor validation, immutable page
shape, strictly descending identities, one-row lookahead, no `OFFSET`, exact
`<` keyset continuation, trusted-key row revalidation, bounded driver output,
and no package commit during owned page reads. Integration/release gates remain
required after this stack is reconciled onto protected `main`.

## Consequences

Operators gain a package-owned bounded traversal primitive suitable for durable
audit export and acquisition diligence. Existing `list_audit_events()` behavior
is unchanged. No migration, release version, or publication authority is added.

The operational tradeoff is explicit: callers choosing independent page calls
accept normal transaction-to-transaction visibility changes; callers requiring a
single snapshot must own and configure that transaction deliberately.
59 changes: 53 additions & 6 deletions docs/checkpoint-audit.md
Original file line number Diff line number Diff line change
Expand Up @@ -89,6 +89,50 @@ action, a database-generated event identity, and an insert-time database
wall-clock timestamp. It does not include provider bodies, prompts, model output,
credentials, DSNs, transport headers, or exception text.

## Export more than one bounded page

Use keyset pagination rather than `OFFSET` when an operator needs to walk older
audit evidence for export to separately governed retention storage:

```python
before = None
while True:
page = store.list_audit_event_page(
"invoice-worker",
"batch-123",
"default",
before_audit_event_id=before,
limit=500,
)
export_to_governed_storage(page.events)
before = page.next_before_audit_event_id
if before is None:
break
```

`before_audit_event_id` is either `None` or a strict positive PostgreSQL `BIGINT`
identity. The query requests at most `limit + 1` rows; the extra row is used only
to determine whether a continuation exists. At most `limit` events are exposed.
The next cursor, when present, is exactly the final returned event identity, and
the following query uses `checkpoint_audit_event_id < before_audit_event_id`.
Rows inserted later with larger identities therefore cannot shift rows already
walked or create the duplicate/skip behavior associated with offset pagination.

This is **not** a multi-page database snapshot. Package-owned calls may observe
changes committed between page requests. A caller that requires all pages from
one PostgreSQL snapshot should start a caller-owned `REPEATABLE READ` or stricter
transaction and repeatedly call `list_audit_event_page_in_transaction()` on that
same cursor. PostgreSQL documents that `REPEATABLE READ` keeps statements in the
transaction on the snapshot established by its first query. The package does not
silently change caller transaction isolation.

The cursor is a navigation boundary, not evidence of completeness, authenticity,
retention, or non-repudiation. Sequence identities can contain gaps and commit
order need not equal identity allocation order. An export consumer must persist
its own export receipt or manifest if it needs evidence that a particular export
set was delivered to immutable storage. A later row that was not visible during
an earlier read belongs to a later export/reconciliation pass.

## Retention and rollback

The database blocks ordinary `UPDATE`, `DELETE`, and `TRUNCATE` against the audit
Expand All @@ -101,12 +145,15 @@ These controls are not cryptographic non-repudiation and do not protect against
PostgreSQL owner, superuser, `BYPASSRLS` role, disabled triggers, or physical
storage administrator. Deployments that require administrator-tamper detection
should replicate or export events to separately governed immutable storage or a
signed/hash-chained evidence system.
signed/hash-chained evidence system. Stable bounded pagination makes that export
operationally tractable but does not itself strengthen the database tamper model.

## Standards and evidence

See [ADR 0009](adr/0009-append-only-checkpoint-audit-trail.md) for the decision
boundary and
[the assurance record](doctoring/checkpoint-audit-trail.md) for threat model,
verification evidence, and APA 7 references to NIST SP 800-53 Rev. 5 AU-3, the
OWASP Logging Cheat Sheet, and PostgreSQL 18 trigger and current-time semantics.
See [ADR 0009](adr/0009-append-only-checkpoint-audit-trail.md) for the base audit
decision, [ADR 0010](adr/0010-bounded-checkpoint-audit-export-pagination.md) for
the export pagination boundary, and
[the export assurance record](doctoring/checkpoint-audit-export-pagination.md)
for failure modes, concurrency semantics, verification evidence, and APA 7
references to NIST SP 800-53 Rev. 5 AU-9 and PostgreSQL 18 transaction-isolation
documentation.
Loading