supply: replay deterministic File change pages from opaque cursors

docs/decisions/0054-acknowledge-file-change-pages-before-cursor-advance.md

@@ -0,0 +1,117 @@
+---
+name: adr-0054-acknowledge-file-change-pages-before-cursor-advance
+version: "1.0.0"
+description: >
+  Activate bounded File readChanges while keeping provider proposals distinct
+  from Control-accepted acquisition checkpoints.
+---
+
+# 0054. Acknowledge File change pages before cursor advance
+
+- Status: accepted
+- Date: 2026-07-25
+- Refines: ADR-0012, ADR-0043, ADR-0044
+
+## Context
+
+Issue #81 activates the first File `readChanges` carrier. A shallow filesystem
+scan can deterministically propose an ordered page, but observing files does not
+prove that ContextControl durably accepted every envelope. Returning the same
+cursor type before and after commit would let callers continue a scan while the
+acquisition checkpoint still points at the previous accepted page.
+
+The provider page also has no delivery audience. Creating publishable import
+jobs during page acceptance would invent FileSourceAccess authority and conflate
+acquisition progress with Runtime visibility.
+
+## Decision
+
+The active File capability advances from immutable v2 to immutable v3 only
+through ContextControl. V3 activates bounded `describeCapabilities`,
+`readChanges`, cursor semantics, checkpoint semantics, and the provider
+checkpoint operation. `discover`, `authorizeAndProject`, deletion execution,
+freshness, and consistency guarantees remain unavailable.
+
+`InitialScan` observes only shallow regular Markdown files under the anchored
+logical root. It orders names by UTF-8 bytes, does not follow symlinks, and emits
+content-free change envelopes containing canonical path, content digest, byte
+length, change kind, Organization, Source, SourceVersion, and scan binding.
+Before returning success, the Provider re-lists the anchored root and
+revalidates every observed file identity; a changed directory membership or
+file identity closes as retryable-unavailable rather than signing a mixed
+snapshot. The database path constraint accepts the exact shallow Markdown
+filename domain owned by `FileImportPath`, including the minimal `.md` name.
+
+The provider signs the canonical whole page and returns a
+`PendingChangeCursor`. ContextControl verifies that page proof, then a narrow
+SECURITY DEFINER transaction requires the same active v3 SourceVersion, exact
+predecessor page, and complete bounded change array. That transaction appends
+the immutable page, every immutable change envelope, and one entry in the
+existing acquisition-checkpoint sequence. Exact replay returns the same receipt;
+out-of-order, changed, foreign, disabled, or stale acceptance returns nothing.
+
+Only after the transaction commits does the PostgreSQL Control adapter wrap the
+pending provider cursor in a separately authenticated `ChangeCursor` bound to
+the accepted page digest, checkpoint reference, and global acquisition sequence.
+Provider pages and Control checkpoints use opposite Ed25519 key directions: each
+consumer holds only the other boundary's public verification key, never its
+private signing authority. The provider accepts only this post-commit cursor
+form for continuation. Keys are injected composition contracts; page and cursor
+values remain opaque and redacted from representations.
+
+Every initial source state receives an opaque scan epoch derived from the
+canonical state plus the current Control-read durable scan head. It is stable
+across Provider restarts while bytes and durable head remain unchanged, but
+advances whenever the observed state changes relative to that head. Every
+continuation page carries that epoch plus its preceding accepted page,
+checkpoint, and sequence. A new epoch also carries the immediately superseded
+epoch so Control can atomically replace the current scan; it cannot name any
+older epoch. Control exposes this head only through the trusted Source progress
+read model, and the caller must recompose `FileChangeSource` from that value
+before Provider I/O. The acceptance transaction compares these bindings with
+the latest accepted File change page for the Source, not merely the latest page
+within one content-derived scan. Re-presenting the same durable state is exact
+replay rather than a new checkpoint; a stale initial page cannot regress a
+newer completed scan. This globally monotonic rule remains crash-safe across
+Provider restart and rejects stale continuation cursors, including ABA source
+state reversion.
+
+Provider-page checkpoints carry no publish outcome and therefore do not create
+gaps in the publication-bearing watermark prefix. Runtime has no privileges on
+page, change, or checkpoint tables and does not consult them as authorization.
+Page acceptance creates no acquisition job, WorkerLease, Revision, candidate,
+Resource policy, or publish watermark. Later scheduling must establish an
+existing File import audience and WorkerLease lineage explicitly.
+
+## Rationale
+
+Acknowledging a signed whole page before issuing its continuation cursor makes
+the durable checkpoint—not Provider process memory—the source of truth. A full
+shallow rescan on each request deliberately preserves restart-safe snapshot
+validation; caching or incremental watching would introduce new invalidation
+and recovery semantics that this fixed point has not proven. Separate signing
+directions and a private cursor-payload encoder also keep page proposal and
+checkpoint minting authority from becoming a shared public API.
+
+## Consequences
+
+- A filesystem observation cannot advance the durable cursor by itself.
+- Provider verification cannot mint a Control checkpoint, and Control
+  verification cannot mint a Provider page.
+- An old scan cannot resume after a newer scan becomes current, including ABA
+  source-state reversion.
+- Whole-page replay is idempotent, while partial insertion cannot commit.
+- Cross-Organization, SourceVersion change, and source disable fail closed.
+- Manual v3 File imports remain available and keep their existing audience/job
+  path; provider changes do not inherit that audience implicitly.
+- Recursive scanning, watching, deletion execution, automatic scheduling,
+  retries, dead-letter handling, and full resync remain inactive.
+- Downgrade is refused after any accepted provider page because deleting or
+  renumbering acquisition history would violate opaque monotonic checkpoints.
+
+## Revisit trigger
+
+Revisit before automatic scheduling, deletion execution, checkpoint compaction,
+key rotation, recursive discovery, or full resync. Any revision must preserve
+post-commit cursor issuance, exact tenant/source/version lineage, whole-page
+atomicity, replay idempotency, and Runtime independence.

docs/decisions/README.md

@@ -45,6 +45,7 @@ kernel, capability separation, and publication visibility model.
 | File Resource deletion | [0042 — Tombstone before cleanup](0042-tombstone-file-resources-before-cleanup.md) | One trusted Control transaction tombstones the active File Resource, advances its Organization Policy Epoch, and records immutable pending cleanup lineage before any physical deletion | Cleanup-defined visibility, caller-authored tenant/epoch/cleanup identity, index deletion as authorization, restore, or native watcher claims |
 | File Source progress | [0043 — Separate acquisition and publication progress](0043-separate-file-acquisition-progress-from-publication-progress.md) | Append accepted changes separately from contiguous Runtime-visibility completion and expose them through an Organization/Source-scoped Control read | One ambiguous checkpoint, skipped publication gaps, Runtime authorization from watermarks, or false standard ProviderPort capability claims |
 | File Source offboarding | [0044 — Disable before cleanup](0044-disable-file-sources-before-cleanup.md) | One trusted Control transaction terminally disables the Source, advances its Organization Policy Epoch, cancels outstanding work, and records immutable pending cleanup lineage | Cleanup-defined revocation, bulk Resource deletion, application-only lifecycle checks, post-disable leases/tickets, or treating progress as authority |
+| File change-page acknowledgement | [0054 — Acknowledge File change pages before cursor advance](0054-acknowledge-file-change-pages-before-cursor-advance.md) | Provider pages remain pending until one whole-page Control transaction appends the page, changes, and existing acquisition checkpoint, after which Control issues the continuation cursor | Provider-observation cursor advance, partial page acceptance, implicit publication audience, a second checkpoint protocol, or Runtime authority from Supply progress |
 | Private delivery ingress | [0045 — Redeem private delivery evidence at ingress](0045-redeem-private-delivery-evidence-at-ingress.md) | One digest-only service/request/asker/audience/epoch-bound DeliveryEvidenceRef constructs private TrustedDeliveryContext inside the current UserActor transaction before content work | Raw trusted delivery facts on the wire, bearer persistence, application-role minting/table reads, alternate Runtime paths, or claiming later M2 carriers |
 | Exact Package egress | [0046 — Bind egress to one exact Package hop](0046-bind-egress-to-one-exact-package-hop.md) | One digest-only grant binds one exact audience-bound Package to one model or channel preflight hop and redeems atomically | Treating Package construction as disclosure authority, arbitrary content at egress, cross-hop reuse, or bypassing final policy |
 | Public OpenAPI v0 | [0047 — Freeze OpenAPI v0 through one Runtime path](0047-freeze-openapi-v0-through-one-runtime-path.md) | One public `/v0/resolve` schema and a hidden provisional v1 bridge share the same sealed Runtime; Package release lineage is read-only from the Learning-published active manifest | Two authorization compositions, caller-authored release facts, Runtime publication/fallback, or in-place mutation of historical snapshots |
@@ -148,3 +149,4 @@ touched:
 - [0044 — Disable File sources before cleanup](0044-disable-file-sources-before-cleanup.md)
 - [0045 — Redeem private delivery evidence at ingress](0045-redeem-private-delivery-evidence-at-ingress.md)
 - [0046 — Bind egress to one exact Package hop](0046-bind-egress-to-one-exact-package-hop.md)
+- [0054 — Acknowledge File change pages before cursor advance](0054-acknowledge-file-change-pages-before-cursor-advance.md)

engine/_opaque.py

@@ -0,0 +1,31 @@
+"""Canonical codecs shared by opaque internal protocol values."""
+
+from __future__ import annotations
+
+import base64
+import binascii
+
+
+def encode_base64url(value: bytes) -> str:
+    """Encode canonical unpadded base64url."""
+
+    return base64.urlsafe_b64encode(value).rstrip(b"=").decode("ascii")
+
+
+def decode_base64url(value: str) -> bytes:
+    """Decode only canonical unpadded base64url."""
+
+    alphabet = "ABCDEFGHIJKLMNOPQRSTUVWXYZabcdefghijklmnopqrstuvwxyz0123456789-_"
+    if not value or any(character not in alphabet for character in value):
+        raise ValueError("invalid base64url")
+    try:
+        decoded = base64.b64decode(
+            value + "=" * (-len(value) % 4),
+            altchars=b"-_",
+            validate=True,
+        )
+    except (binascii.Error, ValueError):
+        raise ValueError("invalid base64url") from None
+    if encode_base64url(decoded) != value:
+        raise ValueError("noncanonical base64url")
+    return decoded

engine/control/__init__.py

@@ -11,7 +11,9 @@
 )
 from engine.control.contracts import (
     FILE_CAPABILITY_MANIFEST,
+    FILE_CHANGE_CAPABILITY_MANIFEST,
     FILE_IMPORT_CAPABILITY_MANIFEST,
+    ActivateFileChangeFeed,
     CapabilityStatus,
     FileCapabilityManifest,
     FileRootRef,
@@ -27,6 +29,28 @@
     SourceResourceKind,
     SourceVersion,
 )
+from engine.control.file_change_pages import (
+    MAX_FILE_CHANGE_PAGE_SIZE,
+    AcceptedChangePage,
+    ChangeCursor,
+    ChangeLimit,
+    ChangePage,
+    FileChangeControlProofs,
+    FileChangeKind,
+    FileChangeProviderOutcome,
+    FileChangeProviderProofs,
+    FileChangeScanHead,
+    FileChangeSource,
+    InitialScan,
+    PendingChangeCursor,
+    ProviderGenericDenied,
+    ProviderInvalidCheckpoint,
+    ProviderOk,
+    ProviderRetryableUnavailable,
+    ProviderUnsupported,
+    SourceChange,
+    VerifiedChangePage,
+)
 from engine.control.file_deletions import (
     FileResourceTombstone,
     TombstoneFileResource,
@@ -53,9 +77,16 @@
 from engine.control.module import ContextControl, ControlStorePort
 
 __all__ = [
+    "MAX_FILE_CHANGE_PAGE_SIZE",
+    "AcceptedChangePage",
+    "ActivateFileChangeFeed",
     "FILE_CAPABILITY_MANIFEST",
+    "FILE_CHANGE_CAPABILITY_MANIFEST",
     "FILE_IMPORT_CAPABILITY_MANIFEST",
     "CapabilityStatus",
+    "ChangeCursor",
+    "ChangeLimit",
+    "ChangePage",
     "ContextControl",
     "ControlOperation",
     "ControlOperatorAuthenticationRejected",
@@ -64,10 +95,17 @@
     "ControlOperatorAuthorityUnavailable",
     "ControlStorePort",
     "FileCapabilityManifest",
+    "FileChangeControlProofs",
+    "FileChangeKind",
+    "FileChangeProviderOutcome",
+    "FileChangeProviderProofs",
+    "FileChangeScanHead",
+    "FileChangeSource",
     "FileImportAudience",
     "FileImportPath",
     "FileImportReceiver",
     "FileRootRef",
+    "InitialScan",
     "FileSourceAcquisitionCheckpoint",
     "FileSourceChangeKind",
     "FileSourceProgress",
@@ -78,10 +116,17 @@
     "FileResourceTombstone",
     "RegisterFileSource",
     "OffboardFileSource",
+    "PendingChangeCursor",
     "PrepareFileImport",
     "PreparedFileImport",
+    "ProviderGenericDenied",
+    "ProviderInvalidCheckpoint",
+    "ProviderOk",
+    "ProviderRetryableUnavailable",
+    "ProviderUnsupported",
     "SourceAclEvidenceMode",
     "SourceControlUnavailable",
+    "SourceChange",
     "SourceContentKind",
     "SourceKind",
     "SourceManifest",
@@ -93,4 +138,5 @@
     "TrustedControlCall",
     "TombstoneFileResource",
     "VerifiedControlOperatorIdentity",
+    "VerifiedChangePage",
 ]

engine/control/authority.py

@@ -19,6 +19,8 @@
 
 
 class ControlOperation(StrEnum):
+    ACCEPT_FILE_CHANGE_PAGE = "accept_file_change_page"
+    ACTIVATE_FILE_CHANGE_FEED = "activate_file_change_feed"
     IMPORT_FILE = "import_file"
     OFFBOARD_FILE_SOURCE = "offboard_file_source"
     REGISTER_SOURCE = "register_source"

engine/control/contracts.py

@@ -98,17 +98,11 @@ class FileCapabilityManifest:
     ingestion_jobs: CapabilityStatus = CapabilityStatus.UNAVAILABLE
 
     def __post_init__(self) -> None:
-        unavailable_statuses = (
-            self.cursor_semantics,
-            self.checkpoint_semantics,
-            self.batch_limits,
+        always_unavailable_statuses = (
             self.freshness,
             self.consistency_guarantees,
-            self.describe_capabilities,
-            self.read_changes,
             self.discover,
             self.authorize_and_project,
-            self.checkpoint,
             self.deletion,
         )
         if (
@@ -119,7 +113,7 @@ def __post_init__(self) -> None:
             or self.projection_fields != ()
             or any(
                 status is not CapabilityStatus.UNAVAILABLE
-                for status in unavailable_statuses
+                for status in always_unavailable_statuses
             )
             or (
                 self.declaration_version == "file-capabilities-v1"
@@ -135,8 +129,45 @@ def __post_init__(self) -> None:
                     or self.ingestion_jobs is not CapabilityStatus.AVAILABLE
                 )
             )
+            or (
+                self.declaration_version == "file-capabilities-v3"
+                and (
+                    any(
+                        status is not CapabilityStatus.AVAILABLE
+                        for status in (
+                            self.cursor_semantics,
+                            self.checkpoint_semantics,
+                            self.batch_limits,
+                            self.describe_capabilities,
+                            self.read_changes,
+                            self.checkpoint,
+                            self.file_source_access,
+                            self.ingestion_jobs,
+                        )
+                    )
+                )
+            )
+            or (
+                self.declaration_version
+                in {"file-capabilities-v1", "file-capabilities-v2"}
+                and any(
+                    status is not CapabilityStatus.UNAVAILABLE
+                    for status in (
+                        self.cursor_semantics,
+                        self.checkpoint_semantics,
+                        self.batch_limits,
+                        self.describe_capabilities,
+                        self.read_changes,
+                        self.checkpoint,
+                    )
+                )
+            )
             or self.declaration_version
-            not in {"file-capabilities-v1", "file-capabilities-v2"}
+            not in {
+                "file-capabilities-v1",
+                "file-capabilities-v2",
+                "file-capabilities-v3",
+            }
         ):
             raise ValueError("File capability manifest is not a recognized snapshot")
 
@@ -172,6 +203,17 @@ def document(self) -> dict[str, object]:
     file_source_access=CapabilityStatus.AVAILABLE,
     ingestion_jobs=CapabilityStatus.AVAILABLE,
 )
+FILE_CHANGE_CAPABILITY_MANIFEST = FileCapabilityManifest(
+    declaration_version="file-capabilities-v3",
+    cursor_semantics=CapabilityStatus.AVAILABLE,
+    checkpoint_semantics=CapabilityStatus.AVAILABLE,
+    batch_limits=CapabilityStatus.AVAILABLE,
+    describe_capabilities=CapabilityStatus.AVAILABLE,
+    read_changes=CapabilityStatus.AVAILABLE,
+    checkpoint=CapabilityStatus.AVAILABLE,
+    file_source_access=CapabilityStatus.AVAILABLE,
+    ingestion_jobs=CapabilityStatus.AVAILABLE,
+)
 
 
 @dataclass(frozen=True, slots=True)
@@ -226,6 +268,20 @@ def __post_init__(self) -> None:
             raise TypeError("SourceRef value must be UUID")
 
 
+@dataclass(frozen=True, slots=True)
+class ActivateFileChangeFeed:
+    """Trusted Control request that carries only one source locator."""
+
+    source_ref: SourceRef
+
+    def __post_init__(self) -> None:
+        if type(self.source_ref) is not SourceRef:
+            raise TypeError("File change feed source_ref must be SourceRef")
+
+    def __reduce__(self) -> NoReturn:
+        raise TypeError("File change feed activation is not serializable")
+
+
 @dataclass(frozen=True, slots=True)
 class SourceVersion:
     """Immutable active source-configuration snapshot returned by Control."""