feat(observability): instrument durable checkpoint operations

AGENTS.md

@@ -174,3 +174,28 @@ add CODEOWNERS-based merge gates until multiple independent maintainers exist.
 - Maintain 100% production statement, branch, and public-docstring coverage with
   deterministic unit, concurrency, migration, rollback, integration, and
   documentation tests.
+
+## OpenTelemetry checkpoint signals
+
+- Keep `OpenTelemetryCheckpointStore` opt-in and dependency-injected. The package
+  must not configure a global tracer provider, meter provider, sampler,
+  processor, exporter, collector, or host resource.
+- Emit only fixed operation names and finite low-cardinality `error.type`
+  classifications. Use `record_exception=False` and
+  `set_status_on_exception=False` because durable checkpoint exceptions may
+  retain protected structured details.
+- Failed checkpoint spans must explicitly set the host OpenTelemetry API's
+  `StatusCode.ERROR` without a description when that optional API is available;
+  successful spans must leave status Unset. Status resolution or mutation is
+  best-effort observer work and must never alter application behavior.
+- Never add tenant, consumer, batch, endpoint, file, digest, cursor, DSN,
+  provider payload, exception message, or dynamic exception-class values to
+  package-owned spans or metrics.
+- Treat tracing, metric, status, optional status-code resolution, and clock
+  failures as best-effort observer failures. They must never alter checkpoint
+  return values, exception identity, compare-and-swap behavior, transaction
+  ownership, commit, or rollback.
+- Keep counter and seconds-based duration histogram behavior deterministic and
+  maintain 100% production statement, branch, and public-docstring coverage for
+  success, conflict, validation, internal-error, Error-status, Unset-status,
+  caller-transaction, confidentiality, and observer-failure paths.

ARCHITECTURE.md

@@ -216,6 +216,48 @@ descriptive snake_case, RLS is enabled and forced, and the rollback refuses to
 drop a non-empty table. The stored digest remains prefix evidence only; durable
 storage does not add provider authentication or full-stream immutability.
 
+## Checkpoint OpenTelemetry observability boundary
+
+`OpenTelemetryCheckpointStore` is an opt-in wrapper around a durable checkpoint
+store. It receives a host-owned tracer and meter through dependency injection and
+does not configure global providers, processors, samplers, exporters,
+collectors, or resources. The wrapped store remains independently usable without
+OpenTelemetry and may be the package-owned PostgreSQL store or a compatible
+host-owned implementation.
+
+Package-owned spans and metrics use only fixed operation, transaction-owner,
+outcome, and finite error-classification attributes. They never contain tenant,
+consumer, batch, endpoint, file, digest, cursor, and DSN values, provider
+payloads, exception messages, or dynamic exception class names. The package
+operation span is deliberately storage-agnostic and does not emit
+`db.system.name` or claim OpenTelemetry database-client semantics. Actual
+database-client spans and database-system attributes belong to the host or
+database instrumentation at the client boundary where the storage technology is
+known.
+
+Automatic exception recording and status-on-exception are disabled because a
+checkpoint exception may retain protected structured state even when its public
+message is bounded. Instead, failed checkpoint spans explicitly set the host
+OpenTelemetry API's `StatusCode.ERROR` without a description when the optional
+API is available, while successful checkpoint spans leave status Unset. This
+preserves standard failure discoverability without exposing exception text in
+status descriptions.
+
+The operation counter records completed loads and saves. The duration histogram
+uses seconds and a monotonic clock, clamping backward or unavailable clock
+evidence to zero. Failures use only `checkpoint_conflict`, `validation_error`, or
+`internal_error`; success omits `error.type`. Tracer, meter, span, export-surface,
+optional status-code resolution, status mutation, and clock failures are
+contained as observer failures. The exact checkpoint return value or application
+exception remains authoritative, so best-effort telemetry cannot change
+checkpoint operation semantics, compare-and-swap, transaction ownership, commit,
+or rollback.
+
+Caller-owned transaction spans cover the package call only and do not claim that
+the surrounding transaction later committed. The host owns telemetry retention,
+access control, alerting, collector availability, database-client
+instrumentation, and any correlation outside this confidential package boundary.
+
 ## Modular interoperability
 
 CWL hosts such as `contextual-orchestrator` and `naruon` supply tenant context
@@ -271,6 +313,11 @@ context-managed early close, and SHA-256 framing sensitivity. Durable-store test
 cover strict consumer identity, caller-owned transaction behavior, idempotent
 repeat, exact compare-and-swap, stale and regressive writers, equal and unequal
 first-writer races, disappearing conflict rows, forced-RLS migration text,
-fail-closed rollback, documentation, and live PostgreSQL persistence. Final merge
-evidence must be regenerated against the integrated base; successful stacked-base
-runs are not reusable release evidence.
+fail-closed rollback, documentation, and live PostgreSQL persistence. Checkpoint
+telemetry tests additionally prove exact delegation, fixed low-cardinality signal
+attributes, storage-agnostic operation spans, seconds-based nonnegative duration,
+confidential failure classification, explicit Error status without descriptions,
+Unset success status, disabled exception recording, and preservation of
+application results and exception identity during ordinary tracer, meter, span,
+status, and clock failures. Final merge evidence must be regenerated against the
+integrated base; successful stacked-base runs are not reusable release evidence.

CHANGELOG.md

@@ -9,6 +9,20 @@ and this project adheres to [Semantic Versioning](https://semver.org/spec/v2.0.0
 
 ### Added
 
+- Optional OpenTelemetry-compatible checkpoint spans and metrics through
+  `OpenTelemetryCheckpointStore`, with dependency-injected tracer and meter,
+  fixed low-cardinality operation and transaction-owner labels, a seconds-based
+  monotonic duration histogram, and the finite failure vocabulary
+  `checkpoint_conflict, validation_error, and internal_error`. Package-owned
+  signals omit tenant, consumer, batch, endpoint, file, digest, cursor, DSN,
+  exception-message, and provider-payload data; automatic exception recording
+  and automatic status-on-exception are disabled. Failed checkpoint spans set
+  the host OpenTelemetry API's `StatusCode.ERROR` without a description when
+  available, while successful checkpoint spans leave status Unset. Optional
+  status resolution, explicit status mutation, ordinary telemetry, and clock
+  failures cannot alter checkpoint results, exception identity,
+  compare-and-swap, commit, rollback, or caller-owned transaction behavior. No
+  SDK/exporter dependency, migration, version bump, or release is included.
 - Optional durable result-checkpoint store through
   `PostgresBatchResultCheckpointStore` and
   `llm_result_stream_checkpoints`, with tenant-qualified consumer identity,

CLAUDE.md

@@ -158,3 +158,25 @@
   while any acknowledgement evidence remains.
 - Maintain deterministic concurrency, migration, rollback, live-PostgreSQL,
   documentation, and 100% production coverage tests.
+
+## OpenTelemetry checkpoint signals
+
+- Keep `OpenTelemetryCheckpointStore` optional, dependency-injected, and free of
+  package-owned global OpenTelemetry configuration.
+- Emit fixed operation, transaction-owner, and outcome labels only. Use a finite
+  low-cardinality error vocabulary and disable automatic exception recording.
+- Use `record_exception=False` and `set_status_on_exception=False` for every
+  package-owned checkpoint span. On failure, explicitly set the host
+  OpenTelemetry API's `StatusCode.ERROR` without a description when available;
+  on success, leave status Unset.
+- Never add tenant, consumer, batch, endpoint, file, digest, cursor, or DSN values
+  to package-owned spans or metrics. Do not add exception messages, dynamic
+  exception class names, provider payloads, or database errors either.
+- The original checkpoint result or exact application exception is authoritative;
+  tracer, metric, clock, optional status-code resolution, and status-mutation
+  failures must not mask or replace application results or exceptions.
+- Preserve package-owned versus caller-owned transaction labels without changing
+  commit, rollback, compare-and-swap, or cursor ownership.
+- Maintain 100% production statement, branch, and public-docstring coverage for
+  success, conflict, validation, internal-error, duration, Error-status,
+  Unset-status, confidentiality, delegation, and telemetry-failure paths.

docs/adr/0008-checkpoint-opentelemetry-observability.md

@@ -0,0 +1,36 @@
+# ADR 0008: Checkpoint OpenTelemetry observability
+
+- Status: Accepted
+- Date: 2026-08-07
+
+## Context
+
+The durable result-checkpoint store provides tenant isolation, exact compare-and-swap advancement, and caller-owned transaction methods. Operators nevertheless need standard latency, outcome, and conflict signals. Logging checkpoint identifiers or exception bodies would create a confidentiality and cardinality risk, while requiring an OpenTelemetry SDK or exporter would compromise standalone operation and could let observer failure alter commit or rollback behavior.
+
+The wrapper may also decorate a compatible host-owned checkpoint store. A package-level checkpoint operation span therefore cannot truthfully assert a database technology or database-client semantic convention: the wrapped implementation may use PostgreSQL, another persistence technology, or no database at all.
+
+## Decision
+
+Add the opt-in `OpenTelemetryCheckpointStore` wrapper. It accepts a dependency-injected OpenTelemetry-compatible tracer and meter and delegates the four existing checkpoint operations without changing their arguments, return values, exception identity, transaction ownership, or storage semantics.
+
+Package-owned spans and metrics never contain resource identifiers. In particular, tenant, consumer, batch, endpoint, file, digest, cursor, and DSN values are excluded. Span names, metric names, operation labels, transaction-owner labels, outcomes, and `error.type` values are fixed finite vocabularies. Automatic exception recording and automatic exception status are disabled because checkpoint exceptions can retain sensitive structured details.
+
+The counter records completed operations. The histogram unit is seconds and records a nonnegative monotonic duration. Success omits `error.type`; failures use only `checkpoint_conflict`, `validation_error`, or `internal_error`. To follow the OpenTelemetry recording-errors contract without exposing exception text, failed checkpoint spans explicitly set OpenTelemetry status Error without a description, while successful checkpoint spans leave status Unset. The package resolves the host OpenTelemetry API's `StatusCode.ERROR` only when that optional API is available; missing or failing observer support never becomes an application dependency.
+
+Package operation spans are storage-agnostic: they do not emit `db.system.name` or claim to be OpenTelemetry database-client spans. When a host needs PostgreSQL client telemetry, the host or database instrumentation layer owns the corresponding database semantic-convention attributes and spans.
+
+Instrumentation is best effort. Tracer start, span entry, attribute mutation, explicit status mutation, span exit, metric creation, metric recording, optional status-code resolution, and clock failures are contained. The original checkpoint result or exact application exception remains authoritative. Exporter, processor, sampler, and provider ownership remains with the host. The package neither configures global OpenTelemetry state nor adds an SDK/exporter dependency.
+
+## Consequences
+
+Operators gain interoperable, low-cardinality checkpoint operation signals without exposing durable identities or provider-controlled data. Hosts may attach their own resource attributes outside the package boundary, but must not reinterpret package omission as authorization to add tenant or remote identifiers to high-cardinality telemetry.
+
+Failed spans are discoverable through standard OpenTelemetry status in addition to the bounded `error.type`, without placing exception messages or status descriptions into package telemetry. Successful operations preserve the default Unset status rather than forcing `Ok`.
+
+Storage-agnostic operation spans remain truthful when the wrapper decorates a compatible host-owned store. PostgreSQL-specific client instrumentation remains independently composable and can coexist with the package operation span without the package fabricating database semantics.
+
+Telemetry cannot prove database commit, replication, or downstream business-effect durability beyond the wrapped method boundary. Caller-owned transaction methods measure the package call only; the caller remains responsible for the surrounding transaction outcome.
+
+## Rollback
+
+Remove the wrapper from host composition and continue using `PostgresBatchResultCheckpointStore` or the compatible host-owned store directly. No database migration, stored state, public checkpoint schema, dependency, or release version needs rollback. Removing instrumentation must not change durable checkpoint data or transaction behavior.

docs/checkpoint-observability.md

@@ -0,0 +1,114 @@
+# Durable checkpoint observability
+
+`OpenTelemetryCheckpointStore` is an optional wrapper for
+`PostgresBatchResultCheckpointStore` or a compatible host-owned checkpoint
+store. It emits one span, one completed-operation count, and one duration
+measurement for each public load or save call. It does not install or configure
+OpenTelemetry and does not assume what storage technology a compatible store
+uses.
+
+## Host setup
+
+Install and configure the OpenTelemetry API, SDK, processors, and exporters in
+the embedding service. Then pass the host-owned tracer and meter explicitly:
+
+```python
+from opentelemetry import metrics, trace
+
+from pg_llm_batch import (
+    OpenTelemetryCheckpointStore,
+    PostgresBatchResultCheckpointStore,
+)
+
+checkpoint_store = PostgresBatchResultCheckpointStore(
+    "postgresql://application-role@database/operations",
+    tenant_scope="tenant-a",
+)
+observed_checkpoints = OpenTelemetryCheckpointStore(
+    checkpoint_store,
+    tracer=trace.get_tracer("buyer.application"),
+    meter=metrics.get_meter("buyer.application"),
+)
+
+current = observed_checkpoints.load(
+    "result-worker",
+    "batch-123",
+    "default",
+)
+```
+
+The wrapper delegates all arguments and returns unchanged. Use
+`load_in_transaction()` and `save_in_transaction()` exactly as on the underlying
+store when checkpoint advancement must share a caller-owned PostgreSQL
+transaction with local business effects. The wrapper does not commit or roll
+back that cursor.
+
+## Signal contract
+
+Spans:
+
+- `pg_llm_batch.checkpoint.load`
+- `pg_llm_batch.checkpoint.save`
+
+Metrics:
+
+- `pg_llm_batch.checkpoint.operation.count`, unit `{operation}`
+- `pg_llm_batch.checkpoint.operation.duration`, unit `s`
+
+Fixed attributes:
+
+- `pg_llm_batch.checkpoint.operation=load|save`
+- `pg_llm_batch.checkpoint.transaction_owner=package|caller`
+- `pg_llm_batch.checkpoint.outcome=success|conflict|validation_error|error`
+- `error.type=checkpoint_conflict|validation_error|internal_error` on failures
+
+Success omits `error.type`. The duration is measured with a monotonic clock and
+is never negative. By contract, failed checkpoint spans explicitly set
+OpenTelemetry status Error without a description, while successful checkpoint
+spans leave status Unset. This keeps failure discovery interoperable without
+copying exception messages into span status descriptions. The wrapper resolves
+the host OpenTelemetry API's `StatusCode.ERROR` only when that optional API is
+available; if status resolution or mutation fails, the checkpoint result or
+exact application exception remains authoritative.
+
+Package operation spans are deliberately storage-agnostic and do not emit
+`db.system.name`. They describe the checkpoint abstraction, not a database client
+call. A host that needs PostgreSQL client spans should enable its own database
+instrumentation at the actual database-client boundary, where OpenTelemetry
+database semantic conventions can be applied truthfully. This preserves the same
+wrapper contract for non-PostgreSQL compatible stores.
+
+## Confidentiality and cardinality
+
+Package-owned telemetry never contains tenant scope, checkpoint consumer name,
+remote batch identifier, endpoint alias, provider file identifier, checkpoint
+digest, database cursor, DSN, provider payload, exception message, exception
+object, or dynamic exception class name. Do not add those values as metric
+attributes in host wrappers; they create confidentiality and unbounded-cardinality
+risk. Use access-controlled logs or a separately reviewed audit store when an
+operator must reconcile a specific durable identity.
+
+Automatic exception recording and automatic status-on-exception are disabled.
+The wrapper instead applies the fixed Error status explicitly, without a status
+description, after classifying a failed operation. It supplies `(None, None,
+None)` when closing the span context so the application exception is not handed
+to observer code through context-manager arguments.
+
+## Failure behavior
+
+Telemetry is best effort. Ordinary tracer, meter, span, exporter-surface, clock,
+optional OpenTelemetry status-code resolution, status mutation, and
+telemetry-originated cancellation failures are contained. They do not change the
+checkpoint result, exact exception object, compare-and-swap decision,
+transaction owner, commit, or rollback behavior. Non-cancellation process-control
+exceptions remain outside this observer-failure guarantee.
+
+A caller-owned transaction span measures only the package method call. It does
+not prove that the surrounding transaction later committed, replicated, or
+completed downstream effects. Observe those boundaries in the embedding service.
+
+## Rollback
+
+Remove `OpenTelemetryCheckpointStore` from service composition and call the
+underlying checkpoint store directly. No schema, migration, persisted row,
+checkpoint digest, or package version needs rollback.