feat(health): observe PostgreSQL scoring-job backlogs

CHANGELOG.md

@@ -5,6 +5,7 @@ All notable product and architecture changes are recorded here. Releases use imm
 ## Unreleased
 
 ### Added
+- PostgreSQL operational-backlog probes classify aggregate outbox, inbox-consumption, data-rights, and scoring-job counts and ages against an explicit caller policy. They expose no payloads, tenant identities, worker identities, or restricted linkage values, and they invent no universal SLO defaults. `classify_postgres_operational_backlog` combines the three families so a healthy outbox cannot hide a stalled scoring queue. Call `apply_backlog_health_index_migration` after the integration, inbox-consumption, and data-rights migrations, and use `apply_scoring_job_migration` so scoring-job readiness indexes from `migrations/0021_scoring_job_health_indexes.sql` are created through the product apply path.
 - Scoring-job cancel and lease-expiry fallback classification lock the current row until the caller transaction ends, so concurrent workers cannot rewrite terminal or unleased evidence.
 - PostgreSQL operational-store readiness probe classifies the supported major version and write-readiness, and fails closed when a caller-declared required relation is missing.
 - PostgreSQL scoring-job cancellation: queued, leased, or retry-scheduled work becomes cancelled without transferring a fence, exact replay is idempotent, and completed or quarantined evidence cannot be rewritten.

docs/OPERABILITY.md

@@ -39,7 +39,7 @@ The implementation must distinguish at least:
 - **liveness** — process can make progress / is not irrecoverably wedged;
 - **readiness** — mandatory dependencies for the selected profile are available enough to accept new work safely;
 - **capability health** — optional/independent capabilities such as authenticated linking, scoring, AI narrative, research registration, or temporal analysis;
-- **backlog health** — durable work is within measured operating bounds and not silently stalled;
+- **backlog health** — durable work is within measured operating bounds and not silently stalled. When a PostgreSQL operational-backlog probe is present, it classifies aggregate outbox, inbox-consumption, data-rights, and scoring-job counts/ages against caller-supplied policy only and must not expose payloads, tenant identities, worker identities, or restricted linkage values. Apply `migrations/0020_backlog_health_indexes.sql` through `apply_backlog_health_index_migration` after the owning table migrations, and apply scoring-job readiness indexes through `apply_scoring_job_migration`, so readiness probes stay bounded as terminal history grows;
 - **data integrity health** — migrations/schema/digests/reconciliation do not indicate incompatible or corrupt state.
 
 Readiness must not fail solely because an optional capability is unavailable if the selected operation can safely proceed without it. Conversely, a process can be live while not ready to accept new state-changing requests.
@@ -161,6 +161,7 @@ The deployed profile must have executable or operator-tested runbooks for at lea
 - outbox publication backlog;
 - inbox processing backlog or poison message;
 - scoring dependency outage and job reconciliation;
+- scoring-job queue, lease, retry, or quarantine backlog;
 - Keyverse/JWKS/federation outage;
 - account-link conflict/adjudication;
 - optional AI provider/orchestrator outage and deterministic fallback verification;
@@ -229,6 +230,10 @@ Never collapse these maturity levels. SOC 2/CSAP readiness work may map evidence
 
 ## 15. References
 
+Beyer, B., Jones, C., Petoff, J., & Murphy, N. R. (Eds.). (2016). *Site reliability engineering: How Google runs production systems*. O'Reilly Media.
+
 International Organization for Standardization & International Electrotechnical Commission. (2023). *ISO/IEC 25010:2023 Systems and software engineering—Systems and software Quality Requirements and Evaluation (SQuaRE)—Product quality model*.
 
+National Institute of Standards and Technology. (2020). *Security and privacy controls for information systems and organizations* (NIST SP 800-53 Rev. 5). https://doi.org/10.6028/NIST.SP.800-53r5
+
 National Institute of Standards and Technology. (2022). *Secure Software Development Framework (SSDF) Version 1.1: Recommendations for mitigating the risk of software vulnerabilities* (NIST SP 800-218). https://doi.org/10.6028/NIST.SP.800-218

docs/TRACEABILITY.md

@@ -40,7 +40,7 @@ An active PR, architecture document, conversation decision, or scheduler plan is
 | Research identity separation | PRD §5, §11 | TRD §14; ERD restricted linkage | ADR-0003, ADR-0006, ADR-0007, ADR-0020 | Partially implemented via research-contribution identity separation; restricted linkage persistence is Target |
 | Research release manifests | PRD §5 | TRD §15 | ADR-0007, ADR-0010 | Target; semantic-data-portal is External dependency |
 | Durable outbox/inbox delivery semantics | PRD §7, §9 | TRD §19–20 | ADR-0014, ADR-0015 | **Partially implemented**: domain contracts in `src/integration.rs`; PostgreSQL 18 outbox/inbox identity, delivery-attempt persistence, and inbox consumption distinct from receipt; live side-effect execution remains Target |
-| Operation-scoped capability health | PRD §7, §13 | `docs/OPERABILITY.md` §3–4; Deployment/Operations | ADR-0011, ADR-0017 | **Implemented** domain health/readiness contract in `src/health.rs` plus `src/postgres_health.rs` PostgreSQL major/write-readiness and caller-declared relation presence; HTTP probes, measured thresholds, and deployment evidence remain Target |
+| Operation-scoped capability health | PRD §7, §13 | `docs/OPERABILITY.md` §3–4; Deployment/Operations | ADR-0011, ADR-0017 | **Implemented** domain health/readiness contract in `src/health.rs` plus `src/postgres_health.rs` PostgreSQL major/write-readiness and caller-declared relation presence; **Active PR** #82 observes aggregate outbox, inbox-consumption, and data-rights backlog counts/ages and applies partial readiness indexes through `apply_backlog_health_index_migration`; **Active PR** #113 observes queued/leased/retry-scheduled/quarantined scoring-job counts and created-at age through `apply_scoring_job_migration`; HTTP probes remain #111; measured thresholds and deployment-profile evidence remain Target |
 | Korean/English exact locale versions | PRD §3.1, §9.9 | TRD §28; instrument release + locale governance | ADR-0013, ADR-0019 | **Partially implemented**: locale is pinned/validated by `src/instrument.rs`; actual English/Korean form content, rights, translation, invariance and serving are Target |
 | WCAG 2.2 AA supported reference client | PRD §9.10 | TRD §27; Quality Attributes | ADR-0002, ADR-0013 | Target; no reference client implementation on evaluated main |
 | EMA/ESM longitudinal flow | PRD §4 | TRD §16; UML longitudinal sequence; logical ERD extension | ADR-0008 | External Gyeot/TEPP dependencies + Target Commons enrollment/normalized-ingestion/orchestration adapter |
@@ -79,7 +79,7 @@ An active PR, architecture document, conversation decision, or scheduler plan is
 | Inbox receipt is not side-effect completion | ADR-0014/0015; UML integration sequence | `src/integration.rs` states/retry semantics; PostgreSQL inbox consumption persists pending/processing/completed and expire-and-reclaim | live adapter crash/retry tests |
 | Liveness is distinct from operation readiness | Operability §3–4; ADR-0017 | **Implemented** in `src/health.rs` and `src/postgres_health.rs`: liveness is modeled independently from operation-scoped readiness and PostgreSQL write-readiness | live transport probes, metrics, and deployment-profile acceptance |
 | Optional capability outage does not fail unrelated work | Operability §3–4; ADR-0011/0017 | **Implemented** in `src/health.rs` and `src/postgres_health.rs`: readiness evaluates only capabilities required by the selected operation and maps PostgreSQL evidence onto that contract | degraded-mode transport/integration tests |
-| Unknown/stalled backlog or unknown/incompatible integrity blocks new state-changing work | Operability §3, §6, §8 | **Implemented** domain contract in `src/health.rs`; `src/postgres_health.rs` fails closed on unsupported/read-only PostgreSQL or a missing required relation | persistence/job backlog metrics, stronger schema probes, alerting, and failure-injection evidence |
+| Unknown/stalled backlog or unknown/incompatible integrity blocks new state-changing work | Operability §3, §6, §8 | **Implemented** domain contract in `src/health.rs`; `src/postgres_health.rs` fails closed on unsupported/read-only PostgreSQL or a missing required relation; **Active PR** #82 classifies durable integration and data-rights backlog evidence against an explicit caller policy; **Active PR** #113 classifies queued/leased/retry-scheduled/quarantined scoring-job evidence the same way | HTTP probe wiring on #111, measured deployment-profile thresholds, alerting, and failure-injection evidence |
 | No operational IDs in public research release | TRD §14–15; Research Governance | architecture policy | release fixture/static/runtime leakage tests |
 | AI optional; deterministic core remains | PRD §9.5; TRD §17; AI Governance | architecture policy | narrative fallback end-to-end test |
 | AI cannot mutate numeric scientific result | AI Governance; ADR-0009, ADR-0018 | architecture policy | product adapter/adversarial mutation tests |
@@ -104,7 +104,7 @@ src/lib.rs
 ├── participant.rs    # stable participant identity + issuer-scoped optional Keyverse account link
 ├── postgres_consent.rs  # PostgreSQL purpose-specific consent ledger persistence
 ├── postgres_data_rights.rs  # PostgreSQL data-rights request and local propagation persistence
-├── postgres_health.rs  # PostgreSQL major/write-readiness and relation-integrity probe
+├── postgres_health.rs  # PostgreSQL major/write-readiness, relation-integrity, and Active PR operational-backlog probes including scoring-job queues
 ├── postgres_inbox_consumption.rs  # PostgreSQL inbox consumption distinct from receipt
 ├── postgres_instrument_release.rs  # PostgreSQL locale-specific instrument-release persistence
 ├── postgres_integration.rs  # PostgreSQL integration evidence/delivery-attempt persistence adapter
@@ -132,7 +132,7 @@ Still-Target logical modules/adapters include remaining product aggregate persis
 
 ### Active implementation work that is not protected-main truth
 
-**Active PR** #76 data-rights processing-start persistence is not protected-main truth until an unchanged reviewed/check-clean head is integrated. Identity-verified requests persist an immutable operation identity and processing-start time under `FOR UPDATE` so later lifecycle composition cannot race the classified row. Dependent-system execution remains outside this slice.
+**Active PR** #82 PostgreSQL operational-backlog observation is not protected-main truth until an unchanged reviewed/check-clean head is integrated. Aggregate outbox, inbox-consumption, and data-rights counts/ages are classified only against an explicit caller policy. Partial readiness indexes ship in `migrations/0020_backlog_health_indexes.sql` and must be applied through `apply_backlog_health_index_migration` after the owning table migrations; a private test `include_str!` is not a product apply path. **Active PR** #113 adds the same content-free observation for queued, leased, retry-scheduled, and quarantined scoring work, with readiness indexes applied through `apply_scoring_job_migration`. HTTP probes remain #111. Deployment-profile thresholds remain outside this slice.
 
 ## 5. ADR traceability by concern
 

migrations/0020_backlog_health_indexes.sql

@@ -0,0 +1,21 @@
+-- Keep readiness probes bounded as durable terminal history grows.
+-- data_rights_propagation_state already has the state/time index created by 0003.
+CREATE INDEX IF NOT EXISTS integration_outbox_pending_health_idx
+    ON integration_outbox (latest_event_at_unix_ms)
+    WHERE current_state = 'pending';
+
+CREATE INDEX IF NOT EXISTS integration_outbox_quarantined_health_idx
+    ON integration_outbox (latest_event_at_unix_ms)
+    WHERE current_state = 'quarantined';
+
+CREATE INDEX IF NOT EXISTS integration_consumption_active_health_idx
+    ON integration_consumption (latest_event_at_unix_ms)
+    WHERE consumption_state IN ('pending', 'processing');
+
+CREATE INDEX IF NOT EXISTS integration_consumption_quarantined_health_idx
+    ON integration_consumption (latest_event_at_unix_ms)
+    WHERE consumption_state = 'quarantined';
+
+CREATE INDEX IF NOT EXISTS data_rights_request_active_health_idx
+    ON data_rights_request_state (requested_at_unix_ms)
+    WHERE current_state IN ('requested', 'identity_verified', 'processing');

migrations/0021_scoring_job_health_indexes.sql

@@ -0,0 +1,8 @@
+-- Keep scoring-job readiness probes bounded as terminal history grows.
+CREATE INDEX IF NOT EXISTS scoring_job_state_active_health_idx
+    ON scoring_job_state (created_at)
+    WHERE scoring_state IN ('queued', 'leased', 'retry_scheduled');
+
+CREATE INDEX IF NOT EXISTS scoring_job_state_quarantined_health_idx
+    ON scoring_job_state (created_at)
+    WHERE scoring_state = 'quarantined';

src/health.rs

@@ -22,6 +22,23 @@ pub enum BacklogHealth {
 }
 
 impl BacklogHealth {
+    /// Combine two independently measured durable-work signals for one readiness decision.
+    ///
+    /// A known stalled backlog takes precedence because accepting more state-changing work is
+    /// already unsafe. Otherwise an unknown signal takes precedence over a healthy signal so a
+    /// missing observation cannot be masked. The result is within bounds only when both signals
+    /// are within their caller-supplied operating bounds.
+    #[must_use]
+    pub const fn combine(self, other: Self) -> Self {
+        if matches!(self, Self::Stalled) || matches!(other, Self::Stalled) {
+            Self::Stalled
+        } else if matches!(self, Self::Unknown) || matches!(other, Self::Unknown) {
+            Self::Unknown
+        } else {
+            Self::WithinBounds
+        }
+    }
+
     const fn accepts_new_work(self) -> bool {
         matches!(self, Self::WithinBounds)
     }