Skip to content
Closed
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
1 change: 1 addition & 0 deletions CHANGELOG.md
Original file line number Diff line number Diff line change
Expand Up @@ -5,6 +5,7 @@ All notable product and architecture changes are recorded here. Releases use imm
## Unreleased

### Added
- PostgreSQL persistence for in-progress `response_event` rows so a two-item path reloads the same answers after restart. Exact replay is idempotent; client, server, sequence, or session rebinding fails closed. Store observed and received times with the event, expose those times on reload, restore them through recovery COPY, and continue the scoring prefix after item 3. Completed snapshots stay on the existing snapshot adapter.
- Scoring-job cancel and lease-expiry fallback classification lock the current row until the caller transaction ends, so concurrent workers cannot rewrite terminal or unleased evidence.
- PostgreSQL operational-store readiness probe classifies the supported major version and write-readiness, and fails closed when a caller-declared required relation is missing.
- PostgreSQL scoring-job cancellation: queued, leased, or retry-scheduled work becomes cancelled without transferring a fence, exact replay is idempotent, and completed or quarantined evidence cannot be rewritten.
Expand Down
6 changes: 3 additions & 3 deletions docs/TRACEABILITY.md
Original file line number Diff line number Diff line change
Expand Up @@ -23,7 +23,7 @@ An active PR, architecture document, conversation decision, or scheduler plan is
| Anonymous core assessment | PRD §3.1, §9.1 | TRD §5, §10; UML anonymous sequence | ADR-0002, ADR-0003, ADR-0005 | Session lifecycle primitives implemented, including creation bound to one published locale-specific release; anonymous credential/HTTP flow is Target |
| Pause/resume | PRD §3.1, §9.1 | TRD §5 | ADR-0005 | **Implemented** in `src/session.rs` with fail-closed transitions |
| Sequence-aware item delivery evidence | PRD §3.1, §9 | TRD §5–7 | ADR-0005, ADR-0010 | **Implemented** domain primitive in `src/item_delivery.rs`; persistence/API delivery orchestration is Target |
| Idempotent response events | PRD §9.2 | TRD §6 | ADR-0005, ADR-0010 | **Implemented** in `src/response.rs` with canonical SHA-256 payload-digest identity; persistence adapter is Target |
| Idempotent response events | PRD §9.2 | TRD §6 | ADR-0005, ADR-0010 | **Implemented** in `src/response.rs` with canonical SHA-256 payload-digest identity; **Active PR** #201 persists and reloads `response_event` under `READ COMMITTED`; HTTP response transport remains Target |
| Immutable response snapshot before scoring | PRD §9.3 | TRD §5–8 | ADR-0005, ADR-0010 | **Implemented** domain semantics in `src/response.rs` |
| Version-pinned scoring | PRD §9.4, §10 | TRD §8 | ADR-0004, ADR-0010 | **Implemented** reusable product-side scoring dispatch contract in `src/scoring.rs` with canonical SHA-256 engine-artifact digest provenance plus `migrations/0011_scoring_request.sql` / `src/postgres_scoring_request.rs` request-identity persistence; live fast-mlsirm integration is Target |
| Bounded asynchronous scoring retry/quarantine with stale-worker fencing | PRD §9.4, §10 | TRD §8; ADR-0015 transaction boundary | ADR-0004, ADR-0010, ADR-0015 | **Implemented** product lifecycle plus PostgreSQL enqueue, claim, retry, completion, expiry recovery, and cancellation without transferring a fence; live fast-mlsirm execution remains Target |
Expand Down Expand Up @@ -55,7 +55,7 @@ An active PR, architecture document, conversation decision, or scheduler plan is
| Server-authoritative session state | TRD §5 | `src/session.rs` + session contract tests, including published-release/locale binding at creation | persistence/API concurrency test |
| Only Active accepts responses | TRD §5–6 | `SessionState::accepts_responses` + response tests | transport-level rejection test |
| Item delivery sequence is positive and evidence-safe | TRD §5–7 | `src/item_delivery.rs` + item-delivery domain tests | durable uniqueness/order/API integration |
| Conflicting idempotency replay fails closed | TRD §6 | `src/response.rs` | DB uniqueness/concurrency test |
| Conflicting idempotency replay fails closed | TRD §6 | `src/response.rs`; **Active PR** #201 `migrations/0020_response_event.sql` unique `(session_ref, client_event_ref)` and `(session_ref, server_sequence)` | HTTP transport rejection test |
| Snapshot requires Completed state | TRD §5–6 | `src/response.rs` | transaction atomicity test with persistence |
| Scoring uses durable snapshot identity | TRD §8 | `src/scoring.rs` requires a canonical SHA-256 engine-artifact digest | live adapter + retry/outbox integration |
| Stale scoring worker cannot complete a newer attempt | TRD §8; ADR-0015 | `src/scoring_job.rs` uses monotonically increasing fencing tokens and rejects stale/expired completion or failure evidence; `src/postgres_scoring_job.rs` persists enqueue, claim, retry, terminal outcomes, expired-lease recovery, and cancellation without transferring a fence | live adapter evidence |
Expand Down Expand Up @@ -132,7 +132,7 @@ Still-Target logical modules/adapters include remaining product aggregate persis

### Active implementation work that is not protected-main truth

**Active PR** #76 data-rights processing-start persistence is not protected-main truth until an unchanged reviewed/check-clean head is integrated. Identity-verified requests persist an immutable operation identity and processing-start time under `FOR UPDATE` so later lifecycle composition cannot race the classified row. Dependent-system execution remains outside this slice.
**Active PR** #201 response-event persist/load is not protected-main truth until an unchanged reviewed/check-clean head is integrated. In-progress answers persist in `response_event` with observed/received time, exact replay, and fail-closed rebinding so a two-item path survives restart. Recovery COPY restores those rows. AS_BUILT names the physical slice. Reload exposes stored times as a persist-side projection and continues the scoring prefix after item 3. Completed snapshot persist remains the existing protected-main adapter. HTTP response transport remains outside this slice. Protected-main `#76` data-rights processing-start persistence is already integrated and is not restated here as active work. Prefer this head over #174, #182, and #53.

## 5. ADR traceability by concern

Expand Down
10 changes: 7 additions & 3 deletions docs/adr/0015-persistence-and-transaction-boundaries.md
Original file line number Diff line number Diff line change
Expand Up @@ -6,9 +6,9 @@
- Scope: Psychometrics Commons-owned durable state, local transactions, migration boundaries, outbox/inbox integration
- Supersedes: none
- Superseded by: none
- Current/as-built status: protected main contains in-memory/domain lifecycle primitives only; active PR #24 carries the first PostgreSQL integration-evidence migration/adapter but is not protected-main truth until merged
- Current/as-built status: protected main persists integration, scoring-job, data-rights, item-delivery, consent, instrument-release, result-snapshot, response-snapshot, scoring-request, and inbox-consumption slices; in-progress `response_event` persist/load is Active PR #201 and is not protected-main truth until merged
- Target status: upstream PostgreSQL 18.x operational persistence with real-database concurrency/crash/recovery evidence and transactional outbox/inbox semantics
- Migration status: active PR #24 introduces only the bounded integration-evidence slice; the remaining product schema still must be established from the logical ERD and this ADR without synthetic provenance backfills
- Migration status: `migrations/0020_response_event.sql` adds the in-progress event ledger with observed/received timestamps; remaining session HTTP and response HTTP families still must be established from the logical ERD without synthetic provenance backfills

## Context

Expand Down Expand Up @@ -76,7 +76,7 @@ A physical migration may split an entity across tables or co-locate value object

### Response recording

One transaction validates the current session state, reserves server sequence, applies the idempotency/uniqueness contract, and stores the accepted response event. Two concurrent requests cannot both create the same logical `client_event_ref`.
One transaction validates the current session state, reserves server sequence, applies the idempotency/uniqueness contract, and stores the accepted response event with distinct observed and received instants (ISO 8601-1). Two concurrent requests cannot both create the same logical `client_event_ref`. The first physical classifier uses `READ COMMITTED` so a concurrent unique-key winner is visible to the exact-replay inspection (Berenson et al., 1995; PostgreSQL Global Development Group, 2026).

### Session completion

Expand Down Expand Up @@ -303,6 +303,10 @@ The physical database technology or decomposition may change if scale, residency

## References

Berenson, H., Bernstein, P., Gray, J., Melton, J., O'Neil, E., & O'Neil, P. (1995). A critique of ANSI SQL isolation levels. *ACM SIGMOD Record, 24*(2), 1–10. https://doi.org/10.1145/568271.223785

International Organization for Standardization. (2019). *Date and time — Representations for information interchange — Part 1: Basic rules* (ISO 8601-1:2019).

PostgreSQL Global Development Group. (2026). *PostgreSQL 18 documentation*.

PostgreSQL Global Development Group. (2026). *PostgreSQL versioning policy*.
4 changes: 4 additions & 0 deletions docs/architecture/AS_BUILT_SCHEMA.md
Original file line number Diff line number Diff line change
Expand Up @@ -25,6 +25,10 @@ The protected-main integration identity is source- and tenant-scoped. A physical

PR #58 (`feat/inbox-consumption-persistence-20260814`) `migrations/0012_integration_consumption.sql` and `src/postgres_inbox_consumption.rs` adapter persist one consumption work item for an existing `integration_inbox` receipt. The slice is **Active PR**, not protected-main truth. It stores pending/processing/completed/quarantined evidence, a monotonically increasing fencing token, a time-bounded processing claim, a durable `side_effect_ref`, and optional completion or quarantine evidence. Receipt-only inbox rows remain uncompleted. A processing claim cannot be stolen by another worker. Expire-and-reclaim returns an expired claim to pending without transferring the crashed worker's fence.

## Active PR response-event physical schema

Active PR #201 `migrations/0020_response_event.sql` and `src/postgres_response_event.rs` persist the in-progress answer ledger so a two-item path can continue after process restart. The slice is **Active PR**, not protected-main truth. It stores opaque `response_event_ref` identity, session binding, client idempotency identity, item version, canonical SHA-256 payload digest, positive `server_sequence`, and distinct `observed_at` / `received_at` timestamps. Exact replay is idempotent and keeps the original times. Client, server, sequence, or session rebinding fails closed. Reload reconstructs `ResponseLedger` in `server_sequence` order under `READ COMMITTED` and exposes stored times as a persist-side projection. HTTP response transport remains outside this slice.

## Protected-main scoring-job physical schema

`migrations/0002_scoring_job_state.sql` maps a bounded physical subset of the logical `scoring_job` aggregate into `scoring_job_state`, owned by `src/postgres_scoring_job.rs`. This is an **Implemented subset** on the named protected-main baseline.
Expand Down
1 change: 1 addition & 0 deletions docs/architecture/ERD.md
Original file line number Diff line number Diff line change
Expand Up @@ -425,6 +425,7 @@ The target ERD deliberately includes several logical entities that are not yet p
- `instrument_release` is the locale-specific publication identity already owned by `src/instrument.rs`. Physical `migrations/0006_instrument_release.sql` persists that one-row aggregate (immutable manifest columns plus `publication_state`); HTTP publication transport remains Target.
- `data_rights_request` and `data_rights_propagation_state` are the first durable export/deletion slice. Physical `migrations/0003_data_rights_propagation.sql` stores requested-state identity plus one local outbox event per dependent system; verification, processing, completion, and dependent-system execution remain Target.
- `item_delivery_event` reflects the already-merged `src/item_delivery.rs` domain primitive; durable persistence/API orchestration is still Target.
- `response_event` is the in-progress answer ledger. Physical `migrations/0020_response_event.sql` on Active PR #201 stores opaque event identity, session binding, client idempotency, item version, payload digest, server sequence, and distinct observed/received timestamps; HTTP response transport remains Target. Completed `response_snapshot` persist is already on protected main.
- `consent_ledger` and `consent_event` persist the already-merged `src/consent.rs` append-only ledger. Physical persistence is carried by Active PR #49 (`migrations/0005_consent_lifecycle.sql`); HTTP consent transport and derived snapshot tables remain Target.
- `participant_identity_link` is the persistence target accepted by ADR-0020. The current `src/participant.rs` `keyverse_subject_ref` field is an application-domain first-link projection, not the future mutable persistence source of truth.
- `longitudinal_enrollment`, `longitudinal_observation_record`, and `temporal_analysis_submission` make the ADR-0008 Commons-owned Gyeot/TEPP orchestration boundary explicit. No TEPP analytical kernel is duplicated here.
Expand Down
2 changes: 2 additions & 0 deletions docs/architecture/UML.md
Original file line number Diff line number Diff line change
Expand Up @@ -66,6 +66,8 @@ classDiagram
+item_version_ref
+payload_digest
+server_sequence
+observed_at
+received_at
}
class ResponseSnapshot {
+response_snapshot_ref
Expand Down
10 changes: 9 additions & 1 deletion docs/doctoring/standards-and-evidence.md
Original file line number Diff line number Diff line change
Expand Up @@ -92,7 +92,11 @@ Product consequences:
- clock skew, impossible ordering, and unknown precision are typed validation
outcomes rather than reasons to rewrite source history;
- analysis-set digests bind the exact observations and time semantics consumed by
temporal, multilevel, cross-classified, or multiple-membership analysis.
temporal, multilevel, cross-classified, or multiple-membership analysis;
- in-progress `response_event` persist uses PostgreSQL `READ COMMITTED` so a
concurrent unique-key winner is visible to exact-replay classification, and
stores observed time separately from platform receipt time (Berenson et al.,
1995; PostgreSQL Global Development Group, 2026).

## Evidence maintenance rules

Expand All @@ -107,6 +111,8 @@ Product consequences:

American Educational Research Association, American Psychological Association, & National Council on Measurement in Education. (2014). *Standards for educational and psychological testing*. American Educational Research Association. https://www.testingstandards.net/

Berenson, H., Bernstein, P., Gray, J., Melton, J., O'Neil, E., & O'Neil, P. (1995). A critique of ANSI SQL isolation levels. *ACM SIGMOD Record, 24*(2), 1–10. https://doi.org/10.1145/568271.223785

International Organization for Standardization. (2022). *ISO/IEC 27001:2022 Information security, cybersecurity and privacy protection—Information security management systems—Requirements* (3rd ed.). https://www.iso.org/standard/27001

International Organization for Standardization. (2023a). *ISO/IEC 23894:2023 Information technology—Artificial intelligence—Guidance on risk management*. https://www.iso.org/standard/77304.html
Expand All @@ -119,6 +125,8 @@ International Organization for Standardization. (2025). *ISO/IEC 42005:2025 Info

International Organization for Standardization. (2019). *ISO 8601-1:2019 Date and time—Representations for information interchange—Part 1: Basic rules* (with Amendment 1:2022). https://www.iso.org/standard/70907.html

PostgreSQL Global Development Group. (2026). *PostgreSQL 18 documentation: Transaction isolation*. https://www.postgresql.org/docs/18/transaction-iso.html

Temoshok, D., Proud-Madruga, D., Choong, Y.-Y., Galluzzo, R., Gupta, S., LaSalle, C., Lefkovitz, N., & Regenscheid, A. (2025). *Digital identity guidelines* (NIST Special Publication 800-63-4). National Institute of Standards and Technology. https://doi.org/10.6028/NIST.SP.800-63-4

World Wide Web Consortium. (2024). *Web Content Accessibility Guidelines (WCAG) 2.2* (W3C Recommendation, 12 December 2024). https://www.w3.org/TR/WCAG22/
Expand Down
53 changes: 53 additions & 0 deletions migrations/0020_response_event.sql
Original file line number Diff line number Diff line change
@@ -0,0 +1,53 @@
-- Opaque response-event references remain data, not SQL syntax. The persistence
-- adapter binds them as query parameters. These checks enforce the canonical
-- identity boundary (nonblank, nonnumeric-like, no outer whitespace).
CREATE TABLE IF NOT EXISTS response_event (
response_event_ref TEXT CONSTRAINT response_event_response_event_ref_not_null NOT NULL
CONSTRAINT response_event_response_event_ref_format_check CHECK (
response_event_ref = btrim(response_event_ref)
AND response_event_ref <> ''
AND NOT (
response_event_ref ~ '[[:digit:]]'
AND response_event_ref ~ '^[[:digit:]+,.eE-]+$'
)
),
session_ref TEXT CONSTRAINT response_event_session_ref_not_null NOT NULL
CONSTRAINT response_event_session_ref_format_check CHECK (
session_ref = btrim(session_ref)
AND session_ref <> ''
AND NOT (
session_ref ~ '[[:digit:]]'
AND session_ref ~ '^[[:digit:]+,.eE-]+$'
)
),
client_event_ref TEXT CONSTRAINT response_event_client_event_ref_not_null NOT NULL
CONSTRAINT response_event_client_event_ref_format_check CHECK (
client_event_ref = btrim(client_event_ref)
AND client_event_ref <> ''
AND NOT (
client_event_ref ~ '[[:digit:]]'
AND client_event_ref ~ '^[[:digit:]+,.eE-]+$'
)
),
item_version_ref TEXT CONSTRAINT response_event_item_version_ref_not_null NOT NULL
CONSTRAINT response_event_item_version_ref_format_check CHECK (
item_version_ref = btrim(item_version_ref)
AND item_version_ref <> ''
AND NOT (
item_version_ref ~ '[[:digit:]]'
AND item_version_ref ~ '^[[:digit:]+,.eE-]+$'
)
),
payload_digest TEXT CONSTRAINT response_event_payload_digest_not_null NOT NULL
CONSTRAINT response_event_payload_digest_format_check CHECK (
payload_digest ~ '^sha256:[0-9a-f]{64}$'
),
server_sequence BIGINT CONSTRAINT response_event_server_sequence_not_null NOT NULL
CONSTRAINT response_event_server_sequence_positive_check CHECK (server_sequence > 0),
observed_at TIMESTAMPTZ CONSTRAINT response_event_observed_at_not_null NOT NULL,
received_at TIMESTAMPTZ CONSTRAINT response_event_received_at_not_null NOT NULL,
CONSTRAINT response_event_pkey PRIMARY KEY (response_event_ref),
CONSTRAINT response_event_session_client_unique UNIQUE (session_ref, client_event_ref),
CONSTRAINT response_event_session_sequence_unique UNIQUE (session_ref, server_sequence),
CONSTRAINT response_event_observed_not_after_received_check CHECK (observed_at <= received_at)
);
1 change: 1 addition & 0 deletions src/lib.rs
Original file line number Diff line number Diff line change
Expand Up @@ -27,6 +27,7 @@ pub mod postgres_inbox_consumption;
pub mod postgres_instrument_release;
pub mod postgres_integration;
pub mod postgres_item_delivery;
pub mod postgres_response_event;
pub mod postgres_response_snapshot;
pub mod postgres_result_snapshot;
pub mod postgres_scoring_job;
Expand Down
Loading
Loading