Skip to content
Draft
1 change: 1 addition & 0 deletions CHANGELOG.md
Original file line number Diff line number Diff line change
Expand Up @@ -5,6 +5,7 @@ All notable product and architecture changes are recorded here. Releases use imm
## Unreleased

### Added
- Restricted research-identity linkage persists one operational participant to one program-scoped research pseudonym, keeps a second program from collapsing into the same identity, and exposes a public-release view that cannot carry the operational reference or linkage-key version. A release fixture loads that view by program and cannot look up a restricted linkage identity. Exact replay is idempotent; conflicting rebinding and padded load identities fail closed.
- Scoring-job cancel and lease-expiry fallback classification lock the current row until the caller transaction ends, so concurrent workers cannot rewrite terminal or unleased evidence.
- PostgreSQL operational-store readiness probe classifies the supported major version and write-readiness, and fails closed when a caller-declared required relation is missing.
- PostgreSQL scoring-job cancellation: queued, leased, or retry-scheduled work becomes cancelled without transferring a fence, exact replay is idempotent, and completed or quarantined evidence cannot be rewritten.
Expand Down
2 changes: 1 addition & 1 deletion docs/RESEARCH_GOVERNANCE.md
Original file line number Diff line number Diff line change
Expand Up @@ -67,7 +67,7 @@ The consent user experience explains, as applicable:

Operational participant identifiers are not used as research identifiers.

A restricted linkage boundary maps an operational participant to a purpose/program-specific research pseudonym. The mapping:
A restricted linkage boundary maps an operational participant to a purpose/program-specific research pseudonym. Active PR #187 persistence for that mapping is not protected-main truth. A public-release fixture loads `public_research_identity` by program and never looks up a restricted linkage identity. The mapping:

- is unavailable to ordinary analytics and public-release workflows;
- is accessed only through an explicit restricted role/purpose;
Expand Down
2 changes: 1 addition & 1 deletion docs/RISK_REGISTER.md
Original file line number Diff line number Diff line change
Expand Up @@ -25,7 +25,7 @@ This register tracks material product, scientific, privacy, security, operationa
| Cross-tenant object reference exposes another user's session/result/research/data-rights state | critical | medium | evidence_required | tenant/resource authorization architecture; transport/persistence negative tests pending |
| Anonymous-to-account link permits account takeover/history theft | critical | medium | evidence_required | dual proof-of-control + Keyverse validation; adapter and adversarial tests pending |
| Keyverse identity role is confused with product/research authorization | high | medium | mitigated_by_architecture | separate domain authorization and separation-of-duties policy; integration tests pending |
| Research release contains operational/Keyverse/linkage identifier | critical | medium | mitigated_by_architecture | restricted linkage + release validation; adversarial release pipeline pending |
| Research release contains operational/Keyverse/linkage identifier | critical | medium | implementation_in_progress | restricted linkage persistence Active PR #187 plus program-scoped `public_research_identity` load; adversarial release pipeline pending |
| High-dimensional/longitudinal release is re-identifiable despite removed direct identifiers | critical | medium | mitigated_by_architecture | privacy-risk review, access class, rare-combination/joinability checks; operational process/evidence pending |
| Research consent is bundled with service use or future release exceeds consent scope | critical | low | implementation_in_progress | purpose-specific consent/research contribution domain contract; UI/API/snapshot enforcement pending |
| Restore resurrects data previously deleted by valid data-rights request | high | medium | mitigated_by_architecture | ADR-0017 deletion reconciliation; real backup/restore drill pending |
Expand Down
9 changes: 6 additions & 3 deletions docs/TRACEABILITY.md
Original file line number Diff line number Diff line change
Expand Up @@ -37,7 +37,7 @@ An active PR, architecture document, conversation decision, or scheduler plan is
| Purpose-specific consent | PRD §5, §9.6 | TRD §12 | ADR-0006 | **Implemented** domain contract in `src/consent.rs` plus `migrations/0005_consent_lifecycle.sql` / `src/postgres_consent.rs` purpose-specific ledgers; HTTP transport remains Target |
| Explicit research contribution + withdrawal | PRD §5 | TRD §12, §14–15 | ADR-0006, ADR-0007 | **Implemented** product-domain lifecycle in `src/consent.rs`; dataset snapshot/release integration is Target |
| Participant export/deletion | PRD §3.1, §9, §11 | TRD §13 | ADR-0006 | **Implemented** domain lifecycle in `src/data_rights.rs` plus `migrations/0003_data_rights_propagation.sql` and `src/postgres_data_rights.rs`; dependent-system execution remains Target |
| Research identity separation | PRD §5, §11 | TRD §14; ERD restricted linkage | ADR-0003, ADR-0006, ADR-0007, ADR-0020 | Partially implemented via research-contribution identity separation; restricted linkage persistence is Target |
| Research identity separation | PRD §5, §11 | TRD §14; ERD restricted linkage | ADR-0003, ADR-0006, ADR-0007, ADR-0020 | **Partially implemented**: research-contribution identity separation exists on evaluated main; **Active PR** #187 restricted linkage persistence binds one operational participant to a program-scoped research identity, keeps multiple programs from collapsing into one pseudonym, and loads only `public_research_identity` columns by program |
| Research release manifests | PRD §5 | TRD §15 | ADR-0007, ADR-0010 | Target; semantic-data-portal is External dependency |
| Durable outbox/inbox delivery semantics | PRD §7, §9 | TRD §19–20 | ADR-0014, ADR-0015 | **Partially implemented**: domain contracts in `src/integration.rs`; PostgreSQL 18 outbox/inbox identity, delivery-attempt persistence, and inbox consumption distinct from receipt; live side-effect execution remains Target |
| Operation-scoped capability health | PRD §7, §13 | `docs/OPERABILITY.md` §3–4; Deployment/Operations | ADR-0011, ADR-0017 | **Implemented** domain health/readiness contract in `src/health.rs` plus `src/postgres_health.rs` PostgreSQL major/write-readiness and caller-declared relation presence; HTTP probes, measured thresholds, and deployment evidence remain Target |
Expand Down Expand Up @@ -108,9 +108,11 @@ src/lib.rs
├── postgres_inbox_consumption.rs # PostgreSQL inbox consumption distinct from receipt
├── postgres_instrument_release.rs # PostgreSQL locale-specific instrument-release persistence
├── postgres_integration.rs # PostgreSQL integration evidence/delivery-attempt persistence adapter
├── postgres_research_identity_linkage.rs # Active PR #187 restricted operational-to-research identity persistence (not protected-main truth)
├── postgres_scoring_job.rs # PostgreSQL scoring enqueue/claim/retry/cancel/terminal persistence
├── postgres_scoring_request.rs # PostgreSQL version-pinned scoring-request identity
├── reference.rs # internal opaque-reference normalization
├── research_identity_linkage.rs # Active PR #187 restricted linkage domain and public-release projection (not protected-main truth)
├── research_release.rs # product-side Research Commons release-evidence gate
├── response.rs # idempotent response ledger + immutable response snapshots
├── result.rs # immutable result provenance/supersession
Expand All @@ -125,14 +127,15 @@ migrations/
├── 0005_consent_lifecycle.sql
├── 0006_instrument_release.sql
├── 0011_scoring_request.sql
└── 0012_integration_consumption.sql
├── 0012_integration_consumption.sql
└── 0025_research_identity_linkage.sql # Active PR #187 restricted linkage persistence (not protected-main truth)
```

Still-Target logical modules/adapters include remaining product aggregate persistence/repositories, public/admin HTTP and event transports, live fast-mlsirm/Keyverse/Gyeot/TEPP/semantic-data-portal adapters, research-release staging, deterministic narrative mapping, longitudinal normalized ingestion, participant identity-link history persistence, runtime health transports/metrics, and Measurement Workbench orchestration.

### Active implementation work that is not protected-main truth

**Active PR** #76 data-rights processing-start persistence is not protected-main truth until an unchanged reviewed/check-clean head is integrated. Identity-verified requests persist an immutable operation identity and processing-start time under `FOR UPDATE` so later lifecycle composition cannot race the classified row. Dependent-system execution remains outside this slice.
**Active PR** #187 restricted research-identity linkage persistence is not protected-main truth until an unchanged reviewed/check-clean head is integrated. Prefer this head over #175 and #162. Operators persist one operational participant to one program-scoped research identity, load the restricted mapping only for authorized research work, and publish only `public_research_identity` columns. A release fixture loads those columns by program from the public view and cannot look up a restricted linkage identity. A second program keeps a distinct research identity. Public releases, Keyverse subjects, and linkage-key material stay outside this slice.

## 5. ADR traceability by concern

Expand Down
2 changes: 2 additions & 0 deletions docs/architecture/ERD.md
Original file line number Diff line number Diff line change
Expand Up @@ -546,6 +546,8 @@ A tenant/resource mismatch or conflicting replay is quarantined/fails closed bef

`research_identity_linkage` is the highest-sensitivity product-owned data structure because it bridges operational participant identity to research pseudonym identity.

Physical persistence (Active PR #187, not protected-main truth) stores `research_program_ref` on both `research_participant` and `research_identity_linkage` as a composite foreign key. That keeps third-normal-form identity (program is an identifying attribute of the research participant) while enforcing one research identity per operational participant and program. The public-release view `public_research_identity` projects only `research_participant_ref` and `research_program_ref`. A release fixture loads that view by program; it does not look up a restricted linkage identity.

Requirements:

- separate database role/authorization policy from normal assessment read paths;
Expand Down
25 changes: 25 additions & 0 deletions docs/doctoring/standards-and-evidence.md
Original file line number Diff line number Diff line change
Expand Up @@ -74,6 +74,27 @@ Product consequences:
- higher-impact AI changes require an impact/risk assessment proportional to intended use and affected participants;
- model/prompt/provider drift is observable and does not silently alter historical results.

## Research de-identification and restricted linkage

Public research releases must not become joinable to operational assessment
identity. ISO/IEC 20889:2018 classifies pseudonymization and other
privacy-enhancing de-identification techniques; ISO/IEC 27559:2022 provides the
current framework for applying those techniques without treating blanking or
blanket masking as a complete control. The product therefore keeps a restricted
linkage store for authorized research contribution and projects only
program-scoped research identities into public release fixtures.

Product consequences:

- operational participant references and linkage-key versions never appear in
public-release projections;
- one person may hold distinct research identities across programs; those
identities are not collapsed;
- authorized research workflows keep the exact linkage they need instead of a
masked substitute that would break contribution, withdrawal, or audit;
- linkage-key material stays in the key-management system; the database stores
only the key-generation version.

## Temporal and provenance evidence

Longitudinal observations distinguish validity time from source-recorded time,
Expand Down Expand Up @@ -117,8 +138,12 @@ International Organization for Standardization. (2024). *ISO/IEC 27001:2022/Amd

International Organization for Standardization. (2025). *ISO/IEC 42005:2025 Information technology—Artificial intelligence (AI)—AI system impact assessment*. https://www.iso.org/standard/42005

International Organization for Standardization. (2018). *ISO/IEC 20889:2018 Privacy enhancing data de-identification—Terminology and classification of techniques*. https://www.iso.org/standard/69373.html

International Organization for Standardization. (2019). *ISO 8601-1:2019 Date and time—Representations for information interchange—Part 1: Basic rules* (with Amendment 1:2022). https://www.iso.org/standard/70907.html

International Organization for Standardization. (2022). *ISO/IEC 27559:2022 Information security, cybersecurity and privacy protection—Privacy enhancing data de-identification framework*. https://www.iso.org/standard/71619.html

Temoshok, D., Proud-Madruga, D., Choong, Y.-Y., Galluzzo, R., Gupta, S., LaSalle, C., Lefkovitz, N., & Regenscheid, A. (2025). *Digital identity guidelines* (NIST Special Publication 800-63-4). National Institute of Standards and Technology. https://doi.org/10.6028/NIST.SP.800-63-4

World Wide Web Consortium. (2024). *Web Content Accessibility Guidelines (WCAG) 2.2* (W3C Recommendation, 12 December 2024). https://www.w3.org/TR/WCAG22/
Expand Down
163 changes: 163 additions & 0 deletions migrations/0025_research_identity_linkage.sql
Original file line number Diff line number Diff line change
@@ -0,0 +1,163 @@
-- Restricted operational-to-research identity mapping.
-- research_program_ref is stored on both tables as part of the composite
-- foreign key so one operational participant can have only one research
-- identity per program without a non-key transitive dependency.

CREATE TABLE IF NOT EXISTS research_participant (
research_participant_ref TEXT
CONSTRAINT research_participant_ref_not_null NOT NULL
CONSTRAINT research_participant_ref_format_check CHECK (
research_participant_ref = btrim(research_participant_ref)
AND research_participant_ref <> ''
AND NOT (
research_participant_ref ~ '[[:digit:]]'
AND research_participant_ref ~ '^[[:digit:]+,.eE-]+$'
)
),
research_program_ref TEXT
CONSTRAINT research_participant_program_ref_not_null NOT NULL
CONSTRAINT research_participant_program_ref_format_check CHECK (
research_program_ref = btrim(research_program_ref)
AND research_program_ref <> ''
AND research_program_ref <> research_participant_ref
AND NOT (
research_program_ref ~ '[[:digit:]]'
AND research_program_ref ~ '^[[:digit:]+,.eE-]+$'
)
),
recorded_at_unix_ms BIGINT
CONSTRAINT research_participant_recorded_at_unix_not_null NOT NULL
CONSTRAINT research_participant_recorded_at_unix_positive_check CHECK (
recorded_at_unix_ms > 0
),
recorded_at TIMESTAMPTZ
CONSTRAINT research_participant_recorded_at_not_null NOT NULL
DEFAULT clock_timestamp(),
CONSTRAINT research_participant_pkey PRIMARY KEY (research_participant_ref),
CONSTRAINT research_participant_program_identity_unique UNIQUE (
research_participant_ref,
research_program_ref
)
);

CREATE TABLE IF NOT EXISTS research_identity_linkage (
linkage_ref TEXT
CONSTRAINT research_identity_linkage_ref_not_null NOT NULL
CONSTRAINT research_identity_linkage_ref_format_check CHECK (
linkage_ref = btrim(linkage_ref)
AND linkage_ref <> ''
AND NOT (
linkage_ref ~ '[[:digit:]]'
AND linkage_ref ~ '^[[:digit:]+,.eE-]+$'
)
),
participant_ref TEXT
CONSTRAINT research_identity_linkage_participant_ref_not_null NOT NULL
CONSTRAINT research_identity_linkage_participant_ref_format_check CHECK (
participant_ref = btrim(participant_ref)
AND participant_ref <> ''
AND participant_ref <> linkage_ref
AND NOT (
participant_ref ~ '[[:digit:]]'
AND participant_ref ~ '^[[:digit:]+,.eE-]+$'
)
),
research_participant_ref TEXT
CONSTRAINT research_identity_linkage_research_ref_not_null NOT NULL
CONSTRAINT research_identity_linkage_research_ref_format_check CHECK (
research_participant_ref = btrim(research_participant_ref)
AND research_participant_ref <> ''
AND research_participant_ref <> participant_ref
AND NOT (
research_participant_ref ~ '[[:digit:]]'
AND research_participant_ref ~ '^[[:digit:]+,.eE-]+$'
)
),
research_program_ref TEXT
CONSTRAINT research_identity_linkage_program_ref_not_null NOT NULL
CONSTRAINT research_identity_linkage_program_ref_format_check CHECK (
research_program_ref = btrim(research_program_ref)
AND research_program_ref <> ''
AND research_program_ref <> participant_ref
AND research_program_ref <> research_participant_ref
AND NOT (
research_program_ref ~ '[[:digit:]]'
AND research_program_ref ~ '^[[:digit:]+,.eE-]+$'
)
),
linkage_key_version TEXT
CONSTRAINT research_identity_linkage_key_version_not_null NOT NULL
CONSTRAINT research_identity_linkage_key_version_format_check CHECK (
linkage_key_version = btrim(linkage_key_version)
AND linkage_key_version <> ''
AND NOT (
linkage_key_version ~ '[[:digit:]]'
AND linkage_key_version ~ '^[[:digit:]+,.eE-]+$'
)
),
recorded_at_unix_ms BIGINT
CONSTRAINT research_identity_linkage_recorded_at_unix_not_null NOT NULL
CONSTRAINT research_identity_linkage_recorded_at_unix_positive_check CHECK (
recorded_at_unix_ms > 0
),
recorded_at TIMESTAMPTZ
CONSTRAINT research_identity_linkage_recorded_at_not_null NOT NULL
DEFAULT clock_timestamp(),
CONSTRAINT research_identity_linkage_pkey PRIMARY KEY (linkage_ref),
CONSTRAINT research_identity_linkage_participant_program_unique UNIQUE (
participant_ref,
research_program_ref
),
CONSTRAINT research_identity_linkage_research_participant_unique UNIQUE (
research_participant_ref
),
CONSTRAINT research_identity_linkage_participant_program_fk FOREIGN KEY (
research_participant_ref,
research_program_ref
) REFERENCES research_participant (
research_participant_ref,
research_program_ref
)
);

CREATE OR REPLACE VIEW public_research_identity AS
SELECT research_participant_ref, research_program_ref
FROM research_identity_linkage;

CREATE OR REPLACE FUNCTION reject_research_identity_mutation()
RETURNS trigger
LANGUAGE plpgsql
AS $$
BEGIN
RAISE EXCEPTION 'restricted research identity evidence is immutable'
USING ERRCODE = '55000';
END;
$$;

DROP TRIGGER IF EXISTS research_participant_immutable_guard
ON research_participant;
CREATE TRIGGER research_participant_immutable_guard
BEFORE UPDATE OR DELETE ON research_participant
FOR EACH ROW
EXECUTE FUNCTION reject_research_identity_mutation();

DROP TRIGGER IF EXISTS research_participant_truncate_guard
ON research_participant;
CREATE TRIGGER research_participant_truncate_guard
BEFORE TRUNCATE ON research_participant
FOR EACH STATEMENT
EXECUTE FUNCTION reject_research_identity_mutation();

DROP TRIGGER IF EXISTS research_identity_linkage_immutable_guard
ON research_identity_linkage;
CREATE TRIGGER research_identity_linkage_immutable_guard
BEFORE UPDATE OR DELETE ON research_identity_linkage
FOR EACH ROW
EXECUTE FUNCTION reject_research_identity_mutation();

DROP TRIGGER IF EXISTS research_identity_linkage_truncate_guard
ON research_identity_linkage;
CREATE TRIGGER research_identity_linkage_truncate_guard
BEFORE TRUNCATE ON research_identity_linkage
FOR EACH STATEMENT
EXECUTE FUNCTION reject_research_identity_mutation();
Loading
Loading