Skip to content
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
6 changes: 6 additions & 0 deletions CHANGELOG.md
Original file line number Diff line number Diff line change
Expand Up @@ -43,6 +43,12 @@
- Added a provider-neutral `fast_mlsirm.factor_retention` contract that records already-computed candidate counts from supported retention methods, rejects duplicate method evidence, and reports `consensus`, `disagreement`, or `insufficient_evidence` without forcing a winner when methods disagree.
- Added deterministic conservative candidate ranges, a fixed transport ceiling, closed method identities, complete fail-closed tests, and scientific doctoring while keeping factor-retention and structural model-selection arithmetic Rust-owned and separate.

#### Governed post-pilot item-bank lifecycle

- Add a factory-sealed, content-addressed post-pilot item-bank lifecycle that requires exact calibration, item-fit, DIF, information, approval, drift, suspension, and retirement evidence before an item can advance through `piloting`, `calibrated`, `approved`, `active`, `suspended`, reactivated, or terminal `retired` states.
- Preserve policy criticality independently of psychometric discrimination, require use-specific approval, link every successor to the exact previous record fingerprint, and retain only source-text-free evidence identities while leaving numerical calibration and item-bank arithmetic Rust-owned.
- Keep tenancy, authorization, identity mapping, persistence, encryption, retention, deletion, human governance, provider SDKs, new estimators, version bumps, and releases outside this reusable-core slice.

#### Governed RAG scoring request

- Reference-free RAG scoring request adapter with privacy-preserving identity channels and fail-closed rejection of raw system configuration content.
Expand Down
7 changes: 7 additions & 0 deletions docs/changelog.d/609-governed-item-bank-lifecycle.md
Original file line number Diff line number Diff line change
@@ -0,0 +1,7 @@
# Governed post-pilot item-bank lifecycle

## Added

- Add a factory-sealed, content-addressed post-pilot item-bank lifecycle that requires exact calibration, item-fit, DIF, information, approval, drift, suspension, and retirement evidence before an item can advance through `piloting`, `calibrated`, `approved`, `active`, `suspended`, reactivated, or terminal `retired` states.
- Preserve policy criticality independently of psychometric discrimination, require use-specific approval, link every successor to the exact previous record fingerprint, and retain only source-text-free evidence identities while leaving numerical calibration and item-bank arithmetic Rust-owned.
- Keep tenancy, authorization, identity mapping, persistence, encryption, retention, deletion, human governance, provider SDKs, new estimators, version bumps, and releases outside this reusable-core slice.
93 changes: 93 additions & 0 deletions docs/doctoring/governed_item_bank_lifecycle.md
Original file line number Diff line number Diff line change
@@ -0,0 +1,93 @@
# Governed Item-Bank Lifecycle Contract

## Scope

This record governs the first post-pilot item-bank lifecycle slice in `fast-mlsirm`. It defines immutable state transitions and source-text-free evidence references. It does **not** estimate item parameters, establish validity, approve an item automatically, create a hosted workflow, or define a physical database schema.

The existing rubric, generated-candidate audit, and verified pilot-admission contracts remain authoritative before this lifecycle begins:

```text
GeneratedItemCandidate
-> CandidateAuditReport
-> PilotCandidateRecord
-> ItemBankLifecycleRecord(piloting)
```

The lifecycle then permits only:

```text
piloting -> calibrated -> approved -> active
active -> suspended -> active
active -> retired
suspended -> retired
```

Every transition creates a new content-addressed record linked to the exact previous record fingerprint. No operational record is edited in place.

## Evidence and interpretation boundary

A lifecycle evidence reference stores only:

- an evidence kind;
- a descriptive evidence identifier; and
- the complete SHA-256 fingerprint of the exact evidence artifact.

The referenced artifact may contain Rust-backed calibration, fit, DIF, information, linking, exposure, drift, approval, suspension, or retirement evidence. The lifecycle layer does not reproduce or reinterpret those calculations.

The initial calibration gate requires references for calibration, item fit, DIF, and item information. This requirement proves only that the governed evidence classes are present; later slices must validate their schemas, estimator identity, uncertainty, population/design scope, parameter recovery, and decision thresholds.

Approval is use-specific. An item cannot enter `approved` without at least one descriptive approved-use identifier, and an `active` record cannot erase that scope.

## Psychometric discrimination versus policy criticality

`PolicyCriticality` is explicitly separate from discrimination, information, fit, or reliability.

- `ordinary` means no additional operational criticality is asserted by this contract.
- `required` means downstream assembly policy requires the item or criterion for the approved use.
- `conjunctive_gate` means failure cannot be offset by a higher aggregate score.

A low-information safety-critical criterion may therefore remain operationally required. Conversely, a highly discriminating item is not automatically safe, fair, valid, or approved.

## Suspension and retirement

Suspension is reversible but requires both a governance suspension record and evidence of a measured concern such as drift or DIF. Reactivation requires new approval plus new drift evidence. Retirement is terminal for new operational use, while historical evidence remains content-addressed for audit reconstruction.

The contract does not authorize physical deletion, retention exceptions, or erasure behavior. Those controls belong to the downstream persistence and governance system.

## Security and privacy boundary

Canonical lifecycle records contain no source, response, prompt, provider-output, or rejected-content text. Complete content fingerprints support exact provenance but are not signatures, identities, anonymity guarantees, authentication credentials, or authorization decisions.

Downstream systems remain responsible for purpose-bound access, tenant isolation, encryption, key management, retention/export/deletion, legal basis, human approval identity, and tamper-evident audit storage.

Provider/model output cannot create calibration, approval, activation, suspension, retirement, repository, merge, release, or deployment authority by identity alone.

## Verification requirements

The implementation requires deterministic tests for:

- exact verified-pilot provenance;
- direct-construction refusal;
- transition graph enforcement;
- required evidence classes;
- use-specific approval;
- cumulative evidence and previous-record linkage;
- evidence-order invariance;
- conflicting evidence identifiers;
- policy-criticality preservation;
- post-construction mutation replay failure;
- exact package-owned child types;
- terminal retirement; and
- redacted stable errors.

No release claim follows until the unchanged integrated head passes full repository CI, security, coverage, packaging, review, and release-acceptance gates.

## References

American Educational Research Association, American Psychological Association, & National Council on Measurement in Education. (2014). *Standards for educational and psychological testing*. American Educational Research Association.

Gierl, M. J., & Lai, H. (2012). The role of item models in automatic item generation. *International Journal of Testing, 12*(3), 273–298. https://doi.org/10.1080/15305058.2011.635830

Mislevy, R. J., Almond, R. G., & Lukas, J. F. (2003). A brief introduction to evidence-centered design. *ETS Research Report Series, 2003*(1), i–29. https://doi.org/10.1002/j.2333-8504.2003.tb01908.x

Williamson, D. M., Xi, X., & Breyer, F. J. (2012). A framework for evaluation and use of automated scoring. *Educational Measurement: Issues and Practice, 31*(1), 2–13. https://doi.org/10.1111/j.1745-3992.2011.00223.x
155 changes: 155 additions & 0 deletions docs/superpowers/plans/2026-08-09-governed-item-bank-lifecycle.md
Original file line number Diff line number Diff line change
@@ -0,0 +1,155 @@
# Governed Item-Bank Lifecycle Contracts Implementation Plan

> **For agentic workers:** REQUIRED SUB-SKILL: Use superpowers:subagent-driven-development (recommended) or superpowers:executing-plans to implement this plan task-by-task. Steps use checkbox (`- [ ]`) syntax for tracking.

**Goal:** Add an immutable, content-addressed, fail-closed lifecycle contract that bridges an already verified `PilotCandidateRecord` into calibrated, approved, active, suspended, reactivated, and retired item-bank states without adding a database or a new numerical estimator.

**Architecture:** Existing `fast_mlsirm.rubric` audit and pilot records remain the only pre-pilot source of truth. A new `item_bank` module owns post-admission lifecycle records, evidence references, transition rules, and exact provenance replay. Numerical evidence remains an opaque fingerprinted reference to existing/future Rust-backed calibration, fit, DIF, information, linking, exposure, and drift outputs.

**Tech Stack:** Python 3.10+, frozen dataclasses, enums, canonical SHA-256 helpers already present in `fast_mlsirm.rubric.models`, pytest, existing rubric candidate/audit fixtures.

## Global Constraints

- Work only in `ContextualWisdomLab/fast-mlsirm` and preserve standalone installation.
- Reuse `PilotCandidateRecord`; do not create another candidate, rubric, assessment, scoring, or pilot schema.
- Add no ORM, database, provider SDK, network call, numerical estimator, likelihood, optimizer, workflow, dependency, version bump, or hosted-product concern.
- Every identifier uses two-or-more-token lower `snake_case`; exact content identities use complete lower-hex SHA-256 fingerprints.
- Operational versions are immutable; transitions create new records rather than mutating earlier records.
- Psychometric discrimination and policy criticality remain separate.
- Raw source, response, prompt, provider output, rejected content, and secrets are not lifecycle fields or error text.
- Added public code requires complete docstrings and 100% statement/branch coverage.

---

### Task 1: Pin the Public Lifecycle and Transition Contract

**Files:**
- Create: `tests/test_rubric_item_bank_lifecycle.py`
- Create: `docs/doctoring/governed_item_bank_lifecycle.md`

**Interfaces:**
- Consumes: `PilotCandidateRecord`, `CandidateLifecycleState.PILOT`, canonical validators from `fast_mlsirm.rubric.models`.
- Produces: test-locked names `ItemBankLifecycleState`, `ItemBankEvidenceKind`, `ItemBankEvidenceReference`, `ItemBankLifecycleRecord`, `PolicyCriticality`, `ItemBankLifecycleError`, `build_item_bank_pilot_record`, and `transition_item_bank_record`.

- [ ] **Step 1: Write failing public-contract tests**

Cover deterministic pilot admission, exact provenance, immutable fingerprints, direct-construction rejection, required evidence by transition, cumulative evidence, order-invariant identities, criticality preservation, no skipped/backward/no-op transitions, terminal retirement, active-use scope, stale-record mutation detection, exact-type validation, redacted errors, and JSON-compatible serialization.

- [ ] **Step 2: Run focused tests and observe RED**

Run: `pytest -q tests/test_rubric_item_bank_lifecycle.py`

Expected: collection failure because `fast_mlsirm.rubric.item_bank` does not yet exist.

- [ ] **Step 3: Commit the RED contract and doctoring**

Commit only tests, this plan, and method/interpretation doctoring. Keep the PR Draft.

### Task 2: Implement Sealed Evidence and Lifecycle Records

**Files:**
- Create: `python/fast_mlsirm/rubric/item_bank.py`
- Modify: `python/fast_mlsirm/rubric/__init__.py`
- Test: `tests/test_rubric_item_bank_lifecycle.py`

**Interfaces:**
- `ItemBankEvidenceReference(evidence_kind, evidence_id, evidence_fingerprint)` validates one source-text-free evidence identity.
- `build_item_bank_pilot_record(pilot_record, *, item_version, policy_criticality)` creates the only initial `PILOTING` record.
- `transition_item_bank_record(current_record, target_state, *, evidence_references, transition_reason_id, approved_use_ids=())` creates a new record linked to the exact previous record fingerprint.

- [ ] **Step 1: Add the enum and redacted error surface**

States: `piloting`, `calibrated`, `approved`, `active`, `suspended`, `retired`.

Evidence kinds: `calibration`, `item_fit`, `dif`, `item_information`, `linking`, `exposure`, `drift`, `approval`, `suspension`, `retirement`.

Criticality: `ordinary`, `required`, `conjunctive_gate`.

- [ ] **Step 2: Implement immutable evidence references**

Require exact enum values, descriptive IDs, and complete SHA-256 fingerprints. Serialize no raw evidence.

- [ ] **Step 3: Implement factory-sealed lifecycle records**

Store the creation-time record fingerprint, derive a 128-bit public handle, and verify the stored fingerprint against freshly canonicalized content before every transition so `object.__setattr__` mutation fails closed.

- [ ] **Step 4: Implement initial pilot record creation**

Require exact `PilotCandidateRecord`, `CandidateLifecycleState.PILOT`, complete pilot/candidate/audit provenance, semantic item version, and no lifecycle evidence yet.

- [ ] **Step 5: Implement transition graph and evidence gates**

Allowed transitions:

```text
piloting -> calibrated -> approved -> active
active -> suspended -> active
active -> retired
suspended -> retired
```

Required newly supplied evidence:

- `calibrated`: calibration + item-fit + DIF + item-information;
- `approved`: approval;
- `active` from approved: no additional kind, but at least one approved use ID;
- reactivation from suspended: approval + drift;
- `suspended`: suspension plus at least one DIF or drift reference;
- `retired`: retirement.

Reject skipping, reversal, no-op, duplicate/conflicting evidence identity, evidence removal, policy-criticality mutation, provenance mutation, and retirement exit.

- [ ] **Step 6: Run focused GREEN and coverage**

Run:

```bash
pytest -q tests/test_rubric_item_bank_lifecycle.py
coverage run --branch -m pytest -q tests/test_rubric_item_bank_lifecycle.py
coverage report -m --include='python/fast_mlsirm/rubric/item_bank.py'
```

Expected: all tests pass; new module reaches 100% statement and branch coverage.

- [ ] **Step 7: Commit implementation**

Commit the module, exports, and tests without changing dependencies or unrelated files.

### Task 3: Release Documentation and Repository Validation

**Files:**
- Create: `docs/changelog.d/609-governed-item-bank-lifecycle.md`
- Modify: `CHANGELOG.md` using `scripts/render_changelog_fragments.py`
- Modify only if required after canonical PR integration: requirements/architecture traceability in the single canonical documentation PR.

**Interfaces:**
- Produces a documented public contract while leaving physical persistence and hosted workflow ownership downstream.

- [ ] **Step 1: Add authoritative changelog fragment**

Describe lifecycle contracts, evidence gates, immutable transitions, downstream persistence boundary, and explicit absence of new psychometric estimates.

- [ ] **Step 2: Render and verify changelog**

Run:

```bash
python scripts/render_changelog_fragments.py --update CHANGELOG.md
python scripts/render_changelog_fragments.py --check CHANGELOG.md
```

- [ ] **Step 3: Run complete relevant validation**

Run focused tests, full Python suite, Rust workspace and PyO3 crate tests, package/reinstall/release acceptance, explicit GPU no-skip, fuzz, Security Scan, and SAST on one unchanged exact head.

- [ ] **Step 4: Current-head review and merge gate**

Keep Draft until exact-head automated review has no valid unresolved finding, repository approval/branch protection is satisfied, and every required check passes. No release/version bump belongs in this bounded contract slice.

## Self-Review

- The plan reuses the existing rubric/audit/pilot hierarchy and introduces only post-pilot lifecycle records.
- Numerical evidence is referenced, not recomputed in Python.
- No physical DB or hosted workflow is introduced.
- Every transition, evidence gate, mutation boundary, terminal state, and interpretation limitation has a deterministic test target.
- The canonical architecture documentation remains owned by PR #604; this feature PR links to it after that writer is available rather than opening a second architecture authority.
17 changes: 13 additions & 4 deletions python/fast_mlsirm/rubric/__init__.py
Original file line number Diff line number Diff line change
@@ -1,4 +1,4 @@
"""Rubric-centered authoring, governed generation, audit, and pilot admission."""
"""Rubric-centered authoring, generation, audit, pilot, and bank lifecycle."""

from .candidates import BinaryAnswerKey as BinaryAnswerKey
from .candidates import CandidateValidationError as CandidateValidationError
Expand Down Expand Up @@ -46,6 +46,14 @@
from .gtheory_pilot import (
build_gtheory_pi_pilot_design as build_gtheory_pi_pilot_design,
)
from .item_bank import ItemBankEvidenceKind as ItemBankEvidenceKind
from .item_bank import ItemBankEvidenceReference as ItemBankEvidenceReference
from .item_bank import ItemBankLifecycleError as ItemBankLifecycleError
from .item_bank import ItemBankLifecycleRecord as ItemBankLifecycleRecord
from .item_bank import ItemBankLifecycleState as ItemBankLifecycleState
from .item_bank import PolicyCriticality as PolicyCriticality
from .item_bank import build_item_bank_pilot_record as build_item_bank_pilot_record
from .item_bank import transition_item_bank_record as transition_item_bank_record
from .models import BlueprintPlan as BlueprintPlan
from .models import DifficultyBand as DifficultyBand
from .models import EvidenceMode as EvidenceMode
Expand All @@ -69,9 +77,10 @@
from .testlet_pilot import build_testlet_pilot_design as build_testlet_pilot_design
from .verified_pilot import PilotCandidateRecord as PilotCandidateRecord

# Preserve the established star-import contract. Audit and pilot types remain
# explicit subpackage attributes and are imported by their documented names,
# but are not added to ``__all__`` until the next public-surface version bump.
# Preserve the established star-import contract. Audit, pilot, and item-bank
# types remain explicit subpackage attributes and are imported by their
# documented names, but are not added to ``__all__`` until the next
# public-surface version bump.
__all__ = [
"MAX_BLUEPRINTS",
"BinaryAnswerKey",
Expand Down
Loading
Loading