diff --git a/ARCHITECTURE.md b/ARCHITECTURE.md index d28c76b9a..f9b5dcfaa 100644 --- a/ARCHITECTURE.md +++ b/ARCHITECTURE.md @@ -1,157 +1,338 @@ -# Architecture — fast-mlsirm +# fast-mlsirm Architecture -Status: living baseline (commercial-hardening loop) -Audience: implementers, reviewers, buyers performing technical due diligence -Related: `docs/prd_trd_summary.md`, `docs/mmle_marginal_lsirm_design.md`, `AGENTS.md`, `CLAUDE.md`, `docs/doctoring/` +Status: **Authoritative living architecture baseline** +Repository: `ContextualWisdomLab/fast-mlsirm` +Last reviewed: 2026-08-11 -## 1. Purpose +This document describes the current and intended architecture of `fast-mlsirm` +using the concerns and viewpoints of ISO/IEC/IEEE 42010:2022. It is the root +navigation point for product requirements, technical requirements, ADRs, +UML/ERD diagrams, and research-to-code traceability. -`fast-mlsirm` is a psychometrics / educational measurement library for -**Multidimensional Latent Space Item Response Models** (MLSIRM / MLS2PLM) and -related IRT tooling. It is designed to: +## 1. Mission and boundary -1. **Stand alone** as a local Python package with a Rust numeric core. -2. **Compose** as a module inside the ContextualWisdomLab ecosystem - (central `.github` governance, `naruon` product surfaces, optional - `contextual-orchestrator` LLM workflows) without requiring those siblings - at import time. -3. Prefer **paper-backed** estimators and **true-parameter recovery** evidence - over keyword heuristics or demo stubs. - -## 2. Layered system view +`fast-mlsirm` is a reusable, domain-neutral psychometric measurement library +for Multidimensional Latent Space Item Response Models (MLSIRM/MLS2PLM) and +related IRT tooling. It owns scientific and numerical measurement truth and +reusable contracts; it does not own a hosted assessment application's runtime +state. It must stand alone as a local Python package with a Rust numerical core +and compose with ContextualWisdomLab products without importing sibling +repositories at runtime. ```text -┌──────────────────────────────────────────────────────────────────────┐ -│ Presentation / product surfaces (optional) │ -│ HTML diagnostics reports · CLI · enterprise sales readiness scripts │ -├──────────────────────────────────────────────────────────────────────┤ -│ Python orchestration (python/fast_mlsirm/) │ -│ fit API · config validation · simulation · I/O · scoring adapters │ -│ multilevel/longitudinal *contracts* (when merged) · CAT / DIF / etc │ -├──────────────────────────────────────────────────────────────────────┤ -│ PyO3 boundary (crates/fast-mlsirm-py → fast_mlsirm._core) │ -├──────────────────────────────────────────────────────────────────────┤ -│ Rust numeric core (crates/mlsirm-core) ★ hot path │ -│ likelihood + analytic gradients · MMLE / multigroup / multilevel │ -│ GPU marginal path (wgpu) · CPU multithreaded rayons-style work │ -│ recovery-contract tests · fuzz targets │ -└──────────────────────────────────────────────────────────────────────┘ +Downstream products and services + Psychometrics Commons / independent callers / research pipelines + | + versioned public contracts + v ++-------------------------------------------------------------------+ +| fast-mlsirm | +| | +| assessment/rubric/scoring contracts -> validation/orchestration | +| | | | +| v v | +| response/rater evidence -> model selection and recovery evidence | +| | | | +| +---------------> Rust numerical core <+ | +| | | +| PyO3 / typed results | +| | | +| reports / release / audit evidence | ++-------------------------------------------------------------------+ + | + optional explicit integrations, never hidden coupling + v + contextual-orchestrator / TEPP / Gyeot / semantic-data-portal / ... ``` -**Rule:** pure numeric work lives in Rust. Python owns validation, packaging, -report rendering, and user-facing contracts. GPU device selection is explicit; -CPU remains the default portable path. +### Owned bounded context + +- assessment, rubric, scoring, item, rater, response, and calibration + contracts; +- CTT/IRT/MIRT and MLSIRM-family numerical functions; +- testlet, many-facet, factor/model-selection, DIF/invariance, linking, + equating, G-theory, CAT, ATA, rotation, and recovery primitives; +- automated-scoring and LLM-judge validation primitives; +- governed rubric/item-bank lifecycles and deterministic scientific, audit, + report, and release evidence. + +### Explicitly outside the bounded context -## 3. Modular MSA stance (standalone + embeddable) +- product HTTP/admin APIs, participant/session/consent/result persistence; +- identity/federation credentials and model-provider secret stores; +- hosted tenant/database migrations, end-user UI, and deployment control planes. -| Boundary | Standalone behavior | Modular import behavior | -| --- | --- | --- | -| Numeric core | `cargo test --workspace`; maturin build | Same artifacts; no network at fit time | -| Python package | `pip install -e .` + `pytest` | Import `fast_mlsirm` without sibling repos | -| Governance workflows | Repo-local `.github/workflows` | Org reusable workflows from ContextualWisdomLab/.github when present | -| LLM automation | Optional; `NVIDIA_NIM_API_KEY` only | Prefer `contextual-orchestrator` when orchestration is required; do not use `COPILOT_GITHUB_TOKEN` for agent paths | -| Data / PII | Local process; no silent masking that blocks scoring | Access control + audit over irreversible PII redaction for production scoring paths | +`ContextualWisdomLab/psychometrics-commons` is a downstream hosted assessment +product. The dependency direction is downstream -> `fast-mlsirm`; never the reverse. +Optional integrations are explicit host adapters, not hidden imports or +cross-service database access. -## 4. Estimation & population structures +## 2. Architecture drivers -Supported population structures on the MMLE path (see `docs/mmle_marginal_lsirm_design.md`): +1. **Scientific defensibility:** interpretation is tied to identification, fit, + recovery, invariance, uncertainty, and appropriate limitations. +2. **Reproducibility:** versioned content-addressed contracts and immutable + revisions make analyses independently reconstructible. +3. **Performance:** production psychometric arithmetic is Rust-first with + low-context-switch CPU parallelism and parity-verified GPU paths where + material. +4. **Safety:** untrusted data, unsupported estimators, provider failures, and + governance uncertainty fail closed rather than becoming silent success. +5. **Composability:** the package remains independently installable while + exposing stable public contracts to CWL services and third parties. +6. **Explainability:** exact numerical values, provenance, model relation, + convergence, and interpretation boundaries remain machine-readable. +7. **Evolution:** changed rubrics, items, models, and calibration artifacts are + new versions or superseding revisions, never silent semantic mutation. -- **single** — independent persons -- **multigroup** — known group membership (DIF / equating contexts) -- **multilevel** — cluster random intercept \(u_c\) (school / class nesting) +## 3. Architectural views -Buyer gap (in flight): **multiple membership** and **longitudinal / temporal -occasion** *contracts* under `fast_mlsirm.multilevel` (content-addressed, -fail-closed) so atomistic fallacy is not forced by a single-level API. Nested -estimation that consumes those contracts remains paper-scoped in Rust. +### 3.1 Layered system and component view -## 5. Data flow (fit) +See [`docs/uml/component.puml`](docs/uml/component.puml). ```text -responses Y [P×I] ──► validate (Python) - │ - ▼ - FitConfig(estimator, backend, device, …) - │ - ┌───────────────┴────────────────┐ - ▼ ▼ - backend=rust (default) backend=numpy (parity) - fast_mlsirm._core pure-Python objective - │ │ - └──────────── fit result ────────┘ - │ - ▼ - FitResult + diagnostics + optional HTML report +Presentation / product surfaces (optional) + HTML diagnostics reports · CLI · release and buyer evidence + | +Python orchestration (python/fast_mlsirm/) + fit API · validation · simulation · I/O · scoring adapters · contracts + | +PyO3 binding registry (crates/fast-mlsirm-py) + | +Rust mlsirm-core (production numerical hot path) + likelihood/gradients · MMLE/multigroup/multilevel · CAT/ATA + CPU parallel execution · GPU marginal path where supported ``` -Recovery evidence path used in CI and release acceptance: +Pure numeric work lives in Rust. Python owns validation, bounded +materialization, packaging, report rendering, and user-facing contracts. A +NumPy implementation may remain only as an explicit reference/parity backend; +it may not silently become a second production engine. GPU device selection is +explicit and CPU is the portable fallback. + +### 3.2 Contract and data-flow view + +See [`docs/uml/scoring-sequence.puml`](docs/uml/scoring-sequence.puml). + +```text +AssessmentSpec + RubricSpecification + -> ScoringRequest + -> Human / AI / external engine + -> ScoreObservation + -> criterion/rater calibration handoff + -> Rust calibration + -> validation / fairness / adjudication / report +``` + +Every trust boundary rechecks content identity rather than trusting display +handles or cached parent objects. Host adapters own transport, authentication, +tenancy, persistence, and provider credentials. + +### 3.3 Rubric/item-bank and lifecycle view + +See [`docs/uml/item-bank-state.puml`](docs/uml/item-bank-state.puml) and +[`docs/uml/item-lifecycle.puml`](docs/uml/item-lifecycle.puml). ```text -simulate(true θ, a, b, ξ, ζ) → fit/estimate → Procrustes align → RMSE / recovery metrics +approved rubric -> deterministic blueprint -> provider-neutral generation + -> untrusted candidate -> structural/evidence/semantic screening + -> pilot -> Rust calibration -> approval -> active monitoring + -> quarantine/suspension/retirement or a new superseding revision ``` -## 6. Security & compliance posture +Candidate-blind generation is the default for benchmark/evaluation banks. +Candidate-aware discovery requires cross-fitting or an equivalent anti-leakage +design. Published or approved revisions do not mutate in place; correcting a +quarantined item creates a new draft identity and records supersession. + +### 3.4 Model-selection and recovery view + +See [`docs/uml/model-selection-sequence.puml`](docs/uml/model-selection-sequence.puml). + +Model selection is multi-stage: determine factor-retention candidates, +classify the structural relation, use relation-appropriate inference, compare +cluster-aware held-out prediction, inspect residual dependence and +DIF/invariance, inspect scoreability and rotation stability, confirm realistic +true-parameter recovery, and choose the simplest model meeting interpretation +requirements. Bifactor, higher-order, testlet, two-tier, many-facet, and +latent-space structures are not interchangeable names. + +### 3.5 Deployment and composition view + +See [`docs/uml/deployment.puml`](docs/uml/deployment.puml). + +`fast-mlsirm` is delivered as a Python package with a compiled Rust extension +and may be embedded in a CLI, notebook, batch worker, service, or hosted +product. Explicit CWL integrations include Psychometrics Commons, Keyverse, +Gyeot, TEPP, `contextual-orchestrator`, `pg-llm-batch`, +`semantic-data-portal`, and EgressWeave. Each host remains independently +operable; no service accesses another service's application database through +this library. + +## 4. Domain and population model + +See [`docs/erd/domain-model.puml`](docs/erd/domain-model.puml) and the +persistence-neutral [`docs/uml/domain-public-contract.puml`](docs/uml/domain-public-contract.puml) +class view. + +The ERD documents reusable identity and cardinality, not ownership of a hosted +relational database. It includes assessment and rubric versions, item +blueprints/candidates/revisions, scoring requests/observations/results, +engine/rater descriptors, calibration designs/reports, item-bank history, and +model-comparison/recovery evidence. Calibration design inputs are a versioned +many-to-many association with observations. + +The population contract must not force an atomistic analysis when the design is +hierarchical, multiply affiliated, or longitudinal: + +- **single:** independent persons; +- **multigroup:** known group membership for DIF/equating contexts; +- **multilevel:** nested cluster effects such as school/class; +- **multiple membership:** weighted membership in more than one cluster; +- **longitudinal:** explicit person, occasion, time origin, and ordering with + temporal validity rules. -- **SAST / supply chain:** CodeQL, Semgrep, OSV, Trivy, Scorecard, Strix (org). -- **CSAP / SOC 2 awareness:** change control via PR + required checks; secrets - never in tree; agent automation uses dedicated NVIDIA NIM credentials, not - review-bot token schemes. -- **PII:** production scoring must not depend on irreversible masking that - destroys person-level measurement; prefer encryption-at-rest / access - control / purpose limitation documented in operability notes. -- **Input hardening:** bounded JSON / hostile control rejection on public - parsers (see security tests). +The `fast_mlsirm.multilevel` contracts are content-addressed and fail closed. +Nested estimators that consume multiple-membership and longitudinal contracts +remain explicitly paper-scoped until their Rust implementation and recovery +evidence are complete; the presence of a contract is not a claim that the +estimator is already production-ready. -## 7. Testing strategy (architecture-level) +## 5. Numerical and scientific architecture -| Layer | What must be true | +### 5.1 Rust ownership and parity + +The Rust core owns production numerical algorithms. Python performs input +validation, provider/domain orchestration, NumPy marshaling, explicit reference +calculations, and report construction. Parity is checked at the identified +mathematical invariant: raw values where identified, Procrustes-aligned +loadings/coordinates under arbitrary rotation, pairwise distances for latent +geometry, and linked/scaled parameter errors after scale alignment. + +### 5.2 Scientific evidence + +True-parameter recovery is a release mechanism. Bias, RMSE, coverage, +convergence, information/function recovery, and realistic simulation are the +primary accuracy evidence; correlation is supplementary order-preservation +evidence and is not parameter recovery or absolute agreement. + +LLM judges are fallible raters. Model family/version, prompt, order/occasion, +assignment, severity, discrimination, bias, and drift are retained whenever +they affect interpretation. Reference-free evaluation is not truth-free: +faithfulness to supplied context and world correctness require different +evidence regimes. + +## 6. Security, privacy, and compliance posture + +### 6.1 Trust boundaries + +- bound before allocate, read, or materialize; +- use closed schemas, reject duplicate keys and non-finite JSON numbers; +- verify evidence spans against exact source revisions; +- sanitize untrusted exception text and never place secrets or uncontrolled + source text in identifiers/evidence logs; +- enforce least-privilege, immutable action pins where practical, central + SAST/dependency gates, exact-head evidence, and stale-head refusal; +- prohibit self-modifying write-capable PR workflows. + +### 6.2 PII and assurance + +The core library must not require blanket masking that destroys measurement +semantics. Prefer purpose limitation, opaque identifiers, minimal fields, +host-owned encryption and access control, auditable linkage, and separation of +identity-bearing hosted data from reusable measurement artifacts. CSAP and SOC +2 control objectives inform change control, access, logging, supply-chain, and +incident evidence; this document does not claim certification. + +LLM automation uses dedicated `NVIDIA_NIM_API_KEY` credentials when a host +authorizes model execution and does not use `COPILOT_GITHUB_TOKEN` for agent +paths. Existing review-agent key schemes are not repurposed. + +## 7. Quality attributes and test strategy + +The principal ISO/IEC 25010:2023 concerns are functional suitability, +performance efficiency, compatibility, accessibility, reliability, security, +maintainability, flexibility, and safety. The corresponding evidence layers +are: + +| Layer | Required evidence | | --- | --- | -| Rust unit | Equation contracts, gradients, multilevel/MMLE edges | -| Recovery | True-parameter recovery / RMSE sentinels (not hard-coded theater) | -| Python API | Config fail-closed, public fit path, report accessibility | -| GPU | Explicit parity smoke vs CPU (Lavapipe in CI) | -| Fuzz | CSV / report / config Atheris budgets on every PR | -| CI matrix | CPython 3.12 **and** 3.14 full pytest; required check name `python` | +| Rust unit | equations, gradients, backend/device and multilevel edges | +| Recovery | seeded true-parameter recovery with RMSE/bias/coverage sentinels | +| Python API | fail-closed configuration, public fit path, real report behavior | +| GPU | explicit CPU parity smoke, including the CI software device where available | +| Fuzz/security | bounded CSV/report/config inputs and hostile-control rejection | +| CI matrix | complete pytest on CPython 3.12 and 3.14, with the required `python` aggregate | + +Realistic tests must measure the software's scientific property: simulated +truth versus estimates for psychometrics, exact expected semantics for reports +and contracts, and parity across supported Rust/CPU/GPU/reference paths. A +green keyword or import-only test is not sufficient evidence of a production +feature. -## 8. Repository map +## 8. Governance documents and conformance + +The repository map is: ```text crates/mlsirm-core/ Rust formulas, GPU marginal, recovery tests crates/fast-mlsirm-py/ PyO3 bindings -python/fast_mlsirm/ public API + orchestration -tests/ Python contracts + recovery integration -docs/ PRD/TRD, designs, doctoring (APA 7th) -docs/doctoring/ paper/standard citations for shipped behaviors -scripts/ release acceptance, sales readiness, changelog render -.github/workflows/ CI, security, governance agents +python/fast_mlsirm/ public API and orchestration +tests/ contract, security, recovery, and integration tests +docs/ PRD/TRD, ADRs, UML/ERD, doctoring and traceability +scripts/ release acceptance, buyer evidence, changelog rendering +.github/workflows/ CI, security, and governance agents ``` -## 9. Governance documents index +`AGENTS.md` and `CLAUDE.md` define operating rules; `ARCHITECTURE.md` defines +system structure; `CHANGELOG.md` and `docs/changelog.d/` define release notes; +`docs/PRD.md` and `docs/TRD.md` define current requirements; `docs/doctoring/` +contains APA 7th source records; and the threat model, test strategy, +operability, and traceability documents define assurance evidence. -| Artifact | Role | -| --- | --- | -| `AGENTS.md` / `CLAUDE.md` | Agent/developer operating rules | -| `ARCHITECTURE.md` (this file) | System structure | -| `CHANGELOG.md` + `docs/changelog.d/` | Fragment-sourced release notes | -| `docs/prd_trd_summary.md` | Product / technical requirements summary | -| `docs/doctoring/*` | APA 7th citations for model/security claims | -| `docs/20b_product_readiness.md` | Commercial readiness gate narrative | -| Threat model / test strategy | Evolved under `docs/` design notes + CI workflows | +The ADR index is [`docs/adr/README.md`](docs/adr/README.md). Material changes +conform only when they preserve bounded-context ownership, Rust/Python +numerical ownership, version/provenance and migration evidence, identification +and recovery evidence for model claims, fail-closed trust boundaries, and +updated requirements/ADR/test/release traceability. The machine-checkable +documentation contract is maintained in +`tests/test_architecture_documentation_contract.py`. -## 10. References (APA 7th) +## 9. References (APA 7th) + +American Educational Research Association, American Psychological Association, +& National Council on Measurement in Education. (2014). *Standards for +educational and psychological testing*. American Educational Research +Association. Fox, J.-P., & Glas, C. A. W. (2001). Bayesian estimation of a multilevel IRT model. *Psychometrika, 66*(2), 271–288. https://doi.org/10.1007/BF02294839 +International Organization for Standardization. (2023). *ISO/IEC 25010:2023 +Systems and software engineering—Systems and software Quality Requirements and +Evaluation (SQuaRE)—Product quality model*. + +International Organization for Standardization. (2023). *ISO/IEC 42001:2023 +Information technology—Artificial intelligence—Management system*. + +International Organization for Standardization, International Electrotechnical +Commission, & Institute of Electrical and Electronics Engineers. (2022). +*ISO/IEC/IEEE 42010:2022 Software, systems and enterprise—Architecture +description*. + Jeon, M., Jin, I. H., Schweinberger, M., & Baugh, S. (2021). Mapping unobserved item-respondent interactions: A latent space item response model with interaction map. *Psychometrika, 86*(2), 378–403. https://doi.org/10.1007/s11336-021-09762-5 -Kang, I., & Jeon, M. (2025). Multidimensional latent space item response models: -A note on the relativity of conditional dependence. *Psychometrika, 90*(2), -799–826. https://doi.org/10.1017/psy.2025.5 +Kang, I., & Jeon, M. (2025). Multidimensional latent space item response +models: A note on the relativity of conditional dependence. *Psychometrika, +90*(2), 799–826. https://doi.org/10.1017/psy.2025.5 -Molenaar, D., & Jeon, M. (2026). Regularized joint maximum likelihood estimation -of latent space item response models. *Psychometrika, 91*, 335–359. +Molenaar, D., & Jeon, M. (2026). Regularized joint maximum likelihood +estimation of latent space item response models. *Psychometrika, 91*, 335–359. https://doi.org/10.1017/psy.2025.10068 + +World Wide Web Consortium. (2024). *Web Content Accessibility Guidelines +(WCAG) 2.2*. diff --git a/CHANGELOG.md b/CHANGELOG.md index 6247e4805..c0c31bcdb 100644 --- a/CHANGELOG.md +++ b/CHANGELOG.md @@ -298,6 +298,19 @@ - Replaced the NumPy reference/fallback EAP expression `(posterior * nodes[None, :]).sum(axis=1)` with the algebraically equivalent matrix-vector product `posterior @ nodes`. This avoids constructing the explicit posterior-shaped broadcast product; NumPy may use optimized BLAS for matrix multiplication when available, while realized runtime remains dependent on array shape, layout, hardware, and the linked numerical library. +#### Canonical product and architecture documentation baseline + +- Replaced the stale MVP-only PRD/TRD authority with canonical `docs/PRD.md` and `docs/TRD.md` requirements covering the current measurement, scoring, rubric/item-generation, model-selection, scientific-evidence, interoperability, security, lifecycle, and release boundaries. +- Added root `ARCHITECTURE.md`, a status-bearing ADR corpus, reviewable PlantUML component/sequence/state/deployment views, a logical reusable-domain ERD, and requirements/research traceability matrices. +- Added a canonical documentation authority index, explicit implementation-maturity/completeness matrix, and machine-checkable documentation contract so missing or stale PRD/TRD/ADR/UML/ERD/traceability/security artifacts remain visible release-maintenance debt rather than silently drifting. +- Added a reusable-core threat model covering provider/JSON replay, native/PyO3 input boundaries, resource and non-finite numerical failures, GPU evidence spoofing, supply-chain/self-modifying CI, credential separation, benchmark contamination, privacy/purpose limitation, and scientific-interpretation abuse while leaving hosted HTTP/session/tenant/database threats downstream. +- Added durable ADRs for converging future Rust-backed features on one canonical PyO3/public-export registry and for preserving legitimate sensitive-data linkage through purpose limitation and minimization rather than blanket masking that changes the measurement design. +- Extended requirements traceability with the conversation-wide invariants that human/LLM judges are fallible raters, correlation is not parameter recovery/absolute agreement, latent-space interaction follows substantive dimension/testlet/facet diagnosis, reference-free is not truth-free, and psychometric discrimination is not business or safety criticality. +- Explicitly deprecated the original narrow `docs/prd_trd_summary.md` as an authoritative requirements source while retaining its historical MLS2PLM MVP context. +- Defined the `fast-mlsirm-cjson-v1` fingerprint preimage, SHA-256 binding, null/ordering/Unicode/number rules, and cross-language normative vector instead of leaving canonical serialization as an interoperability assumption. +- Added the persistence-neutral `docs/uml/domain-public-contract.puml` view, indexed every UML source including the compatibility alias, modeled versioned calibration-design inputs as a many-to-many association, and made corrected quarantined items new immutable revisions. +- Added complete APA 7 research records and scope summaries for LLM-RUBRIC, AutoNuggetizer/TREC RAG, EvalGen, the 2025 AutoNuggetizer follow-up and 2026 reflective rubric research, plus NIST AI RMF governance inputs with explicit non-certification language. + #### Release cut 0.7.0 - Project version is bumped to 0.7.0 in `pyproject.toml`, @@ -317,6 +330,12 @@ ### Fixed +#### Subgroup validation evidence fails closed + +- Automated-scoring subgroup SMD gates reject requested subgroups with fewer + than two paired cases or zero human variance instead of silently skipping + them and reporting a vacuous pass. + #### Serving bundle export requires Rust core - `export_serving_bundle` fails closed when the compiled Rust core is unavailable @@ -326,6 +345,11 @@ - Restricted the public `FitConfig.estimator` vocabulary to the implemented `jmle` and `mmle` fitting paths, so unsupported `em` and `bayes` requests fail during configuration validation instead of entering a fitting path that later raises `NotImplementedError`. +#### Strict JSON artifact interoperability + +- Governed JSON artifact writers now reject `NaN`, positive infinity, and negative infinity instead of emitting Python's non-standard JSON numeric extensions, preserving RFC 8259 interoperability and atomic publication failure. +- Non-finite serialization errors use a bounded package-owned message without reflecting the rejected artifact payload. + #### Node-rule fail-closed validation - Public polytomous and 2PL fitters reject non-string integration-rule controls diff --git a/docs/PRD.md b/docs/PRD.md new file mode 100644 index 000000000..cb62e08b5 --- /dev/null +++ b/docs/PRD.md @@ -0,0 +1,281 @@ +# fast-mlsirm Product Requirements Document + +Status: **Authoritative product requirements baseline** +Repository: `ContextualWisdomLab/fast-mlsirm` +Last reviewed: 2026-08-09 + +## 1. Product definition + +`fast-mlsirm` is the reusable, domain-neutral measurement and psychometric computation layer for ContextualWisdomLab and for independent Python/Rust consumers. It provides governed measurement contracts, psychometric estimation and diagnostics, automated-scoring calibration/validation primitives, item/rubric lifecycle primitives, simulation/recovery evidence, and deterministic reports. + +It is **not** the hosted Psychometrics Commons application. HTTP APIs, participant/session/consent/result lifecycle, product databases and migrations, UI, tenancy, deployment composition, and research-release orchestration belong to `ContextualWisdomLab/psychometrics-commons` or another owning bounded context. `fast-mlsirm` must remain independently installable and must not depend on hosted product code. + +## 2. Product outcome + +The product shall make measurement decisions defensible from the moment a construct/rubric is specified through scoring, psychometric calibration, model selection, validation, reporting, and governed item/model lifecycle. + +The product is successful when a technical team can answer, with reproducible evidence: + +1. **What construct and scoring contract was used?** +2. **Which exact item, rubric, response, rater/engine, task revision, and software artifact produced each observation?** +3. **Which psychometric model was fit, under which identification assumptions?** +4. **Are scores recoverable, reliable enough for their stated interpretation, invariant enough for the intended comparison, and free of known unresolved model-fit or fairness blockers?** +5. **How much uncertainty remains, and what evidence would change the decision?** +6. **Can the exact analysis be rebuilt and independently checked from immutable provenance?** + +## 3. Primary users + +### PRD-PER-001 Psychometric researcher + +Needs simulation, true-parameter recovery, model comparison, diagnostics, and reproducible research-grade outputs without relying on opaque legacy package behavior. + +### PRD-PER-002 Assessment engineer + +Needs versioned assessment/rubric/scoring contracts, calibrated item/rater data, linking, CAT/ATA, DIF/invariance, and governed release evidence. + +### PRD-PER-003 AI evaluation engineer + +Needs to treat LLM judges as fallible raters rather than truth, calibrate judge severity/bias/drift, build evidence-grounded rubrics/items, and compare RAG/LLM systems on measurement-aware scales. + +### PRD-PER-004 Automated-scoring validation lead + +Needs human/AI/external scorer observations, many-facet calibration, agreement beyond correlation, range-use evidence, fairness/DIF, adjudication routing, and audit reports. + +### PRD-PER-005 Downstream product/service team + +Needs stable, provider-neutral, content-addressed contracts and Rust-backed outputs that can be composed into hosted applications without importing product-specific persistence or UI assumptions. + +## 4. Product principles + +### PRD-PRN-001 Measurement before aggregation + +Raw scores, RAGAS values, LLM judgments, or human ratings are observations, not truth. The system shall preserve the facets and conditions needed to model measurement error before deriving consequential summaries. + +### PRD-PRN-002 Rust owns production psychometric arithmetic + +Likelihoods, gradients, Hessians, optimization, information, psychometric scoring/ranking, and other production mathematical kernels are Rust-owned. Python may orchestrate, validate, marshal, report, and retain transparent reference implementations for parity/fallback where explicitly governed. + +### PRD-PRN-003 Correlation is not accuracy + +Validation shall not treat correlation alone as proof of parameter recovery, agreement, calibration, fairness, or validity. Where true parameters are known, bias, MAE/RMSE, interval/SE coverage, convergence, response/information recovery, and backend parity are first-class evidence. + +### PRD-PRN-004 Hierarchy and time are first-class + +Scientifically relevant designs shall support or explicitly model multilevel, cross-classified, multiple-membership, testlet/local-dependence, repeated-measurement, temporal, and drift structure rather than flattening observations into an atomistic single level. + +### PRD-PRN-005 Fail closed on unidentified interpretation + +Unknown model relations, disconnected designs, incomplete provenance, unsupported contract major versions, non-finite results, scoreability failures, or missing required evidence shall not silently produce a preferred model, operational score, or release-ready result. + +### PRD-PRN-006 Content-addressed reproducibility + +Published/reusable measurement artifacts shall be immutable or superseded by new versions, with deterministic fingerprints for the exact construct/rubric/task/model/provenance content relevant to interpretation. + +### PRD-PRN-007 Modular MSA compatibility without hidden coupling + +The package shall expose stable versioned interfaces usable independently and by CWL services. It shall not require another service's database, ORM, HTTP type, UI component, deployment manifest, or ambient credential. + +## 5. Current product capabilities + +The following are implemented on protected `main` as of this baseline unless explicitly marked otherwise: + +- MLS2PLM-family binary simulation and point estimation, including `MIRT`, `MLSRM`, `MLS2PLM`, `ULSRM`, and `ULS2PLM` constraints. +- Rust-backed likelihood/gradient/distance kernels through PyO3/maturin, with NumPy reference/fallback paths and parity tests. +- Missing-response handling, optimization, recovery, fit and dimensionality diagnostics. +- Fixed-item calibration/linking, CAT item-information selection, ATA form assembly. +- Response-process diagnostics, model-fit summaries, multigroup/multilevel-context summaries exposed by current APIs. +- Rubric-centered schemas and deterministic bounded item-blueprint/generation-contract compilation. +- Governed assessment/scoring contracts and provenance-aware automated essay and enterprise-issue adapters added through the v0.7.0-era scoring work. +- Rust-backed criterion many-facet calibration/reporting paths for governed scoring workflows. +- Standalone accessible HTML audit/report artifacts. +- Release, benchmark, procurement, buyer-packet, PR-queue and provenance evidence builders. + +Open PRs and issues may contain additional capabilities. They are **not** considered accepted product behavior until protected integration. + +## 6. Functional requirements + +### 6.1 Canonical measurement contracts + +**PRD-FR-001** The package shall own one canonical `AssessmentSpec` family and one canonical `RubricSpecification` family for reusable assessment domains. Duplicate parallel schemas are prohibited. + +**PRD-FR-002** Assessment/rubric/scoring contracts shall carry deterministic content identity, schema/version identity, construct identity, scoring/calibration/validation policy references, and bounded metadata. + +**PRD-FR-003** Scoring observations shall distinguish `scored`, `abstained`, `failed`, and `excluded` semantics; terminal states shall not be coerced to a low score. + +**PRD-FR-004** Rater/engine identity, task identity and task revision, response identity/revision, assessment/rubric identity, and criterion identity shall remain separately auditable. + +### 6.2 Rubric-to-item lifecycle + +**PRD-FR-010** The package shall support the lifecycle: + +`Rubric -> Blueprint -> Generation Contract -> Candidate -> Screening -> Pilot -> Calibration -> Approved Item Bank -> Monitoring -> Revision/Retirement`. + +**PRD-FR-011** Benchmark generation shall support candidate-blind evidence-grounded criteria. Candidate-aware criterion discovery, when implemented, shall be isolated through cross-fitting or a separate training/diagnostic bank. + +**PRD-FR-012** Canonical criteria shall support atomic, evidence-grounded judgments where appropriate; holistic score descriptions may be compiled compatibility views rather than the sole source of truth. + +**PRD-FR-013** Generated provider output shall be treated as untrusted and shall be bounded, closed-schema, replay-resistant, provenance-bound, and checked for duplicate keys, non-finite numbers, answer-key integrity, evidence-span integrity, and response-format consistency before psychometric use. + +**PRD-FR-014** Semantic screening shall be able to represent answerability, construct alignment, ambiguity, distractor quality, redundancy, leakage, evidence entailment/support, and content/bias review without conflating them with structural JSON validity. + +### 6.3 Automated scoring + +**PRD-FR-020** Human, LLM, external-model, deterministic, and future scorer implementations shall map to a shared scoring-engine/rater observation contract. + +**PRD-FR-021** The automated-scoring validation path shall support ordinal many-facet calibration, rater severity, criterion-specific bias, range-use diagnostics, drift evidence, agreement, DIF/fairness evidence, and human-review/adjudication routing. + +**PRD-FR-022** Human ratings are measurements with error and shall not automatically be treated as error-free true scores. + +**PRD-FR-023** Generated feedback is not required for the measurement core. When a downstream system adds feedback, feedback must not silently alter the numerical score or provenance-bound psychometric evidence. + +### 6.4 Reference-free RAG and LLM-as-a-Judge measurement + +**PRD-FR-030** The product shall distinguish groundedness/faithfulness, world correctness, retrieval relevance/coverage, answer utility/completeness, robustness, abstention/calibration, and citation/evidence attribution where the evidence regime permits them. + +**PRD-FR-031** LLM judges shall be representable as rater facets with model/provider/prompt/occasion/version provenance. + +**PRD-FR-032** Reference-free evaluation shall not claim world correctness or absolute recall when the available evidence supports only context-grounded or pooled-corpus claims. + +**PRD-FR-033** Query-derived probes from the same question/response shall be able to retain query/testlet identity to avoid pseudo-replication. + +### 6.5 Measurement models and model selection + +**PRD-FR-040** The model portfolio shall distinguish unidimensional, correlated multidimensional, bifactor, higher-order, testlet, two-tier, multifaceted, and latent-space structures by actual parameter constraints rather than names. + +**PRD-FR-041** Model relation shall be classified as regular nested, boundary/singular nested, nonlinear-constraint nested, strictly non-nested, overlapping/indistinguishable, or unknown as evidence supports. + +**PRD-FR-042** Non-nested preference shall require formal distinguishability evidence before selection. Boundary models shall use boundary-aware/bootstrap procedures when ordinary chi-square likelihood-ratio theory is invalid. + +**PRD-FR-043** Model selection shall combine relation-appropriate inferential comparison with held-out cluster-aware predictive evidence, residual dependence, scoreability, DIF/invariance, stability, and true-structure recovery. + +**PRD-FR-044** Bifactor model fit shall not imply general or specific-score interpretability. Scoreability evidence shall be reported separately. + +### 6.6 Factor retention and rotation + +**PRD-FR-050** Factor retention shall be a separate decision from structural model selection. + +**PRD-FR-051** Exploratory rotation shall not expose a universal-best criterion claim. Rotation solutions shall retain criterion, optimizer, start/basin, convergence/stationarity, sign/permutation alignment, and stability evidence. + +**PRD-FR-052** Criterion selection shall compare candidates using criterion-neutral recovery/stability/theory evidence rather than raw objective values from incomparable criteria. + +### 6.7 Multilevel, multiple-membership, and longitudinal measurement + +**PRD-FR-060** Reusable contracts shall represent explicit context dimensions, context identities, membership weights, repeated occasions, and temporal state specifications without inferring random-effect families from labels. + +**PRD-FR-061** Multiple-membership weights and temporal ordering shall be provenance-bound. Elapsed-time effects shall not be claimed unless the fitted model actually parameterizes elapsed-time transitions. + +**PRD-FR-062** Future Rust estimators for these contracts shall establish identification and true-parameter recovery before release as production estimators. + +### 6.8 Reporting and evidence + +**PRD-FR-070** Reports shall separate exact machine-readable values from human-readable summaries and preserve provenance needed to reconstruct the analysis. + +**PRD-FR-071** HTML outputs shall be script-free by default where feasible, use restrictive content-security policy where relevant, and meet the repository's accessibility contracts. + +**PRD-FR-072** Reliability, validity, fairness, fit, convergence, and deployment readiness shall remain distinct concepts in report language. + +### 6.9 Lifecycle and release + +**PRD-FR-080** Measurement/rubric/item/model artifacts shall have explicit lifecycle states where lifecycle management is exposed, e.g. `draft -> audited/screened -> pilot -> calibrated -> approved -> active -> suspended/quarantined -> retired`. + +**PRD-FR-081** A release shall be created only from an exact integrated protected head with required CI, security, coverage, packaging, provenance/SBOM, reproducibility, compatibility, review, rollback/migration, and release-acceptance evidence. + +**PRD-FR-082** Release notes and `CHANGELOG.md` shall match the released artifact and authoritative changelog-fragment workflow. + +## 7. Quality requirements + +The quality model follows the concerns of ISO/IEC 25010:2023 while adding psychometric/scientific evidence requirements. + +### PRD-NFR-001 Functional suitability + +Public contracts and numerical outputs shall match documented semantics; unsupported interpretations fail closed. + +### PRD-NFR-002 Performance efficiency + +Computationally material numerical kernels shall use Rust and low-context-switch CPU parallelism; GPU paths shall be added when beneficial and must have parity evidence. Resource use shall be explicitly bounded for caller-controlled dimensions and expensive fallback workspaces. + +### PRD-NFR-003 Compatibility/interoperability + +The wheel/package shall work independently. Cross-service composition shall use versioned contracts/artifacts rather than database coupling. + +### PRD-NFR-004 Usability/accessibility + +CLI, Python API, documentation, and standalone reports shall expose understandable errors and evidence. HTML/report surfaces target WCAG 2.2 AA-relevant semantics without claiming full conformance absent audit. + +### PRD-NFR-005 Reliability + +Long-running studies/subprocesses shall have operation-appropriate deadlines, fail-closed timeout evidence, deterministic seeds where relevant, idempotent/reproducible artifact generation, and no stale evidence reuse. + +### PRD-NFR-006 Security + +Untrusted data/provider output shall be bounded and validated. CI uses least privilege, immutable action pins where practical, supply-chain/security scanning, and no secrets in evidence artifacts. Security failures shall not be converted to success merely to unblock a merge. + +### PRD-NFR-007 Maintainability + +Public Python/Rust docs shall be beginner-readable. Owned production code targets exact 100% statement/branch coverage plus line/function coverage where tooling exposes it. Architecture/ADR/traceability documentation shall change with governing contracts. + +### PRD-NFR-008 Scientific validity + +Parameter recovery, uncertainty coverage, model identification, model relation, scoreability, DIF/invariance, and local-dependence assumptions shall be tested and reported within the scope supported by evidence. + +## 8. Data and privacy requirements + +`fast-mlsirm` is not the system of record for participant identity or hosted operational data. + +- Public/reusable artifacts should use opaque nonnumeric identifiers where durable identity is required. +- Provider text, response text, source content, and PII shall not be duplicated into audit artifacts unless explicitly required by a reusable contract and appropriately bounded. +- Prefer purpose limitation, authorization, minimization, encryption by the host, restricted linkage, hashes/fingerprints, and retention control rather than blanket PII masking that destroys measurement utility. +- Hosted storage, data residency, DSAR, consent, and tenant isolation are downstream product responsibilities unless a reusable library primitive explicitly owns a local artifact format. + +## 9. Non-goals + +The following are not product responsibilities unless a future approved ADR changes the boundary: + +- hosted participant/session/consent/result APIs; +- hosted tenant databases or migrations; +- SSO/SCIM/identity credentials; +- hosted UI/workbench deployment; +- clinical diagnosis or treatment recommendation; +- employment/admission/credit/insurance/legal automated decision authority; +- unconditional claims that one factor rotation, model, LLM judge, or rubric is universally optimal; +- treating `kaefa`, `aFIPC`, or `nonnest2` as runtime/build/release dependencies or sole scientific oracles; +- claiming SOC 2, CSAP, ISO, WCAG, or regulated-device certification solely from repository controls. + +## 10. Product roadmap by bounded capability + +Priority is evidence-driven; open PRs and protected-main state override this ordering when dependencies demand it. + +1. Drain/merge current PR queue safely and remove obsolete duplicates. +2. Complete canonical documentation/ADR/traceability baseline. +3. Complete multilevel/contextual/longitudinal reusable contracts and move estimators to Rust only after recovery design is accepted. +4. Complete rubric-generation trust boundary and semantic screening. +5. Complete artificial-crowd calibration and governed item-bank lifecycle. +6. Complete automated-scoring range/discrimination/drift/fairness evidence. +7. Complete relation-safe factor retention and structural model selection. +8. Complete bifactor scoreability and adaptive rotation product APIs with primary-source traceability. +9. Add buyer-facing workbench only when public domain contracts stabilize; host UI remains downstream. +10. Mature release/provenance/performance/security evidence for enterprise procurement. + +## 11. Acceptance and traceability + +Each material requirement must map to at least one of: + +- a public API or schema; +- a Rust/PyO3/Python implementation module; +- a realistic test/recovery study; +- an ADR explaining the decision and alternatives; +- a release/readiness gate. + +The mapping is maintained under `docs/traceability/`. + +## References + +American Educational Research Association, American Psychological Association, & National Council on Measurement in Education. (2014). *Standards for educational and psychological testing*. American Educational Research Association. + +International Organization for Standardization. (2023). *ISO/IEC 25010:2023 Systems and software engineering—Systems and software Quality Requirements and Evaluation (SQuaRE)—Product quality model*. + +International Organization for Standardization. (2023). *ISO/IEC 42001:2023 Information technology—Artificial intelligence—Management system*. + +International Organization for Standardization, International Electrotechnical Commission, & Institute of Electrical and Electronics Engineers. (2018). *ISO/IEC/IEEE 29148:2018 Systems and software engineering—Life cycle processes—Requirements engineering*. + +World Wide Web Consortium. (2024). *Web Content Accessibility Guidelines (WCAG) 2.2*. diff --git a/docs/README.md b/docs/README.md new file mode 100644 index 000000000..c872b2463 --- /dev/null +++ b/docs/README.md @@ -0,0 +1,51 @@ +# fast-mlsirm documentation authority + +This index distinguishes governing product/architecture documents from implementation history and method-local evidence. + +## Canonical architecture package + +| Document | Governing purpose | +|---|---| +| [`../ARCHITECTURE.md`](../ARCHITECTURE.md) | System of interest, bounded contexts, dependency direction, component/data/deployment/scientific views | +| [`PRD.md`](PRD.md) | Product requirements, users, workflows, non-goals, acceptance boundaries | +| [`TRD.md`](TRD.md) | Technical realization, numerical/runtime/security/resource/release requirements | +| [`adr/README.md`](adr/README.md) | Durable architecture/scientific decision log and status history | +| [`standards_watch.md`](standards_watch.md) | Published governing standards versus draft/revision watch items; no certification shortcut | +| [`verification_validation_plan.md`](verification_validation_plan.md) | Software, numerical, scientific, scoring/RAG, recovery, security, packaging and exact-artifact V&V evidence | +| [`uml/README.md`](uml/README.md) | PlantUML component, sequence, lifecycle, model-selection, deployment and reusable domain/public-contract views | +| [`uml/domain-public-contract.puml`](uml/domain-public-contract.puml) | Persistence-neutral reusable domain/public-contract classes and construction rules | +| [`erd/domain-model.puml`](erd/domain-model.puml) | Logical reusable-domain artifact relationships; **not** a hosted ORM schema | +| [`traceability/requirements-matrix.md`](traceability/requirements-matrix.md) | PRD/TRD/ADR -> protected-main implementation/evidence maturity | +| [`traceability/research-basis.md`](traceability/research-basis.md) | Scientific/standards evidence and APA 7 reference mapping | +| [`documentation_coverage.md`](documentation_coverage.md) | Documentation completeness states, remaining P0/P1/P2 gaps and maintenance gate | +| [`security/threat-model.md`](security/threat-model.md) | Reusable-core trust/threat model; hosted product threats remain downstream | +| [`doctoring/`](doctoring/) | Method/security/interoperability evidence and conservative implementation boundaries | +| [`../AGENTS.md`](../AGENTS.md), [`../CLAUDE.md`](../CLAUDE.md) | Agent/developer operating rules aligned to this architecture | +| [`../CHANGELOG.md`](../CHANGELOG.md) | User-visible released/unreleased change history | + +`prd_trd_summary.md` is historical and must not compete with `PRD.md` and `TRD.md` as a requirements source. + +## Authority and status + +1. Protected-main source/tests define executable behavior. +2. Accepted ADRs define governing architecture/scientific decisions. +3. PRD/TRD define product/technical requirements and non-claims. +4. `ARCHITECTURE.md`, UML/ERD and the threat model define coherent system views. +5. The standards watch defines which published editions may govern claims and which drafts/revisions are only monitored. +6. The V&V plan defines what evidence is needed to verify software behavior and validate scientific/product interpretations. +7. Method-specific doctoring and primary literature justify local scientific/interoperability details. +8. Proposed ADRs, open PRs/issues and plans describe future/active work and are not released capability merely because they exist. + +A conversation or PR body is discovery evidence until the durable decision is captured in the documents above. + +## Implementation history + +`docs/superpowers/specs/` and `docs/superpowers/plans/` preserve bounded design/implementation history. They do not automatically remain normative after implementation. If a plan creates a durable product/architecture/scientific decision, update the canonical PRD/TRD/ADR/traceability set. + +## Completeness gate + +A material change is incomplete if it creates a contradiction among code, accepted ADRs, PRD/TRD, architecture diagrams, security/threat model, standards status, V&V evidence, traceability, doctoring or release evidence. The documentation-contract test and `documentation_coverage.md` make these gaps visible; a missing or stale canonical artifact is release-maintenance debt rather than harmless prose drift. + +## Cross-repository boundary + +`fast-mlsirm` is the standalone reusable measurement/psychometric core. `ContextualWisdomLab/psychometrics-commons` or another owning downstream service is responsible for hosted HTTP/session/consent/tenant/RBAC/UI/database/deployment lifecycle. Architecture documents here may define interoperable reusable artifacts and versioned handoffs without creating a shared application database or reverse product dependency. diff --git a/docs/TRD.md b/docs/TRD.md new file mode 100644 index 000000000..2f3dffe67 --- /dev/null +++ b/docs/TRD.md @@ -0,0 +1,257 @@ +# fast-mlsirm Technical Requirements Document + +Status: **Authoritative technical requirements baseline** +Repository: `ContextualWisdomLab/fast-mlsirm` +Last reviewed: 2026-08-09 + +## 1. Purpose + +This TRD turns the product requirements in [`PRD.md`](PRD.md) into implementation, interface, evidence, and release constraints for the reusable `fast-mlsirm` package. + +The document follows ISO/IEC/IEEE 29148 requirements-engineering principles and the architecture concerns of ISO/IEC/IEEE 42010. It does not replace method-specific design documents or primary methodological literature. + +## 2. System boundary + +### TRD-BOUND-001 Owned by fast-mlsirm + +- versioned domain-neutral `AssessmentSpec`, `RubricSpecification`, scoring/observation and calibration contracts; +- psychometric model configuration and result contracts; +- simulation, recovery, fit and model-selection evidence; +- CTT/IRT/MIRT, MLSIRM/MLS2PLM, testlet, many-facet and related reusable numerical capabilities; +- factor retention, bifactor scoreability, rotation and relation-safe model comparison where implemented; +- DIF/invariance/fairness, linking/equating, CAT/ATA and G-theory primitives; +- automated-scoring and LLM-judge validation primitives; +- governed rubric/item-bank primitives and reusable reports; +- package/release/provenance evidence. + +### TRD-BOUND-002 Not owned by fast-mlsirm + +- hosted HTTP/admin APIs; +- participant/session/consent/result lifecycle; +- hosted product persistence/migrations and multi-tenant database administration; +- end-user authentication, SSO/SCIM/passkeys; +- UI deployment and product navigation; +- provider credential stores; +- operational research-release catalogs. + +These belong to downstream bounded contexts such as `ContextualWisdomLab/psychometrics-commons`, Keyverse, semantic-data-portal, contextual-orchestrator, or another explicitly versioned service. + +## 3. Repository architecture + +```text +python/fast_mlsirm/ Public Python API, validation/orchestration, reports, + transparent reference/fallback paths +crates/mlsirm-core/ Rust psychometric/numerical source of truth +crates/fast-mlsirm-py/ PyO3 bindings and Python transport +scripts/ Release, evidence, governance, study runners +fuzz/ Python/Rust fuzz targets and corpora +tests/ Public contract, regression, delegation, parity tests +docs/ PRD/TRD, method docs, ADRs, doctoring, diagrams, + traceability, release evidence guidance +``` + +## 4. Technical requirements + +### 4.1 Numerical ownership and precision + +**TRD-NUM-001** Production likelihoods, gradients, Hessians/information matrices, psychometric scoring/ranking, optimization, item/factor information, and other mathematically material kernels shall be implemented in Rust before being considered production-owned. + +**TRD-NUM-002** Python reference implementations may exist for numerical parity, diagnostics, controlled fallback, or research transparency, but shall not silently diverge into a second production formula. + +**TRD-NUM-003** Every formula-contract change shall update parameterization, likelihood, analytic derivatives, simulation, recovery, Python/Rust parity, public documentation, and method citations as one coherent model-design change. + +**TRD-NUM-004** The existing simple-structure MLS2PLM specialization is preserved unless a dedicated full-vector discrimination model path is introduced. Local algebra/performance changes shall not silently reinterpret the model. + +**TRD-NUM-005** Caller-controlled dimensions, array sizes, item counts, bootstrap counts, JSON sizes, subprocess workloads, and fallback workspaces shall be bounded before allocation or expensive execution. + +**TRD-NUM-006** Non-finite caller input and non-finite intermediate/result states shall fail with bounded, non-secret-bearing errors when the mathematical contract requires finiteness. + +### 4.2 CPU and GPU execution + +**TRD-PERF-001** Computationally material Rust workloads shall use coarse-grained or otherwise low-context-switch CPU parallelism when concurrency improves throughput without violating determinism or memory ceilings. + +**TRD-PERF-002** GPU is a Rust device path, not an independent public model/backend. GPU implementations shall have CPU/Rust parity evidence at the level appropriate to the algorithm, including invariant-aware comparisons for non-identifiable coordinates. + +**TRD-PERF-003** A GPU fallback shall not be presented as a GPU success. Tests requiring GPU execution shall prove a real adapter/kernel path ran and did not skip when the acceptance contract says GPU is required. + +**TRD-PERF-004** f32 GPU arithmetic shall not be treated as evidence-equivalent to f64 CPU arithmetic without explicit error tolerances and recovery/parity studies. + +### 4.3 PyO3/public API + +**TRD-API-001** PyO3 bindings shall expose typed, bounded domain results rather than unstable error-string parsing. + +**TRD-API-002** Secondary extension initializers or feature bindings shall be registered through one canonical Rust/Python export structure so independent feature PRs can coexist without overwriting module initialization. + +**TRD-API-003** Public Python APIs shall validate shape, type, bounds, identifiers, schema versions and obvious semantic invariants before delegating numerical work to Rust, while avoiding duplicate numerical computation. + +**TRD-API-004** Public error codes/paths are part of the contract. Caller-controlled text and provider exceptions shall not be echoed into durable error evidence unless explicitly safe. + +**TRD-API-005** Public durable identifiers shall be descriptive opaque strings where identity must survive serialization; numeric database-style IDs shall not be introduced into reusable public contracts solely for implementation convenience. + +### 4.4 Canonical serialization and provenance + +**TRD-PROV-001** Reusable immutable artifacts shall have deterministic canonical serialization and full cryptographic fingerprinting where content identity matters. + +**TRD-PROV-002** Human-readable handles may be shortened representations but shall never substitute for the authoritative full fingerprint in replay or integrity decisions. + +**TRD-PROV-003** Schema version and semantic/domain revision shall remain separate dimensions. Changing a rubric/task/calibration revision shall change content identity even if the wire schema is unchanged. + +**TRD-PROV-004** Aggregate objects shall replay/verify package-owned child artifacts at trust boundaries rather than trusting cached display identifiers. + +**TRD-PROV-005** Exact task revisions, response-content revisions, assessment/rubric identities, engine/rater identities and source/evidence revisions shall be preserved separately where their conflation changes the interpretation. + +### 4.5 Rubric and generated-item trust boundary + +**TRD-RUB-001** `RubricSpecification` is the canonical rubric source; scoring or domain adapters shall reference it rather than define competing rubric schemas. + +**TRD-RUB-002** Blueprint compilation shall be deterministic, bounded and content-addressed. + +**TRD-RUB-003** Generation contracts shall use closed response-format-specific schemas with bounded text/collections and typed answer-key semantics. + +**TRD-RUB-004** Provider output shall be parsed as untrusted input. The parser shall reject duplicate object keys, `NaN`/infinities, oversized payloads, unknown/missing fields, provenance mismatches, invalid score order/coverage, option/answer-key inconsistencies, undeclared source IDs, and invalid evidence spans. + +**TRD-RUB-005** Structural schema conformance shall not imply psychometric/content acceptance. Separate screening shall record answerability, construct alignment, evidence support, ambiguity, distractor quality, redundancy, leakage and bias/content-review results. + +**TRD-RUB-006** Candidate-aware rubric/criterion discovery shall not evaluate the same candidate set used to discover criteria unless cross-fitting or another approved anti-leakage design is used. + +### 4.6 Automated scoring and rater measurement + +**TRD-SCR-001** Human and automated scoring shall project into shared rater/engine observation contracts. + +**TRD-SCR-002** `scored`, `abstained`, `failed`, and `excluded` shall remain distinguishable through calibration and audit boundaries. + +**TRD-SCR-003** Many-facet calibration shall bind person/respondent, task/task-revision, and rater/engine axes without substituting response IDs for stable respondent/system identities when the estimator interprets a person effect. + +**TRD-SCR-004** Designs shall explicitly validate respondent-task and task-rater connectedness where those effects must be separately identified. + +**TRD-SCR-005** Automated-scoring validation shall provide agreement evidence appropriate to ordinal ratings, plus descriptive bias/range-use evidence. Pearson/Spearman correlation may be supplementary only. + +**TRD-SCR-006** Future generalized rater discrimination/range-restriction/drift estimators require separate identification and true-parameter recovery contracts before release. + +### 4.7 Reference-free RAG/LLM judge measurement + +**TRD-RAG-001** Evaluation records shall carry system-run, query/testlet, judge family/model/version, prompt/occasion, evidence regime, and criterion identities when available. + +**TRD-RAG-002** Groundedness shall not be relabeled world correctness. Pooled-corpus coverage shall not be relabeled absolute corpus recall. Claims are constrained by the evidence universe. + +**TRD-RAG-003** LLM judges are raters; judge severity/bias/discrimination/drift and family dependence are measurement concerns, not assumed truth. + +**TRD-RAG-004** Candidate-independent perturbation anchors shall be supported for evaluator validation where feasible, such as unsupported-claim insertion, evidence deletion, distractor/citation swaps, and meaning-preserving paraphrases. + +### 4.8 Factor/model structure and relation-safe comparison + +**TRD-MOD-001** Factor count/retention and structural model choice are separate workflows. + +**TRD-MOD-002** The system shall not infer nestedness solely from model names. Actual loading, variance, proportionality, and boundary constraints determine relation class. + +**TRD-MOD-003** Regular nested comparisons may use appropriate LR tests; boundary/singular comparisons require boundary-aware or parametric-bootstrap evidence; strictly non-nested/overlapping comparisons require formal distinguishability before a selection statistic can produce preference. + +**TRD-MOD-004** Same-question/probe or same-system repeated observations shall use cluster-aware aggregation/resampling rather than treating all response cells as independent. + +**TRD-MOD-005** Final model selection shall include held-out prediction at operationally relevant cluster levels and recovery/model-selection simulations under realistic generating structures. + +### 4.9 Bifactor scoreability + +**TRD-BIF-001** A declared general factor must satisfy the documented applicability contract before general-factor ECV/item-ECV/scoreability quantities are returned. + +**TRD-BIF-002** Standardized loadings and uniquenesses shall satisfy the documented variance identity within a bounded numerical tolerance. + +**TRD-BIF-003** PUC shall be returned only for structures for which its interpretation is defined by the implemented contract. + +**TRD-BIF-004** Omega derived from latent-response standardization shall be labeled as latent-response reliability and shall not be presented as categorical observed-score reliability. + +**TRD-BIF-005** Intermediate sums and denominators shall be checked for finite overflow/underflow before reliability indices are returned. + +### 4.10 Rotation + +**TRD-ROT-001** Rotation criteria implement a shared Rust criterion interface separated from the optimizer. + +**TRD-ROT-002** Orthogonal and oblique optimization shall report convergence/stationarity and reject singular/degenerate transformations. + +**TRD-ROT-003** Deterministic multi-start shall report best observed solution and basin/start evidence; it shall not claim mathematical global optimality. + +**TRD-ROT-004** Sign/permutation alignment shall preserve semantically privileged columns when the criterion requires them, such as a designated bifactor general factor or target labels. + +**TRD-ROT-005** Selection across criteria shall use common recovery/stability/theory/degeneracy evidence, not incomparable raw criterion objective values. + +### 4.11 Multilevel/multiple-membership/temporal contracts + +**TRD-MLT-001** Context membership shall include explicit context dimension and context identity. + +**TRD-MLT-002** Multiple-membership weights shall be validated rather than silently renormalized when the public contract promises exact caller-supplied membership. + +**TRD-MLT-003** Cross-classified designs shall maintain dimension-qualified identities. + +**TRD-MLT-004** Temporal occasions shall retain ordering/time provenance and explicitly separate discrete occasion-step AR effects from continuous-time parameterizations. + +**TRD-MLT-005** Numerical multilevel/longitudinal estimators shall remain proposed until Rust implementations pass identification and true-parameter recovery studies. + +### 4.12 Testing and scientific evidence + +**TRD-TEST-001** Owned production Python statement/branch coverage target is 100%; public docstrings and Rust documentation shall be complete and beginner-readable. + +**TRD-TEST-002** Tests shall use realistic measurement cases, not only type/shape smoke tests. + +**TRD-TEST-003** True-parameter simulations shall report bias, MAE/RMSE, SE/interval coverage, convergence, and relevant model-specific recovery. Scale/rotation/linking alignment occurs before parameter error calculation. + +**TRD-TEST-004** Same numerical contract implemented on CPU/GPU/Python-reference paths shall have parity tests using invariants appropriate to identification (e.g. distances/Procrustes rather than raw coordinates where necessary). + +**TRD-TEST-005** Heavy literature/recovery studies shall run on scheduled/manual/release paths when too expensive for every PR; bounded smoke/recovery sentinels remain on PRs. Scientific gates are not deleted to reduce CI latency. + +**TRD-TEST-006** Monte Carlo acceptance bounds shall be specified prospectively from scientific/statistical reasoning and shall not be fitted to one observed random seed's result. + +### 4.13 LLM/provider tests and automation + +**TRD-LLM-001** Model-backed GitHub tests/actions use `NVIDIA_NIM_API_KEY` through GitHub Secrets when a model call is materially necessary. `COPILOT_GITHUB_TOKEN` is prohibited for autonomous development scheduling. + +**TRD-LLM-002** `contextual-orchestrator` is preferred as a provider-neutral orchestration integration when suitable, but remains a read-only external dependency while its own writer loop is active. + +**TRD-LLM-003** Deterministic gates must remain executable without model credentials when the feature being validated does not require a model call. + +**TRD-LLM-004** Deep orchestration must be justified with comparable-budget evidence versus simpler routing, including task decomposition, recursion/workflow depth, role-specific reasoning effort, and ablations where relevant. + +### 4.14 Security and supply chain + +**TRD-SEC-001** PR checks include repository policy and central security scanning; known HIGH/CRITICAL dependency findings shall be remediated or narrowly documented as verified false positives rather than ignored by weakening gates. + +**TRD-SEC-002** GitHub Actions shall use least privilege and immutable action pins where practical. Write-capable self-modifying branch workflows are prohibited. + +**TRD-SEC-003** Provider/source/response text and secrets shall not leak into exception messages, audit IDs, logs, test artifacts, or generated reports unless the contract explicitly permits the content. + +**TRD-SEC-004** Persistence/PII controls that belong to hosted applications shall not be emulated in the core library. Reusable artifacts prefer fingerprints, opaque identities and data minimization. + +### 4.15 Documentation and architecture governance + +**TRD-DOC-001** Canonical documents are `docs/PRD.md`, `docs/TRD.md`, root `ARCHITECTURE.md`, `docs/adr/README.md`, the accepted/proposed ADR corpus, `docs/uml/`, `docs/erd/`, and `docs/traceability/`. + +**TRD-DOC-002** `docs/prd_trd_summary.md` is a historical summary and shall point to the canonical PRD/TRD rather than remain an independent authority. + +**TRD-DOC-003** Material ownership/API/model/lifecycle/security/release changes require architecture/ADR impact review in the same change or an explicit no-impact assertion. + +**TRD-DOC-004** Documentation structure shall be machine-checked for required canonical files, valid ADR statuses and core ownership-boundary consistency. + +## 5. Release acceptance + +A release candidate is accepted only when the exact integrated protected head provides evidence for: + +1. required Python/Rust/PyO3 tests and 100% owned coverage policy; +2. package build and clean reinstall/import; +3. explicit Rust-primary backend assertion; +4. GPU-required tests when the released claim requires GPU; +5. fuzz/security/SAST/supply-chain gates; +6. realistic recovery or parity studies for changed psychometric kernels; +7. release artifact digests/SBOM/provenance and reproducibility evidence; +8. documentation/changelog/version consistency; +9. zero valid unresolved review/security findings; +10. repository approval/branch-protection policy; +11. migration/compatibility/rollback evidence for changed serialized contracts. + +## 6. Standards and primary evidence baseline + +- AERA, APA, & NCME (2014), *Standards for Educational and Psychological Testing*. +- ISO/IEC/IEEE 29148:2018, requirements engineering. +- ISO/IEC/IEEE 42010:2022, architecture description. +- ISO/IEC 25010:2023, product quality model. +- ISO/IEC 42001:2023, AI management-system controls where AI lifecycle governance is relevant. +- W3C WCAG 2.2, report/accessibility concerns. +- Method-specific psychometric primary literature recorded in `AGENTS.md`, doctoring records, and relevant ADRs. diff --git a/docs/adr/0000-template.md b/docs/adr/0000-template.md new file mode 100644 index 000000000..b62542eef --- /dev/null +++ b/docs/adr/0000-template.md @@ -0,0 +1,80 @@ +# ADR-NNNN: Decision title + +Status: Proposed +Date: YYYY-MM-DD +Supersedes: none +Superseded by: none + +## Context + +Describe the observed product/scientific/technical problem and current protected-main behavior. Separate facts from assumptions, active PR work, and future plans. + +## Decision drivers + +- Driver one. +- Driver two. + +## Ownership and dependency direction + +State the owning bounded context/repository and which dependencies are allowed or forbidden. Explicitly note whether the decision changes the `fast-mlsirm` reusable-core vs Psychometrics Commons hosted-product boundary. + +## Decision + +State the decision precisely enough that code/tests/documentation can determine compliance. If only part of the decision is implemented, mark the ADR Proposed or distinguish the accepted invariant from proposed implementation. + +## Invariants / acceptance evidence + +1. Invariant tied to a test, recovery study, security check, or other exact evidence. +2. Invariant tied to a failure/degraded/recovery rule. + +## Non-goals and claims not made + +List adjacent capabilities or interpretations this decision does not authorize. + +## Consequences and trade-offs + +### Benefits + +- Benefit. + +### Costs / risks + +- Cost or risk. + +## Alternatives considered + +### Alternative A + +Why it was considered and rejected/deferred. + +### Alternative B + +Why it was considered and rejected/deferred. + +## Failure, degraded, and recovery behavior + +Describe fail-closed/fallback behavior, retry/idempotency if applicable, operator evidence, recovery/rollback, and how a failed migration or rollout is handled. + +## Security and privacy implications + +Cover new credentials/permissions, data classification, PII/sensitive evidence, native/provider trust, supply-chain surface, retention, auditability, and threat-model changes where applicable. + +## Compatibility, migration, and rollback + +Define public API/schema/artifact compatibility, migration steps, old-artifact interpretation, rollback and how supersession preserves decision history. + +## Verification and release evidence + +List required unit/property/fuzz/security tests, Rust↔Python or CPU↔GPU parity, true-parameter recovery/coverage, documentation/traceability, package/release evidence, or downstream contract checks before the implementation may be called Accepted/released. + +## Research and standards basis + +Use APA 7 references to primary peer-reviewed methods and current official standards/specifications where material. Mark preprints as preprints; do not use legacy software output as the scientific oracle when the original method is available. + +## Follow-ups + +Record deliberately deferred bounded work, with owning issue/PR only as a tracking aid rather than the decision authority. + +## Reversal / supersession conditions + +State what evidence, standard, product-boundary change, or implementation failure should trigger a new superseding ADR instead of silently editing this accepted history. diff --git a/docs/adr/0001-domain-neutral-measurement-boundary.md b/docs/adr/0001-domain-neutral-measurement-boundary.md new file mode 100644 index 000000000..689b4cc2c --- /dev/null +++ b/docs/adr/0001-domain-neutral-measurement-boundary.md @@ -0,0 +1,72 @@ +# ADR-0001: Domain-neutral measurement boundary + +Status: **Accepted** +Date: 2026-08-09 + +## Context + +`fast-mlsirm` has grown from an MLS2PLM-focused toolkit into a reusable measurement layer with assessment/rubric/scoring contracts, automated-scoring adapters, psychometric diagnostics, item/rater calibration and release evidence. ContextualWisdomLab also has downstream hosted/application bounded contexts, especially Psychometrics Commons. + +Without an explicit boundary, the library can accidentally absorb participant/session state, identity, hosted persistence, UI and deployment logic. That would make the numerical core harder to install independently, create circular repository dependencies, and duplicate ownership already assigned to other CWL services. + +## Decision + +`fast-mlsirm` owns reusable domain-neutral measurement contracts and scientific/numerical capabilities: + +- Assessment/Rubric/Scoring contracts and observations; +- calibration, CTT/IRT/MIRT, MLSIRM/MLS2PLM, facets, testlets and related kernels; +- model diagnostics, linking, DIF/invariance/fairness, CAT/ATA and G-theory; +- factor retention/model comparison, bifactor scoreability, rotation and recovery; +- automated-scoring and LLM-judge measurement primitives; +- governed rubric/item-bank primitives and portable scientific/audit reports. + +The hosted product boundary belongs downstream. In particular `ContextualWisdomLab/psychometrics-commons` owns hosted/public/admin APIs, participant/session/response/consent/result lifecycle, product databases/migrations, tenant/resource authorization, UI/reference client behavior, deployment composition and research-release orchestration. + +The dependency direction is: + +```text +hosted/downstream product -> fast-mlsirm +``` + +and never: + +```text +fast-mlsirm -> hosted product +``` + +Other CWL services are explicit optional integrations, not hidden implementation dependencies. + +## Invariants and evidence + +- The package must install and execute without Psychometrics Commons source or runtime. +- No product ORM/database, HTTP route, session/consent or UI type may become a required `fast_mlsirm` dependency. +- Cross-repository composition uses versioned APIs/contracts/immutable artifacts. +- Hosted product state must not be recreated under a library-local assessment runtime service. +- `AGENTS.md` and `CLAUDE.md` carry the same boundary. + +## Consequences + +Benefits: + +- independent adoption and testing; +- clear MSA ownership and security authority; +- lower coupling between scientific evolution and product deployment; +- reusable measurement contracts across essay, RAG, enterprise-issue and other domains. + +Costs: + +- downstream adapters are required for persistence/transport; +- some end-to-end features require cross-repository integration tests rather than one monolith. + +## Alternatives considered + +1. **Make fast-mlsirm the hosted platform.** Rejected because it couples scientific kernels to application infrastructure. +2. **Keep only numerical functions and move all contracts downstream.** Rejected because versioned measurement/scoring contracts are reusable scientific primitives and must stay adjacent to the numerical interpretation they govern. + +## Reversal conditions + +Supersede this ADR only if the organization intentionally redefines repository bounded contexts and provides a migration plan preserving independent scientific/numerical reuse. + +## References + +International Organization for Standardization, International Electrotechnical Commission, & Institute of Electrical and Electronics Engineers. (2022). *ISO/IEC/IEEE 42010:2022 Software, systems and enterprise—Architecture description*. diff --git a/docs/adr/0002-rust-first-numerical-ownership.md b/docs/adr/0002-rust-first-numerical-ownership.md new file mode 100644 index 000000000..8d27b0ecc --- /dev/null +++ b/docs/adr/0002-rust-first-numerical-ownership.md @@ -0,0 +1,60 @@ +# ADR-0002: Rust-first numerical ownership + +Status: **Accepted** +Date: 2026-08-09 + +## Context + +The repository exposes Python APIs while supporting computationally intensive psychometric estimation, diagnostics, calibration and simulation. Maintaining independent Python and Rust production formulas creates drift risk, doubles verification burden and makes CPU/GPU ownership ambiguous. At the same time, transparent NumPy paths are valuable for parity, research inspection and controlled fallback. + +## Decision + +Rust is the production source of truth for mathematically material psychometric computation, including likelihoods, gradients, Hessians/information matrices, optimization, scoring/ranking, item/factor information and other numerical kernels when promoted to production capability. + +Python may: + +- validate and bound inputs; +- marshal NumPy arrays; +- orchestrate domains/providers; +- expose typed results; +- render reports; +- retain governed reference/fallback calculations where a parity contract exists. + +Python shall not become an independently evolving second production formula. + +The public backend architecture may expose `auto`, `rust` and governed `numpy` reference/fallback choices for APIs that currently support them. Rust is the preferred resolved production backend when the extension is available. GPU is a Rust device path, not a third psychometric formula implementation. + +The canonical PyO3 layer must support feature growth without independent PRs overwriting initialization/export structure. Secondary module symbols, if retained, must be registered through one auditable binding architecture. + +## Numerical invariants + +- Formula changes update Rust and any governed reference path together. +- Analytic derivatives are checked against independent finite-difference or equivalent oracles where practical. +- CPU/GPU parity uses identification-aware invariants rather than raw non-identifiable coordinates. +- Caller-controlled allocation sizes are bounded before allocation. +- Non-finite numerical boundaries fail closed where the model requires finite values. +- Computationally material CPU parallelism should minimize task/thread context-switch overhead. + +## Consequences + +Benefits: + +- one production arithmetic authority; +- safer GPU/CPU parity and performance work; +- clearer auditability and reproducibility; +- Python remains ergonomic without owning scientific numerics. + +Costs: + +- new mathematical features require Rust/PyO3 work before product release; +- pure-Python prototypes cannot be declared production estimators without migration. + +## Alternatives considered + +1. **NumPy as primary backend, Rust as optional accelerator.** Rejected as the long-term architecture because production behavior can diverge and high-cost computations remain Python-owned. +2. **Rust-only API.** Rejected because Python remains the principal research/product integration surface. +3. **Independent CPU/GPU model implementations.** Rejected; GPU must share the same mathematical contract and parity evidence. + +## Reversal conditions + +A different numerical owner requires an ADR demonstrating equivalent or stronger correctness, performance, packaging, parity and scientific-evidence guarantees. diff --git a/docs/adr/0003-content-addressed-measurement-contracts.md b/docs/adr/0003-content-addressed-measurement-contracts.md new file mode 100644 index 000000000..5d6787778 --- /dev/null +++ b/docs/adr/0003-content-addressed-measurement-contracts.md @@ -0,0 +1,112 @@ +# ADR-0003: Content-addressed measurement contracts + +Status: **Accepted** +Date: 2026-08-09 + +## Context + +Psychometric interpretation depends on exact construct, rubric, task, response, rater/engine, calibration and software revisions. Human-readable IDs alone do not prove content identity, while mutable objects can make later audits falsely appear to reproduce the original analysis. + +## Decision + +Reusable measurement artifacts use versioned, canonical and content-addressed contracts wherever semantic identity affects interpretation. + +Key rules: + +1. Schema/wire version and semantic/domain revision are separate fields. +2. Authoritative content identity uses deterministic canonical serialization and a full cryptographic digest where the contract exposes fingerprints. +3. Short public handles may aid display but never replace full fingerprint comparison at a trust boundary. +4. Aggregate artifacts replay/verify package-owned child objects when crossing a trust boundary rather than trusting parent caches or display IDs. +5. Logical task ID and exact task revision remain separate. +6. Stable respondent/system identity and response artifact/revision remain separate. +7. Rater/engine descriptor identity and individual rater identity remain separately representable where many-facet interpretation requires them. +8. Published/approved artifacts are immutable; corrections create superseding revisions. +9. Caller/provider text is bounded and sensitive content is not embedded in error identifiers or audit hashes beyond the explicit canonical input. + +## Canonical fingerprint preimage contract + +The v1 fingerprint preimage contract is named `fast-mlsirm-cjson-v1` and is +part of every exposed fingerprint record together with the digest algorithm +`sha-256`. Implementations MUST reject an unknown canonicalization version or +digest algorithm rather than silently selecting a local default. + +For `fast-mlsirm-cjson-v1`, the preimage is the UTF-8 encoding of compact JSON +with no trailing newline, using these rules: + +1. Object keys are unique strings and are ordered by Unicode scalar-value order; + arrays retain their declared order. +2. A contract's declared field set is authoritative. Omitted fields and explicit + `null` are different values; no serializer may add, omit, or coerce fields. +3. Strings are preserved as supplied after valid UTF-8 validation. No implicit + Unicode normalization is performed, so composed and decomposed spellings + have different identities unless the contract normalizes them before + construction. +4. Integers are signed 64-bit values. Floating-point values are finite; negative + zero is serialized as `0.0`. Non-finite values are rejected. Numbers use the + shortest round-trippable JSON decimal representation of the package + canonicalizer. +5. Whitespace is omitted, escaping follows the UTF-8 JSON serializer, and the + SHA-256 digest is computed over the exact resulting bytes. + +The package-owned implementation and any Rust/other-language implementation +MUST pass the same normative vectors. For example, after the v1 validation and +negative-zero normalization, this value: + +```json +{"a":null,"items":[2,1.5],"n":0.0,"z":"café"} +``` + +has UTF-8 SHA-256 +`b2384fee029c793d3e661b5a741b318155a401482a7495d057bb696a2711c9c5`. +The vector binds the version, encoding, key ordering, null retention, array +ordering, Unicode preservation, numeric formatting and digest calculation; a +cross-language implementation is not interoperable until it reproduces it. + +## Primary contract family + +```text +RubricSpecification + | + v +AssessmentSpec / policy references + | + +--> task + exact task revision + | + v +ScoringRequest + | + v +ScoreObservation / ScoringResult + | + v +Calibration design/report +``` + +Rubric-centered item generation extends the same chain: + +```text +RubricSpecification -> ItemBlueprint -> GenerationContract + -> Candidate -> Screening/Calibration -> Item-bank revision +``` + +## Invariants + +- A one-byte semantic change that participates in canonical content changes the fingerprint. +- Reusing an identifier with changed canonical content fails closed at governed boundaries. +- Source/evidence spans are verified against the exact source revision when source evidence is part of the contract. +- Unknown major schema versions do not silently downgrade. +- Serialization is deterministic across input ordering where ordering is not semantically meaningful and preserves declared ordering where it is meaningful. + +## Consequences + +This increases object/schema discipline and migration work but provides replay protection, reproducible research, audit evidence, safe caching and reliable cross-service composition. + +## Alternatives considered + +- **Database integer IDs as identity.** Rejected because they identify rows, not semantic content, and are not portable across independent deployments. +- **Mutable named revisions.** Rejected because an old result could silently point at new content. +- **Hash everything including uncontrolled raw text in every report.** Rejected; only contract-relevant content is included, with privacy/data-minimization boundaries. + +## Reversal conditions + +A replacement must provide equal or stronger deterministic replay, compatibility, privacy and cross-deployment identity guarantees. diff --git a/docs/adr/0004-governed-rubric-item-bank-lifecycle.md b/docs/adr/0004-governed-rubric-item-bank-lifecycle.md new file mode 100644 index 000000000..c14a425ca --- /dev/null +++ b/docs/adr/0004-governed-rubric-item-bank-lifecycle.md @@ -0,0 +1,105 @@ +# ADR-0004: Governed rubric and item-bank lifecycle + +Status: **Proposed** +Date: 2026-08-09 + +## Context + +The numerical core can analyze calibrated observations, but a defensible measurement system also needs to construct, screen, calibrate, assemble, version and retire the criteria/items being measured. Ad hoc `prompt -> LLM item -> use immediately` workflows mix item generation with validation and can introduce candidate leakage, redundant criteria, unverifiable evidence, drift and version ambiguity. + +Protected `main` already contains canonical rubric-centered blueprint and provider-neutral generation-contract primitives. The complete governed item-bank lifecycle is not yet fully protected-integrated, so this ADR remains Proposed. + +## Decision + +Build the reusable lifecycle as: + +```text +RubricSpecification + -> Measurement Blueprint + -> Generation Contract + -> untrusted Generated Candidate + -> Structural Validation + -> Evidence/Semantic Screening + -> Artificial-Crowd / Calibration Pilot + -> Rust Psychometric Calibration + -> Information/Content-Constrained Assembly + -> Approved Item Bank + -> Monitoring / DIF / Drift / Exposure + -> Quarantine / Retirement / New Rubric Revision +``` + +### Generation modes + +- **Benchmark mode:** candidate-blind; criteria are generated from task contract and independent evidence, not target-system answers. +- **Diagnostic mode:** candidate-aware discovery may be used only with cross-fitting or an equivalent separation between criterion discovery and scored candidates. +- **Training mode:** may evolve criteria separately but must not contaminate a fixed benchmark bank. + +### Criterion design + +The canonical model supports rich internal criterion contracts; adapters may compile them to external rubric formats. Atomic binary/nominal criteria are preferred where one independently verifiable decision exists. Holistic ordinal levels remain valid when the construct genuinely requires ordinal synthesis. + +### Lifecycle states + +A representative lifecycle is: + +`draft -> audited -> screened -> pilot -> calibrated -> approved -> active -> suspended/quarantined -> retired`. + +A production/approved revision is immutable. Semantic changes create a new rubric/item revision and require linking/recovery evidence when scores must remain comparable. + +## Required screening dimensions + +- construct alignment; +- criterion atomicity; +- answerability/applicability; +- evidence grounding and provenance; +- ambiguity; +- direction/polarity validity; +- distractor/option integrity where applicable; +- redundancy/local dependence; +- candidate leakage; +- language/domain bias and future DIF risk; +- execution cost/resource bounds. + +## Calibration/assembly principles + +Raw LLM-proposed weights are not psychometric item information. Once pilot data exist, calibrated item/factor information, fit, DIF, rater facets, residual dependence, content constraints, anchor/linking needs, cost and exposure guide assembly. + +Safety/policy-critical criteria may be conjunctive gates rather than compensable score weights. + +## Consequences + +Benefits: + +- closes the upstream gap between rubric design and psychometric calibration; +- makes generated items auditable and versionable; +- supports living item banks without silently changing score meaning; +- creates a differentiating closed loop for AI/assessment evaluation. + +Costs: + +- requires orchestration, screening and lifecycle APIs beyond current blueprint compilation; +- real-model pilots can be expensive and must use bounded provider orchestration; +- linking/monitoring adds stateful host requirements, though reusable artifact contracts remain library-owned. + +## Alternatives considered + +1. **Generate a 1–5 rubric per question and average scores.** Rejected as insufficiently decomposed and uncalibrated. +2. **Keep a permanently static item bank.** Rejected as the only strategy; fixed banks remain supported but cannot address evolving AI/evaluation domains. +3. **Generate criteria from the candidate being scored.** Rejected for benchmark use because it double-dips; permitted only in isolated/cross-fitted diagnostic/training modes. + +## Acceptance before status becomes Accepted + +- trusted parser/screening path protected-integrated; +- at least one end-to-end candidate -> pilot -> Rust calibration workflow; +- governed item-bank revision/lifecycle contract; +- DIF/drift/linking evidence for version changes; +- realistic offline and bounded live-model tests; +- documentation and release evidence. + +## References + +Hashemi, H., Eisner, J., Rosset, C., Van Durme, B., & Kedzie, C. (2024). *Initial nugget evaluation results for the TREC 2024 RAG Track with the AutoNuggetizer framework*. arXiv:2411.09607. + +Hashemi, H., et al. (2024). LLM-Rubric: A multidimensional, calibrated approach to automated evaluation of natural language texts. *Proceedings of ACL 2024*. + +Shankar, S., Zamfirescu-Pereira, J. D., Hartmann, B., Parameswaran, A. G., & Arawjo, I. (2024). Who validates the validators? Aligning LLM-assisted evaluation of LLM outputs with human preferences. *Proceedings of UIST 2024*. diff --git a/docs/adr/0005-automated-scoring-raters.md b/docs/adr/0005-automated-scoring-raters.md new file mode 100644 index 000000000..2cdd9ec25 --- /dev/null +++ b/docs/adr/0005-automated-scoring-raters.md @@ -0,0 +1,69 @@ +# ADR-0005: Human and automated scorers are fallible raters + +Status: **Accepted** +Date: 2026-08-09 + +## Context + +Automated-scoring and LLM-as-a-Judge systems are often validated by correlating machine scores with one human score or by averaging multiple judges. That can hide rater severity, range compression, criterion bias, drift, shared shortcuts and human measurement error. A high correlation preserves rank but does not prove agreement, calibration, fairness or true-parameter recovery. + +The protected codebase already contains governed scoring contracts, essay/enterprise adapters and Rust-backed criterion many-facet calibration/reporting paths. + +## Decision + +Human, LLM and external automated scorers are represented as raters/engines producing observations under explicit assessment/rubric/task revisions. No rater is automatically the truth source. + +The scoring architecture must preserve: + +- rater/engine identity and revision; +- criterion and rubric identity; +- task and exact task revision; +- respondent/system-run identity; +- response artifact/revision identity; +- terminal observation state; +- evidence references where the scoring contract requires them; +- prompt/occasion/model version where an LLM judge is used. + +Many-facet calibration is the baseline mechanism for separating respondent/person, task/item and rater severity effects when the design identifies them. Future generalized rater discrimination, criterion-specific bias, range restriction and time-varying severity/drift require explicit model-identification and recovery evidence before production release. + +## Validation evidence + +Correlation may be shown as supplementary association evidence. Acceptance decisions use appropriate combinations of: + +- QWK, exact and adjacent agreement for ordinal ratings; +- absolute error/bias where a defensible reference scale exists; +- rater severity and fit; +- paired range/dispersion evidence; +- DIF/subgroup error and invariance evidence; +- human-human degradation/comparator evidence when available; +- true-parameter recovery in simulation; +- drift/retest stability across model/prompt/occasion revisions. + +`abstained`, `failed`, `excluded` and `scored` states remain distinct. Abstention or infrastructure failure is not converted to the lowest content score. + +## Identification invariants + +- The respondent/person axis must represent the entity whose latent property is interpreted; response IDs do not substitute for that entity when repeated tasks exist. +- Respondent-task and task-rater graphs must be connected enough for the effects being estimated. +- Multiple raters may score the same exact response revision. +- One respondent-task cell cannot silently bind multiple response revisions. + +## Consequences + +This supports defensible automated essay scoring, enterprise issue evaluation and LLM judge calibration. It also means a simple scorer wrapper cannot be declared validated merely because average agreement is high. + +## Alternatives considered + +- **Treat consensus/majority vote as truth.** Rejected because correlated rater errors and severity remain hidden. +- **Use one expert human as gold.** Rejected as the default scientific model; human anchors can be valuable but should retain rater uncertainty unless independently established as an authoritative answer key. +- **Use raw machine-human correlation as the primary gate.** Rejected because correlation is insensitive to additive/scale bias and depends on sample heterogeneity. + +## References + +American Educational Research Association, American Psychological Association, & National Council on Measurement in Education. (2014). *Standards for educational and psychological testing*. + +Bland, J. M., & Altman, D. G. (1986). Statistical methods for assessing agreement between two methods of clinical measurement. *The Lancet, 327*(8476), 307–310. + +Uto, M., & Ueno, M. (2020). A generalized many-facet Rasch model and its Bayesian estimation using Hamiltonian Monte Carlo. *Behaviormetrika, 47*, 469–496. + +Williamson, D. M., Xi, X., & Breyer, F. J. (2012). A framework for evaluation and use of automated scoring. *Educational Measurement: Issues and Practice, 31*(1), 2–13. diff --git a/docs/adr/0006-relation-safe-model-selection.md b/docs/adr/0006-relation-safe-model-selection.md new file mode 100644 index 000000000..9a2da7c3b --- /dev/null +++ b/docs/adr/0006-relation-safe-model-selection.md @@ -0,0 +1,76 @@ +# ADR-0006: Relation-safe factor and measurement-model selection + +Status: **Accepted** +Date: 2026-08-09 + +## Context + +The product supports or is extending multiple model structures: unidimensional and correlated MIRT, bifactor, higher-order, testlet, two-tier, multifaceted and latent-space models. These structures are not reliably classifiable as nested or non-nested by their names. Additional factor/testlet/latent-space effects may also create boundary or singular null hypotheses where ordinary chi-square LR theory is invalid. + +Selecting the model with the largest in-sample likelihood, lowest BIC, or most attractive plot can overfit and can create unsupported score interpretations. + +## Decision + +Factor retention and structural model selection are separate decisions. + +### Stage 1: substantive factor-retention candidates + +Use data-type-appropriate retention evidence such as exploratory MIRT/EFA, parallel/MAP/network methods where appropriate, fit/residual diagnostics, theory and cross-validation to define a small candidate factor-count set. + +### Stage 2: structural relation classification + +Classify each pair from actual parameter constraints: + +- regular nested; +- boundary/singular nested; +- nonlinear-constraint nested; +- strictly non-nested; +- overlapping/indistinguishable; +- unknown. + +Higher-order versus bifactor, testlet versus bifactor, or a latent-space extension must not be hard-coded as non-nested solely from model labels. + +### Stage 3: relation-appropriate inferential evidence + +- regular nested -> appropriate likelihood-ratio/robust equivalent; +- boundary/singular -> boundary-aware or parametric-bootstrap LR; +- strictly non-nested/overlapping -> formal Vuong distinguishability before normal-theory selection; +- unknown -> no model preference until relation is established. + +A numerical positive variance of casewise log-likelihood differences is not the full formal Vuong distinguishability test. + +### Stage 4: operational predictive evidence + +Use cluster-aware held-out likelihood at the level relevant to deployment, such as query/testlet, respondent/system, rater family or domain. Random response-cell splitting is avoided when it leaks the same person/query/rater into train and validation. + +### Stage 5: interpretation evidence + +Inspect residual dependence, DIF/invariance, factor determinacy/score reliability, bifactor scoreability, rotation/stability and external validity as appropriate. + +### Stage 6: recovery + +Simulate realistic generating structures and evaluate model-selection accuracy plus parameter bias/RMSE/coverage/convergence. + +### Selection rule + +Prefer the simplest model whose predictive performance is practically competitive and whose identification, residual, invariance, scoreability and recovery conditions support the intended interpretation. + +## Consequences + +The system may return `indeterminate`, `requires_distinguishability_test`, or `requires_likelihood_ratio` rather than a winner. This is intended product safety, not missing functionality. + +## Alternatives considered + +- **AIC/BIC-only selection.** Rejected as insufficient for flexible and boundary models. +- **Always select bifactor when it fits better.** Rejected because bifactor flexibility does not establish scoreability. +- **Always add latent space for residual fit.** Rejected; latent space is residual interaction after substantive/facet/testlet structure and must improve held-out/recovery evidence. + +## References + +Cai, L. (2010). A two-tier full-information item factor analysis model with applications. *Psychometrika, 75*, 581–612. + +Preacher, K. J., Zhang, G., Kim, C., & Mels, G. (2013). Choosing the optimal number of factors in exploratory factor analysis: A model selection perspective. *Multivariate Behavioral Research, 48*, 28–56. + +Rijmen, F. (2010). Formal relations and an empirical comparison among the bi-factor, the testlet, and a second-order multidimensional IRT model. *Journal of Educational Measurement, 47*, 361–372. + +Schneider, L., Chalmers, R. P., Debelak, R., & Merkle, E. C. (2020). Model selection of nested and non-nested item response models using Vuong tests. *Multivariate Behavioral Research, 55*, 664–684. diff --git a/docs/adr/0007-multilevel-multiple-membership-temporal.md b/docs/adr/0007-multilevel-multiple-membership-temporal.md new file mode 100644 index 000000000..1b65624d4 --- /dev/null +++ b/docs/adr/0007-multilevel-multiple-membership-temporal.md @@ -0,0 +1,70 @@ +# ADR-0007: Multilevel, multiple-membership and temporal structure are first-class + +Status: **Proposed** +Date: 2026-08-09 + +## Context + +Psychometric and AI-evaluation observations commonly sit inside schools, teams, organizations, prompts, testlets, documents, clients, time periods or other overlapping contexts. Repeated observations also evolve over time. Flattening those structures into independent rows can produce atomistic fallacy, understate uncertainty, confound stable traits with context effects and drift, and misinterpret temporal dependence. + +A current open PR contains reusable contract work for nested, cross-classified, multiple-membership and longitudinal designs, but it is not yet protected-integrated. Numerical estimators for the full structures are not accepted production behavior. Therefore this ADR remains Proposed. + +## Decision + +The architecture treats the following as distinct, explicit structures: + +- nested context; +- cross-classified context; +- weighted multiple membership; +- multiple-membership multiple-classification; +- testlet/shared-stimulus local dependence; +- repeated longitudinal occasions; +- discrete occasion-step autoregression; +- future continuous-time state transitions; +- rater/model/prompt drift. + +### Contract rules + +1. Context membership names an explicit `context_dimension_id` and dimension-scoped `context_id`. +2. Membership weights are provenance-bound and validated under the public contract; they are not silently inferred from labels. +3. Every observation carries the context dimensions required by the declared design. +4. Repeated occasions preserve respondent/system identity and exact temporal ordering/provenance. +5. A discrete occasion-step AR coefficient is not interpreted as elapsed-time decay. Continuous-time interpretation requires a separately parameterized model. +6. Local/testlet effects and substantive dimensions remain conceptually distinct. + +### Numerical release rule + +A new Rust estimator for these structures is not production-ready until realistic simulation establishes: + +- identification under supported designs; +- parameter bias and RMSE; +- SE/interval coverage; +- convergence and failure classification; +- behavior under sparse/unbalanced membership; +- CPU/GPU parity where a GPU path exists; +- comparison against simpler models using relation-safe procedures. + +## Consequences + +The library can represent scientifically realistic designs before every estimator is implemented. Contract availability does not imply estimation capability or causal interpretation. + +This architecture avoids forcing product-specific tenant/org structures into the core; contexts are provider/domain-neutral. + +## Alternatives considered + +- **Flatten all observations.** Rejected due to atomistic/ecological inference risk and underestimated dependency. +- **Use latent space to absorb all dependence.** Rejected because known hierarchy/time/testlet structures should be modeled explicitly before residual interactions. +- **Treat timestamps as labels only forever.** Rejected as a long-term architecture; timestamps are preserved so explicit temporal estimators can be added without data-model migration. + +## Acceptance before status becomes Accepted + +- governed contracts merged to protected main; +- architecture/serialization tests pass; +- at least one Rust estimator or clear handoff contract exists for a supported multilevel/temporal inference use case; +- recovery evidence meets the numerical release rule. + +## References + +Fox, J.-P., & Glas, C. A. W. (2001). Bayesian estimation of a multilevel IRT model. *Psychometrika, 66*, 271–288. + +Uto, M. (2022). A Bayesian many-facet Rasch model with Markov modeling for rater severity drift. *Behavior Research Methods, 55*, 3910–3928. diff --git a/docs/adr/0008-true-parameter-recovery-ci.md b/docs/adr/0008-true-parameter-recovery-ci.md new file mode 100644 index 000000000..b1f55e19f --- /dev/null +++ b/docs/adr/0008-true-parameter-recovery-ci.md @@ -0,0 +1,67 @@ +# ADR-0008: True-parameter recovery is core scientific CI evidence + +Status: **Accepted** +Date: 2026-08-09 + +## Context + +A numerical psychometric implementation can produce plausible-looking estimates and high correlations while being systematically biased, on the wrong scale, overconfident, or unstable near important parameter regions. In AI evaluation, human raw scores also contain rater and task effects; correlation with them is not a sufficient accuracy or validity claim. + +The repository already uses simulation/recovery, Rust/NumPy parity and literature-design studies. This ADR makes the evidence hierarchy explicit. + +## Decision + +For estimators with known generating parameters, the default scientific acceptance evidence is: + +1. identify and align the model scale/rotation/linking; +2. calculate parameter bias; +3. calculate MAE and/or RMSE; +4. evaluate SE bias and nominal interval coverage when uncertainty is exposed; +5. evaluate convergence/failure rate; +6. evaluate model-specific function recovery such as response probabilities, thresholds, information, distances or factor/loadings after appropriate alignment; +7. evaluate CPU/GPU/reference parity where multiple execution paths implement the same contract. + +Correlation may be reported as supplementary order-preservation evidence but is not an accuracy gate. + +### Scale/identification rule + +Raw RMSE is invalid when parameters are unidentified up to scale, sign, permutation, rotation, reflection or translation. The recovery harness must apply the same identification or accepted alignment used for interpretation before error metrics. + +Examples: + +- latent-space positions -> Procrustes/aligned positions or distance matrices; +- multidimensional loadings -> sign/permutation/rotation alignment; +- linked IRT scales -> fixed anchors or a documented linking transform; +- rater severity -> identified centering/reference constraints. + +### CI strategy + +PR CI should contain bounded sentinel/recovery tests that catch scientific regressions without exhausting the merge queue. Expensive paper-design Monte Carlo studies remain scheduled/manual/release evidence with deterministic manifests proving coverage of the intended study inventory. + +Monte Carlo acceptance rules are chosen prospectively from theory/sampling precision, not retrofitted to one observed seed outcome. + +## Consequences + +Benefits: + +- catches errors hidden by correlation; +- makes numerical changes reviewable against scientific consequences; +- supports Rust/GPU evolution without trusting implementation resemblance alone. + +Costs: + +- realistic simulations are computationally expensive; +- recovery thresholds require method-specific justification; +- some models need careful alignment before metrics are meaningful. + +## Alternatives considered + +- **Correlation-only validation.** Rejected because positive affine bias can preserve correlation perfectly. +- **Golden file of one fit.** Rejected as insufficient; deterministic regression is useful but cannot establish recovery across data-generating conditions. +- **Run every heavy study on every PR.** Rejected for queue/schedulability reasons; the studies are retained in scheduled/release evidence rather than deleted. + +## References + +Bland, J. M., & Altman, D. G. (1986). Statistical methods for assessing agreement between two methods of clinical measurement. *The Lancet, 327*(8476), 307–310. + +Svetina, D., Valdivia, A., Underhill, S., Dai, S., & Wang, X. (2017). Parameter recovery in multidimensional item response theory models under complexity and nonnormality. *Applied Psychological Measurement, 41*(7), 530–544. diff --git a/docs/adr/0009-adaptive-rotation-selection.md b/docs/adr/0009-adaptive-rotation-selection.md new file mode 100644 index 000000000..b880009dd --- /dev/null +++ b/docs/adr/0009-adaptive-rotation-selection.md @@ -0,0 +1,101 @@ +# ADR-0009: Adaptive rotation uses criterion registry, multi-start and empirical selection + +Status: **Accepted** +Implementation maturity: **Protected-main CPU baseline implemented; GPU/additional-criterion/recovery expansion remains planned** +Date: 2026-08-09 + +## Context + +Exploratory factor rotation does not have one universally optimal criterion. Criterion behavior depends on loading complexity, factor correlation, cross-loadings, sample size, target information and local optima. Comparing raw objective values across different criteria is also invalid because those objectives have different definitions and scales. + +Protected main now contains the Rust-backed rotation criterion registry, deterministic multi-start optimizer, criterion-neutral selector/evidence surfaces, Python/PyO3 public API, package-root rotation exports, method doctoring and regression coverage. The governing policy in this ADR therefore describes current protected-main behavior. GPU batching, additional criterion families and broader simulation/recovery evidence remain future increments and do not make the implemented CPU policy Proposed. + +## Decision + +Rotation is structured in three layers. + +### 1. Criterion registry + +Each analytic criterion implements a common Rust value/gradient contract. The optimizer does not contain criterion-specific algebra except through that interface. + +The protected-main registry contains orthomax/Crawford-Ferguson, oblimin, geomin, target/PST, information, component-loss, bifactor, tandem and invariant-simplicity families documented by `available_rotation_criteria()`. Procedural or derivative-free criteria may use explicit separate adapters rather than being forced into an invalid gradient contract. + +### 2. Optimizer and solution search + +The Rust optimizer supports appropriate orthogonal/oblique geometry and reports: + +- criterion value; +- projected gradient/stationarity; +- transform/pattern/structure/factor-correlation matrices; +- termination reason; +- deterministic multi-start evidence; +- best-start index and best-observed basin support; +- sign/permutation canonicalization where semantically valid. + +Finite multi-start returns the **best observed solution**, not proof of a global optimum. + +### 3. Criterion-neutral selector + +Criteria are compared using common evidence such as: + +- loading/simple-structure complexity; +- cross-loading sparsity; +- degeneracy/near-singular factor-correlation penalties; +- convergence and basin support; +- bootstrap stability and Tucker congruence after global assignment/sign alignment when replicates are supplied; +- target/theory agreement when externally provided; and +- split-sample or simulation recovery when the study design supplies it. + +The selector exposes explicit evidence policies, but policy weights/ranks are documented choices, not universal scientific constants. Objective values from unlike criteria are never treated as directly comparable merely because they are scalar. + +## Numerical constraints + +- Matrix operations used by a criterion must respect its mathematical domain; SPD log determinants use SPD-safe decomposition such as Cholesky rather than sign-ambiguous row pivot heuristics. +- PST/target weights follow the exact documented criterion semantics. Binary-mask PST does not silently accept arbitrary continuous weights. +- Hyperparameters are included in solution provenance. +- CPU multi-start uses coarse parallelism to limit context switching. +- GPU batching is released only after objective/gradient/stationarity and selected-basin parity evidence; current protected-main rotation provenance identifies the CPU backend rather than implying GPU execution. + +## Invariants and current acceptance evidence + +The Accepted baseline is evidenced by protected-main source/tests/docs including: + +- `crates/mlsirm-core/src/rotation/` for criteria, optimizer, matrix helpers and selector; +- `crates/fast-mlsirm-py/src/rotation_bindings.rs` for the native binding; +- `python/fast_mlsirm/rotation.py` and `rotation_selection.py` for public validation/marshalling/report surfaces; +- package-root rotation exports; +- `tests/test_rotation*.py` for public API, selection, validation and target-alignment contracts; and +- `docs/adaptive_factor_rotation.md` for the current scientific/operational boundary. + +Future criteria, GPU execution and stronger population-recovery studies require their own exact-head parity/recovery evidence before their claims become implemented. Their absence does not invalidate the current criterion-neutral CPU architecture. + +## Consequences + +The public product can answer both “what is the best solution we observed for this criterion?” and “which criterion is best supported for this use case?” without conflating the two. The second answer remains conditional on candidate set, extraction model, data/resampling design and policy; it is not a universal criterion-ranking claim. + +## Failure and interpretation boundaries + +- Non-convergence, singular/invalid domains, unsupported criterion/mode combinations and invalid target/weight semantics fail explicitly rather than silently changing objective meaning. +- A finite set of starts cannot certify the global optimum. +- Bifactor-oriented rotations do not establish substantive bifactor scoreability or justify a general score by themselves. +- Criterion selection evidence does not replace factor-retention, structural-model comparison, held-out prediction, invariance/DIF or true-structure recovery when those questions govern interpretation. + +## Alternatives considered + +- **Default to varimax.** Rejected as universal policy. +- **Choose the criterion with the numerically smallest objective.** Rejected because objectives are not cross-criterion comparable. +- **Use one random start.** Rejected because local minima can dominate results. +- **Claim finite multi-start global optimization.** Rejected as scientifically unsupported. + +## Planned evolution without changing this Accepted decision + +- GPU batching after parity and basin-selection evidence; +- additional scientifically justified criterion families; +- richer bootstrap/simulation recovery evidence and buyer-facing selection reports; and +- supersession through a later ADR if the registry/optimizer/selector ownership model itself changes. + +## References + +Bernaards, C. A., & Jennrich, R. I. (2005). Gradient projection algorithms and software for arbitrary rotation criteria in factor analysis. *Educational and Psychological Measurement, 65*(5), 676–696. https://doi.org/10.1177/0013164404272507 + +Browne, M. W. (2001). An overview of analytic rotation in exploratory factor analysis. *Multivariate Behavioral Research, 36*(1), 111–150. https://doi.org/10.1207/S15327906MBR3601_05 diff --git a/docs/adr/0010-llm-orchestration-and-credentials.md b/docs/adr/0010-llm-orchestration-and-credentials.md new file mode 100644 index 000000000..e90d680d8 --- /dev/null +++ b/docs/adr/0010-llm-orchestration-and-credentials.md @@ -0,0 +1,71 @@ +# ADR-0010: LLM orchestration and credential boundary + +Status: **Accepted** +Date: 2026-08-09 + +## Context + +Some `fast-mlsirm` features and validation studies may use LLMs for item generation, semantic screening, artificial-crowd responses or LLM-as-a-Judge experiments. Model calls introduce provider credentials, untrusted outputs, cost/rate limits, non-determinism and possible cross-service coupling. Autonomous development agents and independent review agents also use separate authority and must not share identities casually. + +## Decision + +### Product/model calls + +- Prefer provider-neutral interfaces. +- Use `contextual-orchestrator` when a reusable orchestration boundary is appropriate, but do not create a reverse source dependency or write there while its owner loop controls that repository. Every cross-repository call MUST bind a versioned request/result schema (for example, `contextual-orchestrator-contract-v1`) or an immutable artifact digest, and the compatibility policy for that contract must be recorded with the caller. +- Treat all model output as untrusted; schema/provenance/semantic validation remains inside the calling workflow. +- Deterministic tests/gates that do not require a model call shall remain executable without model credentials. + +### Credentials + +- Model-backed GitHub tests/agents use the existing GitHub Secret `NVIDIA_NIM_API_KEY` when the model call requires NVIDIA NIM. +- `COPILOT_GITHUB_TOKEN` is not used for autonomous development scheduling. +- Review-agent credentials/identities remain distinct from development-agent credentials and are not rewritten merely to simplify automation. +- Secrets are materialized only in the step/path that needs them and must not appear in logs, report payloads, error strings or generated audit identifiers. + +### Autonomous development scheduling + +GitHub Actions autonomous development uses an immutably pinned OpenCode Agent design when repository automation performs model-backed development work. The scheduler does not manufacture approval, weaken branch protection or make advisory model output merge authority. + +### Test-time compute/orchestration + +When complex LLM orchestration is used, architecture decisions should compare simple single-model routing with deeper orchestration under comparable budgets. Relevant dimensions include: + +- workflow stages; +- task decomposition; +- recursion depth; +- tool/access lists; +- role-specific reasoning effort; +- number/family of model calls; +- ablation of deeper reasoning/orchestration. + +Correctness, evidence quality, reproducibility and controllability are more important than minimizing latency when the research task explicitly prioritizes inference quality. + +## Consequences + +Benefits: + +- no provider SDK becomes part of psychometric numerical truth; +- secrets and review authority remain separated; +- model-free deterministic CI stays usable; +- LLM orchestration can evolve without rewriting measurement contracts. + +Costs: + +- live-model validation requires explicit external service availability; +- orchestration evidence can be expensive; +- some end-to-end tests remain bounded/scheduled rather than always-on PR gates. + +## Alternatives considered + +1. **Call one vendor SDK directly from core numerical modules.** Rejected because it couples measurement truth to a provider. +2. **Require model credentials for all tests.** Rejected because deterministic package/scientific gates must not depend on unrelated provider availability. +3. **Use development-agent credentials for independent review.** Rejected because it collapses separation of duties and cannot create a legitimate independent approval. + +## Reversal conditions + +Supersede if CWL adopts a new organization-wide provider/agent credential architecture that provides equivalent least privilege, auditability, independent review separation and deterministic no-model gates. + +## Research traceability + +Orchestration-depth policies should be documented against current primary research, including Fugu/Conductor/TRINITY-class work or later stronger evidence, when those policies materially affect a released workflow. These studies inform experiments; they do not automatically mandate deep multi-agent orchestration for every task. diff --git a/docs/adr/0011-canonical-pyo3-public-export-registry.md b/docs/adr/0011-canonical-pyo3-public-export-registry.md new file mode 100644 index 000000000..72576e180 --- /dev/null +++ b/docs/adr/0011-canonical-pyo3-public-export-registry.md @@ -0,0 +1,68 @@ +# ADR-0011: Converge Rust-backed features on one canonical PyO3/public-export registry + +Status: **Proposed** +Date: 2026-08-09 + +## Context + +`fast-mlsirm` exposes Rust numerical functionality through PyO3 and composes a large Python package-root API. As independent feature PRs add Rust-backed modules (for example scoreability, rotation, future multilevel/time kernels), each can be tempted to add its own secondary `PyInit_*` symbol, loader shim, `_legacy_init.py` rewrite, or competing `python/fast_mlsirm/__init__.py` composition. Even when each PR works alone, sequential merges can produce import collisions, hidden initialization order, duplicate marshalling conventions, or one feature silently dropping another's export. + +The current protected package already has working public export composition. This ADR does not declare that those existing paths are broken. It defines the direction required before future Rust feature proliferation makes them unmaintainable. + +## Decision + +Adopt one repository-owned **canonical binding and public-export registry architecture** for Rust-backed feature modules. + +The target design shall: + +1. keep `crates/mlsirm-core` as numerical authority and `crates/fast-mlsirm-py` as the single reviewed native binding crate; +2. register Rust-backed Python functions/types from feature-scoped binding modules through one explicit composition point; +3. make the Python package-root export set derive from one maintained composition layer rather than feature PRs independently rewriting `__init__.py`/legacy init state; +4. use one marshalling/error/array-ownership convention per result family and document exceptions; +5. support additive feature registration without runtime source rewriting, dynamic compilation, import-time network access, or mutable plugin discovery; +6. fail import/build tests when two features claim the same public symbol or incompatible module-initialization path; and +7. preserve backwards-compatible public imports through explicit deprecation/alias policy rather than hidden import fallback. + +A secondary extension symbol may exist only as an explicitly designed module of this registry with cross-platform wheel/import evidence; it may not be invented independently by a feature PR as the easiest local integration. + +## Invariants and acceptance evidence + +- A wheel containing two or more Rust-backed feature families imports all of them in the same interpreter/process on every supported CI platform. +- Package-root exports include the union of intended public symbols with no order-dependent loss. +- `maturin`/PyO3 build metadata has one source of truth. +- Native exceptions are mapped through reviewed typed/bounded Python errors; Python does not parse Rust error strings to determine scientific status. +- NumPy arrays returned across the boundary have explicit ownership/immutability/shape semantics. +- An import test starts from a clean environment and never rewrites source or builds a second native module at runtime. +- Feature PRs include Rust -> PyO3 -> Python delegation tests rather than reimplementing the numerical calculation in Python. + +## Consequences and trade-offs + +A centralized registry creates a shared integration hotspot and can require small coordination when parallel feature PRs add bindings. That cost is preferable to multiple incompatible extension initializers and package-root compositions. Feature implementations remain modular internally; only registration/export authority is centralized. + +## Alternatives considered + +### Independent `PyInit_*` module per feature + +Rejected as the default. It can work for isolated features but multiplies wheel/import/platform contracts and creates merge-order conflicts. + +### One monolithic binding source file + +Rejected. A single initializer/composition point does not require a single unmaintainable source file; feature binding modules can remain separated and registered explicitly. + +### Runtime plugin discovery/dynamic compilation + +Rejected for the core package. It weakens reproducibility, offline packaging, supply-chain review and release provenance. + +### Python reimplementation when binding integration is difficult + +Rejected for production numerical paths by ADR-0002. Binding work is part of releasing a Rust numerical feature. + +## Reversal / supersession conditions + +A future stable PyO3 or Python packaging mechanism that provides independently versioned subextensions with stronger reproducibility and lower integration risk may supersede this decision, but only after cross-platform wheel/import evidence and a migration plan for current public imports. + +## References + +PyO3 Project. (2026). *PyO3 user guide* [Software documentation]. https://pyo3.rs/ + +Python Software Foundation. (2026). *Extending and embedding the Python interpreter: Defining extension modules*. Python 3.14 documentation. https://docs.python.org/3.14/extending/extending.html diff --git a/docs/adr/0012-purpose-limited-sensitive-data.md b/docs/adr/0012-purpose-limited-sensitive-data.md new file mode 100644 index 000000000..5b538bf0e --- /dev/null +++ b/docs/adr/0012-purpose-limited-sensitive-data.md @@ -0,0 +1,57 @@ +# ADR-0012: Preserve measurement utility through purpose-limited sensitive-data handling + +Status: **Accepted** +Date: 2026-08-09 + +## Context + +Psychometric, enterprise, longitudinal and human-rating workflows can require exact participant/source/group/context/occasion linkage. Blanket masking at every boundary can destroy repeated-measure, multilevel, multiple-membership, adjudication, DIF/fairness and evidence-trace relationships. At the same time, duplicating raw PII or sensitive source text into every calibration/report/provenance artifact unnecessarily increases privacy, breach and audit scope. + +## Decision + +`fast-mlsirm` does **not** use blanket PII masking as its default reusable-core privacy architecture. It uses purpose limitation, data minimization, separated identity/evidence domains, authorization, bounded retention interfaces, selective disclosure and exact provenance. + +- Raw sensitive content enters a computation only when the caller's authorized scientific/business purpose requires it. +- Durable reusable artifacts prefer opaque ids, content digests, bounded metadata and governed source references over copied raw text. +- Hosted participant/account identity resolution, consent, tenant authorization, encryption keys, residency, retention/deletion and data-subject workflows remain owned by Psychometrics Commons or the appropriate downstream data-owning service. +- Protected attributes used for DIF/fairness remain governed inputs; they are not generalized into unrestricted report metadata. +- Provider/model calls receive sensitive data only across an explicit authorized provider boundary, and provider exception text is not copied into durable audit evidence. +- If the measurement design requires exact linkage and that linkage is not authorized/available, the operation fails rather than silently flattening or substituting masked pseudo-values that change the estimand. + +A digest or pseudonymous identifier is not automatically treated as anonymous merely because plaintext PII is absent. + +## Invariants / evidence + +1. Canonical measurement/result artifacts omit raw sensitive source text unless their public contract explicitly requires it. +2. Source-free audit/provenance outputs retain enough immutable identity to reconstruct authorized evidence without copying it into every artifact. +3. Cross-object identity/replay checks prevent one person's/source's evidence from being rebound to another artifact. +4. Logging and error paths do not echo provider credentials, arbitrary source content or uncontrolled PII. +5. Downstream hosts can revoke/expire source access according to policy without requiring historical non-content scientific fingerprints to be rewritten, where applicable law/policy allows those fingerprints to remain. +6. Any new raw sensitive field in a canonical contract requires versioned schema/privacy review. +7. No documentation may represent masking removal as exemption from legal, contractual, consent or security obligations. + +## Consequences and trade-offs + +This architecture is more demanding than blanket redaction: data ownership, access and purpose must be explicit. It preserves legitimate scientific/operational utility while minimizing unnecessary proliferation of sensitive content. + +## Alternatives considered + +### Mask every identifier/value before measurement + +Rejected as a universal design because it can invalidate longitudinal, hierarchical, multiple-membership, rater, adjudication and evidence-trace workflows. + +### Persist all raw evidence for maximum reproducibility + +Rejected because it expands sensitive-data scope far beyond the minimum needed for reproducible measurement. + +### Move identity into fast-mlsirm + +Rejected by ADR-0001. The reusable core consumes governed identifiers/references; it does not become the hosted identity database. + +## Reversal / supersession conditions + +A superseding decision is required if the reusable package itself begins owning durable participant/customer identity or hosted retention/deletion lifecycle. That would also require reconsidering ADR-0001 and the logical persistence boundary. + +## Standards/control basis + +This ADR is designed to support evidence for privacy/security management and SOC 2/CSAP-oriented controls without claiming certification. The exact legal/privacy requirements and control mapping are owned by the data controller/host and must be kept current for its jurisdiction and customer obligations. diff --git a/docs/adr/0013-continuous-execution-and-documentation-governance.md b/docs/adr/0013-continuous-execution-and-documentation-governance.md new file mode 100644 index 000000000..9c647d766 --- /dev/null +++ b/docs/adr/0013-continuous-execution-and-documentation-governance.md @@ -0,0 +1,100 @@ +# ADR 0013: Continuous execution and canonical documentation governance + +Status: **Proposed** +Date: 2026-08-09 +Decision owners: fast-mlsirm maintainers +Scope: Repository development loop, architecture documentation, and release evidence + +## Context + +`fast-mlsirm` combines Rust numerical kernels, Python contracts and orchestration, psychometric diagnostics, automated-scoring interfaces, Rubric-to-item authoring, and enterprise evidence generation. Durable decisions have historically appeared across source code, feature documents, pull-request bodies, issues, agent instructions, and research notes. That distribution creates two risks: + +1. an execution loop can stop after describing a blocker or completing one small action even though other safe work exists; and +2. two documentation branches can independently claim authority over PRD, TRD, architecture, ADR, UML, ERD, threat-model, or traceability state. + +Both failures increase integration latency and allow shipped behavior, active-PR behavior, and roadmap intent to be confused. + +## Decision + +### Work-conserving execution + +Every autonomous invocation maintains a fresh executable queue and treats commits, merges, review requests, CI reruns, RCA conclusions, documentation updates, and issue closures as intermediate events. After each event the loop selects the next safe action. A blocked merge or external review blocks only that action. The loop ends only when the finite invocation budget is exhausted or all remaining work is non-actionable under current authority. + +The loop writes only `ContextualWisdomLab/fast-mlsirm`. Repositories with dedicated writer loops are read-only dependencies. Before every write, the exact branch head, live base tip, target blob or ref, relevant review state, and active-writer evidence are refreshed. Source movement or an active writer makes only the affected branch read-only for the remainder of the invocation. + +### One canonical documentation writer + +At most one active branch may serve as the canonical cross-cutting documentation authority. It owns changes to: + +- `docs/PRD.md`; +- `docs/TRD.md`; +- root `ARCHITECTURE.md`; +- `docs/adr/` and its index; +- UML, logical ERD, threat model, and verification/validation views; +- requirements and research traceability; +- documentation coverage and maturity state. + +A competing documentation branch must either contribute unique content to the canonical branch and close, or explicitly supersede the canonical branch with recorded lineage. Parallel authority is prohibited. + +### Documentation maturity vocabulary + +Every capability described by canonical documents is labelled or otherwise unambiguously classified as one of: + +- **IMPLEMENTED / ACCEPTED:** present on protected main and supported by current evidence; +- **ACTIVE PR:** implemented on an unmerged branch and not shipped; +- **PROPOSED:** accepted design direction without completed implementation; +- **PLANNED:** roadmap work without an accepted implementation contract; +- **DOWNSTREAM:** owned by another bounded context such as Psychometrics Commons; +- **REJECTED / SUPERSEDED:** intentionally not part of the governing design. + +Unmerged work must never be promoted to protected-main capability in PRD, TRD, Architecture, README, commercial-readiness, or buyer evidence. + +### Documentation completeness + +A substantive contract change is documentation-complete only when every applicable artifact is updated: + +- public behavior and API documentation; +- PRD and TRD requirements; +- an ADR for durable architecture choices; +- UML or ERD when components, data ownership, cardinality, or lifecycle changes; +- requirements-to-ADR-to-implementation/evidence traceability; +- APA 7 primary-source doctoring and equation-to-source traceability for scientific claims; +- identification, interpretation, migration, and rollback boundaries; +- realistic recovery, benchmark, or operational evidence; +- changelog and version when release semantics change. + +Documentation does not replace implementation. Conversely, unresolved architecture ambiguity is a product defect and may be selected as the next executable work item when product branches are blocked. + +## Consequences + +### Positive + +- Development invocations continue useful work instead of terminating on one blocker. +- PRD, TRD, Architecture, ADR, UML, ERD, threat-model, and traceability views converge on one authority. +- Buyer and reviewer documents cannot silently promote roadmap work to shipped capability. +- Repository-writer conflicts are scoped rather than freezing the entire run. +- Documentation completeness becomes auditable and testable. + +### Costs and limitations + +- Canonical documentation changes may need rebasing after accepted product PRs merge. +- Mature traceability requires ongoing maintenance rather than a one-time documentation sprint. +- The execution loop cannot manufacture external approval, credentials, permissions, or evidence. +- This ADR governs repository process; it does not authorize automatic merge or release outside normal protection. + +## Compliance and verification + +Repository tests should fail when required canonical files, maturity vocabulary, ADR status fields, traceability links, or machine-renderable diagram sources disappear. Pull-request review must compare documentation claims with protected-main public exports and exact-head implementation evidence. + +## Alternatives considered + +1. **Status-report-first automation.** Rejected because it consumes run budget without changing repository state. +2. **One documentation PR per feature with no canonical spine.** Rejected because cross-cutting contracts diverge and supersession becomes circular. +3. **Treat README and PR descriptions as sufficient architecture documentation.** Rejected because neither provides stable decision status, model boundaries, data ownership, or requirements traceability. +4. **Freeze all work while one branch waits for CI or review.** Rejected because unrelated safe actions remain executable. + +## References + +International Organization for Standardization, International Electrotechnical Commission, & Institute of Electrical and Electronics Engineers. (2022). *ISO/IEC/IEEE 42010:2022 Software, systems and enterprise—Architecture description*. + +International Organization for Standardization, International Electrotechnical Commission, & Institute of Electrical and Electronics Engineers. (2018). *ISO/IEC/IEEE 29148:2018 Systems and software engineering—Life cycle processes—Requirements engineering*. diff --git a/docs/adr/README.md b/docs/adr/README.md new file mode 100644 index 000000000..83adc9353 --- /dev/null +++ b/docs/adr/README.md @@ -0,0 +1,67 @@ +# fast-mlsirm Architecture Decision Records + +This directory is the authoritative decision log for architecture, scientific interpretation, trust boundaries, and cross-repository ownership decisions that materially affect `fast-mlsirm`. + +Template for new material decisions: `docs/adr/0000-template.md`. The template is guidance, not a live ADR and is therefore intentionally excluded from the decision index below. + +## Status vocabulary + +- **Accepted** — implemented or governing current protected-main behavior/policy. +- **Proposed** — desired design that is not yet fully implemented or protected-integrated. +- **Deprecated** — retained for history but no longer governs new work. +- **Superseded** — replaced by a named later ADR. + +A conversation, issue, PR body, design note, or paper summary is not an Accepted decision by itself. Accepted ADRs must match current code/policy or explicitly describe an accepted invariant whose implementation is tracked. + +## Decision index + +| ADR | Status | Decision | +|---|---|---| +| [0001](0001-domain-neutral-measurement-boundary.md) | Accepted | `fast-mlsirm` owns reusable measurement/psychometric contracts and kernels; hosted runtime belongs downstream. | +| [0002](0002-rust-first-numerical-ownership.md) | Accepted | Rust owns production psychometric arithmetic; Python validates/orchestrates/reports and retains governed reference/fallback paths. | +| [0003](0003-content-addressed-measurement-contracts.md) | Accepted | Assessment/rubric/scoring artifacts use canonical versioned, content-addressed provenance and replay verification. | +| [0004](0004-governed-rubric-item-bank-lifecycle.md) | Proposed | Build candidate-blind evidence-grounded rubric/item generation into a governed psychometric item-bank lifecycle. | +| [0005](0005-automated-scoring-raters.md) | Accepted | Human and automated scorers are fallible raters; calibration/validation must model rater effects and preserve terminal states. | +| [0006](0006-relation-safe-model-selection.md) | Accepted | Factor retention and structural model choice are distinct; model comparison is relation-safe and fail-closed when distinguishability is unknown. | +| [0007](0007-multilevel-multiple-membership-temporal.md) | Proposed | Multilevel, cross-classified, multiple-membership and temporal structure are first-class; Rust estimators require recovery evidence before production release. | +| [0008](0008-true-parameter-recovery-ci.md) | Accepted | True-parameter recovery/coverage, not correlation alone, is the core scientific CI evidence for numerical estimators. | +| [0009](0009-adaptive-rotation-selection.md) | Accepted | Protected main uses a Rust criterion registry, deterministic multi-start and criterion-neutral empirical selection; no universal best criterion or global-optimum claim. GPU/additional-criterion expansion remains separately gated. | +| [0010](0010-llm-orchestration-and-credentials.md) | Accepted | Model-backed automation uses provider-neutral boundaries, NVIDIA NIM credentials where needed, and never uses Copilot credentials for development scheduling. | +| [0011](0011-canonical-pyo3-public-export-registry.md) | Proposed | Future Rust-backed features converge on one reviewed PyO3/public-export registry instead of competing extension initializers/import rewrites. | +| [0012](0012-purpose-limited-sensitive-data.md) | Accepted | Preserve valid measurement linkage through purpose-limited sensitive-data handling rather than blanket masking or raw-data proliferation. | +| [0013](0013-continuous-execution-and-documentation-governance.md) | Proposed | Keep autonomous work work-conserving and enforce one canonical cross-cutting documentation writer with explicit maturity states. | + +## ADR completeness rule + +A material decision should have an ADR when it changes one or more of: + +- repository/bounded-context ownership; +- public serialized contract or versioning rule; +- psychometric model parameterization/identification/interpretation; +- numerical backend ownership or precision policy; +- PyO3/native binding/public-export authority; +- security/privacy/trust/credential boundary; +- model-selection or scientific acceptance rule; +- lifecycle/release governance; +- cross-repository dependency direction. + +Method-local implementation details that do not change such a decision belong in method documentation or code, not a new ADR. + +## Required ADR sections + +Each ADR should include: + +1. Status and date. +2. Context/problem. +3. Decision. +4. Invariants/acceptance evidence. +5. Consequences and trade-offs. +6. Alternatives considered. +7. Failure/degraded/recovery behavior where applicable. +8. Security/privacy implications where applicable. +9. Compatibility/migration/rollback and reversal/supersession conditions. +10. Verification/release evidence and references where research/standards materially govern the decision. + +## Consistency rule + +Accepted ADRs, protected-main code/tests, `docs/PRD.md`, `docs/TRD.md`, root architecture, UML/ERD, the reusable-core threat model, requirements traceability and release evidence must not contradict one another. A changed accepted decision is superseded through a new ADR rather than silently rewriting history. diff --git a/docs/changelog.d/canonical-architecture-baseline.md b/docs/changelog.d/canonical-architecture-baseline.md new file mode 100644 index 000000000..18ac3f326 --- /dev/null +++ b/docs/changelog.d/canonical-architecture-baseline.md @@ -0,0 +1,14 @@ +# Canonical product and architecture documentation baseline + +## Changed + +- Replaced the stale MVP-only PRD/TRD authority with canonical `docs/PRD.md` and `docs/TRD.md` requirements covering the current measurement, scoring, rubric/item-generation, model-selection, scientific-evidence, interoperability, security, lifecycle, and release boundaries. +- Added root `ARCHITECTURE.md`, a status-bearing ADR corpus, reviewable PlantUML component/sequence/state/deployment views, a logical reusable-domain ERD, and requirements/research traceability matrices. +- Added a canonical documentation authority index, explicit implementation-maturity/completeness matrix, and machine-checkable documentation contract so missing or stale PRD/TRD/ADR/UML/ERD/traceability/security artifacts remain visible release-maintenance debt rather than silently drifting. +- Added a reusable-core threat model covering provider/JSON replay, native/PyO3 input boundaries, resource and non-finite numerical failures, GPU evidence spoofing, supply-chain/self-modifying CI, credential separation, benchmark contamination, privacy/purpose limitation, and scientific-interpretation abuse while leaving hosted HTTP/session/tenant/database threats downstream. +- Added durable ADRs for converging future Rust-backed features on one canonical PyO3/public-export registry and for preserving legitimate sensitive-data linkage through purpose limitation and minimization rather than blanket masking that changes the measurement design. +- Extended requirements traceability with the conversation-wide invariants that human/LLM judges are fallible raters, correlation is not parameter recovery/absolute agreement, latent-space interaction follows substantive dimension/testlet/facet diagnosis, reference-free is not truth-free, and psychometric discrimination is not business or safety criticality. +- Explicitly deprecated the original narrow `docs/prd_trd_summary.md` as an authoritative requirements source while retaining its historical MLS2PLM MVP context. +- Defined the `fast-mlsirm-cjson-v1` fingerprint preimage, SHA-256 binding, null/ordering/Unicode/number rules, and cross-language normative vector instead of leaving canonical serialization as an interoperability assumption. +- Added the persistence-neutral `docs/uml/domain-public-contract.puml` view, indexed every UML source including the compatibility alias, modeled versioned calibration-design inputs as a many-to-many association, and made corrected quarantined items new immutable revisions. +- Added complete APA 7 research records and scope summaries for LLM-RUBRIC, AutoNuggetizer/TREC RAG, EvalGen, the 2025 AutoNuggetizer follow-up and 2026 reflective rubric research, plus NIST AI RMF governance inputs with explicit non-certification language. diff --git a/docs/doctoring/llm_orchestration_test_time_compute.md b/docs/doctoring/llm_orchestration_test_time_compute.md new file mode 100644 index 000000000..62ead3072 --- /dev/null +++ b/docs/doctoring/llm_orchestration_test_time_compute.md @@ -0,0 +1,134 @@ +# LLM Orchestration and Test-Time Compute Doctoring + +Date reviewed: 2026-08-09 +Scope: LLM-backed assessment, rubric/item generation, scoring/judging, and autonomous-development integration policy for `fast-mlsirm`. + +## Decision summary + +`fast-mlsirm` does not assume that deeper multi-agent orchestration is intrinsically better than a single capable model. When an LLM-backed feature uses orchestration, compute allocation is an explicit experimental/design variable. At minimum, validation distinguishes: + +- single-model routing; +- parallel sampling/diversification; +- sequential revision/reflection; +- verifier/aggregator stages; +- heterogeneous worker/model selection; +- role assignment; +- communication/access topology; +- recursive/deeper orchestration; +- reasoning-effort allocation by role/stage; and +- total test-time compute budget. + +Performance claims require comparable-budget ablations where scientifically meaningful. Latency is not the primary objective; correctness, evidence quality, reproducibility, controllability, and bounded failure behavior are. + +## Evidence reviewed + +### Conductor + +Nielsen et al.'s *Learning to Orchestrate Agents in Natural Language with the Conductor* (ICLR 2026; arXiv:2512.04388) trains a relatively small coordinator with reinforcement learning to produce worker-specific instructions and communication topologies over heterogeneous LLM pools. The reported architecture can select itself recursively, creating an adaptive test-time-compute axis. This supports making topology, decomposition, worker selection, and recursion first-class orchestration controls rather than hard-coded workflow assumptions. + +### TRINITY + +Xu et al.'s *TRINITY: An Evolved LLM Coordinator* (ICLR 2026; arXiv:2512.04695) uses a compact coordinator that selects a worker model and assigns roles such as Thinker, Worker, or Verifier over multiple turns. The coordinator is optimized with an evolutionary strategy and is explicitly budget-sensitive. This supports role-specific model/reasoning policies and learned/dynamic delegation, while not implying that the exact TRINITY coordinator should be embedded into this library. + +### Sakana Fugu + +Sakana AI describes Fugu as a production multi-agent orchestration system that coordinates pools of frontier models and can use recursive/adaptive coordination. Fugu is treated here as product/operational evidence from its developer, not as an independent peer-reviewed scientific source. The architectural implication is provider/model substitutability behind a stable orchestration boundary, not a requirement to depend on Fugu. + +### Test-time scaling for agents + +Zhu et al. (2025) systematically study agent test-time scaling strategies including parallel sampling, sequential revision, verification/merging, and rollout diversity. Their results support representing these knobs independently in evaluation instead of treating "more agents" as a single scalar configuration. + +### Equal-compute caution + +Tran and Kiela (2026) report that single-agent systems can match or outperform multi-agent systems on multi-hop reasoning when thinking-token budgets are controlled. This is important counter-evidence against attributing gains to architecture when a multi-agent condition simply spends more inference compute. `fast-mlsirm` therefore requires comparable-budget or explicitly budget-conditioned ablations when choosing between a single scorer/router and deeper orchestration. + +Wunderlich et al. (2026) additionally compare self-consistency, self-refinement, debate, and mixture-of-agents over multiple compute configurations and analyze Pareto efficiency. Their results support reporting quality together with test-time compute rather than collapsing every orchestration design into a single unqualified accuracy figure. + +## Architectural requirements derived from the evidence + +### ORCH-001 — Provider-neutral execution boundary + +The measurement core does not own provider SDKs or credentials. An owning adapter/service receives an immutable task/rubric/scoring contract and returns bounded typed output/provenance. + +### ORCH-002 — Explicit stage graph + +A multi-stage workflow records at least: + +- stage identity/type; +- model/engine identity; +- role; +- input/provenance references; +- allowed tools/access list; +- reasoning-effort/budget parameter where the provider exposes one; +- parent/recursive depth; +- stop/verification outcome; and +- token/call/compute usage where measurable. + +### ORCH-003 — Comparable-budget ablation + +When claiming that orchestration improves evaluation/generation quality, compare against an appropriate single-model or simpler-scaffold baseline under a defensible compute budget. At minimum distinguish total model calls and token/reasoning budget; where provider APIs make exact compute unavailable, document the observable budget proxy and limitation. + +### ORCH-004 — Role-specific reasoning effort + +Reasoning effort may differ by role. Examples: + +- routing/classification: lower or bounded reasoning when deterministic evidence is sufficient; +- item/evidence generation: higher reasoning where construct/evidence synthesis is required; +- verifier/auditor: independent evidence-focused reasoning; and +- final psychometric/numerical calculation: no LLM reasoning; use deterministic Rust core. + +A role label alone does not justify higher effort; ablate it when it materially affects cost/quality. + +### ORCH-005 — Recursion and decomposition bounds + +Recursive/decomposed workflows require explicit maximum depth, stage/call count, token/time/compute budget, and termination behavior. A child-agent failure must have a stable classification rather than producing unbounded retry/decomposition. + +### ORCH-006 — Access-list authority + +An orchestrator may select only from predeclared tools/data/model capabilities authorized by the owning application. Model text cannot expand its own access list, repository permissions, secret scope, merge authority, or release authority. + +### ORCH-007 — Evidence-preserving aggregation + +Aggregation must retain which worker/stage produced which claim/evidence. Majority vote or final synthesis may be an output strategy, but it cannot erase dissent/provenance needed for rater calibration, uncertainty, or audit. + +### ORCH-008 — Psychometric separation + +LLM workers/judges generate observations, evidence units, candidates, or qualitative review signals. Likelihoods, parameter estimation, IRT/MIRT/facet calibration, factor/model comparison, ranking/scoring kernels, DIF/linking, and numerical uncertainty remain deterministic Rust-owned computation. + +### ORCH-009 — Live-test credentials + +Repository live model tests and autonomous-development model calls use GitHub Secret `NVIDIA_NIM_API_KEY`. `COPILOT_GITHUB_TOKEN` is not a model execution credential for this project. Existing independent review-agent credential identities/scopes are not repurposed by product tests. + +### ORCH-010 — Deterministic versus live gates + +Deterministic schema, parser, routing-policy, budget, provenance, and fallback tests run without live model access wherever possible. Live model tests are bounded conformance/quality experiments and cannot be the sole evidence for deterministic contract behavior. + +## Example experimental matrix + +| Dimension | Example conditions | +|---|---| +| orchestration depth | single call; router+worker; worker+verifier; recursive coordinator | +| worker count | 1; 2; 4; adaptive | +| model heterogeneity | same-family; mixed-family/provider | +| roles | no roles; thinker/worker/verifier; custom evidence roles | +| reasoning effort | fixed; role-specific; adaptive | +| decomposition | none; fixed; coordinator-generated | +| access list | read-only evidence; retrieval; bounded tools | +| budget | matched token/call budget; quality-first bounded budget | +| aggregation | single output; listwise verifier; voting; evidence-preserving synthesis | + +Report quality/error/recovery metrics together with call/token/budget evidence. The software should not optimize for speed at the expense of scientific validity, but unbounded compute is also not an acceptable product contract. + +## References — APA 7th + +Nielsen, S., Cetin, E., Schwendeman, P., Sun, Q., Xu, J., & Tang, Y. (2026). *Learning to orchestrate agents in natural language with the Conductor*. International Conference on Learning Representations. arXiv:2512.04388. + +Sakana AI. (2026, April 24). *Sakana Fugu: A multi-agent orchestration system as a foundation model*. + +Tran, D., & Kiela, D. (2026). *Single-agent LLMs outperform multi-agent systems on multi-hop reasoning under equal thinking token budgets*. arXiv:2604.02460. + +Wunderlich, F. V., Kaesberg, L. B., Wahle, J. P., Ruas, T., & Gipp, B. (2026). Multi-agent reasoning improves compute efficiency: Pareto-optimal test-time scaling. In *Proceedings of the 64th Annual Meeting of the Association for Computational Linguistics (Volume 4: Student Research Workshop)* (pp. 1–14). Association for Computational Linguistics. https://doi.org/10.18653/v1/2026.acl-srw.1 + +Xu, J., Sun, Q., Schwendeman, P., Nielsen, S., Cetin, E., & Tang, Y. (2026). *TRINITY: An evolved LLM coordinator*. International Conference on Learning Representations. arXiv:2512.04695. + +Zhu, K., Li, H., Wu, S., Xing, T., Ma, D., Tang, X., Liu, M., Yang, J., Liu, J., Jiang, Y. E., Zhang, C., Lin, C., Wang, J., Zhang, G., & Zhou, W. (2025). *Scaling test-time compute for LLM agents*. arXiv:2506.12928. diff --git a/docs/documentation_coverage.md b/docs/documentation_coverage.md new file mode 100644 index 000000000..d528eca08 --- /dev/null +++ b/docs/documentation_coverage.md @@ -0,0 +1,112 @@ +# Architecture documentation completeness and maintenance matrix + +Status: **Authoritative maintenance audit** +Last reviewed: 2026-08-09 + +This matrix answers whether the repository has enough durable documentation to reconstruct current product intent, technical boundaries, architecture, decisions, logical data relationships, threat model, standards status, verification/validation obligations, scientific evidence and release obligations without mining chat history or PR bodies. + +## Status vocabulary + +- **IMPLEMENTED** — canonical document exists on this branch and describes protected-main behavior/policy without relying on unmerged code. +- **ACTIVE PR** — durable requirement/decision is known, but the corresponding runtime/scientific feature is still under open PR; documentation must not call it released. +- **PLANNED** — accepted/proposed direction with incomplete implementation/evidence. +- **DOWNSTREAM** — owned by Psychometrics Commons or another service; fast-mlsirm documents only the versioned boundary/handoff. +- **REJECTED/SUPERSEDED** — considered or historical design that must not silently return as current authority. + +## Canonical documentation coverage + +| Documentation capability | Before canonical baseline | Current target | Status | Maintenance / remaining gap | +|---|---|---|---|---| +| Product requirements | narrow/stale early-MVP summary plus feature plans | `docs/PRD.md` | IMPLEMENTED | update when buyer workflow/non-goal changes | +| Technical requirements | scattered agent/doctoring/spec rules | `docs/TRD.md` | IMPLEMENTED | update on numerical/runtime/security/resource/release policy changes | +| Root architecture | no root architecture authority | `ARCHITECTURE.md` | IMPLEMENTED | keep current vs proposed explicit | +| Documentation authority/index | feature folders only | `docs/README.md` | IMPLEMENTED | new canonical categories must be linked here | +| Architecture decision log | decisions scattered across AGENTS/plans/PRs | `docs/adr/README.md`, ADRs | IMPLEMENTED | material decision needs status-bearing ADR/supersession | +| Standards status/watch | standards mixed into doctoring without one normative/watch registry | `docs/standards_watch.md` | IMPLEMENTED | verify official edition/publication status before material release claims; drafts stay watch-only until adopted | +| Verification and validation plan | method-specific checks without one evidence hierarchy | `docs/verification_validation_plan.md` | IMPLEMENTED | update when a new estimator/scorer/generalization claim changes recovery, anti-leakage, resource, security or release evidence | +| Component/UML views | no coherent canonical set | `docs/uml/*.puml` | IMPLEMENTED | update when module/dependency/lifecycle changes | +| Logical ERD | absent | `docs/erd/domain-model.puml` | IMPLEMENTED | remains logical/persistence-neutral; physical hosted DB is downstream | +| Requirements traceability | absent | `docs/traceability/requirements-matrix.md` | IMPLEMENTED | update maturity/evidence with material feature PRs | +| Scientific/standards basis | strong but scattered doctoring | `docs/traceability/research-basis.md` + doctoring | IMPLEMENTED | keep APA 7, primary/current standards and preprint status honest | +| Reusable-core threat model | implicit in security feature docs | `docs/security/threat-model.md` | IMPLEMENTED | update on new trust/native/provider/artifact boundaries | +| Documentation contract CI | absent | `tests/test_architecture_documentation_contract.py` | ACTIVE PR | this PR enforces the complete ADR-template/UML/index/source-hygiene set; protected-main integration is required before calling the baseline IMPLEMENTED | +| Release/changelog evidence | managed changelog and release docs already exist | changelog fragment + existing release controls | IMPLEMENTED | render fragment before Ready/merge according to repo policy | +| Operational runbook for hosted product | intentionally not owned here | Psychometrics Commons/operator docs | DOWNSTREAM | link only when a versioned integration requires it | +| Physical DB schema/migrations | intentionally not owned here | Psychometrics Commons/owning host | DOWNSTREAM | do not manufacture ORM from logical ERD | +| Tenant/RBAC/SSO/SCIM/UI/billing | not a reusable core concern | hosted product/services | DOWNSTREAM | retain dependency direction only | + +## Conversation-wide scientific/product coverage + +| Work family | Documentation maturity | Runtime/evidence maturity | State | +|---|---|---|---| +| Fallible human/LLM raters and many-facet calibration | PRD/TRD + ADR-0005 + traceability + V&V | baseline facets/scoring exists; generalized discrimination/range/drift remains incremental | IMPLEMENTED / PLANNED extensions | +| Correlation is not parameter recovery/agreement | ADR-0008 + PRD/TRD + research basis + V&V | recovery/simulation evidence exists across model families | IMPLEMENTED governance | +| Reference-free RAG measurement | PRD/TRD + traceability + V&V | no single canonical end-to-end RAG observation adapter/bank workflow on protected main | PLANNED | +| Dynamic evidence-grounded rubric generation | PRD/TRD + ADR-0003/0004 + item UML/ERD + V&V | strong rubric/generation/audit/pilot primitives exist | IMPLEMENTED primitives / PLANNED closed loop | +| Governed item-bank lifecycle | ADR-0004 + state diagram + V&V | pilot/admission/lifecycle pieces exist; unified approve/active/link/drift/exposure/retire workflow remains incomplete | PLANNED/partial | +| Bifactor / higher-order / testlet / two-tier / many-facet relation | ADR-0006 + model-selection UML/research basis + V&V | family-specific features/evidence vary | IMPLEMENTED policy / partial family coverage | +| Latent-space residual interaction | architecture/PRD/TRD + V&V | supported model family exists, but must follow substantive dimension/testlet/facet diagnosis | IMPLEMENTED with interpretation gate | +| Formal non-nested distinguishability | ADR-0006 + traceability + V&V | fail-closed comparison exists where formal family inputs are incomplete; full score/information metadata still needed | PLANNED extension | +| Adaptive rotation criterion selection | ADR-0009 + PRD/TRD + protected-main adaptive-rotation doctoring + V&V | protected main exposes the Rust-backed criterion registry, deterministic multi-start optimizer, criterion-neutral selector/report surfaces and public Python API; GPU batching, additional criteria and broader recovery evidence remain future increments | IMPLEMENTED / PLANNED extensions | +| Multilevel/multiple-membership/cross-classified contracts | ADR-0007 + UML/ERD/PRD/TRD + V&V | active PR exists; dedicated namespace not accepted until protected-main evidence | ACTIVE PR | +| Temporal/longitudinal/drift models | ADR-0007 + PRD/TRD + V&V | design/primitives exist; continuous-time estimator claims require separate recovery | PLANNED/partial | +| Automated essay scoring calibration/validation | PRD/TRD + ADR-0005 + V&V | governed essay contracts/calibration/validation/reporting exist; rater-range/discrimination/drift extensions remain active | IMPLEMENTED baseline / ACTIVE extensions | +| Enterprise issue measurement | PRD/TRD + traceability + V&V | reusable evidence/calibration adapters exist | IMPLEMENTED measurement; causal intervention utility DOWNSTREAM/policy | +| Factor retention | PRD/TRD + ADR-0006 + V&V | diagnostics exist; unified evidence API remains a gap | PLANNED extension | +| Rust-first numerical core / CPU+GPU parity | ADR-0002 + TRD + V&V | current model-specific support/evidence varies by kernel | IMPLEMENTED architecture, feature-specific evidence required | +| Canonical PyO3/public-export registry | ADR-0011 | current exports work, but future feature PRs must converge rather than creating competing initialization schemes | PLANNED hardening | +| PII/purpose limitation | ADR-0012 + threat model + PRD/TRD + V&V abuse cases | source-free/digest-based contracts exist where possible; hosted access/retention is downstream | IMPLEMENTED reusable policy / DOWNSTREAM operations | +| LLM orchestration/model credentials | ADR-0010 + V&V | repository/org automation policy exists | IMPLEMENTED governance | +| Continuous execution and canonical docs ownership | ADR-0013 + documentation contract | repository process contract is proposed by the canonical docs PR; runtime scheduler state is external to shipped package capability | PROPOSED governance | + +## P0 documentation gaps + +A P0 gap blocks treating the architecture package as complete: + +- the canonical documentation baseline or its complete contract test is still only on an open PR rather than protected main; +- missing canonical PRD or TRD; +- missing root architecture boundary; +- missing ADR index/status for a material cross-cutting decision; +- missing standards status registry when a release or buyer claim relies on standards; +- missing V&V plan for a new scientific/scoring/generalization claim; +- missing UML/ERD or persistence-neutral domain/public-contract view for a major public-contract or lifecycle change; +- missing reusable-core threat model after a new trust boundary; +- traceability that falsely marks an active/planned feature as protected-main implemented; +- a stale historical summary that competes with the canonical requirements source; or +- architecture claims that move hosted product DB/HTTP/tenant/RBAC ownership into fast-mlsirm without a superseding ADR. + +## P1 documentation gaps + +P1 gaps do not automatically block unrelated development, but must be repaired before release of the affected capability: + +- missing method-specific doctoring/primary source; +- missing recovery/scoreability interpretation boundary; +- missing V&V evidence class or resampling/generalization unit for the changed claim; +- missing failure/recovery/rollback instructions for a changed public artifact; +- missing privacy/security abuse case for new provider/native/artifact surfaces; or +- missing changelog/release evidence for a user-visible accepted capability. + +## P2 improvements + +- richer rendered architecture diagrams/site navigation; +- downstream hosted-workbench Figma/UX links; +- automated link/PlantUML rendering checks beyond the current source contract; +- generated traceability views from contract metadata; and +- buyer/operator views that consume these artifacts without becoming a second source of truth. + +## Maintenance gate + +Every material PR should answer: + +1. Did product requirements or non-goals change? +2. Did a technical invariant/trust/resource/release rule change? +3. Did a durable architecture/scientific decision change or need supersession? +4. Did an applicable published standard edition/status or watch item change? +5. Did the required software/numerical/scientific V&V evidence or generalization unit change? +6. Did component/data/lifecycle/deployment/ERD views change? +7. Did the threat model gain a new asset/actor/abuse case? +8. Did an implementation maturity state change (PLANNED/ACTIVE -> protected-main IMPLEMENTED)? +9. Did source/test evidence change enough that traceability is stale? +10. Is the changelog/release evidence synchronized? + +If yes, update the corresponding canonical document in the same PR or record a precise downstream/no-change justification. Documentation drift is treated as a repository defect rather than post-release cleanup. diff --git a/docs/erd/domain-model.puml b/docs/erd/domain-model.puml new file mode 100644 index 000000000..08a0bac5e --- /dev/null +++ b/docs/erd/domain-model.puml @@ -0,0 +1,149 @@ +@startuml +hide circle +skinparam linetype ortho +skinparam shadowing false + +title fast-mlsirm logical reusable-domain model + +entity assessment_specification { + * assessment_id : opaque string + * assessment_version : semantic version + * assessment_fingerprint : sha256 + -- + construct_id : string + rubric_fingerprint : sha256 +} + +entity rubric_specification { + * rubric_id : opaque string + * rubric_version : semantic version + * rubric_fingerprint : sha256 + -- + construct_id : string + response_format : enum +} + +entity rubric_level { + * rubric_level_id : opaque string + -- + rubric_fingerprint : sha256 + score_value : ordinal value + label_text : string +} + +entity item_blueprint { + * blueprint_id : opaque string + * blueprint_fingerprint : sha256 + -- + rubric_fingerprint : sha256 + task_family : string + evidence_mode : enum + difficulty_band : enum +} + +entity item_candidate { + * candidate_id : opaque string + * candidate_fingerprint : sha256 + -- + blueprint_fingerprint : sha256 + contract_fingerprint : sha256 + lifecycle_state : enum +} + +entity engine_descriptor { + * engine_id : opaque string + * engine_fingerprint : sha256 + -- + engine_kind : enum + model_version : string +} + +entity scoring_request { + * request_id : opaque string + * request_fingerprint : sha256 + -- + assessment_fingerprint : sha256 + rubric_fingerprint : sha256 + task_id : opaque string + task_revision_fingerprint : sha256 + respondent_id : opaque string + response_id : opaque string + response_content_fingerprint : sha256 +} + +entity score_observation { + * observation_id : opaque string + * observation_fingerprint : sha256 + -- + request_fingerprint : sha256 + engine_fingerprint : sha256 + criterion_id : opaque string + observation_state : enum + score_value : optional ordinal value +} + +entity calibration_design { + * design_id : opaque string + * design_fingerprint : sha256 + -- + criterion_id : opaque string + assessment_fingerprint : sha256 + connected_state : enum +} + +entity calibration_design_input { + * design_input_id : opaque string + * design_input_fingerprint : sha256 + -- + design_fingerprint : sha256 + observation_fingerprint : sha256 + inclusion_role : enum + source_revision : semantic version +} + +entity calibration_report { + * report_id : opaque string + * report_fingerprint : sha256 + -- + design_fingerprint : sha256 + model_family : string + convergence_state : enum +} + +entity item_bank_entry { + * bank_entry_id : opaque string + * bank_revision_fingerprint : sha256 + -- + candidate_fingerprint : sha256 + lifecycle_state : enum + calibration_report_fingerprint : sha256 +} + +entity model_comparison_evidence { + * comparison_id : opaque string + * comparison_fingerprint : sha256 + -- + model_relation : enum + selection_state : enum +} + +rubric_specification ||--|{ rubric_level : defines +rubric_specification ||--o{ assessment_specification : referenced_by +rubric_specification ||--o{ item_blueprint : compiles +item_blueprint ||--o{ item_candidate : generates +assessment_specification ||--o{ scoring_request : governs +engine_descriptor ||--o{ score_observation : produces +scoring_request ||--o{ score_observation : binds +calibration_design ||--o{ calibration_design_input : selects +score_observation ||--o{ calibration_design_input : included_in +calibration_design ||--o{ calibration_report : fitted_as +item_candidate ||--o{ item_bank_entry : versioned_as +calibration_report ||--o{ item_bank_entry : supports +calibration_report }o--o{ model_comparison_evidence : compared_by + +note bottom +This is a logical domain ERD, not a hosted database schema. +Persistence, tenant tables, participant identity, sessions and consent are owned downstream. +end note + +@enduml diff --git a/docs/prd_trd_summary.md b/docs/prd_trd_summary.md index 131f68af8..2055d281a 100644 --- a/docs/prd_trd_summary.md +++ b/docs/prd_trd_summary.md @@ -1,96 +1,36 @@ -# fast-mlsirm PRD/TRD Summary +# fast-mlsirm PRD/TRD Summary — Deprecated -## Product Goal +Status: **Deprecated as an authoritative requirements source** +Last reviewed: 2026-08-09 -`fast-mlsirm` provides fast simulation, fitting, and recovery diagnostics for -Multidimensional Latent Space Item Response Models, especially MLS2PLM: +This file was the original narrow MVP summary for an MLS2PLM-focused prototype. The product has since expanded to governed assessment/rubric/scoring contracts, rubric-centered item generation, automated-scoring calibration/validation, enterprise adapters, broader diagnostics and release evidence. Several statements in the historical summary are therefore stale, including the earlier NumPy-primary/Rust-optional architecture and the earlier roadmap that treated ordinal/GPU work as unexplored future scope. -```text -logit P(Y_pi = 1) = a_i * theta_p,d(i) + b_i - gamma * distance(xi_p, zeta_i) -``` - -The package is aimed at psychometrics, educational measurement, mental-health -assessment, item diagnostics, adaptive testing research, and production-scale -binary response scoring pipelines. - -For sale and support purposes, the current product is a commercial beta for -technical users. It can be packaged, verified, and supported for the documented -local API/CLI workflows, but it is not a regulated decision product, hosted -platform, or full ordinal/Bayesian estimation system. - -## MVP Scope - -Must have: - -- Canonical MLS2PLM simulation. -- `gamma=0` no-CD simulation. -- `MIRT`, `MLSRM`, and `MLS2PLM` model constraints. -- Missing response exclusion. -- Likelihood and analytic gradient. -- Adam and L-BFGS-style optimizers. -- Procrustes alignment and recovery reports. -- Python API and CLI. -- Rust core formulas for likelihood and gradient. -- Optional PyO3/maturin binding for using the Rust likelihood and gradient from - Python fitting. - -Explicitly out of MVP: - -- Full HMC/NUTS Bayesian sampling. -- Ordinal graded response models. -- Real-time adaptive testing. -- GUI dashboards. -- Automatic psychological construct naming. - -## Architecture +Use the following canonical documents instead: -The intended architecture is Python API first, Rust numerical core second: +- [Product Requirements Document](PRD.md) +- [Technical Requirements Document](TRD.md) +- [Root architecture description](../ARCHITECTURE.md) +- [Architecture Decision Record index](adr/README.md) +- [UML diagram index](uml/README.md) +- [Logical reusable-domain ERD](erd/domain-model.puml) +- [Requirements-to-implementation traceability](traceability/requirements-matrix.md) +- [Research-to-architecture traceability](traceability/research-basis.md) -```text -python/fast_mlsirm/ - config, simulation, objective, fit, diagnostics, cli - -crates/mlsirm-core/ - model structs, stable likelihood, analytic gradients, Rust tests - -crates/fast-mlsirm-py/ - PyO3 module exposed as fast_mlsirm._core -``` - -The default Python backend is vectorized NumPy. The optional Rust backend uses -the same core formula through a PyO3/maturin extension and can be selected with -`FitConfig(backend="rust")`, `FitConfig(backend="auto")`, or -`fast-mlsirm fit --backend`. Source and editable installs build that extension -with maturin and therefore require a Rust toolchain; NumPy remains the default -runtime backend after installation. The PyO3 crate is built through maturin and -validated through Python backend parity tests, while `cargo test --workspace` -covers the standalone Rust core. +## Historical scope retained for context -## Formula Contract +The initial product goal was fast simulation, fitting and recovery diagnostics for the simple-structure MLS2PLM specialization: -For item `i` assigned to factor `d_i`: +> The following equation is historical MVP notation only. It is retained to +> explain the origin of this deprecated summary and is not the current runtime +> contract. Current simple-structure MLS2PLM usage follows the canonical +> parameterization in [`docs/papers/mls2plm-canonical-equations.md`](papers/mls2plm-canonical-equations.md), +> including `a_i = exp(alpha_i)`, `gamma = exp(tau)`, the declared distance +> term, and the package's finite-value/epsilon rules. ```text -eta_pi = exp(alpha_i) * theta_p,d_i + b_i - exp(tau) * r_pi -r_pi = sqrt(sum_k (xi_pk - zeta_ik)^2 + eps) -loss = softplus(eta_pi) - y_pi * eta_pi -``` - -The NLL gradient uses: - -```text -e_pi = sigmoid(eta_pi) - y_pi +logit P(Y_pi = 1) = a_i * theta_p,d(i) + b_i - gamma * distance(xi_p, zeta_i) ``` -and applies L2 regularization to `theta`, `xi`, `zeta`, `b`, `alpha`, and -`tau` where those parameters are active for the selected model. - -## Roadmap +The early architecture separated a Python API from a Rust numerical core and PyO3 binding. That separation remains conceptually valid, but the current governing architecture is stricter: production psychometric arithmetic is Rust-first, while Python owns validation/orchestration/reporting and transparent governed reference/fallback paths. -1. Stabilize Python reference formulas and tests. -2. Maintain NumPy/Rust objective parity through PyO3/maturin tests. -3. Add block-mode likelihood/gradient execution. -4. Add benchmark harness and repeated recovery-grid runner. -5. Add sparse/missing optimized kernels. -6. Explore JAX/GPU and ordinal response extensions as separate model/runtime - design work. +The old MVP roadmap is intentionally not reproduced here because it is no longer the authoritative backlog. Current bounded requirements and proposed work are recorded in `PRD.md`, `TRD.md`, ADR statuses, open issues/PRs, and the traceability matrix. diff --git a/docs/security/threat-model.md b/docs/security/threat-model.md new file mode 100644 index 000000000..76b8df580 --- /dev/null +++ b/docs/security/threat-model.md @@ -0,0 +1,119 @@ +# fast-mlsirm reusable-component threat model + +Status: **Authoritative reusable-core threat model** +Last reviewed: 2026-08-09 + +This document models threats owned by the reusable `fast-mlsirm` package and its CI/release boundaries. It deliberately does **not** duplicate hosted-product threats for HTTP endpoints, sessions, consent, tenant/RBAC administration, product databases, billing, customer data-rights workflows or deployment control planes; those belong to Psychometrics Commons or the service that owns them. + +## 1. Assets and trust boundaries + +### Protected assets + +- exact assessment/rubric/scoring contract identity and provenance; +- unmodified item/evidence/calibration/model/recovery artifacts; +- psychometric formula/parameterization/gradient semantics; +- exact numerical output, convergence and uncertainty evidence; +- model relation, scoreability and validity boundaries; +- Python↔PyO3↔Rust memory/shape/type contracts; +- provider/model credentials and reviewer/merge credentials as **separate authorities**; +- CI/release source identity, SBOM/provenance and exact tested artifact; +- benchmark/calibration bank confidentiality where required; +- sensitive source/evidence data only to the extent the reusable computation is explicitly authorized to receive it. + +### Principal trust boundaries + +```text +caller Python objects + -> bounded Python validation + -> PyO3/numpy native boundary + -> Rust numerical core + +rubric/blueprint + -> generation request + -> external/untrusted provider output + -> strict deterministic parser + -> semantic screening + -> psychometric pilot/calibration + +PR source + -> CI/security/scientific evidence + -> independent review/branch protection + -> package/release artifact +``` + +## 2. Threat inventory and controls + +| Threat | Failure mode | Required control | Evidence / recovery | +|---|---|---|---| +| Untrusted JSON/member ambiguity | duplicate keys, NaN/Infinity, unknown fields, deep/oversized payload create parser disagreement | strict bounded JSON, duplicate-member rejection, finite numbers, closed schemas, depth/count/bytes ceilings | hostile parser tests; reject before candidate construction | +| Provider replay/provenance substitution | output for rubric/blueprint/request A is rebound to B | complete content fingerprints, exact echoed identities, source/cardinality checks, recomputed candidate/execution identities | replay/forgery tests; regenerate as new identity | +| Evidence/source fabrication | provider cites undeclared source/span or a span absent from source | exact source ids/digests, bounded verbatim-span validation, later semantic entailment screening | reject structurally; quarantine semantically invalid candidate | +| Benchmark contamination / double dipping | a candidate response defines the rubric used to score itself | candidate-blind benchmark generation; candidate-aware discovery only with separated cross-fit discovery/scoring evidence | fold-isolation tests and separate bank provenance | +| Package artifact mutation | built wheel/sdist/SBOM/provenance no longer corresponds to tested head | exact-head build identity, immutable artifact digests, SBOM/attestation/release-acceptance verification | abort release; rebuild from protected head | +| Self-modifying CI / source laundering | PR-controlled workflow rewrites source and pushes the implementation it claims to test | CI validates reviewed source only; no branch-local self-removing/encoded-patch writer for scientific implementation; least privilege | source-writer absence/permissions tests; close unsafe workflow path | +| PyO3/native shape/type confusion | malformed dimensions/types reach unsafe/native arithmetic or wrong marshalling | bounded Python validation plus Rust validation at native trust boundary; checked products before allocation; typed results | Python/Rust hostile input tests, Miri/fuzz where useful, fail before computation | +| Numeric overflow/non-finite output | dimension products, exponentials, likelihoods or diagnostics overflow and silently return misleading values | checked integer/byte products, finite input/intermediate/output contracts, stable log-domain/scaled algorithms where method permits | boundary/property tests; fail closed with bounded diagnostic | +| CPU oversubscription/resource exhaustion | nested thread pools, enormous workspaces or unconstrained starts/studies starve host/CI | coarse-grained Rust parallelism, explicit worker/batch/workspace ceilings, no unbounded iterable materialization, separate heavy studies from PR smoke | resource-bound tests/metrics; reject oversized workload before allocation | +| GPU evidence spoofing | CPU/software fallback is reported as GPU success | backend/device identity in evidence; no-skip device test; CPU/GPU result parity with declared tolerance | fail GPU claim when device kernel did not execute | +| Scientific model misuse | flexible model fit is represented as validity/scoreability or correlation as recovery | relation-aware comparison, identification/recovery/coverage, scoreability, DIF/invariance/local-dependence and interpretation gates | return indeterminate/not-scoreable instead of preferred/valid | +| Rater/judge authority confusion | LLM/human score treated as truth or model can approve its own PR | fallible-rater contracts; reviewer/merge credentials separate from model credentials; independent review policy | many-facet/agreement/drift evidence; repository protection remains authoritative | +| Credential cross-contamination | provider subprocess receives repo-write/reviewer/OIDC secrets not needed for generation | explicit secret allowlist, stripped child environments, NVIDIA NIM provider credential separated from reviewer/merge authority | workflow/provider environment tests; fail closed when required model secret absent | +| Privacy overcollection | raw PII/source text replicated into durable artifacts/logs/provider errors | purpose limitation, data minimization, opaque references/digests, provider-exception redaction, downstream data-owner authorization/retention | source-free audit tests; revoke source access without mutating non-content provenance where policy allows | +| Blanket masking destroys scientific design | masking prevents longitudinal/context/rater/participant linkage and silently changes estimand | do not substitute masked pseudo-values; use authorized linkage/identity separation and minimum required protected attributes | fail if required authorized linkage unavailable rather than alter design | +| Supply-chain dependency compromise | dependency/action/tool change executes attacker code or changes scientific build | immutable action/source pins where practical, lockfiles, Security Scan/SAST/OSV/SBOM, package/release acceptance | update/remove dependency; narrow documented false-positive suppression only | +| Documentation/decision drift | code and PRD/TRD/ADR/UML/ERD/threat/release docs contradict each other | canonical authority map, status-bearing ADRs, traceability and documentation-contract CI | block release until corrected/superseded | +| Scientific-integrity recovery failure | test threshold, benchmark or model relation is changed after observing an inconvenient result | fail-first tests, prospective recovery targets, exact-head provenance, explicit superseding ADR/doctoring for method change | preserve failed evidence; fix model/design or justify new method independently | + +## 3. Abuse cases + +### A. Malicious item generator returns plausible but rebound JSON + +An external model returns a syntactically valid item but echoes a different rubric/blueprint id and invents a supporting source span. The parser must reject before construction. A provider verdict that it is “valid” has no authority. + +### B. Valid structure, invalid meaning + +A candidate references a real source span but the span does not entail the keyed answer, or the item is ambiguous. Structural acceptance is insufficient; semantic/content screening quarantines the item before pilot/operational use. + +### C. Oversized numerical request + +A caller supplies valid-looking arrays whose derived pairwise/workspace dimensions exceed safe memory. Checked size/byte products must reject before node grids, dense distance matrices, or other dominant allocations are created. + +### D. Misleading high model fit + +A bifactor/latent-space model improves in-sample fit but specific scores are not reliable/recoverable or the extra structure is unsupported out of sample. The system must not turn fit into a released scoring interpretation. + +### E. Review model shares repository write authority + +A model subprocess is given the same credential used for independent approval/merge and can write its own acceptance evidence. This violates the authority boundary even if the model is trustworthy; model and reviewer/merge credentials must remain separate. + +## 4. Privacy and PII strategy + +`fast-mlsirm` preserves legitimate analytical linkage while minimizing raw sensitive-content proliferation: + +- exact sensitive values are used only for an explicit approved computation; +- identity resolution, customer/participant lifecycle, residency, encryption keys, retention/deletion and data-subject handling remain owned by the downstream data controller/service; +- durable reusable artifacts prefer opaque ids, digests and bounded metadata; +- protected attributes used for DIF/fairness remain governed data and are not casually copied into reports; +- a digest is not assumed anonymous merely because it is not plaintext; and +- absence of authorized linkage is an error when the scientific design requires linkage—it is not repaired by silently flattening or pseudorandom masking. + +## 5. Scientific misuse and human governance + +The package provides measurement evidence, not autonomous consequential decisions. A downstream system that uses scores for employment, admission, insurance, credit, diagnosis/treatment, discipline, legal rights or other high-impact decisions must establish the appropriate validation, human/governance, authorization, monitoring and legal basis outside this reusable-core threat model. + +Enterprise issue priority additionally requires causal outcome/intervention/cost/utility policy; psychometric discrimination is not business/safety criticality. + +## 6. Availability and degraded modes + +- Provider rate limits/outages block only the provider-backed action; deterministic validation/numerical work continues where possible. +- GPU unavailability can trigger a documented CPU path where the API permits it, but GPU evidence becomes unavailable rather than successful. +- External reviewer/check latency is not a reason to mutate scientific acceptance criteria; unrelated safe work continues. +- A malformed or disconnected measurement design fails before fitting when detectable; the library does not silently coerce it to a simpler estimand. + +## 7. Security maintenance gate + +Material PRs must update this threat model when they introduce a new trust boundary, persistence/credential authority, native execution surface, artifact mutation path, provider/evidence flow, or scientific interpretation that changes the abuse cases above. Hosted product threats are linked rather than duplicated. + +## 8. Standards and evidence basis + +The architecture maps to the repository's current standards/research basis in `docs/traceability/research-basis.md`, including secure software development, AI risk/governance, architecture description, requirements engineering and testing/measurement standards. This document supports SOC 2/CSAP readiness evidence but does not claim certification. diff --git a/docs/standards_watch.md b/docs/standards_watch.md new file mode 100644 index 000000000..aef3b6ca5 --- /dev/null +++ b/docs/standards_watch.md @@ -0,0 +1,88 @@ +# Standards and research watch + +## Status and use + +This registry is part of the proposed canonical architecture baseline. It separates current published sources used as governing references from drafts, revisions, and emerging research that are monitored but not treated as normative. Catalog status and edition numbers must be revalidated from the issuing body's official publication record before every release that claims alignment. + +`fast-mlsirm` does not claim certification or conformance merely because a standard is cited. Applicability, control implementation, independent assessment, operating evidence, and downstream responsibilities remain separate questions. + +## Governing published references + +| Area | Published reference used by the architecture baseline | Repository use | +|---|---|---| +| Requirements engineering | ISO/IEC/IEEE 29148:2018 | PRD/TRD quality, requirement attributes, traceability, verification and change control | +| Architecture description | ISO/IEC/IEEE 42010:2022 | stakeholders, concerns, viewpoints, views, correspondence and decision records | +| Product quality | ISO/IEC 25010:2023 | functional suitability, performance efficiency, compatibility, interaction capability, reliability, security, maintainability, flexibility and safety quality evidence | +| AI management system | ISO/IEC 42001:2023 | lifecycle governance, roles, change control, documentation and operating evidence; no certification claim | +| AI impact assessment | ISO/IEC 42005:2025 | downstream use-context impact assessment and recorded human/governance decisions | +| AI risk management | ISO/IEC 23894:2023 | risk identification, analysis, treatment, monitoring and communication | +| AI risk framework | NIST AI RMF 1.0 | Govern, Map, Measure and Manage control framing | +| Generative AI profile | NIST AI 600-1 | model/provider risks, content provenance, human oversight, evaluation and incident considerations | +| Testing validity | *Standards for Educational and Psychological Testing* (2014) | score interpretation, fairness, reliability/precision, validation and use boundaries | +| Web accessibility | WCAG 2.2 | standalone HTML report semantics, focus, contrast, reflow, target size, status messages and non-hover exact-value channels | + +## Normative-versus-watch policy + +1. A published, applicable edition may govern a requirement or ADR after its exact edition and official source are recorded. +2. A committee draft, working draft, public consultation, amendment proposal, revision project, or announced future edition is a **watch item**, not a normative requirement. +3. When a new edition is published, maintainers perform a delta assessment before changing repository requirements. The assessment records superseded clauses, migration impact, implementation evidence, release impact, and downstream ownership. +4. A citation does not establish implementation, certification, legal compliance, or suitability for a regulated decision. +5. Scientific method claims use primary peer-reviewed papers where available. Package documentation or legacy software can be a numerical comparison source but does not replace primary methodological validation. + +## Active revision projects verified for this baseline + +These entries make known revision activity explicit so a release does not mistake a still-current published edition for an abandoned line of work. They remain **non-normative watch evidence** until a replacement is published and adopted through the repository decision process. + +| Published baseline retained | Current revision/watch evidence | Repository treatment | +|---|---|---| +| ISO/IEC/IEEE 29148:2018 | ISO project `ISO/IEC/IEEE DIS 29148` (`standard/94091`) reached stage **30.99** on **2026-07-10**, recorded by ISO as CD approved for registration as DIS | Keep 29148:2018 as the current published requirements-engineering baseline. Recheck the ISO project before release; do not treat the DIS as a published replacement. If a new edition publishes, perform a requirement/traceability delta assessment and adopt it only through an ADR or equivalent reviewed change. | +| NIST AI RMF 1.0 (NIST AI 100-1) | NIST's AI RMF program states in 2026 that AI RMF 1.0 is being revised | Keep AI RMF 1.0 as the published framework baseline and NIST AI 600-1 as the published Generative AI Profile. Track the revision, but do not freeze an unpublished successor into normative repository behavior. | + +## Active watch items + +The following topics are monitored because revisions or new evidence may change future requirements. Their exact project identifiers and publication state must be checked against official sources at review time. + +- the ISO/IEC/IEEE DIS 29148 revision project and any later publication replacing ISO/IEC/IEEE 29148:2018; +- revisions to architecture-description standards, including any successor work to ISO/IEC/IEEE 42010:2022; +- updates to the ISO/IEC 25000 SQuaRE family affecting measurement or quality models; +- implementation guidance and conformity-assessment practice for ISO/IEC 42001 and ISO/IEC 42005; +- the announced NIST AI RMF 1.0 revision, Generative AI Profile updates, implementation resources, and evaluation guidance; +- revision of the *Standards for Educational and Psychological Testing*; +- later W3C accessibility recommendations and techniques beyond WCAG 2.2; +- current primary evidence on LLM-as-a-Judge calibration, dynamic rubrics, automatic item generation, test-time scaling, multi-agent verification, and correlated evaluator error; +- current primary evidence for multilevel, multiple-membership, longitudinal, testlet, bifactor, two-tier, latent-space, factor-retention and model-selection methods. + +## Release review checklist + +Before a release or buyer evidence packet uses a standards claim: + +- verify official publication status, edition, title, and issuing body; +- recheck explicit revision projects above and record whether their publication state changed; +- identify the exact requirement or concern affected; +- link the requirement to an ADR, implementation, test, and evidence artifact; +- distinguish core-library obligations from Psychometrics Commons or another downstream host; +- record gaps, compensating controls, migration needs, and rejected applicability; +- remove language that implies certification, conformity, safety, fairness, validity, or legal compliance without independent evidence; +- preserve the published edition as governing until an adopted replacement is approved through an ADR. + +## APA 7 reference record + +American Educational Research Association, American Psychological Association, & National Council on Measurement in Education. (2014). *Standards for educational and psychological testing*. American Educational Research Association. + +International Organization for Standardization & International Electrotechnical Commission. (2023a). *ISO/IEC 23894:2023 Information technology—Artificial intelligence—Guidance on risk management*. + +International Organization for Standardization & International Electrotechnical Commission. (2023b). *ISO/IEC 25010:2023 Systems and software engineering—Systems and software quality requirements and evaluation (SQuaRE)—Product quality model*. + +International Organization for Standardization & International Electrotechnical Commission. (2023c). *ISO/IEC 42001:2023 Information technology—Artificial intelligence—Management system*. + +International Organization for Standardization & International Electrotechnical Commission. (2025). *ISO/IEC 42005:2025 Information technology—Artificial intelligence—AI system impact assessment*. + +International Organization for Standardization, International Electrotechnical Commission, & Institute of Electrical and Electronics Engineers. (2018). *ISO/IEC/IEEE 29148:2018 Systems and software engineering—Life cycle processes—Requirements engineering*. + +International Organization for Standardization, International Electrotechnical Commission, & Institute of Electrical and Electronics Engineers. (2022). *ISO/IEC/IEEE 42010:2022 Software, systems and enterprise—Architecture description*. + +National Institute of Standards and Technology. (2023). *Artificial intelligence risk management framework (AI RMF 1.0)* (NIST AI 100-1). https://doi.org/10.6028/NIST.AI.100-1 + +National Institute of Standards and Technology. (2024). *Artificial intelligence risk management framework: Generative artificial intelligence profile* (NIST AI 600-1). https://doi.org/10.6028/NIST.AI.600-1 + +World Wide Web Consortium. (2024, December 12). *Web content accessibility guidelines (WCAG) 2.2*. https://www.w3.org/TR/WCAG22/ diff --git a/docs/traceability/requirements-matrix.md b/docs/traceability/requirements-matrix.md new file mode 100644 index 000000000..5f944ba5a --- /dev/null +++ b/docs/traceability/requirements-matrix.md @@ -0,0 +1,94 @@ +# Requirements, decisions, implementation and evidence matrix + +Status: **Authoritative traceability baseline** +Last reviewed: 2026-08-09 + +This matrix makes the major product requirements discoverable without reconstructing decisions from chat history or PR bodies. It deliberately distinguishes **protected-main implementation**, **active/open work**, **future research**, and **downstream ownership**. + +| Requirement family | PRD / TRD IDs | ADR | Protected-main implementation/evidence | State | +|---|---|---|---|---| +| Repository ownership | PRD-PRN-007, TRD-BOUND-001/002 | ADR-0001 | `AGENTS.md`, `CLAUDE.md`; package boundary in `python/fast_mlsirm/` | Accepted | +| Rust numerical ownership | PRD-PRN-002, TRD-NUM-001..006 | ADR-0002 | `crates/mlsirm-core/`, `crates/fast-mlsirm-py/`, backend/parity tests | Accepted | +| Canonical PyO3/public exports | TRD-API / numerical integration | ADR-0011 | current package exports exist; future Rust feature modules must converge on one registry instead of independent initializer/import rewrites | Proposed hardening | +| Assessment/scoring contracts | PRD-FR-001..004, TRD-API/P-ROV/SCR | ADR-0003, ADR-0005 | `python/fast_mlsirm/scoring/contracts.py` and bounded submodules | Accepted | +| Rubric/blueprint/generation | PRD-FR-010..014, TRD-RUB-001..006 | ADR-0003, ADR-0004 | `python/fast_mlsirm/rubric/`: models/compiler/contracts/generation/candidates/audit/pilot modules | Partial / evolving | +| Governed item bank lifecycle | PRD-FR-010, FR-080 | ADR-0004 | Pilot/admission/lifecycle primitives exist; complete approved-bank linking/exposure/monitoring/retirement workflow remains evolving; issue #609 tracks the planned canonical closed loop | Proposed/partial | +| Automated essay scoring | PRD-FR-020..023, TRD-SCR | ADR-0005 | governed essay score, calibration, validation and HTML report modules/tests from v0.7.0-era work | Accepted baseline / evolving diagnostics | +| Enterprise issue evaluation | PRD-FR-020..023 | ADR-0005 | `fast_mlsirm.scoring.enterprise_issue` adapters, governed observations/calibration/reporting | Accepted reusable adapter; causal action/utility policy is downstream | +| Reference-free RAG measurement | PRD-FR-030..033, TRD-RAG | ADR-0005, ADR-0006 | shared measurement primitives exist, but no canonical end-to-end RAG observation/calibration pipeline is accepted on protected main; issue #607 tracks the governed adapter/evidence-regime contract | Proposed | +| Fallible human/LLM raters | PRD-FR-020..033, TRD-SCR/RAG | ADR-0005 | common observation/scoring contracts, facets/agreement/validation primitives | Accepted principle; generalized discrimination/range/drift extensions require separate recovery | +| Model relation/comparison | PRD-FR-040..043, TRD-MOD | ADR-0006 | relation-safe comparison primitives and diagnostics where merged; formal family-wide distinguishability remains work in progress | Partial | +| Bifactor scoreability | PRD-FR-044, TRD-BIF | ADR-0006 | protected-main package exposes bifactor scoreability surfaces; interpretation still depends on the evidence contract and model relation | Accepted bounded capability / evolving evidence | +| Factor retention | PRD-FR-040/050, TRD-MOD-001 | ADR-0006 | dimensionality diagnostics exist; unified retention + structural-selection evidence workflow remains a product gap tracked by issue #608 | Partial / planned integration | +| Latent-space residual interaction | PRD-FR-040..052, TRD-MOD | ADR-0006 | MLSIRM family on protected main | Accepted only after substantive dimension/testlet/facet diagnosis; not a substitute for omitted structure | +| Adaptive rotation | PRD-FR-051/052, TRD-ROT | ADR-0009 | protected main contains `crates/mlsirm-core/src/rotation/`, PyO3 bindings, `python/fast_mlsirm/rotation.py`, `rotation_selection.py`, package-root exports, criterion-neutral selection and rotation regression/doctoring evidence | Accepted CPU baseline / planned GPU and broader recovery extensions | +| True-parameter recovery | PRD-PRN-003, TRD-TEST-003..006 | ADR-0008 | simulation/recovery reports, Rust/NumPy parity, scheduled statistical studies/recovery contracts | Accepted | +| Correlation vs recovery/agreement | PRD-PRN-003, scoring validity requirements | ADR-0008, ADR-0005 | recovery/simulation, agreement/QWK/facets evidence | Accepted: correlation is supplementary association evidence, never sole proof of parameter recovery or interchangeability | +| Multilevel/multiple-membership/temporal | PRD-FR-060..062, TRD-MLT | ADR-0007 | contextual summaries exist; full reusable contract PR remains open and Rust estimator recovery is future work | Proposed/partial / active PR | +| Accessible standalone reports | PRD-FR-070..072, NFR-004 | ADR-0005 | report renderers, exact-value exports, WCAG-focused regression/doctoring | Accepted/evolving | +| Sensitive data / PII utility | privacy/security requirements | ADR-0012 | source-free/digest/opaque-id provenance where implemented; provider error redaction; hosted identity/retention downstream | Accepted reusable policy / Downstream operations | +| Reusable-core threat model | security/release requirements | ADR-0001/0002/0003/0005/0010/0012 | `docs/security/threat-model.md`, Security Scan/SAST/fuzz/resource tests | Accepted documentation baseline; feature-specific controls evolve | +| LLM credentials/orchestration | TRD-LLM-001..004 | ADR-0010 | repo/org automation contracts; deterministic paths avoid unnecessary model credentials | Accepted governance | +| Scientific vs business/safety criticality | enterprise/automated scoring decision boundary | ADR-0005 | measurement outputs remain separate from causal intervention/cost/utility policy; critical safety/business gates are not derived from psychometric discrimination alone | Accepted boundary | +| Release/provenance | PRD-FR-080..082, TRD release section | ADR-0003, ADR-0008 | release acceptance, commercial evidence, buyer packet, SBOM/provenance/readiness builders | Accepted baseline | +| Documentation architecture completeness | documentation governance | ADR index + `docs/README.md` | `ARCHITECTURE.md`, PRD/TRD, UML, logical ERD, domain/public-contract class view, traceability, threat model, documentation contract | ACTIVE PR; becomes an Accepted baseline only after the canonical set and contract test are protected-main integrated | + +## Key source locations + +### Canonical public contract composition + +- `python/fast_mlsirm/scoring/contracts.py` +- `python/fast_mlsirm/rubric/__init__.py` +- `python/fast_mlsirm/__init__.py` + +### Numerical source of truth + +- `crates/mlsirm-core/` +- `crates/fast-mlsirm-py/` + +### Scientific and product evidence + +- `tests/` +- `fuzz/` +- `docs/doctoring/` +- `docs/changelog.d/` +- release/recovery/governance scripts under `scripts/` + +### Architecture/security maintenance + +- `ARCHITECTURE.md` +- `docs/PRD.md` +- `docs/TRD.md` +- `docs/adr/README.md` +- `docs/uml/` +- `docs/erd/domain-model.puml` +- `docs/security/threat-model.md` +- `docs/documentation_coverage.md` + +## Conversation-wide interpretation invariants + +These principles are intentionally repeated here because silently losing them would change product behavior even if file names and APIs remained stable: + +1. **LLM and human judges are fallible raters.** Rater identity does not make an observation truth; severity, disagreement, bias/range/occasion and drift are measurement evidence where the design supports them. +2. **Correlation is not parameter recovery or absolute agreement.** Scientific estimator claims use aligned bias/MAE/RMSE/coverage/convergence/probability/information recovery as applicable; scorer interchangeability needs agreement/calibration evidence beyond association. +3. **Latent space follows substantive diagnosis.** Multidimensional, bifactor/higher-order, testlet/two-tier and rater/task/occasion structure are represented before residual latent-space interaction is added; latent geometry may not absorb an omitted scientific dimension by default. +4. **Psychometric discrimination is not business/safety criticality.** Item/judge discrimination measures how well observations separate latent levels. Policy-critical failure, causal action value, expected loss or regulatory severity are separate decision/governance layers and may require conjunctive gates or downstream utility models. +5. **Reference-free is not truth-free.** Groundedness to supplied context can be evaluated without a gold answer, but world correctness, completeness and absolute retrieval recall require stronger evidence. +6. **Context and time are part of the design.** Nested/cross-classified/multiple-membership/repeated/temporal structure is not silently flattened when the intended inference depends on it. + +## Documentation authority + +The documentation authority order is: + +1. protected-main source and tests for executable behavior; +2. accepted ADRs for governing architectural/scientific decisions; +3. `docs/PRD.md` and `docs/TRD.md` for product/technical requirements; +4. root `ARCHITECTURE.md`, UML/ERD and reusable-core threat model for system views; +5. method-specific/doctoring documentation and primary literature; +6. proposed ADRs/open issues/PRs for future work. + +PR bodies, automation handoffs and conversations are evidence/discovery inputs but are not authoritative after their decisions have been captured here. + +## Maintenance rule + +A PR that materially changes a public contract, bounded-context ownership, numerical owner, native binding/export authority, model interpretation, lifecycle, trust/privacy boundary, scientific acceptance criterion or release requirement must update this matrix or explicitly demonstrate that the existing mapping remains correct. A maturity row may move to Accepted/implemented only when the corresponding code is on protected main with the required exact-head evidence. diff --git a/docs/traceability/research-basis.md b/docs/traceability/research-basis.md new file mode 100644 index 000000000..b15ee0540 --- /dev/null +++ b/docs/traceability/research-basis.md @@ -0,0 +1,193 @@ +# Research-to-architecture basis + +Status: **Authoritative research traceability baseline** +Last reviewed: 2026-08-09 + +This document records why major scientific/product directions exist. It does not promote every research idea to accepted functionality. `Accepted`, `Proposed`, and `Open` below match the ADR status/implementation evidence. + +## 1. MLSIRM / MLS2PLM numerical core — Accepted + +Architecture effect: + +- retain the existing simple-structure MLS2PLM as an explicit specialization rather than silently claiming the full discrimination-vector model; +- preserve latent-distance interaction as a residual person-item interaction construct; +- use identification-aware recovery/parity for latent coordinates. + +Primary basis: + +- Jeon, M., Jin, I. H., Schweinberger, M., & Baugh, S. (2021). Mapping unobserved item-respondent interactions: A latent space item response model with interaction map. *Psychometrika, 86*(2), 378–403. +- Kang, I., & Jeon, M. (2025). Multidimensional latent space item response models: A note on the relativity of conditional dependence. *Psychometrika, 90*(2), 799–826. +- Molenaar, D., & Jeon, M. (2026). Regularized joint maximum likelihood estimation of latent space item response models. *Psychometrika, 91*, 335–359. + +## 2. Reference-free RAG as measurement — Proposed + +Architecture effect: + +- RAGAS/LLM-judge outputs are observations, not truth; +- separate groundedness, correctness, retrieval relevance/coverage, utility/completeness, robustness, abstention and citation attribution; +- retain query/testlet, judge family/model, prompt/occasion and system-run identities; +- calibrate judges/facets before interpreting aggregate system quality; +- use multidimensional/bifactor/testlet structure before residual latent-space interaction. + +Research basis: + +- Es, S., James, J., Espinosa-Anke, L., & Schockaert, S. (2024). RAGAS: Automated evaluation of retrieval augmented generation. *Proceedings of EACL 2024*. +- Saad-Falcon, J., Khattab, O., Potts, C., & Zaharia, M. (2024). ARES: An automated evaluation framework for retrieval-augmented generation systems. *Proceedings of NAACL 2024*. +- Jeon/Kang latent-space work above for residual interaction structure. +- Many-facet measurement literature for evaluator severity and design connectedness. + +Open question: a canonical end-to-end RAG observation schema and Rust joint estimator are not yet accepted product behavior. + +## 3. Multidimensional, bifactor, testlet and latent-space hierarchy — Accepted decision rule / evolving implementation + +Architecture effect: + +- correlated substantive dimensions, a possible general bifactor dimension, testlet/local-dependence effects and latent-space residual interactions are complementary layers rather than substitutes; +- bifactor fit does not automatically authorize general/specific scores; +- latent space is added only after known substantive/facet/testlet structure when held-out/recovery evidence supports it. + +Primary basis: + +- Rodriguez, A., Reise, S. P., & Haviland, M. G. (2016). Evaluating bifactor models: Calculating and interpreting statistical indices. *Psychological Methods, 21*, 137–150. +- Rijmen, F. (2010). Formal relations and an empirical comparison among the bi-factor, the testlet, and a second-order multidimensional IRT model. *Journal of Educational Measurement, 47*, 361–372. +- Cai, L. (2010). A two-tier full-information item factor analysis model with applications. *Psychometrika, 75*, 581–612. +- Kang & Jeon (2025) for conditional-dependence relativity. + +## 4. Relation-safe model selection — Accepted + +Architecture effect: + +- determine factor retention separately from structural model choice; +- determine nestedness/overlap from actual constraints; +- formal distinguishability precedes non-nested preference; +- boundary/singular nulls require boundary-aware/bootstrap evidence; +- cluster-aware held-out prediction and true-structure recovery supplement inferential tests. + +Primary basis: + +- Schneider, L., Chalmers, R. P., Debelak, R., & Merkle, E. C. (2020). Model selection of nested and non-nested item response models using Vuong tests. *Multivariate Behavioral Research, 55*, 664–684. +- Preacher, K. J., Zhang, G., Kim, C., & Mels, G. (2013). Choosing the optimal number of factors in exploratory factor analysis: A model selection perspective. *Multivariate Behavioral Research, 48*, 28–56. +- Fujimoto, K. A., & Falk, C. F. (2024). The accuracy of Bayesian model fit indices in selecting among multidimensional item response theory models. *Educational and Psychological Measurement, 84*, 217–244. + +## 5. Dynamic evidence-grounded rubric/item bank — Proposed + +Architecture effect: + +- rubric -> blueprint -> candidate generation -> screening -> artificial crowd -> calibration -> adaptive/governed item bank -> rubric revision; +- benchmark criteria are candidate-blind; candidate-aware discovery is cross-fitted/separate; +- criteria carry evidence regime, provenance, version/lifecycle identity; +- structural schema validity is distinct from construct/content validity; +- item information and fit replace ad hoc LLM weights after calibration. + +Research basis: + +- Hashemi, H., Eisner, J., Rosset, C., Van Durme, B., & Kedzie, C. (2024). LLM-RUBRIC: A multidimensional, calibrated approach to automated evaluation of natural language texts. In *Proceedings of the 62nd Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers)* (pp. 13806–13834). Association for Computational Linguistics. https://aclanthology.org/2024.acl-long.745/ + Scope: calibrates multiple rubric-question outputs to human annotations and motivates treating automated judges as fallible, multidimensional raters rather than truth oracles. +- Pradeep, R., Thakur, N., Upadhyay, S., Campos, D., Craswell, N., & Lin, J. (2024). *Initial nugget evaluation results for the TREC 2024 RAG track with the AutoNuggetizer framework* (arXiv preprint arXiv:2411.09607). https://arxiv.org/abs/2411.09607 + Scope: describes atomic information nuggets, human post-editing/calibration and semantic assignment for reference-free RAG evaluation; it does not justify lexical-only matching or an absolute-truth claim. +- Shankar, S., Zamfirescu-Pereira, J. D., Hartmann, B., Parameswaran, A. G., & Arawjo, I. (2024). Who validates the validators? Aligning LLM-assisted evaluation of LLM outputs with human preferences. In *Proceedings of the 37th Annual ACM Symposium on User Interface Software and Technology*. Association for Computing Machinery. https://doi.org/10.1145/3654777.3676450 + Scope: introduces EvalGen and the criteria-drift problem, supporting human alignment, held-out validation and explicit validator provenance. +- Pradeep, R., Thakur, N., Upadhyay, S., Campos, D., Craswell, N., & Lin, J. (2025). *The great nugget recall: Automating fact extraction and RAG evaluation with large language models* (arXiv preprint arXiv:2504.15068). https://arxiv.org/abs/2504.15068 + Scope: reports AutoNuggetizer variants calibrated against human-based conditions and explicitly notes remaining per-topic diagnostic limitations. +- Norgaila, E., Daniela, L., & Kalniņa, D. (2026). Reflective prompt engineering for assessment rubric optimization: An empirical study of human–AI alignment. *Technology, Knowledge and Learning, 31*, 1023–1038. https://doi.org/10.1007/s10758-026-09979-2 + Scope: provides a recent rubric-refinement and human–AI alignment study; it is evidence for an experiment, not a release-time validity guarantee or a substitute for psychometric calibration. + +## 6. Automated essay scoring / automated scoring as many-facet measurement — Accepted baseline / evolving + +Architecture effect: + +- scorer output is governed observation evidence, not an oracle score; +- human and AI raters share a calibration framework; +- severity, agreement, range use, DIF/fairness, drift and adjudication are separate evidence dimensions; +- raw human-AI correlation is insufficient as the primary validity claim. + +Primary basis: + +- Williamson, D. M., Xi, X., & Breyer, F. J. (2012). A framework for evaluation and use of automated scoring. *Educational Measurement: Issues and Practice, 31*(1), 2–13. +- Uto, M., & Ueno, M. (2020). A generalized many-facet Rasch model and its Bayesian estimation using Hamiltonian Monte Carlo. *Behaviormetrika, 47*, 469–496. +- AERA, APA, & NCME (2014), *Standards for Educational and Psychological Testing*. + +## 7. Parameter recovery over correlation — Accepted + +Architecture effect: + +- align scale first; +- require bias, MAE/RMSE, SE/interval coverage, convergence and function/information recovery where relevant; +- correlation is supplementary only. + +Primary basis: + +- Svetina, D., Valdivia, A., Underhill, S., Dai, S., & Wang, X. (2017). Parameter recovery in multidimensional item response theory models under complexity and nonnormality. *Applied Psychological Measurement, 41*(7), 530–544. +- Bland, J. M., & Altman, D. G. (1986). Statistical methods for assessing agreement between two methods of clinical measurement. *The Lancet, 327*(8476), 307–310. + +## 8. Multilevel, multiple-membership and temporal measurement — Proposed + +Architecture effect: + +- prevent atomistic flattening; +- explicit context dimensions and weighted memberships; +- separate repeated occasion ordering from continuous-time dynamics; +- require Rust estimator identification/recovery before production claims. + +Primary basis: + +- Fox, J.-P., & Glas, C. A. W. (2001). Bayesian estimation of a multilevel IRT model. *Psychometrika, 66*, 271–288. +- Uto, M. (2022). A Bayesian many-facet Rasch model with Markov modeling for rater severity drift. *Behavior Research Methods, 55*, 3910–3928. + +## 9. Adaptive factor rotation — Proposed + +Architecture effect: + +- no universal-best criterion; +- criterion registry separated from optimizer; +- deterministic multi-start and solution-basin diagnostics; +- common empirical selector using stability/recovery/theory rather than raw cross-criterion objective values. + +Primary basis: + +- Bernaards, C. A., & Jennrich, R. I. (2005). Gradient projection algorithms and software for arbitrary rotation criteria in factor analysis. *Educational and Psychological Measurement, 65*(5), 676–696. + +## 10. Enterprise issue measurement — reusable adapter accepted; causal decision layer outside current measurement core + +Architecture effect: + +- evidence, counterevidence, stakeholder perspective and candidate intervention are preserved distinctly; +- criterion observations feed the shared scoring/facets architecture; +- a latent measurement score is not itself an expected business intervention value; +- causal outcome/utility optimization belongs in a separate decision layer or downstream bounded context unless a reusable decision-theory primitive is explicitly added. + +The long-term decision concept discussed in research is expected net intervention value, but it is not an accepted `fast-mlsirm` psychometric-kernel requirement today. + +## 11. LLM orchestration depth — Accepted governance, open product-specific policy + +Architecture effect: + +- use provider-neutral orchestration and NVIDIA NIM credentials where needed; +- preserve deterministic no-model gates; +- compare simple routing versus deeper agent orchestration under comparable budgets before hard-coding a complex topology; +- treat Fugu/Conductor/TRINITY-class results as research input, not a universal mandate. + +The complete APA 7 records and evidence classification for Conductor, TRINITY +and the vendor-described Fugu system are maintained in +[`docs/doctoring/llm_orchestration_test_time_compute.md`](../doctoring/llm_orchestration_test_time_compute.md). +That record distinguishes peer-reviewed/preprint research from vendor product +evidence and keeps the implementation claim bounded to an experiment plan. + +## 12. Standards baseline + +Architecture/product documentation also uses: + +- ISO/IEC/IEEE 29148:2018 for requirements engineering; +- ISO/IEC/IEEE 42010:2022 for architecture-description concerns/viewpoints; +- ISO/IEC 25010:2023 for product-quality concerns; +- ISO/IEC 42001:2023 for AI lifecycle/governance concerns where applicable; +- WCAG 2.2 for accessible report surfaces; +- AERA/APA/NCME 2014 Testing Standards for evidence, fairness and score-use interpretation. +- National Institute of Standards and Technology. (2023). *Artificial intelligence risk management framework (AI RMF 1.0)* (NIST AI 100-1). https://doi.org/10.6028/NIST.AI.100-1 + Governance scope: Govern, Map, Measure and Manage are used as a risk-management input; this citation is not a certification or conformity claim. +- National Institute of Standards and Technology. (2024). *Artificial intelligence risk management framework: Generative artificial intelligence profile* (NIST AI 600-1). https://doi.org/10.6028/NIST.AI.600-1 + Governance scope: provider/model risk, provenance, evaluation, human oversight and incident considerations; this citation is not a certification or conformity claim. + +## Maintenance rule + +When a new research finding changes a released formula, score interpretation, factor/model relation, validity gate, item/rubric lifecycle or rater-evaluation rule, update the relevant ADR, requirement IDs and this traceability record in the same reviewed change. diff --git a/docs/uml/README.md b/docs/uml/README.md new file mode 100644 index 000000000..9be39b063 --- /dev/null +++ b/docs/uml/README.md @@ -0,0 +1,19 @@ +# UML and architecture diagrams + +These PlantUML files are source-controlled architecture views. They are explanatory contracts, not generated screenshots. + +| Diagram | Purpose | +|---|---| +| [`component.puml`](component.puml) | Repository/component ownership and external integration boundary. | +| [`scoring-sequence.puml`](scoring-sequence.puml) | Assessment/rubric -> scoring -> observation -> Rust calibration -> validation/report flow. | +| [`model-selection-sequence.puml`](model-selection-sequence.puml) | Relation-safe factor/model selection and recovery flow. | +| [`item-lifecycle.puml`](item-lifecycle.puml) | Governed item/rubric lifecycle and immutable revision semantics. | +| [`item-bank-state.puml`](item-bank-state.puml) | Compatibility alias for the canonical governed item lifecycle view. | +| [`deployment.puml`](deployment.puml) | Package deployment and downstream-host/service composition boundary. | +| [`domain-public-contract.puml`](domain-public-contract.puml) | Persistence-neutral domain/public-contract classes, construction invariants and host-adapter boundary. | + +## Update rule + +A change to ownership, public contract flow, model-selection decision logic, artifact lifecycle, or deployment/integration boundary must update the affected diagram in the same PR or document why the view is unchanged. + +PlantUML sources are intentionally kept in the repository so diffs are reviewable and renderers can reproduce diagrams without sharing binary design assets. diff --git a/docs/uml/component.puml b/docs/uml/component.puml new file mode 100644 index 000000000..9b26e7dd8 --- /dev/null +++ b/docs/uml/component.puml @@ -0,0 +1,64 @@ +@startuml +skinparam componentStyle rectangle +skinparam shadowing false +skinparam packageStyle rectangle + +title fast-mlsirm component architecture + +actor "Independent Python/Rust Consumer" as Consumer +component "Psychometrics Commons\n(downstream hosted product)" as Commons <> +component "Caller-supplied Host Adapter" as HostAdapter <> + +package "fast-mlsirm" { + component "Python Public API / CLI" as PyAPI + component "Assessment + Rubric Contracts" as Contracts + component "Scoring + Domain Adapters" as Scoring + component "Validation + Model Selection" as Validation + component "Reports + Audit Evidence" as Reports + component "PyO3 Binding Registry" as PyO3 + component "Rust Psychometric Core" as RustCore + component "CPU Parallel Runtime" as CPU + component "GPU Kernels" as GPU + component "Release / Recovery / Governance Evidence" as Evidence +} + +component "contextual-orchestrator" as Orchestrator <> +component "TEPP" as TEPP <> +component "Gyeot" as Gyeot <> +component "semantic-data-portal" as Portal <> +component "EgressWeave" as Egress <> + +Consumer --> PyAPI : package API +Commons --> HostAdapter : supplies hosted boundary +Commons --> PyAPI : versioned contracts/results +PyAPI --> Contracts +PyAPI --> Scoring +PyAPI --> Validation +Scoring --> Contracts +Validation --> Contracts +Scoring --> PyO3 : numeric handoff +Validation --> PyO3 : numeric handoff +PyO3 --> RustCore +RustCore --> CPU +RustCore --> GPU : optional device path +PyAPI --> Reports +Reports --> Contracts +Evidence --> PyAPI : acceptance / provenance + +Scoring ..> Orchestrator : optional LLM orchestration +PyAPI ..> TEPP : explicit integration only +PyAPI ..> Gyeot : explicit integration only +PyAPI ..> Portal : immutable research artifact handoff +HostAdapter --> Egress : host-controlled external egress + +note right of RustCore + Production psychometric arithmetic owner. + Python does not duplicate released numerical kernels. +end note + +note bottom of Commons + Owns HTTP/session/consent/result DB/UI/deployment. + fast-mlsirm never depends on Commons product code. +end note + +@enduml diff --git a/docs/uml/deployment.puml b/docs/uml/deployment.puml new file mode 100644 index 000000000..5bdb38aac --- /dev/null +++ b/docs/uml/deployment.puml @@ -0,0 +1,51 @@ +@startuml +skinparam shadowing false +skinparam componentStyle rectangle + +title fast-mlsirm deployment and composition boundary + +node "Independent Consumer" as Independent { + artifact "Python wheel" as Wheel + component "Python Process / Notebook / Batch" as PythonHost + artifact "fast_mlsirm._core\nRust cdylib" as Core + PythonHost --> Wheel + Wheel --> Core +} + +node "Hosted Psychometrics Product" as Hosted { + component "Psychometrics Commons" as Commons + database "Product persistence\n(owned downstream)" as ProductDB + component "Auth / tenant / consent / sessions" as ProductRuntime + artifact "fast-mlsirm wheel" as HostedWheel + Commons --> ProductRuntime + ProductRuntime --> ProductDB + Commons --> HostedWheel : versioned package/API +} + +cloud "Optional CWL Services" as Services { + component "contextual-orchestrator" as Orchestrator + component "Keyverse" as Keyverse + component "TEPP" as TEPP + component "Gyeot" as Gyeot + component "semantic-data-portal" as Portal + component "EgressWeave" as Egress +} + +Hosted ..> Orchestrator : explicit LLM orchestration contract +Hosted ..> Keyverse : identity/federation contract +Hosted ..> TEPP : temporal/event integration +Hosted ..> Gyeot : EMA/ESM collection integration +Hosted ..> Portal : research artifact/provenance integration +Hosted ..> Egress : external-egress policy integration + +note right of Wheel +Contains Python API plus compiled Rust numerical core. +No hosted DB, HTTP router, tenant identity, or deployment state. +end note + +note bottom of Hosted +Host owns transport, authn/authz, tenancy, persistence, +retention, deployment and service credentials. +end note + +@enduml diff --git a/docs/uml/domain-public-contract.puml b/docs/uml/domain-public-contract.puml new file mode 100644 index 000000000..e856fcaba --- /dev/null +++ b/docs/uml/domain-public-contract.puml @@ -0,0 +1,109 @@ +@startuml +skinparam classAttributeIconSize 0 +skinparam shadowing false +skinparam packageStyle rectangle + +title fast-mlsirm reusable domain and public-contract view + +interface "VersionedContract" as VersionedContract { + +schema_version : string + +semantic_revision : string + +fingerprint : sha256 + +canonicalization : string + +digest_algorithm : string +} + +class AssessmentSpec { + +assessment_id : opaque string + +rubric_fingerprint : sha256 + +construct_id : string + +build() : VersionedContract +} + +class RubricSpecification { + +rubric_id : opaque string + +rubric_version : semantic version + +rubric_fingerprint : sha256 + +levels : immutable sequence +} + +class ItemBlueprint { + +blueprint_id : opaque string + +blueprint_fingerprint : sha256 + +generation_seed : unsigned integer + +evidence_mode : enum +} + +class GenerationContract { + +contract_id : opaque string + +contract_fingerprint : sha256 + +source_revisions : immutable sequence + +provider_boundary : untrusted adapter +} + +class ScoringRequest { + +request_id : opaque string + +request_fingerprint : sha256 + +task_revision_fingerprint : sha256 + +response_content_fingerprint : sha256 +} + +class ScoreObservation { + +observation_id : opaque string + +observation_fingerprint : sha256 + +engine_fingerprint : sha256 + +observation_state : enum +} + +class CalibrationDesign { + +design_id : opaque string + +design_fingerprint : sha256 + +design_input_revision : semantic version + +connected_state : enum +} + +class CalibrationReport { + +report_id : opaque string + +report_fingerprint : sha256 + +convergence_state : enum + +model_family : string +} + +interface "Caller-supplied HostAdapter" as HostAdapter { + +publish(contract : VersionedContract) + +request(request : ScoringRequest) + +external_egress(policy : versioned policy) +} + +VersionedContract <|.. AssessmentSpec +VersionedContract <|.. RubricSpecification +VersionedContract <|.. ItemBlueprint +VersionedContract <|.. GenerationContract +VersionedContract <|.. ScoringRequest +VersionedContract <|.. ScoreObservation +VersionedContract <|.. CalibrationDesign +VersionedContract <|.. CalibrationReport + +RubricSpecification "1" --> "*" ItemBlueprint : compiles +RubricSpecification "1" --> "*" AssessmentSpec : referenced by +AssessmentSpec "1" --> "*" ScoringRequest : governs +ItemBlueprint "1" --> "*" GenerationContract : binds +ScoringRequest "1" --> "*" ScoreObservation : produces +CalibrationDesign "1" --> "*" CalibrationReport : fitted as +CalibrationDesign "*" --> "*" ScoreObservation : versioned design inputs +HostAdapter ..> VersionedContract : exchanges immutable artifacts + +note right of VersionedContract + Construction validates identifiers, schema/revision, cardinality, + UTF-8/finite values and trust-boundary provenance. The canonical + preimage binds fast-mlsirm-cjson-v1 and sha-256. +end note + +note bottom of ScoreObservation + Published artifacts are immutable. A correction creates a new + revision with a new fingerprint and an explicit supersedes link. + A host adapter owns HTTP, sessions, tenants, persistence and + external egress; the reusable core never calls the hosted product. +end note + +@enduml diff --git a/docs/uml/item-bank-state.puml b/docs/uml/item-bank-state.puml new file mode 100644 index 000000000..68ab684f7 --- /dev/null +++ b/docs/uml/item-bank-state.puml @@ -0,0 +1 @@ +!include item-lifecycle.puml diff --git a/docs/uml/item-lifecycle.puml b/docs/uml/item-lifecycle.puml new file mode 100644 index 000000000..50c751143 --- /dev/null +++ b/docs/uml/item-lifecycle.puml @@ -0,0 +1,38 @@ +@startuml +skinparam shadowing false + +title Governed item lifecycle + +[*] --> Draft +Draft --> Audited +Audited --> Screened +Screened --> Pilot +Pilot --> Calibrated +Calibrated --> Approved +Approved --> Active +Active --> Suspended +Suspended --> Active +Suspended --> Retired +Calibrated --> Quarantined +Pilot --> Quarantined +Quarantined --> Retired +state "Draft (new revision)" as CorrectedDraft +Quarantined --> CorrectedDraft : create corrected revision +CorrectedDraft --> Audited : new fingerprint +CorrectedDraft ..> Quarantined : supersedes +Active --> Retired +Draft --> Retired + +note right of Active +Approved and active revisions are immutable. +The quarantined revision is retained or terminated; it is never edited in place. +A correction is a new draft with a new revision fingerprint and an explicit +supersedes relationship. +end note + +note left of Pilot +Benchmark criteria are candidate-blind. +Candidate-aware discovery is separated from scored candidates. +end note + +@enduml diff --git a/docs/uml/model-selection-sequence.puml b/docs/uml/model-selection-sequence.puml new file mode 100644 index 000000000..9e07ec765 --- /dev/null +++ b/docs/uml/model-selection-sequence.puml @@ -0,0 +1,59 @@ +@startuml +skinparam shadowing false + +title Relation-safe measurement-model selection + +actor Analyst +participant "Factor Retention" as Retention +participant "Candidate Model Fits" as Fits +participant "Relation Classifier" as Relation +participant "Inferential Comparator" as Infer +participant "Held-out Validator" as CV +participant "Residual / DIF /\nScoreability Diagnostics" as Diagnostics +participant "Recovery Simulation" as Recovery +participant "Selection Policy" as Select + +Analyst -> Retention : propose substantive factor counts +Retention --> Analyst : candidate counts + uncertainty +Analyst -> Fits : fit correlated MIRT / bifactor / higher-order / testlet / two-tier / facets / latent-space +Fits --> Analyst : exact fit artifacts + case/cluster likelihood evidence + +Analyst -> Relation : classify from actual constraints/boundaries +Relation --> Analyst : regular_nested | boundary_nested | nonlinear_nested | non_nested | overlapping | unknown + +alt regular nested + Analyst -> Infer : LR / robust LR +else boundary or singular + Analyst -> Infer : parametric-bootstrap / boundary-aware LR +else strictly non-nested or overlapping + Analyst -> Infer : formal distinguishability first + Infer --> Analyst : distinguishable? + alt distinguishable + Analyst -> Infer : non-nested selection statistic + else not established + Infer --> Analyst : no preferred model + end +else unknown + Relation --> Analyst : fail closed; establish relation +end + +Analyst -> CV : leave-query/person/testlet/rater/domain-out as appropriate +CV --> Analyst : cluster-aware predictive evidence +Analyst -> Diagnostics : fit + local dependence + DIF/invariance + scoreability/stability +Diagnostics --> Analyst : interpretation evidence +Analyst -> Recovery : realistic true-structure simulation +Recovery --> Analyst : selection accuracy + bias/RMSE/coverage/convergence +Analyst -> Select : combine evidence under declared policy +Select --> Analyst : simplest supported model or indeterminate + +note over Relation,Infer +Model names do not define nestedness. +A positive numeric likelihood-difference variance is not, by itself, +the formal Vuong distinguishability test. +end note + +note over Select +A better in-sample fit does not automatically authorize a score interpretation. +end note + +@enduml diff --git a/docs/uml/scoring-sequence.puml b/docs/uml/scoring-sequence.puml new file mode 100644 index 000000000..758b398a2 --- /dev/null +++ b/docs/uml/scoring-sequence.puml @@ -0,0 +1,51 @@ +@startuml +skinparam shadowing false + +title Governed automated-scoring sequence + +actor Caller +participant "AssessmentSpec /\nRubricSpecification" as Spec +participant "ScoringRequest Factory" as Request +participant "ScoringEngine\n(Human / AI / External)" as Engine +participant "Observation Boundary" as Obs +participant "Facets Design Builder" as Facets +participant "Rust Calibration Core" as Rust +participant "Validation / Fairness" as Validate +participant "Report / Adjudication" as Report + +Caller -> Spec : select exact approved versions/fingerprints +Caller -> Request : create request(response revision, task revision, engine policy) +Request -> Spec : replay contract provenance +Spec --> Request : verified +Request --> Caller : content-addressed ScoringRequest + +Caller -> Engine : execute exact request +Engine --> Caller : untrusted/raw engine result +Caller -> Obs : construct governed ScoreObservation +Obs -> Request : replay request + criterion + evidence + engine identity +Request --> Obs : verified +Obs --> Caller : scored | abstained | failed | excluded + +Caller -> Facets : assemble governed observations +Facets -> Obs : replay child provenance +Facets -> Facets : validate respondent-task and task-rater design +Facets --> Caller : criterion-specific design(s) + +Caller -> Rust : fit exact design +Rust --> Caller : estimates + convergence + trace evidence +Caller -> Validate : validate fit/agreement/DIF/range/drift evidence +Validate --> Caller : validation evidence + review triggers +Caller -> Report : build source-text-minimized audit report +Report --> Caller : exact-value JSON/HTML + human review route + +note over Obs,Facets +Terminal states remain distinct. +Abstention/failure is never coerced into a low score. +end note + +note over Engine,Obs +Engine output is not trusted merely because it is valid JSON. +Provider/rater identity and exact task/response revision remain separate. +end note + +@enduml diff --git a/docs/verification_validation_plan.md b/docs/verification_validation_plan.md new file mode 100644 index 000000000..0ea308ffb --- /dev/null +++ b/docs/verification_validation_plan.md @@ -0,0 +1,359 @@ +# Verification and Validation Plan — fast-mlsirm + +Status: canonical V&V baseline +Date: 2026-08-09 + +## 1. Purpose + +This plan defines how `fast-mlsirm` establishes evidence that an implementation: + +1. conforms to its declared software/contract requirements (**verification**); +2. recovers the intended scientific behavior under realistic data-generating conditions (**scientific validation**); +3. is safe and reliable enough for the package's declared use boundary without overclaiming regulated/high-stakes fitness (**product validation**). + +No single metric is treated as sufficient. In particular, unit-test success, code coverage, correlation with another scorer, or in-sample model fit alone does not establish validity. + +## 2. Evidence hierarchy + +```mermaid +flowchart BT + UNIT[Unit / property / parser tests] --> INT[Integration / PyO3 / package tests] + INT --> NUM[Numerical parity / gradient / invariant tests] + NUM --> REC[True-parameter and structure recovery] + REC --> PRED[Held-out / cluster-aware predictive evidence] + PRED --> VALID[Interpretation / fairness / scoreability evidence] + VALID --> REL[Exact-head release acceptance and provenance] +``` + +Evidence at a higher layer may depend on lower layers but does not erase lower-layer failures. + +## 3. Verification classes + +### VV-SW-001 — Unit and public-contract tests + +Every public contract and stable failure mode shall have deterministic unit tests covering: + +- normal construction/use; +- empty/minimum/maximum boundaries; +- invalid types, Boolean-as-integer, non-finite values, malformed IDs; +- duplicate/conflicting values; +- post-construction mutation/replay where relevant; +- stable non-reflective error codes/paths; +- immutable/canonical output semantics. + +### VV-SW-002 — Property and metamorphic tests + +Use invariants when one example cannot establish correctness, including: + +- order-insensitive canonicalization where specified; +- score/likelihood invariance under valid label/sign/permutation transforms; +- global count-scale invariance where the estimator mathematically has it; +- covariance preservation under valid factor rotation; +- linking transformation identities; +- seed and thread-count determinism where promised; +- round-trip serialization/replay. + +### VV-SW-003 — Python↔Rust delegation and parity + +For Rust-owned features: + +- Python public API must be shown to delegate to the intended native surface; +- independent reference/oracle comparisons use tolerances justified by precision/algorithm; +- validation errors and result fields must preserve typed semantics across PyO3; +- Python wrappers must not recompute production statistics to work around a native error. + +### VV-SW-004 — CPU/GPU evidence + +A GPU claim requires: + +- an actual GPU/device adapter selection in the test evidence; +- a test that fails when GPU execution is skipped if the gate claims GPU evidence; +- CPU/GPU objective/result/recovery parity appropriate to non-identifiability and precision; +- bounded memory/resource behavior; +- no separate formula/interpretation semantics. + +### VV-SW-005 — Packaging and import + +Supported release environments require: + +- source/editable build where advertised; +- wheel/sdist build and metadata validation; +- compiled PyO3 import; +- package-root public exports; +- binding-crate tests when omitted from root workspace; +- dependency/lock integrity; +- clean installed-environment smoke tests. + +### VV-SW-006 — Security and adversarial tests + +Cover the threat model's relevant boundaries: + +- hostile iterables/subclasses/conversions; +- excessive dimensions/allocations; +- duplicate/unknown JSON fields and non-finite JSON; +- evidence/source span spoofing; +- path traversal/symlink/race boundaries in tooling; +- secret-shaped subprocess/model-provider failures; +- stale/replay provenance; +- prompt-injection content treated as inert data; +- no PR-controlled self-modifying writer workflow. + +## 4. Scientific validation classes + +### VV-SCI-001 — Parameter recovery + +Where true parameters are known, report as applicable: + +\[ +\operatorname{Bias}(\hat\psi)=E(\hat\psi-\psi) +\] + +\[ +\operatorname{RMSE}(\hat\psi)=\sqrt{E[(\hat\psi-\psi)^2]} +\] + +and interval/SE coverage, convergence/failure rate, boundary behavior, and decision-relevant function recovery. + +Before computing component-wise recovery, align non-identified representations: + +- IRT location/scale linking; +- MIRT sign/rotation/permutation; +- latent-space Procrustes or invariant pairwise distances; +- factor-rotation sign/permutation alignment. + +A high Pearson/Spearman correlation is supplementary and cannot replace absolute recovery. + +### VV-SCI-002 — Response/information recovery + +When downstream use depends more directly on response functions/information than raw parameters, evaluate: + +- ICC/category response probability error; +- item/test information recovery; +- posterior/score calibration; +- cut-score/classification consistency; +- CAT item-selection regret/length when relevant. + +### VV-SCI-003 — Factor retention recovery + +Candidate factor-retention procedures shall be evaluated under realistic combinations of: + +- sample size; +- item/variable count; +- factor count and correlation; +- weak/minor/cross loadings; +- response type/category count; +- nonnormality/missingness; +- local dependence/testlets; +- multilevel/rater structure. + +Report selection confusion matrices rather than one average accuracy when multiple true structures are simulated. + +### VV-SCI-004 — Structural model recovery + +For correlated MIRT, bifactor, higher-order, testlet, two-tier, multifaceted, and latent-space candidates: + +- test whether the intended relation classifier is correct; +- assess formal-test Type I/selection behavior under the appropriate relation; +- compare held-out/cluster-aware predictive likelihood; +- evaluate residual local dependence; +- recover structural/loadings/rater/testlet/interaction parameters; +- evaluate scoreability/invariance before authorizing score interpretation. + +### VV-SCI-005 — Bifactor scoreability + +When a bifactor solution is used for scores, evaluate the indicators appropriate to the score representation, including ECV/PUC, omega hierarchical/general and subscale evidence, construct replicability/factor determinacy, stability, and external incremental validity when available. A fit improvement alone does not authorize a total/subscale score. + +### VV-SCI-006 — Rotation recovery + +Adaptive rotation shall be evaluated with known population loadings/targets using: + +- Tucker congruence after globally optimal factor assignment/sign alignment; +- loading RMSE/target RMSE where identified; +- bootstrap/split-sample stability; +- solution-basin support/entropy; +- criterion-selection frequency by population condition; +- factor-correlation and degeneracy diagnostics. + +The term "best observed" is used for finite multi-start results; global optimality is not claimed. + +### VV-SCI-007 — Multilevel/multiple-membership recovery + +Validation data must reproduce the real design features: + +- cluster/context dimensions; +- cross-classification; +- membership weights; +- unbalanced group sizes; +- sparse/disconnected assignment patterns; +- rater/task facets when relevant. + +Compare parameter/SE/coverage behavior against atomistic misspecification so the benefit of the structure is empirically demonstrated. + +### VV-SCI-008 — Temporal/longitudinal recovery + +Validate ordering, missing occasions, unequal follow-up patterns, drift/state parameters, random intercept/slope effects, and revision boundaries. A discrete-step model is evaluated by step count; a future continuous-time model must include interval-sensitive generating processes and recovery. + +## 5. Automated scoring / LLM-as-a-Judge validation + +### VV-AI-001 — Rater calibration + +Human and LLM scorers are treated as fallible raters. Evidence should include: + +- rater severity; +- criterion-specific effects; +- discrimination/consistency/range-use where modeled; +- prompt/order/occasion drift; +- connectedness of the assignment graph; +- agreement metrics only as descriptive companions. + +### VV-AI-002 — Agreement and absolute error + +When a defensible target or audited score exists, report an appropriate set of: + +- bias/MAE/RMSE; +- exact and adjacent agreement; +- QWK; +- absolute-agreement ICC or concordance where appropriate; +- Bland-Altman or conditional-error evidence; +- calibration slope/intercept for probabilistic outcomes; +- subgroup/score-region errors. + +### VV-AI-003 — Construct and shortcut validity + +Test that automated scorers respond to construct-relevant changes and resist construct-irrelevant shortcuts, including where appropriate: + +- evidence insertion/removal; +- unsupported claims/contradictions; +- citation swaps; +- paraphrase/order invariance; +- verbosity/length perturbations; +- unanswerable cases and abstention; +- adversarial/prompt-injection content; +- language/domain subgroup shifts. + +### VV-AI-004 — Reference-free RAG evaluation + +Reference-free means no gold answer is required for some constructs; it does not mean truth is known. Validation must distinguish: + +- context groundedness; +- query/answer relevance; +- retrieval relevance/utilization; +- correctness against an authoritative/pooled evidence regime when available; +- completeness/obligation coverage proxy; +- robustness and abstention. + +RAG candidate answers must not be used to discover the final benchmark rubric unless the design uses explicit cross-fitting or separate discovery/evaluation banks. + +### VV-AI-005 — Rubric/item generation + +Validate the item/rubric lifecycle in layers: + +1. structural schema/provenance; +2. evidence/source correctness; +3. construct alignment/atomicity/ambiguity/answerability; +4. duplication/leakage/bias/adversarial risks; +5. artificial-crowd/human pilot; +6. item fit/information/DIF/local dependence; +7. linking/anchor/version evidence; +8. operational exposure/drift/retirement. + +## 6. Generalization and resampling design + +Random response-cell splitting is prohibited when it leaks the same person/query/testlet/rater/domain/occasion across train and validation in a way inconsistent with the intended generalization claim. + +Use the unit matching the claim, for example: + +- leave-person/system-out; +- leave-query/testlet-out; +- leave-rater/family-out; +- leave-domain/language-out; +- temporal forward validation; +- cluster/bootstrap blocks aligned to the dependency structure. + +## 7. Coverage policy + +Owned production code targets 100% statement and branch coverage plus function/line/region coverage where available. Coverage is necessary but not sufficient: + +- exclusions cannot hide production behavior; +- a test that only executes a line without asserting the contract is inadequate; +- fail-first tests must reach the intended production boundary rather than fail during setup/import/fixture construction; +- documentation, schema and workflow contracts may use source/structure tests when runtime instrumentation is not meaningful. + +## 8. Performance and resource validation + +Performance work must record: + +- environment/hardware/toolchain; +- problem shape and data type; +- warmup/repetition/statistical summary; +- peak workspace/allocation evidence when relevant; +- numerical equivalence/recovery evidence; +- CPU thread count / GPU adapter and precision; +- no universal speed claim outside the measured environment. + +Memory/resource limits are product requirements when caller-controlled dimensions can produce denial of service. + +## 9. Release V&V gate + +A release candidate must bind evidence to one exact protected source head and exact artifacts. Required categories include as applicable: + +- Python tests and coverage; +- Rust workspace tests, clippy/rustdoc where policy requires; +- PyO3 binding tests/import; +- package/wheel/sdist build/reinstall; +- explicit GPU evidence for GPU claims; +- fuzz/property/security scans; +- dependency/SBOM/provenance evidence; +- accessibility/exact-value report regressions; +- changelog/version correctness; +- method/recovery evidence for changed scientific behavior; +- zero valid unresolved current-head review findings; +- independent approval where required by policy; +- release acceptance/buyer evidence generated from the exact artifact. + +Queued, pending, cancelled, skipped-required, predecessor-head, synthetic-only, status-only, or stale-base evidence is not passing evidence. + +## 10. Failure triage + +Every failing gate is classified before remediation: + +```text +symptom +→ exact first failing boundary +→ immediate cause +→ technical root cause +→ systemic/control cause if material +→ correction owner +→ smallest feasible fix +→ focused RED/GREEN +→ full exact-head evidence +``` + +A setup/test-harness defect must be fixed before changing production code based on that failure. A central infrastructure/reviewer methodology defect is not repaired by weakening a product test. + +## 11. Documentation evidence + +Documentation is subject to V&V: + +- canonical architecture documents must exist and remain internally consistent; +- Mermaid/diagram source must be parseable by the supported renderer in CI/review when tooling is available; +- PRD/TRD/ADR requirements must be traceable to implementation/evidence maturity; +- source/standards claims must be verified and citations kept current; +- planned features must not be described as protected-main capabilities; +- superseded material must be removed or explicitly marked. + +`tests/test_architecture_documentation_contract.py` is the initial executable regression for this architecture spine. + +## 12. References — APA 7th + +American Educational Research Association, American Psychological Association, & National Council on Measurement in Education. (2014). *Standards for educational and psychological testing*. American Educational Research Association. + +Bland, J. M., & Altman, D. G. (1986). Statistical methods for assessing agreement between two methods of clinical measurement. *The Lancet, 327*(8476), 307–310. https://doi.org/10.1016/S0140-6736(86)90837-8 + +Kane, M. T. (2013). Validating the interpretations and uses of test scores. *Journal of Educational Measurement, 50*(1), 1–73. https://doi.org/10.1111/jedm.12000 + +Schneider, L., Chalmers, R. P., Debelak, R., & Merkle, E. C. (2020). Model selection of nested and non-nested item response models using Vuong tests. *Multivariate Behavioral Research, 55*(5), 664–684. https://doi.org/10.1080/00273171.2019.1664280 + +Svetina, D., Valdivia, A., Underhill, S., Dai, S., & Wang, X. (2017). Parameter recovery in multidimensional item response theory models under complexity and nonnormality. *Applied Psychological Measurement, 41*(7), 530–544. https://doi.org/10.1177/0146621617707507 + +Williamson, D. M., Xi, X., & Breyer, F. J. (2012). A framework for evaluation and use of automated scoring. *Educational Measurement: Issues and Practice, 31*(1), 2–13. https://doi.org/10.1111/j.1745-3992.2011.00223.x diff --git a/tests/test_architecture_documentation_contract.py b/tests/test_architecture_documentation_contract.py new file mode 100644 index 000000000..dbc281d3d --- /dev/null +++ b/tests/test_architecture_documentation_contract.py @@ -0,0 +1,275 @@ +"""Contracts for the repository's canonical architecture documentation set.""" + +from __future__ import annotations + +from pathlib import Path +import re + + +ROOT = Path(__file__).resolve().parents[1] + +REQUIRED_DOCUMENTS = ( + "ARCHITECTURE.md", + "docs/README.md", + "docs/PRD.md", + "docs/TRD.md", + "docs/adr/README.md", + "docs/adr/0000-template.md", + "docs/standards_watch.md", + "docs/verification_validation_plan.md", + "docs/uml/README.md", + "docs/uml/component.puml", + "docs/uml/scoring-sequence.puml", + "docs/uml/model-selection-sequence.puml", + "docs/uml/item-lifecycle.puml", + "docs/uml/item-bank-state.puml", + "docs/uml/deployment.puml", + "docs/uml/domain-public-contract.puml", + "docs/erd/domain-model.puml", + "docs/traceability/requirements-matrix.md", + "docs/traceability/research-basis.md", + "docs/security/threat-model.md", + "docs/documentation_coverage.md", +) + +ADR_STATUS_RE = re.compile( + r"^Status: (?:\*\*)?(Accepted|Proposed|Deprecated|Superseded)(?:\*\*)?[ \t]*$", + re.MULTILINE, +) + + +def _read(path: str) -> str: + """Return repository UTF-8 text for a documentation contract path.""" + return (ROOT / path).read_text(encoding="utf-8") + + +def test_canonical_architecture_documentation_files_exist() -> None: + """Keep requirements, decisions, V&V, diagrams, security, and traceability discoverable.""" + missing = [path for path in REQUIRED_DOCUMENTS if not (ROOT / path).is_file()] + assert missing == [] + + +def test_every_indexed_adr_exists_and_declares_supported_status() -> None: + """Prevent the ADR index from pointing at missing or statusless decisions.""" + index = _read("docs/adr/README.md") + linked = re.findall(r"\]\((\d{4}[^)]+\.md)\)", index) + assert linked + assert "0011-canonical-pyo3-public-export-registry.md" in linked + assert "0012-purpose-limited-sensitive-data.md" in linked + assert "0013-continuous-execution-and-documentation-governance.md" in linked + for relative_path in linked: + adr_path = ROOT / "docs" / "adr" / relative_path + assert adr_path.is_file(), relative_path + assert ADR_STATUS_RE.search(adr_path.read_text(encoding="utf-8")), relative_path + + +def test_adr_template_uses_the_parseable_status_grammar() -> None: + """Keep the new-ADR template compatible with the status parser.""" + template = _read("docs/adr/0000-template.md") + assert re.search( + r"^Status: (Accepted|Proposed|Deprecated|Superseded)$", + template, + re.MULTILINE, + ) + + +def test_every_uml_source_is_indexed_and_has_safe_plantuml_boundaries() -> None: + """Require the complete UML set and reject malformed or nested sources.""" + index = _read("docs/uml/README.md") + sources = sorted((ROOT / "docs/uml").glob("*.puml")) + assert sources + for source in sources: + assert source.name in index, source.name + text = source.read_text(encoding="utf-8") + if source.name == "item-bank-state.puml": + assert text == "!include item-lifecycle.puml\n" + continue + assert text.lstrip().startswith("@startuml"), source.name + assert text.count("@startuml") == 1, source.name + assert text.count("@enduml") == 1, source.name + assert text.rstrip().endswith("@enduml"), source.name + assert "<<<<<<<" not in text, source.name + assert not re.search(r"^!include\s+[/~]", text, re.MULTILINE), source.name + + +def test_canonical_documents_state_the_hosted_product_boundary() -> None: + """Do not let core-library documentation drift into hosted-runtime ownership.""" + architecture = _read("ARCHITECTURE.md") + prd = _read("docs/PRD.md") + trd = _read("docs/TRD.md") + + for text in (architecture, prd, trd): + assert "psychometrics-commons" in text.lower() + assert "independ" in text.lower() + assert "never the reverse" in architecture + assert "hosted HTTP/admin APIs" in trd + + +def test_legacy_prd_trd_summary_is_explicitly_deprecated() -> None: + """Historical MVP notes must not compete with the canonical PRD and TRD.""" + summary = _read("docs/prd_trd_summary.md") + assert "Deprecated as an authoritative requirements source" in summary + assert "[Product Requirements Document](PRD.md)" in summary + assert "[Technical Requirements Document](TRD.md)" in summary + + +def test_root_architecture_links_to_canonical_views() -> None: + """Keep the root navigation graph connected to its diagram and decision sources.""" + architecture = _read("ARCHITECTURE.md") + required_links = ( + "docs/uml/component.puml", + "docs/uml/scoring-sequence.puml", + "docs/uml/model-selection-sequence.puml", + "docs/uml/item-bank-state.puml", + "docs/uml/deployment.puml", + "docs/erd/domain-model.puml", + "docs/adr/README.md", + ) + for target in required_links: + assert target in architecture + assert (ROOT / target).is_file(), target + + +def test_item_bank_state_alias_does_not_nest_plantuml_documents() -> None: + """The compatibility alias must include one complete diagram without nested start/end markers.""" + alias = _read("docs/uml/item-bank-state.puml") + assert alias == "!include item-lifecycle.puml\n" + lifecycle = _read("docs/uml/item-lifecycle.puml") + assert lifecycle.count("@startuml") == 1 + assert lifecycle.count("@enduml") == 1 + + +def test_documentation_index_and_completeness_matrix_cover_security_and_gaps() -> None: + """Canonical navigation must expose threat, completeness, and current-vs-planned state.""" + index = _read("docs/README.md") + coverage = _read("docs/documentation_coverage.md") + for target in ( + "../ARCHITECTURE.md", + "PRD.md", + "TRD.md", + "adr/README.md", + "standards_watch.md", + "verification_validation_plan.md", + "security/threat-model.md", + "documentation_coverage.md", + "traceability/requirements-matrix.md", + ): + assert target in index + for state in ("IMPLEMENTED", "ACTIVE PR", "PLANNED", "DOWNSTREAM"): + assert state in coverage + assert "P0 documentation gaps" in coverage + assert "Canonical PyO3/public-export registry" in coverage + + +def test_standards_watch_separates_published_sources_from_watch_items() -> None: + """Draft standards and future revisions cannot silently become normative contracts.""" + standards = _read("docs/standards_watch.md") + for reference in ( + "ISO/IEC/IEEE 29148:2018", + "ISO/IEC/IEEE 42010:2022", + "ISO/IEC 25010:2023", + "ISO/IEC 42001:2023", + "ISO/IEC 42005:2025", + "ISO/IEC 23894:2023", + "NIST AI RMF 1.0", + "NIST AI 600-1", + "WCAG 2.2", + ): + assert reference in standards + assert "ISO/IEC/IEEE DIS 29148" in standards + assert "2026-07-10" in standards + assert "being revised" in standards + assert "watch item" in standards.lower() + assert "does not claim certification" in standards.lower() + + +def test_verification_validation_plan_requires_recovery_not_correlation() -> None: + """Scientific acceptance must retain recovery, alignment, and anti-leakage evidence.""" + validation = _read("docs/verification_validation_plan.md") + for concept in ( + "True-parameter and structure recovery", + "MIRT sign/rotation/permutation", + "Bias", + "RMSE", + "Bifactor scoreability", + "Multilevel/multiple-membership recovery", + "Temporal/longitudinal recovery", + "Rater calibration", + "Reference-free RAG evaluation", + "Random response-cell splitting is prohibited", + ): + assert concept in validation + assert "correlation" in validation.lower() + assert "cannot replace absolute recovery" in validation + + +def test_requirements_traceability_names_core_contract_sources_and_interpretation_rules() -> None: + """Pin executable sources and conversation-wide scientific interpretation boundaries.""" + trace = _read("docs/traceability/requirements-matrix.md") + assert "python/fast_mlsirm/scoring/contracts.py" in trace + assert "python/fast_mlsirm/rubric/__init__.py" in trace + assert "crates/mlsirm-core/" in trace + assert "crates/fast-mlsirm-py/" in trace + assert "LLM and human judges are fallible raters" in trace + assert "Correlation is not parameter recovery or absolute agreement" in trace + assert "Latent space follows substantive diagnosis" in trace + assert "Psychometric discrimination is not business/safety criticality" in trace + assert "Reference-free is not truth-free" in trace + + +def test_reusable_threat_model_covers_core_trust_and_misuse_boundaries() -> None: + """Keep security, privacy, resource, and scientific-integrity threats explicit.""" + threat = _read("docs/security/threat-model.md") + for concept in ( + "Untrusted JSON/member ambiguity", + "Provider replay/provenance substitution", + "PyO3/native shape/type confusion", + "Numeric overflow/non-finite output", + "CPU oversubscription/resource exhaustion", + "GPU evidence spoofing", + "Scientific model misuse", + "Credential cross-contamination", + "Benchmark contamination / double dipping", + "Blanket masking destroys scientific design", + "Self-modifying CI / source laundering", + "Scientific-integrity recovery failure", + ): + assert concept in threat + assert "hosted-product threats" in threat + assert "does not claim certification" in threat + + +def test_pyo3_and_sensitive_data_adrs_preserve_future_integration_boundaries() -> None: + """Prevent feature PRs from recreating native-export and privacy architecture drift.""" + pyo3 = _read("docs/adr/0011-canonical-pyo3-public-export-registry.md") + privacy = _read("docs/adr/0012-purpose-limited-sensitive-data.md") + assert "one canonical PyO3/public-export registry" in pyo3 + assert "runtime source rewriting" in pyo3 + assert "Proposed" in pyo3 + assert "does **not** use blanket PII masking" in privacy + assert "purpose limitation" in privacy + assert "Psychometrics Commons" in privacy + + +def test_research_basis_and_llm_credential_adr_keep_primary_boundaries() -> None: + """Architecture evidence must preserve primary-source and model-credential authority.""" + research = _read("docs/traceability/research-basis.md") + llm = _read("docs/adr/0010-llm-orchestration-and-credentials.md") + assert "APA" in research + assert "NVIDIA" in llm + assert "COPILOT_GITHUB_TOKEN" in llm + + +def test_documentation_contract_distinguishes_implemented_rotation_from_active_work() -> None: + """Protected-main rotation must be Accepted without promoting unrelated active work.""" + trace = _read("docs/traceability/requirements-matrix.md") + coverage = _read("docs/documentation_coverage.md") + rotation_adr = _read("docs/adr/0009-adaptive-rotation-selection.md") + + assert "Accepted CPU baseline / planned GPU and broader recovery extensions" in trace + assert "Proposed/partial / active PR" in trace + assert "IMPLEMENTED / PLANNED extensions" in coverage + assert "ACTIVE PR" in coverage + assert "PLANNED/partial" in coverage + assert "Status: **Accepted**" in rotation_adr + assert "GPU/additional-criterion/recovery expansion remains planned" in rotation_adr diff --git a/tests/test_architecture_execution_governance_contract.py b/tests/test_architecture_execution_governance_contract.py new file mode 100644 index 000000000..a3044996c --- /dev/null +++ b/tests/test_architecture_execution_governance_contract.py @@ -0,0 +1,45 @@ +"""Contracts for feasibility-first, work-conserving repository governance docs.""" + +from __future__ import annotations + +from pathlib import Path + + +ROOT = Path(__file__).resolve().parents[1] + + +def _read(path: str) -> str: + """Return repository UTF-8 text for one canonical governance document.""" + return (ROOT / path).read_text(encoding="utf-8") + + +def test_execution_governance_is_linked_across_canonical_architecture() -> None: + """ADR 0013 must be discoverable from technical, architecture, and traceability views.""" + architecture = _read("ARCHITECTURE.md") + trd = _read("docs/TRD.md") + traceability = _read("docs/traceability/requirements-matrix.md") + + assert "ADR-0013" in architecture + assert "ADR-0013" in trd + assert "ADR-0013" in traceability + + +def test_execution_governance_preserves_feasibility_and_single_writer_invariants() -> None: + """Canonical docs must explain why one blocker/action cannot terminate useful work.""" + adr = _read("docs/adr/0013-continuous-execution-and-documentation-governance.md") + trd = _read("docs/TRD.md") + traceability = _read("docs/traceability/requirements-matrix.md") + + for concept in ( + "work-conserving", + "exact branch head", + "active writer", + "Parallel authority is prohibited", + "non-actionable under current authority", + ): + assert concept.lower() in adr.lower() + + assert "feasibility" in trd.lower() + assert "single-writer" in trd.lower() + assert "work-conserving" in traceability.lower() + assert "single-writer" in traceability.lower()