diff --git a/ARCHITECTURE.md b/ARCHITECTURE.md index 5d5b236c0..5ec22acb2 100644 --- a/ARCHITECTURE.md +++ b/ARCHITECTURE.md @@ -128,7 +128,7 @@ All languages share global topic identities and latent document coordinates. Lan Repeated report vocabulary is modeled through corpus-background, template, section, style, copied-text, prompt, modality, and substantive-topic sources. It is not silently removed by stopword lists, TF-IDF, or BM25. -Topic proportions are compositional. ESEM and network analysis consume logistic-normal latent coordinates or orthonormal log-ratio coordinates, with posterior uncertainty propagated through plausible values or a joint model. +Topic proportions are compositional (Aitchison, 1982). ESEM and network analysis consume logistic-normal latent coordinates or orthonormal log-ratio coordinates, with posterior uncertainty propagated through plausible values or a joint model (Asparouhov & Muthén, 2009; Asparouhov et al., 2018; Marsh et al., 2014). The product topic-estimator contract is TRSL-TM (ADR 0012); an STM-style logistic-normal family is the reference, not a shipped-backend claim (Blei & Lafferty, 2006; Roberts et al., 2014, 2019). TDT/CHRONOS event intelligence remains an accepted-target boundary (Allan, 2002; Anagnostopoulos et al., 2013). ## Compute architecture @@ -141,3 +141,25 @@ PostgreSQL is the reference relational store. Database objects use two-or-more-w ## Security and trust boundaries Documents and LLM outputs are untrusted. Exact spans, JSON Schema, size/depth limits, Unicode validity, prompt-injection isolation, provider allowlists, no-tool execution, tenant isolation, immutable audit events, dependency pinning, SBOM, provenance, and reproducible releases are mandatory. LLM live tests use `NVIDIA_NIM_API_KEY`; `COPILOT_GITHUB_TOKEN` is forbidden. + +## References + +The full APA 7th register is [`docs/research/standards-and-literature.md`](docs/research/standards-and-literature.md). Method claims on this page use: + +Aitchison, J. (1982). The statistical analysis of compositional data. *Journal of the Royal Statistical Society: Series B, 44*(2), 139–177. https://doi.org/10.1111/j.2517-6161.1982.tb01195.x + +Allan, J. (Ed.). (2002). *Topic detection and tracking: Event-based information organization*. Kluwer Academic Publishers. + +Anagnostopoulos, E., Batsakis, S., & Petrakis, E. G. M. (2013). CHRONOS: A reasoning engine for qualitative temporal information in OWL. *Procedia Computer Science, 22*, 70–77. https://doi.org/10.1016/j.procs.2013.09.082 + +Asparouhov, T., Hamaker, E. L., & Muthén, B. (2018). Dynamic structural equation models. *Structural Equation Modeling, 25*(3), 359–388. https://doi.org/10.1080/10705511.2017.1406803 + +Asparouhov, T., & Muthén, B. (2009). Exploratory structural equation modeling. *Structural Equation Modeling, 16*(3), 397–438. https://doi.org/10.1080/10705510903008204 + +Blei, D. M., & Lafferty, J. D. (2006). Dynamic topic models. In *Proceedings of the 23rd International Conference on Machine Learning* (pp. 113–120). ACM. https://doi.org/10.1145/1143844.1143859 + +Marsh, H. W., Morin, A. J. S., Parker, P. D., & Kaur, G. (2014). Exploratory structural equation modeling: An integration of the best features of exploratory and confirmatory factor analysis. *Annual Review of Clinical Psychology, 10*, 85–110. https://doi.org/10.1146/annurev-clinpsy-032813-153700 + +Roberts, M. E., Stewart, B. M., Tingley, D., Lucas, C., Leder-Luis, J., Gadarian, S. K., Albertson, B., & Rand, D. G. (2014). Structural topic models for open-ended survey responses. *American Journal of Political Science, 58*(4), 1064–1082. https://doi.org/10.1111/ajps.12103 + +Roberts, M. E., Stewart, B. M., & Tingley, D. (2019). stm: An R package for structural topic models. *Journal of Statistical Software, 91*(2), 1–40. https://doi.org/10.18637/jss.v091.i02 diff --git a/CHANGELOG.md b/CHANGELOG.md index ea6f62460..f4bdc07d8 100644 --- a/CHANGELOG.md +++ b/CHANGELOG.md @@ -71,7 +71,7 @@ All notable changes to TEPP are documented here. The format follows Keep a Chang - ADR 0014 separating accepted design, protected-main implementation, scientific/product claim promotion, and release evidence authority. - ADR 0015 separating autonomous model proposal, deterministic verification, publication, independent review, and merge/release authority. - ADR 0016 separating Event Ontology observation, TDT detection/tracking, CHRONOS schema prediction, symbolic temporal consistency, and promoted transition authority. -- Verified APA 7 research traceability for ICLR 2026 TRINITY and Conductor, the 2026 Sakana Fugu technical report, ISO/IEC 42001:2023, ISO/IEC 23894:2023, NIST AI RMF/GAI Profile, AICPA Trust Services Criteria, and KISA CSAP guidance. +- Added APA 7 research traceability for ICLR 2026 TRINITY and Conductor, the 2026 Sakana Fugu technical report, ISO/IEC 42001:2023, ISO/IEC 23894:2023, NIST AI RMF/GAI Profile, AICPA Trust Services Criteria, and KISA CSAP guidance. - Eight-phase delivery roadmap and Temporal/Event Foundation implementation plan. - Immutable evidence, six-clock temporal semantics, interval reasoning, event ontology, typed relation graph, and time-varying multiple-membership contracts. - Shared-latent multilingual topic measurement architecture with native lexical channels and language-profile validation. @@ -92,6 +92,7 @@ All notable changes to TEPP are documented here. The format follows Keep a Chang ### Changed +- Added APA 7th method citations (Allen 1983; ISO 24617-1:2012; Hobbs & Pan 2017; Fox & Glas 2001; AERA/APA/NCME 2014; Blei & Lafferty 2006; Roberts et al. 2014, 2019; Chang & Blei 2009; Mimno et al. 2009; Asparouhov & Muthén 2009; Asparouhov et al. 2018; Marsh et al. 2014; Aitchison 1982; Allan 2002; Li et al. 2021; Anagnostopoulos et al. 2013) into ADRs 0002–0005, 0012, and 0016, plus TRACEABILITY/ARCHITECTURE/TRD method rows. Clarified that TRSL-TM is the product contract, STM-style logistic-normal is the reference family, ESEM/DSEM/TDT/CHRONOS remain accepted-target, and merged PRs #8/#9—not superseded drafts #5/#6—are the protected-main temporal lineage. - Refreshed the live gap-baseline inventory to the 2026-08-24T05:41:54Z GitHub snapshot (118 open PRs / 48 drafts / 12 issues; protected-main `c45be17a9dbce95ef81cee230e9d128abc7160ac`), binding each operator-gap current diff --git a/DOCUMENTATION.md b/DOCUMENTATION.md index 9e7952e3e..d0f517ded 100644 --- a/DOCUMENTATION.md +++ b/DOCUMENTATION.md @@ -61,4 +61,4 @@ The documentation graph is **design-sufficient** when a reviewer can reconstruct It is **protected-main-sufficient** only after the canonical documents are integrated on protected `main`, remain semantically current with live code, and their required exact-head documentation/security/review gates pass. An active documentation PR can therefore be design-sufficient while the protected branch remains documentation-insufficient. -At the time of this review, immutable evidence records/exact spans, the Rust workspace quality foundation, and typed six-clock values/uncertain intervals (PR #8) are implemented-main. PR #9 is the active-PR that replays Task 4 Allen interval algebra and bounded path-consistency reasoner work onto that protected-main temporal foundation. Superseded PRs #5 and #6 remain historical lineage only. Event ontology, PostgreSQL persistence, shared-latent topic estimation, GPU kernels, TDT/CHRONOS intelligence, longitudinal ESEM/DSEM, visual analytics, production HTTP services, and deployment assurance remain later accepted-target or deployment-owned work. +As of 2026-08-19, protected `main` at `7c29e7c971d7940e1fb3def1ed3aae2d1bc8ad4a`, immutable evidence records/exact spans, the Rust workspace quality foundation, typed six-clock values/uncertain intervals (merged PR #8), and Allen interval algebra/bounded path-consistency (merged PR #9) are implemented-main. Superseded PRs #5 and #6 remain historical lineage only and are not current-product claims. Shared-latent topic estimation, GPU kernels, TDT/CHRONOS intelligence, longitudinal ESEM/DSEM, visual analytics, production HTTP services, and deployment assurance remain later accepted-target or deployment-owned work unless a [`docs/TRACEABILITY.md`](docs/TRACEABILITY.md) row records a narrower implemented-main or partial subset. Unmerged or draft PRs are not implemented-main claims. diff --git a/docs/API_CONTRACT.md b/docs/API_CONTRACT.md index 8980f1ded..6d261b14b 100644 --- a/docs/API_CONTRACT.md +++ b/docs/API_CONTRACT.md @@ -1,7 +1,7 @@ # TEPP API and Modular Integration Contract **Status:** Accepted target contract; exact endpoints are introduced only with executable services. -**Last reviewed:** 2026-08-16 +**Last reviewed:** 2026-08-19 ## 1. Authority boundary @@ -14,18 +14,20 @@ Current protected main exposes Rust library/domain contracts. The active PR adds | Contract | Owner | Consumers | Maturity | |---|---|---|---| | evidence record/span wire v1 | TEPP `evidence_core` | future TEPP services/adapters | implemented-main | -| temporal clock/interval wire | `temporal_core` | relation/event/persistence | active-PR #5 | -| interval relation/reasoner API | `temporal_core` | event/relation validation | active-PR #6 | +| temporal clock/interval wire | `temporal_core` | relation/event/persistence | implemented-main (`temporal-core/v1`; wire `schema_version=1`; merged PR #8; [`wire_contract.rs`](../crates/temporal_core/tests/wire_contract.rs), [`schema_semantics_contract.rs`](../crates/temporal_core/tests/schema_semantics_contract.rs), [`temporal-event-foundation.md`](validation/temporal-event-foundation.md)) | +| interval relation/reasoner API | `temporal_core` | event/relation validation | implemented-main (`temporal-core/v1`; in-memory reasoner; merged PR #9; Allen, 1983; [`relation_contract.rs`](../crates/temporal_core/tests/relation_contract.rs), [`reasoner_contract.rs`](../crates/temporal_core/tests/reasoner_contract.rs), [`temporal-event-foundation.md`](validation/temporal-event-foundation.md)) | | event/relation/membership API | future TEPP crates/services | naruon, analytics, UI | accepted-target | | semantic/topic measurement API | future TEPP measurement service | naruon, batch jobs, visual analytics | accepted-target | | LLM interpretation provider port | `tepp_api` orchestration router + future HTTP gateway | contextual-orchestrator | partial | | model/artifact/export API | `tepp_api` export envelopes + future HTTP service | standalone UI/CWL consumers | partial | -| analysis-run request/accepted contracts | `tepp_api` v1 wire DTOs | naruon, orchestrator, UI | active-PR | +| analysis-run request/accepted contracts | `tepp_api` v1 wire DTOs | naruon, orchestrator, UI | implemented-main (merged PR #21) | ## 3. Versioning Every externally consumable contract has an explicit semantic contract version independent of software package version. Breaking changes require a new contract version, migration/compatibility notes, contract tests, and an ADR when they change measurement meaning, temporal semantics, ontology, evidence identity, or authorization. +The temporal semantic contract is `temporal-core/v1`. Its JSON representation keeps `schema_version: 1` as a separate wire-schema field; changing either identifier requires its own compatibility evidence. + Wire payloads: - reject unknown fields unless a version explicitly defines extensibility; diff --git a/docs/DOCUMENTATION_ASSESSMENT.md b/docs/DOCUMENTATION_ASSESSMENT.md index ccbfb9678..cbe100ea2 100644 --- a/docs/DOCUMENTATION_ASSESSMENT.md +++ b/docs/DOCUMENTATION_ASSESSMENT.md @@ -1,7 +1,7 @@ # TEPP Whole-Conversation Documentation Assessment -**Assessment date:** 2026-08-12 -**Scope:** Durable TEPP decisions established in the product conversation, approved PRD v0.4/planning pack, protected-main implementation through merged PR #7, canonical Task 3 replacement PR #8, and legacy Task 4 PR #6. +**Assessment date:** 2026-08-19 +**Scope:** Durable TEPP decisions established in the product conversation, approved PRD v0.4/planning pack, and protected-main implementation through the current TRACEABILITY matrix. Merged PRs #8 and #9 are the temporal lineage; superseded PRs #5 and #6 are historical only. **Verdict:** **The canonical documentation/ADR graph is protected-main authority after PR #7; implementation maturity remains independently tracked and is not promoted by documentation completeness.** ## 1. Assessment rule @@ -58,7 +58,7 @@ The ADR set was not fully clear before the PR #7 reconciliation because accepted - adding ADRs 0012–0016 for previously unowned durable decisions; and - making `scripts/validate_documentation.py` fail when numbered ADR files/index diverge or required ADR metadata/sections disappear. -PR #8 now exercises that policy by updating ADR 0002, the ADR index, TRD/UML/Test Strategy/Traceability, and this assessment to identify the canonical Task 3 replacement lineage rather than leaving stale PR #5 maturity claims. +Merged PR #8 exercised that policy by identifying the canonical Task 3 replacement lineage rather than leaving stale PR #5 maturity claims. Merged PR #9 later replayed Task 4 Allen algebra onto that foundation. Those pull requests are lineage, not living drafts. ## 4. Durable conversation decisions covered @@ -91,9 +91,9 @@ The canonical graph explicitly preserves: Documentation completeness must not be confused with product completeness. -- **implemented-main:** Rust workspace/quality foundation, immutable evidence/exact-span boundary, typed six-clock/uncertain interval foundation (PR #8), and canonical documentation/ADR authority graph through PR #7/#8. -- **active-PR:** PR #9 Allen relation algebra and bounded path-consistency reasoner replayed onto protected-main temporal foundation; promote only after exact-head gates and merge. -- **accepted-target:** Event ontology/graph, multilevel estimators beyond the membership network surface, persistence/splits, multilingual semantic units, TRSL-TM topic measurement, GPU compute, model selection, TDT/CHRONOS, ESEM/DSEM, networks/clusters, interpretation, visual analytics, autonomous product-development authority, and production service APIs. +- **implemented-main:** Rust workspace/quality foundation, immutable evidence/exact-span boundary, typed six-clock/uncertain interval foundation (merged PR #8), Allen relation algebra and bounded path-consistency (merged PR #9), and the canonical documentation/ADR authority graph. See [`docs/TRACEABILITY.md`](TRACEABILITY.md) for the exact capability rows. +- **active-PR:** only capabilities whose TRACEABILITY row still says `active-PR`. Unmerged or draft PRs are not implemented-main claims. +- **accepted-target:** multilingual semantic units, TRSL-TM topic measurement, GPU compute, model selection, TDT/CHRONOS, ESEM/DSEM, networks/clusters, interpretation, visual analytics, autonomous product-development authority, production service APIs, and multilevel estimators beyond the membership network surface. - **partial:** selected repository-quality and standalone crate boundaries are implemented, while complete estimator/service/release authorities remain target work. - **deployment-owned/external-assurance:** production infrastructure controls, measured SLO/RPO/RTO, CSAP certification, SOC 2 attestation and jurisdiction-specific legal determinations. @@ -111,4 +111,4 @@ No additional large parallel documentation pack should be created merely for com ## 7. Exit criterion -The canonical documentation design is already protected-main authority through merged PR #7. Future documentation fitness is event-driven and must stay synchronized with actual implementation lineage. PR #8 can promote the Task 3 temporal capability only after its unchanged exact head satisfies all live repository gates and is merged to protected main; the same rule applies independently to the future Task 4 replay. +The canonical documentation design is already protected-main authority. Future documentation fitness is event-driven and must stay synchronized with actual implementation lineage. Unmerged or draft PRs cannot promote a capability to implemented-main; only exact-head protected-main integration and the live TRACEABILITY row can. diff --git a/docs/OPERABILITY.md b/docs/OPERABILITY.md index 3f0c42543..3a8291634 100644 --- a/docs/OPERABILITY.md +++ b/docs/OPERABILITY.md @@ -3,7 +3,7 @@ **Status:** Accepted target operating baseline with current maturity explicit. **Last reviewed:** 2026-08-13 -TEPP is still an implementation-stage research/product platform. Protected main currently contains the Rust workspace/evidence foundation; PR #5/#6 add temporal foundations. Database, model fitting, GPU, services, visual analytics, and production deployment are later targets. This guide defines the operating evidence those stages must satisfy rather than claiming they already exist. +TEPP is still an implementation-stage research/product platform. Protected main currently contains the Rust workspace/evidence foundation plus implemented-main temporal primitives (merged PRs #8 and #9). Superseded PRs #5 and #6 are historical lineage only. Database adapters are partial; model fitting, GPU, services, visual analytics, and production deployment are later targets. This guide defines the operating evidence those stages must satisfy rather than claiming they already exist. Unmerged or draft PRs are not implemented-main claims. ## Operating principles diff --git a/docs/TEST_STRATEGY.md b/docs/TEST_STRATEGY.md index 5def6776b..3180012d1 100644 --- a/docs/TEST_STRATEGY.md +++ b/docs/TEST_STRATEGY.md @@ -23,9 +23,9 @@ Verify immutable source bytes/text, canonical SHA-256, UUIDv7 identifiers, size ## Temporal tests -Canonical Task 3 replacement PR #8 must prove six nominal clock types cannot be accidentally interchanged, strict known-offset RFC 3339/UTC normalization, precision retention, interval boundary semantics, unknown/open intervals, reversed/empty rejection, strict wire schemas, schema/runtime parity, and non-reflecting errors. PR #5 is historical TDD lineage only and its old checks/reviews are not current evidence. +Protected-main `temporal_core` (merged PR #8) must prove six nominal clock types cannot be accidentally interchanged, strict known-offset RFC 3339/UTC normalization, precision retention, interval boundary semantics, unknown/open intervals, reversed/empty rejection, strict wire schemas, schema/runtime parity, and non-reflecting errors. Superseded PR #5 is historical TDD lineage only and its old checks/reviews are not current evidence. -Task 4 must prove all 13 Allen relations, inverse/composition laws, independent composition verification, proper-interval classification, bounded path-consistency, contradiction evidence, provenance, resource limits, and atomic rollback. Legacy PR #6 contains this work on the superseded #5 stack; before promotion it must be replayed onto PR #8 or its exact protected-main descendant and pass fresh tests/coverage/review. It must not overclaim global satisfiability. +Task 4 on protected main (merged PR #9) must prove all 13 Allen relations (Allen, 1983), inverse/composition laws, independent composition verification, proper-interval classification, bounded path-consistency, contradiction evidence, provenance, resource limits, and atomic rollback. Superseded PR #6 is historical lineage on the discarded #5 stack, not a current-product claim. Path consistency must not be documented as unrestricted global satisfiability. ## Leakage tests @@ -45,7 +45,7 @@ Generate corpora with known topic prevalence/content parameters, covariance, cov ## Psychometric validation -For ESEM/DSEM simulations evaluate loading/factor/path recovery, bias, RMSE, confidence/credible interval coverage, convergence, configural/metric/scalar or partial invariance as required, multilevel/multiple-membership effects, within/between decomposition, irregular-time dynamics, and posterior plausible-value propagation. +For ESEM/DSEM simulations (Asparouhov & Muthén, 2009; Asparouhov et al., 2018; Marsh et al., 2014) evaluate loading/factor/path recovery, bias, RMSE, confidence/credible interval coverage, convergence, configural/metric/scalar or partial invariance as required, multilevel/multiple-membership effects, within/between decomposition, irregular-time dynamics, and posterior plausible-value propagation. These psychometric targets remain accepted-target. ## Network/cluster validation @@ -66,3 +66,7 @@ Simulation thresholds account for Monte Carlo standard error and interval uncert ## Release acceptance A release requires one integrated protected head with all relevant scientific, numerical, security, migration, packaging, SBOM/provenance, accessibility, operational, and independent-review evidence passing. Planning validation, superseded-branch results, and local-only results are supporting evidence, not release proof. + +## References + +The full APA 7th register is [`docs/research/standards-and-literature.md`](research/standards-and-literature.md). Method names used above cite Allen (1983) for interval algebra and Asparouhov & Muthén (2009), Asparouhov et al. (2018), and Marsh et al. (2014) for ESEM/DSEM. diff --git a/docs/TRACEABILITY.md b/docs/TRACEABILITY.md index d1b4acc7c..cb0098966 100644 --- a/docs/TRACEABILITY.md +++ b/docs/TRACEABILITY.md @@ -1,7 +1,7 @@ # TEPP Requirements, Research, and Evidence Traceability **Status:** Accepted cross-cutting traceability baseline -**Last reviewed:** 2026-08-20 +**Last reviewed:** 2026-08-19 The full APA 7th standards/literature register remains `docs/research/standards-and-literature.md`. This matrix links durable requirements to their owning decisions and implementation/evidence maturity without duplicating the bibliography. @@ -10,27 +10,30 @@ The full APA 7th standards/literature register remains `docs/research/standards- | immutable source evidence and exact spans | PRD; Architecture; ADR 0008 | `evidence_core`, Task 2 tests/doctoring; `persistence_postgres` source-artifact SQL insert/lookup plus idempotent retry (#40 implemented-main); typed `text_segment` byte-span SQL (active PR) | implemented-main | | Rust numerical authority / CPU `f64` reference | ADR 0001 | current workspace foundation; future estimators | partial | | Rust workspace/quality foundation | ADR 0007 | workspace/CI/repository contract | implemented-main | -| six distinct clocks and uncertain intervals | PRD; ADR 0002 | PR #8 `temporal_core` on protected main; PR #5 historical only | implemented-main | -| Allen relation algebra/bounded closure | ADR 0002; temporal research | PR #9 `temporal_core` path-consistency on protected main | implemented-main | -| forward-only transition subgraph | PRD; ADR 0002/0003 | `relation_graph` on protected main; downstream transition/split enforcement remains accepted-target | implemented-main | +| six distinct clocks and uncertain intervals | PRD; ADR 0002; ISO 24617-1:2012; Hobbs & Pan (2017) | merged PR #8 `temporal_core` on protected main; PR #5 historical lineage only | implemented-main | +| Allen relation algebra/bounded closure | ADR 0002; Allen (1983) | merged PR #9 `temporal_core` path-consistency on protected main | implemented-main | +| forward-only transition subgraph | PRD; ADR 0002/0003 | `relation_graph` on protected main | implemented-main | | event ontology/evidence mentions | PRD; ADR 0003 | `event_core` mention/instance separation on protected main; `persistence_postgres` mention SQL implemented-main refuses mention-as-instance; event-instance SQL (#39 implemented-main) refuses inverted windows; full intelligence stack remaining | partial | -| time-varying cross-classified multiple membership | PRD; ADR 0003 | `membership_core` network on protected main; multilevel estimators remaining | partial | +| time-varying cross-classified multiple membership | PRD; ADR 0003; Fox & Glas (2001) for multilevel IRT; AERA/APA/NCME (2014) | `membership_core` network on protected main; Fox & Glas does not establish an arbitrary multiple-membership estimator; multilevel estimators remain | partial | | leakage-safe availability/cutoff snapshots | PRD; ADR 0002/0013 | `corpus_split` on protected main | implemented-main | | recovery metrics (RMSE, bias, coverage, graph, temporal order, Monte Carlo SE gates) | PRD; Test Strategy; ADR 0007/0014 | `validation_core` on protected main (PR #19); SE-aware Monte Carlo gates included | implemented-main | | PostgreSQL bitemporal/lineage persistence | ADR 0013; Architecture/ERD | `persistence_postgres` migration contracts, in-memory adapters, live SQL session/document SQL port, tenant RLS (`0002` + session GUC/role helpers), `DATABASE_URL` SQLx gate, optional `live-sqlx` `PgPool` driver, exact-head live PostgreSQL CI with isolation proof, append-only immutability triggers (`0004`), temporal interval ordering CHECKs (`0005`), typed membership assignment (`0006` implemented-main), event-relation/mention/instance SQL (#37–#39 implemented-main), source-artifact SQL (#40 implemented-main), audit-event SQL (#41 implemented-main), concurrent document-write stress (#43 implemented-main), backup/restore integrity revalidation (#44 implemented-main), typed `text_segment` SQL insert/cutoff lookup (active PR); remaining physical ERD constraints including `document_record` FK on `text_segment` | partial | | known-truth temporal/event simulation manifests | PRD; TRD; Test Strategy | `tepp_simulation` on protected main; recovery metrics in `validation_core` | implemented-main | -| versioned service/API contracts and exports | PRD; API contract; ADR 0011/0013 | `tepp_api` analysis-run/export/JSON-LD/GraphML contracts on protected main (PR #21); HTTP service remaining accepted-target | partial | +| analysis-run/export DTO and artifact contracts | PRD; API contract; ADR 0011/0013 | `tepp_api` analysis-run/export/JSON-LD/GraphML contracts on protected main (merged PR #21) | implemented-main | +| production HTTP service routing | PRD; API contract; ADR 0011 | No deployed HTTP service; endpoint shapes remain a future service target | partial | | immutable split/run/reproducibility manifests | ADR 0013; ERD | `tepp_api` reproducibility manifest contract on protected main; `persistence_postgres` append-only SQL insert/lookup for `reproducibility_manifest`, `corpus_split_manifest`, `model_run`, and `model_artifact` (migration `0003`); full physical ERD constraints remaining | partial | -| multilingual shared latent semantic space | PRD; ADR 0004 | future semantic/concept/topic crates | accepted-target | -| TRSL-TM temporal/relational topic posterior and backend compatibility | ADR 0012; ADR 0004 | future `topic_measurement` | accepted-target | +| multilingual shared latent semantic space | PRD; ADR 0004; Mimno et al. (2009); Blei & Lafferty (2006); Roberts et al. (2014, 2019) | future semantic/concept/topic crates | accepted-target | +| TRSL-TM temporal/relational topic posterior and backend compatibility | ADR 0012; ADR 0004; Chang & Blei (2009); Blei & Lafferty (2006); Roberts et al. (2014, 2019) | future `topic_measurement`; TRSL-TM is the product contract, STM-style logistic-normal the reference family | accepted-target | | global P0 topic identity with activity/dormancy/reactivation | ADR 0012 | future topic lineage/activity state | accepted-target | | no default stopword deletion / no TF-IDF-BM25 inferential weighting | ADR 0004/0012; PRD/TRD | future semantic/method-source model | accepted-target | | report template/section/copied/style/modality method effects | ADR 0004/0012; PRD/TRD | simulation truth factors implemented; estimator-side method model remains future | partial | | candidate K statistical/Pareto gates + blinded LLM review | ADR 0012; research | future `model_selection` | accepted-target | -| compositional topic correlation / stable clustering | ADR 0005/0012; research | future `network_analysis` | accepted-target | -| posterior ESEM / longitudinal invariance / DSEM | ADR 0005 | future `psychometric_core` | accepted-target | +| compositional topic correlation / stable clustering | ADR 0005/0012; Aitchison (1982) | future `network_analysis` | accepted-target | +| posterior ESEM / longitudinal invariance / DSEM | ADR 0005; Asparouhov & Muthén (2009); Asparouhov et al. (2018); Marsh et al. (2014); AERA/APA/NCME (2014) | future `psychometric_core` | accepted-target | | CPU bounded multithreading + GPU/VRAM streaming/parity | ADR 0001/0006 | future `compute_backend` | accepted-target | -| TDT detection/tracking vs CHRONOS schema/prediction/temporal consistency | ADR 0016; PRD/research | `event_core` evidence-layer admission and known-stream first-story rates on PR #50; full tracking/calibration and schema extraction remain future | active-PR | +| TDT detection/tracking | ADR 0016; Allan (2002) | future `event_intelligence` | accepted-target | +| neural event-schema induction and prediction | ADR 0016; Li et al. (2021) | future `event_intelligence` | accepted-target | +| symbolic qualitative temporal consistency | ADR 0016; Anagnostopoulos et al. (2013) | future `event_intelligence` | accepted-target | | evidence-bounded LLM interpretation | ADR 0010/0012; PRD | `tepp_api` router plus future `interpretation_gateway` | partial | | adaptive direct/verify/committee/conductor test-time compute | ADR 0010; `docs/LLM_ORCHESTRATION.md` | `tepp_api::route_orchestration`, ablation record, and credential-free contextual-orchestrator binding on protected main; live execution and learned conductor calibration remain future | partial | | purpose-bound PII handling without blanket masking | ADR 0009; `docs/PRIVACY_DATA_GOVERNANCE.md` | `tepp_api` export authorization, elevated re-identification, and provider-payload minimization are implemented-main; migration `0007` retention/deletion/legal-hold SQL contracts are implemented-main; deployment/provider evidence remains accepted-target | partial | diff --git a/docs/TRD.md b/docs/TRD.md index c669620f1..cb85f5709 100644 --- a/docs/TRD.md +++ b/docs/TRD.md @@ -1,7 +1,7 @@ # TEPP Technical Requirements Document **Status:** Accepted technical baseline aligned to approved PRD v0.4 -**Last reviewed:** 2026-08-12 +**Last reviewed:** 2026-08-16 ## 1. Technical objective @@ -9,9 +9,9 @@ TEPP is a multilingual temporal-event psychometrics platform whose executable co ## 2. Current implementation maturity -Protected main contains the Rust workspace foundation, immutable evidence records, exact source spans, strict versioned evidence JSON, stable content-redacting errors, repository quality contracts, and the canonical ADR/documentation authority graph merged through PR #7. PR #8 is the canonical active Task 3 replacement adding typed six-clock temporal values and uncertain intervals on that exact protected-main lineage. Conflicted PR #5 is superseded implementation lineage only. Legacy PR #6 contains Allen relation algebra and bounded path-consistency work stacked on #5; it cannot advance as implementation evidence until its unique Task 4 work is replayed onto PR #8 or its protected-main descendant and all exact-head gates are reacquired. +Protected main contains the Rust workspace foundation, immutable evidence records, exact source spans, strict versioned evidence JSON, stable content-redacting errors, repository quality contracts, and the canonical ADR/documentation authority graph. Typed six-clock values and uncertain intervals are implemented-main (merged PR #8 / `temporal_core`). Allen interval algebra and bounded path-consistency are implemented-main (merged PR #9 / `temporal_core`; Allen, 1983). Superseded PRs #5 and #6 are historical lineage only and are not current-product claims. -The remaining PRD architecture — event ontology, relation graph, multiple membership, PostgreSQL persistence, leakage-safe corpus splits, simulations, multilingual semantic units, topic measurement, GPU/VRAM compute, model selection, TDT/CHRONOS, longitudinal ESEM/DSEM, network analysis, interpretation, and visual analytics — is accepted-target, not as-built. +Capability maturity for later layers is recorded in [`docs/TRACEABILITY.md`](TRACEABILITY.md). Multilingual semantic units, TRSL-TM topic measurement, GPU/VRAM compute, model selection, TDT/CHRONOS (Allan, 2002; Anagnostopoulos et al., 2013), longitudinal ESEM/DSEM (Asparouhov & Muthén, 2009; Asparouhov et al., 2018; Marsh et al., 2014), network analysis, interpretation, and visual analytics remain accepted-target unless a TRACEABILITY row says otherwise. Unmerged or draft PRs are not implemented-main claims. ## 3. Immutable evidence requirements @@ -21,13 +21,13 @@ Evidence wire formats are explicitly versioned, reject unknown fields, and recon ## 4. Temporal requirements -TEPP treats event/valid time, assertion time, document time, system time, available time, and knowledge cutoff as distinct nominal types. Analyses enforce `available_time <= knowledge_cutoff`. Exact, uncertain, open-ended, and unknown intervals preserve source precision and boundary semantics. +TEPP treats event/valid time, assertion time, document time, system time, available time, and knowledge cutoff as distinct nominal types. Analyses enforce `available_time <= knowledge_cutoff`. Exact, uncertain, open-ended, and unknown intervals preserve source precision and boundary semantics. Interval topology follows Allen (1983); event/time-marking vocabulary is aligned with ISO-TimeML (International Organization for Standardization, 2012) and may map outward to OWL-Time (Hobbs & Pan, 2017). -PR #8 implements the typed-value, interval, wire, and schema primitives for this requirement. Historical-snapshot enforcement remains owned by the future persistence/corpus-split layers; active-PR primitives must not be described as protected-main enforcement before merge. +Merged PR #8 implements the typed-value, interval, wire, and schema primitives for this requirement on protected `main`. Historical-snapshot enforcement is owned by persistence/corpus-split layers (see TRACEABILITY); an unmerged PR is not protected-main enforcement. Forward state-transition/input→process→outcome edges must satisfy temporally valid partial order. Retrospective, revision, citation, translation, support, and contradiction edges may point backward as provenance but never create reverse state transitions. -The Task 4 bounded Allen closure represented by replay PR #9, once replayed onto the canonical lineage and independently revalidated, establishes path consistency only within its stated algebra/limits; it must not be documented as a proof of global satisfiability for unrestricted disjunctive interval networks. +The Task 4 bounded Allen closure on protected `main` (merged PR #9) establishes path consistency only within its stated algebra/limits; it must not be documented as a proof of global satisfiability for unrestricted disjunctive interval networks. ## 5. Event/relation/membership target @@ -43,9 +43,9 @@ Language support is a validation claim, not a feature flag: each language profil ## 7. Topic and psychometric target -Shared-latent temporal/relational topic estimation provides posterior uncertainty and covariate effects. Topic proportions are compositional; downstream correlation/ESEM uses logistic-normal coordinates or appropriate orthonormal log-ratio coordinates rather than naïve raw-proportion Pearson correlation. +Shared-latent temporal/relational topic estimation provides posterior uncertainty and covariate effects. The product contract is TRSL-TM (ADR 0012); an STM-style logistic-normal family is the reference, not a shipped-backend claim (Blei & Lafferty, 2006; Roberts et al., 2014, 2019). Topic proportions are compositional (Aitchison, 1982); downstream correlation/ESEM uses logistic-normal coordinates or appropriate orthonormal log-ratio coordinates rather than naïve raw-proportion Pearson correlation. -Longitudinal ESEM/DSEM must distinguish stable between-unit differences from within-unit temporal change, test measurement invariance where comparisons require it, account for irregular intervals when necessary, and propagate topic-posterior uncertainty through plausible values or joint estimation. +Longitudinal ESEM/DSEM (Asparouhov & Muthén, 2009; Asparouhov et al., 2018; Marsh et al., 2014) must distinguish stable between-unit differences from within-unit temporal change, test measurement invariance where comparisons require it (American Educational Research Association, American Psychological Association, & National Council on Measurement in Education, 2014), account for irregular intervals when necessary, and propagate topic-posterior uncertainty through plausible values or joint estimation. These psychometric targets remain accepted-target. ## 8. Compute requirements @@ -72,3 +72,33 @@ Documents and LLM outputs are untrusted data. Live model tests use `NVIDIA_NIM_A ## 13. Quality and release Production Rust line and branch coverage are exactly 100%; public API rustdoc is complete; format/build/Clippy/tests/rustdoc/supply-chain/security gates are warning-free/current-head. Releases additionally require validated migrations/rollback, SBOM/provenance, reproducible artifacts, current protected-head review/security, CHANGELOG/version consistency, operational recovery, and no unresolved scientific blocker. + +## References + +The full APA 7th register is [`docs/research/standards-and-literature.md`](research/standards-and-literature.md). Method claims in this TRD use: + +Aitchison, J. (1982). The statistical analysis of compositional data. *Journal of the Royal Statistical Society: Series B, 44*(2), 139–177. https://doi.org/10.1111/j.2517-6161.1982.tb01195.x + +Allan, J. (Ed.). (2002). *Topic detection and tracking: Event-based information organization*. Kluwer Academic Publishers. + +Allen, J. F. (1983). Maintaining knowledge about temporal intervals. *Communications of the ACM, 26*(11), 832–843. https://doi.org/10.1145/182.358434 + +American Educational Research Association, American Psychological Association, & National Council on Measurement in Education. (2014). *Standards for educational and psychological testing*. American Educational Research Association. + +Anagnostopoulos, E., Batsakis, S., & Petrakis, E. G. M. (2013). CHRONOS: A reasoning engine for qualitative temporal information in OWL. *Procedia Computer Science, 22*, 70–77. https://doi.org/10.1016/j.procs.2013.09.082 + +Asparouhov, T., Hamaker, E. L., & Muthén, B. (2018). Dynamic structural equation models. *Structural Equation Modeling, 25*(3), 359–388. https://doi.org/10.1080/10705511.2017.1406803 + +Asparouhov, T., & Muthén, B. (2009). Exploratory structural equation modeling. *Structural Equation Modeling, 16*(3), 397–438. https://doi.org/10.1080/10705510903008204 + +Blei, D. M., & Lafferty, J. D. (2006). Dynamic topic models. In *Proceedings of the 23rd International Conference on Machine Learning* (pp. 113–120). ACM. https://doi.org/10.1145/1143844.1143859 + +Hobbs, J. R., & Pan, F. (2017). *Time ontology in OWL* (W3C Recommendation). World Wide Web Consortium. https://www.w3.org/TR/owl-time/ + +International Organization for Standardization. (2012). *Language resource management—Semantic annotation framework (SemAF)—Part 1: Time and events (SemAF-Time, ISO-TimeML)* (ISO 24617-1:2012). + +Marsh, H. W., Morin, A. J. S., Parker, P. D., & Kaur, G. (2014). Exploratory structural equation modeling: An integration of the best features of exploratory and confirmatory factor analysis. *Annual Review of Clinical Psychology, 10*, 85–110. https://doi.org/10.1146/annurev-clinpsy-032813-153700 + +Roberts, M. E., Stewart, B. M., Tingley, D., Lucas, C., Leder-Luis, J., Gadarian, S. K., Albertson, B., & Rand, D. G. (2014). Structural topic models for open-ended survey responses. *American Journal of Political Science, 58*(4), 1064–1082. https://doi.org/10.1111/ajps.12103 + +Roberts, M. E., Stewart, B. M., & Tingley, D. (2019). stm: An R package for structural topic models. *Journal of Statistical Software, 91*(2), 1–40. https://doi.org/10.18637/jss.v091.i02 diff --git a/docs/UML.md b/docs/UML.md index 456fec770..ee51de6b5 100644 --- a/docs/UML.md +++ b/docs/UML.md @@ -36,7 +36,7 @@ flowchart LR INT --> ART ``` -On current protected main the workspace/evidence foundation and canonical documentation/ADR authority graph are implemented. Typed six-clock values/uncertain intervals are on canonical replacement PR #8. Legacy PR #6 contains Task 4 Allen/path-consistency work on the superseded PR #5 stack and is not current-lineage implementation evidence until replayed and revalidated. Later boxes are accepted-target. +On current protected main the workspace/evidence foundation, canonical documentation/ADR authority graph, typed six-clock values/uncertain intervals (merged PR #8), and Allen interval algebra/bounded path-consistency (merged PR #9; Allen, 1983) are implemented. Superseded PRs #5 and #6 are historical lineage only. Later boxes such as TRSL-TM, TDT/CHRONOS, and ESEM/DSEM remain accepted-target unless TRACEABILITY records a narrower subset. ## Evidence-to-analysis sequence @@ -67,7 +67,7 @@ sequenceDiagram Artifact-->>Analyst: accessible tables/graphs/manifest ``` -PR #8 provides typed clock/interval primitives only. Persistence/corpus-split enforcement of historical cutoff eligibility and graph integration remain accepted-target rather than as-built leakage protection. +Merged PR #8 provides typed clock/interval primitives on protected `main`. Leakage-safe corpus splits are implemented-main per TRACEABILITY; remaining graph/estimator integration follows owning ADRs. An unmerged PR is not as-built leakage protection. ## Six-clock availability state rule @@ -82,7 +82,7 @@ stateDiagram-v2 analysis_snapshot --> [*] ``` -A later document may report an earlier event, but it cannot be inserted into an earlier historical analysis before its availability time. PR #8 implements the typed clock/interval primitives; persistence/split enforcement of `cutoff_eligible` remains accepted-target. +A later document may report an earlier event, but it cannot be inserted into an earlier historical analysis before its availability time. Merged PR #8 implements the typed clock/interval primitives; `corpus_split` cutoff eligibility is implemented-main per TRACEABILITY. ## Relation authority view @@ -140,8 +140,8 @@ GPU may be absent or fall back to CPU. A GPU result is not accepted merely becau stateDiagram-v2 [*] --> workspace_foundation workspace_foundation --> immutable_evidence: protected_main - immutable_evidence --> six_clock_temporal: PR_8_active_replacement - six_clock_temporal --> interval_reasoning: Task_4_replay_required + immutable_evidence --> six_clock_temporal: PR_8_merged_implemented_main + six_clock_temporal --> interval_reasoning: PR_9_merged_implemented_main interval_reasoning --> event_relation_membership: accepted_target event_relation_membership --> persistence_and_splits: accepted_target persistence_and_splits --> topic_measurement: accepted_target @@ -152,7 +152,7 @@ stateDiagram-v2 ### Legacy stack note -PR #5 is superseded/conflicted lineage for Task 3. PR #6 remains a legacy Draft carrying Task 4 implementation history but is based on that superseded lineage. Its unique behavior must be replayed onto PR #8 or the exact protected-main descendant after PR #8 merges; old checks/reviews do not transfer. +PR #5 is superseded/conflicted lineage for Task 3. PR #6 is superseded Task 4 history on that discarded stack. The protected-main implementation is merged PR #8 plus merged PR #9; those drafts are not current-product claims. Old checks/reviews do not transfer as current-head evidence. ## Maintenance rule diff --git a/docs/adr/0002-six-clock-temporal-semantics.md b/docs/adr/0002-six-clock-temporal-semantics.md index 758a94192..df96df387 100644 --- a/docs/adr/0002-six-clock-temporal-semantics.md +++ b/docs/adr/0002-six-clock-temporal-semantics.md @@ -1,6 +1,7 @@ # ADR 0002 — Six-clock temporal semantics and leakage prevention **Decision status:** Accepted +**Implementation maturity:** partial — typed six-clock values and uncertain intervals are implemented-main on protected `main` (merged PR #8 / `temporal_core`); Allen interval algebra and bounded path-consistency are implemented-main on protected `main` (merged PR #9 / `temporal_core`). Superseded PRs #5 and #6 are historical lineage only and are not current-product claims. Downstream estimator, event-intelligence, and remaining persistence-policy uses of these primitives follow their owning ADRs and [`docs/TRACEABILITY.md`](../TRACEABILITY.md). **Implementation maturity:** active-PR — unmerged PR #8 is the canonical Task 3 replacement implementing typed clocks/intervals against the current protected-main lineage; superseded/conflicted PR #5 is historical lineage only; downstream transition/split enforcement remains accepted-target **Date:** 2026-08-05 **Supersedes:** None. ADR 0013 owns persistence/split representation; ADR 0016 owns event-intelligence reasoning above these temporal primitives. @@ -13,13 +14,13 @@ TEPP also needs uncertain, open, overlapping, and irregular intervals. A total o ## Decision -TEPP stores event/valid time, assertion time, document time, system time, availability time, and model knowledge cutoff as different typed values. Uncertain and open intervals retain boundary semantics, source precision, and provenance. +TEPP stores event/valid time, assertion time, document time, system time, availability time, and model knowledge cutoff as different typed values. Uncertain and open intervals retain boundary semantics, source precision, and provenance. Interval topology follows Allen's thirteen elementary relations (Allen, 1983). Event and time-marking vocabulary is aligned with ISO-TimeML (International Organization for Standardization, 2012). Outward instant/interval relation names may map to OWL-Time (Hobbs & Pan, 2017). These sources define the temporal algebra and annotation contract; they do not by themselves claim a complete TimeML corpus annotator or unrestricted global satisfiability. Historical analysis includes evidence only when its governed availability interval is fully eligible for the analysis cutoff. The practical invariant is `available_time <= knowledge_cutoff`; uncertain availability that can extend beyond the cutoff fails closed unless a versioned policy explicitly defines a conservative admissible interpretation. Forward transition, state-change, and input→process→outcome edges require a valid event-time partial order. Citation, revision, translation, support, contradiction, summary, and retrospective-reporting edges may point to the past but cannot become reverse transitions. Derived interval relations retain source evidence and the reasoner claim boundary. -Legacy PR #6 contains Allen/path-consistency work stacked on the superseded PR #5 lineage. It may narrow possible interval relations, but it is not eligible to advance until its unique Task 4 work is replayed on the canonical PR #8 lineage (or the exact protected-main descendant after PR #8 merges) and revalidated. Path consistency is not documented as unrestricted global satisfiability. +Superseded PRs #5 and #6 retain earlier clock and Allen/path-consistency work on a discarded lineage. They are historical audit evidence only. The protected-main implementation is the merged PR #8 temporal foundation plus the merged PR #9 replay of Task 4 interval algebra; those pull requests are lineage, not a living unmerged product. Path consistency is not documented as unrestricted global satisfiability. ## Non-goals @@ -54,8 +55,16 @@ Wire/database/API contracts retain clock type, interval boundaries, precision, a Property and integration tests cover nominal clock separation, interval algebra, uncertain/open boundaries, timezone/DST normalization, contradiction detection, transition cycles, historical snapshots, delayed availability, cutoff-crossing uncertainty, rolling-origin/relation-aware partitioning, and synthetic truth with known event and document processes. -PR #8 must re-run exact-head repository, line/branch coverage, rustdoc, dependency/security, and current-review gates on the replacement lineage. Historical checks from PR #5 do not transfer as merge evidence even though the tested production/test blobs preserve its TDD implementation lineage. +Protected-main `temporal_core` must continue to pass exact-head repository, line/branch coverage, rustdoc, dependency/security, and current-review gates. Historical checks from superseded PRs #5 and #6 do not transfer as current-head merge evidence even when later PRs preserved their TDD implementation lineage. ## Rollback and supersession Rollback selects the previous temporal contract/version and recomputes dependent snapshots/artifacts; it never reinterprets already-published evidence silently. Supersede only with a contract that explicitly migrates all six clock meanings and preserves or deliberately changes leakage semantics with new validation evidence. + +## References + +Allen, J. F. (1983). Maintaining knowledge about temporal intervals. *Communications of the ACM, 26*(11), 832–843. https://doi.org/10.1145/182.358434 + +Hobbs, J. R., & Pan, F. (2017). *Time ontology in OWL* (W3C Recommendation). World Wide Web Consortium. https://www.w3.org/TR/owl-time/ + +International Organization for Standardization. (2012). *Language resource management—Semantic annotation framework (SemAF)—Part 1: Time and events (SemAF-Time, ISO-TimeML)* (ISO 24617-1:2012). diff --git a/docs/adr/0003-relational-event-multiple-membership.md b/docs/adr/0003-relational-event-multiple-membership.md index 6b7cd347e..0fe84436d 100644 --- a/docs/adr/0003-relational-event-multiple-membership.md +++ b/docs/adr/0003-relational-event-multiple-membership.md @@ -1,13 +1,14 @@ # ADR 0003 — Relational event ontology and time-varying multiple membership **Decision status:** Accepted +**Implementation maturity:** partial — membership network and event mention/instance separation are implemented-main; typed relation graph with forward-only transitions is implemented-main. Multilevel psychometric estimators remain accepted-target. Remaining persistence details follow ADR 0013 and [`docs/TRACEABILITY.md`](../TRACEABILITY.md). **Implementation maturity:** partial — membership network and event mention/instance separation implemented-main; typed relation graph with forward-only transitions implemented-main; multilevel estimators and persistence remain accepted-target **Date:** 2026-08-05 **Supersedes:** None. ADR 0016 owns TDT/CHRONOS event-intelligence task semantics; this ADR remains authoritative for ontology, relation, role, and membership structure. ## Context -Documents, passages, events, entities, revisions, translations, projects, organizations, templates, authors, and time-varying roles are not independent observations. Treating each document as an atom can produce atomistic fallacy, inflate effective sample size, leak related variants across validation splits, and erase the fact that one observation can belong to several non-nested contexts at once. +Documents, passages, events, entities, revisions, translations, projects, organizations, templates, authors, and time-varying roles are not independent observations. Treating each document as an atom can produce atomistic fallacy, inflate effective sample size, leak related variants across validation splits, and erase the fact that one observation can belong to several non-nested contexts at once (American Educational Research Association, American Psychological Association, & National Council on Measurement in Education, 2014; Fox & Glas, 2001). Customer, partner, and competitor are especially contextual roles rather than permanent entity types. The same organization can occupy different roles across projects, events, markets, and time. @@ -19,6 +20,8 @@ Authors, departments, organizations, customers, partners, competitors, projects, Observed relation evidence, inferred relations, and promoted transition edges remain distinct. Relation absence is not silently interpreted as evidence of no relationship. +This ADR names a **relational event ontology** and time-varying membership structure. It does not adopt, and TEPP does not implement, a statistical relational-event-model (REM) estimator family. Multilevel/non-independence measurement follows Fox and Glas (2001) and the clustered-observation discipline in the *Standards for Educational and Psychological Testing* (American Educational Research Association et al., 2014). Production multilevel IRT/ESEM/DSEM estimators remain accepted-target under ADR 0005. + ## Non-goals - do not force observations into one hierarchy; @@ -55,3 +58,9 @@ Tests recover known event/relation graphs and membership effects, reject invalid ## Rollback and supersession Rollback restores the previous ontology/relation/membership contract and revalidates dependent model artifacts. Supersede only through an ADR that preserves explicit evidence provenance and non-nested/multiple-membership semantics or deliberately changes the estimand with corresponding PRD and validation updates. + +## References + +American Educational Research Association, American Psychological Association, & National Council on Measurement in Education. (2014). *Standards for educational and psychological testing*. American Educational Research Association. + +Fox, J.-P., & Glas, C. A. W. (2001). Bayesian estimation of a multilevel IRT model using Gibbs sampling. *Psychometrika, 66*(2), 271–288. https://doi.org/10.1007/BF02294839 diff --git a/docs/adr/0004-shared-multilingual-latent-space.md b/docs/adr/0004-shared-multilingual-latent-space.md index c8da25b55..f5cdc730f 100644 --- a/docs/adr/0004-shared-multilingual-latent-space.md +++ b/docs/adr/0004-shared-multilingual-latent-space.md @@ -15,6 +15,8 @@ The product therefore needs a language-independent analytical contract while pre TEPP learns one shared latent semantic and topic space across languages. Equivalent meanings share concept prototypes, topic identities, and document coordinates. Language-specific morphology, script, syntax, lexical emissions, and content deviations remain explicit rather than being forced to match. +This ADR owns the multilingual measurement substrate. The product topic-estimator contract is **Temporal Relational Shared-Latent Topic Measurement (TRSL-TM)** under ADR 0012. Temporal topic identity follows the dynamic topic-model family (Blei & Lafferty, 2006). An STM-style logistic-normal document-coordinate model (Roberts et al., 2014, 2019) is the **reference family**, not a claim that every compliant backend is already shipped. Implementation maturity for this ADR and for TRSL-TM remains accepted-target. + Original text and exact source spans are preserved. Segmentation and morphology are language-tailored. Universal POS/dependency information may act as soft source evidence or priors but does not authorize irreversible deletion. LLM-proposed semantic units must resolve to exact source evidence and a versioned concept/semantic contract; unknown meaning can remain unresolved rather than being forced into a known concept. Stopword deletion is not the default; TF-IDF and BM25 are not inferential weights for the statistical estimator. Repeated template/section/copied/style/prompt/modality/background wording is modeled as method/background structure so boilerplate does not masquerade as substantive latent meaning. @@ -57,3 +59,11 @@ Parallel/comparable corpora and human-reviewed evidence test exact-span F1, conc ## Rollback and supersession Rollback selects the prior validated language/concept/profile version and does not silently remap already-published artifacts. Supersede only through a decision that maintains cross-language comparability and explicit validity status or intentionally changes the measurement target with new PRD/validation evidence. + +## References + +Blei, D. M., & Lafferty, J. D. (2006). Dynamic topic models. In *Proceedings of the 23rd International Conference on Machine Learning* (pp. 113–120). ACM. https://doi.org/10.1145/1143844.1143859 + +Roberts, M. E., Stewart, B. M., & Tingley, D. (2019). stm: An R package for structural topic models. *Journal of Statistical Software, 91*(2), 1–40. https://doi.org/10.18637/jss.v091.i02 + +Roberts, M. E., Stewart, B. M., Tingley, D., Lucas, C., Leder-Luis, J., Gadarian, S. K., Albertson, B., & Rand, D. G. (2014). Structural topic models for open-ended survey responses. *American Journal of Political Science, 58*(4), 1064–1082. https://doi.org/10.1111/ajps.12103 diff --git a/docs/adr/0005-posterior-esem-dsem.md b/docs/adr/0005-posterior-esem-dsem.md index 09e5b0ce5..2209c07b7 100644 --- a/docs/adr/0005-posterior-esem-dsem.md +++ b/docs/adr/0005-posterior-esem-dsem.md @@ -13,11 +13,13 @@ TEPP also needs to distinguish stable between-unit differences from within-unit ## Decision -Topic proportions are not treated as error-free ordinary indicators. TEPP uses logistic-normal latent coordinates or valid orthonormal log-ratio coordinates and propagates topic posterior uncertainty through plausible values or a joint text-measurement/structural model. +Topic proportions are not treated as error-free ordinary indicators. Compositional parts are not ordinary Euclidean measurements (Aitchison, 1982). TEPP uses logistic-normal latent coordinates or valid orthonormal log-ratio coordinates and propagates topic posterior uncertainty through plausible values or a joint text-measurement/structural model. -Before ESEM/SEM interpretation, each higher-order construct is classified as reflective, formative/composite, network, or unresolved. Reflective indicators may use ESEM/set-ESEM; formative structures use composite/formative models; interacting structures use network models. A good global fit statistic is not authority to reinterpret a formative/network structure as reflective. +Before ESEM/SEM interpretation, each higher-order construct is classified as reflective, formative/composite, network, or unresolved. Reflective indicators may use exploratory structural equation modeling (Asparouhov & Muthén, 2009; Marsh et al., 2014); formative structures use composite/formative models; interacting structures use network models. A good global fit statistic is not authority to reinterpret a formative/network structure as reflective. -Longitudinal analysis evaluates measurement invariance at the level needed for the claimed comparison, supports partial/approximate or time-varying loadings where scientifically justified, separates stable between-unit components from within-unit temporal change, and handles irregular intervals through appropriate discrete- or continuous-time dynamics. +Longitudinal analysis evaluates measurement invariance at the level needed for the claimed comparison (American Educational Research Association, American Psychological Association, & National Council on Measurement in Education, 2014), supports partial/approximate or time-varying loadings where scientifically justified, separates stable between-unit components from within-unit temporal change, and handles irregular intervals through dynamic structural equation models (Asparouhov et al., 2018). + +This ADR remains **accepted-target**. Naming ESEM/DSEM and compositional coordinates as the model-family contract is not a protected-main implementation claim. Input/process/intervention/outcome paths obey event-time order. Temporal precedence, document linkage, event tracking, or model prediction alone do not justify causal language. @@ -57,3 +59,15 @@ Synthetic studies recover loadings, cross-loadings, factors, lagged/direct/indir ## Rollback and supersession Rollback selects the last validated structural-model version and compatible upstream model artifact. Supersede only if a later decision preserves explicit construct classification, uncertainty propagation, invariance/longitudinal evidence, and claim discipline or deliberately changes those estimands with a PRD update. + +## References + +Aitchison, J. (1982). The statistical analysis of compositional data. *Journal of the Royal Statistical Society: Series B, 44*(2), 139–177. https://doi.org/10.1111/j.2517-6161.1982.tb01195.x + +American Educational Research Association, American Psychological Association, & National Council on Measurement in Education. (2014). *Standards for educational and psychological testing*. American Educational Research Association. + +Asparouhov, T., Hamaker, E. L., & Muthén, B. (2018). Dynamic structural equation models. *Structural Equation Modeling, 25*(3), 359–388. https://doi.org/10.1080/10705511.2017.1406803 + +Asparouhov, T., & Muthén, B. (2009). Exploratory structural equation modeling. *Structural Equation Modeling, 16*(3), 397–438. https://doi.org/10.1080/10705510903008204 + +Marsh, H. W., Morin, A. J. S., Parker, P. D., & Kaur, G. (2014). Exploratory structural equation modeling: An integration of the best features of exploratory and confirmatory factor analysis. *Annual Review of Clinical Psychology, 10*, 85–110. https://doi.org/10.1146/annurev-clinpsy-032813-153700 diff --git a/docs/adr/0012-temporal-relational-shared-latent-topic-measurement.md b/docs/adr/0012-temporal-relational-shared-latent-topic-measurement.md index c3d5085fd..cca839523 100644 --- a/docs/adr/0012-temporal-relational-shared-latent-topic-measurement.md +++ b/docs/adr/0012-temporal-relational-shared-latent-topic-measurement.md @@ -13,14 +13,16 @@ Without a dedicated decision, implementations could silently drift toward indepe ## Decision -TEPP adopts **Temporal Relational Shared-Latent Topic Measurement (TRSL-TM)** as the model-family contract. The initial reference backend is a temporal/relational shared-latent STM-style estimator with logistic-normal document coordinates and posterior uncertainty. Alternative polylingual or neural/contextual backends are allowed only when they satisfy the same versioned evidence, temporal, relational, posterior, invariance, and interoperability contracts. +TEPP adopts **Temporal Relational Shared-Latent Topic Measurement (TRSL-TM)** as the product model-family contract. TRSL-TM is the accepted-target estimator identity; it is not a claim that a production topic backend is already shipped on protected `main`. + +The reference family is a temporal/relational shared-latent STM-style estimator with logistic-normal document coordinates and posterior uncertainty (Roberts et al., 2014, 2019; Chang & Blei, 2009). Temporal topic identity over a modeled period follows the dynamic topic-model family (Blei & Lafferty, 2006): one global topic identity set may change in prevalence without silently becoming a new topic merely because a time slice was fitted independently. Alternative polylingual or neural/contextual backends are allowed only when they satisfy the same versioned evidence, temporal, relational, posterior, invariance, and interoperability contracts. For the first production line: - one global topic identity set is selected across the modeled period; - topics may be active, dormant, or reactivated over time without losing identity; - topic birth/split/merge/retirement is a later explicit lineage extension, not an implicit consequence of fitting unrelated time slices; -- multilingual semantic/concept evidence shares topic identities, while native lexical/morphological channels remain language-specific; +- multilingual semantic/concept evidence shares topic identities, while native lexical/morphological channels remain language-specific (Mimno et al., 2009); - repeated template, section, copied-text, style, modality, source, and corpus-background effects are modeled explicitly as method/background structure; - stopword deletion is not the default preprocessing rule; - TF-IDF and BM25 are not inferential weights for the statistical topic estimator; @@ -66,3 +68,15 @@ Required evidence includes known-truth topic/covariate/covariance recovery, bias ## Rollback and supersession Rollback selects the last validated model/backend contract and immutable model artifact. Supersede only with evidence that the new model family preserves or explicitly and deliberately changes the estimand, with corresponding PRD/ADR and migration updates. + +## References + +Blei, D. M., & Lafferty, J. D. (2006). Dynamic topic models. In *Proceedings of the 23rd International Conference on Machine Learning* (pp. 113–120). ACM. https://doi.org/10.1145/1143844.1143859 + +Chang, J., & Blei, D. M. (2009). Relational topic models for document networks. In *Proceedings of the 12th International Conference on Artificial Intelligence and Statistics* (pp. 81–88). PMLR. + +Mimno, D., Wallach, H. M., Naradowsky, J., Smith, D. A., & McCallum, A. (2009). Polylingual topic models. In *Proceedings of the 2009 Conference on Empirical Methods in Natural Language Processing* (pp. 880–889). Association for Computational Linguistics. + +Roberts, M. E., Stewart, B. M., & Tingley, D. (2019). stm: An R package for structural topic models. *Journal of Statistical Software, 91*(2), 1–40. https://doi.org/10.18637/jss.v091.i02 + +Roberts, M. E., Stewart, B. M., Tingley, D., Lucas, C., Leder-Luis, J., Gadarian, S. K., Albertson, B., & Rand, D. G. (2014). Structural topic models for open-ended survey responses. *American Journal of Political Science, 58*(4), 1064–1082. https://doi.org/10.1111/ajps.12103 diff --git a/docs/adr/0016-tdt-chronos-event-intelligence-boundary.md b/docs/adr/0016-tdt-chronos-event-intelligence-boundary.md index 45731fdb1..47f79d55d 100644 --- a/docs/adr/0016-tdt-chronos-event-intelligence-boundary.md +++ b/docs/adr/0016-tdt-chronos-event-intelligence-boundary.md @@ -9,13 +9,15 @@ TEPP uses event intelligence for segmentation, linking, first-story detection, event tracking, schema instantiation, prediction, and temporal-consistency reasoning. These functions have different epistemic meanings. If they are collapsed into one opaque “event model,” a predicted event can be confused with an observed event, a topic link with a causal edge, or path consistency with proof of a complete real-world chronology. +The named intelligence-boundary sources already recorded in [`docs/research/standards-and-literature.md`](../research/standards-and-literature.md) are Topic Detection and Tracking (Allan, 2002), neural event-schema induction/prediction (Li et al., 2021), and CHRONOS qualitative temporal reasoning over OWL (Anagnostopoulos et al., 2013). This ADR remains **accepted-target**: naming those families is not a protected-main implementation claim. + ## Decision TEPP separates three event-intelligence layers: 1. **Event Ontology layer** owns versioned event instances, mentions, roles/arguments, subevents, evidence spans, places, products/outcomes, and provenance. Observed mentions are fallible evidence; event instances are modeled objects. -2. **TDT-style detection/tracking layer** owns story/event segmentation, link detection, new/first-story detection, topic/event detection, and longitudinal tracking. Its outputs are probabilistic measurement/detection evidence and require calibration and false-positive/false-negative evaluation. -3. **CHRONOS-style reasoning layer** has two explicit subcontracts: semantic/neural event-schema extraction/prediction and symbolic/qualitative temporal-consistency reasoning. A predicted event/schema completion remains hypothetical until supported by later evidence. Temporal reasoning validates consistency/partial order under its stated algebra and resource bounds; it does not prove unrestricted global satisfiability unless a later implementation explicitly does so. +2. **TDT-style detection/tracking layer** owns story/event segmentation, link detection, new/first-story detection, topic/event detection, and longitudinal tracking as defined by the TDT task family (Allan, 2002). Its outputs are probabilistic measurement/detection evidence and require calibration and false-positive/false-negative evaluation. +3. **CHRONOS-style reasoning layer** has two explicit subcontracts: semantic/neural event-schema extraction/prediction (Li et al., 2021) and symbolic/qualitative temporal-consistency reasoning in the sense of CHRONOS (Anagnostopoulos et al., 2013). A predicted event/schema completion remains hypothetical until supported by later evidence. Temporal reasoning validates consistency/partial order under its stated algebra and resource bounds; it does not prove unrestricted global satisfiability unless a later implementation explicitly does so. Transition edges admitted to the state/input-process-outcome graph remain governed by ADR 0002/0003 and cannot be created merely because TDT/CHRONOS predicts or links two events. Retrospective evidence and schema predictions remain provenance/hypothesis edges until independently promoted. @@ -52,3 +54,11 @@ Required evidence includes segmentation accuracy, link precision/recall, first-s ## Rollback and supersession Rollback selects the last validated event-intelligence model/reasoner version and preserves previously issued artifacts with their version/status. Supersede only with an ADR that maintains explicit separation of observed evidence, inferred event identity/relation, prediction, temporal consistency, and transition authority. + +## References + +Allan, J. (Ed.). (2002). *Topic detection and tracking: Event-based information organization*. Kluwer Academic Publishers. + +Anagnostopoulos, E., Batsakis, S., & Petrakis, E. G. M. (2013). CHRONOS: A reasoning engine for qualitative temporal information in OWL. *Procedia Computer Science, 22*, 70–77. https://doi.org/10.1016/j.procs.2013.09.082 + +Li, M., Li, S., Wang, Z., Huang, L., Cho, K., Ji, H., Han, J., & Voss, C. (2021). The future is not one-dimensional: Complex event schema induction by graph modeling for event prediction. In *Proceedings of the 2021 Conference on Empirical Methods in Natural Language Processing* (pp. 5203–5215). Association for Computational Linguistics. https://doi.org/10.18653/v1/2021.emnlp-main.422 diff --git a/docs/adr/README.md b/docs/adr/README.md index 61dfe6421..705b0a6d1 100644 --- a/docs/adr/README.md +++ b/docs/adr/README.md @@ -6,6 +6,9 @@ Read [`ADR_POLICY.md`](ADR_POLICY.md) first. **Decision status and implementatio | ADR | Decision | Decision status | Implementation maturity | Clarification / supersession | |---|---|---|---|---| +| [0001](0001-rust-first-modular-msa.md) | Rust-first numerical core and CPU `f64` reference | Accepted | partial | ADR 0011 owns cross-service/MSA authority; 0001 retains numerical/backend authority. | +| [0002](0002-six-clock-temporal-semantics.md) | Six-clock temporal semantics and fail-closed historical leakage prevention | Accepted | partial | Typed clocks/intervals (merged PR #8) and Allen/path-consistency (merged PR #9) are implemented-main; superseded PRs #5/#6 are historical lineage only. Downstream estimator and remaining persistence-policy uses stay with their owning ADRs. | +| [0003](0003-relational-event-multiple-membership.md) | Relational event ontology and time-varying cross-classified multiple membership | Accepted | partial | Membership network/roles and forward-only relation graph are implemented-main; multilevel estimators remain accepted-target. This is an ontology/membership contract, not a statistical REM paper. ADR 0016 owns event-intelligence tasks. | | [0001](0001-rust-first-modular-msa.md) | Rust-first numerical core and CPU `f64` reference | Accepted | partial | Workspace foundation is implemented-main; checkpoint-versus-estimator authority is `checkpoint_authority` on the active PR. ADR 0011 owns cross-service/MSA authority; 0001 retains numerical/backend authority. | | [0002](0002-six-clock-temporal-semantics.md) | Six-clock temporal semantics and fail-closed historical leakage prevention | Accepted | active-PR | Unmerged PR #8 is the canonical Task 3 replacement implementing typed clocks/intervals against the current protected-main lineage; conflicted PR #5 is superseded lineage. Later graph/split enforcement remains target work. | | [0003](0003-relational-event-multiple-membership.md) | Relational event ontology and time-varying cross-classified multiple membership | Accepted | partial | Weighted time-varying membership network/roles are active-PR (PR #12); full multilevel estimators, graph ontology, and persistence remain accepted-target. ADR 0016 owns event-intelligence tasks. | diff --git a/docs/research/standards-and-literature.md b/docs/research/standards-and-literature.md index ab68b4763..87825eca4 100644 --- a/docs/research/standards-and-literature.md +++ b/docs/research/standards-and-literature.md @@ -10,9 +10,11 @@ Asparouhov, T., Hamaker, E. L., & Muthén, B. (2018). Dynamic structural equatio Asparouhov, T., & Muthén, B. (2009). Exploratory structural equation modeling. *Structural Equation Modeling: A Multidisciplinary Journal, 16*(3), 397–438. https://doi.org/10.1080/10705510903008204 +Fox, J.-P., & Glas, C. A. W. (2001). Bayesian estimation of a multilevel IRT model using Gibbs sampling. *Psychometrika, 66*(2), 271–288. https://doi.org/10.1007/BF02294839 + Marsh, H. W., Morin, A. J. S., Parker, P. D., & Kaur, G. (2014). Exploratory structural equation modeling: An integration of the best features of exploratory and confirmatory factor analysis. *Annual Review of Clinical Psychology, 10*, 85–110. https://doi.org/10.1146/annurev-clinpsy-032813-153700 -TEPP applies these sources to construct definition, score interpretation, reliability, validity evidence, uncertainty, consequences, longitudinal invariance, ESEM cross-loadings, and DSEM. Topic outputs are treated as fallible indicators or components only after their construct role is evaluated. +TEPP applies these sources to construct definition, score interpretation, reliability, validity evidence, uncertainty, consequences, longitudinal invariance, ESEM cross-loadings, DSEM, and multilevel/non-independence measurement. Fox and Glas (2001) support Bayesian multilevel IRT estimation; they are not evidence for every cross-classified or multiple-membership estimator TEPP may later implement. Topic outputs are treated as fallible indicators or components only after their construct role is evaluated. Clustered or cross-classified observations are not flattened into independent rows (Fox & Glas, 2001; American Educational Research Association et al., 2014). ## Structural, correlated, dynamic, relational, and multilingual topic models @@ -58,6 +60,8 @@ Raw topic proportions are not ordinary Euclidean measurements. TEPP uses logisti ## Time, events, and topic detection and tracking +Allen, J. F. (1983). Maintaining knowledge about temporal intervals. *Communications of the ACM, 26*(11), 832–843. https://doi.org/10.1145/182.358434 + International Organization for Standardization. (2012). *Language resource management—Semantic annotation framework (SemAF)—Part 1: Time and events (SemAF-Time, ISO-TimeML)* (ISO Standard No. 24617-1:2012). Hobbs, J. R., & Pan, F. (2017). *Time ontology in OWL* (W3C Recommendation). World Wide Web Consortium. https://www.w3.org/TR/owl-time/ @@ -66,7 +70,9 @@ Allan, J. (Ed.). (2002). *Topic detection and tracking: Event-based information Anagnostopoulos, E., Batsakis, S., & Petrakis, E. G. M. (2013). CHRONOS: A reasoning engine for qualitative temporal information in OWL. *Procedia Computer Science, 22*, 70–77. https://doi.org/10.1016/j.procs.2013.09.082 -TEPP uses interval and partial-order reasoning, bitemporal availability, leakage-safe cutoffs, TDT segmentation/link/detection/first-story/tracking tasks, and separate neural/symbolic event-schema and temporal-consistency layers. +Li, M., Li, S., Wang, Z., Huang, L., Cho, K., Ji, H., Han, J., & Voss, C. (2021). The future is not one-dimensional: Complex event schema induction by graph modeling for event prediction. In *Proceedings of the 2021 Conference on Empirical Methods in Natural Language Processing* (pp. 5203–5215). Association for Computational Linguistics. https://doi.org/10.18653/v1/2021.emnlp-main.422 + +TEPP uses Allen interval algebra and partial-order reasoning, ISO-TimeML/OWL-Time vocabulary, bitemporal availability, leakage-safe cutoffs, TDT segmentation/link/detection/first-story/tracking tasks, neural event-schema induction/prediction, and separate symbolic temporal-consistency reasoning (Allen, 1983; International Organization for Standardization, 2012; Hobbs & Pan, 2017; Allan, 2002; Li et al., 2021; Anagnostopoulos et al., 2013). Allen, J. F. (1983). Maintaining knowledge about temporal intervals. *Communications of the ACM, 26*(11), 832–843. https://doi.org/10.1145/182.358434. Interval relations constrain temporal reasoning; they do not make support, contradiction, summary, or `outcome_of` a state transition.