Skip to content
Closed
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension


Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
2 changes: 1 addition & 1 deletion ARCHITECTURE.md
Original file line number Diff line number Diff line change
Expand Up @@ -102,7 +102,7 @@ boundaries above remain the target modular MSA architecture.
| `operational_log` | operational logs; `try_record` is the only recording API; source text and source identity are not loggable; `persistence_postgres` `audit_event` inserts call the same gate |
| `service_tls` | production TLS bind gates and rustls server config |
| `derived_sensitivity` | derived topic/factor/relation outputs inherit source sensitivity |
| `longitudinal_core` | active-PR: within/between decomposition; refuse between-as-within; component RMSE |
| `longitudinal_core` | within/between decomposition; refuse between-as-within; component RMSE; event-time lagged-correlation standardization from lagged covariance plus both occasion-specific marginal variances with exact binary64 covariance-bound validation; arithmetic boundary only, not a DSEM/ctsem/state-transition estimator |
| `topic_lineage` | global topic identity across active/dormant/reactivated states |
| `network_analysis` | compositional cluster-pair gates; raw simplex is not Euclidean |
| `interpretation_gateway` | evidence-bounded LLM interpretations; not estimators or observed facts |
Expand Down
2 changes: 2 additions & 0 deletions CHANGELOG.md
Original file line number Diff line number Diff line change
Expand Up @@ -38,6 +38,8 @@ All notable changes to TEPP are documented here. The format follows Keep a Chang

## [Unreleased]

- `longitudinal_core` adds event-time lagged-correlation standardization that requires lagged covariance plus both occasion-specific marginal variances, enforces the exact binary64 covariance bound, and uses bounded division order to avoid representable overflow/underflow. The invalid one-sided covariance/earlier-variance `expected_autocorrelation` API is removed; this arithmetic boundary does not claim a DSEM/ctsem/state-transition estimator.

- `event_core` adds bounded Allen interval-consistency classification, atomic path-consistency closure, contradiction/resource refusals, and an explicit dependency-error fallback without claiming unrestricted global satisfiability.

- `psychometric_core` recovers the Driver, Oud, and Voelkle (2017, Table 2, p. 12 `MANIFESTTRAITVAR`; §7.1, p. 19; p. 16 `MANIFESTTRAITVARstd`; footnote 4; 2017-era ctsem `summary.ctsemFit.R`; JSS PDF re-opened 2026-08-27T14:20Z from https://www.jstatsoft.org/index.php/jss/article/download/v077i05/1104) scalar standardised manifest-trait variance on current main after `0ce16e8` dropped the pre-consolidation code while research notes already named the map (register items 83–84). Table 2 names `MANIFESTTRAITVAR` `Ψ_τ` the additional time-invariant variance-covariance on the measurement level and sets it `NULL` when there is no manifest trait. Equation 5 writes `Γ ~ N(τ, Ψ)` and names that covariance the manifest traits. Section 7.1 names manifest traits stable individual differences in indicator levels, distinct from process-level `TRAITVAR` `φ_ξ`. Page 16 prints standardised matrices with the suffix `std` when appropriate. The printed example on p. 16 is `discreteDRIFTstd`, not `MANIFESTTRAITVARstd`. Footnote 4 standardises using only the relevant variance, not the total. The relevant variance for that named indicator-level correlation is `MANIFESTTRAITVAR`, not process-level `TRAITVAR` and not residual `MANIFESTVAR` `θ`. The 2017-era source forms `MANIFESTTRAITVARstd` only when `MANIFESTTRAITVAR != 0`, as `solve(sqrt(diag(MANIFESTTRAITVAR) + ridging)) %&% MANIFESTTRAITVAR` when `verbose = TRUE`. OpenMx `%&%` is `t(A) %*% B %*% A`. Unlike `TRAITVARstd`, that formation adds `diag(c(ridging), n.manifest)`. The default `ridging = FALSE` adds 0, not `0.0001`; that ridge is a numerical hack and is not this exact map. The scalar correlation is `ψ / ψ = 1` after strictly positive `MANIFESTTRAITVAR`. Form strictly positive `ψ` first, then `1 / √ψ`, then `(1 / √ψ) ψ (1 / √ψ)`. Unstandardised `MANIFESTTRAITVAR` is defined for a zero trait; standardised `MANIFESTTRAITVAR` is not. Zero `MANIFESTTRAITVAR` skips forming `MANIFESTTRAITVARstd` in the 2017-era source and fails closed here. Indicator-level trait variance is an event-time structural quantity, so a non-event clock fails closed. `MANIFESTTRAITVAR` does not require stable `a < 0`. Distinct positive `ψ` recover the same 1. `trait / trait = 1` is `TRAITVARstd` and recovers the same number and remains a distinct named quantity. `θ` is `MANIFESTVAR` and is measurement error, not this correlation. Meredith (1993) remains unread (web search 2026-08-27T14:20Z: Springer/Cambridge Core paywalled; Unpaywall historically `is_oa: false`; Springer `content/pdf` is an HTML stub). Mislevy (1991, *Psychometrika, 56*, 177–196) remains unread on the same terms (DOI `10.1007/bf02294457`). Still not a Kalman filter, not a matrix `expm`, not ESEM estimation, not DSEM, and not ctsem estimation.
Expand Down
2 changes: 1 addition & 1 deletion crates/longitudinal_core/Cargo.toml
Original file line number Diff line number Diff line change
@@ -1,6 +1,6 @@
[package]
name = "longitudinal_core"
description = "Within/between decomposition gates and component RMSE."
description = "Longitudinal modeling gates, within/between decomposition, temporal association standardization, and recovery metrics."
version.workspace = true
edition.workspace = true
rust-version.workspace = true
Expand Down
336 changes: 336 additions & 0 deletions crates/longitudinal_core/src/association.rs
Original file line number Diff line number Diff line change
@@ -0,0 +1,336 @@
//! Event-time lagged association standardization.
//!
//! This module belongs to the Longitudinal Modeling bounded context. It
//! standardizes a lagged covariance only when both marginal variances are
//! available. A one-sided covariance/initial-variance ratio is deliberately
//! not exposed as an autocorrelation.

use crate::LongitudinalError;

/// Decompose a positive finite binary64 value into an exact integer
/// significand and a power-of-two exponent.
fn positive_binary_components(value: f64) -> (u64, i32) {
let bits = value.to_bits();
let exponent_bits = ((bits >> 52) & 0x7ff) as i32;
let fraction = bits & ((1_u64 << 52) - 1);
if exponent_bits == 0 {
(fraction, -1074)
} else {
((1_u64 << 52) | fraction, exponent_bits - 1023 - 52)
}
}

/// Compare two positive `u128 * 2^exponent` values without overflowing the
/// integer significands.
fn scaled_integer_leq(
left_significand: u128,
left_exponent: i32,
right_significand: u128,
right_exponent: i32,
) -> bool {
let left_bits = (u128::BITS - left_significand.leading_zeros()) as i32;
let right_bits = (u128::BITS - right_significand.leading_zeros()) as i32;
let left_order = left_exponent + left_bits;
let right_order = right_exponent + right_bits;
if left_order != right_order {
return left_order < right_order;
}
if left_exponent == right_exponent {
return left_significand <= right_significand;
}
if left_exponent > right_exponent {
let shift = (left_exponent - right_exponent) as u32;
return (left_significand << shift) <= right_significand;
}
let shift = (right_exponent - left_exponent) as u32;
left_significand <= (right_significand << shift)
Comment thread
seonghobae marked this conversation as resolved.
Resolved
}

/// Test the Cauchy–Schwarz covariance bound exactly for the supplied binary64
/// inputs rather than using a rounded floating-point square-root product.
fn covariance_within_binary_bound(
lagged_covariance: f64,
earlier_total_variance: f64,
later_total_variance: f64,
) -> bool {
let covariance_magnitude = lagged_covariance.abs();
if covariance_magnitude == 0.0 {
return true;
}
let (covariance_significand, covariance_exponent) =
positive_binary_components(covariance_magnitude);
let (earlier_significand, earlier_exponent) =
positive_binary_components(earlier_total_variance);
let (later_significand, later_exponent) =
positive_binary_components(later_total_variance);

let covariance_square =
u128::from(covariance_significand) * u128::from(covariance_significand);
let variance_product =
u128::from(earlier_significand) * u128::from(later_significand);
scaled_integer_leq(
covariance_square,
covariance_exponent * 2,
variance_product,
earlier_exponent + later_exponent,
)
}

/// Recover a Pearson correlation for an event-time lag from its covariance and
/// both marginal variances.
///
/// For observations at event times `t` and `t + Δ`, the correlation is
///
/// `Cov(Y_t, Y_{t+Δ}) / sqrt(Var(Y_t) * Var(Y_{t+Δ}))`.
///
/// Requiring both marginals is essential for nonstationary processes. Driver,
/// Oud, and Voelkle (2017) provide the continuous-time state-transition and
/// covariance components from which occasion-specific marginals can be built;
/// they do not justify replacing the second marginal variance with the first
/// when the process is nonstationary.
///
/// `event_interval` is semantically required to be strictly positive so a
/// measurement-occasion or method facet cannot be passed as an untyped lag.
/// This function does not infer either marginal variance and does not estimate
/// a state process.
///
/// # Errors
///
/// Returns [`LongitudinalError::InvalidTemporalAssociationInput`] for
/// non-finite inputs, [`LongitudinalError::NonPositiveMarginalVariance`] when
/// either marginal variance is not strictly positive,
/// [`LongitudinalError::NonPositiveEventInterval`] for a non-positive event
/// interval, and [`LongitudinalError::CovarianceBoundViolation`] when the
/// supplied covariance is incompatible with the two marginal variances.
pub fn recover_event_time_lagged_correlation(
lagged_covariance: f64,
earlier_total_variance: f64,
later_total_variance: f64,
event_interval: f64,
) -> Result<f64, LongitudinalError> {
if !lagged_covariance.is_finite()
|| !earlier_total_variance.is_finite()
|| !later_total_variance.is_finite()
|| !event_interval.is_finite()
{
return Err(LongitudinalError::InvalidTemporalAssociationInput);
}
if earlier_total_variance <= 0.0 || later_total_variance <= 0.0 {
return Err(LongitudinalError::NonPositiveMarginalVariance);
}
if event_interval <= 0.0 {
return Err(LongitudinalError::NonPositiveEventInterval);
}
Comment thread
seonghobae marked this conversation as resolved.
Resolved
if !covariance_within_binary_bound(
lagged_covariance,
earlier_total_variance,
later_total_variance,
) {
return Err(LongitudinalError::CovarianceBoundViolation);
}

let earlier_scale = earlier_total_variance.sqrt();
let later_scale = later_total_variance.sqrt();
// Divide by the smaller scale first. The exact covariance-bound gate above
// guarantees the intermediate magnitude cannot exceed the remaining
// marginal scale, while this order avoids underflow when the marginals are
// separated by hundreds of binary exponents.
let (first_scale, second_scale) = if earlier_scale <= later_scale {
(earlier_scale, later_scale)
} else {
(later_scale, earlier_scale)
};
Comment thread
seonghobae marked this conversation as resolved.
Resolved
let correlation = (lagged_covariance / first_scale) / second_scale;

// Finite positive marginals plus the exact covariance-bound gate guarantee
// that both divisions stay finite. Clamping only absorbs final
// square-root/division rounding at a valid ±1 boundary; it cannot admit an
// over-bound covariance.
Ok(correlation.clamp(-1.0, 1.0))
}

#[cfg(test)]
mod tests {
use super::{recover_event_time_lagged_correlation, scaled_integer_leq};
use crate::LongitudinalError;

#[test]
fn nonstationary_marginals_do_not_use_the_earlier_variance_twice() {
let recovered = recover_event_time_lagged_correlation(1.5, 1.0, 4.0, 1.0)
.expect("valid nonstationary covariance should standardize");
assert!((recovered - 0.75).abs() < f64::EPSILON * 4.0);
}

#[test]
fn covariance_bound_is_fail_closed() {
assert_eq!(
recover_event_time_lagged_correlation(2.000_000_000_1, 1.0, 4.0, 1.0),
Err(LongitudinalError::CovarianceBoundViolation)
);
assert_eq!(
recover_event_time_lagged_correlation(-2.000_000_000_1, 1.0, 4.0, 1.0),
Err(LongitudinalError::CovarianceBoundViolation)
);
}

#[test]
fn exact_binary_bound_rejects_one_ulp_excess_for_both_signs() {
let variance = 2.0_f64;
let one_ulp_above = f64::from_bits(variance.to_bits() + 1);
assert_eq!(
recover_event_time_lagged_correlation(one_ulp_above, variance, variance, 1.0),
Err(LongitudinalError::CovarianceBoundViolation)
);
assert_eq!(
recover_event_time_lagged_correlation(-one_ulp_above, variance, variance, 1.0),
Err(LongitudinalError::CovarianceBoundViolation)
);
}

#[test]
fn exact_binary_bound_accepts_extreme_and_subnormal_boundaries() {
assert_eq!(
recover_event_time_lagged_correlation(f64::MAX, f64::MAX, f64::MAX, 1.0),
Ok(1.0)
);
assert_eq!(
recover_event_time_lagged_correlation(-f64::MAX, f64::MAX, f64::MAX, 1.0),
Ok(-1.0)
);
let minimum_subnormal = f64::from_bits(1);
assert_eq!(
recover_event_time_lagged_correlation(
minimum_subnormal,
minimum_subnormal,
minimum_subnormal,
1.0,
),
Ok(1.0)
);
assert_eq!(
recover_event_time_lagged_correlation(
-minimum_subnormal,
minimum_subnormal,
minimum_subnormal,
1.0,
),
Ok(-1.0)
);
}

#[test]
fn gross_subnormal_bound_violation_is_classified_before_division() {
let minimum_subnormal = f64::from_bits(1);
assert_eq!(
recover_event_time_lagged_correlation(
1.0,
minimum_subnormal,
minimum_subnormal,
1.0,
),
Err(LongitudinalError::CovarianceBoundViolation)
);
}

#[test]
fn unequal_scales_do_not_underflow_a_representable_correlation() {
let recovered = recover_event_time_lagged_correlation(
f64::MIN_POSITIVE,
f64::MIN_POSITIVE,
f64::MAX,
1.0,
)
.expect("valid unequal-scale covariance should remain representable");
assert!(recovered > 0.0);
assert!(recovered.is_finite());
}

#[test]
fn scale_order_is_symmetric_when_the_earlier_marginal_is_larger() {
let recovered = recover_event_time_lagged_correlation(1.5, 4.0, 1.0, 1.0)
.expect("reversing marginal scale order should still standardize");
assert!((recovered - 0.75).abs() < f64::EPSILON * 4.0);
}

#[test]
fn zero_covariance_is_valid() {
assert_eq!(
recover_event_time_lagged_correlation(0.0, 1.0, 4.0, 1.0),
Ok(0.0)
);
}

#[test]
fn scaled_integer_comparison_covers_alignment_directions() {
assert!(scaled_integer_leq(1, 2, 4, 0));
assert!(!scaled_integer_leq(5, 0, 1, 2));
assert!(scaled_integer_leq(3, 0, 6, -1));
assert!(scaled_integer_leq(6, -1, 3, 0));
assert!(!scaled_integer_leq(7, -1, 3, 0));
}

#[test]
fn exact_boundary_correlations_are_allowed() {
assert_eq!(
recover_event_time_lagged_correlation(2.0, 1.0, 4.0, 1.0),
Ok(1.0)
);
assert_eq!(
recover_event_time_lagged_correlation(-2.0, 1.0, 4.0, 1.0),
Ok(-1.0)
);
}

#[test]
fn square_root_scaling_avoids_avoidable_variance_product_overflow() {
let variance = f64::MAX / 4.0;
let recovered = recover_event_time_lagged_correlation(
variance / 2.0,
variance,
variance,
1.0,
)
.expect("representable standardized covariance should remain representable");
assert!((recovered - 0.5).abs() < 1.0e-15);
}

#[test]
fn every_non_finite_input_position_fails_closed() {
assert_eq!(
recover_event_time_lagged_correlation(f64::NAN, 1.0, 1.0, 1.0),
Err(LongitudinalError::InvalidTemporalAssociationInput)
);
assert_eq!(
recover_event_time_lagged_correlation(0.0, f64::INFINITY, 1.0, 1.0),
Err(LongitudinalError::InvalidTemporalAssociationInput)
);
assert_eq!(
recover_event_time_lagged_correlation(0.0, 1.0, f64::INFINITY, 1.0),
Err(LongitudinalError::InvalidTemporalAssociationInput)
);
assert_eq!(
recover_event_time_lagged_correlation(0.0, 1.0, 1.0, f64::INFINITY),
Err(LongitudinalError::InvalidTemporalAssociationInput)
);
}

#[test]
fn either_non_positive_marginal_fails_closed() {
assert_eq!(
recover_event_time_lagged_correlation(0.0, 0.0, 1.0, 1.0),
Err(LongitudinalError::NonPositiveMarginalVariance)
);
assert_eq!(
recover_event_time_lagged_correlation(0.0, 1.0, 0.0, 1.0),
Err(LongitudinalError::NonPositiveMarginalVariance)
);
}

#[test]
fn non_positive_event_interval_fails_closed() {
assert_eq!(
recover_event_time_lagged_correlation(0.0, 1.0, 1.0, 0.0),
Err(LongitudinalError::NonPositiveEventInterval)
);
Comment thread
seonghobae marked this conversation as resolved.
Resolved
}
}
Loading
Loading