Skip to content
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension


Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
1 change: 1 addition & 0 deletions ARCHITECTURE.md
Original file line number Diff line number Diff line change
Expand Up @@ -61,6 +61,7 @@ boundaries above remain the target modular MSA architecture.
| `tepp_simulation` | known-truth temporal/event data generation |
| `validation_core` | RMSE, bias, coverage, graph, and Monte Carlo metrics |
| `tepp_api` | versioned DTO, schema, and export contracts |
| `model_selection` | statistical/Pareto candidate-`K` gates; LLM votes are not numerical authority |
| `checkpoint_authority` | a model checkpoint is not the CPU `f64` estimator |

No crate exposes placeholder production behavior in Task 1. This prevents an
Expand Down
1 change: 1 addition & 0 deletions CHANGELOG.md
Original file line number Diff line number Diff line change
Expand Up @@ -6,6 +6,7 @@ All notable changes to TEPP are documented here. The format follows Keep a Chang

### Added

- `model_selection` candidate-`K` gates: statistical candidates require `K >= 2` and finite held-out log-likelihood/complexity, a Pareto front excludes dominated alternatives, LLM votes cannot define the numerical optimum, and selected `K` recovers known truth with computed RMSE.
- `event_core` mention-confidence Brier score: known-truth binary outcomes recover a computed Brier of 0 for perfect forecasts and 0.25 for constant 0.5, with empty or mismatched streams failing closed.
- `membership_core` nested ICC: CPU `f64` unbalanced ANOVA recovers a known cluster ICC and refuses to treat cross-classified or multiple-membership designs as a single hierarchy (ADR 0003).
- `persistence_postgres` typed `text_segment` SQL: insert/lookup of exact UTF-8 half-open byte spans on the existing `0006` table, cutoff-eligible document reads (`available_time <= knowledge_cutoff`), and live recovery of a known `hello` span. No new migration number (`#45` still owns `0007`).
Expand Down
4 changes: 4 additions & 0 deletions Cargo.lock

Some generated files are not rendered by default. Learn more about how customized files appear on GitHub.

2 changes: 2 additions & 0 deletions Cargo.toml
Original file line number Diff line number Diff line change
Expand Up @@ -11,6 +11,7 @@ members = [
"crates/tepp_simulation",
"crates/validation_core",
"crates/tepp_api",
"crates/model_selection",
"crates/checkpoint_authority",
]
default-members = [
Expand All @@ -24,6 +25,7 @@ default-members = [
"crates/tepp_simulation",
"crates/validation_core",
"crates/tepp_api",
"crates/model_selection",
"crates/checkpoint_authority",
]

Expand Down
1 change: 1 addition & 0 deletions README.md

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

📝 Info: README prose still says eleven crates

README.md still reads "The eleven bounded crates" while the block below and the workspace now list 12 crates after model_selection was added. Stale prose.

(Refers to this code)

Open in Devin Review

Was this helpful? React with 👍 or 👎 to provide feedback.

Original file line number Diff line number Diff line change
Expand Up @@ -22,6 +22,7 @@ crates/corpus_split
crates/tepp_simulation
crates/validation_core
crates/tepp_api
crates/model_selection
crates/checkpoint_authority
```

Expand Down
17 changes: 17 additions & 0 deletions crates/model_selection/Cargo.toml
Original file line number Diff line number Diff line change
@@ -0,0 +1,17 @@
[package]
name = "model_selection"
description = "Statistical and Pareto candidate-K gates that refuse LLM numerical authority."
version.workspace = true
edition.workspace = true
rust-version.workspace = true
license.workspace = true
authors.workspace = true
repository.workspace = true
homepage.workspace = true
readme.workspace = true
keywords.workspace = true
categories.workspace = true
publish = false

[lints]
workspace = true
156 changes: 156 additions & 0 deletions crates/model_selection/src/candidate.rs
Original file line number Diff line number Diff line change
@@ -0,0 +1,156 @@
//! Candidate topic counts with statistical diagnostics.

use crate::ModelSelectionError;

/// One candidate `K` together with the diagnostics that may admit it.
#[derive(Clone, Copy, Debug, PartialEq)]
pub struct ModelCandidate {
candidate_k: u32,
held_out_log_likelihood: Option<f64>,
complexity: Option<f64>,
llm_vote_only: bool,
}

impl ModelCandidate {
/// Construct a statistically supported candidate.
///
/// # Errors
///
/// Returns [`ModelSelectionError::NonPositiveCandidateK`] when `candidate_k`
/// is less than two, or [`ModelSelectionError::InvalidDiagnostic`] when a
/// diagnostic is non-finite.
pub fn statistical(
candidate_k: u32,
held_out_log_likelihood: f64,
complexity: f64,
) -> Result<Self, ModelSelectionError> {
if candidate_k < 2 {
return Err(ModelSelectionError::NonPositiveCandidateK);
}
if !held_out_log_likelihood.is_finite() || !complexity.is_finite() || complexity < 0.0 {
return Err(ModelSelectionError::InvalidDiagnostic);
}
Ok(Self {
candidate_k,
held_out_log_likelihood: Some(held_out_log_likelihood),
complexity: Some(complexity),
llm_vote_only: false,
})
}

/// Construct a candidate whose only support is an LLM vote.
///
/// The vote may later recommend among statistically admissible candidates.
/// It cannot itself define the numerical optimum.
///
/// # Errors
///
/// Returns [`ModelSelectionError::NonPositiveCandidateK`] when `candidate_k`
/// is less than two.
pub fn llm_vote_only(candidate_k: u32) -> Result<Self, ModelSelectionError> {
if candidate_k < 2 {
return Err(ModelSelectionError::NonPositiveCandidateK);
}
Ok(Self {
candidate_k,
held_out_log_likelihood: None,
complexity: None,
llm_vote_only: true,
})
}
Comment thread
coderabbitai[bot] marked this conversation as resolved.

/// Return the candidate topic count.
#[must_use]
pub const fn candidate_k(self) -> u32 {
self.candidate_k
}

/// Return whether this candidate carries finite statistical diagnostics.
#[must_use]
pub const fn is_statistically_supported(self) -> bool {
match (self.held_out_log_likelihood, self.complexity) {
(Some(_), Some(_)) => !self.llm_vote_only,
_ => false,
}
}

/// Held-out log-likelihood when the candidate is statistically supported.
#[must_use]
pub const fn held_out_log_likelihood(self) -> Option<f64> {
self.held_out_log_likelihood
}

/// Complexity penalty (larger is worse) when statistically supported.
#[must_use]
pub const fn complexity(self) -> Option<f64> {
self.complexity
}

/// Return whether the candidate is an LLM vote without statistical support.
#[must_use]
pub const fn is_llm_vote_only(self) -> bool {
self.llm_vote_only
}

/// Return whether `self` Pareto-dominates `other` on likelihood and complexity.
#[must_use]
pub fn dominates(self, other: Self) -> bool {
let (Some(self_ll), Some(self_complexity), Some(other_ll), Some(other_complexity)) = (
self.held_out_log_likelihood,
self.complexity,
other.held_out_log_likelihood,
other.complexity,
) else {
return false;
};
let no_worse = self_ll >= other_ll && self_complexity <= other_complexity;
let strictly_better = self_ll > other_ll || self_complexity < other_complexity;
no_worse && strictly_better
}
}

#[cfg(test)]
mod tests {
use super::ModelCandidate;
use crate::ModelSelectionError;

#[test]
fn statistical_candidate_accessors_and_dominance_cover_branches() {
let better = ModelCandidate::statistical(4, -10.0, 5.0).expect("better");
let worse = ModelCandidate::statistical(8, -20.0, 9.0).expect("worse");
assert_eq!(better.candidate_k(), 4);
assert_eq!(better.held_out_log_likelihood(), Some(-10.0));
assert_eq!(better.complexity(), Some(5.0));
assert!(better.is_statistically_supported());
assert!(!better.is_llm_vote_only());
assert!(better.dominates(worse));
assert!(!worse.dominates(better));
assert!(!better.dominates(better));

let llm = ModelCandidate::llm_vote_only(3).expect("valid llm candidate");
assert!(llm.is_llm_vote_only());
assert!(!llm.is_statistically_supported());
assert!(!llm.dominates(better));
assert!(!better.dominates(llm));
assert_eq!(
ModelCandidate::statistical(0, -1.0, 1.0),
Err(ModelSelectionError::NonPositiveCandidateK)
);
assert_eq!(
ModelCandidate::statistical(2, -1.0, -0.1),
Err(ModelSelectionError::InvalidDiagnostic)
);
assert_eq!(
ModelCandidate::statistical(2, f64::NAN, 1.0),
Err(ModelSelectionError::InvalidDiagnostic)
);
assert_eq!(
ModelCandidate::statistical(2, -1.0, f64::INFINITY),
Err(ModelSelectionError::InvalidDiagnostic)
);
assert_eq!(
ModelCandidate::llm_vote_only(1),
Err(ModelSelectionError::NonPositiveCandidateK)
);
}
}
60 changes: 60 additions & 0 deletions crates/model_selection/src/error.rs
Original file line number Diff line number Diff line change
@@ -0,0 +1,60 @@
//! Fail-closed model-selection errors.

use std::fmt;

/// A fail-closed model-selection error.
#[derive(Clone, Copy, Debug, Eq, PartialEq)]
#[non_exhaustive]
pub enum ModelSelectionError {
/// Candidate `K` was less than two.
NonPositiveCandidateK,
Comment thread
seonghobae marked this conversation as resolved.
/// A diagnostic was non-finite or otherwise unusable.
InvalidDiagnostic,
/// No candidates were supplied.
EmptyCandidateSet,
/// An LLM vote was asked to define the numerical optimum.
LlmVoteIsNotStatisticalAuthority,
}

impl fmt::Display for ModelSelectionError {
fn fmt(&self, formatter: &mut fmt::Formatter<'_>) -> fmt::Result {
let message = match self {
Self::NonPositiveCandidateK => "candidate k must be at least two",
Self::InvalidDiagnostic => "invalid model-selection diagnostic",
Self::EmptyCandidateSet => "empty model-selection candidate set",
Self::LlmVoteIsNotStatisticalAuthority => "llm vote is not statistical authority",
};
formatter.write_str(message)
}
}

impl std::error::Error for ModelSelectionError {}

#[cfg(test)]
mod tests {
use super::ModelSelectionError;

#[test]
fn error_messages_are_stable() {
for (error, message) in [
(
ModelSelectionError::NonPositiveCandidateK,
"candidate k must be at least two",
),
(
ModelSelectionError::InvalidDiagnostic,
"invalid model-selection diagnostic",
),
(
ModelSelectionError::EmptyCandidateSet,
"empty model-selection candidate set",
),
(
ModelSelectionError::LlmVoteIsNotStatisticalAuthority,
"llm vote is not statistical authority",
),
] {
assert_eq!(error.to_string(), message);
}
}
}
Loading
Loading