Skip to content

Add confidence-interval (ACI) classification for CAT (catIrt-verified) - #257

Merged
seonghobae merged 2 commits into
mainfrom
seonghobae-ci-classify
Jul 26, 2026
Merged

Add confidence-interval (ACI) classification for CAT (catIrt-verified)#257
seonghobae merged 2 commits into
mainfrom
seonghobae-ci-classify

Conversation

@seonghobae

@seonghobae seonghobae commented Jul 25, 2026

Copy link
Copy Markdown
Contributor

Implements the confidence-interval (ACI) classification stopping rule for CAT as mlsirm_core::exposure::ci_classify with PyO3 binding py_ci_classify and thin Python wrapper fast_mlsirm.ci_classify.

Stacked on #256 (SPRT classification, seonghobae-sprt).

Contract (adversarial spec review: REDUCED-SCOPE approved)

  • Single cut, binary responses, D = 1 logistic 3PL.
  • Interim EAP per response prefix on a fixed 41-point uniform grid over [-4, 4], standard-normal log prior -0.5 theta^2 (matches the crate eap_interim convention).
  • SE = EAP posterior SD (sqrt(E[(theta - theta_hat)^2])), matching catIrt eapEst.R.
  • Interval theta_hat +/- z_crit * se vs theta_cut; FIRST STRICT crossing decides (lower > cut -> above, upper < cut -> below; equality continues), matching catIrt termCI.R.
  • Traces for ALL supplied responses; entries past n_used are offline counterfactual replay values (same convention as sprt_classify).
  • z_crit is caller-supplied; for confidence level L pass qnorm((1+L)/2) (catIrt conf.lev parameterization).

Citation governance

  • READ: catIrt R/termCI.R, R/eapEst.R, man/catIrt.Rd at pinned commit c9e979e4812c27d95d367a7f097edfe8e93ac8eb.
  • NOT read (historical/background only, marked in all docstrings): Kingsbury & Weiss (1983); Thompson (2007) CI method section; Eggen & Straetmans (2000). The spec reviewer could not obtain inspectable Thompson method text, so Thompson attribution is deliberately blocked from "verified" wording.

Evidence

  • Pinned 17-digit oracle from independent Python recomputation (spec review): all four traces match the crate at 1e-12; decision="above", n_used=5 with the first-strict-crossing anchor lower_trace[3] <= 0 < lower_trace[4].
  • Rust tests: oracle, below/continue paths, full error paths, MC-500 #[ignore] structural invariants (executed: passed).
  • Mutation kills EXECUTED (backup/mutate/test/restore, restored green):
    • M1 swapped above/below decisions -> 2 tests FAIL
    • M2 point-estimate-vs-cut instead of CI bound -> 2 tests FAIL
    • M3 posterior variance instead of SD (drop sqrt) -> 1 test FAIL (crossing moves to k=4; killed by the k=5 anchor)
    • M4 n_used off-by-one -> 2 tests FAIL
  • pyd rebuilt; binding smoke-tested against the oracle (exact match); pytest TestCiClassify 5 passed (complex-input rejection, exact 0/1 response validation before casts, bool responses, core validation propagation).

Adversarial impl-review outcome

Reviewer independently reproduced the pinned oracle (no repo imports; matched at 1e-12), re-ran all Rust/pytest suites, executed adversarial edge cases (single item, cut outside [-4,4], extreme a=1000, huge/tiny z_crit, bad/complex inputs — all finite, correct), and re-executed the M3 drop-sqrt mutation kill (FAIL then restored green).

  • CONFIRMED-DEFECT (behavior): none.
  • DOC-DEFECT: PyO3 docstring omitted Eggen & Straetmans (2000) from the NOT-read citation list. Fixed in a5d605b.

Status: CLEAN after fix.

Summary by CodeRabbit

  • 새로운 기능

    • CAT에서 신뢰구간(ACI) 기반 단일 컷 분류를 지원합니다.
    • 결과로 above, below, continue 판단과 사용된 응답 수를 제공합니다.
    • 능력 추정치, 표준오차 및 신뢰구간 경계에 대한 진단용 추적 정보를 제공합니다.
    • Python 패키지의 공개 API에서 해당 기능을 사용할 수 있습니다.
  • 문서

    • 새로운 ACI 분류 기능과 동작 방식이 변경 기록에 추가되었습니다.
  • 버그 수정

    • 응답 값, 복소수 입력 및 분류 파라미터에 대한 입력 검증을 강화했습니다.

Base automatically changed from seonghobae-sprt to seonghobae-epv July 26, 2026 09:03
@coderabbitai

coderabbitai Bot commented Jul 26, 2026

Copy link
Copy Markdown
Contributor

Review Change Stack

📝 Walkthrough

Walkthrough

고정 41점 능력 격자에서 ACI 신뢰구간 분류를 수행하는 Rust 코어와 Python API가 추가되었습니다. 분류 결과, 사용 응답 수, 단계별 진단 트레이스를 반환하며 입력 검증과 회귀 테스트가 포함되었습니다.

Changes

ACI 분류 기능

Layer / File(s) Summary
Rust ACI 분류 계약과 알고리즘
crates/mlsirm-core/src/exposure.rs, tests/unit/exposure_tests.rs
CiResultci_classify를 추가해 누적 EAP·posterior SE·신뢰구간을 계산하고, strict crossing에 따라 above, below, continue를 판정합니다. 입력 검증과 구조적 불변성 테스트도 추가되었습니다.
Python API 연결과 공개 검증
crates/fast-mlsirm-py/src/lib.rs, python/fast_mlsirm/exposure.py, python/fast_mlsirm/__init__.py, tests/test_paper_features.py, CHANGELOG.md
PyO3 바인딩과 Python 공개 심볼을 연결하고, 입력 변환·검증 및 진단 트레이스 반환을 구현했습니다. Python 회귀 테스트와 CHANGELOG 항목도 추가되었습니다.

Estimated code review effort: 4 (Complex) | ~45 minutes

Sequence Diagram(s)

sequenceDiagram
  participant PythonCaller
  participant fast_mlsirm_ci_classify
  participant py_ci_classify
  participant core_ci_classify
  PythonCaller->>fast_mlsirm_ci_classify: 입력 정규화 및 검증
  fast_mlsirm_ci_classify->>py_ci_classify: 분류 파라미터와 응답 전달
  py_ci_classify->>core_ci_classify: Rust 코어 호출
  core_ci_classify-->>py_ci_classify: decision, n_used, traces 반환
  py_ci_classify-->>PythonCaller: Python dict 반환
Loading

Suggested reviewers: copilot

🚥 Pre-merge checks | ✅ 5
✅ Passed checks (5 passed)
Check name Status Explanation
Description Check ✅ Passed Check skipped - CodeRabbit’s high-level summary is enabled.
Title check ✅ Passed CAT용 confidence-interval(ACI) 분류 기능 추가를 정확하게 요약한 제목입니다.
Docstring Coverage ✅ Passed Docstring coverage is 100.00% which is sufficient. The required threshold is 80.00%.
Linked Issues check ✅ Passed Check skipped because no linked issues were found for this pull request.
Out of Scope Changes check ✅ Passed Check skipped because no linked issues were found for this pull request.
✨ Finishing Touches
📝 Generate docstrings
  • Create stacked PR
  • Commit on current branch
🧪 Generate unit tests (beta)
  • Create PR with unit tests
  • Commit unit tests in branch seonghobae-ci-classify

Comment @coderabbitai help to get the list of available commands.

Base automatically changed from seonghobae-epv to main July 26, 2026 09:32
seonghobae and others added 2 commits July 26, 2026 18:43
Implement mlsirm_core::exposure::ci_classify: single-cut binary-response
classification by interim EAP (fixed 41-point [-4,4] grid, standard-normal
log prior) with SE = EAP posterior SD; interval theta_hat +/- z_crit * se
against theta_cut with STRICT first-crossing decisions and full
theta/se/lower/upper counterfactual traces.

Verified against R catIrt termCI.R/eapEst.R/catIrt.Rd at commit
c9e979e4812c27d95d367a7f097edfe8e93ac8eb (READ). Kingsbury & Weiss (1983),
Thompson (2007), and Eggen & Straetmans (2000) NOT method-section verified;
cited as historical/background only.

Tests: pinned 17-digit independent-Python oracle (all four traces at
1e-12), below/continue paths, full error paths, MC-500 #[ignore]
structural invariants (passed). Mutation kills executed: M1 swapped
decisions (2 tests fail), M2 point-estimate-vs-cut (2 fail), M3
variance-instead-of-SD (1 fail), M4 n_used off-by-one (2 fail); original
restored green. PyO3 py_ci_classify + Python wrapper with
validate-before-cast (complex rejection, exact 0/1 responses); pytest
TestCiClassify 5 passed.

Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com>
Impl-review doc-defect: the PyO3 docstring omitted Eggen & Straetmans
(2000) from the NOT-read historical citations required by the approved
citation-governance wording (Rust core, Python wrapper, and CHANGELOG
already carried it).

Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com>
Copilot AI review requested due to automatic review settings July 26, 2026 09:43
@seonghobae
seonghobae force-pushed the seonghobae-ci-classify branch from dc6f32e to d0a3573 Compare July 26, 2026 09:43

Copilot AI left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Pull request overview

This PR adds confidence-interval (ACI) classification as a new CAT stopping-rule path across the Rust core (mlsirm_core::exposure::ci_classify), the PyO3 extension (py_ci_classify), and a thin Python wrapper (fast_mlsirm.ci_classify), with pinned-oracle and validation tests to match the approved reduced-scope contract.

Changes:

  • Implement ci_classify in Rust, returning decision + n_used and full theta/se/lower/upper traces.
  • Expose ci_classify through the PyO3 module and a Python wrapper with pre-cast input validation.
  • Add Rust + Python tests (pinned oracle, below/continue paths, error paths, and MC invariants) and document in CHANGELOG.md.

Reviewed changes

Copilot reviewed 7 out of 7 changed files in this pull request and generated 2 comments.

Show a summary per file
File Description
tests/unit/exposure_tests.rs Adds Rust unit tests for ci_classify (oracle + invariants + error paths).
tests/test_paper_features.py Adds Python wrapper tests ensuring the binding/wrapper match pinned outputs and validate inputs.
python/fast_mlsirm/exposure.py Adds the public Python ci_classify wrapper calling _core.py_ci_classify.
python/fast_mlsirm/init.py Re-exports ci_classify from the package top-level.
crates/mlsirm-core/src/exposure.rs Implements the core ci_classify algorithm and result struct.
crates/fast-mlsirm-py/src/lib.rs Adds the py_ci_classify PyO3 binding and registers it in the module.
CHANGELOG.md Documents the new feature, contract summary, and citation governance notes.

💡 Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.

Comment thread crates/mlsirm-core/src/exposure.rs
Comment thread tests/unit/exposure_tests.rs

@coderabbitai coderabbitai Bot left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Actionable comments posted: 1

🧹 Nitpick comments (1)
python/fast_mlsirm/exposure.py (1)

774-798: 📐 Maintainability & Code Quality | 🔵 Trivial | ⚡ Quick win

sprt_classify와 거의 동일한 입력 검증 로직 중복.

복소수 거부 → ndim 체크 → c 기본값 처리 → responses 0/1 캐스팅으로 이어지는 이 블록은 바로 위 sprt_classify(L676-700)와 거의 한 글자씩 동일합니다. 공통 헬퍼(예: _validate_binary_response_inputs(a, b, c, responses))로 추출하면 향후 검증 로직 수정 시 두 곳을 동기화할 필요가 없어집니다.

🤖 Prompt for AI Agents
Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.

In `@python/fast_mlsirm/exposure.py` around lines 774 - 798, Extract the
duplicated validation flow from sprt_classify and the shown function into a
shared helper such as _validate_binary_response_inputs(a, b, c, responses). Move
complex-value rejection, 1-D checks, c default handling, and binary response
validation/casting into that helper, then have both callers reuse its normalized
return values without changing existing validation behavior.
🤖 Prompt for all review comments with AI agents
Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.

Inline comments:
In `@python/fast_mlsirm/exposure.py`:
- Around line 747-771: Update the ci_classify docstring’s References list to add
the formal Nydick (2014) citation for the READ R catIrt sources identified in
the Source status paragraph, matching the existing references in the
corresponding Rust and PyO3 documentation. Keep the historical NOT-read
citations unchanged.

---

Nitpick comments:
In `@python/fast_mlsirm/exposure.py`:
- Around line 774-798: Extract the duplicated validation flow from sprt_classify
and the shown function into a shared helper such as
_validate_binary_response_inputs(a, b, c, responses). Move complex-value
rejection, 1-D checks, c default handling, and binary response
validation/casting into that helper, then have both callers reuse its normalized
return values without changing existing validation behavior.
🪄 Autofix (Beta)

Fix all unresolved CodeRabbit comments on this PR:

  • Push a commit to this branch (recommended)
  • Create a new PR with the fixes

ℹ️ Review info
⚙️ Run configuration

Configuration used: Organization UI

Review profile: CHILL

Plan: Pro Plus

Run ID: 19ae477c-efc0-470c-92de-f99cfe75a005

📥 Commits

Reviewing files that changed from the base of the PR and between dfb57d0 and d0a3573.

📒 Files selected for processing (7)
  • CHANGELOG.md
  • crates/fast-mlsirm-py/src/lib.rs
  • crates/mlsirm-core/src/exposure.rs
  • python/fast_mlsirm/__init__.py
  • python/fast_mlsirm/exposure.py
  • tests/test_paper_features.py
  • tests/unit/exposure_tests.rs

Comment thread python/fast_mlsirm/exposure.py
@seonghobae
seonghobae merged commit 11e81fa into main Jul 26, 2026
34 checks passed
@seonghobae
seonghobae deleted the seonghobae-ci-classify branch July 26, 2026 09:50
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants