feat: Owen-approximate posterior-predictive EPV + Wald SPRT classification (reduced scope) - #286
Merged
Conversation
…er Linden, 1998; reduced scope) Adds epv_select to mlsirm_core::exposure: for each unadministered item, the predictive success probability p* = c + (1-c) Phi((mu-b)/sqrt(1/a^2+sig2)) weights the two Owen-update outcome posterior variances, EPV = p* sig2+ + (1-p*) sig2-, and the argmin (lowest-index ties) is selected. Explicitly NOT vdL-98 exact MEPV (which uses point probabilities and exact numerical posteriors); the reduced scope is documented in code, docstring, and CHANGELOG per the adversarial spec review. PyO3 binding py_epv_select + Python wrapper fast_mlsirm.epv_select (arrays validated before lossy casts). Tests: pinned crate-parity oracles (5-item pool, 17-digit spec-review values, 5e-7 abs), delegation discriminator (argmin EPV=2 vs max-info=4 vs b-matching=1), administered-masking invariance, error paths, and a 500-rep Monte-Carlo invariant suite (#[ignore]). Mutation kills executed: argmin->argmax (3 fail), drop guessing floor (2 fail), swap sig2+/- weights (2 fail), posterior-predictive->point probability (2 fail). Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com>
…citation Adversarial impl-review findings: (1) complex inputs were laundered through the float64 cast, silently discarding imaginary parts — now rejected with ValueError before any cast, regression-tested; (2) the ERIC ED424235 research-report citation said 98-01 but the read source is Research Report 96-01. Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com>
Contributor
📝 WalkthroughWalkthroughOwen 근사 posterior-predictive EPV 기반 CAT 아이템 선택이 Rust 코어에 구현되고 PyO3 및 Python API로 공개되었습니다. 입력 검증, administered 마스킹, 고정 오라클, 오류 경로와 Monte Carlo 테스트 및 변경 로그가 추가되었습니다. ChangesEPV 아이템 선택
Estimated code review effort: 3 (Moderate) | ~20 minutes Sequence Diagram(s)sequenceDiagram
participant PythonCaller
participant py_epv_select
participant mlsirm_core_epv_select
participant owen_update
PythonCaller->>py_epv_select: 입력 배열과 사전분포 전달
py_epv_select->>mlsirm_core_epv_select: contiguous 변환값 전달
mlsirm_core_epv_select->>owen_update: 정답 및 오답 상태 업데이트
owen_update-->>mlsirm_core_epv_select: 분산 성분 반환
mlsirm_core_epv_select-->>py_epv_select: 선택 인덱스와 점수 배열 반환
py_epv_select-->>PythonCaller: Python dict 반환
Suggested reviewers: 🚥 Pre-merge checks | ✅ 4 | ❌ 1❌ Failed checks (1 warning)
✅ Passed checks (4 passed)
✨ Finishing Touches📝 Generate docstrings
🧪 Generate unit tests (beta)
Comment |
There was a problem hiding this comment.
Pull request overview
Note
Copilot was unable to run its full agentic suite in this review.
Adds an Owen-approximate posterior-predictive EPV (expected posterior variance) item selection criterion to the Rust core and exposes it via the Python API, with pinned-oracle parity tests.
Changes:
- Implement
mlsirm_core::exposure::epv_selectand return full-lengthepvandpredictivevectors plus selected index. - Add PyO3 binding (
py_epv_select) and a Python wrapper (fast_mlsirm.epv_select) with input validation. - Add Rust and Python tests including pinned numeric oracles, masking behavior, and error-path checks; document the new feature in the changelog.
Reviewed changes
Copilot reviewed 7 out of 7 changed files in this pull request and generated 1 comment.
Show a summary per file
| File | Description |
|---|---|
| tests/unit/exposure_tests.rs | Adds Rust unit tests for epv_select (pinned-oracle, masking, error paths, MC invariants). |
| tests/test_paper_features.py | Adds Python parity tests for fast_mlsirm.epv_select mirroring Rust oracles and validation. |
| python/fast_mlsirm/exposure.py | Adds Python epv_select wrapper with dtype/shape checks and calls into Rust binding. |
| python/fast_mlsirm/init.py | Re-exports epv_select from the public Python package API. |
| crates/mlsirm-core/src/exposure.rs | Implements core EPV selection logic and result type in Rust. |
| crates/fast-mlsirm-py/src/lib.rs | Exposes py_epv_select via PyO3 and wires it into the module init. |
| CHANGELOG.md | Documents the new EPV selection feature and its intended scope/approximation. |
💡 Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.
…ope) (#256) * feat: Wald SPRT classification for CAT (single-cut binary, reduced scope) Single-cut binary-response sequential probability ratio test for IRT classification testing: point hypotheses at theta_cut -/+ delta, cumulative binary log-likelihood ratio under the D=1 logistic 3PL, inclusive first-crossing decisions against log Wald boundaries A=ln((1-beta)/alpha), B=ln(beta/(1-alpha)) -> above/below/continue with 1-based n_used and a full counterfactual llr_trace diagnostic. Verified against READ sources: R catIrt termSPRT.R/logLik.brm.R/p.brm.R and Thompson (2007, doi:10.7275/fq3r-zz60, p. 7). Reckase (1983) and Eggen (1999) cited as historical origins via Thompson (NOT directly read). Adversarial spec review verdict: REDUCED-SCOPE (no multi-cut, polytomous, or D=1.7 compatibility). Tests: pinned 17-digit interior-crossing oracle (also the final-LLR-only discriminator), below-decision first-crossing invariants on a heterogeneous pool, continue path, full error-path coverage, 500-rep Monte-Carlo structural invariants (#[ignore]). Executed mutation kills: swapped boundaries (2 fail), dropped guessing floor (1), collapsed null hypothesis theta0=theta_cut (2), off-by-one n_used (1); restore verified green. Python wrapper validates before casts (complex laundering, exact 0/1 responses) and returns crate outputs only. Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com> * fix(sprt): stable log-space LLR at saturating parameters; mark Wald (1947) NOT read Adversarial impl review found that extreme-but-valid parameters (e.g. a = 50, delta = 20 -> z = +/-1000) saturate the 2PL response probability to numerical 0/1, making ln(p1/p0) or ln((1-p1)/(1-p0)) non-finite and returning a spurious Err. Rewrite the increments in stable log space: ln(1-P) = ln(1-c) - softplus(z) (the ln(1-c) terms cancel in the wrong- response increment) and ln(P) = -softplus(-z) when c = 0 (for c > 0 the direct form is bounded below by c). Regression test pins the finite +/-1000 LLRs and the ln-bounded c > 0 case, reading crate outputs. Also mark Wald (1947) as NOT read in the Python docstring, PyO3 docstring, and CHANGELOG (it already was in the core module comment). Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com> --------- Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com>
![coderabbitai[bot]](https://avatars.githubusercontent.com/in/347564?s=60&v=4){kind=small}
Contributor
There was a problem hiding this comment.
Actionable comments posted: 2
🤖 Prompt for all review comments with AI agents
Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.
Inline comments:
In `@crates/mlsirm-core/src/exposure.rs`:
- Around line 1332-1387: Update epv_select so owen_update failures for
administered items do not abort the entire scoring operation, since those items
cannot be selected. Preserve error propagation for unadministered items, and
represent administered-item failures with the existing appropriate NaN or
sentinel behavior while keeping their exclusion from selection intact.
In `@python/fast_mlsirm/exposure.py`:
- Around line 556-589: Update the reference number in the docstrings for
owen_update and owen_cat from Research Report 98-01 to Research Report 96-01,
matching the existing epv_select documentation. Leave the surrounding reference
text unchanged.
🪄 Autofix (Beta)
Fix all unresolved CodeRabbit comments on this PR:
- Push a commit to this branch (recommended)
- Create a new PR with the fixes
ℹ️ Review info
⚙️ Run configuration
Configuration used: Organization UI
Review profile: CHILL
Plan: Pro Plus
Run ID: 1b188b4e-d74c-4800-98c5-6c5ff9cdb053
📒 Files selected for processing (7)
CHANGELOG.mdcrates/fast-mlsirm-py/src/lib.rscrates/mlsirm-core/src/exposure.rspython/fast_mlsirm/__init__.pypython/fast_mlsirm/exposure.pytests/test_paper_features.pytests/unit/exposure_tests.rs
Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com>
seonghobae
changed the title
There was a problem hiding this comment.
Pull request overview
Copilot reviewed 7 out of 7 changed files in this pull request and generated 1 comment.
Comments suppressed due to low confidence (1)
CHANGELOG.md:129
- The PR description/title focus on EPV selection, but this PR also introduces the new public API
fast_mlsirm.sprt_classify(and documents it here). Please update the PR title/description (or explicitly scope/justify the SPRT addition) so the stated PR purpose matches the actual changes.
- **Wald SPRT classification for CAT** (`fast_mlsirm.sprt_classify`; in
`mlsirm_core::exposure`). Single-cut binary-response sequential probability
ratio test: point hypotheses at `theta_cut -/+ delta`, cumulative binary
log-likelihood ratio under the D=1 logistic 3PL, and inclusive
first-crossing decisions against the log Wald boundaries
`A = ln((1-beta)/alpha)`, `B = ln(beta/(1-alpha))` -> `"above"`/`"below"`/
`"continue"` with 1-based `n_used`; the full `llr_trace` is returned as an
offline diagnostic (entries past `n_used` are counterfactual replay
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Scope note
Iteration 24 of the autonomous paper-implementation loop. Stacked on #254 (seonghobae-ccat). ## What Owen-approximate posterior-predictive EPV item selection —
fast_mlsirm.epv_select, Rust coremlsirm_core::exposure::epv_select(+ PyO3py_epv_select). For each unadministered item under the Owen normal posteriorN(mu, sig2): - predictive success probabilityp*_i = c_i + (1 - c_i) Phi((mu - b_i) / sqrt(1/a_i^2 + sig2))(identicaldtoowen_update) -EPV_i = p*_i sig2_i^+ + (1 - p*_i) sig2_i^-with both outcome variances fromowen_update- select argmin over unadministered items; lowest-index ties;epv/predictivereturned for the whole pool. ## Adversarial spec-verify (BEFORE implementing): REDUCED-SCOPE The reviewer confirmed the posterior-predictive formula is mathematically correct but ruled that van der Linden (1998), catREPV.R, and mirtCAT define MEPV with point probabilitiesP_i(mu)and exact numerical posterior variances, not posterior-predictive weights + Owen updates. Accepted contract: an explicitly labeled "Owen-approximate posterior-predictive EPV" with prescribed doc wording (used verbatim in code, docstring, and CHANGELOG). This is deliberately NOT exact vdL-98 MEPV. Source status: van der Linden (1998) READ as ERIC ED424235 (Research Report 96-01); catREPV.Rand mirtCATselection_criteria.RREAD; Owen (1975) and the Psychometrika body NOT read (documented in code). ## Evidence - Pinned crate-parity oracles (spec-review exact arithmetic, 17 digits, 5e-7 abs tol) for bothpredictiveandepvover a 5-item pool; argmin = item 1. - Delegation discriminator (spec-review pool): argmin EPV = 2 while max-Fisher-info = 4 and Owen b-matching = 1 — a delegating mutant cannot satisfy all three. - Administered masking: scoring covers the whole pool; masking affects selection only (runner-up promotion pinned, vectors unchanged to 1e-15). - Error paths: empty pool, length mismatch, nonpositivea, non-finiteb/c/mu,cout of [0,1),sig2 <= 0, all-administered, degenerate Owen posterior propagation. - MC-500#[ignore]invariant suite: predictive finite in[c_i, 1](closed — erfc saturation), EPV finite > 0, selected always unadministered, ties lowest-index. (StrictEPV < sig2deliberately NOT asserted: single-outcome Owen variance can genuinely increase and roundoff inflation of 8.9e-16 was observed.) - 4 executed mutation kills (temp-file backup/restore, re-run verified green): - M1 argmin -> argmax: 3 tests fail - M2 drop guessing floor (p* = Phi(d)): 2 fail - M3 swapsig2+/sig2-weights: 2 fail - M5 posterior-predictive -> point probabilityP_i(mu): 2 fail - Python wrapper validates arrays BEFORE lossy casts; 4 pytest parity tests green.cargo test -p mlsirm-core --lib -- epv: 4 passed (+1 ignored MC-500 passed with--ignored).pytest -k EpvSelect: 4 passed. ## References - van der Linden, W. J. (1998). Bayesian item selection criteria for adaptive testing. Psychometrika, 63(2), 201-216. https://doi.org/10.1007/BF02294775 - Owen, R. J. (1975). A Bayesian sequential procedure for quantal response in the context of adaptive mental testing. JASA, 70(350), 351-356. - Magis, D., & Raiche, G. (2012). catR. JSS, 48(8). https://doi.org/10.18637/jss.v048.i08Adversarial impl-review (AFTER implementing): CLEAN after fixes (b2a711a)
a/b/cwere laundered through the float64 cast (imaginary parts silently dropped). Fixed:np.iscomplexobjrejection before any cast + 3 regression error-path tests.Summary by CodeRabbit
새 기능
버그 수정