test(irt): add all-correct/all-incorrect item degeneracy robustness (0점/만점 item side) - #354
test(irt): add all-correct/all-incorrect item degeneracy robustness (0점/만점 item side)#354seonghobae wants to merge 1 commit into
Conversation
Item-side counterpart to the existing extreme-person (0점/만점) coverage: an item every person answers correctly, or every person answers incorrectly, drives the unregularized MLE item difficulty toward +/-inf (the 'separation' problem, analogous to complete separation in logistic regression). The regularized estimator (ridge penalty on b/theta/alpha) must keep every parameter finite. Verified against the built Rust core backend (backend=rust); full tests/test_irt_stability.py is green (9 passed). Grounding: perfect/zero response patterns yield infinite MLE ability estimates, and sparse/degenerate item columns yield overly extreme MLE item difficulty; weakly-informative priors / ridge regularization act as stabilizers that shrink these extremes (Bayesian IRT partial-pooling; item difficulty typically in [-3, 3]). Refs: ETS RM-20-06 Basic Concepts of IRT; Stan User's Guide 1.11 Item-Response Theory Models (regularizing priors). Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_016wtuYFp4E22QnEU1bFMhsr
|
Warning Review limit reached
Next review available in: 3 minutes Enable usage-based reviews in Billing to review now. Otherwise, wait until the next included review is available. How can I continue?After more reviews become available, a review can be triggered using the To avoid repeated limits, reduce automatic review volume by pausing incremental auto-reviews earlier, using label-based review opt-in, excluding WIP or generated PR titles, or requesting reviews manually when the PR is ready. If your team needs uninterrupted high-volume reviews, an organization admin can enable usage-based reviews. How do review limits work?CodeRabbit enforces per-developer PR review limits for each organization. Most developers receive the normal plan review availability. For paid Pro and Pro+ PR reviews, CodeRabbit uses adaptive limits for sustained high-volume activity. When a developer's recent PR review activity reaches the 95th percentile or higher among CodeRabbit users, additional reviews become available more gradually as earlier reviews age out of the rolling window. Please refer docs for additional details. Review details⚙️ Run configurationConfiguration used: Organization UI Review profile: CHILL Plan: Pro Plus Run ID: 📒 Files selected for processing (1)
Comment |
|
중복 정리로 닫습니다. 이 PR의 item-side 0점/만점(all-correct/all-incorrect item) 유한성 검증은 먼저 열린 #350의 Generated by Claude Code |
Summary
Adds one IRT robustness test to
tests/test_irt_stability.py:test_fit_stays_finite_with_all_correct_and_all_incorrect_items.The existing suite already covers the extreme-person (0점/만점) case — persons with all-correct
[1,1]or all-incorrect[0,0]rows (test_fit_handles_missing_by_design_axes_and_extreme_scores,test_hessian_...). This adds the item-side counterpart: an item every person answers correctly, and an item every person answers incorrectly.Why it matters
All-correct / all-incorrect item columns are the degenerate case for item-parameter estimation: the unregularized MLE item difficulty diverges to ±∞ (the separation problem, directly analogous to complete separation in logistic regression). The regularized estimator (ridge penalty on
b/theta/alpha) must keep every parameter finite. This test locks in that guarantee.Verification (executed, not asserted from reading)
pip install -e .[dev]);resolve_backend(FitConfig().backend) == "rust".pytest tests/test_irt_stability.py::test_fit_stays_finite_with_all_correct_and_all_incorrect_items→1 passed.pytest tests/test_irt_stability.py→9 passed.test_rust_parity.pystill green in the same run: 72 passed across stability+parity).Research grounding
Perfect/zero response patterns yield infinite MLE ability estimates, and sparse/degenerate item columns yield overly extreme MLE item difficulty; weakly-informative priors / ridge regularization act as stabilizers that shrink these extremes (Bayesian IRT partial-pooling; item difficulty conventionally in
[-3, 3]).Risk
Minimal — additive test only; no production code, formula, or contract touched.
Generated by Claude Code