Skip to content

test(irt): add all-correct/all-incorrect item degeneracy robustness (0점/만점 item side) - #354

Closed
seonghobae wants to merge 1 commit into
mainfrom
claude/contextualwisdomlab-audit-governance-fb7470
Closed

test(irt): add all-correct/all-incorrect item degeneracy robustness (0점/만점 item side)#354
seonghobae wants to merge 1 commit into
mainfrom
claude/contextualwisdomlab-audit-governance-fb7470

Conversation

@seonghobae

Copy link
Copy Markdown
Contributor

Summary

Adds one IRT robustness test to tests/test_irt_stability.py: test_fit_stays_finite_with_all_correct_and_all_incorrect_items.

The existing suite already covers the extreme-person (0점/만점) case — persons with all-correct [1,1] or all-incorrect [0,0] rows (test_fit_handles_missing_by_design_axes_and_extreme_scores, test_hessian_...). This adds the item-side counterpart: an item every person answers correctly, and an item every person answers incorrectly.

Why it matters

All-correct / all-incorrect item columns are the degenerate case for item-parameter estimation: the unregularized MLE item difficulty diverges to ±∞ (the separation problem, directly analogous to complete separation in logistic regression). The regularized estimator (ridge penalty on b/theta/alpha) must keep every parameter finite. This test locks in that guarantee.

Verification (executed, not asserted from reading)

  • Built the Rust core locally (pip install -e .[dev]); resolve_backend(FitConfig().backend) == "rust".
  • New test passes: pytest tests/test_irt_stability.py::test_fit_stays_finite_with_all_correct_and_all_incorrect_items1 passed.
  • Full file green: pytest tests/test_irt_stability.py9 passed.
  • No formula/estimator/gradient change — a pure test addition, so Rust↔NumPy parity is untouched (test_rust_parity.py still green in the same run: 72 passed across stability+parity).

Research grounding

Perfect/zero response patterns yield infinite MLE ability estimates, and sparse/degenerate item columns yield overly extreme MLE item difficulty; weakly-informative priors / ridge regularization act as stabilizers that shrink these extremes (Bayesian IRT partial-pooling; item difficulty conventionally in [-3, 3]).

Risk

Minimal — additive test only; no production code, formula, or contract touched.


Generated by Claude Code

Item-side counterpart to the existing extreme-person (0점/만점) coverage: an
item every person answers correctly, or every person answers incorrectly,
drives the unregularized MLE item difficulty toward +/-inf (the 'separation'
problem, analogous to complete separation in logistic regression). The
regularized estimator (ridge penalty on b/theta/alpha) must keep every
parameter finite. Verified against the built Rust core backend (backend=rust);
full tests/test_irt_stability.py is green (9 passed).

Grounding: perfect/zero response patterns yield infinite MLE ability estimates,
and sparse/degenerate item columns yield overly extreme MLE item difficulty;
weakly-informative priors / ridge regularization act as stabilizers that shrink
these extremes (Bayesian IRT partial-pooling; item difficulty typically in
[-3, 3]). Refs: ETS RM-20-06 Basic Concepts of IRT; Stan User's Guide 1.11
Item-Response Theory Models (regularizing priors).

Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_016wtuYFp4E22QnEU1bFMhsr
@coderabbitai

coderabbitai Bot commented Jul 29, 2026

Copy link
Copy Markdown
Contributor

Warning

Review limit reached

@seonghobae, you've reached your PR review limit, so we couldn't start this review.

Next review available in: 3 minutes

Enable usage-based reviews in Billing to review now. Otherwise, wait until the next included review is available.
You're only billed for reviews past your plan's rate limits ($0.25/file).

How can I continue?

After more reviews become available, a review can be triggered using the @coderabbitai review command as a PR comment. Alternatively, push new commits to this PR.

To avoid repeated limits, reduce automatic review volume by pausing incremental auto-reviews earlier, using label-based review opt-in, excluding WIP or generated PR titles, or requesting reviews manually when the PR is ready. If your team needs uninterrupted high-volume reviews, an organization admin can enable usage-based reviews.

How do review limits work?

CodeRabbit enforces per-developer PR review limits for each organization. Most developers receive the normal plan review availability.

For paid Pro and Pro+ PR reviews, CodeRabbit uses adaptive limits for sustained high-volume activity. When a developer's recent PR review activity reaches the 95th percentile or higher among CodeRabbit users, additional reviews become available more gradually as earlier reviews age out of the rolling window.

Please refer docs for additional details.

Review details
⚙️ Run configuration

Configuration used: Organization UI

Review profile: CHILL

Plan: Pro Plus

Run ID: c918cc3f-750d-46bf-8541-8d73bce8fa9b

📥 Commits

Reviewing files that changed from the base of the PR and between a3123a2 and 6d9cf7d.

📒 Files selected for processing (1)
  • tests/test_irt_stability.py

Comment @coderabbitai help to get the list of available commands.

Copy link
Copy Markdown
Contributor Author

중복 정리로 닫습니다. 이 PR의 item-side 0점/만점(all-correct/all-incorrect item) 유한성 검증은 먼저 열린 #350의 test_fit_is_finite_for_perfect_and_zero_score_persons_and_items가 이미 포함하는 범위입니다 — #350은 perfect/zero person 행렬과 item 행렬을 모두 적합하고 파라미터 전 블록·item/person-fit 진단의 유한성까지 검증하며, 같은 tests/test_irt_stability.py를 대상으로 하므로 병합 충돌도 확정적입니다. #350을 canonical로 유지합니다. (정리 관례 #156#159, #351/#353과 동일)


Generated by Claude Code

@seonghobae seonghobae closed this Jul 29, 2026
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants