Add icc intraclass correlation coefficients (Shrout-Fleiss taxonomy, irr 0.85) - #309
Closed
seonghobae wants to merge 9 commits into
Closed
Add icc intraclass correlation coefficients (Shrout-Fleiss taxonomy, irr 0.85)#309seonghobae wants to merge 9 commits into
seonghobae wants to merge 9 commits into
Conversation
Reimplements CRAN irr 0.85 kappam.fleiss() (R source READ and normative; Fleiss 1971 and Conger 1980 NOT READ, cited as model origins only) in the Rust core (mlsirm_core::agreement::fleiss_kappa) with a thin PyO3 binding and NumPy wrapper: - classification-table agreement, classic (sum p_j^2) and exact (sum p_j^2 - (1/nr) sum s2_j) chance agreement; kappa, Fleiss' z test, and category-wise kappas (classic mode; NaN for empty categories, matching R's 0/0) - listwise row drop for missing ratings (negative code / NaN) - documented API deviations: index codes 0..k-1 with explicit/inferred k, error on degenerate 1 - chanceP = 0 (R returns NaN), size caps Evidence: exact-Fraction oracle anchors FK1-FK5 (classic kappa 139/399, exact 37/102, category kappas [1/21, 31/91, 43/63]); 6 mutants EXECUTED and killed (agreeP centering, row-vs-column chance sums, exact==classic, variance sign, missing-as-category, pjk centering); MC-500 subject/rater permutation-invariance test (#[ignore], executed); cargo 847 pass; TestFleiss pytest pass. Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com>
Impl-review round 1 findings: - MAJOR: uint64 values above i64::MAX wrapped negative via astype(int64) and were silently dropped as missing; now rejected before conversion. - MINOR: explicit k accepted lossy coercions (3.9, '3', bool); now requires a true integer (int or np.integer, bool excluded). Regression tests added to TestFleiss.test_validation. Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com>
Transcribed from CRAN irr 0.85 R/icc.R (READ, normative source; Shrout & Fleiss 1979, McGraw & Wong 1996, Bartko 1966 NOT READ, cited as origins only). Rust core computes all six variants (oneway/twoway x consistency/agreement x single/average) from one-pass ANOVA mean squares, the F test of H0: icc = r0 (two-way agreement via Satterthwaite df, preserving the R quirk that both units' CI bounds reuse the nr-scaled plug-in form, icc.R lines 139-141), and unclamped confidence bounds. Listwise NaN row drop; Inf rejected; degenerate zero-variance / icc=1 pivots error instead of leaking non-finite output. Evidence: exact-Fraction oracle on Shrout-Fleiss Table 2 (all six coefficients + scipy F CI pins), 6 EXECUTED mutation kills (MSw divisor, quantile df order, agreement denominator, r0-in-F, dimension map, CI plug-in), MC-500 permutation invariance + Spearman-Brown single/average bridge for all three families. cargo 853 pass; pytest paper suite 340 pass. Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com>
Contributor
|
Important Review skippedAuto reviews are disabled on base/target branches other than the default branch. Please check the settings in the CodeRabbit UI or the ⚙️ Run configurationConfiguration used: Organization UI Review profile: CHILL Plan: Pro Plus Run ID: You can disable this status message by setting the Use the checkbox below for a quick retry:
✨ Finishing Touches🧪 Generate unit tests (beta)
Comment |
Impl-review finding (MINOR): object arrays of Python/numpy bools were silently coerced to 0.0/1.0 before the bool-dtype check, bypassing the boolean rejection contract. Scan object arrays for bool elements before the float64 conversion; regression test added. Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com>
Round-2 impl-review finding (MINOR): an object array whose element is a 0-D np.ndarray of dtype bool coerced to 1.0 and bypassed the boolean rejection. The object-array scan now also rejects ndarray elements with boolean dtype; regression test added. Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com>
Round-3 impl-review finding (MINOR): 0-D object-dtype ndarrays wrapping a bool still bypassed the scan. The guard now iteratively unwraps 0-D ndarray elements of any dtype via .item() before the bool check, closing the wrapper-nesting family of bypasses at the root; regression tests for object-wrapped bool and np.bool_ added. 0-D float ndarray elements remain accepted and bitwise-match the plain-float result. Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com>
Replace the boolean blacklist with a numeric whitelist: after unwrapping 0-D ndarrays, only int/float/np.integer/np.floating scalars are accepted (bool excluded as an int subclass). This closes the np.void structured- scalar bypass and any future exotic-scalar coercion path. Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com>
Five review rounds produced successive bypasses of per-element vetting (object bools, nested 0-D wrappers, np.void, timedelta64, self- referential 0-D arrays hanging the unwrap loop, __float__-lying int/float subclasses). Numeric data never requires object dtype, so the wrapper now rejects it categorically, eliminating the entire coercion attack class. Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com>
…ce) (#311) * Add kripp_alpha Krippendorff's alpha for inter-rater agreement Rust reimplementation of CRAN irr 0.85 kripp.alpha() (R/kripp.alpha.R, READ and normative; Krippendorff 1980 NOT READ, cited as method origin only). Coincidence matrix over unordered rater pairs with the irr divisor quirk preserved verbatim (mc = #nonmissing-1 per column only when any value is missing, else 1), all four metrics (nominal, ordinal half-endpoint weights, interval, ratio), alpha = 1 when fewer than two observed levels. Documented deviations: all-missing, infinities, and ratio zero-sum level pairs are explicit errors. Evidence: exact-Fraction oracle anchors K1-K4 (nominal 113/152, ordinal 108577/133160, interval 951/1120, ratio 18222619/22852465, nmv=40; no-NA quirk pin 43/72 vs m-1 mutant's 11/18), 6-mutant EXECUTED kill map, MC-500 permutation invariance. cargo 858 pass, pytest paper suite 344 pass. Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com> * Reject integer rating levels beyond 2**53 in kripp_alpha wrapper Review finding (MAJOR): int64/uint64 rating labels beyond 2**53 are not exactly representable as float64, so distinct levels silently collapsed during the cast — complete disagreement returned alpha=1 with a single level. The wrapper now rejects any integer array with values outside [-2**53, 2**53]; the boundary itself remains accepted (exactly representable). Regression test executes the reviewer's repro and was red-green verified against the unguarded version. Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com> --------- Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com>
3 tasks
Contributor
Author
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Implements
icc— intraclass correlation coefficients for inter-rater reliability, the complete Shrout-Fleiss taxonomy, transcribed from CRAN irr 0.85R/icc.R(READ, normative source).Stacked on #308 (seonghobae-fleiss).
What
icc()+IccResultincrates/mlsirm-core/src/reliability.rs: all six variants (modeloneway/twoway ×typeconsistency/agreement ×unitsingle/average) from one-pass ANOVA mean squares (MSr, MSw, MSc, MSe; sample-variance divisor n−1), the F test of H0: icc = r0, and unclamped confidence bounds. Reuses the existing privatef_cdf/f_quantile.na.omit); infinities rejected; degenerate zero-variance / icc=1 pivots return explicit errors instead of leaking non-finite output.icc) + thin Python wrapper withIccResultdataclass (masked/complex/object/bool rejected before casts).Sources
R/icc.R— READ line by line, normative.Evidence
#[ignore]): subject/rater permutation invariance + Spearman-Brown single↔average bridge for all three families, all values read from crate outputs.Adversarial implementation review (rounds 1-6)
Six review rounds executed against the wrapper and Rust core. Rounds 1-5
each found an object-dtype coercion bypass (object bools, nested 0-D
ndarray wrappers, 0-D object wrappers, np.void structured scalars, then
timedelta64 / self-referential 0-D arrays / float-lying subclasses).
Fixes 88c56c0, 21a544c, c51ad5f, b9e12f5 hardened per-element vetting;
9ebd18f resolved the class root-cause by rejecting object-dtype arrays
outright (numeric data never requires object dtype). Round 6: CLEAN ? no
remaining input bypass or R-semantics defect confirmed.