Add fleiss_kappa: Fleiss' multi-rater kappa with exact (Conger) variant - #308
Merged
Conversation
Reimplements CRAN irr 0.85 kappam.fleiss() (R source READ and normative; Fleiss 1971 and Conger 1980 NOT READ, cited as model origins only) in the Rust core (mlsirm_core::agreement::fleiss_kappa) with a thin PyO3 binding and NumPy wrapper: - classification-table agreement, classic (sum p_j^2) and exact (sum p_j^2 - (1/nr) sum s2_j) chance agreement; kappa, Fleiss' z test, and category-wise kappas (classic mode; NaN for empty categories, matching R's 0/0) - listwise row drop for missing ratings (negative code / NaN) - documented API deviations: index codes 0..k-1 with explicit/inferred k, error on degenerate 1 - chanceP = 0 (R returns NaN), size caps Evidence: exact-Fraction oracle anchors FK1-FK5 (classic kappa 139/399, exact 37/102, category kappas [1/21, 31/91, 43/63]); 6 mutants EXECUTED and killed (agreeP centering, row-vs-column chance sums, exact==classic, variance sign, missing-as-category, pjk centering); MC-500 subject/rater permutation-invariance test (#[ignore], executed); cargo 847 pass; TestFleiss pytest pass. Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com>
Contributor
|
Important Review skippedAuto reviews are disabled on base/target branches other than the default branch. Please check the settings in the CodeRabbit UI or the ⚙️ Run configurationConfiguration used: Organization UI Review profile: CHILL Plan: Pro Plus Run ID: You can disable this status message by setting the Use the checkbox below for a quick retry:
✨ Finishing Touches🧪 Generate unit tests (beta)
Comment |
Impl-review round 1 findings: - MAJOR: uint64 values above i64::MAX wrapped negative via astype(int64) and were silently dropped as missing; now rejected before conversion. - MINOR: explicit k accepted lossy coercions (3.9, '3', bool); now requires a true integer (int or np.integer, bool excluded). Regression tests added to TestFleiss.test_validation. Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com>
3 tasks
Contributor
Author
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Implements Fleiss' multi-rater kappa for nominal agreement, with the exact (Conger) chance-agreement variant ? iteration 67 of the autonomous paper-implementation loop. Stacked on #307.
Source basis
R/kappam.fleiss.R(full 113-line source) +man/kappam.fleiss.Rd. Algorithm source of truth; every formula verified line-by-line against it.What's in
fleiss_kappa+FleissKappaResultinmlsirm_core::agreement: agreeP with the ?nr centering, classic chance ?p?, exact chance ?p? ? (1/nr)?s?? (sample variance over per-rater category proportions; algebraic reduction of R'ssum(apply(rtab,2,var)*(nr-1)/nr)/(nr-1)verified), Fleiss' large-sample z/p (classic), category-wise kappas with NaN for empty categories (R's 0/0 preserved), listwise row drop for missing.fleiss_kappa/FleissKappaResultdataclass infast_mlsirm.validation(NaN = missing, k inferred or explicit, hardened input validation).0..k-1(negative = missing, so negative numeric labels must be remapped), error on degenerate1 ? chanceP = 0(R returns NaN), size caps, always-computed classic detail, exact mode returns NaN z/p + empty detail (R returns neither).Evidence
#[ignore]permutation-invariance test (subject rows + rater columns, classic and exact, category detail), executed and passing. Disclosure: invariance cannot detect wrong-but-symmetric formulas; the FK value pins are the discriminating anchors. p-value pins use abs 5e-7 (crate erfc is the Numerical Recipes approximation, |err| < 1.2e-7).mlsirm-core --lib847 passed; pytesttest_paper_features.py336 passed.Adversarial impl-review outcome