Skip to content

Add fleiss_kappa: Fleiss' multi-rater kappa with exact (Conger) variant - #308

Merged
seonghobae merged 2 commits into
seonghobae-brattfrom
seonghobae-fleiss
Jul 31, 2026
Merged

Add fleiss_kappa: Fleiss' multi-rater kappa with exact (Conger) variant#308
seonghobae merged 2 commits into
seonghobae-brattfrom
seonghobae-fleiss

Conversation

@seonghobae

@seonghobae seonghobae commented Jul 27, 2026

Copy link
Copy Markdown
Contributor

Implements Fleiss' multi-rater kappa for nominal agreement, with the exact (Conger) chance-agreement variant ? iteration 67 of the autonomous paper-implementation loop. Stacked on #307.

Source basis

  • READ and normative: CRAN irr 0.85, R/kappam.fleiss.R (full 113-line source) + man/kappam.fleiss.Rd. Algorithm source of truth; every formula verified line-by-line against it.
  • NOT READ (cited as origins only): Fleiss (1971), Psychological Bulletin 76(5), 378-382; Conger (1980), Psychological Bulletin 88(2), 322-328.

What's in

  • Rust core fleiss_kappa + FleissKappaResult in mlsirm_core::agreement: agreeP with the ?nr centering, classic chance ?p?, exact chance ?p? ? (1/nr)?s?? (sample variance over per-rater category proportions; algebraic reduction of R's sum(apply(rtab,2,var)*(nr-1)/nr)/(nr-1) verified), Fleiss' large-sample z/p (classic), category-wise kappas with NaN for empty categories (R's 0/0 preserved), listwise row drop for missing.
  • PyO3 binding (plain name) + NumPy wrapper fleiss_kappa / FleissKappaResult dataclass in fast_mlsirm.validation (NaN = missing, k inferred or explicit, hardened input validation).
  • Documented API deviations from R: index codes 0..k-1 (negative = missing, so negative numeric labels must be remapped), error on degenerate 1 ? chanceP = 0 (R returns NaN), size caps, always-computed classic detail, exact mode returns NaN z/p + empty detail (R returns neither).

Evidence

  • Adversarial spec-verify: APPROVED (independent exact-rational recomputation of FK1/FK2/FK4 from the R source).
  • Exact-Fraction oracle anchors FK1?FK5, executed: classic ? = 139/399 (var 181/10830, z = 2.6947398540854888), exact ? = 37/102 (?s?? = 3/50), missing-drop reproduces FK1 bitwise with subjects_used = 5, category ? = [1/21, 31/91, 43/63] (var? = 1/30), empty category NaN.
  • 6 mutants EXECUTED and killed: MU1 agreeP drops ?nr; MU2 chanceP from row sums (1/5 ? 67/200 on the 5?4 fixture); MU3 exact ? classic; MU4 variance second-term sign; MU5 missing-as-category; MU6 pjk drops m?nr?p? centering.
  • MC-500 #[ignore] permutation-invariance test (subject rows + rater columns, classic and exact, category detail), executed and passing. Disclosure: invariance cannot detect wrong-but-symmetric formulas; the FK value pins are the discriminating anchors. p-value pins use abs 5e-7 (crate erfc is the Numerical Recipes approximation, |err| < 1.2e-7).
  • Suites: cargo mlsirm-core --lib 847 passed; pytest test_paper_features.py 336 passed.

Adversarial impl-review outcome

  • Round 1: FINDINGS(2) -- MAJOR: uint64 > i64::MAX wrapped negative via astype(int64) and rows were silently dropped as missing; MINOR: explicit k lossy-coerced from 3.9/"3"/bool. Both fixed in 11bc1ed (unsigned-range guard before conversion; strict integer k check).
  • Round 2: CLEAN -- fixes verified by execution (reproductions now raise ValueError; uint8/16/32/64 covered; k=None/int/np.integer paths intact; regression tests read wrapper raise paths).

Reimplements CRAN irr 0.85 kappam.fleiss() (R source READ and normative;
Fleiss 1971 and Conger 1980 NOT READ, cited as model origins only) in the
Rust core (mlsirm_core::agreement::fleiss_kappa) with a thin PyO3 binding
and NumPy wrapper:

- classification-table agreement, classic (sum p_j^2) and exact
  (sum p_j^2 - (1/nr) sum s2_j) chance agreement; kappa, Fleiss' z test,
  and category-wise kappas (classic mode; NaN for empty categories,
  matching R's 0/0)
- listwise row drop for missing ratings (negative code / NaN)
- documented API deviations: index codes 0..k-1 with explicit/inferred k,
  error on degenerate 1 - chanceP = 0 (R returns NaN), size caps

Evidence: exact-Fraction oracle anchors FK1-FK5 (classic kappa 139/399,
exact 37/102, category kappas [1/21, 31/91, 43/63]); 6 mutants EXECUTED
and killed (agreeP centering, row-vs-column chance sums, exact==classic,
variance sign, missing-as-category, pjk centering); MC-500 subject/rater
permutation-invariance test (#[ignore], executed); cargo 847 pass;
TestFleiss pytest pass.

Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com>
@coderabbitai

coderabbitai Bot commented Jul 27, 2026

Copy link
Copy Markdown
Contributor

Important

Review skipped

Auto reviews are disabled on base/target branches other than the default branch.

Please check the settings in the CodeRabbit UI or the .coderabbit.yaml file in this repository. To trigger a single review, invoke the @coderabbitai review command.

⚙️ Run configuration

Configuration used: Organization UI

Review profile: CHILL

Plan: Pro Plus

Run ID: 5cc27061-5015-46e6-ab0d-a86a650a7ee3

You can disable this status message by setting the reviews.review_status to false in the CodeRabbit configuration file.

Use the checkbox below for a quick retry:

  • 🔍 Trigger review
✨ Finishing Touches
🧪 Generate unit tests (beta)
  • Create PR with unit tests
  • Commit unit tests in branch seonghobae-fleiss

Comment @coderabbitai help to get the list of available commands.

Impl-review round 1 findings:
- MAJOR: uint64 values above i64::MAX wrapped negative via astype(int64)
  and were silently dropped as missing; now rejected before conversion.
- MINOR: explicit k accepted lossy coercions (3.9, '3', bool); now
  requires a true integer (int or np.integer, bool excluded).

Regression tests added to TestFleiss.test_validation.

Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com>
@seonghobae
seonghobae merged commit 607ad56 into seonghobae-bratt Jul 31, 2026
6 checks passed
@seonghobae
seonghobae deleted the seonghobae-fleiss branch July 31, 2026 12:38
@seonghobae

Copy link
Copy Markdown
Contributor Author

Note: this PR was squash-merged into a non-main stacked base, so its commits did not land on main via this PR. The feature set reached main via #374 (stack tip integration) after #290.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant