Add kripp_alpha Krippendorff's alpha (irr 0.85 kripp.alpha, READ source) - #311
Merged
Conversation
Rust reimplementation of CRAN irr 0.85 kripp.alpha() (R/kripp.alpha.R, READ and normative; Krippendorff 1980 NOT READ, cited as method origin only). Coincidence matrix over unordered rater pairs with the irr divisor quirk preserved verbatim (mc = #nonmissing-1 per column only when any value is missing, else 1), all four metrics (nominal, ordinal half-endpoint weights, interval, ratio), alpha = 1 when fewer than two observed levels. Documented deviations: all-missing, infinities, and ratio zero-sum level pairs are explicit errors. Evidence: exact-Fraction oracle anchors K1-K4 (nominal 113/152, ordinal 108577/133160, interval 951/1120, ratio 18222619/22852465, nmv=40; no-NA quirk pin 43/72 vs m-1 mutant's 11/18), 6-mutant EXECUTED kill map, MC-500 permutation invariance. cargo 858 pass, pytest paper suite 344 pass. Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com>
Contributor
|
Important Review skippedAuto reviews are disabled on base/target branches other than the default branch. Please check the settings in the CodeRabbit UI or the ⚙️ Run configurationConfiguration used: Organization UI Review profile: CHILL Plan: Pro Plus Run ID: You can disable this status message by setting the Use the checkbox below for a quick retry:
✨ Finishing Touches🧪 Generate unit tests (beta)
Comment |
Review finding (MAJOR): int64/uint64 rating labels beyond 2**53 are not exactly representable as float64, so distinct levels silently collapsed during the cast — complete disagreement returned alpha=1 with a single level. The wrapper now rejects any integer array with values outside [-2**53, 2**53]; the boundary itself remains accepted (exactly representable). Regression test executes the reviewer's repro and was red-green verified against the unguarded version. Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com>
3 tasks
Contributor
Author
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
Iteration 69 of the autonomous paper-implementation loop: kripp_alpha — Krippendorff's alpha for inter-rater agreement, a Rust reimplementation of CRAN irr 0.85
kripp.alpha()(R/kripp.alpha.R— READ and normative; Krippendorff 1980 NOT READ, cited as method origin only).Stacked on #309 (icc).
What
kripp_alpha(ratings, nraters, nsubjects, method) -> Result<KrippResult, String>incrates/mlsirm-core/src/reliability.rs: coincidence matrix over unordered rater pairs per subject, with the irr divisor quirk preserved verbatim (mc = #nonmissing − 1per column ONLY when the matrix contains any missing value, else 1 — complete-data alpha intentionally differs from the m−1 convention), diagonal increment2/mc, mirror by assignment,nmatchval= total cell mass.alpha = 1 − (nmatchval−1)·Σ(utcm·δ²)/Σ(nc_c·nc_k·δ²).kripp_alpha/KrippResultwith the hardened input policy (masked-array reject, object-dtype outright reject, complex/bool reject, dtype-kind fiu).Evidence chain
-p mlsirm-core --lib858 passed / 0 failed; pytesttests/test_paper_features.py344 passed.Test discipline
Every assert reads crate/wrapper outputs (no test-local recomputation identities); exact
==pins where dyadic-exact, rel 1e-15 otherwise; error-contract tests assert specific messages.Adversarial implementation review outcome
FINDINGS(2)): (MAJOR) integer rating labels beyond 2^53 silently collapsed to the same float64 level — complete disagreement returned alpha = 1 with a single level; fixed in 0332c7f by rejecting integer arrays with values outside [-2^53, 2^53] (boundary itself accepted), with a red-green-verified regression test executing the reviewer's int64/uint64 repros. (MINOR) spec/implementation contract mismatch on object dtype — resolved by updating the spec to the icc final policy (object dtype rejected outright; per-element vetting retired as an unwinnable arms race).CLEAN): all round-1 repros now raise ValueError; ±2^53 boundary and in-range integers verified against crate output (2-level disagreement alpha = −0.5); empty dims, uint64::MAX, int64::MIN, huge float values (by-design accepted as exact f64s), and NaN paths probed; cargo ka_ 5 passed / 1 ignored; pytest TestKripp 5 passed; spec dtype-policy paragraph confirmed to match the implementation.Full review log: session evidence file
kripp_impl_review.md(rounds 1–2, executed repros and outputs).