Skip to content

Add kripp_alpha Krippendorff's alpha (irr 0.85 kripp.alpha, READ source) - #311

Merged
seonghobae merged 2 commits into
seonghobae-iccfrom
seonghobae-kripp
Jul 31, 2026
Merged

Add kripp_alpha Krippendorff's alpha (irr 0.85 kripp.alpha, READ source)#311
seonghobae merged 2 commits into
seonghobae-iccfrom
seonghobae-kripp

Conversation

@seonghobae

@seonghobae seonghobae commented Jul 27, 2026

Copy link
Copy Markdown
Contributor

Summary

Iteration 69 of the autonomous paper-implementation loop: kripp_alpha — Krippendorff's alpha for inter-rater agreement, a Rust reimplementation of CRAN irr 0.85 kripp.alpha() (R/kripp.alpha.R — READ and normative; Krippendorff 1980 NOT READ, cited as method origin only).

Stacked on #309 (icc).

What

  • Rust core kripp_alpha(ratings, nraters, nsubjects, method) -> Result<KrippResult, String> in crates/mlsirm-core/src/reliability.rs: coincidence matrix over unordered rater pairs per subject, with the irr divisor quirk preserved verbatim (mc = #nonmissing − 1 per column ONLY when the matrix contains any missing value, else 1 — complete-data alpha intentionally differs from the m−1 convention), diagonal increment 2/mc, mirror by assignment, nmatchval = total cell mass.
  • All four distance metrics: nominal, ordinal (half-endpoint coincidence-row-sum weights), interval, ratio; alpha = 1 − (nmatchval−1)·Σ(utcm·δ²)/Σ(nc_c·nc_k·δ²).
  • Fewer than 2 observed levels → alpha = 1 (R line 45).
  • Documented deviations from R: all-missing matrix, infinities, and ratio level pairs summing to zero are explicit errors (R would return alpha=1, propagate, or emit Inf/NaN).
  • PyO3 binding + thin Python wrapper kripp_alpha / KrippResult with the hardened input policy (masked-array reject, object-dtype outright reject, complex/bool reject, dtype-kind fiu).

Evidence chain

  • Spec-verify (adversarial, BEFORE implementation): APPROVED-WITH-CHANGES; all 5 mandatory changes adopted (all-missing → Err labeled intentional divergence; ratio zero-sum unconditional error; explicit Python dtype policy; MU2 mutant pin 11/18; tolerance wording).
  • Exact-Fraction oracle (EXECUTED, all-pass): K1 = irr man-page 4×12 nmm matrix — nominal 113/152, ordinal 108577/133160, interval 951/1120, ratio 18222619/22852465, nmv = 40, levels = 5. K2 = 3×5 complete-data no-NA quirk pin: nominal 43/72 (the m−1 mutant would give 11/18). K3 = hand-derived 2×4 fixture (independently re-derived by the spec reviewer). K4 = single-level alpha = 1.
  • Mutation kill map: 6/6 mutants EXECUTED and KILLED (MU1 diagonal 1/mc, MU2 mc always m−1, MU3 ordinal full-weight sum, MU4 interval |δ| not δ², MU5 nmv off-diagonal only, MU6 num × nc product).
  • MC-500: rater/subject permutation invariance across all four metrics (#[ignore], passes).
  • Full suites: cargo -p mlsirm-core --lib 858 passed / 0 failed; pytest tests/test_paper_features.py 344 passed.

Test discipline

Every assert reads crate/wrapper outputs (no test-local recomputation identities); exact == pins where dyadic-exact, rel 1e-15 otherwise; error-contract tests assert specific messages.

Adversarial implementation review outcome

  • Round 1 (FINDINGS(2)): (MAJOR) integer rating labels beyond 2^53 silently collapsed to the same float64 level — complete disagreement returned alpha = 1 with a single level; fixed in 0332c7f by rejecting integer arrays with values outside [-2^53, 2^53] (boundary itself accepted), with a red-green-verified regression test executing the reviewer's int64/uint64 repros. (MINOR) spec/implementation contract mismatch on object dtype — resolved by updating the spec to the icc final policy (object dtype rejected outright; per-element vetting retired as an unwinnable arms race).
  • Round 2 (CLEAN): all round-1 repros now raise ValueError; ±2^53 boundary and in-range integers verified against crate output (2-level disagreement alpha = −0.5); empty dims, uint64::MAX, int64::MIN, huge float values (by-design accepted as exact f64s), and NaN paths probed; cargo ka_ 5 passed / 1 ignored; pytest TestKripp 5 passed; spec dtype-policy paragraph confirmed to match the implementation.

Full review log: session evidence file kripp_impl_review.md (rounds 1–2, executed repros and outputs).

Rust reimplementation of CRAN irr 0.85 kripp.alpha() (R/kripp.alpha.R,
READ and normative; Krippendorff 1980 NOT READ, cited as method origin
only). Coincidence matrix over unordered rater pairs with the irr
divisor quirk preserved verbatim (mc = #nonmissing-1 per column only
when any value is missing, else 1), all four metrics (nominal, ordinal
half-endpoint weights, interval, ratio), alpha = 1 when fewer than two
observed levels. Documented deviations: all-missing, infinities, and
ratio zero-sum level pairs are explicit errors.

Evidence: exact-Fraction oracle anchors K1-K4 (nominal 113/152,
ordinal 108577/133160, interval 951/1120, ratio 18222619/22852465,
nmv=40; no-NA quirk pin 43/72 vs m-1 mutant's 11/18), 6-mutant
EXECUTED kill map, MC-500 permutation invariance. cargo 858 pass,
pytest paper suite 344 pass.

Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com>
@coderabbitai

coderabbitai Bot commented Jul 27, 2026

Copy link
Copy Markdown
Contributor

Important

Review skipped

Auto reviews are disabled on base/target branches other than the default branch.

Please check the settings in the CodeRabbit UI or the .coderabbit.yaml file in this repository. To trigger a single review, invoke the @coderabbitai review command.

⚙️ Run configuration

Configuration used: Organization UI

Review profile: CHILL

Plan: Pro Plus

Run ID: 0b511ef5-dfc0-46d6-8b56-2f498c0f51f9

You can disable this status message by setting the reviews.review_status to false in the CodeRabbit configuration file.

Use the checkbox below for a quick retry:

  • 🔍 Trigger review
✨ Finishing Touches
🧪 Generate unit tests (beta)
  • Create PR with unit tests
  • Commit unit tests in branch seonghobae-kripp

Comment @coderabbitai help to get the list of available commands.

Review finding (MAJOR): int64/uint64 rating labels beyond 2**53 are not
exactly representable as float64, so distinct levels silently collapsed
during the cast — complete disagreement returned alpha=1 with a single
level. The wrapper now rejects any integer array with values outside
[-2**53, 2**53]; the boundary itself remains accepted (exactly
representable). Regression test executes the reviewer's repro and was
red-green verified against the unguarded version.

Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com>
@seonghobae
seonghobae merged commit 495e77b into seonghobae-icc Jul 31, 2026
6 checks passed
@seonghobae
seonghobae deleted the seonghobae-kripp branch July 31, 2026 12:38
@seonghobae

Copy link
Copy Markdown
Contributor Author

Note: this PR was squash-merged into a non-main stacked base, so its commits did not land on main via this PR. The feature set reached main via #374 (stack tip integration) after #290.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant