Add Wollack-style omega answer-copying statistic - #259
Conversation
|
Warning Review limit reached
Next review available in: 2 minutes Enable usage-based reviews in Billing to review now. Otherwise, wait until the next included review is available. How can I continue?After more reviews become available, a review can be triggered using the To avoid repeated limits, reduce automatic review volume by pausing incremental auto-reviews earlier, using label-based review opt-in, excluding WIP or generated PR titles, or requesting reviews manually when the PR is ready. If your team needs uninterrupted high-volume reviews, an organization admin can enable usage-based reviews. How do review limits work?CodeRabbit enforces per-developer PR review limits for each organization. Most developers receive the normal plan review availability. For paid Pro and Pro+ PR reviews, CodeRabbit uses adaptive limits for sustained high-volume activity. When a developer's recent PR review activity reaches the 95th percentile or higher among CodeRabbit users, additional reviews become available more gradually as earlier reviews age out of the rolling window. Please refer docs for additional details. Review details⚙️ Run configurationConfiguration used: Organization UI Review profile: CHILL Plan: Pro Plus Run ID: 📒 Files selected for processing (12)
✨ Finishing Touches🧪 Generate unit tests (beta)
Comment |
dc6f32e to
d0a3573
Compare
![opencode-agent[bot]](https://avatars.githubusercontent.com/in/1549082?s=60&v=4){kind=small}
There was a problem hiding this comment.
Pull request overview
OpenCode cannot approve yet because required coverage evidence did not pass.
Review outcome
1. HIGH .github/workflows/opencode-review.yml:1 - Coverage evidence did not prove required test/docstring evidence
-
Problem: The required coverage-evidence job result was
failure, so OpenCode cannot establish approval sufficiency for this head. -
Root cause: Automated approval is only valid when the same-head coverage-evidence job proves supported repository test suites passed and configured docstring gates passed or were advisory, or reports not applicable because no supported source files or package manifests exist. Missing, failed, skipped, unavailable, or unsupported-tooling test evidence is a blocker.
-
Fix: Install or configure the repository test/docstring evidence tooling when source files or package manifests exist, rerun the current-head coverage-evidence job, and approve only after it reports
successwith required evidence or explicit no-source not-applicable evidence. -
Regression test: Keep the approval branch checking
needs.coverage-evidence.result == successbefore posting APPROVE, and publish REQUEST_CHANGES when coverage-evidence blocker states such as cancelled, skipped, failed, unsupported-tooling, or below-100 evidence are present. -
Result: REQUEST_CHANGES
-
Reason: coverage-evidence result was
failure, so required test/docstring evidence was not proven for current head0bced3c4a1300fcf0c5f7177fb1579fc903468b5. -
Head SHA:
0bced3c4a1300fcf0c5f7177fb1579fc903468b5 -
Workflow run: 30197002761
-
Workflow attempt: 1
Coverage evidence
Coverage evidence job did not run or did not publish coverage evidence.
Changed-File Evidence Map
flowchart LR
PR["PR changed files"] --> Evidence["OpenCode bounded evidence"]
Evidence --> S1["Changed file (10 files)"]
S1 --> I1["repository behavior"]
I1 --> Conflict["Merge conflict blocks this path"]
Conflict --> V1["required checks"]
Evidence --> S2["Test (5 files)"]
S2 --> I2["regression suite"]
I2 --> Conflict["Merge conflict blocks this path"]
Conflict --> V2["targeted test run"]
OpenCode Review Overview
Pull request overviewOpenCode cannot approve yet because required coverage evidence did not pass. Review outcome1. HIGH .github/workflows/opencode-review.yml:1 - Coverage evidence did not prove required test/docstring evidence
Coverage evidenceCoverage evidence job did not run or did not publish coverage evidence. Changed-File Evidence Mapflowchart LR
PR["PR changed files"] --> Evidence["OpenCode bounded evidence"]
Evidence --> S1["Changed file (10 files)"]
S1 --> I1["repository behavior"]
I1 --> Conflict["Merge conflict blocks this path"]
Conflict --> V1["required checks"]
Evidence --> S2["Test (5 files)"]
S2 --> I2["regression suite"]
I2 --> Conflict["Merge conflict blocks this path"]
Conflict --> V2["targeted test run"]
Merge Conflict Guidance
gh pr checkout 259 --repo ContextualWisdomLab/fast-mlsirm
git fetch origin seonghobae-ci-classify
git merge --no-ff origin/seonghobae-ci-classify # or: git rebase origin/seonghobae-ci-classify
git status --short
# resolve files, then git add <resolved-files>
# merge path: git commit
# rebase path: git rebase --continue
git push origin HEAD:seonghobae-wollack-omega
# rebase path only: git push --force-with-lease origin HEAD:seonghobae-wollack-omega |
0bced3c to
d15abdc
Compare
…ensionality test Implements the original (non-bootstrap) DIMTEST with caller-supplied confirmatory AT1/AT2 partitions, PT raw-score grouping (J_k >= 20), ML variance denominators, the Nandakumar & Stout (1992/1993) refined bias-correction denominator S_k, T_L/T_B statistics, and one-sided p-value. Formulas transcribed from the inspected ERIC ED351383 technical report (pre-publication of Nandakumar & Stout, 1993, JEBS). - Rust core in mlsirm-core detect.rs with pinned 17-digit oracle (independent Python reference), exact-null bits fixture, error paths, and 500-rep Monte Carlo size/power test (#[ignore]). - 5/5 mutation kills executed (sign swap, J_k-1 denominator, dropped bias correction, Jmin relaxation, two-sided p). - PyO3 binding py_dimtest + thin Python wrapper with validate-before- cast input checks; exports and pytest coverage. Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com>
…group oracle anchors - Python wrapper now rejects non-numeric (e.g. string) response and index arrays via dtype-kind checks BEFORE any astype coercion, with regression tests for string responses and string indices. - Rust: factor dimtest grouping (dimtest_pt_groups) and per-group computation (dimtest_group_diag) out of dimtest_stat, and add a test pinning all 8x2 per-group intermediates (J_k, mean, sigma_k^2, sigma_U,k^2, mu4_k, delta4_k, S_k^2, contribution) for AT1 and AT2 against the independent oracle at 1e-12, closing the compensating-error audit gap. Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com>
Implements the standardized answer-similarity index omega for a suspected copier-source pair: h counts identical observed options, p_i is the copier's model-implied probability of the source's observed option, omega = (h - sum p_i)/sqrt(sum p_i (1-p_i)) with a one-sided upper-tail normal p-value. Citation governance: formula verified against two independently READ implementations (CRAN CopyDetect similarity1.r/similarity2.r and the aberrance package compute_OMG); Wollack (1997) itself NOT read (access blocked) and cited only as implemented by those sources. CopyDetect's printed docs flip the sign but both source files use (h-E)/sqrt(V); the source convention is implemented. Scope: omega only - no g2/GBT/K-index, no continuity correction, no missing responses. Rust core mlsirm_core::security::wollack_omega with full input validation; PyO3 py_wollack_omega; Python wrapper fast_mlsirm. wollack_omega with pre-cast dtype validation (complex/string/fractional rejection). Tests: pinned independent oracle at 1e-12 (p at 5e-7 via crate erfc), structural single-item-extension invariant, error paths, 500-rep Monte Carlo size/power (#[ignore]); 3 executed mutation kills (V-vs-sqrt(V), copier-prob lookup, two-sided p). Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com>
…ance authorship Impl-review round 1 found: (MAJOR) the Python wrapper silently accepted boolean response arrays as option indices 0/1 — bool dtype is now rejected before any cast, with a regression test; (MINOR) the aberrance package citation claimed authors/year not established from the sources actually read — the citation now names only the package and the read source files. Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com>
Review round 2 (MINOR): the sign-conflict disclosure existed in the PyO3 docstring, Python docstring, and CHANGELOG but not in the Rust module documentation. All four surfaces now carry it. Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com>
* Add K-index answer-copying statistic (CopyDetect-faithful port) Implement the K-index of matching incorrect answers exactly as in the CRAN CopyDetect package's internal k() (R/similarity1.r, READ; corroborated by R/similarity2.r): number-incorrect subgroup keyed on the copier's score (including the copier and, when scores match, the source - CopyDetect convention, source exclusion NOT applied), p = mean(emp_agg)/ws, and the binomial upper tail P(Bin(ws, p) >= m) computed with a coefficient-free f64 term recurrence (no factorial overflow, smaller-tail summation, [0,1] clamp). Holland (1996, ETS RR-96-07) and Sotaridona & Meijer (2002, JEM 39(2)) were NOT read; the statistic is cited only as implemented by CopyDetect. Sotaridona & Meijer (2001, RR-01-07, ERIC ED467373) was read for background corroboration of the binomial framework only. The aberrance package was checked and contains no K-index. Rust core mlsirm_core::security::k_index + KIndexResult; PyO3 py_k_index; Python fast_mlsirm.k_index with pre-cast validation (complex/bool/non-binary rejected, ws==0 degenerate error). Tests: pinned oracle (independent exact-binomial Python oracle, all 7 result fields), subgroup-membership invariants (copier always included; source included when its score matches), degenerate branches (m==0 -> K=1, p==1 -> K=1), error paths, and a 500-rep Monte Carlo (#[ignore]) checking null size < 0.10 and power > 0.5 under 90% full-response copying. Three executed mutation kills documented: tail flip, p denominator wc-for-ws, subgroup keyed on ws. Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com> * Fix binomial tail underflow in k_index at extreme p / large n The linear-space term recurrence started from (1-p)^n, which underflows to zero for cases like n=1000, p=0.99 (K wrongly returned 0 instead of 0.58304080330109709). Sum the upper-tail terms directly in log space (log-term recurrence + max-shifted exponential sum): no complement subtraction, no cancellation, no underflow. Regression test pins the extreme-p case against an exact fractions.Fraction reference computed on the exact binary value of 0.99, plus the moderate oracle case. Found by adversarial implementation review. Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com> --------- Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com>
d15abdc to
c20a2f2
Compare
Dismissing stale bot request after branch rebuild and successful required checks.
1 participant
Implements the Wollack-style omega answer-copying statistic as
fast_mlsirm.wollack_omega(Rust coremlsirm_core::security::wollack_omega, PyO3py_wollack_omega). Stacked on #258 (DIMTEST).Contract
h= count of items where copier and source chose the same option.p_i = P_i[source_i]— the COPIER's model-implied probability of the SOURCE's observed option (caller supplies fitted option probabilities, e.g. from a nominal response model).omega = (h - sum p_i) / sqrt(sum p_i (1 - p_i)), one-sided upper-tail normal p-value.Citation governance
similarity1.r/similarity2.r) and the independent aberrance package (compute_OMG) — formula verified against both.(E-h)/sqrt(V)but both source files compute(h-E)/sqrt(V)with an upper-tail p; the source convention is implemented.Evidence
OmegaResultfields.Adversarial impl-review