fix(validation): harden judge category controls

docs/changelog.d/912-judge-category-control-safety.md

@@ -0,0 +1,9 @@
+# Judge category-count control hardening
+
+## Security
+
+- Validate the public `validate_judge(..., k=...)` category count before compiled-core discovery.
+- Accept exact built-in integers and genuine concrete NumPy integer scalars while rejecting booleans, subclasses, and arbitrary integer-conversion protocol providers without executing caller conversion callbacks.
+- Marshal only a trusted built-in integer into the existing Rust-owned judge-validation computation; psychometric/fairness formulas, thresholds, and result schemas are unchanged.
+
+Closes #912.

docs/doctoring/judge_category_control_safety.md

@@ -0,0 +1,39 @@
+# Judge category-count control safety
+
+## Scope
+
+Issue #912 hardens only the Python validation/marshalling boundary for the public `validate_judge(..., k=...)` category-count control. The Rust-owned agreement, quadratic-weighted kappa, degradation, standardized-mean-difference/fairness, gate-decision arithmetic, governed thresholds, and result schema are unchanged.
+
+Protected `main` previously imported the compiled core before validating `k` and repeatedly invoked `int(k)`. Because Python integer subclasses and arbitrary integer-protocol objects may supply executable conversion methods, that order allowed caller-controlled conversion code to run while establishing a security- and fairness-relevant control.
+
+The bounded correction admits only an exact built-in `int` or an exact concrete NumPy integer scalar identity, normalizes a trusted NumPy scalar once, enforces the existing `2..=1000` domain, completes label/policy marshalling, and only then imports the Rust core. Booleans, Python/NumPy subclasses, and arbitrary conversion-protocol providers fail before conversion callbacks or Rust dispatch. NumPy scalar-type admission uses identity comparisons rather than set membership so a caller-controlled scalar metaclass cannot inject `__hash__` or `__eq__` execution into the trust decision.
+
+## Verification contract
+
+The regression suite must prove all of the following on the exact PR head:
+
+- hostile Python integer subclasses execute zero `__int__` callbacks;
+- hostile NumPy integer subclasses execute zero `__int__` callbacks;
+- arbitrary integer-protocol providers execute zero callbacks;
+- caller-controlled NumPy scalar metaclasses execute zero hashing/equality callbacks during type admission;
+- invalid exact category counts fail before compiled-core import;
+- genuine concrete NumPy integer scalars remain compatible and arrive at the Rust boundary as an exact built-in integer; and
+- no judge-validation numerical formula or policy threshold changes.
+
+The initial test commit `7cf9eb6c2937020b5e755b5ae0a6cc2380fc068d` records the conversion/native-discovery RED contract, and `db0cb5848d317d39f197933043e289e00cdf522b` supplies its first bounded GREEN. A second RED at `3979f064b3e32fe44892cab369f8ffc1e3af4d73` demonstrates that hashed type-container membership would still execute caller-controlled metaclass hooks; `54fe33b2dd9d2a287c04635f2acba7bfc94f10fa` replaces that admission with identity-only comparisons. Hosted exact-head evidence remains authoritative over these remembered identities and must be refetched before lifecycle or integration decisions.
+
+## Standards and research basis
+
+The security references below govern input-validation and development-process evidence; they are not psychometric validity authorities. OWASP ASVS 5.0.0 is the latest stable ASVS release as checked on 2026-08-16; OWASP separately labels its bleeding-edge build as preview-only. NIST SP 800-218 Rev. 1 / SSDF 1.2 remains an Initial Public Draft, so this work retains final SSDF 1.1 as the normative NIST baseline and records the draft only as a standards-watch item.
+
+### References (APA 7th)
+
+MITRE. (2026). *CWE-1287: Improper validation of specified type of input* (CWE Version 4.20). Common Weakness Enumeration. https://cwe.mitre.org/data/definitions/1287.html
+
+National Institute of Standards and Technology. (2022). *Secure software development framework (SSDF) version 1.1: Recommendations for mitigating the risk of software vulnerabilities* (NIST Special Publication 800-218). U.S. Department of Commerce. https://doi.org/10.6028/NIST.SP.800-218
+
+National Institute of Standards and Technology. (2025). *Secure software development framework (SSDF) version 1.2: Recommendations for mitigating the risk of software vulnerabilities* (NIST Special Publication 800-218 Rev. 1, Initial Public Draft). U.S. Department of Commerce. https://csrc.nist.gov/pubs/sp/800/218/r1/ipd
+
+OWASP Foundation. (2025). *OWASP application security verification standard 5.0.0*. https://github.com/OWASP/ASVS/tree/v5.0.0_release
+
+Williamson, D. M., Xi, X., & Breyer, F. J. (2012). A framework for evaluation and use of automated scoring. *Educational Measurement: Issues and Practice, 31*(1), 2–13. https://doi.org/10.1111/j.1745-3992.2011.00223.x

python/fast_mlsirm/validation.py

@@ -16,6 +16,48 @@
 import numpy as np
 
 
+MAX_JUDGE_CATEGORIES = 1_000
+_TRUSTED_NUMPY_INTEGER_SCALAR_TYPES = tuple(
+    np.dtype(code).type
+    for code in ("b", "B", "h", "H", "i", "I", "l", "L", "q", "Q", "p", "P")
+)
+
+
+def _is_exact_numpy_integer_scalar_type(value_type: type) -> bool:
+    """Return whether ``value_type`` is a package-trusted NumPy integer type.
+
+    Identity comparisons deliberately avoid hashing or equality on a
+    caller-controlled metaclass. This keeps type admission inert even for a
+    NumPy scalar subclass that overrides metaclass ``__hash__`` or ``__eq__``.
+    """
+    return any(value_type is trusted_type for trusted_type in _TRUSTED_NUMPY_INTEGER_SCALAR_TYPES)
+
+
+def _trusted_judge_category_count(value: object) -> int:
+    """Return a trusted built-in category count without caller coercion callbacks.
+
+    Only an exact built-in :class:`int` or an exact concrete NumPy integer scalar
+    identity is normalized. Python/NumPy subclasses, booleans, and arbitrary
+    integer-protocol providers are rejected before ``int`` can execute caller
+    code. The returned value is always an exact built-in integer suitable for
+    downstream NumPy validation and PyO3 marshalling.
+    """
+    value_type = type(value)
+    if value_type is int:
+        normalized = value
+    elif _is_exact_numpy_integer_scalar_type(value_type):
+        normalized = int(value)
+    else:
+        raise ValueError("k (number of categories) must be an integer")
+
+    if normalized < 2:
+        raise ValueError("k (number of categories) must be >= 2")
+    if normalized > MAX_JUDGE_CATEGORIES:
+        # k drives a dense k-by-k confusion matrix in the Rust core.
+        raise ValueError(f"k (number of categories) must be <= {MAX_JUDGE_CATEGORIES}")
+    return normalized
+
+
 def _validate_labels(a, name: str, *, k: int | None = None, n: int | None = None) -> np.ndarray:
     """Validate caller-supplied category labels before the uint32 conversion the
     Rust gate expects: reject non-1-D, wrong-length, non-finite, non-integer,
@@ -130,27 +172,20 @@ def validate_judge(
     degradation criterion; ``subgroup`` labels each observation for the
     fairness SMD.
     """
-    from . import _core  # computation lives in the Rust core
-
     active_policy = policy if policy is not None else ValidationPolicy()
     if not isinstance(active_policy, ValidationPolicy):
         raise TypeError("policy must be a ValidationPolicy")
 
-    MAX_JUDGE_CATEGORIES = 1_000
-    if int(k) < 2:
-        raise ValueError("k (number of categories) must be >= 2")
-    if int(k) > MAX_JUDGE_CATEGORIES:
-        # k drives a dense k-by-k confusion matrix in the Rust core.
-        raise ValueError(f"k (number of categories) must be <= {MAX_JUDGE_CATEGORIES}")
-    judge_v = _validate_labels(judge, "judge", k=int(k))
-    human_v = _validate_labels(human, "human", k=int(k), n=judge_v.shape[0])
+    category_count = _trusted_judge_category_count(k)
+    judge_v = _validate_labels(judge, "judge", k=category_count)
+    human_v = _validate_labels(human, "human", k=category_count, n=judge_v.shape[0])
     kwargs: dict[str, Any] = {}
     if human_human is not None:
         kwargs["human_a"] = _validate_labels(
-            human_human[0], "human_a", k=int(k), n=judge_v.shape[0]
+            human_human[0], "human_a", k=category_count, n=judge_v.shape[0]
         )
         kwargs["human_b"] = _validate_labels(
-            human_human[1], "human_b", k=int(k), n=kwargs["human_a"].shape[0]
+            human_human[1], "human_b", k=category_count, n=kwargs["human_a"].shape[0]
         )
     if subgroup is not None:
         sg = _validate_labels(subgroup, "subgroup", n=judge_v.shape[0])
@@ -159,10 +194,13 @@ def validate_judge(
         _uniq, sg_compact = np.unique(sg, return_inverse=True)
         kwargs["subgroup"] = sg_compact.astype(np.uint32)
     kwargs.update(active_policy.rust_kwargs())
+
+    from . import _core  # computation lives in the Rust core
+
     res = _core.validate_scoring(
         judge_v,
         human_v,
-        int(k),
+        category_count,
         **kwargs,
     )
     gates = [dict(g) for g in res["gates"]]
@@ -267,6 +305,8 @@ def fleiss_kappa(
         category_z=np.asarray(res["category_z"]),
         category_p=np.asarray(res["category_p"]),
     )
+
+
 @dataclass
 class LightKappaResult:
     """Result of :func:`light_kappa`: mean pairwise unweighted Cohen's kappa

tests/test_validation_category_control_safety.py

@@ -0,0 +1,156 @@
+"""Security regressions for judge category-count validation."""
+
+from __future__ import annotations
+
+import builtins
+from types import SimpleNamespace
+
+import numpy as np
+import pytest
+
+import fast_mlsirm
+from fast_mlsirm import validation
+
+
+class _HostileInt(int):
+    """Integer subclass whose conversion callback records execution."""
+
+    calls = 0
+
+    def __int__(self) -> int:
+        type(self).calls += 1
+        return 2
+
+
+class _HostileNumpyInt(np.int64):
+    """NumPy integer subclass whose conversion callback records execution."""
+
+    calls = 0
+
+    def __int__(self) -> int:
+        type(self).calls += 1
+        return 2
+
+
+class _HostileScalarMeta(type):
+    """Metaclass proving scalar-type admission cannot hash caller types."""
+
+    calls = 0
+
+    def __hash__(cls) -> int:
+        type(cls).calls += 1
+        return type.__hash__(np.int64)
+
+    def __eq__(cls, other: object) -> bool:
+        type(cls).calls += 1
+        return type.__eq__(cls, other)
+
+
+class _MetaclassHostileNumpyInt(np.int64, metaclass=_HostileScalarMeta):
+    """NumPy scalar subclass with caller-controlled type hash/equality hooks."""
+
+
+class _IntegerProtocolProvider:
+    """Arbitrary integer protocol provider that must never be invoked."""
+
+    calls = 0
+
+    def __int__(self) -> int:
+        type(self).calls += 1
+        return 2
+
+
+def _fake_core(calls: list[tuple[int, type[int]]]) -> SimpleNamespace:
+    """Return a deterministic Rust-boundary stand-in for marshalling tests."""
+
+    def validate_scoring(judge, human, k, **kwargs):
+        del judge, human, kwargs
+        calls.append((k, type(k)))
+        return {
+            "gates": [],
+            "exact_agreement": 1.0,
+            "adjacent_agreement": 1.0,
+            "pass": True,
+        }
+
+    return SimpleNamespace(validate_scoring=validate_scoring)
+
+
+@pytest.mark.parametrize(
+    "control_type",
+    [_HostileInt, _HostileNumpyInt, _IntegerProtocolProvider],
+)
+def test_validate_judge_rejects_executable_integer_controls_without_callbacks(
+    monkeypatch: pytest.MonkeyPatch,
+    control_type: type,
+) -> None:
+    """Rejected category controls cannot run conversion code or Rust dispatch."""
+    control_type.calls = 0
+    rust_calls: list[tuple[int, type[int]]] = []
+    monkeypatch.setattr(fast_mlsirm, "_core", _fake_core(rust_calls), raising=False)
+
+    value = control_type(2) if control_type is not _IntegerProtocolProvider else control_type()
+    with pytest.raises(ValueError):
+        validation.validate_judge(np.array([0, 1]), np.array([0, 1]), k=value)
+
+    assert control_type.calls == 0
+    assert rust_calls == []
+
+
+def test_validate_judge_rejects_hostile_scalar_metaclass_without_callbacks(
+    monkeypatch: pytest.MonkeyPatch,
+) -> None:
+    """Scalar-type admission cannot dispatch caller metaclass hash/equality hooks."""
+    _HostileScalarMeta.calls = 0
+    rust_calls: list[tuple[int, type[int]]] = []
+    monkeypatch.setattr(fast_mlsirm, "_core", _fake_core(rust_calls), raising=False)
+
+    with pytest.raises(ValueError):
+        validation.validate_judge(
+            np.array([0, 1]),
+            np.array([0, 1]),
+            k=_MetaclassHostileNumpyInt(2),
+        )
+
+    assert _HostileScalarMeta.calls == 0
+    assert rust_calls == []
+
+
+def test_validate_judge_rejects_invalid_k_before_core_import(
+    monkeypatch: pytest.MonkeyPatch,
+) -> None:
+    """An invalid exact category count fails before compiled-core discovery."""
+    real_import = builtins.__import__
+    core_import_calls = 0
+
+    def guarded_import(name, globals=None, locals=None, fromlist=(), level=0):
+        nonlocal core_import_calls
+        if "_core" in fromlist:
+            core_import_calls += 1
+            raise AssertionError("compiled core discovered before k validation")
+        return real_import(name, globals, locals, fromlist, level)
+
+    monkeypatch.setattr(builtins, "__import__", guarded_import)
+
+    with pytest.raises(ValueError, match="must be >= 2"):
+        validation.validate_judge(np.array([0, 1]), np.array([0, 1]), k=1)
+
+    assert core_import_calls == 0
+
+
+@pytest.mark.parametrize(
+    "k",
+    [np.int8(2), np.uint16(2), np.int32(2), np.uint64(2)],
+)
+def test_validate_judge_normalizes_genuine_numpy_category_counts(
+    monkeypatch: pytest.MonkeyPatch,
+    k: np.integer,
+) -> None:
+    """Supported concrete NumPy integer scalars marshal as exact Python ints."""
+    rust_calls: list[tuple[int, type[int]]] = []
+    monkeypatch.setattr(fast_mlsirm, "_core", _fake_core(rust_calls), raising=False)
+
+    verdict = validation.validate_judge(np.array([0, 1]), np.array([0, 1]), k=k)
+
+    assert verdict.passed is True
+    assert rust_calls == [(2, int)]