feat(rag): govern perturbation anchors for privacy-preserving scoring

docs/changelog.d/777-rag-perturbation-anchors.md

@@ -0,0 +1,6 @@
+# Governed RAG perturbation anchors
+
+## Added
+
+- Added source-free, content-addressed reference-free RAG perturbation anchors with finite preregistered construct/direction semantics for unsupported claims, contradictions, irrelevant context, required-evidence removal, citation swaps, semantic paraphrases, style-only rewrites, and unanswerable queries.
+- Require distinct governed baseline and perturbed request fingerprints and fail closed on unknown perturbation semantics or malformed identities. Expected directions are validation hypotheses, not claims that an observed system actually changed or that an evaluator is ground truth.

docs/doctoring/rag_scoring_request_privacy.md

@@ -1,15 +1,47 @@
-# Governed RAG scoring request privacy
+# Governed RAG scoring request and perturbation-anchor doctoring
 
-## Standards
+## Standards and research basis
 
 American Educational Research Association, American Psychological Association, & National Council on Measurement in Education. (2014). *Standards for educational and psychological testing*. American Educational Research Association.
 
+Es, S., James, J., Espinosa-Anke, L., & Schockaert, S. (2024). RAGAS: Automated evaluation of retrieval augmented generation. In *Proceedings of the 18th Conference of the European Chapter of the Association for Computational Linguistics: System Demonstrations* (pp. 150–158). Association for Computational Linguistics. https://doi.org/10.18653/v1/2024.eacl-demo.16
+
 National Institute of Standards and Technology. (2020). *Security and privacy controls for information systems and organizations* (NIST SP 800-53 Rev. 5). https://doi.org/10.6028/NIST.SP.800-53r5
 
-## Rationale
+Saad-Falcon, J., Khattab, O., Potts, C., & Zaharia, M. (2024). ARES: An automated evaluation framework for retrieval-augmented generation systems. In *Proceedings of the 2024 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies (Volume 1: Long Papers)* (pp. 338–354). Association for Computational Linguistics. https://doi.org/10.18653/v1/2024.naacl-long.20
+
+Williamson, D. M., Xi, X., & Breyer, F. J. (2012). A framework for evaluation and use of automated scoring. *Educational Measurement: Issues and Practice, 31*(1), 2–13. https://doi.org/10.1111/j.1745-3992.2011.00223.x
+
+## Privacy and provenance rationale
+
+Reference-free RAG evaluation records provenance identities (system configuration, retrieval run, query revision fingerprints) without embedding raw query, context, answer, or source text. Managed identifiers must reject free-form content so the canonical scoring artifact cannot become a side channel. The host application may retain source material only under its own purpose-bound security, privacy, retention, and authorization controls.
+
+## Controlled perturbation anchors
+
+`RAGPerturbationAnchor` preregisters a finite construct-specific expectation against two distinct governed scoring-request fingerprints. The package currently represents these hypotheses:
 
-Reference-free RAG evaluation records provenance identities (system configuration, retrieval run, query revision fingerprints) without embedding raw query, context, or answer text. Managed identifiers must reject free-form content so the canonical scoring artifact cannot become a side channel.
+- unsupported claim or explicit contradiction -> `grounded_generation` decreases;
+- irrelevant context -> `retrieval_relevance` decreases;
+- required-evidence removal -> `coverage_or_completeness_proxy` decreases;
+- citation-target swap -> `citation_attribution` decreases;
+- semantic query paraphrase or style-only rewrite -> `robustness` remains approximately invariant; and
+- unanswerable query -> `answerability_and_abstention` increases.
+
+These are controlled-validation expectations, not observed scores, statistical tests, causal effects, ground-truth labels, or evidence that a particular judge can identify world correctness. Context-groundedness remains conditional on the declared evidence regime. A successful anchor regression therefore demonstrates contract identity and preregistered direction semantics only; empirical validation still requires governed observations, calibrated evaluator/rater evidence, subgroup/DIF and drift analysis where applicable, and recovery or human-audit evidence commensurate with the intended interpretation.
+
+## Numerical and ownership boundary
+
+The anchor layer performs no likelihood, calibration, thresholding, uncertainty estimation, scoring aggregation, retrieval, provider call, or truth adjudication. New production psychometric/numerical arithmetic remains Rust-owned with model-appropriate convergence and recovery evidence. Human, rule, AI, and LLM evaluator outputs remain fallible observations rather than truth by identity.
 
 ## Implementation
 
-`python/fast_mlsirm/scoring/rag.py` — `build_rag_scoring_request` validates `system_configuration_id` via `descriptive_identifier` and allowlists caller metadata.
+`python/fast_mlsirm/scoring/rag.py`:
+
+- `build_rag_scoring_request` validates `system_configuration_id` via `descriptive_identifier` and allowlists caller metadata;
+- `RAGPerturbationKind` and `RAGPerturbationDirection` close the perturbation vocabulary;
+- `RAGPerturbationAnchor` derives package-owned construct/direction semantics, rejects no-op baseline/perturbed request pairs, and content-addresses immutable anchor content; and
+- `build_rag_perturbation_anchor` exposes only governed identities and perturbation kind, with no raw-content parameters.
+
+`tests/test_scoring_rag_perturbation_anchors.py` binds the finite mapping, deterministic fingerprints, fail-closed malformed/unknown inputs, distinct request identities, and source-free public signature.
+
+This bounded contract advances issue #607 stage-2 validation infrastructure. It does not by itself establish RAG system validity, a scalar quality score, evaluator interchangeability, retrieval recall, world correctness, calibration, DIF invariance, or release readiness.

python/fast_mlsirm/scoring/rag.py

@@ -8,11 +8,14 @@
 from __future__ import annotations
 
 from collections.abc import Iterable, Mapping
+from dataclasses import dataclass, field
 from enum import Enum
+from types import MappingProxyType
 from typing import Any
 
-from ._contract_safety import enum_value, freeze_metadata
+from ._contract_safety import artifact_digest, enum_value, freeze_metadata
 from ._validation import (
+    CanonicalContract,
     assessment_error,
     descriptive_identifier,
     fingerprint,
@@ -41,6 +44,174 @@ class RAGCandidateVisibility(str, Enum):
     CANDIDATE_VISIBLE_CROSSFIT = "candidate_visible_crossfit"
 
 
+class RAGPerturbationKind(str, Enum):
+    """Finite controlled perturbations with preregistered construct semantics."""
+
+    UNSUPPORTED_CLAIM = "unsupported_claim"
+    EXPLICIT_CONTRADICTION = "explicit_contradiction"
+    IRRELEVANT_CONTEXT = "irrelevant_context"
+    REQUIRED_EVIDENCE_REMOVAL = "required_evidence_removal"
+    CITATION_TARGET_SWAP = "citation_target_swap"
+    SEMANTIC_QUERY_PARAPHRASE = "semantic_query_paraphrase"
+    STYLE_ONLY_REWRITE = "style_only_rewrite"
+    UNANSWERABLE_QUERY = "unanswerable_query"
+
+
+class RAGPerturbationDirection(str, Enum):
+    """Expected direction for one controlled perturbation construct."""
+
+    DECREASE = "decrease"
+    INVARIANT = "invariant"
+    INCREASE = "increase"
+
+
+_PERTURBATION_EXPECTATIONS = MappingProxyType(
+    {
+        RAGPerturbationKind.UNSUPPORTED_CLAIM: (
+            "grounded_generation",
+            RAGPerturbationDirection.DECREASE,
+        ),
+        RAGPerturbationKind.EXPLICIT_CONTRADICTION: (
+            "grounded_generation",
+            RAGPerturbationDirection.DECREASE,
+        ),
+        RAGPerturbationKind.IRRELEVANT_CONTEXT: (
+            "retrieval_relevance",
+            RAGPerturbationDirection.DECREASE,
+        ),
+        RAGPerturbationKind.REQUIRED_EVIDENCE_REMOVAL: (
+            "coverage_or_completeness_proxy",
+            RAGPerturbationDirection.DECREASE,
+        ),
+        RAGPerturbationKind.CITATION_TARGET_SWAP: (
+            "citation_attribution",
+            RAGPerturbationDirection.DECREASE,
+        ),
+        RAGPerturbationKind.SEMANTIC_QUERY_PARAPHRASE: (
+            "robustness",
+            RAGPerturbationDirection.INVARIANT,
+        ),
+        RAGPerturbationKind.STYLE_ONLY_REWRITE: (
+            "robustness",
+            RAGPerturbationDirection.INVARIANT,
+        ),
+        RAGPerturbationKind.UNANSWERABLE_QUERY: (
+            "answerability_and_abstention",
+            RAGPerturbationDirection.INCREASE,
+        ),
+    }
+)
+
+
+@dataclass(frozen=True)
+class RAGPerturbationAnchor(CanonicalContract):
+    """Source-free identity and expected direction for a controlled RAG perturbation.
+
+    The anchor binds two distinct governed scoring-request fingerprints to one
+    finite perturbation kind. ``expected_construct`` and ``expected_direction``
+    are package-derived preregistration metadata; they do not assert that an
+    observed system response actually moved in the expected direction.
+    """
+
+    anchor_id: str
+    baseline_request_fingerprint: str
+    perturbed_request_fingerprint: str
+    perturbation_kind: RAGPerturbationKind
+    expected_construct: str = field(init=False)
+    expected_direction: RAGPerturbationDirection = field(init=False)
+
+    def __post_init__(self) -> None:
+        """Normalize identity fields and derive immutable perturbation semantics."""
+        normalized_anchor_id = descriptive_identifier(
+            self.anchor_id,
+            "anchor_id",
+            "$.anchor_id",
+        )
+        normalized_baseline = fingerprint(
+            self.baseline_request_fingerprint,
+            "baseline_request_fingerprint",
+            "$.baseline_request_fingerprint",
+        )
+        normalized_perturbed = fingerprint(
+            self.perturbed_request_fingerprint,
+            "perturbed_request_fingerprint",
+            "$.perturbed_request_fingerprint",
+        )
+        normalized_kind = enum_value(
+            self.perturbation_kind,
+            RAGPerturbationKind,
+            "rag_perturbation_kind",
+            "$.perturbation_kind",
+        )
+        if normalized_baseline == normalized_perturbed:
+            raise assessment_error(
+                "identical_rag_perturbation_requests",
+                "$.perturbed_request_fingerprint",
+                "baseline and perturbed request fingerprints must differ",
+            )
+        expected_construct, expected_direction = _PERTURBATION_EXPECTATIONS[
+            normalized_kind
+        ]
+        object.__setattr__(self, "anchor_id", normalized_anchor_id)
+        object.__setattr__(
+            self,
+            "baseline_request_fingerprint",
+            normalized_baseline,
+        )
+        object.__setattr__(
+            self,
+            "perturbed_request_fingerprint",
+            normalized_perturbed,
+        )
+        object.__setattr__(self, "perturbation_kind", normalized_kind)
+        object.__setattr__(self, "expected_construct", expected_construct)
+        object.__setattr__(self, "expected_direction", expected_direction)
+
+    def _content_dict(self) -> dict[str, Any]:
+        """Return canonical anchor content without the derived content digest."""
+        return {
+            "anchor_id": self.anchor_id,
+            "baseline_request_fingerprint": self.baseline_request_fingerprint,
+            "perturbed_request_fingerprint": self.perturbed_request_fingerprint,
+            "perturbation_kind": self.perturbation_kind.value,
+            "expected_construct": self.expected_construct,
+            "expected_direction": self.expected_direction.value,
+        }
+
+    @property
+    def anchor_fingerprint(self) -> str:
+        """Return SHA-256 over the complete immutable perturbation-anchor content."""
+        return artifact_digest(self)
+
+    def to_dict(self) -> dict[str, Any]:
+        """Return canonical anchor content plus its deterministic fingerprint."""
+        return {
+            **self._content_dict(),
+            "anchor_fingerprint": self.anchor_fingerprint,
+        }
+
+
+def build_rag_perturbation_anchor(
+    *,
+    anchor_id: str,
+    baseline_request_fingerprint: str,
+    perturbed_request_fingerprint: str,
+    perturbation_kind: RAGPerturbationKind | str,
+) -> RAGPerturbationAnchor:
+    """Build one content-addressed controlled RAG perturbation anchor.
+
+    The contract intentionally stores only bounded identities and package-owned
+    preregistration semantics. It does not accept raw query, context, response,
+    or source text and performs no scoring, retrieval, or truth adjudication.
+    """
+    return RAGPerturbationAnchor(
+        anchor_id=anchor_id,
+        baseline_request_fingerprint=baseline_request_fingerprint,
+        perturbed_request_fingerprint=perturbed_request_fingerprint,
+        perturbation_kind=perturbation_kind,
+    )
+
+
 _MANAGED_METADATA_KEYS = frozenset(
     {
         "rag_evidence_regime",
@@ -220,5 +391,9 @@ def build_rag_scoring_request(
 __all__ = [
     "RAGCandidateVisibility",
     "RAGEvidenceRegime",
+    "RAGPerturbationAnchor",
+    "RAGPerturbationDirection",
+    "RAGPerturbationKind",
+    "build_rag_perturbation_anchor",
     "build_rag_scoring_request",
 ]