diff --git a/docs/changelog.d/777-rag-perturbation-anchors.md b/docs/changelog.d/777-rag-perturbation-anchors.md new file mode 100644 index 000000000..6060db870 --- /dev/null +++ b/docs/changelog.d/777-rag-perturbation-anchors.md @@ -0,0 +1,7 @@ +# Governed RAG perturbation anchors + +## Added + +- Added source-free, content-addressed reference-free RAG perturbation anchors with finite preregistered construct/direction semantics for unsupported claims, contradictions, irrelevant context, required-evidence removal, citation swaps, semantic paraphrases, style-only rewrites, and unanswerable queries. +- Require canonical governed baseline and perturbed `ScoringRequest` values, reject unrelated or mixed-axis pairs, and bind each anchor to exact perturbation specification/run fingerprints while serializing only source-free identities. +- Distinguish literature-aligned constructs from package-owned model-design hypotheses. Every expected direction remains a validation hypothesis, not a claim that the cited papers established the exact perturbation, that an observed system actually changed, or that an evaluator is ground truth. diff --git a/docs/doctoring/rag_scoring_request_privacy.md b/docs/doctoring/rag_scoring_request_privacy.md index 2d315713e..f0e3dbc67 100644 --- a/docs/doctoring/rag_scoring_request_privacy.md +++ b/docs/doctoring/rag_scoring_request_privacy.md @@ -1,15 +1,59 @@ -# Governed RAG scoring request privacy +# Governed RAG scoring request and perturbation-anchor doctoring -## Standards +## Standards and research basis American Educational Research Association, American Psychological Association, & National Council on Measurement in Education. (2014). *Standards for educational and psychological testing*. American Educational Research Association. +Es, S., James, J., Espinosa-Anke, L., & Schockaert, S. (2024). RAGAS: Automated evaluation of retrieval augmented generation. In *Proceedings of the 18th Conference of the European Chapter of the Association for Computational Linguistics: System Demonstrations* (pp. 150–158). Association for Computational Linguistics. https://doi.org/10.18653/v1/2024.eacl-demo.16 + National Institute of Standards and Technology. (2020). *Security and privacy controls for information systems and organizations* (NIST SP 800-53 Rev. 5). https://doi.org/10.6028/NIST.SP.800-53r5 -## Rationale +Saad-Falcon, J., Khattab, O., Potts, C., & Zaharia, M. (2024). ARES: An automated evaluation framework for retrieval-augmented generation systems. In *Proceedings of the 2024 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies (Volume 1: Long Papers)* (pp. 338–354). Association for Computational Linguistics. https://doi.org/10.18653/v1/2024.naacl-long.20 + +Williamson, D. M., Xi, X., & Breyer, F. J. (2012). A framework for evaluation and use of automated scoring. *Educational Measurement: Issues and Practice, 31*(1), 2–13. https://doi.org/10.1111/j.1745-3992.2011.00223.x + +## Privacy and provenance rationale + +Reference-free RAG evaluation records provenance identities (system configuration, retrieval run, query revision fingerprints) without embedding raw query, context, answer, or source text. Managed identifiers must reject free-form content so the canonical scoring artifact cannot become a side channel. The host application may retain source material and controlled-transformation evidence only under its own purpose-bound security, privacy, retention, and authorization controls. + +## Controlled perturbation anchors + +`RAGPerturbationAnchor` preregisters a finite construct-specific expectation against two canonical governed `ScoringRequest` values. The builder verifies that the requests share assessment, rubric, system-run, task-family, occasion, evidence-regime, candidate-visibility, system-configuration, engine-policy, and split provenance. It then enforces one permitted changed axis: + +- response perturbations preserve query and retrieval provenance and change the response artifact; +- retrieval perturbations preserve the query and response artifact and change retrieval provenance; and +- query perturbations change the governed query revision and may produce new retrieval and response artifacts. + +The anchor also records exact `perturbation_specification_fingerprint` and `perturbation_run_fingerprint` values. These identities bind the source-free numerical artifact to the host's governed transformation protocol and execution evidence. The core does not claim that hashes alone prove a semantic transformation such as “unsupported claim” or “semantic paraphrase”; the authorized host must retain that evidence. + +### Scientific claim boundary -Reference-free RAG evaluation records provenance identities (system configuration, retrieval run, query revision fingerprints) without embedding raw query, context, or answer text. Managed identifiers must reject free-form content so the canonical scoring artifact cannot become a side channel. +RAGAS and ARES directly motivate separable grounded-generation/faithfulness and retrieval/context-relevance constructs. Accordingly, unsupported-claim, explicit-contradiction, and irrelevant-context anchors are marked `literature_aligned_construct`. This label means the named construct aligns with those primary sources; it does **not** mean the papers established this package's exact perturbation or expected direction. + +The following mappings are package-owned `model_design_hypothesis` values because the cited RAGAS and ARES papers do not directly establish them: + +- required-evidence removal -> `coverage_or_completeness_proxy` decreases; +- citation-target swap -> `citation_attribution` decreases; +- semantic query paraphrase or style-only rewrite -> `robustness` remains approximately invariant; and +- unanswerable query -> `answerability_and_abstention` increases. + +Unsupported claim or explicit contradiction -> `grounded_generation` decreases, and irrelevant context -> `retrieval_relevance` decreases, are also preregistered expected directions rather than observations or causal conclusions. Every anchor direction in this package remains a hypothesis to test. + +These anchors are not observed scores, statistical tests, causal effects, ground-truth labels, or evidence that a particular judge can identify world correctness. Context-groundedness remains conditional on the declared evidence regime. A successful anchor regression demonstrates canonical identity, allowed-axis validation, governed transformation provenance, and preregistered direction semantics only. Empirical validation still requires governed observations, calibrated evaluator/rater evidence, subgroup/DIF and drift analysis where applicable, and recovery or human-audit evidence commensurate with the intended interpretation. + +## Numerical and ownership boundary + +The anchor layer performs no likelihood, calibration, thresholding, uncertainty estimation, scoring aggregation, retrieval, provider call, or truth adjudication. New production psychometric/numerical arithmetic remains Rust-owned with model-appropriate convergence and recovery evidence. Human, rule, AI, and LLM evaluator outputs remain fallible observations rather than truth by identity. ## Implementation -`python/fast_mlsirm/scoring/rag.py` — `build_rag_scoring_request` validates `system_configuration_id` via `descriptive_identifier` and allowlists caller metadata. +`python/fast_mlsirm/scoring/rag.py`: + +- `build_rag_scoring_request` validates `system_configuration_id` via `descriptive_identifier` and allowlists caller metadata; +- `RAGPerturbationKind`, `RAGPerturbationDirection`, and `RAGPerturbationConstructBasis` close the perturbation vocabulary and distinguish literature-aligned constructs from model-design hypotheses; +- `RAGPerturbationAnchor` is factory-sealed, derives package-owned construct/direction semantics, and content-addresses the request pair plus transformation specification/run provenance; and +- `build_rag_perturbation_anchor` accepts canonical governed request objects, rejects unrelated or mixed-axis pairs, serializes only fingerprints, and exposes no raw-content parameters. + +`tests/test_scoring_rag_perturbation_anchors.py` binds finite semantics, deterministic identities, factory sealing, and the source-free public signature. `tests/test_scoring_rag_perturbation_relationships.py` binds kind-specific changed axes, unrelated-pair rejection, transformation provenance, and the scientific claim-basis distinction. + +This bounded contract advances issue #607 stage-2 validation infrastructure. It does not by itself establish RAG system validity, a scalar quality score, evaluator interchangeability, retrieval recall, world correctness, calibration, DIF invariance, or release readiness. diff --git a/python/fast_mlsirm/scoring/rag.py b/python/fast_mlsirm/scoring/rag.py index 8194d9ac1..c870d2ee7 100644 --- a/python/fast_mlsirm/scoring/rag.py +++ b/python/fast_mlsirm/scoring/rag.py @@ -8,11 +8,14 @@ from __future__ import annotations from collections.abc import Iterable, Mapping +from dataclasses import InitVar, dataclass, field from enum import Enum +from types import MappingProxyType from typing import Any -from ._contract_safety import enum_value, freeze_metadata +from ._contract_safety import artifact_digest, enum_value, freeze_metadata from ._validation import ( + CanonicalContract, assessment_error, descriptive_identifier, fingerprint, @@ -41,6 +44,504 @@ class RAGCandidateVisibility(str, Enum): CANDIDATE_VISIBLE_CROSSFIT = "candidate_visible_crossfit" +class RAGPerturbationKind(str, Enum): + """Finite controlled perturbations with preregistered construct semantics.""" + + UNSUPPORTED_CLAIM = "unsupported_claim" + EXPLICIT_CONTRADICTION = "explicit_contradiction" + IRRELEVANT_CONTEXT = "irrelevant_context" + REQUIRED_EVIDENCE_REMOVAL = "required_evidence_removal" + CITATION_TARGET_SWAP = "citation_target_swap" + SEMANTIC_QUERY_PARAPHRASE = "semantic_query_paraphrase" + STYLE_ONLY_REWRITE = "style_only_rewrite" + UNANSWERABLE_QUERY = "unanswerable_query" + + +class RAGPerturbationDirection(str, Enum): + """Expected direction for one controlled perturbation construct.""" + + DECREASE = "decrease" + INVARIANT = "invariant" + INCREASE = "increase" + + +class RAGPerturbationConstructBasis(str, Enum): + """Whether cited primary sources directly align with the named construct.""" + + LITERATURE_ALIGNED_CONSTRUCT = "literature_aligned_construct" + MODEL_DESIGN_HYPOTHESIS = "model_design_hypothesis" + + +_PERTURBATION_EXPECTATIONS = MappingProxyType( + { + RAGPerturbationKind.UNSUPPORTED_CLAIM: ( + "grounded_generation", + RAGPerturbationDirection.DECREASE, + RAGPerturbationConstructBasis.LITERATURE_ALIGNED_CONSTRUCT, + ), + RAGPerturbationKind.EXPLICIT_CONTRADICTION: ( + "grounded_generation", + RAGPerturbationDirection.DECREASE, + RAGPerturbationConstructBasis.LITERATURE_ALIGNED_CONSTRUCT, + ), + RAGPerturbationKind.IRRELEVANT_CONTEXT: ( + "retrieval_relevance", + RAGPerturbationDirection.DECREASE, + RAGPerturbationConstructBasis.LITERATURE_ALIGNED_CONSTRUCT, + ), + RAGPerturbationKind.REQUIRED_EVIDENCE_REMOVAL: ( + "coverage_or_completeness_proxy", + RAGPerturbationDirection.DECREASE, + RAGPerturbationConstructBasis.MODEL_DESIGN_HYPOTHESIS, + ), + RAGPerturbationKind.CITATION_TARGET_SWAP: ( + "citation_attribution", + RAGPerturbationDirection.DECREASE, + RAGPerturbationConstructBasis.MODEL_DESIGN_HYPOTHESIS, + ), + RAGPerturbationKind.SEMANTIC_QUERY_PARAPHRASE: ( + "robustness", + RAGPerturbationDirection.INVARIANT, + RAGPerturbationConstructBasis.MODEL_DESIGN_HYPOTHESIS, + ), + RAGPerturbationKind.STYLE_ONLY_REWRITE: ( + "robustness", + RAGPerturbationDirection.INVARIANT, + RAGPerturbationConstructBasis.MODEL_DESIGN_HYPOTHESIS, + ), + RAGPerturbationKind.UNANSWERABLE_QUERY: ( + "answerability_and_abstention", + RAGPerturbationDirection.INCREASE, + RAGPerturbationConstructBasis.MODEL_DESIGN_HYPOTHESIS, + ), + } +) + +_RESPONSE_PERTURBATIONS = frozenset( + { + RAGPerturbationKind.UNSUPPORTED_CLAIM, + RAGPerturbationKind.EXPLICIT_CONTRADICTION, + RAGPerturbationKind.CITATION_TARGET_SWAP, + RAGPerturbationKind.STYLE_ONLY_REWRITE, + } +) +_RETRIEVAL_PERTURBATIONS = frozenset( + { + RAGPerturbationKind.IRRELEVANT_CONTEXT, + RAGPerturbationKind.REQUIRED_EVIDENCE_REMOVAL, + } +) +_QUERY_PERTURBATIONS = frozenset( + { + RAGPerturbationKind.SEMANTIC_QUERY_PARAPHRASE, + RAGPerturbationKind.UNANSWERABLE_QUERY, + } +) +_COMMON_REQUEST_FIELDS = ( + "assessment_fingerprint", + "rubric_id", + "rubric_fingerprint", + "construct_id", + "response_format", + "granularity", + "respondent_id", + "task_id", + "task_family_id", + "occasion_id", + "criterion_ids", + "allowed_scores", + "schema_version", +) +_RAG_RELATION_METADATA_KEYS = frozenset( + { + "rag_evidence_regime", + "rag_candidate_visibility", + "rag_system_configuration_id", + "rag_system_configuration_fingerprint", + "rag_retrieval_run_fingerprint", + "rag_query_revision_fingerprint", + } +) +_RELATION_AXIS_METADATA_KEYS = frozenset( + {"rag_retrieval_run_fingerprint", "rag_query_revision_fingerprint"} +) +_RAG_PERTURBATION_ANCHOR_TOKEN = object() + + +@dataclass(frozen=True) +class RAGPerturbationAnchor(CanonicalContract): + """Verified identity and preregistered direction for one RAG perturbation. + + The factory binds two canonical governed scoring requests and validates the + kind-specific changed axis before this source-free artifact is constructed. + ``construct_basis`` distinguishes constructs aligned with the cited RAGAS / + ARES dimensions from package-owned model-design hypotheses. Every expected + direction remains a preregistered hypothesis rather than an observed effect. + """ + + anchor_id: str + baseline_request_fingerprint: str + perturbed_request_fingerprint: str + perturbation_specification_fingerprint: str + perturbation_run_fingerprint: str + perturbation_kind: RAGPerturbationKind + expected_construct: str = field(init=False) + expected_direction: RAGPerturbationDirection = field(init=False) + construct_basis: RAGPerturbationConstructBasis = field(init=False) + _anchor_token: InitVar[object | None] = None + + def __post_init__(self, _anchor_token: object | None) -> None: + """Reject direct construction and derive immutable perturbation semantics.""" + if _anchor_token is not _RAG_PERTURBATION_ANCHOR_TOKEN: + raise assessment_error( + "unverified_rag_perturbation_anchor", + "$", + "use build_rag_perturbation_anchor", + ) + normalized_anchor_id = descriptive_identifier( + self.anchor_id, + "anchor_id", + "$.anchor_id", + ) + normalized_baseline = fingerprint( + self.baseline_request_fingerprint, + "baseline_request_fingerprint", + "$.baseline_request_fingerprint", + ) + normalized_perturbed = fingerprint( + self.perturbed_request_fingerprint, + "perturbed_request_fingerprint", + "$.perturbed_request_fingerprint", + ) + normalized_specification = fingerprint( + self.perturbation_specification_fingerprint, + "perturbation_specification_fingerprint", + "$.perturbation_specification_fingerprint", + ) + normalized_run = fingerprint( + self.perturbation_run_fingerprint, + "perturbation_run_fingerprint", + "$.perturbation_run_fingerprint", + ) + normalized_kind = enum_value( + self.perturbation_kind, + RAGPerturbationKind, + "rag_perturbation_kind", + "$.perturbation_kind", + ) + if normalized_baseline == normalized_perturbed: + raise assessment_error( + "identical_rag_perturbation_requests", + "$.perturbed_request_fingerprint", + "baseline and perturbed request fingerprints must differ", + ) + expected_construct, expected_direction, construct_basis = ( + _PERTURBATION_EXPECTATIONS[normalized_kind] + ) + object.__setattr__(self, "anchor_id", normalized_anchor_id) + object.__setattr__( + self, + "baseline_request_fingerprint", + normalized_baseline, + ) + object.__setattr__( + self, + "perturbed_request_fingerprint", + normalized_perturbed, + ) + object.__setattr__( + self, + "perturbation_specification_fingerprint", + normalized_specification, + ) + object.__setattr__( + self, + "perturbation_run_fingerprint", + normalized_run, + ) + object.__setattr__(self, "perturbation_kind", normalized_kind) + object.__setattr__(self, "expected_construct", expected_construct) + object.__setattr__(self, "expected_direction", expected_direction) + object.__setattr__(self, "construct_basis", construct_basis) + + def _content_dict(self) -> dict[str, Any]: + """Return canonical anchor content without the derived content digest.""" + return { + "anchor_id": self.anchor_id, + "baseline_request_fingerprint": self.baseline_request_fingerprint, + "perturbed_request_fingerprint": self.perturbed_request_fingerprint, + "perturbation_specification_fingerprint": ( + self.perturbation_specification_fingerprint + ), + "perturbation_run_fingerprint": self.perturbation_run_fingerprint, + "perturbation_kind": self.perturbation_kind.value, + "expected_construct": self.expected_construct, + "expected_direction": self.expected_direction.value, + "construct_basis": self.construct_basis.value, + } + + @property + def anchor_fingerprint(self) -> str: + """Return SHA-256 over the complete immutable perturbation-anchor content.""" + return artifact_digest(self) + + def to_dict(self) -> dict[str, Any]: + """Return canonical anchor content plus its deterministic fingerprint.""" + return { + **self._content_dict(), + "anchor_fingerprint": self.anchor_fingerprint, + } + + +def _canonical_rag_request(value: Any, name: str) -> ScoringRequest: + """Return one canonical RAG request with complete managed provenance.""" + if not isinstance(value, ScoringRequest): + raise assessment_error( + f"invalid_{name}", + f"$.{name}", + f"{name} must be a ScoringRequest", + ) + metadata = thaw_json_value(value.metadata) + if not isinstance(metadata, dict) or not ( + _RAG_RELATION_METADATA_KEYS.issubset(metadata) + ): + raise assessment_error( + f"invalid_{name}", + f"$.{name}.metadata", + f"{name} must contain complete managed RAG provenance", + ) + query_revision = fingerprint( + metadata["rag_query_revision_fingerprint"], + "rag_query_revision_fingerprint", + f"$.{name}.metadata.rag_query_revision_fingerprint", + ) + if query_revision != value.task_revision_fingerprint: + raise assessment_error( + f"invalid_{name}", + f"$.{name}.metadata.rag_query_revision_fingerprint", + "RAG query revision must replay the shared task revision", + ) + fingerprint( + metadata["rag_system_configuration_fingerprint"], + "rag_system_configuration_fingerprint", + f"$.{name}.metadata.rag_system_configuration_fingerprint", + ) + fingerprint( + metadata["rag_retrieval_run_fingerprint"], + "rag_retrieval_run_fingerprint", + f"$.{name}.metadata.rag_retrieval_run_fingerprint", + ) + descriptive_identifier( + metadata["rag_system_configuration_id"], + "rag_system_configuration_id", + f"$.{name}.metadata.rag_system_configuration_id", + ) + enum_value( + metadata["rag_evidence_regime"], + RAGEvidenceRegime, + "rag_evidence_regime", + f"$.{name}.metadata.rag_evidence_regime", + ) + enum_value( + metadata["rag_candidate_visibility"], + RAGCandidateVisibility, + "rag_candidate_visibility", + f"$.{name}.metadata.rag_candidate_visibility", + ) + return value + + +def _request_metadata(request: ScoringRequest) -> dict[str, Any]: + """Return mutable JSON metadata from one immutable scoring request.""" + metadata = thaw_json_value(request.metadata) + if not isinstance(metadata, dict): # pragma: no cover - sealed request invariant + raise assessment_error( + "invalid_rag_perturbation_request", + "$.metadata", + "RAG request metadata must be a mapping", + ) + return metadata + + +def _raise_relationship_error(message: str) -> None: + """Raise one stable kind-specific relationship failure.""" + raise assessment_error( + "invalid_rag_perturbation_relationship", + "$.perturbed_request", + message, + ) + + +def _validate_common_request_provenance( + baseline: ScoringRequest, + perturbed: ScoringRequest, + baseline_metadata: Mapping[str, Any], + perturbed_metadata: Mapping[str, Any], +) -> None: + """Reject pairs that differ outside controlled RAG treatment axes.""" + if any( + getattr(baseline, field_name) != getattr(perturbed, field_name) + for field_name in _COMMON_REQUEST_FIELDS + ): + raise assessment_error( + "unrelated_rag_perturbation_requests", + "$.perturbed_request", + "requests must share assessment, rubric, system run, task, " + "and occasion provenance", + ) + baseline_invariants = { + key: value + for key, value in baseline_metadata.items() + if key not in _RELATION_AXIS_METADATA_KEYS + } + perturbed_invariants = { + key: value + for key, value in perturbed_metadata.items() + if key not in _RELATION_AXIS_METADATA_KEYS + } + if baseline_invariants != perturbed_invariants: + raise assessment_error( + "unrelated_rag_perturbation_requests", + "$.perturbed_request.metadata", + "requests must share evidence, visibility, system, policy, " + "and split provenance", + ) + + +def _validate_perturbation_relationship( + baseline: ScoringRequest, + perturbed: ScoringRequest, + kind: RAGPerturbationKind, +) -> None: + """Validate the exact governed request relation permitted for ``kind``.""" + baseline_metadata = _request_metadata(baseline) + perturbed_metadata = _request_metadata(perturbed) + _validate_common_request_provenance( + baseline, + perturbed, + baseline_metadata, + perturbed_metadata, + ) + if baseline.request_id == perturbed.request_id: + _raise_relationship_error( + "baseline and perturbed request identifiers must differ" + ) + + same_query = ( + baseline.task_revision_fingerprint == perturbed.task_revision_fingerprint + ) + same_retrieval = ( + baseline_metadata["rag_retrieval_run_fingerprint"] + == perturbed_metadata["rag_retrieval_run_fingerprint"] + ) + same_response = ( + baseline.response_content_fingerprint + == perturbed.response_content_fingerprint + ) + + if kind in _RESPONSE_PERTURBATIONS: + if not same_query or not same_retrieval or same_response: + _raise_relationship_error( + "response perturbations must change only the governed response artifact" + ) + if baseline.response_id == perturbed.response_id: + _raise_relationship_error( + "a changed response artifact requires a distinct response identifier" + ) + return + + if kind in _RETRIEVAL_PERTURBATIONS: + if not same_query or same_retrieval or not same_response: + _raise_relationship_error( + "retrieval perturbations must change only retrieval provenance" + ) + if ( + baseline.response_id != perturbed.response_id + or baseline.response_character_count != perturbed.response_character_count + or baseline.response_unit_count != perturbed.response_unit_count + ): + _raise_relationship_error( + "retrieval perturbations must preserve the exact governed " + "response artifact" + ) + return + + if kind in _QUERY_PERTURBATIONS: + if same_query: + _raise_relationship_error( + "query perturbations require a distinct governed query revision" + ) + return + + raise AssertionError(f"unhandled RAG perturbation kind: {kind}") + + +def build_rag_perturbation_anchor( + *, + anchor_id: str, + baseline_request: ScoringRequest, + perturbed_request: ScoringRequest, + perturbation_specification_fingerprint: str, + perturbation_run_fingerprint: str, + perturbation_kind: RAGPerturbationKind | str, +) -> RAGPerturbationAnchor: + """Build one verified, source-free controlled RAG perturbation anchor. + + Both inputs must be canonical governed requests. The factory validates + shared provenance and the kind-specific changed axis, then binds the pair + to an externally governed perturbation specification and execution run. It + stores only content-addressed identities, accepts no raw query, context, + response, or source text, and performs no scoring or truth adjudication. + """ + normalized_baseline = _canonical_rag_request( + baseline_request, + "baseline_request", + ) + normalized_perturbed = _canonical_rag_request( + perturbed_request, + "perturbed_request", + ) + normalized_specification = fingerprint( + perturbation_specification_fingerprint, + "perturbation_specification_fingerprint", + "$.perturbation_specification_fingerprint", + ) + normalized_run = fingerprint( + perturbation_run_fingerprint, + "perturbation_run_fingerprint", + "$.perturbation_run_fingerprint", + ) + normalized_kind = enum_value( + perturbation_kind, + RAGPerturbationKind, + "rag_perturbation_kind", + "$.perturbation_kind", + ) + if ( + normalized_baseline.request_fingerprint + == normalized_perturbed.request_fingerprint + ): + raise assessment_error( + "identical_rag_perturbation_requests", + "$.perturbed_request", + "baseline and perturbed requests must differ", + ) + _validate_perturbation_relationship( + normalized_baseline, + normalized_perturbed, + normalized_kind, + ) + return RAGPerturbationAnchor( + anchor_id=anchor_id, + baseline_request_fingerprint=normalized_baseline.request_fingerprint, + perturbed_request_fingerprint=normalized_perturbed.request_fingerprint, + perturbation_specification_fingerprint=normalized_specification, + perturbation_run_fingerprint=normalized_run, + perturbation_kind=normalized_kind, + _anchor_token=_RAG_PERTURBATION_ANCHOR_TOKEN, + ) + + _MANAGED_METADATA_KEYS = frozenset( { "rag_evidence_regime", @@ -220,5 +721,10 @@ def build_rag_scoring_request( __all__ = [ "RAGCandidateVisibility", "RAGEvidenceRegime", + "RAGPerturbationAnchor", + "RAGPerturbationConstructBasis", + "RAGPerturbationDirection", + "RAGPerturbationKind", + "build_rag_perturbation_anchor", "build_rag_scoring_request", ] diff --git a/tests/test_scoring_rag_perturbation_anchors.py b/tests/test_scoring_rag_perturbation_anchors.py new file mode 100644 index 000000000..4a43019a7 --- /dev/null +++ b/tests/test_scoring_rag_perturbation_anchors.py @@ -0,0 +1,231 @@ +"""Canonical identity contracts for governed RAG perturbation anchors.""" + +from __future__ import annotations + +import hashlib +import inspect +from pathlib import Path +import runpy +from typing import Any + +import pytest + +from fast_mlsirm.scoring import AssessmentSpecError, ScoringRequest +from fast_mlsirm.scoring.rag import ( + RAGPerturbationAnchor, + RAGPerturbationDirection, + RAGPerturbationKind, + build_rag_perturbation_anchor, + build_rag_scoring_request, +) + +_FIXTURES = runpy.run_path( + str(Path(__file__).with_name("scoring_execution_fixtures.py")) +) +assessment = _FIXTURES["assessment"] +rubric = _FIXTURES["rubric"] + +QUERY_FP = hashlib.sha256(b"rag-anchor-query-revision").hexdigest() +SYSTEM_FP = hashlib.sha256(b"rag-anchor-system-configuration").hexdigest() +RETRIEVAL_FP = hashlib.sha256(b"rag-anchor-retrieval-run").hexdigest() +RESPONSE_FP = hashlib.sha256(b"rag-anchor-response-content").hexdigest() +SECOND_RESPONSE_FP = hashlib.sha256(b"rag-anchor-response-content-second").hexdigest() +THIRD_RESPONSE_FP = hashlib.sha256(b"rag-anchor-response-content-third").hexdigest() +SPECIFICATION_FP = hashlib.sha256(b"rag-anchor-specification").hexdigest() +RUN_FP = hashlib.sha256(b"rag-anchor-run").hexdigest() + + +def _request(**overrides: Any) -> ScoringRequest: + """Build one deterministic canonical RAG scoring request.""" + values: dict[str, Any] = { + "request_id": "rag_anchor_baseline_request", + "assessment": assessment(), + "rubric": rubric(), + "query_id": "refund_policy_query", + "query_revision_fingerprint": QUERY_FP, + "query_testlet_id": "evidence_review", + "evidence_regime": "retrieved_context", + "candidate_visibility": "candidate_blind", + "system_configuration_id": "retrieval_stack_a", + "system_configuration_fingerprint": SYSTEM_FP, + "system_run_id": "retrieval_stack_a_run_001", + "response_id": "generated_response_001", + "retrieval_run_fingerprint": RETRIEVAL_FP, + "response_content_fingerprint": RESPONSE_FP, + "occasion_id": "evaluation_wave_001", + "criterion_ids": ("grounded_generation", "answer_relevance"), + "response_character_count": 412, + "response_unit_count": 7, + "metadata": {"evaluation_split": "offline_holdout"}, + } + values.update(overrides) + return build_rag_scoring_request(**values) + + +def _anchor( + *, + kind: RAGPerturbationKind | str, + perturbed: ScoringRequest, + baseline: ScoringRequest | None = None, + anchor_id: str | None = None, +) -> RAGPerturbationAnchor: + """Build one deterministic response-axis anchor.""" + normalized_kind = RAGPerturbationKind(kind) + return build_rag_perturbation_anchor( + anchor_id=anchor_id or f"{normalized_kind.value}_anchor", + baseline_request=_request() if baseline is None else baseline, + perturbed_request=perturbed, + perturbation_specification_fingerprint=SPECIFICATION_FP, + perturbation_run_fingerprint=RUN_FP, + perturbation_kind=normalized_kind, + ) + + +@pytest.mark.parametrize( + ("kind", "construct", "direction"), + ( + ("unsupported_claim", "grounded_generation", "decrease"), + ("explicit_contradiction", "grounded_generation", "decrease"), + ("citation_target_swap", "citation_attribution", "decrease"), + ("style_only_rewrite", "robustness", "invariant"), + ), +) +def test_response_perturbation_has_one_construct_specific_expected_direction( + kind: str, + construct: str, + direction: str, +) -> None: + """Response perturbations preserve preregistered construct directions.""" + anchor = _anchor( + kind=kind, + perturbed=_request( + request_id=f"{kind}_request", + response_id=f"{kind}_response", + response_content_fingerprint=SECOND_RESPONSE_FP, + ), + ) + + assert anchor.perturbation_kind is RAGPerturbationKind(kind) + assert anchor.expected_construct == construct + assert anchor.expected_direction is RAGPerturbationDirection(direction) + + +def test_perturbation_enums_are_finite_and_explicit() -> None: + """The public contract must not accept arbitrary perturbation semantics.""" + assert {member.value for member in RAGPerturbationKind} == { + "unsupported_claim", + "explicit_contradiction", + "irrelevant_context", + "required_evidence_removal", + "citation_target_swap", + "semantic_query_paraphrase", + "style_only_rewrite", + "unanswerable_query", + } + assert {member.value for member in RAGPerturbationDirection} == { + "decrease", + "invariant", + "increase", + } + + +def test_anchor_is_content_addressed_and_identity_bearing() -> None: + """The request pair and transformation provenance participate in identity.""" + baseline = _request() + perturbed = _request( + request_id="unsupported_claim_request", + response_id="generated_response_002", + response_content_fingerprint=SECOND_RESPONSE_FP, + ) + first = _anchor( + kind="unsupported_claim", + baseline=baseline, + perturbed=perturbed, + ) + replay = _anchor( + kind="unsupported_claim", + baseline=baseline, + perturbed=perturbed, + ) + second = _anchor( + kind="unsupported_claim", + baseline=baseline, + perturbed=_request( + request_id="unsupported_claim_second_request", + response_id="generated_response_003", + response_content_fingerprint=THIRD_RESPONSE_FP, + ), + ) + + assert first.anchor_fingerprint == replay.anchor_fingerprint + assert first.anchor_fingerprint != second.anchor_fingerprint + payload = first.to_dict() + assert payload["baseline_request_fingerprint"] == baseline.request_fingerprint + assert payload["perturbed_request_fingerprint"] == perturbed.request_fingerprint + assert payload["perturbation_specification_fingerprint"] == SPECIFICATION_FP + assert payload["perturbation_run_fingerprint"] == RUN_FP + assert payload["perturbation_kind"] == "unsupported_claim" + assert payload["expected_construct"] == "grounded_generation" + assert payload["expected_direction"] == "decrease" + assert payload["construct_basis"] == "literature_aligned_construct" + + +def test_anchor_requires_distinct_baseline_and_perturbed_requests() -> None: + """A no-op request pair cannot masquerade as perturbation evidence.""" + request = _request() + with pytest.raises(AssessmentSpecError) as caught: + _anchor( + kind="style_only_rewrite", + baseline=request, + perturbed=request, + anchor_id="no_op_anchor", + ) + assert caught.value.code == "identical_rag_perturbation_requests" + + +def test_anchor_rejects_unknown_semantics() -> None: + """Unknown perturbation semantics fail closed.""" + with pytest.raises(AssessmentSpecError) as caught: + build_rag_perturbation_anchor( + anchor_id="unknown_anchor", + baseline_request=_request(), + perturbed_request=_request( + request_id="unknown_perturbed_request", + response_id="unknown_response", + response_content_fingerprint=SECOND_RESPONSE_FP, + ), + perturbation_specification_fingerprint=SPECIFICATION_FP, + perturbation_run_fingerprint=RUN_FP, + perturbation_kind="improve_everything", + ) + assert caught.value.code == "invalid_rag_perturbation_kind" + + +def test_anchor_cannot_be_constructed_without_relationship_validation() -> None: + """Direct construction cannot bypass canonical pair validation.""" + with pytest.raises(AssessmentSpecError) as caught: + RAGPerturbationAnchor( + anchor_id="unverified_anchor", + baseline_request_fingerprint=RESPONSE_FP, + perturbed_request_fingerprint=SECOND_RESPONSE_FP, + perturbation_specification_fingerprint=SPECIFICATION_FP, + perturbation_run_fingerprint=RUN_FP, + perturbation_kind=RAGPerturbationKind.UNSUPPORTED_CLAIM, + ) + assert caught.value.code == "unverified_rag_perturbation_anchor" + + +def test_anchor_contract_cannot_store_raw_query_context_or_answer_text() -> None: + """Perturbation evidence is source-free identity metadata, not content.""" + parameters = inspect.signature(build_rag_perturbation_anchor).parameters + assert not set(parameters).intersection( + { + "query_text", + "question_text", + "context_text", + "retrieved_text", + "answer_text", + "response_text", + "source_text", + } + ) diff --git a/tests/test_scoring_rag_perturbation_relationships.py b/tests/test_scoring_rag_perturbation_relationships.py new file mode 100644 index 000000000..826d5ee0b --- /dev/null +++ b/tests/test_scoring_rag_perturbation_relationships.py @@ -0,0 +1,270 @@ +"""Relationship contracts for governed RAG perturbation anchors.""" + +from __future__ import annotations + +import hashlib +import inspect +from pathlib import Path +import runpy +from typing import Any + +import pytest + +from fast_mlsirm.scoring import AssessmentSpecError, ScoringRequest +from fast_mlsirm.scoring.rag import ( + RAGPerturbationConstructBasis, + RAGPerturbationKind, + build_rag_perturbation_anchor, + build_rag_scoring_request, +) + +_FIXTURES = runpy.run_path( + str(Path(__file__).with_name("scoring_execution_fixtures.py")) +) +assessment = _FIXTURES["assessment"] +rubric = _FIXTURES["rubric"] + +QUERY_FP = hashlib.sha256(b"rag-anchor-query").hexdigest() +SECOND_QUERY_FP = hashlib.sha256(b"rag-anchor-query-second").hexdigest() +SYSTEM_FP = hashlib.sha256(b"rag-anchor-system").hexdigest() +OTHER_SYSTEM_FP = hashlib.sha256(b"rag-anchor-other-system").hexdigest() +RETRIEVAL_FP = hashlib.sha256(b"rag-anchor-retrieval").hexdigest() +SECOND_RETRIEVAL_FP = hashlib.sha256(b"rag-anchor-retrieval-second").hexdigest() +RESPONSE_FP = hashlib.sha256(b"rag-anchor-response").hexdigest() +SECOND_RESPONSE_FP = hashlib.sha256(b"rag-anchor-response-second").hexdigest() +SPECIFICATION_FP = hashlib.sha256(b"rag-anchor-specification").hexdigest() +RUN_FP = hashlib.sha256(b"rag-anchor-run").hexdigest() + + +def _request(**overrides: Any) -> ScoringRequest: + """Build one deterministic canonical RAG scoring request.""" + values: dict[str, Any] = { + "request_id": "rag_anchor_baseline_request", + "assessment": assessment(), + "rubric": rubric(), + "query_id": "refund_policy_query", + "query_revision_fingerprint": QUERY_FP, + "query_testlet_id": "evidence_review", + "evidence_regime": "retrieved_context", + "candidate_visibility": "candidate_blind", + "system_configuration_id": "retrieval_stack_a", + "system_configuration_fingerprint": SYSTEM_FP, + "system_run_id": "retrieval_stack_a_run_001", + "response_id": "generated_response_001", + "retrieval_run_fingerprint": RETRIEVAL_FP, + "response_content_fingerprint": RESPONSE_FP, + "occasion_id": "evaluation_wave_001", + "criterion_ids": ("grounded_generation", "answer_relevance"), + "response_character_count": 412, + "response_unit_count": 7, + "metadata": {"evaluation_split": "offline_holdout"}, + } + values.update(overrides) + return build_rag_scoring_request(**values) + + +def _anchor( + *, + kind: RAGPerturbationKind | str, + perturbed: ScoringRequest, + baseline: ScoringRequest | None = None, +): + """Build one anchor with deterministic transformation provenance.""" + return build_rag_perturbation_anchor( + anchor_id=f"{RAGPerturbationKind(kind).value}_anchor", + baseline_request=_request() if baseline is None else baseline, + perturbed_request=perturbed, + perturbation_specification_fingerprint=SPECIFICATION_FP, + perturbation_run_fingerprint=RUN_FP, + perturbation_kind=kind, + ) + + +def _assert_relationship_error(kind: str, perturbed: ScoringRequest) -> None: + """Assert one stable invalid-relationship error.""" + with pytest.raises(AssessmentSpecError) as caught: + _anchor(kind=kind, perturbed=perturbed) + assert caught.value.code == "invalid_rag_perturbation_relationship" + + +def test_anchor_builder_accepts_canonical_requests_not_opaque_fingerprints() -> None: + """The boundary must inspect the governed pair before storing identities.""" + parameters = inspect.signature(build_rag_perturbation_anchor).parameters + assert {"baseline_request", "perturbed_request"}.issubset(parameters) + assert { + "perturbation_specification_fingerprint", + "perturbation_run_fingerprint", + }.issubset(parameters) + assert "baseline_request_fingerprint" not in parameters + assert "perturbed_request_fingerprint" not in parameters + + +def test_response_perturbation_binds_only_a_changed_response_artifact() -> None: + """Response kinds preserve query/retrieval provenance and change response.""" + baseline = _request() + perturbed = _request( + request_id="rag_anchor_response_perturbed_request", + response_id="generated_response_002", + response_content_fingerprint=SECOND_RESPONSE_FP, + response_character_count=430, + response_unit_count=8, + ) + + anchor = _anchor( + kind=RAGPerturbationKind.UNSUPPORTED_CLAIM, + baseline=baseline, + perturbed=perturbed, + ) + + assert anchor.baseline_request_fingerprint == baseline.request_fingerprint + assert anchor.perturbed_request_fingerprint == perturbed.request_fingerprint + assert anchor.perturbation_specification_fingerprint == SPECIFICATION_FP + assert anchor.perturbation_run_fingerprint == RUN_FP + + +def test_retrieval_perturbation_binds_only_a_changed_retrieval_artifact() -> None: + """Retrieval kinds preserve query/response while changing retrieval.""" + baseline = _request() + perturbed = _request( + request_id="rag_anchor_retrieval_perturbed_request", + retrieval_run_fingerprint=SECOND_RETRIEVAL_FP, + ) + + anchor = _anchor( + kind=RAGPerturbationKind.IRRELEVANT_CONTEXT, + baseline=baseline, + perturbed=perturbed, + ) + + assert anchor.baseline_request_fingerprint == baseline.request_fingerprint + assert anchor.perturbed_request_fingerprint == perturbed.request_fingerprint + + +def test_query_perturbation_requires_a_changed_query_revision() -> None: + """Query kinds preserve task identity while changing its revision.""" + baseline = _request() + perturbed = _request( + request_id="rag_anchor_query_perturbed_request", + query_revision_fingerprint=SECOND_QUERY_FP, + retrieval_run_fingerprint=SECOND_RETRIEVAL_FP, + response_id="generated_response_002", + response_content_fingerprint=SECOND_RESPONSE_FP, + response_character_count=215, + response_unit_count=4, + ) + + anchor = _anchor( + kind=RAGPerturbationKind.SEMANTIC_QUERY_PARAPHRASE, + baseline=baseline, + perturbed=perturbed, + ) + + assert anchor.baseline_request_fingerprint == baseline.request_fingerprint + assert anchor.perturbed_request_fingerprint == perturbed.request_fingerprint + + +def test_unrelated_system_configuration_pair_fails_closed() -> None: + """A different evaluated system cannot masquerade as one perturbation.""" + with pytest.raises(AssessmentSpecError) as caught: + _anchor( + kind="unsupported_claim", + perturbed=_request( + request_id="rag_anchor_other_system_request", + system_configuration_id="retrieval_stack_b", + system_configuration_fingerprint=OTHER_SYSTEM_FP, + response_id="generated_response_002", + response_content_fingerprint=SECOND_RESPONSE_FP, + ), + ) + assert caught.value.code == "unrelated_rag_perturbation_requests" + + +def test_response_kind_rejects_a_simultaneous_retrieval_change() -> None: + """A response perturbation cannot include a second changed treatment.""" + _assert_relationship_error( + "unsupported_claim", + _request( + request_id="rag_anchor_mixed_response_request", + response_id="generated_response_002", + response_content_fingerprint=SECOND_RESPONSE_FP, + retrieval_run_fingerprint=SECOND_RETRIEVAL_FP, + ), + ) + + +def test_retrieval_kind_rejects_a_simultaneous_response_change() -> None: + """A retrieval perturbation must keep the response artifact fixed.""" + _assert_relationship_error( + "irrelevant_context", + _request( + request_id="rag_anchor_mixed_retrieval_request", + retrieval_run_fingerprint=SECOND_RETRIEVAL_FP, + response_id="generated_response_002", + response_content_fingerprint=SECOND_RESPONSE_FP, + ), + ) + + +def test_query_kind_rejects_an_unchanged_query_revision() -> None: + """Changing only the response is not evidence of a query perturbation.""" + _assert_relationship_error( + "semantic_query_paraphrase", + _request( + request_id="rag_anchor_unchanged_query_request", + response_id="generated_response_002", + response_content_fingerprint=SECOND_RESPONSE_FP, + ), + ) + + +def test_construct_basis_distinguishes_sources_from_model_design() -> None: + """Unsupported mappings remain explicit model-design hypotheses.""" + assert RAGPerturbationConstructBasis.LITERATURE_ALIGNED_CONSTRUCT.value == ( + "literature_aligned_construct" + ) + assert RAGPerturbationConstructBasis.MODEL_DESIGN_HYPOTHESIS.value == ( + "model_design_hypothesis" + ) + + literature_aligned = _anchor( + kind="unsupported_claim", + perturbed=_request( + request_id="rag_anchor_grounded_perturbed_request", + response_id="generated_response_002", + response_content_fingerprint=SECOND_RESPONSE_FP, + ), + ) + model_design = _anchor( + kind="citation_target_swap", + perturbed=_request( + request_id="rag_anchor_citation_perturbed_request", + response_id="generated_response_002", + response_content_fingerprint=SECOND_RESPONSE_FP, + ), + ) + + assert literature_aligned.construct_basis is ( + RAGPerturbationConstructBasis.LITERATURE_ALIGNED_CONSTRUCT + ) + assert model_design.construct_basis is ( + RAGPerturbationConstructBasis.MODEL_DESIGN_HYPOTHESIS + ) + + +def test_relation_provenance_fingerprints_fail_closed() -> None: + """The external controlled-transform protocol and run are content-addressed.""" + perturbed = _request( + request_id="rag_anchor_bad_relation_request", + response_id="generated_response_002", + response_content_fingerprint=SECOND_RESPONSE_FP, + ) + with pytest.raises(AssessmentSpecError) as caught: + build_rag_perturbation_anchor( + anchor_id="bad_relation_anchor", + baseline_request=_request(), + perturbed_request=perturbed, + perturbation_specification_fingerprint="not-a-sha256", + perturbation_run_fingerprint=RUN_FP, + perturbation_kind="unsupported_claim", + ) + assert caught.value.code == "invalid_perturbation_specification_fingerprint"