From f75d3768811f01e12ba4eb22842ddc88e98a2fa4 Mon Sep 17 00:00:00 2001 From: Seongho Bae Date: Wed, 12 Aug 2026 19:03:19 +0900 Subject: [PATCH 1/7] rebuild: #798 unique product delta on current main --- CHANGELOG.md | 12 ++ .../777-rag-perturbation-anchors.md | 6 + docs/doctoring/rag_scoring_request_privacy.md | 42 ++++- python/fast_mlsirm/scoring/rag.py | 177 +++++++++++++++++- .../test_scoring_rag_perturbation_anchors.py | 149 +++++++++++++++ 5 files changed, 380 insertions(+), 6 deletions(-) create mode 100644 docs/changelog.d/777-rag-perturbation-anchors.md create mode 100644 tests/test_scoring_rag_perturbation_anchors.py diff --git a/CHANGELOG.md b/CHANGELOG.md index e1bdeeeb7..32064d04c 100644 --- a/CHANGELOG.md +++ b/CHANGELOG.md @@ -69,6 +69,11 @@ decision owner instead of hard-coding Williamson high-stakes cutoffs only in Python. +#### Governed RAG perturbation anchors + +- Added source-free, content-addressed reference-free RAG perturbation anchors with finite preregistered construct/direction semantics for unsupported claims, contradictions, irrelevant context, required-evidence removal, citation swaps, semantic paraphrases, style-only rewrites, and unanswerable queries. +- Require distinct governed baseline and perturbed request fingerprints and fail closed on unknown perturbation semantics or malformed identities. Expected directions are validation hypotheses, not claims that an observed system actually changed or that an evaluator is ground truth. + #### Architecture baseline documentation - Root `ARCHITECTURE.md` describing layered Rust-primary numeric core, Python @@ -552,6 +557,13 @@ vocabulary and mark parallel-analysis control bounds and essay-report native dark-mode accents as ancestral after their integration. +#### Retire competing hourly review-repair caller + +- Remove the repository-local hourly review-repair GitHub Actions caller so only + the organization single-writer control plane schedules mutation loops, matching + ADR-0013 continuous-execution governance after failed startup evidence for the + local caller. + #### Multilevel hostile numeric callback rejection - Multilevel membership weights and AR(1) coefficients now admit only exact diff --git a/docs/changelog.d/777-rag-perturbation-anchors.md b/docs/changelog.d/777-rag-perturbation-anchors.md new file mode 100644 index 000000000..395a6ffd5 --- /dev/null +++ b/docs/changelog.d/777-rag-perturbation-anchors.md @@ -0,0 +1,6 @@ +# Governed RAG perturbation anchors + +## Added + +- Added source-free, content-addressed reference-free RAG perturbation anchors with finite preregistered construct/direction semantics for unsupported claims, contradictions, irrelevant context, required-evidence removal, citation swaps, semantic paraphrases, style-only rewrites, and unanswerable queries. +- Require distinct governed baseline and perturbed request fingerprints and fail closed on unknown perturbation semantics or malformed identities. Expected directions are validation hypotheses, not claims that an observed system actually changed or that an evaluator is ground truth. diff --git a/docs/doctoring/rag_scoring_request_privacy.md b/docs/doctoring/rag_scoring_request_privacy.md index 2d315713e..d01e061a8 100644 --- a/docs/doctoring/rag_scoring_request_privacy.md +++ b/docs/doctoring/rag_scoring_request_privacy.md @@ -1,15 +1,47 @@ -# Governed RAG scoring request privacy +# Governed RAG scoring request and perturbation-anchor doctoring -## Standards +## Standards and research basis American Educational Research Association, American Psychological Association, & National Council on Measurement in Education. (2014). *Standards for educational and psychological testing*. American Educational Research Association. +Es, S., James, J., Espinosa-Anke, L., & Schockaert, S. (2024). RAGAS: Automated evaluation of retrieval augmented generation. In *Proceedings of the 18th Conference of the European Chapter of the Association for Computational Linguistics: System Demonstrations* (pp. 150–158). Association for Computational Linguistics. https://doi.org/10.18653/v1/2024.eacl-demo.16 + National Institute of Standards and Technology. (2020). *Security and privacy controls for information systems and organizations* (NIST SP 800-53 Rev. 5). https://doi.org/10.6028/NIST.SP.800-53r5 -## Rationale +Saad-Falcon, J., Khattab, O., Potts, C., & Zaharia, M. (2024). ARES: An automated evaluation framework for retrieval-augmented generation systems. In *Proceedings of the 2024 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies (Volume 1: Long Papers)* (pp. 338–354). Association for Computational Linguistics. https://doi.org/10.18653/v1/2024.naacl-long.20 + +Williamson, D. M., Xi, X., & Breyer, F. J. (2012). A framework for evaluation and use of automated scoring. *Educational Measurement: Issues and Practice, 31*(1), 2–13. https://doi.org/10.1111/j.1745-3992.2011.00223.x + +## Privacy and provenance rationale + +Reference-free RAG evaluation records provenance identities (system configuration, retrieval run, query revision fingerprints) without embedding raw query, context, answer, or source text. Managed identifiers must reject free-form content so the canonical scoring artifact cannot become a side channel. The host application may retain source material only under its own purpose-bound security, privacy, retention, and authorization controls. + +## Controlled perturbation anchors + +`RAGPerturbationAnchor` preregisters a finite construct-specific expectation against two distinct governed scoring-request fingerprints. The package currently represents these hypotheses: -Reference-free RAG evaluation records provenance identities (system configuration, retrieval run, query revision fingerprints) without embedding raw query, context, or answer text. Managed identifiers must reject free-form content so the canonical scoring artifact cannot become a side channel. +- unsupported claim or explicit contradiction -> `grounded_generation` decreases; +- irrelevant context -> `retrieval_relevance` decreases; +- required-evidence removal -> `coverage_or_completeness_proxy` decreases; +- citation-target swap -> `citation_attribution` decreases; +- semantic query paraphrase or style-only rewrite -> `robustness` remains approximately invariant; and +- unanswerable query -> `answerability_and_abstention` increases. + +These are controlled-validation expectations, not observed scores, statistical tests, causal effects, ground-truth labels, or evidence that a particular judge can identify world correctness. Context-groundedness remains conditional on the declared evidence regime. A successful anchor regression therefore demonstrates contract identity and preregistered direction semantics only; empirical validation still requires governed observations, calibrated evaluator/rater evidence, subgroup/DIF and drift analysis where applicable, and recovery or human-audit evidence commensurate with the intended interpretation. + +## Numerical and ownership boundary + +The anchor layer performs no likelihood, calibration, thresholding, uncertainty estimation, scoring aggregation, retrieval, provider call, or truth adjudication. New production psychometric/numerical arithmetic remains Rust-owned with model-appropriate convergence and recovery evidence. Human, rule, AI, and LLM evaluator outputs remain fallible observations rather than truth by identity. ## Implementation -`python/fast_mlsirm/scoring/rag.py` — `build_rag_scoring_request` validates `system_configuration_id` via `descriptive_identifier` and allowlists caller metadata. +`python/fast_mlsirm/scoring/rag.py`: + +- `build_rag_scoring_request` validates `system_configuration_id` via `descriptive_identifier` and allowlists caller metadata; +- `RAGPerturbationKind` and `RAGPerturbationDirection` close the perturbation vocabulary; +- `RAGPerturbationAnchor` derives package-owned construct/direction semantics, rejects no-op baseline/perturbed request pairs, and content-addresses immutable anchor content; and +- `build_rag_perturbation_anchor` exposes only governed identities and perturbation kind, with no raw-content parameters. + +`tests/test_scoring_rag_perturbation_anchors.py` binds the finite mapping, deterministic fingerprints, fail-closed malformed/unknown inputs, distinct request identities, and source-free public signature. + +This bounded contract advances issue #607 stage-2 validation infrastructure. It does not by itself establish RAG system validity, a scalar quality score, evaluator interchangeability, retrieval recall, world correctness, calibration, DIF invariance, or release readiness. diff --git a/python/fast_mlsirm/scoring/rag.py b/python/fast_mlsirm/scoring/rag.py index 8194d9ac1..ba94da8f7 100644 --- a/python/fast_mlsirm/scoring/rag.py +++ b/python/fast_mlsirm/scoring/rag.py @@ -8,11 +8,14 @@ from __future__ import annotations from collections.abc import Iterable, Mapping +from dataclasses import dataclass, field from enum import Enum +from types import MappingProxyType from typing import Any -from ._contract_safety import enum_value, freeze_metadata +from ._contract_safety import artifact_digest, enum_value, freeze_metadata from ._validation import ( + CanonicalContract, assessment_error, descriptive_identifier, fingerprint, @@ -41,6 +44,174 @@ class RAGCandidateVisibility(str, Enum): CANDIDATE_VISIBLE_CROSSFIT = "candidate_visible_crossfit" +class RAGPerturbationKind(str, Enum): + """Finite controlled perturbations with preregistered construct semantics.""" + + UNSUPPORTED_CLAIM = "unsupported_claim" + EXPLICIT_CONTRADICTION = "explicit_contradiction" + IRRELEVANT_CONTEXT = "irrelevant_context" + REQUIRED_EVIDENCE_REMOVAL = "required_evidence_removal" + CITATION_TARGET_SWAP = "citation_target_swap" + SEMANTIC_QUERY_PARAPHRASE = "semantic_query_paraphrase" + STYLE_ONLY_REWRITE = "style_only_rewrite" + UNANSWERABLE_QUERY = "unanswerable_query" + + +class RAGPerturbationDirection(str, Enum): + """Expected direction for one controlled perturbation construct.""" + + DECREASE = "decrease" + INVARIANT = "invariant" + INCREASE = "increase" + + +_PERTURBATION_EXPECTATIONS = MappingProxyType( + { + RAGPerturbationKind.UNSUPPORTED_CLAIM: ( + "grounded_generation", + RAGPerturbationDirection.DECREASE, + ), + RAGPerturbationKind.EXPLICIT_CONTRADICTION: ( + "grounded_generation", + RAGPerturbationDirection.DECREASE, + ), + RAGPerturbationKind.IRRELEVANT_CONTEXT: ( + "retrieval_relevance", + RAGPerturbationDirection.DECREASE, + ), + RAGPerturbationKind.REQUIRED_EVIDENCE_REMOVAL: ( + "coverage_or_completeness_proxy", + RAGPerturbationDirection.DECREASE, + ), + RAGPerturbationKind.CITATION_TARGET_SWAP: ( + "citation_attribution", + RAGPerturbationDirection.DECREASE, + ), + RAGPerturbationKind.SEMANTIC_QUERY_PARAPHRASE: ( + "robustness", + RAGPerturbationDirection.INVARIANT, + ), + RAGPerturbationKind.STYLE_ONLY_REWRITE: ( + "robustness", + RAGPerturbationDirection.INVARIANT, + ), + RAGPerturbationKind.UNANSWERABLE_QUERY: ( + "answerability_and_abstention", + RAGPerturbationDirection.INCREASE, + ), + } +) + + +@dataclass(frozen=True) +class RAGPerturbationAnchor(CanonicalContract): + """Source-free identity and expected direction for a controlled RAG perturbation. + + The anchor binds two distinct governed scoring-request fingerprints to one + finite perturbation kind. ``expected_construct`` and ``expected_direction`` + are package-derived preregistration metadata; they do not assert that an + observed system response actually moved in the expected direction. + """ + + anchor_id: str + baseline_request_fingerprint: str + perturbed_request_fingerprint: str + perturbation_kind: RAGPerturbationKind + expected_construct: str = field(init=False) + expected_direction: RAGPerturbationDirection = field(init=False) + + def __post_init__(self) -> None: + """Normalize identity fields and derive immutable perturbation semantics.""" + normalized_anchor_id = descriptive_identifier( + self.anchor_id, + "anchor_id", + "$.anchor_id", + ) + normalized_baseline = fingerprint( + self.baseline_request_fingerprint, + "baseline_request_fingerprint", + "$.baseline_request_fingerprint", + ) + normalized_perturbed = fingerprint( + self.perturbed_request_fingerprint, + "perturbed_request_fingerprint", + "$.perturbed_request_fingerprint", + ) + normalized_kind = enum_value( + self.perturbation_kind, + RAGPerturbationKind, + "rag_perturbation_kind", + "$.perturbation_kind", + ) + if normalized_baseline == normalized_perturbed: + raise assessment_error( + "identical_rag_perturbation_requests", + "$.perturbed_request_fingerprint", + "baseline and perturbed request fingerprints must differ", + ) + expected_construct, expected_direction = _PERTURBATION_EXPECTATIONS[ + normalized_kind + ] + object.__setattr__(self, "anchor_id", normalized_anchor_id) + object.__setattr__( + self, + "baseline_request_fingerprint", + normalized_baseline, + ) + object.__setattr__( + self, + "perturbed_request_fingerprint", + normalized_perturbed, + ) + object.__setattr__(self, "perturbation_kind", normalized_kind) + object.__setattr__(self, "expected_construct", expected_construct) + object.__setattr__(self, "expected_direction", expected_direction) + + def _content_dict(self) -> dict[str, Any]: + """Return canonical anchor content without the derived content digest.""" + return { + "anchor_id": self.anchor_id, + "baseline_request_fingerprint": self.baseline_request_fingerprint, + "perturbed_request_fingerprint": self.perturbed_request_fingerprint, + "perturbation_kind": self.perturbation_kind.value, + "expected_construct": self.expected_construct, + "expected_direction": self.expected_direction.value, + } + + @property + def anchor_fingerprint(self) -> str: + """Return SHA-256 over the complete immutable perturbation-anchor content.""" + return artifact_digest(self) + + def to_dict(self) -> dict[str, Any]: + """Return canonical anchor content plus its deterministic fingerprint.""" + return { + **self._content_dict(), + "anchor_fingerprint": self.anchor_fingerprint, + } + + +def build_rag_perturbation_anchor( + *, + anchor_id: str, + baseline_request_fingerprint: str, + perturbed_request_fingerprint: str, + perturbation_kind: RAGPerturbationKind | str, +) -> RAGPerturbationAnchor: + """Build one content-addressed controlled RAG perturbation anchor. + + The contract intentionally stores only bounded identities and package-owned + preregistration semantics. It does not accept raw query, context, response, + or source text and performs no scoring, retrieval, or truth adjudication. + """ + return RAGPerturbationAnchor( + anchor_id=anchor_id, + baseline_request_fingerprint=baseline_request_fingerprint, + perturbed_request_fingerprint=perturbed_request_fingerprint, + perturbation_kind=perturbation_kind, + ) + + _MANAGED_METADATA_KEYS = frozenset( { "rag_evidence_regime", @@ -220,5 +391,9 @@ def build_rag_scoring_request( __all__ = [ "RAGCandidateVisibility", "RAGEvidenceRegime", + "RAGPerturbationAnchor", + "RAGPerturbationDirection", + "RAGPerturbationKind", + "build_rag_perturbation_anchor", "build_rag_scoring_request", ] diff --git a/tests/test_scoring_rag_perturbation_anchors.py b/tests/test_scoring_rag_perturbation_anchors.py new file mode 100644 index 000000000..de1b13e57 --- /dev/null +++ b/tests/test_scoring_rag_perturbation_anchors.py @@ -0,0 +1,149 @@ +"""Fail-first contracts for governed reference-free RAG perturbation anchors.""" + +from __future__ import annotations + +import hashlib +import inspect + +import pytest + +from fast_mlsirm.scoring import AssessmentSpecError +from fast_mlsirm.scoring.rag import ( + RAGPerturbationDirection, + RAGPerturbationKind, + build_rag_perturbation_anchor, +) + +BASELINE_FP = hashlib.sha256(b"rag-anchor-baseline-request").hexdigest() +PERTURBED_FP = hashlib.sha256(b"rag-anchor-perturbed-request").hexdigest() +SECOND_FP = hashlib.sha256(b"rag-anchor-second-perturbed-request").hexdigest() + + +@pytest.mark.parametrize( + ("kind", "construct", "direction"), + ( + ("unsupported_claim", "grounded_generation", "decrease"), + ("explicit_contradiction", "grounded_generation", "decrease"), + ("irrelevant_context", "retrieval_relevance", "decrease"), + ("required_evidence_removal", "coverage_or_completeness_proxy", "decrease"), + ("citation_target_swap", "citation_attribution", "decrease"), + ("semantic_query_paraphrase", "robustness", "invariant"), + ("style_only_rewrite", "robustness", "invariant"), + ("unanswerable_query", "answerability_and_abstention", "increase"), + ), +) +def test_perturbation_kind_has_one_construct_specific_expected_direction( + kind: str, + construct: str, + direction: str, +) -> None: + """Known perturbations preserve their preregistered construct direction.""" + anchor = build_rag_perturbation_anchor( + anchor_id=f"{kind}_anchor", + baseline_request_fingerprint=BASELINE_FP, + perturbed_request_fingerprint=PERTURBED_FP, + perturbation_kind=kind, + ) + + assert anchor.perturbation_kind is RAGPerturbationKind(kind) + assert anchor.expected_construct == construct + assert anchor.expected_direction is RAGPerturbationDirection(direction) + + +def test_perturbation_enums_are_finite_and_explicit() -> None: + """The public contract must not accept arbitrary perturbation semantics.""" + assert {member.value for member in RAGPerturbationKind} == { + "unsupported_claim", + "explicit_contradiction", + "irrelevant_context", + "required_evidence_removal", + "citation_target_swap", + "semantic_query_paraphrase", + "style_only_rewrite", + "unanswerable_query", + } + assert {member.value for member in RAGPerturbationDirection} == { + "decrease", + "invariant", + "increase", + } + + +def test_anchor_is_content_addressed_and_identity_bearing() -> None: + """The exact baseline/perturbation pair participates in anchor identity.""" + first = build_rag_perturbation_anchor( + anchor_id="unsupported_claim_anchor", + baseline_request_fingerprint=BASELINE_FP, + perturbed_request_fingerprint=PERTURBED_FP, + perturbation_kind=RAGPerturbationKind.UNSUPPORTED_CLAIM, + ) + replay = build_rag_perturbation_anchor( + anchor_id="unsupported_claim_anchor", + baseline_request_fingerprint=BASELINE_FP, + perturbed_request_fingerprint=PERTURBED_FP, + perturbation_kind="unsupported_claim", + ) + second = build_rag_perturbation_anchor( + anchor_id="unsupported_claim_anchor", + baseline_request_fingerprint=BASELINE_FP, + perturbed_request_fingerprint=SECOND_FP, + perturbation_kind="unsupported_claim", + ) + + assert first.anchor_fingerprint == replay.anchor_fingerprint + assert first.anchor_fingerprint != second.anchor_fingerprint + payload = first.to_dict() + assert payload["baseline_request_fingerprint"] == BASELINE_FP + assert payload["perturbed_request_fingerprint"] == PERTURBED_FP + assert payload["perturbation_kind"] == "unsupported_claim" + assert payload["expected_construct"] == "grounded_generation" + assert payload["expected_direction"] == "decrease" + + +def test_anchor_requires_distinct_baseline_and_perturbed_requests() -> None: + """A no-op request pair cannot masquerade as perturbation evidence.""" + with pytest.raises(AssessmentSpecError) as caught: + build_rag_perturbation_anchor( + anchor_id="no_op_anchor", + baseline_request_fingerprint=BASELINE_FP, + perturbed_request_fingerprint=BASELINE_FP, + perturbation_kind="style_only_rewrite", + ) + assert caught.value.code == "identical_rag_perturbation_requests" + + +def test_anchor_rejects_unknown_semantics_and_malformed_fingerprints() -> None: + """Unknown perturbations and non-content-addressed inputs fail closed.""" + with pytest.raises(AssessmentSpecError) as caught: + build_rag_perturbation_anchor( + anchor_id="unknown_anchor", + baseline_request_fingerprint=BASELINE_FP, + perturbed_request_fingerprint=PERTURBED_FP, + perturbation_kind="improve_everything", + ) + assert caught.value.code == "invalid_rag_perturbation_kind" + + with pytest.raises(AssessmentSpecError) as caught: + build_rag_perturbation_anchor( + anchor_id="bad_fingerprint_anchor", + baseline_request_fingerprint="not-a-sha256", + perturbed_request_fingerprint=PERTURBED_FP, + perturbation_kind="unsupported_claim", + ) + assert caught.value.code == "invalid_baseline_request_fingerprint" + + +def test_anchor_contract_cannot_store_raw_query_context_or_answer_text() -> None: + """Perturbation evidence is source-free identity metadata, not a content store.""" + parameters = inspect.signature(build_rag_perturbation_anchor).parameters + assert not set(parameters).intersection( + { + "query_text", + "question_text", + "context_text", + "retrieved_text", + "answer_text", + "response_text", + "source_text", + } + ) From 4f23815a5b1f31a9a3f33044f0f5630aa5a35403 Mon Sep 17 00:00:00 2001 From: Seongho Bae Date: Thu, 13 Aug 2026 09:10:46 +0900 Subject: [PATCH 2/7] test(rag): bind anchors to governed request relationships --- ..._scoring_rag_perturbation_relationships.py | 240 ++++++++++++++++++ 1 file changed, 240 insertions(+) create mode 100644 tests/test_scoring_rag_perturbation_relationships.py diff --git a/tests/test_scoring_rag_perturbation_relationships.py b/tests/test_scoring_rag_perturbation_relationships.py new file mode 100644 index 000000000..d4b43450d --- /dev/null +++ b/tests/test_scoring_rag_perturbation_relationships.py @@ -0,0 +1,240 @@ +"""Fail-first relationship contracts for governed RAG perturbation anchors.""" + +from __future__ import annotations + +import hashlib +import inspect +from pathlib import Path +import runpy +from typing import Any + +import pytest + +from fast_mlsirm.scoring import AssessmentSpecError, ScoringRequest +from fast_mlsirm.scoring.rag import ( + RAGPerturbationConstructBasis, + RAGPerturbationKind, + build_rag_perturbation_anchor, + build_rag_scoring_request, +) + +_FIXTURES = runpy.run_path( + str(Path(__file__).with_name("scoring_execution_fixtures.py")) +) +assessment = _FIXTURES["assessment"] +rubric = _FIXTURES["rubric"] + +QUERY_FP = hashlib.sha256(b"rag-anchor-query").hexdigest() +SECOND_QUERY_FP = hashlib.sha256(b"rag-anchor-query-second").hexdigest() +SYSTEM_FP = hashlib.sha256(b"rag-anchor-system").hexdigest() +OTHER_SYSTEM_FP = hashlib.sha256(b"rag-anchor-other-system").hexdigest() +RETRIEVAL_FP = hashlib.sha256(b"rag-anchor-retrieval").hexdigest() +SECOND_RETRIEVAL_FP = hashlib.sha256(b"rag-anchor-retrieval-second").hexdigest() +RESPONSE_FP = hashlib.sha256(b"rag-anchor-response").hexdigest() +SECOND_RESPONSE_FP = hashlib.sha256(b"rag-anchor-response-second").hexdigest() + + +def _request(**overrides: Any) -> ScoringRequest: + """Build one deterministic canonical RAG scoring request.""" + values: dict[str, Any] = { + "request_id": "rag_anchor_baseline_request", + "assessment": assessment(), + "rubric": rubric(), + "query_id": "refund_policy_query", + "query_revision_fingerprint": QUERY_FP, + "query_testlet_id": "evidence_review", + "evidence_regime": "retrieved_context", + "candidate_visibility": "candidate_blind", + "system_configuration_id": "retrieval_stack_a", + "system_configuration_fingerprint": SYSTEM_FP, + "system_run_id": "retrieval_stack_a_run_001", + "response_id": "generated_response_001", + "retrieval_run_fingerprint": RETRIEVAL_FP, + "response_content_fingerprint": RESPONSE_FP, + "occasion_id": "evaluation_wave_001", + "criterion_ids": ("grounded_generation", "answer_relevance"), + "response_character_count": 412, + "response_unit_count": 7, + "metadata": {"evaluation_split": "offline_holdout"}, + } + values.update(overrides) + return build_rag_scoring_request(**values) + + +def _assert_relationship_error(kind: str, perturbed: ScoringRequest) -> None: + """Assert one stable invalid-relationship error.""" + with pytest.raises(AssessmentSpecError) as caught: + build_rag_perturbation_anchor( + anchor_id=f"{kind}_anchor", + baseline_request=_request(), + perturbed_request=perturbed, + perturbation_kind=kind, + ) + assert caught.value.code == "invalid_rag_perturbation_relationship" + + +def test_anchor_builder_accepts_canonical_requests_not_opaque_fingerprints() -> None: + """The boundary must inspect the governed pair before storing its identities.""" + parameters = inspect.signature(build_rag_perturbation_anchor).parameters + assert {"baseline_request", "perturbed_request"}.issubset(parameters) + assert "baseline_request_fingerprint" not in parameters + assert "perturbed_request_fingerprint" not in parameters + + +def test_response_perturbation_binds_only_a_changed_response_artifact() -> None: + """Response kinds preserve query/retrieval provenance and change response content.""" + baseline = _request() + perturbed = _request( + request_id="rag_anchor_response_perturbed_request", + response_id="generated_response_002", + response_content_fingerprint=SECOND_RESPONSE_FP, + response_character_count=430, + response_unit_count=8, + ) + + anchor = build_rag_perturbation_anchor( + anchor_id="unsupported_claim_anchor", + baseline_request=baseline, + perturbed_request=perturbed, + perturbation_kind=RAGPerturbationKind.UNSUPPORTED_CLAIM, + ) + + assert anchor.baseline_request_fingerprint == baseline.request_fingerprint + assert anchor.perturbed_request_fingerprint == perturbed.request_fingerprint + + +def test_retrieval_perturbation_binds_only_a_changed_retrieval_artifact() -> None: + """Retrieval kinds preserve query and response artifacts while changing retrieval.""" + baseline = _request() + perturbed = _request( + request_id="rag_anchor_retrieval_perturbed_request", + retrieval_run_fingerprint=SECOND_RETRIEVAL_FP, + ) + + anchor = build_rag_perturbation_anchor( + anchor_id="irrelevant_context_anchor", + baseline_request=baseline, + perturbed_request=perturbed, + perturbation_kind=RAGPerturbationKind.IRRELEVANT_CONTEXT, + ) + + assert anchor.baseline_request_fingerprint == baseline.request_fingerprint + assert anchor.perturbed_request_fingerprint == perturbed.request_fingerprint + + +def test_query_perturbation_requires_a_changed_query_revision() -> None: + """Query kinds preserve the governed task identity while changing its revision.""" + baseline = _request() + perturbed = _request( + request_id="rag_anchor_query_perturbed_request", + query_revision_fingerprint=SECOND_QUERY_FP, + retrieval_run_fingerprint=SECOND_RETRIEVAL_FP, + response_id="generated_response_002", + response_content_fingerprint=SECOND_RESPONSE_FP, + response_character_count=215, + response_unit_count=4, + ) + + anchor = build_rag_perturbation_anchor( + anchor_id="semantic_query_paraphrase_anchor", + baseline_request=baseline, + perturbed_request=perturbed, + perturbation_kind=RAGPerturbationKind.SEMANTIC_QUERY_PARAPHRASE, + ) + + assert anchor.baseline_request_fingerprint == baseline.request_fingerprint + assert anchor.perturbed_request_fingerprint == perturbed.request_fingerprint + + +def test_unrelated_system_configuration_pair_fails_closed() -> None: + """A different evaluated system cannot masquerade as one controlled perturbation.""" + with pytest.raises(AssessmentSpecError) as caught: + build_rag_perturbation_anchor( + anchor_id="unrelated_system_anchor", + baseline_request=_request(), + perturbed_request=_request( + request_id="rag_anchor_other_system_request", + system_configuration_id="retrieval_stack_b", + system_configuration_fingerprint=OTHER_SYSTEM_FP, + response_id="generated_response_002", + response_content_fingerprint=SECOND_RESPONSE_FP, + ), + perturbation_kind="unsupported_claim", + ) + assert caught.value.code == "unrelated_rag_perturbation_requests" + + +def test_response_kind_rejects_a_simultaneous_retrieval_change() -> None: + """A response perturbation cannot silently include a second changed treatment.""" + _assert_relationship_error( + "unsupported_claim", + _request( + request_id="rag_anchor_mixed_response_request", + response_id="generated_response_002", + response_content_fingerprint=SECOND_RESPONSE_FP, + retrieval_run_fingerprint=SECOND_RETRIEVAL_FP, + ), + ) + + +def test_retrieval_kind_rejects_a_simultaneous_response_change() -> None: + """A retrieval perturbation must keep the scored response artifact fixed.""" + _assert_relationship_error( + "irrelevant_context", + _request( + request_id="rag_anchor_mixed_retrieval_request", + retrieval_run_fingerprint=SECOND_RETRIEVAL_FP, + response_id="generated_response_002", + response_content_fingerprint=SECOND_RESPONSE_FP, + ), + ) + + +def test_query_kind_rejects_an_unchanged_query_revision() -> None: + """Changing only the response is not evidence of a query perturbation.""" + _assert_relationship_error( + "semantic_query_paraphrase", + _request( + request_id="rag_anchor_unchanged_query_request", + response_id="generated_response_002", + response_content_fingerprint=SECOND_RESPONSE_FP, + ), + ) + + +def test_construct_basis_distinguishes_sources_from_model_design() -> None: + """Unsupported literature mappings remain explicit model-design hypotheses.""" + assert RAGPerturbationConstructBasis.LITERATURE_ALIGNED_CONSTRUCT.value == ( + "literature_aligned_construct" + ) + assert RAGPerturbationConstructBasis.MODEL_DESIGN_HYPOTHESIS.value == ( + "model_design_hypothesis" + ) + + literature_aligned = build_rag_perturbation_anchor( + anchor_id="grounded_generation_anchor", + baseline_request=_request(), + perturbed_request=_request( + request_id="rag_anchor_grounded_perturbed_request", + response_id="generated_response_002", + response_content_fingerprint=SECOND_RESPONSE_FP, + ), + perturbation_kind="unsupported_claim", + ) + model_design = build_rag_perturbation_anchor( + anchor_id="citation_hypothesis_anchor", + baseline_request=_request(), + perturbed_request=_request( + request_id="rag_anchor_citation_perturbed_request", + response_id="generated_response_002", + response_content_fingerprint=SECOND_RESPONSE_FP, + ), + perturbation_kind="citation_target_swap", + ) + + assert literature_aligned.construct_basis is ( + RAGPerturbationConstructBasis.LITERATURE_ALIGNED_CONSTRUCT + ) + assert model_design.construct_basis is ( + RAGPerturbationConstructBasis.MODEL_DESIGN_HYPOTHESIS + ) From 0cb1643fb508832dd821aae0aa4b2d49e903279d Mon Sep 17 00:00:00 2001 From: Seongho Bae Date: Thu, 13 Aug 2026 09:21:41 +0900 Subject: [PATCH 3/7] test(rag): bind anchors to transform provenance --- ..._scoring_rag_perturbation_relationships.py | 120 +++++++++++------- 1 file changed, 75 insertions(+), 45 deletions(-) diff --git a/tests/test_scoring_rag_perturbation_relationships.py b/tests/test_scoring_rag_perturbation_relationships.py index d4b43450d..826d5ee0b 100644 --- a/tests/test_scoring_rag_perturbation_relationships.py +++ b/tests/test_scoring_rag_perturbation_relationships.py @@ -1,4 +1,4 @@ -"""Fail-first relationship contracts for governed RAG perturbation anchors.""" +"""Relationship contracts for governed RAG perturbation anchors.""" from __future__ import annotations @@ -32,6 +32,8 @@ SECOND_RETRIEVAL_FP = hashlib.sha256(b"rag-anchor-retrieval-second").hexdigest() RESPONSE_FP = hashlib.sha256(b"rag-anchor-response").hexdigest() SECOND_RESPONSE_FP = hashlib.sha256(b"rag-anchor-response-second").hexdigest() +SPECIFICATION_FP = hashlib.sha256(b"rag-anchor-specification").hexdigest() +RUN_FP = hashlib.sha256(b"rag-anchor-run").hexdigest() def _request(**overrides: Any) -> ScoringRequest: @@ -61,28 +63,44 @@ def _request(**overrides: Any) -> ScoringRequest: return build_rag_scoring_request(**values) +def _anchor( + *, + kind: RAGPerturbationKind | str, + perturbed: ScoringRequest, + baseline: ScoringRequest | None = None, +): + """Build one anchor with deterministic transformation provenance.""" + return build_rag_perturbation_anchor( + anchor_id=f"{RAGPerturbationKind(kind).value}_anchor", + baseline_request=_request() if baseline is None else baseline, + perturbed_request=perturbed, + perturbation_specification_fingerprint=SPECIFICATION_FP, + perturbation_run_fingerprint=RUN_FP, + perturbation_kind=kind, + ) + + def _assert_relationship_error(kind: str, perturbed: ScoringRequest) -> None: """Assert one stable invalid-relationship error.""" with pytest.raises(AssessmentSpecError) as caught: - build_rag_perturbation_anchor( - anchor_id=f"{kind}_anchor", - baseline_request=_request(), - perturbed_request=perturbed, - perturbation_kind=kind, - ) + _anchor(kind=kind, perturbed=perturbed) assert caught.value.code == "invalid_rag_perturbation_relationship" def test_anchor_builder_accepts_canonical_requests_not_opaque_fingerprints() -> None: - """The boundary must inspect the governed pair before storing its identities.""" + """The boundary must inspect the governed pair before storing identities.""" parameters = inspect.signature(build_rag_perturbation_anchor).parameters assert {"baseline_request", "perturbed_request"}.issubset(parameters) + assert { + "perturbation_specification_fingerprint", + "perturbation_run_fingerprint", + }.issubset(parameters) assert "baseline_request_fingerprint" not in parameters assert "perturbed_request_fingerprint" not in parameters def test_response_perturbation_binds_only_a_changed_response_artifact() -> None: - """Response kinds preserve query/retrieval provenance and change response content.""" + """Response kinds preserve query/retrieval provenance and change response.""" baseline = _request() perturbed = _request( request_id="rag_anchor_response_perturbed_request", @@ -92,30 +110,30 @@ def test_response_perturbation_binds_only_a_changed_response_artifact() -> None: response_unit_count=8, ) - anchor = build_rag_perturbation_anchor( - anchor_id="unsupported_claim_anchor", - baseline_request=baseline, - perturbed_request=perturbed, - perturbation_kind=RAGPerturbationKind.UNSUPPORTED_CLAIM, + anchor = _anchor( + kind=RAGPerturbationKind.UNSUPPORTED_CLAIM, + baseline=baseline, + perturbed=perturbed, ) assert anchor.baseline_request_fingerprint == baseline.request_fingerprint assert anchor.perturbed_request_fingerprint == perturbed.request_fingerprint + assert anchor.perturbation_specification_fingerprint == SPECIFICATION_FP + assert anchor.perturbation_run_fingerprint == RUN_FP def test_retrieval_perturbation_binds_only_a_changed_retrieval_artifact() -> None: - """Retrieval kinds preserve query and response artifacts while changing retrieval.""" + """Retrieval kinds preserve query/response while changing retrieval.""" baseline = _request() perturbed = _request( request_id="rag_anchor_retrieval_perturbed_request", retrieval_run_fingerprint=SECOND_RETRIEVAL_FP, ) - anchor = build_rag_perturbation_anchor( - anchor_id="irrelevant_context_anchor", - baseline_request=baseline, - perturbed_request=perturbed, - perturbation_kind=RAGPerturbationKind.IRRELEVANT_CONTEXT, + anchor = _anchor( + kind=RAGPerturbationKind.IRRELEVANT_CONTEXT, + baseline=baseline, + perturbed=perturbed, ) assert anchor.baseline_request_fingerprint == baseline.request_fingerprint @@ -123,7 +141,7 @@ def test_retrieval_perturbation_binds_only_a_changed_retrieval_artifact() -> Non def test_query_perturbation_requires_a_changed_query_revision() -> None: - """Query kinds preserve the governed task identity while changing its revision.""" + """Query kinds preserve task identity while changing its revision.""" baseline = _request() perturbed = _request( request_id="rag_anchor_query_perturbed_request", @@ -135,11 +153,10 @@ def test_query_perturbation_requires_a_changed_query_revision() -> None: response_unit_count=4, ) - anchor = build_rag_perturbation_anchor( - anchor_id="semantic_query_paraphrase_anchor", - baseline_request=baseline, - perturbed_request=perturbed, - perturbation_kind=RAGPerturbationKind.SEMANTIC_QUERY_PARAPHRASE, + anchor = _anchor( + kind=RAGPerturbationKind.SEMANTIC_QUERY_PARAPHRASE, + baseline=baseline, + perturbed=perturbed, ) assert anchor.baseline_request_fingerprint == baseline.request_fingerprint @@ -147,25 +164,23 @@ def test_query_perturbation_requires_a_changed_query_revision() -> None: def test_unrelated_system_configuration_pair_fails_closed() -> None: - """A different evaluated system cannot masquerade as one controlled perturbation.""" + """A different evaluated system cannot masquerade as one perturbation.""" with pytest.raises(AssessmentSpecError) as caught: - build_rag_perturbation_anchor( - anchor_id="unrelated_system_anchor", - baseline_request=_request(), - perturbed_request=_request( + _anchor( + kind="unsupported_claim", + perturbed=_request( request_id="rag_anchor_other_system_request", system_configuration_id="retrieval_stack_b", system_configuration_fingerprint=OTHER_SYSTEM_FP, response_id="generated_response_002", response_content_fingerprint=SECOND_RESPONSE_FP, ), - perturbation_kind="unsupported_claim", ) assert caught.value.code == "unrelated_rag_perturbation_requests" def test_response_kind_rejects_a_simultaneous_retrieval_change() -> None: - """A response perturbation cannot silently include a second changed treatment.""" + """A response perturbation cannot include a second changed treatment.""" _assert_relationship_error( "unsupported_claim", _request( @@ -178,7 +193,7 @@ def test_response_kind_rejects_a_simultaneous_retrieval_change() -> None: def test_retrieval_kind_rejects_a_simultaneous_response_change() -> None: - """A retrieval perturbation must keep the scored response artifact fixed.""" + """A retrieval perturbation must keep the response artifact fixed.""" _assert_relationship_error( "irrelevant_context", _request( @@ -203,7 +218,7 @@ def test_query_kind_rejects_an_unchanged_query_revision() -> None: def test_construct_basis_distinguishes_sources_from_model_design() -> None: - """Unsupported literature mappings remain explicit model-design hypotheses.""" + """Unsupported mappings remain explicit model-design hypotheses.""" assert RAGPerturbationConstructBasis.LITERATURE_ALIGNED_CONSTRUCT.value == ( "literature_aligned_construct" ) @@ -211,25 +226,21 @@ def test_construct_basis_distinguishes_sources_from_model_design() -> None: "model_design_hypothesis" ) - literature_aligned = build_rag_perturbation_anchor( - anchor_id="grounded_generation_anchor", - baseline_request=_request(), - perturbed_request=_request( + literature_aligned = _anchor( + kind="unsupported_claim", + perturbed=_request( request_id="rag_anchor_grounded_perturbed_request", response_id="generated_response_002", response_content_fingerprint=SECOND_RESPONSE_FP, ), - perturbation_kind="unsupported_claim", ) - model_design = build_rag_perturbation_anchor( - anchor_id="citation_hypothesis_anchor", - baseline_request=_request(), - perturbed_request=_request( + model_design = _anchor( + kind="citation_target_swap", + perturbed=_request( request_id="rag_anchor_citation_perturbed_request", response_id="generated_response_002", response_content_fingerprint=SECOND_RESPONSE_FP, ), - perturbation_kind="citation_target_swap", ) assert literature_aligned.construct_basis is ( @@ -238,3 +249,22 @@ def test_construct_basis_distinguishes_sources_from_model_design() -> None: assert model_design.construct_basis is ( RAGPerturbationConstructBasis.MODEL_DESIGN_HYPOTHESIS ) + + +def test_relation_provenance_fingerprints_fail_closed() -> None: + """The external controlled-transform protocol and run are content-addressed.""" + perturbed = _request( + request_id="rag_anchor_bad_relation_request", + response_id="generated_response_002", + response_content_fingerprint=SECOND_RESPONSE_FP, + ) + with pytest.raises(AssessmentSpecError) as caught: + build_rag_perturbation_anchor( + anchor_id="bad_relation_anchor", + baseline_request=_request(), + perturbed_request=perturbed, + perturbation_specification_fingerprint="not-a-sha256", + perturbation_run_fingerprint=RUN_FP, + perturbation_kind="unsupported_claim", + ) + assert caught.value.code == "invalid_perturbation_specification_fingerprint" From 8c25fce12d54f56fa20ab96212e5f6f27e652281 Mon Sep 17 00:00:00 2001 From: Seongho Bae Date: Thu, 13 Aug 2026 09:23:52 +0900 Subject: [PATCH 4/7] fix(rag): validate governed perturbation relationships --- python/fast_mlsirm/scoring/rag.py | 371 ++++++++++++++++++++++++++++-- 1 file changed, 351 insertions(+), 20 deletions(-) diff --git a/python/fast_mlsirm/scoring/rag.py b/python/fast_mlsirm/scoring/rag.py index ba94da8f7..c870d2ee7 100644 --- a/python/fast_mlsirm/scoring/rag.py +++ b/python/fast_mlsirm/scoring/rag.py @@ -8,7 +8,7 @@ from __future__ import annotations from collections.abc import Iterable, Mapping -from dataclasses import dataclass, field +from dataclasses import InitVar, dataclass, field from enum import Enum from types import MappingProxyType from typing import Any @@ -65,63 +65,139 @@ class RAGPerturbationDirection(str, Enum): INCREASE = "increase" +class RAGPerturbationConstructBasis(str, Enum): + """Whether cited primary sources directly align with the named construct.""" + + LITERATURE_ALIGNED_CONSTRUCT = "literature_aligned_construct" + MODEL_DESIGN_HYPOTHESIS = "model_design_hypothesis" + + _PERTURBATION_EXPECTATIONS = MappingProxyType( { RAGPerturbationKind.UNSUPPORTED_CLAIM: ( "grounded_generation", RAGPerturbationDirection.DECREASE, + RAGPerturbationConstructBasis.LITERATURE_ALIGNED_CONSTRUCT, ), RAGPerturbationKind.EXPLICIT_CONTRADICTION: ( "grounded_generation", RAGPerturbationDirection.DECREASE, + RAGPerturbationConstructBasis.LITERATURE_ALIGNED_CONSTRUCT, ), RAGPerturbationKind.IRRELEVANT_CONTEXT: ( "retrieval_relevance", RAGPerturbationDirection.DECREASE, + RAGPerturbationConstructBasis.LITERATURE_ALIGNED_CONSTRUCT, ), RAGPerturbationKind.REQUIRED_EVIDENCE_REMOVAL: ( "coverage_or_completeness_proxy", RAGPerturbationDirection.DECREASE, + RAGPerturbationConstructBasis.MODEL_DESIGN_HYPOTHESIS, ), RAGPerturbationKind.CITATION_TARGET_SWAP: ( "citation_attribution", RAGPerturbationDirection.DECREASE, + RAGPerturbationConstructBasis.MODEL_DESIGN_HYPOTHESIS, ), RAGPerturbationKind.SEMANTIC_QUERY_PARAPHRASE: ( "robustness", RAGPerturbationDirection.INVARIANT, + RAGPerturbationConstructBasis.MODEL_DESIGN_HYPOTHESIS, ), RAGPerturbationKind.STYLE_ONLY_REWRITE: ( "robustness", RAGPerturbationDirection.INVARIANT, + RAGPerturbationConstructBasis.MODEL_DESIGN_HYPOTHESIS, ), RAGPerturbationKind.UNANSWERABLE_QUERY: ( "answerability_and_abstention", RAGPerturbationDirection.INCREASE, + RAGPerturbationConstructBasis.MODEL_DESIGN_HYPOTHESIS, ), } ) +_RESPONSE_PERTURBATIONS = frozenset( + { + RAGPerturbationKind.UNSUPPORTED_CLAIM, + RAGPerturbationKind.EXPLICIT_CONTRADICTION, + RAGPerturbationKind.CITATION_TARGET_SWAP, + RAGPerturbationKind.STYLE_ONLY_REWRITE, + } +) +_RETRIEVAL_PERTURBATIONS = frozenset( + { + RAGPerturbationKind.IRRELEVANT_CONTEXT, + RAGPerturbationKind.REQUIRED_EVIDENCE_REMOVAL, + } +) +_QUERY_PERTURBATIONS = frozenset( + { + RAGPerturbationKind.SEMANTIC_QUERY_PARAPHRASE, + RAGPerturbationKind.UNANSWERABLE_QUERY, + } +) +_COMMON_REQUEST_FIELDS = ( + "assessment_fingerprint", + "rubric_id", + "rubric_fingerprint", + "construct_id", + "response_format", + "granularity", + "respondent_id", + "task_id", + "task_family_id", + "occasion_id", + "criterion_ids", + "allowed_scores", + "schema_version", +) +_RAG_RELATION_METADATA_KEYS = frozenset( + { + "rag_evidence_regime", + "rag_candidate_visibility", + "rag_system_configuration_id", + "rag_system_configuration_fingerprint", + "rag_retrieval_run_fingerprint", + "rag_query_revision_fingerprint", + } +) +_RELATION_AXIS_METADATA_KEYS = frozenset( + {"rag_retrieval_run_fingerprint", "rag_query_revision_fingerprint"} +) +_RAG_PERTURBATION_ANCHOR_TOKEN = object() + @dataclass(frozen=True) class RAGPerturbationAnchor(CanonicalContract): - """Source-free identity and expected direction for a controlled RAG perturbation. + """Verified identity and preregistered direction for one RAG perturbation. - The anchor binds two distinct governed scoring-request fingerprints to one - finite perturbation kind. ``expected_construct`` and ``expected_direction`` - are package-derived preregistration metadata; they do not assert that an - observed system response actually moved in the expected direction. + The factory binds two canonical governed scoring requests and validates the + kind-specific changed axis before this source-free artifact is constructed. + ``construct_basis`` distinguishes constructs aligned with the cited RAGAS / + ARES dimensions from package-owned model-design hypotheses. Every expected + direction remains a preregistered hypothesis rather than an observed effect. """ anchor_id: str baseline_request_fingerprint: str perturbed_request_fingerprint: str + perturbation_specification_fingerprint: str + perturbation_run_fingerprint: str perturbation_kind: RAGPerturbationKind expected_construct: str = field(init=False) expected_direction: RAGPerturbationDirection = field(init=False) + construct_basis: RAGPerturbationConstructBasis = field(init=False) + _anchor_token: InitVar[object | None] = None - def __post_init__(self) -> None: - """Normalize identity fields and derive immutable perturbation semantics.""" + def __post_init__(self, _anchor_token: object | None) -> None: + """Reject direct construction and derive immutable perturbation semantics.""" + if _anchor_token is not _RAG_PERTURBATION_ANCHOR_TOKEN: + raise assessment_error( + "unverified_rag_perturbation_anchor", + "$", + "use build_rag_perturbation_anchor", + ) normalized_anchor_id = descriptive_identifier( self.anchor_id, "anchor_id", @@ -137,6 +213,16 @@ def __post_init__(self) -> None: "perturbed_request_fingerprint", "$.perturbed_request_fingerprint", ) + normalized_specification = fingerprint( + self.perturbation_specification_fingerprint, + "perturbation_specification_fingerprint", + "$.perturbation_specification_fingerprint", + ) + normalized_run = fingerprint( + self.perturbation_run_fingerprint, + "perturbation_run_fingerprint", + "$.perturbation_run_fingerprint", + ) normalized_kind = enum_value( self.perturbation_kind, RAGPerturbationKind, @@ -149,9 +235,9 @@ def __post_init__(self) -> None: "$.perturbed_request_fingerprint", "baseline and perturbed request fingerprints must differ", ) - expected_construct, expected_direction = _PERTURBATION_EXPECTATIONS[ - normalized_kind - ] + expected_construct, expected_direction, construct_basis = ( + _PERTURBATION_EXPECTATIONS[normalized_kind] + ) object.__setattr__(self, "anchor_id", normalized_anchor_id) object.__setattr__( self, @@ -163,9 +249,20 @@ def __post_init__(self) -> None: "perturbed_request_fingerprint", normalized_perturbed, ) + object.__setattr__( + self, + "perturbation_specification_fingerprint", + normalized_specification, + ) + object.__setattr__( + self, + "perturbation_run_fingerprint", + normalized_run, + ) object.__setattr__(self, "perturbation_kind", normalized_kind) object.__setattr__(self, "expected_construct", expected_construct) object.__setattr__(self, "expected_direction", expected_direction) + object.__setattr__(self, "construct_basis", construct_basis) def _content_dict(self) -> dict[str, Any]: """Return canonical anchor content without the derived content digest.""" @@ -173,9 +270,14 @@ def _content_dict(self) -> dict[str, Any]: "anchor_id": self.anchor_id, "baseline_request_fingerprint": self.baseline_request_fingerprint, "perturbed_request_fingerprint": self.perturbed_request_fingerprint, + "perturbation_specification_fingerprint": ( + self.perturbation_specification_fingerprint + ), + "perturbation_run_fingerprint": self.perturbation_run_fingerprint, "perturbation_kind": self.perturbation_kind.value, "expected_construct": self.expected_construct, "expected_direction": self.expected_direction.value, + "construct_basis": self.construct_basis.value, } @property @@ -191,24 +293,252 @@ def to_dict(self) -> dict[str, Any]: } +def _canonical_rag_request(value: Any, name: str) -> ScoringRequest: + """Return one canonical RAG request with complete managed provenance.""" + if not isinstance(value, ScoringRequest): + raise assessment_error( + f"invalid_{name}", + f"$.{name}", + f"{name} must be a ScoringRequest", + ) + metadata = thaw_json_value(value.metadata) + if not isinstance(metadata, dict) or not ( + _RAG_RELATION_METADATA_KEYS.issubset(metadata) + ): + raise assessment_error( + f"invalid_{name}", + f"$.{name}.metadata", + f"{name} must contain complete managed RAG provenance", + ) + query_revision = fingerprint( + metadata["rag_query_revision_fingerprint"], + "rag_query_revision_fingerprint", + f"$.{name}.metadata.rag_query_revision_fingerprint", + ) + if query_revision != value.task_revision_fingerprint: + raise assessment_error( + f"invalid_{name}", + f"$.{name}.metadata.rag_query_revision_fingerprint", + "RAG query revision must replay the shared task revision", + ) + fingerprint( + metadata["rag_system_configuration_fingerprint"], + "rag_system_configuration_fingerprint", + f"$.{name}.metadata.rag_system_configuration_fingerprint", + ) + fingerprint( + metadata["rag_retrieval_run_fingerprint"], + "rag_retrieval_run_fingerprint", + f"$.{name}.metadata.rag_retrieval_run_fingerprint", + ) + descriptive_identifier( + metadata["rag_system_configuration_id"], + "rag_system_configuration_id", + f"$.{name}.metadata.rag_system_configuration_id", + ) + enum_value( + metadata["rag_evidence_regime"], + RAGEvidenceRegime, + "rag_evidence_regime", + f"$.{name}.metadata.rag_evidence_regime", + ) + enum_value( + metadata["rag_candidate_visibility"], + RAGCandidateVisibility, + "rag_candidate_visibility", + f"$.{name}.metadata.rag_candidate_visibility", + ) + return value + + +def _request_metadata(request: ScoringRequest) -> dict[str, Any]: + """Return mutable JSON metadata from one immutable scoring request.""" + metadata = thaw_json_value(request.metadata) + if not isinstance(metadata, dict): # pragma: no cover - sealed request invariant + raise assessment_error( + "invalid_rag_perturbation_request", + "$.metadata", + "RAG request metadata must be a mapping", + ) + return metadata + + +def _raise_relationship_error(message: str) -> None: + """Raise one stable kind-specific relationship failure.""" + raise assessment_error( + "invalid_rag_perturbation_relationship", + "$.perturbed_request", + message, + ) + + +def _validate_common_request_provenance( + baseline: ScoringRequest, + perturbed: ScoringRequest, + baseline_metadata: Mapping[str, Any], + perturbed_metadata: Mapping[str, Any], +) -> None: + """Reject pairs that differ outside controlled RAG treatment axes.""" + if any( + getattr(baseline, field_name) != getattr(perturbed, field_name) + for field_name in _COMMON_REQUEST_FIELDS + ): + raise assessment_error( + "unrelated_rag_perturbation_requests", + "$.perturbed_request", + "requests must share assessment, rubric, system run, task, " + "and occasion provenance", + ) + baseline_invariants = { + key: value + for key, value in baseline_metadata.items() + if key not in _RELATION_AXIS_METADATA_KEYS + } + perturbed_invariants = { + key: value + for key, value in perturbed_metadata.items() + if key not in _RELATION_AXIS_METADATA_KEYS + } + if baseline_invariants != perturbed_invariants: + raise assessment_error( + "unrelated_rag_perturbation_requests", + "$.perturbed_request.metadata", + "requests must share evidence, visibility, system, policy, " + "and split provenance", + ) + + +def _validate_perturbation_relationship( + baseline: ScoringRequest, + perturbed: ScoringRequest, + kind: RAGPerturbationKind, +) -> None: + """Validate the exact governed request relation permitted for ``kind``.""" + baseline_metadata = _request_metadata(baseline) + perturbed_metadata = _request_metadata(perturbed) + _validate_common_request_provenance( + baseline, + perturbed, + baseline_metadata, + perturbed_metadata, + ) + if baseline.request_id == perturbed.request_id: + _raise_relationship_error( + "baseline and perturbed request identifiers must differ" + ) + + same_query = ( + baseline.task_revision_fingerprint == perturbed.task_revision_fingerprint + ) + same_retrieval = ( + baseline_metadata["rag_retrieval_run_fingerprint"] + == perturbed_metadata["rag_retrieval_run_fingerprint"] + ) + same_response = ( + baseline.response_content_fingerprint + == perturbed.response_content_fingerprint + ) + + if kind in _RESPONSE_PERTURBATIONS: + if not same_query or not same_retrieval or same_response: + _raise_relationship_error( + "response perturbations must change only the governed response artifact" + ) + if baseline.response_id == perturbed.response_id: + _raise_relationship_error( + "a changed response artifact requires a distinct response identifier" + ) + return + + if kind in _RETRIEVAL_PERTURBATIONS: + if not same_query or same_retrieval or not same_response: + _raise_relationship_error( + "retrieval perturbations must change only retrieval provenance" + ) + if ( + baseline.response_id != perturbed.response_id + or baseline.response_character_count != perturbed.response_character_count + or baseline.response_unit_count != perturbed.response_unit_count + ): + _raise_relationship_error( + "retrieval perturbations must preserve the exact governed " + "response artifact" + ) + return + + if kind in _QUERY_PERTURBATIONS: + if same_query: + _raise_relationship_error( + "query perturbations require a distinct governed query revision" + ) + return + + raise AssertionError(f"unhandled RAG perturbation kind: {kind}") + + def build_rag_perturbation_anchor( *, anchor_id: str, - baseline_request_fingerprint: str, - perturbed_request_fingerprint: str, + baseline_request: ScoringRequest, + perturbed_request: ScoringRequest, + perturbation_specification_fingerprint: str, + perturbation_run_fingerprint: str, perturbation_kind: RAGPerturbationKind | str, ) -> RAGPerturbationAnchor: - """Build one content-addressed controlled RAG perturbation anchor. + """Build one verified, source-free controlled RAG perturbation anchor. - The contract intentionally stores only bounded identities and package-owned - preregistration semantics. It does not accept raw query, context, response, - or source text and performs no scoring, retrieval, or truth adjudication. + Both inputs must be canonical governed requests. The factory validates + shared provenance and the kind-specific changed axis, then binds the pair + to an externally governed perturbation specification and execution run. It + stores only content-addressed identities, accepts no raw query, context, + response, or source text, and performs no scoring or truth adjudication. """ + normalized_baseline = _canonical_rag_request( + baseline_request, + "baseline_request", + ) + normalized_perturbed = _canonical_rag_request( + perturbed_request, + "perturbed_request", + ) + normalized_specification = fingerprint( + perturbation_specification_fingerprint, + "perturbation_specification_fingerprint", + "$.perturbation_specification_fingerprint", + ) + normalized_run = fingerprint( + perturbation_run_fingerprint, + "perturbation_run_fingerprint", + "$.perturbation_run_fingerprint", + ) + normalized_kind = enum_value( + perturbation_kind, + RAGPerturbationKind, + "rag_perturbation_kind", + "$.perturbation_kind", + ) + if ( + normalized_baseline.request_fingerprint + == normalized_perturbed.request_fingerprint + ): + raise assessment_error( + "identical_rag_perturbation_requests", + "$.perturbed_request", + "baseline and perturbed requests must differ", + ) + _validate_perturbation_relationship( + normalized_baseline, + normalized_perturbed, + normalized_kind, + ) return RAGPerturbationAnchor( anchor_id=anchor_id, - baseline_request_fingerprint=baseline_request_fingerprint, - perturbed_request_fingerprint=perturbed_request_fingerprint, - perturbation_kind=perturbation_kind, + baseline_request_fingerprint=normalized_baseline.request_fingerprint, + perturbed_request_fingerprint=normalized_perturbed.request_fingerprint, + perturbation_specification_fingerprint=normalized_specification, + perturbation_run_fingerprint=normalized_run, + perturbation_kind=normalized_kind, + _anchor_token=_RAG_PERTURBATION_ANCHOR_TOKEN, ) @@ -392,6 +722,7 @@ def build_rag_scoring_request( "RAGCandidateVisibility", "RAGEvidenceRegime", "RAGPerturbationAnchor", + "RAGPerturbationConstructBasis", "RAGPerturbationDirection", "RAGPerturbationKind", "build_rag_perturbation_anchor", From e04731e8199da8614bc90bee8a9ed6075dc58ddf Mon Sep 17 00:00:00 2001 From: Seongho Bae Date: Thu, 13 Aug 2026 09:24:48 +0900 Subject: [PATCH 5/7] test(rag): migrate anchors to governed request pairs --- .../test_scoring_rag_perturbation_anchors.py | 180 +++++++++++++----- 1 file changed, 131 insertions(+), 49 deletions(-) diff --git a/tests/test_scoring_rag_perturbation_anchors.py b/tests/test_scoring_rag_perturbation_anchors.py index de1b13e57..4a43019a7 100644 --- a/tests/test_scoring_rag_perturbation_anchors.py +++ b/tests/test_scoring_rag_perturbation_anchors.py @@ -1,22 +1,84 @@ -"""Fail-first contracts for governed reference-free RAG perturbation anchors.""" +"""Canonical identity contracts for governed RAG perturbation anchors.""" from __future__ import annotations import hashlib import inspect +from pathlib import Path +import runpy +from typing import Any import pytest -from fast_mlsirm.scoring import AssessmentSpecError +from fast_mlsirm.scoring import AssessmentSpecError, ScoringRequest from fast_mlsirm.scoring.rag import ( + RAGPerturbationAnchor, RAGPerturbationDirection, RAGPerturbationKind, build_rag_perturbation_anchor, + build_rag_scoring_request, ) -BASELINE_FP = hashlib.sha256(b"rag-anchor-baseline-request").hexdigest() -PERTURBED_FP = hashlib.sha256(b"rag-anchor-perturbed-request").hexdigest() -SECOND_FP = hashlib.sha256(b"rag-anchor-second-perturbed-request").hexdigest() +_FIXTURES = runpy.run_path( + str(Path(__file__).with_name("scoring_execution_fixtures.py")) +) +assessment = _FIXTURES["assessment"] +rubric = _FIXTURES["rubric"] + +QUERY_FP = hashlib.sha256(b"rag-anchor-query-revision").hexdigest() +SYSTEM_FP = hashlib.sha256(b"rag-anchor-system-configuration").hexdigest() +RETRIEVAL_FP = hashlib.sha256(b"rag-anchor-retrieval-run").hexdigest() +RESPONSE_FP = hashlib.sha256(b"rag-anchor-response-content").hexdigest() +SECOND_RESPONSE_FP = hashlib.sha256(b"rag-anchor-response-content-second").hexdigest() +THIRD_RESPONSE_FP = hashlib.sha256(b"rag-anchor-response-content-third").hexdigest() +SPECIFICATION_FP = hashlib.sha256(b"rag-anchor-specification").hexdigest() +RUN_FP = hashlib.sha256(b"rag-anchor-run").hexdigest() + + +def _request(**overrides: Any) -> ScoringRequest: + """Build one deterministic canonical RAG scoring request.""" + values: dict[str, Any] = { + "request_id": "rag_anchor_baseline_request", + "assessment": assessment(), + "rubric": rubric(), + "query_id": "refund_policy_query", + "query_revision_fingerprint": QUERY_FP, + "query_testlet_id": "evidence_review", + "evidence_regime": "retrieved_context", + "candidate_visibility": "candidate_blind", + "system_configuration_id": "retrieval_stack_a", + "system_configuration_fingerprint": SYSTEM_FP, + "system_run_id": "retrieval_stack_a_run_001", + "response_id": "generated_response_001", + "retrieval_run_fingerprint": RETRIEVAL_FP, + "response_content_fingerprint": RESPONSE_FP, + "occasion_id": "evaluation_wave_001", + "criterion_ids": ("grounded_generation", "answer_relevance"), + "response_character_count": 412, + "response_unit_count": 7, + "metadata": {"evaluation_split": "offline_holdout"}, + } + values.update(overrides) + return build_rag_scoring_request(**values) + + +def _anchor( + *, + kind: RAGPerturbationKind | str, + perturbed: ScoringRequest, + baseline: ScoringRequest | None = None, + anchor_id: str | None = None, +) -> RAGPerturbationAnchor: + """Build one deterministic response-axis anchor.""" + normalized_kind = RAGPerturbationKind(kind) + return build_rag_perturbation_anchor( + anchor_id=anchor_id or f"{normalized_kind.value}_anchor", + baseline_request=_request() if baseline is None else baseline, + perturbed_request=perturbed, + perturbation_specification_fingerprint=SPECIFICATION_FP, + perturbation_run_fingerprint=RUN_FP, + perturbation_kind=normalized_kind, + ) @pytest.mark.parametrize( @@ -24,25 +86,23 @@ ( ("unsupported_claim", "grounded_generation", "decrease"), ("explicit_contradiction", "grounded_generation", "decrease"), - ("irrelevant_context", "retrieval_relevance", "decrease"), - ("required_evidence_removal", "coverage_or_completeness_proxy", "decrease"), ("citation_target_swap", "citation_attribution", "decrease"), - ("semantic_query_paraphrase", "robustness", "invariant"), ("style_only_rewrite", "robustness", "invariant"), - ("unanswerable_query", "answerability_and_abstention", "increase"), ), ) -def test_perturbation_kind_has_one_construct_specific_expected_direction( +def test_response_perturbation_has_one_construct_specific_expected_direction( kind: str, construct: str, direction: str, ) -> None: - """Known perturbations preserve their preregistered construct direction.""" - anchor = build_rag_perturbation_anchor( - anchor_id=f"{kind}_anchor", - baseline_request_fingerprint=BASELINE_FP, - perturbed_request_fingerprint=PERTURBED_FP, - perturbation_kind=kind, + """Response perturbations preserve preregistered construct directions.""" + anchor = _anchor( + kind=kind, + perturbed=_request( + request_id=f"{kind}_request", + response_id=f"{kind}_response", + response_content_fingerprint=SECOND_RESPONSE_FP, + ), ) assert anchor.perturbation_kind is RAGPerturbationKind(kind) @@ -70,71 +130,93 @@ def test_perturbation_enums_are_finite_and_explicit() -> None: def test_anchor_is_content_addressed_and_identity_bearing() -> None: - """The exact baseline/perturbation pair participates in anchor identity.""" - first = build_rag_perturbation_anchor( - anchor_id="unsupported_claim_anchor", - baseline_request_fingerprint=BASELINE_FP, - perturbed_request_fingerprint=PERTURBED_FP, - perturbation_kind=RAGPerturbationKind.UNSUPPORTED_CLAIM, + """The request pair and transformation provenance participate in identity.""" + baseline = _request() + perturbed = _request( + request_id="unsupported_claim_request", + response_id="generated_response_002", + response_content_fingerprint=SECOND_RESPONSE_FP, + ) + first = _anchor( + kind="unsupported_claim", + baseline=baseline, + perturbed=perturbed, ) - replay = build_rag_perturbation_anchor( - anchor_id="unsupported_claim_anchor", - baseline_request_fingerprint=BASELINE_FP, - perturbed_request_fingerprint=PERTURBED_FP, - perturbation_kind="unsupported_claim", + replay = _anchor( + kind="unsupported_claim", + baseline=baseline, + perturbed=perturbed, ) - second = build_rag_perturbation_anchor( - anchor_id="unsupported_claim_anchor", - baseline_request_fingerprint=BASELINE_FP, - perturbed_request_fingerprint=SECOND_FP, - perturbation_kind="unsupported_claim", + second = _anchor( + kind="unsupported_claim", + baseline=baseline, + perturbed=_request( + request_id="unsupported_claim_second_request", + response_id="generated_response_003", + response_content_fingerprint=THIRD_RESPONSE_FP, + ), ) assert first.anchor_fingerprint == replay.anchor_fingerprint assert first.anchor_fingerprint != second.anchor_fingerprint payload = first.to_dict() - assert payload["baseline_request_fingerprint"] == BASELINE_FP - assert payload["perturbed_request_fingerprint"] == PERTURBED_FP + assert payload["baseline_request_fingerprint"] == baseline.request_fingerprint + assert payload["perturbed_request_fingerprint"] == perturbed.request_fingerprint + assert payload["perturbation_specification_fingerprint"] == SPECIFICATION_FP + assert payload["perturbation_run_fingerprint"] == RUN_FP assert payload["perturbation_kind"] == "unsupported_claim" assert payload["expected_construct"] == "grounded_generation" assert payload["expected_direction"] == "decrease" + assert payload["construct_basis"] == "literature_aligned_construct" def test_anchor_requires_distinct_baseline_and_perturbed_requests() -> None: """A no-op request pair cannot masquerade as perturbation evidence.""" + request = _request() with pytest.raises(AssessmentSpecError) as caught: - build_rag_perturbation_anchor( + _anchor( + kind="style_only_rewrite", + baseline=request, + perturbed=request, anchor_id="no_op_anchor", - baseline_request_fingerprint=BASELINE_FP, - perturbed_request_fingerprint=BASELINE_FP, - perturbation_kind="style_only_rewrite", ) assert caught.value.code == "identical_rag_perturbation_requests" -def test_anchor_rejects_unknown_semantics_and_malformed_fingerprints() -> None: - """Unknown perturbations and non-content-addressed inputs fail closed.""" +def test_anchor_rejects_unknown_semantics() -> None: + """Unknown perturbation semantics fail closed.""" with pytest.raises(AssessmentSpecError) as caught: build_rag_perturbation_anchor( anchor_id="unknown_anchor", - baseline_request_fingerprint=BASELINE_FP, - perturbed_request_fingerprint=PERTURBED_FP, + baseline_request=_request(), + perturbed_request=_request( + request_id="unknown_perturbed_request", + response_id="unknown_response", + response_content_fingerprint=SECOND_RESPONSE_FP, + ), + perturbation_specification_fingerprint=SPECIFICATION_FP, + perturbation_run_fingerprint=RUN_FP, perturbation_kind="improve_everything", ) assert caught.value.code == "invalid_rag_perturbation_kind" + +def test_anchor_cannot_be_constructed_without_relationship_validation() -> None: + """Direct construction cannot bypass canonical pair validation.""" with pytest.raises(AssessmentSpecError) as caught: - build_rag_perturbation_anchor( - anchor_id="bad_fingerprint_anchor", - baseline_request_fingerprint="not-a-sha256", - perturbed_request_fingerprint=PERTURBED_FP, - perturbation_kind="unsupported_claim", + RAGPerturbationAnchor( + anchor_id="unverified_anchor", + baseline_request_fingerprint=RESPONSE_FP, + perturbed_request_fingerprint=SECOND_RESPONSE_FP, + perturbation_specification_fingerprint=SPECIFICATION_FP, + perturbation_run_fingerprint=RUN_FP, + perturbation_kind=RAGPerturbationKind.UNSUPPORTED_CLAIM, ) - assert caught.value.code == "invalid_baseline_request_fingerprint" + assert caught.value.code == "unverified_rag_perturbation_anchor" def test_anchor_contract_cannot_store_raw_query_context_or_answer_text() -> None: - """Perturbation evidence is source-free identity metadata, not a content store.""" + """Perturbation evidence is source-free identity metadata, not content.""" parameters = inspect.signature(build_rag_perturbation_anchor).parameters assert not set(parameters).intersection( { From 5878dd7fd1f00943f0232c5fca01981f86307d92 Mon Sep 17 00:00:00 2001 From: Seongho Bae Date: Thu, 13 Aug 2026 09:25:58 +0900 Subject: [PATCH 6/7] docs(rag): separate provenance from scientific hypotheses --- docs/doctoring/rag_scoring_request_privacy.md | 30 +++++++++++++------ 1 file changed, 21 insertions(+), 9 deletions(-) diff --git a/docs/doctoring/rag_scoring_request_privacy.md b/docs/doctoring/rag_scoring_request_privacy.md index d01e061a8..f0e3dbc67 100644 --- a/docs/doctoring/rag_scoring_request_privacy.md +++ b/docs/doctoring/rag_scoring_request_privacy.md @@ -14,20 +14,32 @@ Williamson, D. M., Xi, X., & Breyer, F. J. (2012). A framework for evaluation an ## Privacy and provenance rationale -Reference-free RAG evaluation records provenance identities (system configuration, retrieval run, query revision fingerprints) without embedding raw query, context, answer, or source text. Managed identifiers must reject free-form content so the canonical scoring artifact cannot become a side channel. The host application may retain source material only under its own purpose-bound security, privacy, retention, and authorization controls. +Reference-free RAG evaluation records provenance identities (system configuration, retrieval run, query revision fingerprints) without embedding raw query, context, answer, or source text. Managed identifiers must reject free-form content so the canonical scoring artifact cannot become a side channel. The host application may retain source material and controlled-transformation evidence only under its own purpose-bound security, privacy, retention, and authorization controls. ## Controlled perturbation anchors -`RAGPerturbationAnchor` preregisters a finite construct-specific expectation against two distinct governed scoring-request fingerprints. The package currently represents these hypotheses: +`RAGPerturbationAnchor` preregisters a finite construct-specific expectation against two canonical governed `ScoringRequest` values. The builder verifies that the requests share assessment, rubric, system-run, task-family, occasion, evidence-regime, candidate-visibility, system-configuration, engine-policy, and split provenance. It then enforces one permitted changed axis: + +- response perturbations preserve query and retrieval provenance and change the response artifact; +- retrieval perturbations preserve the query and response artifact and change retrieval provenance; and +- query perturbations change the governed query revision and may produce new retrieval and response artifacts. + +The anchor also records exact `perturbation_specification_fingerprint` and `perturbation_run_fingerprint` values. These identities bind the source-free numerical artifact to the host's governed transformation protocol and execution evidence. The core does not claim that hashes alone prove a semantic transformation such as “unsupported claim” or “semantic paraphrase”; the authorized host must retain that evidence. + +### Scientific claim boundary + +RAGAS and ARES directly motivate separable grounded-generation/faithfulness and retrieval/context-relevance constructs. Accordingly, unsupported-claim, explicit-contradiction, and irrelevant-context anchors are marked `literature_aligned_construct`. This label means the named construct aligns with those primary sources; it does **not** mean the papers established this package's exact perturbation or expected direction. + +The following mappings are package-owned `model_design_hypothesis` values because the cited RAGAS and ARES papers do not directly establish them: -- unsupported claim or explicit contradiction -> `grounded_generation` decreases; -- irrelevant context -> `retrieval_relevance` decreases; - required-evidence removal -> `coverage_or_completeness_proxy` decreases; - citation-target swap -> `citation_attribution` decreases; - semantic query paraphrase or style-only rewrite -> `robustness` remains approximately invariant; and - unanswerable query -> `answerability_and_abstention` increases. -These are controlled-validation expectations, not observed scores, statistical tests, causal effects, ground-truth labels, or evidence that a particular judge can identify world correctness. Context-groundedness remains conditional on the declared evidence regime. A successful anchor regression therefore demonstrates contract identity and preregistered direction semantics only; empirical validation still requires governed observations, calibrated evaluator/rater evidence, subgroup/DIF and drift analysis where applicable, and recovery or human-audit evidence commensurate with the intended interpretation. +Unsupported claim or explicit contradiction -> `grounded_generation` decreases, and irrelevant context -> `retrieval_relevance` decreases, are also preregistered expected directions rather than observations or causal conclusions. Every anchor direction in this package remains a hypothesis to test. + +These anchors are not observed scores, statistical tests, causal effects, ground-truth labels, or evidence that a particular judge can identify world correctness. Context-groundedness remains conditional on the declared evidence regime. A successful anchor regression demonstrates canonical identity, allowed-axis validation, governed transformation provenance, and preregistered direction semantics only. Empirical validation still requires governed observations, calibrated evaluator/rater evidence, subgroup/DIF and drift analysis where applicable, and recovery or human-audit evidence commensurate with the intended interpretation. ## Numerical and ownership boundary @@ -38,10 +50,10 @@ The anchor layer performs no likelihood, calibration, thresholding, uncertainty `python/fast_mlsirm/scoring/rag.py`: - `build_rag_scoring_request` validates `system_configuration_id` via `descriptive_identifier` and allowlists caller metadata; -- `RAGPerturbationKind` and `RAGPerturbationDirection` close the perturbation vocabulary; -- `RAGPerturbationAnchor` derives package-owned construct/direction semantics, rejects no-op baseline/perturbed request pairs, and content-addresses immutable anchor content; and -- `build_rag_perturbation_anchor` exposes only governed identities and perturbation kind, with no raw-content parameters. +- `RAGPerturbationKind`, `RAGPerturbationDirection`, and `RAGPerturbationConstructBasis` close the perturbation vocabulary and distinguish literature-aligned constructs from model-design hypotheses; +- `RAGPerturbationAnchor` is factory-sealed, derives package-owned construct/direction semantics, and content-addresses the request pair plus transformation specification/run provenance; and +- `build_rag_perturbation_anchor` accepts canonical governed request objects, rejects unrelated or mixed-axis pairs, serializes only fingerprints, and exposes no raw-content parameters. -`tests/test_scoring_rag_perturbation_anchors.py` binds the finite mapping, deterministic fingerprints, fail-closed malformed/unknown inputs, distinct request identities, and source-free public signature. +`tests/test_scoring_rag_perturbation_anchors.py` binds finite semantics, deterministic identities, factory sealing, and the source-free public signature. `tests/test_scoring_rag_perturbation_relationships.py` binds kind-specific changed axes, unrelated-pair rejection, transformation provenance, and the scientific claim-basis distinction. This bounded contract advances issue #607 stage-2 validation infrastructure. It does not by itself establish RAG system validity, a scalar quality score, evaluator interchangeability, retrieval recall, world correctness, calibration, DIF invariance, or release readiness. From b6af39594d91bd116a7c31225703863460569ebf Mon Sep 17 00:00:00 2001 From: Seongho Bae Date: Thu, 13 Aug 2026 09:26:29 +0900 Subject: [PATCH 7/7] docs(changelog): record governed perturbation relationships --- docs/changelog.d/777-rag-perturbation-anchors.md | 3 ++- 1 file changed, 2 insertions(+), 1 deletion(-) diff --git a/docs/changelog.d/777-rag-perturbation-anchors.md b/docs/changelog.d/777-rag-perturbation-anchors.md index 395a6ffd5..6060db870 100644 --- a/docs/changelog.d/777-rag-perturbation-anchors.md +++ b/docs/changelog.d/777-rag-perturbation-anchors.md @@ -3,4 +3,5 @@ ## Added - Added source-free, content-addressed reference-free RAG perturbation anchors with finite preregistered construct/direction semantics for unsupported claims, contradictions, irrelevant context, required-evidence removal, citation swaps, semantic paraphrases, style-only rewrites, and unanswerable queries. -- Require distinct governed baseline and perturbed request fingerprints and fail closed on unknown perturbation semantics or malformed identities. Expected directions are validation hypotheses, not claims that an observed system actually changed or that an evaluator is ground truth. +- Require canonical governed baseline and perturbed `ScoringRequest` values, reject unrelated or mixed-axis pairs, and bind each anchor to exact perturbation specification/run fingerprints while serializing only source-free identities. +- Distinguish literature-aligned constructs from package-owned model-design hypotheses. Every expected direction remains a validation hypothesis, not a claim that the cited papers established the exact perturbation, that an observed system actually changed, or that an evaluator is ground truth.