feat(rag): govern reference-free RAG scoring requests - #613
Conversation
|
Important Review skippedAuto incremental reviews are disabled on this repository. Please check the settings in the CodeRabbit UI or the ⚙️ Run configurationConfiguration used: Path: .coderabbit.yaml Review profile: CHILL Plan: Pro Plus Run ID: You can disable this status message by setting the Use the checkbox below for a quick retry:
📝 WalkthroughWalkthroughAdds a governed RAG scoring adapter. It validates evidence, visibility, metadata, and fingerprints, maps RAG identities to canonical scoring axes, attaches managed provenance, and creates criterion-level ChangesRAG scoring adapter
Estimated code review effort: 3 (Moderate) | ~25 minutes Sequence Diagram(s)sequenceDiagram
participant RAGCaller
participant build_rag_scoring_request
participant ScoringRequest
RAGCaller->>build_rag_scoring_request: RAG identities, enums, fingerprints, and response
build_rag_scoring_request->>ScoringRequest: construct criterion-level request
ScoringRequest-->>RAGCaller: canonical ScoringRequest
Possibly related issues
Possibly related PRs
🚥 Pre-merge checks | ✅ 5✅ Passed checks (5 passed)
✨ Finishing Touches📝 Generate docstrings
🧪 Generate unit tests (beta)
Thanks for using CodeRabbit! It's free for OSS, and your support helps us grow. If you like it, consider giving us a shout-out. Comment |
|
@opencode-agent address Take a bounded writer lease on this Draft branch only if a final refetch still shows exact head Strict red-green scope for issue #607 first slice:
If any test expectation conflicts with an established protected-main shared scoring invariant, preserve the established invariant and make the narrowest test/API correction with an explicit reason rather than forking the shared contract. |
|
@opencode-agent One current RED-contract correction is required before implementing the adapter. Protected-main Preserve the existing calibration contract:
Correct the fail-first test/API before production implementation so RED specifies the existing shared scoring/calibration semantics rather than creating a parallel incompatible RAG interpretation. Keep every other bounded scope instruction unchanged. |
|
@opencode-agent address Take a bounded writer lease on this Draft branch only if a final refetch still shows exact head Current exact-head CI
|
|
@coderabbitai review |
|
![coderabbitai[bot]](https://avatars.githubusercontent.com/in/347564?s=60&v=4){kind=small}
There was a problem hiding this comment.
Actionable comments posted: 2
🤖 Prompt for all review comments with AI agents
Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.
Inline comments:
In `@python/fast_mlsirm/scoring/rag.py`:
- Around line 50-96: Update _rag_metadata in
python/fast_mlsirm/scoring/rag.py#L50-L96 to validate caller metadata against a
non-content schema or explicit allowlist, rejecting raw answer, query, context,
or source text even under unreserved keys while preserving package-managed
provenance fields. Add rejection tests for these raw-content cases in
tests/test_scoring_rag_request.py#L99-L136.
- Around line 109-112: Update python/fast_mlsirm/scoring/rag.py lines 109-112 to
add a separate response_id input while retaining system_configuration_id as
package-managed metadata; in the request construction at lines 171-180, map
respondent_id to system_run_id, response_id to response_id, and include the
configuration identifier in managed metadata. Update
tests/test_scoring_rag_request.py lines 83-95 to assert these corrected
identities, and lines 139-154 to verify distinct runs produce distinct
respondent identities for the same configuration.
🪄 Autofix
Fix all unresolved CodeRabbit comments on this PR:
- Push a commit to this branch (recommended)
- Create a new PR with the fixes
ℹ️ Review info
⚙️ Run configuration
Configuration used: Path: .coderabbit.yaml
Review profile: CHILL
Plan: Pro Plus
Run ID: f808ae08-fa98-4df3-9238-d9230cbda08a
📒 Files selected for processing (2)
python/fast_mlsirm/scoring/rag.pytests/test_scoring_rag_request.py
|
@opencode-agent address Take a bounded writer lease only if a final refetch still shows Draft source head CodeRabbit's two current-head findings are valid against issue #607's canonical hierarchy ( Then implement only the smallest GREEN in
Keep Draft. Do not weaken either CodeRabbit finding, add a second scoring hierarchy, add provider/RAGAS dependencies, alter Rust numerical kernels, touch #604, approve/merge/release, or broaden the slice beyond issue #607 stage 1. |
|
@coderabbitai review Please review exact current head |
|
|
|
@coderabbitai review Review exact current reconciled head |
|
Tip For best results, initiate chat on the files or code changes. 🧩 Analysis chain🏁 Script executed: #!/bin/bash
set -euo pipefail
expected_head='fdf0559cc0bf1bf3ef2941a2693fd57eb07a99c1'
expected_base='8db4bf358b0a469915d6c5e336054f4a4f9c6b46'
echo '== checked-out and expected commits =='
git rev-parse HEAD
git rev-parse "$expected_head"
git rev-parse "$expected_base"
git merge-base "$expected_base" "$expected_head"
echo
echo '== changed paths from requested base to requested head =='
git diff --name-status "$expected_base" "$expected_head"
echo
echo '== diff check =='
git diff --check "$expected_base" "$expected_head"
echo
echo '== workflow runs =='
gh run view 31324471491 --json databaseId,status,conclusion,headSha,name,url || true
gh run view 31324471463 --json databaseId,status,conclusion,headSha,name,url || true
gh run view 31324471464 --json databaseId,status,conclusion,headSha,name,url || true
echo
echo '== scoring module candidates =='
fd -t f . python/fast_mlsirm/scoring tests | sort
echo
echo '== outlines =='
ast-grep outline python/fast_mlsirm/scoring/rag.py --view expanded
ast-grep outline tests/test_scoring_rag_request.py --view expandedLength of output: 15296 🏁 Script executed: #!/bin/bash
set -euo pipefail
echo '== RAG adapter =='
sed -n '1,240p' python/fast_mlsirm/scoring/rag.py
echo
echo '== focused RAG tests =='
sed -n '1,290p' tests/test_scoring_rag_request.py
echo
echo '== shared contract definitions =='
rg -n -C 4 --type py '^(class ScoringRequest\b|def .*scoring.*request\b|def .*request.*scoring\b|def .*metadata\b|_METADATA|metadata:)' python/fast_mlsirm/scoring/contracts.py python/fast_mlsirm/scoring/authorization.py python/fast_mlsirm/scoring/_contract_safety.py
echo
echo '== shared contract outlines =='
ast-grep outline python/fast_mlsirm/scoring/contracts.py --match 'ScoringRequest|create|build|validate' --view expanded
ast-grep outline python/fast_mlsirm/scoring/authorization.py --view expanded
echo
echo '== package public surface =='
cat -n python/fast_mlsirm/scoring/__init__.py
echo
echo '== exact current diff =='
git diff --no-ext-diff --unified=80 8db4bf358b0a469915d6c5e336054f4a4f9c6b46..fdf0559cc0bf1bf3ef2941a2693fd57eb07a99c1 -- python/fast_mlsirm/scoring/rag.py tests/test_scoring_rag_request.pyLength of output: 44091 🏁 Script executed: #!/bin/bash
set -euo pipefail
echo '== authorization request factory =='
sed -n '1,185p' python/fast_mlsirm/scoring/authorization.py
echo
echo '== execution request model and factory =='
ast-grep outline python/fast_mlsirm/scoring/execution.py --match 'ScoringRequest|build_scoring_request|to_dict|request_fingerprint' --view expanded
rg -n -C 8 --type py '^(class ScoringRequest\b|def build_scoring_request\b|def .*fingerprint\b|def to_dict\b)' python/fast_mlsirm/scoring/execution.py
echo
echo '== shared metadata preflight =='
sed -n '250,450p' python/fast_mlsirm/scoring/_contract_safety.py
echo
echo '== identifier and fingerprint validators =='
rg -n -C 5 --type py '^(def (fingerprint|identifier|.*_id|.*metadata)|class .*Error)' python/fast_mlsirm/scoring/_validation.py
echo
echo '== metadata safety test cases =='
rg -n -C 5 --type py '(metadata.*(string|text|content)|freeze_metadata|invalid_.*metadata|metadata.*value)' \
tests/test_scoring_contract_metadata.py \
tests/test_scoring_contract_sensitive_metadata.py \
tests/test_scoring_contract_preflight_safety.py \
tests/test_scoring_contract_serialization_security.py
echo
echo '== direct RAG call-field validation tests =='
rg -n -C 3 --type py '(_request\(|system_configuration_id|system_run_id|response_id|query_id|query_testlet_id|occasion_id|evaluation_split)' tests/test_scoring_rag_request.pyLength of output: 41613 🏁 Script executed: #!/bin/bash
set -euo pipefail
base='8db4bf358b0a469915d6c5e336054f4a4f9c6b46'
echo '== ScoringRequest validation and factory =='
sed -n '415,625p' python/fast_mlsirm/scoring/execution.py
sed -n '850,970p' python/fast_mlsirm/scoring/execution.py
echo
echo '== descriptive identifier validation =='
sed -n '90,145p' python/fast_mlsirm/scoring/_validation.py
echo
echo '== RAG request references in current tree =='
rg -n -i -C 3 '(reference-free|RAGAS|ARES|RAGChecker|rag scoring|retrieved-context|world correctness|absolute recall)' \
README.md docs python tests || true
echo
echo '== changelog conventions and RAG fragments =='
fd -t f . docs/changelog.d | sort | tail -n 60
rg -n -i -C 3 '(changelog\.d|fragment|RAG|reference-free)' \
README.md docs tests/test_changelog_fragment_contract.py pyproject.toml .github 2>/dev/null || true
echo
echo '== same documentation search on requested base =='
git grep -n -i -E 'reference-free|RAGAS|ARES|RAGChecker|rag scoring|retrieved-context|world correctness|absolute recall' "$base" -- README.md docs python 2>/dev/null || true
echo
echo '== request IDs and metadata values passed through tests =='
rg -n -C 2 --type py 'metadata=\{|evaluation_split|metadata\[' tests/test_scoring_*.py | head -n 260Length of output: 201532
The source head and base match the requested commits. Only the two RAG paths differ. I found two current-head issues.
You are interacting with an AI system. |
|
@opencode-agent address Take a bounded writer lease only if a final refetch still shows Draft head A fresh exact-current CodeRabbit review identified two valid issues on predecessor Strict RED→GREEN scope:
Keep Draft. Do not alter canonical architecture PR #604, dependencies, workflows, version/release, provider SDKs, Rust/PyO3 numerical kernels, scoring schemas, review/merge state, or expand into RAG observations/calibration. Stop after one exact-head GREEN/renderer commit and report evidence. |
|
@opencode-agent address Take a bounded writer lease only if a final refetch still shows Draft head The source fix and required evidence are now present: Finish only the deterministic evidence closure for this exact head:
If the branch or protected base moves before your write, leave a receipt and stop rather than reconciling destructively. |
|
@opencode-agent address Take a bounded writer lease only if a final refetch still shows Draft head The branch has already been reconciled non-destructively with current protected main as a two-parent merge commit and now differs from main only by the four RAG contract paths. Preserve the accepted RAG contract and all current-main report UX behavior. Complete only deterministic release-note cleanup:
Keep Draft. Do not alter RAG request semantics, provider boundaries, dependencies/workflows/version/release, canonical architecture PR #604, mark Ready, approve, merge, or expand into calibration/model selection. Stop after exact-head deterministic evidence. |
|
@opencode-agent address Take a bounded writer lease on PR #613 only if a final refetch still shows exact head Current RCA is deterministic and narrow: exact-head Security/SAST are green and the feature tests pass; CI has one repository-contract failure because Execute only this bounded integration repair:
Do not change the RAG contract, model arithmetic, dependencies, workflows, credentials, version, release state, or architecture docs. Keep the PR Draft. Report/push the minimal audited diff only. |
|
Superseded bookkeeping note: an earlier exact-current writer handoff already covers head |
|
@opencode-agent address Take a bounded writer lease only if a final refetch still shows Draft head This PR has only four unique product paths for the governed RAG scoring-request boundary. Preserve the current validation/provenance contract and Rust-only future psychometric arithmetic. Non-destructively reconcile current protected main, including accepted #618 behavior. Then render/check |
|
@opencode-agent address Take a bounded writer lease only if a final refetch still shows Draft head Do not change the RAG contract. Run the focused RAG request tests, then synchronize only managed release notes with the authoritative changelog renderer ( |
|
Exact-head review finding on Keep the current source-writer lease intact; do not race the active changelog handoff. Once that writer is demonstrably finished, add a fail-first regression using a raw-content-shaped |
|
@jules address Fallback bounded source-writer handoff for exact Draft head Address the current-head privacy/content-minimization defect before release-note cleanup. First add a fail-first regression using a raw-content-shaped direct After RED→GREEN, run focused RAG request/security/fingerprint tests, then synchronize managed release notes only with |
|
@opencode-agent address Fresh bounded source/privacy repair for issue #607 after protected-main movement. Immediately before dispatch: Draft #613 exact head A current source/privacy defect is independently reproduced from the exact head:
Keep Draft. Do not touch canonical docs #604, dependencies/workflows/version/release, Ready/approval/merge state, or expand into RAG metrics/inference/retrieval/persistence. Stop source writes after one verified update; fresh exact-head CI/Security/SAST/review returns to the maintainer loop. |
|
@opencode-agent address Superseding exact-current privacy repair after #631 merged. Final pre-dispatch identities: Draft #613 head The exact head still contains the accepted #607 RAG adapter and the independently reproduced managed-identity privacy defect: caller-controlled Keep Draft. No RAG metric arithmetic/provider inference/retrieval/persistence, new schema hierarchy, dependency/workflow/version/release changes, canonical docs #604, Ready/approval/merge. Stop source writes after one coherent verified repair; fresh exact-head full CI/Security/SAST/review belongs to the maintainer loop. |
|
@jules address Fallback sole branch writer for exact Draft #613 head Preserve the accepted #607 RAG request contract and exact fail-first history. Reconcile the two accepted CodeQL 4.37.6 protected-main commits non-destructively. Then reproduce the managed-identity privacy RED at the public RAG request boundary: a raw-content-shaped caller Keep Draft and stop source writes after one coherent verified update. Do not add RAG metric arithmetic, provider inference/retrieval/persistence, another scoring schema, dependencies/workflows/version/release changes, canonical docs #604, Ready/approval/merge state, or create another PR. Fresh exact-head full CI/Security/SAST/current-head review belongs to the maintainer loop. |
|
Superseded by clean-lineage Draft #691 from exact protected |
1 participant
Buyer-visible gap
Issue #607 identifies a missing measurement boundary for reference-free RAG evaluation. Protected
mainhas reusable Assessment/Rubric/Scoring contracts and psychometric calibration primitives, but no canonical adapter that preserves query revision, system configuration versus stochastic run, generated-response identity, retrieval-run identity, evidence regime, and candidate visibility in one governed request.Test-first implementation
This branch began with an intentional RED contract in
tests/test_scoring_rag_request.py, then added the minimal validation/marshalling implementation inpython/fast_mlsirm/scoring/rag.pywhile reusing the authoritative sharedScoringRequest. Valid review findings were repaired test-first: caller metadata is allowlisted (evaluation_splitonly), package-managed RAG provenance cannot be spoofed, raw question/context/answer/source content cannot be smuggled under arbitrary metadata keys, and shared identity projectsrespondent_id=system_run_idwhile preserving a distinct generatedresponse_idandsystem_configuration_id.Bounded contract
prompt_only,retrieved_context,pooled_corpus,authoritative_corpus,human_anchor;candidate_blindorcandidate_visible_crossfit;Python owns validation/provenance marshalling only. Future likelihood, calibration, hierarchical, DIF, uncertainty, or other psychometric arithmetic remains Rust-owned.
Scientific boundary
Reference-free is not truth-free. Grounded generation, answer relevance, retrieval relevance, evidence utilization, coverage/completeness proxies, citation attribution, robustness, and answerability/abstention remain distinct constructs. A retrieved-context evidence regime alone cannot authorize world-correctness, absolute retrieval-recall, fairness, or deployment-validity claims. Canonical architecture/traceability changes belong to #604 rather than a competing documentation PR.
Exact-current identity and evidence
Freshly revalidated against GitHub state for this update:
main:8db4bf358b0a469915d6c5e336054f4a4f9c6b46;754f0716b072e551d682adca2639801aaad2e68b;compare_commits(main, head):ahead, merge base exactly current protected main, with only four net product paths: the RAG implementation, its tests, doctoring, and changelog fragment;31328633599: Rust/PyO3, package/reinstall/release acceptance, GPU-smoke and fuzz succeed; Python reaches the complete suite and fails onlytests/test_changelog_fragment_contract.py::test_repository_changelog_is_rendered_from_current_fragmentsbecauseCHANGELOG.mdhas not yet been regenerated fromdocs/changelog.d/607-rag-scoring-request-boundary.md;Any older body text or comments naming
fdf0559...are predecessor/historical claims and are not exact-current-head evidence.Remaining Draft gate
Keep Draft. The next source mutation must render/check the authoritative changelog without changing the RAG contract, then require fresh unchanged-head full CI/Security/SAST, current-head automated review, zero valid unresolved findings, and the repository's actual approval/branch-protection policy. If the head or protected base moves, discard predecessor acceptance evidence and refetch from scratch.
Advances #607.