Add contextual polytomous LLM judge and IRT response contract - #733
Conversation
|
Important Review skippedAuto incremental reviews are disabled on this repository. Please check the settings in the CodeRabbit UI or the ⚙️ Run configurationConfiguration used: Path: .coderabbit.yaml Review profile: CHILL Plan: Pro Plus Run ID: You can disable this status message by setting the Use the checkbox below for a quick retry:
📝 WalkthroughWalkthroughThe pull request adds a provider-neutral LLM judge, strict structured-result parsing, IRT response validation, public package exports, tests, and README documentation. ChangesLLM Judge and IRT Integration
Estimated code review effort: 4 (Complex) | ~45 minutes Sequence Diagram(s)sequenceDiagram
participant Caller
participant ContextualOrchestratorJudge
participant ContextualOrchestrator
participant LLMJudgeResult
Caller->>ContextualOrchestratorJudge: submit task, answer, and criteria
ContextualOrchestratorJudge->>ContextualOrchestrator: route evaluation prompt
ContextualOrchestrator-->>ContextualOrchestratorJudge: return JSON response and trace usage
ContextualOrchestratorJudge->>LLMJudgeResult: validate scores or categories
LLMJudgeResult-->>Caller: return score, acceptance, metadata, and IRT row
🚥 Pre-merge checks | ✅ 5✅ Passed checks (5 passed)
✨ Finishing Touches📝 Generate docstrings
🧪 Generate unit tests (beta)
Thanks for using CodeRabbit! It's free for OSS, and your support helps us grow. If you like it, consider giving us a shout-out. Comment |
![coderabbitai[bot]](https://avatars.githubusercontent.com/in/347564?s=60&v=4){kind=small}
There was a problem hiding this comment.
Actionable comments posted: 2
🧹 Nitpick comments (8)
python/fast_mlsirm/llm_judge.py (4)
180-186: 🎯 Functional Correctness | 🔵 Trivial | ⚡ Quick winRemove the redundant
intcall and clamp the lower bound.
math.flooralready returns anint, which Ruff reports as RUF046.LLMJudgeResultcan also be constructed directly with a score outside 0..1, because the dataclass does not validatecriterion_scores. A negative score then produces a negative category index, whichvalidate_irt_response_matrixlater rejects with a confusing message.♻️ Proposed change
return tuple( - min( - n_categories - 1, - int(math.floor(float(self.criterion_scores[criterion_id]) * n_categories)), - ) + min( + n_categories - 1, + max(0, math.floor(float(self.criterion_scores[criterion_id]) * n_categories)), + ) for criterion_id in criterion_ids )🤖 Prompt for AI Agents
Verify each finding against current code. Fix only still-valid issues, skip the rest with a brief reason, keep changes minimal, and validate. In `@python/fast_mlsirm/llm_judge.py` around lines 180 - 186, Update the category computation in LLMJudgeResult to remove the redundant int conversion around math.floor and clamp the calculated category to a minimum of 0 as well as the existing maximum of n_categories - 1, ensuring out-of-range criterion_scores always produce valid category indices.Source: Linters/SAST tools
425-433: 📐 Maintainability & Code Quality | 🔵 Trivial | 💤 Low value
__all__ordering does not match Ruff RUF022 in two modules. Ruff's isort-style order lists SCREAMING_SNAKE_CASE constants before class-like names, and both new modules mix the two groups.
python/fast_mlsirm/llm_judge.py#L425-L433: moveMAX_JUDGE_CATEGORIES,MAX_JUDGE_CRITERIA, andMAX_JUDGE_TEXT_CHARACTERSaboveContextualOrchestratorJudge.python/fast_mlsirm/irt_contract.py#L80-L80: moveMIN_IRT_ITEMSbeforeIRTItemType.🤖 Prompt for AI Agents
Verify each finding against current code. Fix only still-valid issues, skip the rest with a brief reason, keep changes minimal, and validate. In `@python/fast_mlsirm/llm_judge.py` around lines 425 - 433, Reorder the __all__ entries in python/fast_mlsirm/llm_judge.py lines 425-433 so MAX_JUDGE_CATEGORIES, MAX_JUDGE_CRITERIA, and MAX_JUDGE_TEXT_CHARACTERS precede ContextualOrchestratorJudge while preserving the remaining order. Also reorder __all__ in python/fast_mlsirm/irt_contract.py line 80 so MIN_IRT_ITEMS precedes IRTItemType, with no other changes.Source: Linters/SAST tools
338-359: 🔒 Security & Privacy | 🔵 Trivial | ⚡ Quick winOptional: harden the prompt delimiters against tag injection.
task,answer, andreference_blockare inserted between literal<task>,<answer>, and<reference>tags. An answer that contains</answer>can close the block and place attacker text in the judge instruction region. The system message tells the model to treat the content as data, which reduces but does not remove the risk. Use per-call random delimiters, or send the payload as a single JSON object.🤖 Prompt for AI Agents
Verify each finding against current code. Fix only still-valid issues, skip the rest with a brief reason, keep changes minimal, and validate. In `@python/fast_mlsirm/llm_judge.py` around lines 338 - 359, Harden the prompt construction in the message-building flow by replacing predictable XML-like delimiters around task, answer, and reference_block with per-call random delimiters or a single JSON payload. Ensure the judge receives all three values as data without allowing embedded closing tags to alter instruction boundaries, while preserving the existing criterion_payload and system guidance.
365-367: 📐 Maintainability & Code Quality | 🔵 Trivial | 💤 Low valueReuse of the
acceptedname hides the validation result.Line 365 binds the parsed advisory value to
accepted. Line 409 overwrites it with the runtime-derived value. The validation at line 366 is still effective, but the shared name suggests the parsed value is used. Rename the parsed value.♻️ Proposed change
- accepted = parsed.get("accepted") - if accepted is not None and not isinstance(accepted, bool): + advisory_accepted = parsed.get("accepted") + if advisory_accepted is not None and not isinstance(advisory_accepted, bool): raise JudgeFormatError("accepted must be a boolean when present")Also applies to: 409-409
🤖 Prompt for AI Agents
Verify each finding against current code. Fix only still-valid issues, skip the rest with a brief reason, keep changes minimal, and validate. In `@python/fast_mlsirm/llm_judge.py` around lines 365 - 367, Rename the parsed advisory value currently assigned from parsed.get("accepted") in the judge-format validation flow to a distinct name, and update the associated None/type check to use it. Keep the runtime-derived accepted value at line 409 under its existing name without changing validation behavior.tests/test_llm_judge.py (2)
121-141: 📐 Maintainability & Code Quality | 🔵 Trivial | ⚡ Quick winAdd coverage for the orchestrator failure paths.
The suite covers malformed JSON and non-integral categories. It does not cover a completion that omits
answer, a completion that omitsrationale, or a completion that is not a mapping. These paths currently raise different exception types, as flagged inpython/fast_mlsirm/llm_judge.pyat lines 363-368. Tests here would lock the error contract.🤖 Prompt for AI Agents
Verify each finding against current code. Fix only still-valid issues, skip the rest with a brief reason, keep changes minimal, and validate. In `@tests/test_llm_judge.py` around lines 121 - 141, Extend the LLM judge tests around ContextualOrchestratorJudge.judge to cover payloads missing answer, payloads missing rationale, and non-mapping completions. Assert the specific exception type and expected error behavior for each path, preserving the existing malformed-JSON and non-integral-category coverage.
66-76: 📐 Maintainability & Code Quality | 🔵 Trivial | 💤 Low valueOptional: use
pytest.raisesfor the failure paths.The manual
try/except/elseblocks duplicate whatpytest.raisesprovides.tests/test_irt_contract.pyalready usespytest.raises. Aligning both files keeps one style and shortens the tests.♻️ Proposed change for one block
- try: - ContextualOrchestratorJudge(_FakeOrchestrator("not json")).judge( - task="task", - answer="answer", - criteria=CRITERIA, - ) - except JudgeFormatError: - pass - else: # pragma: no cover - raise AssertionError("invalid judge response should fail closed") + with pytest.raises(JudgeFormatError): + ContextualOrchestratorJudge(_FakeOrchestrator("not json")).judge( + task="task", + answer="answer", + criteria=CRITERIA, + )Add
import pytestat the top of the file.Also applies to: 113-118, 151-161
🤖 Prompt for AI Agents
Verify each finding against current code. Fix only still-valid issues, skip the rest with a brief reason, keep changes minimal, and validate. In `@tests/test_llm_judge.py` around lines 66 - 76, Refactor the failure-path assertions in test_judge_rejects_malformed_decisions_and_derives_acceptance and the additional blocks around the referenced cases to use pytest.raises instead of manual try/except/else handling. Add the pytest import and preserve each test’s expected JudgeFormatError assertion and fail-closed behavior.tests/test_irt_contract.py (1)
41-47: 📐 Maintainability & Code Quality | 🔵 Trivial | 💤 Low valueOptional: escape the
matchpattern and add category-bound coverage.
match="0..2"is a regex. The dots match any character, so the test can pass on an unrelated message. Ruff reports RUF043 for the same reason. The suite also does not cover rejection of an out-of-rangen_categories, orn_categoriespassed withitem_type="dichotomous".♻️ Proposed change
- with pytest.raises(ValueError, match="0..2"): + with pytest.raises(ValueError, match=re.escape("0..2")): validate_irt_response_matrix([[0, 3]], "polytomous", n_categories=3) + with pytest.raises(ValueError, match="n_categories is only valid"): + validate_irt_response_matrix([[0, 1]], "dichotomous", n_categories=2) + with pytest.raises(ValueError, match="n_categories"): + validate_irt_response_matrix([[0, 1]], "polytomous", n_categories=1)Add
import reat the top of the file.🤖 Prompt for AI Agents
Verify each finding against current code. Fix only still-valid issues, skip the rest with a brief reason, keep changes minimal, and validate. In `@tests/test_irt_contract.py` around lines 41 - 47, Update test_polytomous_contract_rejects_invalid_categories_and_shape to use an escaped or otherwise literal regex for the “0..2” error match, then add coverage for invalid n_categories bounds and for supplying n_categories with item_type="dichotomous", asserting the appropriate ValueError messages.Source: Linters/SAST tools
README.md (1)
92-99: 📐 Maintainability & Code Quality | 🔵 Trivial | ⚡ Quick winRestore the ADR links and pin them to an immutable ref.
Both
mainURLs return404. Correct the paths or restore the documents, then link to a commit or version tag instead ofmain.🤖 Prompt for AI Agents
Verify each finding against current code. Fix only still-valid issues, skip the rest with a brief reason, keep changes minimal, and validate. In `@README.md` around lines 92 - 99, Update the ADR 0005 and ADR 0006 links in the README to use valid document paths, and replace the mutable main branch reference with an immutable commit or version tag. Preserve the existing ADR references and descriptive text.Source: Coding guidelines
🤖 Prompt for all review comments with AI agents
Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.
Inline comments:
In `@python/fast_mlsirm/llm_judge.py`:
- Around line 72-78: Update JudgeCriterion.__post_init__ to validate
criterion_id, description, and weight types before calling regex, string, or
numeric operations, raising ValueError for invalid non-string or non-numeric
inputs. Apply the same conversion protection in _criteria around mapping weight
values so invalid inputs consistently raise ValueError rather than TypeError.
- Around line 363-368: Update the response-processing flow around _bounded_text
calls for completion["answer"] and parsed["rationale"] so malformed
model-controlled values raise JudgeFormatError instead of plain ValueError.
Preserve the existing validation and bounded-text behavior while translating
these failures into the exception type callers already handle.
---
Nitpick comments:
In `@python/fast_mlsirm/llm_judge.py`:
- Around line 180-186: Update the category computation in LLMJudgeResult to
remove the redundant int conversion around math.floor and clamp the calculated
category to a minimum of 0 as well as the existing maximum of n_categories - 1,
ensuring out-of-range criterion_scores always produce valid category indices.
- Around line 425-433: Reorder the __all__ entries in
python/fast_mlsirm/llm_judge.py lines 425-433 so MAX_JUDGE_CATEGORIES,
MAX_JUDGE_CRITERIA, and MAX_JUDGE_TEXT_CHARACTERS precede
ContextualOrchestratorJudge while preserving the remaining order. Also reorder
__all__ in python/fast_mlsirm/irt_contract.py line 80 so MIN_IRT_ITEMS precedes
IRTItemType, with no other changes.
- Around line 338-359: Harden the prompt construction in the message-building
flow by replacing predictable XML-like delimiters around task, answer, and
reference_block with per-call random delimiters or a single JSON payload. Ensure
the judge receives all three values as data without allowing embedded closing
tags to alter instruction boundaries, while preserving the existing
criterion_payload and system guidance.
- Around line 365-367: Rename the parsed advisory value currently assigned from
parsed.get("accepted") in the judge-format validation flow to a distinct name,
and update the associated None/type check to use it. Keep the runtime-derived
accepted value at line 409 under its existing name without changing validation
behavior.
In `@README.md`:
- Around line 92-99: Update the ADR 0005 and ADR 0006 links in the README to use
valid document paths, and replace the mutable main branch reference with an
immutable commit or version tag. Preserve the existing ADR references and
descriptive text.
In `@tests/test_irt_contract.py`:
- Around line 41-47: Update
test_polytomous_contract_rejects_invalid_categories_and_shape to use an escaped
or otherwise literal regex for the “0..2” error match, then add coverage for
invalid n_categories bounds and for supplying n_categories with
item_type="dichotomous", asserting the appropriate ValueError messages.
In `@tests/test_llm_judge.py`:
- Around line 121-141: Extend the LLM judge tests around
ContextualOrchestratorJudge.judge to cover payloads missing answer, payloads
missing rationale, and non-mapping completions. Assert the specific exception
type and expected error behavior for each path, preserving the existing
malformed-JSON and non-integral-category coverage.
- Around line 66-76: Refactor the failure-path assertions in
test_judge_rejects_malformed_decisions_and_derives_acceptance and the additional
blocks around the referenced cases to use pytest.raises instead of manual
try/except/else handling. Add the pytest import and preserve each test’s
expected JudgeFormatError assertion and fail-closed behavior.
🪄 Autofix
Fix all unresolved CodeRabbit comments on this PR:
- Push a commit to this branch (recommended)
- Create a new PR with the fixes
ℹ️ Review info
⚙️ Run configuration
Configuration used: Path: .coderabbit.yaml
Review profile: CHILL
Plan: Pro Plus
Run ID: 3bbd1d86-ba2c-4d1c-9bcd-e3de89572a2a
📒 Files selected for processing (6)
README.mdpython/fast_mlsirm/__init__.pypython/fast_mlsirm/irt_contract.pypython/fast_mlsirm/llm_judge.pytests/test_irt_contract.pytests/test_llm_judge.py
|
@opencode-agent address Bounded exact-current repair handoff for PR #733. Before writing, refetch source head Two current unresolved CodeRabbit findings are valid and must be fixed test-first on the final integrated branch: (1) public Also verify the still-current review nits before finalizing: use JSON or otherwise injection-safe data framing instead of predictable XML-like task/answer/reference delimiters; fix the broken mutable contextual-orchestrator ADR links by pinning valid immutable refs/paths; make category projection fail closed or validate criterion scores rather than silently manufacturing categories from out-of-range values; fix literal-regex/category-bound tests and RUF022 ordering where applicable. Do not merely clamp scientifically invalid scores into a valid category without an explicit validated contract. Keep provider calls outside the psychometric core and preserve contextual-orchestrator as a read-only external dependency here. Do not persist raw task/answer/reference content beyond the call boundary or broaden provider/reviewer credentials. After fixes, require focused tests, |
|
@coderabbitai review |
✅ Action performedReview finished.
|
There was a problem hiding this comment.
Actionable comments posted: 2
Caution
Some comments are outside the diff and can’t be posted inline due to platform limitations.
⚠️ Outside diff range comments (3)
python/fast_mlsirm/llm_judge.py (3)
241-256: 🩺 Stability & Availability | 🟠 Major | ⚡ Quick winBound criteria iteration during normalization.
The maximum is checked only after the loop ends. An infinite iterable of valid criteria never reaches Line 255. It can hang the judge before any orchestrator call.
Reject the input when
normalizedreachesMAX_JUDGE_CRITERIAinside the loop.Proposed fix
normalized: list[JudgeCriterion] = [] for value in values: + if len(normalized) >= MAX_JUDGE_CRITERIA: + raise ValueError(f"criteria must contain 1..{MAX_JUDGE_CRITERIA} values") if isinstance(value, JudgeCriterion): criterion = valueAs per coding guidelines: "For fuzzed arbitrary input, code must either succeed or raise a documented benign exception; panics, hangs, ... are bugs."
🤖 Prompt for AI Agents
Verify each finding against current code. Fix only still-valid issues, skip the rest with a brief reason, keep changes minimal, and validate. In `@python/fast_mlsirm/llm_judge.py` around lines 241 - 256, Update _criteria to enforce the MAX_JUDGE_CRITERIA limit during iteration, raising the existing documented ValueError as soon as normalized reaches the maximum rather than waiting for the iterable to end. Preserve the current normalization of valid JudgeCriterion and mapping values, and retain the final check for empty input.Source: Coding guidelines
128-128: 🎯 Functional Correctness | 🟡 Minor | ⚡ Quick winValidate criterion-score keys before sorting.
LLMJudgeResultis a public API. Mixedcriterion_scoreskeys causesorted()to raiseTypeErrorbeforeJudgeFormatError. Validate thatcriterion_scoresis a mapping with string keys before sorting. Validate thatcriterion_categories, when provided, is a mapping before converting its keys to a set.🤖 Prompt for AI Agents
Verify each finding against current code. Fix only still-valid issues, skip the rest with a brief reason, keep changes minimal, and validate. In `@python/fast_mlsirm/llm_judge.py` at line 128, Update the LLMJudgeResult validation flow before sorting criterion_scores: verify criterion_scores is a mapping whose keys are all strings, raising JudgeFormatError for invalid input instead of allowing sorted() to raise TypeError. When criterion_categories is provided, validate that it is a mapping before converting its keys to a set; preserve normal processing for valid inputs.Source: Coding guidelines
113-210: 🎯 Functional Correctness | 🟡 Minor | ⚡ Quick winAdd a primary psychometric citation to ADR 0006.
ADR 0006 already keeps equal-width projection experimental and blocks production IRT claims. It does not provide a full primary-source basis for mapping continuous scores to equal-width MLSIRM/MLS2PLM categories. Keep
LLMJudgeResult.to_irt_rownon-production until validated cut-points and calibration evidence exist.🤖 Prompt for AI Agents
Verify each finding against current code. Fix only still-valid issues, skip the rest with a brief reason, keep changes minimal, and validate. In `@python/fast_mlsirm/llm_judge.py` around lines 113 - 210, Add a primary psychometric citation to ADR 0006 supporting the mapping of continuous scores into equal-width MLSIRM/MLS2PLM categories, while preserving the existing experimental and non-production status of LLMJudgeResult.to_irt_row until validated cut-points and calibration evidence are available.Source: Coding guidelines
🧹 Nitpick comments (3)
tests/test_llm_judge.py (3)
164-177: 🎯 Functional Correctness | 🔵 Trivial | ⚡ Quick winTest malformed answer and rationale types.
The current cases cover missing fields and a non-mapping completion. Add completions with a non-string
answerand a non-stringrationale. Each case must raiseJudgeFormatErrorinstead of leakingValueErrororTypeError.The PR objective requires
JudgeFormatErrorfor malformed completion answers and rationales.🤖 Prompt for AI Agents
Verify each finding against current code. Fix only still-valid issues, skip the rest with a brief reason, keep changes minimal, and validate. In `@tests/test_llm_judge.py` around lines 164 - 177, Extend test_judge_rejects_missing_or_malformed_model_fields with completion mappings whose answer and rationale fields contain non-string values, and assert each raises JudgeFormatError. Ensure the cases continue covering the existing missing-field and non-mapping inputs.
155-161: 🎯 Functional Correctness | 🔵 Trivial | ⚡ Quick winAdd out-of-range category regression cases.
This test covers only a fractional value. Also test
-1andcategory_countforcategory_count=3. Both values must raiseJudgeFormatError; otherwise an accidental clamp or acceptance bug can create invalid IRT rows.The PR objective requires category projection to fail closed or validate criterion scores instead of silently accepting out-of-range values.
🤖 Prompt for AI Agents
Verify each finding against current code. Fix only still-valid issues, skip the rest with a brief reason, keep changes minimal, and validate. In `@tests/test_llm_judge.py` around lines 155 - 161, Add regression cases alongside the existing fractional-value test in the contextual judge tests, using category scores of -1 and category_count (3) with category_count=3. Assert that each raises JudgeFormatError, preserving the existing integer-format validation and ensuring out-of-range scores are rejected rather than clamped or accepted.
73-76: 🔒 Security & Privacy | 🔵 Trivial | ⚡ Quick winCover all untrusted prompt fields and JSON edge cases.
The test checks only
taskandanswer. Pass a reference answer and values containing quotes, newlines, and instruction-like text. Assert thattask,answer,reference, andcriteriaround-trip throughjson.loadsas data.The PR objective requires injection-safe JSON or equivalent framing for task, answer, and reference data.
🤖 Prompt for AI Agents
Verify each finding against current code. Fix only still-valid issues, skip the rest with a brief reason, keep changes minimal, and validate. In `@tests/test_llm_judge.py` around lines 73 - 76, Expand the test around the orchestrator prompt payload to supply reference-answer and criteria values containing quotes, newlines, and instruction-like text, alongside task and answer. After parsing with json.loads, assert that task, answer, reference, and criteria exactly round-trip as data, covering all untrusted prompt fields and confirming injection-safe framing.
🤖 Prompt for all review comments with AI agents
Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.
Inline comments:
In `@python/fast_mlsirm/llm_judge.py`:
- Around line 80-85: Update the weight validation in the criterion
initialization flow to accept only exact built-in int and float types before
calling float(), rejecting numeric subclasses and preventing custom __float__
execution. Preserve the existing TypeError for non-numeric values and the
finite-number ValueError for conversion failures.
In `@tests/test_llm_judge.py`:
- Around line 179-185: Update test_judge_criteria_reject_invalid_runtime_types
to expect ValueError for invalid JudgeCriterion fields, and add mapping-based
_criteria cases covering invalid criterion identifiers, descriptions, and
weights. Include a weight object that detects or rejects unintended
conversion-hook invocation, preserving the stable ValueError contract for both
public JudgeCriterion and _criteria validation.
---
Outside diff comments:
In `@python/fast_mlsirm/llm_judge.py`:
- Around line 241-256: Update _criteria to enforce the MAX_JUDGE_CRITERIA limit
during iteration, raising the existing documented ValueError as soon as
normalized reaches the maximum rather than waiting for the iterable to end.
Preserve the current normalization of valid JudgeCriterion and mapping values,
and retain the final check for empty input.
- Line 128: Update the LLMJudgeResult validation flow before sorting
criterion_scores: verify criterion_scores is a mapping whose keys are all
strings, raising JudgeFormatError for invalid input instead of allowing sorted()
to raise TypeError. When criterion_categories is provided, validate that it is a
mapping before converting its keys to a set; preserve normal processing for
valid inputs.
- Around line 113-210: Add a primary psychometric citation to ADR 0006
supporting the mapping of continuous scores into equal-width MLSIRM/MLS2PLM
categories, while preserving the existing experimental and non-production status
of LLMJudgeResult.to_irt_row until validated cut-points and calibration evidence
are available.
---
Nitpick comments:
In `@tests/test_llm_judge.py`:
- Around line 164-177: Extend
test_judge_rejects_missing_or_malformed_model_fields with completion mappings
whose answer and rationale fields contain non-string values, and assert each
raises JudgeFormatError. Ensure the cases continue covering the existing
missing-field and non-mapping inputs.
- Around line 155-161: Add regression cases alongside the existing
fractional-value test in the contextual judge tests, using category scores of -1
and category_count (3) with category_count=3. Assert that each raises
JudgeFormatError, preserving the existing integer-format validation and ensuring
out-of-range scores are rejected rather than clamped or accepted.
- Around line 73-76: Expand the test around the orchestrator prompt payload to
supply reference-answer and criteria values containing quotes, newlines, and
instruction-like text, alongside task and answer. After parsing with json.loads,
assert that task, answer, reference, and criteria exactly round-trip as data,
covering all untrusted prompt fields and confirming injection-safe framing.
🪄 Autofix
Fix all unresolved CodeRabbit comments on this PR:
- Push a commit to this branch (recommended)
- Create a new PR with the fixes
ℹ️ Review info
⚙️ Run configuration
Configuration used: Path: .coderabbit.yaml
Review profile: CHILL
Plan: Pro Plus
Run ID: 98402257-7f0c-41f7-8f1e-7ac0f3340a4c
📒 Files selected for processing (5)
README.mdpython/fast_mlsirm/irt_contract.pypython/fast_mlsirm/llm_judge.pytests/test_irt_contract.pytests/test_llm_judge.py
🚧 Files skipped from review as they are similar to previous changes (3)
- README.md
- python/fast_mlsirm/irt_contract.py
- tests/test_irt_contract.py
![opencode-agent[bot]](https://avatars.githubusercontent.com/in/1549082?s=60&v=4){kind=small}
There was a problem hiding this comment.
Pull request overview
OpenCode cannot approve yet because required coverage evidence did not pass.
Review outcome
1. HIGH .github/workflows/opencode-review.yml:1 - Coverage evidence did not prove required test/docstring evidence
-
Problem: The required coverage-evidence job result was
failure, so OpenCode cannot establish approval sufficiency for this head. -
Root cause: Automated approval is only valid when the same-head coverage-evidence job proves supported repository test suites passed and configured docstring gates passed or were advisory, or reports not applicable because no supported source files or package manifests exist. Missing, failed, skipped, unavailable, or unsupported-tooling test evidence is a blocker.
-
Fix: Install or configure the repository test/docstring evidence tooling when source files or package manifests exist, rerun the current-head coverage-evidence job, and approve only after it reports
successwith required evidence or explicit no-source not-applicable evidence. -
Regression test: Keep the approval branch checking
needs.coverage-evidence.result == successbefore posting APPROVE, and publish REQUEST_CHANGES when coverage-evidence blocker states such as cancelled, skipped, failed, unsupported-tooling, or below-100 evidence are present. -
Result: REQUEST_CHANGES
-
Reason: coverage-evidence result was
failure, so required test/docstring evidence was not proven for current headf1be6656b02b89bda7f82174091c87b7d9263bcc. -
Head SHA:
f1be6656b02b89bda7f82174091c87b7d9263bcc -
Workflow run: 31498432384
-
Workflow attempt: 1
Coverage evidence
Coverage evidence job did not run or did not publish coverage evidence.
Changed-File Evidence Map
flowchart LR
PR["PR changed files"] --> Evidence["OpenCode bounded evidence"]
Evidence --> S1["Changed file (4 files)"]
S1 --> I1["repository behavior"]
I1 --> R1["Review risk: Changed file (4 files)"]
R1 --> V1["required checks"]
Evidence --> S2["Test (2 files)"]
S2 --> I2["regression suite"]
I2 --> R2["Review risk: Test (2 files)"]
R2 --> V2["targeted test run"]
OpenCode Review Overview
Pull request overviewOpenCode cannot approve yet because required coverage evidence did not pass. Review outcome1. HIGH .github/workflows/opencode-review.yml:1 - Coverage evidence did not prove required test/docstring evidence
Coverage evidenceCoverage evidence job did not run or did not publish coverage evidence. Changed-File Evidence Mapflowchart LR
PR["PR changed files"] --> Evidence["OpenCode bounded evidence"]
Evidence --> S1["Changed file (4 files)"]
S1 --> I1["repository behavior"]
I1 --> R1["Review risk: Changed file (4 files)"]
R1 --> V1["required checks"]
Evidence --> S2["Test (2 files)"]
S2 --> I2["regression suite"]
I2 --> R2["Review risk: Test (2 files)"]
R2 --> V2["targeted test run"]
|
There was a problem hiding this comment.
Pull request overview
OpenCode cannot approve yet because required coverage evidence did not pass.
Review outcome
1. HIGH .github/workflows/opencode-review.yml:1 - Coverage evidence did not prove required test/docstring evidence
-
Problem: The required coverage-evidence job result was
failure, so OpenCode cannot establish approval sufficiency for this head. -
Root cause: Automated approval is only valid when the same-head coverage-evidence job proves supported repository test suites passed and configured docstring gates passed or were advisory, or reports not applicable because no supported source files or package manifests exist. Missing, failed, skipped, unavailable, or unsupported-tooling test evidence is a blocker.
-
Fix: Install or configure the repository test/docstring evidence tooling when source files or package manifests exist, rerun the current-head coverage-evidence job, and approve only after it reports
successwith required evidence or explicit no-source not-applicable evidence. -
Regression test: Keep the approval branch checking
needs.coverage-evidence.result == successbefore posting APPROVE, and publish REQUEST_CHANGES when coverage-evidence blocker states such as cancelled, skipped, failed, unsupported-tooling, or below-100 evidence are present. -
Result: REQUEST_CHANGES
-
Reason: coverage-evidence result was
failure, so required test/docstring evidence was not proven for current headbcfa4e9f02bbd03811cb330d91b73a0e0302943b. -
Head SHA:
bcfa4e9f02bbd03811cb330d91b73a0e0302943b -
Workflow run: 31504043242
-
Workflow attempt: 1
Coverage evidence
Coverage evidence job did not run or did not publish coverage evidence.
Changed-File Evidence Map
flowchart LR
PR["PR changed files"] --> Evidence["OpenCode bounded evidence"]
Evidence --> S1["Changed file (4 files)"]
S1 --> I1["repository behavior"]
I1 --> R1["Review risk: Changed file (4 files)"]
R1 --> V1["required checks"]
Evidence --> S2["Test (2 files)"]
S2 --> I2["regression suite"]
I2 --> R2["Review risk: Test (2 files)"]
R2 --> V2["targeted test run"]
|
Safety hold at current head |
|
Documentation follow-up on current head
This is a documentation-only follow-up; rerun the full required checks and current-head review. The prior neutral/no-report Strix result remains insufficient evidence until a structured report is produced. |
|
Linked security remediation on current head
Local validation: YAML contract, Ruff, and the targeted metadata test pass. New required checks and a fresh structured Strix report are required on |
|
Post-merge audit record: PR #733 merged as The merged tree does contain the Dependabot cooldown policy and its regression test. This comment preserves the governance discrepancy as follow-up evidence; it is not treated as proof that the review/Strix acceptance boundary was satisfied. |
1 participant
Summary
ContextualOrchestratorJudgethat routes every judge call through contextual-orchestratorLLMJudgeResult.to_irt_row()plus a multi-item dichotomous/polytomous response-matrix validatorValidation
uv run --project . --with pytest --with hypothesis pytest -q(3247 passed, 1 existing overflow warning)uv run --project . --with pytest --with hypothesis pytest -q tests/test_llm_judge.py tests/test_irt_contract.py(9 passed)python3 -m compileall -q python testsRelated
.Jules/palette.mdworking-tree change is intentionally not included.Summary by CodeRabbit
New Features
Documentation