fix(judge): normalize invalid criterion container errors

[seonghobae]

Test-first reliability gap

Protected main 914127ba227d3e02d0564aeeb4f27d76137610f9 now exposes the contextual LLM-judge contract from #733. The public ContextualOrchestratorJudge.judge(criteria=...) path normalizes malformed criterion fields to stable ValueError, but a criterion element that is neither JudgeCriterion nor a mapping still raises TypeError directly from _criteria.

That is inconsistent with the package's benign arbitrary-input/fuzzing contract and with adjacent criterion validation. This Draft starts with an intentional test-only RED requiring the same stable ValueError surface for a non-contract criterion element.

Valid RED

tests/test_llm_judge.py::test_judge_criteria_reject_non_contract_values_with_value_error must reach _criteria through the installed public judge and fail because current production raises TypeError. Setup/import/fixture/changelog failure is not valid evidence.

Minimum GREEN after RED proof

• change only the non-JudgeCriterion/non-mapping _criteria element error from TypeError to the existing stable ValueError validation family;
• preserve accepted criterion normalization, bounded iteration, criterion type/weight hardening, prompt JSON framing, scoring/category semantics, injected orchestrator behavior and public exports;
• run the focused judge suite and the full applicable Python 3.12/3.14, security/SAST and package gates on one unchanged head;
• add a small authoritative changelog fragment only after GREEN if repository policy requires it.

No numerical psychometric arithmetic, contextual-orchestrator code, dependencies/workflows, version/release, hosted state, or canonical #604 documentation belongs in this slice.

Summary by CodeRabbit

• Bug Fixes

  • Invalid judging criteria now consistently raise a ValueError with a clear, stable error message.

• Tests

  • Added coverage to verify handling of unsupported criterion values.

[coderabbitai]

Warning

Review limit reached

@seonghobae, you've reached your PR review limit, so we couldn't start this review.

Next review available in: 59 minutes

You've used all free OSS reviews for now. Wait for the free limit to reset to keep reviewing this public repository.

How can I continue?

After more reviews become available, a review can be triggered using the @coderabbitai review command as a PR comment. Alternatively, push new commits to this PR.

To avoid repeated limits, reduce automatic review volume by pausing incremental auto-reviews earlier, using label-based review opt-in, excluding WIP or generated PR titles, or requesting reviews manually when the PR is ready. If your team needs uninterrupted high-volume reviews, an organization admin can enable usage-based reviews.

How do review limits work?

CodeRabbit enforces per-developer PR review limits for each organization. Most developers receive the normal plan review availability.

For paid Pro and Pro+ PR reviews, CodeRabbit uses adaptive limits for sustained high-volume activity. When a developer's recent PR review activity reaches the 95th percentile or higher among CodeRabbit users, additional reviews become available more gradually as earlier reviews age out of the rolling window.

Please refer docs for additional details.

Review details

⚙️ Run configuration

Configuration used: Path: .coderabbit.yaml

Review profile: CHILL

Plan: Pro Plus

Run ID: f8232833-c36f-4ba3-8677-3edcbeca11b6

📥 Commits

Reviewing files that changed from the base of the PR and between 914127b and 8a38917.

📒 Files selected for processing (2)

• python/fast_mlsirm/llm_judge.py
• tests/test_llm_judge.py

📝 Walkthrough

Walkthrough

The criterion parser now raises ValueError for invalid criterion values. Tests verify the exception type and stable message.

Changes

Criterion validation

Layer / File(s)	Summary
Invalid criterion validation and coverage python/fast_mlsirm/llm_judge.py, tests/test_llm_judge.py	_criteria raises ValueError for unsupported criterion values. Tests verify the exception and message, including direct module execution.

Estimated code review effort: 1 (Trivial) | ~5 minutes

Possibly related PRs

• ContextualWisdomLab/fast-mlsirm#733: Refines _criteria validation and adds related regression coverage.

✨ Finishing Touches 💡 1

🛠️ Fix failing CI checks 💡

• Create stacked PR
• Commit on current branch

📝 Generate docstrings

• Create stacked PR
• Commit on current branch

🧪 Generate unit tests (beta)

• Create PR with unit tests
• Commit unit tests in branch fix/judge-invalid-criteria-error-contract

---

Thanks for using CodeRabbit! It's free for OSS, and your support helps us grow. If you like it, consider giving us a shout-out.

❤️ Share

• X
• Mastodon
• Reddit
• LinkedIn

_{Comment @coderabbitai help to get the list of available commands.}

[seonghobae]

@coderabbitai review

Review exact current head 8a38917ceee545c55405d8c2033178c4b8efa6a8 only. The intentional RED proved the public judge raised TypeError for a non-contract criterion element; the source fix changes only that branch to the stable adjacent ValueError validation family. Current unchanged-head CI, Security Scan and SAST are green. Focus on error-surface consistency, accepted criterion behavior, arbitrary-input/fuzz safety, and unintended API regressions. Do not mutate the branch or infer approval from predecessor evidence.

[coderabbitai]

@seonghobae I will review pull request #739 at exact head 8a38917ceee545c55405d8c2033178c4b8efa6a8. I will check the requested error surface and regression scope only.

⚠️ Action not completed

Review rate limited.

Note: CodeRabbit is an incremental review system and does not re-review already reviewed commits. This command is applicable only when automatic reviews are paused.