Validate the evaluator's LLM provider selection - #3998
Conversation
The ids arrive inside the params JSON blob, so they never passed through a ModelChoiceField queryset: neither team ownership nor provider/model type compatibility was checked. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
|
Warning Review limit reached
Next review available in: 59 minutes Enable usage-based reviews in Billing to review now. Otherwise, wait until the next included review is available. How can I continue?After more reviews become available, a review can be triggered using the To avoid repeated limits, reduce automatic review volume by pausing incremental auto-reviews earlier, using label-based review opt-in, excluding WIP or generated PR titles, or requesting reviews manually when the PR is ready. If your team needs uninterrupted high-volume reviews, an organization admin can enable usage-based reviews. How do review limits work?CodeRabbit enforces per-developer PR review limits for each organization. Most developers receive the normal plan review availability. For paid Pro and Pro+ PR reviews, CodeRabbit uses adaptive limits for sustained high-volume activity. When a developer's recent PR review activity reaches the 95th percentile or higher among CodeRabbit users, additional reviews become available more gradually as earlier reviews age out of the rolling window. Please refer docs for additional details. Review details⚙️ Run configurationConfiguration used: Path: .coderabbit.yaml Review profile: CHILL Plan: Pro Plus Run ID: 📒 Files selected for processing (3)
Thanks for using CodeRabbit! It's free for OSS, and your support helps us grow. If you like it, consider giving us a shout-out. Comment |
![codescene-delta-analysis[bot]](https://avatars.githubusercontent.com/in/53921?s=60&v=4){kind=small}
There was a problem hiding this comment.
Gates Failed
(1 file with Complex Conditional)
Our agent can fix these. Install it.
Gates Passed
Reason for failure
| Enforce advisory code health rules | Violations | Code Health Impact | |
|---|---|---|---|
| forms.py | 1 advisory rule | 4.69 → 4.56 | Suppress |
Quality Gate Profile: Clean Code Collective
Install CodeScene MCP: safeguard and uplift AI-generated code. Catch issues early with our IDE extension and CLI tool.
| # The picker only offers models matching the chosen provider's type (see | ||
| # ``_evaluator_parameter_values``), so a mismatch means the ids were not submitted | ||
| # through the UI. Left unchecked it fails at run time inside get_llm_service. | ||
| if provider is not None and provider_model is not None and provider.type != provider_model.type: |
There was a problem hiding this comment.
❌ New issue: Complex Conditional
EvaluatorForm._validate_llm_provider_selection has 1 complex conditionals with 2 branches, threshold = 2
1 participant
Product Description
Picking an LLM provider or model the team doesn't have access to, or a model that doesn't belong to the selected provider's type, is now rejected on save with a message naming the problem — instead of saving and failing later when the evaluator runs.
Technical Description
Evaluator.paramsis a JSON blob, sollm_provider_idandllm_provider_model_idnever passed through aModelChoiceFieldqueryset. Nothing checked team ownership, and nothing checked that the model's provider type matched the provider. Both checks now run inEvaluatorForm.clean.The team check is the one that matters: an id from another team would have run this team's evaluations against another team's credentials.
Also adds
configure_evaluator_llm_providernext toEvaluatorFactory, which leaves the provider ids out ofparamsand so builds an evaluator this validation rejects.Migrations
None.
Demo
The picker only ever offers models matching the chosen provider's type, so the type error is reachable only by a hand-crafted post.
Docs and Changelog