Extract evaluator LLM provider references into FK fields

[snopoke]

Closes #3977. Part 2 is #3995. Supersedes #3992, which carried this plus four follow-ons.

#3998 has merged; this now targets main.

Product Description

An evaluator whose LLM provider or model has been deleted now says so — "has no LLM provider configured, edit the evaluator and select a provider and model" — instead of failing mid-run with a bare lookup error.

Technical Description

Evaluator.params held llm_provider_id/llm_provider_model_id as plain integers, so deleting a provider left a dangling id (see #3974, and one such row in prod today). These become real FKs with SET_NULL.

params keeps its copy of both ids: the evaluator form is generated from the pydantic schema in evaluators.py, so the ids have to stay in the schema for the UI to edit them. The FK is the authoritative reference — Evaluator.save derives the columns from params on every write (dangling ids resolve to null rather than tripping the constraint) and get_evaluator_params() reads them back off the FKs at run time. Deriving in save rather than in the form is what keeps cloning, bootstrap data, and factories consistent without each remembering. Same shape as Node._sync_resource_fk_fields.

get_related_evaluators_queryset is gone — the usages page reaches evaluators through the reverse FK now.

Two consequences worth calling out, because they have no corresponding line of code:

• Deleting an LlmProviderModel an evaluator uses is now blocked, the moment the reverse FK exists — get_candidate_relations_to_delete picks it up and the existing guard does the rest. The flows that legitimately need the model gone (_replace_custom_model_with_global, remove_deprecated_models) repoint evaluators first.
• The delete button disappears for such a model, via has_related_objects.

_replace_custom_model_with_global also runs from migrations, where Evaluator may be absent from the app state entirely. It raises unless the FK column is genuinely not in the database yet, so a future migration missing the evaluations.0018 dependency fails with instructions rather than a deferred FK violation at commit.

Migrations

• The migrations are backwards compatible

Old code ignores the new columns, so the schema change and backfill are safe to apply ahead of cutover. But an old web worker that creates or edits an evaluator after the backfill writes params only, leaving both FKs null — and after cutover those evaluators hit the "no LLM provider configured" error despite valid params, recoverable only by re-saving each one. Nothing re-syncs them automatically. Small window, small blast radius, but worth deciding deliberately: re-run the backfill after cutover, or hold this and make params a read-through fallback for one release.

The backfill deliberately leaves dangling ids null. Prod has exactly one such evaluator (KMC - Coverage Eval, team Vaccine_Coach); it was already broken at run time and still needs a human to re-pick the model — the original id can't be recovered.

Demo

Docs and Changelog

• This PR requires docs/changelog update

[coderabbitai]

Warning

Review limit reached

@snopoke, you've reached your PR review limit, so we couldn't start this review.

Next review available in: 19 minutes

Enable usage-based reviews in Billing to review now. Otherwise, wait until the next included review is available.
You're only billed for reviews past your plan's rate limits ($0.25/file).

How can I continue?

After more reviews become available, a review can be triggered using the @coderabbitai review command as a PR comment. Alternatively, push new commits to this PR.

To avoid repeated limits, reduce automatic review volume by pausing incremental auto-reviews earlier, using label-based review opt-in, excluding WIP or generated PR titles, or requesting reviews manually when the PR is ready. If your team needs uninterrupted high-volume reviews, an organization admin can enable usage-based reviews.

How do review limits work?

CodeRabbit enforces per-developer PR review limits for each organization. Most developers receive the normal plan review availability.

For paid Pro and Pro+ PR reviews, CodeRabbit uses adaptive limits for sustained high-volume activity. When a developer's recent PR review activity reaches the 95th percentile or higher among CodeRabbit users, additional reviews become available more gradually as earlier reviews age out of the rolling window.

Please refer docs for additional details.

Review details

⚙️ Run configuration

Configuration used: Path: .coderabbit.yaml

Review profile: CHILL

Plan: Pro Plus

Run ID: 5b3576c0-a7ac-4f0f-8ff7-57482f556945

📥 Commits

Reviewing files that changed from the base of the PR and between 9792615 and a0c0620.

📒 Files selected for processing (18)

• api-schemas/export.yml
• apps/conftest.py
• apps/data_migrations/management/commands/remove_deprecated_models.py
• apps/data_migrations/tests/test_remove_deprecated_models.py
• apps/evaluations/evaluators.py
• apps/evaluations/migrations/0018_evaluator_llm_provider_fks.py
• apps/evaluations/migrations/0019_backfill_evaluator_llm_provider_fks.py
• apps/evaluations/models.py
• apps/evaluations/tests/test_evaluator_llm_provider_fks.py
• apps/evaluations/tests/test_evaluator_views.py
• apps/evaluations/tests/test_migration_0019_backfill_evaluator_llm_provider_fks.py
• apps/service_providers/llm_service/default_models.py
• apps/service_providers/tests/test_default_models.py
• apps/service_providers/tests/test_models.py
• apps/service_providers/tests/test_usages.py
• apps/service_providers/usages.py
• apps/teams/management/commands/clone_team.py
• apps/utils/deletion.py

---

Thanks for using CodeRabbit! It's free for OSS, and your support helps us grow. If you like it, consider giving us a shout-out.

❤️ Share

• X
• Mastodon
• Reddit
• LinkedIn

_{Comment @coderabbitai help to get the list of available commands.}

[SmittieC]

LGTM. A few clarifying questions only