fix(frontend): reconcile model registrations

[MatejKosec]

Overview:

Add periodic, source-of-truth reconciliation so a frontend eventually rebuilds its local model registry after missed discovery events or transient registration failures.

The scope follows Neelay Shah’s later clarification, recorded on issue #10732: /health, /live, and request-level process readiness remain unchanged, while the existing serving-readiness filter continues to exclude incomplete models from GET /v1/models.

Details:

• Query DiscoveryQuery::AllModels every 30 seconds and process model-card identities deterministically.
• Retry a registration when the per-instance card, WorkerSet, or WorkerSet checksum does not match discovery.
• Run reconciliation in a single-flight background task so slow registration or stale cleanup does not pause discovery-event processing.
• Remove stale local cards while keeping deletion retryable after a transient discovery-query failure and continuing past an individual stale-entry error.
• Preserve the existing /v1/models serving-readiness filter and leave frontend health and process-readiness semantics unchanged.

Validation:

• cargo fmt --all -- --check
• cargo test -p dynamo-llm --no-default-features discovery::watcher::tests --lib — 12 passed
• cargo test -p dynamo-llm --no-default-features serving_ready_excludes_incomplete_namespace --lib — 1 passed
• cargo clippy -p dynamo-llm --no-default-features --lib -- -D warnings
• Repository pre-commit hooks passed

Where should the reviewer start?

1. ModelWatcher::watch in lib/llm/src/discovery/watcher.rs for the single-flight background scheduling.
2. ModelWatcher::reconcile and handle_delete in the same file for retry and stale-removal behavior.
3. is_registration_complete for the card, WorkerSet, and checksum convergence condition.
4. ModelManager::get_model_card_keys in lib/llm/src/discovery/model_manager.rs for the reconciliation snapshot accessor.

Related Issues

🔗 This PR is linked to an issue:

• Closes Frontend can serve partial model registry without registration reconciliation #10732

Summary by CodeRabbit

• New Features

  • Added automatic periodic reconciliation to keep model discovery state aligned over time.
  • Improved recovery of models that were only partially registered, reducing the need for manual intervention.

• Bug Fixes

  • Better handling of stale or missing model entries, helping remove outdated records more reliably.
  • Improved cleanup flow so temporary discovery issues are less likely to disrupt local model state.

[coderabbitai]

No actionable comments were generated in the recent review. 🎉

ℹ️ Recent review info

⚙️ Run configuration

Configuration used: Path: .coderabbit.yaml

Review profile: CHILL

Plan: Enterprise

Run ID: b3c13ea4-7fe7-4927-8a08-58b662744f4f

📥 Commits

Reviewing files that changed from the base of the PR and between 02d2e8b and 962d2a2.

📒 Files selected for processing (1)

• lib/llm/src/discovery/watcher.rs

🚧 Files skipped from review as they are similar to previous changes (1)

• lib/llm/src/discovery/watcher.rs

---

Walkthrough

This PR adds periodic reconciliation to the discovery watcher, retries incomplete model registrations, removes stale local cards, and adds a key accessor plus a completeness test.

Changes

Discovery reconciliation

Layer / File(s)	Summary
ModelManager key accessor lib/llm/src/discovery/model_manager.rs	Adds get_model_card_keys() returning cloned keys from the internal cards map for reconciliation cleanup.
Reconciliation helpers lib/llm/src/discovery/watcher.rs	Adds the reconciliation interval, supporting imports, model_card_instance_id, and is_registration_complete.
Periodic watch loop lib/llm/src/discovery/watcher.rs	Refactors watch to combine discovery event handling with periodic reconciliation using tokio::select! and JoinSet.
Reconcile and retry flow lib/llm/src/discovery/watcher.rs	Implements snapshot reconciliation, retry scheduling for incomplete registrations, and reordered stale-card deletion.
Completeness test lib/llm/src/discovery/watcher.rs	Adds a unit test for is_registration_complete across model-card, WorkerSet, and removal states.

Estimated code review effort: 4 (Complex) | ~45 minutes

🚥 Pre-merge checks | ✅ 5

✅ Passed checks (5 passed)

Check name	Status	Explanation
Title check	✅ Passed	The title is specific and matches the main change: reconciling model registrations in the frontend.
Description check	✅ Passed	The description follows the required template and includes Overview, Details, reviewer start points, and linked issue reference.
Linked Issues check	✅ Passed	The PR adds periodic reconciliation, deterministic retries, stale cleanup, and preserves the required health/readiness semantics for #10732.
Out of Scope Changes check	✅ Passed	The changes stay focused on model-registration reconciliation and cleanup, with no unrelated feature work shown in the summary.
Docstring Coverage	✅ Passed	Docstring coverage is 100.00% which is sufficient. The required threshold is 80.00%.

---

_{Comment @coderabbitai help to get the list of available commands.}

[coderabbitai]

Actionable comments posted: 2

🤖 Prompt for all review comments with AI agents

Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.

Inline comments:
In `@lib/llm/src/discovery/watcher.rs`:
- Around line 340-341: The reconciliation path in DiscoveryWatcher is blocking
the watch loop because the reconciliation tick awaits `reconcile` inline and
stale cleanup in `handle_delete` can also stall on in-flight work. Refactor the
`reconciliation.tick()` branch and the stale-entry cleanup path so discovery
event processing in the watch loop stays responsive, for example by moving
`reconcile` to a single-flight background task or making cleanup
incremental/non-blocking while preserving the existing `reconcile` and
`handle_delete` behavior.
- Around line 119-125: The registration-complete check in watcher.rs is only
verifying that a WorkerSet exists, so it can incorrectly treat a stale or
mismatched local registration as complete. Update the condition built around
get_model_card, get_model, and has_worker_set in the reconciliation logic to
also require WorkerSet checksum compatibility with the saved card mdcsum before
returning true. Use the existing ws_key path and the model/card lookup flow to
compare the stored WorkerSet metadata against the current card checksum, so
retry_model_registration still runs when the local state is out of sync.

🪄 Autofix (Beta)

Fix all unresolved CodeRabbit comments on this PR:

• Push a commit to this branch (recommended)
• Create a new PR with the fixes

---

ℹ️ Review info

⚙️ Run configuration

Configuration used: Path: .coderabbit.yaml

Review profile: CHILL

Plan: Enterprise

Run ID: 803a19d2-babc-4a87-bbad-5fcbc5fcd110

📥 Commits

Reviewing files that changed from the base of the PR and between af3c1ee and 02d2e8b.

📒 Files selected for processing (2)

• lib/llm/src/discovery/model_manager.rs
• lib/llm/src/discovery/watcher.rs

[devin-ai-integration]

Devin Review found 1 potential issue.

[MatejKosec]

@coderabbitai review

[coderabbitai]

✅ Action performed

Review finished.

Note: CodeRabbit is an incremental review system and does not re-review already reviewed commits. This command is applicable only when automatic reviews are paused.