fix(frontend): use --model-path metadata instead of worker MDC paths

[Pernekhan]

Overview:

We deploy the frontend and workers on separate hosts. Workers hold the engine weights at one path (say /data/<model> on the GPU node), and the frontend only needs the small metadata files (config, tokenizer, chat template) at some path on the CPU node. Those two paths often don't match.

Today the frontend just trusts the worker's MDC paths. If they don't resolve locally, it tries to grab the files from HuggingFace using the worker's local path as a repo id — which obviously can't work. There's no way to tell the frontend "use my own files instead."

This PR adds that. Pass --model-path to the frontend and its files take precedence over what the worker advertises. The worker still owns everything else (kv cache size, runtime config, etc.).

Details:

In do_worker_set_registration, before download_config, we replace the worker MDC's metadata-file slots (model_info, tokenizer, prompt_formatter, chat_template_file, gen_config) with whatever the frontend's --model-path loaded. Without --model-path, the watcher gets None and behaves exactly like main.

Internal API:

• ModelDeploymentCard::override_files_with(&mut self, other: &Self)
• ModelWatcher::set_local_card(Option<ModelDeploymentCard>)

3 unit tests in lib/llm/tests/model_card.rs. Existing tests still pass. 5 files changed, +80 / −1.

There's a bigger plan in DEP #8749 (workers self-host metadata over HTTP). This PR is a smaller, complementary change for deployments that can already mount the metadata files into the frontend pod.

Where should the reviewer start?

• lib/llm/src/model_card.rs — override_files_with, five lines.
• lib/llm/src/discovery/watcher.rs — the call site in do_worker_set_registration, just before download_config.
• lib/llm/src/entrypoint/input/http.rs and …/grpc.rs — same one-liner deciding whether to forward the frontend's card.

Related Issues: (use one of the action keywords Closes / Fixes / Resolves / Relates to)

• Relates to Frontend model discovery fails on local filesystem paths and empty HuggingFace files #7389 (Bug 1) — same root cause as ours; we fix it via a different mechanism. Bugs 2 and 3 from that report are out of scope.

Summary by CodeRabbit

• New Features

  • Added ability to override model configuration during discovery across gRPC and HTTP endpoints.

• Tests

  • Added test coverage for model configuration override behavior.

[copy-pr-bot]

This pull request requires additional validation before any workflows can run on NVIDIA's runners.

Pull request vetters can view their responsibilities here.

Contributors can view more details about this message here.

[github-actions]

👋 Hi Pernekhan! Thank you for contributing to ai-dynamo/dynamo.

Just a reminder: The NVIDIA Test Github Validation CI runs an essential subset of the testing framework to quickly catch errors.Your PR reviewers may elect to test the changes comprehensively before approving your changes.

🚀

[coderabbitai]

Walkthrough

The PR adds support for per-watcher local card overrides in the model discovery system. ModelWatcher gains an optional local_card field that callers can install via a public setter. During worker set registration, discovered cards are merged with the local override using a new override_files_with method before config download. Both gRPC and HTTP entry points compute and inject local cards during setup.

Changes

Model Card Override & Watcher Integration

Layer / File(s)	Summary
Data Shape lib/llm/src/discovery/watcher.rs	ModelWatcher adds private local_card: Option<ModelDeploymentCard> field initialized to None.
Card Override Logic lib/llm/src/model_card.rs	New ModelDeploymentCard::override_files_with(&mut self, other: &ModelDeploymentCard) method clones model_info, tokenizer, prompt_formatter, chat_template_file, and gen_config from the source card.
Watcher Setter & Registration lib/llm/src/discovery/watcher.rs	Public set_local_card setter added; do_worker_set_registration applies the override via card.override_files_with(local) before config download when local_card is present.
Entry Point Wiring lib/llm/src/entrypoint/input/grpc.rs, lib/llm/src/entrypoint/input/http.rs	Both gRPC and HTTP setup paths compute local_card from the model, extend run_watcher signature to accept it, and call set_local_card on the watcher before starting discovery.
Tests & Validation lib/llm/tests/model_card.rs	Three new test cases verify field replacement, preservation of runtime fields during override, and clearing of slots when source provides no required files.

Estimated code review effort

🎯 2 (Simple) | ⏱️ ~12 minutes

🚥 Pre-merge checks | ✅ 4 | ❌ 1

❌ Failed checks (1 warning)

Check name	Status	Explanation	Resolution
Title check	⚠️ Warning	The PR title mentions 'frontend' and '--model-path metadata' but the changes primarily involve backend watcher/discovery logic, not UI frontend code.	Revise the title to reflect the actual changes, e.g., 'fix(watcher): allow frontend --model-path to override worker metadata' or 'feat(discovery): support local card override in model watcher'.

✅ Passed checks (4 passed)

Check name	Status	Explanation
Description check	✅ Passed	The description covers all required template sections with clear explanations of the problem, solution, affected files, and related issues.
Docstring Coverage	✅ Passed	Docstring coverage is 100.00% which is sufficient. The required threshold is 80.00%.
Linked Issues check	✅ Passed	Check skipped because no linked issues were found for this pull request.
Out of Scope Changes check	✅ Passed	Check skipped because no linked issues were found for this pull request.

_{✏️ Tip: You can configure your own custom pre-merge checks in the settings.}

Tip

💬 Introducing Slack Agent: The best way for teams to turn conversations into code.

Slack Agent is built on CodeRabbit's deep understanding of your code, so your team can collaborate across the entire SDLC without losing context.

• Generate code and open pull requests
• Plan features and break down work
• Investigate incidents and troubleshoot customer tickets together
• Automate recurring tasks and respond to alerts with triggers
• Summarize progress and report instantly

Built for teams:

• Shared memory across your entire org—no repeating context
• Per-thread sandboxes to safely plan and execute work
• Governance built-in—scoped access, auditability, and budget controls

One agent for your entire SDLC. Right inside Slack.

👉 Get started

---

Thanks for using CodeRabbit! It's free for OSS, and your support helps us grow. If you like it, consider giving us a shout-out.

❤️ Share

• X
• Mastodon
• Reddit
• LinkedIn

_{Comment @coderabbitai help to get the list of available commands and usage tips.}

[coderabbitai]

Actionable comments posted: 1

🤖 Prompt for all review comments with AI agents

Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.

Inline comments:
In `@lib/llm/src/model_card.rs`:
- Around line 601-609: override_files_with currently updates
model_info/tokenizer/prompt_formatter/chat_template_file/gen_config but does not
clear the cached checksum used by mdcsum(), causing stale checksum reads; after
cloning those fields in override_files_with, reset the OnceLock that caches the
mdcsum (the cached checksum field used by mdcsum, e.g. mdc_sum or similar) so
subsequent calls to mdcsum() recompute the checksum; ensure you reference the
cached checksum field by its actual name and clear or reinitialize it inside
override_files_with immediately after the clones.

🪄 Autofix (Beta)

Fix all unresolved CodeRabbit comments on this PR:

• Push a commit to this branch (recommended)
• Create a new PR with the fixes

---

ℹ️ Review info

⚙️ Run configuration

Configuration used: Path: .coderabbit.yaml

Review profile: CHILL

Plan: Pro

Run ID: b189791e-cd25-4ae5-ba37-71cbae2edfe3

📥 Commits

Reviewing files that changed from the base of the PR and between e06a3d3 and 88c3234.

📒 Files selected for processing (5)

• lib/llm/src/discovery/watcher.rs
• lib/llm/src/entrypoint/input/grpc.rs
• lib/llm/src/entrypoint/input/http.rs
• lib/llm/src/model_card.rs
• lib/llm/tests/model_card.rs

[Pernekhan]

Both review comments addressed in 67cba1c:

1. @dynamo-ops — moved override_files_with to the very top of handle_put, before any mdcsum/save_model_card call. Both branches (existing-worker-set, concurrent-recovery, first-time registration) now see the overridden card consistently. Workers joining an existing WorkerSet compare and store with the overridden checksum.

2. @coderabbitai — override_files_with now resets self.checksum = OnceLock::new(). Five of the seven mdcsum inputs are mutated by the override, so the cache must be invalidated. New unit test test_override_files_with_invalidates_mdcsum_cache pins this.

Local checks still green: cargo build/clippy/fmt clean, cargo test --test model_card 11/11, cargo test --test http-service 4/4 (regression).

[coderabbitai]

Tip

For best results, initiate chat on the files or code changes.

@Pernekhan Thank you for the thorough follow-up! The fix looks well-reasoned — resetting self.checksum = OnceLock::new() in override_files_with is the correct approach given that five of the seven mdcsum inputs are mutated, and backing it with test_override_files_with_invalidates_mdcsum_cache ensures the invariant is locked in for future refactors. Great work!

---

If you found this review helpful, would you consider giving us a shout-out on X?

Thank you for using CodeRabbit!

[Pernekhan]

Pushed ec710a7: gated the override on display_name match.

Caught a multi-model edge case: if the frontend was launched with --model-path for one model and a worker registers a different model, the previous version of this PR would have clobbered the new model's tokenizer/config with the frontend's. With the gate local.display_name == card.display_name, the override is now scoped to the model the frontend was launched for — workers for any other model flow through unchanged like upstream main.

Cases:

• Frontend --model-name X --model-path …, worker registers X → override applies.
• Frontend --model-name X --model-path …, worker registers Y → skipped, Y handled like today.
• Frontend with no --model-path → local_card is None, no override.

Local checks still green: cargo build/clippy/fmt clean, 11/11 model_card tests, 4/4 http-service regression.

[nnshah1]

Heads-up — PR #9057 (frontend MDC verify-and-cache pipeline) is landing the same --model-path plumbing as part of a broader resolve+blake3+content-addressed-cache flow. After both merge I'll consolidate so we end up with a single integration point. Behavior preserved for your deployment, no action needed here.

What the consolidation looks like: local_card and override_files_with collapse into a single local_model_path: Option<PathBuf> on ModelWatcher. The override is applied at URI synthesis time (in checked_file_uri) instead of by swapping CheckedFile slots wholesale, which means worker-published checksums stay authoritative and any drift surfaces via blake3 verification rather than getting silently masked.

Final fallback chain in checked_file_uri (per CheckedFile slot, applied after coercing path-only slots to a synthetic file:// URL so the logic is uniform):

1. cf.url() is http://, https://, or hf:// → use as-is. Worker self-host and HF Hub always win; --model-path never overrides them.
2. cf.url() is file:// and the file exists on the frontend host → use as-is. Shared-mount deployments keep working.
3. cf.url() is file://, the file is missing, AND --model-path/<basename> exists → rewrite to file://<--model-path>/<basename>. This is the rung that fires for your deployment: the worker's path string isn't reachable on the frontend, but the operator's --model-path directory has the same files.
4. Otherwise → fall back to hf://<source_path>/<basename> (existing legacy behavior).

What this means for your deployment: rung 3 fires, the operator's --model-path provides the bytes, those bytes get blake3-verified against the worker-published checksum, and the verified copy lands in ~/.cache/dynamo/mdc/. The verification step is the one functional difference from this PR's override_files_with — but in your topology you're running the same files on both sides, so checksums match and the verification is a no-op. If they ever didn't match, that'd be real drift the operator should know about.

Multi-model robustness: because rung 3 only fires when --model-path/<basename> actually exists, a frontend that discovers other models via DRT discovery (whose files aren't in --model-path) falls through to rung 4 (hf://) for those models instead of erroring. With override_files_with's wholesale swap, every discovered model would get the local card's checksums regardless.

Plumbing diff at consolidation time: drop local_card field + set_local_card setter on ModelWatcher, drop ModelDeploymentCard::override_files_with, drop the if local.display_name == card.display_name hook in do_worker_set_registration. Replace with local_model_path: Option<PathBuf> field + set_local_model_path setter (same shape, simpler payload), threaded through download_config(local_model_path). The entrypoint/input/{http,grpc}.rs plumbing keeps the same source — (!model.path().as_os_str().is_empty()).then(|| model.path().to_path_buf()) — just hands the watcher a PathBuf instead of a whole ModelDeploymentCard.

Happy to loop you in on the consolidation when it goes up — if any of the rungs above don't match what your deployment needs, we can adjust. PR #9057: #9057

[nnshah1]

/ok to test

[copy-pr-bot]

/ok to test

@nnshah1, there was an error processing your request: E1

See the following link for more information: https://docs.gha-runners.nvidia.com/cpr/e/1/

[nnshah1]

LGTM — happy to land this now. Consolidation onto the URI-synthesis-time override in #9057 will be a mechanical follow-up; details in the explainer comment above.

[nnshah1]

/ok to test 50881fe

[nnshah1]

/ok to test 50881fe

[nnshah1]

Heads-up — PR #9057 just merged (squash c4f214d77e) and includes the --model-path plumbing as part of a broader frontend MDC verify-and-cache pipeline. Your fix is now subsumed by that change: when the frontend passes --model-path, the resolve pipeline falls back to it for any worker-side file:// path that isn't reachable on the frontend host, then blake3-verifies the local copy against the worker's MDC before serving it. The 4-rung chain handles HTTP/file/hf URIs uniformly, so the DeepInfra topology works regardless of whether self-hosting metadata is enabled on the worker.

Marking this as draft for now since the capability is upstream — feel free to close it, or repurpose it for any follow-up gaps (e.g. multi-model display-name routing) you'd like to revisit. Thanks again for the contribution and for pushing on this case — the discussion here directly shaped the 4-rung design.

[nnshah1]

And to be clear — really appreciate you flagging the DeepInfra deployment topology and contributing the fix. That use case (frontend host has the model files at a different path than the worker advertises) wasn't on our radar, and your PR is what surfaced it. The merged version generalized the approach, but the feature is yours — thanks for pushing on this. 🙏

[Pernekhan]

#9057 addresses what we wanted. Thanks for quickly shipping it.