feat(self-host): flip DYN_SELF_HOST_METADATA default to ON (gh-8749)

[nnshah1]

Summary

Capstone of the gh-8749 stack: workers now advertise their MDC files over the system_status_server by default instead of requiring shared storage.

What changes

lib/llm/src/local_model.rs (single file, +12/-37 after squash):

• self_host_metadata_default(None) returns true
• Explicit falsy env values 0/false/no/off (case-insensitive, trimmed) opt out
• Anything else stays on
• Hard-fail (anyhow::bail!) at first registration when self_host_metadata=true but DYN_SYSTEM_PORT is unset — surfaces misconfig at startup instead of WARN spam plus silently degraded hf:// MDC advertisement

Behavior enumeration

Without self-host (DYN_SELF_HOST_METADATA=0)

1. Worker downloads model from HF → local snapshot dir (whatever revision HF's main resolves to at that moment).
2. Worker builds MDC with hf://<repo>/<filename> URLs and mdcsum = fold of (basename, blake3) over its resolved files.
3. Worker publishes MDC via discovery.
4. Frontend reads MDC from discovery.
5. Frontend downloads independently from HF — may resolve a different revision if HF's main moved between the worker's download and the frontend's.
6. Frontend blake3-verifies the downloaded files against MDC.
7. Match → serve. Mismatch → hard error at startup, operator must intervene.

With self-host (default, DYN_SYSTEM_PORT=9090 set — operator-managed deployments)

1. Worker downloads model from HF → local snapshot.
2. Worker builds MDC and rewrites URLs to http://<worker>:9090/v1/metadata/<ns>/<component>/<endpoint>/<slug>/<suffix>/<filename> pointing at its own copy.
3. Worker publishes MDC via discovery.
4. Frontend reads MDC.
5. Frontend fetches from the worker's HTTP endpoint — exact bytes the worker used to build the MDC.
6. Frontend blake3-verifies → always matches.
7. Frontend caches locally under ~/.cache/dynamo/mdc/by-slug/<slug>/<mdcsum>/; subsequent workers with the same mdcsum are cache hits.

With self-host on but DYN_SYSTEM_PORT unset (bare / non-operator upgrade)

Worker anyhow::bail!s at first model registration with an actionable message: set DYN_SYSTEM_PORT or opt out with DYN_SELF_HOST_METADATA=0.

HF revision-mismatch and a possible follow-up

The self-host path solves the HF-mismatch problem by construction — worker-served bytes are the source of truth. The non-self-host path detects mismatches (blake3 fails and the frontend errors) but doesn't converge on the same revision.

The hf:// URL format the worker rewrites into already supports a revision segment: parse_hf_uri in lib/llm/src/model_card.rs:640 handles hf://repo[@rev]/filename. The construction site (local_model.rs:619) currently builds hf://<source_path>/ without a revision, but the resolved snapshot's SHA is knowable (HF cache stores files under snapshots/<sha>/).

Embedding the resolved SHA in the hf:// URL — e.g. hf://Qwen/Qwen3-0.6B@abcdef.../config.json — would let the frontend pin its download to the same revision the worker used, eliminating the mismatch class entirely in the non-self-host path. Small change; out of scope for this PR but a natural follow-up if the opt-out path is expected to be used long-term.

Why now

All prerequisites are in main:

• ✅ feat(self-host): worker hosts metadata files via system_status_server (gh-8749) #8855 worker HTTP hosting on system_status_server
• ✅ feat(self-host): frontend MDC verify-and-cache pipeline (gh-8749) #9057 frontend MDC verify-and-cache pipeline
• ✅ fix(self-host): harvest non-weight siblings into slug_dir (gh-8749) #9610 harvest non-weight siblings into slug_dir
• ✅ feat(self-host): worker advertises non-typed metadata siblings (gh-8749) #9707 worker advertises non-typed metadata siblings via extra_files
• ✅ feat(self-host): native preprocessor consumes resolved slug_dir (gh-8749) #10037 native preprocessor consumes resolved local_dir
• ✅ fix(backends): use engine's runai-pulled local dir for register_model #10599 register_model uses engine's runai-pulled local dir on object-storage URIs
• ✅ feat(self-host): auto-clean metadata-registry on detach (gh-8749) #10351 auto-clean MetadataArtifactRegistry on detach (LoRA swap hygiene)

Opt-out path

Existing deployments that need to fall back to shared-storage MDC delivery can pin DYN_SELF_HOST_METADATA=0 in their pod spec. That gives operators a rollback lever without needing an image swap.

Test coverage — honest read

Existing e2e: tests/frontend/test_self_host_metadata.py::test_worker_serves_metadata_via_http — a single post_merge test that spins up a mocker with DYN_SELF_HOST_METADATA=true explicitly set, then curls the /v1/metadata/... route.

Gaps against this PR:

• The existing test explicitly sets the env, so it doesn't validate the flipped default (would pass either way).
• No test for the anyhow::bail! path when DYN_SYSTEM_PORT is unset.
• No test for the opt-out path (=0).
• The file's docstring flags follow-ups that never landed: frontend verify-and-cache assertions, multi-replica + LoRA cases.

Recommendation before final merge: drop the explicit DYN_SELF_HOST_METADATA=true from the existing test so it validates the default, and add one test for the bail! path. Both are small and belong in a follow-up commit on this PR.

Docs

Only code-facing documentation was touched (the doc comment on ENV_SELF_HOST_METADATA). No user-facing docs currently describe the two modes. If we want to broaden reach, options:

1. Add a section to docs/reference/glossary.md explaining "self-hosted vs shared-storage MDC delivery."
2. Add a release-note entry for the default flip.
3. Update the operator's helm chart README to note DYN_SYSTEM_PORT is now required (it already sets 9090 by default).

Test plan

• cargo test -p dynamo-llm --lib passes locally
• cargo fmt --check / cargo clippy -p dynamo-llm --lib --no-deps clean
• Follow-up commit: update existing e2e to validate default, add bail! test
• CI e2e green

🤖 Generated with Claude Code

[github-actions]

🌿 Fern Docs Preview: https://nvidia-preview-019f43fe-abe3-728a-992a-a46fbcd9ba0d.docs.buildwithfern.com/dynamo/dev

[devin-ai-integration]

Devin Review found 3 potential issues.

[coderabbitai]

Walkthrough

The default behavior of LocalModelBuilder::self_host_metadata is flipped: metadata rewriting is now enabled by default unless DYN_SELF_HOST_METADATA is explicitly set to a falsy value. Documentation is updated accordingly. move_to_self_host now fails with an error instead of warning when DYN_SYSTEM_PORT is unset. An outdated unit test module is removed.

Changes

Self-host metadata default behavior change

Layer / File(s)	Summary
Env default parsing and docs lib/llm/src/local_model.rs	self_host_metadata_default now returns true unless the environment variable is explicitly set to a falsy value (0/false/no/off), reversing the prior truthy-allowlist logic; doc comments for the builder and its method are updated to reflect the default-on behavior.
Fail-fast on missing port and test cleanup lib/llm/src/local_model.rs	move_to_self_host now uses anyhow::bail! to error out when DYN_SYSTEM_PORT is unset while rewriting is enabled, replacing the prior warn-and-continue behavior; the env_self_host_metadata_tests module testing the old parsing logic is removed.

Estimated code review effort: 2 (Simple) | ~10 minutes

🚥 Pre-merge checks | ✅ 4 | ❌ 1

❌ Failed checks (1 warning)

Check name	Status	Explanation	Resolution
Description check	⚠️ Warning	The description is detailed, but it misses the required Related Issues section and the reviewer-start guidance from the template.	Add the required Related Issues section with the issue link/confirmation, and include a short "Where should reviewer start?" note.

✅ Passed checks (4 passed)

Check name	Status	Explanation
Docstring Coverage	✅ Passed	No functions found in the changed files to evaluate docstring coverage. Skipping docstring coverage check.
Linked Issues check	✅ Passed	Check skipped because no linked issues were found for this pull request.
Out of Scope Changes check	✅ Passed	Check skipped because no linked issues were found for this pull request.
Title check	✅ Passed	The title clearly summarizes the main change: flipping the self-host metadata default on.

---

_{Comment @coderabbitai help to get the list of available commands.}

[coderabbitai]

Actionable comments posted: 1

🤖 Prompt for all review comments with AI agents

Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.

Inline comments:
In `@lib/llm/src/local_model.rs`:
- Around line 49-55: Add table-driven test coverage for
self_host_metadata_default to lock in the default-on behavior. Create cases for
None, whitespace-padded and mixed-case falsey inputs ("0", "false", "no",
"off"), truthy inputs, and arbitrary strings, and assert the function returns
the expected bool for each. Place the tests near self_host_metadata_default so
future changes to the normalization logic in this helper are caught.

🪄 Autofix (Beta)

Fix all unresolved CodeRabbit comments on this PR:

• Push a commit to this branch (recommended)
• Create a new PR with the fixes

---

ℹ️ Review info

⚙️ Run configuration

Configuration used: Path: .coderabbit.yaml

Review profile: CHILL

Plan: Enterprise

Run ID: 81084b5e-4b8a-4b19-89c8-f34c09d99bf5

📥 Commits

Reviewing files that changed from the base of the PR and between 689372a and fc16f80.

📒 Files selected for processing (1)

• lib/llm/src/local_model.rs