fix(runtime): prefer the routed source address in automatic host selection

[nvpohanh]

[by Claude Code]

Summary

Since #13126, automatic host selection (for DYN_TCP_RPC_HOST, DYN_TCP_RESPONSE_STREAM_HOST and DYN_EVENT_PLANE_HOST when unset, and for wildcard hosts) takes the first usable address of any up interface in operating-system enumeration order. Before #13126 it used local_ip()/local_ipv6(), which on Linux return the preferred source address of the kernel's route to an external address.

On hosts where a node-local interface enumerates before the cluster interface, the new behavior binds and advertises an address that other nodes cannot reach. We hit this on GB300 nodes whose BMC Redfish host interface (bmc_redfish0, which carries the same node-local 10.0.1.2 on every node) is listed first. Every prefill and decode worker advertised 10.0.1.2, so the frontend failed with TCP connection to 10.0.1.2:<port> failed / no endpoints available to route work for every worker on another node. The same nodes work with an older Dynamo build (1.5.0.dev20260914, before #13126), which binds the cluster address.

This change restores the routing-table preference while keeping #13126's guarantees:

• local_candidates() first asks local_ip() and local_ipv6() for the routed source address in each family and seeds the candidates with it only if an up interface in the same snapshot carries that address. It then considers the snapshot in enumeration order as before.
• Failed probes are ignored, so IPv6-only hosts, hosts without a route, and stub resolvers that return errors behave exactly as before.
• Explicit hosts, IP literals and named interfaces are unchanged. Wildcard advertisement and resolve_advertise_ip_for_bind() also benefit, because they share local_candidates().
• The runtime-configuration reference now describes the preference.

local_ip() on Linux (local-ip-address 0.6.13) sends one RTM_GETROUTE netlink request for 192.0.2.0 (IPv6: 2001:db8::) and reads RTA_PREFSRC; on other Unix platforms it returns the first non-loopback IPv4 interface address, which matches the existing enumeration order.

Validation

• cargo test --locked -p dynamo-runtime --lib -- utils::ip_resolver pipeline::network::tcp::server transports::event_plane system_status_server: 128 passed, including two new resolver tests:
  • routed_address_is_preferred_over_earlier_interfaces: a BMC-style interface enumerated first; checks automatic selection, the IPv4 wildcard, resolve_advertise_ip_for_bind() for 0.0.0.0 and ::, and that explicit interface lookup still returns the BMC address.
  • routed_address_outside_up_snapshot_is_ignored: routed addresses on a down interface or absent from the snapshot are ignored.
  • ipv6_only_inventory_uses_global_ipv6_without_family_probes became ..._when_family_probes_fail: failed probes do not change the IPv6-only result. The exact probe call-count assertions and the unused StubResolver probe counters were removed in review.
• cargo clippy --locked --no-deps -p dynamo-runtime --all-targets -- -D warnings passed.
• cargo fmt -p dynamo-runtime -- --check and git diff --check passed.
• python3 docs/fern/scripts/docs_lint.py: 0 errors.

🤖 Generated with Claude Code

[copy-pr-bot]

This pull request requires additional validation before any workflows can run on NVIDIA's runners.

Pull request vetters can view their responsibilities here.

Contributors can view more details about this message here.

[github-actions]

🌿 Fern Docs Preview: https://nvidia-preview-01a10ee1-fd4e-768e-b27d-3643096f3191.docs.buildwithfern.com/dynamo/dev

[coderabbitai]

Navigate logical layers of code changes, visualize relationships, and explore their blast radius.

No actionable comments were generated in the recent review. 🎉

ℹ️ Recent review info

⚙️ Run configuration

• Configuration used: Repository: ai-dynamo/dynamo/.coderabbit.yaml
• Review profile: CHILL
• Plan: Enterprise
• Run ID: a85402a4-5e48-4e27-8a5a-ae976b7f46e2

📥 Commits

Reviewing files that changed from the base of the PR and between 519e735 and 353d94f.

📒 Files selected for processing (2)

• docs/fern/pages/reference/components/runtime-configuration.mdx
• lib/runtime/src/utils/ip_resolver.rs

Included review availability: This review used your included allowance. Your plan provides up to 12 included reviews per hour; 11 remain after this review.

---

Walkthrough

Automatic local address selection now prefers routed IPv4 and IPv6 addresses when they appear on an up interface. The runtime configuration reference documents this preference for DYN_TCP_RPC_HOST and DYN_EVENT_PLANE_HOST.

Changes

Routed address selection

Layer / File(s)	Summary
Select and document routed addresses lib/runtime/src/utils/ip_resolver.rs, docs/fern/pages/reference/components/runtime-configuration.mdx	The resolver probes routed addresses and prioritizes results present in the up-interface snapshot. Interface enumeration remains the fallback. Tests cover successful probes, failed probes, wildcard advertisement, named-interface selection, and routed addresses absent from the snapshot. The reference describes the automatic selection preference.

Priority: ➖ Normal

Estimated code review effort: 3 (Moderate) | ~20 minutes

Merge Risk: ⚪ Minimal · up to 353d9

The change appears mergeable after normal checks; no actionable issue was identified in the supplied changes.

🚥 Pre-merge checks | ✅ 3 | ❌ 2

❌ Failed checks (2 warnings)

Check name	Status	Explanation	Resolution
Docstring Coverage	⚠️ Warning	Docstring coverage is 50.00% which is insufficient. The required threshold is 80.00%. Docstring coverage is scoped to functions touched by this diff. Analyzed 8 functions across 1 files. (1 skipped: 1…	Write docstrings for the functions missing them to satisfy the coverage threshold.
Description check	⚠️ Warning	The description explains the problem, proposed changes, affected behavior, and validation results. It omits the required Related Issues section and does not state whether this PR links to an issue.	Add the required Related Issues section. Either list the related issue using the required format or confirm that no related issue exists.

✅ Passed checks (3 passed)

Check name	Status	Explanation
Linked Issues check	✅ Passed	Check skipped because no linked issues were found for this pull request.
Out of Scope Changes check	✅ Passed	Check skipped because no linked issues were found for this pull request.
Title check	✅ Passed	The title clearly and concisely describes the main change: preferring the routed source address during automatic host selection.

Full details: Docstring Coverage

Explanation

Docstring coverage is 50.00% which is insufficient. The required threshold is 80.00%. Docstring coverage is scoped to functions touched by this diff. Analyzed 8 functions across 1 files. (1 skipped: 1 unsupported.)

• Fix all pre-merge checks with AI

• Autopilot · Keep fixing CodeRabbit findings and required CI, and resolving merge conflicts

---

_{Comment @coderabbitai help to get the list of available commands.}

[jthomson04]

/ok to test bb745db

[devin-ai-integration]

⏹️ Dynamo PR CI cancelled — run 37400200087 (attempt 1) on 9a5dafed46

Gate checks: ✅ backend-status-check · ✅ deploy-status-check · ❌ dynamo-status-check

Other	Jobs
dynamo-runtime	⏹️ 1

Failure details

No jobs failed. 1 job was cancelled: dynamo-runtime / image / Build multi-arch cuda13.0 ran past the 1h job time limit during Build and Push Test Image (GitHub annotation: "The job has exceeded the maximum execution time of 1h0m0s"). Nothing failed before it, so this isn't a fail-fast cancel, and no one cancelled it by hand. dynamo-status-check is red only because of this job. Rerunning failed jobs is likely enough.

For agents

{"pr": 15628, "run_id": 37400200087, "run_attempt": 1, "head_sha": "9a5dafed46f70a1b6b7ed1a4301944e397113b8f", "failures": [], "cancelled": [{"job": "dynamo-runtime / image / Build multi-arch cuda13.0", "job_id": 112065537018, "step": "Build and Push Test Image", "reason": "The job has exceeded the maximum execution time of 1h0m0s", "log_cmd": "gh run view --job 112065537018 -R ai-dynamo/dynamo --log"}]}

_{Posted automatically by Devin for run 37400200087. Updated on every full-CI run of this PR.}

[biswapanda]

lgtm

[biswapanda]

/ok to test 9a5dafe

[dynamo-review-agent]

Previously reported defects still present:

• Original discussion: The previously raised redundant parameter case is still present: routed_address_outside_up_snapshot_is_ignored still iterates over both "198.51.100.1" and "192.0.2.99". After stale0 is filtered from the up snapshot, the 198.51.100.1 case observes the same absent-routed-address behavior as the retained 192.0.2.99 case, while down-interface filtering remains protected by automatic_selection_excludes_down_interfaces_but_explicit_lookup_keeps_them.
• Original discussion: The previously reported redundant "198.51.100.1" parameter case is still present in routed_address_outside_up_snapshot_is_ignored; the retained "192.0.2.99" case exercises the same absent-from-up-snapshot path while the existing down-interface test covers filtering.