fix(runtime): support IPv6-only IP resolution

[jthomson04]

Summary

• Fix local address selection on IPv6-only hosts. Prefer usable non-loopback IPv4, then IPv6, then IPv4 loopback, then IPv6 loopback. On Unix, automatic selection uses one interface snapshot and excludes down interfaces; explicit interface lookup remains unchanged.
• Accept IP literals, bracketed IPv6, wildcards, interface names, and aliases. Trim surrounding host whitespace for direct Rust callers. Canonicalize IPv4-mapped addresses before classification; mapped wildcards select a concrete advertised address through the IPv4 wildcard rules.
• Keep bind and advertised addresses separate with typed SocketAddr values. Wildcards can switch families to select a usable address, and loopback fallback prefers the requested family. Publish addresses and actual ports served by the TCP and system-status listeners.
• Preserve the public IpResolver and ServerOptions::interface APIs. The new list_up_afinet_netifas() method has a default for existing custom resolvers. Cache only successful non-loopback results from local_ip_for_advertise(); TCP servers resolve once at startup.
• Document interface-state filtering, mapped-address handling, host whitespace and empty-value rules, and TCP versus QUIC binding. Keep port parsing unchanged and update the affected workspace lockfiles.

This change follows #7627 and #7632. Fixes #7619.

Limitation

Direct ZMQ publishers still bind an IPv4 wildcard. Automatic advertisement stays in that family and falls back to IPv4 loopback when no eligible non-loopback IPv4 address exists, so remote subscribers cannot connect. An explicit override must identify a reachable IPv4 address served by the listener. An IPv6 override does not enable IPv6 listening.

Validation

Local validation for f1b5aea3c0088b92c9caf6448cc21dcab23581f9:

• 74 focused runtime tests passed: 15 resolver, 44 TCP-server, 10 direct-ZMQ, 2 environment-registry, 2 response-port, and 1 system-status test. Tests ran serially and include real IPv6 and mapped-wildcard binds, port connectivity, padded host values, down-interface filtering, and legacy resolver compatibility.
• cargo clippy --locked --no-deps -p dynamo-runtime --all-targets -- -D warnings passed.
• Root and Python binding formatting checks passed.
• Docs lint passed with 0 errors; 8 warnings are outside the edited page.
• fern check and fern docs broken-links passed after generating the ignored API reference pages.
• git diff --check passed.

CI results are separate from these local checks.

[devin-ai-integration]

Devin Review found 2 potential issues.

[coderabbitai]

No actionable comments were generated in the recent review. 🎉

ℹ️ Recent review info

⚙️ Run configuration

Configuration used: Path: .coderabbit.yaml

Review profile: CHILL

Plan: Enterprise

Run ID: 83015aaf-1ff5-482a-843a-fd82ef714dee

📥 Commits

Reviewing files that changed from the base of the PR and between 975d6a1 and 86be9ef.

📒 Files selected for processing (2)

• lib/runtime/src/pipeline/network/tcp/server.rs
• lib/runtime/src/utils/ip_resolver.rs

---

Walkthrough

Changes

TCP IPv6 address handling

Layer / File(s)	Summary
Local IP resolution flow lib/runtime/src/pipeline/network/tcp/server.rs, lib/runtime/src/utils/ip_resolver.rs	A shared resolver retries IPv6 after IPv4 errors, preserves non-address errors, and supports loopback fallback. Tests cover strategy errors and IPv6 results.
Typed TCP address propagation lib/runtime/src/pipeline/network/tcp/server.rs	TcpStreamServer stores IpAddr values and uses SocketAddr values for startup, listener binding, logging, and registration.
Resolver and registration validation lib/runtime/src/pipeline/network/tcp/server.rs	Tests verify IPv6 fallback, IPv4 error preservation, and socket-address registration formatting.

Estimated code review effort: 3 (Moderate) | ~20 minutes

🚥 Pre-merge checks | ✅ 5

✅ Passed checks (5 passed)

Check name	Status	Explanation
Linked Issues check	✅ Passed	The changes retry IPv6 after all IPv4 errors, preserve relevant failures, and use typed socket addresses, meeting issue #7619.
Out of Scope Changes check	✅ Passed	All reported changes and regression tests support IPv6-only resolution, error handling, or valid TCP socket addressing.
Docstring Coverage	✅ Passed	No functions found in the changed files to evaluate docstring coverage. Skipping docstring coverage check.
Title check	✅ Passed	The title clearly identifies the primary change: adding IPv6-only IP resolution support in the runtime.
Description check	✅ Passed	The description provides detailed change scope, limitations, validation results, and issue references. It does not use the required template headings and omits the required Related Issues selection an…

✨ Finishing Touches 💡 1

🛠️ Fix failing CI checks 💡

• Create stacked PR
• Commit on current branch

---

_{Comment @coderabbitai help to get the list of available commands.}

[jthomson04]

Code review — IPv6-only IP resolution

15 findings. 11 are inline below; 4 land in files this PR doesn't touch, so they're listed here.

The one to look at first: the headline fix likely doesn't work on Linux — see the comment on fallback_loopback. local_ipv6() returns LocalIpAddressNotFound on typical IPv6-only Linux hosts, so such a host falls through to fallback_loopback and advertises [::1], while the interface list it already fetched has the routable global address in it.

The branch type-checks clean (cargo check -p dynamo-runtime --lib --tests, exit 0).

---

Findings in files outside this diff

These are pre-existing, but this PR materially widens the conditions under which they fire — local_ip_for_advertise() now returns IPv6 in many more cases than before (IPv4 probe errors rather than only LocalIpAddressNotFound, IPv4 addresses newly classed unusable, and loopback-only hosts).

1. lib/runtime/src/transports/event_plane/mod.rs:412 — ZMQ binds IPv4 wildcard, advertises a possibly-IPv6 address.
ZmqPubTransport::bind("tcp://0.0.0.0:0") (L399) listens on IPv4 only, then L412-413 builds public_endpoint = format!("tcp://{}:{}", local_ip_for_advertise(), actual_port). On a host where the IPv4 probe now fails or yields an unusable address but IPv6 succeeds — the exact new retry path this PR adds — that becomes tcp://[2001:db8::5]:54321. Subscribers connecting to the discovered endpoint get ECONNREFUSED, because the publisher socket is bound to 0.0.0.0. Same mismatch on a loopback-only host (tcp://[::1]:54321).

2. lib/llm/src/local_model.rs:720 — self URL can name a family the listener doesn't serve.
DEFAULT_SYSTEM_HOST = "0.0.0.0" (config.rs:18) → spawn_system_status_server("0.0.0.0", port) listens on IPv4 only. self_host_base_url sees system_host == "0.0.0.0" and substitutes local_ip_for_advertise(). Result: http://[2001:db8::5]:8081 or http://[::1]:8081 pointing at an IPv4-only listener → connection refused for every consumer.

3. lib/runtime/src/component/endpoint.rs:287 — the request plane still advertises the raw 0.0.0.0 string.
The bind-vs-advertise separation this PR introduces lives only inside TcpStreamServer, so the identical 0.0.0.0 bug is still live one module over. With DYN_TCP_RPC_HOST=0.0.0.0, manager.rs:270 binds 0.0.0.0:port (correct) and endpoint.rs:287-305 publishes TransportType::Tcp("0.0.0.0:5678/<id>/<name>") to discovery. TcpRequestClient::parse_address parses it fine and TcpStream::connect("0.0.0.0:5678") resolves to localhost on the client's node, so remote callers silently talk to themselves or fail. The new resolve_host_or_interface/ResolvedHost machinery handles exactly this but is only reachable from TcpStreamServer::new_with_resolver; local_model.rs:720 also still carries a hand-rolled "0.0.0.0" | "::" | "[::]" special case. Three call sites, three diverging wildcard policies — worth routing all three through the new resolver.

4. lib/runtime/src/config/environment_names.rs:479 — DYN_TCP_RESPONSE_STREAM_HOST docs describe the old behavior.
The comment still reads "Host/interface for the TCP response stream server. If unset, the server auto-detects a routable local IP." The variable now also accepts plain IPv4/IPv6 literals, bracketed IPv6 ([2001:db8::1]), and wildcards (0.0.0.0, ::, [::]) with distinct bind-vs-advertise behavior — and it now hard-rejects values it previously accepted (alias names containing :, link-local/multicast/broadcast literals). Nothing else in the repo mentions the variable, so operators have no way to learn the new grammar or the new rejection rules. Relatedly, ServerOptions::interface (server.rs:78) is now misnamed — it's a host-or-interface field.

[jthomson04]

Addressed the four review-body findings in 697d854a4d:

1. Direct ZMQ: derives the public endpoint from the actual bound SocketAddr and selects only a same-family advertisement address.
2. System status: adds SystemStatusServerInfo::advertised_socket_addr() and uses it for the LLM self-host URL and Python binding.
3. Request plane: resolves the raw DYN_TCP_RPC_HOST value when the shared server starts, stores the concrete advertised SocketAddr, and publishes that address after the real port is known.
4. Documentation and compatibility: documents the accepted literal, wildcard, and interface grammar plus family switching and fallback caching; retains ServerOptions::interface with clear compatibility documentation.

Focused local validation passed: 9 resolver tests, 36 serial TCP server tests, 1 request-plane test, 1 event-plane test, 1 system-status test, and 2 no-default-feature dynamo-llm tests. Runtime and LLM clippy passed with warnings denied; root and Python Rust format checks, the Python binding check, Fern structure check, and Fern broken-link check also passed.

[github-actions]

🌿 Fern Docs Preview: https://nvidia-preview-01a0b0f8-0c06-702e-bd95-f2a0811d474f.docs.buildwithfern.com/dynamo/dev

[dagil-nvidia]

Review of head 4eb4612: 1 must-fix, 5 consider. All gates pass, CI is green on this head, the resolver core is sound and well-tested, and every finding from the prior review rounds is verified addressed.

Must-fix

1. lib/runtime/src/config/environment_names.rs:737 and docs/fern/pages/reference/components/runtime-configuration.mdx:142 - both changed surfaces claim "Enumeration errors and loopback fallbacks are retried and are not cached," but for the two TCP servers these docs sit under, a loopback fallback is never retried: both servers resolve through once-cells, so a server that comes up advertising loopback holds it for the process lifetime; only an Err leaves the cell empty. The retry sentence is true only of local_ip_for_advertise(), which neither env var governs. Suggested scoping: "Automatic selection for advertised URLs caches only successful non-loopback results and re-resolves after enumeration errors and loopback fallbacks. The TCP servers resolve once at startup; a loopback fallback persists until restart, and an enumeration error fails server startup." Mirror in the env comment.

Consider

2. ip_resolver.rs:244 - the wildcard loopback fallback ignores the requested family: :: on a dual-loopback host binds 0.0.0.0 and advertises 127.0.0.1 even though ::1 is usable in the requested family; either try the requested family's loopback first or document the IPv4 preference.
3. event_plane/mod.rs:454 - the direct ZMQ publisher still binds 0.0.0.0, so IPv6-only support does not extend to the direct ZMQ event plane (it now advertises a family-consistent but unreachable loopback); worth a scope note or follow-up.
4. ip_resolver.rs:352 - tcp_rpc_host_from_env() has zero in-repo callers, is pub re-exported, and returns the raw env string; with the new grammar an external caller gets an interface name or wildcard as a URL host. Deprecate or route through the resolver.
5. manager.rs:100 - DYN_TCP_RPC_HOST/DYN_TCP_RPC_PORT are read as raw literals and absent from the canonical env registry while this PR documents them publicly; register the constants.
6. runtime-configuration.mdx:133 - "unspecified ... rejected" reads as contradicting the wildcard acceptance two sentences earlier, and "unscoped link-local rejected" implies scoped is accepted when it cannot even parse. Suggested: "Multicast, broadcast, and IPv6 link-local addresses are rejected; wildcards are accepted as bind addresses and are never advertised."

One adjacent pre-existing worth a one-line fix while in the area: lib/backend-common/src/rl.rs:67 carries a third copy of the old self-host-URL logic (substituting local_ip_for_advertise() on an unspecified bind), which can advertise a cross-family unreachable URL - the exact class this PR fixes in lib.rs and local_model.rs; info.advertised_socket_addr() closes it.

Verified sound: bracket round-trip through both TCP clients; the config grammar matches the docs including loud failures on malformed literals; the selection order and family-switch claims all confirmed; all 15 prior self-review findings and both bot findings addressed at head.

Comment-only review; severity calls stay with the human reviewers.

[jthomson04]

@dagil-nvidia Addressed the review feedback in 33edfb486a:

1. Scoped the cache and retry documentation to local_ip_for_advertise(). The TCP server docs now state that resolution occurs once at startup, loopback persists until restart, and enumeration errors fail startup.
2. Wildcard fallback now tries loopback in the requested family first. The new regression test covers dual-loopback, cross-family fallback, and an empty inventory.
3. Added a PR limitation that direct ZMQ still binds tcp://0.0.0.0; this PR only keeps its advertised address in the bound family.
4. Deprecated tcp_rpc_host_from_env() without removing the public API.
5. Added DYN_TCP_RPC_HOST and DYN_TCP_RPC_PORT to the canonical environment registry and replaced the raw string uses.
6. Clarified that wildcards are accepted bind addresses and are never advertised, while multicast, broadcast, and IPv6 link-local addresses are rejected.

The adjacent RL self-host path now uses SystemStatusServerInfo::advertised_socket_addr().

Validation passed: 10 resolver tests, 2 environment registry tests, 41 serial TCP server tests, the focused system-status and RL tests, formatting, and warning-denied clippy for dynamo-runtime and dynamo-backend-common. Fern still stops on the 26 generated API reference pages missing from this checkout, which is the same repository-level gap seen on the parent head.

[TaekyungHeo]

Additional dual-stack cross-node reproduction from Aria ComputeLab SC qualification:

• Interface ens6f0np0 carries IPv4 10.78.x.x and IPv6 link-local fe80::...%ens6f0np0.
• Dynamo vLLM 1.1 and SGLang 1.1/1.3.1 intermittently select the link-local address when DYN_TCP_RESPONSE_STREAM_HOST=ens6f0np0, then fail with Failed to start TcpListender on fe80::...:0: Invalid argument (os error 22).
• Supplying the IPv4 literal instead fails in released images with Interface not found: 10.78.x.x.
• One vLLM 1.3.1 run succeeded on the same fabric while another reached the IPv6 failure, demonstrating nondeterministic selection rather than a deterministic interface policy.
• Control rows on the same two-node Pyxis/TCP fabric pass: Native vLLM 4027060, Native SGLang 4027098, Native TensorRT-LLM 4026356, Dynamo TensorRT-LLM 4027434.

This matches this PR's need for one deterministic selection pass that prefers non-loopback IPv4 and accepts explicit literals. These are serving-protocol qualification canaries, not performance claims.

[copy-pr-bot]

This pull request requires additional validation before any workflows can run on NVIDIA's runners.

Pull request vetters can view their responsibilities here.

Contributors can view more details about this message here.

[harryskim]

Docs-only review, scoped to docs/fern/pages/reference/components/runtime-configuration.mdx. I checked each statement against the code on this head.

Accurate as written: the unset-selection order matches resolve_local_host/AddressCandidates; the wildcard family-switch matches resolve_wildcard; the link-local/multicast/broadcast rules match is_usable; the caching paragraph matches cached_local_ip_for_advertise; eth0:1 does resolve as an interface name (looks_like_ip_literal requires >= 2 colons); and the QUIC sentence holds, since TcpStreamServer::local_address() returns the advertised address that quic_response_server() reuses.

Two things I'd resolve before merge, both one-or-two-sentence fixes, flagged inline:

1. The "accepts the same values" claim for DYN_TCP_RESPONSE_STREAM_HOST is not true for whitespace — DYN_TCP_RPC_HOST does not trim, and the trimming sentence was deleted from the response-stream entry even though the behavior still holds there.
2. The DYN_EVENT_PLANE_HOST note understates a user-visible behavior change: on an IPv6-only host, automatic selection now advertises IPv4 loopback rather than the IPv6 address.

The rest of the inline notes are non-blocking follow-ups.

One caveat on verification: the PR description reports fern check and fern docs broken-links as blocked by absent generated API reference pages, so the docs build was not confirmed green for this change. The edits are balanced ParamField blocks, so I'd expect them to render, but that is inspection rather than a build.

[rmccorm4]

Three P2 findings reproduced during local review of 0d7c61ee9b.

[rmccorm4]

Summarizing TLDR of the changes

Situation	Before	After
IPv6-only machine	IPv4 detection could fail without finding the working IPv6 address.	Checks interface addresses directly; prefers usable IPv4, then IPv6.
Interface has multiple addresses	Could pick an unsuitable IPv6 link-local address.	Uses a consistent preference order and rejects unsuitable addresses. Automatic selection also skips down interfaces.
Server listens on a wildcard such as 0.0.0.0	Some paths advertised the wildcard itself, which remote machines cannot use as a destination.	Listens on the wildcard but advertises a concrete address.
Dynamically assigned port	Different paths constructed addresses independently.	Publishes the actual port assigned to the listener.

[dmitry-tokarev-nv]

Approved

I verified the address handling by running it. Every place the change builds or splits an endpoint string round-trips for both address families, the new tests are real, and the two open items are low.

Verified: string round-trips, downstream consumers, and a simulated IPv6-only host

Built at head f1b5aea3c0 in a worktree, macOS ARM64, cargo test -p dynamo-runtime --no-default-features. Merge base a10a46a8a0, dated 2026-09-17, so the branch is current.

Endpoint strings, run end to end:

Built at	Form for IPv6	Read back by	Result
tcp/server.rs:214 advertised address	[2001:db8::10]:1234	local_address(), TcpStreamConnectionInfo	parses back to the same SocketAddr
component/endpoint.rs:346 transport	[2001:db8::10]:1234/2a/generate	TcpRequestClient::parse_address	addr=[2001:db8::10]:1234 endpoint="2a/generate"
event_plane/mod.rs:87 ZMQ endpoint	tcp://[2001:db8::20]:4321	ZMQ connect string	family kept, port kept
rl.rs:93, local_model.rs:895, lib.rs:1475	http://[::1]:8080	URL consumers, admin.py	brackets present
tcp/client.rs:368 TLS name	from [2001:db8::10]:1234	ServerName::try_from	Ok(IpAddress(V6(...)))

A zone identifier is refused and never silently reformatted: fe80::1%en0 and [2001:db8::1%eth0] both give InvalidLiteral. Link-local unicast is not a candidate in the first place.

Simulated IPv6-only host, because this machine has no IPv6-only interface. I gave TcpStreamServer a resolver whose whole inventory is IPv6, then bound and connected for real:

IPV6ONLY advertised=[::1]:61178
IPV6ONLY connected peer=Ok([::1]:61178)
EXPLICITLL  err="... interface has no usable IP address: eth0"   # only fe80::1 present
LINKLOCALONLY advertised=127.0.0.1:61180                          # automatic path, warns
EMPTYINV      advertised=127.0.0.1:61176                          # automatic path, warns
IPV6GLOBAL    err="... Can't assign requested address (os error 49)"

So an empty result or a link-local only interface does not fail startup. It warns and serves loopback. An enumeration error does fail startup, which tcp_server_preserves_interface_enumeration_error pins. The base tree reached the same 127.0.0.1 in those cases, so this is not new.

The new tests are real. Five mutations of the changed code, each turning tests red, with the unmutated tree green at 65 passed:

Mutation	Tests that turned red
prefer IPv6 before IPv4	interface_alias_preserves_order..., mapped_interface_addresses...
drop the IPv6 brackets in format_host	transient_fallbacks_are_not_cached_but_success_is
drop the IFF_UP filter	system_inventory_keeps_only_up_ip_addresses
drop to_canonical in consider	mapped_interface_addresses_use_ipv4_classification
drop trim in resolve_host_or_interface	5 tests, including two real binds in tcp::server

The lane is rust-tests (.), .github/workflows/pre-merge.yml:437, cargo test --locked --all-targets. It ran at head f1b5aea3c0 and passed, so the new tests executed.

Three event-plane ZMQ tests fail on this Mac. They fail the same way on the merge base, so they are not from this change.

Two open items, both low, both already stated in the PR body and the docs page. Neither blocks. Skipped on purpose: I did not run this on a real IPv6-only host, and I did not exercise the QUIC response listener under a wildcard host.

[A-Ishak]

pls release 🙏

[dagil-nvidia]

Thanks for the push on this. The IPv6 fix, plus the follow-up in #15324 that it needs, is too large to hand-port safely into 1.5.1, which ships today. Both are on main and will be in 1.6.0. Until then, the nightly containers have them.