Dashboard server sometimes only listens on IPv6 loopback address by default instead of both IPv4 and IPv6

[benjamincburns]

Issue

While helping @OnlyOneJMJQ troubleshoot an issue with the truffle dashboard, we encountered a problem where the dashboard's HTTP server was listening only on the IPv6 loopback interface, while his truffle migrate command was attempting to connect to the IPv4 interface. This resulted in a mix of ECONNREFUSED and ECONNTIMEOUT errors from the truffle migrate process as it was attempting to connect to the dashboard server process.

Steps to Reproduce

1. Ensure that you don't have dashboard.host specified in your truffle-config.js
2. In one terminal instance run truffle dashboard to spin up the dashboard process
3. In another terminal run truffle migrate --network dashboard

Note: this very likely only reproduces with Node v17+ (see last paragraph of "underlying cause" discussion below).

Expected Behavior

The truffle migrate command should succeed as expected.

Actual Results

A mix of ECONNREFUSED and ECONNTIMEOUT errors are reported by the truffle migrate process, and that process exits with a failure code.

Short-term workaround

This issue can be worked around by explicitly setting dashboard.host to "127.0.0.1" or "::1" in truffle-config.js, and ensuring that the value of the dashboard network config (the string, function, or object assigned to networks.dashboard) matches.

Underlying cause

Most modern operating systems have two entries for localhost in their /etc/hosts (or in the case of Windows, C:\Windows\system32\drivers\etc\hosts) files. The first entry usually resolves to the IPv4 loopback address (127.0.0.1), and the second usually resolves to the IPv6 loopback address (::1).

For some reason when we initialize the dashboard's instance of http.Server via express with a hostname value of localhost, it's electing to listen only on the IPv6 loopback interface. To address this, back in node v11.4.0 they added an ipv6Only field to the options object that's passed into the net.Server.listen method.

What's confusing here is that ipv6Only defaults to false, which should trigger "dual stack" support for net.Server.listen, and by extension, http.Server.listen, as http.Server.listen is identical to net.Server.listen.

However in the dashboard code we don't appear to call httpServer.listen directly. Rather, we appear to call the listen method of the ExpressJS app object.

In Node v17.0.0 they switched dns.lookup to default to verbatim=true, which per nodejs/node#39987 (the PR that introduced the change) has the effect of defaulting to IPv6 addresses on name resolution over IPv4 addresses. My strong suspicion is that rather than just passing the unmodified host string off to the built-in listen method, Express is attempting to resolve the hostname and instead providing the resolved IP address. As a result, I strongly suspect that this issue doesn't reproduce prior to node v17.

Proposed fix

Either we need to change up the code to avoid calling expressApp.listen and instead call httpServer.listen ourselves (this can be done by creating the http.Server instance ourselves by calling http.createServer({ ... }, this.expressApp!)), or we need to take extra steps to resolve the IP addresses for the given host and be sure to listen on both protocols.

Environment

• Operating System: macOS 12.6
• Truffle version (truffle version): v5.6.3 (core: 5.6.3)
• node version (node --version): v18.12.0

[benjamincburns]

Hmm, it doesn't seem that ExpressJS is the culprit. When we call expressApp.listen, express is just proxying that call over to httpServer.listen (permalink points to current, at time of writing, HEAD of 4.x branch). 🤔🤔🤔

[benjamincburns]

On further inspection, I don't think the net package makes any effort to bind to both IPv4 and IPv6 addresses on a call to listen.

I traced the code through to where it opens up the file descriptor to listen on the port, and the logic is clearly mutually exclusive.

I have however confirmed the behaviour of the listen function with respect to hostname resolution. That is, listen calls an internal function named lookupAndListen, which uses the signature of dns.lookup that only returns the first-resolved address for the given hostname.

Personally I regard this as a NodeJS bug. Ideally lookupAndListen would pass { all: true } to dns.lookup, and listen on each distinct IP address that's resolved for a given hostname.

In the meantime my suggestion is that we implement exactly that functionality in our code as a workaround to this problem. Doing this will have the effect of fixing the issue not only for the localhost hostname, but also for all other hostnames that resolve dual addresses.

[benjamincburns]

Also related:

• Unable to connect to establish a TCP connection (ECONNREFUSED) nodejs/node#40702
• Happy Eyeballs support (address IPv6 issues in Node 17) nodejs/node#41625
  • See this Wikipedia article on the Happy Eyeballs algorithm
  • this likely will fix this issue when implemented, but said fix won't apply to current versions of node

The first issue above suggests alternative workaround that is arguably simpler to implement than the one I proposed in my previous comment: simply setting dns.setDefaultResultOrder("ipv4first") at process start.

However to work properly this workaround will need to be set on the calling clients as well. Non-node clients like browsers and curl implement the Happy Eyeballs algorithm, and therefor would still work with this workaround in place.

That said, I think the more complex workaround that I mentioned above comes with fewer caveats, likely isn't all that complex to implement, and will be more robust overall.

[benjamincburns]

Looks like changing it to explicitly listen on both protocols will work.

Note that I haven't changed the message bus yet (will need to do that as part of this), but for only the dashboard server's http.Server instance here's the before/after view as shown by lsof -i -P | grep LISTEN:

Before:

node      43379 bburns   23u  IPv4 0x8449cb6c5435c875      0t0  TCP localhost:63144 (LISTEN)
node      43379 bburns   24u  IPv4 0x8449cb6c54345c35      0t0  TCP localhost:63145 (LISTEN)
node      43379 bburns   27u  IPv4 0x8449cb6c4fd76745      0t0  TCP localhost:24012 (LISTEN)

After:

node      44080 bburns   25u  IPv6 0x8449cb75e5f647bd      0t0  TCP localhost:63164 (LISTEN)
node      44080 bburns   26u  IPv6 0x8449cb75e5f64f3d      0t0  TCP localhost:63165 (LISTEN)
node      44080 bburns   29u  IPv6 0x8449cb75e5f665bd      0t0  TCP localhost:24012 (LISTEN)
node      44080 bburns   30u  IPv4 0x8449cb6c4fd87d65      0t0  TCP localhost:24012 (LISTEN)