Implement RFC6555 Happy Eyeballs support for dual stack IPv4 and IPv6 hosts

[pkazmier]

[disclaimer: Rust experience < 1 week]

While writing, literally, my first Rust program, I needed an HTTP client to fetch XKCD comic info. After a little searching, I discovered Reqwest (and thus Hyper). Much to my dismay, I discovered that 25% of the time, my HTTP requests were timing out on my MacBook. To assist in my troubleshooting, I made a small test program to rule out user error given my limited Rust experience:

extern crate reqwest;
extern crate serde_json;

fn main() {
    let mut resp = reqwest::get("http://xkcd.com/1849/info.0.json").unwrap();
    let json: serde_json::Value = resp.json().unwrap();
    println!("Response: {:?}", json["num"]);
}

The problem still persisted (although not consistently), so I also wrote a small test program in Go to compare results. In addition, I fired up Wireshark on my laptop to get packet captures the next time the problem occurred. Here are my findings:

• xkcd.com has four A and four AAAA DNS resource records for IPv4 and IPv6 addresses.
• The Rust test program attempts to connect to one of the IPv6 addresses, but the remote host does not respond, which is why it times out. The packet capture confirms as it shows only a series of unacknowledged SYN packets being sent from my laptop.

     96 0.000000       MacBook               2a04:4e42:200::67     TCP      98     53620 → 443 [SYN] Seq=0 Win=65535 Len=0 MSS=1412 WS=32 TSval=456198563 TSecr=0 SACK_PERM=1
    104 1.001939       MacBook               2a04:4e42:200::67     TCP      98     [TCP Retransmission] 53620 → 443 [SYN] Seq=0 Win=65535 Len=0 MSS=1412 WS=32 TSval=456199563 TSecr=0 SACK_PERM=1
    113 1.002891       MacBook               2a04:4e42:200::67     TCP      98     [TCP Retransmission] 53620 → 443 [SYN] Seq=0 Win=65535 Len=0 MSS=1412 WS=32 TSval=456200563 TSecr=0 SACK_PERM=1
    115 1.001134       MacBook               2a04:4e42:200::67     TCP      98     [TCP Retransmission] 53620 → 443 [SYN] Seq=0 Win=65535 Len=0 MSS=1412 WS=32 TSval=456201564 TSecr=0 SACK_PERM=1
    117 1.000064       MacBook               2a04:4e42:200::67     TCP      98     [TCP Retransmission] 53620 → 443 [SYN] Seq=0 Win=65535 Len=0 MSS=1412 WS=32 TSval=456202564 TSecr=0 SACK_PERM=1
    134 1.002307       MacBook               2a04:4e42:200::67     TCP      98     [TCP Retransmission] 53620 → 443 [SYN] Seq=0 Win=65535 Len=0 MSS=1412 WS=32 TSval=456203564 TSecr=0 SACK_PERM=1
    145 2.001778       MacBook               2a04:4e42:200::67     TCP      98     [TCP Retransmission] 53620 → 443 [SYN] Seq=0 Win=65535 Len=0 MSS=1412 WS=32 TSval=456205564 TSecr=0 SACK_PERM=1
    167 4.002416       MacBook               2a04:4e42:200::67     TCP      98     [TCP Retransmission] 53620 → 443 [SYN] Seq=0 Win=65535 Len=0 MSS=1412 WS=32 TSval=456209564 TSecr=0 SACK_PERM=1
    218 8.012844       MacBook               2a04:4e42:200::67     TCP      98     [TCP Retransmission] 53620 → 443 [SYN] Seq=0 Win=65535 Len=0 MSS=1412 WS=32 TSval=456217564 TSecr=0 SACK_PERM=1
    484 16.062809      MacBook               2a04:4e42:200::67     TCP      98     [TCP Retransmission] 53620 → 443 [SYN] Seq=0 Win=65535 Len=0 MSS=1412 WS=32 TSval=456233564 TSecr=0 SACK_PERM=1
    726 32.051029      MacBook               2a04:4e42:200::67     TCP      82     [TCP Retransmission] 53620 → 443 [SYN] Seq=0 Win=65535 Len=0 MSS=1412 SACK_PERM=1

• While the issue was happening with my Rust program, both the Go program and a simple curl to xkcd.com worked successfully. In both cases, the packet captures show that a SYN packet is sent to the IPv6 address first, and goes unanswered. Then 200ms (curl) / 300ms (Go) later, a new SYN packet is sent to the IPv4 address, which does respond and acknowledge, so the HTTP GET succeeds.

     77 0.002330       MacBook               2a04:4e42:200::67     TCP      98     53782 → 443 [SYN] Seq=0 Win=65535 Len=0 MSS=1412 WS=32 TSval=456993693 TSecr=0 SACK_PERM=1
     79 0.285284       MacBook               151.101.0.67          TCP      78     53783 → 443 [SYN] Seq=0 Win=65535 Len=0 MSS=1460 WS=32 TSval=456993994 TSecr=0 SACK_PERM=1
     80 0.007752       151.101.0.67          MacBook               TCP      74     443 → 53783 [SYN, ACK] Seq=0 Ack=1 Win=26960 Len=0 MSS=1360 SACK_PERM=1 TSval=1551435559 TSecr=456993994 WS=512
     81 0.000078       MacBook               151.101.0.67          TCP      66     53783 → 443 [ACK] Seq=1 Ack=1 Win=132096 Len=0 TSval=456994002 TSecr=1551435559
     82 0.000106       MacBook               151.101.0.67          TLSv1.2  243    Client Hello
     83 0.008295       151.101.0.67          MacBook               TLSv1.2  1414   Server Hello

• I also confirmed that when the IPv6 hosts were responding that all three (Rust, Go, and curl) fulfilled the HTTP GET request via the IPv6 host, and not the IPv4 host.

• After a little research, it appears that they are implementing RFC6555 or the Happy Eyeballs algorithm due to the unreliability of IPv6 today (as evidenced by the sporadic connectivity I have to the xkcd.com IPv6 hosts). The algorithm essentially sends a request to the IPv6 address first, but if it does not respond in a short time frame, the host should try to connect to the IPv4 address. I then went to the source code for both Go and Curl to confirm.

  • Here is a link to the RFC: https://tools.ietf.org/html/rfc6555
  • Here is a link to the algorithm used by Chrome and Mozilla: https://tools.ietf.org/html/rfc6555#section-6
  • Here is a link to the Go implementation: https://github.com/golang/go/blob/master/src/net/dial.go#L46
  • Here is a link to the Curl implementation:
    https://github.com/curl/curl/blob/001822153201e1e9b0246bc64a3797eb18f5ed61/lib/connect.h#L43

Interestingly, it appears Go chooses to implement this in their networking stack, so as to not make it a burden for applications. I'm not familiar with Rust, Tokio, or Hyper, but it looks like the DNS resolution and initiation of a TCP connection to a specific IP address is done at the Hyper layer, https://github.com/hyperium/hyper/blob/master/src/client/connect.rs#L205 , so I'm not sure the same opportunity exists here in the current shape of these libraries.

In summary, as a user with a dual IP stack (IPv4 and IPv6), I need a reliable HTTP client that works in the face of the inherent IPv6 instability that exists in today's Internet. RFC6555 was designed to help improve the client experience as the world shifts from IPv4 to IPv6, without resorting to techniques to further cement in IPv4 (i.e. disabling IPv6 stack for example).

[seanmonstar]

This is certainly something that should be done. As to where exactly, I don't know, but we can discuss that! And as part of the discussion, if anyone wants to try to implement this out-of-tree, it should be possible to test using hyper::Client with it, by just implementing hyper::client::Connect. With that, a client can be configured to use it.

After discussing and experimenting, we should be able to merge the solution into the right place (either here in hyper, or up the stack if that makes sense.)

[seanmonstar]

On second thought, a fix here in hyper wouldn't be that much code, actually. It'd simply adjust the HttpConnector to filter the resolved addresses into IPv4 and IPv6, and then start resolving IPv6 addresses first, selecting on a tokio_core::reactor::Timeout, and then start racing the against IPv4 connects.

[pkazmier]

It would seem to be simple to implement in Hyper as a short-term fix, but I wonder if there is an opportunity for Tokio to provide Happy Eyeballs capability, so every Hyper (or other network client) that comes along does not have to do so. This would mean that the DNS resolution and TCP connection establishment would need to be moved down into Tokio. Happy Eyeballs is something that most developers aren't going to be aware of, and it probably should be provided lower down in the networking stack. But in the meantime, I'm all for a short-term Hyper-only solution as the client is not reliable for me in its current state.

[mehcode]

@seanmonstar Any hope of including this in the upcoming v0.12 release? If what you said still holds true, it's not much code in hyper for a short-term fix.

Most of my "real world" (in production) issues come from my stuff trying to use dodgy ipv6 connections.

[seanmonstar]

I'd merge a PR fixing this, certainly!

As for myself, my time is short, I'm spread thin over so many other things :(

[estk]

I'm interested in taking this on as my next issue, but it wont be until im able to wrap up by current pr. That one turned out to be a Doozy!

[pkazmier]

It is great to see this addressed in Hyper. It is, however, unfortunate that it had to be addressed in Hyper instead of in a lower-layer of the networking stack for Rust. Others not using Hyper or HTTP are going to encounter the exact same issue. Most languages solve this lower in the stack to avoid this very situation. Unfortunately, I'm not familiar with Rust enough to suggest the ideal location.