No IPV6 for some CDN routes

[alandillon]

[alandillon]

[Krinkle]

Please confirm that using the same device and network connection, https://ipv6-test.com/ reports back as "IPv6 Supported".

And that e.g. the following example from Wikipedia does respond for you over IPv6-only:

$ curl -I -6 'https://en.wikipedia.org/favicon.ico'
HTTP/2 200
...

[vpereira01]

I also have issues with IPv6 and code.jquery.com similar to this and previous one reported.

Can't reach https://code.jquery.com/jquery-3.6.0.min.js on Windows and Linux with IPv6 is enabled.
Using ISP DNS server or Cloudflare DNS server has the same behavior.

IPv6 error

user@host:~$ curl -6 -v https://code.jquery.com/jquery-3.6.0.min.js
*   Trying 2001:4de0:ac18::1:a:3b:443...
* Connected to code.jquery.com (2001:4de0:ac18::1:a:3b) port 443 (#0)
* ALPN, offering h2
* ALPN, offering http/1.1
* successfully set certificate verify locations:
*  CAfile: /etc/ssl/certs/ca-certificates.crt
*  CApath: /etc/ssl/certs
* TLSv1.3 (OUT), TLS handshake, Client hello (1):
* TLSv1.3 (IN), TLS handshake, Server hello (2):
* TLSv1.3 (IN), TLS handshake, Encrypted Extensions (8):
* TLSv1.3 (IN), TLS handshake, Certificate (11):
* TLSv1.3 (IN), TLS handshake, CERT verify (15):
* TLSv1.3 (IN), TLS handshake, Finished (20):
* TLSv1.3 (OUT), TLS change cipher, Change cipher spec (1):
* TLSv1.3 (OUT), TLS handshake, Finished (20):
* SSL connection using TLSv1.3 / TLS_AES_128_GCM_SHA256
* ALPN, server accepted to use h2
* Server certificate:
*  subject: CN=*.jquery.com
*  start date: Jul 14 00:00:00 2021 GMT
*  expire date: Aug 14 23:59:59 2022 GMT
*  subjectAltName: host "code.jquery.com" matched cert's "*.jquery.com"
*  issuer: C=GB; ST=Greater Manchester; L=Salford; O=Sectigo Limited; CN=Sectigo RSA Domain Validation Secure Server CA
*  SSL certificate verify ok.
* Using HTTP2, server supports multi-use
* Connection state changed (HTTP/2 confirmed)
* Copying HTTP/2 data in stream buffer to connection buffer after upgrade: len=0
* Using Stream ID: 1 (easy handle 0x559b34b6a5e0)
> GET /jquery-3.6.0.min.js HTTP/2
> Host: code.jquery.com
> user-agent: curl/7.74.0
> accept: */*
> 
* OpenSSL SSL_read: Connection reset by peer, errno 104
* Failed receiving HTTP2 data
* OpenSSL SSL_write: Broken pipe, errno 32
* Failed sending HTTP2 data
* Connection #0 to host code.jquery.com left intact
curl: (56) OpenSSL SSL_read: Connection reset by peer, errno 104

Wikipedia IPv6 check

user@host:~$ curl -I -6 'https://en.wikipedia.org/favicon.ico'
HTTP/2 200 
date: Thu, 07 Apr 2022 18:01:19 GMT
server: mw1326.eqiad.wmnet

IPv4 no issue

user@host:~$ curl -4 https://code.jquery.com/jquery-3.6.0.min.js > a.txt
  % Total    % Received % Xferd  Average Speed   Time    Time     Time  Current
                                 Dload  Upload   Total   Spent    Left  Speed
100 89501  100 89501    0     0   794k      0 --:--:-- --:--:-- --:--:--  794k

PS: Also tested IPV6 with different http versions, using CURL command options, but same result.

[Krinkle]

Can you confirm that this is not influenced by local middleware or adapters such as Intel's Killer Control Center?

#77
#80 (comment)

[alandillon]

It was not. This was related to a hotspot on verizon which I no longer have access to. We turned on IPV6 functionality on the hotspot and it worked then.

But sadly I am no longer able to reproduce this with that hotspot.

[vpereira01]

I can confirm that Intel's Killer Control Center seems unrelated since I see the issue on Windows and Linux.

Also tested in my network with an iPhone/Safari opening the URL directly and it fails sometimes, probably due to browser IPv4 fallback. it seems that browsers have an automatic IPv4 fallback and that's why this issue probably goes unnoticed https://en.wikipedia.org/wiki/Happy_Eyeballs .

Did further testing using curl and the issue seems related to TLS sessions somehow. Check my terminal logs https://gist.github.com/vpereira01/805d742131eee657fa1dbd8810c6588f#file-curl-log

All request fails except the one where this is seen

> GET /jquery-3.6.0.min.js HTTP/2
> Host: code.jquery.com
> user-agent: curl/7.74.0
> accept: */*
> 
{ [5 bytes data]
* TLSv1.3 (IN), TLS handshake, Newsession Ticket (4):
{ [217 bytes data]
* TLSv1.3 (IN), TLS handshake, Newsession Ticket (4):
{ [217 bytes data]
* old SSL session ID is stale, removing
{ [5 bytes data]
* Connection state changed (MAX_CONCURRENT_STREAMS == 100)!
} [5 bytes data]
< HTTP/2 200 

[Krinkle]

@vpereira01 Thanks, I will file an issue with Highwinds support to look into this.

Meanwhile, could you confirm if this happens with other sites that use this vendor? From a quick search and verifying the headers, it seems https://www.maketecheasier.com/ also uses HW for example.

[vpereira01]

Tested with a couple of URLs and the issue really seems to be at Highwinds.

An easier to test is https://ifood.tv/ which has URLs that directly access an Highwinds host and the behaviour is similar when most requests fail https://gist.github.com/vpereira01/d3e5544f05b3bfe52ee3932368cfb2ed#file-curl-hw-ipv6-ifoodtv-log

Also tested another URL, from an adult site, which uses CNAME and the behavior was the same.

[Krinkle]

@vpereira01

Highwinds support wrote:

Our network team made tests from all of our points of presence against https://code.jquery.com/jquery-3.6.0.min.js with IPv6 enabled in cURL and was unable to replicate the problem. Can you inquire if the client/clients have this problem still?
What is their location? What OS are they running the tests on?
What is their cURL version? Can they use wget -6 as well for testing? Can they do a packet capture?

(I'm aware the snippet I shared already included your cURL version.)

[vpereira01]

Decided to dig a bit more and the issue is caused by:

• My home router IPv6 compatibility issue
• Highwinds CDN IPv6 compatibility issue

My router seems to be changing IPv6 Flow Label values during a connection which makes Highwinds CDN drop the connection. I was able to confirm this by disabling IPv6 Flow Labels (always set it to zero) which made the requests work as expected ("script" bellow).

My ISP provided router behavior seems to be a bug but I think this is also Highwinds CDN IPv6 compatibility issue given other CDNs don't show this behavior.

Furthermore, given this post by a Fastly CDN engineer it seems that Highwinds should review their usage of IPv6 Flow Labels.

How I confirmed the issue:

user@host:~# sysctl -n net.ipv6.auto_flowlabels
1

user@host:~# curl -6 https://code.jquery.com/jquery-3.6.0.min.js -o /dev/null
  % Total    % Received % Xferd  Average Speed   Time    Time     Time  Current
                                 Dload  Upload   Total   Spent    Left  Speed
  0     0    0     0    0     0      0      0 --:--:-- --:--:-- --:--:--     0
curl: (56) OpenSSL SSL_read: Connection reset by peer, errno 104

user@host:~# sysctl -w net.ipv6.auto_flowlabels=0
net.ipv6.auto_flowlabels = 0

user@host:~# curl -6 https://code.jquery.com/jquery-3.6.0.min.js -o /dev/null
  % Total    % Received % Xferd  Average Speed   Time    Time     Time  Current
                                 Dload  Upload   Total   Spent    Left  Speed
100 89501  100 89501    0     0   816k      0 --:--:-- --:--:-- --:--:--  816k

user@host:~# sysctl -w net.ipv6.auto_flowlabels=1
net.ipv6.auto_flowlabels = 1

user@host:~# curl -6 https://code.jquery.com/jquery-3.6.0.min.js -o /dev/null
  % Total    % Received % Xferd  Average Speed   Time    Time     Time  Current
                                 Dload  Upload   Total   Spent    Left  Speed
  0     0    0     0    0     0      0      0 --:--:-- --:--:-- --:--:--     0
curl: (56) OpenSSL SSL_read: Connection reset by peer, errno 104

Please share this information with Highwinds and I would advise to ask them to improve their IPv6 compatibility.

[Krinkle]

@vpereira01 Thanks. Highwinds support tells me their engineers actively looking into it now, specifically with awareness of IPv6 Flow Labels.

They did in addition mention that any of the following would be of great help meanwhile:

• Source IPv4 and IPv6 addresses.
• MTR from those source addresses towards code.jquery.com, or any other affected site on our network.
• Date/time of request(s) that failed and request/response headers, or even a packet dump from failed request(s).

They acknowledged that some of these were already in your gists.

[jdoupe]

@vpereira01 - This may have actually been resolved for you (Portugal, likely hitting SP/Highwinds PoP in Madrid)

Further confirmation or any of the other information requested above would be greatly helpful and appreciated!

[vpereira01]

Yes, I can confirm the requests are successful now, great :)

Thanks

[ghost]

@vpereira01 I'm Nick and a senior member of the StackPath support team. We'd like to thank you for helping identify the IPv6 Flow Label issues. As a token of appreciation, we'd like to send some goodies to you. If you can, can you shoot an email to [email protected] and reference this post and my name so we can get the ball rolling?

[vpereira01]

A nice surprise on a Friday 🍻 , will do that.

[alandillon]

Glad my original post was able to help reach a resolution to this issue! Awesome!