[release/1.0.0] HttpClientHandler tests failed with SSL Connect error on Debian and OSX

[Priya91]

In both master and release, hence tagging this issue for RTM, pending investigation.

MESSAGE:
System.Net.Http.HttpRequestException : An error occurred while sending the request.\n---- System.Net.Http.CurlException : SSL connect error
+++++++++++++++++++
STACK TRACE:
at System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess(Task task) at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task) at System.Runtime.CompilerServices.ConfiguredTaskAwaitable`1.ConfiguredTaskAwaiter.GetResult() at System.Net.Http.HttpClient.<FinishSendAsync>d__58.MoveNext() --- End of stack trace from previous location where exception was thrown --- at System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess(Task task) at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task) at System.Runtime.CompilerServices.TaskAwaiter`1.GetResult() at System.Net.Http.Functional.Tests.HttpClientHandlerTest.<SendAsync_RequestVersion20_ResponseVersion20IfHttp2Supported>d__69.MoveNext() --- End of stack trace from previous location where exception was thrown --- at System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess(Task task) at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task) --- End of stack trace from previous location where exception was thrown --- at System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess(Task task) at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task) --- End of stack trace from previous location where exception was thrown --- at System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess(Task task) at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task) ----- Inner Stack Trace ----- at System.Net.Http.CurlHandler.ThrowIfCURLEError(CURLcode error) at System.Net.Http.CurlHandler.MultiAgent.FinishRequest(StrongToWeakReference`1 easyWrapper, CURLcode messageResult)

Test runs here
http://dotnet-ci.cloudapp.net/job/dotnet_corefx/job/release_1.0.0/job/debian8.4_debug/1/

http://dotnet-ci.cloudapp.net/job/dotnet_corefx/job/release_1.0.0/job/debian8.4_release/1/

http://dotnet-ci.cloudapp.net/job/dotnet_corefx/job/release_1.0.0/job/osx_debug/78/

http://dotnet-ci.cloudapp.net/job/dotnet_corefx/job/master/job/debian8.4_release/24/testReport/

[ericeil]

I cannot repro this locally. The test connects to a remote, public, service (https://revoked.grc.com/). It seems likely that this service was simply having temporary issues when these tests ran.

If this ends up being a common occurrence, we should find a way to make these tests more robust (use a local server, or something). But for now, I don't think there's anything to do here.

[stephentoub]

If this ends up being a common occurrence, we should find a way to make these tests more robust (use a local server, or something). But for now, I don't think there's anything to do here.

Yeah, we've been trying to move off of these remote servers to local loopback servers, but it's been a bit slow-going. In this case, we needed a server capable of HTTP/2.

[Priya91]

This is reproing consistently on release debian runs, reopening this.

@ericeil FYI

[davidsh]

Yeah, we've been trying to move off of these remote servers to local loopback servers, but it's been a bit slow-going. In this case, we needed a server capable of HTTP/2.

We can't have all of our tests use loopback servers. We still need end-to-end coverage. And many times, bugs will not found when network i/o goes thru the loopback adapter. So, we need to consider this before changing more tests to loopback.

What we need, instead, is an Xunit attribute with [Retries=n] so that the test can be retried if needed due to network congestion.

[davidsh]

cc: @CIPop

[stephentoub]

We can't have all of our tests use loopback servers. We still need end-to-end coverage. And many times, bugs will not found when network i/o goes thru the loopback adapter. So, we need to consider this before changing more tests to loopback.

All? No, I'm not suggesting "all". But more? Yes. For example, there's no strong reason why the majority of the tests we have using the echo server endpoints for testing things like redirection need to go off to a machine in the cloud.

What we need, instead, is an Xunit attribute with [Retries=n] so that the test can be retried if needed due to network congestion.

Retries don't help when servers out of our control go down for periods of time. That was one of the primary reliability issues with some of the certificate-related tests, where the target servers would go down for periods of time, issues that largely went away by moving the innerloop tests to use a loopback server where possible and moving the tests that went off-box to outerloop.

That said, I don't actually think that's the case here. I think the version of curl installed on those machines is having trouble with HTTP/2 + HTTPS.

[stephentoub]

This looks like a machine configuration issue. I ran:

curl https://http2.akamai.com/ --trace-ascii /dev/stdout

via the Jenkins Script Console on one of the Debian VMs, and I get this:

== Info: Hostname was NOT found in DNS cache
== Info:   Trying 23.38.225.180...
== Info: Connected to http2.akamai.com (23.38.225.180) port 443 (#0)
== Info: successfully set certificate verify locations:
== Info:   CAfile: none
  CApath: /etc/ssl/certs
== Info: SSLv3, TLS handshake, Client hello (1):
=> Send SSL data, 512 bytes (0x200)
0000: ......+.....<c...B...A.[#zi.....].P]....v.0.,.(.$.........k.j.9.
0040: 8.....2...*.&.......=.5.../.+.'.#[email protected].).
0080: %.......<./...A...................].........http2.akamai.com....
00c0: .......4.2..................................................... 
0100: ................................................................
0140: ................................................................
0180: ................................................................
01c0: ................................................................
== Info: SSLv3, TLS handshake, Server hello (2):
<= Recv SSL data, 89 bytes (0x59)
0000: ...U....H.�...". +.......v..T......T_. .A>G.+....m(....8..(..}h(
0040: ....m...0................
== Info: SSLv3, TLS handshake, CERT (11):
<= Recv SSL data, 3770 bytes (0xeba)
0000: .........n0..j0..R.......p.W]x.7.&.+....8.%.L0...*.H........0..1
0040: .0...U....NL1.0...U....Amsterdam1%0#..U....Verizon Enterprise So
0080: lutions1.0...U....Cybertrust1.0,..U...%Verizon Akamai SureServer
00c0:  CA G14-SHA20...160103235701Z..170103235652Z0n1.0...U....US1.0..
0100: .U....CA1.0...U....Santa Clara1!0...U....Akamai Technologies Inc
0140: .1.0...U....http2.akamai.com0.."0...*.H.............0......... .
0180: /..z....I....x.....y..Y.....Ww.w!....c.R.........k.....G4...D..?
01c0: .=4...Q...So.....qvB[ ........o..G.w..*A....\...G...{2v.y_KL4...
0200: 7.U_'....6..Cp.)..Q..7......�FGt...�.*..'.1D..n+;J.G:.}..;e.6|*.
0240: ..:}.D./H.iH.p..<..,..i9._K..[;...2....N....�P...rA...&.l...W...
0280: ......0...0...U.......0.0L..U. .E0C0A..+.....>.20402..+........&
02c0: https://secure.omniroot.com/repository0....+..........0..0-..+..
0300: ...0..!http://vassg142.ocsp.omniroot.com06..+.....0..*https://ca
0340: cert.a.omniroot.com/vassg142.crt06..+.....0..*https://cacert.a.o
0380: mniroot.com/vassg142.der0...U....0...http2.akamai.com0...U......
03c0: .....0...U.%..0...+.........+.......0...U.#..0.......sw....KM...
0400: 3..r.0>..U...70503.1./.-http://vassg142.crl.omniroot.com/vassg14
0440: 2.crl0...U.......w...1.&r....hY8...\0...*.H.............7.....5:
0480: ..%#I4;..]o...g....(......IG........RW*.<).t..-.Q}...�..S[..E..w
04c0: ..C7....)F....)_{C."\04.i.X.+..&.e.$......S..%....,\m.s=..@..$tY
0500: (I..P...h(j...j.w2..V#../...g...o......E..X.<..T.d...@[email protected](z.�
0540: /.s<..h|#t4 c.a>$Ye<@}...J+...h.uRM..D...:.%...W..-..-....#0...0
0580: ...........'.k0...*.H........0Z1.0...U....IE1.0...U....Baltimore
05c0: 1.0...U....CyberTrust1"0 ..U....Baltimore CyberTrust Root0...140
0600: 402143610Z..210402143552Z0..1.0...U....NL1.0...U....Amsterdam1%0
0640: #..U....Verizon Enterprise Solutions1.0...U....Cybertrust1.0,..U
0680: ...%Verizon Akamai SureServer CA G14-SHA20.."0...*.H............
06c0: .0.........n..i......d2jY.... ..H.....G..9@. ].....p.....b....^5
0700: .v.#.o..F5YZ\..#... .I.?...".Cy......wp..\.......(,.......GY....
0740: &.E#Z.....K...`............oc(nr.I..r.........g..W......A.C..mJ.
0780: ..p%.f...m.<....V...u3.(K...K.3P..S.........`.w...._-...6.$..|..
07c0: Zl.H*�>.`..........0...0...U.......0.......0L..U. .E0C0A..+.....
0800: >.20402..+........&https://secure.omniroot.com/repository0....+.
0840: .........0..02..+.....0..&http://ocsp.omniroot.com/baltimoreroot
0880: 09..+.....0..-https://cacert.omniroot.com/baltimoreroot.crt09..+
08c0: .....0..-https://cacert.omniroot.com/baltimoreroot.der0...U.....
0900: ......0...U.#..0.....Y0.GX....T6.{:..M.0B..U...;0907.5.3.1http:/
0940: /cdp1.public-trust.com/CRL/Omniroot2025.crl0...U..........sw....
0980: KM...3..r.0...*.H...............z.r.7.a.s|.j......p.%2..o;.j(=.Q
09c0: ..~...H..w8..V.....ee...t).........e.......(...........z.@....}.
0a00: ...&.n.A;....pH.n�..w%..b.R...9g.".w...l..8.'_..=D...K..V..M{..O
0a40: ..4$r....f*.J...'D.w......,z..."....Y.4..a.. 3..L];.>..-.T...T..
0a80: ..............Y..l_>iU.9.uP.>2...0...0..~........'..0...*.H.....
0ac0: ...0u1.0...U....US1.0...U....GTE Corporation1'0%..U....GTE Cyber
0b00: Trust Solutions, Inc.1#0!..U....GTE CyberTrust Global Root0...12
0b40: 0418163618Z..180813163517Z0Z1.0...U....IE1.0...U....Baltimore1.0
0b80: ...U....CyberTrust1"0 ..U....Baltimore CyberTrust Root0.."0...*.
0bc0: H.............0..........."..=W.&r..y.)........[.+).d..]....m.(.
0c00: .b.b......8.!..A+.R{[email protected].(...%......
0c40: g.?....R./...pp...........3zw.....hDBH......^`........Y..Y..c..c
0c80: ...}]..z.......^.>_...i..96ru.wRM...,.=..#S?.$.!\..)..:..n.:k.tc
0cc0: 3.h.1.x.v....]*..M..'.9........G0..C0...U.......0.......0J..U. .
0d00: C0A0?..U. .0705..+........)http://cybertrust.omniroot.com/reposi
0d40: tory0...U...........0....U.#...0�.y.w0u1.0...U....US1.0...U....G
0d80: TE Corporation1'0%..U....GTE CyberTrust Solutions, Inc.1#0!..U..
0dc0: ..GTE CyberTrust Global Root....0E..U...>0<0:.8.6.4http://www.pu
0e00: blic-trust.com/cgi-bin/CRL/2018/cdp.crl0...*.H.................F
0e40: .........h.h........-.Kn.i*.+.....s....._m.P....@~...s..][..A.7.
0e80: .. ..4j.....7...q..}.D......"v.dQ5.$....2.....*\z'...H:q.C
== Info: SSLv3, TLS alert, Server hello (2):
=> Send SSL data, 2 bytes (0x2)
0000: .0
== Info: SSL certificate problem: unable to get local issuer certificate
== Info: Closing connection 0

Note the line at the end about being "unable to get local issuer certificate". @bartonjs, am I understanding correctly that this means the needed certificate for verification isn't installed on the machine?

[bartonjs]

It's like the OpenSsl library is broken on this distro, and not including the provided intermediate certs in the chainwalk.

openssl s_client -connect http2.akamai.com:443 -showcerts =>

CONNECTED(00000003)
---
Certificate chain
 0 s:/C=US/ST=CA/L=Santa Clara/O=Akamai Technologies Inc./CN=http2.akamai.com
   i:/C=NL/L=Amsterdam/O=Verizon Enterprise Solutions/OU=Cybertrust/CN=Verizon Akamai SureServer CA G14-SHA2
-----BEGIN CERTIFICATE-----
MIIFajCCBFKgAwIBAgIUcBxXXXj4N+Mm5iv24NAQOMQlpEwwDQYJKoZIhvcNAQEL
BQAwgY0xCzAJBgNVBAYTAk5MMRIwEAYDVQQHEwlBbXN0ZXJkYW0xJTAjBgNVBAoT
HFZlcml6b24gRW50ZXJwcmlzZSBTb2x1dGlvbnMxEzARBgNVBAsTCkN5YmVydHJ1
c3QxLjAsBgNVBAMTJVZlcml6b24gQWthbWFpIFN1cmVTZXJ2ZXIgQ0EgRzE0LVNI
QTIwHhcNMTYwMTAzMjM1NzAxWhcNMTcwMTAzMjM1NjUyWjBuMQswCQYDVQQGEwJV
UzELMAkGA1UECBMCQ0ExFDASBgNVBAcTC1NhbnRhIENsYXJhMSEwHwYDVQQKExhB
a2FtYWkgVGVjaG5vbG9naWVzIEluYy4xGTAXBgNVBAMTEGh0dHAyLmFrYW1haS5j
b20wggEiMA0GCSqGSIb3DQEBAQUAA4IBDwAwggEKAoIBAQDgIAEv5K564bQSs0mu
49SAeMy+x48Ned72WekIoeCsV3e3dyHrt5GHY7xSHYroGsO51PWFa7TEw8DQRzTH
/LtEzN0/+z00BZCiUZPNuVNvjbCR4KxxdkJbIIoI6Jfe1KWvb9iUR8N3iOMqQcju
LstchNSWR6+ED3sydgZ5X0tMNB7OCDcGVV8npM2uBDaAD0Nw5ym6jVHttjewBv/g
yv5/Rkd09owKf6wqwY8n8DFE599uKztKBUc6v32rhDtl5DZ8KrkKgDp9xEThL0jT
aUiJcJEFPKbMLIu7aTmUX0sXy1s7C4IPMpUDEY5O7cUI039QjvUbckGvpdUmzWwa
o71XAgMBAAGjggHeMIIB2jAMBgNVHRMBAf8EAjAAMEwGA1UdIARFMEMwQQYJKwYB
BAGxPgEyMDQwMgYIKwYBBQUHAgEWJmh0dHBzOi8vc2VjdXJlLm9tbmlyb290LmNv
bS9yZXBvc2l0b3J5MIGvBggrBgEFBQcBAQSBojCBnzAtBggrBgEFBQcwAYYhaHR0
cDovL3Zhc3NnMTQyLm9jc3Aub21uaXJvb3QuY29tMDYGCCsGAQUFBzAChipodHRw
czovL2NhY2VydC5hLm9tbmlyb290LmNvbS92YXNzZzE0Mi5jcnQwNgYIKwYBBQUH
MAKGKmh0dHBzOi8vY2FjZXJ0LmEub21uaXJvb3QuY29tL3Zhc3NnMTQyLmRlcjAb
BgNVHREEFDASghBodHRwMi5ha2FtYWkuY29tMA4GA1UdDwEB/wQEAwIFoDAdBgNV
HSUEFjAUBggrBgEFBQcDAQYIKwYBBQUHAwIwHwYDVR0jBBgwFoAU+L36r3N3xscb
+UtNEafRM6+vchEwPgYDVR0fBDcwNTAzoDGgL4YtaHR0cDovL3Zhc3NnMTQyLmNy
bC5vbW5pcm9vdC5jb20vdmFzc2cxNDIuY3JsMB0GA1UdDgQWBBS2d/4QBDG6JnKM
06/naFk4CaANXDANBgkqhkiG9w0BAQsFAAOCAQEAN48MlYbhNTqe/SUjSTQ7nNxd
b4Klh2cIq68FKMiKLqvmAElHzN2IuBzo+phSVyqGPCnTdBzeLb9RfY7g9n8OE1Nb
89hF18x3F5dDN/b5LucpRr4I8ZYpX3tD+yJcMDSqaa9Y6ivZACanZaUkA73utdne
U/7lJc74DcYsXG3Lcz3lH0Cb0SR0WShJ96xQ1R/kaChq2sfFapR3Mv/mViPi/y+q
DfhnlYrbb7qHB5GumUWrzFjPPJsfVK1kwMbVQBP9QKRnRyh6y38vuHM8/xxofCN0
NCBj8GE+JFllPEB9s8oXSiuQ5qJo/XVSTfXcRMOf8zrrJaqTzVcbFi0VAi3jrg==
-----END CERTIFICATE-----
 1 s:/C=NL/L=Amsterdam/O=Verizon Enterprise Solutions/OU=Cybertrust/CN=Verizon Akamai SureServer CA G14-SHA2
   i:/C=IE/O=Baltimore/OU=CyberTrust/CN=Baltimore CyberTrust Root
-----BEGIN CERTIFICATE-----
MIIFHzCCBAegAwIBAgIEByekazANBgkqhkiG9w0BAQsFADBaMQswCQYDVQQGEwJJ
RTESMBAGA1UEChMJQmFsdGltb3JlMRMwEQYDVQQLEwpDeWJlclRydXN0MSIwIAYD
VQQDExlCYWx0aW1vcmUgQ3liZXJUcnVzdCBSb290MB4XDTE0MDQwMjE0MzYxMFoX
DTIxMDQwMjE0MzU1MlowgY0xCzAJBgNVBAYTAk5MMRIwEAYDVQQHEwlBbXN0ZXJk
YW0xJTAjBgNVBAoTHFZlcml6b24gRW50ZXJwcmlzZSBTb2x1dGlvbnMxEzARBgNV
BAsTCkN5YmVydHJ1c3QxLjAsBgNVBAMTJVZlcml6b24gQWthbWFpIFN1cmVTZXJ2
ZXIgQ0EgRzE0LVNIQTIwggEiMA0GCSqGSIb3DQEBAQUAA4IBDwAwggEKAoIBAQDd
bp4CaQK1o5kuCGQyalnzxp6mIAfSSNGok8fqR4+DOUDXIF2Nmrqr2HDsnYjRvWL2
2+ydXjUBdgMj5W/Sr0Y1WVpc0agjwevpINRJ1j8A2Kgi3kN5gazppJL1d3AFHly2
oPeQpM2rKCyQwucPw68cR1nVhC7fJgdFI1rG6JDIhUuMFh5g+QET8RQf5ugU7cXS
b2MobnKMSa4IcseTlbQLDK6PmmeE9Vcb24HXF51BEUMZvW1Khe2PcCWrZqv2+m0c
PKvtF71WhOHbdTOyKEuZjvlLgjNQn5JT7fqtD5Wco/LLYPB3HckBi18thr6/Nrgk
lhN8wYZabMFIKn8+k2DFAgMBAAGjggG3MIIBszASBgNVHRMBAf8ECDAGAQH/AgEC
MEwGA1UdIARFMEMwQQYJKwYBBAGxPgEyMDQwMgYIKwYBBQUHAgEWJmh0dHBzOi8v
c2VjdXJlLm9tbmlyb290LmNvbS9yZXBvc2l0b3J5MIG6BggrBgEFBQcBAQSBrTCB
qjAyBggrBgEFBQcwAYYmaHR0cDovL29jc3Aub21uaXJvb3QuY29tL2JhbHRpbW9y
ZXJvb3QwOQYIKwYBBQUHMAKGLWh0dHBzOi8vY2FjZXJ0Lm9tbmlyb290LmNvbS9i
YWx0aW1vcmVyb290LmNydDA5BggrBgEFBQcwAoYtaHR0cHM6Ly9jYWNlcnQub21u
aXJvb3QuY29tL2JhbHRpbW9yZXJvb3QuZGVyMA4GA1UdDwEB/wQEAwIBxjAfBgNV
HSMEGDAWgBTlnVkwgkdYzKz6CFQ2hns6tQRN8DBCBgNVHR8EOzA5MDegNaAzhjFo
dHRwOi8vY2RwMS5wdWJsaWMtdHJ1c3QuY29tL0NSTC9PbW5pcm9vdDIwMjUuY3Js
MB0GA1UdDgQWBBT4vfqvc3fGxxv5S00Rp9Ezr69yETANBgkqhkiG9w0BAQsFAAOC
AQEAgNl67XIFN49hqnN8mmr8/gHiGYFwByUysPBvO8dqKD3kUYfmfoLsrkinsXc4
wtZWr4/yAfxlZRAJ93QptQ6S7pCY0YiiZbfNnA6nhpgovK4Vg7Ya1x3sGdp6jkD5
mRXVfaW6q/0mmG6cQTu2gRjscEjXbn+m4Xcl1t1i6FLzjBY5Z+IiDXcu+xFs5N04
tCdfA6g9ROLyhEuE/Vamnk17ohZPB/U0JHKlovoWZiqkSg7IDSdEnHfUEhCH0gAs
eruOiCKRFb6iWco04BxhlIYgM83nTF07kj7L1i3qVPr7r1T1qMULyouHAOaf5pW/
t8SjWfUWbF8+aVWAOfZ1UBQ+Mg==
-----END CERTIFICATE-----
 2 s:/C=IE/O=Baltimore/OU=CyberTrust/CN=Baltimore CyberTrust Root
   i:/C=US/O=GTE Corporation/OU=GTE CyberTrust Solutions, Inc./CN=GTE CyberTrust Global Root
-----BEGIN CERTIFICATE-----
MIIEFTCCA36gAwIBAgIEByeO7TANBgkqhkiG9w0BAQUFADB1MQswCQYDVQQGEwJV
UzEYMBYGA1UEChMPR1RFIENvcnBvcmF0aW9uMScwJQYDVQQLEx5HVEUgQ3liZXJU
cnVzdCBTb2x1dGlvbnMsIEluYy4xIzAhBgNVBAMTGkdURSBDeWJlclRydXN0IEds
b2JhbCBSb290MB4XDTEyMDQxODE2MzYxOFoXDTE4MDgxMzE2MzUxN1owWjELMAkG
A1UEBhMCSUUxEjAQBgNVBAoTCUJhbHRpbW9yZTETMBEGA1UECxMKQ3liZXJUcnVz
dDEiMCAGA1UEAxMZQmFsdGltb3JlIEN5YmVyVHJ1c3QgUm9vdDCCASIwDQYJKoZI
hvcNAQEBBQADggEPADCCAQoCggEBAKMEuyKrmD1X6CZymrV51Cni4eiVgLGw41uO
KymaZN+hXe2wCQVt2yguzmKiYv60iNoS6zjrIZ3AQSsBUnuId9Mcj8e6uYi1agnn
c+gRQKfRzMpijS3ljwumUNKoUMMo6vWrJYeKmpYcqWe4PwzV9/lSEy/CG9VwcPCP
wBLKBsua4dnKM3p31vjsufFoREJIE9LAwqSuXmD+tqYF/LTdB1kC1FkYmGP1pWPg
kAx9XbIGevOF6uvUA65ehD5f/xXtabz5OTZydc93Uk3zyZAsuT3lySNTPx8kmCFc
B5kpvcY67Oduhjprl3RjM71oGDHweI12v/yejl0qhqdNkNwnGjkCAwEAAaOCAUcw
ggFDMBIGA1UdEwEB/wQIMAYBAf8CAQMwSgYDVR0gBEMwQTA/BgRVHSAAMDcwNQYI
KwYBBQUHAgEWKWh0dHA6Ly9jeWJlcnRydXN0Lm9tbmlyb290LmNvbS9yZXBvc2l0
b3J5MA4GA1UdDwEB/wQEAwIBBjCBiQYDVR0jBIGBMH+heaR3MHUxCzAJBgNVBAYT
AlVTMRgwFgYDVQQKEw9HVEUgQ29ycG9yYXRpb24xJzAlBgNVBAsTHkdURSBDeWJl
clRydXN0IFNvbHV0aW9ucywgSW5jLjEjMCEGA1UEAxMaR1RFIEN5YmVyVHJ1c3Qg
R2xvYmFsIFJvb3SCAgGlMEUGA1UdHwQ+MDwwOqA4oDaGNGh0dHA6Ly93d3cucHVi
bGljLXRydXN0LmNvbS9jZ2ktYmluL0NSTC8yMDE4L2NkcC5jcmwwDQYJKoZIhvcN
AQEFBQADgYEAkx3+i65G7MupD6vl78qyaBZo2I/6E6mvs8st50tujmkqwisQCo32
rnO2ufsU/V9tuFC2xIrWQH7Xw8tz3MldW6+wQbU36+rcIJHENGr0ofOWnTeGl+Fx
pN19+kSElK7XCQQidg9kUTWpJA/5C9sy2sL+wbkqXHonE8qxSDpx0EM=
-----END CERTIFICATE-----
---
Server certificate
subject=/C=US/ST=CA/L=Santa Clara/O=Akamai Technologies Inc./CN=http2.akamai.com
issuer=/C=NL/L=Amsterdam/O=Verizon Enterprise Solutions/OU=Cybertrust/CN=Verizon Akamai SureServer CA G14-SHA2
---
No client certificate CA names sent
---
SSL handshake has read 4414 bytes and written 421 bytes
---
New, TLSv1/SSLv3, Cipher is ECDHE-RSA-AES256-GCM-SHA384
Server public key is 2048 bit
Secure Renegotiation IS supported
Compression: NONE
Expansion: NONE
SSL-Session:
    Protocol  : TLSv1.2
    Cipher    : ECDHE-RSA-AES256-GCM-SHA384
    Session-ID: C...2
    Session-ID-ctx: 
    Master-Key: B...B
    Key-Arg   : None
    PSK identity: None
    PSK identity hint: None
    SRP username: None
    TLS session ticket lifetime hint: 7200 (seconds)
    TLS session ticket:
    0000 - b8 11 77 9e b1 c8 64 9a-79 ef da e9 25 d0 67 c9   ..w...d.y...%.g.
    0010 - 75 01 1d ae c5 1e 5f 2b-14 d6 18 0c 50 60 25 df   u....._+....P`%.
    0020 - 2b e4 07 5e 12 ff 0a bc-ed 3c 8c 9d 90 17 12 30   +..^.....<.....0
    0030 - ad 00 2a 59 cc 22 a7 7c-db ac 86 61 1c 83 35 cf   ..*Y.".|...a..5.
    0040 - c2 f8 b7 76 de 63 25 41-0f fe 6f b1 68 b7 e8 b5   ...v.c%A..o.h...
    0050 - e3 82 af cf f8 b7 22 05-f5 b0 53 42 d7 bd b9 f3   ......"...SB....
    0060 - bb ce 07 34 7c 76 5f 88-55 28 d8 68 94 53 2e cd   ...4|v_.U(.h.S..
    0070 - f1 e5 db a5 5d 6c cd 41-5c a5 74 1c af ee 30 53   ....]l.A\.t...0S
    0080 - 06 42 c4 aa b7 2e ce 0d-84 7b ac 1a d3 b4 bb 98   .B.......{......
    0090 - ea 9f 98 5c 19 6e c8 fd-0a 19 21 d7 9b a5 d6 2a   ...\.n....!....*

    Start Time: 1465488265
    Timeout   : 300 (sec)
    Verify return code: 20 (unable to get local issuer certificate)
---
closed

Note that it has a 0 (the server cert), 1 (the low intermediate), and 2 (the high intermediate). Akamai follows the rules and didn't send 3 (the root).

I saved 0 to akamai.cer, 1 to akamai.1.cer, and 2 to akamai.2.cer.

openssl verify -verbose akamai.cer akamai.1.cer akamai.2.cer =>

akamai.cer: C = US, ST = CA, L = Santa Clara, O = Akamai Technologies Inc., CN = http2.akamai.com
error 20 at 0 depth lookup:unable to get local issuer certificate
akamai.1.cer: OK
akamai.2.cer: OK

Okay, so it doesn't have a problem with the intermediates... is the chain legal?

openssl verify -verbose -CAfile akamai.1.cer akamai.cer akamai.1.cer akamai.2.cer =>

akamai.cer: OK
akamai.1.cer: OK
akamai.2.cer: OK

So.... I don't know why OpenSSL is refusing to honor the input chain presented on the TLS connection... because that's why the chain is given in the first place...

[bartonjs]

Okay, I have an answer for Debian. Ready for crazy? Here we go:

TLS says you're supposed to send your server identity certificate and all intermediates (but not the root). In this case, what we get on the connection is:

 0 s:/C=US/ST=CA/L=Santa Clara/O=Akamai Technologies Inc./CN=http2.akamai.com
   i:/C=NL/L=Amsterdam/O=Verizon Enterprise Solutions/OU=Cybertrust/CN=Verizon Akamai SureServer CA G14-SHA2
 1 s:/C=NL/L=Amsterdam/O=Verizon Enterprise Solutions/OU=Cybertrust/CN=Verizon Akamai SureServer CA G14-SHA2
   i:/C=IE/O=Baltimore/OU=CyberTrust/CN=Baltimore CyberTrust Root
 2 s:/C=IE/O=Baltimore/OU=CyberTrust/CN=Baltimore CyberTrust Root
   i:/C=US/O=GTE Corporation/OU=GTE CyberTrust Solutions, Inc./CN=GTE CyberTrust Global Root

(For the uninitiated: s=subject, i=issuer... notice how the s on one line is equal to the i on the preceding).

libssl takes these certificates and shoves them into the "untrusted' bucket of an X509_STORE_CTX (aka chain builder). "untrusted" doesn't mean "bad", it means "extra context from before I start doing hard work"... the "untrusted" part means "not a source of trust".

libcrypto takes this bucket and starts to build a chain. It builds, well, precisely this chain. It then asks "does the last thing sign itself?", and the answer is "no", so it decides more work is required.

Now that more work is required, it says "okay, hash the name of the issuer of the topmost certificate, and look for a file by that name in a provided store directory" (/etc/ssl/certs is the only one provided). In this case, it happens to care about c692a373.0 (the .0 is in the case of collision).

Sure enough, that file isn't found. Why? Because Debian removed all RSA-1024 certificates from the CA bundle, and GTE CyberTrust Root was an RSA-1024 cert.

What's bad about this is that "Baltimore CyberTrust Root" is... well... also a root. That represents a cross-signed certificate authority. But since OpenSSL already found the Baltimore cert that was provided by Akamai OpenSSL won't look into the store any further.

I must have accidentally ended up on a non-Debian machine when doing my previous testing, because the GTE CyberTrust Root cert was most definitely there (well, the long form was, the hashed name symlink might not have been); and openssl verify on the Baltimore-signed-by-GTE cert succeeded.

SSH'd into a Debian 8(.4?) machine I get:

dotnet-bot@deb84-setup:~$ openssl verify akamai.*
akamai.0.cer: C = US, ST = CA, L = Santa Clara, O = Akamai Technologies Inc., CN = http2.akamai.com
error 20 at 0 depth lookup:unable to get local issuer certificate
akamai.1.cer: OK
akamai.2.cer: C = IE, O = Baltimore, OU = CyberTrust, CN = Baltimore CyberTrust Root
error 20 at 0 depth lookup:unable to get local issuer certificate

akamai.0.cer fails because it's issuer (akamai.1.cer) isn't well known.
akamai.1.cer succeeds because when it wasn't pre-fed information about akamai.2.cer it finds the Baltimore root in /etc/ssl/certs.
akamai.2.cer failing is what causes this problem.

Fixes:

• Akamai could remove the Baltimore-by-GTE cert from their TLS context
• I think OpenSSL made their chain building smarter in 1.0.2, probably to fix problems like this.
• Debian users can manually copy back in the RSA-1024 certs that were overzealously deleted.

Turns out, we're not the first to have noticed this: https://bugs.debian.org/cgi-bin/bugreport.cgi?bug=812708

So, I think we'll want to put in release notes somewhere that (in case they weren't already painfully aware) Debian and TLS chain building aren't really in a happy place right now. The people on that bugreport thread mostly seem to be borrowing Ubuntu's ca-certificates contents to get back online.

[joshfree]

Thank you for the detailed investigation @bartonjs. Marking this as release notes so we record and reference the known Debian issue

[joshfree]

@bartonjs can you also prepare a PR to disable the 1 failing test for debian in order to get green badges in the /release/1.0.0 branch?

[bartonjs]

@joshfree I'm on it.

[bartonjs]

Release note contents:

Debian users may experience unexpected failure when using SSL/TLS

When new Root Certificate Authorities are being created it is not uncommon for the new CA public key to be "cross certified" by an existing Certificate Authority to boostrap the trust relationship into existing environments. For example, the "Baltimore CyberTrust Root" CA was cross-certified by the existing "GTE CyberTrust Global Root" CA. This process is usually considered to make clients more accepting.

Metadata from the server can cause OpenSSL 1.0.1 to consider the cross-certified certificate chain without considering the direct-root chain. Combined with Debian's removal of some older trusted Root Certificate Authorities in the 20141019+deb8u1 version of the ca-certificates package, these certificate chains will be incomplete (and therefore untrusted).

Microsoft has no specific guidance to offer users affected by this configuration state. This is currently tracked as bug 812488 in the Debian bug system (https://bugs.debian.org/cgi-bin/bugreport.cgi?bug=812488).