Skip to content

Troubleshooting Certificate Issues

George Hicken edited this page Dec 6, 2016 · 1 revision

There is basic usage documentation here for certificate configuration, with theoretical and conceptual details via the links on the wikipedia article.

In VIC, certificates are used in three distinct ways

  1. prove to clients accessing the VCH endpointVM that it is the server they expect
  2. authenticate the client attempting to connect to the endpointVM, either for API access or to vic-admin
  3. validate a registry from which the endpointVM has been instructed to pull an image

Most of the diagnostics will involve attempting to determine the validity of the certificates in question. For that, this guide is a good starting point. Almost all references for doing this on the internet will be using the Linux version of openssl client binary.

The certificates are stored in guestinfo keys and are base64 encoded - the server validation section below gives an example of how to extract the server certificate directly from the vmx, but this is applicable to other guestinfo keys as well.

Server validation

The certificate used for host validation is stored in the following guestinfo key:

# As of 0.8 
[root@esx-a:/vmfs/volumes/57624375-a9f1e7c5-6773-000c29162ea1] cat skullcanyon/*.vmx | grep guestinfo | grep HostCertificate
guestinfo.vice./cert/HostCertificate/Key@secret = "<snip>"
guestinfo.vice./cert/HostCertificate/Cert = "<snip>"

The certificates are base64 encoded while in guestinfo so must be unencoded before you can inspect them directly - the following is an example of doing so for the public part of the certificate (that which would be stored in server-cert.pem). The private key is encrypted, as indicated by @secret in the guestinfo key name and must be unencrypted after decoding using the key in guestinfo.ovfEnv:

$ echo LS0tLS1CRUdJTiBDRVJUSUZJQ0FURS0tLS0tCk1JSURBakNDQWVxZ0F3SUJBZ0lRSERORCtqbWNKbUxSZ2VlbnBwSnFtREFOQmdrcWhraUc5dzBCQVFzRkFEQVMKTVJBd0RnWURWUVFLRXdka1pXWmhkV3gwTUI0WERURTJNVEl3TlRFME1qUXpNRm9YRFRFM01USXdOakUwTWpRegpNRm93RWpFUU1BNEdBMVVFQ2hNSFpHVm1ZWFZzZERDQ0FTSXdEUVlKS29aSWh2Y05BUUVCQlFBRGdnRVBBRENDCkFRb0NnZ0VCQUt0WXNuRDNaSlRKam40eDhpb2FPZytVem1ybkpSclFudVZ3d1lSZU9HdzNlZW1RZ09wS081aUwKMmlsSk10cEFCeTVyS24vNWhLYzQ3YVJHdHZuWUZac1BNN3pKY0FsMlU5dEViYnpnbHB1Q0J6dzlBcGZVaFlKawpoM1JEUE5ILzE4QmJnSThLSit6cnE3TWlvNzhEMDZwOW5HL1ZPZUl5ZmpQSnZEUnY5djMrRWJBTDMrenM3dkVXCmhCbGZuRVIwS2I5MDBZS0xQWEljdjExcGRvd0VMSmVHaTA5ZXN4RmZidEM5YmhabVIra2VLTThNR0VlSk1TVTIKUW5ORGJiVGR2WXhTTWR4clFEenBxRU9YTyt2NjYweTJyZEhndk9JcWNEL2d5Q25LakRHRE5uckZWYW9MeGNydgpNTm9IMVNHbE5rQVZpcjZQR1dVR001VDBXYlhSVnQ4Q0F3RUFBYU5VTUZJd0RnWURWUjBQQVFIL0JBUURBZ09vCk1CTUdBMVVkSlFRTU1Bb0dDQ3NHQVFVRkJ3TUJNQXdHQTFVZEV3RUIvd1FDTUFBd0hRWURWUjBPQkJZRUZOY0cKT09zd1pDMlBFVlBkT0JsRTFOSWRmQTJ3TUEwR0NTcUdTSWIzRFFFQkN3VUFBNElCQVFBb0RTNzFMYkdDbGtGaQowbjNLNzJaSkVGWFd6dHZmaVBsYTN3THcwd3d2VmRDemFLVEM2NjI5U1pQVHpjYyt4ZzFOc1hrampVUnJRaVo4Ck85NXlveW5MUndmM1VrOUtUMHcyVWVnd0kydm5WRjRyV0JtMExqd3E1RFVmWVR5bi9wVU4wMXdmNWxKQW1SQ0QKYVNUYzNHU3NacVdQeUd3ZDQ0MG1zNHpERlNNTXllNlREblkrQzMrQjU3TjVIZmdlK0d1ZjNFN3RlcHUzV2FXWgpIa2hxS0VPemdCN29KcmVTbWZDUU95czhNanZEdGtOcDRJeGthcFFMRm1FWWRDUS91Vk1Ua2thU1llV3h3ZHVPCnRueTFIYWVIM1BrcklYdmNqd1lUWjB5RlVRM1c5d3dTb2pWSWFFdmo0cUR2S1pFVkRxZllhK0krOWxNRFpCMDUKK2VNdE5oZE4KLS0tLS1FTkQgQ0VSVElGSUNBVEUtLS0tLQo= \
  | base64 --decode
-----BEGIN CERTIFICATE-----
MIIDAjCCAeqgAwIBAgIQHDND+jmcJmLRgeenppJqmDANBgkqhkiG9w0BAQsFADAS
MRAwDgYDVQQKEwdkZWZhdWx0MB4XDTE2MTIwNTE0MjQzMFoXDTE3MTIwNjE0MjQz
MFowEjEQMA4GA1UEChMHZGVmYXVsdDCCASIwDQYJKoZIhvcNAQEBBQADggEPADCC
AQoCggEBAKtYsnD3ZJTJjn4x8ioaOg+UzmrnJRrQnuVwwYReOGw3eemQgOpKO5iL
2ilJMtpABy5rKn/5hKc47aRGtvnYFZsPM7zJcAl2U9tEbbzglpuCBzw9ApfUhYJk
h3RDPNH/18BbgI8KJ+zrq7Mio78D06p9nG/VOeIyfjPJvDRv9v3+EbAL3+zs7vEW
hBlfnER0Kb900YKLPXIcv11pdowELJeGi09esxFfbtC9bhZmR+keKM8MGEeJMSU2
QnNDbbTdvYxSMdxrQDzpqEOXO+v660y2rdHgvOIqcD/gyCnKjDGDNnrFVaoLxcrv
MNoH1SGlNkAVir6PGWUGM5T0WbXRVt8CAwEAAaNUMFIwDgYDVR0PAQH/BAQDAgOo
MBMGA1UdJQQMMAoGCCsGAQUFBwMBMAwGA1UdEwEB/wQCMAAwHQYDVR0OBBYEFNcG
OOswZC2PEVPdOBlE1NIdfA2wMA0GCSqGSIb3DQEBCwUAA4IBAQAoDS71LbGClkFi
0n3K72ZJEFXWztvfiPla3wLw0wwvVdCzaKTC6629SZPTzcc+xg1NsXkjjURrQiZ8
O95yoynLRwf3Uk9KT0w2UegwI2vnVF4rWBm0Ljwq5DUfYTyn/pUN01wf5lJAmRCD
aSTc3GSsZqWPyGwd440ms4zDFSMMye6TDnY+C3+B57N5Hfge+Guf3E7tepu3WaWZ
HkhqKEOzgB7oJreSmfCQOys8MjvDtkNp4IxkapQLFmEYdCQ/uVMTkkaSYeWxwduO
tny1HaeH3PkrIXvcjwYTZ0yFUQ3W9wwSojVIaEvj4qDvKZEVDqfYa+I+9lMDZB05
+eMtNhdN
-----END CERTIFICATE-----
$

To validate the certificate offered by the endpointVM one can connect via browser to vic-admin on port 2378 and use the browser to inspect the certificate, however it can be hard to get useful detail from the browsers beyond basic checks. Port 2376 is the TLS port for API access, so checking that as well ensures that you are following the same path as the docker client binary.

For a self-signed certificate you will likely receive a warning, ERR_CERT_AUTHORITY_INVALID, but the browser will allow you to continue.

openssl on linux provides much more detail about the certificate than is easily available from most browsers.

--no-tlsverify deployment

This certificate was generated as part of a deployment with --no-tlsverify - this means that the certificate cannot be verified via a standard public CA. In this case the server certificate is self-signed, meaning there is no CA against which it can be validated. This provides TLS encryption, but no protection against man-in-the-middle attacks:

$ openssl s_client -connect 192.168.78.241:2376
CONNECTED(00000003)
depth=0 O = default
verify error:num=20:unable to get local issuer certificate
verify return:1
depth=0 O = default
verify error:num=21:unable to verify the first certificate
verify return:1
---
Certificate chain
 0 s:/O=default
   i:/O=default
---
Server certificate
-----BEGIN CERTIFICATE-----
MIIDAjCCAeqgAwIBAgIQHDND+jmcJmLRgeenppJqmDANBgkqhkiG9w0BAQsFADAS
MRAwDgYDVQQKEwdkZWZhdWx0MB4XDTE2MTIwNTE0MjQzMFoXDTE3MTIwNjE0MjQz
MFowEjEQMA4GA1UEChMHZGVmYXVsdDCCASIwDQYJKoZIhvcNAQEBBQADggEPADCC
AQoCggEBAKtYsnD3ZJTJjn4x8ioaOg+UzmrnJRrQnuVwwYReOGw3eemQgOpKO5iL
2ilJMtpABy5rKn/5hKc47aRGtvnYFZsPM7zJcAl2U9tEbbzglpuCBzw9ApfUhYJk
h3RDPNH/18BbgI8KJ+zrq7Mio78D06p9nG/VOeIyfjPJvDRv9v3+EbAL3+zs7vEW
hBlfnER0Kb900YKLPXIcv11pdowELJeGi09esxFfbtC9bhZmR+keKM8MGEeJMSU2
QnNDbbTdvYxSMdxrQDzpqEOXO+v660y2rdHgvOIqcD/gyCnKjDGDNnrFVaoLxcrv
MNoH1SGlNkAVir6PGWUGM5T0WbXRVt8CAwEAAaNUMFIwDgYDVR0PAQH/BAQDAgOo
MBMGA1UdJQQMMAoGCCsGAQUFBwMBMAwGA1UdEwEB/wQCMAAwHQYDVR0OBBYEFNcG
OOswZC2PEVPdOBlE1NIdfA2wMA0GCSqGSIb3DQEBCwUAA4IBAQAoDS71LbGClkFi
0n3K72ZJEFXWztvfiPla3wLw0wwvVdCzaKTC6629SZPTzcc+xg1NsXkjjURrQiZ8
O95yoynLRwf3Uk9KT0w2UegwI2vnVF4rWBm0Ljwq5DUfYTyn/pUN01wf5lJAmRCD
aSTc3GSsZqWPyGwd440ms4zDFSMMye6TDnY+C3+B57N5Hfge+Guf3E7tepu3WaWZ
HkhqKEOzgB7oJreSmfCQOys8MjvDtkNp4IxkapQLFmEYdCQ/uVMTkkaSYeWxwduO
tny1HaeH3PkrIXvcjwYTZ0yFUQ3W9wwSojVIaEvj4qDvKZEVDqfYa+I+9lMDZB05
+eMtNhdN
-----END CERTIFICATE-----
subject=/O=default
issuer=/O=default
---
No client certificate CA names sent
Peer signing digest: SHA384
Server Temp Key: ECDH, P-256, 256 bits
---
SSL handshake has read 1380 bytes and written 431 bytes
---
New, TLSv1/SSLv3, Cipher is ECDHE-RSA-AES256-GCM-SHA384
Server public key is 2048 bit
Secure Renegotiation IS supported
Compression: NONE
Expansion: NONE
No ALPN negotiated
SSL-Session:
    Protocol  : TLSv1.2
    Cipher    : ECDHE-RSA-AES256-GCM-SHA384
    Session-ID: BCD16EF4DE424992289389D3726132D204D7EB356F0580A0CC39C07C67B916E0
    Session-ID-ctx:
    Master-Key: 82A2F058B4A74BC7EC968833A7C84A20206154170825EF34EB1305215E38AAE85831DEB4F1E6EF40D932C1C81CD7AEC5
    Key-Arg   : None
    PSK identity: None
    PSK identity hint: None
    SRP username: None
    TLS session ticket:
    0000 - eb 4b 5d ea 75 6e 8d d9-0f 9b 4c 09 c8 ca b4 27   .K].un....L....'
    0010 - 64 76 ca bf a0 af a5 aa-ec ca 70 b9 a4 58 e4 cb   dv........p..X..
    0020 - 2d 16 2f 5c 6e 35 73 f5-21 b5 e5 45 18 1b cb 57   -./\n5s.!..E...W
    0030 - ba 7e fc e7 2c c1 e0 07-52 f4 c6 5b 53 f9 30 5e   .~..,...R..[S.0^
    0040 - 3e 3a fc fa 21 46 40 43-de 4d 0f ec 76 f0 09 51   >:[email protected]
    0050 - f6 ae 0d 81 fa 1a f7 30-e8 2f 02 55 fc e2 13 93   .......0./.U....
    0060 - fe e6 b0 3f e8 6e 52 53-15 c8 3f e3 97 83 50 25   ...?.nRS..?...P%
    0070 - 43 ee 67 03 4e ce b5 1f-                          C.g.N...

    Start Time: 1481034333
    Timeout   : 300 (sec)
    Verify return code: 21 (unable to verify the first certificate)
---

Client authentication

When the client is authenticating via a certificate a custom certificate authority is required, one that has signed (whether directly or via Intermediate CA) the offered certificate.

# As of 0.8
[root@esx-a:/vmfs/volumes/57624375-a9f1e7c5-6773-000c29162ea1] cat skullcanyon-tls/*.vmx | grep guestinfo | grep CertificateAuthorities
guestinfo.vice./cert/CertificateAuthorities = "<snip>"

To use openssl to connect to the server in this case requires that we provide appropriate client certificates and CA.

All but the 3rd example where we provide both client certificate and CA we see the following error:

sslv3 alert bad certificate:s3_pkt.c:1472:SSL alert number 42

This will be seen under various conditions and I've yet to learn how to differentiate between the client objecting to the server certificate (e.g. certificate not yet valid), and the server rejecting the client certificate (which is the case below).

without client cert or CA

vagrant@devbox:~/vic$ openssl s_client -connect 192.168.78.127:2376
CONNECTED(00000003)
depth=0 O = SKull Canyon VIC test + O = 192.168.78.127, CN = 192.168.78.127
verify error:num=20:unable to get local issuer certificate
verify return:1
depth=0 O = SKull Canyon VIC test + O = 192.168.78.127, CN = 192.168.78.127
verify error:num=21:unable to verify the first certificate
verify return:1
140181647439512:error:14094412:SSL routines:ssl3_read_bytes:sslv3 alert bad certificate:s3_pkt.c:1472:SSL alert number 42
140181647439512:error:140790E5:SSL routines:ssl23_write:ssl handshake failure:s23_lib.c:177:
---
Certificate chain
 0 s:/O=SKull Canyon VIC test/O=192.168.78.127/CN=192.168.78.127
   i:/O=SKull Canyon VIC test/O=192.168.78.127
---
Server certificate
-----BEGIN CERTIFICATE-----
MIIDmjCCAoKgAwIBAgIRAJB+IPbkuHc58yMC+JRAe5owDQYJKoZIhvcNAQELBQAw
NzE1MBwGA1UEChMVU0t1bGwgQ2FueW9uIFZJQyB0ZXN0MBUGA1UEChMOMTkyLjE2
OC43OC4xMjcwHhcNMTYxMjA1MTQ1MTA2WhcNMTcxMjA2MTQ1MTA2WjBQMTUwHAYD
VQQKExVTS3VsbCBDYW55b24gVklDIHRlc3QwFQYDVQQKEw4xOTIuMTY4Ljc4LjEy
NzEXMBUGA1UEAxMOMTkyLjE2OC43OC4xMjcwggEiMA0GCSqGSIb3DQEBAQUAA4IB
DwAwggEKAoIBAQDMFnxA0IORg7Qvh92YrJlCwUI6y1o79eEPzCWmzyxqoD5gcYJs
8Vmv/0Aqxff8wNidGAQLSRmiYgLIzZHa8s5O0CAyXmVD+Fm1Ud2o2osfXlWNBfxm
YpeTMGYfTzNfFKZEuOuUNuS6Fmu762T1IXvi40SbmItaAdAa863S3Wz5+o8jpCtR
rMoidFhtOq5EhT8f4+xL33uLjNqmuRYQlKIuOU2ai79oNsPa4zb4l7OpnAmq1bvB
oRaPSlvA9abWuoGSwPQ9mWgf+B+8RiIuBTffbGMx7vZ7ui+ateQ4N7wZQYJLsCTX
InC5KppQx/i7U293zYSGndDn4BuP0T6DP0knAgMBAAGjgYcwgYQwDgYDVR0PAQH/
BAQDAgOoMBMGA1UdJQQMMAoGCCsGAQUFBwMBMAwGA1UdEwEB/wQCMAAwHQYDVR0O
BBYEFMvCuEazatA3SPxC/sylEWAFMP6MMB8GA1UdIwQYMBaAFB+WZESv7om0JUhg
n9/qUiRH3dmfMA8GA1UdEQQIMAaHBMCoTn8wDQYJKoZIhvcNAQELBQADggEBAJ8A
0NwsETrruSkNbg+pPTDtYfBZnoT+lsK73CrO5XRXGMF0dRbYZsAlPgy+sqXG64Q5
82CXXc1O3xPSj7AanoD9wrPp9BcTCu7/Vc+rLqwZ6kZD92cCD7/nXk+L11iCGaKg
rZ1uXe2u2ckVQ0J9aISMCsSCDLOz+RVxuWt1PLKxaBl64mIa1J1+WPDIQnOVUM9l
4aV+yAJV5bkj9PKSIJT47bmrAbr84Hqpov+xgfOTzkC2ZHPJoBt9FnAw+RHmYL8d
o72gxd4NFpYZP0+2nMs19ThofuBzZY6w5DTAFSzmdDHQpjIfoF5gG5So1PMhz44C
Y4aN5pcJozEN0gbYZwY=
-----END CERTIFICATE-----
subject=/O=SKull Canyon VIC test/O=192.168.78.127/CN=192.168.78.127
issuer=/O=SKull Canyon VIC test/O=192.168.78.127
---
Acceptable client certificate CA names
/O=SKull Canyon VIC test/O=192.168.78.127
Client Certificate Types: RSA sign, ECDSA sign
Requested Signature Algorithms: RSA+SHA256:ECDSA+SHA256:RSA+SHA384:ECDSA+SHA384:RSA+SHA1:ECDSA+SHA1
Shared Requested Signature Algorithms: RSA+SHA256:ECDSA+SHA256:RSA+SHA384:ECDSA+SHA384:RSA+SHA1:ECDSA+SHA1
Peer signing digest: SHA384
Server Temp Key: ECDH, P-256, 256 bits
---
SSL handshake has read 1440 bytes and written 138 bytes
---
New, TLSv1/SSLv3, Cipher is ECDHE-RSA-AES256-GCM-SHA384
Server public key is 2048 bit
Secure Renegotiation IS supported
Compression: NONE
Expansion: NONE
No ALPN negotiated
SSL-Session:
    Protocol  : TLSv1.2
    Cipher    : ECDHE-RSA-AES256-GCM-SHA384
    Session-ID:
    Session-ID-ctx:
    Master-Key: A9FD19D4C6E0779E8D5A604F4E3BC646DBF748BAA209F7C5A997C7E193860F58DABC1D8C7E1A598BD34D5083D588A014
    Key-Arg   : None
    PSK identity: None
    PSK identity hint: None
    SRP username: None
    Start Time: 1481035945
    Timeout   : 300 (sec)
    Verify return code: 21 (unable to verify the first certificate)
---

with CA but no client cert:

vagrant@devbox:~/vic$ openssl s_client -connect 192.168.78.127:2376 -CAfile bin/skullcanyon-tls/ca.pem
CONNECTED(00000003)
depth=1 O = SKull Canyon VIC test + O = 192.168.78.127
verify return:1
depth=0 O = SKull Canyon VIC test + O = 192.168.78.127, CN = 192.168.78.127
verify return:1
140373344253592:error:14094412:SSL routines:ssl3_read_bytes:sslv3 alert bad certificate:s3_pkt.c:1472:SSL alert number 42
140373344253592:error:140790E5:SSL routines:ssl23_write:ssl handshake failure:s23_lib.c:177:
---
Certificate chain
 0 s:/O=SKull Canyon VIC test/O=192.168.78.127/CN=192.168.78.127
   i:/O=SKull Canyon VIC test/O=192.168.78.127
---
Server certificate
-----BEGIN CERTIFICATE-----
MIIDmjCCAoKgAwIBAgIRAJB+IPbkuHc58yMC+JRAe5owDQYJKoZIhvcNAQELBQAw
NzE1MBwGA1UEChMVU0t1bGwgQ2FueW9uIFZJQyB0ZXN0MBUGA1UEChMOMTkyLjE2
OC43OC4xMjcwHhcNMTYxMjA1MTQ1MTA2WhcNMTcxMjA2MTQ1MTA2WjBQMTUwHAYD
VQQKExVTS3VsbCBDYW55b24gVklDIHRlc3QwFQYDVQQKEw4xOTIuMTY4Ljc4LjEy
NzEXMBUGA1UEAxMOMTkyLjE2OC43OC4xMjcwggEiMA0GCSqGSIb3DQEBAQUAA4IB
DwAwggEKAoIBAQDMFnxA0IORg7Qvh92YrJlCwUI6y1o79eEPzCWmzyxqoD5gcYJs
8Vmv/0Aqxff8wNidGAQLSRmiYgLIzZHa8s5O0CAyXmVD+Fm1Ud2o2osfXlWNBfxm
YpeTMGYfTzNfFKZEuOuUNuS6Fmu762T1IXvi40SbmItaAdAa863S3Wz5+o8jpCtR
rMoidFhtOq5EhT8f4+xL33uLjNqmuRYQlKIuOU2ai79oNsPa4zb4l7OpnAmq1bvB
oRaPSlvA9abWuoGSwPQ9mWgf+B+8RiIuBTffbGMx7vZ7ui+ateQ4N7wZQYJLsCTX
InC5KppQx/i7U293zYSGndDn4BuP0T6DP0knAgMBAAGjgYcwgYQwDgYDVR0PAQH/
BAQDAgOoMBMGA1UdJQQMMAoGCCsGAQUFBwMBMAwGA1UdEwEB/wQCMAAwHQYDVR0O
BBYEFMvCuEazatA3SPxC/sylEWAFMP6MMB8GA1UdIwQYMBaAFB+WZESv7om0JUhg
n9/qUiRH3dmfMA8GA1UdEQQIMAaHBMCoTn8wDQYJKoZIhvcNAQELBQADggEBAJ8A
0NwsETrruSkNbg+pPTDtYfBZnoT+lsK73CrO5XRXGMF0dRbYZsAlPgy+sqXG64Q5
82CXXc1O3xPSj7AanoD9wrPp9BcTCu7/Vc+rLqwZ6kZD92cCD7/nXk+L11iCGaKg
rZ1uXe2u2ckVQ0J9aISMCsSCDLOz+RVxuWt1PLKxaBl64mIa1J1+WPDIQnOVUM9l
4aV+yAJV5bkj9PKSIJT47bmrAbr84Hqpov+xgfOTzkC2ZHPJoBt9FnAw+RHmYL8d
o72gxd4NFpYZP0+2nMs19ThofuBzZY6w5DTAFSzmdDHQpjIfoF5gG5So1PMhz44C
Y4aN5pcJozEN0gbYZwY=
-----END CERTIFICATE-----
subject=/O=SKull Canyon VIC test/O=192.168.78.127/CN=192.168.78.127
issuer=/O=SKull Canyon VIC test/O=192.168.78.127
---
Acceptable client certificate CA names
/O=SKull Canyon VIC test/O=192.168.78.127
Client Certificate Types: RSA sign, ECDSA sign
Requested Signature Algorithms: RSA+SHA256:ECDSA+SHA256:RSA+SHA384:ECDSA+SHA384:RSA+SHA1:ECDSA+SHA1
Shared Requested Signature Algorithms: RSA+SHA256:ECDSA+SHA256:RSA+SHA384:ECDSA+SHA384:RSA+SHA1:ECDSA+SHA1
Peer signing digest: SHA384
Server Temp Key: ECDH, P-256, 256 bits
---
SSL handshake has read 1440 bytes and written 138 bytes
---
New, TLSv1/SSLv3, Cipher is ECDHE-RSA-AES256-GCM-SHA384
Server public key is 2048 bit
Secure Renegotiation IS supported
Compression: NONE
Expansion: NONE
No ALPN negotiated
SSL-Session:
    Protocol  : TLSv1.2
    Cipher    : ECDHE-RSA-AES256-GCM-SHA384
    Session-ID:
    Session-ID-ctx:
    Master-Key: B38B7699B526461A7C7B112271695DA5386AAE29450724B0DA7593602BC2F1907C9180748582FCE4E01429CAAE982861
    Key-Arg   : None
    PSK identity: None
    PSK identity hint: None
    SRP username: None
    Start Time: 1481036001
    Timeout   : 300 (sec)
    Verify return code: 0 (ok)
---

with both CA and client cert:

vagrant@devbox:~/vic$ openssl s_client -connect 192.168.78.127:2376 -CAfile bin/skullcanyon-tls/ca.pem -cert bin/skullcanyon-tls/cert.pem -key bin/skullcanyon-tls/key.pem
CONNECTED(00000003)
depth=1 O = SKull Canyon VIC test + O = 192.168.78.127
verify return:1
depth=0 O = SKull Canyon VIC test + O = 192.168.78.127, CN = 192.168.78.127
verify return:1
---
Certificate chain
 0 s:/O=SKull Canyon VIC test/O=192.168.78.127/CN=192.168.78.127
   i:/O=SKull Canyon VIC test/O=192.168.78.127
---
Server certificate
-----BEGIN CERTIFICATE-----
MIIDmjCCAoKgAwIBAgIRAJB+IPbkuHc58yMC+JRAe5owDQYJKoZIhvcNAQELBQAw
NzE1MBwGA1UEChMVU0t1bGwgQ2FueW9uIFZJQyB0ZXN0MBUGA1UEChMOMTkyLjE2
OC43OC4xMjcwHhcNMTYxMjA1MTQ1MTA2WhcNMTcxMjA2MTQ1MTA2WjBQMTUwHAYD
VQQKExVTS3VsbCBDYW55b24gVklDIHRlc3QwFQYDVQQKEw4xOTIuMTY4Ljc4LjEy
NzEXMBUGA1UEAxMOMTkyLjE2OC43OC4xMjcwggEiMA0GCSqGSIb3DQEBAQUAA4IB
DwAwggEKAoIBAQDMFnxA0IORg7Qvh92YrJlCwUI6y1o79eEPzCWmzyxqoD5gcYJs
8Vmv/0Aqxff8wNidGAQLSRmiYgLIzZHa8s5O0CAyXmVD+Fm1Ud2o2osfXlWNBfxm
YpeTMGYfTzNfFKZEuOuUNuS6Fmu762T1IXvi40SbmItaAdAa863S3Wz5+o8jpCtR
rMoidFhtOq5EhT8f4+xL33uLjNqmuRYQlKIuOU2ai79oNsPa4zb4l7OpnAmq1bvB
oRaPSlvA9abWuoGSwPQ9mWgf+B+8RiIuBTffbGMx7vZ7ui+ateQ4N7wZQYJLsCTX
InC5KppQx/i7U293zYSGndDn4BuP0T6DP0knAgMBAAGjgYcwgYQwDgYDVR0PAQH/
BAQDAgOoMBMGA1UdJQQMMAoGCCsGAQUFBwMBMAwGA1UdEwEB/wQCMAAwHQYDVR0O
BBYEFMvCuEazatA3SPxC/sylEWAFMP6MMB8GA1UdIwQYMBaAFB+WZESv7om0JUhg
n9/qUiRH3dmfMA8GA1UdEQQIMAaHBMCoTn8wDQYJKoZIhvcNAQELBQADggEBAJ8A
0NwsETrruSkNbg+pPTDtYfBZnoT+lsK73CrO5XRXGMF0dRbYZsAlPgy+sqXG64Q5
82CXXc1O3xPSj7AanoD9wrPp9BcTCu7/Vc+rLqwZ6kZD92cCD7/nXk+L11iCGaKg
rZ1uXe2u2ckVQ0J9aISMCsSCDLOz+RVxuWt1PLKxaBl64mIa1J1+WPDIQnOVUM9l
4aV+yAJV5bkj9PKSIJT47bmrAbr84Hqpov+xgfOTzkC2ZHPJoBt9FnAw+RHmYL8d
o72gxd4NFpYZP0+2nMs19ThofuBzZY6w5DTAFSzmdDHQpjIfoF5gG5So1PMhz44C
Y4aN5pcJozEN0gbYZwY=
-----END CERTIFICATE-----
subject=/O=SKull Canyon VIC test/O=192.168.78.127/CN=192.168.78.127
issuer=/O=SKull Canyon VIC test/O=192.168.78.127
---
Acceptable client certificate CA names
/O=SKull Canyon VIC test/O=192.168.78.127
Client Certificate Types: RSA sign, ECDSA sign
Requested Signature Algorithms: RSA+SHA256:ECDSA+SHA256:RSA+SHA384:ECDSA+SHA384:RSA+SHA1:ECDSA+SHA1
Shared Requested Signature Algorithms: RSA+SHA256:ECDSA+SHA256:RSA+SHA384:ECDSA+SHA384:RSA+SHA1:ECDSA+SHA1
Peer signing digest: SHA384
Server Temp Key: ECDH, P-256, 256 bits
---
SSL handshake has read 3307 bytes and written 2398 bytes
---
New, TLSv1/SSLv3, Cipher is ECDHE-RSA-AES256-GCM-SHA384
Server public key is 2048 bit
Secure Renegotiation IS supported
Compression: NONE
Expansion: NONE
No ALPN negotiated
SSL-Session:
    Protocol  : TLSv1.2
    Cipher    : ECDHE-RSA-AES256-GCM-SHA384
    Session-ID: D821B00605B61A6B39719803D060EFAC7B7395F9C797185D542AD2C362C772CD
    Session-ID-ctx:
    Master-Key: 98245E7180AE21B249F21397B139775CAFE241A8D4D64A3A9A9AA6D9826B696720268E2B2418ECDC9058C00B11FA4FED
    Key-Arg   : None
    PSK identity: None
    PSK identity hint: None
    SRP username: None
    TLS session ticket:
    0000 - 74 a5 4b f3 4c 45 5d fc-91 eb 99 60 d1 d0 f0 ae   t.K.LE]....`....
    0010 - 29 84 89 fd b4 57 5f 47-fa 1a 14 42 a4 fe 2a 54   )....W_G...B..*T
    0020 - 42 9e c2 3c 32 93 8e 65-fb d9 57 01 c1 4b 7d 5f   B..<2..e..W..K}_
    0030 - 20 e4 43 9e ae 04 f0 69-13 41 9e ea 00 a3 6e fb    .C....i.A....n.
    0040 - 33 92 ae 09 df 7d 7b a7-34 f4 0c a9 08 22 db 37   3....}{.4....".7
    0050 - 79 db 4a 94 57 27 46 88-3c 72 88 c8 4b 95 c7 73   y.J.W'F.<r..K..s
    0060 - 48 b6 96 3e c2 70 f5 eb-b0 7a a6 38 94 2a 76 6b   H..>.p...z.8.*vk
    0070 - e1 bb a3 c6 0e 05 de 41-63 50 e3 3f 6e 26 99 8c   .......AcP.?n&..
    0080 - c6 73 94 3e f6 c3 86 b3-ef 62 67 06 94 24 5e d6   .s.>.....bg..$^.
    0090 - 33 ad 19 38 66 cd 82 00-52 36 75 a3 96 3a 06 e7   3..8f...R6u..:..
    00a0 - 9a 23 5b 70 b6 7d 6c f1-93 e6 49 f5 3b ca ee 49   .#[p.}l...I.;..I
    00b0 - 8f 26 3d 48 cc a9 92 49-37 23 c1 70 ea d1 66 7c   .&=H...I7#.p..f|
    00c0 - a5 6f 57 20 05 e2 fa bd-f6 a8 82 92 68 da 1b c6   .oW ........h...
    00d0 - f3 2d 21 ae 22 5b 94 c4-67 c1 54 f0 d0 d4 e9 c8   .-!."[..g.T.....
    00e0 - 04 93 c5 3e bd b5 50 c6-aa a1 5b 6a 8c 56 8b b6   ...>..P...[j.V..
    00f0 - a3 af a2 42 1a c8 07 d1-77 51 17 c1 ef 87 63 0c   ...B....wQ....c.
    0100 - 0b b0 b3 1a 68 47 6b 0b-fd 91 0e 7d 31 19 89 0f   ....hGk....}1...
    0110 - c0 d4 65 2b aa 74 92 27-72 c8 09 70 70 18 49 3e   ..e+.t.'r..pp.I>
    0120 - ce 29 92 36 ab 69 ca a9-35 b2 47 73 93 7f b8 f9   .).6.i..5.Gs....
    0130 - 46 44 10 f0 f8 3b 7f d3-c8 11 89 8c 7e 3c 54 7f   FD...;......~<T.
    0140 - f7 2d a7 32 cb c0 f3 05-3c 32 76 dd 81 d5 c9 83   .-.2....<2v.....
    0150 - 02 0e 5b 0d 63 b5 0b 83-d7 b2 e2 11 05 a8 a8 fc   ..[.c...........
    0160 - db ae dd 43 04 e4 58 98-24 ce 9b 29 0f 08 02 4f   ...C..X.$..)...O
    0170 - 67 e3 a3 45 0e 32 ab ab-ad bc b6 72 70 89 32 00   g..E.2.....rp.2.
    0180 - 9a eb 5e f2 8e 6d 34 a8-d2 b4 61 8d 2a 01 35 49   ..^..m4...a.*.5I
    0190 - 3f 40 f9 3e 95 33 9d 94-b2 da bf 87 9a 86 03 57   ?@.>.3.........W
    01a0 - a6 b5 e1 8a a1 b3 b6 47-d0 6d e5 3c ff 21 f0 fc   .......G.m.<.!..
    01b0 - 34 a9 74 4a 0a d4 06 72-47 34 3d 8d fb 24 0d 24   4.tJ...rG4=..$.$
    01c0 - 24 18 30 84 06 66 aa 33-ea 98 c9 f9 20 3d 54 57   $.0..f.3.... =TW
    01d0 - b9 08 4f f6 0e 88 ad 1c-ab 7f 55 1a 93 ae 64 c3   ..O.......U...d.
    01e0 - 57 5e 3c 45 fe d0 3d 75-95 3b 4a df 6d 81 65 87   W^<E..=u.;J.m.e.
    01f0 - e5 50 7d 9c d7 ea 9e 62-39 07 f1 d8 9c 32 54 45   .P}....b9....2TE
    0200 - e9 dd 5c 62 7d 7c 17 59-39 f2 f3 d7 c4 fa 8c ab   ..\b}|.Y9.......
    0210 - 3f 18 56 5d 6e 2a 49 d4-20 fc cc 42 f4 89 dc ac   ?.V]n*I. ..B....
    0220 - 4c 8a f5 1f eb 83 ac 9a-31 e1 e8 56 99 d0 83 60   L.......1..V...`
    0230 - f0 b0 d4 9b 25 13 f4 61-f9 06 a2 10 94 cb 7e 4d   ....%..a......~M
    0240 - 73 2a ba 4f 8e 2e fb 35-af 31 76 31 56 31 cf 28   s*.O...5.1v1V1.(
    0250 - 42 c2 49 36 a6 ec 8b 84-82 66 8c 47 c6 80 75 4f   B.I6.....f.G..uO
    0260 - 8d 7f 31 1f 9a c4 97 f3-4b 1a a1 56 e8 53 0a 9b   ..1.....K..V.S..
    0270 - 64 0d e7 88 0b c0 e6 2c-c6 a3 1b c5 2f 51 18 40   d......,..../Q.@
    0280 - 8b c2 6d 9b 24 3f 87 44-db 2a 3d 8e 7b 5f 3b 30   ..m.$?.D.*=.{_;0
    0290 - 0b c6 69 e0 08 ba f3 ff-6b b5 90 5a bc 83 0c ff   ..i.....k..Z....
    02a0 - 27 4d f0 cb da 5c 12 dd-30 29 ad b3 84 e9 40 25   'M...\..0)....@%
    02b0 - 6d 1d 02 83 02 4f e5 8d-dd 70 47 8c 8a e0 ec 1e   m....O...pG.....
    02c0 - 0e 11 ae 06 b8 7c e0 3c-53 77 66 88 23 22 18 bf   .....|.<Swf.#"..
    02d0 - 7b 1d c4 e3 75 34 eb 38-a3 65 7c d7 7d b5 38 7a   {...u4.8.e|.}.8z
    02e0 - 63 11 4c 55 ba 6f 7e 1c-a8 bc 0f 87 c0 ee e6 68   c.LU.o~........h
    02f0 - 08 27 22 ee 77 ff b2 f0-6e 64 ee f3 68 c1 f2 f1   .'".w...nd..h...
    0300 - 61 0a 01 71 34 30 d2 a2-ab 20 12 3c 91 86 08 f9   a..q40... .<....
    0310 - 6f 86 b7 b1 ba 5a f9 55-61 f7 b2 a8 21 d2 b5 1a   o....Z.Ua...!...
    0320 - da f2 ba 4e f6 b4 72 b0-ef 3c 3e 70 ea 9b 1a 07   ...N..r..<>p....
    0330 - 2d f2 f7 0f 99 13 05 01-e7 d8 ed 1c 63 a0 75 14   -...........c.u.
    0340 - 2b 80 a5 a0 7c 13 ec 91-31 ac 01 c4 ac df c9 74   +...|...1......t
    0350 - 43 c6 5a 80 25 8f 2b 20-e5 cc 7e 73 b9 2b 2a 17   C.Z.%.+ ..~s.+*.
    0360 - bb 4c 36 de a1 c5 9b e2-5a b7 77 30 32 93 b4 45   .L6.....Z.w02..E
    0370 - 8a 69 ad 35 9e 21 d0 e8-87 a2 65 72 e3 56 28 b5   .i.5.!....er.V(.
    0380 - 43 1a 55 de 8b 24 87 6f-3f 65 cb fd c5 78 e0 47   C.U..$.o?e...x.G
    0390 - 7b 53 4b 3b 02 3b 52 cc-cd a7 97 91 fd 63 f0 9e   {SK;.;R......c..
    03a0 - b6 d2 5c 53 1e 76 60 e3-34 4c bd 65 43 3e 6d d5   ..\S.v`.4L.eC>m.
    03b0 - b9 e9 af de c3 57 14 a4-c2 43 f2 91 0e 9c 02 36   .....W...C.....6
    03c0 - 0e 32 98 77 36 e8 2f b6-d6 59 c1 79 e5 6b 6b 18   .2.w6./..Y.y.kk.
    03d0 - 7c 52 de 25 b0 53 90 4c-df 9e 6a 8f a9 61 20 c8   |R.%.S.L..j..a .
    03e0 - b7 b0 09 b3 b4 80 7b 13-0b 2a 4f 1a 55 3c 85 e8   ......{..*O.U<..
    03f0 - be 8a 53 4c 68 c9 63 20-4e 23 dc d4 c1 54 65 96   ..SLh.c N#...Te.
    0400 - 24 df 0c 2e 86 03 06 d6-24 15 c5 18 db b8 b2 d1   $.......$.......
    0410 - 94 35 af d3 09 0e 19 aa-57 78 fc 96 db de 86 36   .5......Wx.....6
    0420 - 10 a6 10 97 6b e2 6e 2b-30 53 41 c1 6a f6 02 8c   ....k.n+0SA.j...
    0430 - c7 6f 86 2f 16 ea af 44-08 db df ed 8a c3 b2 56   .o./...D.......V
    0440 - a0 5a 95 0a 5c 0d 58 86-45 07 97 1e fc e0 ff 30   .Z..\.X.E......0
    0450 - dc de 06 b8 3b 27 76 3e-f9 61 4a a9 c3 a6 e0 2b   ....;'v>.aJ....+
    0460 - c6 1e b3 9d 08 c6 ba 7c-87 2b 6f a5 0a 72 08 90   .......|.+o..r..
    0470 - 5d 81 90 a6 3b 18 f2 e6-3f aa 85 87 4e 31 e8 cb   ]...;...?...N1..
    0480 - 59 28 98 09 44 7e ab db-68 13 72 5f d1 83 86 98   Y(..D~..h.r_....
    0490 - 76 ce 85 e9 d1 4c fa 68-3f cf a9 a6 39 62 bb 63   v....L.h?...9b.c
    04a0 - a0 f0 72 d2 c5 3d 12 38-39 dc 23 5e c1 1e 45 84   ..r..=.89.#^..E.
    04b0 - eb 49 ac 65 4a b3 ff 9c-24 4e ad a1 c5 ee 05 f9   .I.eJ...$N......
    04c0 - ff 97 29 67 44 99 0b cf-72 8d 91 76 e5 00 3e e3   ..)gD...r..v..>.
    04d0 - 9c 6d 7e 87 7f 96 33 70-52 9c ec 7a 67 7a cb be   .m~...3pR..zgz..
    04e0 - 9d d7 67 4a 33 8b 7e 5e-1b d6 5c 90 f3 65 e2 33   ..gJ3.~^..\..e.3
    04f0 - a3 59 3f 74 15 f2 49 c1-56 d6 a5 67 0a de 9f 6e   .Y?t..I.V..g...n
    0500 - aa 13 54 ff ad 76 2f 6f-23 6b c9 31 1a bb ee f7   ..T..v/o#k.1....
    0510 - e8 34 ab ae 0f 90 20 42-f7 30 88 e0 1c 6a 09 1c   .4.... B.0...j..
    0520 - 55 dd e4 8b 44 97 85 54-5e d0 1c 1c f6 a1 82 a6   U...D..T^.......
    0530 - b7 7c be 61 89 86 4d f3-3f bb 34 5a 16 14 62 90   .|.a..M.?.4Z..b.
    0540 - 65 aa 94 b1 1a a5 60 42-57 a2 3d 75 98 85 aa 12   e.....`BW.=u....
    0550 - bf f2 4e 23 d9 e8 46 2a-b5 26 07 01 e7 8f e7 dd   ..N#..F*.&......
    0560 - 98 5d b0 ea ae 47 58 ac-f9 82 95 54 36 ef 6c af   .]...GX....T6.l.
    0570 - 66 37 c3 70 aa d1 6d b0-51 f0 d1 e0 d4 7c 2a 84   f7.p..m.Q....|*.
    0580 - 18 9f 0b b2 c7 50 3e f6-cb bc 38 92 aa eb 89 9b   .....P>...8.....
    0590 - 52 18 62 13 1a ea ad f6-8a 20 c4 b5 b0 39 59 10   R.b...... ...9Y.
    05a0 - 91 1d 25 dc 81 2d eb 67-1e ab 4e 24 2e 2b 9f ff   ..%..-.g..N$.+..
    05b0 - a7 a8 b5 d5 24 eb e1 7d-d0 ea be 9f b1 96 bb c7   ....$..}........
    05c0 - e6 42 2a 9c eb ab e8 36-59 18 fa 81 de 91 36 8a   .B*....6Y.....6.
    05d0 - 6f 10 48 14 a6 5e b8 b8-38 03 09 e6 23 84 74 0b   o.H..^..8...#.t.
    05e0 - e4 df 76 15 e9 f0 6e 2b-95 6a aa 09 ab 65 48 7b   ..v...n+.j...eH{
    05f0 - 83 9f b6 c6 9f e0 8b 4f-7d 69 56 b8 37 08 99 69   .......O}iV.7..i
    0600 - 0a 7d 1d 09 c0 73 b6 9c-2e 33 ef c8 55 39 a1 52   .}...s...3..U9.R
    0610 - c5 b7 85 8d 49 f7 c0 00-81 a7 d1 33 65 93 30 d1   ....I......3e.0.
    0620 - d0 46 9e 82 19 54 87 d2-97 4b 37 3c c3 03 ec bb   .F...T...K7<....
    0630 - 51 53 e6 71 a9 ef 50 7a-42 30 f3 03 be c3 1b 7a   QS.q..PzB0.....z
    0640 - f6 fb ed 73 31 28 37 22-c2 14 02 06 87 c7 68 27   ...s1(7"......h'
    0650 - d1 e8 e2 db ce 15 db 9a-05 10 b2 f8 d9 0f 3c 3e   ..............<>
    0660 - 06 fd 0c 7f 31 14 92 05-bd 0c cd 20 64 1c d2 31   ....1...... d..1
    0670 - 0c d0 15 20 a0 c1 fe 08-22 a7 a5 0c 76 f8 dc 3a   ... ...."...v..:
    0680 - e6 73 7b c6 9c 7f 9d f9-d7 df 93 02 a9 7a dd af   .s{..........z..
    0690 - 1d 29 db e3 73 55 7f 10-06 c3 a0 82 9b db 77 84   .)..sU........w.
    06a0 - fc 76 ef 7f 75 4c c3 a2-85 7b 67 0e e1 8d 37 70   .v..uL...{g...7p
    06b0 - 9c 69 27 9a de f9 df b9-64 c2 6c 0d 92 07 71 26   .i'.....d.l...q&
    06c0 - 93 9d 4a 5c 8d 97 3a ad-2b 68 ca f6 5c c7 25 3e   ..J\..:.+h..\.%>
    06d0 - 92 74 6c 3a 41 9a 2b bb-6f cb 1f 9c 96 b9 bc 27   .tl:A.+.o......'
    06e0 - 50 4b 23 70 d4 d4 fb 2f-ce d3 b9 c5 c1 c1 de ad   PK#p.../........
    06f0 - 2e 5a a0 23 56 63 23 f2-b6 23 14 4f 42 5f 19 2a   .Z.#Vc#..#.OB_.*
    0700 - b7 48 ce f2 43 45 64 e3-71 f1 8b 36 ba ad d7 fb   .H..CEd.q..6....

    Start Time: 1481036049
    Timeout   : 300 (sec)
    Verify return code: 0 (ok)
---

Registry validation

Validation of registries is done using the standard set of root certificates provided with PhotonOS, with the addition of any CAs provided via the --registry-ca option. Of particular note is that the CA provided for client authentication is not used for validating registries - if that is desired then the CA should be specified again via the --registry-ca option.

# As of 0.8