After upgrading requests-credssp to 2.0.0 ansible cannot connect to Windows 2012R2

[marcinbarczynski]

Error from ansible-playbook:

fatal: [win12]: UNREACHABLE! => {"changed": false, "msg": "credssp: Server did not response with a CredSSP token after step TLS Handshake - actual 'Negotiate, Basic realm=\"WSMAN\", CredSSP'", "unreachable": true}

Here are the versions of packages:

$ pip freeze
ansible==2.9.27
certifi==2021.10.8
cffi==1.15.0
charset-normalizer==2.0.12
cryptography==36.0.2
idna==3.3
Jinja2==3.1.1
MarkupSafe==2.1.1
ntlm-auth==1.5.0
pyasn1==0.4.8
pycparser==2.21
pyOpenSSL==22.0.0
pyspnego==0.5.1
pywinrm==0.4.2
PyYAML==6.0
requests==2.27.1
requests-credssp==2.0.0
requests-ntlm==1.1.0
six==1.16.0
urllib3==1.26.9
xmltodict==0.12.0

Windows 10 works fine.

Switching to requests-credssp==1.3.1 fixes the problem.
I have full access to the machines so if you need more info or logs, I'll be happy to provide them for you.

[jborean93]

Thanks for the bug report, the problem is interesting and I wonder if the problem is due to a failure to negotiate a common cipher suite with older Windows OSs. The older requests-credssp used pyOpenSSL which uses whatever OpenSSL that cryptography was built with. The latest version of this library uses pyspnego which utilises the ssl module builtin to Python which could be linked to a different OpenSSL version and potentially constrained by OS settings.

From my limited understanding of how it all work you should be able to use the following to determine what cipher suites each method can offer

import ssl

import OpenSSL.SSL
import spnego.tls

ctx = spnego.tls.default_tls_context(usage="initiate").context
print("Builtin ssl info")
print(ssl.OPENSSL_VERSION)
print([c["name"] for c in ctx.get_ciphers()])

print("pyOpenSSL info")
pyopenssl_ctx = OpenSSL.SSL.Context(OpenSSL.SSL.TLSv1_2_METHOD)
print(OpenSSL.SSL.Connection(pyopenssl_ctx, None).get_cipher_list())

You can also run python -m OpenSSL.debug to get a decent snapshot of both the builtin OpenSSL version and what cryptography/pyOpenSSL is linked to.

For example on my Fedora host this is what I have

Builtin ssl info
OpenSSL 1.1.1n  FIPS 15 Mar 2022
['TLS_AES_256_GCM_SHA384', 'TLS_CHACHA20_POLY1305_SHA256', 'TLS_AES_128_GCM_SHA256', 'TLS_AES_128_CCM_SHA256', 'ECDHE-ECDSA-AES256-GCM-SHA384', 'ECDHE-RSA-AES256-GCM-SHA384', 'ECDHE-ECDSA-AES128-GCM-SHA256', 'ECDHE-RSA-AES128-GCM-SHA256', 'ECDHE-ECDSA-CHACHA20-POLY1305', 'ECDHE-RSA-CHACHA20-POLY1305', 'ECDHE-ECDSA-AES256-SHA384', 'ECDHE-RSA-AES256-SHA384', 'ECDHE-ECDSA-AES128-SHA256', 'ECDHE-RSA-AES128-SHA256', 'DHE-RSA-AES256-GCM-SHA384', 'DHE-RSA-AES128-GCM-SHA256', 'DHE-RSA-AES256-SHA256', 'DHE-RSA-AES128-SHA256']
pyOpenSSL info
['TLS_AES_256_GCM_SHA384', 'TLS_CHACHA20_POLY1305_SHA256', 'TLS_AES_128_GCM_SHA256', 'ECDHE-ECDSA-AES256-GCM-SHA384', 'ECDHE-RSA-AES256-GCM-SHA384', 'DHE-RSA-AES256-GCM-SHA384', 'ECDHE-ECDSA-CHACHA20-POLY1305', 'ECDHE-RSA-CHACHA20-POLY1305', 'DHE-RSA-CHACHA20-POLY1305', 'ECDHE-ECDSA-AES128-GCM-SHA256', 'ECDHE-RSA-AES128-GCM-SHA256', 'DHE-RSA-AES128-GCM-SHA256', 'ECDHE-ECDSA-AES256-SHA384', 'ECDHE-RSA-AES256-SHA384', 'DHE-RSA-AES256-SHA256', 'ECDHE-ECDSA-AES128-SHA256', 'ECDHE-RSA-AES128-SHA256', 'DHE-RSA-AES128-SHA256', 'ECDHE-ECDSA-AES256-SHA', 'ECDHE-RSA-AES256-SHA', 'DHE-RSA-AES256-SHA', 'ECDHE-ECDSA-AES128-SHA', 'ECDHE-RSA-AES128-SHA', 'DHE-RSA-AES128-SHA', 'RSA-PSK-AES256-GCM-SHA384', 'DHE-PSK-AES256-GCM-SHA384', 'RSA-PSK-CHACHA20-POLY1305', 'DHE-PSK-CHACHA20-POLY1305', 'ECDHE-PSK-CHACHA20-POLY1305', 'AES256-GCM-SHA384', 'PSK-AES256-GCM-SHA384', 'PSK-CHACHA20-POLY1305', 'RSA-PSK-AES128-GCM-SHA256', 'DHE-PSK-AES128-GCM-SHA256', 'AES128-GCM-SHA256', 'PSK-AES128-GCM-SHA256', 'AES256-SHA256', 'AES128-SHA256', 'ECDHE-PSK-AES256-CBC-SHA384', 'ECDHE-PSK-AES256-CBC-SHA', 'SRP-RSA-AES-256-CBC-SHA', 'SRP-AES-256-CBC-SHA', 'RSA-PSK-AES256-CBC-SHA384', 'DHE-PSK-AES256-CBC-SHA384', 'RSA-PSK-AES256-CBC-SHA', 'DHE-PSK-AES256-CBC-SHA', 'AES256-SHA', 'PSK-AES256-CBC-SHA384', 'PSK-AES256-CBC-SHA', 'ECDHE-PSK-AES128-CBC-SHA256', 'ECDHE-PSK-AES128-CBC-SHA', 'SRP-RSA-AES-128-CBC-SHA', 'SRP-AES-128-CBC-SHA', 'RSA-PSK-AES128-CBC-SHA256', 'DHE-PSK-AES128-CBC-SHA256', 'RSA-PSK-AES128-CBC-SHA', 'DHE-PSK-AES128-CBC-SHA', 'AES128-SHA', 'PSK-AES128-CBC-SHA256', 'PSK-AES128-CBC-SHA']

pyOpenSSL: 21.0.0
cryptography: 36.0.2
cffi: 1.15.0
cryptography's compiled against OpenSSL: OpenSSL 1.1.1n  15 Mar 2022
cryptography's linked OpenSSL: OpenSSL 1.1.1n  15 Mar 2022
Python's OpenSSL: OpenSSL 1.1.1n  FIPS 15 Mar 2022
Python executable: /home/jborean/.pyenv/versions/psrp-310/bin/python
Python version: 3.10.2 (main, Jan 18 2022, 12:56:09) [GCC 11.2.1 20211203 (Red Hat 11.2.1-7)]
Platform: linux
sys.path: ['/home/jborean/dev/pypsrp', '/home/jborean/.pyenv/versions/3.10.2/lib/python310.zip', '/home/jborean/.pyenv/versions/3.10.2/lib/python3.10', '/home/jborean/.pyenv/versions/3.10.2/lib/python3.10/lib-dynload', '/home/jborean/.pyenv/versions/psrp-310/lib/python3.10/site-packages', '/home/jborean/dev/pypsrp/src']

You can see that pyOpenSSL offers more ciphers than what the builtin module which is most likely due to my OS settings limited what I can use. You can see what cipher suites are available for Server 2012 R2 on this page https://docs.microsoft.com/en-us/windows/win32/secauthn/tls-cipher-suites-in-windows-8-1 but you will have to use https://testssl.sh/openssl-iana.mapping.html to map the OpenSSL format to the RFC format used on the Windows docs. Try and see if there is a common cipher suite that's available in the builtin ssl list and what your Windows host offers.

Finally to test it out you can use a specific cipher suite the following command to verify whether a cipher suite is supported on the Windows host by doing

echo "n" | openssl s_client -connect hostname:3389 -tls1_2 -cipher ECDHE-ECDSA-AES256-GCM-SHA384

[marcinbarczynski]

Thanks for the answer. I ran the script and it turned out that in both methods have ciphers supported by Windows 2012R2:

• spnego

{'TLS_DHE_RSA_WITH_AES_256_GCM_SHA384', 'TLS_DHE_RSA_WITH_AES_128_GCM_SHA256'}

• pyOpenSSL

{'TLS_RSA_WITH_AES_128_CBC_SHA', 'TLS_RSA_WITH_AES_128_CBC_SHA256', 'TLS_RSA_WITH_AES_256_CBC_SHA', 'TLS_DHE_RSA_WITH_AES_256_GCM_SHA384', 'TLS_DHE_RSA_WITH_AES_128_CBC_SHA', 'TLS_DHE_RSA_WITH_AES_128_GCM_SHA256', 'TLS_RSA_WITH_AES_256_GCM_SHA384', 'TLS_DHE_RSA_WITH_AES_256_CBC_SHA', 'TLS_RSA_WITH_AES_128_GCM_SHA256', 'TLS_RSA_WITH_AES_256_CBC_SHA256'}

TLS_DHE_RSA_WITH_AES_256_GCM_SHA384 maps to DHE-RSA-AES256-GCM-SHA384, so I ran:

$ echo "n" | openssl s_client -connect hostname:3389 -tls1_2 -cipher DHE-RSA-AES256-GCM-SHA384
CONNECTED(00000003)
Can't use SSL_get_servername
depth=0 CN = hostname
verify error:num=20:unable to get local issuer certificate
verify return:1
depth=0 CN = hostname
verify error:num=21:unable to verify the first certificate
verify return:1
139731019007296:error:141A318A:SSL routines:tls_process_ske_dhe:dh key too small:../ssl/statem/statem_clnt.c:2149:
---
Certificate chain
 0 s:CN = hostname
   i:CN = hostname
---
Server certificate
-----BEGIN CERTIFICATE-----
...
-----END CERTIFICATE-----
subject=CN = hostname

issuer=CN = hostname

---
No client certificate CA names sent
---
SSL handshake has read 1494 bytes and written 119 bytes
Verification error: unable to verify the first certificate
---
New, (NONE), Cipher is (NONE)
Server public key is 2048 bit
Secure Renegotiation IS supported
Compression: NONE
Expansion: NONE
No ALPN negotiated
SSL-Session:
    Protocol  : TLSv1.2
    Cipher    : 0000
    Session-ID: 960A0000E2DA049CFFA704EE9CDA04D345099811181FCF2F0C0B87F30D101948
    Session-ID-ctx: 
    Master-Key: 
    PSK identity: None
    PSK identity hint: None
    SRP username: None
    Start Time: 1649428527
    Timeout   : 7200 (sec)
    Verify return code: 21 (unable to verify the first certificate)
    Extended master secret: yes
---

and the command exited with 1.

I tried ciphers from the second list and it turned out that for TLS_RSA_WITH_AES_256_CBC_SHA256 (AES256-SHA256) the command exits with 0:

$ echo "n" | openssl s_client -connect hostname:3389 -tls1_2 -cipher AES256-SHA256
CONNECTED(00000003)
Can't use SSL_get_servername
depth=0 CN = hostname
verify error:num=20:unable to get local issuer certificate
verify return:1
depth=0 CN = hostname
verify error:num=21:unable to verify the first certificate
verify return:1
---
Certificate chain
 0 s:CN = hostname
   i:CN = hostname
---
Server certificate
-----BEGIN CERTIFICATE-----
...
-----END CERTIFICATE-----
subject=CN = hostname

issuer=CN = hostname

---
No client certificate CA names sent
---
SSL handshake has read 931 bytes and written 470 bytes
Verification error: unable to verify the first certificate
---
New, TLSv1.2, Cipher is AES256-SHA256
Server public key is 2048 bit
Secure Renegotiation IS supported
Compression: NONE
Expansion: NONE
No ALPN negotiated
SSL-Session:
    Protocol  : TLSv1.2
    Cipher    : AES256-SHA256
    Session-ID: BE4B0000FB1E27CF93EEE6486A37CD3391205006439D33EB5743E73CD90F20C8
    Session-ID-ctx: 
    Master-Key: ...
    PSK identity: None
    PSK identity hint: None
    SRP username: None
    Start Time: 1649429195
    Timeout   : 7200 (sec)
    Verify return code: 21 (unable to verify the first certificate)
    Extended master secret: yes
---
DONE

I am a bit surprised because in both cases I get Verify return code: 21 (unable to verify the first certificate).
Do you have any suggestions what to do next?

[jborean93]

The error 1 with TLS_DHE_RSA_WITH_AES_256_GCM_SHA384 seems to indicate the server does not offer a certificate strong enough that the client can accept. The 2nd option you tried TLS_RSA_WITH_AES_256_CBC_SHA256 does work but based on your output the builtin ssl library does not offer this as a choice for the client. You can see that while the cert was exchanged even with DHE, it failed to negotiate the cipher (value is 0000) (more on this further below).

I am a bit surprised because in both cases I get Verify return code: 21 (unable to verify the first certificate).

This is to be expected for the RDP certificate. It is typically an ephemeral cert generated by Windows that is self signed. Unless you've manually set an explicit certificate or use AD CS to configure a cert for RDP then it will use this self signed certificate. The reason why I asked you to test with RDP (port 3389) is that CredSSP will use a similar ephemeral certificate and be subject to the same limitations as RDP. You can control what certificate CredSSP over RDP uses by setting the thumbprint with

Set-Item -Path WSMan:\localhost\Service\CertificateThumbprint -Value $thumbprint

Do you have any suggestions what to do next?

From what I can see is that the only common cipher with the builtin ssl module and the server are DHE based ciphers and the server does not offer a strong enough certificate the client will accept. The pyOpenSSL library also offers some RSA without DH which is acceptible for use based on your system policies. I have no idea why the building ssl library does not offer these same ciphers but I can have a guess is that it's not allowed due to the lack of forward secrecy with those ciphers.

This leads into the next point which is system wide configuration of OpenSSL. In my /etc/ssl/openssl.cnf I have a file with the following entry

[ crypto_policy ]

.include = /etc/crypto-policies/back-ends/opensslcnf.config

This file contains the following:

CipherString = @SECLEVEL=2:kEECDH:kRSA:kEDH:kPSK:kDHEPSK:kECDHEPSK:kRSAPSK:-aDSS:-3DES:!DES:!RC4:!RC2:!IDEA:-SEED:!eNULL:!aNULL:!MD5:-SHA384:-CAMELLIA:-ARIA:-AESCCM8
Ciphersuites = TLS_AES_256_GCM_SHA384:TLS_CHACHA20_POLY1305_SHA256:TLS_AES_128_GCM_SHA256:TLS_AES_128_CCM_SHA256
TLS.MinProtocol = TLSv1.2
TLS.MaxProtocol = TLSv1.3
DTLS.MinProtocol = DTLSv1.2
DTLS.MaxProtocol = DTLSv1.2
SignatureAlgorithms = ECDSA+SHA256:ECDSA+SHA384:ECDSA+SHA512:ed25519:ed448:rsa_pss_pss_sha256:rsa_pss_pss_sha384:rsa_pss_pss_sha512:rsa_pss_rsae_sha256:rsa_pss_rsae_sha384:rsa_pss_rsae_sha512:RSA+SHA256:RSA+SHA384:RSA+SHA512:ECDSA+SHA224:RSA+SHA224

The key part here is that @SECLEVEL=2 is set which based on the documentation here is

Security level set to 112 bits of security. As a result RSA, DSA and DH keys shorter than 2048 bits and ECC keys shorter than 224 bits are prohibited. In addition to the level 1 exclusions any cipher suite using RC4 is also prohibited. SSL version 3 is also not allowed. Compression is disabled.

When I manually set this to =1 then openssl s_client is able to successfully negotiate the DHE based cipher you've shared. The error you get from openssl also matches that the certificate provided by the server does not meet the security requirements the client has set. I'm honestly not sure why as the ephemeral certificate has a length of 2048 bits but there must be some other reason why it is rejected.

This might also be why the builtin ssl module does not offer ciphers that pyOpenSSL offers. The RSA ciphers you've shared do not offer forward secrecy. If your system wide configuration is set to not allow these then the ssl module will not allow it. This is in constrast with pyOpenSSL which is, AFAIK, not using these system policies as it uses a static copy of OpenSSL in the Python wheel. It could also potentially be that the system OpenSSL being used does not offer the RSA ciphers and the copy of OpenSSL with pyOpenSSL is based on a newer version.

What I suggest you try next

The first thing I would suggest is to share your OS information and how you've installed Python 3, Ansible, requests-credssp, pyOpenSSL. Some of these details are highly dependent on how it was installed, pip vs dnf/apt. This can help me try and replicate the problem and offer more suggestions.

Failing that this is what I would suggest

• See if there is an update to OpenSSL/Python3
  • It might be compiled with an older OpenSSL version that doesn't offer support for the RSA keys
  • I personally don't think this is a problem and it's more than likely you have a system policy to not allow these keys
• Upgrade the certificate used by CredSSP through WinRM
  • Use the PowerShell snippet to set a stronger certificate, maybe with 4096 bits as the key length
  • This should hopefully be enough to pass the DHE key length restriction
  • You could potentially try the same cert as the WinRM HTTPS endpoint if you have one
• Find where your openssl.cnf file is located and set the level to SECLEVEL=1
  • This isn't really that great, there's a reason why 2 is the default these days as DH is broken with weak certificates

Unfortunately this is a complex problem that is only going to get harder in the future as Server 2012 ages and more ciphers get disabled through OpenSSL by default. It doesn't help but it's a sign of why upgrading the OS is important today as things like TLS gets hardened over time and older OS' don't necessarily get the newer protocols and cipher suites backported.

[nathangiuliani]

I had the same issue today. I haven't ran Ansible on my server 2012R2 boxes for a while, and just upgraded to Ubuntu 22.04 - so it's possibly related to the OpenSSL and/or Python updates.

Using requests-credssp 2.0 resulted in the server 2012 r2 errors shown above only, and worked fine on everything else (server 2016, 2019 and 2022).

I fixed the server 2012 R2 errors by using my WinRM cert for CredSSP also. I also needed to grant NT AUTHORITY\NETWORK SERVICE permissions to the certificate's private key. I could share some PowerShell for this if it helps anyone.

Interestingly, using requests-credssp < 2.0 resulted in this error, which I didn't investigate given that 2.0 is current.

credssp: SpnegoError (1): unsupported hash type md4, Context: Unable to negotiate common mechanism

[jborean93]

That is a separate issue jborean93/pyspnego#37. Python on Ubuntu 22.04 doesn't come with md4 enabled which is required for NTLM authentication. You really should be getting Kerberos authentication working and avoiding NTLM but until that linked issue is fixed NTLM won't work.

[kheifer]

@jborean93 I’m having similar issues between python 3.9 and 3.11. Same underlying modules in every step but no success on 3.10/3.11. Is 3.10/3.11 not supported? Is there any way to add cypher suites to ansible/winrm/python

[jborean93]

Unfortunately no, this library is at the mercy of whatever OpenSSL library Python is linked to and the policies that library needs to follow.

[kheifer]

Bummer, with some changes for python 3.10+ and the default ciphers credssp fails on win2012r2 and python 3.10+. I know ansible added support for win get url and get url to use specific ciphers