hashlib should probably load "default" OpenSSL provider on OpenSSL 3.x explicitly

[mgorny]

Bug report

FWIU OpenSSL 3.x disables loading the default provider automatically if one loads a provider explicitly before calling any MD-related function. Since hashlib normally relies on the MDs provided by the default OpenSSL provider, perhaps it should load them explicitly to ensure that they are present. This would also ensure that the loaded OpenSSL providers are consistent whether hashlib is loaded prior to the script loading other providers or not.

By default:

>>> import hashlib
>>> sorted(hashlib.algorithms_available)
['blake2b', 'blake2s', 'md5', 'md5-sha1', 'sha1', 'sha224', 'sha256', 'sha384', 'sha3_224', 'sha3_256', 'sha3_384', 'sha3_512', 'sha512', 'sha512_224', 'sha512_256', 'shake_128', 'shake_256', 'sm3']
>>> hashlib.new("sha512_256")
<sha512_256 _hashlib.HASH object @ 0x7fcccbff2c50>

But if I load the legacy provider first:

>>> import ctypes
>>> ctypes.CDLL("libssl.so").OSSL_PROVIDER_load(None, b"legacy")
-1589238480
>>> import hashlib
>>> sorted(hashlib.algorithms_available)
['blake2b', 'blake2s', 'md4', 'md5', 'mdc2', 'ripemd160', 'sha1', 'sha224', 'sha256', 'sha384', 'sha3_224', 'sha3_256', 'sha3_384', 'sha3_512', 'sha512', 'shake_128', 'shake_256', 'whirlpool']
>>> hashlib.new("sha512_256")
Traceback (most recent call last):
  File "/usr/lib/python3.11/hashlib.py", line 160, in __hash_new
    return _hashlib.new(name, data, **kwargs)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
ValueError: [digital envelope routines] unsupported

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
  File "/usr/lib/python3.11/hashlib.py", line 166, in __hash_new
    return __get_builtin_constructor(name)(data)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/usr/lib/python3.11/hashlib.py", line 123, in __get_builtin_constructor
    raise ValueError('unsupported hash type ' + name)
    ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
ValueError: unsupported hash type sha512_256
>>> import _hashlib
>>> _hashlib.openssl_md_meth_names
frozenset({'whirlpool', 'ripemd160', 'mdc2', 'md4'})

but if I load both default and legacy providers, I get the full set:

>>> import ctypes
>>> ctypes.CDLL("libssl.so").OSSL_PROVIDER_load(None, b"legacy")
-265107616
>>> ctypes.CDLL("libssl.so").OSSL_PROVIDER_load(None, b"default")
-265087936
>>> import hashlib
>>> sorted(hashlib.algorithms_available)
['blake2b', 'blake2s', 'md4', 'md5', 'md5-sha1', 'mdc2', 'ripemd160', 'sha1', 'sha224', 'sha256', 'sha384', 'sha3_224', 'sha3_256', 'sha3_384', 'sha3_512', 'sha512', 'sha512_224', 'sha512_256', 'shake_128', 'shake_256', 'sm3', 'whirlpool']
>>> hashlib.new("sha512_256")
<sha512_256 _hashlib.HASH object @ 0x7ffb1937f6b0>
>>> import _hashlib
>>> _hashlib.openssl_md_meth_names
frozenset({'sha512_256', 'sha3_512', 'sm3', 'sha512_224', 'sha1', 'md5', 'mdc2', 'sha3_256', 'blake2s', 'ripemd160', 'sha3_224', 'sha256', 'whirlpool', 'sha3_384', 'sha384', 'sha224', 'sha512', 'shake_256', 'md5-sha1', 'blake2b', 'md4', 'shake_128'})

Your environment

• CPython versions tested on: 3.11.0b1
• Operating system and architecture: Gentoo Linux amd64
• OpenSSL version: 3.0.3

[tomato42]

On the other hand, if the system in in FIPS mode, lack of default provider may instead be conscious and deliberate action by the system administrator.

Necessity to explicitly load both default and legacy providers if you want to use legacy algorithms is a deliberate choice by openssl.

[h-vetinari]

I don't think FIPS mode should be a concern here. I doubt that python itself can run (unmodified) in a FIPS environment, and the vast, vast majority of users are not affected by rules for USG agencies and contractors.

Said otherwise, someone trying to use this in a FIPS environment will have bigger issues than whether the legacy provider is loaded (which wouldn't be present on their system anyway). While it could ostensibly be wrapped with a try:/except: or a config knob, the default should surely be to load the legacy provider.

[tomato42]

1. FIPS is just an example of a most popular configuration where you are not using the default provider, there definitely be others too

2. Even if there are industries which do not have a legal requirement to be running FIPS certified software, they may still choose to do so because of other reasons (e.g. banking)

3. someone trying to use this in a FIPS environment will have bigger issues

   Python already has interfaces for working in FIPS-like environments, that's what usedforsecurity is for

4. legacy provider is loaded (which wouldn't be present on their system anyway)

   Not true, what's in scope of FIPS is protection of user data. If a cryptographic implementation isn't used for that, then FIPS doesn't care about it. You totally can use legacy provider when the system is in FIPS mode.

[thesamesam]

In any case, an easy way to load it would be nice, even if it's not default in Python.

[gpshead]

If algorithms are not provided at all in usedforsecurity=False mode by OpenSSL, hashlib falls back to our built-ins: https://github.com/python/cpython/blob/main/Lib/hashlib.py#L126-L141 per __get_openssl_constructor()

[gpshead]

If there are openssl providers people configure via whatever means rather than whatever a "default" was for the past 20 years that, for example, do not provide sha1 and md5 at all, hashlib will quite happily do its job and provide those itself. This is working as intended as far as I'm concerned.