hashlib.algorithms_available lists algorithms that are not available in OpenSSL 3.0 default provider

[tiran]

BPO	47101
Nosy	@tiran, @miss-islington
PRs	bpo-47101: list only activated algorithms in hashlib.algorithms_available #32076 [3.9] bpo-47101: list only activated algorithms in hashlib.algorithms_available (GH-32076) #32084 [3.10] bpo-47101: list only activated algorithms in hashlib.algorithms_available (GH-32076) (GH-32085) #32085

^{Note: these values reflect the state of the issue at the time it was migrated and might not reflect the current state.}

Show more details

GitHub fields:

assignee = None
closed_at = None
created_at = <Date 2022-03-23.14:28:49.732>
labels = ['type-bug', '3.9', '3.10', '3.11', 'extension-modules', 'library']
title = 'hashlib.algorithms_available lists algorithms that are not available in OpenSSL 3.0 default provider'
updated_at = <Date 2022-03-23.21:15:34.062>
user = 'https://github.com/tiran'

bugs.python.org fields:

activity = <Date 2022-03-23.21:15:34.062>
actor = 'christian.heimes'
assignee = 'none'
closed = False
closed_date = None
closer = None
components = ['Extension Modules', 'Library (Lib)']
creation = <Date 2022-03-23.14:28:49.732>
creator = 'christian.heimes'
dependencies = []
files = []
hgrepos = []
issue_num = 47101
keywords = ['patch']
message_count = 6.0
messages = ['415877', '415878', '415880', '415907', '415909', '415912']
nosy_count = 2.0
nosy_names = ['christian.heimes', 'miss-islington']
pr_nums = ['32076', '32084', '32085']
priority = 'normal'
resolution = None
stage = 'patch review'
status = 'open'
superseder = None
type = 'behavior'
url = 'https://bugs.python.org/issue47101'
versions = ['Python 3.9', 'Python 3.10', 'Python 3.11']

[tiran]

Hubert Kario wrote in https://bugzilla.redhat.com/show_bug.cgi?id=2054702

Description of problem:
The hashlib.algorithms_available set includes algorithms like ripemd160 and whirlpool, those algorithms are not usable unless openssl legacy provider is loaded. Since it's not loaded, and the hashlib module won't load it, any attempt to use them fails.

Version-Release number of selected component (if applicable):
python3-3.9.10-1.el9.x86_64
openssl-3.0.1-5.el9.x86_64

How reproducible:
always

Steps to Reproduce:
0. start python3

1. from hashlib import algorithms_available
2. algorithms_available
3. import hashlib
4. a = {(name, hashlib.new(name).digest_size) for name in algorithms_available}

Actual results:
{'sha3_384', 'blake2s', 'sha384', 'sha512_224', 'md5', 'sha3_512', 'md5-sha1', 'sha3_256', 'shake_128', 'sm3', 'sha256', 'sha512', 'sha1', 'shake_256', 'blake2b', 'whirlpool', 'sha512_256', 'sha3_224', 'sha224', 'ripemd160', 'md4'}

Traceback (most recent call last):
  File "/usr/lib64/python3.9/hashlib.py", line 164, in __hash_new
    return _hashlib.new(name, data, **kwargs)
ValueError: [digital envelope routines] unsupported

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
  File "<stdin>", line 1, in <setcomp>
  File "/usr/lib64/python3.9/hashlib.py", line 170, in __hash_new
    return __get_builtin_constructor(name)(data)
  File "/usr/lib64/python3.9/hashlib.py", line 127, in __get_builtin_constructor
    raise ValueError('unsupported hash type ' + name)
ValueError: unsupported hash type whirlpool

Expected results:
{'sha3_384', 'blake2s', 'sha384', 'sha512_224', 'md5', 'sha3_512', 'md5-sha1', 'sha3_256', 'shake_128', 'sm3', 'sha256', 'sha512', 'sha1', 'shake_256', 'blake2b', 'sha512_256', 'sha3_224', 'sha224'}

{('blake2b', 64), ('sha512', 64), ('md5-sha1', 36), ('sha3_512', 64), ('md5', 16), ('sha224', 28), ('shake_128', 0), ('sm3', 32), ('blake2s', 32), ('sha1', 20), ('shake_256', 0), ('sha512_256', 32), ('sha3_224', 28), ('sha3_256', 32), ('sha3_384', 48), ('sha384', 48), ('sha256', 32), ('sha512_224', 28)}

Additional info:
If the legacy provider is loaded, then the algorithms should be listed and should work.

It may be caused by Python using the deprecated EVP_MD_do_all() method instead of the EVP_MD_do_all_provided() method

[tiran]

Hubert's suggested solution EVP_MD_do_all_provided() worked almost straight forward. The function signature is a bit different and I got "undefined" in the result set. Filtering out NID_undef got right of it.

[tiran]

$ ./python Tools/ssl/multissltests.py --openssl 3.0.2 --steps modules
$ ./python -c "import hashlib; print(hashlib.algorithms_available)"
{'blake2b', 'sha512', 'sm3', 'shake_128', 'md5', 'sha3_256', 'sha224', 'sha512_224', 'sha3_384', 'sha384', 'md5-sha1', 'sha3_224', 'shake_256', 'sha3_512', 'sha512_256', 'sha1', 'sha256', 'blake2s'}

[miss-islington]

New changeset 48e2010 by Christian Heimes in branch 'main':
bpo-47101: list only activated algorithms in hashlib.algorithms_available (GH-32076)
48e2010

[miss-islington]

New changeset ec3589f by Miss Islington (bot) in branch '3.9':
bpo-47101: list only activated algorithms in hashlib.algorithms_available (GH-32076)
ec3589f

[tiran]

New changeset 1b6acaa by Christian Heimes in branch '3.10':
[3.10] bpo-47101: list only activated algorithms in hashlib.algorithms_available (GH-32076) (GH-32085)
1b6acaa

[mhsmith]

@tiran: It looks like this has been fixed: should the issue be closed?

[h-vetinari]

It looks like this has been fixed: should the issue be closed?

Hashlib still needs to load the legacy provider to achieve parity with 1.1.1 builds, but that's tracked in #92876.