Make SHA256 compression runtime pluggable

[furszy]

Tackling the long-standing request #702.

Right now we ship our own SHA256 implementation, a standard baseline version that does not take advantage of any hardware-optimized instruction, and it cannot be accessed by the embedding application - it is for internal usage only.

This means embedding applications often have to implement or include a different version for their use cases, wasting space on constrained environments, and in performance-sensitive setups it forces them to use a slower path than what the platform provides. Many projects already rely on tuned SHA-NI / ARMv8 / or other hardware-optimized code, so always using the baseline implementation we ship within the library is not ideal.

These changes allow users to supply their own SHA256 compression function at runtime, while preserving the existing default behavior for everyone else. This is primarily intended for environments where the available SHA256 implementation is detected dynamically and recompiling the library with a different implementation is not feasible (equivalent build-time functionality will come in a follow-up PR).

It introduces a new API:

secp256k1_context_set_sha256_transform_callback(ctx, fn_transform)

This function installs the optimized SHA256 compression into the secp256k1_context, which is then used by all internal computations. Important: The provided function is verified to be output-equivalent to the original one.

As a quick example, using this functionality in Bitcoin-Core will be very straightforward: furszy/bitcoin-core@f68bef0

[fjahr]

I guess this is more of a draft for initial review but I will spell it out anyway: It's currently missing some CI coverage for building with an external library as well as some docs. Curious what people think in terms of docs for something like this: Should there be extensive guidance in which context this is safe to use or would users be left on their own to try it out? Or are we assuming all users that go to this length are competent enough to judge if it's a good idea to bring their own sha256 or use the systems one?

[furszy]

Thanks for the feedback fjahr!

I guess this is more of a draft for initial review but I will spell it out anyway: It's currently missing some CI coverage for building with an external library as well as some docs.

Yeah, can create a CI job testing the compile-time pluggable compression function very easily. Thanks for reminding that.

And about the missing docs; yeah. I didn't add it because we currently don't have much documentation outside configure.ac / CMakeLists.txt. Happy to create a file for it.

Curious what people think in terms of docs for something like this: Should there be extensive guidance in which context this is safe to use or would users be left on their own to try it out? Or are we assuming all users that go to this length are competent enough to judge if it's a good idea to bring their own sha256 or use the systems one?

It’s not that we’re letting people plug in whatever they want. Both introduced features have guardrails:

1. Compile-time: we run all current tests against the provided implementation, plus a runtime self-test.
2. Runtime: besides the runtime self-test, have introduced an "equivalence check" that hashes known inputs and compares them against the library’s internal implementation before accepting the external function.

So you can bring your own compression function, but it still has to prove it behaves exactly like ours bit-for-bit.

Also, the target user here is someone who's actually written a hardware-optimized SHA256. It’s not like they stumbled into this by accident.

[real-or-random]

Concept ACK, great to see a PR for this!

[real-or-random]

nit: I think this one should also be an enable-style configure option instead of a with-style configure options; with is for compiling with external packages e.g., with-libxyz. And what the user provides here is not a package.

[hebasto]

Concept ACK.

[furszy]

Update pushed, thanks for the deep review real_or_random!
I haven’t fully finished addressing all the comments yet, but I'll hopefully finish tomorrow. So many good topics.

[sipa]

A few high-level comments:

• I don't think we should do link-time plugging in the same PR as runtime plugging, and when we do, I don't think it should be done through the "module" abstractions. We use modules for exporting functionality from the library, not for offering replacement of internal dependencies (I find it strange to have base functionality depend on a module!).
• "round" is a term of art in cipher design, and SHA-256 has exactly 64 rounds. Using the same term with another meaning is confusing.
• I would try hard to avoid storing the transformation function pointer inside the secp256k1_sha256 struct. Reviewing many lines that just add an extra parameter being pass through is not hard.

[furszy]

Per feedback, this is how it would look if we pass the function everywhere furszy@8149fc1 vs how it looks when the function lives in the sha256 struct (an updated version of what we currently have here): furszy@eb57082

Let me know what you guys think.

[sipa]

@furszy From a quick glance, the passing of the function everywhere is fine I think.

I'd suggest passing a pointer to secp256k1_hash_context to the sha256 functions rather than the function pointer itself. That makes it easier to (if ever) add additional fields to that context.

[furszy]

Updated per feedback. We now pass the hash context to all functions that need it, without storing any state inside the sha256 struct. The API arg "rounds" has also been renamed to "blocks" to prevent confusion with existing terms. Also, the commit introducing the compile-time feature has been removed for inclusion at a later stage (see #1777 (comment) for more details).

[real-or-random]

Approach ACK

[furszy]

Updated per feedback. Thanks @real-or-random!

[real-or-random]

two nits on the last commit (I like that one!)

[sipa]

Posthumous reACK 4d92a08