libstore: Add the ability to use different auth methods with Http Binary Caches

[georgyo]

Motivation

Right now the only auth methods allowed with http caches was the default basic auth method. This meant if you wanted to have any other mechanism for authentication you requiring patching the code.

My personal motivation is enabling ?authmethod=negotiate.

Context

This PR is very similar to #10568 but made to be generic and support as many methods from libcurl as we can reasonably support.

Priorities and Process

Add 👍 to pull requests you find important.

The Nix maintainer team uses a GitHub project board to schedule and track reviews.

[georgyo]

@roberth Do you have any thoughts on this version of the PR?

[fricklerhandwerk]

Discussed in Nix maintainer meeting:

• @edolstra: this could done be in a more general way building on top of Pluggable authentication #9857
  • this change only enables authentication methods for the HTTP binary cache and nothing else
  • pluggable auth does not deliver the feature right now though
    • that other PR does two things actually, splitting it may make it easier to merge
• we'd merge this PR if it didn't introduce a setting that we'd have to support forever
  • this is risky given this topic is currently under development
• we recommend trying to implement the feature using pluggable auth, trying out the branch
  • a valuable contribution would also be to rebase the pluggable auth PR
  • another thing would be doing some research whether Git credential helpers can be used to get the authentication method

[nixos-discourse]

This pull request has been mentioned on NixOS Discourse. There might be relevant details there:

https://discourse.nixos.org/t/2024-06-24-nix-team-meeting-minutes-155/47739/1

[georgyo]

An alternative approach to choosing the exact curl auth method would be to set to CURLAUTH_ANY, which is the git default. The pros here is that this will enable curl to upgrade the auth method to the most secure.

This is how Git behaves when talking to an HTTP repo and allows it to handle both Basic Auth and things like Negotiate. The downside to that approach is if the server advertise negotiation there is no way to force git to use basic.

Setting CURLAUTH_ANY would enable kerberos, (and digest/ntlm) support in nix without also introducing a flag that would have to be supported forever.

[roberth]

Exposing existing curl functionality seems sensible to me, without the added complexity of #9857

I've taken the liberty to push a commit with a partial test. iirc @georgyo had some suggestions to make it cover more of their use case. Something WebDAV related, and maybe more (didn't manage to write that down, sorry)

[roberth]

Suggested change

	curl_easy_setopt(req, CURLOPT_USERNAME, "");
	curl_easy_setopt(req, CURLOPT_PASSWORD, "");
	// Initialize the auth stack. It needs to have a username, password to trigger that. Otherwise kerberos isn't going to work.
	curl_easy_setopt(req, CURLOPT_USERNAME, "");
	curl_easy_setopt(req, CURLOPT_PASSWORD, "");

[roberth]

Suggested change

	// Our writeCallbackWrapper does not support rewinding which breaks
	// negotiate/kerberos auth over http/2.
	// Our writeCallbackWrapper does not support rewinding which breaks
	// negotiate/kerberos auth over http/2.
	// Curl would need to retry a request, after consuming part of our stream, and we currently don't have
	// a way to recover that initial already consumed part.