Git 2.32 cannot access git repositories secured using client certificates when using cURL schannel backend

[pascalmuller]

• I was not able to find an open or closed issue matching what I'm seeing

Setup

• Which version of Git for Windows are you using? Is it 32-bit or 64-bit?

$ git --version --build-options

git version 2.32.0.windows.1
cpu: x86_64
built from commit: 4c204998d0e156d13d81abe1d1963051b1418fc0
sizeof-long: 4
sizeof-size_t: 8
shell-path: /bin/sh
feature: fsmonitor--daemon

• Which version of Windows are you running? Vista, 7, 8, 10? Is it 32-bit or 64-bit?

$ cmd.exe /c ver

Microsoft Windows [Version 10.0.19041.1052]

• What options did you set as part of the installation? Or did you choose the
  defaults?

# One of the following:
> type "C:\Program Files\Git\etc\install-options.txt"
> type "C:\Program Files (x86)\Git\etc\install-options.txt"
> type "%USERPROFILE%\AppData\Local\Programs\Git\etc\install-options.txt"
$ cat /etc/install-options.txt

Editor Option: VIM
Custom Editor Path:
Default Branch Option:
Path Option: Cmd
SSH Option: OpenSSH
Tortoise Option: false
CURL Option: WinSSL
CRLF Option: CRLFAlways
Bash Terminal Option: MinTTY
Git Pull Behavior Option: Merge
Use Credential Manager: Core
Performance Tweaks FSCache: Enabled
Enable Symlinks: Disabled
Enable Pseudo Console Support: Enabled
Enable FSMonitor: Disabled

• Any other interesting things about your environment that might be related
  to the issue you're seeing?

In the repositories where the problem occurs, a p12 client certificate is used.

Details

• Which terminal/shell are you running Git from? e.g Bash/CMD/PowerShell/other

Bash

• What commands did you run to trigger this issue? If you can provide a
  Minimal, Complete, and Verifiable example
  this will help us understand the issue.

$ git fetch

• What did you expect to occur after running these commands?

Git fetches changes.

• What actually happened instead?

This error is shown:

fatal: unable to access 'https://cmstash.cm.ict.nl/scm/pcs/deltawebclient.git/': schannel: next InitializeSecurityContext failed: SEC_E_ILLEGAL_MESSAGE (0x80090326) - This error usually occurs when a fatal SSL/TLS alert is received (e.g. handshake failed). More detail may be available in the Windows System event log.

(This error is shown for all operations involving the remote repository, and is not limited to fetch)

• If the problem was occurring with a specific repository, can you provide the
  URL to that repository to help us with testing?

I cannot provide the URL to the repository, as this is a repository only available to employees of the company I work at. Note that p12 certificates are used as a security measure. This seems to be related to the problem.

Important: I already did some research. This is what I figured out so far:

• This problem does not occur in the previous version ( Git for Windows v2.31.1 )
• I could not find more information in the Windows System event log, even though the error suggests this.
• I tried setting GCM_TRACE=1 and GIT_TRACE_CURL=1 before executing the fetch. This provided me with some more information:

14:49:34.276162 http.c:774              == Info: schannel: disabled automatic use of client certificate
14:49:34.313837 http.c:774              == Info: schannel: next InitializeSecurityContext failed: SEC_E_ILLEGAL_MESSAGE (0x80090326) - This error usually occurs when a fatal SSL/TLS alert is received (e.g. handshake failed). More detail may be available in the Windows System event log.
14:49:34.314072 http.c:774              == Info: Closing connection 0

• The first line of this piece of output is interesting. The repositories causing problems require a p12 client certificate I have installed.
• I believe this might be caused by the upgrade to cURL v7.77.0. Looking in the cURL repository led me to this commit:

commit 54e747501626b81149b1b44949119d365db82004
Author: Jay Satiro <[email protected]>
Date:   Sat Feb 27 16:27:31 2021 -0500

    schannel: Disable auto credentials; add an option to enable it

    - Disable auto credentials by default. This is a breaking change
      for clients that are using it, wittingly or not.

    - New libcurl ssl option value CURLSSLOPT_AUTO_CLIENT_CERT tells libcurl
      to automatically locate and use a client certificate for
      authentication, when requested by the server.

    - New curl tool options --ssl-auto-client-cert and
      --proxy-ssl-auto-client-cert map to CURLSSLOPT_AUTO_CLIENT_CERT.

    This option is only supported for Schannel (the native Windows SSL
    library). Prior to this change Schannel would, with no notification to
    the client, attempt to locate a client certificate and send it to the
    server, when requested by the server. Since the server can request any
    certificate that supports client authentication in the OS certificate
    store it could be a privacy violation and unexpected.

    Fixes https://github.com/curl/curl/issues/2262
    Reported-by: Jeroen Ooms
    Assisted-by: Wes Hinsley
    Assisted-by: Rich FitzJohn

    Ref: https://curl.se/mail/lib-2021-02/0066.html
    Reported-by: Morten Minde Neergaard

    Closes https://github.com/curl/curl/pull/6673

• The problem this leaves me with, is that I can't find a way to set this option in Git, so the old behaviour can be opted in to. I would expect a configuration value (starting with "http.schannel"), but I cannot find one. I was actually very happy with the behaviour out of the box (where the certificate was automatically picked up from the Windows certificate store and sent to the server).

I hope this analysis helps to figure out a solution for anyone experiencing this problem.

[pascalmuller]

I've tested the release candidates between 2.31.1 and the latest release. Git for Windows 2.32.0-rc1 which "Comes with cURL v7.76.1." was the last version to work without issues.

[dscho]

Excellent analysis! In curl/curl@54e7475, I indeed find no way other than setting that flag to re-enable the client certificate.

So here is what I would suggest (but I won't have time to implement any time soon): Implement support for http.sslAutoClientCert, which is ignored for cURL versions prior to v7.77.

This might sound hard, but it really is not. Basically, all you need to imitate is 93aef7c. It is all in there: the documentation update, the file-local variable, parsing the config variable, and setting the option. The only thing I would do differently is to ignore the setting (i.e. not warn) for prior cURL versions. (And I would of course make the condition #if LIBCURL_VERSION_NUM >= 0x074d00, as 0x4d == 77.)

Your mission, should you choose to accept it 😁:

1. install Git for Windows' SDK,
2. sdk cd git,
3. edit http.c, imitating the above-mentioned commit
4. build Git via make -j$(nproc)
5. test in-place via ./git --exec-path="$PWD" -c http.sslAutoClientCert=true ls-remote <url>?
6. once it works, edit Documentation/config/http.txt
7. commit
8. open a PR

[dscho]

Oh, I should also mention that there is a way without installing Git for Windows' SDK:

1. fork git-for-windows/git if you haven't done that yet
2. clone from your fork
3. edit http.c as described above
4. commit and push to your fork
5. in your fork's Action tab, select git-artifacts and trigger a build manually from your branch (if I were you, I would also restrict to build only portable-x86_64, to save time)
6. once that build has passed, download the portable-x86_64 artifact and test locally
7. continue as in step 6 above.

This might get you started quicker than getting the full Git for Windows SDK, at the expense of a longer cycle time in case of build failures or in case a tiny bug crept in and needs to be debugged.