Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

frequent System.Threading.Tasks crashes w/ macOS Big Sur #250

Closed
4 of 16 tasks
jaz2038 opened this issue Dec 16, 2020 · 22 comments
Closed
4 of 16 tasks

frequent System.Threading.Tasks crashes w/ macOS Big Sur #250

jaz2038 opened this issue Dec 16, 2020 · 22 comments
Labels
bug A bug in Git Credential Manager external Relating to an external partner, team, or library platform:osx Specific to the macOS platform

Comments

@jaz2038
Copy link

jaz2038 commented Dec 16, 2020

Which version of GCM Core are you using?
Git Credential Manager version 2.0.289-beta+22bd06dc57 (macOS, .NET Core 3.1.10)

From a terminal, run git-credential-manager-core version and paste the output.

Which Git host provider are you trying to connect to?

  • Azure DevOps
  • Azure DevOps Server (TFS/on-prem)
  • GitHub
  • GitHub Enterprise
  • Bitbucket
  • Other - please describe

Can you access the remote repository directly in the browser using the remote URL?

From a terminal, run git remote -v to see your remote URL.

  • Yes
  • No, I get a permission error
  • No, for a different reason - please describe

[Azure DevOps only] What format is your remote URL?

[Azure DevOps only] If the account picker shows more than one identity as you authenticate, check that you selected the same one that has access on the web.

  • Not applicable
  • I only see one identity
  • I checked each identity and none worked

Expected behavior

I am authenticated and my Git operation completes successfully.

Actual behavior

About 1/5 times after I do a git pull or git push, I get this crash - I didn't see this until I updated to macOS Big Sur (originally seen this with macOS 11.0.1, still seeing it with macOS 11.1). After the crash I have to reauthenticate

Unhandled exception. System.IO.FileLoadException: Could not load file or assembly 'System.Threading.Tasks, Version=4.1.2.0, Culture=neutral, PublicKeyToken=b03f5f7f11d50a3a'. A device attached to the system is not functioning.
 (0x8007001F)
File name: 'System.Threading.Tasks, Version=4.1.2.0, Culture=neutral, PublicKeyToken=b03f5f7f11d50a3a'
   at System.Diagnostics.ProcessWaitState.WaitForExit(Int32 millisecondsTimeout)
   at System.Diagnostics.Process.WaitForExitCore(Int32 milliseconds)
   at System.Diagnostics.Process.WaitForExit()
   at Microsoft.Git.CredentialManager.GitProcessConfiguration.TryGet(String name, String& value)
   at Microsoft.Git.CredentialManager.Settings.GetSettingValues(String envarName, String section, String property)+MoveNext()
   at System.Linq.Enumerable.TryGetFirst[TSource](IEnumerable`1 source, Boolean& found)
   at System.Linq.Enumerable.FirstOrDefault[TSource](IEnumerable`1 source)
   at Microsoft.Git.CredentialManager.Settings.TryGetSetting(String envarName, String section, String property, String& value)
   at Microsoft.Git.CredentialManager.Settings.get_CredentialNamespace()
   at Microsoft.Git.CredentialManager.CommandContext..ctor()
   at Microsoft.Git.CredentialManager.Program.Main(String[] args)
error: /usr/local/share/gcm-core/git-credential-manager-core get died of signal 6
Password for xxx: 
fatal: Authentication failed for xxx

(where xxx is my repo name)

Logs

Set the environment variables GCM_TRACE=1 and GIT_TRACE=1 and re-run your Git command. Review and redact any private information and attach the log.

06:28:57.811123 git.c:440               trace: built-in: git pull
06:28:57.824999 run-command.c:663       trace: run_command: git fetch --update-head-ok --recurse-submodules=on
06:28:57.831071 git.c:440               trace: built-in: git fetch --update-head-ok --recurse-submodules=on
06:28:57.847951 run-command.c:663       trace: run_command: GIT_DIR=.git git-remote-https origin XXX_GITREPO
06:28:58.198763 run-command.c:663       trace: run_command: '/usr/local/share/gcm-core/git-credential-manager-core get'
Unhandled exception. System.IO.FileLoadException: Could not load file or assembly 'System.Threading.Tasks, Version=4.1.2.0, Culture=neutral, PublicKeyToken=b03f5f7f11d50a3a'. A device attached to the system is not functioning.
 (0x8007001F)
File name: 'System.Threading.Tasks, Version=4.1.2.0, Culture=neutral, PublicKeyToken=b03f5f7f11d50a3a'
   at System.Diagnostics.ProcessWaitState.WaitForExit(Int32 millisecondsTimeout)
   at System.Diagnostics.Process.WaitForExitCore(Int32 milliseconds)
   at System.Diagnostics.Process.WaitForExit()
   at Microsoft.Git.CredentialManager.GitProcessConfiguration.TryGet(String name, String& value)
   at Microsoft.Git.CredentialManager.Settings.GetSettingValues(String envarName, String section, String property)+MoveNext()
   at System.Linq.Enumerable.TryGetFirst[TSource](IEnumerable`1 source, Boolean& found)
   at System.Linq.Enumerable.FirstOrDefault[TSource](IEnumerable`1 source)
   at Microsoft.Git.CredentialManager.Settings.TryGetSetting(String envarName, String section, String property, String& value)
   at Microsoft.Git.CredentialManager.Settings.get_CredentialNamespace()
   at Microsoft.Git.CredentialManager.CommandContext..ctor()
   at Microsoft.Git.CredentialManager.Program.Main(String[] args)


error: /usr/local/share/gcm-core/git-credential-manager-core get died of signal 6
Password for XXX_GITREPO:
06:29:10.839226 run-command.c:663       trace: run_command: '/usr/local/share/gcm-core/git-credential-manager-core erase'
06:29:13.375876 ...er/Application.cs:69 trace: [RunInternalAsync] Git Credential Manager version 2.0.289-beta+22bd06dc57 (macOS, .NET Core 3.1.10) 'erase'
06:29:13.392271 ...mmands/Command.cs:63 trace: [ExecuteAsync] Start 'erase' command...
06:29:13.413929 ...mmands/Command.cs:74 trace: [ExecuteAsync] Detecting host provider for input:
06:29:13.415383 ...mmands/Command.cs:75 trace: [ExecuteAsync] 	protocol=https
06:29:13.415571 ...mmands/Command.cs:75 trace: [ExecuteAsync] 	host=dev.azure.com
06:29:13.415590 ...mmands/Command.cs:75 trace: [ExecuteAsync] 	path=XXX_GITREPO
06:29:13.415597 ...mmands/Command.cs:75 trace: [ExecuteAsync] 	username=XXX_USERNAME
06:29:13.415606 ...mmands/Command.cs:75 trace: [ExecuteAsync] 	password=XXX_PWD
06:29:13.452930 ...viderRegistry.cs:129 trace: [GetProvider] Performing auto-detection of host provider.
06:29:13.454881 ...mmands/Command.cs:77 trace: [ExecuteAsync] Host provider 'Azure Repos' was selected.
06:29:13.455872 ...sHostProvider.cs:108 trace: [EraseCredentialAsync] Erasing stored credential in store with service=XXX_GITREPO account=...
06:29:13.530190 ...sHostProvider.cs:111 trace: [EraseCredentialAsync] Credential was successfully erased.
06:29:13.530255 ...mmands/Command.cs:81 trace: [ExecuteAsync] End 'erase' command...
fatal: Authentication failed for XXX_GITREPO
@mjcheetham
Copy link
Collaborator

Hello @jaz2038, thanks for reporting this!

Since you say you've "upgraded to" Big Sur, can I assume this is an Intel-based Mac, and not one of the new Apple Silicon M1 Macs?

Assuming this is an Intel Mac, can you please check if the following file exists:

/usr/local/share/gcm-core/System.Threading.Tasks.dll

Assume it does exist, are you running any anti-virus software, or running in a VM (or is there something "interesting" with your setup)?

@mjcheetham mjcheetham added platform:osx Specific to the macOS platform bug A bug in Git Credential Manager labels Dec 18, 2020
@jaz2038
Copy link
Author

jaz2038 commented Dec 18, 2020

yes correct, intel based Macbook Pro (15" 2019).
yes that file does exist.
yes - running an antivirus but didn't see this issue running same antivirus w/ previous macOS - I can experiment with it off to validate though. not running anything else that's really special with my setup, no VM, etc...

@jaz2038
Copy link
Author

jaz2038 commented Dec 18, 2020

@mjcheetham I see same errors w/ and w/o antivirus running.

@mjcheetham
Copy link
Collaborator

Hi @jaz2038, from the limited searching I've done it seems the specific error message coming from the .NET Runtime "A device attached to the system is not functioning" is due to an unknown I/O failure at the runtime-OS layer (ERROR_GEN_FAILURE is the runtime error value).

I'm not entirely sure how to go about debugging this issue further. Can you try any/all of the following:

  1. Uninstall GCM Core and then reinstall

    • If installed via Homebrew:
      brew --cask reinstall git-credential-manager-core
    • Otherwise:
      sudo /usr/local/share/gcm-core/uninstall.sh and then download and install the latest GCM Core release
  2. Download the .NET SDK and see if the same error occurs on a "Hello, World!" executable?

  3. ☢️ The Nuclear Option ☢️:
    Try installing macOS Big Sur fresh on another APFS volume (apparently APFS's volume handling makes this easy?) OR at least in a VM or another Mac running Big Sur.

I hate suggesting the latter option. Might also be interesting to reach out on the dotnet/runtime repository about help with this?

@mjcheetham mjcheetham added the external Relating to an external partner, team, or library label Jan 4, 2021
@jaz2038
Copy link
Author

jaz2038 commented Jan 7, 2021

@mjcheetham thx for looking deeper here - #1/reinstalling didn't help. As for #2, I'm actually building several .net apps that aren't hitting this. Yeah I'm hesitant to try out #3/nuclear option but I can reach out to dotnet/runtime to see if there is any possible help there. If you have any other builds you'd like to try, I'm game for beta testing/getting more debug information for you as well.

@johnterickson
Copy link

@mjcheetham I have a 100% repro on this. Every git push from my Big Sur repros this on the first attempt, but then works fine the second attempt. I know this sounds crazy. I am running Microsoft corporate Intune which does run Defender.

johnerickson@Filatovm18 lutcomp % git push
Unhandled exception. System.IO.FileLoadException: Could not load file or assembly 'System.Threading.Tasks, Version=4.1.2.0, Culture=neutral, PublicKeyToken=b03f5f7f11d50a3a'. A device attached to the system is not functioning.
 (0x8007001F)
File name: 'System.Threading.Tasks, Version=4.1.2.0, Culture=neutral, PublicKeyToken=b03f5f7f11d50a3a'
   at System.Diagnostics.ProcessWaitState.WaitForExit(Int32 millisecondsTimeout)
   at System.Diagnostics.Process.WaitForExitCore(Int32 milliseconds)
   at System.Diagnostics.Process.WaitForExit()
   at Microsoft.Git.CredentialManager.GitProcessConfiguration.TryGet(String name, String& value)
   at Microsoft.Git.CredentialManager.Settings.GetSettingValues(String envarName, String section, String property)+MoveNext()
   at System.Linq.Enumerable.TryGetFirst[TSource](IEnumerable`1 source, Boolean& found)
   at System.Linq.Enumerable.FirstOrDefault[TSource](IEnumerable`1 source)
   at Microsoft.Git.CredentialManager.Settings.TryGetSetting(String envarName, String section, String property, String& value)
   at Microsoft.Git.CredentialManager.Settings.get_CredentialNamespace()
   at Microsoft.Git.CredentialManager.CommandContext..ctor()
   at Microsoft.Git.CredentialManager.Program.Main(String[] args)


error: /usr/local/share/gcm-core/git-credential-manager-core get died of signal 6
Username for 'https://@github.com': [[A
johnerickson@Filatovm18 lutcomp % git push
Enumerating objects: 9, done.
Counting objects: 100% (9/9), done.
Delta compression using up to 8 threads
Compressing objects: 100% (5/5), done.
Writing objects: 100% (5/5), 483 bytes | 483.00 KiB/s, done.
Total 5 (delta 3), reused 0 (delta 0)
remote: Resolving deltas: 100% (3/3), completed with 3 local objects.
To https://github.com/johnterickson/lutcomp.git
   83ab641..7f21647  master -> master

@derrickstolee
Copy link
Contributor

@mjcheetham I have a 100% repro on this. Every git push from my Big Sur repros this on the first attempt, but then works fine the second attempt.

@johnterickson: a clarification, please: do you mean first attempt per shell session? Does it repro with a new Terminal tab, or restarting Terminal app? Or do you need to restart the computer?

@johnterickson
Copy link

Great question @derrickstolee. Of course, by saying the above I was able to jinx it and it won't happen. I think it has something to do with time and/or sleep. I will try to repro. What logs info should I be capturing? Also, if I can get a repro, I will try excluding the GCM Core folder from Mac Defender and see if that is the key.

@johnterickson
Copy link

@derrickstolee Ok the 100% (3/3) repro is to reboot and immediately try to git push.

By adding git-credential-manager-core to the Defender exclusion list, I no longer get the error.

@haydenlewis
Copy link

I am experiencing a similar issue, albeit when pushing to GitHub for a project unrelated to .NET Core development.

Unhandled exception. System.IO.FileLoadException: Could not load file or assembly 'System.Threading.Tasks, Version=4.1.2.0, Culture=neutral, PublicKeyToken=b03f5f7f11d50a3a'. A device attached to the system is not functioning.
 (0x8007001F)
File name: 'System.Threading.Tasks, Version=4.1.2.0, Culture=neutral, PublicKeyToken=b03f5f7f11d50a3a'
   at System.Diagnostics.ProcessWaitState.WaitForExit(Int32 millisecondsTimeout)
   at System.Diagnostics.Process.WaitForExitCore(Int32 milliseconds)
   at System.Diagnostics.Process.WaitForExit()
   at Microsoft.Git.CredentialManager.Interop.Posix.PosixEnvironment.TryLocateExecutable(String program, String& path)
   at Microsoft.Git.CredentialManager.EnvironmentExtensions.LocateExecutable(IEnvironment environment, String program)
   at Microsoft.Git.CredentialManager.CommandContext..ctor()
   at Microsoft.Git.CredentialManager.Program.Main(String[] args)


error: /usr/local/share/gcm-core/git-credential-manager-core get died of signal 6

Any suggestions to help resolve this issue would be much appreciated.

@fabiansalamo
Copy link

git-credential-manager-core

This did it for me, as I was running defender as well.

@givinalis
Copy link

The solution by @johnterickson worked perfectly for me:

  • Open Microsoft Defender (if this is running on your mac)
  • Click on Actions on the top bar and select Exclusions
  • From the pop up displayed select Add new > Process then enter git-credential-manager-core

You should be good to go.

@haydenlewis
Copy link

haydenlewis commented Mar 1, 2021

I am not running Microsoft Defender, so I remain uncertain how to resolve this issue.

@aaroncrespo
Copy link

I have added the exception, and still see the issue in about 3:4 git interactions.

@jmezach
Copy link

jmezach commented Mar 8, 2021

I'm suffering from the same issue. My organization isn't using Microsoft Defender but Sophos but the issues are very similar. Running git-credential-manager-core a couple of times results in different errors until at some point it just works. Then after a while the issue comes back and I need to run it again a couple of times before it magically fixes itself.

@jaz2038
Copy link
Author

jaz2038 commented Mar 9, 2021

@mjcheetham Problem is definitely related to use of antivirus (ESET in my case) - by temporarily disabling realtime protection everything works fine. Latest errors look slightly different w/ the latest version of git-credential-manager-core that moved to .net 5 (2.0.374-beta+161775c536):

Unhandled exception. System.TypeInitializationException: The type initializer for 'System.Net.Sockets.SocketAsyncEngine' threw an exception. ---> System.IO.FileLoadException: Could not load file or assembly 'System.Collections.Concurrent, Version=5.0.0.0, Culture=neutral, PublicKeyToken=b03f5f7f11d50a3a'. A device attached to the system is not functioning. (0x8007001F) File name: 'System.Collections.Concurrent, Version=5.0.0.0, Culture=neutral, PublicKeyToken=b03f5f7f11d50a3a' at System.Net.Sockets.SocketAsyncEngine..ctor() at System.Net.Sockets.SocketAsyncEngine.CreateEngines() at System.Net.Sockets.SocketAsyncEngine..cctor() --- End of inner exception stack trace --- at System.Net.Sockets.SafeSocketHandle..ctor(IntPtr preexistingHandle, Boolean ownsHandle) at System.Diagnostics.Process.OpenStream(Int32 fd, FileAccess access) at System.Diagnostics.Process.StartCore(ProcessStartInfo startInfo) at System.Diagnostics.Process.Start() at Microsoft.Git.CredentialManager.GitProcessConfiguration.TryGet(String name, String& value) at Microsoft.Git.CredentialManager.Settings.GetSettingValues(String envarName, String section, String property)+MoveNext() at System.Linq.Enumerable.TryGetFirst[TSource](IEnumerable1 source, Boolean& found)
at System.Linq.Enumerable.FirstOrDefault[TSource](IEnumerable1 source) at Microsoft.Git.CredentialManager.Settings.TryGetSetting(String envarName, String section, String property, String& value) at Microsoft.Git.CredentialManager.Settings.get_CredentialNamespace() at Microsoft.Git.CredentialManager.CommandContext..ctor() at Microsoft.Git.CredentialManager.Program.Main(String[] args)

@aaroncrespo
Copy link

There are known issues with EndpointSecurity.framework + git + open() and SIG handling. You can try the latest Big Sur Beta to check if that's the issue, I know it helped me.

@CoooWeee
Copy link

same issue on Big Sur 11.2.3

@aaroncrespo
Copy link

I should have been more clear but check 11.3.3

@jaz2038
Copy link
Author

jaz2038 commented Mar 13, 2021

@aaroncrespo thx a ton - upgrading to Big Sur 11.3 Beta 3 fixes the issue for me.

@sashaweiss
Copy link

@johnterickson great tip - that seems to be helping with me so far. (Big Sur 11.2.3, Intel Mac with Microsoft Defender running.)

@vtbassmatt
Copy link
Contributor

Appears to be fixed with a newer .NET, Defender, and/or Big Sur release 😁

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
bug A bug in Git Credential Manager external Relating to an external partner, team, or library platform:osx Specific to the macOS platform
Projects
None yet
Development

No branches or pull requests