macOS dotnet CLI path for global tools is bad - Installer issue

[jrdodds]

Describe the bug

On macOS when a .Net tool is installed globally, it will not be found in the PATH.

The dotnet installer does attempt to make the directory for globally installed tools part of the PATH, but the mechanism the installer is relying on doesn't work as the installer apparently expects.

To Reproduce

1. Install dotnet on macOS using the Microsoft Installer from https://dotnet.microsoft.com/en-us/download.
2. Install a tool globally, e.g. dotnet tool install -g Microsoft.dotnet-httprepl
3. Assuming zsh which is the default shell on macOS (Catalina and newer), run the command which httprepl
4. The output of the which command will be "httprepl not found" indicating that the httprepl tool can’t be found in the PATH.

Further technical details

Why does this happen?

macOS has a mechanism for adding paths to the PATH environment variable. The files in /etc/paths.d are used by the path_helper tool to build up the PATH environment variable.

The installer for dotnet is adding two files for paths:

• /etc/paths.d/dotnet
• /etc/paths.d/dotnet-cli-tools

The dotnet file contains

/usr/local/share/dotnet

The dotnet-cli-tools file contains

~/.dotnet/tools

However, paths in these files should be literal. The tilde (~) is not expanded to be the current user home directory. Environment variable references are also not expanded.

bash

The default standard shell on macOS Catalina (released in 2019) and newer is zsh. Prior to Catalina the default standard shell was bash. The bash shell appears to expand the tilde at a later point for an interactive shell so this issue may not manifest for bash users. However, the files created in /etc/paths.d by the dotnet installer need to support all shells.

Can't Specify the Current User Home Directory

The tilde can't be expected to be expanded to be the current user home directory. ${HOME} also can’t be used. There is not a way to specify the current user home directory in a file in /etc/paths.d.

I don’t know if not supporting the current user home directory is by design or is an omission.

Given the current functionality, the dotnet installer should not create the /etc/paths.d/dotnet-cli-tools file at all.

Workaround for Individual Users

In the .profile file, the user can either add the following line to add the needed path:

# dotnet-cli-tools
export PATH=$PATH:~/.dotnet/tools

or, assuming the bad path exists in the PATH, add the following ‘fix-up’ to replace any tildes in the PATH (equivalent to bash):

export PATH=${PATH//\~/$HOME}

The .zprofile may be set to source the .profile file with the following line:

[[ -e ~/.profile ]] && emulate sh -c 'source ~/.profile'

[jrdodds]

see also
#22588
#22214

[baronfel]

Thanks @jrdodds, this is the definitive summary of the MacOS issues.

I agree that we should not set any paths on /etc/paths.d that are user-specific.

If we want to maintain automatic PATH modification, then I agree that the best path for the installer going forward would be to add a $HOME-based path, but I after reading this post I think modifying the user's .zshrc is the appropriate place for it. And then there are other concerns we'll need to tackle as well - what if the user has set another shell? We'll need to have a fallback that informs the user that the dotnet tools path isn't in PATH, and that fallback sounds like modifications to the dotnet CLI's output to me.

[jrdodds]

Hi @baronfel, The installer should ask before modifying the user's .zshrc file or any of the other shell configuration files. It's very bad etiquette to silently modify any of these files. (That's part of the reason for the path_helper.)

The post that you reference leaves out the $HOME/.zprofile file which is read before the .zshrc file. (See Order of execution and purpose of .profile vs .zshrc)

The installer could test for the running shell (by checking the $SHELL env var) and could modify the appropriate .*profile file (or .*rc file). That doesn't solve the case where the user switches shells or where the user is interactively running one shell and writing shell scripts in another shell. For lowest common denominator compatibility, scripts are often written to use the Bourne shell (sh).

The .profile file is for the Bourne shell and it's common to share across shells via the .profile file but that is behavior that must be configured.

You might consider the example of HomeBrew. The brew command has a shellenv argument. A dotnet shellenv that is eval'd could return an export line to add the tools directory to the PATH if ~/.dotnet/tools is not already in the PATH. If the eval command is added to multiple shell configuration files that are read in the same session, the directory won't be duplicated in PATH.

Thanks

[baronfel]

You might consider the example of HomeBrew. The brew command has a shellenv argument. A dotnet shellenv that is eval'd could return an export line to add the tools directory to the PATH if ~/.dotnet/tools is not already in the PATH. If the eval command is added to multiple shell configuration files that are read in the same session, the directory won't be duplicated in PATH.

This is my preferred solution, and what I was meaning by a reference to new work in the CLI. Thanks for the additional links about the order of execution, I'll check them out 👍

[kazo0]

Just a bump, I am still experiencing this issue

[cicorias]

be great to get an update as just had a team of folks run through this on their Macs.

[ssbarnea]

I am amazed how this ended up being implemented because it is well known that user of tilde (~) is invalid inside PATH environments. While most shells will accept and expand the ~, many tools will break. Tilde should never be part of PATH.

[cert01]

This "~" was causing all manner of mischief for me by simply existing in my $PATH.
Rather than add the "export PATH=${PATH//~/$HOME}" to the user .zshrc, I Modified the /etc/paths.d/dotnet-cli-tools file to fix it until package resolution comes down the pipeline.

# MACOS zsh  >
brew info dotnet-sdk
#    ==> dotnet-sdk: 7.0.101,ea9de698-864c-4ddb-8ff3-1bf068c6b1a7,cf5d147f2c167317394058deb4d4b0a7
#    https://www.microsoft.com/net/core#macos
#    /opt/homebrew/Caskroom/dotnet-sdk/7.0.101,ea9de698-864c-4ddb-8ff3-1bf068c6b1a7,cf5d147f2c167317394058deb4d4b0a7 (179.1MB)
#    From: https://github.com/Homebrew/homebrew-cask/blob/HEAD/Casks/dotnet-sdk.rb
#    ==> Name
#    .NET SDK
#    ==> Description
#    Developer platform
#    ==> Artifacts
#    dotnet-sdk-7.0.101-osx-arm64.pkg (Pkg)
#    /usr/local/share/dotnet/dotnet (Binary)
#    ==> Analytics
#    install: 2,069 (30 days), 6,115 (90 days), 23,319 (365 days)
# MACOS zsh  >
sw_vers
#    ProductName:		macOS
#    ProductVersion:		13.0.1
#    BuildVersion:		22A400
#  MACOS zsh  >
cat /etc/paths.d/dotnet-cli-tools
~/.dotnet/tools%

Note: the % shows unterminated line in file

# MACOS zsh  >
[[ $PATH = *~/.dotnet/tool* ]] && export PATH=${PATH//\~/$HOME}
echo "\$HOME/.dotnet/tools" | sudo tee /etc/paths.d/dotnet-cli-tools

[AndreuCodina]

Failing on my Macbook M1. I've executed the command:

dotnet tool install -g --add-source "https://api.nuget.org/v3/index.json" --ignore-failed-sources upgrade-assistant

And the output is the next:

Tools directory '/Users/andreu/.dotnet/tools' is not currently on the PATH environment variable.
If you are using zsh, you can add it to your profile by running the following command:

cat << \EOF >> ~/.zprofile

Add .NET Core SDK tools

export PATH="$PATH:/Users/andreu/.dotnet/tools"
EOF

And run zsh -l to make it available for current session.

You can only add it to the current session by running the following command:

export PATH="$PATH:/Users/andreu/.dotnet/tools"

You can invoke the tool using the following command: upgrade-assistant
Tool 'upgrade-assistant' (version '0.4.421302') was successfully installed.

upgrade-assistant

zsh: command not found: upgrade-assistant

echo $PATH

...:~/.dotnet/tools:...

[agross]

I believe tilde is not expanded in PATH.

[0xced]

Quoting myself from Global tools are not found on zsh (a duplicate of this issue) which was filed on May 21, 2018!

Have you considered simply creating the /etc/paths.d/dotnet-cli-tools file with /Users/username/.dotnet/tools (with the current user name) instead of ~/.dotnet/tools?

Pros:

• Works on all shells (bash, zsh, etc.)
• No need to print instructions on how to make cli tools available for your shell, works out of the box

Cons:

• Only works for the user who installed the .NET Core SDK

In my opinion, the pros largely outweight the cons. Not having .NET cli tools work out of the box is a shame.

Unfortunately, I have not been heard by the dotnet sdk team and this issue is still not fixed 5 years later. 🤷‍♀️

By the way, Rick Strahl blogged about it in Dotnet Tool Component not found on the Mac a few days ago.

[jrdodds]

Have you considered simply creating the /etc/paths.d/dotnet-cli-tools file with /Users/username/.dotnet/tools (with the current user name) instead of ~/.dotnet/tools?

That's a really bad idea because the /etc/paths.d/dotnet-cli-tools file is used for all users on the machine.

[0xced]

I just submitted #35411 to address this issue by writing the full path (/Users/username/.dotnet/tools) into /etc/paths.d/dotnet-cli-tools.

Let's hope it gets merged and this PATH nightmare will finally be over. 🤞🏻

[baronfel]

I agree with @jrdodds - the multi-user aspect is why we haven't done this very simple change this entire time.

I still maintain that the correct thing to do is to provide a command (like dotnet tool env <SHELL> or something along those lines (would probably look at other ecosystems for consistency here)) that would write out the correct syntax for the given shell, and have the users put a call to eval $(dotnet tool env <SHELL>) in their profile - we would print this in the first-run output.

[0xced]

Maybe we can try to be pragmatic? This issue has existed since the introduction of the dotnet global tools and caused headaches to gazillion of developers. The proposed pull request solves this issue for developers installing the SDK themselves on their account and then using it on their account which I'd wager is more than 99% of the macOS dotnet developers. I'm pretty sure that the multi-user aspect exists only in the absolute but not in the reality of people using the dotnet SDK.

[jrdodds]

The directions to change the .profile are simple, it is a trivial one-time change, and it 'plays by the rules'. Is enabling command completion also a "nightmare"? I'm not asking to be snarky. Many people are very uncomfortable with making changes in the various shell configuration files. Part of the problem is that there is a lot of bad information about which file is used for what. I don't think we should contribute to the mess by essentially breaking the rules for convenience.

[0xced]

The directions to change the .profile are simple, it is a trivial one-time change, and it 'plays by the rules'.

Yes, absolutely. Yet it's way too easy to miss those instructions, leading to the nightmare I was talking about. There's even issue #3992 (PrintAddPathInstructionIfPathDoesNotExist should be more visible) acknowledging this.

I have added this to my .zshrc many years ago so this is not a problem for me personally anymore.

# See https://github.com/dotnet/sdk/issues/9415#issuecomment-406915716
export PATH=$HOME/.dotnet/tools:$PATH

It's just that people keep hitting this issue years after years after years. While not perfect, adding the full path is a pragmatic solution that certainly improves the current situation. 🤷‍♀️

Many people are very uncomfortable with making changes in the various shell configuration files.

Which should be an argument for the dotnet global tool PATH story to work out of the box, isn't it? 😉

Note: the ~ is expanded by bash but not by zsh, so the pull request could be improved to only include the full path on zsh but not on bash where it works as currently implemented.

[b4ux1t3]

My two cents as a Mac user:

Adding the home expanded path to a system-wide path isn't going to be an issue.

If multiple users need to install dotnet tools, and they are all added to the global path, nothing bad will happen. The users will be unable to see their peers' home directories and the path lookup will just skip it.

The only downside here is that one user can accidentally break dotnet tools for all other users if they have root privileges... Which they could do anyway.

There is absolutely no reason not to take the pragmatic approach here. We're talking about solving a huge pain point for thousands of users, which doesn't impact other users in any way. That is the very definition of a win-win in this industry.

[jrdodds]

If multiple users need to install dotnet tools, and they are all added to the global path, nothing bad will happen.

Dotnet only needs to be installed once to be available to all users on the machine but only the user that installed dotnet will have their expanded user tools directory in the /etc/paths.d/dotnet-cli-tools file. If each user were expected to re-run the installer, the installer would need to change so that it doesn't just overwrite the file.

[jrdodds]

path_helper (which reads /etc/paths.d and /etc/manpaths.d) is an Apple creation. Having a way for applications to add to the PATH and MANPATH without touching a user's shell configuration files is a good idea. But the path_helper implementation has problems. Relevant to the discussion here, path_helper doesn't have a designed way to support user home paths. That's a glaring omission. Perhaps the person who implemented path_helper thought it was acceptable to make the assumption that only the bash shell would ever be used.

As noted in "Paths in /etc/paths.d are not expanded. Can user home directories be supported?", I filed FB9751632 with Apple which remains in an open state.