Use DYLD_FALLBACK_LIBRARY_PATH for dylib_path_envvar on macOS

[glandium]

When loading and linking a dynamic library or bundle, dlopen
searches in LD_LIBRARY_PATH, DYLD_LIBRARY_PATH, PWD, and
DYLD_FALLBACK_LIBRARY_PATH.
In the Mach-O format, a dynamic library has an "install path."
Clients linking against the library record this path, and the
dynamic linker, dyld, uses it to locate the library.
dyld searches DYLD_LIBRARY_PATH before the install path.
dyld searches DYLD_FALLBACK_LIBRARY_PATH only if it cannot
find the library in the install path.
Setting DYLD_LIBRARY_PATH can easily have unintended
consequences.

See https://users.rust-lang.org/t/subprocess-and-dynamic-library-linking-problem-interaction/7873/10
See https://trac.macports.org/ticket/57692
See https://bugzilla.mozilla.org/show_bug.cgi?id=1536486

This is the same change as was applied to cargo in
rust-lang/cargo#6355

[ehuss]

I've been meaning to look at making this change, but never got around to it. This also has a benefit that it improves performance on MacOS 10.13 and newer, though that probably doesn't affect most people.

You have to be careful when setting DYLD_FALLBACK_LIBRARY_PATH because it overrides the default of $(HOME)/lib:/usr/local/lib:/lib:/usr/lib. That means anything that was relying on that default fallback will now fail. See rust-lang/cargo#6625. I think this will probably need to retain the default and append to it.

I'm also curious if setting the library path is necessary at all. rustc uses @rpath, so it shouldn't have trouble finding its own libraries. Alex mentioned it might be due to plugins here. Probably not worth removing, but it would be good to know why this is set at all.

Testing this will be difficult. rustup doesn't really have a nightly testing channel, so it's difficult to pre-flight a change like this. It'll be tough to know who it will affect.

[kinnison]

@ehuss Does that mean you'd recommend we wait to merge this until it can be amended, or should we consider merging this anyway and perhaps opening an issue to look at doing the append approach?

[ehuss]

I think it would be safer to include the defaults.

[kinnison]

@glandium Are you able to amend the PR according to @ehuss 's suggestion?

[glandium]

I'll get to it next week.

[kinnison]

@glandium Awesome. Thank you for doing this.

[bors]

☔ The latest upstream changes (presumably #1754) made this pull request unmergeable. Please resolve the merge conflicts.

[kinnison]

Thank you @glandium for your work here. LGTM.