Add target-specific RUSTFLAGS variants

[jonhoo]

What does this PR try to resolve?

This is a continuation from the discussion in #10395 (comment), and is aimed at being able to solve problems like #4423 by extending the current RUSTFLAGS environment variable to also have per-target variants as in rust-lang/cc-rs#9.

To quote the unstable docs I wrote for the feature:

The -Z targeted-rustflags argument tells Cargo to also look for
target-specific variants of the RUSTFLAGS and
CARGO_ENCODED_RUSTFLAGS variables (and the same for RUSTDOCFLAGS).
The following variants are supported:

1. <var>_<target> - for example, RUSTFLAGS_X86_64_UNKNOWN_LINUX_GNU
2. <build-kind>_<var> - for example, HOST_RUSTFLAGS or TARGET_RUSTFLAGS

HOST_RUSTFLAGS allows setting flags to be passed to rustc for
host-artifacts (like build scripts) when cross-compiling, or all
artifacts when not cross-compiling. See also the host-config feature.
Note that Cargo considers any --target as cross-compiling, even if
the specified target is equal to the host's target triple.
TARGET_RUSTFLAGS specifically applies to all artifacts being built for
the target triple specified with --target.

This feature interacts with the target-applies-to-host setting. When
set to false, <var>_<host target> will not apply to artifacts
built for the host, like build scripts.

More specific flags always take precedence over less specific ones, with
ties broken in favor of CARGO_ENCODED_RUSTFLAGS. So, for example, if
RUSTFLAGS_$TARGET and TARGET_CARGO_ENCODED_RUSTFLAGS are both
specified, RUSTFLAGS_$TARGET would be used.

How should we test and review this PR?

The PR includes two new tests that check for the expected behavior of these flags, and how they behave when more than one is provided.

[rust-highfive]

r? @alexcrichton

(rust-highfive has picked a reviewer for you, use r? to override)

[jonhoo]

I did, as part of this, discover a problem with target-applies-to-host and rustflags as currently implemented (even without this PR). Currently, kind.is_host() returns true for all artifacts when --target is not specified. This means that the function that tries to determine the appropriate rust flags (env_args) has no way to discern host artifacts from target artifacts in that setting. Which in turn means that it cannot correctly implement target-applies-to-host = false when --target isn't specified. The code in Cargo today ends up not taking the fall-through path here for any compilation artifacts

cargo/src/cargo/core/compiler/build_context/target_info.rs

Lines 616 to 619 in 0a3f2b4

	if kind.is_host() {
	if target_applies_to_host && requested_kinds == [CompileKind::Host] {
	// This is the past Cargo behavior where we fall back to the same logic as for other
	// artifacts without --target.

and instead takes the path intended for host artifacts here

cargo/src/cargo/core/compiler/build_context/target_info.rs

Lines 621 to 625 in 0a3f2b4

	// In all other cases, host artifacts just get flags from [host], regardless of
	// --target. Or, phrased differently, no `--target` behaves the same as `--target
	// <host>`, and host artifacts are always "special" (they don't pick up `RUSTFLAGS` for
	// example).
	return Ok(rustflags_from_host(config, flags, host_triple)?.unwrap_or_else(Vec::new));

The net result of this is that if --target isn't specified, and target-applies-to-host = false, neither RUSTFLAGS, target.*.rustflags, nor build.rustflags is used for any artifact, and [host] (if set) applies to all artifacts.

This limitation continues to apply with this PR — if --target is not passed and target-applies-to-host = false, RUSTFLAGS_$TARGET and TARGET_RUSTFLAGS do not apply at all, and HOST_RUSTFLAGS applies to all artifacts (relevant code).

To fix this, I think we need a way to tell env_args whether the compilation unit under consideration is a host artifact like a plugin or build script separately from just kind.is_host(). Unfortunately, I'm not familiar enough with that part of the code-base to figure out how feasible that is. I suspect this is what this now-removed comment was alluding to:

cargo/src/cargo/core/compiler/build_context/target_info.rs

Lines 595 to 602 in fb47df7

	// We *want* to apply RUSTFLAGS only to builds for the
	// requested target architecture, and not to things like build
	// scripts and plugins, which may be for an entirely different
	// architecture. Cargo's present architecture makes it quite
	// hard to only apply flags to things that are not build
	// scripts and plugins though, so we do something more hacky
	// instead to avoid applying the same RUSTFLAGS to multiple targets
	// arches:

[jonhoo]

/cc @messense given #4423 (comment)

[alexcrichton]

Thanks for the PR, but I'm going to be stepping down from the Cargo team so I'm going to un-assign myself from this. The Cargo team will help review this when they get a chance.

[bors]

☔ The latest upstream changes (presumably #10486) made this pull request unmergeable. Please resolve the merge conflicts.

[bors]

☔ The latest upstream changes (presumably #10566) made this pull request unmergeable. Please resolve the merge conflicts.

[weihanglo]

Sorry for the extremely late review 😅

The implementation itself works for me, though I am entirely not sure whether this is a thing we are going to merge. As you know, we already have a few stable and unstable features interacting with each other in this area. It might be more confusing for users who want to tweak their compiler flags. Specifically, we have a bunch of ways to set them

• CARGO_ENCODED_RUSTFLAGS
• RUSTFLAGS
• target.<cfg|triple>.rustflags
• build.rustflags
• profile.<name>.rustflags
• cargo rustc -- <some flags>
• cargo build --config <value>

And maybe more. I could have missed some. I wonder how we teach newcomers about this mess 😞

I totally understand the motivations behind your pull request and other features. Just feel a bit overwhelmed.

[weihanglo]

How does this interact with artifact dependencies (RFC 3028)?
If an artifact dependency specified with a { …, target = "<triple>" }, even without --target specified, Cargo will still try to build that dependency on that target platform.

• Should it receive HOST_RUSTFLAGS or TARGET_RUSTFLAGS?
• If it is a build-deps, which one it should get?

[weihanglo]

Is it possible to make an example of this precedence chain? Such as

cargo/src/doc/src/reference/profiles.md

Lines 422 to 430 in 99ff79e

	The precedence for which value is used is done in the following order (first
	match wins):
	1. `[profile.dev.package.name]` — A named package.
	2. `[profile.dev.package."*"]` — For any non-workspace member.
	3. `[profile.dev.build-override]` — Only for build scripts, proc macros, and
	their dependencies.
	4. `[profile.dev]` — Settings in `Cargo.toml`.
	5. Default values built-in to Cargo.

[weihanglo]

Why setting discriminant manually, supposed rustc can derive Ord with default discriminants?

[weihanglo]

(nit) I believe here we can use allow_generic.then(…) 🎉

[weihanglo]

If str::split is a no-op for empty string, could we reduce these to one line?

[weihanglo]

These new functions and enum are not immediately clear to me. Should we add some docs for them?

[weihanglo]

Could we also update doc comment for fn env_args?