Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Configure hosts separately from targets when --target is specified. #9322

Merged
merged 12 commits into from
Jun 1, 2021
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
13 changes: 9 additions & 4 deletions src/cargo/core/compiler/build_context/target_info.rs
Original file line number Diff line number Diff line change
Expand Up @@ -682,10 +682,15 @@ impl<'cfg> RustcTargetData<'cfg> {
) -> CargoResult<RustcTargetData<'cfg>> {
let config = ws.config();
let rustc = config.load_global_rustc(Some(ws))?;
let host_config = config.target_cfg_triple(&rustc.host)?;
let host_info = TargetInfo::new(config, requested_kinds, &rustc, CompileKind::Host)?;
let mut target_config = HashMap::new();
let mut target_info = HashMap::new();
let target_applies_to_host = config.target_applies_to_host()?;
let host_info = TargetInfo::new(config, requested_kinds, &rustc, CompileKind::Host)?;
let host_config = if target_applies_to_host {
config.target_cfg_triple(&rustc.host)?
} else {
config.host_cfg_triple(&rustc.host)?
};

// This is a hack. The unit_dependency graph builder "pretends" that
// `CompileKind::Host` is `CompileKind::Target(host)` if the
Expand All @@ -695,8 +700,8 @@ impl<'cfg> RustcTargetData<'cfg> {
if requested_kinds.iter().any(CompileKind::is_host) {
let ct = CompileTarget::new(&rustc.host)?;
target_info.insert(ct, host_info.clone());
target_config.insert(ct, host_config.clone());
}
target_config.insert(ct, config.target_cfg_triple(&rustc.host)?);
};
jameshilliard marked this conversation as resolved.
Show resolved Hide resolved

let mut res = RustcTargetData {
rustc,
Expand Down
4 changes: 4 additions & 0 deletions src/cargo/core/features.rs
Original file line number Diff line number Diff line change
Expand Up @@ -600,6 +600,8 @@ unstable_cli_options!(
namespaced_features: bool = ("Allow features with `dep:` prefix"),
no_index_update: bool = ("Do not update the registry index even if the cache is outdated"),
panic_abort_tests: bool = ("Enable support to run tests with -Cpanic=abort"),
host_config: bool = ("Enable the [host] section in the .cargo/config.toml file"),
target_applies_to_host: bool = ("Enable the `target-applies-to-host` key in the .cargo/config.toml file"),
patch_in_config: bool = ("Allow `[patch]` sections in .cargo/config.toml files"),
rustdoc_map: bool = ("Allow passing external documentation mappings to rustdoc"),
separate_nightlies: bool = (HIDDEN),
Expand Down Expand Up @@ -787,6 +789,8 @@ impl CliUnstable {
"panic-abort-tests" => self.panic_abort_tests = parse_empty(k, v)?,
"jobserver-per-rustc" => self.jobserver_per_rustc = parse_empty(k, v)?,
"configurable-env" => self.configurable_env = parse_empty(k, v)?,
"host-config" => self.host_config = parse_empty(k, v)?,
"target-applies-to-host" => self.target_applies_to_host = parse_empty(k, v)?,
"patch-in-config" => self.patch_in_config = parse_empty(k, v)?,
"features" => {
// For now this is still allowed (there are still some
Expand Down
10 changes: 10 additions & 0 deletions src/cargo/util/config/mod.rs
Original file line number Diff line number Diff line change
Expand Up @@ -1486,6 +1486,16 @@ impl Config {
.try_borrow_with(|| self.get::<RustdocExternMap>("doc.extern-map"))
}

/// Returns true if the `[target]` table should be applied to host targets.
pub fn target_applies_to_host(&self) -> CargoResult<bool> {
target::get_target_applies_to_host(self)
}

/// Returns the `[host]` table definition for the given target triple.
pub fn host_cfg_triple(&self, target: &str) -> CargoResult<TargetConfig> {
target::load_host_triple(self, target)
}

/// Returns the `[target]` table definition for the given target triple.
pub fn target_cfg_triple(&self, target: &str) -> CargoResult<TargetConfig> {
target::load_target_triple(self, target)
Expand Down
54 changes: 49 additions & 5 deletions src/cargo/util/config/target.rs
Original file line number Diff line number Diff line change
Expand Up @@ -19,7 +19,7 @@ pub struct TargetCfgConfig {
pub other: BTreeMap<String, toml::Value>,
}

/// Config definition of a `[target]` table.
/// Config definition of a `[target]` table or `[host]`.
#[derive(Debug, Clone)]
pub struct TargetConfig {
/// Process to run as a wrapper for `cargo run`, `test`, and `bench` commands.
Expand Down Expand Up @@ -64,18 +64,62 @@ pub(super) fn load_target_cfgs(config: &Config) -> CargoResult<Vec<(String, Targ
Ok(result)
}

/// Returns true if the `[target]` table should be applied to host targets.
pub(super) fn get_target_applies_to_host(config: &Config) -> CargoResult<bool> {
if config.cli_unstable().target_applies_to_host {
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Can an entry in the unstable book get added for this?

(personally I think it's fine if this is gated by -Zhost-config too)

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Can an entry in the unstable book get added for this?

Added.

(personally I think it's fine if this is gated by -Zhost-config too)

Well we require -Ztarget-applies-to-host to be enabled if -Zhost-config is enabled to ensure we appropriately test the stabilization ordering at the moment since we plan to stabilize the target-applies-to-host option before the host-config feature.

Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Can you detail more why these should be stabilized separately? I would imagine that there's no need for them to be separate and they should be stabilized at the same time. It seems like it would be odd if we provide a way to configure host/target separately but then we don't provide a way to configure the host.

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Reason is that we don't want to stabilize host-config without the new defaults since that complicates testing logic, so we first stabilize target-applies-to-host so that everyone can choose what behavior they want(retain existing or use new) ahead of the default change. For common use cases the host tables aren't actually needed so target-applies-to-host should usually be sufficient anyways to make cross compilation usable with cargo in most cases.

Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

@alexcrichton This came out of discussions about the path to future stabilization, and the path to changing the defaults, and how those will interlock.

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Instead what I mean is that there's a singular -Z flag which gates these two new features in Cargo, the target-applies-to-host configuration option and [host] configuration table. Both of those are completely ignored if the -Z flag isn't passed. If the -Z flag is passed then they're both respected as if both were stable as-is in this PR. The default of target-applies-to-host, regardless of the -Z flag, will be true. Once this is all stable we'll flip that to false.

The issue is that the implementation for one of these features(the [host] table feature) depends on the new defaults effectively. The -Z flag for target-applies-to-host does not change the defaults, however the -Z flag for host-config does as the host table implementation needs the new defaults.

I understand there are literally two features here that are separated in the code. I am asking why, at an interface level, we are bothering to give users two different knobs. I don't think that the knobs are going to be stable at different times and otherwise only the "all on" and "all off" knob configurations are what we'd like to get feedback on feature-wise.

Well they are deliberately separated due to the dependency of the host table feature on the new defaults essentially, so they would be stabilized at different times as outlined here. We have essentially one flag for the first step(target-applies-to-host setting) and one for the other(host table) with enforced ordering.

Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I don't think [host] depends on new defaults, it just depends on target-applies-to-host = false. That can happen either with new defaults or the variable itself. Personally I don't think this is a reason to have two features.

To reiterate my thoughts from before, I disagree that these two features should be stabilized separately. They're pretty intertwined and all the hard stuff with one is still present for the other.

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I don't think [host] depends on new defaults, it just depends on target-applies-to-host = false. That can happen either with new defaults or the variable itself.

Sure.

Personally I don't think this is a reason to have two features.

That by itself isn't the reason for having the features separated, it's a testing issue mostly for both automated tests and those testing with the -Z flag, allowing the use of a host-config feature with the old defaults increases the combinations we need to test for and results in confusing/unclear behavior such as the host table being ignored if the old defaults are used(in the case that target-applies-to-host is not set before the default change). In addition we want to test the new default behavior as well so tying the new defaults to the host-table feature flag seems logical there as then either the [host] table or the new defaults can be tested with the -Z host-config flag together.

To reiterate my thoughts from before, I disagree that these two features should be stabilized separately. They're pretty intertwined and all the hard stuff with one is still present for the other.

To some degree they are intertwined but it's mostly a one way dependency, [host] depends on target-applies-to-host = false but the opposite isn't really true, if we're going to change the defaults in the future it ends up being a lot easier/clearer to just tie that to the [host] table feature to simplify testing of the new defaults and avoid any unclear behavior for some combinations. Keep in mind also that the [host] table feature should only be needed for extremely exotic use cases, the target-applies-to-host feature should be sufficient to fix the broken builds in nearly all cases I can think of so I think it's fine delaying the [host] table feature stabilization to simplify the migration testing.

Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

As I've said many times before, I do not think it is useful to split up these features. I do not think that we can stabilize them on separate schedules, also as I've mentioned before.

This means, to me, the only two states of Cargo that we care about are both features on and both features off. Having one feature on and one off is not a useful state in Cargo since we'll never actually ship that to users. This, to me, means it's not worthwhile to test. It's testing functionality that no one should use, so I don't think we should even expose it and that would simplify things.

I'm honestly not really entirely certain why you're insistent on keeping these separate? As a maintainer of Cargo I'm offering my thoughts to help guide this implementation, but you seem very resolute in your ways that you want precisely the PR as-is right now. I don't really know how to reconcile that honestly. I've tried to respond to all the points you're bringing up but the discussion seems to be very circular where it's just coming around to the same points again and again.

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I do not think that we can stabilize them on separate schedules, also as I've mentioned before.

I guess I'm missing what the issue with this is I guess.

This means, to me, the only two states of Cargo that we care about are both features on and both features off.

There's still the default change as well, which I also wanted to test, sure, we could combine the others and separate that if you want or just not test the new defaults I supposed but that seemed to me to result in a more confusing migration process as only allowing the host config table feature with the new defaults seem to simplify testing of that feature+default combination.

Having one feature on and one off is not a useful state in Cargo since we'll never actually ship that to users. This, to me, means it's not worthwhile to test. It's testing functionality that no one should use, so I don't think we should even expose it and that would simplify things.

Well after the new defaults are applied one shouldn't need to set target-applies-to-host = false ever to use host tables, so I was just trying to avoid needing to set target-applies-to-host = false in cases where one also sets a host table as we probably won't have as good test coverage of all the combinations, we'll still probably be fine but I am less confident in regards to what the implementation correctness would be with them combined and having the default be introduced after the host table feature.

I'm honestly not really entirely certain why you're insistent on keeping these separate? As a maintainer of Cargo I'm offering my thoughts to help guide this implementation, but you seem very resolute in your ways that you want precisely the PR as-is right now. I don't really know how to reconcile that honestly. I've tried to respond to all the points you're bringing up but the discussion seems to be very circular where it's just coming around to the same points again and again.

I'm just trying to clarify the issues I'm running into as it's unclear what the best approach to dealing with them in a satisfactory way is.

Well we inherently have two sets of changes either way, the target-applies-to-host feature needs to be introduced as part of the first, sure we could introduce a host table at the same time as that but we still have the default change down the line anyways so that's why I tied the new defaults to the host table to just have those both as part of the second set of changes.

It's unclear how you want me to implement the change though, is either of these approaches what you have in mind:

  • Tie target-applies-to-host to host table feature and don't bother testing new defaults at all.
  • Tie target-applies-to-host to host table feature and add a different -Z flag which enables the new defaults for testing.

I think the existing approach is less confusing but either of those would still work fine if that's what you think is best.

if let Ok(target_applies_to_host) = config.get::<bool>("target-applies-to-host") {
Ok(target_applies_to_host)
} else {
Ok(!config.cli_unstable().host_config)
}
} else {
if config.cli_unstable().host_config {
anyhow::bail!(
"the -Zhost-config flag requires the -Ztarget-applies-to-host flag to be set"
);
} else {
Ok(true)
}
}
}

/// Loads a single `[host]` table for the given triple.
pub(super) fn load_host_triple(config: &Config, triple: &str) -> CargoResult<TargetConfig> {
if config.cli_unstable().host_config {
jameshilliard marked this conversation as resolved.
Show resolved Hide resolved
let host_triple_prefix = format!("host.{}", triple);
let host_triple_key = ConfigKey::from_str(&host_triple_prefix);
let host_prefix = match config.get_cv(&host_triple_key)? {
Some(_) => host_triple_prefix,
None => "host".to_string(),
};
load_config_table(config, &host_prefix)
} else {
Ok(TargetConfig {
runner: None,
rustflags: None,
linker: None,
links_overrides: BTreeMap::new(),
})
}
}

/// Loads a single `[target]` table for the given triple.
pub(super) fn load_target_triple(config: &Config, triple: &str) -> CargoResult<TargetConfig> {
load_config_table(config, &format!("target.{}", triple))
}

/// Loads a single table for the given prefix.
fn load_config_table(config: &Config, prefix: &str) -> CargoResult<TargetConfig> {
// This needs to get each field individually because it cannot fetch the
// struct all at once due to `links_overrides`. Can't use `serde(flatten)`
// because it causes serde to use `deserialize_map` which means the config
// deserializer does not know which keys to deserialize, which means
// environment variables would not work.
let runner: OptValue<PathAndArgs> = config.get(&format!("target.{}.runner", triple))?;
let rustflags: OptValue<StringList> = config.get(&format!("target.{}.rustflags", triple))?;
let linker: OptValue<ConfigRelativePath> = config.get(&format!("target.{}.linker", triple))?;
let runner: OptValue<PathAndArgs> = config.get(&format!("{}.runner", prefix))?;
let rustflags: OptValue<StringList> = config.get(&format!("{}.rustflags", prefix))?;
let linker: OptValue<ConfigRelativePath> = config.get(&format!("{}.linker", prefix))?;
// Links do not support environment variables.
let target_key = ConfigKey::from_str(&format!("target.{}", triple));
let target_key = ConfigKey::from_str(prefix);
let links_overrides = match config.get_table(&target_key)? {
Some(links) => parse_links_overrides(&target_key, links.val, config)?,
None => BTreeMap::new(),
Expand Down
55 changes: 55 additions & 0 deletions src/doc/src/reference/unstable.md
Original file line number Diff line number Diff line change
Expand Up @@ -641,6 +641,61 @@ cargo +nightly -Zunstable-options -Zconfig-include --config somefile.toml build

CLI paths are relative to the current working directory.

### target-applies-to-host
* Original Pull Request: [#9322](https://github.com/rust-lang/cargo/pull/9322)
* Tracking Issue: [#9453](https://github.com/rust-lang/cargo/issues/9453)

The `target-applies-to-host` key in a config file can be used set the desired
behavior for passing target config flags to build scripts.

It requires the `-Ztarget-applies-to-host` command-line option.

The current default for `target-applies-to-host` is `true`, which will be
changed to `false` in the future, if `-Zhost-config` is used the new `false`
default will be set for `target-applies-to-host`.

```toml
# config.toml
target-applies-to-host = false
```

```console
cargo +nightly -Ztarget-applies-to-host build --target x86_64-unknown-linux-gnu
```

### host-config
* Original Pull Request: [#9322](https://github.com/rust-lang/cargo/pull/9322)
* Tracking Issue: [#9452](https://github.com/rust-lang/cargo/issues/9452)

The `host` key in a config file can be used pass flags to host build targets
such as build scripts that must run on the host system instead of the target
system when cross compiling. It supports both generic and host arch specific
tables. Matching host arch tables take precedence over generic host tables.

It requires the `-Zhost-config` and `-Ztarget-applies-to-host` command-line
options to be set.

```toml
# config.toml
[host]
linker = "/path/to/host/linker"
[host.x86_64-unknown-linux-gnu]
linker = "/path/to/host/arch/linker"
[target.x86_64-unknown-linux-gnu]
linker = "/path/to/target/linker"
```

The generic `host` table above will be entirely ignored when building on a
`x86_64-unknown-linux-gnu` host as the `host.x86_64-unknown-linux-gnu` table
takes precedence.

Setting `-Zhost-config` changes the default for `target-applies-to-host` to
`false` from `true`.

```console
cargo +nightly -Ztarget-applies-to-host -Zhost-config build --target x86_64-unknown-linux-gnu
```

### unit-graph
* Tracking Issue: [#8002](https://github.com/rust-lang/cargo/issues/8002)

Expand Down
Loading