Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Follow-up work for download-rustc #81930

Closed
12 of 13 tasks
jyn514 opened this issue Feb 9, 2021 · 25 comments
Closed
12 of 13 tasks

Follow-up work for download-rustc #81930

jyn514 opened this issue Feb 9, 2021 · 25 comments
Labels
A-contributor-roadblock Area: Makes things more difficult for new contributors to rust itself A-download-rustc Area: The `rust.download-rustc` build option. C-tracking-issue Category: A tracking issue for an RFC or an unstable feature. T-bootstrap Relevant to the bootstrap subteam: Rust's build system (x.py and src/bootstrap)

Comments

@jyn514
Copy link
Member

jyn514 commented Feb 9, 2021

These are all the features for download-rustc that didn't land in the initial MVP (#79540).

Required before this is made a default

Nice to have

@jyn514 jyn514 added T-bootstrap Relevant to the bootstrap subteam: Rust's build system (x.py and src/bootstrap) T-dev-tools Relevant to the dev-tools subteam, which will review and decide on the PR/issue. C-tracking-issue Category: A tracking issue for an RFC or an unstable feature. A-contributor-roadblock Area: Makes things more difficult for new contributors to rust itself labels Feb 9, 2021
@jyn514 jyn514 self-assigned this Feb 9, 2021
@jyn514
Copy link
Member Author

jyn514 commented Feb 24, 2021

Right now only rustdoc is ever rebuilt. This is not ideal and should probably at least compile compiler tools (rustfmt, clippy, miri). #79540 (comment)

While working on this, I found that the handling of ENABLE_DOWNLOAD_STAGE0 is broken: if a Step is called as part of ensure() instead of through maybe_run, it will still be built even if the constant is set to false. Not yet sure how to fix this.

Note that this has local changes enabling the constant for clippy:

diff --git a/src/bootstrap/tool.rs b/src/bootstrap/tool.rs
index 5c874f69bd9..c82b78c1c71 100644
--- a/src/bootstrap/tool.rs
+++ b/src/bootstrap/tool.rs
@@ -564,6 +564,7 @@ pub struct Cargo {
 impl Step for Cargo {
     type Output = PathBuf;
     const DEFAULT: bool = true;
+    const ENABLE_DOWNLOAD_RUSTC: bool = true;
     const ONLY_HOSTS: bool = true;
 
     fn should_run(run: ShouldRun<'_>) -> ShouldRun<'_> {
@@ -647,6 +648,7 @@ pub struct $name {
         impl Step for $name {
             type Output = Option<PathBuf>;
             const DEFAULT: bool = true; // Overwritten below
+            const ENABLE_DOWNLOAD_RUSTC: bool = true;
             const ONLY_HOSTS: bool = true;
 
             fn should_run(run: ShouldRun<'_>) -> ShouldRun<'_> {
$ x.py build -v src/tools/clippy
detected default triple x86_64-unknown-linux-gnu
Updating only changed submodules
Submodules updated in 0.01 seconds
using downloaded stage1 artifacts from CI (commit fe1bf8e05c39bdcc73fc09e246b7209444e389bc)
running: /home/joshua/rustc3/build/x86_64-unknown-linux-gnu/stage0/bin/cargo build --manifest-path /home/joshua/rustc3/src/bootstrap/Cargo.toml
    Finished dev [unoptimized + debuginfo] target(s) in 0.09s
running: /home/joshua/rustc3/build/bootstrap/debug/bootstrap build -v src/tools/clippy
finding compilers
CC_x86_64-unknown-linux-gnu = "cc"
CFLAGS_x86_64-unknown-linux-gnu = ["-ffunction-sections", "-fdata-sections", "-fPIC", "-m64"]
CXX_x86_64-unknown-linux-gnu = "c++"
CXXFLAGS_x86_64-unknown-linux-gnu = ["-ffunction-sections", "-fdata-sections", "-fPIC", "-m64"]
AR_x86_64-unknown-linux-gnu = "ar"
running sanity check
auto-detected local-rebuild 1.52.0-nightly
learning about cargo
> Assemble { target_compiler: Compiler { stage: 1, host: TargetSelection { triple: "x86_64-unknown-linux-gnu", file: None } } }
  > Assemble { target_compiler: Compiler { stage: 0, host: TargetSelection { triple: "x86_64-unknown-linux-gnu", file: None } } }
  < Assemble { target_compiler: Compiler { stage: 0, host: TargetSelection { triple: "x86_64-unknown-linux-gnu", file: None } } }
  > Sysroot { compiler: Compiler { stage: 1, host: TargetSelection { triple: "x86_64-unknown-linux-gnu", file: None } } }
  < Sysroot { compiler: Compiler { stage: 1, host: TargetSelection { triple: "x86_64-unknown-linux-gnu", file: None } } }
< Assemble { target_compiler: Compiler { stage: 1, host: TargetSelection { triple: "x86_64-unknown-linux-gnu", file: None } } }
> Clippy { compiler: Compiler { stage: 1, host: TargetSelection { triple: "x86_64-unknown-linux-gnu", file: None } }, target: TargetSelection { triple: "x86_64-unknown-linux-gnu", file: None }, extra_features: [] }
  > ToolBuild { compiler: Compiler { stage: 1, host: TargetSelection { triple: "x86_64-unknown-linux-gnu", file: None } }, target: TargetSelection { triple: "x86_64-unknown-linux-gnu", file: None }, tool: "clippy-driver", path: "src/tools/clippy", mode: ToolRustc, is_optional_tool: true, source_type: InTree, extra_features: [] }
    > Rustc { target: TargetSelection { triple: "x86_64-unknown-linux-gnu", file: None }, compiler: Compiler { stage: 1, host: TargetSelection { triple: "x86_64-unknown-linux-gnu", file: None } } }
      > Std { target: TargetSelection { triple: "x86_64-unknown-linux-gnu", file: None }, compiler: Compiler { stage: 1, host: TargetSelection { triple: "x86_64-unknown-linux-gnu", file: None } } }
        > StartupObjects { compiler: Compiler { stage: 1, host: TargetSelection { triple: "x86_64-unknown-linux-gnu", file: None } }, target: TargetSelection { triple: "x86_64-unknown-linux-gnu", file: None } }
        < StartupObjects { compiler: Compiler { stage: 1, host: TargetSelection { triple: "x86_64-unknown-linux-gnu", file: None } }, target: TargetSelection { triple: "x86_64-unknown-linux-gnu", file: None } }
        c Assemble { target_compiler: Compiler { stage: 1, host: TargetSelection { triple: "x86_64-unknown-linux-gnu", file: None } } }
        > Libdir { compiler: Compiler { stage: 1, host: TargetSelection { triple: "x86_64-unknown-linux-gnu", file: None } }, target: TargetSelection { triple: "x86_64-unknown-linux-gnu", file: None } }
          c Sysroot { compiler: Compiler { stage: 1, host: TargetSelection { triple: "x86_64-unknown-linux-gnu", file: None } } }
        < Libdir { compiler: Compiler { stage: 1, host: TargetSelection { triple: "x86_64-unknown-linux-gnu", file: None } }, target: TargetSelection { triple: "x86_64-unknown-linux-gnu", file: None } }
        c Libdir { compiler: Compiler { stage: 1, host: TargetSelection { triple: "x86_64-unknown-linux-gnu", file: None } }, target: TargetSelection { triple: "x86_64-unknown-linux-gnu", file: None } }
        c Sysroot { compiler: Compiler { stage: 1, host: TargetSelection { triple: "x86_64-unknown-linux-gnu", file: None } } }
        c Sysroot { compiler: Compiler { stage: 1, host: TargetSelection { triple: "x86_64-unknown-linux-gnu", file: None } } }
        c Sysroot { compiler: Compiler { stage: 1, host: TargetSelection { triple: "x86_64-unknown-linux-gnu", file: None } } }
        c Assemble { target_compiler: Compiler { stage: 1, host: TargetSelection { triple: "x86_64-unknown-linux-gnu", file: None } } }
        > StdLink { compiler: Compiler { stage: 1, host: TargetSelection { triple: "x86_64-unknown-linux-gnu", file: None } }, target_compiler: Compiler { stage: 1, host: TargetSelection { triple: "x86_64-unknown-linux-gnu", file: None } }, target: TargetSelection { triple: "x86_64-unknown-linux-gnu", file: None } }
          c Libdir { compiler: Compiler { stage: 1, host: TargetSelection { triple: "x86_64-unknown-linux-gnu", file: None } }, target: TargetSelection { triple: "x86_64-unknown-linux-gnu", file: None } }
          c Libdir { compiler: Compiler { stage: 1, host: TargetSelection { triple: "x86_64-unknown-linux-gnu", file: None } }, target: TargetSelection { triple: "x86_64-unknown-linux-gnu", file: None } }
        < StdLink { compiler: Compiler { stage: 1, host: TargetSelection { triple: "x86_64-unknown-linux-gnu", file: None } }, target_compiler: Compiler { stage: 1, host: TargetSelection { triple: "x86_64-unknown-linux-gnu", file: None } }, target: TargetSelection { triple: "x86_64-unknown-linux-gnu", file: None } }
      < Std { target: TargetSelection { triple: "x86_64-unknown-linux-gnu", file: None }, compiler: Compiler { stage: 1, host: TargetSelection { triple: "x86_64-unknown-linux-gnu", file: None } } }
      c Assemble { target_compiler: Compiler { stage: 1, host: TargetSelection { triple: "x86_64-unknown-linux-gnu", file: None } } }
      c Assemble { target_compiler: Compiler { stage: 1, host: TargetSelection { triple: "x86_64-unknown-linux-gnu", file: None } } }
      c Std { target: TargetSelection { triple: "x86_64-unknown-linux-gnu", file: None }, compiler: Compiler { stage: 1, host: TargetSelection { triple: "x86_64-unknown-linux-gnu", file: None } } }
      c Libdir { compiler: Compiler { stage: 1, host: TargetSelection { triple: "x86_64-unknown-linux-gnu", file: None } }, target: TargetSelection { triple: "x86_64-unknown-linux-gnu", file: None } }
      c Sysroot { compiler: Compiler { stage: 1, host: TargetSelection { triple: "x86_64-unknown-linux-gnu", file: None } } }
      c Sysroot { compiler: Compiler { stage: 1, host: TargetSelection { triple: "x86_64-unknown-linux-gnu", file: None } } }
      c Sysroot { compiler: Compiler { stage: 1, host: TargetSelection { triple: "x86_64-unknown-linux-gnu", file: None } } }
      c Sysroot { compiler: Compiler { stage: 1, host: TargetSelection { triple: "x86_64-unknown-linux-gnu", file: None } } }
      c Sysroot { compiler: Compiler { stage: 1, host: TargetSelection { triple: "x86_64-unknown-linux-gnu", file: None } } }
      > Llvm { target: TargetSelection { triple: "x86_64-unknown-linux-gnu", file: None } }
      < Llvm { target: TargetSelection { triple: "x86_64-unknown-linux-gnu", file: None } }
      c Assemble { target_compiler: Compiler { stage: 1, host: TargetSelection { triple: "x86_64-unknown-linux-gnu", file: None } } }
      > RustcLink { compiler: Compiler { stage: 1, host: TargetSelection { triple: "x86_64-unknown-linux-gnu", file: None } }, target_compiler: Compiler { stage: 1, host: TargetSelection { triple: "x86_64-unknown-linux-gnu", file: None } }, target: TargetSelection { triple: "x86_64-unknown-linux-gnu", file: None } }
        c Libdir { compiler: Compiler { stage: 1, host: TargetSelection { triple: "x86_64-unknown-linux-gnu", file: None } }, target: TargetSelection { triple: "x86_64-unknown-linux-gnu", file: None } }
        c Libdir { compiler: Compiler { stage: 1, host: TargetSelection { triple: "x86_64-unknown-linux-gnu", file: None } }, target: TargetSelection { triple: "x86_64-unknown-linux-gnu", file: None } }
      < RustcLink { compiler: Compiler { stage: 1, host: TargetSelection { triple: "x86_64-unknown-linux-gnu", file: None } }, target_compiler: Compiler { stage: 1, host: TargetSelection { triple: "x86_64-unknown-linux-gnu", file: None } }, target: TargetSelection { triple: "x86_64-unknown-linux-gnu", file: None } }
    < Rustc { target: TargetSelection { triple: "x86_64-unknown-linux-gnu", file: None }, compiler: Compiler { stage: 1, host: TargetSelection { triple: "x86_64-unknown-linux-gnu", file: None } } }
    c Libdir { compiler: Compiler { stage: 1, host: TargetSelection { triple: "x86_64-unknown-linux-gnu", file: None } }, target: TargetSelection { triple: "x86_64-unknown-linux-gnu", file: None } }
    c Sysroot { compiler: Compiler { stage: 1, host: TargetSelection { triple: "x86_64-unknown-linux-gnu", file: None } } }
    c Sysroot { compiler: Compiler { stage: 1, host: TargetSelection { triple: "x86_64-unknown-linux-gnu", file: None } } }
    c Sysroot { compiler: Compiler { stage: 1, host: TargetSelection { triple: "x86_64-unknown-linux-gnu", file: None } } }
    c Sysroot { compiler: Compiler { stage: 1, host: TargetSelection { triple: "x86_64-unknown-linux-gnu", file: None } } }
    c Sysroot { compiler: Compiler { stage: 1, host: TargetSelection { triple: "x86_64-unknown-linux-gnu", file: None } } }
  < ToolBuild { compiler: Compiler { stage: 1, host: TargetSelection { triple: "x86_64-unknown-linux-gnu", file: None } }, target: TargetSelection { triple: "x86_64-unknown-linux-gnu", file: None }, tool: "clippy-driver", path: "src/tools/clippy", mode: ToolRustc, is_optional_tool: true, source_type: InTree, extra_features: [] }
< Clippy { compiler: Compiler { stage: 1, host: TargetSelection { triple: "x86_64-unknown-linux-gnu", file: None } }, target: TargetSelection { triple: "x86_64-unknown-linux-gnu", file: None }, extra_features: [] }
c Assemble { target_compiler: Compiler { stage: 1, host: TargetSelection { triple: "x86_64-unknown-linux-gnu", file: None } } }
> CargoClippy { compiler: Compiler { stage: 1, host: TargetSelection { triple: "x86_64-unknown-linux-gnu", file: None } }, target: TargetSelection { triple: "x86_64-unknown-linux-gnu", file: None }, extra_features: [] }
  > ToolBuild { compiler: Compiler { stage: 1, host: TargetSelection { triple: "x86_64-unknown-linux-gnu", file: None } }, target: TargetSelection { triple: "x86_64-unknown-linux-gnu", file: None }, tool: "cargo-clippy", path: "src/tools/clippy", mode: ToolRustc, is_optional_tool: true, source_type: InTree, extra_features: [] }
    c Rustc { target: TargetSelection { triple: "x86_64-unknown-linux-gnu", file: None }, compiler: Compiler { stage: 1, host: TargetSelection { triple: "x86_64-unknown-linux-gnu", file: None } } }
    c Libdir { compiler: Compiler { stage: 1, host: TargetSelection { triple: "x86_64-unknown-linux-gnu", file: None } }, target: TargetSelection { triple: "x86_64-unknown-linux-gnu", file: None } }
    c Sysroot { compiler: Compiler { stage: 1, host: TargetSelection { triple: "x86_64-unknown-linux-gnu", file: None } } }
    c Sysroot { compiler: Compiler { stage: 1, host: TargetSelection { triple: "x86_64-unknown-linux-gnu", file: None } } }
    c Sysroot { compiler: Compiler { stage: 1, host: TargetSelection { triple: "x86_64-unknown-linux-gnu", file: None } } }
    c Sysroot { compiler: Compiler { stage: 1, host: TargetSelection { triple: "x86_64-unknown-linux-gnu", file: None } } }
    c Sysroot { compiler: Compiler { stage: 1, host: TargetSelection { triple: "x86_64-unknown-linux-gnu", file: None } } }
  < ToolBuild { compiler: Compiler { stage: 1, host: TargetSelection { triple: "x86_64-unknown-linux-gnu", file: None } }, target: TargetSelection { triple: "x86_64-unknown-linux-gnu", file: None }, tool: "cargo-clippy", path: "src/tools/clippy", mode: ToolRustc, is_optional_tool: true, source_type: InTree, extra_features: [] }
< CargoClippy { compiler: Compiler { stage: 1, host: TargetSelection { triple: "x86_64-unknown-linux-gnu", file: None } }, target: TargetSelection { triple: "x86_64-unknown-linux-gnu", file: None }, extra_features: [] }
> Assemble { target_compiler: Compiler { stage: 1, host: TargetSelection { triple: "x86_64-unknown-linux-gnu", file: None } } }
  > Assemble { target_compiler: Compiler { stage: 0, host: TargetSelection { triple: "x86_64-unknown-linux-gnu", file: None } } }
  < Assemble { target_compiler: Compiler { stage: 0, host: TargetSelection { triple: "x86_64-unknown-linux-gnu", file: None } } }
  > Sysroot { compiler: Compiler { stage: 1, host: TargetSelection { triple: "x86_64-unknown-linux-gnu", file: None } } }
  < Sysroot { compiler: Compiler { stage: 1, host: TargetSelection { triple: "x86_64-unknown-linux-gnu", file: None } } }
< Assemble { target_compiler: Compiler { stage: 1, host: TargetSelection { triple: "x86_64-unknown-linux-gnu", file: None } } }
> Clippy { compiler: Compiler { stage: 1, host: TargetSelection { triple: "x86_64-unknown-linux-gnu", file: None } }, target: TargetSelection { triple: "x86_64-unknown-linux-gnu", file: None }, extra_features: [] }
  > ToolBuild { compiler: Compiler { stage: 1, host: TargetSelection { triple: "x86_64-unknown-linux-gnu", file: None } }, target: TargetSelection { triple: "x86_64-unknown-linux-gnu", file: None }, tool: "clippy-driver", path: "src/tools/clippy", mode: ToolRustc, is_optional_tool: true, source_type: InTree, extra_features: [] }
    > Rustc { target: TargetSelection { triple: "x86_64-unknown-linux-gnu", file: None }, compiler: Compiler { stage: 1, host: TargetSelection { triple: "x86_64-unknown-linux-gnu", file: None } } }
      > Std { target: TargetSelection { triple: "x86_64-unknown-linux-gnu", file: None }, compiler: Compiler { stage: 1, host: TargetSelection { triple: "x86_64-unknown-linux-gnu", file: None } } }
        > StartupObjects { compiler: Compiler { stage: 1, host: TargetSelection { triple: "x86_64-unknown-linux-gnu", file: None } }, target: TargetSelection { triple: "x86_64-unknown-linux-gnu", file: None } }
        < StartupObjects { compiler: Compiler { stage: 1, host: TargetSelection { triple: "x86_64-unknown-linux-gnu", file: None } }, target: TargetSelection { triple: "x86_64-unknown-linux-gnu", file: None } }
        c Assemble { target_compiler: Compiler { stage: 1, host: TargetSelection { triple: "x86_64-unknown-linux-gnu", file: None } } }
        > Libdir { compiler: Compiler { stage: 1, host: TargetSelection { triple: "x86_64-unknown-linux-gnu", file: None } }, target: TargetSelection { triple: "x86_64-unknown-linux-gnu", file: None } }
          c Sysroot { compiler: Compiler { stage: 1, host: TargetSelection { triple: "x86_64-unknown-linux-gnu", file: None } } }
        < Libdir { compiler: Compiler { stage: 1, host: TargetSelection { triple: "x86_64-unknown-linux-gnu", file: None } }, target: TargetSelection { triple: "x86_64-unknown-linux-gnu", file: None } }
        c Libdir { compiler: Compiler { stage: 1, host: TargetSelection { triple: "x86_64-unknown-linux-gnu", file: None } }, target: TargetSelection { triple: "x86_64-unknown-linux-gnu", file: None } }
        c Sysroot { compiler: Compiler { stage: 1, host: TargetSelection { triple: "x86_64-unknown-linux-gnu", file: None } } }
        c Sysroot { compiler: Compiler { stage: 1, host: TargetSelection { triple: "x86_64-unknown-linux-gnu", file: None } } }
        c Sysroot { compiler: Compiler { stage: 1, host: TargetSelection { triple: "x86_64-unknown-linux-gnu", file: None } } }
        c Sysroot { compiler: Compiler { stage: 1, host: TargetSelection { triple: "x86_64-unknown-linux-gnu", file: None } } }
Building stage1 std artifacts (x86_64-unknown-linux-gnu -> x86_64-unknown-linux-gnu)
running: "/home/joshua/rustc3/build/x86_64-unknown-linux-gnu/stage0/bin/cargo" "build" "--target" "x86_64-unknown-linux-gnu" "-Zbinary-dep-depinfo" "-j" "8" "--release" "--features" "panic-unwind backtrace compiler-builtins-c" "--manifest-path" "/home/joshua/rustc3/library/test/Cargo.toml" "--message-format" "json-render-diagnostics"
   Compiling core v0.0.0 (/home/joshua/rustc3/library/core)
^C  Building [======>                     ] 9/33: core                              
Build completed unsuccessfully in 0:00:01

@jyn514
Copy link
Member Author

jyn514 commented Feb 24, 2021

lol I also found that this completely ignores all test suites, making it pretty useless:

Not running bootstrap::test::RustdocUi because its artifacts have been downloaded from CI (`download-rustc` is set)

jyn514 added a commit to jyn514/rust that referenced this issue Feb 24, 2021
This was introduced as part of the MVP for `download-rustc`.
Unfortunately, it doesn't work very well:

- Steps are ignored by default, which makes it easy to leave out a step
that should be built. For example, the MVP forgot to enable any tests,
so it was *only* possible to build locally.
- It didn't work correctly even when it was enabled: calling
  `builder.ensure()` would completely ignore the constant and rebuild the
  step anyway. This has no obvious fix since `ensure()` has to return a
  `Step::Output`.

Instead, this handles `download-rustc` in `impl Step for Rustc` and
`impl Step for Std`, which to my knowledge are the only build steps that
don't first go through `impl Step for Sysroot` (`Rustc` is used for
the `rustc-dev` component).

See rust-lang#79540 (comment)
and rust-lang#81930 for further context.

Here are some example runs with these changes and `download-rustc`
enabled:

```
$ x.py build src/tools/clippy
Building stage1 tool clippy-driver (x86_64-unknown-linux-gnu)
    Finished release [optimized] target(s) in 1m 09s
Building stage1 tool cargo-clippy (x86_64-unknown-linux-gnu)
    Finished release [optimized] target(s) in 0.11s
$ x.py test src/tools/clippy
Updating only changed submodules
Submodules updated in 0.01 seconds
    Finished dev [unoptimized + debuginfo] target(s) in 0.09s
Building stage1 tool clippy-driver (x86_64-unknown-linux-gnu)
    Finished release [optimized] target(s) in 0.09s
Building rustdoc for stage1 (x86_64-unknown-linux-gnu)
    Finished release [optimized] target(s) in 0.28s
    Finished release [optimized] target(s) in 15.26s
     Running build/x86_64-unknown-linux-gnu/stage1-tools/x86_64-unknown-linux-gnu/release/deps/clippy_driver-8b407b140e0aa91c
test result: ok. 592 passed; 0 failed; 3 ignored; 0 measured; 0 filtered out
$ x.py build src/tools/rustdoc
Building rustdoc for stage1 (x86_64-unknown-linux-gnu)
    Finished release [optimized] target(s) in 41.28s
Build completed successfully in 0:00:41
$ x.py test src/test/rustdoc-ui
Building stage0 tool compiletest (x86_64-unknown-linux-gnu)
    Finished release [optimized] target(s) in 0.12s
Building rustdoc for stage1 (x86_64-unknown-linux-gnu)
    Finished release [optimized] target(s) in 0.10s
test result: ok. 105 passed; 0 failed; 0 ignored; 0 measured; 0 filtered out; finished in 8.15s
$ x.py build compiler/rustc
    Finished dev [unoptimized + debuginfo] target(s) in 0.09s
Build completed successfully in 0:00:00
```

Note a few things:

- Clippy depends on stage1 rustc-dev artifacts, but rustc didn't have to
  be recompiled. Instead, the artifacts were copied automatically.
- All steps are always enabled. There is no danger of forgetting a step,
  since only the entrypoints have to handle `download-rustc`.
- Building the compiler (`compiler/rustc`) automatically does no work.
@camelid
Copy link
Member

camelid commented Feb 25, 2021

Required before this is made a default

I don't think this should be enabled by default until this is addressed as well so that it won't use the CI rustc if you've made local changes to rustc. Otherwise it would be really frustrating for people making changes to rustc and rustdoc (et al.).

@jyn514
Copy link
Member Author

jyn514 commented Feb 25, 2021

@camelid that's why it has a warning. Do you think it should force you to disable the option? That seems like it could be frustrating.

@jyn514
Copy link
Member Author

jyn514 commented Feb 25, 2021

I definitely don't think it should silently use the in-tree compiler if you make changes, it should always do what you specify in config.toml.

@camelid
Copy link
Member

camelid commented Feb 25, 2021

I was thinking that there could be a download-rustc = "if-no-local-changes" option or something like that, where it would only download rustc if compiler/ isn't dirty.

@jyn514
Copy link
Member Author

jyn514 commented Feb 25, 2021

Ah ok, that seems useful. I don't think it needs to block turning this on by default, though.

@camelid
Copy link
Member

camelid commented Feb 25, 2021

Ah ok, that seems useful. I don't think it needs to block turning this on by default, though.

Won't this be annoying to new rustc contributors who have to dig around in their settings?

@jyn514
Copy link
Member Author

jyn514 commented Feb 25, 2021

@camelid I found the confusion. When I say "by-default", I mean that I want to add a new x.py setup tools option that's specifically intended for rustdoc/clippy/rustfmt contributors and has download-rustc = true. I don't want to turn it on for the whole repository, that wouldn't make sense (or at least, not until if-no-changes is implemented).

@camelid
Copy link
Member

camelid commented Feb 25, 2021

Ah, I had a feeling we were talking past each other :)

Sounds good to me, as long as the warning is helpful enough that if people start doing compiler contributions they don't get stuck.

jyn514 pushed a commit to jyn514/rust that referenced this issue Mar 1, 2021
…imulacrum

Remove `ENABLE_DOWNLOAD_RUSTC` constant

`ENABLE_DOWNLOAD_RUSTC` was introduced as part of the MVP for `download-rustc` as a way not to rebuild artifacts that have already been downloaded. Unfortunately, it doesn't work very well:

- Steps are ignored by default, which makes it easy to leave out a step
that should be built. For example, the MVP forgot to enable any tests,
so it was only possible to *build* locally.
- It didn't work correctly even when it was enabled: calling
  `builder.ensure()` would completely ignore the constant and rebuild the
  step anyway. This has no obvious fix since `ensure()` has to return a
  `Step::Output`.

Instead, this handles `download-rustc` in `impl Step for Rustc` and
`impl Step for Std`, which to my knowledge are the only build steps that
don't first go through `impl Step for Sysroot` (`Rustc` is used for
the `rustc-dev` component).

See rust-lang#79540 (comment) and rust-lang#81930 for further context.

Here are some example runs with these changes and `download-rustc`
enabled:

```
$ x.py build src/tools/clippy
Building stage1 tool clippy-driver (x86_64-unknown-linux-gnu)
    Finished release [optimized] target(s) in 1m 09s
Building stage1 tool cargo-clippy (x86_64-unknown-linux-gnu)
    Finished release [optimized] target(s) in 0.11s
$ x.py test src/tools/clippy
    Finished dev [unoptimized + debuginfo] target(s) in 0.09s
Building stage1 tool clippy-driver (x86_64-unknown-linux-gnu)
    Finished release [optimized] target(s) in 0.09s
Building rustdoc for stage1 (x86_64-unknown-linux-gnu)
    Finished release [optimized] target(s) in 0.28s
    Finished release [optimized] target(s) in 15.26s
     Running build/x86_64-unknown-linux-gnu/stage1-tools/x86_64-unknown-linux-gnu/release/deps/clippy_driver-8b407b140e0aa91c
test result: ok. 592 passed; 0 failed; 3 ignored; 0 measured; 0 filtered out
$ x.py build src/tools/rustdoc
Building rustdoc for stage1 (x86_64-unknown-linux-gnu)
    Finished release [optimized] target(s) in 41.28s
Build completed successfully in 0:00:41
$ x.py test src/test/rustdoc-ui
Building stage0 tool compiletest (x86_64-unknown-linux-gnu)
    Finished release [optimized] target(s) in 0.12s
Building rustdoc for stage1 (x86_64-unknown-linux-gnu)
    Finished release [optimized] target(s) in 0.10s
test result: ok. 105 passed; 0 failed; 0 ignored; 0 measured; 0 filtered out; finished in 8.15s
$ x.py build compiler/rustc
    Finished dev [unoptimized + debuginfo] target(s) in 0.09s
Build completed successfully in 0:00:00
```

Note a few things:

- Clippy depends on stage1 rustc-dev artifacts, but rustc didn't have to
  be recompiled. Instead, the artifacts were copied automatically.
- All steps are always enabled. There is no danger of forgetting a step,
  since only the entrypoints have to handle `download-rustc`.
- Building the compiler (`compiler/rustc`) automatically does no work.

Helps with rust-lang#81930.

r? `@Mark-Simulacrum`
GuillaumeGomez added a commit to GuillaumeGomez/rust that referenced this issue Mar 1, 2021
…lacrum

Always compile rustdoc with debug logging enabled when `download-rustc` is set

Previously, logging at DEBUG or below would always be silenced, because
rustc compiles tracing with the `static_max_level_info` feature. That
makes sense for release artifacts, but not for developing rustdoc.

Instead, this compiles two different versions of tracing: one in the
release artifacts, distributed in the sysroot, and a new version
compiled by rustdoc. Since `rustc_driver` is always linked to the
version of sysroot, this copy/pastes `init_env_logging` into rustdoc.

To avoid compiling an unnecessary version of tracing when
`download-rustc` isn't set, this adds a new `using-ci-artifacts`
feature for rustdoc and passes that feature in bootstrap.

Addresses rust-lang#81930. This builds on rust-lang#79540.

r? `@Mark-Simulacrum`
@jyn514
Copy link
Member Author

jyn514 commented Apr 5, 2021

It would be nice to support cross-compiling the standard library. Right now this gives an assertion failure I think.

Err it looks like x.py build library/std always does nothing. So --target is also ignored.

Dylan-DPC-zz pushed a commit to Dylan-DPC-zz/rust that referenced this issue Apr 5, 2021
Add `x.py setup tools` which enables `download-rustc` by default

Helps with rust-lang#81930. I know I said in that issue that I should fix that rebasing rebuilds bootstrap, but the compile time improvement is so good I think it's ok to leave that fix for later (I still plan to work on it). I think all the outright bugs have been fixed :)

This builds on rust-lang#83368 so I can set the option to `if-unchanged`.

r? `@Mark-Simulacrum`
Dylan-DPC-zz pushed a commit to Dylan-DPC-zz/rust that referenced this issue Apr 5, 2021
Add `x.py setup tools` which enables `download-rustc` by default

Helps with rust-lang#81930. I know I said in that issue that I should fix that rebasing rebuilds bootstrap, but the compile time improvement is so good I think it's ok to leave that fix for later (I still plan to work on it). I think all the outright bugs have been fixed :)

This builds on rust-lang#83368 so I can set the option to `if-unchanged`.

r? ``@Mark-Simulacrum``
JohnTitor added a commit to JohnTitor/rust that referenced this issue Apr 5, 2021
Add `x.py setup tools` which enables `download-rustc` by default

Helps with rust-lang#81930. I know I said in that issue that I should fix that rebasing rebuilds bootstrap, but the compile time improvement is so good I think it's ok to leave that fix for later (I still plan to work on it). I think all the outright bugs have been fixed :)

This builds on rust-lang#83368 so I can set the option to `if-unchanged`.

r? ```@Mark-Simulacrum```
@jyn514
Copy link
Member Author

jyn514 commented Apr 6, 2021

Oh huh, this is an unexpected side-effect: since #82739, building compiletest also requires building stage0 std first. I think using master libstd is intentional since #59264? I don't think it necessarily needs to change, maybe just documented somewhere?

Anyway, that fact that just works without build errors makes me a lot more confident we're choosing the right abstractions in bootstrap :)

@camelid
Copy link
Member

camelid commented Apr 11, 2021

Reposting from this Zulip discussion as requested by @jyn514:

I just ran into something confusing with download-rustc: normal x.py build will build rustdoc and put it in stage1. However, x.py build with download-rustc enabled will put it in stage2. This means that rustdoc +stage1 ... won't run my local, modified version of rustdoc if download-rustc is set. Specifically, rustdoc +stage1 ... seems to run nightly rustdoc (or something static).

I'm guessing this is because rustdoc built with download-rustc is being built with stage2 rustc, thus it is stage2 rustdoc, whereas rustdoc built locally without download-rustc is usually built with stage1 rustc, thus it is stage1 rustdoc. However, this behavior was very confusing to me and it took me a bunch of experimenting to figure out why my changes weren't showing up in the rustdoc output.

I imagine it could be even more confusing for new contributors since we recommend using stage1 builds, but then when they use download-rustc they have to remember that it's actually a stage2 build and adjust their commands accordingly. Plus, it's frustrating even if you know what's going on to switch between stage1 and stage2 depending on whether you're working on rustc or rustdoc (or a different tool).

Is there any way to solve this problem? I imagine there are multiple possible solutions, some better than others.

@ehuss ehuss removed the T-dev-tools Relevant to the dev-tools subteam, which will review and decide on the PR/issue. label Jan 18, 2022
@jyn514 jyn514 removed their assignment Feb 15, 2022
@jyn514
Copy link
Member Author

jyn514 commented Sep 13, 2022

I just ran into something confusing with download-rustc: normal x.py build will build rustdoc and put it in stage1. However, x.py build with download-rustc enabled will put it in stage2. This means that rustdoc +stage1 ... won't run my local, modified version of rustdoc if download-rustc is set. Specifically, rustdoc +stage1 ... seems to run nightly rustdoc (or something static).

@bjorn3 suggests deleting the stage1 and stage0-sysroot directories when using download-rustc, to avoid caching issues like this. rustdoc +stage1 still won't do the right thing, but it will at least give a hard error instead of silently using the wrong version.

@camelid
Copy link
Member

camelid commented Sep 14, 2022

Great idea! That would be very helpful.

@jyn514
Copy link
Member Author

jyn514 commented Oct 7, 2022

I just ran into something confusing with download-rustc: normal x.py build will build rustdoc and put it in stage1. However, x.py build with download-rustc enabled will put it in stage2. This means that rustdoc +stage1 ... won't run my local, modified version of rustdoc if download-rustc is set. Specifically, rustdoc +stage1 ... seems to run nightly rustdoc (or something static).

bjorn3 suggests deleting the stage1 and stage0-sysroot directories when using download-rustc, to avoid caching issues like this. rustdoc +stage1 still won't do the right thing, but it will at least give a hard error instead of silently using the wrong version.

Done in 973ff03

@jyn514
Copy link
Member Author

jyn514 commented Apr 9, 2023

Give a better error message when rebasing an old branch (I think commits are kept at most 160 days?). #79540 (comment)

I'm going to cross this off the list - right now download-ci-llvm gives a useful error first and I think it's very rare to want to download rustc but not LLVM.

downloading https://ci-artifacts.rust-lang.org/rustc-builds/8ce3204af9463db3192ea1eb31c45c2f6d4b5ae6/rust-dev-nightly-x86_64-unknown-linux-gnu.tar.xz
curl: (22) The requested URL returned error: 404                                                                                

error: failed to download llvm from ci

help: old builds get deleted after a certain time
help: if trying to compile an old commit of rustc, disable `download-ci-llvm` in config.toml:

[llvm]
download-ci-llvm = false

Dylan-DPC added a commit to Dylan-DPC/rust that referenced this issue Apr 10, 2023
Fix `x check --stage 1` when download-rustc is enabled

Helps with rust-lang#81930
JohnTitor added a commit to JohnTitor/rust that referenced this issue Apr 11, 2023
…ertlarsan68

Fix `x test ui --target foo` when download-rustc is enabled

Previously, we would never build the target std, only the host std:

```
; x t tests/ui/attributes --target wasm32-unknown-unknown
Building bootstrap
    Finished dev [unoptimized] target(s) in 0.02s
Building stage0 library artifacts (x86_64-unknown-linux-gnu)
    Finished release [optimized] target(s) in 0.08s
Building tool compiletest (stage0)
    Finished release [optimized] target(s) in 0.09s
Check compiletest suite=ui mode=ui (x86_64-unknown-linux-gnu -> wasm32-unknown-unknown)
thread 'main' panicked at 'called `Result::unwrap()` on an `Err` value: Error { depth: 0, inner: Io { path: Some("/home/jyn/src/rust2/build/x86_64-unknown-linux-gnu/stage2/lib/rustlib/wasm32-unknown-unknown/lib"), err: Os { code: 2, kind: NotFound, message: "No such file or directory" } } }', src/tools/compiletest/src/main.rs:842:31
```

Helps with rust-lang#81930.
Dylan-DPC added a commit to Dylan-DPC/rust that referenced this issue Apr 11, 2023
…ertlarsan68

Fix `x test ui --target foo` when download-rustc is enabled

Previously, we would never build the target std, only the host std:

```
; x t tests/ui/attributes --target wasm32-unknown-unknown
Building bootstrap
    Finished dev [unoptimized] target(s) in 0.02s
Building stage0 library artifacts (x86_64-unknown-linux-gnu)
    Finished release [optimized] target(s) in 0.08s
Building tool compiletest (stage0)
    Finished release [optimized] target(s) in 0.09s
Check compiletest suite=ui mode=ui (x86_64-unknown-linux-gnu -> wasm32-unknown-unknown)
thread 'main' panicked at 'called `Result::unwrap()` on an `Err` value: Error { depth: 0, inner: Io { path: Some("/home/jyn/src/rust2/build/x86_64-unknown-linux-gnu/stage2/lib/rustlib/wasm32-unknown-unknown/lib"), err: Os { code: 2, kind: NotFound, message: "No such file or directory" } } }', src/tools/compiletest/src/main.rs:842:31
```

Helps with rust-lang#81930.
compiler-errors added a commit to compiler-errors/rust that referenced this issue Apr 11, 2023
…ertlarsan68

Fix `x test ui --target foo` when download-rustc is enabled

Previously, we would never build the target std, only the host std:

```
; x t tests/ui/attributes --target wasm32-unknown-unknown
Building bootstrap
    Finished dev [unoptimized] target(s) in 0.02s
Building stage0 library artifacts (x86_64-unknown-linux-gnu)
    Finished release [optimized] target(s) in 0.08s
Building tool compiletest (stage0)
    Finished release [optimized] target(s) in 0.09s
Check compiletest suite=ui mode=ui (x86_64-unknown-linux-gnu -> wasm32-unknown-unknown)
thread 'main' panicked at 'called `Result::unwrap()` on an `Err` value: Error { depth: 0, inner: Io { path: Some("/home/jyn/src/rust2/build/x86_64-unknown-linux-gnu/stage2/lib/rustlib/wasm32-unknown-unknown/lib"), err: Os { code: 2, kind: NotFound, message: "No such file or directory" } } }', src/tools/compiletest/src/main.rs:842:31
```

Helps with rust-lang#81930.
compiler-errors added a commit to compiler-errors/rust that referenced this issue Apr 12, 2023
…ertlarsan68

Fix `x test ui --target foo` when download-rustc is enabled

Previously, we would never build the target std, only the host std:

```
; x t tests/ui/attributes --target wasm32-unknown-unknown
Building bootstrap
    Finished dev [unoptimized] target(s) in 0.02s
Building stage0 library artifacts (x86_64-unknown-linux-gnu)
    Finished release [optimized] target(s) in 0.08s
Building tool compiletest (stage0)
    Finished release [optimized] target(s) in 0.09s
Check compiletest suite=ui mode=ui (x86_64-unknown-linux-gnu -> wasm32-unknown-unknown)
thread 'main' panicked at 'called `Result::unwrap()` on an `Err` value: Error { depth: 0, inner: Io { path: Some("/home/jyn/src/rust2/build/x86_64-unknown-linux-gnu/stage2/lib/rustlib/wasm32-unknown-unknown/lib"), err: Os { code: 2, kind: NotFound, message: "No such file or directory" } } }', src/tools/compiletest/src/main.rs:842:31
```

Helps with rust-lang#81930.
@jyn514 jyn514 added the A-download-rustc Area: The `rust.download-rustc` build option. label Apr 12, 2023
matthiaskrgr added a commit to matthiaskrgr/rust that referenced this issue Apr 30, 2023
…lacrum

download-rustc: Give a better error message if artifacts can't be dowloaded

It should be very rare in practice to happen; people would need to both have `download-ci-llvm` disabled and `download-rustc` enabled. I think it may be more common if we start turning this on by default, though.

Helps with rust-lang#81930.

Before:
```
downloading https://ci-artifacts.rust-lang.org/rustc-builds/bf5cad8e775fb326465e5c1b98693e5d259da156/rust-std-nightly-x86_64-unknown-linux-gnu.tar.xz
curl: (22) The requested URL returned error: 404
```

After:
```
downloading https://ci-artifacts.rust-lang.org/rustc-builds/bf5cad8e775fb326465e5c1b98693e5d259da156/rust-std-nightly-x86_64-unknown-linux-gnu.tar.xz
curl: (22) The requested URL returned error: 404

error: failed to download pre-built rustc from CI

note: old builds get deleted after a certain time
help: if trying to compile an old commit of rustc, disable `download-rustc` in config.toml:

[rust]
download-rustc = false
```
@jyn514
Copy link
Member Author

jyn514 commented Jun 6, 2023

Some tests fail when download-rustc is enabled: https://github.com/rust-lang/rust/issues?q=is%3Aissue+is%3Aopen+label%3AA-testsuite+label%3AA-download-rustc

this is now fixed well enough that x test by default succeeds :) x test --stage 1 rustdoc is still broken, but that should be extremely rare to need to run, and I don't think we need a whole tracking issue for it.

@jyn514 jyn514 closed this as completed Jun 6, 2023
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
A-contributor-roadblock Area: Makes things more difficult for new contributors to rust itself A-download-rustc Area: The `rust.download-rustc` build option. C-tracking-issue Category: A tracking issue for an RFC or an unstable feature. T-bootstrap Relevant to the bootstrap subteam: Rust's build system (x.py and src/bootstrap)
Projects
None yet
Development

No branches or pull requests

3 participants