Follow-up work for `download-rustc` #81930

jyn514 · 2021-02-09T16:38:34Z

These are all the features for download-rustc that didn't land in the initial MVP (#79540).

Required before this is made a default

Download a separate beta compiler for stage0 and only use CI artifacts for stage1 onward. Right now, every time you rebase, stage0 tools (bootstrap, tidy, ...) will have to be recompiled. Additionally running x.py setup tools will compile rustbuild twice. Allow building rustdoc without first building rustc (MVP) #79540 (comment), Use the beta compiler for building bootstrap tools when download-rustc is set #82739
Compile a new version of tracing so that rustdoc still gets debug logging (since CI artifacts always disable debug and trace logging). Always compile rustdoc with debug logging enabled when download-rustc is set #81932
Right now only rustdoc is ever rebuilt. This is not ideal and should probably at least compile compiler tools (rustfmt, clippy, miri). Allow building rustdoc without first building rustc (MVP) #79540 (comment), Remove ENABLE_DOWNLOAD_RUSTC constant #82480
Using git log --author=bors sometimes breaks. This should use git merge-base instead, or use some other workaround that doesn't break. Allow building rustdoc without first building rustc (MVP) #79540 (comment), Fix commit detected when using download-rustc #82740
This breaks during release week (but only release week). Download a more recent LLVM version if src/version is modified #83350

Nice to have

Add download-rustc = "if-unchanged" which rebuilds rustc instead of warning when compiler/ has been modified. Follow-up work for download-rustc #81930 (comment), Add download-rustc = "if-unchanged" #83368
x.py check --stage 0 doesn't work. Follow-up work for download-rustc #81930 (comment), Use the beta compiler for building bootstrap tools when download-rustc is set #82739
x.py check --stage 1 src/tools/rustdoc doesn't work. Follow-up work for download-rustc #81930 (comment), fixed in Fix x check --stage 1 when download-rustc is enabled #110121
Add x.py setup tools to enable this conveniently (it doesn't make sense to use this for compiler developers). This shouldn't be added until people have used this feature for a little bit and it works reasonably well. Add x.py setup tools which enables download-rustc by default #83370
Enable verification for the commit tarballs. This requires first signing nightly commits; right now the .sha256 URLs gave a 404. This seems fairly simple to fix but also not high priority.
Give a better error message when rebasing an old branch (I think commits are kept at most 160 days?). Allow building rustdoc without first building rustc (MVP) #79540 (comment)
It would be nice to support cross-compiling the standard library - fixed in Fix x test ui --target foo when download-rustc is enabled #110113
Some tests fail when download-rustc is enabled: https://github.com/rust-lang/rust/issues?q=is%3Aissue+is%3Aopen+label%3AA-testsuite+label%3AA-download-rustc
Combine download-rustc and download-ci-llvm into the same option. This needs quite a bit more design work. Allow building rustdoc without first building rustc (MVP) #79540 (comment), Unify download-* settings in config.toml #83369 This doesn't seem useful.

The text was updated successfully, but these errors were encountered:

jyn514 · 2021-02-24T06:09:38Z

Right now only rustdoc is ever rebuilt. This is not ideal and should probably at least compile compiler tools (rustfmt, clippy, miri). #79540 (comment)

While working on this, I found that the handling of ENABLE_DOWNLOAD_STAGE0 is broken: if a Step is called as part of ensure() instead of through maybe_run, it will still be built even if the constant is set to false. Not yet sure how to fix this.

Note that this has local changes enabling the constant for clippy:

diff --git a/src/bootstrap/tool.rs b/src/bootstrap/tool.rs
index 5c874f69bd9..c82b78c1c71 100644
--- a/src/bootstrap/tool.rs
+++ b/src/bootstrap/tool.rs
@@ -564,6 +564,7 @@ pub struct Cargo {
 impl Step for Cargo {
     type Output = PathBuf;
     const DEFAULT: bool = true;
+    const ENABLE_DOWNLOAD_RUSTC: bool = true;
     const ONLY_HOSTS: bool = true;
 
     fn should_run(run: ShouldRun<'_>) -> ShouldRun<'_> {
@@ -647,6 +648,7 @@ pub struct $name {
         impl Step for $name {
             type Output = Option<PathBuf>;
             const DEFAULT: bool = true; // Overwritten below
+            const ENABLE_DOWNLOAD_RUSTC: bool = true;
             const ONLY_HOSTS: bool = true;
 
             fn should_run(run: ShouldRun<'_>) -> ShouldRun<'_> {

$ x.py build -v src/tools/clippy
detected default triple x86_64-unknown-linux-gnu
Updating only changed submodules
Submodules updated in 0.01 seconds
using downloaded stage1 artifacts from CI (commit fe1bf8e05c39bdcc73fc09e246b7209444e389bc)
running: /home/joshua/rustc3/build/x86_64-unknown-linux-gnu/stage0/bin/cargo build --manifest-path /home/joshua/rustc3/src/bootstrap/Cargo.toml
    Finished dev [unoptimized + debuginfo] target(s) in 0.09s
running: /home/joshua/rustc3/build/bootstrap/debug/bootstrap build -v src/tools/clippy
finding compilers
CC_x86_64-unknown-linux-gnu = "cc"
CFLAGS_x86_64-unknown-linux-gnu = ["-ffunction-sections", "-fdata-sections", "-fPIC", "-m64"]
CXX_x86_64-unknown-linux-gnu = "c++"
CXXFLAGS_x86_64-unknown-linux-gnu = ["-ffunction-sections", "-fdata-sections", "-fPIC", "-m64"]
AR_x86_64-unknown-linux-gnu = "ar"
running sanity check
auto-detected local-rebuild 1.52.0-nightly
learning about cargo
> Assemble { target_compiler: Compiler { stage: 1, host: TargetSelection { triple: "x86_64-unknown-linux-gnu", file: None } } }
  > Assemble { target_compiler: Compiler { stage: 0, host: TargetSelection { triple: "x86_64-unknown-linux-gnu", file: None } } }
  < Assemble { target_compiler: Compiler { stage: 0, host: TargetSelection { triple: "x86_64-unknown-linux-gnu", file: None } } }
  > Sysroot { compiler: Compiler { stage: 1, host: TargetSelection { triple: "x86_64-unknown-linux-gnu", file: None } } }
  < Sysroot { compiler: Compiler { stage: 1, host: TargetSelection { triple: "x86_64-unknown-linux-gnu", file: None } } }
< Assemble { target_compiler: Compiler { stage: 1, host: TargetSelection { triple: "x86_64-unknown-linux-gnu", file: None } } }
> Clippy { compiler: Compiler { stage: 1, host: TargetSelection { triple: "x86_64-unknown-linux-gnu", file: None } }, target: TargetSelection { triple: "x86_64-unknown-linux-gnu", file: None }, extra_features: [] }
  > ToolBuild { compiler: Compiler { stage: 1, host: TargetSelection { triple: "x86_64-unknown-linux-gnu", file: None } }, target: TargetSelection { triple: "x86_64-unknown-linux-gnu", file: None }, tool: "clippy-driver", path: "src/tools/clippy", mode: ToolRustc, is_optional_tool: true, source_type: InTree, extra_features: [] }
    > Rustc { target: TargetSelection { triple: "x86_64-unknown-linux-gnu", file: None }, compiler: Compiler { stage: 1, host: TargetSelection { triple: "x86_64-unknown-linux-gnu", file: None } } }
      > Std { target: TargetSelection { triple: "x86_64-unknown-linux-gnu", file: None }, compiler: Compiler { stage: 1, host: TargetSelection { triple: "x86_64-unknown-linux-gnu", file: None } } }
        > StartupObjects { compiler: Compiler { stage: 1, host: TargetSelection { triple: "x86_64-unknown-linux-gnu", file: None } }, target: TargetSelection { triple: "x86_64-unknown-linux-gnu", file: None } }
        < StartupObjects { compiler: Compiler { stage: 1, host: TargetSelection { triple: "x86_64-unknown-linux-gnu", file: None } }, target: TargetSelection { triple: "x86_64-unknown-linux-gnu", file: None } }
        c Assemble { target_compiler: Compiler { stage: 1, host: TargetSelection { triple: "x86_64-unknown-linux-gnu", file: None } } }
        > Libdir { compiler: Compiler { stage: 1, host: TargetSelection { triple: "x86_64-unknown-linux-gnu", file: None } }, target: TargetSelection { triple: "x86_64-unknown-linux-gnu", file: None } }
          c Sysroot { compiler: Compiler { stage: 1, host: TargetSelection { triple: "x86_64-unknown-linux-gnu", file: None } } }
        < Libdir { compiler: Compiler { stage: 1, host: TargetSelection { triple: "x86_64-unknown-linux-gnu", file: None } }, target: TargetSelection { triple: "x86_64-unknown-linux-gnu", file: None } }
        c Libdir { compiler: Compiler { stage: 1, host: TargetSelection { triple: "x86_64-unknown-linux-gnu", file: None } }, target: TargetSelection { triple: "x86_64-unknown-linux-gnu", file: None } }
        c Sysroot { compiler: Compiler { stage: 1, host: TargetSelection { triple: "x86_64-unknown-linux-gnu", file: None } } }
        c Sysroot { compiler: Compiler { stage: 1, host: TargetSelection { triple: "x86_64-unknown-linux-gnu", file: None } } }
        c Sysroot { compiler: Compiler { stage: 1, host: TargetSelection { triple: "x86_64-unknown-linux-gnu", file: None } } }
        c Assemble { target_compiler: Compiler { stage: 1, host: TargetSelection { triple: "x86_64-unknown-linux-gnu", file: None } } }
        > StdLink { compiler: Compiler { stage: 1, host: TargetSelection { triple: "x86_64-unknown-linux-gnu", file: None } }, target_compiler: Compiler { stage: 1, host: TargetSelection { triple: "x86_64-unknown-linux-gnu", file: None } }, target: TargetSelection { triple: "x86_64-unknown-linux-gnu", file: None } }
          c Libdir { compiler: Compiler { stage: 1, host: TargetSelection { triple: "x86_64-unknown-linux-gnu", file: None } }, target: TargetSelection { triple: "x86_64-unknown-linux-gnu", file: None } }
          c Libdir { compiler: Compiler { stage: 1, host: TargetSelection { triple: "x86_64-unknown-linux-gnu", file: None } }, target: TargetSelection { triple: "x86_64-unknown-linux-gnu", file: None } }
        < StdLink { compiler: Compiler { stage: 1, host: TargetSelection { triple: "x86_64-unknown-linux-gnu", file: None } }, target_compiler: Compiler { stage: 1, host: TargetSelection { triple: "x86_64-unknown-linux-gnu", file: None } }, target: TargetSelection { triple: "x86_64-unknown-linux-gnu", file: None } }
      < Std { target: TargetSelection { triple: "x86_64-unknown-linux-gnu", file: None }, compiler: Compiler { stage: 1, host: TargetSelection { triple: "x86_64-unknown-linux-gnu", file: None } } }
      c Assemble { target_compiler: Compiler { stage: 1, host: TargetSelection { triple: "x86_64-unknown-linux-gnu", file: None } } }
      c Assemble { target_compiler: Compiler { stage: 1, host: TargetSelection { triple: "x86_64-unknown-linux-gnu", file: None } } }
      c Std { target: TargetSelection { triple: "x86_64-unknown-linux-gnu", file: None }, compiler: Compiler { stage: 1, host: TargetSelection { triple: "x86_64-unknown-linux-gnu", file: None } } }
      c Libdir { compiler: Compiler { stage: 1, host: TargetSelection { triple: "x86_64-unknown-linux-gnu", file: None } }, target: TargetSelection { triple: "x86_64-unknown-linux-gnu", file: None } }
      c Sysroot { compiler: Compiler { stage: 1, host: TargetSelection { triple: "x86_64-unknown-linux-gnu", file: None } } }
      c Sysroot { compiler: Compiler { stage: 1, host: TargetSelection { triple: "x86_64-unknown-linux-gnu", file: None } } }
      c Sysroot { compiler: Compiler { stage: 1, host: TargetSelection { triple: "x86_64-unknown-linux-gnu", file: None } } }
      c Sysroot { compiler: Compiler { stage: 1, host: TargetSelection { triple: "x86_64-unknown-linux-gnu", file: None } } }
      c Sysroot { compiler: Compiler { stage: 1, host: TargetSelection { triple: "x86_64-unknown-linux-gnu", file: None } } }
      > Llvm { target: TargetSelection { triple: "x86_64-unknown-linux-gnu", file: None } }
      < Llvm { target: TargetSelection { triple: "x86_64-unknown-linux-gnu", file: None } }
      c Assemble { target_compiler: Compiler { stage: 1, host: TargetSelection { triple: "x86_64-unknown-linux-gnu", file: None } } }
      > RustcLink { compiler: Compiler { stage: 1, host: TargetSelection { triple: "x86_64-unknown-linux-gnu", file: None } }, target_compiler: Compiler { stage: 1, host: TargetSelection { triple: "x86_64-unknown-linux-gnu", file: None } }, target: TargetSelection { triple: "x86_64-unknown-linux-gnu", file: None } }
        c Libdir { compiler: Compiler { stage: 1, host: TargetSelection { triple: "x86_64-unknown-linux-gnu", file: None } }, target: TargetSelection { triple: "x86_64-unknown-linux-gnu", file: None } }
        c Libdir { compiler: Compiler { stage: 1, host: TargetSelection { triple: "x86_64-unknown-linux-gnu", file: None } }, target: TargetSelection { triple: "x86_64-unknown-linux-gnu", file: None } }
      < RustcLink { compiler: Compiler { stage: 1, host: TargetSelection { triple: "x86_64-unknown-linux-gnu", file: None } }, target_compiler: Compiler { stage: 1, host: TargetSelection { triple: "x86_64-unknown-linux-gnu", file: None } }, target: TargetSelection { triple: "x86_64-unknown-linux-gnu", file: None } }
    < Rustc { target: TargetSelection { triple: "x86_64-unknown-linux-gnu", file: None }, compiler: Compiler { stage: 1, host: TargetSelection { triple: "x86_64-unknown-linux-gnu", file: None } } }
    c Libdir { compiler: Compiler { stage: 1, host: TargetSelection { triple: "x86_64-unknown-linux-gnu", file: None } }, target: TargetSelection { triple: "x86_64-unknown-linux-gnu", file: None } }
    c Sysroot { compiler: Compiler { stage: 1, host: TargetSelection { triple: "x86_64-unknown-linux-gnu", file: None } } }
    c Sysroot { compiler: Compiler { stage: 1, host: TargetSelection { triple: "x86_64-unknown-linux-gnu", file: None } } }
    c Sysroot { compiler: Compiler { stage: 1, host: TargetSelection { triple: "x86_64-unknown-linux-gnu", file: None } } }
    c Sysroot { compiler: Compiler { stage: 1, host: TargetSelection { triple: "x86_64-unknown-linux-gnu", file: None } } }
    c Sysroot { compiler: Compiler { stage: 1, host: TargetSelection { triple: "x86_64-unknown-linux-gnu", file: None } } }
  < ToolBuild { compiler: Compiler { stage: 1, host: TargetSelection { triple: "x86_64-unknown-linux-gnu", file: None } }, target: TargetSelection { triple: "x86_64-unknown-linux-gnu", file: None }, tool: "clippy-driver", path: "src/tools/clippy", mode: ToolRustc, is_optional_tool: true, source_type: InTree, extra_features: [] }
< Clippy { compiler: Compiler { stage: 1, host: TargetSelection { triple: "x86_64-unknown-linux-gnu", file: None } }, target: TargetSelection { triple: "x86_64-unknown-linux-gnu", file: None }, extra_features: [] }
c Assemble { target_compiler: Compiler { stage: 1, host: TargetSelection { triple: "x86_64-unknown-linux-gnu", file: None } } }
> CargoClippy { compiler: Compiler { stage: 1, host: TargetSelection { triple: "x86_64-unknown-linux-gnu", file: None } }, target: TargetSelection { triple: "x86_64-unknown-linux-gnu", file: None }, extra_features: [] }
  > ToolBuild { compiler: Compiler { stage: 1, host: TargetSelection { triple: "x86_64-unknown-linux-gnu", file: None } }, target: TargetSelection { triple: "x86_64-unknown-linux-gnu", file: None }, tool: "cargo-clippy", path: "src/tools/clippy", mode: ToolRustc, is_optional_tool: true, source_type: InTree, extra_features: [] }
    c Rustc { target: TargetSelection { triple: "x86_64-unknown-linux-gnu", file: None }, compiler: Compiler { stage: 1, host: TargetSelection { triple: "x86_64-unknown-linux-gnu", file: None } } }
    c Libdir { compiler: Compiler { stage: 1, host: TargetSelection { triple: "x86_64-unknown-linux-gnu", file: None } }, target: TargetSelection { triple: "x86_64-unknown-linux-gnu", file: None } }
    c Sysroot { compiler: Compiler { stage: 1, host: TargetSelection { triple: "x86_64-unknown-linux-gnu", file: None } } }
    c Sysroot { compiler: Compiler { stage: 1, host: TargetSelection { triple: "x86_64-unknown-linux-gnu", file: None } } }
    c Sysroot { compiler: Compiler { stage: 1, host: TargetSelection { triple: "x86_64-unknown-linux-gnu", file: None } } }
    c Sysroot { compiler: Compiler { stage: 1, host: TargetSelection { triple: "x86_64-unknown-linux-gnu", file: None } } }
    c Sysroot { compiler: Compiler { stage: 1, host: TargetSelection { triple: "x86_64-unknown-linux-gnu", file: None } } }
  < ToolBuild { compiler: Compiler { stage: 1, host: TargetSelection { triple: "x86_64-unknown-linux-gnu", file: None } }, target: TargetSelection { triple: "x86_64-unknown-linux-gnu", file: None }, tool: "cargo-clippy", path: "src/tools/clippy", mode: ToolRustc, is_optional_tool: true, source_type: InTree, extra_features: [] }
< CargoClippy { compiler: Compiler { stage: 1, host: TargetSelection { triple: "x86_64-unknown-linux-gnu", file: None } }, target: TargetSelection { triple: "x86_64-unknown-linux-gnu", file: None }, extra_features: [] }
> Assemble { target_compiler: Compiler { stage: 1, host: TargetSelection { triple: "x86_64-unknown-linux-gnu", file: None } } }
  > Assemble { target_compiler: Compiler { stage: 0, host: TargetSelection { triple: "x86_64-unknown-linux-gnu", file: None } } }
  < Assemble { target_compiler: Compiler { stage: 0, host: TargetSelection { triple: "x86_64-unknown-linux-gnu", file: None } } }
  > Sysroot { compiler: Compiler { stage: 1, host: TargetSelection { triple: "x86_64-unknown-linux-gnu", file: None } } }
  < Sysroot { compiler: Compiler { stage: 1, host: TargetSelection { triple: "x86_64-unknown-linux-gnu", file: None } } }
< Assemble { target_compiler: Compiler { stage: 1, host: TargetSelection { triple: "x86_64-unknown-linux-gnu", file: None } } }
> Clippy { compiler: Compiler { stage: 1, host: TargetSelection { triple: "x86_64-unknown-linux-gnu", file: None } }, target: TargetSelection { triple: "x86_64-unknown-linux-gnu", file: None }, extra_features: [] }
  > ToolBuild { compiler: Compiler { stage: 1, host: TargetSelection { triple: "x86_64-unknown-linux-gnu", file: None } }, target: TargetSelection { triple: "x86_64-unknown-linux-gnu", file: None }, tool: "clippy-driver", path: "src/tools/clippy", mode: ToolRustc, is_optional_tool: true, source_type: InTree, extra_features: [] }
    > Rustc { target: TargetSelection { triple: "x86_64-unknown-linux-gnu", file: None }, compiler: Compiler { stage: 1, host: TargetSelection { triple: "x86_64-unknown-linux-gnu", file: None } } }
      > Std { target: TargetSelection { triple: "x86_64-unknown-linux-gnu", file: None }, compiler: Compiler { stage: 1, host: TargetSelection { triple: "x86_64-unknown-linux-gnu", file: None } } }
        > StartupObjects { compiler: Compiler { stage: 1, host: TargetSelection { triple: "x86_64-unknown-linux-gnu", file: None } }, target: TargetSelection { triple: "x86_64-unknown-linux-gnu", file: None } }
        < StartupObjects { compiler: Compiler { stage: 1, host: TargetSelection { triple: "x86_64-unknown-linux-gnu", file: None } }, target: TargetSelection { triple: "x86_64-unknown-linux-gnu", file: None } }
        c Assemble { target_compiler: Compiler { stage: 1, host: TargetSelection { triple: "x86_64-unknown-linux-gnu", file: None } } }
        > Libdir { compiler: Compiler { stage: 1, host: TargetSelection { triple: "x86_64-unknown-linux-gnu", file: None } }, target: TargetSelection { triple: "x86_64-unknown-linux-gnu", file: None } }
          c Sysroot { compiler: Compiler { stage: 1, host: TargetSelection { triple: "x86_64-unknown-linux-gnu", file: None } } }
        < Libdir { compiler: Compiler { stage: 1, host: TargetSelection { triple: "x86_64-unknown-linux-gnu", file: None } }, target: TargetSelection { triple: "x86_64-unknown-linux-gnu", file: None } }
        c Libdir { compiler: Compiler { stage: 1, host: TargetSelection { triple: "x86_64-unknown-linux-gnu", file: None } }, target: TargetSelection { triple: "x86_64-unknown-linux-gnu", file: None } }
        c Sysroot { compiler: Compiler { stage: 1, host: TargetSelection { triple: "x86_64-unknown-linux-gnu", file: None } } }
        c Sysroot { compiler: Compiler { stage: 1, host: TargetSelection { triple: "x86_64-unknown-linux-gnu", file: None } } }
        c Sysroot { compiler: Compiler { stage: 1, host: TargetSelection { triple: "x86_64-unknown-linux-gnu", file: None } } }
        c Sysroot { compiler: Compiler { stage: 1, host: TargetSelection { triple: "x86_64-unknown-linux-gnu", file: None } } }
Building stage1 std artifacts (x86_64-unknown-linux-gnu -> x86_64-unknown-linux-gnu)
running: "/home/joshua/rustc3/build/x86_64-unknown-linux-gnu/stage0/bin/cargo" "build" "--target" "x86_64-unknown-linux-gnu" "-Zbinary-dep-depinfo" "-j" "8" "--release" "--features" "panic-unwind backtrace compiler-builtins-c" "--manifest-path" "/home/joshua/rustc3/library/test/Cargo.toml" "--message-format" "json-render-diagnostics"
   Compiling core v0.0.0 (/home/joshua/rustc3/library/core)
^C  Building [======>                     ] 9/33: core                              
Build completed unsuccessfully in 0:00:01

jyn514 · 2021-02-24T14:58:34Z

lol I also found that this completely ignores all test suites, making it pretty useless:

Not running bootstrap::test::RustdocUi because its artifacts have been downloaded from CI (`download-rustc` is set)

This was introduced as part of the MVP for `download-rustc`. Unfortunately, it doesn't work very well: - Steps are ignored by default, which makes it easy to leave out a step that should be built. For example, the MVP forgot to enable any tests, so it was *only* possible to build locally. - It didn't work correctly even when it was enabled: calling `builder.ensure()` would completely ignore the constant and rebuild the step anyway. This has no obvious fix since `ensure()` has to return a `Step::Output`. Instead, this handles `download-rustc` in `impl Step for Rustc` and `impl Step for Std`, which to my knowledge are the only build steps that don't first go through `impl Step for Sysroot` (`Rustc` is used for the `rustc-dev` component). See rust-lang#79540 (comment) and rust-lang#81930 for further context. Here are some example runs with these changes and `download-rustc` enabled: ``` $ x.py build src/tools/clippy Building stage1 tool clippy-driver (x86_64-unknown-linux-gnu) Finished release [optimized] target(s) in 1m 09s Building stage1 tool cargo-clippy (x86_64-unknown-linux-gnu) Finished release [optimized] target(s) in 0.11s $ x.py test src/tools/clippy Updating only changed submodules Submodules updated in 0.01 seconds Finished dev [unoptimized + debuginfo] target(s) in 0.09s Building stage1 tool clippy-driver (x86_64-unknown-linux-gnu) Finished release [optimized] target(s) in 0.09s Building rustdoc for stage1 (x86_64-unknown-linux-gnu) Finished release [optimized] target(s) in 0.28s Finished release [optimized] target(s) in 15.26s Running build/x86_64-unknown-linux-gnu/stage1-tools/x86_64-unknown-linux-gnu/release/deps/clippy_driver-8b407b140e0aa91c test result: ok. 592 passed; 0 failed; 3 ignored; 0 measured; 0 filtered out $ x.py build src/tools/rustdoc Building rustdoc for stage1 (x86_64-unknown-linux-gnu) Finished release [optimized] target(s) in 41.28s Build completed successfully in 0:00:41 $ x.py test src/test/rustdoc-ui Building stage0 tool compiletest (x86_64-unknown-linux-gnu) Finished release [optimized] target(s) in 0.12s Building rustdoc for stage1 (x86_64-unknown-linux-gnu) Finished release [optimized] target(s) in 0.10s test result: ok. 105 passed; 0 failed; 0 ignored; 0 measured; 0 filtered out; finished in 8.15s $ x.py build compiler/rustc Finished dev [unoptimized + debuginfo] target(s) in 0.09s Build completed successfully in 0:00:00 ``` Note a few things: - Clippy depends on stage1 rustc-dev artifacts, but rustc didn't have to be recompiled. Instead, the artifacts were copied automatically. - All steps are always enabled. There is no danger of forgetting a step, since only the entrypoints have to handle `download-rustc`. - Building the compiler (`compiler/rustc`) automatically does no work.

camelid · 2021-02-25T01:58:15Z

Required before this is made a default

I don't think this should be enabled by default until this is addressed as well so that it won't use the CI rustc if you've made local changes to rustc. Otherwise it would be really frustrating for people making changes to rustc and rustdoc (et al.).

jyn514 · 2021-02-25T02:18:35Z

@camelid that's why it has a warning. Do you think it should force you to disable the option? That seems like it could be frustrating.

jyn514 · 2021-02-25T02:19:15Z

I definitely don't think it should silently use the in-tree compiler if you make changes, it should always do what you specify in config.toml.

camelid · 2021-02-25T02:22:13Z

I was thinking that there could be a download-rustc = "if-no-local-changes" option or something like that, where it would only download rustc if compiler/ isn't dirty.

jyn514 · 2021-02-25T02:22:55Z

Ah ok, that seems useful. I don't think it needs to block turning this on by default, though.

camelid · 2021-02-25T02:26:20Z

Ah ok, that seems useful. I don't think it needs to block turning this on by default, though.

Won't this be annoying to new rustc contributors who have to dig around in their settings?

jyn514 · 2021-02-25T02:29:36Z

@camelid I found the confusion. When I say "by-default", I mean that I want to add a new x.py setup tools option that's specifically intended for rustdoc/clippy/rustfmt contributors and has download-rustc = true. I don't want to turn it on for the whole repository, that wouldn't make sense (or at least, not until if-no-changes is implemented).

camelid · 2021-02-25T02:37:10Z

Ah, I had a feeling we were talking past each other :)

Sounds good to me, as long as the warning is helpful enough that if people start doing compiler contributions they don't get stuck.

…imulacrum Remove `ENABLE_DOWNLOAD_RUSTC` constant `ENABLE_DOWNLOAD_RUSTC` was introduced as part of the MVP for `download-rustc` as a way not to rebuild artifacts that have already been downloaded. Unfortunately, it doesn't work very well: - Steps are ignored by default, which makes it easy to leave out a step that should be built. For example, the MVP forgot to enable any tests, so it was only possible to *build* locally. - It didn't work correctly even when it was enabled: calling `builder.ensure()` would completely ignore the constant and rebuild the step anyway. This has no obvious fix since `ensure()` has to return a `Step::Output`. Instead, this handles `download-rustc` in `impl Step for Rustc` and `impl Step for Std`, which to my knowledge are the only build steps that don't first go through `impl Step for Sysroot` (`Rustc` is used for the `rustc-dev` component). See rust-lang#79540 (comment) and rust-lang#81930 for further context. Here are some example runs with these changes and `download-rustc` enabled: ``` $ x.py build src/tools/clippy Building stage1 tool clippy-driver (x86_64-unknown-linux-gnu) Finished release [optimized] target(s) in 1m 09s Building stage1 tool cargo-clippy (x86_64-unknown-linux-gnu) Finished release [optimized] target(s) in 0.11s $ x.py test src/tools/clippy Finished dev [unoptimized + debuginfo] target(s) in 0.09s Building stage1 tool clippy-driver (x86_64-unknown-linux-gnu) Finished release [optimized] target(s) in 0.09s Building rustdoc for stage1 (x86_64-unknown-linux-gnu) Finished release [optimized] target(s) in 0.28s Finished release [optimized] target(s) in 15.26s Running build/x86_64-unknown-linux-gnu/stage1-tools/x86_64-unknown-linux-gnu/release/deps/clippy_driver-8b407b140e0aa91c test result: ok. 592 passed; 0 failed; 3 ignored; 0 measured; 0 filtered out $ x.py build src/tools/rustdoc Building rustdoc for stage1 (x86_64-unknown-linux-gnu) Finished release [optimized] target(s) in 41.28s Build completed successfully in 0:00:41 $ x.py test src/test/rustdoc-ui Building stage0 tool compiletest (x86_64-unknown-linux-gnu) Finished release [optimized] target(s) in 0.12s Building rustdoc for stage1 (x86_64-unknown-linux-gnu) Finished release [optimized] target(s) in 0.10s test result: ok. 105 passed; 0 failed; 0 ignored; 0 measured; 0 filtered out; finished in 8.15s $ x.py build compiler/rustc Finished dev [unoptimized + debuginfo] target(s) in 0.09s Build completed successfully in 0:00:00 ``` Note a few things: - Clippy depends on stage1 rustc-dev artifacts, but rustc didn't have to be recompiled. Instead, the artifacts were copied automatically. - All steps are always enabled. There is no danger of forgetting a step, since only the entrypoints have to handle `download-rustc`. - Building the compiler (`compiler/rustc`) automatically does no work. Helps with rust-lang#81930. r? `@Mark-Simulacrum`

…lacrum Always compile rustdoc with debug logging enabled when `download-rustc` is set Previously, logging at DEBUG or below would always be silenced, because rustc compiles tracing with the `static_max_level_info` feature. That makes sense for release artifacts, but not for developing rustdoc. Instead, this compiles two different versions of tracing: one in the release artifacts, distributed in the sysroot, and a new version compiled by rustdoc. Since `rustc_driver` is always linked to the version of sysroot, this copy/pastes `init_env_logging` into rustdoc. To avoid compiling an unnecessary version of tracing when `download-rustc` isn't set, this adds a new `using-ci-artifacts` feature for rustdoc and passes that feature in bootstrap. Addresses rust-lang#81930. This builds on rust-lang#79540. r? `@Mark-Simulacrum`

jyn514 · 2021-04-05T14:27:33Z

It would be nice to support cross-compiling the standard library. Right now this gives an assertion failure I think.

Err it looks like x.py build library/std always does nothing. So --target is also ignored.

Add `x.py setup tools` which enables `download-rustc` by default Helps with rust-lang#81930. I know I said in that issue that I should fix that rebasing rebuilds bootstrap, but the compile time improvement is so good I think it's ok to leave that fix for later (I still plan to work on it). I think all the outright bugs have been fixed :) This builds on rust-lang#83368 so I can set the option to `if-unchanged`. r? `@Mark-Simulacrum`

Add `x.py setup tools` which enables `download-rustc` by default Helps with rust-lang#81930. I know I said in that issue that I should fix that rebasing rebuilds bootstrap, but the compile time improvement is so good I think it's ok to leave that fix for later (I still plan to work on it). I think all the outright bugs have been fixed :) This builds on rust-lang#83368 so I can set the option to `if-unchanged`. r? ``@Mark-Simulacrum``

Add `x.py setup tools` which enables `download-rustc` by default Helps with rust-lang#81930. I know I said in that issue that I should fix that rebasing rebuilds bootstrap, but the compile time improvement is so good I think it's ok to leave that fix for later (I still plan to work on it). I think all the outright bugs have been fixed :) This builds on rust-lang#83368 so I can set the option to `if-unchanged`. r? ```@Mark-Simulacrum```

jyn514 · 2021-04-06T18:58:28Z

Oh huh, this is an unexpected side-effect: since #82739, building compiletest also requires building stage0 std first. I think using master libstd is intentional since #59264? I don't think it necessarily needs to change, maybe just documented somewhere?

Anyway, that fact that just works without build errors makes me a lot more confident we're choosing the right abstractions in bootstrap :)

camelid · 2021-04-11T21:11:43Z

Reposting from this Zulip discussion as requested by @jyn514:

I just ran into something confusing with download-rustc: normal x.py build will build rustdoc and put it in stage1. However, x.py build with download-rustc enabled will put it in stage2. This means that rustdoc +stage1 ... won't run my local, modified version of rustdoc if download-rustc is set. Specifically, rustdoc +stage1 ... seems to run nightly rustdoc (or something static).

I'm guessing this is because rustdoc built with download-rustc is being built with stage2 rustc, thus it is stage2 rustdoc, whereas rustdoc built locally without download-rustc is usually built with stage1 rustc, thus it is stage1 rustdoc. However, this behavior was very confusing to me and it took me a bunch of experimenting to figure out why my changes weren't showing up in the rustdoc output.

I imagine it could be even more confusing for new contributors since we recommend using stage1 builds, but then when they use download-rustc they have to remember that it's actually a stage2 build and adjust their commands accordingly. Plus, it's frustrating even if you know what's going on to switch between stage1 and stage2 depending on whether you're working on rustc or rustdoc (or a different tool).

Is there any way to solve this problem? I imagine there are multiple possible solutions, some better than others.

jyn514 · 2022-09-13T19:06:37Z

I just ran into something confusing with download-rustc: normal x.py build will build rustdoc and put it in stage1. However, x.py build with download-rustc enabled will put it in stage2. This means that rustdoc +stage1 ... won't run my local, modified version of rustdoc if download-rustc is set. Specifically, rustdoc +stage1 ... seems to run nightly rustdoc (or something static).

@bjorn3 suggests deleting the stage1 and stage0-sysroot directories when using download-rustc, to avoid caching issues like this. rustdoc +stage1 still won't do the right thing, but it will at least give a hard error instead of silently using the wrong version.

camelid · 2022-09-14T15:53:04Z

Great idea! That would be very helpful.

jyn514 · 2022-10-07T14:13:47Z

I just ran into something confusing with download-rustc: normal x.py build will build rustdoc and put it in stage1. However, x.py build with download-rustc enabled will put it in stage2. This means that rustdoc +stage1 ... won't run my local, modified version of rustdoc if download-rustc is set. Specifically, rustdoc +stage1 ... seems to run nightly rustdoc (or something static).

bjorn3 suggests deleting the stage1 and stage0-sysroot directories when using download-rustc, to avoid caching issues like this. rustdoc +stage1 still won't do the right thing, but it will at least give a hard error instead of silently using the wrong version.

Done in 973ff03

jyn514 · 2023-04-09T17:49:18Z

Give a better error message when rebasing an old branch (I think commits are kept at most 160 days?). #79540 (comment)

I'm going to cross this off the list - right now download-ci-llvm gives a useful error first and I think it's very rare to want to download rustc but not LLVM.

downloading https://ci-artifacts.rust-lang.org/rustc-builds/8ce3204af9463db3192ea1eb31c45c2f6d4b5ae6/rust-dev-nightly-x86_64-unknown-linux-gnu.tar.xz
curl: (22) The requested URL returned error: 404                                                                                

error: failed to download llvm from ci

help: old builds get deleted after a certain time
help: if trying to compile an old commit of rustc, disable `download-ci-llvm` in config.toml:

[llvm]
download-ci-llvm = false

Fix `x check --stage 1` when download-rustc is enabled Helps with rust-lang#81930

…ertlarsan68 Fix `x test ui --target foo` when download-rustc is enabled Previously, we would never build the target std, only the host std: ``` ; x t tests/ui/attributes --target wasm32-unknown-unknown Building bootstrap Finished dev [unoptimized] target(s) in 0.02s Building stage0 library artifacts (x86_64-unknown-linux-gnu) Finished release [optimized] target(s) in 0.08s Building tool compiletest (stage0) Finished release [optimized] target(s) in 0.09s Check compiletest suite=ui mode=ui (x86_64-unknown-linux-gnu -> wasm32-unknown-unknown) thread 'main' panicked at 'called `Result::unwrap()` on an `Err` value: Error { depth: 0, inner: Io { path: Some("/home/jyn/src/rust2/build/x86_64-unknown-linux-gnu/stage2/lib/rustlib/wasm32-unknown-unknown/lib"), err: Os { code: 2, kind: NotFound, message: "No such file or directory" } } }', src/tools/compiletest/src/main.rs:842:31 ``` Helps with rust-lang#81930.

…lacrum download-rustc: Give a better error message if artifacts can't be dowloaded It should be very rare in practice to happen; people would need to both have `download-ci-llvm` disabled and `download-rustc` enabled. I think it may be more common if we start turning this on by default, though. Helps with rust-lang#81930. Before: ``` downloading https://ci-artifacts.rust-lang.org/rustc-builds/bf5cad8e775fb326465e5c1b98693e5d259da156/rust-std-nightly-x86_64-unknown-linux-gnu.tar.xz curl: (22) The requested URL returned error: 404 ``` After: ``` downloading https://ci-artifacts.rust-lang.org/rustc-builds/bf5cad8e775fb326465e5c1b98693e5d259da156/rust-std-nightly-x86_64-unknown-linux-gnu.tar.xz curl: (22) The requested URL returned error: 404 error: failed to download pre-built rustc from CI note: old builds get deleted after a certain time help: if trying to compile an old commit of rustc, disable `download-rustc` in config.toml: [rust] download-rustc = false ```

jyn514 · 2023-06-06T23:08:11Z

Some tests fail when download-rustc is enabled: https://github.com/rust-lang/rust/issues?q=is%3Aissue+is%3Aopen+label%3AA-testsuite+label%3AA-download-rustc

this is now fixed well enough that x test by default succeeds :) x test --stage 1 rustdoc is still broken, but that should be extremely rare to need to run, and I don't think we need a whole tracking issue for it.

jyn514 self-assigned this Feb 9, 2021

jyn514 mentioned this issue Feb 9, 2021

Always compile rustdoc with debug logging enabled when download-rustc is set #81932

Merged

jyn514 mentioned this issue Feb 24, 2021

Remove ENABLE_DOWNLOAD_RUSTC constant #82480

Merged

jyn514 mentioned this issue Mar 4, 2021

Use the beta compiler for building bootstrap tools when download-rustc is set #82739

Merged

camelid mentioned this issue Apr 13, 2021

Support ./x.py run miri #76666

Closed

ehuss removed the T-dev-tools Relevant to the dev-tools subteam, which will review and decide on the PR/issue. label Jan 18, 2022

jyn514 removed their assignment Feb 15, 2022

jyn514 mentioned this issue Sep 19, 2022

Delete the stage1 and stage0-sysroot directories when using download-rustc #102002

Closed

This was referenced Mar 5, 2023

tests/ui/allocator/no_std-alloc-error-handler* fail when download-rustc is enabled #108767

Closed

x test --stage 2 std uses the wrong stage when download-rustc is enabled #108768

Closed

jyn514 mentioned this issue Apr 9, 2023

Fix x test ui --target foo when download-rustc is enabled #110113

Merged

This was referenced Apr 9, 2023

download-rustc: Give a better error message if artifacts can't be dowloaded #110118

Merged

E0460 long description is too optimistic about cargo's abilities #110120

Open

Fix x check --stage 1 when download-rustc is enabled #110121

Merged

Dylan-DPC added a commit to Dylan-DPC/rust that referenced this issue Apr 10, 2023

Rollup merge of rust-lang#110121 - jyn514:check-stage1, r=ozkanonur

6f2fd3e

Fix `x check --stage 1` when download-rustc is enabled Helps with rust-lang#81930

jyn514 added the A-download-rustc Area: The `rust.download-rustc` build option. label Apr 12, 2023

jyn514 mentioned this issue Apr 20, 2023

Redesign bootstrap stages rust-lang/compiler-team#619

Closed

3 tasks

jyn514 closed this as completed Jun 6, 2023

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Follow-up work for `download-rustc` #81930

Follow-up work for `download-rustc` #81930

jyn514 commented Feb 9, 2021 •

edited

Loading

jyn514 commented Feb 24, 2021

jyn514 commented Feb 24, 2021

camelid commented Feb 25, 2021 •

edited

Loading

jyn514 commented Feb 25, 2021

jyn514 commented Feb 25, 2021

camelid commented Feb 25, 2021

jyn514 commented Feb 25, 2021

camelid commented Feb 25, 2021

jyn514 commented Feb 25, 2021

camelid commented Feb 25, 2021

jyn514 commented Apr 5, 2021

jyn514 commented Apr 6, 2021 •

edited

Loading

camelid commented Apr 11, 2021

jyn514 commented Sep 13, 2022

camelid commented Sep 14, 2022

jyn514 commented Oct 7, 2022

jyn514 commented Apr 9, 2023

jyn514 commented Jun 6, 2023

Follow-up work for download-rustc #81930

Follow-up work for download-rustc #81930

Comments

jyn514 commented Feb 9, 2021 • edited Loading

Required before this is made a default

Nice to have

jyn514 commented Feb 24, 2021

jyn514 commented Feb 24, 2021

camelid commented Feb 25, 2021 • edited Loading

jyn514 commented Feb 25, 2021

jyn514 commented Feb 25, 2021

camelid commented Feb 25, 2021

jyn514 commented Feb 25, 2021

camelid commented Feb 25, 2021

jyn514 commented Feb 25, 2021

camelid commented Feb 25, 2021

jyn514 commented Apr 5, 2021

jyn514 commented Apr 6, 2021 • edited Loading

camelid commented Apr 11, 2021

jyn514 commented Sep 13, 2022

camelid commented Sep 14, 2022

jyn514 commented Oct 7, 2022

jyn514 commented Apr 9, 2023

jyn514 commented Jun 6, 2023

Follow-up work for `download-rustc` #81930

Follow-up work for `download-rustc` #81930

jyn514 commented Feb 9, 2021 •

edited

Loading

camelid commented Feb 25, 2021 •

edited

Loading

jyn514 commented Apr 6, 2021 •

edited

Loading