Clean up dependency tracking in Rustbuild [1/2] #50904

collin5 · 2018-05-19T20:43:30Z

Initial refactor of the Mode enum. Still a WIP
Ref #50509

Mark-Simulacrum · 2018-05-19T20:48:33Z

src/bootstrap/lib.rs

-    /// Build librustc and compiler libraries, placing output in the "stageN-rustc" directory.
-    Librustc,
+    /// Build librustc, codegen and compiler libraries, placing output
+    /// in the "stageN-rustc" directory.


This comment should be modified for Codegen since we place its output in -codegen directory.

Mark-Simulacrum · 2018-05-19T20:49:16Z

src/bootstrap/lib.rs

+            Mode::ToolTest => "-tools",
+            Mode::Codegen => "-rustc",
+            Mode::Rustc => "-rustc",
+            Mode::ToolRustc => "-tools",


Let's instead combine all of the tools into one pattern or have them after each other so it's clearer what's going on

Mark-Simulacrum · 2018-05-19T20:51:24Z

src/bootstrap/tool.rs

        let compiler = if builder.force_use_stage1(compiler, target) {
            builder.compiler(1, compiler.host)
        } else {
            compiler
        };

-        for &cur_mode in &[Mode::Libstd, Mode::Libtest, Mode::Librustc] {
+        for &cur_mode in &[Mode::ToolStd, Mode::ToolTest, Mode::ToolRustc] {


I believe we're calling CleanTools with Mode::Rustc instead of RustcTool in at least some places. Let's change the field to cause: Mode (or a different name if you can come up with something better) and switch these back to Mode::Std, Mode::Test and Mode::Rustc.

Hi, @Mark-Simulacrum Just to make sure I'm getting this right. Do you mean we should change the field mode: Mode on CleanTools to cause: Mode? Also, should the values just remain as they were? Thanks.

Yes, change the field to cause, and change values to Mode::Std vs. Mode::ToolStd since the cause was Std, not a tool.

Got it. Thanks again!

Mark-Simulacrum · 2018-05-19T20:53:22Z

src/bootstrap/tool.rs

@@ -202,7 +202,7 @@ impl Step for ToolBuild {
                return None;
            }
        } else {
-            let cargo_out = builder.cargo_out(compiler, Mode::Tool, target)
+            let cargo_out = builder.cargo_out(compiler, Mode::ToolRustc, target)


This should be self.mode instead

Mark-Simulacrum · 2018-05-19T20:53:50Z

src/bootstrap/tool.rs

@@ -218,7 +218,7 @@ pub fn prepare_tool_cargo(
    command: &'static str,
    path: &'static str,
 ) -> Command {
-    let mut cargo = builder.cargo(compiler, Mode::Tool, target, command);
+    let mut cargo = builder.cargo(compiler, Mode::ToolRustc, target, command);


We'll want prepare_tool_cargo to take the mode as an argument; that'll make future changes easier. Generally speaking we should be threading tool modes through so that we never just pick "ToolRustc" for no reason.

Mark-Simulacrum · 2018-05-19T20:55:53Z

src/bootstrap/tool.rs

@@ -587,7 +587,7 @@ impl<'a> Builder<'a> {
        let host = &compiler.host;
        let mut lib_paths: Vec<PathBuf> = vec![
            PathBuf::from(&self.sysroot_libdir(compiler, compiler.host)),
-            self.cargo_out(compiler, Mode::Tool, *host).join("deps"),
+            self.cargo_out(compiler, Mode::ToolRustc, *host).join("deps"),


Same as above, this should thread mode through from arguments

collin5 · 2018-05-27T22:14:13Z

Hi @Mark-Simulacrum, this should now be ready for another look. Thanks!

Mark-Simulacrum · 2018-05-27T22:25:10Z

src/bootstrap/test.rs

@@ -264,6 +264,7 @@ impl Step for Rls {

        let mut cargo = tool::prepare_tool_cargo(builder,
                                                 compiler,
+                                                 Mode::Rustc,


This should be Mode::ToolRustc as we're building a tool

Oops! let me fix this ASAP

Mark-Simulacrum · 2018-05-27T22:25:24Z

src/bootstrap/test.rs

@@ -318,6 +319,7 @@ impl Step for Rustfmt {

        let mut cargo = tool::prepare_tool_cargo(builder,
                                                 compiler,
+                                                 Mode::Rustc,


Same here, Mode::ToolRustc

Mark-Simulacrum · 2018-05-27T22:34:39Z

src/bootstrap/builder.rs

@@ -662,7 +662,7 @@ impl<'a> Builder<'a> {
            cargo.env("RUSTDOC_LIBDIR", self.rustc_libdir(self.compiler(2, self.config.build)));
        }

-        if mode == Mode::Tool {
+        if [Mode::ToolRustc, Mode::ToolStd, Mode::ToolTest].iter().any(|m| &mode == m) {


Let's implement an is_tool method on Tool.

By this, do you mean we should use a closure is_tool that checks if mode is in ToolRustc, ToolStd, etc?

Not quite -- method on Tool, not a closure.

It makes sense now. Thanks!

rust-highfive · 2018-05-27T23:35:13Z

The job x86_64-gnu-llvm-3.9 of your PR failed on Travis (raw log). Through arcane magic we have determined that the following fragments from the build log may contain information about the problem.

Click to expand the log.

[00:45:16] ...............................................................i....................................
[00:45:20] ....................................................................................................
[00:45:25] ....................................................................................................
[00:45:31] ............................................................................................i.......
[00:45:34] ..........iiiiiiiii...................................................
[00:45:34] 
[00:45:34] travis_fold:start:test_ui_nll
travis_time:start:test_ui_nll
Check compiletest suite=ui mode=ui compare_mode=nll (x86_64-unknown-linux-gnu -> x86_64-unknown-linux-gnu)
---
[00:46:21] ...............................................................i....................................
[00:46:25] ....................................................................................................
[00:46:30] ....................................................................................................
[00:46:36] ............................................................................................i.......
[00:46:38] ..........iiiiiiiii...................................................
[00:46:38] 
[00:46:38]  finished in 64.166
[00:46:38] travis_fold:end:test_ui_nll

---
[01:21:37]    Compiling rand v0.4.2
[01:21:38]    Compiling aho-corasick v0.6.4
[01:21:44]    Compiling tempdir v0.3.7
[01:22:16]    Compiling minifier v0.0.11
[01:22:19] error[E0523]: found two different crates with name `rand` that are not distinguished by differing `-C metadata`. This will result in symbol conflicts between the two.
[01:22:19]    |
[01:22:19] 48 | extern crate tempdir;
[01:22:19]    | ^^^^^^^^^^^^^^^^^^^^^
[01:22:19] 
[01:22:19] 
[01:22:19] error[E0523]: found two different crates with name `rand` that are not distinguished by differing `-C metadata`. This will result in symbol conflicts between the two.
[01:22:19]    |
[01:22:19] 48 | extern crate tempdir;
[01:22:19]    | ^^^^^^^^^^^^^^^^^^^^^
[01:22:19] 
---
[01:22:19] 
[01:22:19] To learn more, run the command again with --verbose.
[01:22:19] 
[01:22:19] 
[01:22:19] command did not execute successfully: "/checkout/obj/build/x86_64-unknown-linux-gnu/stage0/bin/cargo" "test" "--target" "x86_64-unknown-linux-gnu" "-j" "4" "--release" "--locked" "--color" "always" "--manifest-path" "/checkout/src/tools/rustdoc/Cargo.toml" "-p" "rustdoc:0.0.0" "--" "--quiet"
[01:22:19] 
[01:22:19] 
[01:22:19] failed to run: /checkout/obj/build/bootstrap/debug/bootstrap test
[01:22:19] Build completed unsuccessfully in 0:39:15
[01:22:19] Build completed unsuccessfully in 0:39:15
[01:22:19] Makefile:58: recipe for target 'check' failed
[01:22:19] make: *** [check] Error 1

The command "stamp sh -x -c "$RUN_SCRIPT"" exited with 2.
travis_time:start:0e26c3ee
$ date && (curl -fs --head https://google.com | grep ^Date: | sed 's/Date: //g' || true)

I'm a bot! I can only do what humans tell me to, so if this was not helpful or you have suggestions for improvements, please ping or otherwise contact @TimNN. (Feature Requests)

rust-highfive · 2018-05-28T01:45:11Z

The job mingw-check of your PR failed on Travis (raw log). Through arcane magic we have determined that the following fragments from the build log may contain information about the problem.

Click to expand the log.

[00:07:29]     Checking minifier v0.0.11
[00:07:30]     Checking rustdoc v0.0.0 (file:///checkout/src/librustdoc)
[00:07:38]     Checking rustdoc-tool v0.0.0 (file:///checkout/src/tools/rustdoc)
[00:07:39]     Finished release [optimized] target(s) in 17.28s
[00:07:39] thread 'main' panicked at 'target_deps_dir.read_dir() failed with No such file or directory (os error 2)', bootstrap/compile.rs:1097:20

[00:07:39] note: Run with `RUST_BACKTRACE=1` for a backtrace.
[00:07:39] travis_time:end:stage0-rustdoc:start=1527467194686142517,finish=1527467211997598076,duration=17311455559

I'm a bot! I can only do what humans tell me to, so if this was not helpful or you have suggestions for improvements, please ping or otherwise contact @TimNN. (Feature Requests)

bors · 2018-05-28T03:34:48Z

☔ The latest upstream changes (presumably #50892) made this pull request unmergeable. Please resolve the merge conflicts.

rust-highfive · 2018-05-29T00:46:15Z

The job x86_64-gnu-llvm-3.9 of your PR failed on Travis (raw log). Through arcane magic we have determined that the following fragments from the build log may contain information about the problem.

Click to expand the log.


[00:05:20] travis_fold:start:tidy
travis_time:start:tidy
tidy check
[00:05:21] tidy error: /checkout/src/bootstrap/doc.rs:808: line longer than 100 chars
[00:05:22] some tidy checks failed
[00:05:22] 
[00:05:22] 
[00:05:22] command did not execute successfully: "/checkout/obj/build/x86_64-unknown-linux-gnu/stage0-tools-bin/tidy" "/checkout/src" "/checkout/obj/build/x86_64-unknown-linux-gnu/stage0/bin/cargo" "--no-vendor" "--quiet"
[00:05:22] 
[00:05:22] 
[00:05:22] failed to run: /checkout/obj/build/bootstrap/debug/bootstrap test src/tools/tidy
[00:05:22] Build completed unsuccessfully in 0:02:03
[00:05:22] Build completed unsuccessfully in 0:02:03
[00:05:22] Makefile:79: recipe for target 'tidy' failed
[00:05:22] make: *** [tidy] Error 1

The command "stamp sh -x -c "$RUN_SCRIPT"" exited with 2.
travis_time:start:012c8946
$ date && (curl -fs --head https://google.com | grep ^Date: | sed 's/Date: //g' || true)

I'm a bot! I can only do what humans tell me to, so if this was not helpful or you have suggestions for improvements, please ping or otherwise contact @TimNN. (Feature Requests)

collin5 · 2018-05-29T12:19:42Z

Fixed! cc @Mark-Simulacrum

Mark-Simulacrum · 2018-05-29T14:13:36Z

src/bootstrap/builder.rs

@@ -662,7 +663,7 @@ impl<'a> Builder<'a> {
            cargo.env("RUSTDOC_LIBDIR", self.rustc_libdir(self.compiler(2, self.config.build)));
        }

-        if mode == Mode::Tool {
+        if Tool::is_tool(mode) {


Could we instead make this inherent, so mode.is_tool()? I think I may have previously mislead you; I intended this to be implemented on Mode.

No worries! Going to fix this ASAP. Thanks

bors · 2018-05-31T01:12:58Z

☔ The latest upstream changes (presumably #51138) made this pull request unmergeable. Please resolve the merge conflicts.

collin5 · 2018-06-03T04:04:07Z

Hi @Mark-Simulacrum, this should be fixed now. Thanks!

Mark-Simulacrum

I think this is the last thing!

Mark-Simulacrum · 2018-06-03T14:06:23Z

src/bootstrap/builder.rs

@@ -1012,7 +1012,7 @@ impl<'a> Builder<'a> {
        // be resolved because MinGW has the import library. The downside is we
        // don't get newer functions from Windows, but we don't use any of them
        // anyway.
-        if mode != Mode::Tool {
+        if mode.is_tool() {


This should be !mode.is_tool().

Oops! missed this.

collin5 · 2018-06-03T14:33:06Z

Fixed! Thank you

Mark-Simulacrum · 2018-06-03T14:35:14Z

If you can clean up the commit history a little bit -- maybe merging in the is_tool changes into one commit -- then I'll r+. Thanks!

make is_tool inherent prop of mode fix errors from rebase resolve issues from review

collin5 · 2018-06-03T14:45:33Z

Done! thank you.

Mark-Simulacrum · 2018-06-03T14:52:15Z

@bors r+

bors · 2018-06-03T14:52:15Z

📌 Commit 36eafe5 has been approved by Mark-Simulacrum

bors · 2018-06-03T17:16:20Z

⌛ Testing commit 36eafe5 with merge be5f17c...

@Mark-Simulacrum

Clean up dependency tracking in Rustbuild [1/2] Initial refactor of the `Mode` enum. Still a WIP Ref #50509 r? @Mark-Simulacrum

bors · 2018-06-03T20:13:04Z

☀️ Test successful - status-appveyor, status-travis
Approved by: Mark-Simulacrum
Pushing be5f17c to master...

@Mark-Simulacrum

Clean up dependency tracking in Rustbuild [2/2] Make `clear_if_dirty` calls in `Builder::cargo` with stamp dependencies for the given Mode. Continuation of #50904 Ref issue #50509 r? @Mark-Simulacrum

@Mark-Simulacrum

Clean up dependency tracking in Rustbuild [2/2] Make `clear_if_dirty` calls in `Builder::cargo` with stamp dependencies for the given Mode. Continuation of #50904 Ref issue #50509 r? @Mark-Simulacrum

@Mark-Simulacrum

Clean up dependency tracking in Rustbuild [2/2] Make `clear_if_dirty` calls in `Builder::cargo` with stamp dependencies for the given Mode. Continuation of #50904 Ref issue #50509 r? @Mark-Simulacrum

rust-highfive assigned Mark-Simulacrum May 19, 2018

rust-highfive added the S-waiting-on-review Status: Awaiting review from the assignee but also interested parties. label May 19, 2018

collin5 force-pushed the b50509 branch 2 times, most recently from c73b239 to 5f40201 Compare May 19, 2018 20:54

Mark-Simulacrum reviewed May 19, 2018

View reviewed changes

Mark-Simulacrum reviewed May 27, 2018

View reviewed changes

collin5 force-pushed the b50509 branch 2 times, most recently from 7f47fb6 to 2c9ded9 Compare May 28, 2018 00:17

collin5 force-pushed the b50509 branch from 2c9ded9 to 97ff715 Compare May 28, 2018 02:17

collin5 force-pushed the b50509 branch from 97ff715 to 072096b Compare May 29, 2018 00:32

collin5 force-pushed the b50509 branch from 072096b to 1dfb6ce Compare May 29, 2018 01:00

Mark-Simulacrum reviewed May 29, 2018

View reviewed changes

Mark-Simulacrum added S-waiting-on-author Status: This is awaiting some action (such as code changes or more information) from the author. and removed S-waiting-on-review Status: Awaiting review from the assignee but also interested parties. labels Jun 2, 2018

Collins Abitekaniza added 4 commits June 3, 2018 04:59

refactor Mode enum

42ee6d5

refactor, make requested changes

ce10910

thread tool modes through

fb949b5

refactor mode to ToolRustc for Rls & Rustfmt

1133397

collin5 force-pushed the b50509 branch from 1dfb6ce to 44c564a Compare June 3, 2018 02:16

kennytm added S-waiting-on-review Status: Awaiting review from the assignee but also interested parties. and removed S-waiting-on-author Status: This is awaiting some action (such as code changes or more information) from the author. labels Jun 3, 2018

Mark-Simulacrum requested changes Jun 3, 2018

View reviewed changes

Mark-Simulacrum approved these changes Jun 3, 2018

View reviewed changes

impl is_tool on Mode enum

36eafe5

make is_tool inherent prop of mode fix errors from rebase resolve issues from review

collin5 force-pushed the b50509 branch from fd5f579 to 36eafe5 Compare June 3, 2018 14:43

Mark-Simulacrum changed the title ~~[WIP] Clean up dependency tracking in Rustbuild~~ Clean up dependency tracking in Rustbuild Jun 3, 2018

Mark-Simulacrum changed the title ~~Clean up dependency tracking in Rustbuild~~ Clean up dependency tracking in Rustbuild [1/2] Jun 3, 2018

bors added S-waiting-on-bors Status: Waiting on bors to run and complete tests. Bors will change the label on completion. and removed S-waiting-on-review Status: Awaiting review from the assignee but also interested parties. labels Jun 3, 2018

bors added a commit that referenced this pull request Jun 3, 2018

Auto merge of #50904 - collin5:b50509, r=Mark-Simulacrum

be5f17c

Clean up dependency tracking in Rustbuild [1/2] Initial refactor of the `Mode` enum. Still a WIP Ref #50509 r? @Mark-Simulacrum

bors merged commit 36eafe5 into rust-lang:master Jun 3, 2018

bors mentioned this pull request Jun 3, 2018

[experimental]: Build LLVM with ThinLTO enabled #51207

Closed

collin5 deleted the b50509 branch June 4, 2018 14:59

collin5 mentioned this pull request Jul 5, 2018

Clean up dependency tracking in Rustbuild [2/2] #52036

Merged

Clean up dependency tracking in Rustbuild [1/2] #50904

Clean up dependency tracking in Rustbuild [1/2] #50904

Conversation

collin5 commented May 19, 2018

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

collin5 May 22, 2018 • edited Loading

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

collin5 commented May 27, 2018

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

rust-highfive commented May 27, 2018

rust-highfive commented May 28, 2018

bors commented May 28, 2018

rust-highfive commented May 29, 2018

collin5 commented May 29, 2018

Choose a reason for hiding this comment

Choose a reason for hiding this comment

bors commented May 31, 2018

collin5 commented Jun 3, 2018

Mark-Simulacrum left a comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

collin5 commented Jun 3, 2018

Mark-Simulacrum commented Jun 3, 2018

collin5 commented Jun 3, 2018

Mark-Simulacrum commented Jun 3, 2018

bors commented Jun 3, 2018

bors commented Jun 3, 2018

bors commented Jun 3, 2018

collin5 May 22, 2018 •

edited

Loading