New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Determine packages to install prior to installing #9793

Merged

bors merged 3 commits into rust-lang:master from nipunn1313:install_parallel2

Aug 19, 2021

Contributor

nipunn1313 commented Aug 15, 2021

Old logic (pseudocode)

for krate in to_install {
    pkg = determine_pkg(krate);
    install_pkg(pkg);
}

New logic

let pkgs = to_install.into_iter(|krate| determine_pkg(krate));
pkgs.into_iter(|pkg| install_pkg(pkg));

This has the short term benefit of dumping most error messages out earlier in the process (eg a typo in the second package name).

Longer term, it might help with #9741 - as only the second loop would be parallelized. First loop shouldn't be parallelized because it would lead to redundant registry/git updates.

rust-highfive assigned ehuss

rust-highfive commented Aug 15, 2021

(rust-highfive has picked a reviewer for you, use r? to override)

rust-highfive added the S-waiting-on-review label

nipunn1313 mentioned this pull request

Cargo installing multiple bin crates doesn't build them in parallel #9741

Open

nipunn1313 force-pushed the install_parallel2 branch from e573d7b to 5b0cce5 Compare

August 15, 2021 17:47

alexcrichton reviewed

View reviewed changes

src/cargo/ops/cargo_install.rs Outdated Show resolved Hide resolved

nipunn1313 added 2 commits

August 16, 2021 13:02


          Refactor install_one into two parts

3beb322


          Refactor additional validation steps to determine_package

f7e89ea

nipunn1313 force-pushed the install_parallel2 branch from 5b0cce5 to e579652 Compare

August 16, 2021 20:49

nipunn1313 commented

View reviewed changes

src/cargo/ops/cargo_install.rs Outdated

    
                  Ok(Some(pkg))

              }

              fn no_track_duplicates(

Contributor Author

nipunn1313 Aug 16, 2021

this ends up being called once before compilation and once after compilation and I couldn't exactly figure out why. The one before makes sense (to short circuit compilation to avoid overwrites).

AFAICT, the call after compilation is simply to get a list of duplicates in the force case? Not sure if we expect the answer to be different the second time? Couldn't figure it out quickly, so kept the behavior the same.

src/cargo/ops/cargo_install.rs Outdated Show resolved Hide resolved

src/cargo/ops/cargo_install.rs Outdated

    
                  config.shell().status("Installing", &pkg)?;

                  let dst = root.join("bin").into_path_unlocked();

                  let (mut ws, rustc, target) = make_ws_rustc_target(config, opts, &source_id, pkg.clone())?;

Contributor Author

nipunn1313 Aug 16, 2021

Called this make_ws_rustc_target twice - once in the validation step (determine_package and once here).
We could consider funneling the result through from the first call into args here, but I figured it would be easier to just call it again.

Member

alexcrichton Aug 18, 2021

I think yeah let's try to funnel the result from the previous step into this one, workspace construction can be somewhat nontrivial since it involves parsing TOML files, so minimizing the calls there I think makes sense. I think this would also handle the "overwrite the pkg var" comment from above as well?

If you'd like I think it's also fine to have some other refactorings here to avoid "pass all the params all the time" and have, for example, a struct with state in it that has methods. I'll leave that up to you though

Contributor Author

nipunn1313 Aug 18, 2021

Great idea! Looks a lot nicer now with install_one as a method on InstallablePackage. Definitely results in only calling make_ws_rustc_target once.

Might be a bit difficult to code review as a bigish change now. Let me know if you have any factoring ideas that might make it easier on you.

src/cargo/ops/cargo_install.rs Outdated

    
            @@ -270,22 +299,6 @@ fn install_one(
          
                      ws.current()?.clone()

Contributor Author

nipunn1313 Aug 16, 2021

This line seems to overwrite pkg with another value (the value for the workspace) for non-git sources.

This means that the second call to make_ws_rustc_target down below uses the new ws pkg instead of the original pkg. I think this is ok, based on my understanding - but wanted to follow up here w/ you @alexcrichton

nipunn1313 force-pushed the install_parallel2 branch from e579652 to 01b239e Compare

August 16, 2021 20:59

Member

alexcrichton commented Aug 18, 2021

Looks good to me, thanks!

nipunn1313 force-pushed the install_parallel2 branch from 01b239e to 5cbb65d Compare

August 18, 2021 18:45

nipunn1313 commented

View reviewed changes

src/cargo/ops/cargo_install.rs

    
                      let install_results: Vec<_> = pkgs_to_install

                          .into_iter()

                          .map(|(krate, installable_pkg)| (krate, installable_pkg.install_one()))

                          .collect();

Contributor Author

nipunn1313 Aug 18, 2021

Here's the core change. We're first creating all the InstallablePackage, and later calling install_one on each of them.

alexcrichton reviewed

View reviewed changes

Member

alexcrichton left a comment

Looks good to me! I've got a question about the error handling but otherwise seems ok to me

src/cargo/ops/cargo_install.rs

    
                                  succeeded.push(krate);

                              }

                              Err(e) => {

                                  crate::display_error(&e, &mut config.shell());

Member

alexcrichton Aug 19, 2021

This seems like we may want to handle errors earlier perhaps? Otherwise since we process all the packages I think this could make it somewhat difficult to associated packages with errors, and it may also run the risk of delaying errors for some time while other packages are being processed?

I'm not entirely certain, though, what sort of errors are happening during the selection of packages, so this may be more reasonable than I expect. I think, though, that this is a behavior change since today it's more of a fail-fast situation, right?

Contributor Author

nipunn1313 Aug 19, 2021

Ideally errors happening during selection will show up next to "Downloading crates" and errors happening during compilation will happen next to "Compiling foo", so it's clear in either case.

Once I add the collect above - we can see the test output.

Member

alexcrichton Aug 19, 2021

Ah ok I think I also missed that this is actually roughly what happened earlier, sorry I got lost in the diff!

nipunn1313 commented

View reviewed changes

src/cargo/ops/cargo_install.rs Outdated

Contributor Author

nipunn1313 Aug 19, 2021

Just noticed that I actually have to collect() here, or else we won't successfully be completing all of the package selection prior to the compilation process. Once I add collect() here, I have to update a few tests, which is a good sanity check.


          Factor into struct InstallablePackage

f9c2d04

nipunn1313 force-pushed the install_parallel2 branch from 5cbb65d to f9c2d04 Compare

August 19, 2021 16:58

nipunn1313 commented

View reviewed changes

tests/testsuite/install_upgrade.rs

+              [DOWNLOADING] crates ...
+              [DOWNLOADED] two v1.0.0 (registry `[..]`)
+              [DOWNLOADING] crates ...
+              [DOWNLOADED] three v1.0.0 (registry `[..]`)

Contributor Author

nipunn1313 Aug 19, 2021

Noticing here that we could leverage the logic of PackageSet to actually download these crates in parallel (similar to when you do a regular Cargo build). Requires more refactoring. Maybe another day. Doesn't seem as important to do - given that the actual installation process ends up downloading 10x more crates as deps.

After this diff, the process for something like cargo install a b c

Download a
Download b
Download c
Install a (downloading all deps of a and compiling them)
Install b
Install c

One improvement could be to combine 1/2/3 into a parallel step
Another improvement could be to combine 4/5/6 into a parallelized step.

Could envision a super advanced improvement to combine all 6 into some kind of engine with a full task dependency graph (which Cargo has inside of it somewhere, but not at the cargo install level).

Member

alexcrichton Aug 19, 2021

Heh yeah I agree that there's lots of possible ways we could improve this, I'm hoping to delay the merging of all 6 steps for as long as we can, but I suspect it's inevitable...

Member

alexcrichton commented Aug 19, 2021

Thanks again!

Contributor

bors commented Aug 19, 2021

📌 Commit f9c2d04 has been approved by alexcrichton

bors added S-waiting-on-bors and removed S-waiting-on-review labels

Contributor

bors commented Aug 19, 2021

⌛ Testing commit f9c2d04 with merge b64c96b...

Contributor

bors commented Aug 19, 2021

☀️ Test successful - checks-actions
Approved by: alexcrichton
Pushing b64c96b to master...

bors merged commit b64c96b into rust-lang:master

This was referenced Aug 19, 2021

Packages can inherit fields from their root workspace #9684

Closed

WIP: Capture dep info with Spanned #9752

Closed

ehuss mentioned this pull request

Update cargo rust-lang/rust#88278

Merged

Dylan-DPC-zz pushed a commit to Dylan-DPC-zz/rust that referenced this pull request


          Rollup merge of rust-lang#88278 - ehuss:update-cargo, r=ehuss

Update cargo

16 commits in e96bdb0c3d0a418e7fcd7fbd69be08abf830b4bc..9b81660b79832f92512edd4c29059a9ff88fcb6c
2021-08-17 22:58:47 +0000 to 2021-08-23 20:04:47 +0000
- Fix panic with build-std of a proc-macro. (rust-lang/cargo#9834)
- Fix typos “a”→“an” (rust-lang/cargo#9821)
- Fix typo in git-authentication.md (rust-lang/cargo#9832)
- Add some debug logging for `cargo fix` (rust-lang/cargo#9831)
- Add documentation about third-party registries. (rust-lang/cargo#9830)
- unset the FIX_ENV when executing the real rustc (rust-lang/cargo#9818)
- Allow crate download by checksum (rust-lang/cargo#9801)
- Emit warning for migrating to unstable edition in stable channel (rust-lang/cargo#9792)
- Warning for no lib dependencies (rust-lang/cargo#9771)
- Temporarily disable extern-html-root-url test. (rust-lang/cargo#9824)
- Move `tmp` test directory. (rust-lang/cargo#9814)
- Fix test incorrectly validating CARGO_PKG_LICENSE_FILE. (rust-lang/cargo#9813)
- Implement `[future-incompat-report]` config section (rust-lang/cargo#9774)
- Bump curl. (rust-lang/cargo#9809)
- Determine packages to install prior to installing (rust-lang/cargo#9793)
- Show feature resolver differences for dev-dependencies. (rust-lang/cargo#9803)

Dylan-DPC-zz pushed a commit to Dylan-DPC-zz/rust that referenced this pull request


          Rollup merge of rust-lang#88278 - ehuss:update-cargo, r=ehuss

76db2ab

Update cargo

16 commits in e96bdb0c3d0a418e7fcd7fbd69be08abf830b4bc..9b81660b79832f92512edd4c29059a9ff88fcb6c
2021-08-17 22:58:47 +0000 to 2021-08-23 20:04:47 +0000
- Fix panic with build-std of a proc-macro. (rust-lang/cargo#9834)
- Fix typos “a”→“an” (rust-lang/cargo#9821)
- Fix typo in git-authentication.md (rust-lang/cargo#9832)
- Add some debug logging for `cargo fix` (rust-lang/cargo#9831)
- Add documentation about third-party registries. (rust-lang/cargo#9830)
- unset the FIX_ENV when executing the real rustc (rust-lang/cargo#9818)
- Allow crate download by checksum (rust-lang/cargo#9801)
- Emit warning for migrating to unstable edition in stable channel (rust-lang/cargo#9792)
- Warning for no lib dependencies (rust-lang/cargo#9771)
- Temporarily disable extern-html-root-url test. (rust-lang/cargo#9824)
- Move `tmp` test directory. (rust-lang/cargo#9814)
- Fix test incorrectly validating CARGO_PKG_LICENSE_FILE. (rust-lang/cargo#9813)
- Implement `[future-incompat-report]` config section (rust-lang/cargo#9774)
- Bump curl. (rust-lang/cargo#9809)
- Determine packages to install prior to installing (rust-lang/cargo#9793)
- Show feature resolver differences for dev-dependencies. (rust-lang/cargo#9803)

bors added a commit to rust-lang-ci/rust that referenced this pull request


          Auto merge of rust-lang#88278 - ehuss:update-cargo, r=ehuss

720a1b2

Update cargo

19 commits in e96bdb0c3d0a418e7fcd7fbd69be08abf830b4bc..f559c109cc79fe413a8535fb620a5a58b3823d94
2021-08-17 22:58:47 +0000 to 2021-08-26 22:54:55 +0000
- Fix test not to rely on `cargo` in PATH. (rust-lang/cargo#9843)
- Improve resolver message to include dependency requirements (rust-lang/cargo#9827)
- Add hint for cargo metadata in environment section (rust-lang/cargo#9836)
- Fix panic with build-std of a proc-macro. (rust-lang/cargo#9834)
- Fix typos “a”→“an” (rust-lang/cargo#9821)
- Fix typo in git-authentication.md (rust-lang/cargo#9832)
- Add some debug logging for `cargo fix` (rust-lang/cargo#9831)
- Add documentation about third-party registries. (rust-lang/cargo#9830)
- unset the FIX_ENV when executing the real rustc (rust-lang/cargo#9818)
- Allow crate download by checksum (rust-lang/cargo#9801)
- Emit warning for migrating to unstable edition in stable channel (rust-lang/cargo#9792)
- Warning for no lib dependencies (rust-lang/cargo#9771)
- Temporarily disable extern-html-root-url test. (rust-lang/cargo#9824)
- Move `tmp` test directory. (rust-lang/cargo#9814)
- Fix test incorrectly validating CARGO_PKG_LICENSE_FILE. (rust-lang/cargo#9813)
- Implement `[future-incompat-report]` config section (rust-lang/cargo#9774)
- Bump curl. (rust-lang/cargo#9809)
- Determine packages to install prior to installing (rust-lang/cargo#9793)
- Show feature resolver differences for dev-dependencies. (rust-lang/cargo#9803)

nipunn1313 deleted the install_parallel2 branch

September 26, 2021 22:42

ehuss added this to the 1.56.0 milestone

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

S-waiting-on-bors