concurrent `Step` mechanism

[onur-ozkan]

It's worth redesigning the Step execution model to allow Steps to run in parallel, which would speed up pretty much any bootstrap invocation. This would be a big win as almost every bootstrap invocation involves running many Steps. Making bootstrap faster would benefit everyone especially those doing frequent build iterations while working on rust-lang/rust.

[bjorn3]

One complication is that there are cases where multiple steps write to the same directory to for example add extra things to the sysroot. With parallelism this would result in non-deterministic behavior.

[onur-ozkan]

Yeah we will need to figure out that kind of problems too.

[the8472]

We can mark individual steps as non-interfering and they can then be run in parallel to a serial strand of everything not marked so.

[Mark-Simulacrum]

This would be a big win as almost every bootstrap invocation involves running many Steps. Making bootstrap faster would benefit everyone especially those doing frequent build iterations while working on rust-lang/rust.

I'd be a bit surprised if this has any major impact on the common case of e.g. x.py test ui - that shouldn't have any opportunity for parallelism not already present today I think?

Can you say more about what cases you expect to benefit from this? Perhaps those can be achieved without the fully general parallelism?

[onur-ozkan]

I was thinking more about the build phases. For example if the stageN compiler is ready for use, we could build all (though in practice, there should be a reasonable concurrency limit) other Steps in parallel using that compiler instead of blocking on each other.

The biggest benefit would come from running a full build with all tools enabled. I guess primarily in CI but I believe there should still be significant improvements in development environments even without full builds. It's hard to predict precisely without actual testing, but I don't think it will be the same as today.

[the8472]

of e.g. x.py test ui

It'd probably be more beneficial when running multiple testsuites, then the tools needed by some of them could be built in parallel.
Anything that runs regular cargo tests could also build the test executables in parallel (via --no-run) and then run them serially since they're embarrassingly parallel anyway.