feat(build-dir): Reorganize build-dir layout

[ranger-ross]

What does this PR try to resolve?

This PR re-organizes the build-dir file layout structure to a layout organized by "build unit" when -Zbuild-dir-new-layout is enabled.
See #15010 for the motivations and design discussions.

Below is file structure generated for a foo crate with a single dependency on syn.

$ tree -a target
target
├── CACHEDIR.TAG
├── debug
│   ├── build
│   │   ├── foo
│   │   │   └── 9f7d8db454c2e3a0
│   │   │       ├── deps
│   │   │       │   ├── foo-9f7d8db454c2e3a0
│   │   │       │   └── foo-9f7d8db454c2e3a0.d
│   │   │       └── fingerprint
│   │   │           ├── bin-foo
│   │   │           ├── bin-foo.json
│   │   │           ├── dep-bin-foo
│   │   │           └── invoked.timestamp
│   │   ├── proc-macro2
│   │   │   ├── 39344080ec273066
│   │   │   │   ├── deps
│   │   │   │   │   ├── libproc_macro2-39344080ec273066.rlib
│   │   │   │   │   ├── libproc_macro2-39344080ec273066.rmeta
│   │   │   │   │   └── proc_macro2-39344080ec273066.d
│   │   │   │   └── fingerprint
│   │   │   │       ├── dep-lib-proc_macro2
│   │   │   │       ├── invoked.timestamp
│   │   │   │       ├── lib-proc_macro2
│   │   │   │       └── lib-proc_macro2.json
│   │   │   ├── 8fc259c340d09182
│   │   │   │   ├── build-script
│   │   │   │   │   ├── build-script-build
│   │   │   │   │   ├── build_script_build-8fc259c340d09182
│   │   │   │   │   └── build_script_build-8fc259c340d09182.d
│   │   │   │   ├── deps
│   │   │   │   └── fingerprint
│   │   │   │       ├── build-script-build-script-build
│   │   │   │       ├── build-script-build-script-build.json
│   │   │   │       ├── dep-build-script-build-script-build
│   │   │   │       └── invoked.timestamp
│   │   │   └── a6c299e456bc1663
│   │   │       ├── build-script-execution
│   │   │       │   ├── invoked.timestamp
│   │   │       │   ├── out
│   │   │       │   ├── output
│   │   │       │   ├── root-output
│   │   │       │   └── stderr
│   │   │       ├── deps
│   │   │       └── fingerprint
│   │   │           ├── run-build-script-build-script-build
│   │   │           └── run-build-script-build-script-build.json
│   │   ├── quote
│   │   │   └── e95944fa29b3b7d5
│   │   │       ├── deps
│   │   │       │   ├── libquote-e95944fa29b3b7d5.rlib
│   │   │       │   ├── libquote-e95944fa29b3b7d5.rmeta
│   │   │       │   └── quote-e95944fa29b3b7d5.d
│   │   │       └── fingerprint
│   │   │           ├── dep-lib-quote
│   │   │           ├── invoked.timestamp
│   │   │           ├── lib-quote
│   │   │           └── lib-quote.json
│   │   ├── syn
│   │   │   └── a0dd360a8d0d85f9
│   │   │       ├── deps
│   │   │       │   ├── libsyn-a0dd360a8d0d85f9.rlib
│   │   │       │   ├── libsyn-a0dd360a8d0d85f9.rmeta
│   │   │       │   └── syn-a0dd360a8d0d85f9.d
│   │   │       └── fingerprint
│   │   │           ├── dep-lib-syn
│   │   │           ├── invoked.timestamp
│   │   │           ├── lib-syn
│   │   │           └── lib-syn.json
│   │   └── unicode-ident
│   │       └── 90d3c33eeb6b75ae
│   │           ├── deps
│   │           │   ├── libunicode_ident-90d3c33eeb6b75ae.rlib
│   │           │   ├── libunicode_ident-90d3c33eeb6b75ae.rmeta
│   │           │   └── unicode_ident-90d3c33eeb6b75ae.d
│   │           └── fingerprint
│   │               ├── dep-lib-unicode_ident
│   │               ├── invoked.timestamp
│   │               ├── lib-unicode_ident
│   │               └── lib-unicode_ident.json
│   ├── .cargo-lock
│   ├── examples
│   ├── foo
│   ├── foo.d
│   └── incremental
│       └── foo-2476dro80nq0n
│           ├── s-hbvjw5x5fa-1d58onh-7sam1xj7vrwy85uyr70f3vu2p
│           │   ├── 1z96gpvj66dpsd06phiqkocsr.o
│           │   ├── 8jaqpsn916x4kajeuk1b7rl12.o
│           │   ├── 8sv45ek5gkoxjdcr3b6nbgu2o.o
│           │   ├── 9yqg9fn9x870xvia9frlvvazg.o
│           │   ├── a5v36uvuoyp00oyvscfno2pce.o
│           │   ├── c2q7ff0mmsk4s1yg7s5ei1ryr.o
│           │   ├── dep-graph.bin
│           │   ├── query-cache.bin
│           │   └── work-products.bin
│           └── s-hbvjw5x5fa-1d58onh.lock
└── .rustc_info.json

36 directories, 62 files

For why incremental is where it is, see #15947 (comment)

As a side effect, we pass a lot more parameters to rustc, likely making cargo -v more annoying, similar to #13941.

How to test and review this PR?

This PR still needs to be more thoroughly tested. Thus far I have been testing on simple test crates.
Also see #15874 for potential test harness improvements that could be used by this PR.

Follow up actions

• Remove redundant -Cextra-filename in files where possible.
• See if we can drop <platform>/<profile> after More granular locking in cargo_rustc #4282
• Re-evaluate if we want platform to be unconditionally included in the build-dir layout
• Is <pkgname>/<hash> good enough or do we need to go with prefixes to reduce the number of items within a directory, see feat(build-dir): Reorganize build-dir layout #15947 (comment)
• Can we simplify how fingerpritns are stored, reducing pressure on path lenghts
• Under the new layout, should cargo clean -p also cleans old layout paths? See feat(build-dir): Reorganize build-dir layout #15947 (comment)

[epage]

CC @Kobzol due to your work on rust-lang/rust#145408. If reducing duplicate search paths speeds up builds, I wonder what the impact will be of having more but pin point focused search paths will be.

[weihanglo]

If reducing duplicate search paths speeds up builds, I wonder what the impact will be of having more but pin point focused search paths will be.

Probably doesn't matter much because you still need to go through all of them to find what a crate need?

Edit: doesn't matter for primary crate, but for dependencies at the very root (like syn), it would be helpful.

[epage]

Probably doesn't matter much because you still need to go through all of them to find what a crate need?

Currently, the search path includes each -Cextra-filename variant of a package's build. With this change, it will only see the variants relevant for this build.

[Kobzol]

I would be a bit worried about perf. in large scenarios (e.g. 1000 crates, which is not that uncommon), as I suspect that rustc does a bunch of linear (hopefully not quadratic) searches through these directories and files in them. I would suggest benchmarking on https://github.com/zed-industries/zed 😆

[weihanglo]

Basically files under a search directory are preloaded and sorted and then binary search on them, so shouldn't be too bad? It may incur more opendir/readdir syscall though.

Like epage mentioned, it also help for transitive dependency loading less files.

But yeah worth some benchmark for larger projects.

[Kobzol]

Ah, I forgot that we do binary search already. In that case it will be probably fine, yeah.

[Kobzol]

I tried it on Zed and didn't see any perf. difference vs master, neither for clean builds nor for incremental rebuilds.

[epage]

I tried it on Zed and didn't see any perf. difference vs master, neither for clean builds nor for incremental rebuilds.

If there is a different, it will most likely appear if you have multiple unique versions for each package, e.g. from

• cargo check
• cargo clippy
• cargo build
• cargo doc
• changing RUSTFLAGS
• changing features

[Kobzol]

I didn't see the rebuild time getting higher for Zed when I added multiple versions from different cargo invocations.

[epage]

As a follow up to this PR, we may want to remove -Cextra-filename where possible since uniqueness is now guaranteed by the directory. Unsure if rustc relies on this for loading of rlibs or if we can only drop the hash from non-rlibs.

[epage]

Another reason we might want to remove -Cextra-filename where possible is to reduce the risk of hitting windows path length issues

[weihanglo]

We got this pretty frequently these days.

---- package::reserved_windows_name stdout ----
running `D:\a\cargo\cargo\target\debug\cargo.exe package`

thread 'package::reserved_windows_name' (8372) panicked at tests\testsuite\package.rs:2997:10:

test failed running `D:\a\cargo\cargo\target\debug\cargo.exe package`
error: process exited with code 0 (expected 101)
--- stdout

--- stderr
warning: manifest has no documentation, homepage or repository.
See https://doc.rust-lang.org/cargo/reference/manifest.html#package-metadata for more info.
   Packaging foo v0.0.1 (D:\a\cargo\cargo\target\tmp\cit\t2560\foo)
    Updating `dummy-registry` index
    Packaged 4 files, 1.4KiB (877B compressed)
   Verifying foo v0.0.1 (D:\a\cargo\cargo\target\tmp\cit\t2560\foo)
 Downloading crates ...
  Downloaded bar v1.0.0 (registry `dummy-registry`)
   Compiling bar v1.0.0
   Compiling foo v0.0.1 (D:\a\cargo\cargo\target\tmp\cit\t2560\foo\target\package\foo-0.0.1)
    Finished `dev` profile [unoptimized + debuginfo] target(s) in 0.55s

Re-running jobs.

[epage]

Thanks!

[bjorn3]

As a follow up to this PR, we may want to remove -Cextra-filename where possible since uniqueness is now guaranteed by the directory. Unsure if rustc relies on this for loading of rlibs or if we can only drop the hash from non-rlibs.

For dylibs and for everything shipped in the sysroot we do still need this. For the former you almost certainly want to copy the dylibs together into a single directory and even if you don't, the linker only records the filenames of dylibs in executables, not their full path. And for the latter, not doing so would cause conflicts between multiple rustc versions installed on a single system. Also it helps speed up loading dependencies when there are multiple dependencies with the same crate name.

As a side effect, we pass a lot more parameters to rustc, likely making cargo -v more annoying, similar to #13941.

Not just that, but there are hard limits on the cli length on most operating systems. This would make it impossible to just copy paste commands into your shell if you exceed the limit even if cargo were to use response files to workaround this limit for executions it does on its own.