rustdoc: follow-up work for --emit=toolchain-shared-resources

[jyn514]

This is a tracking issue for the unstable feature --emit=toolchain-shared-resources, which requires -Z unstable-options. This feature is not meant to ever be stabilized; it's used by docs.rs to avoid copying the same files for each crate. Users other than docs.rs are not supported.

Unresolved Questions

• Can we get rid of unversioned-shared-resources completely? It's unclear what the difference between a versioned and unversioned resource is; I don't know if rustdoc can promise its files will never change even across toolchains.
• main.js is currently marked as a toolchain resource, which is wrong if the user passes --theme in RUSTDOCFLAGS.

Implementation history

• Initial implementation: rustdoc: Add unstable option to only emit shared/crate-specific files #83478
• Use in docs.rs Upload exactly the shared files needed and give precedence to global files over local ones docs.rs#1340
• Remove --print=unversioned-files: Remove --print unversioned-files from rustdoc  #85770

Why does this feature exist?

Docs.rs has been having quite a lot of trouble with essential files. Here are the recent developments:

1. rustdoc added a new crates.js file a few weeks ago, breaking search for all newly built crates Search within crate is broken: "Uncaught TypeError: crates is undefined" docs.rs#1300
2. dtolnay added crates.js and also some other essential files that were missing Add crates.js and woff2 fonts to required files docs.rs#1301
3. It turned out the other essential files had a typo, breaking all builds Static resources corresponding to 20210305-1.52.0-nightly-caca2121f are missing docs.rs#1305. the typo was fixed by Fix spelling of FiraSans-*.woff2 docs.rs#1308
4. rustdoc wanted to start adding more essential files (Remove theme.js file #82732, Update Source Code Pro and include italics #80705) and I got fed up of dealing with it so I opened Don't hard-code essential rustdoc files docs.rs#1312
5. Don't hard-code essential rustdoc files docs.rs#1312 had a bug (Don't hard-code essential rustdoc files docs.rs#1312 (comment)) if we copy more files than necessary. I opened Give precedence to local shared files over global ones docs.rs#1324 giving precedence to local shared files over global ones which avoids the bug.
6. Give precedence to local shared files over global ones docs.rs#1324 broke docs that weren't using --extern-html-root-url (Docs for releases during 2018-{10-12} appear unstyled docs.rs#1327 (comment)). The tracking issue for rebuilding those is Docs built during 2018-{10-12} link to the wrong url for static resources docs.rs#1181.

In summary, docs.rs can't copy fewer shared files than rustdoc generates, and due to 2 months of broken builds about 3 years ago, it also can't copy more. So there has to be a way to know exactly which files should be shared and which shouldn't.