feat: rebuild(-ish) rustdoc json for different version of same crates

[motorailgun]

Fix #16291.

What does this PR try to resolve?

As discussed in the linked issue, cargo rustdoc --output-format=json doesn't rebuild for same crate name but with different versions. This is because current cargo implementation around rustdoc only specifies output paths/names by only its crate names, but not with version(hash) info.

This PR forces new layout for json output, and uplift the file into backward-compatible place.

How to test and review this PR?

• Can we consider just forcing new layout for --output-format=json? (and stop uplifting)
• Uplifting will always happen even if uplifted json is fresh, and/or new layout is manually specified. (could not figure out how to prevent); however it is just a copy operation if docs JSON are already built and fresh.

[rustbot]

r? @epage

rustbot has assigned @epage.
They will have a look at your PR within the next two weeks and either review your PR or reassign to another reviewer.

Use r? to explicitly pick a reviewer

Why was this reviewer chosen?

The reviewer was selected based on:

• Owners of files modified in this PR: @ehuss, @epage, @weihanglo
• @ehuss, @epage, @weihanglo expanded to ehuss, epage, weihanglo
• Random selection from ehuss, epage, weihanglo

[motorailgun]

Updated to use new directory layout.

[rustbot]

This PR was rebased onto a different master commit. Here's a range-diff highlighting what actually changed.

Rebasing is a normal part of keeping PRs up to date, so no action is needed—this note is just to help reviewers.

[weihanglo]

Can we add new tests first, and the fix and snapshots in the second commit? So that we can see the behavior change in the snapshot updates. See https://epage.github.io/dev/pr-style/#c-test.

[motorailgun]

Ugh, sorry about my stupidness of forgetting atomic commits again... thank you for pointing out!

[motorailgun]

Note: updated PR message reflecting changes.

[weihanglo]

View changes since the review

Sorry I didn't really review this carefully until this round.

I feel like this is the not the correct place to uplift. In Cargo we track whether a OutputFile will be uplifted through its hard_link. If we make rustdoc JSON as part of that, we'll get uplifting logic for free. We would probably get uplifting logic for free if we set it here:

cargo/src/cargo/core/compiler/build_runner/compilation_files.rs

Line 537 in 428d326

hardlink: None,

---

Also, this seems to unconditionally uplift rustdoc JSON for every root unit. Not wrong but a bit error-prone as it assumes every root unit is rustdoc (which is true actually as output-format is available only in cargo rustdoc.

[weihanglo]

View changes since the review

We have cargo_util::paths::link_or_copy though we should leverage OutputFile::hardlink

[weihanglo]

View changes since the review

I wonder if we actually don't need this guard unstable_optionts here?

The place we set UserIntent::Doc { json: true } has already checked -Zunstable-options presence:

cargo/src/bin/cargo/commands/rustdoc.rs

Lines 56 to 72 in 35f8202

	let output_format = if let Some(output_format) = args._value_of("output-format") {
	gctx.cli_unstable()
	.fail_if_stable_opt("--output-format", 12103)?;
	output_format.parse()?
	} else {
	OutputFormat::Html
	};
	let mut compile_opts = args.compile_options_for_single_package(
	gctx,
	UserIntent::Doc {
	deps: false,
	json: matches!(output_format, OutputFormat::Json),
	},
	Some(&ws),
	ProfileChecking::Custom,
	)?;

So that may imply we also don't need the is_rustdoc_json_output at all