Question re rustdoc reproducibility

[smoelius]

When I run cargo rustdoc -- -Z unstable-options --output-format=json on the standard library on an Ubuntu machine, I get the same JSON file each time. This has held regardless of whether I run the command locally or on GitHub's ubuntu-latest runner.

If I add --target x86_64-unknown-linux-gnu and run the command on GitHub's macos-latest runner, I get a slightly different file.

Is this expected?

Generally speaking, what is required to ensure reproducibility of cargo rustdoc's JSON output?

Thank you in advance, and sorry if this is not the right place to ask these questions.

[aDotInTheVoid]

Is this expected?

Honestly I'm not sure. What's the difference between the two files?

Generally speaking, what is required to ensure reproducibility of cargo rustdoc's JSON output?

Also not sure here, sorry. We do test that rustdoc is reproducible in some circumstances here, but that's running exactly the same binary.

More broadly, I'd be happy to take review/merge fixes to rustdoc that increase reproducibility.

[smoelius]

@aDotInTheVoid Thank you very much for the reply. I have attached two JSON files. Both were generated by running the following command in the root of Rust's library directory at commit 4392847410ddd67f6734dd9845f9742ff9e85c83:

rustdoc --target x86_64-unknown-linux-gnu -- -Z unstable-options --output-format=json

std-ubuntu.json was generated on an Ubuntu machine. std-mac.json was generated on a machine running macOS Sequoia 15.2.

I can't see an easy way to summarize the differences. However, they seem to include differences in how ids are assigned. For example, in std-ubuntu.json, this is 11002:

    "11002": {
      "crate_id": 1,
      "kind": "function",
      "path": [
        "core",
        "slice",
        "raw",
        "from_raw_parts"
      ]
    },

And in std-macos.json, this is 11002:

    "11002": {
      "crate_id": 1,
      "kind": "struct",
      "path": [
        "core",
        "core_arch",
        "simd",
        "splat",
        "JustOne"
      ]
    },

Do you have any theories as to how this could occur?

std-ubuntu.json
std-macos.json

[aDotInTheVoid]

Do you have any theories as to how this could occur?

Id’s are assigned in order of their appearance to the visitor that collects items.

https://github.com/rust-lang/rust/blob/b76036c88c1fa7b63f412e777e185336848c52d6/src/librustdoc/json/conversions.rs#L154-L158

So the Id’s being different will be caused by something else being different.

Can you reproduce this on a more minimal example?

[smoelius]

These are more minimal, but I will continue to try to find a smaller example.

Both of these were produced with the following command in the same commit of the Rust compiler:

cargo +nightly-2024-10-22 rustdoc --target x86_64-unknown-linux-gnu --manifest-path=sysroot/Cargo.toml -- -Z unstable-options --output-format=json

Here is the earliest difference:

    "292": {
      "crate_id": 12,
      "path": [
        "rustc_demangle",
        "legacy",
        "Demangle"
      ],
      "kind": "struct"
    },

    "292": {
      "crate_id": 6,
      "path": [
        "libc",
        "unix",
        "linux_like",
        "linux",
        "gnu",
        "timex"
      ],
      "kind": "struct"
    },

sysroot-ubunutu.json
sysroot-macos.json

[smoelius]

I hope this is not getting annoying, but here is a smaller example still. Each was produced by running the following command on a library whose lib.rs contained only #![no_std] (inspired by cargo-public-api):

cargo +nightly-2024-10-22 rustdoc --target x86_64-unknown-linux-gnu --manifest-path=./test-apis/example_api-v0.1.0/Cargo.toml -Zbuild-std -- -Z unstable-options --output-format=json

example_api-ubuntu.json
example_api-macos.json

Please forgive me for not knowing rustdoc internals, but the code you cited in #44 (comment) suggests that ids are generated when the JSON is rendered. Is this correct?

[aDotInTheVoid]

I hope this is not getting annoying

No, not at all.

After formatting the files with oj and oj -p 100.9 ./example_api-ubuntu.json > sorted.ubuntu.json && oj -p 100.9 ./example_api-macos.json > sorted.macos.json the diff is alot nicer:

diff --git a/./sorted.macos.json b/./sorted.ubuntu.json
index 60d302c..3050564 100644
--- a/./sorted.macos.json
+++ b/./sorted.ubuntu.json
@@ -285,35 +285,35 @@
     "264": {"crate_id": 1, "kind": "struct", "path": ["core", "core_arch", "simd", "i8x4"]},
     "265": {"crate_id": 1, "kind": "trait", "path": ["core", "ops", "bit", "BitXor"]},
     "266": {"crate_id": 1, "kind": "trait", "path": ["core", "iter", "traits", "marker", "TrustedStep"]},
-    "267": {
+    "267": {"crate_id": 1, "kind": "trait", "path": ["core", "ops", "range", "RangeBounds"]},
+    "268": {
       "crate_id": 1,
       "kind": "trait",
       "path": ["core", "core_simd", "simd", "num", "int", "SimdInt"]
     },
-    "268": {"crate_id": 1, "kind": "module", "path": ["core", "iter"]},
-    "269": {"crate_id": 1, "kind": "struct", "path": ["core", "core_arch", "x86", "cpuid", "CpuidResult"]},
+    "269": {"crate_id": 2, "kind": "struct", "path": ["compiler_builtins", "int", "big", "i256"]},
     "27": {"crate_id": 1, "kind": "struct", "path": ["core", "asserting", "Capture"]},
-    "270": {"crate_id": 1, "kind": "struct", "path": ["core", "char", "EscapeDefault"]},
-    "271": {"crate_id": 1, "kind": "struct", "path": ["core", "panic", "panic_info", "PanicMessage"]},
-    "272": {"crate_id": 1, "kind": "trait", "path": ["core", "marker", "FnPtr"]},
-    "273": {
+    "270": {"crate_id": 1, "kind": "struct", "path": ["core", "core_arch", "x86", "cpuid", "CpuidResult"]},
+    "271": {"crate_id": 1, "kind": "struct", "path": ["core", "char", "EscapeDefault"]},
+    "272": {"crate_id": 1, "kind": "module", "path": ["core", "iter"]},
+    "273": {"crate_id": 1, "kind": "struct", "path": ["core", "panic", "panic_info", "PanicMessage"]},
+    "274": {"crate_id": 1, "kind": "trait", "path": ["core", "marker", "FnPtr"]},
+    "275": {
       "crate_id": 1,
       "kind": "struct",
       "path": ["core", "num", "nonzero", "private", "NonZeroU16Inner"]
     },
-    "274": {"crate_id": 1, "kind": "struct", "path": ["core", "mem", "transmutability", "Assume"]},
-    "275": {
+    "276": {"crate_id": 1, "kind": "struct", "path": ["core", "mem", "transmutability", "Assume"]},
+    "277": {
       "crate_id": 1,
       "kind": "struct",
       "path": ["core", "iter", "adapters", "take_while", "TakeWhile"]
     },
-    "276": {"crate_id": 1, "kind": "struct", "path": ["core", "str", "pattern", "TwoWaySearcher"]},
-    "277": {"crate_id": 1, "kind": "struct", "path": ["core", "str", "BytesIsNotEmpty"]},
-    "278": {"crate_id": 1, "kind": "trait", "path": ["core", "ops", "range", "RangeBounds"]},
-    "279": {"crate_id": 1, "kind": "struct", "path": ["core", "slice", "iter", "ArrayWindows"]},
+    "278": {"crate_id": 1, "kind": "struct", "path": ["core", "str", "pattern", "TwoWaySearcher"]},
+    "279": {"crate_id": 1, "kind": "struct", "path": ["core", "str", "BytesIsNotEmpty"]},
     "28": {"crate_id": 1, "kind": "trait", "path": ["core", "ops", "drop", "Drop"]},
-    "280": {"crate_id": 1, "kind": "enum", "path": ["core", "ffi", "c_void"]},
-    "281": {"crate_id": 2, "kind": "struct", "path": ["compiler_builtins", "int", "big", "i256"]},
+    "280": {"crate_id": 1, "kind": "struct", "path": ["core", "slice", "iter", "ArrayWindows"]},
+    "281": {"crate_id": 1, "kind": "enum", "path": ["core", "ffi", "c_void"]},
     "282": {"crate_id": 1, "kind": "struct", "path": ["core", "str", "iter", "RMatches"]},
     "283": {"crate_id": 1, "kind": "trait", "path": ["core", "intrinsics", "AggregateRawPtr"]},
     "284": {"crate_id": 1, "kind": "struct", "path": ["core", "ffi", "c_str", "FromBytesUntilNulError"]},
@@ -780,10 +780,10 @@
     "601": {"crate_id": 1, "kind": "struct", "path": ["core", "cell", "Ref"]},
     "602": {"crate_id": 1, "kind": "trait", "path": ["core", "convert", "Into"]},
     "603": {"crate_id": 1, "kind": "trait", "path": ["core", "ops", "deref", "DerefPure"]},
-    "604": {"crate_id": 1, "kind": "struct", "path": ["core", "str", "pattern", "MultiCharEqPattern"]},
+    "604": {"crate_id": 1, "kind": "trait", "path": ["core", "hash", "BuildHasher"]},
     "605": {"crate_id": 1, "kind": "struct", "path": ["core", "core_arch", "simd", "i32x2"]},
-    "606": {"crate_id": 1, "kind": "trait", "path": ["core", "convert", "TryInto"]},
-    "607": {"crate_id": 1, "kind": "trait", "path": ["core", "hash", "BuildHasher"]},
+    "606": {"crate_id": 1, "kind": "struct", "path": ["core", "str", "pattern", "MultiCharEqPattern"]},
+    "607": {"crate_id": 1, "kind": "trait", "path": ["core", "convert", "TryInto"]},
     "608": {"crate_id": 1, "kind": "struct", "path": ["core", "option", "Iter"]},
     "609": {"crate_id": 1, "kind": "struct", "path": ["core", "core_arch", "x86", "splat", "JustOne"]},
     "61": {"crate_id": 1, "kind": "trait", "path": ["core", "ops", "arith", "MulAssign"]},

The difference is entirely in the paths field, so it's unclear immediately how to minimize further. It looks like (but I've not checked for sure) that all the same items are present, just with different Ids.

but the code you cited in #44 (comment) suggests that ids are generated when the JSON is rendered. Is this correct?

Yep. The Id's (that are used in JSON), are decided in the librustdoc::clean -> rustdoc_types pass (aka rendering)

[smoelius]

Yep. The Id's (that are used in JSON), are decided in the librustdoc::clean -> rustdoc_types pass (aka rendering)

Do you think it could be possible that the items are stored in different orders on the two platforms?

[aDotInTheVoid]

Do you think it could be possible that the items are stored in different orders on the two platforms?

Yeah, but I'm not really sure why. I guess you could either try:

1. Make a #![no_core] reproduction that's smaller.
2. Take the existing reproduction and use RUSTDOC_LOG=rustdoc::json=trace to try to see where things go different. You may need to build rustdoc locally if you want to add more logging.

(Or some combo of the two. Tracing may expose where in core things diverge and let you write a smaller example. Or once you have a small example of things diverging, the tracing may tell you why)

[smoelius]

Just recording a note here for myself.

This command:

        dbg!(&self.cache.external_paths.keys().collect::<Vec<_>>());

when added between these two lines:

        debug!("Constructing Output");
        let output_crate = types::Crate {

lists DefIds in different orders on Linux and macOS.

I haven't yet figured out why.

[aDotInTheVoid]

Summarizing @smoelius findings that were in other tickets:

• It turns out that while FxHashMap iteration order is deterministic on a single platform (because there's no randomization by the hasher), Hashbrown will still give different iteration order on different platforms, due to different simd implementations (sse2::Group::match_tag misimplementation? hashbrown#612). This is considered correct behaviour from HashBrown
• It turns out this was being used inside of rustdoc: Change one FxHashMap to FxIndexMap in librustdoc rust#138871
• And it wasn't flagged by rustc::potential_query_instability, because it didn't look at IntoIterator: Extend QueryStability to handle IntoIterator implementations rust#139345

@smoelius, with that last PR merged, are outputs now reproducible between OS's?