rustdoc: add hash to filename of toolchain files #101702

jsha · 2022-09-11T20:54:59Z

All static files used by rustdoc are now stored in static.files/ and their filenames include a hash of their contents. Their filenames no longer include the contents of the --resource-suffix flag. This clarifies caching semantics. Anything in static.files can use Cache-Control: immutable because any updates will show up as a new URL.

Invocation-specific files like crates-NN.js, search-index-NN.js, and sidebar-items-NN.js still get the resource suffix.

This has a useful side effect: once toolchain files aren't affected by resource suffix, it will become possible for docs.rs to include crate version in the resource suffix. That should fix a caching issue with /latest/ URLs: rust-lang/docs.rs#1593. My goal is that it should be safe to serve all rustdoc JS, CSS, and fonts with infinite caching headers, even when new versions of a crate are uploaded in the same place as old versions.

The --disable-minification flag is removed because it would vary the output of static files based on invocation flags. Instead, for rustdoc development purposes it's preferable to symlink static files to a non-minified copy for quick iteration.

Example listing:

$ cd build/x86_64-unknown-linux-gnu/doc/ && find . | egrep 'js$|css$' | egrep -v 'sidebar-items|implementors' | sort
./crates1.65.0.js
./rust.css
./search-index1.65.0.js
./source-files1.65.0.js
./static.files/ayu-2bfd0af01c176fd5.css
./static.files/dark-95d11b5416841799.css
./static.files/light-c83a97e93a11f15a.css
./static.files/main-efc63f77fb116394.js
./static.files/normalize-76eba96aa4d2e634.css
./static.files/noscript-5bf457055038775c.css
./static.files/rustdoc-7a422337900fa894.css
./static.files/scrape-examples-3dd10048bcead3a4.js
./static.files/search-47f3c289722672cf.js
./static.files/settings-17b08337296ac774.js
./static.files/settings-3f95eacb845293c0.css
./static.files/source-script-215e9db86679192e.js
./static.files/storage-26d846fcae82ff09.js

Fixes #98413

rustbot · 2022-09-11T20:55:01Z

Some changes occurred in HTML/CSS/JS.

cc @GuillaumeGomez, @Folyd, @jsha

rust-highfive · 2022-09-11T20:55:02Z

r? @GuillaumeGomez

(rust-highfive has picked a reviewer for you, use r? to override)

Nemo157 · 2022-09-11T22:06:56Z

It will be necessary to figure out how docs.rs will handle this. Currently the shared resource handling is a massive hack that allows a static resource request from anywhere but will only lookup the file from / (so won't work if those files are in a sub-directory). This is probably a good opportunity to figure out how to improve that for future doc builds.

https://github.com/rust-lang/docs.rs/blob/4abe7e9c9bd1298ec3af01878af554a5b60797e7/src/web/mod.rs#L165-L177

bors · 2022-09-16T17:27:58Z

☔ The latest upstream changes (presumably #101895) made this pull request unmergeable. Please resolve the merge conflicts.

jsha · 2022-09-29T05:52:12Z

It will be necessary to figure out how docs.rs will handle this. Currently the shared resource handling is a massive hack that allows a static resource request from anywhere but will only lookup the file from / (so won't work if those files are in a sub-directory). This is probably a good opportunity to figure out how to improve that for future doc builds.

https://github.com/rust-lang/docs.rs/blob/4abe7e9c9bd1298ec3af01878af554a5b60797e7/src/web/mod.rs#L165-L177

I filed rust-lang/docs.rs#1863 documenting my current understanding of how this hack works.

Assuming that's correct, I think there's a fairly straightforward way to make docs.rs' SharedResourceHandler work with this change: We can add code such that if the path begins with /static.files/, SharedResourceHandler looks up the full path in storage, rather than just the last component of the path. So long as that change rolls out to docs.rs before this PR lands, we should be good.

jsha · 2022-09-29T07:04:24Z

Another possible way to interface with docs.rs is demonstrated here: ea51f8a

In other words, the default value of --static-root-path is ../static.files/ (insert appropriate number of ../ depending on depth). The nice thing about this approach is that it makes it much harder to miss a mistake like the one fixed in #83094 (confusing "root path" with "static root path" because they default to the same value).

If we adopt this approach, docs.rs would have to make some changes in sync with the rustdoc release:

pass --static-root-path /rustdoc.static/ instead of --static-root-path /.
for each new toolchain, after running with --emit=toolchain-shared-resources, copy the contents of doc/static.files/ to /rustdoc.static/ in storage.

(note I specifically chose the different directory name rustdoc.static to make it clear it doesn't have to have the same name).

Nemo157 · 2022-09-29T10:34:22Z

I like being able to configure the static-root-path explicitly. To make synchronization easier I wonder if we could land that flag first (with it being a no-op), then update docs.rs to use it and copy doc/static.files if it exists, then land this change using the flag.

(I'll also try and come up with a way we can test the behavior of docs.rs against this branch so we can confirm the changes work before it's merged).

jsha · 2022-09-30T05:21:13Z

I'm a little confused - --static-root-path already exists, and docs.rs sets it to /. So, no need to land a flag.

That said, we could make a change to docs.rs ahead of this one:

start storing essential files in /rustdoc.static/
start passing --static-root-path /rustdoc.static/
adjust SharedResourceHandler to treat any paths beginning with /rustdoc.static/ as full paths rather than just filenames.

Nemo157 · 2022-09-30T11:02:10Z

Ah, for some reason I thought the current flag was --root-path. So IIUC with this change nothing would actually need to change on the docs.rs side (we would just start getting files with the hash in the "essential files" instead of the resource-suffix). But it would probably still be good for us to change the path to make it easier to apply caching headers and make it more understandable.

With ea51f8a we would have to change something because --static-root-path would correspond to docs/static.files/* rather than docs/* (we could pre-prepare for that by detecting whether the static.files directory exists and either copy from in there or from the root).

src/librustdoc/config.rs

src/librustdoc/html/static/css/rustdoc.css

GuillaumeGomez · 2022-10-05T21:40:20Z

I think there are a few issues coming up from this change. If I understand correctly, the --resource-suffix option will be added to the filename in addition to the file's hash. For predictability, I'm not sure if it's a good thing (or if we even care).

The content of static_files.rs is duplicated between the array and the struct, which isn't great. I don't think it can be done much better than that though. Maybe we could make a macro I guess so everything would be set in one place to prevent a mismatch between the struct fields and the array elements. What do you think?

I'm really not a big fan to have to update css files manually when a static file is updated. The AUTOREPLACE thing wasn't great but at least it was automated. It should be pretty rare though since we don't update them often so I guess it's a minor concern.

Another very minor concern: why static.files as folder name and not static_files for example?

One last point: I think we'll need a perf check to see if there is any impact with having such a big struct (I think not but better be aware of it if there is an impact).

Overall it seems like a really nice improvement. :)

jsha · 2022-10-05T21:48:16Z

If I understand correctly, the --resource-suffix option will be added to the filename in addition to the file's hash. For predictability, I'm not sure if it's a good thing (or if we even care).

No, there are two types of files now:

static files (they get a hash)
invocation specific files (they get a --resource-suffix)

Another very minor concern: why static.files as folder name and not static_files for example?

I wanted a name that couldn't conflict with a package or module name. I don't feel strongly about the choice and am happy to take suggestions.

Maybe we could make a macro I guess so everything would be set in one place to prevent a mismatch between the struct fields and the array elements. What do you think?

This is a good idea. I'll see what I can do.

GuillaumeGomez · 2022-10-06T08:41:18Z

Thanks for the answers, it all make more sense. Keeping the . in the folder name is the simplest solution then I think.

bors · 2022-10-06T09:11:18Z

☔ The latest upstream changes (presumably #102726) made this pull request unmergeable. Please resolve the merge conflicts.

…th, r=notriddle Fix invalid background-image file name This is a follow-up of rust-lang#101702. Apparently the image hash was the wrong one. You can see the error in https://doc.rust-lang.org/nightly/core/primitive.u16.html?search=hello too. I really need to check if I can adds check for resources load errors in `browser-ui-test`. cc `@jsha` r? `@notriddle`

…th, r=notriddle Fix invalid background-image file name This is a follow-up of rust-lang#101702. Apparently the image hash was the wrong one. You can see the error in https://doc.rust-lang.org/nightly/core/primitive.u16.html?search=hello too. I really need to check if I can adds check for resources load errors in `browser-ui-test`. cc ``@jsha`` r? ``@notriddle``

…th, r=notriddle Fix invalid background-image file name This is a follow-up of rust-lang#101702. Apparently the image hash was the wrong one. You can see the error in https://doc.rust-lang.org/nightly/core/primitive.u16.html?search=hello too. I really need to check if I can adds check for resources load errors in `browser-ui-test`. cc ```@jsha``` r? ```@notriddle```

…th, r=notriddle Fix invalid background-image file name This is a follow-up of rust-lang#101702. Apparently the image hash was the wrong one. You can see the error in https://doc.rust-lang.org/nightly/core/primitive.u16.html?search=hello too. I really need to check if I can adds check for resources load errors in `browser-ui-test`. cc ````@jsha```` r? ````@notriddle````

…th, r=notriddle Fix invalid background-image file name This is a follow-up of rust-lang#101702. Apparently the image hash was the wrong one. You can see the error in https://doc.rust-lang.org/nightly/core/primitive.u16.html?search=hello too. I really need to check if I can adds check for resources load errors in `browser-ui-test`. cc `````@jsha````` r? `````@notriddle`````

…th, r=notriddle Fix invalid background-image file name This is a follow-up of rust-lang#101702. Apparently the image hash was the wrong one. You can see the error in https://doc.rust-lang.org/nightly/core/primitive.u16.html?search=hello too. I really need to check if I can adds check for resources load errors in `browser-ui-test`. cc ``````@jsha`````` r? ``````@notriddle``````

…-errors, r=notriddle Add check in GUI test for file loading failure Since rust-lang#101702, some resources location need to be updated in case their content changed because then their hash will change too. This will prevent errors like rust-lang#104114 to happen again. The second commit is to prevent CORS errors: when a file is linked from a file itself imported, the web browser considers they come from a different domain and therefore triggers the error. The option tells the web browser to ignore this case. cc `@jsha` r? `@notriddle`

…-errors, r=notriddle Add check in GUI test for file loading failure Since rust-lang#101702, some resources location need to be updated in case their content changed because then their hash will change too. This will prevent errors like rust-lang#104114 to happen again. The second commit is to prevent CORS errors: when a file is linked from a file itself imported, the web browser considers they come from a different domain and therefore triggers the error. The option tells the web browser to ignore this case. cc ``@jsha`` r? ``@notriddle``

…-errors, r=notriddle Add check in GUI test for file loading failure Since rust-lang#101702, some resources location need to be updated in case their content changed because then their hash will change too. This will prevent errors like rust-lang#104114 to happen again. The second commit is to prevent CORS errors: when a file is linked from a file itself imported, the web browser considers they come from a different domain and therefore triggers the error. The option tells the web browser to ignore this case. cc ```@jsha``` r? ```@notriddle```

…ation, r=notriddle Fix missing minification for static files It's a fix for rust-lang#101702. The problem was that `Path::ends_with` doesn't do what we thought it does: it checks if the entire item is the last path part, no just if the "path string" ends with the given argument. So instead, I just used the `extension()` method to get the information we want. cc `@jsha` r? `@notriddle` PS: Is it worth it to add a CI test to ensure that the minification was performed on JS and CSS files or not?

…r=notriddle Add check in GUI test for file loading failure Since rust-lang/rust#101702, some resources location need to be updated in case their content changed because then their hash will change too. This will prevent errors like rust-lang/rust#104114 to happen again. The second commit is to prevent CORS errors: when a file is linked from a file itself imported, the web browser considers they come from a different domain and therefore triggers the error. The option tells the web browser to ignore this case. cc ```@jsha``` r? ```@notriddle```

…anup, r=GuillaumeGomez rustdoc: remove unused ID `mainThemeStyle` This was added in rust-lang#47620 and used to build the URL of the theme stylesheets. It isn't used any more, because rust-lang#101702 changed it so that the URL was supplied in a `<meta>` tag, which also provides the hashes of the files.

Rollup merge of rust-lang#115655 - notriddle:notriddle/rustdoc-fe-cleanup, r=GuillaumeGomez rustdoc: remove unused ID `mainThemeStyle` This was added in rust-lang#47620 and used to build the URL of the theme stylesheets. It isn't used any more, because rust-lang#101702 changed it so that the URL was supplied in a `<meta>` tag, which also provides the hashes of the files.

The static files placement by `rustdoc` changed in Rust 1.67.0 [1], but the custom code we have to replace the logo in the generated HTML files did not get updated. Thus update it to have the Linux logo again in the output. Hopefully `rustdoc` will eventually support a custom logo from a local file [2], so that we do not need to maintain this hack on our side. Link: rust-lang/rust#101702 [1] Link: rust-lang/rfcs#3226 [2] Fixes: 3ed03f4 ("rust: upgrade to Rust 1.68.2") Cc: [email protected] Signed-off-by: Miguel Ojeda <[email protected]>

The static files placement by `rustdoc` changed in Rust 1.67.0 [1], but the custom code we have to replace the logo in the generated HTML files did not get updated. Thus update it to have the Linux logo again in the output. Hopefully `rustdoc` will eventually support a custom logo from a local file [2], so that we do not need to maintain this hack on our side. Link: rust-lang/rust#101702 [1] Link: rust-lang/rfcs#3226 [2] Fixes: 3ed03f4 ("rust: upgrade to Rust 1.68.2") Cc: [email protected] Tested-by: Benno Lossin <[email protected]> Link: https://lore.kernel.org/r/[email protected] Signed-off-by: Miguel Ojeda <[email protected]>

The static files placement by `rustdoc` changed in Rust 1.67.0 [1], but the custom code we have to replace the logo in the generated HTML files did not get updated. Thus update it to have the Linux logo again in the output. Hopefully `rustdoc` will eventually support a custom logo from a local file [2], so that we do not need to maintain this hack on our side. Link: rust-lang/rust#101702 [1] Link: rust-lang/rfcs#3226 [2] Fixes: 3ed03f4 ("rust: upgrade to Rust 1.68.2") Cc: [email protected] Tested-by: Benno Lossin <[email protected]> Reviewed-by: Andreas Hindborg <[email protected]> Link: https://lore.kernel.org/r/[email protected] Signed-off-by: Miguel Ojeda <[email protected]>

[ Upstream commit cfd9672 ] The static files placement by `rustdoc` changed in Rust 1.67.0 [1], but the custom code we have to replace the logo in the generated HTML files did not get updated. Thus update it to have the Linux logo again in the output. Hopefully `rustdoc` will eventually support a custom logo from a local file [2], so that we do not need to maintain this hack on our side. Link: rust-lang/rust#101702 [1] Link: rust-lang/rfcs#3226 [2] Fixes: 3ed03f4 ("rust: upgrade to Rust 1.68.2") Cc: [email protected] Tested-by: Benno Lossin <[email protected]> Reviewed-by: Andreas Hindborg <[email protected]> Link: https://lore.kernel.org/r/[email protected] Signed-off-by: Miguel Ojeda <[email protected]> Signed-off-by: Sasha Levin <[email protected]>

…r=notriddle Add check in GUI test for file loading failure Since rust-lang/rust#101702, some resources location need to be updated in case their content changed because then their hash will change too. This will prevent errors like rust-lang/rust#104114 to happen again. The second commit is to prevent CORS errors: when a file is linked from a file itself imported, the web browser considers they come from a different domain and therefore triggers the error. The option tells the web browser to ignore this case. cc ```@jsha``` r? ```@notriddle```

jsha added T-rustdoc Relevant to the rustdoc team, which will review and decide on the PR/issue. A-rustdoc-ui Area: Rustdoc UI (generated HTML) labels Sep 11, 2022

rust-highfive assigned GuillaumeGomez Sep 11, 2022

rust-highfive added the S-waiting-on-review Status: Awaiting review from the assignee but also interested parties. label Sep 11, 2022

jsha mentioned this pull request Sep 11, 2022

rustdoc: adjust static file names for better cache configuration #98413

Closed

This comment has been minimized.

Sign in to view

jsha force-pushed the static-files2 branch from a89d029 to 3d75fc8 Compare September 12, 2022 01:27

This comment has been minimized.

Sign in to view

jsha mentioned this pull request Sep 19, 2022

invocation specific rustdoc static files are cached too long on /latest/ rust-lang/docs.rs#1593

Open

jsha force-pushed the static-files2 branch from 3d75fc8 to 749fa1a Compare September 19, 2022 22:09

This comment has been minimized.

Sign in to view

jsha force-pushed the static-files2 branch from 749fa1a to 0bf7ad3 Compare September 19, 2022 23:26

This comment has been minimized.

Sign in to view

jsha mentioned this pull request Sep 29, 2022

Update comment about buggy static file serving rust-lang/docs.rs#1863

Merged

jsha mentioned this pull request Oct 4, 2022

Change static root path rust-lang/docs.rs#1869

Closed

GuillaumeGomez reviewed Oct 5, 2022

View reviewed changes

src/librustdoc/config.rs Show resolved Hide resolved

GuillaumeGomez reviewed Oct 5, 2022

View reviewed changes

src/librustdoc/html/static/css/rustdoc.css Show resolved Hide resolved

rustbot added this to the 1.67.0 milestone Nov 5, 2022

GuillaumeGomez mentioned this pull request Nov 7, 2022

Fix invalid background-image file name #104114

Merged

GuillaumeGomez mentioned this pull request Nov 9, 2022

Add check in GUI test for file loading failure #104201

Merged

GuillaumeGomez mentioned this pull request Nov 14, 2022

Fix missing minification for static files #104404

Merged

syphar mentioned this pull request Dec 28, 2022

rustdoc referenced search.js and settings.js not via static-root-path rust-lang/docs.rs#1979

Open

jsha deleted the static-files2 branch January 16, 2023 18:08

tspiteri mentioned this pull request Jan 27, 2023

rustdoc: update Source Serif 4 from 4.004 to 4.005 #107354

Merged

notriddle mentioned this pull request Sep 7, 2023

rustdoc: remove unused ID mainThemeStyle #115655

Merged

lolbinarycat mentioned this pull request Sep 2, 2024

[rustdoc] re-add --disable-minification #129908

Open

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

rustdoc: add hash to filename of toolchain files #101702

rustdoc: add hash to filename of toolchain files #101702

jsha commented Sep 11, 2022 •

edited

Loading

rustbot commented Sep 11, 2022

rust-highfive commented Sep 11, 2022

This comment has been minimized.

Nemo157 commented Sep 11, 2022

This comment has been minimized.

bors commented Sep 16, 2022

This comment has been minimized.

This comment has been minimized.

jsha commented Sep 29, 2022

jsha commented Sep 29, 2022

Nemo157 commented Sep 29, 2022 •

edited

Loading

jsha commented Sep 30, 2022

Nemo157 commented Sep 30, 2022 •

edited

Loading

GuillaumeGomez commented Oct 5, 2022

jsha commented Oct 5, 2022

GuillaumeGomez commented Oct 6, 2022

bors commented Oct 6, 2022

rustdoc: add hash to filename of toolchain files #101702

rustdoc: add hash to filename of toolchain files #101702

Conversation

jsha commented Sep 11, 2022 • edited Loading

rustbot commented Sep 11, 2022

rust-highfive commented Sep 11, 2022

This comment has been minimized.

Nemo157 commented Sep 11, 2022

This comment has been minimized.

bors commented Sep 16, 2022

This comment has been minimized.

This comment has been minimized.

jsha commented Sep 29, 2022

jsha commented Sep 29, 2022

Nemo157 commented Sep 29, 2022 • edited Loading

jsha commented Sep 30, 2022

Nemo157 commented Sep 30, 2022 • edited Loading

GuillaumeGomez commented Oct 5, 2022

jsha commented Oct 5, 2022

GuillaumeGomez commented Oct 6, 2022

bors commented Oct 6, 2022

jsha commented Sep 11, 2022 •

edited

Loading

Nemo157 commented Sep 29, 2022 •

edited

Loading

Nemo157 commented Sep 30, 2022 •

edited

Loading