Making docs optional again

[MJDSys]

See as docs are required again due to #903, can we track what's necessary to make them optional again here? As one of my laptops doesn't have space for the docs (16G disk :( ), I'd be interested in helping fix the relevant issues. The two mentioned issues I've seen are:

• rustup doc should deal with absent doc instead of silently failing #901, which would require a useful message/action when docs are requested and not present
• Make rustup-init install docs by default #891, which would install the docs by default to help new people
• Teach rustup/rustbuild about optional components #1006 Add support for default installed optional features to manifest

Are there any others? And if you prefer to track this information elsewhere, please close this bug and point me in the right direction!

[vks]

Another problem are network file systems. For me it takes minutes to install the docs, the rest finishes almost instantly.

[pravic]

I'd prefer to leave docs as optional since I use the online version only and don't want to waste disk space or increase rustup update time.

[brson]

The major missing feature is that the manifest format needs to be extended to understand optional features: #1006.

[Boscop]

Yes, please make docs optional again! We work a lot with Raspberry Pis where space is very limited!

[wagenet]

This would certainly be nice for CI.

[Boscop]

And when updating a nightly or when different projects use different nightlies (with a rust-toolchain file) it always takes a long time to install/update them, it would be much faster without the docs.

[pravic]

The major missing feature is that the manifest format needs to be extended to understand optional features

Is it still? Because now we have plenty of optional components in rustup (rust-std-*, rust-src, clippy-preview, etc).

The problem is (among others) that $toolchain/share/doc contains nearly 15000 files which is a big pressure on disk during toolchain updates - on Windows it takes a really significant amount of time to install rust-docs.

From #903:

It seems that most people are comforted knowing docs are available by default, so I want to turn them back on, then add some more options to remove them.

I am fine with default (though I really doubt that those who really need docs can't install them as a component), but let's make them as opt-out, so rustup would remember during the next update that rust-docs doesn't need to be downloaded.

[Boscop]

I can confirm that it takes a long time to install the docs on windows, and I'm dealing with multiple nightlies for different projects (with rust-toolchain files)..

But I'd prefer if the docs were opt-in, because most people only use the online docs. An internet connection is required to be able to use Rust anyway..

[Arnavion]

The major missing feature is that the manifest format needs to be extended to understand optional features

Is it still? Because now we have plenty of optional components in rustup (rust-std-*, rust-src, clippy-preview, etc).

@pravic The issue is that the manifest does not have a way express opt-out components, only opt-in.

[shepmaster]

The playground docker images are also about 300MB larger than they need to be because of the docs (which I'm working on removing at the moment).

[Geobert]

Installation of docs makes rustup hang at the end of the install process: #1464

[pronebird]

It takes ages for us to install Rust on Travis/Windows. Any progress on sorting this out? Really delays builds by 10 minutes or more which is impossible to work in productive way without wasting 2 hours on CI each day.

[nrc]

I have a plan for profiles which will address this. I hope to implement in the next few weeks

[johnthagen]

@nrc I'm sure this is nothing new to you, but wanted to provide another data point. Building a "Hello World" executable on Travis CI takes 8x longer on Windows. The vast majority of the time is spent installing rust-docs.

https://travis-ci.org/johnthagen/min-sized-rust/builds/465623377

The profile work will have a great impact on CI. ❤️

[Kampfkarren]

@nrc What's the status on that? rust-docs takes minutes to install when I never use them.

[thedavidmeister]

@brson @nrc is there anything i can do to help progress this?

[ashthespy]

@kinnison Any way the community could help? If there are smaller sub tasks, I don't mind trying my hand at it.

[kinnison]

There is a draft PR at #1673 which needs updating against the reorganisation of the codebase and such. I'm not sure how much more work will be needed after that, though @nrc will hopefully have something to say on what his plans are.

[ice1000]

I am very looking forward to see an update on this issue.

[bestia-dev]

"rustup docs" can be smart: if there is offline docs use it else go online.
The existing required "docs component" could be for online use.
A new optional "DocsOffline component" could be for offline docs.
Does it make sense?

[DoumanAsh]

Until there is no way to opt-out of docs, they should not be installed by default

error: component 'rust-docs' for target 'x86_64-pc-windows-msvc' is required for toolchain 'nightly-x86_64-pc-windows-msvc' and cannot be removed

This is especially making me happy

[softprops]

I'm fine with docs by default as long as there is a way to opt out for use cases where they are not needed.

[thedavidmeister]

we moved to azure + appveyor "enterprise" and built images containing rust already installed

it was the only way to avoid timeouts

it's not cheap tho

[ashthespy]

There is hope!

[softprops]

Is there a place where I can read up on profiles. I've only found this open pull without a description #1673

[kinnison]

Currently there's not anything concrete to read about, sorry, no.

[ashthespy]

@softprops
Maybe this gives you a starting point?

let profiles = &[
                ("minimal", vec!["rustc", "cargo", "rust-std"]),
                ("default", vec!["rustc", "cargo", "rust-std", "rust-docs"]),
                (
                    "complete",
                    vec!["rustc", "cargo", "rust-std", "rust-docs", "rust-src"],
                ),
            ];

[nrc]

Is there a place where I can read up on profiles. I've only found this open pull without a description

You can see the initial design discussion at https://internals.rust-lang.org/t/pre-rfc-improve-access-to-rustup-components/8318

[softprops]

Thanks @ashthespy. I'll follow this issue's progress.

For context I'm interested in slimming down the docker builder image for aws lambda softprops/lambda-rust#1. I really just need to cargo build tool chain. The bundled docs artifacts aren't useful in this context and yet they aren't currently possible to opt out of. This minimal profile looks great!

[shepmaster]

@kinnison is this issue addressed by the minimal profile now?

[shepmaster]

This appears to do what I want and not include the docs!

curl https://sh.rustup.rs -sSf | sh -s -- -y --profile minimal --default-toolchain "${channel}"

info: downloading installer
info: profile set to 'minimal'
info: syncing channel updates for 'stable-x86_64-unknown-linux-gnu'
info: latest update on 2019-09-26, rust version 1.38.0 (625451e37 2019-09-23)
info: downloading component 'cargo'
info: downloading component 'rust-std'
info: downloading component 'rustc'
info: installing component 'cargo'
info: installing component 'rust-std'
info: installing component 'rustc'

❤️

[vks]

It is also possible again to remove the rust-docs component (independently of the chosen profile).

[shepmaster]

@vks yup, I was very happy to do

for tc in 1.0.0 1.2.0 1.4.0 1.5.0 1.12.1 1.14.0 1.15.1 1.16.0 1.17.0 1.18.0 1.19.0 1.20.0 1.21.0 1.22.1 1.23.0 1.24.1 1.25.0 1.26.2 1.27.2 1.28.0 1.29.2 1.30.1 1.31.0 1.31.1 1.32.0 1.34.0 1.35.0; do
    rustup component remove --toolchain $tc rust-docs
done

Note that this doesn't work for very old releases (like 1.0.0), which predate the concept of components. However, I'll take the savings for the versions which do!

[kinnison]

I believe this is now covered so I'm going to close the issue. Thanks all for your patience.

[leodotcloud]

Leaving a trail for those who are looking to do this with the offline installer:

export RUST_VERSION=1.60.0 && \
    export TMP_RUST_DIR=/tmp/rust && \
    mkdir -p "${TMP_RUST_DIR}" && \
    cd "${TMP_RUST_DIR}" && \
    curl -sLf "https://static.rust-lang.org/dist/rust-${RUST_VERSION}-x86_64-unknown-linux-gnu.tar.gz" |  \
      tar xvzf - -C "${TMP_RUST_DIR}" --strip-components=1 --exclude=rust-docs && \
    ./install.sh --without=rust-docs && \
    cd /tmp && rm -rf "${TMP_RUST_DIR}"