Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

use xz files for stage0 download if on system with xz #65757

Closed
stefson opened this issue Oct 24, 2019 · 4 comments · Fixed by #65932
Closed

use xz files for stage0 download if on system with xz #65757

stefson opened this issue Oct 24, 2019 · 4 comments · Fixed by #65932
Labels
C-enhancement Category: An issue proposing an enhancement or a PR with one. T-bootstrap Relevant to the bootstrap subteam: Rust's build system (x.py and src/bootstrap)

Comments

@stefson
Copy link

stefson commented Oct 24, 2019

I noticed that if ./x.py is run from a git checkout, the build script downloads the rust-std and rustc, and cargo stage0 archives from static.rust-lang.org in the tar.gz format. Those are a lot bigger in size than the also delivered tar.xz files, usually adds some 30% in download time and consumed bandwith. Do you think you could possibly change the script to use tar.xz in the future?

here's a snipet from my log:

downloading https://static.rust-lang.org/dist/2019-09-26/rust-std-1.38.0-x86_64-unknown-linux-gnu.tar.gz.sha256
running: curl -# -y 30 -Y 10 --connect-timeout 30 --retry 3 -Sf -o /var/tmp/portage/dev-lang/rust-999/temp/tmprecglk92.sha256 https://static.rust-lang.org/dist/2019-09-26/rust-std-1.38.0-x86_64-unknown-linux-gnu.tar.gz.sha256
################################################################################### 100,0%
downloading https://static.rust-lang.org/dist/2019-09-26/rust-std-1.38.0-x86_64-unknown-linux-gnu.tar.gz
running: curl -# -y 30 -Y 10 --connect-timeout 30 --retry 3 -Sf -o /var/tmp/portage/dev-lang/rust-999/temp/tmp_2fjvg1r https://static.rust-lang.org/dist/2019-09-26/rust-std-1.38.0-x86_64-unknown-linux-gnu.tar.gz
################################################################################### 100,0%
verifying /var/tmp/portage/dev-lang/rust-999/temp/tmp_2fjvg1r
@jonas-schievink jonas-schievink added T-bootstrap Relevant to the bootstrap subteam: Rust's build system (x.py and src/bootstrap) C-enhancement Category: An issue proposing an enhancement or a PR with one. labels Oct 24, 2019
@Mark-Simulacrum Mark-Simulacrum changed the title bootstrap uses tar.gz packed stage0's instead of the smaller tar.xz ones use xz files for stage0 download if on system with xz Oct 24, 2019
@guanqun
Copy link
Contributor

guanqun commented Oct 27, 2019

I did a quick check and the boostrap script uses tarfile module to do the unpacking, and as our build system need to support at least python 2.7 which unfortunately doesn't contain the support of xz out of box. see the reference here: https://docs.python.org/2/library/tarfile.html
however, when we transition to python 3, then this tarfile supports xz: https://docs.python.org/3/library/tarfile.html

@guanqun
Copy link
Contributor

guanqun commented Oct 29, 2019

I created a PR to use .tar.xz if x.py is called with python3. I tried locally with msvc and it works fine.

@stefson
Copy link
Author

stefson commented Oct 29, 2019

does this work depending on wether it's either

python2 ./x.py build for using tar.gz

or python3 ./x.py build for using tar.xz then?

@guanqun
Copy link
Contributor

guanqun commented Oct 29, 2019

that's right. because python 2 doesn't have xz support in its tarball library, see the link above. we could import other libraries but that would require extra dependencies.

JohnTitor added a commit to JohnTitor/rust that referenced this issue Nov 13, 2019
JohnTitor added a commit to JohnTitor/rust that referenced this issue Nov 13, 2019
@bors bors closed this as completed in 4ac230a Nov 13, 2019
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
C-enhancement Category: An issue proposing an enhancement or a PR with one. T-bootstrap Relevant to the bootstrap subteam: Rust's build system (x.py and src/bootstrap)
Projects
None yet
Development

Successfully merging a pull request may close this issue.

3 participants