Disk usage 2-3x somewhere around 1.48

[infinity0]

Not sure if anyone else is seeing this, but the Debian builds of rustc started to take up way more disk space around 1.48, in some cases more than a 3x increase:

arch	1.47 → 1.48	1.48 → 1.48	1.49 beta → 1.49
all [*]	7 GB	20.32 GB	21.73 GB
amd64	8.36 GB	32.69 GB	34.62 GB
arm64	7.61 GB	25.03 GB	26.85 GB
armel	7.39 GB	9.95 GB	11.21 GB
armhf	7.38 GB	9.94 GB	11.2 GB
i386	7.74 GB	27.29 GB	28.84 GB
mips64el	9.47 GB	33.6 GB	36.63 GB
mipsel	8.94 GB	11.5 GB	12.85 GB
powerpc	7.64 GB	9.88 GB	(failed)
ppc64	8.12 GB	33.06 GB	35.1 GB
ppc64el	14.4 GB	39.48 GB	41.75 GB
riscv64	7.7 GB	37.61 GB	(failed)
s390x	8.24 GB	39.31 GB	42.09 GB
sparc64	7.97 GB	31.57 GB	33.39 GB

Data from Debian rustc logs; the column headings mean (version of stage0 used -> stable version being built)

[*] architecture-independent artifacts such as libstd:wasm32 and docs; builds rustc during the process but this is discarded

LLVM 11 was used for all of these builds.

[jyn514]

@infinity0 is this with incremental enabled? Could you paste debian/config.toml?

[infinity0]

We are not enabling incremental, the relevant parts of config.toml is as follows, but this hasn't changed across the 3 different builds listed in the above table:

[build]
submodules = false
vendor = true
locked-deps = false
verbose = 2

[install]
prefix = "/usr"

[target.DEB_BUILD_RUST_TYPE]
llvm-config = "LLVM_DESTDIR/usr/lib/llvm-LLVM_VERSION/bin/llvm-config"
linker = "DEB_BUILD_GNU_TYPE-gcc"

[target.DEB_HOST_RUST_TYPE]
llvm-config = "LLVM_DESTDIR/usr/lib/llvm-LLVM_VERSION/bin/llvm-config"
linker = "DEB_HOST_GNU_TYPE-gcc"

[target.wasm32-wasi]
wasi-root = "/usr"

[llvm]
link-shared = true

[rust]
jemalloc = false
optimize = true
dist-src = false

channel = "stable"

debuginfo-level = 2
debuginfo-level-std = 2
rpath = false
remap-debuginfo = true

verbose-tests = true
backtrace-on-ice = true

We also have some Debian-specific patches but these are generally fairly trivial and also haven't changed significantly across the builds.

One thing that may be significant, is the first column used LLVM 11.0.0 but the others used LLVM 11.0.1. Does rust official builds record the disk usage? If so it would be interesting to see this info for before and after PR #80796.

[Mark-Simulacrum]

We don't track those metrics (yet), but it may be possible to pull them from logs. Do you have a tarball or some other way of seeing what is using the disk space in your case?

[klensy]

I've see

1.40.0+dfsg1-5 (sid)

amd64	Maybe-Successful	2020-01-05 17:11:43	x86-ubc-01	1h 50m	20.07 GB

So it isn't specific to version mentioned?

[nagisa]

What is the actual thing that's being measured here?

[infinity0]

@Mark-Simulacrum no sadly not, it all gets discarded. I can probably reproduce by rebuilding an old version but that'll take some time.

@nagisa Extra disk space consumed by the build process.

It's possibly debuginfo-related - the architectures using the least space are the ones where we are setting debuginfo-level = 0 due to #45854, generally the 32-bit architectures. But it's weird that it's changing across versions when the config is staying the same.

[apiraino]

Issue has been discussed during the compiler team meeting on Zulip.

We acknowledge this is definitively a major annoyance and we need to improve/investigate our reporting on our builds (currently lacking a bit of depth in this regard).

@rustbot label -I-prioritize +P-medium

[apiraino]

Could some bisecting help isolating why the debug artifacts have been growing so much?

@rustbot ping icebreakers-cleanup-crew

[rustbot]

Hey Cleanup Crew ICE-breakers! This bug has been identified as a good
"Cleanup ICE-breaking candidate". In case it's useful, here are some
instructions for tackling these sorts of bugs. Maybe take a look?
Thanks! <3

cc @AminArria @camelid @chrissimpkins @contrun @DutchGhost @elshize @h-michael @HallerPatrick @hdhoang @hellow554 @henryboisdequin @imtsuki @JamesPatrickGill @kanru @KarlK90 @LeSeulArtichaut @MAdrianMattocks @matheus-consoli @mental32 @nmccarty @Noah-Kennedy @pard68 @PeytonT @pierreN @Redblueflame @RobbieClarken @RobertoSnap @robjtede @SarthakSingh31 @shekohex @sinato @smmalis37 @steffahn @Stupremee @tamuhey @turboladen @woshilapin @yerke

[chordtoll]

I was able to reproduce this one locally, and it appears to be a change in how cargo resolves the value of the debuginfo parameter to pass to rustc. With everything else held constant, a build using cargo rust-1.42.0 uses ~8G of space, and cargo rust-1.46.0 uses ~33G. I'm working on bisecting further, and hope to have a specific changeset identified in a few days.

[chordtoll]

Looks like the change is caused by rust-lang/cargo#5506, which stabilizes the config-profile feature. We've been setting the CARGO_PROFILE_RELEASE_DEBUG = 2 environment variable, and prior to this merge, this was ignored, since we were not setting the -Z config-profile option.

[chordtoll]

I'm not yet sure where that environment variable is set, but my initial impression is that this is likely a misconfiguration in the debian build config. I'll investigate further.

[pnkfelix]

Visting for P-high review.

Based on the discussion thread here, I am inferring that this is a misconfiguration issue in the debian build config, as stated by @chordtoll above.

(While it would be nice to figure out ways to reduce the disk size overhead of our debuginfo-level=2 generation, that is not a P-high issue, I think.)

Closing as "working as expected"

[wesleywiser]

I agree, this does seem to be expected as a result of more crates being compiled with debuginfo=2. If however, there is some specific set of crates that don't need to be compiled with debuginfo=2 and lowering that would make a noticeable difference in terms of disk space used, I think it would potentially be reasonable to expose a configuration option in config.toml for those.