BinaryBuilder.jl: Construct The Future

[staticfloat]

Big BB overhaul! Highlights:

• BB is now Artifact-aware! It generates JLL packages as a part of its deployment now. See this WIP blog post for more info on these.
• Incidientally, compiler shard downloads are also now handled by the Pkg.Artifacts system, and so BB itself now requires Julia 1.3+.
• Compilers are now JIT-generated wrapper scripts; just calling cc will now invoke a shell script that calls ${target}-gcc, which in turn will call /opt/${target}/bin/${target}-gcc ${platform_cflags} ${platform_ldflags} which allows us to not have to worry so much when build systems don't include $(LDFLAGS) automatically.

Things yet to do before merging this:

• Fix docker runner (we can probably do something clever with mounts with our new artifacts support)
• As an extra bonus, get docker working on Windows as well! (Two small fixes for running Docker on Windows #418) :D
• Fix qemu runner (upgrade it to use artifacts instead of downloading things directly, etc...) skipping for now, docker will be the official way to do this on Mac/Windows
• Teach the wizard about JLL packages.
• Make the wizard not try to deploy to Travis, just open PRs against Yggdrasil.

[visr]

The "get docker working on Windows" checkbox above may not hold quite yet :)

On julia 1.3rc4 with BINARYBUILDER_RUNNER=docker, and Docker running, I tried:

using BinaryBuilder; BinaryBuilder.runshell(Linux(:x86_64, libc=:glibc))

This gives the following error message from Docker

docker: Error response from daemon: OCI runtime create failed: container_linux.go:345: starting container process caused "exec: \"/docker_entrypoint.sh\": stat /docker_entrypoint.sh: no such file or directory": unknown.

full error

docker: Error response from daemon: OCI runtime create failed: container_linux.go:345: starting container process caused "exec: \"/docker_entrypoint.sh\": stat /docker_entrypoint.sh: no such file or directory": unknown.
ERROR: failed process: Process(`docker run --rm --privileged -w '/\workspace\' -v 'C:\Users\visser_mn\.julia\artifacts\2c08629dc0616843c016da9659137d1f0121fc38:/opt/x86_64-linux-gnu/PlatformSupport-2019.10.11:ro' -v 'C:\Users\visser_mn\.julia\artifacts\f2124900465e4fa506f8b1e274074eb92d68d46b:/opt/x86_64-linux-gnu/GCCBootstrap-4.8.5:ro' -v 'C:\Users\visser_mn\.julia\artifacts\a2ee62c58a436ef3c6405c77ac00477d82dc967f:/opt/x86_64-linux-musl/LLVMBootstrap-8.0.0:ro' -v 'C:\Users\visser_mn\.julia\artifacts\96b763f0eb3690e28ca2a9a01840dedca4410acb:/opt/x86_64-linux-musl/PlatformSupport-2019.10.11:ro' -v 'C:\Users\visser_mn\.julia\artifacts\2e6f62648ec0497cc25774aa175471ad41b6172a:/opt/x86_64-linux-musl/GCCBootstrap-4.8.5:ro' -v 'D:\repo\julia\builder\GEOSBuilder:/workspace' -v 'C:\Users\visser_mn\AppData\Local\Temp\jl_NvZBYO:/opt/bin' -e HOSTFC=x86_64-linux-musl-gfortran -e HOSTSTRIP=x86_64-linux-musl-strip -e OBJCOPY_BUILD=x86_64-linux-musl-objcopy -e HISTFILE=/meta/.bash_history -e HOSTRANLIB=x86_64-linux-musl-ranlib -e VERBOSE=false -e BUILD_STRIP=x86_64-linux-musl-strip -e RUSTUP_TOOLCHAIN=stable-x86_64-unknown-linux-gnu -e LLVM_TARGET=x86_64-linux-gnu -e BUILD_AS=x86_64-linux-musl-as -e RUSTC=rustc -e BUILD_READELF=x86_64-linux-musl-readelf -e LD_FOR_BUILD=x86_64-linux-musl-ld -e NM_FOR_BUILD=x86_64-linux-musl-nm -e HOSTCC=x86_64-linux-musl-gcc -e BUILD_CC=x86_64-linux-musl-gcc -e CXX_BUILD=x86_64-linux-musl-g++ -e RUSTUP_HOME=/opt/x86_64-linux-gnu -e HOSTREADELF=x86_64-linux-musl-readelf -e OBJC=objc -e prefix=/workspace/destdir -e READELF_FOR_BUILD=x86_64-linux-musl-readelf -e OBJDUMP_BUILD=x86_64-linux-musl-objdump -e AS_BUILD=x86_64-linux-musl-as -e CMAKE_TARGET_TOOLCHAIN=/opt/x86_64-linux-gnu/x86_64-linux-gnu.cmake -e PKG_CONFIG_PATH=/workspace/destdir/lib/pkgconfig:/workspace/destdir/share/pkgconfig -e AR_FOR_BUILD=x86_64-linux-musl-ar -e FC_FOR_BUILD=x86_64-linux-musl-gfortran -e nproc=8 -e CC=cc -e target=x86_64-linux-gnu -e HOSTOBJDUMP=x86_64-linux-musl-objdump -e LD_BUILD=x86_64-linux-musl-ld -e HOSTCXX=x86_64-linux-musl-g++ -e bindir=/workspace/destdir/bin -e CCACHE_COMPILERCHECK=content -e HOSTNM=x86_64-linux-musl-nm -e AR_BUILD=x86_64-linux-musl-ar -e LIPO_BUILD=x86_64-linux-musl-lipo -e nbits=64 -e MACHTYPE=x86_64-linux-musl -e RANLIB_BUILD=x86_64-linux-musl-ranlib -e CARGO_BUILD_TARGET=x86_64-unknown-linux-gnu -e CC_BUILD=x86_64-linux-musl-gcc -e DSYMUTIL_BUILD=llvm-dsymutil -e OBJDUMP_FOR_BUILD=x86_64-linux-musl-objdump -e ZERO_AR_DATE=1 -e CXX=c++ -e LD_LIBRARY_PATH=/usr/local/lib64:/usr/local/lib:/usr/local/lib:/usr/lib:/lib64:/lib:/opt/x86_64-linux-musl/x86_64-linux-musl/lib64:/opt/x86_64-linux-musl/x86_64-linux-musl/lib:/opt/x86_64-linux-gnu/x86_64-linux-gnu/lib64:/opt/x86_64-linux-gnu/x86_64-linux-gnu/lib:/opt/x86_64-linux-gnu/x86_64-linux-gnu/lib64:/opt/x86_64-linux-gnu/x86_64-linux-gnu/lib:/workspace/destdir/lib64:/workspace/destdir/lib -e HOSTOBJCOPY=x86_64-linux-musl-objcopy -e HOSTLIPO=x86_64-linux-musl-lipo -e BUILD_RANLIB=x86_64-linux-musl-ranlib -e STRIP_BUILD=x86_64-linux-musl-strip -e HOSTAR=x86_64-linux-musl-ar -e USER=julia -e GOPATH=/workspace/.gopath -e GO=go -e rust_host=x86_64-unknown-linux-gnu -e BUILD_OBJDUMP=x86_64-linux-musl-objdump -e exeext= -e SOURCE_DATE_EPOCH=0 -e NM_BUILD=x86_64-linux-musl-nm -e CXX_FOR_BUILD=x86_64-linux-musl-g++ -e BUILD_OBJCOPY=x86_64-linux-musl-objcopy -e rust_target=x86_64-unknown-linux-gnu -e USE_CCACHE=false -e FC=gfortran -e HOSTDSYMUTIL=llvm-dsymutil -e BUILD_FC=x86_64-linux-musl-gfortran -e CC_FOR_BUILD=x86_64-linux-musl-gcc -e GOARM=7 -e CARGO_HOME=/opt/x86_64-linux-gnu -e CARGO=cargo -e dlext=so -e SRC_NAME= -e PATH=/opt/bin:/opt/x86_64-linux-gnu/bin:/opt/x86_64-linux-musl/bin:/usr/local/bin:/usr/local/sbin:/usr/bin:/usr/sbin:/bin:/sbin:/workspace/destdir/bin -e FC_BUILD=x86_64-linux-musl-gfortran -e AS_FOR_BUILD=x86_64-linux-musl-as -e BUILD_AR=x86_64-linux-musl-ar -e MESON_HOST_TOOLCHAIN=/opt/x86_64-linux-musl/x86_64-linux-musl.meson -e BUILD_CXX=x86_64-linux-musl-g++ -e CMAKE_HOST_TOOLCHAIN=/opt/x86_64-linux-musl/x86_64-linux-musl.cmake -e proc_family=intel -e BUILD_LD=x86_64-linux-musl-ld -e V=false -e STRIP_FOR_BUILD=x86_64-linux-musl-strip -e PKG_CONFIG_SYSROOT_DIR=/workspace/destdir -e TERM=screen -e MESON_TARGET_TOOLCHAIN=/opt/x86_64-linux-gnu/x86_64-linux-gnu.meson -e OBJCOPY_FOR_BUILD=x86_64-linux-musl-objcopy -e RANLIB_FOR_BUILD=x86_64-linux-musl-ranlib -e BUILD_LIPO=x86_64-linux-musl-lipo -e GOCACHE=/workspace/.gocache -e HOSTLD=x86_64-linux-musl-ld -e 'HIDDEN_PS1=\[\]sandbox\[\]:\[\]${PWD//$WORKSPACE/\\$\{WORKSPACE\}}\[\] \$ ' -e DSYMUTIL_FOR_BUILD=llvm-dsymutil -e BUILD_DSYMUTIL=llvm-dsymutil -e LIPO_FOR_BUILD=x86_64-linux-musl-lipo -e LLVM_HOST_TARGET=x86_64-linux-musl -e READELF_BUILD=x86_64-linux-musl-readelf -e libdir=/workspace/destdir/lib -e BUILD_NM=x86_64-linux-musl-nm -e bb_target=x86_64-linux-gnu -e WORKSPACE=/workspace -e HOSTAS=x86_64-linux-musl-as -ti -i julia_binarybuilder_rootfs:v2019.10.17 /bin/bash -l`, ProcessExited(127)) [127]

Stacktrace:
 [1] pipeline_error at .\process.jl:525 [inlined]
 [2] #run#569(::Bool, ::typeof(run), ::Cmd) at .\process.jl:440
 [3] run at .\process.jl:438 [inlined]
 [4] #run_interactive#254(::Nothing, ::Nothing, ::Nothing, ::Bool, ::typeof(BinaryBuilder.run_interactive), ::BinaryBuilder.DockerRunner, ::Cmd) at C:\Users\visser_mn\.julia\dev\BinaryBuilder\src\DockerRunner.jl:194
 [5] #run_interactive at .\none:0 [inlined]
 [6] #runshell#189 at C:\Users\visser_mn\.julia\dev\BinaryBuilder\src\Runner.jl:606 [inlined]
 [7] #runshell at .\none:0 [inlined]
 [8] #runshell#190(::Bool, ::Base.Iterators.Pairs{Union{},Union{},Tuple{},NamedTuple{(),Tuple{}}}, ::typeof(BinaryBuilder.runshell), ::Type{BinaryBuilder.DockerRunner}, ::Linux) at C:\Users\visser_mn\.julia\dev\BinaryBuilder\src\Runner.jl:610
 [9] runshell(::Type{BinaryBuilder.DockerRunner}, ::Linux) at C:\Users\visser_mn\.julia\dev\BinaryBuilder\src\Runner.jl:610
 [10] #runshell#188(::Base.Iterators.Pairs{Union{},Union{},Tuple{},NamedTuple{(),Tuple{}}}, ::typeof(BinaryBuilder.runshell), ::Linux) at C:\Users\visser_mn\.julia\dev\BinaryBuilder\src\Runner.jl:602
 [11] runshell(::Linux) at C:\Users\visser_mn\.julia\dev\BinaryBuilder\src\Runner.jl:602
 [12] top-level scope at REPL[1]:1

Searching online I found https://stackoverflow.com/q/38905135. It seems it could be a Windows line endings thing, or perhaps an bash/ash shebang thing. Not sure how I can easily check that though.

[staticfloat]

I've just checked the last box, so tomorrow I'm likely going to merge this (even though the tests need to be rewritten) then start in on the tests.

@visr I hear you, I will look into it, it's most likely that the docker import failed and you have a docker image that is empty. Try clearing out your binarybuilder rootfs image (find it with docker images then docker rmi <id> and run it again.

[visr]

Ok thanks for the tip. It does help relieving one error, but then I bump into issues with tar -c -C $(rootfs_path) . in the same pipeline (tar is not available and wsl tar gives other path issues).

But don't let it stop you from merging, I can always make separate issues for Windows, I think there are a few more. In any case I still need 1872752.