Skip to content

Move riscv64 to Azure Linux #1098

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Merged
merged 1 commit into from
Jun 17, 2024
Merged

Move riscv64 to Azure Linux #1098

Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
1 commit
Select commit
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
34 changes: 34 additions & 0 deletions src/azurelinux/3.0/net9.0/cross/riscv64-alpine/Dockerfile
View file
Open in desktop
Original file line number Diff line number Diff line change
@@ -0,0 +1,34 @@
FROM mcr.microsoft.com/dotnet-buildtools/prereqs:azurelinux-3.0-crossdeps-builder-net9.0-local AS builder
ARG ROOTFS_DIR=/crossrootfs/riscv64

RUN /scripts/eng/common/cross/build-rootfs.sh riscv64 alpine3.20 --skipunmount

RUN TARGET_TRIPLE="riscv64-alpine-linux-musl" && \
cmake -S llvm-project.src/runtimes -B runtimes \
-DCMAKE_BUILD_TYPE=Release \
-DCMAKE_ASM_COMPILER=clang \
-DCMAKE_C_COMPILER=clang \
-DCMAKE_CXX_COMPILER=clang++ \
-DCMAKE_ASM_COMPILER_TARGET="$TARGET_TRIPLE" \
-DCMAKE_C_COMPILER_TARGET="$TARGET_TRIPLE" \
-DCMAKE_CXX_COMPILER_TARGET="$TARGET_TRIPLE" \
# We're going to link the static libraries we build here into PIC images, so build the static libraries as PIC.
-DCMAKE_POSITION_INDEPENDENT_CODE=ON \
-DCMAKE_SYSROOT="$ROOTFS_DIR" \
# Specify linker to use for exes directly for CMake toolchain detection
-DCMAKE_EXE_LINKER_FLAGS="-fuse-ld=lld" \
# Don't search for tools in the sysroot as we're cross-compiling
-DCMAKE_FIND_ROOT_PATH_MODE_PROGRAM="NEVER" \
-DLLVM_USE_LINKER=lld \
-DLLVM_ENABLE_RUNTIMES="libcxx" \
-DLIBCXX_ENABLE_SHARED=OFF \
-DLIBCXX_HAS_MUSL_LIBC=ON \
-DLIBCXX_CXX_ABI=libstdc++ \
-DLIBCXX_CXX_ABI_INCLUDE_PATHS="$ROOTFS_DIR/usr/include/c++/13.2.1/;$ROOTFS_DIR/usr/include/c++/13.2.1/$TARGET_TRIPLE" && \
cmake --build runtimes -j $(getconf _NPROCESSORS_ONLN) && \
cmake --install runtimes --prefix "$ROOTFS_DIR/usr"

FROM mcr.microsoft.com/dotnet-buildtools/prereqs:azurelinux-3.0-crossdeps-llvm-net9.0-local
ARG ROOTFS_DIR=/crossrootfs/riscv64

COPY --from=builder $ROOTFS_DIR $ROOTFS_DIR
50 changes: 50 additions & 0 deletions src/azurelinux/3.0/net9.0/cross/riscv64/Dockerfile
View file
Open in desktop
Original file line number Diff line number Diff line change
@@ -0,0 +1,50 @@
FROM mcr.microsoft.com/dotnet-buildtools/prereqs:azurelinux-3.0-crossdeps-builder-net9.0-local AS builder
ARG ROOTFS_DIR=/crossrootfs/riscv64

# Install the latest debootstrap
RUN tdnf remove -y debootstrap && \
git clone --depth 1 "https://git.launchpad.net/ubuntu/+source/debootstrap" /debootstrap && \
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

from what I can see noble was added in debootstrap 1.0.134 so I'd prefer getting the .tar.gz from https://ftp.debian.org/debian/pool/main/d/debootstrap/ instead of git

Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

also would be good to file an issue tracking this so we can remove it once azurelinux is updated

Copy link
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

It is a shell script, not an executable. We will need to setup a fakechroot etc. to make it work from the tarball, see https://src.fedoraproject.org/rpms/debootstrap/blob/rawhide/f/debootstrap.spec. BTW, these steps are taken from the readme of lanchpad official package, to run without installing. Lets keep this for now. I will remove it once newer version of azlinux is published in a few months.

Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I looked into this a bit more and it seems the only relevant change is that scripts/noble is a symlink to scripts/gutsy. I think we can just create that symlink?

Copy link
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This step took 8s (out of 145mins)

2024-06-16T18:06:48.9108010Z #7 [builder 2/4] RUN tdnf remove -y debootstrap &&     git clone --depth 1 "https://git.launchpad.net/ubuntu/+source/debootstrap" /debootstrap &&     chmod a+x /debootstrap/debootstrap &&     ln -s /debootstrap/debootstrap -t /usr/local/bin
2024-06-16T18:06:49.0377831Z #7 0.284 Loaded plugin: tdnfrepogpgcheck
2024-06-16T18:06:50.5655548Z #7 1.812 
2024-06-16T18:06:50.5655730Z #7 1.812 Removing:
2024-06-16T18:06:50.5656173Z #7 1.812 debootstrap      noarch       1.0.128+nmu2-1.azl3 @System      193.98k     0.00b
2024-06-16T18:06:50.5658780Z #7 1.812 
2024-06-16T18:06:50.5659290Z #7 1.812 Total installed size: 193.98k
2024-06-16T18:06:50.5659535Z #7 1.812 Total download size:   0.00b
2024-06-16T18:06:50.6800990Z #7 1.927 Testing transaction
2024-06-16T18:06:50.8734176Z #7 1.930 Running transaction
2024-06-16T18:06:50.8734709Z #7 1.974 Removing: debootstrap-1.0.128+nmu2-1.azl3.noarch
2024-06-16T18:06:50.8735050Z #7 2.120 Cloning into '/debootstrap'...
2024-06-16T18:06:56.8048205Z #7 DONE 8.1s

riscv64 is a community-supported platform, I think we can live with it for a while. Unless you are feeling too strongly about it, I'd say lets keep it for now and we can delete it once tdnf picks up the latest version in a few weeks or so. (we have been using and trusting launchpad aka Canonical packages forever, so there is no security risk)

LMK 😅

Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

@am11 it's not about the time, I'm more worried about cloning directly from git HEAD. We don't really know when Azure Linux will bump to new debootstrap and if a simple RUN ln -s gutsy /usr/share/debootstrap/scripts/noble works too I prefer that :)

In fact I tried that and it seems to work for me locally.

Copy link
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Lets merge this to unblock others and then follow-up on it. I don't wana run another 4 hours CI job for it. That git-clone step hasn't changed in years and hasn't broken anyone. It can sustain a few days. Fair?

Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I agree, sounds good.

Copy link
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Upstream PR microsoft/azurelinux#9422

chmod a+x /debootstrap/debootstrap && \
ln -s /debootstrap/debootstrap -t /usr/local/bin

RUN /scripts/eng/common/cross/build-rootfs.sh riscv64 noble --skipunmount

RUN TARGET_TRIPLE="riscv64-linux-gnu" && \
CLANG_MAJOR_VERSION=$(clang --version | grep -oP "(?<=version )\d+") && \
cmake -S llvm-project.src/runtimes -B runtimes \
-DCMAKE_BUILD_TYPE=Release \
-DCMAKE_ASM_COMPILER=clang \
-DCMAKE_C_COMPILER=clang \
-DCMAKE_CXX_COMPILER=clang++ \
-DCMAKE_ASM_COMPILER_TARGET="$TARGET_TRIPLE" \
-DCMAKE_C_COMPILER_TARGET="$TARGET_TRIPLE" \
-DCMAKE_CXX_COMPILER_TARGET="$TARGET_TRIPLE" \
# We're going to link the static libraries we build here into PIC images, so build the static libraries as PIC.
-DCMAKE_POSITION_INDEPENDENT_CODE=ON \
-DCMAKE_SYSROOT="$ROOTFS_DIR" \
# Specify linker to use for exes directly for CMake toolchain detection
-DCMAKE_EXE_LINKER_FLAGS="-fuse-ld=lld" \
# Don't search for tools in the sysroot as we're cross-compiling
-DCMAKE_FIND_ROOT_PATH_MODE_PROGRAM="NEVER" \
-DLLVM_USE_LINKER=lld \
-DLLVM_ENABLE_RUNTIMES="libcxx;compiler-rt" \
-DLIBCXX_ENABLE_SHARED=OFF \
-DLIBCXX_CXX_ABI=libstdc++ \
-DLIBCXX_CXX_ABI_INCLUDE_PATHS="$ROOTFS_DIR/usr/include/c++/13/;$ROOTFS_DIR/usr/include/$TARGET_TRIPLE" \
-DCOMPILER_RT_CXX_LIBRARY="libcxx" \
-DCOMPILER_RT_STATIC_CXX_LIBRARY=ON \
-DCOMPILER_RT_BUILD_SANITIZERS=OFF \
-DCOMPILER_RT_BUILD_MEMPROF=OFF \
# The libfuzzer build in LLVM doesn't correctly forward the required CMake properties to the "fuzzed libc++" build
# so skip it here.
-DCOMPILER_RT_BUILD_LIBFUZZER=OFF \
-DCOMPILER_RT_DEFAULT_TARGET_ONLY=ON \
-DCOMPILER_RT_INSTALL_PATH="/usr/local/lib/clang/$CLANG_MAJOR_VERSION" && \
cmake --build runtimes -j $(getconf _NPROCESSORS_ONLN) && \
cmake --install runtimes --prefix "$ROOTFS_DIR/usr"

FROM mcr.microsoft.com/dotnet-buildtools/prereqs:azurelinux-3.0-crossdeps-llvm-net9.0-local
ARG ROOTFS_DIR=/crossrootfs/riscv64

COPY --from=builder /usr/local/lib/clang /usr/local/lib/clang/
COPY --from=builder $ROOTFS_DIR $ROOTFS_DIR
2 changes: 1 addition & 1 deletion src/azurelinux/3.0/net9.0/crossdeps-builder/amd64/Dockerfile
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -70,7 +70,7 @@ RUN mkdir llvm-project.src && \
-DCMAKE_BUILD_TYPE=Release \
-DCMAKE_C_COMPILER=clang \
-DCMAKE_CXX_COMPILER=clang++ \
-DLLVM_TARGETS_TO_BUILD="X86;AArch64;ARM;PowerPC;SystemZ" \
-DLLVM_TARGETS_TO_BUILD="X86;AArch64;ARM;PowerPC;RISCV;SystemZ" \
-Wno-dev \
-DLLVM_ENABLE_PROJECTS="clang;lld;clang-tools-extra" && \
make -j $(getconf _NPROCESSORS_ONLN) && \
Expand Down
25 changes: 25 additions & 0 deletions src/azurelinux/manifest.json
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -215,6 +215,31 @@
}
]
},
{
"platforms": [
{
"dockerfile": "src/azurelinux/3.0/net9.0/cross/riscv64",
"os": "linux",
"osVersion": "azurelinux3.0",
"tags": {
"azurelinux-3.0-cross-riscv64-net9.0-$(System:TimeStamp)-$(System:DockerfileGitCommitSha)": {},
"azurelinux-3.0-cross-riscv64-net9.0$(FloatingTagSuffix)": {}
}
}
]
}, {
"platforms": [
{
"dockerfile": "src/azurelinux/3.0/net9.0/cross/riscv64-alpine",
"os": "linux",
"osVersion": "azurelinux3.0",
"tags": {
"azurelinux-3.0-cross-riscv64-alpine-net9.0-$(System:TimeStamp)-$(System:DockerfileGitCommitSha)": {},
"azurelinux-3.0-cross-riscv64-alpine-net9.0$(FloatingTagSuffix)": {}
}
}
]
},
{
"platforms": [
{
Expand Down
35 changes: 0 additions & 35 deletions src/ubuntu/22.04/cross/riscv64-alpine/Dockerfile
View file
Open in desktop

This file was deleted.

16 changes: 0 additions & 16 deletions src/ubuntu/22.04/cross/riscv64/Dockerfile
View file
Open in desktop

This file was deleted.

26 changes: 0 additions & 26 deletions src/ubuntu/manifest.json
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -278,19 +278,6 @@
}
]
},
{
"platforms": [
{
"dockerfile": "src/ubuntu/22.04/cross/riscv64",
"os": "linux",
"osVersion": "jammy",
"tags": {
"ubuntu-22.04-cross-riscv64-$(System:TimeStamp)-$(System:DockerfileGitCommitSha)": {},
"ubuntu-22.04-cross-riscv64$(FloatingTagSuffix)": {}
}
}
]
},
{
"platforms": [
{
Expand All @@ -304,19 +291,6 @@
}
]
},
{
"platforms": [
{
"dockerfile": "src/ubuntu/22.04/cross/riscv64-alpine",
"os": "linux",
"osVersion": "jammy",
"tags": {
"ubuntu-22.04-cross-riscv64-alpine-$(System:TimeStamp)-$(System:DockerfileGitCommitSha)": {},
"ubuntu-22.04-cross-riscv64-alpine$(FloatingTagSuffix)": {}
}
}
]
},
{
"platforms": [
{
Expand Down