Skip to content
Merged
Show file tree
Hide file tree
Changes from 9 commits
Commits
Show all changes
30 commits
Select commit Hold shift + click to select a range
64b7d8d
feat: Add support for manylinux_2_28 and use it to build x86_64 or aa…
jackluo923 Jun 26, 2025
917cecd
fix: resolved python lint issues
jackluo923 Jun 26, 2025
53f5974
fix: resolved docs issues
jackluo923 Jun 26, 2025
73563d5
fix: resolved issues raised by code-rabbit
jackluo923 Jun 26, 2025
0b81039
fix: resolved more python linting issue
jackluo923 Jun 26, 2025
5ab5946
fix: resolved issues with github workflow
jackluo923 Jun 26, 2025
117ff00
fix: reduced the scope of the PR
jackluo923 Jun 27, 2025
9112825
Revert "fix: resolved issues with github workflow"
jackluo923 Jun 27, 2025
b18c2f3
fix: resolved issues raised by coderabbit
jackluo923 Jun 27, 2025
c6aea71
Merge branch 'main' into manylinux_2_28
jackluo923 Jun 27, 2025
258ba7b
Merge branch 'main' into manylinux_2_28
jackluo923 Jun 29, 2025
ab09ee3
fix: added back a line that wasn't supposed to be removed
jackluo923 Jun 29, 2025
af88773
Restructure docs before adding new, unpublished container images.
kirkrodrigues Jun 29, 2025
8378fdb
refactor: Remove unnecessary comments from build scripts; Make style …
kirkrodrigues Jun 29, 2025
9c5c624
Use latest conventions in install-all.sh
kirkrodrigues Jun 29, 2025
c31a6c0
Use latest conventions in install-packages-from-source.sh; Add commen…
kirkrodrigues Jun 29, 2025
dfa89e9
Use latest conventions in install-prebuilt-packages.sh; Remove unnece…
kirkrodrigues Jun 29, 2025
c744237
Simplify comment.
kirkrodrigues Jun 29, 2025
8cb738f
Rename images for consistency.
kirkrodrigues Jun 29, 2025
a35809d
Add new images to docs.
kirkrodrigues Jun 29, 2025
b1716df
Edit comment.
kirkrodrigues Jun 29, 2025
e916ecb
Don't lock base image version so that we can tell if something breaks…
kirkrodrigues Jun 29, 2025
e1809e8
Improve comment.
kirkrodrigues Jun 29, 2025
502a31c
Rename manylinux scripts directory.
kirkrodrigues Jun 29, 2025
f96a13a
Update more docs.
kirkrodrigues Jun 29, 2025
10dc855
Set minimum CMake version to 3.31.
kirkrodrigues Jun 30, 2025
02d257d
Add link to manylinux 2.28.
kirkrodrigues Jun 30, 2025
31e9874
Update components/core/CMakeLists.txt
kirkrodrigues Jun 30, 2025
8fd2fa0
Revert "Update more docs."
kirkrodrigues Jun 30, 2025
57cda64
Replace manylinux 2.28 with manylinux_2_28.
kirkrodrigues Jun 30, 2025
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
7 changes: 5 additions & 2 deletions components/core/CMakeLists.txt
Original file line number Diff line number Diff line change
Expand Up @@ -72,7 +72,10 @@ if (CLP_USE_STATIC_LIBS)
if (APPLE)
set(CLP_STATIC_LIBS_UNSUPPORTED_PLATFORM "macOS")
elseif (EXISTS "/etc/centos-release")
set(CLP_STATIC_LIBS_UNSUPPORTED_PLATFORM "CentOS")
# For now, only manylinux_2_28 (AlmaLinux 8 based) supports static linking
if(NOT "${CENTOS_RELEASE_CONTENT}" MATCHES "AlmaLinux")
set(CLP_STATIC_LIBS_UNSUPPORTED_PLATFORM "CentOS")
endif()
Comment thread
coderabbitai[bot] marked this conversation as resolved.
Resolved
endif()

if (DEFINED CLP_STATIC_LIBS_UNSUPPORTED_PLATFORM)
Expand Down Expand Up @@ -212,7 +215,7 @@ endif()
# Find and setup libcurl
# By default, CURL does not provide static libraries
if(CLP_NEED_CURL)
find_package(CURL 7.68.0 REQUIRED)
find_package(CURL 7.61.1 REQUIRED)
if(CURL_FOUND)
message(STATUS "Found CURL ${CURL_VERSION_STRING}")
else()
Expand Down
Original file line number Diff line number Diff line change
@@ -0,0 +1,23 @@
FROM quay.io/pypa/manylinux_2_28_aarch64:2025.06.25-1

# Binaries built on manylinux_2_28 (AlmaLinux 8 based) is expected to be
# compatible with other distros using glibc 2.28 or later, including:
# - Debian 10+
# - Ubuntu 18.10+
# - Fedora 29+
# - CentOS/RHEL 8+

WORKDIR /root

RUN mkdir -p ./tools/scripts/lib_install
COPY ./tools/scripts/lib_install ./tools/scripts/lib_install

RUN ./tools/scripts/lib_install/manylinux_2_28/install-all.sh

# Remove cached files
RUN dnf clean all && rm -rf /var/cache/dnf /tmp/* /var/tmp/*

# DO NOT FLATTEN THE IMAGE!!!
# The consequence of flattening the image is that we lose shell startup scripts
# and environment modification from /etc/profile.d, /etc/bashrc, etc.
# These are important for manylinux, otherwise, default system compilers are used, etc.

ghost Jun 26, 2025

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

🧹 Nitpick (assertive)

Consider adding a non-root user

Building images as root is fine, but running later stages (or CI tests) under root can mask permission bugs. Creating clp user (USER clp) after installs improves security and aligns with best-practice.

🧰 Tools
🪛 Checkov (3.2.334)

[LOW] 1-23: Ensure that HEALTHCHECK instructions have been added to container images

(CKV_DOCKER_2)


[LOW] 1-23: Ensure that a user for the container has been created

(CKV_DOCKER_3)

🤖 Prompt for AI Agents
In
components/core/tools/docker-images/clp-env-base-manylinux_2_28_aarch64/Dockerfile
around lines 20 to 23, the Dockerfile currently runs as root, which can mask
permission issues. To fix this, add a non-root user named 'clp' after the
installation steps by creating the user and switching to it with the USER
directive. This improves security and aligns with best practices by ensuring
subsequent commands and tests run with limited permissions.

Original file line number Diff line number Diff line change
@@ -0,0 +1,31 @@
#!/bin/bash

set -euo pipefail

# Get the directory this script is in
script_dir="$( cd "$( dirname "${BASH_SOURCE[0]}" )" &> /dev/null && pwd )"
component_root=${script_dir}/../../../

# Build aarch64 image, will automatically use QEMU emulate if not on native platform
build_cmd=(
docker buildx build
--platform linux/arm64
--tag clp-core-dependencies-manylinux_2_28_aarch64:dev
"$component_root"
--file "${script_dir}/Dockerfile"
--load
)

# If in a git repo, add labels for revision and source
if command -v git >/dev/null && git -C "$script_dir" rev-parse --is-inside-work-tree >/dev/null ;
then
build_cmd+=(
--label "org.opencontainers.image.revision=$(git -C "$script_dir" rev-parse HEAD)"
--label "org.opencontainers.image.source=$(git -C "$script_dir" remote get-url origin)"
)
fi

echo "Running: ${build_cmd[*]}"

# Execute the docker buildx command
"${build_cmd[@]}"
Original file line number Diff line number Diff line change
@@ -0,0 +1,23 @@
FROM quay.io/pypa/manylinux_2_28_x86_64:2025.06.25-1

# Binaries built on manylinux_2_28 (AlmaLinux 8 based) is expected to be
# compatible with other distros using glibc 2.28 or later, including:
# - Debian 10+
# - Ubuntu 18.10+
# - Fedora 29+
# - CentOS/RHEL 8+

WORKDIR /root

RUN mkdir -p ./tools/scripts/lib_install
COPY ./tools/scripts/lib_install ./tools/scripts/lib_install

RUN ./tools/scripts/lib_install/manylinux_2_28/install-all.sh

# Remove cached files
RUN dnf clean all && rm -rf /var/cache/dnf /tmp/* /var/tmp/*

# DO NOT FLATTEN THE IMAGE!!!
# The consequence of flattening the image is that we lose shell startup scripts
# and environment modification from /etc/profile.d, /etc/bashrc, etc.
# These are important for manylinux, otherwise, default system compilers are used, etc.
Comment thread
jackluo923 marked this conversation as resolved.
Resolved
Original file line number Diff line number Diff line change
@@ -0,0 +1,31 @@
#!/bin/bash

set -euo pipefail

# Get the directory this script is in
script_dir="$( cd "$( dirname "${BASH_SOURCE[0]}" )" &> /dev/null && pwd )"
component_root=${script_dir}/../../../

# Build x86_64 image, will automatically use QEMU emulate if not on native platform
build_cmd=(
docker buildx build
--platform linux/amd64
--tag clp-core-dependencies-manylinux_2_28_x86_64:dev
"$component_root"
--file "${script_dir}/Dockerfile"
--load
)
Comment on lines +10 to +

ghost Jun 26, 2025

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

🧹 Nitpick (assertive)

Image context may be unintentionally broad

Using ${component_root} (three levels up) as the build context ships the entire repo into Docker, slowing builds and exposing secrets in the context tarball. If only the Dockerfile and install scripts are required, set the context to "${script_dir}" or add a .dockerignore.

🤖 Prompt for AI Agents
In
components/core/tools/docker-images/clp-env-base-manylinux_2_28_x86_64/build.sh
around lines 10 to 17, the Docker build context is set to ${component_root},
which is too broad and includes the entire repo, causing slow builds and
potential secret exposure. Change the build context to "${script_dir}" to limit
it to only the Dockerfile and install scripts, or alternatively add a
.dockerignore file to exclude unnecessary files from the context.


# If in a git repo, add labels for revision and source
if command -v git >/dev/null && git -C "$script_dir" rev-parse --is-inside-work-tree >/dev/null ;
then
build_cmd+=(
--label "org.opencontainers.image.revision=$(git -C "$script_dir" rev-parse HEAD)"
--label "org.opencontainers.image.source=$(git -C "$script_dir" remote get-url origin)"
)
fi

echo "Running: ${build_cmd[*]}"

# Execute the docker buildx command
"${build_cmd[@]}"
4 changes: 2 additions & 2 deletions components/core/tools/scripts/lib_install/libarchive.sh
Original file line number Diff line number Diff line change
Expand Up @@ -63,8 +63,8 @@ fi
cd ${extracted_dir}
mkdir -p cmake-build-release
cd cmake-build-release
# NOTE: Disable expat so the static libarchive doesn't look for it at link time
cmake -DENABLE_EXPAT=OFF ../
# NOTE: Disable expat and openssl so the static libarchive doesn't look for it at link time
cmake -DENABLE_EXPAT=OFF -DENABLE_OPENSSL=OFF ../
make -j${num_cpus}

# Check if checkinstall is installed
Expand Down
Original file line number Diff line number Diff line change
@@ -0,0 +1,18 @@
#!/usr/bin/env bash

# Exit on any error
set -e

# Exit on failures hidden inside pipes (e.g., curl | tar)
set -o pipefail

# Error on undefined variable
set -u

script_dir="$( cd "$( dirname "${BASH_SOURCE[0]}" )" &> /dev/null && pwd )"

Comment on lines +3 to +7

ghost Jun 30, 2025

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

🧹 Nitpick (assertive)

Declare script_dir as a readonly constant

The value of script_dir is meant to stay immutable for the lifetime of the script.
Marking it readonly makes that intent explicit and guards against accidental reassignment.

-script_dir="$( cd "$( dirname "${BASH_SOURCE[0]}" )" &> /dev/null && pwd )"
+readonly script_dir="$( cd "$( dirname "${BASH_SOURCE[0]}" )" &> /dev/null && pwd )"
🤖 Prompt for AI Agents
In components/core/tools/scripts/lib_install/manylinux_2_28/install-all.sh
around lines 3 to 7, the variable script_dir is assigned but not marked
readonly. To prevent accidental reassignment and make the intent clear, declare
script_dir as a readonly variable immediately after its assignment by adding the
readonly keyword before or after the assignment.

"${script_dir}/install-prebuilt-packages.sh"
"${script_dir}/install-packages-from-source.sh"

# TODO: https://github.com/y-scope/clp/issues/795
"${script_dir}/../check-cmake-version.sh"
Original file line number Diff line number Diff line change
@@ -0,0 +1,22 @@
#!/usr/bin/env bash

# Exit on any error
set -e

# Exit on failures hidden inside pipes (e.g., curl | tar)
set -o pipefail

# Error on undefined variable
set -u

script_dir="$( cd "$( dirname "${BASH_SOURCE[0]}" )" &> /dev/null && pwd )"
lib_install_scripts_dir="${script_dir}/.."

Comment on lines +3 to +8

ghost Jun 29, 2025

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

🧹 Nitpick (assertive)

Declare constants as readonly for defensive programming

script_dir and lib_install_scripts_dir are effectively constants after initialisation.
Marking them readonly prevents accidental reassignment later in the script (or in sourced code) and makes intent explicit.

-script_dir="$( cd "$( dirname "${BASH_SOURCE[0]}" )" &> /dev/null && pwd )"
-lib_install_scripts_dir="${script_dir}/.."
+readonly script_dir="$( cd "$( dirname "${BASH_SOURCE[0]}" )" &> /dev/null && pwd )"
+readonly lib_install_scripts_dir="${script_dir}/.."
📝 Committable suggestion

‼️ IMPORTANT
Carefully review the code before committing. Ensure that it accurately replaces the highlighted code, contains no missing lines, and has no issues with indentation. Thoroughly test & benchmark the code to ensure it meets the requirements.

Suggested change
set -eu
set -o pipefail
script_dir="$( cd "$( dirname "${BASH_SOURCE[0]}" )" &> /dev/null && pwd )"
lib_install_scripts_dir="${script_dir}/.."
set -eu
set -o pipefail
readonly script_dir="$( cd "$( dirname "${BASH_SOURCE[0]}" )" &> /dev/null && pwd )"
readonly lib_install_scripts_dir="${script_dir}/.."
🤖 Prompt for AI Agents
In
components/core/tools/scripts/lib_install/manylinux_2_28/install-packages-from-source.sh
around lines 3 to 8, the variables script_dir and lib_install_scripts_dir are
assigned once and should be treated as constants. To prevent accidental
reassignment and clarify intent, declare both variables as readonly immediately
after their assignment by adding the readonly keyword before or after their
initialization.

# NOTE: The remaining installation scripts depend on boost, so we install it beforehand.
"${lib_install_scripts_dir}"/install-boost.sh 1.87.0

"$lib_install_scripts_dir"/liblzma.sh 5.8.1
"$lib_install_scripts_dir"/lz4.sh 1.10.0
"$lib_install_scripts_dir"/zstandard.sh 1.5.7
"$lib_install_scripts_dir"/libarchive.sh 3.8.0
"${lib_install_scripts_dir}"/msgpack.sh 7.0.0
Original file line number Diff line number Diff line change
@@ -0,0 +1,68 @@
#!/usr/bin/env bash

# Exit on any error
set -e

# Exit on failures hidden inside pipes (e.g., curl | tar)
set -o pipefail

# Error on undefined variable
set -u

dnf install -y \
gcc-c++ \
java-11-openjdk \
libcurl-devel \
mariadb-connector-c-devel \
openssl-devel \
zlib-devel \
zlib-static \
jq

# Determine architecture for `task` release to install
rpm_arch=$(rpm --eval "%{_arch}")
case "$rpm_arch" in
"x86_64")
task_pkg_arch="amd64"
;;
"aarch64")
task_pkg_arch="arm64"
;;
*)
echo "Error: Unsupported architecture - $rpm_arch"
exit 1
;;
esac

# Install `task`
# NOTE: We lock `task` to a version < 3.43 to avoid https://github.com/y-scope/clp/issues/872
tmp_rpm=$(mktemp -t --suffix ".rpm" task-pkg.XXXXXXXXXX) || exit 1
task_pkg_path="$tmp_rpm"
curl \
--fail \
--location \
--output "$task_pkg_path" \
--show-error \
"https://github.com/go-task/task/releases/download/v3.42.1/task_linux_${task_pkg_arch}.rpm"
dnf install --assumeyes "$task_pkg_path"
rm -rf "$task_pkg_path"

Comment thread
jackluo923 marked this conversation as resolved.
Resolved
# The bundled CMake 4.x version is too new; downgrading to CMake 3.27 is
# necessary to resolve build issues with third-party libraries. Even after
# we added the "-DCMAKE_POLICY_VERSION_MINIMUM=3.5" flag, compilation issues
# in libraries such as googletest-src and yaml-cpp still persist.
# See sample error below:
# CMake Error at _deps/googletest-src/CMakeLists.txt:4 (cmake_minimum_required):
# Support for CMake versions below 3.5 has been dropped.
#
# Adjust the VERSION argument’s minimum value, or use the <min>...<max> syntax
# to indicate the required CMake version range.
#
# Alternatively, you can try adding -DCMAKE_POLICY_VERSION_MINIMUM=3.5
export PLATFORM=$(uname -m)
Comment thread
jackluo923 marked this conversation as resolved.
Resolved
export CMAKE_VERSION=3.27.9
export CMAKE_RELEASE_URL=https://github.com/Kitware/CMake/releases/download/v${CMAKE_VERSION}/cmake-${CMAKE_VERSION}-linux-${PLATFORM}.tar.gz
echo "Downgrading CMAKE to 3.27.9"
curl -L "${CMAKE_RELEASE_URL}" | tar -xzf -
mv "cmake-${CMAKE_VERSION}-linux-${PLATFORM}" "/opt/cmake-${CMAKE_VERSION}"
ln -sf "/opt/cmake-${CMAKE_VERSION}/bin/cmake" /usr/local/bin/cmake

ghost Jun 29, 2025

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Can we use pip3 to install cmake with a specified version? It worked on our private containers.