Arm Runner authentication broken w/ .Net 6.0 & .Net 8.0 on Ubuntu 24.x, fixed with .Net 9.0. "The SSL connection could not be established, see inner exception." #3505
Labels
bug
Something isn't working
Comments
satmandu
changed the title
|
This may be connected to dotnet/runtime#101444 Which was apparently fixed in dotnet/runtime#102410 But just doing a dotnet 8 build of the current runner from master still has the same issue.... |
satmandu
added a commit
to satmandu/runner
that referenced
this issue
Oct 23, 2024
See actions#3505 dotnet/runtime#101444 dotnet/runtime#96460 dotnet/runtime#102410 (Looks like this has not been backported to .Net 8.x...) Signed-off-by: Satadru Pramanik <[email protected]>
satmandu
added a commit
to satmandu/runner
that referenced
this issue
Oct 23, 2024
This updates .Net to 9.0.100-rc.2.24474.11 . See actions#3505 dotnet/runtime#101444 dotnet/runtime#96460 dotnet/runtime#102410 (Looks like this has not been backported to .Net 8.x...) Signed-off-by: Satadru Pramanik <[email protected]>
|
So it appears that this is broken with both .Net 6.0 and .Net 8.0, but fixed with .Net 9.0. I patched from top of tree to use .Net 9.0 RC2, and I am no longer having authentication failures: I have opened an issue to request that the likely responsible Y2038 fixes be backported from .Net 9.0 to 8.x: |
satmandu
changed the title
|
After patching to use .Net 9.0 this is what I get: |
satmandu
changed the title
satmandu
added a commit
to satmandu/runner
that referenced
this issue
Nov 13, 2024
This updates .Net to 9.0.100-rc.2.24474.11 . See actions#3505 dotnet/runtime#101444 dotnet/runtime#96460 dotnet/runtime#102410 (Looks like this has not been backported to .Net 8.x...) Signed-off-by: Satadru Pramanik <[email protected]>
satmandu
added a commit
to satmandu/runner
that referenced
this issue
Nov 13, 2024
This updates .Net to 9.0.100-rc.2.24474.11 . See actions#3505 dotnet/runtime#101444 dotnet/runtime#96460 dotnet/runtime#102410 (Looks like this has not been backported to .Net 8.x...) Signed-off-by: Satadru Pramanik <[email protected]>
satmandu
added a commit
to satmandu/runner
that referenced
this issue
Nov 13, 2024
This updates .Net to 9.0.100-rc.2.24474.11 . See actions#3505 dotnet/runtime#101444 dotnet/runtime#96460 dotnet/runtime#102410 (Looks like this has not been backported to .Net 8.x...) Signed-off-by: Satadru Pramanik <[email protected]>
|
I am also experiencing this issue. I cannot use actions runner on armv7. |
|
🤔 Thanks for bring this up. |
Yes I really hope this gets resolved. 👍 But to me it looks like you either pause releasing for armv7 or you skip dotnet 8 and got to dotnet 9 directly. |
|
Would your runner work if we disable TLS validation? |
This sounds unsafe I am now using the latest release from @satmandu and no longer have this issue |
|
I've been meaning to submit a PR to update to .Net 9.0, which fixes the underlying issue, and was officially released yesterday! (As previously mentioned this fix will not be backported to .Net 8.x .) But feel free to just compare my main branch to the one here to see what I changed. I rebased yesterday before rebuilding, and the version I built I'm now using in docker containers on a RPI5 for our armv7l Chromebrew unit tests. |
|
(Essentially I'm just using .Net 9.0 on top of the latest Ubuntu LTS, which the MS dev containers require for .Net 9.0 support.) |
satmandu
added a commit
to satmandu/runner
that referenced
this issue
Nov 14, 2024
This updates .Net to 9.0.100-rc.2.24474.11 . See actions#3505 dotnet/runtime#101444 dotnet/runtime#96460 dotnet/runtime#102410 (Looks like this has not been backported to .Net 8.x...) Signed-off-by: Satadru Pramanik <[email protected]>
|
PR Submitted: #3573 Also @TingluoHuang |
|
@satmandu your branch fixed the issue I had when emulating armv7. Through our deployment scheme on actual hardware I still get the same error, which I thought would be related to this issue: |
|
@malik-n I can't speak to your specific issue. This is our setup, which we use for our Unit-Test.yml . Maybe you can see how your setup differs from ours? This is the Dockerfile I am currently using, with # syntax=docker/dockerfile:1-labs
FROM ubuntu:24.10
ENV TZ='America/New_York'
ENV DEBIAN_FRONTEND=noninteractive
# install python and the packages the your code depends on along with jq so we can parse JSON
# add additional packages as necessary
RUN apt-get update -y \
&& apt-get upgrade -y \
&& apt-get install -y --no-install-recommends \
curl jq build-essential libssl-dev libffi-dev python3 python3-venv python3-dev python3-pip \
&& apt-get install -y ca-certificates libicu-dev ruby inetutils-ping vim git gh sudo file golang \
&& git config --global advice.detachedHead false \
&& echo TZ='America/New_York' >> /etc/environment
SHELL ["/bin/bash", "-o", "pipefail", "-c"]
# Make sure golang is up to date, and rebuild gh-cli off of that.
ARG GO_VERSION=$GO_VERSION
ENV GO_VERSION=$GO_VERSION
ARG GH_CLI_VERSION=$GH_CLI_VERSION
ENV GH_CLI_VERSION=$GH_CLI_VERSION
RUN echo "GH_CLI_VERSION is ${GH_CLI_VERSION}" \
&& echo "GO_VERSION is ${GO_VERSION}" \
&& apt-get update -y \
&& apt-get upgrade -y \
&& apt remove gh -y \
&& mkdir /home/work \
&& cd /home/work \
&& git clone --depth 1 --branch ${GH_CLI_VERSION} https://github.com/cli/cli.git gh-cli \
&& cd gh-cli \
&& GO_LDFLAGS="-s -w" GOOS=linux CGO_ENABLED=0 make install \
&& cd .. \
&& rm -rf gh-cli \
&& apt-get remove golang* -y
RUN install -m 0755 -d /etc/apt/keyrings \
&& curl -fsSL https://download.docker.com/linux/ubuntu/gpg -o /etc/apt/keyrings/docker.asc \
&& chmod a+r /etc/apt/keyrings/docker.asc \
&& echo \
"deb [arch=$(dpkg --print-architecture) signed-by=/etc/apt/keyrings/docker.asc] https://download.docker.com/linux/ubuntu \
$(. /etc/os-release && echo "$VERSION_CODENAME") stable" | \
tee /etc/apt/sources.list.d/docker.list > /dev/null \
&& apt-get update \
&& apt-get install -y docker-ce docker-ce-cli containerd.io docker-buildx-plugin docker-compose-plugin \
&& apt-get upgrade -y \
&& useradd -m docker -g docker \
&& echo "docker ALL=(ALL) NOPASSWD: ALL" >> /etc/sudoers
# Get the github runner version, cd into the user directory, download and unzip the github actions runner
ARG RUNNER_VERSION=$RUNNER_VERSION
ENV RUNNER_VERSION=$RUNNER_VERSION
RUN --mount=type=bind,rw,target=/input echo "RUNNER_VERSION is ${RUNNER_VERSION}" \
&& case $( dpkg --print-architecture ) in amd64) RUNNER_ARCH=x64;; arm64) RUNNER_ARCH=arm64;; armhf) RUNNER_ARCH=arm;; *) echo "Runner ARCH not determined." && exit 1 ;; esac \
&& echo "RUNNER_ARCH is ${RUNNER_ARCH}" \
&& cd /home/docker && mkdir actions-runner && cd actions-runner \
&& cp /input/actions-runner-linux-${RUNNER_ARCH}-${RUNNER_VERSION}.tar.gz . \
&& tar xzf ./actions-runner-linux-${RUNNER_ARCH}-${RUNNER_VERSION}.tar.gz && rm ./actions-runner-linux-${RUNNER_ARCH}-${RUNNER_VERSION}.tar.gz
# && curl -O -L https://github.com/actions/runner/releases/download/v${RUNNER_VERSION}/actions-runner-linux-${RUNNER_ARCH}-${RUNNER_VERSION}.tar.gz \
# && tar xzf ./actions-runner-linux-${RUNNER_ARCH}-${RUNNER_VERSION}.tar.gz && rm ./actions-runner-linux-${RUNNER_ARCH}-${RUNNER_VERSION}.tar.gz
RUN <<EOF1
tee -a /start.sh <<RUNNERSTARTEOF
#!/bin/bash
CONTAINER_ORIGINAL_RUNNER_VERSION="$RUNNER_VERSION"
TZ='America/New_York'
ORGANIZATION=\$ORGANIZATION
ACCESS_TOKEN=\$ACCESS_TOKEN
case \$( dpkg --print-architecture ) in
amd64) RUNNER_ARCH=x64;;
arm64) RUNNER_ARCH=arm64;;
armhf) RUNNER_ARCH=arm;;
*) echo "Runner ARCH not determined." && exit 1 ;;
esac
echo "RUNNER_ARCH is \$RUNNER_ARCH"
cd /home/docker/actions-runner
verlte() {
[ "\$1" = "`echo -e "\$1\n\$2" | sort -V | head -n1`" ]
}
verlt() {
[ "\$1" = "\$2" ] && return 1 || verlte \$1 \$2
}
RUNNER_VERSION=\$(git -c 'versionsort.suffix=-' ls-remote --exit-code --refs --sort='version:refname' --tags https://github.com/actions/runner '*.*.*' | tail --lines=1 | cut --delimiter='/' --fields=3 | sed 's/v//') \
if verlt \$CONTAINER_ORIGINAL_RUNNER_VERSION \$RUNNER_VERSION ; then
echo "Updating GitHub Actions Runner to: \$RUNNER_VERSION"
curl -O -L https://github.com/actions/runner/releases/download/v\${RUNNER_VERSION}/actions-runner-linux-\${RUNNER_ARCH}-\${RUNNER_VERSION}.tar.gz \
&& tar xzf ./actions-runner-linux-\${RUNNER_ARCH}-\${RUNNER_VERSION}.tar.gz && rm ./actions-runner-linux-\${RUNNER_ARCH}-\${RUNNER_VERSION}.tar.gz
fi
./config.sh --unattended --url https://github.com/\${ORGANIZATION} --token \${ACCESS_TOKEN}
cleanup() {
echo "Removing runner..."
./config.sh remove --token \${ACCESS_TOKEN}
}
trap 'cleanup; exit 130' INT
trap 'cleanup; exit 143' TERM
./run.sh & wait \$!
RUNNERSTARTEOF
chmod +x /start.sh
EOF1
# since the config and run script for actions are not allowed to be run by root,
# set the user to "docker" so all subsequent commands are run as the docker user
USER docker
RUN git config --global advice.detachedHead false
# use cmd instead of entrypoint for easier debugging...
CMD ["./start.sh"]This is the script I am using to build our container: #!/bin/bash
# build_docker_runner
: "${REPOSITORY:=satmandu}"
GH_CLI_VERSION="$(git -c 'versionsort.suffix=-' ls-remote --exit-code --refs --sort='version:refname' --tags https://github.com/cli/cli '*.*.*' | grep -v "release.\\|weekly." | tail --lines=1 | cut --delimiter='/' --fields=3)"
echo "GH_CLI_VERSION is $GH_CLI_VERSION"
GO_VERSION="$(git -c 'versionsort.suffix=-' ls-remote --exit-code --refs --sort='version:refname' --tags https://github.com/golang/go '*.*.*' | grep -v "release.\|weekly." | tail --lines=1 | cut --delimiter='/' --fields=3)"
echo "GO_VERSION is $GO_VERSION"
# RUNNER_VERSION="$(git -c 'versionsort.suffix=-' ls-remote --exit-code --refs --sort='version:refname' --tags https://github.com/actions/runner '*.*.*' | tail --lines=1 | cut --delimiter='/' --fields=3 | sed 's/v//')"
RUNNER_VERSION='2.321.1'
echo "RUNNER_VERSION is $RUNNER_VERSION"
case $( uname -m ) in
x86_64) PLATFORM=( linux/amd64 );;
aarch64) PLATFORM=( linux/arm/v7 linux/arm64 );;
armv7l|armv8l) PLATFORM=( linux/arm/v7 );;
*) echo "Docker platform to build not determined." && exit 1 ;;
esac
function join_by { local IFS="$1"; shift; echo "$*"; }
BUILD_PLATFORMS="$(join_by , "${PLATFORM[@]}")"
echo "BUILD_PLATFORMS: $BUILD_PLATFORMS"
BUILD_PLATFORM_TAGS="$(for i in "${PLATFORM[@]}" ; do echo " --tag ${REPOSITORY}/chromebrewgithubrunner:latest-${i//\//_} " ; done | tr -d '\n')"
echo "BUILD_PLATFORM_TAGS: $BUILD_PLATFORM_TAGS"
cd "/localnet/chromeos/github/runner"
dangling_images=$(docker images --filter "dangling=true" -q --no-trunc)
[[ -n "$dangling_images" ]] && docker rmi -f $(docker images --filter "dangling=true" -q --no-trunc)docker buildx rm builder
docker buildx create --name builder --driver docker-container --use --driver-opt env.BUILDKIT_STEP_LOG_MAX_SIZE=10485760
docker buildx inspect --bootstrap
buildx_cmdline="env PROGRESS_NO_TRUNC=1 docker buildx build \
--no-cache \
--build-arg GH_CLI_VERSION=${GH_CLI_VERSION} \
--build-arg GO_VERSION=${GO_VERSION} \
--build-arg RUNNER_VERSION=${RUNNER_VERSION} \
--push --platform ${BUILD_PLATFORMS} \
--tag ${REPOSITORY}/chromebrewgithubrunner:${RUNNER_VERSION} \
--tag ${REPOSITORY}/chromebrewgithubrunner:runner-${RUNNER_VERSION} \
--tag ${REPOSITORY}/chromebrewgithubrunner:latest \
${BUILD_PLATFORM_TAGS} \
."
echo "build command is ${buildx_cmdline}"
rm -rf crewrunner-"${PLATFORM//\//_}"-build.log
$buildx_cmdline 2>&1 | tee -a crewrunner-"${PLATFORM//\//_}"-build.logThis is our systemd service file: # /etc/systemd/system/github-runner.service
[Unit]
Description=Github Actions Runner
Requires=docker.service
After=network.target docker.service
[Service]
Restart=always
User=runner
ExecStartPre=/usr/bin/docker pull satmandu/chromebrewgithubrunner:latest-linux_arm_v7
ExecStart=/home/runner/bin/runner.sh
ExecStop=/usr/bin/docker stop -t 30 runner-linux_arm_v7
[Install]
WantedBy=default.targetThis is the script we invoke from that systemd service: #!/bin/bash
# runner.sh
if [[ -n $GITHUB_ORGANIZATION ]]; then
echo "GITHUB_ORGANIZATION variable not set!"
exit 1
fi
if [[ -n $GITHUB_ACCESS_TOKEN ]]; then
echo "GITHUB_ACCESS_TOKEN variable not set!"
exit 1
fi
case $( uname -m ) in
x86_64) RUN_PLATFORM=( linux/amd64 );;
aarch64) RUN_PLATFORM=( linux/arm/v7 );;
armv7l|armv8l) RUN_PLATFORM=( linux/arm/v7 );;
*) echo "Docker platforms to run not determined." && exit 1 ;;
esac
function join_by { local IFS="$1"; shift; echo "$*"; }
RUN_PLATFORMS="$(join_by , "${RUN_PLATFORM[@]}")"
echo "Runner platforms are: $RUN_PLATFORMS"
for PLATFORM in "${RUN_PLATFORM[@]}"
do
docker pull satmandu/chromebrewgithubrunner:latest-"${PLATFORM//\//_}"
if docker container ls -a | grep -q runner-"${PLATFORM//\//_}"; then
docker container start -a runner-"${PLATFORM//\//_}"
else
docker run \
--privileged \
--init \
-v /var/run/docker.sock:/var/run/docker.sock \
--env ORGANIZATION="$GITHUB_ORGANIZATION" \
--env ACCESS_TOKEN="$GITHUB_ACCESS_TOKEN" \
--env TZ="America/New_York" \
--name runner-"${PLATFORM//\//_}" \
--platform "${PLATFORM}" \
satmandu/chromebrewgithubrunner:latest-"${PLATFORM//\//_}"
fi
done |
|
(It's clearly working for us...) |
|
Thank you for this information. |
That could be a cause. We're just starting docker, which gives no problems. |
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
The runner fails to authenticate with Github when using the arm runner in an Ubuntu 24.x based container. Logs indicate a "The remote certificate is invalid because of errors in the certificate chain: NotTimeValid" error. This appears to be due to arm32 Y2038 compatibility issues with the underlying Ubuntu/Debian operating systems moving to 64-bit time.
Fixes are in .Net 9.0, but will not be backported to .Net 8.0 as per dotnet/core#9285
See:
dotnet/runtime#101444
dotnet/runtime#96460
dotnet/runtime#102410
Working patch (Just adjusting to use .Net 9.0):
satmandu#1
I requested a backport at dotnet/runtime#109161 but it's clear that this isn't going to happen and that for newer linux distribution based usage on arm32 .Net 9.0 will be needed.
This is what I see:
Digging in I see this error when running
./config.sh --check --url https://github.com/${ORGANIZATION} --pat ${ACCESS_TOKEN}(withDOTNET_SYSTEM_NET_HTTP_USESOCKETSHTTPHANDLER=0set):Looking at one of these InternetCheck logs in more detail:
w/o
DOTNET_SYSTEM_NET_HTTP_USESOCKETSHTTPHANDLER=0set:To Reproduce
See Docker image setup info below.
Runner Version and Platform
v2.320 (though v2.319 didn't work either.)
OS of the machine running the runner? Linux (Raspberry PI OS arm64 with an armv7l/armhf docker container cotnaining the runner.)
Job Log Output
Runner and Worker's Diagnostic Logs
Looking at the _diag Runner logs, I see the same thing:
This is the Dockerfile I am using:
I build the container on a Raspberry Pi 5B running the 64-bit Raspberry Pi OS with 4k pages set via booting with the kernel8.img (which allows me to use armv7l/armv8l functionality) using:
I can start the container thus:
The text was updated successfully, but these errors were encountered: