Skip to content

[CI][Docker][Cortex-M]Update scripts to update ci_cortexm to Ubuntu 20.04 #13736

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Merged
merged 8 commits into from
Jan 11, 2023
Merged

[CI][Docker][Cortex-M]Update scripts to update ci_cortexm to Ubuntu 20.04 #13736

Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
8 commits
Select commit Hold shift + click to select a range
38ed935
Update nrfjprog script and LLVM script
mehrdadh Jan 6, 2023
c484dba
cleanup
mehrdadh Jan 6, 2023
3488fa9
change llvm install to apt
mehrdadh Jan 9, 2023
6599e3c
add SHA to nrfjprog installation script
mehrdadh Jan 9, 2023
b819d5d
trigger
mehrdadh Jan 9, 2023
c1456ae
fix key
mehrdadh Jan 9, 2023
3901fab
update apt key for llvm
mehrdadh Jan 10, 2023
08ce7f8
add python3.7 install option to script
mehrdadh Jan 10, 2023
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
2 changes: 1 addition & 1 deletion apps/microtvm/reference-vm/base-box/base_box_setup_common.sh
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -46,7 +46,7 @@ sudo apt-install-and-clear -y --no-install-recommends \
cmake=3.22.2-0kitware1ubuntu18.04.1 cmake-data=3.22.2-0kitware1ubuntu18.04.1 \

# Python
sudo ~/ubuntu_install_python.sh
sudo ~/ubuntu_install_python.sh 3.7
rm -f ~/ubuntu_install_python.sh

# Poetry deps
Expand Down
2 changes: 1 addition & 1 deletion docker/Dockerfile.ci_arm
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -56,7 +56,7 @@ RUN bash /install/ubuntu_install_llvm.sh
ENV TVM_VENV /venv/apache-tvm-py3.7
COPY python/bootstrap/lockfiles /install/python/bootstrap/lockfiles
COPY install/ubuntu_install_python.sh /install/ubuntu_install_python.sh
RUN bash /install/ubuntu_install_python.sh
RUN bash /install/ubuntu_install_python.sh 3.7
ENV PATH ${TVM_VENV}/bin:$PATH
ENV PYTHONNOUSERSITE 1 # Disable .local directory from affecting CI.

Expand Down
8 changes: 4 additions & 4 deletions docker/Dockerfile.ci_cortexm
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -17,7 +17,7 @@

# CI docker CPU env
# tag: v0.62
FROM ubuntu:18.04
FROM ubuntu:20.04

COPY utils/apt-install-and-clear.sh /usr/local/bin/apt-install-and-clear

Expand All @@ -38,15 +38,15 @@ RUN bash /install/ubuntu_install_googletest.sh
ENV TVM_VENV /venv/apache-tvm-py3.7
COPY python/bootstrap/lockfiles /install/python/bootstrap/lockfiles
COPY install/ubuntu_install_python.sh /install/ubuntu_install_python.sh
RUN bash /install/ubuntu_install_python.sh
RUN bash /install/ubuntu_install_python.sh 3.7
ENV PATH ${TVM_VENV}/bin:$PATH
ENV PYTHONNOUSERSITE 1 # Disable .local directory from affecting CI.

COPY install/ubuntu_install_python_package.sh /install/ubuntu_install_python_package.sh
RUN bash /install/ubuntu_install_python_package.sh

COPY install/ubuntu1804_install_llvm.sh /install/ubuntu1804_install_llvm.sh
RUN bash /install/ubuntu1804_install_llvm.sh
COPY install/ubuntu2004_install_llvm.sh /install/ubuntu2004_install_llvm.sh
RUN bash /install/ubuntu2004_install_llvm.sh
Copy link
Contributor

@asparkhi asparkhi Jan 10, 2023 •
edited
Loading

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

@mehrdadh, thanks for this upgrade. Since Ubuntu20.04 has default installation of python3.8 and python3.7 EOL is June 2023 as per https://peps.python.org/pep-0537/#lifespan. Since I have not seen any other discussions, my assumption is that we still do wish to support older version until its EOL. We most likely would require python3.7 installation from https://launchpad.net/~deadsnakes/+archive/ubuntu/ppa.

Copy link
Member Author

@mehrdadh mehrdadh Jan 10, 2023

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

@ashutosh-arm do we need python3.7 for cortexm CI image? If so, we should add it in a separate PR.
This PR doesn't update the image yet.

Copy link
Contributor

@asparkhi asparkhi Jan 10, 2023 •
edited
Loading

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Yes, we will need to support python3.7 until its EOL. Considering this, the next thing that pops up in my brain is when the ci_cortexm gets updated eventually corresponding to this (current) change, will the default venv still remain at python3.7 or we would move cortex-m docker image to use install/ubuntu2004_install_python.sh?

tvm/docker/Dockerfile.ci_cortexm

Line 38 in 92da138

ENV TVM_VENV /venv/apache-tvm-py3.7

Copy link
Member Author

@mehrdadh mehrdadh Jan 10, 2023

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

install/ubuntu2004_install_python.sh installs python3.8 as well. We have to add another script to install 3.7 in ubuntu20.04 if we need it.

Switching the ci image to 3.8 doesn't mean we're not supporting 3.7 anymore. Could you elaborate if there is a specific reason that this image has to run 3.7? We have switched other images like hexagon to 3.8 already.

Copy link
Member Author

@mehrdadh mehrdadh Jan 10, 2023

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I see your point, I will change the scripts to install python3.7 in ci_cortexm.

Copy link
Member Author

@mehrdadh mehrdadh Jan 11, 2023

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

08ce7f8 addresses your concern.

Copy link
Contributor

@asparkhi asparkhi Jan 11, 2023

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Thanks @mehrdadh


# Rust env (build early; takes a while)
COPY install/ubuntu_install_rust.sh /install/ubuntu_install_rust.sh
Expand Down
2 changes: 1 addition & 1 deletion docker/Dockerfile.ci_cpu
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -37,7 +37,7 @@ RUN bash /install/ubuntu_install_googletest.sh
ENV TVM_VENV /venv/apache-tvm-py3.7
COPY python/bootstrap/lockfiles /install/python/bootstrap/lockfiles
COPY install/ubuntu_install_python.sh /install/ubuntu_install_python.sh
RUN bash /install/ubuntu_install_python.sh
RUN bash /install/ubuntu_install_python.sh 3.7
ENV PATH ${TVM_VENV}/bin:$PATH
ENV PYTHONNOUSERSITE 1 # Disable .local directory from affecting CI.

Expand Down
2 changes: 1 addition & 1 deletion docker/Dockerfile.ci_gpu
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -44,7 +44,7 @@ RUN bash /install/ubuntu_install_googletest.sh /googletest
ENV TVM_VENV /venv/apache-tvm-py3.7
COPY python/bootstrap/lockfiles /install/python/bootstrap/lockfiles
COPY install/ubuntu_install_python.sh /install/ubuntu_install_python.sh
RUN bash /install/ubuntu_install_python.sh
RUN bash /install/ubuntu_install_python.sh 3.7
ENV PATH ${TVM_VENV}/bin:$PATH
ENV PYTHONNOUSERSITE 1 # Disable .local directory from affecting CI.

Expand Down
2 changes: 1 addition & 1 deletion docker/Dockerfile.ci_hexagon
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -40,7 +40,7 @@ RUN bash /install/ubuntu_install_googletest.sh
ENV TVM_VENV /venv/apache-tvm-py3.8
COPY python/bootstrap/lockfiles /install/python/bootstrap/lockfiles
COPY install/ubuntu_install_python.sh /install/ubuntu_install_python.sh
RUN bash /install/ubuntu_install_python.sh
RUN bash /install/ubuntu_install_python.sh 3.8
ENV PATH ${TVM_VENV}/bin:$PATH
ENV PYTHONNOUSERSITE 1 # Disable .local directory from affecting CI.

Expand Down
2 changes: 1 addition & 1 deletion docker/Dockerfile.ci_i386
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -52,7 +52,7 @@ ENV PYTHONNOUSERSITE 1 # Disable .local directory from affecting CI.
ENV TVM_VENV /venv/apache-tvm-py3.7
COPY python/bootstrap/lockfiles /install/python/bootstrap/lockfiles
COPY install/ubuntu_install_python.sh /install/ubuntu_install_python.sh
RUN bash /install/ubuntu_install_python.sh
RUN bash /install/ubuntu_install_python.sh 3.7
ENV PATH ${TVM_VENV}/bin:$PATH

COPY install/ubuntu_install_python_package.sh /install/ubuntu_install_python_package.sh
Expand Down
2 changes: 1 addition & 1 deletion docker/Dockerfile.ci_lint
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -29,7 +29,7 @@ RUN apt-install-and-clear -y wget git sudo make parallel
ENV TVM_VENV /venv/apache-tvm-py3.7
COPY python/bootstrap/lockfiles /install/python/bootstrap/lockfiles
COPY install/ubuntu_install_python.sh /install/ubuntu_install_python.sh
RUN bash /install/ubuntu_install_python.sh
RUN bash /install/ubuntu_install_python.sh 3.7
ENV PATH ${TVM_VENV}/bin:$PATH
ENV PYTHONNOUSERSITE 1 # Disable .local directory from affecting CI.

Expand Down
2 changes: 1 addition & 1 deletion docker/Dockerfile.ci_minimal
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -37,7 +37,7 @@ RUN bash /install/ubuntu_install_googletest.sh
ENV TVM_VENV /venv/apache-tvm-py3.7
COPY python/bootstrap/lockfiles /install/python/bootstrap/lockfiles
COPY install/ubuntu_install_python.sh /install/ubuntu_install_python.sh
RUN bash /install/ubuntu_install_python.sh
RUN bash /install/ubuntu_install_python.sh 3.7
ENV PATH ${TVM_VENV}/bin:$PATH
ENV PYTHONNOUSERSITE 1 # Disable .local directory from affecting CI.

Expand Down
2 changes: 1 addition & 1 deletion docker/Dockerfile.ci_riscv
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -38,7 +38,7 @@ RUN bash /install/ubuntu_install_googletest.sh
ENV TVM_VENV /venv/apache-tvm-py3.7
COPY python/bootstrap/lockfiles /install/python/bootstrap/lockfiles
COPY install/ubuntu_install_python.sh /install/ubuntu_install_python.sh
RUN bash /install/ubuntu_install_python.sh
RUN bash /install/ubuntu_install_python.sh 3.7
ENV PATH ${TVM_VENV}/bin:$PATH
ENV PYTHONNOUSERSITE 1 # Disable .local directory from affecting CI.

Expand Down
2 changes: 1 addition & 1 deletion docker/Dockerfile.ci_wasm
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -35,7 +35,7 @@ RUN bash /install/ubuntu_install_googletest.sh
ENV TVM_VENV /venv/apache-tvm-py3.7
COPY python/bootstrap/lockfiles /install/python/bootstrap/lockfiles
COPY install/ubuntu_install_python.sh /install/ubuntu_install_python.sh
RUN bash /install/ubuntu_install_python.sh
RUN bash /install/ubuntu_install_python.sh 3.7
ENV PATH ${TVM_VENV}/bin:$PATH
ENV PYTHONNOUSERSITE 1 # Disable .local directory from affecting CI.

Expand Down
2 changes: 1 addition & 1 deletion docker/Dockerfile.demo_android
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -31,7 +31,7 @@ RUN bash /install/ubuntu_install_core.sh
ENV TVM_VENV /venv/apache-tvm-py3.7
COPY python/bootstrap/lockfiles /install/python/bootstrap/lockfiles
COPY install/ubuntu_install_python.sh /install/ubuntu1804_install_python.sh
RUN bash /install/ubuntu1804_install_python.sh
RUN bash /install/ubuntu1804_install_python.sh 3.7
ENV PATH ${TVM_VENV}/bin:$PATH
ENV PYTHONNOUSERSITE 1 # Disable .local directory from affecting CI.

Expand Down
2 changes: 1 addition & 1 deletion docker/Dockerfile.demo_rocm
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -29,7 +29,7 @@ RUN bash /install/ubuntu_install_core.sh
ENV TVM_VENV /venv/apache-tvm-py3.7
COPY python/bootstrap/lockfiles /install/python/bootstrap/lockfiles
COPY install/ubuntu_install_python.sh /install/ubuntu_install_python.sh
RUN bash /install/ubuntu_install_python.sh
RUN bash /install/ubuntu_install_python.sh 3.7
ENV PATH ${TVM_VENV}/bin:$PATH
ENV PYTHONNOUSERSITE 1 # Disable .local directory from affecting CI.

Expand Down
2 changes: 1 addition & 1 deletion docker/Dockerfile.demo_vitis_ai
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -35,7 +35,7 @@ RUN bash /install/ubuntu_install_vitis_ai_core.sh
ENV TVM_VENV /venv/apache-tvm-py3.7
COPY python/bootstrap/lockfiles /install/python/bootstrap/lockfiles
COPY install/ubuntu_install_python.sh /install/ubuntu_install_python.sh
RUN bash /install/ubuntu_install_python.sh
RUN bash /install/ubuntu_install_python.sh 3.7
ENV PATH ${TVM_VENV}/bin:$PATH
ENV PYTHONNOUSERSITE 1 # Disable .local directory from affecting CI.

Expand Down
35 changes: 35 additions & 0 deletions docker/install/ubuntu2004_install_llvm.sh
View file
Open in desktop
Original file line number Diff line number Diff line change
@@ -0,0 +1,35 @@
#!/bin/bash
# Licensed to the Apache Software Foundation (ASF) under one
# or more contributor license agreements. See the NOTICE file
# distributed with this work for additional information
# regarding copyright ownership. The ASF licenses this file
# to you under the Apache License, Version 2.0 (the
# "License"); you may not use this file except in compliance
# with the License. You may obtain a copy of the License at
#
# http://www.apache.org/licenses/LICENSE-2.0
#
# Unless required by applicable law or agreed to in writing,
# software distributed under the License is distributed on an
# "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
# KIND, either express or implied. See the License for the
# specific language governing permissions and limitations
# under the License.

set -euxo pipefail

echo deb http://apt.llvm.org/focal/ llvm-toolchain-focal main\
>> /etc/apt/sources.list.d/llvm.list

echo deb http://apt.llvm.org/focal/ llvm-toolchain-focal-13 main\
>> /etc/apt/sources.list.d/llvm.list

apt-key adv --keyserver keyserver.ubuntu.com --recv-keys 15CF4D18AF4F7421

apt-get update && apt-install-and-clear -y \
llvm-9 llvm-10 llvm-11 llvm-12 llvm-13 \
clang-9 libclang-9-dev \
clang-10 libclang-10-dev \
clang-11 libclang-11-dev \
clang-12 libclang-12-dev \
clang-13 libclang-13-dev
24 changes: 17 additions & 7 deletions docker/install/ubuntu_install_nrfjprog.sh
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -21,19 +21,29 @@ set -u
set -o pipefail
set -x

TEMP_PATH_NAME="nrfjprog_tmp_path"

# Install JLink
NRF_COMMANDLINE_TOOLS_FILE=nRFCommandLineToolsLinuxamd64.tar.gz
NRF_COMMANDLINE_TOOLS_URL=https://nsscprodmedia.blob.core.windows.net/prod/software-and-other-downloads/desktop-software/nrf-command-line-tools/sw/versions-10-x-x/10-12-1/nrfcommandlinetools10121linuxamd64.tar.gz
NRF_COMMANDLINE_TOOLS_INSTALLER=nRF-Command-Line-Tools_10_12_1_Linux-amd64.deb
JLINK_LINUX_INSTALLER=JLink_Linux_V688a_x86_64.deb
NRF_COMMANDLINE_TOOLS_URL=https://nsscprodmedia.blob.core.windows.net/prod/software-and-other-downloads/desktop-software/nrf-command-line-tools/sw/versions-10-x-x/10-18-1/nrf-command-line-tools-10.18.1_linux-amd64.tar.gz
JLINK_LINUX_INSTALLER=JLink_Linux_V780c_x86_64.deb
NRF_COMMANDLINE_TOOLS_SHA=5611536ca3377d64131ccd51232f9e33cde6d289b03ea33db0581a1288be8b0b10f995e2d60fdd4a3ce5a5c7b12bc85ddc672b282c9af8c5808707ab41543a7d

cd ~
mkdir -p nrfjprog
mkdir -p ${TEMP_PATH_NAME}
wget --no-verbose -O $NRF_COMMANDLINE_TOOLS_FILE $NRF_COMMANDLINE_TOOLS_URL
echo "$NRF_COMMANDLINE_TOOLS_SHA $NRF_COMMANDLINE_TOOLS_FILE" | sha512sum --check

cd nrfjprog
cd ${TEMP_PATH_NAME}
tar -xzvf "../${NRF_COMMANDLINE_TOOLS_FILE}"
apt-install-and-clear -y "./${JLINK_LINUX_INSTALLER}"
apt-install-and-clear -y "./${NRF_COMMANDLINE_TOOLS_INSTALLER}"

# Install nrfjprog
NRF_DEB_FILE=nrf-command-line-tools_amd64.deb
NRF_DEB_FILE_SHA=1f0339e16d50345ddde9757c2a4211361bcc78ff7371aac09decfffa809d86329001f5bc135f33dd154000a8f0da8bee4a0e80d3865ceff229f63ff9ace5ea95
wget --no-verbose -O $NRF_DEB_FILE https://nsscprodmedia.blob.core.windows.net/prod/software-and-other-downloads/desktop-software/nrf-command-line-tools/sw/versions-10-x-x/10-18-1/nrf-command-line-tools_10.18.1_amd64.deb
echo "$NRF_DEB_FILE_SHA $NRF_DEB_FILE" | sha512sum --check
apt-install-and-clear -y ./$NRF_DEB_FILE

cd ..
rm -rf nrfjprog "${NRF_COMMANDLINE_TOOLS_FILE}"
rm -rf ${TEMP_PATH_NAME} "${NRF_COMMANDLINE_TOOLS_FILE}"
23 changes: 20 additions & 3 deletions docker/install/ubuntu_install_python.sh
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -27,25 +27,42 @@ if [ -z "${TVM_VENV+x}" ]; then
exit 2
fi

if [ "$#" -lt 1 ]; then
echo "Usage: docker/install/ubuntu_install_python.sh <PYTHON_VERSION>"
exit -1
fi
PYTHON_VERSION=$1

if [ "${PYTHON_VERSION}" != "3.7" ] && [ "${PYTHON_VERSION}" != "3.8" ]; then
echo "Only 3.7 and 3.8 versions are supported in this script."
exit -1
fi

apt-get update

# Ensure lsb-release is installed.
apt-install-and-clear -y \
lsb-core

apt-install-and-clear -y software-properties-common

release=$(lsb_release -sc)
if [ "${release}" == "bionic" ]; then
PYTHON_VERSION=3.7
if [ "${PYTHON_VERSION}" == "3.8" ]; then
echo "Only 3.7 is supported for bionic in this script."
exit -1
fi
elif [ "${release}" == "focal" ]; then
PYTHON_VERSION=3.8
if [ "${PYTHON_VERSION}" == "3.7" ]; then
add-apt-repository -y ppa:deadsnakes/ppa
fi
else
echo "Don't know which version of python to install for lsb-release ${release}"
exit 2
fi

# Install python and pip. Don't modify this to add Python package dependencies,
# instead modify install_python_package.sh
apt-install-and-clear -y software-properties-common
apt-install-and-clear -y \
acl \
python${PYTHON_VERSION} \
Expand Down