Skip to content

[SPARK-45096][INFRA] Optimize apt-get install in Dockerfile #42842

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Closed
zhengruifeng wants to merge 2 commits into from
Closed

[SPARK-45096][INFRA] Optimize apt-get install in Dockerfile #42842

Changes from all commits
Commits
Show all changes
2 commits
Select commit Hold shift + click to select a range
d857a52
init
zhengruifeng Aug 22, 2023
2426087
address comments
zhengruifeng Sep 7, 2023
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
50 changes: 35 additions & 15 deletions dev/infra/Dockerfile
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -24,19 +24,44 @@ ENV FULL_REFRESH_DATE 20221118
ENV DEBIAN_FRONTEND noninteractive
ENV DEBCONF_NONINTERACTIVE_SEEN true

ARG APT_INSTALL="apt-get install --no-install-recommends -y"

RUN apt-get clean
RUN apt-get update
RUN $APT_INSTALL software-properties-common git libxml2-dev pkg-config curl wget openjdk-8-jdk libpython3-dev python3-pip python3-setuptools python3.8 python3.9
RUN update-alternatives --set java /usr/lib/jvm/java-8-openjdk-amd64/jre/bin/java
RUN apt-get update && apt-get install -y \
software-properties-common \
git \
pkg-config \
curl \
wget \
openjdk-8-jdk \
gfortran \
libopenblas-dev \
liblapack-dev \
build-essential \
gnupg \
ca-certificates \
pandoc \
libpython3-dev \
python3-pip \
python3-setuptools \
python3.8 \
python3.9 \
r-base \
libcurl4-openssl-dev \
qpdf \
zlib1g-dev \
libssl-dev \
libpng-dev \
libharfbuzz-dev \
libfribidi-dev \
libtiff5-dev \
libgit2-dev \
libxml2-dev \
libjpeg-dev \
libfontconfig1-dev \
libfreetype6-dev \
&& rm -rf /var/lib/apt/lists/*

RUN curl -sS https://bootstrap.pypa.io/get-pip.py | python3.9

RUN add-apt-repository ppa:pypy/ppa
RUN apt update
RUN $APT_INSTALL gfortran libopenblas-dev liblapack-dev
RUN $APT_INSTALL build-essential

RUN mkdir -p /usr/local/pypy/pypy3.8 && \
curl -sqL https://downloads.python.org/pypy/pypy3.8-v7.3.11-linux64.tar.bz2 | tar xjf - -C /usr/local/pypy/pypy3.8 --strip-components=1 && \
Expand All @@ -45,19 +70,14 @@ RUN mkdir -p /usr/local/pypy/pypy3.8 && \

RUN curl -sS https://bootstrap.pypa.io/get-pip.py | pypy3

RUN $APT_INSTALL gnupg ca-certificates pandoc
RUN echo 'deb https://cloud.r-project.org/bin/linux/ubuntu focal-cran40/' >> /etc/apt/sources.list
RUN gpg --keyserver hkps://keyserver.ubuntu.com --recv-key E298A3A825C0D65DFD57CBB651716619E084DAB9
RUN gpg -a --export E084DAB9 | apt-key add -
RUN add-apt-repository 'deb https://cloud.r-project.org/bin/linux/ubuntu focal-cran40/'
RUN apt update
RUN $APT_INSTALL r-base libcurl4-openssl-dev qpdf libssl-dev zlib1g-dev

RUN Rscript -e "install.packages(c('knitr', 'markdown', 'rmarkdown', 'testthat', 'devtools', 'e1071', 'survival', 'arrow', 'roxygen2', 'xml2'), repos='https://cloud.r-project.org/')"

# See more in SPARK-39959, roxygen2 < 7.2.1
Copy link
Member

@Yikun Yikun Sep 7, 2023

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

So L58 not change is intentional, right?

Copy link
Contributor Author

@zhengruifeng zhengruifeng Sep 7, 2023 •
edited
Loading

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

no, should be also moved in L27, let me update

RUN apt-get install -y libcurl4-openssl-dev libgit2-dev libssl-dev libxml2-dev \
libfontconfig1-dev libharfbuzz-dev libfribidi-dev libfreetype6-dev libpng-dev \
libtiff5-dev libjpeg-dev
RUN Rscript -e "install.packages(c('devtools'), repos='https://cloud.r-project.org/')"
RUN Rscript -e "devtools::install_version('roxygen2', version='7.2.0', repos='https://cloud.r-project.org')"

Expand Down