Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Remove unnecessary intermediate layers in base-conda Dockerfile #5697

Merged
merged 8 commits into from
Jan 29, 2021
Merged

Remove unnecessary intermediate layers in base-conda Dockerfile #5697

Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
8 commits
Select commit Hold shift + click to select a range
c5ce784
[docker][base-conda] Combine ENV+COPY instructions
sumanthratna Jan 28, 2021
23cfc96
[docker][base-cuda] Combine ENV+COPY instructions
sumanthratna Jan 28, 2021
8b92804
[docker][base-xla] Combine ENV+COPY instructions
sumanthratna Jan 28, 2021
64c39ff
[docker][base-cuda] Fix COPY instruction
sumanthratna Jan 28, 2021
b0a13b2
[docker][base-xla] Fix quote in ENV
sumanthratna Jan 28, 2021
4eeebe3
[docker][base-xla] Fix $PATH in ENV
sumanthratna Jan 28, 2021
9b9a633
[docker][base-conda] Fix COPY instruction
sumanthratna Jan 28, 2021
dc60878
chlog
Borda Jan 29, 2021
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
2 changes: 2 additions & 0 deletions CHANGELOG.md
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -29,6 +29,8 @@ The format is based on [Keep a Changelog](http://keepachangelog.com/en/1.0.0/).

- Fixed a race condition in `ModelCheckpoint` when checking if a checkpoint file exists ([#5144](https://github.com/PyTorchLightning/pytorch-lightning/pull/5144))

- Remove unnecessary intermediate layers in Dockerfiles ([#5697](https://github.com/PyTorchLightning/pytorch-lightning/pull/5697))


## [1.1.6] - 2021-01-26

Expand Down
39 changes: 20 additions & 19 deletions dockers/base-conda/Dockerfile
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -59,21 +59,21 @@ RUN apt-get update -qq && \
rm -rf /root/.cache && \
rm -rf /var/lib/apt/lists/*

ENV PATH="/root/miniconda3/bin:$PATH"
ENV LD_LIBRARY_PATH="/root/miniconda3/lib:$LD_LIBRARY_PATH"
ENV CUDA_TOOLKIT_ROOT_DIR="/usr/local/cuda"

ENV HOROVOD_GPU_OPERATIONS=NCCL
ENV HOROVOD_WITH_PYTORCH=1
ENV HOROVOD_WITHOUT_TENSORFLOW=1
ENV HOROVOD_WITHOUT_MXNET=1
ENV HOROVOD_WITH_GLOO=1
ENV HOROVOD_WITHOUT_MPI=1
#ENV MAKEFLAGS="-j$(nproc)"
ENV MAKEFLAGS="-j1"
ENV TORCH_CUDA_ARCH_LIST="3.7;5.0;6.0;7.0;7.5"

ENV CONDA_ENV=lightning
ENV \
PATH="/root/miniconda3/bin:$PATH" \
LD_LIBRARY_PATH="/root/miniconda3/lib:$LD_LIBRARY_PATH" \
CUDA_TOOLKIT_ROOT_DIR="/usr/local/cuda" \
HOROVOD_GPU_OPERATIONS=NCCL \
HOROVOD_WITH_PYTORCH=1 \
HOROVOD_WITHOUT_TENSORFLOW=1 \
HOROVOD_WITHOUT_MXNET=1 \
HOROVOD_WITH_GLOO=1 \
HOROVOD_WITHOUT_MPI=1 \
# MAKEFLAGS="-j$(nproc)" \
MAKEFLAGS="-j1" \
TORCH_CUDA_ARCH_LIST="3.7;5.0;6.0;7.0;7.5" \
CONDA_ENV=lightning

COPY environment.yml environment.yml

# conda init
Expand All @@ -90,10 +90,11 @@ RUN conda create -y --name $CONDA_ENV python=${PYTHON_VERSION} pytorch=${PYTORCH
conda clean -ya && \
rm environment.yml

ENV PATH /root/miniconda3/envs/${CONDA_ENV}/bin:$PATH
ENV LD_LIBRARY_PATH="/root/miniconda3/envs/${CONDA_ENV}/lib:$LD_LIBRARY_PATH"
# if you want this environment to be the default one, uncomment the following line:
ENV CONDA_DEFAULT_ENV=${CONDA_ENV}
ENV \
PATH /root/miniconda3/envs/${CONDA_ENV}/bin:$PATH \
LD_LIBRARY_PATH="/root/miniconda3/envs/${CONDA_ENV}/lib:$LD_LIBRARY_PATH" \
# if you want this environment to be the default one, uncomment the following line:
CONDA_DEFAULT_ENV=${CONDA_ENV}

COPY ./requirements/extra.txt requirements-extra.txt
COPY ./requirements/test.txt requirements-test.txt
Expand Down
29 changes: 15 additions & 14 deletions dockers/base-cuda/Dockerfile
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -32,11 +32,11 @@ ARG CMAKE_VERSION=3.18.4

SHELL ["/bin/bash", "-c"]
# https://techoverflow.net/2019/05/18/how-to-fix-configuring-tzdata-interactive-input-when-building-docker-images/
ENV DEBIAN_FRONTEND=noninteractive
ENV TZ=Europe/Prague

ENV PATH="$PATH:/root/.local/bin"
ENV CUDA_TOOLKIT_ROOT_DIR="/usr/local/cuda"
ENV \
DEBIAN_FRONTEND=noninteractive \
TZ=Europe/Prague \
PATH="$PATH:/root/.local/bin" \
CUDA_TOOLKIT_ROOT_DIR="/usr/local/cuda"

RUN apt-get update -qq && \
apt-get install -y --no-install-recommends \
Expand Down Expand Up @@ -68,15 +68,16 @@ RUN apt-get update -qq && \
rm -rf /root/.cache && \
rm -rf /var/lib/apt/lists/*

ENV HOROVOD_GPU_OPERATIONS=NCCL
ENV HOROVOD_WITH_PYTORCH=1
ENV HOROVOD_WITHOUT_TENSORFLOW=1
ENV HOROVOD_WITHOUT_MXNET=1
ENV HOROVOD_WITH_GLOO=1
ENV HOROVOD_WITHOUT_MPI=1
#ENV MAKEFLAGS="-j$(nproc)"
ENV MAKEFLAGS="-j1"
ENV TORCH_CUDA_ARCH_LIST="3.7;5.0;6.0;7.0;7.5"
ENV \
HOROVOD_GPU_OPERATIONS=NCCL \
HOROVOD_WITH_PYTORCH=1 \
HOROVOD_WITHOUT_TENSORFLOW=1 \
HOROVOD_WITHOUT_MXNET=1 \
HOROVOD_WITH_GLOO=1 \
HOROVOD_WITHOUT_MPI=1 \
# MAKEFLAGS="-j$(nproc)" \
MAKEFLAGS="-j1" \
TORCH_CUDA_ARCH_LIST="3.7;5.0;6.0;7.0;7.5"

COPY ./requirements.txt requirements.txt
COPY ./requirements/ ./requirements/
Expand Down
19 changes: 11 additions & 8 deletions dockers/base-xla/Dockerfile
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -25,8 +25,9 @@ SHELL ["/bin/bash", "-c"]

ARG CONDA_VERSION=4.7.12
# for skipping configurations
ENV DEBIAN_FRONTEND=noninteractive
ENV CONDA_ENV=lightning
ENV \
DEBIAN_FRONTEND=noninteractive \
CONDA_ENV=lightning

# show system inforation
RUN lsb_release -a && cat /etc/*-release
Expand All @@ -53,8 +54,9 @@ RUN apt-get update -qq && \
rm -rf /root/.cache && \
rm -rf /var/lib/apt/lists/*

ENV PATH="/root/miniconda3/bin:$PATH"
ENV LD_LIBRARY_PATH="/root/miniconda3/lib:$LD_LIBRARY_PATH"
ENV \
PATH="/root/miniconda3/bin:$PATH" \
LD_LIBRARY_PATH="/root/miniconda3/lib:$LD_LIBRARY_PATH"
COPY environment.yml environment.yml

RUN conda create -y --name $CONDA_ENV && \
Expand All @@ -67,10 +69,11 @@ RUN conda create -y --name $CONDA_ENV && \
conda clean -ya && \
rm environment.yml

ENV PATH /root/miniconda3/envs/${CONDA_ENV}/bin:$PATH
ENV LD_LIBRARY_PATH="/root/miniconda3/envs/${CONDA_ENV}/lib:$LD_LIBRARY_PATH"
# if you want this environment to be the default one, uncomment the following line:
ENV CONDA_DEFAULT_ENV=${CONDA_ENV}
ENV \
PATH=/root/miniconda3/envs/${CONDA_ENV}/bin:$PATH \
LD_LIBRARY_PATH="/root/miniconda3/envs/${CONDA_ENV}/lib:$LD_LIBRARY_PATH" \
# if you want this environment to be the default one, uncomment the following line:
CONDA_DEFAULT_ENV=${CONDA_ENV}

# Disable cache
RUN pip --version && \
Expand Down