Make pip install faster in Docker build for Complement testing

[MadLittleMods]

Make pip install faster in Docker build for Complement testing.

If files have changed in a COPY command, Docker will invalidate all of the layers below. So I changed the order of operations to install all dependencies before we COPY synapse /synapse/synapse/. This allows Docker to use our cached layer of dependencies even when we change the source of Synapse and speed up builds dramatically! 53.5s -> 3.7s builds 🤘

As an alternative, I did try using BuildKit caches but this still took 30 seconds overall on that step. 15 seconds to gather the dependencies from the cache and another 15 seconds to Installing collected packages.

Fix #9364

How does it perform?

On the first run and cold cache, it takes 53.5s to install the dependencies and now on the second run, we just have to wait 3.7s for the synapse package itself to install!

First run takes 53.5s to install all of the python dependencies. And 3.7s to install the synapse package itself.

./scripts-dev/complement.sh
[+] Building 96.7s (19/19) FINISHED
 => [internal] load build definition from Dockerfile                                                                                                                     0.0s
 => => transferring dockerfile: 2.30kB                                                                                                                                   0.0s
 => [internal] load .dockerignore                                                                                                                                        0.0s
 => => transferring context: 35B                                                                                                                                         0.0s
 => [internal] load metadata for docker.io/library/python:3.8-slim                                                                                                       0.3s
 => CACHED [builder  1/10] FROM docker.io/library/python:3.8-slim@sha256:1389669225e7fa05a9bac20d64551b6b6d84ee3200330d8d8de74c6d2314fdc7                                0.0s
 => [internal] load build context                                                                                                                                        0.1s
 => => transferring context: 59.84kB                                                                                                                                     0.1s
 => [builder  2/10] RUN echo "asdf"                                                                                                                                      0.4s
 => [builder  3/10] RUN apt-get update && apt-get install -y     build-essential     libffi-dev     libjpeg-dev     libpq-dev     libssl-dev     libwebp-dev     libxm  33.6s
 => [builder  4/10] COPY scripts /synapse/scripts/                                                                                                                       0.1s
 => [builder  5/10] COPY MANIFEST.in README.rst setup.py synctl /synapse/                                                                                                0.2s
 => [builder  6/10] COPY synapse/__init__.py /synapse/synapse/__init__.py                                                                                                0.1s
 => [builder  7/10] COPY synapse/python_dependencies.py /synapse/synapse/python_dependencies.py                                                                          0.0s
+ => [builder  8/10] RUN pip install --prefix="/install" --no-warn-script-location     /synapse[all]                                                                     53.5s
 => [builder  9/10] COPY synapse /synapse/synapse/                                                                                                                       0.2s
+ => [builder 10/10] RUN pip install --prefix="/install" --no-deps --no-warn-script-location /synapse                                                                     3.7s
 => CACHED [stage-1 2/5] RUN apt-get update && apt-get install -y     curl     gosu     libjpeg62-turbo     libpq5     libwebp6     xmlsec1     && rm -rf /var/lib/apt/  0.0s
 => [stage-1 3/5] COPY --from=builder /install /usr/local                                                                                                                1.2s
 => [stage-1 4/5] COPY ./docker/start.py /start.py                                                                                                                       0.0s
 => [stage-1 5/5] COPY ./docker/conf /conf                                                                                                                               0.1s
 => exporting to image                                                                                                                                                   1.7s
 => => exporting layers                                                                                                                                                  1.7s
 => => writing image sha256:568efab19bf9bc7b1ef09801646cf48b7850491d64cf02da631375d1e706346e                                                                             0.0s
 => => naming to docker.io/matrixdotorg/synapse:latest                                                                                                                   0.0s

Make a change to one of the source files like synapse/rest/client/v1/room.py which would normally invalidate all of the Docker cache layers everything before.

Second run we just have to wait for the 3.8s to install the synapse packages.

./scripts-dev/complement.sh
[+] Building 8.9s (18/18) FINISHED
 => [internal] load build definition from Dockerfile                                                                                                                     0.0s
 => => transferring dockerfile: 37B                                                                                                                                      0.0s
 => [internal] load .dockerignore                                                                                                                                        0.0s
 => => transferring context: 35B                                                                                                                                         0.0s
 => [internal] load metadata for docker.io/library/python:3.8-slim                                                                                                       0.4s
 => [builder 1/9] FROM docker.io/library/python:3.8-slim@sha256:1389669225e7fa05a9bac20d64551b6b6d84ee3200330d8d8de74c6d2314fdc7                                         0.0s
 => [internal] load build context                                                                                                                                        0.1s
 => => transferring context: 95.36kB                                                                                                                                     0.1s
 => CACHED [builder 2/9] RUN apt-get update && apt-get install -y     build-essential     libffi-dev     libjpeg-dev     libpq-dev     libssl-dev     libwebp-dev     l  0.0s
 => CACHED [builder 3/9] COPY scripts /synapse/scripts/                                                                                                                  0.0s
 => CACHED [builder 4/9] COPY MANIFEST.in README.rst setup.py synctl /synapse/                                                                                           0.0s
 => CACHED [builder 5/9] COPY synapse/__init__.py /synapse/synapse/__init__.py                                                                                           0.0s
 => CACHED [builder 6/9] COPY synapse/python_dependencies.py /synapse/synapse/python_dependencies.py                                                                     0.0s
+ => CACHED [builder 7/9] RUN pip install --prefix="/install" --no-warn-script-location     /synapse[all]                                                                 0.0s
 => [builder 8/9] COPY synapse /synapse/synapse/                                                                                                                         0.2s
+ => [builder 9/9] RUN pip install --prefix="/install" --no-deps --no-warn-script-location /synapse                                                                       3.8s
 => CACHED [stage-1 2/5] RUN apt-get update && apt-get install -y     curl     gosu     libjpeg62-turbo     libpq5     libwebp6     xmlsec1     && rm -rf /var/lib/apt/  0.0s
 => [stage-1 3/5] COPY --from=builder /install /usr/local                                                                                                                1.1s
 => [stage-1 4/5] COPY ./docker/start.py /start.py                                                                                                                       0.0s
 => [stage-1 5/5] COPY ./docker/conf /conf                                                                                                                               0.0s
 => exporting to image                                                                                                                                                   1.7s
 => => exporting layers                                                                                                                                                  1.7s
 => => writing image sha256:fb2d712394ebb3dca26f1d8b5ff20c3ed68ddb59523db44cebce9a7d7a52d4e0                                                                             0.0s
 => => naming to docker.io/matrixdotorg/synapse:latest                                                                                                                   0.0s

Dev notes

BuildKit cache

Using BuildKit caches,

• https://abe-winter.github.io/caching/2020/05/26/runmount.html
• https://stackoverflow.com/a/58021389/796832
• https://gist.github.com/RainJayTsai/16979ee1687546f93827ba78e5657196

# syntax = docker/dockerfile:experimental

RUN --mount=type=cache,target=/install pip install --prefix="/install" --no-warn-script-location \
        /synapse[all]

Doesn't seem to work or speed things up. Not sure what I am missing. Still 30 seconds.

Changing the app COPY to be after the install

If any of the files we COPY in change, that invalidates all later layers: we’ll need to rerun pip install, for example.

So only copying what is necessary for the install and doing that before copying all of synapse, https://pythonspeed.com/articles/docker-caching-model/

# Copy just what we need to pip install
COPY scripts /synapse/scripts/
COPY MANIFEST.in README.rst setup.py synctl /synapse/
COPY synapse/__init__.py /synapse/synapse/__init__.py
COPY synapse/python_dependencies.py /synapse/synapse/python_dependencies.py
# To speed up rebuilds, install all of the dependencies before we copy over
# the whole synapse project so that we this layer in the Docker cache can be
# used while you develop on the source
RUN pip install --prefix="/install" --no-warn-script-location \
    /synapse[all]

Previous outdated problems

COPY setup.py /synapse/setup.py
COPY synapse/__init__.py /synapse/synapse/__init__.py
COPY synapse/python_dependencies.py /synapse/synapse/python_dependencies.py

RUN pip install --prefix="/install" --no-warn-script-location \
        /synapse[all]

This results in an error currently:

Update: To fix this, just needed to copy over README.rst from Synapse as well. This was being used by setup.py.

$ ./scripts-dev/complement.sh

 => ERROR [builder  7/10] RUN pip install --prefix="/install" --no-warn-script-location         /synapse[all]                                                            0.9s
------
 > [builder  7/10] RUN pip install --prefix="/install" --no-warn-script-location         /synapse[all]:
#14 0.662 Processing /synapse
#14 0.815     ERROR: Command errored out with exit status 1:
#14 0.815      command: /usr/local/bin/python -c 'import sys, setuptools, tokenize; sys.argv[0] = '"'"'/tmp/pip-req-build-ltrplihy/setup.py'"'"'; __file__='"'"'/tmp/pip-req-build-ltrplihy/setup.py'"'"';f=getattr(tokenize, '"'"'open'"'"', open)(__file__);code=f.read().replace('"'"'\r\n'"'"', '"'"'\n'"'"');f.close();exec(compile(code, __file__, '"'"'exec'"'"'))' egg_info --egg-base /tmp/pip-pip-egg-info-t3dwmd5d
#14 0.815          cwd: /tmp/pip-req-build-ltrplihy/
#14 0.815     Complete output (7 lines):
#14 0.815     Traceback (most recent call last):
#14 0.815       File "<string>", line 1, in <module>
#14 0.815       File "/tmp/pip-req-build-ltrplihy/setup.py", line 86, in <module>
#14 0.815         long_description = read_file(("README.rst",))
#14 0.815       File "/tmp/pip-req-build-ltrplihy/setup.py", line 72, in read_file
#14 0.815         with open(file_path) as f:
#14 0.815     FileNotFoundError: [Errno 2] No such file or directory: '/tmp/pip-req-build-ltrplihy/README.rst'
#14 0.815     ----------------------------------------
#14 0.816 WARNING: Discarding file:///synapse. Command errored out with exit status 1: python setup.py egg_info Check the logs for full command output.
#14 0.816 ERROR: Command errored out with exit status 1: python setup.py egg_info Check the logs for full command output.
------
executor failed running [/bin/sh -c pip install --prefix="/install" --no-warn-script-location         /synapse[all]]: exit code: 1

Other stuff

$ docker run -it --entrypoint /bin/bash matrixdotorg/synapse:latest

root@d970b1f1386b:/# which generate_signing_key.py
/usr/local/bin/generate_signing_key.py

root@d970b1f1386b:/# python
>>> import synapse.util
>>> synapse.util.__file__
'/usr/local/lib/python3.8/site-packages/synapse/util/__init__.py'

Pull Request Checklist

• Pull request is based on the develop branch
• Pull request includes a changelog file. The entry should:
  • Be a short description of your change which makes sense to users. "Fixed a bug that prevented receiving messages from other servers." instead of "Moved X method from EventStore to EventWorkerStore.".
  • Use markdown where necessary, mostly for code blocks.
  • End with either a period (.) or an exclamation mark (!).
  • Start with a capital letter.
• Pull request includes a sign off
• Code style is correct (run the linters)

[MadLittleMods]

Auto-formatting from VSCode to make everything 4 spaces

[MadLittleMods]

Our new step installs all dependencies before we COPY the synapse source, not just ones without wheels.

[MadLittleMods]

All of the new goodness is here!

If files have changed in a COPY command, Docker will invalidate all of the layers below. So I changed the order of operations to install all dependencies before we COPY synapse /synapse/synapse/. This allows Docker to use our cached layer of dependencies even when we change the source of Synapse and speed up builds dramatically!

[MadLittleMods]

I learned a lot about setuptools and Python pip installs for this 😅

setup.py, setuptools, the package itself, install_requires, extras_require 🥴

[MadLittleMods]

@richvdh Assigned you because I saw you touched the Docker layer caching stuff before. Feel free to assign to the general reviewer group though.

[callahad]

Not a formal review, but this looks reasonable enough to me.

Bit weird to install /synapse[all] twice, but if it works...

[anoadragon453]

Bit weird to install /synapse[all] twice, but if it works...

There might be a better way to represent it, but it makes sense. The first is for installing all of our dependencies, the second is just for installing Synapse's files.

I presume that if we add a new dependency to the project then we'd need to go back and run both again.

I tried this out with the Complement Workers stack of docker images and it looks to work just as well, and is much faster to make changes indeed!

[MadLittleMods]

@anoadragon453 What's needed to merge this now? (I see approval)

[anoadragon453]

Oh sorry, we typically expect team members to merge their PRs after they've been approved (assuming you have the ability to merge PRs?). But I probably should have mentioned that.

So feel free to merge, otherwise I can do so for you 🙂

[anoadragon453]

I'll take that as me merging it then 🙂

[MadLittleMods]

Thanks @anoadragon453! I just pushed up the change and went to bed (to let the CI run).