This guide answers the following questions:
- How do I install the proper dependencies?
- How can a developer upgrade packages in the project?
- Why isn't pip-compile upgrading to the absolute latest?!
- Why pip-tools?
# Activate your virtual env
...
# Install pip-tools
pip install pip-tools
# Make your env exactly mirror production
pip-sync requirements.txt
# Or make your env mirror a development version of production
pip-sync dev_requirements.txt
# Make changes to either dev_requirements.in or requirements.in
...
# Generate new production requirements
pip-compile -o requirements.txt requirements.in
# Generate new development requirements (depends on previous command!!)
pip-compile -o dev_requirements.txt dev_requirements.in requirements.txt
NOTE: If you forget how a requirements file is generated, the instructions are at the top of it.
The second command deserves more explanation. The second specification passed to pip-compile is the generated file from the first pip-compile. This ensures that the development environment includes the exact versions included in the production environment.
There is a nuance in how pip-compile generates the requirements. When you pass specifications to pip-compile you might assume that it uses the specs to find the latest possible packages. It doesn't work that way. If it did, then every time compile was run, your depencies could change drastically. While you want to be up-to-date, you don't want to have to track down issues because you installed many new versions simultaneously.
It's 1977 -what a great year- you run pip-compile for the first time. And it
behaves exactly as you would expect. Your requirements.in includes a single
line: django.
$ pip-compile -o requirements.txt requirements.in
...
django=0.1.3
...
You fast-forward in time to the current year. You run the same command.
$ pip-compile -o requirements.txt requirements.in
...
django=0.1.3
...
You know there is a newer version of django but nothing changed in the
generated requirements.
Your second command is actually more like:
$ pip-compile -o requirements.txt requirements.txt requirements.in
So pip-compile actually uses your existing requirements to guide how it generates the dependencies. This means that you the developer decide when to upgrade, compiling doesn't do it for you.
To recover from django circa 1977.
$ pip-compile -P django requirements.in
Pip-tools solve a singular problem.
When you deploy to production, you want to specify an exact set of packages versions to be installed. This makes it possible to have highly reproducible builds.
However, development of an application requires constantly changing dependencies. It's much easier to specify loose requirements than exact versions for several reasons. Subsequent installs slowly migrate your application forward. This ensures you get bug-fixes, new-features, and that larger migrations are less painful.
Pip-tools is a suite of tools to address the competing desires of production and development environments.
pip-compile accepts a loose set of requirements and generates a strict set of requirements. When you want to upgrade, simply adjust your loose requirements to include the new dependency, and regenerate the strict dependencies (see the section above on upgrading packages).
Suppose we define the file loose.txt:
django>1.3
Then we generate the strict version
$ pip-compile -o strict.txt loose.txt
The file strict.txt now contains:
#
# This file is autogenerated by pip-compile
# To update, run:
#
# pip-compile --output-file strict.txt /dev/stdin
#
django==1.11.2
pytz==2017.2 # via django
Pretty neat, because we didn't know we needed pytz and we'd like to keep it
that way!
Now you just need to run:
# Note: you should already be inside of a virtual env
pip-sync strict.txt
This will ensure that your virtual environment exactly mirrors strict.txt.
You could run pip install strict.txt but that fails to remove packages,
which makes it easy to forget to include dependencies in revision control
because they already exist in your environment.