Switch to pipenv or other enhanced dependency manager? At least partially

[abulte]

Following #2182 and #2172.

We regularly have trouble with transitive dependencies breaking "randomly" on different environments:

• circle CI (behaves either as a brand new env or a previous install, depending on cache)
• local env (usually not a problem, that's where we fix things)
• production and staging
• brand new environments (people doing pip install udata or using https://github.com/opendatateam/docker-udata)

Example: jsonschema gets upgraded, either because it's not pinned and thus installed at the latest version in a new env, or because it's pinned for various reasons (as happened in #2172). jsonschema==3.0.1 depends on six>=1.11.0, but sometimes an older version is already installed. In those cases, pip install won't upgrade six, thus breaking the install.

Proposed minimal solution: generate requirement files from pipenv lock -r after creating a venv with pipenv install -r requirements/install.pip. This has the advantage of pinning all the dependencies, thus leaving no room for discrepancies among different environments. In our example, if jsonschema==3.0.1 is pinned in install.pip, six==1.12.0 will be pinned in the new install.pip.

By generating a standard pip requirement file, we do not need to change our production deploy process (just do a pip install -r). Still, there are some impacts:

• we need to switch to pipenv on our local envs and maintain a Pipfile and Pipfile.lock beside the usual requirement file (or just do generate the "extended" requirements once with pipenv and then still rely on pip? I'm quite sure this will lead to troubles, and it's cleaner to just list our core dependencies in a Pipfile)
• https://pyup.io will try to update each and every dependencies in our project, which will quickly become a mess. There's seems to be partial support of Pipfile though Support Pipfiles pyupio/pyup#197
• we need to generate the requirements files every time we add something to Pipfile: maybe this be automated at release time?

We could also switch completely to pipenv, including in our deploy process, but I feel this is quite touchy.

[abulte]

For reference, this is the requirement file produced by pipenv after initialising it with install.pip:

-i https://pypi.org/simple
amqp==2.5.0
aniso8601==6.0.0
asn1crypto==0.24.0
attrs==19.1.0
authlib==0.10
awesome-slugify==1.6.5
babel==2.6.0
bcrypt==3.1.6
billiard==3.5.0.5
bleach==3.1.0
blinker==1.4
celery==4.1.1
celerybeat-mongo==0.1.0
certifi==2019.3.9
cffi==1.12.3
chardet==3.0.4
click==6.7
commonmark==0.8.1
cryptography==2.7
elasticsearch-dsl==2.2.0
elasticsearch==2.4.1
enum34==1.1.6 ; python_version < '3'
factory-boy==2.11.1
faker==1.0.5
flask-babelex==0.9.3
flask-caching==1.4.0
flask-cdn==1.5.3
flask-fs==0.6.1
flask-gravatar==0.5.0
flask-login==0.4.1
flask-mail==0.9.1
flask-mongoengine==0.9.5
flask-navigation==0.2.0
flask-principal==0.4.0
flask-restplus==0.12.1
flask-security==3.0.0
flask-sitemap==0.3.0
flask-themes2==0.1.4
flask-wtf==0.14.2
flask==1.0.2
functools32==3.2.3.post2 ; python_version < '3'
future==0.17.1
geojson==2.4.1
html2text==2018.1.9
idna==2.8
ipaddress==1.0.22 ; python_version < '3'
isodate==0.6.0
itsdangerous==1.1.0
jinja2==2.10.1
jsonschema==3.0.1
kombu[redis]==4.2.1
lxml==4.3.3
markupsafe==1.1.1
mongoengine==0.16.3
msgpack-python==0.4.8
netaddr==0.7.19
passlib==1.7.1
pillow==6.0.0
pycparser==2.19
pydenticon==0.3.1
pyliblzma==0.5.3
pymongo==3.7.2
pyparsing==2.4.0
pyrsistent==0.15.2
python-dateutil==2.8.0
pytz==2019.1
pyyaml==5.1
rdflib-jsonld==0.4.0
rdflib==4.2.2
redis==2.10.6
regex==2019.6.5
requests==2.21.0
simplejson==3.16.0
six==1.12.0
speaklater==1.3
stringdist==1.0.9
text-unidecode==1.2
tlds==2019060500
ujson==1.35
unicodecsv==0.14.1
unidecode==0.4.21
urllib3==1.24.3
vine==1.3.0
voluptuous==0.10.5
webencodings==0.5.1
werkzeug==0.14.1
wtforms-json==0.3.3
wtforms==2.2.1
xmltodict==0.12.0

[abulte]

Maybe this could help w/o switching away from pip https://github.com/jazzband/pip-tools/