Deploy fails due to outdated Pipfile.lock #704

adamkowalczyk · 2018-05-29T15:38:13Z

Our heroku builds have started failing today with the error message

Your Pipfile.lock (<local hash>) is out of date. Expected: (<new hash>)

I notice that you have bumped the pipenv version today, and I have also updated my local pip (10.0.1) and and pipenv (2018.05.18) installations. However, I am unable to get my Pipfile.lock hash to change to <new hash> -running pipenv lock does not change the hash at all.

I am at a loss as to what is causing my local Pipfile.lock hash to be different to the one generated in the heroku build, or how to fix the problem.

The text was updated successfully, but these errors were encountered:

adamkowalczyk · 2018-05-30T11:39:41Z

Rolling back to v134 has fixed this issue for now.

Before the deploy fails due to the outdated Pipfile.lock hash, there are a number of lines logged along the lines of
Adding -e git+https://github.com/quru/django-waffle@730ee99b892ced05630fe8be61317b79540a0054#egg=django-waffle to Pipfile's [packages]… relating to our packages installed via github url, which we do not normally see. Is something happening here to invalidate our Pipfile.lock?

Our error output:

Counting objects: 180, done.
Delta compression using up to 32 threads.
Compressing objects: 100% (113/113), done.
Writing objects: 100% (180/180), 52.50 KiB | 0 bytes/s, done.
Total 180 (delta 117), reused 114 (delta 58)
remote: Compressing source files... done.        
remote: Building source:        
remote: 
remote: -----> Deleting 215 files matching .slugignore patterns.        
remote: -----> stunnel app detected        
remote: -----> Moving the configuration generation script into app/bin        
remote: -----> Moving the start-stunnel script into app/bin        
remote: -----> stunnel done        
remote: -----> Python app detected        
remote:  !     The latest version of Python 3 is python-3.6.5 (you are using python-3.6.4, which is unsupported).        
remote:  !     We recommend upgrading by specifying the latest version (python-3.6.5).        
remote:        Learn More: https://devcenter.heroku.com/articles/python-runtimes        
remote: -----> Installing pip        
remote: -----> Installing dependencies with Pipenv 2018.5.18…        
remote:        Installing -e git+https://github.com/photocrowd/dj-stripe@b8229cdc0ea400737ab5d34a6f2bee5665229f45#egg=dj-stripe…        
remote:        Looking in indexes: https://pypi.python.org/simple        
remote:        Obtaining dj-stripe from git+https://github.com/photocrowd/dj-stripe@b8229cdc0ea400737ab5d34a6f2bee5665229f45#egg=dj-stripe        
remote:          Cloning https://github.com/photocrowd/dj-stripe (to revision b8229cdc0ea400737ab5d34a6f2bee5665229f45) to ./src/dj-stripe        
remote:        Requirement already satisfied: django!=1.9,>=1.8 in ./.heroku/python/lib/python3.6/site-packages (from dj-stripe) (2.0.3)        
remote:        Requirement already satisfied: stripe>=1.22.2 in ./.heroku/python/lib/python3.6/site-packages (from dj-stripe) (1.79.1)        
remote:        Requirement already satisfied: django-model-utils>=2.2 in ./.heroku/python/lib/python3.6/site-packages (from dj-stripe) (3.1.1)        
remote:        Requirement already satisfied: django-braces>=1.8.0 in ./.heroku/python/lib/python3.6/site-packages (from dj-stripe) (1.12.0)        
remote:        Requirement already satisfied: jsonfield>=1.0.3 in ./.heroku/python/lib/python3.6/site-packages (from dj-stripe) (2.0.2)        
remote:        Requirement already satisfied: pytz>=2015.4 in ./.heroku/python/lib/python3.6/site-packages (from dj-stripe) (2018.3)        
remote:        Requirement already satisfied: requests>=0.8.8 in ./.heroku/python/lib/python3.6/site-packages (from stripe>=1.22.2->dj-stripe) (2.18.4)        
remote:        Requirement already satisfied: chardet<3.1.0,>=3.0.2 in ./.heroku/python/lib/python3.6/site-packages (from requests>=0.8.8->stripe>=1.22.2->dj-stripe) (3.0.4)        
remote:        Requirement already satisfied: idna<2.7,>=2.5 in ./.heroku/python/lib/python3.6/site-packages (from requests>=0.8.8->stripe>=1.22.2->dj-stripe) (2.6)        
remote:        Requirement already satisfied: certifi>=2017.4.17 in ./.heroku/python/lib/python3.6/site-packages (from requests>=0.8.8->stripe>=1.22.2->dj-stripe) (2018.4.16)        
remote:        Requirement already satisfied: urllib3<1.23,>=1.21.1 in ./.heroku/python/lib/python3.6/site-packages (from requests>=0.8.8->stripe>=1.22.2->dj-stripe) (1.22)        
remote:        Installing collected packages: dj-stripe        
remote:          Running setup.py develop for dj-stripe        
remote:        Successfully installed dj-stripe        
remote:                
remote:        Adding -e git+https://github.com/photocrowd/dj-stripe@b8229cdc0ea400737ab5d34a6f2bee5665229f45#egg=dj-stripe to Pipfile's [packages]…        
remote:        Installing -e git+https://github.com/matagus/django-pagination-py3@78d0ae7414b290f17e5fd825d6a25248cad7efbd#egg=django-pagination-py3…        
remote:        Looking in indexes: https://pypi.python.org/simple        
remote:        Obtaining django-pagination-py3 from git+https://github.com/matagus/django-pagination-py3@78d0ae7414b290f17e5fd825d6a25248cad7efbd#egg=django-pagination-py3        
remote:          Cloning https://github.com/matagus/django-pagination-py3 (to revision 78d0ae7414b290f17e5fd825d6a25248cad7efbd) to ./src/django-pagination-py3        
remote:        Installing collected packages: django-pagination-py3        
remote:          Running setup.py develop for django-pagination-py3        
remote:        Successfully installed django-pagination-py3        
remote:                
remote:        Adding -e git+https://github.com/matagus/django-pagination-py3@78d0ae7414b290f17e5fd825d6a25248cad7efbd#egg=django-pagination-py3 to Pipfile's [packages]…        
remote:        Installing -e git+https://github.com/quru/django-waffle@730ee99b892ced05630fe8be61317b79540a0054#egg=django-waffle…        
remote:        Looking in indexes: https://pypi.python.org/simple        
remote:        Obtaining django-waffle from git+https://github.com/quru/django-waffle@730ee99b892ced05630fe8be61317b79540a0054#egg=django-waffle        
remote:          Cloning https://github.com/quru/django-waffle (to revision 730ee99b892ced05630fe8be61317b79540a0054) to ./src/django-waffle        
remote:        Installing collected packages: django-waffle        
remote:          Running setup.py develop for django-waffle        
remote:        Successfully installed django-waffle        
remote:                
remote:        Adding -e git+https://github.com/quru/django-waffle@730ee99b892ced05630fe8be61317b79540a0054#egg=django-waffle to Pipfile's [packages]…        
remote:        Installing -e git+https://github.com/rtpg/django-floppyforms@e118e79c77664ea08ae72f59ea02412da7d9b3a8#egg=django-floppyforms…        
remote:        Looking in indexes: https://pypi.python.org/simple        
remote:        Obtaining django-floppyforms from git+https://github.com/rtpg/django-floppyforms@e118e79c77664ea08ae72f59ea02412da7d9b3a8#egg=django-floppyforms        
remote:          Cloning https://github.com/rtpg/django-floppyforms (to revision e118e79c77664ea08ae72f59ea02412da7d9b3a8) to ./src/django-floppyforms        
remote:        Installing collected packages: django-floppyforms        
remote:          Running setup.py develop for django-floppyforms        
remote:        Successfully installed django-floppyforms        
remote:                
remote:        Adding -e git+https://github.com/rtpg/django-floppyforms@e118e79c77664ea08ae72f59ea02412da7d9b3a8#egg=django-floppyforms to Pipfile's [packages]…        
remote:        Creating a virtualenv for this project…        
remote:        Using /app/.heroku/python/bin/python (3.6.4) to create virtualenv…        
remote:        Already using interpreter /app/.heroku/python/bin/python        
remote:        Using base prefix '/app/.heroku/python'        
remote:        New python executable in /app/.local/share/virtualenvs/build_f780e88524ece5fe8b1cb04f2864448f-dSm8IqzX/bin/python        
remote:        Installing setuptools, pip, wheel...done.        
remote:                
remote:        Virtualenv location: /app/.local/share/virtualenvs/build_f780e88524ece5fe8b1cb04f2864448f-dSm8IqzX        
remote:        Your Pipfile.lock (a71053) is out of date. Expected: (967902).        
remote:        Aborting deploy.        
remote:  !     Push rejected, failed to compile Python app.        
remote: 
remote:  !     Push failed        
remote: Verifying deploy...        
remote: 
remote: !	Push rejected to photocrowd-staging.        
remote: 
To [email protected]:photocrowd-staging.git
 ! [remote rejected] HEAD -> master (pre-receive hook declined)
error: failed to push some refs to '[email protected]:photocrowd-staging.git'
Exited with code 1

and our Pipfile:

[[source]]

url = "https://pypi.python.org/simple"
verify_ssl = true
name = "pypi"


[packages]

django = ">=2.0.3"
dj-stripe = {git = "https://github.com/photocrowd/dj-stripe", editable = true, ref = "b8229cdc0ea400737ab5d34a6f2bee5665229f45"}
"django-pagination-py3" = {git = "https://github.com/matagus/django-pagination-py3", editable = true, ref = "78d0ae7414b290f17e5fd825d6a25248cad7efbd"}
django-waffle = {git = "https://github.com/quru/django-waffle", editable = true, ref = "730ee99b892ced05630fe8be61317b79540a0054"}
django-floppyforms = {git = "https://github.com/rtpg/django-floppyforms", editable = true, ref = "e118e79c77664ea08ae72f59ea02412da7d9b3a8"}
babeldjango = "*"
coverage = "*"
django-braces = "*"
django-choice-object = "*"
django-compressor = "*"
django-countries = "*"
django-cursor-pagination = "*"
django-dirtyfields = "*"
django-mptt = "*"
django-orderable = "*"
django-prices = "<1.0.0"
django-redis-sessions = "*"
django-reversion = "*"
django-sendfile = "*"
django-timezone-field = "*"
djangorestframework = "*"
feincms = "*"
graphene = "*"
graphene-django = "*"
markdown = "*"
satchless = "*"
twitter-text = "*"
celery = "*"
collectfast = "*"
django-heroku-memcacheify = "*"
django-pylibmc = "*"
django-storages = "*"
django-webpack-loader = "*"
djrill = "*"
gunicorn = "*"
hirefire = "*"
newrelic = "*"
pillow = "*"
"psycopg2" = "*"
raven = "*"
redis = "*"
attrs = "*"
cyordereddict = "*"
dj-database-url = "*"
django-debug-toolbar = "*"
"enum34" = "*"
factory-boy = "*"
fakeredis = "*"
"flake8" = "*"
inflect = "*"
isort = "*"
mock = "*"
more-itertools = "*"
numpy = "*"
pyflakes = "*"
singledispatch = "*"
sphinx = "*"
unicodecsv = "*"
"boto3" = "*"
django-babel = "*"
faker = "*"
prices = "<1.0.0"
sphinx-rtd-theme = "*"


[dev-packages]



[requires]

python_version = "3.6"

jxltom · 2018-06-01T05:26:10Z

Save issue with #702

This is obviously a bug related with latest pipenv. It may related with pypa/pipenv#2219

Using pipenv version v11.9.0 or older version can fix issue for now.

heroku/heroku-buildpack-python#704

zyv · 2018-06-30T09:54:38Z

Got hit by the same bug, raised support ticket #602020, but sadly got no help so far 👎

How did you manage to downgrade the buildpack @adamkowalczyk, by forking it to your account, amending master and specifying it for the app instead of heroku/python buildpack?

I was hoping that I would be able to disable Pipfile support in the buildpack to have it fall back to pre-generated requirements.txt in some other way than removing Pipfile altogether, but, of course naively setting SKIP_PIPENV_INSTALL didn't work.

I think we would really benefit from a toggle to disable pipenv, the latest releases have been very problematic. It's great when it works, but when it doesn't you're just stuck. At least I'm able to generate a lock file with 2018.5.18, although on Heroku it still comes up with a different hash causing this issue (?!), but 2018.6.25 isn't even able to resolve our dependencies...

jxltom · 2018-06-30T10:09:56Z

@zyv Pin your buildpack version by https://github.com/heroku/heroku-buildpack-python#v130 or any other version should work.

edmorley · 2018-06-30T10:44:38Z

setting SKIP_PIPENV_INSTALL didn't work

Environment variables set in the app's config are not set in the environment during the build by default - instead individual steps during the build have to load them in from ENV_DIR (using the sub_env helper). See:
https://devcenter.heroku.com/articles/buildpack-api#bin-compile

I presume this is intended to prevent the build being broken by an app having variables set that shouldn't be, but it does mean that it's currently not possible to set SKIP_PIPENV_INSTALL manually (perhaps the maintainers of this buildpack would be open to changing this?).

For now an alternative workaround (instead of pinning to an old buildpack version), would be to use the pre_compile hook to delete Pipfile and Pipfile.lock before they are detected and used:
https://github.com/heroku/heroku-buildpack-python/blob/master/bin/steps/hooks/pre_compile

To do that, create a script named bin/pre_compile in your app's repository, containing:

#!/usr/bin/env bash

# Make non-zero exit codes & other errors fatal:
# http://redsymbol.net/articles/unofficial-bash-strict-mode/
set -euo pipefail

# Prevent heroku-buildpack-python from using pipenv instead of pip and requirements.txt.
rm Pipfile Pipfile.lock

zyv · 2018-06-30T10:49:23Z

Hey @edmorley, that sounds like a great workaround, thank you! I'll give it a whirl and report back.

edmorley · 2018-06-30T10:54:12Z

You're welcome! I should add a caveat that if pipenv is not currently reliable enough (either in general or just via this buildpack) that it's probably best to remove Pipfile and Pipfile.lock from the app repositories entirely (and rely on requirements.txt) rather than just remove on Heroku - otherwise it reduces dev-prod parity and unless tested in CI will inevitably result in dependency drift/errors in the future.

zyv · 2018-06-30T11:10:24Z

Just implemented it and it works like a charm!

This will allow us to keep dependencies under pipenv management / enjoy its support for working with virtualenvs, and, at the same time, hopefully never again waste days on fixing pipenv related problems on Heroku / always have a stable requirements.txt fallback.

@edmorley as to your additional point, thanks for this valuable hint as well, I'll just add a check to our CI script to compare requirements.txt and pipenv lock -r output to avoid forgetting to update requirements.txt, and hitting resulting problems w.r.t. systems parity.

I wish Heroku engineers were as helpful and responsive as you guys, so glad I stumbled upon this issue 👍

This also starts tracking which Pipenv version users are selecting to use. Closes #702 Closes #704 Closes #706 Closes #727

gedankenstuecke · 2018-07-13T18:01:16Z

Just to report some success in solving it and another work around:

@mcescalante and I ran into the same issue today. In our case it was fixed by removing a requirements that was to be installed from GitHub (c.f. https://github.com/OpenHumans/oh-fitbit-integration/pull/16/files#diff-1e61a31bf9b94805f869dc4137ec1885L20).

After removing the corresponding requirement the build on heroku would go through without any problems. As an alternative to that heroku buildpacks:set https://github.com/heroku/heroku-buildpack-python\#v134 did the trick as well.

CaseyFaist · 2018-10-11T20:38:40Z

Hey y'all! We just released a fix for a bug where installing pipenv caused pip to upgrade to latest - v144. If the latest release doesn't help with this issue, I'd love to know 👍

adamkowalczyk · 2018-10-17T15:23:52Z

@CaseyFaist No, doesn't look like anything has changed. Just tried a deploy with the latest version (v141) and the build failed.
Your Pipfile.lock (daa25b) is out of date. Expected: (78064f).

jxltom · 2018-10-18T01:07:29Z

@adamkowalczyk It is because this issue is caused by pipenv 2018.5.18. So this issue is not actually solved. Pin pipenv to version newer than 2018.6.25 can fix this.

However, other pipenv version has other critical issues. So maybe it is better to stay in this way until a stable pipenv version is released.

adamkowalczyk · 2018-10-18T09:05:07Z

@jxltom Thanks for the info. Are you aware of a pipenv issue relating to this problem? I can't seem to find one.

jxltom · 2018-10-18T09:10:26Z

@adamkowalczyk It should be pypa/pipenv#2219 and fixed in pypa/pipenv#2301

MalikRumi · 2019-06-29T21:12:16Z

This thread has been very helpful. I am back up and running on Heroku. Just one thing: When you change the name of the pipfile, or move them elsewhere, don't forget to commit those changes!

jxltom mentioned this issue Jun 1, 2018

Downgrade pipenv version to v11.9.0 #706

Closed

cuducos added a commit to InternetSemLimites/PublicAPI that referenced this issue Jun 13, 2018

Stop using Pipfile due to a bug in pipenv

08f158a

heroku/heroku-buildpack-python#704

cuducos added a commit to InternetSemLimites/PublicAPI that referenced this issue Jun 13, 2018

Stop using Pipfile due to a bug in pipenv

3661876

heroku/heroku-buildpack-python#704

harrywhite4 mentioned this issue Jul 6, 2018

Update pipenv version #727

Closed

sigmavirus24 added a commit that referenced this issue Jul 10, 2018

Allow users to override the version of pipenv used

9190900

This also starts tracking which Pipenv version users are selecting to use. Closes #702 Closes #704 Closes #706 Closes #727

sigmavirus24 mentioned this issue Jul 10, 2018

Upgrade pipenv and allow users to override the version installed #732

Closed

mcescalante mentioned this issue Jul 13, 2018

Remove requests-respectful from pipfile OpenHumans/oh-fitbit-integration#16

Merged

jxltom mentioned this issue Oct 5, 2018

Test https://github.com/heroku/heroku-buildpack-python/issues/704 jxltom/python-package-management-tests#1

Closed

umitanuki mentioned this issue Jan 8, 2019

ModuleNotFoundError: No module named 'numpy.core._multiarray_umath' alpacahq/pylivetrader#73

Closed

adamkowalczyk closed this as completed Feb 15, 2019

jerr0328 mentioned this issue May 24, 2019

Upgrade Pipenv to 2018.11.26 #820

Closed

paulmelnikow mentioned this issue Mar 10, 2020

Allow other pipenv versions than 2018.5.18 #906

Closed

abooodbah mentioned this issue Jul 9, 2020

issues after updating pipenv pypa/pipenv#4369

Closed

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Deploy fails due to outdated Pipfile.lock #704

Deploy fails due to outdated Pipfile.lock #704

adamkowalczyk commented May 29, 2018 •

edited

Loading

adamkowalczyk commented May 30, 2018 •

edited

Loading

jxltom commented Jun 1, 2018 •

edited

Loading

zyv commented Jun 30, 2018

jxltom commented Jun 30, 2018

edmorley commented Jun 30, 2018 •

edited

Loading

zyv commented Jun 30, 2018

edmorley commented Jun 30, 2018 •

edited

Loading

zyv commented Jun 30, 2018

gedankenstuecke commented Jul 13, 2018

CaseyFaist commented Oct 11, 2018

adamkowalczyk commented Oct 17, 2018

jxltom commented Oct 18, 2018 •

edited

Loading

adamkowalczyk commented Oct 18, 2018

jxltom commented Oct 18, 2018

MalikRumi commented Jun 29, 2019

Deploy fails due to outdated Pipfile.lock #704

Deploy fails due to outdated Pipfile.lock #704

Comments

adamkowalczyk commented May 29, 2018 • edited Loading

adamkowalczyk commented May 30, 2018 • edited Loading

jxltom commented Jun 1, 2018 • edited Loading

zyv commented Jun 30, 2018

jxltom commented Jun 30, 2018

edmorley commented Jun 30, 2018 • edited Loading

zyv commented Jun 30, 2018

edmorley commented Jun 30, 2018 • edited Loading

zyv commented Jun 30, 2018

gedankenstuecke commented Jul 13, 2018

CaseyFaist commented Oct 11, 2018

adamkowalczyk commented Oct 17, 2018

jxltom commented Oct 18, 2018 • edited Loading

adamkowalczyk commented Oct 18, 2018

jxltom commented Oct 18, 2018

MalikRumi commented Jun 29, 2019

adamkowalczyk commented May 29, 2018 •

edited

Loading

adamkowalczyk commented May 30, 2018 •

edited

Loading

jxltom commented Jun 1, 2018 •

edited

Loading

edmorley commented Jun 30, 2018 •

edited

Loading

edmorley commented Jun 30, 2018 •

edited

Loading

jxltom commented Oct 18, 2018 •

edited

Loading