Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Errors during the install of pip/setuptools/wheel don't cause the build to fail #1002

Closed
edmorley opened this issue Jul 22, 2020 · 0 comments · Fixed by #1007
Closed

Errors during the install of pip/setuptools/wheel don't cause the build to fail #1002

edmorley opened this issue Jul 22, 2020 · 0 comments · Fixed by #1007
Assignees
@edmorley
Labels
bug

Comments

@edmorley
Copy link
Member

Whilst much of the buildpack is run with bash's exit on error enabled, the python setup step disables that mode temporarily using set +e:

heroku-buildpack-python/bin/steps/python

Line 3 in 156b07c

set +e

Most of the logic in that file is written to take this into account, with the exception of the pip/setuptools/wheel install here:

heroku-buildpack-python/bin/steps/python

Lines 159 to 160 in 156b07c

/app/.heroku/python/bin/python "$GETPIP_PY" pip=="$PIP_UPDATE" &> /dev/null
/app/.heroku/python/bin/pip install "$ROOT_DIR/vendor/setuptools-39.0.1-py2.py3-none-any.whl" &> /dev/null

If either of those commands fail (eg PyPI down, transient network issue, pip/buildpack bug), then the build continues regardless, leaving the Python install without a copy of pip/setuptools/wheel - which will cause confusing failures in later buildpack steps.

To compound the issue, the stderr from those commands is sent to /dev/null, making it seem like the step succeeded / obscuring the reason for the failed install.

We should:

  1. Re-enable exit-on-error mode for these steps
  2. Either redirect only stdout to /dev/null, or remove all redirection and use the --quiet options amongst others to suppress non-error output
@edmorley edmorley self-assigned this Jul 22, 2020
@edmorley edmorley mentioned this issue Jul 23, 2020
Merged
edmorley added a commit that referenced this issue Jul 23, 2020
@edmorley
Update pip/setuptools/wheel and refactor installation
776a1f9 
The versions installed by the buildpack have been updated as follows:
* pip:
  - If using Python 3.4: No change (already using the last to support 3.4)
  - If using pipenv: No change (need to update to a newer pipenv first)
  - For everything else: `20.0.2` -> `20.1.1`
* setuptools:
  - If using Python 3.4: `39.0.1` -> `43.0.0` (latest for 3.4)
  - If using Python 2.7: `39.0.1` -> `44.1.1` (latest for 2.7)
  - For everything else: `39.0.1` -> `47.1.1` (until #1006 fixed)
* wheel:
  - If using Python 3.4: `unpinned` -> `0.33.6`
  - For everything else: `unpinned` -> `0.34.2`

This fixes #949 and fixes #1005, and means packages that rely on newer
setuptools will now install successfully.

Changelogs:
https://pip.pypa.io/en/stable/news/
https://setuptools.readthedocs.io/en/latest/history.html#v47-1-1
https://wheel.readthedocs.io/en/latest/news.html

In addition:
* Installed versions are now deterministic (fixes #1000, fixes #1003)
* The build output now includes the versions used, making it easier to
  debug future upgrades (closes #939)
* Errors during pip/setuptools/wheel install now correctly fail the
  build, and stderr is no longer sent to `/dev/null` (fixes #1002)
* Setuptools is no longer installed twice (fixes #1001)
* Everything that is downloaded is now used (fixes #999)
* `--no-cache` and `--disable-version-check` are now used, saving
  unnecessary work and preventing creation of unwanted files in `/app`
* The `PIP_UPDATE` env var no longer leaks into subprocesses.

As part of fixing version pinning, we now use pip itself to determine
whether the installed packages are up to date, since parsing pip's
output is fragile (eg #1003).

This means `pip install` is now called every time, however this is a
no-op for repeat builds where the versions have not changed, since
unless `--upgrade` is specified pip does not hit the index (PyPI) if
requirements are satisfied.

For the installation itself `get-pip.py` is no longer used, since:
- It uses `--force-reinstall`, which is unnecessary here and would slow
  down repeat builds (given we call pip install every time now).
  Trying to work around this by using `get-pip.py` only for the initial
  install, and real pip for subsequent updates would mean we lose
  protection against cached broken installs, plus significantly
  increase the version combinations test matrix.
- It means downloading pip twice (once embedded in `get-pip.py`, and
  again during the install, since `get-pip.py` can't install the
  embedded version directly)
- We would still have to manage several versions of get-pip.py, to
  support older Pythons.

We don't use `ensurepip` since:
- Not all of the previously generated Python runtimes on S3 include it
- We would still have to upgrade pip afterwards
- The versions of pip/setuptools bundled with ensurepip differ greatly
  depending on Python version, and we could easily start using a CLI
  flag for the first pip install before upgrade that isn't supported
  on all versions, without even knowing it (unless we test against
  hundreds of Python archives).

The new pip wheel assets on S3 were generated using:

```
$ pip download --no-cache pip==19.1.1
Collecting pip==19.1.1
  Downloading pip-19.1.1-py2.py3-none-any.whl (1.4 MB)
  Saved ./pip-19.1.1-py2.py3-none-any.whl
Successfully downloaded pip

$ pip download --no-cache pip==20.1.1
Collecting pip==20.1.1
  Downloading pip-20.1.1-py2.py3-none-any.whl (1.5 MB)
  Saved ./pip-20.1.1-py2.py3-none-any.whl
Successfully downloaded pip

$ aws s3 sync . s3://lang-python/common/ --exclude "*" --include "*.whl" --acl public-read --dryrun
(dryrun) upload: ./pip-19.1.1-py2.py3-none-any.whl to s3://lang-python/common/pip-19.1.1-py2.py3-none-any.whl
(dryrun) upload: ./pip-20.1.1-py2.py3-none-any.whl to s3://lang-python/common/pip-20.1.1-py2.py3-none-any.whl

$ aws s3 sync . s3://lang-python/common/ --exclude "*" --include "*.whl" --acl public-read
upload: ./pip-19.1.1-py2.py3-none-any.whl to s3://lang-python/common/pip-19.1.1-py2.py3-none-any.whl
upload: ./pip-20.1.1-py2.py3-none-any.whl to s3://lang-python/common/pip-20.1.1-py2.py3-none-any.whl
```
edmorley added a commit that referenced this issue Jul 29, 2020
@edmorley
Correctly handle failed pip/setuptools/wheel installs (#1007)
eb37945 
They are now displayed in the build output (instead of being sent to
`/dev/null`) and fail the build early instead of failing later in
`bin/steps/pip-install`.

Fixes #1002.
edmorley added a commit that referenced this issue Jul 29, 2020
@edmorley
Correctly handle failed pip/setuptools/wheel installs (#1007)
00e70ff 
They are now displayed in the build output (instead of being sent to
`/dev/null`) and fail the build early instead of failing later in
`bin/steps/pip-install`.

Fixes #1002.
dryan pushed a commit to dryan/heroku-buildpack-python that referenced this issue Nov 19, 2020
@edmorley @dryan
Correctly handle failed pip/setuptools/wheel installs (heroku#1007)
a398457 
They are now displayed in the build output (instead of being sent to
`/dev/null`) and fail the build early instead of failing later in
`bin/steps/pip-install`.

Fixes heroku#1002.
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees

@edmorley edmorley

Labels
bug
Projects
None yet
Milestone
No milestone
Development

Successfully merging a pull request may close this issue.

Improve pip/setuptools/wheel installation heroku/heroku-buildpack-python
1 participant
@edmorley