Tests: Add additional coverage of editable package installation #1251
Conversation
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Previously the editable installation mode test only checked whether the build completed successfully, not whether the installed editable packages worked. Now the packages are tested via their entrypoints in the following scenarios: 1. During the `bin/post_compile` hook 2. By a later buildpack in the build 3. At runtime In particular, (2) catches the issue described in #1006 when setuptools is upgraded, and (1) + (3) will be useful to verify the solution to #1006 hasn't broken the other cases. GUS-W-10047026. [skip-changelog]
joshwlewis
approved these changes
Oct 18, 2021
edmorley
added a commit
that referenced
this pull request
Oct 19, 2021
Currently the build system performs builds in a different directory (`/tmp/build_<hash>`) to which the app will be run at runtime (`/app`). This means that any hardcoded paths in the slug must be rewritten by the buildpack, so the app still works after being moved. One such case of hardcoded paths, are the `.pth` and `.egg-link` files that are created in the `site-packages` directory by Pip/setuptools for editable package installs (aka develop mode). The most common way someone might use editable mode is via `-e <package specifier>` entries in their `requirements.txt` file. Until now, the Python buildpack rewrote paths inside these files during the compile itself, which meant the build-time paths were no longer present when subsequent buildpacks ran. This happened to work due to an interaction of legacy setuptools behaviour and a buildpack bug, but stops working in setuptools 47.2.0 or later - as described in #1006. Longer term we would like to stop building in one location and running the app from another, so that the path rewriting isn't required at all. However that change breaks some other buildpacks so requires a long-term transition plan. In the meantime, this change moves path rewriting to a `.profile.d/` script, so that it occurs at runtime, and so after all other buildpacks have run. Additional test coverage of editable packages was added previously in #1251 and #1253, and has confirmed that this new `profile.d/` script approach will prevent the issues in #1006 when setuptools is updated in a future PR. There is one subtle implication of moving this path rewriting, in that subsequent cached builds will no longer see the existing package as being already installed, so won't uninstall if before reinstalling it (as seen in the test log output change). However this is not believed to have any significant impact. Fixes #1006. (And so unblocks updating to the newer setuptools required for #1248.)
edmorley
added a commit
that referenced
this pull request
Oct 19, 2021
…#1252) Currently the build system performs builds in a different directory (`/tmp/build_<hash>`) to which the app will be run at runtime (`/app`). This means that any hardcoded paths in the slug must be rewritten by the buildpack, so the app still works after being moved. One such case of hardcoded paths, are the `.pth` and `.egg-link` files that are created in the `site-packages` directory by Pip/setuptools for editable package installs (aka develop mode). The most common way someone might use editable mode is via `-e <package specifier>` entries in their `requirements.txt` file. Until now, the Python buildpack rewrote paths inside these files during the compile itself, which meant the build-time paths were no longer present when subsequent buildpacks ran. This happened to work due to an interaction of legacy setuptools behaviour and a buildpack bug, but stops working in setuptools 47.2.0 or later - as described in #1006. Longer term we would like to stop building in one location and running the app from another, so that the path rewriting isn't required at all. However that change breaks some other buildpacks so requires a long-term transition plan. In the meantime, this change moves path rewriting to a `.profile.d/` script, so that it occurs at runtime, and so after all other buildpacks have run. Additional test coverage of editable packages was added previously in #1251 and #1253, and has confirmed that this new `profile.d/` script approach will prevent the issues in #1006 when setuptools is updated in a future PR. There is one subtle implication of moving this path rewriting, in that subsequent cached builds will no longer see the existing package as being already installed, so won't uninstall if before reinstalling it (as seen in the test log output change). However this is not believed to have any significant impact. Fixes #1006. (And so unblocks updating to the newer setuptools required for #1248.) GUS-W-7828034.
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
Previously the editable installation mode test only checked whether the build completed successfully, not whether the installed editable packages worked.
Now the packages are tested via their entrypoints in the following scenarios:
bin/post_compilehookIn particular, (2) catches the issue described in #1006 when setuptools is upgraded (example failure), and (1) + (3) will be useful to verify the solution to #1006 hasn't broken the other cases.
GUS-W-10047026.
[skip changelog]