virtualenv: use version 20 to support Python 3.9 #8873
Merged
Conversation
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
|
I'm new to homebrew dev, so sorry if I've not found a previous discussion. Is there a reason not to use the |
Good question. You're correct, the only Python versions we have currently are 3.7, 3.8, and 3.9. If I had to guess, the reason we aren't using it is that this code was introduced before the According to the
I don't think any of these are breaking points for us, though. Do any other maintainers have an opinion? |
5 tasks
|
This looks fine to me, but I do not know that Python-related code well enough to predict if it could have unexpected downsides. |
This was referenced Oct 7, 2020
MikeMcQuaid
approved these changes
Oct 7, 2020
|
This might have broken For example in the flake 8 formula, we expect this path to exist: But it does not get created. Only On Mac I do not see this issue. It might also be due to something else, I am still investigating. |
|
On Linux, the formula resources are getting installed outside of the virtualenv. It seems like on Linux, brewed Python is being executed rather than the virtualenv Python. macOSLinux |
|
We can observe the defective behavior by running the virtualenv pip and checking the value of macOS: % /usr/local/opt/flake8/libexec/bin/pip debug
WARNING: This command is only meant for debugging. Do not use this with automation for parsing and getting these details, since the output and options of this command may change without notice.
pip version: pip 20.2.3 from /usr/local/Cellar/flake8/3.8.4_1/libexec/lib/python3.9/site-packages/pip (python 3.9)
sys.version: 3.9.0 (default, Oct 6 2020, 04:17:54)
[Clang 12.0.0 (clang-1200.0.32.2)]
sys.executable: /usr/local/Cellar/flake8/3.8.4_1/libexec/bin/pythonLinux: # /home/linuxbrew/.linuxbrew/Cellar/flake8/3.8.4_1/libexec/bin/pip debug
WARNING: This command is only meant for debugging. Do not use this with automation for parsing and getting these details, since the output and options of this command may change without notice.
pip version: pip 20.2.3 from /home/linuxbrew/.linuxbrew/Cellar/flake8/3.8.4_1/libexec/lib/python3.9/site-packages/pip (python 3.9)
sys.version: 3.9.0 (default, Oct 7 2020, 09:11:45)
[GCC 5.4.0 20160609]
sys.executable: /home/linuxbrew/.linuxbrew/opt/[email protected]/bin/python3.9Applying this patch makes the install work correctly, but this is a horrible hack and I'd rather figure out why this is happening in the first place. diff --git a/Library/Homebrew/language/python.rb b/Library/Homebrew/language/python.rb
index b5523b4ed..e42100e2e 100644
--- a/Library/Homebrew/language/python.rb
+++ b/Library/Homebrew/language/python.rb
@@ -317,9 +317,11 @@ module Language
def do_install(targets)
targets = Array(targets)
- @formula.system @venv_root/"bin/pip", "install",
- "-v", "--no-deps", "--no-binary", ":all:",
- "--ignore-installed", *targets
+ with_env("PYTHONEXECUTABLE" => @venv_root/"bin/python") do
+ @formula.system @venv_root/"bin/pip", "install",
+ "-v", "--no-deps", "--no-binary", ":all:",
+ "--ignore-installed", *targets
+ end
end
end
end |
|
I don't have any experience with linux so it's hard for me to try to help. Is this broken for all formulae that use a virtualenv or just Python 3.9 formulae? |
|
Seems to be broken for python3.8 formulae as well. |
Shall we apply the horrible hack for now and then iterate on a better solution? Bonus points if we add a test to |
|
|
Here is the content of the sitecustomize.py file on Linux: Commenting out the 2 last lines does fix the issue, so that's a first hint in the direction we have to look. |
|
The difference is that: /home/linuxbrew/.linuxbrew/Cellar/[email protected]/3.9.0/bin/python3 reads sitecustomize.py On Linux the second case should not read the sitecustomize.py file |
|
There is a /home/linuxbrew/.linuxbrew/Cellar/flake8/3.8.4_1/libexec/lib/python3.9/site-packages/homebrew_deps.pth file which brings in the main site-package on Linux. This file does not exist on Mac. |
7 tasks
iMichka
added a commit
to iMichka/brew
that referenced
this pull request
Oct 8, 2020
If added, this makes the virtualenv read the main site-package from brewed Python, and especially makes it read our sitecustomize.py file, which will modify the sys.executable path. See the full discussion at: Homebrew#8873 I also took the opportunity to not include test deps, as these will be not be installed, so the .pth file should not contains references to site-packages from test deps. Previous packages on Linux did already contain the wrong lines in the pth file, for example: cat /home/linuxbrew/.linuxbrew/Cellar/aws-google-auth/0.0.36_1/libexec/lib/python3.8/site-packages/homebrew_deps.pth import site; site.addsitedir('/home/linuxbrew/.linuxbrew/opt/[email protected]/lib/python3.8/site-packages') import site; site.addsitedir('/home/linuxbrew/.linuxbrew/opt/libxml2/lib/python3.8/site-packages') This might have caused subtle bugs for some packages but not for others.
|
I have a patch for this: #8880 |
Sign up for free
to subscribe to this conversation on GitHub.
Already have an account?
Sign in.
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
brew stylewith your changes locally?brew testswith your changes locally?brew manlocally and committed any changes?See Homebrew/homebrew-core#62201 (comment) and Homebrew/discussions#31
Update the
homebrew-virtualenvresource to version 20.0.33 to support virtual environments using Python 3.9. This was attempted previously in #7941.I tested this locally and was able to build a formula using a Python 3.9 formula locally without any problems. I'm not yet sure what the consequences are to adding the four new
homebrew-resources.