Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Python 3.9 mass migration #62560

Closed
wants to merge 165 commits into from
Closed

Python 3.9 mass migration #62560

wants to merge 165 commits into from

Conversation

fxcoudert
Copy link
Member

Leaving out all independent formulas (see #62201) is it doable to migrate all the rest in one huge PR?

@fxcoudert
Copy link
Member Author

fxcoudert commented Oct 11, 2020

So, what did we learn from the first round? It ran for 53 hours, and built 35 bottles. But, most of that time is spent in a couple of formulas: rust takes 3 hours to build, and 6 more hours to test all its dependents; node is a couple of hours; llvm also. Also, it tried many times to rebuild libxml2 from source, and kept failing.

Failures

libxml2 fails to build, upstream has a patch (https://gitlab.gnome.org/GNOME/libxml2/-/issues/149):

libxml.c:297:15: error: expected '(' after 'if'
    } else if PyUnicode_Check (ret) {
              ^

scipy failure reported there: scipy/scipy#12935
this commit should fix it: scipy/scipy@b8e4706

scipy fails with many errors:

scipy/sparse/linalg/dsolve/SuperLU/SRC/clacon2.c:175:5: error: implicit declaration of function 'ccopy_' is invalid in C99 [-Werror,-Wimplicit-function-declaration]
scipy/sparse/linalg/dsolve/SuperLU/SRC/dmach.c:62:9: error: implicitly declaring library function 'strncmp' with type 'int (const char *, const char *, unsigned long)' [-Werror,-Wimplicit-function-declaration]
scipy/sparse/linalg/dsolve/SuperLU/SRC/ilu_cdrop_row.c:196:3: error: implicit declaration of function 'scopy_' is invalid in C99 [-Werror,-Wimplicit-function-declaration]
scipy/sparse/linalg/dsolve/SuperLU/SRC/ilu_zdrop_row.c:196:3: error: implicit declaration of function 'dcopy_' is invalid in C99 [-Werror,-Wimplicit-function-declaration]
scipy/sparse/linalg/dsolve/SuperLU/SRC/smach.c:62:9: error: implicitly declaring library function 'strncmp' with type 'int (const char *, const char *, unsigned long)' [-Werror,-Wimplicit-function-declaration]
scipy/sparse/linalg/dsolve/SuperLU/SRC/zlacon2.c:175:5: error: implicit declaration of function 'zcopy_' is invalid in C99 [-Werror,-Wimplicit-function-declaration]
scipy/sparse/linalg/dsolve/SuperLU/SRC/slacon2.c:160:16: error: implicit declaration of function 'idamax_' is invalid in C99 [-Werror,-Wimplicit-function-declaration]

uhd failure due to Boost >= 1.73: EttusResearch/uhd#347
Trying to patch here: #62726


apache-arrow fails to build with:

2020-10-10T19:57:41.2826500Z CMake Error at cmake_modules/Findre2.cmake:65 (add_library):
2020-10-10T19:57:41.2827150Z   add_library cannot create imported target "RE2::re2" because another target
2020-10-10T19:57:41.2827850Z   with the same name already exists.
2020-10-10T19:57:41.2828250Z Call Stack (most recent call first):
2020-10-10T19:57:41.2829030Z   /usr/local/lib/cmake/grpc/gRPCConfig.cmake:21 (find_package)
2020-10-10T19:57:41.2829760Z   cmake_modules/ThirdpartyToolchain.cmake:2453 (find_package)
2020-10-10T19:57:41.2830370Z   CMakeLists.txt:484 (include)

filebeat fails to build:

2020-10-11T10:57:09.7355860Z     In file included from src/decode.c:868:
2020-10-11T10:57:09.7357300Z     /private/tmp/pip-install-mabhh0iv/pillow/src/libImaging/Jpeg2K.h:11:10: fatal error: 'openjpeg.h' file not found
2020-10-11T10:57:09.7358200Z     #include <openjpeg.h>
2020-10-11T10:57:09.7358520Z              ^~~~~~~~~~~~
2020-10-11T10:57:09.7358790Z     1 error generated.
2020-10-11T10:57:09.7360810Z     error: command '/usr/local/Homebrew/Library/Homebrew/shims/mac/super/clang' failed with exit code 1
2020-10-11T10:57:09.7362610Z     ----------------------------------------
2020-10-11T10:57:09.7367680Z ERROR: Command errored out with exit status 1: /private/tmp/filebeat-20201011-14100-1d89or3/src/github.com/elastic/beats/filebeat/build/python-env/bin/python3 -u -c 'import sys, setuptools, tokenize; sys.argv[0] = '"'"'/private/tmp/pip-install-mabhh0iv/pillow/setup.py'"'"'; __file__='"'"'/private/tmp/pip-install-mabhh0iv/pillow/setup.py'"'"';f=getattr(tokenize, '"'"'open'"'"', open)(__file__);code=f.read().replace('"'"'\r\n'"'"', '"'"'\n'"'"');f.close();exec(compile(code, __file__, '"'"'exec'"'"'))' install --record /private/tmp/pip-record-mpr08bmy/install-record.txt --single-version-externally-managed --compile --install-headers /private/tmp/filebeat-20201011-14100-1d89or3/src/github.com/elastic/beats/filebeat/build/python-env/include/site/python3.9/Pillow Check the logs for full command output.

fail2ban fails to build:

2020-10-11T10:43:33.8332690Z �[34m==>�[0m �[1mmake dirhtml SPHINXBUILD=sphinx-build�[0m
2020-10-11T10:43:33.8334620Z sphinx-build -b dirhtml -d ""build/doctrees   . ""build/dirhtml
2020-10-11T10:43:33.8335480Z Running Sphinx v3.2.1
2020-10-11T10:43:33.8335900Z 
2020-10-11T10:43:33.8336840Z Extension error:
2020-10-11T10:43:33.8338670Z Could not import extension numpydoc (exception: No module named 'numpydoc')
2020-10-11T10:43:33.8339620Z make: *** [dirhtml] Error 2

fdroidserver fails to build:

2020-10-11T10:54:33.2724310Z     RuntimeError: Polyfit sanity test emitted a warning, most likely due to using a buggy Accelerate backend. If you compiled yourself, see site.cfg.example for information. Otherwise report this to the vendor that provided NumPy.

brew linkage --test gst-editing-services
==>�FAILED
Broken dependencies:
  /usr/local/opt/[email protected]/Frameworks/Python.framework/Versions/3.8/Python ([email protected])
No broken library linkage detected
==>�Testing gst-editing-services
/usr/bin/sandbox-exec -f /private/tmp/homebrew20201009-83220-15xzthx.sb ruby -W0 -I $LOAD_PATH -- /usr/local/Homebrew/Library/Homebrew/test.rb /usr/local/Homebrew/Library/Taps/homebrew/homebrew-core/Formula/gst-editing-services.rb --verbose --retry
/usr/local/Cellar/gst-editing-services/1.18.0/bin/ges-launch-1.0 --ges-version
dyld: Library not loaded: /usr/local/opt/[email protected]/Frameworks/Python.framework/Versions/3.8/Python
  Referenced from: /usr/local/Cellar/gst-editing-services/1.18.0/lib/libges-1.0.0.dylib
  Reason: image not found
brew linkage --test gst-plugins-bad
==>�FAILED
Broken dependencies:
  /usr/local/opt/srt/lib/libsrt.1.dylib (srt)
brew install --build-from-source findomain
==>�FAILED
Downloading https://github.com/Edu4rdSHL/findomain/archive/2.1.4.tar.gz
Downloading from https://codeload.github.com/Findomain/Findomain/tar.gz/2.1.4
Error: SHA256 mismatch
Expected: e669e5297c3112ec4e256bef286028220dcb9d46cd8a31e6965cbbc3d2ae3199
  Actual: dd193f3b2d81b05769efbf2a2c7af7479395617204343c84fb5d7e1b0bec9d83
brew install --build-from-source mdbook
==>�FAILED
Downloading https://github.com/rust-lang/mdBook/archive/v0.4.3.tar.gz
Downloading from https://codeload.github.com/rust-lang/mdBook/tar.gz/v0.4.3
Error: SHA256 mismatch
Expected: e1a60eb877a95c6cf8832cb158a8f09b2d641206fe853378905bcdb962902935
  Actual: 3a3c4da76ff47943dbd5d490567a3d283cf85889f16d4d96ae04328371c1a817
brew install --build-from-source spotifyd
==>�FAILED
error: failed to run custom build command for `coreaudio-sys v0.2.3`
brew audit llvm --online --git --skip-style
==>�FAILED
Error: 1 problem in 1 formula detected
llvm:
  * Packages have been installed for:
      Python 3.8
    but this formula depends on:
      Python 3.9
brew audit vapoursynth --online --git --skip-style
==>�FAILED
Error: 1 problem in 1 formula detected
vapoursynth:
  * python modules have explicit framework links
    These python extension modules were linked directly to a Python
    framework binary. They should be linked with -undefined dynamic_lookup
    instead of -lpython or -framework Python.
      /usr/local/opt/vapoursynth/lib/python3.9/site-packages/vapoursynth.so

@fxcoudert
Copy link
Member Author

fxcoudert commented Oct 14, 2020

10.13 tests are over in 37 hours, and the results are not so bad!

brew install --verbose --build-bottle apache-arrow
brew install --verbose --build-bottle afflib
brew install --verbose --build-bottle borgbackup
brew install --build-from-source apache-arrow
brew install --only-dependencies --verbose --build-bottle dvc
brew install --verbose --build-bottle dvc
brew install --verbose --build-bottle fail2ban
brew install --verbose --build-bottle fdroidserver
brew install --verbose --build-bottle filebeat
brew install --verbose --build-bottle lc0
brew install --verbose --build-bottle mypy
brew install --verbose --build-bottle ola
brew install --verbose --build-bottle onnxruntime
brew install --verbose --build-bottle salt
brew test --retry --verbose bowtie2
brew test --retry --verbose gspell
brew test --verbose dnsviz
brew test --verbose eralchemy
brew test --verbose gdb
brew test --verbose gdbgui
brew test --verbose gnuradio
brew test --verbose lensfun
brew test --verbose mitmproxy
brew test --verbose zim
brew audit cryptominisat --onlinea --git --skip-style
brew audit gtk-doc --online --git --skip-style
brew audit libpqxx --online --git --skip-style
brew audit tbb --online --git --skip-style
brew audit uhd --online --git --skip-style
brew audit vapoursynth --online --git --skip-style

@fxcoudert
Copy link
Member Author

fxcoudert commented Oct 14, 2020

Build failures

Test failures

  • bowtie says env: python3: No such file or directory
  • gspell is timing out
  • dnsviz has:
 /usr/local/Cellar/dnsviz/0.8.2_4/libexec/lib/python3.9/site-packages/M2Crypto/X509.py:44: SyntaxWarning: "is not" with a literal. Did you mean "!="?
   value.strip('0123456789abcdefABCDEF:') is not '':
 pygraphviz is required, but not installed.
 An exception occurred within a child process:
   BuildError: Failed executing: /usr/local/Cellar/dnsviz/0.8.2_4/bin/dnsviz graph -r example.com.json -Thtml -o /dev/null
  • eralchemy has a linking issue:
 ImportError: dlopen(/usr/local/Cellar/eralchemy/1.2.10_4/libexec/lib/python3.9/site-packages/pygraphviz/_graphviz.cpython-39-darwin.so, 2): Symbol not found: _PyIOBase_Type
   Referenced from: /usr/local/Cellar/eralchemy/1.2.10_4/libexec/lib/python3.9/site-packages/pygraphviz/_graphviz.cpython-39-darwin.so
   Expected in: flat namespace
  in /usr/local/Cellar/eralchemy/1.2.10_4/libexec/lib/python3.9/site-packages/pygraphviz/_graphviz.cpython-39-darwin.so
  • gnuradio has a weird failure:
 Failed to execute: python3
 An exception occurred within a child process:
   BuildError: Failed executing: python3 /private/tmp/gnuradio-test-20201013-8925-1mnc7ia/test.py

Audit failures

 uhd:
   * python modules have explicit framework links
     These python extension modules were linked directly to a Python
     framework binary. They should be linked with -undefined dynamic_lookup
     instead of -lpython or -framework Python.
       /usr/local/opt/uhd/lib/python3.9/site-packages/uhd/libpyuhd.so
 vapoursynth:
   * python modules have explicit framework links
     These python extension modules were linked directly to a Python
     framework binary. They should be linked with -undefined dynamic_lookup
     instead of -lpython or -framework Python.
       /usr/local/opt/vapoursynth/lib/python3.9/site-packages/vapoursynth.so
   * Files were found with references to the Homebrew shims directory.
     The offending files are:
       lib/libcryptominisat5.5.6.dylib
   * Files were found with references to the Homebrew shims directory.
     The offending files are:
       libexec/bin/gtkdoc-depscan
       share/gtk-doc/python/gtkdoc/config.py

Expired certificates:

@fxcoudert
Copy link
Member Author

The most time-consuming formulas remaining in the PR are: glib gobject-introspection pyqt protobuf bazel libtorch [email protected] gdal vtk ansible emscripten gnuradio kibana libtensorflow libtensorflow@1 mame nest nwchem opencv opencv@3 semgrep wxpython

Moving some of those away from this PR…

@fxcoudert
Copy link
Member Author

Great. New style audits blocking all testing.

Formula/libxml2.rb:60:10: C: GitLab patches should end with .patch, not .diff:
  https://gitlab.gnome.org/nwellnhof/libxml2/-/commit/e4fb36841800038c289997432ca547c9bfef9db1.
    url "https://gitlab.gnome.org/nwellnhof/libxml2/-/commit/e4fb36841800038c289997432ca547c9bfef9db1.diff"
         ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
Formula/mitmproxy.rb:197:10: C: GitHub patches should end with .patch, not .diff:
  https://github.com/mitmproxy/mitmproxy/commit/8e5e43de.diff?full_index=1
    url "https://github.com/mitmproxy/mitmproxy/commit/8e5e43de.diff?full_index=1"
         ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^

Be right back 😭

@dtrodrigues
Copy link
Member

redex failure due to new boost:

libredex/Show.cpp:12:10: fatal error: 'boost/io/detail/quoted_manip.hpp' file not found
#include <boost/io/detail/quoted_manip.hpp>
         ^~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~

PR in #63266

@BrewTestBot
Copy link
Member

:shipit: @fxcoudert has triggered a merge.

@fxcoudert
Copy link
Member Author

Thanks @dtrodrigues
I'm going to merge as-is, and we can deal with the test failures and redex build failure in a second time.

@fxcoudert
Copy link
Member Author

10.15

brew audit vapoursynth --online --git --skip-style
brew audit gtk-doc --online --git --skip-style
brew audit uhd --online --git --skip-style
brew audit cryptominisat --online --git --skip-style
brew audit csound --online --git --skip-style
brew audit cxxtest --online --git --skip-style
brew audit dnsviz --online --git --skip-style
brew test --verbose gnuradio
brew audit mitmproxy --online --git --skip-style
brew audit mkvtomp4 --online --git --skip-style
brew install --verbose --build-bottle redex

10.14

brew audit vapoursynth --online --git --skip-style
brew audit gtk-doc --online --git --skip-style
brew audit uhd --online --git --skip-style
brew audit anime-downloader --online --git --skip-style
brew audit cmark-gfm --online --git --skip-style
brew audit cmark --online --git --skip-style
brew audit code-server --online --git --skip-style
brew audit cryptominisat --online --git --skip-style
brew audit csound --online --git --skip-style
brew audit gnuradio --online --git --skip-style
brew test --verbose gnuradio
brew audit gprof2dot --online --git --skip-style
brew audit rdiff-backup --online --git --skip-style
brew install --verbose --build-bottle redex
brew audit redex --online --git --skip-style

10.13

brew audit vapoursynth --online --git --skip-style
brew audit gtk-doc --online --git --skip-style
brew audit uhd --online --git --skip-style
brew audit cryptominisat --online --git --skip-style
brew audit genometools --online --git --skip-style
brew audit git-cola --online --git --skip-style
brew audit gitfs --online --git --skip-style
brew audit gitless --online --git --skip-style
brew test --verbose gnuradio
brew audit libproxy --online --git --skip-style
brew fetch --retry pytouhou --build-bottle --force
brew install --verbose --build-bottle pytouhou
brew install --verbose --build-bottle redex

@BrewTestBot
Copy link
Member

⚠️ @fxcoudert bottle publish failed.

@BrewTestBot
Copy link
Member

:shipit: @fxcoudert has triggered a merge.

@BrewTestBot
Copy link
Member

⚠️ @fxcoudert bottle publish failed.

@fxcoudert
Copy link
Member Author

==> Checking remote file homebrew/bottles/gnuradio-3.8.2.0_3.catalina.bottle.tar.gz
/usr/bin/curl --disable --globoff --show-error --user-agent Linuxbrew/2.5.6-112-g0d56c47\ \(Linux\;\ x86_64\ Ubuntu\ 18.04.5\ LTS\)\ curl/7.58.0 --header Accept-Language:\ en --retry 3 --fail --silent --head https://dl.bintray.com/homebrew/bottles/gnuradio-3.8.2.0_3.catalina.bottle.tar.gz
/usr/bin/curl --disable --globoff --show-error --user-agent Linuxbrew/2.5.6-112-g0d56c47\ \(Linux\;\ x86_64\ Ubuntu\ 18.04.5\ LTS\)\ curl/7.58.0 --header Accept-Language:\ en --fail --progress-bar --silent --retry 3 --fail --silent --output /dev/null https://api.bintray.com/packages/homebrew/bottles/gnuradio
==> Uploading homebrew/bottles/gnuradio/3.8.2.0_3/gnuradio-3.8.2.0_3.catalina.bottle.tar.gz
/usr/bin/curl --disable --globoff --show-error --user-agent Linuxbrew/2.5.6-112-g0d56c47\ \(Linux\;\ x86_64\ Ubuntu\ 18.04.5\ LTS\)\ curl/7.58.0 --header Accept-Language:\ en --fail --progress-bar --silent --retry 3 --fail --upload-file gnuradio--3.8.2.0_3.catalina.bottle.tar.gz --header X-Checksum-Sha2:\ 17573d65b40db37b8354c9ac75b59df89f91630fe89d154c487d50f8b841fb7d --user brewtestbot:****** https://api.bintray.com/content/homebrew/bottles/gnuradio/3.8.2.0_3/gnuradio-3.8.2.0_3.catalina.bottle.tar.gz
curl: Can't open 'gnuradio--3.8.2.0_3.catalina.bottle.tar.gz'!
curl: try 'curl --help' or 'curl --manual' for more information
Error: Failure while executing; `/usr/bin/curl --disable --globoff --show-error --user-agent Linuxbrew/2.5.6-112-g0d56c47\ \(Linux\;\ x86_64\ Ubuntu\ 18.04.5\ LTS\)\ curl/7.58.0 --header Accept-Language:\ en --fail --progress-bar --silent --retry 3 --fail --upload-file gnuradio--3.8.2.0_3.catalina.bottle.tar.gz --header X-Checksum-Sha2:\ 17573d65b40db37b8354c9ac75b59df89f91630fe89d154c487d50f8b841fb7d --user brewtestbot:****** https://api.bintray.com/content/homebrew/bottles/gnuradio/3.8.2.0_3/gnuradio-3.8.2.0_3.catalina.bottle.tar.gz` exited with 26. Here's the output:
curl: Can't open 'gnuradio--3.8.2.0_3.catalina.bottle.tar.gz'!
curl: try 'curl --help' or 'curl --manual' for more information

I have no idea why this happens…

@jonchang
Copy link
Contributor

GitHub Actions is having an outage.

@BrewTestBot
Copy link
Member

:shipit: @fxcoudert has triggered a merge.

@BrewTestBot
Copy link
Member

⚠️ @fxcoudert bottle publish failed.

@fxcoudert fxcoudert mentioned this pull request Oct 22, 2020
5 tasks
@BrewTestBot
Copy link
Member

:shipit: @jonchang has triggered a merge.

@BrewTestBot
Copy link
Member

⚠️ @jonchang bottle publish failed.

This was referenced Oct 22, 2020
@jonchang jonchang closed this in d6edb2a Oct 23, 2020
@jonchang
Copy link
Contributor

Couldn't figure out how to fix it in time so just did a manual merge overnight. 🥳

@fxcoudert fxcoudert deleted the py39 branch October 23, 2020 08:24
@fxcoudert
Copy link
Member Author

Many thanks @jonchang !

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
python Python use is a significant feature of the PR or issue python-3.9-migration
Projects
None yet
Development

Successfully merging this pull request may close these issues.

5 participants