New methods of building/installing PyThemis #1023
Merged
Conversation
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Add command that creates a .whl package for Python, current modern format to be installed into virtual environmants
Lagovas
reviewed
Sep 25, 2023
Update script for make target pythemis_make_wheel, remove venv path from PATH if detected, so that building wheel always works properly. Add make target pythemis_install_wheel to install created wheel into currently active venv. Update CI config to 1) Use Ubuntu 20.04 (since the're some issues with themispp tests) 2) Use new pythemis_install_wheel command instead of hardcoded filename
Rename pythemis_install -> pythemis_old_install here as well
vixentael
reviewed
Sep 25, 2023
Makefile
Outdated
|
|
||
| # pythemis_make_os_pkg: CMD = TODO | ||
| # pythemis_make_os_pkg: | ||
| # TODO |
Use wheel content to create the .deb package. Add CI job to test with this specific installation method.
Lagovas
reviewed
Sep 27, 2023
iamnotacake
changed the title
Lagovas
reviewed
Oct 2, 2023
* Rename pythemis_deb -> deb_python * Remove some comments, TODOs * Add libthemis to PyThemis deb deps
Since python3-pythemis now depends on libthemis, we need to install the dependency before installing Python wrapper. Updated Makefile to conditionally include C++ and Java wrappers in deb target so that one could build only Themis itself on systems without Java (jni.h is required to to build Java wrapper). This may not be the best way, but it uses the least amount of changes. Potential improvements: * Split deb target into deb_themis, deb_themispp, deb_java (will need to copy-paste some vars as well, and to the same to rpm for consistency) * Add new target like deb_themis_install and others, just like did it with pythemis_install_deb
Lagovas
reviewed
Oct 10, 2023
| @@ -607,6 +609,25 @@ endif | |||
| @echo -n "pythemis install " | |||
| @$(BUILD_CMD_) | |||
|
|
|||
| pythemis_make_wheel: CMD = cd src/wrappers/themis/python/ && python3 setup.py bdist_wheel | |||
There was a problem hiding this comment.
last comment. can we clean generated artifacts with clean target? this command creates new folder with source codes and wheel. or if we can specify destination folder, let's use $(BIN_PATH). It's already mentioned in the clean target
|
lgtm, it's ready to merge. but let's make all tests green and after that merge. |
Last version that still works with current MSRV 1.58. New ones want 1.60 or newer.
Same reason as with 'log', to avoid increasing MSRV when only tests depend on this.
|
looks like we need update rust dependencies again. let's up the version to 1.60. looks like a lot of version upgraded their dependencies |
Lagovas
approved these changes
Nov 28, 2023
vixentael
approved these changes
Nov 28, 2023
G1gg1L3s
added a commit
to G1gg1L3s/themis
that referenced
this pull request
Dec 4, 2023
commit 784033b Author: Anatolii Lishchynskyi <[email protected]> Date: Tue Nov 28 21:45:12 2023 +0200 New methods of building/installing PyThemis (cossacklabs#1023) Refactor Makefile: * new target pythemis_make_wheel to create a .whl Python package, current modern format to be installed into virtual environmants * new target pythemis_install_wheel to install it in currently active virtualenv * new target deb_python that builds a .deb package for system-wide installation of PyThemis * new target pythemis_install_deb, alias for pythemis_deb + apt install of the created pkg * new target rpm_python, similar to deb_python * new target pythemis_install_rpm, pythemis_install_deb Update GitHub Actions workflow, test .whl and .deb installation See cossacklabs/product-docs/pull/317 for related docs update commit 30578c8 Author: Anatolii Lishchynskyi <[email protected]> Date: Mon Nov 27 17:53:02 2023 +0200 Fix clippy and fmt issues, update MSRV (cossacklabs#1039) Fix clippy and fmt issues Update MSRV to 1.60 Freeze test deps so they compile on Rust 1.60, with no effect on themis itself (does not use those frozen crates) commit 6111766 Author: Anatolii Lishchynskyi <[email protected]> Date: Tue Nov 21 17:08:51 2023 +0200 Update emscripten requirements and WASM CI job (cossacklabs#1036) * Bump emsdk version to 3.1.47 Produces module importable in Node v18 With older (emsdk 3.0.0) version, generated `libthemis.js` that should load `libthemis.wasm` fails due to some internal autogenerated code working differently on v16 and v18 * Add link flag for WASM builds Needed to make new emscripten produce working module * Update Node testing versions Add v18 that should now work, also add v20 that is in active development as of now, but if tests are green then why not? * Switched integration tests to use v16 * Removed testing of quite old and deprecated v10 * Updated version of BoringSSL submodule to a newer one (not the latest though) commit 05cac26 Author: Anatolii Lishchynskyi <[email protected]> Date: Thu Nov 9 22:24:35 2023 +0200 CI fixes (cossacklabs#1034) Rix Rust CI builds (cossacklabs#1032) * Pin log dependency to 0.4.17 * Pin byteorder dependency to 1.4.3 Last versions that still work with current MSRV 1.58. Fix sanitizers CI job (cossacklabs#1033) * Switch to GCC 10 * Install `libgcc-10-dev` that provides file `libtsan_preinit.o` needed for thread sanitizer Change priority of PREFIX in CFLAGS and LDFLAGS (cossacklabs#1031) * Change include dir priority in CFLAGS Move 'CFLAGS += -I/usr/local/include' below engine selection macros, so that if OS have OpenSSL header files installed in /usr/local/include, it won't mess with vendored BoringSSL or any other selected engine. * Put /usr/local/lib in separate LDFLAGS Introduce ADDITIONAL_LDFLAGS macro and put it after both LDFLAGS and CRYPTO_ENGINE_LDFLAGS during linking. * Add optional verbose logging to Makefile Add VERBOSE option to Makefile. If set, print executed command (compiler, linker etc) even for successful runs, and run ldd on created shared libraries. * Enable verbose builds on macOS jobs commit 06d52f4 Author: Anatolii Lishchynskyi <[email protected]> Date: Tue Sep 26 13:15:37 2023 +0300 Fix make target rbthemis_uninstall (cossacklabs#1022) * Fix make target rbthemis_uninstall * Add RbThemis uninstall step to CI commit 8609650 Author: Nazar Serhiichuk <[email protected]> Date: Mon Jul 10 22:57:46 2023 +0300 Avoid specific versions in README (cossacklabs#1016) Just to avoid responsibility of updating it during release (•_•) ( •_•)>⌐■-■ (⌐■_■) commit 3219654 Author: Martin Arista <[email protected]> Date: Mon Jul 10 11:54:25 2023 -0400 Update README.md (cossacklabs#1015) update links for maven and java/kotlin links
Lagovas
pushed a commit
that referenced
this pull request
Dec 5, 2023
* Pre-release Themis 0.15.0 (#1011) * Fix rust issues (pin log, run bindgen) (#1005) * rust: Pin log version to =0.4.18 The 0.4.19 requires rustc 1.60, but currently we support 1.58. Pinning it is not a big deal since it's development dependecy for tests and examples. * rust: Regenerate and update lib.rs bindgen was updated again and changed something which resulted in new output (seems like some internal constants are removed). * Pythemis: introduce `pyproject.toml` (#1006) * pythemis: Add pyproject.toml Since setup.py is deprecated, let's try moving to the pyproject.toml and configuring it with the same data as in setup.py. Use setuptools as a backend for no particular reasons ¯\_(ツ)_/¯, just because the name is familiar and we have no reasons to not use it or use something else. Keep the old setup.py for backward compatibility so old systems can try to build the package. For now, keep 0.14.0, we will bump the version in another PR. * makefile: Use pyproject.toml for installing pythemis According to this [1] article, the correct command is pip install . in the project's root. Let's try that. Also, the other option is python -m build --wheel which builds the package but doesn't install it. We can provide something like `pythemis_build` for it for example. [1]: https://godatadriven.com/blog/a-practical-guide-to-setuptools-and-pyproject-toml/ * pythemis: Update classifiers to Python3.6+ With many hours and docker containers I tested that themis actually works up to python 3.4. The other versions require some changes in makefile so they are more like "grey area". However, python3.5 is deprecated and it produces warning like "DEPRECATION: Python 3.5 reached the end of its life on..." so many libraries don't support it. Instead they start with 3.6 which will do as well, I guess. Though, actually python3.6 is also deprecated [1]. The same will be true for python3.7 in a couple of days (Jun 27 2023), so the question is, should we declare support of these versions? [1]: https://devguide.python.org/versions/ * pythemis: Extend range of supported py versions * Update changelog * Run and pin bindgen (#1008) * rust-themis: Update bindgen It updated and broke something again 🤦 * rust-themis: Pin bindgen version It is pretty unstable with its frequent releases, so let's pin it. * Update changelog * Bump wrapper versions to 0.15.0 (#1007) * changelog: Add 0.15.0 summary * themis-core: Update version * pythemis: Update version * pythemis: Fix 8-year old typo in AUTHORS :) * rbthemis: Update version * jsthemis: Update versions * wasm-themis: Update versions * android-themis: Update version * rust-themis: Update versions * react-native-themis: Update versions * pythemis: https in AUTHORS Co-authored-by: vixentael <[email protected]> * rust-themis: Update bench versions Somehow missed that. * changelog: Forgot to mention rust 1.58 * changelog: Mention the new iteration count --------- Co-authored-by: vixentael <[email protected]> * Bump embedded BoringSSL (#1004) * Bump BoringSSL and fix makefile This is not the latest BoringSSL version yet, because there are a couple of fixes. So, treat it as the first. Here we also fix our makefile because the BoringSSL team fixed bug with the strange behaviour of absolute path to symbols.txt [1]. [1]: https://boringssl.googlesource.com/boringssl/+/8c75ed046f799f1d8b805036b1dea9c5ec0a0fb5%5E%21/#F0 * Bump BoringSSL and fix opaque EVP As OpenSSL, BoringSSL made many types opaque, so it will require updating some of the code to not use fields. * Bump BoringSSL again and fix RSA The same issue - RSA type became opaque, so we need to use accessors similar to what Openssl had. * Bump BoringSSL once more This is (hoperfully) the last bump. This time without issues but we will see what CI says. * Make bignum_to_bytes accept const bignum* It will prevent some of the warnings. This function doesn't mutate bignum anyway. * Update changelog * boringssl: Bump once again * msys2: Update hashes temporarily This are test values because we will move the tag. But for now, let's just test it. * phpthemis: Update version for the sake of testing They will fail probably, but just out of curiosity let's try to run the tests. * Update date of the release Solstice! --------- Co-authored-by: vixentael <[email protected]> * msys2: Update checksums (#1012) It's the egg-chicken problem: we can update those hashes only after the release. But then, the release tag will not point to the updated hashes. * Squashed commit of the following: commit 784033b Author: Anatolii Lishchynskyi <[email protected]> Date: Tue Nov 28 21:45:12 2023 +0200 New methods of building/installing PyThemis (#1023) Refactor Makefile: * new target pythemis_make_wheel to create a .whl Python package, current modern format to be installed into virtual environmants * new target pythemis_install_wheel to install it in currently active virtualenv * new target deb_python that builds a .deb package for system-wide installation of PyThemis * new target pythemis_install_deb, alias for pythemis_deb + apt install of the created pkg * new target rpm_python, similar to deb_python * new target pythemis_install_rpm, pythemis_install_deb Update GitHub Actions workflow, test .whl and .deb installation See cossacklabs/product-docs/pull/317 for related docs update commit 30578c8 Author: Anatolii Lishchynskyi <[email protected]> Date: Mon Nov 27 17:53:02 2023 +0200 Fix clippy and fmt issues, update MSRV (#1039) Fix clippy and fmt issues Update MSRV to 1.60 Freeze test deps so they compile on Rust 1.60, with no effect on themis itself (does not use those frozen crates) commit 6111766 Author: Anatolii Lishchynskyi <[email protected]> Date: Tue Nov 21 17:08:51 2023 +0200 Update emscripten requirements and WASM CI job (#1036) * Bump emsdk version to 3.1.47 Produces module importable in Node v18 With older (emsdk 3.0.0) version, generated `libthemis.js` that should load `libthemis.wasm` fails due to some internal autogenerated code working differently on v16 and v18 * Add link flag for WASM builds Needed to make new emscripten produce working module * Update Node testing versions Add v18 that should now work, also add v20 that is in active development as of now, but if tests are green then why not? * Switched integration tests to use v16 * Removed testing of quite old and deprecated v10 * Updated version of BoringSSL submodule to a newer one (not the latest though) commit 05cac26 Author: Anatolii Lishchynskyi <[email protected]> Date: Thu Nov 9 22:24:35 2023 +0200 CI fixes (#1034) Rix Rust CI builds (#1032) * Pin log dependency to 0.4.17 * Pin byteorder dependency to 1.4.3 Last versions that still work with current MSRV 1.58. Fix sanitizers CI job (#1033) * Switch to GCC 10 * Install `libgcc-10-dev` that provides file `libtsan_preinit.o` needed for thread sanitizer Change priority of PREFIX in CFLAGS and LDFLAGS (#1031) * Change include dir priority in CFLAGS Move 'CFLAGS += -I/usr/local/include' below engine selection macros, so that if OS have OpenSSL header files installed in /usr/local/include, it won't mess with vendored BoringSSL or any other selected engine. * Put /usr/local/lib in separate LDFLAGS Introduce ADDITIONAL_LDFLAGS macro and put it after both LDFLAGS and CRYPTO_ENGINE_LDFLAGS during linking. * Add optional verbose logging to Makefile Add VERBOSE option to Makefile. If set, print executed command (compiler, linker etc) even for successful runs, and run ldd on created shared libraries. * Enable verbose builds on macOS jobs commit 06d52f4 Author: Anatolii Lishchynskyi <[email protected]> Date: Tue Sep 26 13:15:37 2023 +0300 Fix make target rbthemis_uninstall (#1022) * Fix make target rbthemis_uninstall * Add RbThemis uninstall step to CI commit 8609650 Author: Nazar Serhiichuk <[email protected]> Date: Mon Jul 10 22:57:46 2023 +0300 Avoid specific versions in README (#1016) Just to avoid responsibility of updating it during release (•_•) ( •_•)>⌐■-■ (⌐■_■) commit 3219654 Author: Martin Arista <[email protected]> Date: Mon Jul 10 11:54:25 2023 -0400 Update README.md (#1015) update links for maven and java/kotlin links * Bump boringssl * Revert "Bump boringssl" This reverts commit dee8388. Okay, this time I checked, the previous commit of boiringssl was newer than one in the master. * msys2: Fix hashes for 0.15.0 release (#1040) Since we already have 0.15.0 assets on GitHub, we can specify those and expect successful builds on stable. --------- Co-authored-by: vixentael <[email protected]>
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
Refactor Makefile:
pythemis_make_wheelto create a.whlPython package, current modern format to be installed into virtual environmantspythemis_install_wheelto install it in currently active virtualenvdeb_pythonthat builds a.debpackage for system-wide installation of PyThemispythemis_install_deb, alias forpythemis_deb+apt installof the created pkgrpm_python, similar todeb_pythonpythemis_install_rpm,pythemis_install_debSee cossacklabs/product-docs/pull/317 for related docs update
Checklist
Benchmark results are attached (if applicable)Public API has proper documentationExample projects and code samples are up-to-date (in case of API changes)