-
Notifications
You must be signed in to change notification settings - Fork 143
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
Merge branch master
into wasm-typescript
#845
Conversation
Our benchmarking harness has decided that it's time for some INNOVATION and for starters released a version which has disabled generation of HTML reports by default, changed their location from current directory to the proper shared target directory, and strongly suggests migrating away from running benchmarks with "cargo bench" in favor of their own "cargo criterion". Well, okay, that's a fine rabbit to chase later, but for now let's at least unbreak our build by adapting to the new reality. (cherry picked from commit d230f57) Also added the same path update for CircleCI which is still running for the stable branch.
* Do not install Python 2 for examples pip has dropped support for Python 2 some time ago -- a version of Python that has been EOL for about a year now. Themis does not support Python 2 officially, yet we're still test with it as long as it's practical. Well, it stopped being practical for this CI job because Ubuntu has removed "python-pip" package from their repos. Remove all lines which install Python 2 testing environment. * Run examples only with Python 3 For quite some time we have been running this job only with Python 3. Now that Python 2 environment installation has been removed, it no longer makes sense to pretend that we'll enable it back. Just run everything with "python3". (cherry picked from commit 68827da)
…#772) Recent release of bindgen 0.56.0 resulted in a change in the order of the derived traits in the generated code. The changelog does not say anything about that, but I doubt this is considered a notable change. However, it breaks our build system which meticulously checks for any changes in the generated code compared to checked in version. It does this so that we don't miss any changes. Well, the CI is going to be using bindgen 0.56.0 from now on, let's use the new trait order as well, whatever it is. (cherry picked from commit 3f8f9b8)
…ssacklabs#773) One of the GitHub Actions jobs is using a pristine Ubuntu container because whatever is provided by GitHub Actions runners causes conflicts when some of the packages are installed. That container has been using Ubuntu 19.10. Ubuntu 19.10 "Eoan Ermine" has reached end of life on 2020-07-17. Normally EOL only means that the distribution stops receiving updates after that date. However, eventually it gets purged from the repositories as well. This is exactly what has happened recently, and since then the builds are all red because they are not able to install packages for a missing distribution. Upgrade to Ubuntu 20.04 LTS "Focal Focca". This should last us at least until 2030 or something. Or well, until GitHub becomes more attentive to whatever stuff they put in their sources.list. (cherry picked from commit 647fb0c)
* Migrate to "afl++" from "afl" The original "afl" [1] has been abandoned and is not maintained since 2017. "afl++" [2] is its replacement. Since it's a fork, it has mostly compatible command line, and more features. [1]: http://lcamtuf.coredump.cx/afl/ [2]: https://github.com/AFLplusplus/AFLplusplus The original afl is currently being phased out from distributions. In particular, it's gone from whatever repo list GitHub Actions use, breaking our build. Migrate to "afl++" instead. Not all operating systems ship with "afl++" though. For example, it's still not available in binary form on macOS via Homebrew. Therefore, don't jump the gun and don't update the Makefile to use "afl++"-only features. (However, it would be nice to use them if available.) * Pin "fuzzing" job to Ubuntu 18.04 As noted in the comment, GitHub Actions runners are schizophrenic and sometimes believe that they come from the future where the latest Ubuntu image works correctly. Don't let them be fooled, use 18.04 for now. (cherry picked from commit 0c6a44d) Ignore changes in CHANGELOG.md, there is no "Unreleased" section on the "stable" branch. It will get here in due time.
Recently released Rust 1.51 includes new Clippy with new warnings: error: name `GetPublicKeyForID` contains a capitalized acronym --> src/wrappers/themis/rust/tests/secure_session.rs:663:6 | 663 | type GetPublicKeyForID = Box<dyn FnMut(&[u8]) -> Option<EcdsaPublicKey>>; | ^^^^^^^^^^^^^^^^^ help: consider making the acronym lowercase, except the initial letter: `GetPublicKeyForId` | = note: `-D clippy::upper-case-acronyms` implied by `-D warnings` = help: for further information visit https://rust-lang.github.io/rust-clippy/master/index.html#upper_case_acronyms error: aborting due to previous error Shut up, Clippy, I name my types however I want to!^K I mean, yes, sure, O Wise Stationery, I will immediately rename the entity as prescribed and will refrain from such practice any further. (cherry picked from commit 734a624)
This is the only dependency of RustThemis and they have managed to break even that. Some bright maintainer head over there thought that it would be a great idea to just yank all 0.x versions from crates.io, breaking our build. (Since RustThemis is a library, we're not using Cargo.lock. If a dependency is yanked, we can't build our stuff.) Well, thanks for not breaking the API in 1.x at least. You're still better than the Tokio team. Note that zeroize requires reasonably new stable version of Rust toolchain. With 0.5.2 this has been 1.31, now it's 1.47. (The current stable version is 1.51. RustThemis tracks that version.)
This is the only dependency of RustThemis and they have managed to break even that. Some bright maintainer head over there thought that it would be a great idea to just yank all 0.x versions from crates.io, breaking our build. (Since RustThemis is a library, we're not using Cargo.lock. If a dependency is yanked, we can't build our stuff.) Well, thanks for not breaking the API in 1.x at least. You're still better than the Tokio team. Note that zeroize requires reasonably new stable version of Rust toolchain. With 0.5.2 this has been 1.31, now it's 1.47. (The current stable version is 1.51. RustThemis tracks that version.) (cherry picked from commit 9b3fc01)
* Add Node.js v16 support Node.js v16 has been released yesterday [1], becoming the new "current" stable release track of Node.js. It's going to turn LTS in October 2021. [1]: https://nodejs.org/en/about/releases/ Node.js v10 release is scheduled for EOL this April. However, we're still going to build and test our JavaScript stuff against it while it's still practical. * Test Node.js examples with recent Node.js versions Oh, I have completely forgot about this job. Let's make it run with all currently versions of Node.js.
* add SPM support with XCF * add swift SPM test project * add macOS swift SPM example, updated iOS one * updates per pr comments * removed cl-openssl from Themis.xcodeproj + edits per pr comments + extra comments * removed example projects * updated xcodeproj version * updated Package.swift * updated changelog * Update CHANGELOG.md Co-authored-by: vixentael <[email protected]> Co-authored-by: vixentael <[email protected]>
Merged stable into release/0.13
* Add SPM support with XCF (cossacklabs#789) * add SPM support with XCF * add swift SPM test project * add macOS swift SPM example, updated iOS one * updates per pr comments * removed cl-openssl from Themis.xcodeproj + edits per pr comments + extra comments * removed example projects * updated xcodeproj version * updated Package.swift * updated changelog * Update CHANGELOG.md Co-authored-by: vixentael <[email protected]> Co-authored-by: vixentael <[email protected]> * fixed automerge issue * fixed typo Co-authored-by: vixentael <[email protected]>
0.13.7 release process: merge release/0.13 to stable
OpenSSL 1.1.1k for iOS/macOS XCF and SPM
merge release 0.13.8 into stable
Update npm lockfile format. We don't get a say here. Update some dependencies with it, resolving some security advisories: - CVE-2020-7774 Prototype Pollution in y18n - CVE-2021-23337 Command Injection in lodash These are not terribly important since they do not affect our users, only potentially affect the CI environment.
* Copy cossacklabs/themis-java-examples/android-example Start by importing the example files. While I could import the history as well, there is not much value in that, so here are just the files. Note that this example uses Gradle, like Themis project itself, but the example is completely independent of the top-level Gradle project. It even uses a different version of Gradle wrapper because that's what Android Studio insists on. I'd like to keep it that way, so that users are able to copy and paste the project directory and get a working example, not dependent on any Themis structure. The example is also tied to a published version of AndroidThemis, not the one being developed in the source tree. Maintainers will have to update the dependency manually, similar to how macOS/iOS examples work. * Bump Gradle version 4.1.2 => 4.1.3 Android Studio suggests this change, let's go with it. I guess we could make a bit of a wider jump to get to the current 6.x versions, but maybe a bit later. * Bump target SDK 26 => 29 * Migrate from "android.support" library to "androidx.appcompat" Another sacrifice to the CADT model because Android Studio is so insistent on it. At least they have "Refactoring | Migrate to AndoridX" which has helped a lot. * Bump androidx.appcompat 1.0.0 => 1.2.0 * Bump androidx.constraintlayout 1.1.3 => 2.0.4 * Configure Kotlin for the project I'm using 1.3.72 because that's what my Android Studio comes with and it's pretty insistent on using that. The latest version is 1.5.0. * Convert Android example to Kotlin Automatic conversion with "Code | Convert Java file to Kotlin file". This works reasonably well for initial conversion, but more work needs to be done manually to fix up some remaining bits. I'm converting Android example to Kotlin because that's how most Android stuff is developed these days. Java is really a legacy language there. Java 16 is the current version. Android has mostly Java 7 with some selected bits of Java 8. Go figure. JavaThemis will have its own separate example using Java specifically. AndroidThemis now has a Themis showcase. * Adjust nullability here and there Automatic conversion did not get the fact that ISessionCallbacks has nullable return types. Maybe that's because they don't have annotation? * Bump Kotlin 1.3.72 => 1.4.32 Android Studio has suddenly discovered that it has some update. Gotta update then, eh. * Use Kotlin-style array comparisons Now it's a flurry of miscellaneous IDE-suggested updates. * Don't use properties where parameters suffice * Relax exception annotations Kotlin is not very keen on Java checked exceptions. All exceptions there are unchecked, runtime ones. Drop all the exception annotations and fine-grained catch clauses, replace them with catch-alls. Currently we don't do any involved exception handling, we just want to prevent app crashes. Specific exception handling will have specific exceptions. * Explicit Themis imports Make sure we import everything we need explicitly. Previously we have also imported a bunch of exception types which we now don't need. * Build Android examples on GitHub Actions Just build. It would be cool to test the resulting application but that opens up a whole new time hell hole, with launching emulators and stuff. I have no desire to dive there at the moment. * Update changelog This is somewhat notable so let's update the changelog too. Once JavaThemis example is moved to the main repository as well, we can finally deprecate and archive that separate repo.
* Copy cossacklabs/themis-java-examples/java-example Start by copying existing example code. Just like with Android, simply copy the content as independent project, without integrating it into the top-level Gradle scripts. * Restructure Gradle project a bit Make it look more like Android example. Introduce its own "settings.gradle" so that Gradle treats it as an independent project instead of freaking out about "settings.gradle" at source tree root. * Run JavaThemis examples on CI Add a job to exercise the example project. Admittedly, it's doing some weird things there, but that's at least something. Ideally, we would like to be testing both versions: the tip one and the released one. * Update changelog Make a couple of entries to note that example project has a new home.
* Add benchmark for Token Protect - Master Key * Add benchmark for Context Imprint - Master Key * Add benchmark for Secure Message, enc/dec - ECDSA * Add benchmark for Secure Message, enc/dec - RSA * Add benchmark for Secure Message, sig/ver - ECDSA * Add benchmark for Secure Message, sig/ver - RSA * Update README Put '✅ complete' in the benchmark table. * Add benchmark for wrapped cell in seal mode Wrapped cell means 'Rust wrapper of Secure Cell' * Add benchmark for wrapped cell in token protect * Add bench for wrapped cell in context imprint mode * Add bench for wrapped cell in seal with passphrase * Change naming of Rust benches `secure_cell_wrapped -> rust_secure_cell` `Wrapped Secure Cell -> RustThemis - Secure Cell` * Add Secure Message benches into CI * Fix CI to run only core benchmarks of Secure Cell Before, CI used the next regexp to filter the benches: "Secure Cell .* passphrase/4 KB". But it matches "Secure Cell..." as well as "RustThemis - Secure Cell...". Replace regexp with an explicit "^Secure Cell...". * Fix CI to run only core benches of Secure Message The same as a prevoius one, plus fix some quotation marks formatting issues. * Move RustThemis benches into separate crate * Remove prefix `rust_` in the RustThemis benchmark filenames * Update benches/README.md Add entry about rust wrapper benches * Update benches/rust/README.md Add the goal of the RustThemis benchmarks. Co-authored-by: vixentael <[email protected]> * Refactor with cargo-clippy Remove some redundant copies Co-authored-by: vixentael <[email protected]> Co-authored-by: Alexei Lozovsky <[email protected]>
* Uninstall CMake 3.18.1 from Android SDK Gradle ends up really confused by this CMake version: FAILURE: Build failed with an exception. * What went wrong: Could not determine the dependencies of task ':android:assembleDebug'. > A problem occurred configuring project ':boringssl'. > Invalid revision: 3.18.1-g262b901 Our builds are fine with older CMake (for now), so let's use that. * Pin runner to macos-10.15 GitHub people are working on adding macOS 11 support for Actions and it's probably going to happen soon. However, I don't really want to be notified about new and exciting ways to break our builds when that finally happens, so let's keep using the old environment for now.
* Cherry-pick: Updated Carthage examples to use XCF (cossacklabs#823) * Fixing Carthage examples embed settings (cossacklabs#827) * fix changelog conflict
* Updated podspec to support OpenSSL-XCF * versions update * update changelog * Cherry-pick: Updated Carthage examples to use XCF (cossacklabs#823) * Fixing Carthage examples embed settings (cossacklabs#827) * Revert "Fixing Carthage examples embed settings (cossacklabs#827)" This reverts commit d9403c5. * Revert "Cherry-pick: Updated Carthage examples to use XCF (cossacklabs#823)" This reverts commit 6c14677. * update changelog * added deprecation notice to podspec and changelog * update changelog
* Prefer GitHub Discussions for general inquiries This feature is still technically in beta but it's already usable. That's a nice way to keep general "question" issues out of the issue tracker which should contain only something actionable. Make Discussions the first link that you'd like to click if you just want to chat about something and not to report a bug. * Link to dicussions from the CONTRIBUTING.md Add this link there too, just so that users will have a chance to know that Discussions is a thing.
By the power of vim and blessing of Git, I hereby confer onto thee a Dukedom. Your titles shall now be Duchess of Eyeos, Marchioness of Swift and Objective Countess. May you govern wisely and your domain prosper.
This resolves CVE-2020-28469 advisory that has been there for a while. Nothing terribly important.
Oh, there a bunch of other tweaks to WasmThemis that I'd like to throw in (tests and examples). So I guess I'll put this PR to draft for the time being, then redo the merge. |
Instead of running all builds every day, run them only once a week. Also, since there are a lot of builds across the entire ecosystem started at the beginning of an hour, offset our builds a bit. We have quite a few builds and I feel a bit sorry for GitHub Actions that have to work so much. No, actually, I'm not. Microsoft has the $. What I do care about is amount of email I get about broken builds. For the past 30 days I have received 19 notifications about something being broken in Objective-C examples or other tasks. Out of which 0 (zero) have been actual, actionable issues, not cloud farts. Now I'm going to receive maybe three emails a week at worst. I can live with this.
* Test GoThemis with more versions of Go Somehow we have forgot to add new versions when they come out. * Run GoThemis tests on CI in a modular way Instead of installing GoThemis and running tests via "make test_go", run them directly from GoThemis module. Starting with Go 1.16, the new testing toolchain does not really like global installation into $GOPATH/src. Just run unit tests for the module. Note that "make test_go" keeps its behavior of testing globally installed GoThemis, if you wish to do so. However, you will need to explicitly disable modules with GO111MODULE=off and install GoThemis into GOPATH. * Run "make test_go" in non-modular, GOPATH mode Currently "make test_go" is similar to script languages in that it tests global installation of GoThemis. However, Go has been steadily migrating away from global package installation since Go 1.11 and the advent of modules. Starting with Go 1.16 the modular approach is the default. Keep the existing behavior of "make test_go" which expects global installation of GoThemis. It will not work without GoThemis installed into GOPATH via GO111MODULE=off go get github.com/cossacklabs/themis/gothemis I don't know whether it's time to deprecate this Makefile target or not. It still kinda works, but it's definitely less useful than it was.
* Fix cross-compilation on macOS with SDK and ARCH Surprisingly, it's possible to cross-compile Themis Core for different SDKs and architectures using our Makefile. For example, make SDK=macosx11.0 ARCH=arm64 will compile for Apple Silicon, provided that your Xcode toolchain is actually capable of building that architecture. You also need to have OpenSSL binaries with desired architecture installed (for which you might need to tweak ENGINE_INCLUDE_PATH and ENGINE_LIB_PATH separately). Or build ENGINE=boringssl to use embedded BoringSSL that we can cross-compile too. However, cross-compilation does not work because the architecture and system library directories are not set at link time, so libthemis.dylib it attempted to be linked for the native target which does not work. Pass proper settings to LDFLAGS at configuration time so that cross-compilation has a chance to work. * Pass cross-compilation options to embedded BoringSSL build If we seem to be cross-compiling, tell CMake what SDK it should use and what architecture it should be building. It knows what to do next. * Test cross-compilation on GitHub Actions Throw in a job to make sure that cross-compilations stays in working order. Unfortunately, we can't actually run unit tests, but hey, at least it compiles.
* Test WasmThemis command-line tools Put it under "Code examples" job because that's the only example code for WasmThemis that we have at the moment. * Modular ES6-style "tools" for WasmThemis In addition to ES5-style tools add those that use ES6 module syntax. This is literally the same code, but with different import syntax. Modern Node.js can work with both. (And make sure it does on CI.) If we are going that route of compiling from TypeScript, these will be the only tools that are going to ultimately work. * Code examples for WasmThemis (ES5-style for Node.js) WasmThemis did not really have any code examples. We bullshit our way in by pretending that JsThemis examples are good, but they actually are not compatible with WasmThemis due to API differences and WebAssembly stuff. Write examples for WasmThemis, start with ES5-style for Node.js. This is basically copypasta of JsThemis examples, just cleaned up a bit. I don't want to invent CI tests for new examples so they are compatible. * Code examples for WasmThemis (ES6-style for Node.js) Now the same thing but with different file name extension, different import syntax, and async function support (though, Node.js APIs are still mostly callback-based). * Fix variable references in Secure Cell + passphrase tool This fails in ES6 mode but not in ES5, that's because ES6 is a bit more strict on what gets captured and survives and what does not. Note that assigning previously undeclared variable *is not* an error in JavaScript (this is JavaScript, after all), but it *depends* whether that variable will be captured (this is JavaScript, you know). * Build WasmThemis only once I know that GitHub Actions compute is kinda free, but still... building WasmThemis eight times for all our jobs is wasteful. We do this in exactly the same Emscripten environment, host Node.js version does not affect the build result. Split out the building part into a separate job, run Themis Core tests there too (with Emscripten's Node.js). Then archive the build directory and upload it as an artifact to be shared with other jobs. Other jobs that run WasmThemis unit-tests and examples will pull the build directory with WebAssembly build results, unpack it, and run then stuff with prebuilt WebAssembly binaries (use that portability for once). This is similar to what users will do: download binaries from npm. Note that unit tests and examples *are* exercised with various Node.js versions. * Make ES5-like code look like ES5 That is, no using "let" and "const" which are actually ES6-only and are allowed in code executed by Node.js only because Node.js. Am I doing this right, mom? * Run WasmThemis jobs when examples change * Fail tests on unhandled rejections Some of our examples don't handle top-level promise rejection because it's bothersome. Make sure that Node.js process exits with a failure if those examples fail due to an unhandled promise rejection. That is, code like this will now fail tests: themis.initialized.then(function() { throw new Error('this should cause non-zero exit code') }) instead of just printing a warning. NODE_OPTIONS can include "--unhandled-rejections" only for Node.js v10 or later. v8 will fail due to an unknown option. And we still test v8. GitHub Actions do not provide a nice way to set environment variables based on the matrix configuration and I don't want to add a job step that does that manually. We care about this setting only when testing examples, so set it just there, where we use Node.js v12+
* Example WasmThemis project with webpack Finally my dream is coming true: interactive example of Themis right in the browser. It's a bit sad that I have to work for my own dreams, but doing what I can with what I got. The first commit adds a skeleton with webpack 5, pulling in WasmThemis and half of the internet of other dependencies. Don't think about it too much, okay? This is normal for JavaScript. Right now we don't do anything, but at least you can see that WasmThemis has loaded successfully. Or not. * Utilities for UTF-8 and base64 conversion JavaScript has no standard library and we'll need to encode JavaScript strings as UTF-8 into Uint8Arrays, Uint8Arrays as base64 in strings, and vice versa. JavaScript developers would pop a library for that. I hate dependencies with a passion of a thousand burning suns, so instead just reuse the code I found on MDN. Thanks, Mozilla! * Secure Cell interactive example Throw in layout, code, and basic styling for interactive example of Secure Cell. You can input things and observe results. Performance is okayish for symmetric keys. Passphrases are slow. Damn slow. I thought about splitting this into commits, but no one reads these messages, right? 🍌 <BANANA MODIFIER VIOLET> So why bother? * Apply Cascading Style Shits These took a while to get them more or less right. I don't envy web designers that have to deal with this shit on a daily basis. I am so exhausted from Safari thinking differently™ that I am not going to clean up the CSS file, just throwing it in as is. (Like, why the hell disabled inputs are 1 pixel shorter that normal ones? Who knows.) But at least now the page looks *somewhat* decently. I don't care for more formatting, as long as it's functional. * Auto-adjust height of textareas Normally text areas don't resize to fit their content, but this looks weird when you enter some significant amount of text. While two lines is good enough to input "test" or "message" there, let's make the text areas grow a bit in response to user input and content changes. * Break error message lines for readability Error messages don't show nicely because they are a bit too long. Make sure they are formatted like this: Error: Themis: SecureCellContextImprint: context must be not empty which looks a bit better. The "source" of the error on one line, and its "body" on the following one. * Preprocess "index.html" with HTML plugin Get "index.html" ready for subresource integrity injection. We'll need to compute SRI and inject it into HTML content. That's why it can't stay a verbatim resource. * Compute SRI hash sum for JavaScript output This is an example project for a security-conscious library and yet we don't use SRI? Ridiculous. Throw in another plugin to compute the hash of the compiler JS bundle and inject that into <script> tags in HTML. * Switch to "wasm-themis" Now that it's released, use the proper release channel instead of my private preview. * Minor CSS tweaks Adjust textarea margins to get some more air between the textarea and the label text above it.
I've redone the merge. Now the branch includes newly added examples and tools for WasmThemis. WasmThemis examples (and integration tests with WasmThemis) are expected to fail due to backwards-incompatible changes on WasmThemis unit tests are expected to pass. |
Conflicts in "wasm/package-lock.json" have been resolved by merging changes from both branches. There were also conflicts in "tools/js/wasm-themis" as 'wasm-typescript' branch has introduced some breaking changes in API. I've reconciled the changes between branches, favoving the "broken" API here. I've also updates other WasmThemis examples to use this API. The plan is to get the CI running, helping with this, and then work on removing backwards-incompatible changes in later commits.
Changed my mind once again and redone the merge. Now all tests are expect to pass, examples are expected to work. Conflicts in There were also conflicts in The plan is to get the CI running, helping with this, and then work on removing backwards-incompatible changes in later commits. |
Well, tests are passing.. |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
I didn't review all the changes, because PRs is huge; you wrote "merge via console"; soooo if there's anything backdoory here, that's on you :D
@vixentael, I don't have a good git fu to teach you on how to review merge commits between branches that both have loads of new commits on them. “Trust me, I'm an engineer!” The answer probably is don't let the situation deteriorate to this state 🤷 |
in next life probably :) |
Well, at least you can check that I've merged the commit I intended to merge, not pushed something completely different. |
Sync changes on
master
withwasm-typescript
, resolve merge conflicts. Most changes are unrelated to WasmThemis, but I want to pull dependency bumps that fixnpm audit
advisories. And speedy Ninja builds for BoringSSL.TODO:
master
Checklist
git push
once approved.