Update Emscripten to 2.0.9 upstream (#760)

* Update Emscripten to 2.0.9 upstream

For quite some time Themis has been pinning Emscripten to an outdated
version of the "fastcomp" flavor because recent versions have been
broken, and the new "upstream" flavor has been broken as well.

However, Emscripten 2.0+ does not support the "fastcomp" flavor anymore
so there is no hope for an fix. Eventually we'd have to migrate to the
"upstream" flavor.

I have tried out the latest Emscripten 2.0.9 upstream and it seems to
be working--after some further tweaks. There are no signs of the issues
that were observed before and prevented Themis Core from being compiled.
With the latest version is can be compiled and run.

Update the Emscripten version we use on CI to 2.0.9. This is also the
version that we'd be building packages with for the next release. Note
that the version is pinned and will need to be updated later, we won't
be tracking the "latest" tag for a while, until we're sure that
WasmThemis works fine with Emscripten 2.0.

* Disable stack protector for Emscripten builds

As noted in the linked issue, the upstream flavor of Emscripten does not
yet support LLVM's stack protector so have it disabled. Otherwise linker
will produce errors about missing symbols, because they're not there.
Some day it may be implemented though so I leave a FIXME in the code.

* Use new allocate() API of Emscripten

Themis Core uses the allocate() function provided by Emscripten runtime
for quick on-stack allocations. This is a semi-private function which
has unstable API. With the new Emscripten version its API has changed,
now it accepts a "slab" parameter which indicates the size of the stack
that needs to be allocated--for the entire function. That is, if we
allocate memory on stack, allocate() may be called only once.

* Stop using deprecated RESERVED_FUNCTION_POINTERS

RESERVED_FUNCTION_POINTERS does not have much effect on the "upstream"
flavor but it was important for the "fastcomp" flavor. Now that since
Emscripten 2.0.1 the "fastcomp" flavor is not supported anymore, this
option has been a synonym for ALLOW_TABLE_GROWTH. Use the new option
instead directly then.

* Work around issues with dead code elimination

For some reason wasm-ld likes to strip *all* (each and every) function
of Themis Core from the resulting *.wasm file. While they are there in
the object files and archives, linker's dead code elimination thinks
they are not needed in the WebAssembly code.

Normally this is prevented by EMSCRIPTEN_KEEPALIVE attribute--which we
still have and it's still defined--but for some reason it does not work.

As a temporary workaround, pass the LINKABLE option to the linker which
disables dead code elimination and other link-time optimizations. This
leads to somewhat bigger WebAssembly file which may be a bit slower,
but at least this prevents issues.

It might be some transient issue with a toolchain so there is a FIXME
in the code. We'd need to keep an eye on it for some future releases.
However, I have not found any similar issues in Emscripten tracker.

.github/workflows/integration.yaml

@@ -75,16 +75,8 @@ jobs:
           cd $HOME
           git clone https://github.com/emscripten-core/emsdk.git
           cd $HOME/emsdk
-          # FIXME(ilammy, 2020-07-07): unpin emsdk version [T1698]
-          # Latest HEAD has a regression, we roll back to last good version.
-          git checkout 92d512faa832b3ff5d6b8bc991b6801e31d8e372
-          # FIXME(ilammy, 2020-06-07): migrate to "upstream" flavor
-          # LLVM flavor of Emscripten has some issues compiling our code,
-          # and latest versions of the fastcomp flavor started giving out
-          # compiler warnings (turned into errors by the build system).
-          # We need to migrate, but for the time being use the old version.
-          ./emsdk install  1.39.16-fastcomp
-          ./emsdk activate 1.39.16-fastcomp
+          ./emsdk install  2.0.9
+          ./emsdk activate 2.0.9
       - name: Install PHP from PPA
         run: |
           sudo apt install --yes software-properties-common

.github/workflows/test-wasm.yaml

@@ -58,16 +58,8 @@ jobs:
           cd $HOME
           git clone https://github.com/emscripten-core/emsdk.git
           cd $HOME/emsdk
-          # FIXME(ilammy, 2020-07-07): unpin emsdk version [T1698]
-          # Latest HEAD has a regression, we roll back to last good version.
-          git checkout 92d512faa832b3ff5d6b8bc991b6801e31d8e372
-          # FIXME(ilammy, 2020-06-07): migrate to "upstream" flavor
-          # LLVM flavor of Emscripten has some issues compiling our code,
-          # and latest versions of the fastcomp flavor started giving out
-          # compiler warnings (turned into errors by the build system).
-          # We need to migrate, but for the time being use the old version.
-          ./emsdk install  1.39.16-fastcomp
-          ./emsdk activate 1.39.16-fastcomp
+          ./emsdk install  2.0.9
+          ./emsdk activate 2.0.9
       - name: Check out code
         uses: actions/checkout@v2
         with:

CHANGELOG.md

@@ -102,6 +102,10 @@ _Code:_
   - `TSSession` initializer now returns an error (`nil`) when given incorrect key type ([#710](https://github.com/cossacklabs/themis/pull/710)).
   - Improved compatibility with Xcode 12 ([#742](https://github.com/cossacklabs/themis/pull/742)).
 
+- **WebAssembly**
+
+  - Updated Emscripten toolchain to the latest version ([#760](https://github.com/cossacklabs/themis/pull/760)).
+
 _Infrastructure:_
 
 - Improved package split making `libthemis` thinner ([#678](https://github.com/cossacklabs/themis/pull/678)).

Makefile

@@ -221,11 +221,17 @@ CFLAGS += -O2 -g
 # so they almost always use stack and need the frame pointer anyway.
 CFLAGS += -fno-omit-frame-pointer
 # Enable runtime stack canaries for functions to guard for buffer overflows.
+# FIXME(ilammy, 2020-10-29): enable stack canaries for WasmThemis too
+# Currently, stack protector is not supported by the "upstream" flavor
+# of Emscripten toolchain. Tracking issue is here:
+# https://github.com/emscripten-core/emscripten/issues/9780
+ifndef IS_EMSCRIPTEN
 ifeq (yes,$(call supported,-fstack-protector-strong))
 CFLAGS += -fstack-protector-strong
 else
 CFLAGS += -fstack-protector
 endif
+endif
 # Enable miscellaneous compile-time checks in standard library usage.
 CFLAGS += -D_FORTIFY_SOURCE=2
 # Prevent global offset table overwrite attacks.

src/wrappers/themis/wasm/src/secure_cell_context_imprint.js

@@ -73,7 +73,7 @@ module.exports = class SecureCellContextImprint {
 
         let status
         /// C API uses "size_t" for lengths, it's defined as "i32" in Emscripten
-        let result_length_ptr = libthemis.allocate(4, 'i32', libthemis.ALLOC_STACK)
+        let result_length_ptr = libthemis.allocate(new ArrayBuffer(4), libthemis.ALLOC_STACK)
         let master_key_ptr, message_ptr, context_ptr, result_ptr, result_length
         try {
             master_key_ptr = utils.heapAlloc(this.masterKey.length)
@@ -148,7 +148,7 @@ module.exports = class SecureCellContextImprint {
 
         let status
         /// C API uses "size_t" for lengths, it's defined as "i32" in Emscripten
-        let result_length_ptr = libthemis.allocate(4, 'i32', libthemis.ALLOC_STACK)
+        let result_length_ptr = libthemis.allocate(new ArrayBuffer(4), libthemis.ALLOC_STACK)
         let master_key_ptr, message_ptr, context_ptr, result_ptr, result_length
         try {
             master_key_ptr = utils.heapAlloc(this.masterKey.length)

src/wrappers/themis/wasm/src/secure_cell_seal.js

@@ -81,7 +81,7 @@ class SecureCellSeal {
 
         let status
         /// C API uses "size_t" for lengths, it's defined as "i32" in Emscripten
-        let result_length_ptr = libthemis.allocate(4, 'i32', libthemis.ALLOC_STACK)
+        let result_length_ptr = libthemis.allocate(new ArrayBuffer(4), libthemis.ALLOC_STACK)
         let master_key_ptr, message_ptr, context_ptr, result_ptr, result_length
         try {
             master_key_ptr = utils.heapAlloc(this.masterKey.length)
@@ -149,7 +149,7 @@ class SecureCellSeal {
 
         let status
         /// C API uses "size_t" for lengths, it's defined as "i32" in Emscripten
-        let result_length_ptr = libthemis.allocate(4, 'i32', libthemis.ALLOC_STACK)
+        let result_length_ptr = libthemis.allocate(new ArrayBuffer(4), libthemis.ALLOC_STACK)
         let master_key_ptr, message_ptr, context_ptr, result_ptr, result_length
         try {
             master_key_ptr = utils.heapAlloc(this.masterKey.length)
@@ -223,7 +223,7 @@ class SecureCellSealWithPassphrase extends SecureCellSeal {
 
         let status
         /// C API uses "size_t" for lengths, it's defined as "i32" in Emscripten
-        let result_length_ptr = libthemis.allocate(4, 'i32', libthemis.ALLOC_STACK)
+        let result_length_ptr = libthemis.allocate(new ArrayBuffer(4), libthemis.ALLOC_STACK)
         let master_key_ptr, message_ptr, context_ptr, result_ptr, result_length
         try {
             master_key_ptr = utils.heapAlloc(this.masterKey.length)
@@ -291,7 +291,7 @@ class SecureCellSealWithPassphrase extends SecureCellSeal {
 
         let status
         /// C API uses "size_t" for lengths, it's defined as "i32" in Emscripten
-        let result_length_ptr = libthemis.allocate(4, 'i32', libthemis.ALLOC_STACK)
+        let result_length_ptr = libthemis.allocate(new ArrayBuffer(4), libthemis.ALLOC_STACK)
         let master_key_ptr, message_ptr, context_ptr, result_ptr, result_length
         try {
             master_key_ptr = utils.heapAlloc(this.masterKey.length)

src/wrappers/themis/wasm/src/secure_cell_token_protect.js

@@ -66,8 +66,10 @@ module.exports = class SecureCellTokenProtect {
 
         let status
         /// C API uses "size_t" for lengths, it's defined as "i32" in Emscripten
-        let result_length_ptr = libthemis.allocate(4, 'i32', libthemis.ALLOC_STACK)
-        let token_length_ptr = libthemis.allocate(4, 'i32', libthemis.ALLOC_STACK)
+        /// allocate() with ALLOC_STACK cannot be called multiple times,
+        /// but we need two size_t values so allocate an array, of a sort.
+        let result_length_ptr = libthemis.allocate(new ArrayBuffer(2 * 4), libthemis.ALLOC_STACK)
+        let token_length_ptr = result_length_ptr + 4
         let master_key_ptr, message_ptr, context_ptr, result_ptr, result_length, token_ptr, token_length
         try {
             master_key_ptr = utils.heapAlloc(this.masterKey.length)
@@ -150,7 +152,7 @@ module.exports = class SecureCellTokenProtect {
 
         let status
         /// C API uses "size_t" for lengths, it's defined as "i32" in Emscripten
-        let result_length_ptr = libthemis.allocate(4, 'i32', libthemis.ALLOC_STACK)
+        let result_length_ptr = libthemis.allocate(new ArrayBuffer(4), libthemis.ALLOC_STACK)
         let master_key_ptr, message_ptr, context_ptr, token_ptr, result_ptr, result_length
         try {
             master_key_ptr = utils.heapAlloc(this.masterKey.length)

src/wrappers/themis/wasm/src/secure_comparator.js

@@ -106,7 +106,7 @@ class SecureComparator {
 
         let status
         /// C API uses "size_t" for lengths, it's defined as "i32" in Emscripten
-        let message_length_ptr = libthemis.allocate(4, 'i32', libthemis.ALLOC_STACK)
+        let message_length_ptr = libthemis.allocate(new ArrayBuffer(4), libthemis.ALLOC_STACK)
         let message_ptr, message_length
         try {
             status = libthemis._secure_comparator_begin_compare(this.comparatorPtr,
@@ -143,7 +143,7 @@ class SecureComparator {
 
         let status
         /// C API uses "size_t" for lengths, it's defined as "i32" in Emscripten
-        let reply_length_ptr = libthemis.allocate(4, 'i32', libthemis.ALLOC_STACK)
+        let reply_length_ptr = libthemis.allocate(new ArrayBuffer(4), libthemis.ALLOC_STACK)
         let request_ptr, reply_ptr, reply_length
         try {
             request_ptr = utils.heapAlloc(request.length)

src/wrappers/themis/wasm/src/secure_keygen.js

@@ -122,8 +122,10 @@ function generateECKeyPair() {
     var err
 
     // C API uses "size_t" for lengths, it's defined as "i32" on Emscripten
-    let private_len_ptr = libthemis.allocate(4, 'i32', libthemis.ALLOC_STACK)
-    let public_len_ptr = libthemis.allocate(4, 'i32', libthemis.ALLOC_STACK)
+    /// allocate() with ALLOC_STACK cannot be called multiple times,
+    /// but we need two size_t values so allocate an array, of a sort.
+    let private_len_ptr = libthemis.allocate(new ArrayBuffer(2 * 4), libthemis.ALLOC_STACK)
+    let public_len_ptr = private_len_ptr + 4
 
     err = libthemis._themis_gen_ec_key_pair(null, private_len_ptr, null, public_len_ptr)
     if (err != ThemisErrorCode.BUFFER_TOO_SMALL) {
@@ -179,7 +181,7 @@ function generateSymmetricKey() {
     var err
 
     // C API uses "size_t" for lengths, it's defined as "i32" on Emscripten
-    let key_len_ptr = libthemis.allocate(4, 'i32', libthemis.ALLOC_STACK)
+    let key_len_ptr = libthemis.allocate(new ArrayBuffer(4), libthemis.ALLOC_STACK)
 
     err = libthemis._themis_gen_sym_key(null, key_len_ptr)
     if (err != ThemisErrorCode.BUFFER_TOO_SMALL) {

src/wrappers/themis/wasm/src/secure_message.js

@@ -73,7 +73,7 @@ class SecureMessage {
 
         let status
         /// C API uses "size_t" for lengths, it's defined as "i32" in Emscripten
-        let result_length_ptr = libthemis.allocate(4, 'i32', libthemis.ALLOC_STACK)
+        let result_length_ptr = libthemis.allocate(new ArrayBuffer(4), libthemis.ALLOC_STACK)
         let private_key_ptr, public_key_ptr, message_ptr, result_ptr, result_length
         try {
             private_key_ptr = utils.heapAlloc(this.privateKey.length)
@@ -134,7 +134,7 @@ class SecureMessage {
 
         let status
         /// C API uses "size_t" for lengths, it's defined as "i32" in Emscripten
-        let result_length_ptr = libthemis.allocate(4, 'i32', libthemis.ALLOC_STACK)
+        let result_length_ptr = libthemis.allocate(new ArrayBuffer(4), libthemis.ALLOC_STACK)
         let private_key_ptr, public_key_ptr, message_ptr, result_ptr, result_length
         try {
             private_key_ptr = utils.heapAlloc(this.privateKey.length)
@@ -205,7 +205,7 @@ class SecureMessageSign {
 
         let status
         /// C API uses "size_t" for lengths, it's defined as "i32" in Emscripten
-        let result_length_ptr = libthemis.allocate(4, 'i32', libthemis.ALLOC_STACK)
+        let result_length_ptr = libthemis.allocate(new ArrayBuffer(4), libthemis.ALLOC_STACK)
         let private_key_ptr, message_ptr, result_ptr, result_length
         try {
             private_key_ptr = utils.heapAlloc(this.privateKey.length)
@@ -271,7 +271,7 @@ class SecureMessageVerify {
 
         let status
         /// C API uses "size_t" for lengths, it's defined as "i32" in Emscripten
-        let result_length_ptr = libthemis.allocate(4, 'i32', libthemis.ALLOC_STACK)
+        let result_length_ptr = libthemis.allocate(new ArrayBuffer(4), libthemis.ALLOC_STACK)
         let public_key_ptr, message_ptr, result_ptr, result_length
         try {
             public_key_ptr = utils.heapAlloc(this.publicKey.length)

src/wrappers/themis/wasm/src/secure_session.js

@@ -216,7 +216,7 @@ class SecureSession {
     connectionRequest() {
         let status
         /// C API uses "size_t" for lengths, it's defined as "i32" in Emscripten
-        let request_length_ptr = libthemis.allocate(4, 'i32', libthemis.ALLOC_STACK)
+        let request_length_ptr = libthemis.allocate(new ArrayBuffer(4), libthemis.ALLOC_STACK)
         let request_ptr, request_length
         try {
             status = libthemis._secure_session_generate_connect_request(
@@ -259,7 +259,7 @@ class SecureSession {
 
         let status
         /// C API uses "size_t" for lengths, it's defined as "i32" in Emscripten
-        let reply_length_ptr = libthemis.allocate(4, 'i32', libthemis.ALLOC_STACK)
+        let reply_length_ptr = libthemis.allocate(new ArrayBuffer(4), libthemis.ALLOC_STACK)
         let message_ptr, reply_ptr, reply_length
         try {
             message_ptr = utils.heapAlloc(message.length)
@@ -315,7 +315,7 @@ class SecureSession {
 
         let status
         /// C API uses "size_t" for lengths, it's defined as "i32" in Emscripten
-        let wrapped_length_ptr = libthemis.allocate(4, 'i32', libthemis.ALLOC_STACK)
+        let wrapped_length_ptr = libthemis.allocate(new ArrayBuffer(4), libthemis.ALLOC_STACK)
         let message_ptr, wrapped_ptr, wrapped_length
         try {
             message_ptr = utils.heapAlloc(message.length)
@@ -368,7 +368,7 @@ class SecureSession {
 
         let status
         /// C API uses "size_t" for lengths, it's defined as "i32" in Emscripten
-        let unwrapped_length_ptr = libthemis.allocate(4, 'i32', libthemis.ALLOC_STACK)
+        let unwrapped_length_ptr = libthemis.allocate(new ArrayBuffer(4), libthemis.ALLOC_STACK)
         let message_ptr, unwrapped_ptr, unwrapped_length
         try {
             message_ptr = utils.heapAlloc(message.length)

src/wrappers/themis/wasm/wasmthemis.mk

@@ -25,7 +25,12 @@ WASM_PRE_JS  = $(abspath $(WASM_PATH)/emscripten/pre.js)
 WASM_PACKAGE = $(BIN_PATH)/wasm-themis.tgz
 
 $(BIN_PATH)/libthemis.js: LDFLAGS += -s EXTRA_EXPORTED_RUNTIME_METHODS=@$(WASM_RUNTIME)
-$(BIN_PATH)/libthemis.js: LDFLAGS += -s RESERVED_FUNCTION_POINTERS=1
+$(BIN_PATH)/libthemis.js: LDFLAGS += -s ALLOW_TABLE_GROWTH
+# FIXME(ilammy, 2020-11-29): rely in EMSCRIPTEN_KEEPALIVE instead of LINKABLE
+# For some reason existing EMSCRIPTEN_KEEPALIVE macros do not work and without
+# LINKABLE flag wasm-ld ends up stripping *all* Themis functions from "*.wasm"
+# output, as if removed by dead code elimination.
+$(BIN_PATH)/libthemis.js: LDFLAGS += -s LINKABLE=1
 $(BIN_PATH)/libthemis.js: LDFLAGS += --pre-js $(WASM_PRE_JS)
 
 $(BIN_PATH)/libthemis.js: CMD = $(CC) -o $@ $(filter %.o %a, $^) $(LDFLAGS)