Support WASI #900

nhynes · 2019-09-18T06:53:20Z

No description provided.

josephlr · 2019-09-18T06:59:35Z

This should (at the very least) add stuff to the CI to make sure that ring builds and the tests pass when run on WASI.

nhynes · 2019-09-18T07:03:31Z

add stuff to the CI

I totally agree. I was actually just looking into doing this for both Wasm and WASI using Travis.

nhynes · 2019-09-18T16:30:51Z

NB: the newly added tests pass. CI appears to be failing because one of them is flaky.

Update: that one works after a rebuild. Now a different one is randomly failing.
Update: Now yet a different random one is failing.

josephlr · 2019-09-18T23:34:35Z

@nhynes I'm not sure why the arm tests are acting up, but we also need to actually run the tests on WASI. It looks like your current CI changes just build the code but don't execute it.

nhynes · 2019-09-19T03:56:22Z

Good point. Turns out that wasm32 won't work without somewhat involved runtime support until the GFp functions are moved to not-assembly.

briansmith · 2019-09-22T18:43:44Z

@nhynes I reopened this because I also want to make forward progress on WASI support.

It is true that some functionality won't be available yet in WebAssembly, but that should actually change soon.

In the meantime, there is a lot of functionality that is available already: basically ring::rand and all the stuff that is based on only digest functions. People are using the stuff that's working in wasm32 applications today.

How are you yourself running (not just building) the tests in WASI? And which specific WASI environment are you targetting?

(It looks like we don't have the wasm32-unknown-unknown tests running in CI in this repo now. I'll see what I can

nhynes · 2019-09-22T20:40:14Z

This is great news, thanks!

I'm compiling to a standard wasi-unstable environment to be used in a custom (blockchain) WASI runtime. I was trying to use ECDSA which is why lld couldn't prune the offending symbols.

Building the tests shows what's needed in not-asm:

First, you'll need the wasi-sdk to build the ring C extensions. Then compile using cargo:

$ RANLIB=/opt/wasi-sdk/bin/llvm-ranlib CC=/opt/wasi-sdk/bin/clang c t --no-run --target wasm32-wasi

Finally, running wasm-nm shows the required functions that either need to be implemented in not-asm or provided by the runtime:

$ wasm-nm -i target/wasm32-wasi/debug/ring-*.wasm
i GFp_p256_scalar_mul_mont
i GFp_p256_scalar_sqr_rep_mont
i GFp_nistz256_add
i GFp_nistz256_mul_mont
i GFp_nistz256_sqr_mont
i GFp_nistz256_point_add
i GFp_nistz256_neg
i GFp_nistz256_point_add_affine
i GFp_nistz256_point_double
i GFp_poly1305_emit
i GFp_poly1305_blocks
i GFp_poly1305_init_asm
i GFp_ChaCha20_ctr32
i GFp_bn_mul_mont
i GFp_gcm_init_clmul
i GFp_gcm_ghash_clmul
i GFp_gcm_gmult_clmul
i GFp_aes_hw_set_encrypt_key
i GFp_aes_nohw_set_encrypt_key
i GFp_aes_nohw_encrypt
i GFp_aes_hw_encrypt
i GFp_aes_hw_ctr32_encrypt_blocks
i GFp_nistz256_select_w5
i GFp_nistz256_select_w7

It goes without saying that getting the wasm32-unknown-unknown tests to run is more difficult, since compiling libring is more involved.

josephlr · 2019-09-23T05:56:57Z

For running wasm32 tests, the getrandom crate currently has CI for both wasm32-wasi and wasm32-unknown-unknown.

We use wasmtime for WASI and the wasm-bindgen tools to run wasm32-unknown-unknown tests. See our .cargo/config and CI script. Ignore any of the stuff about cargo web, it's not a platform ring wants to support (I think).

This lets us test the more conventional wasm32-unknown-unknown/wasm-bindgen targets on Node, Chrome, and Firefox, as well as testing on the non-Web, non-JS WASI.

Obviously this is easier for us, as we have no C code, but if you can figure out how to build ring, you should be able to use something like our setup.

dunnock · 2019-11-20T09:52:14Z

Actually after applying mentioned changes by @josephlr tests still do not run due to dependency on C library which is required for ring to work.. It was mentioned above but I thought it is due to non standard setup:
...ImportNotFound { namespace: "env", name: "GFp_aes_hw_encrypt" }, ImportNotFound { namespace: "env", name: "GFp_aes_hw_ctr32_encrypt_blocks" }, ImportNotFound { namespace: "env", name: "GFp_poly1305_emit" }, ImportNotFound { namespace: "env", name: "GFp_poly1305_blocks" }, ImportNotFound { namespace: "env", name: "GFp_poly1305_init_asm" }, ImportNotFound { namespace: "env", name: "LIMBS_add_mod" }, ImportNotFound { namespace: "wasi_unstable", name: "random_get" }...

dunnock · 2019-11-23T12:15:40Z

@nhynes @briansmith I still want to make it to WASI even with custom functions: at least it will allow to run TlsStream in a customized WASI runtime. Direct approach I am following so far is to compile test runner exporting C functions into WASI environment, so that tests compiled to wasm32-wasi target might call those functions via a wrapper over original export "C".

I started building WASI runtime, though there are few complications:

code duplication around exported functions and data, see one of function implementations https://github.com/dunnock/ring/blob/wasi-support/util/wasi/src/poly1305.rs due to types are private to module. What would help is making those types public, unless there is a reason they should stay private
code duplication would lead to incompatibility changes
wasiringruntime is separate crate to not pollute ring codebase, though since it links to same C libraries it heavily reuses build script and C artifacts, e.g. https://github.com/dunnock/ring/blob/wasi-support/util/wasi/build.rs

I want to start discussion and get your opinion before spending more time on this, is it feasible approach? Is there any better way to work around those complexities?

One of possible approaches would be to convert all crypto code to pure rust, though probably it won't provide required crypto warranties and is it too big effort for benefit?

Another possible options might be for wasi runtime would just export custom top level cryptography functions so that TlsStream would use those functions to make it work - what would be candidates?

Or probably WASI runtime should just export TlsStream Sink/Send...

briansmith · 2020-11-24T17:33:26Z

build.rs

@@ -489,6 +490,16 @@ fn build_library(
                .and_then(|f| f.to_str())
                .expect("No filename"),
        );
+
+        if &target.os == "wasi" {
+            let ranlib_cmd = std::env::var("RANLIB");


Please write a comment explaining why this chunk of code is here.

Wouldn't we need to output a cargo:rerun-if-env-changed=RANLIB here?

briansmith · 2020-11-24T17:35:31Z

include/GFp/base.h

@@ -87,6 +87,9 @@
 #elif defined(__mips__) && defined(__LP64__)
 #define OPENSSL_64_BIT
 #define OPENSSL_MIPS64
+#elif defined(__wasm32__)
+#define OPENSSL_32_BIT
+#define OPENSSL_X86_64


It definitely isn't right to define OPENSSL_X86_64 for WASI.

We already have:

#elif defined(__wasm__) #define OPENSSL_32_BIT

Is that not good enough? I think we can just revert this chunk.

briansmith · 2020-11-24T17:36:53Z

mk/travis.sh

@@ -102,6 +102,11 @@ else
  target_dir=target/$TARGET_X/debug
 fi

+if [[ "$TARGET_X" =~ "wasm32-" ]]; then
+  cargo build -vv -j2 ${mode-} ${FEATURES_X-} --target=$TARGET_X


I will leave a comment in the issue about the CI/CD.

briansmith · 2020-11-24T17:39:13Z

src/rand.rs

+
+    #[inline]
+    pub fn chunk(dest: &mut [u8]) -> Result<usize, error::Unspecified> {
+        wasi::wasi_unstable::random_get(dest).map_err(|_| error::Unspecified)?;


It seems like, rather than adding that heavyweight dependency, we should just call __wasi_random_get directly and avoid that crate.

briansmith · 2020-11-24T22:40:46Z

My plan is to, soon, have C or Rust implementations of all features of ring, so that all ring features will work on WebAssembly, MIPS, etc.

I understand that right now some people are working around the functions that ring is currently missing by substituting their own. This will break as we add the C/Rust versions of those functions to ring. Also, there are other changes in the pipeline that would break such an approach. And, such an approach has never been guaranteed to work; we frequently change the "ABI" of the internal functions of ring, so it's pretty unreasonable to plug in implementations from the outside.

briansmith · 2021-10-06T17:59:14Z

@nhynes Thanks for this PR. It seems to have been dormant for a long time and needs a lot of changes to rebase on top of recent changes. I will leave a comment in Issue #1043 about recent progress and status regarding WASI support in ring.

nhynes force-pushed the wasi branch from fab3fb8 to 49712fa Compare September 18, 2019 07:53

nhynes force-pushed the wasi branch from 57739d1 to c164393 Compare September 18, 2019 18:38

nhynes closed this Sep 19, 2019

briansmith reopened this Sep 22, 2019

nhynes force-pushed the wasi branch from 4b143fe to 01cd007 Compare September 22, 2019 20:39

tomaka mentioned this pull request Mar 14, 2020

Restore WASI support tomaka/redshirt#165

Closed

nhynes added 4 commits May 2, 2020 03:08

Support WASI

0a181c7

Add wasm32 to CI

0a4957a

Trigger rebuild

edd0434

Build libring for wasi

fc8edaa

nhynes force-pushed the wasi branch from 01cd007 to fc8edaa Compare May 2, 2020 03:10

Geal mentioned this pull request Aug 18, 2020

wasm32-wasi doesn't build #1043

Closed

briansmith requested changes Nov 24, 2020

View reviewed changes

briansmith closed this Oct 6, 2021

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Support WASI #900

Support WASI #900

nhynes commented Sep 18, 2019

josephlr commented Sep 18, 2019

nhynes commented Sep 18, 2019 •

edited

Loading

nhynes commented Sep 18, 2019 •

edited

Loading

josephlr commented Sep 18, 2019

nhynes commented Sep 19, 2019

briansmith commented Sep 22, 2019

nhynes commented Sep 22, 2019 •

edited

Loading

josephlr commented Sep 23, 2019 •

edited

Loading

dunnock commented Nov 20, 2019 •

edited

Loading

dunnock commented Nov 23, 2019

briansmith Nov 24, 2020

briansmith Nov 24, 2020

briansmith Nov 24, 2020

briansmith Nov 24, 2020

briansmith commented Nov 24, 2020

briansmith commented Oct 6, 2021

Support WASI #900

Support WASI #900

Conversation

nhynes commented Sep 18, 2019

josephlr commented Sep 18, 2019

nhynes commented Sep 18, 2019 • edited Loading

nhynes commented Sep 18, 2019 • edited Loading

josephlr commented Sep 18, 2019

nhynes commented Sep 19, 2019

briansmith commented Sep 22, 2019

nhynes commented Sep 22, 2019 • edited Loading

josephlr commented Sep 23, 2019 • edited Loading

dunnock commented Nov 20, 2019 • edited Loading

dunnock commented Nov 23, 2019

briansmith Nov 24, 2020

Choose a reason for hiding this comment

briansmith Nov 24, 2020

Choose a reason for hiding this comment

briansmith Nov 24, 2020

Choose a reason for hiding this comment

briansmith Nov 24, 2020

Choose a reason for hiding this comment

briansmith commented Nov 24, 2020

briansmith commented Oct 6, 2021

nhynes commented Sep 18, 2019 •

edited

Loading

nhynes commented Sep 18, 2019 •

edited

Loading

nhynes commented Sep 22, 2019 •

edited

Loading

josephlr commented Sep 23, 2019 •

edited

Loading

dunnock commented Nov 20, 2019 •

edited

Loading