Make a deterministic build system

[ethanfrey]

See CosmWasm/wasmd#268

Multiple compiles of the same code should give us the same dlls.

I think this requires removing all concurrency from compilation and a few other tricks... definitely increasing build times a lot, so it should be an option, not default. But we should do this for tagged releases, so we can get deterministic (and signable) go binaries

[webmaster128]

We have build containers now for the .dylib, .so and .a builds. Rust strives towards reproducible builds in general and is doing pretty good. I think the only thing that is really missing here is that we should not mount the host's target folder into the guest, which happens as part of -v $(shell pwd):/code. Instead we should have a guest-only target folder, as in

docker run --rm -v "$(pwd)":/code \
  --mount type=volume,source="$(basename "$(pwd)")_cache",target=/code/target \
  --mount type=volume,source=registry_cache,target=/usr/local/cargo/registry \
  cosmwasm/rust-optimizer:0.10.5

[yihuang]

https://github.com/yihuang/cosmwasm-plus-plus/blob/integration-tests/nix/default.nix
How about nix? I managed to build wasmd and wasmvm together with nix.

[ethanfrey]

I think the main issue is the compilation step of wasmvm returns a different dll each time. Maybe if we limit it to one compiler thread (and 4-8x slower), it will produce deterministic output, as with the smart contracts.

Nix is a good idea if we have dependency issues (like linking native C libs causing issues). It may also be a more efficient solution to the Docker files we use now.

[maurolacy]

I think the main issue is the compilation step of wasmvm returns a different dll each time.

Interesting. My take is that it should be possible to instruct the compiler / linker to produce deterministic output. Maybe with some compiler flags, disabling some optimizations, or something like that.

I can take a look at this at some point if you want.

[yihuang]

Is there an issue in the rust side? I was only aware of un-deterministic caused by working path, which should be fixed with option --remap-path-prefix.

[webmaster128]

My take is that it should be possible to instruct the compiler / linker to produce deterministic output.

I second that. For Rust, deterministic builds are considered standard and when builds are not deterministic, we can file this as a bug. It is well possible that this does not work here for the shared libs that are a bit exotic for Rust, but we should have all the community support to get this fixed.

[ethanfrey]

I believe it is due to multiple compiler threads and varying linking order.

I know I set codegen-units = 1 to get deterministic builds for the CosmWasm contracts (over a year ago)
https://github.com/CosmWasm/cosmwasm/blob/main/contracts/hackatom/Cargo.toml#L19

I have not tried building the wasmvm with that option and checking the sha256sum of the resultant DLLs. I would be interested in the results if anyone wants to try.

[maurolacy]

Some relevant links, after a quick search for "rustc deterministic":

• CI for deterministic / reproducible builds rust-lang/rust#75362
• https://internals.rust-lang.org/t/verifying-rustc-releases-with-reproducible-builds/4502/4
• Bit-for-bit deterministic / reproducible builds rust-lang/rust#34902
• https://users.rust-lang.org/t/testing-out-reproducible-builds/9758

Some info may be outdated. According to the last link, and particularly if we keep the same build system, deterministic builds are very likely.

[webmaster128]

This seems to work well for the .so and .dylib builds. But the .dll on Windows changes even without changes to the libwasmvm source code (ba3ecc0).