Test and document non-nix build

[TerrorJack]

Closes #11.

[amesgen]

Re https://github.com/tweag/ghc-wasm-miso-examples/actions/runs/8742531268/job/23991017268?pr=12:
I have seen deno hash failures before (https://gitlab.haskell.org/ghc/ghc-wasm-meta/-/merge_requests/54#note_553186), they seem to update release artifacts in-place...

[TerrorJack]

Maybe it's not that simple; I bumped ghc-wasm-meta and deno hash check is failing again. It's simple to get rid of deno in ghc-wasm-meta but I'd really like to figure out what the heck has gone wrong here..

[TerrorJack]

@amesgen I ran ./autogen.js in a clean nixos/nix:2.21.2 container and there's a hash change, but not in 2.20.6. So it's highly likely a nix bug that introduced builtins.fetchTarball hash change in 2.21 and what is worse, I've lazily marked the nix job as fragile and have not noticed the mismatch issue for quite a while.......

[amesgen]

Wow, that's surprising. I bisected this in Nix, and it is due to NixOS/nix#10038 (cabee98 is the first bad commit).

Maybe the problem is that https://github.com/denoland/deno/releases/download/v1.42.1/deno-x86_64-unknown-linux-gnu.zip is not a tarball, but a zip file? According to the docs of builtins.fetchTarball:

The file must be a tape archive (.tar) compressed with gzip, bzip2 or xz.

In any case, https://gitlab.haskell.org/ghc/ghc-wasm-meta/-/merge_requests/60 looks good as a solution 👍

[TerrorJack]

My understanding of builtins.fetchTarball is it's capable of handling any archive format supported by libarchive, but I've now been corrected the hard way. Anyway I've removed allow_failure: true in the nix job so at least this shouldn't go unnoticed again..

[TerrorJack]

@hasufell I've now added CI check as well as build instructions for non-nix users in the README. Would you mind giving it another try?

[hasufell]

wasm32-wasi-cabal does not exist when installing via ghcup. I changed it to:

cabal build --with-ghc=wasm32-wasi-ghc --with-ghc-pkg=wasm32-wasi-ghc-pkg --with-hsc2hs=wasm32-wasi-hsc2hs build ghc-wasm-miso-examples

And then got the error:

$ ./build.sh
Building for dev
Downloading
https://ghc.gitlab.haskell.org/head.hackage/package/text-short-0.1.5.tar.gz
Downloading
https://ghc.gitlab.haskell.org/head.hackage/package/free-5.2.tar.gz
Downloading
https://ghc.gitlab.haskell.org/head.hackage/package/lens-5.2.3.tar.gz
Error: cabal: Cannot build the package build, it is not in this project
(either directly or indirectly). If you want to add it to the project then
edit the cabal.project file.

[hasufell]

And then got the error:

That was a mistake on my part. With this patch I'm able to run it:

diff --git a/frontend/build.sh b/frontend/build.sh
index f3fb6e3..0c1cca1 100755
--- a/frontend/build.sh
+++ b/frontend/build.sh
@@ -19,7 +19,7 @@ rm -rf dist
 mkdir dist
 cp ./*.html dist/

-wasm32-wasi-cabal build ghc-wasm-miso-examples
+cabal build --with-ghc=wasm32-wasi-ghc --with-ghc-pkg=wasm32-wasi-ghc-pkg --with-hsc2hs=wasm32-wasi-hsc2hs ghc-wasm-miso-examples
 hs_wasm_path=$(find .. -name "*.wasm")

 "$(wasm32-wasi-ghc --print-libdir)"/post-link.mjs \

So we need some ghcup compatibility and make minimal assumptions.

[TerrorJack]

So we need some ghcup compatibility and make minimal assumptions.

I can change build.sh to handle absense of wasm32-wasi-cabal, since it's a minimal wrapper around cabal anyway. However, other than passing --with-prog=..., wasm32-wasi-cabal also adds CABAL_DIR to specify a different cabal directory from the default location, to prevent cabal from confusing host/wasm cabal stores when ghc version number happens to match.

How does ghcup handle this case and how should CABAL_DIR be set when the wasm toolchain is managed by ghcup, do you have some suggestion here?

[hasufell]

to prevent cabal from confusing host/wasm cabal stores when ghc version number happens to match

That's a cabal bug. ghcup does nothing about that.

[TerrorJack]

build.sh now falls back to passing --with-prog flags to cabal when wasm32-wasi-cabal is not present, I believe that should make the build script work for wasm toolchains managed by ghcup.

[amesgen]

Given that build.sh now also has support for the ghcup way of install the WASM backend (ie when wasm32-wasi-cabal is not present), we could also link to the corresponding ghcup instructions. I just tested that everything works when using ghcup with wasm32-wasi-ghc-9.10.0.20240412, see #10 (comment).

[TerrorJack]

Maybe. Though it looks like the wasm bindist metadata is maintained manually, I'd really like to have a proper CI story to update that before recommending ghcup to manage the wasm ghc bindists.

[hasufell]

I'd really like to have a proper CI story to update that before recommending ghcup to manage the wasm ghc bindists.

What does "proper CI story" mean?

Unless you're proposing some "nightly" style thing, the proper way of doing redistribution is doing manual updates, including working gpg signatures (not machine keys).

I don't see a problem with that.

[TerrorJack]

Unless you're proposing some "nightly" style thing

Yeah, though rather weekly than nightly, I was thinking that it would be nice to have weekly CI of https://github.com/amesgen/ghc-wasm-bindists to push updated bindist artifacts to ghcup cross yaml metadata, rather than having to update manually.

But that's indeed a separate concern worth discussion elsewhere. I'll update README to include ghcup instructions later.

[TerrorJack]

I have updated README to point to ghcup user manual about setting up wasm toolchain, and also added a CI job to test ghcup builds.

[hasufell]

to push updated bindist artifacts to ghcup cross yaml metadata

That won't be possible. Machine generated metadata has to be kept strictly separate. So that would mean the wasm project hosts their own yaml metadata of ghcup.

That is totally fine.

You might be interested in GHCs scripts: https://gitlab.haskell.org/ghc/ghc/-/tree/master/.gitlab/rel_eng/mk-ghcup-metadata?ref_type=heads