Note an exception to the platform.os recommendation for wasi

[imjasonh]

As discussed in today's call.

Wasm (WebAssembly) is an exciting new space, and the Wasm community has smartly chosen to adopt OCI as a packaging and distribution format for Wasm applications.

Unfortunately, today, they mainly build applications using docker build and Dockerfiles where .wasm files are COPYed onto scratch base images, resulting in images whose configs report them as linux/amd64 images by default.

If Wasm applications want to be distributed alongside images for other platforms, especially alongside actual linux/amd64 images, this may be problematic.

Unfortunately, OCI's guidance around selecting a platform to report is lacking for this use case. Repurposing Go's set of supported GOOS and GOARCH values is insufficient here, since Go currently doesn't have good support for Wasm outside of JS-adjacent use cases. In the years since Go added support for JS/Wasm, Wasm has "escaped" the browser, and is now used directly on some platforms, aided by WASI, the WebAssembly System Interface. Regular Go tooling has no support for Wasi today -- see discussion here: golang/go#31105 -- though a separate project, TinyGo, does support Wasm and WASI in constrained scenarios.

This PR proposes defining an exception to the recommendation to use an appropriate GOOS value, when the Wasm application relies on WASI.

Diverging from Go here provides some small opportunity for trouble for us down the road:

1. Go could add official support for WASI, but choose to do so using a GOOS value that isn't wasi -- this would confuse our guidance more, since there'd be a valid GOOS for WASI that isn't our recommended .platform.os value.
2. Go could add support for something that isn't WASI, but choose to do so using the GOOS value wasi -- this would confuse our guidance since wasi in OCI-land would mean one thing, and wasi in Go-land would mean something else.

Either of these scenarios seem unlikely, and we'd have some options in either case to avoid confusion. Trying to plan for infinite scary futures is hard.

Having this expected behavior defined would help Wasm image builders and platforms agree on shared values to interoperate better.

Alternatively, we could just do nothing for now, and recommend the Wasm community build using os=wasi "off the record", gathering feedback and watching the community until their behavior settles. Maybe Go will have WASI support by then, alleviating some potential risks above.

cc @squillace @devigned

[sudo-bmitch]

Overall I'm in favor. Before putting my own LGTM on this one, I'm looking for:

1. Something from the Go team (link to past discussions is good) talking about their decision of the GOOS variable for WASM so we know we aren't stepping on an intentional reason to avoid GOOS=wasi.
2. An example image to test tooling against.
3. Give those that work with different tooling (runtimes, registries, etc) a bit of time to weigh in.

[SteveLasker]

Isn't the WASI another great example of a new artifactType?
Seems we'd want to maintain the OS type, where WASI isn't an OS, but rather a type, similar to what the singularity folks did.

[sudo-bmitch]

I've seen other examples with unikernels where overloading the OS makes sense. It is a runable image (with the right runtime) and can be listed in an index of images, so I see a lot of value to setting the platform on an image. Is there more on artifactType that would compel that model?

[SteveLasker]

What's interesting here is wasm may use a config, whereas many other artifact types don't.

I think we want to make it super obvious to the various runtimes that this is NOT a container image, so don't try to load it. It's not a Helm chart, so don't try and process it as helm. Only runtimes that know how to load a wasm should move forward attempting to process it. And, scanners should know how to scan it, uniquely as a WASM, as opposed to all the other artifact types.

So, aligning with the new OCI Artifact work, I'd suggest:
"artifactType": "application/vnd.bca.wasm"
and
"config.mediaType": "application/vnd.bca.wasm.config.v1+json"
Within the config, use the platform/architecture values, and other config values that make sense.

EDIT:
...and os=wasi makes sense in the WASM config object.

[imjasonh]

Overall I'm in favor. Before putting my own LGTM on this one, I'm looking for:

1. Something from the Go team (link to past discussions is good) talking about their decision of the GOOS variable for WASM so we know we aren't stepping on an intentional reason to avoid GOOS=wasi.

golang/go#31105 is the closest I've been able to find here.

2. An example image to test tooling against.

You're in luck!

I've been hacking on a tool today to build these according to this proposed spec change: https://github.com/imjasonh/wasimg

$ go run ./ fake.wasm gcr.io/jason-chainguard-public/wasimg
gcr.io/jason-chainguard-public/wasimg@sha256:b44bb1118f59398f76542c1d9beef95ab5f0408374e54e26822b239b5368aca2

3. Give those that work with different tooling (runtimes, registries, etc) a bit of time to weigh in.

+1

[imjasonh]

Re: artifacts

I'll mostly defer to Wasm folks about how they want to package and distribute their applications. If they'd prefer to package in OCI Image manifests with a config that says os=wasi, architecture=wasm, then this PR satisfies that.

If they'd prefer to package in an OCI Artifact manifest, we should try to help give guidance about how best to do that -- it won't work in all registries today, but perhaps that community is comfortable with that limitation. This does leave the open question of how they specify any variants (e.g., spin vs slight), but maybe artifactType fits there too. I'll defer to the Wasm community's preferences, with OCI's guidance.

We do agree that the current state of packaging in a Docker-media-typed image that reports as linux/amd64 is likely not the best path to be on, and we should encourage folks to get off it as soon as possible, before inertia takes over.

[SteveLasker]

Thanks @imjasonh, just a clarification, I wasn't suggesting using the new Artifact manifest, for the exact points you mentioned. I was referring to the OCI Artifact (non manifest) for differentiating the type. I thought we made an addition to image manifest to also support artifactType and the config.mediaType to enable differentiation, but I just double checked and realized artifactType is only on the new artifact manifest.

The real point I was making was differentiating a WASM from a container image. Using a different config.mediaType you can accomplish this, and setting the os=wasi to provide for the abstraction.

[imjasonh]

The real point I was making was differentiating a WASM from a container image. Using a different config.mediaType you can accomplish this, and setting the os=wasi to provide for the abstraction.

Here is where I'd like Wasm folks to tell us what they want, instead of just dictating it from OCI-land.

Today they're packaging using linux/amd64 Docker images, and we all agree that's wrong. They can continue to package in OCI image manifests with help from this PR, and retain the ability to push to any conformant OCI registry. If the Wasm community would prefer to move to OCI Artifacts instead, they can, and we can help guide them, but there are other downsides.

In any case, it's worth noting that the benefits of the reference types work should be able to help them attach SBOMs, signatures, ice cream flavors, whatever, to their Images-or-Artifacts, which is TBH really the part I'm most interested in exploring personally.

[squillace]

We do agree that the current state of packaging in a Docker-media-typed image that reports as linux/amd64 is likely not the best path to be on, and we should encourage folks to get off it as soon as possible, before inertia takes over.

OH yes. we're already figuring out how that projection happens -- turns out it's a default docker build effect, but we can specify this platform using buildx so we're already repushing with wasi/wasm. So in the shorter time frame, this will work fine for us in this specific case and for everyone in the ecosystem to not have docker go bonk with no reason why. So that's good!

As a result, here's my take:

1. Yes, we should use wasi/wasm platform information in order to inform current tools that this is not a platform they understand OR that it takes the responsibility to understand it. I'm totes fine with that. And that will work with registries that don't do artifacts yet (Docker Hub is the main one here). So, fine! we can do this we already have with our images that do this.
2. All future work in this ecosystem will require directed graphs, and OCI artifacts is this path (wasm-to-oci takes this route and that's the core tool for almost the entire server-side wasm ecosystem and it is also the work being done with Wasm components in the BCA) so within the year, my ecosystem expectation is that OCI artifacts will fully dominate.
3. I do have to say, however, that regarding platforms stuff and oci, no one cares about what GO does or does not do, so while that might be an implementation detail, I could not be more uninterested than I am about what go supports. I'm sure there's a reason for this in this discussion, but it has little to do with metadata for images itself.

Does this help? I'm fine with ensuring there's a useful exception for this case in the shorter term while this area fleshes itself out.

So I'm good with that aspect of the platform exception note.

[squillace]

just background information: something like what https://github.com/engineerd/wasm-to-oci#how-does-this-work does is what is the sweet spot here.

[imjasonh]

2. All future work in this ecosystem will require directed graphs, and OCI artifacts is this path (wasm-to-oci takes this route and that's the core tool for almost the entire server-side wasm ecosystem and it is also the work being done with Wasm components in the BCA) so within the year, my ecosystem expectation is that OCI artifacts will fully dominate.

That sounds good. Just to clarify, there's nothing about packaging the Wasm module in an OCI Image (not Artifact) that precludes it from being a member of a DAG of things. Artifacts (and Images) can point to Images using subject today, or more likely the fallback tag scheme defined in the spec.

3. I do have to say, however, that regarding platforms stuff and oci, no one cares about what GO does or does not do, so while that might be an implementation detail, I could not be more uninterested than I am about what go supports. I'm sure there's a reason for this in this discussion, but it has little to do with metadata for images itself.

+1

Drafting off of Go's conventions was convenient, but this example highlights the shortcomings.

[squillace]

@imjasonh yes, we can do the dag that way in the future, too, and yes the images can be a part of the dag as well in different ways. And thanks about the go stuff; that makes total sense.

[flouthoc]

+1 to this PR, as of now crun's implementation of running webassembly workload relies on a hack where it identifies image based on a annotation module.wasm.image/variant=compat, I think if this gets accepted it can help unifying standards for distributing OCI image containing webassembly workloads.

FWIW I also think that wasi also deserves a separate config block in runtime-spec but I am not entirely sure about it and that is a topic for different PR. :)

[giuseppe]

Diverging from Go here provides some small opportunity for trouble for us down the road:

1. Go could add official support for WASI, but choose to do so using a GOOS value that isn't wasi -- this would confuse our guidance more, since there'd be a valid GOOS for WASI that isn't our recommended .platform.os value.
2. Go could add support for something that isn't WASI, but choose to do so using the GOOS value wasi -- this would confuse our guidance since wasi in OCI-land would mean one thing, and wasi in Go-land would mean something else.

I think extending these fields is a good idea. Should in the WebAssembly case be platform=wasm so the os field can be left to specify the wasi interface if there is need to?

[sudo-bmitch]

1. Something from the Go team (link to past discussions is good) talking about their decision of the GOOS variable for WASM so we know we aren't stepping on an intentional reason to avoid GOOS=wasi.

golang/go#31105 is the closest I've been able to find here.

Rereading that issue a few times, I'm not sure that wasi/wasm is diverging from GOOS/GOARCH, so much as getting a step ahead of the Go development. If that's the case, I wouldn't even call this an exception, but just a note on the forward-looking usage of that field.

Regardless of how we phrase it, I hope the WASM devs have enough direction in this issue to continue their work with OCI images.

no one cares about what GO does or does not do

My concern is less current uses and more future compatibility. A sizeable chunk of containers are built using Go. There is a lot of code already out there that splits the platform string and assumes it is GOOS/GOARCH, sets the variables, and runs a Go cross compile to generate binaries for other platforms. My goal is to avoid breaking code like that. In the short term I think we are safe from that since I don't think many people are using Go for wasm containers. And in the long term it looks like Go will use these values.

[squillace]

sounds fine to me, @sudo-bmitch. My point is that I don't care about any particular language, so it is just an irrelevant point to me. But if the platform value we choose here is good for any or all of them, that's a good thing. :-)

[imjasonh]

Docker announced support for wasm in technical preview: https://www.docker.com/blog/docker-wasm-technical-preview/

Relevant to this discussion:

docker run -dp 8080:8080 --name=wasm-example --runtime=io.containerd.wasmedge.v1 --platform=wasi/wasm32 michaelirwin244/wasm-example

--platform=wasi/wasm32 – This specifies the architecture of the image we want to use. By leveraging a Wasm architecture, we don’t need to build separate images for the different architectures. The Wasm runtime will do the final step of converting the Wasm binary to machine instructions.

edit to clarify: I think we should see if we can get Docker to change this to wasi/wasm. I'd argue that the precedent is that 32-bit runtimes, e.g., arm, are just arm, and the 64-bit variant is arm64. If wasm is currently 32-bit, it should be wasm.

Of course, now that it's out there, it may be hard to change, and it'll get harder the longer it's out there.

[imjasonh]

cc @chris-crone

I saw your talk at Cloud Native Wasm Day about building and distributing Wasm applications using Docker's tooling: https://www.youtube.com/watch?v=3j915xoDovs -- how hard would it be at this point to change tooling for wasm images to use the platform wasi/wasm, instead of wasi/wasm32?

Even if it's too late and we decide to stick with wasi/wasm32, OCI will need to document this as an exception to the current spec, since there's no GOOS/GOARCH that corresponds to those values.

[chris-crone]

@imjasonh chatting internally at Docker with @justincormack, @tianon, @tonistiigi, @milosgajdos we think that wasi/wasm makes sense to use going forward.

The argument that we don't append 32 for other 32-bit architectures resonates with us.

I had chosen wasm32 for the demo as that was what Rust did for its target and I didn't see a reason not to be precise.

[imjasonh]

Thanks @chris-crone that's great to hear! I think with that detail sorted we should proceed with locking this into the spec.

Can I get a vote @sajayantony @sudo-bmitch @vbatts @jonjohnsonjr

[sajayantony]

@vbatts would be good to get your eyes on this. If there are no concerns then we can get this merged.

[vrothberg]

@flouthoc @giuseppe @rhatdan PTAL

I think the change makes sense.
LGTM from my end

[giuseppe]

LGTM

[flouthoc]

LGTM

[imjasonh]

FYI there is a new proposal to support WASI as a Go compilation target, where GOOS=wasi.

If this proposal is accepted and implemented with GOOS=wasi -- this does not appear to be a controversial part of this proposal -- then we can remove the exception in image-spec added by this PR, since the preceding sentence about using GOOS values holds.

[squillace]

Well, that's kinda nice. :-)

[squillace]

I'm certainly excited about the Go proposal!

[imjasonh]

An update on golang/go#58141 for those not following: it now sounds like they'll have GOOS=wasip1 for "wasi preview 1", which is expected to be followed by "wasi preview 2" (GOOS=wasip2) and eventually (years) an LTS release called wasi.

I don't know if there's anything we need to do on this image-spec issue. As written today, all of those would roll up into the OCI platform wasi/wasm, and something somewhere would have to determine whether the wasm blob inside the image is p1, p2, LTS, etc.

I don't know enough about how the wasm community plans to use OCI to know whether this will be disruptive -- is detecting this hard? Do we expect multi-platform images to contain variants for wasip1 and wasip2 in the same index manifest? Another option would be to wait for 58141 to be finalized and change our recommendation to prefer wasip1/wasm, wasip2/wasm, etc.

[squillace]

Hi @imjasonh , I'm of course watching that work with anticipation. Hard to see the variants as a registry issue, off the top of my head, and hence a spec issue: they're all wasi, just versions of pre-1.0. I'd expect to be able to push and pull but blow on execution with the proper exception.

Should we be treating this the same way as, for example, aarch64/armv7/armv8 and so on? I could see that after 1.0, but hey, there it is. I like what go is doing, but I fail to see this as anything but a oci dist registry implementation detail. I'm surely overlooking something.

[tianon]

The discussion on the Go issue makes it sound like p1 and p2 will be very incompatible at runtime; would there be a publisher use case for compiling and publishing software for both? (Thus needing an image index to somehow be able to differentiate them and letting the runtimes pick the one they can run?)

If so, one option (besides adjusting the os value) could be os.version (since unlike Go, we already have a precedent for OS variant), but if this situation with incompatibility is supposed to calm down after 1.0 then maybe that's premature.

(No strong opinions and not officially an image spec maintainer, just throwing out an idea to try to help clarify 😇)

[tianon]

(The variants of arm and the handling of OS version are defined by this spec, not the registry, so definitely relevant here 👍)

[squillace]

The discussion on the Go issue makes it sound like p1 and p2 will be very incompatible at runtime; would there be a publisher use case for compiling and publishing software for both? (Thus needing an image index to somehow be able to differentiate them and letting the runtimes pick the one they can run?)

From my perspective, they ARE incompatible at runtime, yes, but are semantically the same thing and I would expect runtimes to handle this difference as there are no major versions yet. I would not expect registries to treat them differently until major versions. Like, for example, armv7/armv8 and so on.

If so, one option (besides adjusting the os value) could be os.version (since unlike Go, we already have a precedent for OS variant), but if this situation with incompatibility is supposed to calm down after 1.0 then maybe that's premature.

Yup, but my take -- also not an OCI maintainer, just a constant user :-) and an Azure Container Registry feature PM on the side -- is that I wouldn't be implementing a service feature or a client tool feature based on the versions specified this early in the game. I'd expect the users of preview modules to understand that blowups are part of the work.

Starting with 1.0, I'd DEFINITELY be into versioning here, including os.version. If I understand things correctly, always in doubt.

[vbatts]

omg y'all open a new issue and don't converse on a closed/merged issue/PR

[squillace]

I like that you think I have some idea what I'm doing. :-)

[codefromthecrypt]

by the way, rust and v8 are beginning to enforce a wasi version similar, if not exactly the same as the Go discussion. I think we shouldn't really keep "wasi" knowing this, especially as the whole thing is new might as well do it right.

e.g. wasip1 like go (previously discussed here), or wasi-preview1 en route in rust.

[codefromthecrypt]

opened an issue to help this important discussion along to some action of some kind #1053