Serialized module stability

[webmaster128]

Summary

In one of our use case we compile and serialize modules with Wasmer 2.0.0 and derialize and run it with Wasmer 2.1.1. This leads to crashes at runtime. As soon as the cache is cleared between the two versions, everything works smoothly.

I digged a bit in the code and have two questions:

1. Do I understand correctly that the commit a603c33 is breaking the module format because SerializableModule changed?
2. Is there any kind of stability guarantee that Wasmer can give users? Would it be possible to expose a module serialization number to let callers know when the format changed?

Additional details

• Using the Universal engine with Singlepass.
• We have a module serialization version already that we could use on our side if we know a breaking change happened (https://github.com/CosmWasm/cosmwasm/blob/66b987766423c154cb0e99a7e28157f7ab3d27cf/packages/vm/src/modules/file_system_cache.rs#L10-L17).

[Amanieu]

1. In Use a standard header for metadata in all serialized modules #2747 we added version checking when loading serialized module, so incompatible modules simply fail to load with an error. This change is not included in 2.1.1 but is in 2.2.0-rc1.
2. As a general rule serialized modules are only compatible with the exact same version of Wasmer that created them.

[webmaster128]

Thank you for the quick response and update. This solves a large part of the problem: random crashes are converted into load time errors.

Now for the context of blockchain, it would be very unfortunate if we had to recompile all the program for every Wasmer bump. Especially when the caller of Wasmer is a library, not an application we can have users thet upgrade Wasmer without us knowing.

Would it be possible as an addition to #2747 to export the version as a compile time constant? This would allow me to do something like this:

// TODO: convert to compile time error once available in Rust
assert_eq!(MetadataHeader::CURRENT_VERSION, 1, "Wasmer module format changed. Please update the expected version accordingly and bump MODULE_SERIALIZATION_VERSION.");

[webmaster128]

In https://github.com/CosmWasm/cosmwasm/blob/7c1519c82e024a8860f01b4a7eb3dfe9295d1371/packages/vm/src/modules/versioning.rs#L14-L39 I implemented a workaround for the missing API that I'm very happy with. It's a bit complicated to compile an emty module to read the version number from the module serialization, but it does the job. At the same time it shows which API we'd need to create one directory per module version, such that re-compiles only need to be done when the version actually changed.

[epilys]

This is tracked in #2916.