out of bounds memory access on aarch64 / others (2)

[kwonoj]

This is reopening #3764 (comment) that I don't have power to reopen existing case.

We upgraded wasmer to latest (3.3), and another case of out of bounds memory access on aarch64 with stacks:

1: RuntimeError: out of bounds memory access
           at memcpy (<module>[28580]:0x9e73bf)
           at <T as alloc::slice::hack::ConvertVec>::to_vec::hbdf0e7783569b0dc (<module>[27521]:0x9a0339)
           at alloc::slice::<impl alloc::borrow::ToOwned for [T]>::to_owned::h7785fa8682e1ebb7 (<module>[27614]:0x9a458b)
           at <str as alloc::string::ToString>::to_string::hc5e46221542e2147 (<module>[21591]:0x7f604b)
           at rkyv::impls::alloc::string::<impl rkyv::Deserialize<alloc::string::String,D> for rkyv::string::ArchivedString>::deserialize::h4cfb68af873c7ac3 (<module>[21510]:0x7f1fe3)
           at <swc_atoms::EncodeJsWord as rkyv::with::DeserializeWith<<alloc::string::String as rkyv::Archive>::Archived,string_cache::atom::Atom<swc_atoms::JsWordStaticSet>,D>>::deserialize_with::hbb70711b43875277 (<module>[10394]:0x3f3eeb)
           at rkyv::with::<impl rkyv::Deserialize<rkyv::with::With<T,W>,D> for F>::deserialize::h16e51719d7cd2dc8 (<module>[656]:0x249b3)
           at swc_ecma_ast::ident::_::<impl rkyv::Deserialize<swc_ecma_ast::ident::Ident,__D> for <swc_ecma_ast::ident::Ident as rkyv::Archive>::Archived>::deserialize::h81123a13599d367d (<module>[9161]:0x3381f2)
           at swc_ecma_ast::ident::_::<impl rkyv::Deserialize<swc_ecma_ast::ident::BindingIdent,__D> for <swc_ecma_ast::ident::BindingIdent as rkyv::Archive>::Archived>::deserialize::hf618eb15f2e65258 (<module>[9163]:0x33898c)
           at swc_ecma_ast::pat::_::<impl rkyv::Deserialize<swc_ecma_ast::pat::Pat,__D> for <swc_ecma_ast::pat::Pat as rkyv::Archive>::Archived>::deserialize::h53bfb2b1f134e37b (<module>[9079]:0x323972)
           at swc_ecma_ast::decl::_::<impl rkyv::Deserialize<swc_ecma_ast::decl::VarDeclarator,__D> for <swc_ecma_ast::decl::VarDeclarator as rkyv::Archive>::Archived>::deserialize::h02cbf2d5181dca2b (<module>[7228]:0x2092ba)
           at rkyv::impls::core::<impl rkyv::DeserializeUnsized<[U],D> for [T]>::deserialize_unsized::h83eaede211d8b669 (<module>[7261]:0x20fdd1)
           at rkyv::impls::alloc::vec::<impl rkyv::Deserialize<alloc::vec::Vec<T>,D> for rkyv::vec::ArchivedVec<<T as rkyv::Archive>::Archived>>::deserialize::he91ff48c8cfed6fa (<module>[7170]:0x1fb46f)
           at swc_ecma_ast::decl::_::<impl rkyv::Deserialize<swc_ecma_ast::decl::UsingDecl,__D> for <swc_ecma_ast::decl::UsingDecl as rkyv::Archive>::Archived>::deserialize::hcc528688d06237c4 (<module>[7227]:0x208ed5)
           at rkyv::impls::core::<impl rkyv::DeserializeUnsized<T,D> for <T as rkyv::Archive>::Archived>::deserialize_unsized::hb9162d1a044e1b57 (<module>[7247]:0x20d8cf)

This PR kwonoj/swc-plugin-coverage-instrument#215 is the offending one to try.

It may not be the wasmer is culprit, but I'm curious for the reason seeing similar errors each time trying to upgrade wasmer. Any ideas to make a preventive action on our end?

[kwonoj]

Actually this time x86 fails same https://github.com/kwonoj/swc-plugin-coverage-instrument/actions/runs/4950612331/jobs/8854459486 🤔

[theduke]

@kwonoj this looks like version mismatched of the artifacts.

Have you switched to the new, safe variants of artifact deserialization?
See https://docs.rs/wasmer/latest/wasmer/struct.Module.html#method.deserialize_checked

In general, you also need to include the Engine::deterministic_id and the metadata version header in artifact cache paths (see https://docs.rs/wasmer-types/latest/src/wasmer_types/serialize.rs.html#210 ).
I realize this isn't currently public, but will be public in the next release.

But using the checked deserializations from above should result in returned errors, in which case you can re-compile the artifact instead.

[kwonoj]

Have you switched to the new, safe variants of artifact deserialization?

No, I don't think we have enabled something similar yet and wanted to have these kind for a long time.

I realize this isn't currently public, but will be public in the next release.

Eager to try out - any eta for having new release?

[kwonoj]

One more thing: looks like deserialize_checked (or deserialize_from_file_checked) is not exposed via wasmer-cache's FileSystemCache?

[theduke]

One more thing: looks like deserialize_checked (or deserialize_from_file_checked) is not exposed via wasmer-cache's FileSystemCache?

Thanks for bringing that up.
We switched the CLI over to use the safe methods, but indeed forgot about the cache.

I will also switch the cache over to use always use the safe methods, because that's the sensible thing to do and the performance overhead is small.

[theduke]

Alright, so after a bit of reflection:

We currently do not plan for another release in the next few weeks, which might also contains some additional changes to the caching layers.

I opened a PR to switch the FilesystemCache to safe deserialization here: #3856 .

But if you don't want to wait (which is understandable), I would recommend just copying the FilesystemCache implementation using the checked_ variant - it's not a lot of code.

[ptitSeb]

@kwonoj is it working for you now?

[kwonoj]

I'm trying to wait for the new upstream release with FileSystemCache instead of copying it myself, so no, not haven't tried suggestions yet.

[ptitSeb]

Is this ticket still relevent @kwonoj ?

[kwonoj]

I haven't had chance to try checked version of load, afaik upstream filesystem cache doesn't have support yet? (

wasmer/lib/cache/src/filesystem.rs

Line 95 in 0ea42da

unsafe fn load(

)

[stale]

This issue has been automatically marked as stale because it has not had recent activity. It will be closed if no further activity occurs. Thank you for your contributions.

[stale]

Feel free to reopen the issue if it has been closed by mistake.