multiple wasmer processes created?

[matsbror]

Summary

I have built the command line tool from source and am trying to benchmark performance. Startup time seems to be very slow and increasing with the number of processors in the system. How can I force wasmer to only use one core?

Additional details

Provide any additional details here.

[Michael-F-Bryan]

This may be because we use a rayon threadpool, and on first use, that threadpool will start up a number of threads proportional to the number of cores on the machine. If so, you might be able to swap out the wasmer_wasix::runtime::task_manager::TokioTaskManager for a different implementation.

[hishamhm]

I seem to be running into the same issue — we're running Wasmer inside Nginx, which already employs a process pool model, and spawns N forks for N processors. Running Wasmer 4 is giving us N*N forks, with the complicating factor that the threads do not seem to be terminated when we do wasm_engine_delete (we're embedding via the Wasm C API so our customization options are limited).

I found this comment in the Rayon repository which looks concerning: "I suspect this is simply due to the fact that we never shutdown the global thread pool, by design. See also rayon-rs/rayon#688", and that Rayon issue is still open: "Consider adding an easier way of synchronously shutting down a thread-pool".

The end-result is that when I run our test suite, which starts a Wasm engine per test, I end up with thousands of threads spawned, which brings the system to a crawl and, most impactfully, breaks running tests under Valgrind: https://github.com/Kong/ngx_wasm_module/actions/runs/6164685265/job/16730923753?pr=413

This is preventing us from upgrading to the latest Wasmer (which we need to because of #4058).