运行几次后总会随机出现错误:out of bounds memory access

[kamylee]

我在wasm中使用了thread，运行几次后总会随机出现错误:out of bounds memory access
我在wasm app中创建了一个线程A用来做http server. 另外又创建了一个线程B pthread_create，用来与外部的websocket服务器通信，线程A一直运行稳定，最近加了线程B后总是随机出现out of bounds memory access的错误。
线程B中用到了native本地提供的api，里面用到了WebSocket，
线程B中用到了jsoncpp库，用来解析Json数据，

运行环境：windows
总是在正常接收几次数据后随机出现：out of bounds memory access的错误，这是wasi线程的问题吗？
频繁调用wasm_runtime_module_malloc wasm_runtime_module_free 就容易出错

CALL_STACK：
#00: 0xd48a7bf4 - $f3908
#1: 0x0007 - free
#2: 0x0036 - $f3898
#3: 0x0036 - $f274
#4: 0x003f - $f272
#5: 0x00cd - $f269
#6: 0x0064 - $f1068
#7: 0x005e - $f1064
#8: 0x0113 - $f1059
#9: 0x0045 - $f851
#10: 0x0038 - $f780
#11: 0x0151 - $f778
#12: 0x0036 - $f777
#13: 0x004d - $f839
#14: 0x0038 - $f1070
#15: 0x0036 - $f1067
#16: 0x003f - $f1063
#17: 0x00f2 - $f1059
#18: 0x008e - $f1059
#19: 0x008e - $f1059
#20: 0x008e - $f1059
#21: 0x0045 - $f851
#22: 0x0038 - $f780
#23: 0x0151 - $f778
#24: 0x0036 - $f777
#25: 0x0a9a - $f1986
#26: 0x08f5 - $f1977
#27: 0x6785 - $f1971
#28: 0x372c - $f1840
#29: 0x0625 - $f1827
#30: 0x1fd0 - $f1826
#31: 0x589d - $f1825
#32: 0x009e - $f1305

#00: 0xfff5a4de - free

[wenyongh]

Hi, is thread B created by thread A (e.g. wasm app calls pthread_create) or created by host native itself? And how do you compile you wasm application, do you refer to the this document?

Note that if Build with libc-WASI, there may be two choices (1) disable malloc/free functions of libc wasi, by removing dlmalloc.o from libc.a, (2) use higher version of wasi-sdk (larger than 20.0) and export malloc/free functions in wasm app, by adding -Wl,--export=malloc -Wl,--export=free for /opt/wasi-sdk/bin/clang.

[kamylee]

I am calling the init() method in wasm through wasm_runtime_call_casm.
Thread 1 and thread 2 are both created by the "init()" method.

wasi-sdk ：I'm using wasi-sdk-22.0+m.

I complied my program like this:

cmake .. -DWAMR_BUILD_LIB_PTHREAD=1 -DWAMR_BUILD_LIB_WASI_THREADS=1 -DWAMR_BUILD_PLATFORM=windows -DWAMR_BUILD_MULTI_MODULE=1 -DWAMR_BUILD_DUMP_CALL_STACK=1
cmake --build . --config Release

Cmakelists.txt:
`add_executable (HttpServer.wasm
${JSONCPP_SOURCES}
"HttpServer.cpp" "Common.cpp"
"thirdparty/llhttp/llhttp.c" "thirdparty/llhttp/http.c" "thirdparty/llhttp/api.c"
"thirdparty/llhttp/WebSocket.cpp" "thirdparty/llhttp/base64/base64.cpp" "thirdparty/llhttp/sha1/sha1.cc"
"tsdb_sample.c" #"kvdb_basic_sample.c" "kvdb_type_blob_sample.c" "kvdb_type_string_sample.c"
"CAppProcessManage.cpp" "CMessageManage.cpp" "CMMap.cpp"
${FLASHDB_SRC}
)

target_compile_options(HttpServer.wasm PRIVATE -pthread )#-g

TARGET_LINK_LIBRARIES(HttpServer.wasm
#cjson
pthread

)

target_link_options(HttpServer.wasm PRIVATE
LINKER:--export=__heap_base
LINKER:--export=__data_end
LINKER:--export=malloc
LINKER:--export=free
LINKER:--export=init
LINKER:--export=onMessage
LINKER:--export=onDataRecv
LINKER:--export=OnDestroy
LINKER:--export=on_connect
LINKER:--shared-memory
#linear memory线性内存 包含:ncludes three parts, data area, auxiliary stack area and heap area.
LINKER:--initial-memory=45875200,--max-memory=65536000 #900*65536=40,632,320
LINKER:-zstack-size=13107200 # aux stack 819200 1638400
LINKER:--no-check-features
LINKER:--allow-undefined
)
`
$ cmake -G "Unix Makefiles" -DWASI_SDK_PREFIX=E:/WorkSpace/DownLoads/wasi-sdk-21.0.m-mingw/wasi-sdk-21.0+m -DCMAKE_TOOLCHAIN_FILE=E:/WorkSpace/DownLoads/wasi-sdk-21.0.m-mingw/wasi-sdk-21.0+m/share/cmake/wasi-sdk.cmake -DCMAKE_SYSROOT=E:/WorkSpace/DownLoads/wasi-sdk-21.0.m-mingw/wasi-sdk-21.0+m/share/wasi-sysroot ..
make

It doesn't completely stop working, but crashes after a few or a dozen times.
I did some tests, and it's very likely that wasm_runtime_module_malloc and wasm_runtime_module_free are the cause.

[wenyongh]

I am a little confused, do you use the same exec_env/module_inst to call the init() function, and are the two threads belongs to a sample wasm instance (e.g. use the same shared linear memory)?

Could you track which line in C source code causes the exception first? Please refer to https://github.com/bytecodealliance/wasm-micro-runtime/tree/main/samples/debug-tools.

[kamylee]

Yes,i use the same exec_env/module_inst to call the init() function. These two threads belong to the same wasm instance.
Can't I create two threads in init()?
I'm trying to use debug-tools.

[wenyongh]

Just curious why not let init() create two threads one time.
Does the init() call pthread_create() to create the thread? And how to pass different thread callbacks for thread A and thread B, I guess you pass a flag to init() and init() passes different thread callback to thread A and B according to the flag? So there are three threads eventually.

[kamylee]

My code is as follows:
`
void* Thread_Tcp_loop(void* Param)
{
...
}
static void* Thread_Test(void* Param)
{
...
}
extern "C" int init(char* strBuf, int bufLen)
{
...
pthread_t tid;
int ret = pthread_create(&tid, NULL, &Thread_Tcp_loop, (void*)iCurID);
if (ret) {
printf("failed to spawn thread: %s", strerror(ret));
}
pthread_detach(tid);

pthread_t tid1;
int ret1 = pthread_create(&tid1, NULL, &Thread_Test, (void*)iCurID);
if (ret1) {
printf("failed to spawn thread: %s", strerror(ret1));
}
pthread_detach(tid1);
...
return 1;
}
`

[wenyongh]

Hi, so you create two threads in init(), then you should only call init() one time in the host native? And could you remove pthread_detach(tid); and pthread_detach(tid1);? A little confused why detach the threads here.

Another issue is that could you wait some time after

ret = pthread_create(&tid, NULL, &Thread_Tcp_loop, (void*)iCurID);
if (ret) {
printf("failed to spawn thread: %s", strerror(ret));
}

e.g., add usleep(..) or pthread_cond_wait to wait until the loop actually launches?

[kamylee]

yes there are two threads in init() and only init() is called only once.

Because I need the init() function to return immediately.so call pthread_detach.
After calling init(), the host program enters a message receiving loop. Once it receives a message from another WASP app, it calls wasm_runtime_module_malloc to pass parameters to the onMessage function inside the wasm.

actually,i added sleep():

And in both of these two threads, the loop does not exit.

[wenyongh]

OK, seems it isn't caused by pthread_detach, but it may be better to put the first pthread_detach after NativeApp_Sleep(10) and put the second pthread_detach after NativeApi_sleep(100).

[kamylee]

I simply use wasm_runtime_module_malloc/free, and there is still a chance of crashing, but it doesn't crash if malloc is not executed.

[wenyongh]

Not sure whether it is caused by wasi-sdk (its libc bytecode of malloc function), or is caused by wamr, could you try compile the wasm app with wasi-sdk-20+threads?

Another way you can try is to remove the dlmalloc.o from wasi-sdk's libc.a, so wasm_runtime_module_malloc will allocate memory from wamr's app heap instead of libc's malloc, please refer to pthread_library.md:

And if you can, had better dump the call stack of wasm app.

[kamylee]

i have upgraded the wasi-sdk from 21 to 24(wasi-sdk-24.0-x86_64-windows)。

After following the steps above, libc.a became smaller, the wasm compiled successfully, but there was an error during execution.

[wenyongh]

Hi, is it caused by wasm_runtime_module_malloc returning 0? Do yo pass --heap-size=n to iwasm, or if you are not using iwasm, could you pass host_managed_heap_size with a value larger than 0 to wasm_runtime_instantiate:

wasm-micro-runtime/core/iwasm/include/wasm_export.h

Lines 696 to 700 in 36d4380

	WASM_RUNTIME_API_EXTERN wasm_module_inst_t
	wasm_runtime_instantiate(const wasm_module_t module,
	uint32_t default_stack_size,
	uint32_t host_managed_heap_size, char *error_buf,
	uint32_t error_buf_size);

[kamylee]

yes，this is exactly how I do it.

[kamylee]

基本确定是线程方面造成的，因为我在调用wasm内的init之前调用wasm_runtime_module_malloc/free达6000次都不崩溃，但放在调用init()之后，并且init里只启动1个线程，在循环6000次期间，如果这个线程里收到了数据，就一定会崩溃，似乎堆栈被破坏了。

for (int i = 0; i < 6000; i++)
{
	printf("wasm_runtime_module_malloc time %d\n", i);
	//在wasm中分配内存
	char* bufferTemp = NULL;
	uint64_t wasmBuffer = wasm_runtime_module_malloc(wasm_module_inst, 100, (void**)&bufferTemp);
	//wasmBufferName = wasm_runtime_module_dup_data(wasm_module_inst, "dasdf", 100);
	if (wasmBuffer != 0)
	{
		strncpy(bufferTemp, "{\"key\":\"value1\"}", 100);
		uint32 argv[2];
		argv[0] = wasmBuffer;     /* pass the buffer address for WASM space */
		argv[1] = 100;
		if (!wasm_runtime_call_wasm(exec_env, onDataRecv, 2, argv))
		{
			const char* errInfo = wasm_runtime_get_exception(wasm_module_inst);
			DLOG << "Native wasm_runtime_call_wasm err msg:" << errInfo;
		}
		printf("wasm_runtime_module_free\n");
		wasm_runtime_module_free(wasm_module_inst, wasmBuffer);
	}
}//6000次不崩溃
...
wasm_runtime_call_wasm(exec_env, init, 2, argv);//内有1个线程，用于接收socket数据
...
如果循环放在这里，当init()内创建的**线程A**收到数据后，就一定会崩溃。

线程A调用NativeApiSocket创建了socket 和使用accept来监听新的连接，accept目前是阻塞模式，当有新的连接接入时会调用wasm内的onConnect(socketId)并传入参数socketId，在onConnect又创建了线程用于调用native本地NativeApiRecv接收当前socketId的数据。我设的线程上限是4个，并未超过线程数量上限。

那我是不能在wasm用线程了吗？还是有其他办法解决这个问题吗？