Thread use question in Windows

[xingkaiyu]

I had seen the issue #547
And I tried to use wasm_runtime_init_thread_env() and wasm_runtime_destroy_thread_env().
My code is like this:
1- call wasm_runtime_init_thread_env()
2- call wasm_runtime_call_wasm() and in wasm app create thread.
3- call wasm_runtime_destroy_thread_env() after wasm_runtime_call_wasm()
It works the first time. And wasm_runtime_init_thread_env returns true.
But the follows all went to crash. And wasm_runtime_init_thread_env return false. (I use the same thread as first time)

Then I just call wasm_runtime_init_thread_env one time in this thread. And I do not call wasm_runtime_destroy_thread_env any more.
Then it works well.

So I have some questions:

1. Is there any bug in wasm_runtime_destroy_thread_env?
2. Can I only call wasm_runtime_init_thread_env one time and never call wasm_runtime_destroy_thread_env in one thread?
3. Can wasm_runtime_init_thread_env be used in other platform such as Android?
4. Which thread don't need to call wasm_runtime_init_thread_env, the thread called wasm_runtime_init, the thread called wasm_runtime_create_exec_env or other?

[wenyongh]

I had seen the issue #547 And I tried to use wasm_runtime_init_thread_env() and wasm_runtime_destroy_thread_env(). My code is like this: 1- call wasm_runtime_init_thread_env() 2- call wasm_runtime_call_wasm() and in wasm app create thread. 3- call wasm_runtime_destroy_thread_env() after wasm_runtime_call_wasm() It works the first time. And wasm_runtime_init_thread_env returns true. But the follows all went to crash. And wasm_runtime_init_thread_env return false. (I use the same thread as first time)

Then I just call wasm_runtime_init_thread_env one time in this thread. And I do not call wasm_runtime_destroy_thread_env any more. Then it works well.

Hi, do you call wasm_runtime_init_thread_env()/destroy_thread_env() in the thread that is created from the wasm app? Or, wasm app invokes pthread_create like API, and in the native, you call wasm_runtime_init_thread_env()? If yes, don't call them, these APIs are only required to call when the thread isn't created by runtime (runtime creating thread normally includes (1) wasm app invokes pthread_create(), (2) developer calls wasm_runtime_spawn_thread(), (3) developer calls os_thread_create()), for example, if developer creates thread by himself with native pthread_create() function, then in the thread callback, he normally should:

void *thread_callback(..)
{
    ret = wasm_runtime_init_thread_env();
    /* check return value */

   create exec env, and call wasm func

   wasm_runtime_destroy_thread_env();
   return NULL;
}

So I have some questions:

1. Is there any bug in wasm_runtime_destroy_thread_env?
2. Can I only call wasm_runtime_init_thread_env one time and never call wasm_runtime_destroy_thread_env in one thread?
3. Can wasm_runtime_init_thread_env be used in other platform such as Android?

Yes, you can have a try.

4. Which thread don't need to call wasm_runtime_init_thread_env, the thread called wasm_runtime_init, the thread called wasm_runtime_create_exec_env or other?

[xingkaiyu]

@wenyongh
Thanks for your reply.
Did you mean that the thead my wasm app run in and call pthread_create must be one kind of these threads:

1. The thread that managed or created by the wamr runtime. Such as threads created by wasm_runtime_spawn_thread and the threads created in wasm app.
2. The thread created by native derectly using os thread maker (like pthread_create) and I called wasm_runtime_init_thread_env in the new thread before run wasm function.
   And did you mean that wasm_runtime_init_thread_env and wasm_runtime_destroy_thread_env should never be called in the first satuation. Or else the thread will be broken and can't create thread in wasm app any more?

But I still don't know how to do...
My situation is that the native rumtime maneger is not the maneger of the process. It is part of the whole application and will be called passively. So some time I don't know which thread now it is. Is there any method like "wasm_runtime_need_init_thread_env" to help me to know whether call or not call wasm_runtime_init_thread_env?

And I have more tests today. And from them I think wasm_runtime_init_thread_env will always go wrong in a thread when called more than one time. No matter whether it is a native created thread or not.

My test step are:

condition1:
1- thread 19964 call wasm_runtime_init wasm_runtime_load wasm_runtime_instantiate wasm_runtime_create_exec_env
2- thread 18144 call wasm_runtime_init_thread_env and get true. Call wasm_app1 to create thread and it works ok. Call wasm_runtime_destroy_thread_env.
3- thread 19964 call wasm_runtime_init_thread_env and get false. Call wasm_app2 to do things but not create thread and it works ok. Call wasm_runtime_destroy_thread_env.
4- thread 18144 call wasm_runtime_init_thread_env again and get false. Call wasm_app1 to create thread and get crash.

condition2:
1- thread 3584 call wasm_runtime_init wasm_runtime_load wasm_runtime_instantiate wasm_runtime_create_exec_env
2- thread 3584 call wasm_runtime_init_thread_env and get false. Call wasm_app2 to do things but not create thread and it works ok. Call wasm_runtime_destroy_thread_env.
3- thread 23040 call wasm_runtime_init_thread_env and get true. Call wasm_app1 to create thread and it works ok. Call wasm_runtime_destroy_thread_env.
4- thread 22100 call wasm_runtime_init_thread_env and get true. Call wasm_app1 to create thread and it works ok. Call wasm_runtime_destroy_thread_env.
5- thread 22100 call wasm_runtime_init_thread_env again and get false. Call wasm_app1 to create thread and get crash.

condition3:
1- thread 23356 call wasm_runtime_init wasm_runtime_load wasm_runtime_instantiate wasm_runtime_create_exec_env
2- thread 23356 call wasm_app2 to do things but not create thread and it works ok. (This time not call wasm_runtime_init_thread_env and wasm_runtime_destroy_thread_env)
3- thread 26204 call wasm_runtime_init_thread_env and get true. Call wasm_app1 to create thread and it works ok. Call wasm_runtime_destroy_thread_env.
4- thread 26204 call wasm_runtime_init_thread_env again and get false. Call wasm_app1 to create thread and get crash.

[wenyongh]

@wenyongh Thanks for your reply. Did you mean that the thead my wasm app run in and call pthread_create must be one kind of these threads:

1. The thread that managed or created by the wamr runtime. Such as threads created by wasm_runtime_spawn_thread and the threads created in wasm app.
2. The thread created by native derectly using os thread maker (like pthread_create) and I called wasm_runtime_init_thread_env in the new thread before run wasm function.
   And did you mean that wasm_runtime_init_thread_env and wasm_runtime_destroy_thread_env should never be called in the first satuation. Or else the thread will be broken and can't create thread in wasm app any more?

Hi, only when the thread isn't managed by runtime, developer needs to call wasm_runtime_init_thread_env/wasm_runtime_destroy_thread_env. Normally in these situations the thread is managed by runtime:

• the main thread that calls wasm_runtime_init() or wasm_runtime_full_init()
• the thread created by wasm app, in which wasm app invokes imported pthread_create() to create thread
• the thread created by wasm_runtime_spawn_thread()
• the thread created by os_thread_create()

And if the thread is created by developer, e.g. using the host pthread_create or _beginthreadex API, and wasm_runtime_init()/full_init() isn't called, then wasm_runtime_init_thread_env/wasm_runtime_destroy_thread_env should be called, and developer had better call them at the beginning and the end of the thread callback func, e.g. in Windows:

unsigned __stdcall thread_wrapper(void *arg) {
    wasm_runtime_init_thread_env();
    create exec env and call wasm func;
    wasm_runtime_destroy_thread_env();
    return 0;
}

_beginthreadex(NULL, 32768, thread_wrapper, arg, 0, NULL);

But I still don't know how to do... My situation is that the native rumtime maneger is not the maneger of the process. It is part of the whole application and will be called passively. So some time I don't know which thread now it is. Is there any method like "wasm_runtime_need_init_thread_env" to help me to know whether call or not call wasm_runtime_init_thread_env?

It is not easy to implement the wasm_runtime_need_init_thread_env like API, can you use thread local variable to check whether thread_env is initied? e.g.

static __declspec(thread) bool thread_env_inited = false;

And set it to true when wasm_runtime_need_init_thread_env is called, and false when wasm_runtime_need_destroy_thread_env is called?

And I have more tests today. And from them I think wasm_runtime_init_thread_env will always go wrong in a thread when called more than one time. No matter whether it is a native created thread or not.

My test step are:

condition1: 1- thread 19964 call wasm_runtime_init wasm_runtime_load wasm_runtime_instantiate wasm_runtime_create_exec_env 2- thread 18144 call wasm_runtime_init_thread_env and get true. Call wasm_app1 to create thread and it works ok. Call wasm_runtime_destroy_thread_env. 3- thread 19964 call wasm_runtime_init_thread_env and get false. Call wasm_app2 to do things but not create thread and it works ok. Call wasm_runtime_destroy_thread_env. 4- thread 18144 call wasm_runtime_init_thread_env again and get false. Call wasm_app1 to create thread and get crash.

thread 19964 should not call wasm_runtime_init_thread_env as it is inited by wasm_runtime_init.

condition2: 1- thread 3584 call wasm_runtime_init wasm_runtime_load wasm_runtime_instantiate wasm_runtime_create_exec_env 2- thread 3584 call wasm_runtime_init_thread_env and get false. Call wasm_app2 to do things but not create thread and it works ok. Call wasm_runtime_destroy_thread_env. 3- thread 23040 call wasm_runtime_init_thread_env and get true. Call wasm_app1 to create thread and it works ok. Call wasm_runtime_destroy_thread_env. 4- thread 22100 call wasm_runtime_init_thread_env and get true. Call wasm_app1 to create thread and it works ok. Call wasm_runtime_destroy_thread_env. 5- thread 22100 call wasm_runtime_init_thread_env again and get false. Call wasm_app1 to create thread and get crash.

Same as above, thread 3584 should not call wasm_runtime_init_thread_env.

condition3: 1- thread 23356 call wasm_runtime_init wasm_runtime_load wasm_runtime_instantiate wasm_runtime_create_exec_env 2- thread 23356 call wasm_app2 to do things but not create thread and it works ok. (This time not call wasm_runtime_init_thread_env and wasm_runtime_destroy_thread_env) 3- thread 26204 call wasm_runtime_init_thread_env and get true. Call wasm_app1 to create thread and it works ok. Call wasm_runtime_destroy_thread_env. 4- thread 26204 call wasm_runtime_init_thread_env again and get false. Call wasm_app1 to create thread and get crash.

The usage seems to be good, normally we only tested that wasm_runtime_init_thread_env and wasm_runtime_destroy_thread_env are called only one time. I fixed an issue for "thread 26204 call wasm_runtime_init_thread_env again and get false" with patch #923, see win_thread.c, could you please try again? Thanks.

[xingkaiyu]

@wenyongh
I read the source code and I think my crash is caused by enable aot in wamr.
bool
wasm_runtime_init_thread_env()
{
#ifdef BH_PLATFORM_WINDOWS
if (os_thread_env_init() != 0)
return false;
#endif

#if WASM_ENABLE_AOT != 0
#ifdef OS_ENABLE_HW_BOUND_CHECK
if (!aot_signal_init()) {
#ifdef BH_PLATFORM_WINDOWS
os_thread_env_destroy();
#endif
return false;
}
#endif
#endif
return true;
}

Then I disabled aot and my application never crash any more. No matter I called wasm_runtime_init_thread_env in wasm_runtime_init thread or not. No matter I called wasm_runtime_destroy_thread_env or not....

And without aot, I think the win_thread code does enough protect to the two method:

1. wasm_runtime_init_thread_env will do nothing when an old data exists.
2. wasm_runtime_destroy_thread_env will do nothing when the thread data is supervisor_thread_data
   So I think they can be called freely.

I think you can check the aot code to find whether there are any mistakes...

[wenyongh]

@xingkaiyu Have you applied the patch #923? When aot is enabled, wasm_runtime_init_thread_env() calls aot_signal_init() also, and aot_signal_init does two things:

1. os_thread_signal_init, which calls SetThreadStackGuarantee with 16KB for stack overflow check, meaning that when these pages are accessed, the stack overflow will happen and the registered exception handler will be triggered to handle it.
2. AddVectoredExceptionHandler, which is to register the exception handler, here it might be not necessary for the sub thread as the main thread has registered it. But I have a test, it also works to register it again.

Could you please check whether the wasm_runtime_init_thread_env()/wasm_runtime_destroy_thread_env() can be called repeatedly in the sub thread? e.g. in the condition 3 above you mentioned. And wondering how you generating the AOT file, did you compile wamrc on windows, and use it to generate the AOT file? Does the AOT file work in single thread mode?

And note that by default the boundary check with exception handler is enabled, so when the wasm app runs into exception (or crash), like "out of bounds memory access" and "native stack overflow", the exception handler will be triggered, we should wait until the exception handling ends. Could you check whether the exception handler works correctly, and if not, debug which exception was thrown? Or is there a simple case to reproduce the issue? Thanks.

[xingkaiyu]

@wenyongh
I am using the WAMR-08-10-2021 and now I just test thread in wasm mode not aot mode. I enable aot in wamr just for convenient. Maybe I will test aot in the future.
And for patch #923. I think the related modification is the on in win_thread.c. And I change the return value from true to 0. Then it is ok now even though I enabled aot mode in wamr.
And I think the wasm_runtime_init_thread_env can still be called freely in any thread. Because either in win_thread.c or in posix_thread.c os_thread_signal_init only do init when thread is not inited and I think AddVectoredExceptionHandler called repeatly is ok. And I think os_thread_env_init and aot_signal_init is harmless except some ram and time waste for the thread not need to use sub-thread in wasm app.
But for wasm_runtime_destroy_thread_env I am not sure. For example, if I do the code like this:

1. create a thread in wasm app. It will use os_thread_wrapper to do os_thread_signal_init
2. call native function in sub thread
3. native call another wasm function and init thread env before calling(I think will do nothing) and desdroy thread env after calling. It will call os_thread_signal_destroy (do nothing in windows but something in posix)
4. native return result to the first wasm function in sub-thread

I don't know if the wasm function will be ok for the following step. Because I destroyed the thread_signal early than os_thread_wrapper.
Maybe I should never call destroy in native.
I will do some more tests...