Fix hot-reload panic on Linux

[Bromeon]

Fixes #648.

Adds a new hot-reloading example. It's pretty bare-bones and can be polished, but that's for the future.

Also added integration tests for hot-reloading. Don't ask me how they work, it's a big patchwork of Bash, Python, GDScript, Rust and UDP 😂

Furthermore, Linux exhibits very strange reloading behavior. I cannot get normal reloading to work, since the reloaded .so library is always stale -- in both WSL2 and the CI runners. People seem to be able to do hot-reloading on Linux though. What I don't quite understand is how a .so is not fully unloaded (thus causing the double-init panic in #648), yet hot-reloading can apparently still work (in the sense that new compiled Rust code is picked up). This may need further investigation, at the moment I'm working around this issue by renaming the .so. Of course that workaround makes the scenario a bit more distant from real hot-reloading done by users.

---

Update: after eternal debugging, it looks like ThreadId may be responsible. It could be related to thread-local storage or syscalls that prevent dlclose() from actually closing the library. Amos' post below goes into more detail.

This also explains why the workaround experimental-threads was successful: multithreading did not validate the thread ID.

I'll leave this for later, but it's important to note that this problem is not limited to gdext. Even if we find a way to avoid ThreadId, user code could always use it. I'm not sure if other languages are also affected, but it may be necessary that Godot establishes a mechanism to rename .so files, like they already do for .dlls on Windows.

See also:

• Faster than Lime: What can prevent dlclose from unloading a library
• Discord thread about the panic issue

[GodotRust]

API docs are being generated and will be shortly available at: https://godot-rust.github.io/docs/gdext/pr-653

[Bromeon]

Merging, as it fixes a long-standing issue with hot reload on Linux and adds integration tests, preventing further regressions.

Regarding ThreadId, a proper strategy is more sophisticated and may likely involve upstream Godot changes. As mentioned above, it might be necessary to adopt a policy to rename .so files after updating, as is already the case for Windows .dlls.

[TitanNano]

I'm not sure if other languages are also affected, but it may be necessary that Godot establishes a mechanism to rename .so files, like they already do for .dlls on Windows.

Hot reloading on macOS does not work either. With your findings here, I get the impression the engine should enable so/dynlib/dll renaming on all platforms.

[Bromeon]

@TitanNano created upstream issue (see link), however simply renaming seems not viable. We probably need to address it on our side. Maybe with the macro that FasterThanLime recommends, or with some compiler flag to the LLVM backend, mirroring -fno-gnu-unique in C++.

Are you sure that macOS experiences the same issue? Not just hot-reloading doesn't work, but specifically because the dynamic library is prevented from being unloaded?

[lilizoey]

I'll see if i can hack in faster than lime's macro and maybe that just works

[lilizoey]

So it does in fact work if i add faster than lime's code (and add back in thread-id checking to actually make hot reloading fail)
Here's my code: https://github.com/lilizoey/gdextension/tree/hacky-hot-reloading-fix

[TitanNano]

Are you sure that macOS experiences the same issue? Not just hot-reloading doesn't work, but specifically because the dynamic library is prevented from being unloaded?

@Bromeon I used 7fd1be9 on macOS and hot reloading would not work properly because the old lib version would not get unloaded. After many hours of going through fasterthanlimes post and looking into the latest available source code for dlopen and dlclose from apple, I gave up because I wasn't able to determine why the dynlib was not being unloaded.
Using the same project source on Linux showed that the project could, in fact, be hot-reloaded. It also worked on macOS when I modified the engine to use the same renaming logic as windows.