Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Godot 3.2.4rc3 mono crashes on exit with specific combination of packed scene and dynamic font #46679

Closed
raineszm opened this issue Mar 5, 2021 · 21 comments · Fixed by #46742

Comments

@raineszm
Copy link

raineszm commented Mar 5, 2021

Godot version:

v3.2.4.rc3.mono.official

OS/device including version:
MacOS 11.2.2

OpenGL ES 3.0 Renderer: Intel(R) Iris(TM) Plus Graphics 650
OpenGL ES Batching: ON

Issue description:

Sorry the title is such a mess. I couldn't figure out a way to summarize this succinctly.

Godot segfaults on exit if the following sequence occurs:

  • Load a scene with a monoscript containing an exported PackedScene variable
  • Change scenes to that packed scene.
  • If the packed scene uses a control with a DynamicFont Godot will now segfault when quitting producing a traceback such as the following.

TRACEBACK.txt

Concretely if I have a button with this script

using Godot;

public class AButton : Button
{
    [Export] private PackedScene Other;
    public override void _Pressed()
    {
        base._Pressed();
        GetTree().ChangeSceneTo(Other);
    }
}

and the scene Other contains a dynamic font, then the game crashes on exit if the scene Other has been loaded.

Steps to reproduce:
The conditions for this are somewhat specific but I've been able to reproduce them reliably with the following.

  • In the example project below. Run the scene AScene.tscn
  • Click the button
  • Quit
    At this point the game crashes on exit and dumps a stacktrace.

Note that no crash occurs if:

  • The second scene isn't loaded
  • The second scene isn't loaded from the packed scene variable
  • The second scene does not use a DynamicFont

Minimal reproduction project:

SampleCrash.zip

@akien-mga
Copy link
Member

Thanks for the detailed bug report! I can't reproduce the crash on Linux, but it might be related to #46411. Can you confirm if it's a regression in 3.2.4 RC 3 or do you also experience the issue with previous releases?

@akien-mga akien-mga added this to the 3.2 milestone Mar 5, 2021
@akien-mga
Copy link
Member

If it's a regression in 3.2.4 RC 3, could you try those two builds to confirm if the "before-pr45618" one fixes the issue?
https://downloads.tuxfamily.org/godotengine/testing/Godot_v3.2.4-rc-before-pr45618_mono_osx.64.zip
https://downloads.tuxfamily.org/godotengine/testing/Godot_v3.2.4-rc-pr45618_mono_osx.64.zip

That's just a hunch, a proper Git bisect would be better but time consuming.

@akien-mga
Copy link
Member

Might actually be the same issue as #46550 which also happens on Ubuntu, so might not be macOS-specific. I still can't reproduce either issue on Mageia 8 or Ubuntu 20.04 myself :(

@raineszm
Copy link
Author

raineszm commented Mar 5, 2021

If it's a regression in 3.2.4 RC 3, could you try those two builds to confirm if the "before-pr45618" one fixes the issue?
https://downloads.tuxfamily.org/godotengine/testing/Godot_v3.2.4-rc-before-pr45618_mono_osx.64.zip
https://downloads.tuxfamily.org/godotengine/testing/Godot_v3.2.4-rc-pr45618_mono_osx.64.zip

That's just a hunch, a proper Git bisect would be better but time consuming.

I can confirm that this bug happens in the pr45618 build but not the before-pr45618 build.

@akien-mga
Copy link
Member

Thanks, that confirms that it's a regression from #45618.

I see that there's some changes to DynamicFont's Mutex in the Mutex refactoring commit. They look correct but maybe it now behaves differently and causes this issue?

@bruvzg
Copy link
Member

bruvzg commented Mar 5, 2021

I can confirm the crash on macOS. I do not see anything related to DynamicFont, but I guess the font destructor can delete textures in the same way:

Stack trace:

[Main Thread]

libsystem_kernel.dylib!semaphore_timedwait_trap (Unknown Source:0)
libmonosgen-2.0.1.dylib!mono_coop_sem_timedwait (Unknown Source:0)
libmonosgen-2.0.1.dylib!mono_domain_finalize (Unknown Source:0)
godot.osx.tools.x86_64.mono!GDMono::_unload_scripts_domain() (/modules/mono/mono_gd/gd_mono.cpp:1048)
godot.osx.tools.x86_64.mono!GDMono::~GDMono() (/modules/mono/mono_gd/gd_mono.cpp:1265)
godot.osx.tools.x86_64.mono!GDMono::~GDMono() (/modules/mono/mono_gd/gd_mono.cpp:1260)
godot.osx.tools.x86_64.mono!void memdelete<GDMono>(GDMono*) (/core/os/memory.h:117)
godot.osx.tools.x86_64.mono!CSharpLanguage::finish() (/modules/mono/csharp_script.cpp:154)
godot.osx.tools.x86_64.mono!ScriptServer::finish_languages() (/core/script_language.cpp:184)
godot.osx.tools.x86_64.mono!Main::cleanup(bool) (/main/main.cpp:2244)
godot.osx.tools.x86_64.mono!main (/platform/osx/godot_main_osx.mm:73)
libdyld.dylib!start (Unknown Source:0)
libdyld.dylib!start (Unknown Source:0)

[Thread causing crash]

libGL.dylib!glDeleteTextures (Unknown Source:0)
godot.osx.tools.x86_64.mono!RasterizerStorageGLES3::Texture::~Texture() (/drivers/gles3/rasterizer_storage_gles3.h:343)
godot.osx.tools.x86_64.mono!RasterizerStorageGLES3::Texture::~Texture() (/drivers/gles3/rasterizer_storage_gles3.h:339)
godot.osx.tools.x86_64.mono!void memdelete<RasterizerStorageGLES3::Texture>(RasterizerStorageGLES3::Texture*) (/core/os/memory.h:117)
godot.osx.tools.x86_64.mono!RasterizerStorageGLES3::free(RID) (/drivers/gles3/rasterizer_storage_gles3.cpp:8009)
godot.osx.tools.x86_64.mono!VisualServerRaster::free(RID) (/servers/visual/visual_server_raster.cpp:72)
godot.osx.tools.x86_64.mono!VisualServerWrapMT::free(RID) (/servers/visual/visual_server_wrap_mt.h:593)
godot.osx.tools.x86_64.mono!ImageTexture::~ImageTexture() (/scene/resources/texture.cpp:444)
godot.osx.tools.x86_64.mono!ImageTexture::~ImageTexture() (/scene/resources/texture.cpp:442)
godot.osx.tools.x86_64.mono!void memdelete<Reference>(Reference*) (/core/os/memory.h:117)
godot.osx.tools.x86_64.mono!godot_icall_Reference_Disposed(_MonoObject*, Object*, unsigned char) (/modules/mono/glue/base_object_glue.cpp:113)
[Unknown/Just-In-Time compiled code] (Unknown Source:0)
libmonosgen-2.0.1.dylib!mono_gc_run_finalize (Unknown Source:0)
libmonosgen-2.0.1.dylib!sgen_gc_invoke_finalizers (Unknown Source:0)
libmonosgen-2.0.1.dylib!finalizer_thread (Unknown Source:0)
libmonosgen-2.0.1.dylib!start_wrapper_internal (Unknown Source:0)
libmonosgen-2.0.1.dylib!start_wrapper (Unknown Source:0)
libsystem_pthread.dylib!_pthread_start (Unknown Source:0)
libsystem_pthread.dylib!thread_start (Unknown Source:0)

It crashes at the glDeleteTextures called from thread (I guess it's Mono GC) with EXC_BAD_ACCESS (code=1, address=0x0).
Not sure why, but IIRC OpenGL require some content switching to use it from another thread.

Edit: crash at editor exit, with empty Mono project opened.

@bruvzg
Copy link
Member

bruvzg commented Mar 5, 2021

With the test project from #46550 (comment) it crashes on glGenTextures call.

[Main Thread]

libobjc.A.dylib!objc_msgSend (Unknown Source:0)
CoreFoundation!__RELEASE_OBJECTS_IN_THE_ARRAY__ (Unknown Source:0)
CoreFoundation!-[__NSArrayM dealloc] (Unknown Source:0)
ColorSync!createProfileData (Unknown Source:0)
ColorSync!ColorSyncProfileGetMD5 (Unknown Source:0)
CoreGraphics!md5_creator (Unknown Source:0)
CoreFoundation!-[__NSDictionaryI __apply:context:] (Unknown Source:0)
CoreGraphics!CGColorSyncTransformCacheGetRetained (Unknown Source:0)
CoreGraphics!CGCMSConverterCreate (Unknown Source:0)
CoreGraphics!CGColorTransformCacheGetConversionType (Unknown Source:0)
CoreGraphics!CGColorTransformConvertColorComponents (Unknown Source:0)
CoreGraphics!CGFontRenderingGetCustomAntialiasingStyle (Unknown Source:0)
CoreGraphics!CGContextDelegateDrawGlyphs (Unknown Source:0)
CoreGraphics!draw_glyphs (Unknown Source:0)
CoreGraphics!CGContextShowGlyphsWithAdvances (Unknown Source:0)
CoreText!EnumerateOverlappingGlyphs(CGContext*, TFont const&, unsigned short const*, long, int, void (CFRange, bool) block_pointer) (Unknown Source:0)
CoreText!CTFontDrawGlyphsWithAdvancesInternal (Unknown Source:0)
UIFoundation!-[NSCoreTypesetter _NSFastDrawString:length:attributes:paragraphStyle:typesetterBehavior:lineBreakMode:rect:padding:graphicsContext:baselineRendering:usesFontLeading:usesScreenFont:scrollable:syncAlignment:mirrored:boundingRectPointer:baselineOffsetPointer:drawingContext:] (Unknown Source:0)
UIFoundation!-[NSCoreTypesetter _stringDrawingCoreTextEngineWithOriginalString:rect:padding:graphicsContext:forceClipping:attributes:stringDrawingOptions:drawingContext:stringDrawingInterface:] (Unknown Source:0)
UIFoundation!__NSStringDrawingEngine (Unknown Source:0)
UIFoundation!_NSStringDrawingCore (Unknown Source:0)
AppKit!_NSDrawTextCell (Unknown Source:0)
AppKit!-[NSTextFieldCell _drawForegroundOfTextLayer] (Unknown Source:0)
AppKit!+[NSAppearance _performWithCurrentAppearance:usingBlock:] (Unknown Source:0)
AppKit!__NSTextLayerDrawForeground_block_invoke (Unknown Source:0)
AppKit!-[NSFocusStack performWithFocusView:inWindow:usingBlock:] (Unknown Source:0)
AppKit!NSTextLayerDrawForeground (Unknown Source:0)
QuartzCore!CABackingStoreUpdate_ (Unknown Source:0)
QuartzCore!invocation function for block in CA::Layer::display_() (Unknown Source:0)
QuartzCore!-[CALayer _display] (Unknown Source:0)
AppKit!__22-[NSTextLayer display]_block_invoke_2 (Unknown Source:0)
AppKit!+[NSAppearance _performWithCurrentAppearance:usingBlock:] (Unknown Source:0)
AppKit!__22-[NSTextLayer display]_block_invoke (Unknown Source:0)
AppKit!-[NSFocusStack performWithFocusView:inWindow:usingBlock:] (Unknown Source:0)
AppKit!-[NSTextLayer display] (Unknown Source:0)
QuartzCore!CA::Layer::display_if_needed(CA::Transaction*) (Unknown Source:0)
QuartzCore!CA::Context::commit_transaction(CA::Transaction*, double, double*) (Unknown Source:0)
QuartzCore!CA::Transaction::commit() (Unknown Source:0)
AppKit!__62+[CATransaction(NSCATransaction) NS_setFlushesWithDisplayLink]_block_invoke (Unknown Source:0)
AppKit!___NSRunLoopObserverCreateWithHandler_block_invoke (Unknown Source:0)
CoreFoundation!__CFRUNLOOP_IS_CALLING_OUT_TO_AN_OBSERVER_CALLBACK_FUNCTION__ (Unknown Source:0)
CoreFoundation!__CFRunLoopDoObservers (Unknown Source:0)
CoreFoundation!__CFRunLoopRun (Unknown Source:0)
CoreFoundation!CFRunLoopRunSpecific (Unknown Source:0)
HIToolbox!RunCurrentEventLoopInMode (Unknown Source:0)
HIToolbox!ReceiveNextEventCommon (Unknown Source:0)
HIToolbox!_BlockUntilNextEventMatchingListInModeWithFilter (Unknown Source:0)
AppKit!_DPSNextEvent (Unknown Source:0)
AppKit!-[NSApplication(NSEvent) _nextEventMatchingEventMask:untilDate:inMode:dequeue:] (Unknown Source:0)
godot.osx.tools.x86_64.mono!OS_OSX::process_events() (/platform/osx/os_osx.mm:3145)
godot.osx.tools.x86_64.mono!OS_OSX::run() (/platform/osx/os_osx.mm:3269)
godot.osx.tools.x86_64.mono!main (/platform/osx/godot_main_osx.mm:71)
libdyld.dylib!start (Unknown Source:0)

[Thread causing crash]

libGL.dylib!glGenTextures (Unknown Source:0)
godot.osx.tools.x86_64.mono!RasterizerStorageGLES2::texture_create() (/drivers/gles2/rasterizer_storage_gles2.cpp:538)
godot.osx.tools.x86_64.mono!VisualServerRaster::texture_create() (/servers/visual/visual_server_raster.h:145)
godot.osx.tools.x86_64.mono!VisualServerWrapMT::texture_create() (/servers/visual/visual_server_wrap_mt.h:82)
godot.osx.tools.x86_64.mono!ImageTexture::ImageTexture() (/scene/resources/texture.cpp:435)
godot.osx.tools.x86_64.mono!ImageTexture::ImageTexture() (/scene/resources/texture.cpp:431)
godot.osx.tools.x86_64.mono!Ref<ImageTexture>::instance() (/core/reference.h:285)
godot.osx.tools.x86_64.mono!DynamicFontAtSize::_bitmap_to_character(FT_Bitmap_, int, int, float) (/scene/resources/dynamic_font.cpp:511)
godot.osx.tools.x86_64.mono!DynamicFontAtSize::_update_char(wchar_t) (/scene/resources/dynamic_font.cpp:610)
godot.osx.tools.x86_64.mono!DynamicFontAtSize::get_char_size(wchar_t, wchar_t, Vector<Ref<DynamicFontAtSize> > const&) const (/scene/resources/dynamic_font.cpp:252)
godot.osx.tools.x86_64.mono!DynamicFont::get_char_size(wchar_t, wchar_t) const (/scene/resources/dynamic_font.cpp:846)
godot.osx.tools.x86_64.mono!Font::get_string_size(String const&) const (/scene/resources/font.cpp:493)
godot.osx.tools.x86_64.mono!MethodBind1RC<Vector2, String const&>::ptrcall(Object*, void const**, void*) (/core/method_bind.gen.inc:1340)
godot.osx.tools.x86_64.mono!godot_icall_1_110(MethodBind*, Object*, _MonoString*, GDMonoMarshal::M_Vector2*) (/modules/mono/glue/mono_glue.gen.cpp:1092)
[Unknown/Just-In-Time compiled code] (Unknown Source:0)

Again OpenGL call from the thread.

@rick551a
Copy link

rick551a commented Mar 5, 2021

Just FYI, error does not occur on Windows 10 on 3.23 stable or 3.24 beta6, 3.2.4rc2 or 3.2.4. rc3

@bruvzg
Copy link
Member

bruvzg commented Mar 5, 2021

For the reference in the pre-#45618 build, glDeleteTextures and glGenTextures are called only from the main thread.

@bruvzg
Copy link
Member

bruvzg commented Mar 5, 2021

Probably reproducibility depends on OpenGL driver implementation, not the platform.

@bruvzg
Copy link
Member

bruvzg commented Mar 5, 2021

The commit causing the issue is Modernize Thread, RWLock, Mutex, Semaphore and Atomics commits seems OK.

It's not caused by difference between std::thread and pthread (if there are any), after replacing all std::thread calls in the os/thread.cpp with pthread equvivalents issue still there.

@bruvzg
Copy link
Member

bruvzg commented Mar 5, 2021

It probably has something to do with ids, at least it's the only thing that is different.

Edit: Yep, it's ID collision between Mono and Godot threads. Mono thread causing crash do have the same it as the main thread (which is also server thread).

if (Thread::get_caller_id() != server_thread) { \

This check fails, and it's calling OpenGL API in the thread, instead of pushing it to command query.

@akien-mga
Copy link
Member

Indeed, see #46411 (comment) where @neikeq has come to the same conclusion (he's preparing a fix).

@bruvzg
Copy link
Member

bruvzg commented Mar 5, 2021

Indeed, see #46411 (comment) where @neikeq has come to the same conclusion (he's preparing a fix).

In case it's useful, here's my patch, which fixes crash on macOS (I have not tested it on any other platforms).

diff --git a/core/os/thread.cpp b/core/os/thread.cpp
index 58b29b9802..61cc31e154 100644
--- a/core/os/thread.cpp
+++ b/core/os/thread.cpp
@@ -41,9 +41,13 @@ void (*Thread::set_priority_func)(Thread::Priority) = nullptr;
 void (*Thread::init_func)() = nullptr;
 void (*Thread::term_func)() = nullptr;
 
-Thread::ID Thread::main_thread_id = 1;
-SafeNumeric<Thread::ID> Thread::last_thread_id{ 1 };
-thread_local Thread::ID Thread::caller_id = 1;
+uint64_t _thread_id_hash(const std::thread::id &p_t) {
+	static std::hash<std::thread::id> hasher;
+	return hasher(p_t);
+}
+
+Thread::ID Thread::main_thread_id = _thread_id_hash(std::this_thread::get_id());
+thread_local Thread::ID Thread::caller_id = _thread_id_hash(std::this_thread::get_id());
 
 void Thread::_set_platform_funcs(
 		Error (*p_set_name_func)(const String &),
@@ -57,7 +61,7 @@ void Thread::_set_platform_funcs(
 }
 
 void Thread::callback(Thread *p_self, const Settings &p_settings, Callback p_callback, void *p_userdata) {
-	Thread::caller_id = p_self->id;
+	Thread::caller_id = _thread_id_hash(p_self->thread.get_id());
 	if (set_priority_func) {
 		set_priority_func(p_settings.priority);
 	}
@@ -73,7 +77,7 @@ void Thread::callback(Thread *p_self, const Settings &p_settings, Callback p_cal
 }
 
 void Thread::start(Thread::Callback p_callback, void *p_user, const Settings &p_settings) {
-	if (id != 0) {
+	if (id != _thread_id_hash(std::thread::id())) {
 #ifdef DEBUG_ENABLED
 		WARN_PRINT("A Thread object has been re-started without wait_to_finish() having been called on it. Please do so to ensure correct cleanup of the thread.");
 #endif
@@ -81,22 +85,22 @@ void Thread::start(Thread::Callback p_callback, void *p_user, const Settings &p_
 		std::thread empty_thread;
 		thread.swap(empty_thread);
 	}
-	id = last_thread_id.increment();
 	std::thread new_thread(&Thread::callback, this, p_settings, p_callback, p_user);
 	thread.swap(new_thread);
+	id = _thread_id_hash(thread.get_id());
 }
 
 bool Thread::is_started() const {
-	return id != 0;
+	return id != _thread_id_hash(std::thread::id());
 }
 
 void Thread::wait_to_finish() {
-	if (id != 0) {
+	if (id != _thread_id_hash(std::thread::id())) {
 		ERR_FAIL_COND_MSG(id == get_caller_id(), "A Thread can't wait for itself to finish.");
 		thread.join();
 		std::thread empty_thread;
 		thread.swap(empty_thread);
-		id = 0;
+		id = _thread_id_hash(std::thread::id());
 	}
 }
 
@@ -109,10 +113,10 @@ Error Thread::set_name(const String &p_name) {
 }
 
 Thread::Thread() :
-		id(0) {}
+		id(_thread_id_hash(std::thread::id())) {}
 
 Thread::~Thread() {
-	if (id != 0) {
+	if (id != _thread_id_hash(std::thread::id())) {
 #ifdef DEBUG_ENABLED
 		WARN_PRINT("A Thread object has been destroyed without wait_to_finish() having been called on it. Please do so to ensure correct cleanup of the thread.");
 #endif
diff --git a/core/os/thread.h b/core/os/thread.h
index 837f9323c3..38763e2ebc 100644
--- a/core/os/thread.h
+++ b/core/os/thread.h
@@ -62,7 +62,6 @@ private:
 	friend class Main;
 
 	static ID main_thread_id;
-	static SafeNumeric<ID> last_thread_id;
 
 	ID id;
 	static thread_local ID caller_id;

Edit: Added default std::thread::id() values instead of zeros.

@kalysti
Copy link
Contributor

kalysti commented Mar 6, 2021

Can confirm. Patch is working under windows to. Needs to be implement to rc4

@RandomShaper
Copy link
Member

@neikeq is also working on it.

The only (minor) downside of this approach is that we lose the human-friendly, serial thread ids, but I don't think that will really be a problem in practice.

If at some point the debugger lets you see the list of threads so the serial numbering is desired, we can always revisit that. Getting the multiple issues related to this fixed is the top priority now, to unblock the release.

That said, let's see what we comes up with.

@neikeq
Copy link
Contributor

neikeq commented Mar 6, 2021

@RandomShaper My idea was something like this:

struct ThreadID {
    ID id;
    bool assigned = false;
};
static ThreadID ID caller_id;
static ID get_caller_id() { if (!caller_id.assigned) { assign_id(); } return caller_id.id; }

Godot threads should already have an id pre-assigned in Thread::callback as we do now.

This solution has the benefit of human-friendly ids. However, maybe your solution is better for performance as there's no if condition in get_caller_id. I don't mind either so whichever you prefer.

@RandomShaper
Copy link
Member

The only thing I can say is that performance beats human-friendly ids, so, if I had to choose, I'd go @bruvzg's way.

@bruvzg
Copy link
Member

bruvzg commented Mar 6, 2021

The only (minor) downside of this approach is that we lose the human-friendly, serial thread ids, but I don't think that will really be a problem in practice.

std::thread::id are not human-friendly indeed, but serial IDs doesn't seem to have one-to-one correspondence with the thread IDs in the debugger anyway (at least with LLDB output).

Here's the same patch as PR - #46742

@RandomShaper
Copy link
Member

std::thread::id are not human-friendly indeed, but serial IDs doesn't seem to have one-to-one correspondence with the thread IDs in the debugger anyway (at least with LLDB output).

I meant the Godot debugger, if it some day it gets features to debug GDScript threads. But, again, that's not relevant now. If that happens at some point, we will just decide how to deal with it then.

@bruvzg
Copy link
Member

bruvzg commented Mar 7, 2021

I meant the Godot debugger, if it some day it gets features to debug GDScript threads. But, again, that's not relevant now. If that happens at some point, we will just decide how to deal with it then.

For Godot debugger we can add get/set_thread_name() and Map<ID, String> to make it even more readable.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment