Feature request: including ARV_IS_BUFFER in this library

[denglerchr]

Good morning,

I am still running into issues with my camera that I cannot seem to resolve. The error I keep getting is that the assertion for ARV_IS_BUFFER fails.

I wanted to ask if it is possible to have the ARV_IS_BUFFER function made available in Julia in this library, so that I could call this check manually already in Julia before calling the arv_buffer_get_stats that keeps crashing for me?

[jwahlstrand]

Hmm, the fact that it's happening in the finalizer is a sign of something deeper. It looks like the buffer is being freed by libaravis but Julia thinks it's still valid. You might be able to prevent it by calling Gtk4.GLib.glib_ref(buf) on the buffer after you pop it -- this adds a reference to the buffer, so if libaravis dereferences it once, the object will still survive. Ordinarily I would be wary of this because of the potential for a massive memory leak, but since the buffers are provided by a ring buffer I don't think that's a big concern. Any memory leak will not grow indefinitely, if I understand things right. Normally, if a reference has to be added it should happen in the method provided by AravisCameras -- I will try to look into it tomorrow.

ARV_IS_BUFFER is a C macro (I think) that just checks that the GType of the object is an ArvBuffer. The equivalent in Julia is:

function ARV_IS_BUFFER(buf)
    Gtk4.GLib.G_OBJECT_CLASS_TYPE(buf) == Gtk4.GLib.g_type(ArvBuffer)
end

Certainly we could add it to AravisCameras but like I said I think the issue is deeper.

[denglerchr]

I am not fully sure about the source of this error, maybe there are even multiple sources? While it always complains about the "ARV_IS_BUFFER" at some point, the julia stack trace is not always the same, see for example the screenshots here #4 where it appeared to be an "InexactError" in Julia (it didnt print more odf the stack trace there unfortunately) , or other cases where the error was preceded by "G_IS_OBJECT ( object) failed".
Ill try the Julia version of ARV_IS_BUFFER though for sure, maybe it will make things more robust.
Also thanks a lot for your help!

[jwahlstrand]

OK, I looked into this. I'm seeing that pop_buffer and try_pop_buffer sometimes return pointers to objects that have been freed already. I'm pretty sure this is caused by a serious bug with how Gtk4.jl handles the memory lifecycle of GObjects. I don't know how to fix it yet, but I did just push some changes that seem to work around the issue, in particular adding a call to glib_ref to the ArvBuffer constructor. This seems to prevent buffers from being freed early. If I have more time this weekend I'll look further into it, but see if this helps.

Gtk4.jl, at this point, is pretty robust when it comes to wrapping libgtk4, but this package managed to uncover a new problem. Thanks for persisting!

[denglerchr]

Good afternoon,

I got back to the camera just now, unfortunately the change seems to cause new errors, especially related to the finalizer if I see it correctly (snippets 1 and 2 below). Also I sometimes get segmentation faults now, mostly without much stacktrace, but once I got some, see snippet 3.

error in running finalizer: ErrorException("val already in a list")
error at ./error.jl:35
push! at ./linked_list.jl:53 [inlined]
_wait2 at ./condition.jl:87
#wait#731 at ./condition.jl:127
wait at ./condition.jl:125 [inlined]
slowlock at ./lock.jl:157
lock at ./lock.jl:147 [inlined]
macro expansion at ./lock.jl:271 [inlined]
delref at /home/christian/.julia/packages/Gtk4/tHG3O/src/GLib/gtype.jl:325
jfptr_delref_5482 at /home/christian/.julia/compiled/v1.11/Gtk4/rFcBQ_pNG21.so (unknown line)
run_finalizer at /cache/build/tester-amdci5-12/julialang/julia-release-1-dot-11/src/gc.c:299
jl_gc_run_finalizers_in_list at /cache/build/tester-amdci5-12/julialang/julia-release-1-dot-11/src/gc.c:389
run_finalizers at /cache/build/tester-amdci5-12/julialang/julia-release-1-dot-11/src/gc.c:435
enable_finalizers at ./gcutils.jl:161 [inlined]
unlock at ./locks-mt.jl:68 [inlined]
push! at ./task.jl:725
enq_work at ./task.jl:813
yield at ./task.jl:900
g_siginterruptible at /home/christian/.julia/packages/Gtk4/tHG3O/src/GLib/signals.jl:244
g_yield at /home/christian/.julia/packages/Gtk4/tHG3O/src/GLib/signals.jl:265
jfptr_g_yield_5790 at /home/christian/.julia/compiled/v1.11/Gtk4/rFcBQ_pNG21.so (unknown line)
jlcapi_g_yield_5872 at /home/christian/.julia/compiled/v1.11/Gtk4/rFcBQ_pNG21.so (unknown line)
uv_dispatch at /home/christian/.julia/packages/Gtk4/tHG3O/src/GLib/signals.jl:363
jfptr_uv_dispatch_5839 at /home/christian/.julia/compiled/v1.11/Gtk4/rFcBQ_pNG21.so (unknown line)
jlcapi_uv_dispatch_5861 at /home/christian/.julia/compiled/v1.11/Gtk4/rFcBQ_pNG21.so (unknown line)
g_main_dispatch at /home/christian/.julia/artifacts/6d8a1f48d4b47dc03fa048d88680be84ae15e5fc/lib/libglib-2.0.so.0 (unknown line)
g_main_context_iterate_unlocked.isra.29 at /home/christian/.julia/artifacts/6d8a1f48d4b47dc03fa048d88680be84ae15e5fc/lib/libglib-2.0.so.0 (unknown line)
g_main_context_iteration at /home/christian/.julia/artifacts/6d8a1f48d4b47dc03fa048d88680be84ae15e5fc/lib/libglib-2.0.so.0 (unknown line)
#237 at /home/christian/.julia/packages/Gtk4/tHG3O/src/GLib/loop.jl:83
unknown function (ip: 0x7f24201dbe4f)
g_sigatom at /home/christian/.julia/packages/Gtk4/tHG3O/src/GLib/signals.jl:211
unknown function (ip: 0x7f24201dbd22)
glib_main at /home/christian/.julia/packages/Gtk4/tHG3O/src/GLib/loop.jl:80
unknown function (ip: 0x7f24201daf9f)
jl_apply at /cache/build/tester-amdci5-12/julialang/julia-release-1-dot-11/src/julia.h:2157 [inlined]
start_task at /cache/build/tester-amdci5-12/julialang/julia-release-1-dot-11/src/task.c:1202

error in running finalizer: AssertionError(msg="Multiple concurrent writes to Dict detected!")
rehash! at ./dict.jl:182
ht_keyindex2_shorthash! at ./dict.jl:314
setindex! at ./dict.jl:356
unknown function (ip: 0x7ff47c1b2fe2)
finalize_gc_unref at /home/christian/.julia/packages/Gtk4/tHG3O/src/GLib/gtype.jl:310
delref at /home/christian/.julia/packages/Gtk4/tHG3O/src/GLib/gtype.jl:328
jfptr_delref_5482 at /home/christian/.julia/compiled/v1.11/Gtk4/rFcBQ_pNG21.so (unknown line)
run_finalizer at /cache/build/tester-amdci5-12/julialang/julia-release-1-dot-11/src/gc.c:299
jl_gc_run_finalizers_in_list at /cache/build/tester-amdci5-12/julialang/julia-release-1-dot-11/src/gc.c:389
run_finalizers at /cache/build/tester-amdci5-12/julialang/julia-release-1-dot-11/src/gc.c:435
run_finalizers at /cache/build/tester-amdci5-12/julialang/julia-release-1-dot-11/src/gc.c:416 [inlined]
ijl_gc_collect at /cache/build/tester-amdci5-12/julialang/julia-release-1-dot-11/src/gc.c:3921
maybe_collect at /cache/build/tester-amdci5-12/julialang/julia-release-1-dot-11/src/gc.c:922 [inlined]
jl_gc_pool_alloc_inner at /cache/build/tester-amdci5-12/julialang/julia-release-1-dot-11/src/gc.c:1325
jl_gc_alloc_ at /cache/build/tester-amdci5-12/julialang/julia-release-1-dot-11/src/julia_internal.h:523 [inlined]
_new_genericmemory_ at /cache/build/tester-amdci5-12/julialang/julia-release-1-dot-11/src/genericmemory.c:56 [inlined]
jl_alloc_genericmemory at /cache/build/tester-amdci5-12/julialang/julia-release-1-dot-11/src/genericmemory.c:99
GenericMemory at ./boot.jl:516 [inlined]
Array at ./boot.jl:578 [inlined]
Array at ./boot.jl:591 [inlined]
Array at ./boot.jl:599 [inlined]
similar at ./abstractarray.jl:868 [inlined]
similar at ./abstractarray.jl:867 [inlined]
similar at ./broadcast.jl:224 [inlined]
similar at ./broadcast.jl:223 [inlined]
copy at ./broadcast.jl:892 [inlined]
materialize at ./broadcast.jl:867 [inlined]
process_linescan_intensity at /home/christian/Desktop/Laser2MQTT/src/hdf5.jl:151
add_frame! at /home/christian/Desktop/Laser2MQTT/src/hdf5.jl:87
#mqtt_forwarder#9 at /home/christian/Desktop/Laser2MQTT/src/mqtt.jl:59
mqtt_forwarder at /home/christian/Desktop/Laser2MQTT/src/mqtt.jl:27 [inlined]
#11 at /home/christian/Desktop/Laser2MQTT/src/Laser2MQTT.jl:28
unknown function (ip: 0x7ff47c1a61df)
jl_apply at /cache/build/tester-amdci5-12/julialang/julia-release-1-dot-11/src/julia.h:2157 [inlined]
start_task at /cache/build/tester-amdci5-12/julialang/julia-release-1-dot-11/src/task.c:1202

.

[19744] signal 11 (128): Segmentation fault
in expression starting at /home/christian/Desktop/Laser2MQTT/test/vrmagic.jl:10
g_type_check_instance_is_fundamentally_a at /home/christian/.julia/artifacts/6d8a1f48d4b47dc03fa048d88680be84ae15e5fc/lib/libgobject-2.0.so.0 (unknown line)
g_object_get_qdata at /home/christian/.julia/artifacts/6d8a1f48d4b47dc03fa048d88680be84ae15e5fc/lib/libgobject-2.0.so.0 (unknown line)
convert_ at /home/christian/.julia/packages/Gtk4/tHG3O/src/GLib/gtype.jl:192
convert at /home/christian/.julia/packages/Gtk4/tHG3O/src/GLib/gtype.jl:182 [inlined]
try_pop_buffer at /home/christian/.julia/packages/AravisCameras/LlUSJ/src/gen/aravis_methods:3083 [inlined]
get_buffer_safe at /home/christian/Desktop/Laser2MQTT/src/cameras.jl:288
#start_acquisition#6 at /home/christian/Desktop/Laser2MQTT/src/cameras.jl:235
start_acquisition at /home/christian/Desktop/Laser2MQTT/src/cameras.jl:210 [inlined]
#10 at /home/christian/Desktop/Laser2MQTT/src/Laser2MQTT.jl:27
unknown function (ip: 0x7fa8ac196b6f)
jl_apply at /cache/build/tester-amdci5-12/julialang/julia-release-1-dot-11/src/julia.h:2157 [inlined]
start_task at /cache/build/tester-amdci5-12/julialang/julia-release-1-dot-11/src/task.c:1202
Allocations: 11384929 (Pool: 11384366; Big: 563); GC: 19
Segmentation fault (core dumped)

[jwahlstrand]

By "the change" do you mean the commit pushed a week ago? That commit partially reverted what was done before because I was seeing segfaults in my code. I've studied libaravis enough to be convinced that the wrapper is now doing the right thing for these methods. I did notice another segfault issue with destroying an ArvStream and creating a new one. I no longer do that (I just reuse the same stream) and have not seen a segfault since.

From the stacktraces it looks like you're seeing issues related to finalizing GObjects in multiple threads. Fixing that might require changes to Gtk4 -- basically we need to add locks around some of the code in GLib/gtype.c. Unfortunately it's the beginning of a work week for me; I will not have time to think about this until next weekend (this is a hobby project).

Does your code run with a single thread?

[denglerchr]

I didnt realise there were multiple changes, bascially "the change" applies to changes since around 20.12
Ill try tomorrow if my Julia program works with a single thread and report back. No worries if you cannot look into it now, it is not critical, I have an almost ready version with https://pypi.org/project/harvesters/ in docker currently. But Julia would probably be better suited for fast acquisition and processing.

[denglerchr]

Good afternoon,
I tried the a single threaded version for the camera acquisition. The program is stable then, I let it run for around 100000 frames, which is a lot further than I get with the multi-threaded version. Looks like the issues lies with GTK4 then.