Ginormous recordings with only a handful of rows obliterate the GC stack #4185
Assignees
Labels
😤 annoying
Something in the UI / SDK is annoying to use
🪳 bug
Something isn't working
⛃ re_datastore
affects the datastore itself
📺 re_viewer
affects re_viewer itself
Comments
teh-cmc
added
🪳 bug
4 tasks
teh-cmc
added a commit
that referenced
this issue
Nov 9, 2023
**Commit by commit, there's renaming involved!** GC will now focus on the oldest-modified recording first. Tried a lot of fancy things, but a lot of stress testing has shown that nothing worked as well as doing this the dumb way. Speaking of stress testing, the scripts I've used are now committed in the repository. Make sure to try them out when modifying the GC code :grimacing:. In general, the GC supports stress much better than I thought/hoped: - `many_medium_sized_single_row_recordings.py`, `many_medium_sized_many_rows_recordings.py` & `many_large_many_rows_recordings.py` all behave pretty nicely, something like this: https://github.com/rerun-io/rerun/assets/2910679/26f67d69-de0e-4002-8936-2ac32c451cc3 - `many_large_single_row_recordings.py` on the other hand is _still_ a disaster (watch til the end, this slowly devolves into a blackhole): https://github.com/rerun-io/rerun/assets/2910679/673ee10c-2eca-4e3e-b285-77714e5c3d61 This is not a new problem (not to me at least 😬), large recordings with very few rows have always been a nightmare on the GC (not specifically the DataStore GC, the GC as a whole through the entire app). I've never had time to investigate why, but now we have an issue for it at least: - #4185 --- - Fixes #1904
|
What am I looking at in the video? 😅 |
The GC going into a death spiral and drowning under its own weight: This might very well be fixed at this point though, I need to try this again with all the improvements that have recently landed. |
|
I can confirm that:
|
|
I randomly reproduced this while working on primary caching: Like, what even is going on here? The store is literally empty, all recordings are gone! Are we leaking some arrow references or something? |
|
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Our GC generally responds well to stress (see scripts in
tests/python/gc_stress), but for some reason recordings with one to a few very large rows run into a degenerate case that completely obliterates it.The exact script to trigger this is many_large_single_row_recordings.py.
In practice this ends up looking like this (watch til the end):
23-11-08_17.00.12.patched.mp4
The text was updated successfully, but these errors were encountered: