Add comments and tests for gc_string_view_batch #1
Conversation
| let gc_array = do_gc(array.clone()); | ||
| compare_string_array_values(&array, &gc_array); | ||
| assert_eq!(array.data_buffers().len(), 5); | ||
| // TODO this is failing now (it always compacts) |
There was a problem hiding this comment.
This test is failing now because the StringView is compacted even though all data in the buffers is pointed to
What I think this is showing is that string views with many long strings but no actual garbage are always compacted, which doesn't seem like a good heuristic
There was a problem hiding this comment.
I agree the heuristic is a bit arbitrary and not really backed by real experiments... I'll think a bit more about this and back to it later today
There was a problem hiding this comment.
FWIW I am going to prototype what the combined filter/coalsece thing we were talking about on apache#11628 might look like
There was a problem hiding this comment.
Prototype is here: apache#11647 (not yet ready for review)
Note this targets apache#11587
Rationale
While reviewing apache#11587 I felt something wasn't quite right with the heuristic, but I couldn't articulate it directly.
So I wrote some tests that show the problem