-
Notifications
You must be signed in to change notification settings - Fork 25.6k
Open
Labels
:Analytics/AggregationsAggregationsAggregations>tech debtTeam:AnalyticsMeta label for analytical engine team (ESQL/Aggs/Geo)Meta label for analytical engine team (ESQL/Aggs/Geo)
Description
When we did #56487 we decided that it was important to do an inventory of all of the memory that aggregations allocate that is not part of BigArrays. We'd like to get everything tracked so we're less reliant on the real memory breaker catching stuff.
-
DeferringBucketCollectorsubclasses aren't tracked. -
matrix_stats'sRunningStatshas a bunch ofHashMaps that aren't being tracked properly. It looks like they don't grow a lot, but if you put in a high enough cardinality agg it could get messy. -
string_statshasMap<Character, LongArray>would could end up taking up a fair bit of untracked space if under a high cardinality agg and there are a bunch of characters. English's would see thatMaphas 64ish entries. Japanese and Chinese look like they'd consistently see a couple thousand entries in the Map. And the array won't work at all for things that aren't on the BMP like Emoji and Egyptian Hieroglyphs and a few unlucky languages. -
top_hitswill create a bunch ofCollectorswhich aren't tracked byBigArrays. They are all fairly careful with memory, but it could use a bit and we aren't tracking it. -
TDigestState, HDR histogram and friends look like they can use a fair bit of untracked memory. We could probably track a max for it or something like that (see also Integrate TDigestState with circuit breakers #99815). For HDR, the HDR histogram library needs to be forked first (Fork HdrHistogram library #95904) -
HyperLogLogPlusPlushas anOpenBitSetwhich has the same behavior as ourBitArraybut it isn't backed toBigArrays. Use standard bit set impl in cardinality #61816 -
DoubleHistogramand friends are also untracked. -
ScriptedMetricis totally untracked and frankly terrifying. -
filters's "compatible" collector can realize a bunch of bit sets in memory. Trigger parent circuit breaker when building scorers in filters aggregation #102511 - The reduction phase is all java object based (see Enable Circuit Breaker tracking in more parts of the aggregations framework #89437)
- While global ordinals memory usage is tracked, the process of building them isn't Check the real memory circuit breaker when building global ordinals #102462
Metadata
Metadata
Assignees
Labels
:Analytics/AggregationsAggregationsAggregations>tech debtTeam:AnalyticsMeta label for analytical engine team (ESQL/Aggs/Geo)Meta label for analytical engine team (ESQL/Aggs/Geo)