ES|QL: Implement heap attack tests for histograms

[JonasKunz]

Part of #140508.

Implements heap attack tests for tdigest and exponential_histogram.
I also thought about adding heap-attack tests for histogram, but the only thing you can do with histogram in ES|QL currently is to cast it to tdigest. So the test would end up testing tdigest again, which seemed pointless to me.

[elasticsearchmachine]

Pinging @elastic/es-storage-engine (Team:StorageEngine)

[not-napoleon]

There is a to_string for histogram, if we want to stress test that.

[not-napoleon]

I think the test itself looks good. It's probably worth adding a version that does the same thing with a legacy histogram, just in case something is bonkers in the casting logic.

What about testing a single, pathologically large histogram?

[not-napoleon]

Do we actually need to do this construction here? we have randomized functions that build both t-digests and exponential histograms already, and the storage classes have the ability to render as json. I don't love adding another place we randomly generate t-digests, especially ones that aren't actually built as t-digests

[JonasKunz]

We want deterministically sized data for these tests, therefore I think this is required.

And for the exponential histogram the code is really just histogram construction plus calling the serializer, so no duplicate logic here.

storage classes have the ability to render as json

This is the case for exponential histograms, but is that also possible for T-Digest? I couldn't find reusable code there. It should definitely be a a future refactor to remove this duplicate serialization logic.

[JonasKunz]

It's probably worth adding a version that does the same thing with a legacy histogram, just in case something is bonkers in the casting logic.

My understanding here is that all heap-attack tests "grow" stuff in memory, e.g. through computations or duplications, in the compute engine. To my knowledge it isn't possible to do this for histogram. So the only option there to make the ciruit breaker trip would be to ingest a lot of data into ES first. That in turn however would make the blockloader circuit breakers trip, which is not the point of this tests (to my understanding).

Therefore I don't see use in adding this kind of test for histogram.

[nik9000]

I think it's good. We might need to do more cruel things later - like ingest a single giant histogram and to_string it. That's probably more compelling if we have a very dense ingest format for histograms.

[nik9000]

Cute.

[nik9000]

What about testing a single, pathologically large histogram?

I'd love to get one of these in to be honest.

I do think it'd be worth doing the other type, just to be paranoid. It's not about the code being the same now, it's about what we'll do in six months without noticing.

What about TO_STRING and TO_TDIGEST? TO_STRING could, for example, not be tracking the memory of each element. That's fine for numbers, but for these biggest types that seems like a thing.

TO_TDIGEST looks to be limited to 2mb histograms. I see that it is using

        List<Double> centroids = new ArrayList<>();
        List<Long> counts = new ArrayList<>();

which isn't tracked. I might send it a 1.99mb histogram and a 2.01mb histogram. Just out of paranoia.

[JonasKunz]

@nik9000 I'm having trouble of constructing test for to_string / to_tdigest which are constructed so that when scaling up they actually explode in those functions and not somewhere else first.

To my understanding this is the general pattern for these tests: Find some way of scaling up the memory usage in a particular function that we want to test for robustness. So for example for CONCAT we can do this easily by adding a single eval doing multiple different CONCATS that the planner cannot optimize away. This way, the memory usage before the EVAL is constant and we grow it targeted in the CONCATS.

The same is not possible for TO_TDIGEST or TO_STRING. E.g. if you do EVAL a = TO_STRING(histo), b = TO_STRING(histo), ... the planner is too smart and optimized these duplicate TO_STRING invocations away.

So the only option would be to gradually grow histo in size. It is currently not possible to construct histograms in ES|QL.
Therefore, the only option to do this would be to do in a loop:

• Insert a large histogram into ES
• Query that single histogram with TO_TDIGEST
• If the circuitbreaker didn't trip, double the size for the next iteration and repeat

However, here we still don't guarantee that we actually test TO_TDIGEST, because we also grow the amount of data we load in contrast to the other tests. That means the circuitbreaker is likely to just trip while blockloading, unless we are lucky and hit the sweetspot where the block just barely fits into the circuit breaker, while the TO_TDIGEST result does not anymore.

For TO_STRING we'd also have to change the testing entirely, because we fail the method call with an IllegalArgumentException if the input is too big instead of causing the circuit breaker to trip. I've actually added a function unit test for that in #140718.

That's why I'm unsure on how to proceed here. If you tell me what approach to take for testing TO_STRING / TO_TDIGEST I can do that. I just think that my approach suggested above doesn't actually test what we want to test.

I do think it'd be worth doing the other type, just to be paranoid. It's not about the code being the same now, it's about what we'll do in six months without noticing.

I can add it as it is not much effort, but I think it will never test the thing we want to test. E.g. I would use the same testing I did for tdigest, but use histogram ES documents and prepend an EVAL histo = TO_TDIGEST(histo) before the STATS. But like explained above, this will be executed with constant memory usage, the memory growth happens in the STATS afterwards, where we are already back on the exact same codepath as for the tdigest tests.

[JonasKunz]

I've added now the tests for the histogram type and keeping it otherwise as-is. If we can resolve my concerns from the previous comment somehow, we can later add tests for TO_STRING.