Reenable uint64 support

[urso]

We currently mask all 64bit unsigned integer values using a 63bit mask. With most counters being ever increasing the results in a simple overflow every 2^63 increments. Fortunately loads of values requiring 64bit are counters that we only compute the rate from, so it's no big problem. But 64bit IDs and and status fields can currently not be stored correctly.

Elasticsearch issue discussing to add uint64 bit integers: elastic/elasticsearch#32434
And in progress PR: elastic/elasticsearch#60050

Once we have uint64 bit support we might be able to remove the 63bit mask. We still might need to check for the Elasticsearch version and apply the mask if we index into an older version. But latest 8.0 the mask should be removed if possible.

[elasticmachine]

Pinging @elastic/integrations-services (Team:Services)

[jpountz]

We still might need to check for the Elasticsearch version and apply the mask if we index into an older version. But latest 8.0 the mask should be removed if possible.

Given that we recommend upgrading Elasticsearch first, Beats should always be on a version <= Elasticsearch, so we shouldn't have to check the Elasticsearch version? I remember Beats used to try to retain compatibility for older Elasticsearch versions, but I thought we had stopped doing this?

[urso]

I remember Beats used to try to retain compatibility for older Elasticsearch versions, but I thought we had stopped doing this?

This is still the case. All Beats 7.x will try to be compatible with all 7.x Elasticsearch versions. The version constraint will be for Agent.

[ruflin]

@urso If my understanding of elastic/elasticsearch#60050 is correct, this is a new field type. For all the fields that we define as long and not unsigned-long in our mappings, we will still need 63bit mask. What I'm getting at, we first need to update the mappings in the packages / modules before we remove this?

[urso]

yes, mappings need to be updated. In code we should use int64 if we want long. The bitmask is only applied to uint64.

[mayya-sharipova]

I am wondering what kind of queries and aggregations are necessary for unsigned_long field (for "id" use case and for "counter" use case)? Do we need any numeric metric aggregations (e.g. sum/avg etc)?

Also do we need mixing longs and unsigned longs when sorting on a field (e.g. search request across indices where a field can be long in one index, and unsigned_long in another)?

[urso]

I am wondering what kind of queries and aggregations are necessary for unsigned_long field (for "id" use case and for "counter" use case)? Do we need any numeric metric aggregations (e.g. sum/avg etc)?

It depends on what the field represents (e.g. physical unit).

If the field is an ID, then counts (and aggregations on the counts) would be the only ones that make sense.

If the field is a "counter", then we often want the derivative (the rate like IO ops).

All in all for metrics there is no real difference for us regarding long vs. unsigned long. The advantage is that we can store the raw values as is (especially if the unsigned long is an ID).

Also do we need mixing longs and unsigned longs when sorting on a field (e.g. search request across indices where a field can be long in one index, and unsigned_long in another)?

It depends when/how we do the change in Beats. e.g. if we enable unsigned long support for new integrations/packages only, we might be somewhat on the safe side.

Will it be an issue? My understanding was that ES treats numeric values for aggregations as double.

[urso]

Uint64 support has been merged into Elasticsearch and seems to become available with 7.11.

ECS ticket: elastic/ecs#900

[mayya-sharipova]

seems to become available with 7.11.

@urso This field type is available from 7.10

[urso]

@mayya-sharipova Thank you :)

[botelastic]

This issue has been automatically marked as stale because it has not had recent activity. It will be closed if no further activity occurs. Thank you for your contributions.