Review mapping templates of beats

[jpountz]

I was looking at templates after the switch to scaled/half floats and found a couple potential issues, maybe we should do a review of these templates before the 5.0 release. For instance:

• apache.status.total_accesses and apache.status.total_kbytes are currently an integers but a high traffic instance could overflow them I think? Should it be a long?
• apache.status.requests_per_sec is a half float, which means that the integer value will become fuzzy after 2048, and rounded to +Infinity after 65504, which are values that could be reached on a high-traffic server that serves small static files. Maybe they should be integers instead? or scalled floats if we really want to be able to have some decimal places.

There are also some inconsistencies, for instance most CPU usage stats seem to have moved from half floats to scaled floats, but the cpu stats of redis are still using half floats? Is it intentional? Similarly, the system load averages are using scaled floats while the apache load averages are using half floats.

[ruflin]

Thanks @jpountz for taking time to review the fields. First the good news for int / long: We automatically convert all integer to float in the templates we generate: https://github.com/elastic/beats/blob/master/metricbeat/metricbeat.template.json#L137 Based on this, all values sent to elasticsearch are mapped as long as we didn't see any disadvantage in this. Please tell us if that isn't a good idea. That doesn't mean we should not update the templates but we saw it as lower priority.

For request_per_sec I had to check the apache docs. It is indeed a float returned by apache and it can have quite a few decimals. The question is probably how many we want to use. As you suggested, scaled_floats would be a better option here.

I agree that we should do a full review of the existing Metricsets for the mapping before beta1 to make sure we use the correct values. I think all the points you pointed out about redis for example are inconsistencies that have to be fixed and I'm quite sure there are more. Currently our focus on the system module.

An additional task we should do when reviewing all event documents is to check for consistency in naming: https://www.elastic.co/guide/en/beats/libbeat/master/event-conventions.html I'm mentioning this here as I think this could be done in one go.

[jpountz]

Based on this, all values sent to elasticsearch are mapped as long as we didn't see any disadvantage in this.

The benefit of integer vs long is indeed quite small, so that would be fine. Thanks for explaining!

[ruflin]

I reviewed all the half_float, scaled_float and float and opened #2430 with the changes. Here some observations:

• All the usage of half_float was removed. half_float is not a good fit for most of our metrics as they can grow > 2048 or even > 65504 (@jpountz Thanks for spotting this, wasn't aware especially of the > 2048 part). All half_float were replaced by scaled_float with scaling_factor 1000
• All the usage of float was replaced by scaling_float. It seems there aren't currently any metrics where more then 3 digits after the commas are needed. Most of these values with scaled_floats are either percentages or average values where the digits after the comma do not matter too much.

Conclusion: scaled_floats are awesome and seem to be the perfect fit for our use case.

As default we currently use scaling_factor: 1000 and we have very few values were we use scaling_factor: 100 as we know the system will only send us two digits. It would be interesting to know what the difference would be if we would go for simplicity also with scaling_factor: 1000 for these values.

[jpountz]

All half_float were replaced by scaled_float with scaling_factor 1000

This might not be an issue, but note that this will make scaled floats use more disk than floats if the amplitude of the values in a field grows beyond 2^32/scaling_factor.

It would be interesting to know what the difference would be if we would go for simplicity also with scaling_factor: 1000 for these values.

This would increase disk usage by about 2.3 bits per value.