Add filtering option for prometheus collector

[ChrsMark]

Drafting what was discussed on #15493 to start implementation specific discussions.For the time being only filters are implemented, but mappings will be added if we agree on this.
@jsoriano I preferred to add the processing on collector layer to avoid touching GetFamilies since it is used by many other places like the state_* metricsets of k8s module. Not sure if we would like to have it in there (in the helper method).

What does this PR do?

This PR introduces filtering options for prometheus collector metricset.

Why is it important?

This is important when someone want to filter out metrics that come in a single response for instance in case of prometheus exporters.

This will be useful in light-modules like ibmmq module where currently we use script processor to filter out metrics as well as in OpenMetrics module where a user may want to explicitly define which metrics want to keep.

How to test this PR locally

1. One can use a nodexporter as an example Prometheus app to collect from. For instance this forked project should be enough: https://github.com/ChrsMark/dockprom. Download and docker-compose up. Then metrics should be exported at http://localhost:9100/metrics.
2. Configuration:

- module: prometheus
  period: 10s
  hosts: ["localhost:9100"]
  metrics_path: /metrics
  metrics_filters:
    include: ["node_filesystem_*"]
    exclude: ["node_filesystem_device_*", "^node_filesystem_readonly$"]

And expect to see only events that match node_filesystem_* but also not node_filesystem_device_* and are not node_filesystem_readonly.
3. Try to remove ^node_filesystem_readonly$ from exclude filters and see that this metric is included in events. One can try different combinations.

Related issues

• Relates [Metricbeat] Prometheus based module should support metrics filtering #15493, [Metricbeat] OpenMetrics Module #14982

[jsoriano]

This approach LGTM, I agree with limiting it to filtering by now, and we can consider adding mapping in the future.

I think we could accept using include_metrics and ignore_metrics at the same time, I added a comment about that.

I preferred to add the processing on collector layer to avoid touching GetFamilies since it is used by many other places like the state_* metricsets of k8s module. Not sure if we would like to have it in there (in the helper method).

👍 we can revisit this later if needed.

[jsoriano]

A couple of comments more, and I'd be ok with continuing with this proposed implementation. Thanks!

[ChrsMark]

Thanks for reviewing! I will work on the comments and I also plan to add some unit-tests maybe for the helper functions so as to secure the logic.

[ChrsMark]

Giving a heads-up on this:

• comments of @jsoriano have been addressed. Also took into consideration proposal of @exekias [Metricbeat] Prometheus based module should support metrics filtering #15493 (comment) to rename ignore configuration option to exclude.
• Unit tests added.
• Default configuration was updated

Still need to update the docs with these new settings description.