Make prometheus plugin output customizable

[binarin]

Proposed Changes

Prometheus plugin always exposes all possible metrics (even in aggregated mode, where the output is smaller, but the same amount of data still needs to be processed), and this can be problematic when there is a large number of connections/channels/queues.

For existing endpoints this change allows to disable some metric families altogether or filter out some vhosts (but this is optional, and without configuration change everything is 100% backwards compatible). This can greatly reduce the amount of data exposed, while still providing enough details for a proper monitoring.

And then there is a new endpoint where the same customizations can be applied on the fly, in case more information is needed, or with different level of details.

Types of Changes

What types of changes does your code introduce to this project?
Put an x in the boxes that apply

• Bug fix (non-breaking change which fixes issue #NNNN)
• New feature (non-breaking change which adds functionality)
• Breaking change (fix or feature that would cause an observable behavior change in existing systems)
• Documentation improvements (corrections, new content, etc)
• Cosmetic change (whitespace, formatting, etc)

Checklist

• I have read the CONTRIBUTING.md document
• I have signed the CA (see https://cla.pivotal.io/sign/rabbitmq)
• I have added tests that prove my fix is effective or that my feature works
• All tests pass locally with my changes
• If relevant, I have added necessary documentation to https://github.com/rabbitmq/rabbitmq-website
• If relevant, I have added this change to the first version(s) in release-notes that I expect to introduce it

Further Comments

[vikinghawk]

This looks really promising!

Our usecase is probably similar to many others... we are only interested in queue message & consumer count at the per-object granularity. But scraping the current all or nothing per-object endpoint with 10k+ queues can take a long time (60+ seconds) and causes high memory usage on the node.

Couple questions:

1. Have you guys decided what releases this will be targeted for? Specifically, is it possible to be backported to 3.8.x?
2. Would it be feasible to add "rabbitmq_queue_consumer_utilisation" to the "queue_consumer_count" group? It would be useful to see the current consumer utilization metrics alongside the count

[binarin]

1. Have you guys decided what releases this will be targeted for? Specifically, is it possible to be backported to 3.8.x?

For now we are planning to do it, unless there will be some unexpected problem.

2. Would it be feasible to add "rabbitmq_queue_consumer_utilisation" to the "queue_consumer_count" group? It would be useful to see the current consumer utilization metrics alongside the count

No, but I've did a bit of benchmarking on a 10k queues/10k publishers/10k consumers rabbit:

• Current per-object takes around 2 minutes.
• queue_coarse_metrics and queue_consumer_count - 1.3 second
• queue_metrics - 8.5 seconds

If you need utilization, you'll be able to scrape it now in a more reasonable fashion anyway.

[luos]

Hi @binarin, seems like this did not get backported by Mergify to v3.8.x. Do you think there is a chance this could be merged into it? Thanks

[sherifkayad]

@binarin similar to @luos I am missing this commit in 3.8.x .. any plans to include it there? .. I submitted an issue for that.