Shovel metrics

[markmasl]

Hi!
We are actively using rabbitmq_shovel and rabbitmq_shovel_management plugins.
At this moment rabbit exporter do not support monitoring of shovel status, so I believe it would be very great to have such an option.
Thanks!

[kbudde]

@markmasl I will have a look into the api and if I find some useful data I'm going to integrate it.
Some data should already be exported in exchange and queue metrics.

[markmasl]

@kbudde Thanks for reply!
Metrics for the shovel connections can be found under /api/shovels.
Anyways, now I realized, that possibly it's not that straight forward, because, as regards to shovel state, rabbitmq returns strings:
"starting","running","terminated". I assume some sort of mapping is necessary, to convert these strings to gauge metric.
What are your thoughts on this?

Here are couple examples of returned json output:
[{"node":"my-rabbit@4a6df52ebc2a", "timestamp":"2019-04-23 10:32:08", "name":"test-shovel", "vhost":"/", "type":"dynamic", "state":"terminated", "reason":"{failed_to_connect_using_provided_uris,\n [{rabbit_amqp091_shovel,make_conn_and_chan,2,\n [{file,\"src/rabbit_amqp091_shovel.erl\"},{line,324}]},\n {rabbit_amqp091_shovel,connect_source,1,\n [{file,\"src/rabbit_amqp091_shovel.erl\"},{line,78}]},\n {rabbit_shovel_worker,handle_cast,2,\n [{file,\"src/rabbit_shovel_worker.erl\"},{line,64}]},\n {gen_server2,handle_msg,2,[{file,\"src/gen_server2.erl\"},{line,1056}]},\n {proc_lib,init_p_do_apply,3,[{file,\"proc_lib.erl\"},{line,249}]}]}"}]

and:
[{"node":"[email protected]","timestamp":"2019-04-17 1:01:11","name":"ADMIN-3779-1","vhost":"/","type":"dynamic","state":"running","src_uri":"amqp://","src_protocol":"amqp091","dest_protocol":"amqp091","dest_uri":"amqps://rabbitmq.example.com:5671/dev-1","src_exchange":"test.exchange","src_exchange_key":"EVENT_SNAPSHOT.#","dest_exchange":"test.event.snapshot.v1"}]

[kbudde]

@markmasl I think the best solution is to a constant metric with value 1 and the state as label.
This way you can sum by state (alert if there are more than x shovels in terminated state, ...) or alert on missing metrics (absent(rabbitmq_shovel_state{state="running"}))

Would this help?

# HELP rabbitmq_shovel_state A metric with a value of constant '1' for a shovel in a certain state
# TYPE rabbitmq_shovel_state gauge
rabbitmq_shovel_state{cluster="rabbit@089d4829cce7",type="dynamic",shovel="test-shovel",self="1",state="terminated",vhost="/"} 1
rabbitmq_shovel_state{cluster="rabbit@089d4829cce7",type="dynamic",shovel="ADMIN-3779-1",self="1",state="running",vhost="/"} 1

[markmasl]

@kbudde
Yes, that's probably the best way to implement this.
And, yes, that certainly will help.

[kbudde]

@markmasl I've created a new release containing the shovel metrics. You can run it the usual way.

[markmasl]

@kbudde Thank you!
Unfortunately I see some problems with scraping shovel metrics.
Running rabbit_Exporter with:
docker run -d --net=container:my-rabbit -e RABBIT_CAPABILITIES=bert,no_sort -e PUBLISH_PORT=9419 -e RABBIT_EXPORTERS=exchange,node,queue,shovel -e LOG_LEVEL=debug kbudde/rabbitmq-exporter:v1.0.0-RC6

Have seen following errors:

time="2019-04-26T09:37:16Z" level=error msg="Error while retrieving data from rabbitHost" error="<nil>" host="http://127.0.0.1:15672" statusCode=406
time="2019-04-26T09:37:16Z" level=warning msg="retrieving shovel failed" error="Error while retrieving data from rabbitHost"

# TYPE rabbitmq_module_up gauge rabbitmq_module_up{cluster="my-rabbit@4a6df52ebc2a",module="exchange",node="my-rabbit@4a6df52ebc2a"} 1 rabbitmq_module_up{cluster="my-rabbit@4a6df52ebc2a",module="node",node="my-rabbit@4a6df52ebc2a"} 1 rabbitmq_module_up{cluster="my-rabbit@4a6df52ebc2a",module="overview",node="my-rabbit@4a6df52ebc2a"} 1 rabbitmq_module_up{cluster="my-rabbit@4a6df52ebc2a",module="queue",node="my-rabbit@4a6df52ebc2a"} 1 rabbitmq_module_up{cluster="my-rabbit@4a6df52ebc2a",module="shovel",node="my-rabbit@4a6df52ebc2a"} 0
But if I curl to port 15672, I get:

[root@localhost rabbitmq_exporter]# curl guest:guest@localhost:15672/api/shovels
[{"node":"my-rabbit@4a6df52ebc2a","timestamp":"2019-04-26 10:27:16","name":"test-shovel","vhost":"/","type":"dynamic","state":"terminated","reason":"{failed_to_connect_using_provided_uris,\n    [{rabbit_amqp091_shovel,make_conn_and_chan,2,\n         [{file,\"src/rabbit_amqp091_shovel.erl\"},{line,324}]},\n     {rabbit_amqp091_shovel,connect_source,1,\n         [{file,\"src/rabbit_amqp091_shovel.erl\"},{line,78}]},\n     {rabbit_shovel_worker,handle_cast,2,\n         [{file,\"src/rabbit_shovel_worker.erl\"},{line,64}]},\n     {gen_server2,handle_msg,2,[{file,\"src/gen_server2.erl\"},{line,1056}]},\n     {proc_lib,init_p_do_apply,3,[{file,\"proc_lib.erl\"},{line,249}]}]}"}]

Could you, please, check what could be wrong?
Thank you in advance!

[kbudde]

Hi,

the shovel plugin does not support bert encoding. The exporter now prefers bert but also accepts json encoding for all requests.
It should work now in all cases.

There will be version available shortly v1.0.0-RC6.1

BR

[markmasl]

@kbudde
Hi,
Have tested finally. Now it works as expected. Many thanks to you!

[kbudde]

Thank you für your feedback.