Plans for Job Configuration?

[ewr]

What are your plans re: job specifications? I see the service discovery question in #6, but I'm thinking more in terms of a LWRP that wrapper cookbooks can call.

I'm still trying to evaluate if Prometheus is going to be what I'm looking for, but I'm glad to help out on implementing that sort of thing over the next few days. I just want to make sure I'm generally pointed in the same direction you'd like to see the cookbook go.

[rayrod2030]

I think ultimately a LWRP is where we need to go in order to allow people to implement their own discovery mechanism and use this cookbook with a wrapper cookbook as you have mentioned. I was thinking of just doing something chef discovery specific using search but even that feature would benefit from using a LWRP to generate the job template so yes it's well worth it to write a job config LWRP.

[ewr]

Am I correct in not seeing any support for prometheus to read multiple config files? If so, that's going to be the complicated bit in terms of a job LWRP—you need to make sure all the LWRP calls get made before you write the single config file.

A similar case would be the iptables-ng cookbook. The rule and chain providers write out individual files, and then delayed notify a ruby block that generates the final output file at the end:

https://github.com/chr4-cookbooks/iptables-ng/blob/01f79041e03aef17fe2212a856970ca928792f09/providers/rule.rb#L66-L67

I'll take a shot at doing something in that direction.

[rayrod2030]

Yeah good point another cookbook that I thought about that does this is the heavy water graphite cookbook: https://github.com/hw-cookbooks/graphite. Multiple resources merge to build one or a few graphite config files.

[rayrod2030]

Also note that the core prometheus project itself is working on adding support for difference discovery services in addition to their DNS based discovery support but it would certainly be useful to have in the cookbook as well.

[rayrod2030]

Some more details on the Accumulator Pattern that's used in the graphite cookbook to take multiple resources and merge them into a file or template: https://github.com/kisoku/chef-accumulator

[ewr]

Neat. I hadn't seen chef-accumulator.

I forgot about the DNS SRV support. We run Consul, which supports DNS SRV, so that might work for part of what I need (what nodes / ports should I query for a given service?).

I'm working on the client cookbook side for a little bit, but will definitely circle back around to this piece by tomorrow morning.

[ewr]

Here's where I'm at in case any of this is useful...

Yesterday, I built out a little cookbook for client nodes. It currently has a node resource that supports grabbing a binary node-exporter, installing it and setting up an Upstart service:

https://github.com/scpr-cookbooks/prometheus-client

I also built a wrapper cookbook that uses that resource to set up the node exporter and register into a Consul service:

https://github.com/scpr-cookbooks/scpr-prometheus-client

On the other side, this wrapper cookbook sets up the server and then sets up our node-exporter job from Consul:

https://github.com/scpr-cookbooks/scpr-prometheus

[rayrod2030]

I meant to reply to this thread yesterday. I actually have tried using the dns based service discovery built into prometheus using consul/SRV records/dnsmasq and could not get it to work. Would be curious to see if you have better luck.

[ewr]

It seems to work for me...

You can check out the scpr-consul cookbook to see how I'm setting up dnsmasq, etc.

[ewr]

Just noticed my prometheus port is wrong there... Need to take the extra colon out of the spec for that job in the default recipe.

[rayrod2030]

Ah cool I have to revisit what I had set up before.

[ewr]

Just FYI since you mentioned using Consul: I'm playing with building an exporter to monitor service health (and eventually keys that should have active locks). Just got the first working pass this morning:

https://github.com/ewr/consul_exporter

Example results:

[rayrod2030]

Very cool. Thanks!

On Wed, Mar 18, 2015 at 9:21 AM, Eric Richardson [email protected]
wrote:

Just FYI since you mentioned using Consul: I'm playing with building an
exporter to monitor service health (and eventually keys that should have
active locks). Just got the first working pass this morning:

https://github.com/ewr/consul_exporter

Example results:

[image: screen shot 2015-03-18 at 9 19 57 am]
https://cloud.githubusercontent.com/assets/204350/6709476/031aa264-cd50-11e4-9808-3583c1722655.png

—
Reply to this email directly or view it on GitHub
#20 (comment)
.

[ewr]

Closed by #22.