Implement Prometheus checksum monitoring

[mwitkow]

Prometheus is a very popular monitoring stack. Unfortunately, it doesn't support strings as values.

Use case:
being able to count the checksums of static and dynamic flags in a pre-defined flagset across many server.

The typical approach in Prometheus (e.g. for build labels) is to export a value 1 with the string inside a label. This allows for easy counting of servers that have that particular value.

The resulting Prometheus metrics would look as follows:

flagz_checksum{set="default",type="dynamic",checksum="9fa85e70"} 1
flagz_checksum{set="default",type="static",checksum="53fd4b02"} 1

These checksums would need to be calculated on-scrape. Prometheus already has support for it with GaugeFunc but it doesn't allow to set labels, and only outputs the value on scrape.

To achieve this, a custom Collector needs to be implemented.

[mwitkow]

@beorn7 any suggestions regarding GaugeFunc-like Collector that exposes a string value as a Label instead of the float64 value?

[beorn7]

This will look something along the following lines:

package main

import (
    "encoding/hex"

    flagz "github.com/mwitkow/go-flagz"
    "github.com/prometheus/client_golang/prometheus"
    "github.com/spf13/pflag"
)

type checksumCollector struct {
    desc *prometheus.Desc
    fs   *pflag.FlagSet
}

func NewChecksumCollector(set, typ string, fs *pflag.FlagSet) prometheus.Collector {
    return &checksumCollector{
        desc: prometheus.NewDesc(
            "flagz_checksum",
            "The xyz checksum of a flag set.",
            []string{"checksum"},
            prometheus.Labels{"set": set, "type": typ},
        ),
        fs: fs,
    }
}

func (cc *checksumCollector) Describe(c chan<- *prometheus.Desc) {
    c <- cc.desc
}

func (cc *checksumCollector) Collect(c chan<- prometheus.Metric) {
    c <- prometheus.MustNewConstMetric(
        cc.desc,
        prometheus.GaugeValue,
        1,
        hex.EncodeToString(flagz.ChecksumFlagSet(cc.fs, nil)),
    )
}

func main() {
    coll := NewChecksumCollector(
        "default", "dynamic",
        nil, // need a real flag set, of course.
    )
    prometheus.MustRegister(coll)
}

[mwitkow]

Thanks @beorn7, that was incredibly helpful!
#13

Maybe there should be an example in Prometheus Go Client for this?

Happy easter :)

[mwitkow]

This is now done, thanks again :)

[beorn7]

I'm currently working on streamlining the way "const" metrics are handled. Initially, they were more an internal hack, but then it turned out that they are super helpful for a lot of use cases.

Use cases are better understood now. So I'll adjust the API to be more straight forward, and then I'll document the more relevant use cases.