Easy way to initialize counter vecs?

[beorn7]

See code like https://github.com/prometheus/prometheus/blob/b0adfea8d5e883cbab5c7e4113f8ea91f7cbc099/rules/manager.go#L75-L78 .

If you know in advance all the values each label can take, it is recommended to initialize all combinations so that each possible counter is exported with zero as a value.

This could be a utility function/method or be done somehow during construction.

This is a concern not specific to client_golang. Other client libraries usually have the same issue.

@fabxc @brian-brazil Feel free to comment…

[brian-brazil]

Python and Java work the same way as Go for this.

[beorn7]

I have an idea how to do this quite neatly. But it's a breaking change. I'll move this issue over to the v0.10 milestone and will flesh out as soon as I find some time.

[beorn7]

I mostly assigned this to myself as a “default” because I used to be the maintainer of this repo. Therefore, I'm un-assigning this from myself now. New maintainers @bwplotka & @kakkoyun, please deal with this as you see fit.

To get my aforementioned idea sketched out, at least vaguely (feel free to consult me for details, whoever might be working on this in the future):

Labels will be managed by a new label package. This will both be used for labels itself as well as label specs. The below are rough ideas for the interface. Needs refinement, and it all has to be done in a way that allows efficient implementations avoiding allocs etc.

func New(name, value string) Label creates a single label. There's a type []Label Labels with the contract of being sorted lexicographically and having no duplicates. It has a Normalize method to make sure of that. There is a convenience constructor func Set(nv ...string) Labels, which takes an even number of arguments (name1 value1 name2 value2 ...). They don't need to be sorted, but if they are, it will not cause allocations.

The above all implement an interface Spec. They are all full specs as they fully describe the label or labels. A func Names(names ...string) Spec is the least specific spec, as it only has the names. But then we can have “partial” specs that allow to specify all possible values a label might have: func WithValues(name string, values ...string) Spec. Perhaps additionally an “open” spec as well that specifies possible values but allows more than the listed ones: func WithIncompleteValues(name string, values ...string) Spec.

The constructor for a CounterVec would then look like func NewCounterVec(opts CounterOpts, labels ...label.Spec) *CounterVec (modulo other changes in v2). Example:

	httpAPIReqs := prometheus.NewCounterVec(
		prometheus.CounterOpts{
			Name: "http_requests_total",
			Help: "How many HTTP requests processed, partitioned by status code and HTTP method.",
		},
		label.WithValues("method", "GET", "POST"),
		label.WithIncompleteValues("code", "200", "404"),
		label.New("endpoint", "/api"), // Full spec, replaces old const labels.
	) // Automatically initializes children for GET/200, POST/200, GET/404, POST/404

	httpAPIReqs.With(label.Set("code", "200", "method", "GET")).Inc()  // Increments pre-initialized child.
	httpAPIReqs.With(label.Set("code", "503", "method", "GET")).Inc()  // Creates new child on the fly and increments it.
	httpAPIReqs.With(label.Set("code", "200", "method", "PUT")).Inc()  // Error because method="PUT" is not allowed.

[ivan-section-io]

FWIW in such situations - I've found myself with a helper function to address this:

// prometheusNewCounterVecZeroed creates prometheus.CounterVec with cartesian product of known labels zeroed
func prometheusNewCounterVecZeroed(opts prometheus.CounterOpts, labelNamesValues prometheusLabelNamesValues) *prometheus.CounterVec {
	labelNames := make([]string, 0, len(labelNamesValues))
	for i := range labelNamesValues {
		labelNames = append(labelNames, i)
	}

	counterVec := prometheus.NewCounterVec(opts, labelNames)
	indexes := make([]int, len(labelNames))
outer:
	for {
		labelValues := make([]string, len(labelNames))
		for i := range indexes {
			labelValues[i] = labelNamesValues[labelNames[i]][indexes[i]]
		}
		counterVec.WithLabelValues(labelValues...)
		for i := range indexes {
			indexes[i]++
			if indexes[i] == len(labelNamesValues[labelNames[i]]) {
				indexes[i] = 0
				continue
			}
			continue outer
		}
		break
	}
	return counterVec
}

It ain't pretty, is likely buggy, but makes Collector definition more readable until I discover how I "should" be doing it.

[beorn7]

Note that #1151 has paved the road towards auto-initialize vectors that have only constrained labels.

[bwplotka]

It seems the Created Timestamp feature can avoid the need of knowing the label values in advance and creating custom logic like above.

Created timestamp is produced on the first occurance of label for *Vec or metric. With this Prometheus (for now behind the created-timestamp-zero-ingestion feature flag) will create a zero sample which will yield precise rate/increase for such cases. In future the plan is to store CT in metadata, and use that for accurate rates as well in similar way.

Feedback welcome, but I think CT might close this issue and need.