expose discarded samples metrics via prometheus /metrics endpoint

[woodsaj]

We use prometheus to scrap active_series counts on a per org basis so we can provide our customers with a unified dashboard that shows usage for both graphite (metrictank) and prometheus(cortex) hosted metrics services.

In addition to the current metrics, we also need to expose "discarded samples" numbers.
These metrics will be similar to the existing metrics we collect:
https://github.com/grafana/metrictank/blob/master/input/input.go#L47-L51
https://github.com/grafana/metrictank/blob/master/mdata/init.go#L31-L39

But will be a counter metric "discarded_samples_total" with labels for OrgID and discarded reason.
eg

var DiscardedSamples = prometheus.NewCounterVec(
	prometheus.CounterOpts{
		Namespace: "metrictank",
		Name: "discarded_samples_total",
		Help: "The total number of samples that were discarded.",
	},
	[]string{"reason", "org"},
)

the reason will be something like

• out-of-order (currently called tank.metrics_too_old)
• new-value-for-timestamp (currently called tank.metrics_too_old )
• received-too-late (currently called tank.add_to_closed_chunk)
• invalid-timestamp (currently called input.%s.metricdata.invalid or input.%s.metricpoint.invalid)
• invalid-interval (currently called input.%s.metricdata.invalid)
• invalid-orgID (currently called input.%s.metricdata.invalid)
• invalid-name (currently called input.%s.metricdata.invalid)
• invalid-mtype (currently called input.%s.metricdata.invalid)
• invalid-tag-format (currently called input.%s.metricdata.invalid)
• unknown-point-id (currently called input.%s.metricpoint.unknown)

[woodsaj]

see also: #1201

[Dieterbe]

This is mostly finished, but needs followup, new-value-for-timestamp
see #1278 (comment)

[fkaleo]

This is mostly finished, but needs followup, new-value-for-timestamp
see #1278 (comment)

Yep, will do with #1201 and #1202