Decide and document the scope of heapster's responsibilities

[bgrant0607]

Since we're extending heapster in various ways for auto-scaling, and we're about to add support for collecting and aggregating custom metrics, we should decide and document what the boundaries for heapster's responsibilities should be. In particular, I don't want it to grow into a general-purpose application and/or infrastructure monitoring, dashboarding, and alerting system, such as Prometheus. There are many such systems, and we shouldn't compete with them.

We should also document how heapster can be integrated with components that provide functionality outside its boundaries. For instance, it is already documented that Heapster supports multiple storage backends, with a few examples. Additionally, my understanding is that we plan to use collectd in order to convert a multitude of data formats to the Prometheus format. Other obvious areas include dashboards, such Graphana, and alerting, such as AlertManager.

We should also clarify intended usage. For instance, I understand collecting a handful of metrics, such as QPS, to drive horizontal pod auto-scaling and other core functionality, but how about collection of hundreds of arbitrary metrics from a single application?

cc @vishh @jszczepkowski @mwielgus @davidopp @lavalamp @smarterclayton

[jimmidyson]

I think this is an incredibly important thing to nail down. I've previously raised a proposal in #645 to utilise Prometheus for metric collection & Heapster to become a REST API with Kubernetes API semantics over the top of Prometheus.

I'd also like to point out the work I've done with Prometheus to provide autodiscovery for master, node, container & application targets for metric collection.

But that doesn't help with nailing down exactly what the function & limits of Heapster is but I feel this would provide a configurable & extensible option to users.

[piosz]

cc @fgrzadkowski

[fgrzadkowski]

Reassigning to @mwielgus who is driving now changes in heapster.

[mwielgus]

The target scope and vision documented in #769. Closing this for now.