How to use in a cluster

[PhillippOhlandt]

Hey,

how would one use this in a cluster of nodes? I don't use it to collect operating system or vm metrics but metrics about my application state. That means every node should give me (kinda) the same metrics.

Has someone done this yet? What is the easiest approach?

[deadtrickster]

Hi, could you please be more specific? The reason I asked is that there is no really difference whether it's a cluster or not. You custom metrics are just metrics, they have instance label attached, etc.

[PhillippOhlandt]

The lib itself works with ETS so the metrics state is not distributed to all members in a cluster. I want that the /metrics endpoint of all of my nodes return the same metrics so I just need to scrape one of my nodes and not all of them. I think I need a wrapper that does the replication of all metric changes across the cluster.

[deadtrickster]

but why?

[PhillippOhlandt]

Because I see my state as one unit and not as a unit per node. Each node will have a different part of the whole state and all entrypoints (/metrics, graphql api, etc.) should show the whole state and not just the one from the current node.

[deadtrickster]

but what you will do if network split, how you'll know the state (and the issues) of the particular node? plus prometheus does exactly this - aggregates.

[PhillippOhlandt]

but what you will do if network split, how you'll know the state

Valid question for everything distributed

plus prometheus does exactly this - aggregates.

yes it does, but I do not want to reconfigure prometheus every time I scale up. I also don't want to limit myself with having each node exposing a webserver.

[deadtrickster]

reconfigure prometheus every time I scale up

I think this is what service discovery for.

I understand that other concerns (incl not having http endpoint) are important to you, but in prometheus world even reachability is a metric itself (and a good source of alerts). If you don't want to have plugs/phoenix, you can look at zero-dependency exporter - https://github.com/deadtrickster/prometheus-httpd.

[PhillippOhlandt]

I actually use prometheus as my main time series long time storage and I don't trust service discovery that much. It also often requires yet another piece of software and from a quick google search, prometheus doesn't seem to support SSDP.