The -target=all flag does not include alertmanager

[jakubgs]

I'm running a 3 host cluster with 1.3.0-rc.1 and I'm trying to run Alertmanager serivce with the following config:

alertmanager:
  external_url: 'https://alerts.example.org/'
  cluster_bind_address: '10.1.33.0:9094'
  cluster_advertise_address: '10.1.33.0'
  data_dir: '/var/tmp/cortex/alertmnanager'
  retention: 120h
  storage:
    type: local
    local:
      path: '/var/tmp/cortex/alerts'

But I don't see the service listening on port 9094 at all:

admin@master-03.do-ams3.metrics.hq:~ % sudo netstat -lpnt | grep cortex
tcp        0      0 10.1.33.0:9095          0.0.0.0:*               LISTEN      27780/cortex        
tcp6       0      0 :::9092                 :::*                    LISTEN      27780/cortex 

And I don't see the service in the return value from /services:

 % curl -s http://10.1.33.0:9092/services | grep -A1 '<tr>' | grep td
					<td>memberlist-kv</td>
					<td>store</td>
					<td>server</td>
					<td>ring</td>
					<td>distributor</td>
					<td>ingester</td>
					<td>query-frontend</td>
					<td>querier</td>
					<td>ruler</td>
					<td>table-manager</td>

Both configured directories exist and are owned by service user.
I'm not sure what more I'm supposed to do.

[pstibrany]

Alert Manager is not part of "all" target, and must be run on its own (by using -target=alertmanager or in config file: target: alertmanager).

[jakubgs]

Ooooooh, that makes sense. I totally missed that. Thanks.

Though question, what's the advantage of running alertmanager via Cortex as opposed to running Alertmanager directly?

[pstibrany]

Though question, what's the advantage of running alertmanager via Cortex as opposed to running Alertmanager directly?

I will leave answering that question to others, as my knowledge of AM is very low :) (Sorry!) (@gotjosh @jtlisi @annanay25?)

[jakubgs]

Though to be fair, when I see -target=all I kinda assume all means "all", not "all except alertmanager".

 % /usr/local/bin/cortex -h | grep -A 1 '\-target'
  -target string
    	The Cortex service to run. Use "-modules" command line flag to get a list of available options. (default "all")

The help message says nothing about it. And I don't see anything about that here:
https://cortexmetrics.io/docs/configuration/arguments/

[pstibrany]

That is a very good point. Can we reopen this issue, and rename it to document "all" target properly?

[jakubgs]

Does make me wonder though, what's stopping alertmanager from running alongside other routines/services in Cortex.

[pstibrany]

Does make me wonder though, what's stopping alertmanager from running alongside other routines in Cortex.

There can be different reasons. One is that multiple Cortex instances in "single-binary" (target=all) mode can work together. There are however some services that require exactly one instance, and if they were started multiple times, they would fail. This is the case for Alert Manager, as far as I understand.

Another reason is that some targets are only used in special situations (eg. "compactor", "flusher").

[jakubgs]

Not sure if that's true for Alertmanager as I have it running in a cluster on my infra without Cortex. There's a flag called --cluster.peer for specifying peers in the alertmanager cluster to connect to. Unless I'm misunderstanding you.

[pstibrany]

Cortex AlertManager is different from Prometheus one, although Cortex reuses parts of the original.

Eg. Cortex' AM supports multi-tenancy, so it actually instantiates one "Prometheus" AM per tenant.

[jakubgs]

Wait, does that mean that Cortex's alertmanager doesn't run in a cluster? Then how does it dedupe alerts?

[pstibrany]

Wait, does that mean that Cortex's alertmanager doesn't run in a cluster? Then how does it dedupe alerts?

I'm not exactly sure how but documentation says that it does deduplicate alerts ;-) (https://cortexmetrics.io/docs/architecture/#alertmanager)

[bboreham]

Duplicate of #1722

Eg. Cortex' AM supports multi-tenancy, so it actually instantiates one "Prometheus" AM per tenant.

One per Cortex alertmanager per tenant, i.e. if you have 3 Cortex alertmanagers and 100 tenants then there are 300 "Prometheus" alertmanagers.

Alert deduplication is done exactly the same as in Prometheus, per tenant.