VMAlertmanager: possibility to customize default matcher labels

[romainlaurent]

Hey folks

I have my Alert Manager running in a cluster, and it's reading VMAlertmanagerConfig from different namespaces. I could add the namespace label through VMAlert, but the metrics are coming from another cluster, and the namespace would be overwritten even though it's needed in the rule.

Would it be possible to customize the default label matcher by configuring the VMAlertmanager operator? We could change it here.

I'd be happy to implement this change if you're okay with it.

[f41gh7]

Hello,

I don't think, that changing default matcher is a good idea. First of all, it was designed for SaaS like metrics stack deployment. When scraping, alerting and notification configurations created at the same namespace. In this case namespace label enforced at scraping and notification configurations.

I'd suggest to disable namespace matcher and add needed matchers manually at VMAlertmanagerConfigs.

Maybe we could add some sort of globalVMAlertmanagerConfigMatchers array at VMAlertmanager side.

Like:

kind: VMAlertmanager
spec:
  enforcedTopRouteMatchers:
   - {namespace != ""}

But it doesn't look like good solution to me.

[romainlaurent]

The solution you propose seems to be a good one.
It would allow for better control over the default matchers.

To provide context, I have no control over the VMAlertmanagerConfigs, which are configurable by the user.
I do have control over the VMAlert, where I can set up static labels, and the VMAlertmanager (knowing the alert manager is shared among multiple users).

This feature would help me significantly to avoid alert collisions between users and undesired label overrides.

[f41gh7]

Feature added at v0.48.3 release.

With new setting spec.enforcedTopRouteMatchers it's possible to enforce matchers for all VMAlertmanagerConfig for particular `VMAlertmanager.

kind: VMAlertmanager
spec:
  enforcedTopRouteMatchers:
  - {namespace!=""}

Must enforce namespace label to present at alert to be able routed into VMAlertmanagerConfig.