"many-to-many matching not allowed" issue in alert rules when monitoring two postgresql applications in two different models with cos.

[sagittariuslee]

Steps to reproduce

1. Deploy cos-lite stack
2. Deploy two postgresql-k8s applications in two different models
3. Relate these two applications with cos

Error message in cos-prometheus log

In prometheus within cos model, we can see logs like:

2024-12-05T21:11:53.560Z [pebble] GET /v1/notices?after=2024-12-05T21%3A10%3A23.543832964Z&timeout=30s 30.000938692s 200
2024-12-05T21:11:59.224Z [prometheus] ts=2024-12-05T21:11:59.224Z caller=group.go:483 level=warn name=PostgresqlConfigurationChanged index=15 component="rule manager" file=/etc/prometheus/rules/juju_spark_f70165e5_grafana-agent.rules group=spark_f70165e5_postgresql-k8s_PostgresqlExporterK8s_alerts msg="Evaluating rule failed" rule="alert: PostgresqlConfigurationChanged\nexpr: '{__name__=~\"pg_settings_.*\",juju_application=\"postgresql-k8s\",juju_model=\"spark\",juju_model_uuid=\"f70165e5-be13-44fe-8722-337c85eb602c\"}\n  != on (__name__) {__name__=~\"pg_settings_([^t]|t[^r]|tr[^a]|tra[^n]|tran[^s]|trans[^a]|transa[^c]|transac[^t]|transact[^i]|transacti[^o]|transactio[^n]|transaction[^_]|transaction_[^r]|transaction_r[^e]|transaction_re[^a]|transaction_rea[^d]|transaction_read[^_]|transaction_read_[^o]|transaction_read_o[^n]|transaction_read_on[^l]|transaction_read_onl[^y]).*\",juju_application=\"postgresql-k8s\",juju_model=\"spark\",juju_model_uuid=\"f70165e5-be13-44fe-8722-337c85eb602c\"}\n  offset 5m'\nlabels:\n  juju_application: postgresql-k8s\n  juju_charm: postgresql-k8s\n  juju_model: spark\n  juju_model_uuid: f70165e5-be13-44fe-8722-337c85eb602c\n  severity: info\nannotations:\n  description: |-\n    Postgres Database configuration change has occurred\n      VALUE = {{ $value }}\n      LABELS = {{ $labels }}\n  summary: Postgresql configuration changed (instance {{ $labels.instance }})\n" err="found duplicate series for the match group {__name__=\"pg_settings_allow_in_place_tablespaces\"} on the right hand-side of the operation: [{__name__=\"pg_settings_allow_in_place_tablespaces\", instance=\"spark_f70165e5-be13-44fe-8722-337c85eb602c_postgresql-k8s_postgresql-k8s/2\", job=\"juju_spark_f70165e5_postgresql-k8s_prometheus_scrape-2_7d3c1b29ce2243676fc0d7b3882b8267b056b8fb74c744c099042c50f28a5668\", juju_application=\"postgresql-k8s\", juju_charm=\"postgresql-k8s\", juju_model=\"spark\", juju_model_uuid=\"f70165e5-be13-44fe-8722-337c85eb602c\", juju_unit=\"postgresql-k8s/2\", server=\"/var/run/postgresql:5432\"}, {__name__=\"pg_settings_allow_in_place_tablespaces\", instance=\"spark_f70165e5-be13-44fe-8722-337c85eb602c_postgresql-k8s_postgresql-k8s/1\", job=\"juju_spark_f70165e5_postgresql-k8s_prometheus_scrape-1_d07997ef47403bb349cf96cd6d4f5a27e846f75e8cc55cd263a2d6c19245ea5a\", juju_application=\"postgresql-k8s\", juju_charm=\"postgresql-k8s\", juju_model=\"spark\", juju_model_uuid=\"f70165e5-be13-44fe-8722-337c85eb602c\", juju_unit=\"postgresql-k8s/1\", server=\"/var/run/postgresql:5432\"}];many-to-many matching not allowed: matching labels must be unique on one side"

Results of Prometheus Query {__name__="pg_settings_allow_in_place_tablespaces"}

|   pg\_settings\_allow\_in\_place\_tablespaces{instance\="spark\_f70165e5-be13-44fe-8722-337c85eb602c\_postgresql-k8s\_postgresql-k8s/1", job\="juju\_spark\_f70165e5\_postgresql-k8s\_prometheus\_scrape-1\_d07997ef47403bb349cf96cd6d4f5a27e846f75e8cc55cd263a2d6c19245ea5a", juju\_application\="postgresql-k8s", juju\_charm\="postgresql-k8s", juju\_model\="spark", juju\_model\_uuid\="f70165e5-be13-44fe-8722-337c85eb602c", juju\_unit\="postgresql-k8s/1", server\="/var/run/postgresql:5432"}   | 0 |
| --- | --- |
|   pg\_settings\_allow\_in\_place\_tablespaces{instance\="spark\_f70165e5-be13-44fe-8722-337c85eb602c\_postgresql-k8s\_postgresql-k8s/2", job\="juju\_spark\_f70165e5\_postgresql-k8s\_prometheus\_scrape-2\_7d3c1b29ce2243676fc0d7b3882b8267b056b8fb74c744c099042c50f28a5668", juju\_application\="postgresql-k8s", juju\_charm\="postgresql-k8s", juju\_model\="spark", juju\_model\_uuid\="f70165e5-be13-44fe-8722-337c85eb602c", juju\_unit\="postgresql-k8s/2", server\="/var/run/postgresql:5432"}   | 0 |
|   pg\_settings\_allow\_in\_place\_tablespaces{instance\="spark\_f70165e5-be13-44fe-8722-337c85eb602c\_postgresql-k8s\_postgresql-k8s/0", job\="juju\_spark\_f70165e5\_postgresql-k8s\_prometheus\_scrape-0\_fb708f950bce87d811689ee3a1001742dd7c153c17ee2c2e9776e016eb643941", juju\_application\="postgresql-k8s", juju\_charm\="postgresql-k8s", juju\_model\="spark", juju\_model\_uuid\="f70165e5-be13-44fe-8722-337c85eb602c", juju\_unit\="postgresql-k8s/0", server\="/var/run/postgresql:5432"}   | 0 |
|   pg\_settings\_allow\_in\_place\_tablespaces{instance\="spark\_6b63e988-dc44-457b-8847-c1e59e4a1ee3\_postgresql-k8s\_postgresql-k8s/2", job\="juju\_spark\_6b63e988\_postgresql-k8s\_prometheus\_scrape-2\_02e9bc7754e782166c6efeb4be53c852636fb358c2e904207089941613afb8be", juju\_application\="postgresql-k8s", juju\_charm\="postgresql-k8s", juju\_model\="spark", juju\_model\_uuid\="6b63e988-dc44-457b-8847-c1e59e4a1ee3", juju\_unit\="postgresql-k8s/2", server\="/var/run/postgresql:5432"}   | 0 |
|   pg\_settings\_allow\_in\_place\_tablespaces{instance\="spark\_6b63e988-dc44-457b-8847-c1e59e4a1ee3\_postgresql-k8s\_postgresql-k8s/1", job\="juju\_spark\_6b63e988\_postgresql-k8s\_prometheus\_scrape-1\_848e1cede82a2b895126a58493f5cf2b2b4424510a44f01884b7dfca10127ca7", juju\_application\="postgresql-k8s", juju\_charm\="postgresql-k8s", juju\_model\="spark", juju\_model\_uuid\="6b63e988-dc44-457b-8847-c1e59e4a1ee3", juju\_unit\="postgresql-k8s/1", server\="/var/run/postgresql:5432"}   | 0 |
|   pg\_settings\_allow\_in\_place\_tablespaces{instance\="spark\_6b63e988-dc44-457b-8847-c1e59e4a1ee3\_postgresql-k8s\_postgresql-k8s/0", job\="juju\_spark\_6b63e988\_postgresql-k8s\_prometheus\_scrape-0\_d082fb6de5aaee65b78041b889c8c77398f28679eebe2d15fe9738aa61768f2a", juju\_application\="postgresql-k8s", juju\_charm\="postgresql-k8s", juju\_model\="spark", juju\_model\_uuid\="6b63e988-dc44-457b-8847-c1e59e4a1ee3", juju\_unit\="postgresql-k8s/0", server\="/var/run/postgresql:5432"}   |

Additional context

Possibly, this rule should be modified.

[syncronize-issues-to-jira]

Thank you for reporting us your feedback!

The internal ticket has been created: https://warthogs.atlassian.net/browse/DPE-6170.

This message was autogenerated

[syncronize-issues-to-jira]

Thank you for reporting us your feedback!

The internal ticket has been created: https://warthogs.atlassian.net/browse/DPE-6171.

This message was autogenerated

[marceloneppel]

Hi, @sagittariuslee!
I planned the fix for this issue for the next pulse.

[lucasgameiroborges]

Hi @sagittariuslee , thank you for the report!

I'll take a closer look at this issue first thing in January. Will keep you updated here.