Error "multiple matches for labels: many-to-one matching must be explicit (group_left/group_right)" on Kubernetes/Networking Dashboards

[gsmith-sas]

When I attempt to deploy 3 of the Grafana dashboards (i.e. Kubernetes/ Networking/ Cluster, Kubernetes/Networking/Namespace (Pods) and Kubernetes / Networking/Namespace (Workload)) with Grafana 11, the dashboards fail to load properly. An error icon appears for each of chart/plot on these dashboards reporting the error "multiple matches for labels: many-to-one matching must be explicit (group_left/group_right)".

I am deploying Grafana as part of the Kube-Prometheus Stack Helm chart which deploys this mix-in. I was originally deploying Grafana 11.0.0 using Kube-Prometheus Stack Helm chart version 60.4.0. But I noticed there was an update released three days ago so I have just redeployed with Grafana 11.1.0 and Kube-Prometheus Stack Helm chart version 61.1.1 and the problem remains. Here's a screenshot showing the error message (for one of the queries):

[dragoangel]

Should be something like sum(sum(rate(container_network_transmit_bytes_total{cluster="$cluster",namespace=~"$namespace"}[$__rate_interval])) by (cluster,namespace,pod) * on (cluster,namespace,pod) GROUP_RIGHT() sum(kube_pod_info{host_network="false"}) without(pod_ip,uid))

[gsmith-sas]

@jkroepke Can you take a look at this issue? It appears you made the most recent changes to the queries, perhaps there is a simple typo? Or, does the current syntax work in your deployment? Thanks!

[jkroepke]

Could someone test this one?

sum by (namespace) (
    rate(container_network_receive_bytes_total{cluster="$cluster",namespace!=""}[$__rate_interval])
  * on (cluster, namespace, pod) group_left ()
    topk by (cluster, namespace, pod) (
      1,
      max by (cluster, namespace, pod) (kube_pod_info{host_network="false"})
    )
)

[gsmith-sas]

That PromQL executes without errors. I've included a screenshot of the resulting chart below. Is that what the chart should look like?

[gsmith-sas]

@jkroepke Do you need additional help testing any other revised PromQL queries for the Kubernetes/Networking dashboards? Or, are they similar enough that you have what you need to get these dashboards back to a good state?

[povilasv]

@jkroepke do you know why need this part? What is the issue here? 🤔

    topk by (cluster, namespace, pod) (
      1,
      max by (cluster, namespace, pod) (kube_pod_info{host_network="false"})
    )

[jkroepke]

If you deploy Statefulsets, than the same pod name in the same namespace/cluster exists, but with different labels (e.g. IP).

Then, I replicate the logic from other alerts:

kubernetes-mixin/rules/apps.libsonnet

Lines 20 to 22 in b836f02

	) * on (%(clusterLabel)s, namespace, pod) group_left(node) topk by (%(clusterLabel)s, namespace, pod) (
	1, max by(%(clusterLabel)s, namespace, pod, node) (kube_pod_info{node!=""})
	)

[povilasv]

Ah, 💯 feel free to open your PR, would definetely merge it #964 👍

[jkroepke]

#964 is not completed yet. I missing the workload/namespace level dashboards yet.

[jkroepke]

#964 is done yet. It would be great, if someone could also test it on own infra.

[gsmith-sas]

@jkroepke I will test these changes when they are available via the Kube-Prometheus Stack Helm chart project. Thank you for fixing this!