Separate pod and user diagnostics

[jnywong]

Context

The current "User Diagnostics" dashboard is technically a "Pod Diagnostics" dashboard.

In JupyterHubs where named servers are enabled, we can have multiple singleuser servers associated with one user.

Changes

This PR separates the two use cases into

• Pod Diagnostics: per-user per-server
• User Diagnostics: per-user

by updating PromQL to make use of the annotation_hub_jupyter_org_username selector of the kube_pod_annotations metric picked up from the _build_common_annotations method of Kubespawner to allow querying on unescaped usernames. This is useful for later associating unescaped usernames with membership of user groups.

Also

• adds snippet to deployment docs on how to configure metricAnnotationsAllowlist
• adds a common $user_name variable that allows you to filter by unescaped jupyter username at the top of the User Diagnostics dashboard
• user diagnostics dashboard does not show data from support namespaces unrelated to user servers, whereas all pod data from all namespaces is retained in the Pod diagnostics dashboard
• fixes a small bug for the "Home Directory Usage" panel that had a missing ~ to regex-match namespaces

Result

Pod diagnostics dashboard

User diagnostics dashboard

Questions

• What should the purpose of the pod diagnostics dashboard be in light of the user diagnostics dashboard? Current implementation retains pod data from all namespaces and removes "Home Directory Usage" panel
• How useful is the User Diagnostics dashboard for hubs that don't enable named servers?

Closes: 2i2c-org/infrastructure#5806

[yuvipanda]

The 'home directory usage' panel in the user diagnostics dashboard is not filtering based on the selected username:

This may be a little tricky because these are escaped, and could be escaped differently than the server names. Am ok with leaving that as is for now but something to flag.

The only difference between the pod and user board is when multiple named servers are present. Currently servername is set as an annotation only when user sets it, while unconditionally set as a "" label regardless. I opened jupyterhub/kubespawner#887 to unconditionally set the annotation as well.

I think there's no real reason to have the pod diagnostics dashboard. We should figure out a way to represent the different named servers in the user diagnostics dashboard, and not have a pod specific one. Maybe we could group by the servername label in addition to the username label for now, and move to annotations once that PR lands?