[Grafana] Web UI password inconsistency when redeploying Helm without deleting persistent database

[SergeiCherevko]

A Notorious Issue with persistent storage password in Grafana helm deployments

When deploying Grafana with Helm for the first time using a persistent database, it creates a login and password for the Grafana UI. However, when you update or redeploy Helm, it generates a new password, but this password is not updated in the persistent database. This leads to login issues and can be quite frustrating.

I deploy Grafana in a playground environment with separate persistent storage. During testing, I often need to recreate the Grafana application multiple times for various reasons, but I do not modify the persistent database.

For example, after redeploying Grafana, I run a script that displays the Grafana URL along with the login credentials:

Grafana is ready to use at https://$$cluster_name-grafana.example.com
Login: admin
Password: $(kubectl get secret grafana -n monitoring -o 'jsonpath={.data.admin-password}' | base64 -d)

However, this password is incorrect because the actual password remains stored in the persistent database, while Helm generates a new one in the Kubernetes Secret. As a result, there is no way to retrieve the correct password unless you manually delete the persistent storage and database—which is extremely annoying.