Problem with persistent storage on NFS and root remapping

[briantopping]

When using persistent storage and the root user is remapped (i.e. to 'nobody'), the run.sh script assumes it can chmod -R grafana:grafana, but it fails. This fails the script as well (IIRC) because it is launched with -e.

It took several hours to figure out everything that was going on and the error messages are not helpful. grafana/grafana#6285 had this same problem, the response was that the volume needs to be writable. This is a nuance of the "writable volume" that's probably not obvious.

Many NFS servers will not allow per-mount changes of such parameters if the allowed networks overlap. This would be true (for instance) if prometheus and grafana were both running on the same server and had different mounts with different map all settings for each different user. As such, this is a blocker.

What can I provide here?

[devKlausS]

We are facing the same problem running Prometheus in K8s with NFS persistent storage. For a quick fix I just removed the following two lines from run.sh

chown -R grafana:grafana "$GF_PATHS_DATA" "$GF_PATHS_LOGS"
chown -R grafana:grafana /etc/grafana

[devKlausS]

You can find my fix here
devKlausS@9041985

[xlson]

Usage of chown will be removed completely in 5.1 (#146). The new image also supports remapping to any user id.

If the chown fails we will no longer fail to boot since 5.0.4 that was released today.

[briantopping]

Hi @xlson, is there a reason to keep the -e? What I was attempting in this description was fine without that flag (I tested and deployed it for my own purposes that way).

Just looking to add robustness. Arguably that can go both ways whether the flag is removed or remains.

Thanks for all your work on a great product!

[xlson]

@briantopping The main idea behind having the flag is to fail fast with the thinking that the script should be designed so that any such problems would be severe enough to warrant it. That said, I'm not entirely sure if there is any such actions left in the script. Will have to think about it wether we still need it.

Thanks!