ironic-ipa-downloader: Permission Denied

[albal]

Hello,

I popped up on Kubernetes Slack last week with an issue a client was having with Metal3 Ironic in that they couldn't deploy hosts and I was asked to raise an issue here. We found that the ironic pod was in Init:CrashLoopBackOff.
From kubectl logs -n baremetal-operator-system ironic-5468f99647-72xw6 -c ironic-ipa-downloader:

+ export http_proxy=
+ http_proxy=
+ export https_proxy=
+ https_proxy=
+ export no_proxy=
+ no_proxy=
+ SHARED_DIR=/shared
+ IPA_BASEURI=https://tarballs.opendev.org/openstack/ironic-python-agent/dib
++ echo master
++ tr / -
+ IPA_BRANCH=master
+ IPA_FLAVOR=centos9
+ FILENAME=ipa-centos9-master
+ FILENAME_EXT=.tar.gz
+ FFILENAME=ipa-centos9-master.tar.gz
+ DESTNAME=ironic-python-agent
+ mkdir -p /shared/html/images /shared/tmp
mkdir: cannot create directory '/shared': Permission denied
mkdir: cannot create directory '/shared': Permission denied

# git submodule status
-9490d85c1655e252a4b93513a65acaffa7dff5fc baremetal-operator

I then lost access to that client but another client today has this issue also - these are where the logs came from above. These were previously working instances.

The volumes are backed by Longhorn but all health data seems good for Longhorn.

Both clients are using Rancher K3S version v1.23.17+k3s1 and the same versions of Longhorn, metal etc.

I'm the process of creating a sandboxed version of the issue where I can experiment with, not disturbing the clients' systems.

[metal3-io-bot]

This issue is currently awaiting triage.
If Metal3.io contributors determine this is a relevant issue, they will accept it by applying the triage/accepted label and provide further guidance.
The triage/accepted label can be added by org members by writing /triage accepted in a comment.

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes-sigs/prow repository.

[tuminoid]

Is there any reproduction steps, or is it just that you notice this behavior when Ironic has crashed/relocated?

Can you also share the versions of BMO, ironic-ipa-downloader and Ironic-image you're using? BMO submodule SHA matches v0.6.0-4-g9490d85c.

[albal]

These are the images I am using:

image: quay.io/metal3-io/ironic-ipa-downloader
image: quay.io/metal3-io/ironic:release-26.0
image: quay.io/metal3-io/baremetal-operator:release-0.8

I've not got to sandboxing this yet. On both clusters ironic was working and then at some point it stops working with the error that is can't write to the shared folder.

[tuminoid]

Hmm, there is a discrepancy here with the BMO submodule SHA and the image version. If the manifests are from BMO 0.6 and the image is from 0.8, this can lead to various errors as BMO contains the Ironic manifests as well.

[Rozzii]

So I was working on a feature related to this project last week. IMO this issue is exactly what the error message say you most likely have no volume backing up the /shared, if the ipa-downloader can't find /shared it will try to create it and probably it tries to create it with a user who has no permissions or on a read only root FS.

Please make sure you have the mount and the volume configured properly for the Ironic Pod. I have checked the script without the whole Ironic thing and it works and it also works in our tests too .

/triage needs-information

[dtantsur]

Another potential issue (especially if using a custom way to deploy Ironic): mismatch between the users used to run Ironic.