Postgresql docker : ‘/var/lib/postgresql/data’: Permission denied

[VasseurLaurent]

Issue type

Bug report

SUMMARY

Hello, I am trying to deploy AWX 19.2 on k3s for testing purpose.
However, it seems that the postgresql docker cannot write on this volume.

You can find below my installation procedure.
Funny things, I successfully deployed an AWX for testing yesterday with the same procedure

I saw some already created issue related to postgresql but I didn't find one with the same issue has mine.

Do you have any idea of what I could do wrong ?

Thank you a lot in advance.

ENVIRONMENT

AWX version: 19.2.2
Operator version: 0.12
Kubernetes version: v1.21.2+k3s1
AWX install method: AWX Operator on k3s cluster

STEPS TO REPRODUCE

Install k3s

curl -sfL https://get.k3s.io | sh -s - --write-kubeconfig-mode 644

Install AWX Operator :

kubectl apply -f https://raw.githubusercontent.com/ansible/awx-operator/0.12.0/deploy/awx-operator.yaml

Create deploy configuration :

apiVersion: awx.ansible.com/v1beta1
kind: AWX
metadata:
  name: example
spec:
  service_type: nodeport
  ingress_type: ingress
  hostname: example.com

And apply it :

kubectl apply -f awx-deploy.yml

EXPECTED RESULTS

Ansible AWX fully installed.

ACTUAL RESULTS

It seems to have an issue with the postgresql docker

NAME                            READY   STATUS             RESTARTS   AGE
awx-operator-545497f7d5-9dwp9   1/1     Running            0          14m
awx-iac-65f4d4b596-lcmwg        4/4     Running            0          13m
awx-iac-postgres-0              0/1     CrashLoopBackOff   7          13m

ADDITIONAL INFORMATION

I did some research and I can find that the postgresql docker cannot write data :

kubectl logs awx-iac-postgres-0
mkdir: cannot create directory ‘/var/lib/postgresql/data’: Permission denied

AWX-OPERATOR LOGS

When I use the command :

kubectl describe pods awx-operator-545497f7d5-9dwp9

I can see the error :

Events:
  Type     Reason       Age   From               Message
  ----     ------       ----  ----               -------
  Normal   Scheduled    18m   default-scheduler  Successfully assigned default/awx-operator-545497f7d5-9dwp9 to ....
  Warning  FailedMount  18m   kubelet            MountVolume.SetUp failed for volume "kube-api-access-pgvn6" : failed to sync configmap cache: timed out waiting for the condition

But I don't know how to debug it.

[chipn]

I'm hitting the same issue, but it appears to be caused by this k3s issue: k3s-io/k3s#3704.

Downgrading to k3s v1.20.9+k3s1 resolved it for me.

[tchellomello]

Thanks for the heads up @chipn Since this is related to the external k3s, I'm going ahead and close this up. Feel free to reopen if needed.

[profhase]

@tchellomello Not sure we should close this - There are certain things that can be done on the operator to fix it in the operator e.g. assure permissions on postgres start

[scott-vick]

This is still a problem with AWX. I installed the previous version of k3s (v1.20.9+k321) and I'm still getting the same "permission denied" error. I've been trying for 4 days to install AWX and I cannot get it to install.

[tchellomello]

@tchellomello Not sure we should close this - There are certain things that can be done on the operator to fix it in the operator e.g. assure permissions on postgres start

Yes it makes sense. The issue is reopened and linked to a PR

[wenottingham]

See #483, closing as a duplicate.