Issues with deploying a RedisCluster on OpenShift

[michaelarnauts]

I've seen a few mentions of this issue here, but all issues are closed, and creating a RedisCluster on OpenShift using this operator still doesn't seem to work.

Note that I'm using the Redis Operator 0.13.0, the latest one that is available in the Operator Hub on OpenShift.

I'm using the following Manifest to create the cluster.

kind: RedisCluster
apiVersion: redis.redis.opstreelabs.in/v1beta1
metadata:
  name: redis-cluster
spec:
  clusterSize: 2
  clusterVersion: v7
  securityContext:
    fsGroup: 1000
    runAsUser: 1000
  persistenceEnabled: true
  kubernetesConfig:
    image: 'quay.io/opstree/redis:v7.0.5'
    imagePullPolicy: IfNotPresent
  redisExporter:
    enabled: false
    image: 'quay.io/opstree/redis-exporter:v1.44.0'
    imagePullPolicy: IfNotPresent
  storage:
    volumeClaimTemplate:
      spec:
        accessModes:
          - ReadWriteOnce
        resources:
          requests:
            storage: 1Gi

I can see it creates a RedisCluster with a name redis-cluster, and a StatefulSet named redis-cluster-leader. However, the Statefulset doesn't contain any pods and has the following error. It seems that this is due to the runAsUser parameter in the securityContext.

create Pod redis-cluster-leader-0 in StatefulSet redis-cluster-leader failed

error: pods "redis-cluster-leader-0" is forbidden: unable to validate against any security context constraint: [
provider "anyuid": Forbidden: not usable by user or serviceaccount,
provider restricted-v2: .spec.securityContext.fsGroup: Invalid value: []int64{1000}: 1000 is not an allowed group, spec.containers[0].securityContext.runAsUser: Invalid value: 1000: must be in the ranges: [1000800000, 1000809999],
provider "restricted": Forbidden: not usable by user or serviceaccount,
provider "nonroot-v2": Forbidden: not usable by user or serviceaccount,
provider "nonroot": Forbidden: not usable by user or serviceaccount,
provider "noobaa": Forbidden: not usable by user or serviceaccount,
provider "noobaa-endpoint": Forbidden: not usable by user or serviceaccount,
provider "hostmount-anyuid": Forbidden: not usable by user or serviceaccount,
provider "machine-api-termination-handler": Forbidden: not usable by user or serviceaccount,
provider "hostnetwork-v2": Forbidden: not usable by user or serviceaccount,
provider "hostnetwork": Forbidden: not usable by user or serviceaccount,
provider "hostaccess": Forbidden: not usable by user or serviceaccount,
provider "ocs-metrics-exporter": Forbidden: not usable by user or serviceaccount,
provider "rook-ceph": Forbidden: not usable by user or serviceaccount,
provider "node-exporter": Forbidden: not usable by user or serviceaccount,
provider "rook-ceph-csi": Forbidden: not usable by user or serviceaccount,
provider "privileged": Forbidden: not usable by user or serviceaccount
]

When I set the securityContext to an empty object (like below), I can see that two pods are created, but I get the permission errors at startup.

kind: RedisCluster
apiVersion: redis.redis.opstreelabs.in/v1beta1
metadata:
  name: redis-cluster
spec:
  clusterSize: 2
  clusterVersion: v7
  securityContext: {}
...

Redis is running without password which is not recommended
/usr/bin/entrypoint.sh: line 22: /etc/redis/redis.conf: Permission denied
/usr/bin/entrypoint.sh: line 32: /etc/redis/redis.conf: Permission denied
sed: /data/nodes.conf: No such file or directory
/usr/bin/entrypoint.sh: line 72: /etc/redis/redis.conf: Permission denied
Running without TLS mode
Starting redis service in cluster mode.....
10:C 05 Apr 2023 09:36:00.658 # oO0OoO0OoO0Oo Redis is starting oO0OoO0OoO0Oo
10:C 05 Apr 2023 09:36:00.658 # Redis version=7.0.5, bits=64, commit=00000000, modified=0, pid=10, just started
10:C 05 Apr 2023 09:36:00.658 # Configuration loaded
10:M 05 Apr 2023 09:36:00.659 * monotonic clock: POSIX clock_gettime
10:M 05 Apr 2023 09:36:00.659 * Running mode=standalone, port=6379.
10:M 05 Apr 2023 09:36:00.660 # WARNING: The TCP backlog setting of 511 cannot be enforced because /proc/sys/net/core/somaxconn is set to the lower value of 128.
10:M 05 Apr 2023 09:36:00.660 # Server initialized
10:M 05 Apr 2023 09:36:00.660 * Ready to accept connections

Inside the container, I observe these permissions:

/data $ id
uid=1000800000(1000800000) gid=0(root) groups=1000800000

/data $ ls -l /etc/redis
total 4K     
-rwxr-xr-x    1 redis    redis        114 Oct 30 13:58 redis.conf

/data $ ls -ln /etc/redis
total 4
-rwxr-xr-x    1 1000     1000           114 Oct 30 13:58 redis.conf

So it seems that this change isn't applied: #4

When I run this locally, on docker, it seems that the image isn't modified like in above PR.

$ docker run -it --entrypoint=/bin/bash quay.io/opstree/redis:v7.0.5
bash-5.1$ ls -lah /etc/redis/
total 12K
drwxr-xr-x    1 redis    redis       4.0K Oct 30 13:58 .
drwxr-xr-x    1 root     root        4.0K Apr  5 09:51 ..
-rwxr-xr-x    1 redis    redis        114 Oct 30 13:58 redis.conf
bash-5.1$

See also:

• Openshift Redis Cluster not starting redis-operator#69 (comment)
• mkdir: can't create directory '/opt/redis': Permission denied redis-operator#19 (comment)

[michaelarnauts]

It seems that the change of #4 has been removed in this change:

fac6855#diff-dd2c0eb6ea5cfc6c4bd4eac30934e2d5746747af48fef6da689e85b752f39557L47

Therefore, /etc/redis/redis.conf isn't writable anymore by the group that should be root.

[michaelarnauts]

I've build my own image like this:

FROM quay.io/opstree/redis:v7.0.5

USER root

RUN chgrp -R 0 /etc/redis/redis.conf && \
    chmod g+rw /etc/redis/redis.conf

USER redis

And modified the Manifest to use my image instead of quay.io/opstree/redis:v7.0.5.

It seems that this allows the cluster to start successfully.

Redis is running without password which is not recommended
sed: /data/nodes.conf: No such file or directory
Running without TLS mode
Starting redis service in cluster mode.....
10:C 05 Apr 2023 11:05:43.782 # oO0OoO0OoO0Oo Redis is starting oO0OoO0OoO0Oo
10:C 05 Apr 2023 11:05:43.782 # Redis version=7.0.5, bits=64, commit=00000000, modified=0, pid=10, just started
10:C 05 Apr 2023 11:05:43.782 # Configuration loaded
10:M 05 Apr 2023 11:05:43.783 * monotonic clock: POSIX clock_gettime
10:M 05 Apr 2023 11:05:43.784 * No cluster configuration found, I'm af4e94b87dc4608b96f3c5b1788487371c3e8c57
10:M 05 Apr 2023 11:05:43.793 * Running mode=cluster, port=6379.
10:M 05 Apr 2023 11:05:43.793 # WARNING: The TCP backlog setting of 511 cannot be enforced because /proc/sys/net/core/somaxconn is set to the lower value of 128.
10:M 05 Apr 2023 11:05:43.793 # Server initialized
10:M 05 Apr 2023 11:05:43.795 * Creating AOF base file appendonly.aof.1.base.rdb on server start
10:M 05 Apr 2023 11:05:43.797 * Creating AOF incr file appendonly.aof.1.incr.aof on server start
10:M 05 Apr 2023 11:05:43.797 * Ready to accept connections

It would be great to be able to add this again to the Dockerfile.

[michaelarnauts]

I've created a PR with this fix here: #25