Hardcoded runAsUser and runAsGroup Values in Helm Chart Cause Deployment Failures on OpenShift

[steveliem]

Description

I am encountering deployment issues with the WSO2 API Platform for Kubernetes (APK) on OpenShift due to hardcoded runAsUser and runAsGroup values in the Helm chart, specifically within the gateway-api-admission-server deployment template. OpenShift assigns user IDs (UIDs) and group IDs (GIDs) dynamically for security reasons, and specifying these values explicitly in the container's security context leads to permission conflicts.

Error Message:

Error creating: pods "gateway-api-admission-server-7998dfb48-" is forbidden: unable to validate against any security context constraint: [provider "anyuid": Forbidden: not usable by user or serviceaccount, ... provider restricted-v2: .containers[0].runAsUser: Invalid value: 65532: must be in the ranges: [1001280000, 1001289999], ...]

Steps to Reproduce

Analysis:

The deployment template includes the following security context:

securityContext:
  allowPrivilegeEscalation: false
  readOnlyRootFilesystem: true
  runAsNonRoot: true
  runAsUser: 65532
  runAsGroup: 65532
  capabilities:
    drop:
      - "ALL"
  seccompProfile:
    type: RuntimeDefault

In OpenShift, the assigned UID range for the project is 1001280000/10000, causing the hardcoded runAsUser: 65532 and runAsGroup: 65532 to fall outside the permissible range, resulting in deployment failures.

Suggested Solution:

To ensure compatibility with OpenShift's dynamic UID and GID assignment, I recommend the following modifications to the Helm chart:

1. Remove Hardcoded Values: Eliminate the explicit runAsUser and runAsGroup settings from the security context in the deployment templates.

2. Utilize Service Accounts: Define a service account for the deployment and allow OpenShift's Security Context Constraints (SCC) to assign appropriate UIDs and GIDs dynamically.

3. Make Security Context Configurable: Introduce Helm chart values that allow users to optionally specify runAsUser and runAsGroup if needed, defaulting to dynamic assignment when not set.

Proposed Deployment Template Adjustment:

apiVersion: apps/v1
kind: Deployment
metadata:
  name: gateway-api-admission-server
  namespace: {{ .Release.Namespace }}
  labels:
    name: gateway-api-admission-server
spec:
  replicas: 1
  selector:
    matchLabels:
      name: gateway-api-admission-server
  template:
    metadata:
      name: gateway-api-admission-server
      labels:
        name: gateway-api-admission-server
    spec:
      serviceAccountName: {{ .Values.gatewaySystem.deployment.serviceAccountName | default "default" }}
      containers:
        - name: webhook
          image: {{ .Values.gatewaySystem.deployment.image }}
          imagePullPolicy: {{ .Values.gatewaySystem.deployment.imagePullPolicy }}
          args:
            - -logtostderr
            - --tlsCertFile=/etc/certs/cert
            - --tlsKeyFile=/etc/certs/key
            - -v=10
            - 2>&1
          ports:
            - containerPort: 8443
              name: webhook
          resources:
            limits:
              memory: 50Mi
              cpu: 100m
            requests:
              memory: 50Mi
              cpu: 100m
          volumeMounts:
            - name: webhook-certs
              mountPath: /etc/certs
              readOnly: true
          securityContext:
            allowPrivilegeEscalation: false
            readOnlyRootFilesystem: true
            runAsNonRoot: true
            {{- if .Values.gatewaySystem.deployment.runAsUser }}
            runAsUser: {{ .Values.gatewaySystem.deployment.runAsUser }}
            {{- end }}
            {{- if .Values.gatewaySystem.deployment.runAsGroup }}
            runAsGroup: {{ .Values.gatewaySystem.deployment.runAsGroup }}
            {{- end }}
            capabilities:
              drop:
                - "ALL"
            seccompProfile:
              type: RuntimeDefault
      volumes:
        - name: webhook-certs
          secret:
            secretName: gateway-api-admission

Proposed Helm Values Addition:

gatewaySystem:
  deployment:
    serviceAccountName: gateway-api-admission
    # Optional: Specify these only if explicit values are required
    # runAsUser: 1001280000
    # runAsGroup: 1001280000

By implementing these changes, the deployment will be more adaptable to environments like OpenShift, which manage security contexts dynamically, thereby preventing permission issues and facilitating smoother deployments.

Thank you for your attention to this matter.

Affected Component

Adapter

Version

1.2.0

Environment Details (with versions)

OpenShift version: 4.16.18
Kubernetes version: v1.29.8+632b078
Channel: stable-4.16

Relevant Log Output

Error creating: pods "gateway-api-admission-server-7998dfb48-" is forbidden: unable to validate against any security context constraint: [provider "anyuid": Forbidden: not usable by user or serviceaccount, provider "quattro-monitoring-thanos-query-frontend-anyuid-scc": Forbidden: not usable by user or serviceaccount, provider "quattro-sealed-secrets-sealed-secrets": Forbidden: not usable by user or serviceaccount, provider restricted-v2: .containers[0].runAsUser: Invalid value: 65532: must be in the ranges: [1001280000, 1001289999], provider "restricted": Forbidden: not usable by user or serviceaccount, provider "containerized-data-importer": Forbidden: not usable by user or serviceaccount, provider "nonroot-v2": Forbidden: not usable by user or serviceaccount, provider "nonroot": Forbidden: not usable by user or serviceaccount, provider "loki": Forbidden: not usable by user or serviceaccount, provider "gateway-api-admission-scc": Forbidden: not usable by user or serviceaccount, provider "hostmount-anyuid": Forbidden: not usable by user or serviceaccount, provider "logging-scc": Forbidden: not usable by user or serviceaccount, provider "kubevirt-controller": Forbidden: not usable by user or serviceaccount, provider "machine-api-termination-handler": Forbidden: not usable by user or serviceaccount, provider "bridge-marker": Forbidden: not usable by user or serviceaccount, provider "hostnetwork-v2": Forbidden: not usable by user or serviceaccount, provider "hostnetwork": Forbidden: not usable by user or serviceaccount, provider "hostaccess": Forbidden: not usable by user or serviceaccount, provider "demo": Forbidden: not usable by user or serviceaccount, provider "scc-admin": Forbidden: not usable by user or serviceaccount, provider "scc-demo": Forbidden: not usable by user or serviceaccount, provider "linux-bridge": Forbidden: not usable by user or serviceaccount, provider "kubevirt-handler": Forbidden: not usable by user or serviceaccount, provider "fluent-bit-logging": Forbidden: not usable by user or serviceaccount, provider "node-exporter": Forbidden: not usable by user or serviceaccount, provider "privileged": Forbidden: not usable by user or serviceaccount]

Related Issues

No response

Suggested Labels

APK SCC Openshift gateway-api-admission-server Helm

[steveliem]

After adjusting the runAsUser and runAsGroup values to fall within the allowed range, the gateway-api-admission-server deployment is now operating correctly. Below is the current status:

$ oc get deploy
NAME                                             READY   UP-TO-DATE   AVAILABLE   AGE
gateway-api-admission-server                     1/1     1            1           65m
wso2-apk-cert-manager                            1/1     1            1           18h
wso2-apk-cert-manager-cainjector                 1/1     1            1           18h
wso2-apk-cert-manager-webhook                    1/1     1            1           18h
wso2-apk-wso2-apk-adapter-deployment             1/1     1            1           18h
wso2-apk-wso2-apk-common-controller-deployment   1/1     1            1           18h
wso2-apk-wso2-apk-config-ds-deployment           1/1     1            1           18h
wso2-apk-wso2-apk-gateway-runtime-deployment     2/2     2            2           18h
wso2-apk-wso2-apk-ratelimiter-deployment         2/2     2            2           18h

$ oc describe deploy gateway-api-admission-server
Name:                   gateway-api-admission-server
Namespace:              wso2-apk
CreationTimestamp:      Fri, 06 Dec 2024 12:24:04 +0100
Labels:                 app.kubernetes.io/managed-by=Helm
                        name=gateway-api-admission-server
Annotations:            deployment.kubernetes.io/revision: 2
                        meta.helm.sh/release-name: wso2-apk
                        meta.helm.sh/release-namespace: wso2-apk
Selector:               name=gateway-api-admission-server
Replicas:               1 desired | 1 updated | 1 total | 1 available | 0 unavailable
StrategyType:           RollingUpdate
MinReadySeconds:        0
RollingUpdateStrategy:  25% max unavailable, 25% max surge
Pod Template:
  Labels:  name=gateway-api-admission-server
  Containers:
   webhook:
    Image:           registry.k8s.io/gateway-api/admission-server:v1.0.0
    Port:            8443/TCP
    Host Port:       0/TCP
    SeccompProfile:  RuntimeDefault
    Args:
      -logtostderr
      --tlsCertFile=/etc/certs/cert
      --tlsKeyFile=/etc/certs/key
      -v=10
      2>&1
    Limits:
      cpu:     100m
      memory:  50Mi
    Requests:
      cpu:        100m
      memory:     50Mi
    Environment:  <none>
    Mounts:
      /etc/certs from webhook-certs (ro)
  Volumes:
   webhook-certs:
    Type:        Secret (a volume populated by a Secret)
    SecretName:  gateway-api-admission
    Optional:    false
Conditions:
  Type           Status  Reason
  ----           ------  ------
  Available      True    MinimumReplicasAvailable
  Progressing    True    NewReplicaSetAvailable
OldReplicaSets:  gateway-api-admission-server-7998dfb48 (0/0 replicas created)
NewReplicaSet:   gateway-api-admission-server-fdbbcd5db (1/1 replicas created)
Events:
  Type    Reason             Age   From                   Message
  ----    ------             ----  ----                   -------
  Normal  ScalingReplicaSet  66m   deployment-controller  Scaled up replica set gateway-api-admission-server-7998dfb48 to 1
  Normal  ScalingReplicaSet  51s   deployment-controller  Scaled up replica set gateway-api-admission-server-fdbbcd5db to 1
  Normal  ScalingReplicaSet  47s   deployment-controller  Scaled down replica set gateway-api-admission-server-7998dfb48 to 0 from 1

$ oc get pod
NAME                                                             READY   STATUS      RESTARTS      AGE
gateway-api-admission-2j2c4                                      0/1     Completed   0             67m
gateway-api-admission-patch-r7hqf                                0/1     Completed   1             67m
gateway-api-admission-server-fdbbcd5db-zhw66                     1/1     Running     0             62s
redis-master-0                                                   1/1     Running     0             18h
wso2-apk-cert-manager-5dc6b554f6-vvg2h                           1/1     Running     0             18h
wso2-apk-cert-manager-cainjector-c9bd4c4f6-qnxcn                 1/1     Running     0             18h
wso2-apk-cert-manager-webhook-7f5c7d4967-8jt5v                   1/1     Running     0             18h
wso2-apk-wso2-apk-adapter-deployment-6c5bd8fbd9-rvrdp            1/1     Running     0             18h
wso2-apk-wso2-apk-common-controller-deployment-d6577c8c6-tlm5p   1/1     Running     0             15h
wso2-apk-wso2-apk-config-ds-deployment-7888979585-c5rzr          1/1     Running     0             18h
wso2-apk-wso2-apk-gateway-runtime-deployment-79b7f67f78-v2588    2/2     Running     0             16h
wso2-apk-wso2-apk-gateway-runtime-deployment-79b7f67f78-zpcv9    2/2     Running     0             16h
wso2-apk-wso2-apk-ratelimiter-deployment-6d45479664-9rd9c        1/1     Running     3 (18h ago)   18h
wso2-apk-wso2-apk-ratelimiter-deployment-6d45479664-hkrw7        1/1     Running     3 (18h ago)   18h

These outputs confirm that the deployment is now functioning as expected and that my above arguments are valid.