The nginx-gateway's pod is not ready

[Yelijah]

Describe the bug
After I deploy nginx gateway fabric by the helm, the nginx-gateway-fabric pod's container - nginx-gateway can't be ready.

To Reproduce
Steps to reproduce the behavior:

1. Deploy nginx gateway fabric by helm : helm install ngf oci://ghcr.io/nginxinc/charts/nginx-gateway-fabric --create-namespace -n nginx-gateway
2. then the pod's status is

$ kubectl get pod -n nginx-gateway
NAME                                        READY   STATUS    RESTARTS   AGE
ngf-nginx-gateway-fabric-5985484bb8-fh8dw   1/2     Running   0          21s

3. View logs on ngf-nginx-gateway-fabric-5985484bb8-fh8dw

$  kubectl logs ngf-nginx-gateway-fabric-5985484bb8-fh8dw -c nginx-gateway -n nginx-gateway
{"level":"info","ts":"2024-03-12T02:21:40Z","logger":"controller-runtime.healthz","msg":"healthz check failed","statuses":[{}]}
{"level":"info","ts":"2024-03-12T02:21:41Z","logger":"controller-runtime.healthz","msg":"healthz check failed","statuses":[{}]}
{"level":"info","ts":"2024-03-12T02:21:42Z","logger":"controller-runtime.healthz","msg":"healthz check failed","statuses":[{}]}
{"level":"info","ts":"2024-03-12T02:21:43Z","logger":"controller-runtime.healthz","msg":"healthz check failed","statuses":[{}]}

4. See error
   All logs of nginx gateway are in the attachment.
   nginx-gateway.log

Expected behavior
My nginx-gateway pod can be ready

Your environment

• Version of the NGINX Gateway Fabric - 1.1.0
• Version of Kubernetes - 1.24.2
• Kubernetes platform - 3 nodes k8s cluster
• Details on how you expose the NGINX Gateway Fabric Pod: Both Nodeport and LoadBalancer can't work well.
• Logs of NGINX container:

$ kubectl -n nginx-gateway logs ngf-nginx-gateway-fabric-5985484bb8-fh8dw -c nginx --limit-bytes=1024
/docker-entrypoint.sh: /docker-entrypoint.d/ is not empty, will attempt to perform configuration
/docker-entrypoint.sh: Looking for shell scripts in /docker-entrypoint.d/
/docker-entrypoint.sh: Launching /docker-entrypoint.d/10-listen-on-ipv6-by-default.sh
10-listen-on-ipv6-by-default.sh: info: /etc/nginx/conf.d/default.conf is not a file or does not exist
/docker-entrypoint.sh: Sourcing /docker-entrypoint.d/15-local-resolvers.envsh
/docker-entrypoint.sh: Launching /docker-entrypoint.d/20-envsubst-on-templates.sh
/docker-entrypoint.sh: Launching /docker-entrypoint.d/30-tune-worker-processes.sh
/docker-entrypoint.sh: Configuration complete; ready for start up
2024/03/12 02:21:19 [notice] 20#20: using the "epoll" event method
2024/03/12 02:21:19 [notice] 20#20: nginx/1.25.3
2024/03/12 02:21:19 [notice] 20#20: built by gcc 12.2.1 20220924 (Alpine 12.2.1_git20220924-r10) 
2024/03/12 02:21:19 [notice] 20#20: OS: Linux 3.10.0-1160.81.1.el7.x86_64
2024/03/12 02:21:19 [notice] 20#20: getrlimit(RLIMIT_NOFILE): 65536:6[root@k8s nginx-gateway-fabric-1.1.0]# 
[root@k8s nginx-gateway-fabric-1.1.0]# 
[root@k8s nginx-gateway-fabric-1.1.0]# kubectl -n nginx-gateway logs ngf-nginx-gateway-fabric-5985484bb8-fh8dw -c nginx
/docker-entrypoint.sh: /docker-entrypoint.d/ is not empty, will attempt to perform configuration
/docker-entrypoint.sh: Looking for shell scripts in /docker-entrypoint.d/
/docker-entrypoint.sh: Launching /docker-entrypoint.d/10-listen-on-ipv6-by-default.sh
10-listen-on-ipv6-by-default.sh: info: /etc/nginx/conf.d/default.conf is not a file or does not exist
/docker-entrypoint.sh: Sourcing /docker-entrypoint.d/15-local-resolvers.envsh
/docker-entrypoint.sh: Launching /docker-entrypoint.d/20-envsubst-on-templates.sh
/docker-entrypoint.sh: Launching /docker-entrypoint.d/30-tune-worker-processes.sh
/docker-entrypoint.sh: Configuration complete; ready for start up
2024/03/12 02:21:19 [notice] 20#20: using the "epoll" event method
2024/03/12 02:21:19 [notice] 20#20: nginx/1.25.3
2024/03/12 02:21:19 [notice] 20#20: built by gcc 12.2.1 20220924 (Alpine 12.2.1_git20220924-r10) 
2024/03/12 02:21:19 [notice] 20#20: OS: Linux 3.10.0-1160.81.1.el7.x86_64
2024/03/12 02:21:19 [notice] 20#20: getrlimit(RLIMIT_NOFILE): 65536:65536
2024/03/12 02:21:19 [notice] 20#20: start worker processes
2024/03/12 02:21:19 [notice] 20#20: start worker process 40
2024/03/12 02:21:19 [notice] 20#20: start worker process 41
2024/03/12 02:21:19 [notice] 20#20: start worker process 42

• NGINX Configuration: kubectl -n nginx-gateway exec <gateway-pod> -c nginx -- nginx -T

$ kubectl exec -it ngf-nginx-gateway-fabric-5985484bb8-fh8dw -n nginx-gateway -c nginx --  nginx -T
nginx: the configuration file /etc/nginx/nginx.conf syntax is ok
nginx: configuration file /etc/nginx/nginx.conf test is successful
# configuration file /etc/nginx/nginx.conf:
load_module /usr/lib/nginx/modules/ngx_http_js_module.so;

worker_processes auto;

pid /var/run/nginx/nginx.pid;
error_log stderr info;

events {
  worker_connections 1024;
}

http {
  include /etc/nginx/conf.d/*.conf;
  include /etc/nginx/mime.types;
  js_import /usr/lib/nginx/modules/njs/httpmatches.js;

  default_type application/octet-stream;

  proxy_headers_hash_bucket_size 512;
  proxy_headers_hash_max_size 1024;
  server_names_hash_bucket_size 256;
  server_names_hash_max_size 1024;
  variables_hash_bucket_size 512;
  variables_hash_max_size 1024;

  sendfile on;
  tcp_nopush on;

  server {
    listen unix:/var/run/nginx/nginx-status.sock;
    access_log off;

    location /stub_status {
        stub_status;
    }
  }
}

# configuration file /etc/nginx/conf.d/config-version.conf:

server {
    listen unix:/var/run/nginx/nginx-config-version.sock;
    access_log off;

    location /version {
        return 200 1;
    }
}

# configuration file /etc/nginx/conf.d/http.conf:


upstream invalid-backend-ref {
    random two least_conn;
    zone invalid-backend-ref 512k;
    
    server unix:/var/lib/nginx/nginx-500-server.sock;
}



server {
    listen unix:/var/lib/nginx/nginx-502-server.sock;
    access_log off;

    return 502;
}

server {
    listen unix:/var/lib/nginx/nginx-500-server.sock;
    access_log off;

    return 500;
}



# Set $gw_api_compliant_host variable to the value of $http_host unless $http_host is empty, then set it to the value
# of $host. We prefer $http_host because it contains the original value of the host header, which is required by the
# Gateway API. However, in an HTTP/1.0 request, it's possible that $http_host can be empty. In this case, we will use
# the value of $host. See http://nginx.org/en/docs/http/ngx_http_core_module.html#var_host.
map $http_host $gw_api_compliant_host {
    '' $host;
    default $http_host;
}

# Set $connection_header variable to upgrade when the $http_upgrade header is set, otherwise, set it to close. This
# allows support for websocket connections. See https://nginx.org/en/docs/http/websocket.html.
map $http_upgrade $connection_upgrade {
    default upgrade;
    '' close;
}

# configuration file /etc/nginx/mime.types:

types {
    text/html                                        html htm shtml;
    text/css                                         css;
    text/xml                                         xml;
    image/gif                                        gif;
    image/jpeg                                       jpeg jpg;
    application/javascript                           js;
    application/atom+xml                             atom;
    application/rss+xml                              rss;

    text/mathml                                      mml;
    text/plain                                       txt;
    text/vnd.sun.j2me.app-descriptor                 jad;
    text/vnd.wap.wml                                 wml;
    text/x-component                                 htc;

    image/avif                                       avif;
    image/png                                        png;
    image/svg+xml                                    svg svgz;
    image/tiff                                       tif tiff;
    image/vnd.wap.wbmp                               wbmp;
    image/webp                                       webp;
    image/x-icon                                     ico;
    image/x-jng                                      jng;
    image/x-ms-bmp                                   bmp;

    font/woff                                        woff;
    font/woff2                                       woff2;

    application/java-archive                         jar war ear;
    application/json                                 json;
    application/mac-binhex40                         hqx;
    application/msword                               doc;
    application/pdf                                  pdf;
    application/postscript                           ps eps ai;
    application/rtf                                  rtf;
    application/vnd.apple.mpegurl                    m3u8;
    application/vnd.google-earth.kml+xml             kml;
    application/vnd.google-earth.kmz                 kmz;
    application/vnd.ms-excel                         xls;
    application/vnd.ms-fontobject                    eot;
    application/vnd.ms-powerpoint                    ppt;
    application/vnd.oasis.opendocument.graphics      odg;
    application/vnd.oasis.opendocument.presentation  odp;
    application/vnd.oasis.opendocument.spreadsheet   ods;
    application/vnd.oasis.opendocument.text          odt;
    application/vnd.openxmlformats-officedocument.presentationml.presentation
                                                     pptx;
    application/vnd.openxmlformats-officedocument.spreadsheetml.sheet
                                                     xlsx;
    application/vnd.openxmlformats-officedocument.wordprocessingml.document
                                                     docx;
    application/vnd.wap.wmlc                         wmlc;
    application/wasm                                 wasm;
    application/x-7z-compressed                      7z;
    application/x-cocoa                              cco;
    application/x-java-archive-diff                  jardiff;
    application/x-java-jnlp-file                     jnlp;
    application/x-makeself                           run;
    application/x-perl                               pl pm;
    application/x-pilot                              prc pdb;
    application/x-rar-compressed                     rar;
    application/x-redhat-package-manager             rpm;
    application/x-sea                                sea;
    application/x-shockwave-flash                    swf;
    application/x-stuffit                            sit;
    application/x-tcl                                tcl tk;
    application/x-x509-ca-cert                       der pem crt;
    application/x-xpinstall                          xpi;
    application/xhtml+xml                            xhtml;
    application/xspf+xml                             xspf;
    application/zip                                  zip;

    application/octet-stream                         bin exe dll;
    application/octet-stream                         deb;
    application/octet-stream                         dmg;
    application/octet-stream                         iso img;
    application/octet-stream                         msi msp msm;

    audio/midi                                       mid midi kar;
    audio/mpeg                                       mp3;
    audio/ogg                                        ogg;
    audio/x-m4a                                      m4a;
    audio/x-realaudio                                ra;

    video/3gpp                                       3gpp 3gp;
    video/mp2t                                       ts;
    video/mp4                                        mp4;
    video/mpeg                                       mpeg mpg;
    video/quicktime                                  mov;
    video/webm                                       webm;
    video/x-flv                                      flv;
    video/x-m4v                                      m4v;
    video/x-mng                                      mng;
    video/x-ms-asf                                   asx asf;
    video/x-ms-wmv                                   wmv;
    video/x-msvideo                                  avi;
}

Additional context
Add any other context about the problem here. Any log files you want to share.

[kate-osborn]

Hi @Yelijah I see the following error in the nginx gateway logs:

{"level":"error","ts":"2024-03-12T02:21:19Z","logger":"eventLoop.eventHandler","msg":"Failed to update NGINX configuration","batchID":1,"error":"failed to reload NGINX: failed to send the HUP signal to NGINX main: operation not permitted"

This error would cause the problem you are seeing where the nginx-gateway Pod isn't reporting as ready.

To resolve you will need to tweak the security context of the nginx-gateway Pod. See this troubleshooting guide for details: https://docs.nginx.com/nginx-gateway-fabric/how-to/monitoring/troubleshooting/.

[Yelijah]

Hi @Yelijah I see the following error in the nginx gateway logs:

{"level":"error","ts":"2024-03-12T02:21:19Z","logger":"eventLoop.eventHandler","msg":"Failed to update NGINX configuration","batchID":1,"error":"failed to reload NGINX: failed to send the HUP signal to NGINX main: operation not permitted"

This error would cause the problem you are seeing where the nginx-gateway Pod isn't reporting as ready.

To resolve you will need to tweak the security context of the nginx-gateway Pod. See this troubleshooting guide for details: https://docs.nginx.com/nginx-gateway-fabric/how-to/monitoring/troubleshooting/.

Thank you for your help