Cannot expose NGF readiness probe (/readyz on 8081) through Gateway Service for Azure Application Gateway health checks.

[avi1512]

Describe the bug

We are running NGINX Gateway Fabric (NGF) on an AKS cluster behind an Azure Application Gateway (AppGW).
NGF is deployed with an internal LoadBalancer Service, whose IP is added to AppGW as a backend.

AppGW requires an HTTP health probe. We attempted to use NGF’s built-in readiness endpoint:

• Path: /readyz
• Port: 8081 (hardcoded probe port inside NGF container)

NGF exposes only port 80 at the Service level, so attempts to expose 8081 via:

• Gateway listeners
• Service ports
• Helm chart overrides
• NginxProxy configuration

all result in NGF becoming Unready, and readiness probes fail with HTTP 404.

NGF logs show:

[warn] conflicting server name "" on 0.0.0.0:8081, ignored
[warn] conflicting server name "" on [::]:8081, ignored 

Additionally, when attempting to re-expose 8081, NGF and the container logs repeated 404s:

[warn] conflicting server name "" on 0.0.0.0:8081, ignored

"GET /readyz HTTP/1.1" 404 146 "-" "kube-probe/1.33"

This makes it impossible to configure Application Gateway health probes for NGF.

To Reproduce

Deploy NGF (2.2.1) on AKS.

Attempt to expose readiness probe via Gateway:

listeners:
  - name: health
    port: 8081
    protocol: HTTP

NGF readiness probe fails with HTTP 404.

Logs show:

conflicting server name "" on 0.0.0.0:8081, ignored

Azure Application Gateway marks backend as Unhealthy.

Expected behavior

We expect one of the following to work:

NGF allows exposing its internal readiness endpoint (/readyz on 8081) via:

• Gateway listeners

• Service ports

• Helm configuration

NGF provides a documented, stable health endpoint on the Service load balancer (e.g., /healthz on port 80), similar to other ingress controllers. The readiness probe port becomes configurable, so users can align it with Azure health probe requirements. Currently, none of these options work.

Your environment

NGF version: 2.2.1

Kubernetes: AKS 1.33

Platform: AKS behind Azure Application Gateway

Exposure: Internal LoadBalancer Service

NGINX container logs (excerpt):

time=2025-12-04T18:11:11.790Z level=INFO msg="NGINX config tested" output="2025/12/04 18:11:11 [notice] 34#34: js vm init njs: 00007FDC5C23DC80\n2025/12/04 18:11:11 [warn] 34#34: conflicting server name \"\" on 0.0.0.0:8081, ignored\nnginx: [warn] conflicting server name \"\" on 0.0.0.0:8081, ignored\n2025/12/04 18:11:11 [warn] 34#34: conflicting server name \"\" on [::]:8081, ignored\nnginx: [warn] conflicting server name \"\" on [::]:8081, ignored\nnginx: the configuration file /etc/nginx/nginx.conf syntax is ok\nnginx: configuration file /etc/nginx/nginx.conf test is successful\n" correlation_id=6d69e1ce-36c3-4641-bfec-f292246ca83f server_type=command
time=2025-12-04T18:11:11.790Z level=INFO msg="Reloading NGINX PID" pid=8 correlation_id=6d69e1ce-36c3-4641-bfec-f292246ca83f server_type=command
2025/12/04 18:11:11 [notice] 8#8: signal 1 (SIGHUP) received from 18, reconfiguring
time=2025-12-04T18:11:11.791Z level=INFO msg="No NGINX error logs found to monitor" correlation_id=6d69e1ce-36c3-4641-bfec-f292246ca83f server_type=command
2025/12/04 18:11:11 [notice] 8#8: reconfiguring
2025/12/04 18:11:11 [notice] 8#8: js vm init njs: 00007F87FA17AA00
2025/12/04 18:11:11 [warn] 8#8: conflicting server name "" on 0.0.0.0:8081, ignored
2025/12/04 18:11:11 [warn] 8#8: conflicting server name "" on [::]:8081, ignored
10.21.144.255 - - [04/Dec/2025:18:11:16 +0000] "GET /readyz HTTP/1.1" 404 146 "-" "kube-probe/1.33"
10.21.144.255 - - [04/Dec/2025:18:11:26 +0000] "GET /readyz HTTP/1.1" 404 146 "-" "kube-probe/1.33"
10.21.144.255 - - [04/Dec/2025:18:11:36 +0000] "GET /readyz HTTP/1.1" 404 146 "-" "kube-probe/1.33"

NGINX config excerpt from NGF container:

server {
    listen 8081;
    location = /readyz {
        return 200;
    }
}

Additional context

#3629

We also attempted to use the new custom readinessProbe port feature (introduced by PR #3629) by modifying the NGF Helm chart to set the readiness probe port to 8082.

Example from the generated NginxProxy resource:

readinessProbe:
  initialDelaySeconds: 5
  port: 8082

The NGF Pod initially starts correctly and shows the updated readiness probe:

Readiness: http-get http://:8082/readyz

However, as soon as we expose port 8082 externally (either via a Gateway listener, Kubernetes Service port, or App Gateway probe), the NGF Pod becomes unhealthy, and readiness fails with 404 Not Found, identical to the behavior on port 8081.

NGF logs show the same conflict warnings but now on port 8082:

2025/12/04 18:11:11 [warn] 8#8: conflicting server name "" on 0.0.0.0:8082, ignored
2025/12/04 18:11:11 [warn] 8#8: conflicting server name "" on [::]:8082, ignored

Followed by repeated 404s for the readiness probe:

"GET /readyz HTTP/1.1" 404 "-" "kube-probe/1.33"

So even with a custom readiness probe port, NGF still will not serve /readyz once the port is exposed outside the Pod, and the Pod falls into a crash loop:

Readiness probe failed: HTTP probe failed with statuscode: 404

This confirms:

The issue is not limited to 8081

Any readiness probe port exposed through a Service or listener results in port conflict warnings and NGF failing readiness

/readyz is not reliably reachable through any externally exposed port, even when configured.

[nginx-bot]

Hi @avi1512! Welcome to the project! 🎉

Thanks for opening this issue!
Be sure to check out our Contributing Guidelines and the Issue Lifecycle while you wait for someone on the team to take a look at this.

[salonichf5]

Hey @avi1512, thanks for raising this and for your interest in NGINX Gateway Fabric.
I was able to reproduce this behavior on our side as well.

Just to confirm: your main requirement is to have a health endpoint that Azure Application Gateway can probe, so it can determine NGF’s health?

As a temporary workaround, you can patch the NGF's control-plane Service to expose the readiness port used by /readyz (for example, by adding a health port that forwards to the same container port as the readiness probe).
This allows AppGW to call /readyz directly on that Service port without needing a Gateway listener on that port.

That said, I agree this should be supported out of the box. We will prioritize this issue on our side to track exposing a health endpoint suitable for external load balancers (like AppGW) as part of the standard NGF installation.

[yugannkt]

Hello @salonichf5

Proposed Solution

Root Cause

The problem occurs because the readiness probe server block is generated without a server_name directive. When a Gateway listener is created on the same port (whether 8081, 8082, or any custom port), NGINX sees two server blocks on the same port with empty/no server_name, resulting in:

[warn] conflicting server name "" on 0.0.0.0:8081, ignored

NGINX then ignores one of the server blocks (typically the readiness probe), causing /readyz to return 404.

Solution

Add a unique server_name directive to the readiness probe server block:

server {
    listen 8081;
    listen [::]:8081;
    server_name nginx-readiness-probe.internal;  # ← Added
    
    location = /readyz {
        access_log off;
        return 200;
    }
}

This allows both the readiness probe and Gateway listener to coexist on the same port without conflict, as NGINX can distinguish them by their server_name.

Would love to hear your thoughts on this approach! Does this align with what you had in mind for supporting health endpoints? I'll open a PR once we agree this is the right direction.

[avi1512]

Just to confirm: your main requirement is to have a health endpoint that Azure Application Gateway can probe, so it can determine NGF’s health?

Hi @salonichf5 , Yes, that’s correct.
Our requirement is to provide Azure Application Gateway with a stable health probe endpoint so it can determine the health of NGF.

Since AppGW needs an HTTP endpoint exposed on the Service IP, we are attempting to use NGF’s built-in readiness endpoint:

Path: /readyz

Port: 8081 (or custom port like 8082)

However, because NGF does not expose this port through the Service by default, AppGW cannot access it unless we manually expose it — and that currently leads to the port conflict and 404 readiness failures described in the issue.

Thanks for the suggestion — the workaround worked successfully. we patched the NGF control-plane Service using the following NginxProxy patch:

After applying this patch:

The Service began exposing port 8082

Azure Application Gateway was able to reach /readyz

Health probe validation succeeded

NGF remained healthy (no readiness probe failures)

So the workaround resolves the issue for now.

I agree this should be supported natively, and I appreciate that you're prioritizing a built-in health endpoint for external load balancers.

I have one follow-up question regarding the behavior we observed:

Before applying the NginxProxy patch, we manually edited the NGF control-plane Service several times using kubectl edit svc nginx-gateway-fabric. Kubernetes showed the changes in the editor, but the Service ports never actually updated—the modifications were silently overridden.

However, after applying the Service patch through the NginxProxy resource, the port was added successfully and the AppGW health probe started working.

Why does manually editing the Service not persist, while patching through the NginxProxy works correctly?
Is NGF reconciling and overwriting the Service definition on every sync, and therefore only changes applied through NginxProxy are honored?

Just want to understand the expected behavior so we follow the correct method going forward.

[salonichf5]

@yugannkt thanks a lot for the thoughtful analysis and proposal.
I agree this is definitely part of the solution — we should avoid cases where the readiness port (e.g. 8081) and data-plane server blocks collide, especially when they end up sharing the same implicit server name.

That said, I believe the original ask from the user was specifically to have port 8081 exposed on the NGF Service so that their Azure Application Gateway can reach the /readyz endpoint. I think a solid fix would be:

• Expose the readiness/health port on the NGF Service, so external LBs like AppGW can probe it directly, and
• Give the readiness server block a distinct server_name to avoid future collisions on the NGINX side if that port is ever reused.

Thanks again for being proactive here — really appreciate your contribution to the project. Would you be interested to work on it? I’d be happy to review a PR that implements this approach and help move it forward.

[salonichf5]

@avi1512 glad to hear the patch worked 🎉
NGF manages the Service and Deployment based on the NginxProxy resource (and Helm values at install time), so any manual changes made directly with kubectl edit don't get reconciled back (have not tried it editing a service myself). That’s why updates applied via NginxProxy persist, while direct edits to the Service do not.

[avi1512]

@avi1512 glad to hear the patch worked 🎉 NGF manages the Service and Deployment based on the NginxProxy resource (and Helm values at install time), so any manual changes made directly with kubectl edit don't get reconciled back (have not tried it editing a service myself). That’s why updates applied via NginxProxy persist, while direct edits to the Service do not.

Thanks, Got it.

[yugannkt]

Hi @salonichf5,

Thank you for the feedback and support! I really appreciate your guidance on this issue.

Server Name Conflict Fix

I'll work on implementing the solution to add a unique server_name directive to the readiness probe server block to prevent conflicts when Gateway listeners share the same port. I'll submit a PR for this fix soon.

Additional Feature Proposal: Generic Additional Service Ports

While analyzing this issue with the user( @avi1512 ) , I identified a broader use case that could benefit us.. Currently, users can only expose ports through:

1. Gateway listener ports (80, 443, etc.)
2. Service patches (JSONPatch/StrategicMerge)

However, there are several operational scenarios where users need to expose additional ports on the Service:

• External load balancer health checks (the original issue Cannot expose NGF readiness probe (/readyz on 8081) through Gateway Service for Azure Application Gateway health checks. #4400)
• External metrics scraping
• NGINX Plus API/dashboard access
• Custom monitoring/debugging ports

Proposed Solution

Add a simple, declarative way to expose additional ports via Helm values:

nginx:
  service:
    additionalPorts:
      - name: health
        port: 8082
        targetPort: 8082
        protocol: TCP
      - name: metrics
        port: 5001
        targetPort: 5001
        protocol: TCP

This provides a first-class, declarative way to expose operational ports without requiring users to write Kubernetes patches for this common scenario.

Would this align with NGF's design principles? Happy to implement the server name fix first, then add this feature in a separate PR, or combine them as you see fit.

[salonichf5]

@yugannkt I think we should only expose the health port out of the box, and let users add any additional ports via patches if they need them. It’s definitely a nice-to-have for usability — @sjberman, what do you think?

For now, the main priority is to expose the health endpoint and ensure we don’t get any future server name collisions with the health server block.