Update nginx configuration to forward request to next service node on 503 response

[dhatri308]

Hi,

We have a case in which if a service pod returns 503 to nginx, we would like to forward the request to next service pod. We added below config to "Ingress" manifest in server snippets but it doesn't seem to work. Can someone pls let us know if this would work ? thank you. More details below.

"metadata": {
    "annotations": {
      "nginx.org/server-snippets":"location /rest/v1/(abc|xyz)  \n{\n  proxy_next_upstream error http_503;\n  proxy_next_upstream_tries 3;\n}
                             }
                     }

[vepatel]

@dhatri308 we'll take at look at this, thanks!

[vepatel]

Hey @dhatri308, are you looking to apply this directive to locations defined by Ingress object or custom location defined in server-snippet?
Please make sure you've enabled -enable-snippets flag in deployment before applying Ingress objects with snippets.

If you're looking for Ingress defined locations, then you should use location-snippets instead:

apiVersion: networking.k8s.io/v1
kind: Ingress
metadata:
  name: cafe-ingress
  annotations:
    nginx.org/location-snippets: |
      proxy_next_upstream error timeout http_503;
spec:
  ingressClassName: nginx
  tls:
  - hosts:
    - cafe.example.com
    secretName: cafe-secret
  rules:
  - host: cafe.example.com
    http:
      paths:
      - path: /tea
        pathType: Prefix
        backend:
          service:
            name: tea-svc
            port:
              number: 80
      - path: /coffee
        pathType: Prefix
        backend:
          service:
            name: coffee-svc
            port:
              number: 80

conf:

# configuration for default/cafe-ingress
upstream default-cafe-ingress-cafe.example.com-coffee-svc-80 {zone default-cafe-ingress-cafe.example.com-coffee-svc-80 256k;
        random two least_conn;
        server 10.112.0.14:8080 max_fails=1 fail_timeout=10s max_conns=0;
        server 10.112.1.8:8080 max_fails=1 fail_timeout=10s max_conns=0;
}

upstream default-cafe-ingress-cafe.example.com-tea-svc-80 {zone default-cafe-ingress-cafe.example.com-tea-svc-80 256k;
        random two least_conn;
        server 10.112.0.15:8080 max_fails=1 fail_timeout=10s max_conns=0;
        server 10.112.1.9:8080 max_fails=1 fail_timeout=10s max_conns=0;
        server 10.112.2.7:8080 max_fails=1 fail_timeout=10s max_conns=0;
}



server {
        listen 80;listen [::]:80;
        listen 443 ssl;listen [::]:443 ssl;
        ssl_certificate $secret_dir_path/default-cafe-secret;
        ssl_certificate_key $secret_dir_path/default-cafe-secret;

        server_tokens on;

        server_name cafe.example.com;

        set $resource_type "ingress";
        set $resource_name "cafe-ingress";
        set $resource_namespace "default";
        if ($scheme = http) {
         return 301 https://$host:443$request_uri;
        }
        location /tea {
         set $service "tea-svc";
         proxy_http_version 1.1;

         proxy_next_upstream error timeout http_503;

         proxy_connect_timeout 60s;
         proxy_read_timeout 60s;
         proxy_send_timeout 60s;
         client_max_body_size 1m;
         proxy_set_header Host $host;
         proxy_set_header X-Real-IP $remote_addr;
         proxy_set_header X-Forwarded-For $proxy_add_x_forwarded_for;
         proxy_set_header X-Forwarded-Host $host;
         proxy_set_header X-Forwarded-Port $server_port;
         proxy_set_header X-Forwarded-Proto $scheme;
         proxy_buffering on;
         proxy_pass http://default-cafe-ingress-cafe.example.com-tea-svc-80;


        }

        location /coffee {
         set $service "coffee-svc";
         proxy_http_version 1.1;

         proxy_next_upstream error timeout http_503;

         proxy_connect_timeout 60s;
         proxy_read_timeout 60s;
         proxy_send_timeout 60s;
         client_max_body_size 1m;
         proxy_set_header Host $host;
         proxy_set_header X-Real-IP $remote_addr;
         proxy_set_header X-Forwarded-For $proxy_add_x_forwarded_for;
         proxy_set_header X-Forwarded-Host $host;
         proxy_set_header X-Forwarded-Port $server_port;
         proxy_set_header X-Forwarded-Proto $scheme;
         proxy_buffering on;
         proxy_pass http://default-cafe-ingress-cafe.example.com-coffee-svc-80;


        }

}

[dhatri308]

Hi @vepatel Thanks for your help. I moved the setting to location snippet and it did work. However all of the location paths in Ingress got these settings. How do I restrict these settings to only few custom ingress locations ? thanks

"nginx.org/location-snippets":"proxy_next_upstream error http_503;\nproxy_next_upstream_tries 3;"

[vepatel]

hi @dhatri308, if you'd like to modify specific paths I'd recommend using VirtualServer resource instead of Ingress. Using location-snippets there will allow you to pick and chose the locations you want to modify.

see: https://docs.nginx.com/nginx-ingress-controller/configuration/virtualserver-and-virtualserverroute-resources/#virtualserverroute
basic VirtualServer example: https://github.com/nginxinc/kubernetes-ingress/tree/v3.7.0/examples/custom-resources/basic-configuration

[pdabelf5]

Hi @dhatri308, as you have discovered, the location-snippets annotation applies to every path in the Ingress. There is not currently a way to filter which paths get the snippet applied.

[dhatri308]

Hi, I have a follow-up question. I see that the request does not get forwarded to next service pod if we have "proxy_next_upstream_tries" set to 1 . But I change this param to 2 and then I see request getting forwarded to next service pod. Could you please let us know if this is expected ? thanks