nginx returns 5-byte sized responses with status code 200

[GiladTrachtenberg]

Environment

Details:

• Nginx version: 1.27.3
• OS kernel version - linux (amd64)

Description

nginx returns absurdly small responses (5 bytes) with status 200 following CDN cache purges when trying to access a single page application (/pay). It only happens in mobile platforms; not in web-based ones:

<redacted_ip_address> - - [04/Oct/2025:10:03:40 +0000] "GET /pay?packageId=<redacted_client_related_info> HTTP/1.1" 200 5 "-" "Mozilla/5.0 (iPhone; CPU iPhone OS 18_7 like Mac OS X) AppleWebKit/605.1.15 (KHTML, like Gecko) Mobile/15E148" "<redacted_ip_address_2>, <redacted_ip_address_3>"

The problem is that I can't seem to reproduce this bug. Here are all of the methods I've tried:

• Trying to replicate in our dev environment (which doesn't have caching) simply doesn't work. When there's no cache, nginx returns totally valid responses.
• Increasing the disk space of the nginx server
• Ruled out any inherent infrastructure issues - it works just fine even without caching, the only exception is mobile platforms. Plus, there are no inherent errors; just absurdly small status 200 responses.
• Configuring the following parameters:
  • client_max_body_size: when setting it to an absurdly small value, it returns an actual error, not status 200.
  • lingering_time: same as the above ^.
  • keepalive_timeout: same as the above ^.
  • worker_processes: makes no difference whether I keep the default value or increase it to 4/8.
  • client_header_buffer_size: same as the above ^ - no difference.
  • large_client_header_buffers: error 400 is returned if the value is too small.
  • proxy_buffering: if it's on and the correlating values are too small, it returns an actual error, not status 200. If it's off, it makes no difference - the situation remains as it is.
  • proxy_buffer_size: returns an error if proxy_buffering is on and value is too small.
  • proxy_busy_buffers: same as the above ^.
  • proxy_temp_file_write_size: returns an error if value is too small; default value makes no difference.
  • proxy_max_temp_file_size: same as the above ^.
• Listening port is not misconfigured - else it wouldn't have worked in all platforms other than mobile.
• SSL issue - that's not it, because the server is configured to listen in HTTP.
• Not an IP issue either, as the IP remains static.
• Not the application buffers as well - if the value is too small, nginx will return error 400.
• Midflight cache purges - unfortunately, that's not it, either. Tested by sending multiple concurrent request mid cache-purge.
• HTTP/2 to HTTP/1.1 mismatch - not the issue. Nginx only supports 1.1. Even when I tried to test with explicit HTTP/2.0 requests, but nginx simply responds with HTTP/1.1 and the issue isn't replicated.

Small Hints of Promise

I created a mirroring environment with the same infrastructure and settings, but in a different domain. I couldn't reproduce the 5 byte-sized responses with status code 200, but two interesting things happened:

• Tried mobile requests to the specific endpoint from the log (which I redacted). Pre-cache/post cache-purge return different-sized responses. In some mobile browsers, it doesn't even load at all. In firefox, however, it loads up just fine. On PC-based clients, cache purges didn't affect anything, regardless of the browswer.
• I modified the config a bit to check if the upstream header is empty/missing. If it is, I set a variable to 1. If it's not empty, the variable defaults to 0. Then, in the individual location blocks, I'd check if that varible was equal to 1 - if so, I'd return status 404. What ended up happening when I tried the specific endpoint from the log (the one I redacted) is that I got error 404. But when I dropped that setting, I was getting the correct page that the endpoint references.

Checklist

• The bug is reproducible with the latest version of nginx. Couldn't reproduce this bug myself.
• The nginx configuration is minimized to the smallest possible to reproduce the issue and doesn't contain third-party modules

nginx configuration

resolver <private_ip> ipv6=off valid=300s;
server {
  server_name <my_domain>.com;

  set $s3_app_bucket_url "<mydomain>.com.s3-website-us-west-2.amazonaws.com";
  set $pay_url "pay.<mydomain>.com.s3-website-us-west-2.amazonaws.com";

  add_header Referrer-Policy "no-referrer-when-downgrade";

  add_header Content-Security-Policy "default-src * 'unsafe-inline' 'unsafe-eval'; script-src * 'unsafe-inline' 'unsafe-eval'; connect-src * 'unsafe-inline'; img-src * data: blob: 'unsafe-inline'; frame-src *; style-src * 'unsafe-inline';" ;

  location /pay {
      add_header Cache-Control "no-cache, no-store, must-revalidate";
      rewrite ^/pay/(.*)$ /$1 break;
      proxy_pass             http://$pay_url;
      proxy_redirect         off;
      proxy_http_version     1.1;
      proxy_set_header       X-Real-IP $remote_addr;
      proxy_buffering        on;
      proxy_buffer_size      128k;
      proxy_buffers          256 128k;
      proxy_busy_buffers_size 256k;
      proxy_temp_file_write_size 256k;
      proxy_max_temp_file_size 1024m;
      proxy_hide_header      x-amz-id-2;
      proxy_hide_header      x-amz-request-id;
      proxy_hide_header      x-amz-meta-server-side-encryption;
      proxy_hide_header      x-amz-server-side-encryption;
      proxy_hide_header      Set-Cookie;
      proxy_ignore_headers   Set-Cookie;
      add_header X-Frame-Options "SAMEORIGIN" always;
      add_header Content-Security-Policy "frame-ancestors 'self'" always;
      proxy_intercept_errors on;
      error_page 404 =200 @pay_index;
  }

  location @pay_index {
      add_header Cache-Control "no-cache, no-store, must-revalidate";
      add_header X-Frame-Options "SAMEORIGIN" always;
      add_header Content-Security-Policy "frame-ancestors 'self'" always;
      proxy_pass http://$pay_url/index.html;
  }

  location / {
      add_header Cache-Control "no-cache, no-store, must-revalidate";
      add_header X-Frame-Options "SAMEORIGIN" always;
      add_header Content-Security-Policy "frame-ancestors 'self'" always;
      proxy_http_version     1.1;
      proxy_redirect         off;
      proxy_set_header       X-Real-IP $remote_addr;
      proxy_hide_header      x-amz-id-2;
      proxy_hide_header      x-amz-request-id;
      proxy_hide_header      x-amz-meta-server-side-encryption;
      proxy_hide_header      x-amz-server-side-encryption;
      proxy_hide_header      Set-Cookie;
      proxy_ignore_headers   Set-Cookie;
      proxy_intercept_errors on;

      proxy_pass http://$s3_app_bucket_url;
      error_page 404 =200 @default_index;
  }

  location @default_index {
      add_header Cache-Control "no-cache, no-store, must-revalidate";
      add_header X-Frame-Options "SAMEORIGIN" always;
      add_header Content-Security-Policy "frame-ancestors 'self'" always;
      proxy_pass http://$s3_app_bucket_url/index.html;
  }
}

[GiladTrachtenberg]

I couldn't reproduce the bug due to the scale of the issue (more on that later), but there's a definitive solution I found based solely in nginx (and more outside of it, but it's irrelevant for now, I guess).

The solution relies on utilizing a custom lua script to catch responses/requests with sizes smaller than an x amount of bytes and retry the request (which works in my case). Something like so:

location /securepay {
    # ... your existing config ...
    
    header_filter_by_lua_block {
        local body_size = tonumber(ngx.var.body_bytes_sent) or 0
        if body_size < 98 then
            ngx.var.retry_needed = "1"
        end
    }
    
    body_filter_by_lua_block {
        if ngx.var.retry_needed == "1" then
            -- Trigger retry by setting error status
            ngx.exit(ngx.HTTP_SERVICE_UNAVAILABLE)
        end
    }
    
    proxy_next_upstream error timeout http_503;
    proxy_next_upstream_tries 3;
}

Regarding the scale of the issue - due to the sheer scale of the user base that translates into significant network traffic, reproducing this bug is easier said than done.