bug: 503 errors occur after restarting APISIX when using Consul and APISIX both in Docker

[caijwei]

Current Behavior

Restarting APISIX while it is handling continuous requests leads to intermittent 503 Service Unavailable responses.

I have prepared a minimal reproducible setup. The file structure looks like this:

├── apisix
│   ├── apisix.yaml
│   └── config.yaml
├── nginx
│   └── default.conf
├── init.sh
├── restart.sh
└── test.sh

Once unpacked, the issue can be reproduced easily on any machine that has Docker and curl installed.

1. Run init.sh to launch Consul and the upstream NGINX container that APISIX will route traffic to.
2. Run restart.sh to start APISIX.
3. Run test.sh, which sends continuous requests to APISIX. At first, all responses should return HTTP status code 200.
4. While test.sh is running, repeatedly execute restart.sh to restart APISIX.
5. After restarting, even when the new APISIX instance is up and running normally, you will still see intermittent 503 errors in the output of test.sh.

This indicates that APISIX fails to route traffic correctly after a restart, despite the upstream and service discovery being healthy.

Expected Behavior

After restarting APISIX, once it has successfully started and is healthy, it should be able to route traffic to upstream services registered in Consul without returning 503 errors.

Error Logs

When test.sh starts to show 503 responses, you can check the APISIX container logs to see detailed error messages. These logs provide insight into why the requests failed, such as upstream discovery or routing issues.

Steps to Reproduce

1. Unpack the minimal test setup files.
2. Run: ./init.sh
3. Start APISIX: ./restart.sh
4. Start testing: ./test.sh
5. In a separate terminal, repeatedly run: ./restart.sh
6. Watch test.sh output — it will gradually show 503 errors.

Environment

apisix-debug.zip

[caijwei]

After testing, you can clean up all related containers and the network created during this run by executing the following command:
docker ps -a --format '{{.Names}}' | grep '^apisix-debug-' | xargs -r docker rm -f && docker network rm apisix-debug

[hanqingwu]

@caijwei , yes , it can reproduce. some work process always 503, some work process always 200.
I think some work process can not callback discovery_consul_callback .
Maybe events:register failed or skip . I try to add some code to trace it .

[hanqingwu]

@caijwei, Can you try to add these params in apisix/config.yaml:

  events:                             # Event distribution module configuration
    module: lua-resty-events          # Sets the name of the events module used.

Also get good status 200.

[caijwei]

@hanqingwu I’d like to provide some additional test details:

The host machine is a MacBook Pro with an M3 chip (Apple Silicon), running APISIX inside Docker. Without explicitly setting worker_processes, the default value is auto, and it starts 8 worker processes as expected.

Here’s the confirmation:

root@4cb79f205ed5:/usr/local/apisix# cat /usr/local/apisix/conf/nginx.conf | grep worker_processes
worker_processes auto;
root@4cb79f205ed5:/usr/local/apisix# ps aux | grep 'nginx: worker process' | grep -v grep | wc -l
8
root@4cb79f205ed5:/usr/local/apisix# lscpu
Architecture:                         aarch64
CPU op-mode(s):                       64-bit
Byte Order:                           Little Endian
CPU(s):                               8
On-line CPU(s) list:                  0-7
Thread(s) per core:                   1
Core(s) per socket:                   8
Socket(s):                            1
Vendor ID:                            0x61
Model:                                0
Stepping:                             0x0
BogoMIPS:                             48.00
Vulnerability Gather data sampling:   Not affected
Vulnerability Itlb multihit:          Not affected
Vulnerability L1tf:                   Not affected
Vulnerability Mds:                    Not affected
Vulnerability Meltdown:               Not affected
Vulnerability Mmio stale data:        Not affected
Vulnerability Reg file data sampling: Not affected
Vulnerability Retbleed:               Not affected
Vulnerability Spec rstack overflow:   Not affected
Vulnerability Spec store bypass:      Mitigation; Speculative Store Bypass disabled via prctl
Vulnerability Spectre v1:             Mitigation; __user pointer sanitization
Vulnerability Spectre v2:             Not affected
Vulnerability Srbds:                  Not affected
Vulnerability Tsx async abort:        Not affected
Flags:                                fp asimd evtstrm aes pmull sha1 sha2 crc32 atomics fphp asimdhp cpuid asimdrdm jscvt fcma lrcpc dcpop sha3 asimddp sha512 asimdfhm dit uscat ilrcpc flagm ssbs sb paca pacg dcpodp flagm2 frint

As per your suggestion, I added the lua-resty-events module configuration to config.yaml and explicitly set worker_processes: 8.

Unfortunately, neither change had any effect. After running restart.sh several times, intermittent 503 errors still occur quite easily.

[hanqingwu]

Can you try to reduce worker_processes to 4 ?

[caijwei]

After setting worker_processes to 4, the first few restarts all returned 200 as expected, but on the 13th time I ran ./restart.sh, a 503 error occurred again.

[caijwei]

After adding events.module and setting worker_processes to 4, 503 errors still occasionally occur, and they follow a consistent pattern.

During the execution of restart.sh, test.sh is continuously running. If the output of test.sh at the moment of restart is 000, then all responses after the restart are consistently 200. However, if the output is 503 at the time of the restart, intermittent 503 errors will continue to appear even after the restart completes.

This behavior does not occur with the original version of the ZIP file I provided earlier — in that version, the output at the moment of restart is always 000, regardless of when the restart is triggered, and 503 errors tend to appear more easily and frequently.

[hanqingwu]

Yes ， I think root cause is that events:register not thread safe . when multi work process concurrent call events:register, some register info will overload by other work process ,so this work process can not get events:post . discovery_consul_callback can not be called . this work process always 503 happen.

So , If your env is not high concurrency , set worker_processes to 1 should fix this temporarily.
Final solution need be discussed.

[Baoyuantop]

Hi @caijwei, I can't reproduce this problem with the script and steps you provided. I set worker_processes to 8, and repeated restart.sh about 10 times. After each execution, a 503 error appeared, but it didn't appear after that.