Add property to configure the queue size for Tomcat

[mhalbritter]

As of now, the request processing queue size of Tomcat can't be adjusted and defaults to an unbounded queue (this is due to the code in org.apache.tomcat.util.net.AbstractEndpoint#createExecutor).

There are two properties named server.tomcat.accept-count and server.tomcat.max-connections but they work on the TCP connection level, and with HTTP/1.1 keep-alive/pipelining and HTTP/2 multiplexing multiple requests onto one connection this doesn't cut it.

We added it for Jetty via this PR where it was decided to not do it for Tomcat as we already have accept-count and max-connections.

Right now, out of the box, a Tomcat application which takes 5 second for a response can be easily overwhelmed because the connections pile up in the queue. Even after stopping the load generator, the application takes some time to recover and to clear the queue.

There's a workaround in code:

@Component
@EnableConfigurationProperties(ServerProperties.class)
class MyProtocolHandlerCustomizer implements TomcatProtocolHandlerCustomizer<ProtocolHandler> {
    private final ServerProperties serverProperties;

    MyProtocolHandlerCustomizer(ServerProperties serverProperties) {
        this.serverProperties = serverProperties;
    }

    @Override
    public void customize(ProtocolHandler protocolHandler) {
        ServerProperties.Tomcat.Threads threads = this.serverProperties.getTomcat().getThreads();
        TaskQueue queue = new TaskQueue(100);
        ThreadPoolExecutor executor = new ThreadPoolExecutor(threads.getMinSpare(), threads.getMax(), 60000, TimeUnit.MILLISECONDS, queue);
        queue.setParent(executor);
        protocolHandler.setExecutor(executor);
    }
}

This limits the queue to 100. When doing it that way, Tomcat closes the socket immediately when the queue is full.

I think we should reconsider adding a server.tomcat.threads.max-queue-capacity property. WDYT?

[mhalbritter]

After some more investigation:

accept-count is directly used as a parameter to the ServerSocketChannel.bind method. The JavaDoc of that parameter says:

The backlog parameter is the maximum number of pending connections on the socket. Its exact semantics are implementation specific. In particular, an implementation may impose a maximum length or may choose to ignore the parameter altogether. If the backlog parameter has the value 0, or a negative value, then an implementation specific default is used.

I would have expected that, when setting this to 1, the OS declines new connection requests after 1 request is in the accept queue. This is not the case on MacOS, this parameter doesn't seem to have any effect, as clients still wait in some queue. This queue is not the executor queue of the protocol handler. I'll check again with Linux if this has an effect there.

On Linux, it seems to work. At least, netstat reports that the listen queue overflowed (MacOS doesn't do that):

netstat -s | grep -i listen
    25265 times the listen queue of a socket overflowed
    25265 SYNs to LISTEN sockets dropped

A client still hangs until the client timeout is reached.

[mhalbritter]

After some discussion and taking a look at Jetty, we decided to add a property which lets users limit the Tomcat queue size. We default that property to the same default Tomcat has (unbounded), because we generally try to stick to the default behavior of each container so that users who are already familiar with Tomcat get a similar experience when using it embedded in Boot. With that property you can configure both Jetty and Tomcat to behave the same way: they print a log when the queue overflows and immediately drop the connection.

We should also revisit the descriptions of server.tomcat.accept-count and server.tomcat.max-connections and see if we can make things more clear what property is doing what.

What I found out so far:

• server.tomcat.accept-count directly relates to the backlog parameter of the ServerSocketChannel and controls the length of the accept queue of that socket in the OS. When this limit is reached, the OS should drop the connection. This may or may not be the case, depending on the socket implementation. On Linux, from a client point of view, it just looks like a dead connection which isn't closed.
• server.tomcat.max-connections is essentially a semaphore guarding the accept call of the server socket. It's checked inside the Acceptor of Tomcat and limits the maximum amount of accepted connections. The semaphore is released when the connection is closed. When reaching this limit, the acceptor thread blocks, and I guess connections then queue up in the accept queue of the socket. Together with accept-count which (should) limit the accept queue size this load-sheds on the server side. The downside is that from the client perspective this looks like a dead connection which is not closed.

After this issue is resolved: To get the behavior of immediately closing the connection when the queue is full and no request handling threads are available, set the property server.tomcat.threads.max-queue-capacity to some useful value depending on your use case. When setting to 0, this immediately closes the connection if all threads are busy.

[mhalbritter]

I looked at how this could be implemented and stumbled over some problems. There's no easy way to customize the int capacity argument from the TaskQueue backing the ThreadPoolExecutor (which is created in org.apache.tomcat.util.net.AbstractEndpoint#createExecutor). You can set a custom executor via org.apache.tomcat.util.net.AbstractEndpoint#setExecutor but this has some ugly consequences:

1. You need to essentially copy the executor setup code from org.apache.tomcat.util.net.AbstractEndpoint#createExecutor
2. When setting a custom executor, Tomcat sets org.apache.tomcat.util.net.AbstractEndpoint#internalExecutor to false, and when the connector is shut down, it doesn't shutdown the executor (see the code in org.apache.tomcat.util.net.AbstractEndpoint#shutdownExecutor).

You also can't get the TaskQueue from the existing executor and modify it. While org.apache.tomcat.util.threads.ThreadPoolExecutor#getQueue returns the queue, it has no setter, cause the workQueue field is final. The org.apache.tomcat.util.threads.TaskQueue class itself is immutable in regards of the capacity, too.

I think a better approach would be to ask the Tomcat team if they could provide a configuration parameter maxQueueSize which is then used in org.apache.tomcat.util.net.AbstractEndpoint#createExecutor. I'll open an issue on the Tomcat tracker.

[evelzi]

Hey there,

Are you still working on this?

I'm not quite sure if I've got it right. Are you saying that multiple tasks get piled up in the task queue when making several requests through a single connection? Does that mean the "task queue" holds all these "request tasks"? I mean, n requests put n elements in the queue, it doesn't matter if they are through the same connection....?
And could the queue potentially keep growing indefinitely, even with just one established connection?

[mhalbritter]

Hey! No, it's involving multiple connections. The load generator closes the connection after some time, and opens a new one. But the closed connection still lingers in the queue until Tomcat wants to work on it, then sees that it's closed, and then removes it from the queue. At least that is what I think, I'm not that familiar with Tomcat internals.

[evelzi]

I have recently conducted a test using HTTP/2 and multiplexing to handle numerous requests within a single connection. I can affirm that this approach allows for the concurrent generation of multiple requests through a single connection, which can potentially lead to the queue expanding well beyond the defined limit set by the "max-connections" parameter.

This underscores the importance of also considering a limit on the queue size, often referred to as the "max queue size."

[ahmedhus]

Tomcat already offers maxQueueSize config property as part of the StandardThreadExecutor . I tried continuing the work you started here and used the tomcat executor instead of creating a custom thread pool executor. However, I had to register the executor so that it will be managed by the tomcat lifecycle.

is there a way to only configure the TaskQueue on the existing handler?

I don't believe this is possible, tomcat create an executor here if none is defined; it uses Integer.MAX

[mhalbritter]

Hey @ahmedhus, thanks for working on that! With your changes, I now have something which works. Unfortunately we missed the merge window for Boot 3.2, but I will merge it in 3.3 then.

Changes are here: https://github.com/mhalbritter/spring-boot/tree/mh/36087-add-property-to-configure-the-queue-size-for-tomcat