QueuedThreadPool: Stopped without executing or closing null

[dzoech]

Jetty version(s)
9.4.44.v20210927

Java version/vendor (use: java -version)
Java 17
we use gcr.io/distroless/java17-debian11:nonroot

OS type/version
Debian 11

Description
We use Spring Boot 2.6.2 with Jetty.

In our logs the following log appears:

WARN [SpringApplicationShutdownHook] QueuedThreadPool: Stopped without executing or closing null

This seems to be the code that writes that warning log:
https://github.com/eclipse/jetty.project/blob/jetty-9.4.44.v20210927/jetty-util/src/main/java/org/eclipse/jetty/util/thread/QueuedThreadPool.java#L299

My understand is that either the BlockingQueue<Runnable> _jobs contains objects that are null, or there is a race condition so that !_jobs.isEmpty() is true but then _jobs.poll() returns null because it is now empty.

I'm not sure if this issue could be related to Spring so I'd like to mention that we implemented a shutdown hook waiting for an ExecutorService:

@Configuration
public class MyConfig {
    ....
    @Bean(destroyMethod = "onShutdown")
    MyWorker createMyWorker() {
        ExecutorService executorService = createExecutorService(); // returns a new ThreadPoolExecutor
        return new MyBean(executorService);
    }
}

public class MyWorker {
    ...
    public boolean onShutdown() throws InterruptedException {
        this.executorService.shutdown();
        boolean terminatedGracefully = executorService.awaitTermination(180, TimeUnit.SECONDS);
        return terminatedGracefully;
    }
}

How to reproduce?
Unfortunately, I cannot reproduce it.

[dzoech]

This issue is still occurring for us. Is there anything more I can do to help here?

[joakime]

Wonder how a null job even entered the queue.

We have handling for null on _jobs.poll() in other places in QTP, why not here?

[joakime]

Yeah, if you have a QueuedThreadPool, and you attempt to execute/offer a null job you'll get an exception.

QueuedThreadPool pool = new QueuedThreadPool();
pool.start();
pool.execute(null);

java.lang.NullPointerException
	at java.base/java.util.Objects.requireNonNull(Objects.java:221)
	at [email protected]/org.eclipse.jetty.util.BlockingArrayQueue.offer(BlockingArrayQueue.java:391)
	at [email protected]/org.eclipse.jetty.util.thread.QueuedThreadPool.execute(QueuedThreadPool.java:756)

[dzoech]

Would extending the if condition like this

else if (job != NOOP || job != null) {
    LOG.warn("Stopped without executing or closing {}", job);
}

be sufficient here? I can open a PR if you like.

[joakime]

@gregw & @sbordet WDYT?

[sbordet]

I think the loop should not test for isEmpty(), but just trying to poll() until it sees null -- a job can be stolen while stopping.

@dzoech care to submit a pull request?

[dzoech]

Sure, I'll follow up with a PR in the next couple of weeks.

[dzoech]

I now created the PR against 9.4 since we are using this version, which is predefined by org.springframework.boot:spring-boot-dependencies:2.7.4.

Is this the correct approach to get it merged?

[sbordet]

@dzoech please make the PR against branch jetty-10.0.x.

Jetty 9.4.x is at End of Community Support, see #7958.