java.lang.NullPointerException: null error on WebDAV Domain due to jetty 9.4.51 bug

[geonmo]

Dear dCache staff,

I found some errors on a WebDAV domain.

2024-08-04T00:36:58.908509+09:00 cms-t2-se01.sdfarm.kr dcache@WebDavDomain[488477]: java.lang.NullPointerException: null
2024-08-04T00:36:58.908509+09:00 cms-t2-se01.sdfarm.kr dcache@WebDavDomain[488477]: #011at org.eclipse.jetty.io.AbstractConnection.lambda$failedCallback$0(AbstractConnection.java:92)
2024-08-04T00:36:58.908509+09:00 cms-t2-se01.sdfarm.kr dcache@WebDavDomain[488477]: #011at org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:883)
2024-08-04T00:36:58.908509+09:00 cms-t2-se01.sdfarm.kr dcache@WebDavDomain[488477]: #011at org.eclipse.jetty.util.thread.QueuedThreadPool$Runner.run(QueuedThreadPool.java:1034)
2024-08-04T00:36:58.908509+09:00 cms-t2-se01.sdfarm.kr dcache@WebDavDomain[488477]: #011at java.base/java.lang.Thread.run(Thread.java:829)

I don't know if it's due to this bug, but if the number of failed transfers at the WebDAV door increases, at some point the WebDAV domain will stop working and even basic functions like serving SSL certificates won't work.

I googled the error and found that it occurred in jetty 9.4.51 and was fixed in 9.4.52. (jetty/jetty.project#9476)

However, I understand that dCache v9.2 uses jetty 9.4.51.

I strongly request that the jetty version upgrade in the next release of dCache v9.2.

Please also check whether this bug can cause any abnormal behavior of a WebDAV domain.

The webdav hang error has significantly reduced the availability of our site.

Regards,

-- Geonmo

[kofemann]

Hi @geonmo ,

Thanks for reporting. Do you know how to reproduce the issue?

[geonmo]

Hello, @kofemann ,

The code to test jetty itself is in issue #9476. (jetty/jetty.project#9476)

In my case, I recently experienced a data upload issue slowdown that brought the issue.

Currently on our site, roughly 450 uploads are consistently failing with timeouts, and the webdav domain is down every 8 hours, as short as 1 hour and as long as 8 hours. I'm assuming that not that many are failing because the timeout time (7200) is long.

I think it might be possible to make the timeout of the webdav transfer task smaller and then test it to keep failing.

[geonmo]

I've seen the release with the jetty version change, thank you for taking care of it. I'll close this issue.