Debug OpenJPEG / JPEG2000 concurrent reading issues

[j08lue]

We observe that larger numbers of concurrent requests for tiles from JPEG2000 files cause an OpenJPEG failure.

For example, the eoAPI demo using dynamic tiling of a mosaic of 14 items in this demo example (search for "searches") consistently triggers the issue.

Log trail leading up to the failure:

│ 2024-10-08T14:38:42.839048805Z raster-eoapi-6ddc84f5f5-4fcrr ERROR 1: opj_get_decoded_tile() failed                                                                │
│ 2024-10-08T14:38:42.839946400Z raster-eoapi-6ddc84f5f5-4fcrr ERROR 1: Stream too short                                                                             │
│ 2024-10-08T14:38:42.840037452Z raster-eoapi-6ddc84f5f5-4fcrr                                                                                                       │
│ 2024-10-08T14:38:42.840051503Z raster-eoapi-6ddc84f5f5-4fcrr ERROR 1: opj_get_decoded_tile() failed                                                                │
│ 2024-10-08T14:38:42.935449486Z raster-eoapi-6ddc84f5f5-4fcrr ERROR 1: Stream too short                                                                             │
│ 2024-10-08T14:38:42.935517308Z raster-eoapi-6ddc84f5f5-4fcrr                                                                                                       │
│ 2024-10-08T14:38:42.935530488Z raster-eoapi-6ddc84f5f5-4fcrr ERROR 1: opj_get_decoded_tile() failed                                                                │
│ 2024-10-08T14:38:42.937411531Z raster-eoapi-6ddc84f5f5-4fcrr ERROR 1: Stream too short                                                                             │
│ 2024-10-08T14:38:42.937477333Z raster-eoapi-6ddc84f5f5-4fcrr                                                                                                       │
│ 2024-10-08T14:38:42.937487304Z raster-eoapi-6ddc84f5f5-4fcrr ERROR 1: opj_get_decoded_tile() failed                                                                │
│ 2024-10-08T14:38:43.837492212Z raster-eoapi-6ddc84f5f5-4fcrr ERROR 1: Stream too short                                                                             │
│ 2024-10-08T14:38:43.837985516Z raster-eoapi-6ddc84f5f5-4fcrr                                                                                                       │
│ 2024-10-08T14:38:43.838053628Z raster-eoapi-6ddc84f5f5-4fcrr ERROR 1: opj_get_decoded_tile() failed                                                                │
│ 2024-10-08T14:39:09.037868300Z raster-eoapi-6ddc84f5f5-4fcrr INFO:     10.42.4.3:44526 - "GET /raster/searches/4f9f8eb46bfc3ae0b769c5e1573a6134/tiles/WebMercatorQ │
│ uad/9/280/164?assets=B04_60m&assets=B03_60m&assets=B02_60m&color_formula=Gamma+RGB+3.2+Saturation+0.8+Sigmoidal+RGB+25+0.35&nodata=0 HTTP/1.1" 204 No Content      │
│ 2024-10-08T14:39:14.441989451Z raster-eoapi-6ddc84f5f5-4fcrr INFO:     Shutting down                                                                               │
│ 2024-10-08T14:39:14.935943105Z raster-eoapi-6ddc84f5f5-4fcrr INFO:     Waiting for connections to close. (CTRL+C to force quit)                                    │
│ 2024-10-08T14:39:21.837942208Z raster-eoapi-6ddc84f5f5-4fcrr INFO:     10.42.4.3:44450 - "GET /raster/searches/4f9f8eb46bfc3ae0b769c5e1573a6134/tiles/WebMercatorQ │
│ uad/9/280/162?assets=B04_60m&assets=B03_60m&assets=B02_60m&color_formula=Gamma+RGB+3.2+Saturation+0.8+Sigmoidal+RGB+25+0.35&nodata=0 HTTP/1.1" 204 No Content

💀

opj_get_decoded_tile is coming from the OpenJPEG lib https://www.openjpeg.org/doxygen/openjpeg_8h.html#aa67a413455bb7e2b53f4e4c2c1e604ef