Taking a lot of time to load from Cog Titiler due to huge number of bands #958
-
|
Download of cog file is faster but taking a lot of time to visualise if there are huge number of bands in it. Is there any way to speed it up or use multi processing for this? Below are the details of the COG
|
Beta Was this translation helpful? Give feedback.
Replies: 1 comment 13 replies
-
Pretty hard to tell without knowing how you're trying to render your file 🤷♂️ |
Beta Was this translation helpful? Give feedback.
-
|
@vincentsarago here are the specific metrics:
|
Beta Was this translation helpful? Give feedback.
-
|
Hi @vincentsarago , Timings Information for dataset where we are experiencing slow rendering:###Get Zoom Levels:
$ tilebench get-zooms <s3url>
{"minzoom": 11, "maxzoom": 12}
###Tile profile:
$ tilebench profile <s3url>
2024-08-06T13:42:24.150651+0530 | TILEBENCH | reading tile: 11-1480-928
{"HEAD": {"count": 0}, "GET": {"count": 6, "bytes": 35646127, "ranges": ["0-16383", "16384-1064959", "1064960-11550719", "11550720-27934719", "27934720-35646126"]}, "Timing": 6.003262042999268}
###Random Tile at Zoom 15:
$ tilebench random <s3url> --zoom 15
15-23679-14861
###Profile Specific Tile at Zoom 15 with Configurations:
$ tilebench profile <s3url> --tile 15-7055-13040 --config GDAL_DISABLE_READDIR_ON_OPEN=EMPTY_DIR
{"HEAD": {"count": 0}, "GET": {"count": 5, "bytes": 35629743, "ranges": ["16384-1064959", "1064960-11550719", "11550720-27934719", "27934720-35646126"]}, "Timing": 6.925491809844971}
$ tilebench profile <s3url> --tile 15-7055-13040 --config GDAL_DISABLE_READDIR_ON_OPEN=FALSE
{"HEAD": {"count": 0}, "GET": {"count": 6, "bytes": 35646127, "ranges": ["0-16383", "16384-1064959", "1064960-11550719", "11550720-27934719", "27934720-35646126"]}, "Timing": 6.576103925704956}
Sentinel-2:###Get Zoom Levels:
$ tilebench get-zooms <s3url>
{"minzoom": 9, "maxzoom": 14}
###Tile profile:
$ tilebench profile <s3url>
2024-08-06T12:57:25.225692+0530 | TILEBENCH | reading tile: 10-219-406
{"HEAD": {"count": 0}, "GET": {"count": 4, "bytes": 376832, "ranges": ["0-16383", "180224-540671"]}, "Timing": 1.7954511642456055}
###Random Tile at Zoom 15:
$ tilebench random <s3url> --zoom 15
15-7055-13040
###Profile Specific Tile at Zoom 15 with Configurations:
$ tilebench profile <s3url> --tile 15-7055-13040 --config GDAL_DISABLE_READDIR_ON_OPEN=EMPTY_DIR
{"HEAD": {"count": 0}, "GET": {"count": 3, "bytes": 49152, "ranges": ["13975552-13991935", "14024704-14057471"]}, "Timing": 1.4548020362854004}
$ tilebench profile <s3url> --tile 15-7055-13040 --config GDAL_DISABLE_READDIR_ON_OPEN=FALSE
{"HEAD": {"count": 0}, "GET": {"count": 4, "bytes": 65536, "ranges": ["0-16383", "13975552-13991935", "14024704-14057471"]}, "Timing": 1.756836175918579}Could you review this and provide any insights or recommendations for optimization? Please let me know if anything else is required from my end. |
Beta Was this translation helpful? Give feedback.
-
|
well to me it seems that the connection to S3 is super slow 6 sec for 6 GET seems a bit weird to me |
Beta Was this translation helpful? Give feedback.
-
|
The COG created by GDAL is INTERLEAVED=PIXEL, hence fetching a subset of band indexes isn't supported. Hence, even if we query it, with a subset of the bands at high level and pass it to rasterio. (Rasterio needs to explicitly download an entire overview containing all bands, which covers the tile) and then extract the bands from it. Ref: Note: The COG specification itself is unopinionated about it as mentioned here. |
Beta Was this translation helpful? Give feedback.
-
|
This makes senses 🙏 When dealing with a high number of bands, having
|
Beta Was this translation helpful? Give feedback.
The COG created by GDAL is INTERLEAVED=PIXEL, hence fetching a subset of band indexes isn't supported. Hence, even if we query it, with a subset of the bands at high level and pass it to rasterio. (Rasterio needs to explicitly download an entire overview containing all bands, which covers the tile) and then extract the bands from it.
Ref:
Note: The COG specification itself is unopinionated about it as mentioned here.