-
Notifications
You must be signed in to change notification settings - Fork 111
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
Build GDAL with internal libtiff to better support COGs #1087
Comments
See PR: #1088 |
@palmerj Thanks for detailing everything. |
Thank you @fjperini |
@palmerj I will apply this change. we can close this issue |
FYI: Debian and Ubuntu packages use external libtiff. :-( |
Anyone know how exactly the internal build of libtiff differs from the external build of libtiff? |
Older versions of official libtiff APIs didn't allow for optimised reading of the tiff header (strile offset/count values from the [Strip|Tile][Offsets/ByteCounts] array.) The TIFFReadFromUserBuffer interface was added in 4.1 so that the user can directly provide the input buffer to decompress, and hence optimise the reading of the COG header. The GDAL internal version of libtiff also has this optimisation. |
Hmm, but TIFFReadFromUserBuffer is part of the public libtiff API?
So why does GDAL need the internal libtiff for this?
…--
bye,
pabs
https://bonedaddy.net/pabs3/
|
But it's a new API in v4.1 (I think) |
According to @Firefishy the external build of 4.1 doesn't work, only the internal one does.
…--
bye,
pabs
https://bonedaddy.net/pabs3/
|
So newer features work out of the box, such as `LERC` in `COG`. See https://gdal.org/drivers/raster/cog.html and similar PR at OSGeo/homebrew-osgeo4mac#1087
Can we please build GDAL with internal Libtiff support?
This will help any QGIS user adding COG raster data from cloud sources (such as AWS S3) or any GDAL scripts querying large COGs.
Background
Currently when using GDAL or projects that depend on GDAL such, reading large Cloud Optimised Geotiffs (COGs) is very slow for random access to pixel data. In some cases large COGs can have many 10s if not 100s of MBs of header data (mostly IFD array data) and If GDAL it is built with external libtiff then the whole Geotiff header needs to be downloaded before pixel data can be fetched. If GDAL is built using internal libtiff the arrays that make up most of the IFD header are not automatically downloaded - only the parts needed to specify a pixel request, which are usually a very small amount of the overall header.
Describe the solution you'd like
Change the GDAL configure script to use
--with-libtiff=internal
. See PR: #1088Describe alternatives you've considered
The libtiff project could be changed to support special compilation option, but this would require changing the existing public API of the library and would be a much harder task and would require some discussion with other libtiff stakeholders to see if this is possible. Even then this would take some time to flow through to an official release.
Additional context
The source code of the internal copy of libtiff in GDAL is actually identical
to libtiff upstream. But the internal copy is compiled with a special option
to do lazy loading of the arrays to support partially downloading the COG header.
The text was updated successfully, but these errors were encountered: