FlatGeoBuf driver loads the whole feature offset index / inefficient for network access

[rouault]

CC @bjornharrtell
See https://lists.osgeo.org/pipermail/gdal-dev/2019-October/050946.html

[mloskot]

Seems like this #1964 added recently by @bjornharrtell is related

[rouault]

Seems like this #1964 added recently by @bjornharrtell is related

Not really :-) #1964 is about writing side, this one about reading side

[bjornharrtell]

Ah yes, I've been semi-aware of this shortcoming.

[bjornharrtell]

I want to take a stab at this but unsure about the best strategy. I would try to do an adaptive block cache for the I/O here but unsure if I should simply try to implement that directly in the driver or if I should attempt to implement at a higher abstraction level. Do you have any guidance @rouault?

[rouault]

I would just suggest implementing the most simple strategy, that is issuing a VSIFRead() to get the offset of just the feature you need at the time where you need it. For local I/O, the operating system will cache things. And for /vsicurl/ and related, they will also have some caching.

[bjornharrtell]

Mmm yes, will be very simple and probably work well for most cases. 👍

[bjornharrtell]

@rouault it seems accessing via vsizip is performing very badly on large files (uncompressed >1GB) and I suspect may not have been so bad before this PR. I've looked through cpl_vsil_gzip but fail to find any limit to the snapshot cache which would explain it... will investigate further when time permits but mabye you have an idea what I should look for? (accessing comparable shapefile data via vsizip performs well)

[rouault]

ah, it might be that the lots of seeks that you need to do might hit hard the snapshot mechanism. but random access in a zip file is going to suck anyway...

[bjornharrtell]

Well, a spat filter query via vsizip shapefile actually performs rather well, even over vsicurl. (!)

[bjornharrtell]

My theory is that because shp is separated into several files you get several caches and less seeking per file. But the cache could/should be smarter than that. Will try to understand it and improve when I can. :)

[bjornharrtell]

But I slowly realize that deflating a single stream requires sequential read unless some kind of extra/custom indexing is done like fx. https://github.com/pauldmccarthy/indexed_gzip#overview.

[bjornharrtell]

Ok some more thought on this and I now believe that simply reading out the feature offset indexes in advance (before getting any feature data) and sorting it (to avoid backwards seeks when reading out features) will be better for both vsicurl and vsizip and possibly compete with the performance I see in shp.

[bjornharrtell]

Looks like there are some compression methods out there that do split the steam into indexed blocks for random access. Most prominent I've found is https://lz4.github.io/lz4/ which is also very fast. Would be fun to eventually support that all the way.

[rouault]

We already have the ZSTD dependency, which has excellent compression_ratio / speed

[bjornharrtell]

Ah yes zstd is quite nice too even if lz4 wins by far in decompression speed. But I fail to find any info about if zstd has out of the box stream splitting which is what I found interesting about lz4 (however I do not know how mature it is). Looks like this is a work in progress for zstd, see facebook/zstd#395.

[bjornharrtell]

And somewhat related project https://blosc.org/ is also interesting.