Expose GDAL VRT connection string parameters in various image endpoints #960
Replies: 6 comments 4 replies
-
|
thanks for starting the discussion @TomAugspurger To be honest I'm not quite sure I see how and where to add those parameters, especially if this is to be used in titiler-pgstac. If it's to be used in a simple TilerFactory created endpoints (e.g. /cog) then I do think it's pretty simpler for user to create there own def DatasetPathParams(
url: Annotated[str, Query(description="Dataset URL")],
sds_name: Annotated[str, Query(description="The name of a subdataset in the source asset to load.")],
sds_bidx: Annotated[List[int], Query(description="The band numbers (between 1 and N) of the source asset to load."),
) -> str:
"""VRT URL from args""
...
app = FastAPI(title="My simple app")
cog = TilerFactory(path_dependency=DatasetPathParams)
app.include_router(cog.router) |
Beta Was this translation helpful? Give feedback.
-
|
I might be mistaken, but I think our TiTiler deployment doesn't expose As you note, getting the parameters through in titiler-pgstac is a challenge. For now, I'm pulling them directly off the vrt_params = {}
if subdataset_name := request.query_params.get("subdataset_name"):
vrt_params["sd_name"] = subdataset_name
if subdataset_bands := request.query_params.getlist("subdataset_bands"):
vrt_params["bands"] = ",".join([str(band) for band in subdataset_bands])
if vrt_params:
params = urllib.parse.urlencode(vrt_params, safe=",")
asset_url = f"vrt:///vsicurl/{asset_url}?{params}"If we go forward with this, I'll like to pass those through properly from the endpoint function to the caller of I'm defining my parameters like @dataclasses.dataclass
class SubdatasetParams(DefaultDependency):
"""Assets, Expression and Asset's band Indexes parameters."""
subdataset_name: Annotated[
Optional[str],
Query(
title="Subdataset name",
description="The name of a subdataset within the asset.",
),
] = None
subdataset_bands: Annotated[
Optional[List[int]],
Query(
title="Subdataset bands",
description="The name of a subdataset within the asset.",
),
] = Noneadding them to the routes like subdataset_params=Depends(self.subdataset_dependency),which makes them available on the request object. I think the main value here would be coordinating on the parameter names used, so that they're standardized across TiTiler deployments. Dunno whether that's worth it or not. |
Beta Was this translation helpful? Give feedback.
-
|
how do you pass the request object to the I don't think it's possible to pass anything to it because it's used in https://github.com/cogeotiff/rio-tiler/blob/f59532afd14abd84dd84d0d0f5dc20947a7181c3/rio_tiler/io/base.py#L455-L543 I understand there is a high level of abstraction between titiler/titiler-pgstac/rio-tiler which makes options passing super hard 😭 I may have an idea to enable this but it will require some change in rio-tiler. Let me open an issue there. Just to make it clear, you want an option to be able to |
Beta Was this translation helpful? Give feedback.
-
|
This might appear like pure magic, but here is how you can customize your titiler-pgstac application and add https://gist.github.com/vincentsarago/81f84f1ef349b9ac1a3a383c4424142f The idea is to create a custom low level
@attr.s
class CustomReader(Reader):
subdataset_name: Optional[str] = attr.ib(default=None)
subdataset_bands: Optional[List[int]] = attr.ib(default=None)
def __attrs_post_init__(self):
vrt_params = {}
if self.subdataset_name:
vrt_params["sd_name"] = self.subdataset_name
if self.subdataset_bands:
vrt_params["bands"] = ",".join([str(band) for band in self.subdataset_bands])
if vrt_params:
params = urllib.parse.urlencode(vrt_params, safe=",")
self.input = f"vrt:///vsicurl/{self.input}?{params}"
super().__attrs_post_init__()those 2 args are passed using the
|
Beta Was this translation helpful? Give feedback.
-
|
microsoft/planetary-computer-apis@main...user/tom/feature/grib-poc has how I'm testing stuff. Your way is certainly shorter :) With that, the requests to
FWIW, if you think these parameters a bit too application-specific to go into TiTiler or TiTiler-pgstac, we can certainly just do it in our application. |
Beta Was this translation helpful? Give feedback.
-
I think generalizing those options is not so simple! event if we add an option as a dependency, it's mostly certain that you'll still need a custom STACReader with a custom If we where adding default options to the TiTiler TileFactory def SdsParams(
url: Annotated[
str,
Query(
...,
description="Dataset url",
),
],
subdataset_name: Annotated[
Optional[str],
Query(
title="Subdataset name",
description="The name of a subdataset within the asset.",
),
] = None,
subdataset_bands: Annotated[
Optional[List[int]],
Query(
title="Subdataset bands",
description="The band index of a subdataset within the asset.",
),
] = None,
):
vrt_params = {}
if subdataset_name:
vrt_params["sd_name"] = subdataset_name
if subdataset_bands:
vrt_params["bands"] = ",".join([str(band) for band in subdataset_bands])
if vrt_params:
params = urllib.parse.urlencode(vrt_params, safe=",")
url = f"vrt:///vsicurl/{url}?{params}"
return urlwhich is pretty simple for users to write and add as their own dependencies. For PgSTAC Mosaic you either need to update the asset url within the # Custom ReaderParameter which is used to pass some options to the low-level `Reader`
# used in the PgSTAC Mosaic Backend
# PgSTACMosaicBackend -> STACReader -> Reader
@dataclass(init=False)
class ReaderParams(DefaultDependency):
"""reader parameters."""
reader_options: Dict = field(init=False)
def __init__(
self,
subdataset_name: Annotated[
Optional[str],
Query(
title="Subdataset name",
description="The name of a subdataset within the asset.",
),
] = None,
subdataset_bands: Annotated[
Optional[List[int]],
Query(
title="Subdataset bands",
description="The band index of a subdataset within the asset.",
),
] = None
):
"""Initialize ReaderParams """
params = {}
if subdataset_name:
params["subdataset_name"] = subdataset_name
if subdataset_bands:
params["subdataset_bands"] = subdataset_bands
self.reader_options = params
# Custom Low-Level reader which will `edit` the dataset url to use
# the VRT form if parameters are provided
# https://gdal.org/drivers/raster/vrt.html#vrt-connection-string
@attr.s
class CustomReader(Reader):
subdataset_name: Optional[str] = attr.ib(default=None)
subdataset_bands: Optional[List[int]] = attr.ib(default=None)
def __attrs_post_init__(self):
vrt_params = {}
if self.subdataset_name:
vrt_params["sd_name"] = self.subdataset_name
if self.subdataset_bands:
vrt_params["bands"] = ",".join([str(band) for band in self.subdataset_bands])
if vrt_params:
params = urllib.parse.urlencode(vrt_params, safe=",")
self.input = f"vrt:///vsicurl/{self.input}?{params}"
super().__attrs_post_init__()
# Custom PgSTACReader (rio_tiler.io.MultiBaseReader)
# which use the `CustomReader` as low level reader
# This Reader will be used for the /collections/{collection_id}/items/{item_id} endpoints
@attr.s
class PgSTACReader(reader.PgSTACReader):
reader: Type[BaseReader] = attr.ib(default=CustomReader)
# CustomSTACReader (based on titiler.pgstac.reader.CustomSTACReader)
# which use the `CustomReader` as low level reader
# This Reader will be used for the /collections/{collection_id} or /searches/{search_id} endpoints
@attr.s
class CustomSTACReader(mosaic.CustomSTACReader):
reader: Type[BaseReader] = attr.ib(default=CustomReader)
# Custom Titiler PgSTAC mosaic backend
# which use the `CustomSTACReader` as low level reader
@attr.s
class PGSTACBackend(mosaic.PGSTACBackend):
reader: Type[BaseReader] = attr.ib(init=False, default=CustomSTACReader) |
Beta Was this translation helpful? Give feedback.
-
|
Is there precedent for using an existing parameter like |
Beta Was this translation helpful? Give feedback.
-
|
FYI over https://gist.github.com/vincentsarago/43b2bf643f7f878761004c8145a5f959 I've explored another way TL&DR, the custom STACReader in the gist, accepts assets to be passed in form of note: using this in titiler, means that you'll need to encode the asset string (because of the ://, ? and & strings) |
Beta Was this translation helpful? Give feedback.
-
|
Thanks Vincent, let me make sure I understood what the gist is doing:
Did I understand that right? I like this solution because it just exposes the VRT URI. If GDAL adds more options in the future, titiler doesn't have to change. I think this approach would also work for NetCDF and HDF5, if it's not too onerous can you try it out with one of our NetCDF datasets from Planetary Computer? |
Beta Was this translation helpful? Give feedback.
-
yes, in addition to the code shared in the gist, I would need to account that maybe some asset could be named
depends what is the asset
Yes
Yes
absolutely 👍
sure |
Beta Was this translation helpful? Give feedback.
-
I'd like to visualize a band out of a large GRIB2 file, or perhaps a subdatset out of a NetCDF file. I can do this efficiently if I can build a GDAL VRT connection string to hand off to rasterio:
gdalinfo "vrt:///vsicurl/https://noaahrrr.blob.core.windows.net/hrrr/hrrr.20240620/conus/hrrr.t00z.wrfsfcf00.grib2?bands=123"(without the
?bands=123then rasterio / GDAL needs to read all the messages, which is slow)I'm using titiler-pgstac, and can override
PgSTACReader._get_asset_infoto build an AssetInfo with a URL that's actually a VRT connection string. From there things just work!I'd like to propose two new parameters to TiTiler's image endpoints, and am curious if it makes sense to "standardize" these in TiTiler itself.
For reasons that aren't entirely clear to me, the various "subdatasets" inside of the HRRR grib files are considered bands of a single dataset, while for a NetCDF file like this GOES asset, they're considered subdatasets:
Perhaps if you had a 3-dimensional array in the NetCDF file then you'd end up with both subdatasets and bands?
A couple notes:
subdataset_bandsisn't 100% technically correct. This is thebandsparameter passed to the VRT query string.vrt_bandswould be a bit more accurate, but I prefer thesubdatasetprefix. If we prefervrt_bandsthen I'd recommend usingvrt_subdatasetorvrt_subdataset_namefor the other parameter.sdparameter for selecting a subdataset by position. I've chosen not to add it since the docs noteBeta Was this translation helpful? Give feedback.
All reactions