Client side hydration

[gadomski]

Items can contain a lot of duplicate information. pgstac has the concept of hydration where repeated information is removed from Items and stored in the item assets on a Collection. Now that item assets are "first class" as of STAC v1.1, I think it's time to explore "client side hydration", where items are returned from a STAC API "deydrated" and then hydrated by the client. We've pulled out the (Rust backed) https://github.com/developmentseed/hydraters/ from pgstac with this in mind.

As I see it, there's a couple of things we'd need to do to proof this out:

• Make a proof-of-concept STAC API extension so a server could advertise that it supports client-side hydration
• Update stac-fastapi-pgstac to ship dehydrated items and advertise it via our new extension ☝🏼
• Update pystac-client to use hydraters to do client-side hydration

[wildintellect]

Does a server need to advertise this? I had been thinking in the case of a server that already does it, there's no action (i.e. pgstac) but in the case of a server that doesn't do it, a client library should be able to take a hydrate=True and pull info from the collection record into items locally.

Or are you saying the user when making an API request should be able to control wether or not item records are hyrdated from collection record data.

PS: the use case we've been discussing internally is the Classification extension which can be very verbose for a lot of categories. https://planetarycomputer.microsoft.com/api/stac/v1/collections/usgs-lcmap-conus-v13/items/LCMAP_CU_032003_2021_V13_CCDC

[gadomski]

Or are you saying the user when making an API request should be able to control wether or not item records are hyrdated from collection record data.

I think the client and the server will have to coordinate, because not all clients will know how to hydrate items. So yes, I think it'd be something like

• Client sees client_side_hydration in the conformsTo on the server
• Client sends GET /search?client_side_hydration=true or some-such
• Server doesn't hydrate items, and sends back dehydrated items and a link to all collections that need to be used to re-hydrate client-side

There's some rough edges and design decisions to work out in there, as I type it (e.g. should we send back the item assets with each search page? probably not...)

[m-mohr]

Is the only advantage that the response payload is potentially smaller? It feels like a lot of added complexity for a debatable benefit IMHO.

This also reminds me of the Commons Extension in STAC back in STAC 0.x, which we abandoned at some point for probably good reasons (which I can't fully remember, but I guess it felt too complex for the added benefit).

[gadomski]

JSON bloat has been a pretty significant issue for data at rest in large scale STAC databases (e.g. Planetary Computer), which is where dehydration came from (citation needed, I'm trying to find a writeup). Rehydration could happen anywhere ... we've played with doing it in the DB and in the server. Since the data are being sent over the wire, to me it makes sense to try pushing it all the way back to the client.

[m-mohr]

Oh, just to clarify: I'm speaking primarily about the API parameter/client-side capabilities. I'm pretty certain that in the database hydration or normalization is a good thing ;-)
I'm just not sure it's worth the effort. Even if you are saving couple of megabytes in transfer (which I suspect is often less than a megabyte?), afterwards you need to still rehydrate so processing time and memory consumption doesn't necessarily go down and afterwards you are usually downloading gigabytes of data anyway... ;-)

[gadomski]

afterwards you are usually downloading gigabytes of data anyway... ;-)

Not in the visualization case, which is where I see this being the most useful. It does overlap a bit with how the fields API extension works, and I imagine in many cases fields could be used instead of dehydration ... 🤔

[jsignell]

Does the STAC 1.1 inheritance model not do away with the need for hydration at least conceptually?

[gadomski]

I think so, yes ... at least how hydration is currently implemented (for assets only, not for item properties).

So maybe we need to think about how to say that a client "supports inheritance", i.e. it looks up asset properties from the item's collection?

[jsignell]

Wouldn't it just be pystac's job to support inheritance? Or put another way: do you ever actually need the hydrated dict?

[gadomski]

You might, if you're saving the item to a place where you might not be able to get to the collection, or if you're moving the item to an new collection.

[jsignell]

ah ha! ok go it now.