Support data URIs in load_url

[soxofaan]

load_url currently only supports HTTP(S) URLs:

openeo-processes/proposals/load_url.json

Lines 12 to 19 in 965bbae

	"name": "url",
	"description": "The URL to read from. Authentication details such as API keys or tokens may need to be included in the URL.",
	"schema": {
	"title": "URL",
	"type": "string",
	"format": "uri",
	"subtype": "uri",
	"pattern": "^https?://"

For a use case we were brainstorming about avoiding the overhead of creating/managing external URLs (for a lot of small files) and came to the idea to load from data URLs where the data can be embedded in base64 inside the process graph, without need for external files/URLs. E.g.

  "lu": {
    "process_id": "load_url",
    "arguments": {
      "url": "data:application/vnd.apache.parquet;base64,UEFSMRUEFRAVFEwVAhUAEgAACBwqAAAAAAAAAB...",  

• additional work for a back-end wouldn't be that much
• would also be a handy feature in clients to allow embedding local data directly in the process graph

[soxofaan]

I can cook up a PR for this is there is more interest for this feature

[clausmichele]

Interesting! There will be the risk to create heavy process graphs probably, but the same happens with inline geoJSON anyway. Good to have another option and happy to try to support it.

[soxofaan]

Indeed, we already had issues with users embedding huge GeoJSON constructs in their process graph, so this would not create a new problem. As a matter of fact, the textual representation of GeoJSON makes it very space-inefficient and data URLs could improve the situation because of binary encoding and compression.

But still, it could be the responsibility of the clients to put reasonable thresholds on this and warn about or forbid excessive payloads

[m-mohr]

This needs more investigation:

• How is the size difference between native GeoParquet and base64 encoding.
  • Online references claim about 4/3 of the original size, VITO to investigate further: @jdries or @soxofaan
• Is there a size limit for data uris?
  • See MDN https://developer.mozilla.org/en-US/docs/Web/URI/Schemes/data for browsers
  • It should probably a low number in megabytes. Due to the larger size of base64, the original size limit would be smaller though.
• This shouldn't be a way to sidetrack the /file_formats. So only file formats would be allowed that are listed as input file formats in that endpoint. There's no way right now to map from file format to accepted media type(s) in openEO.
  • Would need an implementation of the open API issue: GET /file_formats: Expose supported data types and media types openeo-api#476
• As loading from a url and from a data URI are somewhat different, we may want to define it in a separate process such as load_data_uri.
• Is a data URI a valid URI according to the uri format in JSON Schema?
  • It's a valid URI: https://en.wikipedia.org/wiki/Data_URI_scheme
  • In principle it should be, but tooling support has to be investigated
• The content-type part of data URIs is optional, but we should probably require it?!
• The base64 encoding is also optional, but we should probably require it?!
• ...?

[soxofaan]

Another use case where data URIs could be part of the solution: users that want to work with sensitive/private (vector) data in non-geojson format that can not be hosted on a public URL.

[VictorVerhaert]

As geojson is limited to using lonlat, this feature would now allow users to pass geometries in a different crs to an openEO backend.

[m-mohr]

Any thoughts or updates on the questions raised in #498 (comment)?

[soxofaan]

How is the size difference between native GeoParquet and base64 encoding. Online references claim about 4/3 of the original size
Is there a size limit for data uris? ... It should probably a low number in megabytes.

Yes, that's true, base64 encoding has this "expanding" property, but I don't think this is a concern to worry about at the level of the openeo-processes. This feature is intended for "small data" use cases (e.g. a single or a couple of polygons in non-EPSG4326 projection). And when users might accidentally use oversize payloads: there is enough opportunity to raise warnings/errors client side.

Also note that GeoJSON itself is actually pretty poor in terms of size efficiency (e.g. a single float value in JSON easily consumes 15 or more characters), so a base64 encoded binary format might actually even be smaller than GeoJSON.

This shouldn't be a way to sidetrack the /file_formats ...

Indeed, it should be aligned with /file_formats, e.g. as raised in Open-EO/openeo-api#476

As loading from a url and from a data URI are somewhat different, we may want to define it in a separate process such as load_data_uri.

I don't see them as different enough to create a separate process . In the web dev world they seem to be used interchangeably (typically to embed images) without much trouble.

Is a data URI a valid URI according to the uri format in JSON Schema? ... In principle it should be, but tooling support has to be investigated

what kind of tooling are you worried about here?

The content-type part of data URIs is optional, but we should probably require it?! The base64 encoding is also optional, but we should probably require it?!

yes, I would also think it's best to require both