Route to import explored datasources automatically

[fm3]

New route to explore and immediately import remote datasets. POST /api/v5/datasets/exploreAndAddRemote with expected body {"remoteUri":"…", "datasetName": "…", "folderPath": "…"} (folderPath is optional, defaults to root)

TODO

• Call add route from wk to datastore after successful explore
• pass user token
• assert valid new name
• prettier error messages in failure case
• design prettier API
• specify target folder by path
  • create path if missing
  • get path object id, pass it to add route in datastore
  • there, forward it to reserveupload
• forbid forward slashes in folder names in sql constraint

Steps to test:

• Call the new route with a valid dataset that does not need credentials (or specify s3 credentials in env vars as introduced in Use s3 auth environment variables if set #7170

Issues:

• fixes Route for adding OME-Zarr datasets directly #7146
• fixes Set uploader when adding a remote dataset #7211

---

• Updated changelog

[fm3]

@philippotto @normanrz do you have opinions on the API?

Right now, the explore route takes a list of ExploreRemoteDatasetParameters (so that multiple layers can be explored into one datasource).

To prototype this, I just added shouldAutoAdd: Boolean and autoAddDatasetName: String to each element in this list, but use only the first.
What do you think would be a useful format for the app-to-app use case described in #7146?
I would suggest we wrap the list in an object, something like

{
  "layersToExplore": [
     {
       "remoteUri": "…",
       "credentialIdentifier": "…",
       "credentialSecret": "…",
     }
  ],
  "shouldAutoAdd": true
  "autoAddDatasetName": "myDataset",
}

[normanrz]

@philippotto @normanrz do you have opinions on the API?

I would like something simple like a GET route /api/datasets/importRemote?uri=s3://bucket/dataset.zarr.

[philippotto]

@philippotto @normanrz do you have opinions on the API?

I would like something simple like a GET route /api/datasets/importRemote?uri=s3://bucket/dataset.zarr.

Shouldn't GET be avoided when the route has side-effects?

[normanrz]

@philippotto @normanrz do you have opinions on the API?

I would like something simple like a GET route /api/datasets/importRemote?uri=s3://bucket/dataset.zarr.

Shouldn't GET be avoided when the route has side-effects?

Fair enough. Then, maybe a POST with

{ "remoteUri": "..." }

[fm3]

I made it to be POST /api/datasets/exploreAndAddRemote with expected body {"remoteUri":"…", "datasetName": "…"}

The passed datasetName must be available in the relevant webknossos organization.

Does this work for you? Or should we automatically generate dataset names to further simplify the api?

[normanrz]

Can you add an optional field for folder path? Ideally, the folders would be created if not existent.

[fm3]

So far folders have only ever been referenced by id. If we now start using path literals with folder names, I’d vote to forbid forward slashes in folder names. There are existing folders with forward slashes in their names, so we would have to introduce a migration for that as well. Do you agree?

[normanrz]

So far folders have only ever been referenced by id. If we now start using path literals with folder names, I’d vote to forbid forward slashes in folder names. There are existing folders with forward slashes in their names, so we would have to introduce a migration for that as well. Do you agree?

Yes, folder names should not have slashes in them. I think we should be as restrictive as we are with dataset and layer names.