RESTful API for visualisations

[lukavdplas]

This is something we discussed as part of #1430, but I think it also has advantages for general code quality.

Right now, visualisation requests to the backend are roughly made in two ways.

For fast visualisations:

• a POST request to an endpoint for that visualisation, with the visualisation parameters as payload. The response returns the results.

For slower visualisations (which may cause timeouts if handled in a single request):

• a POST request to an endpoint for that visualisation, with the visualisation parameters as payload. The backend returns the ID of a celery task
• repeated POST requests to /api/task_status with a list of celery task IDs in the payload. The response returns the status (done/working/error) of the task, and the results the task is done.
• a POST request to /api/abort_tasks with a list of celery task IDs in the payload. This will cancel those tasks in celery.

(The latter two request involve lists of tasks because a visualisation often involves multiple tasks for parallellisation.)

A most RESTful implementation would be:

• a POST request to create a new visualisation object.
• repeated GET requests to check if the status / results of the visualisation have been updated.
• a DELETE request to remove the visualisation and cancel any open tasks for it.

Potentially, you could also support a PATCH request to update the parameters of the visualisation. (The backend would then cancel open tasks and create a new task to fetch results.)

This would make sense if visualisation objects are also represented as objects in the database, so this API can interact with that. That sounds fine to me; ideally with a periodic job to remove visualisations older than, say, 24 hours.

Advantages:

• a more conventional and consistent API
• the current abort_task / task_status views are a minor security issue, as anyone is authorised to view or cancel any celery task. (This is mitigated by the fact that celery task IDs are long, random strings.) With this setup, we could actually check if someone is authorised to view the results of a visualisation.
• cleaner division of labour between backend and frontend; for instance, right now, the frontend needs to know something about the internal task structure of a visualisation
• possibilities for caching in the future?

[jgonggrijp]

Terminology nitpick: you are proposing to use more appropriate HTTP verbs. This has almost nothing to do with RESTfulness; see this slightly polemic blog post by the inventor of the words "REST" and "HATEOAS" about what it entails to work RESTful. We almost never do this at the RSL, although RDF data are intrinsically somewhat RESTful.

[lukavdplas]

I wouldn't claim to use this terminology accurately, so you're probably right 🙃

Anyway, perhaps explicating HTTP methods in my description is a bit distracting; I'm afraid it sounds like that was my main point, but I would make the same proposal if the current API was already using the methods I proposed.

Besides some of the other points mentioned, the goal is to create a more predictable relationship between what you provide as the payload of a request, the response you get, and what kind of "follow-up actions" would be accepted.

At the moment, you send data to one endpoint, then get an array of random strings as a response, and then send those to a different endpoint to figure out the results. Regardless of the HTTP methods used, this results in a somewhat idiosyncratic API that can only be navigated with a manual.

The alternative I propose is that you would send the data of a resource you want to create (a new visualisation object), the server returns the created resource with an identifier, and you can ask the same endpoint (with the ID added) to fetch the latest version of the resource.

I used the term RESTful because this change would incidentally mean that you shift from the way views were implemented in the Flask backend, where each method of each endpoint is implemented as an independent function with no standardisation or framework of any kind, to building a view the way django-rest-framework expects APIs work. But I may be misunderstanding what part of DRF is RESTful.

[jgonggrijp]

So "more in line with the canonical Django REST Framework way of doing things". Of course, this is a good idea in any case. I did not mean to suggest otherwise.

Despite the name, using DRF in the canonical way does not automatically make a RESTful API. Nowadays, everyone uses the words "REST API" when they just mean "JSON API", including the author of DRF. Real REST is rarely done because it requires more careful design on the server side and more intelligence on the client side.

All of this is just a terminology side remark though. I agree with what you are proposing.

[lukavdplas]

Ah, I see, thanks for explaining!