Skip to content
This repository was archived by the owner on Mar 21, 2024. It is now read-only.

Dumps API - Make dump creation an asynchronous task #139

Merged
merged 11 commits into from
Jul 7, 2022
Merged

Dumps API - Make dump creation an asynchronous task #139

Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
11 commits
Select commit Hold shift + click to select a range
0d46319
wip
gmourier Apr 4, 2022
7323588
Make a dump creation a visible asynchronous task
gmourier Apr 12, 2022
7634dcd
Add precisions
gmourier Apr 12, 2022
928292c
Update open-api.yml
gmourier Apr 12, 2022
b3840ed
Add ommited type field for summarized task response
gmourier Apr 12, 2022
2638281
Add future possibilities
gmourier Apr 12, 2022
065dd84
Apply suggestions from code review
gmourier Apr 13, 2022
b7dc284
Precise that indexUid can be null
gmourier Apr 13, 2022
d7f35a5
Precise priorization of dumpCreation task over other task types
gmourier May 18, 2022
fa404b2
Keep taskUid for 202 response
gmourier May 18, 2022
c81dac0
remove dumps.get API key action
gmourier Jun 7, 2022
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
93 changes: 12 additions & 81 deletions open-api.yaml
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -341,6 +341,9 @@ components:
description: Value for the primaryKey field encountered if any for indexCreation or indexUpdate task.
settings:
$ref: '#/components/schemas/settings'
dumpUid:
type: string
description: Identifier generated for the dump for dumpCreation task.
error:
description: Error object containing error details and context when a task has a failed status.
$ref: '#/components/schemas/error'
Expand Down Expand Up @@ -541,37 +544,6 @@ components:
- numberOfDocuments
- isIndexing
- fieldDistribution
dumps:
type: object
additionalProperties: false
examples:
- uid: 20210719-114144097
status: done
startedAt: '2021-07-19T14:31:16.920473Z'
finishedAt: '2021-07-19T14:32:16.920473Z'
title: dumps
description: ''
properties:
uid:
type: string
example: 20200929-114144097
description: ' Unique identifier for the dump'
status:
type: string
enum:
- in_progress
- failed
- done
example: in_progress
description: Status of the dump
startedAt:
$ref: '#/components/schemas/timestamp'
finishedAt:
$ref: '#/components/schemas/timestamp'
required:
- uid
- status
- startedAt
error:
title: error
type: object
Expand Down Expand Up @@ -620,7 +592,6 @@ components:
- settings.update
- stats.get
- dumps.create
- dumps.get
- version
indexes:
type: array
Expand Down Expand Up @@ -887,13 +858,15 @@ components:
- uid
- indexUid
- status
- type
- enqueuedAt
examples:
Example:
value:
taskUid: 0
indexUid: movies
status: enqueued
type: documentAddition
enqueuedAt: '2021-07-19T14:31:16.920473Z'
'204':
description: No Content
Expand Down Expand Up @@ -996,59 +969,17 @@ paths:
content:
application/json:
schema:
$ref: '#/components/schemas/dumps'
$ref: '#/components/responses/202'
examples:
Created:
'202':
value:
uid: 20210719-114144097
status: in_progress
startedAt: '2021-07-19T14:31:16.920473Z'
'401':
$ref: '#/components/responses/401'
'/dumps/{dumpUid}/status':
get:
operationId: dumps.get
summary: Get dump status
description: |
Get the status of a dump creation process using the uid returned after calling the [dump creation route](https://docs.meilisearch.com/reference/api/dump.html#create-a-dump).

The returned status could be:

* `in_progress`: Dump creation is in progress.
* `failed`: An error occured during dump process, and the task was aborted.
* `done`: Dump creation is finished and was successful.
tags:
- Dumps
security:
- apiKey: []
responses:
'200':
description: Ok
content:
application/json:
schema:
$ref: '#/components/schemas/dumps'
taskUid: 0
indexUid: null
status: enqueued
type: dumpCreation
enqueuedAt: '2021-07-19T14:31:16.920473Z'
'401':
$ref: '#/components/responses/401'
'404':
description: Not Found
content:
application/json:
schema:
$ref: '#/components/schemas/error'
examples:
Not Found:
value:
message: Dump `20201001-110357260` not found
code: dump_not_found
type: invalid_request
link: 'https://docs.meilisearch.com/errors#dump_not_found'
parameters:
- schema:
type: string
name: dumpUid
in: path
required: true
/health:
get:
operationId: health.check
Expand Down
29 changes: 20 additions & 9 deletions text/0060-tasks-api.md
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -21,10 +21,10 @@ As writing is asynchronous for most of Meilisearch's operations, this API makes
| field | type | description |
|------------|---------|-------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------|
| uid | integer | Unique sequential identifier |
| indexUid | string | Unique index identifier |
| indexUid | string | Unique index identifier. This field is `null` when the task type is `dumpCreation`. |
| batchUid | integer | Identify in which batch a task has been grouped by auto-batching. It corresponds to the first task uid grouped within a batch. See [0096-auto-batching.md](0096-auto-batching.md) |
| status | string | Status of the task. Possible values are `enqueued`, `processing`, `succeeded`, `failed` |
| type | string | Type of the task. Possible values are `indexCreation`, `indexUpdate`, `indexDeletion`, `documentAdditionOrUpdate`, `documentDeletion`, `settingsUpdate` |
| type | string | Type of the task. Possible values are `indexCreation`, `indexUpdate`, `indexDeletion`, `documentAdditionOrUpdate`, `documentDeletion`, `settingsUpdate`, `dumpCreation` |
| details | object | Details information for a task payload. See Task Details part. |
| error | object | Error object containing error details and context when a task has a `failed` status. See [0061-error-format-and-definitions.md](0061-error-format-and-definitions.md) |
| duration | string | Total elapsed time the engine was in processing state expressed as an `ISO-8601` duration format. Times below the second can be expressed with the `.` notation, e.g., `PT0.5S` to express `500ms`. Default is set to `null`. |
Expand All @@ -36,12 +36,12 @@ As writing is asynchronous for most of Meilisearch's operations, this API makes

##### Summarized `task` Object for `202 Accepted`

| field | type | description |
|------------|---------|----------------------------------------------------------------------------------|
| taskUid | integer | Unique sequential identifier |
| indexUid | string | Unique index identifier |
| status | string | Status of the task. Value is `enqueued` |
| type | string | Type of the task. |
| field | type | description |
|------------|---------|---------------------------------|
| taskUid | integer | Unique sequential identifier |
| indexUid | string | Unique index identifier. This field is `null` when the task type is `dumpCreation`. |
| status | string | Status of the task. Value is `enqueued` |
| type | string | Type of the task. |
| enqueuedAt | string | Represent the date and time as `RFC 3339` format when the task has been enqueued |


Expand All @@ -68,6 +68,7 @@ As writing is asynchronous for most of Meilisearch's operations, this API makes
| documentAdditionOrUpdate |
| documentDeletion |
| settingsUpdate |
| dumpCreation |

> 👍 Type values follow a `camelCase` naming convention.

Expand Down Expand Up @@ -119,6 +120,16 @@ As writing is asynchronous for most of Meilisearch's operations, this API makes
| distinctAttribute | `distrinctAttribute` payload string |
| displayedAttributes | `displayedAttributes` payload array |

##### dumpCreation

| name | description |
| ----- | ------------ |
| dumpUid | The generated uid of the dump |

Since the creation of a dump is not a task associated with a particular index, it is only present on the `GET` - `/tasks` and `GET` - `tasks/:task_uid` endpoints.

Fully qualified and summarized task objects related to a dump creation display a `null` `indexUid` field.

#### 5. Examples

e.g. A fully qualified `task` object in an `enqueued` state.
Expand Down Expand Up @@ -702,4 +713,4 @@ n/a
- Use Hateoas capability to give direct access to a `task` resource.
- Add dedicated task type names modifying a sub-setting. e.g. `SearchableAttributesUpdate`.
- Add an archived state for old `tasks`.
- Indicate the `API Key` identity that added a `task`.
- Add the `API Key` identity that added a `task`.
23 changes: 0 additions & 23 deletions text/0061-error-format-and-definitions.md
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -849,29 +849,6 @@ HTTP Code: `404 Not Found`

---

## dump_already_processing

`Synchronous`

### Context

This error occurs when the user tries to launch the creation of a new dump while a creation is already being processed.

### Error Definition

Http Code: `409 Conflict`

```json
{
"message": "A dump is already processing. You must wait until the current process is finished before requesting another dump.",
"code": "dump_already_processing",
"type": "invalid_request",
"link": "https://docs.meilisearch.com/errors#dump_already_processing"
}
```

---

## invalid_content_type

`Synchronous`
Expand Down
1 change: 0 additions & 1 deletion text/0085-api-keys.md
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -169,7 +169,6 @@ Only the master key allows managing the API keys.
| settings.update | Provides access to `POST / DELETE` `/indexes/:authorizedIndexes/settings` and `/indexes/:authorizedIndexes/settings/*` routes. |
| stats.get | Provides access to `GET` `/stats/`. **⚠️Non-authorized `indexes` are omitted from the response on `/stats`**. Also add access to `GET` `/indexes/:authorizedIndexes/stats`. |
| dumps.create | Provides access to `POST` `/dumps` route. **As dumps are not scoped by indexes, a restriction on `indexes` does not affect this action.** |
| dumps.get | Provides access to `GET` `/dumps/:dumpUid` route. **As dumps are not scoped by indexes, a restriction on `indexes` does not affect this action.** |
| version | Provides access to `GET` `/version` route.

---
Expand Down
65 changes: 20 additions & 45 deletions text/0105-dumps-api.md
View file
Open in desktop
Original file line number Diff line number Diff line change
@@ -1,7 +1,4 @@
- Title: Dumps API
- Start Date: 2022-01-12

# Dumps
# Dumps API and CLI

## 1. Summary

Expand All @@ -16,19 +13,18 @@ The dumps exist to upgrade a MeiliSearch instance from a previous version to a m
### 3.1. Summary Key Points

- A dump creation can be scheduled from the MeiliSearch API using the `POST - /dumps` endpoint.
- A dump creation status can be tracked using the `GET - /dumps/{uid}/status` endpoint.
- A dump creation status can be tracked using the `GET - /tasks/:task_uid` endpoint.
- MeiliSearch can only create one dump at a time.
- By default, dumps are created in a folder named `dumps`, and can be found in the same directory as the MeiliSearch binary.
- The `dumps` directory can be customized using the `--dumps-dir` configuration option. If the dump directory does not already exist when the dump creation process is called, MeiliSearch will create it.
- If MeiliSearch is restarted after a dump creation, the dump's status will not appear on the `GET - /dumps/:uid/status` endpoint.
- A `.dump` file can be imported using the `--import-dump` command-line flag.
- The MeiliSearch server starts when the dump is fully imported and indexed.
- By default, importing a dump when a database already exists (a non-empty data.ms folder in the same directory as the MeiliSearch binary) will stop the process and throw an error.
- When using the command-line flag `--ignore-dump-if-db-exists=true`, MeiliSearch will use the existing database to start an instance instead of throwing an error. The dump will be ignored.
- By default, trying to import a dump that does not exist, will stop the process and throw an error.
- When using the command-line flag `--ignore-missing-dump`, MeiliSearch will continue its process and not throw an error.
- When a dump is being imported, the http API is not available. Meilisearch can't receive read or write requests.
- When a dump is being created, the task queue can receive other future operations to perform later but can't process any additional tasks during the dump creation.
- `dumpCreation` task takes priority over enqueued `tasks`. This means that if a `dumpCreation` task is created, it will be directly processed when the current processing task finishes even if other tasks have been enqueued before.

---

Expand All @@ -41,45 +37,18 @@ Create a dump
##### 3.2.1.1. Body Payload
N/A

##### 3.2.1.2. Response

`202 Accepted`

```json
{
"uid": "20220112-151751438",
"status": "in_progress",
"startedAt": "2022-01-12T15:17:51.438881Z"
}
```

##### 3.2.1.3. Errors

- 🔴 If Meilisearch is secured, accessing this route without the `Authorization` header returns a [missing_authorization_header](0061-error-format-and-definitions.md#missing_authorization_header) error.
- 🔴 If Meilisearch is secured, accessing this route with a key that does not have permissions (i.e. other than the master-key) returns an [invalid_api_key](0061-error-format-and-definitions.md#invalid_api_key) error.
- 🔴 Attempting to create a dump while a dump is already being created return an [dump_already_in_progress](0061-error-format-and-definitions.md#dump_already_in_progress) error.
##### 3.2.1.2. Response Definition

#### 3.2.2 GET `/dumps/:dump_uid/status`
When the request is successful, Meilisearch returns the HTTP code `202 Accepted`. The response's content is the summarized representation of the received asynchronous task.

Get a dump status
See [Summarized `task` Object for `202 Accepted`](0060-tasks-api.md#summarized-task-object-for-202-accepted).

##### 3.2.2.1. Response
The uid of the generated dump can be found in the task details.

`200 Success`

```json
{
"uid": "20220112-151751438",
"status": "in_progress",
"startedAt": "2022-01-12T15:17:51.438881Z"
}
```

##### 3.2.2.2. Errors
##### 3.2.1.3. Errors

- 🔴 If Meilisearch is secured, accessing this route without the `Authorization` header returns a [missing_authorization_header](0061-error-format-and-definitions.md#missing_authorization_header) error.
- 🔴 If Meilisearch is secured, accessing this route with a key that does not have permissions (i.e. other than the master-key) returns an [invalid_api_key](0061-error-format-and-definitions.md#invalid_api_key) error.
- 🔴 Attempting to access a dump details that does not exist returns a [dump_not_found](0061-error-format-and-definitions.md#dump_not_found) error.

---

Expand Down Expand Up @@ -131,18 +100,24 @@ The following table describes which version of the dump correspond to which vers
| `v4` | `v0.25.0` and later | - |

What does "Highest compatibility dump version" means?

For maintainance reasons, we cannot guarantee the compatibility from old dump versions to the newest ones.
Concretely, if the user wants to upgrade from Meilisearch `v0.19.0` (dump `v1`) to `v0.26.0` (dump `v4`), migration should be done in two steps. First, import your `v0.19.0` dump into an instance running any version of Meilisearch between v0.21 and v0.24. Second, export another dump from this instance and import it to a final instance running with `v0.26.0`.

## 2. Technical Aspects
## 4. Technical Aspects

### 2.1. Dump Creation
### 4.1. Dump Creation

When a dump is being created, the task queue can receive other future operations to perform later but can't process any additional tasks during the dump creation.
When a dump is being created, the task queue can receive other future operations to perform.

### 2.2. Importing a dump
### 4.2. Importing a dump

When a dump is being imported, the http API is not available. Meilisearch can't receive read or write requests.

## 3. Future Possibilities
- Make dump creation a `task`
### 4.3. Dump Creation Task Priority

Dump creation tasks have priority over other task types. If a `dumpCreation` task is enqueued, it will be directly processed when the current processing task finishes even if other tasks have been enqueued before.

## 5. Future Possibilities

- Give information about the import of a dump within the tasks.