Skip to content
This repository was archived by the owner on Mar 21, 2024. It is now read-only.

Specify the new fetch documents route #234

Closed
Parent: Release v1.2.0
wants to merge 12 commits into from
Closed

Specify the new fetch documents route #234

Show file tree
Hide file tree
Changes from 4 commits
Commits
Show all changes
12 commits
Select commit Hold shift + click to select a range
174cafd
specify the new fetch documents route
irevoire May 4, 2023
6ef42de
update open-api
irevoire May 4, 2023
5e2c6d8
try to be nice with bumpsh
irevoire May 4, 2023
3141e26
specify the new analytics of the get/fetch documents route
irevoire May 10, 2023
52ef6c6
group the analytics behind already used groups
irevoire May 10, 2023
f50255b
fix a placeholder text
irevoire May 10, 2023
9af7b72
group the get and fetch route under a common section
irevoire May 10, 2023
8173e1a
rename the section to contain the name of both routes
irevoire May 16, 2023
826f693
clarify how to send the parameter to both routes
irevoire May 16, 2023
f03e4ff
fix the type of the parameters
irevoire May 16, 2023
af875c7
add the new fetch route to the api keys actions
irevoire May 16, 2023
c63508f
capitalize json
irevoire May 16, 2023
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
60 changes: 60 additions & 0 deletions open-api.yaml
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -1664,6 +1664,7 @@ paths:
- $ref: '#/components/parameters/limit'
- $ref: '#/components/parameters/offset'
- $ref: '#/components/parameters/fields'
- $ref: '#/components/parameters/filter'
post:
operationId: indexes.documents.create
summary: Add or replace documents
Expand Down Expand Up @@ -1805,6 +1806,65 @@ paths:
description: Not Found
parameters:
- $ref: '#/components/parameters/indexUid'
'/indexes/{indexUid}/documents/fetch':
post:
operationId: indexes.documents.fetch
summary: Get Documents
description: |
Get [documents](https://docs.meilisearch.com/learn/core_concepts/documents.html) by batch.
tags:
- Documents
security:
- apiKey: []
responses:
'200':
description: Ok
content:
application/json:
schema:
type: object
properties:
results:
type: array
items:
$ref: '#/components/schemas/document'
limit:
$ref: '#/components/schemas/limit'
offset:
$ref: '#/components/schemas/offset'
total:
$ref: '#/components/schemas/total'
required:
- results
- limit
- offset
- total
examples:
Example:
value:
results:
- id: 25684
title: American Ninja 5
poster: 'https://image.tmdb.org/t/p/w1280/iuAQVI4mvjI83wnirpD8GVNRVuY.jpg'
overview: 'When a scientists daughter is kidnapped, American Ninja, attempts to find her, but this time he teams up with a youngster he has trained in the ways of the ninja.'
release_date: 725846400
- id: 45881
title: The Bridge of San Luis Rey
poster: 'https://image.tmdb.org/t/p/w500/4X7quIcdkc24Cveg5XdpfRqxtYA.jpg'
overview: "The Bridge of San Luis Rey is American author Thornton Wilder's second novel, first published in 1927 to worldwide acclaim. It tells the story of several interrelated people who die in the collapse of an Inca rope-fiber suspension bridge in Peru, and the events that lead up to their being on the bridge.[ A friar who has witnessed the tragic accident then goes about inquiring into the lives of the victims, seeking some sort of cosmic answer to the question of why each had to die. The novel won the Pulitzer Prize in 1928."
release_date: 1072915200
limit: 20
offset: 0
total: 2
'401':
$ref: '#/components/responses/401'
'404':
description: Not Found
parameters:
- $ref: '#/components/parameters/limit'
- $ref: '#/components/parameters/offset'
- $ref: '#/components/parameters/fields'
- $ref: '#/components/parameters/filter'
'/indexes/{indexUid}/documents/delete-batch':
post:
operationId: indexes.documents.removeBatch
Expand Down
32 changes: 31 additions & 1 deletion text/0034-telemetry-policies.md
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -44,6 +44,8 @@ The collected data is sent to [Segment](https://segment.com/). Segment is a plat
| Documents Added | Aggregated event on all received requests via the `POST` - `/indexes/:indexUid/documents` route during one hour or until a batch size reaches `500Kb`. |
| Documents Updated | Aggregated event on all received requests via the `PUT` - `/indexes/:indexUid/documents` route during one hour or until a batch size reaches `500Kb`. |
| Documents Deleted | Aggregated event on all received requests via the `DELETE` - `/indexes/:indexUid/documents`, `DELETE` - `/indexes/:indexUid/documents/:documentId`, `POST` - `indexes/:indexUid/documents/delete-batch` routes during one hour or until a batch size reaches `500Kb`. |
| Documents Fetched GET | Aggregated event on all received requests via the `GET` - `/indexes/:indexUid/documents` or `GET` - `/indexes/:indexUid/documents/:doc_id` routes during one hour or until a batch size reaches `500Kb`. |
| Documents Fetched POST | Aggregated event on all received requests via the `POST` - `/indexes/:indexUid/documents/fetch` routes during one hour or until a batch size reaches `500Kb`. |
| Index Created | Occurs when an index is created via `POST` - `/indexes`. |
| Index Updated | Occurs when an index is updated via `PUT` - `/indexes/:indexUid`. |
| Indexes Swapped | Occurs when indexes are swapped via `POST` - `/swap-indexes`. |
Expand Down Expand Up @@ -179,7 +181,11 @@ The collected data is sent to [Segment](https://segment.com/). Segment is a plat
| indexes.total_single_index | The total number of calls where only one index where queried. | `2007` | `Documents Searched by Multi-Search POST`
| `swap_operation_number` | The number of swap operation given in `POST /swap-indexes` API call | 2 | `Indexes Swapped` |
| `matching_strategy.most_used_strategy` | Most used word matching strategy among all search requests in this batch | `last` | `Documents Searched POST`, `Documents Searched GET` |
| `per_document_id` | `true` if `DELETE /indexes/:indexUid/documents/:documentUid` endpoint was used in this batch, otherwise `false` | false | `Documents Deleted` |
| `per_document_id` | `true` if `DELETE /indexes/:indexUid/documents/:documentUid` or `GET /indexes/:indexUid/documents/:doc_id` endpoint was used in this batch, otherwise `false` | false | `Documents Deleted`, `Documents Fetched GET` |
| `per_filter` | `true` if `POST /indexes/:indexUid/documents/fetch` or `GET /indexes/:indexUid/documents/` endpoint was used with a filter in this batch, otherwise `false` | false | `Documents Fetched POST`, `Documents Fetched GET` |
| `total_received` | Total number of request received in this batch | 325 | `Documents Fetched GET`, `Documents Fetched POST` |
| `max_limit` | Highest value given for the `limit` parameter in this batch | 60 | `Documents Fetched POST`, `Documents Fetched GET` |
| `max_offset` | Highest value given for the `offset` parameter in this batch | 1000 | `Documents Fetched POST`, `Documents Fetched GET` |
| `clear_all` | `true` if `DELETE /indexes/:indexUid/documents` endpoint was used in this batch, otherwise `false` | false | `Documents Deleted` |
| `per_batch` | `true` if `POST /indexes/:indexUid/documents/delete-batch` endpoint was used in this batch, otherwise `false` | false | `Documents Deleted` |

Expand Down Expand Up @@ -395,6 +401,30 @@ This property allows us to gather essential information to better understand on
| clear_all | `true` if `DELETE /indexes/:indexUid/documents` endpoint is called in the aggregated event, otherwise `false` | `false` |
| per_batch | `true` if `POST /indexes/:indexUid/documents/delete-batch` endpoint is called in the aggregated event, otherwise `false` | `false` |

## `Documents Fetched GET`

> The Documents Fetched GET event is sent once an hour or when a batch reaches the maximum size of `500kb`.

| Property name | Description | Example |
|---------------|-------------|---------|
| `total_received` | Total number of request received in this batch | 325 | `Documents Fetched GET`, `Documents Fetched POST` |
| `per_document_id` | `true` if `GET /indexes/:indexUid/documents/:doc_id` endpoint was used in this batch, otherwise `false` | false |
| `per_filter` | `true` if `GET /indexes/:indexUid/documents` endpoint was used with a filter in this batch, otherwise `false` | false |
| `max_limit` | Highest value given for the `limit` parameter in this batch | 60 |
| `max_offset` | Highest value given for the `offset` parameter in this batch | 1000 |

## `Documents Fetched POST`

> The Documents Fetched POST event is sent once an hour or when a batch reaches the maximum size of `500kb`.

| Property name | Description | Example |
|---------------|-------------|---------|
| `total_received` | Total number of request received in this batch | 325 | `Documents Fetched GET`, `Documents Fetched POST` |
| `per_document_id` | `false` | false |
| `per_filter` | `true` if `POST /indexes/:indexUid/documents/fetch` endpoint was used with a filter in this batch, otherwise `false` | false |
| `max_limit` | Highest value given for the `limit` parameter in this batch | 60 |
| `max_offset` | Highest value given for the `offset` parameter in this batch | 1000 |

## `Settings Updated`

| Property name | Description | Example |
Expand Down
23 changes: 23 additions & 0 deletions text/0061-error-format-and-definitions.md
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -878,6 +878,29 @@ HTTP Code: `400 Bad Request`

---

## invalid_document_filter

`Synchronous`

### Context

This error occurs if a value with a different type than `String`, `Array of String` or `Array of array of String` for `filter` is specified.

### Error Definition

HTTP Code: `400 Bad Request`

```json
{
"message": "`:deserr_helper`",
"code": "invalid_document_filter",
"type": "invalid_request",
"link": "https://docs.meilisearch.com/errors#invalid_document_filter"
}
```

---

## invalid_document_limit

`Synchronous`
Expand Down
39 changes: 28 additions & 11 deletions text/0124-documents-api.md
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -20,21 +20,25 @@ Documents are stored inside indexes.

Manipulate documents of a Meilisearch index.

- [3.1.1. `GET` - `indexes/:index_uid/documents`](#311-get---indexesindexuiddocuments)
- [3.1.1. `GET` - `indexes/:index_uid/documents`](#311-get-indexesindexuiddocuments-and-post-indexesindexuiddocumentsfetch)
- [3.1.2. `GET` - `indexes/:index_uid/documents/:document_id`](#312-get---indexesindexuiddocumentsdocumentid)
- [3.1.3. `POST` - `indexes/:index_uid/documents`](#313-post---indexesindexuiddocuments)
- [3.1.4. `PUT` - `indexes/:index_uid/documents`](#314-put---indexesindexuiddocuments)
- [3.1.5. `DELETE` - `indexes/:index_uid/documents`](#315-delete---indexesindexuiddocuments)
- [3.1.6. `DELETE` - `indexes/:index_uid/documents/:document_id`](#316-delete---indexesindexuiddocumentsdocumentid)
- [3.1.7. `POST` - `indexes/:index_uid/documents/delete-batch`](#317-post---indexesindexuiddocumentsdelete-batch)

#### 3.1.1. `GET` - `indexes/:index_uid/documents`
#### 3.1.1. `GET` - `indexes/:index_uid/documents` and `POST` - `indexes/:index_uid/documents/fetch`
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

An unsolicited comment/feedback/question: is it wise to have two endpoints that do almost exactly the same thing, but in slightly different ways?

From a documentation perspective, it makes it harder for us to provide clear guidance on recommended practices and best use cases: "you can use A to do X, but you can also use B. B is slightly better for Y, but A works fine as well." This often results in users having to spend more time than they would want to on making a decision that might not be that important.

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Yeah, personally, I think we should deprecate the GET version of the route, same for the search.
To me, basically;

  • The get route can be slightly easier to use with curl
  • The post route is better on absolutely everything else (like sending big filters or using the array syntax)

Copy link
Member

@guimachiavelli guimachiavelli May 9, 2023 •
edited
Loading

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Deprecating GET would be fine for us. In case this is not possible (e.g. API stability and versioning), the second best solution would be to get clear confirmation from product and/or engine teams that we can recommend users to prefer POST and discourage GET usage.

IMHO, with this change GET becomes a net negative. Being slightly easier to use in a relatively small use-case is outweighed by the effort required in choosing between similar options.

Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

@guimachiavelli totally fine on our side to recommend users to use POST instead of GET. We can even go so far as to warn users that GET will be deprecated in the near future.

Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Brilliant, @macraig, thanks! We'll keep this in mind when documenting these changes and add you as a one of the reviewers to ensure our recommendations are aligned with the API design's intent 👍

Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

For what it's worth, we'd also be very happy if GET /search were to be deprecated.

Copy link
Member

@brunoocasali brunoocasali May 12, 2023 •
edited
Loading

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

So, just to be sure are we going to deprecate the GET index/{index_uid}/documents right away or are we waiting until the v1.3?

Cause, I can already mark those getDocuments occurrences as deprecated and point the users to the new fetchDocuments method.

Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

@brunoocasali We won't mark it as deprecated for v1.2, we are waiting for v1.3

Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Should we (integrations) follow that? Cause I don't see a world where waiting til v1.3 will make much difference for the users, a deprecation notice is not the same as "this method does not work anymore. Take this exception in your face instead".

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Once we deprecate it, it'll continue to work until the v2 of meilisearch, right?
The user will never get an exception, I think? (but maybe you could deprecate it in the integration and get rid of the method before meilisearch v2 🤔)


Meilisearch exposes 2 routes to get the documents:
- GET `indexes/:index_uid/documents`
- POST `indexes/:index_uid/documents/fetch`

List all documents of a Meilisearch index.

The query parameters `offset` and `limit` permit browsing through all documents of an index.

Meilisearch orders documents depending on the hash of their id.
Meilisearch orders documents depending on the order they were inserted in the db.

##### 3.1.1.1. Path Parameters

Expand All @@ -51,11 +55,12 @@ Unique identifier of an index.

##### 3.1.1.2. Query Parameters

| Field | Type | Required |
|--------------------------|--------------------------|----------|
| `offset` | Integer / `null` | false |
| `limit` | String / `null` | false |
| `fields` | String / `null` | false |
| Field | Type | Required |
|--------------------------|---------------------------|----------|
| `offset` | Integer / `null` | false |
| `limit` | String / `null` | false |
| `fields` | Array of Strings / `null` | false |
| `filter` | filter / `null` | false |

###### 3.1.1.2.1. `offset`

Expand All @@ -75,7 +80,7 @@ Sets the maximum number of documents to be returned by the current request.

###### 3.1.1.2.3. `fields`

- Type: String
- Type: Array of Strings
- Required: False
- Default: `*`

Expand All @@ -92,6 +97,17 @@ If `fields` is not specified, all attributes from the documents are returned in

> The index setting `displayedAttributes` has no impact on this endpoint.

###### 3.1.1.2.4. `filter`

- Type: String | Array of array of Strings
- Required: False
- Default: null

Refine the results by selecting documents that match the given filter.
In the case of the POST route, it is possible to send the filter in the form of an array of array of strings akin to the search route.

Attributes used as filter criteria must be added to the `filterableAttributes` list of an index settings. See [Filterable Attributes Setting API](0123-filterable-attributes-setting-api.md).

##### 3.1.1.3. Response Definition

A `results` array representing documents as JSON objects.
Expand Down Expand Up @@ -161,6 +177,7 @@ Gives the total number of documents that can be browsed in the related index.
- 🔴 Sending a value with a different type than `Integer` or `null` for `offset` will return a [invalid_document_offset](0061-error-format-and-definitions.md#invalid_document_offset) error.
- 🔴 Sending a value with a different type than `Integer` or `null` for `limit` will return a [invalid_document_limit](0061-error-format-and-definitions.md#invalid_document_limit) error.
- 🔴 Sending a value with a different type than `String` or `null` for `fields` will return a [invalid_document_fields](0061-error-format-and-definitions.md#invalid_document_fields) error.
- 🔴 Sending a value with a different type than `filter` or `null` for `filter` will return a [invalid_document_filter](0061-error-format-and-definitions.md#invalid_document_filter) error.

#### 3.1.2. `GET` - `indexes/:index_uid/documents/:document_id`

Expand Down Expand Up @@ -252,7 +269,7 @@ This endpoint accepts various content-type:

- [`application/json`](0135-indexing-json.md)
- [`text/csv`](0028-indexing-csv.md)
- [`application/x-ndjson`](0028-indexing-ndjson.md)
- [`application/x-ndjson`](0029-indexing-ndjson.md)

##### 3.1.3.1. Path Parameters

Expand Down Expand Up @@ -326,7 +343,7 @@ This endpoint accepts various content-type:

- [`application/json`](0135-indexing-json.md)
- [`text/csv`](0028-indexing-csv.md)
- [`application/x-ndjson`](0028-indexing-ndjson.md)
- [`application/x-ndjson`](0029-indexing-ndjson.md)

##### 3.1.4.1. Path Parameters

Expand Down