diff --git a/.github/workflows/bump.yml b/.github/workflows/bump.yml index dd6bf1bf..32fe86e0 100644 --- a/.github/workflows/bump.yml +++ b/.github/workflows/bump.yml @@ -4,18 +4,18 @@ on: pull_request: branches: - main + - develop jobs: api-diff: - name: Check diff on Bump + if: ${{ github.event_name == 'pull_request' }} + name: Check API diff on Bump runs-on: ubuntu-latest steps: - - name: Checkout - uses: actions/checkout@v2 - - name: Comment pull request with API diff - uses: bump-sh/github-action@v1 + - uses: actions/checkout@v2 + - uses: bump-sh/github-action@v1 with: - doc: 0772e3a7-2923-4d62-95e3-c4339c525968 + doc: meilisearch token: ${{secrets.BUMP_TOKEN}} file: open-api.yaml command: diff diff --git a/open-api.yaml b/open-api.yaml index 4415eb94..3e9086a3 100644 --- a/open-api.yaml +++ b/open-api.yaml @@ -124,6 +124,7 @@ components: length: 5 - start: 155 length: 5 + description: '' properties: _formatted: type: object @@ -140,7 +141,9 @@ components: - string - number description: Retrieve attributes of the document. `attributesToRetrieve` controls these fields. - description: '' + _geoDistance: + type: number + description: 'Using _geoPoint({lat}, {lng}) built-in sort rule at search leads the engine to return a _geoDistance within the search results. This field represents the distance in meters of the document from the specified _geoPoint.' documentId: oneOf: - type: number @@ -156,6 +159,9 @@ components: - String: `"something > 1 AND genres=comedy AND (genres=horror OR title=batman)"` - Mixed: `["something > 1 AND genres=comedy", "genres=horror OR title=batman"]` + > info + > _geoRadius({lat}, {lng}, {distance_in_meters}) built-in filter rule can be used to filter documents within a geo circle. + > warn > Attribute(s) used in `filter` should be declared as filterable attributes. See [Filtering and Faceted Search](https://docs.meilisearch.com/reference/features/filtering_and_faceted_search.html). example: @@ -361,7 +367,7 @@ components: - attribute - sort - exactness - - release_date:asc + - 'release_date:asc' examples: [] filterableAttributes: type: array @@ -617,6 +623,8 @@ components: > warn > Attribute(s) used in `sort` should be declared as sortable attributes. See [Sorting](https://docs.meilisearch.com/reference/features/sorting.html). + > info + > _geoPoint({lat}, {long}) built-in sort rule can be used to sort documents around a geo point. filter: name: filter in: query @@ -632,8 +640,20 @@ components: - String: `something > 1 AND genres=comedy AND (genres=horror OR title=batman)` - Mixed: `["something > 1 AND genres=comedy", "genres=horror OR title=batman"]` + > info + > _geoRadius({lat}, {lng}, {distance_in_meters}) built-in filter rule can be used to filter documents within a geo circle. + > warn > Attribute(s) used in `filter` should be declared as filterable attributes. See [Filtering and Faceted Search](https://docs.meilisearch.com/reference/features/filtering_and_faceted_search.html). + Content-Type: + name: Content-Type + in: header + required: true + schema: + type: string + enum: + - application/json + description: Payload content-type responses: '202': description: Accepted @@ -685,6 +705,10 @@ components: type: apiKey in: header name: X-Meili-API-Key + description: |- + An API key is a token that you provide when making API calls. Include the token in a header parameter called `X-Meili-API-Key`. + + Example: `X-Meili-API-Key: 8fece4405662dd830e4cb265e7e047aab2e79672a760a12712d2a263c9003509` examples: {} tags: - name: Indexes @@ -912,6 +936,8 @@ paths: errorLink: 'https://docs.meilisearch.com/errors#invalid_index_uid' '401': $ref: '#/components/responses/401' + parameters: + - $ref: '#/components/parameters/Content-Type' '/indexes/{indexUid}': get: operationId: indexes.get @@ -984,6 +1010,8 @@ paths: $ref: '#/components/responses/401' '404': description: Not Found + parameters: + - $ref: '#/components/parameters/Content-Type' delete: operationId: indexes.remove summary: Delete Index @@ -1048,6 +1076,9 @@ paths: > info > If the provided index does not exist, it will be created. + + > info + > Use the reserved `_geo` object to add geo coordinates to a document. `_geo` is an object made of `lat` and `lng` field. tags: - Documents security: @@ -1059,6 +1090,7 @@ paths: schema: type: array items: null + examples: {} responses: '202': $ref: '#/components/responses/202' @@ -1066,10 +1098,21 @@ paths: $ref: '#/components/responses/401' '413': $ref: '#/components/responses/413' + parameters: + - schema: + type: string + enum: + - application/json + - text/csv + - application/x-ndjson + in: header + name: Content-Type + required: true + description: The content-type associated with the format to be indexed put: operationId: indexes.documents.upsert summary: Add or update documents - description: | + description: |- Add a list of documents or update them if they already exist. If you send an already existing document (same [id](https://docs.meilisearch.com/learn/core_concepts/documents.html#primary-key)) the old document will be only partially updated according to the fields of the new document. Thus, any fields not present in the new document are kept and remained unchanged. @@ -1078,6 +1121,9 @@ paths: > info > If the provided index does not exist, it will be created. + + > info + > Use the reserved `_geo` object to add geo coordinates to a document. `_geo` is an object made of `lat` and `lng` field. tags: - Documents security: @@ -1106,6 +1152,17 @@ paths: $ref: '#/components/responses/401' '413': $ref: '#/components/responses/413' + parameters: + - schema: + type: string + enum: + - application/json + - text/csv + - application/x-ndjson + in: header + name: Content-Type + description: The content-type associated with the format to be indexed + required: true delete: operationId: indexes.documents.removeAll summary: Delete all documents @@ -1157,6 +1214,8 @@ paths: $ref: '#/components/responses/401' '404': description: Not Found + parameters: + - $ref: '#/components/parameters/Content-Type' parameters: - $ref: '#/components/parameters/indexUid' '/indexes/{indexUid}/documents/{documentId}': @@ -1389,6 +1448,8 @@ paths: $ref: '#/components/responses/401' '404': description: Not Found + parameters: + - $ref: '#/components/parameters/Content-Type' parameters: - $ref: '#/components/parameters/indexUid' '/indexes/{indexUid}/settings': @@ -1446,7 +1507,7 @@ paths: - attribute - sort - exactness - - release_date:asc + - 'release_date:asc' distinctAttribute: ean13 searchableAttributes: - title @@ -1543,7 +1604,7 @@ paths: - attribute - sort - exactness - - release_date:asc + - 'release_date:asc' distinctAttribute: ean13 searchableAttributes: - title @@ -1563,6 +1624,8 @@ paths: $ref: '#/components/responses/401' '404': description: Not Found + parameters: + - $ref: '#/components/parameters/Content-Type' delete: operationId: indexes.settings.reset summary: Reset settings @@ -1631,6 +1694,8 @@ paths: $ref: '#/components/responses/401' '404': description: Not Found + parameters: + - $ref: '#/components/parameters/Content-Type' delete: operationId: indexes.settings.synonyms.reset summary: Reset synonyms @@ -1675,6 +1740,9 @@ paths: summary: Update sortable attributes description: | Update the list of [sortableAttributes](https://docs.meilisearch.com//reference/features/settings.html#sortable-attributes) of an index. + + > info + > In order to enable sorting capabilities on geographic data, the `_geo` field must be added as a sortableAttribute. tags: - Settings security: @@ -1694,6 +1762,8 @@ paths: $ref: '#/components/responses/401' '404': description: Not Found + parameters: + - $ref: '#/components/parameters/Content-Type' delete: operationId: indexes.settings.sortable-attributes.reset summary: Reset sortable attributes @@ -1767,6 +1837,8 @@ paths: $ref: '#/components/responses/401' '404': description: Not Found + parameters: + - $ref: '#/components/parameters/Content-Type' delete: operationId: indexes.settings.stopWords.reset summary: Reset stop-words @@ -1836,6 +1908,8 @@ paths: $ref: '#/components/responses/401' '404': description: Not Found + parameters: + - $ref: '#/components/parameters/Content-Type' delete: operationId: indexes.settings.rankingRules.reset summary: Reset ranking rules @@ -1888,6 +1962,9 @@ paths: description: | Update the [filterable attributes](https://docs.meilisearch.com/reference/features/settings.html#filterable-attributes) of an index. + > info + > In order to enable filtering capabilities on geographic data, the `_geo` field must be added as a filterableAttribute. + > info > If the provided index does not exist, it will be created. tags: @@ -1907,6 +1984,8 @@ paths: schema: $ref: '#/components/schemas/filterableAttributes' description: '' + parameters: + - $ref: '#/components/parameters/Content-Type' delete: operationId: indexes.settings.filterableAttributes.reset summary: Reset Filterable Attributes @@ -1972,6 +2051,8 @@ paths: $ref: '#/components/responses/401' '404': description: Not Found + parameters: + - $ref: '#/components/parameters/Content-Type' delete: operationId: indexes.settings.distinctAttribute.reset summary: Reset distinct attribute @@ -2036,6 +2117,8 @@ paths: $ref: '#/components/responses/401' '404': description: Not Found + parameters: + - $ref: '#/components/parameters/Content-Type' delete: operationId: indexes.settings.searchableAttributes.reset summary: Reset searchable attributes @@ -2100,6 +2183,8 @@ paths: $ref: '#/components/responses/401' '404': description: Not Found + parameters: + - $ref: '#/components/parameters/Content-Type' delete: operationId: indexes.settings.displayedAttributes.reset summary: Reset displayed attributes @@ -2336,7 +2421,7 @@ paths: value: commitSha: b46889b5f0f2f8b91438a08a358ba8f05fc09fc1 commitDate: '2021-07-08' - pkgVersion: 0.22.0 + pkgVersion: 0.23.0 '401': $ref: '#/components/responses/401' security: [] diff --git a/text/0028-indexing-csv.md b/text/0028-indexing-csv.md new file mode 100644 index 00000000..0e975381 --- /dev/null +++ b/text/0028-indexing-csv.md @@ -0,0 +1,243 @@ +- Title: Indexing CSV +- Start Date: 2021-04-9 +- Specification PR: [PR-#28](https://github.com/meilisearch/specifications/pull/28) +- Discovery Issue: n/a + +# Indexing CSV + +## 1. Functional Specification + +### I. Summary + +To index documents, the body of the add documents request has to match a specific format. That specific format is then parsed and tokenized inside MeiliSearch. After which, the documents added are in the pool of searchable and returnable documents. + +A [CSV](https://en.wikipedia.org/wiki/Comma-separated_values) data format is broadly used to store and exchange data in a simple format. + +Also, in order to boost write performance CSV data format is more suited than JSON for consequent datasets, as keys are not duplicated for every document. + +#### Summary Key Points + +- The header of the csv payload allows to name the attributes and type them. +- `text/csv` Content-Type header is now supported. +- The error cases have been strengthened and completed. See Errors part. + +### II. Motivation + +We want to provide our users with an always improved usage experience. Currently, the engine only accepts JSON format as a data source. We want to give users the possibility of another simple data format, well known, to use. Thus, give them more versatility at the data source choices for the indexing (add and update) step. + +Since most SQL engines or SQL clients can easily dump data as CSV, it will facilitate MeiliSearch adoption by extending the indexing step on a wider range of customer cases than before. + +Writing performance is also considered as a motivation since CSV parsing is less CPU and memory intensive than parsing Json due to the streamable capability of the CSV format. + +### III.Explanation + +#### CSV Formatting Rules + +While there's [RFC 4180](https://tools.ietf.org/html/rfc4180) as a try to add a specification for CSV format, we will find a lot of variations from that. MeiliSearch features capabilities requires CSV data to be formatted the proper way to be parsable by the engine. + +- CSV data format needs to contain a first line representing the list of attributes with the optionally chosen type separated from the attribute name by `:` character. The type is case insensitive. + +> An attribute can be specificed with two types: `string` or `number`. A missing type will be interpreted as a `string` by default. +> +> Valid headline example: "id:number","title:string","author","price:number" + +- The following CSV lines will represent a document for MeiliSearch. +- A `,` character must separate each cell. +- A CSV value should be enclosed in double-quotes when it contains a comma character or a newline to escape it. +- Using double-quotes to enclose fields, then a double-quote appearing inside a field must be escaped by preceding it with another double quote as mentioned in [RFC 4180](https://tools.ietf.org/html/rfc4180). +- Float value should be written with a `.` character, like `3.14`. +- CSV text should be encoded in UTF8. +- The format can't handle array cell values. We are providing `nd-json` format to deal with theses types of attribute in a easier way. + +##### Example with a comma inside a cell + +Given the CSV payload +``` +"id:number","label","price:number","colors","description" +"1","t-shirt","4.99","red","Thus, you will rock at summer time." +``` +the search result should be displayed as +```json +{ + "hits": [ + { + "id": 1, + "label": "t-shirt", + "price": 4.99, + "colors": "red", + "description": "Hey, you will rock at summer time." + } + ], + ... +} +``` + +##### Example with a double quote inside a cell + +Given the CSV payload +``` +"id:number","label","price","colors","description" +"1","t-shirt","4.99","red","Hey, you will ""rock"" at summer time." +``` +the search result should be displayed as +```json +{ + "hits": [ + { + "id": 1, + "label": "t-shirt", + "price": "4.99", + "colors": "red", + "description": "Hey, you will rock at summer time.", + } + ], + ... +} +``` + +> Note that the price attribute was not typed as a number. By default, MeiliSearch type it as a string. + +#### API Endpoints + +> Each API endpoints mentioned above will now require a `text/csv` as `Content-Type` header to be processed as CSV data. + +**As a developer, I want to upload a CSV payload of documents so that end-user can search them** + +**POST documents** `/indexes/:indexUid/documents` + +```bash +curl \ + -X POST 'http://localhost:7700/indexes/movies/documents' \ + -H 'Content-Type: text/csv' \ + --data--binary ' + "id","label","price:number","colors","description"\n + "1","hoodie","19.99","purple","Hey, you will rock at summer time." + ' +``` +> 202 Accepted - Response + +**PUT documents** `/indexes/:indexUid/documents` + +```bash +curl \ + -X PUT 'http://localhost:7700/indexes/movies/documents' \ + -H 'Content-Type: text/csv' \ + --data-binary ' + "id","label","price:number","colors","description"\n + "1","hoodie","19.99","purple","Hey, you will rock at summer time." + ' +``` +> 202 Accepted - Response + +##### Errors + +- 🔴 Omitted `Content-Type` header will lead to a 415 Unsupported Media Type - **missing_content_type** error code. +- 🔴 Sending an empty `Content-Type` will lead to a 415 Unsupported Media Type - **invalid_content_type** error code. +- 🔴 Sending a different `Content-Type` than `application/json`, `application/x-ndjson` or `text/csv` will lead to 415 Unsupported Media Type **invalid_content_type** error code. +- 🔴 Sending an empty payload will lead to a 400 Bad Request - **missing_payload** error code. +- 🔴 Sending a different payload type than the `Content-Type` header should return a 400 Bad Request - **malformed_payload** error code. +- 🔴 Sending a payload excessing the limit will lead to a 413 Payload Too Large - **payload_too_large** error code. +- 🔴 Sending an invalid CSV format will lead to a 400 bad_request - **malformed_payload** error code. +- 🔴 Sending a CSV header that does not conform to the specification will lead to a 400 bad_request - **malformed_payload** error code. + +##### Errors Definition + +## missing_content_type + +### Context + +This error occurs when the Content-Type header is missing. + +### Error Definition + +HTTP Code: `415 Unsupported Media Type` + +```json +{ + "message": "A Content-Type header is missing. Accepted values for Content-Type are: :contentTypeList", + "code": "missing_content_type", + "type": "invalid_request", + "link": "https://docs.meilisearch.com/errors#missing_content_type" +} +``` + +- The `:contentTypeList` is inferred when the message is generated. The values are separated by a `,` char. e.g. `application/json`, `text/csv`. + +--- + +## invalid_content_type + +### Context + +This error occurs when the provided content-type is not handled by the API method. + +### Error Definition + +HTTP Code: `415 Unsupported Media Type` + +```json +{ + "message": "The Content-Type :contentType is invalid. Accepted values for Content-Type are: :contentTypeList", + "code": "invalid_content_type", + "type": "invalid_request", + "link": "https://docs.meilisearch.com/errors#invalid_content_type" +} +``` + +- The `:contentTypeList` is inferred when the message is generated. The values are separated by a `,` char. e.g. `application/json`, `text/csv`. + +--- + +## missing_payload + +### Context + +This error occurs when the client does not provide a mandatory payload to the request. + +### Error Definition + +HTTP Code: `400 Bad Request` + +```json +{ + "message": "A :payloadType payload is missing.", + "code": "missing_payload", + "type": "invalid_request", + "link": "https://docs.meilisearch.com/errors#missing_payload" +} +``` + +- The `:payloadType` is inferred when the message is generated. e.g. `json`, `ndjson`, `csv` + +--- + +## malformed_payload + +### Context + +This error occurs when the format sent in the payload is malformed. The payload contains a syntax error. + +### Error Definition + +HTTP Code: `400 Bad Request` + +```json + "message": ":syntaxErrorHelper. The :payloadType payload provided is malformed.", + "code": "malformed_payload", + "type": "invalid_request", + "link": "https://docs.meilisearch.com/errors#malformed_payload" +``` + +- The `:payloadType` is inferred when the message is generated. e.g. `json`, `ndjson`, `csv` +- The `:syntaxErrorHelper` is inferred when the message is generated. + +--- + +## 2. Technical details +n/a + +## 3. Future possibilities + +- Provide an interface in the future dashboard to upload CSV data into an index and optionally provide the headers types. +- Set a payload limit directly related to the type of data format. Currently, the payload size is equivalent to [JSON payload size](https://docs.meilisearch.com/reference/features/configuration.html#payload-limit-size). Metrics on feature usage and configuration update should help to choose a better suited value for this type of data format. +- Accepts more common CSV separators diff --git a/text/0029-indexing-ndjson.md b/text/0029-indexing-ndjson.md new file mode 100644 index 00000000..bf2dcdd5 --- /dev/null +++ b/text/0029-indexing-ndjson.md @@ -0,0 +1,216 @@ +- Title: Indexing NDJSON +- Start Date: 2021-04-12 +- Specification PR: [PR-#29](https://github.com/meilisearch/specifications/pull/29) +- Discovery Issue: n/a + +# Indexing NDJSON + +## 1. Functional Specification + +### I. Summary + +To index documents, the body of the add documents request has to match a specific format. That specific format is then parsed and tokenized inside MeiliSearch. After which, the documents added are in the pool of searchable and returnable documents. + +An [NDJSON](http://ndjson.org/) data format is easier to use than a CSV format because it propose a convenient format for storing structured data. + +#### Summary Key Points + +- `application/x-ndjson` Content-Type header is now supported. +- The error cases have been strengthened and completed. See Errors part. + +### II. Motivation + +Currently, the engine only accepts JSON format as a data source. We want to give users the possibility of another simple data format to use. Thus, give them more versatility at the data source choices for the indexing step. + +Writing performance is also a motivation since JSON Lines data parsing is less CPU and memory-intensive than parsing standard JSON. When new lines represent separate entries it makes the NDJSON data streamable, thus, more suited for indexing a consequent data set. + +While we give the ability to Meilisearch to ingest CSV data for indexing in this [specification](https://github.com/meilisearch/specifications/pull/28), we are aware of the limitations of CSV so we also want to provide a format that is easy to validate. Handling the validity of a CSV can be frustrating and difficult. Only strings can be managed within a CSV. In addition, there is no official specification except [RFC 4180](https://tools.ietf.org/html/rfc4180) which is not sufficient for all data scheme. + +Representing nested structures in a JSON object is easy and convenient. + +### III. Explanation + +Newline-delimited JSON (`ndjson`), line-delimited JSON (`ldjson`), JSON lines (`jsonl`) are three terms expressing the same formats primarily intended for JSON streaming. + +As of now, we will use `ndjson` in the next parts to refer to a data format that represents JSON entries separated by a new line character. + +- Each entries will represent a document for MeiliSearch. +- Each entries should be a valid JSON object. +- The data should be encoded in UTF-8. + +#### Example of a valid NJSON + +Given the NDJSON payload +''' +{"id":1, "label": "t-shirt", "price": 4.99, "colors": ["red", "green", "blue"]} +{"id":499, "label": "hoodie", "price": 19.99, "colors": ["purple"]} +''' +the search result should be displayed as +```json +{ + "hits": [ + { + "id": 1, + "label": "t-shirt", + "price": 4.99, + "colors": [ + "red", + "green", + "blue" + ], + }, + { + "id": 499, + "label": "hoodie", + "price": 19.99, + "colors": [ + "purple" + ], + } + ], + ... +} +``` + +#### API Endpoints + +> Each API endpoints mentioned above will now require a `application/x-ndjson` as `Content-Type` header to be processed as NDJSON data. + +**As a developer, I want to upload a NDJSON payload of documents so that end-user can search them** + +**POST documents** `/indexes/:indexUid/documents` + +```bash +curl \ + -X POST 'http://localhost:7700/indexes/movies/documents' \ + -H 'Content-Type: application/x-ndjson' \ + --data-binary ' + {"id":1, "label": "t-shirt", "price": 4.99, "colors": ["red", "green", "blue"]}\n + {"id":499, "label": "hoodie", "price": 19.99, "colors": ["purple"]} + ' +``` +> 202 Accepted - Response + +**PUT documents** `/indexes/:indexUid/documents` + +```bash +curl \ + -X PUT 'http://localhost:7700/indexes/movies/documents' \ + -H 'Content-Type: application/x-ndjson' \ + --data-binary ' + {"id":1, "label": "t-shirt", "price": 4.99, "colors": ["red", "green", "blue"]}\n + {"id":499, "label": "hoodie", "price": 19.99, "colors": ["purple"]} + ' +``` +> 202 Accepted - Response + +##### Errors + +- 🔴 Omitted `Content-Type` header will lead to a 415 Unsupported Media Type - **missing_content_type** error code. +- 🔴 Sending an empty `Content-Type` will lead to a 415 Unsupported Media Type - **invalid_content_type** error code. +- 🔴 Sending a different `Content-Type` than `application/json`, `application/x-ndjson` or `text/csv` will lead to 415 Unsupported Media Type **invalid_content_type** error code. +- 🔴 Sending an empty payload will lead to a 400 Bad Request - **missing_payload** error code. +- 🔴 Sending a different payload type than the `Content-Type` header should return a 400 Bad Request - **malformed_payload** error code. +- 🔴 Sending a payload excessing the limit will lead to a 413 Payload Too Large - **payload_too_large** error code. +- 🔴 Sending an invalid ndjson format will lead to a 400 bad_request - **malformed_payload** error code. + +##### Errors Definition + +## missing_content_type + +### Context + +This error occurs when the Content-Type header is missing. + +### Error Definition + +HTTP Code: `415 Unsupported Media Type` + +```json +{ + "message": "A Content-Type header is missing. Accepted values for the Content-Type header are: :contentTypeList", + "code": "missing_content_type", + "type": "invalid_request", + "link": "https://docs.meilisearch.com/errors#missing_content_type" +} +``` + +- The `:contentTypeList` is inferred when the message is generated. The values are separated by a `,` char. e.g. `application/json`, `text/csv`. + +--- + +## invalid_content_type + +### Context + +This error occurs when the provided content-type is not handled by the API method. + +### Error Definition + +HTTP Code: `415 Unsupported Media Type` + +```json +{ + "message": "The Content-Type :contentType is invalid. Accepted values for the Content-Type header are: :contentTypeList", + "code": "invalid_content_type", + "type": "invalid_request", + "link": "https://docs.meilisearch.com/errors#invalid_content_type" +} +``` + +- The `:contentTypeList` is inferred when the message is generated. The values are separated by a `,` char. e.g. `application/json`, `text/csv`. + +--- + +## missing_payload + +### Context + +This error occurs when the client does not provide a mandatory payload to the request. + +### Error Definition + +HTTP Code: `400 Bad Request` + +```json +{ + "message": "A :payloadType payload is missing.", + "code": "missing_payload", + "type": "invalid_request", + "link": "https://docs.meilisearch.com/errors#missing_payload" +} +``` + +- The `:payloadType` is inferred when the message is generated. e.g. `json`, `ndjson`, `csv` + +--- + +## malformed_payload + +### Context + +This error occurs when the format sent in the payload is malformed. The payload contains a syntax error. + +### Error Definition + +HTTP Code: `400 Bad Request` + +```json + "message": ":syntaxErrorHelper. The :payloadType payload provided is malformed.", + "code": "malformed_payload", + "type": "invalid_request", + "link": "https://docs.meilisearch.com/errors#malformed_payload" +``` + +- The `:payloadType` is inferred when the message is generated. e.g. `json`, `ndjson`, `csv` +- The `:syntaxErrorHelper` is inferred when the message is generated. + +--- + +## 2. Technical details +n/a + +## 3. Future possibilities + +- Provide an interface in the future dashboard to upload NDJSON data into an index. +- Set a payload limit directly related to the type of data format. Currently, the payload size is equivalent to [JSON payload size](https://docs.meilisearch.com/reference/features/configuration.html#payload-limit-size). Metrics on feature usage and configuration update should help to choose a better suited value for this type of data. diff --git a/text/0059-geo-search.md b/text/0059-geo-search.md new file mode 100644 index 00000000..24270da0 --- /dev/null +++ b/text/0059-geo-search.md @@ -0,0 +1,272 @@ +- Title: Geosearch +- Start Date: 2021-08-02 +- Specification PR: [#59](https://github.com/meilisearch/specifications/pull/59) +- Discovery Issue: [#42](https://github.com/meilisearch/product/issues/42) + +# Geosearch + +## 1. Functional Specification + +### I. Summary + +The purpose of this specification is to add a first iteration of the **geosearch** feature to give geo-filtering and geosorting capabilities at search time. + +#### Summary Key points + +- Documents MUST have a `_geo` reserved object to be geosearchable. +- Filter documents by a given geo radius using the built-in filter `_geoRadius({lat}, {lng}, {distance_in_meters})`. It is possible to cumulate several geosearch filters within the `filter` field. +- Sort documents in ascending/descending order around a geo point. e.g. `_geoPoint({lat}, {lng}):asc`. +- It is possible to filter and/or sort by geographical criteria of the user's choice. +- `_geo` must be set as a filterable attribute to use geo filtering capabilities. +- `_geo` must be set as a sortable attribute to use geo sort capabilities. +- There is no `geo` ranking rule that can be manipulated by the user. This one is automatically integrated in the ranking rule `sort` by default and activated by sorting using the `_geoPoint({lat}, {lng})` built-in sort rule. +- Using `_geoPoint({lat}, {lng})` in the `sort` parameter at search leads the engine to return a `_geoDistance` within the search results. This field represents the distance in meters of the document from the specified `_geoPoint`. +- Add an `invalid_geo_field` error. +- Add an alternative message for `invalid_sort` and `invalid_filter` error to handle reserved keywords. +- `invalid_criterion` is renamed to `invalid_ranking_rule` and add an alternative message to handle reserved keywords. + +### II. Motivation + +According to our user feedback, the lack of a geosearch feature is mentioned as one of the biggest deal-breakers for choosing MeiliSearch as a search engine. A search engine must offer this feature. Some use cases specifically require integrated geosearch capabilities. Moreover, a lot of direct competitors offer it. Users today must find workarounds like using geohash to be able to geosearch documents. We hope to better serve the needs of users by implementing this feature. It allows multiplying the use-cases to which MeiliSearch can respond. + +### III. Technical Explanations + +#### **As a developer, I want to add geospatial coordinates to a document so that the document can be geosearchable.** + +- Introduce a reserved field `_geo` for documents to store geo spatial data from an **object** made of `lat` and `lng` fields for a **JSON format**. +- Introduce a reserved column `_geo` for documents to store geo spatial data from a **string** made of `lat,lng` for a **CSV format**. + +##### **JSON Format** + +**`_geo` field definition** + +- Name: `_geo` +- Type: Object +- Format: `{lat:float, lng:float}` +- Not required + +> 💡 if `_geo` is found in the document payload, `lat` and `lng` are required. +> 💡 `lat` and `lng` must be of float value. + +##### **CSV Format** + +Following the format already defined in the https://github.com/meilisearch/specifications/pull/28/files specification for document indexing from a CSV format. A reserved column `_geo` can be added to specify the geographical coordinates of a document. + +csv format example +``` +"id:number","label","brand","_geo" +"1","F40","Ferrari","48.862725,2.287592" +``` + +**`_geo` column definition** + +- Name: `_geo` +- Type: String +- Format: `"lat:float,lng:float"` +- Not required + +#### POST Add or replace documents `/indexes/{indexUid}/documents` + +##### Request body +```json +[ + { + "id": 1, + "label": "F40", + "brand": "Ferrari", + "_geo": { + "lat": 48.862725, + "lng": 2.287592 + } + } +] +``` + +##### 202 Accepted - Response body + +```json +{ + "updateId": 1 +} +``` + +#### PUT Add or replace documents `/indexes/{indexUid}/documents` + +##### Request body + +```json +[ + { + "id": 1, + "brand": "F40 LM", + "brand": "Ferrari", + "_geo": { + "lat": 48.862725, + "lng": 2.287592 + } + } +] +``` + +##### 202 Accepted - Response body + +```json +{ + "updateId": 2 +} +``` + +- 🔴 Giving a bad formed `_geo` that do not conform to the format causes the `update` payload to fail. A new `invalid_geo_field` error is given in the `update` object. + +##### Errors Definition + +## invalid_geo_field + +### Context + +This error occurs when the `_geo` field of a document payload is not valid. + +### Error Definition + +```json +{ + "message": "The document with the id: `:documentId` contains an invalid _geo field: :syntaxErrorHelper.", + "code": "invalid_geo_field", + "type": "invalid_request", + "link": "https://docs.meilisearch.com/errors#invalid_geo_field" +} +``` + +- The `:documentId` is inferred when the message is generated. +- The `:syntaxErrorHelper` is inferred when the message is generated. + +--- + +### **As an end-user, I want to filter documents within a geo radius.** + +- Introduce a `_geoRadius({lat}, {lng}, {distance_in_meters})` built-in filter rule to `filter` documents in a geo radius.shape. + +**`_geoRadius` built-in filter rule definition** + +- Name: `_geoRadius` +- Signature: ({lat:float}:required, {lng:float}:required, {distance_in_meters:int}:required) +- Not required +- `distance_in_meters` only accepts positive value. + +> The `_geo` field has to be set in `filterableAttributes` setting by the developer to activate geo filtering capabilities at search. + +#### GET Search `/indexes/{indexUid}/search` + +``` +...&filter="brand=Mercedes AND _geoRadius(48.862725, 2.287592, 2000)"` +``` + +#### POST Search `/indexes/{indexUid}/search` + +```json +{ + "filter": ["brand = Ferrari", "_geoRadius(48.862725, 2.287592, 2000)"] +} +``` + +- 🔴 Specifying parameters that do not conform to the `_geoRadius` signature causes the API to return an `invalid_filter` error. The error message should indicate how `_geoRadius` should be used. See `_geoRadius` built-in filter rule definition part. +- 🔴 Using `_geo`, `_geoDistance`, `_geoPoint` in a filter expression cause the API to return an `invalid_filter` error. `message` should be `:reservedKeyword is a reserved keyword and thus can't be used as a filter expression.` + +--- + +### **As an end-user, I want to sort documents around a geo point.** + +- Introduce a `_geoPoint({lat}, {lng})` function parameter to `sort` documents around a central point. + +**`_geoPoint` built-in sort rule definition** + +- Name: `_geoPoint` +- Signature: ({lat:float}:required, {lng:float}:required) +- Not required + +Following the [`sort` specification feature](https://github.com/meilisearch/specifications/pull/55): +> The `_geo` field has to be set in `sortableAttributes` setting by the developer to activate geo sorting capabilities at search. +> +>There is no `geo` ranking rule as such. It is in fact within the `sort` ranking rule in an obfuscated way. +> +>`_geoPoint` built-in sort rule can sort documents in ascending/descending order. + +#### GET Search `/indexes/{indexUid}/search` + +``` + ...&sort=_geoPoint({lat, lng}):asc,price:desc +``` + +#### POST Search `/indexes/{indexUid}/search` + +```json +{ + "sort": "_geoPoint({lat, lng}):asc,price:desc" +} +``` +- 🔴 Specifying parameters that do not conform to the `_geoPoint` signature causes the API to return an `invalid_sort` error. The error message should indicate how `_geoPoint` should be used. See `_geoPoint` built-in sort rule definition part. +- 🔴 Using `_geo`, `_geoDistance`, `_geoRadius` in a sort expression cause the API to return an `invalid_sort` error. `message` should be `:reservedKeyword is a reserved keyword and thus can't be used as a sort expression.` + +--- + +### **As an end-user, I want to know the document distance when I am sorting around a geo point.** + +- Introduce a `_geoDistance` parameter to the search result `hit` object. + +**`_geoDistance` field definition** + +- Name: `_geoDistance` +- Description: Return document distance when the end-user sorts document from a `_geoPoint` in meters. +- Type: int +- Not required + +> 💡 `_geoDistance` response field is only computed and shown when the end-user have sorted documents around a `_geoPoint`. So if the end-user filters documents using a `_geoRadius` built-in filter without sorting them around a `_geoPoint`, this field `_geoDistance` will not appear in the search response. + +--- + +### `invalid_criterion` error changes + +The error is currently marked as an internal error thus the name is not explicit and consistent with the term `Ranking Rule` a user can encounter in the documentation and in the API resource name. A new definition of this error is proposed. + +#### invalid_ranking_rule + +#### Context + +This error is raised asynchronously when the user tries to specify an invalid ranking rule in the ranking rules setting. + +#### Error Definition + +```json + "message": ":rankingRule ranking rule is invalid. Valid ranking rules are Words, Typo, Sort, Proximity, Attribute, Exactness and custom ranking rules." + "code": "invalid_ranking_rule" + "type": "invalid_request" + "link": "https://docs.meilisearch.com/errors#invalid_ranking_rule" +``` + +- 🔴 Specifying an invalid ranking rule name raises an `invalid_ranking_rule` error. See `message` defined in the error definition part. +- 🔴 Specifying a custom ranking rule with `_geo` or `_geoDistance` raises an `invalid_ranking_rule` error. The message is `:reservedKeyword is a reserved keyword and thus can't be used as a ranking rule.`. +- 🔴 Specifying a custom ranking rule with `_geoPoint` raises an `invalid_ranking_rule` error. The message is `_geoPoint is a reserved keyword and thus can't be used as a ranking rule. _geoPoint can only be used for sorting at search time`. +- 🔴 Specifying a custom ranking rule with `_geoRadius` raises an `invalid_ranking_rule` error. The message is `_geoRadius is a reserved keyword and thus can't be used as a ranking rule. _geoRadius can only be used for filtering at search time`. +--- + +### IV. Finalized Key Changes + +- Add a `_geo` reserved field on JSON and CSV format to index a geo point coordinates for a document. +- Add a `_geoPoint(lat, lng)` built-in sort rule. +- Add a `_geoRadius(lat, lng, distance_in_meters)` built-in filter rule. +- Return a `_geoDistance` in `hits` objects representing the distance in meters computed from the `_geoPoint` built-in sort rule. + +## 2. Technical Aspects + +### I. Measuring + +- `filterableAttribute` setting definition to evaluate `_geo` presence. +- `sortableAttribute` setting definition to evaluate `_geo` presence. + +## 3. Future Possibilities + +- Add built-in filter to filter documents within `polygon` and `bounding-box`. +- Handling array of geo points in the document object. +- Handling multiple geo formats for the `_geo` field. e.g. "{lat},{lng}", a geohash etc. +- Handling distance in other formats (like the imperial format). **It's easy to implement on the user side though.** +- Handling position in other formats. It seems that [degrees and minutes](https://www.pacioos.hawaii.edu/voyager-news/lat-long-formats/) are also used a lot. **It's easy to implement on the user side though.** \ No newline at end of file