Composite embedders

[Kerollmops]

Related product team resources: PRD (internal only)
Related product discussion:

Motivation

Search speed and latency reduction when using AI and embedders.

Usage

https://meilisearch.notion.site/Composite-embedder-usage-14a4b06b651f81859dc3df21e8cd02a0?pvs=4

TODO

Reminders when modifying the API

• Update the openAPI file with utoipa:
  • If a new module has been introduced, create a new structure deriving the OpenAPI proc-macro and nest it in the main openAPI structure.
  • If a new route has been introduced, add the path decorator to it and add the route at the top of the file in its openAPI structure.
  • If a structure which is deserialized or serialized in the API has been introduced or modified, it must derive the schema or the IntoParams proc-macro.
    If it's a new structure you must also add it to the big list of structures in the main OpenApi structure.
  • Once everything is done, start Meilisearch with the swagger flag: cargo run --features swagger, open http://localhost:7700/scalar on your browser, and ensure everything works as expected.
  • For more info, refer to this presentation.

Reminders when modifying the Setting API

• Ensure the new setting route is at least tested by the test_setting_routes macro
• Ensure Analytics are fully implemented
  • /settings/my-new-setting configurated in the make_setting_routes macro
  • global /settings route configurated in the update_all function
• Ensure the dump serializing is consistent with the /settings route serializing, e.g., enums case can be different (camelCase in route and PascalCase in the dump)

Special cases when adding a setting for an experimental feature

• ⚠️ API stability: The setting does not appear on the main settings route when the feature has never been enabled (e.g. mark it Unset when returned from the index in this situation. See an example)
• The setting cannot be set when the feature is disabled, either by the main settings route or the subroute (see validate_settings function)
• If possible, the setting is reset when the feature is disabled (hard if it requires reindexing)

Impacted teams

@meilisearch/docs-team, @meilisearch/integration-team, @meilisearch/cloud-team