[Inference API] Add Alibaba Cloud AI Search Model support to Inference API

[weizijun]

Add Alibaba Cloud AI Search Model support to Inference API.

Prerequisites to Model Creation

An Alibaba Cloud Account with Alibaba Cloud Opensearch access
An api key used to access Alibaba Cloud AI Search Model

Inference Model Creation:

PUT _inference/{task_type}/{inference_model_id}
{
  "service": "alibabacloud-ai-search",
  "service_settings": {
    "api_key": "{{api_key}}",
    "service_id": "<<service_id>>",
    "host": "<<host>",
    "workspace": "<<workspace_name>>",
    <<ADDITIONAL SERVICE SETTINGS (see below)>>,
  }
  "task_settings": {
    <<TASK SETTINGS (see below)>>
  }
}

Support task_type

We have now added support for these task_types:

• text_embedding
• sparse_embedding
• rerank

We also support other task_types, such as completion, document analyze, document split, image analyze, query analyze.
Here is the document: https://help.aliyun.com/zh/open-search/search-platform/product-overview/introduction-to-search-platform

Testing

Creating the inference endpoint for Alibaba Cloud AI Search

PUT _inference/text_embedding/os-embeddings
{
  "service": "alibabacloud-ai-search",
  "service_settings": {
    "api_key": "{{API_KEY}}",
    "service_id": "ops-text-embedding-001",
    "host" : "default-j01.platform-cn-shanghai.opensearch.aliyuncs.com",
    "workspace" : "default"
  }
}

Performing text_embedding inference:

POST _inference/text_embedding/os-embeddings
{
    "input": "What is Elastic?"
}

[elasticsearchmachine]

Pinging @elastic/search-inference-team (Team:Search - Inference)

[elasticsearchmachine]

Pinging @elastic/ent-search-eng (Team:SearchOrg)

[saikatsarkar056]

@davidkyle, @weizijun

After this backend PR has been merged, we need to add alibaba to inference endpoints view with proper icon and display fields in the front-end. This is an example PR for Amazon Bedrock. Thank you.

[weizijun]

After this backend PR has been merged, we need to add alibaba to inference endpoints view with proper icon and display fields in the front-end. This is an example PR for Amazon Bedrock. Thank you.

Ok, I will try to add alibaba to inference endpoints view PR. Thank you!
And about this PR, Is there anything else to add to this PR?

[davidkyle]

This looks really good thank you!

I don't see any problems but this is a large PR so I'd like to review the changes 1 more time and test the inference against an Alibaba AI model myself.

[davidkyle]

@weizijun I resolved the conflict in TransportVersions.java the CI build should run now

[davidkyle]

Fantastic I was able to create text embeddings, sparse embeddings and rerank documents in Alibaba with this change.

The comments I've left are all related to the configuration and are suggestions to improve the validation. Otherwise the PR is good Thank You ❤️

[davidkyle]

LGTM

[davidkyle]

Thanks @weizijun this is great to have another reranker, sparse embeddings and a text embedding service added. I've asked one of the Elasticsearch writers to create the documentation for this change.

[weizijun]

Thanks @weizijun this is great to have another reranker, sparse embeddings and a text embedding service added. I've asked one of the Elasticsearch writers to create the documentation for this change.

Thanks @davidkyle , What can I do to help with document writing?

[serenachou]

@weizijun is there any chance you'd want to add a notebook into https://github.com/elastic/elasticsearch-labs/tree/main/notebooks/integrations to show a few examples? We can use that for a blog post that we can write together too. there's a few tutorials we can also add Alibaba code examples to : https://www.elastic.co/guide/en/elasticsearch/reference/master/semantic-search-semantic-text.html or https://www.elastic.co/guide/en/elasticsearch/reference/master/semantic-search-inference.html depending on the use case. cc @szabosteve

[davidkyle]

@elasticmachine test this please

[weizijun]

@weizijun is there any chance you'd want to add a notebook into https://github.com/elastic/elasticsearch-labs/tree/main/notebooks/integrations to show a few examples?

OK, Let me see, and I will add some examples.

[davidkyle]

@elasticmachine test this please

[davidkyle]

@elasticmachine test this please

[weizijun]

Thanks to @davidkyle for merging this PR. I will write a PR document for our inference service.