[CLOUDP-367240] Add support of auto embeddings for vector search using mongot config

[viveksinghggits]

Summary

This PR adds the support for auto embeddings in case of vector search in MCK. Without auto embeddings if a customer has to use vector search, they will have to generate vector embeddings for their data manually (themselves) and after the vector embeddings are generated, they can use the vector search.
With the help of auto embeddings, if customers opt in, they won't have to generate the vector embeddings by themselves. If configured properly, mongot will do it for the customers when the vector search index is created. mongot looks for specific fields (embedding) in it's config to decide it has to do auto embedding or not.

This PR adds support to enable users to configure the mongot config so that mongot can do auto embedding, and to do that we are exposing the autoEmbedding field in the MongoDBSearch CR. Using this field customers will be able to configure embedding field of mongot config.

k explain mongodbsearch.spec.autoEmbedding
GROUP:      mongodb.com
KIND:       MongoDBSearch
VERSION:    v1

FIELD: autoEmbedding <Object>


DESCRIPTION:
    Configure MongoDB Search's automatic generation of vector embeddings using
    an embedding model service.
    `embedding` field of mongot config is generated using the values provided
    here.

FIELDS:
  embeddingModelAPIKeySecret    <Object> -required-
    EmbeddingModelAPIKeySecret would have the name of the secret that has two
    keys
    query-key and indexing-key for embedding model's API keys.

  providerEndpoint      <string>
    <no description>


k explain mongodbsearch.spec.autoEmbedding.embeddingModelAPIKeySecret
GROUP:      mongodb.com
KIND:       MongoDBSearch
VERSION:    v1

FIELD: embeddingModelAPIKeySecret <Object>


DESCRIPTION:
    EmbeddingModelAPIKeySecret would have the name of the secret that has two
    keys
    query-key and indexing-key for embedding model's API keys.

FIELDS:
  name  <string>
    Name of the referent.
    This field is effectively required, but due to backwards compatibility is
    allowed to be empty. Instances of this type with an empty value here are
    almost certainly wrong.
    More info:
    https://kubernetes.io/docs/concepts/overview/working-with-objects/names/#names

Should we make EmbeddingModelAPIKeySecret mandatory in CR if the ProviderEndpoint is specified?

Proof of Work

Install the locally built version of search using locally run MCK operator, make sure MongoDBSearch resource is in running state. And create the search resource using below manifest

kubectl apply --context "${K8S_CTX}" -n "${MDB_NS}" -f - <<EOF
apiVersion: mongodb.com/v1
kind: MongoDBSearch
metadata:
  name: ${MDB_RESOURCE_NAME}
spec:
  # no need to specify source.mongodbResourceRef if MongoDBSearch CR has the same name as MongoDB CR
  # the operator infer it automatically
  resourceRequirements:
    limits:
      cpu: "3"
      memory: 5Gi
    requests:
      cpu: "2"
      memory: 3Gi
  autoEmbedding:
    providerEndpoint: https://api.voyageai.com/v1/embeddings
    embeddingModelAPIKeySecret: voyage-api-keys
EOF

and then go ahead and create the vector search index

use sample_mflix;

db.movies.createSearchIndex("vector_index", "vectorSearch",
      { "fields": [ {
        "type": "autoEmbed",
        "path": "plot",
        "modality": "text",
        "model": "voyage-3.5-lite"
      } ] });

query the data to make sure search is working

db.movies.aggregate([
   {
     "$vectorSearch": {
       "index": "vector_index",
       "path": "plot",
       "query": "spy crime",
       "numCandidates": 150,
       "limit": 10,
       "quantization": "scalar"
     }
   },
   {
     "$project": {
       "_id": 0,
       "plot": 1,
       "title": 1,
       "score": { "$meta": "vectorSearchScore" }
     }
   }
 ]);

E2E test will be part of another PR.

go test  -run ^TestEnsureEmbeddingConfig -v
=== RUN   TestEnsureEmbeddingConfig_APIKeySecretAndProviderEndpont
--- PASS: TestEnsureEmbeddingConfig_APIKeySecretAndProviderEndpont (0.00s)
=== RUN   TestEnsureEmbeddingConfig_WOAutoEmbedding
--- PASS: TestEnsureEmbeddingConfig_WOAutoEmbedding (0.00s)
=== RUN   TestEnsureEmbeddingConfig_JustAPIKeys
--- PASS: TestEnsureEmbeddingConfig_JustAPIKeys (0.00s)
PASS
ok      github.com/mongodb/mongodb-kubernetes/controllers/searchcontroller      0.704s

Checklist

• Have you linked a jira ticket and/or is the ticket in the title?
• Have you checked whether your jira ticket required DOCSP changes?
• Have you added changelog file?
  • use skip-changelog label if not needed
  • refer to Changelog files and Release Notes section in CONTRIBUTING.md for more details

[github-actions]

⚠️ (this preview might not be accurate if the PR is not rebased on current master branch)

MCK 1.6.2 Release Notes

Bug Fixes

• Fixed an issue where monitoring agents would fail after disabling TLS on a MongoDB deployment.
• Persistent Volume Claim resize fix: Fixed an issue where the Operator ignored namespaces when listing PVCs, causing conflicts with resizing PVCs of the same name. Now, PVCs are filtered by both name and namespace for accurate resizing.
• Fixed a panic that occurred when the domain names for a horizon was empty. Now, if the domain names are not valid (RFC 1123), the validation will fail before reconciling.
• Fixed an issue where the Operator could crash when TLS certificates are configured using the certificatesSecretsPrefix field without additional TLS settings.

[fealebenpae]

I think this looks good, thanks for your hard work!

[anandsyncs]

LGTM, please fix the linting issues.