Support configurable chunking in semantic_text fields

[kderusso]

Supports configurable chunking in semantic_text fields that will override the model defaults.

Example script:

PUT my-chunking
{
  "mappings": {
    "properties": {
      "default_chunked_inference_field": {
        "type": "semantic_text"
      },
      "custom_chunked_inference_field": {
        "type": "semantic_text",
        "chunking_settings": {
          "strategy": "word",
          "max_chunk_size": 10,
          "overlap": 5
        }
      }
    }
  }
}

PUT my-chunking/_doc/1
{
  "default_chunked_inference_field": "Elasticsearch is an open source, distributed, RESTful, search engine which is built on top of Lucene internally and enjoys all the features it provides.",
  "custom_chunked_inference_field": "Elasticsearch is an open source, distributed, RESTful, search engine which is built on top of Lucene internally and enjoys all the features it provides."
}


GET my-chunking/_search
{
  "query": {
    "bool": {
      "should": [
        {
          "semantic": {
            "field": "default_chunked_inference_field",
            "query": "what is elasticsearch"
          }
        },
        {
          "semantic": {
            "field": "custom_chunked_inference_field",
            "query": "what is elasticsearch"
          }
        }
      ]
    }
  },
  "highlight": {
    "fields": {
      "default_chunked_inference_field": {
        "type": "semantic",
        "number_of_fragments": 5
      },
      "custom_chunked_inference_field": {
        "type": "semantic",
        "number_of_fragments": 5
      }
    }
  }
}

[Mikep86]

Looks good, thank you for the iterations on this!

[Mikep86]

I noticed a completely unrelated bug in SentenceBoundaryChunkingSettings while reviewing this: equals does not consider sentenceOverlap

[kderusso]

We can spin up a quick and dirty followup PR for that, not addressing it here.

[kderusso]

#126250

[elasticsearchmachine]

💔 Backport failed

Status	Branch	Result
❌	8.x	Commit could not be cherrypicked due to conflicts

You can use sqren/backport to manually backport by running backport --upstream elastic/elasticsearch --pr 121041

[kderusso]

💚 All backports created successfully

Status	Branch	Result
✅	8.x

Questions ?

Please refer to the Backport tool documentation