Pre-filter nested fields in knn queries

[ezorita]

Description

By reading some of the latest bugfixes in 8.13.0 and the documentation it seems it is still not possible to pre-filter a knn search on nested mappings, such as:

{
   "mappings":{
      "properties":{
         "title":{
            "type":"keyword"
         },
         "paragraphs":{
            "type":"nested",
            "properties":{
               "language":{
                  "type":"keyword"
               },
               "vector":{
                  "type":"dense_vector",
                  "dims":3,
                  "similarity":"cosine",
                  "index":true
               }
            }
         }
      }
   }
}

then searching the following would never match any document:

{
   "query":{
      "nested":{
         "path":"paragraphs",
         "query":{
            "knn":{
               "field":"paragraphs.vector",
               "query_vector":[
                  0.5,
                  0.5,
                  0.5
               ],
               "num_candidates":5,
               "filter":{
                  "bool":{
                     "must":[
                        {
                           "match":{
                              "paragraphs.language":"EN"
                           }
                        }
                     ]
                  }
               }
            }
         }
      }
   }
}

or

{
   "query":{
      "nested":{
         "path":"paragraphs",
         "query":{
            "knn":{
               "field":"paragraphs.vector",
               "query_vector":[
                  0.5,
                  0.5,
                  0.5
               ],
               "num_candidates":5,
               "filter":{
                  "nested":{
                     "path":"paragraphs",
                     "query":{
                        "bool":{
                           "must":[
                              {
                                 "match":{
                                    "paragraphs.language":"EN"
                                 }
                              }
                           ]
                        }
                     }
                  }
               }
            }
         }
      }
   }
}

In the current implementation the filter scope is limited to the parent document, thus limiting other functionalities of nested documents.

I wonder how hard it would be to extend the filter context to nested documents?

Thank you!

[ppaanngggg]

Yes, same problem with @ezorita , the pre-filter of nested fields is a key feature to me.

I think the best DSL to this case should like this:

{
   "knn":{
      "query_vector": [0.5, 0.5, 0.5],
      "field": "paragraphs.vector",
      "k": 10,
      "num_candidates": 100,
      "filter": {
         "nested": {
            "path": "paragraphs",
            "query": {
               "term": {"paragraphs.language": "EN"}
            },
            "inner_hits": {
               "size": 5
            }
         }
      }
   }
}

[xflashxx]

Same for me

[xflashxx]

Any updates on whether this is doable at all? 🤗

[mpaluch92]

I presume that this is doable, as Opensearch offers it as well: https://opensearch.org/docs/latest/search-plugins/knn/nested-search-knn/#k-nn-search-with-filtering-on-nested-fields

[elasticsearchmachine]

Pinging @elastic/es-search-relevance (Team:Search Relevance)

[GinoBerardelli]

This would be great!

[dstith-ip]

Upvote for this feature!

[gui11aume]

Good feature.

[mayya-sharipova]

Thanks for highlighting the importance of this issue, we will look into it.