post filter not applied to field collapsing inner hits?

[qwerty4030]

Elasticsearch version: 5.3.3

Plugins installed: [discovery-ec2, repository-s3, x-pack]

JVM version : 1.8.0_131 Java HotSpot(TM) 64-Bit Server VM

OS version : 3.13.0-121-generic AWS Ubuntu 14.04 LTS

Description of the problem including expected versus actual behavior: I recently started using the new field collapsing functionality and I noticed that the search post_filter is not applied to the inner_hits. This was surprising behavior to me because the post_filter applies to the collapsed search hits just fine and I did not see anything that said it wouldn't work. However I'm not 100% sure if this would be a bug or enhancement.

We use the post_filter for the typical faceted navigation use case and we wanted to leverage the new field collapse with inner_hits to add more variety to our search results. I guess a workaround would be to use a top_hits + filter aggregation to do the field collapse? Are there any significant performance differences between using the top_hits aggregation vs the new collapse functionality?

Steps to reproduce:

POST test-index/test-type/1
{
  "value":1,
  "common":1
}

POST test-index/test-type/2
{
  "value":2,
  "common":1
}

POST test-index/test-type/_search
{
  "collapse": {
    "field": "common",
    "inner_hits": {
      "name": "common"
    }
  },
  "post_filter": {
    "term": {
      "value": 1
    }
  }
}

Note the search result inner_hits contains both doc 1 and 2, but I'm expecting to see only doc 1 because the post_filter does not match doc 2.

[martijnvg]

@jimczi I think the reason why this happens is because the collapse collector is executed before the filtered collector in QueryPhase#execute(...)? (line 259 collectors.addFirst(topDocsFactory); )

[jimczi]

@martijnvg the collector chain is reversed when it is built from the factories so the filtered collector is before the top docs collector and works fine on the collapsed hits. This issue is about applying the post_filter to the expand phase (the inner_hits retrieval), currently it is just ignored.
I agree that it should be applied so it can be considered as a bug.

Are there any significant performance differences between using the top_hits aggregation vs the new collapse functionality?

Yes, the field collapsing is only applied to the top_hits of the query so it should be faster than using an combo agg terms + top_hits.

[martijnvg]

@jimczi Ah I forgot that the chain is reversed. Thx for the explanation.