Multiple aggregations are not parsed correctly in ISearchResponse object

[tagg7]

I encountered an error while performing a query of the following form:-

var response = client.Search(s => s
.Query(...)
.Aggregations(a => a
   .TopHits(...)
   .Terms(...)

In the response, only a single aggregation is parsed and available in ISearchResponse.Aggregations (always TopHits). I can see from the raw response that the Terms query was also returned.

[gmarz]

Hey @tagg7 that's strange...I'm not able to reproduce this. Here's the test I'm running:

var response = this.Client.Search<ElasticsearchProject>(s => s
  .Size(0)
  .Aggregations(a => a
    .TopHits("my-top-hits", th => th
      .Size(1)
    )
    .Terms("my-terms", ts => ts
      .Field(p => p.Name)
    )
  )
);

response.IsValid.Should().BeTrue();

var topHits = response.Aggs.TopHitsMetric("my-top-hits");
topHits.Should().NotBeNull();
topHits.Hits<ElasticsearchProject>().Count().Should().BeGreaterThan(0);

var terms = response.Aggs.Terms("my-terms");
terms.Should().NotBeNull();
terms.Items.Count().Should().BeGreaterThan(0);

And everything passes.

Can you maybe provide some more info? The exact query you're executing with NEST and the full raw response?

Also, the top hits aggregation isn't very useful as a top-level aggregation, which is how you're using it. It's really intended to be a sub-aggregator, i.e:

.Aggregations(a => a
    .Terms("my-terms", ts => ts
        ...
        .Aggregations(aa => aa
            .TopHits("my-top-hits", th => th
                ...
        )

Not sure if that's maybe throwing off your expectations?

[tagg7]

So I simplified my code a bit and this is it:

var response = client.Search<WebsiteResult>(s => s
    .Size(0)
    .Query(q => q
        .MatchAll()
    )
    .Aggregations(a => a
        .TopHits("top_hits_oldest", tha => tha
            .Size(1)
        )
        .Terms("day_of_week_terms", t => t
            .Field(x => x.Timestamp)
        )
    )
);

I did have additional options set (ie. actually sorting the TopHits result by a field), but I had the same problem even with this minimal query. Note that the Timestamp field is just a DateTime object.

The HTTP request was:

{
  "size": 0,
  "aggs": {
    "top_hits_oldest": {
      "top_hits": {
        "size": 1
      }
    },
    "day_of_week_terms": {
      "terms": {
        "field": "timestamp"
      }
    }
  },
  "query": {
    "match_all": {}
  }
}

And the HTTP response was:

{
    "took": 3,
    "timed_out": false,
    "_shards": {
        "total": 5,
        "successful": 5,
        "failed": 0
    },
    "hits": {
        "total": 12,
        "max_score": 0,
        "hits": []
    },
    "aggregations": {
        "top_hits_oldest": {
            "hits": {
                "total": 12,
                "max_score": 1,
                "hits": [
                    {
                        *Item Result* (Redacted)
                    }
                ]
            }
        },
        "day_of_week_terms": {
            "doc_count_error_upper_bound": 0,
            "sum_other_doc_count": 2,
            "buckets": [
                {
                    "key": 1418861390212,
                    "key_as_string": "2014-12-18T00:09:50.212Z",
                    "doc_count": 1
                },
                {
                    "key": 1418861390368,
                    "key_as_string": "2014-12-18T00:09:50.368Z",
                    "doc_count": 1
                }
            ]
        }
    }
}

Thank you for the quick response!

[tagg7]

Okay, I think I figured out the problem. I renamed the terms aggregation to something without the string "terms" in it and now it's working...

[gmarz]

@tagg7 Okay, great. Having terms in the name of the aggregation shouldn't make any difference though. Is it possible that you were maybe retrieving the aggs result from the response by the wrong name?

[tagg7]

@gmarz No, nothing like that. There was only ever a single aggs result until I changed the terms name.

[gmarz]

@tagg7 Yikes, this is indeed a bug. The problem has to due with the order in which the aggregation results come back from ES.

Depending on the name, in some cases the top hits aggregation is returned first other times the terms aggregation.

For instance:

{
  "size": 0,
  "aggs": {
    "my-top-hits": {
      "top_hits": {
        "size": 1
      }
    },
    "b": {
      "terms": {
        "field": "name",
        "size": 20
      }
    }
  }
}

returns

"aggregations": {
      "b": {
         "doc_count_error_upper_bound": 0,
         "sum_other_doc_count": 0,
         "buckets": [
            ...
         ]
      },
      "my-top-hits": {
         "hits": {
            "total": 1,
            "max_score": 1,
            "hits": [
                ...
            ]
         }
      }
   }
}

while

{
  "size": 0,
  "aggs": {
    "my-top-hits": {
      "top_hits": {
        "size": 1
      }
    },
    "a": {
      "terms": {
        "field": "name",
        "size": 20
      }
    }
  }
}

returns

"aggregations": {
  "my-top-hits": {
     "hits": {
        "total": 1,
        "max_score": 1,
        "hits": [
            ...
        ]
     }
  },
  "a": {
     "doc_count_error_upper_bound": 0,
     "sum_other_doc_count": 0,
     "buckets": [
        ...
     ]
  }
}

The bug in NEST is that when the top_hits result is returned first, the parsing code is thrown off. We can't rely on ES returning the results in a certain order, and it shouldn't matter regardless.

We will fix this ASAP. Thanks for reporting!

[tagg7]

Awesome, glad you were able to get to the bottom of it. Thanks for your help!