Histogram aggregation fails on NaN

[thomas11]

Elasticsearch version (bin/elasticsearch --version):

Version: 5.6.1, Build: 667b497/2017-09-14T19:22:05.189Z, JVM: 1.8.0_141

Plugins installed: none

JVM version (java -version):

openjdk version "1.8.0_92"
OpenJDK Runtime Environment (Zulu 8.15.0.1-win64) (build 1.8.0_92-b15)
OpenJDK 64-Bit Server VM (Zulu 8.15.0.1-win64) (build 25.92-b15, mixed mode)

OS version (uname -a if on a Unix-like system):

Windows 10

Description of the problem including expected versus actual behavior:

The histogram aggregation fails on a field of type double when "NaN" is indexed.

Steps to reproduce:

Index:

{
    "mappings": {
        "docs": {
            "dynamic": false,
            "properties": {
                "BaseRate": {
                    "type": "double"
                }
            }
        }
    }
}

Test document (one is enough):

        {
            "BaseRate": "NaN"
        }

Aggregation request:

{
    "query": {
        "match_all": {}
    },
    "aggregations": {
        "BaseRate": {
            "histogram": {
                "field": "BaseRate",
                "interval": "10"
            }
        }
    }
}

This causes:

[2017-09-25T18:07:20,671][WARN ][r.suppressed             ] path: /nan-histogram/docs/_search, params: {index=nan-histogram, type=docs}
org.elasticsearch.action.search.SearchPhaseExecutionException:
        at org.elasticsearch.action.search.AbstractSearchAsyncAction.onPhaseFailure(AbstractSearchAsyncAction.java:272) [elasticsearch-5.6.1.jar:5.6.1]
        at org.elasticsearch.action.search.FetchSearchPhase$1.onFailure(FetchSearchPhase.java:92) [elasticsearch-5.6.1.jar:5.6.1]
        at org.elasticsearch.common.util.concurrent.ThreadContext$ContextPreservingAbstractRunnable.onFailure(ThreadContext.java:623) [elasticsearch-5.6.1.jar:5.6.1]
        at org.elasticsearch.common.util.concurrent.AbstractRunnable.run(AbstractRunnable.java:39) [elasticsearch-5.6.1.jar:5.6.1]
        at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149) [?:1.8.0_141]
        at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) [?:1.8.0_141]
        at java.lang.Thread.run(Thread.java:748) [?:1.8.0_141]
Caused by: java.lang.IndexOutOfBoundsException: Index: 0, Size: 0
        at java.util.ArrayList.rangeCheck(ArrayList.java:653) ~[?:1.8.0_141]
        at java.util.ArrayList.get(ArrayList.java:429) ~[?:1.8.0_141]
        at org.elasticsearch.search.aggregations.bucket.histogram.InternalHistogram.reduceBuckets(InternalHistogram.java:274) ~[elasticsearch-5.6.1.jar:5.6.1]
        at org.elasticsearch.search.aggregations.bucket.histogram.InternalHistogram.doReduce(InternalHistogram.java:360) ~[elasticsearch-5.6.1.jar:5.6.1]
        at org.elasticsearch.search.aggregations.InternalAggregation.reduce(InternalAggregation.java:119) ~[elasticsearch-5.6.1.jar:5.6.1]
        at org.elasticsearch.search.aggregations.InternalAggregations.reduce(InternalAggregations.java:77) ~[elasticsearch-5.6.1.jar:5.6.1]
        at org.elasticsearch.action.search.SearchPhaseController.reduceAggs(SearchPhaseController.java:513) ~[elasticsearch-5.6.1.jar:5.6.1]
        at org.elasticsearch.action.search.SearchPhaseController.reducedQueryPhase(SearchPhaseController.java:490) ~[elasticsearch-5.6.1.jar:5.6.1]
        at org.elasticsearch.action.search.SearchPhaseController.reducedQueryPhase(SearchPhaseController.java:408) ~[elasticsearch-5.6.1.jar:5.6.1]
        at org.elasticsearch.action.search.SearchPhaseController$1.reduce(SearchPhaseController.java:725) ~[elasticsearch-5.6.1.jar:5.6.1]
        at org.elasticsearch.action.search.FetchSearchPhase.innerRun(FetchSearchPhase.java:102) ~[elasticsearch-5.6.1.jar:5.6.1]
        at org.elasticsearch.action.search.FetchSearchPhase.access$000(FetchSearchPhase.java:45) ~[elasticsearch-5.6.1.jar:5.6.1]
        at org.elasticsearch.action.search.FetchSearchPhase$1.doRun(FetchSearchPhase.java:87) ~[elasticsearch-5.6.1.jar:5.6.1]
        at org.elasticsearch.common.util.concurrent.ThreadContext$ContextPreservingAbstractRunnable.doRun(ThreadContext.java:638) ~[elasticsearch-5.6.1.jar:5.6.1]
        at org.elasticsearch.common.util.concurrent.AbstractRunnable.run(AbstractRunnable.java:37) ~[elasticsearch-5.6.1.jar:5.6.1]
        ... 3 more

Probable root cause:

In InternalHistogram.reduceBuckets, the loop compares top.current.key != key (line 272 in 5.6.1). Both variables are doubles, in this case with the value Double.NaN, and Nan != Nan.

[colings86]

This should be fixed in 6.0 by #25826 which ensures that out of range, infinite and NaN values are rejected at index time.

[thomas11]

And what about the invalid values that are already indexed? The fix for this can't be to re-index all data.

[thomas11]

Seems like changing from == to Double.compare(double a, double b) == 0 would do the trick.

There are two ways in which comparisons performed by this method differ from those performed by the Java language numerical comparison operators (<, <=, ==, >=, >) when applied to primitive double values:

• Double.NaN is considered by this method to be equal to itself and greater than all other double values (including Double.POSITIVE_INFINITY).
• 0.0d is considered by this method to be greater than -0.0d.

[jpountz]

And what about the invalid values that are already indexed?

This is unfortunate that we had this leniency in the past indeed. I'm afraid this histogram issue is only one of the issues you could hit with NaNs in your index. Using a range query with both bounds open should work in order to filter out nans in case you need it.

I see the switch to Double.compare as a good move regardless of this issue. Would you like to open a PR?

[thomas11]

Not sure if you're automatically notified @jpountz, I submitted PR 26856. Thanks!

[jpountz]

Fixed by #26856