query cache used memory calculating is not correct, which cause non-stopping old gc.

[xzer]

Describe the feature:

Elasticsearch version (bin/elasticsearch --version): 5.4.3

Plugins installed: []

JVM version (java -version): 1.8.0_131

OS version (uname -a if on a Unix-like system):Debian 3.16.43-2

Description of the problem including expected versus actual behavior:

We have a cluster with 6 nodes, and with 2gb query cache size configured: [indices.queries.cache.size: 2g], and with 31g jvm heap size, with 24g old gen configured.

We found that our used old gen size is keeping increasing and then finally fall into CMS gc continually, by memory sampling by jvisualvm, we found that there are over than 8M count of long[], and with size almost 10GB. at the first time, we guess there may be memory leak, which causes this, but after we clear our query cache as following:

curl -XPOST 'http://127.0.0.1:9200/_cache/clear?query=true&pretty'

after the one more time CMS gc, we noticed that out jvm heap got more about 10GB free memory than before.

and also, after clear query cache, we did memory sampling again, we got about 3M count of long[] and with size 2.6GB.

It seems that query cache is holding at least 7-8GB memory in fact even we configured it to 2GB.

(I am sorry that I input wrong count, but the memory usage was correct.)

[sudpaw]

I think indices.queries.cache.count takes precedence over the size. It is set to 10K by default. We have seen similar issues and reducing this has helped keeping the heap usage in check. I cannot say how much you should reduce this though. Try with some numbers to find out. This requires rebooting the node.

[xzer]

        "query_cache" : {
          "memory_size_in_bytes" : 2116055192,
          "total_count" : 116187745,
          "hit_count" : 22459516,
          "miss_count" : 93728229,
          "cache_size" : 113600,
          "cache_count" : 2341426,
          "evictions" : 2227826
        }

before we clear the query cache, we got stat as above (from one node), it has 2.3k count of cached objects, but it obviously has hit the memory size limit since it keeps the memory usage at 2GB, and the almost same stats on all nodes.

As you explained, if the count takes precedence, I think the memory usage will not keep at just 2GB level, or I misunderstand it?

[jpountz]

The thing is that the count of the setting is the number of unique queries that may be cached while the stats' count is the number of (query, segment) pairs that are cached, which is typically larger than the number of unique queries that exist in the cache.

I have seen that memory accounting issue recently as well and I can't think of a good way to fix it: we could fix some queries to report memory usage but the problem is that this information becomes hidden if a query is wrapped into another query, which makes the problem almost impossible to fix in the general case. My best guess would be to reduce the default value of indices.queries.cache.count in order to make this problem less likely to occur.

[xzer]

I think there are things to do:

• at least, the document should contain warning that asks user to check the real memory usage under the real work load

• Since the memory size cannot be calculated correctly, why we just get rid of the configuration item? A wrong calculated size will only mislead users and cause unexpected failure.

• Also, we can add a api such as '/_cache/analyze', which can dump the cache to a fake stream to calculate the real size of cached data. Then we can get the real ratio between the calculated size and real size, so that we can adjust the calculated size by the ratio. Further, we can even run the analyze in a fixed period automatically, and also, by sampling.

Finally, not only query cache size, but also request cache has the same issue. As spin-off of another issue we reported (#26943), after we clear the request cache, we got more about free 4g memory rather than the configured size of 2g.

[jpountz]

Thanks to #26949, https://issues.apache.org/jira/browse/LUCENE-8058 and more generally the fact that we now have the ability to disable caching on queries that use a lot of memory, things should be much better in Elasticsearch 6.2.