Detected Java Memory Leak on Google Cloud Datastore API 1.0.2 (the problem start from version 0.13.0-beta)

[suttiwat]

Before: I've used Google Cloud Datastore API/Java 0.8.0 (beta) since December 2016 on Google App Engine Flex/Java 8/Jetty it's perfect and work well.

After: I update Google Cloud Datastore Java API to 1.0.2 (GA) last week, I detected OOM-Killer kill java process frequently. And it's only one thing that I change. So I decide to roll-back to 0.8.0. Everything looks good as the same. No OOM-Killer kill the Java process anymore.

So just need to raise this issue for you all first.

Update as of 30 May 2017:

Today, I spend time to test this case to find which version start to have OOM (out of memory) problem. Now I can assure that google-cloud-datastore 0.12.0-beta is the LAST version that OOM that kill the Java process doesn't happened on my environment. 0.13.0 onward will create the OOM problem.

I observed 2 things during this OOM problems

1. In Java Runtime itself, If I checked the free memory (via Runtime.getRuntime().freeMemory()) It's show a lot of memory available in free.

2. But If I see at the OS-level by using top command. I observed that the java memory in resident (RES) growing and growing up and take 90% of memory in the system and then process kswapd0 take action and then I see OOM-killing in the log (via Google Cloud Console Log)

So the suspect for this case, it should be the library that allocated the native memory (not allocated from the JRE itself, because of #1 freeMemory() is plenty of free available area)

I've found this topic netty/netty#6221 talking about netty library. User updated to 4.1.7-Final this users got OOM. And this user also tested with netty 4.1.6, it's the last version that work fine. Which is the same version use in google-cloud-datastore 0.12.0-beta.

Hope this help you investigate the problem of the API soon. Thanks !!

[vam-google]

@suttiwat thanks for a very detailed description and deep investigation. We did recently conduct memory-usage profiling of our libraries and did not found any leaks, but that did not include possible native memory allocations. Also we did experience large memory consumption by netty library in our tests, so the case you are describing is likely.

[vam-google]

The corresponding netty memory leak issue https://github.com/netty/netty/issues/6221issue was fixed by netty/netty#6252 and should be present in netty starting from version netty-4.1.8.Final. google-cloud-java currently depends on a newer 4.1.11.Final.

Also specifically the google-cloud-datastore module does not depend on netty libraries anymore (though many other modules of google-cloud-java still do).

Closing the issue.