KPL Silently Swallows Exceptions from Deamon Threads

[jeremysears]

The Deamon class will swallow and not report any exceptions that are throw by its deamon threads. The deamon threads are initiated by running ExecutorService.submit(Runnable). The submit method returns a Future object. If the Runnable throws, its exception is added to the Future object. However, Deamon never looks at the Future objects, so any errors are never reported. Instead of calling ExecutorService.submit(Runnable), as currently written, this should probably call ExecutorService.execute(Runnable) instead. Changing this to use execute allows the Thread's UncaughtExceptionHandler or the global default UncaughtExceptionHandler to properly handle the failure. At a minimum this allows the error to be logged.

This situation could arise in several scenarios. For example, if a deamon thread runs into an OutOfMemoryError or if a RuntimeException thrown from a custom AWSCredentialsProvider used by the credential refresh thread.

In the event that one of these threads throws, the KPL will no longer operate correctly. Consequently, some restart logic, or a hard shutdown should probably be initiated in failure conditions.

[jeremysears]

You can reproduce this by hardcoding an exception in any of the Deamon's thread's run methods. The thread will abort, but nothing will be logged.

[jeremysears]

I've prepared a pull request which will at least allow any errors to be logged. I'll submit it once it passes our staging environment. However, I haven't addressed restart/shutdown logic in the pull request.

[pfifer]

Thanks for reporting this. I'm looking at your PR as well.

[jeremysears]

What is the ETA for merging the related PR?

[lipmice]

I'm running into this as well. It would be good to have these errors logged to flag the error and enable cleanup/restart.