corrupted disk should be removed automatically if index couldn't be written in for a long time

[garyelephant]

If disk encounters disk error ,such as input/output err, this corrupted disk should be removed automatically from es node if index couldn't be written in for a long time.

Currently ops must update config and restart es node.

[2016-08-31 10:43:18,568][WARN ][gateway                  ] [10.148.67.11] [slb_itvresult-2016.08.28.05]: failed to write index state
java.nio.file.FileSystemException: /data/slot9/elasticsearch/es02/nodes/0/indices/slb_itvresult-2016.08.28.05: Input/output error
        at sun.nio.fs.UnixException.translateToIOException(UnixException.java:91)
        at sun.nio.fs.UnixException.rethrowAsIOException(UnixException.java:102)
        at sun.nio.fs.UnixException.rethrowAsIOException(UnixException.java:107)
        at sun.nio.fs.UnixFileSystemProvider.checkAccess(UnixFileSystemProvider.java:308)
        at java.nio.file.Files.createDirectories(Files.java:746)
        at org.elasticsearch.gateway.MetaDataStateFormat.write(MetaDataStateFormat.java:136)
        at org.elasticsearch.gateway.MetaStateService.writeIndex(MetaStateService.java:135)
        at org.elasticsearch.gateway.GatewayMetaState.clusterChanged(GatewayMetaState.java:160)
        at org.elasticsearch.gateway.Gateway.clusterChanged(Gateway.java:185)
        at org.elasticsearch.cluster.service.InternalClusterService.runTasksForExecutor(InternalClusterService.java:610)
        at org.elasticsearch.cluster.service.InternalClusterService$UpdateTask.run(InternalClusterService.java:772)
        at org.elasticsearch.common.util.concurrent.PrioritizedEsThreadPoolExecutor$TieBreakingPrioritizedRunnable.runAndClean(PrioritizedEsThreadPoolExecutor.java:231)
        at org.elasticsearch.common.util.concurrent.PrioritizedEsThreadPoolExecutor$TieBreakingPrioritizedRunnable.run(PrioritizedEsThreadPoolExecutor.java:194)
        at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
        at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
        at java.lang.Thread.run(Thread.java:745)

[clintongormley]

Duplicates of #18417 and #18279