Fail shard if IndexShard#storeStats runs into an IOException

[bleskes]

If that happens something is totally wrong. We currently just bubble up the exception to the stats caller.

Relates to #18279 (comment)

[elasticmachine]

Pinging @elastic/es-distributed

[clintongormley]

@bleskes won't we keep trying to allocate shards to the emptiest mount point (ie the one with the failure?) and almost never assign to the other (working) mount points?

[bleskes]

@clintongormley we will first try another node. If we end up retrying the node we will probably use the same mount point indeed. If it fails immediately, we will only try 5 times and stop. If the failure is more subtle we will probably rinse repeat later. I agree this is not ideal but this is consistent with how we deal with other failures. I think what you mean is a bigger problem - tracking node/path health and avoiding assigning shards to it in general. That one is a way bigger fish.

[evanvolgas]

Does #16745 address

we should check if we can write on the datapath before we allocate
mentioned here #18279 (comment)?

I don't think that other issue addresses what Simon's talking about there. The PR you linked is just a node launch start up check, is it not? That's not really the same thing as a path reachable check on shard allocation.

Inre

I think what you mean is a bigger problem - tracking node/path health and avoiding assigning shards to it in general. That one is a way bigger fish.

But isn't that exactly what #18279 is about? I'm actually a little unclear why you closed that ticket. Can you share your thinking a bit more and/or revisit #18279 and whether it makes sense to leave that ticket open as a high hanging fruit?

[milan15]

@bleskes As a beginner, if no one is working on this i want to try and take this. I hope it would help me with increased understanding and give me commit/review process experience.

[milan15]

@bleskes in storeStats, added line to fail shard with message and passing IOException reference. let me know if anything is wrong/missing here.

[bleskes]

@EvanV I responded on the other issue.

@milan15 the direction looks good. Can you open an PR and we can iterate there? I would love to have some testing but I wanted to first do some research to give you guidance. You'd need some kind of a mock directory implementation that throws exceptions. If you open a PR, I can offer help there if needed.

[milan15]

@bleskes i did it here #29078 . thanks for offering your help.