Prioritize recovery of system indices

[jaymode]

During startup, system indices should be prioritized over the recovery of normal indices/user data so that the stack may be fully operational once user data is recovered.

Going beyond that, system indices themselves may need to be prioritized in terms of recovery operations as some system components may depend on others. We may be able to reuse the order that exists on indices today in terms of prioritizing which system indices get recovered first.

[elasticmachine]

Pinging @elastic/es-distributed (:Distributed/Recovery)

[pugnascotia]

@jaymode Does this issue only seek to change the current recovery order? Do we also need to think about preventing non-system indices from beginning recovery until the system indices have completed?

It looks like PriorityComparator is the right place to adjust the recovery order. We'd probably need to look at implementing a new AllocationDecider sub-class if we need to do recovery blocking.

[jaymode]

@pugnascotia I think we should allow for concurrent system and non-system index recoveries.