feature/desired-balance-allocator remaining work

[idegtiarenko]

Description

Following tasks need to be completed before DesiredBalanceShardsAllocator could be merged to master. Please note this list is likely not complete.

• Enable CI on feature branch
• Use new allocation api in AutoCreateAction
• Use new allocation api in MetadataRolloverService
• Use new allocation api in TransportRolloverAction
• Use new allocation api in MetadataIndexStateService
• Use new allocation api in MetadataUpdateSettingsService
• Use new allocation api in DelayedAllocationService
• Use new allocation api in LocalAllocateDangledIndices
• Use new allocation api in RestoreService
• Add a listener to the AllocationService#disassociateDeadNodes
• Add a listener to the AllocationService#applyFailedShards
• Remove AllocationService#reroute implementation without listener
• Handle AllocationCommands: record and apply them during next balance calculation cycle
• revisit logging, make sure it could be used for real use case debugging (Revisit loggers and log levels for desired balance allocator #90440)
• check and remove unnecessary todo/comments/logs in the feature branch

Optional things or ones that could be implemented after merging back to the main branch

• Refactor DesiredBalanceShardsAllocator so that desired balance changed branch is not causing another balance compute cycle (Simplify reconciliation #89900)
• Implement test synchronous implementation
• Use above in existing ESAllocationTestCase tests (Fix failing tests on feature/desired-balance-allocator branch #86429)

Additional questions

• primary balancing: desired balance might assign all primaries to the same node in a small cluster -> ordering during reconciliation
• allocating unallocated shards has a higher priority to rebalancing. Are there situations when it is not possible to allocate new shard without rebalancing first? Related to this decider -> we should allocate shards somewhere in the cluster that could not be allocated according to the balance
• take into account disk space while simulating (simulate disk usage during balance calculation #90061)
• If the shard can not start (after relocation) then reconcile might infinitely retry that (Shard relocations are retried indefinitely #79445)
• handle ignored shards -> if shard is ignored but has shard.unassignedInfo.lastAllocatedNodeId set it should be treated as assigned to that node to prevent unwanted shard relocations (Initialize shards assuming they would be placed on original nodes #90600)

[elasticsearchmachine]

Pinging @elastic/es-distributed (Team:Distributed)