slurm: wait for all nodes being ready

[dpausp]

Implement ready_many similar to drain_many which waits until the desired state is reached, with a configurable timeout. If nodes failed to become ready, they are reported in the timeout exception. Usually we expect the nodes to become ready almost immediately so there shouldn't be a noticeable waiting period.

ready_many is now used by the all-nodes ready subcommand.

PL-131739

@flyingcircusio/release-managers

Release process

Impact:

Changelog:

• slurm: the fc-slurm all-nodes ready command which is used by our automated maintenance system now waits for the nodes to actually become ready by checking their reported state (PL-131739).

Security implications

• Security requirements defined? (WHERE)
  • availability: make sure that the node state reported by slurm matches the expected state
• Security requirements tested? (EVIDENCE)
  • checked in the test slurm cluster that ready all terminates properly when nodes are ready
  • automated test checks normal execution and timeout