Validate the racks

[sjeropkipruto]

Status: Work in progress 🚧

Description

we want to validate the singularity racks because adding new instances in racks that singularity doesn't know about causes scheduling issues

Issue: https://git.hubteam.com/HubSpot/PaaS-Run/issues/1102

Todos

• specify the valid racks in the singularity configuration
• implement the validation checks to confirm the racks are known by singularity
• testing
• rolling out

[ssalinas]

I think this is close, just needs a tweak. We likely still want to be able to schedule some things on the smaller racks we add, but don't want a singular under sized rack to limit placement among the others. I think the hard coded list should be what informs if we choose RACK_SATURATED or not, rather than rejecting a new/unseen rack altogether. Essentially this gives us a more hardcoded count of 'expectedRacks' versus 'validRacks' if that makes sense

[ssalinas]

Close, but this is still a bit too restrictive. This will essentially not let anything schedule on unexpected racks, but doesn't solve the issue that the number of racks will still make it think that something is saturated too early.

Check out the isRackOk method. That is where we want to hook in. There is a line in there like:

double numPerRack = numDesiredInstances / (double) rackManager.getNumActive();
    if (racksAccountedFor < rackManager.getNumActive()) {
...

The rackManager.getNumActive() is what gets us right now when we accidentally do something like add a single host in a new rack. Singularity RackManager then thinks there are 4 for example, but only 3 actually have sufficient capacity for scheduling. We want to do something like using the count of expectedRacks here vs rackManager.getNumActive() (probably falling back to rackManager in the case that the expectedRacks is empty/unspecified, which should be the default)

[ssalinas]

Looking good, if all tests are still passing let's give it a go in staging 👍

[WH77]

Verified that scheduling still works normally on staging, added a unit test just to be safe.

[ssalinas]

🚢