[Core] Update V2 Autoscaler to support scheduling using Node labels and LabelSelector API

[ryanaoleary]

Why are these changes needed?

This PR updates the GCS state manager to append the labels set on a Node and retrieved from GCS to the dynamic_node_labels on the NodeState passed to the autoscaler. This PR also updates the V2 autoscaler to consider labels when calling _compute_score to schedule a ResourceRequest on a Node, assigning a higher score to a node type with labels that satisfy a LabelSelector.

Related issue number

#51564

Checks

• I've signed off every commit(by using the -s flag, i.e., git commit -s) in this PR.
• I've run scripts/format.sh to lint the changes in this PR.
• I've included any doc changes needed for https://docs.ray.io/en/master/.
  • I've added any new APIs to the API Reference. For example, if I added a
    method in Tune, I've added it in doc/source/tune/api/ under the
    corresponding .rst file.
• I've made sure the tests are passing. Note that there might be a few flaky tests, see the recent failures at https://flakey-tests.ray.io/
• Testing Strategy
  • Unit tests
  • Release tests
  • This PR is not tested :(

[ryanaoleary]

@MengjinYan I ended up implementing the logic to pass labels from GCS to the autoscaler and consider labels in the autoscaler when scheduling in the same PR because it made it easier to test the change.

[can-anyscale]

some tests are failing, could be related @ryanaoleary

[MengjinYan]

Looks like we should also add the logic to parse the LabelSelector from the GCS GetClusterStatusReply in autoscaler to populate the label selector information.

Also, is it possible to add end to end test with label selector?

[MengjinYan]

@rueian Can you take a look at the PR as well from autoscaler's perspective to see if we missed anything? Thanks!

[MengjinYan]

Nit: I think we should also change the _sort_resource_request logic in _try_schedule to add labels to the sorting mechanism, as there can resource requests with exact same resource requirements but different label selectors

[ryanaoleary]

Sounds good to me, done in 8cb1816. I had it sort by the length of the constraints of the first selector, similar to how we sort based on the length of the placement constraints.

[rueian]

There is already a repeated LabelSelectorConstraint field in the LabelSelector. Why do we need repeated here?

[ryanaoleary]

The LabelSelectorConstraints are the conditions associated with one label selector with AND semantics (i.e. we require a node to have labels accelerator-type="TPU" and market-type="SPOT" if those two respective constraints are specified). The repeated label selector is so that we can support fallback with label selectors (i.e. the user specifies two separate label selectors with their own constraint(s), and we fallback to the second one if the first is unsatisfiable).

[dayshah]

generally just minor nits, not really blockers

[dayshah]

are we using std sets/maps with this somewhere instead of absl ones. Would prefer sticking to absl and just defining the absl hash functions

[ryanaoleary]

We are using std::hash in task_spec.h here for the scheduling class descriptor. I think since we're using absl::flat_hash_map in that class I can just go ahead and switch it all to use absl hash functions instead.

[ryanaoleary]

Done in 9b3bbbe.