Support dynamic node role

[ylwu-amzn]

OpenSearch doesn't support ML node role currently, check DiscoveryNodeRole.java, no ML role defined.

We need dedicated ML node for ML plugin

We have released a new ML plugin ml-commons in 1.3 and we are planning to adding more models.

ML model generally consuming more resources, especially for training process. We are going to support bigger ML models which might require more resources and special hardware like GPU. As OpenSearch doesn’t support ML node, we dispatch ML task to data node only. That means if user want to train some big model, they need to scale up all data nodes which seems costly and not reasonable. If we can support dedicated ML node, user don’t need to scale up their data node at all, just need to configure a new ML node(with different settings, more powerful instance type) and add it to cluster. And we can separate resource usage better by running ML task on dedicated node which can reduce impact to other tasks like search/ingestion.

What changes we need to make

OpenSearch

• Add new node type ml in DiscoveryNodeRole.java
• Support configuring ML node role in opensearch.yml
• Security: should be ok as ML node communicate with other nodes via transport action.

[Updated solution]: We plan to support user configuring any node role name in opensearch.yml file. Node will read the unknown roles as dynamic role rather than break node start process. Dynamic role abbreviations will be the same with full name. To avoid confusion, we enforce the node role full name and abbreviation case-insensitive(must be lower-case). We will return all role's full name including both built-in and dynamic roles in a new field node.roles when user call _cat/nodes API. Check more details in discussions below.

ML plugin

• If ML node exists, dispatch ML task to ML node; otherwise, dispatch to data node. Support dedicated ml node ml-commons#79

[dblock]

I like the problem that is being solved here (some tasks may compete/work better on specialized hardware and provisioning only some nodes to be GPU-enabled vs. others not be GPU enabled is cheaper) and understand what we're trying to accomplish (send certain tasks to certain nodes).

My suggestion is to generalize this to avoid solving this problem for each type of workload.

This can be treated as a request routing problem. For example, I'd like to be able to direct all search requests to search-only nodes (separate read/write). In #2859 I'd like to route requests in a weighted round robin. In #2447 I'd like not to have to install plugins on all nodes, and thus isolate execution between plugins across different hardware nodes (same as only some nodes are "ml" nodes because they have the "ml" plugin installed).

I think the generic version of this is to 1) make node roles dynamic (vs. hard-coded in Java), 2) make nodes advertise roles at runtime and be able to drop/add roles without restarts, 3) allow plugins to register and advertise roles at runtime, 4) enable routing to nodes that support a set of roles.

It doesn't have to be an existing "role" concept, it could be a new "capability" or "tag" concept.

If we implement this generically, a node that has the ML plugin installed would advertise a "plugin:ml" role and requests can be routed to it. A node that has a specific kind of GPU could advertise "os:gpu" and "os:gpu:NVIDIA T4". Round robin routing could then be policy-based with potential fallback (e.g. prefer nodes with "os:gpu" at 2x weight). Will need a way to say that an ML workload must be routed to an ML-capable node, but some other workload has a preference of ML-capable node and can fallback to a non-ML-capable data node.

[penghuo]

Two questions,

1. Does ML node contains index data?
2. Do we consider term compute node instead of ml node. In my option, ML algorithms is a type of computation task. but we could schedule any computation task on compute node, e.g. OpeSearch pipeline aggregation could execute distributively on compute node.

[jainankitk]

Can you add some more details on ML node interaction with storage layer? More specifically:

How is the ML node handled from shard allocation perspective? If ML nodes don't have any shards like master, do they fetch data from other nodes while processing ML task? Or we temporarily relocate the shard to ML nodes?

[reta]

It doesn't have to be an existing "role" concept, it could be a new "capability" or "tag" concept.

I like the idea of introducing the capabilities (fe search, ingest, data, ...). In this case the built-in roles could be assembled out of capabilities (to support existing features) but the routing / placement policies could rely on capabilities instead. For some capabilities we still need the support in core (fe could node store data or not), but others could be added dynamically (like gpu, ssd, ...) at any time.

[ylwu-amzn]

@penghuo Thanks for the question/suggestion

1. Does ML node contains index data?

Node can have 1 or multiple node roles. If a node has only "ml" node role, it won't store index data (no shard on it). If a node has both "data" and "ml" node roles, shard will be allocated to it. Basically only allocate shard to node which has "data" node role.

2. Do we consider term compute node instead of ml node. In my option, ML algorithms is a type of computation task. but we could schedule any computation task on compute node, e.g. OpeSearch pipeline aggregation could execute distributively on compute node.

Good point. I think it's good to use a more generic role name like "computation", so any computation-intensive tasks can run on "computation" nodes.

[ylwu-amzn]

Can you add some more details on ML node interaction with storage layer? More specifically:

How is the ML node handled from shard allocation perspective? If ML nodes don't have any shards like master, do they fetch data from other nodes while processing ML task? Or we temporarily relocate the shard to ML nodes?

@jainankitk thanks, this is a good question.
We are not going to change the shard allocation part, at least for the short-term view. It's possible that we meet data transfer bottleneck in future, then we may make some improvement like move ML model closer to data(like run model on shard level) or move data closer to ML model (transfer data to ML node). But this is not one-way-door. We can make continuous improvement and the improvement could be in ML plugin rather than OpenSearch core, unless that improvement is general and will benefit other use cases.

[ylwu-amzn]

@reta @dblock thanks for your suggestion. Like this

It doesn't have to be an existing "role" concept, it could be a new "capability" or "tag" concept.

After reading all comments and brainstorming possible solutions, I think we have multiple options to support dedicated node to run specific tasks, and ML task is just one example, maybe we can support other special tasks like image processing.

The main reasons why we need dedicated nodes for specific tasks like ML are

1. ML task uses shared resource of data nodes which may impact core functions. We can limit the memory usage and thread pool of ML tasks, but still possible to impact core function like searching and indexing.
2. Costly to scale. If some ML tasks needs special hardware like GPU, we need to scale up all data nodes which is costly and unreasonable.

We may have these options to solve the problems. Welcome any more suggestions/options.

Option1: add new node role

OpenSearch doesn't support ML node role currently, check DiscoveryNodeRole.java , no ML role defined.
Add a new role for ML/computation-intensive tasks like “ml” or more generally “computation”. Prototype code change link . I started a multi-node cluster (1ML node, 2 data nodes, 3 master nodes) and it works well, all ML tasks routes to ML node and no shard allocated to ML node.

Pros:

1. Less effort. Just leverage current node role framework.
2. ML node doesn't has shards. ML node failure has no impact on cluster state.

Cons:

1. Node roles are configured in opensearch.yml file. User can’t change node role on the fly.
2. If we have a new task type which needs to run on its own dedicated node in future, we need to add a new node role. But that depends on how possible it looks that we are going to add many new task types and if the new task types can use the general "computation" node role.

Option 2: enhance current node role framework

We keep some reserved/predefined roles like master/data/ingest. Enhance current node role framework to support any custom role and can assign/change node role on the fly.

Need to consider some challenges:

1. If change ML node role to data node role, shard will be allocated to it. If change data node role to ML, shard will be relocated to other data nodes. Shard moving between nodes may cause cluster performance down or unstable state. We can add some limitation like user can’t change role between data role with other roles, or user can’t change from/to reserved roles like master/data.
2. Assume one node role is dedicated for some kind of task, for example “ML” node role for ML tasks, “image_processing” role is for processing images tasks. If we change a node role from “ML” to “image_processing”, the ML tasks on it must stop immediately and maybe migrate to other ML node. This flexibility is good to let user repurpose node efficiently, but also risky to stop some running tasks especially if the task doesn’t support migration to other node. So user should be careful about changing node roles.
3. We need to persist the node role information somewhere (like OpenSearch index, or some local state file), so node will pick up correct latest node role after restarting. Compared with the static opensearch.yml file, we need to pay more attention on data consistency.

Pros:

1. Support assigning/changing node role on the fly. User can repurpose node easily.

Cons

1. There are some challenges to solve.
2. User needs to pay attention when assign/change node role on the fly. More flexibility, also more risk.

Option 3: keep current node role framework and add new node tag/capability

We build a brand new concept: "capability" or "tag". For easier discussion, here I just use "tag".

This option doesn’t change node role, so it’s safe for shard allocation and cluster management. But still have same second&third challenge of Option2.

Pros:

1. No impact to current node role
2. Support assigning/changing node tags on the fly.

Cons:

1. If a cluster only has master or data nodes. Adding new node tag can’t separate ML task from data node, it still uses shared resources of data nodes.

Option 4: get rid of current node role framework, build brand new node tag/capability framework

This options looks similar to option2, just name role as "tag"/"capability". We may think what other feature we can build in this new framework.

Pros:

1. Flexible. We can build any new features in this new framework.

Cons:

1. BWC: support old node roles in previous versions.
2. Same challenges of Option2

My conclusion by comparing these options

After comparing these options, I think Option1 is the easiest, and Option2 and Option3 will support more use case like assigning/changing node role/tag on the fly, but also bring more challenges. Option4 is something similar to Option2, but we may have more flexibility to build new features.

So, I wound suggest follow Option1 for short-term goal. And we can support Option2 later so user have flexibility to change node roles easily with some limitation (fe, can't change data roles). I think Option3 and Option4 are some long term solution, we can follow these options if we have some user cases which can't be solved by node role framework.

[dblock]

Option 1 solves a super narrow use-case, and its main advantage is "less effort". Is the prototype misleading? It doesn't say how calls will be routed to that node. I imagine you'll modify ml-commons/k-nn to say "route this to an ML node" - do you have that code? What happens if I don't have any nodes in my cluster with such role? Will that node sit idle when it's not busy? Does introducing a new node type have other impact, such as upgrades?

Is there a simple version of option 2 where there's no API to change node roles on the fly, but you can define custom node roles in the node's opensearch.yml and have the ML plugin prefer routing requests to a node that has the "ml" role?

node.roles: [ data, ingest, ml, image_processing, fizzbuzz ]

[reta]

Is there a simple version of option 2 where there's no API to change node roles on the fly, but you can define custom node roles in the node's opensearch.yml and have the ML plugin prefer routing requests to a node that has the "ml" role?

I think this would be the simplest starting point, we could also bake in the "dynamic" nature of roles (on the fly) for all custom ones but not for builtin ones (those deemed "static").

[ylwu-amzn]

Thanks @dblock .

I imagine you'll modify ml-commons/k-nn to say "route this to an ML node" - do you have that code?

Yes, in my prototype we need to filter out ml nodes, then route request. This is the code change ylwu-amzn/ml-commons@6883925.

What happens if I don't have any nodes in my cluster with such role?

That depends on plugin's logic. In my prototype, if ml node doesn't exist, will route request to data nodes. This prototype just demonstrate we can route request to ml node only if it exists. We may change this later, like only route request to ml nodes, and will throw exception if no ml node.

Will that node sit idle when it's not busy?

Yes, that's true. We should explain this in our documentation. If user don't run heavy ML tasks, maybe just run it on data node is enough. We need more experiments to find out the best configuration.

Does introducing a new node type have other impact, such as upgrades?

When upgrade, we need to consider BWC for the data layer and code logic. For data layer, the shard only allocated on data nodes, should be ok. For code logic, each plugin should consider BWC for their own case especially for mixed cluster. For ml-commons, ML tasks on old version routes to data nodes, so old tasks still consumes shared resource on data nodes, but new tasks only routes to ml nodes, both tasks can run independently as we don't have distributed model run on both data and ml nodes.

Is there a simple version of option 2 where there's no API to change node roles on the fly, but you can define custom node roles in the node's opensearch.yml and have the ML plugin prefer routing requests to a node that has the "ml" role?

Agree with @reta , this is the simplest and we can add "dynamic" nature for custom roles later (this is not one-way-door). Actually this is what I mean in Option1, In my prototype, I configured ml roles in opensearch.yml file, like this

node.name: ml-node-1
node.roles: [ ml ]

And add ml node role in OpenSearch core, check code change in my repo ylwu-amzn@b0915f4