Adaptive Flops Partitioning Strategy

[DevEmilio96]

Issue: #284

Since I cannot test Exo on a real cluster, I have made the following assumptions, and I would like to know which ones are true and which are false:

1. Exo does not partition the model across multiple machines.
2. instead, all machines execute the entire model.
3. The partitioning strategy applies to inference partitioning, specifically its layers.
4. All intermediate states of the chat are stored on all machines.
5. It is not possible to reassign partitions during tensor processing.
6. Partitions can only be reassigned between prompts.
7. Compared to the model, inference computation requires much less memory. For example, if the model takes up 16GB,
   after each query the machines retain intermediate states ranging from around 100MB to 2GB, depending on the
   complexity of the query and the length of the output to be generated.
8. Inter-node latency is not a critical factor in a non-geographically distributed cluster.

AdaptiveFlopsPartitioningStrategy:

1. Initial partitioning is based on FLOPs (floating point operations) rather than memory.
2. After each prompt, partitions are reassigned based on the performance of the nodes.
3. It implements the update_node_performance method, which can be modified to include other relevant metrics.

[dtnewman]

How do the other nodes find out about this node's performance measurements?

[DevEmilio96]

thanks for the comment, I added a commit that should answer your question

[AlexCheema]

Closing as we will rework partitioning strategies.